Understanding excess mortality What is the fairest way to compare COVID-19 deaths internationally?

Introduction

Charts comparing COVID-19 deaths across countries are appearing daily in our newsfeeds. Done well, these international comparisons can help us to understand how different national strategies and policies have affected the spread and severity of COVID-19 outbreaks. But sometimes what is presented in these neat charts is not quite as straightforward as it seems, and can draw misleading conclusions. Here, we summarise some of the ways of comparing countries to help commentators make sense of data on deaths.

Understanding data on deaths

A simple place to start is to look at mortality in different countries via a pair of metrics – the total number of deaths and the deaths per million population (see Table 1). Each of these metrics gives a quite different picture of the countries performing ‘well’ or ‘poorly’. The USA, for example, moves from being ranked number one in total number of deaths, to number nine when adjusted per capita (with the caveat that US deaths do not yet seem to have stabilised). Belgium makes the reverse move, going from number six in overall deaths to being the country with the most deaths per million population. On this measure, the UK – following the recent inclusion of care home deaths – now has the third highest number of deaths in the world.

Table 1: Comparing highest ranked countries by total COVID-19 deaths and by deaths per million

Excess deaths: a better comparator?

While these metrics are intuitive and simple, using them as comparators is problematic for the reasons discussed shortly. Increasingly, therefore, commentators are turning to excess deaths to assess different countries’ COVID-19 mortality. This is typically used to count the deaths caused by an event – such as a very cold winter or a pandemic – that would not have occurred if the event had not happened. In this case, we use a measure that compares a country’s weekly deaths in 2020, with weekly deaths in previous years. In England and Wales (the only parts of the UK the ONS reports for), the ONS compares 2020 weekly deaths for all causes to the average of the same week in the previous 5 years.

Excess deaths is a better measure of the impact of COVID-19 on mortality, and a fairer way to make international comparisons. There are three reasons for this.

1. The total number of deaths reported across all causes (and therefore excess deaths) is not skewed by differences in defining and reporting COVID-19 deaths across countries. Some countries, including Italy, are only reporting COVID-19 deaths within hospitals while others, including France, are reporting deaths more widely from the community. Definitions also vary, with the UK broadly defining anyone who died with a positive COVID-19 diagnosis as having died because of it. Conversely, in Germany, even if an individual has COVID-19 when they die their death may not be attributed to it.
2. Measuring excess deaths allows us to understand the full impact of COVID-19 on mortality, including deaths indirectly related to COVID-19. For instance, in Belgium, on 2 days in March, 400 excess deaths were reported not directly attributable to COVID-19. These deaths may be linked to delays in seeking care for other conditions or lower capacity in the Belgian health care system  to deal with other conditions.
3. Excess deaths enables more valid international comparisons. It allows us to compare a country’s present with their own past, lessening concerns about different population age structures, reporting approaches, or health system capacity. A good comparison measure is the percentage increase of deaths in 2020, over deaths in previous years. EuroMOMO – a source frequently used by commentators – analyses excess deaths using z-scores. This is a measure of the increase relative to normal weekly fluctuations, rather than absolute change – but this is not as good for cross-country comparisons and is less intuitively understandable.

Will the UK have the worst excess mortality in Europe?

Looking at excess deaths data increases the apparent COVID-19 death toll for all countries, including England and Wales. In England and Wales, it shows deaths from the week commencing 21 March as between 10% (in the early stages of the outbreak) and 113% higher (at the peak) than the average of the previous 5 years.  

In addition, if we look at the percentage increase in mortality (Figure 2), not only is the England and Wales peak the joint highest with Spain, the increase remains high while in other countries it has fallen significantly. Recent commentators have suggested that the UK is heading for much higher excess deaths than other European nations – and will trail only the USA in terms of the impact of COVID-19. This is not yet a fair conclusion.

This is because different countries’ data is at different stages of completeness. The ONS, which stands out among national statistical agencies for the quality, frequency and timeliness of its reporting, has data to 24 April. Data from France’s statistical agency is only up to 20 April although corrections are expected for the most recent week. In Italy, it is only for a proportion of the municipalities up to the end of March and then for a smaller proportion of municipalities up to 15 April. Incomplete data is shown by dotted lines in the Figure 2 below. The apparent declines in these countries may turn out not to be ‘real’ – particularly in France and Sweden.

Applications and limitations of the excess deaths metric

The excess deaths measure can be used in several ways to shed light on different aspects of performance.

First, we use it to estimate the number of excess deaths in a given country, rather than simply the percentage increase (see Table 2). How many extra deaths does the curve for England and Wales suggest? Do we have more excess deaths than Italy? What does this tell us about the absolute scale, pace and impact of the outbreak? While this is easy to calculate from the weekly excess deaths figures (for England and Wales the figure is around 38,000), comparisons between countries, at this stage, are very tricky. This is because mortality data from countries is at different stages of being reported – as described above. Accurate comparisons of excess deaths will need to wait.

Second, we can look at the proportion of excess deaths attributable specifically to COVID-19. This partly gives us a sense of how comprehensive an individual country’s counting of COVID-19 mortality is. Of the 21,261 excess deaths in France, 92% were recorded as COVID-19 deaths. In England and Wales only 71% were being recorded as COVID-19 deaths. Germany, despite concerns that their tight definition of deaths attributable to COVID-19 was underestimating their mortality, is actually accounting for a similar proportion of their excess deaths as other European nations. In time, we should also begin to be able to unpick what is causing the unaccounted for deaths in different countries – whether these are ‘missed’ COVID-19 deaths, or if these are indirect deaths caused by overloaded health care systems, or delays in patients seeking care for other conditions.

Table 2: Excess mortality in selected European countries

*The start date is when we start counting excess mortality. We start when 2020 deaths are at least 5% above baseline. End date is the latest available data. COVID-19 deaths are also counted within this timeframe.

** Excess deaths are from a subset of municipalities reporting; this cannot be compared to  COVID-19 deaths which are for whole of the country

Making fair international comparisons

By any measure, the UK is one of the countries most affected by COVID-19 in the world. The UK has the third highest total number of COVID-19 deaths and fourth highest number of deaths per million population in the world. Looking at excess mortality – which helps us to include COVID-19 deaths that may not have occurred in hospitals – suggests we are doing worse than Italy, France and Germany, and comparably to Spain. But we have not reached the end of the crisis and data reporting across countries is at different stages. It is too soon to conclude what the ultimate totals will be.

International comparisons will continue to become more robust and insightful as our data improves. Analysis should therefore be updated – and new countries added – as data becomes available.

In making these international comparisons, researchers and journalists should bear two things in mind.

1. Excess deaths is the best way to compare COVID-19’s impact between countries. When using this metric percentage change above baseline, rather than z-scores, gives a more meaningful and understandable comparison. Be aware that different countries use different baselines, that more recent data is likely to be updated, and that countries are at different stages of their outbreak.
2. If comparing COVID-19 mortality rates between different countries, be aware that (a) some countries count only hospital-based deaths (b) some countries count both COVID-19 confirmed and suspected deaths, and (c) testing regimes differ between countries.