Comparison of pandemic excess mortality in 2020–2021 across different empirical calculations

Abstract

Different modeling approaches can be used to calculate excess deaths for the COVID-19 pandemic period. We compared 6 calculations of excess deaths (4 previously published [3 without age-adjustment] and two new ones that we performed with and without age-adjustment) for 2020–2021. With each approach, we calculated excess deaths metrics and the ratio R of excess deaths over recorded COVID-19 deaths. The main analysis focused on 33 high-income countries with weekly deaths in the Human Mortality Database (HMD at mortality.org) and reliable death registration. Secondary analyses compared calculations for other countries, whenever available. Across the 33 high-income countries, excess deaths were 2.0–2.8 million without age-adjustment, and 1.6–2.1 million with age-adjustment with large differences across countries. In our analyses after age-adjustment, 8 of 33 countries had no overall excess deaths; there was a death deficit in children; and 0.478 million (29.7%) of the excess deaths were in people <65 years old. In countries like France, Germany, Italy, and Spain excess death estimates differed 2 to 4-fold between highest and lowest figures. The R values’ range exceeded 0.3 in all 33 countries. In 16 of 33 countries, the range of R exceeded 1. In 25 of 33 countries some calculations suggest R > 1 (excess deaths exceeding COVID-19 deaths) while others suggest R < 1 (excess deaths smaller than COVID-19 deaths). Inferred data from 4 evaluations for 42 countries and from 3 evaluations for another 98 countries are very tenuous. Estimates of excess deaths are analysis-dependent and age-adjustment is important to consider. Excess deaths may be lower than previously calculated.

1. Introduction

Many studies estimate excess deaths in specific locations, countries, regions, or worldwide during the COVID-19 pandemic (Karlinski and Kobak, 2021; The Economist; COVID-19 Excess Mortality Collaborators, 2022; World Health Organization). Excess deaths reflect a composite of deaths from SARS-CoV-2 infection plus indirect effects of the pandemic (e.g. health system strain) and measures taken (Ioannidis, 2021; Kiang et al., 2020). It has been argued (Islam, 2022; Vandenbroucke, 2021) that excess deaths are a more appropriate measure of impact than recorded COVID-19 deaths. Recorded COVID-19 deaths may be under- or over-counted in different time periods and locations (Ioannidis, 2021) and do not capture indirect effects of the pandemic and the measures taken. However, excess deaths calculations require modeling of the expected deaths that entails many assumptions and analytical choices. To obtain excess deaths estimates one needs to define a control (reference) pre-pandemic period, use some model for extrapolating expected deaths in the pandemic period and compare them against observed deaths. There are many different possibilities on how to select the pre-pandemic reference period and on how to model data and extrapolations (Nepomuceno et al., 2022; Islam et al., 2021). A major analytical decision is whether to account for changes in the age structure of the population over time. With an aging population (particularly in high-income countries), mortality rates may increase over time, countering the anticipated decrease in mortality from better healthcare and overall human progress. These changes may be better addressed by considering age structure (Nepomuceno et al., 2022; Kowall et al., 2021; Gianicolo et al., 2021; Stang et al., 2020) rather than simply relying on regression trends of overall population data regardless of age.

Another pivotal dilemma in excess death calculations is what sources of data to use; and which countries are considered to have sufficiently reliable data. Death registration is sub-standard in most countries around the world: many deaths remain unrecorded (Adair and Lopez, 2018; Mikkelsen et al., 2015). Information on causes of death has flaws even in the most developed countries (D'Amico et al., 1999; Zellweger et al., 2019) while the pandemic generated new death coding challenges (Fedeli et al., 2021). Changes in deaths over time may be confounded by changes in death registration and recorded COVID-19 deaths depend on coding. Even population counts have uncertainty and this applies also to age-stratified estimates from different sources using different imputations to estimate age-stratified population for recent years. Even in high-income countries, most of them have not had a formal census performed for many years. Finally, several highly visible studies that attempted to calculate excess mortality world-wide (Karlinski and Kobak, 2021; The Economist; COVID-19 Excess Mortality Collaborators, 2022; World Health Organization), first estimated excess mortality in countries with trustworthy death data on all-cause mortality; then, they extrapolated across other countries worldwide, using the profile of various characteristics in these countries versus those with trustworthy data. Consequently, proper calculation of excess mortality in countries with most trustworthy mortality data has critical importance even for worldwide estimates.

Here, we compare the results of different evaluations that have attempted to calculate global excess mortality during the COVID-19 pandemic in 2020–2021 (Karlinski and Kobak, 2021; The Economist; COVID-19 Excess Mortality Collaborators, 2022; World Health Organization). We focus primarily on high-income countries with the most reliable death registration systems and discuss the implications for extrapolations to a global level. We compared 4 widely publicized excess death calculations published in eLife, Economist, the Lancet, and by the World Health Organization (WHO) (Karlinski and Kobak, 2021; The Economist; COVID-19 Excess Mortality Collaborators, 2022; World Health Organization; Estimated excess death count based, 2021; Estimated excess death count from, 2021) and included also our calculations. Our calculations explicitly explored what difference it would make to model deaths using separate death and population data for age strata. Besides raw excess mortality estimates, we focused on the ratio of excess mortality over recorded COVID-19 deaths. This ratio is critical in understanding whether excess mortality confers more information compared with just focusing on routinely recorded COVID-19 deaths. Ideally, one would like to see consistency in this ratio regardless of modeling and analytical choices, while large inconsistency would put the added value of excess mortality calculations in question.

2. Materials and methods

2.1. Compared published excess mortality estimates

We considered four previously published pandemic excess mortality evaluations (Karlinski and Kobak, 2021; The Economist; COVID-19 Excess Mortality Collaborators, 2022; World Health Organization; Fedeli et al., 2021; Estimated excess death count based, 2021) that have calculated both country-specific and global estimates and which use diverse methods to extrapolate from pre-pandemic reference periods to the pandemic period.

Karlinsky and Kobak published their evaluation in eLife in mid-2021 (Karlinski and Kobak, 2021); the Economist team released their estimates in late 2021 (The Economist); the COVID-19 Excess Mortality Collaborators published their estimates in the Lancet in early 2022 considering the two-year period 2020–2021 (COVID-19 Excess Mortality Collaborators, 2022); and WHO released in May 2022 its updated estimates covering the same two year period 2020–2021. We call these four evaluations for convenience eLife, Economist, Lancet, and WHO respectively. Both the eLife and Economist models allow updating of excess mortality estimates over time and we have used the Our World in Data resource (Fedeli et al., 2021; Estimated excess death count based, 2021) that includes such updates. We used the estimates of excess mortality for the 2020–2021 two-year period for all 4 evaluations to maximize comparability. The three evaluations used also different sources for capturing the recorded numbers of COVID-19 deaths, which resulted in mostly minor discrepancies. Again, to maximize comparability, we used the same set of recorded numbers of COVID-19 for comparing against the excess mortality estimates of each evaluation: this set is identical to the set used by the Lancet evaluation, with the exception of Spain and UK where the Lancet numbers of recorded deaths were too high by 10% and 16%, respectively (perhaps due to clerical error) and where we used the Johns Hopkins data instead (Johns Hopkins CoronaVirus Resource Center).

The reported cumulative counts of COVID-19 deaths are taken from the Johns Hopkins Repository as reported by Our World in Data. The same values are reported by two of the four methods we use: eLife and Economist. Lancet reported values in https://ghdx.healthdata.org/sites/default/files/record-attached-files/IHME_EM_COVID_19_2020_2021_DATA_Y2022M03D10.CSV are sometimes quite different. Specifically for Spain, United Kingdom, Kazakhstan, Mexico, Russia, Georgia & Tajikistan where discrepancies are over 10% and some times as large as 115% (Russia is 651,000 in Lancet and 302,671 in OWID).

For each of the 4 evaluations, we extracted information on the following methodological features: reference period selected; modeling of reference period (static average, linear, spline, Poisson seasonality, other); exclusion of heat waves, wars, natural disasters, other; unit of modeling data (week, month, quarter, other); pandemic time period covered in the original publication/release; source of data for all-cause mortality; source of data for COVID-19 deaths used in original analysis; age and/or gender adjustment in calculations of excess deaths (if yes, how); any other adjustment in the calculations (if yes, specify); eligibility criteria and number of countries modeled directly; eligibility criteria and number of countries inferred from the directly modeled countries; and how these were inferred. Details on the data sources and modeling methods for these 4 evaluations can be found in references Karlinski and Kobak (2021), The Economist, COVID-19 Excess Mortality Collaborators (2022), World Health Organization, Fedeli et al. (2021) and Karlinsky and Kobak methods.

2.2. Evaluation considering age strata in the calculations

We performed also our own calculation of excess deaths focused on considering the impact of age-adjustment on the calculations. Age-adjustments have a long tradition in demography when comparing mortality across different regions with different age-structure (see e.g. references Kitagawa (1964) and Keiding and Clayton (2014)) and with changing (e.g. aging) populations, they are essential in computing excess mortality. We considered three pre-pandemic years (2017–2019) as the reference period. We used the following age strata: 0–14, 15–64, 65–75, 75–85, and >85 years old. For each age stratum, we obtained the average mortality, the number of deaths per million for the population of the specific age stratum. Then we extrapolated to the two pandemic years, again correcting for the population size in the specific age stratum. Finally, expected deaths were summed across the population strata. An illustrative worked example of the age-adjusted calculations is shown in Fig. 1 using the data for Germany. This analytical approach corrects for changes in total population of a country over time, as well as changes in the proportion of elderly people. If the total population and/or the proportion of older people is increasing in recent years over time, the expected deaths by the age-adjusted scheme will be more compared to analyses that do not consider population changes and age-stratification. The inverse will happen, if the total population and/or the proportion of older people is decreasing in recent years over time. We used the Human Mortality Database (https://www.mortality.org), (Dong et al., 2020) specifically the Short Term Mortality Fluctuations file (https://www.mortality.org/Public/STMF/Outputs/stmf.csv) that includes weekly all-cause deaths and mortality values allowing the population size to be calculated as their ratio for each week.

Fig. 1.

Schematic Representation of the Calculation of Age Adjusted Excess Death. Populations and total amounts of death on a single year are computed as sum of weekly data and used to obtain the mortality values for the reference years (2017–2019) for each of the age strata. The average of this value is taken as the reference mortality for that strata. Expected deaths for a non COVID scenario for 2020 and 2021 are obtained from the population and reference mortality data for each strata. Excess deaths are calculated as the difference between the actual deaths and the expected deaths. Expected and excess deaths for the non-age adjusted case are also reported for comparison. The table reports the real values for Germany as an example. This table is provided as Excel in the Supplement so that method can be easily used on other data.

Some clarifications are required for these analyses. First, on the calculation of the Weekly Population from the weekly Death Count and Weekly Mortality given in the MDH file stmf.csv. As Population = Death/Mortality, when Death = 0, Mortality = 0, so that Population is not defined. In these cases, we get the Population value from adjacent weeks with non-zero Death in the same year. Another problem is that some years, like 2015 & 2020 have an extra week 53, a leap week. This is because 52 weeks is 364 days whereas an average year is 365.25 days (a leap year every fourth year adds an extra day). Since mortality data are reported weekly, these years would get an apparent extra week of deaths. To correct this issue, we consider a standard year to be 365.25/7 = 52.1786 weeks and distribute weekly counts uniformly into years. Thus 2017 gets all its 52 week plus 0.1786 of week 1 of 2018. Year 2018 loses 0.1786 of week 1 and needs to get an additional 2 × 0.1786 of week 1 of 2019, and so on. In this way all the five years 2017–2021 are adjusted to have 52.1786 weeks.

We averaged the mortality values for each age-stratum over the reference years. We did not exclude any deaths or periods due to heat waves or other natural or man-made events, since it is subjective to arbitrate which ones should be excluded; moreover, we wanted to compare the COVID-19 pandemic with three recent years that had not raised any concerns about undue levels of deaths. We also repeated the excess death calculations using the same exact process but without considering age-strata.

We used population data, including also age-strata, from mortality.org (Wilmoth et al., 2007; Jdanov et al., 2020) for consistency across all evaluations in calculating excess deaths per million even though different evaluations had used different sources originally, e.g. WHO had used population data from the World Population Prospects 2019 with projections (United Nations, 2019).

2.3. Countries considered in main comparison

We focused our primary comparison on countries that have excellent death registration, are high income, and include data with weekly deaths in the Human Mortality Database. These countries have the most reliable evidence to allow proper excess death calculations and changes in deaths over time cannot be due to changes in death registration (e.g. either improvements in death registration during the pandemic as countries put more resources on capturing information or worsening of death registration during the pandemic due to the pandemonium and acute death peaks). We defined high-income countries by World Bank criteria (The World by Income and Region). The Short Term Mortality Fluctuations file in the Human Mortality Database includes detailed weekly deaths for the period 2017–2019 and also for the pandemic years 2020–2021 on 35 countries, all of which except Bulgaria and Russia are high-income, thus 33 countries were included in the main analyses. Of note, Canada and Australia did not have data for the entire 2 years of 2020–2021 and we performed calculations that cover the available time periods.

2.4. Excess death metrics of interest

We calculated excess deaths for the full two years 2020–2021 for each country, E. We expressed them also as percentage above the expected deaths, E_p and as excess deaths per million E_m. E.g. if 60,000 deaths happened in 2020–2021 in one country of 4,000,000 people and 50,000 were expected, the percentage E_P is (60,000–50,000)/50,000 = 12% and E_m is (60,000–50,000)/4,000,000 = 2500 per million. As the main metric of interest, we used the ratio of estimated excess mortality E divided by the number of officially recorded COVID-19 deaths during 2020–2021, hence called the excess ratio R. E.g. R = 1.20 means that the estimated excess mortality is 20% higher than the officially recorded COVID-19 deaths and R = 0.90 means that the estimated excess mortality is 10% lower than the officially recorded COVID-19 deaths.

2.5. Analyses

We calculated the total number of excess deaths across the eligible countries with each of the different calculation methods, and as compared with the total recorded COVID-19 deaths. We also calculated the Pearson correlation coefficients of excess death metrics across the eligible countries based on the different calculation methods (a full list of correlation coefficient estimates appears in Supplementary Table 5).

We noted in how many of the 33 eligible countries the different calculations of excess deaths had consistently R above 1 or R below 1 (i.e. they all agreed that excess deaths were more than the recorded COVID-19 deaths or they all agreed that excess deaths were less than the recorded COVID-19 deaths); and in how many countries had R values that were all within a range of 0.3 between the highest and lowest estimate (a higher range means that divergence in the excess death estimates exceeds 30% of the recorded COVID-19 deaths); and within a range of 1 (a higher range means that the divergence in the excess death estimates exceeds the number of recorded COVID-19 deaths itself).

For the remaining, non-eligible countries, we evaluated excess death metrics and their comparison across different evaluations as exploratory analyses.

All analyses were done independently by two analysts (ML and FZ) and then compared notes with arbitration (including discussion with the third author, JPAI) for any disagreements until both analysts obtained the same results.

3. Data availability

All data are in the manuscript, tables, and supplementary tables and in the publicly available databases listed in Supplementary Links to Data.

4. Results

4.1. Characteristics of the compared excess death calculations

Table 1 summarizes the main features of the compared excess death calculations. As shown, the evaluations differed substantially in defining the reference period, the choice of analytical model, exclusions, use of adjustments, eligibility criteria, data sources, extrapolations and imputations.

Table 1.

Main features of the construction of the compared evaluations of excess deaths.

	eLife	Lancet	Economist	WHO	Levitt
Reference period years	2015–2019	2010 (or earliest available)-February 2020	Unclear, not mentioned	2015–2019 (countries with monthly historical data); 2000–2019 (country with annual historical data)	2017–2019
Modeling of reference period	Linear fit	Ensemble of 6 models (weighted): 4 using splines with different placement of the last knot, one Poisson, and one taking 2019 only	Machine learning. Mix of boosted Gradient, Random Forest and Bootstrapping.	Sum of an annual trend (thin-plated spline) and a within-year seasonal variation (cyclic cubic spline)	Static average
Exclusions	Heat waves	Heat waves	Unclear, not mentioned	Not mentioned	No
Time unit of modeling data	Weekly (preferred) or monthly or quarterly	Weekly or monthly	Weekly for most, some monthly	Monthly	Weekly
Pandemic time period covered in the original publication/release	Varies per country, mostly 2020 to mid-2021, exact start in 2020 depends on availability of weekly (week 10), monthly (March), or quarterly (January) data	2020–2021 (acknowledged potential problem with late registration for last weeks/months)	2020 to late 2021	2020–2021 (had also released early estimates for 2020)	2020–2021
Pandemic time period covered in the current comparative analysis	2020–2021	2020–2021	2020–2021	2020–2021	2020–2021
Source of data for all-cause mortality	Human Mortality Database, others	World Mortality Database, Human Mortality Database, European Statistical Office	World Mortality Database, Human Mortality Database, others	Eurostat, Human Mortality Database, World Mortality Database	Human Mortality Database
Source of data for COVID-19 deaths used in original paper	Johns Hopkins	Apparently Johns Hopkins (although too high for Spain and UK)	Unclear	Not used	Johns Hopkins
Source of data for COVID-19 deaths used in the current comparative analysis	Johns Hopkins	Johns Hopkins	Johns Hopkins		Johns Hopkins
Age adjustment	No	No (authors stated that they may adjust for age in future work)	No	Yes (excess deaths summed across 7 age strata)	Yes (excess deaths summed across 5 age strata), also done without age-adjustment
Gender adjustment in calculations	No	No	No	Yes	No
Any other adjustment	No	Under-registration corrected for countries with <95% death registration	Probably no (unclear)	No	No
Eligibility criteria for countries modeled directly	Weekly, monthly or quarterly data available for at least one pre-pandemic year and for pandemic period	Weekly or monthly data available for any pre-pandemic years and for pandemic period	Data availability (unclear about details)	Data availability (Age and sex specific death for 2020 aggregated to 5-year age bands), excluding the countries that have experienced conflict, small population numbers, incomplete deaths and/or erratic/implausible age-pattern	Weekly data available in Human Mortality Database from 2017 onwards
Number of countries modeled directly	103 in the publication. 77 with data to December 2021	74 countries and territories in the publication	78 countries apparently had mortality data, but it seems that all countries were included in the machine learning	50	36
Eligibility criteria for countries inferred from the directly modeled countries	None	All countries considered	Unclear	All countries. All data for 2021 were inferred	None
Number of countries inferred from the directly modeled countries	None	Remaining world	Remaining world	Remaining world	None
How were they inferred?	Not applicable	LASSO regression, selected 15 covariates related to pandemic (e.g. seroprevalence) and to background population health metrics (e.g. Healthcare Access and Quality Index)	Machine learning as above; totally impossible to reproduce based on thinly presented information, 121 indicators considered	K-mean clustering. Countries are divided into 5 clusters with different values of, Human Development Index Mean age at death, Crude excess rate	Not Applicable

4.2. Comparison of excess deaths metrics in 33 countries with most-reliable data

33 countries were included in the main analysis, selected because they were high-income according to the World Bank, and they had detailed available weekly data on observed deaths in the three pre-pandemic years and also during 2020–2021 (Table 2 ). These countries had a total population of almost 1 billion and almost 20 million deaths in 2020–2021, of which 9.5% were recorded as COVID-19 deaths. The total of calculated excess deaths in these 33 countries ranged from 2.0 million (eLife) to 2.8 million (Lancet), with WHO (2.1 million) and Economist (2.2 million) being closer to eLife. Our own calculations without age adjustment gave a similar total (2.3 million), but with age-adjustment the excess deaths were only 1.6 million. The estimated total excess mortality across the 33 countries with our age-adjusted analyses was 810,516 deaths for 2020 and 799,346 for 2021 (Supplementary Table 1).

Table 2.

Excess death estimates for 2020–2021 according to 6 evaluations in the 33 eligible countries*.

Country	2021 Population (millions) from HMD**	Two Years Actual All-Cause Death from HMD	Two Year Excess Death per eLife	Two Year Excess Death per Economist	Two Year Excess Death per Lancet	Two Year Excess Death per WHO	Two Year Excess Death per Levitt Age-Adjusted	Two Year Excess Death per Levitt Not Age-Adjusted
Australia	24.547	277,603	−11,639	−9500	−18,100	−14,258	−14,460	−2116
Austria	8.935	180,363	15,261	16,877	18,300	11,941	13,007	15,343
Belgium	11.494	239,201	20,613	23,364	32,800	17,919	13,958	19,036
Canada	36.108	586,135	13,474	23,548	43,700	22,018	21,829	37,938
Chile	17.960	263,154	38,894	38,094	37,200	38,698	31,640	45,021
Croatia	4.051	119,871	16,826	19,186	22,900	17,178	12,205	16,050
Czechia	10.730	269,137	41,480	43,942	49,100	37,040	34,079	43,262
Denmark	5.864	111,772	913	2453	10,400	3716	−3157	2390
Estonia	1.332	34,559	3172	3774	5630	3374	2675	3346
Finland	5.548	112,800	2662	4469	8780	2858	−716	4345
France	65.467	1,297,407	78,910	97,390	155,000	81,849	57,767	96,831
Germany	82.533	2,005,701	88,446	113,242	203,000	194,987	54,740	128,557
Greece	10.711	274,725	24,177	25,269	25,400	19,394	20,515	29,551
Hungary	9.762	296,496	35,811	41,714	53,800	36,499	27,813	36,090
Iceland	0.362	4640	50	−35	−314	−10	−142	11
Israel	9.293	99,437	7203	7967	9280	6178	3201	5421
Italy	59.630	1,454,193	167,816	190,872	259,000	160,800	115,690	166,373
Latvia	1.906	63,088	6979	7851	12,400	7668	6046	7023
Lithuania	2.802	90,523	16,008	17,396	20,000	17,253	11,283	12,274
Luxembourg	0.635	9106	57	314	1070	69	109	171
Netherlands	17.479	339,242	28,495	33,017	45,500	29,213	17,969	32,020
New Zealand	5.013	67,586	−2787	−2566	−872	−2678	−4118	−1826
Norway	5.408	82,491	1101	1986	742	−100	−2994	−182
Poland	38.482	998,284	157,247	171,806	214,000	157,531	149,722	182,454
Portugal	10.323	248,658	20,677	24,530	40,400	20,449	16,286	25,602
Slovakia	5.480	131,782	24,131	25,538	25,400	24,320	18,662	23,786
Slovenia	2.103	47,090	4953	5492	6980	5584	3944	5617
South Korea	51.631	621,862	7529	6967	4630	6289	−30,286	33,417
Spain	47.511	948,016	102,991	115,685	162,000	103,935	68,720	95,964
Sweden	10.408	184,326	9926	11,976	18,100	11,253	−367	3666
Switzerland	8.688	146,969	11,394	13,539	15,500	8247	5640	10,139
United Kingdom	67.145	1,353,941	136,795	148,889	169,000	148,896	87,307	125,716
United States	329.995	6,849,500	961,032	1,017,655	1,130,000	932,460	871,295	1,116,088
TOTALS	969.336	19,809,658	2,030,597	2,242,701	2,780,726	2,110,570	1,609,862	2,319,376

HMD, Human Mortality Database short-term mortality fluctuation file stmf.csv downloaded from https://www.mortality.org/Public/STMF/Outputs/stmf.csv on 1-May-2022.

Data is given for full two-year period 1-Jan-2020 to 31-Dec-2021 for all countries except for Australia to 2021 week 48 and Canada to 2021 week 48. Because we use a standard year of 365.25 days (52.1786 weeks), the two years 2020 & 2021 are 104.357 weeks. The HMD Total Deaths, Expected Deaths and Excess Deaths for Australia and Canada are all smaller than they would be if data for these two locations were no delayed.

Data in the HMD are summed over weeks available. This means that both the Total Deaths and Population are incomplete for Australia (to week 47) and Canada (to week 48).

There were very large differences across countries and this was evident also when excess deaths were estimated as a proportion above the observed deaths, E_p (Supplementary Table 2), deaths per million population, E_m (Supplementary Table 3) and ratio of excess deaths over recorded COVID-19 deaths, R (Table 3 ) metrics. Australia, New Zealand, and in some analyses also Iceland showed no excess deaths at all, even without age-adjustment. With age-adjustment, however, in our calculations several other countries such as Norway, Finland, Denmark, Sweden, and South Korea also showed no excess deaths. In the WHO age-adjusted calculations, no excess deaths were seen in Australia, New Zealand, Iceland, and Norway. Among countries with excess deaths in all analyses, the excess death estimate differed by 2–4 times between the highest (non-age-adjusted) and lowest (age-adjusted) estimates in countries like France, Germany, Italy, and Spain. Age-adjustment led to modest reductions of estimates for the USA.

Table 3.

Ratio of excess deaths over recorded COVID-19 deaths for 33 countries.

Country	Reported COVID-19 Deaths from OWID	Excess Death/Reported per eLife	Excess Death/Reported Economist	Excess Death/Reported per Lancet	Excess Death/Reported per WHO	Excess Death/Reported Age-Adjusted	Excess Death/Reported per Levitt Not Age-Adjusted
Australia	2253	−5.17	−4.22	−8.03	−6.33	−6.42	−0.94
Austria	13,733	1.11	1.23	1.33	0.87	0.95	1.12
Belgium	28,331	0.73	0.82	1.16	0.63	0.49	0.67
Canada	30,570	0.44	0.77	1.43	0.72	0.71	1.24
Chile	39,115	0.99	0.97	0.95	0.99	0.81	1.15
Croatia	12,538	1.34	1.53	1.83	1.37	0.97	1.28
Czechia	36,129	1.15	1.22	1.36	1.03	0.94	1.20
Denmark	3267	0.28	0.75	3.18	1.14	−0.97	0.73
Estonia	1932	1.64	1.95	2.91	1.75	1.38	1.73
Finland	1714	1.55	2.61	5.12	1.67	−0.42	2.53
France	123,805	0.64	0.79	1.25	0.66	0.47	0.78
Germany	111,925	0.79	1.01	1.81	1.74	0.49	1.15
Greece	20,790	1.16	1.22	1.22	0.93	0.99	1.42
Hungary	39,186	0.91	1.06	1.37	0.93	0.71	0.92
Iceland	37	1.35	−0.95	−8.49	−0.27	−3.82	0.29
Israel	8243	0.87	0.97	1.13	0.75	0.39	0.66
Italy	137,402	1.22	1.39	1.88	1.17	0.84	1.21
Latvia	4570	1.53	1.72	2.71	1.68	1.32	1.54
Lithuania	7387	2.17	2.35	2.71	2.34	1.53	1.66
Luxembourg	915	0.06	0.34	1.17	0.08	0.12	0.19
Netherlands	20,999	1.36	1.57	2.17	1.39	0.86	1.52
New Zealand	51	−54.65	−50.31	−17.10	−52.51	−80.75	−35.81
Norway	1305	0.84	1.52	0.57	−0.08	−2.29	−0.14
Poland	97,054	1.62	1.77	2.20	1.62	1.54	1.88
Portugal	18,955	1.09	1.29	2.13	1.08	0.86	1.35
Slovakia	16,635	1.45	1.54	1.53	1.46	1.12	1.43
Slovenia	5589	0.89	0.98	1.25	1.00	0.71	1.01
South Korea	5625	1.34	1.24	0.82	1.12	−5.38	5.94
Spain	89,405	1.15	1.29	1.81	1.16	0.77	1.07
Sweden	15,310	0.65	0.78	1.18	0.74	−0.02	0.24
Switzerland	12,217	0.93	1.11	1.27	0.68	0.46	0.83
United Kingdom	148,737	0.92	1.00	1.14	1.00	0.59	0.85
United States	827,887	1.16	1.23	1.36	1.13	1.05	1.35
TOTAL OR MEDIANS	1,883,611	1.09	1.22	1.36	1.00	0.71	1.15

OWID refers to Our World in Data master COVID-19 file down loaded from https://covid.ourworldindata.org/data/owid-covid-data.csv on 22-Apr-2022.

The pairwise correlations between eLife, Economist, WHO and our two calculations (with and without age-adjustments) were consistently extremely high for R (all r ≥ 0.977). The same picture was seen largely for these 5 calculations for E_p (all r ≥ 0.930) and for E_m (all r ≥ 0.951). The Lancet calculations had modestly lower correlation with the other 4 evaluations (range r = 0.808–0.841 for R, range r = 0.835–0.919 for E_p, 0.917–0.958 for E_m). The correlations for R between different evaluations, however, were modest/poor when Australia and New Zealand (that were outliers with very negative values of R) were excluded, (range −0.386 to 0.657). The two other metrics E_p (range, 0.794–0.989) and E_m (range 0.904–0.996) show good correlation values even when these outliers are excluded.

Even when correlations were high, given the substantial differences in the absolute estimates of excess deaths with the different calculations in each country, the range of R values was always large (Table 3). The R values’ range always exceeded 0.3 in all 33 countries. In 16 of 33 countries, the range of R across different evaluations exceeded 1, i.e. it was as large as the number of recorded COVID-19 deaths itself. 6 countries had consistently R > 1, i.e. more excess deaths than recorded COVID-19 deaths, with all empirical evaluations. Conversely, only 2 countries, Australia and New Zealand had consistently R < 1. The large majority of countries (25 of 33) had some calculations suggesting R > 1 (i.e. excess deaths greater than COVID-19 deaths) and others suggesting R < 1 (excess deaths smaller than COVID-19 deaths).

Table 4 shows the break-down of excess deaths per age stratum in each country for our age-adjusted analyses. In total, 0.478 of the 1.609 million excess deaths were in people <65 years old (29.7%), but the percentage varied widely across countries. 30 of 33 countries had death deficit for children 0–14 years old (all, except for Iceland, Luxembourg, and Netherland that also had minimal excess deaths in children) and overall across all 33 countries there was a death deficit of 7737 deaths for children 0–14 years old. 10 countries had death deficit even for people <65 years old. Conversely, for 4 countries, more than 25% of the excess deaths was in people <65 years old (Canada 45.5%, USA 41.6%, Chile 37.8%, UK 28.7%).

Table 4.

Excess deaths per age strata in the 33 countries of the main analysis

Country	Mean HMD 2020 & 2021 Population (millions)	Excess deaths in 0–14 years	Excess deaths in 15–64 years	Excess deaths in 65–74 years	Excess deaths in 75–84 years	Excess deaths in >85 years	Excess death for all ages	Percentage of excess deaths <65 years old
Australia	24.547	−151	−1196	−1802	−5485	−5825	−14,460	No excess
Austria	8.935	−55	1812	1449	5673	4129	13,007	13.5%
Belgium	11.494	−307	675	2707	4071	6812	13,958	2.6%
Canada	36.108	1113	8894	4429	4558	2835	21,829	45.8%
Chile	17.960	−857	12,810	7551	7592	4545	31,640	37.8%
Croatia	4.051	−25	1192	3639	4196	3203	12,206	9.6%
Czechia	10.730	−144	4261	9906	11,614	8442	34,079	12.1%
Denmark	5.864	−21	−882	−194	−1643	−417	−3157	No excess
Estonia	1.332	−9	530	538	741	875	2675	19.5%
Finland	5.548	−35	−350	308	−652	12	−716	No excess
France	65.467	−562	−3076	13,541	8666	39,198	57,767	No excess
Germany	82.533	−143	12,197	10,066	34,161	−1541	54,740	22.0%
Greece	10.711	−70	3569	4710	3500	8805	20,515	17.1%
Hungary	9.762	−94	4593	10,362	8676	4277	27,813	16.2%
Iceland	0.362	19	6	−14	−139	−14	−142	a
Israel	9.293	−269	203	1168	668	1431	3201	−2.1%
Italy	59.630	−568	12,066	21,888	40,081	42,223	115,690	9.9%
Latvia	1.906	−49	1082	1248	2254	1511	6046	17.1%
Lithuania	2.802	−56	2584	2403	3567	2785	11,283	22.4%
Luxembourg	0.635	40	−58	−64	1	189	109	No excess
Netherlands	17.479	14	1241	3566	7175	5973	17,969	7.0%
New Zealand	5.013	−62	−512	−797	−1301	−1446	−4118	No excess
Norway	5.408	−63	−452	−433	−1540	−506	−2994	No excess
Poland	38.482	−574	19,293	47,295	39,417	44,291	149,722	12.5%
Portugal	10.323	−110	1959	2877	4277	7283	16,286	11.4%
Slovakia	5.480	−36	3467	6293	6211	2728	18,663	18.4%
Slovenia	2.103	−13	−49	691	1418	1897	3944	No excess
South_Korea	51.631	−774	−431	−7474	−10,335	−11,272	−30,286	No excess
Spain	47.511	−203	9788	13,194	14,342	31,598	68,720	13.9%
Sweden	10.408	−41	−682	2	−422	776	−367	No excess
Switzerland	8.688	−16	175	492	1075	3914	5640	2.8%
United_Kingdom	67.145	−762	25,852	19,276	20,476	22,465	87,307	28.7%
United_States	329.995	−2858	365,676	216,688	169,623	122,167	871,295	41.6%
TOTAL	969.336	−7737	486,236	395,508	382,515	353,341	1,609,862	29.7%

^aNo excess overall, small excess in >65 years old.

4.3. Other countries

eLife, Economist, Lancet, and WHO had estimates of excess deaths in 42 additional countries (Supplementary Table 4). These countries had a total population of almost 1.4 billion, the total reported COVID-19 deaths were almost 2.3 million and the excess death estimates were about double with all 4 evaluations (4.7 million per eLife, 4.8 million per Economist, 5.5 million per Lancet, 4.4 million per WHO) with higher correlations between eLife, Economist and WHO and more modest correlations with Lancet for all three excess death metrics. For several countries, differences across evaluations were very large. Two countries had death deficits by some evaluations, but not with others (Singapore range −1770 to 1776 and Japan range −19,469 to 111,000).

Another 98 countries (total population 5.3 billion) had excess death values obtained only per Economist, Lancet and WHO (Supplementary Table 5) for a total of 10.7, 9.8, and 7.9 million, respectively (6–9 times higher than recorded COVID-19 deaths). The correlation of the calculations was modest for all three metrics (r = 0.499–0.570, r = 0.552–0.662, r = 0.530–0.801 for E_p, E_m, and R, respectively). Very large differences across different calculations for the same country were very common. For 11 countries, there was an estimated death deficit based on some calculation but not with all 3. For China, the range was extreme (452,669 excess deaths per Economist; −52,064 deficit per WHO).

5. Discussion

Across 33 high-income countries with total population of approximately 1 billion and highly thorough death registration, different empirical estimates of excess deaths in 2020–2021 ranged widely from 1.6 million, i.e. substantially fewer than the 1.9 million recorded COVID-19 deaths, to 2.8 million, almost a million more. Countries with highest estimates of excess deaths were more or less the same in all evaluations; and countries with the most favorable picture performed well across the different evaluations. However, large differences emerged in the magnitude of country-specific estimates. The largest divergence was produced by whether age adjustment was used in calculating excess deaths. Age-adjusted estimates were lower. They are more appropriate, and COVID-19 has impressive age-related risk gradient for death (Estimated excess death count from, 2021; O'Driscoll et al., 2021; Axfors and Ioannidis, 2022). Modest changes in age structure, in particular with aging populations, may produce major differences in estimates. With age-adjustment, several high-income countries showed no or minimal excess deaths; and for many others, the estimates of excess deaths markedly decreased. However, even age-adjusted excess death estimates differed substantially depending on the choice of data source for age-stratified population in the recent years as well as modeling choices. High correlation coefficients between different methods in many of our assessments may be misleadingly optimistic for agreement, as they do not capture the large absolute differences in estimates between different methods. Moreover, correlations may be less meaningful for such derivative measures and they can be largely driven by the minority of the extreme countries (those that clearly did very well and those that did very poorly).

Our age-adjusted analyses also revealed large differences across countries in the proportion of excess deaths accounted by non-elderly age strata. A few countries had more than a quarter of the excess deaths in people <65 years old. There are several possible factors that may explain this pattern. Adverse risk profile of the non-elderly populations, including high prevalence of obesity; inequalities and disadvantaged populations without good health care; and increased fatalities due to opioid overdose and other non-COVID-19 causes of excess death may explain in part the USA pattern. Also for Canada, USA and UK, many deaths in elderly people may have occurred in long-term care residents with very limited life expectancy. Deaths from SARS-CoV-2 among patients with limited life expectancy result in no excess deaths if the time window assessed after their infection is shorter than their life expectancy (Ballin et al., 2022). E.g., patients in palliative care with life expectancy of <12 months and who died in 2020, they would have been expected to die before the end of 2021 even in the absence of COVID-19 infection, thus their COVID-19 deaths would not be captured in excess death calculations covering the whole 2020–2021 2-year period.

The ratio R of excess deaths over recorded COVID-19 deaths varied substantially across different evaluations. Only 5 countries in Eastern Europe and USA had consistently R > 1, i.e. more excess deaths than recorded COVID-19 deaths. Conversely, only Australia and New Zealand had consistently R < 1. R values varied widely for all countries. In most countries, the uncertainty in the range of excess death estimates exceeded the number of recorded COVID-19 deaths itself. This large variability questions to what extent excess deaths can give much better insights on the total pandemic toll than COVID-19 recorded deaths (Islam, 2022). Excess death calculations are dependent on how they are calculated. In some cases, like Eastern Europe, they can tell that COVID-19 deaths have been undercounted and/or the numbers of other deaths have escalated during the pandemic. However, they cannot differentiate the relative contribution of these two factors nor can they give a precise estimate of either or their combination. In a few countries with limited SARS-CoV-2 deaths they can be reassuring that indirect pandemic effects and measures did not escalate fatalities, at least during 2020–2021. However, for most countries, excess death calculations are so model-dependent that they should be seen with great caution.

We observed that total excess death estimates in the 33 high-income countries with most reliable data were similar for 2021 than for 2020. This was seen despite the availability of effective vaccination options in 2021 and it may reflect the higher percentage of people infected in 2021 than in 2020 in many countries that had low population infection rates in 2020. Excess deaths in the two calendar years differed markedly within single countries and variability may continue to be seen in 2022.

Besides the high-income countries with meticulous weekly death registration data, excess death calculations for the rest of the world are very tenuous exercises. For 42 countries where eLife, Economist, Lancet and WHO generated estimates, on average excess deaths were ∼2-times the COVID-19 recorded deaths and among another 98 countries where Economist and Lancet provided estimates, overall R was 6–9. However, given the low reliability of the data, the immense uncertainty surrounding these estimates cannot be overstated. More importantly, these calculations offer no causal insights. Excess deaths may be due to the virus, the indirect pandemic effects, or/and disruptive measures taken, even more so in countries with very frail healthcare systems, widespread poverty, and/or even high rates of hunger. The 2020–2021 crisis may have indeed resulted in many deaths, but causes may be very complex.

For some of these additional countries, calculations were run based on available mortality data. Even then, in most cases death registration is unreliable and the impact of changes in death registration during the pandemic compound any calculation. Perhaps age-adjustment would also lead to different estimates in these countries, as for the 33 most data-reliable countries. Even for these 33 countries, population estimates (including age-stratified population counts) are projected with different methods and typically no formal population census has been performed for several years. This adds further uncertainty to excess death calculations that are highly susceptible to minor differences especially in the population of elderly strata. How populations are imputed and even what population is considered (January 1 versus mid-year) can also make a difference occasionally. For most countries, excess death calculations are indirectly imputed from the countries with mortality data. The methods employed by the Economist are not described in sufficient detail to allow probing validity and reproducibility. Lancet and WHO calculations provide more elaboration, mostly proving their complexity. The uncertainty in excess death estimates apparently far exceeds the width of published confidence intervals.

Hence, extrapolations from the 33 main analysis countries to other countries need to be extremely cautious. Among the total excess deaths, deaths due specifically to viral infection in the other countries may be proportionally far less. Other countries have far smaller percentage of elderly people and very few nursing home residents. Conversely, they have far more frail health systems, societies, and economies. Therefore, probably indirect effects of the pandemic and measures were perhaps more important contributors to total excess deaths than SARS-CoV-2 infection.

Nepomuceno et al. have also assessed the impact of different analytical choices on excess death calculations for 2020 (Vandenbroucke, 2021). They find modest differences among different approaches in countries with reliable death registration. Age-stratification was also shown to be important in previous assessments for specific countries (Baum, 2022; K ö nig et al., 2022; De Nicola et al., 2022; De Nicola and Kauermann, 2022). Germany is a classic example. Our age-adjusted estimate is 55,000 excess deaths, while without age-adjustment we calculated 129,000 excess deaths and Lancet calculated 203,000 excess deaths – compared with 111,000 COVID-19 reported deaths. De Nicola et al. (De Nicola et al., 2022; De Nicola and Kauermann, 2022) estimated only approximately 30,000 deaths for Germany for 2020–2021 by using an even more refined method than ours that employs detailed life tables, while we only used 5 age strata assuming homogeneity within each stratum. In countries like Germany, the population may be aging not only overall but also within each age group. Thus, finer age adjustment will give more appropriate estimates of excess death and these are likely to be even smaller than what we report.Baum (Baum, 2022) calculated only 22,000 excess deaths after age adjustment. In Germany, the number of people aged >80 years increased from 4.8 million in 2016 to 5.8 million in 2020, so consideration of age is crucial (Kowall et al., 2021).

We caution that it would be an over-simplification to infer that lack of age adjustment is always causing excess deaths to be exaggerated. Other modeling choices may also have major impact on the calculations. For example, for Italy, age-adjusted estimates are smaller than the non-age-adjusted estimates, but the range of estimates is substantial even between age-adjusted analyses (116,000–160,000) and between non-age-adjusted analyses (167,000–258,00 deaths). The Italian National Institute for Statistics has released a report of 178,000 excess deaths as of the end of January 2022, including 145,000 due to COVID-19, thus placing an estimate in the middle between age-adjusted and unadjusted analyses. Careful, in-depth analysis of both death certificates and medical records is needed to get a better handle of age-stratification effects and a better sense of the relative contribution of COVID-19 versus other causes towards excess mortality. For example, as already mentioned above, the large share of excess deaths in people <65 years old in the USA may be due to a considerable extent to an increase in deaths from overdose/opiates in 2020–2021 (as escalation of an ongoing problem that worsened due to pandemic disruption); and myocardial infarction due to missed health care for this acute condition. Disadvantaged populations were also hit the most during the crisis and careful dissection of the contributions of inequalities and societal marginalization towards excess COVID-19 and non-COVID-19 deaths is necessary. For non-high-income countries, the impact of age stratification is extremely difficult to assess at the moment, but it is very likely that excess deaths especially in young people are largely due to the major impact of disruption, poor care, and indirect effects of aggressive measures taken.

Further diversity stems from whether modeling anticipates increasing life expectancy (decreasing mortality rate) over time (Kowall et al., 2021). This anticipation may calculate spuriously high excess deaths: calculations assume a desired mortality rate lower than even attained in reality in the past. Medical and overall progress cannot guarantee continuing to decrease overall mortality in high-income countries, especially in old, frail people. In fact, care of such people may have deteriorated over time in recent years (e.g. with privatization and deterioration of long-term care) and the pandemic brought this to light (Akhtar-Danesh et al., 2022). Two of the 4 previous studies even used splines for generating the expected values for 2020–2021. With splines the extrapolation strongly depends on how knots are chosen and this can markedly affect the results. In all, models that expect that mortality should continuously decrease over the years (let alone with the steep slopes that some splines may generate) may have totally unrealistic expectations and may calculate more excess deaths even if there has been a death deficit but not as prominent as expected under spurious expectations of major decreases versus past mortality. Excess deaths then spuriously emerge against a phantom of optimistic expectations. Data from much longer-term periods of observation suggest multiple, overlapping, complex long-term trends in winter mortality (Jones and Ponomarenko, 2022). Finally, some other models may diverge in their calculations, if they exclude certain periods. E.g., the popular Euromomo model using a Serfling model in its core but excludes weeks of high influenza activity from the modeling: thus it generates high excess death estimates (Schöley, 2021).

Some other caveats should be discussed. First, not only finer adjustment for age (e.g. in more narrow age bins), but also more comprehensive adjustment for other factors (e.g. gender, frailty, long-term care facility residence, comorbidities) may be able to offer even more accurate estimates of excess deaths. Second, data on deaths are not final for a while even for high-income countries and also previous analyses of excess deaths may correct or improve their calculations downstream. Since we performed these analyses, WHO has recently made some corrections (Van Noorden, 2022), especially for Germany and Sweden, in response to criticisms. Two core noted problems were the spurious nature of extrapolations of splines and the use of mortality data that did not match the raw data from the national statistical offices. As a result of the corrections, the Germany estimate of excess deaths decreased by 37% while that of Sweden increased by 19%. During the revision of our manuscript we have also repeated in mid-June 2022 our age-adjusted analyses to consider available data including all 2020–2021 and data on as many weeks of 2022 as they may be available in different countries in the Human Mortality Database. The excess deaths during early 2022 for the 8 countries that had death deficit in our previous 2020–2021 calculations are as follows: Australia 279 (8 weeks), Denmark 713 (21 weeks), Finland 2175 (17 weeks), Iceland 135 (13 weeks), Korea 17,105 (13 weeks), Norway 843 (20 weeks), New Zealand −366 (19 weeks), Sweden −244 (19 weeks). Therefore, with the exception of Finland, they continue to have overall a death deficit in the total period 2020–2022. We have also noted that 5757 deaths in Sweden were not assigned to a specific week for 2020–2021 in the 5 age bin data file in Human Mortality Database. However, these deaths were included in another file of Human Mortality Database where data were presented in 24 age bins of 5-years width each and also separately for men and women. When we calculated excess deaths for Sweden in 2020–2021 using the complete 24 age bins, the excess deaths for the 2 years were 63 and the estimate became a death deficit (−220) after considering also gender stratification. Among other countries, there was substantial variability in their 2022 performance to-date versus the 2020–2021 performance, e.g. Netherlands had only 1650 excess deaths in first 21 weeks, while Greece already had 8801 excess deaths in the first 13 weeks. We caution that recent HMD data may not be complete and already noted some inconsistencies against national sources. Updates as well as corrections may change the exact excess death estimates again.

Third, there can be debate on whether/how natural disasters and wars should be excluded.

Heat waves, and other natural disasters (many of which are accentuated by climate change) may lead to increasingly unstable estimates for specific countries and years in the future, but probably had limited overall impact to-date for most high-income countries in our analyses - with occasional exceptions. E.g. the highest excess death week for Greece during 2020–2021 was an August 2021 week when a heat wave was compounded by major fires exposing acutely half the population of the country to very toxic atmospheric pollution, as the authorities failed to control the fires.

Fourth, long-term effects of both the pandemic and measures taken on healthcare, other aspects of health, education, society, and economy remain uncaptured in the 2020–2021 window. Comparisons of different approaches to excess death calculations should continue for longer periods of follow-up, also in the post-pandemic endemic phase. The boundaries between pandemic and endemic phase can be debated (Ioannidis, 2022). Regardless, the relative performance of different countries and their excess death ranking may change substantially over time. Analyzing specific causes of death may be informative, but suffers from major misclassification even in high-income countries.

Fifth, we provide for the 6 compared methods point estimates of excess deaths, without accompanying measures of precision, such as confidence intervals. Some of the published results to-date provide also such measures of uncertainty and we refer to the respective publications for their perusal. We believe that the published estimates of uncertainty are underestimated or even grossly underestimated, as they consider only some sources of uncertainty. As we have demonstrated, there are many degrees of freedom in the analysis of these data, each component in the calculations has its own uncertainty, and the uncertainties multiply. As an illustrative example, we estimated that if we consider all possible combinations of 2017, 2018, and 2019 as baseline, the mean (or median) standard deviation of the excess death estimates across the 33 countries is already 1.3% of expected deaths. Adding variability due to diverse modeling or ignoring of time trends and diverse/erratic source data would increase uncertainty substantially.

Acknowledging these caveats, our analyses map the magnitude and uncertainty of excess deaths during 2020–2021. In countries with reliable data, age-adjustment suggests that the number of excess deaths is lower than what has been previously published in calculations without age-adjustment. Excess death calculations convey some broad picture, especially for countries that fared very well or very poorly. Large differences in the impact of a pandemic across countries has been seen also in previous pandemics (Viboud et al., 2016, 2005) for reasons that often remain largely unexplained. In depth assessments with death certificate audits, medical record audits, and autopsies may yield more granular insights about deaths and their causes, but these approaches also have limitations and feasibility challenges. For most countries worldwide, the tremendous uncertainty in the sparse data and indirect inferences should be mostly a call for improving the completeness and accuracy of death registration and investment in more rigorous demography infrastructures in the future.

Credit author statement

M.L.: Conceptualization; Data curation; Formal analysis; Funding acquisition; Investigation; Writing – original draft; Writing – review & editing. F.Z.: Data curation; Formal analysis; Investigation; Writing – original draft; Writing – review & editing. J.P.A.I.: Conceptualization; Data curation; Formal analysis; Investigation; Writing – original draft; Writing – review & editing.

Funding

NIHR35 GM122543 (Michael Levitt). The work of John Ioannidis is supported by an unrestricted gift from Sue and Bob O'Donnell.

Data statement

All data are in the manuscript and in publicly available datasets.

Ethical approval

Not applicable.

Declaration of competing interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Appendix A. Supplementary data

The following is the Supplementary data to this article: