Excess Mortality as a Tool to Monitor the Evolution of Health Emergencies: Choices, Challenges, and Future Directions

Excess mortality has become one of the most popular metrics to assess the impact of the COVID-19 pandemic across the world. Excess mortality has three key advantages relative to alternatives: (1) it requires relatively little data (in the most basic form, a historical time series of annual deaths), (2) it does not depend on whether causes of death are accurately assigned on death certificates, and (3) it can be compared across space and time.

EXCESS MORTALITY ESTIMATES AND THEIR INTERPRETATION

In this issue of AJPH, Vandenbroucke and Pearce (p. 593) discuss the many merits of the excess mortality metric and focus on its role as an inferential tool to compare the “performance” of different countries or subpopulations in containing the COVID-19 pandemic. As they note, this type of analysis is akin to a difference-in-differences design, albeit usually approached with less rigor than in the causal inference literature. The key assumption behind this approach is that there are no time-varying differences between the two units being compared (typically two countries). For example, differences in excess mortality in two countries would be informative about the effectiveness of the policies they implemented only if no other time-varying factor could explain them. Since policies are not randomly assigned and countries were on different mortality trajectories before the onset of the pandemic, the assumption of no time-varying differences is unlikely to hold in many comparisons. Vandenbroucke and Pearce reach the same conclusion and exhort researchers to more carefully discuss how the assumptions might be violated and how the violations would affect the interpretation of the results. We think that this discussion can be made more rigorous if five crucial choices are carefully considered.

DESIGNING AN EXCESS MORTALITY ANALYSIS

The first choice a researcher faces is how to estimate the number of deaths that would have been observed in the absence of the COVID-19 pandemic. The most basic method to obtain this counterfactual consists of averaging annual deaths over a number of prepandemic years. This approach, used by many early studies of excess mortality,¹ has the advantage of being simple. However, it ignores trends in mortality and population change; furthermore, unless combined with a statistical analysis, it does not lend itself to probabilistic statements on excess mortality. More robust methods consider linear and nonlinear time trends as well as seasonality, most commonly through either the inclusion of harmonics with varying periodicities² or with the use of seasonal³ or nonseasonal autoregressive integrated moving average (ARIMA) models, and deliver both point estimates and uncertainty intervals for expected mortality. Typically, the more flexible approaches are attractive when few units are considered (e.g., national deaths stratified by age) but can become computationally challenging if applied to subnational geographic units.^4,5

The second key choice faced by researchers is how many years to include in the baseline period on which the counterfactual is calibrated. When this decision has a substantial effect on the estimates, multiple options should be explored and the robustness of the analysis’s key findings should be assessed.⁶

A third choice, also discussed by Vandenbroucke and Pearce, is whether to adjust for or stratify by important covariates, especially age. Whether such adjustments are needed or desirable depends on the amount of between-group heterogeneity and whether the principal aim is to conduct a causal analysis or descriptive study. In the first case, adjusting for potential confounders is crucial. However, in the second case, where description rather than causal inference is the goal, controlling for factors such as age and average income might be counterproductive as the estimates would no longer reflect the number of excess deaths that actually occurred. In such an instance, stratification may be a better approach as it allows for an examination of subgroup differences while maintaining the descriptive integrity of the estimates. For example, if excess mortality estimates are to be used to understand where to allocate recovery funds, we may prefer an estimate of the actual number of deaths that occurred (overall and across strata) rather than a synthetic estimate reflecting a hypothetical world in which all units shared similar characteristics.

A fourth choice is which level of spatial and temporal granularity to use. Regarding spatial units, national-level analyses have the advantage of fewer data requirements and offer appealing units for international comparisons. However, in countries where the spread of COVID-19 was geographically heterogeneous, national analyses might underestimate the impact of the pandemic. This limitation is particularly important when conducting cross-country or subnational comparisons and is closely related to the point made by Heuveline and Tzen that comparisons between small and densely populated areas (where virus transmission is easier) and large and sparsely populated ones (where the virus faces “natural” barriers) can be misleading.⁷ A similar point can be made regarding temporal granularity. In countries, such as the United States, in which clear waves in COVID-19 and excess deaths have been documented,⁵ yearly estimates might be unsatisfactory, especially when combined with a focus on national rather than subnational trends.

A fifth and final choice faced by researchers is which mortality indicator to present. Typical choices include crude excess death rates, age-standardized excess death rates, and age-specific excess death rates. Absolute counts of excess deaths, in total or by age, are also popular, together with relative excess presented as percentage or proportional increases in mortality. As pointed out by Vandenbroucke and Pearce, this choice should reflect assumptions about whether excess mortality would be additive (absolute measures would be more appropriate for comparisons) or multiplicative (relative measures would be more appropriate for comparisons). Another important consideration should be how interpretable different measures are. In general, we have found relative measures to have an advantage in this dimension as they do not require demographic knowledge to be interpreted. The key choices for modeling excess mortality are summarized in Figure 1.

FIGURE 1—

Key Factors to Consider When Estimating an Excess Mortality Model

Note. SARIMA = seasonal autoregressive integrated moving average. The figure illustrates the major choices and considerations that may be relevant to designing an excess mortality analysis, including how to construct a reasonable counterfactual, how to deal with potentially different time trends across units, which level of granularity to choose, and whether to adjust for covariates.

EXCESS MORTALITY BEYOND THE COVID-19 PANDEMIC

While excess mortality analyses have become popular in the public health community during the COVID-19 pandemic, they have been used before the pandemic to study influenza-associated deaths⁸ and heat-related mortality,⁹ and more recently to compare US mortality with the mortality of peer nations.^10,11 These applications are valuable because they convert abstract public health issues into a quantity that is easy to understand for researchers across different fields, policymakers, and the general public.

Despite its many strengths, excess mortality as a metric also has some weaknesses. Death is often preceded by visible symptoms and, in the case of a viral disease, by an infection, possibly resulting in hospitalization. As such, other metrics like test positivity, percentage of emergency department visits diagnosed as COVID-19, hospitalizations, and severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) RNA levels in wastewater provide better early indicators of the evolution of a health emergency. In addition, deaths are usually reported and made available to researchers with a lag. For example, mortality data for the United States are only 65% complete within two weeks and 85% after four weeks, and remain only 95% complete after eight weeks.¹² These reporting lags further limit the usefulness of excess mortality as a real-time indicator. However, most alternative indicators do not guarantee complete coverage and, aside from hospitalizations, are more useful in measuring the spread rather than in assessing the intensity of a health emergency. Excess mortality thus stands out as a tool to retrospectively evaluate the effectiveness of measures aimed at reducing the negative health impact of a health emergency.

The numerous advantages of the excess mortality metric suggest that it is likely to remain an essential tool for monitoring emerging threats to population health moving forward. Future research should continue to develop, refine, and standardize excess mortality modeling tools to assist with future public health preparedness and response efforts.