Multiple testing and the effect of NPIs on the spread of SARS-CoV-2

You Li and colleagues¹ estimate average associations between imposing and lifting eight non-pharmaceutical interventions (NPIs) and the reproduction number (R) of severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) in the first half of 2020 across 131 countries through a regression analysis with daily data. Since changes in the status of different NPIs often occurred either jointly or in close temporal proximity within each country, their individual associations are generally difficult to disentangle from observational data and are naturally subject to substantial statistical uncertainty.² This uncertainty is unfortunately not adequately captured by the 95% CIs reported by Li and colleagues.¹ In particular, they do not reflect the fact that multiple NPIs are considered simultaneously, and they do not account for possible temporal and spatial dependence between datapoints.

To see the scope of the simultaneity issue, consider the association between NPI-status changes and the percentage shift in R after 28 days. With lengths between 30 and 72 percentage points, the corresponding 95% CIs reported in the right column of table 1 in the Article are quite wide to begin with. But with 16 estimates to account for in this case alone, a simple Bonferroni correction³ would further widen each 95% CI by about half. Although there are other statistical adjustments that might not result in quite as much stretch, it is safe to predict that 95% CIs that correctly account for multiple comparisons would be much wider than the ones presented in table 1, would all cover a zero change in R, and would overlap substantially.

It is therefore not possible to deduce from this kind of data with conventional levels of statistical certainty that imposing or lifting any particular NPI is associated with a non-zero change in R after 28 days, or that any particular NPI works better than any of the others under consideration (analogous comments apply to estimates for other timepoints). Given the substantial statistical uncertainty, individual point estimates should also not be interpreted as precise predictions of the effect of future interventions.