On the fallibility of simulation models in informing pandemic responses

As of April 24, 2020, the severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) pandemic has led to over 2·7 million confirmed cases, and 190 000 reported deaths worldwide. To reduce the transmission and mortality of this newly emerging infection, countries have needed to make rapid policy decisions on the basis of scarce early data with many uncertainties.

In the early stages of pandemics, mathematical models can provide valuable insights into transmission dynamics, help to predict disease spread, and evaluate control measures.¹ However, models are only valid within the limits of the parameters examined. As reliable parameter estimates are rarely available early in a new pandemic, best-guess estimates are used, which need to be constantly reviewed as new real-world data emerge. Estimating how sensitive the model is to changes in its parameters can provide useful information about validity when parameters are uncertain. Interpreting models without considering these factors can lead to flawed inferences, which can have far-reaching effects when they inform public health policy.

We illustrate the potential impact that flawed model inferences can have on public health policy with the model described in The Lancet Global Health by Joel Hellewell and colleagues,² which is part of the scientific evidence informing the UK Government's response to coronavirus disease 2019 (COVID-19).³ On March 12, 2020, the UK Government decided to cease community testing and contact tracing, claiming that the scientific evidence did not support these strategies, as the UK had entered the so-called delay phase of the epidemic. These actions were consistent with the conclusions of Hellewell and colleagues that these measures were unlikely to be able to bring the epidemic under control in under 12 weeks: “In most plausible outbreak scenarios, case isolation and contact tracing alone is insufficient to control outbreaks, and in some scenarios even near perfect contact tracing will still be insufficient, and further interventions would be required to achieve control.”²

A key parameter of the model was the delay between a case becoming symptomatic and being isolated.² Two median delays were modelled—3·83 days (short) and 8·09 days (long)—on the basis of data from the SARS-CoV epidemic, and an empirical distribution calculated from the early phase of the SARS-CoV-2 outbreak in Wuhan.² However, by this time, several rapid tests with a turnaround time of less than 4 h had been developed internationally, with some having received relevant regulatory approvals.⁴ Soon after, South Korea and Singapore began drive-through testing and had developed the capacity to test rapidly and deliver results within a single day, making a 1-day delay scenario plausible.5, 6 Indeed, recent models examining interventions for COVID-19 control in Singapore have assumed a 1-day delay between symptom onset and quarantine.⁷

We find that inferences of the model from Hellewell and colleagues² are very sensitive to the parameter of onset-to-isolation delay (appendix). Using the authors' original code, we demonstrate that when the delay is changed to a median of 1 day, the model predicts the probability of controlling the epidemic within 12 weeks to be more than 80%, with 30–60% (ie, considerably less than near perfect) contact tracing (depending on the proportion of pre-symptomatic cases at a given time). These results suggest that rapid testing, contact tracing, and isolation could be effective strategies to control transmission.

At the time that the UK Government decided to cease community testing, real-world data from several other countries were pointing to the effectiveness of testing and contact tracing, potentially at odds with the evidence that the government was following. On March 12, 2020 (16 days after the tenth SARS-CoV-2 death in both South Korea and Italy), South Korea had 66 reported deaths compared with Italy's 827 (appendix). The epidemic trajectories in South Korea, Japan, Taiwan, and Singapore have been vastly different from most European countries, with far fewer deaths (appendix). Although multiple aspects of the populations and systems of these countries could have contributed to these differences, one key commonality is the early adoption of an approach focused on testing, isolation, and contact tracing in the countries with slower trajectories. Indeed, many of these countries did not institute nationwide lockdown measures at all or until much later, after the curve had flattened, potentially indicating the impact of case detection-based strategies on transmission (appendix). These detection-based strategies might have allowed the countries to control transmission without stringent lockdown measures and school closures, thus avoiding the disruption that these entail. Instead, they were able to implement targeted restrictions in response to observed outbreaks (eg, church closures in South Korea).⁸ By contrast, many European countries have had to impose lockdowns much earlier in the timeline of their epidemics (appendix).

Empirical, real-world data must be considered alongside mathematical models when devising pandemic responses. Models are fallible and scientists and policy makers must be mindful that an over-reliance on models, and a lack of caution in interpreting them, could be a costly exercise.