Development and dissemination of infectious disease dynamic transmission models during the COVID-19 pandemic: what can we learn from other pathogens and how can we move forward?

Abstract

The current COVID-19 pandemic has resulted in the unprecedented development and integration of infectious disease dynamic transmission models into policy making and public health practice. Models offer a systematic way to investigate transmission dynamics and produce short-term and long-term predictions that explicitly integrate assumptions about biological, behavioural, and epidemiological processes that affect disease transmission, burden, and surveillance. Models have been valuable tools during the COVID-19 pandemic and other infectious disease outbreaks, able to generate possible trajectories of disease burden, evaluate the effectiveness of intervention strategies, and estimate key transmission variables. Particularly given the rapid pace of model development, evaluation, and integration with decision making in emergency situations, it is necessary to understand the benefits and pitfalls of transmission models. We review and highlight key aspects of the history of infectious disease dynamic models, the role of rigorous testing and evaluation, the integration with data, and the successful application of models to guide public health. Rather than being an expansive history of infectious disease models, this Review focuses on how the integration of modelling can continue to be advanced through policy and practice in appropriate and conscientious ways to support the current pandemic response.

Introduction

Although the current COVID-19 pandemic has brought infectious disease transmission models into prominence, there is a long history of the use of mathematical and statistical frameworks to understand infectious disease transmission and inform public health interventions.1, 2, 3, 4, 5 The oldest known disease transmission model dates back to Daniel Bernoulli, who first used a probability-based model in 1766 to evaluate the effectiveness of smallpox inoculation, the process of introducing smallpox into the skin to cause a more mild illness than if the virus acquired by the usual respiratory route, but one that induces immunity.⁶ This precursor to modern-day vaccination was not without risk, and Bernoulli used his model to evaluate the likelihood of death following inoculation against the gain in life expectancy from evading infection. William Farr was perhaps the first to attempt to calibrate (or fit) a model of disease transmission when he noted the similarity between the timeseries of smallpox incidence and the normal distribution.⁷ In 1889, En’ko⁸ investigated heterogeneity in measles transmission, using models to explain the possible mechanisms that give rise to variable epidemic sizes. In 1927, Kermack and McKendrick⁹ expanded on existing models to develop the foundation of the modern-day susceptible-infected-recovered mass-action model using differential equations. These models, and many others, have formed the basis of modern disease modelling, which has been a crucial tool in improving understanding of best practices for infectious disease control. By combining mathematical or statistical formalism with epidemiological data and an understanding of biological mechanisms, infectious disease models enable the evaluation of the effect of public health interventions like vaccines, projection of future disease burden in various contexts, and answering fundamental questions, such as why some people are uninfected in an outbreak.

During the COVID-19 pandemic, transmission models have been used to provide rigorous guidance to inform decision making for infectious disease control at an unprecedented rate. Transmission models are often separated into two classes (though overlap exists): mechanistic models, in which individuals or populations are modelled as moving between discrete, exclusive health states via defined transitions, such as infection or recovery, and the process of transmission is modelled to represent the specific mechanisms of infection;¹⁰ and statistical models, in which transmission processes or the entire population are not necessarily explicitly modelled, but key associations are leveraged to estimate or predict transmission variables or disease risk.11, 12, 13, 14 The statistical models category would include so-called forecasting models, which are purely phenomenological.⁵ Both forms of transmission models can be used, broadly, for either of two purposes: first, to project disease burden, given a model structure and estimated or assumed parameter values; or second, to test competing hypotheses (ie, models) on what mechanisms best explain observed disease dynamics. Often, models are first fit to data, allowing for in-sample model assessment; and then, crucially for public health purposes, used to generate out-of-sample projections of disease transmission in the future or in other settings assumed to have similar transmission dynamics.

Models are crucial tools to estimate key epidemiological parameters, to understand and project the effect of intervention policies, to evaluate the presence or effect of transmission or disease history phenomena (eg, superspreading events, waning immunity), and to provide short-term and long-term projections of disease risk or burden with customised assumptions of disease transmission and intervention strategies. Which of these applications are the most feasible and the most useful will depend on the context of disease transmission: under the timescale of model development and with few data during an emerging outbreak, short-term projections and estimates of basic disease variables might take precedence over exploration of natural history phenomena that occur over longer timescales. Previous major triumphs of disease modelling include examining the effect of vaccination campaigns (eg, effect of routine and supplementary vaccination programmes for measles) and the necessary amount of interventions for disease control or elimination (eg, required vector control amount for malaria control), identifying important epidemiological features of disease outbreaks (eg, measles honeymoons) and endemic transmission (eg, different seasonal transmission dynamics for measles), and predicting heterogeneity among epidemiological dynamics in different sociodemographic, geographical, or population contexts (eg, effect of different seasonal forcing, or birth rates and population mixing, on transmission).

The use of both statistical and mechanistic disease models has increased substantially during the COVID-19 pandemic, primarily with the aim of estimating epidemiological variables or projecting case and mortality counts over time. As of Sept 27, 2020, there have been more than 5000 modelling analyses published in peer-reviewed journals, excluding preprint servers, since the start of the epidemic in December, 2019. These analyses have greatly aided the pandemic response worldwide, from providing the first estimates of the basic reproductive number, R₀, to highlighting areas or populations that needed improved surveillance,15, 16, 17, 18 quantifying targets for intervention strategies such as contact tracing,19, 20, 21, 22 and keeping decision makers and the public informed of current and potential future burdens.23, 24, 25, 26, 27 However, models and their results can be misrepresented, misinterpreted, or even mis-specified. As with any other scientific enterprise, inappropriate assumptions and the misinterpretation of data or results can cause modelling efforts to be faulty. Importantly, a subset of COVID-19 models have relied on opaque or inappropriate methods that do not accurately convey uncertainty. Reliance on these models can lead to fallacious policy decisions and the erosion of public confidence in modelling results.5, 28, 29

Therefore, there is one central question: how can the use of mathematical transmission models be rigorously tested and evaluated in the context of an emerging pandemic where key epidemiological features might be unknown, information to guide responses and policy is urgently needed, and data might be few and rapidly changing? To answer this question, expertise needs to be drawn from a range of disciplines, in particular, epidemiology, public health, immunology, data science, statistics, ecology, and scientific communication; and historical instances in which modelling has been successful—or unsuccessful—in guiding infectious disease prevention and control should be considered. Below, we discuss a few ongoing and historical examples in the field of mathematical transmission modelling for public health applications. We then outline aspects from these examples that have resulted in success and conclude with recommendations for future modelling efforts of emerging outbreaks.

What makes modelling efforts successful: past examples

In this section, we outline briefly the history and features of several successful and less successful efforts where models were developed and investigated to inform a public health response.2, 8 These efforts span multiple pathogens, hosts, routes of transmission, and transmission settings (emerging, endemic, or nearing elimination). Although no two outbreaks or transmission scenarios are alike, a common thread in the examples discussed here is the use of models in real time to develop policy decisions. Shared in the examples below of key public health questions that have been addressed through modelling, and mirrored in the ongoing COVID-19 outbreak, are the problems of: few and possibly unreliable data, potentially uncertain transmission routes and model structures, and developing and communicating actionable modelling results. Comparing and analysing previous outbreaks across a range of settings allows for a holistic view of best practices and potential pitfalls for disease modellers.

Malaria: how effective does vector control need to be?

Models of malaria transmission were some of the first mechanistic models used to assess public health interventions. Between 1908 and 1921, Ross developed a series of mathematical formulations of malaria transmission following an unsuccessful mosquito larval control trial.30, 31 At the time, the long-standing scientific belief was that mosquito populations should be completely eliminated to eradicate malaria, an unattainable goal. However, with the use of a theoretical framework based on the mosquito-human transmission process and the mosquito lifecycle, Ross provided evidence that malaria transmission could be contained with only the partial control, rather than the extinction, of mosquito populations. This early work laid the foundation for particular metrics to monitor transmission that are still used, such as the prevalence rate and entomological inoculation rate. Following this work, Macdonald extended Ross's model to inform control strategies for the WHO Global Malaria Eradication Programme.30, 31 Importantly, this work evaluated the usefulness of additional vector control measures, like insecticides, and their overall effectiveness in reducing malaria transmission in high transmission regions of sub-Saharan Africa. These early models were highly successful in four areas: first, illustrating the benefit of developing biologically realistic theoretical formulations; second, identifying key epidemiological values, and outlining the necessary data to estimate them; third, outlining and addressing clear questions with precise communication, integration, and motivation from the eradication programmes; and finally, restricting analyses to evaluating the effectiveness of different interventions as opposed to directly forecasting the burden, which was not yet an attainable goal with the available knowledge of malaria transmission and computational methods.

HIV and AIDS: what is the effectiveness of different testing and treatment policies?

Models of HIV and AIDS, including models of population-level transmission and within-host viral dynamics, have been used to identify patterns of transmission and risk structure, the effect of treatment and individual changes in the immune response with antiretroviral therapy, and the emergence and propagation of drug-resistant variants.32, 33 Arguably the most successful use of population-level (as opposed to within-host) models has been in producing and evaluating estimates for the intensity and frequency of various treatment and prevention measures needed to reach control targets in forward simulations of incidence and prevalence.34, 35, 36, 37, 38, 39 As with many emerging pathogens, back-calculation methods were also commonly used to estimate relevant transmission parameters (eg, the incubation period) and historical infection incidence from AIDS incidence data.⁴⁰ Importantly, these methods also provided a scientifically supported approach to project AIDS incidence at various stages of disease progression with surprising simplicity and without the need for more complex model structures or data that were not readily available early on in the global pandemic.⁴⁰

Increasingly, more realistic formulations of projection models have been developed to add social, demographic, and biological realism to the population risk structure, with a particular focus on burden within key populations and the role of heterogeneous sexual networks.41, 42, 43 The understanding that preferred or assortative mixing, concurrency in sexual partnerships, and scale-free contact networks might lead to greater disease burden and faster growth than expected under proportionate mixing assumptions was developed largely from models of HIV transmission.42, 43, 44, 45 In addition to these theoretical advances in infectious disease dynamics, important public health decisions have been guided by the long-term projections of HIV incidence estimated with the use of mathematical transmission models, including the adoption or recommendations to scale-up antiretroviral therapy, universal test and treatment, and treatment as prevention.39, 46, 47 Several of the most successful modelling efforts were integrated with long-running cohort studies or clinical trials, which provided crucial data on intervention effectiveness to modellers, and enabled the integration of modelling results into policy decisions and on-the-ground public health activities.47, 48 But variability in model structure, complexity, and variable choice might be amplified over the decades-long timescale for which projections of HIV are often made, considering the innate complexity of HIV transmission. Sensitivity analyses and model comparisons, often done by formalised working groups, have been crucial to identifying a general consensus (eg, that antiretroviral therapy has the potential to substantially reduce infections, if access and adherence to it are high)34, 47 and exploring possible uncertainties and variability in modelling results.⁴⁹ At present, modelling is widely used to guide national and international programmes; for example, the Spectrum/AIDS impact model is used in more than 170 countries to estimate key HIV transmission and control indicators.⁵⁰

Measles: how should vaccinations be deployed?

Measles is one of the earliest pathogens to be modelled, with transmission models dating back to at least the late 1800s.⁸ Since the wide-scale deployment of a safe and effective vaccine, transmission models have informed and guided international, national, and local immunisation programmes.8, 51 By providing a clear way to estimate spatial, seasonal, and age-specific transmission rates, dynamic compartmental models have been used to evaluate the effect of novel interventions, control strategies, and elimination strategies. In addition, an array of theoretical and applied work based on long-term mortality and morbidity data has informed current understanding of the relative effect of birth rates, seasonal forcing, recolonisation, and extinction events, and age-specific mixing rates on producing variable epidemic patterns.52, 53, 54, 55, 56, 57 Importantly, models have been used to understand the effectiveness of various vaccination campaign designs on local and global measles elimination in both low-income and high-income settings. The balance between applied and basic science has formed robust literature focusing on measles dynamics. For example, the WHO Strategic Group of Experts includes models as a key tool for the successful control and hopeful elimination of the disease.⁵⁸ The successes of integrated modelling-public health efforts have largely relied on a foundation of clear, applied questions, along with an abundance of long time series data and multiple model fitting approaches. When coupled with the pathogen's simple life history, models can readily address frequently applied questions such as: where are the areas of the population that routine vaccination coverage is not reaching, what is the likelihood of transmission in susceptible populations, and how to best design effective additional vaccination campaigns. Measles dynamics highlight the success associated with the use of extensive epidemiological data across a wide range of geographical, population and demographic settings to inform model structure, estimate variables, and test model validity. Additionally, the history of measles modelling shows the benefits of a goal-oriented approach for establishing modelling priorities and of rigorous model evaluation by multiple research groups over time.

Rubella: what is the effect of vaccination strategies on congenital rubella syndrome burden?

Routine vaccination for rubella has additional complications compared with those for the measles vaccination, in that insufficient vaccine coverage can increase overall disease burden by increasing the average age of infection and subsequently the risk of congenital rubella syndrome among pregnant women, relative to the more mild form of the disease that occurs in early childhood.51, 59, 60, 61 As a result, those leading vaccination programmes should weigh the risk of increased congenital rubella syndrome burden against the benefits of an overall reduction in rubella infections when considering introducing rubella vaccine into their routine programme. This calculation depends on the demographic characteristics of the population, the risk of an outbreak, and the probable vaccination coverage. Infectious disease transmission models have been used to clearly identify the risks and benefits of introducing rubella to a vaccine programme and therefore have been an integral part of guidance policies for routine vaccination requirements in various countries. The success of rubella modelling in addressing what the appropriate circumstances are for introducing a rubella vaccine into a population relies on three components. First, the ability to build off of the substantial history of measles transmission modelling (with the use of common terminology and validated models); second, clear policy questions and guidance; and third, subsequent model refinements to the structure of the model following the identification of possible vaccination strategies.4, 61

Foot and mouth disease: how should the epidemic be controlled?

During the 2001 foot and mouth disease outbreak in the UK, multiple models were used to predict the disease dynamics and inform control measures.⁶² This situation was one of the first instances when models were used during an epidemic to support the decision making process. Models were used both to predict the epidemic trajectory (with stochasticity) as it was occurring, and to compare different control measures.⁶² Models used during this time were able to capture the number of cases at a moment in time, approximate the spatial concentration of cases, and roughly estimate the overall magnitude of the outbreak.63, 64, 65, 66 Direct communication between modellers, veterinarians, and policy makers, as models were being developed and as more epidemiological data were becoming available, was facilitated via a centralised working group.⁶⁷ Therefore, this outbreak served as the first example of the regular integration of emerging data into modelling efforts during an ongoing outbreak to inform decision making (in this situation, the recommendation was to cull infected animals and potentially exposed animals in nearby facilities).⁶⁸ That multiple models were developed allowed for model comparison to identify robust results, and helped to support the notion that modelling results were not biased by model structure or complexity. These efforts were not without controversy, though; some contend that the few models used to justify large-scale animal culling were unverified, based largely on the same data and assumptions about transmission, and that the policies they informed were too rigid.62, 67, 69 There was concern that model estimates were presented as inappropriately precise, and that models geared towards understanding a national outbreak were not sufficiently capturing the local context.62, 68

Ebola: what will be the magnitude, burden, duration, and areas affected by the Ebola outbreak?

Although the 2014–2015 Ebola outbreak in west Africa probably gave rise to more real-time models than any previous outbreak scenario, it is also arguably the least successful example of integrating mathematical models with a public health response. There were an estimated 125 models identified and developed during this outbreak to answer a range of questions, including predicting the spatial and temporal extent of the outbreak, overall burden and required hospital or treatment centre capacity, and subsequent effects on other health services.70, 71, 72, 73 Several models sought to evaluate the use of ring vaccination strategies to control emerging outbreaks or mitigate established outbreaks.74, 75, 76 Most models attempted to forecast burden, but there were few or inappropriately aggregated data that missed local patterns, a rapidly changing situation on the ground, and suffered from little integration with response activities in the most highly affected countries.71, 77, 78 Often, there was little collaboration and transparency between modelling efforts, and there were substantial delays to the results being made public.70, 77 Post-hoc analyses of these models found that their ability to accurately forecast burden was low; forecasts were unreliable further than 1–2 weeks in the future, a timeframe far surpassed by the modelling attempts during the outbreak.70, 71, 72, 73, 79 Substantial differences in model results, including some criticised at the time as clearly unreasonable or reliant on assumptions known to be unfounded, further complicated the interpretation and application of model results.79, 80, 81 Such disagreement led to a real-time analysis critiquing avoidable errors in Ebola modelling.⁷³ The difficulty in engaging local stakeholders and the community, particularly those on the front lines of epidemic response, combined with these siloed, conflicting, and even sensationalist modelling results fed into a general distrust that prevented models from being able to provide results that would inform effective and implementable public health interventions.82, 83

Summary of lessons learned

We note several common features among the successful efforts (table ).

Table.

Recommendations for successful public health modelling efforts

	Specific actions
Integrate modelling into decision making in the early stages of the outbreak response	Develop models with succinct, actionable outputs (eg, estimates of health-care system needs under various intervention strategies; risk of infection or death for key populations) that address the specific needs of policy makers and stakeholders; continually refine models to address the changing needs of policy makers and stakeholders and to incorporate new data or knowledge about disease transmission. Clearly communicate to policy makers and stakeholders any changes to model and consequences of these changes
Integrate data in both construction and evaluation stages	Evaluate and constrain models with empirical data. At minimum, ensure that the results are consistent with the observed data. Ideally, use data to quantify the uncertainty or bias in model performance and to improve model accuracy via advanced fitting methods; incorporate uncertainty in the data, including report processes if possible, into modelling results. Consider how biases in the data would propagate through to model outcomes
Ensure reproducible and transparent modelling practices	Make model code publicly accessible and easily reproducible; publish results publicly (eg, through preprint servers); where available, participate in modelling consortia or other collaborative modelling efforts to confront and explore assumptions in model structure and data uncertainty; share modelling results in accessible ways to appropriate audiences. Pay special attention to communicating the assumptions and uncertainties in modelling results
Contextualise modelling results	Consider the specific context in which policy recommendations are to be made; adapt models to cover the population of interest, transmission patterns, or behaviours, and potential interventions being considered; explicitly state assumptions and possible biases underlying modelling results. Clarify the heterogeneities and questions that the model can address and the heterogeneities that the model ignores or simplifies; present modelling results with an appropriate degree of uncertainty, and, if applicable, over a relevant time frame

First, successful efforts integrate modelling into decision making in the early stages of the outbreak response by developing models that produce succinct, actionable outputs addressing the specific needs of policy makers and stakeholders. Second, successful models integrate data in both construction and evaluation stages, ensuring that the results are consistent with the observed data and knowledge of the epidemiology of the pathogen of interest. The integration with data should include some quantification of uncertainty in parameter estimation and model output. Thus, as the epidemic progresses, these models can be evaluated and refined in terms of both their structure and complexity and users obtain a deeper understanding of exactly which questions these models can address. Third, successful efforts make use of reproducible and transparent modelling practices, thus facilitating collaborative research and peer review by the scientific and public health communities. Finally, successful modelling is properly contextualised, both in development and in producing results relevant to and informed by current public health practice. Modelling results that are not grounded in a particular context are unlikely to produce timely, actionable results. In creating and consuming these models, the context-specific implications of results should be actively addressed at the peril of misinterpretation or misuse, leading to suboptimal and at times biased public health policies.

Of the five example pathogens discussed, the use of modelling in the COVID-19 pandemic probably best resembles that of the 2014–2015 Ebola outbreak: many models are produced independently, with little real-time evaluation or scientific review of model results, few (or biased) data are available, and knowledge of how severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) is transmitted changes continually. However, one difference is that in many settings modelling efforts for COVID-19 have been integrated with public health decision making at multiple levels. The magnitude of the public health emergency response provides more support and demand than ever for modelling results to inform decision making, presenting an opportunity to integrate models into public health emergency and response on an unprecedented scale.

How can the successes and pitfalls of past transmission modelling efforts help to inform SARS-CoV-2 transmission modelling?

A universal feature of transmission models is the need to evaluate their usefulness and validity within the context of their intended use. This aspect has been key to making previous modelling efforts successful, since it directly informs the choice of model structure, variables, and outputs, and makes these results more relevant to decision makers. Model evaluation most commonly refers to a formal quantitative assessment of how well the model output matches known data; exactly which data or target population is used, and which metric is used to establish how well the model and data match depends on many factors, as well as a more holistic review of the usefulness and reasonableness of a model and its assumptions.

Importance and challenges of continued model refinement

As knowledge of transmission, disease progression, and the effectiveness of interventions continue to improve over the course of an epidemic, transmission models should be refined to reflect these changes and the current needs of decision makers and stakeholders. Model refinement might reflect both the data and the underlying assumptions about processes driving transmission patterns. For example, at the start of the pandemic it was reasonable to use estimates of the serial interval, generation time, and transmission rate from SARS-CoV for SARS-CoV-2 before many data were available. However, as new data and analyses emerged, these variable values were no longer valid. Similarly, initial models of SARS-CoV-2 might not have included asymptomatic or pre-symptomatic transmission, given the inadequate understanding of the pathogen at that time.⁸⁴ This assumption might underestimate transmission and accordingly overestimate the effectiveness of some interventions, such as temperature screenings.⁸⁵ The validity of modelling results might depend most on these structural assumptions, which should continually be interrogated and evaluated via model comparison and sensitivity analyses.86, 87

What is reasonable to include in a model will change over time; a model that might be a useful tool early in an outbreak could no longer be valid weeks later. In fact, the most effective models should render themselves obsolete, as they guide policy and intervention implementation that change the course of the previously forecasted outbreaks. This presents an additional challenge; though, in that changing recommendations and contradicting results following feedback between modelling results, decision makers, and policy makers, can cause distrust of modelling. The evolution of model results over the course of an epidemic should be communicated carefully and with consideration of the target audience.⁸⁸ Models should be constrained by empirical data as much as possible; however, in scenarios where this is not possible, researchers are under obligation to evaluate and communicate the limitations of their modelling approach.

Continual model refinement does not require continually more complex models. Perhaps the most crucial decision in modelling is to decide which heterogeneities (eg, contact structure, age-specific risk of severe disease or mortality, seasonality) should be included to derive valid or useful results, which heterogeneities are not needed, and which heterogeneities have sufficient data to include in a model. Model parsimony is often a strength, in that the data required and the assumptions inherent to the model are fewer, which has contributed to the enduring popularity of compartmental models such as those derived by Kermack and McKendrick.⁹ Although it might appear that more complex models, in making fewer abstractions about the disease transmission process, might better capture all aspects of an outbreak, such models rely on many more latent assumptions that can imply and amplify the bias or uncertainty in model results.⁸⁹ Balancing the need or request to build a highly specific model with the constraints of available data and knowledge is key to generating useful, interpretable, and actionable model results.

Importance and challenges of fitting models to data

Grounding models in data is one way to address concerns around model validity. Data are crucial to better inform key aspects of models, including variable terms, structure, and intervention scenarios. However, a model used to guide policy is only as good as the data it relies on—therefore, a model's reliance on data is a double-edged sword, particularly in emerging epidemics. The most successful examples of models used to inform policy (eg, measles) have greatly relied on long time series data, spanning more than 100 years, of incident cases coupled with information on vaccination in multiple populations and countries to estimate variables, test model complexity, and inform model structure. Such rich data do not yet exist for the current COVID-19 pandemic; however, when available data are used methodologically and with careful interpretation of possible biases, they are nonetheless invaluable in developing realistic models.

Reliance on data largely comes in two forms: clinical, laboratory, or epidemiological data used to externally estimate key variable input into the model; or data on a model outcome to which the model is directly fit. Data availability for either purpose might be scarce or delayed in emerging epidemic scenarios. Biases in data because of over representation of some groups, varying access to equitable health care, and low spatial resolution, can also skew a model's ability to resolve an underlying mechanism (eg, age-specific risk of infection) or result in flawed projections.71, 73 Although methods exist to explicitly consider reporting and measurement processes, when fitting models or estimating key variables these methods generally assume a constant reporting rate or a simple functional form (eg, a logistic response function), when in reality reporting is likely to change non-linearly in an emerging outbreak.⁹⁰

Data drawn from multiple sources present an additional challenge, in that estimates from one source (eg, transmission rates derived from early serological analyses) might differ from estimates drawn from other sources (eg, transmission rates derived from early exponential growth time series). Whether these differences are because of differing methodological assumptions, confounding factors, temporal trends, or reflect the true differences in estimates across populations, can rarely be established. For example, estimates of the proportion of COVID-19 asymptomatic infections have ranged from 12% to 60%, but are probably confounded by age and differences in study design or case definitions.⁸⁴ Perhaps one of the more pernicious challenges of using data to fit transmission models is the use of aggregated data that can mask local variance in transmission patterns, restricting the accuracy of local projections that might be most helpful to stakeholders guiding response.29, 62, 71

Just as models require continual refinement, data are frequently changing during an outbreak scenario. For example, incorporating newly available case data for model fitting⁹¹ or estimating the serial interval⁹² has led to multiple estimates of the R₀ that sometimes conflict, despite it being a fundamental transmission variable. Data are likely to become more complete or less prone to errors over time. Common estimation procedures, such as trajectory matching assuming an exponential growth of the outbreak, are particularly sensitive to the omnipresent errors in early available data. As the outbreak continues and community transmission persists, data might be more reliable, but the assumptions about the initial exponential growth underpinning these methods are no longer valid and will probably lead to incorrect variable estimation. Advanced methods, including approximate Bayesian computation, data augmentation, and particle filtering, allow for more inclusion of multiple data sources under flexible distributional assumptions,93, 94, 95 though these methods alone cannot resolve issues of validity or uncertainty in data. Incorporating the inherent stochasticity of both disease transmission and the reporting processes when fitting a model to data is computationally and technically complicated, but crucial to fully capture the range of plausible model outcomes.

Importance and challenges of clear, reciprocal communication and interpretation

In epidemics, transmission models are primarily developed and used to answer policy-relevant questions. The successful modelling efforts previously outlined (malaria, measles, foot and mouth disease, and rubella) were all guided by clear policy goals or intervention scenarios and informed by policy makers. As transmission patterns and response to the pandemic change, the questions of interest to stakeholders—and the questions that feasibly can be answered with models—will also change. Early models will probably focus on estimating baseline transmission rates and populations most likely to be at a high risk of infection or severe outcomes on the basis of preliminary data. As more data become available, direct forecasts will become increasingly accurate. Later models might focus on estimating and comparing the effect of intervention strategies that have been or that could be implemented. As a result, one model cannot possibly answer every question of interest during these types of outbreaks, and models should regularly be tailored to address the most pressing questions. At the same time, modellers should communicate clearly the assumptions or possible biases arising from the choice of model structure and data, as described earlier. In the longer term, modellers should develop a formal integration or partnership with public health agencies to facilitate sustainable modelling efforts for public health policy.⁹⁶

For example, one common policy question in the COVID-19 pandemic is: when will we hit the maximum intensive care unit bed capacity? Although statistical, curve-fitting models might be useful for some short-term forecasts of intensive care unit bed use, they are generally not as useful for longer term forecasts that rely on knowledge of the mechanistic underpinnings of disease transmission. Furthermore, as testing and interventions change, the demographic characteristics of the infected population also change, leading to different intensive care unit bed usage. In general, simple, compartmental mechanistic models might be ideal for exploring the relative effect of intervention strategies or varying assumptions of the transmission process, but are less suited for precise forecasts, particularly in the absence of epidemiological data to which models can be fit. Such models are also ill-equipped to consider individual-level transmission phenomena (eg, the role of household transmission or the effectiveness of contact tracing), which are better addressed with models at the individual-level, that are agent-based, and that are of the mechanistic, network, or statistical branching process type. Useful though they might be, long-term forecasts are likely to be unreliable in nearly all contexts, though especially so early in an outbreak when much is unknown about the routes and frequency of transmission or the interventions available to control the spread.

Interpreting model results accurately and communicating them effectively is essential for policy and public health officials to make decisions to improve health and for the public to engage in intervention use.⁷⁷ The interpretation and dissemination of mathematical models, however, is inherently challenging as models are used to inform a diverse set of stakeholders ranging from policy makers and other scientists to journalists, and by proxy the general public.5, 97, 98 For the interpretation of model results, it is important to at least describe the type of model used and what questions this model can address (eg, a planning model allows for the comparison of different intervention scenarios), to explicitly state the model assumptions, and to provide the results with estimates of uncertainty that appropriately propagate uncertainty from underlying data and methods of estimation, when possible. Accurately depicting model traits such as uncertainty is crucial for producing useful and timely model interpretations, particularly during the exponential growth phase of epidemics. Model results can often appear deceptively precise, though point estimates typically represent an aggregate, probabilistic estimate of what might happen. For example, models cannot predict precisely when intensive care unit bed capacity will be reached. Rather they show, within bounds of uncertainty, the likelihood that intensive care units will exceed capacity in a time frame and which interventions might reduce that risk. Comparing various models, which is an increasingly common practice, is another method to help provide a broader range of estimates to inform decision making. Groups have even started providing guidelines on how these comparisons should be done and explored.⁸⁶ Without appropriate measurements of uncertainty and transparency in model assumptions, model results might give rise to misinformation and poor decision making on behalf of policy and public health officials.²⁸

Conclusion

As the COVID-19 epidemic changes regionally, new questions or public health policies arise and require their own evaluation. As a consequence, an abundance of mathematical models and analyses are thrust onto the public and scientific body. The use of models has changed as the COVID-19 pandemic has progressed, and as a result there will be more substantial qualitative and quantitative variation in the models developed by academic and industry groups worldwide. Ultimately, only a subset of these models will be functionally integrated with the ongoing public health response. As historic examples have shown, transmission models can be used to help inform and guide public health policy; however, their success depends on integrating the biology of the pathogen, specifics about the population of interest, and epidemiological parameters, with a clear focus on the public health policy implications of the results, all at the same time as communicating the uncertainty and the limitations of current knowledge and model results.

Search strategy and selection criteria

We searched PubMed for the terms “covid” OR “covid19” OR “covid-19” OR “sars-cov-2” OR “novel coronavirus” AND “model” OR “modeling” OR “modelling” published from Dec 1, 2019 to Sept 20, 2020, to identify published modelling efforts for COVID-19. Similar searches were done in PubMed, medRxiv, Google Scholar, and Web of Science for related papers for each of the specific modelling efforts discussed. References were included based on their relevance to the broad scope of this Review. Only papers published in English were reviewed.