Transparency assessment of COVID-19 models

The COVID-19 pandemic has strained societal structures and created a global crisis. Scientific models have a crucial role in mitigating harm from the pandemic, by estimating the spread of outbreaks of the virus and analysing the effects of public health policies. The context-sensitive and time-sensitive measures provided by COVID-19 models offer real population health impacts and are of great importance. However, these models must be completely transparent before policies and insights are enacted.

Transparency is a cornerstone of scientific methodology, and efforts to improve transparency and reproducibility of research have been increasing over the past decade.¹ Researchers have called for complete transparency of COVID-19 models.² An absence of transparency in the design, development, and analysis of these models reduces the trust in their timely messages and limits their reproducibility, impeding scientists from verifying the findings and improving the model's performance. Many modellers have already shared the details of their models openly. However, the overall status of transparency of COVID-19 models remains unknown. We assessed whether COVID-19 modellers adhere to best practices in reporting and documentation; we did not evaluate whether a model's projections are correct.

To systematically evaluate the transparency of COVID-19 models, we reviewed a sample of models that have earned global attention and been referenced in governmental public health efforts. We first collected models that included a methods report from the US Centers for Disease Control and Prevention's compilation,³ then identified the most-referenced models through Google Scholar and the PlumX News Mentions metric. This search took place on June 13, 2020, and resulted in the identification of 29 models. Due to the urgency created by the pandemic, preprints and project websites made available in advance of research publication have had an essential role during the crisis.⁴ Therefore, we included models from these sources (n=14), in addition to models in peer-reviewed publications (n=15).

We assessed these models against 27 binary criteria to evaluate the transparency of their reports. The transparency assessment criteria were adopted from several transparency checklists,5, 6, 7 and include two main themes. First, the specificity of model items, including but not limited to discussion of assumptions, parameterisation, codes, and sensitivity analyses, and second, general research items, such as disclosure of research limitations, funding, and potential conflicts of interest. Two researchers reviewed the full text and appendix of each modelling report. A third reviewer helped to discuss the discrepancies between the first two reviews to reach a resolution.

Each of the 27 criteria were satisfied by an average of 22 (76%) of 29 models in our sample. Eight criteria were satisfied by more than 90% of the models, but most criteria were satisfied by a much smaller percentage of models (appendix p 1). For example, seven (24%) of 29 models did not report the equations used, nine (31%) did not report their estimated parameters, 13 (45%) did not share all of their longitudinal data, and 15 (52%) did not report their code (appendix p 2). Only four articles (14%) satisfied more than 90% of our transparency checklist items.

This evaluation shows that models that are not fully transparent can still posit analytical insights and inform policy. Rather than presenting recommendations at face value, modellers must ensure that their claims are independently verifiable. The scientific and modelling communities need to make transparency the norm, rather than the exception. Otherwise, they risk losing the faith of policy makers and the public.

Such consequences were observed for a model released on March 30, 2020,⁸ which was often cited by government agencies, including the White House.⁹ Amid concerns about the model's projections, the scientific community was frustrated by the absence of information regarding codes and specific model details. Another high-profile preprint model published on March 16, 2020,¹⁰ faced similar scrutiny after predicting 510 000 deaths in a scenario with no interventions, prompting researchers to attempt replication. When the code was released about 6 weeks later, several bugs and issues with the modelling assumptions were unearthed. Issues like these suggest that governments should not rely on a small number of models to inform policy. Instead, policy makers can mitigate potential harm by aggregating the available models and synthesising their results to help inform action.

A crucial element of model transparency is achieved by providing codes. One concern for researchers is that their code is disorganised and cannot be evaluated or run by laypeople hoping to replicate their efforts. However, even messy code can provide a framework for replication and generate useful dialogue, as seen on platforms like GitHub. Still, well documented code is preferable. Of the 14 (48%) of 29 articles which reported their codes, 13 provided helpful detailed documentation either directly in the file or in supplementary material. We encourage modellers who hope to impact perceptions and policy to follow transparent research practices and release their codes in a timely manner for public evaluation.

Many journals ask for transparency statements and encourage scientists to report the details in supplementary documents. Although journals should continue to further enhance their transparency requirements, they cannot control the full transparency of publications. Additionally, peer review cannot be fully relied upon for the most technical aspects of a model, especially if papers do not provide documentation. During crises such as COVID-19, preprints provide fast information delivery. Therefore, journals' transparency policies have minimal effect. Models which were still preprints or project websites satisfied an average of 70% of our transparency criteria, compared with 80% by peer-reviewed articles. The responsibility to provide transparency remains largely with the modellers, even though the peer-review process can help address these omissions.

Reporting a fully documented and transparent model can be difficult, but this effort has both tangible and intangible benefits for modellers and policy makers. With the urgency created by a global pandemic, modellers might justify prioritising speed of reporting rather than transparency. However, poor transparency of models that directly impact public health policies (and therefore human lives) can be catastrophic. Model transparency does not necessarily equate to model quality; however, models with little documentation cannot be assessed for quality at all. Hence, all models must be fully transparent for both scientific and ethical purposes.

This online publication has been corrected. The corrected version first appeared at thelancet.com/lancetgh on November 18, 2020