Statistical successes and failures during the COVID‐19 pandemic: Comments on Ellenberg and Morris

The ongoing HIV/AIDS and COVID‐19 pandemics have exerted an extraordinary effect on human health, and they are prime examples of the unique and ever‐present threat of emerging infectious diseases. In their article, Ellenberg and Morris describe the critical role of statistical thinking in responding to both pandemics. This includes modeling the outbreak, tracking reporting trends, characterizing the natural history of disease, and evaluating interventions. Statisticians have a unique opportunity to contribute during public health crises, to ensure that analyses are rigorous, inference is sound, and policy decisions are driven by the data.

Though we remain in the midst of the COVID‐19 pandemic, sufficient time has elapsed that we can reflect upon some successes and failures. These reveal lessons that we may carry forward into the future.

A Success—Vaccines and Leveraging Existing Networks. The rapid identification of safe and efficacious COVID‐19 vaccines is a standout achievement for the scientific community. A noteworthy contributor to this success was that the COVID‐19 Prevention Network (CoVPN), formed by the US National Institutes of Health, relied heavily upon prior experience testing HIV vaccines. ¹ For over 20 years, the HIV Vaccine Trials Network (HVTN) has been hard at work implementing trials and developing associated statistical methods. ² HVTN statisticians played a critical role in CoVPN, advising vaccine companies working with Operation Warp Speed on their trial protocols. The end result is higher quality and more standardized results, enabling easier comparisons for regulators and for the public.

Repurposing infrastructure put in place for other diseases has been an important strategy to efficiently conduct pandemic research. Many systems for influenza have been extended to include COVID‐19. The Seattle Flu Study, created in 2018, detected some of the earliest SARS‐CoV‐2 infections in the United States. ³ Leveraging the established relationships investigators had with their community, they were able to quickly stand up new SARS‐CoV‐2 studies in households and homeless shelters. Another positive example of a study that quickly adapted is the REMAP‐CAP platform trial for simultaneous evaluation of multiple treatments for community‐acquired pneumonia. ⁴ In response to the COVID‐19 pandemic, they implemented a pandemic appendix that enabled already participating trial sites to begin testing COVID‐19 therapeutics. These successes highlight the value of building multi‐functional studies, including planning in advance how to accommodate emerging infectious diseases.

A Failure—National COVID Data and Missing Standardization. Unfortunately, there have also been failures in our response. Ellenberg and Morris document a range of data challenges familiar to statisticians even outside the context of a fast‐moving pandemic—missing information, measurement error, lagged reporting, and lack of standardization. Infectious diseases are inherently local, so we like to study trends on a fine spatial scale. We want to disaggregate across ages, occupations, and racial/ethnic groups to track risk and target interventions. Individuals may have diverse reasons for seeking testing, thus it is useful to distinguish between those presenting with and without symptoms. Yet it has been incredibly challenging to find summary data with this level of detail. Notably, different states use different definitions for reporting, including different categorizations for age and racial/ethnic group. ⁵ Though the Centers for Disease Control is rapidly expanding the functionality of their online dashboard, ⁶ this utility should have been available much earlier.

As statisticians, we must be advocates for data quality and accessibility. The best treatment for these data woes is prevention, and statisticians have a key role in highlighting why we need high‐quality data to guide the response. This includes demanding more transparency in the methods used to report and calculate statistics, which is crucial for making comparisons across locations. Even something as simple as the total number of SARS‐CoV‐2 tests conducted can be very complicated, as the COVID Tracking Project has described in their blog. ⁷ Given their deep understanding of COVID data, the COVID Tracking Project group has also offered public recommendations for a national dashboard and tracking vaccinations. ⁸ We can also be advocates for making data easy to access. For many states, data have not been consistently available in machine‐readable format to enable automated retrieval. Statisticians can highlight the importance of open data to encourage the many independent data analysis and modeling efforts that have yielded useful insights.

A Work in Progress—Statisticians and the Media. A reporter recently commented to me, “SCIENCE IS THE NEWS NOW.” Given the dramatic and persistent impacts of this pandemic on daily life, new scientific insights are constantly in demand. Of course statisticians can contribute to the cause through their scholarly work—be it designing a clinical trial, analyzing surveillance data, or modeling disease dynamics—but there are other opportunities to have a wider impact. The landscape is moving so quickly that peer‐reviewed journals are straining to keep up, and so science is happening in pre‐prints, on dashboards, in opinion pieces, and even on social media. To maintain quality, more eyes are needed for informal peer review. As statisticians, we can offer a range of skills to improve public understanding and correct misconceptions. As a small example, I have now taught the concepts of test sensitivity, specificity, and positive predictive value to my students and at least a dozen reporters with questions about antibody testing.

Unlike the early days of the HIV pandemic, there are many more avenues to engage with the public. These efforts fall under the heading of service, which does not always get its due in academic circles, but as a community we should acknowledge the immense value this work adds to the public discussion. The time required can range from modest, for example, a 15‐minute chat with a journalist who will then translate your insights into a larger story, to more intense, for example, maintaining a weekly blog. Social media is a particularly accessible way to join the conversation. Scientists can stay up to date on the latest research and layer on top of these their own interpretations. Journalists are very active on social media, and they may reach out for comment if they find your insights to be uniquely informative. Of course media training is not a part of our curriculum, and some statisticians may be uncomfortable talking to the press. A blog post entitled “Top tips to statisticians communicating through the media, especially in the time of COVID‐19” offers advice, an act of service in itself. ⁹ The helpful tips include “don't be lured out of your comfort zone” and “keep off the (statistical) jargon.” Adding my own, as statisticians we may be most intrigued by the details, but reporters are most interested in the big picture. Learn how to strike a balance.

As noted by Ellenberg and Morris, during the COVID‐19 and HIV pandemics, “[s]tatisticians and other data scientists with expertise in integrating and synthesizing information across multiple data types… play a critical role in proper interpretation of these data.” We are able to bring nuance and insight to an often over‐simplified public discussion, where an out of context point estimate can turn into headline and then into a media frenzy. We can enter into conversations early to encourage high quality study designs, and support sound statistical inference along the way. Yet our role can also extend far beyond data analysts. Whether your research focuses on infectious diseases, genetics, air pollution, neuroimaging or another area, statisticians can also be scientific leaders. Statisticians can testify in front of governments. By viewing ourselves as well‐rounded scientists, we are able to achieve an even greater level of impact.

Dean N. Statistical successes and failures during the COVID‐19 pandemic: Comments on Ellenberg and Morris. Statistics in Medicine. 2021;40:2515–2517. 10.1002/sim.8934

DATA AVAILABILITY STATEMENT

Data sharing is not applicable to this article as no new data were created or analyzed in this study.