Observational studies provide crucial information early during epidemics and pandemics, but they often suffer from methodological shortcomings, which can be resolved.
Scientific research is a necessary part of epidemic preparedness and response. Observational studies, in which the intervention and outcome(s) of interest are not under the researcher’s control, are used in epidemics to describe basic properties of a pathogen and its transmission; clinical symptoms; associations between interventions and patient outcomes; and the effectiveness of public health measures to curb disease spread.
Early importance
An example of a type of observational study that is particularly important for epidemic research is a prospective cohort study. These studies enroll populations of individuals who have a particular exposure or similar characteristics, and researchers collect data to evaluate possible outcomes associated with their exposure. For example, the Immunophenotyping Assessment in a COVID-19 Cohort (IMPACC) study was a prospective cohort study that launched in May 20201. It followed newly hospitalized, SARS-CoV-2-positive individuals to understand clinical and immunological disease manifestations. The study results defined a set of clinical characteristics to assist clinicians with diagnosis and treatment; it also set the stage to evaluate individuals suffering from long COVID.
Observational study designs are ideal early in epidemics because of their speed and ease of implementation across settings, lower cost relative to other study designs, and flexibility in integrating pre-existing data sources, such as historical clinical data, census data and previous study results, to control for variables that may not be collectable. Beyond generating information to improve public health responses, rigorous observational studies can inform the design of subsequent randomized, controlled trials (RCTs) of novel interventions, ultimately reducing morbidity and saving lives2.
An observational study can be an ethically superior design early in epidemics because the risk–benefit tradeoff is frequently simpler than for other study designs, such as RCTs. For instance, because they do not directly provide experimental interventions, observational studies do not cause any intervention-related adverse events3. There are also settings in which RCTs are impossible to conduct, for example when an epidemic is emerging and outcomes are still so rare that achieving sufficient enrollment and statistical power in RCTs is infeasible4.
Yet despite their major potential for scientific and social value5, observational trials in recent epidemics and pandemics have failed to address priority research questions and suffered from important methodological shortcomings, generating false leads for investigators and policy-makers and contributing to scientific misinformation and mistrust. Targeted reforms that are neither resource nor time intensive can address these problems.
Methodological shortcomings
Some observational studies in recent epidemics generated information that did not have the potential (ex ante) to lead to significant health benefits or did not address vital research questions, such as clinical presentation, host specificity or transmissibility. For example, one meta-analysis assessing the association between ABO blood type and risk of SARS-CoV-2 infection found 314 relevant papers with data collected in 20206. Although conducting a few high-quality studies of blood type and risk of infection may have been justified, in the context of a global pandemic and without a clearly actionable finding, the existence of upward of 300 studies seems to be of low social value when considering the priority research questions in a public health emergency.
Many observational studies in recent epidemics were conducted in ways that were methodologically or otherwise flawed, reducing the likelihood that the results could lead to substantial health benefits. In particular, studies variously lacked data standards such as defined units and vocabularies for the management of data across studies that examined the same intervention or outcome; did not measure or incorrectly measured confounding factors; or used incorrect design and analytic methods7,8. Infamously, poorly conducted and heavily biased observational studies of hydroxychloroquine resulted in its use as treatment for COVID-19, causing patients to receive incorrect treatment, given that it was ultimately found ineffective, and disrupting the early pandemic response7. Post hoc strategies to address methodologic biases are moreover limited and can result in conflicting evidence at best or compound incorrect or harmful evidence at worst9.
Observational studies, especially if conducted early in an epidemic, may also suffer from small samples and inconsistencies in sample selection, limiting their generalizability from the sample to the broader population (external validity) and their power to detect significant results. Geographical dispersion of events may mean that individual research teams have few cases on which to build a study, and those cases may be more reflective of the particular features of a study site than of the outbreak in question7,8. This, too, reduces the likelihood that study results can lead to significant health benefits.
Indeed, observational studies may have negative social value if their findings undermine the epidemic response. Observational studies, not least because of the speed with which they can be conducted, may spread low-quality or spurious information, thereby informing major and potentially irrevocable decisions in the early epidemic. These decisions cost lives due to the adoption of ineffective interventions and the abandonment of effective ones, divert limited resources for healthcare and research, and lead to overall poor policymaking7.
Even well-designed observational studies are difficult to communicate and easily misinterpreted by policy-makers, journalists and the public in the often rapidly evolving situation of an epidemic. Residual confounding, bias and study estimators (such as odds ratios) in observational studies are less clear than those in clinical trials and are more varied study to study, meaning that interpreting study results may take more time and be less straightforward to non-experts. There is a real information hazard if the results and limitations of those studies are not well communicated.
Master protocols
Given the limitations of observational studies identified above, reforms are needed to address these limitations, thereby promoting the social value of observational studies in future epidemics (Table 1).
Table 1 |.
Issues and proposed solutions for reforming observational studies
| Issue | Proposed solutions | Contribution to social value |
|---|---|---|
| Addressing the right research question | Master protocols to establish priority research questions | Allocates scarce scientific resources to the most pressing questions in an epidemic |
| Methodological issues | Master protocols and data standards to harmonize study design, data collection, analysis and reporting | Increases methodological rigor of studies and reduces scientific waste |
| Small sample sizes | Collaborate and consolidate data to improve study quality | Generates externally valid knowledge for clinical or public health practice |
| Misinterpretation by stakeholders such as health policy-makers, journalists and the public | Lay scientific methods summary in publications | Mitigates the risk of misinterpretation of observational study results |
Master observational study protocols should be developed to establish priority research questions during infectious disease outbreaks, helping to guarantee that study results lead to health benefits. Master protocols also help ensure appropriate participant and measurement variable selection while reducing bias, increasing the likelihood that health benefits will result. Readily available protocols will aid in prioritizing important information for outbreak response, such as basic reproduction number, symptoms, prognosis for different risk groups and effectiveness of nonpharmaceutical interventions. These protocols should be developed in consultation with a diverse segment of the research community to ensure that priorities and outcome measures are robust in advance of the next epidemic. Ongoing protocol development for multisite RCTs can inform this process10,11.
Data consistency
Open data standards should be developed and adopted to improve consistency in data collection, especially of outcomes such as case definition or intervention effectiveness12. Harmonized data standards enable data comparison across studies and reduce the burden of managing mountains of incompatible data, increasing the likelihood that observational studies will provide socially valuable results. Lessons can be learned from clinical data standards that help harmonize and standardize clinical data from electronic health records and claims data13.
Research groups should be empowered to collaborate and consolidate observational study data into larger samples to improve research quality and make it more likely that the results will improve clinical or public health practice. This goes beyond merely harmonizing and centralizing participating sites’ electronic health records. Rather, research sponsors should establish funding opportunities and large-scale collaborative research networks that create a shared sense of purpose and trust (ref. 10; https://www.recoverytrial.net/).
Research communication
Conducting observational studies across settings via a network unified in protocol and data standards would provide high-quality evidence to policymakers and prevent the controversy associated with multiple low-quality studies providing conflicting information. It would also reduce delays in effective policy action due to incomplete or incorrect results that require rolling back recommendations, while allowing scientists to focus finite resources on rigorously addressing priority research questions.
Reformed observational studies should, when published, be accompanied by appropriate explanatory text to guide their interpretation, such as the “Key Findings” section provided by some journals at the beginning of articles to provide context. Including an equally prominent, plain-language interpretation of the statistical claims and limitations of observational studies could mitigate the risk of intentional or unintentional misinterpretation. Journals could require authors to provide such lay scientific method summaries after peer review and acceptance but before publication, or work with authors and in-house staff to craft these summaries.
Patient privacy
Reforming observational studies as proposed has the potential to improve their social value, but it also poses challenges. Happily, these are relatively easy to overcome. Sharing, using and reusing data from observational studies can significantly increase their scientific and social value. However, these practices may also result in broad dissemination of participants’ protected health information. Data standards and sharing will need to align with existing ethical norms for protecting participants’ data privacy and confidentiality.
The informed consent process for observational studies is frequently less robust than that for interventional studies, creating a barrier to understanding the true meaning and extent of data sharing. even if participants are not ultimately harmed by this practice, a lack of informed discussion before data collection can undermine public trust in health research. Fortunately, appropriate data collection, management, use and reuse efforts can promote social value without compromising participants’ rights or interests. This has been demonstrated by the incorporation of research participant preferences about data storage, management and reuse into modern studies involving data-sharing activities14.
Adoption of standards
A second challenge is the adoption of data and protocol standards in observational studies by the scientific community, especially absent a coordinating body. Here, professional societies, research sponsors, regulators, journal editors and journalists can play critical roles in requiring or incentivizing the adoption of these standards. For example, professional societies could champion the standards as the best practice in the field. In the USA, the national Institutes of Health (nIH) could clarify that specific standards for observational studies are required per its new data management and sharing policy, or it could spearhead the design and implementation of observational study standards for nIH-sponsored research that set an example for the field. The US Food and Drug Administration (FDA) requires the use of specific data standards for new Drug Applications and Biologics License Applications and can refuse to receive any electronic submission whose study data do not conform to those specified in the FDA Data Standards Catalog. These programs could be extended to observational trials that support regulatory approval. Prominent journals or consortia such as the Committee on Publication ethics could require that submitted manuscripts adhere to data and protocol standards, as some journals have done by requiring that manuscripts follow the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines. Journalists could also recognize master protocol-compliant research as a standard of excellence in the field in their reporting.
Finally, lowering barriers to implementation through free access to training materials, open standards and data to facilitate research would provide an incentive to form an observational study user community. Designing new standards for observational studies and training researchers to adopt them as a part of routine scientific inquiry would also develop capacity for producing high-quality results during the next crisis.
At the regulatory level, standardizing ethical and scientific review of master protocols and data standards within and between countries would allow simultaneous collection of high-quality observational data for public health response and research use, rather than the conduct of research using data and samples collected solely for response purposes without the use of rigorous (or any) epidemiologic methods. Adoption of data standards and communications practices by journals will prepare them for the next rush of epidemic research and will prime media and policymakers to understand the statistical claims therein.
An ethical imperative
Observational studies, done properly, are a lifeline, especially in a crisis. These proposed reforms to improve the social value of epidemic observational studies require modest investment by research sponsors, professional societies, academic journals and observational trialists themselves, without raising new ethical concerns. This makes pursuing these reforms an ethical imperative, as they will save lives at low cost to the scientific and policy communities. The global pandemic that has killed more than 6 million people and fundamentally reshaped the world continues. now that the acute phase is over, there is a critical opportunity to begin planning for the next pandemic and develop protocols and policies that can also be used in response to other global health challenges.
Acknowledgements
This article was supported in part by the division of intramural research of the National Institute of Allergy and Infectious Diseases and The Clinical Center, NIH. N.G.E. received funding from the Greenwall Foundation, the Davis Educational Foundation and the US Air Force Office of Scientific Research. The views expressed here are those of the authors and do not necessarily reflect the policies of the National Institutes of Health or the US Department of Health and Human Services.
Footnotes
Competing interests
I.G.C. reports serving on the bioethics advisory board of Illumina Serves and the bioethics council of Bayer.
References
- 1.IMPACC Manuscript Writing Team. Sci. Immunol 6, eabf3733 (2021). [DOI] [PMC free article] [PubMed] [Google Scholar]
- 2.Paneth N & Joyner M J. Clin. Invest 131, e146392 (2021). [DOI] [PMC free article] [PubMed] [Google Scholar]
- 3.Rid A & Emanuel EJ The Lancet 384, 1896–1899 (2014). [DOI] [PubMed] [Google Scholar]
- 4.Cohen J Science 10.1126/science.aav3996 (2018). [DOI] [Google Scholar]
- 5.Rid A Perspect. Biol. Med 63, 293–312 (2020). [DOI] [PubMed] [Google Scholar]
- 6.Banchelli F et al. J. Clin. Med 11, 3029 (2022). [DOI] [PMC free article] [PubMed] [Google Scholar]
- 7.Alexander PE et al. J. Clin. Epidemiol 123, 120–126 (2020). [DOI] [PMC free article] [PubMed] [Google Scholar]
- 8.Alirol E et al. BMC Med. Ethics 18, 43 (2017). [DOI] [PMC free article] [PubMed] [Google Scholar]
- 9.Boudesseul J et al. Front. Public Health 9, 722458 (2021). [DOI] [PMC free article] [PubMed] [Google Scholar]
- 10.Ogburn EL et al. Science 368, 1198–1199 (2020). [DOI] [PMC free article] [PubMed] [Google Scholar]
- 11.Dean NE et al. N. Engl. J. Med 382, 1366–1369 (2020). [DOI] [PMC free article] [PubMed] [Google Scholar]
- 12.Morton SC et al. J. Clin. Epidemiol 71, 3–10 (2016). [DOI] [PubMed] [Google Scholar]
- 13.Institute of Medicine. Patient Safety: Achieving a New Standard for Care; 10.17226/10863 (National Academies, 2004). [DOI] [PubMed] [Google Scholar]
- 14.Omberg L et al. Nat. Biotechnol 40, 480–487 (2022). [DOI] [PubMed] [Google Scholar]