The culture of experimentation in epidemiology—50 years later

Experimentation in epidemiology has seen mixed attention over the past decades. Trials, including those that employ random assignment, are the most obvious examples of experimentation that epidemiologists use, and undoubtedly discuss, especially in terms of the randomized controlled trial (RCT) as a model or theoretical ambition that other study designs, including cohort and case-control studies, are intended to approximate.¹ Whether or not epidemiologists conducting cohort and case-control studies ever really intended their studies to approximate RCTs, or instead intended them to serve as stand-alone research contributions to larger bodies of work, alongside trials, is a continuing debate.^2,3 But the question remains as to whether the field of epidemiology, and the peer-reviewed journals upon which it stands, like the International Journal of Epidemiology (IJE), have fostered trials, the benefits of a culture of experimentation,⁴ and the untapped scientific curiosity and discovery that accompanies a culture of experimentation.

Three recent original research articles published in IJE would argue otherwise. They include a double-blind RCT of the effects of micronutrient supplementation on child growth in over 8000 women; a cluster-randomized, placebo-controlled trial of antibiotic distribution to children in more than 500 communities; and another cluster-randomized trial of the effects of improved water supply and sanitation on pregnancy outcomes in over 200 catchment areas.^5–7 All three of these published trials illustrate success in taking a scientific risk and testing important questions in field-based epidemiological contexts that pose unique challenges to investigators and risk in terms of the possible success, logistically and scientifically, to their studies. Two of these publications detail group-randomized trials whose study designs raise the possibility of added benefits to public health and new methodological ground for epidemiology.

Yet in 2021, the IJE published only 10 papers about trials or trial methods, including these three, totalling about 3% of its output that calendar year. The findings and topics of these trials are not the focus of this discussion. What is of interest here is that the authors took their trials to conclusion and chose to publish them in a leading epidemiology journal. Trial papers like these are often cited and can have significant impact on public health, suggesting that the IJE and other leading epidemiology journals could, and perhaps should, be leaders in publishing trials and trial methods, including community-based public health RCTs. This begs the question of how often have trials been published in the IJE over the past 50 years since the journal was established in 1972 and, if we knew this, is there something to be learned with respect to the field of epidemiology more broadly?

Are trials growing in epidemiology, or just a handful of isolated studies each year?

In 1948 Austin Bradford Hill took note of a well-regarded trial of streptomycin and tuberculosis which influenced subsequent case-control investigations into the causes of cancer.⁸ There is, of course, a long-standing research literature and classroom pedagogy illustrating the connections between RCTs and observational designs such as case-control studies.⁹ So what then has occurred as to the continued use of RCTs, and trials in general, for the field of epidemiology? I ask this with no assumption that the RCT is some sort of gold standard—that claim has been appropriately questioned for some time.² However, I do ask the question from the perspective of considering RCTs and trials as just another type of study design, with their own unique and valuable contributions to health promotion and disease prevention. If considered shoulder-to-shoulder with other unique and valuable study designs, such as cohort or case-control studies, we could, and perhaps should, assume that some level of parity in terms of the application of these various study designs has evolved over the past half-century in epidemiology.

This parity does not seem to be the case if we consider the publication activity of one of epidemiology’s oldest and most cited journals, the IJE. A 50-year PubMed review of all IJE publications bears this out. To put some numbers behind this claim, I visited and downloaded data via searches on PubMed.gov, which is hosted by the National Institutes of Health/National Library of Medicine’s National Center for Biotechnology Information.¹⁰ After restricting to years 1972 to 2021, the following searches strings were submitted: (i) (‘International Journal of Epidemiology’[Journal]); (ii) (‘International Journal of Epidemiology’[Journal]) AND trial[Title/Abstract]; (iii) (‘International Journal of Epidemiology’[Journal]) AND cohort[Title/Abstract]; and (iv) (‘International Journal of Epidemiology’[Journal]) AND case-control[Title/Abstract]. These searches effectively ascertained mention of the terms ‘trial’, ‘cohort’ and ‘case-control’ in the titles or abstracts of all IJE publications from 1972 to 2021. The year 2022 was omitted as it was incomplete at the time of this analysis. Articles included peer-reviewed submissions of original research as well as other article types such as reviews and editorials.

The immediate online reported results showed that from 1972 to 2021, a total of n = 9022 articles were published in the IJE. Over this same period in IJE, n = 230 (2.5%) articles mentioned trials, n = 1864 (20.7%) mentioned cohort studies and n = 678 (7.5%) mentioned case-control studies. On average, roughly one in 40 articles in IJE over the time period mentioned or involved trials, whereas almost one in three articles in IJE over the same time period mentioned or involved cohort or case-control studies. In only 2 years (1980 and 1984) did the percentage of articles mentioning trials exceed 5%. A clear upward trend is seen in the combination of cohort and case-control studies. This trend seems to have been driven somewhat by case-control studies over the first two decades of the time period. However, it was overwhelmingly contributed to by a growing preponderance of publications mentioning cohort studies after that. The trend for trials effectively remained flat over the entire 50-year period (Figures 1 and 2).

Figure 1.

Percentages of articles by different study designs published in the International Journal of Epidemiology over the past 50 years

Figure 2.

Percentages of articles by specific study designs published in the International Journal of Epidemiology over the past 50 years; second-order polynomial trends are shown to facilitate visual comparisons

In answering the original question about whether trials are growing in the field of epidemiology in general, it is important to state outright that this is an analysis of a single epidemiology journal. The IJE is one of five general epidemiology journals, and it is possible that the trends shown here could be different if these epidemiology journals were jointly considered (a worthwhile analysis for the future). This analysis also does not distinguish epidemiologists who publish their papers, trials or otherwise in non-epidemiologialc journals. It also does not capture papers on trials that were submitted by epidemiologists for publication and rejected. Despite these limitations, the IJE is a leading journal for the field of epidemiology, with one of the highest impact factors and readership of all epidemiology journals. Thus, valuable insights can be gleaned from analysing trends in the IJE.

Why may a lack of parity have evolved between trials and other study designs?

Mainly due to the straightforward causal identification afforded by random assignment, it has been stated that ‘all things being equal, RCTs are still preferred to observational studies’.⁸ But all things are not always equal. Intervention, manipulation or random assignment may be physically impossible (for instance, assigning people to a condition like obesity), unethical or not in a situation of equipoise (for instance, assigning people to smoke cigarettes or possess guns¹¹) or simply unaffordable.³

Even if a particular trial idea crossed these hurdles of being physically possible, ethical and affordable, there are many other reasons why investigators may choose to pursue alternative study designs in attempting to answer specific research questions. Not least among these is the considerable challenge of conducting intervention studies and trials, compared with other epidemiological study designs. Trials have a high level of scrutiny and uncertainty in terms of the logistics and timing challenges of prospective research; requiring that a small number of primary outcomes be registered ahead of time; requiring data, protocol and safety reviews; requiring highly detailed record-keeping in adhering to government regulations; and unique ethical, participant and community-relations challenges. The politics of experimentation can be especially burdensome and resource intensive.^12,13 Although other study designs are also subject to many of these same challenges in execution, all these parameters are not always as concentrated in such a relatively short period of time as they are with trials.

Epidemiology is often plagued with what is not possible, instead of the possible, compounding scepticism about launching new trials. Epidemiology students and early career scientists are raised on a steady diet of academic criticism and perfection, less so on scientific risk-taking. This does not bode well when contemplating an endeavour as challenging and unpredictable as a major RCT conducted outside the comfortable boundaries of the academy. The act of conducting a trial often requires a high level of innovation and painstaking patience in terms of creating and/or choosing an intervention or manipulation to test. Something completely new must usually be involved and, despite the reputation of discovery that science has been popularly assigned, crossing this high bar of creativity can often be quite challenging for scientists.

Another reason that the occurrence and discussion of trials may not have grown in epidemiology is the relative assurance involved in conducting cohort studies. This is supported by the data presented here for IJE. Although often expensive, cohort studies by design require long periods of study time and as such provide a certain level of longer-term job security, possibly even across generations of researchers, than intervention trials. In doing so, the establishment of a study cohort may also carry a certain level of sustained reputational benefit for the researchers and institutions that sponsor it. Trials and interventional research are often shorter-term and seemingly less rewarding.

Trials may also face difficulties recruiting subjects for inconvenient experimental interventions and end up with volunteers who do not represent their target study population. The ‘forgotten randomization’, namely population random sampling, very often takes a back seat to random assignment; external validity gets sacrificed to internal validity, although this need not be the case. Finally, epidemiologists interested in field trials of community-based interventions may be confused or turned off by the hegemonic misuse and misapplication of the word ‘clinical’ to describe all trials. The term clinical, from Greek, means ‘at the bedside’ and clearly refers to research that involves patients and is within medical or healthcare settings. Community-based intervention trials are not clinical trials. Corrections to this exclusionary mislabelling are occurring, although noteworthy misuses remain, e.g. ClinicalTrials.gov.

What might be done to redress this lack of parity between trials and other study designs?

Other fields of study and sectors of the economy have now embraced experimentation and manage to conduct thousands of trials each year. Pharmaceutical and biotech advancements hinge on RCTs as part of a larger regulatory structure, but increasingly so do social and economic policies.^4,14,15 Given this, here are five recommendations in terms of striking a better balance between trials and other study designs in epidemiology, as follows.

Have leading epidemiology journals take a proactive interest in publishing trials and other intervention studies

Structural changes by journal editors at IJE, and possibly other epidemiology journals in the field of epidemiology more broadly, could be leading actions in redressing the apparent under-appreciation of trials and intervention studies. In addition to simply recognizing the importance of trials and intervention studies, special journal sections, reviewer expertise pools and journal issues could be pursued, among other ideas. Epidemiologists have critical insights into the conduct of trials and, to take this idea a step further, epidemiologists should be leading trials much more often, maybe even redirecting their finite but valuable efforts partially away from the conduct of cohort and case-control studies to newly explore the discomfort of being the principal investigators of trials.

Grow the culture of experimentation in epidemiology as a means to public health action and greater relevance for the field

Given the limited attention that trials have received in IJE these past decades, and possibly by extension in other epidemiology journals, it stands to reason that more RCTs, done ethically with transparency and reproducibility and within a reasonable time frame to produce meaningful, actionable and perhaps policy-relevant results, may be in order. Although some questions asked in trials may also be answerable via other study designs, the fundamental participation and learning of the scientist in the treatment, intervention or action under study itself cannot be experienced in other study designs. A more robust culture of experimentation in epidemiology could thus have the added value of adding critical new scientific experience and insight while also growing the relevance, scientific standing and real-world impact of the field.¹⁶ This expansion would require that corresponding structures be created in the classroom and for early career epidemiologists who place greater emphasis and value on experiments as opportunities for learning, and not on whether any single experiment succeeds or fails.⁴

Expand organizational systems and resources that support large-scale experimentation in epidemiology

If a widespread culture of experimentation is to grow in epidemiology, systems and resources to support this will need to be expanded or implemented de novo in some settings. Although some types of trials may be relatively inexpensive, e.g. N-of-1 trials,¹⁷ the vast majority of trials are not and appropriate recognition and budgeting for the full costs of all types of trials will be required. Trials that test community-based and group-randomized interventions will need to be on par with biomedical and clinical trials in terms of the amount of resources they receive (financial and logistical), especially under the expectation that clinical and non-clinical trials be regulated under the same policies. As a related activity, the field should interrogate the public health return on investment from the decades-long proliferation of cohort studies. The redirection of resources toward trials and intervention studies may be justified as part of a broader value proposition in a resource-limited scientific environment.

Democratize experimentation and embrace its conduct beyond medical care and the clinic

More trials and studies of interventions, co-produced and initiated with communities, will go far to raise the ethical choice and use of interventions, the quality of the evidence and the standing of the field. If communities are involved from the ground up, they can guide what is of greatest value to test, know they are not being manipulated and then continue as necessary partners in subsequent implementation science activities. Increased conduct of ethical, community- and patient-engaged implementation trials of processes, not outcomes that have already been demonstrated to be effective, will be specially critical to advancing population health.¹⁸ This democratization would apply to both clinical and non-clinical trials and would be accompanied by better development of experimental mixed methods, for instance between trialists and ethnographers.¹⁹ In addition, greater attention to random sampling alongside random assignment, in the pursuit of better generalizability, as well as to field interventions and group-randomized trials like the ones referred to earlier, is warranted. The mislabelling of community field trials as ‘clinical trials’ should cease.

Elevate other study designs that seek to test interventions but may not involve experimental manipulation

Other approaches to intervention testing should also be advanced, beyond a strict focus on RCTs and other studies in which the investigators themselves perform manipulations or administer treatments. Many researchers, including social epidemiologists, have called for greater use of natural or found experiments.²⁰ Other epidemiologists have developed approaches such as Mendelian randomization or instrumental variables techniques more broadly as ways to meaningfully get at causal inference without experimental manipulation.²¹ This is often the case again for social epidemiologists who cannot plausibly manipulate study factors, although these other approaches to testing interventions also have applications elsewhere in epidemiology.

Conclusion

Fifty years later in the IJE, the culture of experimentation appears to still be severely limited. As one of the leading contemporary journals in epidemiology, this is unacceptable and may also be an indicator of what is going on in the field more broadly. There are many possible reasons for this lack of attention to and pursuit of scientific trials, despite the early origins of trials in the field and the recognition by the field of epidemiology that trials and experimentation are essential to the scientific endeavour we as a community pursue. Although the culture of experimentation need not be the dominant culture in epidemiology, its excessive under-recognition these past 50 years should be rectified in terms of a greater push for original research, methods development and pedagogy.

In its inattention to trials and intervention studies, the IJE, and possibly the field of epidemiology more broadly, may have missed out on several key opportunities and developments over the past half-century. This need not continue as the field moves ahead. A more vibrant culture of experimentation is the gateway to numerous innovations, not to mention epidemiological advances that can improve the health of populations locally and globally. In not overtly promoting a culture of experimentation, the field of epidemiology may effectively be ceding the fundamental tasks of discovery and generation of public health solutions to others—other disciplines, scientists and policy makers etc.²² In addition, the opportunity to learn unique things from the process of experimentation itself has been lost. There is value and critical learning to be gained in the kinaesthetic action of conducting experiments, over and above the cognitive work of the science. In not building a fuller body of research when possible, from observation to experiment to implementation, epidemiology may be providing incomplete evidence around solutions to our most complex health problems.

Data availability

PubMed.gov, NIH National Library of Medicine, National Center for Biotechnology Information, [https://pubmed.ncbi.nlm.nih.gov] (3 April 2022, date last accessed).

Funding

Author funded in part by National Institutes of Health grants R01HD095609 and R01AA024941.