Adapt or die: how the pandemic made the shift from EBM to EBM+ more urgent

Abstract

Evidence-based medicine (EBM’s) traditional methods, especially randomised controlled trials (RCTs) and meta-analyses, along with risk-of-bias tools and checklists, have contributed significantly to the science of COVID-19. But these methods and tools were designed primarily to answer simple, focused questions in a stable context where yesterday’s research can be mapped more or less unproblematically onto today’s clinical and policy questions. They have significant limitations when extended to complex questions about a novel pathogen causing chaos across multiple sectors in a fast-changing global context. Non-pharmaceutical interventions which combine material artefacts, human behaviour, organisational directives, occupational health and safety, and the built environment are a case in point: EBM’s experimental, intervention-focused, checklist-driven, effect-size-oriented and deductive approach has sometimes confused rather than informed debate. While RCTs are important, exclusion of other study designs and evidence sources has been particularly problematic in a context where rapid decision making is needed in order to save lives and protect health. It is time to bring in a wider range of evidence and a more pluralist approach to defining what counts as ‘high-quality’ evidence. We introduce some conceptual tools and quality frameworks from various fields involving what is known as mechanistic research, including complexity science, engineering and the social sciences. We propose that the tools and frameworks of mechanistic evidence, sometimes known as ‘EBM+’ when combined with traditional EBM, might be used to develop and evaluate the interdisciplinary evidence base needed to take us out of this protracted pandemic. Further articles in this series will apply pluralistic methods to specific research questions.

Introduction

Two and a half years into the COVID-19 pandemic, it is time to take stock. What began as—we assumed—an acute respiratory illness rapidly sweeping the world has become a prolonged global crisis with medical, social, economic and political dimensions. The SARS-CoV-2 virus has proved tenacious and shifting. It causes a complex multisystem disease which disproportionately impacts minority ethnic groups and the sick, poor, old and disadvantaged. It produces significant mortality and—in some—prolonged sequelae. Effective and safe vaccines were produced rapidly, but uptake has been patchy and highly transmissible variants continue to spread and mutate. Coordinated disinformation campaigns have weakened the public health response.

Despite a quarter of a million scientific papers on COVID-19, some basic issues remain contested. How exactly does the virus spread? How effective are non-pharmaceutical interventions—masks, distancing, closure of buildings, remote working and learning, lockdowns—in reducing transmission, and what are their trade-offs? How can we make schools, hospitals and other public buildings safe? How can we protect workers and the public without closing down the economy? How cn we reduce the shocking inequalities that have characterised this pandemic?

This paper explains why we need to go beyond lip-service to evidential pluralism to fully answer those questions. While acknowledging the primacy of randomised controlled trials (RCTs) and meta-analyses of RCTs for estimating the efficacy of drugs and vaccines, we highlight and extend the arguments made by some scholars within the evidence-based medicine (EBM) movement for ‘EBM+’,^{1 2} defined as an approach which systematically considers mechanistic evidence (studies which aim to explain which factors and interactions are responsible for a phenomenon³) on a par with probabilistic clinical and epidemiological studies. Our central argument is that for some aspects of the pandemic, especially those characterised by a combination of complexity (multiple variables interacting dynamically with a high degree of uncertainty), urgency (decisions needed in days not years) and threat (the consequences of not acting could be catastrophic), mechanistic evidence has been mission-critical and RCTs difficult or impossible. Thousands of lives were likely lost as a result of what was incorrectly claimed to be an “evidence-based” approach—dismissing or downgrading mechanistic evidence, overvaluing findings from poorly designed or irrelevant RCTs, and advocating for inaction where RCT evidence was lacking. The pandemic is an epistemic opportunity for the EBM movement to come to better understand, debate and embrace EBM+.

Evidence hierarchies: not fit for all purposes

Every student of EBM is taught that the hierarchy of evidence—with meta-analyses of RCTs at the top and so-called ‘less trustworthy’ evidence (case studies, mechanistic evidence, expert opinion) at the bottom (figure 1A)⁴ —is a useful heuristic but does not apply in all circumstances. But guidance to apply the hierarchy flexibly to take account of—for example—the nature of the research question, the degree of complexity involved, whether RCTs of the necessary size and scope are actually feasible, and whether policy decisions can or should be put on hold pending definitive results—has been honoured more in the breach than the observance. From Guyatt et al’s⁵ paper announcing EBM as a ‘new paradigm’ in medicine to their 2008 Grading of Recommendations, Assessment, Development and Evaluation statement (which offers eight domains of strengths and limitations of the overall evidence base, including observational studies as well as RCTs)⁶ and the Oxford Centre for EBM 2011 Levels of Evidence Document,⁷ there remains within EBM orthodoxy an explicitly hierarchical relationship between probabilistic clinical and epidemiological evidence (invariably placed at the top of the hierarchy) and the kinds of evidence which help establish causal mechanisms (invariably placed at or near the bottom).

Figure 1.

One version of EBM’s hierarchy of evidence, showing evolution over time to encourage a more flexible approach. (Reproduced under CC-BY-4.0 Licence from Murad et al ⁴). EBM, evidence-based medicine.

The cornerstone of EBM as taught to students—ask focused questions in population-intervention (or exposure)-comparison-outcome (PICO) format; search for RCTs and, if these are unavailable, work down the hierarchy of evidence; appraise studies for risk of bias; and combine them using meta-analysis—was developed for a stable context in which predictions based on yesterday’s research map unproblematically onto today’s policy questions (eg, what actions should we take to prevent and manage coronavirus?). Despite their theoretical strengths for some kinds of question, and despite heroic achievements in the pandemic with fast-track platform trials such as RECOVERY,⁸ PRINCIPLE⁹ and PANORAMIC,¹⁰ RCTs are not a panacea (table 1).¹¹ Modifications of the hierarchy of evidence (eg, by separating the layers with dotted or wavy lines, as in figure 1B, and using systematic review as a lens through which primary evidence is interpreted, as in figure 1C ⁴ fall short of the epistemic shift that is now needed.

Table 1.

Strengths and limitations of RCTs and meta-analyses for pandemic research

‘Quality’ features of RCTs and meta-analyses of RCTs

Reason why these designs are sometimes problematic in a pandemic

Randomisation, if successful, will ensure that comparisons between exposed and unexposed groups are fair—that is, that groups are effectively identical in all respects except for their exposure to the drug, vaccine or public health intervention in question. Clinical equipoise and informed consent helps ensure that a trial is ethical. Factorial, pick-the-winner, ring vaccination and other advanced RCT designs could potentially add helpful information about complexity and speed the generation of knowledge in a crisis. Meta-analysis allows objective and systematic assessment of individual trials and appropriate weighting by risk of bias. ‘Living’ systematic reviews can provide timely summaries of RCT evidence, as occurred with drugs and vaccines.

Because communicable diseases (by definition) spread through communities, it may be impossible to ensure that cross-contamination of arms does not occur. Even rapid RCTs take time to plan and execute; in most cases even a rapidly conducted RCT will generate less timely information than real-time analysis of observational data already in hand. It may be impossible to power trials adequately to evaluate the full range of interactions of interest. It may be unethical to avoidably expose study participants to a virulent pathogen RCTs are not the preferred study design for questions of aetiology (eg, ‘how does the SARS-CoV-2 virus spread?’) Meta-analysis involves subjective judgments by the reviewer about how rigorously an effect was measured and how well context and confounders were taken into account in primary studies. If the individual RCTs in a meta-analysis include unfair comparisons, or if their findings are incorrectly combined, flawed summaries, false reassurance and inappropriate policy could result.

RCTs, randomised controlled trials.

Strengthening causal claims through mechanistic evidence

Mechanistic evidence includes a wide range of designs, including in vitro experiments, biomedical imaging, autopsy, established theory, animal experiments, aerosol science, engineering research and simulations.³ Like all evidence, mechanistic studies can be of variable quality and definitiveness. Table 2 shows some preliminary criteria for grading mechanistic evidence, based partly on previous studies,^12–14 which would benefit from further elaboration.

Table 2.

A suggested hierarchy of evidence for mechanistic evidence (drawing partly on previous publications)^{12 14}

Level 1 (strongest)	Necessary and sufficient conditions for causality (eg, multiple features of the causal chain) supported by multiple independent studies, confirmed by multiple independent research groups using accepted best research methods. No high-quality disconfirming studies found.
Level 2	Indicators of causality (eg, more than one feature of the causal chain) supported, especially if by more than one method and confirmed by independent studies. No high-quality disconfirming studies found.
Level 3	Suggestion of causality (eg, one feature of the causal chain) supported by multiple independent studies. No high-quality disconfirming studies found.
Level 4	Sparse evidence supporting feature(s) of the causal chain. Disconfirming studies found but these are not definitive.
Level 5 (weakest)	No supporting studies. High-quality disconfirming studies found.

No one kind of evidence stands alone. Rather, as encapsulated in the Bradford Hill indicators^15–17 (table 3), evidence of mechanisms must be combined with probabilistic evidence from clinical trials (and, where appropriate, evidence from non-randomised comparative and observational studies) to make a strong case for causality.¹³ This is because real-world circumstances often differ from the ones in which a probabilistic estimate (such as an effect size in an RCT) was demonstrated; evidence supporting a plausible mechanism affirms (or not) that the causal relationship is likely to be stable across settings.¹³ The presence of high-quality and consistent mechanistic evidence greatly increases the external validity of an RCT finding; absence of such evidence should make us question RCT findings.³ Public health interventions involve two kinds of mechanisms: the upstream causes of disease (for COVID-19 ‘mechanisms involving family structures and interaction patterns, occupational behaviour, urban density, housing occupation and overcrowding, workplace and retail environment structures and organisation, as well as local social, economic and cultural variation’—Aronson JK, page 688² and the ‘causes’ through which an interventions might act (eg, attitudes, beliefs, capability, personal resources).²

Table 3.

Bradford-Hill indicators of causality, showing the importance of both probabilistic evidence (eg, from RCTs) and mechanistic evidence (eg, from laboratory studies)

Indicator	Explanation	Type	Preferred types of evidence
Strength of association	A strong association is more likely to be causal than a weak one	Both	RCTs or unbiased observational studies (eg, longitudinal cohort) which allow calculation of a measure of association such as relative risk or OR, and adjustment for confounding.
Consistency	Multiple observations made by different observers with different instruments mean the association is more likely to be causal	Both	Systematic review. Exploration of heterogeneity via tools such as meta-regression may help identify and quantify inconsistency in effects across studies.
Specificity	If an outcome is best predicted by one primary factor, the causal claim is more credible	Neither	A problematic indicator, as it is common for a single exposure to be causally associated with multiple outcomes (eg, tobacco smoking and heart disease, stroke, cancer, etc).
Temporality	A cause must precede an effect	Both	Any longitudinal design for example, clinical trial, non-cross-sectional observational study.
Biological gradient	There should be a direct ‘dose-response’ relationship between the independent variable (eg, a risk factor) and the dependent one (eg, people’s status on the disease variable)	Both	Basic science (eg, toxicology-based risk analysis), varying-dose RCTs, unbiased observational studies with adjustment for confounding.
Plausibility	An association is more likely to be causal if there is a rational and theoretical basis for it	Mechanistic	Basic science
Coherence	An association is more likely to be causal if it coheres with other knowledge (ie, does not conflict with what is known about the variables under study and there are no plausible competing theories or rival hypotheses)	Both	Systematic review; integration of both mechanistic AND probabilistic evidence is key here.
Experimental manipulation	Any related research that is based on experiments will make a causal inference more plausible	Both	Basic science or RCT (which may be considered ‘epidemiological experiments’, as exposure is defined by the investigator).
Analogy	Sometimes a commonly accepted phenomenon in one area can be applied to another area	Both	Requires a broad understanding of all relevant fields; potentially subject to logical fallacy.

Adapted from Hill and Bradford Hill,¹⁵ incorporating important subsequent insights from Russo and Williamson¹⁶ and Fedak et al. ¹⁷

RCT, randomised controlled trial.

Mechanistic evidence is inherently explanatory. It helps address the question ‘how might this effect be produced?’. In the early months of the pandemic, the grammar of our research was prematurely fixed and PICO-constrained (eg, ‘what is the effect size of masks in preventing respiratory infections?’). This narrowed our focus onto probabilistic studies of interventions (‘masks-on’), comparisons (‘masks-off’) and predefined outcomes (infections in the wearer) expressed in statistical terms (see masking example in box 1 ^18–28). We suppressed our scientific imagination. We failed sufficiently to wonder at the novelty of the disease and the significance of its unique patterns of spread such as super-spreading events, overdispersion, transmission in the absence of symptoms, and indoor predominance—all of which should have raised mechanistic hypotheses about a predominantly airborne mechanism of transmission.²⁹

Box 1. Public masking—‘naked statistics’ or evidential pluralism?

Mortality rates differed dramatically between Asian countries which introduced public masking very early in the pandemic and Western countries which did not.¹⁸ But was this relationship causal?

The ‘naked statistics view of evidence-based medicine (EBM)’¹² would answer this question exclusively from randomised controlled trial (RCT) evidence, rejecting all other study designs as less trustworthy. The DANMASK researchers, for example, randomised 6024 people to being advised to wear masks outside the home or not.¹⁹ The primary outcome (infection with SARS-CoV-2) occurred in 42 people (1.8%) of the intervention arm and 53 (2.1%) of the control arm—a difference that was not statistically significant. This finding was described by some in the EBM community thus: ‘now that we have properly rigorous scientific research we can rely on, the evidence shows that wearing masks in the community does not significantly reduce the rates of infection’.²⁰

DANMASK had numerous design flaws²¹: it was underpowered; it focused only on whether masks protect the wearer rather than whether they reduce overall community transmission; it occurred at a time of extremely low incidence of COVID-19; it had an intervention period of only 1 month; and—as the authors themselves observed¹⁹—the 95% CIs were compatible with a 46% reduction to a 23% increase in infection with masking (ie, it was inconclusive rather than negative). An editorial in the Cochrane Database of Systematic Reviews encouraged policy-makers to adopt the precautionary principle and ‘act on incomplete evidence’ for several reasons: the urgency of the situation (and the risks of doing nothing), the multiple complexities and complex chains of causation for this particular behavioural intervention, and the potential benefits, over time, of even a non-statistically significant reduction in transmission rate.²²

An EBM+ approach would consider the DANMASK trial and other RCTs (such as a large community masking study in Bangladesh villages²³ on their merits, but also take account of studies which throw light on the mechanism by which masking might work—including.

• Laboratory, real-world case studies and mathematical modelling studies supporting an airborne route of transmission.²⁹

• Engineering studies which established the filtration properties of different kinds of masks.^{24 25}

• Studies on the psychological, sociocultural and wider structural causes of whether people actually wear masks.²⁶

• Studies confirming that masking does not appear to have other unintended psychosocial consequences such as risk compensation²⁷ or increased face touching.²⁸

When the pandemic began, we immediately sought ‘robust’ evidence from new RCTs.³⁰ But overapplication of this heuristic led to narrowly focused reviews,³¹ excluded experts from other disciplines, and closed our minds to the mechanistic explanations we could have built from well-conducted laboratory studies,³² animal studies,³³ modelling studies,³⁴ engineering studies³⁵ or careful analysis of real-world events³⁶—not to mention prepandemic studies on the transmission of comparable respiratory viruses.³⁷

Dynamic mechanisms: complex interventions in complex systems

A complex intervention (eg, facial protection with masks or respirators) consists of multiple components acting independently and inter-dependently at multiple levels (eg, material artefact, person, group, organisation, system), such that it is difficult to identify the ‘active ingredient’.³⁸ Facial protection can be home-made or produced to formal technical standards; humans may agree—or not—to mask; masking may be socially expected, organisationally required or legally mandated; the threshold case rate for introducing or reintroducing masking may vary; global supply may be unreliable; masking is often paired with other interventions, introducing confounders.

A complex system—such as the one for delivering public health (which includes government, health service providers and the public)—is characterised by a number of features:

• Emergence (the system evolves over time, in ways that cannot be fully anticipated).

• Adaptation (the system adjusts to accommodate change).

• Feedback (either positive, where change in one factor reinforces another factor, or negative, where change in one factor diminishes another).

• Self-organisation (eg, a public health department adapts a national programme to suit local constraints and priorities).³⁹

Particular outcomes emerge as a result of multiple interactions in the system.

Whereas studies of drug and vaccine efficacy can be interpreted within the traditional biomedical paradigm (table 4, column 2),⁴⁰ findings from such trials may not map to complex, real-world settings for multiple social and behavioural reasons. Studies how complex interventions generate outcomes in complex systems require a new paradigm (table 4, column 3) with designs that can capture dynamic change, accommodate non-linearity and embrace uncertainty.⁴¹

Table 4.

Traditional biomedical versus complex systems paradigms (reproduced and adapted under CC-BY-4.0 licence from Greenhalgh and Papoutsi)⁴¹

	Traditional biomedical paradigm	Complex systems paradigm
Scientific truth viewed as	Singular (ie, there is one set of facts which will be found by using the correct scientific methods)	Multiple (ie, there are many legitimate scientific perspectives on a problem, so multiple methods may be needed)
Goal of research	Establishing the truth, which is seen as universal and generalisable	Exploring tensions; generating insights; revealing multiple perspectives
Assumed model of causality	Linear, cause-and-effect; a fixed input has a fixed effect size	Emergent: many interacting influences, but none has a fixed ‘effect size’
Typical format of research question	‘What is the effect size of the intervention on the predefined outcome, and is it statistically significant?’	‘What combination of influences has generated this phenomenon? What does the intervention of interest contribute? What happens to the system and its actors if we intervene in a particular way? What are the unintended consequences elsewhere in the system?’
Good research defined in terms of	Methodological rigour, standardisation, precision	Strong theory, flexible methods, pragmatic adaptation to changing circumstances
Most valued study designs	Randomised controlled trials, meta-analyses	No single design is ‘most valued’. A combination of designs—for example, multilevel intervention studies, naturalistic case studies, modelling studies—is preferred
Purpose of theorising	Disjunctive (simplification and abstraction)	Conjunctive (drawing together different parts of a complex problem)
Approach to data collection	Continue collecting data until the dataset is complete and agreed	Data will never be complete; decisions must sometimes be made on the basis of incomplete or contested data
Analytic focus	Dualisms: comparing A with B; finding the influence of X on Y	Dualities: inter-relationships and tensions between A, B, C and other emergent phenomena

One approach to researching these generative mechanisms is the in-depth case study, a mixed-method design in which data collection, interpretation and synthesis are oriented to producing a rich description of a phenomenon in context.⁴² Hypotheses are explored and tested by constructing a plausible story (narrative synthesis) in which all data—qualitative and quantitative—are accounted for. Local public health interventions, even when part of national policy, are typically path-dependent (shaped and constrained by what has happened here previously) and emergent (they grow organically over time); they tend to be implemented and improved through testing and iteration in the real world.⁴³ As Ogilvie et al ⁴³ have pointed out, this pragmatic approach is not ‘weaker’ than an RCT but complementary to it (figure 2). Observations from real-world case studies inform hypothesis-driven questions about the impact of particular interventions under controlled conditions, which can be tested in RCTs. Conversely, RCTs may produce important findings which inform development and real-world testing of an intervention. These authors contrast the ‘brick wall’ of a meta-analysis (in which every primary study contributes a similar ‘brick’) with the ‘dry stone wall’ of a mixed-method narrative review⁴⁴ (in which different kinds of primary evidence are combined to illuminate a problem from multiple angles).⁴⁵

Figure 2.

Combining controlled trials with evaluation of real-world action via case studies. (Reproduced and adapted under CC-BY-4.0 Licence from Ogilvie et al ⁴³). RCT, randomised controlled trial.

Generative mechanisms in complex systems can also be studied quantitatively using modelling. Models (simplified versions of reality) can be used for a variety of purposes—to describe epidemic growth, estimate the likely impacts of interventions, simulate airflow inside buildings, and synthesise data from multiple sources in a manner analogous to systematic review and meta-analysis.⁴⁶ Model projections can be useful for presenting plausible counterfactuals, and for quantifying the impact of uncertainty. Policy-makers’ use of modelling studies permit considering multiple ‘what if’ scenarios and sensitivity assessment as policy options develop. For communicable disease outbreaks, models can capture the non-independence of events that are the defining attribute of these diseases, and allow quantification of indirect effects on individuals who do not receive a given intervention⁴⁶—for example creation of herd effects via immunisation.

As with all research designs, real-world case studies^{42 47} and modelling⁴⁶ can be done well or badly, and the findings of such studies can be more or less relevant to a policy decision that is being contemplated in a different context or setting.

Mechanisms in the physical world: standards as evidence

Engineers use knowledge of the natural world to craft designs which benefit and improve society. That design process takes account of physical and technical properties and capabilities along with social needs and assessment of impact. Central to the science of engineering is a high reliance on standards, which are continuously developed to improve on the state of the art.⁴⁸ Indeed, quality in engineering evidence is defined in terms of relevant accepted standards and codes (eg, on indoor air quality controls for healthcare facilities).⁴⁹

New engineering problems are addressed largely by creatively applying and actively improving existing standards and models. Designing medical equipment such as ventilators or diagnostic imaging systems does not begin de novo but with a standard selection of component products built and tested to meet established standards. Such devices are not iteratively constructed and tested to confirm success. Most aspects are determined analytically using numerical models and design tools. Designs are constrained to meet established national standards that dictate performance and safety requirements. RCTs have no place in such design work.

Engineering knowledge rarely generates a single ‘truth’. Rather, it tends to produce a range of options which vary in how well they meet social needs and satisfy design and resource constraints. There is usually more than one way of designing a medical device to meet safety and performance standards. The success of the engineering method is illustrated by the rarity of harmful failures of engineered systems. When rare failures occur, there follows a detailed investigation of causes, and application of that knowledge to future designs.⁵⁰

Building engineering contributes crucial evidence on how to protect people from airborne pathogens, since the use and impact of personal protective equipment is influenced by (among other things) the quality of indoor air.³⁵ Furthermore, the mechanisms by which filtering facepiece respirators work are long-established and well-understood. Robust certification systems, standards and workplace usage protocols for respirators currently minimise exposure to occupational hazards for millions of workers worldwide. Nobody would propose an RCT comparing these products with less effective protection to ‘prove’ their value in protecting against chemical contamination in a lead smelter—yet such studies are currently being proposed for determining whether respirators protect against biological hazards in healthcare environments.

Human mechanisms: social and behavioural evidence

The human (or social) sciences seek to explain why people act in the way they do, taking account of wider influences such as cultural norms or economic constraints. The Medical Research Council’s framework on developing and evaluating complex interventions emphasises the need for developing a programme theory that takes account of the many influences on human (and organisational and system) behaviour.³⁹

One such framework is the capability-opportunity-motivation-behaviour model,⁵¹ whose lead developer, Professor Susan Michie, was an adviser to UK’s Scientific Advisory Group on Emergencies. In relation to wearing masks or respirators, people may not be physically able to do so, or they may lack key knowledge (eg, they may wrongly consider themselves at low risk, wrongly believe that cloth masks, surgical masks and respirators are equally effective, or underestimate the importance of a good fit); because they lack motivation (including conscious beliefs and subconscious emotional reactions), or because they lack opportunity (eg, their employer bans masks or social pressures convey negative norms).

Other frameworks, such as those based on social ecology theory, place greater emphasis on the multiple layers of influence which shape and constrain human behaviour (including individual, interpersonal, organisational, community and public policy).⁵²

Without a careful exploration of these wider influences, experimental studies of complex interventions may produce misleading findings.

Conclusion

The pandemic brought complex and fast-changing problems. We have argued that EBM’s traditional methods and quality standards, which favour probabilistic evidence from clinical trials, must be extended to place greater emphasis on other sources of evidence. We are not, however, arguing for an anything-goes approach to evidence or what one reviewer of an earlier draft of this paper called ‘a throwback to the 1970s’—a time before much of the rigorous scholarship of EBM had been built.

An urgent debate is needed within the EBM/evidence-based healthcare community on how, in what circumstances and to what extent EBM should evolve into EBM+. We hope to contribute to that debate with further articles on how an EBM +approach can enhance and extend EBM’s important contribution to pandemic science.

Ethics statements

Patient consent for publication

Not applicable.

Ethics approval

Not applicable.