From a pyramid to an amphitheatre (triangle to circle): embracing the totality of evidence for the postmarketing evaluation of the safety of therapeutic interventions

Abstract

Objectives

To propose a novel approach to evidence-based medicine (EBM) for medication safety. To critically evaluate the appropriateness of the traditional pyramid hierarchy of evidence for pharmacovigilance.

Key arguments

For over three decades, the traditional pyramid hierarchy of evidence has guided the practice of EBM, primarily in evaluating the efficacy of medications. While the traditional pyramid has a place in evaluating evidence for efficacy outcomes, this model is usually inappropriate for the evaluation of safety outcomes of therapeutic interventions in the postmarketing setting. In the context of pharmacovigilance, a rigid hierarchy fails to accommodate the complexity and diversity of data required for evaluation of medication safety.

This article critically evaluates the appropriateness of this traditional hierarchy of evidence for decision-making in pharmacovigilance and argues for using a non-hierarchical approach to how evidence is appraised in this context. It explores the expanding role of real-world data (RWD) and real-world evidence, which increasingly contribute to therapeutic decision-making in the postmarketing setting, noting that clinical trials and real-world studies are complementary rather than competitive. To address the limitations of the traditional EBM model in the context of pharmacovigilance, we propose an ‘amphitheatre (circular) approach’ which embraces the totality of evidence available for the safety of an intervention with no rigid hierarchy and emphasises the integration of diverse data sources to answer research questions and provide healthcare.

Conclusions

The amphitheatre of evidence offers a more flexible, inclusive and pragmatic alternative to the traditional pyramid hierarchy for assessing safety outcomes in pharmacovigilance. By embracing the totality of evidence, this model uses a more holistic approach to support clinical decision-making and protection of public health. While this review is focused on pharmacotherapeutic interventions in the postmarketing setting, the proposed novel approach may be generalised to a broader evaluation of therapeutic and diagnostic interventions.

Introduction

Spontaneous adverse drug reaction (ADR) reporting by health professionals and patients through a network of local, regional, national and international reporting systems remains the cornerstone of pharmacovigilance. Unsolicited ADR reports in a broadened postmarketing patient population, including those excluded from clinical trials, remain crucial to inform the identification of new or rare ADRs that were not identified in pre-marketing clinical trials or smaller postmarketing surveillance studies.^{1 2} However, spontaneous reporting data has well-documented limitations, including underreporting of ADRs, lack of data regarding population exposure (ie, no denominator) and missing data, which challenges the assessment of causal association based on spontaneous reporting alone.² As such, ADRs emerging as safety signals usually serve as triggers for additional investigation of possible causal relationships, using multiple sources relevant to the ADR, drug and population of interest.

Therefore, there is a clear need to move beyond spontaneous reporting to consider a diverse range of data collected through a wide variety of study designs, including studies using real-world data (RWD), in the postmarketing phase. Well-designed, high-quality, non-randomised studies may complement or surpass randomised controlled trials (RCTs) to address limitations in terms of the safety of medicines, due to larger sample sizes, longer follow-up times and more generalisable populations representative of real-world settings.3,5 Various study designs exist to monitor the ongoing safety profile of an authorised drug, each with their own strengths and weaknesses, that may complement or fill gaps identified regarding safety from clinical trials.⁶ Although limitations exist for non-randomised or observational research methods, including susceptibility to bias, confounding, missing information and misclassification, these studies remain crucial to pharmacovigilance.⁷ Indeed, the field of observational research has made progress in attempts to minimise bias and confounding, particularly through guidelines such as the Strengthening the Reporting of Observational studies in Epidemiology⁸ and Guidelines for Good Pharmacoepidemiology Practices.⁹ Despite this, observational research remains disregarded by some due to a misguided opinion that it ranks lower on the traditional evidence hierarchy for Evidence-Based Medicine (EBM). It is possibly considered simpler to discount findings of observational research in favour of those from RCTs, as the critical analysis of multiple strands of real-world evidence (RWE) requires a more ‘laborious’ analysis.¹⁰ However, when it comes to decision-making regarding the safety of medicines, each study design and data source should be considered in a complementary, equivalent and synergistic nature to best characterise a benefit-risk profile throughout the drug’s lifecycle.¹¹

Monitoring the safety of medicines is a continuous process which spans the lifecycle of a drug.¹² In drug development, studies are conducted in different phases with increasing numbers of patients to determine the safe dosing (phase I) and tolerability of the dose range in different patient groups (phase II) and to examine the efficacy and refine the benefit-risk profile of the drug before it reaches the market (phase III).¹² These studies are required to obtain the marketing authorisations for the products. Clinical trials are conducted in accordance with the study protocol. During these clinical experiments, medicines are used for limited durations in a relatively small number of patients with strict eligibility and exclusion criteria. Often, these study populations are not fully representative of the patients who would be using the medication in the real-world setting. Therefore, the postmarketing monitoring of the safety of medicines, known as pharmacovigilance, is essential to ensure the safe and effective use of drugs in real-world clinical practice and to fill gaps in the evidence present in pre-marketing trials of therapeutic interventions.¹² Postmarketing safety surveillance often includes a combination of Phase IV trials addressing both safety and effectiveness, observational postauthorisation safety studies and monitoring of reported adverse events.

Thus, pharmacovigilance decisions about a drug are often based on a number of available evidence sources,¹³ ranging from spontaneous reports of suspected ADRs reported by healthcare professionals or patients to published case reports, intensive monitoring schemes, open access cohorts such as compassionate use cohorts, unblinded extensions of RCTs or results of formal research such as real-world observational studies or clinical trials including pragmatic real-world trials with safety outcomes.¹³ Over time, there has been a gradual shift to consider a wider range of evidence as part of the decision-making process for safety of therapeutic interventions; the types of evidence used in European marketing authorisation withdrawals for a safety reason have increased.^{14 15} While spontaneous reports and published case reports remain the most frequently used data sources for regulatory decision-making in Europe, other sources of evidence have been increasingly used in pharmacovigilance decision-making in recent years. These sources of evidence provide important safety information which is essential for decision making.13,15

Types of evidence used to assess the efficacy of therapeutic interventions have been expressed in a hierarchy as part of EBM for the last 30 years, typically presented as a pyramid.¹⁶ Although various versions of the evidence pyramid have been described over time, the most ‘robust’ type of evidence is always presented at the top of this hierarchy, with the supposed ‘weaker’ study designs placed at the bottom. However, it is important to note that the traditional evidence hierarchy may not be best placed to consider the type of evidence necessary to inform decisions in postmarketing drug safety and pharmacovigilance. For example, RCTs with safety endpoints are rarely available during pharmacovigilance decisions, while observational real-world studies, which are positioned lower than RCTs in the evidence pyramid, are frequently used. As such, the applicability of the pyramidal hierarchy of evidence must be challenged when considering the safety of medicines in the wider patient population receiving therapeutic interventions in real-world clinical settings.

The hierarchy of evidence in EBM

EBM involves the identification and use of the most current high-quality evidence, combined with clinical expertise and patient values to support clinical decision-making.¹⁷ A major component of the practice of EBM is therefore critiquing and classifying the available evidence to make a judgement on its value, based on its robustness, quality or rigour. This is usually approached using a hierarchical system, which allows interpretation of the reliability of information for formulating recommendations and clinical decision-making.^{18 19}

A suggested hierarchy involving the levels of evidence for use in EBM was originally described in 1979 and has been in use in an evolving format since then.¹⁸ The original evidence hierarchy was intended as a reference to examine the quality of available evidence for an intervention.¹⁸ In 1989, Sackett et al described the hierarchical levels of evidence (table 1) in the context of EBM for the use of antithrombotic agents.²⁰ High-quality RCTs were positioned at the top of the hierarchy due to being considered the most robust study design. In an updated iteration, a well-designed meta-analysis of high-quality RCTs became the highest ranked within the hierarchy (table 1).

Table 1. Levels of evidence based on Sackett^{20 39}.

Evidence level	Study type
1a (strong)	Well-designed meta-analysis, or two or more ‘high’-quality RCTs that show similar findings
1b (moderate)	One RCT of ‘high’ quality
2a (limited)	At least one ‘fair’ quality RCT
2b (limited)	At least one well-designed non-experimental study: non-randomised controlled trial, quasi-experimental studies, cohort studies with multiple baselines, or single-subject series with multiple baselines
3 (consensus)	Agreement by an expert panel, a group of professionals in the field or a number of pre-post design studies with similar results
4 (conflicting)	Conflicting evidence of two or more equally designed studies
5 (no evidence)	No well-designed studies: ‘poor’ quality RCTs; only case studies/case descriptions or cohort studies/single subject series with no multiple baselines)

RCT, randomised controlled trial.

Due to the hierarchical nature of levels of evidence, their visual representation as a pyramid is most common. Most people in the field of EBM will be familiar with such a figure; at the time of writing, a web search for ‘evidence pyramid’ returned over 47 million results.

In the evidence pyramid, study designs are ranked by rigour, strength and precision of results (figure 1). Each level of the pyramid represents both the rigour and the quantity of the available evidence, organising study designs according to their susceptibility to bias. At the pinnacle, ‘stronger’ study designs are less intrinsically susceptible to bias (RCTs, meta-analyses) and are considered the ‘gold standard’ of evidence with the highest impact on informing clinical guidelines and patient care.²¹ Further down the pyramid, results of studies may be affected by bias and confounding, considered ‘weaker’ study design and are more abundant in the literature.^{22 23}

Figure 1. The traditional pyramid of evidence (authors’ own).

The pyramid can be further divided into sections representing filtered and unfiltered information (figure 1). Filtered evidence uses secondary data sources which have undergone screening and assessment of quality, including assessment for risk of bias as part of the study design. Synthesised evidence falls under this category and, as such, is positioned at the top of the pyramid. Conversely, unfiltered evidence includes individual studies and journal articles. When using this evidence, the user must make their own conclusions on the validity and generalisability of results and the overall quality of the research.

Limitations of the evidence pyramid for the safety of medicines

The practice of EBM is one of the major advances in the progress of medicine and healthcare. Good pharmacovigilance practice is dependent on the availability of good-quality data. Both benefit-risk evaluation of therapeutic interventions and pharmacovigilance decisions must be evidence based to protect patients effectively. However, to ensure high-quality decisions, the practice of evidence-based pharmacovigilance must evaluate the nature of the data used. RWD from non-interventional observational studies provides most of the information used in postmarketing pharmacovigilance, with a small contribution from RCTs. Therefore, for assessing the safety of medicines, the greatest limitation of the traditional evidence pyramid is that while RCTs and meta-analyses comprise the pinnacle of quality, in fact these studies may not be the most appropriate in design to study safety outcomes. Notably, Sackett²⁰ provides a caveat that, whenever possible, recommendations should be based on firm results of RCTs, but ‘for many disorders, RCTs have never been (and, arguably, never could be) carried out, and the only information for generating some of the recommendations comes from uncontrolled clinical observations.’ Real-world studies and RCTs are complementary, not competitive. Whenever possible, the most efficient method must be used to answer a specific research question. The evidence must be evaluated in its totality with no a priori assumptions about a hierarchical structure putting one method above another. For example, in some cases, a meta-analysis of RCTs is the best method to answer the question, while for other issues, individual case reports are the most appropriate data source that provides the answer. However, in the traditional hierarchy of evidence, these sources are at opposite ends of the spectrum.

Indeed, RCTs provide vital evidence in the pre-marketing phases of drug development regarding the efficacy of drugs before use in the real-world patient populations. The strength of RCTs lies in the balancing of measured and unmeasured confounding, such as prognostic factors, across intervention and control groups through random allocation.²⁴ When well designed and with a large sample size, RCTs can achieve high internal validity.²⁴ However, RCTs themselves may be susceptible to both random and systematic error (ie, chance and bias). Although treatment groups are assigned at random, non-adherence to the intervention may occur, which could result in the underestimation of treatment effect and potential adverse effects. In long-term trials, the baseline balancing of risk factors between the two groups achieved by randomisation may become progressively imbalanced with extended follow-up. As such, the benefits of random assignment become less apparent in some long-term studies due to differential attrition and change in risk factor distribution.¹⁰

Further limitations of RCTs exist when considering the safety of therapeutic interventions. For example, RCTs designed to evaluate efficacy are usually limited in statistical power to detect adverse events (AEs) due to relatively small sample sizes, lower incidence of AEs compared with efficacy outcomes and short study periods, leading to possible inaccuracies of rare or delayed AEs.⁶ Other than factors relating to statistical power, RCTs often include strict inclusion criteria usually representative of a healthier, younger population. RCTs are considered more robust when limiting the variability of confounding factors; however, this often compromises the representativeness of the study population to the broader patient population exposed to the intervention in real-world clinical settings.¹⁰ Similarly, because of strict inclusion/exclusion criteria, patients included in RCTs usually have a less severe or less complicated form of disease and fewer comorbidities and/or concomitant medications than real-world patients, limiting the external validity of any safety data collected.²⁵ For example, extremes of age (elderly and children) and pregnant and lactating women are usually excluded from clinical trial populations due to concerns about increased risk of adverse effects or poorer tolerability in these groups, or risk to the fetus or breastfeeding child associated with drug exposure. Additionally, generalisability is also often limited due to poor representativeness of sex, ethnicity, social class and nationality in RCT study populations compared with the real world.¹⁰ When a drug is authorised for use, the benefit-risk balance is based on effects of the therapeutic intervention on patients selected for inclusion in RCTs; thus, it may not be generalisable to patient groups who were excluded from RCTs or those who receive the intervention for a longer duration than the RCT. Furthermore, patients enrolled in RCTs receive the study medication for the indications listed in the study protocol, while in the real world, off-label use can occur with implications for both the safety and efficacy of the medicinal product. The above limitations should also be considered when interpreting results of meta-analyses comprising only or mostly RCTs. Although a powerful tool to pool data on the efficacy of drugs, elements that influence the validity of individual studies could affect meta-analyses of their results, which should be considered when evaluating findings of meta-analyses.²⁶

Importantly, conducting RCTs after a drug is marketed may be considered unethical where there are known risks associated with the investigational drug, or if randomisation to a placebo or an alternative with a less favourable benefit-risk balance denies patients access to the drug they could benefit from. To achieve marketing authorisation, medicines should have demonstrated efficacy within the target population; depriving patients from receiving an effective therapeutic intervention is likely to be inappropriate, particularly when there are no alternatives with an acceptable benefit-risk balance. While randomisation and blinding are essential to the internal validity of RCTs, the aim of a clinical trial is to establish the efficacy of an intervention where there is insufficient and/or an absence of evidence.²⁷ This makes the process of randomisation inappropriate or impractical in postmarketing studies. However, lack of randomisation also remains a source of bias in observational research which needs to be considered when interpreting safety findings.¹¹

Recent developments in RCTs such as the introduction of Umbrella and Basket trials²⁸ and adaptive studies—for example the Recovery trial for the treatment of COVID-19 therapeutic agents²⁹—need to be evaluated in terms of bridging the gaps between real-world studies and RCTs.

The importance of real-world evidence in EBM

The use of RWE can overcome some of the limitations of RCTs discussed above. When well-designed, observational studies can answer research questions effectively. The choice of study design should consider the study’s requirements, including the medicinal product and/or AEs of interest. Therefore, there is no one-size-fits-all approach to considering the value of evidence: it depends on the issue under investigation. The advantages of real-world observational studies include representation of real-world usage with few exclusions, the possibility to include large study populations and the ability to use long observation periods. Nonetheless, disadvantages of observational studies include limitations relating to their internal validity, such as bias, confounding, missing data and misclassification. A careful critique and evaluation of studies of observational design is required to assess the impact of their inherent limitations on their findings.

In recent years, EBM has seen a major shift in the type and complexity of drugs requiring safety monitoring, including biologics and gene therapy, with suitable adjustments in the types of evidence and methodologies used in both clinical trial and observational research settings. The rapid growth in the fields of genomics, immunology and epigenetics, alongside advancements in big data, machine learning, artificial intelligence and computational biology, will catalyse change. Although some developments have been made in clinical trial methodologies, such as master protocols allowing for several sub-trials and even parallel observational research, studies using RWD are increasing in popularity. A multitude of sources of RWD are emerging, including data collected through wearable technology and social media, that increases the size and richness of data available for RWE generation.^{7 30}

This demand for increased RWE applies not only to drugs during the development phases but also remains important throughout the product’s lifecycle. In 2023, biologics accounted for approximately 40% of new drugs evaluated by both the FDA and EMA.^{31 32} However, monitoring the ongoing benefit-risk profile of biologics in the postmarketing period requires additional considerations compared with small molecule drugs.³² Biologics tend to be prescribed for complex and often rare clinical conditions, meaning data from pre-marketing clinical trials (considered the gold standard in the traditional evidence pyramid) cannot necessarily be extrapolated to patients using the medicine in real-world clinical settings. Therefore, considering multiple diverse sources of observational data generated from real-world use of the intervention provides a more suitable body of evidence to inform the safety and effectiveness of drugs developed via emerging technologies.

Embracing the totality of safety information

EBM has evolved to Evidence-Based Practice (EBP) which is broader than the earlier definitions of EBM. The Sicily statement on EBP requires that decisions about healthcare are based on the best available, current, valid and relevant evidence.³³ In addition, the statement emphasises the role of patients in the decisions made about their health. According to the Sicily Statement, the evidence used in EBM is comprehensive and beyond adherence to the evidence hierarchy.³³

Considering the limitations of RCTs, pharmacovigilance has evolved to include the science and activities essential to fill any gaps raised from clinical trials regarding the safety profile of the intervention, while further characterising the safety profile when use is expanded to a greater and more diverse population.³⁴ To protect patient health, pharmacovigilance decisions informing regulatory action should be based on all available evidence, which includes current, high-quality evidence from suitable RWD sources. Common sources of diverse RWE for postmarketing safety monitoring are summarised in table 2.

Table 2. Examples of common sources of RWE for postmarketing safety monitoring^{2 4 5 35}.

Source of data	Examples
Spontaneous reports of ADRs	National or regional spontaneous reporting databases Global spontaneous reports databases
Electronic health records	Specialist ambulatory care records Databases for healthcare systems Primary care medical records Pharmacy dispensing records Hospital inpatient, outpatient and/or discharge records Emergency care records
Administrative data on healthcare service/claims	Administrative healthcare claims databases Diagnostic tests and procedures reimbursement records Exemption from co-payment exemptions
Observational cohorts (with collection of primary data)	Postauthorisation safety studies (PASS) or postauthorisation effectiveness studies (PAES) Biobanks (with prospective follow-up)
Disease/condition-specific registries	Disease registries Cancer registries Pregnancy registries Congenital anomaly registries Death registries
Drug/device registries	Drug registries Medical device registries
Patients	Patient-reported outcomes: Health surveys or questionnaires Interviews and/or focus groups registries Patient-generated health data: Wearable medical/personal devices o Mobile applications Social media Internet-based tools
Healthcare professionals	Survey of prescribers Chart review Clinical audit

ADR, adverse drug reaction; RWE, real-world evidence.

Significant progress has been made to increasingly consider multiple types of information for pharmacovigilance decision-making through various initiatives such as the EMA’s Real-World Evidence Framework to Support EU Regulatory Decision-Making.³⁵ In Europe between 1999 and 2016, the evidence sources used for regulatory decision-making increased, with a clear shift towards the inclusion of RWD to bridge the gap between RCTs and the real-world conditions of clinical practice.13,1535

Introducing the ‘amphitheatre (circular) approach’

In ancient Rome, amphitheatres were venues that brought people from different places and social classes together. The events held within these grand structures were a shared experience that surpassed social boundaries, allowing people from different backgrounds to come together and enjoy the spectacles. Although the primary purpose of amphitheatres was entertainment, they also served as a platform for learning and sharing knowledge. A similar platform should be employed for the safety of medicines; by bringing information from disparate sources into the same discussion, we can learn and advance EBM.

This concept is not new; in fact, it has long been the responsibility of healthcare professionals, scientists and researchers to consider the totality of evidence when addressing research questions and decisions concerning the safety of the public. However, the traditional evidence pyramid, where safety data offered by study designs other than RCTs are largely undervalued and/or disregarded, needs to be deconstructed in the era of RWE. We therefore propose that evidence should be represented not as a hierarchical pyramid but as an amphitheatre that encompasses the totality of all relevant sources of RWE at a similar level of importance, considering the individual strengths and weaknesses of each source in the context of the research question or issue under consideration. This ‘amphitheatre approach’ draws on ideas of Rawlins³⁶ in the 2008 Royal College of Physicians’ Harveian Oration that emphasised the illusory ‘rules’ of evidence hierarchies and that decision-makers should instead use clinical and scientific judgement to appraise evidence from both experimental and/or observational sources, where best for an intervention. Additionally, there have been previous suggestions for new representations of the evidence pyramid, ranging from the inclusion of ‘wavy’ lines between the study designs rather than solid, based on various domains of quality (figure 2),¹⁶ to a circular model (figure 3) that explores the efficacy of an intervention complemented by observational or non-experimental methods to describe applicability to clinical settings.^{37 38} The amphitheatre approach advances these ideas to embracing the totality of evidence from diverse RWD sources, considering no one source of evidence more valuable than another as each data source has a different message, use and applicability and/or generalisability to populations in the real world. The amphitheatre approach integrates a diverse range of sources, ensuring that all are represented ‘in the arena’, integrating strategically selected RWD into a robust and high-quality evaluation of the benefit-risk profile most applicable to the real-world clinical setting.

Figure 2. An adapted evidence pyramid with ‘wavy’ lines (Murad, 2016).¹⁶.

Figure 3. A circular hierarchy of evidence (Walach, 2006).³⁸ RCT, randomised controlled trial.

The amphitheatre approach aims to disrupt traditional evidence hierarchies for EBM and to progress the understanding of the benefit-risk profiles of therapeutic interventions more efficiently. As the ancient pyramids were replaced with other structures in the history of human civilisation, the pyramid of evidence should be replaced by a modern approach to help advance EBM. However, for this approach to truly have a lasting impact, a shift is needed in the attitudes and practices of a wide range of multidisciplinary actors in the interests of the benefit-risk of healthcare interventions, particularly from physicians and researchers who rely solely on data from RCTs and clinical trials. The adoption of an amphitheatre approach promises a positive impact on public health, bringing together the totality of evidence more efficiently for the safe use of therapeutic interventions in clinical practice.

Conclusions

There is no doubt that the application of EBM has contributed to major advances in medicine and healthcare. Although the pyramidal hierarchy of evidence has a place in EBM, when considering the safety outcomes and benefit-risk balance of therapeutic interventions used in real-world clinical settings, this hierarchy is not the most appropriate approach. Considering the increasing quantity and quality of other data sources, including RWD increasingly contributing to regulatory decision-making, there is a clear need for a new approach to embrace these diverse sources for the evaluation of safety outcomes in EBM. The amphitheatre approach involves multiple sources of evidence bringing value to the arena, each with their respective strengths and weaknesses. With these considered, the ongoing evaluation of the benefit-risk of drugs and other therapeutic interventions in real-life populations may be based on the most appropriate data sources reflective of real-world use. This shift will require a change in the mindset of clinicians and the research community to challenge existing perceptions and opinions in EBM. However, as the ancient pyramids were replaced with other structures in the history of human civilisation, we should embrace new ways of thinking to most efficiently enhance decision-making in biomedical research and healthcare.