ABSTRACT
Data from sources other than traditional randomized clinical trials are known as real‐world data (RWD), and the evidence derived from the review and analysis of RWD is known as real‐world evidence (RWE). RWD and RWE are used increasingly throughout the lifecycle of medicinal products to provide evidence about their effectiveness and safety. Recent regulatory guidance about RWE and approvals based on the use of RWE to demonstrate beneficial effects of products have created an urgency to develop generally accepted processes that promote trust in the evidence‐generation process. A recent report from a working group of the Council for International Organizations of Medical Science (CIOMS) describes the use of RWE for decision making in the lifecycle of medical products, describes RWD and data sources, discusses key scientific considerations in the generation of RWE, and discusses ethical and governance issues related to the use of RWD. This paper provides a high‐level summary of this report. More work remains to be done to globally harmonize practices and guidance for using RWD and RWE for regulatory decision making, thereby maximizing the benefits they can bring to patient care and public health.
Keywords: decision making, lifecycle, medicine, public health, real‐world data, real‐world evidence, regulation
Summary.
In recent years, real‐world evidence use has evolved, driven by recognition and acceptance by regulators, payers, and health technology assessment bodies to answer specific research questions.
Each real‐world data source has its strengths and limitations, and scientific evaluation of the fitness of a real‐world data source for a given study is essential in choosing a data source.
Study design decisions affect the validity and generalisability of the study results and thus are essential to the generation of fit‐for‐purpose real‐world evidence.
There is an urgent need for regulators to provide principles and for regulators to come together to harmonize the approach taken on ethics and governance issues.
More work remains to be done to globally harmonize practices and guidance for using real‐world data and real‐world evidence for regulatory decision making.
1. Introduction
Randomized controlled trials (RCTs) are considered the gold standard design for evaluating the benefits of medicinal products. Randomization, the key feature of RCTs, provides a mechanism by which randomized groups are balanced, on average, with respect to baseline factors, both measured and unmeasured. Beginning with the enactment of the 1962 Kefauver–Harris Drug Amendments to the US Food Drug and Cosmetic Act and analogous laws in other countries, RCTs became the norm for demonstrating efficacy [1]. However, a number of important research questions are challenging to address with randomized designs, such as the study of rare adverse effects, long‐term drug effects, and the treatment of rare diseases and other conditions where patients may be reluctant to be randomized. Further, many pre‐approval trials have enrolled participants who were not fully representative of the population who used the product or were conducted under conditions that do not represent those under which the new product would be used in typical health care settings once it was approved. This tendency led to concerns about an efficacy‐effectiveness gap between outcomes observed in RCTs (efficacy) compared with real‐world circumstances (effectiveness) [2]. Such concerns have led to increasing use of real‐world data (RWD) and real‐world evidence (RWE), defined below, to inform regulatory and clinical decisions about medical products.
While various definitions of RWD have been proposed (see Table 1, for examples), there is currently no consensus definition. However, a recent International Council for Harmonization of Technical Requirements for Pharmaceuticals for Human Use (ICH) Reflection Paper [9] indicates a willingness on behalf of ICH to start creating harmonized guidance, including terms and definitions for this area.
TABLE 1.
Some definitions of real‐world data.
| Organization | Definition of RWD |
|---|---|
| US Food and Drug Administration [3] | Data relating to patient health status and/or the delivery of health care routinely collected from a variety of sources. Examples of RWD include data derived from electronic health records (EHRs), medical claims data, data from product or disease registries, and data gathered from other sources (such as digital health technologies) that can inform on health status. |
| European Medicines Agency [4] | Routinely collected data relating to a patient's health status or the delivery of health care from a variety of sources other than traditional clinical trials. |
| Joint International Society for Pharmacoepidemiology (ISPE) – International Society for Pharmacoeconomics and Outcomes Research (ISPOR) Special Task Force on Real‐World Evidence in Health Care Decision Making [5] | Data obtained outside the context of randomized controlled trials (RCTs) generated during routine clinical practice. |
| International Society for Pharmacoeconomics and Outcomes Research (ISPOR) [6] | Data relating to areas such as patient health status and/or healthcare delivery not collected in conventional RCTs. Examples of RWD are EHRs; wearables; medical claims data; surveys; and product, patient, and disease registries. |
| RAND Corporation [7] | Data collected during the routine delivery of care and its reimbursement. This type of data, referred to as real‐world data, includes patient registries, EHRs, healthcare claims databases, and patient‐generated data and is defined by its production outside of a research study. |
| Innovative Medicines Initiative Get Real Project [8] | An umbrella term for data regarding the effects of health interventions (e.g., safety, effectiveness, resource use) that are not collected in the context of highly‐controlled RCT's. Instead, RWD can either be primary research data collected in a manner which reflects how interventions would be used in routine clinical practice or secondary research data derived from routinely collected data. Data collected include, but are not limited to, clinical and economic outcomes, patient‐reported outcomes (PRO) and health‐related quality of life (HRQoL). RWD can be obtained from many sources including patient registries, electronic medical records, and claims databases. |
RWE is evidence derived from the review and/or analysis of RWD [10].
The Council for International Organizations of Medical Sciences (CIOMS) recently issued a consensus report [11] to inform discussions about the use of RWD and RWE for regulatory and health care decision making, product authorization, reimbursement, and clinical use. This review provides a high‐level summary of that report.
1.1. Real‐World Evidence for Decision Making During the Product Lifecycle
RWE can be used to inform a broad range of regulatory, coverage, and clinical decisions by various kinds of stakeholders, including patients, clinicians, regulators, health technology assessment (HTA) agencies, healthcare payers, and biopharmaceutical and medical device companies. RWE has a key role to play in supporting decision‐making along the lifecycle of medicinal products, as illustrated in Figure 1 [12, 13].
FIGURE 1.

Examples of using real‐world data in the stages of the medicine lifecycle. Note: This figure has been adapted from the original figure in the CIOMS WG XIII report. RCT, randomized controlled trial; SOC, system organ class. Source: Orpha Strategy.
For RWE to support regulatory decision making, relevant stakeholders should implement a transparent process of planning, reporting, and assessing RWE. A structured template for planning and reporting on the implementation of RWE studies (STaRT‐RWE) has developed a such a template for RWE studies of the safety and effectiveness of treatments [14]. A proposal for HARmonized Protocol Template to Enhance Reproducibility (HARPER) can facilitate study protocol development and enhance transparency and reporting [15].
2. Sources of Real‐World Data
RWD from existing health care databases, including insurance claims, electronic health records (EHRs), and registry databases is widely accepted and has been used for decades, mainly for safety evaluation, risk management, and to support benefit–risk evaluation of medicinal products [16, 17]. These RWD sources have many strengths, including longitudinality, large population size, and ready accessibility of the data, allowing for faster study completion. Their limitations include the unavailability of information on key confounding factors and that populations are often limited to patients with certain conditions or exposed to specific medicinal products.
Those considering any given RWD source can evaluate if the RWD study population would contribute clinically meaningful information to guide their decision‐making for some populations of interest, even if not all subgroups of interest. When existing data sources are not suitable to answer the questions at hand, either because of a lack of information or under‐representativeness of the study population, new RWD with primary data collection, or with data collected on an ad‐hoc basis specifically for the study, may be needed. Because the data are collected specifically to answer a set of study questions, these data sources can potentially be more effective in answering those questions.
The availability of many different RWD sources presents a unique opportunity to perform the same study using different sources. A set of organized but distinct RWD sources analyzed separately using the same protocol for the same study is usually called a federated system [18, 19, 20, 21, 22, 23, 24, 25]. The use of multiple databases can enlarge the sample size, broaden representativeness, enrich the data, and prolong the study follow‐up time.
In addition to existing health care data and data collected on research participants recruited by clinicians, direct enrolment of participants by health systems and others without clinician involvement is becoming more common. For example, the ADAPTABLE (Aspirin Dosing: A Patient‐Centric Trial Assessing Benefits and Long‐Term Effectiveness) trial recruited participants directly through electronic health records and patient portals and randomized them to either 81 or 325 mg of daily aspirin, independently of health care providers. All study visits were completed within the web portal and did not require clinic visits [26].
Other important RWD sources are spontaneous reporting systems (SRSs) that record individual case reports that are spontaneously reported or solicited. One of the limitations of SRSs is that all observations reflect reported events, whereas the populations from which these events arise are not known. Other weaknesses include under‐reporting, stimulated reporting, differential reporting, and poor data quality in terms of validity and completeness [27, 28]. While SRSs are usually not an ideal basis for causal inference, they have been an important source for signal detection since the 1960s [29, 30] and are likely to remain so for the foreseeable future.
Cross‐sectional survey databases, such as the US National Health and Nutrition Examination Survey database, are other RWD sources that can play a key role in the evaluation of the burden or prevalence of diseases. Because survey participants are often systematically sampled to be representative of the population, estimates of prevalence can be generalized to that population.
The introduction of new technologies, such as those related to remote care, and the increased use of mobile devices, have provided new sources of information that can be generated with unprecedented volume, speed, and complexity and require a different set of data management and analytical methods. These emerging RWD sources include biosensor data, patient experience data, curated EHRs combined with ancillary data, text or images from radiology information systems, output from laboratory information systems, and output from structured genomics investigation. Although the utility of these emerging sources is still becoming apparent, with the rapid development of modern computing and advanced analytics, it is just a matter of time before they will also be used as key RWD sources in the context of regulatory decision making.
3. Real‐World Evidence for Regulatory Use: Key Considerations
In each phase of the product lifecycle, research questions for the inclusion of RWE may arise. The research question of interest is defined considering the evidence gaps, followed by the selection of an appropriate data source (primary or secondary), a study design, and a protocol [31].
3.1. Assessing Real‐World Data to Generate Evidence
Scientific fitness‐for‐purpose is critical for the selection of one or more databases. The attributes of a RWD source need to be suitable for the research question or study purpose, including the size and representativeness of the study population and the availability of key variables about exposures, outcomes, and covariates, including confounders with the method of data collection. A single database may not be sufficient for a given research question, and multiple databases may be needed. If so, the principles of scientific considerations of fitness‐for‐purpose apply to each database [32]. When multiple data sources are used, proper work processes for data ingestion and harmonization of datasets into a common data model are extremely important. If data linkage is needed, the right data elements for patient linking must be present. A data feasibility assessment framework called the Structured Process to Identify Fit‐For‐Purpose Data is a systematic process for assessing feasibility to determine if a data source is fit for decision making [30].
The US Food and Drug Administration's (FDA's) draft guidance requests sponsors that submit non‐interventional studies for regulatory review to take responsibility for all activities related to the design, conduct, and oversight of the studies [33]. Data quality frameworks provide structured approaches to assess the quality of RWD [32, 33, 34]. The recent European Medicines Agency framework emphasizes several key dimensions of data quality that are crucial for evaluating the reliability and usability of RWD, including reliability, extensiveness, coherence, timeliness, and relevance [34].
3.2. Study Design and Methods
The choice of study design depends on the research question, type, and availability of relevant data, and feasibility of the study. The strengths and limitations of different study designs must be carefully considered to ensure the validity of the study results. RWD/RWE can address research questions like estimating disease incidence/prevalence for orphan medicines, measuring background adverse event rates, selecting comparators for single‐arm studies, and providing descriptive evidence to support label changes. RWD can also be used to explore treatment patterns, healthcare resource use, and patient‐reported outcomes, enriching regulatory decision making beyond safety and efficacy assessments. Valid design choices are essential to ensure the validity of a study's findings, and errors in design are unlikely to be remediable in a study's analysis. Emulating a hypothetical randomized trial for designing studies using RWD is an approach that seeks to address the limitations of nonrandomized studies in evaluating the safety and effectiveness of medical interventions. There are several advantages to this approach, including that it clarifies thinking while making crucial design decisions such as inclusion criteria, duration of follow‐up, and study endpoints, and reduces the potential for introducing error [35]. Given its advantages, target trial emulation is becoming more common and accepted, although it is not always straightforward to conceptualize every research question as an emulated trial.
The successful implementation of a real‐world study hinges on identifying the population that would most benefit from a given therapy or intervention. Identifying a clinically relevant anchor point in time is critical, as it establishes the temporality between potential confounders, the exposure, and the outcome. Historical controls differ from contemporaneous controls in terms of timing for cohort entry. A contemporaneous control arm can be created if all of the RWD was generated on or after the first patient receiving the study agent was enrolled. To account for any potential temporal changes in disease treatment or outcomes, contemporaneous control cohorts are preferable to historical controls. Constructs such as race and ethnicity merit additional care in the design and analysis of studies to generate RWE. The use of race or ethnicity and decisions regarding the representation of those who provide these data should be informed by an understanding of the community's interest in seeing themselves in the results, while respecting privacy concerns. As best practices are evolving in this area [36], researchers are advised to seek up‐to‐date expert guidance on measurement, analysis, and reporting of race or ethnicity [37].
Outcome definitions of RWD studies refer to the specific endpoints or measures that are used to evaluate the effectiveness or safety of a particular intervention or exposure in the study population. Selecting a clinical outcome measure in the real‐world assessment of medicine effectiveness and safety involves careful consideration of disease or condition factors and data sources [38]. Outcome definitions described in the CIOMS report include clinical, patient‐reported, surrogate, and economic outcomes. Composite endpoints, with two or more component endpoints, are often used as study endpoints when the individual events included in the score are rare, and/or when it makes biological and clinical sense to group them. Using composite endpoints in RWD studies confers advantages by amalgamating multiple, related outcomes. Selecting the appropriate exposure definition is critical in the real‐world assessment of medicine effectiveness and safety. The “as‐started” exposure definition, which is analogous to the intention‐to‐treat principle in RCTs, follows patients from the start of their treatment until the end of follow‐up, regardless of treatment discontinuation [39]. This exposure definition answers the clinical question of whether to initiate a medicine versus another; it is about the intent of treatment initiation. In the time‐varying exposure definition, patients are followed from a cohort entry point, and their exposure status is allowed to vary over time.
Confounding is one of the biggest challenges in working with RWD and plays an even more significant role when comparing treatment effectiveness with safety. A confounder can be defined as a variable for which control by design or analytical adjustment is required to obtain unbiased estimates of the effect of an exposure on the outcome under study. In nonrandomized studies and hybrid clinical trial designs, it is important to assess the potential impact of residual confounding. When using RWD, assessment of systematic error (bias) is critical for a study aimed at evaluating a medicinal product's treatment effect. The role of bias in epidemiology and pharmacoepidemiology has been described in many guidelines and reference works.
Pragmatic randomized clinical trials are largely thought to answer important and relevant questions about the real‐world effects of treatments in post‐approval routine clinical practice settings [40]. They typically include a broader and more diverse study population of patients who are eligible to receive study interventions as part of routine clinical practice. Research participants are recruited from clinical practice settings. Primary and secondary outcomes could be collected from claims or EHRs, or collected through limited electronic case report forms, with or without adjudication. While such trials can incorporate pragmatic elements, they can still have features to maintain rigorous standards for data collection.
External controls, often derived from past traditional trials, have been used as a control arm for single‐arm trials. More recently, external controls derived from RWD are increasingly being used as controls for single‐arm trials, especially for serious and rare diseases where a trial is not feasible or where randomization is unethical [41]. Regulations and guidance documents have indicated circumstances where historical control arm designs can be used. US Code of Federal Regulations 21CFR 314.126 [42] indicates that historical control designs are usually reserved for special circumstances. ICH E10 (2001) [43] describes selection strategies for control groups in clinical trials intended to demonstrate efficacy. The US FDA has issued a guidance document on externally controlled trials [41, 44].
3.3. Real‐World Evidence Reporting‐Transparency and Trust
Recent regulatory approvals based on RWE created an urgency to develop generally accepted processes that promote trust in the evidence‐generation process [45]. Registration of RWD studies has been proposed to improve transparency, trust, and research replicability. In the CIOMS report, quality tools for RWD studies and RWE reporting are introduced [46, 47, 48, 49].
In 2020, an extensive US‐based study team launched the RCT DUPLICATE initiative (Randomized, Controlled Trials Duplicated Using Prospective Longitudinal Insurance Claims: Applying Techniques of Epidemiology) to compare the findings of RCTs relevant to regulatory decision making with the findings of noninterventional RWE that emulate the trial design as closely as possible in a consistent, transparent, and reproducible process that would be acceptable to regulators [50, 51]. Some of the main takeaways from RCT DUPLICATE included: (1) simple measures of agreement in results between RCTs and RWD studies lack nuance and will not tell the whole story; (2) while residual bias or random error is always a potential explanation for observed divergence in results between a trial and a RWD study, when the divergence is driven by design emulation differences, the database study could be accurately targeting a different effect (for a different research question) than the trial; (3) given low adherence in clinical practice, it can be challenging to replicate trial findings for outcomes with a long induction window or time varying hazard over extended follow up; and (4) while comparisons of RCT and RWD studies typically use the result of a single trial as a reference standard, this does not take into account the uncertain replicability of a trial's findings even by other trials. Other similar RCT emulation projects have also found that when the data and design are fit‐for‐purpose, non‐randomized database studies can reach similar conclusions about drug effects as randomized trials [47, 48, 49, 50, 51, 52]. The real benefit of non‐randomized, non‐interventional RWD studies is in how they can complement rather than replace evidence from RCTs in extending knowledge about the effects of medical products in post‐approval settings.
4. Ethics and Governance
An important ethical issue is the efficacy‐effectiveness gap, the difference between the efficacy observed in clinical trials and the effectiveness observed in real‐life settings. The broader use of RWE to evaluate efficacy as well as safety is justified not only by a need for stronger and broader evidence and to include neglected groups in the evidence base [53], but also by concerns that evidence from RCTs sometimes does not translate well into real‐world use. Trials also tend to under‐report harm, given the narrow eligibility that limits exposure to certain groups until safety is proven [54]. Evidence shows that the efficacy‐effectiveness gap worsens disease response and survival outcomes and increases toxicity in the clinical setting [55]. This gap provides an ethical argument to incorporate RWE into decision making.
Other ethical and governance issues discussed in the report are informed consent, privacy, and data protection, as summarized below.
4.1. Consent
Perhaps the most important ethical issue concerning the use of RWD is informed consent. In many cases, patient data is routinely used for service evaluation and audit without explicit consent being sought. For example, some healthcare providers in the United Kingdom simply display posters informing patients about this. If RWD is to be used more, then routine data linkage with patient records for research, in addition to evaluation and audit, may be a next step, which raises some issues regarding confidentiality and privacy. However, given that such data are often used without explicit consent for other purposes, it might be argued that seeking active consent for research use is disproportionate given the potential benefits of research. The need for consent for clinical trials comparing standard‐of‐care interventions has sparked significant debate. While some argue that all trials should require comprehensive consent, others contend that trials limited to standard treatments do not need consent, since the interventions are already accepted practices [56].
4.2. Privacy and Data Protection
The purpose of data protection legislation is to protect the fundamental rights and interests of citizens in relation to the processing of personal data that relates to them. On the one hand, this can be satisfied in many situations where sensitive personal data about individuals are processed, for example, in relation to banking details or in other commercial transactions that place citizens in vulnerable situations in relation to their personal data, through the safeguard of a clickwrap or click‐to‐sign consent. On the other hand, highly regulated areas such as medical research, with multiple safeguards and independent scrutiny, are made almost impossible to negotiate. RWD is in danger of being so restricted by data protection laws and regulations that it becomes impossible to work with, whereas in practice it is an area where the interests of individual citizens are robustly protected, more so than in many commercial situations imposed on consumers, and where the outcomes that the RWD research pursues are clearly in the public interest and in the interests of protecting human dignity.
4.3. Broad Data Protection Landscape
From its common international roots in the late 1970s [46], data protection law has shared a common language and basic shape [57]. The underpinning idea is that the individual citizen has human rights, particularly privacy rights in relation to the processing of their personal data. According to the European Union's General Data Protection Regulation (EU GDPR 2016/679), lawful processing is prescribed to include, although not exclusively, two fundamental elements:
Processing must be on the legal bases for the processing of personal data; and
Data subjects must be given information about the identity and contact details of the data controller and the purpose and nature of the processing of the personal data.
Data protection and processing are the major unresolved conceptual and technical issues for the use of RWD. Strong data protection law should not be seen as an obstacle or barrier to the effective processing of personal data, and therefore, where there are unresolved technical issues, they must be resolved. A well‐functioning personal data protection regime is essential to the acceptance and operation of RWD processing.
4.4. Information Provision
Separately from the requirement for a legal basis for personal data processing, those who process personal data must inform the data subjects of their identity, contact details, and the purpose for and nature of the processing they propose. This is not a requirement for informed consent in all cases. This requirement acknowledges that the data subject has rights that they can only engage when they are aware that processing is taking place. It allows, in certain circumstances, for data subjects to opt out or modify their participation in certain processing and is therefore a necessary part of the process.
4.5. De‐Identifying the Data
Data protection law operates only on personal data, meaning data that identify an individual natural person or that are capable of doing so when linked to other data. The easiest example to comprehend is pseudonymised data. Personal data have certain identifiers, for example, a name or address, which are replaced with a code. “Pseudonymization” of data means replacing any information that could be used to identify an individual with a pseudonym or, in other words, a value that does not allow the individual to be directly identified. The level of pseudonymising needed to reasonably protect persons from being identified without hindering researchers' ability to address questions of clinical and public importance merits further consideration.
There is an urgent need for regulators to provide principles, and for governments to come together to harmonize the approach taken on ethics and governance issues. The lack of guidance at least gives an opportunity for strong guidance to be created now to fill the gaps.
5. Conclusions and Future Directions
In recent years, RWE has evolved, driven by recognition and acceptance by regulators, payers, and HTA bodies to answer specific research questions.
Each RWD source has its strengths and limitations, and scientific evaluation of the fitness of a RWD source for a given study is essential in choosing a data source. Study design decisions affect the validity and generalisability of the study results, and thus are essential to the generation of fit‐for‐purpose RWE. Ethics and governance issues concerning RWD and RWE include the efficacy‐effectiveness gap, inferred consent, and privacy and data protection. The ethical frameworks and legislation must be established to enable RWE to be used in regulatory decision‐making.
More work remains to be done to globally harmonize practices and guidance for using RWD and RWE for regulatory decision making.
Members of the Council for International Organizations of Medical Sciences (CIOMS) Working Group XIII.
| Name | Company/Organization | Country |
|---|---|---|
| Alteri, Enrica | Formerly European Medicines Agency (EMA) | Switzerland |
| Atsuta, Yoshiko | Japanese Data Center for Hematopoietic Cell Transplantation/Aichi Medical University School of Medicine | Japan |
| Aubrun, Elodie | Novartis Pharma AG | Switzerland |
| Azoulay, Laurent | McGill University | USA |
| Baumfeld Andre, Elodie | Roche | USA |
| Blackburn, Stella | IQVIA | UK |
| Boerstoel, Mariette | Bristol Myers Squibb | USA |
| Brookland, Thomas | Roche | USA |
| Campbell, Ulka | Pfizer | USA |
| Crane, Gracy | Roche | UK |
| Goettsch, Wim | National Health Care Institute, Diemen/Utrecht University | Netherlands |
| Gomez‐Caminero, Andres | Merck Sharp & Dohme | USA |
| Gomez‐Reino, Elisa | Alexion | USA |
| Haenisch, Britta | Federal Institute for Drugs and Medical Devices (BfArM) | Germany |
| Hennessy, Sean | University of Pennsylvania | USA |
| Heß, Steffen | Federal Institute for Drugs and Medical Devices (BfArM) | Germany |
| Hill, Sanna | Council for International Organizations of Medical Sciences (CIOMS) Switzerland | Switzerland |
| Irs, Alar | State Agency of Medicines | Estonia |
| Ishiguro, Akihiro | Pharmaceuticals and Medical Devices Agency (PMDA) | Japan |
| Iyasu, Solomon | Formerly Merck Sharp & Dohme | USA |
| Jonsson Funk, Michele | University of North Carolina at Chapel Hill | USA |
| Juhaeri, Juhaeri | Sanofi | USA |
| Junji, Moriya | Pharmaceuticals and Medical Devices Agency (PMDA) | Japan |
| Lambert, Laurie | Canadian Agency for Drugs and Technologies in Health (CADTH) | Canada |
| Li, Jie | Food and Drug Administration (FDA) | USA |
| de Luise, Cynthia | Pfizer | USA |
| Machlitt, Andrea | Bayer | USA |
| Mayer, Miguel‐Angel | Hospital del Mar Barcelona/Universitat Pompeu Fabra Barcelona | Spain |
| Mera, Robertino | Gilead | USA |
| Nishioka, Kinue | Pharmaceuticals and Medical Devices Agency (PMDA) | Japan |
| Nomura, Manami | Pharmaceuticals and Medical Devices Agency (PMDA) | Japan |
| Nonaka, Takahiro | Osaka Metropolitan University | Japan |
| Rägo, Lembit | Council for International Organizations of Medical Sciences (CIOMS) Switzerland | Switzerland |
| Rubino, Heather | Pfizer | USA |
| Sato, Daisaku | Pharmaceuticals and Medical Devices Agency (PMDA) | Japan |
| Schiel, Anja | Norwegian Medical Products Agency (NOMA), Scientific Advice Working Party, European Medicines Agency (EMA), and MPG and JSC member HTA‐Coordination Group (HTA‐CG) | Norway |
| Shaw, David | Maastricht University/University of Basel | Switzerland |
| Soares, Monica | Brazilian Health Regulatory Agency (ANVISA) | Brazil |
| Stingl, Julia | University Hospital of Rheinisch‐Westfälische Technische Hochschule (RWTH) Aachen | Germany |
| Townend, David | University of London | UK |
| Wakao, Rika | Pharmaceuticals and Medical Devices Agency (PMDA) | Japan |
| Wang, Shirley | Harvard Medical School | USA |
| Wicherski, Julia | Federal Institute for Drugs and Medical Devices (BfArM) | Germany |
| Wormser, David | Novartis Pharma AG | USA |
| Zint, Kristina | Boehringer Ingelheim International GmbH | Germany |
Conflicts of Interest
The authors declare no conflicts of interest.
Funding: The authors received no specific funding for this work.
Contributor Information
Sean Hennessy, Email: hennessy@pennmedicine.upenn.edu.
Council for International Organizations of Medical Sciences (CIOMS) Working Group XIII:
Alteri Enrica, Atsuta Yoshiko, Aubrun Elodie, Azoulay Laurent, Baumfeld Andre Elodie, Blackburn Stella, Boerstoel Mariette, Brookland Thomas, Campbell Ulka, Crane Gracy, Goettsch Wim, Gomez‐Caminero Andres, Gomez‐Reino Elisa, Haenisch Britta, Hennessy Sean, Heß Steffen, Hill Sanna, Irs Alar, Ishiguro Akihiro, Iyasu Solomon, Jonsson Funk Michele, Juhaeri Juhaeri, Junji Moriya, Lambert Laurie, Li Jie, De Luise Cynthia, Machlitt Andrea, Mayer Miguel‐Angel, Mera Robertino, Nishioka Kinue, Nomura Manami, Nonaka Takahiro, Rägo Lembit, Rubino Heather, Sato Daisaku, Schiel Anja, Shaw David, Soares Monica, Stingl Julia, Townend David, Wakao Rika, Wang Shirley, Wicherski Julia, Wormser David, and Zint Kristina
References
- 1. Greene J. A. and Podolsky S. H., “Reform, Regulation, and Pharmaceuticals—The Kefauver–Harris Amendments at 50,” New England Journal of Medicine 367, no. 16 (2012): 1481–1483, 10.1056/NEJMp1210007. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 2. Templeton A. J., Booth C. M., and Tannock I. F., “Informing Patients About Expected Outcomes: The Efficacy‐Effectiveness Gap,” Journal of Clinical Oncology 38, no. 15 (2020): 1651–1654, 10.1200/JCO.19.02035. [DOI] [PubMed] [Google Scholar]
- 3. US FDA , “Real‐World Evidence,” May 16, 2024, https://www.fda.gov/science‐research/science‐and‐research‐special‐topics/real‐world‐evidence.
- 4. Cave A., Kurz X., and Arlett P., “Real‐World Data for Regulatory Decision Making: Challenges and Possible Solutions for Europe,” Clinical Pharmacology & Therapeutics 106, no. 1 (2019): 36–39, 10.1002/cpt.1426. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 5. Berger M. L., Sox H., Willke R. J., et al., “Good Practices for Real‐World Data Studies of Treatment and/or Comparative Effectiveness: Recommendations From the Joint ISPOR‐ISPE Special Task Force on Real‐World Evidence in Health Care Decision Making,” Pharmacoepidemiology and Drug Safety 26, no. 9 (2017): 1033–1039, 10.1002/pds.4297. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 6. The Professional Society for Health Economics and Outcomes Research (ISPOR) , “About Real‐World Evidence,” accessed September 15, 2022, https://www.ispor.org/strategic‐initiatives/real‐world‐evidence/about‐real‐world‐evidence.
- 7. Timbie J. W., Kim A. Y., and Concannon T. W., “Use of Real‐World Evidence for Regulatory Approval and Coverage of Medical Devices: A Landscape Assessment,” Value in Health 24, no. 12 (2021): 1792–1798. [DOI] [PubMed] [Google Scholar]
- 8. Makady A., de Boer A., Hillege H., Klungel O., Goettsch W., and (on behalf of GetReal Work Package 1) , “What Is Real‐World Data? A Review of Definitions Based on Literature and Stakeholder Interviews,” Value in Health 20, no. 7 (2017): 858–865. [DOI] [PubMed] [Google Scholar]
- 9. “ICH Reflection Paper, Endorsed by the ICH Assembly on 4 June 2024, Updated May 2024. Pursuing Opportunities for Harmonisation in Using Real‐World Data to Generate Real‐World Evidence, with a focus on Effectiveness of Medicines,” ICH, accessed June28, 2024, https://admin.ich.org/sites/default/files/2024‐06/ICH_Reflection%20Paper_RWE_2024_0521_EndorsedbyICHAssembly_2024_0604.pdf.
- 10. Zuidgeest M. G. P., Goetz I., Meinecke A. K., et al., “The GetReal Trial Tool: Design, Assess and Discuss Clinical Drug Trials in Light of Real World Evidence Generation,” Journal of Clinical Epidemiology 149 (2022): 244–253. [DOI] [PubMed] [Google Scholar]
- 11. Council for International Organizations of Medical Sciences (CIOMS) , Real‐World Data And Real‐World Evidence In Regulatory Decision Making. CIOMS Working Group Report (Council for International Organizations of Medical Sciences (CIOMS), 2024) 10.56759/kfxh6213. [DOI] [PubMed] [Google Scholar]
- 12. Orpha Strategy , “Patient‐Centric Real‐World Evidence,” accessed October 2022, https://www.orphastrategy.com/real‐world‐evidence/.
- 13. Bate A., Juniper J., Lawton A. M., and Thwaites R. M., “Designing and Incorporating a Real World Data Approach to International Drug Development and Use: What the UK Offers,” Drug Discovery Today 21, no. 3 (2016): 400–405, 10.1016/j.drudis.2015.12.002. [DOI] [PubMed] [Google Scholar]
- 14. Wang S. V., Pinheiro S., Hua W., et al., “STaRT‐RWE: Structured Template for Planning and Reporting on the Implementation of Real World Evidence Studies,” BMJ 372 (2021): m4856, 10.1136/bmj.m4856. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 15. Wang S. V., Pottegård A., Crown W., et al., “HARmonized Protocol Template to Enhance Reproducibility of Hypothesis Evaluating Real‐World Evidence Studies on Treatment Effects: A Good Practices Report of a Joint ISPE/ISPOR Task Force,” Value in Health 25, no. 10 (2022): 1663–1672, 10.1016/j.jval.2022.09.001. [DOI] [PubMed] [Google Scholar]
- 16. Mazzaglia G., Straus S. M., Arlett P., et al., “Study Design and Evaluation of Risk Minimization Measures: A Review of Studies Submitted to the European Medicines Agency for Cardiovascular, Endocrinology, and Metabolic Drugs,” Drug Safety 41, no. 2 (2017): 191–202, 10.1007/s40264-017-0604-4. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 17. Santoro A., Genov G., Spooner A., Raine J., and Arlett P., “Promoting and Protecting Public Health: How the European Union Pharmacovigilance System Works,” Drug Safety 40, no. 10 (2017): 855–869, 10.1007/s40264-017-0572-8. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 18. Chen R. T., Glasser J. W., Rhodes P. H., et al., “Vaccine Safety Datalink project: a new tool for improving vaccine safety monitoring in the United States. The Vaccine Safety Datalink Team,” Pediatrics 99, no. 6 (1997): 765–773, 10.1542/peds.99.6.765. [DOI] [PubMed] [Google Scholar]
- 19. Ross T. R., Ng D., Brown J. S., et al., “The HMO Research Network Virtual Data Warehouse: A Public Data Model to Support Collaboration,” EGEMS 2, no. 1 (2014): 1049, 10.13063/2327-9214.1049. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 20. Platt R., Davis R., Finkelstein J., et al., “Multicenter Epidemiologic and Health Services Research on Therapeutics in the HMO Research Network Center for Education and Research on Therapeutics,” Pharmacoepidemiology and Drug Safety 10, no. 5 (2001): 373–377, 10.1002/pds.607. [DOI] [PubMed] [Google Scholar]
- 21. Observational Health Data Sciences and Informatics (OHDSI) The Observational Medical Outcomes Partnership (OMOP) , “Standardized Data: The OMOP Common Data Model,” accessed September 15, 2022, https://ohdsi.org/data‐standardization/.
- 22. Observational Health Data Sciences and Informatics (OHDSI) , “OMOP Common Data Model,” accessed June 2, 2023, https://ohdsi.github.io/CommonDataModel/.
- 23. European Health Data & Evidence Network (EHDEN) , “Becoming the Trusted Open Science Community Built With Stanadardised Health Data via a European Fedarated Network,” accessed July 3, 2023, https://ehden.eu/.
- 24. European Medicines Agency (EMA) , “Data Analysis and Real World Interrogation Network (DARWIN EU),” accessed March 11, 2024, https://www.darwin‐eu.org.
- 25. Curtis L. H., Weiner M. G., Boudreau D. M., et al., “Design Considerations, Architecture, and Use of the Mini‐Sentinel Distributed Data System,” Pharmacoepidemiology and Drug Safety 21 (2012): 23–31, 10.1002/pds.2336. [DOI] [PubMed] [Google Scholar]
- 26. Marquis‐Gravel G., Roe M. T., Robertson H. R., et al., “Rationale and Design of the Aspirin Dosing—A Patient‐Centric Trial Assessing Benefits and Long‐Term Effectiveness (ADAPTABLE) Trial,” JAMA Cardiology 5, no. 5 (2020): 598–607, 10.1001/jamacardio.2020.0116. [DOI] [PubMed] [Google Scholar]
- 27. Hazell L. and Shakir S. A., “Under‐Reporting of Adverse Drug Reactions: A Systematic Review,” Drug Safety 29, no. 5 (2006): 385–396, 10.2165/00002018-200629050-00003. [DOI] [PubMed] [Google Scholar]
- 28. Rawlins M. D., “Spontaneous Reporting of Adverse Drug Reactions. I: The Data,” British Journal of Clinical Pharmacology 26, no. 1 (1988): 1–5, 10.1111/j.1365-2125.1988.tb03356.x. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 29. Almenoff J., Tonning J. M., Gould A. L., et al., “Perspectives on the Use of Data Mining in Pharmacovigilance,” Drug Safety 28, no. 11 (2005): 981–1007, 10.2165/00002018-200528110-00002. [DOI] [PubMed] [Google Scholar]
- 30. Bate A. and Evans S. J., “Quantitative Signal Detection Using Spontaneous ADR Reporting,” Pharmacoepidemiology and Drug Safety 18, no. 6 (2009): 427–436, 10.1002/pds.1742. [DOI] [PubMed] [Google Scholar]
- 31. Duke Margolis Center for Health Policy , “A Framework for Regulatory Use of Real‐World Evidence,” accessed June 2, 2023, https://healthpolicy.duke.edu/publications/framework‐regulatory‐use‐real‐world‐evidence.
- 32. Gatto N. M., Campbell U. B., Rubinstein E., et al., “The Structured Process to Identify Fit‐For‐Purpose Data: A Data Feasibility Assessment Framework,” Clinical Pharmacology and Therapeutics 111, no. 1 (2022): 122–134, 10.1002/cpt.2466. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 33. The United States Food and Drug Administration (FDA) , “Considerations for the Use of Real‐World Data and Real‐World Evidence to Support Regulatory Decision‐Making for Drug and Biological Products. Guidance Document,” accessed October 15, 2022, https://www.fda.gov/media/154714/download.
- 34. European Medicines Agency , “Data Quality Framework for EU Medicines Regulation, EMA/326985/2023,” accessed April 21, 2024, https://www.ema.europa.eu/system/files/documents/regulatory‐procedural‐guideline/data‐quality‐framework‐eu‐medicines‐regulation_en_1.pdf.
- 35. Hernán M. A. and Robins J. M., “Using Big Data to Emulate a Target Trial When a Randomized Trial Is Not Available,” American Journal of Epidemiology 183, no. 8 (2016): 758–764, 10.1093/aje/kwv254. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 36. “Initial Proposals For Updating OMB's Race and Ethnicity Statistical Standards,” accessed June 4, 2023, https://www.federalregister.gov/documents/2023/01/27/2023‐01635/initial‐proposals‐for‐updating‐ombs‐race‐and‐ethnicity‐statistical‐standards.
- 37. Flanagin A., Frey T., Christiansen S. L., and AMA Manual of Style Committee , “Updated Guidance on the Reporting of Race and Ethnicity in Medical and Science Journals,” Journal of the American Medical Association 326, no. 7 (2021): 621–627, 10.1001/jama.2021.13304. [DOI] [PubMed] [Google Scholar]
- 38. Velentgas P., Dreyer N., Noujah P., Smith S. R., and Torchia M. M., eds., Developing a Protocol for Observational Comparative Effectiveness Research: A User's Guide (Agency for Health Care Research and Quality (U.S.), Government Printing Office, 2013). [PubMed] [Google Scholar]
- 39. Schneeweiss S. and Patorno E., “Conducting Real‐World Evidence Studies on the Clinical Outcomes of Diabetes Treatments,” Endocrine Reviews 42, no. 5 (2021): 658–690, 10.1210/endrev/bnab007. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 40. Franklin J. M., Pawar A., Martin D., et al., “Nonrandomized Real‐World Evidence to Support Regulatory Decision Making: Process for a Randomized Trial Replication Project,” Clinical Pharmacology & Therapeutics 107, no. 4 (2020): 817–826, 10.1002/cpt.1633. [DOI] [PubMed] [Google Scholar]
- 41. European Medicines Agency , Reflection Paper on Establishing Efficacy Based on Single‐Arm Trials Submitted as Pivotal Evidence in a Marketing Authorisation, Considerations on Evidence From Single‐Arm Trials accessed July 16, 2024, https://www.ema.europa.eu/en/documents/scientific‐guideline/draft‐reflection‐paper‐establishing‐efficacy‐based‐single‐arm‐trials‐submitted‐pivotal‐evidence‐marketing‐authorisation_en.pdf.
- 42. Codes of Federal Regulations , “21CFR 314.126. Adequate and Well‐Controlled Studies,” accessed May 9, 2023, https://www.ecfr.gov/current/title‐21/chapter‐I/subchapter‐D/part‐314/subpart‐D/section‐314.126.
- 43. European Medicines Agency (EMA) , ICH E10 Choice of Control Group in Clinical Trials—Scientific Guideline accessed May 9, 2023, https://www.ema.europa.eu/en/ich‐e10‐choice‐control‐group‐clinical‐trials‐scientific‐guideline.
- 44. The United States Food and Drug Administration (FDA) , “Considerations for the Design and Conduct of Externally Controlled Trials for Drug and Biological Products. Guidance for Industry. Draft Guidance,” accessed May9, 2023, https://www.fda.gov/media/164960/download.
- 45. Orsini L. S., Berger M., Crown W., et al., “Improving Transparency to Build Trust in Real‐World Secondary Data Studies for Hypothesis Testing—Why, What, and How: Recommendations and a Road Map From the Real‐World Evidence Transparency Initiative,” Value in Health 23, no. 9 (2020): 1128–1136, 10.1016/j.jval.2020.04.002. [DOI] [PubMed] [Google Scholar]
- 46. Von Elm E., Altman D. G., Egger M., et al., “The Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) Statement: Guidelines for Reporting Observational Studies,” International Journal of Surgery 12, no. 12 (2014): 1495–1499, 10.1016/j.ijsu.2014.07.013. [DOI] [PubMed] [Google Scholar]
- 47. Morgan R. L., Thayer K. A., Santesso N., et al., “A Risk of Bias Instrument for Non‐randomized Studies of Exposures: A users' Guide to Its Application in the Context of GRADE,” Environment International 122 (2019): 168–184, 10.1016/j.envint.2018.11.004. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 48. Langan S. M., Schmidt S. A., Wing K., et al., “The Reporting of Studies Conducted Using Observational Routinely Collected Health Data Statement for Pharmacoepidemiology (RECORD‐PE),” BMJ 363 (2018): k3532, 10.1136/bmj.k3532. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 49. Benchimol E. I., Smeeth L., Guttmann A., et al., “The REporting of Studies Conducted Using Observational Routinely‐Collected Health Data (RECORD) Statement,” PLoS Medicine 12, no. 10 (2015): e1001885, 10.1371/journal.pmed.1001885. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 50. Franklin J. M., Patorno E., Desai R. J., et al., “Emulating Randomized Clinical Trials With Nonrandomized Real‐World Evidence Studies,” AHA Journals 143, no. 10 (2020): 1002–1013, https://www.ahajournals.org/doi/abs/10.1161/CIRCULATIONAHA.120.051718. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 51. RCT Duplicate . https://www.rctduplicate.org/.
- 52. Wang S. V., Schneeweiss S., and RCT‐DUPLICATE Initiative , “Emulation of Randomized Clinical Trials With Nonrandomized Database Analyses: Results of 32 Clinical Trials,” Journal of the American Medical Association 329, no. 16 (2023): 1376–1385, 10.1001/jama.2023.4221. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 53. US FDA , “Diversity Plans to Improve Enrollment of Participants from Underrepresented Racial and Ethnic Populations in Clinical Trials Guidance for Industry. DRAFT GUIDANCE (This Guidance Document Is Being Distributed for Comment Purposes Only),” accessed July 16 2024, https://www.fda.gov/media/157635/download.
- 54. Seruga B., Templeton A. J., Badillo F. E., Ocana A., Amir E., and Tannock I. F., “Under‐Reporting of Harm in Clinical Trials,” Lancet Oncology 17, no. 5 (2016): 209–219, 10.1016/S1470-2045(16)00152-2. [DOI] [PubMed] [Google Scholar]
- 55. International Federation of Pharmaceutical Manufacturers & Associations (IFPMA) , “Patients, Product Development Partnerships and Private Health Sector Representatives Call for the Rapid Ratification of the African Medicines Agency Treaty to Speed Up Timelines to Vaccines and Medicines Approval,” 2021, accessed April 15, 2023, https://www.ifpma.org/news/patients‐product‐development‐partnerships‐and‐private‐health‐sector‐representatives‐call‐for‐the‐rapid‐ratification‐of‐the‐african‐medicines‐agency‐treaty‐to‐speed‐up‐timelines‐to‐vaccines‐and‐medici/.
- 56. Wendler D., Dickert N. W., Silbergleit R., Kim S. Y., and Brown J., “Targeted Consent for Research on Standard of Care Interventions in the Emergency Setting,” Critical Care Medicine 45, no. 1 (2017): 105–110, 10.1097/CCM.0000000000002023. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 57. Council of Europe , “The Convention for the Protection of Individuals With Regard to Automatic Processing of Personal Data CETS No. 108” (1981) (accessed April 2023, https://rm.coe.int/1680078b37) and “Directive 95/46/EC of the European Parliament and of the Council of 24 October 1995,” (accessed April 15, 2023, https://eur‐lex.europa.eu/TodayOJ/) on the Protection of Individuals With Regard to the Processing of Personal Data and on the Free Movement of Such Data.
