Comparison of two assessments of real‐world data and real‐world evidence for regulatory decision‐making

Abstract

Real‐world data (RWD) and real‐world evidence (RWE) are increasingly used to support regulatory decision making, but regulatory agencies and stakeholders may apply different definitions for RWD and use different criteria to determine when analysis of such data are considered RWE in decisions on drug approvals. To explore this issue, we reviewed two prominent publications that operationalized the definitions of RWD and RWE when describing the use of RWE in drug approvals by the US Food and Drug Administration (FDA) and the European Medicines Agency (EMA). Both publications considered noninterventional (observational) studies, RWD as a comparator arm for a single‐arm trial, product‐related literature reviews, and RWD to support clinical trial implementation (e.g., to identify potential participants) as generating RWE. In contrast, inconsistencies were identified regarding types of data sources and study designs that were considered as not generating RWE. For example, a lack of agreement existed regarding whether RWE is generated when RWD describe therapeutic contexts or are used in phase I/II interventional trials, open‐label extension studies, or pharmacovigilance activities. These discrepancies highlight opportunities to develop a consistent understanding of the role of RWE in regulatory decision making for drug approvals among regulatory agencies and stakeholders.

Study Highlights.

• WHAT IS THE CURRENT KNOWLEDGE ON THE TOPIC?

Given that no universal definitions for real‐world data (RWD) and real‐world evidence (RWE) currently exist, a lack of clarity can arise regarding how RWD and RWE are utilized across different regulatory agencies.

• WHAT QUESTION DID THIS STUDY ADDRESS?

How have the definitions of RWD and RWE been applied in reports of regulatory decision making processes across different regulatory agencies to support drug approvals?

• WHAT DOES THIS STUDY ADD TO OUR KNOWLEDGE?

This study focuses on the definitions of RWD and RWE from two different regulatory agencies and compares two prominent analyses in terms of the application of RWD and RWE.

• HOW MIGHT THIS CHANGE CLINICAL PHARMACOLOGY OR TRANSLATIONAL SCIENCE?

This report underscores the need for consistent tracking and reporting of how RWD can generate RWE to support drug approvals.

INTRODUCTION

Interest is increasing in the use of real‐world data (RWD) and real‐world evidence (RWE) to support regulatory decision making, especially in the context of demonstrating the effectiveness of medical products. Despite this interest, a lack of universal definitions for RWD and RWE ¹ contributes to misunderstanding of the role of such data and evidence in regulatory decision making. In addition, given that different jurisdictions are subject to specific laws and regulations, individual regulatory agencies and stakeholders can apply different operational definitions of RWD and RWE when describing corresponding activities and accomplishments.

Various reports ² , ³ , ⁴ , ⁵ , ⁶ , ⁷ have attempted to characterize various aspects of RWD and RWE, with one review ⁶ identifying 38 definitions of RWD in the literature. Of note, the US Food and Drug Administration (FDA) defines RWD as “data relating to patient health status and/or the delivery of health care routinely collected from a variety of sources […].” ⁸ RWD can come from electronic health records (EHRs), claims and billing activities, product and disease registries, and data gathered from other sources that can inform on health status, such as digital health technologies in non‐research settings. RWE is defined as the “clinical evidence regarding the usage and potential benefits or risks of a medical product derived from analysis of RWD.” ⁸

Also of note, the European Medicines Agency (EMA), as described in Cave et al., defines RWD as “routinely collected data relating to a patient's health status or the delivery of health care from a variety of sources other than traditional clinical trials.” ⁹ In the same publication, the authors indicated they excluded “traditional clinical trials even if single arm but would incorporate data from pragmatic clinical trials [as RWD] if data were collected remotely through an electronic health record or other observational data source and solely under conditions of normal clinical care.” ⁹ RWE was defined as the “information derived from analysis of RWD.” ⁹

Some reports have quantified or evaluated the extent of RWD and RWE in regulatory submissions. For example, a report by Flynn et al. ⁴ assessed the contribution of RWE in new marketing authorization applications (MAAs) and extensions of indication (EOIs) applications submitted to the EMA in 2018 and 2019. As another example, Purpura et al. ⁵ conducted a systematic review of publicly available review documents for FDA approvals of new drugs and biological products, from January 2019 to June 2021, to assess the role of RWE studies in these approvals. Although these two publications ⁴ , ⁵ evaluated data from two different regulatory agencies, insights can be gained by a side‐by‐side comparison of how definitions of RWD/RWE are used to support drug approvals. The current research compares the operational definitions of (i.e., criteria for) RWD/RWE in these two prominent publications to assess whether different approaches can lead to different assessments of the same types of data sources and design structures.

METHODS

We conducted a comparative review of the analyses by Flynn et al. ⁴ and Purpura et al. ⁵ on the role of RWD and RWE in the drug approval process. Flynn et al. ⁴ delineated how conceptual definitions of RWD and RWE provided by Cave et al. ⁹ were translated into specific criteria, as described in their table 1, “Types of real‐world data/real‐world evidence included/excluded in new MAAs and EOIs for already authorized products.” Purpura et al. ⁵ described their methods of analysis but did not include complete details regarding operationalized criteria. Accordingly, our review relied on examples of FDA approvals that were identified in that article along with the associated FDA approval documents—specifically, for approvals listed in their Table 1, “Categorization applied to applicants' intended use of RWE in NDAs and BLAs,” as well as for 37 listed approvals in their table S7, “FDA's feedback on use of RWE and/or the RWE studies, January 2019 to June 2021.” Of note, our review focused on final drug approval decisions as reported by Flynn et al. ⁴ and Purpura et al., ⁵ and therefore does not assess the use of RWD in all stages of drug development.

Two co‐authors independently identified and extracted the criteria used in each publication to categorize specific types of data and study designs as generating RWE. If the reasoning used to classify the data and study design was not fully described in the publication, cited approvals were reviewed to assess alignment with a criterion extracted from the publication. Differences between co‐authors were adjudicated through discussion with the third co‐author.

RESULTS

The criteria used to classify specific types of data and study designs as generating or not generating RWE to support drug approval are shown in Table 1. Given that Flynn et al. ⁴ typically provided descriptions of operational criteria for identifying RWD and RWE, we were able to closely align the criteria in Table 1 with their publication. For example, the criteria we extracted as generating RWE included the use of RWD as a comparator arm in a single‐arm trial, RWD to support clinical trial implementation, noninterventional studies, and product‐related literature reviews. We further assessed that Flynn et al. ⁴ did not consider various scenarios as generating RWE: non‐product related literature reviews; aggregated data from multiple sources with unclear attribution; open‐label follow‐up studies of clinical trial patients; active and routine pharmacovigilance activities; surveys not based on individual patients; and all phases of interventional clinical trials, except for phase III/IV studies containing RWD.

TABLE 1.

What is considered RWE in two representative publications?

Criteria	Flynn et al. 4	Purpura et al. 5
Patient‐level data from a noninterventional study	✓	✓
Use of RWD as comparator arm for a single‐arm trial	✓	✓
Use of RWD to support clinical trial implementation	✓	✓
Product‐related literature review	✓	*
Non‐product related literature review	✘	*
Use of aggregated data from multiple sources with unclear attribution	✘	–
Phase I/II interventional study	✘	–
Phase I/II interventional study (with RWD)	–	*
Phase III/IV interventional study without RWD	✘	–
Open‐label follow‐up study of clinical trial patients	✘	*
Active and routine pharmacovigilance activities	✘	*
Survey not based on individual patients	✘	–

Note: ✓, indicates the criterion was directly identified in the publication as generating RWE to support regulatory decision‐making. ✘, indicates the criterion was directly identified in the publication as not generating RWE to support regulatory decision‐making. –, indicates that no statement was evident as to whether the criterion would generate RWE. *, indicates the criterion was indirectly identified in our review of the publication.

Abbreviations: RWD, real‐world data; RWE, real‐world evidence.

Also shown in Table 1, Purpura et al. ⁵ identified an “observational head‐to‐head study” (i.e., noninterventional study), RWD serving as a “control in an externally controlled trial” (i.e., comparator arm for a single‐arm trial), and RWD serving as a “reference for clinical trial adverse events” as examples of how RWE is generated to support product safety and/or effectiveness. These authors also described an example of RWE to support therapeutic context, such when “RWD establishes burden of disease to demonstrate need for the product (incidence and prevalence study).” ⁵ We adjudicated this scenario to broadly align with criteria from Flynn et al. for both product‐related literature reviews (i.e., “on real‐world safety data of the product in other indications” ⁴ ) and non‐product related literature reviews (i.e., “those related to the natural history of the targeted disease or comorbidities associated with the disease” ⁴ ). In addition, Purpura et al. ⁵ included the FDA product approvals based on phase I/II studies that utilized RWD as generating substantial or supportive RWE for product safety and/or efficacy. Our review of this publication also found the FDA product approvals that included open label follow‐up studies of clinical trial patients, as well as active and routine pharmacovigilance activities, as generating RWE in the analysis.

DISCUSSION

We identified similarities and differences when comparing approaches used in two prominent stakeholder publications ⁴ , ⁵ regarding categorization of the use of RWE to support drug approvals. Several criteria were identified as generating RWE in both publications, including use of patient‐level data from noninterventional studies, use of RWD as a comparator arm for a single‐arm trial, use of RWD to support clinical trial implementation, and product‐related literature reviews. In contrast, other criteria for scenarios generating RWE did not align, including discrepancies as to whether evidence used to provide therapeutic context is considered RWE, or whether RWE is generated from phase I/II interventional trials, open‐label extension studies, or active and routine pharmacovigilance activities.

The two publications also differed regarding the use of RWD to support therapeutic context, specifically involving literature reviews. Whereas Purpura et al. ⁵ describes the use of RWD to support therapeutic context (i.e., to establish “burden of disease to demonstrate need for the product”) as an example of RWE—relevant to both product‐related and non‐product related literature reviews, as described by Flynn et al. ⁴ —the analysis by Flynn et al. ⁴ excludes the use of non‐product‐related literature reviews as generating RWE. Flynn et al., ⁴ however, did denote “data collection on disease epidemiology” as constituting RWE. Although details regarding the specific types of studies encompassed in literature reviews were not specified, individual regulatory agencies may not consider such reviews as RWE for their purposes, given that therapeutic context and disease epidemiology do not directly evaluate a drug‐outcome association. For example, although not considered RWE, RWD can be used to improve efficiencies of drug development programs, such as assessing the feasibility of clinical trials. ¹⁰ , ¹¹

When considering phases of interventional trials, Flynn et al. ⁴ broadly excluded phase I/II interventional studies as generating RWE, whereas Purpura et al. ⁵ considered the FDA drug approvals to incorporate RWE if the phase I/II clinical trials included RWD elements in their studies. For example, their table S7 lists a phase I/II trial as the basis of approval for selumetinib ¹² and a phase II trial as the basis of approval for triheptanoin ¹³ (as supportive evidence), whereas lonafarnib ¹⁴ is listed with both a phase I/II trial and a phase II trial (as substantial evidence). Whether various regulatory agencies would consider phase I/II trials as providing RWE for product safety and/or efficacy is a representative issue identified by this review.

As another consideration, open‐label studies that incorporate RWD can also generate RWE, but this classification can be nuanced and challenging to interpret. For example, Purpura et al. ⁵ considered the FDA drug approval for viloxazine extended‐release capsules ¹⁵ to incorporate RWD/RWE. The approval package included Study 812P310, a single‐arm open‐label extension study which provided long‐term safety data. In this open‐label study, however, the data were generated from patients who already had participated in randomized, controlled trials (RCTs) designed to assess the safety and efficacy of the product, and the patients continued to be followed up based on existing RCT protocols. A recent publication by Bloomfield‐Clagett et al. ⁷ did not consider RWE to be generated from interventional studies that have traditionally been an integral part of drug development programs, including open‐label extension studies focused on safety. Considering such data to have generated RWE may cause confusion in the stakeholder community.

Pharmacovigilance activities represent another challenge in terminology. The approval package for viloxazine ¹⁵ in Purpura et al. ⁵ (as described above) referenced the assistance of the FDA's Division of Pharmacovigilance in capturing European postmarketing adverse event reports to identify potential safety concerns along with the use of the FDA's Adverse Event Reporting System (FAERS). The FDA regularly conducts active and routine pharmacovigilance activities, and in instances where such data are analyzed by a regulatory agency, it may be misleading to suggest that such long‐standing activities regarding safety reflect the recent interest in using RWE for medical product approvals regarding effectiveness.

Another issue, as mentioned previously, is whether the use of RWD to support clinical trial implementation generates RWE. Both publications indicated that the use of RWD to support clinical trial implementation would generate RWE. Specifically, Flynn et al. ⁴ stated that “in the context of clinical trials, the definition [of RWE] includes the use of observational data to support and complement RCTs, even types of clinical trial considered interventional.” Depending on what “support” and similar terms represent, however, “RWD” have been routinely used for many years to enable clinical trial implementation (e.g., using EHRs to find relevant patients), but when the resulting evidence of drug effectiveness and/or safety is derived from data generated from the RCT, such evidence is not generally considered RWE. ¹¹ Without suggesting that either of these two approaches is inherently wrong, these inconsistencies highlight the need for a common understanding of how RWD are used to generate RWE to support regulatory decision making.

As a strength of our work, although various reports ² , ³ , ⁴ , ⁵ , ⁶ refer to the role of RWD and RWE in regulatory decision making, we conducted a focused review of Flynn et al. ⁴ and Purpura et al. ⁵ These two publications represent prominent evaluations of RWD/RWE submitted to two regulatory agencies, and their examples of data and study designs as generating or not generating RWE allowed for direct comparability. We used a structured approach to assess the publications, and multiple reviewers evaluated the information that was collected. As limitations when considering our findings, all operational definitions used in generating the publications were not available for review, and we were unable to ascertain all drug approvals that were described as including RWE in the Purpura et al. ⁵ publication.

Although we were able to review the examples of drug approvals included in supplemental materials of Purpura et al., ⁵ it is possible that other approvals categorized as including RWD/RWE would have met criteria in Table 1. Furthermore, even though neither report provides the direct viewpoint of the corresponding regulatory agency, Flynn et al. ⁴ had access to information unavailable in the public domain and received comments from multiple EMA committees, whereas Purpura et al. ⁵ utilized publicly accessible approval documents containing documented FDA feedback. In addition, our review was limited to the final drug approval decisions, as discussed in Flynn et al. ⁴ and Purpura et al., ⁵ and we acknowledge that RWD can play an important role throughout the life cycle of the drug‐development process.

In conclusion, inconsistencies in characterizing RWE can lead to confusion across the scientific community as well as create conflicting expectations among stakeholders and regulators regarding the submission and review of RWE involving drug development. In this context, the FDA has published a series of guidance documents that help sponsors describe data and design elements when submitting applications containing RWD/RWE. ¹⁶ Continued efforts to develop a uniform understanding of the RWE landscape would benefit sponsors in planning their study designs as well as allow regulatory agencies to identify, review, and characterize such evidence.

AUTHOR CONTRIBUTIONS

All authors wrote the manuscript, designed and performed the research, and analyzed the data.

FUNDING INFORMATION

No funding was received for this work.

CONFLICT OF INTEREST STATEMENT

The authors declared no competing interests for this work. Ms. Yuan, as a Presidential Management Fellow, was on rotational assignment at the FDA during the conduct of this project.

Yuan L, Rahman M, Concato J. Comparison of two assessments of real‐world data and real‐world evidence for regulatory decision‐making. Clin Transl Sci. 2023;17:e13702. doi: 10.1111/cts.13702