The Role of Real‐World Evidence in FDA‐Approved New Drug and Biologics License Applications

Abstract

The US Food and Drug Administration (FDA) is open to accepting real‐world evidence (RWE) to support its assessment of medical products. However, RWE stakeholders lack a shared understanding of FDA’s evidentiary expectations for the use of RWE in applications for new drugs and biologics. We conducted a systematic review of publicly available FDA approval documents from January 2019 to June 2021. We sought to quantify, by year, how many approvals incorporated RWE in any form, and the intended use of RWE in those applications. Among approvals with RWE intended to support safety and/or effectiveness, we classified whether and how those studies impacted FDA’s benefit‐risk considerations, whether those studies were incorporated into the product label, and the therapeutic area of the medical product. Finally, we qualified FDA’s documented feedback where available. We found that 116 approvals incorporated RWE in any form, with the proportion of approvals incorporating RWE increasing each year. Of these approvals, 88 included an RWE study intended to provide evidence of safety or effectiveness. Among these 88 approvals, 65 of the studies influenced FDA’s final decision and 38 were included in product labels. The 88 approvals spanned 18 therapeutic areas. FDA’s feedback on RWE study quality included methodological issues, sample size concerns, omission of patient level data, and other limitations. Based on these findings, we would anticipate that future guidance on FDA’s evidentiary expectations of RWE use will incorporate fit‐for‐purpose real‐world data selection and careful attention to study design and analysis.

Biopharmaceutical companies and their development partners increasingly use data from real‐world settings to generate evidence that can support regulatory decision making and approvals of their manufactured medical products. The use of such real‐world evidence (RWE) can complement ¹ or, in some cases, serve as an alternative to ² evidence traditionally yielded by randomized controlled trials (RCTs). For example, the use of external control arms ³ can have considerable benefits, including accelerating the development process or reducing burden on trial participants. RWE can also provide investigators the opportunity to ask more questions and to understand broader, more diverse populations, as compared to RCTs.

The US Food and Drug Administration (FDA; also referred to here as the Agency) has taken significant steps to advance the use of RWE in regulatory decision making. This momentum has grown after the 21st Century Cures Act passed in December 2016; the act required FDA to develop a program for evaluating the use of RWE to support new indications for already‐approved drugs and fulfill postapproval study requirements. In 2018, FDA published a framework for its RWE program and is currently drafting guidance on its regulatory expectations regarding the use of RWE in medical product approvals. ⁴ As part of its broad impact, FDA’s framework has helped to promote common definitions for real‐world data (RWD; defined as “data relating to patient health status and/or the delivery of health care routinely collected from a variety of sources” ⁴ ) and RWE (defined as “the clinical evidence about the usage and potential benefits or risks of a medical product derived from analysis of RWD” ⁴ ). This is part of a worldwide interest in the use of RWE, including by regulatory agencies, such as the European Medicines Agency (EMA), ¹ , ⁵ the Pharmaceuticals and Medical Devices Agency (PMDA) in Japan, ⁶ and the National Medical Products Administration (NMPA) in China. ⁷

Much of FDA’s regulatory use of RWE to date has been in the context of postmarket surveillance through programs like FDA’s Sentinel initiative, a system initially designed to aid the Agency in evaluating medical product safety, ⁸ and which has more recently expanded in scope. Current goals include enhancing Sentinel’s ability to evaluate effectiveness ⁸ and extending the evaluation of safety. ⁹ In parallel, FDA also created a pilot project in 2008 to provide for the evaluation of vaccine effectiveness ¹⁰ and, in 2017, launched the Biologics Effectiveness and Safety (BEST) system to enhance FDA’s use and analysis of data to assure the safety and effectiveness of biological products. ¹¹

Beyond postmarket surveillance, FDA considers RWE studies as part of the evidence package for submissions seeking authorization to market new medical products, including new drug applications (NDAs) and biologics license applications (BLAs). ⁴ , ¹² These submissions are reviewed and decisions are rendered primarily by two centers within the Agency: the Center for Drug Evaluation and Research (CDER), which focuses on drug products and therapeutic biological products, and the Center for Biologics Evaluation and Research (CBER), which focuses on biological products including vaccines. Studies submitted as part of a sponsor’s overall evidence package will be considered in decision making. Approvals will then be based on, among other things, studies that provide “substantial evidence” (in CDER decisions) or “primary evidence” (in CBER decisions), which make the primary case for product safety and effectiveness, and “supportive evidence,” which can serve to bolster the case. Submitted studies providing therapeutic context can help reviewers understand the landscape of the disease (such as disease prevalence and incidence) and any current standard of care, but may not directly affect decision making. Many submitted studies will be presented in the FDA‐approved product label of an approved or licensed drug or biological product. As such, RWE studies have the potential to be directly influential in prescribers’ decision making if FDA approves a product label that includes submitted RWE studies. ¹³ , ¹⁴

FDA and others have directly and indirectly provided insights into FDA’s current approaches by speaking to and publishing select examples of successful and unsuccessful uses of RWE studies in medical product approvals, ¹⁵ , ¹⁶ , ¹⁷ , ¹⁸ , ¹⁹ , ²⁰ , ²¹ , ²² and FDA has provided publicly available guidance for industry and staff. ²³ , ²⁴ , ²⁵ , ²⁶ However, RWE methodology is evolving, and on many important topics, RWE stakeholders lack a shared understanding of FDA’s expectations around the use of RWE, particularly in the context of NDAs and BLAs.

To date, there are no published systematic assessments of the use of RWE studies in FDA‐approved NDAs and BLAs, nor on FDA’s feedback on and acceptance of such studies. In this paper, we address this gap by presenting a systematic review of FDA’s written approval documents involving RWE studies in NDAs and BLAs to determine whether and how FDA incorporated that evidence in its final decision for approval. We summarize trends in recent use of RWE in FDA‐approved NDAs and BLAs, as well as whether and how RWE supported FDA’s approval decision. By distilling these reviews, we also seek to identify best practices for avoiding common design and analysis pitfalls identified by the Agency.

METHODS

Data

We examined a variety of FDA’s public resources to identify approvals from January 2019 through June 2021, and extracted the public documents available about such approvals. Our sources included FDA CDER’s Drugs@FDA, an Agency database of approved drugs; ²⁷ CDER’s list of novel drug approvals; ²⁸ CDER’s manual on NDA classification codes; ²⁹ CBER’s list of biological approvals by year; ³⁰ and publicly accessible review and approval documents. We also referenced FDA’s Spectrum of Diseases by Therapeutic Area Found in Written Requests ³¹ to align to their taxonomy of therapeutic areas.

For products approved by CDER, the publicly accessible approval documents ²⁷ we reviewed were FDA approval letters, product labels, advisory committee meeting materials, and published review documents. The FDA review documents we examined were the Multi‐Discipline Reviews, Integrated Reviews, Summary Reviews, Clinical Reviews, Other Reviews, Administrative and Correspondence Documents, and Statistical Reviews.

For products licensed by CBER, the publicly accessible review and approval documents ³² we reviewed were FDA approval letters, product labels, Summary Basis for Regulatory Action (product review document), and related items. Examples of related items included BLA Clinical Review Memoranda, Statistical Review Memoranda, Team Meeting Summaries, Summary of Mid‐Cycle Communication Teleconferences with CBER, and Records of Telephone Conversation.

Approval inclusion and exclusion criteria

We first included all NDAs approved by CDER for new molecular entities (NMEs) from January 1, 2019, through June 30, 2021. Based on FDA’s classification system, ²⁹ this included NDA type 1 (“new molecular entity”) and type 9 (“new indication or claim, drug not to be marketed under type 9 NDA after approval”), as these are the core mechanisms for FDA to consider marketing authorization for new molecules. We excluded the remainder of FDA’s NDA classifications. Specifically, we excluded types 2 through 5, as these classifications pertain to new formulations, new combinations, or other administrative or “technical” modifications to existing molecular entities. We excluded type 6 as it is no longer actively used, and type 7 because it pertains to drugs marketed prior to 1962. We excluded type 8, as those pertain to drugs moving from prescription to over‐the‐counter. Finally, we excluded type 10, as it is used for duplicates of drug products that are included in pending or approved NDAs. We also excluded approval documents for non‐NME 505(b)(2) applications, which are non‐NMEs with a new indication, dosage form, or regimen supported by evidence packages not generated by the applicant. Additionally, we excluded medical gas approvals and unidentified NDAs.

We included all approval documents for original biologic products licensed by CBER or CDER over the same time period. To focus the analysis principally on novel drugs examined by the CDER, and vaccines and biologics examined by the CBER and CDER, we excluded assays, solutions, and blood products, including blood grouping reagents, coagulation factors, antihemophilic factors, pooled immunoglobulin, plasminogen, and source plasma.

We did not consider FDA‐approved efficacy supplements (e.g., additions or modifications of indications or claims) because FDA does not routinely release approval documents for these supplements to the general public; indeed, of the 443 FDA‐approved efficacy supplements, FDA only published approval documents for 29 medical products. Finally, for practical reasons, we excluded FDA‐approved NDAs and BLAs where the associated approval documents were not yet available on our study cutoff date of June 30, 2021.

Review of approvals and data abstraction

In analyzing approvals, we sought (i) to quantify what proportion of approvals incorporated RWE studies in any manner, and (ii) of those, how the applicant intended to make use of the RWE studies (i.e., to support therapeutic context, safety, and/or effectiveness). Among the approvals that included an RWE study to support medical product safety and/or effectiveness, we sought to classify (3a) how, if at all, the RWE study supported FDA’s benefit‐risk considerations, (3b) whether the RWE study and/or its findings were referenced in the product label, and (3c) in which therapeutic area the medical product with an RWE study fell. We further (4) qualified FDA’s documented feedback on these RWE studies to understand FDA’s evidence expectations. We additionally reported all findings stratified by year to observe any evolving time trends in FDA’s use of RWE.

Proportion of approvals incorporating RWE

To measure the proportion of included approvals that incorporated RWE studies (item 1), we evaluated whether the approval drew upon RWD in any form. Following FDA’s definition of RWD, ⁴ we considered an approval to include RWD if the approval’s included studies contained data from medical claims, electronic health records, disease and product registries, and/or used secondary analysis of patient‐ and physician‐reported outcomes. We examined all associated documents associated with each approval, reviewing each section and each reference for RWD. In some cases, the Agency clearly denoted a study as using RWD. In other cases, we assessed the underlying publication or study to determine whether it should be characterized as RWD. Only completed studies were considered.

Intended use of RWE

To classify how the applicant intended to make use of the RWE studies (item 2) in the context of the application, we categorized each RWE study by the applicant’s presumed intent. We developed three non‐mutually exclusive categories: RWE used to support the application’s therapeutic context (e.g., prevalence and incidence of a disease), RWE studies to support the demonstration of product safety, and RWE studies to support the demonstration of product effectiveness. To determine how to categorize the study, we used the applicant’s expressed intent of the RWE study where it was explicitly stated. Where the applicant did not state the intent, we categorized the study based on which section of the approval documents the RWE study was discussed (e.g., therapeutic context section or benefit‐risk section; Table  1 ).

Table 1.

Categorization applied to applicants’ intended use of RWE in NDAs and BLAs

Category	Sources of information on RWE study	Examples
RWE supports therapeutic context	RWE study appeared in either of: CDER’s published reviews section on therapeutic context CBER’s Summary Basis for Regulatory Action section on disease background	RWD establishes burden of disease to demonstrate need for the product (incidence and prevalence study)
RWE supports the demonstration of product safety	RWE study appeared in any of: Benefit‐Risk Assessment Review of Safety Product label	RWD serves as a reference for clinical trial adverse event rates (descriptive study) RWD provides both treatment and control information in a head‐to‐head study RWD serves as a control in an externally controlled trial
RWE supports the demonstration product effectiveness	RWE study appeared in any of: Benefit‐Risk Assessment (NDA/BLA) Conclusions on the Substantial Evidence of Effectiveness (NDA) Integrated Overview of Efficacy (BLA) Sources of Clinical Data and Review Strategy (NDA) Sources of Clinical Data and Other Information Considered in the Review (BLA) Statistical and Clinical Evaluation (NDA/BLA) Discussion of Individual Studies/Clinical Trials (BLA) Product label (NDA/BLA)	RWD serves as a treatment and comparator in a observational head‐to‐head study (comparative study) RWD serves as a control in an externally controlled trial (comparative study)

BLA, biologics license application; CDER, Center for Drug Evaluation and Research; NDA, new drug application; RWD, real‐world data; RWE, real‐world evidence.

Use of RWE to support of benefit‐risk considerations

We further examined the subset of studies that we classified as intending to support product safety and/or effectiveness. For our item (3a), we quantified how, if at all, the submitted RWE study supported FDA’s benefit‐risk considerations (Table  2 ). ²⁴ Benefit‐risk assessment is an FDA framework for regulatory decisions determined based on whether the benefit of a product outweighs known and potential risks, and relies on underlying assessments of safety and effectiveness. ²⁴

Table 2.

Categorization applied to how submitted RWE supported FDA’s benefit‐risk considerations

Category	Definition	Source documentation considered
Substantial evidence (CDER) 23 or primary evidence (CBER)	FDA’s documents explicitly used the phrase “substantial evidence” (CDER) or “primary evidence” (CBER)	For CDER‐approved products, published reviews section on Conclusions on the Substantial Evidence of Effectiveness (Section 1.2) For CBER‐licensed products, Summary Basis for Regulatory Action sections on Introduction (Section 1) and Recommendations and Risk/ Benefit Assessment (Section 11) Product label
Supportive evidence	FDA’s documents noted how the RWE study influenced its decision and/or the RWE study was referenced in the label	For CDER‐approved products, published reviews sections Benefit‐Risk Assessment (Section 1.3), Sources of Clinical Data and Review Strategy (Section 7), (8.2) Review of Safety (Section 8.2), and Conclusions and Recommendations (Section 8.4) For CBER‐licensed products, Summary of Basis for Regulatory Action sections Clinical/Statistical/Pharmacovigilance (Section 6) and Safety (Section 7) Product label
Not adequate for decision making	FDA’s documents affirmatively stated that the RWE study was not evidentiary; in some cases FDA spoke to the study limitations	Approval documents in full
RWE studies that FDA did not address	We did not identify any comments on or references to the RWE study in FDA documentation	Approval documents in full

CBER, Center for Biologics Evaluation and Research; CDER, Center for Drug Evaluation and Research; FDA, US Food and Drug Administration; RWE, real‐world evidence.

To measure whether the study was supportive of the benefit‐risk assessment, we applied methods developed through a preliminary review of FDA documentation (see below) and classified studies as informing FDA’s benefit‐risk assessment if they provided substantial evidence (by CDER), primary evidence (by CBER), or supportive evidence. Studies that we classified as not informing benefit‐risk assessment were those that, from the FDA reviews, appeared inadequate to support FDA’s decision making, or were simply not addressed in the reviews.

From the learnings of our preliminary review, we identified studies that served as substantial or primary evidence by looking specifically for those phrases (Table  2 ). ²⁴ As an example, in CDER’s multi‐discipline review of the tuberculosis medication pretomanid, we noted the phrase “the Applicant has provided substantial evidence of effectiveness… in a single phase 3 clinical trial… compared to [RWD] historical controls.” ³³ (In this application, the controls were drawn as individual patients observed in an electronic medical record system over a 6 year period, and were considered both as a group and as individually matched to patients in the treated arm.) Similarly, in CBER’s Summary Basis for Regulatory Action on onasemnogene abeparvovec‐xioi, a gene therapy for spinal muscular atrophy (SMA), we noted the phrase “comparison of the results of the ongoing clinical trial to available natural history data of infants with SMA provides primary evidence of the effectiveness of [the product]” ³⁴ (bolding added by the authors).

Categorization as supportive evidence was more qualitative (Table  2 ), ²⁴ with phrases such as “sufficient evidence,” “no formal statistical comparisons were made,” and “the study did, nevertheless, demonstrate…” taken as indicators. For example, in a multi‐discipline review of selumetinib, a therapy for children with neurofibromatosis type 1 (NF1) and symptomatic, inoperable plexiform neurofibromas (PNs)—a rare disease— FDA wrote “a natural history [RWE] study of NF1… was submitted with the application to provide external control data; however, no formal statistical comparisons were made by the FDA. The study did, nevertheless, demonstrate that a key characteristic of NF1 PN is the uncommon occurrence of spontaneous regression such that the observed tumor responses in [the] SPRINT [clinical study] are concluded to be the effect of the drug.” ³⁵ In another example, in CDER’s summary review of the coronavirus disease 2019 (COVID‐19) treatment remdesivir, the compiled safety database was comprised of a phase III clinical trial as well as data from administration of the medication under Emergency Use Authorization and in a compassionate use program. We noted FDA’s direct use of the word “support”: “The Agency deemed the safety database adequate to support the NDA.” ³⁶

Categorization as not adequate to support decision making was also qualitative based on direct statements and/or documented issues with submitted RWE. For example, one review contained both, stating “Due to major methodological issues (including immortal time bias, selection bias, misclassification, confounding, and missing data), the FDA does not consider these [RWE] results adequate to support regulatory decision making.” ³⁷

Finally, for our category of studies that went unaddressed, we noted cases where FDA neither directly addressed the submitted RWE positively or negatively, nor made reference to the RWE in the approval documents. More broadly, not using or addressing the submitted RWE did not necessarily imply that FDA did not approve the application, but rather that the RWE portion did not appear to affect the decision making.

Use of RWE in product labels

Among the approvals considered, we assessed the associated product labels to identify which included RWE submitted as part of the application (item 3b). If a product label included either the RWE study name or its findings, we categorized the approval as having RWE referenced in the product label.

Use of RWE in therapeutic areas

Among the approvals considered, we cross‐referenced the medication with FDA and external resources to identify the product’s primary therapeutic area (item 3c). Our main source was FDA’s Spectrum of Diseases by Therapeutic Area Found in Written Requests document, ³¹ which maps many diseases to one of 20 therapeutic areas. Based on each drug’s primary indication, we categorized therapeutic area according to this FDA taxonomy; if the disease treated by the medication was not included in FDA’s list, we categorized the medical product’s therapeutic area based on our understanding of the disease and how applicants titled their clinical development programs and marketing franchises.

FDA’s documented feedback on use of RWE

Finally, to categorize FDA’s publicly available feedback according to key themes (item 4), we reviewed FDA documents on the approvals that included an RWE study to inform medical product safety or effectiveness, either successfully or not. We sought to better understand whether FDA deemed the evidence adequate in its decision making, and if not, why not.

We developed four non‐mutually exclusive categories: feedback noting methodological issues (e.g., immortal time bias, confounding, and lack of comparability between trial data and RWD); sample size concerns (e.g., underpowered or unstable estimates); omission of patient‐level data (i.e., lack of transparency, lack of ability for FDA to re‐analyze the data and/or perform subsequent analysis or re‐analysis); and other limitations that did not fit the categories above. We applied the categorization after considering FDA’s documented feedback on the reported reasons for an RWE study's not being adequate for decision making.

The data abstraction for all of our classifications was conducted by two investigators (C.P. and N.H.) who independently assessed the approval documents for each medical product. As necessary, reviewers resolved any discordance via a discussion of relevant passages with input solicited from other investigators not otherwise involved.

RESULTS

Selection of approvals

Among the 378 FDA‐approved NDAs or BLAs identified from January 1, 2019, to June 30, 2021, we determined that 136 (36%) of applications met our inclusion criteria. Among the 242 that did not, 237 (98%) were excluded because they were outside our criteria for new molecular entities or original biologic products (Figure  1 , Table  S1 ), and an additional 5 (2%) were excluded because documentation was not yet available for public review as of the study’s June 30, 2021 cutoff (Table  S3 ).

Figure 1.

Inclusion of FDA‐approved NDAs and BLAs between January 2019 and June 2021. BLA, biologics license application; FDA, US Food and Drug Administration; NDA, new drug application; NME, new molecular entity; RWE, real‐world evidence.

We found that 116 approvals among the 136 (85%) included RWE in any form (item 1). The proportion of approvals including an RWE study increased from 2019 to 2021, with 38 of 51 (75%) approvals including an RWE study in 2019, 53 of 59 (90%) in 2020, and 25 of 26 (96%) in the first half of 2021 (Table  3 ).

Table 3.

Observed intended use of RWE in included NDAs/BLAs, January 2019 to June 2021

Included NDAs and BLAs	2019 n = 51 approvals	2020 n = 59 approvals	2021 through June 30 n = 26 approvals	Total n = 136 approvals
Incorporated RWE for any purpose	38 (75%)	53 (90%)	25 (96%)	116 (85%)
Used RWE to provide therapeutic context	25 (49%)	36 (61%)	22 (85%)	83 (61%)
Used RWE to support safety and/or effectiveness	27 (53%)	46 (78%)	15 (58%)	88 (65%)
Safety only	17 (33%)	21 (36%)	5 (19%)	43 (32%)
Effectiveness only	7 (14%)	6 (10%)	2 (8%)	15 (11%)
Safety and effectiveness	3 (6%)	19 (32%)	8 (31%)	30 (22%)

Categories are not mutually exclusive.

BLA, biologics license application; NDA, new drug application; RWE, real‐world evidence.

In categorizing the applicants’ intended use of RWE studies (item 2), we determined that a high proportion supplied RWE with the intent to provide evidence of product safety or effectiveness, with 88 of 136 (65%) of approvals using RWE studies to provide such evidence. 83 of 136 (61%) used RWE studies with the intent to provide therapeutic context (Table  3 ). The use of RWE studies to provide therapeutic context appeared to be trending upwards from 2019 to 2021 (from 49% in 2019 to 85% in the first half of 2021), whereas use of RWE studies to provide evidence of safety or effectiveness increased from 27 of 51 (53%) in 2019, to 46 of 59 (78%) in 2020, but decreased to 15 of 16 (58%) in the first half of 2021.

Table  S4 notes the 20 FDA‐approved NDAs and BLAs without an RWE study, as well as the 28 FDA‐approved NDAs and BLAs where the applicant only intended to use RWE studies to support therapeutic context.

For item (3a), we further categorized the 88 approvals intended to support product safety or effectiveness by assessing whether and how the RWE studies supported FDA’s assessment of benefit‐risk. We found that among these approvals, FDA appeared to use 65 of the 88 RWE studies (74%) in its assessments, with 8 (9%) classified as substantial/primary evidence, and 57 (65%) as supportive evidence. The RWE studies appeared inadequate for FDA decision making in 13 (15%) cases, whereas in the remaining 10 (11%) cases, FDA did not directly address the provided RWE study in its published reviews (Table  4 ). Although the relatively small numbers make it difficult to identify trends, it appears that RWE is serving at least as supportive evidence in a growing proportion of approved applications over the time period studied. Separately, among the 88 approvals, 38 (43%) of the product labels referenced the RWE studies and/or findings (item 3b).

Table 4.

Observed FDA use of applicant‐submitted RWE in considered approvals, January 2019 to June 2021

FDA’s use of RWE	2019 n = 27 approvals	2020 n = 46 approvals	2021 through June 30 n = 15 approvals	Total n = 88 approvals
Substantial or primary evidence	3 (11%)	4 (9%)	1 (7%)	8 (9%)
Supportive evidence	16 (59%)	30 (65%)	11 (73%)	57 (65%)
Not adequate for decision making	6 (22%)	4 (9%)	3 (20%)	13 (15%)
RWE studies that FDA did not address	2 (7%)	8 (17%)	0 (0%)	10 (11%)

FDA, US Food and Drug Administration; RWE, real‐world evidence.

These 88 approvals spanned 16 therapeutic areas (Tables  5 , S5 ), with RWE submitted most commonly in oncology (30 of 43 approvals in that therapeutic area), infectious disease (16 of 17), neuroscience (13 of 25), and endocrinology and metabolism (8 of 14). FDA considered the evidence substantial/primary or supportive most commonly in the same therapeutic areas: oncology (16 of 43 approvals in that therapeutic area), infectious disease (12 of 17), neuroscience (11 of 25), and endocrinology and metabolism (6 of 14). However, when including RWE in the product label, the most common therapeutic areas differed somewhat: infectious disease (11 of 17 approvals in that therapeutic area), neuroscience (9 of 25), endocrinology and metabolism (4 of 14), oncology (3 of 43), and radiology (3 of 6).

Table 5.

Observed use of RWE in FDA approvals by therapeutic area, January 2019 to June 2021

Therapeutic area	Total approvals per therapeutic area of n = 136 approvals	RWE intended to support safety and/or effectiveness in n = 88 approvals	RWE used as substantial/primary or supportive evidence in n = 65 approvals	RWE studies referenced in product label in n = 38 approvals
Oncology	43	30 (70%)	16 (37%)	3 (7%)
Neuroscience	25	13 (52%)	11 (44%)	9 (36%)
Infectious disease	17	16 (94%)	12 (71%)	11 (65%)
Endocrinology and metabolism	14	8 (57%)	7 (50%)	4 (29%)
Radiology	6	6 (100%)	6 (100%)	3 (50%)
Hematology	5	2 (40%)	2 (40%)	1 (20%)
Dermatology	4	2 (2%)	2 (3%)	1 (3%)
Ophthalmology	4	0 (0%)	0 (0%)	0 (0%)
Gastroenterology	3	2 (66%)	2 (66%)	2 (66%)
Allergy	2	1 (50%)	0 (0%)	0 (0%)
Anesthesiology	2	1 (50%)	1 (50%)	1 (50%)
Cardiovascular	2	2 (100%)	2 (100%)	1 (100%)
Gynecology	2	1 (50%)	1 (50%)	1 (50%)
Inflammation and immunology	2	0 (0%)	0 (0%)	0 (0%)
Urology	2	1 (50%)	1 (50%)	1 (50%)
Autoimmune	1	1 (1%)	1 (2%)	0 (0%)
Cosmetic	1	1 (100%)	0 (0%)	0 (0%)
Respiratory	1	1 (100%)	1 (100%)	0 (0%)

FDA, US Food and Drug Administration; RWE, real‐world evidence.

FDA provided publicly documented feedback on the RWE studies in 37 of 88 approvals that included an RWE study to support medical product safety and/or effectiveness (item 4). Across all documented studies, the issues noted in FDA’s feedback were methodological issues (n = 23), sample size concern (n = 8), omission of patient‐level data (n = 3), and other limitations (n = 13). Some studies had multiple issues identified. Examples of each category are in Table  6 , with further detail in Tables  S6 and S7 .

Table 6.

Examples of FDA’s feedback on use of RWE and/or the submitted RWE study

Category of FDA’s feedback	Product’s primary basis of approval	FDA feedback on example studies
Methodological issues	Phase II, single‐arm study	Inadequate due to missing data, differences in follow‐up and response assessment, population heterogeneity, and bias in end point assessment 51
	Single‐arm study	Supportive with critiques: small sample size; lack of comparator arm; limitations of retrospective observational data; time‐to‐event end points not interpretable 52
	Phase II, single‐arm cohort Phase I/II study	Inadequate due to determination and validation of endpoints, selection bias, confounding factors, and generalizability 53 , 54
	Phase II study	Inadequate due to variability in natural history of disease, imprecision of population matching, and selection bias 55 , 56
General limitations	Phase I study	Inadequate without detail 57
	Phase III study	RWE did not overrule safety signal from clinical trials 58
	Phase II study	Premise of RWE study deemed controversial 59
Sample size concern	Phase II trial and phase I/II study	Inadequate data collection 60 , 61
	Phase II, single‐arm study	Inadequate due to limited data 62
	Three phase III studies	Limited sample size precluded it from FDA’s consideration for safety and effectiveness 63 , 64
Omission of patient‐level data	Phase II study	Supportive because it strengthened clinical relevance, although “… FDA cannot independently verify these results” 65

FDA, US Food and Drug Administration; RWE, real‐world evidence.

DISCUSSION

Regulatory agencies worldwide are considering how they may incorporate RWE into their decision‐making processes. In this study, we detailed how over the past two and a half years, both the quantity of RWE submitted and its impact on FDA’s decision making—particularly as supportive evidence—have significantly increased and, downstream, that RWE studies are appearing in US product labels. However, we also saw that not all RWE was accepted as the applicant intended; in a large number of cases, FDA either did not consider the RWE at all or found it inadequate to support an approval. In these cases, a consistent theme emerged: methodologically‐sound RWE rooted in principled study design and analysis supported FDA’s decisions, whereas studies that FDA found not to meet this bar bore far less impact.

Most approvals where RWE studies impacted decision making were in oncology, infectious disease, and neuroscience. Although we are not able to draw a direct line between these therapeutic areas and why they make up the largest portion of RWE use, we note that they have some common features: there is substantial drug development effort in each of these areas, the seriousness of many diseases and conditions in these areas commonly leads to eligibility for expedited programs ³⁸ (which may in turn look to rapid evidence generation), and they may use expanded access programs ³⁹ (which may generate data on real‐world experience). Further, in certain diseases within these areas, small patient populations, impracticality of running randomized trials, or ethical issues with using a placebo control arm may be favorable for the use of RWE.

Studies that support regulatory approvals are rooted in good study design, appropriate data collection, and thoughtful data analysis; this is true whether the study is a randomized or a nonrandomized, real‐world design. We saw instances where FDA directly addressed this in their comments. For example, in one case, an FDA document stated, “The Agency agreed to the use of the external contemporaneous control in the matched analysis because the trial investigators and Applicant took adequate measures to minimize potential sources of bias. The trial investigators undertook rigorous efforts to identify patients with [Hutchinson‐Gilford progeria syndrome] globally and offered all patients the same opportunity to participate in the natural history study, reducing potential selection bias. Known potential confounders (age at treatment initiation, variant status, sex, and continent of residence) were adequately addressed in the analyses.” ⁴⁰

However, as noted above, FDA also cited instances where the submitted evidence did not meet the threshold to be included in their decision making, either directly stating that and/or identifying methodological issues in RWE generation. One review stated, “There are several methodological limitations of this comparative [RWE] analysis outlined below that impact the interpretability of the study results,” then went on to list four specific issues in detail: missing data, differences in the endpoint definitions or follow‐up period used to ascertain them, treatment heterogeneity among multiple standard of care therapies, and selection bias due to lack of comparability. ⁴¹

Reliable RWE is built on using fit‐for‐purpose RWD, and data‐related challenges were cited in FDA’s documented feedback. In one example, FDA found that patients represented in the selected RWD source were not comparable to those in an associated clinical trial, saying “differential selection of comparison groups, specifically, that trial patients were enrolled from academic medical centers primarily from Europe, while [RWD] controls were enrolled from community oncology centers in the US.” ⁴² To avoid issues like this, a focus on identifying RWD that meet expectations around relevance and reliability, ⁴³ ensuring the data speak to the question at hand and are high‐quality, is critical. In assessing relevancy, applicants can ensure that the data represent the intended population, that key variables such as exposure, outcome, and confounders are captured and measurable, and that data are sufficiently longitudinal for the study. In assessing quality, applicants can assess accuracy, completeness, and consistency. Our research suggests that spending substantial effort on this phase can avoid some of the downstream issues that lead to RWE studies not being adequate for decision making.

With fit‐for‐purpose data in hand, identifying an appropriate study design is a key next step. Framing questions in a meaningful way for the ultimate decision‐making audience and enforcing designs and analyses that are transparent, auditable, and reproducible is crucial. ⁴⁴ , ⁴⁵ One reliable design strategy applicants can use is designing the RWE study to emulate either an actual or hypothetical target trial. ⁴⁶ , ⁴⁷ , ⁴⁸ In emulating a target trial, researchers are advised to mimic all design aspects of a randomized trial, and use confounding identification and adjustment to mitigate bias in the absence of randomization. This ensures that key areas of design, including eligibility criteria, treatment strategies, start and end of follow‐up, outcomes, causal contrast, and the analysis plan are methodologically sound.

In some instances in the documented feedback, the importance of communication with FDA in the planning, design, and review phases of an RWE study was referenced. In the planning phase, early and frequent communication with FDA helps to ensure “good RWE” ⁴⁹ by providing a channel for study design feedback and an opportunity to adjust any issues. Feedback provided in formal meetings provides an early signal on whether a particular RWE approach is likely to support FDA’s benefit‐risk assessment. In one case where the submitted RWD was deemed not adequate for decision making, FDA highlighted that lack of prior communication about the RWD: “As previously mentioned, without having reviewed and consented to a protocol and statistical analysis plan (SAP), FDA cannot be certain that the protocol and SAP were pre‐specified and unchanged during the data selection and analyses. This uncertainty and the knowledge that subsequent unmasked analyses have been performed could lead to overly optimistic conclusions.” ³⁷

However, in another case where the RWE was ultimately deemed substantial, FDA took specific note of how the applicant heeded the Agency's early feedback. In this case, the applicant had originally published the mortality data of their medical product prior to creation of a SAP. As a result, the Agency stated in their meeting to discuss the initial clinical study, “Ideally, analysis plans for marketing applications should be submitted to the FDA for agreement before analyzing the data; this was a missed opportunity for the program and may be a review issue.” ⁴⁰ Noting this, the applicant formally proposed to conduct the same analysis again as a part of the SAP. In the final approval, FDA stated: “The Applicant agreed with the Agency’s recommendation and revised the SAP with a cut‐off date of June 1, 2019 for follow‐up. The SAP was also revised based on Agency feedback pertaining to the matching approach and censoring. The revised SAP was deemed acceptable by the Agency prior to NDA submission, with the caveat that analyses were retrospectively planned and considered post‐hoc. During the course of the NDA review, issues with the analyses were identified and adequately resolved.” The RWE, in this case, was deemed substantial evidence. ⁴⁰

In the review phase, applicants should allow FDA reviewers transparent access to the analyses (e.g., ability to trace data provenance, understand all data transformations, and see the order in which methods are applied), and provide them the option to reanalyze and/or conduct post hoc sensitivity analyses on the data. In our review, we found that FDA may place importance on being able to independently analyze data, in one case stating “While FDA considers data from Study X2102 and [RWD] Study X2401 to be supportive, [the sponsor] did not submit this data, and thus FDA cannot independently verify these results.” ⁵⁰

We note three limitations. First, our systematic review covered all of 2019 and 2020, but only the first half of 2021. In past years, the majority of decisions have occurred in the second half of the year, so trends including 2021 data should be interpreted in this light. Second, our evidence base is limited to FDA‐approved NDAs and BLAs because complete response letters (which FDA sends when it cannot approve an application in its current form) are confidential between the applicant and the Agency. In addition, documents regarding efficacy supplements are not necessarily confidential but are also not routinely made public, although they may be available through a Freedom of Information Act request. Third, due to the low sample size, we were not able to draw strong conclusions about the role of RWE (e.g., “RWE was most likely to be deemed substantial evidence if…”), and we did not feel it appropriate to identify trends by design or analytic choice (e.g., “X type of RWE most often provided substantial evidence”), largely because of the heterogeneity in uses of RWE.

CONCLUSIONS

As the use of RWE in regulatory applications increases, there is opportunity to improve both the understanding of the expectations of worldwide regulatory agencies around the utility, substance and quality of RWE studies, as well as applicants’ adherence to such expectations. Over the next years, we expect that further clarification of these expectations will emerge, but feel confident that they will be built upon the foundations of fit‐for‐purpose RWD selection and careful attention to study design and analysis. As questions remain as to what specific guidances will be provided or how well they are being followed, further studies such as these, augmented with specific demonstration projects, can continue to build empirical evidence on how RWE can meet regulatory agencies’ expectations and improve public health.

FUNDING

This study was funded internally by Aetion.

CONFLICT OF INTEREST

All authors are employees of and hold stock options or equity in Aetion, Inc. No other interests are declared.

Supporting information

Tables S1–S9