FRAME: Framework for Real‐World Evidence Assessment to Mitigate Evidence Uncertainties for Efficacy/Effectiveness – An Evaluation of Regulatory and Health Technology Assessment Decision Making

Abstract

Real‐World Evidence (RWE) is increasingly used in submissions to regulatory agencies and health technology assessment bodies (HTAbs) to support the efficacy and effectiveness of new medicines and indications. However, there is limited information on the RWE characteristics that impact its role in approval and reimbursement decisions. To investigate these characteristics, we developed FRAME: a Framework for Real‐world evidence Assessment to Mitigate Evidence uncertainties for efficacy/effectiveness. We compiled a list of medicinal product indications where RWE supported the efficacy of interventional trials or assessed effectiveness in observational settings. FRAME was applied to a prioritized subset of these submissions to authorities from North America, Europe, and Australia. For each product indication, we extracted information on characteristics describing the submission, clinical context, strength of evidence, and process factors from publicly available assessment reports. Of the 87 identified medicinal product indications, 15 were prioritized, covering 68 submissions and 76 RWE studies across 11 authorities in scope. Four main results emerged: (i) low granularity within assessment reports on the analyzed variables, limiting the learnings from analyzing them; (ii) variability in how RWE was assessed within and across regulatory agencies and HTAbs; (iii) a positive association between the proportion of positive comments from authorities on RWE studies and their impact on decision making. Particularly, a large effect size was consistently noted when RWE was considered primary evidence; and (iv) limited use of advanced RWE study designs. These findings support five recommendations to enhance shared learning on RWE, clarify its evidentiary value, and generate evidence to better support authorities' decision making.

Study Highlights.

• WHAT IS THE CURRENT KNOWLEDGE ON THE TOPIC?

Real‐World Evidence (RWE) is increasingly used in approval and reimbursement submissions. However, knowledge is lacking on the characteristics of RWE to support efficacy and effectiveness claims that impact its role in approval and funding decisions.

• WHAT QUESTION DID THIS STUDY ADDRESS?

Which characteristics of RWE used to support efficacy/effectiveness claims impact regulatory and Health Technology Assessment body decision making in North America, Europe, and Australia?

• WHAT DOES THIS STUDY ADD TO OUR KNOWLEDGE?

This study systematically identified and analyzed a comprehensive set of characteristics describing the use and consideration of RWE regarding efficacy/effectiveness claims in authority decision making. We examined a wide range of products in multiple therapeutic areas, spanning numerous authorities and regions.

• HOW MIGHT THIS CHANGE CLINICAL PHARMACOLOGY OR TRANSLATIONAL SCIENCE?

The research results may facilitate multi‐stakeholder dialogue and inform actions aimed at improving shared learning on RWE, clarifying its evidentiary value, and facilitating the generation of evidence that better meets authorities' needs.

Real‐World Evidence (RWE) is recognized as an important component of the totality of evidence for medicinal products. ¹ , ² , ³ , ⁴ , ⁵ , ⁶ , ⁷ , ⁸ It is increasingly used in submissions to regulatory agencies and health technology assessment bodies (HTAb) (hereafter, collectively, authorities) to support the approval and funding of new medicines and indications. ⁹ , ¹⁰ , ¹¹ , ¹² , ¹³

Recognizing RWE's increasing importance, several authorities have developed strategies, frameworks, guidance and pilot programmes on its use to support their evidentiary needs. In 2018 the US Food and Drug Administration (FDA) developed a framework for evaluating the potential use of RWE and, as part of it, in 2022 launched the FDA Advancing RWE Program. ¹⁴ , ¹⁵ The European Medicines Agency (EMA) and the European Medicines Regulatory Network have also established a framework to enable better integration of RWE into regulatory decisions. ¹⁶ , ¹⁷ , ¹⁸ In addition, HTAbs, including England's National Institute for Health and Care Excellence (NICE), France's Haute Autorité de Santé (HAS) and Canada's Drug Agency (CDA‐AMC) have established RWE framework and guidance. ⁴ , ⁵ , ¹⁹

Whilst RWE has long been used in post‐marketing safety evaluation and in characterizing disease epidemiology, it has been used less often to inform authorities' decision making around efficacy or effectiveness. Therefore, details are scarcer about how authorities consider RWE for such decision making. To fully harness the opportunity for RWE in this area, there is a need for a more granular understanding of the characteristics that impact its acceptability and use. This could increase predictability and consistency of interpretation, facilitating harmonization efforts across authorities. In turn, this could contribute to making evidence generation more efficient, enhancing patient access to medicines.

A growing base of literature has quantified the use of RWE for efficacy/effectiveness evaluations and the role it played in those decisions. ⁹ , ¹¹ , ¹² , ²⁰ , ²¹ , ²² , ²³ , ²⁴ However, to our knowledge, no systematic investigation has identified a comprehensive set of characteristics and examined how these are considered in authority decision making across multiple products, therapeutic areas, study designs, and authorities. To address this gap, we developed FRAME: a Framework for Real‐world evidence Assessment to Mitigate Evidence uncertainties for efficacy/effectiveness. The methodological framework consists of four steps (Figure S1 ): (i) identify the characteristics that could impact the consideration of RWE in authorities' decision making; (ii) establish a consistent and reproducible process for extracting and analyzing information related to each of these characteristics; (iii) extract and categorize the information, and (iv) analyze it using both qualitative and quantitative methods.

We implemented FRAME on publicly available assessment reports from a subset of submissions from authorities in North America, Europe and Australia in order to:

• Summarize the decision context and intended purpose of the RWE.

• Describe the assessment reports' level of granularity.

• Investigate convergences and divergences in how RWE was considered across authorities, and

• Examine which characteristics might be associated with an increasing role of RWE in decision making.

By taking a holistic approach to this multi‐factorial research question, we seek to support the identification of, and create multi‐stakeholder dialogue on, the optimal approaches that align with authorities' needs and increase confidence in the use of RWE to support efficacy/effectiveness.

MATERIALS AND METHODS

Authorities in scope

Five regulatory agencies and six HTAb in North America, Europe, and Australia were included to provide additional insights on the generalizability of results and on the consistency across authorities. Authorities were selected based on the public availability of their assessment reports and the language these were written in (English, German and French).

We included the following regulatory agencies: EMA (European Union), Medicines and Healthcare products Regulatory Agency (MHRA; UK), FDA (USA), Health Canada (HC; Canada), and Therapeutic Goods Administration (TGA; Australia). Where possible, HTAbs were chosen to correspond with the regulatory agencies: Federal Joint Committee (G‐BA; Germany), HAS (France), NICE (England), Institute for Clinical and Economic Review (ICER; USA), CDA‐AMC (Canada) and Pharmaceutical Benefits Advisory Committee (PBAC; Australia).

Identification of product indications: Search and prioritization strategy

We conducted a two‐stage search and prioritization strategy (Figure S2 ).

Stage 1: Search – We compiled an extensive list of medicinal product indication (hereafter product) submissions that included RWE to support the efficacy of interventional trials or assess effectiveness in observational settings. Sources included publications on the use of RWE in authority decisions identified through a targeted review, ¹¹ , ²⁰ , ²¹ , ²² , ²³ , ²⁴ gray literature including presentations from authorities, ²⁵ and case studies from relevant data/technology companies. ²⁶ , ²⁷

Stage 2: Prioritization – For the identified submissions, we screened authorities' assessment reports to confirm the use of RWE for efficacy/effectiveness and gather additional information to determine which to prioritize for in‐depth review.

For the prioritization, the following was considered:

• Select submissions dated between January 2017 and June 2024 to reflect the period associated with increasing interest around RWE.

• Identify at least one submission per authority in scope, and to enable comparisons, prioritize products assessed by multiple authorities.

• Include both submissions with a positive and negative opinion, as well as where RWE contributed to decision making and where it did not.

• Ensure a balanced range of values across the following factors that could impact the contribution of RWE to authority decisions: (i) types of application (new Marketing Authorisation Applications—MAA, Extension of Indication—EoI), (ii) therapeutic area, (iii) orphan designation, (iv) rarity, and (v) study design.

Regarding the study designs, we classified them following the FDA guidance ²⁸ and corresponding literature. ²⁹ , ³⁰ , ³¹ This led to consider the following spectrum: Randomized Controlled Trials (RCT) that use RWD (shortened to RCT with RWD); Single‐Arm Trials (SAT) analyzed with an external control arm that relies on RWD (SAT contextualized); and observational studies.

Characteristics identified and analyzed in each submission (step 1 of FRAME)

We identified relevant submission characteristics and then divided them into two sets of variables.

The first set characterized the submission and type of RWE, including information such as indication, dates, procedural pathway, study design, and data source (43 variables; Table S1 ).

The second set, representing the core of our analysis, included the RWE's role in authority decisions, and the characteristics that could have impacted it. These latter variables (30 variables; Figure 1 , Table S2 ) were determined based on an analysis of authorities' guidance, publicly available case studies from FDA, literature on best practice for generating high‐quality RWE ³² , ³³ , ³⁴ , ³⁵ and the authors' experience. They can be grouped into three areas:

• Clinical context (12 variables): these describe circumstances that align with authority considerations for more flexibility in evidentiary approach and tolerance of uncertainty, such as disease severity, unmet need, or RCTs' ethical or feasibility challenges.

• Strength of evidence (16 variables): these include characteristics of the different evidence sources: pre‐clinical, clinical interventional, and non‐interventional studies. The focus was on the latter, for which a detailed list of 13 attributes was compiled. These were categorized as follows: data source ³² (5 variables), study design ³³ , ³⁴ , ³⁵ (7 variables), and treatment effect size (1 variable).

• Process (2 variables): these capture clear procedural recommendations from authorities.

Figure 1.

Key characteristics identified through FRAME which could impact authorities' decision making on RWE for efficacy/effectiveness. RCT, randomized controlled trial; RWE, real‐world evidence.

The three areas can be conceptualized as a continuum of applicant control. Clinical context encompasses factors that sponsors have limited to no control over (e.g., condition severity). Process refers to factors over which the sponsor does have control (e.g., early interaction with the authority). Strength of evidence includes both sponsor‐independent (e.g., data availability) and sponsor‐dependent (e.g., study design) elements.

Methods for variable extraction and categorization (steps 2 and 3 of FRAME)

We developed a standardized data extraction form to capture relevant information about the identified variables from authorities' publicly available assessment reports (Table S3 ). For each HTAb, data were extracted solely from the final recommendation reports, rather than also including sources like clinical or economic assessment reports and committee papers. This was to provide an accurate reflection of the key factors mentioned in the final decision.

To increase the consistency and reproducibility of data extraction, we created detailed definitions for each variable that may have impacted the role of RWE in decision making (Figure 1 , definitions available in Table S2 ). In addition, the information extracted was summarized into categories. For the overall role of RWE, these were: primary, supportive, neutral/unclear, discussed and not used, or not addressed. To note that an RWE study could be primary or supportive to either a positive or negative overall outcome for the product submission. The variables that may have impacted the RWE's role were summarized with a positive (+), mixed (~), or negative (−) value. Detailed definitions were also created for each of these categories (Table S2 ); examples of their application can be found in Table S4 .

A minimum of two independent researchers systematically reviewed the authorities' reports to extract quotes and assign summary values. Any disagreement over the extracted quotes and their categorization was resolved through discussion with a panel of at least three independent reviewers.

Analysis (step 4 of FRAME)

Data extracted were analyzed both qualitatively and quantitatively. The qualitative analysis aimed to create case studies, identify common RWE themes, and highlight areas of convergence or divergence between authorities.

The quantitative analysis characterized the submission with focus on the RWE component. For each variable in our framework, we calculated the number of times it was addressed in the authorities' assessment reports, along with the distribution of the variable's corresponding summary values (+, ~, −). To investigate patterns in authorities' consideration of RWE, we stratified the proportion of summary values (+, ~, −) for each variable by the different categories of RWE's role. All analyses were also stratified by authority and product. When comparing results across authorities, analyses were based on the common subset of products assessed. However, for clarity, only aggregate results including all products assessed by an authority are presented in the results section. Inconsistencies with the comparison based on the common subset of products, if any, were noted in the text.

RESULTS

Characteristics of the submissions reviewed

The stage 1 search strategy identified 87 products, from which we prioritized 15 for detailed review (Table 1 , for a detailed characterization Table S5 and Figures S3 , S4 ).

Table 1.

Role of RWE in authority decision making by RWE study

Active substance	Brand name	Type of application	Therapeutic area	RWE study	EMA (n = 16)	MHRA (n = 0)	FDA (n = 16)	HC (n = 5)	TGA (n = 4)	G‐BA (n = 6)	HAS (n = 7)	NICE (n = 8)	ICER (n = 3)	CDA‐AMC (n = 7)	PBAC (n = 4)
idecabtagene vicleucel	Abecma	MAA	Oncology	NDS‐MM‐003				a		a	a			–a
				SLR
				MAMMOTH
erdafitinib	Balversa	MAA	Oncology	BRIDGE
				Flatiron‐FMI
avelumab	Bavencio	MAA	Oncology	100070_Obs001						–				c
atidarsagene autotemcel	Libmeldy	MAA	Neurology	Expanded access
				Natural history
lutetium Lu 177 Dotatate	Lutathera	MAA	Oncology	ERASMUS
131I‐omburtamab	Omblasyts	MAA	Oncology	CGCCR	–		–
entrectinib	Rozlytrek	MAA	Oncology	WO40977/Flatiron DB						–d	–	d		d
alpelisib	Vijoice	MAA	Oncology	EPIK‐P1	–
onasemnogene abeparvovec	Zolgensma	MAA	Neurology	PNCR & NeuroNext				e					b
				LT‐001						–			b
blinatumomab	Blincyto	EoI	Oncology	Study 20120148
				Neuf Study
palbociclib	Ibrance	EoI	Oncology	Flatiron DB
				Study A5481097
tenecteplase	Metalyse	EoI	Cardiovascular	AcT
				Additional RWE
catridecacog	NovoThirteen	EoI	Hematology	PASS NN1841‐3868
abatacept	Orencia	EoI	Rheumatology	IM101841/CIBMTR	–			f
tacrolimus	Prograf	EoI	Immunology	506‐CL‐3001

CDA‐AMC, Canada's Drug Agency; CGCCR, Central German Childhood Cancer Registry; CIBMTR, Center for International Blood and Marrow Transplant Research; DB, database; EMA, European Medicines Agency; EoI, Extension of Indication; FDA, Food and Drug Administration; G‐BA, Federal Joint Committee; HAS, Haute Autorité de Santé; HC, Health Canada; ICER, Institute for Clinical and Economic Review; MAA, Marketing Authorisation Applications; MHRA, Medicines and Healthcare products Regulatory Agency; NICE, National Institute for Health and Care Excellence; PASS, post authorisation safety study; PBAC, Pharmaceutical Benefits Advisory Committee; PNCR, Pediatric Neuromuscular Clinical Research Database; RWE, real‐world evidence; SLR, systematic literature review; TGA, Therapeutic Goods Administration.

– Submission not approved or withdrawn.

^a HC and HAS did not name NDS‐MM‐003 but referred to it descriptively. G‐BA assessed NDS‐MM‐003 together with PREAMBLE, while CDA‐AMC reviewed it with MAMMOTH.

^b Zolgensma was discussed, but no HTA final recommendation report was issued as it was too early in clinical development.

^c CDA‐AMC referenced Study 100070_Obs001 without naming it.

^d G‐BA, NICE, and CDA‐AMC referred to WO40977 as a study using the Flatiron Health Database, without naming it.

^e HC did not name PNCR or NeuroNext, calling them “natural history studies.” NICE reviewed PNCR, NeuroNext, ENDEAR, and De Sanctis et al. (2016), selecting NeuroNext as the key supportive natural history study.

^f HC described IM101841 without naming the study.

The prioritized products included nine MAAs and six EoIs. Seven products had orphan designation from all regulatory agencies, four from at least one agency, and four had none. Oncology was the most represented therapeutic area (n = 9), and one product was for a non‐rare disease.

All products were submitted to at least one of the regulatory agencies in scope, while seven were submitted also to at least one HTAb.

The 15 products corresponded to a total of 68 submissions across authorities, 38 to regulatory agencies, and 30 to HTAbs. FDA and EMA had the most submissions (n = 13), while ICER (n = 2) and MHRA (n = 1) had the fewest (Figure S4 ). Nine of the 68 submissions were not approved, four by regulators and five by HTAb (Table S6 ).

Characteristics of the RWE studies

Analysis of the 68 assessment reports showed that eight did not reference an RWE study, despite an indication that such a study had been submitted (i.e., one was referenced in other authority reports for the same product). This was most common in the HC assessment reports and included the only MHRA submission; therefore, no MHRA results are included in subsequent analyses.

The remaining assessment reports revealed a total of 76 RWE studies, representing 23 unique studies and covering all three types of designs identified by FDA guidance (Figure S5 ):

• SAT contextualized was the most common design (n = 12), primarily used in MAAs (n = 10) and for oncology products (n = 9). Most studies employed population‐level matching (n = 8), while three used naïve comparisons.

• Observational studies (n = 10) were evenly distributed between EoIs (n = 6) and MAAs (n = 4), as well as between oncology (n = 5) and non‐oncology products (n = 5). All were cohort studies; the majority were non‐comparative (n = 8).

• RCT with RWD (n = 1) was included in an EoI for a non‐rare disease. This represented an RCT linked to registry data providing information on patient baseline characteristics and outcomes.

Most studies relied on secondary data collection (n = 16), particularly for SAT contextualized, followed by primary data collection (n = 4), and both primary and secondary data collection (n = 3).

Granularity of information in public assessment reports

The analysis of the 30 variables identified as potentially impacting the role of RWE in authority decisions (Figure 1 ) revealed heterogeneity across authorities in the types and numbers of variables commented on (Figure 2 ).

Figure 2.

Average number of variables commented on by authorities per submission. x = number of product submissions; y = number of RWE studies assessed by the authority. Percentages indicate the percentage of variables commented on by the authority across all product submissions. For a detailed breakdown of the average number of variables commented on in each area, see Table S7 . FDA, Food and Drug Administration; EMA, European Medicines Agency; TGA, Therapeutic Goods Administration; HC, Health Canada; PBAC, Pharmaceutical Benefits Advisory Committee; CDA‐AMC, Canada's Drug Agency; ICER, Institute for Clinical and Economic Review; HAS, Haute Autorité de Santé; NICE, National Institute for Health and Care Excellence; G‐BA, Federal Joint Committee.

On average, FDA and PBAC commented on at least 50% of the variables, while TGA, HC, and G‐BA commented on less than one third.

Clinical context variables were most consistently discussed. On average, authorities commented on 49% to 63% of these variables, except for G‐BA and HC, whose assessment reports included 15% and 33%, respectively.

Discussion of Strength of evidence variables varied widely among regulators, ranging from 5% (HC) to 71% (FDA). In contrast, there was more consistency across HTAbs, with inclusion of between 33% (NICE) and 44% (PBAC) of these variables.

Discussion of Process variables was below 20% for all authorities except PBAC, EMA, and FDA: 33%, 63%, and 81% respectively.

Role of RWE in authority decision making from the assessment report

In the sample of submissions analyzed, RWE had a primary role in 8 (20%) regulatory and 3 (9%) HTAb assessments, and a supportive role in 19 (46%) and 20 (57%) respectively. There were few instances where the contribution of RWE was neutral/unclear, 3 (7%) and 0 (0%) respectively. RWE was discussed and not used in 9 (22%) and 12 (34%) cases, and not addressed in 2 (5%) regulatory and 0 (0%) HTAb reports (Figure S6 ).

Nine studies were assessed by at least two regulatory agencies and two HTAbs, enabling comparison across authorities (Table 1 ).

Convergence was observed in the assessment of five of these studies. For instance, submitted RWE was considered either primary or supportive by all six authorities evaluating the two RWE studies for atidarsagene autotemcel (Libmeldy), and all but G‐BA among those assessing the PNCR & NeuroNext studies for onasemnogene abeparvovec (Zolgensma). Five authorities considered the RWE study for the entrectinib (Rozlytrek) submission inadequate, and two did not address it in their reports.

Divergences were also observed. For instance, for avelumab (Bavencio), six authorities found RWE supportive, its role was unclear for HC, and it was considered inadequate for G‐BA and HAS. Similarly, while regulatory agencies and NICE considered the RWE for lutetium Lu 177 dotatate (Lutathera) supportive, HAS and CDA‐AMC found it inadequate. The greatest divergence was seen in the assessment of Study 20120148 for blinatumomab (Blincyto). EMA, NICE, CDA‐AMC, and PBAC considered it supportive; its role was unclear in the FDA's assessment, while HC, G‐BA, and TGA found it inadequate or did not comment on it.

When comparing assessments in common across authorities, some alignment was observed between EMA and FDA, with nine out of 13 RWE studies assessed by both agencies as playing similar roles in decision making. In contrast, HC showed a higher proportion of assessments where RWE did not impact decisions compared to FDA and EMA.

For HTAbs, a similar profile emerged between HAS and G‐BA: six RWE studies had comparable roles in decision making, while one study was considered supportive by HAS but not mentioned by G‐BA. Similarities were also observed among NICE, CDA‐AMC, and PBAC. In contrast, we found divergence between these groups (HAS and G‐BA vs NICE, CDA‐AMC, and PBAC).

Qualitative analyses

In this section, we illustrate the type of analyses conducted with one relevant example.

Case study: Onasemnogene abeparvovec (Zolgensma)

Onasemnogene abeparvovec (Zolgensma) first received regulatory approval from the FDA in May 2019 for treatment of patients <2 years of age with spinal muscular atrophy (SMA) with bi‐allelic mutations in the survival motor neuron 1 (SMN1) gene. While the precise indication varied slightly across authorities, this was followed by a conditional approval from EMA and approvals by HC and TGA. Reimbursement approvals from HAS, CDA‐AMC, NICE, and PBAC followed (Figure S7 ). G‐BA identified evidence gaps, and the added benefit was determined to not be proven.

RWE purpose: SAT Contextualized

Submissions included two open‐label SATs, a phase I dose‐escalating and a phase III, with two natural history studies. These were the Pediatric Neuromuscular Clinical Research Database (PNCR) and NeuroNext studies, derived from research databases and used as historical comparators. While not in focus for this case study, in some submissions a long‐term observational follow‐up of the phase I trial was also included to support the maintenance of safety and efficacy.

RWE's role: Primary or supportive across most agencies

All authorities that assessed onasemnogene abeparvovec considered the RWE studies to be supportive in providing evidence for efficacy, with FDA noting that “the comparison of Phase III trial results and the natural history data provided primary evidence.” ³⁶ The exception was G‐BA, which did not reference the studies within the decision justification.

Clinical context

There was consistency across authorities in recognizing the severity, rarity, and unmet need (italics represent variables in Table 2 ). It was broadly recognized that “the natural history of infantile‐onset SMA is well‐documented and follows a relatively predictable course that can be objectively measured and verified” ³⁶ (known disease characteristics).

Table 2.

Onasemnogene abeparvovec (Zolgensma): summary values based on the information extracted from the reviewed documentation, by authority

RWE name and role	EMA	FDA	HC	TGA	G‐BA	HAS	NICE	ICER	CDA‐AMC	PBAC
Study name	PNCR & NeuroNext	PNCR & NeuroNext	PNCR	PNCR & NeuroNext		PNCR & NeuroNext	NeuroNext		PNCR & NeuroNext	PNCR & NeuroNext
RWE's role	Supportive	Primary	Supportive	Supportive	Not referenced	Supportive	Supportive		Supportive	Supportive

Clinical context	EMA	FDA	HC	TGA	G‐BA	HAS	NICE	ICER	CDA‐AMC	PBAC
Severity of the condition
Rare disease
Orphan designation
Unmet need/public health impact
Lack of alternative treatments
Off label use
Ethical concerns RCT
Feasibility concerns RCT
Product health equity advantages
Product administration
Knowledge of previous active substance use
Known disease characteristics

Strength of evidence		EMA	FDA	HC	TGA	G‐BA	HAS	NICE	ICER	CDA‐AMC	PBAC
RWE data	Reliability
	Extensiveness
	Coherence
	Timeliness
	Relevance
RWE study design	Generalizability
	Exposure, follow‐up, covariates, endpoints
	Sample size
	Statistical methods
	Bias/comparability
	Confounding
	Sensitivity analyses
RWE effect size
Interventional trial
Mechanistic considerations
Safety

Process	EMA	FDA	HC	TGA	G‐BA	HAS	NICE	ICER	CDA‐AMC	PBAC
Predefined protocol/SAP
Authority interactions on RWE

Positive (+)

Neutral/mix (~)

Negative (‐)

Not commented

CDA‐AMC, Canada's Drug Agency; EMA, European Medicines Agency; FDA, Food and Drug Administration; G‐BA, Federal Joint Committee; HAS, Haute Autorité de Santé; HC, Health Canada; ICER, Institute for Clinical and Economic Review; NeuroNext, The Network for Excellence in Neuroscience Clinical Trials; NICE, National Institute for Health and Care Excellence; PBAC, Pharmaceutical Benefits Advisory Committee; PNCR, Paediatric Neuromuscular Clinical Research Database; RCT, randomized control trial; RWE, real‐world evidence; SAP, statistical analysis plan; TGA, Therapeutic Goods Administration.

Some variation was observed in the level of granularity in assessment reports for the clinical context variables. For instance, only FDA and PBAC commented on the feasibility concerns of an RCT, and HC and TGA did not reference the unmet need.

HTAbs commented on a wider set of elements. All noted the product administration advantages of having a single intravenous infusion; NICE noted the potential increase in health disparities as a result of the product requiring administration at specialist centers (product health equity advantages).

Strength of evidence of RWE: What did authorities say

Although the same studies were submitted, their assessment varied across authorities based on the public assessment reports reviewed.

There was a consistent lack of commentary on RWE data source variables, with only PBAC highlighting that timeliness was a consideration given the changes in clinical practice over time.

Regarding RWE study design variables, NICE raised questions around the generalizability of NeuroNext given differences in national clinical practice. In the US, where the study was conducted, the patient population comprised “a high proportion of people who have a tracheostomy unlike best supportive care in the NHS.” ³⁷

Almost all authorities commented on bias/comparability between populations in the SATs and RWE studies, covering a spectrum of conclusions:

• Populations could be compared (+): FDA reviewers noted that the populations “can be compared”, ³⁷ EMA and TGA acknowledged that although the RWD cohort showed “less severe disease as expressed by the older age”, this was “not considered a major issue since the potential bias this creates, is not in favor of Zolgensma”. ³⁸

• Populations do not appear to be strictly comparable (~): HAS acknowledged uncertainties that could create a bias in either direction. “Patients in the RWD cohort were older, suggesting less severe disease”, however a higher proportion required “nutritional and ventilatory support related to more advanced disease”. ³⁹ PBAC noted that “natural history studies may not adequately capture improvements in supportive care over time”. ⁴⁰

• Populations are not comparable (−): NICE stated that the SAT included a presymptomatic population which can develop a range of SMA types, some less severe than type 1. They concluded that “the comparison with natural history studies including only type 1 SMA is not appropriate”. ³⁶ CDA‐AMC mentioned that the RWD cohort were more severe, having “a lower CHOP INTEND score and required more feeding support and more ventilatory support” and that clinical practice had evolved considerably from the time of the RWE studies (Table S9 ). They concluded that the comparison “did not allow for unbiased estimates of treatment effect”. ⁴¹

Authorities were consistent regarding the large effect size of the results, except for HC which did not mention this aspect. For example, FDA noted that “overall the likelihood appears remote that such factors could account for all, or a substantial part of the difference observed”, ³⁷ and EMA stated that “the survival and motor milestones achieved largely exceed the natural history of SMA type 1”. ³⁸

A detailed review of the assessment can be found in Table S8 .

Quantifying regulatory and HTAbs assessments

Assigning summary values to each of the variables potentially impacting the role of RWE enabled analysis of the most to least commented variables and their type: positive (+), mixed (~), or negative (−).

Within the Clinical context area, the most frequently discussed variables were severity, rarity, and lack of alternative treatment, mentioned in at least 94% of regulatory and 85% of HTA reports. This was followed by unmet need (85% and 82% respectively) and authorities' knowledge of disease characteristics (73% and 78% respectively, Figure S8 ). The vast majority of comments on these variables confirmed the severity, rarity, and unmet need of the disease object of the submission.

HTAbs commented more frequently than regulatory agencies on health equity (0% for regulatory agencies and 37% for HTAbs) and product administration advantages (18% and 52% respectively).

In the Strength of evidence area, the most discussed variables were effect size (63% for regulatory agencies and 83% for HTAbs) and bias/comparability (61% and 69%, respectively; Figure 3 ). In contrast, variables describing RWD sources were the least acknowledged, with 37% of RWE studies lacking comments on all of the variables in this category (29% for regulatory agencies and 47% for HTAbs), compared to 5% for the study design category and 26% for effect size.

Figure 3.

Frequency of positive, neutral, and negative summary values across all submissions for RWE variables. HTAbs, health technology assessment bodies; RWE, real‐world evidence.

The type of commentary in the Strength of evidence area was primarily negative (37% for regulatory agencies and 47% for HTAbs), addressing uncertainties regarding data sources and study designs. Positive commentary was more common among regulatory agencies (33%) than HTAbs (16%). HTAbs' positive comments mainly focused on effect size, while they spread across nearly all considered variables for regulatory agencies.

When analyzing Strength of evidence variables by RWE's role (Figure 4 ), the proportion of positive comments was highest when RWE was deemed primary. This decreased as its impact on decision making diminished. In instances where RWE was discussed and not used, most comments were negative or mixed.

Figure 4.

Frequency of positive, neutral, and negative summary values across all submissions for RWE variables, stratified by the role of RWE in decision making. HTAbs, health technology assessment bodies; RWE, real‐world evidence.

A large effect size was consistently noted in submissions where RWE was considered primary, while it was never mentioned in cases where RWE was deemed neutral or inadequate for decision making. Other variables that showed some association between the type of comments from authorities (positive, mixed and negative) and RWE's role were relevance for data sources, exposure/endpoints, generalizability, sample size, and sensitivity analyses for the study design.

DISCUSSION

The main objective of this research was to empirically investigate which characteristics of RWE, submitted to support efficacy/effectiveness claims, impacted authorities' decision making. We investigated 15 products submitted to five regulatory agencies and six HTAbs, totaling 76 assessments of RWE studies. For each, we systematically identified, categorized, and analyzed 74 variables:

• 43 describing the submission and type of RWE.

• 1 summarizing RWE's role in decision making.

• 30 relating to characteristics that could influence RWE's role in decision making.

Here, we present four main findings:

Low granularity within publicly available assessment reports

The frequency with which authorities commented on the 30 variables that could influence RWE's role (Figure 2 ) suggests that the granularity of assessment reports was low on average.

This may be partly due to the extensive list of variables used. Some, such as off‐label use (or health equity considerations for regulatory agencies), may not be applicable in every assessment. In addition, reviewers may not have systematically reported on all the variables they assessed, focusing instead on those deemed more important based on the submitted evidence or suggested by internal reporting guidelines. The latter could also explain some of the variability seen across authorities.

Clinical context was the most frequently commented area, yet no authority addressed more than two‐thirds of its variables on average. A more detailed characterization of the clinical context could clarify when RWE might be appropriate. For example, ethical or feasibility concerns around RCTs were only noted in one‐third of assessment reports, leaving unclear whether RWE was considered in decision making even when traditional methods were feasible.

In the Strength of evidence area, bias/comparability and effect size were the most frequently commented variables, reflecting authority reviewers main RWE concerns. In contrast, variables related to data sources were commented on the least often. More comments on data sources would have been expected given the importance of the topic—study design and results cannot compensate for data not being fit‐for‐purpose—authorities' interest in this topic, ³² , ⁴² , ⁴³ and as most analyzed RWE studies relied on secondary data collection.

Variability in how RWE is assessed by authorities

On the role of RWE, analysis revealed cases of both convergence and divergence across authorities (Table 1 ). Notably, even when authorities agreed on RWE's role, they were not necessarily aligned on the underlying factors driving their decision. For example, in the assessment of the PNCR & NeuroNext study for onasemnogene abeparvovec (Zolgensma), authorities reached similar conclusions on RWE's role despite their differing assessments on bias/comparability. Unsurprisingly, there was significantly more variability in the Strength of evidence area than in Clinical context.

Differences in the assessment of clinical evidence are not new in authority decision making. ⁴⁴ , ⁴⁵ , ⁴⁶ This can be partly attributed to differences in remits, as evidenced by HTAbs commenting more than regulators on variables such as generalizability and health equity. Divergence can also be caused by differing perceptions of RWE's value. This might explain why HTAbs had three times more negative than positive comments regarding the RWE Strength of evidence variables, whereas regulatory agencies had a more balanced distribution. The use of different methodologies could also contribute, as seen when comparing HAS and G‐BA, which use a relative clinical benefit assessment methodology, versus NICE, CDA‐AMC, and PBAC, which use a cost‐effectiveness methodology.

The example of bias/comparability for the PNCR & NeuroNext study suggests that other factors may also play a role. These could include variability among reviewers, the submission of RWE studies that differ slightly despite sharing a name, or variations in how sponsors reported RWE.

Effect size appears to be the variable most associated with RWE's role

Comments related to Clinical context were largely positive, which confirmed the circumstances in which the use of RWE is expected: rare and highly severe diseases with unmet needs. This also suggests that Clinical context is a contributing factor to RWE's role in decision making.

In contrast, Strength of evidence variables elicited a wider range of comment types, with negative commentary being the most common (Figure 3 ). This could reflect reviewers' uncertainty regarding the RWE approach.

The positive correlation between the number of positive comments and the impact of RWE (Figure 4 ) suggests consistency between reviewers' assessments and the weight of RWE in their final decisions. However, individual studies revealed exceptions.

Effect size seemed to be the main characteristic when RWE was considered primary; in 9 of the 11 studies classified as such, a large effect size was noted. The two remaining cases were tenecteplase (Metalyse), the only example of RCT using RWD, that involved a non‐inferiority trial involving randomization where a large effect size was not needed, and alpelisib (Vijoice) EMA submission. In the latter, RWE from an expanded access program served as the main clinical evidence and, as such, had a primary role. The perceived uncertainty on the magnitude of the effect size could not attenuate the broad range of uncertainties surrounding the RWE study, and it contributed to the subsequent application withdrawal.

The importance of a large effect size, along with its frequent mention alongside bias/comparability, suggests that reviewers are aware of the potential for bias in RWE. A large effect size enhances their confidence that the observed effect is unlikely to be explained by the presumed bias.

Few submissions with advanced RWE study design

Of the 10 cohort studies submitted, only two employed a comparative design, and none of the 12 SAT contextualized used individual patient matching. Further, none of the RWE studies, as presented in the assessment reports, explicitly referenced frameworks like the target trial emulation or estimand. This finding aligns with existing literature. ⁴⁷

Possible explanations for the absence of advanced designs may include a relative lack of sponsor expertise and experience, as well as awareness of the challenges associated with such studies. There may also be a perception of limited experience in interpreting these designs by authorities. It is also important to consider the timing of the submissions included in our analysis, most of which preceded 2022, along with the fact that these studies were planned well in advance of them. Nevertheless, this result may also indicate that simplicity has inherent advantages.

Recommendations

We propose five recommendations based on these findings (Figure 5 ).

Figure 5.

Key recommendations from the FRAME research and corresponding benefits for the different stakeholders. RWE, real‐world evidence.

The first three address challenges encountered in learnings from publicly available assessment reports. First, a higher level of granularity in assessment reports can be achieved if a more structured approach to presenting assessments is established. Templates or checklists could be developed, informed by the variables used in this research and in the relevant literature. Such a structured approach can also be applied to how sponsors present the evidence in their submissions.

Second, assessment reports could also introduce a structured section that describes the submitted RWE, detailing data sources, study design, and other pertinent details. Third, an ongoing public repository to track, characterize, and capture insights on future submissions that include RWE to support efficacy/effectiveness could be established.

These measures aim to enhance transparency and facilitate shared learnings about how authorities evaluate RWE and its evidentiary value. This could benefit all stakeholders. Researchers, sponsors, and authorities could have more informed scientific discussions on RWE, helping them to identify research priorities to make progress collaboratively. Authorities would derive learnings that could be integral to further implementing their RWE frameworks. Sponsors would achieve a clearer understanding and predictability on how authorities interpret and use RWE.

The fourth recommendation, informed by the results around variability across authorities, is to strengthen collaboration on initiatives aiming at defining consistent terminology and common principles for RWE assessment. Such initiatives have been advocated in the literature ³⁰ , ³¹ , ⁴⁸ and by an ICH reflection paper on pursuing opportunities for harmonization in generating RWE. ⁴⁹ These efforts recognize the benefits of a greater consistency in how RWE is interpreted and evaluated, which will in turn help promote a better understanding and awareness of RWE, its use and role in decision making. Our findings could aid these initiatives by highlighting areas of inconsistency and in offering reflections on the underlying rationale for key divergences.

The fifth recommendation is drawn from the findings on the characteristics potentially associated with the impact of RWE. Decision‐support tools that evaluate the feasibility of using RWE to support efficacy/effectiveness claims could be developed or enhanced by our results. These tools could assist sponsors in deciding whether to invest additional resources into developing an RWE approach, considering its characteristics and those of their clinical development program.

Strengths and limitations

The research has some limitations. It was exploratory in nature and did not aim to provide a comprehensive review of all submissions using RWE to support efficacy/effectiveness claims. Instead, it focused on a detailed analysis of a selected subset of products representing a variety of scenarios.

The research relied on publicly available decision documentation, which may not fully capture all reviewers' considerations. For instance, a comparison between our analysis of onasemnogene abeparvovec (Zolgensma) with that of Bakker, ¹⁶ which also had access to non‐public EMA documents, revealed two additional study design considerations beyond the two we identified.

Another limitation relates to the data extraction and categorization process (+, ~, −). Although significant effort was made to establish a transparent and reproducible methodology, there may have been instances of quotes interpreted without full context or where categorization was ambiguous, particularly for vague or unclear quotes. This limitation also applies to the main variable of the analysis: the role of RWE. Authorities base their decisions on the totality of evidence and do not always provide clear explanations regarding how different pieces of evidence were considered or their relative weighting in their decision making.

The research has several strengths. A thoughtful approach was applied to the prioritization of products and, in particular, to the identification of the variables to be collected and analyzed, which cover multiple areas. Significant effort was made to minimize any subjectivity by creating detailed definitions for each variable and incorporating relevant examples to clarify these definitions, enhancing the reproducibility of the research. In addition to having multiple reviewers, a final harmonization round was conducted to further check for consistency.

To our knowledge, this is the only research that has identified and analyzed such a comprehensive set of characteristics across a range of products and study designs spanning numerous authorities and regions.

CONCLUSION

We took a holistic and systematic approach to identify the characteristics associated with the role of RWE to support efficacy/effectiveness claims in authorities' decision making. We hope that the key findings and corresponding recommendations will facilitate multi‐stakeholder dialogue on this topic to increase the confidence in the use of RWE. This, in turn, could contribute to the generation of better evidence, enhancing patient access to innovative treatments.

FUNDING

The research was funded by Bayer AG.

CONFLICT OF INTEREST

G. Candore, C. Martin, B. Wolf are employees at Bayer Pharma AG. P. Bolot is an employee and shares holder at Bayer AG, former employee at GSK. M. Soriano Gabarró is an employee at Bayer AG, former employee at GSK holding GSK shares. M.J. Mills has done consultancy work with Bayer AG, Merck, and MSD. J. Wasem, P.G. Kanavos and M. Sculpher are paid consultants of Bayer AG. All other authors declared no competing interests for this work.

AUTHOR CONTRIBUTIONS

G.C. C.M. M.J.M., A.S., A.L., D.C‐M., D.C., B.W., P.B., J.W., M.S.G., and M.S. wrote the manuscript; G.C., C.M., M.J.M., A.S., A.L., J.W., M.S.G., P.G.K., and M.S. designed the research; G.C., C.M., M.J.M., A.S., A.L., D.C‐M., D.C., B.W., P.B., J.W., M.S.G., P.G.K., and M.S. performed the research; G.C., M.J.M., A.S., A.L., D.C‐M., D.C., B.W., P.B., and M.S.G. analyzed the data.

Supporting information

Figure S1.

Table S1.