The role and value of real-world evidence in health technology decision-making in France, Germany, Italy, Spain, and the UK: insights on external control arms

Abstract

Real-world evidence (RWE) is increasingly used and accepted by health technology assessment (HTA) bodies as supportive evidence to inform the approval of new technologies. However, the criteria driving RWE acceptance are often unclear.

This study aims to improve understanding of the role and value of RWE in HTA decision-making and outline the best practices in building real-world external control arms (ECAs).

A mixed approach of a targeted literature review and HTA expert interviews was applied. The HTA reports of ten selected technologies and the expert interviews from France, Germany, Italy, Spain, and the UK informed the criteria driving the acceptance of RWE. Overall, the UK and Spanish HTA bodies are more receptive to accepting RWE, whereas the French and German are the least accepting. When RWE is used to substantiate efficacy claims, the level of scrutiny from regulators and HTA bodies is considerably higher than when RWE has different intended uses. Representativeness of the data source, overall transparency in the study and robust methodologies are the key criteria driving RWE acceptance across markets.

Introduction

The increasing digitalization of healthcare data offers novel opportunities to leverage real-world data (RWD) to inform the evaluation of health technologies and advance our understanding of diseases and care pathways. The European Medicines Agency (EMA), and the Food and Drug Administration (FDA) encourage the use of RWD to generate real-world evidence (RWE) to support decision-making (1–4). Overall, the submission of RWE to regulators and health technology assessment (HTA) bodies is increasing (5, 6).

Simultaneously, technological advances like genomic sequencing and its growing integration into healthcare systems contribute to better disease characterization, thereby increasing the diagnosis rates of rare and ultra-rare diseases (7–13). These advancements enable the development of targeted and advanced therapy medicinal products (ATMPs), often aimed at addressing high unmet needs in rare diseases. Orphan and targeted therapies, including ATMPs, often face challenges in adhering to conventional randomized controlled trial (RCT) methodologies, with one key challenge being the recruitment of eligible patients (14). Hence, implementing robust traditional RCTs suitable for regulators and HTAs may be unfeasible in this context (15). As a result, the number of new technologies using single-arm trials (SATs), instead of RCTs, has increased over the last decade (16). In this context, leveraging RWE to reduce the uncertainty around SATs, such as through external control arms (ECAs), has become increasingly relevant. Still, in the context of assessing treatment effectiveness, RWE generated from RWD can also be leveraged to support label expansions (17). Additionally, RWE can inform other aspects of the HTA process such as characterizing the size of the target population.

Despite the increasing integration of RWE into HTA, the acceptance and evaluation criteria for RWE vary between countries and often remain unclear. To date, most research examining the acceptability of RWE focuses on regulatory decision-making bodies, such as the FDA or EMA, rather than European HTA bodies (5, 6, 1 8–20). Although HTA bodies have recently published RWE guidance documents and position papers, the appraisal criteria remain vague and fail to address all potential uses of RWD (21–26). Hence, we conducted this study to inform the value of RWE for European HTA, and the criteria driving the acceptance and design of robust ECAs to support HTA decision-making.

Methods

This review implemented several approaches to unravel the value of RWE for European HTA decision-making focusing on five agencies: the Agencia Espanola de medicamentos y productos sanitarios (AEMPS) in Spain, the Gemeinsamer Bundersausschus (G-BA) in Germany, the Haute Autorité de Santé (HAS) in France, and the National Institute for Health and Care Excellence (NICE) in the United Kingdom (UK), and Agenzia Italiana del Farmaco (AIFA) from Italy. The information gathered to inform the acceptance criteria and the value of RWE for decision-making was extracted from three main sources: HTA documents containing RWE, HTA expert interviews, and RWE guidance documents published by the HTAs included in this study.

This study adhered to the definition of RWD provided by the EMA and the International Society for Pharmacoeconomics and Outcomes Research (ISPOR), as health data collected outside the context of RCTs and RWE as any meaningful and robust analysis of RWD (6, 2 7).

Literature Review

A targeted literature review was conducted to identify examples of technologies that used RWE in their regulatory evidence submissions. The search was limited to English, from 2015 to 2022 in the MEDLINE database using the following search string: (Real-world [ti] OR RWD [ti] OR RWE [ti] OR observational study [ti] OR External Control Arm [ti]) AND (Regulatory approval [ti] OR Regulatory Agencies [ti] OR European Medicine Agency [ti] OR EMA [ti] OR FDA [ti] OR food and drug administration [ti]). Only technologies that received marketing authorization from the EMA were included. The objective of the literature review was to identify examples of technologies using RWE in their submission package extracting insights regarding the criteria driving the acceptability and value of RWE. Systematically capturing all the technologies approved using RWE was out of scope, since after reviewing more than 10 technology reports, the insights extracted were considered redundant. Ten technologies spanning 12 indications were included to represent a wide range of RWE use cases and therapeutic areas (6, 1 8–20,27).

The HTA reports were retrieved and reviewed to assess the inclusion, acceptability, and potential impact of RWE on the final reimbursement decision. The reimbursement status and consequent availability of the technologies in their respective markets were also extracted and complemented with the NAVLIN database. For Italy, only the technology’s reimbursement status was retrieved from the Italian Gazzetta Ufficiale, as no HTA reports were available.

Differences exist in the criteria used by HTA bodies in the EU4 and UK to appraise technologies. The AEMPS and AIFA largely focus on budgetary impact analyses, whereas HAS and the G-BA prioritize clinical relevance over economic considerations. In contrast, the UK’s NICE places significant emphasis on cost-effectiveness analyses and incorporates quality-adjusted life years (QALYs) and incremental cost-effectiveness ratios (ICERs) to guide reimbursement decisions within the National Health Service (NHS).

Reimbursement status was categorized into (1) fully reimbursed when the reimbursed population matches the population indicated in the EMA label, (2) conditionally reimbursed, when post-marketing commitments to submit additional evidence are imposed, (3) restricted reimbursement, when the reimbursed population is narrower than that indicated in the EMA label and (4) not reimbursed when the technology is not approved by HTA bodies for funding.

The RWE acceptance status in each technology’s HTA assessment was determined based on references or appraisal of RWE and categorized into five groups: (1) RWE accepted to support efficacy, (2) RWE accepted as “disease background” (e.g., the burden of disease or natural history of the disease) (3) RWE “neutral connotation,” when the RWE was identified, however, no further evaluation of the evidence was found, (4) RWE is not included, and (5) RWE is not accepted when RWE was identified and appraised in the HTA report and a negative connotation(s) regarding the acceptability of the RWE was identified or inferred.

A set of best practices to support the design and implementation of an ECA was developed based on the RWE appraisal extracted from the HTA reports, complemented by the expert interviews and guidance documents reviewed.

Semi-structured expert interviews

Six online semi-structured 90-minute interviews with six country-specific HTA experts were conducted (two for the UK, and one expert each from France, Germany, Italy, and Spain) between September 2022 and December 2022. The experts recruited were selected for having at least 10 years’ experience in the HTA process of their respective country of expertise, such as being involved in dossier submissions and/or holding positions in the HTA body.

An interview guide (Supplementary Information) was developed that probed the perceptions, experiences, perceived obstacles to and factors driving the uptake, and preparedness to integrate RWE into HTA decision-making. Interviews were audio recorded and then transcribed. Information from both the literature review and supplemental interviews was reviewed by three researchers.

Results

Characterization of the selected technologies and RWE purpose

A total of 95 studies were identified in the literature review. After reviewing 17 technologies, ten technologies spanning 12 indications that met the inclusion criteria were selected to represent a wide range of RWE use cases across different therapeutic areas and to provide a comprehensive overview of the criteria and uses of RWE by HTAs. The selected technologies were divided into two groups: those using RWE to support efficacy claims (e.g., through ECAs or indication expansion) and those using RWE to inform the natural history of the disease (e.g., to characterize disease background or epidemiology) (Table 1). All the technologies included were targeted rare diseases and received orphan drug designation (ODD) by the EMA, except for atezolizumab in small cell lung cancer. The technologies using RWE to support efficacy claims included six ECAs, and two registry studies demonstrating the technology’s efficacy in subpopulations not included in the pivotal trials (Eculizumab in Atypical Hemolytic Uremic Syndrome [aHUS] for a pediatric line expansion, and in paroxysmal nocturnal hemoglobinuria [PNH] in patients without prior blood transfusions) (Table 1).

Table 1.

Selected technologies, corresponding disease areas, and RWE descriptions

INN	Name	Year of EMA approval	Disease	Type of RWD	Description of RWD
RWE submitted to support efficacy claim
Avelumab	Bavencio	2017	First-line Merkel Cell Carcinoma (MCC)	External Control Arm	An ECA compared avelumab with data from a retrospective observational study 100070-Obs001 designed to evaluate outcomes under current clinical practices in MCC, including 1 L and 2 L patients from the US and Europe (57–59) (R)
Avelumab	Bavencio	2017	Second-line Merkel Cell Carcinoma (MCC)
Blinatumomab	Blincyto	2018	Acute Lymphoblastic Leukemia (ALL)		An ECA comparing Blinatumomab with continued chemotherapy using data from a retrospective study (study20120148)
Tisagenlecleucel	Kymriah	2018	Relapsed or Refractory Diffuse Large B-cell Lymphoma (R/R DLBCL)		An ECA comparing Tisagenlecleucel to the historical standard of care using different data sources (SCHOLAR–1, ZUMA–1, CORAL, Eyre, and PIX301) (60)
Axicabtagene Ciloleucel	Yescarta	2018			An ECA comparing Axicabtagene ciloleucel to salvage therapy using historical data from two sources (SCHOLAR–1 and JULIET) (61)
Onasemnogene Abeparvovec	Zolgensma	2020	Spinal Muscular Atrophy (SMA)		An ECA comparing Onasemnogene abeparvovec with natural history studies to estimate outcomes for best supportive care: PNCR, NeuroNEX, ENDEAR (versus Spinraza) and a study by De Sactis et al (2016) (62)
Eculizumab	Soliris	2007	Paroxysmal Nocturnal Hemoglobinuria (PNH)	Efficacy and Natural History of the Disease	An observational non-interventional registry (M07–001) was used to evaluate the efficacy of Eculizumab in PNH patients with no history of blood transfusion, which was not included in the clinical trial (63, 64)
Eculizumab	Soliris	2011	Atypical hemolytic uremic syndrome (aHUS)	Pediatric line expansion	A retrospective observational study (C09–001; N = 30) was used to support the line extension to pediatric patients (n = 19) (65)
RWE submitted to inform the disease background
Lutetium (177Lu) Oxodotreotide	Lutathera	2017	Gastroenteropancreatic Neuroendocrine Tumors (GEP NETs)	Compassionate Use Program	A compassionate use study from a single center in the Netherlands called ERASMUS, with data collected on 1,214 patients. (63, 64, 76)
Velmanase Alfa	Lamzede (2018)	2018	Alpha-mannosidosis	Natural History of the Disease	Data and studies from Orphanet were used for the disease epidemiology and natural history of the disease (67)
Atezolizumab	Tecentriq	2019	Extensive Stage Small Cell Lung Cancer (ES-SCLC)	RWD input for Economic Modelling	Data from Flatiron, ESCAP 2011, and KBP IMPower were used for the extrapolation of the overall survival of the control group (68)
Afamelanotide	Scenesse	2014	Erythropoietic Protoporphyria	Post Authorization Safety Study (PASS)	Following the EMA’s regulatory approval, the company was required to conduct a PASS, where QoL data was collected and later used in HTA submission (69)

Abbreviations: INN: International Nonproprietary Name; aHUS: atypical hemolytic uremic syndrome; ALL: acute lymphoblastic leukemia; EC: European Commission; ECA: external control arm; EMA: European Medicines Agency; ES-SCLC: Extensive-stage small-cell lung cancer; GEP NET: gastroenteropancreatic neuroendocrine tumor; HTA: health technology assessment; MCC: Merkel cell carcinoma; N: sample size; ODD: Orphan Drug Designation; PNH: paroxysmal nocturnal hemoglobinuria; QoL: quality of life; R/R DLBCL: relapsed or refractory diffuse large B-cell lymphoma; RWD: real-world data; RWE: real-world evidence; and SMA: spinal muscular.

Acceptance status of RWE and ECA by each HTA body

The overall acceptance of RWE and reimbursement status across selected technologies from 12 indications are captured for each HTA body, illustrated in Figure 1. Additionally, the specific acceptance status of the submitted RWE and the reimbursement status for each technology per HTA are outlined in Table 2. Despite all technologies being approved by the EMA, not all secured reimbursement from the assessed HTAs; some were only reimbursed for subpopulations of the labeled population. Only onasemnogene abeparvovec and eculizumab (for aHUS) received reimbursement in all markets within the study scope. Overall, the NICE (UK) and AEMPS (Spain) had the highest acceptance rate, accepting RWE from nine and seven of 12 indications, respectively. In contrast, the HAS (France) and the G-BA (Germany) accepted RWE from six and two of the 12 indications, respectively (Table 2). Several technologies (e.g., avelumab, blinatumomab, tisagenlecleucel, axicabtagene ciloleucel, and velmanase alfa) had their RWE rejected by some HTA bodies; however, they still received reimbursement in the respective countries.

Figure 1.

Overview of the RWE acceptance status and reimbursement decision among the selected 12 technologies.

Table 2.

RWE Acceptance and reimbursement status of each technology per HTA

Technology	AEMPS (Spain)		AIFA (Italy)***	G-BA (Germany)		HAS (France)		NICE (UK)
	Reimbursement	RWE	Reimbursement	Reimbursement	RWE	Reimbursement	RWE	Reimbursement	RWE
RWE for Efficacy (External Control Arm)
Avelumab (Bavencio) in 1 L MCC	Fully reimbursed	RWE partially accepted	Fully reimbursed	Fully reimbursed	RWE not accepted	Not reimbursed	RWE not accepted	Fully reimbursed	RWE accepted
Avelumab (Bavencio) in 2 L in MCC	Fully reimbursed	RWE partially accepted	Fully reimbursed	Fully reimbursed	RWE not accepted	Conditional Reimbursement	RWE not accepted, but requested post-marketing RWE	Fully reimbursed	RWE accepted
Blinatumomab (Blincyto)	Not reimbursed	RWE not accepted	Fully reimbursed	Fully reimbursed	RWE not accepted	Conditional Reimbursement	RWE not accepted	Restricted reimbursement	RWE accepted
Tisagenlecleucel (Kymriah)	Fully reimbursed	RWE accepted to	Fully reimbursed	Fully reimbursed	RWE not accepted	Conditional Reimbursement	RWE not accepted, but requested post-marketing RWE	Fully reimbursed	RWE accepted
Axicabtagene ciloleucel (Yescarta)	Fully reimbursed	RWE accepted to	Fully reimbursed	Fully reimbursed	RWE accepted	Conditional Reimbursement	RWE not accepted, but requested post-marketing RWE	Fully reimbursed	RWE accepted
Onasemnogene abeparvovec (Zolgensma)	Restricted reimbursement	RWE accepted	Fully reimbursed	Fully reimbursed	RWE accepted to support efficacy	Conditional reimbursement	RWE accepted and requested post-marketing RWE	Restricted reimbursement	RWE accepted
Other types of RWE
Eculizumab (Soliris) in aHUS	Fully reimbursed	RWE accepted to support efficacy	Fully reimbursed	Fully reimbursed	Report not available	Conditional reimbursement	RWE accepted to support efficacy	Restricted reimbursement	RWE accepted to support efficacy
Eculizumab (Soliris) in PNH	Fully reimbursed	RWE accepted to support efficacy	Fully reimbursed	Fully reimbursed	Report not available	Restricted reimbursement	RWE not included	Reimbursed, report not available	Report not available
Lutetium Lu 177 dotatate (Lutathera)	Fully reimbursed	RWE neutral	Fully reimbursed	Fully reimbursed	Report not available	Conditional reimbursement	RWE accepted to support efficacy and safety	Fully reimbursed	RWE accepted for disease background
Velmanase Alfa (Lamzede)	Not reimbursed	RWE accepted as disease background	Fully reimbursed	Fully reimbursed	RWE not accepted	Conditional reimbursement	RWE accepted for disease background, requested post-marketing RWE	Restricted reimbursement	RWE not accepted
Atezolizumab (Tecentriq)	Fully reimbursed	RWE not included	Fully reimbursed	Fully reimbursed	RWE not included	Fully reimbursed	RWE accepted for disease background	Restricted reimbursement	RWE accepted for disease background
Afamelanotide (Scenesse)	Not reimbursed	RWE neutral	Not reimbursed	Fully reimbursed	RWE neutral	Fully reimbursed	RWE accepted for disease background, requested post-marketing RWE	Not reimbursed	RWE neutral

Abbreviations: 1 L: first-line; 2 L: second-line; AEMPS: Agencia Española de Medicamentos y Productos Sanitarios; aHUS: atypical hemolytic uremic syndrome; AIFA: Agenzia Italiana del Farmaco; ECA: external control arm; G-BA: Gemeinsame Bundesausschuss; HAS: Haute Autorité de Santé; HTA: health technology assessment; NICE: National Institute for Health and Care Excellence; PNH: paroxysmal nocturnal hemoglobinuria; and RWE: real-world evidence.

*No distinction in the Gazzetta Ufficiale between first-line and second-line Avelumab.

For Lutetium Lu 177 dotatate no assessment was made by the G-BA (Germany) as Lutetium Lu 177 dotatate is expected to impose a negligible cost on the healthcare system.

^***No information could be found regarding reimbursement restrictions or post-marketing commitments for AIFA.

Best practices for ECA design and implementation

In this study, six technologies using an SAT with a supportive ECA were included. Critical analyses of the technology’s HTA reports, expert interviews, and guidance documents published by HTAs led to the development of best practices for designing and implementing an ECA to supportan SAT. Eight key categories were identified for consideration in the design of an SAT + ECA. For each recommendation, an excerpt from HTA reports supporting the recommendation is presented (Table 3).

Table 3.

Best practices for the design of an SAT with an RW-ECA with illustrative quotes from the selected HTA reports

Category	ECA Best Practices	Source (example)
SAT + ECA feasibility	Demonstrate the need to run a SAT + ECA (e.g., unfeasible or unethical to perform an RCT due to rare disease) Engage with regulatory agencies to validate the SAT + ECA study design, protocol, and selected RWD sources Pressure test SAT + ECA strategy with payer/HTA experts to assess acceptability and anticipate Market Access implications Leverage individual country HTA early advice channels to pressure test SAT + ECA design directly with payers	“The absence of direct comparison with a clinically relevant comparator must be justified by the company and may be accepted by the TC in certain situations, such as concomitant developments, specific populations for whom extrapolation of efficacy can be performed on the basis of pharmacokinetic data or real-life data” (70)
ECA Study Design	Pre-specify the ECA design, data source, clinically meaningful thresholds, and QoL hierarchy in the protocol to reduce bias in the analysis Pre-specify the strategy to deal with missing data Consider including multiple control groups, to increase the likelihood of matching and reduce bias Identify and list confounders in a pre-specified manner with the help of KOLs, preferably before RWD identification Collect and analyze safety data from the ECA Consider pre-specified threshold-crossing approaches as the large magnitude of treatment effects is less likely to be the result of bias or confounding alone (71)	“A systematic and exhaustive identification of a priori confounders and prognostic factors should have been performed before the indirect comparison” (Avelumab [Bavencio], MCC, HAS) (72) “No safety data was collected in the control population nor accompanying therapies, precluding an assessment of the benefit-harm profile”(Blinatumomab [Blincyto], ALL, G-BA) (74) “The quality of the studies [ECA] was not assessed in detail, in particular, the completeness (percentage of missing data) and how the missing data was managed” (Avelumab [Bavencio], MCC, HAS) (72)
RWD source Selection	Demonstrate that the selected RWD source is the most adequate to address the research need (e.g., through literature review, consider sample size, access, and data availability) Select RWD sources with access to patient-level data, as they allow for more advanced analysis and adjustment of confounders Validate data extraction by more than one person	“The committee regretted that the data extraction was performed by only one person per center, introducing potential bias due to lack of validation of the extraction process” (Blinatumomab [Blincyto], ALL, G-BA) (73) The committee appreciated that “The ECA started with a systematic literature review of the available observational data” (Onasemnogene abeparvovec [Zolgensma], SMA, HAS) (74)
Representativeness and Comparability	Ensure inclusion and exclusion criteria are as close as possible between the intervention group and ECA (e.g., disease severity, baseline characteristics, prior treatment lines), and if using historical controls, consider if treatment pathways are still relevant Ensure that endpoints’ definitions and follow-up time are similar between SAT and ECA	“Patients in the historical cohort do not have the same characteristics as those in phase II, especially the patients that already received HSCT” (Blinatumomab [Blincyto], ALL, HAS) (75) “The differences in the populations studied between tisagenlecleucel [Kymriah], and the possible pharmacological alternatives make indirect comparison difficult, so it is not possible to establish their comparative efficacy.” (Tisagenlecleucel [Kymriah], DLBCL, AEMPS) (76)
Endpoint selection	Prioritize objective endpoints which can be tracked reliably with RWD Agree on the validity of the selected endpoints based on the RWD source with KOLs and health authority/HTA Confirm the validity of the selected endpoints with independent blinded reviewers Define primary and secondary outcomes as similar as possible between SAT and ECA	“The definition of one of the variables (independent sitting position) was different between the two studies STRIVE US and START” (Onasemnogene abeparvovec [Zolgensma], SMA, HAS) (74) “In addition, the extent of the observed effects is not of such a magnitude that it can be ruled out that the effects are solely due to systematic bias or Incidental findings” (Avelumab [Bavencio], 1 L MCC, G-BA) (77)
Bias Reduction	Demonstrate the quality of the RWD source, e.g., using REQUEST tool (78) Limit differences in observational time to avoid immortal time bias, if present implement time-adjusted analysis Minimize systemic and incidental bias by employing appropriate statistical methods Stratify data by centers/investigators that participated or not to detect any bias or difference in the data collected	“The entire observation period is viewed critically, as this differs significantly between the studies, and information on the actual observation period is missing both in the studies justifying approval and in the external controls. Observation time-adjusted effect estimates that take these differences into account in the analysis would be adequate for this” (79)
Statistical Analysis	Leverage appropriate statistical methodologies e.g., propensity-score matching, ensuring comparability of the intervention and control arms based on key covariates (e.g., disease severity, age, previous treatments, time since diagnosis, and clinical pathway) Implement sensitivity analysis, where various assumptions and scenarios are explored Implement quality assurance of the analysis and code used	“At clarification, the ERG requested that the company performs a propensity score weighting analysis to compare avelumab with chemotherapy” (Avelumab [Bavencio], MCC, NICE) (80)
Transparency	Present the criteria used to select centers/RWD sources for ECA Register the study protocol before implementing the study (clinicaltrial.gov or RWE transparency network) Publish the analytical code used Present study results using developed reporting tools (e.g., START-RWE)	“The selection criteria used to select the centers for the historical control was not shown” (Blinatumomab [Blincyto], ALL, G-BA) (73)

Abbreviations: AEMPS: Agencia Española de Medicamentos y Productos Sanitarios; ALL: acute lymphoblastic leukemia; DLBCL: diffuse large B-cell lymphoma; ECA: external control arm; ERG: Evidence Review Group; G-BA: Gemeinsame Bundesausschuss; HAS: Haute Autorité de Santé; HSCT: hematopoietic stem cell transplantation; HTA: health technology assessment; KOL: key opinion leader; MCC: Merkel cell carcinoma; NICE: National Institute for Health and Care Excellence; QoL: quality of life; RCT: randomized controlled trial; RW: real-world; RWD: real-world data; RWE: real-world evidence; SAT: single-arm trial; SMA: spinal muscular atrophy; and TC: Transparency Commission.

Country profile: use and acceptance of RWD and RWE

Finally, country profiles based on three main themes – common usages of RWE, RWE supporting efficacy, and early advice on the RWE plan – are summarized in Table 4, illustrating how RWE is employed, its goals, level of acceptance, and value across each country. The data were primarily gathered from the expert interviews, complemented by HTA reports and HTA-specific guidance, which supported the interview findings (Table 4).

Table 4.

Use and acceptability of RWD and RWE in the EU4 + UK across a range of use cases

	AEMPS (Spain)	AIFA (Italy)	G-BA (Germany)	HAS (France)	NICE (UK)
Common usage of RWE
Size of the target population	Common practice and relevant	Common practice and relevant	Common practice and relevant	Common practice and relevant	Common practice and relevant
Safety	Common practice and relevant	Common practice and relevant	Common practice and relevant	Common practice and relevant	Common practice and relevant
Identification of the SoC	Information on SoC is provided by clinical experts from pre-appointed Evaluation Nodes during the technology assessment	Common practice, performed by the AIFA through their administrative records and expert consultation	Common practice and relevant for the identification of the appropriate comparator therapy (ACT)	Relatively common, may be used coupled with other sources (e.g., clinical guidelines)	Common practice
Description of the natural history of the disease	Relatively common practice but most of the time not considered relevant, as this perspective is provided by the clinical experts taking part in the assessment	Relatively common practice may be considered relevant in rare diseases where the natural history of the disease is not well-defined	Uncommon, may be done to highlight the unmet need or burden of disease and in some cases to define a different ACT	Relatively common practice, and relevant, especially for rare diseases	Relatively common practice and relevant, especially for rare diseases
Description of the treatment pathway	Not considered relevant, as this perspective is provided by the clinical experts participating in the appraisal. In some cases, the patient pathway is considered rather than the treatment pathway	Common practice performed by the AIFA	Common practice and relevant	Common practice, to define where the new technology would be integrated into the current therapeutic strategy	Common practice and increasingly relevant. One of NICE strategic ambitions for 2021–2026 is to implement “dynamic, living guideline recommendations” that extend across the whole care pathway and stakeholders, to drive the rapid adoption of best practices and latest innovations through RWD
Description of the patient’s burden	It is becoming more common and relevant	Currently not formally accepted but there are ongoing discussions on how this could be included in the assessment	Uncommon, yet PROs collected in clinical trials are considered relevant for the technology appraisal	Uncommon and not considered relevant unless for rare diseases (PRO from clinical trials are accepted under certain conditions)	Uncommon, certain qualitative studies may be considered relevant in the case of rare and ultra-rare diseases
Description of the societal burden (e.g., caregivers, productivity loss)	Increasingly common	Although there is no formal process yet, many companies are including this in the pricing and reimbursement dossier	Uncommon and not considered relevant unless in for very rare diseases with high unmet need	Uncommon and not considered relevant	Relevant for rare diseases with high unmet needs. Limited environmental impact with sustainable practices is valued
RWD to inform Economic Modeling	RWD to evaluate the economic burden may be relevant when demonstrating that the technology has a positive impact on the healthcare system to support pricing and reimbursement decisions	Economic evaluation is not performed in Italy	Economic evaluation is not performed by the G-BA. Price negotiation considers the final benefit assessment granted by the G-BA	Technologies that aim to obtain an ASMR I-III or expect sales of >20 M EUR and < 50 M EUR must submit a cost-effectiveness model, when sales are expected to exceed 50 M EUR a budget impact model is also required. RWD can be used to provide costs and utilities. RWD can also be used when economic models are provided post-marketing, namely, to demonstrate the impact of integrating the new technology in the healthcare system	Cost-effectiveness models are mostly populated with clinical trial data, however RWD can and is used to inform costs and outcomes of SoC/comparator. Models enriched with RWD are more likely to receive a positive review by the ERG than models that lack RWD (81)
Implementation of payer agreements	Increasingly common, outcome-based agreements are performed at the regional level, with data collected through VALTERMED, especially for costlier technologies	Common practice, Italy frequently establishes payer agreements, setting up a specific monitoring registry. AIFA uses the RWD collected to monitor outcomes and utilization (access to the collected data is restricted to AIFA)	Uncommon. A law was introduced in 2019 that allows for RWD collection to inform performance agreements. Tisagenlecleucel and Axicabtagene ciloleucel were subject to outcome-based agreements in the 12-month free-pricing period	Performance-based agreements are avoided by CEPS; however, price- or volume-based agreements relying on RWE (e.g., PMSI) are common	Relatively common, however, it is seen as administratively burdensome and complex to implement
RWD collected in the context of Early Access programs	RWD can be collected in early access programs to inform decision-making but it is not mandatory	Relatively common practice, depending on the type of program in place the data may or may not be accessible to the manufacturer	In exceptional cases when Early Access programs are implemented, RWD collection is expected (82) (safety data collection is mandatory)	A common practice for technologies that meet the following criteria: hints of high efficacy and favorable safety profile, innovation, rare disease or high burden, high unmet need, absence of effective treatment, and no clinical trial in France). A recent reform of the early access program requires manufacturers to collect RWD on efficacy, safety and QoL. The RWD collected from EAP is used for technology appraisal	Relatively common, RWD collection is mandatory
Post-marketing technology re-appraisal	Relatively common, this is performed using the VALTERMED platform or independent RWD submitted by the manufacturer	Common practice, Italy frequently establishes payer agreements and uses RWD to monitor outcomes and utilization (access to the collected data is exclusively reserved for the AIFA)	The new law was introduced in 2019 allowing the G-BA to collect RWD for orphan drugs and technologies with conditional approval. The first example is Onasemnogene abeparvovec	The common practice in France and the HAS-published guidance for RWD studies is more focused on technology re-appraisal than first assessment	Common practice, through the Cancer Drug Fund, although the expert indicated that re-assessment most often relies on mature trial data rather than on RWD
RWE Supporting Efficacy Claims
ECA acceptability	Only in exceptional cases, where an RCT is considered unethical, rare/ ultra-rare diseases, or when two technologies are developed within the same time period	The AIFA does not have a defined position on ECAs, and the EMA assessment is followed	Low, G-BA experts estimated that out of the 100 historical control arms submitted to the G-BA, only 10 were accepted	ECA is believed to be legitimate when direct comparisons are not feasible, otherwise, they should not be used	ECA is considered biased, only a few exceptional cases warrant the use of ECAs
Acceptability of RWE as the main source of efficacy data	Low – RWE is considered a biased source of evidence for informing efficacy and is only accepted in exceptional cases	Low – RWE is seen as a source of supportive evidence and not as a main source of efficacy data.	Low/Very-low – Seen as a biased and confounded source of evidence. Inadequate for robust quantification of benefit.	Low – Seen as a biased and confounded source of evidence but for specific cases, it may be the only source of data available in which case it will be considered. RWD may be considered a complementary source of data to the RCT	Low – it is a source of evidence with low internal validity
Early advice on RWE plan
Possibility to review the evidence generation plan prior to submission	Yes, through AEMPS which will be re-evaluated at the regional level	Early scientific advice is possible before the submission of the P&R dossier	Early advice on the full evidence generation strategy, including RWD is possible to obtain, however, it may be detrimental if the advice is not followed	Possible for first submissions if the technology fulfills three criteria: the disease is severe, the last phase of development has not begun, and there is a high unmet need. For the re-assessment stage, the protocol must be reviewed by HAS and discussed with the RWD team if the study is requested by HAS	Under the new NICE RWE Framework, the “NICE Scientific Advice” is strongly encouraged to validate RWE strategies planned to be part of the overall evidence-generation plan
Availability of national RWD sources	VALTERMED is the national registry of the NHS but is only available to the health ministry. Other smaller publicly available registries exist	AIFA manages all mandatory post-reimbursement registries. Although these registries are sponsored by the manufacturer, the data is not accessible to manufacturers. Registries covering different diseases (not run by AIFA) are available.	Data from the German sick funds are expensive and complex to access. German disease registries are available and considered a suitable RWD source. However, patient-level data is rarely accessible to privates. Technology-specific registries can rarely be used outside the technology setting as they are not considered representative	Manufacturers must go through certified companies to access the national claims database: SNDS Disease registries exist and access must be determined directly with registry holders	NICE is committed to enabling access to data when necessary to support technology appraisal, through the NHS digital Large EMR representative data sources exist e.g., CPRD, SACT

Abbreviations: ACT: appropriate comparator therapy; AEMPS: Agencia Española de Medicamentos y Productos Sanitarios; ASMR: Amélioration du Service Médical Rendu; CEPS: Comité Economique des Produits de Santé; CPRD: Clinical Practice Research Datalink; ECA: external control arm; ERG: Evidence Review Group; EMR: Electronic Medical Records; G-BA: Gemeinsame Bundesausschuss; HAS: Haute Autorité de Santé; HTA: health technology assessment; NHS: National Health Service; NICE: National Institute for Health and Care Excellence; PRO: Patient-reported Outcomes; P&R: Pricing and Reimbursement; QoL: quality of life; RCT: randomized controlled trial; RW: real-world; RWD: real-world data; RWE: real-world evidence; SACT: Systemic Anti-Cancer Therapy; SAT: single-arm trial; SNDS: Système National Des Données De Santé; SoC: Standard of Care; and TC: Transparency Commission.

Representativeness of the RWD source was highlighted as a key criterion, driving the evaluation and acceptance of RWD. Most HTA experts who were interviewed mentioned a preference for national data, followed by Europe, the United States (US), and the rest of the world. However, experts agreed that while representativeness is crucial for informing economic models, identifying the standard of care (SoC), and estimating the size of the target population, the similarity between the trial and the external patient characteristics is more relevant for assessing comparative effectiveness. RWD used to characterize the natural history of the disease, humanistic and clinical burden, and the unmet need is also acceptable from non-national sources if clinical and treatment practices are similar to those of the country in question.

Transparency in RWD collection and analysis was the second most-cited criterion driving acceptability and trust in RWD. The NICE (UK) RWE Framework and the HAS (France) RWD study guidance both recommend publishing the protocol before conducting the study to raise confidence in RWD (22, 2 3). The recommended platforms for the publication of RWD studies include the EMA RWD Catalogues (28), the RWE Registry, endorsed by the ISPOR RWE Transparency Initiative, and the Clinical Trial.gov registry (29–33).

Regarding the common uses of RWE, these include informing the natural history of the disease, characterizing the epidemiology, the clinical unmet need, identifying the national SoC, and informing the safety of newly introduced technologies. The identification of the appropriate SoC is particularly relevant in Germany, as the G-BA evaluates new technologies against the pre-defined appropriate comparative therapy (ACT). Robust RWE can be used to challenge the G-BA-endorsed ACT, which can have a strong impact on the final benefit assessment and price of the technology being evaluated. Similarly in France, the new technology is evaluated in the context of a selected appropriate comparator. In Italy and in the UK, RWE can also help to identify the SoC; whereas, at the national level in Spain, the SoC is often defined by specialized physicians consulted by the AEMPS (Spain) (Table 4).

RWE extends beyond initial technology assessments. In the UK, Italy, and Francecollecting RWD during the post-marketing phase is crucial for re-evaluating technologies. This practice is well-established, as evidenced by initiatives such as the Cancer Drug Fund (CDF) in the UK, post-marketing authorization studies in France, and the AIFA registries in Italy. Notably, in France, the use of RWD in technology re-evaluation is a primary focus outlined in the HAS (France) RWD Guidelines (“RWD studies for the evaluation of drugs and medical devices, 2021”) (23) and is already well-established with 11 post-marketing RWD studies requested in 2022 concerning 10 technologies (34). The collection of RWD for re-evaluation of technologies is critical for maintaining reimbursement, confirming the added benefit of technologies conditionally approved due to uncertainties in areas of high unmet need, and assessing the impact of integrating the technology into the healthcare system (23) (Table 4).

The practice of requesting RWE for re-evaluation has spread to Germany and Spain (36, 3 7). In Germany, a new law was introduced in 2019 which allows the G-BA to request RWD collection following the initial assessment of technologies with conditional approval or approval in an orphan disease (36). Historically, the G-BA and IQWiG have been critical of including RWE in decision-making; however, the law states that any additional RWE mandated by the G-BA must be used in the reassessments by both the G-BA and IQWiG. In 2021, onasemnogene abeparvovec became the first technology to undergo this process. The G-BA mandated the collection of RWD to monitor the real-world effectiveness and safety of onasemnogene abeparvovec. Novartis, the manufacturer, is using the RESTORE registry, which will include nearly 500 children with SMA from the US and Europe, to collect this data over 15 years (37, 38).

In Spain, VALTERMED was introduced by the Spanish National Health System in 2019 as a national registry designed to devaluate the therapeutic value of high-impact medicines in real clinical practice. The initiative aims to enhance transparency and optimize healthcare decisions by collecting and analyzing RWD on the effectiveness of medicines with significant clinical and economic impacts, as well as supporting pay-for-performance agreements (49). As of December 2022, VALTERMED has included 22 drugs, most of which are orphan drugs (63 percent) and ATMPs (40). Due to privacy concerns, data from the VALTERMED registry is only available to the Spanish Minister of Health (MoH) Table 4).

In Italy, the AIFA oversees 283 therapeutic indication-based registries. Among these, 182 registries focus solely on drug utilization, while 35 registries include additional financial-based agreements, and 60 registries utilize performance-based risk-sharing agreements (41, 4 4). Although supporting the achievement of economic sustainability the registries represent a significant operational burden that can delay access to treatments, according to the expert interviewed (Table 4).

The NICE (UK) RWE Framework highlights the ambition to create “dynamic, living clinical guideline recommendations,” (22) where the agency aims to regularly update its guidance documents with the latest RWE. An example of the use of RWE to update clinical guidelines can be found in the uptake of injectable long-acting antipsychotics, where multiple RWE studies showed that these treatments result in better adherence and therefore, are more effective compared with their oral comparators (42–44) (Table 4).

Discussion

This study provides evidence that the utilization and scrutiny of RWE vary by its specific application and among different HTA bodies. Additionally, the use of RWE in HTA decision-making is increasing, as evidenced by Germany’s new law enabling the G-BA to request RWE and the establishment of VALTERMED in Spain.

Overall, the NICE (UK) and AEMPS (Spain) had the highest acceptance rate of RWE, accepting RWE from nine and seven of 12 indications, respectively. In contrast, the HAS (France) and the G-BA (Germany) accepted RWE from six and two of the 12 indications, respectively. The non-acceptability of RWE was not correlated with the reimbursement decision since several technologies were reimbursed despite the rejection of RWE. In Germany, new pharmaceuticals are reimbursed by default, however, the level of reimbursement and price are dictated by the level of added benefit versus the ACT.

The use of RWE to characterize the natural history of the disease, epidemiology, and identification of SoC is widely accepted. However, some uses of RWE are more established than others, such as supporting economic models in the UK and France, or post-marketing studies to support payer agreements in Italy. Our findings indicate that the quality of the data, relevance, methodological robustness, and disease context (e.g., rare disease, high unmet need) are key determinants of its acceptance, which is consistent with previous findings (45, 4 9). Additionally, this study demonstrates that transparency is key to establishing trust in the methodological validity of an RWE study, as highlighted by the latest NICE RWE framework (22).

The utilization of RWE in combination with clinical trials to demonstrate treatment effects results in increased scrutiny. Our research highlights that HTA bodies tend to be skeptical of the need for and validity of SATs with RWE-based ECAs, as this study design is prone to bias and confounding. Traditional RCTs remain the gold standard for demonstrating the safety and efficacy of new technologies. Consequently, the rationale supporting the non-feasibility of a comparative trial is a key driver for the acceptability of such a study design. While regulators and HTAs may hold different perspectives on what constitutes a valid rationale for accepting an SAT, which is assessed on a case-by-case scenario, there is a consensus that under special conditions (e.g., rare and ultra-rare diseases where there may be violations of equipoise), traditional RCTs may be infeasible or unethical, and SATs with ECAs may provide the best available evidence. This is also recognized by the EMA when stating that RCTs may “need to be complemented by other methodologies to address research questions where a traditional RCT may be unfeasible or unethical.” (15) Similarly, the AEMPS (Spain) accepts that certain situations are not amenable to RCTs, such as in the case of onasemnogene abeparvovec in SMA, where “the rapidly progressive and fatal nature of the forms studied makes the use of placebo unfeasible.” (46) Additionally, Traditional RCTs have limitations, including stringent inclusion and exclusion criteria that enhance internal validity but may result in findings that are not representative of the broader population, as the included subjects tend to be less severe and more homogenous in terms of comorbidities, ethnicity, and socio-economic background. Hybrid trial designs, such as pragmatic trials that combine RCT characteristics with real-world conditions, are gaining attention; however, their acceptance by regulators and HTAs remains debatable (47–51).

While the criteria cited by HTAs for rejecting evidence derived from ECAs are similar, the acceptance of the same evidence package varies widely across the evaluated countries. High-quality RWD allowing for robust statistical analyses are very important for HTA bodies when appraising RW-ECA. The impact of the quality of the RWD used in the ECA may be illustrated in the case of axicabtagene ciloleucel and tisagenlecleucel, where both technologies used the same data source for their ECA SCHOLAR-1 (an international retrospective research study pooling data from two randomized trials and two academic databases) (52), but only axicabtagene ciloleucel’s manufacturer had access to the individual patient data. While the ECA data from axicabtagene ciloleucel was accepted by the AEMPS (Spain), G-BA (Germany), and NICE (UK), tisagenlecleucel’s ECA was only accepted by the AEMPS and NICE. The increased level of robustness of axicabtagene ciloleucel’s ECA, given the access to patient-level data, may have contributed to a more favorable appraisal such as the acceptance of the ECA data by the G-BA, and achievement of ASMR III in France (indicating a moderate improvement in the medical benefit), while tisagenlecleucel’s ECA was not accepted by the G-BA, and tisagenlecleucel received an ASMR IV in France, (indicating a minor improvement).

The risk of bias was cited by HTAs as one of the main criteria for rejecting data from ECA; however, neither the sponsors nor the evaluating committee has attempted to evaluate or quantify the presence and impact of bias, highlighting the importance of the systematically using quantitative bias analysis methods to enable a more objective assessment of bias in RWE studies (53, 5 8).

Overall, this study demonstrates differences in the degree of RWE acceptance among the included HTAs. This represents a challenge that the future European Joint Clinical Assessment (JCA) will bring (54). For a harmonized evaluation of RWE, particularly RW-ECA, member states and their respective HTA bodies should align on the requirements and acceptability of such data across various use cases.

Study limitations

One limitation of this study is the limited number of technologies covered, as well as the representation of each HTA body by a single expert (except for the UK where two experts were interviewed), which may limit the generalizability and completeness of the findings. However, selecting a restricted number of technologies allowed a more granular analysis of the HTA reports that may be generalizable to similar RWD types and disease contexts (e.g., rare diseases).

Another limitation is that it is challenging to pinpoint the specific impact of the submitted RWE for the final HTA reimbursement decision. Velmanase alfa in Spain represents an example where the submitted RWE was accepted and used to inform the epidemiology of the disease; however, the technology was not reimbursed by the AEMPS (Spain) due to the lack of statistically significant changes in the trial’s functional outcomes, which is unrelated to the quality of the RWE (55). Several technologies (e.g., avelumab, blinatumomab, tisagenlecleucel, axicabtagene ciloleucel, and velmanase alfa) had their RWE rejected by the HTA bodies; however, they still received reimbursement (most cases, conditional reimbursement with mandatory post-marketing studies) despite the rejection of the RWE. Highlighting the multitude of factors that influence the final HTA decision, such as cost-effectiveness, the disease’s unmet need, disease prevalence, patient engagement, and political pressures, which are challenging to capture and evaluate. In some cases, interpreting the acceptability of ECA data from HTA reports is challenging, as seen in the AEMPS report for avelumab in first- and second-line MCC. While it is stated that “the comparison of avelumab with the chemotherapy presents large methodological problems,” the comparative data from the observational studies are presented at length and compared with the results of the SAT, which itself received criticism. In the first-line setting, the conclusion was that the efficacy results, although preliminary, present a “duration of response not achievable with chemotherapy,” which was derived from the ECA. While in second-line Merkel Cell Carcinoma (MCC), the lack of a comparator arm “prevents us from being certain about the magnitude and relevance of the benefit.” Thus, we considered that the ECA data were accepted in the first line (where the conclusion refers to challenges associated with the clinical trial and not the ECA) but not in second line (where the lack of an appropriate comparator is specifically mentioned in the conclusion) (56).

Conclusion

Based on the RWE guidelines reviewed, HTA reports, and expert interviews, this study highlights that RWE is becoming an integral component of evidence submitted to HTA bodies alongside clinical trials. RWE is used for various purposes, including estimating disease epidemiology, characterizing standard of care, assessing the safety of new technologies post-marketing, supporting economic models, and payer agreements in some regions. It is particularly crucial in the re-evaluation of technologies following conditional approvals. When RWE is used to support efficacy claims, HTA bodies apply stringent scrutiny. Methodological rigor, population comparability between clinical trials and external controls, along with transparency, are key factors influencing RWE acceptance. As the number of registered SATs grows annually, developing robust RWE strategies aligned with regulatory and HTA requirements becomes increasingly essential. Enhanced collaboration and transparency among stakeholders are needed to create integrated evidence-generation plans that address the needs of patients, healthcare professionals, regulators, and HTA bodies.

Supporting information

Vidalis et al. supplementary material

Vidalis et al. supplementary material

Supplementary material

The supplementary material for this article can be found at http://doi.org/10.1017/S0266462324004720.

Author contributions

Catia C Proenca designed the study, wrote the manuscript, performed the research, and analyzed the data. Oscar Dumoulin contributed to the writing of the manuscript, performed the research, and analyzed the data. Alexa Vidalis contributed to the writing of the manuscript, performed the research, and analyzed the data. Mrunmayee Godbole contributed to writing the manuscript and the qualitative analyses.

Funding

This project was funded by Alira Health.

Competing interest

Oscar Dumoulin, Alexa Vidalis, Catia C Proenca, and Mrunmayee Godbole were all employed by Alira Health while contributing to this project. Alira Health is a healthcare consulting company providing services to pharmaceutical and MedTech companies. The work conducted in this project was fully funded by Alira Health and was not associated with any external organization.