The Use of Real-World Data for Estimating Relative Treatment Effects in NICE Health Technology Assessment Submissions: A Review

Abstract

This paper investigates the current use of real-world data (RWD) for estimating relative treatment effects in National Institute for Health and Care Excellence (NICE) health technology assessment (HTA) submissions. This review included 64 HTA submissions, which accounted for approximately 11% of the total NICE HTA submissions between January 2016 and December 2023. The main sources of RWD considered in the submissions were disease registries and electronic health records. RWD were primarily used to provide an external control arm to enable comparisons within single-arm trials and to inform long-term treatment effects when extrapolating survival data beyond the trial follow-up. Adjustments for potential systematic differences between treatment groups have improved over time; however, approximately one-third of the submissions still relied on unadjusted treatment comparisons. We found that approximately 10% of NICE HTA submissions used RWD to directly inform treatment effects estimation. Over one-third of the submissions relied on naïve and/or unadjusted treatment comparisons, which may have introduced biases. To ensure that RWD provide sound evidence for HTA, submissions should follow published guidelines, including the NICE real-world evidence (RWE) framework.

Supplementary Information

The online version contains supplementary material available at 10.1007/s40273-024-01449-w.

Key Points for Decision Makers

There has been a gradual increase in the use of RWD for estimating comparative effectiveness in UK NICE HTAs, particularly to enable comparisons between single-arm trials and RWD-derived external control arms.

The study found that over one-third of NICE HTA submissions still relied on unadjusted treatment comparisons, which may have introduced biases and potentially misleading inferences.

Future submissions should consider principled study design frameworks, such as the target trial emulation, as these can bring greater transparency and confidence to the use of RWD for estimating treatment effects for HTA purposes.

Introduction

Randomised controlled trials (RCTs) play a crucial role in informing reimbursement decisions since they are considered the gold standard for establishing the relative effectiveness and cost effectiveness of health technologies. In many instances, evidence from RCTs may fall short of the needs of HTAs. For example, the disease may be rare, leading to difficulties in recruiting sufficient participants to detect statistically significant differences between arms. In addition, RCTs may be unable to capture longer-term treatment effects or include all relevant comparators and outcomes required for HTA purposes. In such cases, RWD, hereafter defined as data collected routinely in electronic health records (EHRs), disease registries and administrative datasets or prospectively in observational studies, can be used to complement RCT evidence to estimate treatment effects.

International HTA agencies are committed to leveraging RWD for reimbursement purposes and are setting standards for its use in HTA. For example, the UK NICE 5-year strategy envisages a more central role for RWD to better inform treatment recommendations and enable more dynamic clinical guidelines [1]. In line with this strategy, NICE published a RWE framework in 2022 to help incorporate RWD into HTA, including guidance on study design and statistical analysis of RWD to improve the quality and transparency of non-randomised evidence in HTA submissions [2]. Other agencies around the world, such as the French Haute Autorité de Santé, the German Institut für Qualität und Wirtschaftlichkeit im Gesundheitswesen (IQWiG), the US Food and Drug Administration (FDA), and the Canadian Agency for Drugs and Technologies in Health (CADTH), also published practical and methodological guidance on the requirements for RWD collection, data quality and data analysis to support the conduct of real-world studies for HTA purposes [3–6].

Recent review studies have synthesised circumstances where RWD have been used in HTA. For example, studies [7–9] reviewed the use of RWD to address evidence gaps when populating decision-analytical models, such as non-RCT evidence related to health-related quality of life, resource use, treatment frequency, and disease prevalence. Others [10, 11] reviewed the extent to which non-comparative studies are used in HTA appraisals across different HTA agencies, including NICE, CADTH, and IQWiG. They found that the use of non-comparative studies in HTA was limited and mainly considered in rare disease settings or when it was impractical to randomise patients to meaningful alternative treatments (comparators). In addition, they highlighted the challenges in developing a robust evidence package, which requires a reliable RWD source, a clinically relevant comparator, and an appropriate analytical approach for treatment comparison.

Although these studies provide a general overview of how RWD have been used in HTA across different HTA bodies, the extent to which RWD are used to directly inform the estimation of treatment effects remains unclear. This matters because evidence on the relative effectiveness of health technologies tends to be central to the decision as to whether the health technology is recommended for use, and because decision makers are acutely aware of the various biases and pitfalls of RWD-based comparative effectiveness studies, not least because most RWD are not collected for research purposes.

This paper examines how RWD have been used to directly derive treatment effects in NICE technology appraisal submissions. Typically, as part of the NICE HTA process, the manufacturer submits a dossier to NICE with all the available evidence (including RWD) on how the new treatment compares to the existing one(s). This dossier is critically appraised by an external assessment group. The NICE HTA committee then consider both the company’s submission and the external assessment group report (known as committee papers) to decide whether the new technology should be recommended for use in the UK national health service. This study reviews the NICE committee final report and the committee papers for a deeper understanding of RWD use in HTA. We pay particular attention to the study design and analytical strategies used for deriving treatment effects, including a critique of the limitations of current RWD use.

Methods

Inclusion Criteria

We reviewed HTA submissions that were published between 1 January 2016 and 31 December 2023 and are available on the NICE website (https://www.nice.org.uk). The search period was chosen to cover recent developments in NICE decision making, such as the more prominent role of RWD in managed access agreements [12], and to complement previous systematic reviews in this area. Complete single and multiple technology appraisals of pharmaceutical drugs and medical devices using RWD across all disease areas were included. Re-appraisals and updates to initial recommendations were also included, so the same health technology may be reviewed more than once.

We excluded submissions for which (1) RWD did not inform the estimation of treatment effects; (2) RWD were only used as supporting evidence via citation of an existing published study, but not utilised in the HTA submission; (3) RWD were included as part of a systematic review; and (4) RWD were only considered by NICE’s evidence review group.

Study Selection

HTA submissions were initially screened to identify relevant studies using the following RWD-related search terms: database, registry, non-randomised, non-RCT, observational, real-world, single-arm, and retrospective. For submissions included after the initial screening, the full text of both the final submission reports and the committee papers were reviewed to assess whether the HTA submission met the eligibility criteria.

The initial screening and full-text review were conducted by the first reviewer. A second reviewer scrutinised a random sample of 20% of the HTA submissions at both stages and validated eligibility for inclusion. Any disagreements between the first and second reviewers were resolved by consulting a third reviewer.

Synthesis

The main unit of analysis was the HTA submission, in which one or more RWD sources may be used. We first presented descriptive statistics of the HTA submission trend over time, RWD sources, disease areas, and NICE recommendations. We then conducted a narrative synthesis of the types of RWD used for estimating treatment effects, the approaches adopted for study design and statistical analysis, estimand, and the extent to which RWD have supported NICE HTA decisions.

Results

Included Submissions

Full details of the identification of relevant NICE submissions are presented in Fig. 1. In total, 569 HTA submissions were published between January 2016 and December 2023, including 549 single and 20 multiple technology appraisals across all disease areas. We excluded 99 terminated submissions and eight withdrawn submissions because the submission did not meet the NICE evidence requirements. We excluded an extra eight submissions that we did not have access to, which were updated and replaced by the new submission or guidance.

Fig. 1.

PRISMA diagram of the included UK NICE HTA submissions

Of the remaining 454 HTA submissions, 195 were excluded after initial screening because they did not include the relevant keywords. Of the 259 HTA submissions assessed for eligibility, 195 were excluded after full-text review. Among the excluded, 70 (36%) used RWD for other purposes (not informing treatment effects), for example, informing input parameters in the decision-analytical model, such as costs, utilities, and transition probabilities, or the natural history of the disease; 59 (30%) submissions were excluded because RWD were considered only by citing existing published studies or real-world clinical practice (59 of 194 [30%]); 58 (30%) submissions included one of the keywords but no RWD were actually considered in the submission (i.e. a single-arm trial was compared with another single-arm trial). The exclusion reasons are summarised in Table S1 in the Supplementary Material.

After a full-text review, we included 64 HTA submissions (58 single and six multiple technology appraisals) that used RWD to directly derive treatment effects.

Characteristics of the Included Submissions

Figure 2 shows the change in the use of RWD in NICE HTA submissions over the review period. Except for 2016, the percentage of appraisals in which RWD were used to inform treatment effects per total submissions varied between 10 and 27%. There was an upward trend in the use of RWD over time, except for 2020.

Fig. 2.

RWD use for relative treatment effects estimation in UK NICE HTA submissions from 2016 to 2023

Sources of RWD included disease registries, EHRs, and medical chart reviews. Most studies were international and multicentre. The main countries collecting RWD through registries were the UK, USA, France, and Germany. US data included a wider range of databases at the regional and/or national level, whereas RWD in other countries tended to come from national registries (e.g. UK Systemic Anti-Cancer Therapy) or single-institution EHRs (e.g., UK Clinical Practice Research Datalink).

A total of 53 submissions received positive recommendations, although approximately 75% of these were made with conditions such as managed access agreements, commercial arrangements, or 2-year stopping rule. Eleven appraisals were recommended for use in the UK Cancer Drugs Fund.

The most common diseases covered in the included submissions were lymphoma, non-small-cell lung cancer, carcinoma outside of the lung, and leukaemia (Fig. S1 in the Supplementary Material). Non-cancer areas included kidney disease, idiopathic pulmonary fibrosis, asthma, mastocytosis, lupus erythematosus, myelofibrosis, cystic fibrosis, hypertrophic cardiomyopathy, and COVID-19.

RWD Use and Sources

External Control Arm

All 64 submissions included in this review considered RWD to construct an external control arm (ECA), which was then compared with a treatment arm in a single-arm trial or RCT. In the latter case, an ECA was required when the control arm of an RCT was not considered adequate because (1) the RCT’s control arm was outside the NICE scope or not reflective of UK clinical practice, (2) the comparator was undefined or inappropriate for UK clinical practice (e.g., country-specific treatment regimens and different lines of therapies), or (3) the randomised sample was small, which resulted in low statistical power due to a lack of an established comparator (e.g. orphan disease with no approved treatments).

The identification of a relevant real-world ECA (e.g., current clinical management, the standard of care) was generally informed by a systematic literature review and clinical expert opinion. However, the suitability and quality of RWD sources were not formally assessed using assessment tools or reporting checklists such as the Data Suitability Assessment Tool [2].

Non-UK real-world ECAs were accepted in 36 submissions (56.3% of 64 submissions). The main justifications for using non-UK ECAs were small sample size and survival endpoint availability. Accepted non-UK real-world ECAs did not necessarily have better data quality than UK real-world ECA; it was the most complete dataset available at the time of analysis. The limitations were acknowledged, such as non-UK setting and assumptions on patient characteristics with RWD; however, they were not treated differently from UK ECAs. Two-thirds (44 submissions) of the total submissions considered RWD for ECA analysis only in the base-case analysis. In nine submissions, RWD were considered in both the base-case and sensitivity analysis, primarily to assess the robustness of the base-case assumptions, for example with respect to the statistical approach. Eleven submissions considered RWD only in sensitivity/scenario analysis. In some instances, RWD provided evidence to complement RCT evidence in respect of disease types (rare tumours) or subgroup effects (e.g. disease severity, tumour expression). In other cases, RWD were considered in sensitivity analysis to assess the generalisability of the findings, for example, by using a jurisdiction-specific disease registry to assess the generalisability of the RCT evidence from elsewhere.

Table 1 summarises the number of RWD sources used in ECA analysis according to its application in base-case versus sensitivity analysis. A total of 108 individual RWD sources, including disease registries, EHRs, and chart reviews, were considered in the 64 included submissions. Although EHRs and disease registries are equally considered in the base-case analysis, the latter are more likely to be considered in the sensitivity analysis than the former.

Table 1.

The number of RWD sources used in external control arm analyses across base-case versus sensitivity/scenario analysis according to the type of RWD. Numbers in brackets are submissions

Data source	Base-case only	Both base-case and sensitivity/scenario analysis	Sensitivity/scenario analysis only
EHRs	37	7	5
Disease registries	36	11	9
Chart reviews	15	6	2
Total	88 (44)	24 (9)	16 (11)

Extrapolation

In total, 12 submissions considered RWD (disease registries in all cases) to directly inform the long-term treatment effects.

In 10 submissions, RWD were used to calibrate the choice of the parametric curve. For example, the survival rates observed in disease registries could determine whether the parametric curves are underestimating or overestimating the survival of the control group. The other two submissions (TA396 and TA562) [13, 14] used RWD to adjust the long-term projections of the case-mixed model to inform treatment effects across different patient subgroups.

Analytical Strategies for Deriving Treatment Effects

External Control Arm Studies

Figure 3 describes the statistical methods used to adjust for differences between treatment and real-world ECAs, focusing on the primary base-case analysis. Over one-third (14 of 44) of submissions performed a naïve comparison (no confounding adjustments) between treatment and the real-world comparator. The proportion of naïve comparisons varied over time but seems to have somewhat decreased in the last few years (Table S2 in the Supplementary Material). When confounding adjustments were performed (30 of 44), weighting was the preferred adjustment method (20 of 30), followed by matching, regression, and simulated treatment comparison (STC). Matching was particularly preferred over weighting or regression approaches when the RWD sample was large and/or model specification was judged more challenging. Potential confounders or effect modifiers were identified by systematic literature review and/or through discussions with clinical experts. Selected measured confounders were defined a priori, but additional confounding factors were sometimes considered in the sensitivity analysis, particularly when there was a lack of overlap in key prognostic factors between the single-arm trial and ECA groups.

Fig. 3.

Statistical methods used to adjust for differences between treatment and external control arm in base-case analysis only (44 submissions)

The estimand of interest was reported in 52 submissions; 39 submissions reported an intention-to-treat analysis (average treatment effect). Appraisals often stated that this estimand was preferred by HTA decision makers because it tends to reflect the effect of the treatment policy in real-world practice (recognising that patients may discontinue or switch to alternative treatments). In nine submissions, the average treatment effect on the treated was adopted instead of the average treatment effect. For example, this was done when there was more than one comparator group and the baseline characteristics differed across the comparator group. Per-protocol effects (treatment effect with strict protocol adherence) were reported in four submissions. In eight submissions, more than one estimand was reported.

Extrapolation

Extrapolation of survival data was done by fitting alternative parametric survival curves to each trial arm and the real-world ECA independently. The most popular parametric distributions were Weibull, exponential, Gompertz, log-logistic, log-normal, and generalised gamma distributions. The choice of parametric survival curve (and hence long-term survival projection) was informed by the goodness-of-fit measures (Akaike information criterion/Bayes information criterion) and clinical expertise. In 20 submissions, the real-world ECA was deemed inappropriate for extrapolation for various reasons, such as (1) short follow-up, (2) inclusion of treatments not observed in UK clinical practice, and (3) key endpoints (e.g., progression-free survival) not collected. In particular, the Systemic Anti-Cancer Therapy registry was deemed too immature to be extrapolated despite UK patient representativeness. In such cases, alternative RWD or trials were chosen to extrapolate survival in the control group.

Discussion

Main Findings

Our review found that the use of RWD in NICE technology appraisals has been gradually increasing to complement randomised evidence for estimating comparative effectiveness. None of the submissions included in this review derived treatment effects based exclusively on RWD. The most frequently used sources of RWD were disease registries and EHRs. RWD were used mainly to construct an ECA, enabling comparisons with single-arm trials, and to inform long-term treatment effects by calibrating survival curve extrapolation.

NICE recommendations highlighted the added uncertainty related to the potential risk of biases in non-randomised RWD studies. While inverse probability weighting has been often considered to adjust for any imbalances (confounding) between treatment and control groups, about one-third of the comparative assessments were still based on unadjusted analyses. The lack of transparency in submissions that used RWD was also highlighted. It was particularly concerning that the development and registration of the study protocol, including design and analysis plans, were absent from the reviewed submissions.

The generalisability of non-UK RWD was examined in 24 submissions. In 17 submissions, non-UK RWD were judged to match the UK population and treatments provided in routine clinical practice. In the other seven submissions, the non-UK RWD reported differences in treatment pathways, healthcare contexts, and definitions of care interventions (e.g. standard of care, best supportive care). However, this lack of generalisability had relatively little impact on the final decision.

Contributions

Previous studies reviewed the use of RWD to populate various parameters, such as utilities and costs, of the decision model [7–9]. Our article adds to this literature by critically reviewing the use of RWD specifically for estimating relative treatment effects. Given the increasing interest in the use of RWD to complement RCT evidence for establishing treatment effects in the HTA setting, there have been concerted efforts to develop guidance on study design and analysis of RWD [2–6, 15–17]. This article provides a snapshot of the extent to which these developments have permeated NICE HTA practice.

A recent study reviewed the use of supplementary external control data in HTAs based on single-arm trials [11]. The study found that about 60% of the comparisons conducted alongside single-arm trials relied on unadjusted comparison approaches. Our results suggest a somewhat more optimistic picture than that reported by Patel et al. [11]. We found that about one-third of ECA studies relied on naïve comparisons. One of the reasons driving these differences is that Patel et al. reviewed HTA submissions included in the IQVIA HTA Accelerator, which covers a wide range of HTA agencies. It may be that NICE pushes more strongly for the reporting of adjusted comparisons than other HTA agencies. Alternatively, given that Patel et al. reviewed HTA submissions from before 2019, the lower number of unadjusted comparisons may simply reflect a greater uptake of statistical approaches recommended in NICE RWD methods guides published in the last few years [2, 18]. A related review by Appiah et al. [19] examined the justification for using ECA sources in NICE and Pharmaceutical Benefits Advisory Committee oncology HTA submissions. As in our review, they found that the selection and justification (e.g. quality and generalisability) of RWD sources for HTA submissions were underreported.

Our study complements these two reviews in several ways. First, our study is not restricted to the use of RWD in single-arm trials. We found 27 submissions that combined RWD with RCTs, for example, when the comparator arm was inappropriate in the UK context. Second, we assessed the use of RWD to derive long-term treatment effects in the extrapolation of survival outcomes. Data from cancer registries were particularly useful to calibrate the choice of the parametric curve when it clearly underestimates or overestimates the survival in the control group. Third, we reviewed both the NICE final technology appraisals and the supporting documents (committee papers). This allowed us to characterise the use of RWD in greater detail. Fourth, our review provides a more recent snapshot of the use of RWD in HTA. This matters because this is an area of rapid development, and methodological guidelines adoption requires time to permeate practice.

Limitations

There are some limitations to our review. First, our review only covered the period since 2016. This was partly motivated by a previous review by Griffiths et al. [10], who explored the role and the type of non-randomised evidence in HTA submissions up to 2016. Second, our review only focused on NICE technology appraisals, and the results may not be generalisable to other HTA agencies outside of the UK given the variations in HTA practice and jurisdictions. For example, a previous review [10] suggested that non-randomised studies were more likely to be used for complementing RCT evidence by NICE than by other HTA agencies such as CADTH and IQWiG. In particular, the latter places a strong emphasis on establishing treatment effectiveness directly from RCTs. Third, this review is based on publicly available information, such as NICE final recommendation reports and committee papers. More detailed information in the manufacturer submission, or communications between the manufacturer and NICE, would have helped provide a more accurate picture of the use of RWD in HTA, including the study design and analytical approaches.

Improving Future RWD Evidence Submission

This review identified some areas in HTA practice that require further improvement to strengthen future RWD evidence generation. First, although there seems to be an improvement in the adjustment for observed differences in patients’ characteristics between treatment groups, a significant number of submissions still relied on naïve comparisons. HTA agencies should continue to encourage future submissions to report population-adjusted treatment comparisons, such as matching and weighting [18] over non-adjusted methods. Second, unmeasured confounding remains the primary concern in non-randomised evidence, and it may result in biased treatment effect estimates, but this has received insufficient attention in HTA submissions. For example, quantitative bias analysis [20] has been widely recommended to explore the potential impact of unmeasured confounding, but this has not permeated HTA practice. Third, the current reporting of RWD studies lacks a detailed description of the study design and analytical assumptions, making it difficult to assess their appropriateness. RWD study design frameworks, such as the target trial emulation [16], can bring more clarity and confidence to the use of RWD in HTA, providing a common template for study protocols to provide greater clarity and transparency about the design and analysis of RWD. HTA agencies should require these protocols to be registered ahead of conducting the actual RWD-based studies. For example, the RWE Transparency Initiative is working with the US FDA to develop a platform where these protocols can be uploaded and accessed only by some users (e.g. HTA agencies) to protect proprietary information [21, 22]. Fourth, the lack of careful alignment between the design of the single-arm trial and RWD-based ECA has been highlighted in several submissions. These discrepancies were related to eligibility criteria or treatment regimens. Manufacturers can avoid these by taking early steps in the design of the single-arm trial (e.g. more pragmatic inclusion criteria and treatment strategies) that will facilitate the comparison with RWD. For example, before the submission, the company may consider the ‘NICE Advice Service’ to seek scientific advice on the quality of the planned RWD sources and on how these can be incorporated robustly into the submission. This review found that only one submission (TA855) [23] explicitly followed the NICE RWE framework. This submission has taken a more careful approach to (1) study design, by considering data assessment tools and clearly specifying important components of the study design, such as the definition of baseline and treatment strategies, and (2) statistical analysis, by carefully justifying the analytical approaches used and conducting several sensitivity analyses to address uncertainties associated with RWD. The review did not find that the methodological quality of the HTA submissions improved after the publication of the RWE framework and these guidelines.

Concluding Remarks

The use of RWD for comparative effect estimation purposes in HTAs has been gradually increasing in the last few years. NICE recognises the more prominent role that RWD can play in complementing RCT evidence, particularly in the context of single-arm trials, but current practice has ample scope for improvement. To ensure that RWD provide sound evidence for HTA, submissions should follow best practices on the study design and analysis of RWD, which have been published widely, including the NICE RWE framework.

Supplementary Information

Below is the link to the electronic supplementary material.

Declarations

Funding

MG is recipient of a NIHR Advanced Fellowship (NIHR302259).

Conflict of interest

All authors declare that they have no conflict of interest.

Availability of data and materials

Not applicable.

Ethics approval

Not applicable.

Author contributions

YC and MG conceived and designed the study. YC performed the review and wrote the first draft of the manuscript. MG reviewed and verified the screening process. SD critically reviewed the manuscript for important intellectual content. All authors reviewed and approved the submitted manuscript.