Use of real-world data and real-world evidence in NICE (UK) health technology appraisals of new therapeutics in oncology: a systematic review

Abstract

Objectives

To quantify and describe the use of real-world data (RWD) in National Institute for Health and Care Excellence (NICE) oncology technology appraisal (TA) final appraisal determination documents.

Design

A systematic literature review was conducted on pharmaceutical NICE oncology TAs published between April 2000 and March 2024 (covering financial years 2000/2001 to 2023/2024 inclusive) extracted on 22 August 2023 (2000/2001 - 2022/2023) and 8 August 2024 (2023/2024).

Data sources

NICE TA final appraisal determination documents.

Eligibility criteria

All pharmaceutical oncology TAs published between April 2000 and March 2024 (financial years 2000/2001 to 2023/2024) that did not go on to be terminated.

Data extraction and synthesis

The data required for eligibility screening was extracted from an Excel file directly from the NICE website, where data related to each TA was extracted using an automated script derived from published sources. TAs were assessed based on prespecified review criteria covering whether an RWD submission was reported by the committee, and if so, which RWD sources were used, alongside the methods reported and any feedback from the committee regarding the use of RWD. Bias was not assessed as part of the study.

Results

Of 310 TAs identified, 135 (48.0%) used RWD. A variety of RWD types were used, mostly from UK or US data sources. 47 TAs (34.8%) leveraged RWD from multiple sources. RWD was mostly used in comparisons of survival (41.5%), to inform utility values (26.7%) and to compare baseline characteristics (19.3%), with matched adjusted indirect comparisons (MAICs) and external control arms (ECAs), seen from 2015 and 2018, respectively. The committee expressed concerns around the RWD presented by the company in 53 TAs (39.2%), the most common being a lack of generalisability to the UK population and/or National Health Service practice and comprehensiveness of the RWD.

Conclusions

This study quantifies the increasing use of diverse RWD sources in NICE oncology TAs, as well as the shift towards more complex methods like MAICs and ECAs. The feedback of the NICE committee highlights key areas of improvement as the generalisability and maturity of the RWD presented.

STRENGTHS AND LIMITATIONS OF THIS STUDY.

• A systematic approach was undertaken ensuring no key National Institute for Health and Care Excellence (NICE) oncology therapeutic appraisal documentation was missed.

• Technology appraisals (TAs) between the 2000/2001 and 2023/2024 financial years were investigated, making it one of the largest reviews of real-world data (RWD) in TAs.

• Our investigation was only based on the final appraisal document, hence if RWD was used in other documentation provided by the company it is not acknowledged in this review, potentially under-representing the overall use of RWD in the complete TA process.

• TAs without documentation (N=58) were excluded creating a potential source of selection bias; however, documentation is usually only removed when a TA has been reassessed or removed from licensing in the National Health Service.

• Our investigation was restricted to NICE oncology TAs; hence, it may be the case that these results are not generalisable to other regulatory bodies.

Introduction

The technology appraisal (TA) process is used by the National Institute for Health and Care Excellence (NICE) to recommend new and existing treatments within the UK National Health Service (NHS).¹ Recommendations for routine provisioning in the NHS are based on both the quality and strength of the clinical evidence available and whether the technology meets predefined standards of cost-effectiveness.

Historically, real-world data (RWD) and real-world evidence (RWE) have been predominantly used to monitor and evaluate the post-market safety of approved drugs (phase IV) in regulatory settings.² However, in 2022, NICE published its Real-World Evidence Framework,³ providing guidance on the best practices around developing RWE to support NICE reimbursement decisions. This followed a 2019 review, conducted by NICE, of its Health Technology Evaluation methods which recognised that justified uses of RWD in appropriate circumstances are valuable in contributing to a comprehensive evidence base. Potential RWD sources for submission to NICE TAs were identified as electronic health records, administrative data, claims data, and patient registries among others, with each having particular strengths and suitability towards particular methods.⁴ Similar guidance has been published by regulatory bodies around the world, like the US Food and Drug Administration⁵ and Medicines and Healthcare products Regulatory Agency,⁴ marking a shift in the role of RWE in addressing key evidence gaps and highlighting a growing acceptance of RWE in the preapproval setting.

The value of RWD is apparent when ethical and/or practical considerations prohibit the execution of a traditional randomised control study. In these settings, RWD provides an opportunity to assess the value of a new treatment despite the known challenges of observational studies.⁶ RWE can contribute to the body of evidence for a new treatment in a variety of ways, from crude comparisons of survival rates from published studies and electronic health records to more complex methodologies, like matched adjusted indirect comparisons^{7 8} (MAICs) or external control arms⁹ (ECAs). MAICs provide a framework for addressing differences between trial populations when comparing results from a trial in which individual patient trial data is available to a trial or other data source where only aggregate data is available.¹ External control arms have gained increasing interest as a way to assess treatment efficacy for single arm trials using patient level electronic health record RWD control.¹⁰

Previous reviews have summarised and compared the growing use of RWD across multiple regulatory and reimbursement bodies¹¹; assessed the suitability of RWD sources used for ECAs¹²; reviewed the contributions of RWD to either clinical effectiveness¹³ or cost-effectiveness^{1 14 15} evidence in NICE TAs or discussed specific case studies of particular methods (ECAs).¹⁶

At present, no studies have investigated how RWD has been used for both clinical evidence and cost-effectiveness analyses, described the statistical methods being used in the studies and documented any direct feedback from the committee around the use of RWD. This review aims to address this gap, with a specific focus on NICE oncology TAs. Greater demand for increased efficiencies in oncology drug development, and subsequently in clinical trials,¹⁷ has led to an expansion of the use of RWD in oncology beyond just the post-approval setting.¹⁸ The use of RWD in oncology TAs has been discussed via case studies¹⁹; assessed in the context of health economic modelling only^{14 15} and has been compared across TA bodies.^{20 21} However, this review aims to better understand the extent to which RWD is being referred to in NICE oncology TAs committee appraisal documents; how many, and which, datasets are being used; and assess the quality of the methods being used.

Methods

Information source and search strategy

All oncology TAs with a decision published between April 2000 and March 2024 (financial years 2000/2001 to 2023/2024) were identified from the NICE website.²² An Excel document of all oncology TAs from 2000/2001 to 2022/2023 was extracted directly from the NICE website on 22 August 2023 and those from 2023/2024 were extracted on 8 August 2024.

Selection criteria

Oncology TAs were included if they were of the ‘pharmaceutical’ technology type using predefined filters from the NICE website, and had documentation available at the time of extraction; hence, guidelines which had since been updated, withdrawn or terminated were excluded. All other recommendation outcomes, including those resubmitted to the Cancer Drugs Fund (CDF), were included in the review.

Inclusion criteria

• Oncology NICE TA.

• Published between April 2000 and March 2024.

• Pharmaceutical technology type.

Exclusion criteria

• Non-oncology NICE TA.

• Medical devices, diagnostic techniques or digital product technology type.

• Terminated appraisals.

Data collection process

The Excel extracted from the NICE website had already been filtered to oncology TAs only and included additional descriptive fields for each TA. These included the technology type; year of publication and single versus multiple technology appraisal status, which were used for both eligibility screening and review. Documentation for each TA was extracted using modified code developed by Polak et al.²³

Review criteria

A systematic literature review was undertaken on all TAs identified in figure 1. The review was conducted based on the guidelines for systematic literature reviews by Khan et al²⁴ with two independent reviewers of the documentation and a specialised statistical reviewer to confirm findings on the type(s) of analysis presented in the submissions.

The reviewers independently assessed each TA against the prespecified review criteria (see online supplemental table 1 below). For all TAs included in the review, the reviewers assessed whether or not RWD was submitted by the company as part of the body of evidence. RWD was taken to be any form of observational patient data (eg, retrospective healthcare records, registries, national audits) or insights derived from such data (eg, real-world treatment pathways, patient characteristics, average survival estimates, utility values). RWD from independently conducted academic studies, not established databases, was also considered an acceptable use of RWD and categorised as ‘public literature’. This review focused on RWD presented by the company and included in the final appraisal determination document; therefore, RWD introduced by the external assessment group only was not considered. Bias was not assessed as part of the literature review.

If the TA was deemed to have included RWD, the reviewers answered further questions on the specific RWD source. Where more than one data source was included in the TA, each data source was considered individually for the assessment of use of data sources. Where more than one method was used for RWD in the TA, each method was considered individually for the assessment of use of methods. Whether RWD was used to support the clinical evidence or cost-effectiveness analyses submitted was recorded based on which subsection of the final appraisal document the RWD was introduced and discussed in. The methods used to compare RWD to the trial data/other data sources were recorded for each TA, focussing specifically on how the data was integrated and used as evidence (see review criteria in online supplemental table 1) as well as any concerns of the committee expressly related to the use of the RWD. In addition, the reviewers marked the strength of the contribution of the RWD to the evidence presented from zero to five, following a definition of each category dependent on a combination of the complexity of the analytical methods applied to generate the RWE and the severity and nature of concern, if any, of the committee in relation to the RWD or methods associated with it. Online supplemental figure 1 details the approach used by the reviewers to categorise the strength of the contribution of RWD.

A pro forma with multiple choice answers to these questions and an area for free text was developed and refined using an initial review of the 20 most recent TA documents. Potential answers are described in online supplemental table 1.

Once all TA documents were reviewed, any discrepancies in findings were discussed between the two reviewers until a consensus was reached. If the two reviewers failed to agree following discussion, the statistical reviewer independently reviewed the documentation and made a final decision.

Analysis of review

All TAs included in the review were included in the analysis. The TAs were grouped by financial year of publication and their results were standardised and aggregated at this level. The counts and corresponding percentage of TAs which used RWD in each financial year were calculated, alongside the frequency of use of each distinct RWD source; the methods used to incorporate RWD into the body of evidence; key concerns of the review committee and the strength of the contribution of RWD to the evidence submitted (as described in online supplemental table 1). Case studies were developed expanding on the key concerns of the review committee and why the concerns impact the evidence.

All analyses were performed in R²⁵ V.4.0.2, using tidyverse,²⁶ readxl²⁷ and ggplot2²⁸ for data visualisation.

Patient and public involvement

Patients and the public were not involved in the development or interpretation of this study.

Results

Of 537 oncology TA recommendations identified between 2000/2001 and 2023/2024, 310 met the inclusion and exclusion criteria for the study. Reasons for exclusion are in figure 1 below. Following exclusions, of the 419 TAs, 51 were removed due to being re-reviews. Finally, of the remaining 368 TAs, 58 were removed due to missing documentation (see figure 1).

Figure 1. Flow diagram of oncology therapeutic appraisal documentation exclusions. n, number; TA, therapeutic appraisal.

TA characteristics and citations are available for the included TAs in online supplemental table 2, along with whether the TA used RWD and the datasets used.

Of the 310 TAs reviewed, 153 (49.4%) used RWD. The number of TAs using RWD has increased over time as demonstrated in figure 2 below. While one study used RWD as early as 2001/2002, through the re-use of published literature, the consistent use of RWD did not begin until 2009/2010 with the number of studies using RWD increasing from 3 (37.5%) in 2009/2010 to 2023 (63.9%) in 2022/2023, although a slight decrease to 20 (58.8%) was noted in 2023/2024.

Figure 2. The distribution of therapeutic appraisals using real world data stratified by financial year of publication. RWD, real-world data; TA, therapeutic appraisal.

Use of RWD was most commonly used to support clinical evidence and cost-effectiveness sections of the TA committee appraisal document, with 82 (53.6%) and 96 (62.7%) TAs containing RWD in these areas, respectively. 33 (21.5%) TAs had reported use of RWD across multiple sections of the appraisal document. Between 2009/2010 and 2023/2024, use of RWD in the clinical evidence section of the committee appraisal has increased from 0% to 70.0% of TAs while use of RWD in the cost-effectiveness section has decreased from 100% to 40.0%.

Data sources came from a variety of sources including data from the UK, USA and published literature. UK datasets, not including published literature, appeared in 76 TAs (49.7%). UK and US datasets generally came from data sources with patient level data. UK datasets included systemic anticancer therapy (SACT)/CDF in 31 TAs, Haematological Malignancy Research Network in 14 TAs, Hospital Episode Statistics/Office for National Statistics/Public Health England datasets in 10 TAs, National Cancer Registration and Analysis Service in 4 TAs. Patient level data from the USA included key datasets such as Surveillance Epidemiology and End Results (SEER) in 5 TAs and Flatiron in 4 TAs. Aggregate level RWD from published literature was used in 100 TAs. Multiple data sources were used in 56 (36.6%) TAs with 1 TA using eight different data sources.

When database use was assessed annually (figure 3), a clear trend towards the use of datasets over published literature has been seen since 2019/2020 from 29.6% of reported databases in 2018/2019 to 74.3% in 2022/2023; however, a slight increase in the use of published literature was seen in 2023/2024. SACT and CDF datasets have shown a large uptake in recent years (since 2020/2021), although a small number of TAs used these datasets earlier in 2017/2018 (figure 3). Also, since 2020/2021, the majority of reported datasets are from the UK (shown in shades of orange in figure 3).

Figure 3. Types of real-world data used in oncology therapeutic appraisals by financial year of publication between 2009/2010 and 2023/2024 facetted by the location the dataset comes from. Note 1: the location of published literature was not considered. Note 2: each therapeutic appraisal may use more than one real-world data. CDF, cancer drugs fund; HES, Hospital Episode Statistics; HMRN, Haematological Malignancy Research Network; NCRAS, National Cancer Registration and Analysis Service; ONS, Office for National Statistics; PHE, Public Health England; SACT, systemic anticancer therapy; SEER, Surveillance Epidemiology and End Results.

Comparisons of survival analyses were the most used at 63 times (41.2% of oncology TAs using RWD), followed by utility values (42 uses, 27.5%) and baseline characteristics (29 uses, 19.0%).

Since 2015/2016, MAIC analyses and since 2018/2019, ECA analyses have been used in TA submissions (figure 4). These methods have shown a rapid uptake, for example, in 2023/2024 MAICs were used in 4 (20.0%) TAs reporting RWD while ECAs were used in 5 TAs (25.0%) accounting for 28.1% of all analyses reported that year. Cost analyses using RWD were rarely reported only appearing in the final appraisal determination document of 11 (7.2%) TAs.

Figure 4. Analyses used in oncology therapeutic appraisals stratified by financial year of publication. Note: therapeutic appraisals may use more than one method for the analysis. ECA, external control arm; MAIC, matched adjusted indirect comparison.

Studies were not compared for bias since no information is included in the final appraisal determination document regarding the peer-review process of documents submitted to the committee. The methods used and committee comments were reviewed to identify which of the datasets were considered to have a ‘strong use’ of RWD, based on six levels ranked 0–5. Results are available in table 1 and the ranking is described in online supplemental table 1. Studies could have multiple rankings where multiple datasets were used. Only 59 (38.6% of TAs using RWD) TAs reported concerns about the RWD used.

Table 1. Strength of the use of RWD in the TAs.

Rating	Number (per cent) of TAs
0 - RWD was discarded	8 (5.2)
1 - reuse of published data	69 (45.1)
2 - weak use of RWD limited to summary statistics or similar	52 (34.0)
3 - use of complex methods with major concerns	19 (12.4)
4 - use of complex methods with minor concerns	5 (3.3)
5 - use of complex methods with no concerns	0 (0)

RWD, real-world data; TA, therapeutic appraisal.

Key concerns that regularly appeared in TAs regarding the RWD included: lack of generalisability to the UK population in 10 (16.9%) TAs or NHS practice in 11 (18.6%) TAs, immaturity of data in 11 (18.6%) TAs, inappropriate modelling approaches in 10 (16.9%) TAs and misalignment of comparator populations in 9 (15.3%) TAs. Four TAs (6.8%) had concerns related to RWD regarding the variables available in the dataset. Regular concerns when data from outside the UK was used included differences between ‘standard of care’ since medications that are not recommended for use in the NHS are recommended for use elsewhere. Furthermore, differences in the ethnicity and age profile lead to uncertainty in the population. The most common of these concerns is described with examples using case studies below.

Case study 1: lack of generalisability to NHS practice – TA766 pembrolizumab for adjuvant treatment of completely resected stage 3 melanoma

During the NICE committee review of this TA, concerns were highlighted around the immaturity of survival data in KEYNOTE-054 since the trial is ongoing. The main concern was that the modelled survival was likely overestimated for pembrolizumab and underestimated for routine surveillance. RWD was used to mitigate these concerns, alongside other trial data, including data from the American Joint Committee on Cancer’s (AJCC) eighth edition of melanoma staging and SEER both datasets from the USA. Concerns were raised about the generalisability to the NHS practice of both SEER and AJCC, along with specific concerns about the AJCC data since three major publications report worse outcomes for people with stage 3 melanoma than those reported by the AJCC. The company also used data from SACT to validate the data sources, both trial and RWD, the population of which was found to be older with fewer people having an Eastern Cooperative Oncology Group score of 0. These differences in population lead to uncertainties about the extent of the survival benefit of pembrolizumab. While uncertainties remained, this TA was ultimately approved with an estimated incremental cost-effectiveness ratio (ICER) less than £30 000 per quality-adjusted life year primarily due to the similar mechanism of action to nivolumab which had been found to be cost-effective and the preferable dosing schedule for many patients and healthcare services.

Case study 2: sample size and immaturity of data – TA898 dabrafenib plus trametinib for treating BRAF V600 mutation-positive advanced non-small-cell lung cancer

Due to a lack of direct comparison between dabrafenib plus trametinib and pembrolizumab plus platinum chemotherapy in BRAF V600 mutation-positive advanced non-small-cell lung cancer, the Flatiron database was used as an indirect comparator using inverse probability of treatment weighting. Concerns about uncertainty were raised due to the small sample size in both the trial and Flatiron datasets and due to the limited follow-up available in Flatiron. This analysis was superseded as the preferred evidence source of the committee by a double-blinded phase III randomised controlled trial, KEYNOTE-189, which ultimately used an MAIC analysis for comparative effectiveness between the two treatment regimens. Ultimately, no RWD was included in the committees’ preferred assumptions for this assessment. Based on the cost-effectiveness analysis, dabrafenib plus trametinib was recommended for use in the NHS; however, the maximum acceptable ICER was required to be at the lower end of the £20 000 to £30 000 range.

Case study 3: missing and misaligned variables – TA653 osimertinib for treating EGFR T790M mutation positive advanced non-small-cell lung cancer

This TA presented data on the use of osimertinib in the NHS using the SACT dataset and was ultimately approved; however, this was due to the treatment meeting the extension to life criterion rather than the usual ICER per QALY requirement of <£30 000. Concerns were raised by the committee since patients in SACT were found to have a lower median overall survival than in the clinical trial, AURA3. There was also misalignment and missingness of performance status data in SACT, suggesting that patients in SACT may have been more unwell than in AURA3, along with other variables completely missing in SACT such as frequency of cerebral metastases. These missing data items meant that patient could not be accurately matched potentially leading to the lower median survival. These differences and differences in the overall survival highlighted areas of uncertainty; however, because of these missing data items it was difficult to determine the source of uncertainty.

Discussion

Our review has identified 153 TAs using RWD from a total of 310 TAs reviewed between 2000/2001 and 2023/2024. We have illustrated the increased use of RWD in NICE oncology TAs, and its increasing importance in the deliberation by the committee as reported in the final appraisal determination document. Our review has also considered the wide variety of data sources used, the increasing use of more complex methodologies and highlighted some key concerns of the review committee regarding the use of RWD.

One key finding of our review is the increased uptake of RWD in oncology TAs. The highest proportion (n=6, 75.0%) of TAs using RWD occurred in 2012/2013; however, the vast majority (83.3%) of datasets were published literature. In contrast, the highest number of TAs using RWD (n=23, 63.9%) occurred in 2022/23, the second most recent year of the review, with 80% of data sources coming from non-published literature sources, in particular patient level data sources such as SACT. Potential reasons behind the increasing uptake of RWD in oncology TAs are likely threefold. First, as aforementioned, many regulatory bodies,¹¹ including NICE,⁴ have published guidelines regarding the use of RWD in regulatory submissions. These guidelines demonstrate the increasing acceptance of RWD to supplement submissions, in particular to ‘resolve gaps in knowledge and drive forward access to innovations for patients’. While these guidelines have only become available in more years, it is likely that the wider research community’s acceptance of RWD will influence changing opinions earlier driving uptake of RWD in HTA submissions. Second, the quality and understanding of RWD is improving, especially with more datasets harnessing specialised RWD. Different types of data sources have strengths and weaknesses lending them towards different analytical methods. Patient level electronic health records lend themselves towards ECA analyses, while claims data often contain detailed information about costings. Registries are often limited in the details they collect and the population they collect data on due to the nature of the forms used to collect data and entry criteria used to establish patient entry to the registry. An example of a dataset increasing in popularity for NICE HTA submissions is SACT, a dataset of SACTs from across NHS England trusts. While use of this dataset remains limited, in 2022/2023, the most used dataset was SACT (n=10, 28.6%) having only been released in September 2019²⁹ and rapidly increased in its use since that time. A second example is medications approved for use in the CDF. A major component of approval for use in the CDF is assessment of the effectiveness and safety of the medication while in use in the NHS; hence, when medications return for full approval, the use of RWD is inherently more common. Further data improvements are now occurring with techniques such as artificial intelligence³⁰ and, in particular, natural language processing³¹ allowing for the extraction of data items stored in unstructured formats such as free text. However, further improvements such as improving data coverage, identification of complex data items and diversity of patients within datasets are still needed to enhance what information can be collected.³² Finally, increased guidance^{4 11} regarding methods and tools for regulatory approvals allows for more robust comparative analyses. While methods such as MAIC and to some extent ECA analyses are ‘tried and tested’ likely leading to their recent increased use, concerns remain about using RWD to draw causal conclusions³³ since it is known that using RWD instead of randomised data is not without risk, highlighting the importance of adhering to methods such as target trial emulation,³⁴ an approach which directly adopts key principles in clinical trial design and methods to ensure that inherent biases from RWD are reduced.

Sources of RWD from the UK, USA and Europe have been identified in oncology TA appraisals, and the increasing use of data sources outside the regulators target healthcare geography has driven research into understanding the external validity, or ‘transportability’ of data from outside the UK for use in NICE HTA submissions. Methods have explored the transportability of data from countries other than the target population, through understanding what differences in patient populations and treatment pathways can drive differences in key patient outcomes. However, given the relative recency of these methods, the utilisation of these methods has had limited uptake in health technology assessments.³⁵ A recent study has demonstrated use of transportability methods to assess differences in patients with advanced non-small cell lung cancer between the USA and UK. Their findings suggest that for this specific population, USA data has the potential to be used where UK data is sparse.³⁶ However, newer more detailed UK datasets are arising, which are able to capture more complex patient information than before, such as biomarkers for progression of disease,³⁷ which may mitigate the need for international data sources in the future.

This review has demonstrated that the most common method for which RWD is used is long-term survival estimates at 41.5%. These methods are regularly used to estimate the long-term survival of patients in health economic analyses forecasting survival past the short-term survival estimates identified in randomised control trials. This method has been frequently used since 2014/2015 improving the estimate of the cost per QALY a key metric in assessing the suitability of funding new medications by the NHS. We have also seen the use of MAICs since 2015/2016 and ECA since 2018/2019. These methods are useful when either ethical implications prevent the comparison of a new treatment to a placebo due to the potential harm a patient may face from the lack of treatment for their symptoms³⁸ such as during single-arm studies, or where clinical trials have already been undertaken outside the UK where the standard of care differs to that of the NHS. In the second case, ECAs can function as an additional arm to the trial population for the specific area the HTA is being assessed, allowing for the transportation of clinical trial results to the relevant population, however, NICE typically prefers subgroup analyses in this case. The increasing availability and access to patient level data likely influences the use of these more robust comparative analyses. MAIC analyses are only appropriate when aggregate data are available, while propensity score methods, which are commonly used in ECA analyses, require access to patient level data. This may explain in part the slightly later introduction of ECA analyses to TA documentation with some large patient level datasets of oncology medications in the UK such as SACT having only been available since 2019. Methods such as ECAs can also solve additional limitations of randomised control trials such as difficulties in identifying/defining control groups and analyses of specific subpopulations for new precision medications. However, additional methods, such as quantitative bias analysis,³⁹ a sensitivity analysis which interrogates the underlying assumptions in RWD analyses, are likely necessary to mitigate the underlying uncertainties which RWD bring.

While 2022/2023 saw the largest number of TAs including real world data, further development of guidance, such as that recently published by ISPOR,⁴⁰ is necessary to ensure data is ‘fit-for-purpose’ and to continue overcoming challenges of merging RWD with randomised control trials. Newer methods such as MAIC and ECA are being used in TA applications; however, concerns are still frequently being raised by the committee about the use of RWD with varying levels of confidence about RWD. In our case studies, we have highlighted concerns about missing data items in particular when matching on patient characteristics, misalignment between clinical trial and RWD populations and hence inappropriate matching between populations, the sample size of RWD, inappropriate modelling approaches and lack of generalisability to the UK population. Since RWD is generally collected for the purpose of diagnosing and treating a medical condition, it is not uncommon for data specific to a clinical trial, such as specific genomic variants of a tumour or a scan at a specific date post diagnosis to not be available or to be hidden within unstructured components of the patient record. This can lead to an inability to apply trial inclusion/exclusion criteria to RWD patients due to the unavailability of the key data items required. These concerns of the committee highlight the need for highly detailed UK data, including both structured and unstructured data. However, recent studies have highlighted the issues with RWD, including the lack of transparency regarding the collection and reporting of these datasets.⁴¹

Based on the findings of this study, it is anticipated that RWD will continue to be increasingly used and referred to in the final appraisal documents of NICE oncology TAs, particularly given the trend towards more targeted therapies for the treatment of cancer,⁴² due to their improved survival and reduced undesirable effects. This highlights the need to ensure that RWD used is fit for purpose, meaning that not only it is of good quality, but it contains all data items needed to ensure an accurate comparison to clinical trial data can be made. At present, few datasets in the UK contain comprehensive information of hospital laboratory tests, medications and imaging data highlighting the need for this type of dataset.

While this is one of the largest reviews of NICE HTA documents to date, our review has limitations. We have based our investigation only on the final appraisal document; hence, it may be the case that RWD was used in other documentation provided by the company about the medication being investigated however the committee decided not to include the evidence in their report. While we have identified multiple different analysis types in our review, the most common type was survival modelling hence if RWD is not used for survival modelling there is a chance the RWD use has not been captured; hence, the use of different methods may be under-represented in this review. Further, certain uses of RWD such as ‘cost analyses’ are more routinely accepted in models and hence may be less likely to be referenced as part of the final appraisal document. Second, the purpose of the RWD use was classified as either for clinical evidence or cost-effectiveness based on which section of the final appraisal document it was mentioned. However, it is possible that RWD used for clinical evidence may also be used in the cost-effective analysis model and not explicitly be stated in the cost-effectiveness section of the final appraisal document, hence its use would not have been captured. Third, there were 58 TAs which did not have documentation which is a potential source of selection bias; however, most of these TAs have no documentation since they were either retracted or later re-reviewed in a new TA process limiting the impact of the selection bias. There were an additional 68 terminated appraisals, which never submitted documentation since it became clear the medicine was not a viable candidate. This is an example of positive result bias, and it is not clear whether RWD was used in these appraisals. Since we only investigated NICE oncology TAs, it may be the case that these results are not generalisable to other regulatory bodies. Finally, this review only covers data up until March 2024 and hence ongoing changes in the use of RWD for NICE HTA submissions may affect the generalisability of the paper to more recent years not covered by this review.

Our study quantifies the increasing appearance of RWD in NICE TA committee documents. It has also shown that a wide variety of datasets are being used with varying concerns surrounding these datasets. The increased approval of RWD to support TA increases the opportunity for and speed of new therapies being approved for the treatment of rare and hard-to-treat diseases improving patient care in the UK.

Data availability statement

Data used in this study is freely available from the original source where the NICE TAs were extracted: https://www.nice.org.uk/guidance/published?from=2000-04-01&to=2024-03-31&ngt=Technology+appraisal+guidance. The most recent version of the spreadsheet covering the oncology disease areas can be found at: https://www.nice.org.uk/what-nice-does/our-guidance/about-technology-appraisal-guidance/technology-appraisal-data-cancer-appraisal-recommendations. The final dataset used for analysis is not available for sharing.