“Don’t Think Twice, It’s All Right”: Using Additional Data to Reduce Uncertainty Regarding Oncologic Drugs Provided Through Managed Access Agreements in England

Abstract

Objectives

The Cancer Drugs Fund (CDF) in England uses managed access agreements to facilitate additional data collection to address uncertainties identified in the appraisals of new drugs. This study reviews the uncertainties highlighted in the original appraisals where recommendations “to use within the CDF” were made and how additional data were used to address these uncertainties in the CDF review appraisals where final decisions on routine commissioning were made.

Methods

The first 24 drugs exiting the 2016 CDF were included in this review. The information about uncertainty and the use of newly collected data were extracted from the original appraisals and the CDF review appraisals. The additional data used in the CDF review appraisals, distinguishing between clinical trial data and real-world data (RWD), were reviewed to assess the extent to which the additional data were able to reduce the original uncertainties.

Results

The recommendation that the drug be routinely commissioned was made in 87.5% of re-appraisals. Uncertainty stemming from immaturity of the survival data in clinical trials was frequently found in appraisals. Later follow-up of clinical trials was used to address this uncertainty, whereas limited use was made of RWD. The Systemic Anti-Cancer Therapy (SACT) dataset is the most frequently used source of RWD. SACT data were mostly used in review appraisals to support the clinical outcomes based on later follow-up of trial participants and to inform modelling of subsequent treatments or treatment duration.

Conclusions

While additionally collected RWD attracted attention when the 2016 CDF was introduced, RWD have not been widely used in CDF review appraisals and (to date) have done little to reduce uncertainty. Experience with these appraisals has highlighted the importance of longer follow-up of clinical trials and the relatively limited role of RWD, in general, and of SACT data in particular.

Supplementary Information

The online version contains supplementary material available at 10.1007/s41669-022-00369-9.

Key Points for Decision Makers

When uncertainties regarding the clinical evidence have been too great for National Institute for Health and Care Excellence (NICE) to recommend routine commissioning, managed access agreements have allowed patients to be treated while additional data are collected.

Immature survival data are an important source of clinical uncertainty, which has largely been addressed by later follow-up of patients in clinical trials rather than by additional real-world data.

Systemic Anti-Cancer Therapy data (an important English source of real-world data) have been used to address a limited number of clinical uncertainties.

Introduction

New oncology drugs receive special treatment in England. Since January 2009, differential valuation of the health benefits of many cancer drugs has been implemented by adopting a higher cost-effectiveness threshold for life-extending, end-of-life treatments within the National Health Service (NHS) [1]. In 2010–2011, the Cancer Drugs Fund (CDF) was introduced to provide cancer patients in England with access to drugs that either had not been appraised by the National Institute for Health and Care Excellence (NICE) or had not been recommended for routine commissioning [2]. In the original model of the CDF, there was an absence of clear entry and exit criteria for drugs. This created unsustainable financial pressure without evidence of patient benefit [3–5]. In 2016, the CDF was revised to provide a more sustainable approach to funding promising new drugs and to collecting additional clinical data [6].

Since the reform of the CDF (from here 2016 CDF), all new oncology drugs are appraised by NICE. The 2016 CDF offers a mechanism for conditional approval. Figure 1 shows possible NICE recommendation options. If uncertainties regarding a drug are too great for it to be recommended for routine commissioning, a recommendation for use within the CDF can be considered [7]. The appraisal committee uses the criteria in Fig. 2 to decide which drugs are eligible to be used within the 2016 CDF [5]. One of these is whether the clinical uncertainty can be addressed with additional data collected while the drug is provided through the CDF. If the appraisal committee recommends use within the CDF, a data collection arrangement (DCA) working group is formed, with representation from NICE and NHS England. The DCA working group reviews the data collection proposal to translate the committee’s uncertainties related to clinical outcome into defined data collection questions [8]. Additional data are collected in line with the DCA and form the basis for the review appraisal of the case for routine commissioning, which is expected to happen normally within 2 years [9].

Fig. 1.

Managed access scheme for new cancer drugs in 2016 CDF. CDF Cancer Drugs Fund, NHS National Health Service, NICE National Institute for Health and Care Excellence

Fig. 2.

2016 CDF entry criteria

During this period, more evidence would be collected on the clinical effectiveness of the drug to resolve the key areas of uncertainty. The CDF review appraisal considers the data that have become available since the original appraisal, together with any change to the patient access scheme or commercial access arrangement proposed by the company. However, changes to the scope of the appraisal such as the population and comparators are not considered during CDF reviews [10]. There are two main options for data collection, ongoing and new clinical trials and real-world data (RWD) from the Systemic Anti-Cancer Therapy (SACT) dataset [8]. Other established cancer registries are also potential data sources for further review.

When introducing the 2016 CDF, a role for RWD, particularly SACT data, was highlighted [8]. SACT data are routinely collected for patients receiving anti-cancer therapies from NHS England providers, as a mandatory collection under the responsibility of Public Health England (PHE) [11]. The SACT dataset is preferred for any data collection of routine chemotherapy practice in England because the infrastructure (including data protection and information governance) is already established, data are already being collected and progress can easily be monitored [8]. PHE, through a cancer data partnership with the NHS, initially reported SACT data to support the re-appraisal of treatments provided by the 2016 CDF. NHS Digital took over this responsibility from PHE, on 1 October 2021, when the latter was replaced by the United Kingdom (UK) Health Security Agency and the Office for Health Improvement and Disparities.

The additional data collection is expected to address areas of uncertainty highlighted by the appraisal committee. In 2020, PHE indicated that, “Real-world data reported by PHE is the primary information used to answer NICE uncertainty for 25% of CDF treatments” [12]. This report did not say how RWD had been used as primary evidence to address the uncertainty issues. Understanding how such data are used is more important than simply counting appraisals reporting SACT data. Moreover, given the increasing interest in RWD, 25% of CDF treatments is a relatively low proportion, and the reasons for this low utilisation of RWD need to be reviewed.

Managed access agreements (MAAs) give opportunities to gather additional evidence that could help to reduce uncertainty when making a final decision. A review of the 24 CDF review appraisals completed to date can document the extent to which this objective has been met by collecting RWD. Moreover, it can identify the challenges and opportunities for use of RWD by NICE. It is timely to review experience with the 2016 CDF, because 24 drugs have now completed their re-appraisal, and a broadly similar fund entitled the Innovative Medicines Fund (IMF) has recently been introduced. This paper reviews the committee’s recommendations following re-appraisal in order to obtain insight into the performance of the 2016 CDF. It focuses particularly on the uncertainties that led to drugs being provided through the CDF and on how clinical and cost-effectiveness evidence considered at the re-appraisal differed from that in the original appraisal.

We find that re-appraisals have largely resulted in recommendations for the routine commissioning of these drugs, which might suggest “Don’t think twice, it’s all right,” as a maxim for these decision-makers. However, a detailed review of each re-appraisal indicates quite limited success in reducing the uncertainties which led to these drugs not being recommended for routine commissioning in the original appraisal. It also highlights the relative importance of longer follow-up of trial participants, compared to the contribution of RWD, in addressing some of the original uncertainties. Among different types of additionally collected data, a particular focus of this study is on the use of SACT data, which was highlighted when the 2016 CDF was introduced.

Methods

NICE technology appraisals (TAs) (https://www.nice.org.uk/guidance) were determined to be eligible if they met the following criteria: (1) the drug was provided for the specific indication through the 2016 CDF following an MAA made between NHS England and the manufacturer and (2) the NICE CDF review appraisal had been completed before 16 August 2022. As a result, 24 appraisals were identified for this review. The terminated appraisal (TA674 pembrolizumab) was included in this review. In this appraisal, the company decided not to make a case after the CDF review started, and consequently, there was sufficient data available to include it in the review.

Data were extracted following a protocol developed to extract information about how RWD have been used in NICE appraisals of oncology medicines [13]. This protocol enables a more comprehensive understanding of the use of RWD in CDF review appraisals by identifying non-parametric and parametric use of RWD in both the base-case and sensitivity analyses. Parametric use of RWD is where such data provide the numerical value of a specific variable in the economic model, whereas non-parametric use is where the data are used to develop the model structure or to support, corroborate or validate assumptions and/or choice of data used to parameterise the model. The distinction is made to facilitate more consistent and comprehensive data extraction and to provide a means of measuring the intensity of the use of RWD in an appraisal.

As this data extraction tool was developed for a more general purpose, a few additional variables were required for the specific purposes of this study (Appendix 1, see the electronic supplementary material). These variables capture references to additional data, especially the SACT data, in the CDF review appraisals and the uncertainties identified in the original appraisals and in the review appraisals. Information about uncertainties was extracted from the Final Appraisal Determinations (FADs) in both the original and CDF review appraisals. Uncertainties were classified as either a “key uncertainty” or “other uncertainty”, following Morrell et al., who reviewed the common types of uncertainty addressed in appraisals of drugs that entered the original CDF and discussed the potential for RWD to resolve these uncertainties [14]. If an uncertainty was described in a section heading or highlighted in the conclusion or in the CDF consideration, this uncertainty was considered as a “key uncertainty”. Any other uncertainty addressed across the appraisal was recorded as “other uncertainty”. The uncertainty in CDF review appraisals was reviewed to assess how much additional data helped to reduce uncertainty. Three categories were used (still uncertain, uncertainty resolved, newly added uncertainty) by comparing the FADs from the original and the subsequent appraisal. Any comments about uncertainty made by the committee were recorded. Given that CDF review appraisals highlighted resolving uncertainty identified in the original appraisals, remaining uncertainties were usually addressed in review appraisals. If an uncertainty was not mentioned in the FAD of the review appraisal, it was classified as “resolved”.

The original and CDF review appraisals were compared in terms of the data used, with particular emphasis on where the additional data came from to address the originally identified uncertainties. RWD were of particular interest because one of the arguments for having MAAs was that they provided opportunities to collect additional data, particularly from routine clinical use of the drug. Data were extracted from the main appraisal documents (final scope, company submission, evidence review group [ERG] report, and FAD). Although most evidence used in decision-making was available in these documents, some parts of the evidence in CDF review appraisals were not fully described. When the assumptions made in original appraisals were followed in CDF reviews, the evidence for these assumptions were not fully described in review appraisals. This was often the case with resource use. In this research, evidence not mentioned in any of the four main documents of the CDF review was assumed to be the same as that in the original appraisal. While this is a reasonable assumption, without access to the underlying economic evaluation models, it cannot be guaranteed that the evidence has not changed. Since this research was restricted to the analysis of data in the public domain, this was a potential limitation.

Another research question was whether the pattern of use of RWD changes or not. While drugs were provided through the CDF, companies could collect their own RWD. Additionally collected data could be used not only to reduce uncertainty but also to support their models with more recent evidence. While it might be anticipated that provision through the CDF would increase the opportunities to use RWD in assessing cost-effectiveness, it was possible that the availability of additional trial data reduced reliance on RWD. Hence, the pattern and intensity of use of RWD were reviewed to see whether these changed over the CDF process. Following the data extraction protocol, patterns were identified from both original and review appraisals. Use of RWD in three specific components of an economic evaluation was defined as major use of RWD (use of RWD in estimating overall survival [OS] for either intervention and comparators, volume of treatment for either intervention and comparators and the choice of comparators). These components are likely to have a major impact on the estimated incremental cost-effectiveness ratio. Along with reviewing use of additional data in addressing identified uncertainties, this pattern and intensity review can give a more comprehensive picture of how NICE has used newly collected RWD in CDF reviews.

Results

CDF Review Recommendations

The recommendations made by the committee, reported in Table 1, were unchanged following re-appraisal in 18 cases. In three further re-appraisals, changes were minor. In the case of atezolizumab (TA739), the change was as a result of a changed marketing authorisation, and in two nivolumab appraisals (TA655 and TA713), guidance was further optimised by requiring no prior programmed cell death protein 1 (PD-1) or programmed death-ligand 1 (PD-L1) inhibitor treatment (reflecting changes in clinical practice). There were three cases where the treatments were not recommended for routine commissioning (TA674, TA692 and TA795). Thus, 87.5% of re-appraisals resulted in a recommendation that the treatment be routinely commissioned.

Table 1.

Summary of characteristics of CDF review appraisals in this review

	TA524	TA531	TA629	TA653	TA655
Technology	Brentuximab vedotin	Pembrolizumab	Obinutuzumab + bendamustine	Osimertinib	Nivolumab
Conditions	CD30-positive Hodgkin lymphoma	Untreated PD-L1-positive metastatic non-small-cell lung cancer	Follicular lymphoma refractory after rituximab	EGFR T790M mutation-positive advanced non-small-cell lung cancer	Advanced squamous non-small-cell lung cancer after chemotherapy
Recommendation for routine commissioning	Optimised**	Recommended	Recommended	Optimised**	Optimised**
Date of FAD	04/2018	03/2018	03/2020	09/2020	09/2020
Original appraisal
TA number	TA446	TA447	TA472	TA416	TA483
Date of FAD	04/2017	05/2017	07/2017	09/2016	09/2017
Key uncertainty in FAD	Post-treatment stem cell transplant rates	Immature survival data	Immature survival data	Immature survival data	Immature survival data
Other uncertainties in FAD	OS and PFS following stem cell transplant	Duration of continued treatment effect Appropriate HRQoL values	Duration of continued treatment effect	Appropriate HRQoL values Indirect treatment comparison Small number of participants after restricting the population	Appropriate HRQoL values Stopping rule Duration of continued treatment effect Methods of extrapolation for OS and PFS
Month between FADs	12	10	32	48	36
Evidence in CDF reviews
Clinical effectiveness evidence	New observational data and SACT data	Clinical trial (KEYNOTE-024)	Clinical trial (GADOLIN)	Clinical trial (AURA pooled, AURA3*)	Clinical trial (CheckMate 017, CheckMate 003)
SACT
PHE report in guidance	No	No	Yes, attached	Yes, attached	Yes, attached
Median follow-up (months)	–	–	12.4	Minimum 4 months	3.1
Number (n)	–	–	92	357	389
Use of SACT in economic evaluation
Base-case analysis (parametric use)	Estimate of stem cell transplant rates after treatment	–	–	–	–
Base-case analysis (non-parametric use)	–	–	–	–	Treatment duration Support trial data
Scenario/sensitivity analysis	–	–	–	–	–

	TA674	TA683	TA684	TA687	TA691
Technology	Pembrolizumab	Pembrolizumab + pemetrexed and platinum chemotherapy	Nivolumab	Ribociclib + fulvestrant	Avelumab
Conditions	Untreated PD-L1-positive locally advanced or metastatic urothelial cancer when cisplatin is unsuitable	Untreated, metastatic, non-squamous non-small cell lung cancer	Adjuvant treatment of completely resected melanoma with lymph node involvement or metastatic disease	Hormone receptor-positive, HER2-negative advanced breast cancer after endocrine therapy	Untreated metastatic Merkel cell carcinoma
Recommendation for routine commissioning	Terminated†	Recommended	Recommended	Recommended	Recommended
Date of FAD	02/2021a	01/2021	02/2021	02/2021	03/2021
Original appraisal
TA number	TA522	TA557	TA558	TA593	TA517
Date of FAD	04/2018	11/2018	11/2018	06/2019	03/2018
Key uncertainty in FAD	Immature survival data Indirect treatment comparison	Immature survival data	Immature survival data Recurrence-free survival data	Indirect treatment comparison (results of network meta-analysis) Extrapolation of PFS	Immature survival data Indirect treatment comparison
Other uncertainties in FAD	Duration of continued treatment effect	Duration of treatment effect Indirect treatment comparison	Subsequent treatment Clinical effectiveness of PD-1 inhibitors as adjuvant treatments	Immature survival data Statistical power as relevant population was subgroup in the trial Time to treatment stopping Post-progression survival	Small number of patients in the trial
Months between FADs	34	26	27	20	36
Evidence in CDF reviews
Clinical effectiveness evidence	Clinical trial (KEYNOTE-361*)	Clinical trial (KEYNOTE-189)	Clinical trial (CheckMate 238)	Clinical trial (MONALEESA-3)	Clinical trial (JAVELIN MERKEL 200 trial: Part B)
SACT
PHE report in guidance	Yes, attached	No	Yes, attached	Yes, attached	Yes, attached
Median follow-up (months)	4.2	–	5.0	3.7	6.0
Number (n)	61	–	299	221	52
Use of SACT in economic evaluation
Base-case analysis (parametric use)	–	–	–	–	–
Base-case analysis (non-parametric use)	–	–	–	–	Treatment duration
Scenario/sensitivity analysis	–	–	Subsequent treatment Time gap between treatments	–	–

	TA692	TA713	TA725	TA736	TA739
Technology	Pembrolizumab	Nivolumab	Abemaciclib + fulvestrant	Nivolumab	Atezolizumab
Conditions	Locally advanced or metastatic urothelial cancer after platinum-containing chemotherapy	Advanced non-squamous non-small-cell lung cancer after chemotherapy	Hormone receptor-positive, HER2-negative advanced breast cancer after endocrine therapy	Recurrent or metastatic squamous cell carcinoma of the head and neck after platinum-based chemotherapy	Untreated PD-L1-positive locally advanced or metastatic urothelial cancer when cisplatin is unsuitable
Recommendation for routine commissioning	Not recommended	Optimised**	Recommended	Recommended	Recommended***
Date of FAD	03/2021	05/2021	08/2021	09/2021	09/2021
Original appraisal
TA number	TA519	TA484	TA579	TA490	TA492
Date of FAD	02/2018	09/2017	03/2019	09/2017	10/2017
Key uncertainty in FAD	Immature survival data	Methods of extrapolation for OS and PFS	Immature survival data	Effectiveness according to PD-L1 expression Long-term OS	Indirect treatment comparison (single-arm)
Other uncertainties in FAD	Duration of continued treatment effect	Appropriate HRQoL values Duration of continued treatment benefit Clinical stopping rule Absence of indirect treatment comparison with relevant comparator (best supportive care)	Time on treatment Heterogeneity of network analysis due to absence of direct treatment comparison	Generalisability of the trial to the UK practice Appropriate time point to extrapolate the trial data Underestimated 2-year PFS Time on treatment Duration of continued treatment effect Stopping rule Appropriate HRQoL values	Immature survival data Small number of patients Duration of continued treatment benefit Time on Treatment Appropriate HRQoL values Effectiveness for PD-L1 subgroup
Months between FADs	37	44	29	48	47
Evidence in CDF reviews
Clinical effectiveness evidence	Clinical trial (KEYNOTE-045)	Clinical trial (CheckMate 057, CheckMate 003)	Clinical trial (MONARCH 2)	Clinical trial (CheckMate 141)	Clinical trial (Imvigor130*)
SACT
PHE report in guidance	Yes, attached	Yes, attached	Yes, attached	No	Yes, attached
Median follow-up (months)	Not available	4.2	4.4	–	9.6
Number (n)	102	43	298	–	64
Use of SACT in economic evaluation
Base-case analysis (parametric use)	–	–	–	–	–
Base-case analysis (non-parametric use)	–	Support trial data	Treatment duration Support trial data	Support trial data	Supporting a choice of survival curve
Scenario/sensitivity analysis	–	–	–	–	–

	TA766	TA770	TA780	TA783	TA784
Technology	Pembrolizumab	Pembrolizumab	Nivolumab + ipilimumab	Daratumumab	Niraparib
Conditions	Adjuvant treatment of completely resected stage 3 melanoma with lymph node involvement	Untreated metastatic squamous non-small-cell lung cancer	Untreated advanced renal cell carcinoma	Relapsed and refractory multiple myeloma	Relapsed, platinum-sensitive high-grade serous epithelial ovarian, fallopian tube or primary peritoneal cancer
Recommendation on routine commissioning	Recommended	Optimised**	Recommended	Optimised**	Optimised**
Date of FAD	12/2021	12/2021	01/2022	01/2022	02/2022
Original appraisal
TA number	TA553	TA600	TA581	TA510	TA528
Date of FAD	11/2018	05/2019	03/2019	12/2017	05/2018
Key uncertainty in FAD	Immature survival data Distant metastases-free survival	Immature survival data	Immature survival data	Indirect treatment comparison	Immature survival data More complete understanding of who would benefit most from treatment using somatic and other testing
Other uncertainties in FAD	Duration of continued treatment effect Reuse of pembrolizumab after use in the adjuvant setting	Time to treatment discontinuation Subsequent treatment	Proportion of people with intermediate- and poor-risk disease	Long-term survival benefit Subsequent treatment	Duration of treatment
Months between FADs	37	28	34	49	39
Evidence in CDF reviews
Clinical effectiveness evidence	Clinical trial (KEYNOTE-054)	Clinical trial (KEYNOTE-407)	Clinical trial (CheckMate 214)	Clinical trial (MMY2002)	Clinical trial (NOVA)
SACT
PHE report in guidance	Yes, attached	No	Yes, attached	Yes, attached	Yes, attached
Median follow-up (months)	15.7	–	10.8	4.3	13.7 (germline BRCA mutation), 5.7 (no germline BRCA mutation)
Number (n)	1,324	–	814	2301	157 (germline BRCA mutation), 859 (no germline BRCA mutation)
Use of SACT in economic evaluation
Base-case analysis (parametric use)	Volume of subsequent treatment	–	–	Volume of subsequent treatment	–
Base-case analysis (non-parametric use)	Validation of OS projection Treatment sequence (subsequent treatment)	–	–	Time to discontinuation Support trial data	–
Scenario/sensitivity analysis	Characteristics of population	–	Subsequent treatment	OS	Time to discontinuation

	TA795	TA796	TA798	TA802
Technology	Ibrutinib	Venetoclax	Durvalumab	Cemiplimab
Conditions	Waldenstrom’s macroglobulinaemia	Chronic lymphocytic leukaemia	Maintenance treatment of unresectable non-small-cell lung cancer after platinum-based chemoradiation	Advanced cutaneous squamous cell carcinoma
Recommendation on routine commissioning	Not recommended	Recommended	Recommended	Optimised**
Date of FAD	04/2022	04/2022	05/2022	05/2022
Original appraisal
TA number	TA491	TA487	TA578	TA592
Date of FAD	09/2017	09/2017	03/2019	05/2019
Key uncertainty in FAD	Pre-progression mortality	Patients’ characteristics Indirect treatment comparison	Immature survival data	Immature survival data Indirect treatment comparison
Other uncertainties in FAD	Long-term survival benefit Indirect treatment comparison	Long-term survival after B-cell receptor pathway inhibitors	Duration of continued treatment effect	Dosing regimens
Months between FADs	52	56	38	36
Evidence in CDF reviews
Clinical effectiveness evidence	SACT dataset	SACT dataset	Clinical trial (PACIFIC)	Clinical trial (Study 1423, EMPOWER CSCC-1)
SACT
PHE report in guidance	Yes, attached	Yes, attached	Yes, attached	Yes, attached
Median follow-up (months)	12.9	20.6	7.3	10.2
Number (n)	859	153 (non-del(17p)/TP53 mutation), 218 (Del(17p)/TP53 mutation)	591	352
Use of SACT in economic evaluation
Base-case analysis (parametric use)	OS PFS (using treatment duration as a surrogate)	OS PFS (using treatment duration as a surrogate)	–	–
Base-case analysis (non-parametric use)	Characteristics of population	Characteristics of population	–	Support trial data
Scenario/sensitivity analysis	Pre-progression mortality	.	–	Characteristics of population

CDF Cancer Drugs Fund, EGFR epidermal growth factor receptor, FAD Final Appraisal Determination, HRQoL health-related quality of life, NICE National Institute for Health and Care Excellence, OS overall survival, PD-1 programmed cell death protein 1, PD-L1 programmed death-ligand 1, PFS progression-free survival, PHE Public Health England, SACT Systemic Anti-Cancer Therapy, TA technology appraisal, UK United Kingdom

*New data not submitted in original appraisal

^†Unable to make recommendation due to the company’s withdrawal

^aDate of CDF engagement termination meeting

**“Optimised” is a recommendation for a smaller group of patients than originally stated by the marketing authorisation. TA recommendations are aligned with NICE reports (https://www.nice.org.uk/about/what-we-do/our-programmes/nice-guidance/nice-technology-appraisal-guidance/data/appraisal-recommendations)

***Indication has been changed from original

Key Uncertainties Addressed in the Original Appraisals

The uncertainties reported in the FADs of the original appraisals are shown in Table 1. Immature survival data in the clinical trials were identified as a common source of uncertainty in most appraisals (67% of appraisals as key uncertainty; 10% of appraisals as other uncertainty). These data increased uncertainty around the size of clinical benefits or long-term benefits. In ten appraisals (42%), an indirect treatment comparison was a source of uncertainty when assessing the clinical benefits. Among them, six appraisals identified this as a key uncertainty. Indirect treatment comparisons were made because of an absence of randomised clinical trials (RCTs) or because relevant comparators were not included in a single RCT. Another source of uncertainty was how clinical effectiveness varied across subgroups defined by the expression of PD-L1. Duration of treatment effect, time on treatment and health-related quality of life (HRQoL) values were frequently noted as sources of uncertainty, but were not identified as key uncertainties in many appraisals.

Use of Additional Data in Economic Evaluation in CDF Reviews

Additional Data from Clinical Trials

The average time gap between the publication of the FADs for the original appraisal and the CDF review was 35.6 months (median 36 months). The main evidence for economic evaluation in CDF review appraisals was from clinical trials. The additional data in 17 CDF review appraisals came from further follow up of patients in the trials featured in the original appraisal. Two CDF review appraisals (TA674, TA739) used data from clinical trials that were not presented in the original appraisals. Another appraisal (TA653) used both later follow-up of a trial and new clinical trial data. Three appraisals (TA524, TA795, TA796) used SACT data along with previously used data as the main evidence for the economic evaluation model.

Since the information about median follow-up was redacted in a few appraisals, five appraisals were excluded to estimate the increase in the duration of follow-up. The average increase in median follow-up between the original appraisal and the CDF review appraisal was 22.2 months (median 22.2 months). The longest increase in median follow-up was 50 months in the appraisal of niraparib (TA784). Two review appraisals (TA629, TA770) reported increases in median follow-up of 6 months.

Additional Real-World Data

SACT data were the predominant type of RWD used in CDF review appraisals. The use of SACT is reviewed in a separate section below. Here, RWD, other than SACT data are reviewed. There were three cases where RWD other than SACT data were used in the CDF review, but not in the original appraisal. In the CDF review appraisal of pembrolizumab (TA766), the company used the registry data (the Surveillance, Epidemiology and End Results [SEER] and American Joint Committee on Cancer [AJCC]) as well as SACT data to validate the survival distribution. This appraisal also used the market share data for subsequent treatment lines in a scenario analysis. The appraisal of niraparib (TA784) used a chart review study for clinical outcomes with the comparator, routine surveillance data, in a scenario analysis. One of the uncertainties in the original appraisal derived from an indirect treatment comparison. The company used RWD in a scenario analysis to investigate the uncertainty around the indirect comparison, whereas they used data from another clinical trial for the base-case analysis. The appraisal of cemiplimab (TA802) used a new retrospective chart review study for comparative evidence as the lack of comparative evidence was highlighted during the original appraisal. However, the comparative effectiveness of cemiplimab remained highly uncertain due to the chart review lacking validity.

The CDF review appraisal of pembrolizumab (TA770) stopped using RWD when extrapolating OS. The company’s original model was criticised due to missing information about the second-line treatments. In the review appraisal, the company dropped these data and used more recent clinical trial data. The appraisal of ibrutinib (TA795) substituted the RWD used in the original appraisal with UK-based registry data, to help address the data gap (progression-free survival [PFS]), which SACT could not provide. Also, these data were used to estimate a rate of pre-progression mortality in a scenario analysis, which was one of the key uncertainties in the original appraisal.

Patterns of use of RWD in the original appraisals and in the CDF review appraisals were compared (Appendix 2, see the electronic supplementary material). Although there were changes in use of RWD, limited use was made of RWD collected during the CDF period. Substantial changes in patterns of RWD use were not found. Consequently, the intensity of the use of RWD has not changed. In the CDF review of pembrolizumab (TA766), RWD were used more broadly for supporting diverse assumptions in the model such as validating survival extrapolation, informing subsequent treatment line and baseline age of population in the model. However, the intensity of use of RWD has not changed much as only one additional component (volume of subsequent treatment) was informed by RWD (in this case SACT data).

Use of SACT Data in CDF Review Economic Evaluations

This study focused on the use of SACT data in CDF review appraisals. SACT data were the most commonly used form of RWD in CDF reviews. Since data collection via SACT was a part of the MAAs, the primary source of additional RWD was substantially the SACT database. Although the SACT dataset was the major vehicle to collect RWD, its overall use was limited. SACT data were not used to update the economic evaluation model in nine out of 24 CDF review appraisals (Table 2). The remaining 15 appraisals made limited use of SACT data. SACT data, newly collected from CDF patients, were used more for non-parametric purposes (11 appraisals), such as validation or corroboration of the model, than for parametric purposes (five appraisals). SACT data featured in both non-parametric and parametric uses in four appraisals (TA766, TA783, TA795, TA796).

Table 2.

Summary of use of SACT data in CDF review appraisals

Type of use	Drug	TA number
Parametric use	Brentuximab vedotin	TA524
	Pembrolizumab	TA766
	Daratumumab	TA783
	Ibrutinib	TA795
	Venetoclax	TA796
Non-parametric use	Nivolumab	TA655
	Avelumab	TA691
	Nivolumab	TA713
	Abemaciclib + fulvestrant	TA725
	Nivolumab	TA736
	Atezolizumab	TA739
	Pembrolizumab	TA766
	Daratumumab	TA783
	Cemiplimab	TA802
	Ibrutinib	TA795
	Venetoclax	TA796
Used in sensitivity/scenario analysis	Nivolumab	TA684
	Pembrolizumab	TA766
	Nivolumab + ipilimumab	TA780
	Daratumumab	TA783
	Niraparib	TA784
	Cemiplimab	TA802
	Ibrutinib	TA795
Not used	Pembrolizumab	TA531
	Obinutuzumab + bendamustine	TA629
	Osimertinib	TA653
	Pembrolizumab	TA674
	Pembrolizumab	TA683
	Ribociclib + fulvestrant	TA687
	Pembrolizumab	TA692
	Pembrolizumab	TA770
	Durvalumab	TA798

CDF Cancer Drugs Fund, SACT Systemic Anti-Cancer Therapy, TA technology appraisal

Parametric Use

Five cases of parametric use were identified. In the CDF review of brentuximab vedotin (TA524), the company used CDF data to inform the rate of subsequent stem cell transplant following treatment with brentuximab vedotin. This was one of the key clinical uncertainties, which was expected to be resolved during the CDF period. A questionnaire sent to consultants identified the rates of stem cell transplant in patients who had brentuximab vedotin as part of the original CDF between April 2013 and March 2016. Another example was the CDF review appraisal of pembrolizumab (TA766). The company used SACT data for the distribution of subsequent treatments administered in the advanced setting for patients in the adjuvant pembrolizumab arm as clinical evidence was incomplete and SACT data were the best available RWD to reflect the clinical practice observed in the CDF. Similar to TA766, in the CDF review appraisal of daratumumab (TA783), SACT data were used to inform subsequent therapies for all comparators. The appraisal of ibrutinib (TA795) has used SACT data as primary clinical evidence in an economic evaluation model. In the original appraisal, the company used a single-arm trial, Study 1118E, for the clinical outcome. Longer-term clinical effects were highly uncertain due to the limited long-term data. In the CDF review, the company revised their base-case analysis using SACT data to calibrate OS for the transition probability (post-progression mortality). Since SACT data did not record disease progression data, the company used other sources of RWD to estimate PFS. Here, treatment duration from SACT data was used to adjust the hazard compared with PFS. The appraisal committee concluded that there was considerable uncertainty around the most appropriate approach to estimating PFS of ibrutinib although an indirect approach to estimate PFS was reasonable. The CDF review appraisal of venetoclax (TA796) also used SACT data as primary clinical evidence in the economic model. Parametric models for OS were explored using SACT data. In this review appraisal, the company assumed that PFS was equivalent to the duration of venetoclax treatment. During the appraisals, the committee concluded that the assumption regarding PFS was plausible and that SACT data were the best available and were acceptable to represent venetoclax efficacy.

Non-parametric Use

Non-parametric use of the SACT dataset has been made in 11 CDF review appraisals. Five forms of non-parametric use, informing characteristics of the study population, updating the subsequent treatment line, validation of survival outcome, treatment duration and corroboration of survival data, were identified. Two CDF reviews used SACT data to validate the choice of survival curves in the model (TA739, TA766). In both original appraisals, extrapolation of the survival data was highly uncertain. Updated clinical trial data directly informed the estimates of OS in the economic evaluation model. The clinical plausibility of the survival distribution selected in the model in the CDF review appraisal was checked with SACT data. The duration of treatment was reviewed in four appraisals (TA655, TA691, TA725, TA783) by seeing to what extent SACT data were aligned with the trial data. This informed the discussion of the generalisability of the trial data to routine clinical practice in NHS England but did not inform the estimates of time-on-treatment directly. In six appraisals (TA655, TA713, TA725, TA736, TA783, TA802), SACT data were used to corroborate the clinical trial evidence. Median OS in SACT data and the overlaid survival curves were usually presented to support the trial data. There was one appraisal where SACT data were used to update the subsequent treatment line in the base-case analysis (TA766) and two appraisals (TA795, TA796) where SACT data were used to inform the characteristics of the study population.

Use of SACT Data in Sensitivity/Scenario Analyses

SACT data were used in six CDF reviews (TA684, TA766, TA780, TA783, TA784, TA802) to explore the impact of alternative assumptions in sensitivity or scenario analyses. In one appraisal (TA784), the company used the time to discontinuation in SACT data at the request of NHS England. The company used the SACT data in a scenario analysis but not in the base-case economic model, due to limited availability of baseline characteristics in the SACT database.

SACT Not Used

Evidence from SACT was not used to either update the economic model or support the evidence in nine appraisals. Three patterns of non-use of SACT were identified. In pattern 1, no information on SACT data was reported in the appraisal documentation nor was the PHE report uploaded (TA531, TA683, TA770). In pattern 2, SACT data were attached, but were not reported in the company submission (TA674, TA692). In pattern 3, the company submission reported SACT data and the PHE reports were attached (TA629, TA653, TA687, TA798), but the SACT data were neither used as corroboration nor used directly in the economic evaluation model. The small number of patients and the limited follow-up periods were given as reasons for not using the SACT data.

Assessment of the Extent to Which Additional Data Reduced the Original Uncertainties

In the CDF review, the technical engagement process was important to discuss the methods with which to deal with uncertainties. Technical engagement is a step where companies get a technical report from the NICE technical team and have a chance to mitigate the remaining uncertainties in the evidence base before appraisal committee meetings [15]. In this process, discussion between ERGs and companies is also allowed. Companies have an opportunity to improve their evidence through this engagement.

Although the technical engagement could help to reduce the methodological challenges, some uncertainties remained. Data from new trials and later follow-up of existing trials were important when it comes to resolving these uncertainties. Uncertainty around immaturity was addressed by clinical trials that had further follow-up. However, later analysis of clinical trials could not solve all immaturity issues. Committees in three review appraisals (TA531, TA684, TA766) still had concerns about the immaturity of survival data. Although the clinical trial captured survival events over a longer period, choice of parametric model to predict OS was highly uncertain in five appraisals (TA655, TA683, TA687, TA692, TA713).

Uncertainty around survival benefit due to indirect treatment comparison was resolved by clinical trials when new RCTs were available. When the original appraisal was based on a single-arm trial while RCTs were ongoing, the review appraisal updated the model based on new phase 3 trials (TA492, TA519). However, if the RCTs didn’t include all relevant comparators, clinical trials had limited scope to reduce uncertainty coming from indirect treatment comparisons. Commonly unresolved uncertainties in CDF review appraisals were the duration of continued treatment effects and the best utility values to use. It was common to use the assumptions previously preferred by committee. Also, clinical experts’ opinions were often used to discuss these issues.

The SACT dataset has rarely been actively used to deal with uncertainties because SACT data were not regarded as robust enough for use in the economic evaluation. A few review appraisals (TA629, TA691, TA725, TA766, TA784) directly indicated that the SACT data were too immature. As later clinical trial data were available, SACT data were less relevant to address the uncertainty around immature data. However, SACT data have provided useful information such as time to treatment discontinuation and subsequent treatment. For example, in the CDF review appraisal of TA581 (TA780), one of the uncertainties was answered by SACT data. The committee preferred to use the proportions based on SACT data to weight the effectiveness estimates by risk group in the clinical trials as the SACT data were expected to inform the true proportion.

Discussion

The central findings of this study of experience to date with CDF review appraisals are the limited role played by SACT data and the importance of longer follow-up of the patients in the clinical trials upon which the original appraisals were based. Reasons for these key features of the review appraisals are not hard to find. The additional data available from SACT are limited in several respects—SACT data are not randomised, and survival data are generally immature given the period during which the CDF provided the treatment. The value of the SACT data may be further limited by the number of patients included and the information recorded. The former is also a direct consequence of the timetable chosen for the CDF review.

The use of clinical trial data in preference to SACT data is partly because the latter are not randomised. Comparisons of SACT data with other groups of patients in terms of PFS and OS potentially introduce bias because of differences between patient groups in the distribution of effect modifiers [16, 17]. However, not all the trials used in the original appraisals were randomised trials. In such cases, this limitation of SACT data is less important. For example, in the recent re-appraisal of ibrutinib for treating Waldenstrom’s macroglobulinaemia (TA795), the committee concluded that the SACT data (n = 823) were more relevant than updated trial data from the single-arm Study 1118E (n = 63) and iNNOVATE arm C (n = 31).

The number of patients available for analysis is often smaller and the length of patient follow-up is shorter in the SACT database than in the original clinical trial. For example, in the re-appraisal of avelumab (TA691), the number of trial participants exceeded that in the SACT data (n = 116 vs n = 52), also median follow-up in JAVELIN was 16 months versus 6 months in the SACT database. Also, the data required for the economic model is more often available from the trial rather than from the SACT database. Potentially important model inputs such as PFS, HRQoL and response rate are not available in the SACT database [18].

Latimer suggests that the problem lies not just with the SACT database itself but is in part a failure to exploit the analytical opportunities these data offer [19]. In reviewing the early entrants to the 2016 CDF, he notes that little information was given as to how the SACT dataset would be analysed. In recent CDF review appraisals, TA795 and TA796, SACT data have been used to a greater extent for OS and PFS estimation through active technical engagements and exploring the plausible ways of using the data. A more coherent analytical plan for assessing comparative effectiveness could facilitate better use of SACT data to support the reduction of uncertainties [19, 20].

It is important to stress that this paper reviews experience with the first 24 drugs to exit the 2016 CDF. It accurately documents this recent experience. It is not claiming that SACT data (or other RWD) cannot play a major role in resolving the clinical uncertainties, which have in turn contributed to uncertainty regarding the cost-effectiveness of many new oncologic drugs. The claim is simply that to date the contribution to resolving clinical uncertainty has been modest. More detailed planning for future analysis and longer periods of data collection might both increase the potential contribution of SACT data. It is noteworthy that the consultation over the IMF, a recently introduced sister fund to the CDF for non-oncologic medicines, made reference to provision for a period not exceeding 5 years [22], as does the recent NICE process and methods manual.

A review of the operation of the 2016 CDF is particularly relevant since NHS England is expanding the use of managed access schemes with the introduction of the IMF. While it is likely that the IMF will operate in a similar fashion to the CDF, it will support “patients with any condition, including those with rare and genetic diseases, to get early access to the most clinically promising treatments where further data are needed to support NICE make recommendations with respect to routine commissioning by the NHS” [21]. Consideration of experience with the CDF can aid understanding of the opportunities and challenges of using additional data to address uncertainties.

Although the use of RWD in CDF review appraisals is an institution-specific issue, the use of RWD in drug appraisals is of more general interest. The Italian Medicines Agency (AIFA) monitoring platform of registries tracks eligible patients and a complete flow of treatment to evaluate the appropriate use of drugs following their approval in the Italian national health system [22]. The data collected are useful sources for verifying the real impact of the initial reimbursement criteria [23]. In Dutch health technology assessment (HTA) reports, RWD have been used for initial decision-making. In conditional financing, a type of MAA, use of RWD to reduce uncertainty has attracted attention [24]. However, a detailed analysis of the utilisation of different forms of RWD in different HTA systems is beyond the scope of this paper.

This paper has focused on the uncertainties in the original appraisal and the additional data considered at the re-appraisal. It has not sought to assess the success or otherwise of the CDF. Patients have had access to these 24 therapies through the CDF following an initial decision not to recommend routine commissioning. Moreover, following re-appraisal, 21 have moved to routine commissioning. In addition, while in the CDF, the drugs have had a price that is deemed cost-effective given the available evidence. An alternative perspective might be that the original clinical uncertainties do not appear to have been markedly reduced and still the re-appraisals have been overwhelmingly positive. This possibly suggests that CDF review appraisals should be regarded as a “review to ensure that the original decision is consistent with the latest evidence”, rather than as a “final chance to make the case”. However, before accepting Bob Dylan’s rejection of re-appraisal and the re-assurance that NICE committees can generally make the correct decision at the first attempt, it is important to recognise that any assessment of the value of the CDF needs to make a judgment regarding the counter-factual, including how the existence of the CDF might be influencing committees’ decision-making and manufacturers’ research activities and pricing decisions.

Conclusions

While additionally collected RWD attracted attention when the 2016 CDF was introduced, RWD were not widely used in CDF review appraisals and (to date) do little to reduce uncertainty. Experience with these appraisals has highlighted the importance of longer follow-up of clinical trials and the relatively limited role of RWD, in general, and SACT data in particular. Although the 2016 CDF, with its MAAs, is a clear improvement on the original CDF, the extent to which the clinical uncertainties have been resolved by additional data is unclear.

Supplementary Information

Below is the link to the electronic supplementary material.

Declarations

Funding

Jiyeon Kang is supported by the Centre for Cancer Biomarkers, University of Bergen, funded by the Research Council of Norway grant number (223250). The funder is not involved in any aspect of the study conduct or the decision to submit the paper for publication.

Conflict of interest

The authors declare no competing interest.

Ethics approval

This study was approved by the Ethics Committee of the London School of Hygiene and Tropical Medicine on 14 November 2019 (17315).

Consent to participate

This study used publicly available data. Consent to participate is not applicable in this study.

Consent for publication

Not applicable.

Availability of data and material

The data analysed during the current study are available on the National Institute for Health and Care Excellence website [https://www.nice.org.uk/guidance/published?ngt=Technology%20appraisal%20guidance&ndt=Guidance].

Code availability

Not applicable.

Author contributions

Both authors contributed to conceptualising and designing the study. JK analysed the data and drafted the manuscript. JC revised the manuscript for important intellectual content and contributed to the methodology.