Evidence Following Conditional NICE Technology Appraisal Recommendations: A Critical Analysis of Methods, Quality and Risk of Bias

Abstract

Background

The National Institute for Health and Care Excellence (NICE) may approve health technologies on condition of more evidence generated only in research (OiR) or only with research (OwR). NICE specifies the information needed to comply with its request, although it may not necessarily guarantee good quality and timely evidence for re-appraisal, before reaching a final decision.

Aim

This study aimed to critically appraise the methods, quality and risk of bias of evidence generated in response to NICE OiR and OwR technology appraisal (TA) and highly specialised technologies (HSTs) recommendations.

Methods

NICE TAs (between March 2000 and September 2020) and HST evaluations (to October 2023) of medicines were reviewed. Conditional recommendations were analysed to identify the evidence requested by NICE for re-appraisal. The new evidence was analysed for compliance with NICE’s request and assessed using the Cochrane Collaboration’s tools for risk of bias in randomised trials and the ROBINS-I tool for non-randomised evidence.

Results

NICE made 54 conditional recommendations from TAs (13 OiR and 41 OwR) and five conditional recommendations for HSTs (all OwR). Of these, 16 TAs presented additional evidence for re-appraisal (9 OiR [69%] and 7 OwR [17%]) and three HSTs (3 OwR [60%]). Two of the nine re-appraised TAs with OiR recommendation and four of the seven OwR complied fully with NICE’s request for further evidence, while all three from the HSTs complied. The majority of re-appraised TAs and HSTs included evidence that was deemed to be at serious, high, moderate or unclear risk of bias. Among the 26 randomised controlled trials from TAs assessed, eight were categorised as having low risk of bias in all domains and ten had at least one domain as a high risk of bias. Reporting was unclear for the remainder. Twenty-two non-randomised studies, primarily single-arm studies, were susceptible to biases mostly due to the selection of participants and to confounding. Two HSTs provided evidence from randomised controlled trials which were classified as unclear or high risk of bias. All non-randomised evidence from HSTs were categorised as moderate or serious risk of bias.

Conclusions

There is widespread non-compliance with agreed data requests and important variation in the quality of evidence submitted in response to NICE conditional approval recommendations. Quality standards ought to be stipulated in respect to evidence contributing to re-appraisals following NICE conditional approval recommendations.

Supplementary Information

The online version contains supplementary material available at 10.1007/s40273-024-01418-3.

Key Points for Decision Makers

There is widespread non-compliance with agreed data requests following conditional NICE technology appraisal recommendations.

The majority of evidence submitted for re-appraisal was judged to be at serious, high, moderate or unclear risk of bias.

Quality standards ought to be stipulated in respect to evidence contributing to re-appraisals following conditional NICE approval recommendations.

Introduction

Pharmaceutical purchasing decisions have historically been based on prices set by their manufacturers and the evidence available at the time of application for reimbursement [1]. More recently, mechanisms for approving medicines conditional on further evidence being generated has gained traction as a mechanism for earlier patient access to new and sometimes innovative medicines [2–7]. Approaches to conditional approval include only in research (OiR) and only with research (OwR) recommendations; with the main distinction being that in OiR, coverage is limited to participants enrolled in research studies, whereas the whole eligible population is covered following an OwR recommendation, but with pre-specified data collected [1, 4, 8].

The National Institute for Health and Care Excellence (NICE) has always had the option to recommend health technologies OiR where there is uncertainty in clinical parameters (along with ‘recommended’, ‘optimised’ [that is, recommended for a sub-group of patients for whom the medicine is clinically indicated] and ‘not recommended’ appraisal decision outcomes [9]). OiR recommendations were more common in the early years of the NICE appraisal process, contrasting with OwR which has become more commonplace following provisions for performance-based risk-sharing arrangements in the Pharmaceutical Price Regulation Scheme and more recently the Voluntary Scheme for Branded Medicines Pricing and Access. Most notably, OwR recommendations have featured extensively in managed access agreements (MAAs) for Cancer Drugs Fund (CDF) medicines, a trend set to continue with the Innovative Medicines Fund for England. MAAs consist of two main components—a data collection arrangement to address clinical uncertainty and a commercial agreement that controls drug costs whilst data are collected [10]. Medicines included in the CDF or the Innovative Medicines Fund list are funded for a specified period and considered for routine commissioning upon review of additional data in a subsequent appraisal.

One source of concern about NICE’s conditional approval process is the quality and reliability of the methods of evidence generation, analysis and reporting [3, 11]. Morrell et al. [12] identified the most common sources of uncertainty among a sample of technology appraisals (TAs) of cancer medicines as being the immaturity of survival data and issues relating to comparators. They commented that these could not be readily resolved by real-world data collection where there are no ongoing trials to provide longer-term data. However, subsequent reviews of cancer medicines appraised by NICE found more extensive use of later follow-up data from clinical trials than from real-world data for addressing uncertainty related to survival estimates [13, 14].

A second area of concern is the increasing focus on OwR in preference to OiR. While the research following OiR recommendation is predominantly funded by manufacturers, OwR incurs a cost to the NHS. The relative value of both approaches ought to be established when conditional approval recommendations are made. Central to this is the need to consider explicitly the health gain foregone by patients denied access to a medicine until the research is completed (OiR) and the health gain foregone by other patients when a medicine is reimbursed that is subsequently found to be insufficiently effective (OwR) [15].

NICE’s conditional approval mechanisms facilitate patient access to specific categories of medicines based on immature evidence and lower certainty around their clinical and cost effectiveness. Providing access to ineffective treatments or treatments with questionable evidence imposes an opportunity cost in terms of impacting on access to medicines and services in other areas of the NHS [16]. The drive to expedite patient access has also led to a wider acceptance of inferior evidence based on non-randomised research [17]. While the collection of further evidence can reduce the uncertainty of re-appraisal decisions, the reversibility of such decisions is potentially limited, although in practice treatment would continue for responders, and not initiated for new patients.

Recognising the limitations of conditional approval, and the variable nature of the evidence available for re-appraisal, the aim of this study was to critically appraise the methods, quality and risk of bias of evidence generated in response to OiR and OwR recommendations by NICE in TAs and evaluations of highly specialised technologies (HSTs).

Methods

Data Sources

All medicines appraised by NICE between March 2000 and September 2020 were identified from a published list of TAs [18] and HSTs up to October 2023 [19]. The final appraisal determinations (or if unavailable, the assessment reports) for TAs and the final evaluation determinations for HSTs were reviewed for whether they were approved conditionally (OiR or OwR). Relevant committee papers and MAAs (where applicable) were retrieved from NICE’s website or, in the case of historical TAs (which had been deleted from the website), from the UK Web Archive online or the UK government National Web Archive, respectively (see Supplementary Table 1 in the electronic supplementary material [ESM]).

Inclusion Criteria

The review was limited to pharmacological interventions, which represent the majority of technologies with an MAA and all technologies on the CDF list. For the purposes of this review, TAs and HSTs were classified as either OiR or OwR based on the following definitions of Walker et al. [8]: “OiR—Coverage of a technology is available only to patients involved in research. OwR—A positive coverage decision is conditioned upon the collection of additional evidence to support continued, expanded or withdrawal of coverage.”

Exclusion Criteria

TAs and HSTs were excluded from the review if they were not recommended or had been terminated due to non-submission by the pharmaceutical company.

Data Extraction

A database was constructed in MS Excel to record details of each TA and HST, including the name of the medicine, clinical indication, NICE recommendation, CDF and MAA status and whether the recommendation was OiR or OwR. Data were extracted from NICE TA and HST guidances on the areas of uncertainty highlighted by the appraisal committees and the recommendations made for further research (study type and data sources). MAAs were also checked to identify any agreement or recommendation that may not have been mentioned in the TA or HST.

Medicines that were re-appraised after conditional approval were identified and the TAs or HSTs reviewed for reference to the new evidence generated. The original (_X) and re-appraised (_Y) TAs (HSTs) are referenced as TA_X → TA_Y (HST_X → HST_Y). Further information about the evidence was then identified from published journal articles and NICE committee papers. Data were subsequently extracted on the source of the clinical evidence (e.g. trials, audits, registries, experts), the research design (randomised controlled trials [RCTs], observational studies, opinion) and methods (randomisation, blinding, consideration of confounding, outcome reporting, data missingness and adherence to intervention). The principal findings of the additional research studies were summarised.

Analysis

The risk of bias from randomised evidence identified for re-appraised medicines was analysed using the Cochrane Collaboration’s tool for assessing risk of bias in randomised trials [20]. The tool considers six domains of bias: selection bias, performance bias, detection bias, attrition bias, reporting bias and other biases. Each domain of bias could be categorised as low risk, high risk, or unclear risk.

Non-randomised evidence was analysed using the Cochrane Collaboration’s Risk Of Bias In Non-randomised Studies of Interventions (ROBINS-I) tool [21], which considers seven domains of bias: two pre-intervention (confounding, selection of participants), one at intervention (classification of intervention) and four post-intervention (deviations from intended interventions, missing data, measurement of outcomes and selection of the reported result). Each domain could be categorised as low, moderate, serious, or critical risk, or no information provided.

Risk of bias assessments were undertaken by YPP and reviewed by DAH.

Results

Included Appraisals

For the period of analysis, NICE conducted 651 TAs and made 968 individual recommendations. Six hundred TAs were of medicines, from which 88 were not recommended and 61 related to terminated appraisals. Of the remaining TAs, 53 were included for analysis. These made 54 recommendations—13 were OiR and 41 satisfied the criteria for classification as OwR. Additionally, some TAs were misclassified in NICE’s database, including TA070 which was listed as recommended, but with a condition of further data collection; TA397, which was listed as optimised and not highlighted as OiR; and TA588 which was an optimised recommendation, but linked to an MAA with conditions for further data collection.

NICE also conducted 28 HST evaluations, of which five were conditionally approved OwR. HST1 was classified as OwR because one of the conditions for approval included a research programme to evaluate when to stop the treatment, although it was the only HST conditionally approved without an MAA (see Fig. 1 and Supplementary Table 2 [in the ESM]).

Fig. 1.

Flow diagram of Technology Appraisals (TAs) and Highly Specialised Technologies (HSTs) included in the review

NICE Recommendations for Further Research

NICE specified the need to address uncertainty in clinical and cost effectiveness in all conditionally approved TAs and HSTs. Recommendations for further research included RCTs, observational studies, or a combination of both (Table 1). For example, among OiR recommendations, in TA093 the committee recommended obtaining data from well-designed (unspecified method) clinical studies and an audit, while in TA111, the committee recommended research, “preferably in the form of RCTs”, and an audit. For TAs with OwR recommendations, data were requested mainly from ongoing trials, and from the Systemic Anti-Cancer Therapy (SACT) and Blueteq registries as secondary sources of information. Only TA581 recommended a new RCT, linked to a post-marketing condition for approval by the European Medicines Agency. Recommendations for all conditionally approved HSTs included the collection of auditable data through specific databases. For instance, for HST1 and HST2, NICE recommended the creation of a national registry; for HST3, data would be collected via the NorthStart database; for HST6, through a disease-specific MAA database; while for HST12 it would be collected through the creation of an Excel SharePoint repository of registering patients.

Table 1.

Study type and data sources

NICE recommendations for further evidence	TAs			HSTs
	OiR (13)	OwR (41)		OwR (5)
		MAA (40)a	Other (1)	MAA (4)	Other (1)
Randomised controlled trial evidence
New	3	1	–	–	–
Ongoing	1	36	–	–	-
Observational studies
SACT		38	–	–	–
Blueteq		34	–	–	–
British Isles Lupus Assessment Group Registry	1	–	–	–	–
National Registry	1		1	1	1
Waldenstrom’s Macroglobulinaemia UK Morrison Registry	–	1	–	–	–
Hospital Episode Statistics	–	1	–	–	–
Bone Marrow (stem cell) Transplant Register	–	2	–	–	–
Medical Data Solutions and Services	–	2	–	–	–
Annual or biannual surveys (patient-reported outcome measures)	–	1	–	–	–
Retrospective chart review	–	1	–	–	–
European Reference Networks for Rare Cancers Genomic Registry	–	1	–	–	–
Flatiron Health and Foundation Medicine Clinico-genomic Database	–	1	–	–	–
Foundation Medicine Genomic Database	–	1	–	–	–
NorthStar database	–	–	–	1	–
Disease-specific MAA database with data from the hypophosphatasia Global Registry	–	–	–	1	–
Excel electronic SharePoint repository	–	–	–	1	–
Collection of auditable data (for the manufacturer to carry out according to NICE recommendations)	5	1	1	4	1
Retrospective, non-interventional trial	–	1	–	–	–
Other: ‘further studies’ (unspecified source)b	7	–	1	–	1
Unspecified ‘ongoing studies’	1	–	1	–	–

CDF Cancer Drug Fund, HSTs highly specialised technologies, MAA managed access agreement, OiR only in research, OwR only with research, RCT randomised controlled trial, SACT Systemic Anti-Cancer Therapy, TAs technology appraisals

− No observations in this category

^aAll TAs with MAA are part of CDF with one exception

^bSeven individual recommendations from six TAs recommended OiR (TA065, TA072, TA093, TA193 [two indications], TA244 and TA467) and TA070 recommended OwR; and HST1

NICE Technology Appraisals (TAs) and Highly Specialised Technologies (HSTs) Subjected to Re-appraisal

Sixteen TAs that included 17 individual recommendations were subsequently re-appraised by NICE.

Nine TAs were originally recommended OiR (including TA193 with two individual recommendations) and of these, NICE requested RCT evidence for three TAs (TA033, TA037 and TA111) and further research from unspecified sources in six TAs (TA065, TA072, TA093, TA193 [for both indications], TA244 and TA467).

There were seven re-appraised TAs that were originally recommended OwR (TA070, TA416, TA446, TA447, TA465, TA472 and TA483). For these, NICE mainly requested evidence from ongoing RCTs, SACT and/or Blueteq databases, with the exception of TA070, which made reference to a national registry, an audit and “good quality studies”; and TA446 which referred to a retrospective, non-interventional trial with data from SACT and Blueteq.

Three HSTs were subjected to NICE re-appraisal. Following HST1, for example, NICE requested national monitoring systems to record the number of people diagnosed; and for HST2, NICE specified that the marketing authorisation holder had to create a patient registry accessible to NHS England.

Evidence Considered in Re-appraisal Following Only in Research (OiR) Recommendation

There were some inconsistencies in what research NICE requested in OiR recommendations and what was subsequently considered during re-appraisal. While two TAs complied fully with NICE’s request for further evidence (TA033 → TA093 and TA288 → TA418), four were only partially compliant (TA072 → CG79, TA093 → CG131, TA111 → TA217 and TA244 → TA461).

RCT evidence was provided in three instances following NICE recommendations (TA033 → TA093, TA111 → TA217 and TA244 → TA461), although RCT data were considered in three other re-appraisals: TA072 → CG79, TA093 → CG131 and TA288 → TA418 (Table 2). However, while TA461 included evidence from two RCTs, there were no data on the comparison of interest that NICE specified in TA244. NICE specified audit data along with additional studies in their OiR recommendations for TA065, TA072 and TA093, but evidence from audits were not available in any re-appraisals. TA217 did not include evidence requested in TA111 of how the technology contributed to health-related quality of life.

Table 2.

Summary of the clinical and economic evidence (unreferenced evidence was extracted from NICE committee papers or final appraisal determinations) considered upon re-appraisal

TA (NICE recommendation [period]) → updated by (new recommendation after appraisal [period])	Clinical evidence		Cost-effectiveness evidence
	Randomised evidence	Non-randomised evidence
NICE technology appraisals with OiR recommendation
TA033 (OiR [2001/02]) → TA093 (remained OiR [2005/06])	[22] 439 patients randomised to raltitrexed or 5-fluororacil (5-FU) + leucovorin; HR for OS was 1.056 (95% CI 0.847–1.317; p = 0.44) [23] 495 patients randomised to raltitrexed or 5-FU + leucovorin median survival 10.9 vs 12.3 months respectively; HR 1.15 (95% CI 0.93–1.42; p = 0.197) [24] 905 patients randomised to 5-FU bolus, and infusion on days 1 and 2, repeated every 14 days, HRs for OS were 0.88 (95% CI 0.70–1.12; p = 0.17) for de Gramont versus Lokich, and 0.99 (0.79–1.25; p = 0.94) for de Gramont versus raltitrexed	The committee heard testimony from the clinical and patient experts who specified that the most important outcomes for people with advanced colorectal cancer are overall survival and quality of life	A US economic analysis of raltitrexed compared with 5-FU/FA reported an incremental cost per life-year gained of US$154,611 (£97,000 in GBP 1999 rates)
TA037 (OiR [2001/02]) → TA137 (recommended [2007/08])	No new evidence provided in response to the OiR recommendation
TA065 (OiR [2003/04]) → (withdrawn [2016/17])	Guidance was withdrawn because it was no longer relevant to clinical practice
TA072 (OiR [2003/04]) → CG79 (2008/09) → NG100 (remained OiR [2018/19])	[25] 309 patients received subcutaneous injections of anakinra (30, 75 or 150 mg) in a long-term extension trial. In those originally allocated to anakinra, the level of improvement was maintained for 48 weeks and the ACR20 response was 51% at week 24 and 46% at week 48. In those originally allocated to placebo, ACR20 response was 34% at week 24 and 51% at week 48 [26] 419 patients randomised to anakinra or placebo. Greater improvements in the Health Assessment Questionnaire disability index scores observed with anakinra at 12 weeks (1.0 mg: −0.35, p < 0.05; 2.0 mg: −0.39, p < 0.01) and 24 weeks (end of study) (1.0 mg: −0.37, p < 0.05; 2.0 mg: −0.51, p < 0.01) [27] 244 patients randomised to receive etanercept only, full-dose etanercept plus anakinra, or half-dose etanercept plus anakinra. Combination therapy with etanercept and anakinra did not add benefit (ACR50 of 41%, compared with 31% in the full-dosage etanercept plus anakinra group), but increased adverse event rate compared with etanercept alone [28] 230 patients received subcutaneous injections of anakinra (30, 75 or 150 mg). Significant reductions in radiographic joint damage were observed in the second 24-week period following anakinra 75 mg/day (p = 0.006) and 150 mg/day (p = 0.008)		The appraisal committee acknowledged that while health economic evidence was not formally reviewed, drug costs and clinical evidence had not changed enough for anakinra to become a cost-effective therapy
TA093 (OiR [2005/06]) → CG131 (remained OiR [2011/12])	[29] 1921 patients randomised to six cycles of 5-FU + leucovorin or eight cycles of raltitrexed. The trial closed prematurely when 17 (1.9%) raltitrexed-related deaths were reported [30] 94 patients with previously untreated metastatic advanced colorectal cancer randomised to raltitrexed followed by oxaliplatin (arm A), or irinotecan followed by raltitrexed (arm B). Overall response rate was 46% for arm A (95% CI 29.5–57.7), and 34% for arm B (95% CI 19.8–48.4)	[31] 91 patients were enrolled to evaluate the efficacy and toxicity of irinotecan in combination with raltitrexed as first-line treatment. The median time to progression was 11.1 months and the median OS was 15.6 months [32] 51 patients were enrolled to intravenous infusion of raltitrexed and oxaliplatin infusion. 47% of patients were still alive at the last follow-up contact, with a median OS of 15 months (range 8–28) [33] 45 patients received raltitrexed followed by oxaliplatin. Median time to progression was 4 months (range 1–12+) and median OS was 9 months (range 1–29+) [34] 44 patients received raltitrexed and oxaliplatin. Median time to disease progression was 6 months (95% CI 4.4–7.6) and OS was 14.8 months (95% CI 11.2–18.4) [35] 48 patients received irinotecan and raltitrexed. Median duration of response was 10 months, while median PFS and OS were 5 and 14 months, respectively [36] 62 patients received irinotecan plus raltitrexed. Median survival was 12.2 months, and median time to progression was 6.3 months [37] 31 patients received a triple combination of Camptosar, oxaliplatin and Tomudex. PFS was 7.3 months and overall median survival 16.6 months [38] 51 patients received raltitrexed plus oxaliplatin. Median time to progression was 18 weeks and median OS was 54.4 weeks	The committee determined that the clinical evidence was not sufficiently robust and there was no cost-effectiveness analysis. They also discussed that trials of raltitrexed in 5-FU/FA intolerant patients had not been conducted and are unlikely to happen, they were therefore concerned that the use of raltitrexed is being denied to a specific subgroup in which it is impossible to obtain direct evidence of effectiveness. Hence, the committee agreed to recommend raltitrexed for this subgroup of patients
TA111 (OiR [2006/07]) → TA217 (recommended in line with marketing authorisation for people with severe Alzheimer's disease [AD] and optimised for people with moderate AD [2010/11])	[39] 252 patients were assigned randomly to receive placebo or memantine. Imputed and observed mean differences in Clinicians’ Interview-Based Impression of Change Plus Caregiver Input (CIBIC-Plus) ratings favoured memantine at 0.3 (p = 0.06) and 0.3 (p = 0.03), respectively, at 28 weeks [40] 404 patients received memantine or placebo. Mean Severe Impairment Battery (SIB) scores (SE) were 0.9 (0.67) vs −2.5 (0.69), respectively (p < 0.001), at 24 weeks [41] 403 patients randomised to memantine or placebo. Primary outcome was change from baseline to 24 weeks on the Alzheimer's Disease Assessment Scale–Cognitive Subscale (ADAS-cog) with mean treatment difference of − 1.9 points (95% CI − 3.1 to − 0.6) favouring memantine [42] 350 patients were randomised to memantine or placebo. SIB change from baseline to week 24 did not show a statistically significant benefit for memantine (− 2.0 vs − 2.5, p = 0.616) [43] 433 participants randomised to memantine or placebo for 24 weeks. There were no statistically significant differences in primary (ADAS-cog, CIBIC-Plus) or secondary outcomes [44] 470 patients randomised to memantine or placebo. Efficacy was primarily assessed as change from baseline in ADAS-cog and CIBIC-plus scores. Differences were not statistically significant at week 24 (− 0.85 [− 2.02 to 0.32]; p = 0.156 and − 0.07 [− 0.30 to 0.15]; p = 0.523)	The committee considered submissions from clinical specialists and patient experts and accepted that memantine may contribute to HRQoL to patients and carers based on anecdotal evidence from patient experts Evidence from observational studies included references to grey literature sources, and the following publications: [45] A survey of 158 patients treated with memantine in combination with an acetylcholinesterase inhibitors (AChEI). Combination therapy was well tolerated for nearly all patients (98%) [46] 382 patients underwent serial clinical evaluations. Those treated with AChEI plus memantine had significantly lower mean annualised rates of deterioration in Blessed Dementia Scale (BDS) and Weintraub Activities of Daily Living Scale scores compared with AChEI alone at 1–2 years [47] 4600 patients randomly selected. There was a 39–50% increase in patients treated with psychotropic drugs before memantine initiation; this trend is reduced after taking memantine [48] 451 patients, free of cholinergic medication, received memantine. At 6–month assessment, 26.8% showed no deterioration and 3.8% showed improvement [49] 943 patients were examined to determine time to nursing home admission and death. Patients prescribed AChEI had a significant delay in admissions (HR 0.37, 95% CI 0.27–0.49); which was significantly augmented with the addition of memantine (HR 0.29, 95% CI 0.11–0.72)	The committee considered that the base case ICER for memantine of £32,100/QALY gained compared with best supportive care in moderate to severe Alzheimer’s disease was likely to be an overestimation In the Assessment Group’s exploratory analysis, memantine dominated best supportive care for the moderate group and was £26,500/QALY gained for the severe group compared with best supportive care. The committee acknowledged the explanation from the Assessment Group that the difference was due to assumptions and concluded that if behavioural effects were considered in the model, as noted by clinical specialists, the ICER would likely be lower
TA193 (OiR: 2 individual recommendations [2010/11]) → a panel reviewed the TA with no new evidence available (remained OiR [2013/14])	No new evidence
TA244 (OiR [2011/12]) → TA461 (optimised to adults with bronchitis if the disease is severe or if the person has had ≥ 2 exacerbations in the previous 12 months despite triple inhaled therapy [2017/18])	[50] 1935 patients randomised to roflumilast or placebo. The rate of moderate-to-severe COPD exacerbations was reduced (roflumilast 0·805 vs placebo 0·927; rate ratio [RR] 0.868 [95% CI 0.753–1.002], p = 0.0529) [51] 2352 patients with severe COPD randomised to roflumilast or placebo. Rate ratio for moderate or severe exacerbations was 0.92 (95% CI 0.81–1.04; p = 0.163) per patient per year	Only clinical experts advised the committee, they emphasised the importance to patients and carers of a new therapy that reduces exacerbations. They also explained how reducing exacerbations in COPD patients may represent a potential decrease in the burden for the NHS	ICERs varied between £16,293 and £30,349 per QALY gained. It was noted that the true ICER may be slightly lower because of the double counting of deaths highlighted by the ERG
TA288 (OiR [2013/14]) → TA418 (recommended [2016/17])	[52] 320 patients randomised to receive placebo or dapagliflozin plus saxagliptin and metformin. The difference in adjusted mean change from baseline in HbA1c between dapagliflozin and placebo as add-on therapy was −0.81% (95% CI −1.06 to −0.55) [53] 216 patients randomised to dapagliflozin or placebo. More patients achieved a therapeutic glycaemic response (HbA1c <7.0% [53 mmol/mol]) with dapagliflozin (31.8%) vs placebo (11.1%) (p < 0.0001)	Patient experts explained the medicine was easy to take and manage compared with injectable treatments. Testimony from clinical experts highlighted other benefits such as the reduction in blood pressure and weight loss	The company model predicted that dapagliflozin would dominate DPP-4 inhibitors, whereas the ERG base case for dapagliflozin was £37,997 per QALY gained. Neither base case had considered all the health-related QALY benefits of dapagliflozin, therefore it was considered that the higher ICER was an overestimate
NICE technology appraisals with OwR recommendation
TA070 (OwR [2003/04]) → TA241 (not recommended [2011/12])	[54] 150 patients with imatinib-resistant chronic-phase CML were randomised 2:1 to dasatinib or imatinib 800 mg. Treatment failure (HR 0.16; p < 0.001) and PFS (HR 0.14; p < 0.001) both favoured dasatinib [55] Phase II trial in which dasatinib demonstrated higher rates of complete hematologic response (93% vs 82%; p = 0.034), and the estimated PFS also favoured dasatinib (unstratified log-rank test; p = 0.0012)	[56] 54 patients received imatinib dose escalation and 20 patients received 800 mg. Complete cytogenetic response (CCyR) was achieved in 27 (37%) patients. The estimated 2-year PFS and OS were 87% and 85%, respectively [57] Phase IV study: 71 patients received imatinib. For evaluable patients, 30.8% of patients achieved CcyR at 6 months, and median time to treatment failure was 18.0 months [58] 90 patients with resistance to standard dose of imatinib underwent imatinib resistant mutation analysis and dose escalation of imatinib. The most common mutation was T315I; 35 (38%) patients responded to dose escalation and had 67% event-free survival at estimated 2 years	The committee concluded that high-dose imatinib was dominated in all models
TA416 (OwR [2016/17]) → TA653 (optimised to adults only if their disease has progressed after first-line treatment with an EGFR tyrosine kinase inhibitor [2020/21])	[59] 419 patients randomised to oral osimertinib or intravenous pemetrexed plus either carboplatin or cisplatin. The median duration of PFS was longer with osimertinib than with platinum therapy plus pemetrexed (10.1 months vs 4.4 months; HR 0.30; 95% CI 0.23–0.41; p < 0.001)	The committee considered clinical and patient experts’ opinions. Patient experts explained the technology may increase HRQoL, especially in the patients’ social life. In addition, clinical experts explained that patients on osimertinib have better clinical outcomes and a lower disease progression [60] 210 patients received osimertinib. At data cut off, 122 (58%) patients remained on treatment. The median duration of follow-up was 13·0 months (interquartile range [IQR] 7.6–14.2). 140 (70%; 95% CI 64–77) of 199 patients achieved an objective response 357 patients (CDF SACT data) received osimertinib. The median survival was 13.9 months (95% CI 12.1–17.6)	The most plausible ICER was estimated between £41,799 and £49,649 per QALY gained if the company’s commercial arrangement is considered
TA446 (OwR [2017/18]) → TA524 (optimised to adults with relapsed or refractory disease, only if they have already had autologous stem cell transplant or they have already had ≥2 previous therapies when autologous stem cell transplant or multi-agent chemotherapy are not suitable [2018/19])	–	The committee only considered clinical experts’ opinions, who explained the technology could potentially be curative in some patients, with or without stem cell transplant UK real-world retrospective observational study: 78 patients received brentuximab vedotin. PFS 5.68 months (95% CI 4.21–17.05), OS 37.2 months (95% CI 17.8 to not reached) [61] 60 patients received brentuximab vedotin. PFS 4.8 months (95% CI 2.96–5.32), OS 74% at 24 months (95% CI 58.0–84.6)	The committee considered an estimate between £16,535 and £17,885 per QALY gained
TA447 (OwR [2017/18]) → TA531 (recommended [2018/19])	[62] 305 patients received either pembrolizumab or investigators’ choice of platinum-based chemotherapy. Median PFS was 10.3 months (95% CI 6.7 to not reached) in the pembrolizumab group versus 6.0 months (95% CI 4.2–6.2) in the chemotherapy group (HR for disease progression or death 0.50; 95% CI 0.37–0.68; p < 0.001)	Clinical experts emphasised the need for well tolerated treatments that improve HRQoL of patients. Patient experts also mentioned the importance of improving the length of life	The committee concluded that all the scenarios presented were plausible, with ICERs ranging from £30,000 to <£50,000 per QALY gained
TA465 (OwR [2017/18]) → (withdrawn [2019–20])	[63] 509 patients randomised to doxorubicin plus olaratumab or placebo. Median OS was 20.37 months in the investigational arm and 19.75 months in the control arm (HR 1.047 [95% CI 0.841–1.303]; p = 0.69)
TA472 (OwR [2017/18]) → TA629 (recommended [2020/21])	[64] 413 patients randomised to obinutuzumab or bendamustine. After a median follow-up of 31.8 months, median PFS was 25.8 months for obinutuzumab and 14.1 months for bendamustine monotherapy (HR 0.57; 95% CI 0.44–0.73; p < 0.001)	The committee considered clinician and patient experts’ opinions. Clinical experts concentrated on the benefits for patients, including HRQoL and mental health	Using the ERG’s preferred random change-point models for both PFS and OS, and also with an updated price for bendamustine, the ICER was estimated at £15,045 per QALY gained
TA483 (OwR [2017/18]) → TA655 (optimised in adults after chemotherapy, only if it is stopped at 2 years of uninterrupted treatment, or earlier if their disease progresses, and they have not had a PD-1 or PD-L1 inhibitor before [2020/21])	[65] 272 patients with previously treated advanced or metastatic squamous non-small-cell lung cancer (NSCLC) randomised to nivolumab or docetaxel. Median OS was 9.2 vs 6.0 months (HR 0.59 [95% CI 0.44–0.79]; p < 0.001)	The committee considered a clinical expert submission and patients’ experiences; the latter focussed not only on their own benefits, but also on the extended family [66] 129 patients received nivolumab. 5-year OS rates were similar for squamous (16%) and non-squamous (15%). OS rates were 15% (95% CI 8–25) 348 patients who received nivolumab after chemotherapy. Median OS was 8.4 months (95% CI 7.2–9.7)	There were two estimates: the company’s preferred ICER for nivolumab compared with docetaxel alone for people with squamous NSCLC at £35,710 per QALY gained and using the committee’s preferred assumptions of 2 years of treatment and 3 years of continued benefit once treatment had stopped, likely to be below £40,168 per QALY gained
NICE Highly specialised technologies with OwR recommendation
HST2 (OwR [2014/15]) → HST19 (recommended [2022/23])	[67] MOR-004 176 participants were randomised to receive elosulfase alfa 2.0 mg/kg/every other week (qow), elosulfase alfa 2.0 mg/kg/week (weekly) or placebo for 24 weeks. The estimated mean effect on the 6MWT vs placebo was 22.5 m (95% CI 4.0–40.9; p = 0.017) for weekly and 0.5 m (95% CI −17.8 to 18.9; p = 0.954) for qow [68] MOR-005 in part 1, patients initially randomized to ERT in the 24-week pivotal study (MOR-004) remained on their regimen (2.0 mg/kg/week or qow); placebo patients were re-randomised to one of the two regimens. During part 2, all patients received elosulfase alfa 2.0 mg/kg/week. Mean increase in the modified per protocol population was 9.2% for forced vital capacity (FVC), 8.8% for forced expiratory volume in 1 s (FEV1), and 6.1% for maximal voluntary ventilation (MVV) after 120 weeks	Qualitative evidence from the Rare Disease Research Partners and the MPS Society [69] MOR-001 data collected from 325 subjects. Mean ± SD from the 6MWT was 212.6 ± 152.2 m, revealing limitations in functional endurance testing, and 30.0 ± 24.0 stairs/min for the 3-minute-stair-climb test [70] MAA data combined with data from MOR trials for 55 patients, including 26 patients previously enrolled in clinical trials and 29 who started ERT after enrolling in the MAA. In patients with both baseline and follow-up data, mean 6MWT distance increased from 217 m at baseline to 244 m after a mean follow-up of 4.9 years	The company's base-case results after technical engagement resulted in an ICER under £300,000 per QALY gained. However, the committee noted that taking its preferred assumptions into account, the ICER using the company's utility values was under £300,000 per QALY gained
HST3 (OwR [2014/15]) → HST22 (recommended [2022/23])		The company explained that it did not use data from the MAA in its economic model because the primary outcome measure used was the North Star Ambulatory Assessment (NSAA). The committee agreed to consider the data from STRIDE and CINRG for this review but also took into account the findings of the MAA MAA with matched controls from the NorthStar register: 59 patients received ataluren compared with 59 patients receiving best supportive care (BSC). Results on the document accessed were confidential but specifies the technology delays the age of ambulation and the majority of functional outcomes such as timed function tests (TFTs) and pulmonary outcomes [71] 213 male patients from the STRIDE registry combined with 400 patients from the CINRG study to capture the natural history of the disease in people with Duchenne muscular dystrophy (DMD) who had best standard of care (SoC). Kaplan–Meier analyses demonstrated that ataluren + SoC significantly delayed age at loss of ambulation and age at worsening performance in timed function tests vs SoC alone (p ≤ 0.05)	Using the committee-preferred assumptions, and a new commercial arrangement submitted for the second committee meeting, the ICER estimate was below £100,000 per QALY gained, although the exact ICER was considered confidential by the company and was not reported
HST6 (OwR [2014/15]) → HST23 (optimised [2022/23])	[72] ENB-009-10 19 people of 13 years to 65 years, with hypophosphatasia (18 of 19 people had paediatric-onset hypophosphatasia) followed for up to 5 years. Change from baseline to week 24 for plasma pyridoxal-5 phosphate (PLP) mean (SD) combined −397.72 (455.249)	[73] ENB-003-08 11 people of 36 months and under with infantile-onset hypophosphatasia, with an extension study (ENB-003-08) that followed 10 people for up to 7 years. Skeletal healing was sustained over 7 years of treatment; all evaluable 14 patients had RGI-C scores ≥+2 at years 6 and 7 [74] ENB-010-10 was a non-randomised, dose-comparison study of asfotase alfa treatment in 69 people of 5 years and under, with perinatal- or infantile-onset hypophosphatasia followed for up to 6 years. During median (minimum, maximum) 2.3 (0.02, 5.8) years of treatment, RGI-C scores improved significantly at month 6 (12.0 [21.7, 13.0]), year 1 (12.0 [22.3, 13.0]), and last assessment (12.3 [22.7, 13.0]; p = 0.0001 all) [75] ENB-006-09 13 people of 5–12 years with paediatric-onset hypophosphatasia, with an extension study ENB-008-10 that followed 12 people for up to 5 years. Skeletal radiographic changes improvement occurred by 6 months and persisted through 5 years (+2.2 [+1.7, +2.7]; p = 0.0005) Real-world data was collected under the MAA (UK MAA data) in people with paediatric-onset hypophosphatasia, regardless of current age, who had Asfotase alfa. Data-cut for the analysis was confidential, but in the committee papers it is claimed that treatment reduces mortality, reduces respiratory support, helps with growth and helps motor function Real-world data from the ongoing Global Hypophosphatasia Registry and data from a prospective, longitudinal telephone-based survey, which was presented in confidence and were referenced as attachments provided on files to NICE The Evaluate and Monitor Physical Performance of Adults Treated with Asfotase Alfa for Hypophosphatasia (EmPATHY): 21 adult patients with paediatric-onset hypophosphatasia contacted for data collection at a single centre in Germany. Mobility improvements in median distance walked was 267.0 metres (interquartile range [IQR]: 0, 368.0), which increased to 320.0 metres (IQR: 234.0, 469.0) after 12 months of treatment	The committee's base-case results at the second evaluation committee meeting resulted in a probabilistic ICER of £98,276 per QALY gained in juvenile-onset hypophosphatasia with the QALY weighting applied

6MWT 6-min walk test, CI confidence interval, CDF Cancer Drug Fund, CML chronic myeloid leukaemia, COPD chronic obstructive pulmonary disease, DSM-IV Diagnostic and Statistical Manual of Mental Disorders, 4th edition, ERG Evidence Review Group, HR hazard ratio, HRQoL health-related quality of life, ICER incremental cost-effectiveness ratio, MAA managed access agreement, NHS National Health Service, NINCDS-ADRDA the National Institute of Neurological and Communicative Diseases and Stroke/Alzheimer’s Disease and Related Disorders Association, OiR only in research, OS overall survival, OwR only with research, PFS progression-free survival, SACT Systemic Anti-Cancer Therapy, TA technology appraisal, ERT Enzyme Replacement Therapy

No new evidence was presented in three re-appraisals (TA037 → TA137, TA065 and TA193). NICE commented in TA137 that it would be difficult to gather relevant information for the OiR indication in question because the medicine was approved as a third-line treatment. NICE decided TA065 was no longer relevant to clinical practice; and a review of TA193 did not find any new evidence.

Evidence Considered in Re-appraisal Following Only with Research (OwR) Recommendation

Of the seven TAs identified as OwR, four complied with NICE’s request for further evidence on re-appraisal (TA416 → TA653, TA446 → TA524, TA465 and TA483 → TA655) and three TAs complied partially (TA070 → TA241, TA447 → TA531 and TA472 → TA629). TA241 did not present the audit data requested and, while TA531 and TA629 included SACT data, this was immature or not considered by the committee to be sufficiently robust.

RCT evidence was available on re-appraisal for five TAs for which NICE requested such data following OwR recommendations (TA416 → TA653, TA447 → TA531, TA465, TA472 → TA629 and TA483 → TA655) (Table 2). TA653 included a survival analysis based on SACT data relating to CDF patients, in addition to the evidence from the two studies requested in TA416. For TA465, NICE requested data from an ongoing trial and from SACT, but the results from the trial indicated no survival advantage, and the TA was withdrawn. TA655 presented a combination of data from an RCT, follow-up data from a previous study and a cohort study.

NICE’s re-appraisal of TA070 in TA241 included RCT evidence in addition to non-randomised studies, but no audit. The re-appraisal of TA446 in TA524 provided the evidence requested and, in addition, a single-arm, open-label, multicentre study (Table 3).

Table 3.

Risk of bias assessment for randomised evidence in updated TAs and HSTs

TA (NICE recommendation) → updated by	Randomised evidence	Domain 1: selection bias		Domain 2: performance bias	Domain 3: detection bias	Domain 4: attrition bias	Domain 5: reporting bias	Domain 6: Other bias	Overall judgment for analysis
		Random sequence generation	Allocation concealment	Blinding of participants and researchers	Blinding of outcome assessment	Incomplete outcome data	Selective reporting
NICE technology appraisals with OiR recommendation
TA033 (OiR) → TA093	Cunningham et al. [22]	?	?	?	?	+	+	+	Unclear
	Cocconi et al. [23]	+	?	−	−	+	+	+	High
	Maughan et al. [24]	+	+	?	+	+	+	−	High
TA072 (OiR) → CG79 → NG100	Nuki et al. [25]	?	?	+	+	+	+	+	Unclear
	Cohen et al. [26]	?	?	+	+	+	+	+	Unclear
	Genovese et al. [27]	+	?	+	+	+	+	+	Unclear
	Bresnihan et al. [28]	?	?	+	+	+	+	+	Unclear
TA093 (OiR) → CG131	Popov et al. [29]	+	+	?	?	+	+	?	Unclear
	Feliu et al. [30]	?	?	?	?	+	+	+	Unclear
TA111 (OiR) → TA217	Reisberg et al. [39]	+	+	+	+	−	+	−	High
	Tariot et al. [40]	+	+	+	+	−	+	−	High
	Peskind et al. [41]	+	+	+	+	+	+	+	Low
	Dyck et al. [42]	+	+	+	+	+	+	+	Low
	Porsteinsson et al. [43]	+	+	+	+	+	+	+	Low
	Bakchine et al. [44]	+	+	+	+	+	+	+	Low
TA244 (OiR) → TA461	Martínez et al. (REACT) [50]	+	+	+	+	+	+	+	Low
	Martínez et al. (RE2SPOND) [51]	+	+	+	+	+	+	+	Low
TA288 (OiR) → TA418	Mathieu et al. (Study 5) [52]	+	+	+	+	+	+	?	Unclear
	Matthaei et al. 2015 [53]	+	+	+	+	+	+	+	Low
NICE technology appraisals with OwR recommendation
TA070 (OwR) → TA241	Kantarjian et al. [54]	?	?	−	−	−	+	−	High
	Kantarjian et al. [55]	?	?	−	−	−	+	−	High
TA416 (OwR) → TA653	Mok et al. (Aura3) [59]	+	?	−	−	+	+	−	High
TA447 (OwR) → TA531	Reck et al. (KEYNOTE-024) [62]	+	?	−	+	−	+	+	High
TA465 (OwR)	Tap et al (ANNOUNCE) [63]	+	+	+	+	+	+	+	Low
TA472 (OwR) → TA629	Cheson et al. (GANDOLIN) [64]	+	?	−	−	+	+	+	High
TA483 (OwR) → TA655	Brahmer et al. (Checkmate 017) [65]	+	?	−	−	+	+	+	High
NICE highly specialised technologies with OwR recommendation
HST2 (OwR) → HST19	MOR-004 [67]	?	?	+	+	+	+	+	Unclear
	MOR-005 [68]	−	?	−	−	?	+	−	High
HST6 (OwR) → HST23	ENB-009-10 [72]	−	?	−	−	+	−	+	High

CG clinical guidance, HST highly specialised technology, NG NICE Guidance, OiR only in research, OwR only with research, TA technology appraisal

Key: + Low risk, − High risk, ? Unclear risk

All re-appraised HSTs complied with NICE’s requested of further evidence. In HST6 → HST23, however, while the company presented descriptive MAA data on clinical outcomes, these were not initially analysed in accordance with NICE’s request (a comparison with registry data), on the basis that there were major baseline differences between populations. A subsequent analysis of the registry data, which compared treatment naïve and experienced patients, was redacted.

Risks of Bias in Evidence Presented in Re-appraisals

Among technology re-appraisals where new data were considered, TA461 and TA465 included evidence that was all judged to be of low risk of bias, two TAs (TA217 and TA418) included at least some evidence that was of low risk of bias and 11 TAs (TA093, TA217, TA241, TA418, TA524, TA531, TA629, TA653 and TA655, including evidence from CG79, CG131) contained evidence with a critical, serious, high, moderate or unclear risk of bias.

Randomised Evidence

Based on the Cochrane risk of bias tool, randomised studies were susceptible mostly to biases due to a lack of blinding of participants and researchers, or blinding of outcome assessment (see Fig. 2).

Fig. 2.

Risk of bias from (A) randomised controlled trials and (B) non-randomised studies, that were considered during re-appraisal of Technology Appraisals (TAs) and Highly Specialised Technologies (HSTs) with Only in Research (OiR) or Only with Research (OwR) conditional approvals

Eight RCTs reported in four TAs (three TAs previously recommend OiR and one TA previously recommended as OwR) were categorised as having low risk of bias in all domains: TA217 [41–44], TA418 [53], TA461 [50, 51] and TA465 [63].

Eight RCTs from four TAs (all previously recommended OiR) had at least one domain categorised as unclear: TA093 [22], CG79 [25–28], CG131 [29, 30] and TA418 [52]. Selection, performance and detection bias were identified as the three main sources of risk of bias that led to this classification.

Ten RCTs from seven TAs (two previously recommended OiR and five previously recommended OwR) were categorised in at least one domain as a high risk of bias: TA093 [23, 24], TA241 [54, 55], TA217 [39, 40], TA531 [62], TA629 [64], TA653 [59] and TA655 [65]. Performance, detection and other bias were identified as the domains that contributed the most to this categorisation.

Evidence presented in HSTs was categorised as unclear or high risk. For instance, trial MOR-004 [67] in HST2 had most domains as low risk of bias, but random sequence generation and allocation concealment were unclear. Trials MOR-05 [68] in HST2 and ENB-009-10 [72] in HST6 had more than two domains classified as high risk of bias.

Non-randomised Evidence

There were six re-appraised TAs with evidence from 22 non-randomised studies—primarily single-arm studies, with the exception of Atri et al. [46] and Lopez et al. [49] in TA217. The evidence came from a combination of studies, including, for example, a case series [56], a phase II single-arm study [60], three phase IV or post-marketing surveillance studies that used patient-reported outcomes [45, 48, 57], a real-world UK observational study (TA524) and evidence from SACT data (TA653 and TA655) (Table 4).

Table 4.

Risk of bias assessment for non-randomised evidence in updated technology appraisals

TA (NICE recommendation] → updated by	Non-randomised evidence	Pre-intervention		At intervention	Post-intervention				Overall judgement
		Domain 1	Domain 2	Domain 3	Domain 4	Domain 5	Domain 6	Domain 7
		Bias due to confounding	Bias in selection of participants into the study	Bias in classification of interventions	Bias due to deviations from intended interventions	Bias due to missing data	Bias in measurement of outcomes	Bias in selection of the reported result
NICE technology appraisals with OiR recommendation
TA093 (OiR) → CG131	Feliu et al. [31]	M	M	L	L	L	M	M	Moderate
	Cortinovis et al. [32]	L	S	L	L	M	M	S	Serious
	Laudani et al. [33]	L	M	L	L	L	M	L	Moderate
	Santini et al. [34]	L	M	L	L	M	M	L	Moderate
	Chiara et al. [35]	L	M	S	M	L	M	L	Serious
	Aparicio et al. [36]	L	S	L	L	C	M	S	Critical
	Maroun et al. [37]	L	M	L	L	S	L	M	Serious
	Vyzula et al. [38]	L	M	L	L	M	L	M	Moderate
TA111 (OiR) → TA217	Hartmann et al. [45]	S	S	L	L	M	S	L	Serious
	Atri et al. [46]	L	M	S	M	S	M	M	Serious
	Vidal et al. [47]	M	S	M	L	M	M	L	Serious
	Clerici et al. [48]	L	S	M	M	S	M	M	Serious
	Lopez et al. [49]	L	S	L	M	L	M	L	Serious
NICE technology appraisals with OwR recommendation
TA070 (OwR) → TA241	Breccia et al. [56]	S	S	S	M	L	M	M	Serious
	Koh et al. [57]	S	S	M	M	S	M	M	Serious
	Rajappa et al. [58]	S	M	S	M	L	M	M	Serious
TA416 (OwR) → TA653	Goss et al. AURA2 [60]	L	M	M	L	L	L	L	Moderate
	CDF SACT data	L	M	M	L	M	M	L	Moderate
TA446 (OwR) → TA524	Real-world UK observational study	S	L	L	L	L	S	M	Serious
	Walewski et al. (C25007) [61]	L	M	L	L	M	L	L	Moderate
TA483 (OwR) → TA655	Gettinger et al. Checkmate 003 [66]	L	S	M	L	L	L	L	Serious
	CDF SACT data	L	M	M	L	M	M	L	Moderate
NICE highly specialised technologies with OwR recommendation
HST2 (OwR) → HST19	MAA data combined with MOR trials data [70]	S	M	S	NI	S	M	L	Serious
HST3 (OwR) → HST22	Data from the STRIDE study, combined with the CINRG study [71]	M	L	NI	M	L	S	L	Serious
	MAA data matched controls from the NorthStar	S	S	M	M	NI	M	M	Serious
HST6 (OwR) → HST23	ENB-003-08 [73]	M	L	M	L	L	L	M	Moderate
	ENB-010-10 [74]	M	M	M	M	S	S	L	Serious
	ENB-006-09 [75]	M	S	S	M	S	L	M	Serious

C critical risk of bias, L low risk of bias, M moderate risk of bias, NI no information, S serious risk of bias

CDF Cancer Drugs Fund, CG clinical guidance, HST highly specialised technology, MAA managed access agreement, NICE National Institute for Health and Care Excellence, OiR only in research, OwR only with research, SACT Systemic Anti-Cancer Therapy, TA technology appraisal

Based on the ROBINS-I tool, the non-randomised studies were susceptible mostly to biases due to the selection of participants and to confounding (Fig. 2). One study (Aparicio et al. [36]) presented in CG131 (previously recommended OiR) had one domain classified as a critical risk of bias. Thirteen studies from five TAs (two previously recommended OiR and three OwR) presented evidence categorised as high risk of bias in at least one domain: TA241 [56–58], CG131 [31, 32, 35, 37], TA217 [45–49], TA524 (real-world observational study) and TA655 [66]. All the other evidence was categorised as a moderate risk of bias (one TA previously recommended OiR and three TAs OwR).

Evidence submitted in re-appraised HSTs were also categorised as moderate or serious risk of bias. All evidence but one (study ENB-003-08 [73] in HST6) was considered to have serious risk of bias in more than one domain.

NICE Recommendations Following Re-appraisal

There were nine TAs with OiR recommendation, of which three (TA037 → TA137, TA111 → TA217 and TA288 → TA418) were recommended in full for at least one indication. Two TAs were optimised (TA111 → TA217 and TA244 → TA461), four remained recommended OiR (TA033 → TA093, TA072 → CG79, TA093 → CG131 and TA193) and TA065 was withdrawn. Of the seven TAs with OwR recommendation, two TAs were subsequently recommended in full (TA472 → TA629 and TA447 → TA531), three TAs were optimised (TA416 → TA653, TA446 → TA524 and TA483 → TA655), one TA was not recommended (TA070 → TA241) and TA465 was withdrawn.

Of the six TAs that complied fully with NICE’s recommendations for further research (either OiR or OwR), one (TA418) was recommended in line with marketing authorisation, three TAs (TA524, TA653 and TA655) were recommended optimised, TA093 remained OiR and TA465 was withdrawn. Among the seven TAs that complied partially with NICE’s recommendations, TA531 and TA629 were recommended in line with marketing authorisation and TA461 was recommended optimised. TA217 received full recommendation for patients with severe disease but optimised for those with moderate disease. CG79 and CG131 remained OiR and TA241 was not recommended. Three TAs did not provide new evidence, but of those, TA137 was recommended, TA193 remained OiR for both indications and TA065 was withdrawn.

All HSTs were recommended after re-appraisal with HST19 and HST2 recommended according to marketing authorisation, while HST23 was recommended optimised.

Discussion

Statement of Principal Findings

Over the period of analysis, NICE conditionally recommended more medicines in technology appraisals OwR (41) than OiR (12). However, more TAs with OiR recommendations were subsequently re-appraised (9) than OwR recommendations (7). Upon re-appraisal, only six TAs (two originally with OiR and four with OwR recommendations) complied fully with NICE’s request for further evidence; three TAs did not comply.

The majority of re-appraised TAs included evidence that was deemed to be at serious, high, moderate or unclear risk of bias, with Aparicio et al. [36] in CG131 categorised as critical. Only eight randomised studies from four appraisals were supported by evidence that was judged to have low risk of bias in all domains. Reliance on observational studies and audits was commonplace among TAs that were subsequently re-appraised. However, the majority of these studies were judged as having high or critical risk of bias. The increasing use of SACT and Blueteq for real-world evidence of benefit from cancer and high-cost medicines is fraught with issues resulting from missing data and confounding, and which could be mitigated through well-designed RCTs.

All five conditionally approved HSTs were recommended OwR. Of these, three were re-appraised and were compliant with NICE’s request of further information. However, all the evidence submitted was categorised as unclear, moderate or high/serious risk of bias.

Overall, the quality of evidence submitted in response to OiR and OwR recommendations was poor.

Comparison with Other Studies

To our best knowledge, this is the first systematic study of the quality and biasedness of evidence resulting from NICE OiR and OwR recommendations. Other researchers, however, have (1) highlighted the challenges for health technology assessment in the face of higher clinical uncertainty relating to conditional marketing authorisation pathways [76]; (2) investigated the use of real-world data within single technology appraisals of cancer medicines [77] or as part of MAA associated with CDF medicines [12, 13] and (3) compared real-world data relating to targeted and non-targeted cancer therapies [14].

Strengths and Limitations

This review provides a comprehensive assessment of the nature and quality of evidence requested and subsequently evaluated by NICE following conditional recommendation. It benefits from the systematic application of two widely utilised tools for the determination of the risks of bias in randomised and observational data. Our findings should be of value in informing the application of the NICE real-world evidence framework [78].

One limitation of this review has been our reliance on the availability of documents on NICE’s website. Some superseded documents have been removed from the website, while many documents are highlighted as being commercially sensitive with text redacted [79]. The lack of transparency in aspects of the CDF has been noted previously [80], although some historical documents could be retrieved from UK web archives [81, 82]. Additionally, as NICE does not explicitly label OwR recommendations as such, assumptions were necessary in relation to those TAs for medicines conditionally recommended via processes other than OiR [8, 83–85]. A third limitation related to the application of tools used to assess the risk of bias being restricted to the information publicly available for those reviewed studies. This was compounded by analyses of SACT and Blueteq data not being available as publications in peer-reviewed journals. Finally, as our study was limited to NICE, its findings may not be fully generalisable to other jurisdictions.

Policy and Research Implications

There are four important policy implications to our research. Firstly, while OwR recommendations were more likely to result in re-appraisal, they rely heavily on data from potentially biased studies. The increasing use of non-randomised studies for providing evidence on clinical effectiveness is problematic, not least as confounding, selection and information bias can undermine the reliability of such data. The cost of incorrect decisions upon re-appraisal can be significant—there is a fallacy to expediting wider access to treatment through reliance on observational data. The limitations of non-randomised data are acknowledged in the NICE real-world evidence framework, which has the specific purpose and aim of improving the quality of real-world evidence to inform NICE’s guidance.

Secondly, non-compliance with NICE-requested evidence generation is widespread. The reasons for this were not apparent in most instances; however, it is concerning both that manufacturers do not always provide the evidence requested and that NICE recommendations are seemingly unaffected by this.

Thirdly, NICE has resisted using value of information methods to help select the specific design and sample size of a proposed study, as well as the value of OiR versus OwR [86]. Instead, NICE assesses the nature and methods of evidence generation (in respect to MAA) according to whether data collection and analyses are feasible in a reasonable timeframe, do not represent an unreasonable burden on patients and the NHS and are likely to support the case for a positive recommendation upon re-appraisal [87]. While conceptually attractive, value of information analyses require detailed specification of the decision problem and quantification of the associated uncertainty. This typically entails a level of complexity that may obfuscate, and a requirement for data that may not be reliably available—although a valid outcome from such an analysis would be evidence to inform the choice of whether a recommendation ought to be OiR or OwR.

Fourthly is the timeliness of data. Others have commented on the delays and costs in acquiring evidence following OiR recommendations—such as in the extreme example of treatments for multiple sclerosis [88]. Our review found evidence to be either immature or had been superseded by RCTs in some re-appraised TAs following OwR recommendations. While the duration of data collection is agreed within data collection agreements, it is well recognised that some medicines with MAA remain on the CDF for extended periods of time [80].

Conclusions

This review highlights important variation in the quality of evidence submitted in response to NICE conditional approval (OiR and OwR) recommendations. It further identifies non-compliance with agreed data requests and discusses the implications in respect to pharmaceutical policy. The increased reliance on real-world evidence raises concerns about the risks of incorrect decisions being made based on inaccurate data. This may be mitigated through more careful considerations of the limitations of observational evidence, and potentially pre-empted by conducting value of information analyses that could inform the nature and choice of additional data collection methods. As a minimum, quality standards ought to be stipulated in respect to evidence contributing to NICE re-appraisals following conditional approval recommendations.

Supplementary Information

Below is the link to the electronic supplementary material.

Declarations

Conflict of interest

This research was supported by the MRC-NIHR Trials Methodology Research Partnership (MR/S014357/1) via a Bangor University hosted PhD studentship. DAH is a Health and Care Research Wales Senior Research Leader (SRL 2022-25-20). Dyfrig Hughes is an Editorial Board Member of PharmacoEconomics. He was not involved in the selection of peer reviewers for the manuscript nor any of the subsequent editorial decisions.

Data availability

The datasets analysed during the current study are available via https://www.nice.org.uk/.

Author contributions

Dyfrig Hughes conceived the study, secured the funding and provided supervision. Dyfrig Hughes and Yankier Pijeira Perez contributed to the study design. Data collection and analysis were performed by Yankier Pijeira Perez. The first draft of the manuscript was written by Yankier Pijeira Perez. Dyfrig Hughes contributed to subsequent drafts, and both authors approved the final manuscript.