Treatment Effect Waning in Immuno-oncology Health Technology Assessments: A Review of Assumptions and Supporting Evidence with Proposals to Guide Modelling

Abstract

Treatment effect waning (TEW) refers to the attenuation of treatment effects over time. Assumptions of a sustained immuno-oncologic treatment effect have been a source of contention in health technology assessment (HTA). We review how TEW has been addressed in HTA and in the wider scientific literature. We analysed company submissions to English language HTA agencies and summarised methods and assumptions used. We subsequently reviewed TEW-related work in the ISPOR Scientific Presentations Database and conducted a targeted literature review (TLR) for evidence of the maintenance of immuno-oncology (IO) treatment effects post-treatment discontinuation. We found no standardised approach adopted by companies in submissions to HTA agencies, with immediate TEW most used in scenario analyses. Independently fitted survival models do however suggest TEW may often be implicitly modelled. Materials in the ISPOR scientific database suggest gradual TEW is more plausible than immediate TEW. The TLR uncovered evidence of durable survival in patients treated with IOs but no evidence that directly addresses the presence or absence of TEW. Our HTA review shows the need for a consistent and appropriate implementation of TEW in oncology appraisals. However, the TLR highlights the absence of direct evidence on TEW in literature, as TEW is defined in terms of relative treatment effects—not absolute survival. We propose a sequence of steps for analysts to use when assessing whether a TEW scenario is necessary and appropriate to present in appraisals of IOs.

Supplementary Information

The online version contains supplementary material available at 10.1007/s40273-024-01423-6.

Key Points for Decision Makers

Treatment effect waning (TEW) signifies the attenuation of treatment effects over time. Assumptions regarding the maintenance of treatment effects associated with immuno-oncology therapies (IOs) following treatment discontinuation have been a source of contention in health technology assessment (HTA).

Our findings highlight an absence of external evidence to inform assumptions of a waning of effect for IO submissions to HTA agencies.

We propose a sequence of steps for analysts to use when assessing whether a TEW scenario is necessary and appropriate to present in appraisals of IOs.

Introduction

Treatment effect waning (TEW) refers to a phenomenon where the effects of a treatment attenuate over time. This means that while a treatment may show effects (e.g. a relative improvement in progression-free survival [PFS]) in the short-term, the benefits may not be sustained in the long-term.

Immuno-oncology therapies (IOs) are a type of biological therapy used to treat cancer [1]. It has been suggested that due to the mechanism of action in modification of the immune system, IOs may provide a durable treatment effect for some patients long after treatment has been discontinued [2]. Explicit assumptions around the maintenance over time of the treatment effect associated with IOs have been a source of contention in health technology assessment (HTA). Given that trial data around the time of evidence submission to HTA agencies are relatively short-term and the treatment is new, there is seldom direct evidence to help predict long-term survival [3].

In the absence of external evidence, a long-term protective treatment effect (even after treatment discontinuation) is a strong assumption when extrapolating beyond clinical trial follow-up. This assumption becomes more intricate due to the presence of treatment stopping rules which are at times in place from trial protocols, and payer agreements. These rules involve the treatment(s) being discontinued at a pre-specified time point (e.g., 2 years) [4]. In this case, a key consideration surrounds whether the treatment effect will be maintained after treatment has been discontinued in patients who have not experienced disease progression while remaining on treatment.

TEW has commonly been incorporated as a scenario analysis for economic evaluations by HTA agencies such as the National Institute for Health and Care Excellence (NICE), commonly by equalling hazards to a control arm at a specified time point (referred to as immediate TEW) [5]. The reason being if a treatment’s clinical benefit decreases over time, its cost-effectiveness will likely worsen, making it less attractive as a treatment option. Previous work that reviewed TEW methods included in NICE technology appraisals (TA)s of IO therapies where a treatment stopping rule was applied found that the application of TEW assumptions varied considerably [4, 5]. Other work investigating differences in decision making across NICE HTAs of nivolumab uncovered inconsistencies in assumptions made by companies, external assessment groups (EAGs), and NICE appraisal committees (AC)s around TEW [6].

The aim of this paper is to understand how TEW has been addressed in HTA. Therefore, we firstly provide an overview of the methods employed in HTA to address the potential for TEW. To accomplish this, we conducted a search of NICE TAs, categorising and summarising the utilised methods. We then reviewed HTAs undertaken by other English language agencies to identify whether different TEW assumptions are made by different agencies. Furthermore, we review work related to TEW that is included in the International Society for Pharmacoeconomics and Outcomes Research (ISPOR) Scientific Presentations Database, which could capture a range of reviews, methods and experiences from HTA submissions. After this, we performed a targeted literature review (TLR) of the scientific literature to identify evidence on the maintenance of IO treatment effects, with a specific focus on effects beyond treatment discontinuation. The aim here was to investigate whether the techniques identified for addressing TEW in HTA could be rationalised based on the available evidence on clinical effectiveness.

Drawing upon our findings, we propose a sequence of steps for analysts to use when assessing whether it is necessary, and appropriate, to present TEW scenario analyses in HTA submissions of IOs.

Methods

We reviewed information from a range of different, complementary data sources. We describe the methods used for each of these reviews below.

Review of TEW in HTA

Technology Appraisal Review

We aimed to identify how the potential for TEW has been addressed in NICE appraisals of IOs that included treatment stopping rules. We conducted a review of NICE appraisals by searching the NICE website (https://www.nice.org.uk) to identify published NICE appraisal documents for IO therapies across different oncology indications until September 2022. Inclusion criteria for appraisals required both the presence of a treatment stopping rule in the clinical trial and specific assumptions around TEW.

We reviewed NICE appraisal documents including company submissions, EAG reports, appraisal consultation documents and final appraisal determinations. We extracted information on each TA including: NICE’s recommendation, disease indication, comparators, pivotal trial name, study design, sample size, follow-up duration, stopping rules, how the treatment effect on survival was modelled, details on TEW (from the company, EAG and committee), any changes in the incremental cost-effectiveness ratio (ICER) with TEW scenarios and whether TEW assumptions were based on any evidence.

In addition, we searched the Scottish Medicines Consortium (SMC), the All Wales Medicines Strategy Group (AWMSG), the Canadian Agency for Drugs and Technologies in Health (CADTH), the Pharmaceutical Benefits Advisory Committee (PBAC, Australia), Pharmac (New Zealand) and the Institute for Clinical and Economic Review (Inst-CER, USA). Further details on the methods used for the TA review are provided in Supplementary Materials (Supplementary Methods).

ISPOR Scientific Presentations Database Review

Here, we aim to identify previous research that had been undertaken that explored TEW in HTA but which had not been published in peer-reviewed journals (and therefore would not be identified in the TLR described below). We developed search terms for the ISPOR scientific presentations database (https://www.ispor.org/heor-resources/presentations-database/search). We included relevant presentations with an accompanying poster published up to (and including) ISPOR Europe 2022. Further details, including the terms used to search titles and abstracts are provided in Supplementary Materials (Supplementary Methods).

Targeted Literature Review of Clinical Evidence

The aim of the TLR was to identify evidence on the treatment effects associated with IOs beyond treatment discontinuation. We searched PubMed on 19 April 2023 to identify papers that specifically analysed or reviewed treatment effects post treatment discontinuation for IO treatments. The searches were conducted limiting articles from the year 2010, which marks the year when the first evidence was published regarding the effectiveness of immunotherapy in oncology, subsequently licensed in 2011 [7]. Articles were excluded that did not describe an assessment of treatment effects or survival beyond treatment discontinuation, i.e. analysing whether survival benefits were maintained after treatment discontinuation.

Further details, including the terms used to search titles and abstracts, are provided in Supplementary Materials (Supplementary Methods).

Results

TEW in HTA

Technology Appraisal Review

In total, 59 NICE TAs in IO indications were identified. Of these, 34 included discussions of TEW. After screening for IO agent clinical trial stopping rules, 18 NICE TAs were included, with publication dates between January 2017 and September 2022. A total of, 13 were recommended by NICE, 3 were not recommended and 2 were entered into the Cancer Drugs Fund (CDF). The 18 NICE TAs are summarised in Table 1.

Table 1.

NICE technology appraisals included and assessments of treatment effect waning assumptions

TA number date and decision	Treatment and disease indication	Pivotal trial name and design	Follow-up durationa	Stopping rule	Survival modelb	TEW—base casec	TEW—scenarios	TEW—ERG	TEW—AC	TEW—ICER
TA428 [9] January 2017 Recommended	Pembrolizumab Lung, NSCLC, 2L+	KEYNOTE-010 Phase III RCT	13 months 24 months	2 years	Independent	None (assumed lifetime treatment effect^)	Immediate. HR for OS set to 1 at 3, 5 and 10 years from start of treatment	HR for OS set to 1 at 3 years	Could not agree on a single scenario	None, £47,844 3 years, £66,707 5 years, £54,629 10 years, £48,503
TA531 [23] July 2018 Recommended	Pembrolizumab Lung, NSCLC, 1L	KEYNOTE-024 Phase III RCT	25.2 months 33 months	2 years	Pooled parametric tested. Independent chosen	None (assumed lifetime treatment effect^)	Immediate. HR for OS set to 1 at 3 and 5 years from start of treatment	No comments	Duration of treatment effect uncertain. Scenarios considered in decision	None, £30,244 3 years, £44,483 5 years, £36,156
TA578/TA798 (ref. [12]) June 2022 CDF	Durvalumab Lung, NSCLC, 2L+	PACIFIC Phase III RCT	14.5 months 24 months	1 year	Independent	None (assumed lifetime treatment effect^)	Immediate. Equal OS hazard after 7.5 and 10 years from start of treatment	Equal hazard after 3 and 5 years preferred scenarios	Equal hazards after 3 and 5 years appropriate for decision making	None, £11,719 3 years, £22,581 5 years, £22,441 10 years, £12,375
TA589 (ref. [25]) July 2019 Recommended	Blinatumomab Blood & bone marrow, 2L+	BLAST Phase II single arm	18-month follow-up period	< 1 year (up to 2 cycles)	Independent	Equal hazard after 11 years	Immediate. Duration of benefits set to 60 months, including those in the relapse-free survival group	No new scenarios explored	AC preferred fixed cure point was 60 months	Base case, £18,818 60 months, £25,034
TA650 (ref. [10]) September 2020 Not recommended	Pembrolizumab Renal cell carcinoma, 1L	KEYNOTE-426 Phase III RCT	17.4 months 27 months	2 year	Independent	None (assumed lifetime treatment effect^)	Immediate. Equal hazard after 10 years from start of treatment for both OS and PFS & gradual TEW between 5 and 10 years based on response	Equal hazard after 5 and 10 years. Re-ran gradual analyses for all (not just responders)	Not enough evidence for lifetime treatment effect^. TEW after 5 years from start of treatment most plausible	None, £59,292 5 years, £133,900 10 years, £86,712 ERG: None, £120,455 5 years, £162,424 10 years, £123,368
TA661 (ref. [14]) November 2020 Recommended	Pembrolizumab Head and neck, 1L	KEYNOTE-048 Phase III RCT	NR	2 year	Independent	None (assumed lifetime treatment effect^)	Immediate. Equal hazard after 3 and 5 years from start of treatment	Equal hazard after 5 years (from starting treatment)	Agreed with the ERG that a 5-year treatment effect was appropriate and consistent with previous TAs	Minimal impact – after the initial 1–2 years, the conditional survival between both arms was similar
TA683 (ref. [19]) March 2021 Recommended	Pembrolizumab Lung, NSCLC, 1L	KEYNOTE-189 Phase III RCT	10.5 months 20.4 months	2 year	Independent	Equal hazards after 5 years from start of treatment	Immediate. Equal hazards after 3 years, 10 years. Lifetime treatment effect^	Gradual waning between 2 and 5 years	Agreed with ERG gradual waning approach	NR
TA692 (ref. [16]) April 2021 Not recommended	Pembrolizumab Bladder, urothelial carcinoma, 2L+	KEYNOTE-045 Phase III RCT	40.9 months 48.9 months	2 year	Independent	Equal hazards after 5 years from start of treatment	Immediate. Equal hazards after 3 and 10 years. Also, conditional waning on those with/without disease control	Equal hazards after 3 years. Also explored 2, 5 and 10 years in scenarios	3-year treatment effect most plausible	Base case, £47,123 3 years, £51,970 ERG Base case, £53,678 2 years, £61,315 5 years, £48,518 10 years, £45,377 None, £44,473
TA705 (ref. [13]) June 2021 Recommended	Atezolizumab Lung, NSCLC, 1L	IMpower110 Phase III RCT	31.3 months 52 months	No stopping rule for Atezolizumab. 2-year for pembrolizumab	Dependent using a time-dependent treatment effect	None (assumed lifetime treatment effect^)	Immediate. Atezolizumab equal hazards after 5 and 8 years	Maintained company base-case and explored several scenarios	AC considered various duration of treatment effect scenarios performed by the ERG	ICERs are pembrolizumab versus atezolizumab, higher ICER suggests atezolizumab is worth funding. Base-case, 560,832 Atezolizumab 60 months, 234,870 and 90 months, 345,711
TA724 (ref. [15]) September 2021 Not recommended	Nivolumab Lung, NSCLC, 1L	CHECKMATE-9LA Phase III RCT	Minimum follow-up 12.7 months	2 year	Independent	Initially none. Updated base-case equal hazards after 5 years (from stopping treatment)	Immediate. Equal hazard after 3 years	Equal hazards after 3 and 5 years explored (after starting treatment)	Treatment effect lasting 3 to 5 years most appropriate	Original submission: None, £29,139 3 year, £35,149
TA737 (ref. [26]) October 2021 Recommended	Pembrolizumab Oesophageal, 1L	KEYNOTE-590 Phase III RCT	12.6 months 33.6 months	2 year	Independent	None. Company claims TEW is reflected in the extrapolation of OS	Gradual waning between 5 and 7 years (equal hazard by 7)	Preferred scenario was waning between 5 and 7 years	All scenarios plausible	None, £41,688 5–7 years, £51,921
TA770 (ref. [24]) February 2022 Recommended	Pembrolizumab Lung, NSCLC, 1L	KEYNOTE-407 Phase III RCT	14.3 months 48 months	2 year	Independent	Equal hazard after 5-years from start of treatment for OS but lifetime treatment effect^ for PFS	Immediate. Equal hazard after 3 and 10 years from start of treatment	Preference of equal hazard for OS at 5 years. Explored TEW for PFS as scenario	Originally preferred TEW after 3 years but company contested with newer data cut. Settled for TEW after 5 years	Original base-case no waning, £38,833 3 years, £39,576
TA772 (ref. [21]) February 2022 Recommended	Pembrolizumab Lymphatic, 3L+	KEYNOTE-204 Phase III RCT	24.7 months 33 months	2 year	Independent	None (assumed lifetime treatment effect^)	Gradual waning between 5 and 7 years from start of treatment (equal hazard by 7) in OS and PFS	Explored gradual waning between 3 and 5 years in OS and PFS	Did not comment on TEW	Base case, £10,133 Waning 5–7 years, £10,282
TA798 (ref. [12]) June 2022 Recommended	Durvalumab Lung, NSCLC, 2L+	PACIFIC Phase III RCT	34.2 months 74.7 months	1 year	Independent	None (assumed lifetime treatment effect^)	Immediate. Equal hazards after 10 years from starting treatment	TEW after 3 years for PFS and 5 years for OS conditional on base-case	TEW after 3 and 5 years both appropriate for decision making	None, £11,719 10 years, £12,375 ERG 3 years, £20,345 5 years, £15,871
TA801 (ref. [18]) June 2022 Recommended	Pembrolizumab Breast, 1L	KEYNOTE-355 Phase III RCT	44.1 months 52.2 months	2 year	Independent	None (assumed lifetime treatment effect^)	Gradual waning from 4 years using SEER data and equal hazard (immediate) after 5 years	Preferred equal hazard after 5 years from start of treatment	5-year treatment effect most appropriate	None, £34,887 Gradual SEER, £31,605 5 years, £42,138
TA802 (ref. [22]) June 2022 Recommended	Cemiplimab Skin, 1L	EMPOWER-CSCC 1 Phase II single arm	NR	22 months	Independent	Equal hazard after 5 years from starting treatment	No waning and gradual waning between 60 and 96 months in OS and PFS	Additional TEW scenarios of equal hazards at 42 and 48 months	NR	ICERs versus chemo: Base case, £36,162 no waning, £26,263 gradual waning 60 and 96 months, £32,466
TA818 (ref. [17]) August 2022 Recommended	Nivolumab Lung, mesothelioma, 1L	CHECKMATE-743 Phase III RCT	29.7 months 37.5 months	2 year	Independent	None (assumed lifetime treatment effect^)	Gradual waning – decrease linearly between year 5 and 10 for both OS and PFS	Equal hazards after 5 years from start of treatment for both OS and PFS	Concluded that it was reasonable to assume some TEW, but duration unclear	None, £77,531 5–10 years, £119,543
TA823 (ref. [20]) September 2022 CDF	Atezolizumab Lung, NSCLC, 2L+	IMpower010 Phase III RCT	32.2 months 57 months	1 year	Independent	Originally equal hazard after 5 years. Subsequently removed and curves adjusted with 5-year cure assumption	None	ERG noted that the original waning improved CE, therefore recommended removing	None	NR

TA, technology appraisal; TEW, treatment effect waning; ERG, evidence review group; AC, appraisal committee; ICER, incremental cost-effectiveness ratio; RCT, randomised controlled trial; HR, hazard ratio; OS, overall survival; NR, not reported; CDF, cancer drugs fund; PFS, progression-free survival; SEER, Surveillance, Epidemiology, and End Results; CE, cost-effectiveness; CEM, cost-effectiveness model

^aMedian and maximum follow-up duration at the time of the appraisal unless otherwise stated. Most recent data cut used in the NICE TA presented if multiple data cuts used

^bTo determine if the survival models being used assumed proportional hazards / constant treatment effect or were independently fitted to each treatment arm (as this could inform whether waning may be ‘inbuilt’)

^cIn some instances, the base-case changed (e.g., after CDF review); we include the most recent base-case

^Where ‘lifetime treatment effect’ is used, it should be noted that we do not know what the lifetime treatment effect is, given that independent survival models were used in 18/19 appraisals

The application of TEW assumptions varied between submissions. In 11 of the 18 TAs, TEW was not applied in the company’s base-case. In 10 of these 11 TAs, independent survival models were used, so it was not clear whether the base case modelled treatment effect was increasing, decreasing or approximately remaining constant. In these instances, the External Assessment Group (EAG) and/or the appraisal committee often deemed scenario analyses that included TEW to be more appropriate for decision making.

Various methods for applying TEW were used across the 18 TAs reviewed. These included: immediate waning, gradual waning, conditional waning, waning involving cure assumptions and waning based on independently fitted survival models which converged in the long term. Table 2 provides a description of each of these approaches, outlining their underlying assumptions. Figure 1 illustrates the three most commonly used approaches—immediate waning, gradual waning and waning based on independent survival models.

Table 2.

Treatment effect waning methods uncovered from NICE TAs

Treatment effect waning method	Description
Immediate waning	Any treatment effect is removed at a pre-specified time point. Usually this involves setting a HR to 1.
Gradual waning	Assumes that the positive effect for the treatment group relative to the comparator group changes over time, reflecting a gradual decrease in the treatment effect. Using this approach, the estimated mortality hazard for the IO gradually converges with that of the comparator arm over the course of the waning period, such that by the end of the gradual waning period, the hazard used to inform the remainder of the model horizon is identical between treatment arms.
Conditional wanning	Within TA692 (pembrolizumab for bladder cancer), the company provided scenarios assuming waning in only those patients who did not achieve disease control. This meant that they assumed a ‘lifetime treatment effect’ for patients achieving disease control (based on best overall response).
Waning involving cure assumptions	Assuming a fixed cure point is not the same as assuming that the treatment effect wanes over time, but in practical terms it has a similar impact because beyond the cure time point the mortality hazards are equalised (usually at background population levels) in the treatment arms being compared, provided that some proportion of patients reach the cure time point in both treatment arms. If the proportion of patients remaining alive at the cure timepoint is larger in the IO group than in the comparator group, the hazard associated with patients who are cured will be applied to more patients in the IO group than in the comparator group. However, given that the same hazard is applied to all patients who remain alive beyond the cure timepoint, this is equivalent to hazards between treatment groups that converge at the cure timepoint.
Independently fitted survival models	If independent survival models are fitted to each treatment arm and the hazard plots show that the curves gradually converge (i.e. the treatment effect is diminishing), then this would imply that any TEW is to some extent already accounted for in the model.

Fig. 1.

A shows a treatment effect expressed as a hazard ratio from a (hypothetical) trial including a modelled treatment effect using independent survival models, a constant treatment effect and different types of treatment effect waning. B and C include plots showing hazard functions (converging hazards [B] and diverging hazards [C]) from a (hypothetical) trial (0–2-year trial period to the left of the vertical dashed lines) with extrapolation over a 30-year time horizon using independent survival models

Independent survival models were used in 17 of the TAs. Dependent models were used in 1 TA, more specifically a time-dependent treatment effect was incorporated. When independent survival models are used, assumptions around the treatment effect are implicit rather than explicit—i.e. the implied treatment effect over time depends upon the ratio of the hazards of the survival models fitted to each treatment arm, which may imply a treatment effect that is increasing, decreasing or constant. This implicit nature of the treatment effect when independent survival models are used is acknowledged in technical support document (TSD) 21, published by NICE’s Decision Support Unit (DSU). This document recommends that plots of the hazards and HRs predicted by independently fitted survival models should always be presented [8].

The use of independent models was not mentioned as a ‘waning approach’ in any of the TAs but is included as these models could (and often do) imply waning. Where hazards gradually converge (i.e. the treatment effect of an intervention is diminishing at a faster rate than the control), then this would imply that any TEW is to some extent already accounted for in the model. A crude illustration of this is shown in Fig. 1B. Figure 1C shows an illustration of hazards diverging and underlines the importance of examining these plots, as outlined in TSD 21 [8].

It is possible to gain some understanding as to whether independently fitted models predicted converging or diverging hazards over time by analysing the results of TEW scenarios compared with base-case scenarios (presented in the final column of Table 1). For example, if waning scenarios had only a minor impact on the incremental cost-effectiveness ratio (ICER), it is likely that the independently fitted survival models already predicted hazards that converged at a timepoint close to that used in the TEW scenario. In contrast, when TEW scenarios have a substantial impact on the ICER this may be because the independently fitted survival models predict diverging hazards or because the models predict hazards that converge at a timepoint substantially after that used in the TEW scenario. In some instances, TEW scenarios substantially increased the value of the ICER (e.g. comparing base-case to immediate at 3 years in TA428 [9]), lifting it above commonly accepted thresholds, therefore making TEW an important driver of uncertainty for decision making.

Few NICE TAs reviewed included a justification for the TEW methods used. It broadly appears companies generally held the view that there was a lack of evidence to justify not applying TEW, whereas EAGs and committees were of the opinion that a lack of evidence was not the same as evidence of absence. The often-missing justification and different perspectives from companies, EAGs and committees potentially led to TEW scenarios being explored without specific rationale for the type and timing of the waning modelled.

The reasons for the TEW assumptions used in different NICE TAs varied. They included referring to expert clinical opinion [10–13], results from certain clinical trials [14–18] and methods from previous NICE TAs for IOs [19–22]. In the appraisals which cited previous trials on the same treatment, the key reasoning used for this was that these trials had longer follow-up data and, therefore, could be used to either confirm or challenge the pre-existing TEW assumptions made by the company. When TAs cited previous TAs to justify their TEW assumption, the TEW assumption made in the previous TA was typically retained by the NICE committee, with this commonly involving immediate waning around 3–5 years. It is possible that this timeframe was used arbitrarily as it was effectively a precedent that had been set for IOs. Few appraisals assessed the clinical reasoning behind these waning timepoints. There was some evidence of precedence by disease type. For example, TA531 (ref. [23]), TA683 (ref. [19]) and TA770 (ref. [24]) for pembrolizumab in first line non-small cell lung cancer indications all explored immediate TEW at 3 years.

When searching the other English language HTA agencies, we encountered several instances where TEW approaches differed between HTA agencies. The most notable difference in broad approach taken was the substantially shorter time horizons often used for the PBAC appraisals (often 7 years versus lifetime). This is relevant in the context of TEW, because TEW assumptions are generally made in the relatively long-term; thus, if shorter time horizons are considered (such as for PBAC submissions or generally for more aggressive cancer types), assumptions around TEW may have a relatively lower impact on cost-effectiveness estimates.

Generally, any other differences were typically the year at which waning began or ended. For example, for PC0250-000/TA737 (pembrolizumab for oesophageal cancer), in the NICE TA, a gradual TEW scenario was explored between years 5 and 7. In the CADTH submission, the company applied the gradual waning between 2 and 5 years as a scenario, which was subsequently used by CADTH as their base-case. Generally, any differences in handling TEW between NICE and CADTH were because of differences in EAG/economic guidance panel for CADTH submissions or committee preference. It was common for CADTH to use TEW scenarios in their base-case, even when the company did not include them in their base-case.

ISPOR Scientific Presentations Database Review

There were 11 ISPOR presentations of relevance identified (Supplementary Material [Supplementary Fig. 1]). Two studies investigated the accuracy of different TEW methods used in past NICE appraisals in lung cancer using subsequently published later data cuts from the relevant trial (pembrolizumab in KEYNOTE-024 and nivolumab in CheckMate-057). Conclusions were similar in that gradually equalising hazards of death (gradual waning, described in Table 2) demonstrated improved predictive accuracy versus immediate TEW [27, 28] for observed and predicted longer-term survival. It is however notable that neither presentation provided information on what their independently fitted survival models implied about the treatment effect over time when waning was not applied.

One review uncovered inconsistencies in assumptions made by companies, EAGs and NICE appraisal committees around TEW and highlighted the need for further guidance for consistent incorporation of TEW methods in HTA submissions [6]. The authors reviewed ten nivolumab NICE TAs and found that TEW was not incorporated into the company’s base case in any of the original submissions. However, TEW was later included in the company's accepted base case in three of these TAs following requests from the EAG or committee. Although, details were lacking due to the review being in poster presentation format and not solely being focussed on TEW.

Kamgar et al. [4] created smooth hazard ratio (HR) plots based on pseudo-individual patient level data from longer-term follow up data (~5 years) of the pivotal trials from TA428 (ref. [9]), TA531 (ref. [23]), TA578 (ref. [11]) and TA692 (ref. [16]). The plots illustrate the ratio of hazards between treatment groups, demonstrating whether the treatment effect remains approximately the same over time, or whether the effect appears to increase or decrease. In the examples, the HR trended towards one in the longer term suggestive of a waning effect. However, when smoothed HRs are fitted to randomised controlled trials (RCTs) with low numbers at risk in the long-term, results become uncertain as has happened in the Kamgar et al example [4].

Finally, other presentations that were included used TEW in analyses, discussed model structures with relevance for TEW or just mentioned TEW in any capacity meaning that the presentations were eligible for inclusion but did not provide information useful for this review. Further information on each of the presentations is provided in Supplementary Material (Supplementary Table 1).

Targeted Literature Review of Clinical Evidence

The initial search identified 880 articles. A total of 799 articles were removed at the abstract screening stage with most failing to mention treatment effects over time following discontinuation of treatment. We subsequently searched the full texts of the remaining articles and provide a breakdown of reasons for exclusion in Supplementary Fig. 2. After screening, there were 30 articles included that mentioned or discussed treatment effects following discontinuation.

Of the 30 papers included, 14 presented clinical trial data, e.g. clinical trials with long-term follow-up or pooled analyses of clinical trials. The articles based on clinical trial data provide longer-term survival outcomes from clinical trials exploring effects of IOs on a range of different cancers (Supplementary Table 2). A total of 10 out of 14 were for nivolumab [29–38], 3 for pembrolizumab [39–41] and 1 for avelumab [42]. Kaplan–Meier plots of the survival outcomes we discuss are presented in main article text for each reference.

Also identified were 11 real-world evidence (RWE) observational analyses (summarised in Supplementary Table 3) [43–53], and 5 were review papers (including systematic reviews, more general reviews and short communications), summarised in Supplementary Table 4 [54–58].

Despite the terminology none of the trial papers identified presented a specific analysis of the relative treatment effect over time. For instance, hazard plots akin to those illustrated in Fig. 1 were not included in any of the trials to examine this aspect explicitly. Therefore, while there is clearly evidence available on the long-term outcomes of IO treatments for overall survival (OS), progression-free survival (PFS) and duration of response (DOR), this evidence does not provide information that directly addresses the potential for TEW as it is applied in HTA.

Similarly, it is difficult to derive direct estimates of the effectiveness of IO treatments beyond treatment discontinuation from the available evidence. Although several studies report information on treatment duration alongside survival estimates, it is not straightforward to determine whether patients that discontinue treatment continue to benefit.

Discussion

Summary

The objective of this work is to provide an overview of the modelling of TEW in economic evaluations of IO therapies. We conducted an extensive search of HTA agency appraisals, specifically focussing on NICE and other English language HTA agencies, to understand the methods employed in appraisals to address the potential for TEW.

TEW in HTA

Our analysis of 18 NICE TAs revealed that there is no standardised approach adopted by companies in their submissions, and the preferences of the EAG and committee can vary across different groups or committees. Most frequently, immediate TEW within a range of 3–5 years from the start of treatment was used. Results for other English language HTA bodies were broadly consistent with NICE, although there were several specific cases where TEW approaches used for related appraisals differed between HTA agencies—even for the same evidence base. This finding is unsurprising given the lack of guidance/evidence around the subject and the different preferences from EAGs and committees. A limitation of reviewing other HTA agencies is that detailed information comparable to that found via NICE was seldom available. This limitation restricts the conclusions that can be drawn from these comparisons.

In the TA review, we observed that when independent survival models are used, the implied treatment effect over time depends upon the ratio of the hazards of the survival models. We believe clarity around the hazards predicted by independently fitted survival models is crucial with respect to TEW. Often, when TEW is not included in an analysis, EAGs and ACs interpret the analysis as if a lifetime treatment effect is being modelled. If the independently fitted models result in converging hazards however, it has already been assumed that the treatment effect wanes over time, even without any explicit TEW being added. We believe that if this was clearly demonstrated through hazard plots, some concerns highlighted around assuming a ‘lifetime treatment effect’ would be allayed. This finding does not appear to have been discussed in the materials we have reviewed.

We performed a search of the ISPOR presentation database to identify relevant research related to TEW in the specific context of HTA. We included 11 presentations in our analysis and categorised each into different groups due to the broad nature of the work submitted to ISPOR. Notably, we discovered two projects that arrived at the same conclusion using trial data assessing pembrolizumab (KEYNOTE-024) and nivolumab (CheckMate-057), suggesting that gradual waning methods based on the equalisation of hazards over time may represent a more suitable approach than immediate waning [27, 28].

The assumption of setting the HR to 1 at a specified timepoint (as seen in other literature) is that there is then no difference in hazards between the treatment group and the control group from that timepoint onwards termed ‘immediate waning’. Applying waning on this immediate basis, implies beneficial effects will suddenly disappear on a specified day—which does not reflect the underlying biology of the disease or the mechanism of action of IOs. Not only does this approach lack face validity but also does not accord with the (limited) clinical evidence. If it is to be assumed that the effect of a treatment attenuates, it seems more realistic to assume a gradual waning of the treatment effect. Related to this, it is apparent that when TEW is applied, it is dealt with as a function of time since beginning treatment. An alternative approach could be to apply waning based on time since discontinuing a treatment—further research on when treatment effects may be expected to begin to wane would be valuable. Extended follow-up of RCTs designed with a stopping rule may also provide a means of assessing and quantifying TEW.

The approach of gradual waning also offers the opportunity to identify or develop an appropriate model to capture the pattern of the waning effect. In the TA review, simple linear models were used when gradual waning was assumed, but these models would often not reflect the underlying biology of the disease and mechanisms of action. The choice of the model will influence the results of the analysis however and should be carefully considered. Limitations and challenges of gradual waning include: (1) potentially biased results from assuming the wrong waning pattern, (2) choosing incorrect timing and (3) increased complexity of analysis (as opposed to immediate waning). Clinical input in two appraisals (TA737 [ref. [26]] and TA683 [ref. [19]]) highlighted that a gradual waning would be more clinically plausible. We did not, however, identify any TAs which used gradual waning in their base case.

Findings from the TLR of Clinical Evidence

We conducted a TLR to identify evidence on the persistence of IO treatment effects beyond treatment discontinuation, with the intention of uncovering whether the techniques employed in an HTA setting could be rationalised based on the available evidence on clinical effectiveness. The TLR uncovered promising long-term survival outcomes in patients treated with IOs, particularly for patients who achieved a CR, who tended to experience more favourable survival outcomes. It is difficult to interpret what the evidence reviewed here means with respect to TEW as the way TEW is understood in HTA is about the effect of the new treatment over time, relative to the comparator. No analyses comparing these long-term hazards between IO-treated patients and comparator groups were found, and so, we recommend future reports of long-term IO outcomes (for instance, secondary analyses) to include smoothed hazards/HRs so that these findings can be used to inform TEW assumptions. We did identify work suggesting that some IO patients may be cured [56]; however, in situations where a proportion of patients in both treatment arms can be considered to be cured, this means that the HR between treatments will revert to 1. This is not waning as such, because the treatment effect hasn’t worn off, but the implications are the same—hazards in both arms equalise.

Limitations

This article has several limitations. With the reviews conducted we believe we have identified the methods used for modelling of TEW, but not of how often TEW is modelled for IOs, as by design we have not identified appraisals where TEW was not included. Furthermore, we are not able to tell what (if anything) has been done about TEW when stopping rules have not been included. The TLR was, by nature a ‘targeted’ review with the specific aim of identifying published research that specifically analysed or reviewed survival post treatment discontinuation for IO treatments. Given the absence of recognised terms and understandings (with many analyses presented in Supplementary Findings, or implicit in figures), it is possible that some relevant research could have been missed. However, we believe that it is unlikely that we have missed any relevant analyses that might have compared long-term hazards between IO-treated patients and comparator groups. Nevertheless, future work should include in-depth systematic reviews to further understand how the scientific literature could inform TEW analyses.

A further limitation is that because our focus was on identifying evidence that analysed treatment effects post treatment discontinuation, our review does not provide a comprehensive overview of evidence on long-term survival associated with IO treatments. Although our conclusions and recommendations are for IOs and TEW in general, we did not attempt to split evidence and approaches by type of IO, though acknowledge it is possible that these different treatments could behave differently over time—particularly where different mechanisms are used, in different conditions.

Proposed Sequence of Steps to Assess TEW

Bringing our findings together, we propose a sequence of steps for analysts to use when assessing whether it is necessary and appropriate to present TEW scenario analyses in HTA (Fig. 2). The findings from this research indicate that TEW is applied inconsistently in HTA, often with precedence and without incorporating evidence. By following these steps, we hope to make TEW as evidence based as possible by harnessing analyses of the trial data and of any relevant external data or information. It is important to note that we do not make recommendations about exactly what should be assumed about TEW; instead, we focus on steps that should be taken to ensure that assumptions made about TEW are based on the best evidence available. Additionally, we acknowledge that our proposed steps could be valuably informed by further research and validation and may require updating in the future.

Fig. 2.

A sequence of steps for assessing the necessity and appropriateness of presenting TEW scenario analyses in health technology assessment

Conclusions

Inconsistencies were observed in the application of TEW across IO oncology appraisals, with TEW often applied without clear scientific rationale or link to the evidence base. Our findings from the TA review demonstrate the need for consistent and appropriate implementation of TEW in HTA oncology appraisals. We have suggested a sequence of steps to help rationalise and justify whether specific TEW scenario analyses are appropriate on a case-by-case basis.

Results from the TLR highlight the absence of direct evidence on TEW in the literature, given that in HTA, TEW is defined in terms of relative treatment effects. We, therefore, conclude that currently, the evidence on TEW is suboptimal; the current body of evidence lacks the necessary analyses to fully grasp the complexities of TEW. As such although the evidence may have been collected, it is not possible at present to justify (or refute) application of TEW conclusively—aside from the implausible assumption of immediate waning. We recommend that studies (particularly secondary analyses of study data) incorporate plots of the relative treatment effect over time to inform future TEW analyses.

Supplementary Information

Below is the link to the electronic supplementary material.

Declarations

Funding

Funding for this study was provided by BMS. Publication of results was not contingent on the sponsor’s approval or censorship of the manuscript.

Conflicts of interests

K.T., N.R.L., T.D. and A.H. are employees of Delta Hat Limited, who received funding from BMS to perform the analyses. N.R.L. has a role at the University of Sheffield that is funded by Yorkshire Cancer Research (award reference number S406NL), but this work was conducted in his time as a Delta Hat Limited employee. S.H., G.O., J.B., C.C., I.K. and D.B. are employees of BMS and hold BMS stock.

Ethics approval

Not applicable.

Informed consent

Not applicable.

Consent for publication

Not applicable.

Availability of data and materials

Not applicable.

Code availability

Not applicable.

Author contributions

K.T., N.L. and D.B. conceived the concept for the manuscript with input from all authors. K.T. and T.D. performed data extractions. K.T. drafted the initial manuscript. All authors contributed to revising and editing the manuscript.