Potential bias associated with modeling the effectiveness of healthcare interventions in reducing mortality using an overall hazard ratio

Abstract

Background.

Clinical trials often report intervention efficacy in terms of the reduction in all-cause mortality between the treatment and control arms (i.e., an overall hazard ratio [oHR]) instead of the reduction in disease-specific mortality (i.e., a disease-specific hazard ratio [dsHR]). Using oHR to reduce all-cause mortality beyond the time horizon of the trial may introduce bias if the relative proportion of other-cause mortality increases with age. We sought to quantify this oHR extrapolation bias and propose a new approach to overcome this bias.

Methods.

We simulated a hypothetical cohort of patients with a generic disease that increased background mortality by a constant additive disease-specific rate. We quantified the bias in terms of the percentage change in life expectancy gains with the intervention under an oHR compared to a dsHR approach as a function of the cohort start age, the disease-specific mortality rate, dsHR, and the duration of the intervention’s effect. We then quantified the bias in a cost-effectiveness analysis (CEA) of implantable cardioverter-defibrillators based on efficacy estimates from a clinical trial.

Results.

For a cohort of 50-year-old patients with a disease-specific mortality of 0.05, a dsHR of 0.5, a calculated oHR of 0.55 and a lifetime duration of effect, the bias was 28%. We varied these key parameters over wide ranges and the resulting bias ranged between 3% and 140%. In the CEA, the use of oHR as the intervention’s effectiveness overestimated quality-adjusted life expectancy by 9% and costs by 3%, biasing the ICER by-6%.

Conclusions.

The use of an oHR approach to model the intervention’s effectiveness beyond the time horizon of the trial overestimates its benefits. In CEAs, this bias could decrease the cost of a QALY, overestimating interventions’ cost-effectiveness.

1. INTRODUCTION

The development of decision models requires the analyst to make several assumptions. It’s important to understand the potential implications of these assumptions. For some situations it may suffice to conduct structural sensitivity analyses to examine the impact of various modeling assumptions [1–4]. For other situations, there are assumptions that seem reasonable but can actually introduce unintended biases in model results [5]. In this paper, we present an assumption that induces an unintended bias and present a less biased approach.

In health economic evaluation we often rely on published estimates of intervention effectiveness from randomized controlled trials (RCTs) to parameterize decision models [6–8]. Trials for which all-cause mortality is the primary endpoint typically report a hazard ratio (HR) as a measure of efficacy [9]. The HR reports an intervention’s efficacy in terms of the reduction in the all-cause mortality rate between treatment and control arms. We refer to this HR as the overall hazard ratio (oHR), which can be estimated using statistical methods from survival analysis [10,11]. An oHR reflects only the reduction in mortality over the time horizon of the trial, which is often relatively short compared to the expected life span of the study population [6].

The U.S. Public Health Service Panel on Cost-Effectiveness in Health and Medicine, the International Society of Pharmacoeconomics and Outcomes Research (ISPOR) Good Research Practices Task Force and the Second Panel on Cost-Effectiveness in Health and Medicine recommend using a time horizon in CEAs that is long enough to capture all relevant future effects [6,12–14]; in particular, a lifetime horizon for interventions that affect survival [8,15–18]. Two general situations of the source of data for extending the time horizon beyond the follow-up period of the trial is to use published summaries of parameters of interest [1,19,20] or to use individual-level data to estimate them by fitting parametric survival functions [21–23]; our study considers the former approach.

A common practice in model development is to specify at least two components of all-cause mortality: (1) a background mortality (i.e., non-disease-specific mortality) that is specific for the age, sex, and race (ASR) of the modeled cohort and can be time dependent (note that one assumes that the disease-specific causes of death in the life tables is either negligible or is separated out), and (2) a disease-specific mortality–the mortality for which the intervention is intended. The disease-specific mortality is only applied to patients with the disease and can either be characterized as a difference in hazards or a hazard ratio between people with and without the disease, so that all-cause mortality could be, respectively, an additive or multiplicative function of background mortality and the disease-specific component, and be estimated as a single parameter from a clinical study [2]. Background mortality rates for all ages are available from published country-specific life tables; thus, extrapolations are not necessary as the simulated cohort ages. However, the analyst will need to make reasonable assumptions regarding the extrapolation of the disease-specific rate in the absence of intervention, which can be informed by population-based studies and clinical judgment. The analyst will also need to make assumptions about extrapolating the intervention effect beyond the time horizon of the trial, and typically a range of assumptions about the duration of the effect are explored. However, it’s also important to consider how one incorporates the efficacy estimate (i.e., oHR) into the model. For example, one could simply apply the estimated oHR as intended (i.e., to reduce all causes of death) for the duration of the intervention effect. However, if background mortality increases with age and all-cause mortality is an additive function of background mortality and a constant disease-specific mortality, then the relative proportion of disease-specific mortality to overall mortality will decrease over time. Thus, assuming that the oHR estimated during a trial (when the proportion of disease-specific death rates are relatively large) applies to the longer term (when the proportion of disease-specific death rates are relatively small) may result in a bias. The aim of this paper is to quantify the bias associated with modeling an intervention’s effectiveness using an oHR estimate beyond the time horizon of the clinical trial and to propose a new approach to overcome this bias.

2. METHODS

To illustrate the magnitude and direction of the bias we used two approaches for modeling an intervention’s effect beyond the time horizon of a clinical trial when the effectiveness estimate is reported as a reduction in all-cause mortality and all-cause mortality is an additive function of background mortality and a constant disease-specific mortality. For the naïve approach, we assumed that the oHR reduced all-cause mortality for the duration of the intervention effect. For our recommended approach, we derive an estimate for the disease-specific hazard ratio (dsHR) –the reduction in the disease-specific mortality rate from the intervention – from the oHR and age- and disease-specific mortality of the trial cohort. We compared the results of modeling an intervention’s effectiveness using both approaches in a simple state-transition model and quantified the bias in terms of differences in life expectancy (LE) gains under the two approaches and reported the bias as a function of the age of the simulated cohort, the disease-specific mortality rate, dsHR, and the duration of the intervention’s effect. We refer to this bias as oHR extrapolation bias. Finally, we illustrate the oHR extrapolation bias with a real-world problem, where we model the cost-effectiveness of implantable cardioverter-defibrillators (ICDs) using efficacy estimates from a clinical trial.

2.1. Definition of terms and bias

The published oHR estimated over the time horizon of a clinical trial as a reduction in the all-cause mortality rate (μ_All) could be defined as

$$oH{R}_{RCT}=\frac{{\widehat{\mu }}_{All}^{\ast }}{{\widehat{\mu }}_{All}},$$ (1)

where ${\widehat{\mu }}_{All}^{\ast }$ and ${\widehat{\mu }}_{All}$ are the average, all-cause mortality rates of the subjects in the treatment and control arms of the clinical trial, respectively. It is reasonable to assume that the average age-, sex-, and race-specific (ASR) background mortality rate, μ_ASR, is the same for the two arms and that only the disease-specific mortality rate, μ_Dis, is affected by the intervention [2,24]. Thus, we can express μ_All as μ_ASR+μ_Dis for both arms in Equation (1) and define oHR as

$$oH{R}_{RCT}=\frac{{\widehat{\mu }}_{ASR}+{\widehat{\mu }}_{Dis}^{\ast }}{{\widehat{\mu }}_{ASR}+{\widehat{\mu }}_{Dis}},$$ (2)

where ${\widehat{\mu }}_{ASR}$ is the average background mortality rate in the trial, and ${\widehat{\mu }}_{Dis}$ and ${\widehat{\mu }}_{Dis}^{\ast }$ are the average disease-specific mortality rates in the absence and presence of intervention, respectively. If the intervention is effective, the mean disease-specific mortality rate will be lower in the treatment arm than in the control arm and thus can be expressed as

$${\widehat{\mu }}_{Dis}^{\ast }={\widehat{\mu }}_{Dis}\ast dsHR,$$ (3)

and dsHR can be defined as

$$dsHR=\frac{{\widehat{\mu }}_{Dis}^{\ast }}{{\widehat{\mu }}_{Dis}}.$$ (4)

We used a decision model to extend mortality beyond the time horizon of the trial and incorporated background mortality from life tables [25] for a specific cohort defined by age, sex, and race μ_ASR(a) that varies with each year of age, a, as the cohort ages in the model. We modeled the intervention’s effectiveness using both an oHR and a dsHR approach. For the oHR approach, we assume that the all-cause mortality rate is reduced each year by a constant factor with the intervention (i.e.,$\left({\mu }_{ASR}\left(a\right)+{\mu }_{Dis}\right)\ast oHR$). For the dsHR approach, we assume that the intervention acts solely on the disease-specific mortality rate, reducing it by a constant factor (i.e.,${\mu }_{ASR}\left(a\right)+{\mu }_{Dis}\ast dsHR$). We quantified the oHR extrapolation bias as the percentage change in gains in expected outcome with intervention vs. no intervention (ΔY) [26] under an oHR approach compared with a dsHR approach:

$$Bias=\frac{\Delta Y_{oHR}-\Delta Y_{dsHR}}{\Delta Y_{dsHR}}\times 100.$$ (5)

Expected outcome Y could be any expected outcome produced by a decision model. Examples of common outcomes are LE, quality-adjusted life expectancy (QALE), and costs. A positive bias implies that the outcome gains using oHR as the intervention’s effectiveness are higher compared to a dsHR approach (i.e., the outcome gains attributed to the intervention will be overestimated under an oHR approach).

2.2. Estimation of dsHR from aggregated data

In practice, we often don’t have estimates of dsHR from clinical trials, in part because of the concerns related to ascertaining cause of death [6]. Instead, it is possible to derive dsHR from the average background mortality for the trial population, the disease-specific mortality and the published oHR following two steps. First, by assuming that mortality in the trial population due to causes other than the disease of interest is the same as the expected mortality in the general population with same demographic characteristics, we can calculate the average background mortality for the trial using estimates from country-specific life tables that is specific to the demographic characteristics (age, sex, race) of the trial population:

$${\widehat{\mu }}_{ASR}=\frac{{\sum }_{a=a_0}^{a_0+T}{\mu }_{ASR}\left(a\right)}{T},$$

where a₀ is the mean age of the study, T is the number of follow-up periods of the trial (in years) and μ_ASR(a) is the background mortality rate for the trial population for each age, a, (by sex and race) and is obtained from country-specific life tables for each a, which reflects the rate of dying from any cause but the disease of interest for that year. Second, by assuming that the disease-specific causes of death in the life tables is negligible (or subtracting out disease-specific causes of death), the disease-specific mortality rate can be derived from the control arm by subtracting the background rate from the overall observed rate. Finally, to obtain an expression for dsHR as a function of a published oHR estimate, ${\widehat{\mu }}_{ASR}$ and ${\widehat{\mu }}_{Dis}$, we substitute ${\widehat{\mu }}_{Dis}^{\ast }$ from Equation (3) into Equation (2) and solve for dsHR:

$$dsHR=\frac{oHR\ast \left({\widehat{\mu }}_{ASR}+{\widehat{\mu }}_{Dis}\right)-{\widehat{\mu }}_{ASR}}{{\widehat{\mu }}_{Dis}}.$$ (6)

All-cause mortality rates in Equation (1) can also be expressed in terms of additional disease-or comorbidity-specific mortality rates that are not affected by the intervention but represent increase mortality compared to background mortality. If all-cause mortality is composed of other mortality rates due to comorbid disease(s) in addition to background mortality and the mortality rate affected by the intervention, a more general approach of Equation (6) is:

$$dsHR=\frac{oHR\ast {\widehat{\mu }}_{All}-\left({\widehat{\mu }}_{All}-{\widehat{\mu }}_{Dis}\right)}{{\widehat{\mu }}_{Dis}}.$$ (7)

The uncertainty of dsHR can be quantified through Monte Carlo methods using either Equation (6) or (7). In general, we assume that there is no uncertainty for background mortality, so the uncertainty of dsHR is mainly derived from the uncertainty of oHR and ${\widehat{\mu }}_{Dis}$. In the supplementary material, we performed an analytical analysis to demonstrate the influence of various model parameters on the dsHR estimate analytically using a comparative statics approach under different parametric functional forms to represent background mortality. We provide an R package (https://github.com/feralaes/dshr) that implements the equations above to facilitate the estimation of dsHR as a function of oHR and other key parameters of the trial under different functional forms for the background mortality.

2.3. A simple disease-treatment model

We simulated a hypothetical cohort of patients using a simple state-transition model [27,28] with two health states, ill and dead. We assumed patients start in the ill state and face a mortality rate equal to the sum of the background mortality rate, μ_ASR(a), and a constant additive disease-specific mortality rate, μ_Dis. We compared a hypothetical intervention with no intervention using both oHR and dsHR effectiveness approaches and quantified the oHR extrapolation bias as the percentage change in LE gains.

We considered a base-case scenario of a cohort of 50-year-old patients in which age-, sex- and race-specific mortality rates were incorporated using data from the U.S. life tables [25] to represent the general population with a disease-specific mortality of 0.05. We assumed a known dsHR of 0.5 and a 5-year time horizon of the clinical trial. We calculated the oHR that one would expect from a trial of patients with an average age of 50 years representative of the US general population in terms of sex and race. Using the time horizon and the study population of the clinical trial we calculated an oHR of 0.55 from Equations (2) and (3).

We evaluated the magnitude of the oHR extrapolation bias as a function of four key parameters: (1) the start age of the simulated cohort (ages 40, 50, 60 and 70 years), (2) the annual disease-specific mortality rate (100 different values between 0.01 and 0.10), (3) the effectiveness of the intervention in reducing the disease-specific mortality rate (101 different values between 0.1 and 0.9), and (4) the duration of the intervention’s effect (five different values between 5 years and lifetime). The parameters of the simple disease-treatment model and their ranges are shown in Table 1. To provide a quantitative measure of the associations between the oHR extrapolation bias and the key parameters, we proposed a computer simulation study where we compute the oHR extrapolation bias from the simple disease-treatment model simulated over multiple combinations of these parameters. We used a meta-analytic regression approach between the oHR extrapolation bias, and the start age of the cohort, disease-specific mortality rate, and dsHR following a full factorial design [29,30]. We fitted two separate meta-analytic multivariate regressions based on two different assumptions about the duration of the intervention effect. First, we fitted a meta-analytic multivariate regression fixing the duration of the intervention effect to be lifetime, which effectively means running the cohort until age 100, resulting in 40,400 simulated scenarios. Second, we incorporated the duration of the intervention effect as a dependent variable from five years to lifetime with ten-year increments, resulting in 202,000 simulated scenarios. We specified a log-log relationship between the oHR extrapolation bias and both the disease-specific mortality rate and dsHR to quantify the percentage change on the oHR extrapolation bias associated with a one-percent change in the dependent variables, which corresponds to a simulation-based elasticity. Additionally, we specified a log-linear relationship between the oHR extrapolation bias and both the start age of the cohort and the duration of the intervention effect to quantify the percentage change on the bias for a one-year change in the independent variables, which corresponds to a simulation-based semi-elasticity.

Table 1:

Model Parameters for Simple-Disease Treatment Model

Parameter	Base-case scenario	Range
Start age of the cohort (years)	50	40–70
Annual disease-specific mortality rate (μDis)	0.05	0.01–0.1
Disease-specific hazard ratio (dsHR)	0.5
Duration of intervention effect	Lifetime	5 years-lifetime

To further illustrate the difference between the dsHR and oHR estimates, we calculated the dsHR from a hypothetical clinical trial for different start ages, values of disease-specific mortality rates and oHR using Equation (6) assuming 1-, 5-, and 10-year time horizons of the trial.

2.4. Cost-effectiveness analysis model

To illustrate the oHR extrapolation bias we performed a CEA of ICDs, which are implantable devices used as a first-line treatment and prophylactic therapy to potentially avert sudden death from cardiac causes, such as ventricular fibrillation and ventricular tachycardia [31]. We evaluated the long-term costs and benefits of prophylactic implantation of ICD in patients with myocardial infarction (MI) and reduced ejection fraction based on the efficacy from the Multicenter Automatic Defibrillator Implantation Trial II (MADIT-II) compared with conventional therapy [32]. The estimated efficacy of ICD in reducing all-cause mortality (i.e., an oHR) from MADIT-II was 0.69. We incorporated parameters for costs and utilities from previous analyses using observational data [33–37]. Model parameters are listed in Table 2.

Table 2:

Model Parameters for the cost-effectiveness analysis of ICDs

Parameter	Value	Source
Clinical Trial Variables
Length of trial (months)	20	[32]
Starting age of population (years)	64	[32]
Clinical Variables
Total risk of death (20 months)	0.198	[32]
Sudden-cardiac mortality risk (20 months)	0.1	[32]
Probability of ICD procedural death	0	[32]
Efficacy of ICD in reducing all-cause mortality (oHR)	0.69	[32]
Frequency of ICD generator replacement (years)	7	[32]
Probability of lead problems requiring surgical intervention (20 months)	0.024	[32]
Costs
Initial hospitalization	$23,314	[33,34]
ICD device	$25,00	[34]
Monthly inpatient cost	$494	[33]
Monthly outpatient cost	$37	[33]
Generator replacement	$21,742	[34]
Utilities
Baseline health state (Control therapy)	0.88 QALY	[33–35]
ICD	0.88 QALY	[33–35]
Decrease in utility if experience a complication from the ICD	1 day	[33–35]
Other variables
Annual discount rate	3%	[14]

Notes: All probabilities are monthly unless stated otherwise. ICD: implantable cardioverter defibrillator.

We simulated a hypothetical cohort of patients that are at risk of three different mortalities: sudden cardiac, non-sudden cardiac, and other non-cardiac background-related mortality. Patients receiving an ICD face a monthly risk of having complications due to the ICD, and this risk persist over the patients’ lifetime. Patients who experience a complication are withdrawn from the intervention (i.e., get their device removed) and switch to conventional therapy. We assumed that the ICD battery would need to be replaced every seven years and that each replacement incurs a fixed cost. We modeled the effectiveness of ICD assuming a lifetime benefit under both the oHR and dsHR approaches. The ICD reduces only sudden-cardiac deaths and thus patients with the ICD face the same non-sudden cardiac death as patients in conventional therapy. We did not model the risk of death from the implantation procedure; patients in the MADIT-II trial did not experience death from the surgical implantation and we considered it of no relevance for our illustration purposes. A schematic diagram of the decision model of the CEA of ICD is illustrated in Figure 1.

Figure 1:

Schematic representation of the decision-analytic model. The square node on the left represents a choice among alternative treatments: ICD implantation as a prophylactic therapy to potentially avert sudden death from cardiac causes or no intervention (conventional therapy). Circles represent chance nodes. Patients in both strategies enter a Markov tree (denoted by a circle with an “M”). Each Markov tree represents the salt states in which the patients can transition at any given cycle. Health states are denoted with ovals with their corresponding names. Patients in the ICD strategy face a monthly risk of complications that will cause them to withdraw from the intervention and switch to conventional therapy. All patients face monthly all-cause mortality composed of three different causes: (1) sudden cardiac death, (2) non-sudden cardiac death, and (3) background mortality. ICD: implantable cardioverter-defibrillators; No Rx: No intervention (conventional therapy).

In MADIT-II the total mortality risk over the 20-month period in the control arm of the trial was 19.8%. The authors estimated a sudden-cardiac mortality due to arrhythmia of 10%. We transformed these risks into rates assuming a constant rate over the 20 months and calculated a dsHR of 0.35 using Equation (7). In addition to background and sudden-cardiac mortalities, total mortality of the population in MADIT-II trial had an extra mortality component from non-sudden cardiac death due to their heart condition. We modeled the extra component associated with non-sudden cardiac death as an additive mortality rate to the background rate (similar to the way we modeled sudden cardiac death). From the characteristics of the trial’s study population and assuming a time horizon of 20 months we used US life tables to estimate the background mortality. To match the total mortality of the clinical trial, we calculated a non-sudden cardiac excess mortality rate of 0.09 over the 20-months period of the trial.

We quantified the oHR extrapolation bias as the percentage change in gains on four outcomes: LE, QALE, costs and incremental cost-effectiveness ratios (ICER) with ICD vs. no intervention under an oHR approach compared to a dsHR approach. All the analyses were conducted in R statistical program [38].

3. RESULTS

3.1. Simple disease-treatment model

Assuming a 50-year-old cohort (inclusive of all sex and race groups) with an annual disease-specific mortality of 0.05, a dsHR of 0.5, an estimated oHR of 0.55, and assuming a lifetime duration of intervention effect, the estimated gains in LE with the hypothetical intervention was 5.7 years under a dsHR approach and 7.3 years under an oHR approach. The magnitude of the oHR extrapolation bias was 28% (i.e., 7.3 years is a 28% increase in the LE gains compared to 5.7 years). To explore the magnitude of the bias across wide ranges of the start age, disease-specific mortality, dsHR, and duration of intervention effect, we quantified the bias as a function of these four key parameters. Figure 2 shows how the oHR extrapolation bias changes with different values for the start age of the cohort (40, 50, 60 and 70), the baseline disease-specific mortality rate (0.01–0.1), dsHR (0.1–0.9), and the duration of intervention effect (lifetime and ten-years). Different colors represent scenarios with varying magnitudes of the bias, ranging between 3%–140% and 1%–29% increase in LE gains across all starting ages of the cohort for a lifetime and ten-year duration of the intervention’s effect, respectively. The magnitude of the oHR extrapolation bias was most sensitive to the disease-specific mortality rate and increased as the disease-specific mortality rate decreased. The oHR extrapolation bias was highest when both the disease-specific mortality rate and dsHR were at their lowest values (i.e., 0.01 and 0.1, respectively), independently of the start age of the cohort.

Figure 2:

Heat maps of the oHR extrapolation bias at different start ages of the simulated cohort as a function of the disease-specific mortality rate (μ_Dis), and the disease-specific hazard ratio (dsHR) assuming a duration of the intervention effect of a lifetime (panel A) and ten years (panel B).

Table 3 presents the meta-analytic regression results on the oHR extrapolation bias for the simple disease-treatment model for two different assumptions about the duration of the intervention effect. Column (1) reports the results for a lifetime duration of the intervention effect. Assuming a lifetime duration of the effect, a one-year increase in the start age increased the oHR extrapolation bias by 2.3% and a 10% decrease in the disease-specific mortality rate increased the bias by 8.1%; a similar decrease in dsHR increased the bias by 5.7%. By incorporating different durations of the intervention effect – ranging from five years to a lifetime — a one-year increase in the duration of the effect increased the bias by 6.1%, Column (2).

Table 3:

Regression coefficients from meta-analytic regression on the oHR extrapolation bias for the simple disease-treatment model. (1) Duration of effect is fixed at lifetime, which effectively means running the cohort until age 100; (2) Duration of effect ranges between 5 years and lifetime.

	Dependent variable:
	ln(Bias)
	(1)	(2)
ln(μDis)	−0.812*** (0.002)	−0.788*** (0.001)
ln(dsHR)	−0.574*** (0.002)	−0.260*** (0.001)
Age	0.023*** (0.0001)	0.067*** (0.0001)
Duration		0.061*** (0.0001)
Constant	−5.237*** (0.007)	−9.870*** (0.006)
Simulated scenarios	40,400	202,000
R2	0.919	0.918

Notes: S.E. in parentheses;

^*p<0.1;

^**p<0.05;

^***p<0.01.

ln: Natural logarithm; μ_Dis: Disease-specific mortality rate; dsHR: Disease-specific hazard ratio.

Figure 3 shows dsHR as a function of oHR for different starting ages of the cohort, values of disease-specific mortality (μ_Dis), and time horizons for a hypothetical clinical trial. The difference between dsHR and oHR increases as μ_Dis deceases, the starting age of the cohort increases and the length of the trial increases.

Figure 3:

Disease-specific hazard ratio (dsHR) calculated as a function of an overall hazard ratio (oHR) for different values of disease-specific mortality (μ_Dis), average initial age of the trial’s study population and time horizons for a hypothetical clinical trial.

3.2. Cost-effectiveness analysis model

We used the estimates shown in Table 2 to evaluate the cost-effectiveness of ICDs for post-MI patients using efficacy information from a clinical trial assuming both oHR and dsHR approaches. Furthermore, we tested the sensitivity of the results to different relationships between non-sudden cardiac death and background mortality that increases with age. We estimated the oHR extrapolation bias as the percentage change in gains in LE, QALE, and costs, as well as the percent change in the ICER. The cohort of patients receiving implantable ICDs has an increase in LE, QALE, and costs relative to conventional therapy under both effectiveness approaches. Under the oHR approach patients with ICDs gain 1.6 years in LE and 1.4 years in QALE compared to conventional therapy. Patients receiving ICDs modeled under the dsHR approach gain 1.5 years in LE and 1.3 years in QALE compared to conventional therapy. The difference in LE and QALE gains between oHR and dsHR approaches compared to conventional therapy represents a bias of 8.8%. Costs are also higher for patients receiving ICD under both modeling approaches, although the magnitude of the bias is not as big as for LE and QALE (2.6%). The difference in magnitude between costs and health outcomes is in part due to the upfront fixed costs of the intervention that are the same under both approaches, such as initial hospitalization costs and cost of the ICD. The difference in health outcomes for patients with ICD under an oHR approach compared to a dsHR translated into a lower ICER for the oHR approach compared to the dsHR approach. The bias in the ICER was-5.7%, meaning that modeling the ICD’s effectiveness using an oHR approach reduced the incremental cost per QALY gained by the intervention compared to a dsHR approach. The outcomes and oHR extrapolation bias for the CEA of ICDs are shown in Table 4.

Table 4:

Discounted^† expected outcomes and bias for the CEA of ICD using efficacy estimates from a clinical trial.

Strategy	Outcome
	LYs	QALYs	Cost ($)	ICER ($/QALY)
No intervention§	5.8	5.1	60,430	-
ICD under oHR approach	7.4	6.5	110,364	35,104
ICD under dsHR approach	7.3	6.4	109,104	37,227
Bias¶	8.8%	8.8%	2.6%	−5.7%

Notes:

^†Annual discount factor is 3%.

^§No intervention refers to conventional therapy.

^¶The bias is measured as the percentage change in LE and QALE gains, incremental costs, and ICERs with ICD vs. no intervention under an oHR approach compared to a dsHR approach.

4. DISCUSSION

We assessed the implications of modeling the long-term effectiveness of healthcare interventions assuming that the reduction in all-cause mortality observed in an RCT persists beyond the time horizon of the trial and proposed a new approach to derive a more accurate estimate of the effectiveness of the intervention on disease-specific mortality. When background mortality increases with age and all-cause mortality is an additive function of background mortality and a constant disease-specific mortality, the relative proportion of disease-specific mortality decreases with age. Thus, extending the oHR observed in a trial (when the contribution of background mortality is relatively small) beyond the time horizon of the trial will result in biased estimates of health outcomes such as LE gains or QALE gains with the intervention.

Using a simple disease-treatment model we quantified the magnitude of the oHR extrapolation bias as a function of the start age of the cohort, the annual disease-specific mortality rate, the dsHR, and the duration of the intervention effect. The magnitude of the bias was most sensitive to the disease-specific mortality rate. The bias increased as both the disease-specific mortality rate and dsHR decrease, and as age of the cohort and the duration of the effect increased. This relationship can be explained in two ways. On the one hand, larger values of disease-specific mortality would result in shorter life expectancy and thus the degree of extrapolation would be less. On the other hand, the older the starting age of the cohort, the relative contribution of background mortality to all mortality increases; thus, the effect of oHR on background mortality increases. The magnitude of the bias was greatest for low values of the disease-specific mortality rate and dsHR assuming a lifetime effect.

We found that the oHR approach overestimated the benefit of an intervention (as measured by LE or QALE gains). By modeling the intervention’s effectiveness under a dsHR approach, individuals only benefit from the intervention in reducing the disease-specific mortality rate, even though the dsHR estimate is further away from 1 than the oHR estimate. In contrast, under an oHR approach individuals face a constant reduction in all-cause mortality over the assumed time for which the intervention will have an effect. If this time is sufficiently long, individuals with the disease receiving the intervention could potentially face a lower mortality compared to those without the disease. For example, we found in our simple disease-treatment model that at age 81 the cohort receiving the intervention under the oHR approach had the same mortality rate it would have had if it never had the disease, assuming a lifetime intervention effect (Figure 4).

Figure 4:

Plot with mortality rates for simulated cohorts of individuals over different scenarios. Rx: Intervention; dsHR: Disease-specific hazard ratio; oHR: Overall hazard ratio.

To show the difference in expected life years over time for a cohort with disease receiving the hypothetical intervention under the two modeling approaches, we plotted the cumulative expected life years over time using the base-case parameter values shown in Table 1 in our simple disease-treatment model (Figure 5). The difference in life years between both approaches was 1.6 years, which represents a 7.9% increase over 50 years.

Figure 5:

Percentage difference in life expectancy (LE) between the two intervention’s effectiveness approaches for a simulated cohort of 50-year-old patients.

It is often the case that clinical trials do not report estimates of dsHR. In these cases, dsHR should be estimated as a function of oHR and other parameters often reported in clinical trials. To facilitate these calculations, we provide an R package (https://github.com/feralaes/dshr) that implements the equations provided in this manuscript. In the absence of these parameters (or when the assumptions that go into the calculation of dsHR are of concern), using the oHR as an estimate of dsHR could be an alternative and conservative approach that would result in a conservative bias. Using oHR as dsHR implies an intervention effectiveness of ${\mu }_{ASR}\left(t\right)+{\mu }_{Dis}\ast oHR$, which would result in a larger mortality rate with intervention compared to the dsHR approach, yielding an underestimation of the LE gain. We recommend that analyst present this approach as a scenario analysis. To illustrate an example, consider the hypothetical cohort for our simple disease-treatment model using the base-case values, using an oHR as an estimate of dsHR resulted in a bias of-11.7% in the LE gain from the intervention.

In the CEA example, we found that modeling the ICD’s effectiveness using an oHR approach reduced the incremental cost per QALY gained by the intervention compared using a dsHR approach. This is of particular relevance as many decisions in healthcare take into consideration the ICER of the intervention compared to conventional therapy (i.e., standard of care). If the ICER of the intervention using an oHR approach turns out to be lower than what it should be, decision makers are more likely to consider an intervention as cost-effective when it might not be below a particular willingness-to-pay threshold.

Our analysis has several limitations. We used relatively simple decision models to demonstrate the effects of the oHR extrapolation bias. Accordingly, we made several assumptions about the disease-specific mortality rate and dsHR. For example, the disease-specific mortality rate likely varies with age and assuming a constant parameter might not be an accurate representation of the disease. However, it is often the case that there is not enough evidence to estimate a time-dependent disease-specific component to long-term mortality. From our results we infer that the closer the disease-specific mortality is to varying at the same rate as background mortality (assuming enough information is available to estimate time-dependent disease-specific mortality) the smaller the bias. If analysts had access to individual-level data then disease-specific mortality rates could be estimated by fitting parametric (or semi-parametric) poly-hazard models as proposed previously by Benaglia et al. [21]. The proposed decision model for the CEA of ICD is based on a CEA proposed more than a decade ago and might not represent current modeling practices of intervention’s effectiveness. However, this model structure has been used in recent CEAs of ICD [39], and effectiveness of ICD are still being reported as oHR [40] and used in recent CEAs [41]. Therefore, our exposition of the potential bias of using oHR as an intervention’s effectiveness beyond the period of clinical trials and our proposed corrections are still relevant. In addition, in our simple disease-treatment model and CEA, we did not let dsHR (or oHR) to vary by age or other characteristics of the population such as sex or race. dsHR could vary by these and other confounders and should be estimated considering these factors.

In summary, we quantified the oHR extrapolation bias as a function of the start age of the cohort, the disease-specific mortality rate, dsHR and the duration of the intervention effect for a simple disease-treatment model. We also propose a new approach to derive an expression of dsHR as a function of the estimated oHR and the characteristics of the trial’s study population to avoid the oHR extrapolation bias. We found that the oHR extrapolation bias has an impact in health and economic outcomes such as LE and QALE gains, incremental costs, and ICERs, which could ultimately influence the decision of the optimal strategy. We recommend modeling the effectiveness of healthcare interventions using a dsHR and whenever there is not a direct estimate of dsHR from a clinical trial, it should be derived from the oHR and the projected background mortality of the study population.

Key Points for Decision Makers.

• The use of an overall hazard ratio (oHR) to model intervention’s effectiveness beyond the time horizon of the trial might allow that diseased individuals will face a lower mortality than if they never had the disease, which will overestimate the benefit of the intervention.

• Using a disease-specific hazard ratio (dsHR) is a more accurate modeling approach to extrapolate the intervention’s effectiveness. If dsHR is not estimated in a trial it should be derived from the oHR and the projected background mortality of the study population.

• We quantify the bias associated with using oHR as an intervention’s effectiveness beyond the time horizon of the clinical trial and derive an expression of dsHR as a function of the estimated oHR and the characteristics of the trial’s study population.