Clinical Value of Predicting Individual Treatment Effects for Intensive Blood Pressure Therapy: A Machine Learning Experiment to Estimate Treatment Effects from Randomized Trial Data

Abstract

Background

The absolute risk reduction (ARR) in cardiovascular events from therapy is generally assumed to be proportional to baseline risk—such that high-risk patients benefit most. Yet newer analyses have proposed using randomized trial data to develop models that estimate individual treatment effects. We tested two hypotheses: first, that models of individual treatment effects could reveal that patients may most benefit from intensive blood pressure therapy that are proportional to baseline risk; and second, that a machine learning approach designed to predict heterogeneous treatment effects—the X-learner meta-algorithm—is equivalent to a conventional logistic regression approach.

Methods and Results

We compared conventional logistic regression to the X-learner approach for prediction of 3-year CVD event risk reduction from intensive (target systolic blood pressure <120 mmHg) versus standard (target <140 mmHg) blood pressure treatment, using individual participant data from the SPRINT (N=9361) and ACCORD-BP (N=4733) trials. Each model incorporated 17 covariates, an indicator for treatment arm, and interaction terms between covariates and treatment. Logistic regression had lower C-statistic for benefit than the X-learner (0.51 [95% CI: 0.49, 0.53] versus 0.60 [95% CI: 0.58, 0.63], respectively). Following the logistic regression’s recommendation for individualized therapy produced restricted mean time until CVD event of 1065.47 days [95% CI: 1061.04, 1069.35], while following the X-learner’s recommendation improved mean time until CVD event to 1068.71 days [95% CI: 1065.42, 1072.08]. Calibration was worse for logistic regression; it over-estimated ARR attributable to intensive treatment (slope between predicted and observed ARR of 0.73 [95% CI: 0.30, 1.14], versus 1.06 [95% CI: 0.74, 1.32] for the X-learner, compared to the ideal of 1). Predicted ARRs using Cox were generally proportional to baseline pre-treatment cardiovascular risk, whereas the X-learner observed—correctly—that individual treatment effects were often not proportional to baseline risk.

Conclusions

Predictions for individual treatment effects from trial data reveal that patients may experience ARRs not simply proportional to baseline CVD risk. Machine learning methods may improve discrimination and calibration of individualized treatment effect estimates from clinical trial data.

Introduction

While models to predict the absolute risk of a disease have been developed for over five decades, the increasing availability of individual participant data from randomized trials has led to a new type of modeling: the development of benefit/risk models designed to help clinicians predict whether an individual patient is more likely to experience meaningful benefits or risks from a therapy (that is, whether they may be at the left tail or right tail of the distribution around the average treatment effect in the trial) ^1–6. Identifying such heterogeneous treatment effects (HTEs)—or systematically different benefits or risks from a medical therapy among some participants in a study, as compared to the study average result—is clinically important for the application of randomized trial data to patient care ⁷. Identifying HTEs can provide clinicians with critical information about whether a therapy with positive average benefit in a trial would be expected to have a similarly positive benefit in a given patient, even greater benefit, no benefit at all, or even harm. Conversely, identifying HTEs can help identify whether a therapy found to have no significant average benefit in a trial may nevertheless be useful for a subset of patients.

Numerous articles have highlighted that the absolute risk reduction in cardiovascular events from therapy is generally assumed to be proportional to a person’s baseline risk—such that high-risk patients will benefit most ^8,9. In this context, it is questionable whether models for individual treatment effects are necessary at all, or whether clinicians can safely assume that a baseline risk model, and the average treatment effect reported in the trial, will generally predict the absolute benefit of a treatment for their patient.

If an individual treatment effect model has value, a methodological question is what approach can best estimate HTEs to incorporate into an individual treatment effect model. Analysts have recommended developing a model by regressing the primary outcome against trial participant characteristics (demographics, biomarkers, etc.), an indicator variable for treatment arm (capturing the average treatment effect), and interaction terms between treatment arm and characteristics (identifying HTEs). Identifying relative HTEs using this multivariable approach has been observed to increase the power of subgroup analyses, as well as to enhance the clinical utility of randomized trial results ¹⁰. Hence, benefit/risk equations for therapeutic decision-making, derived using the multivariable approach, have been increasingly published in high-profile medical journals for treatments such as statin therapy, intensive blood pressure therapy, and antiplatelet therapy ^1,3,5. Yet HTE effects may be biased if equations are overfit to a single trial population, or if they erroneously include irrelevant interaction terms due to collinearity ^11,12. Important covariates may not be known in advance, and complex interactions between many subtle covariates may be missed when performing a standard linear regression—which tends to privilege simple combinations of well-characterized risk factors over complex, non-linear, or subtle combinations of factors¹².

To overcome these limitations, machine learning (ML) approaches have been increasing considered for identifying HTEs. ML approaches can identify complex combinations of non-linear interactions that may predispose to an outcome in a subtle way, but also use estimation and cross-validation methods to avoid overfitting. Although ML approaches have been increasingly applied to predict overall disease risk using large datasets (e.g., early disease detection through analysis of electronic medical records or registries) ^12–16, they have not been rigorously evaluated for specifically detecting HTEs from clinical trial data.

Here, we tested two hypotheses: first, that models of individual treatment effects would reveal that benefits from intensive blood pressure therapy are simply proportional to baseline risk—such that individual risk models would not offer clinical value that routine cardiovascular risk models do not; and second, that a machine learning approach specifically designed to estimate heterogeneous treatment effects from individual participant data in clinical trials—a meta-algorithm known as the X-learner¹⁷—would not be more accurate than a conventional regression approach typically used to create clinical decision rules.

Methods

Our code and analytic methods have been made available to other researchers for purposes of replicating the procedure¹⁸. The data has been made available to other researchers for purposes of reproducing the results, at https://biolincc.nhlbi.nih.gov/home.

We compared metrics of HTE discrimination (for higher versus lower treatment absolute risk reduction, ARR) and calibration (for degree of estimated treatment ARR versus observed treatment ARR) when estimating the cardiovascular disease (CVD) risk reduction over three years from intensive blood pressure treatment (targeting a systolic blood pressure <120 mmHg) versus standard treatment (targeting systolic pressure <140 mmHg). To do so, we developed HTE models from the SPRINT and ACCORD-BP randomized trials^19,20. We focused on a comparison between a conventional logistic regression approach and the X-learner machine learning approach, though in sensitivity analyses (see Supplementary Methods), we explored Cox regression as well as alternative machine learning methods.

Data

We employed individual participant data from the SPRINT and ACCORD-BP trials. The SPRINT trial (N = 9361) was a randomized, controlled, open-label trial of intensive versus standard blood pressure treatment among adults without type 2 diabetes mellitus, conducted at 102 clinical sites in the United States between November 2010 and August 2015¹⁹. The trial was stopped early after a median follow-up of 3.3 years due to a significantly lower rate of the primary composite CVD outcome in the intensive treatment group than in the standard treatment group. Inclusion criteria for the SPRINT trial included: age at least 50 years, systolic blood pressure 130 to 180 mm Hg, and increased CVD event risk [defined as either clinical or subclinical CVD other than stroke; chronic kidney disease, excluding polycystic kidney disease, with an estimated glomerular filtration rate (eGFR) of between 20 and 60 ml/min/1.73m²; a 10-year Framingham risk score of at least 15%; or age at least 75 years]. Exclusion criteria included having diabetes mellitus or a prior stroke. By contrast, the ACCORD-BP trial (N = 4733) was a randomized, controlled, open-label trial of intensive versus standard blood pressure treatment among adults with type 2 diabetes mellitus, conducted at 77 clinical sites in the United States and Canada between January 2003 and June 2009, with a mean follow-up of 4.7 years²⁰. Inclusion criteria for the ACCORD-BP trial included: age at least 40 years with CVD or at least 55 years with anatomical evidence of substantial atherosclerosis, albuminuria, left ventricular hypertrophy or at least two additional CVD risk factors (dyslipidemia, hypertension, smoking, or obesity); systolic blood pressure 130 to 180 mm Hg taking three or fewer blood pressure agents and having a 24-hour protein excretion rate less than 1g; and type 2 diabetes mellitus with a hemoglobin A1c level of at least 7.5%. Exclusion criteria included having a body mass index greater than 45 kg/m², serum creatinine greater than 1.5 mg/dL, or other serious illness. Because the published composite primary outcomes differed between the SPRINT and ACCORD-BP trials, we utilized the disaggregated outcome variables in the ACCORD-BP dataset to construct the CVD composite outcome matching the SPRINT definition, which was myocardial infarction, other acute coronary syndromes, stroke, heart failure, or death from cardiovascular causes¹⁹. For the purposes of derivation and validation we employed a combined dataset of SPRINT and ACCORD-BP, with an indicator variable for trial equal to 1 for the ACCORD-BP participants and 0 for SPRINT participants; summary statistics of the combined and individual datasets are available in Supplemental Table 1. We also repeated derivation and validation using just one or the other of the trial datasets, as a robustness check.

Model Development

Inverse probability of censorship weighting.

To handle the time-to-event nature of the trial data, we dichotomized outcomes as occurrence of CVD event within 3 years, and used inverse propensity of censorship weighting (IPCW) to account for an informative censoring distribution^21,22. Specifically, a Cox regression model was fit using all covariates to estimate participant-specific distributions over time to censoring. Participants whose outcomes were censored prior to the time of interest were excluded from the dataset used to fit models. Remaining participants were weighted by the inverse of their estimated probability of not being censored by the time of interest (if CVD event did not occur before the time of interest), or by the time of event (if CVD event did occur before the time of interest).

Conventional approach.

The conventional approach involved a logistic regression model which included all 17 potential covariates, a dummy variable for treatment arm, and all possible interaction terms between the dummy variable for treatment arm and each covariate. This approach was chosen to reflect the strategy adopted by numerous recent HTE modeling analyses using clinical trial data^1,2,23. It is distinct from the alternative risk-based multivariable HTE assessment approach that involves only a single composite interaction term between a well-established risk score for a primary outcome (e.g., the Framingham risk score for cardiovascular disease, or its more recent variants) and the treatment^24,25.

Machine learning approach.

We compared the conventional approach of developing a logistic regression model, above, to a range of alternative machine learning algorithms adopted in the literature (Supplementary Methods). The primary algorithm we focus on in the main text is the X-learner approach with random forest base learners^17,26. The X-learner is a meta-algorithm specifically designed for estimating individual treatment effects. We trained the learner through a three-step process: we first estimated expectations of outcomes given predictors under control and treatment separately, then estimated imputed treatment effects as the difference between expected outcomes and actual outcomes for each individual, and finally predicted the treatment effects as weighted averages of the estimated imputed treatment effects (Figure 1)¹⁷.

Figure 1:

Conceptualization of the X-learner approach to detect heterogeneous treatment effects from trial data. We first estimate expectations of outcomes given predictors under control and treatment separately, then estimate imputed treatment effects (defined as the difference between expected outcomes and actual outcomes), and finally predict individual treatment effects as weighted averages of the estimated imputed treatment effects. For each step in the process we use random forests with 1000–2000 trees as base learners, and predict out-of-bag samples to prevent over-fitting.

These estimation steps could be performed using any machine learning or regression method as base learners. We chose to employ random forests, in which decision trees were built that separate the population into subsets based on combinations of characteristics that help identify those with higher versus lower treatment effects, and then an individual’s treatment effect was the average of thousands of trees built from subsamples of the dataset. We chose the X-learner with random forests as base learners because random forests have been known to outperform older methods by inherently accounting for interactions among multiple variables as branches of each decision tree in the forest²⁷.

In sensitivity analyses (see Supplementary Methods), we explored alternative common machine learning methods and validated the superiority of the X-learner approach. We note as well that the X-learner has been proven to be unbiased in predicting the difference in treatment effect between study arms, unlike older machine learning methods that can be biased by focusing on variables that predict on the absolute rate of events (e.g., risk of CVD) rather than heterogeneous treatment effects (e.g., how individual features affect the treatment’s ability to reduce the risk of CVD)^17,27.

Outcome metrics

Our pre-specified outcome metrics included: (i) the C-statistic for benefit²⁸, which is a variant of the common C-statistic (area under the receiver operating characteristic curve) designed to specifically calculate the ability for a model to discriminate between people having more benefit versus less benefit from a treatment (rather than just higher versus lower overall CVD risk), (ii) the decision value of the 3-year restricted mean survival time (RMST)²⁹, which is an unbiased estimate of the RMST for patients under the policy implied by the model (that is, treat patients with predicted absolute risk reduction suggesting lower CVD risk with intensive treatment and do not treat patients with zero or negative predicted absolute risk reduction), and (iii) the slope of the calibration line between predicted absolute risk reduction (in quintiles) and observed absolute risk reduction.

The C-statistic for benefit was calculated by first matching patient pairs across the two study arms on predicted risk reduction at 3 years, then calculating the proportion of matched pairs with unequal observed benefit, in which the pair receiving greater treatment benefit was observed to do so. The decision value of the 3-year RMST was calculated by computing the Kaplan-Meier estimates of survival curves for subgroups of predicted benefit (predicted absolute risk reduction suggesting lower CVD risk with intensive treatment) and no benefit (zero or negative predicted absolute risk reduction). The area under each Kaplan-Meier curve, up to 3 years, is defined as the RMST for each subgroup. We then take the average RMST over all participants of the subgroup to which they were assigned – this is an unbiased estimate of the RMST under the treatment choice recommended by the model. The calibration curve was constructed by aggregating participants into quintiles of predicted absolute risk reduction (where quintiles were chosen to ensure adequate sample size for stable curve estimates²⁸). For each quintile, observed absolute risk reduction was defined as the difference between the intensive versus standard treatment arm CVD rate at 3 years, using Kaplan-Meier estimates to account for censoring. Discrimination was further assessed by comparing bootstrap confidence intervals for observed ARRs in each of the subgroups of predicted benefit and no benefit. Lastly, predicted treatment effects in both the logistic regression approach and the X-learner approach were compared to deciles of baseline pre-treatment cardiovascular risk, where the latter was estimated using the ACC/AHA ASCVD risk score estimator²⁵, to assess whether treatment effects were simply proportional to baseline CVD risk (higher benefit for participants with higher baseline risk), thus potentially obviating the need for an HTE model. We also compared predicted treatment effects to deciles of baseline pre-treatment cardiovascular risk where the latter was estimated using 10-year Framingham risk scores²⁴.

It is well-known that models validated using the same data upon which they were trained will be too optimistic due to over-fitting³⁰. To account for this, the logistic regression model was corrected for optimism by calculating C-for-benefit and decision value RMST over 250 bootstrap samples of the dataset, stratified by treatment arm and presence of CVD event. The average difference between performance on the bootstrap dataset (upon which the logistic regression model was derived) and the original dataset (upon which the model was validated) was defined as the optimism²⁸, and subtracted from apparent performance estimates. In the X-learner approach, we calculated predicted risk reduction for each participant only using trees in the random forest base learners for which that participant was out-of-sample, so no participant was used for both derivation and validation of any individual tree. As a result, no optimism correction was necessary for the X-learner approach. All confidence intervals were constructed by bootstrap re-sampling of the dataset, stratified by treatment arm and presence of CVD event.

Analyses were performed in Python (version 3.6.2, The Python Software Foundation, Wilmington, Delaware) and R (version 3.5.1, The R Foundation for Statistical Computing, Vienna, Austria).

Results

Conventional approach

The conventional logistic regression model produced a model with 17 main terms, a dummy variable for the treatment arm, and 17 HTE interaction terms between the main terms and the treatment arm variable. In descending order, the interaction terms with largest HTE effect sizes (based on the absolute change in effect size based on a 1 unit change in covariate Z-score) were the interaction term between treatment and statin usage (with statin usage corresponding to less benefit), sex (with being female corresponding to less benefit), HDL (with higher HDL corresponding to more benefit), and DBP (with higher DBP corresponding to more benefit). No single variable in isolation clearly predicted higher or lower treatment benefit; rather, the groups with benefit and with no benefit had similar demographic and clinical characteristics in univariate analysis, except for non-significantly higher baseline systolic blood pressure and fewer ACCORD participants in the group predicted to benefit from intensive treatment (Table 1).

Table 1:

Summary statistics of participants in the combined dataset of the SPRINT and ACCORD-BP trials, partitioned into predicted sub-groups of benefit or no benefit as determined by machine learning (left) and conventional (right) methods.

Covariates	Mean [SD] using Machine Learning		Mean [SD] using Conventional
	Benefit (N=9763)	No benefit (N=3841)	Benefit (N=11029)	No benefit (N=2575)
Age, yrs	66.67 (9.13)	65.34 (8.07)	67.51 (8.85)	61.08 (6.79)
Female, fraction	0.39 (0.49)	0.43 (0.50)	0.34 (0.47)	0.66 (0.47)
Black, fraction	0.30 (0.46)	0.26 (0.44)	0.29 (0.45)	0.28 (0.45)
Hispanic, fraction	0.09 (0.29)	0.10 (0.30)	0.06 (0.23)	0.26 (0.44)
Systolic blood pressure (mmHg)	141.20 (16.06)	135.67 (13.75)	140.66 (15.61)	135.25 (15.01)
Diastolic blood pressure (mmHg)	78.93 (11.23)	73.58 (11.16)	78.15 (11.62)	74.30 (10.19)
Number of blood pressure treatment classes	1.78 (1.06)	1.81 (1.05)	1.80 (1.06)	1.76 (1.02)
Current Smoker, fraction	0.10 (0.30)	0.07 (0.26)	0.10 (0.30)	0.07 (0.26)
Former Smoker, fraction	0.44 (0.50)	0.46 (0.50)	0.47 (0.50)	0.35 (0.48)
Aspirin, fraction	0.51 (0.50)	0.53 (0.50)	0.53 (0.50)	0.47 (0.50)
Statin, fraction	0.45 (0.50)	0.65 (0.48)	0.43 (0.50)	0.82 (0.39)
Serum creatinine (mg/dL)	1.01 (0.31)	1.04 (0.35)	1.03 (0.29)	1.00 (0.43)
Total cholesterol (mg/dL)	191.70 (41.51)	189.33 (43.54)	192.00 (41.64)	186.86 (43.83)
High-density lipoprotein cholesterol (mg/dL)	51.78 (14.29)	48.27 (14.47)	52.39 (14.50)	43.94 (11.91)
Triglycerides (mg/dL)	133.13 (79.20)	180.02 (191.34)	127.95 (73.82)	225.29 (223.00)
Body mass index (kg/m2)	30.69 (5.79)	30.58 (5.73)	30.59 (5.88)	30.93 (5.31)
ACCORD-BP participants, fraction	0.27 (0.44)	0.50 (0.50)	0.26 (0.44)	0.65 (0.48)

The benefit bucket consists of participants predicted to have absolute risk reduction (ARR) > 0, and the no benefit bucket consists of participants predicted to have ARR ≤ 0.

The conventional logistic regression model had a corrected (bootstrap-adjusted) C-statistic for benefit of 0.51 (95% CI: 0.49, 0.53; Table 2), meaning that it was only slightly better than a flip of a coin for correctly discriminating which of two participants in the dataset would have more versus less absolute risk reduction from intensive blood pressure therapy. Note that the C -for-benefit statistic is in general much more conservative than the traditional C-statistic, which only assess the ability of a model to detect higher versus lower absolute CVD risk, not absolute risk reduction; the traditional C-statistic is determined primarily by the main effect terms (e.g., age, sex, tobacco smoking) rather than the HTE interaction terms of a model (e.g., interaction between treatment arm and HDL, or treatment arm and statin use). The corrected (bootstrap-adjusted) decision value of the 3-year restricted mean survival time was 1062.86 days (95% CI: 1058.43, 1066.74), which is the estimate of the 3-year RMST under the treatment recommended by the model (that is, treat participants with positive predicted risk reduction under treatment and do not treat patients with zero or negative predicted risk reduction). For comparison, a baseline policy of prescribing intensive treatment for participants in the SPRINT trial and standard treatment for those in the ACCORD-BP trial results in 3-year decision RMST of 1061.24 days (95% CI: 1057.37, 1064.10).

Table 2:

Discrimination and calibration metrics for risk reduction predictions [95% confidence intervals].

		Machine Learning	Conventional
Discrimination
	Apparent C-for-benefit (higher is better)	0.60 [0.58 0.63]	0.54 [0.52 0.56]
	C-for-benefit optimism	0.00	0.03
	Corrected C-for-benefit	0.60 [0.58 0.63]	0.51 [0.49 0.53]
	Apparent decision value RMST, days (higher is better)	1068.71 [1065.42 1072.08]	1065.47 [1061.04 1069.35]
	Decision value RMST optimism, days	0.00	2.61
	Corrected decision value RMST, days	1068.71 [1065.42 1072.08]	1062.86 [1058.43 1066.74]
Calibration
	Slope (ideally 1)	1.06 [0.74 1.32]	0.73 [0.30 1.14]
	Intercept (Ideally 0)	−0.00 [−0.01 0.00]	0.00 [−0.01 0.01]

For discrimination, we calculated the C-for-benefit statistic and adjusted for optimism with bootstrap samples. The conventional method required correction because it was both trained and validated on the same dataset, while the machine learning method does not because we evaluated out-of-sample predictions. We also calculated the corrected 3-year decision value RMST (in days), and note that a baseline policy of prescribing intensive treatment for in the SPRINT trial and standard treatment for those in the ACCORD-BP trial results in 3-year decision RMST of 1061.24 days (95% CI: 1057.37, 1064.10). For calibration, we recorded the slope and intercept of the calibration curve fitted to quintiles of predicted risk reduction. Confidence intervals and optimism were calculated through 250 bootstrap samples. RMST: restricted mean survival time.

Discrimination was further assessed by bucketing participants into subgroups of predicted benefit (those with predicted ARR suggesting lower CVD risk with intensive treatment) and no benefit (those with zero or negative predicted ARR). We found that the conventional logistic regression model was able to have a positive observed risk reduction of 0.0225 (a 2.3% absolute risk reduction, 95% CI: 1.8%, 2.7%) among the group predicted to have benefit. However, the conventional Cox model was not as accurate in predicting the no benefit group, which had an observed risk reduction confidence interval crossing the zero observed risk reduction line, and an average risk increase of 0.0107 (1.1% absolute risk increase) but wide 95% confidence intervals spanning from a 2.7% increase to a 0.8% decrease (Figure 2).

Figure 2:

Participants with uncensored outcomes at 3 years are grouped into predicted buckets of benefit (ARR > 0) and no benefit (ARR ≤ 0) via machine learning and conventional methods. Using these buckets, we bootstrap 95% confidence intervals for the observed absolute risk reduction (ARR) in each bucket, and calculate corresponding P values via the Wald test. The X-learner machine learning approach yielded more discriminative buckets than the conventional logistic regression approach, with the benefit bucket exhibiting higher observed ARR and the no benefit bucket exhibiting lower observed ARR.

The conventional Cox model also had worse calibration, by over-estimating the ARR attributable to intensive blood pressure treatment (Table 2; Figure 3). The slope of the calibration line between the predicted ARR (x-axis on Figure 3) and observed ARR (y-axis on Figure 3, difference between intensive and standard treatment arm CVD event rates in quantiles of predicted risk reduction) was 0.73 (95% CI: 0.30, 1.14), as compared to the ideal of 1 (Table 2).

Figure 3:

Calibration plots for predicted absolute risk reduction versus observed absolute risk reduction (difference between intensive and standard treatment arms in cardiovascular event rates) using the X-learner machine learning approach and the conventional logistic regression approach, evaluated at quintiles of predicted absolute risk reduction and using Kaplan-Meier estimates to adjust for censoring.

Predicted treatment effects using the traditional logistic regression approach were generally proportional to baseline pre-treatment cardiovascular risk, with the range of predicted treatment benefit being proportionately higher with higher baseline risk (Figure 4, Supplemental Table 2). When compared to baseline pre-treatment cardiovascular risk as defined by Framingham risk scores instead of ACC/AHA ASCVD risk scores, results did not meaningfully differ (Supplemental Figure 1).

Figure 4:

Predicted risk reduction across deciles of predicted baseline CVD risk. We employed predictions of baseline risk calculated by ACC/AHA ASCVD risk score estimates to group trial participants into baseline risk deciles and compare the median and quartiles of predicted absolute risk reduction (ARR) from intensive blood pressure treatment across each decile. The logistic regression approach generally predicted absolute risk reduction to be proportional to baseline risk, whereas the X-learner machine learning approach predicted wider ranges of risk reduction per decile, not necessarily proportional to baseline risk.

When the discrimination and calibration outcome metrics were computed separately on the SPRINT and ACCORD-BP datasets, rather than on the combined set alone, results did not meaningfully differ between the trials (Supplemental Tables 3-4, Supplemental Figures 2-3). When outcome metrics were computed for 5-year outcomes instead of 3-year outcomes using the ACCORD-BP dataset (the trial for which the majority of participants had follow-up times through 5 years, unlike SPRINT), results did not meaningfully differ (Supplemental Table 5).

Machine learning approach

The X-learner machine learning approach predicted more benefit from intensive blood pressure treatment for older participants, those not on statins, and those with lower triglycerides, but few measures differed remarkably between the predicted benefit and predicted no benefit subgroups (Table 1), suggesting complex interactions rather than simple univariate subgroups were identified by the approach.

The X-learner machine learning approach had a corrected C-statistic for benefit of 0.60 (95% CI: 0.58, 0.63; Table 2), which was significantly better than the conventional logistic regression approach for correctly discriminating which of two participants in the dataset would have more versus less absolute risk reduction from intensive blood pressure therapy. The corrected 3-year decision value of restricted mean survival time was 1068.71 days (95% CI: 1065.42, 1072.08), which is higher than the corresponding statistic using the logistic regression approach. For comparison, a treatment approach of prescribing intensive treatment for those in the SPRINT trial and standard treatment for those in the ACCORD-BP trial resulted in a 3-year decision RMST of 1061.24 days (95% CI: 1057.37, 1064.10).

The X-learner approach was able to produce a positive observed ARR of 0.0356 (a 3.6% ARR, 95% CI: 2.9%, 4.2%) among the group predicted to have benefit. Unlike the logistic regression approach, the machine learning approach was also accurate in predicting the no benefit group, which had an observed risk reduction of −0.0313 (3.1% absolute risk increase; 95% CI: 1.6%, 4.7%; Figure 2).

The machine learning approach also had better calibration than the logistic regression model (Table 2; Figure 3). The slope of the calibration line between the predicted absolute risk reduction (x-axis on Figure 3) and observed absolute risk reduction (y-axis on Figure 3) was 1.06 (95% CI: 0.74, 1.32; Table 2).

Predicted treatment effects using the machine learning approach were not generally proportional to baseline pre-treatment cardiovascular risk, with the range of predicted treatment benefit not being proportionately increased with higher baseline risk (Figure 4). When compared to baseline pre-treatment cardiovascular risk as defined by Framingham risk scores, results did not meaningfully differ (Supplemental Figure 1).

When the discrimination and calibration outcome metrics were computed separately on the SPRINT and on the ACCORD-BP datasets, rather than on the combined set alone, results did not meaningfully differ (Supplemental Tables 3-4, Supplemental Figures 2-3).

To aid interpretability of the X-learner approach, we calculated variable importance plots²⁶ (Supplemental Figure 4) and partial dependence plots³¹ (Supplemental Figure 5), which help identify which variables were most critical in estimating the HTEs (and the directionality of the coefficient in terms of increasing or decreasing benefit) and show how much the ARR estimates change across the range of each variable’s values (displaying any non-linearities). As shown in the Supplementary Methods, the X-learner approach using random forest base learners was superior to alternative machine learning methods, including an X-learner with a linear base learners, causal forests, random survival forests, and Cox regression with interaction terms (Supplemental Table 6).

Discussion

We compared a conventional logistic regression approach for modeling heterogeneous treatment effects to a machine learning approach called X-learner, applying both approaches to individual participant data from randomized trials of intensive blood pressure treatment. The machine learning approach also revealed correctly that an individual patient’s predicted absolute benefit from intensive treatment was not necessarily proportional to their baseline CVD risk. This contradicts prior hypotheses that simply calculating baseline risk will be sufficient to guide therapy⁹, highlighting the clinical importance of HTE risk estimation for making individual treatment effect estimates. We believe that the difference in survival between treating everyone as the average in a trial and calculating individual treatment effects is meaningful, given that it is statistically significant and larger than the survival difference often observed in randomized trials of many therapeutics. We also observed that the machine learning approach had significantly better discriminative ability, evident in higher C-for-benefit and decision value RMST statistics. The machine learning approach also partitioned participants into a “benefit” subgroup that observed a higher empirical absolute risk reduction than the “no benefit” subgroup, while the difference between subgroups was more modest for the logistic regression model. Finally, the machine learning approach had better calibration than the logistic regression model, which over-estimated the absolute risk reduction attributable to intensive blood pressure treatment.

Our specific modeling approach poses several advances over prior literature, particularly in rigorously comparing the predicted versus absolute risk reduction from therapy, rather than simply calculating discrimination or calibration statistics on overall CVD event rates¹. This is important because outcome metrics related to overall absolute CVD risk will be driven by major well-known CVD risk factors (e.g., age, sex, tobacco smoking), and smaller interaction terms that critically define HTEs (e.g., interactions between treatment arm and biomarkers) will be lost amidst the more dominant overall CVD risk terms when calculating conventional discrimination and calibration metrics. Hence, an HTE model may appear “good” by traditional C-statistics and calibration curves, but actually have incorrect HTE terms. We found that direct assessments of observed CVD event rate reductions in held-out samples of data, and plotting a calibration curve between predicted and observed risk reduction (rather just predicted versus observed CVD event rates) revealed benefits of the machine learning approach tested here. Our approach also advanced beyond prior papers modeling the SPRINT and ACCORD-BP datasets, in that we found the X-learner approach captured complex interactions and had improved discrimination and calibration as compared to older approaches using logistic regression, which can capture interaction terms but suffer from the limitations of linear modeling, the risk of overfitting, and the risk of false positive results with more complex interactions are tested³⁰. By contrast, the X-learner approach used here uses out-of-sample testing to reduce the risk of overfitting, and a tree-based approach to enable complex non-linear interaction terms to be incorporate¹¹. We present our decision value RMST as a generic outcome metric for evaluating HTE predictions on clinical trial datasets. In future analyses, our method of evaluation can help determine when the X-learner method (or any other HTE estimation method) would provide improvement over the standard assumption that the average result was generalizable to everyone, indicating in which cases HTE analysis is useful or not.

There are nevertheless important limitations to our analysis. First, the machine learning approach applied here, like many machine learning approaches, is more complex and difficult to visualize than standard regression modeling. Hence, we communicated the results of our approach by sharing the statistical code to apply to any given patient or other datasets, as well as by plotting variable importance and partial dependence plots to visualize which variables the learner is using and how it is transforming those variables into estimated absolute risk reductions. The results cannot be easily captured in a single equation, since the X-learner forest modeling approach assembles thousands of decision trees to produce a prediction rather than a single set of coefficients in a linear model, but the X-learner model can be automatically built into websites and electronic medical records for implementation. Ongoing efforts to improve the communication and transparency of machine learning methods are needed as part of further research. Additionally, the SPRINT and ACCORD-BP datasets remain selective populations of trial participants, and are therefore not as varied as patients in real clinical settings. Hence, a necessary next step would be to prospectively compare individuals predicted as benefitting or having no benefits from the various models derived here, and observe their real-world outcomes to assess the generalizability and importance of our findings.

Nevertheless, in the meantime, our results from this study suggest that the X-learner machine learning approach may be superior to conventional logistic regression modeling for estimating HTEs, particularly from the perspective of avoiding miscalibration of HTE models. Using methods sensitive to the interactions between treatment arm and covariates, and derived through repeated cross-validation, may be critical to ensuring accurate models of HTEs, and simply assuming that absolute risk reduction is necessarily proportional to baseline disease risk may not accurately capture true variations in patient risk or benefit from a therapy.

What is Known

• Modelling individualized treatment effects can help clinicians identify which patients are more or less likely to benefit from treatment, as compared to the average result from a randomized trial. It is often assumed that the absolute risk reduction an individual will experience from a cardiovascular treatment will be proportional to their baseline cardiovascular risk.

• We explored how alternative machine learning methods may help to identify individualized treatment effects from randomized trial data, and whether such effects were necessarily proportional to baseline risk.

What the Study Adds

• Machine learning methods revealed that absolute risk reductions from treatment were often not simply proportional to baseline cardiovascular risk, meaning that calculation of individualized treatment effects rather than simply calculating baseline risk would potentially help direct therapy to people most likely to benefit.

• A machine learning method called the X-learner may improve discrimination and calibration of individualized treatment effect estimates from clinical trial data, as compared to other machine learning methods and standard logistic regression modeling.