Spending Analysis of Machine Learning–Based Communication Nudges in Oncology

Abstract

BACKGROUND

Serious illness conversations (SICs) in the outpatient setting may improve mood and quality of life among patients with cancer and decrease aggressive end-of-life care. Interventions informed by behavioral economics may increase rates of SICs between oncology clinicians and patients, but the impact of these interventions on end-of-life spending is unknown.

METHODS

This study is a secondary analysis of a stepped-wedge cluster randomized, controlled trial that involved nine medical oncology practices and their high-risk patients at a large academic institution between June 2019 and April 2020. The study included 1187 patients who were identified by a machine-learning algorithm as high risk of 180-day mortality and who died by December 2020. The patients were randomly assigned to standard of care (controls) or to a behavioral intervention designed to increase clinician-initiated SICs. We abstracted spending — defined as inflation-adjusted costs for acute care (inpatient plus emergency room), office/outpatient care, intravenous systemic therapy, other therapy (e.g., radiation), long-term care, and hospice — from the institution’s accounting system, and we captured spending at inpatient, outpatient, and pharmacy settings. To evaluate intervention impacts on spending, we used a two-part model, first using logistic regression to model zero versus nonzero spending and second using generalized linear mixed models with gamma distribution and log-link function to model daily mean spending in the last 180days of life. Models were adjusted for clinic and wedge fixed effects, and they were clustered at the oncologist level. For all patients with at least one SIC within 6 months of death, we also calculated their mean daily spending before and after SIC.

RESULTS

Median age at death was 68years (interquartile range, 15.5), 317 patients (27%) were Black or of ethnicities other than white, and 448 patients (38%) had an SIC. The intervention was associated with lower unadjusted mean daily spending in the last 6 months of life for the intervention group versus controls ($377.96 vs. $449.92; adjusted mean difference, −$75.33; 95% confidence interval, −$136.42 to −$14.23; P=0.02), translating to $13,747 total adjusted savings per decedent and $13 million in cumulative savings across all decedents in the intervention group. Compared with controls, patients in the intervention group incurred lower mean daily spending for systemic therapy (adjusted difference, −$44.59; P=0.001), office/outpatient care (−$9.62; P=0.001), and other therapy (−$8.65; P=0.04). The intervention was not associated with differences in end-of-life spending for acute care, long-term care, or hospice. Results were consistent for spending in the last 1 and 3 months of life and after adjusting for age, race, and ethnicity. For patients with SICs, mean daily spending decreased by $37.92 following the first SIC ($329.87 vs. $291.95).

CONCLUSIONS

A machine learning–based, behaviorally informed intervention to prompt SICs led to end-of-life savings among patients with cancer, driven by decreased systemic therapy and outpatient spending. (Funded by the Penn Center for Precision Medicine and the National Institutes of Health; ClinicalTrials.gov number, NCT03984773.)

Introduction

Patients with advanced cancer often receive treatment and acute care that is discordant with their preferences at the end of life.^1–3 Serious illness conversations (SICs) with patients with cancer about their goals and treatment preferences may improve their mood and quality of life and decrease health care use.^4–8 However, most patients with cancer die without a documented SIC.⁹ The dearth of SICs with such patients can be attributed to numerous factors, including inaccurate prognostication and lack of clinicians’ willingness to engage in the conversations, which may result in an excess of low-value care.^10–12

Initiating SICs after diagnosis to facilitate care that is concordant with patients’ goals may decrease unwanted care and associated health care expenditures near the end of life.^13,14 Indeed, several retrospective studies have confirmed an association between earlier SICs and reduced net costs of care, including decreased hospital out-of-pocket expenses.^15–17 However, prospective trials of SIC interventions have largely not demonstrated cost savings,^4,18 perhaps because the interventions used were insufficient to target clinicians’ interactions with high-risk patients. To address this short-coming, we conducted a stepped-wedge cluster randomized, controlled trial that evaluated the effect of nudges to clinicians to prompt SICs with high-risk patients. Such patients were previously identified by a validated machine-learning (ML) algorithm to be at high risk for death within 6 months.¹⁹ A preliminary analysis at 16weeks showed that the intervention was associated with significant increases in SIC rates among all patients and high-risk patients,²⁰ and a long-term analysis at 40weeks demonstrated that the intervention decreased end-of-life systemic therapy use.²¹

In this current secondary analysis of the trial, we evaluated the impact of SIC nudges to clinicians on end-of-life spending in a cohort of decedents who died during the trial’s observation period. We hypothesized that patients who received the intervention would have lower daily spending in the last 6 months of life than those who did not receive the intervention.

Methods

STUDY DESIGN

This study represents a secondary analysis of a stepped-wedge randomized trial (NCT03984773) that randomly assigned nine medical oncology practices and their high-risk patients at a large academic institution to a 40-week behavioral intervention designed to increase SICs versus standard of care between June 2019 and April 2020.¹⁹ High-risk patients, defined as having a 10% or higher risk of 180-day mortality, were identified using a prospectively validated ML algorithm. The intervention comprised three components: weekly emails to clinicians comparing their SIC rates against their peers’ rates, weekly lists of six or more forthcoming encounters with high-risk patients, and opt-out reminder texts to clinicians on the morning of encounters with high-risk patients. After a 4-week baseline wedge, in which all groups remained in usual care, groups were randomly assigned to the intervention in 4-week wedges until all clinics received the intervention by the fifth wedge (week 17) and were followed up through week 40. Eligible clinicians were physicians, physician assistants, and nurse practitioners who provided oncology care at eight subspecialty clinics and one general oncology clinic. Clinicians were excluded if they cared only for patients with benign hematologic or genetic disorders, saw fewer than 12 patients classified as high risk by the algorithm in either the preintervention or postintervention period, or had not undergone SIC training at the time of trial initiation. Eligible patients had encounters at one of the clinics during the study period. Patients were excluded if they had a documented SIC or advanced care planning conversation before the start of the trial or if they were enrolled in another ongoing trial of palliative care at the time of advanced cancer diagnosis.

For the current analysis, the sample included 1187 enrolled patients with complete data who died by December 2020. Decedents were assigned to standard of care (controls) or to an intervention group on the basis of the intervention status on the date of their last clinic encounter. A total of 957 (80.6%) patients received the intervention, and 230 (19.4%) received the standard of care. Decedent status was defined using the Social Security Administration Death Master File matched to patients by social security number and date of birth and an institutional registry linked to the National Death Index.^19,22 The study was approved by the institutional review board, which granted a waiver of informed consent owing to minimal risk.

ML ALGORITHM

This study was a secondary analysis of a previously published randomized trial that evaluated the impact of a behavioral intervention informed by a gradient-boosted ML algorithm used to predict 180-day mortality among outpatients with cancer¹⁹; the current study analyzed end-of-life spending among select patients. We trained the model using a cohort of 26,525 adult patients who had outpatient encounters with oncology or hematology/oncology specialties at 1 of 11 University of Pennsylvania Hospital System (UPHS) outpatient sites between February 1, 2016, and July 1, 2016. All data on patient information were abstracted from Clarity, an Epic Systems Corporation reporting database that contains individual electronic health records (EHRs) for patients. Our dataset included three broad classes of variables that are commonly available in EHRs: demographic variables, Elixhauser comorbidities, and laboratory and select electrocardiogram data (Table S1 in the Supplementary Appendix). The full code behind our model is available at https://github.com/pennsignals/eol-onc. For all variables, we used several standard feature selection strategies, such as removing zero-variance features and highly correlated variables, and we identified 559 structured features as inputs. The study population was randomly split into a training cohort of 18,567 patients (70%) and a validation cohort of 7958 patients (30%). In evaluating model performance in the holdout set, the model had adequate discrimination in predicting 180-day mortality, with an area under the receiver operating characteristic curve (AUC) of 0.87 (95% confidence interval [CI], 0.85 to 0.89), and it had greater AUC and positive predictive value than a logistic regression model using backward selection.

The model was validated prospectively on a cohort of 24,582 eligible patients with outpatient oncology encounters between March 1, 2019, and April 30, 2019.²³ A 60-day baseline period was selected, during which the algorithm ran silently for all encounters, and clinicians were not exposed to algorithm predictions. During the baseline period, a database of all required structured EHR data was updated nightly. The algorithm ran automatically once a week on Thursdays at 7:00 a.m. and used EHR data that were updated on the previous night to generate risk predictions of 180-day mortality for each patient whose encounter was scheduled for the coming Monday through Friday during the baseline period. In this full prospective validation cohort, the AUC of the model was 0.89 (95% CI, 0.88 to 0.90), and the model was well calibrated at a threshold of 10% risk of 180-day mortality. Further details regarding algorithm specifications, features, and validation are described in previous publications.^19,23

DATA COLLECTION

Spending information for study participants was abstracted from the hospital cost-accounting system used by UPHS. Hospital spending reported in this study included direct costs, such as for labor, supplies, laboratory, and perioperative administration, as well as indirect costs, such as for facility management and overhead. The data abstracted from the accounting system represented the consumer price index medical inflation-adjusted operating cost of care at UPHS-affiliated clinics and hospitals rather than the amount charged to the patient’s insurance or the amount that the hospital was reimbursed for the patient’s care.

We calculated the overall total and daily health care spending for all patients as well as daily spending separated by the following visit types: acute care use, office/outpatient, systemic therapy, hospice, other therapy, and rehabilitation (rehab) or long-term care (LTC) (Table 1 and Fig. S1).

Table 1.

Care Use and Spending Items Collected from the Randomized Control Trial.^*

Spending Category	Source†	Type of Spending Included	Description
Acute care use	Hospital claims	Emergency department visits, inpatient hospitalizations	Emergency room and hospitalization visits
Systemic therapy	Hospital claims with line items	Intravenous and oral chemotherapy, injectable hormone therapy, and intravenous immunotherapy	Oral/injectable systemic therapy medication and administration
Hospice	Hospital claims	Inpatient hospice	Inpatient hospice services
Office/outpatient	Hospital claims	Office and outpatient visits, endoscopy/bronchoscopy	Clinician visits, including tests and procedures, if applicable
Other therapy	Hospital claims	Day surgery, radiation/oncology	Treatments serviced outside the outpatient and acute care setting
Rehab/LTC	Hospital claims	Long-term acute care hospital, outpatient intensive detoxification, inpatient rehabilitation, wound care	LTC services, skilled nursing care, and rehabilitation

^*LTC denotes long-term care; and Rehab, rehabilitation.

^†Source indicates the codes and cost items used to group together cost sectors. All data were abstracted from the hospital cost accounting system used by the University of Pennsylvania.

We collected current procedural terminology (CPT) codes and dates of service for all services rendered during patient visits throughout the trial period. Acute care use spending included all hospitalizations and emergency department services. Emergency department visit spending included all spending identified by the appropriate CPT codes in addition to spending for other services (i.e., laboratories) during the same encounter. Outpatient physician spending and all services indicated in the system as “outpatient,” except for those associated with systemic therapy, were incorporated in outpatient care spending. All spending associated with hospice care was abstracted from inpatient hospice facilities and UPHS-affiliated hospices, and therefore, it was not included as part of outpatient spending.

Spending information from UPHS pharmacies was abstracted to determine the spending for intravenous systemic therapy, which encompassed both drug and administration spending. Oral systemic therapy costs were determined from hospital and outpatient pharmacy spending. Other therapy spending included costs for day surgery as well as recurring radiation oncology appointments and associated therapies. Spending for rehab/LTC was defined as spending associated with long-term acute care, outpatient intensive detoxification, inpatient rehab services, and wound care during the study period.

OUTCOMES

The primary outcomes of interest were mean total and daily health care spending during the last 6 months of life. Mean spending during the last 3 months and mean spending during the last 1 month of life were secondary outcomes. For each time point, spending was stratified on the basis of the visit types defined: acute care use, office/outpatient, systemic therapy, hospice, other therapy, and rehab/LTC. Per-decedent spending rather than overall spending was analyzed because we expected that the spending impacts of the SIC intervention would be manifested through avoided care at the end of life, which is when most spending occurs for patients with cancer and is a consistent finding across the literature.³ As a result, the mechanism of initiating SICs would directly reduce costs at the end of life by avoiding unnecessary care.

STATISTICAL ANALYSIS

Baseline cohort characteristics were compared across patients in the intervention and control groups using t-tests for continuous variables and chi-square tests for categorical variables. Because of the nonnormal nature of the spending data, for all unadjusted spending outcomes, a nonparametric permutation test with 10,000 permutations was used. Each permutation was sampled without replacement, and the resulting intervention and control frequencies matched the originally observed intervention and control frequencies. To evaluate intervention impacts on spending for each of the visit types and time points, we used a mixed two-part model. The first part modeled a logistic regression with a binary outcome of zero spending versus nonzero spending, and the second part modeled a generalized linear mixed model with a gamma distribution and a log-link function for patients with positive spending. These models controlled for clinic-group effects and wedge period. As a sensitivity analysis, similar two-part models were built with the added covariates of age, race, and insurance; cubic splines with three knots were used for age. In all two-part models, predicted spending was stochastically aggregated across both parts, and all models were clustered by clinician, with an independent correlation structure, because of possible dependence between patients with the same clinician. Type III tests were conducted for all covariates in both model parts, and the marginal difference between intervention and control spending was calculated.

Our primary analysis assessed the spending effect of a behavioral intervention to prompt SICs among high-risk individuals; patients in both intervention and control groups may or may not have received SICs. To estimate the direct impact of an SIC in a per-protocol fashion, for all patients with at least one SIC within 6 months of death, we calculated their mean daily spending before SIC and mean daily spending after SIC as well as the timing of the SIC in both control and intervention groups. We then calculated the mean daily spending change after SIC as the difference between main daily spending before and after SIC. The estimand for this exploratory, hypothesis-generating analysis was the average decrease in spending following an SIC.

SAS v0.4 (SAS Institute) and Stata v17.0 (StataCorp) were used for all analyses. We considered a two-tailed P<0.05 as statistically significant in our primary analysis; in our secondary analyses examining intervention impacts across spending categories, we used the Bonferroni correction for testing four hypotheses and defined significance as a two-tailed P=0.01.

Results

COHORT DESCRIPTION

Of the 1187 patients in our analysis, 448 (38%) received an SIC, the median age of the cohort at death was 68years of age (interquartile range, 61–76.5), and 317 (27%) patients were Black or of ethnicities other than white. Of the 957 patients in the intervention group, 373 (39.0%) received an SIC, and of the 230 patients in the control group, 75 (32.6%) received an SIC. Compared with the control group, the intervention group was older (mean age, 67.7 vs. 64.0years of age; P<0.001) and had higher proportions of female patients (47.5% vs. 43.0%; P<0.001), Black patients (20.7% vs. 13.0%; P<0.001), and Medicare-insured patients (62.8% vs. 48.3%; P<0.001). Baseline characteristics are further summarized in Table 2.

Table 2.

Baseline Cohort Characteristics among Patients in the Intervention and Control Groups.^*

Characteristic	Control	Intervention	Standardized Mean Difference
Total cohort — n (%)	230 (19.4%)	957 (80.6%)	1.55
Age at death — yr, mean (SD)	64.02 (12.1)	67.74 (12.3)	0.31
SIC before death — n (%)	75 (32.6%)	373 (39.0%)	0.13
Female patients — n (%)	99 (43.0%)	455 (47.5%)	0.09
Race — n (%)
White	180 (78.3%)	690 (72.1%)	−0.14
Black	30 (13.0%)	198 (20.7%)	0.21
Other	20 (8.7%)	69 (7.2%)	−0.06
High-risk before death — n (%)	180 (78.3%)	696 (72.7%)	−0.13
Insurance — n (%)
Commercial	105 (45.7%)	307 (32.1%)	−0.28
Medicare	111 (48.3%)	601 (62.8%)	0.30
Medicaid	14 (6.1%)	49 (5.1%)	−0.04
Disease focus of oncology clinic — n (%)
Breast	67 (29.1%)	214 (22.4%)	−0.16
Central nervous system plus melanoma	52 (22.6%)	179 (18.7%)	−0.10
Gastrointestinal	29 (12.6%)	95 (9.9%)	−0.08
General oncology	18 (7.8%)	79 (8.3%)	0.02
Genitourinary	11 (4.8%)	84 (8.8%)	0.16
Myeloma	22 (9.6%)	150 (15.7%)	0.18
Lymphoma	13 (5.7%)	86 (9.0%)	0.13
Thoracic	18 (7.8%)	70 (7.3%)	−0.02

^*Values are presented as number (percentage) unless otherwise specified. P values were obtained using Pearson’s chi-square test for categorical variables and t-test for continuous variables. Standardized mean differences were calculated for all characteristics. SD denotes standard deviation; and SIC, serious illness conversation.

SPENDING DIFFERENCES AT THE END OF LIFE

In unadjusted analyses, mean daily spending during the last 6 months of life was lower in the intervention group than in the control group ($377.96 vs. $449.92); observed cost savings were most prominent for end-of-life systemic therapy ($112.97 vs. $164.32), office/outpatient ($25.96 vs. $38.61), and other therapy ($29.42 vs. $39.77). There were no meaningful effects on end-of-life acute care ($201.75 vs. $200.42), LTC ($8.90 vs. $6.28), or hospice ($0.89 vs. $0.22) spending (Fig. 1 and Table S2).

Figure 1.

Daily and Total Health Care Spending per Sector (in Dollars) for the Intervention and Control Groups.

Spending is stratified by 1, 3, and 6 months before death. LTC denotes long-term care; Rehab, rehabilitation; and SNF, skilled nursing facility.

In the primary analyses, the intervention was associated with a lower mean daily spending of $75.33 (95% CI, −$136.42 to −$14.23) in the last 6 months of life. Intervention effects were significant for systemic therapy (-$44.59; 95% CI, −$70.23 to -$18.95) and office/outpatient (−$9.62; 95% CI, −$15.39 to −$3.85) spending. When translating mean daily spending to total end-of-life spending savings, the intervention was associated with a decreased spending of $13,747 (95% CI, −$24,897 to −$2598) in the last 6 months of life. There were no significant between-group spending differences for hospice or rehab/LTC visit types (Table 3).

Table 3.

Adjusted Differences in the Mean Daily Spending between the Intervention and Control Groups in the Last 1, 3, and 6 Months of Life Stratified by Sector of Care.^*

	Last 6 Months of Life†				Last 3 Months of Life				Last 1 Month of Life
Spending Type	Mean Cost (SD), Control	Mean Cost (SD), Intervention	Difference (95% CI)	P Value	Mean Cost (SD), Control	Mean Cost (SD), Intervention	Difference (95% CI)	P Value	Mean Cost (SD), Control	Mean Cost (SD), Intervention	Difference (95% CI)	P Value
Overall	452.68 (27.89)	377.35 (12.66)	−75.33 (−136.42 to −14.23)	0.02	490.47 (40.04)	427.95 (18.99)	−62.52 (−151.55 to 26.50)	0.17	511.91 (63.99)	449.83 (25.31)	−62.08 (−212.93 to 88.77)	0.42
Acute care	196 (19.09)	202.93 (8.48)	6.93 (−35.76 to 49.63)	0.75	263.04 (26.56)	272.68 (14.77)	9.64 (−64.76 to −4.75)	0.76	364.98 (46.75)	351.95 (21.04)	−13.03 (−119.91 to 93.86)	0.81
Systemic therapy	158.91 (12.35)	114.32 (5.62)	−44.59 (−70.23 to −18.95)	0.001	135.3 (13.62)	100.55 (6.32)	−34.76 (−64.76 to −4.75)	0.02	100.06 (14.83)	66.17 (6.74)	−33.89 (−66.23 to −1.56)	0.04
Office/outpatient	34.23 (3.29)	24.61 (1.06)	−9.62 (−15.39 to −3.85)	0.001	28.05 (2.02)	22.36 (0.98)	−5.69 (−9.73 to −1.65)	0.006	18.84 (2.55)	15.05 (1.14)	−3.8 (−9.39 to 1.79)	0.18
Other therapy‡	38.32 (4.04)	29.66 (1.98)	−8.65 (−16.70 to −0.60)	0.04	35.27 (5.28)	23.11 (2.34)	−12.16 (−23.60 to −0.72)	0.04	—	—	—	—

^*Both primary and secondary analyses included eligible patients diagnosed with cancer who had encounters at one of the clinics between June 2019 and April 2020. Patients were excluded if they had a documented serious illness conversation or advanced care planning conversation before the start of the trial or if they were enrolled in another ongoing trial of early palliative care. CI denotes confidence interval; and SD, standard deviation.

^†All models were mixed two-part models. The first part modeled a logistic regression with a binary outcome of zero cost versus nonzero spending, and the second part modeled a generalized linear mixed model with a gamma distribution and a log-link function for patients with positive spending. All models were adjusted for clinician-group and wedge-period effects.

^‡Because of sample size restrictions, we were unable to estimate adjusted spending for hospice and rehabilitation/long-term care for all time periods and other therapies for the last month of life.

In secondary analyses, compared with the control group, the intervention group had lower unadjusted mean daily spending in the last 3 months of life ($431.80 vs. $473.20) and in the last 1 month of life ($814.46 vs. $947.18). Associations were directionally consistent with the 6-month analysis at 3 months (adjusted difference associated with intervention, −$62.52; 95% CI, −$151.55 to $26.50) and 1 month (adjusted difference, −$62.08; 95% CI, −$212.93 to $88.77).

SPENDING CHANGE AFTER SIC

In an exploratory analysis, among the 448 patients who received at least one SIC in the last 6 months of life, mean daily spending decreased by $37.92 following the first SIC ($329.87 before SIC vs. $291.95 after SIC) (Table S3). The average duration between the first SIC and death was 73.6days (60.1days for the control group and 76.3days for the intervention group). Among the 293 patients who received exactly one SIC in the last 6 months of life, the duration between SIC and death was 52.0days for the control group and 65.2days for the intervention group.

Discussion

In this secondary analysis of a randomized, controlled trial among 1187 patients with cancer receiving care in a large academic health system, an intervention that combined ML mortality predictions with behavioral nudges significantly reduced mean daily spending at the end of life, resulting in $13,747 in total adjusted savings per decedent in the last 6 months of life. Across all 957 decedents in the intervention group, this amount translated to more than $13 million in cumulative savings during the last 6 months of life. Savings were observed for end-of-life systemic therapy, office/outpatient, and other therapy spending. Similarly, among patients who received at least one SIC in the last 6 months of life, the main daily spending decreased by $37.92 following the first SIC. These findings suggest that an ML-based intervention to initiate early SICs may help patients use resources in a more cost-effective manner, which can lead to a long-term reduction in end-of-life spending. The intervention group had lower systemic therapy spending at all time points, indicating that these patients avoided intensive chemotherapy regimens throughout the last 6 months of life. These effects also compared favorably with a previous analysis of this trial, which showed that systemic therapy use at the end of life was significantly decreased following the intervention,²¹ as well as with broader retrospective studies that indicated that discussions about advanced treatment goals and preferences are associated with reduced chemotherapy near the end of life.^1–3,5–7

Furthermore, we observed that the between-group spending differences for hospice or LTC visit types in the analysis were nonsignificant. Changes in LTC may have been harder to achieve by nudges to clinicians alone because of the many factors downstream of initial conversations that influence LTC and hospice decisions. Mechanistically, SICs may promote greater scrutiny by oncologists and patients of the role of systemic therapy near the end of life, which may translate to reduced therapy and outpatient spending. Whereas previous prospective studies have investigated the role of early SICs in reducing health care spending, only a handful have demonstrated spending reduction from an intervention promoting SICs alone.^24–38 These studies primarily involved the use of a communication-based intervention with multiple follow-up visits and often focused on specific patient populations, including nursing home residents and patients with mental illness.^39,40 Similarly, among trials that reported cost savings following SIC implementation, there was evidence of reduced hospitalizations and fewer emergency department visits within the last year of life.^33,39

Moreover, mean daily spending decreased by $37.92 in the intervention group following an SIC, and total mean daily spending decreased by $75.33 compared with controls. This suggests that there was likely decreased spending even for patients who did not have documented SICs in the intervention group. ML-based risk predictions can explain this observation in that exposure to mortality estimates may promote more cost-effective behavior by clinicians and patients at the end of life. Additionally, behavioral nudges to clinicians in the form of peer comparisons, performance reports, and text messages could have influenced high-value behavior independent of SICs, thus leading to greater total cost savings observed in the adjusted analysis. It is also possible that greater awareness of SICs through the behavioral intervention normalized physicians toward higher-quality end-of-life care, as was suggested in a post hoc qualitative analysis evaluating clinicians’ perspectives regarding the trial.⁴¹ Further, because SICs were captured from dedicated sections within the EHR or smart phrases in progress notes, it is plausible that the intervention led to conversations that were not documented as official SICs.²⁰ Therefore, the savings reported in this analysis can be attributed to more precise targeting of SICs for patients with higher risk of mortality as well as to the implementation of more direct interventions to nudge clinicians’ behavior.

Overall, our study adds to this growing literature by being the first to show that an ML-based behavioral intervention results in substantial spending reductions in routine oncologic care. We also show that initiating SICs can reduce systemic therapy use at the end of life, which translates to less end-of-life systemic therapy spending and consequently, less overall spending. Other evaluations of ML-based interventions, such as that of the SHIELD-RT (System for High-Intensity Evaluation During Radiation Therapy) trial, demonstrated similar findings in that ML-based evaluations of high-risk patients undergoing radiotherapy decreased spending on emergency room visits and hospitalizations.^42,43 More broadly, the results of this trial showcase a rare instance of patient preferences and spending reductions being aligned as a result of an ML-based intervention. One advantage of this ML-based behavioral intervention, compared with potentially time-consuming educational efforts to improve serious illness care, is its scalability and low setup costs. Additionally, the threshold of ML algorithms could be modified to adjust to different risk distributions in other care settings. The savings associated with the intervention were mainly attributed to lower systemic therapy and outpatient use with unchanged hospitalization spending. Practices and health systems participating in value-based payment models may be particularly incentivized to adopt such an intervention. However, other practices that are reimbursed primarily through fee-for-service programs may be less incentivized to adopt interventions that decrease end-of-life systemic therapy use. Given that such savings stemming from SICs are likely concordant with patients’ and caregivers’ goals, it is paramount to ensure that reimbursement models do not disincentivize potential adoption of algorithm-based interventions that promote serious illness communication.

LIMITATIONS

This study had several limitations. First, given that the data from this study were obtained from a singular academic health system, we had limited visibility into spending incurred outside of that system. Costs may differ from rates negotiated by other health systems on the basis of payer contracts as well as by Medicare, and as such, they could affect the magnitude of total cost savings in other care settings. However, our analysis reflects real-world costs across a diverse payer mix in a representative academic health system across many diseases. Thus, we expect that these results are generalizable and reflect the perspective of similar academic health systems. Additionally, the study participants may not be representative of the general population of oncologists and patients with cancer, and this may include selection bias associated with drawing a postrandomization sample of decedents. For instance, significant baseline differences were observed between the intervention and control groups because of the variation in patient populations of the included clinics as well as the durations of the period for which each clinic was in the control or intervention groups. We were able to adjust for these clinic-group and wedge-period effects in our primary analysis, and we additionally controlled for age, race, and insurance status in a separate sensitivity analysis that supported the primary analysis. The ratio of patients who received the intervention was also similar to the primary trial and remained consistent across wedges, which suggests that differences in sample size were almost entirely attributed to the stepped-wedge design (Table S4). Finally, we lacked sufficient data to calculate mean daily spending for hospice and LTC use in the last 1 and 3 months of life, and we could not assess hospice use outside of the University of Pennsylvania facility. Nevertheless, by limiting the cohort to patients in the cancer center area who received primary oncologic care at UPHS, we selected for a population that primarily used hospice and oncology pharmacy services from the institution.

Conclusions

An intervention using ML-based mortality predictions to prompt clinicians to engage in early SICs with high-risk patients led to significant cost savings in the last 6 months of life. These savings were driven by decreases in systemic therapy and outpatient spending. This study shows that the integration of automated risk predictions alongside behavioral interventions may reduce end-of-life health care spending and should be considered as part of value-driven interventions to improve goal-concordant care.