Effect of Integrating Machine Learning Mortality Estimates With Behavioral Nudges to Clinicians on Serious Illness Conversations Among Patients With Cancer: A Stepped-Wedge Cluster Randomized Clinical Trial

Christopher R Manz; Ravi B Parikh; Dylan S Small; Chalanda N Evans; Corey Chivers; Susan H Regli; C William Hanson; Justin E Bekelman; Charles A L Rareshide; Nina O’Connor; Lynn M Schuchter; Lawrence N Shulman; Mitesh S Patel

doi:10.1001/jamaoncol.2020.4759

. 2020 Oct 15;6(12):e204759. doi: 10.1001/jamaoncol.2020.4759

Effect of Integrating Machine Learning Mortality Estimates With Behavioral Nudges to Clinicians on Serious Illness Conversations Among Patients With Cancer

A Stepped-Wedge Cluster Randomized Clinical Trial

Christopher R Manz ^1,^✉, Ravi B Parikh ^2,^3,⁴, Dylan S Small ⁵, Chalanda N Evans ^2,⁶, Corey Chivers ⁷, Susan H Regli ⁷, C William Hanson ⁷, Justin E Bekelman ^2,³, Charles A L Rareshide ^2,⁶, Nina O’Connor ², Lynn M Schuchter ^2,³, Lawrence N Shulman ^2,³, Mitesh S Patel ^2,^3,^4,⁶

¹Department of Population Sciences, Dana-Farber Cancer Institute, Boston, Massachusetts

²Department of Medicine, Perelman School of Medicine, University of Pennsylvania, Philadelphia

³Penn Center for Cancer Care Innovation, Abramson Cancer Center, University of Pennsylvania, Philadelphia

⁴Corporal Michael J. Crescenz Veterans Affairs Medical Center, Philadelphia, Pennsylvania

⁵Wharton School of the University of Pennsylvania, Philadelphia

⁶Penn Medicine Nudge Unit, Philadelphia, Pennsylvania

⁷University of Pennsylvania Health System, Philadelphia

Accepted for Publication: July 28, 2020.

Published Online: October 15, 2020. doi:10.1001/jamaoncol.2020.4759

^✉

Corresponding Author: Christopher R. Manz, MD, Dana-Farber Cancer Institute, 450 Brookline Avenue, 11th Floor, Boston, MA 02215 (christopher_manz@dfci.harvard.edu).

Author Contributions: Drs Manz and Parikh had full access to all the data in the study and take responsibility for the integrity of the data and the accuracy of the data analysis. Drs Manz and Parikh contributed equally to this work and are co–first authors.

Concept and design: Manz, Parikh, Small, Chivers, Regli, Hanson, O’Connor, Schuchter, Shulman, Patel.

Acquisition, analysis, or interpretation of data: Manz, Parikh, Small, Evans, Chivers, Bekelman, Rareshide, O’Connor, Shulman, Patel.

Drafting of the manuscript: Manz, Parikh, Evans, Bekelman.

Critical revision of the manuscript for important intellectual content: Manz, Parikh, Small, Chivers, Regli, Hanson, Bekelman, Rareshide, O’Connor, Schuchter, Shulman, Patel.

Statistical analysis: Manz, Parikh, Small, Chivers, Rareshide.

Obtained funding: Manz, Parikh.

Administrative, technical, or material support: Parikh, Evans, Chivers, Regli, O’Connor, Shulman, Patel.

Supervision: Parikh, Regli, Hanson, O’Connor, Schuchter, Shulman, Patel.

Conflict of Interest Disclosures: Dr Parikh reported receiving personal fees from GNS Healthcare and the Cancer Study Group and grants from Penn Center for Precision Medicine and the MUSC Transdisciplinary Collaborative Center in Precision Medicine and Minority Men’s Health, and grants and nonfinancial support from the Conquer Cancer Foundation outside the submitted work. Dr Bekelman reported receiving personal fees from the Centers for Medicare and Medicaid Services, Optum, CVS Health, the National Comprehensive Cancer Network, and UnitedHealthcare and grants from Pfizer, North Carolina Blue Cross Blue Shield, Embedded Healthcare, and UnitedHealth Group outside the submitted work. Dr Patel reported being an advisory board member of and receiving stock options from Life.io and personal fees from Catalyst Health LLC, HealthMine Services, and Holistic Industries outside the submitted work. No other disclosures were reported.

Funding/Support: This study was funded by the Penn Center for Precision Medicine Accelerator Fund (to Drs Manz and Parikh), award T32 GM075766-14 to Dr Manz from the National Institute of General Medical Sciences, and the University of Pennsylvania Health System through the Penn Medicine Nudge Unit.

Role of the Funder/Sponsor: The funders had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; and decision to submit the manuscript for publication.

Meeting Presentation: This paper was presented at the Annual Meeting of the American Society of Clinical Oncology; May 29, 2020; virtual; and at the Annual Research Meeting of the Academy Health; July 28, 2020; virtual.

Data Sharing Statement: See Supplement 3.

Additional Contributions: Peter E. Gabriel, MD, MSE, Penn Medicine, provided assistance with database curation; and Jennifer Braun, MHA, BSN, Penn Medicine, provided guidance on interpretation. Neither person receive financial compensation for their contributions.

^✉

Corresponding author.

PMCID: PMC7563672 PMID: 33057696

Key Points

Question

What is the effect of delivering machine learning mortality predictions with behavioral nudges to oncology clinicians on the rate of serious illness conversations with patients with cancer?

Findings

In this stepped-wedge cluster randomized clinical trial that included 14 607 patients with cancer, the intervention led to a significant increase in serious illness conversations from approximately 1% to 5% of all patient encounters and from approximately 4% to 15% of encounters with patients having high predicted mortality risk.

Meaning

Machine learning mortality predictions combined with behavioral nudges to clinicians led to an increased rate of serious illness conversations for patients with cancer.

Abstract

Importance

Serious illness conversations (SICs) are structured conversations between clinicians and patients about prognosis, treatment goals, and end-of-life preferences. Interventions that increase the rate of SICs between oncology clinicians and patients may improve goal-concordant care and patient outcomes.

Objective

To determine the effect of a clinician-directed intervention integrating machine learning mortality predictions with behavioral nudges on motivating clinician-patient SICs.

Design, Setting, and Participants

This stepped-wedge cluster randomized clinical trial was conducted across 20 weeks (from June 17 to November 1, 2019) at 9 medical oncology clinics (8 subspecialty oncology and 1 general oncology clinics) within a large academic health system in Pennsylvania. Clinicians at the 2 smallest subspecialty clinics were grouped together, resulting in 8 clinic groups randomly assigned to the 4 intervention wedge periods. Included participants in the intention-to-treat analyses were 78 oncology clinicians who received SIC training and their patients (N = 14 607) who had an outpatient oncology encounter during the study period.

Interventions

(1) Weekly emails to oncology clinicians with SIC performance feedback and peer comparisons; (2) a list of up to 6 high-risk patients (≥10% predicted risk of 180-day mortality) scheduled for the next week, estimated using a validated machine learning algorithm; and (3) opt-out text message prompts to clinicians on the patient’s appointment day to consider an SIC. Clinicians in the control group received usual care consisting of weekly emails with cumulative SIC performance.

Main Outcomes and Measures

Percentage of patient encounters with an SIC in the intervention group vs the usual care (control) group.

Results

The sample consisted of 78 clinicians and 14 607 patients. The mean (SD) age of patients was 61.9 (14.2) years, 53.7% were female, and 70.4% were White. For all encounters, SICs were conducted among 1.3% in the control group and 4.6% in the intervention group, a significant difference (adjusted difference in percentage points, 3.3; 95% CI, 2.3-4.5; P < .001). Among 4124 high-risk patient encounters, SICs were conducted among 3.6% in the control group and 15.2% in the intervention group, a significant difference (adjusted difference in percentage points, 11.6; 95% CI, 8.2-12.5; P < .001).

Conclusions and Relevance

In this stepped-wedge cluster randomized clinical trial, an intervention that delivered machine learning mortality predictions with behavioral nudges to oncology clinicians significantly increased the rate of SICs among all patients and among patients with high mortality risk who were targeted by the intervention. Behavioral nudges combined with machine learning mortality predictions can positively influence clinician behavior and may be applied more broadly to improve care near the end of life.

Trial Registration

ClinicalTrials.gov Identifier: NCT03984773

This stepped-wedge cluster randomized clinical trial assesses the effect of a clinician-directed intervention combining machine learning mortality predictions with behavioral nudges vs usual care on motivating serious illness conversations between clinicians and patients with cancer.

Introduction

Patients with cancer often receive treatment and acute care use, particularly near the end of life, that does not align with their preferences or goals of care.^1,2,3,4,5 Early discussions about goals and treatment preferences may lead to better perceived quality of life, reduced emotional distress, and decreased health care use near the end of life.^6,7,8 However, most patients with cancer die without a documented discussion about goals and treatment preferences.^9,10

There are several reasons for the dearth of such discussions between oncologists and patients with cancer. First, oncologists correctly identify fewer than half of their patients who will die within 6 to 12 months.^11,12 Overoptimistic assessments of short-term mortality risk may result in inadequate identification of patients who will benefit from a timely discussion.^13,14,15 Second, it is often difficult to change clinician behavior and practice patterns. Insights from behavioral economics, including changing default options and delivering performance feedback, have been shown to alter clinician behavior in a variety of clinical settings.^16,17,18,19

Serious illness conversations (SICs) are a type of advance care planning discussion and are structured conversations between clinicians, patients, and families intended to explore prognostic awareness, treatment priorities, and goals of care.²⁰ In a prior randomized clinical trial, an intervention that provided clinicians with a Serious Illness Conversation Guide ameliorated anxiety and depression among patients with cancer.^21,22 The objective of the present stepped-wedge cluster randomized clinical trial was to test the effect of an intervention that integrated real-time machine learning (ML) mortality predictions with behavioral nudges on motivating SICs between oncologists and high-risk patients. We used a validated ML algorithm that has been shown to accurately predict 180-day mortality risk.²³ We hypothesized that this multimodal intervention would increase the rate of SICs between oncology clinicians and all patients as well as between oncology clinicians and high-risk patients.

Methods

Study Design

This was a stepped-wedge cluster randomized clinical trial (NCT03984773) among 9 medical oncology clinics in the University of Pennsylvania Health System.²⁴ The trial included five 4-week wedges and was conducted from June 17 to November 1, 2019. The trial evaluated the effect of delivering ML mortality estimates with behavioral nudges on the rate of SICs among patients with cancer. The mortality prediction algorithm has been previously validated.²³ This trial followed the Consolidated Standards of Reporting Trials (CONSORT) reporting guideline. The trial protocol was approved by the University of Pennsylvania institutional review board (trial protocol in Supplement 1),²⁵ which also granted a waiver for the requirement to obtain informed consent because this study was an evaluation of a health system initiative that posed minimal risk to clinicians and patients.

Study Sample

Eligible clinicians were physicians, physician assistants, and nurse practitioners who provided oncology care at 8 subspecialty clinics (breast, central nervous system, gastrointestinal, genitourinary, lymphoma, melanoma, myeloma, and thoracic or head and neck) and 1 general oncology clinic (n = 78). Clinicians at the 2 smallest subspecialty clinics (melanoma and central nervous system malignant neoplasms) were grouped together, resulting in 8 clinic groups. Clinicians were excluded if they cared only for patients with benign hematologic or genetic disorders (n = 3) saw fewer than 12 patients classified as high risk by the algorithm in either the preintervention or postintervention period (n = 6), or had not undergone SIC training at the time of trial initiation (n = 4).

Eligible patients had encounters at 1 of the clinics during the study period. Patients were excluded if they had a documented SIC or advanced care planning conversation (ACP) prior to the start of the trial, or if they were enrolled in another ongoing trial of early palliative care. Genetics encounters were also excluded.

Randomization

The 8 clinic groups were randomized to the 4 intervention wedge periods, stratified by SIC rate (above or below the median). The study senior author (M.S.P.) and data analyst (C.A.L.R.) were blinded to wedge assignment until analysis of the primary end point was complete. Blinding was not feasible for investigators who were oncologists practicing at Penn Medicine.

Interventions

All clinicians included in the trial were trained in the use of the Serious Illness Conversation Guide (Ariadne Labs) at least 3 months prior to the start of the trial (range, 5-18 months).²⁰ The default advance care planning template for clinicians was based on this guide.

For approximately 1 year before the trial began, clinicians in all clinic groups received weekly emails that reported their individual SIC performance and blinded peer comparison in 2 bar graphs: 1 for cumulative SICs performed since that clinician’s SIC training date and 1 for SICs performed in the prior week. This continued during the control period of the trial and was turned off once the intervention was implemented. In the first 4-week wedge, all clinic groups remained in usual care (control). In each subsequent 4-week period, 2 clinic groups (intervention) received the intervention. By the start of the fifth wedge, all clinic groups were receiving the intervention (eFigure 1 in the Supplement 2).

The intervention was directed toward clinicians and included 3 main components. First, every Thursday, each clinician received an email with performance feedback indicating the number of SICs he or she performed in the prior 4 weeks. The email also contained a peer comparison message; this message differed for clinicians who performed below or above 8 SICs or who were among the top 10 performers of SICs during the prior 4 weeks. Samples of the email text are shown in eFigure 2 in the Supplement 2. Performance feedback and peer comparisons have been shown to be successful in motivating clinician decision-making.^17,26,27

Second, the email included a link to a secure dashboard in which clinicians could review patients scheduled for the following week in their clinic who had a high risk of mortality; these were described as patients “who may benefit from an SIC.” Mortality risk was determined by a previously validated ML algorithm, which used structured electronic health record (EHR) data to predict risk of 180-day mortality.² Every Thursday morning, the algorithm generated mortality risk predictions for all patients with a scheduled encounter during the following week with an eligible clinician. Clinicians could view a personalized list of up to 6 patients with the highest predicted 180-day mortality risk (“high-risk” patients) via a secure dashboard. Patients with a mortality risk below 10% were not included in the list. Mortality predictions were updated weekly; thus, patient assignment as “high risk” could change from encounter to encounter. Information in the dashboard included patient identifiers, the author and date of any prior SICs, and a checkbox to opt out of reminder texts. Using opt out (as opposed to opt in) as a default has been successful in different health care settings to change physician behavior.^{28,29,30,31,32} The prediction algorithm silently identified high-risk patients during the control periods, but these lists were not shared with the clinicians via email or text.

Third, clinicians received a text message reminding them to consider having an SIC with any high-risk patient. For the high-risk patients on the clinicians’ dashboard, clinicians were sent this message on the morning of the patient’s encounter, unless they opted out or an SIC was performed in the prior 2 months.

Outcome Measures

The primary outcome was change in SIC rate among all patient encounters. Secondary outcomes included changes in rates of (1) SICs among high-risk patients and (2) ACPs for the overall sample and for the high-risk subgroup.

The electronic medical record had an ACP section in which advance care planning notes were documented. Any note documented in this section was classified as an ACP. The subset of notes that included a preformatted SIC template was classified as an SIC. Thus, ACPs encompassed both SIC and non-SIC documentation about treatment goals and wishes.

Statistical Analysis

A priori power calculations used data from the University of Pennsylvania Health System to estimate the rate of SICs. Assuming a baseline SIC rate of 0.65%, we estimated that this trial would have 80% power to detect a 60% relative increase in SIC rate using a 2-sided α of 0.05 as our threshold for statistical significance.

All trial clinicians and their patients were included in the intention-to-treat analysis. Analyses were conducted using the first patient encounter in each wedge; subsequent encounters were excluded to avoid biasing results toward patients with numerous encounters. Patients seen by physician assistants and nurse practitioners were allocated to their corresponding oncologist. Because care planning documentation can occur days to weeks after an encounter, SICs and ACPs within 28 days of the encounter with no intervening encounter were counted in the primary outcome. We performed a sensitivity analysis that defined the primary outcome using a shorter, 7-day period after the index encounter to count SICs and ACPs.

Similar to prior work,⁷ we fit models for the outcome measures based on generalized estimating equations with a logit link and an independence correlation structure using oncologist as the clustering unit. The main models used patient-encounter level observations, clinic-group and wedge-period fixed effects, and a binary indicator for intervention implementation. To test the robustness of our findings, we fit a fully adjusted model that also included age, sex, marital status, patient-reported race/ethnicity, insurance status, and Charlson Comorbidity Index. To obtain the adjusted difference and 95% CIs in the percentage points, we used the bootstrap method, resampling patients 1000 times.^33,34 Resampling of patients was conducted by the oncologist variable to maintain clustering at the oncologist level. Analyses were conducted between January 6 and March 9, 2020. Two-sided hypothesis tests used a significance level of P ≤ .05. All analyses were conducted using SAS, version 9.4 (SAS Institute Inc).

Results

The sample consisted of 78 clinicians (of whom 42 were oncologists), 14 607 patients, and 26 059 patient encounters (Figure). Patients’ mean (SD) age was 61.9 (14.2) years. Of 14 607 patients, 8141 (55.7%) were female, 10 285 (70.4%) were non-Hispanic White, 6876 (47.1%) were commercially insured, 6806 (46.7%) were covered by Medicare, and 925 (6.3)% were covered by Medicaid (Table 1). Patients in the control and intervention groups had similar characteristics. High-risk encounters accounted for 2125 of 12 170 patient encounters (17.5%) in the control groups and 1999 of 13 889 patient encounters (14.4%) in the intervention groups. The median 180-day mortality risk per patient-encounter was similar in the control and intervention groups (4% [interquartile range {IQR}, 2%-13%] vs 4% [IQR, 2%-12%]). Among high-risk patients, the patients’ characteristics and median mortality risk (34% [IQR, 20%-53%] vs 36% [IQR, 20%-56%]) were also similar between control and intervention groups (eTable 1 in Supplement 2). The number of patient encounters and SIC rates by clinic and control vs intervention group are shown in eTable 2 in Supplement 2.

Table 1. Patient-Encounter Characteristics.

Characteristic	No. (%) of patient encounters		P value
Characteristic	Control (n = 12 170)	Intervention (n = 13 889)	P value
Sociodemographic characteristics
Age, mean (SD), y	62.5 (13.8)	61.3 (14.5)	<.001
Female	6426 (52.8)	7576 (54.5)	.005
Race/ethnicity
Non-Hispanic White	8629 (70.9)	9709 (69.9)	.02
Non-Hispanic Black	2436 (20.0)	2772 (20.0)
Other	1105 (9.1)	1408 (10.1)
Insurance
Commercial	5467 (44.9)	6522 (47.0)	<.001
Medicare	5953 (48.9)	6321 (45.5)
Medicaid	750 (6.2)	1046 (7.5)
Clinical imblances^a
Breast	2003 (16.5)	1520 (10.9)	<.001
Central nervous system + melanoma	762 (6.3)	1188 (8.6)
Gastrointestinal	624 (5.1)	2770 (19.9)
General oncology	2171 (17.8)	3757 (27.1)
Genitourinary	1792 (14.7)	451 (3.2)
Myeloma	2592 (21.3)	707 (5.1)
Lymphoma	541 (4.4)	2289 (16.5)
Thoracic	1685 (13.8)	1207 (8.7)
Clinical characteristics
Charlson Comorbidity Index, median (IQR)	3 (2-8)	3 (2-8)	.02
Predicted mortality risk, median (IQR), %^b	4 (2-13)	4 (2-12)	<.001
High-risk patient encounters (% of total)	2125 (17.5)	1999 (14.4)	<.001

Open in a new tab

Abbreviation: IQR, interquartile range.

^{^a}

Imbalances between control and intervention groups were due to staggered intervention start times in the stepped-wedge design.

^{^b}

Mortality risk at 180 days as predicted by the mortality prediction algorithm.

Serious Illness Conversations

Among all patients, SICs were conducted in 155 of 12 170 control encounters (1.3%) and 632 of 13 889 intervention encounters (4.6%). Among high-risk patients, SICs were conducted in 77 of 2125 control encounters (3.6%) and 304 of 1999 intervention encounters (15.2%) (eFigure 3 in Supplement 2). In adjusted analyses of all patients, the intervention led to a significant increase in the SIC rate (adjusted difference in percentage points, 3.3; 95% CI, 2.3-4.5; P < .001) (Table 2). In adjusted analyses of 4124 high-risk patient encounters, there was a significant increase in the SIC rate (adjusted difference in percentage points, 11.6; 95% CI, 8.2-12.5; P < .001). Full model results are shown in eTable 3 and eTable 4 in Supplement 2. Sensitivity analyses showed similar results (eTables 5, 6, 7, and 12 in Supplement 2).

Table 2. Adjusted Changes in Serious Illness Conversations and in Advanced Care Planning.

Conversation	No./total No. (%) of encounters		Adjusted difference for intervention relative to control, percentage points (95% CI)^a	P value
Conversation	Control	Intervention		P value
Serious illness encounters
All patients	155/12 170 (1.3)	632/13 889 (4.6)	3.3 (2.3-4.5)	<.001
High-risk patients	77/2125 (3.6)	304/1999 (15.2)	11.6 (8.2-15.5)	<.001
Advanced care planning encounters
All patients	231/12 170 (1.9)	680/13 889 (4.9)	3.0 (2.1-4.1)	.001
High-risk patients	124/2125 (5.8)	350/1999 (17.5)	11.7 (8.4-15.7)	<.001

Open in a new tab

^{^a}

Adjusted for intervention, clinic group, and wedge.

Advanced Care Planning Conversations

Among all patients, ACPs were conducted in 231 of 12 170 control encounters (1.9%) and 680 of 13 889 intervention encounters (4.9%). Among high-risk patients, ACPs were conducted in 124 of 2125 control encounters (5.8%) and 350 of 1999 intervention encounters (17.5%). In adjusted analyses of all patients, the intervention led to a significant increase in the ACP rate (adjusted difference in percentage points, 3.0; 95% CI, 2.1-4.1; P = .001) (Table 2). In adjusted analyses of high-risk patients, the intervention led to a significant increase in the ACP rate (adjusted difference in percentage points, 11.7; 95% CI, 8.4-15.7; P < .001). Full model results are shown in eTable 8 and eTable 9 in Supplement 2. Sensitivity analyses showed similar results (eTables 10, 11 and 12 in Supplement 2). No adverse events were reported.

Discussion

In this randomized clinical trial conducted in 9 oncology practices, an intervention that combined ML mortality predictions with behavioral nudges significantly increased rates of SICs and ACPs. The intervention led to a 4-fold increase (3.6%-15.2%) in SICs among high-risk patients targeted by the intervention. To our knowledge, this is one of the first randomized clinical trials testing the effect of real-time ML predictions and behavioral nudges on clinical care. There are several important implications of our findings.

First, our intervention resulted in a large increase in SIC rates, especially among patients at highest risk of 180-day mortality. Although prior studies have assessed SIC and ACP rates among decedents,^6,7 our study showed that a behaviorally informed intervention led to a rapid increase in such conversations among all patients with cancer. Although the optimal SIC prevalence among patients with cancer is unknown, the effect of this intervention compares favorably with other published interventions on increasing the number of such conversations.^35,36,37 Discussions about goals and treatment preferences have been associated with less aggressive care for cancer patients near the end of life, including increased hospice utilization rates and reduced chemotherapy and hospitalization rates near the end of life.^{1,2,3,4,5,6,7,8} Furthermore, documentation of such discussions ensures that patients’ wishes are recorded and available across care settings in a health system. Our study has particular importance as oncologists and patients now grapple with care delays and competing risks due to the coronavirus disease 2019 pandemic. Indeed, documentation of goals and wishes has been recognized by several oncology guideline bodies as a key priority to ensure goal-concordant care during the pandemic. A uniform approach to the automated identification of high-risk patients that increases the rate of SICs can be combined with new care models, such as telemedicine, to meet this need.

Targeting SICs to high-risk patients at risk of short-term mortality may result in improved goal-concordant care, which was not shown in a previous randomized clinical trial of an intervention consisting primarily of training on the Serious Illness Conversation Guide.^21,22 The cohort in that trial consisted of individuals who died. The use of that cohort may explain why overall rates of SICs in the control and intervention groups were much higher than those in the present trial, which consisted of all patients (decedents and nondecedents) with cancer, including many with curable early-stage cancers for whom SICs would not be expected. In addition, the relatively low SIC prevalence among even high-risk patients after the intervention may be due to the exclusion of patients with an existing SIC at the start of the present trial, a short follow-up period, clinician time constraints, the performance of SICs that were not documented in a trackable fashion in the EHR, and misclassification of patients as high risk. Subsequent research should determine longer-term clinical and quality-of-life outcomes associated with SICs prompted by this intervention.

Second, although several trials have prospectively integrated ML risk predictions into clinical care, few studies have shown that interventions that involve ML predictions can positively influence clinical care or decision-making.^38,39 Our intervention illustrates that generalizable behavioral principles combined with ML predictions may inform more accurate, individualized prognoses that clinicians and patients use to guide treatment decisions and ensure receipt of goal-concordant care. A key innovation of the present study was integrating behavioral economic principles into the ML-based intervention, including peer comparison, performance feedback, and opt-out reminders, which have been used successfully in other clinical settings to change clinician behavior.^{16,17,18,19,40} Future implementation of artificial intelligence or ML tools in clinical practice should explore similar mechanisms of delivering predictions and integrating behavioral principles to maximally improve clinician and patient decision-making. Yet these behavioral nudges may not be equally necessary and effective. Although the present study cannot disentangle the effects of individual intervention components, setting the default to a desired behavior—in this instance, defaulting clinicians to receive notifications encouraging SICs with high-risk patients—is a powerful nudge that has been widely applied in medicine to change behavior and may be the most impactful component.²⁸ Future research should use head-to-head comparisons to identify which intervention components deliver the most benefit and how an integrated EHR-based mortality prediction tool can be generalized to other health systems.

Third, although our intervention effect was strongest in the high-risk patients flagged by the algorithm, we still detected an increase of nearly 2 percentage points in SICs among patients who were not deemed high-risk by the algorithm. These “off-target” patients comprised the majority of our cohort. To our knowledge, off-target effects of an intervention to promote supportive care have not been shown. This off-target effect may have occurred because the clinicians who received the intervention became accustomed to actively considering mortality risk in their clinical practice, resulting in an increase in SICs for patients who may have a poor prognosis but were not flagged by the algorithm as high risk.

Strengths and Limitations

Our study has several strengths, including the application of behavioral insights into an ML-based intervention, integration of accurate individual-level mortality risk predictions at the point of care, a stepped-wedge design that controlled for cancer site–specific uptake of SICs, and a large sample of patients across several different cancer types and in both tertiary academic and general oncology practices. This study has several limitations. First, this study was performed in a single academic health system, and the participants may not be representative of the general population of oncologists or patients with cancer. However, given the prominence of EHRs in oncology care and the many examples of EHR-based ML algorithms across EHR types, the principles behind this intervention could theoretically be applied in any cancer care setting.⁴¹ Second, early ACPs and SICs are surrogates for goal-concordant care and decreased aggressive end-of-life care. Longer-term studies are needed to assess whether algorithm-based interventions improve clinical outcomes for patients. Third, our intervention consisted of several components, including a real-time prediction algorithm, a secure list of high-risk patients, peer comparisons, performance reports, and text messages. Future work is needed to assess the individual effects of each of these interventions.

Conclusions

An intervention combining ML mortality risk predictions and behavioral nudges led to an increase in SICs among patients with cancer. This study provides guidance for more robust integration of automated risk predictions alongside behavioral interventions in future prospective studies related to and outside of SICs in oncology.

Supplement 1.

Trial Protocol

Click here for additional data file.^{(553.4KB, pdf)}

Supplement 2.

eFigure 1. Stepped-Wedge Design Overview

eFigure 2. Example Clinician Emails by Performance

eFigure 3. Serious Illness Conversations for High-Risk Patients by Clinic

eTable 1. High-Risk Patient Encounter Characteristics

eTable 2. Patient Encounters and SIC Rates by Clinic and Control vs Intervention

eTable 3. Regression Table for Serious Illness Conversation Model Including All Patients

eTable 4. Regression Table for Serious Illness Conversation Model Including High Risk Patients Only

eTable 5. Fully Adjusted Models for Serious Illness Conversations (SICs) for All Patients and for High Risk Patients Only

eTable 6. Serious Illness Conversations (SICs) for All Patients, Documented Within 7 Days

eTable 7. Serious Illness Conversations (SICs), Adjusted for Clinician Time From SIC Guide Training to Trial Start Date

eTable 8. Regression Table for Advance Care Planning Model Including All Patients

eTable 9. Regression Table for Advance Care Planning Model Including High Risk Patients Only

eTable 10. Fully Adjusted Models for Advance Care Planning (ACPs) for All Patients and for High Risk Patients Only

eTable 11. Adjusted Change in Advance Care Planning Conversations (SICs), Documented Within 7 days

eTable 12. Adjusted Change in Serious Illness Conversations (SICs) and Advanced Care Planning for Non-High-Risk Patients

Click here for additional data file.^{(670.2KB, pdf)}

Supplement 3.

Data Sharing Statement

Click here for additional data file.^{(18.9KB, pdf)}

References

1.Emanuel EJ, Young-Xu Y, Levinsky NG, Gazelle G, Saynina O, Ash AS. Chemotherapy use among Medicare beneficiaries at the end of life. Ann Intern Med. 2003;138(8):639-643. doi: 10.7326/0003-4819-138-8-200304150-00011 [DOI] [PubMed] [Google Scholar]
2.Earle CC, Neville BA, Landrum MB, Ayanian JZ, Block SD, Weeks JC. Trends in the aggressiveness of cancer care near the end of life. J Clin Oncol. 2004;22(2):315-321. doi: 10.1200/JCO.2004.08.136 [DOI] [PubMed] [Google Scholar]
3.Earle CC, Landrum MB, Souza JM, Neville BA, Weeks JC, Ayanian JZ. Aggressiveness of cancer care near the end of life: is it a quality-of-care issue? J Clin Oncol. 2008;26(23):3860-3866. doi: 10.1200/JCO.2007.15.8253 [DOI] [PMC free article] [PubMed] [Google Scholar]
4.Chastek B, Harley C, Kallich J, Newcomer L, Paoli CJ, Teitelbaum AH. Health care costs for patients with cancer at the end of life. J Oncol Pract. 2012;8(6):75s-80s. doi: 10.1200/JOP.2011.000469 [DOI] [PMC free article] [PubMed] [Google Scholar]
5.Wen F-H, Chen J-S, Su P-J, et al. . Terminally ill cancer patients’ concordance between preferred life-sustaining treatment states in their last six months of life and received life-sustaining treatment states in their last month: an observational study. J Pain Symptom Manage. 2018;56(4):509-518.e3. doi: 10.1016/j.jpainsymman.2018.07.003 [DOI] [PubMed] [Google Scholar]
6.Bernacki RE, Block SD; American College of Physicians High Value Care Task Force . Communication about serious illness care goals: a review and synthesis of best practices. JAMA Intern Med. 2014;174(12):1994-2003. doi: 10.1001/jamainternmed.2014.5271 [DOI] [PubMed] [Google Scholar]
7.Wright AA, Zhang B, Ray A, et al. . Associations between end-of-life discussions, patient mental health, medical care near death, and caregiver bereavement adjustment. JAMA. 2008;300(14):1665-1673. doi: 10.1001/jama.300.14.1665 [DOI] [PMC free article] [PubMed] [Google Scholar]
8.Brinkman-Stoppelenburg A, Rietjens JA, van der Heide A. The effects of advance care planning on end-of-life care: a systematic review. Palliat Med. 2014;28(8):1000-1025. doi: 10.1177/0269216314526272 [DOI] [PubMed] [Google Scholar]
9.Schubart JR, Levi BH, Bain MM, Farace E, Green MJ. Advance care planning among patients with advanced cancer. J Oncol Pract. 2019;15(1):e65-e73. doi: 10.1200/JOP.18.00044 [DOI] [PubMed] [Google Scholar]
10.National Quality Forum : Palliative and end-of-life care 2015-2016. Draft report for comment June 20, 2016. Accessed June 10, 2020. http://www.qualityforum.org/Projects/n-r/Palliative_and_End-of-Life_Care_Project_2015-2016/Draft_Report_for_Comment.aspx
11.Chow E, Harth T, Hruby G, Finkelstein J, Wu J, Danjoux C. How accurate are physicians’ clinical predictions of survival and the available prognostic tools in estimating survival times in terminally ill cancer patients? a systematic review. Clin Oncol (R Coll Radiol). 2001;13(3):209-218. doi: 10.1007/s001740170078 [DOI] [PubMed] [Google Scholar]
12.Ermacora P, Mazzer M, Isola M, et al. . Prognostic evaluation in palliative care: final results from a prospective cohort study. Support Care Cancer. 2019;27(6):2095-2102. doi: 10.1007/s00520-018-4463-z [DOI] [PubMed] [Google Scholar]
13.Weeks JC, Cook EF, O’Day SJ, et al. . Relationship between cancer patients’ predictions of prognosis and their treatment preferences. JAMA. 1998;279(21):1709-1714. doi: 10.1001/jama.279.21.1709 [DOI] [PubMed] [Google Scholar]
14.Rose JH, O’Toole EE, Dawson NV, et al. . Perspectives, preferences, care practices, and outcomes among older and middle-aged patients with late-stage cancer. J Clin Oncol. 2004;22(24):4907-4917. doi: 10.1200/JCO.2004.06.050 [DOI] [PubMed] [Google Scholar]
15.White N, Reid F, Harris A, Harries P, Stone P. A systematic review of predictions of survival in palliative care: how accurate are clinicians and who are the experts? PLoS One. 2016;11(8):e0161407. doi: 10.1371/journal.pone.0161407 [DOI] [PMC free article] [PubMed] [Google Scholar]
16.Brown B, Gude WT, Blakeman T, et al. . Clinical Performance Feedback Intervention Theory (CP-FIT): a new theory for designing, implementing, and evaluating feedback in health care based on a systematic review and meta-synthesis of qualitative research. Implement Sci. 2019;14(1):40. doi: 10.1186/s13012-019-0883-5 [DOI] [PMC free article] [PubMed] [Google Scholar]
17.Navathe AS, Emanuel EJ. Physician peer comparisons as a nonfinancial strategy to improve the value of care. JAMA. 2016;316(17):1759-1760. doi: 10.1001/jama.2016.13739 [DOI] [PubMed] [Google Scholar]
18.Patel MS, Volpp KG, Asch DA. Nudge units to improve the delivery of health care. N Engl J Med. 2018;378(3):214-216. doi: 10.1056/NEJMp1712984 [DOI] [PMC free article] [PubMed] [Google Scholar]
19.Patel MS, Volpp KG, Small DS, et al. . Using active choice within the electronic health record to increase influenza vaccination rates. J Gen Intern Med. 2017;32(7):790-795. doi: 10.1007/s11606-017-4046-6 [DOI] [PMC free article] [PubMed] [Google Scholar]
20.Bernacki R, Hutchings M, Vick J, et al. . Development of the Serious Illness Care Program: a randomised controlled trial of a palliative care communication intervention. BMJ Open. 2015;5(10):e009032. doi: 10.1136/bmjopen-2015-009032 [DOI] [PMC free article] [PubMed] [Google Scholar]
21.Paladino J, Bernacki R, Neville BA, et al. . Evaluating an intervention to improve communication between oncology clinicians and patients with life-limiting cancer: a cluster randomized clinical trial of the Serious Illness Care Program. JAMA Oncol. 2019;5(6):801-809. doi: 10.1001/jamaoncol.2019.0292 [DOI] [PubMed] [Google Scholar]
22.Bernacki R, Paladino J, Neville BA, et al. . Effect of the Serious Illness Care Program in outpatient oncology: a cluster randomized clinical trial. JAMA Intern Med. 2019;179(6):751-759. doi: 10.1001/jamainternmed.2019.0077 [DOI] [PMC free article] [PubMed] [Google Scholar]
23.Parikh RB, Manz C, Chivers C, et al. . Machine learning approaches to predict 6-month mortality among patients with cancer. JAMA Netw Open. 2019;2(10):e1915997. doi: 10.1001/jamanetworkopen.2019.15997 [DOI] [PMC free article] [PubMed] [Google Scholar]
24.Machine-generated mortality estimates and nudges to promote advance care planning discussion among cancer patients. ClinicalTrials.gov identifier: NCT03984773. Accessed September 2, 2020. https://clinicaltrials.gov/ct2/show/NCT03984773
25.Manz CR, Parikh RB, Evans CN, et al. . Integrating machine-generated mortality estimates and behavioral nudges to promote serious illness conversations for cancer patients: design and methods for a stepped-wedge cluster randomized controlled trial. Contemp Clin Trials. 2020;90:105951. doi: 10.1016/j.cct.2020.105951 [DOI] [PMC free article] [PubMed] [Google Scholar]
26.Patel MS, Kurtzman GW, Kannan S, et al. . Effect of an automated patient dashboard using active choice and peer comparison performance feedback to physicians on statin prescribing: the prescribe cluster randomized clinical trial. JAMA Netw Open. 2018;1(3):e180818. doi: 10.1001/jamanetworkopen.2018.0818 [DOI] [PMC free article] [PubMed] [Google Scholar]
27.Meeker D, Linder JA, Fox CR, et al. . Effect of behavioral interventions on inappropriate antibiotic prescribing among primary care practices: a randomized clinical trial. JAMA. 2016;315(6):562-570. doi: 10.1001/jama.2016.0275 [DOI] [PMC free article] [PubMed] [Google Scholar]
28.Halpern SD, Ubel PA, Asch DA. Harnessing the power of default options to improve health care. N Engl J Med. 2007;357(13):1340-1344. doi: 10.1056/NEJMsb071595 [DOI] [PubMed] [Google Scholar]
29.Patel MS, Day SC, Halpern SD, et al. . Generic medication prescription rates after health system–wide redesign of default options within the electronic health record. JAMA Intern Med. 2016;176(6):847-848. doi: 10.1001/jamainternmed.2016.1691 [DOI] [PMC free article] [PubMed] [Google Scholar]
30.Patel MS, Day S, Small DS, et al. . Using default options within the electronic health record to increase the prescribing of generic-equivalent medications: a quasi-experimental study. Ann Intern Med. 2014;161(10)(suppl):S44-S52. doi: 10.7326/M13-3001 [DOI] [PubMed] [Google Scholar]
31.Sharma S, Guttmann D, Small DS, et al. . Effect of introducing a default order in the electronic medical record on unnecessary daily imaging during palliative radiotherapy for adults with cancer: a stepped-wedge cluster randomized clinical trial. JAMA Oncol. 2019;5(8):1220. doi: 10.1001/jamaoncol.2019.1432 [DOI] [PMC free article] [PubMed] [Google Scholar]
32.Delgado MK, Shofer FS, Patel MS, et al. . Association between electronic medical record implementation of default opioid prescription quantities and prescribing behavior in two emergency departments. J Gen Intern Med. 2018;33(4):409-411. doi: 10.1007/s11606-017-4286-5 [DOI] [PMC free article] [PubMed] [Google Scholar]
33.Efron B, Tibshirani R. An Introduction to the Bootstrap. New York, NY: Chapman & Hall; 1993. doi: 10.1007/978-1-4899-4541-9 [DOI] [Google Scholar]
34.Davison AC, Hinkley DV. Bootstrap Methods and Their Application. New York, NY: Cambridge University Press; 1997. doi: 10.1017/CBO9780511802843 [DOI] [Google Scholar]
35.Schofield G, Kreeger L, Meyer M, et al. . Implementation of a quality improvement programme to support advance care planning in five hospitals across a health region. BMJ Support Palliat Care. 2015;5(1):91-94. doi: 10.1136/bmjspcare-2013-000574 [DOI] [PubMed] [Google Scholar]
36.Temel JS, Greer JA, Gallagher ER, et al. . Electronic prompt to improve outpatient code status documentation for patients with advanced lung cancer. J Clin Oncol. 2013;31(6):710-715. doi: 10.1200/JCO.2012.43.2203 [DOI] [PubMed] [Google Scholar]
37.Bestvina CM, Polite BN. Implementation of advance care planning in oncology: a review of the literature. J Oncol Pract. 2017;13(10):657-662. doi: 10.1200/JOP.2017.021246 [DOI] [PubMed] [Google Scholar]
38.Titano JJ, Badgeley M, Schefflein J, et al. . Automated deep-neural-network surveillance of cranial images for acute neurologic events. Nat Med. 2018;24(9):1337-1341. doi: 10.1038/s41591-018-0147-y [DOI] [PubMed] [Google Scholar]
39.Lin H, Li R, Liu Z, et al. . Diagnostic efficacy and therapeutic decision-making capacity of an artificial intelligence platform for childhood cataracts in eye clinics: a multicentre randomized controlled trial. EClinicalMedicine. 2019;9:52-59. doi: 10.1016/j.eclinm.2019.03.001 [DOI] [PMC free article] [PubMed] [Google Scholar]
40.Emanuel EJ, Ubel PA, Kessler JB, et al. . Using behavioral economics to design physician incentives that deliver high-value care. Ann Intern Med. 2016;164(2):114-119. doi: 10.7326/M15-1330 [DOI] [PubMed] [Google Scholar]
41.LeBlanc TW, Back AL, Danis M, Abernethy AP. Electronic health records (EHRs) in the oncology clinic: how clinician interaction with EHRs can improve communication with the patient. J Oncol Pract. 2014;10(5):317-321. doi: 10.1200/JOP.2014.001385 [DOI] [PMC free article] [PubMed] [Google Scholar]

Associated Data

This section collects any data citations, data availability statements, or supplementary materials included in this article.

Supplementary Materials