Integrating Machine-Generated Mortality Estimates and Behavioral Nudges to Promote Serious Illness Conversations for Cancer Patients: Design and Methods for a Stepped-Wedge Cluster Randomized Controlled Trial

Abstract

Introduction

Patients with cancer often receive care that is not aligned with their personal values and goals. Serious illness conversations (SICs) between clinicians and patients can help increase a patient’s understanding of their prognosis, goals and values.

Methods and analysis

In this study, we describe the design of a stepped-wedge cluster randomized trial to evaluate the impact of an intervention that employs machine learning-based prognostic algorithms and behavioral nudges to prompt oncologists to have SICs with patients at high risk of short-term mortality. Data are collected on documented SICs, documented advance care planning discussions, and end-of-life care utilization (emergency room and inpatient admissions, chemotherapy and hospice utilization) for patients of all enrolled clinicians.

Conclusion

This trial represents a novel application of machine-generated mortality predictions combined with behavioral nudges in the routine care of outpatients with cancer. Findings from the trial may inform strategies to encourage early serious illness conversations and the application of mortality risk predictions in clinical settings.

Introduction

Patients with cancer often undergo cancer care, including costly therapy and acute care utilization that is discordant with their values, goals, and preferences, particularly at the end of life. Serious illness conversations (SICs) between clinicians, patients, and families help identify a patient’s prognostic awareness and explores their priorities and goals.^1–9 Furthermore, early discussions regarding a patient’s goals and values can improve goal-concordant care, resulting in better quality of life and reduced emotional distress and health spending.^10–12 For those reasons, many United States quality metrics – including those from the American Society of Clinical Oncology Quality Oncology Practice Initiative (QOPI) – track documentation of patients’ goals and wishes at the end of their life. Still, most patients with advanced cancer die without a documented advance care planning discussion or serious illness conversation.^{13, 14}

A key reason for this gap may be that oncologists routinely overestimate life expectancy of patients with advanced cancer. Oncologists’ prognostic estimates approximate actual patient survival only 20% of the time, contributing to low rates of documented advance care planning and more aggressive care near the end of life.^15,16 Existing prognostic aids in oncology are rarely used because they do not apply to most cancers, do not identify most patients who will die within one year, and require time-consuming data input.^17–20

Electronic health record (EHR)-based predictive algorithms may improve clinicians’ prognostication and decision-making.^21–23 Automated prognostic assessments can be integrated into workflows to remind clinicians about the need for SICs, reliably discriminate between patients at high versus low risk of mortality, and minimize choice overload by identifying patients appropriate for early SICs. While EHR-based predictive algorithms have shown promise in acute care settings, they have not been applied in the outpatient setting to prompt SICs.²⁴

Behavioral economic principles may further engage clinicians to act on these mortality estimates and increase frequency of SICs. Feedback on personal performance combined with social incentives in the form of peer comparisons may encourage clinicians to adjust their behaviors based on their social ties and connections.^25–27 Making predictions salient to providers by sending opt-out reminders can improve adherence to evidence-based interventions.²⁸ These strategies have demonstrated the ability to overcome barriers to behavior change, and combined with mortality predictions, may drive an increase in SICs.

This paper describes the design and methods for a pragmatic, stepped-wedge randomized controlled trial, which is currently ongoing, that applies machine-generated mortality predictions and behavioral nudges to prompt initiation and documentation of SIC discussions between oncology clinicians and their patients at highest risk of short-term mortality. We compared results of the intervention to usual care alone. This study is innovative because it (1) is one of the first applications of real-time machine learning-based predictions in routine oncology practice; (2) combines mortality predictions with behavioral nudges in the form of performance reports, peer comparison, and text message reminders; and (3) is a multifaceted approach to increase SICs – a metric that has been difficult to meaningfully increase in other quality improvement attempts.^29–31

Methods and analysis

Study Overview

This study includes oncology clinicians practicing at nine oncology clinics within a large academic health system. Clinicians receive an intervention that combines patient mortality predictions and behavioral nudges to encourage clinicians to have and document SICs with their patients. The study uses a pragmatic, stepped-wedge cluster randomized design to evaluate the impact of the intervention on documented SICs for patients seen over a 40-week study period (16-week intervention period and 24-week post-intervention period) by oncology clinicians at the nine sites.

This trial utilizes Conversation Connect, an electronic health record (EHR)-based machine learning algorithm that uses real-time patient EHR data, including demographic information, comorbidities, lab values, and encounters with the health system over the prior six months, to estimate individuals’ risk of dying in the subsequent six months. The prediction algorithm has been validated and shown to have good discrimination and positive predictive value.³² In the pilot application of this mortality algorithm, clinicians indicated that 59% of patients identified by the algorithm were appropriate for an SIC discussion.³²

Patient Involvement Statement

Patients were not involved in the design of this study. However, the study’s purpose is to improve communication between patients and clinicians about patient care preferences; the outcome selection of increasing Serious Illness Conversations is based upon prior literature demonstrating that cancer patients are open to having SICs with their oncologists and that SICs can lead to earlier, more frequent and better quality conversations about cancer care preferences, as well as improve patient depression and anxiety.^6–8 In addition, the Serious Illness Conversation Guide that is the foundation for Serious Illness Conversations used in this trial was developed with patient input.³⁴

Participant Recruitment

Subject enrollment is ongoing. Clinicians (medical oncologists, nurse practitioners, and physician assistants) at two practice locations within a large academic health system were eligible for participation in the study. Clinicians and their patients were automatically included in the trial if they cared for adult patients with cancer at the Pennsylvania Hospital Oncology clinic (a general oncology practice), or one of the following eight disease-specific clinics at the Perelman Center for Advanced Medicine (a tertiary academic oncology practice with disease-site specific clinics): breast, gastrointestinal, genitourinary, lymphoma, melanoma, central nervous system, myeloma and thoracic/head and neck. Of the eight disease site-specific clinics in the academic practice, two clinics (central nervous system tumors and melanoma) had a small number of clinicians and were grouped together. The investigators discussed the trial intervention at practice and disease site-specific meetings prior to trial enrollment to obtain clinician feedback on the trial intervention. Clinicians were excluded from the trial if they (1) solely cared for patients with benign hematologic or genetic disorders; (2) saw fewer than 12 high-risk patients in either the pre- or post-intervention periods; or (3) had not undergone SIC training at the time of trial initiation.³³ Only patients of clinicians enrolled in the trial receiving medical oncology care at the University of Pennsylvania Health System were eligible for participation in the study. Patients visits for genetics consultations will be excluded from the analysis, though these patients could be enrolled for other visits with a medical oncologist enrolled in the study (e.g. the patient’s visit with their oncologist would be included but their visit for a genetics evaluation would not be included).

Randomization

Randomization has been completed. There were a total of eight oncology clinics – seven disease-specific clinics and one general oncology practice – that were randomized. Clinics were stratified by those above and below the median baseline SIC rate (0.65 SICs per 100 unique patients) and electronically randomized through computer-generated random numbers to one of the four wedges using a block size of two clinics, such that each block had one clinic above the median SIC rate and one clinic below the median SIC rate. The senior author and statistician were blinded to the randomization sequence, and the co-primary authors assigned participants to interventions and implemented the randomization via email. A wedge started the intervention every four weeks over a period of 16 weeks until all clinics received the intervention (Figure 1). The study follow-up period is 24 weeks post-intervention, for a total study duration of 40 weeks.

Figure 1.

Intervention rollout among participating clinics

Interventions

Control arm: standard communication

As standard of care, all enrolled clinicians had been trained in the Serious Illness Conversation Guide at least three months prior to starting the intervention.³³ For approximately one year prior to trial enrollment, clinicians received personalized weekly automated emails that contained two bar graphs that compared their individual SIC performance to the blinded performance of others in their disease-specific clinic: one displaying the cumulative total number of SICs ever performed, and another displaying the number of SICs within the last week. Clinicians had SICs with patients as they deemed appropriate. Clinicians continued this standard of care until they received the intervention.

Intervention arm: mortality estimates and email and text nudges

Each clinic was assigned to one of four wedge start dates, as described above. On the start date, clinicians received a weekly automated email on Thursday for the upcoming week that provided information on how many SICs they had in the prior 4 weeks and a peer comparison message (Figure 2A). The ten clinicians with the most SICs documented in the prior four weeks received one of two messages: (1) a message that referred to the institutional goal of 1–2 SICs per week, if they had fewer than 8 SICs in those four weeks; or (2) a positive feedback message if they had ≥8 SICs over a four-week period (Figure 2B). All other providers received a peer comparison message that detailed the number of oncology clinicians who documented more SICs than them in the prior four weeks.

Figure 2.

Example peer comparison emails

The email also contained a link to a secure web interface, Conversation Connect (Figure 3). Upon logging in, clinicians could view a list of up to six of their patients scheduled for a visit in the coming week with the highest-risk of machine-predicted six-month mortality. The interface indicated the patient’s name, appointment day and time, presence or absence of a documented SIC, and an option to opt-out of the default reminder text messages. Patients with a documented SIC in the prior 2 months had a default opt-in option (the text reminder box was unchecked).

Figure 3.

Conversation Connect interface

At 8 AM on the day of the patient’s visit, clinicians received a secure text message reminding them to consider an SIC for the patients on their list. Clinicians did not receive reminder text messages for patients who already had a documented SIC in the prior 60 days unless the clinician opted in on the Conversation Connect interface. Clinicians could choose to withdraw from receiving emails or text messages at any point, in which case they would revert to receiving no emails related to SIC performance. Beginning on December 26, 2019, in response to clinician feedback, the email prompt intervention was revised to no longer give providers information on how the number of serious illness conversations over the past month compared to peer oncology providers. The other elements of the email remained unchanged.

Outcome Measures

The primary outcome is the change in the number of documented SICs per 100 unique patient visits, before and after the intervention, during the four-week lead-in period (baseline) and 16-week intervention period. The secondary outcomes are the changes during the four-week lead-in period and 16-week intervention period in (1) the number of documented SICs per 100 unique high-risk patient visits (the patients flagged by our algorithm to be presented to clinicians every week; in the pre-intervention period, clinicians were not shown the list of patients); and (2) the number of documented Advance Care Planning (ACP) notes (a broader measure of care planning that may or may not include elements about end-of-life care preferences) per 100 unique patient visits. Additional secondary outcomes assess the previously described outcomes over the four-week lead-in period and 40-week study period.

Exploratory outcome measures are changes in healthcare utilization metrics over the four-week lead-in period and 40-week study period. These metrics include the number of outpatient urgent care clinic visits for oncology patients, and, for decedents, any of the following in the last thirty days of life: receipt of chemotherapy, the number of emergency department admissions, the number of inpatient hospital admissions, and the number of intensive care unit admissions.

Statistical Analysis

Analysis of the trial is ongoing. Baseline data was collected prior to the start of the intervention and outcome data was to be collected from the EHR at the end of the primary outcome (16 weeks) and secondary outcomes (16 and 40 weeks). Interim analysis was to occur at the end of the primary outcome (16 weeks) and final analysis was to occur at the end of all secondary outcomes (40 weeks). There will be no early termination of the trial due to large intervention effect, as each group was to eventually receive the intervention after 16 weeks. All analyses use intention-to-treat with the patient as the unit of analysis, clustering at the level of the oncologist. MP and CR were blinded to the intervention order until the end of collection of the primary outcome, but the remaining study investigators and participating clinicians are involved in implementation of the intervention and cannot be blinded. MP and CR could be unblinded if there were any patient safety or ethical concerns about the trial or its analysis. Patients with a documented SIC or ACP before the lead-in period are excluded from the analysis. Patients with lung cancer who are enrolled in an ongoing clinical trial of a palliative care intervention are also excluded, as this intervention may confound the effects of our intervention. Physician assistants and nurse practitioners also receive the intervention; for purposes of the analyses, they are associated with the oncologist with whom they primarily work.

The wedges are separated by four-week intervals with a four-week pre-intervention lead-in period. To avoid bias due to varying frequency of follow-up visits between disease groups, the primary analysis is based upon the presence or absence of a documented SIC at the patient- wedge level. Patients are assigned to the wedge in which they first saw the oncology team. The primary outcome is expressed as a standardized rate of documented SIC discussions (number of documented SIC notes / 100 unique patient visits). In the main adjusted analysis, we fit models using generalized estimating equations using intervention (yes vs. no), group (oncology site-specific clinics) and period (temporal variable at the wedge stage) fixed effects, clustered by attending oncologist and use a two-sided alpha of 0.05 as the threshold for statistical significance.

To test the robustness of our findings, we are conducting several sensitivity analyses. First, we fit a model that incorporates patient observations in multiple wedges but where the patient is removed from the sample once an SIC or ACP, respectively, is documented. Second, we adjust for the number of visits that a patient has during the data collection period, available patient characteristics and comorbidities such as age, sex, race, marital status, insurance status, select lab values and the Charlson Comorbidity Index. Third, we include patients enrolled in a palliative care lung cancer trial, as many of these patients may appear in high-risk lists for lung cancer clinicians.

Data management and monitoring

Details of data management and quality assurance, including protection of patient confidentiality, can be found in the protocol (see Supplement). The investigators provide oversight for the study evaluation of this health system initiative. Clinicians use their clinical judgment to determine the appropriateness of initiating ACPs with patients, in accordance with standard of care, and the institutional review board deemed that there was no need for an independent data monitoring committee. An approved protocol to monitor for adverse events including loss of confidentiality is followed and additional support for patients with symptoms of psychological distress is available through the Department of Medicine. The trial will be audited regularly through the Abramson Cancer Center Clinical Trials Scientific Review and Monitoring Committee, which is independent from the investigators. Important protocol modifications, including changes to eligibility criteria, outcomes, and analyses, will be communicated to all relevant parties, including the investigators, Institutional Review Board, and trial participants.

Sample size

Power calculations were performed for the primary outcome: change in the number of documented SICs per 100 unique patient visits. Using retrospective baseline data from our health system, we estimated that there would be at least 80% power to detect change in the SIC rate of 0.8 per 100 patient visits. This estimate assumes that the baseline rate was 1.4 per 100 visits with standard deviation of 0.018, and a significance level with a two-sided alpha of 0.05.

Trial results will be communicated by publication and presentation. The study protocol (last modified December 6, 2019)is available in the Supplement. All changes to the protocol have been approved by the University of Pennsylvania Instituitional Review Board. All study investigators will have access to the final dataset for analysis. Statistical code from analysis will be made available upon request.

Results

Collection of data for all outcomes will be complete in April 2020.

Discussion

This trial represents the first application, to our knowledge, of machine-generated mortality predictions combined with behavioral nudges to the routine care of outpatients with cancer. This trial assesses the feasibility and effectiveness of combining machine-generated prognoses and behavioral nudges to improve goal-concordant care for patients with cancer. While the trial is not powered to assess the impact of the intervention on end-of-life utilization, a positive impact of the intervention on the trial’s primary and secondary endpoints of documented SICs may be a surrogate marker that can be rigorously tested in a larger, multi-institutional trial. It also provides important pragmatic insights into how clinicians interact with the intervention, including whether clinicians access the website regularly or opt out of texts, which informs further applications of machine-generated prognoses in the clinical setting.

The trial has several limitations. Most patients in the trial are seen in tertiary academic oncology clinics based in the United States, though patients seen in the general oncology practice that may be more representative of the general United States oncology population are also included. Non-specialty and non-academic clinicians may respond to the intervention differently. The trial does not include patients with gynecologic malignancies (as these patients are treated by gynecologic oncologists in a different division of the health system) or patients with leukemia or myelodysplastic syndromes (as our prior work noted that the mortality prediction algorithm was not as accurate in those populations). The mortality prediction algorithm may not be equally accurate in all disease subtypes, which may result in different levels of clinician trust in the predictions. Secular trends towards increased documentation of SICs may also influence the trial’s ability to detect a meaningful change due to the intervention.

Increasing goal-concordant care and reducing unwanted aggressive care at the end of life has been a persistent challenge in oncology practice. This trial uses a pragmatic clinical trial design to evaluate if machine-generated mortality predictions and behavioral nudges can increase documented SICs. Results from this trial may provide a model for future analytics-driven interventions in end-of-life care.