Electronic Health Record Mortality Prediction Model for Targeted Palliative Care Among Hospitalized Medical Patients: a Pilot Quasi-experimental Study

Abstract

Background

Development of electronic health record (EHR) prediction models to improve palliative care delivery is on the rise, yet the clinical impact of such models has not been evaluated.

Objective

To assess the clinical impact of triggering palliative care using an EHR prediction model.

Design

Pilot prospective before-after study on the general medical wards at an urban academic medical center.

Participants

Adults with a predicted probability of 6-month mortality of ≥ 0.3.

Intervention

Triggered (with opt-out) palliative care consult on hospital day 2.

Main Measures

Frequencies of consults, advance care planning (ACP) documentation, home palliative care and hospice referrals, code status changes, and pre-consult length of stay (LOS).

Key Results

The control and intervention periods included 8 weeks each and 138 admissions and 134 admissions, respectively. Characteristics between the groups were similar, with a mean (standard deviation) risk of 6-month mortality of 0.5 (0.2). Seventy-seven (57%) triggered consults were accepted by the primary team and 8 consults were requested per usual care during the intervention period. Compared to historical controls, consultation increased by 74% (22 [16%] vs 85 [63%], P < .001), median (interquartile range) pre-consult LOS decreased by 1.4 days (2.6 [1.1, 6.2] vs 1.2 [0.8, 2.7], P = .02), ACP documentation increased by 38% (23 [17%] vs 37 [28%], P = .03), and home palliative care referrals increased by 61% (9 [7%] vs 23 [17%], P = .01). There were no differences between the control and intervention groups in hospice referrals (14 [10] vs 22 [16], P = .13), code status changes (42 [30] vs 39 [29]; P = .81), or consult requests for lower risk (< 0.3) patients (48/1004 [5] vs 33/798 [4]; P = .48).

Conclusions

Targeting hospital-based palliative care using an EHR mortality prediction model is a clinically promising approach to improve the quality of care among seriously ill medical patients. More evidence is needed to determine the generalizability of this approach and its impact on patient- and caregiver-reported outcomes.

Electronic supplementary material

The online version of this article (10.1007/s11606-019-05169-2) contains supplementary material, which is available to authorized users.

INTRODUCTION

Hospitals nationwide have invested in palliative care programs to improve the quality of care for seriously ill patients.¹ Hospital-based palliative care consultation has been shown to decrease length of stay (LOS), readmission rates, and health care costs.^2–6 Palliative care has also been shown to increase survival in some patient populations.⁷ Emerging evidence further suggests that earlier palliative care consultation leads to greater family satisfaction with care.⁸ Despite these benefits, the frequency and timing of consultation is highly variable among patients with non-cancer diagnoses,⁹ due largely to a reliance upon clinicians to identify palliative care needs and refer patients for consultation. Thus, innovative and systematic strategies are needed to augment clinician referral of patients most likely to benefit from earlier palliative care consultation.^10–12

There has been a growing interest in triggering palliative care consultation using several different criteria, such as selected diagnoses,^{13, 14} disease-specific prognostic indicators,^{15, 16} or a patient’s location.^17–19 Although sound in principle, these approaches are nonspecific and assume palliative care needs and are therefore unlikely to be scalable given the growing number of seriously ill patients and the limited palliative care specialist workforce in the USA.^{20, 21} The rapid growth of data science combined with the breadth of available data in the electronic health record (EHR) may allow for more timely identification of patients based on actual palliative care needs using predictive analytics. Indeed, several machine learning EHR models recently developed to predict individual risk of mortality and other adverse health outcomes among hospitalized patients represent an important step in this direction.^22–27 Such prediction models offer great promise for systematically identifying patients most in need, yet none have been coupled with a palliative care intervention to evaluate clinical impact.^{28, 29}

We hypothesized that Palliative Connect, an intervention that combines predictive analytics with early triggered consultation, would improve the delivery of hospital-based palliative care. Thus, we conducted a pilot study to evaluate the clinical impact of triggering palliative care consultation (with clinician opt-out) on hospital day 2 among a diverse general medical population with high predicted risk of death within 6 months. We also explored the acceptability of Palliative Connect among clinicians and spill-over effects among lower risk patients.

METHODS

This pilot study was conducted among general medical patients on the hospitalist services at the Hospital of the University of Pennsylvania, a 791-bed tertiary care institution with over 34,000 adult admissions annually. Nine hospitalist teams care for an average of 85 patients daily. Each team is staffed by one attending hospitalist physician and either a certified registered nurse practitioner or three resident physicians. A multidisciplinary palliative care program that was established 9 years ago completes approximately 10 new consults and maintains a daily census of approximately 60 each day.

An 8-week intervention period was established a priori from December 1, 2017, to February 8, 2018 (excluding the holiday week) and compared with an 8-week historical control period. There were no other palliative care initiatives on the hospitalist services during the study period. This project was determined to qualify as quality improvement by the University of Pennsylvania Institutional Review Board; need for informed consent was waived.

Palliative Connect Intervention

Most existing EHR machine learning mortality prediction models are limited by inclusion of a restricted population,^{22, 23, 25} reliance on data only available at the end of a hospitalization,²⁴ or use of computationally complex methods that can hinder clinician interpretability and future deployment into the EHR production environment,^{27, 30} and none have been externally validated. Thus, we first developed and validated an EHR mortality prediction model with this health system’s data using a machine learning approach and regression analysis. We chose the predicted outcome of death within 6 months to identify seriously ill patients who would likely benefit from advance care planning (ACP) discussions in a timeframe sufficient for such discussions to potentially influence the likelihood that they receive goal-concordant care in the final stages of disease.^{31, 32}

We trained the supervised machine learning model using 64,246 admissions among 46,305 unique patients (excluded patients < 18 years of age and observation, obstetrics, hospice, or rehabilitation admissions) to three large urban hospitals at the University of Pennsylvania Health System in 2016; all used EPIC v2017 (EPIC Systems Corporation, Verona, WI). Characteristics between the training and test cohorts are summarized in eTable1 in the online supplement. Six-month mortality was determined from the date of hospital admission and using death dates in the EHR or Social Security Death Index (SSDI).³³ The 6-month mortality rates were 9.9% and 10.1% in the training and test sets, respectively. See the online supplement for additional details of the machine learning methods used for prediction model development and analyses. The final prediction model retained 35 covariates (age, gender, admission type, 18 Elixhauser comorbidities, and 14 laboratory values) significantly associated with risk of six-month mortality (online supplement eTable2). The model demonstrated excellent discrimination in the test set, with a c-statistic of 0.86 (95% CI 0.84–0.88) and good calibration, with the expected mortality within the 95% CIs of observed mortality except at the extreme deciles where there were few cases with the outcome or low sample sizes to inform the model.³⁴ eFigure1 and eFigure2 in the online supplement summarize the performance characteristics of the final model in the test set.

We then performed a chart review to determine the mortality risk (Palliative Connect score) threshold above which patients were most likely to need a palliative care consult for ACP discussion and documentation.^{35, 36} Two palliative medicine specialists (KC and NO) reviewed 30 random charts using data available in the EHR through hospital day 1 while blinded to the predicted mortality risk. Among those with a Palliative Connect score of ≥ 0.3 (13/30), the inter-rater reliability was 100% (Cohen’s Kappa = 1.0).

Finally, we built an intranet-based user interface to bring eligible patients to the attention of the palliative care team each day in a user-friendly format, and to track communication with the primary team clinicians (online supplement eFigure3). Patients ≥ 18 years of age at the time of hospital admission, on the hospitalist service, and with a Palliative Connect score ≥ 0.3 were eligible for a triggered consult using the intervention workflow shown in Fig. 1. Recognizing that patient identification alone is likely insufficient to change clinicians’ palliative care referral patterns, we applied insights from behavioral economics regarding the use of default (opt-out) options to nudge clinicians to deliver recommended care without restricting their choices.^37–39 Thus, the palliative care triage nurse offered a triggered consult to the primary clinician of patients on the list in descending order until a maximum of two consults were accepted. This approach preserved the palliative care team’s capacity for usual care consults without adding resources.⁴⁰ Remaining patients were carried over on the list each day until they were offered a triggered consult or discharged or transferred to another service. The content of triggered consults was left to the discretion of the palliative care clinician.

Figure 1.

Study process flow for palliative connect intervention. Each weekday, a Palliative Connect score (predicted risk of death within 6 months) was calculated for patients on hospital day 2. Patients with a score ≥ 0.3 populated a web-based user interface list in order from highest to lowest risk (actual prediction not shown). The palliative care team’s triage nurse called the primary clinicians of patients in descending order to offer an opt-out of the triggered consult until the maximum of two consults were accepted. Remaining patients were carried over on the list each day until they were offered a triggered consult, or they were discharged or transferred to another service. Consults requested per usual care were accepted. Palliative care clinicians and hospitalists completed surveys and clinical data was obtained from the clinical data warehouse. Figure 1 and 2 contains poor quality of text inside the artwork. Please do not re-use the file that we have rejected or attempt to increase its resolution and re-save. It is originally poor, therefore, increasing the resolution will not solve the quality problem. We suggest that you provide us the original format. We prefer replacement figures containing vector/editable objects rather than embedded images. Preferred file formats are eps, ai, tiff and pdf.fully vectorized .eps files attached along with original powerpoint figure file

Historical Patient Controls

To create a comparable cohort of control patients, we ran the prediction risk algorithm silently for 8 weeks prior to the intervention period. Patients were selected for inclusion in the control group using the same eligibility criteria as the intervention group, followed by a stochastic simulation⁴¹ that mimicked the descending Palliative Connect scores on the user interface, the triggered consult decline rate observed in the intervention period, and the limit of two accepted triggered consults per day. This approach could not account for other potential unmeasured factors in the opt-out decision process.

Outcomes

For all control and intervention patients, we evaluated several palliative care processes and outcomes, including completed consults (triggered and usual care), pre-consult length of stay (LOS), ACP documentation (defined by the presence of a completed “ACP note” type in the EHR), home palliative care and hospice referrals, change in code status (defined by a new do-not-attempt-resuscitation [DNAR] order in the EHR), hospital mortality, hospital LOS, intensive care unit (ICU) admission and LOS, and all-cause readmission within 30 days of discharge.

For intervention patients, we explored acceptability of the intervention by the proportions of (1) triggered consults declined by a patient and/or caregiver after the primary clinician accepted it; (2) hospitalists who agreed or strongly agreed that triggering palliative care for patients most likely to benefit is acceptable; and (3) completed consults determined to be appropriate by the palliative care clinician. We also collected triggered consult decline reasons. Clinician data were collected from post-consult surveys.

Finally, we evaluated for spill-over effects on palliative care consultation rates among two non-study populations on the hospitalist services: (1) patients with a risk score < 0.3 and (2) patients with a risk score ≥ 0.3 who were discharged or transferred services before their clinician was offered a triggered consult. All clinical data were retrieved from Penn Data Store, a data warehouse that includes EHR and post-discharge coded data.

Statistical Analyses

Analyses for this pilot study were primarily descriptive using summary statistics as appropriate. The analytic sample included all admissions offered a triggered consult. Comparisons between intervention and control groups were performed using Pearson’s chi-square or Fisher exact tests for categorical variables, and two-tailed t tests and Wilcoxon rank-sum tests for normally and non-normally distributed continuous variables, respectively. Statsmodels v0.6.1⁴² in the Python programming language was used for all analyses. Data are presented as number (percent) unless otherwise specified. Findings were considered statistically significant at P < 0.05. The datasets generated and/or analyzed during the study are not publicly available due to protected patient data, but may be made available from the corresponding author on reasonable request.

RESULTS

During the 8-week intervention period, 946 medical admissions to the hospitalist services were automatically screened in the EHR, 134 (14) of which were offered a triggered a consult (Fig. 2). Patient demographics and mortality risk scores in the control (n = 138) and intervention cohorts were similar (Tables 1 and 2).

Figure 2.

Palliative Connect study screening and enrollment. Screening occurred automatically in the EHR every day at 0630. The palliative care triage nurse called the primary clinicians of patients with a Palliative Connect score ≥ 0.3.

Table 1.

Patient Characteristics

	No. (%)*
Characteristic	Control	Intervention
No. of admissions	138	134
No. of patients	125	125
Age, median (IQR), years	72.7 (66.0, 81.6)	72.9 (63.02 83.0)
Female	52 (38)	60 (45)
Race
White	82 (59)	70 (52)
Black	49 (36)	59 (44)
Asian	5 (4)	2 (2)
Other/unknown	2 (1)	3 (2)
Married	81 (59)	74 (55)
Admission type urgent	138 (100)	133 (99)
Elixhauser Index, median (IQR)	8 (6, 12)	9 (6, 12)
Palliative Connect score, mean (SD)	0.5 (0.2)	0.5 (0.2)
Primary discharge diagnosis category†
Cardiac	16 (12)	15 (12)
Endocrinologic	3 (2)	2 (2)
Gastrointestinal	27 (20)	23 (17)
Hematologic/oncologic	22 (16)	29 (22)
Infectious	36 (26)	28 (21)
Musculoskeletal	1 (1)	1 (1)
Neurologic	5 (4)	6 (4)
Pulmonary	12 (9)	12 (9)
Renal	15 (11)	17 (13)
Rheumatologic	1 (1)	1 (1)

IQR interquartile range, SD standard deviation

*χ² testing was used to analyze categorical data; t testing was used for continuous data; P values were non-significant for all comparisons between control and intervention groups

^†Numbers may not add up to 100% due to rounding

Table 2.

Palliative Connect Outcomes

Measure	No. (%)		P value
	Control (n = 138)	Intervention (n = 134)
Palliative care consults	22 (16)	85 (63)	< .001
Pre-consult length of stay, median (IQR), d	2.6 (1.1, 6.2)	1.2 (0.8, 2.7)	.02
Advance care planning documentation	23 (17)	37 (28)	.03
Change in code status	42 (30)	39 (29)	NS
New home palliative care referral	9 (7)	23 (17)	.01
Hospice referral	14 (10)	22 (16)	NS
In-hospital mortality	6 (4)	2 (2)	NS
Hospital length of stay, median (IQR), days	5.7 (3.7, 9.1)	5.7 (3.7, 10.0)	NS
ICU admission	31 (23)	18 (13)	NS
ICU length of stay, median (IQR), days	4.3 (1.4, 6.0)	2.7 (1.9, 4.3)	NS
30-day all-cause readmission*	29 (22)	26 (20)	NS

NS non-significant, IQR interquartile range, ICU intensive care unit

*Denominator includes all patients who were discharged from the hospital alive with a minimum follow-up duration of 30 days after discharge

Process Measures and Outcomes

During the intervention period, primary clinicians accepted 77 (57) triggered consults and 8 additional consults were requested per usual care, with 3 occurring after the triggered consult was declined. Table 2 summarizes the clinical findings from this study. Palliative care consults increased by 74% in the intervention group compared to control (22 [16] vs 85 [63], P < .001), and occurred 1.4 days earlier after admission (2.6 [1.1, 6.2] vs 1.2 [0.8, 2.7], P = .02). There was a 38% increase in ACP documentation (23 [17] vs 37 [28], P = .03) and 61% increase in home palliative care referrals (9 [7] vs 23 [17], P = .01) during the intervention period. There were fewer ICU admissions, in-hospital deaths and 30-day readmissions, and more hospice referrals among the intervention group compared to historical controls, but these differences were not statistically significant. The median hospital LOS and proportion of patients with a change in code status were unchanged between the groups.

Acceptability of Triggered Palliative Care

Among the 57 (43) declined triggered consults, the most common reason cited by primary clinicians was “the patient has no palliative care needs at this time” (Table 3). No accepted triggered consults were declined by a patient or caregiver. Of the hospitalists who completed a post-consult survey (13/29; RR 45%), 10 (77) agreed or strongly agreed that triggering palliative care consults (with opt-out) for patients most likely to benefit was acceptable, and 3 (23) neither agreed nor disagreed. For all completed consults among the intervention cohort (81/85; RR 95%), palliative care clinicians were more likely to report that traditionally requested consults were appropriate compared to triggered consults (7 [100] vs 49 [66]; P = .04). However, when the 33 additional consults requested for patients with a risk score < 0.3 were included, there was no difference in reported appropriateness (P = .18).

Table 3.

Reasons Provided by Primary Team for Declining Triggered Palliative Care Consult (n = 57)

Reasons for declined triggered consults	No. (%)*
No palliative care needs at this time	26 (46)
Primary team meeting palliative care needs	4 (7)
Discharge anticipated soon	12 (21)
Hospice already consulted	5 (9)
Palliative care already involved†	9 (16)
Other	1 (2)

*Numbers do not add up to 100% due to rounding

^†Includes hospital and/or outpatient palliative care involvement

Spill-Over Effects of Triggered Palliative Care

Consultations among hospitalist admissions with a Palliative Connect score < 0.3 remained unchanged between the intervention and control periods (33/798 [4] vs 48/1004 [5]; P = .48), and similarly for admissions with a risk score ≥ 0.3 who were discharged or transferred services before the trigger (0/11 [0] vs 1/17 [6]; P = .41).

DISCUSSION

Increasing access to palliative care is a major focus of national efforts to improve the quality of serious illness care.^{43, 44} This pilot study suggests that using an EHR mortality prediction model to trigger palliative care may improve timely identification of seriously ill medical patients and increase consultation among this vulnerable population. Specifically, we found that palliative care consultation increased nearly fourfold compared to historical controls and occurred a day and a half earlier, along with higher ACP documentation rates.

The Palliative Connect prediction model identified hospitalized patients with a high risk of death within 6 months of admission who would not have otherwise received a consult. When nudged to consider palliative care for these patients, the primary team accepted more than half of the triggered consults. For declined consults, clinicians most often cited either a lack of palliative care needs or a belief that palliative care needs were already being met. Importantly, we did not assess clinician-patient agreement in these cases. Striking a balance between the needs of patients, hospitalist teams, and palliative care clinicians was essential to ensure feasibility and acceptability of this intervention. Primary teams were not prevented from requesting consults at any time during the study, but rather by limiting the triggered consults seen each day the palliative care team was still able to see all consults requested per usual care. However, it is evident from the triggered consult declination rates and palliative care clinicians’ assessments of consult appropriateness that further evaluation is needed to define the optimal trigger timing across different diseases and care settings.

Another important result of Palliative Connect was the increase in referrals to home palliative care. Such increased access to palliative care across the care continuum is essential to improving the quality of serious illness care, as patients spend the majority of time out of the hospital and the evidence for community-based palliative care to improve patient and caregiver outcomes is strong.⁴⁵ We also saw a trend toward increased hospice referrals, which is consistent with findings from a retrospective study of outcomes after palliative care consultation conducted in this study’s health system.⁵ Coupled with the increase in ACP documentation, this suggests that implementation of Palliative Connect may facilitate delivery of goal-concordant care, although this remains an elusive outcome to directly measure. Importantly, despite more palliative care consults and ACP documentation, there was not an increase in new DNAR orders during the intervention period. This finding should alleviate, at least in part, persistent misconceptions that palliative care is analogous to hospice or limiting treatment options.

In addition to providing new insights on key questions about clinician acceptability of triggering palliative care for hospitalized medical patients and its potential impact on important palliative care process metrics, this study also revealed promising trends in clinical outcomes. There were clinically meaningful reductions in-hospital mortality, 30-day all-cause readmissions, ICU admissions, and ICU LOS. Although we did not design this pilot study to detect statistically significant differences, these early findings are supported by existing observational evidence of the value of palliative care.

The Palliative Connect intervention has several key strengths that lend it to real-world applicability, including (1) systematic identification of patients at risk for poor health outcomes, (2) risk stratification to allow for selection of a desired threshold, (3) delivery of actionable information to clinical teams in real time, and (4) preservation of primary clinician decision autonomy. Moreover, as there is significant heterogeneity across hospitals and palliative care programs,^{46, 47} this intervention design enables adaptation to meet local implementation needs with subsequent evaluation of outcomes. For example, the optimal number of triggered consults seen each day, type of clinician nudge, and mortality risk threshold are likely to differ by site based on available resources, the culture of palliative care, and patient case-mix. Similarly, considerations to mitigate potentially unmanageable increases in consult volume²¹ and to facilitate scalability and sustainability might include using the prediction model to deliver primary clinician-directed educational interventions or nudges for “primary” palliative care before a consult is triggered, or for direct referrals to community-based palliative care, although it is not yet clear which patients would benefit from which intervention. Finally, incorporation of the perspectives and preferences of patients and caregivers for implementation could enhance acceptability.

Limitations

There are several important limitations of this study. First, this was a non-randomized, single-center study among general medical patients that used historical controls, so whether Palliative Connect could be generalizable to other inpatient populations is yet unknown. We opted not to randomize individual patients because of the potential for significant spill-over effects of increased consultation among non-study patients being treated by the same primary clinician, and alternate study design options that would randomize at the clinician or unit level were not practical for this small pilot study. Thus, differences in outcomes between the historical control and intervention groups may have been attributable to unmeasured patient characteristics, temporal trends, or a Hawthorne Effect among the primary team or palliative care clinicians. Second, this study was underpowered to detect statistically meaningful differences in clinical outcomes, so caution is warranted in interpretation of the intervention’s effectiveness. Third, a mortality risk-based trigger relies on the widely held belief that patients with higher risk have greater palliative care needs. In reality, patients’ actual palliative care needs are influenced by a combination of their prognosis, diagnoses, clinical team, and sociodemographic factors. Finally, all prediction model clinical decision-making tools are at risk for perpetuating human biases reflected in the underlying data, and for capturing practice patterns and case-mix at one time point.³⁰ Thus, rigorous assessments of fairness to promote health equity⁴⁸ and periodic model re-evaluation and recalibration are essential to ensure that the model remains valid and useful.⁴⁹

CONCLUSION

The Palliative Connect intervention demonstrated promising evidence of clinician acceptability and clinical impact among a general medical population at a large academic hospital. Such innovative palliative care delivery approaches that target patients most likely to benefit are critical to the field’s ability to sustainably provide high-value care. A randomized evaluation of Palliative Connect is needed in a more diverse population and setting to determine its effectiveness, including assessments of patient- and caregiver-reported outcomes.

Authors’ Contributions

Study concept and design: KC, CC, MB, SR, MD, NO

Collection, management, analysis, and interpretation of the data: KC, CC, MB, SR, LP, MD, NO

Drafted or critically revised the manuscript for important intellectual content: KC, CC, MB, SR, LP, MD, NO

All authors read and approved the final manuscript.

Funding

This work was funded in part by a career development award from the National Palliative Care Research Center (KRC).

Compliance with Ethical Standards

Conflict of Interest

The authors declare that they do not have a conflict of interest.