Palliative Care Exposure Relative to Predicted Risk of 6-Month Mortality in Hospitalized Adults

Abstract

Context:

The optimal strategy for implementing mortality-predicting algorithms to facilitate clinical care, prognostic discussions, and palliative care interventions remains unknown.

Objectives:

To develop and validate a real-time predictive model for 180-day mortality using routinely available clinical and laboratory admission data and determine if palliative care exposure varies with predicted mortality risk.

Methods:

Adult admissions between 10/1/2013 and 10/1/2017 were included for the model derivation. A separate cohort was collected between 1/1/2018 and 7/31/2020 for validation. Patients were followed for 180 days from discharge, and logistic regression with selected variables was used to estimate patients’ risk for mortality.

Results:

In the model derivation cohort, 7963 events of 180-day mortality (4.5% event rate) were observed. Median age was 53.0 (IQR 24.0–66.0) with 92,734 females (52.5%). Variables with strongest association with 180-day mortality included: Braden Score (OR 0.83; 95% CI 0.82–0.84); admission Do Not Resuscitate orders (OR 2.61; 95% CI 2.43–2.79); admission service and admission status. The model yielded excellent discriminatory ability in both the derivation (c-statistic 0.873; 95% CI 0.870–0.877; Brier score 0.04) and validation cohorts (c-statistic 0.844; 95% CI 0.840–0.847; Brier score 0.072). Inpatient palliative care consultations increased from 3% of minimal-risk encounters to 41% of high-risk encounters (p<0.01).

Conclusion:

We developed and temporally validated a predictive mortality model for adults from a large retrospective cohort, which helps quantify the potential need for palliative care referrals based on risk strata. Machine learning algorithms for mortality require clinical interpretation, and additional studies are needed to design patient-centered and risk-specific interventions.

INTRODUCTION

Advances in machine learning provide an opportunity to predict healthcare outcomes with greater accuracy, potentially assisting in clinical prognostication and shared decision-making (1–3). In particular, machine learning approaches utilizing electronic health record (EHR) data have been shown to accurately predict short-term mortality in general medicine settings (4–8). While some models have focused on the prediction of inpatient mortality for all adult patients regardless of illness acuity and severity (9), others have tested algorithms for patients with a particular serious illness such as cancer (10–12). The latter model’s implementation is now being evaluated in a randomized clinical trial to encourage both oncologists and high-risk patients to engage in serious illness conversations (13,14), to increase the frequency, timing, and documentation of such discussions (NCT04867850).

With the development of predictive models for mortality, there has been a parallel interest to improve delivery of palliative care (PC) as a mechanism to address unmet needs and optimize care at the end of life (15). Though a few studies have sought to leverage predictive mortality models to trigger palliative care consultations (PCCs) (16–18), it remains unknown whether this is the best strategy to facilitate clinical care and prognostic discussions for all patients. A more nuanced approach may be warranted to proactively address unforeseen consequences and understand the specific limitations when implementing mortality models in larger healthcare systems (19).

Herein, we developed and validated a model using routinely available clinical and laboratory EHR data upon admission to predict 180-day mortality in adult patients within our academic medical center. To evaluate current practice patterns, we analyzed PC utilization according to mortality risk groups using our validation cohort. Our objectives were to not only develop a risk-prediction model that could be easily integrated within our EHR, but to also determine if PC exposure objectively varied across risk strata.

METHODS

Data Source

This study was conducted at an academic, quaternary healthcare center in a major metropolitan city in the United States. Study cohorts were derived from inpatient admissions. Patient EHRs were initially from a home-grown integrated interface (model development), and later from the clinical information systems, EPIC (model validation). The change in EHR system occurred on November 2, 2017. To avoid potential negative effects of the EHR transition on the quality of the data, the time period for the derivation cohort ended before this date, and the time period for the validation cohort began after this date. This project was approved by the Institutional Review Board, which waived the need for informed consent for the use of identifiable data.

Model Development and Performance

The primary objective of this study was to develop a real-time predictive model for six-month mortality in adult patients using routinely collected clinical and laboratory data within the first 24 hours of admission. All hospitalizations with an order placed for inpatient admission were included. There were no exclusion criteria for model development, as it was designed to adapt to all levels of acuity and severity of medical illness. Prior outpatient PC visits, inpatient PCCs from previous hospitalizations, prior advance care planning or goals-of-care conversations, and presence of advance directives did not exclude patient-encounters from analysis for model development. Distinct admission encounters for the same patient were included in our analysis.

The derivation cohort consisted of 176,672 inpatient admissions between October 1, 2013 and October 31, 2017. EHR data elements from patient demographics, admission information, and laboratory test results were identified as candidate predictors. The earliest recorded values for each variable were used for model-building purposes. Mean value imputation was used for missing data. Model performance was evaluated for gender and race in the derivation cohort, to ensure that no subpopulations were inadvertently harmed or discriminated against. A separate temporal cohort of 113,511 inpatient admissions from January 1, 2018 to July 31, 2020 was used to validate the model.

All patients were followed for 180 days within discharge to determine the event rate of mortality. Outpatient mortality events were captured by notifications of a patient’s death through hospice facilities or by family members. Mortality events were verified by standard operational processes through our institution’s Decedent Affairs and Office for Quality, Safety, and Risk Prevention.

Relationship with Palliative Care Exposure

PC exposure was examined within our validation cohort using predefined risk groups according to percent predicted mortality: minimal-risk <10%, low-risk 10–24%, moderate-risk 25–49%, and high-risk 50–100%. Though recent studies have used a 30–40% mortality risk threshold for high-risk vs low-risk categorization (11,16), we elected to further stratify into four risk groups to identify specific patient populations that could benefit from a variety of interventions. Through electronic data capture, we collected: 1) prior PC exposure, defined as the presence of either inpatient PCCs or outpatient referrals within six months prior to the admission encounter, 2) the presence of inpatient PCCs and 3) time to inpatient PCCs during the current admission encounter. We also analyzed the relationship and frequency distribution between predicted risk of 180-day mortality and PCC status along a continuous scale. Lastly, we evaluated the characteristics and range of discharge disposition relative to each risk strata.

Statistical Analysis

Patient variables were univariately summarized using counts and proportions for categorical variables and medians with interquartile ranges (IQR) for continuous variables. Potential predictor variables were tested for association with 180-day mortality using an uncorrected chi-square for categorical and a Mann-Whitney U test for continuous variables. A two-sided P-value of less than 0.05 was used to indicate statistical significance. No adjustments were made for multiple comparisons.

Logistic regression was used to estimate patients’ risk for 180-day mortality at the encounter level. Variables were considered for inclusion in the model based on expert clinical opinion, availability within the electronic health record within 24 hours of admission, and previously established association with mortality. Applying a conservative limit of 15 mortality events per predictor variable, it was determined that the number of events in our development cohort was sufficient to support a large pool of potential variables without overfitting our model. Approximately 50 variables were considered a priori as candidate predictors. Patient race was excluded as a candidate predictor variable to avoid introducing bias. Final inclusion in the model was ultimately based on univariate and multivariate strength of association with 180-day mortality. Restricted cubic splines with 5 knots were used for continuous variables to test whether our model would be improved with the addition of non-linear terms. Performance was compared between the linear and non-linear models. Model interactions were not pre-specified and thus, interaction terms were not included.

Model performance was assessed using visual examination of calibration curves, c-statistic with 95% confidence interval (CI), Brier scores, and bootstrap validation. A temporal validation was performed to further assess the accuracy and stability of the model. Model predicted risk distribution and accuracy were checked within patient sub-populations using median and IQR of predicted risk and c-statistic with 95% CI. An uncorrected chi-square test was used to test for differences in proportion of PC exposure across risk categories. Time to PCC orders was summarized by risk category. The relationship between predicted risk and PC exposure was also examined using smoothed spline curves juxtaposed with rug plots for risk frequency.

Statistical analysis was performed using R version 4.0.3. This study adheres to the Transparent Reporting of a Multivariable Prediction Model for Individual Prognosis and Diagnosis (TRIPOD) reporting guideline for prediction model development and validation (20).

RESULTS

Patient Characteristics

There were 176,672 adult patient admission encounters in the derivation cohort and 113,511 in the temporal validation cohort. The characteristics of these hospitalizations in both cohorts are described in Table 1. Median age and gender distribution were similar, with slightly more admissions for women compared to men, in both cohorts. The majority of patients identified as white (79.7%) in the derivation cohort. The validation cohort demonstrated higher percentages of do not resuscitate (DNR) orders on admission (5.9% vs 4.1% derivation) and inpatient PCCs (6.1% vs 3.9% derivation); however, the percentage of patients who died in hospital during their encounter was stable at 3% across cohorts.

Table 1:

Characteristics of the overall cohort population. Data presented as N (%) for categorical variables and median (interquartile ranges [IQR]) for continuous.

Characteristic	Derivation Cohort(N = 176672)	Validation Cohort(N = 113511)
Age in years	53.0 (34.0 – 66.0)	56.0 (37.0 – 69.0)
Gender
Female	92734 (52.5)	59296 (52.2)
Male	83936 (47.5)	54211 (47.8)
Unknown	2 (0.0)	4 (0.0)
Racea
White	140874 (79.7)
African American	26365 (14.9)
Asian	2362 (1.3)	-
American Indian or Alaska Native	432 (0.2)
Pacific Islander	158 (0.1)
Unknown/Declined/Other	6481 (3.7)
Admission Status
Emergency	108201 (61.2)	58079 (51.2)
Elective	43764 (24.8)	26903 (23.7)
Urgent	22523 (12.7)	25907 (22.8)
Trauma Center	2166 (1.2)	2135 (1.9)
Unknown/Missing	18 (0.0)	487 (0.4)
Admission Service
Obstetrics	18726 (10.6)	12373 (10.9)
Cardiology	16154 (9.1)	10146 (8.9)
Trauma	12350 (7.0)	7678 (6.8)
Psychiatry	11425 (6.5)	9 (0.0)
Orthopedics	10314 (5.8)	6332 (5.6)
Pulmonary	8942 (5.1)	7086 (6.2)
General Surgery	7979 (4.5)	3538 (3.1)
Neurology	7966 (4.5)	5199 (4.6)
Hematology	6616 (3.7)	3552 (3.1)
Oncology	4779 (2.7)	3111 (2.7)
Urology	3843 (2.2)	1868 (1.6)
Other	67578 (38.3)	52619 (46.4)
Transfer from Outside Hospital	35346 (20.0)	29933 (22.1)
Number of Hospitalizations in 12 Months Prior to Admission	0 (0 – 1)	0 (0 – 1)
DNR Diagnosis on Admission	7182 (4.1)	6648 (5.9)
Albuminb	3.5 (2.9 – 3.9)	3.7 (3.2 – 4.1)
RDWc	14.1 (13.1 – 15.6)	14.1 (13.1 – 15.9)
O2 Saturationd	99.0 (98.0 – 100.0)	98.0 (96.0 – 99.0)
BUNe	15.0 (11.0 – 23.0)	16.0 (11.0 – 24.0)
Plateletsf	227.0 (174.0 – 290.0)	230.0 (171.0 – 299.0)
MCVg	90.0 (86.0 – 94.0)	89.0 (85.0 – 94.0)
WBCh	9.3 (6.8 – 12.8)	9.4 (6.7 – 13.2)
Length of Stay	3.0 (2.0 – 6.0)	3.0 (2.0 – 6.0)
Palliative Care During Admission	6929 (3.9)	6871 (6.1)
Died in Hospital	5372 (3.0)	3380 (3.0)
Died within 180 Days of Discharge	7963 (4.5)	10658 (9.4)

^{a –}Race: Patient race was not collected in validation cohort because it was not part of the model risk calculation

^{b –}[ALBUMIN] Missing for 122401 (69.3%) in derivation cohort and 68429 (60.3%) in temporal validation cohort.

^{c –}[RDW] Missing for 24887 (14.1%) in derivation cohort and 38752 (34.1%) in temporal validation cohort.

^{d –}[O2 Sat] Missing for 144705 (81.9%) in derivation cohort and 817 (0.7%) in temporal validation cohort.

^{e –}[BUN] Missing for 29440 (16.7%) in derivation cohort and 20501 (18.1%) in temporal validation cohort.

^{f –}[PLTCT] Missing for 21163 (12.0%) in derivation cohort and 38852 (34.2%) in temporal validation cohort.

^{g –}[MCV] Missing for 24776 (14.0%) in derivation cohort and 38696 (34.1%) in temporal validation cohort.

^{h –}[WBC] Missing for 24948 (14.1%) in derivation cohort and 38715 (34.1%) in temporal validation cohort.

Model Description

In the derivation cohort, a total of 7963 events of mortality within 180 days (4.5% event rate) were observed. Our final logistic regression model included 16 predictor variables: age, gender, admission status (emergency, elective, trauma-center, urgent, unknown/missing), admission service, transfer status (i.e., patient transferred from an outside facility), number of hospitalizations in the last 12 months, DNR present on admission, emergency department admission flag (i.e., patient admitted through the emergency department), Braden score, O₂ saturation, and six admission laboratory values: albumin, RDW, MCV, WBC, BUN, and platelet count. This final model was fit using 7963 mortality events and 29 degrees of freedom (274.6 events per degree of freedom), which demonstrates that our model does not overfit the data. Figure 1 summarizes the 16 variables by descending order of importance and by adjusted strength for prediction, as quantified by model chi-square statistics. Odds ratios of each predictive variable are simultaneously presented. Variables that exhibited the strongest association with 180-day mortality included: Braden Score (OR 0.83; 95% CI 0.82–0.84; P<0.01), DNR presence (OR 2.61; 95% CI 2.43–2.79; P<0.01), admission status, and admission service, with highest odds ratios from inpatient malignant hematology and solid tumor oncology services (eTable 1). Of note, DNR presence was defined by having a DNR order placed by the admitting clinician within 24 hours of a patient’s hospital admission. Any DNR orders made prior to the admission encounter and/or resuscitation preferences documented on advance directives or Physician Orders for Life-Sustaining Treatment (POLST) forms must have been confirmed by the admitting clinician, with a new order placed in the EHR for risk calculation. DNR orders made by healthcare teams after the initial 24-hour window were not included in the model.

Figure 1:

Visual representation of the final predictors used in the derived 180-day mortality risk prediction model. Scores are ordered by the adjusted strength of the predictor as quantified by model chi-square statistic. The figure also includes OR for each variable, along with 95% CI and p-values.

The final model displayed excellent discriminatory ability within our model-fitting cohort (c-statistic 0.873; 95% CI 0.870–0.877; Brier score 0.04). In comparing this model to the model including non-linear terms, it was determined that the non-linear model did not offer a large enough increase in predictive accuracy (c-statistic 0.876; 95% CI 0.872–0.879; Brier score 0.04) to justify the additional complexity. The correlation matrix and model standard errors were reviewed, and no evidence of substantial multicollinearity was found. Evaluation of the model by patient subpopulations for gender and race confirmed the model’s performance and consistency across subgroups (eTable 2). In concordance with our model performance, there was greater frequency of higher predicted risk scores (>10%) in those patients who died within 180 days of discharge, and conversely, there was a greater frequency of lower predicted risk scores (<10%) in those patients who did not die within 180 days of discharge (eFigure 1).

Model Temporal Validation

In the temporal validation cohort, a total of 10,658 events of mortality within 180 days (9.4% event rate) were observed. When applying the original model and coefficients to the validation cohort, the model again displayed excellent discriminatory ability (c-statistic 0.844; 95% CI 0.840–0.847; Brier score 0.07). Due to a difference in mortality event rate in our validation cohort (9.4% vs 4.5% derivation, p<0.01), the calibration curves (Figure 2) show the model tended to under-estimate risk of mortality on average.

Figure 2:

Calibration curves for 180-day mortality risk prediction model. Left panel shows performance in the model derivation cohort, right panel shows performance in validation cohort. C-statistic in model derivation cohort = 0.873, 95% CI 0.870 to 0.877. C-statistic in validation cohort = 0.844, 95% CI 0.840 to 0.847.

Risk Stratification and Palliative Care Utilization:

Patient encounters from the validation cohort were stratified according to their predicted risk (Table 2). Prior PC exposure increased as predicted mortality risk increased, spanning from 1% of minimal-risk to 11% of high-risk encounters (chi-square test statistic: 2744.7, p<0.01). PCCs during the current encounter increased from 3% of minimal-risk encounters to 19% of low-risk, 33% of moderate-risk, and 41% of high-risk encounters (chi-square test statistic: 13501.4, p<0.01). Time to PCC remained relatively stable across risk categories, with a median of 2.6 days (IQR 0.9–6.6) in minimal-risk encounters and a median of 2.0 days (IQR 0.7–4.8) in high-risk encounters. As clinically expected, fewer patients were discharged home (9% in high-risk vs 79% in minimal-risk) and more patients died during their admission (46% high-risk vs 1% in minimal-risk) as predicted mortality increased. Figure 3 depicts spline curves to further characterize the relationship between predicted risk on a continuous scale versus proportion of patient encounters with PC exposure. As predicted risk increases, the proportion of patients with prior PC exposure only modestly increases and does so uniformly. However, when assessing the relationship with PCCs during the current admission encounter, we observe an increase in PCC rates as risk increases from 0 to 75%, followed by a decrease in PCC rates as risk increases from 75% to 100%.

Table 2:

Table shows palliative care order characteristics and discharge disposition by admission risk from the validation cohort. Data presented as % (N).

	Minimal Risk[0, .1) N = 96219	Low Risk[.1, .25) N = 11640	Moderate Risk[.25, .5) N = 4236	High Risk[.5, 1] N = 1416
Palliative Care Order (Prior to Admission)	0.01 (1134)	0.06 (647)	0.09 (388)	0.11 (160)
Palliative Care Order (Current Admission)	0.03 (2718)	0.19 (2168)	0.33 (1408)	0.41 (577)
Time to Palliative Care Order Placement (Days)1	2.6 (0.9 to 6.6)	2.8 (1.0 to 6.0)	2.4 (0.8 to 6.0)	2.0 (0.7 to 4.8)
Discharge Disposition
Home/Self Care	0.79 (75667)	0.38 (4436)	0.18 (755)	0.09 (131)
Home Health Care	0.09 (8316)	0.14 (1572)	0.11 (466)	0.05 (76)
Skilled Nursing Facility	0.05 (4898)	0.20 (2327)	0.24 (1012)	0.17 (237)
Rehab Facility	0.03 (3244)	0.10 (1165)	0.08 (331)	0.04 (61)
Left Against Medical Advice	0.01 (1095)	0.01 (84)	0.00 (19)	0.00 (3)
Hospice (Home)	0.00 (291)	0.03 (330)	0.06 (239)	0.06 (81)
Hospice (Medical Facility)	0.00 (297)	0.02 (287)	0.05 (232)	0.07 (99)
Expired in Hospital	0.01 (776)	0.08 (956)	0.22 (953)	0.46 (656)
Other	0.02 (1635)	0.04 (483)	0.05 (229)	0.05 (72)

^1-Time to palliative care order placement was calculated for those patients with an order placed during their current encounter. Data presented as median and IQR time in days.

Figure 3:

Smoothed splines showing relationship between admission predicted risk of 180-day mortality and rate of palliative care consultation in validation cohort population. Rug plot above x-axis shows frequency of admission predicted risk values. Left panel shows rate of palliative care consultation during current admission encounter, right panel shows rate of palliative care consultation in 6-months prior to current admission encounter.

DISCUSSION:

In this large, single-center retrospective study, we developed and validated a predictive mortality risk tool for adult patients, irrespective of illness acuity or severity, upon hospital admission. Our 180-day mortality model demonstrated excellent discrimination and calibration, using routinely collected clinical and laboratory data within the first 24 hours of admission. Compared to other predictive mortality models (1,2,4–7,9,10,17,18,21), our algorithm’s strengths are based on: a) its large cohort size for derivation, b) readily available combination of clinical and laboratory data spanning different admission diagnoses, c) generalizability across medical conditions, and d) its ability to predict mortality regardless of location within six months of hospital discharge. In addition, our model showed consistent performance across racial and gender subgroups. Race and ethnicity were purposefully excluded from the model development to avoid the potential of unintended consequences that could propagate healthcare inequities. In developing an admission risk score for a patient’s mortality, our model provides an informational value that can help healthcare teams to decide, based on each patient’s unique clinical context, whether proactive initiation or re-engagement in conversations about personal goals and values is appropriate.

Furthermore, our data add to existing literature through a granular evaluation of practice patterns on exposure to PC in relation to mortality risk. PC has been well-recognized as a medical discipline that can maximize quality of life through alleviation of suffering and promotion of adaptive coping for those facing serious illnesses (22–25). Although, PCC is appropriate at any point in a patient’s illness trajectory, timing of referral may vary based on an individual’s evolving needs and clinician comfort level in caring for those who are seriously ill, thereby requiring a graded approach (26,27). To ensure the delivery of goal-concordant care, it is vital that clinical teams engage patients in conversations about their values, goals, and treatment preferences, and that these discussions occur in both an iterative and timely manner. However, application of PC principles in the era of machine learning, with regard to mortality predictions, requires careful consideration and tailored communication strategies.

Our data demonstrate that as predicted mortality increases, past and present exposure to PC simultaneously increases. Nearly 41% of patients in the highest risk category received an inpatient PCC, with most consults occurring within the first few days of hospitalization; in contrast, only 11% of these patients had been exposed to PC within six months prior to admission. When depicting this relationship on a continuous scale, we observe a unique inflection point in the proportion of patients who received a PCC as predicted mortality increases beyond 75%. This finding suggests that for the minority of patients with the highest admission risk score and the poorest clinical prognosis, primary teams may have greater certainty and confidence in transitioning to end-of-life care and may not require PCC assistance. In essence, our model adds less value for those patients with greater prognostic certainty (i.e., patients with either the lowest or highest risk scores), and instead, may be most helpful for patients who lie in the middle of the risk continuum.

Our model currently does not trigger or prescribe any specific course of care. It instead establishes a validated risk score predicting 180-day mortality, which we believe can alert clinicians for the need to engage patients in goals-of-care conversations, recognizing that such conversations do not require an automatic PCC. Additional testing is needed to determine our model’s acceptability in clinical practice. Differences in predicted risk may therefore necessitate different strategies for patient-centered interventions. For example, patients at higher risk may benefit from more urgent dialogue in the inpatient setting while those at lower risk may follow with their outpatient providers. Future studies should test implementation of the model in a randomized pragmatic fashion, with communication interventions designed to evaluate potential benefits and harms (such as increased emotional distress for patients or caregivers), to explore each patient’s illness understanding in the context of their hospital stay and medical history, and to overall determine if mortality algorithms can be helpful to guide resource allocation in a personalized manner.

Our study has several limitations. This study was conducted at a single academic center with a well-established PC program. Our findings may not be generalizable across all healthcare systems; PC resources may also vary per healthcare setting. Moreover, DNR presence on admission may have been influenced by patient decision-making in a hospital environment, the 24-hour window for variable consideration, and diverse ordering practices among admitting clinicians. In addition, reasons for PCC were not available for analysis, and therefore one cannot assume that the content of PCCs focused on goals-of-care. Regarding consistency and quality control in capturing mortality, there are inherent limitations in relying on hospice agencies or family members for notification of outpatient deaths. There were few but notable differences in our validation cohort (Table 1). The psychiatry admission service did not account for patients directly admitted to a separate psychiatric hospital; this was offset by psychiatry admission indicating a protective factor for mortality in our model (eTable 1). The higher event rate of 180-day mortality likely reflects the increased capture of outpatient mortality with a change in EHR systems; as a result, our model may underestimate the risk of mortality on average.

The optimal strategy for implementing mortality-predicting algorithms to facilitate clinical care remains unknown. Output risk scores from predicted mortality models should be interpreted and used cautiously in large healthcare systems. The application of these models should not be used with the sole intention of improving quality metrics, such as inpatient observed to expected mortality ratios (28), nor should it be used to automatically enact a change in system-wide practice patterns, such as triggering a shift in disposition planning or increasing hospice and PC referrals (19). Instead, machine learning has the potential to serve as a readily accessible and useful adjunct, or rather, a numerical result that still requires clinical judgement and interpretation. Therefore, the primary motive for mortality-predicting models should be to maximize the opportunity for patients and families to reflect on and make informed decisions about their end-of-life care in a given clinical context, concordant with their personal values and care preferences.

In summary, we developed and temporally validated a predictive mortality model for adult patients admitted at our healthcare institution. Our mortality risk model helps quantify the potential need for palliative care referrals based on risk strata. Machine learning algorithms for mortality require clinical interpretation, and additional studies are needed to design patient-centered and risk-specific interventions.

KEY MESSAGE.

This article describes the development and validation of a predictive mortality risk model using sixteen clinical and laboratory variables routinely available on adult inpatient admission. This model helps quantify the potential need for palliative care referrals based on risk strata.