The “Surprise Question” Asked of Emergency Physicians May Predict 12-Month Mortality among Older Emergency Department Patients

Abstract

Background: Identification of older adults with serious illness (life expectancy less than one year) who may benefit from serious illness conversations or other palliative care interventions in the emergency department (ED) is difficult.

Objectives: To assess the performance of the “surprise question (SQ)” asked of emergency physicians to predict 12-month mortality.

Design: We asked attending emergency physician “Would you be surprised whether this patient died in the next 12 months?” regarding patients ≥65 years old that they had cared for that shift. We prospectively obtained death records from Massachusetts Department of Health Vital Records.

Setting: An urban, university-affiliated ED.

Measurement: Twelve-month mortality.

Results: We approached 38 physicians to answer the SQ, and 86% participated. The mean age of our cohort was 76 years, 51% were male, and 45% had at least one serious illness. Out of 207 patients, the physicians stated that they “would not be surprised” if the patient died in the next 12 months for 102 of the patients (49%); 44 of the 207 patients (21%) died within 12 months. The SQ demonstrated sensitivity of 77%, specificity of 56%, positive predictive value of 32%, and negative predictive value of 90%. When combined with other predictors, the model sorted the patient who lived from the patient who died correctly 72% of the time (c-statistic = 0.72).

Conclusion: Use of the SQ by emergency physicians may predict 12-month mortality in older ED patients and may help emergency physicians identify older adults in need of palliative care interventions.

Introduction

The majority (75%) of older adults with life-limiting illnesses visit the emergency department (ED) during the last six months of life.¹ ED visits often mark an inflection point in these patients' illness trajectories, signaling a more rapid rate of decline.^2,3 Many of these patients have not formulated their goals for end-of-life care,⁴ and the majority (56–99%) do not have advance directives available at the time of ED presentation.⁵ Most of these patients have priorities other than simply living as long as possible,⁶ yet they may receive aggressive care that does not align with their goals.⁷ The ED provides a point in time and a location to identify patients who would benefit from formulating their goals for future medical care. Older adults with limited life expectancy (less than one year) are likely to benefit from a discussion about their goals of care.⁸ Validated tools to identify older adults with limited life expectancy, such as PREDICT criteria (palliative care status, nursing home resident, current diagnosis of cancer, intensive care unit [ICU] admission, ≥2 hospitalizations in the past year, and age >55 years—score ≥13 yields 95.3% [95% confidence interval (CI): 93–96%] specificity and 53.9% [95% CI: 48–60%] sensitivity for predicting 1-year mortality),⁹ are cumbersome to administer in the time-pressured ED environment. Current clinical practice is constrained by the lack of feasible and reliable approaches to identify patients with limited life expectancy most likely to benefit from palliative care consultations in the ED or immediately upon admission, and/or serious illness conversations that have been shown to improve quality of life¹⁰ and receipt of goal concordant care.¹¹

The “surprise question (SQ),” worded as “Would you be surprised if this patient died in the next 12 months?” relies only on the clinicians' overall clinical impression to predict 12-month mortality among patients with life-limiting illness. The SQ has shown sensitivity of 21–84% and specificity of 51–94% in prior studies^12–19 and may be uniquely fitting to predict mortality in the time-pressured ED setting. However, the SQ has never been studied to predict 12-month mortality in an ED population.

We aimed to test the accuracy of the SQ as a predictor of 12-month mortality among older ED patients. Furthermore, we aimed to examine whether adding the PREDICT,⁹ a validated 12-month mortality prediction rule for older adults hospitalized from the ED, improves the performance of the SQ.

Methods

Study design

We conducted a prospective cohort study at an academic, urban ED with an annual volume of 60,000 visits, 24% of whom are ≥65 years of age. The study protocol was approved by the appropriate institutional review board of our hospital.

Participants and procedures

We included patients during three consecutive weeks in October 2015 if they received care in the ED and were ≥65 years of age. One hour before the end of each clinical shift, research assistants (RAs) asked the attending emergency physicians, “Would you be surprised if your patient died in the next 12 months?” on patients ≥65 years who they had cared for. Enrollment took place at the conclusion of the day, evening, and overnight shifts and included weekdays and weekends. We excluded shifts wherein physicians were unwilling to participate, as well as shifts when the RAs were physically unavailable. Another trained RA identified components of the PREDICT criteria (palliative care status, nursing home resident, current cancer diagnosis, ICU admission, ≥2 hospitalizations in the past year, and age >55 years)⁹ in our medical record using a standardized chart abstraction method.²⁰ This RA was blinded to the data collected in the ED. To determine the quality of chart abstraction, the principal investigator (PI) independently reabstracted data from 10% of patient medical records. We obtained the 12-month death records from the Massachusetts Registry of Vital Records and Statistics.

Variables

The primary outcome was the accuracy of the SQ in predicting 12-month mortality. We included two additional variables in our multivariable regression model: PREDICT criteria and emergency physicians' number of years after residency training. We hypothesized that adding PREDICT (a validated predictor) may improve the diagnostic accuracy of the SQ. We also hypothesized that emergency physicians with more clinical experience (number of years after residency training) would perform better at predicting mortality using the SQ.

Analyses

Patients were divided into two groups per the SQ classification. We used binary logistic regression to assess the association between 12-month mortality and our independent variables. We performed univariate analysis for the SQ, PREDICT score, and number of years after residency training, respectively, and multivariable analysis including all three variables. We determined the sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV) using Stata v.14.0 (StataCorp, College Station, TX) with p < 0.05 for statistical significance. To evaluate the inter-rater agreement of the chart abstraction, we used proportion of agreement rather than kappa coefficient because of high concordance, resulting in paradoxical kappa from unbalanced marginal totals.²¹ Similar to prior studies,²² we assessed model discrimination (the ability of the model to correctly identify those who died from those who survived) by calculating a c-statistic.

Results

During 69 eligible ED shifts in our study period, 38 physicians provided care to ≥1 patient ≥65 years of age. The physicians answered the SQ during 55 shifts (80%), and physicians declined to participate during 14 shifts (20%). Fifteen shifts were ineligible due to RA unavailability. Overall, 33 physicians (86%) participated. All emergency physicians were board eligible or certified. On average, physicians had been in practice for 9.2 years and 67% were male. These 33 physicians screened 207 patients with mean patient age of 75 years (SD 7.5); 51% were male, and 44% had at least one life-limiting illness (Table 1). There was 90% agreement in chart-abstracted data between the RA and PI.

Table 1.

Patient Characteristics Grouped by Physician Response to the “Surprise Question” (N = 207)

	“Yes, would be surprised” (n = 105)	“No, would not be surprised” (n = 102)	p
Age in years [mean (SD)]	75 (7.4)	76 (7.7)	0.233
Female (%)	48.1	49.2	0.233
PREDICT score [mean (SD)]	7.7 (6)	7.5 (7.0)	0.784
Dementia	12	26	0.016
Cancer	11	29	0.002
ESRD	4	6	0.548
COPD	1	9	0.011
CHF	6	18	0.011
ESLD	2	2	0.977
Septic shock	1	4	0.185
AIDS	1	0	0.309
Hip fracture	0	1	0.323
Trauma	0	1	0.323
Palliative care status	3	7	0.211
Nursing home status	24	21	0.538
ICU admission	5	5	0.963
≥2 Hospital admissions in past 12 months	59	46	0.043

AIDS, acquired immunodeficiency syndrome; CHF, congestive heart failure; COPD, chronic obstructive pulmonary disease; ESLD, end-stage liver disease; ESRD, end-stage renal disease; ICU, intensive care unit.

Out of 207 patients, the physicians stated that they “would not be surprised” if the patient died in the next 12 months for 102 of the patients (49%); 44 of the 207 patients (21%) died within 12 months (10 in “would be surprised” and 34 in “would not be surprised” groups). The odds of death at 12 months were four times higher in patients for whom emergency physicians answered, “No, I would not be surprised” compared with patients for whom emergency physicians answered, “Yes, I would be surprised” (odds ratio [OR] = 4.4, 95% CI: 2.1–9.5, p < 0.001).

Discrimination and test characteristics of the SQ, PREDICT criteria, and physician experience on univariate and multivariable analyses are given in Table 2. When tested alone, the SQ had the following predictive performance: sensitivity 77%, specificity 56%, PPV 32%, and NPV 90% with c-statistic of 0.67. In the multivariable model including the SQ, PREDICT criteria, and physician experience, the c-statistic improved to 0.72, whereas sensitivity (77%), specificity (56%), PPV (32%), and NPV (90%) remained unchanged (Table 2).

Table 2.

Evaluation of the “Surprise Question” and the PREDICT Score as Predictors for 12-Month Mortality

	Univariate analysis
Predictors or confounders	Odds ratio	95% Confidence intervals
PREDICT criteria	1.0	0.9–1.0
Physician experience	1.0	1.0–1.1
“Surprise question”
“Yes, I would be surprised.”	Reference
“No, I would not be surprised.”	4.4	2.0–9.5
C-statistic	0.67

	Multivariable analysisa
“Surprise question”
“Yes, I would be surprised.”	Reference
“No, I would not be surprised.”	4.0	1.8–8.7
C-statistic	0.72

^aMultivariable analysis includes the PREDICT score, physician experience, and “surprise question.”

Discussion

Emergency physicians asserted that they would “not be surprised” if nearly half of their older ED patients died in the next 12 months. When categorized by emergency physicians to be “not surprised,” patients had nearly fourfold increase in odds of death. Combining the SQ (physicians' overall clinical impression), PREDICT criteria (validated prognostic criteria for older adults hospitalized from the ED, obtained from chart abstraction), and physician experience, our multivariable model sorted patients who died from patients who lived correctly 72% of the time (c-statistic = 0.72). Our findings suggest that the SQ, when combined with other predictors for mortality that are easily obtainable from electronic health records (EHR), may be a valuable tool for identifying older adults who would benefit from serious illness conversations and potentially other palliative care services in or after the ED. Since manual assessment of PREDICT criteria is cumbersome, integrating PREDICT as an automatic EHR trigger and relying physicians to only answer the SQ may improve the feasibility of implementation and predictive performance in the ED setting. If the SQ is implemented in the ED, it will likely not change the emergency physicians' practice significantly²³; rather, it might increase the demand of palliative care services after the ED visit.

Our study expands on the available literature on the association between the SQ and 12-month mortality in other patient populations, and shows that, in the ED setting, the SQ has comparable sensitivity and specificity to the SQ in previously studied clinical settings (e.g., outpatient oncology and outpatient nephrology clinics). Similar to previous studies,^{12,14,17–19} we found a low PPV and a high NPV. The magnitude of association was lower in our study (OR = 4.4) than in some of the prior studies (OR = 3–11).^15–17 This finding could be explained by the fact that emergency physicians' perception of prognosis may be influenced by the high acuity and acute needs for medical care of patients presenting to the ED.^24,25 As such, our patients were older (75 vs. 60–66 years^{12,14,17–19}) and had higher 12-month mortality (21% vs. 6–12%^{12,14,17–19}) than those in prior studies. Emergency physicians' perception of acuity may be blunted (downward bias on the magnitude of association), yet the prevalence of 12-month mortality was higher (improves the predictive values). These unique clinical situations that emergency physicians face likely contributed to our findings.

Our study has several limitations. We collected data on consecutive shifts, yet physicians refused to participate in 14 out of 69 shifts. The response rate was adequate (86%, 33 out of 38 eligible physicians), but may not reflect what the use of this tool would be in other settings. Shifts where physicians refused to participate or RA was unavailable may have included patients with high mortality, and the direction of bias is unpredictable. Our sampling strategy, small sample size, and limited death records to those reported in Massachusetts likely produced nondifferential classification bias and limited our power to detect the true difference. Our prevalence of 12-month mortality may be higher than that of average EDs due to higher hospital acuity; however, the test characteristic of the SQ should not change in other institutions. We were only able to ask our physicians to make a snapshot judgment of their patients. Sampling bias could have occurred if the SQ was asked before clinical deterioration of the patients (e.g., physicians stated that they “would be surprised” if patient died when he/she had not seen that the patient became hypotensive and had an elevated serum lactate later in his/her ED stay.). We were unable to control for potential confounding by Emergency Severity Index, chief complaint, and length of stay due to small sample size and large number of chief complaints. Some of this information is likely incorporated into board-certified emergency physicians' overall clinical assessment of their patients. We did not include other clinicians (e.g., nurses) to better understand how their overall assessment of the patients may defer from the emergency physicians' assessment. Further study may be warranted to investigate whether other members of the clinical team in the ED can reliably answer the SQ to improve the scalability of implementation. Lastly, we followed recommended practices for chart abstraction,²⁰ yet there was a 10% disagreement between the abstracters.

Conclusion

The SQ asked of emergency physicians may be valuable to predict 12-month mortality in older ED patients, enhancing access to appropriate conversations and palliative care services for this population. Further research is warranted to improve the predictive accuracy of the SQ in the ED setting by exploring other predictors.

Author Disclosure Statement

No competing financial interests exist.