TO THE EDITOR
Palliative care improves the value of care for patients with serious illnesses, but resource constraints necessitate targeting palliative care interventions to patients who need them most.1 The “Surprise Question” (SQ) – “Would you be surprised if this patient died in the next 12 months?” – has emerged as an attractive, simple solution for identifying patients who might benefit from palliative care.2,3 Despite optimism about the potential of the SQ to identify primary care patients who would benefit from palliative care,4 there is no evidence on its performance in this setting.
METHODS
Study Cohort
We identified patients screened for high-risk care management program at a large academic primary care group practice, for whom physicians (PCPs) answered the SQ in 2013. We assumed a “No” answer represented physician prediction of high one-year mortality risk.
Primary outcome was mortality one year after SQ response, ascertained by linkage to Social Security Administration data. Demographics and comorbidities were drawn from electronic health records. We assessed SQ performance for predicting one-year mortality using area under the receiver-operating curve (AUC), sensitivity, and positive predictive value; we also calculated the odds ratio (OR) of a “No” response for one-year mortality using logistic regression.
To determine the incremental benefit of the SQ for predicting one-year mortality over and above routinely-collected administrative data, we calculated the integrated discrimination improvement (IDI)5 of adding SQ response to a multivariate logistic regression model of mortality on age, sex, and comorbidity score.6 IDI measures change in sensitivity and specificity between two models.
RESULTS
Patients in the study were predominantly female (60.3%). Mean age was 65, and 43.2% had 3 or more comorbidities. High-risk patients (SQ answer of “No”) had 4.36 times higher odds of dying than low risk patients (SQ answer: “Yes”; 95% CI 2.63–7.22, p<0.001). Table 1 shows performance of the SQ as a screening test for 1-year mortality. Sensitivity of the SQ was 20.5% and specificity 94.4%, giving positive and negative likelihood ratios of 3.66 and 0.84, respectively. Given the one-year mortality rate of 6.6% in this population, positive and negative predictive values were 20.2% and 94.5%, respectively. Area under the curve (AUC) was 0.57.
TABLE 1.
Tabulation of Surprise Question Response and Patient Vital Status After One Year
| Vital Status at one year from SQ | Total (%) | |||
|---|---|---|---|---|
| Deceased | Alive | |||
| SQ Response | No | 23 (Death predicted, accurate) | 91 (Death predicted, inaccurate) | 114 (6.6%) |
| Yes | 89 (Alive predicted, inaccurate) | 1,534 (Alive predicted, accurate) | 1,623 (93.4%) | |
| Total (%) | 112 (6.4%) | 1,625 (93.6%) | 1,737 | |
Abbreviations: Surprise Question (SQ)
In multivariate analysis, a PCP prediction of high risk via the SQ remained strongly associated with one-year mortality (OR: 2.52, 95% CI 1.46–4.34), over and above age, sex, or comorbidity score (Table 2). Predictive performance of the logistic regression model, however, was not significantly affected by SQ response: IDI was 0.88% (95% CI −0.14%–1.9%).
TABLE 2.
Multivariate Logistic Regression With an Outcome of Vital Status After One Year
| Odds Ratio [95% Confidence Interval] | p-value | |
|---|---|---|
| Age | 1.05 [1.03,1.06] | <0.001 |
| Male Sex | 1.61 [1.08, 2.42] | 0.02 |
| Gagne Comorbidity Score | 1.23 [1.15, 1.32] | <0.001 |
| Surprise Question Response = “No” | 2.52 [1.46, 4.34] | 0.001 |
DISCUSSION
We found that PCP prediction of high mortality risk via the SQ failed to identify the majority of deaths, making it is a poor screening tool for one-year mortality in a heterogeneous primary care population. Adding the SQ response to a validated one-year mortality prediction model6 did not improve the discriminative ability of that model. These findings are mostly consistent with prior studies examining the SQ in renal disease and cancer.2,3 {Moss, 2008 #1237; Moss, 2010 #1240; Moss, 2008 #1237; Moroni, 2014 #1256}Importantly, even if mortality were wholly predictable, poor near-term prognosis is only one of many triggers for palliative care initiation. Neither prognosis nor the SQ explicitly account for symptoms and other burdens of serious illness that can indicate a need for palliative care.
While these results suggest caution in using the SQ in isolation to identify poor prognosis patients in the primary care setting, the SQ did contain large amounts of signal for predicting mortality: it was strongly and significantly associated with one-year mortality, and this effect was noted over and above known predictors such as age and comorbidities. Understanding this signal, and incorporating it into more advanced predictive algorithms, could be useful topics for future research.
Acknowledgments
Funding/Support: The development and implementation of the high-risk primary care population management program at Brigham and Women’s hospital is funded by the Brigham and Women’s Physicians Organization and Brigham and Women’s Hospital. Dr. Obermeyer is supported by NIH grant: DP5 OD012161. Part of the analysis was supported by a grant from the National Institute for Health Care Management to Dr Obermeyer and Mr Powers.
Footnotes
Conflict of Interest Disclosures: None of the authors have any conflicts of interest to disclose.
Role of the Funder/Sponsor: No funder had any role in design and conduct of the study; collection, management, analysis, and interpretation of the data; or preparation, review, or approval of the manuscript.
Previous Presentation(s): None
Author Contributions: Drs. Lakin and Obermeyer had full access to all the data in the study and take responsibility for the integrity of the data and the accuracy of the data analysis.
Study concept and design: Drs. Lakin, Obermeyer, Bernacki, Cunningham, and Block
Acquisition, analysis, or interpretation of data: Drs. Obermeyer, Bernacki, and Lakin, Ms. Robinson, Mr. Powers
Drafting of the manuscript: All authors
Critical revision of the manuscript for important intellectual content: All authors
Statistical analysis: Dr. Obermeyer, Ms. Robinson, Mr. Powers
Obtained funding: Dr. Obermeyer
Administrative, technical, or material support: Ms. Robinson
Study supervision: Drs. Obermeyer and Bernacki
References
- 1.Institute of Medicine. Dying in America: Improving Quality and Honoring Individual Preferences Near the End of Life. Washington, DC: National Academies Press; 2014. [PubMed] [Google Scholar]
- 2.Moroni M, Zocchi D, Bolognesi D, et al. The ‘surprise’ question in advanced cancer patients: A prospective study among general practitioners. Palliative medicine. 2014;28(7):959–964. doi: 10.1177/0269216314526273. [DOI] [PubMed] [Google Scholar]
- 3.Moss AH, Ganjoo J, Sharma S, et al. Utility of the “surprise” question to identify dialysis patients with high mortality. Clin J Am Soc Nephrol. 2008;3(5):1379–1384. doi: 10.2215/CJN.00940208. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 4.Pattison M, Romer AL. Improving Care Through the End of Life: launching a primary care clinic-based program. Journal of palliative medicine. 2001;4(2):249–254. doi: 10.1089/109662101750290335. [DOI] [PubMed] [Google Scholar]
- 5.Pencina MJ, D’Agostino RB, Pencina KM, Janssens AC, Greenland P. Interpreting incremental value of markers added to risk prediction models. Am J Epidemiol. 2012;176(6):473–481. doi: 10.1093/aje/kws207. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 6.Gagne JJ, Glynn RJ, Avorn J, Levin R, Schneeweiss S. A combined comorbidity score predicted mortality in elderly patients better than existing scores. J Clin Epidemiol. 2011;64(7):749–759. doi: 10.1016/j.jclinepi.2010.10.004. [DOI] [PMC free article] [PubMed] [Google Scholar]