The “surprise question” for predicting death in seriously ill patients: a systematic review and meta-analysis

Abstract

BACKGROUND:

The surprise question — “Would I be surprised if this patient died in the next 12 months?” — has been used to identify patients at high risk of death who might benefit from palliative care services. Our objective was to systematically review the performance characteristics of the surprise question in predicting death.

METHODS:

We searched multiple electronic databases from inception to 2016 to identify studies that prospectively screened patients with the surprise question and reported on death at 6 to 18 months. We constructed models of hierarchical summary receiver operating characteristics (sROCs) to determine prognostic performance.

RESULTS:

Sixteen studies (17 cohorts, 11 621 patients) met the selection criteria. For the outcome of death at 6 to 18 months, the pooled prognostic characteristics were sensitivity 67.0% (95% confidence interval [CI] 55.7%–76.7%), specificity 80.2% (73.3%–85.6%), positive likelihood ratio 3.4 (95% CI 2.8–4.1), negative likelihood ratio 0.41 (95% CI 0.32–0.54), positive predictive value 37.1% (95% CI 30.2%–44.6%) and negative predictive value 93.1% (95% CI 91.0%–94.8%). The surprise question had worse discrimination in patients with noncancer illness (area under sROC curve 0.77 [95% CI 0.73–0.81]) than in patients with cancer (area under sROC curve 0.83 [95% CI 0.79–0.87; p = 0.02 for difference]). Most studies had a moderate to high risk of bias, often because they had a low or unknown participation rate or had missing data.

INTERPRETATION:

The surprise question performs poorly to modestly as a predictive tool for death, with worse performance in noncancer illness. Further studies are needed to develop accurate tools to identify patients with palliative care needs and to assess the surprise question for this purpose.

The surprise question (SQ) was developed more than a decade ago and has been suggested as a simple test to identify patients who might benefit from hospice and palliative care (HPC).¹ It involves a clinician reflecting on the question, “Would I be surprised if this patient died in the next 12 months?”. It was thought that the SQ would correct for a physician’s tendency to overestimate prognosis² by asking the physician to consider whether death in the coming year is possible rather than probable. The surprise question has been widely promoted^3,4 and adopted into frameworks for assessing hospice and palliative care needs.^5,6

In the past few years, several studies have reported on the accuracy of the SQ for a different purpose: as a prognostic test of intermediate-term death in different patient populations. These studies sought to determine whether an answer of “no” (hereafter SQ+) predicts intermediate-term death. We conducted a systematic review of the literature to determine the performance characteristics of the SQ in predicting death and the methodologic characteristics of these studies.

Methods

Search strategy

We searched MEDLINE (from 1946 to week 2 of October 2016), MEDLINE in process (to Oct. 19, 2016), Embase (1947 to Oct. 19, 2016), Cochrane Central Register of Controlled Trials (to September 2016), Cochrane Database of Systematic Reviews (from 2005 to Oct. 19, 2016), PsycINFO (from 1806 to week 2 of October 2016), Cumulative Index to Nursing and Allied Health Literature (CINAHL; from 1961 to Oct. 20, 2016), Web of Science (Oct. 19, 2016), SCOPUS, PubMed and Google Scholar. Details of the search strategy are available in Appendix 1 (available at www.cmaj.ca/lookup/suppl/doi:10.1503/cmaj.160775/-/DC1).

Study selection

Two authors (J.D. and R.G.) independently screened citations to identify potentially relevant articles or abstracts. Potentially relevant citations were retrieved and reviewed independently in full text by the same 2 authors to determine whether they met inclusion criteria: a prospective cohort study with SQ asked of study participants, with prospective follow-up for the primary outcome (death at least 6 months after SQ asked). We did not consider designs where the SQ was asked retrospectively because of the potential for biased ratings based on knowledge of patient outcome. Reviewers were not masked to the study author, institution or journal. Because we did not wish to limit calculations to sensitivity, we excluded studies that provided data for outcomes only for patients who were SQ+, but we attempted to contact authors of these studies to determine whether outcomes were available for patients identified as SQ– (yes answer to SQ).

Data abstraction and methodologic quality

Two authors (J.D. and N.A.) abstracted in duplicate the following data from each study: setting; population; proportion of eligible patients enrolled; proportion of eligible patients with outcomes data; number of SQ evaluators for each patient and measure of agreement, if any; and outcomes (true positives, false positives, true negatives and false negatives). A true positive was defined as an answer of no to SQ for a patient who died. Two reviewers (J.D. and N.A.) also assessed the risk of bias of each study using the Quality in Prognosis Studies tool.^7,8 Disagreements between reviewers at any stage of the review were resolved by consensus.

Statistical analysis

For each study, we constructed a 2 × 2 table (predictor, SQ; outcome, death) and calculated the incidence of death in the study, sensitivity, specificity, positive likelihood ratio (LR+), negative likelihood ratio (LR−), positive (PPV) and negative (NPV) predictive values, and diagnostic odds ratio (DOR). The DOR is the ratio of odds of a prediction of death (SQ+) in a patient who dies divided by the odds in a patient who lives, calculated as LR+ divided by LR−. In studies reporting more than 1 clinician rating the SQ we used the study’s definition of SQ+, which could require consensus or at least 1 rater.

We constructed a summary receiver operating characteristic (sROC) curve using a Rutter–Gatsonis hierarchical sROC model.⁹ The ROC curve plots the true positive rate (sensitivity) against the false-positive rate (1 – specificity), and the area under the ROC curve (AUC) is a measure of discrimination, with 1 denoting perfect discrimination and 0.5 denoting no better than chance. Discrimination is the probability, given a randomly selected pair of patients, of whom one lives and the other dies, that the survivor will be coded as SQ− and the decedent coded as SQ+. The hierarchical model was appropriate given that the scale parameter β, which provides for asymmetry in the sROC model, was not significantly different from zero, and the distribution of the random errors was normal on visual inspection.

Based on the hierarchical sROC model, we determined the meta-analytic summaries of DOR, LR+ and LR−, sensitivity and specificity. Using the DOR, we derived the AUC and its SE based on the formula described by Walter.¹⁰ We also presented summary PPV and NPV for illustration, and acknowledged the following limitations: heterogeneity may be greater than for test characteristics of sensitivity and specificity, and average predictive values relate to test utilization at some average, but unknown, incidence of death.¹¹

For the predefined subgroup analyses of patients with cancer and those with noncancer illness, there were fewer studies; therefore, we used univariable random effects models, with each study weighted by the inverse of its variance for the given parameter. Parameters for these subgroups were compared using the z test. We report heterogeneity of summary estimates of diagnostic performance using the I² measure, which is the percentage of total variability across studies that is attributable to heterogeneity rather than chance, and used published guidelines for low (I² 25%–49%), moderate (I² 50%–74%), and high (I² ≥ 75%) heterogeneity.¹² To calculate I², we used univariable models for all studies together and for the subgroups with and without cancer. Heterogeneity measures were not available for AUC estimates, because they were derived from the DOR.

We assessed for publication bias using the regression test of asymmetry described by Deeks and colleagues,¹³ which is based on a plot of the inverse of the square root of the sample size versus the log of the DOR. The analysis was performed in SAS version 9.3 (SAS Institute) using the MetaDAS macro for the hierarchical sROC model and R version 3.2.0 for the univariable meta-analysis. The sROC plot was generated using RevMan version 5.3 (Cochrane Community). Ethics approval was not required for this study.

Results

We identified 792 citations, of which 26^1,14–38 were potentially relevant (Figure 1). Of these studies, 7 did not include outcomes data for patients who were SQ–^14–19,36 that were confirmed in the publication^15–18,36 or after contact with study authors.^14,19 In 3 studies, the SQ was used only to predict 7- and/or 30-day mortality.^29,37,38 Sixteen remaining studies of SQ to predict death met inclusion criteria^{1,20–28,30–35} (Table 1) and were included in the meta-analysis. One study provided data for a derivation cohort in the manuscript.²⁰ We also obtained data from a validation cohort from the study authors, which we analyzed separately in the numerical description and meta-analyses but considered with the derivation cohort as a single study otherwise.

Figure 1:

Study selection for the systematic review. SQ = surprise question, SQ− = response to SQ is yes.

Table 1:

Characteristics of studies included in the systematic review

Study, year	Centre	Diagnosis/procedure/practice	No. of eligible patients	Proportion of eligible patients enrolled, %	Proportion of enrolled patients with SQ response, %	Proportion of patients with “No” as SQ response, %	Incidence of death, % (95% CI)	No. of evaluators per patient; total no. of evaluators per study	Follow-up, mo
Barnes et al.,21 2008	16 general practice clinics	Congestive heart failure	1555	231/1555 (15)*	212/231 (92)	76/212 (36)	6.1 (3.4–10.0)	1 general practitioner; unclear	12
Moss et al.,1 2008	3 dialysis units looked after by 1 nephrology group	Hemodialysis	150 (consecutive)	147/150 (98)	147/147 (100)	34/147 (23)	15.0 (9.6–21.8)	1 nurse practitioner; 3	12
Cohen et al.,20 2010	5 dialysis units looked after by 1 nephrology group	Hemodialysis	1026 (consecutive)†	1026/1026 (100)	874/1026 (85)	127/874 (15)	Derivation, 6.0 (4.0–8.7); validation, 8.4 (6.0–11.5)	1 nephrologist; unclear	6
Moss et al.,26 2010	1 cancer centre	Cancer	853 (consecutive outpatients)	853/853 (100)	826/853 (97)	130/826 (16)	8.6 (6.8–10.7)	1 oncologist; 4	12
Da Silva Gane et al.,30 2013	3 dialysis units	Hemodialysis	344 (prevalent)	344/344 (100)	344/344 (100)	220/344 (64)	15.1 (11.5–19.3)	1 nephrologist; 6‡	12
Pang et al.,22 2013	1 dialysis unit	Peritoneal dialysis	367 (prevalent)	367/367 (100)	367/367 (100)	109/367 (30)	12.0 (8.8–15.8)	3 nephrologists;§ 3	12
Reilly et al.,23 2013	1 inpatient ward	Respiratory disease	Unclear	85 (randomly selected); unclear denominator	85/85 (100)	67/85 (79)	32.9 (23.1–44.0)	Respirologists (n unclear); unclear	12
Khan et al.,24 2014	1 medical–surgical ICU	Critically ill (mixed)	500 (consecutive)	500/500 (100)	500/500 (100)	238/500 (48)	36.0 (31.8–40.4)	Intensivists (n unclear); unclear	6
Moroni et al.,25 2014	42 general practitioners (unclear number of clinics)	Cancer	Unclear	231 (enrolled by participating physicians)¶	231/231 (100)	126/231 (55)	45.0 (38.5–51.7)	1 general practitioner; 42	12
Vick et al.,28 2015	Single centre	Cancer	4779	“All patients seen” by participating clinicians; unclear no. of nonparticipating physicians	4779/4779 (100)	732/4779 (15)	10.0 (9.2–10.9)	1 oncologist or nurse practitioner; 76	12
Gerlach et al.,33 2016	Single centre	Cancer	828 (consecutive outpatients)	828/828 (100)	828/828 (100)	146/828 (18)	17.0 (14.5–19.8)	1 oncologist or palliative care physician; 13	18
Amro et al.,31 2016	2 dialysis units looked after by 1 nephrology group	Hemodialysis	201 (prevalent)	201/201 (100)	201/201 (100)	50/201 (25)	19.4 (14.2–25.6)	1 nephrologist; 9	12
Lefkowits et al.,35 2016	Single centre	Cancer	263 (prevalent, receiving chemotherapy or radiotherapy)	263/263 (100)	263/263 (100)	87/263 (33)	20.5 (15.8–25.9)	1 gynecology–oncology physician; 7 physicians††	12
Carmen et al.,32 2016	1 dialysis unit	Hemodialysis	49 receiving hemodialysis for more than 3 mo in 2014; unclear if incident or prevalent	49/49 (100)	49/49 (100)	20/49 (41)	18.4 (8.8–32.0)	“Medical staff”; unclear	12
Lakin et al.,34 2016	Single centre	Primary care practice	Unclear	Patients screened for “high-risk care management program” over 18-month period, unclear number of eligible patients	1737/1737 (100)	114/1737 (7)	6.4 (5.3–7.7)	1 primary care physician; unclear	12

Note: CI = confidence interval, ICU = intensive care unit, SQ = surprise question, SQ+ = response to SQ is no, SQ− = response to SQ is yes.

^*Interest in participation expressed by 748 patients; 587 of these were asked for demographic information and 542 returned the survey they were sent; 11/16 general practice clinics agreed to participate, for a total of 231 patients.

^†Data from the derivation cohort (n = 512) were published. We obtained data for the validation cohort (n = 514) from the authors.

^‡SQ+ (“I would not be surprised if this patient died in the next 12 mo”) was defined as a positive response from one of the nephrologist evaluators.

^§SQ+ (“I would not be surprised if this patient died in the next 12 mo”) was defined as a positive response from any of the 3 evaluators.

^¶Eight of 50 (16.0%) eligible general practitioners refused to enrol their patients.

^**SQ+ and SQ− for each patient were defined by consensus.

^††Data for physician responses are reported here for consistency with other studies.

The 16 studies (17 cohorts) enrolled a total of 11 621 patients (mean, 683 patients per cohort, range 49–4779; Table 1). All were prospective studies enrolling patients in 1 centre or group^{1,20,22–24,26–28,30–35} or several clinics.^21,25 Five included only patients with cancer, ^{25,26,28,33,35} 7 included patients with renal failure,^{1,20,22,27,30–32} 2 included patients with end-stage heart²¹ or lung²³ disease, 1 included a heterogeneous population of patients with critical illness²⁴ and 1 involved a primary care practice.³⁴ The SQ was answered by physicians in 13 studies (including oncologists,^26,33,35 nephrologists,^{20,22,30–32} respirologists,²³ intensivists,²⁴ and general practitioners^21,25,34), specialist nurses in 1 study,¹ either a physician or a nurse in 1 study²⁸ and a multidisciplinary team of 2 physicians and a nurse in another study.²⁷ In each case, the assessors were familiar with the patient. Thirteen studies asked a 12-month SQ,^{1,21–23,25–28,30–32,34,35} 2 asked a 6-month SQ^20,24 and 1 asked an 18-month SQ.³³

In 13 studies reporting the number of patients enrolled and the eligible population,^{1,20–22,24,26,27,30–35} the median participation rate was 100% (interquartile range [IQR] 98.5–100). Three studies reported incomplete availability of the SQ result for enrolled patients (85%,²⁰ 92%²¹ and 97%¹). Of note, 1 study reported that the 12% of eligible patients receiving hemodialysis without a recorded SQ response (because the physician did not answer the question) had higher mortality than those with a response.²⁰ Among 2 studies reporting provider participation, 1 reported that 8 of 50 (16%) general practitioners declined to participate,²⁵ and another found that 5 of 16 (31%) general practitioner clinics declined.²¹ No study reported incomplete availability of status of death for patients with an SQ response.

In most studies, only 1 clinician answered the SQ for each patient.^{1,20,21,23–26,31,33,35} One study asked the SQ to a multidisciplinary team, which arrived at an answer by consensus.²⁷ In 1 study of patients receiving hemodialysis, every physician and nurse working with that patient was asked the SQ independently; to be consistent, we considered that the patient was SQ+ if either of the 2 physicians attending in their unit responded “no” to the SQ.³⁰ One study of patients receiving peritoneal dialysis²² asked the SQ to 3 nephrologists for each patient and reported slight to fair³⁹ agreement (κ 0.34–0.41).

We assessed the methodologic quality of the included studies using the 6 domains of the Quality in Prognosis Studies tool⁷ (Table 2) and assigned an overall risk of bias for each study according to the highest risk score in any domain. The rationale behind our judgment of risk is provided in Appendix 1. Only 2 studies were thought to have a low risk of bias in all domains; 10 had a moderate risk of bias in at least 1 domain, and 4 had a high risk of bias in at least 1 domain. The test of funnel plot asymmetry was not significant, suggesting no evidence of publication bias (p = 0.3).

Table 2:

Risk of bias for studies included in the systematic review

Study, year	Domain 1: Study participation	Domain 2: Study attrition	Domain 3: Prognostic factor measurement	Domain 4: Outcome measurement	Domain 5: Study confounding	Domain 6: Statistical analysis and reporting	Overall risk of bias
Barnes et al.,21 2008	High	Low	Low	Low	Moderate	Low	High
Moss et al.,1 2008	Low	Low	Low	Low	Moderate	Low	Moderate
Cohen et al.,20 2010	High	Low	Low	Low	Moderate	Low	High
Moss et al.,26 2010	Moderate	Low	Low	Low	Low	Low	Moderate
Da Silva Gane et al.,30 2013	Low	Low	Low	Low	Moderate	Low	Moderate
Pang et al.,22 2013	Low	Moderate	Low	Low	Moderate	Low	Moderate
Reilly et al.,23 2013	Moderate	Low	Low	Low	Moderate	Low	Moderate
Khan et al.,24 2014	Low	Low	Low	Low	Moderate	Low	Moderate
Moroni et al.,25 2014	Moderate	Low	Low	Low	Low	Low	Moderate
Feyi et al.,27 2015	Low	Low	Low	Moderate	High	Low	High
Vick et al.,28 2015	Moderate	Moderate	Low	Low	Low	Low	Moderate
Gerlach et al.,33 2016	Low	Low	Low	Low	Low	Low	Low
Amro et al.,31 2016	Low	Low	Low	Low	Moderate	Low	Moderate
Lefkowits et al.,35 2016	Low	Low	Low	Low	Low	Low	Low
Carmen et al.,32 2016	Low	Low	Low	Low	Moderate	Low	Moderate
Lakin et al.,34 2016	High	Low	Low	Low	Low	Low	High

Note: Appendix 1 (available at www.cmaj.ca/lookup/suppl/doi:10.1503.cmaj.160775/-/DC1) provides the details of how judgments were made for the individual domains.

The median incidence of death for the 17 cohorts was 15.1% (IQR 8.6%–20.5%). Prognostic properties of each included study are listed in Table 3. We contacted 6 authors to clarify or obtain additional data.^{21,26,30,31,33,35} Using data from all 17 cohorts (Table 4), the meta-analytic estimates were as follows: of sensitivity 67.0% (95% confidence interval [CI] 55.7%–76.7%), specificity 80.2% (73.3%–85.6%), LR+ 3.4 (95% CI 2.8–4.1), LR– 0.41 (95% CI 0.32–0.54), PPV 37.1% (95% CI 30.2%–44.6%) and NPV 93.1% (95% CI 91.0%–94.8%). The sROC model showed moderate discriminatory ability (Figure 2), with an AUC of 0.81 (95% CI 0.78–0.86).

Table 3:

Prognostic properties of studies included in the systematic review

Study, year	Sensitivity, % (95% CI)	Specificity, % (95% CI)	LR+ (95% CI)	LR− (95% CI)	PPV, % (95% CI)	NPV, % (95% CI)	DOR (95% CI)
Barnes et al.,21 2008	78.6 (49.2–95.3)	67.2 (60.2–73.7)	2.4 (1.7–3.4)	0.32 (0.12–0.87)	14.5 (7.5–24.4)	97.8 (93.7–99.5)	7.50 (2.50–22.54)
Moss et al.,1 2008	45.5 (24.4–67.8)	80.8 (72.8–87.3)	2.4 (1.3–4.2)	0.68 (0.46–1.00)	29.4 (15.1–47.5)	89.4 (82.2–94.4)	3.51 (1.58–7.78)
Cohen et al.,20 2010 (derivation cohort)	63.0 (42.4–80.6)	87.4 (83.8–90.4)	5.0 (3.4–7.3)	0.42 (0.26–0.69)	24.3 (14.8–36.0)	97.3 (95.2–98.7)	11.77 (5.85–23.67)
Cohen et al.,20 2010 (validation cohort)	47.2 (30.4–64.5)	89.8 (86.3–92.6)	4.6 (2.9–7.3)	0.59 (0.43–0.8)	29.8 (18.4–43.4)	94.9 (92.1–96.9)	7.85 (4.25–14.51)
Moss et al.,26 2010	74.6 (62.9–84.2)	89.8 (87.4–91.9)	7.3 (5.7–9.4)	0.28 (0.19–0.42)	40.8 (32.2–49.7)	97.4 (95.9–98.5)	25.93 (15.88–42.34)
Da Silva Gane et al.,30 2013	57.7 (43.2–71.3)	74.7 (69.3–79.5)	2.3 (1.7–3.1)	0.57 (0.41–0.78)	28.8 (20.4–38.6)	90.8 (86.5–94.2)	4.02 (2.41–6.70)
Pang et al.,22 2013	61.4 (45.5–75.6)	74.6 (69.5–79.3)	2.4 (1.8–3.3)	0.52 (0.35–0.76)	24.8 (17.0–34.0)	93.4 (89.7–96.1)	4.67 (2.69–8.10)
Reilly et al.,23 2013	100 (87.7–100)	31.6 (19.9–45.2)	1.4 (1.2–1.7)	0.05 (0.00–0.87)	41.8 (29.8–54.5)	100 (81.5–100)	26.70 (2.44–291.88)
Khan et al.,24 2014	82.2 (75.8–87.5)	71.9 (66.6–76.7)	2.9 (2.4–3.5)	0.25 (0.18–0.34)	62.2 (55.7–68.4)	87.8 (83.2–91.5)	11.82 (8.08–17.29)
Moroni et al.,25 2014	83.7 (75.1–90.2)	69.3 (60.5–77.2)	2.7 (2.1–3.6)	0.24 (0.15–0.37)	69.0 (60.2–77.0)	83.8 (75.3–90.3)	11.55 (6.74–19.79)
Feyi et al.,27 2015	66.7 (50.5–80.4)	77.9 (70–84.6)	3.0 (2.1–4.4)	0.43 (0.28–0.66)	48.3 (35.0–61.8)	88.3 (81.2–93.5)	7.07 (3.74–13.36)
Vick et al.,28 2015	57.9 (53.4–62.4)	89.4 (88.5–90.3)	5.5 (4.9–6.2)	0.47 (0.42–0.52)	38.0 (34.4–41.6)	95.0 (94.3–95.7)	11.66 (9.81–13.85)
Gerlach et al.,33 2016	48.2 (39.7–56.8)	88.6 (86.0–90.9)	4.3 (3.2–5.6)	0.58 (0.50,0.69)	46.6 (38.3–55.0)	89.3 (86.7–91.5)	7.27 (5.17–10.22)
Amro et al.,31 2016	56.4 (39.6–72.2)	82.7 (76.0–88.2)	3.3 (2.1–5.0)	0.53 (0.37–0.76)	44.0 (30.0–58.7)	88.7 (82.6–93.3)	6.19 (3.29–11.65)
Lefkowits et al.,35 2016	64.8 (50.6–77.3)	75.1 (68.7–80.8)	2.6 (1.9–3.5)	0.47 (0.32–0.68)	40.2 (29.9–51.3)	89.2 (83.7–93.4)	5.56 (3.25–9.52)
Carmen et al.,32 2016	77.8 (40.0–97.2)	67.5 (50.9–81.4)	2.4 (1.4–4.2)	0.33 (0.10–1.14)	35.0 (15.4–59.2)	93.1 (77.2–99.2)	7.27 (1.74–30.40)
Lakin et al.,34 2016	20.5 (13.5–29.2)	94.4 (93.2–95.5)	3.7 (2.4–5.6)	0.84 (0.77–0.93)	20.2 (13.2–28.7)	94.5 (93.3–95.6)	4.36 (2.85–6.65)

Note: CI = confidence interval, DOR = diagnostic odds ratio, LR+ = positive likelihood ratio, LR− = negative likelihood ratio, NPV = negative predictive value, PPV = positive predictive value.

Table 4:

Meta-analytic estimates of prognostic properties^*

Parameter	Estimate for all patients (95% CI) n = 11 621	Heterogeneity (I2), %	Estimate for patients with noncancer illness (95% CI) n = 2457	Heterogeneity (I2), %	Estimate for patients with cancer (95% CI) n = 6927	Heterogeneity (I2), %	p value†
DOR	8.21 (6.21–10.87)	0	5.94 (4.57–7.81)	0	10.69 (7.06–16.19)	28.0	0.02
LR+	3.4 (2.8–4.1)	0	2.7 (2.1–3.6)	0	4.2 (2.9–6.0)	17.4	0.06
LR−	0.41 (0.32–0.54)	0	0.53 (0.46–0.61)	0	0.41 (0.32–0.53)	39.0	0.09
Sensitivity, %	67.0 (55.7–76.7)	88.7	60.7 (52.6–68.1)	40.0	66.4 (54.1–76.8)	89.2	0.4
Specificity, %	80.2 (73.3–85.6)	96.5	75.9 (67.6–82.6)	92.7	84.3 (77.6–89.3)	95.0	0.08
PPV, %	37.1 (30.2–44.6)	89.5	31.3 (25.0–38.3)	67.8	46.8 (36.4–57.5)	90.1	0.02
NPV, %	93.1 (91.0–94.8)	86.4	93.2 (90.5–95.2)	68.6	92.4 (87.3–95.6)	94.3	0.7
AUC	0.81 (0.78–0.84)	NA	0.77 (0.74–0.8)	NA	0.83 (0.79–0.87)	NA	0.02

Note: AUC = area under the ROC curve, CI = confidence interval, DOR = diagnostic odds ratio, LR+ = positive likelihood ratio, LR− = negative likelihood ratio, NA = not applicable, NPV = negative predictive value, PPV = positive predictive value.

^*Parameter estimates for all patients (16 studies, 17 cohorts) used hierarchical summary receiver operating characteristic (sROC) models. For cancer studies involving patients with cancer (5), studies involving patients with noncancer illness (9, 10 cohorts) and for all measures of heterogeneity (including for all studies combined), analyses used univariable models (see text for details). Heterogeneity measures are not available for AUC, which is derived from DOR.

^†The z test was used for p values to compare studies involving patients with cancer with studies involving patients with noncancer illness.

Figure 2:

Summary receiver operating characteristic (sROC) curve. Each rectangle represents 1 study; the width of the rectangle is proportional to the standard error (SE) of the sensitivity, and the height is proportional to the SE of the specificity. The summary point is the meta-analytic sensitivity and specificity obtained from the hierarchical sROC model, and the dotted area is the 95% confidence interval region, based on the same model.

Prognostic properties of the SQ were better in studies involving patients with cancer than those involving patients with noncancer illness (Table 4); meta-analytic estimates of DOR, PPV and AUC were statistically significantly different between these patient populations. However, even in patients with cancer,^{25,26,28,33,35} likelihood estimates would generate small changes in pretest probability, with a pooled LR+ of 4.2 (95% CI 2.9–6.0) and LR− of 0.41 (95% CI 0.32–0.53). In studies involving patients with noncancer illness, ^{1,20–23,27,30–32} likelihood ratios were less helpful (pooled LR+ 2.7 [95% CI 2.1–3.6] and LR− 0.53 [95% CI 0.46–0.61]). Heterogeneity was absent or low for DOR and LRs, but generally high for sensitivity, specificity, PPV and NPV for both cancer and noncancer studies.

Interpretation

In this systematic review, we identified 16 studies with 17 distinct cohorts that studied the SQ as a tool to predict death in patients with serious illness. Overall, pooled results suggest poor to modest accuracy of the SQ for predicting death at 12 months, with low sensitivity and positive predictive values for the studied populations. The pooled likelihood ratios are in the range of those that generate small changes from pretest to posttest probability.⁴⁰ Prognostic performance was worse for noncancer illness, missing more than one-third of those who died and more than two-thirds of positive results proved to be false. The NPV of the SQ was high, meaning that SQ–patients (“I would be surprised if this patient died”) had a high probability of living. However, this finding was largely driven by the low prevalence of death in the included studies (e.g., if 12% of the population dies, then a coin flip would yield an NPV of 88%). One study also reported only a slight to fair interobserver reliability, and most included studies were felt to have at least a moderate risk of bias.

Our results may not be surprising to some, because physicians are inaccurate prognosticators,² and the SQ was not originally conceived as a prognostic tool but rather as a screening test for patients who might benefit from a palliative approach. The first reported use of the SQ, albeit without outcomes data, was among a primary care population in Tacoma, Washington.⁴¹ Since then, the SQ has become widely used and promoted as a trigger for palliative or end-of-life interventions,^3,4 and has been incorporated as a screening test into the Gold Standards Framework in the United Kingdom⁵ and the Necesidades Paliativas (NECPAL) program in Catalonia.⁶

The basic assumption of the SQ as a screening test is that patients in their final year of life may have unmet palliative care needs. In general, a screening test should have high sensitivity; lower specificity is acceptable, if the cost of a false positive is minimal, or if each positive on the screening test is followed by a more specific confirmatory test. In the case of the SQ, the sensitivity is not high and, when applied to some populations, the number of positive results could be large: 7.7% of the population over age 65 years in the Osona region of Catalonia,⁶ up to one-third of inpatients in 1 UK study⁴² and 78% of patients admitted with advanced congestive heart failure in the abstract of 1 study in the United States.⁴³ The studies included in this meta-analysis had SQ+ rates that ranged from 7% to 79%. It is possible that the SQ is more accurate for identifying those with unmet palliative needs than those in the final year of life. But if being in the final year of life is a surrogate for unmet end-of-life care needs, including palliative care, then our results show that the SQ will simultaneously generate referrals of hospice and palliative care for patients who may not benefit from assessment and miss many patients who might benefit.

The high false-positive rate may not be surprising, because clinicians answering the SQ must consider whether death is possible rather than probable. However, several studies highlighted other concerns about the systematic use of the SQ. First, clinicians may only have moderate agreement when answering the SQ, with more experienced clinicians generally more accurate than those with less experience.^30,44 Second, although many of the single-centre studies showed a high response rate, some clinicians may be reluctant to adopt the SQ into routine practice;^21,25,45 1 study reported that discomfort among staff led to the SQ being dropped as a trigger for a care bundle for patients in hospital.⁴⁶ This discomfort is an important problem for a test that depends on clinician participation. Third, the poorer sensitivity and positive likelihood ratio in patients with noncancer illness, such as those with end-stage organ failure and frailty,^47,48 suggests that the SQ may be particularly unhelpful in this population. Fourth, physicians may be reluctant to commit to a positive answer on the SQ even if they suspect that the patient is dying.^20,49 Thus, the SQ may have more success as a trigger for hospice and palliative care when used in selected settings, such as oncology, with a small number of experienced assessors.

It is possible that a false-positive SQ result would be beneficial, because a palliative approach or referral to hospice and palliative care may benefit patients who are not in the last year of life. The American Society of Clinical Oncology recommends “early” palliative care referral for patients with cancer,⁵⁰ but there is no consensus about the ideal timing of the referral. Two large randomized controlled trials (RCTs) showed a benefit to “early” referral to hospice and palliative care for patients with newly detected metastatic cancer,^51,52 but neither used the SQ as a trigger. A recent analysis of RCTs of hospice and palliative care found that authors used “no clear definitions” of hospice and palliative care patients, and that there was a “lack of consensus concerning the attributes of illnesses needing palliation.”⁵³ Although Advance Care Planning may be appropriate for patients at any stage of illness, some authors have highlighted the “Goldilocks phenomenon,” in which such planning may be ineffective or deleterious if implemented too early or too late.³ Furthermore, patients with noncancer illness may benefit from earlier referral to hospice and palliative care, because they may have a longer period of high support needs.⁵⁴ But these patients can survive for years even with poor prognostic signs, suggesting that the burden on most hospice and palliative care services would be unsustainable if they were routinely referred early. Given the high false-positive rate, any hospice and palliative care service using the SQ as a referral trigger would need to screen out many referrals to avoid being overwhelmed. The broader issue of limited hospice and palliative care resources may be addressed by initiatives to educate and better support primary care providers in delivering palliative care without referral to hospice and palliative care services.

Ultimately, our findings do not indicate that the SQ is a “bad” or “good” test, but clinicians could benefit from better ways to identify patients with palliative health trajectories. One study reported improved prognostic accuracy by combining SQ with clinical predictors among a population of patients receiving hemodialysis.²⁰ Conversely, 1 study in our review reported that the SQ no longer predicted death when included in a multivariable model with the Clinical Frailty Scale.³² The Gold Standards Framework⁵ and NECPAL⁶ both use the SQ serially with general and disease-specific indicators to identify those with hospice and palliative care needs, but the Supportive and Palliative Care Indicators Tool (SPICT)¹⁵ uses these indicators without the SQ, which was removed. Further studies will be needed to determine whether the SQ combined with other clinical indicators improves the identification of patients with hospice and palliative care needs.

Another study used administrative data for all nonpsychiatric admissions of adult patients to hospitals in Ontario and Alberta, and to the Brigham and Women’s Hospital in Boston to derive a score with excellent discrimination (AUC 0.92) for predicting 1-year mortality among patients admitted to hospital for nonpsychiatric indications.^55,56 If the findings are reproducible, the score could be implemented to generate automatic prompts for clinicians to consider discussions of advanced care planning or referral to hospice and palliative care. However, without a consistent definition of a patient who is “palliative” or a consensus about what problems hospice and palliative care is meant to address, we do not have a gold standard to which we can compare any of these triggers.

Strengths and limitations

Strengths of this systematic review include its novelty; a rigorous and transparent search strategy involving multiple databases; independent duplicate review for study selection, data abstraction and quality assessment; efforts to obtain unpublished data from primary study authors; and robust statistical methods. Limitations include the predominant conduct of the studies, in single centres with a small number of evaluators, which limited generalizability and assessment of interrater reliability. Many studies had methodologic limitations that were primarily related to incomplete participation of eligible patients and lack of detail on whether a SQ+ response triggered a discussion with the patient regarding limitations of life-sustaining treatments. These limitations may have led to overestimates of the prognostic performance of the SQ. In addition, the meta-analytic estimates of predictive values will differ in populations with different risks of death.

Conclusion

In summary, the SQ is intended to be a simple and feasible screening test to identify patients with hospice and palliative care needs, but it performs poorly to modestly when used to predict death at 6 to 18 months, with poorer performance among patients with noncancer illness. Based on these findings, the SQ should not be used as a stand-alone prognostic tool, and we do not know whether it is more accurate for identifying patients with unmet palliative needs than it is for those in the final year of life. The high false-positive rate for SQ may be of concern if it used as a routine trigger for time-consuming, costly or poorly available assessments for hospice and palliative care. Developing accurate, reliable and automated means of identifying patients with hospice and palliative care needs in a variety of settings remains a high-priority area of research.