The Accuracy of Probabilistic Versus Temporal Clinician Prediction of Survival for Patients with Advanced Cancer: A Preliminary Report

The accuracy of probabilistic and temporal clinician prediction of survival for advanced cancer patients admitted to an acute palliative care unit were assessed and compared. Factors associated with prediction accuracy were identified.

Abstract

Clinicians have limited accuracy in the prediction of patient survival. We assessed the accuracy of probabilistic clinician prediction of survival (CPS) and temporal CPS for advanced cancer patients admitted to our acute palliative care unit, and identified factors associated with CPS accuracy. Eight physicians and 20 nurses provided their estimation of survival on admission by (a) the temporal approach, “What is the approximate survival for this patient (in days)?” and (b) the probabilistic approach, “What is the approximate probability that this patient will be alive (0%–100%)?” for ≥24 hours, 48 hours, 1 week, 2 weeks, 1 month, 3 months, and 6 months. We also collected patient and clinician demographics. Among 151 patients, the median age was 58 years, 95 (63%) were female, and 138 (81%) had solid tumors. The median overall survival time was 12 days. The median temporal CPS was 14 days for physicians and 20 days for nurses. Physicians were more accurate than nurses. A higher accuracy of temporal physician CPS was associated with older patient age. Probabilistic CPS was significantly more accurate than temporal CPS for both physicians and nurses, although this analysis was limited by the different criteria for determining accuracy. With the probabilistic approach, nurses were significantly more accurate at predicting survival at 24 hours and 48 hours, whereas physicians were significantly more accurate at predicting survival at 6 months. The probabilistic approach was associated with high accuracy and has practical implications.

Introduction

The ability to prognosticate accurately has great implications for clinical decision making, particularly in the end-of-life setting for patients with advanced cancer. Many important decisions, such as chemotherapy use, hospice referral, advance care planning, discharge planning, and personal finances, are dependent on the length of expected survival [1–5]. Various prognostic factors and prognostic models, such as the Palliative Prognostic Score and the Palliative Prognostic Index, have been developed [6–8]. However, current prognostic models are relatively cumbersome and inaccurate. To date, clinician prediction of survival (CPS) remains the most practical and hence most commonly used approach to estimate prognosis [9].

CPS is the subjective formulation of survival based on clinical experience and knowledge of prognostic factors and the natural history of disease [10]. In this approach, the clinician is typically asked “How long do you think the patient will live?” and a temporal answer is provided. CPS has a number of key advantages over existing prognostic models, including its convenience (no need for additional data), quick response, and the provision of an easily interpretable answer. However, temporal CPS has consistently been shown to overestimate survival times [11]. In this approach, clinicians are forced to project an “expiration date” for their patients, which could make them feel uncomfortable. Furthermore, temporal CPS is highly clinician dependent, making it a less reliable tool [12, 13].

There is a pressing need to develop better approaches to CPS, which would allow clinicians to provide more accurate survival predictions for patients and their families, and to better facilitate clinical decision making. To better account for the uncertain nature of prognostication, we examined a probabilistic approach to CPS by asking clinicians “What is the approximate probability that this patient will be alive (0%–100%)?” for a number of defined time frames. In this prospective study, we compared the accuracy of probabilistic CPS and temporal CPS, and identified factors associated with CPS accuracy using these two approaches. We hypothesized that the probabilistic approach is more accurate than the temporal approach.

Patients and Methods

Study Setting and Criteria

The Institutional Review Board at The University of Texas MD Anderson Cancer Center approved this study. All clinicians who participated in this study signed an informed consent form prior to enrollment. The Institutional Review Board provided waiver of consent for patient participation.

Consecutive patients with advanced cancer who were ≥18 years old and admitted to the acute palliative care unit (APCU) at MD Anderson Cancer Center between April 2010 and July 2010 were included in this study. This 12-bed APCU provides intensive symptom support and transition of care for patients with advanced cancer and their families. It is staffed by an interdisciplinary team of physicians, nurses, social workers, physiotherapists, pharmacists, and a chaplain [14].

Patient Characteristics

Upon admission, patient demographics such as age, sex, race, education, religion, and cancer diagnosis were retrieved from our institutional electronic database.

Clinician Characteristics

All physicians from the Department of Palliative Care and Rehabilitation Medicine at the MD Anderson Cancer Center participated in this study. These physicians were board certified in palliative care and provided patient care following standardized clinical protocols. All bedside nurses who worked in the APCU during the study period were also enrolled in this study. These nurses received specialized training in palliative care and cared for two to four patients per shift. We collected various clinician characteristics including age, sex, profession (physician or nurse), religion, years of clinical experience, and years of palliative care experience.

CPS

Upon admission, both the attending physician and bedside nurse most responsible for the patient's care independently filled out an eight-question form inquiring about CPS using two different approaches: (a) a temporal approach, “What is the approximate survival for this patient (in days)?” and (b) a probabilistic approach, “What is the approximate probability that this patient will be alive (0%–100%)?” for ≥24 hours, 48 hours, 1 week, 2 weeks, 1 month, 3 months, and 6 months. Clinicians were asked to circle an answer of 0%–100% in 10% increments that best reflected the probability of survival for the given time point. The time frames were chosen for their relevance to clinical decision making. Specifically, recognition of “actively dying” is related to 24-hour and 48-hour survival predictions, hospice transfer is dependent on 1-week and 2-week survival predictions, eligibility for chemotherapy and other treatments is dependent on a 1-month survival prediction, and a goals-of-care discussion is related to 3-month and 6-month survival predictions. Prognostic estimates were not shared with patients or their families.

Actual Survival

Overall survival was calculated from study entry to death, or censored at the day of last follow-up if death was not observed. Vital status was obtained from electronic chart review and the Cancer Registry Vital Statistics Database.

Statistical Analysis

We summarized the baseline demographics and accuracy of CPS using descriptive statistics, including medians, means, standard deviations, ranges, and frequencies.

The cutoffs for defining accuracy of CPS were defined a priori. For the temporal approach, an estimation of the number of days of survival was considered accurate if it fell within ±33.3% of the actual survival. This cutoff was previously defined in other published studies [13, 15]. For the probabilistic approach, an estimation of the probability of survival was considered to be accurate if either (a) the patient died and the clinician endorsed a survival probability ≤30% or (b) the patient survived and the clinician selected a survival probability ≥70%. These cutoffs were specifically chosen after discussion among 10 palliative care specialists to be stringent yet clinically meaningful. The clinician had to be relatively confident that the patient would likely die (i.e., ≤30% chance of survival) or live (i.e., ≥70% chance of survival) to be considered accurate. A survival probability of 40%–60% indicated uncertainty and ambivalence and thus was coded as inaccurate regardless of the survival outcome.

We compared the accuracy of CPS between the probabilistic approach and the temporal approach using the McNemar test. This same test was also used to compare the accuracy between physicians and nurses. To determine patient factors associated with CPS accuracy for physicians, we included patient age, sex, race, malignancy (solid versus hematological tumors), duration of CPS, as well as a number of clinical signs and symptoms collected on admission, including palliative performance status, delirium, pneumonia, and dysphagia, in a multivariate regression model. The same process was conducted for CPS accuracy for nurses.

To determine clinician factors associated with CPS accuracy, we first calculated the accuracy rate of each physician and nurse. We then conducted univariate analysis for temporal and probabilistic accuracy using the Mann-Whitney test for categorical variables (i.e., sex, religion, profession) and the Spearman correlation test for continuous, nonparametric variables (i.e., years of clinical experience, years of palliative care experience).

A two-sided p-value <.05 was considered to be statistically significant. The Statistical Package for the Social Sciences (SPSS, version 16.0; SPSS Inc., Chicago, IL) was used for statistical analyses.

Results

Patient Characteristics

In total, 151 patients were enrolled in this study. Table 1 summarizes the patient demographics on APCU admission. Lung, gastrointestinal, and gynecologic cancers represented the most common malignancies. The median palliative performance status was 40% (interquartile range, 30%–50%). The median duration between cancer diagnosis and APCU admission was 19 months (interquartile range, 9–43 months). The median overall survival duration was 12 days (95% confidence interval, 9–15 days). At the time of study analysis, 127 (84%) patients had died.

Table 1.

Patient characteristics on APCU admission

^aUnless otherwise specified.

Abbreviation: APCU, acute palliative care unit.

Clinician Characteristics

Eight physicians and 20 nurses were involved in this study. Clinician characteristics are shown in Table 2. Physicians had longer clinical experience and palliative care experience than nurses.

Table 2.

Clinician characteristics

^aUnless otherwise specified.

Temporal CPS

The median temporal CPS was 14 days (95% confidence interval, 11–17 days) for physicians and 20 days (95% confidence interval, 17–23 days) for nurses. Figure 1 shows that both physicians and nurses systematically overestimated survival times, and nurses were more optimistic than physicians. Among the patients who died (n = 127), actual survival was correlated with temporal CPS by physicians (r = 0.63; p < .001) and nurses (r = 0.50; p < .001). Physicians' and nurses' estimates were also correlated with each other (r = 0.54; p < .001).

Figure 1.

Predicted survival versus actual survival by physicians and nurses for 127 patients who died. The diagonal line depicts perfect concordance, with dots above suggesting overestimation and dots below suggesting underestimation. This plot shows that nurses (♦) were more optimistic than physicians (■) in estimating survival. A logarithmic scale is used for both x-axis and y-axis.

The accuracy of temporal CPS is shown in Figure 2, demonstrating that physicians were more accurate than nurses (32% versus 18%; p < .001).

Figure 2.

Accuracy of clinician prediction of survival (CPS) by the traditional and probabilistic approaches. Probabilistic CPS was significantly more accurate than temporal CPS (p < .001 for all paired comparisons between the temporal approach and each probabilistic time frame; i.e., column 1 versus column 3, 4, 5, 6,7, 8, and 9 for physicians; column 2 versus column 10, 11, 12, 13, 14, 15, and 16 for nurses; McNemar test). We also compared between physicians and nurses for each prognostication question, and the significant p-values are indicated with paired asterisks.

Probabilistic CPS

The median probabilistic CPS percentages provided by physicians were 90%, 80%, 60%, 40%, 10%, 0%, and 0% for survival at ≥24 hours, 48 hours, 1 week, 2 weeks, 1 month, 3 months, and 6 months, respectively. In contrast, the median probabilistic CPS percentages provided by nurses were 100%, 100%, 80%, 60%, 35%, 10%, and 0%, respectively. Overall, only a small proportion of clinicians expressed uncertainty (i.e., 40%–60% probability of being alive) with probabilistic CPS. An uncertain answer was provided by 16% of physicians and 7% of nurses for survival at 24 hours, 19% of physicians and 6% of nurses for survival at 48 hours, 25% of physicians and 19% of nurses for survival at 1 week, 32% of physicians and 22% of nurses for survival at 2 weeks, 22% of physicians and 29% of nurses for survival at 1 month, 8% of physicians and 10% of nurses for survival at 3 months, and 0% of physicians and 7% of nurses for survival at 6 months.

Probabilistic CPS was significantly more accurate than temporal CPS for both physicians and nurses (p < .001 for all paired comparisons between the temporal approach and each individual time frame, McNemar test) (Fig. 2). The respective accuracy rates for predictions of survival at ≥24 hours, 48 hours, 1 week, 2 weeks, 1 month, 3 months, and 6 months were 71%, 66%, 58%, 56%, 67%, 86%, and 96% for physicians and 91%, 86%, 61%, 53%, 60%, 79%, and 88% for nurses. Comparing between physicians and nurses, we found that nurses were significantly more accurate in their prediction of survival at 24 hours (91% versus 71%; p < .001) and at 48 hours (86% versus 66%; p < .001). In contrast, physicians were significantly more likely than nurses to provide accurate prognostication for the 6-month time point (96% versus 88%; p = .006) (Fig. 2).

Predictors of Accuracy of CPS

We examined both patient and clinician characteristics for factors associated with better accuracy of CPS. The accuracy of temporal CPS was associated with older patient age (p = .019) (Fig. 3) but no other patient or clinician characteristic, such as clinician age, sex, and years of experience. There was also wide interphysician variation in temporal CPS (Fig. 3).

Figure 3.

Accuracy of temporal clinician prediction of survival (CPS) by clinician and patient age. The y-axis represents the predicted values from a multivariate logistic regression examining temporal accuracy of CPS as a function of clinician and patient age. The x-axis represents the actual patient age (in years) for each observation. Patient age was a significant predictor of clinician accuracy (p = .019). On average, a significantly more accurate temporal CPS was provided for older patients than for younger patients, regardless of which clinician provided the estimate.

Other than the nursing profession for the 24-hour and 48-hour time point estimates, we found no other patient or clinician characteristics that were associated with higher accuracy for probabilistic CPS.

Discussion

In this prospective study, we compared probabilistic CPS and temporal CPS in patients with advanced cancer admitted to an APCU. We found that the probabilistic approach was associated with greater accuracy than with the temporal CPS approach, suggesting that the way we ask about prognosis can have an important impact on its accuracy. For temporal CPS, older patient age was associated with higher accuracy. For probabilistic CPS, nurses were highly accurate in the last 48 hours of life, whereas physicians were better at longer term prognostication.

The low accuracy rate for temporal CPS is consistent with the 20%–40% rates reported in the literature [11, 13, 15]. This could partly be related to the sensitive nature of the temporal question, and simply that the clinician response has no upper limit. Clinicians may err on the side of caution and express their uncertainty by providing longer projected survival.

Few studies have compared the prognostic accuracy of temporal CPS between physicians and nurses. We found that physicians were more accurate with the temporal approach than nurses, who were more optimistic in their prognostication. Llorera et al. [15] also reported that oncologists were better at predicting survival than nurses. Similar to other studies [16], we found that some physicians consistently prognosticate more accurately than others (Fig. 3), suggesting that there may be specific clinician-related factors contributing to better accuracy. The literature suggests that greater clinician experience and a shorter duration of the patient–physician relationship are associated with better temporal CPS accuracy [13]. However, we did not identify any association between the accuracy of temporal CPS and the length of clinical and palliative care experience. This may be explained by the fact that we only examined clinicians practicing in palliative care, who tend to have strong expertise with survival prediction. Thus, we may be observing a saturation effect in which additional years of clinical experience do not contribute to further accuracy in prognostication once a basic competency is attained. Further studies are needed to examine how experience can affect accuracy. Qualitative studies may also have a role in identifying novel predictive factors for CPS accuracy.

In addition to clinician-related factors, patient-related factors have also been shown to be associated with greater prognostic accuracy. Most recently, Clément-Duchéne et al. [17] reported higher accuracy in patients with lower quality of life. Interestingly, our analysis showed that older patient age was an independent factor associated with higher temporal CPS accuracy (Fig. 3). One potential explanation is that younger patients may not show the same prognostic signs as older patients. Alternatively, it may be psychologically more difficult for physicians to give younger patients a poor prognosis. Further studies are required to examine this phenomenon.

To overcome the inaccuracy and inconsistency associated with temporal CPS, various investigators have suggested alternative questions to foretelling outcomes. Some researchers have proposed the “surprise question,” in which clinicians were asked “Would I be surprised if this patient died in 1 year?” This concept resulted in an accuracy rate of 88%, with a positive predictive value of 41% and negative predictive value of 97% [18]. A challenge with this approach is that it forces a binary response with a “yes” or “no” answer. To better account for uncertainty when predicting prognosis, a probabilistic approach could be used instead. Forster and Lynn [19] asked clinicians to provide “the number of months that would include 90% of all deaths of patients exactly like this.” In other studies, clinicians estimated the chance of patient survival at a specific time point (e.g., 6 month), and the accuracy was analyzed using the Brier Score [20–22]. Our current study examined the probabilistic approach using clinically relevant time points and easy-to-interpret cutoffs. Unlike temporal CPS and the surprise question, our approach allows clinicians to express their degree of uncertainty regarding a patient's survival. It also frames the question to support clinical decision making. For instance, if a clinician is evaluating a patient for hospice eligibility, the prognosis question can be framed as “What is the approximate probability that this patient will be alive for ≥6 months?” An answer ≥70% would suggest that this patient may not be eligible for hospice from a survival standpoint. In another scenario, an estimate ≤30% for a ≥48 hour chance of survival may suggest that the patient would benefit from staying in the hospital rather than being discharged home. Further research is necessary to examine potential practical applications for probabilistic CPS.

To our knowledge, this is the first study to directly compare probabilistic CPS with temporal CPS. We found that the probabilistic approach was superior to the temporal approach in predicting survival. Although nurses still overestimated with the probabilistic model, they were much more accurate than with the temporal approach. The high accuracy rate can be attributed to the use of the probability question, which is easier to ask and answer, provided that the respondents were comfortable with the concept of probability. This approach, coupled with the fact that we tested the questions with a group of experienced clinicians and in a patient population with a relatively short survival time, made it possible to achieve a high degree of accuracy. Interestingly, only a minority of clinicians expressed uncertainty when using this approach, supporting the clinical utility of probabilistic CPS.

By asking multiple questions with the probabilistic approach, we were able to gain important insights into how accuracy varies within different time frames. Both physicians and nurses were accurate when the patient's survival time was in terms of months and days, but not weeks. This finding is consistent with other studies demonstrating that intermediate survival is difficult to predict [23, 24]. We postulate that longer term prognostication (i.e., months of survival) requires knowledge of the natural history of the disease, whereas very short-term prognostication (i.e., days of survival) requires recognition of the signs of dying. Both of these are relatively well-established knowledge domains. In contrast, prognostication in the intermediate time frame involves understanding the risk for developing acute life-threatening chaotic events, such as pneumonia and thromboembolism. These events often happen 1–4 weeks before death, and the outcomes are more difficult to predict because of treatment and interpatient variability. Development of prognostic tools for this time frame may augment clinician accuracy.

The survival time of our cohort was relatively short, with approximately two thirds of patients discharged alive. Accurate prognostication is critical for this cohort because many important therapeutic (e.g., discontinuation of aggressive therapies) and discharge (e.g., home, inpatient hospice, home hospice) decisions are made during an APCU admission, all of which are dependent on projected survival. We found that nurses were highly accurate in the last 48 hours of life, whereas physicians were better with predictions earlier in the disease trajectory. This observation is not surprising given that nurses spend more time at a patient's bedside and are thus able to pick up imminent signs of dying. Our findings highlight the importance of incorporating nursing input into clinical decision making in an interdisciplinary environment.

This study has several limitations. First, we were only able to include a small number of clinicians in this study, which precludes detailed analysis regarding clinician characteristics predictive of CPS accuracy. Our preliminary findings strongly support the need to conduct multicenter studies with larger sample sizes. Second, the accuracy rate is very much dependent on how accuracy is defined. The accuracy criterion for temporal CPS is particularly stringent for patients with a short expected survival duration, and this should be taken into consideration when interpreting our findings. We identified our cutoffs for probabilistic CPS a priori based on discussions with the research team, and employed practical and easily interpretable cutoffs. We did not use the Brier Score to analyze probabilistic CPS, but focused on the accuracy for individual patients instead. This is similar to tumor response, in which we need to judge the outcome by individuals rather than in aggregate. Third, this study was conducted in an APCU at a tertiary care cancer center with a unique patient population with a very short prognosis, which may limit its generalizability. Validation in other palliative care settings and also earlier in the disease trajectory would be beneficial. Our observation that the patterns of probabilistic estimation in physicians and nurses mirrored each other lends support to the applicability of this approach.

Conclusion

This study demonstrated that the way we ask and interpret the prognosis question can have an important impact on its accuracy and clinical utility. By reframing the dreaded question “How long do I have?” experienced clinicians may be able to intuitively formulate survival estimates with a higher degree of accuracy. The probabilistic question proposed in this study is easy to use, and the practical time frames can potentially be linked to clinical decision making, particularly when used in conjunction with other prognostic tools [25]. Our study further highlights the differential yet complementary roles of physicians and nurses in predicting prognosis. Further research and education are necessary to improve CPS and ultimately patient care.

Author Contributions

Conception/Design: David Hui, Kelly Kilgore, Linh Nguyen, Stacy Hall, Julieta Fajardo, Tonye P. Cox-Miller, Shana L. Palla, Wadih Rhondali, Jung Hun Kang, Sun Hyun Kim, Egidio Del Fabbro, Donna S. Zhukovsky, Suresh Reddy, Ahmed Elsayem, Shalini Dalal, Rony Dev, Paul Walker, Sriram Yennu, Akhila Reddy, Eduardo Bruera

Provision of study material or patients: Egidio Del Fabbro, Donna S. Zhukovsky, Suresh Reddy, Ahmed Elsayem, Shalini Dalal, Rony Dev, Paul Walker, Sriram Yennu

Collection and/or assembly of data: David Hui, Kelly Kilgore, Linh Nguyen, Stacy Hall, Julieta Fajardo, Tonye P. Cox-Miller, Wadih Rhondali, Jung Hun Kang, Sun Hyun Kim, Egidio Del Fabbro, Donna S. Zhukovsky, Suresh Reddy, Ahmed Elsayem, Shalini Dalal, Rony Dev, Paul Walker, Sriram Yennu, Akhila Reddy

Data analysis and interpretation: David Hui, Shana L. Palla, Eduardo Bruera

Manuscript writing: David Hui, Eduardo Bruera

Final approval of manuscript: David Hui, Kelly Kilgore, Linh Nguyen, Stacy Hall, Julieta Fajardo, Tonye P. Cox-Miller, Shana L. Palla, Wadih Rhondali, Jung Hun Kang, Sun Hyun Kim, Egidio Del Fabbro, Donna S. Zhukovsky, Suresh Reddy, Ahmed Elsayem, Shalini Dalal, Rony Dev, Paul Walker, Sriram Yennu, Akhila Reddy, Eduardo Bruera