A Comparison of the Accuracy of Clinician Prediction of Survival versus the Palliative Prognostic Index

Abstract

Context

Survival predictions for advanced cancer patients impact many aspects of care, but the accuracy of clinician prediction of survival (CPS) is low. Prognostic tools such as the Palliative Prognostic Index (PPI) have been proposed to improve accuracy of predictions. However, it is not known if PPI is better than CPS at discriminating survival.

Objective

We compared the prognostic accuracy of CPS to PPI in patients with advanced cancer.

Methods

This was a prospective study in which palliative care physicians at our tertiary care cancer center documented both the PPI and CPS in hospitalized patients with advanced cancer. We compared the discrimination of CPS and PPI using concordance statistics, area under the receiver-operating characteristics curve (AUC), net reclassification index (NRI) and integrated discrimination improvement for 30-day survival and 100-day survival.

Results

215 patients were enrolled with a median survival of 109 days and a median follow up of 239 days. The AUC for 30-day survival was 0.76 (95% confidence interval [CI] 0.66–0.85) for PPI and 0.58 (95% CI 0.47–0.68) for CPS (P<0.0001). Using the NRI, 67% of patients were correctly reclassified using PPI instead of CPS for 30-day survival (P=0.0005). CPS and PPI had similar accuracy for 100-day survival (AUC 0.62 vs. 0.64; P=0.58).

Conclusion

We found that PPI was more accurate than CPS when used to discriminate survival at 30 days, but not at 100 days. This study highlights the reason and timing for using PPI to facilitate survival predictions.

Introduction

As patients approach the end of life, it becomes crucial to accurately predict the length of survival, for many treatment decisions hinge on this determination.(1,2) Knowing whether patients’ life expectancy is days, weeks, or months allows physicians to make appropriate healthcare recommendations and allows patients to make many personal and healthcare choices that align with their values.(3) Clinicians frequently use their experience and clinical reasoning skills, rather than objective data, to estimate survival.(4) Previous research indicates that clinician prediction of survival (CPS) is inaccurate – only 20% of predictions fall within 33% of actual survival time and 63% of predictions overestimate patient survival.(5–7)

Given the importance of accurate life expectancy predictions, and the inaccuracy of clinical gestalt forming these predictions, some experts have proposed the use of validated prognostic models that incorporate multiple established prognostic markers. The Palliative Prognostic Index (PPI) is a validated prognostic model for patients with advanced cancer.(8–11) It consists of five independently predictive variables: performance status (based on the Palliative Performance Scale), oral intake, edema, dyspnea at rest, and delirium. PPI has an advantage over other prognostic tools in that it does not require laboratory values or CPS, has been validated in both solid tumors and hematologic malignancies, and can easily and reliably be performed in a number of settings by expert and non-expert physicians or nurses.(9,10,12)

Despite the development of PPI and other prognostic scores, clinicians continue to rely on their clinical gestalt for survival prediction. One potential reason is that they believe that their judgment already incorporates many variables in PPI and could potentially be more accurate with the added weight of their experience. To date, no studies have directly compared the accuracy of PPI to CPS. With a direct comparison, we would more fully understand the role of PPI in prognostication. In this prospective study, we compared the accuracy of PPI and CPS in survival prediction using established metrics. We hypothesized that PPI is more accurate than CPS.

Patients and methods

Study setting and criteria

This was a secondary analysis of a prospective study designed to examine novel prognostic markers in patients with advanced cancer. Patients enrolled in the study were adults with advanced cancer, hospitalized at MD Anderson Cancer Center in Houston, TX, and consulted on by the palliative care team. Patients with delirium, contraindications to bioelectric impedance analysis, or inability to use the hand dynamometer were excluded. The Institutional Review Board at MD Anderson Cancer Center reviewed and approved the study protocol. Participants were enrolled between 9/22/2011 and 1/26/2013 after providing written informed consent.

Data collection

We collected patient demographics such as sex, age, race and cancer diagnosis during study enrollment (Table 1). PPI score consists of five variables: Palliative Performance Scale, amount of oral intake, edema (absence/presence), dyspnea at rest (absence/presence), and delirium (absence/presence).(13) The Palliative Performance Scale awards points for PPI according to the following scale: 10–20 - four points, 30–50 - two and a half points, and higher than 60 - zero points. Patients receive two and a half points if their oral intake is a few mouthfuls or less, one point if intake is reduced (but more than mouthfuls), one point for edema, three and a half points for dyspnea at rest, and four points for delirium. Each variable is independently scored and then summed; the total score ranges from 0 to 15 and patients are placed into one of three groups: 0–4, 4.5–6, 6.5–15.(13–15) A score of 4 or less predicts a survival of greater than 6 weeks (sensitivity of 80%, specificity of 77%) and a score of 6 or more predicts a survival less than 3 weeks (sensitivity 80%, specificity of 85%).(13)

Table 1.

Patient characteristics (N=215)

Characteristics	N (%)1
Age, average (range)	54.9 (22 – 79)
Sex
Female	126 (58.6)
Male	89 (41.4)
Ethnicity
White	142 (66.1)
Black	43 (20)
Hispanic	28 (13)
Others	2 (0.9)
Education
High school or lower	114 (53)
College	72 (33.5)
Advanced	29 (13.5)
Cancer
Breast	27 (12.6)
Gastrointestinal	70 (32.6)
Genitourinary	19 (8.8)
Gynecological	23 (10.7)
Head and neck	10 (4.7)
Hematological	12 (5.6)
Others	18 (8.4)
Respiratory	36 (16.7)
Overall survival in days, median (95% confidence interval)	109 (71–133)
Clinician prediction of survival (CPS) in days, median (Q1-Q3)	90 (60–150)
Palliative prognostic index (PPI score)
0 – 4 points	160 (74.4)
4.5 – 6 points	38 (17.7)
6.5 – 15 points	17 (7.9)
PPS in PPI, average (range)	55.7 (20 – 90)
Oral intake in PPI
Normal	69(32.1)
Moderately reduced	96(44.7)
Severely reduced	50(23.3)
Edema in PPI
Absent (0 point)	160 (74.4)
Present (1 point)	55 (25.6)
Dyspnea rest in PPI
Absent (0 point)	188 (87.4)
Present (1 point)	27 (12.6)
Delirium in PPI
Absent (0 point)	215 (100)
Present (1 point)	0

CPS, clinician prediction of survival; PPI, palliative prognostic index

¹unless otherwise specified

There were 18 board certified palliative care faculty involved in patient evaluation. The primary palliative care physician for each patient estimated the length of survival at study enrollment and this estimation was used for CPS. Actual length of survival of study participants was collected from electronic medical records and institutional databases.

Statistical analysis

The sample size justification was previously reported and was based on having at least 10 events (i.e. deaths) for each prognostic variable in a multivariable Cox Proportional Hazards regression model.(16,17) All patients with the variables required to calculate a PPI score were included in this study.

We summarized the baseline demographics using descriptive statistics, including means, medians, percentages, interquartile ranges (IQRs), ranges and 95% confidence intervals (CI).

We compared the accuracy of CPS and PPI in relation to overall survival for all enrolled subjects by assessing their discrimination ability with C-index, the area under the receiver-operating characteristics curve (AUC), net reclassification index (NRI), and integrated discrimination improvement (IDI) as described in Hui et al.(17) Discrimination determines the ability of a prognostic tool to differentiate between patients who remained alive or died by a specific time frame. The C-index is defined as the probability of concordance between predicted and observed responses. It is a global index commonly used to validate the prediction ability of a model. A value of 0.5 indicates that the prognostic tool predicts the outcome no better than chance whereas a value of 1 indicates a perfect prediction. We evaluated how well each prediction model classified the survival outcome (alive, dead) at 30 days and at 100 days, and graphed the AUC for each. The 30-day survival is often used in mortality statistics and can be used to evaluate the quality of care at the end-of-life. The 100-day survival was chosen because many procedures are contraindicated if life expectancy falls within this time frame, and it approximates the median overall survival for this cohort (109 days). The AUC evaluates the ability of a model to classify a binary outcome as its threshold varies and it shows the relationship between the test’s sensitivity and specificity. Similar to the C-index, 0.5 indicates no discrimination and 1.0 indicates perfect discrimination. The NRI assesses the ability of a new model to re-classify subjects compared to an old model into binary event or no-event categories. The index assigns a score of +1 for correctly reclassified subjects, −1 for incorrectly reclassified subjects, and 0 for subjects who are not reclassified. Scores are summed in each group, event and non-event, and divided by the number of subjects in the corresponding group. The NRI score is the sum of these two values from event and non-event groups. We also reported the percentage of subjects who were correctly reclassified. The IDI determines whether adding a new risk factor improves the discrimination slope of a test. It calculates the average probability of an event for both event and non-event groups and measures how much the average probability has increased with the addition of a new risk factor compared to the old model. An improved new model gives an increased predicted probability for events, compared with non-events.

Statistical analyses were carried out in Statistical Analysis System (SAS version 9.2, SAS Institute, Cary, North Carolina) and R version 3.1.3. A P-value of <0.05 is considered statistically significant.

Results

Patient characteristics

Two hundred fifteen of 222 (97%) patients enrolled in this study had full PPI and CPS score data. Patient characteristics and distribution of the PPI score are shown in Table 1. At the time of analysis, 136 of 215 (63%) patients had died, with a median overall survival of 109 days (95% CI 71–133 days). The median follow-up was 239 days (IQR 180–261 days).

Accuracy of CPS and PPI

The C-index for CPS was 0.58 (95% CI 0.47–0.68) (Table 2). The lower limit of the 95% CI for the concordance index of CPS was below 0.5, suggesting that it had limited prognostic utility. The AUC analysis for CPS showed similar findings for both 30-day survival (0.58, 95% CI 0.47–0.68) and 100-day survival (0.62, 95% CI 0.54–0.70) (Table 2, Figure 1).

Table 2.

Discriminatory ability and relative performance of CPS, PPI

	Concordance index (95% CI)	AUC (95% CI) for 30-day survival	AUC (95% CI) for 100-day survival	IDI for 30-day survival3	IDI for 100-day survival3
CPS	0.58 (0.47, 0.68)	0.58 (0.47, 0.68)	0.62 (0.54, 0.70)	--	--
PPI	0.62 (0.51, 0.73)	0.76 (0.66, 0.85)	0.64 (0.56, 0.72)	--	--
Difference between PPI and CPS4	0.2 (0.15, 0.25) 1	0.18 (0.09, 0.27) 2	0.03 (−0.06, 0.12)2	0.15 (0.08, 0.21)	0.03 (−0.01, 0.07)
P-value	<0.0001	0.0001	0.58	<0.0001	0.1478

Abbreviations: CPS, clinician prediction of survival; PPI, palliative prognostic index; CI, confidence interval; AUC, area under the receiver-operating characteristics curve; IDI, integrated discrimination improvement

¹A positive value indicates better discrimination for concordance index

²A positive value indicates better discrimination for AUC

³The relative change in slope is shown based on the integrated discrimination improvement. A positive value indicates better discrimination for PPI compared to CPS

⁴The difference between PPI and CPS is a statistical comparison that calculates A-B and the appropriate standard deviation for this difference. A p-value <0.05 indicates that the difference between PPI and CPS were significantly significant

Figure 1. Discrimination of CPS and PPI score.

These receiver-operating characteristics curves plot sensitivity vs. 1-specificity for (A) 30-day survival and (B) 100-day survival. PPI has larger area under the curve and thus better performance compared to CPS.

The C-index for PPI was 0.62 (95% CI 0.51–0.73; P≤0.0001), which was significantly higher than CPS (Table 2). PPI also had a significantly greater AUC at 30 days (0.76, 95% CI 0.66–0.85; P=0.0001) but not at 100 days (0.64, 95% CI 0.56–0.72; P=0.58) compared to CPS.

Net reclassification index and Integrated Discrimination Improvement

As shown in Table 3, net reclassification index analysis revealed that 67% of patients were correctly reclassified using PPI rather than CPS. In patients who died within 30 days, PPI predicted their life expectancy more accurately than CPS 58% of the time. In patients who were still alive after 30 days, PPI predicted closer to their length of survival 76% of the time compared to CPS. At the 100-day point, no significant difference was detected.

Table 3.

Net Reclassification Table

	PPI better than CPS N (%)1	CPS better than PPI N (%)1	NRI (from CPS to PPI)2(95% CI)	P-value
Died within 30 days	19 (58)	14 (42)	67% (31%, 103%)	0.0005
Alive after 30 days	123 (76)	39 (24)
Died within 100 days	43 (47)	48 (53)	20% (−9%, 48%)	0.19
Alive after 100 days	55 (63)	33 (37)

Abbreviations: CI, confidence interval; CPS, clinician prediction of survival; NRI, net reclassification improvement; PPI, palliative prognostic index

¹We applied logistic regression modeling to compute the probability of an outcome of interest (e.g. death within 30 or 100 days) for each prognostication approach (i.e. CPS or PPI). We then compared the outcomes for each approach. Each cell shows the number of patients in which the probability of having the outcome based on one prognostication approach is closer to predicting the outcome than the other approach, along with the row percentage in parenthesis.

²The percentage of subjects who were correctly reclassified using PPI instead of CPS. A positive value indicates better discrimination.

Similarly, integrated discrimination improvement analysis showed that the discrimination slope of PPI was 15% (95% CI 8%–21%; P<0.0001) higher than CPS for 30-day survival. However, PPI and CPS did not differ significantly for 100-day survival (difference 3%; 95% CI −1%–7%; P=0.15)(Table 2).

Discussion

This study compared the accuracy of PPI and palliative care physicians in estimating length of survival of patients with advanced cancer who were hospitalized in a tertiary care cancer center. We found that PPI was consistently more accurate than CPS when used to discriminate survival at 30 days, but not at 100 days. It also highlights that there is opportunity for continued development of more accurate prediction tools.

PPI was developed to prognosticate survival for patients with advanced cancer with a relatively short life expectancy. It was initially derived in patients with a median survival of approximately 1 month, and was subsequently validated in similar patient populations.(9,13) This score excels at discriminating patients with only a few weeks of survival remaining (i.e. <3 weeks) from those with a longer life expectancy (i.e. >6 weeks). For this reason, it is to be expected that we found that the AUC was higher at 30 days than at 100 days, suggesting that PPI may not be as useful for patients earlier in the disease trajectory with many months of survival. This differential discriminatory ability in prognostic scores was also observed in a previous study, and suggests that clinicians should apply the prognostic factors/tools to the right population (advanced cancer) at the right time (to differentiate weeks vs. months of life expectancy).(17) Notably, prognostic scores for advanced cancer patients with longer-term survival are available, such as the Glasglow Prognostic Score which requires both albumin and C-reactive protein.(15,18,19)

To our knowledge, this is the first study to directly compare PPI and CPS and confirm that this prognostic model is more accurate than clinician intuition. Morita et al. conducted a study with two groups, one in which physicians estimated CPS and another where they referenced PPI and allowed it to guide their judgment and influence their survival estimation. Their team found that using PPI to guide the creation of CPS resulted in improved accuracy of CPS.(20) This finding is consistent with the literature examining other prognostic tools. Gwilliam et al. developed the Prognosis in Palliative Care Study predictor model (PiPS) which was found to independently predict survival at two weeks and two months and was found to perform better than CPS.(21) More recently, we compared the Palliative Prognostic Score (PaP) score to CPS and also found that CPS was less accurate.(17)

Despite the consistency of research showing that prognostic tools are more accurate than CPS, clinicians still rely on CPS. Potential explanations may include the ease of application and clinicians’ preference to rely on their own judgment. Clinicians have biases that may impact the accuracy of their prognostication, including the recency bias (predicting survival for future patients based on survival of recently seen patients), over- or under-valuing certain prognostic factors, and becoming emotionally attached to a patient.(22) For many of these reasons, the Steering Committee of the European Association for Palliative Care recommended that physicians use prognostication tools in addition to CPS in order to best predict life expectancy.(23)

Prognostic scores are superior to CPS because they are more accurate, objective, and reproducible. One way to facilitate more widespread use and adoption of actuarial prognostication tools is to make prognostic calculators easily available and accessible. This could be accomplished by a website (e.g. www.predictsurvival.com) or a smart phone application with various prognosticating equations that allow input of requisite data with an “answer” screen with the results of different tools based on the data input. Such a calculator would eliminate the need to remember the variables and scoring systems associated with each test and would facilitate interpretation of the results. Based on the outputs from various tools, clinicians could refine or become more confident in their survival estimations. The use of these tools may support clinical decision-making, although more research is needed in the advanced cancer setting.

The results of this study should be considered with the following limitations. First, this is a single center study conducted in an acute palliative care unit at a comprehensive cancer center. Some patient characteristics, such as the age, race and cancer biology may differ from other settings. Further studies are thus needed to confirm our findings in other patient populations. Second, because this was a secondary analysis of a study which required patients’ active involvement, we excluded patients with delirium, which is a well-established prognostic factor and one of five PPI variables. By excluding some patients with a shorter survival, we may introduce selection bias and limit the generalizability of our findings. Future studies would ideally include patients with a diagnosis of delirium. Third, the median survival of our cohort was 109 days, when PPI has only been validated in patient populations with weeks of survival. The increased length of survival for our cohort may have affected the discriminatory ability of PPI. However, we found that it was still more accurate than CPS in predicting 30-day survival, which may indicate that PPI is more generalizable than previously thought. More research would be required to further explore this issue. Finally, CPS is dependent on physician experience. While all physicians in this study were practicing in an academic tertiary care center and board certified in palliative care, we did not specifically document their level of experience. Larger studies are needed to examine how much clinical experience could impact CPS in this setting.

This prospective study found that PPI was more accurate than CPS for 30-day survival. It supports the use of PPI over CPS to determine patients’ life expectancies when applied under the appropriate circumstances. This study also highlights the need to develop more accurate prognostic tools and to make them readily available to practicing clinicians.