Prospective Comparison of Prognostic Scores in Palliative Care Cancer Populations

Predicting prognosis in advanced cancer aids physicians in clinical decision making and can help patients and their families to prepare for the time ahead. It can be confirmed that all four prognostic scores used in palliative care studies accurately identify classes of patients with different survival probabilities. The Palliative Prognostic Score has been extensively validated and shows high accuracy and reproducibility in different settings.

Abstract

Purpose.

Predicting prognosis in advanced cancer aids physicians in clinical decision making and can help patients and their families to prepare for the time ahead.

Materials and Methods.

This multicenter, observational, prospective, nonrandomized population-based study evaluated life span prediction of four prognostic scores used in palliative care: the original Palliative Prognostic Score (PaP Score), a variant of PaP Score including delirium (D-PaP Score), the Palliative Performance Scale, and the Palliative Prognostic Index.

Results.

A total of 549 patients were enrolled onto the study. Median survival of the entire group was 22 days (95% confidence intervals [95% CI] = 19–24). All four prognostic models discriminated well between groups of patients with different survival probabilities. Log-rank tests were all highly significant (p < .0001). The PaP and D-PaP scores were the most accurate, with a C index of 0.72 (95% CI = 0.70–0.73) and 0.73 (95% CI = 0.71–0.74), respectively.

Conclusion.

It can be confirmed that all four prognostic scores used in palliative care studies accurately identify classes of patients with different survival probabilities. The PaP Score has been extensively validated and shows high accuracy and reproducibility in different settings.

Introduction

The three main components of medical intervention are diagnosis, therapy, and prognosis. Of these, prognosis is the least studied aspect in scientific literature. As proof of this, a Medline search produced 7,184,331 citations for the term “diagnosis”, 6,244,916 for “therapy”, and only 896,636 for “prognosis” [1]. Prognostic data can help physicians to decide whether to continue with antineoplastic therapies (increasingly used with palliative intent in end-of-life care) or whether the time has come to consider hospice and palliative care programs [2–4]. In fact, although the use of palliative care in combination with specific antineoplastic treatments in the early stages of the disease is significantly favorable, at a certain point palliative care as the only treatment becomes appropriate [5–8]. Major issues in clinical palliative care emerge during the last three months of life [9]. The theme of prognosis is critical in terms of how it is formulated and communicated by physicians to patients and relatives. The present paper focuses on the former topic in an attempt to overcome the difficulties in formulating prognostication reported in the literature. Physicians often lack confidence in formulating and communicating prognosis [10–15]. Clinical prediction of survival (CPS) alone is fairly inaccurate and often presents a bias oriented towards overestimation. Its prognostic capability can potentially be improved by assessing and evaluating a number of clinical symptoms and syndromes. It has been seen that prognostic factors in early phases of cancer are related to pathological findings, correct diagnosis, and appropriate therapy; in contrast, clinical factors take on a greater importance in palliative care [16]. Specific manifestations are highly indicative of prognosis in far advanced and palliative phases, for example, symptoms related to nutritional status (Cancer Anorexia-Cachexia Syndrome), Performance Status Indexes, symptoms such as dyspnea and delirium, and biological parameters, for example, leukocytosis, lymphocytopenia, and C-reactive protein [14, 17, 18]. Numerous authors have tried to integrate specific prognostic factors into prognostic scores used in palliative settings [19–32] with the aim of providing clinicians with easy-to-use support tools.

A Task Force of Cancer Experiences Collaborative recently produced an experts' consensus paper to identify and define priorities in prognostication research [33]. Initially 40 questions were found as potentially interesting by the 25 experts, subsequently reduced to five top questions: (1) How valid are prognostic tools? (2) Can we use prognostic criteria as entry criteria for research? (3) How do we judge the impact of a prognostic score in clinical practice? (4) What is the best way of presenting survival data to patients? (5) What is the most user-friendly validated tool?

During the last decade, prognostic scores such as the Palliative Prognostic Score (PaP score) [19, 20], Palliative Prognostic Index (PPI) [21], and Palliative Performance Scale (PPS) [22, 23] have been presented by leading researchers and subsequently validated by others in different case series. Our own PaP Score has a number of strengths and weaknesses (see Discussion); for example, it does not include “delirium”, which is considered an important prognostic factor. We recently compared the original PaP score with a version of the same score modified to include the delirium symptom (Delirium-PaP Score [D-PaP]) [34]. Results showed that D-PaP accuracy was substantially superimposable with that of the original PaP and did not highlight any potential advantage of the modified score over the original version in clinical practice [34]. Our study presents a prospective comparison between the PaP Score, D-PaP, and two other prognostic scores proposed in the literature, PPS and PPI.

Materials and Methods

This observational, prospective, multicenter, cohort study was conducted on patients consecutively admitted to three Italian hospices from June 2009 to October 2010. During the study period, all patients referred to the hospice were considered eligible, and all were cancer cases. All patients discharged were followed until death or study closure time (January 31, 2011). Patients underwent a full blood count a maximum of 1 week before the assessment so that an evaluation of leukocytosis and lymphocyte percentage was available. Prognostic scores were completed by the physician for all patients on the first day of admission to hospice. Routinely recorded clinical and administrative data included age, gender, and diagnosis, while other information was also collected for the PaP score: presence or absence of dyspnea, anorexia, Karnofsky Performance status (KPS), clinical prediction of survival (based on the clinical experience of the physician), evaluation of leukocytosis, and lymphocyte percentage. The PaP score was utilized as originally built and validated [19, 20]. Total scores can range from 0 to 17.5, and the index is used to classify patients into one of three groups, each with a different probability of survival at 30 days: group A, probability of 30-day survival >70% (score ≤5.5); group B, probability of 30-day survival 30%–70% (score 5.6–11.0); group C, probability of 30-day survival <30% (score >11.0).

D-PaP is a revised version of the original PaP score to which the symptom of delirium has been added [34]. Delirium, as other symptoms, may be assessed in different ways. In PPI it was counted as absent when not present and if caused solely by a single medication and potentially reversible. Otherwise, for D-PaP, delirium was evaluated as present or absent in the opinion of the clinician without looking for the exact reason. It was also assessed with CAM (Confusion Assessment Method) in a subgroup of 269 patients [35]. Total D-PaP scores ranged from 0 to 19.5 and classified patients into three groups according to 30-day survival probability: group A, survival probability >70% (score ≤7.0); group B, survival probability 30%–70% (score 7.1–12.5); group C, survival probability <30% (score >12.5). Results on the diagnostic accuracy of D-PaP when delirium was assessed with CAM or with clinical judgment were superimposable (data not shown). The PPS score [22, 23] is a modification of the Karnofsky Performance scale in which ambulation, activity, self care, intake, and conscious level are considered. PPS is divided into 11 categories, from healthy (100%) to death (0%), and patients are grouped into 3 classes (10–20, 30–50, ≥60), as reported in the literature [21].

PPI identifies five variables (oral intake, presence or absence of edema, dyspnea at rest, delirium, and PPS) that are independently predictive of survival [21]. The total PPI score is calculated by summing the partial scores, and ranges from 0 to 15. Patients are classified into three groups (group A, PPI ≤2.0; group B, PPI 2.1–4.0; group C, PPI >4.0).

Our study was approved by the local Ethical Committee (a single Committee for all three centers) and was conducted in accordance the ethical standards laid down in the 1964 Declaration of Helsinki. The study was partially funded by Istituto Oncologico Romagnolo from 2009 to 2010.

Statistical Analysis

Survival curves were estimated using the product-limit method of Kaplan–Meier [36] and compared by the log-rank statistic test [37]. The discriminating ability of the prognostic models was assessed using Harrell's C index [38], which is an extension of the area under the receiver operating characteristic curve [39] in the case of right-censored survival data. It is calculated by looking at all usable pairs of samples that are comparable and calculating the probability of these pairs showing concordance between the ranking of the predicted failure times and that of the observed times. A C index value of 0.5 represents no discriminating ability and a value of 1.0 represents perfect discrimination. The corresponding 95% confidence intervals (95% CI) of the C index of prognostic models were obtained by bootstrapping [40].

Sensitivity, specificity, positive predictive value, negative predictive value, accuracy, and relative 95% CI were calculated at two different time points (21st and 30th day of follow-up) usually used in literature for this setting of cancer population, even if they are not ideal measures in the context of censored survival data and can be applied only when the outcome is dichotomous. For each time point there were no censored data and dichotomous outcome was defined (death/alive) using the best cutoff for each prognostic score. Statistical analyses were carried out using SAS Statistical software (version 9.1, SAS Institute, Cary, NC) and R software (http://www.r-project.org); p values of <.05 were considered statistically significant.

Results

A total of 549 patients were recruited in three Italian hospices. Median patient age was 71 years (range 18–94); 269 (49%) were male, and 280 (51%) were female. Gastrointestinal cancer was the most frequent primary tumor (37.5%), followed by respiratory (18.9%) and genitourinary (17.8%) cancers. At the time of the statistical analysis, 23 patients were still alive (censored data) and 526 had died. Median overall survival of the entire population was 22 days (95% CI = 19–24) (Figure 1). Patient distribution on the basis of presence or not of certain elements, that is, the symptoms, syndromes, and biological data that make up the four studied scores, is reported in Table 1. Of note, both PaP and D-PaP distributions were quantitatively superimposable (33% and 43.5% in group A, 40.4% and 33.4% in group B, and 26.6% and 23.1% in group C for PaP and D-PaP scores, respectively), whereas both PPS and PPI had an underrepresented risk subgroup (1.5% and 1.5%, 57.2% and 29.1%, and 41.3% and 69.4% for PPS and PPI, respectively). The last two scores thus actually subdivided the population into two rather than three groups. Survival estimates and survival curves of the four prognostic models are shown in Table 2 and Figure 2A–D. All four models identified groups with different prognoses, and log-rank tests were all highly significant (p < .0001). D-PaP and PaP were the most accurate scores, with a C index of 0.73 (95% CI = 0.71–0.74) for D-PaP and 0.72 (95% CI = 0.70–0.73) for PaP. PPI and PPS showed a slightly worse performance with C indexes of 0.62 and 0.63, respectively. Calculating the pairwise difference between the C index of our prognostic models highlighted that the performance of these two last models was only slightly lower, with the difference in discriminating accuracy being <10% with respect to that of the PaP and D-PaP scores. Accuracy in terms of sensitivity, specificity, positive predictive value, and negative predictive value at the two cutoff times chosen was high for all the scores (Table 3). The accuracy of PPS did not exceed 50% and varied from 70.3% to 88.0% in the other scores, reaching a maximum in the PaP score at 30 days with a cutoff of 5 (88.0%) and a minimum in the PPI score at 21 days with a cutoff of 5 (70.3%).

Figure 1.

Kaplan-Meyer overall survival curve for the entire population. Abbreviation: OS, overall survival.

Table 1.

Prognostic scores in palliative care populations

Abbreviations: CPS, clinical prediction of survival; D-PaP, PaP Score including delirium; KPS, Karnofsky Performance status; PaP, Palliative Prognostic Score; PPI, Palliative Prognostic Index; PPS, Palliative Performance Scale; Total WBC, total white blood count.

Table 2.

Overall survival estimates of the four prognostic scores

Abbreviations: CI, confidence intervals; D-PaP, PaP Score including delirium; n.r., not reached; OS, overall survival; PaP, Palliative Prognostic Score; PPI, Palliative Prognostic Index; PPS, Palliative Performance Scale.

Figure 2.

Overall survival curves for the low (group A), intermediate (group B), and high (group C) risk groups defined by (A) Palliative Prognostic Score (PaP Score), (B) Delirium-Palliative Prognostic Score (D-PaP Score), (C) Palliative Prognostic Index (PPI), and (D) Palliative Performance Scale (PPS). All four models discriminated well, and log-rank tests were all highly significant (p < .0001). Abbreviation: OS, overall survival.

Table 3.

Prediction accuracy of the assessed scores

^aPPS alone accuracy <50% (see text).

^bWe chose to show the best performance cutoff for each score.

Abbreviations: CI, confidence intervals; D-PaP, PaP Score including delirium; NPV, negative predictive value; PaP, Palliative Prognostic Score; PPI, Palliative Prognostic Index; PPV, positive predictive value.

The accuracy at 30 days of follow-up of CPS alone is 75.6%, 88.0% for PaP score, 79.6% for D-PaP, 72.3% for PPI, and <50% for PPS. Thus, in our experience, the accuracy of CPS alone was increased by the two versions of the PaP score.

Discussion

Several prognostic scores have been built in palliative care populations, and a number have undergone external validation [9, 14, 41]. In 2005 the Research Network of the European Association for Palliative Care published six evidence-based clinical recommendations on prognostic factors in advanced cancer, one of which strongly advocated the clinical usefulness of prognostic scores [14].

In our prospective multicenter study we compared the four main prognostic scores proposed in palliative care literature in an attempt to identify the score with the best accuracy in palliative care patients with very advanced cancer. All four scores showed a statistically significant predictive capacity, although PaP and D-PaP score would seem to identify more homogeneous subgroups in terms of survival.

Each of the four prognostic scores has its own strengths and weaknesses. The original PaP score was built for a population of Italian patients undergoing palliative care [19, 20] and subsequently validated in a wide range of cancer and non-cancer patient populations in different disease stages and settings [42–51]. Controversy has arisen regarding the use of the PaP score. First, it has been argued that the inclusion of the CPS may reduce score objectivity [52, 53]. Some physicians, especially inexperienced ones, may experience difficulty in formulating prognoses, limiting the use of the PaP score because it requires CPS. It is obvious that, while such tools must not be used as substitutes for clinical judgment, all contain an element of subjectivity. This is why it is recommended that CPS be used in combination with more objective parameters [54, 55]. These two factors are incorporated into the PaP score, which was originally built for a cancer population (excluding hematological and kidney cancers) undergoing palliative care [19, 20]. Other studies have since highlighted its efficacy in all cancer types and also in non-cancer populations [44, 45, 48]. A further criticism is that the score requires a blood sample to be taken. Although laboratory tests can be carried out as part of routine clinical practice, they are impractical when death is near or patients are reluctant. It has been seen that, in all reports on the PaP Score, blood samples are only taken as part of routine clinical practice [19, 20, 48]. It has been pointed out that no prior method of assessment or predetermined definition of the symptom involved is provided in the original papers [48, 52], whereas subsequent articles report such issues [46].

Finally, the PaP score has been criticized because it does not include the “delirium” symptom, which has proven prognostic in other studies [14, 47]. This criticism could be made of any score because all are built around a series of variables chosen by researchers and then organized on the basis of those proving significant at multivariate analysis. Initial selections may miss a critical factor, and this can thus be considered as a universal methodological shortcoming [52]. To evaluate the impact of this missing symptom on PaP score performance, we recently carried out a retrospective cohort study of 361 terminally ill cancer patients from 14 Italian palliative care centers consecutively entered in hospice programs [34], using a “validation by calibration” approach originally proposed by van Houwelingen and co-workers [56] and adapted by Miceli and colleagues [57]. The overall performance of the revised score was superimposable with that of the original score, suggesting that modification of the PaP score was not needed.

The prognostic capacity of PPS is widely accepted [22, 23, 58–63], with the literature reporting a number of cutoffs for its prognostic capability; we chose the cutoff used by Morita et al. [21]. Because of the exclusively subjective nature of the tool, some difficulties may emerge when assessing patients at higher PPS levels (the same as those of KPS) [58], and it has thus been incorporated into the PPI score [21, 55]. The PPI was built and internally validated in 1999 [21]; prognostic factors included are PPS, oral intake, edema, dyspnea at rest, and delirium. Its use in combination with clinical prediction of survival has improved the accuracy of CPS, substantially reducing the percentage of “serious errors” [55]. Successively, PPI was tested by other authors [64–67]. A point in favor of PPS and/or PPI is that they can be performed by either nurses or physicians. One limitation of PPI is that it was originally built in a Japanese population, which may lead to different results for other ethnicities.

Our study included all patients admitted to three hospices in the study period, independently of type of cancer and without any selected recruitment criteria. This approach was chosen to reduce the probability of distorting findings, and our sample can be considered representative of all palliative cancer populations. Moreover, taking into account the prospective design of the study, there are no missing data for any component of the prognostic score or for clinical outcome. One of the potential limitations of our study is the fact that the three hospices included in this validation study have been working together for years, and their medical staff are well versed in dealing with these problems. However, the fact that each of the scores has been widely validated means that results can be applied to other contexts.

In recent years, some authors have proposed an increasing number of scores as simple as possible and begun to prospectively compare the performance of different scores [24, 29, 30, 68–71]. Hyodo and colleagues built and validated the Japan Palliative Oncology Study-Prognostic Index (JPOS-PI) and compared it with the PaP Score and a simplified PPI [28]. In this preliminary study, the new score and the PaP index showed a similar performance, whereas the simplified PPI did not discriminate between low- and intermediate-risk groups. JPOS-PI is similar to the PaP score in its conception and includes CPS. The performance status variable is not present in the JPOS-PI. However, these data are congruent with a “historical” paper of ours entitled “Clinical prediction of survival is more accurate than the Karnofsky Performance Status in estimating life span of terminally ill cancer patients” [72]. Stiel et al. compared two prognostic scores, PPI and PaP [73], concluding that both yielded similar results, with a better performance in predicting poor prognosis. Tavares reported that PPS alone was less accurate than PaP and PPI, with the former being slightly more accurate than the latter but both showing problems in the intermediate prognosis group [74].

Finally, a recent paper by Gwilliam et al. [75] proposed a new score, Prognosis in Palliative Care Study, which, using a blood test, would seem to estimate survival better than either a doctor or a nurse, providing a robust rationale for treatment decision making. However, further validation is needed before it can be used in routine clinical practice.

Conclusion

It has been shown that different prognostic factors can be added to CPS, and that CPS itself can be integrated into a comprehensive prognostic tool. We can also affirm that all prognostic scores proposed in palliative care studies are highly predictive of classes of patients with different survival probabilities. The most widely used palliative care scores are the PaP score, D-PaP score, PPI, and PPS, and each has proven capable of accurately predicting different risk classes for patient survival. In our experience, all four scores showed statistically significant predictive capacity. We found that the PaP score and especially D-PaP subdivided populations into more homogeneous subgroups and showed slightly better overall accuracy than PPI or PPS in terms of C index, possibly because PaP (and its variant) includes sophisticated indicators such as CPS and blood cell count. PPS and PPI, however, demonstrated fairly high C index, only 10% lower than that of PaP. PPS and/or PPI may also be performed by either nurses or physicians. Moreover, PPI was originally built in a Japanese population, which may lead to different results in other ethnicities. Our results suggest that PaP is useful when a more accurate prognostication is needed, while the other scores can be used when a rapid and simple evaluation is sufficient.

Author Contributions

Conception/Design: Marco Maltoni, Emanuela Scarpi, Oriana Nanni

Provision of study material or patients: Marco Maltoni, Cristina Pittureri, Francesca Martini, Luigi Montanari, Elena Amaducci, Stefania Derni, Laura Fabbri, Marta Rosati

Collection and/or assembly of data: Marco Maltoni, Emanuela Scarpi

Data analysis and interpretation: Marco Maltoni, Emanuela Scarpi, Oriana Nanni

Manuscript writing: Marco Maltoni, Emanuela Scarpi, Oriana Nanni

Final approval of manuscript: Marco Maltoni, Emanuela Scarpi, Cristina Pittureri, Francesca Martini, Luigi Montanari, Elena Amaducci, Stefania Derni, Laura Fabbri, Marta Rosati, Dino Amadori, Oriana Nanni