External validation of a prognostic score predicting overall survival for patients with brain metastases based on extracranial factors

Highlights

• •We provide an external validation of the extracranial prognostic score developed by Nieder et al.
• •This extracranial score could effectively define the patients with very short survival.
• •A new combination of extracranial score with DS-GPA-score achieved an even better discrimination

Abstract

Purpose

The aim of our study was an external validation of the extracranial prognostic score predicting survival of patients with brain metastases receiving cranial irradiation on data from a single institution.

Materials and methods

A retrospective analysis of 524 patients with brain metastases treated with cranial radiotherapy in a single tertiary center was performed. Three predictive scores were calculated and assessed for their ability to discriminate prognostic groups: (i) The Recursive Partitioning Analysis (RPA) score (available for 524 patients); (ii) the Diagnosis-Specific Graded Prognostic Assessment (DS-GPA) score (464 patients); (iii) the extracranial score (EC-S) developed by Nieder et al. which is based on serum albumin, lactate dehydrogenase (LDH) and the number of extracranial organs involved (157 patients). Discrimination of each score was assessed by Gönen & Heller’s concordance probability estimate (CPE). The calibration was checked by comparing median survival estimates of each risk group with the corresponding values of the datasets from which the scores were derived. Finally, a multivariable Cox regression model was built by using the least absolute shrinkage and selection operator on a large number of variables including all three scores.

Results

With a CPE = 0.626 ± 0.022, the EC-S had the best discriminatory power. The EC-S also appeared to be better calibrated and had the best ability to separate patients with a very poor prognosis: patients with combination of low albumin, elevated LDH and more than 1 extracranial organ with metastatic involvement had a median survival time of only 0.6 months (CI95% 0.1–1.1) and a hazard ratio for death of 6.36 (2.67–15.14) compared to patients with no extracranial metastases and normal levels of albumin and LDH. In the multivariable Cox model serum albumin, LDH, treatment modality, DS-GPA and EC-S were retained as prognostic factors. An ad hoc combination of both DS-GPA and EC-S into a new score was possible for 134 patients and indicated a slightly better discrimination (CPE = 0.636 ± 0.023) than either DS-GPA or EC-S alone.

Conclusions

This study provides an independent validation of the prognostic EC-S which was the best prognostic model for defining the patients who obviously did not benefit from radiation therapy of brain metastases in terms of overall survival. The combination of the EC-S with the established DS-GPA score resulted in a slight increase in discriminatory ability. The new EC-GPA score needs further validation in larger patient cohorts.

1. Introduction

The development of appropriate patient selection criteria for tumor specific treatment including chemo- and radiotherapy (RT) is the cornerstone of modern precision oncology. With regard to life-threatening conditions such as brain-disseminated cancer it is widely believed and anticipated that aggressive antitumor treatment should be started as soon as possible to prolong survival and maintain quality of life. However, it has long been recognized that some patients will not derive any profit from active treatment while other will do. In order to discriminate those patients potentially benefitting from antitumor therapy, several prognostic scores such as the RTOG Recursive Partitioning Analysis (RPA) [1] score or the Graded Prognostic Assessment (GPA) score [2] were developed and validated [3], [4]. Although offering a possibility to separate the survival curves, i.e., identifying patient groups with better and worse prognosis, these models have not sufficient predictive ability for deciding which patients will not derive sufficient benefit from brain RT, which is important within the context of counselling patients about their prognosis and treatment options. It is known that several widely available and cheap blood tests such as albumin and lactate dehydrogenase (LDH) can be used as surrogate parameters in survival prediction [5], [6]. The number of extracranial organ systems involved has also been found to be an independent and highly significant predictor of overall survival [7], [8]. The combination of those three parameters was used by Nieder et al. to develop a simple extracranial score (EC-S) as a reasonable addition to the aforementioned intracranial prognostic factors helping to identify the patients with brain metastases with very bad prognosis in whom best supportive care could be the best choice [9]. The main goal of this study was an independent validation of the EC-S as a possible tool for predicting very limited survival and to compare it with the RPA and disease-specific GPA (DS-GPA) scores.

2. Materials and methods

We extracted all available patient records referring to a diagnosis of brain metastases from the clinical RT software Mosaiq© from the Department of Radiation Oncology at the University hospital Wuerzburg. Overall survival (OS) data from 524 patients treated between 04.02.2008 and 08.11.2017 were available. Brain MRI was obligatory as a part of staging for patients with primarily non-metastasized lung cancer and malignant melanoma. In other cases the clinical symptoms of brain metastases prompted cranial imaging. In our clinic the vast majority of tumor patients receive their treatment recommendation after discussion in interdisciplinary tumor boards. Further diagnostic work-up was performed according to the advice of the tumor board and based basically on national guidelines. Prognostic scores for each patient were determined as originally described [1], [2]. For calculating the EC-S one point was counted for each elevated LDH, decreased albumin and more than one extracranial site of metastatic involvement, so that the final score ranged from 0 to 3 (3 indicating the worst prognosis) [6].

Elevated LDH was defined as above 250 U/l, and decreased albumin was defined as below 3.5 g/l according to the normal levels of the local laboratory. LDH and albumin measurements were only considered if taken within 2 weeks before the first fraction of RT. Since both blood tests are not mandatory in our radiotherapy department, only 157 out of 524 extracted patient records contained information on all 3 extracranial prognostic parameters. In contrast, the RPA score could be computed for all, and DS-GPA score for 466 patients. Only 134 cases had both DS-GPA and EC-S available. An overview of all relevant patient characteristics is given in Table 1.

Table 1.

Patient characteristics (n = 524). Abbreviations used in the table: renal cell carcinoma (RCC), Karnofsky performance score (KPS), whole brain radiotherapy (WBRT), radiotherapy (RT).

Parameter	N		%
	(whole)	EC-S available	(whole)	EC-S available
Gender
Female	242	64	46	41
Male	282	93	54	59
Age (years)	Median 63	Median 63	Range (20–92)	Range (21–86)
Primary
Lung	270	87	51	55
Breast	66	7	13	5
RCC	22	7	4	5
Melanoma	74	34	14	22
Head and neck	6	0	1	0
GI	37	7	7	5
CUP	19	6	4	4
other	30	9	6	6
KPS
KPS >70%	228	70	43	44
KPS ≤70%	297	87	57	56
RPA class
1	133	34	25	22
2	262	86	50	55
3	129	37	25	24
DS-GPA class
1	190	61	41	43
2	154	38	33	27
3	92	26	20	18
4	28	16	6	11
EC-S
0	15		10
1	69		44
2	60		38
3	13		8
Molecular target with therapeutic relevance present
Yes	97	30	19	19
No	424	126	81	81
Primary controlled
Controlled	305	83	58	53
Not controlled	219	74	42	47
Number of brain metastases
One	143	40	27	26
Two or three	107	34	20	22
Multiple	274	83	52	53
Extracranial metastases
No	101	23	19	15
Single organ	152	40	29	26
Multiple	271	94	52	60
Treatment modality
Surgery + adjuvant RT or stereotactic radiosurgery	103	51	20	32
WBRT and others	421	106	93	68

Actuarial survival from the first day of whole-brain (WB) or other RT was calculated using the Kaplan-Meier method and compared between different groups with the Log-rank test. Discrimination of each score was assessed by Gönen & Heller’s concordance probability estimate (CPE) for the Cox model [10]. The concordance probability is a general measure of discriminatory power of a nonlinear statistical model, with a probability of 0.5 indicating random discrimination and 1 perfect discrimination. The calibration (external validity) was checked by comparing median survival estimates of each risk group with the corresponding values of the datasets from which the scores were derived [11]. To determine the most important prognostic factors in multivariable analysis, Cox regression was used. The following covariates from Table 1 were judged as putatively important prognostic factors: treatment modality (surgery + adjuvant RT or stereotactic radiosurgery/WBRT and others), age (<65/≥65 years as used in the RPA score), gender, baseline Karnofsky performance score (KPS; ≤70/>70), primary tumor type (8 strata in total), presence of a molecular target with therapeutic relevance (yes/no), the number of extracranial organs involved (0/1/>1), number of brain metastases (1/2 or 3/>3), serum albumin (normal/decreased), LDH (normal/elevated), RPA score, DS-GPA score and EC-S. Those variables were available for 140 patients of which 119 had died. Given the large number of variables compared to the number of events, we conducted variable selection using the LASSO method which shrinks regression coefficients of less important variables to 0 and typically yields lower estimation variance than stepwise selection methods [12]. The optimal penalty parameter λ was determined based on 10-fold cross validation and used for determining the most important predictor variables. These selected variables were then used to build a new predictive model. We adhered to the TRIPOD criteria to assure the transparence of our data presentation and analysis [13]. Statistical analysis was performed with IBM-SPSS-25© and R version 3.5.0.

3. Results

Most patients in the whole cohort had multiple brain metastases (52%) and multiple extracranial metastases (52%) (Table 1). The median KPS was 80, range 30–100. The most frequent primary tumor was lung cancer (51.3%), followed by malignant melanoma (14.3%) and breast cancer (12.7%). Albumin and LDH measurements prior to RT were available for 165 and 260 patients respectively, and 157 patients had both proteins measured. 132 events were registered in this latter group and 25 cases were censored. We used the DS-GPA (available for 464 patients with 410 events and 54 censored cases) and RPA scores (available for 524 patients with 464 events and 60 censored cases) as established reference to compare with EC-S. The results are summarized in Table 2, and Fig. 1 and Fig. 2 show the Kaplan-Meier survival curves for patients stratified according to the DS-GPA score and EC-S, respectively. Significant survival differences between all groups within each prognostic score were seen except for groups 0 and 1 of the EC-S (p = 0.974) which had similar median survival estimates (Table 2 and Fig. 2). However, the EC-S had the best discriminatory power as judged by Gönen & Heller’s CPE. The EC-s was also the best score for discriminating patients with a particularly poor prognosis, since its worst prognostic class possessed the largest hazard ratio (6.36) compared to the most favorable class. A comparison between the median survival estimates of our cohort and those of the datasets from which the different scores had been derived revealed differences between the survival predictions for the derivation datasets and our data. In this respect, the predictions for classes 0, 2 and 4 of the EC-S appeared to be the best calibrated.

Table 2.

Results concerning the calibration and discrimination of the three scores applied to our dataset. Gönen & Heller’s CPE is an estimate of the concordance probability of the Cox models. Abbreviations used in the table:Confidence interval (CI), hazard ratio (HR), overall survival (OS).

	RPA			DS-GPA				EC-S
	1	2	3	3.5–4.0	2.5–3.0	1.5–2.0	0–1.0	0	1	2	3
N	133	262	129	28	92	154	190	15	69	60	13
# events	108	232	124	17	75	135	183	9	56	54	13
Median OS	8.38	5.16	1.77	12.91	7.85	7.36	2.14	7.9	8.8	2.0	0.6
Median OS 95% CI	6.80–10.41	3.91–6.70	1.38–2.27	9.56-NA	6.74–14.26	5.16–9.36	1.81–2.86	3.9–12	5.7–10.9	1.4–3.9	0.1–1.1
HR	1	1.47	2.86	1	1.64	2.26	4.12	1	1.05	2.31	6.36
HR SE		0.12	1.14		0.27	0.26	0.26		0.36	0.36	0.44
HR 95% CI		1.16–1.85	2.19–3.73		0.97–2.78	1.36–3.75	2.50–6.81		0.52–2.14	1.14–4.69	2.67–15.14
Median OS in derivation dataset	7.1	4.2	2.3	16.7 (14.7–18.8)	9.6 (8.7–10.6)	5.4 (4.9–5.9)	3.1 (2.8–3.5)	9.0	3.5	2.3	0.7
Gönen & Heller’s CPE	0.5938 ± 0.0116			0.6110 ± 0.0123				0.6258 ± 0.0220

Fig. 1.

Actuarial Kaplan-Meier survival plot for patients with available DS-GPA-class (n = 466). Differences between groups significant in pairwise comparison (Log rank, p < 0.05).

Fig. 2.

Actuarial Kaplan-Meier survival plot for patients with available EC-S (n = 157). Difference between all but groups 0 and 1 were significant in pairwise comparison (Log rank, p < 0.05).

Using the LASSO method to build a multivariable prognostic Cox model from our own data (140 patients and 119 events), the following variables were selected: Treatment modality, Albumin, LDH, DS-GPA score and EC-S. Using 5- or 20-fold instead of 10-fold cross validation for finding the optimal LASSO penalty parameter did not change this variable selection result. The regression coefficients of the final model fitted with these variables are given in Table 3. The CPE of the final Cox model was 0.7230 ± 0.0212, indicating a significant increase in discriminatory power compared to every score on its own (Table 2).

Table 3.

Prognostic factors and regression coefficients in the final Cox model obtained after LASSO variable selection.

	Coefficient	Hazard ratio	p-Value
Albumin <3.5 g/l	0.677 ± 0.403	1.97	0.093
LDH <250 U/l	−0.072 ± 0.570	0.93	0.900
Treatment: Whole brain RT and no (radio-)surgery	0.448 ± 0.246	1.57	0.068
DS-GPA: 2.5–3.0	1.295 ± 0.488	3.65	0.008
DS-GPA: 1.5–2.0	1.590 ± 0.483	4.90	0.001
DS-GPA: 0.5–1.0	2.181 ± 0.509	8.86	1.8 × 10−5
EC-S: 1	−0.733 ± 0.449	0.48	0.1979
EC-S: 2	0.438 ± 0.724	1.55	0.545
EC-S: 3	0.297 ± 0.996	1.35	0.766

Given the selection of both DS-GPA and EC-S into the final Cox model, we heuristically combined both scores into a new “EC-GPA” score with 4 categories (Table 4). Stratification of the 134 patients for which the new EC-GPA combination could be calculated resulted in a clear separation of the survival curves (Fig. 3). The hazard ratios of EC-GPA classes 1, 2 and 3 compared to class 0 were 4.84 (95% CI 1.42–16.46), 9.26 (2.83–29.97) and 31.64 (8.36–119.76). Furthermore, with a CPE = 0.6355 ± 0.0230 the discrimination between the four prognostic groups was slightly increased compared to the EC-S alone (CPE = 0.6258 ± 0.0220).

Table 4.

Calculation of the combined EC-DS-GPA score.

DS-GPA	EC-S
	0	1	2	3
1	2	2	2	3
2	2	2	2	3
3	1	1	1	3
4	0	0	1	3

Fig. 3.

Actuarial Kaplan-Meier survival plot for patients with ES-GPA-score (n = 134). Difference between all groups significant in pairwise comparison (Log rank, p < 0.05).

4. Discussion

A problem of proper patient and/or therapy selection has not lost its importance since the beginning of local therapy of brain metastasis. Selecting patients who have a chance to derive any benefit from antitumor treatment is a prerequisite for omitting unneeded treatment of those who have very limited survival prognosis.

Some discouraging results of more aggressive local treatment of brain metastases [14], [15] challenged clinicians and statisticians. Several methodologies on the way to prognostic tools for discrimination of patient populations benefiting from more aggressive treatment were developed [2], [16] and validated [17], [18], [19].

Despite the validation of the RPA score we believe that it is not helpful in answering any of the above questions. Sperduto et al. modified the original GPA score obviously due to the need to better stratify patients with brain metastases. The DS-GPA and finally Lung-molGPA [20] and Melanoma-molGPA [21] scores were developed and externally validated [19]. Nevertheless, even these most recent scores have limited ability of reliably predicting individual patient prognosis, sometimes classifying several long term survivors into the group with the worst prognosis and vice versa [21].

The role of extracranial factors such as performance status, extent of extracranial metastases or control of the primary tumor provided some additional valuable information about an individual’s prognosis [22]. A further refinement of the individual prognosis within the group of patients with very limited survival was possible after inclusion of such widely available and cheap biochemical surrogate parameters such as LDH and albumin [6].

The application of the EC-S to our data yielded similar results as in the original derivation study by Nieder et al. [6]. First, with the exception of the fairly good prognosis class 1, the EC-S appeared well calibrated with median OS differences between our data and the derivation data not larger than 1.1 months (Table 2). Second, the EC-S had the highest discriminatory power as judged by Gönen & Heller’s CPE. Third, the EC-S performed better than the RPA or DS-GPA score in separating the group of patients with very poor prognosis. However, in contrast to the DS-GPA score, the EC-S was not able to separate the two groups of patients with a good and fairly good prognosis in our data. This indicates some miscalibration of the model underlying the EC-S for patients with more favorable prognoses, so that survival predictions for new patients are not necessarily reliable.

In building a multivariable prognostic Cox regression model on our dataset, the RPA score was not selected as a prognostic factor, while both the DS-GPA and EC-S were. Due to these findings we pursued the idea of combining these two scores. We built 4 classes in the collective of 134 patients with both known DS-GPA and EC-S as displayed in Table 4. Application of the new combined score (EC-GPA) resulted in a clear separation of the survival curves (Fig. 3) and yielded slightly better discrimination between prognostic groups than either the DS-GPA or EC-S alone, although the CPEs of the EC-S and EC-GPA scores overlap within their uncertainties due to the small sample size. We believe that, although acquired heuristically, the EC-GPA score combination could be a good prognostic tool which should be evaluated in future studies using larger combined or independent datasets.

It is obvious that the paradigm of reserving stereotactic radiosurgery for treating patients with the most favorable prognoses changed over time. Low toxicity and wide availability of stereotactic radiosurgery led to its more frequent application and omission of WBRT [23] despite still controversial evidence for a clinical benefit and clear concerns from detailed analysis of available randomized trials [24], [25]. Prognostic scores were not analyzed in the EORTC 22952–26001 trial, and only the RPA-score was used in the trial of Yamamoto et al. The stratification according to GPA score in the secondary analysis of Aoyama et al. demonstrated clear benefit of WBRT in combination with stereotactic radiosurgery in the group with best prognosis also in terms of OS.

Since the publication of the QUARTZ-trial there is a good level of evidence that in preselected patients the use of very hypofractionated WBRT has a limited effect on OS and quality of life [26]. Due to several limitations of the trial such as intention-to-treat analysis (ca. 20% of patients in the WBRT group did not receive WBRT), a low treatment dose of 20 Gy, an obvious negative selection of patients in both groups and more aggressive antitumor therapy in the control arm there are still some questions about its practice changing role. Despite the main conclusion of this trial, younger patients did derive a clear benefit from WBRT in terms of overall survival. GPA was not a significant variable in the survival analysis, probably because of the limited number of patients with a high score and limited statistical power to address this research question.

Our study shows that incorporation of extracranial factors into a prognostic model significantly improves discriminatory power. Nevertheless, our study has some limitations due to its retrospective nature and limited sample size, especially for the subset of patients for which the EC-S could be calculated. Furthermore, it was not planned a priori to develop a new prognostic score – the combination of DS-GPA and EC-S was rather a data-driven heuristic approach for improving the discrimination of various patient groups with distinctly different prognoses. Due to the limited data quality associated with the retrospective nature of this study, the ad-hoc definition of the combined EC-GPA score should be seen as a limitation. We prefer to consider the EC-GPA score as a hypothetical possible way for further refinement of both the DS-GPA and EC-S that should however be evaluated in future studies.

5. Conclusions

In conclusion, our study provides an independent validation of the prognostic EC-S developed by Nieder et al. [6]. Their prognostic model which is based solely on extracranial factors appeared to have a higher external validity than both the RPA and DS-GPA scores when applied to the unselected patients from our hospital. It was also the best prognostic model for defining the patients who obviously did not benefit from RT of brain metastases at least in terms of OS. The combination of the EC-S with the established DS-GPA score resulted in a slight gain of discriminatory ability. Further validation of the EC-S and the new EC-GPA score will be pursued in an ongoing analysis of combined patient cohorts from different tertiary care centers.

Conflict of interest

None.