Predicting survival of patients with spinal metastatic disease using PathFx 3.0 – A validation study of 668 patients in Sweden

Abstract

Introduction

PathFx is a computer-based prediction model for estimating survival of patients with bone metastasis. The model has been validated in several studies, but this is the first validation using exclusively patients with spinal metastases.

Research question

Is PathFx 3.0 a tool useful for predicting survival for patients with spinal metastatic disease?

Material and methods

668 patients (67% male, median age 67 years) presenting with spinal metastases at two university hospitals in Sweden 1991–2014 were included. Of those, the majority (82%, n ​= ​551) underwent surgery. Data on all patients was analyzed with PathFx version 3.0, generating a probability of survival at 1, 3, 6, 12, 18 and 24 months. The predictions were compared to real survival data and the precision in estimation was evaluated with Receiver-Operating Characteristic curve (ROC) analysis where the Area Under Curve (AUC) was calculated. Brier score and decision curve analyses were also assessed.

Results

The AUC for 1-, 3-, 6- and 12 months survival predictions were 0.64 (95% CI 0.5–0.71), 0.71 (95% CI 0.67–0.75), 0.70 (95% CI 0.66–0.77) and 0.74 (95% CI 0.70–0.78). For 18- and 24 months survival the AUC were 0.74 (95% CI 0.69–0.78) and 0.76 (95% CI 0.72–0.81). The Brier scores were all 0.23 or lower depending on the estimated survival time.

Discussion and conclusion

PathFx 3.0 is a reasonably reliable tool for predicting survival in patients with spinal metastatic disease. As the PathFx computer model can be updated to reflect advancements in oncology, we suggest this type of model, rather than rigid point-based scoring systems, to be used for estimating survival in patients with metastatic spinal disease in the future.

Highlights

• •Predictive scoring systems for spinal metastatic disease lag behind the advancements in oncology
• •PathFx 3.0 is a reasonably reliable as a tool for predicting survival for patients with spinal metastatic disease
• •AI-based tools like PathFX 3.0 are easier to keep updated than rigid point-based scoring systems

1. Introduction

Spinal metastatic disease is a complication of several common types of cancer and the incidence is rising (Torre et al., 2016). Available treatments range from supportive care to extensive spinal surgery, where the latter has a high risk of adverse events. There is strong evidence that surgery for spinal metastatic disease can improve the quality of life of the patient, but the associated risks must be put in relation to the expected survival (Patchell et al., 2005; Fehlings et al., 2016; Dea et al., 2014).

There are several point-based scoring systems available to facilitate treatment decisions for patients with spinal metastatic disease. Historically, the systems have been based on retrospective studies on cohorts treated several decades ago. Recent evaluations of older scoring systems suggest that their accuracy is questionable and that they tend to underestimate survival, as they lag behind the evolution in oncology (Carrwik et al., 2019; Hibberd and Quan, 2017; Pollner et al., 2018; Mezei et al., 2020; Tabourel et al., 2021).

This highlights the need for a less rigid prediction algorithm, reflecting recent oncological advancements. PathFx is a free online-based prediction model, developed for estimating survival of patients with pathologic fractures. By entering clinical data in a web interface, the user will get survival estimations at different time points based on data from previous patients using a statistic model.

PathFx has been validated in several studies and shows a higher degree of precision compared to other prediction models for pathologic fractures. However, the precision and usability in cohorts with spinal metastases have not been investigated. The aim of this study was to validate PathFx with a cohort treated for spinal metastatic disease and evaluate whether it is useful as a prediction tool in this population (Overmann et al., 2020; Anderson et al., 2020; Meares et al., 2019).

2. Material & methods

Two databases of patients with spinal metastatic disease from Karolinska University Hospital and Uppsala University Hospital in Sweden containing 668 patients treated 1991–2014 were merged. Most of the patients (81%) were surgically treated. Sixty-seven percent were male and the median age was 67 years. All patients from the Uppsala cohort (n ​= ​315) were treated surgically as the cases were extracted from Swespine, the national Swedish registry for spine surgery (Table 1).

Table 1.

Characteristics of included patients.

Factor	Surgery group	Non-surgery group	All
Patients included	551 (82.5%)	117 (17.5%)	668
Age (years)	67 (12)	67 (13)	67 (12)
Mean (SD)
Sex
Male	367 (66.6%)	81 (69.2%)	448 (67.1%)
Female	184 (33.4%)	36 (30.8%)	220 (32.9%)
Visceral metastases
Yes	186 (33,8%)	56 (47.9%)	242 (36.2&)
No	365 (66,2%)	61 (52.1%)	426 (63.8%)
Missing data	0	0	0
Skeletal metastases
Solitary	145 (26.3%	7 (6.0%)	152 (22.8%)
Multiple	406 (73.7%)	110 (94.0%)	516 (77.2%)
Missing data	0	0	0
Lymph node metastases
Yes	2 (0.4%)	2 (1.7%)	4 (0.6%)
No	129 (23.4%)	59 (50.4%)	188 (28.1%)
Missing data	420 (76.2%)	56 (47.9%)	476 (71.3%)
Pathologic fracture
Yes	261 (47.4%)	52 (44.4%)	313 (46.9%)
No	284 (51.5%)	64 (54.7%)	348 (52.1%)
Missing data	6 (1.1%)	1 (0.9%)	7 (1.0%)
Frankel class
A	21 (3.8%)	10 (8.5%)	31 (4.6%)
B	49 (8.9%)	11 (9.4%)	60 (9.0%)
C	241 (43.7%)	30 (25.6%)	271 (40.6%)
D	145 (26.3%)	40 (34.2%)	185 (27.7%)
E	95 (17.2%)	25 (21.4%)	120 (18.0%)
Missing data	0	1 (0.9%)	1 (0.1%)
Hemoglobin (mg/dl)
Mean (SD)	125.1 (16.8)	117.6 (18.6)	123.8 (17.4)
Missing data	22	4	26 (3.9%)

Demographic and clinical data was extracted from each patient's record. The data includes age, sex, primary oncologic diagnosis, number of bone metastases (solitary or multiple), presence or absence of lymph node metastases, presence or absence of visceral metastases, levels of hemoglobin, absolute lymphocyte count and Eastern Cooperative Oncology Group (ECOG) performance status. The neurologic function was reported according to the Frankel scale A-E, where grade A means complete neurologic injury with no motor or sensory function below the level of injury and grade E means normal function. The Regional Ethical Review Board in Uppsala (ref 2012/133) and the Regional Ethical Review Board in Stockholm (ref 2012/272–31/4 and 2019–06189) approved the study.

The data was entered into a spreadsheet and uploaded to PathFX 3.0 as a batch from file and all the parameters were used in the analysis. Using PathFX 3.0, six Bayesian belief networks designed to estimate the likelihood of survival for each patient after 1, 3, 6, 12, 18 and 24 months were created using the bnlearn package in R Version 3.5.1 (R Foundation for Statistical Computing, Vienna, Austria). The Bayesian analysis is a statistical method that facilitates data-driven learning to estimate likelihood of an outcome based on the observed data. The estimations were then compared to true survival data from the medical records and the precision was assessed by several methods.

The discriminatory ability of each estimation of survival was evaluated by receiver operating characteristic (ROC) curve analysis. A ROC area under the curve (AUC) of minimum 0.7 was considered as an acceptable predictive value. The accuracy of each prediction model was assessed by calculating the Brier score, a statistical method where lower score means higher accuracy for the prediction model. Calibration curves plotting the expected outcome and the observed outcome for each time estimation were calculated as well.

To evaluate the clinical usefulness of the predictions, a decision curve analysis was performed for every estimation. The decision curve analysis assesses whether the estimation by PathFx 3.0 is useful in a clinical setting, compared to the estimation that all or none of the patients will survive past the given time point in each estimation.

3. Results

The AUC in the survival estimations varies from 0.64 (1 month survival) to 0.76 (24 months survival), with a tendency to higher AUC for longer survival estimations (Fig. 1, Fig. 2, Fig. 3, Fig. 4, Fig. 5, Fig. 6).

Fig. 1.

Reciever operator characteristic (ROC) curve for external validation with 1 month model.

Fig. 2.

Reciever operator characteristic (ROC) curve for external validation with 3 month model.

Fig. 3.

Reciever operator characteristic (ROC) curve for external validation with 6 month model.

Fig. 4.

Reciever operator characteristic (ROC) curve for external validation with 12 month model.

Fig. 5.

Reciever operator characteristic (ROC) curve for external validation with 18 month model.

Fig. 6.

Reciever operator characteristic (ROC) curve for external validation with 24 month model.

The calibration curves follow the same pattern, with low precision in the 1-month survival estimation. For estimations of 12 months and above, the estimations by PathFx tend to overestimate rather than underestimate survival (Fig. 7, Fig. 8, Fig. 9, Fig. 10, Fig. 11, Fig. 12).

Fig. 7.

Calibration curve for the 1 month survival model.

Fig. 8.

Calibration curve for the 3 month survival model.

Fig. 9.

Calibration curve for the 6 month survival model.

Fig. 10.

Calibration curve for the 12 month survival model.

Fig. 11.

Calibration curve for the 18 month survival model.

Fig. 12.

Calibration curve for the 24 month survival model.

Decision curve analyses at different time points show that PathFx 3.0 performed better than a dichotomous model assuming that all patients should be deceased or alive at a given time point, except for the prediction of survival after one month (Fig. 13, Fig. 14, Fig. 15, Fig. 16, Fig. 17, Fig. 18).

Fig. 13.

Decision Curve Analysis (DCA) for the 1 month survival model.

Fig. 14.

Decision Curve Analysis (DCA) for the 3 month survival model.

Fig. 15.

Decision Curve Analysis (DCA) for the 6 month survival model.

Fig. 16.

Decision Curve Analysis (DCA) for the 12 month survival model.

Fig. 17.

Decision Curve Analysis (DCA) for the 18 month survival model.

Fig. 18.

Decision Curve Analysis (DCA) for the 24 month survival model.

The Brier scores ranged from 0.09 to 0.23 depending on the estimations, where the lowest score was seen in the 1-month predictions (Table 2).

Table 2.

Brier scores for different survival estimates.

Estimated survival, months	Brier score (95% CI)
1	0.09 (0.08–0.11)
3	0.20 (0.18–0.22)
6	0.22 (0.21–0.25)
12	0.21 (0.19–0.22)
18	0.20 (0.19–0.22)
24	0.17 (0.15–0.19)

4. Discussion

The range of available treatments for patients with spinal metastatic disease highlights the need for reliable prognostication. The ideal prognostic model should eliminate the risk of overtreating as well as undertreating the patient and be adjustable to the oncological advancements.

This is the first validation study of PathFx only including patients with spinal metastatic disease and the model shows good reliability, with exception for very short expected survival. The results are in line with previous studies including patients with other bone metastasis, not only in the spine (Overmann et al., 2020; Anderson et al., 2020; Forsberg et al., 2017). A part of this cohort has previously been validated with older scoring systems (Tomita, Tokuhashi, Revised Tokuhashi and Bauer scores) showing similar results (Carrwik et al., 2019). As opposed to the scoring systems evaluated in the previous study, PathFx 3.0 has a tendency to overestimate rather than underestimate survival, especially in survival estimations of 12 months and longer.

Precision for estimated one month survival is the lowest among the tested time frames, with an AUC of 0.64. A possible explanation is the low number of patients deceased within one month after treatment, making the training set limited. Furthermore, the study population does not represent a normal cohort of patients with spinal metastatic disease, since there is a high number of patients treated surgically, indicating an assumed better prognosis. Somewhat contradictory, the Brier Score for the one month survival estimation group is the lowest, which illustrates the weakness of the Brier Score when forecasting rare events and makes it less suitable for measuring accuracy in clinical decisions (Assel et al., 2017).

Further studies are needed to compare PathFx 3.0 to other scoring systems using the same methodology. A similar method using ROC curve analysis was used by Ahmed et al. in a study evaluating nine predictive scoring systems in a cohort of 176 patients treated surgically due to spinal metastatic disease (Ahmed et al., 2018). In that study, six out of nine scoring systems had a higher AUC at one month predicted survival compared to PathFX 3.0 in our study. However, at three months (90 days) predicted survival, none of the nine scoring systems had a higher AUC than 0.70 while PathFX 3.0 had an AUC of 0.71 ​at the same time point in our material.

This study has several limitations. The combined dataset includes both patients treated surgically and non-surgically, but the dataset from Uppsala University Hospital contains only surgically treated patients. Furthermore, all cases are from tertiary referral centres offering multidisciplinary treatments, which means the results may not be applicable in other settings such as countries with lower economic possibilities.

We will never see the perfect prediction tool with 100% accuracy, but we believe that the open-source construction of PathFx 3.0 in combination with the ability to validate the prediction models with machine learning is the way forward, rather than older rigid point-based scoring system.

Declaration of competing interest

The authors declare the following financial interests/personal relationships which may be considered as potential competing interests: Rikard Wedin reports a relationship with Prognostix AB, the company hosting the PathFx web site.