Comparing the Accuracy of Seven Scoring Systems in Predicting Survival of Lung Cancer Patients With Spinal Metastases

Study Design.

Retrospective case series.

Objective.

To investigate the accuracy of seven scoring systems for the prediction of survival in lung cancer patients with spinal metastases (SPM).

Summary of Background Data.

Although survival scoring systems have been developed for surgical decision-making, the reliability and validity of these models are unclear for specific cancer types. As the prevalence of patients with lung cancer increases, it is imperative to determine the accuracy of these models for lung cancer patients with SPM.

Materials and Methods.

This is a retrospective study of a cohort of lung cancer patients with SPM who underwent spine surgery between 2019 and 2021 at two centers. The optimal area under the curve (AUC) was calculated to evaluate the accuracy of seven candidate scoring systems at 3, 6, and 12 months. Calibration and decision curve analysis was used for further validation.

Results.

A total of 166 patients (mean age: 58.98±10.94; 105 males and 61 females) with SPM were included. The median postoperative survival was 12.87±0.93 months. The modified Bauer score, revised Tokuhashi score, Linden score, Tomita score, the Skeletal Oncology Research Group nomogram, and the New England Spinal Metastasis Score in prediction survival at 3, 6, and 12 months showed a slightly weaker AUC (range 0.464–0.659). The AUC of the Katagiri-New score in predicting 1-year survival for lung cancer patients was the highest (0.708; range 0.619–0.798). The decision curve analysis showed that the Katagiri-New score led to a greater net benefit than the strategies of changing management for all patients or none of the patients.

Conclusions.

This study suggests that the most commonly used models have limitations in predicting survival in patients undergoing spinal surgery for metastatic lung cancer and underestimate survival. In this sample of lung cancer patients, the Katagiri-New Scoring system score had the best performance in predicting 1-year survival.

Level of Evidence.

4.

Over 50% of patients with end-stage cancer develop skeletal metastases, and the spine is the most common location for metastases.^1,2 Lung cancer represents a common primary source of spinal metastases (SPM)³, and lung cancer remains the major cause of cancer-related mortality globally (18.4% of cancer-related deaths).^4,5

To guide an appropriate surgical intervention, several scoring systems have been developed for the prediction of survival following SPM. These include the revised Tokuhashi (2005),⁶ Tomita Score,⁷ Katagiri-New score,⁸ New England Spinal Metastasis Score (NESMS),⁹ modified Baur,¹⁰ van der Linden,¹¹ and Skeletal Oncology Research Group (SORG) nomogram.¹² The revised Tokuhashi score and Tomita score are the most widely used scoring systems, and both of them consider lung cancer as one of the poorest primary tumors in terms of survival.¹³ In 2014, Katagiri et al.⁸ classified lung cancer into suitable and unsuitable cases for targeted therapy. Suitable cases are typically moderately aggressive tumors, whereas unsuitable cases are considered extremely aggressive and fast-growing tumors. In 2015, Ghori et al.⁹ added walking function and serum albumin level based on Bauer’s revised score and developed the NESMS for patients with skeletal metastasis. In 2010, Schwab’s group used machine learning to develop a prediction model for the 1-year and 90-day mortality in SPM, which was incorporated into an open Internet platform¹² and demonstrated outstanding predictive performance.¹⁴

However, the development of treatment for lung cancer has been advancing rapidly in recent decades. For targeted therapy, tyrosine kinase inhibitors have been used extensively, improving survival and clinical outcome significantly.¹⁵ The importance of the EGFR mutation has been reported as well.¹⁶ In addition, PD-1/PD-L1 inhibitors have shown longer progression-free survival benefits in untreated stage III/IV lung cancer patients, either in combination or monotherapy.^17,18 The application of osteoprotective agents, including bisphosphonates and denosumab, was reported to increase survival in patients with SPM.^19–21 Furthermore, spondylectomy of the spinal tumor enhanced functional outcomes with an effective reconstruction of spinal stability.²² Combinations of these advanced therapies have led to significantly increased survival for advanced lung cancer.

Although these models have been reported to achieve significant accuracy,^{6,8,9,12,23,24} the reliability and accuracy of these scoring systems have not been externally validated in large cohorts of patients with lung cancer. We conducted a retrospective study of lung cancer patients with SPM who underwent spinal procedures at two Chinese centers. The present study investigated survival, and the optimal area under the curve (AUC), and confirmed whether available prognostic scoring systems can accurately predict the life expectancy of lung cancer patients for surgical guidance.

MATERIALS AND METHODS

Study Population

The ethical committee of Guangdong Provincial People’s Hospital and The First Affiliated Hospital of Guangzhou Medical University approved the research protocol. All of the participants gave informed consent. In this retrospective study, we collected clinical data on patients with SPM who underwent spinal procedures in the Department of Orthopedics across two centers between January 1, 2019 and March 31, 2022.

Inclusion criteria included the following: (1) Patients diagnosed with lung cancer with SPM who have symptoms of spinal cord compression, neurological impairment, and pain; or patients with spinal instability according to the Spinal Instability Neoplastic Score (SINS); (2) The primary lesion or SPM lesion was pathologically confirmed as lung cancer; and (3) Patients who were in the good physical condition and without serious comorbidities, and who were willing to undergo surgery.

Exclusion criteria included: (1) Missing/incomplete clinical information and follow-up data; (2) Patients who died unexpectedly from surgical complications or other nontumor causes.

Data Collection and Follow-up

Data were retrospectively abstracted to obtain information on the primary tumor, age, sex, neurological deficits, visceral or brain metastasis, surgery procedures, number and segment of SPM, Epidural Spinal Cord Compression, treatment modalities (chemotherapy, targeted therapy, and immunotherapy), surgical segments, pathologic types, Eastern Cooperative Oncology Group scale, Frankel grade, and so on. All patients were followed up every 3 months by telephone or outpatient visits; the time of death was recorded in all cases. Overall survival time for patients with SPM is computed from the operation to the date of death or final follow-up. The last follow-up was on March 31, 2022. Each patient was examined using the seven distinct scoring systems: Tomita, revised Tokuhashi, Van der Linden, Katagiri-New, modified Bauer, NESMS, and SORG nomogram.

Statistical Analysis

The statistical analyses were conducted using SPSS 22.0 (IBM Corp, Armonk, NY, USA). Kaplan–Meier plot and Cox regression were used for survival analysis. An analysis of receiver operating characteristic (ROC) curves was performed to establish AUC at three months, six months, and one year. The DeLong test was used to compare differences between the various AUCs. P<0.05 indicated statistical significance. Generally, an AUC between 0.7 and 0.8 indicated that the model helped assess survival prognosis; a poor model has <0.7 and an excellent prediction model has >0.8.

A subgroup analysis was conducted by stratifying the patients by way of surgery performed (eg, decompression and instrumentation surgery, minimally invasive surgery (MIS)). Univariable Cox regression, calibration (calibration plot, intercept, and calibration slope), decision curve analysis (DCA),²⁵ and AUC were performed to assess the results. Statistical analysis was performed using R Language software (version 3.6.2).

RESULTS

The demographic and clinical characteristics are summarized in Table 1. A total of 166 patients with a mean age of 57.98±10.94 years underwent assessment. Among them, 152 (91.6%) had nonsmall cell lung cancer and 14 (8.4%) had small cell lung cancer. The study involved spinal cord decompression and segmental instrumentation in 133 and minimally invasive surgery in 33. Of a total of 166 patients in our study, 85 (51.2%) patients had received preoperative or postoperative chemotherapy, 94 (56.6%) were treated with targeted therapy, and 36 (21.6%) had immunotherapy. The postoperative median survival for all patients with confirmed SPM was 12.87±0.93 months (Fig. 1). The 3-month survival rate was 88.4% (n=144), the 6-month survival rate was 73.7% (n=106), and the 12-month survival rate was 58.8% (n=65). Of a total of 166 patients in our study, 85(51.2%) had received preoperative or postoperative chemotherapy, 94 (56.6%) were treated with targeted therapy, and 36 (21.6%) had immunotherapy.

TABLE 1.

Characteristics of the 166 Lung Cancer Patients With Spinal Metastasis

Characteristics	Values
Age (±SD, y)	58.98±10.94
Sex
Male	105 (63.3)
Female	61 (36.7)
Pathologic type
nonsmall cell lung cancer	152 (91.6)
small cell lung cancer	14 (8.4)
No. spinal metastases
<3	69 (41.6)
≥3	97 (58.4)
Visceral metastasis
No	53 (31.9)
Yes	113 (68.1)
Spinal metastases at first clinical diagnosis
No	75 (45.2)
Yes	91 (54.8)
Brain metastasis
No	113 (68.1)
Yes	53 (31.9)
Adjuvant treatment
preoperative or postoperative chemotherapy	85 (51.2)
targeted therapy	94 (56.6)
immunotherapy	36 (21.7)
Median postoperative survival±SD, mo	12.87±0.93

Numbers in parentheses refer to percentages.

Figure 1.

Kaplan–Meier cumulative survival rate for the 166 patients. The 3-month survival rate is 88.4% (n=144), the 6-month survival rate is 73.7% (n=106), and the 12-month survival rate is 58.8% (n=65).

The survival analysis of candidate scoring systems is shown in Figure 2, and details of scoring results for each subgroup are presented in Table 2. In addition, according to the SORG nomogram, the mean SORG score was 268.6 (range: 137.4–406.6), and the 30-day, 90-day, and 1-year median survival probability was 85.0%, 71.5%, and 39.0%, respectively.

Figure 2.

Kaplan-Meier curves (log-rank test) of our series for the prognostic subgroups of the six scoring systems. A, Tomita score (P=0.002); B, Revised Tokuhashi score (P=0.008); C, New Katagiri score (P<0.001); D, Modified Bauer score (P=0.191); E, Van der Linden score (P=0.006); and F, NESMS (P<0.001). Note: Cum survival, cumulative survival. NESMS indicates New England Spinal Metastasis Score.

TABLE 2.

Median Survival Time in Different Scoring Systems and AUC Based on ROC Analysis at 3, 6, and 12 Months

Subgroup of Scoring Systems	n	Median OS (mo)	AUC at 3 mo	AUC at 6 mo	AUC at 12 mo
Tomita	—	—	0.620 (0.486, 0.755)	0.568 (0.463, 0.672)	0.585 (0.486, 0.684)
4–5 (good)	10	20.7±3.5*	—	—	—
6–7 (moderate)	59	22.9±9.9	—	—	—
8–10 (poor)	97	12.9±0.9	—	—	—
Revised Tokuhashi	—	—	0.637 (0.518, 0.756)	0.665 (0.574, 0.757)	0.642 (0.546, 0.737)
0–8 (poor)	151	12.5±0.8	—	—	—
9–11 (moderate)	15	27.3±2.8*	—	—	—
Katagiri-New	—	—	0.640 (0.511, 0.769)	0.638 (0.543, 0.734)	0.708 (0.619, 0.798)
0–3 (good)	7	25.1±2.9*	—	—	—
4–6 (moderate)	112	14.6±4.9	—	—	—
7–10(poor)	47	7.1±0.7	—	—	—
NESMS	—	—	0.593 (0.456, 0.730)	0.619 (0.516, 0.722)	0.652 (0.557, 0.747)
0(worse)	19	6.5±3.6	—	—	—
1(poor)	50	10.4±2.7	—	—	—
2(moderate)	97	26.1±2.2*	—	—	—
Modified Bauer	—	—	0.575 (0.447, 0.703)	0.571 (0.470, 0.671)	0.591 (0.493, 0.690)
0–1(poor)	156	12.6±0.9	—	—	—
2(moderate)	10	20.7±3.6*	—	—	—
Linden	—	—	0.659 (0.553, 0.785)	0.642 (0.542, 0.742)	0.608 (0.511, 0.706)
0-3(poor)	130	11.8±1.0	—	—	—
4(moderate)	36	33.8±9.8	—	—	—
SORG	166	12.9±0.9	0.639 (0.513, 0.764)	None	0.634 (0.537, 0.730)

Note: values are given as the AUC (95% confidence interval [CI]).

^*Top asterisks () represent the mean overall survival.

AUC indicates area under the curve; OS, overall survival; ROC, receiver operating characteristic.

All AUC-related data are presented in Table 2 and Figure 3. The AUC of the modified Bauer Score at about 3-month survival was the lowest and poorest (0.575; Fig. 3A), and the Linden score had the best performance (0.659). According to the ROC analysis of half-year survival (Fig. 3B), the accuracy of the Linden score was still the highest and reached 0.642, while that of the Tomita score was the poorest (0.568). However, ROC at 1-year postoperative survival, the Katagiri-New score presented an AUC of 0.708, indicating that the model helped assess survival prognosis.

Figure 3.

Time-dependent ROC curves of survival scoring systems in lung cancer are as follows. A, The ROC curves for the seven prognostic spinal scoring systems at 3 months. The Linden score had the best performance (AUC: 0.659). B, The ROC curves for six prognostic scores at 6 months. The revised Tokuhashi had the best performance (AUC: 0.665). C, The ROC curves for seven prognostic scores at 12 months. The Katagiri-New score had the best performance (AUC: 0.708). AUC indicates area under the curve; ROC, receiver operating characteristic.

The DCA showed (Fig. 4) that the Katagiri-New score led to a greater net benefit than the strategies of changing management for all patients or none of the patients.

Figure 4.

Decision curve analysis with net benefit by threshold probability for 12-month survival.

Subgroup Analysis

Univariable Cox regression analysis showed no statistically significant difference in the two surgical subgroups in one-year survival (HR 0.51; 95% CI 0.25–1.07; P=0.08 for MIS). When stratified by surgery, the Katagiri-New score had the best performance (AUC=0.72; Table 3) in the decompression group and showed similar accuracy (AUC=0.68) in the MIS group. The SORG nomogram is the most accurate scoring system (AUC=0.82) in the MIS group. However, DCA in the MIS group showed that SORG did not provide a good net benefit. The strategy using this model for none of the patients achieved better outcomes than using it for any patient. Moreover, the calibration intercept is −0.06 [95%CI: −0.147-0.025], and the calibration slope is 1.7 [95%CI: −1.517-1.907]. Furthermore, the observations in the MIS group show very low calibration (Supplemental digital content, http://links.lww.com/BRS/C51).

TABLE 3.

The Subgroup Analysis of Surgical Procedures at 12-month AUC

	Decompression (n=133)	MIS(n=33)
Tomita	0.59 (0.48, 0.70)	0.58 (0.34, 0.82)
Tokuhashi	0.64 (0.54, 0.75)	0.57 (0.32, 0.82)
Katagiri-New	0.72 (0.62, 0.82)	0.68 (0.44, 0.91)
NESMS	0.63 (0.53, 0.74)	0.72 (0.49, 0.95)
Modified Baur	0.58 (0.47, 0.69)	0.66 (0.42, 0.89)
Linden	0.60 (0.49, 0.71)	0.57 (0.33, 0.81)
SORG	0.58 (0.34, 0.82)	0.82 (0.59, 1.00)

Note: AUC indicates area under the curve.

Values are given as the AUC (95% confidence interval [CI]).

DISCUSSION

To the best of our knowledge, this is the first study focusing on a specific cancer type to evaluate the most commonly used prognostic scores in a large sample of lung cancer patients with SPM, including the revised Tokuhashi score, Linden score, modified Bauer score, Tomita score, Katagiri-New score, NESMS, and SORG nomogram.^{6–9,12,20,23,26} This information can help surgeons select the optimal tool to stratify patients, predict survival, and guide surgical decision-making in the setting of lung cancer. Our study demonstrated that:(1) the Katagiri-New score had the best performance (AUC=0.708) in predicting 1-year survival for lung cancer; (2) except for the above score, other scoring systems performed poorly (AUC<0.7) in predicting survival.

Although most of these models can classify patients into prognostic groups that have a significant difference in survival, they cannot identify patients with comparable accuracy. The revised Tokuhashi and Tomita scores gained popularity in clinical decision-making; both of them rated lung cancer as a rapidly growing tumor. The revised Tokuhashi score is often considered the most accurate predictive scoring system for SPM from many primary tumors.^9,23,27,28 Lee et al.²⁴ found an accuracy of 0.66 and 0.64 in lung cancer when performing a systematic review and meta-analysis of the two scores. This result is consistent with our findings. Therefore, it is reasonable to believe that both scoring systems would underestimate the lung cancer patient’s survival.

The Katagiri-New score incorporated some abnormal laboratory indicators and had an AUC>0.7 in predicting survival at both 6 and 12 months.⁸ This scoring system enrolled only 59 (7.9%) patients with surgical intervention and included prior chemotherapy as an indicator, which is still a controversial factor.²⁹ In this research, we found that the accuracy of the 1-year survival rate was (0.712 [0.602–0.822]; P<0.05). A possible reason is that this score categorized lung cancer as a moderate or fast-growing tumor in the context of molecularly targeted drugs.

In 2015, the NESMS proposed by American scholars included a revised version of the Bauer score, walking function, serum albumin, and other indicators, and an external validation (161 patients) found 80% accuracy in 1-year survival.⁹ However, the NESMS still had poor performance (AUC: 0.667) at predicting 1-year survival after surgery. That report is consistent with the results of our external validation.

The original Bauer score, first published in 1995, was introduced to address not only spinal but also extremity metastases.³⁰ The modified Bauer score reported by Leithner et al.¹⁰ excluded scoring for pathologic fractures and was able to distinguish between poor, moderate, and good prognoses. In our series, the accuracy of the modified Bauer score was 0.559, 0.554, and 0.559, respectively, indicating its limitations in predicting survival for patients with lung cancer. The reason for this could be at the time of design; the original study only included 6 (2.5%) lung cancer patients with SPM and only 35 (13.8%) cases in the revised version.^30,31 Furthermore, there were limited treatments for lung cancer at the time the system was developed.

Schwab et al.³² reported that they created a classic scoring nomogram (SORG) that was more reliable at predicting survival time, with an accuracy of 0.76 (30 d), 0.74 (90 d), and 0.77 (365 d). Smeijers et al.¹⁴ concluded that the SORG nomogram enjoyed a superior performance in survival prediction for surgery in spinal metastases. Although the external validation of the SORG nomogram indicated a superior predictive ability,^33–35 the nomogram performed poorly in estimating survival with an accuracy of 0.618 for 90 days, and 0.621 for 365 days in the present study.

Advanced therapies have increased the survival of patients with SPM. More evidence suggests that spinal surgery does bring better outcomes, with an improvement in quality of life, addressing the neurologic deficit, stabilizing the spine, and pain relief.^36–39 Most scoring systems focus on 3-, 6- and 12-month survival as time points. At these points, a trade-off must be made between the risk of complications and the benefit. In general, clinicians believe that anticipated 3-month survival is still a threshold for candidacy for surgery.^36,40 For the group with survival < 3 months, major surgery requiring weeks or even months of recovery is not suitable, and the risks outweigh the benefit. The surgical complication rate (commonly over 20%) increases with the scope and complexity of the surgical intervention.^41,42 Our results suggest that most of the scores significantly underestimate the survival time of patients with lung cancer, which may result in an incorrect decision. In cases where the expected OS is between 3 and 12 months, an individualized and multi-disciplinary consideration should be taken into the survival benefits for patients with lung cancer.

In our series, the median survival time was 12.87±0.93 months (95%CI: 11.0–14.7); the 6-month survival rate was 73.7%, and the 12-month survival rate was 58.8%. This is consistent with other survival times from comparable studies.^13,43

Differentiations in genetics, type of cancer, and demographics among distinct populations may influence the validity of the predictive models. In our study, the included patients were all from China. The Chinese population may be more homogeneous. In addition, most patients in China are concentrated in large tertiary general hospitals rather than being limited to a specialist hospital in a particular region due to the country’s health care system. Therefore, our work may be reflective of the typical lung cancer patient with SPM in China.

Our study has several inherent limitations. First, all patient data were extracted from medical records, and there is the possibility of incomplete data or inaccurate information. Second, all patients enrolled in this study were offered spinal surgery, introducing a selection bias. Third, the lack of long-term follow-up is recognized as another limitation.

CONCLUSIONS

In summary, we assessed the accuracy of seven scoring systems in lung cancer patients with SPM. This study suggests that the most commonly used prediction models have limitations in prognosticating the survival of patients undergoing spinal surgery for metastatic lung cancer and underestimate survival. In our study, the Katagiri-New system score had the best performance in predicting 1-year survival.

Key Points

• The study suggests that the most commonly used prediction models have limitations in predicting the survival of lung cancer patients with spinal metastases.

• The Katagiri-New Scoring system score had the best performance in predicting 1-year survival for lung cancer patients.

• Future development of new prognostic models should take into account advances in cancer treatment.

Supplementary Material

brs-48-1009-s001.docx ^{(102.1KB, docx)}