External validation and adaptation of a dynamic prediction model for patients with high‐grade extremity soft tissue sarcoma

Abstract

Background and Objectives

A dynamic prediction model for patients with soft tissue sarcoma of the extremities was previously developed to predict updated overall survival probabilities throughout patient follow‐up. This study updates and externally validates the dynamic model.

Methods

Data from 3826 patients with high‐grade extremity soft tissue sarcoma, treated surgically with curative intent were used to update the dynamic PERsonalised SARcoma Care (PERSARC) model. Patients were added to the model development cohort and grade was included in the model. External validation was performed with data from 1111 patients treated at a single tertiary center.

Results

Calibration plots show good model calibration. Dynamic C‐indices suggest that the model can discriminate between high‐ and low‐risk patients. The dynamic C‐indices at 0, 1, 2, 3, 4, and 5 years after surgery were equal to 0.697, 0.790, 0.822, 0.818, 0.812, and 0.827, respectively.

Conclusion

Results from the external validation show that the dynamic PERSARC model is reliable in predicting the probability of surviving an additional 5 years from a specific prediction time point during follow‐up. The model combines patient‐, treatment‐specific and time‐dependent variables such as local recurrence and distant metastasis to provide accurate survival predictions throughout follow‐up and is available through the PERSARC app.

Acronyms

CI

confidence interval

DM

distant metastasis

DOS

dynamic overall survival

eSTS

extremity soft tissue sarcomas

HR

hazard ratio

LR

local recurrence

OS

overall survival

PERSARC

PERsonalised SARcoma Care

1. INTRODUCTION

Extremity soft tissue sarcomas (eSTS) not only represent a wide variety of histological subtypes, sizes, and grades but also affect patients of all age groups. This reflects the clear and substantial differences in their clinical course and prognosis. ¹ As treatment protocols differ for specific patients between institutes and countries, several prognostic prediction models for overall survival (OS) and local recurrence (LR) have been developed. ² , ³ , ⁴ , ⁵ , ⁶ , ⁷ , ⁸ However, these models are designed to estimate prognosis at the time of treatment or diagnosis and do not take new events that occur during treatment and follow‐up into account. In addition, they do not account for possible time‐varying effects of baseline risk factors.

A dynamic prediction model for patients with eSTS was therefore developed, the dynamic PERsonalised SARcoma Care (PERSARC) model, to predict the probability of surviving an additional 5 years from a prediction time point during follow‐up. ⁹ Before the introduction of the dynamic PERSARC model, prediction models for eSTS patients were limited to predictions from baseline, for example, time of surgery or diagnosis. ² , ³ , ⁴ , ⁵ , ⁶ , ⁷ , ⁸ The dynamic PESARC model uses updated patient information such as the occurrence of LR and distant metastasis (DM) which become available during follow‐up, to update predictions over time. Additionally, it accounts for the time‐varying effects of histology subtype and surgical margin on survival. The dynamic model has been internally validated through the use of cross‐validation, but so far, no external validation has been performed. As the original publication on dynamic PERSARC did not account for the grade, the model is updated to meet current clinical demands and improve possibilities for implementation.

The aim of this study was to update and improve the existing dynamic prediction model as well as to validate it using a large external data set. The model was adapted in two ways: (1) new patients were added to the model development cohort, and (2) the grade of disease was included in the model.

2. MATERIALS AND METHODS

2.1. Study design

In this study, the original dynamic prediction model developed by Rueten‐Budde et al. ⁹ was updated and externally validated, using a retrospectively collected cohort of patients with eSTS. The model development data were augmented for the update and contained data from Leiden University Medical Center, Royal Orthopaedic Hospital, Netherlands Cancer Institute, Mount Sinai Hospital, the Norwegian Radium Hospital, Aarhus University Hospital, Skåne University Hospital, Medical University Graz, Royal Marsden Hospital, Erasmus MC Cancer Institute, Radboud University Medical Center, University Medical Center Groningen, Haukeland University Hospital, Helios Klinikum Berlin‐Buch, MedUni Vienna, Vienna General Hospital, and the EORTC trial 62931, a randomized controlled trial which studied the effect of intensive adjuvant chemotherapy on several outcome measures.

External data were provided by Istituto Nazionale dei Tumori. For both, the model development and external cohort data were collected from centers between January 1, 2000, and December 31, 2014. Data from the EORTC trial 62931, which is part of the development cohort, were collected between February 1995, and December 2003.

The outcome of interest was OS, defined as the time from surgery to death due to any cause or last recorded follow‐up. The dynamic model predicts 5‐year dynamic overall survival (DOS) from a particular prediction time point during follow‐up. For example, at 1‐year postsurgery, the model predicts the probability of surviving an additional 5 years (therefore until 6‐year postsurgery). To determine the predictive performance of the model, calibration and discrimination were evaluated with the external data set. Ethical approval for this study was waived by the Institutional Review Board CME (G16.022), because clinical data were collected from medical records and were pseudo‐anonymized.

2.2. Patients and variables

Selection and exclusion criteria were identical for the model development cohort and the external cohort. ⁹ All patients were selected from the sarcoma registry based on histological diagnosis from each hospital. Histologically, tumors were classified according to the WHO's criteria ¹ and patients were grouped into eight categories. Included eSTS subtypes included high‐grade (FNCLCC Grades II and III ¹⁰ ) angiosarcoma, malignant peripheral nerve sheath tumor, synovial sarcoma, spindle cell sarcoma, myxofibrosarcoma, liposarcoma, leiomyosarcoma, malignant fibrous histiocytoma/undifferentiated pleomorphic sarcoma, (pleomorphic) soft tissue sarcomas not‐otherwise‐specified, epithelioid sarcoma, clear cell sarcoma, rhabdomyosarcoma (adult form), conventional fibrosarcoma, giant cell sarcoma, malignant granular cell tumor, unclassified soft tissue sarcoma, and undifferentiated sarcoma.

Patients were excluded if they were initially treated without curative intent, presented with LR or DM, had Kaposi's or rhabdomyosarcoma (pediatric form), had tumor in their abdomen, thorax, head, or neck, or received isolated limp perfusion as (neo‐) adjuvant treatment.

Three types of risk factors were included in the dynamic model. Patient‐specific predictors assessed at baseline: age (years), tumor size by the largest diameter measured at pathological examination (cm), tumor depth in relation to investing fascia (deep/superficial), grade (II/III), and histological subtype according to the WHO classification. ¹ Treatment‐related predictors measured at baseline: radiotherapy ((neo)adjuvant/no radiotherapy), surgical margin categorized according to the categorical R‐system, “R0” for negative margin and “R1–2” for a positive margin with tumor cells in the inked surface of the resection margin. ¹¹ Risk factors measured during follow‐up: LR defined as the presence of pathological and/or radiologically confirmed tumor at the site where it was originally detected, more than 2 months after primary surgery and DM defined as radiological evidence of systemic spread of tumor distant from the primary tumor site.

The original dynamic prediction model was based on 2232 patients. ⁹ For the revised model, additional data were collected resulting in 3826 patients for the development of the updated dynamic model. For external validation, 1111 patients were considered.

2.3. Statistical analysis

The dynamic prediction model developed in Rueten‐Budde et al. ⁹ was revised by adding more patients and the variable grade to the model. The prediction model was based on landmark methodology. Technical details about landmark models for dynamic prediction are provided in van Houwelingen and Putter. ¹² Additionally, the association between chemotherapy and survival was investigated.

The predictive ability of the updated model was assessed in terms of calibration and discrimination using an external data set. Model discrimination refers to how well the model is able to discriminate between high‐ and low‐risk patients; dynamic C‐indices ¹² were computed to evaluate the performance of the model. A C‐index equal to one corresponds to perfect discrimination and a C‐index of 0.5 means that the model predicts just as well as flipping a coin. ¹³ Model calibration on the external data refers to how well predicted and observed survival probabilities have similar values and was assessed with yearly calibration plots.

Calibration plots visualize calibration at a particular prediction time point (e.g., 1‐year postsurgery). Patients at risk at a specific time were divided into eight prognostic groups based on their predicted survival. This means that the dynamic model was used to predict 5‐year DOS for patients in the external data set and based on these probabilities risk groups were made. Five years after the prediction time point (e.g., 6‐year postsurgery), the observed survival probabilities of the risk groups were estimated by applying Kaplan–Meier's method. In the calibration, each point represents a risk group. If the points lay on the diagonal (x = y), predicted and observed survival are the same, implying that the predictions for the risk groups were perfect. The arbitrary choice for the number of risk groups was made based on the number of patients at risk over time; initially, 1111 patients were at risk, however, 5 years after surgery only 529 patients remain in the risk set. To have a reasonable number of patients per risk group even at 5‐year postsurgery, eight risk groups were chosen.

The items on the checklist of the Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis (TRIPOD) were considered during model development. ¹⁴ Statistical analyses were performed in the R‐software environment. ¹⁵ All p values were derived from a two‐sided test and p < .05 was considered significant.

3. RESULTS

The model was developed on a cohort of 3826 patients with median follow‐up equal to 6.00 years (95% confidence interval [CI] = 5.86–6.18), assessed with the reverse Kaplan–Meier method. ¹⁶ The external validation cohort consisted of 1111 patients with a median follow‐up equal to 6.89 years (95% CI = 6.47–7.61). Table 1 provides a summary of the patient characteristics for the cohort used to develop the dynamic model and the external cohort.

Table 1.

Patient demographics for the two cohorts used to develop and to validate the model

Characteristics	Development	External
Total	3826	1111
Age, mean (SD)	59.40 (18.10)	55.46 (17.03)
Gender (%)
Female	1680 (43.9)	504 (45.4)
Male	2011 (52.6)	607 (54.6)
Unknown	135 (3.5)	0 (0.0)
Tumor size in cm, mean (SD)	9.04 (5.77)	8.33 (5.66)
Margin (%)
R1–2	515 (13.5)	142 (12.8)
R0	3028 (79.1)	969 (87.2)
Unknown	283 (7.4)	0 (0.0)
Histology (%)
Myxofibrosarcoma	689 (18.0)	197 (17.7)
MPNST	261 (6.8)	60 (5.4)
Synovial sarcoma	411 (10.7)	122 (11.0)
MFH/UPS and NOS	1204 (31.5)	202 (18.2)
Spindle cell	191 (5.0)	0 (0.0)
LMS	368 (9.6)	150 (13.5)
LPS	388 (10.1)	167 (15.0)
Other	314 (8.2)	213 (19.2)
Tumor depth (%)
Deep	2493 (65.2)	802 (72.2)
Superficial	912 (23.8)	309 (27.8)
Unknown	421 (11.0)	0 (0.0)
Grade
2	639 (16.7)	432 (38.9)
3	3111 (81.3)	679 (61.1)
Unknown	76 (2.0)	0 (0.0)
Radiotherapy (%)
No radiotherapy	1331 (34.8)	474 (42.7)
Neoadjuvant	517 (13.5)	138 (12.4)
Adjuvant	1878 (49.1)	499 (44.9)
Unknown	100 (2.6)	0 (0.0)
Chemotherapy (%)
No	3189 (83.4)	739 (66.5)
Yes	470 (12.3)	372 (33.5)
Unknown	167 (4.4)	0 (0.0)

Note: Histology “Other”, angiosarcoma, malignant rhabdoid tumor, alveolar soft part sarcoma, epithelioid sarcoma, clear cell sarcoma, embryonal rhabdomyosarcoma, rhabdomyosarcoma (adult form), giant cell sarcoma, malignant granular cell tumor, conventional fibrosarcoma, unclassified soft tissue sarcoma, and undifferentiated sarcoma; Tumor depth: relative to the investing fascia.

Abbreviations: cm, centimeters; LMS, leiomyosarcoma; LPS, liposarcoma; MFH/UPS, malignant fibrous histiocytoma/undifferentiated pleomorphic sarcoma; MPNST, malignant peripheral nerve sheath tumor; sarcoma – NOS, (pleomorphic) soft tissue sarcomas not‐otherwise‐specified; SD, standard deviation.

Figure 1 shows Kaplan–Meier survival curves for both development and external cohort.

Figure 1.

Kaplan–Meier curves for development and external cohort [Color figure can be viewed at wileyonlinelibrary.com]

An overview of the number of patients at risk in the development and external data set is given in Figure 2 together with information about the disease status. In the development cohort, in total 1602 patients died, 241 patients developed LR, 949 DM, and 385 developed both. In the external cohort, 306 patients died, 70 had LR, 279 DM, and 77 developed both.

Figure 2.

Number of patients at risk in development and external data set, respectively. Red: patients with local recurrence (LR) and distant metastasis (DM); pink: patients with DM; green: patients with LR; blue: patients without LR or DM [Color figure can be viewed at wileyonlinelibrary.com]

Table 2 shows hazard ratios (HRs) together with 95% CI for the risk factors included in the revised dynamic model. Age and tumor size are both modeled by a linear and a quadratic term (age in steps of 10 years and size in steps of 1 cm). This means that the HRs consist of two components: the linear (HR_lin) and the quadratic effect (HR_quad). For example, for the risk factor age, the HR of an 80‐year‐old compared to a 60‐year‐old patient (reference) is equal to

$$H{R}_{\mathit{lin}}^{\mathit{step}}\times H{R}_{\mathit{quad}}^{\mathit{ste}{p}^2}={1.366}^2\times {1.052}^4=2.285$$

where “step” corresponds to the age difference between the two patients in units of 10 years.

Table 2.

Dynamic prediction model for overall survival: hazard ratio (HR) along with 95% confidence interval (n = 3826)

	HR	95% CI	p
Covariates with time‐constant effects
Age (ref: 60 years, per 10 years)
Age	1.366	1.304–1.431	<.001
Age2	1.052	1.028–1.076	<.001
Tumor size (ref: 0 cm, per 1 cm)
Size	1.158	1.116–1.202	<.001
Size2	0.996	0.995–0.998	<.001
Tumor depth (superficial vs. deep)	0.790	0.673–0.927	.004
Grade (3 vs. 2)	1.425	1.174–1.730	<.001
Radiotherapy (RT)
No RT	1
Neoadjuvant	0.719	0.583–0.886	.002
Adjuvant	0.818	0.716–0.936	.003
Local recurrence (yes vs. no)	2.232	1.892–2.634	<.001
Distant metastasis (yes vs. no)	6.446	5.662–7.338	<.001
Covariates with time‐varying effects
Prediction time (ref: time of surgery, per year)
t p	0.507	0.415–0.621	<.001
t p 2	1.095	1.050–1.141	<.001
Histology
Constant
Myxofibrosarcoma	1
MPNST	2.132	1.633–2.783	<.001
Synovial sarcoma	1.458	1.145–1.856	.002
MFH/UPS and NOS	1.207	1.004–1.452	.045
Spindle cell	1.396	1.054–1.848	.020
LMS	1.065	0.819–1.386	.638
LPS	0.915	0.706–1.185	.501
Other	1.419	1.095–1.841	.008
Linear time‐varying effect
Myxofibrosarcoma	1
MPNST	0.845	0.669–1.068	.159
Synovial sarcoma	1.261	1.037–1.534	.020
MFH/UPS and NOS	1.002	0.851–1.179	.981
Spindle cell	1.058	0.824–1.357	.660
LMS	1.166	0.941–1.444	.160
LPS	1.010	0.812–1.256	.929
Other	0.863	0.663–1.124	.276
Quadratic time‐varying effect
Myxofibrosarcoma	1
MPNST	1.000	0.947–1.056	1.000
Synovial sarcoma	0.939	0.897–0.983	.007
MFH/UPS and NOS	1.009	0.976–1.044	.585
Spindle cell	0.972	0.906–1.043	.434
LMS	0.989	0.946–1.034	.636
LPS	1.011	0.967–1.058	.622
Other	1.019	0.963–1.078	.510
Margin
Constant
R0 versus R1–2	0.827	0.698–0.981	.029
Linear time‐varying effect
R0 versus R1–2	1.334	1.114–1.597	.002
Quadratic time‐varying effect
R0 versus R1–2	0.954	0.918–0.990	.014

Note: Histology “Other”, angiosarcoma, malignant rhabdoid tumor, alveolar soft part sarcoma, epithelioid sarcoma, clear cell sarcoma, embryonal rhabdomyosarcoma, rhabdomyosarcoma (adult form), giant cell sarcoma, malignant granular cell tumor, conventional fibrosarcoma, unclassified soft tissue sarcoma, and undifferentiated sarcoma; Tumor depth: relative to the investing fascia.

Abbreviations: CI, confidence interval; HR, hazard ratio; LMS, leiomyosarcoma; LPS, liposarcoma; MFH/UPS, malignant fibrous histiocytoma/undifferentiated pleomorphic sarcoma; MPNST, malignant peripheral nerve sheet tumor; sarcoma – NOS, (pleomorphic) soft tissue sarcomas not‐otherwise‐specified; tp, prediction time points.

Surgical margin and histology subtype are modeled as time‐varying variables, which means that the effect on the outcome changes over time. For example, the HR 1‐year postoperative for a patient with R0 margin compared to an R1–2 margin is equal to

$$\begin{array}{ll}\mathit{HR}=& [\mathit{\text{constant}}\times {(\mathit{\text{linear time}}-\mathit{\text{va rying effect}})}^{t_p}\\ \hspace{1em}& \times {\mathit{quadratic}\mathit{time}-\mathit{\text{varying effect}}}^{{t_p}^2}]=0.827\times 1.334\\ \hspace{1em}& \times 0.954=1.052\end{array}$$

where t_p = 1 and t_p ² = 1. The HR changes from 0.827 at the time of surgery to 1.052 a year later. The model shows that the effect of surgical margin changes from being protective at surgery time to having no effect on survival after 1 year.

In a preliminary analysis, the association of risk factors to the outcome of chemotherapy treatment (yes (neoadjuvant or adjuvant) vs. no) was evaluated. Most baseline risk factors showed a significant association (age, tumor size, depth, histology, radiotherapy, and grade). Country of treatment was significantly associated with chemotherapy treatment. This means that, correcting for the other risk factors (age, tumor size, depth, margin, histology, radiotherapy, and grade) in the model, countries had different approaches in giving chemotherapy treatment. The association of chemotherapy to survival was investigated by including this risk factor in the dynamic model and no significant effect was found (chemotherapy yes vs. no; HR = 1.131; 95% CI = 0.946–1.352; p = .178). Chemotherapy was therefore not included in the updated dynamic prediction model.

The quality of the model can be assessed from the calibration plots (Figure 3A–F). Each point in the plot represents a risk group; the figure shows they are relatively close to the diagonal line implying that predictions are accurate. Figure 3 also suggests that the model generally slightly underestimated survival.

Figure 3.

Calibration plots for predictions of 5‐year DOS from 0‐, 1‐, 2‐, 3‐, 4‐, and 5‐year postsurgery. DOS, dynamic overall survival [Color figure can be viewed at wileyonlinelibrary.com]

The discriminative ability of the model was assessed with dynamic C‐indices, with values equal to 0.697, 0.790, 0.822, 0.818, 0.812, and 0.827 at 0, 1, 2, 3, 4, and 5 years after surgery respectively. High values for the C‐indices are due to the strong predictive value of DM on survival. A patient who experiences DM has a worse prognosis compared to a patient without DM.

4. DISCUSSION AND CONCLUSIONS

The previously developed dynamic prediction model has been updated and successfully externally validated. The sample size of the model development cohort was increased from 2232 to 3826 patients and the risk factor grade was added to the updated model. ⁹ The model can estimate the probability of surviving an additional 5 years from a prediction time point during follow‐up. It can be used from the time of surgery up until 5‐year postsurgery for patients with high‐grade eSTS treated with curative intent.

Even though calibration plots showed that predicted survival was close to observed survival the model generally underestimated survival in the external cohort. Kaplan–Meier curves estimated for the development and external cohort indicate that the external cohort had better survival. There are several possible reasons for the underestimation of survival: the effect of risk factors might be different in the development cohort compared to the external cohort, or patients might differ in terms of an unobserved covariate which might affect survival and cannot be taken into account.

The association of chemotherapy with survival is controversial, and its indication greatly depends on other risk factors. When added to the dynamic model, chemotherapy showed no significant association with survival.

The updated dynamic prediction models is implemented in the updated PERSARC application; available for free at the Apple Store and Google Play Store.

PERSARC Study Group

Will Aston, Han Bonenkamp, Dario Callegaro, P. D. Sander Dijkstra, Peter C. Ferguson, Anthony M. Griffin, Alessandro Gronchi, Dirk J. Grünhagen, Rick L. Haas, Andrew Hayes, Lee M. Jeys, Johnny Keller, Minna K. Laitinen, Andreas Leithner, Katja Maretty‐Kongstad, Rob Pollock, Florian Posch, Myles Smith, Maria A. Smolle, Emelie Styring, Per‐Ulf Tunn, Jos A. van der Hage, Robert J. van Ginkel, Winan J. van Houdt, Kees Verhoef, Madeleine Willegger, Julie J. Willeumier, Reinard Windhager, Jay S. Wunder, Olga Zaikova.

CONFLICT OF INTERESTS

Authors Anja J. Rueten‐Budde, Veroniek M. van Praag, and Marta Fiocco have nothing to disclose. Author Michiel A. J. van de Sande reports grants from Daiichi Sankyo, outside the submitted work.

SYNOPSIS

A dynamic prediction model for soft tissue sarcoma was externally validated. It takes into account the evolution of the disease after surgery and predicts dynamic overall survival later on in follow‐up.

Rueten‐Budde AJ, van Praag VM, van de Sande MAJ, Fiocco M. External validation and adaptation of a dynamic prediction model for patients with high‐grade extremity soft tissue sarcoma. J Surg Oncol. 2021;123:1050–1056. 10.1002/jso.26337