Multi-institutional Development and External Validation of a Nomogram to Predict Recurrence After Curative Resection of Pancreatic Neuroendocrine Tumors

Abstract

Objective:

To develop a nomogram estimating the probability of recurrence free at 5 years after resection for localized grade 1 (G1)/ grade 2 (G2) pancreatic neuroendocrine tumors (PanNETs).

Background:

Among patients undergoing resection of PanNETs, approximately 17% experience recurrence. It is not established which patients are at risk, with no consensus on optimal follow-up.

Method:

A multi-institutional database of patients with G1/G2 PanNETs treated at 2 institutions was used to develop a nomogram estimating the rate of freedom from recurrence at 5 years after curative resection. A second cohort of patients from 3 additional institutions was used to validate the nomogram. Prognostic factors were assessed by univariate analysis using Cox regression model. The nomogram was internally validated using bootstrap resampling method and on the external cohort. Performance was assessed by concordance index (c-index) and a calibration curve.

Results:

The nomogram was constructed using a cohort of 632 patients. Overall, 68% of PanNETs were G1, the median follow-up was 51 months, and we observed 74 recurrences. Variables included in the nomogram were the number of positive nodes, tumor diameter, Ki-67, and vascular/perineural invasion. The model bias-corrected c-index from the internal validation was 0.85, which was higher than European Neuroendocrine Tumors Society/American Joint Committee on Cancer 8th staging scheme (c-index 0.76, P=<0.001). On the external cohort of 328 patients, the nomogram c-index was 0.84 (95% confidence interval 0.79–0.88).

Conclusion:

Our externally validated nomogram predicts the probability of recurrence-free survival at 5 years after PanNETs curative resection, with improved accuracy over current staging systems. Estimating individual recurrence risk will guide the development of personalized surveillance programs after surgery.

The incidence and prevalence of pancreatic neuroendocrine tumors (PanNETs) has increased during the past decade, and currently, PanNETs represent the second most frequent indication for pancreatic surgery.¹ Surgical resection is the first line of treatment for patients with localized PanNETs, resulting in cure in 70% to 90% of cases.^2–4 Almost 95% of resected well-differentiated PanNETs are grade 1 (G1) or grade 2 (G2) tumors4 exhibiting a Ki-67 labeling index <20%.⁵ These tumors are characterized by a heterogeneous risk of recurrence, depending on several clinical and pathological factors. It has not been well established which patients are at significant risk of recurrence, and therefore there is no consensus on the optimal follow-up with wide variations in surveillance protocols between institutions.^6–8 Currently, both the American Joint Committee on Cancer (AJCC) and the European Neuroendocrine Tumors Society (ENETS) staging systems stratify localized G1/G2 tumors according to the tumor node metastasis (TNM) system. However, assessing the likelihood of recurrence by these approaches for patients with PanNETs can be inaccurate as they rely on only the extent of the disease, whereas other grade-related factors that contribute to the risk of recurrence are ignored.⁹ Given the significant heterogeneity of grade-related factors in G1 and G2 tumors, wide variations in recurrence risk could be accounted for by these factors which are often not included.

Nomograms are statistical predictive models that use a simple graphical representation to estimate the individualized risk of a clinical event and have recently emerged as an accurate tool to estimate prognosis in oncology.^9–11 Compared with the traditional staging system, they also allow incorporation of continuous variables proven to be prognostic, rather than a less informative broad cut-off. Over the years, other nomograms for PanNETs have been proposed, however, with only minor advantages over the conventional staging systems, and with no clear impact on clinical practice.^12–15

In this study, we sought to develop, and externally validate, a new model that accurately predicts the individual risk of recurrence after curative resection of localized G1/G2 PanNETs. We constructed a nomogram using data from multiple high-volume institutions and we then compared the predictive ability of this nomogram over the current staging systems. Predicting the risk of recurrence offers the potential to improve personalized surveillance schedules, determine clinical trial eligibility, and compare results across studies and different institutions.

METHODS

Patients and Data Collection

This study was approved by a waiver of authorization from each of the 5 participating organizations’ Institutional Review Boards. Prospectively maintained databases at Memorial Sloan Kettering (MSK) (New York, NY, USA), Verona University Hospital (VUH) (Verona, Italy), Johns Hopkins Hospital (JHH) (Baltimore, MD, USA), Glasgow Royal Infirmary (GRI) (Glasgow, UK), and Royal North Shore Hospital (RNSH) (Sydney, Australia) were queried for patients who underwent resection for G1 or G2 PanNETs between 2000 and 2016. Patients with a familial syndrome, evidence of metastatic disease, residual R2 disease, postoperative mortality, lack of Ki-67 labeling index on pathology report, and those receiving neoadjuvant or adjuvant therapy were excluded from the study. For the purpose of the study, with regard to the pathologic nodal status, we considered “N0”: patients who had lymph node removal and no metastatic nodes, and those who had an Nx status after parenchyma-spearing resection, such as enucleation and central pancreatectomy. Resected PanNETs were then classified according to the WHO grading system and staged according to the ENETS/AJCC 8th staging system specific for well-differentiated neoplasms.^16,17

Statistical Analysis

Disease and treatment characteristics were summarized using median and range for continuous variables, and frequency and percentages for categorical variables. Time to recurrence (TTR) was calculated from the date of curative surgery until the date of the first recurrence and estimated using Kaplan-Meier methods. Recurrence was identified through routine computed tomography (CT) scans at 6 months after surgery and then every year from the first follow-up. Patients who died without a recurrence (n = 29) were censored at the date of death. A Cox proportional-hazards model was used to study the association between possible risk factors and recurrence.

Nomogram Construction

The nomogram was constructed based on patients treated at MSK and VUH (n = 632). Recognizing that this was a slow growing disease and we observed 74 recurrence at the time of study, our ability to construct a complex model was limited because there should be 10 to 15 events per covariate in the model to avoid the risk of overfitting.¹⁸ Variables significantly associated with TTR from univariate analysis at P < 0.05 were entered into the regression model and the possible prognostic factors were identified based on examining the results from best subsets regression¹⁹ and according to the clinical judgment and to a possible cause-effect relationship. Positive lymph nodes, Ki-67, lesion size, R status, vascular invasion, and perineural invasion were selected as the potential candidates for the final prediction model. Vascular invasion and perineural invasion were combined into 1 composite factor. R status was further dropped from the final model as over 95% of patients had undergone R0 resection. To allow flexibility in representing nonlinear covariate effect on outcome, the number of positive lymph node, largest lesion size, and Ki-67 were modeled using restricted cubic splines.²⁰

Nomogram Validation

The internal validation was performed on MSK and VUH cohorts (n= 620) using bootstrap with 100 resampling method. Biascorrected c-index was used to internally evaluate the discriminative power of this prediction tool.²⁰ Bias-corrected c-index was also calculated for the ENETS/AJCC 8th staging systems for well-differentiated tumor and the WHOgrading classification, which are commonly used in clinical practice to stage and classify Pan-NETs, and for the AJCC 8th staging system for pancreatic neuroendocrine carcinoma (PanNEC). Each of the 3 indices was compared with the c-index from the nomogram using methods proposed by Kang at al.²¹

The external validation was performed on the cohort of patients treated at JHH, GRI, and RNSH (n = 328). Model performance was evaluated by assessing c-index proposed by Gönen et al,²² and calibration curve on the external validation. Concordance probability is a measurement of discrimination,²² and its interpretation is similar to that of the area under the receiver-operating characteristic curve.²³ It is the probability that given 2 randomly selected patients, the patient who recurred first had a higher probability of recurrence. In addition, to measure the ability to discriminate, models were evaluated with calibration curves in which predicted outcome from the nomogram versus observed outcome from Kaplan-Meier is graphically depicted to further access model’s ability to accurately estimate prognosis.²⁰ The calibration plot provides a visual interpretation of model’s performance, but does not lend itself to a hard and fast decision rule. The error bars represent the 95% confidence interval (CI) around the observed values. If the points fall on or near 45-degree line, the model is said to have good calibration. If the points fall above the 45-degree line, the model is said to underestimate the 5-year recurrence-free probability and overestimate the risk of recurrence. On the contrary, if the points fall below the 45-degree line, the model is said to overestimate the recurrence-free probability and underestimate the risk of recurrence. Specific ways of recalibration would depend on the pattern of deviations from the 45-degree line.

All analyses were performed either in SAS (SAS Institute Inc., Cary, NC) or in R (R Foundation for Statistical Computing, Vienna, Austria). All P values were 2-sided. P values <0.05 were considered to indicate statistical significance.

RESULTS

During the study period, 912 patients underwent pancreatic resection for G1/G2 PanNETat MSK and VUH. Of these, 280 (31%) were excluded due to distant metastatic disease identified at the time of operation (n = 87), the presence of a hereditary syndrome (n = 38), the use of neoadjuvant (n =36) or adjuvant treatments (n = 9), postoperative mortality (n = 1), documented R2 status (n = 5), and lack of Ki-67 on pathological report (n = 104). In all, 632 patients were included, and their clinical and pathologic characteristics are listed in Table 1. Median age was 57 years (range 19–85 years), and in 48% of cases, the PanNET was incidentally discovered. Overall, 90 patients (14%) had a functional PanNET, 429 (68%) had a G1 tumor, and 203 (32%) had a G2 tumor. Median tumor diameter was 2 cm (range 0.4–13.5 cm) and median Ki-67 was 2% (0.3%–20%). At the time of analysis, 76 patients (12%) had experienced a recurrence, with a median time to recurrence of 37 months (range 1–126).

TABLE 1.

Clinicopathological Characteristics of the Internal Cohort

Characteristics	Training Cohort (n = 632)	MSK (n = 226)	VUH (n = 406)
Age, yrs, median (range)	57 (19–85)	59 (27–83)	55 (19–85)
Sex, n (%)
Female	321 (51)	116 (51)	205 (51)
Male	311 (49)	110 (49)	201 (49)
Functional, n (%)
No	540 (85)	215 (95)	325 (80)
Yes	90 (14)	11 (5)	79 (19)
Unknown	2 (1)	0 (0)	2 (1)
Multifocal, n (%)
No	618 (98)	216 (96)	402 (99)
Yes	14 (2)	10 (4)	5 (1)
Primary pancreatic sites, n (%)
Head	213 (34)	68 (30)	145 (36)
Body/tail	411 (65)	155 (69)	256 (63)
Multiple site	8 (1)	3 (1)	5 (1)
Surgical procedure, n (%)
Pancreaticoduodenectomy	171 (27)	63 (28)	108 (27)
Distal pancreatectomy	279 (44)	124 (55)	155 (38)
Central pancreatectomy	68 (11)	21 (9)	47 (12)
Enucleation	102 (16)	18 (8)	84 (21)
Total pancreatectomy	11 (2)	0 (0)	11 (3)
Other	1 (0)	0 (0)	1 (0)
Mini-invasive surgery, n (%)
No	495 (78)	174 (77)	321 (79)
Yes	137 (22)	52 (23)	85 (21)
Grade, n (%)
G1	429 (68)	156 (69)	273 (67)
G2	203 (32)	70 (31)	133 (33)
Tumor diameter, cm, median (range)	2 (0.4, 13.5)	2.1 (0.5, 13.5)	1.8 (0.4, 13.5)
Ki-67, %, median (range)	2 (0.3, 20)	2 (0.3, 20)	2 (1, 20)
No. of nodes, median (range)	10 (0–91)	8 (0–59)	11 (0–91)
No. of positive nodes, median (range)	3 (1–34)	2.5 (1–25)	3 (1–34)
R status, n (%)
R0	597 (94.5)	212 (94)	385 (95)
R1	35 (5.5)	14 (6)	21 (5)
Vascular invasion, n (%)
No	447 (71)	145 (64)	302 (74)
Yes	174 (27)	81 (36)	93 (23)
Not available	11 (2)	0 (0)	11 (3)
Perineural invasion, n (%)
No	483 (76)	162 (72)	321 (79)
Yes	130 (21)	64 (28)	66 (16)
Not available	19 (3)	0 (0)	19 (5)
ENETS/AJCC 8th stage, n (%)
I	273 (43)	85 (38)	188 (46)
IIA	140 (22)	57 (25)	83 (21)
IIB	81 (13)	44 (19)	37 (9)
IIIB	138 (22)	40 (18)	98 (24)

Nomogram

Median follow-up among survivors was 51 months, and we observed 74 patients with recurrence at the time of analysis. Outcome was reported as 5-year freedom from recurrence. Univariate analysis identified older age, nonfunctional tumor status, increased Ki-67 value, tumor grade, tumor diameter, number of positive nodes, R status, and the presence of vascular and perineural invasion, to be associated with recurrence (Table 2).

Table 2.

Univariate Analysis of Risk Factors Associated With Disease Recurrence in the Training Cohort

Characteristics	No. of Events	HR (95% CI)	P
Age			0.037
Per 1-year increase		1.02 (1–1.04)
Sex			0.66
Female	36	Ref
Male	40	1.11 (0.7–1.75)
Functional			0.016
No	72	Ref
Yes	4	0.29 (0.11–0.79)
Primary pancreatic site			0.539
Head	22	Ref
Body/tail	53	1.32 (0.8–2.18)
Multiple site	1	1 (0.13–7.4)
Mini-invasive procedure			0.531
No	66	Ref
Yes	10	0.81 (0.41–1.58)
Tumor diameter			<0.001
Per 1-unit increase		1.31 (1.24–1.39)
Ki-67%			<0.001
Per 1-unit increase		1.19 (1.15–1.23)
Grade			<0.001
G1	16	Ref
G2	60	11.3 (6.47–19.72)
R status			<0.001
R0	63	Ref
R1	13	4.32 (2.37–7.87)
No. of positive nodes			<0.001
Per 1-unit increase		1.14 (1.10–1.18)
Vascular invasion			<0.001
No	22	Ref
Yes	54	8.55 (5.14–14.21)
Perineural invasion			<0.001
No	33	Ref
Yes	42	5.91 (3.72–9.4)

After excluding 12 patients with missing data in at least 1 of these variables, the nomogram was constructed using the following variables: number of positive nodes in the specimen, Ki-67 value, tumor diameter, and presence of vascular or perineural invasion (Fig. 1). We did not include functional status in the nomogram because functional tumors recurred significantly less than nonfunctional tumor [hazard ratio (HR) 0.29, 95% CI 0.11–0.79, P = 0.016], but were also smaller (mean size 1.7 vs 2.8 cm; P =0.05), had lower Ki-67 value (mean 2.2% vs 23.3%; P = 0.05), and a lower likelihood of having perineural (7% vs 23.5%; P < 0.05) and vascular (7% vs 31.5%; P = 0.05) invasion compared with nonfunctional PanNET.

FIGURE 1.

Nomogram predicting the probability of 5-year recurrence-free survival. Points are assigned for number of positive lymph nodes, Ki-67, tumor diameter, presence of vascular invasion or perineural invasion, by drawing a line upward from the corresponding values to the “Points” line. The sum of these 3 points, plotted on the “Total points” line, corresponds to predictions of 5-year recurrence-free probabilities.

Nomogram Validation

The nomogram c-index on the internal cohort was 0.85, and this was superior to predictions based on the ENETS/AJCC 8th staging system for well-differentiated PanNET (c-index 0.76, P < 0.001), on the AJCC 8th for PanNEC (c-index 0.79, P < 0.001) (Fig. 2) and the WHO grade classification (c-index 0.76, P < 0.001). The external validation of the nomogram was conducted on the external cohort with no missing data in variables (n = 328). The clinicopathologic characteristics of the cohorts are shown in Table 3. The median age was 59 years (range 17–87 years), 71% of the lesions were G1 and 29% G2. Median tumor diameter was 2 cm (range 0.5–16 cm), and the median Ki-67 was 2% (range 0.1%–20%). The median follow-up among survivors was 40 months, and 30 patients had developed recurrence at the time of the study. The nomogram was employed to score each patient from this cohort, with a c-index of 0.84 (95% CI 0.79–0.88). The calibration plot for this cohort is shown in Fig. 3.

FIGURE 2.

Time to recurrence by (A) ENETS/AJCC 8th staging system for PanNET and (B) AJCC 8th ed. staging system for PanNEC.

TABLE 3.

Clinicopathological Characteristics of the External Cohort

Variable	Validation Cohort (n = 328)	JHH (n = 219)	GRI/RNSH* (n = 109)
Age, median (range)	59 (17, 87)	59 (17, 87)	61 (18, 87)
Sex, n (%)
Male	175 (53)	122 (56)	53 (49)
Female	153 (47)	97 (44)	56 (52)
Functional, n (%)
No	268 (82)	195 (89)	73 (67)
Yes	54 (16)	24 (11)	30 (27.5)
Not available	6 (2)	0 (0)	6 (5.5)
Grade, n (%)
G1	233 (71)	156 (71)	77 (71)
G2	95 (29)	63 (29)	32 (29)
Ki-67, %, median (range)	2 (0.1, 20)	2 (0.1, 20)	1.5 (0.5, 20)
Tumor size, cm, median (range)	2 (0.5, 16)	1.9 (0.5, 10.5)	2 (0.8, 16)
No of positive nodes, median (range)	0 (0, 19)	0 (0, 19)	0 (0, 18)
Vascular invasion, n (%)
No	257 (78)	180 (82)	77 (71)
Yes	71 (22)	39 (18)	32 (29)
Perineural invasion, n (%)
No	274 (83.5)	175 (80)	99 (91)
Yes	54 (16.5)	44 (20)	10 (9)

^*GRI and RNSH cohorts are presented together because were managed from the rgical team.

FIGURE 3.

Calibration plot for predction of of 5-year recurrence-free survival on external cohort. The x-axis represents the nomogram-predicted probability of RFS and the y-axis represents the observed fraction with evidence of RFS. Perfect prediction corresponds to the 45° line. Points estimated below the 45° line correspond to nomogram overall prediction whereas points situated above the 45° line correspond to nomogram under prediction.

DISCUSSION

In recent years, pancreatic neuroendocrine neoplasms have been increasingly diagnosed, and currently represent the second most common indication for pancreatic surgery, after pancreatic adenocarcinoma.¹ The majority of well-differentiated PanNETs that undergo resection are characterized by a favorable prognosis, with only 13% to 17% of patients experiencing recurrence during postoperative follow-up.¹⁵ Currently, there is no indication for adjuvant therapy for PanNETs after resection, regardless of the pathological characteristic of the tumor. A large number of patients are therefore included in surveillance programs after resection; however, there is no consensus on the optimal frequency of the visits and type of investigations to be performed.⁸ Indeed, different surveillance protocols have been proposed, using CT scans, MRI scans, or octreoscan or gallium-68-based PET, every 6 or 12 months, according to the different international societies.^8,24 No follow-up protocols adjusted to the risk of recurrence are available, and, as a consequence, many patients undergo potentially unnecessary imaging studies for a long period. For example, data from the current study shows that patients who underwent resection for T1 PanNET (tumor diameter less than 2 cm) do not recur before 3 years; therefore, in these cases, a longer period before the first follow-up may be reasonably argued. These considerations underscore the need for more accurate prognostic models to stratify patients according to the risk of recurrence, allowing the development of personalized surveillance programs and a better distribution of health resources.

The aim of the present study was, therefore, to develop a clinical tool that predicts recurrence for individual patients after curative resection of G1/G2 PanNET in the absence of adjuvant treatment. By considering a wide variety of prognostic factors and complex mathematical relationships, the current nomogram individualizes the risk of recurrence for each patient and demonstrated a higher accuracy than the current staging systems and the WHO grade classification.We combined in the same model: variables included in the ENETS/AJCC 8th staging system, those included in the WHO grade classification and other prognostic pathological variables, as vascular or perineural invasion, which are not a part of the TNM system. In addition, the nomogram assigns points based on the exact Ki-67 proliferative index, the tumor diameter, and the number of positive lymph nodes in a continuous, but not linear, fashion, improving the predictive accuracy of the model. Incorporating nonlinear variables is clinically relevant and is exemplified by the Ki-67 value, as prior studies have demonstrated that small variations in the Ki-67 value result in significant differences in prognosis.^12,25 The number of metastatic lymph nodes also appears to be clinically important rather than the simple dichotomization into a binary variable (positive vs negative). Our data indeed demonstrated a 1.14-fold increased risk of developing recurrence for each metastatic lymph node, in line with recent studies showing the number of positive lymph nodes to be independently associated with recurrence.^26,27

The model’s ability to predict outcomes was assessed using the c-index, which expresses the ability of the nomogram to distinguish between patients who present the event from those who do not. A value of 0.5 indicates that the model is no better than chance, a value above 0.70 generally identify a good model, and a value above 0.80 a strong model, whereas a c-index of 1.0 indicates a perfect prediction model.⁹ Our nomogram achieved a c-index of 0.85 in the training cohort, and the strength of the model was then confirmed by a c-index of 0.84 in the external validation cohort. In addition, the calibration plot demonstrated an almost perfect accuracy of our model in predicting recurrence-free probability in patients with low risk of recurrence (5-year recurrence-free probability >80%). In patients with a recurrence-free probability ranging between 55% and 70%, the nomogram was less accurate, underestimating it by about 15. However, in our opinion, these patients still present a significant risk of recurrence that warrants regular surveillance schedules and patient counseling.

The strengths of this study are represented by the large sample size, the multi-institutional nature of the data, and the validation of an external population, and also the inclusion of continuous variables into the model. Also, the proposed nomogram relies on only 4 variables that are easily evaluated on the surgical specimen and should be provided in the pathological report, significantly decreasing its complexity. The effect of the functional status on recurrence was dependent upon these variables. Functional tumors were therefore included in the construction of the nomogram making the nomogram broadly applicable for all well-differentiated PanNETs, regardless of the functional status.

Multiple prior efforts at nomogram development have been made for patients with PanNET.^13–15 These studies, however, used smaller cohorts of patients, did generally not have external validation, or included neuroendocrine tumors from other gastrointestinal sites. A recent multi-institutional study from Europe by Genç et al¹⁵ developed a nomogram to predict recurrence on a cohort of 211 patients with no external validation. Only categorical variables were included, largely limiting the range of possible scores and with no clear improvements compared with the TNM staging systems. A second nomogram was proposed by the US Neuroendocrine Tumor Study Group¹³ on a large cohort of 754 gastroenteropancreatic tumors and was independently but not externally validated. This model was not specific for pancreatic tumors, representing a relevant limitation as PanNETs have demonstrated different patterns and timescales of recurrence compared with neuroendocrine tumors from other gastrointestinal sites.^28,29

The present study does have limitations. Given the retrospective and multicentric nature of the study, we cannot exclude that some pathological features might not have been evaluated uniformly across the institutions. In particular, tumor heterogeneity and subjectivity in hot spots in the Ki-67 calculation may have led to variations in reporting the Ki67 index.^30,31 Similarly, we included in our model vascular and perineural invasion as features of aggressive behavior, and, because PanNETs are highly vascularized tumors, it may be difficult to distinguish true vascular invasion from tumor-related vascularity.³¹ Finally, lymphadenectomy was not performed in all patients, and therefore we cannot exclude that some of these patients might be understaged due to the lack of appropriate nodal sampling. However, we found that these biases were controlled because nomogram predictions were well-calibrated between the training and the external validation cohort. Finally, recurrence after PanNET resection may occur up to 10 years after surgery, whereas the current nomogram was developed on patients who were under surveillance for a median time of 51 months. A longer follow-up period will therefore be required to improve the nomogram.

As future perspectives, recent genetic and gene expression studies have demonstrated exciting avenues for PanNETs prognostication as they identify molecular alterations, including in the alternative lengthening of telomeres and in mammalian target of rapamycin pathways, which yield prognostic and biological significance.^32–34 In the near future, clinical and pathological features could be integrated with genomic data to further improve the predictive ability of the model.

CONCLUSIONS

In conclusion, we have presented an externally validated nomogram that accurately predicts 5-year recurrence after curative resection of PanNETs, and that improves upon current TNM staging systems and the WHO grade classification. This model will enable the development of surveillance programs based on the individual risk of recurrence and facilitate design future adjuvant therapy clinical trials in high-risk patients.