Validation of a nomogram to predict the risk of cancer in patients with intraductal papillary mucinous neoplasm and main duct dilatation of 10mm or less

Abstract

Background:

Intraductal papillary mucinous neoplasm (IPMN) is premalignant pancreatic lesion. International guidelines offer limited predictors of individual risk. A nomogram to predict individual IPMN malignancy risk was released, with good diagnostic performance based on a large cohort of Asian patients with IPMN. The present study validated a nomogram to predict malignancy risk and invasiveness of IPMN using both Eastern and Western cohorts.

Methods:

Clinicopathological and radiological data from patients who underwent pancreatic resection for IPMN at four centres each in Eastern and Western countries were collected. After excluding patients with missing data for at least one malignancy predictor in the nomogram (main pancreatic duct diameter, cyst size, presence of mural nodule, serum carcinoembryonic antigen and carbohydrate antigen (CA) 19-9 levels, and age).

Results:

In total, data from 393 patients who fit the criteria were analysed, of whom 265 were from Eastern and 128 from Western institutions. Although mean age, sex, log value of serum CA19-9 level, tumour location, main duct diameter, cyst size and presence of mural nodule differed between the Korean/Japanese, Eastern and Western cohorts, rates of malignancy and invasive cancer did not differ significantly. Areas under the receiver operating characteristic (ROC) curve values for the nomogram predicting malignancy were 0·745 for Eastern, 0·856 for Western and 0·776 for combined cohorts; respective values for the nomogram predicting invasiveness were 0·736, 0·891 and 0·788.

Conclusions:

External validation of the nomogram showed good performance in predicting cancer in both Eastern and Western patients with IPMN lesions.

Introduction

Pancreatic cancer has a reported 5-year survival rate of 8 per cent¹. Late diagnosis and poor response to systemic chemotherapy are the main reasons for this dismal prognosis. Less than 20 per cent of patients with pancreatic cancer are candidates for curative resection at diagnosis². Identifying precancerous lesions that are likely to progress to pancreatic cancer could improve outcomes.

Intraductal papillary mucinous neoplasm (IPMN) is the best known radiologically identifiable premalignant lesion of pancreatic cancer. The apparent incidence of IPMN is increasing because of increased awareness of the disease and advances in diagnostic radiological imaging³. Although observation may be sufficient for small IPMNs, invasive IPMNs showed worse prognosis than other gastrointestinal cancers, with a 5-year survival rate of 27–60 per cent⁴. Therefore, studies to find appropriate indications for surgical treatment are continuing. Complete resection is generally recommended for IPMNs with dilatation of the main pancreatic duct (MPD) exceeding 10 mm because of the high risk of progression to malignancy, but treatment and surveillance of IPMNs with MPD dilatation of 10 mm or less are controversial. International Consensus Guidelines⁵ published in 2012 suggested several features indicative of malignant transformation, including the dilatation or calibre change in MPD, large cysts, mural nodules and septal wall thickening. In 2017, the guidelines⁶ added raised serum carbohydrate antigen (CA) 19-9 level as a worrisome feature, based on studies^7-9 that showed a relationship between high CA 19-9 levels and malignant IPMN. The guidelines recommended careful surveillance or resection of IPMN with these features. However, because each worrisome feature had a different statistical value and some patients have more than two or three such features, the guidelines have limited quantitative predictive value for malignancy risk in individual patients.

To establish treatment standards and to predict individual risk of malignancy in patients with IPMN, a collaborative study was undertaken and a nomogram to predict malignancy in IPMN was proposed in 2017¹⁰. However, worldwide application of this nomogram requires it to be validated in cohorts of different ethnicities. This study was designed to validate the nomogram’s predictive value for malignancy in IPMN.

Methods

All data were collected and analysed at the Department of Surgery and the Department of Statistics, Seoul National University, Korea. This study was approved by the institutional review board of each hospital (H-1609-055-790).

Nomogram description and validation cohort

A nomogram to predict malignancy in IPMN was proposed in 2017¹⁰. The area under the receiver operating characteristic (ROC) curve (AUC) of the nomogram was 0·783 for malignancy. Data were collected for 2258 patients who underwent surgery for IPMN in nine Korean and 13 Japanese centres. Several radiological findings and serum tumour markers (CA19-9, carcinoembryonic antigen (CEA)) were used to establish the malignancy prediction model for IPMN. Based on these variables, a malignancy predictive nomogram was developed (Fig. 1) and published as an open, web-based program for ease of accessibility (http://statgen.snu.ac.kr/software/nomogramIPMN).

Fig. 1. Nomogram predicting individual risk of cancer in intraductal papillary mucinous neoplasm.

CEA, carcinoembryonic antigen; CA19-9, carbohydrate antigen 19-9; MPD, main pancreatic duct. Figure adapted from Jang et al.¹⁰ Copyright (c) 2017, with permission from Wolters Kluwer Health, Inc.

The explicit multiple logistic regression models in equation form are as follows:

$$\begin{gathered} \text{Malignancy model: logit}\left(\frac{{\pi }_i}{1-{\pi }_i}\right) \\ =-5\cdot 654857+0\cdot 0198\left(\text{age}\right)+0\cdot 2549\left(\log (\text{CEA})\right) \\ +0\cdot 2395\left(\log (\text{CA19-9})\right)+0\cdot 2413\left(\text{main duct size}\right) \\ +0\cdot 02\left(\text{cyst size}\right)+1\cdot 055\left(\text{mural nodule}\right). \\ \text{Invasive model: logit}\left(\frac{{\pi }_i}{1-{\pi }_i}\right) \\ =-5\cdot 90679+0\cdot 00541\left(\text{age}\right)+0\cdot 2675\left(\log (\text{CEA})\right) \\ +0\cdot 5085\left(\log (\text{CAI9-9})\right)+0\cdot 1784\left(\text{main duct size}\right) \\ +0\cdot 02\left(\text{cyst size}\right)+1\cdot 028\left(\text{mural nodule}\right). \end{gathered}$$

External validation was done using external cohorts of patients who underwent surgery for IPMN at multiple institutions in Eastern and Western countries where the prevalence of pancreatic cancer is high, and to explore its clinical value in a multinational cohort.

Model construction

It was ensured that this prediction model validation procedure followed the 22 reporting guidelines of the TRIPOD statement¹¹.

Patients and definitions

This study included patients who underwent pancreatic resection for IPMN from China, Taiwan, Europe and the USA, including Peking University First Hospital (China), Zhongshan Hospital (China), National Taiwan University Hospital (Taiwan), Taipei Veterans General Hospital (Taiwan), Academic Medical Centre (the Netherlands), Karolinska University Hospital (Sweden), Columbia University Medical Center (USA) and Johns Hopkins Hospital (USA), between 1997 and 2017 (Table S1, supporting information).

As in the earlier study, patients with MPD dilatation over 10 mm on radiological imaging or endoscopic ultrasonography (EUS) were excluded. A standard case form was used to collect clinicopathological and radiological data, including factors predictive of malignancy reported in the previous study. Tests for diagnosis and surveillance of IPMN differed between countries for several reasons, such as differences in medical insurance coverage. Preoperative enhanced CT, MRI or EUS was commonly used to measure cyst size, MPD diameter and establish mural nodule status. All detectable mural nodules from radiological images were checked regardless of size, as in the previous study. Preoperative levels of serum tumour markers were measured. For this study, complete-case analysis was undertaken; patients who had data missing for any of the predictive factors in the nomogram were excluded.

As in the previous study¹⁰, malignancy was defined as high-grade dysplasia and invasive IPMN. The nomogram to predict invasiveness in IPMN was developed separately in a previous study¹⁰, and external validations of two nomograms were performed separately in the present study.

Statistical analysis

Continuous variables are reported as mean(s.d.) or median (range), and compared using two-sided Student t test or one-way ANOVA across cohorts. Pearson’s χ² test was used for categorical variables. Malignancy and invasiveness in patients with IPMN were predicted by the previous IPMN nomogram. The Hosmer-Lemeshow test was used to evaluate goodness of fit for logistic regression, from which the IPMN nomogram was built. The AUC was used to estimate the nomogram’s predictive accuracy for malignancy and invasiveness in the Eastern, Western and combined (Eastern and Western) cohorts. Sensitivity, specificity and balanced accuracy (BA) were evaluated for the three cohorts.

$$\begin{gathered} \text{Sensitivity}=\frac{{\sum }_{i=1}^{n_1}I({\widehat{y}}_{i1}\ge \delta )}{n_1} \\ \text{Specificity}=\frac{{\sum }_{i=1}^{n_2}I({\widehat{y}}_{i2}<\delta )}{n_2} \\ \text{BA}=\frac{1}{2}(\text{sensitivity + specificity}) \end{gathered}$$

where ${\widehat{y}}_{i1}$ is the risk score for the ith individual in the malignancy (or invasive group), ${\widehat{y}}_{i2}$ is the risk score for the ith individual in the benign group, n₁ is the number of samples in the malignancy (or invasive group), n₂ is the number of samples in the benign group, and δ is the cut-off value. Statistical significance was defined at P < 0·050. Statistical analysis was performed using R version 3.1.0 (R Foundation for Statistical Computing, Vienna, Austria) and SPSS® version 21 (IBM, Armonk, New York, USA).

Results

Data were collected on 354 patients from Eastern centres and 781 from Western centres. Finally, a cohort of 393 patients from Eastern (265) and Western (128) centres was used for external validation of the nomogram (Table S2, supporting information).

Clinicopathological characteristics of patients in the Eastern and Western cohorts are shown in Table 1 and compared with data from the original Korean/Japanese cohort used in developing the nomogram. The three cohorts differed significantly in age, sex, log value of serum CA19-9, tumour location, MPD diameter, cyst size and presence of mural nodules.

Table 1.

Comparison of characteristics of validation cohorts

	Korea/Japan (n = 1914)	Eastern (n = 265)	Western (n = 128)	P§	Standardized difference between Eastern and Western
Age (years)*	65·1(9·5)	64·1(10·8)	69·6(8·4)	< 0·001¶	−57·18
Sex ratio (M : F)	1183 : 731	151 : 114	66 : 62	0·030	−10·19
Serum CEA (ng/ml)†‡	2·0 (0·1–176·0)	2·1 (0·1–312·5)	2·1 (0·5–48·2)	0·366¶	16·14
Serum CA19-9 (units/ml)†‡	12·0 (0·1–4533·0)	19·3 (0·1–118 552·5)	16·0 (1·0–9897·0)	< 0·001¶	2·61
Tumour location				< 0·001	–
Head	1181 (61·7)	157 (59·2)	53 (41·4)
Body–tail	656 (34·3)	88 (33·2)	67 (52·3)
Diffuse	77 (4·0)	20 (7·5)	8 (6·3)
Main duct diameter (mm)*	4·8(2·5)	4·5(2·4)	3·9(2·6)	0·001¶	23·96
Cyst size (mm)*	32(15)	33(22)	24(15)	< 0·001¶	49·50
Mural nodule-positive	973 (50·8)	157 (59·2)	111 (86·7)	< 0·001	−65·06
Malignancy	734 (38·3)	95 (35·8)	44 (34·4)	0·519
Invasive cancer	366 (19·1)	61 (23·0)	18 (14·1)	0·098

Values in parentheses are percentages unless indicated otherwise; values are

^*mean(s.d.) and

^†median (range).

^‡These values were log-transformed before performing the analysis. CEA, carcinoembryonic antigen; CA19-9, carbohydrate antigen 19-9.

^§Pearson χ² test, except

^¶ANOVA.

External validation of nomogram

Fig. 2 shows the Hosmer–Lemeshow goodness-of-fit tests for the malignancy prediction model in the combined cohort of 393 patients. Each error bar in the figure shows the bootstrap confidence interval that indicates how accurate the probability of each cluster is. To construct the confidence interval, 1000 bootstrap samples were generated. In the Hosmer–Lemeshow test, higher P values indicate the adequacy of the logistic regression model. Both the 20 per cent (P = 0·534) and 10 per cent (P = 0·279) quantile scales showed strong concordance between predicted probability and percentage malignancy.

Fig. 2. Calibration of nomogram.

a Twenty per cent and b 10 per cent quantile scale. Error bars represent bootstrap confidence intervals. a P = 0·534, b P = 0·279 (Hosmer–Lemeshow test).

The AUCs for predicting malignancy in IPMNs were 0·745 for the Eastern cohort, 0·856 for the Western cohort and for the 0·776 combined cohort (Fig. 3a); respective values for predicting invasiveness were 0·736, 0·891 and 0·788 (Fig. 3b). For malignancy, at a probability of 0·37 (which maximized BA in the previous study), sensitivity, specificity and BA were 0·71, 0·66 and 0·68 respectively for Eastern; 0·64, 0·92 and 0·78 for Western; and 0·68, 0·74 and 0·71 for combined cohorts (Table 2). For invasiveness, at a probability of 0·20 (which maximized the BA in the previous study), sensitivity, specificity and BA were 0·69, 0·69 and 0·69 respectively for Eastern; 0·61, 0·90 and 0·76 for Western; and 0·67, 0·76 and 0·72 for combined cohorts. For malignancy, the nomogram’s BAs were maximal at probabilities of 0·48 for the Eastern, 0·29 for the Western and 0·35 for the combined cohort (Table 2). For invasiveness, the nomogram’s BAs were maximal at probabilities of 0·20, 0·11 and 0·11 respectively.

Fig. 3. Receiver operating characteristic (ROC) curves of nomograms for Eastern and Western cohorts.

a Area under curve (AUC) for prediction of malignancy: Eastern cohort 0·745, Western cohort 0·856, combined cohort 0·776; b AUC for prediction of invasiveness: Eastern cohort 0·736, Western cohort 0·891, combined cohort 0·788.

Table 2.

Diagnostic value of nomograms for malignancy and invasiveness of intraductal papillary mucinous neoplasms in Eastern and Western cohorts

	AUC	Sensitivity	Specificity	BA	Cut-off†	Sensitivity	Specificity	BA
Malignancy		Cut-off 0·37*
Eastern	0·745 (0·608, 0·805)	0·71 (0·59, 0·78)	0·66 (0·63, 0·77)	0·68 (0·63, 0·75)	0·48	0·63 (0·58, 0·77)	0·80 (0·68, 0·81)	0·72 (0·65, 0·77)
Western	0·856 (0·759, 0·917)	0·64 (0·47, 0·76)	0·92 (0·91, 1·00)	0·78 (0·71, 0·86)	0·29	0·71 (0·62, 0·88)	0·88 (0·77, 0·92)	0·79 (0·72, 0·87)
Combined	0·776 (0·719, 0·818)	0·68 (0·58, 0·74)	0·74 (0·73, 0·83)	0·71 (0·68, 0·77)	0·35	0·71 (0·60, 0·75)	0·74 (0·73, 0·83)	0·72 (0·68, 0·77)
Invasiveness		Cut-off 0·20*
Eastern	0·736 (0·667, 0·801)	0·69 (0·55, 0·79)	0·69 (0·62, 0·75)	0·69 (0·61, 0·75)	0·20	0·69 (0·59, 0·82)	0·69 (0·60, 0·73)	0·69 (0·62, 0·75)
Western	0·891 (0·808, 0·950)	0·61 (0·45, 0·88)	0·90 (0·82, 0·94)	0·76 (0·66, 0·89)	0·11	0·94 (0·66, 1·00)	0·75 (0·71, 0·86)	0·85 (0·71, 0·90)
Combined	0·788 (0·731, 0·834)	0·67 (0·57, 0·76)	0·76 (0·71, 0·80)	0·72 (0·66, 0·77)	0·11	0·87 (0·72, 0·90)	0·58 (0·54, 0·64)	0·73 (0·65, 0·75)

Values in parentheses are 95 per cent confidence intervals.

^*Cut-off value that maximized balanced accuracy (BA) in previous study

^†cut-off value that maximized BA for each cohort. AUC, area under curve.

Discussion

In this international validation study, a nomogram was used to predict the probability of malignancy and invasiveness of IPMN in individual patients. External validation analysis in combined Eastern and Western cohorts showed a predictive AUC value of 0·776 for malignancy, and indicated that the nomogram could be applied to patients with IPMN worldwide and of differing ethnicities.

Surgery to prevent cancer by resecting precursor lesions such as IPMN could be an important method of improving the dismal prognosis of patients with pancreatic cancer. However, risk–benefit outcomes of pancreatectomy with respect to perioperative morbidity and mortality, and loss of endocrine and exocrine function, would require precise selection of patients whose IPMNs are at high risk of malignant transformation, considering the relatively dormant nature of many IPMNs. The algorithm suggested by International Consensus Guidelines for treatment of IPMN^5,9 cannot predict malignancy risk for individual patients with various characteristics. To overcome this limitation, several nomograms or scoring systems to predict malignancy in IPMN have been proposed^12-16. Most of these, however, were based on small numbers of enrolled patients and the results were not validated externally.

A nomogram to predict IPMN malignancy was first suggested in 2010¹³, and validated using a cohort from three hospitals in Japan¹⁵. Four factors – sex, type of lesion, size of mural nodule and pancreatic juice cytology – were selected for the nomogram. This nomogram showed good diagnostic performance in the external validation study, with an AUC value of 0·760. However, the study population was relatively small; furthermore, obtaining pancreatic juice involves an invasive procedure with risk of complications, and its cytology reportedly provides high specificity but low sensitivity and is highly dependent on individual pathologists^17-19.

A multi-institutional nomogram to predict IPMN dysplasia grade was developed and validated recently using data from 1028 patients who underwent pancreatic resection for IPMN at three centres in the USA¹⁶. Age, sex, size of cyst, presence of mural nodule, weight loss, symptoms and MPD dilatation were included as scoring factors in the nomogram. The nomogram was developed with a training data set of 720 patients and validated with an independent validation data set of 308 patients. The concordance index was 0·81 at validation, which indicated a high level of discrimination. In this study, continuous variables such as cyst size and MPD diameter were categorized into groups and scored as such. Arbitrary categorization of continuous variables could alter the malignancy risk calculated by means of the nomogram. An increased serum CA19-9 level was found to be a significant predictor of malignancy for IPMN in previous studies^7-9,20,21, was included as a worrisome feature in the most recent revision of the International Consensus Guidelines⁶, and was identified as a predictive risk factor in univariable analysis in the US study¹⁰. However, the CA19-9 level was not included in subsequent analysis because 40 per cent of the patients did not have CA19-9 levels available. External validation of a nomogram is important to confirm that it can be applied to patients outside of the cohort. This US nomogram was not validated externally using other ethnic groups.

The nomogram to predict IPMN malignancy risk used data from 2258 Korean and Japanese patients and was developed in the authors’ previous study¹⁰. Multivariable analysis with 200 × cross-validations indicated six factors (MPD diameter, cyst size, presence of mural nodules, serum CEA and CA19-9 levels, and age) that were most predictive of malignancy. The nomogram is based on these six variables and offers robust predictive power for malignancy (AUC 0·783).

Although the nomogram described in the previous study¹⁰ was based on a large cohort, these patients were all treated in Korea or Japan. For worldwide application of the nomogram, external validation with a Western cohort was required, and the authors therefore proceeded with this multinational collaborative study. To apply the nomogram, all variables included in the nomogram must be present; 89 patients in the Eastern cohort and 653 in the Western cohort with data missing were therefore excluded. In most instances, the missing information was serum tumour marker levels. An increase in tumour marker concentrations was a significant predictor of malignancy in the authors’ previous study¹⁰, with an odds ratio similar to that of mural nodules, and reportedly associated with malignancy in other studies^7,8. For this reason, although a relatively large number of patients were excluded owing to lack of serum tumour marker data, only patients with all six variables (including serum tumour marker levels) were included this validation study. Other characteristics of included and excluded patients were compared to verify the differences between the two groups (Table S2, supporting information). There were no significant differences between included and excluded patients in the Eastern cohort. In the Western cohort, although age, cyst location, mean MPD diameter and mean cyst size differed significantly between the two groups, sex, presence of mural nodule, and rates of malignancy and invasive cancer were similar.

The probability of malignancy can be calculated using this nomogram. Raising the cut-off value for deciding resection increases the specificity and decreases the sensitivity, and might be adjusted in consideration of a patient’s general condition, willingness to undergo surgery and surgical risk. According to the present results, if the probability of malignancy calculated using the nomogram in an individual patient is over 0·30 or 0·40, resection is worth considering, depending on the patient’s general condition and life expectancy. If the calculated probability of invasiveness is 0·11 or above, surgical resection could be strongly recommended to the patient. It will be more appropriate and helpful to patients to recommend resection according to the probability of malignancy calculated by means of the nomogram, which reflects multiple risk factors, than deciding treatment based on any single risk factor. For younger and healthier patients, cut-off values for deciding surgical resection could be lower than those suggested here, in consideration of relatively good ability to tolerate major surgery.

This study has several limitations. First, a large number of patients were excluded owing to lack of data, especially preoperative serum tumour marker levels in Western patients. For wider use of the nomogram based on radiological findings, the authors are planning to develop another version of the nomogram without serum tumour markers, using data sets collected from Korean/Japanese, Eastern and Western centres. Second, this study was based on retrospective data from patients who underwent surgery. Further prospective validation studies based on preoperative cohorts and patients under surveillance are needed to confirm the clinical value of this nomogram.