Validation of the ENDPAC model: Identifying new-onset diabetics at risk of pancreatic cancer

Abstract

Background:

Patients with new-onset diabetes are known to be at a higher risk of developing pancreatic cancer. The Enriching New-Onset Diabetes for Pancreatic Cancer (ENDPAC) model was recently developed to identify new-onset diabetics with this higher risk. Further validation is needed before the ENDPAC model is implemented as part of a screening program to identify pancreatic cancer.

Methods:

A retrospective case-control study was performed; a cohort of patients with new-onset diabetes was identified using hemoglobin A1c. Patients were scored by the ENDPAC model and then divided based on whether pancreatic cancer was diagnosed after the diagnosis of diabetes. The performance of the model was assessed globally and at different cutoffs.

Results:

There were 6254 controls and 48 cases of pancreatic cancer. Bivariate analysis showed that patients with pancreatic cancer lost weight before diagnosis while controls gained weight (−0.93 kg/m2 vs. 0.45 kg/m2, p < 0.00*). Cases had a more significant increase in their HbA1C from one year before (1.3% vs. 0.82%, p = 0.02). Smoking and pancreatitis rates were higher in cases compared to controls (p < 0.00*). The area under the curve (AUC) of the ENDPAC model was 0.72. A score >1 was the optimal cutoff. At this cutoff, the sensitivity was 56%, specificity was 75%, and pancreatic cancer prevalence increased from 0.78% at baseline to 1.7%.

Conclusion:

The ENDPAC model was validated in an independent cohort of patients with new-onset diabetes.

Introduction

Pancreatic cancer is third in all cancer-related deaths in the United States (US) [1]. Often diagnosed at a late stage, treatment options are limited, and only 15–20% of cases can be resected [2]. Despite the evident importance of screening for this disease, the US Preventive Services Task Force (USPSTF) currently recommends against screening asymptomatic adults for pancreatic cancer due to the high number of potentially false-positive cases [3]. While certain high-risk groups would benefit from screening, such as patients with familial pancreatic cancer, Lynch syndrome, and other genetic disorders [4], there is no large scale screening method to identify patients with early-onset pancreatic cancer.

Despite this, in the last several years, attention has been drawn to screening new-onset diabetics for their risk of developing pancreatic cancer. Diabetes is an independent risk factor for developing pancreatic cancer [5]. It is estimated that the incidence of pancreatic cancer among patients with new-onset diabetes after 50 is near 1%, a 6–8 fold increase from the general population [6]. Given the prevalence of diabetes in the US, screening new-onset diabetics can unmask many patients with early-onset pancreatic cancer.

Several models have been proposed to isolate a high-risk cohort of patients with new-onset diabetics who would benefit from screening. The Enriching New-Onset Diabetes for Pancreatic Cancer (ENDPAC) model, developed by Sharma et al. was designed to consider three variables that have previously been shown to differentiate diabetes secondary to pancreatic cancer from type 2 diabetes. These variables are age at diagnosis of diabetes, change in weight, and change in blood glucose from the year preceding diagnosis. Unlike type 2 diabetics, patients with diabetes secondary to pancreatic cancer will lose weight before the diagnosis of diabetes [7]. The progression of hyperglycemia is more rapid for patients with early-onset pancreatic cancer than type 2 diabetics [8]. Additionally, new-onset diabetics who develop pancreatic cancer tend to be older than type 2 diabetics [9]. In this setting, the ENDPAC model used these patterns to provide a score reflecting the risk of a patient with diabetes developing pancreatic cancer [10].

The ENDPAC model has been validated in one separate study in which it performed well. In the study by Chen et al. the prevalence of pancreatic cancer was noted to be 2.0% in the high-risk cohort, relative to 0.7% at baseline, and the area under the curve was 0.75 [11]. However, in that study, it was also noted that the criteria required to apply the ENDPAC model might be too restrictive; the model requires two abnormal glycemic values paired together to diagnose diabetes. This concern is valid. This study aimed to validate the ENDPAC model in a separate cohort of new-onset diabetic patients and to validate it utilizing a less restrictive definition of diabetes.

Materials and methods

Study setting

This study’s population was obtained as specialized data from TrinetX after obtaining approval from the West Virginia University Institutional Review Board. TrinetX is a global health research network that links healthcare information across multiple healthcare organizations to facilitate clinical trials and observational studies. The original dataset requested from TrinetX was a dataset of all patients who had a hemoglobin A1c (HbA1c) > 6.5, preceded by an HbA1C < 6.5. No other restrictions were placed on this initial dataset, which spanned 1,288,858 patients across 32 healthcare organizations.

Patient population

From this dataset, patients were selected who met the glycemic definition of new-onset diabetes. For this study, the following conditions had to be met to diagnose new-onset diabetes:

1. An HbA1c > 6.5 was preceded by at least one HbA1c < 6.5 in the past 6e24 months. The date the HbA1c > 6.5 was obtained was defined as the index date.

2. No HbA1c > 6.5 before the index date.

3. No exposure to anti-diabetic medications occurred up until three months before the index date.

4. There was no history of pancreatic cancer before the index date.

5. Age greater than 50 at index date.

This definition was adopted from the study by Sharma et al. [10]. For this study, HbA1c did not have to be paired with another abnormal glycemic value to diagnose diabetes. From the original cohort, 107,305 patients met these criteria for new-onset diabetes.

Variables

The ENDPAC model requires three variables to score a patient. These variables are [1] age at diagnosis of diabetes [2], weight change (kg), and [3] change in blood glucose from one year before diagnosis. In order to determine the weight change, body mass index (BMI) values were obtained at the index date (within 90 days) and approximately one year before diagnosis (within six months to two years preceding diagnosis) for every patient that had these values recorded. These values were subsequently converted into bodyweight measurements (kg) by the following formula: BMI x height² (m²). The body height needed for this conversion was imputed from the National Health and Nutrition Examination Survey [12]. Body height measures were assigned for every patient based on the year of diagnosis of diabetes, age, and gender.

In order to determine the change in blood glucose from the year before the diagnosis of diabetes, two HbA1c measurements were obtained for each patient, one measurement at diagnosis and another preceding diagnosis following the same time restrictions that were applied for BMI measurements. HbA1c measurements were subsequently converted into estimated average glucose measurements by formula 28.7 × A1C − 46.7 [10].

In total, there were five measurements needed for an individual to be scored by the ENDPAC model.

1. Age at diagnosis

2. BMI at diagnosis

3. BMI preceding diagnosis

4. HbA1c at diagnosis

5. HbA1c preceding diagnosis

The variables of interest were converted into sub-scores (glucose, age, and weight) according to the rules as shown in the supplemental materials by Sharma et al. [10]. The complete ENDPAC score is a summation of the three sub-scores. Of the original cohort of new onset-diabetics, 15,539 had the elements needed to have a complete ENDPAC score.

Outcome identification

The outcome was a diagnosis of pancreatic cancer within four years of diabetes onset. The ICD10 code C25.* defined the outcome, accompanied by either of the following within one year.

1. Histological confirmation. Patients were cross-matched with the ICD-0 codes 8000, 8010, 8140, 8500, and 8520.

2. The administration of chemotherapy. TrinetX keeps a separate database of chemotherapy administration for any patient in the dataset. Potential cases were cross-matched with this dataset.

3. Procedural intervention. Patients who had endoscopic (ERCP) or surgical (pancreatectomy, pancreaticoduodenectomy, or other pancreatic resection types) intervention. Endoscopic intervention required placement of a biliary or pancreatic stent. The CPTcodes used were 43273, 43274, 43276, or 48100–48160.

Once cases were identified, controls (patients with ENDPAC scores who do not meet the criteria for a diagnosis of pancreatic cancer) were filtered to include only patients who had consecutive annual outpatient clinic appointments for at least four years after diagnosis of diabetes. This requirement was employed to assure that no controls were included who may have been lost to followup. In the final cohort, 48 patients with pancreatic cancer and 6254 control cases were identified.

Statistical analysis

Baseline characteristics were compared between the two groups using student’s t-test for ordinal data and Fisher’s exact test for categorical data. The mean ENDPAC sub-scores and composite score were compared accordingly.

As a binomial classifier, the ENDPAC model categorizes a patient’s risk of developing pancreatic cancer based on their score in relation to a designated cutoff. Patients scoring above that cutoff would need follow-up testing to confirm the presence of pancreatic cancer. At every possible cutoff, sensitivity, specificity, positive predictive value, negative predictive value, and the original cohort’s percentage to be further tested were calculated. Sensitivity was defined as the number of cases (patients with pancreatic cancer) with ENDPAC scores above the respective cutoff divided by the total number of cases in the study. Specificity was defined as the total number of controls (patients without pancreatic cancer) with scores below the respective cutoff divided by the total number of controls in the study. The positive predictive value was defined as the number of cases with scores above the respective cutoff divided by the total number of patients with scores above that cutoff. The negative predictive value was the total number of controls with scores below the respective cutoff divided by the total number of patients with scores below that cutoff. The percentage of patients who would need further screening for each cutoff was defined as the number of patients with scores above the respective cutoff divided by the total number of patients in the cohort.

The Youden index was calculated for each cutoff to determine the optimal cutoff. The Youden index, which is defined as sensitivity + specificity − 1, is commonly utilized in the study of binomial classification to identify the cutoff that maximizes both the sensitivity and specificity of a test [13]. The cutoff with the highest Youden index reflects the “optimal” cutoff. The model’s concordance was assessed by the receiver operating curve and the area under the curve (AUC).

Results

Table 1 shows the baseline characteristics of the final cohort. Patients who developed pancreatic cancer had a higher HbA1c at diagnosis compared to those who did not. They also had a more significant change in their HbA1c from the year before. Their BMIs were lower at diagnosis and decreasing from the year before diagnosis. Both a history of tobacco abuse and pancreatitis occurred at a higher rate in patients who later developed pancreatic cancer. Among the sub-scores, both glucose and weight sub-score were significantly different between the two groups.

Table 1.

Baseline characteristics of the final cohort. A bivariate analysis was done comparing relevant demographic, metabolic, and social characteristics between cases and controls.

Baseline characteristics
		Pancreatic cancer		p-value
		Yes	No
		N = 48	n = 6253
Clinical variables	A1C at diagnosis (%)	7.3	6.9	0.04*
	A1C 1 year before (%)	6.0	6.1	0.03*
	Change in A1C (%)	1.3	0.82	0.02*
	BMI at diagnosis (kg/m2)	28.	31.	<0.00*
	BMI 1 year before (kg/m2)	29.	31.	0.12
	Change in BMI (kg/m2)	−0.93	0.45	<0.00*
	Height (meters)	1.67	1.67	0.82
Demographics	Age	68.	67.	0.15
	Gender (Male)	54%	50%	0.57
	Caucasian	81%	76%	0.50
	Affican American	10%	13%	0.83
	Other race	6.3%	11%	0.36
	Prior Tobacco use	39%	21%	<0.00*
	History of Pancreatitis	16%	2%	<0.00*
Scores	Glucose	1.09	0.61	0.02*
	Weight	3.07	−0.01	<0.00*
	Age	0.35	0.18	0.10
	Total	3.1	−0.01	<0.00*

P values with * indicate a significant difference (p-value <0.05).

Table 2 shows sensitivities and specificities at different cutoffs. The Youden index showed that a cutoff score of greater than 1 was the optimal cutoff point. At this cutoff, the model’s sensitivity was 56%, and the specificity was 75%. In Sharma’s original study, a score greater than two was defined as the high-risk cohort. In this study, a cutoff >2 had a sensitivity of 48%, and the Youden index was only marginally decreased at a cutoff score >2 compared to >1 (0.29 vs. 0.31).

Table 2.

Sensitivity and Specificity of the ENDPAC model at different cutoffs. Patients are defined as high or low risk for each cutoff depending on their score to the designated cutoff. The Youden index was calculated for each cutoff.

Sensitivity and Specificity of the ENDPAC model at different cutoffs
	Sensitivity (%)	Specificity (%)	Youden index
> −7	100	1.7	0.017
> −6	98	4.5	0.024
> −5	96	9.9	0.057
> −4	94	14	0.08
> −3	92	22	0.14
> −2	90	29	0.18
> −1	88	43	0.30
>0	71	56	0.26
> 1	56	75	0.31
>2	48	81	0.29
>3	42	88	0.30
>4	35	91	0.26
>5	27	94	0.21
>6	25	96	0.21
>7	23	98	0.21
>8	15	99	0.14
>9	8.0	100	0.08
>10	0	100	−0.00
>11	0	100	−0.00

In Table 3, positive predictive value (PPV), negative predictive value (NPV), and the percentage of patients who would need further screening are shown at different cutoff scores. At a score of greater than 1 (the optimal cutoff), the PPV was 1.7%, and NPV was 100%. Before applying the ENDPAC model, the prevalence of patients who would later develop pancreatic cancer in the cohort was 0.78%. After applying the model, this prevalence increased by a factor of 2.2 to 1.7% in the high-risk cohort (patients who scored > 1). At this cutoff, 25% of the original population would need to be further screened to confirm the presence of pancreatic cancer.

Table 3.

Positive predictive value (PPV), negative predictive value (NPV), and percentage of the original cohort who would require further screening at different cutoff points.

PPV, NPV, and % of the population to be further screened
Cutoff	Positive Predictive Value (%)	Negative Predictive Value (%)	% to be further screened
> −7	0.78	100	98
> −6	0.78	100	96
> −5	0.81	100	90
> −4	0.83	100	86
> −3	0.9	100	78
> −2	0.96	100	71
> −1	1.2	100	58
>0	1.2	100	45
> 1	1.7	100	25
>2	1.9	100	19
>3	2.6	99	12
>4	2.9	99	9.5
>5	3.4	99	6
>6	4.5	99	4.2
>7	8.8	99	2
>8	9.6	99	1.2
>9	15	99	0.43
>10	0	99	0.095
>11	0	99	0.016

Fig. 1 shows the receiver operating curve of the ENDPAC model. The AUC of the model was 0.72.

Fig. 1.

Receiver-operative Characteristic (ROC) Curve of the ENDPAC model. The Area Under The Curve (AUC) was 0.72.

Discussion

In this validation study, the ENDPAC model was assessed in its ability to predict pancreatic cancer among a cohort of new-onset diabetics. The first step in this was to validate the underlying relationships from which the ENDPAC model derives its performance. These relationships include weight loss (as opposed to weight gain) preceding diabetes, more rapidly progressive diabetes, and older age at diagnosis.

The most crucial relationship, arguably, is the pattern of weight loss, as this is the sub-score from which a patient may gain or lose the most points from the model. This relationship was demonstrated in Table 1; the weight sub-score showed the highest contrast of all sub-scores between type 2 diabetics and early onset pancreatic cancer. This relationship has been demonstrated previously as an early sign of pancreatic cancer [14–16]. Although the mechanism underlying this process is still being investigated, it has been shown that the conversion of subcutaneous adipose tissue [17] and the upregulation of pro-inflammatory cytokines [18] both play a role in promoting early weight loss in pancreatic cancer. Unlike patients with pancreatic cancer, type 2 diabetics will typically gain weight preceding the onset of diabetes.

Another feature distinguishing early onset pancreatic cancer from type 2 diabetes is the onset of glycemic dysregulation; it is more rapid for patients who would later develop pancreatic cancer. While patients with type 2 diabetes develop hyperglycemia 7–8 years before diagnosis [19], patients with pancreatic cancer develop hyperglycemia within 24–36 months of diabetes onset [8,14,16]. This effect was demonstrated in this study; patients who later developed pancreatic cancer had a higher HbA1c at diagnosis of diabetes, a lower HbA1c preceding diagnosis, and a greater change in HbA1c from the previous year than type 2 diabetes. This finding translated into higher glucose sub-scores. The underlying mechanism behind this phenomenon may be multifactorial. It is theorized that the pancreatic tumor’s mass effect affects insulin secretion, as patients with pancreatic cancer will often have an improvement in their diabetes following pancreatectomy [20,21]. The increased production of certain peptides (adrenomedullin, vanin-1) may also promote insulin resistance [22].

This study validated the risk factors known to be associated with pancreatic cancer. Patients who developed pancreatic cancer had higher rates of pancreatitis before diagnosis, tobacco abuse, and obesity. These are known risk factors for pancreatic cancer, and the validation of these relationships reaffirms this study’s results.

The ENDPAC model demonstrated a reasonable ability to differentiate patients with type 2 diabetes from diabetics who would later develop pancreatic cancer. This ability was supported by its AUC of 0.72 and its high NPV across all cutoffs. This high NPV was a function of the overall low prevalence of pancreatic cancer in the cohort (and among new-onset diabetics in general). However, this low prevalence also decreased the PPV, as even at the optimal cutoff point, the prevalence of pancreatic cancer above the cutoff was 1.7%. At this cutoff, there would be a significant number of type 2 diabetics with false-positive scores. Despite this limitation, Sharma’s original study showed that certain medical conditions could cause false-positive scores. Among these conditions included recent steroid use, active malignancy, and end-stage disease. In a previous analysis of the ENDPAC model among patients in West Virginia, steroids were demonstrated to cause false elevated ENDPAC scores [23]. Going forward, identifying and excluding patients with these medical conditions may further improve the model’s performance.

The ENDPAC model was designed as part of a multi-step process to screen for pancreatic cancer. Patients scoring above the designated cutoff would ultimately need confirmation testing with imaging studies (endoscopic ultrasound, computed tomography) to be diagnosed with pancreatic cancer. At the optimal cutoff (a score > 1), this may not be a feasible option; 25% of the new-onset diabetics would still need to follow up imaging to identify pancreatic cancer. Given that over a million people are diagnosed with new-onset diabetes annually, confirmatory imaging’s financial cost may not be worth the benefit at this cutoff. At higher cutoffs, this becomes less of an issue as the prevalence of pancreatic cancer among patients scoring above the cutoff (the PPV of the model) increases, and fewer patients from the original cohort need follow up imaging. However, at higher cutoffs, the sensitivity of the ENDPAC model decreases. These issues may be partially alleviated by eliminating the burden of false-positive scores. As suggested in Sharma’s original study, another strategy would be to employ secondary serological testing for patients with an “intermediate” risk of pancreatic cancer (who score near the optimal cutoff) to enrich this portion of the cohort further. For example, CA 19–9 is the only FDA approved biomarker for pancreatic cancer, but it is approved for its surveillance following therapeutic outcome and not in its early detection [24]. Identifying an appropriate biomarker would help enrich the cohort of patients with an intermediate risk of developing pancreatic cancer and alleviate the financial burden of imaging.

There was a discrepancy in the ENDPAC model’s performance compared between Sharma’s original study and subsequent validation studies. In the original study, a score greater than two was set as defining a high-risk cohort, and the prevalence of patients who developed pancreatic cancer at this cutoff was 3.6%. However, in Chen’s validation study, the prevalence at this cutoff was increased to only 2.0%, and in this study, the prevalence was 1.9%. Also, the AUC of the ENDPAC model in this study (AUC = 0.72) is more comparable to Chen’s validation study (AUC = 0.75) than Sharma’s original study (AUC = 0.84). These findings reflect a decline in the predictive power of the ENDPAC model from its original study, but this is not altogether surprising as models tend to do better on the data from which they were created [25]. There are differences between the datasets that may have played a role in this discrepancy. More pancreatic cancer cases were used to validate the model in this study and Chen’s study (48 and 99 cases, respectively) than Sharma’s original study (9 cases). This small sample may have biased the estimate of the model’s performance. Also, the ENDPAC model had been derived from a predominantly Caucasian population, and the Caucasian element of Chen’s study had a higher prevalence of patients who subsequently developed pancreatic cancer compared to other races (3.0% among Caucasians compared to 2.0% among all races). In this study, Caucasians had a marginally higher prevalence of pancreatic cancer (2.0% among Caucasians compared to 1.9% among all races) at a cutoff greater than 2. Whether this is a limitation of the ENDPAC model’s generalizability will need further investigation.

The model’s performance in this study was comparable to Chen’s study despite using only one abnormal value (HbA1C > 6.5) to diagnose diabetes. There were 8,245,405 total HbA1C values in this dataset compared to only 493,482 fasting glucose values; requiring the pairing of the A1C and fasting glucose to diagnose diabetes would have dramatically decreased the number of screenable patients. In Chen’s study, it was noted that there were roughly 500,000 unique HbA1C values and 100,000 unique fasting glucose measurements annually. More patients in that study had an ENDPAC score derived from a previous (one year before diagnosis) HbA1C value than a fasting glucose value. In the real world setting, these findings suggest the ENDPAC model can, and should, be applied to a broader group that is diagnosed with diabetes through HbA1C only.

There are several limitations to this study. First, its retrospective nature meant that many patients were missing values; of the original cohort of new-onset diabetics, 14% of them had the elements needed to be scored. This quality may reflect the real-world setting that could limit the application of the ENDPAC model. Another limitation was the imputation of body height. While body height values were included in the data set, their prevalence was so low that there would be too few pancreatic cancer cases with derivable scores to test the model. To assess the deviation of the imputed height values from actual height values, we calculated the mean difference between the two values in the final cohort of patients; the mean difference among controls (n = 576) was <0.00 m ( ±0.08 m) and among cases (n = 4) was 0.02 m ( ±0.07 m). This difference was minimal, and it increased our confidence in the imputed values.

In conclusion, this study validated the ENDPAC model’s performance in identifying new-onset diabetics at risk of developing pancreatic cancer. The model increased the prevalence of pancreatic cancer in the high-risk cohort (score of 2 or greater) to 1.9%. Given the current imaging modalities available for diagnosing pancreatic cancer, screening costs may still be prohibitively expensive in the enriched cohort without a secondary screening strategy. Despite this, the model has been demonstrated to be simple, generalizable, and reasonably accurate. These features make it attractive as an initial screening tool. If the results of this study are prospectively confirmed, the ENDPAC model will be an essential first step in developing a generalized screening program for pancreatic cancer.