Abstract
INTRODUCTION:
Early detection of pancreatic ductal adenocarcinoma (PDAC) improves survival. However, screening recommendations are limited to individuals with hereditary risk, accounting for only 10% of PDAC. We explore the feasibility of developing and validating an electronic health record-based model to identify high-risk individuals for PDAC screening within the asymptomatic general population.
METHODS:
Using multivariable Cox regression, we developed a diagnostic model to predict time to PDAC within 3 years in the Veterans Health Administration. We evaluated the final model using internal and temporally separate data sets using Akaike Information Criterion, Harrell c statistic, calibration curves, and sensitivity/specificity corresponding to a 3-year risk screening threshold of 1%.
RESULTS:
Among 9,351,261 individuals, 26,119 (0.3%) developed PDAC (107.6 cases per 100,000 person-years) within 3 years. The final model included age, pancreatic cyst, pancreatitis, smoking status, history of a localized solid tumor, race/ethnicity, and body mass index. Glucose and albumin values were highly important, in addition to other metabolic, inflammatory, and liver-related laboratory values. The c statistic (95% CI) was 0.75 (0.75–0.76) in development, 0.75 (0.75–0.76) in internal validation, and 0.74 (0.73–0.75) in temporal validation. At a 3-year risk threshold of 1.0%, 11% of the population would undergo screening, capturing 30% of the PDAC cases.
DISCUSSION:
We demonstrate good model discrimination in independent data. Compared with current screening practices targeting only genetically predisposed individuals, its implementation could identify 3 times as many PDAC cases. However, predictors beyond the electronic health record (EHR) may be needed to further improve the feasibility of generalized screening.
KEYWORDS: sporadic pancreatic cancer, risk prediction, screening, early detection
INTRODUCTION
Because most cases of pancreatic ductal adenocarcinoma (PDAC) are diagnosed at advanced stages, PDAC will soon be the second deadliest cancer in the United States (1,2). Early detection of PDAC improves survival, and gastroenterology guidelines (3–5) recommend screening asymptomatic individuals with a known family history or genetic mutation, but this accounts for only 10% of all cases.
Previously, we found that an early detection strategy enriching for individuals with ≥1% PDAC risk within 3 years of new-onset diabetes was cost-effective (6). Furthermore, a population-based early detection model for clinician use should meet the following criteria: (i) use routinely available, clinically interpretable information, (ii) be noninvasive, (iii) have diagnostic accuracy, (iv) cause minimal harm, and (v) detect cancer when it is amenable to effective treatment.
Fifteen studies have focused on PDAC prediction in the asymptomatic general population (7–21). Ten studies did not report the accuracy (i.e., discrimination and/or calibration) of the model (8,9,12–18,21). Three additional studies were less clinically applicable, using cohorts and controls who were younger (baseline age: 45–49 years [20] or age at end of follow-up: 56 years [11,19]) and healthier than the at-risk population for PDAC. Two remaining studies (7,22) developed models on individuals in Kaiser Permanente Southern California (KPSC) and TriNetX. However, the follow-up interval for both was only 18 months, and the average onset of PDAC was 6 months in the first cohort. Given the duration between onset of disease and the clinical diagnosis of PDAC is estimated to be 3 years and beyond, an 18-month follow-up interval might miss a substantial number of resectable, early stage PDAC (23,24).
In this study, we leverage routinely available clinical data from the Veterans Health Administration (VA), the largest integrated healthcare system in the United States with a robust, longitudinal electronic health record (EHR). We aimed to develop and validate a diagnostic PDAC early detection model to enrich for asymptomatic individuals at risk of subclinical PDAC within the general population for one time subsequent targeted screening and identify the most salient clinical factors to personalize PDAC risk.
METHODS
Study design and source of data
We performed a retrospective cohort study within the VA, which captures demographics and longitudinal data from outpatient and inpatient encounters and associated medical diagnoses, laboratory results, and medications (25). We included Medicare data for both comorbid conditions and the outcome. The study was granted a waiver of informed consent by the West Haven VA IRB (1695518). Reporting followed transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD)+AI guidelines (see Supplementary Figure 1, Supplementary Digital Content, http://links.lww.com/CTG/B454) (26).
Study population
We included individuals ages 20–100 years old with at least 1 inpatient or outpatient encounter between October 1, 2001, and September 30, 2020 (fiscal year 2002–2020). To generate the index date, we randomly selected a visit date at least 18 months after the first VA encounter with a concurrent common laboratory value (albumin, alanine aminotransferase [ALT], aspartate aminotransferase [AST], cholesterol, creatinine, estimated glomerular filtration rate [eGFR], hemoglobin, platelets, white blood cells [WBC], glucose, and total bilirubin) during the same calendar year, representing PDAC risk for veterans in care across all years (27,28). We excluded individuals with PDAC or metastatic solid tumor at baseline.
Outcome
To assess for subclinical PDAC, our outcome was PDAC within 3 years of the index date, defined as either 1 inpatient or two 2 outpatient encounters for PDAC (see Supplementary Table 1, Supplementary Digital Content, http://links.lww.com/CTG/B454) in the VA or VA-linked Medicare data as previously used to define other VA conditions (29,30). We also included PDAC as defined by Medicare encounters given the average age of onset of PDAC is 70 years old (31) and the high prevalence of dual VA-Medicare users (32).
Predictors
We considered 43 candidate predictors based on those previously identified in the literature and routinely available in the EHR, including age, race/ethnicity, sex, body mass index (BMI) (underweight, normal, overweight, or obese), smoking status (current, former, never, or uninterpretable [contradictory or unclear smoking status]), alcohol use disorder, diabetes, acute and chronic pancreatitis, chronic hepatitis C, cholecystectomy/acute and chronic cholecystitis, gallstones, pancreatic cysts, other pancreatic diseases, Charlson comorbidity index (0, ≥1, ≥2 index points, and all individual components) (27), and common laboratory values (albumin, ALT, AST, cholesterol, creatinine, eGFR, hemoglobin, platelets, WBC, glucose, and total bilirubin). We defined medical conditions based on 1 inpatient or 2 outpatient diagnoses in the VA or Medicare data before the index date (see Supplementary Table 1, Supplementary Digital Content, http://links.lww.com/CTG/B454). Laboratory values were the closest to the index date chosen from 18 months before to 14 days after. Hepatitis C history was based on diagnostic codes and laboratory test results (33).
Missing data
We eliminated variables with greater than 30% missing values and replaced extreme laboratory values with either the first or 99th percentile to avoid undue influence from extreme values. For laboratory tests missing less than 30% of values, we substituted the median value of the laboratories, consistent with the real-world clinical assumption that a missing laboratory value is likely assumed normal and how clinicians reason through incomplete information (34).
Statistical analysis
We split our sample into development and 2 validation sets ([i] individuals with an index year of 2011 to test temporal generalizability and potential drift and [ii] random 30% of the remaining individuals for internal validation). The remaining 70% was used for development.
In the development cohort, we constructed multivariable Cox regression models with complete case analysis of nonlaboratory values to predict PDAC within 3 years. Patients were followed until the earliest of PDAC, death, 2 years after the last encounter, 3 years after baseline date, or December 31, 2021 (latest available Medicare data). First, we determined the appropriate functional forms of continuous variables, plotting associations across the range of observed values and transforming the variables using fractional polynomials as necessary. We also created a model using categorical variables for ease of clinical interpretability. We incrementally tested the candidate predictors, retaining variables with a statistically significant Z score (P < 0.05) and decrease in Akaike Information Criterion by more than 10 compared with the prior model, which could yield the maximum improvement of Harrell c statistic and identify the most parsimonious prediction model for clinical interpretability. We tested for multicollinearity of the final model using variance inflation factor (VIF).
We multiplied the regression coefficients from the final model of each variable (derived from development data only) by the values of each individual and summed them to create a linear predictor (Xβ) in each data set. Using the baseline survival at 3 years in the development data set, we derived predicted probability of PDAC based on each Xbeta. For both the development and validation data sets, we plotted calibration curves of Kaplan-Meier estimated observed PDAC probability vs predicted probability at 3 years.
For ease of interpretation, we created an individual risk prediction by rescaling Xβ to range from 0 to 100. Specifically, we divided each individual Xβ by the range of Xβ values in the development set and multiplied by 100. Using the risk scores as the sole variable input, we measured calibration in the development and validation samples by plotting observed Kaplan-Meier estimated incidence vs. risk score. We separated out the distribution of risk scores through histogram, highlighting only distributions with ≥10 events per score. Finally, as an additional sensitivity analysis, we used a Fine and Gray subdistribution hazards model to account for the competing risk of death on PDAC diagnosis.
RESULTS
Sample characteristics
We identified 9,351,261 eligible individuals, with median age 65 years (IQR 53–75), male 93%, non-Hispanic White 65%, and nearly 20% Black non-Hispanic or Hispanic (Table 1). Overall, 26,119 were diagnosed with PDAC within 3 years (107.6 cases per 100,000 person-years, median time to PDAC: 1.20 years). Of these, 10,559 PDAC cases (40.4%) were diagnosed at least 18 months after the index date. Our development (n = 6,250,709) and validation data sets (year 2011: n = 423,118; random set: n = 2,677,434) reflect the overall spreads. Overall, 8,323,199 individuals (89.0%) were available for complete case analysis.
Table 1.
Patient characteristics in development and validation cohorts
| Total (n = 9,351,261) | Development (n = 6,250,709) | Validation (Year 2011, n = 423,118) | Validation (remaining 30%, n = 2,677,434) | |
| PDAC (n, %) | 26,119 (0.3%) | 17,444 (0.3%) | 1,234 (0.3%) | 7,441 (0.3%) |
| Median age (IQR) | 65.0 (53.0–75.0) | 65.0 (53.0–75.0) | 64.0 (54.0–76.0) | 65.0 (53.0–75.0) |
| Sex (n, %) | ||||
| Male | 8,729,073 (93.3%) | 5,833,103 (93.3%) | 397,880 (94.0%) | 2,498,090 (93.3%) |
| Female | 622,188 (6.7%) | 417,606 (6.7%) | 25,238 (6.0%) | 179,344 (6.7%) |
| Race (n, %) | ||||
| White | 6,071,654 (64.9%) | 4,047,805 (64.8%) | 290,437 (68.6%) | 1,733,412 (64.7%) |
| Black | 1,307,639 (14.0%) | 874,031 (14.0%) | 59,223 (14.0%) | 374,385 (14.0%) |
| Hispanic | 459,654 (4.9%) | 308,211 (4.9%) | 19,423 (4.6%) | 132,020 (4.9%) |
| Other | 420,461 (4.5%) | 280,930 (4.5%) | 18,701 (4.4%) | 120,830 (4.5%) |
| Unknown | 1,091,853 (11.7%) | 739,732 (11.8%) | 35,334 (8.4%) | 316,787 (11.8%) |
| Median BMI (IQR; n,%) | 28.5 (25.2–32.4) | 28.6 (25.3–32.5) | 28.5 (25.2–32.3) | 28.6 (25.3–32.5) |
| Underweight | 116,584 (1.2%) | 78,111 (1.2%) | 5,109 (1.2%) | 33,364 (1.2%) |
| Normal weight | 2,010,973 (21.5%) | 1,342,611 (21.5%) | 93,140 (22.0%) | 575,222 (21.5%) |
| Overweight | 3,415,071 (36.5%) | 2,281,091 (36.5%) | 157,008 (37.1%) | 976,972 (36.5%) |
| Obese | 3,600,755 (38.5%) | 2,408,377 (38.5%) | 160,733 (38.0%) | 1,031,645 (38.5%) |
| Missing | 207,878 (2.2%) | 140,519 (2.2%) | 7,128 (1.7%) | 60,231 (2.2%) |
| Smoking (n, %) | ||||
| Never smoker | 2,615,718 (28.0%) | 1,758,899 (28.1%) | 102,432 (24.2%) | 754,387 (28.2%) |
| Current smoker | 2,222,154 (23.8%) | 1,480,354 (23.7%) | 108,609 (25.7%) | 633,191 (23.6%) |
| Former smoker | 2,540,347 (27.2%) | 1,707,493 (27.3%) | 102,284 (24.2%) | 730,570 (27.3%) |
| Uninterpretable | 944,980 (10.1%) | 635,475 (10.2%) | 37,149 (8.8%) | 272,356 (10.2%) |
| Missing | 1,028,062 (11.0%) | 668,488 (10.7%) | 72,644 (17.2%) | 286,930 (10.7%) |
| Alcohol use disorder (n, %) | 1,179,363 (12.6%) | 785,004 (12.6%) | 58,588 (13.8%) | 335,771 (12.5%) |
| Medical comorbidities (n, %) | ||||
| Myocardial infarction | 437,702 (4.7%) | 292,133 (4.7%) | 20,472 (4.8%) | 125,097 (4.7%) |
| Congestive heart failure | 944,487 (10.1%) | 633,282 (10.1%) | 39,993 (9.5%) | 271,212 (10.1%) |
| Peripheral vascular disease | 965,013 (10.3%) | 645,120 (10.3%) | 43,389 (10.3%) | 276,504 (10.3%) |
| Cerebrovascular event | 870,018 (9.3%) | 580,273 (9.3%) | 41,152 (9.7%) | 248,593 (9.3%) |
| Dementia | 289,645 (3.1%) | 194,413 (3.1%) | 11,714 (2.8%) | 83,518 (3.1%) |
| COPD | 1,574,588 (16.8%) | 1,052,557 (16.8%) | 70,734 (16.7%) | 451,297 (16.9%) |
| Connective tissue disease | 161,512 (1.7%) | 108,436 (1.7%) | 6,934 (1.6%) | 46,142 (1.7%) |
| Peptic ulcer disease | 136,005 (1.5%) | 90,924 (1.5%) | 5,658 (1.3%) | 39,423 (1.5%) |
| Diabetes | 2,401,216 (25.7%) | 1,603,958 (25.7%) | 111,562 (26.4%) | 685,696 (25.6%) |
| Hemiplegia | 96,580 (1.0%) | 64,851 (1.0%) | 4,066 (1.0%) | 27,663 (1.0%) |
| Chronic kidney disease | 2,401,216 (25.7%) | 1,603,958 (25.7%) | 111,562 (26.4%) | 685,696 (25.6%) |
| Localized solid tumor | 983,125 (10.5%) | 655,651 (10.5%) | 47,829 (11.3%) | 279,645 (10.4%) |
| HIV | 38,582 (0.4%) | 25,800 (0.4%) | 1,772 (0.4%) | 11,010 (0.4%) |
| Liver disease | 369,477 (4.0%) | 247,580 (4.0%) | 15,592 (3.7%) | 106,305 (4.0%) |
| Cholecystitis | 88,699 (1.4%) | 7,143 (1.7%) | 37,723 (1.4%) | |
| Gallstones | 410,763 (4.4%) | 272,243 (4.4%) | 21,539 (5.1%) | 116,981 (4.4%) |
| Pancreatitis | 156,023 (1.7%) | 103,535 (1.7%) | 7,943 (1.9%) | 44,545 (1.7%) |
| Pancreatic cyst | 23,968 (0.3%) | 15,963 (0.3%) | 1,053 (0.2%) | 6,952 (0.3%) |
| Chronic HCV | 273,617 (3.4%) | 180,971 (3.3%) | 15,349 (4.0%) | 77,297 (3.3%) |
This table shows the demographic and clinical characteristics of participants included in the total cohort and separated by development and 2 validation cohorts.
BMI, body mass index; COPD, chronic obstructive pulmonary disease; HCV, hepatitis C virus; PDAC, pancreatic ductal adenocarcinoma.
PDAC prediction
In the final Cox regression model from the development set, the strongest nonlaboratory predictors in order of decreasing importance (z-scores) were age, race/ethnicity, pancreatic cyst, pancreatitis, smoking status, BMI, and history of a prior localized cancer. When modeling age categorically compared with ages 50–54, the hazards of PDAC increased every 5 years until age 85. Compared with nonsmoking status, current smoking (aHR 1.35, 95% CI 1.29–1.42) or uninterpretable smoking status (aHR 1.20, 95% CI 1.14–1.26) was associated with higher PDAC risk. Overweight or obese status was negatively associated with 3-year risk of PDAC compared with individuals of normal BMI. Additional nonlaboratory predictors included peptic ulcer disease, diabetes, and alcohol use. For example, a 70-year-old male current smoker with a history of pancreatitis would have 3.69 increased HR of PDAC (3-year cumulative incidence of 1.21%) vs a 65-year-old male nonsmoker (3-year cumulative incidence of 0.38%). Multicollinearity was limited (mean VIF 1.81, max VIF 4.94).
Among laboratory values, glucose, albumin, and bilirubin contributed the most to the prediction model. Additional metabolic (cholesterol), inflammatory (WBC, hemoglobin), liver-related (AST, ALT), and kidney-related (eGFR) laboratory values were also included. We included the model using categorical variables (Table 2) and fractional polynomials (see Supplementary Table 2, Supplementary Digital Content, http://links.lww.com/CTG/B454) with similar results. Similarly, our Fine and Gray subdistribution hazards model, which accounted for the competing risk of death, showed consistent findings (see Supplementary Table 3, Supplementary Digital Content, http://links.lww.com/CTG/B454).
Table 2.
Final prediction model
| HR | 95% CI | Z | |
| Age | |||
| 20–39 | 0.04 | 0.03–0.06 | −16.14 |
| 40–44 | 0.27 | 0.21–0.35 | −9.52 |
| 45–49 | 0.49 | 0.41–0.59 | −7.71 |
| 50–54 (reference) | — | — | |
| 55–59 | 1.71 | 1.53–1.91 | 9.45 |
| 60–64 | 2.55 | 2.3–2.84 | 17.37 |
| 65–69 | 3.51 | 3.16–3.89 | 23.63 |
| 70–74 | 4.20 | 3.78–4.67 | 26.44 |
| 75–79 | 5.10 | 4.57–5.69 | 29.24 |
| 80–84 | 5.35 | 4.78–5.98 | 29.28 |
| 85–89 | 5.17 | 4.58–5.83 | 26.54 |
| 90–100 | 4.24 | 3.62–4.96 | 17.87 |
| Female | 0.79 | 0.7–0.88 | −4.14 |
| Alcohol use disorder | 0.92 | 0.87–0.98 | −2.76 |
| Smoking status | |||
| Never smoker (reference) | — | — | |
| Current smoker | 1.35 | 1.29–1.42 | 12.29 |
| Former smoker | 1.01 | 0.97–1.05 | 0.31 |
| Uninterpretable | 1.20 | 1.14–1.26 | 6.88 |
| Race/ethnicity | |||
| White (reference) | — | — | |
| Black | 1.18 | 1.12–1.25 | 6.13 |
| Hispanic | 0.83 | 0.75–0.91 | −3.86 |
| Other | 1.04 | 0.96–1.13 | 0.89 |
| Unknown | 1.90 | 1.82–1.98 | 29.93 |
| Diabetes | 1.17 | 1.13–1.22 | 7.57 |
| Pancreatic cyst | 3.90 | 3.47–4.4 | 22.47 |
| Pancreatitis | 2.28 | 2.11–2.45 | 21.85 |
| Localized solid tumor | 1.32 | 1.27–1.38 | 12.85 |
| BMI | |||
| Underweight | 1.09 | 0.97–1.24 | 1.42 |
| Normal weight (reference) | — | — | |
| Overweight | 0.88 | 0.84–0.91 | −6.19 |
| Obese | 0.78 | 0.75–0.82 | −10.59 |
| Peptic ulcer disease | 1.21 | 1.09–1.34 | 3.69 |
| Albumin (g/dL) | |||
| <3.5 (reference) | — | — | |
| 3.50–3.69 | 0.85 | 0.79–0.92 | −3.99 |
| 3.70–3.79 | 0.87 | 0.8–0.94 | −3.32 |
| 3.80–3.99 | 0.81 | 0.75–0.87 | −6.07 |
| 4.0–4.09 | 0.79 | 0.73–0.86 | −5.69 |
| 4.10–4.19 | 0.81 | 0.76–0.86 | −6.31 |
| 4.2–4.39 | 0.78 | 0.73–0.84 | −6.65 |
| ≥4.40 | 0.75 | 0.7–0.8 | −7.87 |
| ALTa (U/L) | 1.08 | 1.06–1.09 | 10.55 |
| ASTb (U/L) | 1.09 | 1.06–1.11 | 7.71 |
| Cholesterola (mmol/L) | 0.99 | 0.98–0.99 | −4.47 |
| eGFRc (mL/min/1.73 m2) | 1.17 | 1.14–1.2 | 11.85 |
| Hemoglobind (g/L) | 0.92 | 0.9–0.93 | −11.03 |
| WBC (K/mm3) | 1.05 | 1.04–1.05 | 10.30 |
| Glucose (mg/dL) | |||
| <88 (reference) | — | — | |
| 88–94 | 0.96 | 0.89–1.03 | −1.05 |
| 95–100 | 1.00 | 0.93–1.08 | 0.10 |
| 101–105 | 1.18 | 1.1–1.26 | 4.82 |
| 106–112 | 1.26 | 1.17–1.35 | 6.14 |
| 113–123 | 1.41 | 1.31–1.52 | 9.30 |
| 124–141 | 1.51 | 1.4–1.62 | 10.66 |
| 142–183 | 1.51 | 1.4–1.63 | 10.52 |
| ≥184 | 1.94 | 1.79–2.1 | 16.47 |
| Bilirubine (mg/dL) | 1.79 | 1.65–1.95 | 13.70 |
This table summarizes the multivariable associations between candidate predictors and time to PDAC in a Cox regression model.
ALT, alanine aminotransferase; AST, aspartate aminotransferase; BMI, body mass index; eGFR, estimated glomerular filtration rate; PDAC, pancreatic ductal adenocarcinoma; WBC, white blood cells.
Units by a factor of 10.
AST > 33 is linear, units by a factor of 10.
eGFR > 82 is linear, units by a factor of 10.
Hemoglobin < 14.4 is linear.
Bilirubin > 0.9 is linear.
Discrimination through c-statistic was consistent in the development (c statistic 0.75, 95% CI 0.75–0.76) and validation samples [temporal index year 2011: 0.74 (0.73–0.75), random sample 0.75 (0.75–0.76)] and within subgroups of age and race/ethnicity (see Supplementary Table 4, Supplementary Digital Content, http://links.lww.com/CTG/B454).
The predicted 3-year PDAC probability ranged from 0% to 14.1% in the development cohort vs 0% to 3.4% (year 2011) and 0% to 8.8% (random 30%) in the validation cohort. Calibration (observed vs predicted probability) was consistent across the overall population and subgroups defined by age and race/ethnicity. The top predicted interval (0.006% of the population) of the development data set had modest overprediction (Figure 1).
Figure 1.
Calibration curves across development and validation cohorts. Calibration plots compare observed and predicted probability in the overall cohort and stratified by age (<65, ≥65 years) and race (White, Black) for both the development and validation cohorts.
For ease of interpretation, we also plotted the observed 3-year probability of PDAC vs generated prediction score across the overall cohort and age and race/ethnicity subgroups. At increasing prediction scores, there was increased observed probability of PDAC in the development and validation sets (Figure 2). At a 3-year risk threshold of 1.0%, 11% of the population would undergo screening, and we could capture more than 30% of the PDAC. Additional model performance characteristics at various risk thresholds are given in Table 3. Most of the risk thresholds for the validation cohort fell within the 95% CI for the development cohort (see Supplementary Figure 2, Supplementary Digital Content, http://links.lww.com/CTG/B454).
Figure 2.
Observed probability vs prediction score. Plots of observed probability vs predicted pancreatic ductal adenocarcinoma score in the overall cohort and stratified by age (<65, ≥65 years) and race (White, Black) for both the development and validation cohorts.
Table 3.
Model test performance
| PDAC score | Development cohort | Validation cohorts | ||||||||||
| Risk of PDAC within 3 yr (%) | 95% CI (%) | Sensitivity (%) | Specificity (%) | PPV (%) | % qualifying for further screening | Risk of PDAC within 3 yr (%) | 95% CI (%) | Sensitivity (%) | Specificity (%) | PPV (%) | % qualifying for further screening | |
| 49 | 0.5 | 0.5–0.6 | 82.0 | 50.1 | 0.4 | 49.9 | 0.5 | 0.5–0.6 | 81.8 | 50.3 | 0.5 | 49.8 |
| 55 | 0.8 | 0.8–0.9 | 42.5 | 81.8 | 0.6 | 18.3 | 0.8 | 0.8–0.9 | 41.7 | 82.0 | 0.6 | 18.0 |
| 57 | 1.0 | 1.0–1.0 | 30.2 | 88.8 | 0.7 | 11.3 | 1.0 | 0.9–1.0 | 29.1 | 89.0 | 0.7 | 11.1 |
| 65 | 2.5 | 2.3–2.6 | 6.0 | 99.1 | 1.7 | 1.0 | 2.4 | 2.2–2.6 | 5.8 | 99.1 | 1.7 | 0.9 |
| 70 | 4.7 | 4.2–5.2 | 2.6 | 99.8 | 3.2 | 0.2 | 4.1 | 3.6–4.8 | 2.4 | 99.8 | 3.0 | 0.2 |
| 72 | 5.5 | 4.9–6.3 | 1.8 | 99.9 | 3.9 | 0.1 | 4.4 | 3.7–5.3 | 1.5 | 99.9 | 3.3 | 0.1 |
| 77 | 7.7 | 6.1–9.6 | 0.6 | 100.0 | 5.0 | 0.0 | 6.7 | 5.0–9.1 | 0.6 | 100.0 | 5.0 | 0.0 |
This table summarizes the performance characteristics of the model (sensitivity, specificity, PPV, % qualifying for further screening) and corresponding PDAC risk at various PDAC scores across the development and validation cohorts.
PDAC, pancreatic ductal adenocarcinoma; PPV, positive predictive value.
DISCUSSION
In this study, we leveraged routine clinical data to stratify risk of PDAC over a 3-year window among 9.4 million individuals in an asymptomatic general VA population. The most important predictors included age, race/ethnicity, pancreatic cyst, pancreatitis, smoking status, BMI, glucose, albumin, and bilirubin values. The median time to PDAC was 1.2 years, and 75% were diagnosed at least 6 months after their index date, a clinically actionable time point for early detection. Obese and overweight status were associated with decreased risk for PDAC diagnosed within 3 years, likely reflecting that our model is identifying subclinical cancer.
Our prediction model incorporates strengths of model development and highlights promising characteristics for application in early detection. First, it uses clinically available, noninvasive clinical, lifestyle, and environmental data routinely captured at a single timepoint in the integrated EHR to identify PDAC at risk thresholds previously determined to be cost-effective (6), facilitating its subsequent implementation through a clinical dashboard (35). Second, we developed our model using clinical reasoning, and clinical-based models have resulted in more successful clinical adoption (36). Third, we derived our PDAC model from the largest national cohort of individuals to date (over 9 million individuals and over 26,000 PDAC cases), representing a diverse community-based population to broaden its applicability and minimize model overfitting (i.e., selecting irrelevant predictors) or underfitting (i.e., failing to include relevant predictors). Fourth, we simulated a real world setting by randomizing the index date to generate the random population-based risk for PDAC across time and imputing median values for common laboratory values with <30% missingness similar to the way a clinician would interpret missing values. Finally, our model performed well in both temporal and internal validation.
Although a direct comparison of discrimination is only possible using a standardized data set and time interval, our study reports model discrimination similar to or higher than the majority of general population models reporting performance (9,11–14,16,19), including a model derived from KPSC and externally validated in the VA (AUC 0.68–0.71). Since the KPSC model only identified PDAC up to 18 months after the index date, half the typically cited window of opportunity for early detection, arguably our model had improved performance while identifying PDAC up to 3 years out. In addition, the KPSC model includes change variables (weight and laboratory values), making it harder to implement. A few previously reported models had higher discrimination (AUC 0.80–0.86) but included variables with limited availability (e.g., meat intake [20], Townsend index [10], exercise status) [15], or were restricted to younger, healthier populations [10,20] than those at-risk for PDAC, limiting the opportunity to capture patients developing PDAC and biasing the model toward the null.
The pancreatic cancer risk prediction model (PRISM) study (22) using TriNetX achieved good discrimination in internal validation (AUC 0.740–0.828). However, the model only predicted PDAC up to 18 months after initial date. Furthermore, TriNetX included more than 80 features many of which were less clinically interpretable/relevant (e.g., dyspnea, clarity of urine, and encounter for immunization). Finally, some included variables are not routinely available in an asymptomatic general population (e.g., carbohydrate antigen 19-9).
Our model achieved good discrimination in validation. At a 1% risk threshold, it would identify more than 30% of the associated PDAC from the general population, a greater proportion of all PDAC cases compared with current screening recommendations (37) and compared with models focused on those with new onset diabetes (24,38). Nevertheless, given PDAC is a rare event, even applying our model at the 1% risk threshold among individuals aged ≥55 year old would still require screening 11% of the general population which is not clinically applicable at this time.
Our study has limitations. Because we wanted to determine how much risk could be identified based on routine clinically available data, we did not include family history or dietary patterns. These data are available in the subset Million Veteran Program biobank (39,40), and we plan to consider them in future work. Second, veterans using the VA for their primary healthcare are more likely to be Black, younger, unmarried, less educated, and have lower household incomes than veterans who do not use the VA healthcare system (41). However, given the increased risk of PDAC among Blacks (42) and the poorer survival among individuals with lower socioeconomic status, oversampling these groups could be a strength and identify more individuals at risk for PDAC (43). Future analyses would need to consider recalibration methods should the model be applied outside VA populations. Third, the VA population is predominantly male, which may limit generalizability to women, and future studies should focus on the female population separately to gain additional insight into the potential differences in outcomes and diagnoses. Although the absolute incidence of PDAC is higher in men, there is no evidence of biological differences between the sexes (31). Finally, future iterations of modeling would incorporate formal cost-effectiveness analysis which is crucial given mixed benefits of screening among various high-risk genetic subgroups at risk of pancreatic cancer (37).
Future PDAC models might be improved by adding novel risk factors such as common genetic risk variants (44) or DNA methylation markers (45) alongside the clinical risk factors we have identified (46). Incorporating radiomics features (47,48) from preexisting imaging might also offer an innovative means of identifying individuals at high risk for subclinical pancreatic cancer (47). In addition, future analyses will prioritize external validation for the goal of evaluating generalizability and direct incorporation into the EHR as a risk score.
In summary, we developed and validated a prediction model using clinically available data to identify the most salient available risk factors for subclinical PDAC in an asymptomatic general population. Future work should focus on identifying which additional variables might further improve the feasibility of PDAC screening in the general population.
CONFLICTS OF INTEREST
Guarantor of the article: Louise Wang, MD, MSCE.
Specific author contributions: L.W., J.T., Y.X.Y., and A.J. contributed to conception and design of the study. L.W., M.S., and J.T. acquired the data. L.W. and J.T. contributed to analysis of data. L.W., J.T., M.S., R.H., C.B., Y.X.Y., and A.J. contributed to drafting the article or revising it critically for important intellectual content; and final approval of the version to be published.
Financial support: L.W. was supported by a VA CDA-2 grant (IK2-BX00589).
Potential competing interests: None to report.
IRB approval statement: The study was granted a waiver of informed consent by the West Haven VA IRB (1695518).
Study Highlights.
WHAT IS KNOWN
✓ Early detection improves survival from pancreatic ductal adenocarcinoma (PDAC).
✓ Current screening guidelines that focus on individuals with hereditary risk only identify 10% of PDAC cases.
WHAT IS NEW HERE
✓ We built a PDAC diagnostic prediction model in the asymptomatic general population using commonly available data in the electronic health record.
✓ Compared with current screening focused on individuals with hereditary risk, its implementation can identify 3 times as many PDAC cases.
✓ We identified the most salient clinical factors to inform the feasibility of screening in the general population.
Supplementary Material
ABBREVIATIONS:
- AIC
Akaike Information Criterion
- ALT
alanine aminotransferase
- AST
aspartate aminotransferasee
- BMI
body mass index
- CA
carbohydrate antigen (as in CA 19-9)
- CI
confidence interval
- COPF
chronic obstructive pulmonary disease
- eGFR
estimated glomerular filtration rate
- GFR
estimated glomerular filtration rate
- HCV
Hepatitis C virus
- HR
hazard ratio
- IQR
interquartile range
- KPSC
Kaiser Permanente Southern California
- PDAC
pancreatic ductal adenocarcinoma
- PPV
positive predictive value
- PRISM
pancreatic cancer risk prediction model
- TRIPOD
transparent reporting of a multivariable prediction model for individual prognosis or diagnosis
- VA
Veterans Health Administration
- VIF
variance inflation factor
- WBC
white blood cell
Footnotes
SUPPLEMENTARY MATERIAL accompanies this paper at http://links.lww.com/CTG/B454
Yu-Xiao Yang and Amy Justice are cosenior authors.
Contributor Information
Janet Tate, Email: Janet.Tate2@va.gov.
Melissa Skanderson, Email: Melissa.Skanderson@va.gov.
Ronald Hauser, Email: ronald.hauser@yale.edu.
Cynthia Brandt, Email: cynthia.brandt@yale.edu.
Yu-Xiao Yang, Email: yangy@pennmedicine.upenn.edu.
Amy Justice, Email: Amy.Justice2@va.gov.
REFERENCES
- 1.Rahib L, Smith BD, Aizenberg R, et al. Projecting cancer incidence and deaths to 2030: The unexpected burden of thyroid, liver, and pancreas cancers in the United States. Cancer Res 2014;74(11):2913–21. [DOI] [PubMed] [Google Scholar]
- 2.Rahib L, Wehner MR, Matrisian LM, et al. Estimated projection of US cancer incidence and death to 2040. JAMA Netw Open 2021;4(4):e214708. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 3.Aslanian HR, Lee JH, Canto MI. AGA clinical practice update on pancreas cancer screening in high-risk individuals: Expert review. Gastroenterology 2020;159(1):358–62. [DOI] [PubMed] [Google Scholar]
- 4.Syngal S, Brand RE, Church JM, et al. ACG clinical guideline: Genetic testing and management of hereditary gastrointestinal cancer syndromes. Am J Gastroenterol 2015;110(2):223–63. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 5.Sawhney MS, Calderwood AH, Thosani NC, et al. ASGE guideline on screening for pancreatic cancer in individuals with genetic susceptibility: Summary and recommendations. Gastrointest Endosc 2022;95(5):817–26. [DOI] [PubMed] [Google Scholar]
- 6.Wang L, Scott FI, Boursi B, et al. Cost-effectiveness of a risk-tailored pancreatic cancer early detection strategy among patients with new-onset diabetes. Clin Gastroenterol Hepatol 2022;20(9):1997–2004.e7. [DOI] [PubMed] [Google Scholar]
- 7.Chen W, Zhou Y, Xie F, et al. Derivation and external validation of machine learning-based model for detection of pancreatic cancer. Am J Gastroenterol 2023;118(1):157–67. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 8.Colditz GA, Atwood KA, Emmons K, et al. Harvard report on cancer prevention volume 4: Harvard cancer risk index. Risk Index Working Group, Harvard Center for Cancer Prevention. Cancer Causes Control 2000;11(6):477-88. [DOI] [PubMed] [Google Scholar]
- 9.Galeotti AA, Gentiluomo M, Rizzato C, et al. Polygenic and multifactorial scores for pancreatic ductal adenocarcinoma risk prediction. J Med Genet 2021;58(6):369–77. [DOI] [PubMed] [Google Scholar]
- 10.Hippisley-Cox J, Coupland C. Development and validation of risk prediction algorithms to estimate future risk of common cancers in men and women: Prospective cohort study. BMJ Open 2015;5(3):e007825. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 11.Kachuri L, Graff RE, Smith-Byrne K, et al. Pan-cancer analysis demonstrates that integrating polygenic risk scores with modifiable risk factors improves risk prediction. Nat Commun 2020;11(1):6084. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 12.Kim DJ, Rockhill B, Colditz GA. Validation of the harvard Cancer Risk Index: A prediction tool for individual cancer risk. J Clin Epidemiol 2004;57(4):332–40. [DOI] [PubMed] [Google Scholar]
- 13.Kim J, Yuan C, Babic A, et al. Genetic and circulating biomarker data improve risk prediction for pancreatic cancer in the general population. Cancer Epidemiol Biomarkers Prev 2020;29(5):999–1008. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 14.Klein AP, Lindström S, Mendelsohn JB, et al. An absolute risk model to identify individuals at elevated risk for pancreatic cancer in the general population. PLoS One 2013;8(9):e72311. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 15.Muhammad W, Hart GR, Nartowt B, et al. Pancreatic cancer prediction through an artificial neural network. Front Artif Intell 2019;2:2. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 16.Nakatochi M, Lin Y, Ito H, et al. Prediction model for pancreatic cancer risk in the general Japanese population. PLoS One 2018;13(9):e0203386. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 17.Pang T, Ding G, Wu Z, et al. A novel scoring system to analyze combined effect of lifestyle factors on pancreatic cancer risk: A retrospective case-control study. Sci Rep 2017;7(1):13657. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 18.Risch HA, Yu H, Lu L, et al. Detectable symptomatology preceding the diagnosis of pancreatic cancer and absolute risk of pancreatic cancer diagnosis. Am J Epidemiol 2015;182(1):26–34. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 19.Salvatore M, Beesley LJ, Fritsche LG, et al. Phenotype risk scores (PheRS) for pancreatic cancer using time-stamped electronic health record data: Discovery and validation in two large biobanks. J Biomed Inform 2021;113:103652. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 20.Yu A, Woo SM, Joo J, et al. Development and validation of a prediction model to estimate individual risk of pancreatic cancer. PLoS One 2016;11(1):e0146473. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 21.Yu Y, Tong Y, Zhong A, et al. Identification of serum microRNA-25 as a novel biomarker for pancreatic cancer. Medicine 2020;99(52):e23863. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 22.Jia K, Kundrot S, Palchuk MB, et al. A pancreatic cancer risk prediction model (prism) developed and validated on large-scale US clinical data. eBioMedicine 2023;98:104888. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 23.Luebeck EG, Curtius K, Jeon J, et al. Impact of tumor progression on cancer incidence curves. Cancer Res 2013;73(3):1086–96. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 24.Sharma A, Kandlakunta H, Nagpal SJS, et al. Model to determine risk of pancreatic cancer in patients with new-onset diabetes. Gastroenterology 2018;155(3):730–9.e3. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 25.Fihn SD, Francis J, Clancy C, et al. Insights from advanced analytics at the Veterans Health Administration. Health Aff 2014;33(7):1203–11. [DOI] [PubMed] [Google Scholar]
- 26.Collins GS, Moons KGM, Dhiman P, et al. TRIPOD+AI statement: Updated guidance for reporting clinical prediction models that use regression or machine learning methods. BMJ 2024;385:e078378. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 27.McGinnis KA, Justice AC, Marconi VC, et al. Combining Charlson comorbidity and VACS indices improves prognostic accuracy for all-cause mortality for patients with and without HIV in the Veterans Health Administration. Front Med 2024;10:1342466. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 28.Tate JP, Sterne JAC, Justice AC. Albumin, white blood cell count, and body mass index improve discrimination of mortality in HIV-positive individuals. AIDS 2019;33(5):903-12. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 29.Re VL, Lim JK, Goetz MB, et al. Validity of diagnostic codes and liver‐related laboratory abnormalities to identify hepatic decompensation events in the Veterans Aging Cohort Study. Pharmacoepidemiol Drug 2011;20(7):689–99. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 30.Bagchi A, Sambamoorthi U, McSpiritt E, et al. Use of antipsychotic medications among HIV-infected individuals with schizophrenia. Schizophrenia Res 2004;71(2-3):435–44. [DOI] [PubMed] [Google Scholar]
- 31.National Cancer Institute. SEER. Cancer stat facts: Pancreatic cancer. Accessed June 1, 2021. https://seer.cancer.gov/statfacts/html/pancreas.html [Google Scholar]
- 32.Hynes DM, Koelling K, Stroupe K, et al. Veterans' access to and use of Medicare and Veterans Affairs Health care. Med Care 2007;45(3):214–23. [DOI] [PubMed] [Google Scholar]
- 33.Park LS, Tate JP, Sigel K, et al. Association of viral suppression with lower AIDS-defining and non–AIDS-defining cancer incidence in HIV-infected veterans: A prospective cohort study. Ann Intern Med 2018;169(2):87–96. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 34.Justice AC, Tate JP, Howland F, et al. Adaption and national validation of a tool for predicting mortality from other causes among men with nonmetastatic prostate cancer. Eur Urol Oncol 2024;7(4):923–32. [DOI] [PubMed] [Google Scholar]
- 35.Atkins D, Makridis CA, Alterovitz G, et al. Developing and implementing predictive models in a learning healthcare system: Traditional and artificial intelligence approaches in the Veterans Health Administration. Annu Rev Biomed Data Sci 2022;5(1):393–413. [DOI] [PubMed] [Google Scholar]
- 36.Makridis CA, Strebel T, Marconi V, et al. Designing COVID-19 mortality predictions to advance clinical outcomes: Evidence from the Department of Veterans Affairs. BMJ Health Care Inform 2021;28(1):e100312. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 37.Wang L, Levinson R, Mezzacappa C, et al. Review of the cost-effectiveness of surveillance for hereditary pancreatic cancer. Fam Cancer 2024;23(3):351–60. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 38.Boursi B, Finkelman B, Giantonio BJ, et al. A clinical prediction model to assess risk for pancreatic cancer among patients with new-onset diabetes. Gastroenterology 2017;152(4):840–50.e3. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 39.Gaziano JM, Concato J, Brophy M, et al. Million Veteran Program: A mega-biobank to study genetic influences on health and disease. J Clin Epidemiol 2016;70(70):214–23. [DOI] [PubMed] [Google Scholar]
- 40.Peters MLB, Eckel A, Seguin CL, et al. Cost-effectiveness analysis of screening for pancreatic cancer among high-risk populations. JCO Oncol Pract 2024;20(2):278–90. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 41.Meffert BN, Morabito DM, Sawicki DA, et al. US veterans who do and do not utilize veterans affairs health care services: Demographic, military, medical, and psychosocial characteristics. Prim Care Companion CNS Disord 2019;21(1):18m02350. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 42.Tavakkoli A, Singal AG, Waljee AK, et al. Racial disparities and trends in pancreatic cancer incidence and mortality in the United States. Clin Gastroenterol Hepatol 2020;18(1):171–8.e10. [DOI] [PubMed] [Google Scholar]
- 43.SEER*Explorer application. Accessed October 4, 2024. https://seer.cancer.gov/statistics-network/explorer/application.html?site=40&data_type=1&graph_type=2&compareBy=sex&chk_sex_1=1&rate_type=2&race=1&age_range=157&stage=101&advopt_precision=1&advopt_show_ci=on&hdn_view=1&advopt_show_apc=on&advopt_display=1
- 44.Khera AV, Chaffin M, Aragam KG, et al. Genome-wide polygenic scores for common diseases identify individuals with risk equivalent to monogenic mutations. Nat Genet 2018;50(9):1219–24. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 45.Michaud DS, Ruan M, Koestler DC, et al. Epigenome-wide association study using prediagnostic bloods identifies new genomic regions associated with pancreatic cancer risk. JNCI Cancer Spectr 2020;4(5):pkaa041. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 46.Wang L, Grimshaw AA, Mezzacappa C, et al. Do polygenic risk scores add to clinical data in predicting pancreatic cancer? A scoping review. Cancer Epidemiol Biomarkers Prev 2023;32(11):1490–7. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 47.Cao K, Xia Y, Yao J, et al. Large-scale pancreatic cancer detection via non-contrast CT and deep learning. Nat Med 2023;29(12):3033–43. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 48.Mukherjee S, Patra A, Khasawneh H, et al. Radiomics-based machine-learning models can detect pancreatic cancer on prediagnostic computed tomography scans at a substantial lead time before clinical diagnosis. Gastroenterology 2022;163(5):1435–46.e3. [DOI] [PMC free article] [PubMed] [Google Scholar]