Multianalyte Serum Biomarker Panel for Early Detection of Pancreatic Adenocarcinoma

Abstract

PURPOSE

We determined whether a large, multianalyte panel of circulating biomarkers can improve detection of early-stage pancreatic ductal adenocarcinoma (PDAC).

MATERIALS AND METHODS

We defined a biologically relevant subspace of blood analytes on the basis of previous identification in premalignant lesions or early-stage PDAC and evaluated each in pilot studies. The 31 analytes that met minimum diagnostic accuracy were measured in serum of 837 subjects (461 healthy, 194 benign pancreatic disease, and 182 early-stage PDAC). We used machine learning to develop classification algorithms using the relationship between subjects on the basis of their changes across the predictors. Model performance was subsequently evaluated in an independent validation data set from 186 additional subjects.

RESULTS

A classification model was trained on 669 subjects (358 healthy, 159 benign, and 152 early-stage PDAC). Model evaluation on a hold-out test set of 168 subjects (103 healthy, 35 benign, and 30 early-stage PDAC) yielded an area under the receiver operating characteristic curve (AUC) of 0.920 for classification of PDAC from non-PDAC (benign and healthy controls) and an AUC of 0.944 for PDAC versus healthy controls. The algorithm was then validated in 146 subsequent cases presenting with pancreatic disease (73 benign pancreatic disease and 73 early- and late-stage PDAC cases) and 40 healthy control subjects. The validation set yielded an AUC of 0.919 for classification of PDAC from non-PDAC and an AUC of 0.925 for PDAC versus healthy controls.

CONCLUSION

Individually weak serum biomarkers can be combined into a strong classification algorithm to develop a blood test to identify patients who may benefit from further testing.

INTRODUCTION

Pancreatic ductal adenocarcinoma (PDAC) is one of the most highly lethal diseases resulting in more than 47,000 deaths in the United States annually and nearly 460,000 deaths worldwide.^1,2 This number is expected to rise³ with projections that PDAC will be the second leading cause of cancer deaths by 2030.⁴ Surgical resection remains the mainstay of curative therapy, with 5-year survival rising to 25% for patients who present with resectable disease.⁵ Advances in pancreatic surgery and the expanded use of neoadjuvant chemotherapy have increased the fraction of patients eligible for surgery and decreased operative morbidity^6-8; however, the majority of patients still present with advanced, unresectable disease for which treatments remain ineffective.⁹ Five-year survival for PDAC remains around 10%, improved from 5% survival rates 20 years ago.¹⁰

CONTEXT

• Key Objective

• To develop a blood-based screening tool to discriminate early-stage pancreatic adenocarcinoma from benign pancreatic disease and healthy subjects.

• Knowledge Generated

• We demonstrate that a multianalyte panel of serum markers that are weak predictors individually could be combined using machine learning into an effective classification tool. The tool showed high specificity and actionable sensitivity in an independent validation cohort.

• Relevance

• This approach could be used as the basis of a primary screening test developed for detection of early-stage pancreatic ductal adenocarcinoma to suitably inform the need for secondary screening using imaging, particularly in groups at high risk for developing pancreatic adenocarcinoma.

An accurate, cost-effective screening protocol could substantially increase the number of patients eligible for surgical resection of newly diagnosed PDAC and greatly improve current PDAC-associated oncologic outcomes. Available evidence suggests at least a 5-year period between the development of malignant founder cells and acquisition of metastatic capacity, offering a window of opportunity for identification of early-stage, potentially curable disease.¹¹ Screening for PDAC is a controversial topic, however, with no currently effective screening tool for detecting early tumors or asymptomatic tumors in the general population.¹² The US Preventative Services Task Force recently discouraged screening for pancreatic cancer in asymptomatic adults as the risks of harm currently outweigh the benefits.¹³

A blood-based test may be useful as a primary screening tool for assessing the need for secondary screening by imaging. Limiting screening to groups with high-risk for developing PDAC would increase the pretest probability and reasonably offset the risk of false-positive diagnoses and costs with the survival benefits of early detection. Individual biomarkers, such as carcinoembryonic antigen and cancer antigen (CA) 19-9, lack the accuracy to be used independently for early-stage PDAC screening; however, a blood-based diagnostic panel relying on multiple biomarkers could be developed to improve sensitivity and specificity of a screening assay^14,15 and could initially be deployed as a prefilter to inform the need for secondary screening by imaging in high-risk groups.

This study describes the performance of a multianalyte, blood-based primary screening test developed for detection of early-stage PDAC to suitably inform the need for secondary screening using imaging. Potential analytes were assembled from a large body of prior studies that investigated circulating biomarkers in early-stage disease.^{16-30,31-50,51-71} We used statistical learning to develop classification algorithms using the relationship between classes on the basis of the changes across analytes. A diagnostic algorithm was devised in a large development cohort and subsequently validated in an independent sample set.

MATERIALS AND METHODS

Subjects and Sample Collection

This study retrospectively evaluated a continuous set of prospectively collected samples from cases and controls and used clinical diagnoses as the reference standard. All patients presenting with suspicion of pancreatic disease and accompanying adults to serve as healthy control subjects were approached for enrollment in a research protocol at the Huntsman Cancer Institute between 2005 and 2019. Blood was collected during normal clinic hours from consenting adults and processed for serum, aliquoted, and frozen within 2 hours of collection. Fasting status was not imposed. Serum samples were stored at –80°C for <5 years or –140°C for <10 years before being assayed. Available samples were periodically queried for inclusion in this study, and samples from all patients meeting the inclusion criterion were assayed. Cases were chosen for inclusion if they had cytologically confirmed stage IA, IB, IIA, or IIB PDAC with an available treatment-naïve serum sample. Control samples from patients with benign pancreatic disease included patients with chronic pancreatitis and intraductal papillary mucinous neoplasm (IPMN, n = 94). Samples for IPMN cases were included if the patient had 3 years of subsequent follow-up without intervention for progression. Healthy control samples were included from accompanying adults (independent of the diagnosis of the primary case) and cases obtained from the University of Salerno (n = 39). Additional control cases included excess sera obtained from a regional reference laboratory. Subject characteristics are provided in Table 1. Informed written consent was obtained from each subject enrolled in the research protocol, and all studies were performed with the approval of the Institutional Review Board at the University of Utah in accordance to the principles of Declaration of Helsinki and the US Common Rule.

TABLE 1.

Subject Characteristics

Enzyme-Linked Immunosorbent Assay

A total of 837 serum samples were assayed for 31 analytes by enzyme-linked immunosorbent assay. Samples were processed in four batches over the course of 5 years. Commercially available kits (Supplementary Table S1, Data Supplement) were used for analyte quantification according to the manufacturers recommended protocols, including primary antibody optimization, where appropriate. The raw data were log transformed and adjusted for sex, age, and batch (fitting a natural spline with 2 df) using a linear model. The resulting adjusted data were normalized by centering and dividing by the standard deviation of the healthy control data.

Data Analysis

Complete data were available for all predictors, and imputation was not required. Statistical analyses and modeling were performed using R version 3.6.1.⁷² For diagnostic performance of individual analytes, areas under the receiver operating characteristic (ROC) curve (AUC) were determined. Bootstrap resampling was used to estimate results from repeated analyses. A random sampling with replacement of calibration samples was performed 2000 times, and the range of values was then recorded and compared with the average result. Ensemble models were built using the caretEnsemble package in R⁷³ by stacking using gradient boosting. For modeling, all data with known class in the 837-sample data set were adjusted together before randomly separating into training (80%) and test (20%) sets. Discriminate algorithms were developed from the training data set for dichotomous classification of controls (including healthy, chronic pancreatitis, and benign IPMN patients) and early-stage PDAC cases. Using the training data set, multiple models were sequentially built to optimize AUC, sensitivity, and specificity, and the resulting probabilities averaged for all models. Method selection and tuning parameters for individual modeling methods were evaluated in an intermediate data subset consisting of 74 healthy control, 60 chronic pancreatitis, 76 IPMN, and 122 early-stage PDAC cases. Individual model performance was assessed by 10-fold cross validation. Ensemble classification models involved first optimizing submodels using individual classification methods ('glmnet', 'svmRadial', 'rf', 'nnet', 'knn' packages in R, Supplementary Table S2, Data Supplement) and then optimizing classification by combining the individual methods by stacking (gbm package in R). Again, performance of the ensemble models was assessed by 10-fold cross validation. Ensemble classification models were then used to predict class probabilities to the samples in the test set using a probability of 0.5 as the classification threshold. The final classification model was then applied to the 186-sample validation set after data adjustment.

RESULTS

Analyte Selection and Performance

Potential analytes were identified from prior studies that evaluated serum or plasma biomarkers in premalignant lesions, prediagnostic samples, or early-stage disease (Supplementary Table S3, Data Supplement). Candidate biomarkers were selected for inclusion in the panel if identified in two or more independent experiments from blood products in early-stage PDAC clinical samples. A subset of promising candidates that did not meet this standard were evaluated in pilot studies, and candidate biomarkers were included for further analysis if the analyte was significantly elevated in early-stage PDAC cases relative to healthy control subject (Supplementary Table S4, Data Supplement). A total of 31 analytes were selected for inclusion in the biomarker panel (Table 2). Most of the analytes are secreted and/or function within the extracellular space, and many participate in diverse biological processes associated with cancer (Supplementary Table S5, Data Supplement).

TABLE 2.

Analytes Included in Final Panel

Levels of the selected analytes were measured in a development set of 837 serum samples from treatment-naïve patients with resectable PDAC, subjects with benign pancreatic disease, and subjects who were apparently healthy (Table 1). The diagnostic performance of the individual biomarkers was evaluated by ROC analyses (Supplementary Table S6, Data Supplement). Twenty-nine of the 31 analytes yielded significantly higher area under the ROC curve than predicted by chance for at least one of the comparisons between PDAC and benign IPMN, chronic pancreatitis, or healthy controls demonstrating their potential for diagnostic discrimination of early-stage PDAC. Two analytes (ANG and MMP2) did not yield significant discrimination in this data set despite compelling evidence in the literature. Since the approach allows for interactions between analytes and because elevation of an individual analyte may occur in some PDAC cases, but not reach significance in aggregate data, ANG and MMP2 were included in the data sets for algorithm development, testing, and validation. In general, the individual analytes were weak classifiers for discriminating PDAC from controls.

Diagnostic Development and Performance

In an attempt to identify a stronger classifier, we used a machine learning approach to combine the information from the individual analytes (Fig 1). The approach proceeded in two steps. First, age, sex, and the 31 analyte levels in samples were provided as predictors to individual machine learning methods. The methods were chosen to represent dissimilar learning techniques,⁷⁴ and pilot studies indicated that all predictors were used to some extent by one or more of the methods (Supplementary Table S7, Data Supplement). The individual methods were used to assign two-class probabilities (early-stage PDAC or non-PDAC control) for each case by optimizing ROC AUC via bootstrap resampling. Second, the outputs of the individual methods were combined by re-sampling into an ensemble model to assign the final class probabilities. The ensemble model was robust to sequential elimination of the individual methods, but each method contributed to model accuracy when specific combinations were considered (Supplementary Table S8, Data Supplement).

FIG 1.

Machine learning schematic for diagnostic classification. Predictors were fed to individual machine learning methods and used to assign two-class probabilities (early-stage PDAC or non-PDAC control) for each case. Outputs of the individual methods were combined into an ensemble model to assign the final class probabilities. GBM, generalized boosted regression model; GLMnet, lasso and elastic-net regularized generalized linear model; KNN, k-nearest neighbors; NNET, neural network; PDAC, pancreatic ductal adenocarcinoma; RF, random forest; SVM, support vector machines with radial basis function kernel.

The development set of 837 serum samples were randomly split into a training set representing 80% of the cases and a hold-out test set of 20% of cases. The two-step machine learning approach was applied to the training set to develop a diagnostic algorithm. The probabilities assigned by the algorithm to the training set resulted in a sensitivity of 92.8%, correctly identifying 141 of 152 PDAC cases. The algorithm yielded a specificity of 100%, correctly identifying all 358 healthy control cases. For benign pancreatic disease, the algorithm correctly identified all 82 chronic pancreatitis cases but misidentified 1 of 77 IPMN cases for an overall specificity of 99.8% for identification of non-PDAC subjects (Table 3). The high specificity of the algorithm was a desirable result, given the rarity of the disease, but is likely a consequence of the higher ratio of controls to cases rather than a design feature since overall diagnostic accuracy (case versus control) was used to optimize the algorithm.

TABLE 3.

Diagnostic Performance

For objective assessment of algorithm accuracy, the resulting algorithm was applied to unused samples from the hold-out test set resulting in a sensitivity of 63.3%, correctly identifying 19 of 30 PDAC cases and a specificity of 97.1% (100 of 101 healthy controls). All 17 IPMN cases were correctly identified, as were 16 of 18 chronic pancreatitis cases, yielding an overall specificity of 96.4% for identification of non-PDAC cases (Table 3). ROC analysis illustrates that the diagnostic ability of the analyte panel for discrimination of PDAC from healthy controls (AUC = 0.944, Fig 2A) was greater than that of CA 19-9 alone (AUC = 0.878, Fig 2C) and for discrimination of PDAC from non-PDAC controls (analyte panel AUC = 0.920, Fig 2B v CA 19-9 AUC = 0.861, Fig 2D).

FIG 2.

Diagnostic performance in the test set. Receiver operating characteristic curves describing the ability of the analyte panel for discrimination of (A) early-stage PDAC cases from (B) healthy controls and all non-PDAC controls including chronic pancreatitis, benign IPMN, and healthy controls (C and D) compared with similar discrimination by CA 19-9 alone in the 20% hold-out test sample set. Bars represent 95% CIs from 2000 bootstrap iterations. AUC, area under the curve; CA, cancer antigen; ChPT, chronic pancreatitis; CON, healthy control; IPMN, intraductal papillary mucinous neoplasm; PDAC, pancreatic ductal adenocarcinoma.

Diagnostic Validation

The experimental design required that clinical information be available for model development, but the validation set was evaluated in the fixed algorithm in a blinded manner. Additional samples, collected over a period of 18 months after the final analyte panel algorithm was constructed using the development set (Table 1), were used to evaluate the diagnostic utility of the algorithm in a real-world setting. The 31 analytes were measured in 186 additional serum samples representing 73 PDAC, 32 chronic pancreatitis, and 41 cystic lesions (27 IPMN, 5 mucinous cystic neoplasm, and 9 serous cystadenoma) cases and 40 samples from healthy control subjects. These data, when supplied to the analyte panel algorithm, correctly identifying 53 of the 73 PDAC cases resulted in a sensitivity of 72.6%, which was better performance than the test set. Classification of controls in the validation set was consistent with results in the test set with a specificity of 95% (38 of 40) for healthy controls and 95.6% (108 of 113) for non-PDAC controls. In a rigorous validation using only those cases that satisfied the same inclusion criterion used for the development set (32 early-stage PDAC, 32 chronic pancreatitis, 40 healthy controls [cystic lesions had less than the required 3 years follow-up]), the algorithm yielded a sensitivity of 81.5% (26 of 32) and specificity of 95% (38 of 40) for healthy controls and 95.8% (69 of 72) for non-PDAC controls.

For the validation set, all cases presenting with pancreatic disease were evaluated. Although the development set only included early-stage resectable PDAC cases, the validation set included >50% late-stage cases (Table 1), reflecting the expected distribution of stages at presentation. For cases with known stage, the analyte panel sensitivity was higher for stage I-II cases (81.3%) than for stage III-IV cases (63.9%), although the comparison was not significant (P = .27 by Fisher's exact test). The development set also had the requirement that IPMN cases have at least 3 years of follow-up with no evidence of progression or high-grade dysplasia, which was not required for the validation set. Thus, cases with cystic lesions in the validation set may have included cases with more advanced disease, which cannot be excluded for the two cystic lesion cases in the validation set, both IPMN, that were classified as PDAC by the algorithm.

As with the test set, ROC analyses of the validation set showed greater diagnostic ability of the analyte panel for discrimination of PDAC than that of CA 19-9 alone. The analyte panel yielded an AUC of 0.925 for binary discrimination of PDAC from healthy controls (Fig 3A) compared with an AUC of 0.891 for CA 19-9 (Fig 3C). For discrimination of PDAC from all non-PDAC controls, the analyte panel had an AUC of 0.919 (Fig 3B) with CA 19-9 yielding an AUC of 0.874 (Fig 3D). For those PDAC cases that had CA 19-9 levels below the diagnostic threshold of 37 U/mL, the algorithm correctly classified 8 of 25 cases. These results confirm that the algorithm devised by machine learning in the development set was consistent in an independent sample set, show improvement over individual biomarkers, and suggest that the test may be useful for routine screening for early-stage disease in susceptible populations.

FIG 3.

Diagnostic performance in the validation set. Receiver operating characteristic curves describing the ability of the analyte panel for discrimination of (A) early-stage PDAC cases from (B) healthy controls and all non-PDAC controls including chronic pancreatitis, cystic lesions (IPMN, MCN, SCA), and healthy controls (C and D) compared with similar discrimination by CA 19-9 alone in the independent validation sample set. Bars represent 95% CIs from 2000 bootstrap iterations. AUC, area under the curve; CA, cancer antigen; ChPT, chronic pancreatitis; CL, cystic lesion; CON, healthy control; IPMN, intraductal papillary mucinous neoplasm; PDAC, pancreatic ductal adenocarcinoma.

Our approach was to use machine learning to uncover the interactions and structure of the data that would indicate the presence of early-stage disease. Changes in levels in individual analytes because of age or sex could potentially mask this structure. Despite adjusting for these factors in individual analytes, it was still possible that age and sex would contribute as effective predictors in the ensemble model, which seems to be justified by our variable importance subanalysis (Supplementary Table S7, Data Supplement). Diagnostic performance without age/sex preprocessing had slightly poorer overall performance (Supplementary Table S9, Data Supplement), although the results were reasonable and probably actionable.

DISCUSSION

Potential biomarkers evaluated for this study were previously identified as elevated in premalignant or early-stage PDAC and largely validated in an extensive development sample set. The selected biomarkers participate in diverse biological processes, including angiogenesis, apoptosis, immune response, inflammatory response, and extracellular signal transduction. One barrier to diagnostic accuracy is that biomarker elevation may indicate unrelated conditions, but since these processes are also relevant to cancer, a large combined panel should be less ambiguous. A serial screening tool, deployed annually or biannually, would tend to obviate any unrelated acute condition that would be expected to resolve over the course of months unless it was also related to a chronic condition such as cancer. A second barrier to diagnostic accuracy is heterogeneity that may arise from biological, behavioral, and temporal differences between cases and controls. Attempts to find single or small sets of biomarkers would be more prone to the confounding of unrelated conditions and disease heterogeneity.

All predictors that met initial selection criterion were included throughout training, testing, and validation phases with the expectation that different subsets of biomarkers would more effectively discriminate cases and controls by accommodating disease heterogeneity.^14,75 The fact that all predictors contributed to diagnostic models supports this expectation (Supplementary Table S7, Data Supplement) and the differential utilization of predictors by the individual methods illustrate that different combinations of biomarkers contributed to discrimination. Derivation of the diagnostic algorithm was designed to be representative, not exhaustive, with the intent of minimizing overfitting and devising an extensible algorithm.

The goal of machine learning is to reveal underlying structure of the data, in this case to assign binary probabilities for designation as either case or control classification. Selection of a single method optimized on one data set may emphasize set-specific or spurious aspects of the data organization that may not faithfully represent the data structure for a different data set. Ensemble methods seek to overcome the limitations of a single machine learning method by using resampling to evaluate performance of multiple machine learning models and then generate a combination of the models that optimizes test performance. This approach has been proven to be as accurate or better than the best individual component.⁷⁶ A general risk for machine learning approaches is overfitting to a data set that is unrepresentative to the general population; however, we found that our model gave almost identical AUC between test and validation, indicating that this algorithm may be generalizable. That bootstrap accuracy of the test set was 0.920 (AUC), and the accuracy of the independent validation set was nearly identical (AUC = 0.919), suggesting that the tradeoffs between undercomputing and overfitting were reasonable. Consistent with expectation, the algorithm performance was greater than the best individual biomarker, CA 19-9, although CA 19-9 performed very well in our sample sets as CA 19-9 assays achieved the highest range of historical accuracy.⁶⁰

Ultimately, proof of efficacy will require prospective evaluation in a larger population demonstrating not only improved survival but also decreased mortality. In addition, as this study was largely a single institution investigation, the results will require validation in a multi-institutional cohort to increase external validity. Any screening test for PDAC must be highly specific to avoid false-positive screens leading to increased morbidity, cost, and emotional distress associated with secondary screening tests and treatment.⁷⁷ The recommendation that screening be limited to high-risk groups⁷⁸ would increase pretest probability and reduce the number of false-positive diagnoses as PDAC has a far lower prevalence in the general population than other commonly screened malignancies. Screening with imaging is relatively low risk.^79,80 However, screening with primary imaging is expensive.^81-83 The analytes and algorithms devised in this study may be useful as a primary screening tool for assessing the need for secondary screening by imaging while limiting concerns of unnecessary morbidity caused by false-positive determinations. The analyses routinely yielded the desirable high specificity, although with lower, but actionable sensitivity. In a serial screening program, the test will have the opportunity to increase overall sensitivity by re-evaluation of negative initial tests.

The analyses show that the general approach to PDAC screening is valid, that our samples generate reproducible signals, and that novel analytes contribute to disease classification. This 31-biomarker assay may be helpful in developing a useful tool for identifying early-stage, asymptomatic PDAC, particularly for screening in high-risk patient populations to guide the use of screening imaging. The results also serve to illustrate the proposed analytical methods which allows for interaction between analytes, subsets, and heterogeneous biological response among subjects. Finally, the approach allows for improvement through incorporation of additional biomarkers, including signals from cell-free DNA, circulating tumor cells, and demographic factors.

PREPRINT VERSION

Preprint version available on https://www.medrxiv.org/content/10.1101/2022.03.03.22271867v1.

DATA SHARING STATEMENT

Deidentified data generated in this study are available within the article and its supplementary data files.

AUTHOR CONTRIBUTIONS

Conception and design: Matthew A. Firpo, Josh Bleicher, Courtney L. Scaife, Sean J. Mulvihill

Financial support: Sean J. Mulvihill

Administrative support: Sean J. Mulvihill

Provision of study materials or patients: Ignacio Garrido-Laguna

Collection and assembly of data: Matthew A. Firpo, Gayatri D. Khanderao, Alessandra Rosati, Katherine E. Poruk, Sama Kamal, Liberato Marzullo, Margot De Marco, Antonia Falco, Armando Genovese, Jessica M. Adler, Douglas G. Adler, Kajsa E. Affolter, Courtney L. Scaife, M. Caterina Turco, Sean J. Mulvihill

Data analysis and interpretation: Matthew A. Firpo, Kenneth M. Boucher, Josh Bleicher, Vincenzo De Laurenzi, Ignacio Garrido-Laguna, Courtney L. Scaife, M. Caterina Turco

Manuscript writing: All authors

Final approval of manuscript: All authors

Accountable for all aspects of the work: All authors

AUTHORS' DISCLOSURES OF POTENTIAL CONFLICTS OF INTEREST

The following represents disclosure information provided by authors of this manuscript. All relationships are considered compensated unless otherwise noted. Relationships are self-held unless noted. I = Immediate Family Member, Inst = My Institution. Relationships may not relate to the subject matter of this manuscript. For more information about ASCO's conflict of interest policy, please refer to www.asco.org/rwc or ascopubs.org/cci/author-center.

Open Payments is a public database containing information reported by companies about payments made to US-licensed physicians (Open Payments).