Validation of a Multiprotein Plasma Classifier to Identify Benign Lung Nodules

Introduction:

Indeterminate pulmonary nodules (IPNs) lack clinical or radiographic features of benign etiologies and often undergo invasive procedures unnecessarily, suggesting potential roles for diagnostic adjuncts using molecular biomarkers. The primary objective was to validate a multivariate classifier that identifies likely benign lung nodules by assaying plasma protein expression levels, yielding a range of probability estimates based on high negative predictive values (NPVs) for patients with 8 to 30 mm IPNs.

Methods:

A retrospective, multicenter, case-control study was performed using multiple reaction monitoring mass spectrometry, a classifier comprising five diagnostic and six normalization proteins, and blinded analysis of an independent validation set of plasma samples.

Results:

The classifier achieved validation on 141 lung nodule-associated plasma samples based on predefined statistical goals to optimize sensitivity. Using a population based nonsmall-cell lung cancer prevalence estimate of 23% for 8 to 30 mm IPNs, the classifier identified likely benign lung nodules with 90% negative predictive value and 26% positive predictive value, as shown in our prior work, at 92% sensitivity and 20% specificity, with the lower bound of the classifier’s performance at 70% sensitivity and 48% specificity. Classifier scores for the overall cohort were statistically independent of patient age, tobacco use, nodule size, and chronic obstructive pulmonary disease diagnosis. The classifier also demonstrated incremental diagnostic performance in combination with a four-parameter clinical model.

Conclusions:

This proteomic classifier provides a range of probability estimates for the likelihood of a benign etiology that may serve as a noninvasive, diagnostic adjunct for clinical assessments of patients with IPNs.

Lung nodules deemed indeterminate lack the features suggestive of benign etiologies¹ and present clinicians with a diagnostic conundrum. Patient and practitioner balance a desire for the certainty of a diagnosis against the tolerance for the unknown, while assessing the risk and yield of an invasive procedure and the likelihood of malignancy. Achieving an early diagnosis of cancer remains a clinical imperative² to improve the dismal 16% 5-year survival of nonsmall-cell lung cancer (NSCLC),³ and also to assuage the immediate concern and anxiety engendered among both patients and physicians upon the identification of such spots.^4–6 The use of computed tomography (CT) technology has grown annually with the number of nodules identified by chest CT scans approaching millions per year, most of which are benign.⁵ Patients with a nodule less than 8 mm in size or having benign radiographic features may be managed expectantly by serial CT scan surveillance.¹ However, those with larger nodules may embark on a diagnostic odyssey,⁷ including positron emission tomography (PET), transthoracic needle aspiration, bronchoscopic biopsy, and/or surgery.¹ Therefore, innovative strategies to identify benign lung nodules may mitigate the diagnostic burden of those considered indeterminate, by providing complementary data for decision-making, minimizing surgical resection of benign processes,⁸ and managing more lung nodules by radiographic surveillance.

Extensive efforts to classify pulmonary nodules using molecular biomarkers, such as DNA, RNA, and proteins, have yielded novel insights into lung cancer pathogenesis, with most having been focused largely on identifying malignant rather than benign lung nodules.^2,9–17 Proteins are attractive as biomarkers because they are the dynamic, functional molecules acting in cell communications,¹⁸ with those of greatest interest often being in low abundance in plasma or serum. Therefore, advances in bioinformatics are at the core of recent progress in the development of diagnostic biomarker classifiers.¹⁹ The current enthusiasm for introducing biomarkers into practice has also heightened expectations for rigor in their validation as diagnostic tools for a targeted or intended use population.²⁰

Our prior work²¹ applied multiple reaction monitoring mass spectrometry^22,23 for the discovery and initial validation of a classifier incorporating plasma protein expression levels to differentiate benign and malignant pulmonary nodules with 90% negative predictive value (NPV). In this study, we performed a validation of a multiprotein plasma classifier that prioritizes the diagnostic parameters of sensitivity and NPV to identify likely benign lesions in patients presenting with 8 to 30 mm lung nodules.

MATERIALS AND METHODS

Validation

The study conforms to Institute of Medicine guidelines²⁰ (Supplemental Table 4, Supplemental Digital Content, http://links.lww.com/JTO/A773) and the Standards for Reporting of Diagnostic Accuracy (STARD) criteria for reporting studies of diagnostic accuracy (Supplemental Table 5, Supplemental Digital Content, http://links.lww.com/JTO/A773).²⁴ Protein expression analyses and computational procedures were performed in a clinical laboratory adhering to the Clinical Laboratory Improvement Amendments of 1988.²⁰

Study Design and Oversight

The overall objective was to validate the performance of an 11-protein classifier in identifying lung nodules with likely benign (i.e., nonmalignant) etiologies (Supplemental Materials, Supplemental Digital Content, http://links.lww.com/JTO/A773), yielding a range of probability estimates for use as a diagnostic adjunct in clinical assessments. A retrospective, case-control study utilized multiple reaction monitoring mass spectrometry to analyze archival plasma samples from subjects enrolled in clinical studies approved by the Ethics Review Board or Institutional Review Boards at multiple institutions, using a blinded data analysis strategy. Management of clinical data complied with the Health Insurance Portability and Accountability Act of 1996.

Study Inclusion and Exclusion Criteria

The subject inclusion criteria were a minimum age of 40 years and any smoking history. The radiologic and pathologic criteria for lung nodule inclusion were a diameter between 8 to 30 mm, a histopathologic diagnosis of NSCLC or a benign process, or a clinical diagnosis of a benign etiology based on stability in size and appearance for 2 years after the baseline CT scan. The subject exclusion criteria included the lack of nodule size or histopathologic diagnosis, follow-up for less than 2 years, or a diagnosis of small-cell lung cancer. The subjects’ spirometry data and the global initiative for chronic obstructive lung disease criteria²⁵ were used to define the presence and severity of chronic obstructive pulmonary disease (COPD). The cancer and benign subgroups were matched for age, gender, smoking history, and nodule size.

Lung Nodule Protein Expression Classifier and Proteomic Analysis

The classifier consists of five diagnostic and six normalization proteins (Table 1), which were fully defined, or “locked-down,” before sample analysis. The five diagnostic proteins were refined from the 13 proteins previously shown to discriminate benign and malignant lung nodules²¹ using stable isotope standards (Supplemental Materials, Supplemental Digital Content, http://links.lww.com/JTO/A773). The six normalization proteins were identified to reduce preanalytical and analytical variations in mass spectroscopic protein quantification.^25a Plasma protein expression assays were performed as previously described²¹ using methods incorporating stable isotope standards (Supplemental Materials, Supplemental Digital Content, http://links.lww.com/JTO/A773).

TABLE 1.

Lung Nodule Protein Expression Classifier Diagnostic and Normalization Proteins

Data Analysis

The first objective was to validate the overall performance of the classifier (Table 1) in identifying benign nodules, using the method of the partial area under the curve (pAUC).²⁶ This objective required that the lower 95% confidence bound of the pAUC bounded by a sensitivity of 0.8 be higher than the corresponding pAUC (0.02) of a nonperforming classifier. The second objective was to validate the performance of the classifier in identifying benign nodules at predefined reference values using binomial testing. This objective required that the lower 95% confidence bound of the fraction of benign samples among samples whose scores were less than or equal to the corresponding reference values be higher than the fraction of benign samples in the study. The fixed-sequence procedure^27,28 was used to control the overall multitesting error rate (α = 0.05) in the study. Statistical analyses were performed using the MannWhitney and Fisher’s exact tests.

RESULTS

Study Cohort

Plasma specimens from 195 subjects with lung nodules at four institutions in different geographic regions of North America initially satisfied the study inclusion criteria, which included a minimum subject age of 40 years, but no stipulated smoking status or pack-year history. Thirty-two candidate samples were excluded due to clinical criteria, and an additional 22 samples were excluded based on laboratory criteria. A total of 141 subjects satisfied all clinical and laboratory criteria, including 78 with cancer and 63 with benign diagnoses, demonstrated no overlap with those involved in the classifier’s development, and were included in the data analysis (Fig. 1 and Table 2). There were no statistically significant differences in subject age, gender, and smoking history or lung nodule size between the cancer and benign groups.

FIGURE 1.

Flow chart of validation study plasma sample availability and exclusions based on clinical and laboratory criteria. The participating centers (n = 4) identified 195 plasma samples from 195 subjects initially satisfying the study inclusion criteria. After completion of clinical data monitoring, 32 candidate samples were excluded due to subject age less than 40 years; nodule size less than 8 mm or larger than 30 mm; nodule cancer pathology other than NSCLC; plasma sample collection after surgery; or provision of duplicate samples from the same subjects. An additional 22 samples were excluded after sample analysis based on laboratory criteria including depletion column quality control, mass spectrometry quality control, or a classifier result outside of the validated analytical range.²¹ A final total of 141 samples from 141 subjects satisfied the prespecified clinical and laboratory criteria, including 78 cancer and 63 benign samples, and were included in the data analysis.

TABLE 2.

Clinical Characteristics of Subjects and Lung Nodules

Identification of Likely Benign Lung Nodules

The use of biomarkers for diagnostic purposes requires validation of the classifier’s performance,²⁰ including the definition of the clinically relevant performance range to impact decision-making. Based on the plasma measurements of five diagnostic and six normalization proteins, the protein expression classifier yields a score from 0 to 1, with each value associated with a sensitivity and a specificity, and lower scores associated with a higher probability of a benign etiology.²¹

The overall performance of the classifier in identifying benign nodules was validated with a lower 95% confidence bound for the pAUC of 0.026, which was greater than the targeted pAUC of 0.02 (Fig. 2 and Supplemental Materials, Supplemental Digital Content, http://links.lww.com/JTO/A773). The performance of the classifier in identifying benign nodules at predefined reference values was tested individually in an increasing, stepwise manner and validated up to a reference value of 0.47 (Supplemental Materials, Supplemental Digital Content, http://links.lww.com/JTO/A773). As shown in Figure 2, the sensitivity of the classifier increases above 70% at 0.47, as the reference values decrease toward the upper-right portion of the receiver operating characteristics (ROC) curve.

FIGURE 2.

Protein expression classifier validation. The protein expression classifier yields a score between 0 and 1, with lower values associated with a higher probability of a benign etiology, based on the identification and quantification of specific plasma proteins (Supplemental Materials, Supplemental Digital Content, http://links.lww.com/JTO/A773). A, The statistical performance of the classifier was validated with a lower 95% confidence bound for the partial area under the curve (pAUC) of 0.026, which was greater than the targeted pAUC of 0.02; the performance of the classifier was validated at predefined reference values from 0.38 to 0.47. Shown are the receiver operating characteristic (ROC) curves based on the raw (circles) and fitted (grey line) data, along with the ROC curve associated with chance (grey dashed line). The maximal classifier score in the study for use as a reference value to assign a likely benign classification was 0.47, using the fixed-sequence procedure.^27,28 Therefore, the region bounded by the grey ROC curve and a sensitivity of 80% represents the classifier’s partial area under the ROC curve (AUC) (shaded in grey). The associated sensitivity and specificity values (%) for the reference value of 0.47 are indicated (dashed lines). Similar data for reference values of 0.39 and 0.36, which correspond to NPVs of 87% and 90%, respectively, are indicated (dashed lines). B, A reference value is a specific score at or below which the classifier may assign a likely benign result to a given plasma specimen, based on the measured values of the classifier’s constituent proteins. Classifier scores between 0.18 and 0.47 may be used as a reference value. Shown are the positive and negative predictive values (PPV and NPV, respectively) for classifier reference values of 0.36, 0.39, and 0.47 using cancer prevalences of 15%, 20%, 23%, and 25%. The value of 0.47 is shown based on the prespecified criterion for validation of the classifier using the fixed sequence procedure; and the values of 0.36 and 0.39 are shown to illustrate a diagnostic performance of 90% NPV and an intermediate value, respectively.

The NPV was determined for each reference value by incorporating the prevalence of NSCLC in the target population with 8 to 30 mm lung nodules, as high NPVs are useful in identifying the absence of malignancy,²⁹ e.g., benign nodules. Due to the lack of a consensus estimate of cancer prevalence for pulmonary nodules,^30–33 a population-based weight-adjusted NSCLC prevalence of 23.1% was used, incorporating data from a national, multicenter chart review study of patients with indeterminate pulmonary nodules (IPNs, n = 377; unpublished data) and the National Lung Screening Trial³⁴ (Supplemental Materials, Supplemental Digital Content, http://links.lww.com/JTO/A773). The NPV also increases from 84% at a reference value of 0.47 to 90% at a reference value of 0.36 (Fig. 2B), corresponding to sensitivity and specificity of 92% and 20%, respectively (Supplemental Fig. 4, Supplemental Digital Content, http://links.lww.com/JTO/A773). The reference value-specific NPVs may be associated with potential false-negative results, possibly resulting in malignant nodules being misclassified as likely benign, as shown in Supplemental Table 7 (Supplemental Digital Content, http://links.lww.com/JTO/A773). No single or combination of clinical parameters was identified that distinguished a given lung nodule sample in the study cohort as a potential “false-negative” result, using either 0.47 or 0.36 as a reference value.

Classifier Independence of NSCLC Predictors

The cancer and benign groups were matched for age, pack-years, and nodule size, similar to what was done in the discovery and initial validation of the classifier proteins.²¹ This enabled further assessment of the relationship of the classifier score to these clinical parameters. With respect to age, smoking history pack-years, and nodule size, none of these parameters correlated with the classifier scores (Supplemental Fig. 5 and Table 8, Supplemental Digital Content, http://links.lww.com/JTO/A773), which is consistent with our previous results.²¹

COPD Status and Classifier Scores

Because COPD is a major risk factor for lung cancer,³⁵ the potential impact of a concurrent COPD diagnosis on classifier scores was determined. Spirometry data were available for 54 (45%) of the 121 subjects with a history of tobacco use, including 33 with a cancer diagnosis and 21 with a benign diagnosis (Fig. 3A). There was no difference between the classifier scores in subjects with malignant lung nodules without or with COPD (Fig. 3B). Similarly, there was no difference observed for classifier scores in subjects with benign lung nodules in the absence or presence of COPD.

FIGURE 3.

Distribution of classifier scores between cancer and benign histopathology by tobacco use and chronic obstructive pulmonary disease (COPD). A, Shown are the correlations of classifier scores by subject gender, tobacco use history, and COPD subgrouped by the global initiative for chronic obstructive lung disease (GOLD) classification system for malignant and benign lung nodules. (Statistical analyses were performed using either the MannWhitney test* or the KruskalWallis test.** Data provided only for those nonsmoking subjects satisfying the GOLD definition of COPD.§) B, Box plots of classifier scores for subjects without and with COPD in association with lung nodules, with either a diagnosis of NSCLC (cancer) or a benign etiology (benign), demonstrate no statistically significant impact of COPD on classifier scores.

Incremental Diagnostic Value

To illustrate the potential added diagnostic value of the protein expression classifier to the clinical assessment of patients with lung nodules, a four-parameter clinical model composed of patient age, gender, and smoking history pack-years together with lung nodule size (clinical) and a model combining the clinical model with the protein expression classifier (clinical + classifier) were evaluated, as shown in Figure 4. The classifier demonstrated an area under the curve (AUC) of 0.615 (95% confidence interval [CI]: 0.524, 0.707), which was greater than that of the clinical model which showed an AUC of 0.591 (95% CI: 0.492, 0.678), whereas the AUC of the combined clinical + classifier model was 0.634 (95% CI: 0.530, 0.724). Similarly, the pAUC of the clinical + classifier model was 0.062 (95% CI: 0.035, 0.088) and greater than the pAUC of 0.041 (95% CI: 0.022, 0.062) for the clinical model alone. The clinical + classifier model significantly improved upon the performance of the clinical model alone based on the likelihood ratio test (p = 0.025). The integrated discrimination improvement index, a metric for evaluating the incremental diagnostic value of biomarkers,³⁶ for the clinical + classifier model was 0.041 (95% CI: 0.006, 0.076; p = 0.021). These data suggest that the protein expression classifier result may augment the diagnostic performance of clinical parameters used by physicians to assess lung nodules.

FIGURE 4.

Incremental diagnostic value of the protein expression classifier to a clinical lung nodule prediction model. Shown are the respective ROC curves for the clinical model⁴⁸ alone (grey dashed line), the protein expression classifier alone (black dashed line), and the model combining the protein expression classifier and the clinical model (solid line). Model performance for the clinical models was evaluated based on 1000 bootstrappings. Area under the ROC curve (AUC) and partial area under the curve (pAUC) at 80% sensitivity (shaded in grey) were calculated with bootstrap bias-corrected 95% confidence intervals (CI). The clinical model was composed of gender and the continuous variables of subject age and smoking history in pack-years (PKY) together with lung nodule size in a logistic regression model (Supplemental Materials, Supplemental Digital Content, http://links.lww.com/JTO/A773). The clinical plus classifier model included an additional parameter—the protein expression classifier score. Of the 141 subjects and lung nodules in the validation study cohort, one cancer sample and one benign sample were removed from the analysis due to missing PKY data; therefore, 139 samples were fitted in the logistic regression models, first with the clinical model alone and then with the clinical plus classifier model. The clinical model alone yielded an AUC of 0.591, whereas the clinical plus classifier model yielded an AUC of 0.634. The clinical model alone yielded a pAUC of 0.041, whereas the clinical plus classifier model yielded a pAUC of 0.062.

DISCUSSION

Although most lung nodules are benign,⁵ the decision to pursue serial CT scan surveillance is often difficult for those characterized as indeterminate (Supplemental Table 3, Supplemental Digital Content, http://links.lww.com/JTO/A773).¹ To address the need for diagnostic adjuncts to the clinical predictors of malignancy, our prior work identified a panel of plasma proteins that discriminates benign from malignant lung nodules based on high sensitivity and high NPV and involves molecular pathways implicated in lung cancer. This study demonstrates successful validation of a protein expression classifier using an independent plasma sample set, yielding a range of NPVs to estimate the probability that a patient’s lung nodule is due to a benign, i.e., nonmalignant, etiology.

By incorporating the expression values of 11 plasma proteins quantified by mass spectrometry, the classifier yields a score that may be translated into a probability that an IPN is benign. Such a probability may be useful to discriminate nodules that are benign from those that are indeterminate at the time of initial assessment.¹ The classifier includes five diagnostic proteins that play roles in diverse signaling pathways implicated in homeostasis and lung cancer pathogenesis. Expression of fructose-1,6-bisphosphate aldolase, an enzyme regulating diverse cellular functions, is upregulated in adenocarcinoma tissues and correlates with the metastatic potential of squamous cell carcinoma.^37,38 Collagen alpha-1 (XVIII) chain is an extracellular matrix protein constituent of vascular and epithelial basement membranes whose expression is strongly associated with poor outcomes in NSCLC.³⁹ Downregulation of the expression of ferritin light chain identified in the early stages of squamous cell carcinoma suggests its potential as a biomarker for early diagnosis.⁴⁰ Tissue expression of galectin-3-binding protein, which is implicated in angiogenesis and cell adhesion, motility and invasion, correlates with poor survival rates in lung cancer patients.^41,42 Thrombospondin-1 is an endogenous angiogenesis inhibitor previously implicated as a circulating diagnostic biomarker discriminatory for lung cancer.^43,44 The 141 validation study plasma samples analyzed, and the 247 patient samples in our prior study, were representative of the classifier’s target population.²¹ Achieving the first-validation objective based on the partial AUC of the ROC curve enabled optimization of the classifier’s sensitivity, and achieving the second-validation objective defined the range of classifier scores used to derive the NPVs, based on the associated sensitivity and a weight-adjusted estimate of NSCLC prevalence for the target population. The importance of cancer prevalence on NPV was demonstrated by comparing the classifier NPVs based on the observed prevalence in the study cohort to the weight-adjusted prevalence estimated for the target population (Supplemental Fig. 4, Supplemental Digital Content, http://links.lww.com/JTO/A773). This study’s cancer prevalence of 55% is artifactually high given the case-control study design and is not representative of the cancer prevalence in the overall population of patients with IPNs. By contrast, use of the 23% cancer prevalence estimate for the target population allowed the calculation of NPVs (for the range of classifier reference values) that would more likely be observed in clinical practice. The potential impact of lower or higher prevalence of NSCLC on the protein expression classifier is shown in Figure 2B, demonstrating that a lower cancer prevalence increases the classifier’s NPV at each reference value, whereas a higher cancer prevalence decreases its performance. By contrast, the corresponding low positive predictive values based on the associated specificities were anticipated due to the prioritization of identifying benign rather than malignant nodules. Thus, the proteomic analysis of a patient’s plasma based on the classifier may be used to derive a probability, ranging from an NPV of 84% that a given individual’s lung nodule is likely due to a benign etiology.

Several aspects of the data suggest that the protein expression classifier may be a useful diagnostic adjunct to the current paradigm for assessing indeterminate lung nodules. There was no correlation between classifier scores and the clinical parameters of age, smoking history pack-years, or nodule size for this cohort, which was also previously demonstrated for this biomarker panel.²¹ The independence of the classifier from these clinical predictors enabled it to augment the diagnostic performance of a four-parameter clinical model as demonstrated by an increase in the AUC of the combined model. The addition of other clinical and imaging parameters, such as nodule location, shape, or structure, may further augment the performance of such models. The lack of correlation between COPD global initiative for chronic obstructive lung disease status and classifier scores suggests that the classifier’s clinical performance may be minimally influenced by a patient’s concurrent diagnosis of mild or moderate COPD. Furthermore, the range of NPVs for the classifier reference values has a minimum of 84%, correlating with a false-negative rate (FNR) of 16% or lower; these classifier performance data compare favorably with the 4% to 28% FNR for quantitatively interpreted PET scans, the 10% to 30% FNR for transthoracic needle biopsy, and the 30% to 70% FNR for bronchoscopic biopsy.¹

The potential limitations of this study derive from specifics of the experimental design relating to the classifier performance priorities and molecular biomarkers. As illustrated by the classifier ROC curve and the associated AUC and pAUC, the developmental prioritization of sensitivity allowed us to develop a classifier with high NPVs, which allows for the accurate identification of a likely benign lung nodule. However, the emphasis on sensitivity and high NPV throughout the discovery and validation of the classifier’s constituent proteins yielded a specificity ranging from 48% to 20% for NPVs ranging from 84% to 90%, respectively, demonstrating that the classifier may identify approximately half of the lung nodules that are benign. Therefore, a higher probability of an accurate assignment of a nodule as likely benign is accompanied by a lower proportion of nodules with that probability. Clinicians may need to consider this performance compromise for the classifier, not only in comparison with similar issues with other diagnostic modalities, but also with respect to their potential morbidities. In addition, the use of archival samples from academic centers, though representative of the target population, may raise questions about the classifier’s prospective performance in the general population. The Institute of Medicine guidelines recognize and sanction the use of archival biospecimens due to the inherent difficulties in their acquisition and their limited availability.²⁰ This limitation is further mitigated for the validation cohort by the analysis of samples from geographically dispersed centers with diverse patient populations, including two not participating in the classifier’s discovery. Nevertheless, future evaluations of the proteomic expression classifier in prospective lung nodule studies may clarify some of these performance issues. Additional limitations in the study cohort include the relative preponderance of adenocarcinoma among the cancer group, the lack of granuloma histologic subtyping, and the size of the study subcohort demonstrating the lack of an impact of COPD status on classifier scores; however, the number of subjects in the validation compares favorably with other proteomic investigations of COPD biomarkers.^45–47

In summary, this work validates the performance of a proteomic classifier that identifies likely benign lung nodules with a high NPV, in patients with a minimum age of 40 who present with a lung nodule 8 to 30 mm in diameter. The data suggest that the classifier may serve as a noninvasive, objective, biology-based, and quantitative diagnostic adjunct to current modalities, such as PET scan, that may help decrease the number of lung nodules categorized as indeterminate and facilitate their diagnostic triage to surveillance imaging. Incorporation of the classifier result early during the initial clinical assessment may provide assurance about the likelihood of a benign etiology, with further reassurance provided by radiographic stability on subsequent chest CT scans. The noninvasive advantage of a blood-based molecular diagnostic test for lung nodules may preclude a diagnostic odyssey⁷ of successively more invasive procedures, reassure patients and physicians about surveillance decisions, decrease concern and anxiety about malignancy, and mitigate the morbidity and costs associated with biopsies and surgery that unfortunately often lead to benign diagnoses.¹

ACKNOWLEDGMENTS

The authors acknowledge the participation of the subjects and clinical research groups at the participating institutions and the contributions of the project teams at Integrated Diagnostics.

Supplementary Material

jto-10-629-s001.pdf ^{(1.8MB, pdf)}