A Blood-Based Proteomic Classifier for the Molecular Characterization of Pulmonary Nodules

Abstract

Each year millions of pulmonary nodules are discovered by computed tomography and subsequently biopsied. As the majority of these nodules are benign, many patients undergo unnecessary and costly invasive procedures. We present a 13-protein blood-based classifier that differentiates malignant and benign nodules with high confidence, thereby providing a diagnostic tool to avoid invasive biopsy on benign nodules. Using a systems biology strategy, 371 protein candidates were identified and a multiple reaction monitoring (MRM) assay was developed for each. The MRM assays were applied in a three-site discovery study (n = 143) on plasma samples from patients with benign and Stage IA cancer matched on nodule size, age, gender and clinical site, producing a 13-protein classifier. The classifier was validated on an independent set of plasma samples (n = 104), exhibiting a high negative predictive value (NPV) of 90%. Validation performance on samples from a non-discovery clinical site showed NPV of 94%, indicating the general effectiveness of the classifier. A pathway analysis demonstrated that the classifier proteins are likely modulated by a few transcription regulators (NF2L2, AHR, MYC, FOS) that are associated with lung cancer, lung inflammation and oxidative stress networks. The classifier score was independent of patient nodule size, smoking history and age, which are risk factors used for clinical management of pulmonary nodules. Thus this molecular test can provide a powerful complementary tool for physicians in lung cancer diagnosis.

Introduction

Computed tomography (CT) identifies millions of pulmonary nodules annually with many being undiagnosed as either malignant or benign (1–3). In many cases, histopathological diagnosis by biopsy techniques such as fine needle aspiration is impossible (due to nodule location) or inconclusive (due to small nodule size). The vast majority of these nodules are benign, but nevertheless many patients with benign nodules undergo unnecessary procedures. It is estimated that only 20% of patients with lung nodules undergoing biopsy or surgery actually have a malignant lung nodule (4). Consequently, there is a high unmet need for a non-invasive clinical test that can discriminate between benign and malignant nodules (5, 6).

The performance and development requirements for a diagnostic test to mitigate the use of invasive and costly medical procedures for lung nodule evaluations are as follows:

First, physicians require a negative test result (i.e. “benign”) to be correct with high probability (over 90%) to ensure malignant nodules are not accidentally eliminated, that is, high negative predictive value (NPV), which is the percentage of correct negative test results. A NPV of 90% reduces the post-test probability of cancer to 10% or lower – a two-fold reduction in cancer risk from the 20% pre-test probability of cancer among patients selected for invasive procedures.

Second, the diagnostic test must frequently provide actionable results for clinical utility and economic benefit. This corresponds to the specificity of the test that is the percentage of benign nodules correctly called benign (i.e. negative) by the test. Specificity indicates the fraction of patients with benign tumors that can be identified confidently by the test. High impact tests such as Oncotype DX for treatment stratification of breast cancers has reported actionable results in approximately 34% of cases (7).

Third, the diagnostic test must be developed and validated on intended use samples from multiple independent sites without demographic bias on key clinical parameters such as age, nodule size, gender, etc. Intended use samples are defined to be radiologically discovered and pathologically confirmed malignant or benign nodules with a diameter of less than 30 mm (Stage IA cancers). The intended use population has a high occurrence of current and former smokers as this is a significant risk factor for lung cancer.

Fourth, development and validation studies should conform to rigorous guidelines for test development such as those recently provided by the Institute of Medicine (IOM) (8).

Previous biomarker studies on lung cancer (9–15) have not achieved optimal development and performance requirements, in particular, the requirement of achieving a NPV of 90% on a multisite validation study with only Stage IA samples.

We present here a 13-protein plasma test, or classifier, meeting the performance and development requirements stated above. We utilized multiple reaction monitoring (MRM, also known as selected reaction monitoring in literature) mass spectrometry (MS) to measure the concentrations of candidate proteins in plasma (16, 17). The benefits of MRM assays include high protein specificity, large multiplexing capacity, high sensitivity (mid atomole), significant dynamic range (10⁶), and both rapid and reliable assay development and deployment. MRM has been used for clinical testing of small molecule analytes for many years, and recently in the development of biologically relevant assays (18–23).

Results

Selection of biomarker candidates for assay development

To identify lung cancer biomarkers in blood that are shed or secreted from lung tumor cells, proteins over-expressed on the cell surface or over-secreted from lung cancer tumor cells relative to normal lung cells were identified from freshly resected lung tumors using organelle isolation techniques combined with mass spectrometry (24, 25)(Table S1). In addition, an extensive literature search for lung cancer biomarkers was performed using public and private resources (Table S2). Both the tissue-sourced biomarkers and literature-sourced biomarkers were required to have evidence of presence in blood (Table S3 and Fig. S1). More details are provided in Supplementary Materials.

Table 1 presents the steps taken in the refinement of these initial 388 protein candidates down to the set of 13 classifier proteins used for validation and performance assessment. The results are presented in the same sequence.

Table 1.

Steps in refining the 388 candidates down to the 13-protein classifier.

Number of Proteins	Refinement	Data
388	Lung cancer associated protein candidates sourced from tissue and literature.	Table S3
371	Number of the 388 protein candidates successfully developed into a MRM assay.	Table S5
190	Number of the 371 MRM protein assays detected in plasma.	Table S5
125	Number of the 190 MRM protein assays detected in at least 50% of cancer or 50% of benign discovery samples.	-
36	Number of the 125 detected proteins that were cooperative.	Table S7
21	Number of the 36 cooperative proteins with robust MRM assays (i.e. no interfering signals, good signal-to-noise, etc.)	Table S7
13	Number of the 21 robust and cooperative proteins with stable logistic regression coefficients.	Table 3

Development of MRM assays

Standard synthetic peptide techniques were used to develop a 371-protein multiplexed MRM assay from the 388 protein candidates. On average over 4 MRM transitions, i.e. predetermined and highly specific mass products (17), per protein were developed (Table S4–S5). Statistical correlation techniques were used to establish a transition-to-protein error rate below 5% ensuring high quality MRM assay development (see Supplementary Materials and Fig. S2). Synthetic peptides could not be developed or confidently identified for 17 candidates. The 371-protein MRM assay was applied to plasma samples from patients with pathologically confirmed benign or malignant nodules to determine how many of the 371 proteins could be detected in plasma. A total of 190 MRM assays were able to detect their target proteins in plasma (51% success rate; see also Table S4 and Fig S2). This success rate compared favorably to similar efforts (16%) to develop large scale MRM assays for the detection of diverse cancer markers in blood (26).

Classifier discovery

A total of 143 samples were obtained from three clinical sites (Table 2). All samples were selected to be consistent with intended use, specifically, having nodule size between 4–30 mm. Cancer and benign samples were pathologically confirmed. After clinical data monitoring, six samples were later found to have nodule size outside the 4–30mm range. Benign and cancer samples were pairwise matched on age, gender, nodule size and clinical site to avoid bias during MRM analysis. One benign sample was lost due to experimental deviations. See Table S6 for the experimental design.

Table 2.

Clinical characteristics of subjects and nodules in the discovery and validation studies.

Characteristics	Cancer	Benign	p-value	Cancer	Benign	p-value
	Discovery Study			Validation Study
Subjects	72	71		52	52
Age (year)*	65 (59–72)	64 (52–71)	0.46†	63 (60–73)	62 (56–67)	0.03†
Gender			1.00‡			0.85‡
Male	29	28		25	27
Female	43	43		27	25
Smoking history Status			0.006‡			0.006‡
Never§	5	19		3	15
Former	60	44		38	29
Current	6	6		11	7
No data	1	2		0	1
Pack-year*¶	37 (20–52)	20 (0–40)	0.001†	40 (19–50)	27 (0–50)	0.09†
Nodules
Size (mm)*	13 (10–16)	13 (10–18)	0.69†	16 (13–20)	15 (12–22)	0.68†
Source			1.00‡			0.89‡
IUCPQ||	14	14		13	12
New York	29	28		6	9
Pennsylvania	29	29		14	13
Vanderbilt	0	0		19	18
Histopathology
Benign diagnosis
Granuloma	-	48		-	26
Hamartoma	-	9		-	6
Scar	-	2		-	2
Other	-	12		-	18
Cancer diagnosis
Adenocarcinoma	41	-		25	-
Squamous cell	3	-		15	-
Large cell	0	-		2	-
Bronchioloalveolar (BAC)	3	-		0	-
Adenocarcinoma/BAC	21	-		5	-
Other	4	-		5	-

^*Data shown are median values with quartile ranges indicated in parentheses.

^†Mann-Whitney test.

^‡Fisher’s exact test.

^§A never smoker is defined as an individual who has a lifetime history of smoking less than 100 cigarettes.

^¶A pack-year is defined as the product of the total number of years of smoking and the average number of packs of cigarettes smoked daily. Pack-year data were not available for 4 cancer and 6 benign subjects in the discovery set and 2 cancer and 3 benign subjects in the validation set.

^||IUCPQ is the Institute Universitaire de Cardiologie et de Pneumologie de Quebec.

The 371-protein MRM assay was applied to the 143 discovery samples and the resulting transition data were analyzed to derive a 13-protein classifier fit to a logistic regression model (Table 3). The steps in the refinement from the 371 proteins to the 13 proteins participating in the classifier are summarized in Table 1. The key step in this refinement was the identification of 36 cooperative proteins. A protein was deemed cooperative if found more frequently on best performing panels than expected by chance alone. This strategy was motivated by the intent to capture the integrated behavior of proteins within lung cancer-perturbed networks. This was a defining step in the discovery of the classifier as the most cooperative proteins were often not the proteins with best individual performance. Full details of the estimation procedure and discovery process appear in Materials and Methods; sample classifier scores are listed in Table S6; the 36 cooperative proteins are provided in Table S7. The applicability of logistic regression modeling to the discovery dataset is provided in Table S8.

Table 3.

The 13-protein logistic regression classifier.

Protein (Human)	Transition	Coefficient*	95% CI*	p-value†	Tissue biomarker‡	Biomarker in literature
LRP1	TVLWPNGLSLDIPAGR_855.00_400.20	−1.59	(−2.39, −1.20)	9.64E-05	EPI	(38)
BGH3	LTLLAPLNSVFK_658.40_804.50	1.73	(1.18, 2.53)	4.37E-03		(39)
COIA1	AVGLAGTFR_446.26_721.40	−1.56	(−2.48, −1.16)	3.90E-04		(40)
TETN	LDTLAQEVALLK_657.39_330.20	−1.79	(−2.95, −1.21)	6.71E-03		(41)
TSP1	GFLLLASLR_495.31_559.40	0.53	(0.35, 0.93)	1.13E-02		(42)
ALDOA	ALQASALK_401.25_617.40	−0.80	(−1.21, −0.51)	1.03E-02	Secreted, EPI	(43)
GRP78	TWNDPSVQQDIK_715.85_260.20	1.41	(0.91, 2.24)	6.24E-03	Secreted, EPI, ENDO	(44)
ISLR	ALPGTPVASSQPR_640.85_841.50	1.40	(0.85, 2.21)	1.09E-02
FRIL	LGGPEAGLGEYLFER_804.40_913.40	0.39	(0.24, 0.66)	1.63E-02	Secreted, EPI, ENDO	(45)
LG3BP	VEIFYR_413.73_598.30	−0.58	(−0.97, −0.28)	6.24E-02	Secreted	(46)
PRDX1	QITVNDLPVGR_606.30_428.30	−0.34	(−0.64, −0.10)	1.54E-01	EPI	(47)
FIBA	NSLFEYQK_514.76_714.30	0.31	(0.16, 0.53)	4.10E-02		(48)
GSLG1	IIIQESALDYR_660.86_338.20	−0.70	(−1.51, −0.19)	1.60E-01	EPI, ENDO
Constant		36.16	(19.72, 59.01)

^*Based on 20,000 Monte Carlo cross validation permutations with a 20% hold-out rate: “Coefficient” is the corresponding median values from the permutations, “95% CI” is the corresponding 95% BCa confidence intervals (49).

^†Based on likelihood ratio test on the whole discovery dataset, comparing the models with or without the corresponding proteins.

^‡See Supplementary Materials and Table S3 for details.

Classifier performance in discovery study

We assessed the performance of the classifier in the discovery study in terms of negative predictive value (NPV) and specificity (SPC) (Fig. 1). When the classifier predicts that a patient from the intended use population has a benign tumor, the NPV is the probability that this prediction is true. NPV can be calculated from the classifier’s sensitivity, specificity and the estimated cancer prevalence in the intended use population (27). Specificity is the fraction of the benign nodules that the classifier is able to detect with high confidence. Thus, the higher the specificity at a given high NPV, the more patients with benign nodules can be rescued from unnecessary invasive procedures.

Fig. 1.

Performance of the classifier on the discovery samples (n=143) and validation samples (n=104). Negative predictive value (NPV) and specificity (SPC) are presented in terms of classifier score. A cancer prevalence of 15% was assumed.

We assumed a cancer prevalence of 15% based on estimates from the literature (4). Note that this prevalence is larger than the prevalence for the much smaller nodules studied in the National Lung Screening Trial (NLST) study (28). The classifier generated a cancer score, ranging from 0 to 1. Any reference value in this range can be defined so that a sample was predicted to be benign if its classifier score was not above the reference value, or malignant otherwise. We plotted in Fig. 1 the NPV and the specificity of the classifier on the discovery samples as a function of the reference value.

Table 4 reports the classifier’s performance on the discovery samples for multiple lung cancer prevalences with selected reference values. For each prevalence, the reference value was selected that corresponded to a discovery NPV of 95%. For a cancer prevalence of 15%, the reference value of 0.60 was selected and the classifier had a NPV of 95% +/−2% and a specificity of 66% +/−11% on the discovery samples, where 95% confidence intervals (CIs) were reported (see Supplementary Materials). The receiver operating characteristic (ROC) curve and the corresponding area under the curve (AUC) of the classifier on the discovery samples were reported in Fig. S3.

Table 4.

Performance of the classifier in discovery and validation at three cancer prevalences.

Dataset	Prevalence (%)	Reference Value	Sensitivity (%)	Specificity (%)	NPV* (%)	PPV (%)
Discovery (n=143)	15	0.60	82	66	95	30
	20	0.46	90	49	95	31
	25	0.42	93	45	95	36
Validation (n=104)	15	0.60	71	44	90	18
	20	0.46	83	29	87	23
	25	0.42	90	27	89	29
Vanderbilt (n=37)	15	0.60	79	56	94	24
	20	0.46	89	33	93	25
	25	0.42	100	28	100	32

The area under the receiver operating characteristic (ROC) curve (AUC) was 0.82, 0.60 and 0.74 in discovery, in validation and for Vanderbilt samples, respectively. Note that partial AUC is the preferred metric for tests maximizing NPV (27).

^*NPV is negative predictive value.

^†PPV is positive predictive value.

Classifier validation

The 13-protein classifier was fully defined before validation was performed, including the identity of the proteins, the logistic regression model, and the reference value used to classify a nodule as benign or malignant. We followed precisely the bright line demarcation of a locked down omics test as defined in the Institute of Medicine (IOM) omics report (8).

A total of 52 cancer and 52 benign samples (Table 2) were used to validate the performance of the 13-protein classifier. All validation samples were from different patients than the discovery samples. In addition, 36% of the validation samples were sourced from a new fourth clinical site, Vanderbilt University. The remaining validation samples were selected randomly from the discovery sites. Samples were consistent with intended use and matched as in the discovery study. The classifier was applied to the validation samples and analyzed (see Materials and Methods). See Table S9 for the experimental design and classifier scores of individual samples.

The performance of the classifier on the validation samples was summarized in Fig. 1, Table 4 and Fig. S3, along with the corresponding performance on the discovery samples. All reference values in Table 4 were selected in discovery and applied directly in validation. For cancer prevalence 15%and reference value 0.60, the classifier had NPV 90% +/−5 % and specificity 44% +/−13 % on the validation samples. The NPV and specificity on the Vanderbilt samples were 94% and 56%, respectively, providing a strong sign that the classifier was not overfit to the discovery sites.

Fig. 2 presents the application of the classifier to all 247 discovery and validation samples. We compared the clinical risk factors of smoking (measured in pack years) and nodule size (proportional to the diameter of each circle) to the classifier score assigned to each sample. Nodule size did not appear to increase with the classifier score. Indeed, both large and small nodules were spread across the classifier score spectrum. To quantify this observation, the Pearson correlation (R) between the classifier score and nodule size, smoking history pack-year and age were calculated (Table 5). The largest R² was 0.05, indicating that all correlations were either non-existing or very weak. The implication of this observation is remarkable. The classifier provides information on the disease status of pulmonary nodules that is independent of the three currently used risk factors for malignancy (age, smoking history and nodule size)(29, 30), and thus provides incremental molecular information of added clinical value. For a similar plot of nodule size vs. classifier score, see Fig. S4. See Table S10 for more details illustrating impact of clinical characteristics on classifier score.

Fig. 2.

Multivariate analysis of clinical (smoking, nodule size) and molecular (classifier score) factors as they relate to cancer and benign samples (n=247) in the discovery and validation studies. Smoking is measured by pack-years on the vertical. Nodule size is represented by circle diameter.

Table 5.

Pearson correlation between classifier score and clinical risk factors.

Risk factor	All samples	Cancer samples	Benign samples
Age (year)	0.190	0.015	0.227
Smoking history (pack year)	0.185	0.089	0.139
Nodule size (mm)	−0.071	−0.081	−0.035

The molecular foundations of the classifier

By design, the 13 classifier proteins were selected from a candidate list of 388 proteins, each with either empirical evidence or literature support of differential expression in lung cancer tissue, or both (Table 3). The only exception was the protein ISLR whose function is currently not well-characterized in the literature despite the fact that it was identified as a lung cancer biomarker in the automatic literature search.

To better understand the specific role each of these 13 proteins have in lung cancer, and their relationship to each other, they were submitted for pathway analysis using Ingenuity Systems (IPA). First, the transcription regulators most likely to cause a modulation of these 13 proteins were identified. With standard IPA analysis parameters, the four most significant nuclear transcription regulators were FOS (proto-oncogene c-Fos), NF2L2 (nuclear factor erythroid 2-related factor 2), AHR (aryl hydrocarbon receptor) and MYC (myc proto-oncogene protein): See Materials and Methods for details. These proteins regulate 12 of the 13 classifier proteins, with ISLR being the exception (see below).

FOS is common to many forms of cancer. NF2L2 and AHR are associated with lung cancer, oxidative stress response and lung inflammation. MYC is associated with lung cancer and oxidative stress response. These four transcription regulators and the 13 classifier proteins, collectively, are also highly associated (p-value 1.0×10⁻⁷) with the same three biological networks, namely, lung cancer, lung inflammation and oxidative stress response. This is summarized in Fig. 3 where the classifier proteins (green), transcription regulators (blue) and the three merged networks (orange) are depicted. Only ISLR is not connected through these three networks to other classifier proteins, although it is connected through cancer networks not specific to lung. In summary, the modulation of the 13 classifier proteins can be linked back to a few transcription regulators highly associated with lung cancer, lung inflammation and oxidative stress response networks; three biological processes reflecting aspects of lung cancer.

Fig. 3.

The 13 classifier proteins (green), 4 transcription regulators (blue) and the three networks (orange lines) of lung cancer, oxidative stress response and lung inflammation. All references are human UniProt identifiers.

Derivation of the classifier

MRM technology enabled the simultaneous exploration of a large number of lung cancer relevant proteins. The definitive step in the derivation of the classifier was the identification of the most cooperative protein biomarkers. Typically, proteins are shortlisted in the discovery process by filtering on individual diagnostic performance. To contrast the difference between filtering proteins based on strong individual performance as opposed to frequency on high performance panels, we calculated a p-value for each of the 21 MRM-robust cooperative proteins using the Mann-Whitney non-parametric test (31). Only 1 of these 21 proteins had a p-value below 0.10 (Table S7). In addition, the cooperative protein score for each of these 21 proteins was calculated. The cooperative protein score measured the frequency at which each protein appeared on high performance panels (see Materials and Methods). Importantly, the p-values and cooperative scores of the 21 MRM-robust cooperative proteins were not correlated (Pearson correlation −0.24, p-value 0.30).

Most informative proteins

Which proteins in the classifier were most informative? To answer this question all possible classifiers were constructed from the set of 21 MRM-robust cooperative proteins (Table S7) and their performance measured. The frequency of each protein among the 100 best performing panels was determined (see “Frequency” column of Table S7). Four proteins (LRP1, COIA1, ALDOA, LG3BP) were highly enriched with 95% of the 100 best classifiers having at least three of these four proteins (p-value < 1.2 ×10⁻⁴¹). The conclusion was that high performance panels of cooperative proteins for pulmonary nodule characterization were similar in composition to one another with a preference for a set of particularly informative (cooperative) proteins. It is expected that optimizations of the classifier on different technological platforms (e.g. sample processing pipelines, mass spectrometry instrumentation, etc.) may result in changes in optimal classifier parameters such as the logistic regression protein coefficients.

Discussion

Classifier score independent of clinical risk factors

Recent work has demonstrated that there is frequently not a correlation between an individual protein’s blood concentration and tumor size (32), as is the case for our signature. In fact, even if individual protein concentrations increased or decreased with tumor size, our classifier score is combination of these 13 protein concentrations, and so, would not necessarily correlate to tumor size. Of clinical importance is that the classifier provides a new metric, independent of current diagnostic risk factors (29, 30), for assessing the molecular status of a lung nodule.

Limitations of work

The main limitation of the work presented is that both discovery and validation studies were conducted using retrospective samples. A prospective study on intended use samples is required to further validate the utility of the classifier for clinical use.

A second limitation is that the molecular classifier presented here is not integrated with clinical risk factors such as nodule size, age and smoking history for the purpose of a single classifier for lung nodule diagnosis. Although an integrated classifier would be ideal, in practice pulmonologists vary broadly in the use of clinical risk factors, and so, it is actually preferable to have a molecular diagnostic test that produces a score independent of clinical risk factors.

Materials and Methods

Discovery study design

A retrospective, multi-center, case-control study was performed using K2-EDTA plasma aliquots previously obtained from subjects who provided informed consent and contributed biospecimens in studies approved by the Ethics Review Board (ERB) or the Institutional Review Boards (IRB) at the IUCPQ or New York University and the University of Pennsylvania, respectively. In addition, plasma samples were provided by study investigators after review and approval of the sponsor’s study protocol by the respective institution’s ERB or IRB, as required. Sample eligibility for the proteomic analysis was based on the satisfaction of the study inclusion and exclusion criteria (Supplementary Materials). Each cancer-benign sample pair was matched in best efforts by gender, nodule size (±10 mm), age (±10 years), smoking history pack-years (±20 pack-years), and by center. The study was powered with a probability of 92% to detect 1.5 fold differences in protein abundance between malignant and benign lung nodules. See Table S6 for more details.

Analysis of plasma samples using MRM-MS

The protocol for MRM-MS analysis of plasma aliquots included immunodepletion on IgY14-Supermix resin columns (Sigma), denaturation, trypsin digestion, and de-salting, followed by reversed-phase liquid chromatography and MRM-MS analysis of the obtained peptide samples (see details in Supplementary Materials).

Development of MRM assays

MRM assays for candidate proteins were developed based on synthetic peptides (17, 26, 33). More details are provided in Supplementary Materials.

Identification of endogenous normalizing proteins

The following criteria were used to identify a transition of a normalization protein: (A) highest median intensity of all transitions from the same protein; (B) detected in all samples; (C) ranking high, as a normalizer, in reducing median technical coefficient of variation (CV); (D) ranking high in reducing median column drift (defined in Supplementary Materials); and (E) possession of low median technical CV and low median biological CV, i.e. median CV of transition intensities that were measured on clinical samples. Six endogenous normalizing proteins were identified (Table S11 and Fig. S5).

Normalization of raw MRM-MS data

Six normalization transitions (Table S11) were used to normalize raw MRM-MS data to reduce sample-to-sample intensity variations within same study. A scaling factor was calculated for each sample so that the intensities of the six normalization transitions of the sample were aligned with the corresponding median intensities of all human plasma standard (HPS) samples. Assuming that N_i,s was the intensity of a normalization transition i in sample s and N̂_i the corresponding median intensity of all HPS samples, then the scaling factor for sample s was given by Ŝ/S_s, where

$$S_s=\mathit{median}(\frac{N_{1,s}}{{\widehat{N}}_1},\frac{N_{2,s}}{{\widehat{N}}_2},\dots ,\frac{N_{6,s}}{{\widehat{N}}_6})$$ (1)

was the median of the intensity ratios and Ŝ was the median of S_s over all samples in the study. For each transition of each sample, its normalized intensity was calculated as

$${\check{I}}_{i,s}=I_{i,s}\ast \widehat{S}/S_s$$ (2)

where I_i,s was the raw intensity.

Logistic Regression Model

The logistic regression classification method (34, 35) was used to combine a panel of transitions into a classifier and to calculate a classification score between 0 and 1 for each sample. The score (P_s) of a sample was determined as

$$P_s=1/[1+exp(-\alpha -{\sum }_{i=1}^N{\beta }_i\ast {\check{I}}_{i,s})],$$ (3)

where Ǐ_i,s was the logarithmically transformed (base 2), normalized intensity of transition i in sample s, β_i was the corresponding logistic regression coefficient, α was a classifier-specific constant, and N was the total number of transitions in the classifier. A sample was classified as benign if P_s was less than or equal to a reference value or cancer otherwise.

Lung nodule classifier development

The classifier development included the following steps:

Normalization of raw MRM-MS data was performed to reduce sample-to-sample intensity variations using a panel of six endogenous proteins as described above. After normalization, MRM-MS data were filtered down to transitions having the highest intensities of the corresponding proteins and satisfying the criterion for detection in a minimum of 50% of the cancer or 50% of the benign samples. A total of 125 proteins satisfied these criteria of reproducible detection. Missing values were replaced by half the minimum detected values of the corresponding transitions in all samples.

Remaining transitions were then used to identify proteins, defined as cooperative proteins, that occurred with high frequency on top-performing protein panels. The cooperative proteins were derived using the following estimation procedure as it was not computational feasible to evaluate the performance of all possible protein panels.

Monte Carlo cross validation (MCCV) (36) was performed on 1×10⁶ panels, each panel comprised of 10 randomly selected proteins and fitted to a logistic regression model, as described above, using a 20% holdout rate and 10² sample permutations. The receiver operating characteristic (ROC) curve of each panel was generated and the corresponding partial area under the ROC curve (AUC) but above the boundary of sensitivity being 90%, defined as the partial AUC (27, 37), was used to assess the performance of the panel. By focusing on the performance of individual panels at high sensitivity region, the partial AUC allowed for the identification of panels with high and reliable performance on NPV. The candidate proteins that occurred in the top 100 performing panels with a frequency greater than that expected by chance were identified empirically as cooperative proteins. For each protein the cooperative score was defined as its frequency on the 100 high performance panels divided by the expected frequency. Highly cooperative proteins had a score of 1.75 or higher (one-sided p value < 0.05), cooperative proteins had a score higher than 1 while non-cooperative proteins had a score of 1 or less. Note that one million panels were sampled to ensure that the 100 top performing panels were exceptional (empirical p value ≤ 10⁻⁴). In addition, panels of size 10 were used in this procedure based on empirical evidence that larger panels did not change the resulting list of cooperative proteins. This also avoided overfitting the logistic regression model. In total, 36 cooperative proteins were identified, including 15 highly cooperative proteins.

Raw chromatograms of all transitions of cooperative proteins were manually reviewed. Proteins with low signal-to-noise ratios and/or showing evidence of any interference were removed from further consideration. In total, 21 cooperative and robust proteins were identified.

Remaining candidate proteins were then evaluated in an iterative, stepwise procedure to derive the final classifier. In each step, MCCV was performed using a holdout rate of 20% and 10⁴ sample permutations to train the remaining candidate proteins to a logistic regression model and to assess the variability, i.e. stability, of the coefficient derived for each protein by the model. The performance of the model was assessed by partial AUC (27). The protein having the least stable coefficient was identified and removed. This procedure was repeated until no protein was left. During this procedure a total of 21 logistic regression classifiers were developed and their respective performances were obtained. Among the 21 classifiers the 13-protein classifier had the optimal performance and was selected as the classifier for validation. Seven of the 13 proteins in the final classifier were highly cooperative.

Proteins in the final classifier were further trained to a logistic regression model by MCCV with a holdout rate of 20% and 2×10⁴ sample permutations. Results for the lung nodule classifier development are summarized in Table S7.

Lung nodule classifier validation

The design of the validation study was identical to that of the discovery study, but involved plasma samples associated with independent subjects not evaluated in the discovery study. Additional specimens were obtained from Vanderbilt University with similar requirements for patient consent, IRB approval, and satisfaction of HIPAA requirements. Of the 104 total cancer and benign samples in the validation study, half were analyzed immediately after the discovery study, while the other half were analyzed later. The study was powered to observe the expected 95% CI of the NPV associated with reference value 0.60. See Supplementary Materials and Table S9 for more details.

The raw MRM-MS dataset in the validation study was normalized in the same way as the discovery dataset. Variability between the discovery and the validation studies was mitigated by utilizing human plasma standard (HPS) samples in both studies as external calibrator: See Supplementary Materials for details. Missing data in the validation study were then replaced by half the minimum detected values of the corresponding transitions in the discovery study. Normalized, calibrated transition intensities were applied to the logistic regression model of the final classifier learned previously in the training phase, from which classifier scores were assigned to individual samples. The performance of the lung nodule classifier on the validation samples was then assessed based on the classifier scores.

IPA pathway analysis

Standard parameters were used. Specifically, in the search for nuclear transcription regulators, requirements were p-value < 0.01 with a minimum of 3 proteins modulated. Significance was determined using a right-tailed Fisher’s exact test using the IPA Knowledge Database as background.

Statistical analysis

All statistical analyses were performed with STATA, MATLAB and/or R. Specific tests and analysis details are indicated at appropriate sections.