Autoantibody Signature Enhances the Positive Predictive Power of Computed Tomography and Nodule-Based Risk Models for Detection of Lung Cancer

Abstract

Introduction

The incidence of pulmonary nodules is increasing with the movement toward screening for lung cancer by low-dose computed tomography. Given the large number of benign nodules detected by computed tomography, an adjunctive test capable of distinguishing malignant from benign nodules would benefit practitioners. The ability of the EarlyCDT-Lung blood test (Oncimmune Ltd., Nottingham, United Kingdom) to make this distinction by measuring autoantibodies to seven tumor-associated antigens was evaluated in a prospective registry.

Methods

Of the members of a cohort of 1987 individuals with Health Insurance Portability and Accountability Act authorization, those with pulmonary nodules detected, imaging, and pathology reports were reviewed. All patients for whom a nodule was identified within 6 months of testing by EarlyCDT-Lung were included. The additivity of the test to nodule size and nodule-based risk models was explored.

Results

A total of 451 patients (32%) had at least one nodule, leading to 296 eligible patients after exclusions, with a lung cancer prevalence of 25%. In 4- to 20-mm nodules, a positive test result represented a greater than twofold increased relative risk for development of lung cancer as compared with a negative test result. Also, when the “both-positive rule” for combining binary tests was used, adding EarlyCDT-Lung to risk models improved diagnostic performance with high specificity (>92%) and positive predictive value (>70%).

Conclusions

A positive autoantibody test result reflects a significant increased risk for malignancy in lung nodules 4 to 20 mm in largest diameter. These data confirm that EarlyCDT-Lung may add value to the armamentarium of the practitioner in assessing the risk for malignancy in indeterminate pulmonary nodules.

Introduction

More than 1.5 million individuals in the United States have lung nodules identified annually, including 63,000 who receive a new lung cancer diagnosis within 2 years.^1,2 This number is growing given the increasing number of computed tomography (CT) scans being performed since the National Lung Screening Trial reported that annual CT screening reduced lung cancer mortality by 20% as compared with chest radiography.³ The problem remains, however, that more than 50% of CT-screened patients have at least one noncalcified nodule, with more than 96% of nodules larger than 4 mm being false-positives.^3,4 Furthermore, there is a high false-positive rate and morbidity associated with biopsy or resection of benign nodules.^2,3,5 Therefore, a critical need exists for an adjunctive test to help evaluate the malignancy potential and reduce the false-positive rate.

EarlyCDT-Lung (Oncimmune Ltd., Nottingham, United Kingdom) detects the presence of serum autoantibodies (AAbs) to a panel of lung cancer–associated antigens using an indirect enzyme-linked immunosorbent assay method.^6,7 A sample is positive if at least one AAb is elevated above a predetermined cutoff. Test specificity was improved in November 2010 by changing from a six-AAb panel to the current seven-AAb panel.⁸ Test performance in routine clinical practice for approximately 1600 patients with unknown nodule status was assessed previously, showing 87% specificity, 41% sensitivity, and a 5.4-fold relative risk of lung cancer in cases with a positive test result.⁹ We demonstrate how the test can assist in evaluation of the malignancy of pulmonary nodules in the clinic with or without lung cancer risk models.

Methods

Patients

A cohort of 1987 Health Insurance Portability and Accountability Act–authorized patients was tested in routine practice between May 2009 and December 2012 by using EarlyCDT-Lung as described previously.^8–10 For this study CT imaging reports from within 6 months of the AAb test were assessed for the presence of pulmonary nodules. The size of the largest noncalcified nodule was recorded. Patients who had invalid AAb tests, were lost to follow-up, or had a history of cancer were excluded. PET scan data were generally unavailable. Although most cancer cases were confirmed by pathology reports, some were diagnosed solely on imaging reports. To reflect this, the analysis considered two cohorts defined by their eligibility criteria (Fig. 1).

Figure 1.

Patient cohorts in the clinical registry of nodule status. Of 1987 patients enrolled, 585 (29%) were excluded either on the basis of being lost to follow-up (LTFU) for all reasons, including practices closed, patients changing practices, and physicians moving practices (n = 493), or on the basis of having received a confirmed cancer diagnosis (Dx) other than lung cancer, a history (Hx) of cancer (n = 78), or an invalid autoantibody test result (n = 14). Of the remaining 1402 patients (71% of the 1987), 451 (339 without cancer and 112 with cancer) had a pulmonary nodule(s) reported, whereas the remaining 951 either did not have a nodule detected by computed tomography (CT), did not have an available CT, were reported as unknown by the physician, or had a CT but with unknown results. In the 451 patients with a nodule, the nodule was calcified in 25 individuals, 75 nodules had no clear size information, and a further 55 were outside the 6-month time window, leaving 296 patients (221 without cancer and 75 with cancer) for the inclusive cohort and 269 (217 without cancer and 52 with cancer) for the exclusive cohort. The EarlyCDT-Lung positivity rates for the excluded, no-nodule, and with-nodule groups were 8%, 11%, and 25%, respectively, with the comparison of the excluded and no-nodule groups being of borderline significance (p = 0.05) and thus giving no clear evidence of bias.

Inclusive Cohort

The clinician’s diagnosis was accepted even if CT or pathology reports were not available. This corresponds most closely to the commercial setting. The inclusive cohort retains as many cancers as possible to enhance statistical power.

Exclusive Cohort

Starting with the inclusive cohort, only those members with data from actual CTs and pathology reports on file, as opposed to office visits or physician phone interviews, were accepted. These criteria correspond most closely to ideal clinical practice.

Statistical Analysis

A model-free analysis summarizing the association between nodule size (<4 mm, 4–20 mm, and ≥20 mm), cancer diagnosis, and test positivity was performed. Diagnostic performance metrics were calculated for the inclusive, exclusive, and exclusive (7AAb panel only) cohorts. Pretest risk was calculated by applying size thresholds above which a patient would be deemed to have a positive result (nodules-only). Posttest risk (positive predictive value [PPV]) was calculated by adding the AAb test result using a “both-positive rule” (i.e., positive for both size threshold and AAb test result). The curve for PPV versus nodule size threshold was compared with that for the combined test (nodules + AAb). Finally, the observed nodule distribution was compared with the Mayo study⁴ to confirm similarity.

A risk-model analysis was applied using the inclusive cohort. The pretest cancer risk was estimated using three published nodule-based models: (1) the GOULD model, which is based on Department of Veterans Affairs data¹¹; (2) the BROCK model, which was developed on the basis of pan-Canadian study data¹²; and (3) the MAYO model, which is based on Mayo study data.^13,14 The calibration of these models over nodule size was examined by comparing observed and predicted cancer rates using χ² goodness-of-fit tests.

The additivity of the AAb test to the models was assessed for 4-mm to 20-mm nodules, which are the most relevant for early detection. The receiver operating characteristic curve for each model was formed by varying the risk threshold above which a patient was deemed positive and calculating specificity and sensitivity at each threshold (model only). Then, at each threshold the data were split by AAb test result and a new curve was formed by applying the both-positive rule (Model + AAb). The sensitivity observed at fixed specificity values was compared across the two curves. Plots of the respective PPVs were also created.

Results

Cohorts

From the original cohort (n = 1987), exclusions led to 296 patients (221 without cancer and 75 with cancer) for the inclusive cohort and 269 (217 without cancer and 52 with cancer) for the exclusive cohort (Table 1 and Supplementary Table 1 for the six-AAb and seven-AAb panels).

Table 1.

Summary of Demographics of Patients with Nodules (Inclusive and Exclusive Data Sets)

Characteristic	Inclusive	Exclusive
Data set size	296	269
Sex (M-to-F)	145:151	133:136
Age, median (range), y	66 (30–89)	66 (30–89)
Smoker (current, ex, no)a	121, 128, 43	110, 113, 42
6AAb, 7AAb	114, 182	103, 166
Lung cancersb	75	52
Tumor type (NSCLC, SCLC, others)	56, 4, 15	44, 2, 6
Staging (early, late, unknown)	34, 21, 20	28, 17, 7

^aFour patients had incomplete smoking data.

^bNumber of cancers diagnosed at up to 6 months’ follow-up.

M, male; F, female; AAb, autoantibody.

Model-Free Analysis

The nodule distribution was most similar to that in the Mayo study (Supplementary Fig. 1). For the exclusive cohort, 22 of 68 of patients with a positive test result (32%) and 30 of 201 of patients with a negative test result (15%) had lung cancer, giving a relative risk of 2.2 (95% confidence interval [CI]: 1.3–3.5, PPV = 32%) (Table 2). For the seven-AAb test the relative risk increased to 2.3 (95% CI: 1.3–3.9, PPV = 40%). For the 4-mm to 20-mm group, the relative risk was 2.7-fold (95% CI: 1.3–5.7), equating to an increase in absolute risk from 13% to 24%, and for the ≥20-mm group, the relative risk was 2.1-fold (95% CI: 1.3–3.3), which equates to an increase in absolute risk from 46% to 77% (Supplementary Fig. 2).

Table 2.

Comparison of Test Performance Characteristics in the Inclusive and Exclusive Cohorts by Nodule Size

Inclusive cohort (n = 296)a
Category	Positive AAb				Negative AAb			RR	DLRp	p Value
Size (mm)	LC	Not LC	Total	PPV	LC	Not LC	Total
<4	1	8	9	—	0	18	18	—	—	—
4 to <20	14	35	49	29%	20	127	147	2.1 (1.2–3.8)	1.9 (1.2–3.1)	0.028
≥20	15	3	18	83%	25	30	55	1.8 (1.3–2.6)	4.1 (1.3–13.0)	0.006
Overall	30	46	76	39%	45	175	220	1.9 (1.3–2.8)	1.9 (1.3–2.8)	0.002
Exclusive cohort (n = 269)b
Category	Positive AAb				Negative AAb			RR	DLRp	p Value
Size (mm)	LC	Not LC	Total	PPV	LC	Not LC	Total
<4	1	8	9	—	0	17	17	—	—	—
4 to <20	11	35	46	24%	12	124	136	2.7 (1.3–5.7)	2.2 (1.3–3.6)	0.018
≥20	10	3	13	77%	18	30	48	2.1 (1.3–3.3)	3.9 (1.2–12.9)	0.014
Overall	22	46	68	32%	30	171	201	2.2 (1.3–3.5)	2.0 (1.3–3.0)	0.003
Exclusive, 7AAb cohort (n = 166)c
Category	Positive AAb				Negative AAb			RR	DLRp	p Value
Size (mm)	LC	Not LC	Total	PPV	LC	Not LC	Total
<4	0	5	5	—	0	13	13	—	—	—
4 to <20	6	14	20	30%	9	73	82	2.7 (1.1–6.8)	2.5 (1.1–5.4)	0.071
≥20	8	2	10	80%	14	22	36	2.1 (1.2–3.4)	4.4 (1.0–18.4)	0.032
Overall	14	21	35	40%	23	108	131	2.3 (1.3–3.9)	2.3 (1.3–4.1)	0.010

Note: p Values for comparing EarlyCDT-Lung results by malignant and nonmalignant nodules were calculated using Fisher’s exact test; 95% binomial confidence intervals are shown. Nodule size coding: small = 3 mm; sub cm = 5 mm; tiny = 2 mm; <4 = 3 mm; 3–4 = 4 mm; <10 = 5 mm; 17–20 = 19 mm.

^aOverall specificity = 79%, sensitivity = 40%, PPV = 39%, NPV = 80%. Comparison of positivity between size categories (excluding <4 mm) (p = 0.95).

^bOverall specificity = 79%, sensitivity = 42%, PPV = 32%, NPV = 85%. Comparison of positivity between size categories (excluding <4 mm) (p = 0.53).

^cOverall specificity = 84%, sensitivity = 38%, PPV = 40%, NPV = 82%. Comparison of positivity between size categories (excluding <4 mm) (p = 0.77).

AAB, autoantibody; LC, lung cancer; PPV, positive predictive value; RR, relative risk; and DLRp, positive diagnostic likelihood ratio, all defined in the text (see Statistical Analysis section).

p Values for comparing EarlyCDT-Lung results by malignant and nonmalignant nodules were calculated using Fisher’s exact test. The 95% binomial confidence intervals are shown.

Risk Model Analysis

For the inclusive cohort, the GOULD and BROCK models, respectively, overestimated and underestimated the risk relative to the actual rates observed (p<0.001). The MAYO estimates were closest to those observed (p = 0.54) (Table 3).

Table 3.

Calibration of Risk Models versus Observed Cancers in the Registry—Inclusive Cohort (n = 296)

Size	n	Observed Cancers	Cancer Rate	Mean Predicted Cancer Risk			Mayo Study
				GOULD	BROCK	MAYO
<4 mm	27	1	4%	15%	—a	5%	0%
4 to <8 mm	75	6	8%	24%	0%	8%	1%
8 to <20 mm	121	28	23%	39%	8%	24%	21%
≥20	73	40	55%	77%	40%	70%	33%
All	296a	75	25%	42%	13%	29%	17%
χ2 testb				p < 0.001	p < 0.001	p = 0.54	p < 0.01

^aBROCK not available for <4 mm.

^bChi-square test for comparison of audit-observed versus expected cancers for each model and Mayo study.

For nodules in the 4- to 20-mm range, the receiver operating characteristic curves for the models show that as specificity surpassed 90%, the sensitivity declined to 30% and less (Fig. 2 for MAYO and Supplementary Figures 3 and 4 for GOULD and BROCK, respectively). When EarlyCDT-Lung was added, the sensitivity was increased at the high-specificity end (left-hand side). More precisely, for a fixed risk threshold, when a biomarker test was added using the both-positives rule, the specificity increased and the sensitivity decreased. For the MAYO model, for example, as the risk threshold moved from 35% to 60%, the specificity for the model-only case increased from 88% to 97% and the sensitivity decreased from 50% to 13% (the PPV stayed at 50%). To control the specificity of the combined test at 88% to 97%, the risk thresholds need to be lowered to 10% to 20%, giving a sensitivity of 40% to 33% (PPV 43%–72%). So for the same false-positive rate, a total of eight extra cancers (a 2.6-fold increase) were declared positive, all (in this case) with a pretest risk between 20% and 60% (Table 4 and Supplementary Table 2). So lower-risk patients are having their cancer detected. This effect was seen only at high specificity levels.

Figure 2.

Partial receiver operating characteristic curves for the MAYO model and with EarlyCDT-Lung added using the both-positive rule. Curves are shown for the MAYO model only (black line); the MAYO model plus EarlyCDT-Lung (dashed line); the proportional line, which is the theoretical line if EarlyCDT-Lung was added to the MAYO model in a strictly proportional (independent) manner (dotted line); and the random line of no diagnostic discrimination (thin black line). Below approximately 8% on the x axis (92% specificity) the combined model and autoantibody test show improved sensitivity at the same specificity. The proportional line follows the observed line quite closely (inclusive cohort, 4 mm–20 mm [n = 208]).

Table 4.

Example Using a Specificity of 97%

Model	Model Only			Model + AAb			Total
MAYO		Negative	Positive		Either Negative	Both Positive
	No LC	163	5	No LC	163	5	168
	LC	35	5	LC	27	13	40
	Total	198	10	Total	190	18	208
Risk		60%			20%
Spec, Sens		97%, 13%			97%, 33%
PPV		50%			72%
RR		2.8 (1.4–5.6)			5.1 (3.2–8.0)
DLRp		4.2 (1.3–13.8)			10.9 (4.1–28.9)

Note: The effect of adding EarlyCDT-Lung at a reduced risk level (60%–20%) is to reclassify eight cancers from false-negative to true positive. This gave high performance statistics: PPV = 72%, RR = 5.1, and DLRp = 10.9.

AAb, autoantibody; LC, lung cancer; Risk, risk threshold; Spec, specificity; Sens, sensitivity; PPV, positive predictive value; RR, relative risk with 95% confidence interval; DLRp, positive diagnostic likelihood ratio with 95% confidence interval.

For fixed risk threshold, for example, a threshold of 30% with the MAYO model, the PPV increased from 48% to 91% when EarlyCDT-Lung was added (Fig. 3 for MAYO and Supplementary Figs. 5 and 6 for GOULD and BROCK, respectively). Virtually all false-positive results have been eliminated (25 down to 1), with approximately half of the cancers being lost (23 down to 11) (Table 5). Also shown is the percentage of subjects whose risk is above the threshold and are thus eligible for a positive combined test.

Figure 3.

Plot of positive predictive value (PPV) versus risk threshold (R) for the MAYO model. Curves are shown for the MAYO model only (black line with dots) and the MAYO model plus EarlyCDT-Lung (gray dashed line with triangles). Also shown is the plot of the percentage of subjects with an individual risk below R on the x axis (black dashed line). Adding EarlyCDT-Lung improves the PPV over the whole range. For a chosen risk threshold, a patient is risk model–positive if his or her calculated individual risk is greater than the threshold. The procedure could be as follows: Choose R (30%, for example) and then read off the model-only PPV for patients with a risk higher than R (48% in this example) and then with an EarlyCDT-Lung–positive result added (91%). Finally, read off the percentage of patients in the population with a risk higher than R (23%). The number of false-positive results is reduced at the expense of fewer cancers detected. So choose the value of R giving the most useful performance (inclusive cohort, 4 mm–20 mm [n = 208]).

Table 5.

Example Using a Risk Threshold of 30% with the MAYO Model

Model	Model Only			Model + AAb			Total
MAYO		Negative	Positive		Either Negative	Both Positive
	No LC	143	25	No LC	167	1	168
	LC	17	23	LC	30	10	40
	Total	160	48	Total	197	11	208
Risk		30%			30%
Spec, Sens		85%, 58%			99%, 25%
PPV		48%			91%
RR		4.5 (2.6–7.7)			6.0 (4.1–8.7)
DLRp		3.9 (2.5–6.1)			42.0 (5.5–318.9)

Note: The effect of adding EarlyCDT-Lung was to eliminate virtually all false-positive results (25 down to 1) at the expense of losing just more than half of the cancers (23 down to 11). This gave high performance statistics: PPV = 91%, RR = 6.0, and DLRp = 42.0.

AAb, autoantibody; LC, lung cancer; Risk, risk threshold; Spec, specificity; Sens, sensitivity; PPV, positive predictive value; RR, relative risk with 95% confidence interval; DLRp, positive diagnostic likelihood ratio with 95% confidence interval.

Discussion

The study showed, under strict eligibility conditions, that the addition of a positive EarlyCDT-Lung AAb test to nodule size alone significantly increased the PPV for malignancy prediction, with relative risks of 2.7-fold for nodules 4 to 20 mm in the largest diameter.

Using nodule-based risk models with positivity thresholds, lowering the risk threshold, and adding the AAb test using a both-positive rule increased sensitivity by reclassifying a proportion of the lower-risk false-negatives to true positives. This was demonstrated in 4- to 20-mm nodules, which is important because malignant nodules smaller than 20 mm are largely stage IA.^4,15 Alternatively, the false-positive rate can be reduced while still detecting a reasonable number of cancers by fixing the risk threshold for the combined test.

The testing was not controlled by a formal protocol and may have been subject to biases, particularly with respect to diagnosis not based on a pathology report, hence the two cohorts. However, the study reflects the clinical setting within which a biomarker test might be expected to operate, and so the findings do have validity. Additionally, as indicated previously,⁹ those with a negative test were also followed up to avoid observer bias and allow unbiased estimates of the relative risk. The risk model analysis is, of course, only approximate. The models themselves were trained on specific data sets with different nodule size distributions and malignancy rates.¹⁴ We hope to confirm the results in a prospective study.

Lung cancer tends to be detected at a later stage as a nodule grows.⁴ A biomarker that suggests an increased probability of malignancy while the nodule is still relatively small could lead to early detection and decreased mortality. We have shown that an AAb test can significantly add to clinicians’ interpretation of pulmonary nodules, both with and without risk models, for nodules smaller than 20 mm in diameter. For larger nodules (e.g., >20 mm) there is less need for a biomarker test because the clinical pathway is better established. For indeterminate nodules and a negative AAb test, the physician should continue to assess patients on the basis of their other risk factors and subsequent scans.