Development and validation of clinical diagnostic models for the probability of malignancy in solitary pulmonary nodules

Abstract

Background

It is critical to develop a non-invasive and accurate method for differentiating between malignant and benign solitary pulmonary nodules. In large sample studies, the effectiveness of the diagnostic prediction model as a tool of assessment of the probability of malignancy is still unclear. The establishment of a diagnostic model based on large samples is needed.

Methods

In this study, 3358 patients diagnosed with a solitary pulmonary nodule between January 2005 and March 2013, were enrolled. All patients received surgery for pulmonary nodule resection. Clinical characters, preoperative biomarker results, and computed tomography scan findings were collected. All patients were randomly separated into a training set (n = 1679) and a test set (n = 1679); we used training sets to build a diagnostic model for the malignancy probability of pulmonary nodules, and applied the test set to validate our model, as well as other published diagnostic models.

Result

Logistic regression analysis identified 11 clinical characteristics as independent predictors of malignancy in patients with a solitary pulmonary nodule. The goodness-of-fit statistic for the model indicated that the observed proportion of malignancies did not differ from the predicted proportion (P = 0.571). The area under the curves of the receiver operator characteristic curve for our model in the training set was 0.935.

Conclusion

As the accuracy of the model was high, we suggest that the diagnostic model can be used as a tool to help guiding clinical decisions, when the clinician cannot make a definitive diagnosis of a solitary pulmonary nodule.

Introduction

A solitary pulmonary nodule (SPN), or “coin lesion,” is an oval lesion ≤3 cm that is completely surrounded by pulmonary parenchyma without other abnormalities.¹ Detection tends to be incidental by chest X-ray imaging or computed tomography (CT) scan. SPNs may be malignant and represent early-stage lung cancer.^2,3 Early-stage lung cancer patients are expected to have a good prognosis if treated expediently: the five-year overall survival rate for stage IA lung cancer is around 80%. Advanced-stage lung cancer patients, however, generally have a much poorer clinical outcome: stage IV lung cancer patients have a five-year overall survival rate of 10%.⁴ Thus, early detection and treatment of malignant SPN is crucial to improving the survival of lung cancer patients. The accurate differentiation of malignant solitary pulmonary nodules from benign lesions is critical for providing treatment for malignant tumor patients and avoiding unnecessary surgery for those with benign lesions.⁵

Currently, diagnosis of SPNs relies mainly on pathological examination, which requires acquisition of the tumor. Surgical resection is a widely used method to extract the nodule. It is also the standard treatment for malignant SPNs, but it should be avoided in cases of benign lesions. Needle biopsy is also applied for differential diagnosis, but it is invasive, potentially risky, and sometimes nondiagnostic.⁶ Observation with serial chest radiographs avoids unnecessary surgery in cases of benign disease, but when malignancy exists,⁷ this approach delays appropriate diagnosis and treatment. Therefore, there is an urgent need to develop new methods to diagnose lung cancer noninvasively with high accuracy. Presently, there are many methods for identifying the malignancy of SPNs, such as clinical characteristics (age, gender, smoking history, family history of cancer, and history of malignancy); CT signs of nodule (size, calcification, spiculation, lobulation, pleural retraction, clear border, and cavity); and results of biomarker tests. Some of these characteristics can help to evaluate the probability of malignancy. The use of models combining some of these features for the differentiation of SPNs has been reported.^8–11

The accuracy of the models in the reported studies was essentially acceptable, but none of them has been validated with a large cohort of more than 1000 patients. A diagnostic model integrating clinical characteristics, CT signs, and biomarker results developed and validated in large cohorts of thousands of individuals could have better potential for clinical use.

The purpose of our present study is twofold: on the one hand, we aim to validate the results of published diagnostic models;^10,11 on the other, we seek to use comprehensive data of a large cohort of patients with SPN to identify an ideal solid and accurate diagnostic model for estimating the probability of malignancy in SPNs.

Materials and methods

Patients and clinical data

A total of 5273 patients diagnosed by chest CT scans or X-ray were continuously enrolled from the Department of Thoracic Surgery in the Cancer Institute & Hospital of the Chinese Academy of Medical Sciences between January 2005 and March 2013. The patients were enrolled according to the following criteria: patients had no antineoplastic therapy, radiotherapy, or chemotherapy prior to surgery, nor cancer diagnosis within one year prior to the operation for SPNs; patients had complete clinical data; postoperative histological diagnosis of patients was not the metastatic cancer of extra-pulmonary organs; and patients had received chest CT scans and biomarker detection within 30 days prior to surgery. There were 912 patients excluded because of a lack of complete clinical data. There were 1003 patients excluded because of the detection of metastatic cancer of the extra-pulmonary organs by postoperative pathological diagnosis; most of these patients had a definite cancer history, and these circumstances serve to remind the physician to treat the SPN more actively prior to resorting to surgery, because although the SPN had a clear border, most physicians would not consider these cases as benign lesions.

The remaining 3358 patients with SPNs (1921 men and 1437 women – ratio 1.34:1) were enrolled. Patients were randomly separated into a training set (n = 1679) and a test set (n = 1679). We used the training set to develop a diagnostic model for the malignancy probability of SPN and applied the test set to validate the model. Clinical data collected included: age, gender, smoking history, smoking quantity (pieces-year), history of malignant tumor, and family history of malignant tumor (Table 1). The medical ethics committee of the Cancer Institute & Hospital of the Chinese Academy of Medical Sciences approved this study.

Table 1.

Univariate analysis of data collected from patients included in training and test sets

Classification	Training set (n = 2274)	Test set (n = 2274)	P-value*
Clinical Characters
Male (%)	58.9	55.5	0.051
Age (years)	57.97	58.28	0.395
Diameter of lesion (cm)	1.96	1.95	0.450
Smoking history (%)	42.5	51.5	0.576
Smoking quantity pieces-year	462.7	473.6	0.615
Previous history of malignant tumor (%)	3.9	3.0	0.154
Family cancer history (%)	13.9	13.5	0.763
Biomarker results
NSE (ng/mL)	11.40	11.64	0.181
CEA (ng/mL)	6.79	6.71	0.114
CYFRA 21-1 (ng/mL)	2.62	2.59	0.387
CA125 (ng/mL)	16.95	16.76	0.883
SCC (ng/mL)	0.819	0.829	0.618
Histological diagnosis malignant (%)	77.2	77.7	0.741
CT scan findings
Clear border (%)	9.4	9.9	0.640
Satellite lesions (%)	2.9	3.2	0.689
Cavity (%)	3.9	4.5	0.388
Lobulation (%)	37.0	38.5	0.393
Spiculation (%)	42.7	41.9	0.675
Calcification (%)	5.9	6.1	0.827
Pleura retraction sign (%)	3.9	3.8	0.788
Position of SPN
LUL (%)	23.6	28.1	0.004
LLL (%)	16.1	17.5	0.310
RUL (%)	30.7	29.7	0.573
RML (%)	7.2	5.2	0.018
RLL (%)	22.4	19.6	0.051

*Univariate analysis: t-tests and chi-squared test were performed for proportional differences or mean differences in variables between training set and test set. CA, carbohydrate antigen; CEA, carcinoembryonic antigen; CT, computed tomography; CYRFA 21-1, cytokeratin 19-fragment marker; LLL, left lower lobe; LUL, left upper lobe; NSE, neuron-specific enolase; RLL, right lower lobe; RML, right middle lobe; RUL, right upper lobe; SCC, squamous cell carcinoma; SPN, solitary pulmonary nodule.

Biomarker detection and computed tomography (CT) scans

All patients included in the training and test sets underwent biomarker series tests and chest CT scans. Serum samples were collected from patients one to 10 days before surgery. Serum tumor markers were detected using a commercially available automatic electrochemiluminescence immuno analyzer (ECLIA) (Roche Cobas 6000e601, Basel, Switzerland), for carcinoembryonic antigen (CEA), neuron-specific enolase (NSE), cytokeratin 19-fragment marker (CYFRA 21-1), and carbohydrate antigen 125 (CA125). Squamous cell carcinoma (SCC) antigen was determined using an automated chemiluminescent immunoassay analyzer (CLIA) (Abbott ARCHITECT i1000SR, Illinois, USA ). The CT findings of SPN in the two sets of patients were reviewed as follows: calcification, spiculation, lobulation, pleural retraction, clear border, satellite lesions (some tiny spots around the SPN, not new nodules), cavity, lung side (right or left), location (lobe), and diameter (mm).

Surgical procedure and histological diagnosis

All patients underwent surgical resection of nodules via two kinds of surgical approach: thoracotomy (n = 2771) and video-assisted thoracoscopic surgery (VATS) (n = 587). Procedure methods included lobectomy, pulmonary wedge resection, pulmonary segmentectomy, and lesion enucleation. A pathologist carried out definitive postoperative histological diagnosis reports, and each of the reports was examined and verified by two other highly qualified pathologists to guarantee accuracy. Non-small-cell lung cancer (NSCLC) was confirmed by histopathology, according to the World Health Organization Classification of Tumours of the Lung.¹² There were 2600 malignant SPNs and 758 benign SPNs in this study (Table 2).

Table 2.

Histological diagnosis of malignant and benign solitary pulmonary nodules (SPNs)

Histological diagnosis	3358	Percentage
Malignant SPNs	2600	100
Adenocarcinoma	1767	67.9
Squamous carcinoma	525	20.2
Small cell lung cancer	95	3.7
Adenosquamous carcinoma	42	1.6
Carcinosarcoma	11	0.4
Large cell carcinoma	25	0.96
Alveolar cell carcinoma	77	2.96
Carcinoid tumour	29	1.12
Others†	29	1.12
Benign SPNs	758	100
Pulmonary tuberculosis	127	16.7
Pulmonary hamartoma	133	17.5
Inflammation	126	16.6
Mycotic infection	14	1.8
Sclerosing hemangioma	44	5.8
Fibrosis nodules	33	4.3
Pulmonary sequestration	7	0.9
Bronchogenic cyst	12	1.6
Hemangioma	12	1.6
Atypical adenomatous hyperplasia	18	2.3
Inflammatory fibroblast tumor	8	1.3
Other‡	24	2.6

†MALT lymphoma n = 1, solitary fibrous tumor (SFT) n = 2, epithelioid hemangioendothelioma n = 2, lymphoepithelial carcinoma n = 1, Langerhans cell histiocytosis n = 1, non-Hodgkin's lymphoma n = 2, malignant fibrous histiocytoma n = 1, Hodgkin's lymphoma n = 1, pulmonary mucoepidermoid carcinoma n = 8, inflammatory myofibroblast cell tumors n = 7, leiomyosarcoma n = 1, spindle cell carcinoma n = 1, signet-ring cell carcinoma n = 1. ‡Fibrous histiocytoma n = 6, leiomyoma n = 5, lipoma = 4, clear cell tumour n = 3, lymph node hyperplasia n = 4, sarcoidosis n = 2.

Statistical analysis and diagnostic model

Categorical variables of clinical characteristics, biomarker results, and CT scan findings were compared using the chi-square test. Continuous variables were compared using the unpaired t-test. The mathematical diagnostic model was devised based on the results of multivariate logistic regression analysis. The step-wise procedure was performed to select independent variables from the statistically significant variables in univariate analysis. The Hosmer-Lemeshow test was used to evaluate the goodness-of-fit of the diagnostic model. For each patient, the probability of malignant SPN was predicted using the multivariable model. Receiver operator characteristic (ROC) curves were created, and the area under the curve (AUC) and the prediction accuracy were calculated to evaluate the diagnostic model. SAS version 9.2 (SAS Institute, Cary, NC, US) and MedCalc version 9.6.2.0 were used for statistical analysis. A P-value of <0.05 (two-sided) was considered statistically significant.

Results

Development of the diagnostic model in the training data set

The clinical characteristics, biomarker results, and radiological presentation of the 1679 patients in the training set (383 benign SPNs, 1296 malignant SPNs) are presented in Table 3. The mean age was 58.6 ± 10.7 years. The prevalence of malignant SPNs was 77.2%. Individuals with malignant nodules were older, were more likely to be current smokers, and had more pieces-years of smoking experience. Participants with malignant nodules were more likely to have a family history of malignant tumor. Malignant nodules were larger, compared with benign ones. Malignant nodules were more likely to have the CT signs of speculation, pleural retraction, and lobulation. There was no significant difference in previous history of malignant tumor and cavity rate between the malignant and benign groups.

Table 3.

Univariate analysis of data collected from patients in the training set

Classification	Malignant (n = 1296)	Benign (n = 383)	P-value*
Clinical Characters
Male (%)	58.9	59.0	0.963
Age (years)	59.5	52.7	<0.001
Diameter of lesion (cm)	2.02	1.76	<0.001
Smoking history (%)	49.2	19.8	<0.001
Smoking quantity pieces-year	502.8	226.6	<0.001
Previous history of malignant tumor (%)	4.4	2.1	0.056
Family cancer history (%)	16.0	6.8	<0.001
Biomarker results
NSE (ng/mL)	11.50	11.08	0.083
CEA (ng/mL)	7.15	2.02	<0.001
CYFRA 21-1 (ng/mL)	2.79	2.04	<0.001
CA125 (ng/mL)	18.14	12.89	<0.001
SCC (ng/mL)	0.87	0.65	<0.001
CT scan findings
Clear border (%)	4.6	25.8	<0.001
Satellite lesions (%)	1.3	8.4	<0.001
Cavity (%)	3.5	5.0	0.208
Lobulation (%)	46.3	5.7	<0.001
Spiculation (%)	52.8	8.6	<0.001
Calcification (%)	0.7	23.5	<0.001
Pleura retraction sign (%)	4.6	1.6	0.010
Position of SPN
LUL (%)	24.2	21.4	0.283
LLL (%)	15.8	17.2	0.561
RUL (%)	32.1	25.8	0.023
RML (%)	6.5	9.7	0.055
RLL (%)	21.4	25.8	0.760

*Univariate analysis: t-tests and chi-squared test were performed for proportional or mean differences in variables between malignant and benign solitary pulmonary nodules (SPNs). CA, carbohydrate antigen; CEA, carcinoembryonic antigen; CT, computed tomography; CYRFA 21-1, cytokeratin 19-fragment marker; LLL, left lower lobe; LUL, left upper lobe; NSE, neuron-specific enolase; RLL, right lower lobe; RML, right middle lobe; RUL, right upper lobe; SCC, squamous cell carcinoma.

All of the variables in the training set that were significantly different between benign and malignant SPNs were put into logistic regression for multivariable analysis to construct a diagnostic model that distinguished between benign and malignant SPNs. The results of multivariable analysis are presented in Table 4. The goodness-of-fit statistic by Hosmer-Lemeshow for the derivation dataset was X² = 11.608, P-value = 0.571. The model defined the probability of malignant SPN as follows:

Table 4.

Logistic regression analysis of training set

Factor	Regression coefficient	P-value	Odds ratio value	95% CI
				Lower	Upper
Age	0.035	<0.001	1.035	1.018	1.052
Smoking history†	1.029	<0.001	2.797	1.903	4.110
Family history† of cancer†	0.974	0.014	2.649	1.528	4.594
CEA‡	0.221	<0.001	1.247	1.131	1.374
CA125‡	0.003	0.723	1.003	0.987	1.018
CYFRA 21-1‡	0.200	0.019	1.221	1.034	1.442
SCC‡	0.136	0.373	1.146	0.849	1.546
Diameter of lesion§	0.633	<0.001	1.884	1.465	2.422
Clear border†	−1.631	<0.001	0.196	0.118	0.325
Satellite lesions†	−1.923	<0.001	0.146	0.053	0.403
Lobulation†	2.673	<0.001	14.485	8.348	25.135
Pleura retraction†	0.315	0.567	1.371	0.466	4.034
Spiculation†	2.027	<0.001	7.590	4.915	11.719
Calcification†	−3.295	<0.001	0.037	0.015	0.092
RUL†	0.249	0.196	1.283	0.879	1.872
Constant	−4.294	0.000	0.014

†Ever versus never; ‡ng/mL; §cm. CI, confidence interval.

Probability of malignant SPN = e^x/(1 + e^x), x = −4.294 + (0.035 × age) + (0.221 × CEA) + (0.200 × CYFRA 21-1) + (1.029 × smoking) + (0.974 × family history of cancer) + (0.633 × diameter of lesion +(−1.631 × clear border) + (−1.923 × satellite lesions) + (2.673 × lobulation) + (−3.295 × calcification) + (2.027 × spiculation); e is the base of natural logarithms, while x is the regression coefficient in the logistic regression.

The criteria were:

The diameter of the lesion is measured in centimeters (cm); CEA = serum CEA level (ng/mL); CYFRA 21-1 = serum CYFRA 21-1 level (ng/mL); Smoking = 1 if smoking history is present (otherwise = 0); family cancer history = 1 if family cancer history present (otherwise = 0); clear border = 1 if clear border present in the SPN(otherwise = 0); satellite lesions = 1 if satellite lesions present in the SPN (otherwise = 0); lobulation = 1 if lobulation present in the SPN (otherwise = 0); calcification = 1 if calcification present in the SPN (otherwise = 0); spiculation = 1 if speculated appearance is present in the SPN (otherwise = 0).

The accuracy of the model was evaluated by ROC curve. For the training set, the AUC of the ROC curve was 0.935 (95% confidence interval [CI]: 0.924–0.945, Fig 1). The probability value of 0.659 was selected as the cut-off point. Individuals with probability value >0.659 were diagnosed with malignant SPN; otherwise the subjects were considered to have benign SPNs. The sensitivity and specificity of this model in the training set was 90.7% and 81.2%, respectively.

Figure 1.

Receiver operator characteristic (ROC) curve for the training set.

Validation of the diagnostic model in the test data set

Clinical and biomarker data of the patients in the test set (n = 1679) were used to validate the accuracy of the diagnostic model. Compared with the training set, there was no significant statistical difference in the test set data. The area under the ROC curve of the test set was 0.917 (95% CI: 0.906–0.929, Fig 2). In addition to this result, the data of the test set were used to make the ROC curve using the diagnostic prediction model created by Swensen¹⁰ and Li¹¹ (Fig 2).

Figure 2.

Comparison of three models validated using the test set. , Test set;, Li; , Swensen.

The Swensen model:

Independent factors were: age, smoking history, cancer history, diameter, spiculation, and site in the upper lobe where probability of malignant SPN = e^x/(1 + e^x)

where X = −6.8272 + (0.0391 × age) + (0.7917 × smoking history) + (1.3388 × cancer history) + (0.1274 × diameter) + (1.0407 × spiculation) + (0.7838 × the upper lobe). The area under the ROC curve of test set was 0.788 (95% CI: 0.779–0.796, Fig 2).

The Li model:

Independent factors were: age, diameter, spiculation, family cancer history, calcification, and clear border. The probability of malignant SPN = e^x/(1 + e^x)

where X = −4.496 + (0.07 × Age) + (0.676 × diameter) + (0.736 × spiculation) + (1.267 × family history of cancer) + (−1.615 × calcification) + (−1.408 clear border). The area under the ROC curve of the test set was 0.819 (95% CI: 0.808–0.829, Fig 2).

Discussion

Patients with pulmonary nodules should be evaluated by estimating the probability of malignancy, performing imaging tests for better characterization of the lesions, evaluating the risks associated with various management alternatives, and eliciting the patients' preferences for management.¹³ In the present study, we used comprehensive and large sample data (including clinical characteristics, CT scan signs, and biomarker results, n = 1679) in the training set to develop a new diagnostic model to estimate the malignant probability of SPNs. We then used large sample data in the test set (n = 1679) to validate the model, which achieved a good result of accuracy (AUC of ROC curve was 0.917). Meanwhile, using the data of the test set, we validated another two diagnostic models that have been widely cited.^10,11 We found that the model we built was more accurate than the other two. All of the patients in our study underwent surgeries for pulmonary nodule resection and obtained postoperative histological diagnosis – thus, the nodules in our study have been diagnosed definitively.

Our model has an accuracy that is slightly higher than the model developed by Swensen and his colleagues at the Mayo Clinic. Similar to their model, ours included positive smoking history, older age, and spiculation – independent predictors of malignant SPNs. However, in our model, we found that family cancer history, lobulation, CEA, and CYFRA 21-1 were independent predictors of malignancy. We found that calcification, satellite lesions, and clear borders in CT scans were also independent predictors of non-malignancy. Furthermore, we did not confirm the Mayo finding that upper lobe nodules were more likely to be malignant. In the Mayo model, researchers assumed that a history of lung or extra-thoracic cancer would be strong predictors of malignant SPNs, but we found that previous history of malignant tumor was not an independent predictor of malignancy. Similar to the AUC of ROC curve of 0.830 calculated by the Mayo Clinic using their own database, we validated the model using our test set database, and the AUC of ROC curve was 0.788.

Our model has an accuracy that is also higher than the model developed by Li and his colleagues.¹¹ Therefore, we suggest that our diagnostic model is used as a tool to help guide the clinical decision when it is difficult to make a definitive diagnosis. If a malignant SPN was found using this model, the physician could treat it more actively. Similar to the Li model, our model included a family history of cancer, older age, and spiculation – independent predictors of malignant SPNs. Concurrently, in addition to calcification and a clear border, we found that satellite lesions in CT scans represented another independent predictor of non-malignancy. Unlike the model built by Li and colleagues, we added preoperative biomarker results into our model and found that CEA and CYFRA 21-1 were also independent predictors of malignancy. High preoperative CEA and CYFRA 21-1 levels were significant to independent diagnosis and prognostic factors in patients with stage I NSCLC.^14–16 Because of the addition of more clinical characteristics, CT scan signs, and biomarkers (especially CEA and CYFRA 21-1) to the data in our test set used to validate our model, the AUC of ROC curve could reach 0.917. Apart from our current study, no diagnostic model of SPNs has been built in China using such a large sample of patients and comprehensive data with clinical characteristics, CT scan signs, and preoperative biomarkers.

Our study has several limitations. Although the present study used a large amount of data from patients, it is a single-center study and, thus, independent validation of other data from other centers is required in future studies. Gould and Cummings have shown the diagnostic and cost effectiveness of strategies for SPN management.^17,18 In China, the number of patients receiving regular check-ups using fluorodeoxyglucose-positron emission tomography/computed tomography (PET/CT) is limited because of cost considerations. Hence, the results from PET/CT were not included in this study. The present work is also a retrospective study. However, in surpassing the previous research, it provides the most comprehensive data collection of both clinical, imaging, and biomarker information, with a definitive pathology diagnosis for each patient. Regarding nodule location, there were a large number of patients in this group with pulmonary tuberculosis, which is known to occur more likely in the superior lobe apicoposterior segment and the lower lobe dorsal segment. In addition, the incidence of tuberculosis in China is higher than in Western countries. This study found that there was no significant correlation between nodule position and the probability of malignancy of SPNs.

Conclusion

While test-validation of the model obtained a similar AUC, our results still need further validation in an independent cohort study of patients with SPNs. The risk of malignancy in patients with SPNs is higher with age, smoking, high biomarker results, and in those who manifest spiculation and lobulation in CT scans. Therefore, clinicians need elaborative consideration for all SPNs, especially in patients with the risk factors noted above.

Disclosure

No authors report any conflict of interest.