Comparison of prediction models with radiological semantic features and radiomics in lung cancer diagnosis of the pulmonary nodules: a case-control study

Abstract

Purpose

To compare the ability of radiological semantic and quantitative texture features in lung cancer diagnosis of pulmonary nodules.

Materials and methods

A total of N = 121 subjects with confirmed non-small-cell lung cancer were matched with 117 controls based on age and gender. Radiological semantic and quantitative texture features were extracted from CT images with or without contrast enhancement. Three different models were compared using LASSO logistic regression: “CS” using clinical and semantic variables, “T” using texture features, and “CST” using clinical, semantic, and texture variables. For each model, we performed 100 trials of fivefold cross-validation and the average receiver operating curve was accessed. The AUC of the cross-validation study (AUC_CV) was calculated together with its 95% confidence interval.

Results

The AUC_CV (and 95% confidence interval) for models T, CS, and CST was 0.85 (0.71–0.96), 0.88 (0.77–0.96), and 0.88 (0.77–0.97), respectively. After separating the data into two groups with or without contrast enhancement, the AUC (without cross-validation) of the model T was 0.86 both for images with and without contrast enhancement, suggesting that contrast enhancement did not impact the utility of texture analysis.

Conclusions

The models with semantic and texture features provided cross-validated AUCs of 0.85–0.88 for classification of benign versus cancerous nodules, showing potential in aiding the management of patients.

Introduction

Management of pulmonary nodules is a problem in clinical scenarios, in part due to increasing use of multislice computed tomography (CT) with contiguous thin sections, considered the gold standard for pulmonary nodule detection [1]. For both screening and incidental findings, it can be challenging to classify pulmonary nodules between benign and malignant at first presentation, and overdiagnosis is a challenging issue [2–4].

It has been accepted by clinicians that the pretest probability of cancer can aid and direct the physician in the diagnosis of pulmonary nodules in a cost-effective way. While many clinicians and radiologists rely on clinical experience to assess the likelihood that a nodule is cancerous, there is recent research using quantitative methods to inform the assessment of a newly discovered pulmonary nodule. Prediction models have been developed that utilize both clinical and radiological data to classify nodules. A few of these models [5–7] have been independently validated to exhibit classification accuracies (as measured by the area under the receiver operator characteristic curve, or AUC) in the range of 0.57 to 0.90 [8–10].

The emerging field of “radiomics” has been recently studied as a method to extract more quantitative information from radiological images to aid clinical decisions. In radiomics, mathematical formulae that quantitatively describe various image qualities (e.g., contrast, homogeneity, texture) are defined in a region and compared to outcomes to form a predictive model. Providing a definition of the radiomics paradigm, Gilles et al stated that “Images Are More than Pictures, They Are Data” [11]. Specifically, radiology images are transformed to higher dimensional data to search for useful correlations that define a useful radiographic phenotype before, during, or after therapy [12]. Recent developments in radiomics indicate that quantitative analysis of images can help discriminate different types of cancer, predict disease outcomes, and determine response to treatment [13–17]. The application of radiomics methods to the management of pulmonary nodules has been suggested [18] and evaluated in patients in at least two previous studies [19, 20].

Quantitative texture features may compliment traditional prediction models for pulmonary nodule diagnosis, and a model that integrates these approaches may lead to increased predictive classification accuracy. In this study, we created individual models with exclusive texture or clinical and semantic features as well as their combination in order to assess the predictive power of each. We also examined the ability of quantitative texture features to compliment clinical data (e.g., smoking history) and semantic imaging features (e.g., spiculated nodule, ground glass opacity), with the goal of determining how radiomics may help with pulmonary nodule classification.

Materials and methods

Patients

Data were prospectively collected from October 2010 to September 2015 for a cohort of 492 subjects enrolled in the Fred Hutch Lung Cancer Early Detection and Prevention Clinic (LCEDC) under an active institutional review board, and written informed consents were acquired. All of the cases were reviewed by a multidisciplinary lung nodule board, and CT images were collected at 3–12-month intervals until they were determined to be malignant or benign. For our retrospective study of the LCEDC data, IRB approval and a waiver of written informed consent were obtained. Inclusion criteria were as follows: (1) Only the patients with at least one pulmonary nodule that was at least 1 mm in diameter, when using lung parenchymal CT display thresholds, on the baseline CT image were included [7]. (2) For patients with malignant tumors, only subjects with nodules that were determined to be non-small cell lung cancer (NSCLC) by histopathology before the consent date of May 2014 were included. (3) For benign nodules, only those confirmed by histopathologic examination of tissue obtained via surgical resection, or the lesion was found to be stable radiographically for at least 2 years of follow-up, or resolved under CT surveillance, were included. There were N = 125 NSCLC case subjects included, which were 1:1 matched with subjects with benign nodules. Matching was first based on gender (exact match), then age (± 2 years). In attempting to match the NSCLC cases with controls for pack years of smoking, we found that matching was not possible due to the limited sample size. Clinical demographics of age, gender, race, BMI, smoking behavior, including current smoking status, pack-years, and histology were recorded. We excluded 12 subjects who did not have qualifying CT scan or did not have full clinical demographic records. Figure 1 summarizes the flowchart of the study enrolling process and reasons for patient exclusion. A total of 238 patients were included in this study.

Fig. 1.

Flowchart of the study enrolling process and reasons for patient exclusion

CT imaging acquisition

The CT scans used in this study were from multiple institutions using local imaging protocols. This resulted in non-uniform imaging protocols across patients in the study. The CT images were acquired with or without contrast enhancement. Details regarding the different protocols and acquisition parameters are in the Supplementary Material.

Nodule segmentation

For each patient, only the largest nodule confirmed by histopathology, or confirmed to be benign by surveillance, was chosen for analysis. Contours defining the nodule region (Fig. 2) were manually delineated slice-by-slice on the transaxial CT images by a thoracic radiologist (W.W., 6 years of experience) using the MIM image viewing and analysis software (MIM Software, Inc.). For the nodules which are larger than 5 mm in long-axis diameter, the slice thickness of the images for contouring is 2.5 mm, while for nodules which are equal or less than 5 mm, the slice thickness for contouring is 1.25 mm. Lesion volumes were first contoured in a mediastinal window (WW 350 HU WL 40 HU) to identify the boundaries with the chest wall or other soft tissues, then in a lung window (WW 1324 HU WL −362 HU) to apture the extent in the lung parenchyma. A 3D volume of interest (VOI) was created to encompass the entire nodule and then converted to a binary mask.

Fig. 2.

Example images illustrating the categories of patient (NSCLC vs. benign, and contrast vs. non-contrast imaging protocol) and pulmonary nodule segmentation of each

Radiomics feature extraction

For each patient, the CT image and corresponding binary VOI mask were imported into the PORTS radiomics software package [21] (details in Supplementary Material). PORTS computed 42 quantitative image and texture metrics from the voxels within the VOI (Supplementary Table S2 and Fig. 3).

Fig. 3.

Examples of PORTS texture analysis showing the quantitative values of four different texture features

Semantic feature extraction

A second thoracic radiologist (S.P., 12 years of experience), blinded to clinical and histologic findings, evaluated semantic features including nodule count, location, size, shape, density, and the visual presence and the types of invasiveness, lymphadenopathy, emphysema, fibrosis, and asbestosis. In total, 13 categories of semantic features from CT images were extracted for the analysis.

Statistical analysis

Univariate tests were performed to assess the difference of the estimated features between the case and control status using the PASW Statistics16.0™ analysis package (SPSS Inc.). The Fisher exact test was used for the categorical data and the Mann-Whitney U test was used for continuous data.

LASSO penalized logistic regression [22, 23] was used as a variable-selection procedure to build a low-dimensional logistic regression model. LASSO was performed using the glmnet package in R version 3.4.3 [24]. We constructed three models based on subsets of variables (details in Supplementary Material). The three models are “CS” using the clinical and semantic variables only, “T” using the texture features as well as the CT scan in kilovoltage peak (kVp), and “CST” using all of the clinical, semantic, and texture variables. In each case, the variables selected by the LASSO were used to fit an ordinary logistic regression model and estimate the regression coefficients. For the resulting models, a receiver operator characteristic curve (ROC) was created and the area under the curve (AUC) was computed to evaluate the prediction capability for each model separately. For each model, we performed 100 trials of fivefold cross-validation, where fourfifths of data was used as training and one-fifth of the data was used for testing. The AUC of the cross-validation study (AUC_CV) was calculated together with its 95% confidence interval.

Since the CT images were approximately equally divided between those with and without contrast enhancement, we also assessed the impact of contrast enhancement on texture features within the CT images by comparing the abilities of the texture-only models (i.e., the model T) to predict nodule status. The goal was to evaluate if there was any impact of contrast enhancement on the texture model and no attempt was made to compare with the CS and CST models described above.

Results

Clinical characteristics

Clinical characteristics for the total 238 patients and their distribution in case or control group are summarized in Table 1. The mean age was 63.8 ± 10.6 years. There were 121 NSCLC subjects and 117 control subjects which included 64 cases with the nodules stable for at least 2 years, 24 cases with the nodules resolved during the follow-up, and 29 cases with confirmed histopathology diagnosis. There was no difference for the mean age, proportion of males, or BMI between the NSCLC and control groups. However, the control and NSCLC groups exhibited some difference in smoking history with the control group containing patients who were former (i.e., not current) smokers or who had never smoked, while there were more current smokers in the NSCLC group (p < 0.05). We further reviewed all cases of former smokers and found that the minimal cessation time was 18 days. For most of the former smokers, the cessation time was longer than 1 year and just 5 cases (5/123) had a cessation time of less than 1 year. The control group had a mean of 25.3 pack-years smoked versus 32.4 pack-years for the NSCLC group; however, there was no significant difference (p = 0.064). We note that reported univariate p values in Table 1 (and Supplementary Table S1) are not claims about the statistical significance of the models, as it relates to the population of all cancer samples, but rather they are reported here for descriptive and comparative purposes only.

Table 1.

Clinical characteristics of participants with NSCLC and benign nodules

Category: data type	Full sample	Benign	NSCLC	p* value
Age: mean (± SD)	63.8 (±10.6)	63.0 (±11.5)	64.4 (±9.52)	0.42
Male: N (%)	106 (44.5)	53 (45.3)	53 (43.8)	0.90
BMI: mean (± SD)	27.8 (±7.2)	27.7 (±7.3)	28.0 (±7.1)	0.49
Pack-years: mean (± SD)	28.9 (±26.0)	25.3 (±22.1)	32.4 (±29.0)	0.06
Smoking history: N (%)				0.04
Never smoker	43 (18.1)	26 (22.2)	17 (14.0)
Former smoker	123 (51.7)	64 (54.7)	59 (48.8)
Current smoker	72 (30.3)	27 (23.1)	45 (37.2)
Histology: N (%)				–
Adenocarcinoma	89 (37.4)	–	89 (73.6)	–
Adenocarcinoma in situ	13 (5.5)	–	13 (10.7)	–
Squamous	17 (7.1)	–	17 (14.0)	–
Large cell	2 (0.8)	–	2 (1.7)	–
Infections	10 (4.2)	10 (8.5)	–	–
Inflammation	17 (7.1)	17 (14.5)	–	–
Choroid hamartoma	1 (0.4)	1 (0.9)	–	–
Fibrosis/scarring	1 (0.4)	1 (0.9)	–	–

p values are uncorrected for multiple comparisons and are used only descriptively

A former smoker is defined as an adult who has smoked at least 100 cigarettes in his or her lifetime but who had quit smoking at the time of interview. This is consistent with the definition of the National Health Interview Survey (NHIS) from Centers for Disease Control and Prevention (CDC)

^*Mann-Whitney U test for continuous data, Fisher exact test for categorical data

Semantic and texture features

The Fisher exact test and Mann-Whitney U test revealed that six categories of semantic features have the potential for distinguishing between cases and controls. (Supplementary Table S1). We report here the unadjusted p values as a means to describe their relative strength for discrimination. These features include emphysema degree (p = 0.01), nodule location (p = 0.02), size including long-axis diameters (L), shortaxis diameters (S), height (H), and the bounding volume maximum length (VM, defined as $VM=\sqrt{L^2+S^2+H^2)}$) (p < 0.001 for each), shape (p < 0.001), invasiveness (p < 0.001), and lymphadenopathy (p < 0.001). No difference was detected between two groups for pattern of emphysema, region of emphysema, fibrosis, nodule count, or nodule density. The distribution of each semantic feature for the two groups is summarized in Supplementary Table S1. With the exception of features F9, F16, F17, and F31, all quantitative texture metrics showed evidence of a difference between the NSCLC and control groups (p < 0.05).

Multivariate and ROC analysis of clinical, semantic, and texture features in classification of cases versus controls

Model T used the texture features and kVp as input variables. The LASSO regression selected 6 (out of 43) variables: F10 (GTSDM-Correlation), F19 (GTSDM-Information Correlation1), F21 (GTSDM-Autocorrelation), F22 (GTSDM-Dissimilarity), F41 (GLZSM-Zone Size Non-Uniformity), and F42 (GLZSM-Zone Size Percentage). The AUC_CV for model T was 0.85 with a 95% confidence interval of 0.71–0.96. Model CS used the clinical and semantic variables only. The LASSO selected 5 (out of 22) variables: smoothness, spiculation, invasiveness, lymphadenopathy, and VM. The AUC_CV was 0.88 with a 95% confidence interval of 0.77–0.96. Model CST combined all of the clinical, semantic, and texture variables. The LASSO selected 7 (out of 64) variables: smoothness, spiculation, invasiveness, lymphadenopathy, VM, F19 (GTSDM-Information Correlation1), and F22 (GTSDM-Dissimilarity). The AUC_CV was 0.88 with a 95% confidence interval of 0.77–0.97. These results are presented in Table 2 and Fig. 4.

Table 2.

Prediction performance of the three different models

Model	Subjects	Variables considered	Number of LASSO-selected top predictor variables	AUC (all data)	AUCCV (95% CI)
T	238	43	6	0.86	0.85 (0.71–0.96)
CS	238	22	5	0.89	0.88 (0.77–0.96)
CST	238	64	7	0.89	0.88 (0.77–0.97)

Fig. 4.

ROCs and cross-validated AUC_CVof three different models. Solid lines are the fivefold cross-validated AUCs and the dotted lines in corresponding colors refer to the 95% confidence interval for each AUC_CV

Comparison of models of contrast versus non-contrast images

There were 125 patients with contrast-enhanced images and 113 patients with non-contrast-enhanced images. The AUC (without cross-validation) of the model T was 0.86 both for images with and without contrast enhancement. When these data are combined into a single population, the AUC was also 0.86. The three ROC curves and AUCs are shown in Supplementary Fig. S1.

Discussion and conclusion

We examined the capabilities of clinical, semantic, and image texture features to assess cancer risk in pulmonary nodules. Our cross-validated multivariate analysis shows that models CS and CST had AUC_CV = 0.88, slightly larger than model T with AUC_CV = 0.85 (Fig. 4). The 95% confidence intervals of all three models show that the ROC performance largely overlapped, demonstrating similar performance of all three models. While the AUC is a global measure accounting for all possible sensitivity and specificity thresholds, for comparison purposes, we can estimate that for a true-positive rate (sensitivity) of 90%, the false-positive rate (1-specificity) ranges from 34 to 38%. If the true-positive rate is relaxed to 80%, the false-positive rate is reduced to a range of 23 to 31% for the three methods (Fig. 4). These models were controlled for gender and age, but not smoking history or smoking pack-years. To check the impact of these factors we separately included smoking history and smoking pack-years into the models. When adjusted for smoking history, the AUC_CV values were 0.88, 0.88, and 0.85 for the CS, CST, and T models. In other words, there was no detectable change compared to the models without any adjustment. When adjusted for smoking pack-years, the AUC_CV values increased very slightly to 0.89, 0.89, and 0.86 for the CS, CST, and T models, respectively—very similar to the performance of the models with the smoking history adjustment. The similar results may be due to a high correlation between smoking pack-years and smoking history.

These results indicate that models based on semantic features determined by an experienced thoracic radiologist will likely slightly outperform models based only on computed texture features. This is supported by the recent work of Liu et al who demonstrated that a set of clinically relevant radiologic features can be readily scored to effectively determine lung cancer risk for solitary pulmonary nodules [25].

Even so, the texture features perform nearly as well and can be used to standardize performance across clinical sites. In addition, work continues in the exploration of radiomics methods with larger cohorts so further performance improvements are also possible.

A key component is the integration of accurate and efficient lung nodule delineation or segmentation methods. Evaluation of this component was beyond the scope of this study, but we note that manual delineation is time-consuming and known to suffer from intra- and inter-reader variability [26]. However, recent studies have shown that some automated and semi-automated lung nodule segmentation tools can reduce variability [27]. In the event an automated or semi-automated lung nodule delineation is incorporated, the radiomics workflow could be very efficient. This might be particularly helpful in instances where radiologists have less lung specific training or expertise.

To our knowledge, the inclusion of quantitative texture features with semantic features in our predictive model is unique among similar studies, including the Mayo Clinic model [6], the Veteran’s Affairs model [5], the Herder model [28], and the model from Brock University [7], which use clinical and limited semantic features. One 2016 study found that a model using texture features performed statistically similarly to that using the Brock University model on the NLST (National Lung Screening Trial) dataset with accuracy of 78% [19]. Another 2018 study of a relative small sample demonstrated a radiomics-only model with the ability to discriminate between malignant and benign pulmonary nodules with an accuracy of 84% [20].

To evaluate if it is reasonable to include both non-contrast-enhanced and contrast-enhanced images in our model, we performed a substudy to compare the texture feature performance in the contrast and non-contrast groups. Our analysis showed that quantitative texture features perform similarly in patient populations with and without contrast-enhanced images (AUC of 0.86). Although one previous study reported that the predictive performance of quantitative texture features is superior in non-contrast-enhanced images (AUC = 0.86) than those with contrast (AUC = 0.83) [29], a more recent study on discrimination of lung-invasive adenocarcinoma validated the radiomics signature in two independent non-contrast-enhanced and contrast-enhanced cohorts and found the performance was similar [30], consistent with our findings. In the discussion for this study, it was hypothesized that the radiomics signature may be independent of the injection of a contrast agent as the selected radiomics features measure the correlation, uniformity, or deviation of the pixels, which reduces the effect of contrast increase of the overall image intensity. Although plausible, this hypothesis should be confirmed. Regardless, as our model is independent with regard to the use of contrast and its combination of images with multiprotocols, this may allow for broader applicability in clinical care.

There were several limitations in our study. First, this was a retrospective analysis with a relatively small population that included prediagnostic (screening) and diagnostic patients. Second, we did not have an independent patient population as a validation cohort, as would be required to assess the accuracy the models. Instead, these models provide a proof of principle regarding the role of radiomic and semantic features to facilitate the clinical assessment of pulmonary nodules. For validation, a follow-up study with an independent patient population is needed. Potentially another limitation is the non-standardized image acquisition and definition of radiological semantic features. There is an unresolved debate on the requirement for standardized protocols versus the need for methods that work in a typical imaging network [31]. In general, we expect standardized acquisition and processing to enable distinguishing smaller differences between tumor phenotypes. In clinical practice, however, a wide variety of scanning techniques and parameters are used. In CT phantom studies, differences in slice thickness and reconstruction algorithm (standard versus lung) were shown to substantially change radiomic feature values [32]. However, recent studies on the reproducibility of features extracted from lung CT images have demonstrated that a subset of radiomic features are reproducible and informative even in the presence of scan protocol differences [33, 34]. In addition, Zhao et al [33] and Lu et al [35] showed that use of both 1.25-mm and 2.5-mm slice thicknesses is acceptable if the use of “lung” image reconstruction filters is avoided, as was the case for our study. Finally, we note that a recent study concluded that voxel-size resampling is an appropriate preprocessing step for image datasets acquired with variable voxel sizes to obtain more reproducible CT features [36]. As noted in the supplementary data, this was the procedure used for the radiomics analysis presented here. Another potential limitation of our approach became apparent for 8 of the 117 control cases that had nodules smaller than 4 mm. Since nodules less than 4 to 6 mm are considered less significant clinically [1, 37], and the small size may prevent calculation of texture metrics, the impact of these smaller lesions is unclear. In our analysis, only one texture metric was not calculable for 7 of the cases with nodules smaller than 4 mm. We estimated these missing values using the k-nearest-neighbor approach by grouping subjects with the similar characteristics of age, race, smoking history, and BMI status [38]. When the most important texture variables were chosen using LASSO regression as described above, the non-calculable texture metric was not selected. Thus, for our study, we believe there was no impact from these few small nodules. However, the issue of non-calculable texture metrics for very small nodules should be considered in future studies. If there is an observable effect, then the use of interpolation to smaller voxels might be considered.

Despite these limitations, we conclude that using semantic and quantitative texture features from CT images has the potential to aid and facilitate the clinical decision or management of pulmonary nodules.

Key Points.

• Pretest probability of cancer can aid and direct the physician in the diagnosis and management of pulmonary nodules in a cost-effective way.

• Semantic features (qualitative features reported by radiologists to characterize lung lesions) and radiomic (e.g., texture) features can be extracted from CT images.

• Input of these variables into a model can generate a pretest likelihood of cancer to aid clinical decision and management of pulmonary nodules.

Abbreviations

CI

Confidence intervals

CT

X-ray computed tomography

NSCLC

Non-small cell lung cancer

VM

The bounding volume maximum length

VOI

Volume of interest