Investigating the robustness Neighborhood Grey Tone Difference Matrix (NGTDM) and Grey Level Co-Occurrence Matrix (GLCM) radiomic features on clinical Computed Tomography systems using Anthropomorphic Phantoms: Evidence from a multivendor study

Abstract

Objective

To determine if optimized imaging protocols across multiple computed tomography (CT) vendors could result in reproducible radiomic features calculated from an anthropomorphic phantom.

Methods

Materials with varying degrees of heterogeneity were placed throughout the lungs of the phantom. Twenty scans of the phantom were acquired on 3 CT manufacturers with chest CT protocols that had optimized protocol parameters. Scans were reconstructed using vendor-specific standards and lung kernels.

The Concordance Correlation Coefficient (CCC) was used to calculate reproducibility between features. For features with high CCC values, Bland-Altman analysis was also used to quantify agreement.

Results

The mean Hounsfield unit (HU) was 32.93 HU (141.7 to −26.5 HU) for the rubber insert and 347.2 HU (−320.9 to −347.7 HU) for the wood insert. Low CCC values < 0.9 were calculated for all features across all scans.

Conclusions

Radiomic features that are derived from the spatial distribution of voxel intensities should be particularly scrutinized for reproducibility in a multi-vendor environment.

Introduction

Technological advancements in medical imaging hardware and software have enabled devices, such as Computed Tomography (CT) scanners, to be particularly helpful in depicting tumor compositional and morphological features with high quality^1–5. Such advances have enabled the rapid progression of fields like radiomics. For example, in patients with lung adenocarcinoma, Coroller, et al.⁶ reported 35 radiomic features to be prognostic for distant metastasis and 12 features that predict survival. However, any value and practicality of noninvasively quantifying tumor characteristics is largely dependent on the repeatability and reproducibility of quantitative imaging features across medical imaging devices^{6–19, 32}. As such, prior to the clinical application of radiomic features for patient care, they must be found to be repeatable and reproducible under the conditions that might be expected in clinical practice or a clinical trial with multiple scanners or sites^{6–19, 32}.

To date, publicly available retrospective patient image datasets and phantoms have been used to identify robust radiomic features. With test/retest CT scans of 56 non-small cell lung carcinoma (NSCLC) patients Hunter, et al.¹¹ identified machine-robust image features, but also reported that reproduciblity and redundancy of radiomic features depended on both the CT scanner and CT image type. To further understand the influence of CT scan parameters on radiomic feature reproducibility Mackin, et al.¹² compared radiomic features calculated for NSCLC tumors from 20 patients with those calculated for 17 scans of a rectangular radiomics phantom. He showed that the variability calculated from different CT scanners is comparable to that found in CT images of NSCLC tumors. In a retrospective clinical trial Solomon, et al.¹⁹ reported on the degree to which radiation dose and iterative reconstruction algorithms affect the robustness of radiomic features extracted from tumors. However, the study was only conducted on General Electric (GE) CT scanners.

Although these studies highlight that variability in radiomic features exist, they are limited in several respects. In particular, several of the studies have not incorporated the effect of the latest advancements in CT hardware and software such as automated tube current modulation (TCM) and iterative reconstruction. All modern TCM systems are designed to automatically modulate the radiation dose and tube current according to a user-selected image quality reference parameter and the patient size/attenuation profile^35,36. The underlying functionality varies across manufacturers, but the overall intent is to reduce noise variation across the inhomogenous body habitus. Due to concerns surrounding the stochastic risks from exposure to low doses of ionizing raidation dose, patient based test/retest studies used to determine radiomic feature robustness may not be practical in a multi-vendor environment¹². Phantoms may be the most practical means to determine robustness, but the phantoms currently being used are either too homogenous in texture or unrealistic in shape (i.e., have a square or rectangular cross section) ¹² for modern CT systems. Inserts consisting of variable textures¹² were incorporated into an adult-sized anthropomorphic phantom to address these limitations. The anthropomorphic phantom consists of attenuation properties similar to human tissue and provides for a realistic context with which to evaluate modern CT scanners. Hence, the purpose of this study was to investigate if imaging protocols optimized for image quality and radiation dose across multiple CT vendors would result in reproducible radiomic features.

Materials and Methods

Scanning Parameters and Anthropomorphic Phantom

Twenty scans of the hybrid anthropomorphic phantom (Figure 1a, b) were acquired on three CT manufacturers: a 64 detector row CT scanner (Discovery CT750 HD; GE Healthcare, Milwaukee, Wisconsin, USA), 32 detector row CT scanner (Somatom Definition AS, Siemens, Forchheim, Germany), and a 16 detector row scanner (Big Bore Brilliance, Phillips Healthcare, Cleveland, OH, USA). All scans were acquired using modified CT chest protocols that used manufacturer-specific TCM. The scan parameters common to each scanner were: tube potential of 120 kVp, 0.7s rotation time, large body filter, 512 × 512 matrix, and a display field of view of 39 cm. All scans were reconstructed using the manufacturer specific standard and lung kernels and a reconstruction slice thickness of 5 mm. The imaging parameters are listed in Table 1.

Figure 1.

(a), a cross section of the anthropomorphic phantom with the rubber and wood inserts placed throughout the lungs. #104 is the rubber insert. #107 is the sycamore wood. (b), axial CT scan of the phantom cross section displayed on the left. The regions of higher electron density are brighter in the image and the regions of lower electron density are not as bright. (c) Cross section shows the placement of ROIs on each of the inserts.

Table 1.

List of Scanners and Scan Parameters Used in the Study.

Manufacturer/Model	Reconstruction Kernel	Scan Type	Slice Thickness	Pixel Spacing (mm)	Spiral Pitch Factor	kVp	Image Quality Reference Parameter	Indicated CTDIvol (mGy)
GE Discovery CT750 HD	Standard/Lung	Helical	5mm	0.762	0.984	120	NI 14	9.33
Phillips Brilliance 16	Standard(B)/Lung (L)	Helical	5 mm	0.762	0.938	120	150 mAs/Slice	9.20
Siemens Somatom Definition AS	Standard – B40f Lung – B60f	Helical	5 mm	0.762	0.951	120	QRM 253	9.34

kVp indicates kilovoltage peak; mAs is milliampere-second. NI = Noise index. QRM = Quality Reference mAs.

The anthropomorphic phantom consists of five simulated tissue types: soft tissue, lung, brain, spinal cord, and bone. The tissue types were engineered to produce photon attenuation values within 1% of those for real tissues for the bone and the soft tissue substitutes, and 3% for the lung tissue substitute over the range of 30–20 000 keV²⁰. Although the anthropomorphic phantom contains different tissue types, two objects with varying degrees of heterogeneity, shredded rubber and sycamore wood, were introduced. Feature values extracted from the rubber insert were previously demonstrated to span the range of feature values found in human tissue, especially NSCLC¹². The rubber insert presented with a porous, random and strong texture, whereas the wood insert consisted of a regular, repeated texture. While these exact patterns may not be found within human tumors, the enhancement or smoothing resulting from different reconstruction kernels will be similar to what is observed in heterogenous or homogenous tissues. For example, a NSCLC tumor may consist of several different densities and sharp discontinuities at the boundaries producing a wide range of Hounsfield unit (HU) values. A lung reconstruction algorithm would seek to over enhance such boundaries, whereas a standard algorithm would tend to reduce the noise and offer a smoother looking image. Similar observations would be noted with the chosen inserts. The inserts placed within the anthropomorphic phantom are shown in Figure 1a.

Quantitative Image Analysis

The first set of features evaluated in this study included the mean and median HU of the rubber and wood inserts. The second sets of features evaluated are based on the spatial distribution of voxel intensities or CT numbers (i.e. HU values) of a CT image. These features included the neighborhood gray-tone difference matrix (NGTDM)²¹ and the Gray Level Co-occurrence Matrix (GLCM)³⁸. A total of 27 features were used in this study, Table 2 provides a listing. Both features were chosen due to their common use in radiomics research investigations^22–26.

Table 2.

NGTDM and GLCM features used in this study.

	Feature Name	Category
1	Busyness	NeighborIntensityDifference3 (Computed from a 3D image matrix)
2	Coarseness
3	Complexity
4	Contrast
5	Texture Strength
6	Autocorrelation	GrayLevelCooccurenceMatrix3 (Computed from a 3D image matrix)
7	Cluster Prominence
8	Cluster Shade
9	Cluster Tendency
10	Contrast
11	Correlation
12	Difference Entropy
13	Dissimilarity
14	Energy
15	Entropy
16	Homogeneity
17	Homogeneity 2
18	Information Measure Corr1
19	Information Measure Corr2
20	InverseDiffMomentNorm
21	InverseDiffNorm
22	InverseVariance
23	MaxProbability
24	SumAverage
25	SumEntropy
26	SumVariance
27	Variance

Description of features can be found in Amadsun and King in ref. 21, Haralick, R. M., et al., ref. 38 and Aerts, H. et al., ref. 9.

The IBEX radiomics software package (MD Anderson) was used to calculate the NGTDM and GLCM features²⁷. For each insert, ROIs were defined as shown in Figure 1c. The NGTDM features were calculated using a 3D image matrix²⁷. The intensity range and number of bins were determined by evaluating the histogram for each respective insert. As a result, for both inserts, the number of bins used was 32. The intensity range varied with reconstruction kernel and insert type. The neighborhood matrix size was 3 in the X-dimension. The slice thickness was set to 5mm, as shown in Table 1. A second set of features derived from the GLCM were also computed using IBEX. The IBEX algorithm calculates features in 3D in thirteen unique directions²⁷. The distance between pixels d, was set to a value of d = 1.

Reproducibility Analysis

The variation in HU for each insert across all scanners and scan types was first explored. Then, while applying vendor specific standard or lung kernels to each scan, the reproducibility of the NGTDM and GLCM features was investigated. The scans were acquired with optimized acquisition parameters where the CTDIvol, voxel dimensions, pitch, and kVp were nearly equivalent. The following six combinations of scans were included in the reproducibility analysis: GE standard kernel (GE-SK) vs. Siemens standard kernel (S-SK), GE-S vs. Phillips standard kernel (P-SK), and S-SK vs. P-SK. The same sequence was used for the lung kernel (LK): GE-LK vs S-LK, GE-LK vs. P-LK, and S-LK vs. P-LK.

In a separate analysis, the reproducibility of NGTDM and GLCM features calculated on scans reconstructed with sharp lung and smoother standard kernels was explored. For this analysis, only the reconstruction kernel was allowed to vary within and between each CT vendor. Reproducibility of the following scan combinations was considered: GE-SK vs. GE-LK, GE-SK vs. S-LK, GE-SK vs. P-LK; S-SK vs. GE-LK, S-SK vs. S-LK, S-SK vs. P-LK; and P-SK vs. GE-LK, P-SK vs. S-LK, and P-SK vs. P-LK.

Statistical Analysis

The Lilliefors test was used to test the normality of all features extracted from the phantom scans. A p value of less than 0.05 indicated statistical significance. The Concordance Correlation Coefficient (CCC) ^29–31 was used to identify the most reproducible radiomic features. The CCC, as proposed by Lawrence and Lin, describes the extent to which paired measurements diverge from perfect agreement, reflecting both systemic differences between repeated measurements and variability^{11, 28,29}.

The CCC is defined as:

$$\mathit{CCC}=\frac{{\sigma }_1{\sigma }_2{\rho }_{12}}{{\sigma }_1^2+{\sigma }_2^2+{\left({\mu }_1-{\mu }_2\right)}^2}$$

Where ${\sigma }_1^2\mathit{and}{\sigma }_2^2$ are the variances for each class, μ₁ and μ₂ are the group means. As suggested by McBride, et al., reproducible features with CCC > 0.9 were defined. A CCC value of greater than 0.9 indicated highly correlated values, whereas those below 0.9 were considered to be poorly correlated^29–31. Kernels and scanning parameters that produced highly correlated results (CCC > 0.9) were identified. All the statistical analysis was performed by software written in MATLAB R2014b on a Microsoft Windows system.

A Bland-Altman analysis and regression analysis was performed next. Due to the large number of possible combinations, plots with regression and the Bland-Altman analysis were limited to the features found to have the highest CCC values. Bland-Altman plots are scatter plots with the vertical axis showing the difference between two variables versus against their average in the horizontal axis³². The mean difference and 95% limits of agreement were reported on each plot. If features from any two scans were comparable, then the differences on the Bland-Altman plot would be small or near zero³². A negative value implied that feature values are greater on one particular scan method relative to another. A positive value indicated lower feature values relative to any particular scan method. The reproducibility coefficient (RPC) was calculated as 1.96 * standard deviation. The coefficient of variation (CV) was also reported within the Bland-Altman plots³².

Results

The normality test revealed that most NGTDM- and GLCM-derived features appeared to have normal distribution (p < 0.05). For those NGTDM and GLCM features found to be non-normal, a log-normal transformation was performed. Figure 2 shows the distribution of HU values for each insert across all scanners. With the rubber insert, the average HU across all scans was found to be 32.93 HU with a range of 141.7 HU to −26.5 HU. The mean HU of the sycamore wood insert was found to be −347.2 HU with a range of −320.9 HU to −347.7 HU.

Figure 2.

Left: Distribution of the HU value for the rubber insert. Right: Distribution of HU for the sycamore wood insert for all scans. Each HU value was extracted using equivalent sized ROIs and at the same slice locations. SK: Standard Kernel, LK: Lung Kernel.

Figure 3 shows the reproducibility of radiomic features between images reconstructed with vendor specific standard kernels. Within the heatmaps, a lighter color indicates a higher CCC value, which suggests greater reproducibility. The corresponding numerical CCC values of these radiomics features are overlaid onto the heatmaps. Figure 3a shows the CCC values for the rubber insert and Figure 3b shows the values for the wood insert. For both the rubber and wood inserts, all radiomic features were found to have CCC values that were less than 0.9. Similarly, Figure 4 compared the reproducibility of radiomic features across CT vendors, but for scans reconstructed with vendor specific lung kernels. In this case as well, all radiomic features were not found to be reproducible across vendors, with CCC < 0.9.

Figure 3.

Heatmap consisting of CCC values for the radiomic features evaluated in this study. The CCC (0 to 1) was computed from repeat CT images acquired with vendor specific standard reconstruction kernels for (a) the rubber insert and (b) wood insert. The lighter the shade of gray, the higher the CCC value.

Figure 4.

Heatmap consisting of CCC values for the radiomic features evaluated in this study. The CCC (0 to 1) was computed from repeat CT images acquired with different reconstruction kernels for (a) the rubber insert and (b) wood insert. The lighter the shade of gray, the higher the CCC value.

When comparing reproducibility of radiomic features extracted from an image reconstructed with a standard kernel to the same features extracted from an image reconstructed with a lung kernel, we found that the CCC is less than 0.9 for all features and across all vendors, see Figure 5a and 5b.

Figure 5.

Heatmap consisting of CCC values for the radiomic features evaluated in this study. The CCC (0 to 1) was computed from repeat CT images acquired with different imaging settings for (a) the rubber insert and (b) wood insert. The lighter the shade of gray, the higher the CCC value.

Regression and Bland-Altman analysis were performed next on the features within Figure 3 and Figure 4 that had the largest CCC values. Figures 6 and 7 show the Bland-Altman analysis with limits of agreement, RPC, CV and the regression analysis for the GLCM homogeneity and GLCM SumEntropy feature calculated for the wood insert. The GLCM homogeneity feature had a CCC of 0.387 between Phillips-SK and Siemens-SK scans. The mean difference was 0.0009, (95% CI, −0.02 to 0.02), with a CV of 3.2% and RPC of 0.02. The CCC of the GLCM SumEntropy feature of the Phillips-SK and Phillips-LK was CCC = 0.785. The Bland-Altman plot in Figure 7 shows that the mean difference for GLCM SumEntropy feature (CCC = 0.785) was slightly less than zero at 0.02 (95% CI, −0.05 to 0.01) with a CV of 0.33% and RPC of 0.03. The slight negative mean difference suggests that the GLCM SumEntropy feature may have had, on average, higher values on the Phillips-LK as compared to the Phillips-SK scans.

Figure 6.

Comparison of GLCM Homogeneity feature calculated on images of the wood insert from Phillips and Siemens CT scans. The scans were reconstructed using the vendor-specific standard kernel. Left: Scatter plot with regression fit and confidence intervals plotted. Slope = 0.33 and intercept = 0.39. Right: Bland-Altman plot with lower and upper limits. RPC = reproducibility coefficient. CV = coefficient of variation.

Figure 7.

Comparison of GLCM SumEntropy feature calculated on images of the wood insert from Phillips CT scans. The scans were reconstructed using the vendor-specific standard and lung kernels. Left: Scatter plot with regression fit and confidence intervals plotted. Slope = 0.90 and intercept = 0.47. Right: Bland-Altman plot with lower and upper limits of agreement. RPC = reproducibility coefficient, CV = coefficient of variation.

Discussion

The anthropomorphic phantom used in this study consisted of realistic tissue attenuation properties and components that are reproducible over time and position (i.e., the phantom is not compressed over time). For such an idealized case, we have demonstrated that with optimized scanning parameters across CT vendors, where voxel dimensions, CTDIvol, pitch, kVp and reconstruction kernels were similar, NGTDM and GLCM radiomic features were found to be not reproducible. The importance of deriving robust, reproducible and repeatable quantitative metrics from CT scans is critical for the progress of radiomics and personalized medicine. With increasingly large numbers of publications emerging about the connection between underlying tumor phenotype and radiomic features, strong consideration must be paid to the robustness of radiomic features, particularly NGTDM and GLCM features. In addition to publications demonstrating linkage of features to tumor phenotype, an equal number of publications are demonstrating the impact that CT acquisition scan parameters have on robustness of radiomic features^6–16. One solution being proposed for robust radiomic features has been the standardization of CT acquisition methods⁴⁰. However, as our results show here, even while using nearly identical scanning parameters for three CT scanners, radiomic features based on the NGTDM and GLCM methods are not consistent across the scanners used in this study. There are specific intrinsic factors within each scanner that cannot be compensated for.

The original inception of CT scanners was not for them to be accurate, quantitative measuring devices as found in calibration style laboratories³². Consistency of the fundamental unit of measurements for all CT scanners, the HU, is dependent on a number of items, such as the scanner detector characteristics and design, calibration methods, and quality control/maintenance⁴¹. Additional uncontrollable factors that vary from patient to patient and limit the accuracy of HU values is the fundamental physics of x-ray interaction that result in beam hardening artifacts⁴¹. As an x-ray beam hardens, the effective energy increases, resulting in voxel intensity or HU values that deviate from expectations. Hence, standardization of imaging protocols alone may not be enough to ensure robust NGTDM and GLCM radiomic features in a multi-vendor environment. For radiomics to progress, the engineering of features that are specific for textures found on 3D medical images, particularly CT images may be necessary, and/or robust correction factors may need to be implemented to ensure accuracy within a multi-vendor environment.

Although the data presented here is unique in that it is derived from anthropomorphic phantoms, some limitations must be discussed. Due to available resources, this study included only three scanners and a subset of image acquisition parameters that are used clinically. Siemens CT scanners offer many additional kernels that are used for different types of exams (e.g., body, head, etc.); however, we focused primarily on a kernel intended to be used with chest CT exams. Further, only images reconstructed with traditional filtered back projection were considered. Iterative reconstruction (IR) algorithms are increasingly being used in practice and are known to produce images with differing textures^19,34. However, due to the proprietary nature and continuous evolution of these IR algorithms, developing a consistent set of radiomics features that accurately describe the tumor microenvironments, or developing correction factors from images reconstructed using IR may not be practically feasible. Additionally, the inserts used within this study were of a fixed, small size, and ROIs used to calculate texture parameters did not include borders of the objects. Hence, the small ROI and small size of the inserts may have introduced bias into the final results. A follow up study will aim to investigate the impact of ROI size on feature reproducibility and also reasons as to why within vendor kernels presented with higher CCC values for some features. Lastly, although the anthropomorphic phantom consists of materials that simulate the attenuation properties of human tissue, it is static, of one size, and issues, such as patient motion, could not be accounted for.

In conclusion, we sought to determine if standardized imaging protocols could produce the most robust radiomic features. However, even under optimized scanning conditions with an idealized anthropomorhpic phantom, the NGTDM and GLCM features extracted from CT images were not reproducible across scanners. Radiomic features that are derived from the spatial distribution of voxel intensities should be particularly scrutinized for accuracy.