Reproducibility and location-stability of radiomic features derived from cone-beam computed tomography: a phantom study

Abstract

Objectives:

This study aims to determine the reproducibility and location-stability of cone-beam computed tomography (CBCT) radiomic features.

Methods:

Centrifugal tubes with six concentrations of K₂HPO₄ solutions (50, 100, 200, 400, 600, and 800 mg ml⁻¹) were imaged within a customized phantom. For each concentration, images were captured twice as test and retest sets. Totally, 69 radiomic features were extracted by LIFEx. The reproducibility was assessed between the test and retest sets. We used the concordance correlation coefficient (CCC) to screen qualified features and then compared the differences in the numbers of them under 24 series (four locations groups * six concentrations). The location-stability was assessed using the Kruskal-Wallis test under different concentration sets; likewise, the numbers of qualified features under six test sets were analyzed.

Results:

There were 20 and 23 qualified features in the reproducibility and location-stability experiments, respectively. In the reproducibility experiment, the performance of the peripheral groups and high-concentration sets was significantly better than the center groups and low-concentration sets. The effect of concentration on the location-stability of features was not monotonic, and the number of qualified features in the low-concentration sets was greater than that in the high-concentration sets. No features were qualified in both experiments.

Conclusions:

The density and location of the target object can affect the number of reproducible radiomic features, and its density can also affect the number of location-stable radiomic features. The problem of feature reliability should be treated cautiously in radiomic research on CBCT.

Introduction

The concept of radiomics, the high-throughput extraction of large amounts of image features from radiographic images, was first proposed by Dutch scholar Philippe Lambin in 2012, ¹ and its advantages lie in maximizing the information potential of radiographic images and serving as a data-driven biomarker in the context of precision medicine. In recent years, with the rapid development of artificial intelligence (AI) and the upgrading of various software and hardware, radiomics has become a hot topic for research in medical imaging.

Cone-beam computed tomography (CBCT) has attracted wide attention in dentistry because of its superior spatial resolution of bone tissue and less radiation exposure compared with traditional computed tomography (CT). Similar to CT, MRI, PET, ultrasound, and other imaging modalities, CBCT has become a platform for numerous scholars attempting to apply radiomics. Through the use of radiomics on CBCT, several researchers had achieved promising results in areas such as jaw diseases, ^2–4 temporomandibular joint diseases, ^5–7 periapical diseases, ^3,8,9 and dental implantation. ¹⁰

Nonetheless, the inherent limitations of CBCT cannot be overlooked. Due to the differences in technical principles between CBCT and CT, confounding imaging factors such as noise, scattered radiation, and beam hardening are more pronounced in CBCT, which increases the unreliability of grey value utilization. ^11–13 Specifically, the randomness of the above confounding factors may lead to the uncertainty of the local grey value, which inevitably affects the reproducibility of some radiomic features. It is also noteworthy that some previous studies proved that the grey value of CBCT can be influenced by the location of the target object in the field of view (FOV). ^14,15 This raises potential hazards for CBCT radiomic research based on grey value.

The main purpose of this study is to explore the reproducibility and location-stability of radiomic features derived from CBCT images, using a customized phantom.

Methods and materials

Phantom

Our phantom design was inspired by the research of Amanda Candemil et al, ¹⁶ in which the phantom was developed for the evaluation of grey level homogeneity. The phantom content was modified according to the purpose of current study. The basic frame of the phantom was composed of a cylindrical container and a 17-hole fixing plate which made of polymethyl methacrylate. Seventeen centrifugal tubes filled with reference K₂HPO₄ solution were loaded in the fixing plate. Six concentrations of K₂HPO₄ aqueous solution, i.e., 0, 100, 200, 400, 600, and 800 mg ml⁻¹, were prepared to mimic the different densities of bones. Before assembling the phantom, we added an appropriate amount of water to the cylindrical container to simulate the soft tissue. Figure 1b shows the assembled phantom.

Figure 1.

Before statistical analysis of the data, we need to finish (a) Solution preparation and loading, (b) Phantom assembling and grouping, (c) Phantom shooting and image acquisition and (d) Features extraction in turn.

Imaging data

The phantom was located in the center of a 17 × 12 cm FOV, which was larger than the phantom size. The images were captured at an interval of 10 min to minimize the influence of the overheating tube on the results. The cylindrical container was kept stationary during the whole image acquisition process, only the centrifugal tubes with varied concentrations of solution were replaced. The CBCT unit was 3D Accuitomo (J. Morita, Kyoto, Japan) with exposure settings of 85 kV, 5.0 mA, 17.5 s, and a voxel size of 0.25 mm. The data were converted to DICOM format for the follow-up analysis.

It needs to be emphasized that the tubes with each concentration of K₂HPO₄ solution were captured twice, and the obtained images were marked as test sets and retest sets. The reproducibility experiment compared the data from test and retest sets, while the location-stability experiment only analyzed the data from the test sets.

Grouping and acquisition of regions of interest (ROI)

We classified the 17 centrifugal tubes into four groups based on their spatial distribution. They were placed to form two concentric circles, with the tube in the center labeled Group 1; the four tubes, on the smaller circle and equidistant from each other, were Group 2; among the 12 tubes on the larger circle, the four tubes closest to Group 2 were classified as Group 3; the remaining eight tubes were labeled as Group 4. (Figure 1b) In order to facilitate statistical analysis, 32 ROIs were generated in each group by combining manual labeling with Monte Carlo random sampling, i.e., a single centrifugal tube in Groups 1, 2, 3, and 4 generated 32, 8, 8, and 4 ROIs, respectively, and 128 ROIs were finally obtained from the four groups. (See appendix A ROI for detailed steps)

Radiomic features extraction

We used the open-source software LIFEx ¹⁷ (version 7.3.0) to extract radiomic features, before which three sampling parameters (spatial resampling, intensity discretization, and intensity rescaling) needed to be set manually. For spatial resampling size, we chose the default value; while we adopted a more cautious attitude toward intensity discretization and intensity rescaling because some published data proved that the “size of bins” ¹⁸ and the “absolute or relative rescaling methods” ¹⁹ would affect the properties of the final feature values. After the preliminary discretization experiments (as depicted in appendix B Discretization experiment), we finally set the input parameters as shown in Table 1.

Table 1.

Setting of compulsory input parameters for each concentration set

Concentration (mg/mL)	Spatial Resampling (mm3)	Intensity Discretization		Intensity Rescaling bounds (absolute)
		Nb of grey levels	Size of bins a
50	0.25 × 0.25×0.25	1000	1.001001	500–1500
100				600–1600
200				800–1800
400				1100–2100
600				1300–2300
800				1500–2500

^a “Size of bins” is automatically generated by the software after setting the “Nb of grey levels” and “Intensity rescaling bounds”.

Sixty-nine features including 17 first-order and 52 second-order features were analyzed in this study. See detail in Table 2.

Table 2.

Features included in the study

First-order features	Second-order features
Intensity-Histogram	GLCM a	GLRLM b	NGTDM c	GLSZM d
Mean	Joint Maximum	Short Runs Emphasis	Coarseness	Small Zone Emphasis
Skewness	Joint Average	Long Runs Emphasis	Contrast	Large Zone Emphasis
Kurtosis	Joint Variance	Low Grey Level Run Emphasis	Busyness	Low Grey Level Zone Emphasis
Median	Joint Entropy Log2	High Grey Level Run Emphasis	Complexity	High Grey Level Zone Emphasis
Standard Deviation	Joint Entropy Log10	Short Run Low Grey Level Emphasis	Strength	Small Zone Low Grey Level Emphasis
Interquartile Range	Difference Average	Short Run High Grey Level Emphasis		Small Zone High Grey Level Emphasis
Mean Absolute Deviation	Difference Variance	Long Run Low Grey Level Emphasis		Large Zone Low Grey Level Emphasis
Robust Mean Absolute Deviation	Difference Entropy	Long Run High Grey Level Emphasis		Large Zone High Grey Level Emphasis
Median Absolute Deviation	Sum Average	Grey Level Non-Uniformity		Grey Level Non-Uniformity
Coefficient Of Variation	Angular Second Moment	Run Length Non-Uniformity		Normalised Grey Level Non-Uniformity
Quartile Coefficient Of Dispersion	Contrast	Run Percentage		Zone Size Non-Uniformity
Entropy Log10	Dissimilarity			Normalized Zone Size Non-Uniformity
Entropy Log2	Inverse Difference			Zone Percentage
Maximum Histogram Gradient	Normalised Inverse Difference			Grey Level Variance
Maximum Histogram Gradient Grey Level	Inverse Difference Moment			Zone Size Variance
Minimum Histogram Gradient	Normalised Inverse Difference Moment			Zone Size Entropy
Minimum Histogram Gradient Grey Level	Correlation
	Autocorrelation
	Cluster Shade
	Cluster Prominence

^agray level co-occurrence matrix

^bgray level run length matrix

^cneighborhood grey tone difference matrix

^dgray level size zone matrix

Statistical analysis

The concordance correlation coefficient (CCC) was used to evaluate the reproducibility of these features between the test and retest sets. A total of 24 series of comparative data (four location groups × six concentrations) were analyzed. With values ranging from −1 to 1, the CCC indicated results ranging from completely inconsistent to completely consistent. ²⁰ We took 0.85 as the cut-off point, which means the consistency was acceptable for a feature with CCC value greater than 0.85. These features with acceptable consistency in any series were subsequently screened as qualified features with good reproducibility.

There is no consensus selection of CCC cut-off point in previous studies about radiomics reproducibility. ^21,22 However, the choice of cut-off point inevitably impacts the number of qualified features, consequently affecting the overall evaluation of reproducibility for radiomic features in each series. Therefore, in order to mitigate the influence of cut-off point selection, we introduced a mean value named CCC*. The CCC* values were calculated as the arithmetic average based on all CCC values of the qualified features which were pre-identified.

For location-stability, the Kruskal-Wallis test was used to determine if there were significant differences among four groups of data in test sets of six concentrations. If the <i>p-value ≥ 0.05, which means the difference was insignificant, the tested feature was considered as a qualified feature with location-stability.

Statistical analysis was carried out in MATLAB (version R2021a) (see appendix C Code for specific information).

Results

Reproducibility

Figure 2 shows the basic distribution of 69 target features’ CCC values in 24 series. A sum of 20 qualified features were selected throughout the screening process, and their names are marked red in Figure 2. It can be seen that no features in NGTDM were superior to cut-off value.

Figure 2.

69 features’ CCC values distribution is shown in this heatmap, with 20 features reproducible in at least one series. The naming rule for the y axis is ”concentration.group”. Abbreviations are explained as follows: GLCM: grey level co-occurrence matrix; GLRLM: grey level run length matrix; NGTDM: neighborhood grey tone difference matrix; GLSZM: grey level size zone matrix.

As shown in Table 3, the numbers of qualified features in Groups 3 and 4 were larger than those in Groups 1 and 2 for each concentration. This indicated the features’ reproducibility of the peripheral area was better than that of the center area. In terms of various concentration series, the numbers of qualified features in the high-concentration sets (400, 600, and 800 mg ml⁻¹) were larger than those in the low-concentration sets (50, 100, and 200 mg ml⁻¹). These concentration-related changes in the number of qualified features were more noticeable in center-located Groups 1 and 2.

Table 3.

Number of qualified features in each series and concentration set

Concentration (mg/ML)	Reproducibility				Location-stability
	Group
	1	2	3	4
50	0	1	13	15	6
100	0	1	14	15	11
200	3	1	15	16	7
400	7	12	15	15	6
600	7	14	15	16	2
800	12	4	18	16	1

Similar to Table 3, in each concentration, the CCC* values in the peripheral area (Groups 3 and 4) were larger than those in the center area (Group 1 and 2). Meanwhile, the phenomenon that the values of CCC* in high-concentration sets were larger than those in low-concentration sets was also more significant in the center area (Groups 1 and 2). (Figure 3)

Figure 3.

The distribution of CCC* among each group in six concentration sets.

Location-stability

After the Kruskal-Wallis test among four groups in each concentration test set, 23 qualified features with location-stability were selected. Their names and p-values in six concentration sets are shown in Table 4.

Table 4.

P value of qualified features in the Kruskal-Wallis test

Potentially Qualified Feature		Concentration(mg/mL)
		50	100	200	400	600	800
Intensity- Histogram	Skewness	0.069	<0.001	<0.001	<0.001	0.010	0.001
	Kurtosis	0.117	0.040	0.023	<0.001	<0.001	0.169
	Quartile Coefficient Of Dispersion	0.074	<0.001	0.008	<0.001	<0.001	<0.001
	Maximum Histogram Gradient	0.017	<0.001	0.009	0.157	<0.001	0.002
	Minimum Histogram Gradient	0.973	0.193	0.150	0.126	0.003	<0.001
GLCM	Joint Maximum	0.004	<0.001	0.002	<0.001	0.294	<0.001
	Normalized Inverse Difference	0.002	0.020	0.002	0.158	<0.001	<0.001
	Inverse Difference Moment	0.045	0.083	<0.001	<0.001	<0.001	<0.001
	Normalized Inverse Difference Moment	0.002	0.009	0.001	0.138	<0.001	<0.001
	Cluster Shade	0.133	<0.001	<0.001	<0.001	0.079	<0.001
GLRLM	Short Runs Emphasis	0.024	0.463	0.114	0.004	<0.001	<0.001
	Long Runs Emphasis	0.002	0.078	0.076	<0.001	<0.001	<0.001
	Grey Level Non-Uniformity	<0.001	<0.001	0.113	<0.001	<0.001	<0.001
	Run Percentage	0.010	0.225	0.074	0.002	<0.001	<0.001
NGTDM	Coarseness	<0.001	0.100	<0.001	0.623	<0.001	<0.001
	Busyness	<0.001	<0.001	0.084	0.049	<0.001	<0.001
GLSZM	Small Zone Emphasis	<0.001	0.067	0.001	<0.001	<0.001	<0.001
	Large Zone Emphasis	0.008	0.176	<0.001	0.006	<0.001	<0.001
	Grey Level Non-Uniformity	<0.001	<0.001	0.053	<0.001	<0.001	<0.001
	Normalized Zone Size Non-Uniformity	<0.001	0.104	0.002	<0.001	<0.001	<0.001
	Zone Percentage	0.207	0.143	0.001	0.002	<0.001	<0.001
	Zone Size Variance	<0.001	0.135	0.001	0.001	0.009	<0.001
	Zone Size Entropy	<0.001	<0.001	<0.001	0.060	<0.001	<0.001

There are significant differences in the number of qualified features for different concentrations. (Table 3) With the increase in concentration, the number of qualified features first increased and then decreased, and the numbers of qualified features in high-concentration sets were obviously less than those in low-concentration sets.

It was surprising that there was no consensus feature that was deemed qualified in both the reproducibility and location-stability experiments.

Discussion

High-throughput extraction of tissue and lesion features, quantitative analysis of data for diagnosis, ^23,24 monitoring disease progression, ^25,26 and predicting clinical endpoints ^27,28 are some applications of radiomics. As the basis of radiomic models, the characteristics of the features can eventually affect the reliability of the model output. In fact, after realizing the importance of feature screening, related research has been widely carried out on major imaging devices, including CT, ^29–31 PET, ³² MRI, ^33–36 and ultrasound. ^37,38 The increased and unevenly distributed confounding factors in CBCT images raised concerns about the reliability of extracted features. ^11–15 In this study, different concentrations of K₂HPO₄ aqueous solution were used to evaluate the reproducibility and location-stability of radiomic features.

In the past, a great deal of effort was made to evaluate feature reproducibility. Several studies indicated that the proportion of reproducible features may be related to equipment, ^39,40 acquisition parameters, ^39–43 image reconstruction, ^44,45 target region ²¹ etc. In our reproducibility experiment, we adopted the same parameter settings for test and retest sets and creatively incorporated concentration and spatial location into the study design.

To achieve this goal, we took the K₂HPO₄ aqueous solutions as the target object and manually set their spatial location in the FOV. Due to the similarity of K₂HPO₄’s effective atomic number to hydroxyapatite, it is frequently employed as a core material in liquid phantoms for simulating X-ray attenuation in bone. Unlike solid phantoms commonly used in CT radiomic studies, ^39,42 liquid phantoms can simulate X-ray attenuation at different densities by adjusting solution concentrations, allowing for experimental observations at various densities of the same object. As mentioned in Matheus Oliveira’s research, ¹⁴ K₂HPO₄ in the range from 50 to 800 mg ml⁻¹ of concentration can be equivalent to the attenuation of trabecular bone with varying degrees of mineralization.

The results showed some radiomic features that were not reproducible even with the same parameter settings, which is similar to some previous reproducibility studies. ^22,46–49 Moreover, the reproducibility of CBCT-derived features was density- and location-related. Specifically, the target object with a higher density or located in the peripheral area of the FOV had a higher number of reproducible features. Confounding imaging factors are partly responsible for this phenomenon due to their various spatial distribution and density dependence. Confounding factors, such as scatter radiation, might be higher in the center area of the FOV, as demonstrated in AK Hunter’s study. ⁵⁰ As for density, based on our results, it is reasonable to hypothesize that the influence of confounding factors in a low-density target might be more obvious than that in a high-density target.

This is probably the first study to assess the location-stability of radiomic features in the same FOV. Previous investigations were mainly focused on the location stability of grey values. Matheus Oliveira et al ¹⁴ demonstrated that the absolute CT numbers vary significantly with anatomical locations, using K₂HPO₄ solutions and a human skull phantom. Azin Parsa’s study showed similar results using a dry human mandible. ¹⁵ The shortcomings of CBCT grey value have been widely acknowledged. It is reasonable to doubt the stability of extracted radiomic features, especially the first-order features which directly reflect the grey value distribution. The results of our experiment confirmed our concern. Most of the features did not have location-stability in any concentration set, and none of the features had location-stability in all concentration sets. There were more location-stable features in low concentration sets. The confounding factors, such as beam hardening, ^51,52 which changes with the density of the target, might be responsible for this phenomenon. The non-monotonic variation in the number of qualified features (first increased and then decreased) may be attributed to the joint action of multiple confounding factors.

The current research has following limitations. First, all the experiments were conducted in a single CBCT equipment, which may limit the generalization of the findings. In addition, we used K₂HPO₄ solution as the reference, ignoring the effect of microstructure such as trabecular bone on ray scattering, which may make our results more optimistic. Finally, the experimental setting of the phantom smaller than FOV completely avoided the influence of exomass ^16,53 and may increase the number of qualified features.

The reliability of radiomic features extracted from CBCT images was not considered in previously related published data which applied this methodology. Our study demonstrated that the density and location of the target object can directly affect the reliability of the features. Meanwhile, the limitations of CBCT-derived features, such as lack of reproducibility or location-stability, or not having both at the same time, also greatly hampered their application scenarios. According to our study results, it is extremely recommended to clarify the priority of reproducibility or location-stability at the beginning in a CBCT-based radiomic research, so as to select appropriate radiomic features accordingly.

The study on radiomic features derived from CBCT is far from enough; therefore, more research needs to be carried out in a simulated clinical environment. In recent years, many methods have emerged to improve the quality of CBCT images, ^54–56 which may be a fundamental way to solve the problem of the features’ reliability and is also one of the directions of future research.

Conclusions

The reproducibility and location-stability of radiomic features derived from CBCT were assessed by a multiconcentration phantom. The density and location of the target object affected the reproducibility of features. Few features met the standard of location-stability, and the location-stability was affected by the density of the target object. When using CBCT as the research platform for radiomics, we should be cautious about the reliability of radiomic features.