Benchmarking Various Radiomic Toolkit Features While Applying the Image Biomarker Standardization Initiative toward Clinical Translation of Radiomic Analysis

Abstract

The image biomarkers standardization initiative (IBSI) was formed to address the standardization of extraction of quantifiable imaging metrics. Despite its effort, there remains a lack of consensus or established guidelines regarding radiomic feature terminology, the underlying mathematics and their implementation across various software programs. This creates a scenario where features extracted using different toolboxes cannot be used to build or validate the same model leading to a non-generalization of radiomic results. In this study, IBSI-established phantom and benchmark values were used to compare the variation of the radiomic features while using 6 publicly available software programs and 1 in-house radiomics pipeline. All IBSI-standardized features (11 classes, 173 in total) were extracted. The relative differences between the extracted feature values from the different software programs and the IBSI benchmark values were calculated to measure the inter-software agreement. To better understand the variations, features are further grouped into 3 categories according to their properties: 1) morphology, 2) statistic/histogram and 3)texture features. While a good agreement was observed for a majority of radiomics features across the various tested programs, relatively poor agreement was observed for morphology features. Significant differences were also found in programs that use different gray-level discretization approaches. Since these software programs do not include all IBSI features, the level of quantitative assessment for each category was analyzed using Venn and UpSet diagrams and quantified using two ad hoc metrics. Morphology features earned lowest scores for both metrics, indicating that morphological features are not consistently evaluated among software programs. We conclude that radiomic features calculated using different software programs may not be interchangeable. Further studies are needed to standardize the workflow of radiomic feature extraction.

Supplementary Information

The online version contains supplementary material available at 10.1007/s10278-021-00506-6.

Introduction

Radiomics has recently become an important research topic as a method of extracting additional latent information from clinical imaging that may help solve a variety of perplexing clinical scenarios. With the advances in image processing and data science, medical images (computed tomography, magnetic resonance, positron emission tomography images, etc.) are converted to minable high-dimensional data, which may carry latent information that can facilitate clinical decision-making. In general, a radiomics pipeline consists of 4 steps: image acquisition, lesion segmentation, radiomics feature extraction and statistical/machine learning-based analysis [1]. A number of studies have shown that any of these 4 steps can cause variability of radiomics analysis [2–9]. For example, currently, different imaging centers use varying image acquisition protocols. The lack of consensus or guidelines on image acquisition across institutions leads to imaging data heterogeneity among different studies [10]. The accuracy of manual segmentation also affects the robustness and stability of radiomics models; however, it is a viable solution in scenarios of contouring highly varying complex tumor boundaries with low tumor–tissue contrast using automated methods [11].

While researchers have recognized the relative lack of consistency and reproducibility, there are a dearth of approaches to address this issue. Many research institutions have either developed their in-house or use open-source programs for radiomic analysis. These programs usually run different implementations of the same features but with different terminologies. In some examples, the same metrics may not generate identical values due to differences in implementing the underlying algorithms, e.g., image intensity normalization based on just the tumor region or on entire image may generate different results.

The IBSI was founded to address the standardization of radiomics features [12]. The IBSI proposed the definitions of 11 commonly used feature classes. A digital phantom and a CT phantom, with their benchmark values of features, have been created for the purpose of standardization.

Recent analyses have been conducted on datasets for evaluating the variation of features among radiomics software programs, showing that large variations were found due to image preprocessing, parameters and algorithm implementations [3–5]. As these studies focused on specific private datasets and limited number of features, the results are not generalizable. In this study, the issue of generalization was addressed by using the IBSI benchmarking to generate reliable results.

Materials and Methods

Digital Phantom

The digital phantom developed by the IBSI is geometrically small, consisting of 5 $\times$ 4 $\times$ 4 voxels. Fig. 1 shows four slices of the phantom. Gray levels are designed as integers and hence, discretization is not required before computing features. Gray levels range from 1 to 9 in the whole volume and from 1 to 6 in the region of interest (ROI). Level 2 and 5 are absent. Voxels are isotropic, with 2.0 $\times$ 2.0 $\times$ 2.0 mm dimensions.

Fig. 1.

IBSI digital phantom. The gray levels are annotated in each voxel. Yellow region indicates the ROI

IBSI-Standardized Features

To comprehensively compare inter-software differences, 11 classes of radiomics features were evaluated in this study: morphology, local intensity, intensity-based statistic, intensity histogram, intensity-volume histogram, gray-level co-occurrence matrix (GLCM), gray-level run length matrix (GLRLM), gray-level size zone matrix (GLSZM), gray-level distance zone matrix (GLDZM), neighborhood gray tone difference matrix (NGTDM) and neighboring gray-level dependence matrix (NGLDM) based features. These features are standardized and summarized from various technical references in the IBSI. According to their mathematical definitions, the 11 classes of radiomics features characterize various properties of the ROI, and they can be grouped into 3 major categories: 1) statistical/histogram, 2) texture and 3) morphology features (Table 1).

Table 1.

Three major categories of 11 classes of IBSI-standardized features. Morphology features are in a category by themselves

Statistical/histogram features	Texture features	Morphology (shape) features
Local intensity	GLCM	Morphology features
Intensity-based statistic	GLRLM
Intensity histogram	GLSZM
Intensity-volume histogram	GLDZM
	NGTDM
	NGLDM

Table 2 details the number of features within the IBSI standards that fall within each of the aforementioned categories. While the IBSI has defined specific taxonomy and terminology, similar features may be named differently in different software programs. As an example, GLCM joint maximum is tagged as maximum probability in some programs, e.g., Pyradiomics [13], QIFE [14] or studies [15–17]. In some software programs [13] , kurtosis is computed with the formula of excess kurtosis, which is an alternative definition of kurtosis. While excess kurtosis is equal to kurtosis – 3, to perform comparisons to IBSI standards the values of kurtosis must be corrected manually after processing through these software programs. To conduct a fair comparison of metrics across the different software programs, we created a glossary of matching radiomic metrics, with their different terminologies, across different radiomics software (Supplementary Table 1).

Table 2.

Radiomics features

	IBSI-standardized	A1	Pyradiomics	MITK	LIFEx	SERA	CaPTk	A2
Morphological features	29	1	14	20	2	25	3	27
Local intensity features	2	0	0	2	0	2	0	1
Intensity-based statistical features	18	6	15	18	5	18	15	18
Intensity histogram features	23	0	2	21	2	19	17	23
Intensity-volume histogram features	7	0	0	7	0	7	0	7
GLCM	25	15	23	25	6	25	6	25
GLRLM	16	11	16	16	11	16	4	16
GLSZM	16	16	16	16	11	16	16	16
GLDZM	16	0	0	0	0	16	0	16
NGTDM	5	5	5	5	3	5	5	5
NGLDM	16	16	13	16	0	16	16	16

Radiomics Software Programs

Six publicly available software programs: Pyradiomics [13, 18, 19], the Medical Imaging Interaction Toolkit (MITK) [20–22], LIFEx [23–25], the Standardized Environment for Radiomics Analysis (SERA) [26–28], Cancer Imaging Phenomics Toolkit (CaPTk) [29–31], a MATLAB library from McGill University (A2, GitHub Repo: radiomics-develop) [12, 32, 33] and an in-house MATLAB library (A1, our affiliation [ref1][ref2][ref3]), are evaluated here (Table 3). These programs were included in our study as they are widely used in radiomics research and span across different software development languages and operating systems.

Table 3.

Radiomics software programs

	Language	Supported Operating System	Significant Dependent Library
A1	MATLAB	Windows, Linux, MacOS	None
Pyradiomics	Python	Windows, Linux, MacOS	SimpleITK
MITK	C++	Windows, Linux, MacOS	the Insight Toolkit (ITK), the Visualization Toolkit (VTK)
LIFEx	Java	Windows, Linux, MacOS	None
SERA	MATLAB	Windows, Linux, MacOS	None
CaPTk	C++	Windows, Linux, MacOS	OpenCV, ITK, VTK
A2	MATLAB	Windows, Linux, MacOS	None

While the underlying algorithm remains the same, its application differs across different software (Table 4). For example, for morphology features, researchers commonly use Marching Cubes algorithms for meshing presentation, as it works efficiently on multiple programming platforms [34]. However, more efficient implementations were proposed by other researchers [35]. In fact, the IBSI also recommended the methodology proposed by Lewiner [36], optimizing the performance and feasibility. Intensity-based statistics and histogram features are already well defined in common image processing field and therefore, their computing results should rely on relatively established programming strategies. For example, for textural features, all the evaluated software programs cite the same publications, where GLCM was proposed by Haralick [37], GLRLM was proposed by Galloway [38], GLSZM was proposed by Thibault [39, 40], GLDZM was derived from Thibault [40], NGTDM was proposed by Amadasun [41] and NGLDM was proposed by Sun [42].

Table 4.

Algorithm references across different software

	A1	Pyradiomics	MITK	LIFEx	SERA	CaPTk	A2	IBSI recommendation
Morphological features	-	Lorensen [34]	-	-	IBSI [12]	Vallières [32]	IBSI [12]	Lorensen [34]
					Varnäs [52]	Thibault [53]		Lewiner [36]
Local intensity features	-	-	-	-	-	-	IBSI [12]	Wahl [54]
								Frings [55]
Intensity-based statistical features	-	-	-	-	IBSI [12]	-	IBSI [12]	-
Intensity histogram features	-	-	-	-	-	Macyszyn [56]	IBSI [12]	-
Intensity-volume histogram features	-	-	-	-	IBSI [12]	-	IBSI [12]	El Naqa [57]
GLCM	Haralick [37]	Haralick [37]	-	Haralick [37]	Haralick [37]	Haralick [37]	IBSI [12]	Haralick [37]
					IBSI [12]			Unser [58]
GLRLM	Galloway [38]	Galloway [38]	-	Xu [59]	Galloway [38]	Galloway [38]	IBSI [12]	Galloway [38]
		Chu [60]
		Xu [59]
		Tang [61]
		Tustison [62]
GLSZM	Thibault [39]	Thibault [39]	-	Thibault [39]	IBSI [12]	Galloway [38]	IBSI [12]	Thibault [40]
						Tang [61]
GLDZM	-	-	-	-	IBSI [12]	-	IBSI [12]	Thibault [40]
NGTDM	Amadasun [41]	Amadasun [41]	-	Amadasun [41]	IBSI [12]	Amadasun [41]	IBSI [12]	Amadasun [41]
NGLDM	Sun [42]	Sun [42]	-	-	IBSI [12]	-	IBSI [12]	Sun [42]

The level of quantitative assessment provided by different radiomic software is different, i.e., not all software programs includes all 173 IBSI-standardized features. Table 2 lists the number of features computed by each software. To better understand the distribution of the 173 different IBSI-defined features across the different radiomics software, an UpSet diagram and Venn diagram (Fig. 2) approach was used. Using this approach, the “popularity” of the 3 major categories of features were analyzed, i.e., how many features were shared by different radiomics software programs was identified. Only the 6 publicly available software programs were considered in the diagrams for the reason that they may be designed for general radiomics analysis, while in-house software may be designed for application-specific scenarios. As the total number of features of each category are different across and within software, two ad hoc metrics to quantify the “popularity” of the each of the 3-feature category, based on the 6 software were defined:

$$\begin{array}{cc}\hfill Popularity1:P_1=& \frac{{\sum }_{i=1}^d{w}_i}{6d},0\le P_1\le 1\hfill \end{array}$$ 1

$$\begin{array}{cc}\hfill Popularity2:P_2=& \frac{{\sum }_{i=1}^d\mathbb{1}(w_i>4)}{d},0\le P_2\le 1\hfill \end{array}$$ 2

where $w_i$ indicates the number of software that support the $i^{\mathit{th}}$ features, d indicates the number of features. Features supported in more than 4 out of 6 software programs are considered as “popular”. These are highlighted in the UpSet diagram in Fig. 2. Intuitively, $P_1$ quantifies the weighted cardinality of the specific group of radiomics features; $P_2$ quantifies the percentage of “popular” features in the specific group. Large values in both metrics indicate that radiomics features are widely supported by various software (Fig. 3).

Fig. 2.

Venn and UpSet diagram for radiomics features across 6 publicly available programs. (a) morphology features. (b) statistics and histogram features (c) texture features, (d) all IBSI features

Fig. 3.

Popularity 1 and Popularity 2 within 3 categories and within all 173 IBSI features

Computational Workflow

In general, the computational workflow can be divided into two stages: 1) preprocessing and 2) feature computation (Fig. 4a). Two preprocessing operations are included in most studies and introduced in the IBSI: voxel interpolation and intensity discretization.

Fig. 4.

(a) General computation workflow. Preprocessing is not required but irremovable in CaPTk and LIFEx, thus enclosed by dash lines. (b) Different workflow for morphology features and non-morphology features

As texture features are sensitive to voxel size variation, interpolation is required. Different algorithms for interpolation, such as nearest neighbor, bilinear and B-Spline interpolation, are used. Intensity discretization is useful for producing a reasonable quantitative analysis and removing noise. Two approaches are defined in the IBSI: 1) fixed bin number and fixed bin width [12]:

$$\begin{array}{cc}\hfill X_{d,k}=& \left\{\begin{array}{cc}\lfloor N_g\frac{X_{gl,k}-X_{gl,min}}{X_{gl,max}-X_{gl,min}}\rfloor +1\hfill & X_{gl,k}<X_{gl,max}\hfill \\ Ng\hfill & X_{gl,k}=X_{gl,max}\hfill \end{array}\right)\hfill \end{array}$$ 3

$$\begin{array}{cc}\hfill X_{d,k}=& \lfloor \frac{X_{gl,k}-X_{gl,min}}{w_b}\rfloor +1\hfill \end{array}$$ 4

where $X_{d,k}$ denotes the discrete intensity for voxel k, $N_g$ denotes the number of bins, $X_{gl,k}$ denotes the original gray level of voxel k, $X_{gl,min}$ and $X_{gl,max}$ denotes the minimum and maximum gray level in the ROI, and $w_b$ denotes the interval width of each bin. Number of bins $N_g$ and width of bins $w_b$ need to be specify for fixed bin number and fixed bin width, respectively.

As the digital phantom was developed using discretized intensities and isotropic voxels, no preprocessing is needed. While most of the radiomics software programs allow modular computing, LIFEx and CaPTk are rigid and preprocessing cannot be skipped. To address this problem, specific parameters of voxel interpolation and intensity discretization are assigned to remove their respective contribution. Expected voxel size is needed to specify for interpolation. Voxel dimensions 2.0 $\times$ 2.0 $\times$ 2.0 mm, which is identical to the original spacing, is set for voxel interpolation because we expect the spacing not to change. For intensity discretization, LIFEx uses fixed bin width approach with $w_b$ equal to 1 and CaPTk use fixed bin number approach with $N_g$ equal to 6. Similar to the setting for interpolation, we expect both of these settings for discretization do not change voxel intensities. In this study, five basic statistic features are chosen to compare mean, median, minimum, maximum discretized intensity and discretized intensity range across the different software, after intensity discretization to reveal basic properties of intensity distribution which has been used to evaluate the different software.

Feature computation is the next step after either voxel interpolation or intensity discretization (Fig. 4a). For non-morphology features, the feature values are directly calculated from the 3D volume data. Morphology features, however, by definition, are defined under polygon meshing representation, leading to a two-stage computation workflow (Fig. 4b). Image volume data are first transformed to meshing representation, then morphology features are calculated based on it.

Table 5 shows the parameters in each step of computation. To minimize the effect of different parameters and generalize the results, parameter settings are designed to maintain the highest consistency. For GLCM, GLRLM, GLSZM, GLDZM, NGTDM and NGLDM features, voxel neighborhoods are measured along all possible directions/angles in 3-dimensional (3D) space, which mean that a central voxel has 26 neighbors. The distance between the central voxel and the neighbor voxel is set as 1 by default. As each direction can generate one GLCM and GLRLM by definition, feature aggregation should be used to summarize feature values from all directions. In general, the information of features from multiple directions are summarized in two different ways. LIFEx averages the features over 26 directions, while the other software programs compute the features after merging matrices from all directions.

Table 5.

Computation parameters

		A1	Pyradiomics	MITK	LIFEx	SERA	CaPTk	A2
Interpolation		-	-	-	2.0 $\times$ 2.0 $\times$ 2.0 mm	-	2.0 $\times$ 2.0 $\times$ 2.0 mm	-
Discretization		-	-	-	Fix bin size of 1.0	-	Fix bin number of 6	-
GLCM	Neighborhood Dimension	3D	3D	3D	3D	3D	3D	3D
	Distance	1	1	1	1	1	1	1
	Aggregation Method	Merging	Merging	Merging	Averaing	Merging	Merging	Merging
GLRLM	Neighborhood Dimension	3D	3D	3D	3D	3D	3D	3D
	Distance	1	1	1	1	1	1	1
	Aggregation Method	Merging	Merging	Merging	Averaing	Merging	Merging	Merging
GLSZM	Neighborhood Dimension	3D	3D	3D	3D	3D	3D	3D
	Distance	1	1	1	1	1	1	1
GLDZM	Neighborhood Dimension	-	-	-	-	3D	-	3D
	Distance	-	-	-	-	1	-	1
NGTDM	Neighborhood Dimension	3D	3D	3D	3D	3D	3D	3D
	Distance	1	1	1	1	1	1	1
NGLDM	Neighborhood Dimension	3D	3D	3D	-	3D	3D	3D
	Distance	1	1	1	-	1	1	1
	Coarseness	0	0	0	-	0	0	0

Results

As the voxel interpolation and intensity discretization are fixed in CaPTk and LIFEx, we compare the agreement of these 5 features first to confirm that preprocessing does not change the voxel intensities (Table 6). Values of mean, median, minimum and maximum intensity calculated by CaPTk are 1 unit less than the benchmark values, while the intensity range is equal. This indicates that CaPTk set the minimum gray level as 0 instead of 1 and all voxel intensities are shifted by -1. LIFEx does not include the 5 features, hence effect of preprocessing is unclear.

Table 6.

Basic features

	A1	Pyradiomics	MITK	LIFEx	SERA	CaPTk	A2	IBSI Benchmark Value
Mean discretized intensity	-	-	2.1643	-	2.1486	1.1486	2.1486	2.15
Median discretized intensity	-	-	1.0156	-	1	0	1	1
Minimum discretized intensity	-	-	1	-	1	0	1	1
Maximum discretized intensity	-	-	6	-	6	5	6	6
Discretized intensity range	-	-	5	-	5	5	5	5

Software agreement for each single feature was measured by relative difference between the calculated feature values and the benchmark values provided by the IBSI (Eq. 5). Significant differences in overall features are noted from CaPTk analysis (Fig. 5), possibly from the effect of intensity shifting in the step of preprocessing. Good agreement was found between the other 6 software programs.

$$\begin{array}{c}\hfill relativedifference=\frac{\mid featurevalue-benchmarkvalue\mid }{benchmarkvalue}\end{array}$$ 5

Fig. 5.

Heatmap for the relative differences between computed features and the IBSI benchmark values. Missing features are denoted in gray color. Lower values are better

To study the effect of intensity shifting, we modified the code for intensity shifting in our in house pipeline and produced the results of GLCM as an example. Results based on the intensity shifting in our in house pipeline and producing the results of GLCM as an example revealed that some features (e.g., joint average, joint entropy, sum average and inverse difference moment) have significant deviation after adding the intensity shifting, while others (e.g., contrast, sum entropy) are unchanged (Fig. 6).

Fig. 6.

GLCM features calculated by A1 with two different settings of minimum intensities

A scatter plot was used to analyze the agreement (measured by relative difference) among different feature classes (Fig. 7). Since CaPTk preprocessed the intensity by shifting -1 unit, which is different to all other software, we considered all the other software programs in Fig. 7. Good agreement was also observed among non-morphology features, while poor agreement is observed for morphology features

Fig. 7.

Scatter plot for the relative differences, concluding the rest 6 software programs (A1, Pyradiomics, MITK, LIFEX, SERA, A2). Larger size of points indicates that more points are overlapped at the underlying location (e.g., a point with Size 50% means that 50% of points within that feature class are equal). Large points around the location of 0 indicate good agreement across radiomics software programs

As morphology features are computed under the meshing representation, the computed meshes may also affect feature values. Among 7 software programs, we extracted the computed meshes from SERA and A2 at their intermediate steps (Fig. 8). Other software programs are not able to provide the meshes data as output, either because they are programmed as “end-to-end” pipelines or are not open-source. The meshing of SERA slightly differs from the one of A2. Values of volumes of them are also different (556 for SERA, 554 for A2).

Fig. 8.

(a) Meshing representations of the digital phantom generated by SERA. Value of volume is 556. (b) Meshing representations of the digital phantom generated by A2. Value of volume is 554

Discussion

This study has demonstrated that the design and implementation of radiomics feature calculation vary across multiple software programs, using a single IBSI standard phantom. In other prior studies [3–5], the comparison and analyses of radiomics reproducibility, software agreement were reported using statistical hypothesis testing [3, 5], spearman correlation coefficients [4] or intraclass-correlation coefficients [5] (ICC), using a set of features from datasets. In addition, this study includes more comprehensive amount of ISBI standardized features, while early studies analyze only those which are common across all software programs utilized in their studies.

Different strategies of intensity discretization have an influence on the resulted feature values. As shown in Fig. 6, two strategies (minimum intensity as 0 vs. minimum intensity as 1) produce different feature values. The reason for different values is that 0 is introduced in the computation; thus, partial terms will be lost. Given the two different strategies, as an example, GLCM joint entropy will have two different mathematical definitions:

$$\begin{array}{c}\hfill -\sum _{i=0}^{N_g-1}\sum _{j=0}^{N_g-1}P(i,j){log}_{2}P(i,j)\end{array}$$ 6

$$\begin{array}{c}\hfill -\sum _{i=1}^{N_g}\sum _{j=1}^{N_g}P(i,j){log}_{2}P(i,j)\end{array}$$ 7

where P(i, j) denotes the element of GLCM at the location of $i^{\mathit{th}}$ row and $j^{\mathit{th}}$ column. According to the definition of GLCM, both cases will result in identical P(i, j), but the second-order statistical features may be affected because zeros are introduced in the calculation. Partial terms of co-occurrence will be lost in the multiplication operation with 0, causing relatively smaller feature values. This discretization strategy is not unique to just CaPTk; other studies also use the same mathematical definition, setting the minimum intensity to zero [43–45]. By definition, the two discretization approaches given by the IBSI (Eq. 3 and Eq. 4), add a constant unit of one to the minimum intensity avoid zeros encountered in the quantized image. Discretization should therefore adhere to the IBSI standard not only for reproducibility, but also for retaining the partial terms in the computation of the co-occurrence matrix to fully and precisely characterize the properties of ROI.

Disagreement among feature values also occurred frequently in the group of morphological features. The computation of morphology features is more complex than non-morphology features. Statistical and textural features are defined unequivocally on the 3D volume of ROI; hence the computation is direct using matrix algebra. The computation of morphology features, by contrast, includes two stages. The source of feature value variation may come from the first stage, where volume data is transformed as meshes data. There are various algorithms for meshing approximation, where different algorithms may result in different meshes and therefore produce different feature values. As Fig. 8 shows, SERA and A2 may use different algorithms and give two different approximations and values of volumes. Therefore, careful consideration needs to be given to the choice of morphometric features used particularly in multi-center radiomics analysis.

The “popularity” of each features is important for analysis of software agreement. With more features included in these software programs, higher level of quantitative assessment for them can be provided. Within 173 IBSI-standardized features, Fig. 2 demonstrates that the 6 publicly available software programs share 1/29 morphology feature, 7/50 statistics and histogram features and 21/94 texture features. Furthermore, both ad hoc metrics, $P_1$ and $P_2$ , have lower values for morphology features (Fig. 3). These indicate that relatively less attention is provided to morphology properties of an ROI compared to its non-morphological properties. This may be true considering the computational complexity behind extracting morphology features, i.e., the algorithms and programming of meshing representations are usually challenging, tedious and error prone [34, 46–48]. Taken together, these reasons justify the poor representation and feature inconsistency of morphological metrics within radiomics software.

The primary aim of most radiomics studies is to identify significant features and build discriminant models. Therefore, it is desirable to improve the model performance by including more candidate features in the statistical analysis. With more features, in general, it is possible to find a better subset of radiomics features to improve the performance, achieving higher accuracy, area under the curve to aid discrimination. In addition, morphology features show significant prognostic power in the application of various cancer types [49–51]. Therefore, despite the computational complexity in extracting morphology features which are rich in information, possibly distinct from that provided by non-morphological metrics, they should be included in analysis. The results reveal that current software programs lack standardization. Benchmarking may be important for reproducibility of a radiomics study. The IBSI digital phantom is a good resource for validating programs with the provided benchmark values for each proposed feature. In most studies, images are analyzed with preprocessing (normalization, transformation, discretization). While the phantom is not able to contribute to the benchmarking of preprocessing directly as it does not require additional processing, it may be indirectly helpful by checking the basic features in Table 6, where the difference of discretization approach are found. The intensity-based statistical features are calculated right after interpolation, which can be used for benchmarking of image interpolation; intensity histogram features are calculated right after gray-level discretization, which can be used for benchmarking of both interpolation and discretization (Fig. 4a).

We recommend that future radiomic studies should provide implementation details about the radiomics software such as the name of the software, its version, whether it is open-source or custom-built, the programming language used to develop it, etc., so that appropriate comparisons of the results across different radiomics studies can be performed. In addition, consistent terminology of the features should be used, ideally adhering to the IBSI terminologies. Features not used in the IBSI list should be thoroughly defined and supported by references, so future studies can validate them.

Limitation

While this study evaluated only 6 publicly available and 1 in-house program, adding more software programs may strengthen our findings. For some programs, computation and code version tracking are not available, leading to the difficulty in matching the program available to the program used to publish results. While some programs are source code readable (e.g., MATLAB toolbox), other programs only provide graphical user interface (GUI) for the convenience of non-professional users and codes are not visible. As an example, LIFEx is programmed in Java, and the source and reason of features deviation are not easy to understand. We have tried to explain the rationale behind the differences in metrics that were significantly different among the software compared in this study (e.g., Fig. 8). However, it is beyond the scope of this study to look at the reason for the difference occur for each feature.

Further analysis could focus on the feature comparison on real-world datasets. With a larger dynamic range of gray levels in the real cases, different imaging modalities, the features agreement may be different. Most studies employ image enhancement, augmentation and transformation before feature extraction. These operations may also affect the degree of variation across radiomics programs.

Conclusion

We evaluated feature agreement of 7 different radiomics software programs. While most first-order and second-order texture features show satisfactory agreement, morphological features show significant variation. In addition, most programs are relatively weak on morphology analysis possibly owing to its computation complexity. Further work is necessary to standardize the calculation of radiomics features across different radiomics software, which is one of the first steps towards the clinical translation of radiomic analysis.

Supplementary Information

Below is the link to the electronic supplementary material.

Declarations

Conflict of Interest

On behalf of all authors, the corresponding author states that there is no conflict of interest.