Radiomics: a new application from established techniques

Abstract

The increasing use of biomarkers in cancer have led to the concept of personalized medicine for patients. Personalized medicine provides better diagnosis and treatment options available to clinicians. Radiological imaging techniques provide an opportunity to deliver unique data on different types of tissue. However, obtaining useful information from all radiological data is challenging in the era of “big data”. Recent advances in computational power and the use of genomics have generated a new area of research termed Radiomics. Radiomics is defined as the high throughput extraction of quantitative imaging features or texture (radiomics) from imaging to decode tissue pathology and creating a high dimensional data set for feature extraction. Radiomic features provide information about the gray-scale patterns, inter-pixel relationships. In addition, shape and spectral properties can be extracted within the same regions of interest on radiological images. Moreover, these features can be further used to develop computational models using advanced machine learning algorithms that may serve as a tool for personalized diagnosis and treatment guidance.

Introduction

Radiological imaging techniques are powerful noninvasive tools used for the detection, differentiation, and diagnosis of different tissue characteristics in patients. These imaging methods include X-Ray, Computed Tomography(CT), Magnetic Resonance Imaging (MRI), Nuclear Medicine (NM) Positron Emission Tomography(PET) and Ultrasound(US). Each of these modalities creates different tissue contrast based on whether the tissue is normal or abnormal. These different tissue contrast mechanisms are exploited by the radiologist to identify patterns to reach a diagnosis. However, each of the radiological images contains more information content not visible to the clinician’s eye and this “hidden” information creates a “radiological texture” which can provide much more information about the tissue of interest than previously thought. Thus, Radiomics was introduced as a “new” method to discover and translate the metrics obtained by using texture and other analysis techniques on radiological images. However, Radiomics is a new application using established techniques, specifically, Texture, Entropy, Haralick and other features^1–4. The novel aspect of Radiomics is the comparison of these measures to surrogate endpoints and the advent of increased computational power available today. The main idea behind Radiomics is that information is “hidden” within the radiological images can be extracted using advanced texture and shape analysis. From this extraction of data, a high dimensional space is created. Intuitively, texture is used by everyone to define objects either visually or by touch. The terms commonly used to describe texture are roughness, smoothness, coarseness, etc., but for digital data, analytical methods were needed to better “describe” the information content within digital objects. Therefore, texture analysis was born, in part, is based on information theory developed in 1948 by Claude Shannon and extended by Haralick, Galloway, and others that incorporated different statistical measures to include gray level matrix operations on the input data^1–4.

Historically, one of the first applications of texture analysis was in analyzing aerial photographs⁶ then later applied to medical and other images^2,7. Currently, texture feature analysis is used in many different areas of research, especially, with increased computational power and digital storage capacity^8–11. Moreover, given a region of interest, shape based features can also be extracted along with texture analysis⁵.

Recently with the availability of increased computer technology to reduce the computational complexity into investigations of using texture and shape analysis led to the development of a new area of research termed Radiomics. By definition, Radiomics is the high throughput extraction of quantitative features from radiological images creating a high dimensional data set followed by data mining for potentially improved decision support^12–14, however, Radiomics is based on texture, shape and gray level statistics within images to discern different relationships compared to clinicopathologic data. But, the correlation to true biological meaning is needed. But, through the use of Radiomic features in medical imaging, it may be possible to decode tissue pathology that is not normally visible to the naked eye. How is this possible? Tumors are spatially and temporally heterogeneous and maybe undersampled during medical procedures, for example, during biopsy. Therefore, the whole tumor is not completely characterized and this could be detrimental in determining treatment options. This challenge can be potentially overcome by using a Radiomics approach which is capable of extracting quantitative features over the whole tumor and the results can be compared to the tumors pathological results or other information, such as, survival¹⁴.

This article reviews and outlines historical, mathematical, and more recent applications of radiomics techniques in detecting tumor characteristics and demonstrates that radiomics methods are promising in the near future for clinical applications. Moreover, this review will give a summary of the state of the art knowledge of radiomics and texture analysis in different cancers, and conclude with an outlook on potential future of radiomics in the era of “personalized medicine” cancer therapies.

Radiomics Algorithm

The framework for radiomics use in clinical settings is shown in Figure 1. The first step involves radiological image acquisition of MRI, CT, PET, etc., depending on the imaging modality used for diagnostic and/or treatment planning. Then, the second step involves identifying region of interest in the acquired images. This region of interest can be either lesion tissue or a normal tissue depending on the application. The identification and segmentation of the tissue of interest is accomplished manually by radiological experts or automated segmentation software. The third step involves extraction of radiomic features from the region of interest based on the texture and shape properties. This step produces a large number of radiomic features based on the statistical, filtering and morphological analysis and creates a high dimensional feature space¹⁴. Then, the “highly” informative features are selected based on the user defined criteria. The final step involves classification of the features to defined by the user, for example, distinguishing malignant tumor from benign tumors or survival.

Figure 1.

Illustration of the radiomics algorithm. (a) Initial Computed Tomography (CT) scan. (b) Segmentation is performed on the lesion using a region of interest(ROI). (c) Radiomic features are extracted from the ROI based on the gray level patterns, inter-voxel relationships, and shape. (d) A subset of the extracted radiomic features is selected for classification (e) The selected features are used as inputs into a classification model to produce a diagnosis or correlation to a prognostic marker. Data from^5,12.

Radiomic feature extraction

Radiomic feature extraction methods are based into primarily three categories consisting of statistical, filtering and morphological features. In this paper, we discuss the fundamental feature extraction methods from all the three categories, however, there are some other feature extraction methods such as gray level size zone matrix based features, Minkowski functionals, and others used in radiomics applications that will not been discussed here^15,16.

Statistical texture features

Statistical texture features characterize the stochastic or random properties of the spatial distribution of gray levels within an image using statistical measures, such as, marginal-probabilities (defined below). Currently, there are two levels of statistical methods used for Radiomics, first and higher order methods as shown in figure 2. These statistical methods have been applied to different imaging modalities used in several different diseases and are outlined below.

Figure 2.

Illustration of statistical texture feature extraction. (a) Segmented tumor image (b) Segmented tumor image quantized to four intensity levels (c) First order statistical features corresponding to first order histogram (d) Higher order statistical features corresponding to 1. GLCM (gray level co-occurrence matrix), 2. GLRLM (gray level run length matrix) and 3. NGTDM (neighborhood gray tone difference matrix). Data from¹⁶.

First order texture statistics

First order texture statistics are based on the first order histogram that describes distribution of voxel intensities in an image. For example, a normalized first order histogram (H) is computed by dividing the voxel intensities (I) in an image into B equally spaced bins and computing the proportion of voxels in each bin as defined by the equation.

$$H(i)=\frac{\#of\mathit{pixels}\mathit{with}\mathit{gray}\mathit{levels}in\{I\epsilon B_i\}}{\sum \#\mathit{pixels}in\mathit{the}\mathit{image}}$$ (1)

Entropy and uniformity are the two commonly used features computed using the histogram. Let H be the first order histogram with B bins, then the entropy¹ and the uniformity are given by the following equations:

$$\mathit{Entropy}=-K{\sum }_{i=1}^{B}H(i)logH(i)$$ (2)

where K is a positive constant and is determined by the units of the application.

However, since we are dealing with bits of information, K =1 and the equation used in most digital applications is given as

$$\mathit{Entropy}=-{\sum }_{i=1}^{B}H(i){log}_{2}H(i)$$ (3)

and

$$\mathit{Uniformity}={\sum }_{i=1}^{B}H{(i)}^2$$ (4)

Entropy measures the inherent randomness in the gray level intensities of an image. The maximum value of entropy (Log₂(B)) occurs when all the gray level intensities in an image occur with equal probability¹. On the other hand, uniformity, measures the uniformity of gray level intensities within an image or ROI. For example, the maximum uniformity of one occurs when all the pixels in an image or ROI have the same gray level intensity.

Both the first and second order statistics can give different values based on the number of bins and the number of bins is a critical parameter defined by the user. If the number of bins is selected either too small or very large, then the histogram may not be able to correctly represent the underlying distribution with the image or ROI. Several methods exist for determining the optimal number of bins without making any assumptions regarding the underlying data distribution^17,18. However, if different sized ROIs in a study have a different number of bins in the first order histograms, then becomes difficult to directly compare the results between studies. To potentially overcome this difficulty, another approach would be to use the same number of bins for all ROIs. Unfortunately, this leads to an issue of dependence of first order statistic features on the size of ROI. For example, if we consider two ROIs of sizes 128 and 1024 voxels and bin the gray level intensities within these two ROIs into 1024 bins per R0I (as we are using the same number of bins), then the entropy values of data distributions with maximum randomness (each voxel has a different intensity) would be equal to 7 for 128 voxel-sized and 10 for 1024 voxel-sized ROIs respectively. Optimal binning is a major issue for all statistical approaches, since they depend on the preprocessing step of image quantization. Thus, some form of feature normalization is required with respect to size in cases when the same number of bins is used for all the ROIs and with respect to the of number of bins when different numbers of bins are used for different ROIs. Typically, these binning issues are not addressed in many publications.

A number of first order statistical texture features have been applied to applications across different medical imaging modalities for characterization of several biological tissues of interest^{5,7,16,19–63,64}. Table 1 summarizes the quantitative values of first order entropy for different organs and pathologies as reported in the literature. Depending on the organ being imaged and the imaging modality, the first order histograms may or may not have been the same across all the applications. However, as shown in Table 1, entropy values are higher in malignant than in corresponding benign tissue.

Table 1.

Quantitative values of first order statistical entropy as reported in the literature

Reference	Tissue	Image/Object type	Pathology
				Benign	Malignant
Sinha et al76	Breast	Normalized radial length of tumor		1.46	1.6
Ertas et al35	Breast	nMITR projection		5.4	6.77
Suo et al57	Urinary bladder	DWI, b=700		4.08	4.06
	Urinary bladder	DWI, b=1500		3.97	3.78
Kierans et al51	Adnexal lesion	ADC		4.54	4.94
				Low grade	High grade
Ryu et al56	Glioma	ADC		6.26	6.86
				Volunteers	Healthy
Brinkmann et al22	Liver	MRS		7.6	4.7
			Healthy adult	Elderly	CSM patients
Cui et al187	Spinal cord	FA map	6.07	6.01	5.32
			Young	Middle aged	Non demented elderly	Elderly (AD)
Chen et al48	Brain	cortical surface structure	1.2	1.24	1.28	1.33

nMITR= normalized maximum intensity-time ratio, FA=fractional anisotropy and AD =Alzheimer’s disease ^{22,35,48,51,56,57,76,187}.

Higher order texture statistics

The features generated from first order statistics provide information about the distribution of voxel intensities in an image but they do not provide any information about the inter-voxel relationships within the image. The inter-voxel relationships in an image can be quantified using several different techniques, the gray level co-occurrence matrix, gray level run length matrix and neighborhood gray tone difference matrix method and they are outlined below^1–4.

Gray level co-occurrence matrix

The spatial distribution of gray level intensities within an image can be extracted by using the gray level co-occurrence matrix (GLCM)². The GLCM is constructed by considering the relationship between voxel pairs and the frequency of each intensity pairs within an image or a region of interest ². The relationship between voxel pairs is characterized by two user defined parameters, the distance (d) and angle (θ) and illustrated in Figure 3a using the parameter, θ. If the number of gray levels in the image is N_g, the number of possible voxel pairs would be N_g × N_g. The frequency of each voxel pair in the image or a region of interest is computed and stored in the N_gxN_g GLCM matrix. The GLCM constructed can be either symmetric ² or asymmetric depending on the ordering of values in voxel pairs. Mathematically, the GLCM of an image of size N_xxN_y with N_g gray levels is given by the following equation²:

$${\mathit{GLCM}}_d^{\theta }(i,j)=\mid \{((r,s),(t,v)):I(r,s)=i,I(t,v)=j\}\mid \forall i,j\epsilon \{1,2,3,\dots ,N_g\}$$ (5)

where (r, s), (t, v) ∈ Nx × Ny; $(t,v)=\{\begin{array}{cc}r+d,s& if\theta =0°\\ r+d,s+d& if\theta =45°\\ r,s+d& if\theta =90°\\ r-d,s+d& if\theta =135°\end{array}$; I represents the image as a function I N_x × N_y → {1,2, …, N_g}; and |·| denotes the cardinality of a set

Figure 3.

(a) Illustration of the inter-pixel relationships characterized by the user defined parameter, θ (b) An example 5 × 5 matrix with gray values ranging from 1 to 5. (c) The resultant symmetric gray level co-occurrence matrix (GLCM) obtained by multiplying the asymmetric GLCM with its transpose.

The GLCM obtained from equation (5) is an asymmetric GLCM and can be converted to a symmetric GLCM by multiplying by the transpose (G_sym = G × G^T).

The range of input parameters is θ ∈{0°, 45°, 90°, 135°}, d ∈ {1, 2, 3,…, n} for a two dimensional image and by extending to a three dimensional image, the range for theta increases to a total of 13 angles. Figure 3 demonstrates an example 5×5 matrix with the corresponding symmetric GLCM for θ = 0° and d = 1.. Therefore, based on different values of theta and d, a large number of GLCMs can be produced and for each GLCM, fourteen textural features were developed by Haralick, et al² and current research into texture analysis has led to construction of more features, with most recent applications using up to twenty-two GLCM based features⁵.

Studies have found no significant differences in changing the number of gray levels (eg, 64, 128, etc) in the image to define the GLCM based features to determine benign form malignant lesions or treatment response^65–66. From the twenty-two GLCM texture features. The most commonly used features are given below:

$$\mathit{ASM}={\sum }_{i=1}^{N_g}{\sum }_{j=1}^{N_g}{(G_{\mathit{norm}}(i,j))}^2$$ (6)

$$\mathit{Entropy}=-{\sum }_{i=1}^{N_g}{\sum }_{j=1}^{N_g}{G}_{\mathit{norm}}(i,j){log}_2(G_{\mathit{norm}}(i,j))$$ (7)

$$\mathit{Contrast}={\sum }_{i=1}^{N_g}{\sum }_{j=1}^{N_g}{\mid i-j\mid }^2{G}_{\mathit{norm}}(i,j)$$ (8)

$$\mathit{Correlation}=\frac{{\sum }_{i=1}^{N_g}{\sum }_{j=1}^{N_g}ij(G_{\mathit{norm}}(i,j)-{\mu }_x(i){\mu }_y(j)}{{\sigma }_x(i){\sigma }_y(j)}$$ (9)

The notation used in the above equations is explained in table 2.

Table 2.

Notation used in the equations for computing texture features using gray level co-occurrence matrix

Gnorm	Normalized gray level co-occurrence matrix
Gx	Marginal probability matrix obtained by summing the rows of Gnorm
Gy	Marginal probability matrix obtained by summing the columns of Gnorm
μx	Mean of Gx
μy	Mean of Gy
σx	Standard deviation of Gx
σy	Standard deviation of Gy

The angular second moment (ASM) feature is the measure of uniformity in an image. A uniform image will have a few gray level transitions with the maximum uniformity of one representing only a single type of gray level transition through the image. In contrast, the entropy feature, increases as different types of gray level transitions are present in an image. Visually an image with higher entropy will appear heterogeneous. Similarly, an image with a higher contrast feature value will have a higher frequency of large intensity differences between each neighbor. A simple example of a high contrast image is a checkerboard. Finally, the correlation feature measures the linear dependence between any two neighboring voxels throughout the image i.e. an image with higher contrast will generally have lower correlation. In addition, texture features obtained from the GLCM are not rotationally invariant. Rotationally invariant features can be obtained by taking mean and range of texture values obtained in all directions (four for 2D images and thirteen for 3D volumes). Computation of rotationally invariant features for 3D is not straightforward because voxel spacing along the third dimension may not be same as voxel spacing in the other two dimensions. Thus, using the same value of d (GLCM input parameter) as voxel spacing in all three dimensions is very important to achieve rotationally invariant texture features.

For many applications, GLCM based features are the most commonly used textural features applied to different medical imaging modalities for characterization of biological tissue of interest^{5,16,19–21,26,28,30,31,34,37,39–41,47,49,50,54,56,58–63,65–101}. These applications used symmetric GLCMs for texture analysis as defined by Haralick et al ²; however recent applications of GLCM based features have also used asymmetric GLCMs with some success⁵. Chen et al. compared the GLCMs generated from 2D slices vs. 3D volume on breast DCE-MRI and showed that 3D GLCM texture features performed significantly better than 2D GLCM texture features in classifying malignant from benign breast lesions⁶⁵.

Gray level run length matrix

The gray level run length matrix (GLRL) is defined as the number of contiguous voxels that have the same gray level value and it characterizes the gray level run lengths of different gray level intensities in any direction³. Elements (i,j) in the matrix represents the number of times, j, a gray level value, i, appears in the image. Based on the direction angle, θ (4 for two dimensions and 13 for three dimensions), different GLRL matrices can be constructed. The inter-pixel relationship based on the user defined parameters of angle, θ and run length, j is illustrated in figure 4a. Mathematically GLRL of an image of size N_x×N_y with N_g gray levels is given by the following equation:

$${\mathit{GLRL}}_{\theta }(i,j)=\mid \{(m,n):\mid \{(k,l)\in Nb(m,n,j,\theta ):I(k,l)=i\}\mid =j\}\mid \forall i,j\epsilon \{1,2,3,\dots ,N_g\}$$ (10)

where (m, n) ∈ Nx × Ny; $Nb(m,n,j,\theta )=\{\begin{array}{ll}\{(m+1,n),(m+2,n),\dots ,(m+j,n)\}\hfill & if\theta =0°\hfill \\ \{(m+1,n+1),(m+2,n+2),\dots ,(m+j,n+j)\}\hfill & if\theta =45°\hfill \\ \{(m,n+1),(m,n+2),\dots ,(m,n+j)\}\hfill & if\theta =90°\hfill \\ \{(m-1,n+1),(m-2,n+2),\dots ,(m-j,n+j)\}\hfill & if\theta =135°\hfill \end{array}$; I represents the image as a function: N_x × N_y → {1,2, …, N_g}; and |·| denotes the cardinality of a set

Figure 4.

(a) Illustration of the inter-pixel relationships characterized by the user defined parameters, angle θ and run length j. (b) Example 5 × 5 matrix with values ranging from 1 to 5. (c) Resultant gray level run length matrix (GLRL) for run lengths of 1 to 5 and θ = 0°.

Galloway et al developed five features from the GLRL matrix viz. short runs emphasis (SRE), long runs emphasis (LRE), gray level nonuniformity (GLN), run length nonuniformity (RLN) and run percentage (RP) given by the following equations:

$$\mathit{SRE}=\frac{{\sum }_{i=1}^{N_g}{\sum }_{j=1}^{N_r}\frac{\mathit{GLRL}(i,j)}{j^2}}{{\sum }_{i=1}^{N_g}{\sum }_{j=1}^{N_r}\mathit{GLRL}(i,j)}$$ (11)

$$\mathit{LRE}=\frac{{\sum }_{i=1}^{N_g}{\sum }_{j=1}^{N_r}{j}^2\mathit{GLRL}(i,j)}{{\sum }_{i=1}^{N_g}{\sum }_{j=1}^{N_r}\mathit{GLRL}(i,j)}$$ (12)

$$\mathit{GLN}=\frac{{\sum }_{i=1}^{N_g}{\left({\sum }_{j=1}^{N_r}\mathit{GLRL}(i,j)\right)}^2}{{\sum }_{i=1}^{N_g}{\sum }_{j=1}^{N_r}\mathit{GLRL}(i,j)}$$ (13)

$$\mathit{RLN}=\frac{{\sum }_{j=1}^{N_r}{\left({\sum }_{i=1}^{N_g}\mathit{GLRL}(i,j)\right)}^2}{{\sum }_{i=1}^{N_g}{\sum }_{j=1}^{N_r}\mathit{GLRL}(i,j)}$$ (14)

$$RP={\sum }_{i=1}^{N_g}{\sum }_{j=1}^{N_r}\frac{\mathit{GLRL}(i,j)}{P}$$ (15)

where N_r is the number of different run lengths measured to compute GLRL matrix and P is the total number of voxels in the image. A total of eleven features based on gray level run length matrices have been proposed in the literature ^3,102,103. Current applications in texture analysis use the complete set of eleven features derived from GLRL matrix⁵.

An example 5×5 input matrix and the corresponding GLRL matrix for horizontal direction are shown in figure 4b and 4c respectively. The GLRL matrices are generally normalized (by the total number of entries in the matrix) in order to facilitate consistency between different scans of different patients as well as different scans for same patient considered across a time interval. GLRL matrix based features have also been applied to a number of applications across different medical imaging modalities for characterization of biological tissue of interest ^{5,16,19,21,28,30,31,34,37,39,41,42,50,54,58,60–63,69,79}.

Neighborhood gray tone difference matrix

The neighborhood gray tone difference matrix (NGTDM) is a texture analysis method based on the visual properties of an image ⁴. Neighborhood gray tone difference matrix (NGTDM) is a one dimensional matrix computed such that the each gray level entry, defined as gt, in the NGTDM is the summation of the differences between all the pixels with gray level value, gt and the average gray level value of its neighborhood. The size of the neighborhood is defined by the user. Mathematically, NGTDM of an image (N_x × N_y) with N_g gray levels is given using the following set of equations:

$$\mathit{ANGT}(i,j)=\frac{1}{W-1}\left({\sum }_{ik=-d}^d{\sum }_{jk=-d}^{d}I(i+ik,j+jk)\right),(ik,jk)\ne (0,0)\forall i\in \{1,2,3,\dots ,N_x\}\mathit{and}j\in \{1,2,3,\dots ,N_y\}$$ (16)

$$\mathit{NGTDM}(gt)={\sum }_{(i,j)\mid I(i,j)=gt}\mid gt-\mathit{ANGT}(i,j)\mid \forall gt\in \{1,2,\dots ,N_g\}$$ (17)

Here, ANGT is the average neighborhood gray tone in the neighborhood of the pixel at position (i,j), W=(2d+1)² and d = size of the neighborhood (e.g. 1 in case of 3 × 3 neighborhood). There are five features derived from the NGTDM and are given by the following equations:

$$\mathit{Coarseness}={\left[\epsilon +{\sum }_{i=1}^{N_g}{P}_i\mathit{NGTDM}(i)\right]}^{-1}$$ (18)

$$\mathit{Contrast}=\left[\frac{1}{N_t(N_t-1)}{\sum }_{i=1}^{N_g}{\sum }_{j=1}^{N_g}{P}_i{P}_j{(i-j)}^2\right]\left[\frac{1}{n^2}{\sum }_{i=1}^{N_g}\mathit{NGTDM}(i)\right]$$ (19)

$$\mathit{Busyness}=\frac{\left[{\sum }_{i=1}^{N_g}{P}_i\mathit{NGTDM}(i)\right]}{\left[{\sum }_{i=1}^{N_g}{\sum }_{j=1}^{N_g}i{P}_i-j{P}_j\right]},P_i\ne 0,P_j\ne 0$$ (20)

$$\mathit{Complexity}={\sum }_{i=1}^{N_g}{\sum }_{j=1}^{N_g}\left\{\frac{\mid i-j\mid }{n^2\left(P_i+P_j\right)}\right\}\{P_i\mathit{NGTDM}(i)+P_j\mathit{NGTDM}(j)\},P_i\ne 0,P_j\ne 0$$ (21)

$$\mathit{Textures}\mathit{strength}=\frac{\left[{\sum }_{i=1}^{N_g}{\sum }_{j=1}^{N_g}\left(P_i+P_j\right){(i-j)}^2\right]}{\left[\epsilon +{\sum }_{i=1}^{N_g}\mathit{NGTDM}(i)\right]}$$ (22)

where P_i is the probability of occurrence of a gray level value I, n is the number of entries in the NGTDM and N_t is the total number of gray levels present in the image.. For example, figure 5 demonstrates the application to a 5×5 gray level input matrix.

Figure 5.

(a) Illustration of the neighborhood around the pixel of interest based on the user defined neighborhood parameter, d (b) Example 5×5 input matrix with values ranging from 1 to 5. (c) Neighborhood gray tone difference matrix for d=1.

The NGTDM features have been developed to correlate the quantitative values of texture features as closely as possible to the visual interpretation of texture by humans. For example, coarseness provides a quantitative measure of local uniformity while contrast provides quantitative information about the difference in the intensity levels of neighboring regions. Similarly, busyness tells us how “busy” (rapid intensity changes) are the neighborhoods in a given image or a region of interest. Furthermore, complexity, as the name suggests, quantifies the complexity of the spatial information present in an image. Finally, texture strength can be defined as characterizing the visual aesthetics of an image. Images with higher texture strength are generally more attractive to look at than images with low texture strength. The NGTDM features have been applied to some medical image applications ^16,49,104.

Morphological features

Morphological features are used in many settings to define the shape of an object. For example, the shape of the tumor can be quantified using fractal dimension^105,106. Fractal dimension is a measurement of irregularity in the shape of the tumor. Different methods proposed in the literature for computing the fractal dimension of a pattern are discussed in¹⁰⁷. Other methods based on volume and surface area of the tumor have been summarized in⁵. However different fractal sets having different textures may have the same fractal dimension values. Consequently, Mandelbrot introduced the concept of lacunarity for characterization of texture in an image ¹⁰⁶. Lacunarity quantifies how fractals fill space, i.e., larger the gaps, higher the lacunarity. Different methods proposed in the literature for computing the fractal dimension of a pattern are discussed in¹⁰⁷. Li et al ¹⁰⁸ evaluated four methods for computing fractal dimension of parenchymal patterns obtained from mammography in assessment of breast cancer risk. The four methods evaluated included conventional box counting method, modified box counting method using linear discriminant analysis (LDA), global Minkowski method and modified Minkowski method using LDA. The authors observed that the advanced methods using LDA resulted in better classification between low and high risk patterns (AUC=0.9 for modified box counting method and AUC=0.93 for modified Minkowski method). Likewise, Guo et al¹⁰⁹ compared five different methods for computing the fractal dimension as well as lacunarity analysis of breast regions obtained from mammography to classify breast masses from normal parenchyma. The fractal dimension methods tested in this study were Reticular cell counting method¹¹⁰, differential box counting method¹¹¹, blanket method¹¹², Fourier power spectrum method¹¹³ and fractional Brownian motion model(FBM) ¹¹⁴. The highest AUC of 0.84 was achieved for the FBM method. Moreover, when FBM method was combined with lacunarity, the AUC increased to 0.90. Morphological features based on Renyl fractal dimension, shape analysis, volume and surface area of the tumor have also been implemented in the literature^5,115,116.

Filtering approaches

The basic idea of filtering approaches is that the original textured image is subjected to some form of linear or nonlinear transform, followed by statistical analysis of the transformed image to obtain texture information. A comparative study of different filtering approaches can be found in¹¹⁷. In this paper, we discuss the filtering techniques commonly used for texture analysis of medical images.

Spatial filtering techniques

Spatial filtering techniques are based on neighborhood operations on the original textured input images. These neighborhood operations are based on filters or kernels of size n × n, where n determines the size of the neighborhood considered by the kernel. Some examples of commonly used filters for texture analysis include statistical filters like average filter, range filter and entropy filter or edge filters like Prewitt filter, Sobel filter, Laplacian filter and Laplacian of Gaussian (LoG) filter. The input image is convolved with the desired kernel to produce filtered images highlighting specific texture information in the original texture image. The resultant filtered images are analyzed using first order statistics (mean, median, standard deviation, etc.). Statistical and edge filtering techniques have been applied to medical images for texture analysis in a number of applications^{7,59,118–126}.

Apart from statistical and edge kernels, special kernels have also been designed for identifying different types of textures. For example, Laws designed three sets of one dimensional convolution masks of different sizes corresponding to different types of textures such as level, edge, spot, wave, ripple, undulation and oscillation^127,128. All the convolution masks were center weighted, symmetric or asymmetric and zero-sum except the level convolution masks. Using a vector product between the masks of the same sizes, different spatial domain filters or kernels of size 3×3, 5×5 and 7×7 were generated by Laws. Laws texture energy measures have been used in filtering input medical images for texture analysis in many applications^39,129–140. Another example of a specially designed kernel is the fractal dimension kernel designed by Al Kadi et al¹⁴¹. Examples of different techniques for filtering in the spatial domain have been illustrated in figure 6. The value of n (size of the kernel neighborhood) is determined by the spatial resolution of the image being filtered. Higher the spatial resolution of the image being filtered, higher the value of neighborhood, n, used for filtering. However, further research is required to standardize the size of the neighborhood for different resolution images acquired using different imaging modalities.

Figure 6.

Illustration of different techniques used for spatial domain filtering (a) statistical kernel (e.g. median filter) (b) Edge kernel (e.g. Laplacian of Gaussian filter) (c) Special kernel (e.g. Fractal dimension filter). Data from^59,141,170

Multi-resolution image scaling

The frequency of variations in the gray level values in a region of interest is dependent on the scale of the region of interest. The frequency content within an image can be analyzed at different scales using wavelets^142–145. Image texture can be analyzed at different scales by representing the image in a pyramid structure. Using discrete wavelet transform, four low resolution images can be obtained from the original image viz. I_LL, I_LH, I_HL and I_HH. By repeatedly applying discrete wavelet transform on I_LL at each level, hierarchical pyramid structure for different resolutions can be created.

Texture analysis can be done by computing statistical texture features at each level or averaging the results across multiple resolutions. Wavelets have been extensively used in multiresolution texture analysis of medical images ^{5,38,50,58–63,70,73,77,88,140,146}. The use of multiresolution can decompose the data into different frequency components, thereby facilitating the study of each component with a resolution matched to its scale. Figure 7 illustrates multiresolution scaling on an example breast MRI diffusion weighted image¹⁴⁷.

Figure 7.

Multiresolution methods applied to a diffusion-weighted image (b=500): a) the original size 256×256; b) compressed image (64×64) at different levels. c) For compression 2D biorthogonal spline wavelets were used. d(h)j, d(v)j and d(d)j respectively are detail components corresponding to vertical, horizontal, and diagonal. aj, is the approximation (coarse) component at decomposition level. Data from ¹⁴⁷

Feature selection

The next step is to select a subset of features that can characterize the tissue of interest. The challenge here is that even using one d value and thirteen theta values, a large number of GLCM features are extracted. Reduction of these features needs to be done in order to avoid the potential problem of overfitting. In the event of overfitting, the statistical model will better reflect noise in the image than the original data. There are a number of strategies available for countering the problem of overfitting. Tpyically, If a particular set of features have been shown to be most relevant in the particular application of interest, then those features can be manually selected for subsequent analysis. In addition, by using regression analysis, correlations between different features can be analyzed and redundant features maybe be removed. Thus, the feature selection procedure can be either supervised or unsupervised. In supervised feature selection, most informative features are selected based on some ground truth knowledge about the task at hand. As the biological meaning of different radiomic features has not yet been established, the ground truth used for feature selection is generally the final class label (e.g. benign or malignant). The supervised feature selection can be accomplished using a filtering or a wrapper method. Filtering methods evaluate one feature at a time for its importance or predictive ability. Some examples of filtering methods include Fisher’s criterion, Wilcoxon rank sum test, Student t-test, etc¹⁴⁸. Parmar et al⁶² compared the performance of fourteen such feature selection methods in predicting overall survival in lung cancer patients using CT scans. They found that the Wilcoxon test based method had the best performance and maximum stability. One major disadvantage of filtering methods is that they do not consider the dependencies between different features that may lead to better prediction. Wrapper methods, on the other hand, evaluate subsets of features for their combined predictive power. As a result, wrapper methods are capable of examining the dependencies between different features. However, due to very high dimensionality of radiomics feature space, evaluation of complete feature subset space is computationally NP-hard¹⁴⁹. Some examples of computationally efficient wrapper methods include greedy forward selection and greedy backward elimination as they do not evaluate all possible feature subsets. Finally, supervised feature selection methods are prone to overfitting and may not scale well across applications.

The unsupervised feature selection approach is based on dimensionality reduction algorithms. Dimensionality reduction algorithms transform the high dimensional feature space into a meaningful representation of its intrinsic dimensionality. Dimensionality reduction algorithms can be linear or nonlinear. Linear dimensionality reduction algorithms assume the high dimensional features to lie on or near a linear subspace of some high dimensional topological space while nonlinear dimensionality reduction algorithms do not rely on the linearity assumption. Some examples of linear dimensionality reduction algorithms include principal component analysis (PCA) and multidimensional scaling (MDS) ^150,151. Examples of nonlinear dimensionality reduction algorithms are Isometric mapping (Isomap), locally linear embedding (LLE) and diffusion map^152–154. A review on different dimensionality reduction algorithms can be found in¹⁵⁵. The dimensionality reduction algorithms are not prone to overfitting; however, but, they may not always give optimal low dimensional representations. In medical imaging, Kassner et alused PCA to transform four GLCM textural features obtained from texture analysis of T1-weighted post contrast brain images for prediction of hemorrhagic transformation in acute ischemic stroke in a set of patients, however, there was no longitudinal data⁹¹. Texture was found to be potentially useful in discriminating hemorrhagic transformation from non-hemorrhagic transformation when compared to visual examination. Both supervised and unsupervised feature selection methods have their advantages and disadvantages. Further research is required to determine an optimal hybrid approach that combines the advantages of both feature selection methods.

Classification

The final step in the radiomics framework is the classification of the features for quantifying the tissue of interest and prediction. This is done using either a supervised or unsupervised classifier. Supervised classifiers require the user to provide input for the patients for which the underlying pathology is already known. Using the data provided by the user, supervised classifiers learn a classification model to categorize any new patient with unknown pathology. Some examples of supervised classifiers include support vector machine (SVM), random forest (RF), k-nearest neighbors (kNN), etc^156–159.

In contrast to supervised classifiers, unsupervised classifiers do not need any training data. Unsupervised classifiers group the patients based on some form of distance metric, such as, Euclidean distance, angular distance, etc. Unsupervised classifiers are very useful when the goal of classification step is to identify a patient or a group of patients in the database that are most similar to the new patient with unknown pathology. The diagnosis and treatment details for the known patients may be then used to efficiently diagnose or determine outcomes for the patient. Some examples of unsupervised classifiers include k-means clustering, hierarchical clustering, consensus clustering, etc ^160–162. A recent study compared the performance of twelve different supervised classifiers in predicting overall survival in lung cancer patients using CT scans and found the random forest classifier to have relatively the best performance with low AUC=0.6⁶². However, since Radiomics is a new and ever expanding area of research and with the development of better classification algorithms, the optimal method is not clearly defined and will depend on the application.

Radiomics applications

In early applications to medical imaging, the majority of methods used Texture analysis, which, then evolved into the current application of Radiomics. An extensive review on the applications of both texture and radiomics analysis in neurologic pathologies can be found in¹⁶³. In this paper, we reviewing the application of these methods to mostly body and cancer applications.

Lung

a) X-Ray

Texture analysis was first used in 1972 for automatic classification of pulmonary disease by Sutton and Hall⁷. They employed texture measures of symmetry, directional gradient and Fourier domain energy on a dataset of 24 patients to classify between lung tissue using linear discriminant analysis. The authors observed that the texture features based on directional gradient gave the best classification accuracy. The training classification accuracy was achieved at 92% while the testing classification accuracy was achieved at 84%. This study was directed towards demonstrating the usefulness of texture in automatic screening of chest radiographs.

b) Computed Tomography

Al-Kadi and Watson implemented the differential box counting based fractal analysis method as well as lacunarity analysis on contrast enhanced (CE) CT images for differentiation of aggressive malignant lung tumors from the nonaggressive ones on a cohort of 15 patients¹⁴¹. The authors achieved an accuracy of 83.3% for distinguishing between these two groups using average fractal dimension. The authors also observed high correlation between the average fractal dimension and tumor uptake of 18FDG obtained using PET.

Ganeshan et al developed texture features using multiscale (fine to coarse) application of spatial domain filtering algorithm LoG (Laplacian of Gaussian) followed by extraction of statistical features of mean gray intensity (MGI) along with entropy (E) and uniformity (U) on unenhanced CT scans of 18 non-smal cell lung cancer patients to correlate with tumor glucose metabolism and stage¹²¹. The authors found significant associations between coarse texture features and tumor standard uptake value (SUV) and fine texture features and tumor stage. In another study, Ganeshan et al used the texture feature of uniformity to associate with patient survival in a study of 54 patients¹²⁴. The authors observed that patients with coarse texture uniformity of less than 0.62 did not survive more than two and a half years. The same research group further extended their research to correlate texture features with histopathological markers of angiogenesis and hypoxia on a dataset of 14 patients with unenhanced as well as CE CT images ¹²⁵. The same texture analysis method was employed with different statistical features (standard deviation (SD) along with mean (MPP) and uniformity (UPP) of positive pixel distribution). The results indicated significant associations between texture features and average intensity of tumor staining with pimonidazole, tumor Glut-1 expression and tumor CD34 expression. The relationships between image features and the histograms generated at different scales using spatial domain filtering algorithm, LoG have also been analyzed by the same group to provide clinical understanding of results¹⁶⁴.

Aerts et al used texture features based on first order statistics, GLCMs, GLRL matrices at multiple scales using wavelets on CT images of 1019 patients with non-small cell lung cancer or head-and-neck(H&N) cancer to associate radiomic features with primary tumor stage as well as patient survival⁵. The radiomic features of first order statistics energy, shape compactness and gray level nonuniformity (GLRL feature) obtained from unscaled data as well as wavelet HLH scaled data were found to be most significant. Good to moderate prediction concordance indices (CIs) of 0.65, 0.69 and 0.69 were achieved on one lung cancer validation dataset of 225 patients and two H&N datasets of 136 and 95 patients. The authors found significant associations between radiomic features and gene expression patterns indicating the utility of radiomic features in characterizing underlying biological mechanisms. Strong correlations were observed between the radiomic features (GLRL gray level non uniformity obtained from unscaled data as well as wavelet HLH scaled data) and cell cycling pathways, demonstrating higher proliferation for more heterogeneous tumors.

Recent studies by the same group achieved low AUCs of 0.66 (lung cancer) using random forest and CIs of 0.6 (lung cancer) and 0.68 (H&N cancer) using consensus clustering on the same set of radiomic features for predicting patient survival, which needs further investigation^62,63. Moreover, using consensus clustering, the authors also found either no or some association between consensus clusters and lung stage (AUC =0.61), lung histology (AUC=0.56), H&N stage (AUC=0.77) and H&N histology (AUC=0.58) based on the poor AUC. However, these are essentially low AUCs and more research is needed.

The same group used additional texture features based on first order statistics obtained after the application of LoG filter in addition to the other radiomic features used previously on a cohort of 182 patients to predict probability of distant metastasis in lung adenocarcinoma in addition to patient survival⁵⁹. The authors found thirty five radiomic features to be somewhat prognostic for distant metastasis (CI>0.6) and twelve features for patient survival (CI=0.55). The authors reported four features based on the LoG filter to be trending for potentially prognostic for distant metastasis and patient survival. A recent study by the same group tested two new features of entropy ratio and convexity for association with patient survival in lung adenocarcinoma using two independent patient cohorts of 61 and 47 patients⁶⁴. The authors found significant association between the two features and patient survival for only the first cohort.

In summary, statistical features obtained from LoG based spatial domain filtering of tumors at coarse level were found to be more predictive in different applications than statistical features obtained at fine level. A LoG filter highlights the edges within an ROI, suggesting that edges found by the filter using a wider Gaussian were more informative than edges produced using a narrow Gaussian. This may be because edges found using a narrow Gaussian are more susceptible to noise in the input image, Furthermore, LoG based features were also found to be more informative than statistical features based on first order statistics, GLCM and GLRL⁵⁹.

c) PET

Cook et al investigated the texture features obtained from NGTDM extracted from FDG PET images of 53 non-small cell lung cancer patients for association with chemoradiotherapy response and survival¹⁰⁴. The texture features were compared to the three survival groups (overall survival (OS), progression free survival (PFS) and local PFS (LPFS) obtained using RECIST (Response Evaluation Criteria in Solid Tumors) criteria. The authors found the texture features of coarseness, busyness and contrast to significantly predict survival in the patient cohort as summarized in Table 3. The authors observed that responders had lower coarseness but higher contrast and busyness than non-responders. This suggests that the texture obtained for tumors corresponding to responders is less uniform with high frequency of intensity changes i.e. they are more heterogeneous than tumors corresponding to nonresponders.

Table 3.

Summary of coarseness, contrast, busyness and complexity values corresponding to responders and non-responders of NSCLC obtained using Lung PET as reported in the literature¹⁰⁴.

Pathology	Coarseness	Contrast	Busyness	Complexity
Responders	0.01	0.11	0.76	1938
Non-responders	0.03	0.04	0.37	1926

d) PET-CT

Vaidya M et al used FDG-PET/CT dataset of 27 patients for characterization of radiotherapy tumor response in non-small cell lung cancer⁴⁷. Texture features of first order statistics and gray level co-occurrence matrix features were combined with the SUV/HU (Halsted units) measurements in this study. The model was built using logistic regression and resampling methods of cross validation and bootstrapping. The authors observed that the first order statistics features obtained from intensity volume histogram correlated more strongly with loco-regional control in contrast with the GLCM based features which correlated more strongly with local control.

Breast

a) Mammogram

Texture analysis was first used on mammograms in 1986 by Magnin et al to evaluate the risk for developing breast cancer ⁶⁷. GLCM based texture features were used in this study. However this study did not yield very good results with reproducibility barely reaching 80%. Wei et al successfully implemented GLCM based texture analysis at multiple scales using wavelets on 672 ROIs to classify between biopsy proven masses and normal parenchyma using linear discriminant analysis⁷⁰. They achieved an AUC of 0.89 on the training set and an AUC of 0.86 on the test set. A study by Chan et al investigated the use of texture analysis on mammograms for associating the presence of clustered microcalcifications with malignant pathology⁷⁵. The authors implemented the GLCM texture features on a set of 86 mammograms followed by a backpropagation artificial neural network classifier to achieve an AUC of 0.88. Multiple studies have since investigated the use of texture analysis on mammograms for detection of masses^{39,77,79,81,88,109,115,131,134,165–168} with an average AUC of 0.87 and a maximum AUC of 0.96 ³⁹ achieved with texture features obtained using first order statistics based features, GLCM based features, GLRL matrix based features and Law’s texture energy measures. The quantitative values of the fractal dimension obtained from breast mammograms corresponding to normal, mass and architectural distortion as reported in the literature are summarized in table 4.

Table 4.

Summary of fractal dimension values corresponding to different breast tissue classes obtained using mammogram as reported in the literature^109,167.

Tissue class	Fractal dimension
Normal	2.64
Mass	2.39
Architectural distortion	2.52

b) Ultrasound

Garra et al performed texture analysis (first order statistics, GLCM and fractal dimension) on breast ultrasound in a cohort of 80 patients²⁰. The authors were able to identify malignant lesions with a sensitivity of 100% and specificity of 80% (78% for fibroadenoma, 73% for cysts and 91% for fibrocystic nodules). The best features identified in this study were GLCM based contrast with θ = 45° and GLCM based correlation with θ = 0°. Numerous studies have since used texture analysis to differentiate between benign and malignant breast lesions using ultrasound ^{83,140,146,169}.

c) MRI

The first application of texure analysis in breast MRI was in 1997 by Sinha et al ⁷⁶. This study included 43 breast cases (23 benign and 20 malignant) and used eight texture features in combination with patient age and the DCE-MRI parameter of maximum enhancement to obtain sensitivity and specificity of 93% and 95% respectively. In reality, it was Radiomics, since they compared the results to an outcome or clinical variable. Moreover, Gibbs et al implemented GLCM based texture features to differentiate between benign and malignant breast tumors on post contrast MRI images from 79 patients⁸⁵. Texture features of variance, entropy and sum entropy were found to be most significant using logistic regression analysis and the ROC analysis resulted in an excellent AUC of 0.92. Similarity, Ertas et al extracted first order statistics from normalized maximum intensity-time ratio (nMITR) projection generated using DCE-MRI to classify between benign and malignant breast tumors in 46 patients³⁵. The texture features of entropy, mean, standard deviation and maximum were observed to be the most significant (p<0.001) with excellent AUCs between 0.86 and 0.97.

Nie et al used GLCM based features along with morphological features to distinguish between benign and malignant tumors using post contrast MRI images of 71 patients⁸⁹. The authors found the texture features of compactness, normalized radial length entropy, volume and GLCM based features of entropy, sum average and homogeneity to most significantly differentiate benign and malignant tumors and obtained an AUC of 0.82. The same group compared the two feature selection and classification methods of logistic regression and artificial neural network for the task of classifying malignant breast tumors from benign breast tumors⁹². The authors found there was no significant difference in the results obtained from the two methods.

Instead of using post contrast MRI images, Karahaliou et al used parametric maps derived from DCE-MRI (e.g. signal enhancement ratio map) to extract GLCM based features⁹⁴. The authors found the GLCM ASM, GLCM entropy and GLCM sum entropy obtained from signal enhancement ratio (SER) map as the most discriminative features producing an AUC of 0.92. Agner et al used kernels based on GLCM features along with other filters on the DCE MRI images in a dataset containing 41 patients for differentiating malignant from benign breast tumors¹⁷⁰. The authors created textural kinetic curves by computing the mean of the filtered image within the region of interest across the DCE sequence of images. The textural kinetic curves were classified using a probabilistic boosting tree and achieved sensitivity and specificity of 99% and 76% respectively. In a recent study, Wang et al used morphological and GLCM texture features in combination with pharmacokinetic parameters obtained from DCE-MRI to classify between benign and malignant breast tumors¹⁰¹. The authors found the texture feature of GLCM entropy, GLCM energy and compactness along with pharmacokinetic parameters of rate constant (k_ep) and volume of plasma (v_p) to be the most discriminative with sensitivity and specificity of 91% and 92% respectively. Cai et al used GLCM based texture features from DCE-MRI in combination with ADC, kinetic curve features and morphological features to distinguish between benign and malignant breast tumors on a cohort of 234 patients⁹⁹. The authors achieved sensitivity and specificity of 85% and 89% respectively. They also tested the same set of features on a validation dataset consisting of 93 patients and achieved sensitivity and specificity of 69% and 91% respectively⁹⁸.

Holi et al applied texture analysis (first order statistics, GLCM, GLRL) on T1-weighted pre contrast, post contrast and subtraction breast MRI datasets from twenty patients in order to associate texture features with histological types of invasive breast cancer (lobular vs. ductal) ⁴². The authors identified the entropy based GLCM features to be the most effective features and were able to achieve a maximum accuracy of 100% using linear discriminant analysis (LDA) and nonlinear discriminant analysis (NDA) on the first subtraction and contrast images.

A recent study by Ahmed et al performed GLCM based features on breast MRI to predict chemotherapy response in 100 breast cancer patients⁶⁶. The authors found the texture features of contrast, variance, difference in variance, sum variance, sum entropy, sum average, cluster shade and cluster prominence showed significant difference between responders and partial responders of chemotherapy when implemented on post contrast images. Parikh et al used multiscale LoG (Laplacian of Gaussian) filter followed by extraction of first order statistical features from T2-weighted MRI of 36 patients to predict chemotherapy response¹⁷¹. The authors found the texture features of entropy and uniformity showed significant different between responders and non-responders with an AUC of 0.84.

In summary, post contrast enhanced MRI was the most frequently used image for texture analysis of breast tumors. This is consistent with the clinical environment where radiologists also use post contrast enhanced MRI to discern textural features corresponding to breast tumors.

The quantitative values of the texture features obtained using post contrast enhanced breast MRI images for benign and malignant lesions reported in the literature are summarized in table 5. The quantitative texture values of entropy, energy, etc obtained from various studies maybe difficult to compared due to several reasons. For example, the use of different magnet strengths (1.5 or 3T), differnent preprocessing steps, binning methods, and gray level normalization. However, a trend in the quantitative values can still be observed, for example, entropy, energy (uniformity), and contrast values tend to be increased compared to benign and/or normal tissue. These metrics may provide an insight into the heterogeneity of the tumor tissue. Based on the current literature, It may be inferred that malignant tumor tissue is more heterogeneous (high entropy, low uniformity and high contrast) than benign tumor tissue and seen in pathological samples¹⁷². This correlation is the impetus for further Radiomics research in developing noninvasive tools for clinical decision.

Table 5.

Summary of quantitative values of four GLCM based features obtained using post contrast enhanced breast MRI images for benign and malignant lesions as reported in the literature ^{65,66,76,85,94}.

Image Metrics					Benign lesion				Malignant lesion
Reference	Acquisition	Pre-processing	gray levels	Dimension	ASM	Entropy	Contrast	Correlation	ASM	Entropy	Contrast	Correlation
Sinha et al76	Post contrast DCE-MRI	None	16	2D	0.02	1.96	34.3	-	0.01	2.27	535.8	-
Ahmed et al66	Post contrast DCE-MRI	H.Eq	16	2D	-	-	-	-	0.01	7.13	8.5	0.8
Gibbs et al65,85	Post contrast DCE-MRI	H,Eq	32	2D	0.004	5.85	25.5	0.84	0.004	5.94	23.2	0.86
Chen et al63	Post contrast DCE-MRI	H.Eq	32	2D	0.01	5.41	52.1	0.6	0.01	5.44	75.8	0.6
Chen et al65	Post contrast DCE-MRI	H.Eq	32	3D	0.01	5.12	67.5	0.5	0.01	5.26	69.2	0.55
Karahiliou et al94	SER map	None	64	2D	0.025	1.71	12.8	0.37	0.01	2.12	23.8	0.51

H.Eq =histogram equalization, SER= signal enhancement ratio and ASM = angular second moment.

Liver

a) Computed Tomography

Mir et al implemented texture analysis based on GLCMs and GLRL on the liver CT images of 60 patients to classify the CT images as normal liver, clearly visible malignancy and invisible malignancy⁶⁹. The authors found the texture features of entropy (normal: 1.65±0.12, visible malignancy: 2.13±0.17, invisible malignancy: 1.64±0.08), local homogeneity (normal: 2.30±0.25, visible malignancy: 1.48±0.21, invisible malignancy: 2.23±0.15) and gray level distribution (normal: 5.54±0.31, visible malignancy: 2.09±0.61, invisible malignancy: 4.72±0.54) to be the most significant. Chen et al used fractal analysis along with GLCM based features to classify the liver tumors as hemangioma or hepatoma in a dataset of 30 patients⁸⁰. Using modified probabilistic neural network classifier, the authors were able to achieve classification accuracy of 83%. In a study by Gletsos et al, GLCM based features were used to classify the CT focal liver lesions of 147 patients into four classes (normal, hepatic cysts, hemangioma and hepatocellular carcinomas) ⁸⁶. The authors used three sequentially placed feed forward neural networks and achieved excellent accuracies of 97%, 100% and 82% in classifying the normal from abnormal, hepatic cysts from others and hemangioma from hepatocellular carcinomas respectively. Huang et al¹⁷³ used autocovariance function on a dataset of 164 liver CT images to classify malignant (hepatocellular carcinomas and colorectal metastases) lesions from benign lesions and achieved moderate sensitivity and specificity of 75% and 88% respectively.

b) MRI

Jirak et al extracted first order statistical and GLCM based texture features using T2-weighted MRI images obtained from 43 patients to classify between healthy and cirrhotic liver²⁶. The authors achieved classification error around 8%. Along with T2-weighted MRI images, Mayerhoefer et al also extracted texture features (first order statistics, GLCM, GLRL matrix) from T1-weighted images to classify focal liver lesions achieving error rates of 12–18% on T2-weighted images and 16–18% to T1-weighted images⁴¹. Fujimoto et al used entropy of ADC to classify normal from abnormal fibrosis stage (AUC =0.94, p<0.001, entropy cutoff = 1.30) ⁴⁶. ADC maps were created using diffusion weighted MRI (b =0 and 1000 s/mm²). O’ Connor et al quantified tumor heterogeneity using first order statistics on voxel-wise K^trans, v_e and v_p and fractal dimension analysis on DCE-MRI data obtained from 10 patients with 26 colorectal cancer liver metastases to predict shrinkage in tumor volume in response to bevacizumab and cytotoxic chemotherapy⁴⁵. The authors found the median v_e, tumor enhancing fraction (E_f) and microvascular uniformity obtained using fractal dimension to be the most significant features and the median classification error was achieved at 12%.

c) Ultrasound

Texture analysis was first implemented on liver ultrasound images in 1985 by Raeth et al to classify liver into normal, diffuse parenchymal and malignant disease on a dataset of 71 patients with an accuracy of 96%¹⁹. Wu et al used multiresolution fractal analysis to classify between normal liver, hepatoma and cirrhosis on a dataset of 40 patients and achieved an accuracy of 90%⁶⁸. The authors observed that multiresolution fractal analysis features outperformed the GLCM based texture features as well as Laws texture energy measures. Sujana et al extracted first order statistical features along with GLCM and GLRL matrix based features to classify liver ultrasound images from 113 patients into normal, hemangioma and malignant categories²¹. Using artificial neural network, the authors were able to achieve classification accuracy of 100%. Horng et al developed a new texture analysis method called texture feature coding to classify liver ultrasound images into normal, hepatitis and cirrhosis on 120 patients (30 training and 90 test images) ⁸⁴. The authors achieved an accuracy of 86.7% as compared to 75.7% obtained by the same group using GLCM, texture spectrum and fractal dimension based features⁷¹. Yoshida et al used multiresolution analysis on a dataset of 44 patients and obtained an AUC of 0.92 in classifying malignant from benign lesions²⁹.

Numerous studies have since used texture analysis to classify focal hepatic lesions using ultrasound^{132,136,174–180}. A comparative study of different texture analysis approaches implemented in the literature can be found in ¹⁸¹. A recent implementation by Mitrea et al extracted GLCM based texture features from the Laws texture energy images obtained from filtering liver ultrasound images and achieved an accuracy of 90% in liver cancer detection¹³⁹.

Colorectal cancer

a) Computed Tomography

A study by Ganeshan et al investigated the use of texture analysis of liver on 28 patients with colorectal cancer¹¹⁹. The authors used multiscale (fine to coarse) application of the spatial domain filtering algorithm LoG (Laplacian of Gaussian) followed by extraction of first order statistical features. The authors observed that the relative scale texture parameter correlated inversely with the corresponding PET SUV metric (r=−0.59, p=0.007) and hepatic phosphorylation index(HPFI) (r=−0.59, p=0.006). There was apositive correlation with the total hepatic perfusion (THP) (r=0.51, p=0.02-) and hepatic portal perfusion (HPP) (r=0.45,0.05) for patients without liver metastases. The same research group used the same set of texture features obtained from liver CT to predict survival in patients with colorectal cancer¹²⁰. The texture feature of uniformity was obtained for the scale ratios of 1.5 (fine) to 2.5 (course) and 2 (medium) to 2.5(coarse) were observed to be significant (p<0.005) prognostic factors for surival. Goh et al extracted fractal dimension based features from colorectal perfusion CT images of 20 patients to differentiate between colon cancer and normal bowel¹⁸². The authors found the fractal dimension and fractal abundance were significantly (p≤0.001) higher for colon cancer (1.71±0.07 and 7.82±0.62) than normal bowel (1.61±0.07 and 6.89±0.47). Cui et al used fractal dimension analysis on CT images obtained from 228 patients to classify the status of lymph nodes in rectal cancer as benign or malignant¹⁸³. The authors were able to classify the malignant nodes from benign nodes with an accuracy of 88%.

Ganeshan et al used their previously developed multiscale texture analysis algorithm on dynamic contrast enhanced CT of the liver obtained from 27 patients to classify between node negative and not positive non metastatic colororectal cancer¹²². The authors observed significant difference in entropy and uniformity of the node negative and node positive patients. Using fine texture entropy of the images obtained between 26 and 30 seconds after contrast injection, the authors were able to achieve sensitivity and specificity of 100% and 71% respectively in identifying node positive patients (entropy cutoff ≤ 0.0807). The same group used texture analysis of the colorectal tumor obtained from contrast enhanced CT of 57 patients to define biomarkers for 5-year survival¹²⁶. The authors used the texture features of fine scale entropy, uniformity, kurtosis, skewness and standard deviation to define the five year survival rate with the corresponding cutoffs at less than 7.89 for entropy, greater than 0.01 for uniformity, less than 2.48 for kurtosis, greater than 20.38 for skewness and less than 61.83 for standard deviation.

Head and Neck

a) Computed Tomography

Leijenaar et al ⁶⁰ chose four radiomic features (Energy, compactness, GLRL non uniformity and wavelet GLRL non uniformity obtained by Aerts et aldescribed previously⁵ to investigate their prognostic ability on 542 oropharyngeal squamous cell carcinoma (OPSCC) patientsthat underwent CT.. The features were weighted in a Cox model to develop and test a prognostic index for validation of the radiomic signatures. The authors observed that the Kaplan-Meiyer survival curves were significantly different (p<0.05) between low and high radiomic signature predictions using a log-rank test, however, the C-index was 0.628 which is lower than reported in previous reports (C index=0.686 and 0.685)⁵

b) PET

El Naqa et al explored the GLCM texture and features on a cohort of nine Head & Neck patients and 14 cervical cancer patients undergoing chemoradiotherapy and imaged with PET⁹⁰. The most discriminative features from the GLCM included energy, contrast, local homogeneity and entropy. The authors were able to achieve moderate to high AUCs of 0.76 (cervical cancer) and 1.0 (H&N) and 0.76 (cervical cancer) on the very small datasets indicating a potential usefulness of texture in PET imaging. For cervical cancer, the GLCM entropy of the CTV (clinical tumor volume) was much lower (3.6) than that of the tumor (5.3). Similar trend was observed for H&N cancer as well where entropy of the CTV was 3.8 and that of the tumor was 4.7.

Esophageal cancer

a) PET

Tixier et al used first order texture features from the GLCM, GLRL matrix and NGTDM on a PET dataset to classify between response in 41 esophageal cancer patients treated with chemoradiation¹⁶. The authors found the texture features of GLCM entropy (sensitivity=79%, specificity=91%) and GLCM angular second moment (sensitivity=88%, specificity=73%) along with size (sensitivity=76%, specificity=91%) and intensity variabilities (sensitivity=76%, specificity=91%) of uniform zones were the most discriminative between responders and non-responders compared to the SUV_max (sensitivity=46%, specificity=91%).

Adnexal lesion

a) MRI

Kierans et al extracted first order statistical features of entropy from ADC maps to classify between benign and malignant adnexal lesions in 37 patients⁵¹. The average entropy ADC was significantly higher for malignant (4.94±0.4) than benign (4.54±0.44) lesions. However, the sensitivity (66.7%) and specificity (82.4%) of the entropy ADC was low in diagnostic performance.

Prostate lesions

a) MRI

Wibmer et al performed GLCM texture analysis on T2-weighted and diffusion weighted MRI of prostate obtained from 147 patients and compare with the Gleason score as well as cancer detection¹⁸⁴. The authors used five GLCM features of inertia, entropy, energy, correlation and homogeneity for analysis. According to the authors, the texture features of cluster shade and cluster prominence “over-emphasize” the other GLCM features such as energy, entropy, homogeneity and contrast and hence were not used. There were a total of 186 cancerous lesions in the 147 patients with 143 lesions in posterior zone (PZ) and 43 lesions in transition zone (TZ). The authors observed similar trend in the GLCM texture features obtained from PZ tissue in both ADC map and T2WI as well as from TZ tissue in ADC map. For all these cases, the GLCM texture features of entropy and inertia were significantly higher for cancer while the remaining three features were significantly lower for cancer than non-cancerous tissue (p<0.008). Furthermore, the authors observed that only the GLCM texture features inertia (p=0.001) and correlation (p=0.04) obtained from T2-weighted were significantly different between TZ cancerous and non-cancerous tissue. Moreover, when compared to Gleason score, GLCM energy was lower while GLCM entropy was higher for increased Gleason scores (p<0.05 for all inter-score GLCM feature values).

Discussion

Radiomics and texture analysis have been widely used for diagnosis and prognosis in many different applications. In fact, some of the initial applications of texture analysis date back to 1970s. The most commonly implemented features across all the applications are based on GLCMs. The reason behind the popularity of GLCMs as the texture analysis method of choice is because the GLCM features analyze the inter-voxel relationships in both voxels that are in immediate neighborhood as well as voxels that are far apart. Moreover, GLCMs are easy to implement and the features are more intuitive to interpret. In spite of being the most popular method, the features identified as the most predictive GLCM features are not consistent across different applications. In fact, the most predictive GLCM features are not even consistent within the same organ, same modality and the same diagnostic application. For example, in breast MRI, the texture features identified as the most important for diagnosis were not consistent across different implementations in the literature. Sinha et al⁷⁶ observed the GLCM features of difference entropy, contrast, sum entropy and inverse difference of moments to be the most predictive, Gibbs et al⁸⁵ observed the GLCM features of variance, entropy and sum entropy as the most predictive and Nie et al⁸⁹ observed the GLCM features of entropy, sum average and homogeneity as the most predictive in classifying benign from malignant breast lesions. Moreover, all the studies demonstrated excellent results. The variations in the radiomics results may arise from the image preprocessing steps such as segmentation, image quantization (for statistical features), neighborhood size (for statistical and filtering features) or the modeling steps of feature selection and classification.

The development of a texture feature set that is consistent across different modalities, organs and pathologies may not be possible or practical. This is because different modalities highlight separate characteristics of the tissue of interest. Furthermore, different organs may have distinct inherent texture. However, it is very important to develop a consistent feature subspace for each application even though different applications may have different feature subspaces. Furthermore, consistency in feature subspaces is also required to understand and correlate the texture features with tissue biology. For example, if the entropy values of benign and malignant tumors are significantly different for one region and not for another, it is not possible to establish any correlation between tumor biology and texture values. Thus, extensive research on large patient datasets is required to standardize the radiomics workflow.

Five year view

Radiomics is a relatively new field of research even though it uses established methods for feature extraction. As discussed earlier, extensive research is required to evaluate the preprocessing steps such as quantization for the statistical methods. In the next five years, a consensus will be achieved on the optimal method for data binning as well as feature normalization for the statistical methods. On the other hand, a standard for optimal neighborhood sizes for different image resolutions corresponding to different imaging modalities will be established for filtering techniques. Establishing the standards for these parameters will require extensive theoretical research along with practical validations. Once the standards for feature extraction are established, the next step would include standardization of feature selection and classification methods.

The majority of texture analysis studies have focused on extracting features from a single slice or 3D volume using a specific imaging modality. However, extracting texture features from a combined dataset of multiple images obtained from multiparametric imaging methods (e.g. MRI) or obtained from different modalities (e.g. PET-CT) might be able to better correlate with tissue biology. Representation learning of multispectral datasets has already been achieved in the computer vision community using advanced machine learning methods such as deep learning^185,186. In the next five years, we will also see radiomic feature representations emerge based on these advanced machine learning techniques. In conclusion, as the field of radiomics matures, radiological reports will also evolve into mineable patient data spaces built using imaging biomarkers, finally realizing the goal of precision medicine for providing an improved decision support system.

Key Issues.

• Radiomics is a new field of research and targeted for precision medicine.

• The mathematical basis for radiomics is well grounded in texture, shape, and information theory.

• Development and identification of radio-clinicopathologic features to radiomics features are critical for wide spread use of radiomics in precision medicine.

• Further research is needed to determine the optimal processing steps needed for reproducible application of radiomics to different imaging applications, for example, imaging modality, histogram binning, voxel spacing, size of the image or ROI, and size of the spatial filtering kernel, to name a few.

• Standardization of “relevant” radiomic features for clinical application is needed for improved diagnostic ability; most studies to date have low to moderate AUCs. Thus, every step of the radiomics framework needs extensive analysis, validation and standardization for radiomics to achieve its true potential as a decision support system.

• Advanced data algorithms will be needed to identify the significant features in the high dimensional feature space created by the radiomics method.

• Prospective trials and follow up studies are needed to fully define the impact of radiomics for diagnosis and precision medicine.