NCTN Assessment on Current Applications of Radiomics in Oncology

Abstract

Radiomics is a fast-growing research area based on converting standard-of-care imaging into quantitative minable data and building subsequent predictive models to personalize treatment. It has been proposed as a study objective in clinical trial concepts and a potential biomarker for stratifying patients across interventional treatment arms. In recognizing the growing importance of radiomics in oncology a group of medical physicists and clinicians from NRG Oncology reviewed the current status of the field and identified critical issues, providing general assessment and early recommendations for incorporation in oncology studies.

Radiomics and Applications

Radiomics introduction

The concept of relating imaging information to predicting prognosis and therapeutic response traces its root to the early days of robotics and computer vision in the 1960s, but its systemic application to quantitative imaging analysis dates to the beginning of the 1980s in areas such as computer-aided detection or diagnosis (CAD) ¹. The application of this approach to biological markers and therapeutic endpoints only started in the past decade, when the concept of personalized medicine arose following the increasing use of genomics. Some early examples include the investigation of correlations between hepatocellular carcinoma imaging phenotypes with gene expression ², and between PET-based features and radiotherapy response ³.

When initial studies to investigate whether MRI-based measurements of breast cancer volume could accurately assess the response to neoadjuvant chemotherapy proved promising ⁴, they were used as a springboard for a series of American College of Radiology Imaging Network (ACRIN) trials. The ACRIN trials investigated using serial MRI studies to predict therapeutic response to chemotherapy; this was called the ‘I-SPY TRIAL’ for ‘Investigation of Serial Studies to Predict Your Therapeutic Response with Imaging And moLecular Analysis’ ⁵.

Since 2010, this field has been formalized with the term “radiomics”⁶. The term originates from the word “radio”, which refers to radiology, the science of acquiring medical images through the use of radiation (e.g., CT, PET, MRI). The suffix “omics” follows from the wholesome notion, which was firstly used in the term genomics to indicate the whole mapping of human genetics⁷. Currently, this process of extraction of massive quantitative information from anatomical/molecular images and relating them to corresponding biological information and clinical endpoints is an emerging field referred to as ‘radiomics’⁸.

Radiomics applications

With the recent advances of imaging techniques, imaging has extended its role to the whole spectrum of cancer management, from detection and diagnosis, to treatment response monitoring, and further risk surveillance. The clinical application of radiomics is expected to play important roles in every aspect of cancer management.

Tumor detection and diagnosis

One of the earliest applications for radiomics-driven method is in tumor detection, with the greatest success in lung and breast imaging. US Food and Drug Administration (FDA)-approved systems are currently being used in the clinic. A recent study utilizing commercially-available clinical tools identified many lung cancers that were initially missed by radiologists in the International Early Lung Cancer Action Program (I-ELCAP) trial ⁹.

Currently the trend moves to a direction called discovery radiomics ^10,11. Instead of using pre-defined radiomics feature set, the image data are directly fed into the discovery engine, where a customized radiomics sequencer is constructed using deep learning architecture such as convolutional neural network (CNN), and then the descriptive radiomics sequencer can be applied to identify normal or abnormal tissues. Besides tumor detection, radiomics features have also been demonstrated helpful to identify various types of lesions. For example, Li et al. suggested that mammographic images contain computer-extractable information, which may distinguish between BRCA1/2 mutation carriers and non-carriers ¹². Grimm et al. also observed that imaging features from DCE-MRI were strongly associated with Luminal A and Luminal B hormonal receptor positive molecular subtypes ¹³. Overall, early detection and identification of tumors could be useful for better stratification of patients and identification of subsequent treatment options.

Treatment outcome prediction for decision making support

Recently, significant interest in utilizing radiomics for early prediction of treatment response has emerged. In predicting pathological complete response (pCR) following neoadjuvant chemoradiation for locally advanced rectal cancer, Nie et al. showed improved prognostic values could be achieved using a voxelized radiomics analysis approach over conventional imaging metrics ¹⁴. Zhang et al., identified MRI-based radiomics as prognostic factor for progression-free survival in patients with nasopharyngeal carcinoma (NPC). The prediction power significantly outweighed than traditional TNM (tumor-node-metastasis) staging ¹⁵. Although the TNM staging is the cornerstone for treatment decision making. It is typically assessed based on gross anatomy information, not reflecting the intra-tumor heterogeneity. While the radiomics approach can characterize the intra-tumor heterogeneity noninvasively thus can add incremental value to the clinical information in assessing the treatment outcome.

Radiomics features from CT, PET/CT and CBCT have also shown predictive value for response to treatment ^16–24. Investigators analyzed the daily non-contrast CTs, acquired during routine IGRT using an in-room CT, from patients with head and neck ²⁵, lung ²⁶ and pancreatic cancers ^27,40. They reported that radiation can induce patient specific changes in CT texture features, and that these changes can be detected in the early phase of radiation therapy. Ohri et al., showed pre-treatment metabolic tumor volume (MTV) and heterogeneity textural metrics on PET/CT can be good prognostic factors for locally advanced non-small cell lung cancer (NSCLC) patients treated with chemo-radiotherapy based on data from ACRIN 6668/RTOG 0235 ²⁸. Buizza et al., in addition, showed the longitudinal temporal and spatial changes from PET/CT image could improve the early survival prediction for chemoradiation treatment²⁹. Radiomics has also been used to predict radiation induced normal tissue toxicities, such as radiation pneumonitis ²¹ or xerostomia ^30,31. These results suggest that a radiomics-based signature may emerge as an accepted imaging biomarker for predicting therapeutic outcome and for improving decision support in cancer treatment.

Risk assessment

Radiomics has also been extended to risk surveillance in several cancers. Liu et al., investigated the association between imaging features and low-grade gliomas (LGG) related epilepsy, and proposed a radiomics-based model for the prediction of associated risk³². Similarly, it is well known that mammographic density is an independent risk factor, and radiomics may provide much more information than breast density. ^33–35. Li et al. investigated breast parenchymal patterns in mammographic images in 456 patients (53 with BRCA1/2 gene carrier, 75 with unilateral cancer and 328 with low-risk) ¹². They demonstrated that women at high risk tend to have dense breasts with coarse and low-contrast texture patterns. Haberle et al. performed a case-control study with 864 cases vs. 418 controls ³⁶. Of the 470 radiomics features explored, 46 remained in the final risk model; the radiomics model outperformed than the conventional risk model with mammographic density. These studies may promote future breast cancer prevention trials to investigate the role of radiomics to measure breast tissue composition in individual woman for personalized risk management.

Radiogenomics

In radiogenomics, the radiomics phenotype is correlated with a genomic profile. The hypothesis is that imaging may provide insight into tumor phenotypes which are driven by the heterogeneity of the genetic evolution. Radiogenomics is a very young field due to the lack of data consisting of both imaging and genomic measurements on the same set of tumors ^37,38.

Recent studies in brain tumors ^37,39,40, lung cancer ^40–42 and breast cancer ^13,43 suggest value for radiogenomics. National shared databases, such as the Cancer Imaging Archive (TCIA) and the Cancer Genome Atlas (TCGA), provide researchers opportunities to explore this field. Using MRI images from TCIA and clinical, histopathologic, and genomic data from TCGA, Li et al. investigated the relationship between MRI imaging phenotypes and multi-gene assays including Oncotype Dx, MammoPrint and PAM50 ⁴³. Multiple linear regression analyses demonstrated significant associations between radiomics signatures and multigene assay recurrence scores. Zinn et al. identified an association between high T2 FLAIR volumes, upregulation of periostin (POSTN), and downregulation of miR-219 using data from TCGA³⁹. They noted high levels of POSTN were associated with mesenchymal tumors and shorter survival and further concluded that this approach may be valuable for identifying new targets for molecular inhibition or future therapies.

Although numerous radiomics features can be extracted from medical images, the method by which tumor pathophysiological processes give rise to imaging phenotypes remain unclear. More studies are required to confirm these associations to further elucidate the biological meaning of the radiomics features.

Radiomics Processes and Components

One goal of radiomics is to convert images into data that can be mined using high throughput computing. The process follows several general steps, as shown in Figure 1: (a) image acquisition, (b) region of interest (ROI) identification and segmentation, (c) quantitative image feature extraction, (d) data mining and informatics analysis.

Figure 1.

The general radiomics study workflow. Step I: Image acquisition; step II: region of interest identification and segmentation; step III: quantitative image feature extraction and Step IV: data mining and informatics analysis.

Image acquisition

The first step of image-based radiomics phenotyping involves image acquisition. Different image acquisitions will provide different, and often complementary information. For example, size or volume-based analysis can be obtained using anatomical MRI or CT. Measurements of perfusion can be determined from a series of dynamic-contrast-enhanced (DCE) MRI or CT acquisitions. Functional MRI, such as diffusion-weighted-imaging (DWI) can be used for tissue microcirculation and cellularity evaluation ¹⁴. Metabolic changes, such as rate of glucose metabolism, can be measured using fluorodeoxyglucose (FDG)-PET ⁴⁴. Emerging functional and molecular imaging methods that are increasingly used in clinical trials may offer additional biomarkers. Human experts can help guide the choice of the imaging modalities tailored to the disease site of interest, clinical endpoints and potential treatment options.

Historically, imaging devices are designed for subjective interpretation of images, allowing clinicians to identify, for example, the presence and the location of a lesion. Subsequent technical innovation has largely focused on improving imaging quality, shortening scanning time, or integrating with treatment machines. Conversely, they are not primarily to provide quantitative measurements in a reproducible manner. The standardization of imaging acquisition protocols is typically lacking, and wide variations in reconstruction or acquisition parameters can exist. Zhao et al. studied repeat computed tomography (CT) data set from lung cancer patients and concluded that smooth and sharp reconstruction algorithms should not be used interchangeably ⁴⁵. Galavis et al. assessed the variability of radiomics features extracted from PET due to different acquisition modes, reconstruction algorithms, post-filtering, and iteration numbers. 40 out of 50 features were shown to have substantial variability, up to 30%⁴⁶. Due to the gradient strength of the scanner, pulse sequence used, method of contrast agent administration, k-space trajectory sampling, and other factors, results from MRI can vary more significantly ^47–49. Because the quality of the radiomics data depends on the reliability of the acquisition protocols used in clinical centers, the impact of these variations on the stability of radiomics features needs to be thoroughly investigated and understood in future studies.

Region of interest (ROI) identification and segmentation

Defining regions of interest is a fundamental task within the practice of oncology. In radiology, human experts identify the presence, location and size of the suspicious areas, for diagnosis, staging or response assessment. In radiation oncology, the human experts must identify the tumor extent for treatment and organ-at-risks (OARs) for radiation sparing. Manual outlining by experienced radiologists or radiation oncologists is often treated as the gold standard. Yet it is labor intensive and suffers from high inter-operator or even intra-operator variabilities. ROIs may be contoured more consistently using semi- or fully automated methods such as thresholding, region growing, classifiers, clustering, Markov random field models, artificial neural networks, deformable models and atlas-guided approaches ^50,51. Although full automation may present a new opportunity for standardized segmentation method, challenges related to complex anatomy or areas with low soft tissue contrast persist, and manual correction of contours by an experienced physician is often required. A rather new idea to avoid segmentation pitfalls is the use of a “digital biopsy”, which samples rather than segmenting the region of interest. This approach has been recently applied to sampling CT lung nodules ⁵². More recently, advanced machine learning based algorithms has been applied for image segmentation or sampling ^53,54. There are also several large initiatives aimed at developing automatic segmentation solutions using deep learning. These include Google’s DeepMind ⁵⁵, Microsoft’s InnerEye ⁵⁶, Mirada’s DLCExpert ⁵⁷, and the Grand Challenges in Biomedical Image Analysis ⁵⁸. These automated segmentation tools have been shown to improve efficiency of structure set generation, particularly for organs-at-risk (OARs). In the near future, deep-learning-based segmentation tools may be robust enough for routine radiomics applications.

Radiomics features

The core of radiomics is the extraction of high-dimensional feature sets to quantify images. The features extracted from images can be divided into static (i.e., a snapshot of enhancement at one point in time) or dynamic (i.e., time variant) features according to the acquisition protocol used at the time of scanning ⁵⁹.

Static image features

Several types of static image features can be applied to radiomics studies, including: (a) morphology-based features which are used to capture three-dimensional (3D) shape characteristics, such as volume and surface area, and sphericity which quantifies how close a 3D volume is to a sphere ⁶⁰. A higher sphericity indicates a round shape while a lower value indicates an irregular or elongated shape, (b) intensity-based features which are used to quantify the gray-level distribution inside the ROI. Examples of first-order intensity-based features include mean, standard deviation, percentiles, kurtosis, and skewness. They are used to characterize the overall intensity variability. While second-order intensity-based features, also referred to as texture features, look into the local distribution. Example metrics include the gray-level co-occurrence matrices (GLCM) ⁶¹, gray-level run length matrices (GLRLM) ⁶², and gray-level size zone matrices (GLSZM) ⁶³. GLCM gives the probability of observing a pair of values in voxels at a given distance in a given direction ⁶¹. GLRLM measures the number of consecutive voxels with the same value aligned in a given direction ⁶² and GLSZM reflects the number of neighboring voxels with the same value ⁶³. Higher-order intensity values may be achieved using image transformations (e.g., Laplacian or Gaussian) ^64,65 with different filter grids (e.g. Laws’ filters) ⁶⁶, which highlight edge structures, or by using wavelet composition ⁶⁷, which characterize sharp transitions in the intensity frequency spectrum.

Dynamic image features

In order to quantify the dynamic behavior of a contrast agent or other tracer within a region (which can be one or more voxels), pharmacokinetic modeling is typically used. In general, pharmacokinetic model considers tissue concentration as a convolution between the arterial input function (AIF) and the residual function for the decay of contrast agent inside the region of interest. The intravascular and interstitial space can be modeled under different assumptions. The most widely used kinetic model, Toft’s model, for example, assumes instantaneous mixing of contrast upon the arrival in the intravascular and interstitial space, while the extended Toft’s model considers a delay effect of the tissue concentration transferred from the artery ⁶⁸. The adiabatic tissue homogeneity model (AATH) is motivated by the fact that the concentration of contrast agent in the extravascular distribution volume changes slowly relative to that in the intravascular space ⁶⁹. Thus, it assumes that there exists a finite transit time for contrast solutes to travel from the arterial to the venous phase. The Patlak model is a linearization of irreversible compartment models in an equilibrium state where the tracers flow into the tissue without leaving ⁷⁰. The Logan model is also a linearization of reversible compartment model in the equilibrium states where the tracers can move freely back into the plasma ⁷⁰. Another approach is to directly fit the residual function using deconvolution without making any additional model assumptions ⁷¹. Typically-derived dynamic image features include regional blood flow (rBF), regional blood volume (rBV), mean transit time (MTT), extraction fraction, permeability surface area product, and most frequently volume transfer constant (Ktrans), and extravascular extracellular volume (v_e). With an increasing emphasis on imaging of the tumor microenvironment, DCE-CT/MRI and FDG-PET have evolved as important functional techniques in this setting.

Overall, the current radiomics pipeline typically incorporates thousands of extracted radiomics features and these are still expected to further widen as the field continues to evolve.

Analytical tools

As in many other “-omics” fields, the number of input variables often far exceeds the number of patients. To reduce the probability of false positives, feature selection as dimension reduction is often needed, and filter-based score-ranking approaches, such as Wilcoxon, Chi-square, principle-component-analysis (PCA) are typically used ⁷². This can be carried out either using univariate methods, as the scoring criterion only depends on the feature relevancy, or multivariate methods using a weighted sum to maximize relevancy and minimize redundancy ^72,73. Feature selection can also be combined with feature classification into a single model; examples include least-absolute-shrinkage-and-selection-operator (LASSO) ⁷⁴ and Elastic Net ⁷⁵.

Once a feature set is obtained, a data-driven model can be constructed. These include supervised and non-supervised approaches ⁷⁶. Unsupervised analysis does not provide an outcome variable, but summary information of the data. The most frequently used graphical display is a heat map, which simultaneously reveals cluster structures in a data matrix ⁷⁷. Supervised analysis, in contrast, creates models that attempt to separate the data with respect to a treatment outcome, such as responders vs. non-responders. Typical classification methods include conventional logistic regression or more advanced machine learning techniques ^78,79.

Outcome modeling by logistic regression

Logistic regression is a common tool for multi-metric modeling. A logit transformation is used:

$$f(x_i)=\frac{e^{g(x_i)}}{1+e^{g(x_i)}},i=1,\dots ,n,$$ (a)

where n is the number of cases (patients), x_i is a vector of the input variable values (i.e.,image features) used to predict f (x_i ) for outcome y_i (e.g., tumor control or toxicity) of the i_th patient,

$$g(x_i)={\beta }_0+\sum _{j=1}^d{\beta }_j{x}_{ij}i=1,\dots ,n,j=1,\dots ,d,$$ (b)

where d is the number of model variables and the β’s are the set of model coefficients determined by maximizing the probability that the data gave rise to the observations. This gives a linear combination of selected features with coefficients of respective weightings to the outcome, providing an intuitive tool for clinicians to interpret the associations between selected variables and the outcome.

Outcome modeling by machine learning

There are a wide class of AI techniques (e.g., neural networks, decision trees, support vector machine), which are able to provide a non-linear association of input variables to the outcome ^80,81. Indeed, prognostic biomarkers developed using these machine-learning methods have increased performance when compared with conventional statistical methods ^78,82,83. Recently, deep learning algorithms such as convolutional neural networks (CNN) have achieved breakthrough prediction power in a variety of medical studies, including detection of lung nodules on CT ^84,85,86, and detection of breast cancer on mammogram ^87,88. A comparison in mortality prediction from chest CT between a deep-learning framework and a standard framework with radiomics features, showed increased accuracy with CNN based classification ⁸⁹. It is anticipated that multi-task learning will help to provide a degree of interpretation for deep learning approaches ^76,81–90. Given enough high-quality data (text and images), it is expected that the role of CNNs will continue to expand in medicine and quantitative imaging. Despite these advances, however, concerted efforts are needed to promote detailed understanding of these approaches, including the relationship between dataset sizes, possible confounders and performance of outcome prediction.

Quality Assurance of the Images and Methodologies

Despite the promise that radiomics may hold for precision medicine, there are significant concerns regarding the lack of reproducibility in results within and across modalities and among multiple institutions. In this section, the challenges associated with the clinical translation of radiomics are highlighted, and recommendations are provided for application in NCTN clinical trials.

Standardization of image acquisition parameters

In recent years, the field has strived to improve standardization by defining standard acquisition protocols. National efforts have been led by the Quantitative Imaging Network (QIN) initiated by the National Cancer Institute (NCI) ^91,92, by Radiological Society of North of America (RSNA) Quantitative Imaging Biomarker Alliance (QIBA) ^93,94, and others ^95–96. The NCI-Quantitative Image Excellence (CQIE) project was initiated in 2010 and the NCTN is a key focus for this effort ^96,97. The CQIE provides PET/CT and MRI phantoms and protocols for site qualification, while QIBA provides consensus ‘profiles’ on the measurement accuracy of quantitative imaging biomarkers and the requirement/procedures needed to achieve this level of accuracy ⁹⁸.

Despite the progress made by these groups, there still no universal acquisition protocols for any imaging modality in clinical practice. For studies involved with radiomics in NTCN trials, therefore, we recommend:

1. a comprehensive description of the image acquisition parameters should be documented, including manufacture, model, types of images (e.g., CT, MRI, contrast-enhanced), contrast agents, image acquisition parameters (e.g., slice-by-slice or three-dimensional acquisition, MRI magnetic field/TR/TE/flip angle, CT tube current/voltage, axial or helical mode), reconstruction package, software version, image resolution, signal-to-noise ratio, management of motion artifact;

2. If clinical trial is being conducted in institutes with the same scanners, the same scanning protocols should be strictly followed. Comparison across institutions with different scanners may be difficult. So it is suggested, when possible, to use each patient as his/her own control and the delta changes instead of the absolute value, or other corrections means for such variability including accounting for contrast;

3. direct measurement from scans with contrast should be used cautiously since uptake and the time from injection to imaging will differ and cause large variability. A practical strategy is to control the normal tissue ROIs as a baseline. For instance, on an individual basis, the average background of parenchyma or muscle can be used to normalize breast scans ^99,100;

4. using radiomics feature(s) that are less dependent to variations in image acquisition protocol and/or platform into the final model.

Standardization of image pre-/post-processing

Following image acquisition, there can still be a large range of voxel intensities and image noise, therefore, filtering procedures may be needed to enhance the signal and reduce the unwanted noise ¹⁰¹. Fave et al. tested the effect of different image pre-processing filters, such as bit-depth resampling and smoothing filters, on radiomics features, concluding that the correlation of extracted radiomics features with clinical outcome changes with different filters ¹⁰². The impact of noise, which directly impacts intensity and GLCM features, has also been studied ¹⁰³. In addition, images obtained on two different scanners may result in different pixel values due to different detector materials, image resolution or acquisition techniques ⁷⁶. A very recent study by Reuze et al., showed GLCM-entropy values were higher on a scanner equipped with time-of-flight capabilities, leading to very different cutoff values for predicting recurrence ⁷⁶. Therefore, some post-image acquisition processing as filtering and normalization is necessary.

To reduce the impact of noise, the SNR from all acquired images should be well-controlled. Some smoothing filtering procedure maybe used so that all images come close to a target spatial resolution value ^104–106. A consequence of this approach is that images with the highest initial resolution will be degraded, which is especially adverse for analysis of image texture. CT images without contrast may be used directly if geometry distortions are calibrated properly and consistency in terms of HU can be obtained. MR images in general are susceptible to different distortions. A typical correction procedure would follow normalization prior to radiomics analysis ¹⁰⁷. PET images tend to have more noise than other imaging modalities used in radiomics analysis, a typical pre-processing procedure to correction procedure has been to use Poisson-to-Gaussian conversion (a root-squared transform) ¹⁰⁸.

Additional procedures include the discretization signal intensities into finite intervals for intensity-based feature analysis, using either absolute (fixed bin size) or relative (fixed number of bins) discretization ¹⁰⁹. The choice of method is important, as the extracted features will vary. It has demonstrated that absolute discretization shows better repeatability, lower sensitivities to changes, and is not volume dependent ^109,76. Thus, absolute discretization with fixed bin size should be adopted for intensity-based radiomics analysis. Each discretization method, however, has advantages and drawbacks and can lead to substantially different results ⁷⁶.

Overall, effects of these pre-/post-processing techniques on variations in radiomics analysis remain as an open area. Although specific recommendations are being worked out by the IBSI group ¹¹⁰ and other, more general pitfalls to avoid including:

• to include a detailed description of filtering technique, noise reduction technique, intensity-correction, intensity discretization and bit-depth resampling, should be provided; and

• to apply radiomics feature(s) that are less dependent to variations in image pre-/post-processing techniques into the final model.

Reproducibility of radiomics features

Differences in segmentation methods are likely to bias the stability of shape metrics and intensity-based features. Parmar et al., performed a stability analysis based on an inter-observer study ¹¹¹. Fifty-six radiomics features, quantifying shape, intensity and texture were extracted from lung CT images. They showed for manual delineation among 5 experienced operators, only 52% of these features had high reproducibility compared to 88% based on semi-automatic segmentation.

Moreover, contours of regions-of-interest (ROIs) are typically stored into two image formats: directly as voxels (e.g., NIfTI), or as (x,y,z) coordinates (e.g., DICOM-RT STRUCT). We need to determine which voxel centers lie within the space enclosed by the contour polygon. Thus different interpolation and partial-volume fraction threshold would also affect the contour-based feature calculation. Researchers showed contour data-rendered volumes exhibited large variations across the commercialized stereotactic radiosurgery (SRS) platforms using both the phantom and patient cases by up to 20% ¹¹².

The methodologies to calculate radiomics feature can vary as well. Radiomics features can be obtained from either a 2D image slice ^113,13 or from a reconstructed 3D volume ¹¹⁴. As another example, GLCM texture analysis can be calculated either by averaging the values of the matrices computed for 13 distinct directions or a single matrix that accounts for tumor co-occurrence information in all 13 directions ^102,115. In addition, features computed from different matrices can have the same name ¹¹⁶. For example, entropy can be computed either from a histogram of intensities or from the GLCM matrix accounting for spatial similarity ¹¹⁷. Thus the impact of different feature identification methods used in radiomics also needs to be carefully studied.

Furthermore, the variability of radiomics indices has been found to be highly feature dependent. Recent studies on mammography datasets reported that robust features were those that described spatial patterns rather than directionality or image intensity¹¹⁸. Deformable registration of CT lung data was shown to alter underlying texture but certain features could still be identified that were robust to registration effects ²¹. Consensus has been reached that first-order and shape features in both CT and CBCT are generally more repeatable than texture or higher-order features ^{119–120,121}. First-order statistically-derived features from a standard uptake value (SUV) histogram are generally robust with respect to segmentation, while texture features consistently showed greater sensitivity to segmentation differences ^111,119,122.

Regarding the computation of radiomics features, we recommend:

1. unambiguous definitions of each radiomics feature should be provided and evaluated;

2. if contouring is involved, describe how ROIs are delineated in the image. Specify if segmentation is performed manually, semi-automatically or automatically, by how many users/experts and how consensus has been formed. The reproducibility of the radiomics features based on segmentations by multiple observers needs to be assessed;

3. the stability and the accuracy of features should be confirmed in terms of calculation algorithms, such as interpolation, criteria to include or exclude voxels from an ROI mask, 2D/3D calculations; or use features that do not require accurate segmentation;

4. additional suggestions include through the use of test-retest scan during the developmental phase (i.e., scanned on the same scanner but repositioned in between scans) with physical phantom and/or patient studies, to compare the means and standard deviations of these results as well as their correlations. Within the radiation oncology work flow, there are instances that could mimic the test-retest scenario. One example is a 4DCT scan if each phase is treated as a separate CT scan. A good practice policy is to eliminate features that prove to be unreliable in the test-retest. To this end, several datasets are public available. Of note, the RIDER dataset allows validation of results in the same set of patients with two scans taken 15mins apart ¹²³. Investigators are encouraged to develop and share additional shared test-retest datasets;

5. again, using radiomics feature(s) that are less dependent to above variabilities to build the final model.

Radiomics phantoms

To help in addressing some of the challenges described above, the design of radiomics specific phantoms is now an active area of research ^124–126. One example is the Credence Cartridge Radiomics (CCR) phantom, which was designed for use in studies of texture feature robustness ¹²⁵. Using the CCR phantom, Mackin et al. investigated the inter-scanner variability of the calculated texture features, looking for clustering effects due to the scanner manufacturer and CT acquisitions parameters ¹²⁷. They subsequently developed a correction technique to reduce or eliminate the variability in radiomics features due to differences in image pixel size based on CT images using the physical CCR phantom ¹²⁸.

Digital phantoms such as the the Zobel and NCAT phantoms have long been used in image processing applications ¹²⁹. It is unclear, however, whether these phantoms are suited for radiomics, and particularly for texture analysis. An alternative approach is to use standardized patterns such as the Brodatz Textures, ¹³⁰ though the accompanying signatures are not necessarily clinically relevant. Alternately, simulated images can be used to determine optimal parameters for feature analysis. McGurk et al. augmented PET images from 30 soft tissue sarcoma patients by varying the extent of axial data combined per slice (‘span’) ¹³¹. Simulated T1-weighted and T2-weighted MR images were acquired by varying the repetition time and echo time in a spin-echo pulse sequence, respectively. The impact of PET and MR image acquisition parameter variation on individual textures was investigated to assess the global response and the predictive properties of a texture-based model. The results suggested that such a process is feasible for identifying an optimal set of image acquisition parameters to improve prediction performance ¹³¹.

Overall, those radiomics phantoms may be helpful in assessing the inter-/intra-scanner variabilities, and thus protocols for regular phantom QA may be worth of being developed, similar to dosimetric study in current radiotherapy QA program, to monitor inter-scan and inter-vendor variability of image-derived features. In addition, phantom studies will also be useful in optimizing imaging protocols and image pre-/post-processing techniques that allow for reliable radiomics characterization. Ultimately, specific acquisition protocols optimized to generate superior radiomics measurements for a given clinical problem should be developed and standardized and thereafter validated using clinical scanners.

Robustness of modeling with radiomics

In the current iteration of radiomics, image features have to be extracted with high throughput, putting a premium on statistical modeling and machine learning algorithm development. Currently there is no consensus on which feature selection or learning methods should be used. Parmar et al. carried out a comparative study using 12 machine-learning algorithms combined with 14 feature selection methods, for radiomics-based prediction of 2-year survival of lung cancer patients treated with radical radiotherapy alone or with chemotherapy ⁷⁸. They concluded the 30% variation was observed for different classification method but not the feature selection algorithm. Given the state of the art, an analysis of the impact of different learning approaches on the stability and robustness of the proposed models is needed.

The general understanding is that simpler models involving few radiomics features are more robust. To avoid over-fitting, a reasonable rule of thumb with feature selection or dimension reduction is that 10 positive samples (patients) are needed for each feature selected for binary classification according to the Harrell’s guidelines ^132,133. For example, radiomics analysis can be performed for 100 positive events, which will result no more than 10 features selected for the final prediction model.

Recently, deep learning has emerged as a productive force across many healthcare disciplines, especially in diagnostic radiology and pathology. However, there is a significant mismatch between the perceived capabilities of AI compared to their actual capabilities in present imaging studies ^134,135. The scarce availability of high quality data, as well as the lack of standardized processes among institutions are the key-barriers. In addition, interpretation of non-linear relationship between input variables vs. outcome, as given by many current “black box” machine learning models, requires high-level expertise in this field. Additional complementary information such as patient demographics, genomics, histology, and biomarkers may be helpful in interpreting the results.

Nevertheless, robust models:

1. need to assess the over-fitting risk by using cross-validation methods during the classification step and controlled by dimension reduction methods.

2. need to be built upon rigorous training, testing and validation. Estimation of predictive performance in single-institution cohorts should include multiple-folded repeated cross validation to minimize the risk of overfitting, while validation with external dataset is highly recommended; and

3. may need to consider to accommodate clinical information, and with covariates of genomic profiles, histology, biomarkers, patient histories etc., to generate clinically understandable and acceptable decisions.

Overview of Commercial and Open-Source Radiomics Systems

Commercial systems

Several commercial applications providing radiomics capabilities applicable to cancer clinical trials have been created, including HealthMyne.com ¹³⁶, Texrad.com ¹³⁷, and Oncoradiomics.com ¹³⁸. The commercial entities offer platforms that automate image analysis and clinical interpretation for radiologists and oncologists. Radiomic feature extraction is included as part of image analysis functionality, and intensity, shape and texture metrics, along with conventional size measurements used in routine image interpretation as RECIST (Response Evaluation Criteria In Solid Tumors), therapy response monitoring, and cancer screening, are generated on approval of structure segmentation. Along with imaging data (CT, MR and PET), these systems can import and display electronic health record (EHR) information via an HL7 interface, and imported and generated data is stored in a minable patient database that can be examined.

Open source systems

There are several open source software packages capable of performing radiomics analysis. Examples include: IBEX ¹³⁹, MaZda ¹⁴⁰, CGITA ¹⁴¹, pyradiomics ^142,143, CERR¹⁴⁴. Because there is no official index of the various open source radiomics packages, this list is not necessarily exhaustive. These open-source packages are typically capable of calculating first-order texture, GLCM, and GLRLM features, but not necessarily more sophisticated ones such as fractal features. Certain packages can calculate texture features in 2D and 3D (IBEX, CGITA, pyradiomics) while others are limited to 2D datasets (MaZda). Furthermore, the ability to alter calculation parameters, such as the number of gray-levels and directions for GLCMs, is not available in all open-source packages (e.g., MaZda). Some packages were tailored for certain imaging modalities. For example, IBEX was initially developed for CT data whereas MaZda was initially developed for MRI studies. As a result, both IBEX and MaZda rescale pixel values in an image to remove any non-negative pixel values ¹⁴⁵. Algorithmic implementation of features in each package could also vary. Some efforts have been made, such as the image biomarker standardization initiative (IBSI) ¹¹⁰, to standardize feature definitions. Some packages such as pyradiomics attempt to adhere to IBSI definitions. Beyond open-source software tools, a number of groups have also developed in-house tools for radiomics analyses. One such example is an open source Matlab that calculates different texture matrices in addition to global metrics. It also supports multivariate model building using a logistic regression model with bootstrapping for order selection ¹⁴⁶. Overall, when selecting an open-source package, the user should verify that it meets their task-specific need and provides expected results.

It is recommended that:

1. investigators provide access to code and datasets to ensure results can be reproduced. The datasets should include images, segmentations and clinical information if available and compliant with HIPAA regulations.

2. investigators promote an open source approach to radiomics software as an important step toward independent validation and to general dissemination of the approach.

Implementation in Clinical Trials and Recommendations

In order for radiomics to gain broad acceptance in clinical medicine, the value that the technology brings to clinical assessment and decision making must be affirmed in the context of clinical trials. Ideally, clinical trials that are conducted across diverse institutional environments and that are agnostic to vendor or technological platform will provide the most robust validation of the significance of radiomics. High quality clinical research is resource intensive and consumes time, energy and resources of physicians, research staff, and patients alike. The recently published roadmap for imaging biomarkers is a notable advancement, showcasing key recommendations for clinical translation of radiomics ¹⁴⁷. These considerations being applied to the role of genomics in clinical trials may also apply to radiomics. Due to the expense of clinical trials, it is crucial that any prospective trials involving radiomics are carefully designed.

1. An important clinical question is what type of studies will be best suitable to test the value of radiomics. Strategies must be defined, mitigate and/or quantify uncertainties, risk and cost associated with any potential biomarker in making research or clinical.

2. Trials need to be carefully designed considering both the initial and any secondary analyses to ensure that the proper data has been collected to support the study questions. Whether there is and how much value of radiomics would add into the current gold standard clinical measurement needs to be verified. It is also prudent to incorporate radiomics studies as exploratory aims in the initial phase of the implementation into clinical trials. It should be clearly specified whether radiomics input will be used to stratify or determine patient eligibility or integrated within the trial to assess the quality of the test.

3. For clinical trials, a standard radiomics data format, similar to the needs defined by the Advanced Technology Consortium to support clinical trials, one of the main proponents for the DICOM-RT standard, may be necessary. Such a standard would need to be developed by researchers, clinicians, and manufacturers together to address the competing needs for robust quantitative evaluation of data. In addition, a guideline for reporting results from radiomics studies should established. A concept of radiomics quality score (RQS) has been proposed as possible evaluation criteria for radiomics studies ¹⁴⁸. The RQS contains sixteen key components that intend to minimize bias and enhance the usefulness of the radiomics models. These recommendations may establish a reporting guidelines for future radiomics studies.

4. For a given trial, being able to share a standard acquisition protocol in support of the trial across participating institutions would establish a strong foundation for meaningful radiomics data and analysis.

Summary

Radiomics is an evolving field that is growing at a rapid pace with great potential to impact the design of future clinical trials in oncology. As was noted in a 2009 report, however, many genomics-based studies have been published that contained significant analytical errors ¹¹⁰, and scientists reported that they failed to replicate 47 out of 53 landmark studies ¹⁴⁹. These findings raise similar concerns regarding the accountability of the statistical analysis, the transparency of the raw data accessibility and validation of results in radiomics studies. As the ultimate goal of radiomics is to build reliable imaging biomarkers to assist clinical decision-making, a prospective investigation must be trained, tested and validated against a completely independent data set, with a systematic study design with uniform treatment delivery, complete reporting of results, and robust statistical analysis. Protocols for standardizing image acquisition, feature extraction, and analysis will help to streamline the process for clinical trials. This can build on already existing expertise in the community in credentialing and evaluation of other metrics (e.g., treatment plan delivery, imaging QA,) to accelerate the process.

Nevertheless, the promise of radiomics is still quite positive. Medical imaging is redefining its role as a valuable data source for precision medicine in the guise of image-based phenotyping. Further, it could characterize tumor heterogeneity at a macroscopic level, a critical limitation of biopsy-based or current genomic approaches, which could potentially provide important complementary information for precision medicine. The successful introduction of these methods into clinical care will require much additional research to determine how underlying driving biologic phenomena are related to the tumor imaging phenotypes. In the foreseeable future, we expect that the data gleaned from oncology examinations will be converted into quantitative data which will be interfaced with knowledge databases to improve the diagnostic and prognostic power for clinical decision support.