Radiomic Analysis: Study Design, Statistical Analysis, and Other Bias Mitigation Strategies

Abstract

Rapid advances in automated methods for extracting large numbers of quantitative features from medical images have led to tremendous growth of publications reporting on radiomic analyses. Translation of these research studies into clinical practice can be hindered by biases introduced during the design, analysis, or reporting of the studies. Herein, the authors review biases, sources of variability, and pitfalls that frequently arise in radiomic research, with an emphasis on study design and statistical analysis considerations. Drawing on existing work in the statistical, radiologic, and machine learning literature, approaches for avoiding these pitfalls are described.

© RSNA, 2022

Summary

This review highlights biases and inappropriate methods used in radiomic research that can lead to erroneous conclusions; addressing these issues will accelerate translation of research to clinical practice and have the potential to positively impact patient care.

Essentials

• ■ Many radiomic research studies are hindered by systematic biases.

• ■ In addition to ongoing initiatives for standardization, improvements in study design, data collection, rigorous statistical analysis, and thorough reporting are needed in radiomic research.

• ■ Insight into potential problems and suggestions for how to circumvent common pitfalls in radiomic studies are provided.

Introduction

Rapid advances in automated methods for extracting large numbers of quantitative features from medical images have led to an explosion of publications exploring combinations of features as imaging biomarkers for diagnosis, clinical prognosis, treatment selection, or other decision support (1,2). Radiomics is a term used to describe both the automatic conversion of medical imaging data to quantifiable features and the quantified features themselves; these features may include well-known imaging descriptors, such as Hounsfield units, or more exploratory features such as gray-level texture or machine learned features. However, a sobering consideration is that only a small fraction of quantitative imaging biomarkers is clinically adopted. Moreover, to our knowledge, there are no radiomic signatures that are identified by way of a high-throughput pipeline in widespread clinical use (3–5). Despite the great promise it holds, radiomics research is susceptible to hidden obstacles (Fig 1).

Figure 1:

Diagram illustrates how the potential of radiomics can be weighed down by multiple sources of bias and variability that are often overlooked and require careful consideration for the field to be successful. Radiomics analysis has shown promise for generating imaging biomarkers, which is well described in the literature.

While considerable progress has been made in radiomic biomarker taxonomy and standardization (6–8), commensurate attention has not been paid to the design and conduct of radiomic studies for imaging biomarker discovery. As a result, many published studies harbor systematic biases or do not include sufficient information for readers to interpret findings in an appropriate context (9). Herein, we discuss study design and statistical analysis considerations for radiomic studies, drawing from our recent experience as collaborating statisticians and computer scientists and as reviewers for journals such as Radiology. It is not our intention to provide an in-depth review of technical radiomic features (eg, gray scale and bin width); we refer readers elsewhere for that (7,10–14). We do not provide a comprehensive list of sources of bias and variability or a ranking of their impact. Instead, our aim is to highlight common pitfalls that we have observed in the design and statistical analysis of radiomic studies and to suggest ways to potentially circumvent them. Our goal is to facilitate high-quality results with the potential for widespread positive impact on patient care.

Study Design Considerations

Radiomic analyses require the availability of patient images acquired as part of clinical practice or a clinical trial. These analyses are susceptible to several different biases (Table 1), because systematic error in the research process can lead to erroneous conclusions. Bias may be minimized by careful study design, calculating sample size to yield sufficient power to detect clinically meaningful differences, and prespecification of hypotheses, research objectives, the nature and sources of variables of interest, potential confounders, and appropriate analysis (Tables 2–4). Best practices for design, conduct, and analysis are well developed for diagnostic, prognostic, and predictive models and are applicable to radiomic studies (15–22). In the paragraphs that follow, we alert readers to design choices that occur frequently and lead to bias.

Table 1:

Frequent Sources of Variability and Bias in Radiomic Analyses

Table 2:

Methods to Prevent Sources of Variability and Bias in Radiomic Analyses

Table 4:

Possible Consequences of Pitfalls in Radiomic Analyses

Table 3:

Examples of Potential Pitfalls in Radiomic Analyses

Definition of Outcome

The primary outcome of interest should be defined at study onset. In analyses evaluating accuracy, the outcome is typically an abnormal condition assessed with a reference standard. In radiomic analyses where there may be no reference standard, the outcome may be the presence of an abnormal condition either at the time of imaging or at a future point, such as with overall or progression-free survival. We use the word “outcome” to refer to both the reference standard in accuracy studies and the condition of interest when there is no reference standard.

Outcome assessment should not rely on information from the modality from which the imaging features are extracted to avoid incorporation bias (23–26). This bias is greatest when the outcome is identified solely from the same image from which the features are measured, and one or more of the features are vital for identifying the condition. However, bias may still be present if a subsequent image from the same modality is used or if the features under study play less of a role in identifying the condition. Intuitively, the magnitude of bias will be related to the magnitude of correlation between the features and the condition.

Dercle et al (27) developed a radiomics signature for predicting overall survival using CT images in 667 patients with colorectal cancer who were enrolled in a clinical trial comparing irinotecan, 5-fluorouracil, and leucovorin (FOLFIRI), alone or in combination with cetuximab. Scans from baseline (pretreatment) and 8 weeks after randomization (on-therapy scans) were divided into training and validation data sets. After comparing the signature with known predictors of sensitivity to FOLFIRI plus cetuximab, KRAS mutation status and the 8-week change in tumor burden, the authors concluded that their radiomic signature outperformed the other known predictors. Notably, if they had used progression-free survival as the outcome rather than overall survival, incorporation bias could have been introduced into the analysis. Progression-free survival is determined from CT scans, as are both the radiomic signature and 8-week change in tumor burden, potentially inducing a correlation between the predictors and the outcome, while the KRAS mutation status predictor would not be subject to the same induced correlation.

Selection of Images for Inclusion

The selection of images to be used, both for training and validating a radiomics model, requires careful consideration.

Some outcomes, such as histologic diagnosis, are only assessed for a subset of patients, based in part on clinical interpretation of imaging results. Limiting the study to these patient images results in verification bias (Tables 1, 2), which is a missing data problem that may lead to estimates of sensitivity that are too high and estimates of specificity that are too low, or in extreme cases, the inability to directly estimate sensitivity and specificity (15,26). Different study designs have been suggested to avoid this bias, and there are several proposed bias-correction methods when verification bias is deemed unavoidable (22,28–31).

Kontos et al (32) analyzed mammograms obtained from routine breast cancer screening using unsupervised clustering to identify four radiomic phenotypes of mammographic parenchymal complexity. To evaluate whether these radiomic phenotypes were associated with improved cancer detection, the authors analyzed scans from a separate case-control study that included women at high risk of breast cancer who were diagnosed with breast cancer. If the case-control study included only women for whom a biopsy result was available, their results would be subject to verification bias.

Discovery studies exploring new feature combinations often use a case-control design (33). If the included patients (cases) have severe, overt disease or their health conditions are more obvious, or the healthy patients (controls) are atypically healthy, spectrum bias is likely present. Also referred to as case-mix bias, spectrum bias is not limited to case-control studies and can lead to estimates of accuracy metrics that are too high (23). Because Kontos et al (32) defined their cases to be women at high risk of breast cancer (because of BRCA1 or BRCA2 mutation or a history of chest radiation, for instance), their analysis may be subject to spectrum bias, while the clinical trial data reported by Dercle et al (27) are less likely to overestimate performance of radiomics-derived predictors. (Note, however, that Kontos et al avoid possible spectrum bias in defining the radiomic phenotype by performing unsupervised clustering in the routine screening population without regard for outcome availability.)

We make the distinction here between spectrum bias and spectrum effect, where the latter is defined as the variation in performance in different populations (34). While not necessarily a bias in the statistical sense (where an estimate is biased if its expected value does not match the true value of the corresponding parameter), it is an important aspect of study design to consider, especially in the context of lack of diversity in medical research (35–38). As has been pointed out, analyses of data sets restricted to patients from a single institution may not be generalizable (32,39). Spectrum effects may be related to catchment populations and institutional procedures for treatment and supportive care; radiomics studies may also demonstrate spectrum effects introduced by differences in scanner hardware, scanning protocols, and image analysis protocols (Tables 1, 2).

Finally, when more than one image per patient is available for analysis, the study design should take into account that these images are not completely independent, and, therefore, do not increase the sample size and study power in the same way that adding completely independent images (patients) would. Specialized statistical analysis methods are required to account for within-patient correlation introduced by assessment of patients at multiple time points or with multiple lesions (40–42).

Training and Validation Data Sets

Appropriate division of data for training and validation of models according to radiomic features is needed to avoid optimistic performance bias (Tables 1, 2) (43). There are different ways this division is handled in practice. For instance, Eslami et al (39) divided CT scans from 624 participants in the Framingham Heart Study into training and validation data sets to develop a radiomic-based risk score characterizing coronary artery calcium. Using a composite outcome consisting of all-cause mortality, nonfatal ischemic stroke, or myocardial infarction, the authors found that adding the radiomic-based risk score to a model containing known predictors of cardiovascular events, including a well-established measure of coronary artery calcium, resulted in substantial improvement in identifying high risk individuals.

When designing a radiomics analysis, independent and mutually exclusive data sets should be used as follows:

Training data are used for data exploration, feature selection, hyper-parameter selection, and model development. Training data may be further divided into subsets, such as test data for model discovery and tuning data for model revisions or hyper-parameter selection. Eslami et al (39) used a “discovery” training data set with repeated cross-validation to select parameters in a random forest model.

Validation data are used to assess performance of a model that was “locked down” (no changes in feature selection, data set–specific standardization, or model parameters) using training data. External validation is preferred and uses independent and distinct data (eg, from a different institution). Internal validation uses hold-out data sets from the same source as the training data, although separated from training data by random selection or a different date range (44). When hold-out data sets are impractical due to modest sample size, cross-validation can be used to assess model performance. However, the cross-validation design must be prespecified to avoid selecting a model based on performance of validation with different k-folds. Because feature selection is likely performed within each cross-validation step, another design prespecification should outline how to select final model features when they are not consistently chosen in each cross-validation subset (45).

Image Analysis Considerations

All biomarkers, whether developed from plasma, tissue, imaging, patient-reported data, or in combination, face potential sources of bias due to assay methodology and analysis. Unique to radiomic analyses are the errors and biases that may arise during image acquisition, processing, and imaging feature quantification (Tables 1, 2) (46–51). Standardization of workflow components (7,52) supports the reproducibility needed to move beyond discovery and into clinical practice and was demonstrated by Eslami et al (39) through their use of PyRadiomics, an open-source radiomics platform that follows suggested feature definitions. However, standardization is not always feasible, most notably in clinical image acquisition (7,50,53,54). Within clinical trials, well-defined patient cohorts and a standardized protocol permit precise biomarker development and exploration of causation. Translation to clinical practice will require thorough testing to identify and account for spectrum effects and to ensure that radiomics-derivedmeasures are robust to data acquisition protocols. These efforts will additionally require broad access to large quantities of images available from real-world data, with clinical annotation.

Statistical Analysis Considerations

Statistical analysis of radiomic features often involves multiple steps, including one or more of the following: dimension reduction, feature selection, model building (or classification), selection of a risk-stratification threshold, fine-tuning of model components, internal validation, and external validation (55–58); bias may arise at multiple points in this process (Tables 1–4).

Exclusion of Indeterminate or Missing Data

Exclusion of missing outcome or feature data from analysis can lead to bias, especially when the reason the feature could not be quantified or extracted is directly related to the feature or the outcome (15,59). All three of the aforementioned examples (Dercle et al [27], Kontos et al [32], and Eslami et al [39]) excluded poor-quality images or images with artifacts and are, therefore, susceptible to resultant bias. This type of image exclusion is very common practice across radiomic studies and is an inherent challenge when quantitatively assessing images. Additionally, the impact of image quality on radiomic features is an active and important area of study (60–63).

Overfitting

Overfitting refers to the situation when a model or classifier is highly optimized for a particular data set, consequently captures noise, and then fails to work well in other data sets either by over- or underestimating risk of the patient outcome (64,65). This results in poor model performance metrics, such as low values for the area under the receiver operating characteristic curve. Despite a high level of awareness in the radiomics literature (1), bias due to overfitting is still commonly observed. Overfitting is most likely to occur when the study sample size is small relative to the number of imaging features evaluated and can also occur in cases where only a small number of imaging features are included, particularly if they are only weakly associated with the outcome (64,66). In all three of our stated examples (27,32,39), the authors aimed to minimize overfitting by including a relatively large number of patients and reducing the number of features that were used.

Multiple Testing

In radiomics, multiple testing is widely recognized as problematic and occurs when many radiomic features are examined without prespecified hypotheses or a method for minimizing false discovery (67–69). Hidden multiple testing may arise at any of several analysis steps. For example, testing multiple optimal cutoffs for clinical decision making is also multiple testing and leads to an increased chance of erroneously finding an association (inflated type I error) as well as overestimation of effect sizes (67,70). Additionally, when there are multiple candidate methods for a step (ie, dimension reduction) and none has been shown to be uniformly superior to others (55–57,71), exploration of several methods may lead to selecting a method based on (spurious) fluctuations in performance in the study sample. As for cross-validation, the methodology should be prespecified based on careful consideration of study aims, data characteristics inherent to the study design, and consequences of false-positive and false-negative errors.

Reporting Considerations

It is not always possible to safeguard against all potential sources of study bias in radiomics research. Therefore, it is imperative that researchers thoroughly report on their imaging data (ie, Digital Imaging and Communications in Medicine [DICOM] header information), methodology, limitations, and any other potential sources of variability. Rigorous reporting enables researchers to build on others’ results and protects against failed attempts to replicate spurious and overstated results. For instance, Eslami et al (39) included a detailed description of their methodology in their supplementary material.

Reporting guidelines developed for other research emphases, such as the Standards for Reporting of Diagnostic Accuracy (STARD) initiative for diagnostic accuracy studies (16), the transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD) statement for prediction models (72), and the Reporting Recommendations for Tumor Marker Prognostic Studies (REMARK) for tumor markers (73), are required by some journals for relevant articles and have many elements that are applicable to radiomic research (74). There has been some previous work aiming to establish parallel initiatives in radiomics (46,47,75). It will be key to consider aspects of study design, data collection, and rigorous statistical analysis moving forward.

Discussion

Radiomic analyses are highly susceptible to bias arising from multiple sources. A unifying theme behind the biases and pitfalls we have outlined is that they can all lead to incorrect inference and a model that erroneously includes or excludes imaging features and, ultimately, performs poorly. While not meant to be an all-encompassing list, the issues we have highlighted arise frequently. Some, such as overfitting and lack of adjusting for multiple testing, are particularly relevant in radiomic studies. Others are issues that may arise equally as frequently in other types of studies but have been highlighted here because we have noticed a lack of awareness of these issues among investigators conducting radiomic studies. All are issues that are broadly applicable to many studies, including those where features are derived by the computer using convolution neural network (deep) approaches. In any analysis, the challenge is to identify the most relevant sources of bias and measurement error.

Although software packages to implement analyses are readily available and increasingly user friendly, if they are not implemented with the necessary expertise or correct guidance, there is a high risk that incorrect conclusions will be drawn from the work. The field of radiomics lies at the intersection of medicine, computer science, and statistics (Fig 2). We contend that to produce clinically meaningful results that positively impact patient care and minimize biases and pitfalls, radiomic analysis requires a multidisciplinary approach with a research team that includes individuals with multiple areas of expertise.

Figure 2:

Diagram shows that the expertise required to ensure meaningful results from radiomic analysis spans a variety of disciplines. Collaboration and knowledge sharing is essential.