Quality Reporting of Radiomics Analysis in Mild Cognitive Impairment and Alzheimer's Disease: A Roadmap for Moving Forward

Abstract

Objective

To evaluate radiomics analysis in studies on mild cognitive impairment (MCI) and Alzheimer's disease (AD) using a radiomics quality score (RQS) system to establish a roadmap for further improvement in clinical use.

Materials and Methods

PubMed MEDLINE and EMBASE were searched using the terms ‘cognitive impairment’ or ‘Alzheimer’ or ‘dementia’ and ‘radiomic’ or ‘texture’ or ‘radiogenomic’ for articles published until March 2020. From 258 articles, 26 relevant original research articles were selected. Two neuroradiologists assessed the quality of the methodology according to the RQS. Adherence rates for the following six key domains were evaluated: image protocol and reproducibility, feature reduction and validation, biologic/clinical utility, performance index, high level of evidence, and open science.

Results

The hippocampus was the most frequently analyzed (46.2%) anatomical structure. Of the 26 studies, 16 (61.5%) used an open source database (14 from Alzheimer's Disease Neuroimaging Initiative and 2 from Open Access Series of Imaging Studies). The mean RQS was 3.6 out of 36 (9.9%), and the basic adherence rate was 27.6%. Only one study (3.8%) performed external validation. The adherence rate was relatively high for reporting the imaging protocol (96.2%), multiple segmentation (76.9%), discrimination statistics (69.2%), and open science and data (65.4%) but low for conducting test-retest analysis (7.7%) and biologic correlation (3.8%). None of the studies stated potential clinical utility, conducted a phantom study, performed cut-off analysis or calibration statistics, was a prospective study, or conducted cost-effectiveness analysis, resulting in a low level of evidence.

Conclusion

The quality of radiomics reporting in MCI and AD studies is suboptimal. Validation is necessary using external dataset, and improvements need to be made to feature reproducibility, feature selection, clinical utility, model performance index, and pursuits of a higher level of evidence.

INTRODUCTION

Alzheimer's disease (AD) is a progressive neurodegenerative disease and is the most leading cause of dementia (1). Mild cognitive impairment (MCI) is often considered a prodromal stage of AD, but patients with MCI are heterogeneous with different rates of progression toward AD (2). The criteria of the National Institute of Neurological Disorders and Stroke-Alzheimer Disease and Related Disorders (3) and the National Institute on Aging and Alzheimer's Association guideline (4) have highlighted the use of neuroimaging for the diagnosis and prognosis of AD. However, traditional imaging biomarkers from structural MRI such as atrophy have limited value because they are not specific for neurodegeneration due to AD, and atrophy occurs as a later event in AD progression (5,6).

Radiomics is the process of converting image data to mineable data for extraction of quantitative radiomics phenotypes using data characterization algorithms. The underlying hypothesis for radiomics is that these features (i.e., shape, first-order, and second-order [texture] features) have the potential to discover hidden information that is inaccessible with single-parameter approaches such as volume, and may reflect genomic, cellular, and metabolic information (7). Although previous radiomics studies in the neuroradiology field have mostly focused on neuro-oncology (8,9,10,11), recently there has been a growing number of studies that performed radiomics analyses on MCI and AD. These studies have demonstrated promising results in differential diagnosis (12,13,14,15,16,17) and prediction of cognitive conversion in MCI and AD patients (18,19,20,21).

Radiomics research has shown great promise for personalized clinical decision making (22). However, the fact that radiomics research is currently performed for academic purposes without clinical translation can be partly attributed to insufficient strategies for imaging biomarker translation, which requires methodology standardization for reproducibility and evaluation of clinical-biomarker correlation and biomarker-outcome correlation (23). Recently, a radiomics quality score (RQS) was proposed to assess the quality of studies (22). Previous studies have assessed RQS in the oncology (24,25) or neuro-oncology fields (26), but to the best of our knowledge, the quality of science in radiomics research studies in MCI and AD is unknown. There is a need to assess the quality of current radiomics studies to provide a roadmap for improvement in future researches.

Therefore, the purpose of our study was to evaluate the quality of reporting of radiomics in MCI and AD studies using RQS. We intended to promote the quality of reporting of radiomics in MCI and AD studies and increase the reliability of radiomics for the diagnosis and prognostic biomarkers of MCI and AD in the clinical setting.

MATERIALS AND METHODS

Systematic Search Strategy and Study Selection

All original research papers using radiomics analysis published up until March 11, 2020 were searched from PubMed MEDLINE (n = 133) and EMBASE (n = 224) databases using the following search term: (“cognitive impairment” OR “Alzheimer” OR “dementia”) AND (“radiomic” OR “texture” OR “radiogenomic”). A total of 357 candidate articles were searched, and the retrieved articles were screened for eligibility. After removing 99 duplicate articles, 219 articles were further excluded for the following reasons: non-radiomics studies (n = 130), conference abstracts (n = 65), technical notes (n = 8), review articles (n = 7), not in the field of interest (n = 6), non-human subject studies (n = 2), and non-brain images (n = 1). Of the remaining 40 articles, studies using less than 10 radiomics features (n = 7), studies with main text in languages other than English (n = 6), and studies assessing only correlation without results of diagnostic or prognostic performance (n = 1) were excluded. Finally, 26 articles were included in analysis (Fig. 1).

Fig. 1. Flow chart of the study selection process.

Analysis of Method Quality Based on RQS

The RQS score consisted of 16 components. The reviewers performed RQS evaluation according to six domains as previously reported (Supplementary Table 1) (25,26). Prior to the evaluation, a research meeting was held to educate the reviewers on the RQS system.

Two reviewers (with six and nine years of experience in radiology, respectively) independently scored the articles for each of the six domains using RQS (Supplementary Materials). If disagreement occurred between the two reviewers, a final decision was made through a consensus.

In addition, additional topics of RQS were discussed by the two reviewers and a consensus was reached for evaluation with consideration to the characteristics of AD and MCI researches (Supplementary Materials). RQS was scored according to the consensus reached for the following topics: ‘image protocol quality’ (domain 1), automatic segmentation for ‘multiple segmentation’ (domain 2), issues for ‘validation’ (domain 2), ‘comparison with the gold standard’ (domain 3), and ‘potential clinical utility’ (domain 3).

Statistical Analysis

The characteristics of articles were reviewed. If the article got at least one point from each item (0–16 items), it was defined as having a basic adherence to RQS for that item. Basic adherence to RQS for 0–16 items was counted. Basic adherence rate (%) was calculated as proportion of the number of articles with basic adherence to number of total articles. RQS score was described as mean scores and standard deviation using descriptive statistics for each item. Percentage of the ideal score (%) was defined as percentage of mean score to ideal score for each item. Total RQS score (−8 to 36) was counted for all articles.

In addition, subgroup analysis was performed to determine whether the reporting quality improved over time (publications before January 1, 2019 [n = 18] and after January 1, 2019 [n = 8]). According to normality test results, either Student's t test or the Mann-Whitney's U-test was applied for comparison. A p value < 0.05 was considered statistically significant. All statistical analyses were performed using R (version 4.0.2; R Foundation for Statistical Computing).

RESULTS

Characteristics of the 26 Included Radiomics Studies in MCI and AD

The characteristics of the 26 included radiomics studies (12,13,14,15,16,17,19,20,21,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43) are summarized in Table 1, Figure 2, and Supplementary Table 2. The median number of subjects in the included articles was 204 (range 86–460). Journal type included 10 clinical journals (38.5%), 9 imaging journals (34.6%), and 7 computer science/neuroscience journals (26.9%). Radiomics analysis was performed to evaluate a diagnostic biomarker (50%), prognostic biomarker (42.3%), or both (7.7%). The purposes of the studies included differential diagnosis (42.3%), prediction of conversion to dementia (42.3%), or both (7.7%). There was no study that assessed molecular/genomic classification or response to treatment. Radiomics analysis of MCI or AD was mainly performed on brain MRI (84.6%), followed by PET (7.7%), or both (7.7%). Of 24 studies with MRI, 21 studies (87.5%) used only T1-weighted images and the remaining 3 (12.5%) used T1-weighted images combined with other sequences such as fluid-attenuation inversion recovery or quantitative susceptibility mapping for feature extraction. Automatic segmentation (76.9%) was more frequently performed than manual segmentation (23.1%). Hippocampal analysis (46.2%) was most frequently used, followed by miscellaneous segmentation (23.1%), white matter and/or gray matter segmentation (19.2%), and whole brain region segmentation (11.5%). Only one study performed external validation (3.8%). Of the 26 studies, 16 studies (61.5%) used an open source database (14 from Alzheimer's Disease Neuroimaging Initiative [ADNI] and two from Open Access Series of Imaging Studies [OASIS]) and the remaining 10 studies (38.5%) used data from a single institute. In terms of magnetic strength, 10 studies (41.7%) utilized a 3.0 Tesla (T) magnet, 10 studies (41.7%) utilized 1.5T, 3 studies (12.5%) utilized both 1.5T and 3T, and 1 study (4.2%) utilized 2.0T.

Table 1. Characteristics of the 26 Included Radiomics Studies.

Article Characteristics	No. of Articles*
No. of subjects	204 (range 86–460)
Journal type
Clinical journal	10 (38.5)
Imaging journal	9 (34.6)
Computer science/neuroscience journal	7 (26.9)
Biomarker
Diagnostic	13 (50)
Prognostic	11 (42.3)
Diagnostic and prognostic	2 (7.7)
Predictive	0 (0)
Topics in MCI and AD
Differential diagnosis	13 (50)†
Prediction of cognitive conversion	13 (50)†
Molecular/genomic classification	0 (0)
Response to treatment	0 (0)
Imaging type
MRI	22 (84.6)
PET	2 (7.7)
MRI and PET	2 (7.7)
Sequence used for feature extraction in MRI studies‡
T1WI	21 (87.5)
T1WI + FLAIR or T1WI + QSM	3 (12.5)
Segmentation
Automatic	20 (76.9)
Manual	6 (23.1)
Anatomy
Hippocampus	12 (46.2)
White matter and/or gray matter	5 (19.2)
Whole brain	3 (11.5)
Miscellaneous	6 (23.1)
External validation
Performed	1 (3.8)
Not performed	25 (96.2)
Studies using open source dataset
ADNI	14 (53.8)
OASIS	2 (7.7)
Non-open source (institutional dataset)	10 (38.5)
Magnetic field strength (tesla)‡
1.5	10 (41.7)
3.0	10 (41.7)
2.0	1 (4.2)
1.5 and 3.0	3 (12.5)

^*Numbers in parentheses are percentages, ^†Two studies overlapping in both differential diagnosis and prediction of conversion to dementia were classified as prognostic purpose, ^‡Data analyzed in studies with MRI. AD = Alzheimer's disease, ADNI = Alzheimer's Disease Neuroimaging Initiative, FLAIR = fluid-attenuated inversion recovery, MCI = mild cognitive impairment, OASIS = Open Access Series of Imaging Studies, QSM = quantitative susceptibility mapping, T1WI = T1-weighted image

Fig. 2. Summary chart of the radiomics studies, according to the (A) number of published studies on radiomics in the AD research field, (B) topics in cognitive impairment and dementia, (C) segmentation method, (D) anatomy, and (E) usage of open source dataset.

ADNI = Alzheimer's Disease Neuroimaging Initiative, GM = gray matter, Misc = miscellaneous, OASIS = Open Access Series of Imaging Studies, WM = white matter

Basic Adherence Rate of the Reporting Quality according to the Six Key Domains

The basic adherence rates to the RQS for a total of 16 items are documented in Table 2. In domain 1, 25 studies (96.2%) used well-documented image protocols or public datasets. Two studies (7.7%) performed imaging at two time scans to evaluate the stability of radiomics features (36,41). There was no study that performed a phantom study. Automatic segmentation was performed in 20 studies (76.9%).

Table 2. Radiomics Quality Score according to the Six Key Domains.

	Basic Adherence Rate (%)	Mean Score (Mean ± Standard Deviation)	Percentage of the Ideal Score (%)
Total 16 items (ideal score 36)	115 (27.6)	3.6 ± 6.6	9.9
Domain 1-protocol quality and stability in image and segmentation (0 to 5 points)	47 (45.2)	2.5 ± 1.0	50
Protocol quality (2 points)	25 (96.2)	1.6 ± 0.6	80.8
Test-retest (1 point)	2 (7.7)	0.1 ± 0.3	7.7
Phantom study (1 point)	0 (0)	0	0
Multiple segmentation (1 point)	20 (76.9)	0.8 ± 0.5	80.8
Domain 2-feature selection and validation (−8 to 8 points)	26 (50.0)	−1.2 ± 5.5	−15.4
Feature reduction or adjustment of multiple testing (−3 or 3 points)	14 (53.8)	0.2 ± 3.1	7.7
Validation (−5, 2, 3, 4, or 5 points)	12 (46.2)	−1.2 ± 4.2	−24.6
Domain 3-biologic/clinical validation and utility (0 to 6 points)	7 (6.7)	0.4 ± 0.8	7.7
Non-radiomics features (1 point)	3 (11.5)	0.1 ± 0.3	11.5
Biologic correlations (1 point)	1 (3.8)	0.0 ± 0.2	3.8
Comparison to “gold standard” (2 points)	3 (11.5)	0.2 ± 0.7	11.5
Potential clinical utility (2 points)	0 (0)	0	0
Domain 4-model performance index (0 to 5 points)	18 (69.2)	0.9 ± 0.7	18.5
Cut-off analysis (1 point)	0 (0)	0	0
Discrimination statistics (2 points)	18 (69.2)	0.8 ± 0.7	46.2
Calibration statistics (2 points)	0 (0)	0	0
Domain 5-high level of evidence (0 to 8 points)	0 (0)	0	0
Prospective study (7 points)	0 (0)	0	0
Cost-effectiveness analysis (1 point)	0 (0)	0	0
Domain 6-open science and data (0 to 4 points)	17 (65.4)	0.8 ± 0.8	19.2

In domain 2, 14 studies (53.8%) performed feature reduction or adjustment of multiple testing. In 14 studies (53.8%), validation was missing. In seven studies (26.9%) (15,19,20,21,30,41,42), validation was done by randomly splitting an open source dataset such as the ADNI database into training and test sets. In the remaining 5 studies (19.2%), validation was executed based on a dataset from the same institute (14,16,17,40,43).

In domain 3, only 3 studies (11.5%) executed multivariate analysis with non-radiomics features (19,38,39). Studies included either clinical and/or genotypes in addition to radiomics features for diagnostic and prognostic models. Only 1 study (3.8%) earned 1 point in the biologic correlation component (42). Only 3 studies (11.5%) compared radiomics with the “gold standard” method (such as hippocampal volume or clinical risk factors) (19,20,21). There was no study that addressed the “clinical utility” component.

In domain 4, 18 studies (69.2%) used discrimination statistics, such as receiver operating characteristic curve or area under the curve with their statistical significance, or a resampling method, such as bootstrapping or cross-validation. Six of the 18 studies (33.3%) used both discrimination statistics and a resampling method (19,21,28,29,38,43). There was no study using either cut-off analysis or calibration statistics.

In domain 5, there was no prospective study or report on the cost-effective analysis. Lastly, in domain 6, 17 studies (65.4%) made either 1 of 4 categories (scan, region of interest, code, representative region of interest with calculated radiomics feature) publicly available. Only one of those made all four components publicly available (37).

Assessment of the RQS

For the 26 radiomics studies, the mean overall RQS score was 3.6 ± 6.6, which was 9.9% of the ideal score (Table 2, Fig. 3). The lowest score was −7, and the highest score was 16, which was 44.4% of the ideal score. When considering each domain, the mean score and percentage of the ideal score was lowest in domain 2 (feature selection and validation) and highest in domain 1 (protocol quality and stability in image and segmentation).

Fig. 3. RQS assessment results according to the six key domains.

RQS = radiomics quality score

Neither feature selection nor validation was executed in 8 studies (30.8%) (29,31,32,33,34,35,36,38), and 1 of these 8 studies had the lowest RQS (34). The study with the highest score achieved the ideal score in protocol quality, multiple segmentation, feature reduction or adjustment of multiple testing, non-radiomics features (cox model analysis including clinical factors), comparison to “gold standard,” and discrimination analysis using C-statistics (19).

Subgroup Analysis

The results of the subgroup analysis according to the publication date is shown in Supplementary Table 3. The mean overall RQS was significantly higher in recently published studies after January 1, 2019 than studies published before January 1, 2019 (8.1 vs. 0.8, p = 0.006). A statistically significant increase was seen in the validation component in domain 2 (3 vs. −5, p = 0.035).

DISCUSSION

Radiomics research in MCI and AD is rapidly growing, and a comprehensive evaluation of the quality of science and reporting at present is critical to ensure progress in the field. This study evaluated radiomics studies in MCI and AD for their quality in the science and reporting using RQS. The basic adherence rate and percentage of ideal RQS were 27.6% and 9.9%, respectively. In terms of validation, radiomics studies were particularly insufficient (3.8%) in performing external validation. None of the studies addressed potential clinical utility nor did they perform calibration statistics. Also, none of the studies performed either prospective study nor cost-effectiveness analysis, resulting in a low level of evidence. Our study indicates that the overall quality was suboptimal in radiomics studies in MCI and AD, requiring significant improvement.

Radiomics research in MCI and AD manifest characteristics that differ from those of neuro-oncology radiomics research, and some of these characteristics provide certain advantages. First, radiomics studies in MCI and AD were highly dependent on the ADNI dataset (comprising 53.8% of the radiomics studies), proving the profound impact of ADNI in MCI and AD research. The well-labeled high-quality open source database from ADNI provided a relatively high basic adherence rate (65.4%) in the open science and data domain for the radiomics research, proving the strength of its database framework. These results differ from the previous studies that assessed the quality of reporting and science of radiomics studies in oncology or neuro-oncology and revealed substantially lower basic adherence rates (3.9% and 5.9%, respectively) in the open science and data domain (25,26). However, most radiomics studies using the ADNI dataset used only the ADNI dataset without true external validation. Since the MRI protocol in ADNI is strictly controlled and relatively homogeneous, future studies performing external validation with either an independent institutional or another open source dataset are warranted to validate the true performance of a radiomics model and to gain clinical significance (44). Second, due to the well-developed automatic segmentation tools for neurodegenerative diseases, the basic adherence rate for multiple segmentation was relatively high (76.9%). This seems to be an advantage in radiomics studies in MCI and AD in contrast to those in oncology, where there is still an insufficiency of validated automatic segmentation tools.

On the other hand, there is also vast room for improvement in future radiomics studies in MCI and AD compared to those in neuro-oncology. First, unlike the radiomics studies in neuro-oncology, which implemented molecular or genomic classification with a rate of 49% (26), radiomics studies in MCI and AD have not yet performed molecular or genomic classification (45). Previous studies have already shown associations between apolipoprotein E genotype, the most robust AD susceptibility gene (45), and hippocampal atrophy (46,47), and radiomics may have the potential to predict the apolipoprotein genotype, which must be explored in future studies. Second, the basic adherence rates of domain 2 (feature selection and validation), domain 3 (biological/clinical validation and utility), and domain 4 (model performance index) were all considerably lower than those reported for neuro-oncology or oncology researches. Specifically, the basic adherence rates for domain 2, domain 3, and domain 4 were 50.0%, 6.7%, and 23.1%, which were all substantially lower than previously reported basic adherence rates of 81.4%, 39.2%, and 45.1% for the neurooncology radiomics research (26). In domain 2, the basic adherence rate for feature selection was 53.8%, which was substantially lower than previous reported rates of 94.1% and 96.1% for the neuro-oncology and oncology researches, respectively (25,26). Since radiomics represent complex high-dimensional data with relatively small samples (“large-p, small-n” data), feature reduction or adjustment of multiple testing is a necessary process when understanding the nature of radiomics to avoid overfitting (48). Also, the basic adherence rate of the validation in domain 2 was 46.2%, which was also lower than previous reported rates of 68.6% and 70.1% in neuro-oncology and oncology researches, respectively, with a mean RQS score below zero (−1.2 ± 4.2). There was no type of validation in 14 studies (53.8%). Moreover, external validation was conducted in only one study (3.8%). In order for radiomics studies to be translated into clinical practice, external validation is a crucial process for generalizing the radiomics model. Also, with regard to domain 3, radiomics research can achieve a higher clinical impact by integrating non-radiomics features and comparing the performance of radiomics to the “gold standard” in MCI and AD; however, the current adherence rates for both are as low as 11.5%. There is also potential for improvement in the use of cut-off analysis and calibration statistics in domain 4. These are important for the application of a radiomics model, and further emphasize the utility of radiomics in clinical settings. Considering the fact that processes such as feature reduction and validation (especially from the same institute) from domain 2, multivariable analysis with non-radiomics features and comparison to “gold standard” from domain 3, and discrimination/calibration statistics from domain 4 are relatively simple processes that can be easily integrated into the radiomics pipeline, we speculate that future studies may achieve higher technical and clinical impact by adhering to these aspects.

We applied the six key domains designed in previous researches, that support the integration of the RQS (25,26). There are several more key domains that require significant improvement. Regarding the technical validation in domain 1, only 2 studies (7.6%) conducted test-retest (37,42) and no studies performed a phantom study, indicating overall insufficiency of data supporting the precision or technical bias. Technical validation is warranted in future studies performing radiomics analysis. In domain 6, only 1 study (3.8%) (37) provided the code in open source, and majority of the studies did not provide clear definitions of the radiomics calculation. For standardization of radiomics features and the reproducibility of the radiomics technique, multi-center trials are needed, and the releasing of the code in open source can accelerate the development of the radiomics field.

It should be noted that RQS is an expert opinion and not a reporting guideline. The suggested RQS may be too idealistic to be qualified (i.e., the phantom study and multiple imaging acquisitions) in clinical settings. Also, the scoring system when using an open source dataset such as ADNI or OASIS is unclear. Nonetheless, pursuit for a higher quality of reporting is inevitable for the future clinical application of radiomics approaches. The recently published radiomics studies showed significantly higher RQS than formerly published studies, which suggests that the quality of science may be further improved in future studies.

There are several limitations in our study. First, there were a relatively small number of articles in MCI and AD radiomics research. We decided to focus on this specific field because there seemed to be an urgent need to review the overall quality of the rapidly increasing radiomics researches, and to provide a roadmap for future studies to improve the methodology and reporting. Second, adherence to several components in the RQS is rarely possible in MCI and AD studies (for example, the ‘biological correlations’ component in domain 3 due to limited histologic confirmation (49) and ‘cut-off analyses’ in domain 4), which may have lowered the overall RQS score. Despite these limitations, our results show that the overall quality of radiomics research in AD and MCI was suboptimal, especially when compared to neuro-oncology radiomics research. Thus, there is scope for improvement in future studies to reach a higher technical and clinical impact.

In conclusion, the current quality of reporting of radiomics studies in MCI and AD is suboptimal. Validation is necessary using an external dataset, and improvements need to be made to feature reproducibility, feature selection, clinical utility, model performance index, and pursuits of a higher level of evidence.

Supplementary Materials

The Data Supplement is available with this article at https://doi.org/10.3348/kjr.2020.0715.

Supplementary Table 1

The Six Key Domains of the RQS

Supplementary Table 2

Characteristics of the Included Studies

Supplementary Table 3

Subgroup Analysis of RQS according to Publish Date