Skip to main content
NCBI home page
The British Journal of Radiology logoLink to The British Journal of Radiology
. 2022 Jun 15;95(1137):20211211. doi: 10.1259/bjr.20211211

Radiomics quality score in renal masses: a systematic assessment on current literature

Afshin Azadikhah 1, Bino Abel Varghese 2,✉, Xiaomeng Lei 3, Chloe Martin-King 4, Steven Yong Cen 5, Vinay Anant Duddalwar 6
PMCID: PMC10996962  PMID: 35671097

Abstract

Objective:

To perform a systematic assessment and analyze the quality of radiomics methodology in current literature in the evaluation of renal masses using the Radiomics Quality Score (RQS) approach.

Methods:

We systematically reviewed recent radiomics literature in renal masses published in PubMed, EMBASE, Elsevier, and Web of Science. Two reviewers blinded by each other’s scores evaluated the quality of radiomics methodology in studies published from 2015 to August 2021 using the RQS approach. Owing to the diversity in the imaging modalities and radiomics applications, a meta-analysis could not be performed.

Results:

Based on our inclusion/exclusion criteria, a total of 87 published studies were included in our study. The highest RQS was noted in three categories: reporting of clinical utility, gold standard, and feature reduction. The average RQS of the two reviewers ranged from 5 ≤ RQS≤19, with the maximum attainable RQS being 36. Very few (7/87 i.e., 8%) studies received an average RQS that ranged from 17 < RQS≤19, which represents studies with the highest RQS in our study. Many (39/87 i.e., 45%) studies received an average RQS that ranged from 13 < RQS≤15. No significant interreviewer scoring differences were observed.

Conclusions:

We report that the overall scientific quality and reporting of radiomics studies in renal masses is suboptimal, and subsequent studies should bolster current deficiencies to improve reporting of radiomics methodologies.

Advances in knowledge:

The RQS approach is a meaningful quantitative scoring system to assess radiomics methodology quality and supports a comprehensive evaluation of the radiomics approach before its incorporation into clinical practice.

Introduction

In the United States, the American Cancer Society estimates that in 2022 there will be 7,9000 new cases of kidney cancer with 1,3920 deaths. 1 Definitive treatments of localized renal cancer include surgical resection and ablative treatments. However, for localized small renal masses, active surveillance which includes clinical follow-up and scheduled imaging evaluation has emerged as a safe alternative management strategy in selected patients. 2 Imaging plays a key role in early diagnosis, and differential diagnosis, as well as in determining tumor stage and following up patients for response to treatment. 3 In clinical practice, visual or qualitative evaluation of imaging is used in patient management, and hence subject to interobserver bias. 4

Radiomics is the quantifiable extraction of objective features/metrics from imaging data. 5–7 This method enables the extraction of additional information beyond qualitative or visual analysis. 8,9 In radiomics, the quantitative imaging features are calculated from different types of digitalized imaging data, including radiographs, CT, MRI, PET, and ultrasound data. 10 The high-dimensional data and the massive quantity of quantitative information are best analyzed by artificial intelligence techniques such as machine-learning (ML) methods. 11 Radiomics studies have been used in numerous cancer models in the improvement of differential diagnosis and prognosis of tumors and accurate prediction of tumor grading. 8,12–14 However, radiomics studies are difficult to compare and correlate to each other as they are often single center-based, retrospective studies with a varying degree of pathological confirmation. 15,16 The purpose of our study is to evaluate the quality of published studies in the radiomic evaluation of renal masses. We hypothesized that the Radiomics Quality Score (RQS) approach can provide systematic guidelines for radiomics evaluation and reporting and help identify areas of improvement within the radiomics workflow to aid clinical translation.

Radiomics workflows

Currently, two broad types of radiomics workflows exist (Figure 1). Firstly, Hand-crafted Radiomics is a conventional workflow that consists of human-engineered metrics. 17 Another radiomics workflow involves deep-learning-based radiomic (DLR). 17,18 Hand-crafted radiomics workflows feature four basic modules: Image acquisition, region of interest (ROI) segmentation, feature extraction, and statistical analysis. Consistent image acquisition is necessary for radiomics analyses to increase the repeatability and comparability of studies. 9 Meanwhile, the robustness of radiomic features can be affected by different methods of imaging acquisition. 19–21 Image segmentation can become time-consuming and subjective depending on the type of segmentation performed. Although automated segmentation is more efficient, the accuracy associated with manual approaches makes it widely used in lots of radiomics applications. 22,23 Feature extraction involves the extraction of quantitative features leading to the creation of high-dimensional quantitative data from the segmented ROI. The extracted data can be classified into three major groups: 1) size/shape-based (morphology) features, 2) statistical/image intensity features 3) texture-based features. 24 The high-dimensional radiomics feature spaces present a challenge for traditional statistical methods, e.g., logistic regression models. Radiomics-based prediction models have often been developed using ML methods. To process reconstructed images for classification or detection, DL utilizes artificial neural network (ANN) algorithms, particularly convolutional neural networks (CNNs). 25,26 These methods extract the features from the images automatically thereby reducing the need for manual processing of images, which is a time-consuming process and subject to operator bias. Hybrid approaches, which combine the strengths of both models, have shown promising results across many cancer types. 17

Figure 1.

Figure 1.

Open in a new tab

Schematic showing the two types of radiomic workflows and their constituent modules.

Methods

We conducted a systematic assessment. Systematic review methods (ref PRISMA guideline 27–29 ) were followed as far as possible given the nature of this assessment (Figure 2). Since in our systematic assessment, we included studies with a variety of different radiomic applications including but not limited to the prediction of tumor malignancy, grade, subtype, necrosis score, molecular features such as gene mutations, metastasis, clinical outcomes, etc., as well as spanning a variety of imaging modalities such as CT, MRI, and PET, we did not perform a companion meta-analysis, but rather only focused on quality assessment of the methodologies of the radiomics studies. The RQS approach has been used in the literature on the radiomic pipelines to evaluate the quality of the internal/external validation process. 11,30 We found appropriate literature published in PubMed (National Center for Biotechnology Information, NCBI), EMBASE (Ovid), Elsevier (Scopus), and Web of Science (WoS), based on our search strategy to perform a systematic assessment and subsequent RQS analysis. To capture contemporary trends in radiomics applications the search was limited to articles published from 2015 until 08-31-2021. In our study, the search terms used to find radiomics studies included the following terms in isolation or combination: “kidney cancer,” “renal cell carcinoma,” “renal mass,” “radiomics,” and “radio-genomics.” Most of these shortlisted studies applied artificial intelligence (AI) to the classification task and were classified as ‘machine learning’ or ‘deep learning’. We repeated searches on all newly identified articles until no further relevant articles were found. Inclusion criteria included: 1) Full-text papers written in English 2) Studies with radiomics analyses performed on the kidney using CT and MRI with or without PET scans. Exclusion criteria removed 1) duplicate literature from these four databases, and 2) case reports and review papers.

Figure 2.

Figure 2.

Open in a new tab

Diagram of systematic assessment workflow. Modified from PRISMA 2020 statement (http://www.prisma-statement.org/)

A systematic and RQS assessment

The research question of our systematic assessment study was: “What are the recent radiomics studies related to renal mass evaluation, how do these studies compare the quality of the radiomics methodology and its reporting in these studies?”

The RQS approach has been used in the literature of radiomic studies to evaluate the quality of the internal/external validation process. 11,30 However, RQS studies within RCC have been limited. 31–34 The RQS approach consists of sixteen criteria evaluating the methodology of radiomics studies with a 36-point scale scores from −8 to +36. 30 Each criterion is assigned a numerical value equivalent to its impact in radiomics studies as indicated in Table 1. From the shortlisted literature, two reviewers (A.A. and B.V.), blinded to each other’s evaluations, scored the studies using the RQS approach. 11 The RQS criteria for each study were categorized and quantified in the Supplementary Table 2, including the authors, the year of publications, journals, radiomic features, and modality. We calculated the total RQS for each article and expressed it as the average of two reviewers' scores.

Table 1.

Description of RQS parameters

Parameters Marks
 1  Image protocol: recorded image protocols in detail (+1) and/or general utilization for reproducibility (+1).  0 _ 2
2 Multiple segmentation: Multiple segmentation is done by different fellows or made by different applications (+1). To study feature validity at different segmentation modules. 0, 1
3
4
5
6
7
8
9
10
11
12
13
14
15
16		 Phantom study: To study features validity, a phantom study is an origin of diversity on the scanners (+1).
Multiple time points: Images at different time points - multiple images for treatment and/or pre-treatment follow-up (+1). To study features validity at different time points.
Feature reduction: To study features validity by feature selection, feature reduction accommodates features for multiple trials. It prevents the selection of corresponding features for a set of samples.
Non-Radiomics: Non-Radiomics features represent an integrated Radiomic study to find the potential correlations between Non-Radiomics and Radiomics features (+1).
Biological correlates: identifying biological correlation provides insight into the biological correspondence with Radiomics (+1).
Cut-off: cut-off investigations identify risk factors or at-risk populations using statistical analyses (+1).
Discrimination and resampling: based on the results of statistical analyses, discrimination is determined by the ROC curve, AUC, concordance statistic. Statistical significance is determined by P-value, confidence interval (+1). The resampling method can be utilized by cross-validation and bootstrap techniques in statistics (+1).
Calibration: calibration statistics are used in predictive analytics by calibration slope, calibration plot, calibration in the large, and determining the statistical significance by P-value, confidence interval (+1). The resampling method can be utilized by cross-validation and bootstrap techniques in statistics (+1).
Prospective: prospective models are used in the trial studies for validation of clinical prediction models to determine the application of specific radiomic biomarkers (+7).
Validation: validation is an established set of data from the same institute (+2), from another institute (+3), two separate institutes (+4), a published study in the past (+4), and three or more separate institutes (+5), or lack of validation (−5).
Gold standard: it determines which model in the Radiomic study is in concordance with the gold standard model (+2).
Clinical utility: it determines whether the Radiomic study is potentially useful (+1) or currently active (+1) in clinical applications.
Cost-effectiveness: it determines whether the Radiomic study is valuable and cost-effective in clinical applications (+1).
Open science: making encoded data available for public and open encrypted data, promotes consistency and the ability to reproduce the study. Open source of scans (+1), open-source of ROI segmentation (+1), open-source of code (+1), calculation radiomic features based on a typical set of ROIs and ROIs are open source (+1).		 0, 1
0, 1
−3, + 3
0, 1
0, 1
0, 1
0 _ 2
0 _ 2
0, + 7
−5, 2 _ 5
0, + 2
0 _ 2
0, 1
0 _ 4
Open in a new tab

The percent adherence was calculated based on the percent proportion of articles that fulfilled each RQS criterion. Studies having the highest score for each RQS criterion were assigned ‘perfect’ adherence to that criterion. Studies having zero or less than zero scores for a given RQS criterion were assigned a ‘non-adherent’ designation. Studies with scores between the perfect and the non-adherent RQS criterion were categorized as studies with ‘some’ adherence. In case of interrater disagreement, adjudicated scores were obtained for the adherence assessment. Interrater agreement was assessed using intraclass correlation coefficient 2-way random with the absolute agreement (ICC 2.1) for the overall score over 16 sections. For each measurement, the prevalence-adjusted and bias-adjusted κ (PABAK) coefficient was used. 35 We evaluated the consistency between reviewers using the κ score and the differences in RQS for each study between reviewer 1 and reviewer two were measured. The agreement and disagreement were calculated between different studies for both reviewers’ scores.

Results

129 full-text manuscripts were identified using the search terms listed. After excluding 42 papers based on duplication and other exclusion criteria, including review papers and case-report studies, we included 87 manuscripts. Based on the date of publication for each database/manuscript, 53, were identified using PubMed, 14 from Ovid, 10 from Scopus, and 10 from WoS. Figure 2 summarizes an overview of our systematic assessment. The results’ view and the calculation of total RQS for each reviewer/article are presented in Supplementary Material 1. The RQS results highlighted the differences between the two independent reviewers. A summary of the average RQS of the reviewers for each study with the details of the present systematic assessment is provided in the Supplementary Material 1.

73 of 87 (84%) published papers reported using AI-augmented processes that used AI algorithms such as ML and DL patterns in their radiomics studies. 73/87 (84%) studies utilized radiomics features for predicting tumor grade associated with renal masses. Overall, 13 of 87 (15%) studies were related to radio-genomics, including gene mutation, and gene expression, or were associated with, biological systems consisting of the immune system and cell-cycle control pathways. 87/87 (100%) of the studies had the gold standard and potential clinical applications however, none of the studies have reported the phantom study, prospective radiomics study, and cost-effectiveness of radiomics in clinical applications such as raising the quality of life.

Of the 87 studies reviewed, 39/87 (45%) had an average RQS ranging between 13 and 15 (13 < RQS≤15) with the maximum attainable RQS being 36. That means the overall quality of reporting of radiomics studies in this cohort was suboptimal. Only 7 of 87 (8%), had an average RQS ranging between 17 and 19 (17 < RQS≤19), which represents the highest RQS range in our assessment (Figure 3).

Figure 3.

Figure 3.

Open in a new tab

Histogram of average RQS representative of the RQS range. Boundaries are indicated by signs, which, as an example (13, 15] means the range between 13 < RQS≤15.

The assessment of adherence of the 87 studies reviewed to the various categories of the RQS showed that the top three categories with perfect adherence belonged to clinical utility, use of gold standard, and feature reduction. The categories with the lowest adherence belonged to reporting of phantom studies, prospective studies, and cost-effectiveness of the radiomics approach (Figure 4).

Figure 4.

Figure 4.

Open in a new tab

Distribution of median RQS across the 16 categories of the RQS approach. The RQS was represented as adjudicated scores.

We found high agreement between the two reviewers with ICC-total = 0.66, 95% CI 0.52–0.77 (Figure 5) across the entire RQS panel of metrics. The average scores of the two reviewers were normally distributed. 65/87 studies acquired a score within the average range (16th to 84th percentile rank). The range, mean, mode, and median values for total average RQS, were found to be 5 to 19 (13.9% to 52.8%), 14.2 (39.4%), 14 (38.9%), and 15 (41.7%), respectively. 80 of 87 studies (92%) received less than 50% maximum score, and none of the studies were assigned a quality score of 0% or 100%.

Figure 5.

Figure 5.

Open in a new tab

Median RQS per category. The overall ICC between the two reviewers across the entire RQS panel was 0.66.

The disagreements in variables of RQS results between interreviewers were mostly observed in the scoring of validation, open science, calibration, clinical utility, and discrimination/resampling sections with reported agreements of 79.31%, 80.46%, 85.06%, 89.66%, and 94.25%, respectively (Figure 5; Table 2).

Table 2.

Agreement for RQS variables between reviewers

Measure Image Protocol Multiple Segmentation Phantom Study Multiple Time Points Feature Reduction Non-Radiomics Biological Correlates Cut-off Discrimination and Resampling Calibration Prospective Validation Gold Standard Clinical Utility Cost-Effectiveness Open Science
Total agree % 95.4 96.55 100 97.7 100 97.7 95.4 97.7 94.25 85.06 100 79.31 100 89.66 100 80.46
Lower confidence limit 0.88 0.89 1 0.92 1 0.92 0.87 0.92 0.86 0.66 1 0.65 1 0.67 1 1
Upper confidence limit 1 1 1 1 1 1 1 1 0.99 0.89 1 0.85 1 0.92 1 1
Open in a new tab

Discussion

Our systematic assessment assessed the quality of recently published literature in renal mass radiomics using the RQS analysis. This paper identifies some of the key limitations impeding its clinical translation and provides some future perspectives. To examine tumor biology in radiomics using RQS, Sanduleanu et al 31 included 41 papers with a distribution of RQS that corresponded to our assessment. Their study focused on biological tumors of all organs, whereas our study evaluated the quality of 87 radiomics studies on renal masses and found the radiomics modality held up to ML and DL in 84% of studies. An assessment of methodological quality by Ursprung et al 32 was performed on 57 studies using RQS (average RQS: 9.4%). Furthermore, a meta-analysis showed moderate heterogeneity in ten studies (odds ratio: 6.24) for the differentiation of angiomyolipoma without visible fat from RCC. Accordingly, Mühlbauer et al 33 evaluated 113 studies about the diagnosis of localized renal tumors using RQS (median RQS: 13.9%) and meta-analysis by analyzing 30 studies for the differentiation of angiomyolipoma, oncocytoma, and unidentified benign renal tumors reported odds ratios of 2.89, 3.08, and 3.57, respectively. The study by Bhandari et al, 34 assessed thirteen studies on CT radiomics for grading and differentiating various subtypes in renal tumors using RQS (median score: 33.3%), by evaluating four studies to differentiate low- and high-grade clear cell RCC with AUC = 0.82–0.978, and one study that differentiated low- versus high-grade chromophobe with AUC = 0.84, and eight studies that differentiated benign renal tumors from various subtypes of RCC (AUC = 0.82–0.96). We conducted a systematic assessment of papers published more recently, and therefore we provided an updated RQS with comparable suboptimal RQS results. Our findings on the quality of radiomics reporting in renal studies using RQS are similar to those reported in the literature on brain metastasis, 36 neurodegenerative disease, 37 and hepatocellular carcinoma, 38 which also reported suboptimal RQS as well.

In the RQS-based categorical assessment of most given scores of all studies, the highest levels of adherence to reporting clinical utility, use of the gold standard, and feature reduction were found. However, reporting cost-effectiveness, conducting prospective studies, and phantom studies had the lowest adherence rates. Therefore, it highlights the need for more carefully designed radiomics studies with a focus on these criteria. In non-kidney related RQS studies, these categories also received low adherence scores. In fact, among the three categories that were least adhered to, prospective studies are the most important, considering that for clinical validation, prospective testing of an imaging biomarker in clinical populations is required. While the adherence of the other two categories carries lower points i.e., two points for cost-effectiveness and one point for phantom studies, they are important as the information regarding if the new technique provides value for money compared with the other currently available techniques and if they are reliable (robust, repeatable, and reproducible), respectively is important for clinical translation of radiomics. While higher adherence in reporting imaging protocols, discrimination/resampling, and validation have been observed, in comparison with the other RQS criteria such as gold standard and clinical utility, there is still a wide variation in the RQS results between different studies. This draws attention to making the radiomics workflow more consistent and comprehensive. However, this is not an easy task considering the different approaches to performing a radiomics study based on study objectives, techniques, and results. 19 Ongoing efforts of standardizing the radiomics methodology and harmonizing its outputs within multicenter settings are underway and showing promising results. 39,40 Digital or virtual phantom-based imaging studies could be rigorously conducted with the pertinent studies to establish imaging standards for the minimum levels of image quality required for radiomics analysis. Assuming an acceptable image quality, heterogeneity in radiomics results may result from the variation of imaging preprocessing before radiomics feature extraction, or due to poor consistency regarding radiomics feature terminology and implementation. Therefore, this further demonstrates the need for increased research in reporting the phantom study, prospective radiomics study, and cost-effectiveness.

Segmentation is one of the main challenges in radiomics. While manual segmentations are commonly used in radiomic studies, they are also subjective to interreader disagreement based on reader proficiency. 41,42 Therefore, with the growth of imaging datasets that are publicly accessible, the need for reliable and efficient segmentation is increasing. 43,44 After validation, DL could also improve consistency in segmentation considering that segmentation is one of the fundamental steps of the radiomics workflow. 44 Some studies use one or more deep networks to create prediction models either by extracting DLR features or as an end-to-end tool up to the prediction task using image classification. 45,46 As with human-engineered radiomic metrics, the effectiveness of DLR features is highly dependent on the quality of the segmentation and the amount of training data. 8 However, segmentation is not a requirement for DL approaches if the desired result is image classification. Additionally, in some reviewed studies, qualitative imaging features that are complementary to quantitative radiomics features were used for ROI characterization.

In our findings, studies that investigated genomic characteristics associated with tumor biology using CT/PET or MRI/PET were reported in 13 of the total 87 studies. In addition, considering the dynamic nature of disease progression, radiomic features of importance may differ before and after treatment, therefore a change in the value of radiomic values named delta-radiomics may be more informative than conventional radiomics metrics. 47 The latter can provide an evaluation of treatment response during the process, and its usefulness has been observed for treatment outcomes on metastatic RCC. 48

A limitation of our study was a failure to evaluate the bias of the reviewed works of literature in our statistical analysis. There were some uncertainties in scoring categories such as clinical utility. For example, many radiomics studies have been investigating older systemic therapies, which no longer represent the current standard. Radiomics is an emerging technology with diverse applications, leading to a scenario where some of the RQS categories may become non-applicable in some scenarios, e.g., in the evaluation of DL-driven radiomics studies where it might not be well-suited to capture many of the challenges of involved in the training of DL networks. For example, having scored for multiple segmentations is meaningless if segmentation is not required in DL approaches, however, DL approaches with and without segmentation exist depending on the application. 45,46 Similarly, phantom studies are of reduced importance when no human interpretable features are extracted. The concept of feature reduction is not directly applicable to DL studies, as DLR metrics show high interfeature correlation, as they are all generated from the same data without the use of prior knowledge. However, rigorous validation steps in DL approaches reduce the risk of model overfitting. Therefore, while it is complicated to use RQS to score DL approaches, it does help identify common limitations with ML approaches. In the current study, DL studies were considered as an extension of ML studies and not evaluated in isolation. Newer RQS-based approaches such as the simplified and reproducible AI quality score (SRQS) reported by Lecointre et al, 49 which is based on simplified RQS categories and additional categories such as model design for DL approaches, better capture the differences in ML and DL in radiomics signature development. The term ‘no adherence’ could simply mean it was not reported, e.g., the study’s quality cannot be assessed whether the cost-effectiveness method was not performed. However, for an important element of the research method, for example, whether used an independent testing cohort, is conducted but not reported, it also means the author ignored the importance of a critical research methodology. Although by reviewing the published literature, we can never find out whether a non-reported item was not completed or ignored to report. Since both actions resulted in the same consequence with a negative impact on the quality of a published research article, we put them in the same bin. Another methodological assessment of radiomics studies is Quality Assessment of Diagnostic Accuracy Studies (QUADAS-2), which is a qualitative tool to evaluate the risk factors for bias in the fields of patient selection, index test, reference standard, and patient flow/timing. 50 However, while QUADAS-2 estimates the bias, it is still qualitative subject to interoperator interpretation bias. 32

Conclusion

Based on a systematic assessment using RQS evaluation, the overall quality of the reviewed radiomics studies assessing renal masses was suboptimal. Implementation and reporting of well-designed prospective radiomic studies and accounting for cost-effectiveness are strongly warranted. Using the RQS approach can provide systematic radiomics evaluation and reporting guidelines and help identify areas of improvement within the radiomics workflow to aid clinical translation.

Supplementary Material

bjr.20211211.suppl-01
bjr.20211211.suppl-01.docx (159.4KB, docx)

Footnotes

The authors Afshin Azadikhah and Bino Abel Varghese contributed equally to the work.

Author disclosures: VAD is a consultant to Radmetrix, Inc and Westat Inc. and is an Advisory Board Member to Deeptek, Inc.

Contributor Information

Afshin Azadikhah, Email: afshin.azadikhah@med.usc.edu, USC Radiomics Laboratory, Department of Radiology, Keck School of Medicine, University of Southern California, Los Angeles, United States .

Bino Abel Varghese, Email: bino.varghese@med.usc.edu, USC Radiomics Laboratory, Department of Radiology, Keck School of Medicine, University of Southern California, Los Angeles, United States .

Xiaomeng Lei, Email: Xiaomeng.Lei@med.usc.edu, USC Radiomics Laboratory, Department of Radiology, Keck School of Medicine, University of Southern California, Los Angeles, United States .

Chloe Martin-King, Email: chloe.martin-king@med.usc.edu, USC Radiomics Laboratory, Department of Radiology, Keck School of Medicine, University of Southern California, Los Angeles, United States .

Steven Yong Cen, Email: cen@usc.edu, USC Radiomics Laboratory, Department of Radiology, Keck School of Medicine, University of Southern California, Los Angeles, United States .

Vinay Anant Duddalwar, Email: Vinay.Duddalwar@med.usc.edu, USC Radiomics Laboratory, Department of Radiology, Keck School of Medicine, University of Southern California, Los Angeles, United States .

REFERENCES

  • 1. Siegel RL, Miller KD, Fuchs HE, Jemal A . Cancer statistics, 2022 . CA Cancer J Clin 2022. ; 72: 7 – 33 . doi: 10.3322/caac.21708 [DOI] [PubMed] [Google Scholar]
  • 2. McIntosh AG, Ristau BT, Ruth K, Jennings R, Ross E, Smaldone MC, et al. . Active surveillance for localized renal masses: tumor growth, delayed intervention rates, and >5-yr clinical outcomes . Eur Urol 2018. ; 74: 157 – 64 . doi: 10.1016/j.eururo.2018.03.011 [DOI] [PubMed] [Google Scholar]
  • 3. Lindenberg L, Mena E, Choyke PL, Bouchelouche K . PET imaging in renal cancer . Curr Opin Oncol 2019. ; 31: 216 – 21 . doi: 10.1097/CCO.0000000000000518 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 4. Belle A, Thiagarajan R, Soroushmehr SMR, Navidi F, Beard DA, Najarian K . Big data analytics in healthcare . Biomed Res Int 2015. ; 2015: 370194 . doi: 10.1155/2015/370194 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 5. Rutman AM, Kuo MD . Radiogenomics: creating a link between molecular diagnostics and diagnostic imaging . Eur J Radiol 2009. ; 70: 232 – 41 . doi: 10.1016/j.ejrad.2009.01.050 [DOI] [PubMed] [Google Scholar]
  • 6. Kumar V, Gu Y, Basu S, Berglund A, Eschrich SA, Schabath MB, et al. . Radiomics: the process and the challenges . Magn Reson Imaging 2012. ; 30: 1234 – 48 . doi: 10.1016/j.mri.2012.06.010 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 7. Parekh V, Jacobs MA . Radiomics: a new application from established techniques . Expert Rev Precis Med Drug Dev 2016. ; 1: 207 – 26 . doi: 10.1080/23808993.2016.1164013 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 8. Avanzo M, Wei L, Stancanello J, Vallières M, Rao A, Morin O, et al. . Machine and deep learning methods for radiomics . Med Phys Mex Symp Med Phys 2020. ; 47: e185 - 202 . doi: 10.1002/mp.13678 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 9. de Leon AD, Kapur P, Pedrosa I . Radiomics in kidney cancer . Magnetic Resonance Imaging Clinics of North America 2019. ; 27: 1 – 13 . doi: 10.1016/j.mric.2018.08.005 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 10. Lu L, Ehmke RC, Schwartz LH, Zhao B, Tian J . Assessing agreement between radiomic features computed for multiple CT imaging settings . PLoS ONE 2016. ; 11: e0166550 : 12 . doi: 10.1371/journal.pone.0166550 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 11. Lambin P, Leijenaar RTH, Deist TM, Peerlings J, de Jong EEC, van Timmeren J, et al. . Radiomics: the bridge between medical imaging and personalized medicine . Nat Rev Clin Oncol 2017. ; 14: 749 – 62 . doi: 10.1038/nrclinonc.2017.141 [DOI] [PubMed] [Google Scholar]
  • 12. Tsougos I, Vamvakas A, Kappas C, Fezoulidis I, Vassiou K . Application of radiomics and decision support systems for breast MR differential diagnosis . Comput Math Methods Med 2018. ; 2018: 7417126 . doi: 10.1155/2018/7417126 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 13. Lee G, Park H, Bak SH, Lee HY . Radiomics in lung cancer from basic to advanced: current status and future directions . Korean J Radiol 2020. ; 21: 159 – 71 . doi: 10.3348/kjr.2019.0630 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 14. Huhdanpaa H, Hwang D, Cen S, Quinn B, Nayyar M, Zhang X, et al. . CT prediction of the fuhrman grade of clear cell renal cell carcinoma (RCC): towards the development of computer-assisted diagnostic method . Abdom Imaging 2015. ; 40: 3168 – 74 . doi: 10.1007/s00261-015-0531-8 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 15. Bhindi B, Thompson RH, Lohse CM, Mason RJ, Frank I, Costello BA, et al. . The probability of aggressive versus indolent histology based on renal tumor size: implications for surveillance and treatment . Eur Urol 2018. ; 74: 489 – 97 . doi: 10.1016/j.eururo.2018.06.003 [DOI] [PubMed] [Google Scholar]
  • 16. Larue R, Defraene G, De Ruysscher D, Lambin P, van Elmpt W . Quantitative radiomics studies for tissue characterization: a review of technology and methodological procedures . Br J Radiol 2017. ; 90: 20160665 . doi: 10.1259/bjr.20160665 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 17. Afshar P, Mohammadi A, Plataniotis KN, Oikonomou A, Benali H . From handcrafted to deep-learning-based cancer radiomics: challenges and opportunities . IEEE Signal Process Mag 2019. ; 36: 132 – 60 . doi: 10.1109/MSP.2019.2900993 [DOI] [Google Scholar]
  • 18. van Timmeren JE, Cester D, Tanadini-Lang S, Alkadhi H, Baessler B . Radiomics in medical imaging-"how-to" guide and critical reflection . Insights Imaging 2020. ; 11( 1 ): 91 . doi: 10.1186/s13244-020-00887-2 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 19. Rizzo S, Botta F, Raimondi S, Origgi D, Fanciullo C, Morganti AG, et al. . Radiomics: the facts and the challenges of image analysis . Eur Radiol Exp 2018. ; 2: 36 . doi: 10.1186/s41747-018-0068-z [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 20. Ligero M, Jordi-Ollero O, Bernatowicz K, Garcia-Ruiz A, Delgado-Muñoz E, Leiva D, et al. . Minimizing acquisition-related radiomics variability by image resampling and batch effect correction to allow for large-scale data analysis . Eur Radiol 2021. ; 31: 1460 – 70 . doi: 10.1007/s00330-020-07174-0 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 21. Shur J, Blackledge M, D’Arcy J, Collins DJ, Bali M, O’Leach M, et al. . MRI texture feature repeatability and image acquisition factor robustness, a phantom study and in silico study . Eur Radiol Exp 2021. ; 5: 2 . doi: 10.1186/s41747-020-00199-6 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 22. Ge L, Chen Y, Yan C, Zhao P, Zhang P, A R, et al. . Study progress of radiomics with machine learning for precision medicine in bladder cancer management . Front Oncol 2019. ; 9: 1296 . doi: 10.3389/fonc.2019.01296 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 23. Sharma N, Aggarwal LM . Automated medical image segmentation techniques . J Med Phys 2010. ; 35: 3 – 14 . doi: 10.4103/0971-6203.58777 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 24. Lei M, Varghese B, Hwang D, Cen S, Lei X, Azadikhah A, et al. . Benchmarking features from different radiomics toolkits/toolboxes using image biomarkers standardization initiative . 2020. .
  • 25. Thrall JH, Li X, Li Q, Cruz C, Do S, Dreyer K, et al. . Artificial intelligence and machine learning in radiology: opportunities, challenges, pitfalls, and criteria for success . J Am Coll Radiol 2018. ; 15: 504 – 8 . doi: 10.1016/j.jacr.2017.12.026 [DOI] [PubMed] [Google Scholar]
  • 26. Aktolun C . Artificial intelligence and radiomics in nuclear medicine: potentials and challenges . Eur J Nucl Med Mol Imaging 2019; 46: 2731–36. 10.1007/s00259-019-04593-0 [DOI] [PubMed] [Google Scholar]
  • 27. McInnes MDF, Moher D, Thombs BD, McGrath TA, Bossuyt PM, and the PRISMA-DTA Group. et al. . Preferred reporting items for a systematic review and meta-analysis of diagnostic test accuracy studies: the PRISMA-DTA statement . JAMA 2018. ; 319: 388 – 96 . doi: 10.1001/jama.2017.19163 [DOI] [PubMed] [Google Scholar]
  • 28. Campbell M, McKenzie JE, Sowden A, Katikireddi SV, Brennan SE, Ellis S, et al. . Synthesis without meta-analysis (swim) in systematic reviews: reporting guideline . BMJ 2020. ; 368: l6890 . doi: 10.1136/bmj.l6890 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 29. Cumpston M, Li T, Page MJ, Chandler J, Welch VA, Higgins JP, et al. . Updated guidance for trusted systematic reviews: a new edition of the cochrane handbook for systematic reviews of interventions . Cochrane Database Syst Rev 2019. ; 10: ED000142 . doi: 10.1002/14651858.ED000142 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 30. Lambin P. Radiomics Quality Score - RQS . The department is headed by prof.dr . 2017. . Available from : https://www.radiomics.world
  • 31. Sanduleanu S, Woodruff HC, de Jong EEC, van Timmeren JE, Jochems A, Dubois L, et al. . Tracking tumor biology with radiomics: A systematic review utilizing A radiomics quality score . Radiother Oncol 2018. ; 127: 349 – 60 . doi: 10.1016/j.radonc.2018.03.033 [DOI] [PubMed] [Google Scholar]
  • 32. Ursprung S, Beer L, Bruining A, Woitek R, Stewart GD, Gallagher FA, et al. . Radiomics of computed tomography and magnetic resonance imaging in renal cell carcinoma-a systematic review and meta-analysis . Eur Radiol 2020. ; 30: 3558 – 66 . doi: 10.1007/s00330-020-06666-3 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 33. Mühlbauer J, Egen L, Kowalewski K-F, Grilli M, Walach MT, Westhoff N, et al. . Radiomics in renal cell carcinoma-A systematic review and meta-analysis . Cancers (Basel) 2021. ; 13( 6 ): 1348 . doi: 10.3390/cancers13061348 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 34. Bhandari A, Ibrahim M, Sharma C, Liong R, Gustafson S, Prior M . CT-based radiomics for differentiating renal tumours: a systematic review . Abdom Radiol (NY) 2021. ; 46: 2052 – 63 . doi: 10.1007/s00261-020-02832-9 [DOI] [PubMed] [Google Scholar]
  • 35. Mak HKF, Yau KKW, Chan BPL . Prevalence-adjusted bias-adjusted kappa values as additional indicators to measure observer agreement . Radiology 2004; 232: 302–3. 10.1148/radiol.2321031974 [DOI] [PubMed] [Google Scholar]
  • 36. Park CJ, Park YW, Ahn SS, Kim D, Kim EH, Kang S-G, et al. . Quality of radiomics research on brain metastasis: A roadmap to promote clinical translation . Korean J Radiol 2022. ; 23: 77 – 88 . doi: 10.3348/kjr.2021.0421 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 37. Won SY, Park YW, Park M, Ahn SS, Kim J, Lee SK . Quality reporting of radiomics analysis in mild cognitive impairment and alzheimer’s disease: A roadmap for moving forward . Korean J Radiol 2020. ; 21: 1345 – 54 . doi: 10.3348/kjr.2020.0715 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 38. Wang Q, Li C, Zhang J, Hu X, Fan Y, Ma K, et al. . Radiomics models for predicting microvascular invasion in hepatocellular carcinoma: A systematic review and radiomics quality score assessment . Cancers (Basel) 2021; 13(22): 5864. 10.3390/cancers13225864 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 39. Da-Ano R, Masson I, Lucia F, Doré M, Robin P, Alfieri J, et al. . Performance comparison of modified combat for harmonization of radiomic features for multicenter studies . Sci Rep 2020. ; 10( 1 ): 10248 . doi: 10.1038/s41598-020-66110-w [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 40. Zwanenburg A, Vallières M, Abdalah MA, Aerts HJWL, Andrearczyk V, Apte A, et al. . The image biomarker standardization initiative: standardized quantitative radiomics for high-throughput image-based phenotyping . Radiology 2020. ; 295: 328 – 38 . doi: 10.1148/radiol.2020191145 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 41. Echegaray S, Gevaert O, Shah R, Kamaya A, Louie J, Kothary N, et al. . Core samples for radiomics features that are insensitive to tumor segmentation: method and pilot study using CT images of hepatocellular carcinoma . J Med Imaging (Bellingham) 2015. ; 2: 041011 . doi: 10.1117/1.JMI.2.4.041011 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 42. Liu Y, Sung K, Yang G, Afshari Mirak S, Hosseiny M, Azadikhah A, et al. . Automatic prostate zonal segmentation using fully convolutional network with feature pyramid attention . IEEE Access 2019. ; 7: 163626 – 32 . doi: 10.1109/ACCESS.2019.2952534 [DOI] [Google Scholar]
  • 43. Liu Y, Yang G, Hosseiny M, Azadikhah A, Mirak SA, Miao Q, et al. . Exploring uncertainty measures in bayesian deep attentive neural networks for prostate zonal segmentation . IEEE Access 2020. ; 8: 151817 – 28 . doi: 10.1109/ACCESS.2020.3017168 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 44. Papadimitroulas P, Brocki L, Christopher Chung N, Marchadour W, Vermet F, Gaubert L, et al. . Artificial intelligence: deep learning in oncological radiomics and challenges of interpretability and data harmonization . Phys Med 2021. ; 83: 108 – 21 . doi: 10.1016/j.ejmp.2021.03.009 [DOI] [PubMed] [Google Scholar]
  • 45. Esteva A, Kuprel B, Novoa RA, Ko J, Swetter SM, Blau HM, et al. . Dermatologist-level classification of skin cancer with deep neural networks . Nature 2017. ; 542: 115 – 18 . doi: 10.1038/nature21056 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 46. Gulshan V, Peng L, Coram M, Stumpe MC, Wu D, Narayanaswamy A, et al. . Development and validation of a deep learning algorithm for detection of diabetic retinopathy in retinal fundus photographs . JAMA 2016. ; 316: 2402 . doi: 10.1001/jama.2016.17216 [DOI] [PubMed] [Google Scholar]
  • 47. Jeon SH, Song C, Chie EK, Kim B, Kim YH, Chang W, et al. . Delta-radiomics signature predicts treatment outcomes after preoperative chemoradiotherapy and surgery in rectal cancer . Radiat Oncol 2019. ; 14( 1 . doi: 10.1186/s13014-019-1246-8 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 48. Goh V, Ganeshan B, Nathan P, Juttla JK, Vinayan A, Miles KA . Assessment of response to tyrosine kinase inhibitors in metastatic renal cell cancer: CT texture as a predictive biomarker . Radiology 2011. ; 261: 165 – 71 . doi: 10.1148/radiol.11110264 [DOI] [PubMed] [Google Scholar]
  • 49. Lecointre L, Dana J, Lodi M, Akladios C, Gallix B . Artificial intelligence-based radiomics models in endometrial cancer: A systematic review . European Journal of Surgical Oncology 2021. ; 47: 2734 – 41 . doi: 10.1016/j.ejso.2021.06.023 [DOI] [PubMed] [Google Scholar]
  • 50. Whiting PF, Rutjes AW, Westwood ME, Mallett S, Deeks JJ, Reitsma JB, et al. . QUADAS-2: a revised tool for the quality assessment of diagnostic accuracy studies . Ann Intern Med 2011. ; 155: 529 – 36 . doi: 10.7326/0003-4819-155-8-201110180-00009 [DOI] [PubMed] [Google Scholar]

Associated Data

This section collects any data citations, data availability statements, or supplementary materials included in this article.

Supplementary Materials

bjr.20211211.suppl-01
bjr.20211211.suppl-01.docx (159.4KB, docx)

Articles from The British Journal of Radiology are provided here courtesy of Oxford University Press

RESOURCES