Discriminating malignant and benign clinical T1 renal masses on computed tomography

Supplemental Digital Content is available in the text

Abstract

The aim of this study was to discriminate malignant and benign clinical T1 renal masses on routinely acquired computed tomography (CT) images using radiomics and machine learning techniques.

Adult patients undergoing surgical resection and histopathological analysis of clinical T1 renal masses were included. Preoperative CT studies in venous phase from multiple referring centers were included, without restriction to specific CT scanners, slice thickness, or degrees of artifacts. Renal masses were segmented and 120 standardized radiomic features extracted. Machine learning algorithms were used to predict malignancy of renal masses using radiomics features and cross-validation. Diagnostic accuracy of machine learning models and assessment by independent blinded radiologists were compared based on the gold standard of histopathologic diagnosis.

A total of 94 patients met inclusion criteria (benign renal masses: n = 18; malignant: n = 76). CT studies from 18 different scanners were assessed with median slice thickness of 2.5 mm and artifacts in 15 cases (15.9%).

Area under the receiver-operating-characteristics curve (AUC) of random forest (random forest [RF], AUC = 0.83) was significantly higher compared to the radiologists (AUC = 0.68, P = .047). Sensitivity was significantly higher for RF versus radiologists (0.88 vs 0.80, P = .045), whereas specificity was numerically higher for RF (0.67 vs 0.50, P = .083).

Although limited by an overall small sample size and few benign renal tumors, a radiomic features and machine learning approach suggests a high diagnostic accuracy for discrimination of malignant and benign clinical T1 renal masses on venous phase CT. The presented algorithm robustly outperforms human readers in a real-life scenario with nonstandardized imaging studies from various referring centers.

1. Introduction

Renal cell carcinoma (RCC) is the most common renal malignancy worldwide, accounting for approximately 175,000 annual cancer-related deaths.^[1] In recent years, RCC incidence has been increasing, which is partially attributable to technical advancements and wider availability of cross-sectional imaging.^[2] In particular, the incidence of small renal masses has increased, now comprising up to 40% of all renal masses in the United States.^[3,4] Although the majority of renal masses are malignant, studies report that 16% to 19% of renal masses are benign.^[5,6]

Renal masses are often incidentally detected on cross-sectional imaging that has been performed for other indications.^[7] In these cases, imaging studies might not be optimized for characterization of renal masses and might, thus, lack crucial information. For example, routine computed tomography (CT) studies might lack contrast enhanced images in corticomedullary renal phase.

This monophasic imaging studies might complicate renal mass assessment, since classical CT features are not available, such as enhancement patterns over time.^[8] In patients with incidentally detected renal masses on monophasic studies, repeated imaging with dedicated multiphasic CT might be performed. Still, this exposes patients to radiation and high doses of iodinated contrast media, and even multiphasic CT studies do not allow unequivocal classification of renal masses in all cases.^[8] Nuclear medicine studies, magnetic resonance imaging, as well as ultrasound and renal biopsy might be indicated to further assess incidental renal masses. Still, these procedures might not be timely available for all patients and are themselves associated with costs and procedural risk. Therefore, the questions remain whether advanced image analyses could aid in assessment of incidental renal masses on monophasic CT studies.

Radiomic feature analyses and machine learning algorithms have recently been shown to perform well on a range of different radiological imaging types, including magnetic resonance imaging and CT.^[9,10] Still, to date there is no literature to evaluate the diagnostic performance of these techniques for the assessment of clinical T1 renal masses in a real-world setting with various CT scanners, acquisition protocols, and potential artifacts.

The aim of this study was to train a machine learning algorithm for discrimination of malignant and benign clinical T1 renal masses that were detected on venous CT and compare its diagnostic performance to experienced radiologists.

2. Material and methods

This retrospective, STARD-compliant study received previous approval by the ethics committee of the University Medical Center Goettingen (No 2/4/17) and is compliant with the Declaration of Helsinki.

2.1. Patient inclusion

Adult patients presenting for surgical resection of renal masses consecutively between 2012 and 2017 at the University Medical Center Goettingen were considered for inclusion if preoperative, contrast-enhanced CT studies in venous phase were available. Only renal masses radiologically staged T1 with maximal diameter of 70 mm in any direction were included in this study. Exclusion criteria were diffuse infiltrative renal disease (ie, lymphoma) and primarily cystic lesions. A study flow-chart is provided in Figure 1.

Figure 1.

Study flow chart and prospective clinical application.

2.2. Radiological Imaging

In a pragmatic, real-life approach, we evaluated contrast-enhanced CT studies in venous phase only. Our study was not restricted to CT studies performed at our tertiary center but included patients that were referred with imaging from external centers as well. No restrictions were made regarding CT scanner, slice thickness, or imaging artifacts.

2.3. Radiomic features

The open source software 3D Slicer was used for renal mass segmentation and radiomic feature analyses.^[11] Radiomic features in 3D slicer are based on standardized and reproducible algorithms in accordance with feature definitions by the Imaging Biomarker Standardization Initiative.^[12,13] A bin width of 25 was chosen.

Renal masses were segmented (delineation of region of interest) on axial venous CT images by 2 radiologists in consensus. The total number of assessed CT slices varied with renal mass size and CT slice thickness.

From the segmented renal masses, a total of 120 distinct radiomic features were analyzed, with extended details provided online.^[12] Radiomic features are subdivided into 8 classes:

1. First-order statistic (describing renal mass voxel intensity)

2. 3D shape features (describing 3-dimensional size and shape of renal mass)

3. 2D shape features (describing 2-dimensional size and shape of renal mass)

4. Gray-level co-occurrence matrix features (GLCM; describing second-order joint probability function of renal mass)

5. Gray-level size zone matrix features (GLSZM; quantifying gray level zones in renal mass)

6. Gray-level run length matrix features (GLRLM; quantifying gray level runs in renal mass)

7. Neighboring gray tone difference matrix features (NGTDM; quantifying the difference between a gray value and average gray value of its neighbors in renal mass)

8. Gray-level dependence matrix features (GLDM; quantifying gray level dependencies of renal mass).

2.4. Renal mass assessment

As criterion standard, all renal mass specimens underwent histopathological assessment at the Department of Pathology, University Medical Center Goettingen. Renal masses were categorized as malignant (including clear cell, chromophobe, and papillary RCC) and benign (including oncocytoma and angiomyolipoma [AML]). Histopathological assessment was performed on partial or radical nephrectomy specimens using hematoxylin-eosin staining, and immunostaining for cytokeratin 7, CD10, CD117, as well as Vimentin, following international recommendations.^[14,15] Diagnosis of AMLs was further based on Melan-A, human melanoma black 45 and actin staining.^[16,17] Representative histopathological slides of malignant and benign renal masses are presented in the appendix.

All CT studies were independently assessed by 2 radiologists (with 3 and 5 years of dedicated experience in abdominal imaging) that were blinded to each other and final histopathological diagnosis. Renal masses were radiologically assessed using a Likert scale defining the probability of malignancy (POM) ranging from 1 (definitely benign renal mass) to 10 (definitely malignant) with increments of 1.

2.5. Machine learning

In a first step, preprocessing of the radiomic features was conducted with centering (subtracting the mean from individual values) and scaling (dividing values by standard deviation) of each feature.

Second, a feature selection was conducted using recursive feature elimination (RFE). For RFE, a full logistic regression model was fit with all potential predictors, ranking the importance of each predictor. At each RFE iteration, only the most important predictors were retained; the model was refitted followed by an assessment of its diagnostic accuracy. In this study, RFE was conducted using 10-fold cross-validation to avoid overfitting. For this cross-validation, the full dataset is divided into 10 subsamples of which 9 subsamples are used as training data and the remaining subset for testing. The process is repeated 10 times using each subset for validation once.

Third, machine learning algorithms were modeled to predict the probability of malignancy of a specific renal mass (outcome) given its radiomic features (predictors). Several machine learning algorithms were considered a priori according to Wolperts’ no free lunch theorem.^[18] The following machine learning algorithms were trained: extreme gradient boosting (XG boost), random forest (RF), neural network, support vector machines (SVM), and k-nearest neighbors. Details for each machine learning algorithm have been published previously and are provided in a short summary in the appendix. All machine learning algorithms were trained and tested using a leave-one-out cross validation: training was conducted on n-1 observations and the model performance tested on the left-out observation. This procedure was repeated n-times to obtain the final model.

2.6. Diagnostic performance assessment

Diagnostic accuracy was assessed using the receiver-operating characteristics curve (ROC) and the area under the ROC curve (AUC). AUCs of radiologists and machine learning algorithms were compared using 2000 bootstrap samples. The Youden Index was used to determine the optimal cutoff values from radiologists and machine learning algorithms for calculation of sensitivity and specificity. The McNemar test was used for comparison of sensitivity and specificity between radiologists and machine learning algorithms. Interobserver agreement was evaluated comparing renal mass assessment between both radiologists; intraobserver agreement was evaluated comparing renal mass assessment of one radiologist at 2 separate timepoints. Inter- and intraobserver agreement were quantified using the inter-/intraobserver correlation coefficient (ICC) ranging from 0–1.^[19] ICC <0.4 was rated as poor, 0.4 to 0.59 as fair, 0.6 to 0.74 as good, and 0.75 to 1 as excellent. Readings of both radiologists were combined to simulate an “average radiologist” for comparison of diagnostic accuracy to the machine learning algorithms.

Machine learning algorithm implementation and statistical analyses were performed using R and RStudio ^[20,21] with the R package “caret.”^[22] An alpha level of 0.05 was chosen to indicate statistical significance. All provided P values are 2-sided.

3. Results

3.1. Study cohort

A total of 94 patients met were included in our study (female, n = 28, 29.8%; male, n = 66, 70.2%) with median age of 64.4 years (interquartile range [IQR]: 54.9–73 years). The histopathological assessment revealed 76 malignant lesions (clear cell RCC, n = 67; papillary RCC, n = 7; chromophobe RCC, n = 2) and 18 benign lesions (oncocytoma, n = 9; AML, n = 9). A study flowchart is provided in the appendix.

3.2. Radiological imaging and assessment

The median renal mass diameter was 46.5 mm (IQR: 35–56.8 mm). Radiological imaging was acquired from 18 different CT scanners (see appendix for further details) with median slice thickness of 2.5 mm (IQR: 1–5 mm). Imaging artifacts were present in 15 CT studies (15.9%).

Interobserver agreement in renal mass assessment between both radiologists was fair with an ICC = 0.513. As shown in Figure 2, interobserver agreement was good for those renal masses with very low or very high average probability of malignancy. Intraobserver agreement for repeated renal mass assessment of one radiologist was fair with ICC = 0.435.

Figure 2.

Dotplot and Whisker-boxplot depicting the discrepancies between both radiologists’ probability of malignancy (POM) assessment of renal lesions. Notably, the radiologists’ discrepancies are smaller in cases with very low (1–2) or very high (9–10) POM.

3.3. Machine learning algorithms versus radiological assessment

Table 1 summarizes the diagnostic accuracy of different machine learning algorithms to predict renal mass malignancy. Among the machine learning algorithms, RF achieved the numerically highest AUC = 0.83.

Table 1.

Diagnostic accuracy of machine learning algorithms and radiologist readers measured by AUC. Sensitivity and specificity obtained using the Youden Index.

As demonstrated in Figure 3, the AUC of RF (0.83) was higher when compared to the radiologists (AUC = 0.68, P = .047). According to the Youden Index, the optimal cut-off to distinguish benign and malignant lesions for the RF algorithm was 67% and for the radiologists a POM 5/10. After dichotomization, RF sensitivity (0.88) was significantly higher than the radiologists (sensitivity = 0.80, P = .045). RF specificity (0.67) was numerically higher, but did not reach statistical significance (radiologist specificity = 0.50, P = 0.083). Notably, cases of classical AMLs with macroscopic fat (n = 5) were assigned a low POM score by both radiologists. Case studies for malignancy prediction of renal masses are provided in Figures 4 and 5.

Figure 3.

Receiver-operating characteristic curves for random forest machine learning algorithm and radiologists. POM = probability of malignancy.

Figure 4.

Case study of a 38-year-old male patient presenting with heterogeneous renal mass of the right upper ventral pole (arrow; A axial plane; B 3D reformation with renal mass highlighted in green). Relevant beam-hardening artifacts are evident from dorsal instrumentation. Probability of malignancy was 99% by random forest machine learning, and rated 10/10 by radiologist 1 and 7/10 by radiologist 2. Histopathology diagnosed a clear cell renal cell carcinoma.

Figure 5.

Case study of a 73 year-old female presenting with heterogeneous right-sided renal mass of the lateral circumference (arrow; A axial plane; B 3D reformation with renal mass highlighted in green). Probability of malignancy was 54% by random forest machine learning (and therefore rated as “benign” using the Youden Index cutoff at 67%), and rated 6/10 radiologist 1 and 7/10 by radiologist 2 (therefore “malignant” using the Youden Index cutoff at 5/10). Diagnosis upon histopathological assessment was oncocytoma.

4. Discussion

The discrimination of malignant and benign renal masses is an ongoing radiological challenge, especially in the case of CT studies not specifically tailored to renal imaging and small renal lesions.

In our study, radiomic features and machine learning algorithms demonstrated a high diagnostic accuracy for prediction of renal mass malignancy on preoperative, venous CT. The final RF algorithm robustly performed on a heterogeneous population with CT studies acquired on a range on different scanners. Even in cases with large slice thickness, beam-hardening or motion artifacts, accurate malignancy predictions were achieved. This robustness corroborates the utility of machine learning algorithms in a real-life clinical scenario.

Among evaluated machine learning algorithms, RF demonstrated superior performance. Due to their algorithmic characteristics, RF excel in cases of high-dimensional and highly correlated data, such as the radiomic features analyzed in this study.^[23]

Compared to 2 experienced radiologists, the RF algorithm demonstrated superior diagnostic accuracy for renal mass assessment. Although sensitivity and specificity were numerically higher for the RF algorithm after dichotomization using the Youden index, no statistically significant differences were evident compared to the radiologists. This could be attributed to a lack of statistical power given the limited sample size.

The observed suboptimal interobserver agreement between both radiologists underlines the need for additional diagnostic tools to reliably assess renal masses. Notably, the radiologists performed well in cases with macroscopic fat, which were accurately rated as AMLs. The overall lower diagnostic accuracy of the radiologists compared to the RF algorithm might therefore be driven by cases with fat-poor AMLs and oncocytomas, which were falsely described as malignant masses by the radiologists.

The study cohorts’ characteristics of our study are in line with population-based analyses, showing a male predominance and peak incidence between age 60 and 70.^[24,25] Further, the proportion of benign renal masses with 19% in our study is comparable to the literature ranging from 20% to 30%.^[5,6]

In renal imaging, machine learning studies are scarce. Recently, Feng et al used SVMs to discriminate fat-poor AMLs from RCC in 58 patients.^[26] Kocak et al^[27] evaluated 68 RCC cases and were able to discriminate histological subtypes with moderate accuracy. Yu et el^[9] evaluated radiomics and machine learning for discrimination of histological RCC subtypes in 119 patients. Initial studies also highlighted that RCC mutational status correlated with CT imaging features and were predictable using machine learning algorithms.^[28,29] Although these studies showed promising results, most of them were designed to evaluate CT studies which were acquired in a standardized and partially multiphasic manner, which might limit their applicability to a heterogeneous clinical setting. Our study, in contrast, demonstrated the feasibility and accuracy of radiomic feature analyses and machine learning algorithms even in a pragmatic scenario with diverse CT scanners and relevant artifacts.

Our RF algorithm using only venous CT imaging demonstrated similar diagnostic accuracy to previous multiphasic studies: using contrast enhancement analyses to differentiate renal mass subgroups in 200 patients with multiphasic CT, Coy et al^[30] reported an AUC = 0.85, which is comparable to findings in our study (AUC = 0.83).

It still remains unclear, how exactly radiomic features discriminate benign and malignant renal masses. One hypothesis might be that radiomic features quantify neovascularity and malignant cell transformation. Another potential mechanism is the correlation of radiomic features with well described alterations in renal cancer cell metabolism, such as the metabolic flux through glycolysis, mitochondrial bioenergetics, and oxidative phosphorylation.^[31–33]

Further, it has to be highlighted that radiomic feature analyses and machine learning are not the only method to discriminate between benign and malignant renal masses. In recent years, there have been several reports on the emerging role of biomarkers in renal cancer diagnosis: for example, Papale et al described RKIP/p-RKIP as an urinary biomarker for ccRCC, whereas Lucarelli et al reported on serum circulating CA 15–3, CA 125 and beta-2 microglobulin as prognostic markers in RCC.^[34,35] Further studies have been published on the prognostic impact of the autocrine motility factor, soluble serum αKlotho, and kynurine pathway in ccRCC.^[36–38] It remains to be evaluated whether the combination of radiomic feature analyses and biomarkers can further assist in accurate renal mass diagnosis.

Our study is not devoid of limitations: first, only patients of white race and from a restricted European region were evaluated, which may limit the generalizability of our findings. Still, there is no literature available on race-specific differences in radiologic appearance of renal mass. Second, our sample size was not large enough to allow for an independent validation dataset and results might thus be overfitted to our specific population. Specifically, the class imbalance with a small number of benign cases limits the generalizability of our findings. Third, we excluded primarily cystic and diffuse infiltrative renal masses, as well as those >70 mm in diameter. Thereby, the proposed machine learning algorithm cannot be applied to any case presenting in clinical routine. Nevertheless, diffuse infiltrative renal disease is clinically rare with primary renal lymphoma reportedly accounting for <1% of renal lesions.^[39] For renal masses of >70-mm diameter, patient stratification is less important than for smaller renal masses, as even benign masses like oncocytomas might be resected due to risk of hemorrhage.^[40] Using a 2-reader consensus approach in our study, the variability of renal mass segmentation and its effect on malignancy prediction were not assessible. Finally, no validation dataset was available for measuring the algorithm's performance in unknown data. To bypass this shortcoming, we used cross-validation which, though being widely accepted as an internal validation method, lags behind a completely independent, external validation dataset, which will be the subject of future studies.

Despite this study's’ apparent limitation, its innovative approach and future implementations should be highlighted: our pragmatic approach carries the potential for developing a clinically applicable software that not only supports radiologists in daily routine, but also multidisciplinary tumor board decisions. Further research should aim to validate and improve our algorithm on independent datasets. Moreover, automated renal mass segmentation has a high potential to streamline implementation of our radiomic and machine learning approach, which further lowers the threshold for clinical application.

5. Conclusions

Our study suggests that radiomic features and machine learning yield good diagnostic accuracy for discrimination of malignant and benign renal masses on CT studies, which was significantly higher than that of radiologists. Although limited by a small sample size and low number of benign renal masses, the presented RF algorithm robustly performs in a real-life scenario with nonstandardized CT studies from various referring centers. Further studies should aim to validate our findings in independent datasets. A future clinical pipeline should incorporate not only radiomic analyses and machine learning algorithms, but also automated detection and segmentation of renal lesions to streamline renal mass diagnostics.

Author contributions

Conceptualization: Johannes Uhlig, Manuel M Nietert, Tim Beißbarth, Joachim Lotz, Hyun S. Kim, Lutz Trojan, Annemarie Uhlig.

Data curation: Johannes Uhlig, Lorenz Biggemann, Manuel M Nietert, Annemarie Uhlig.

Formal analysis: Johannes Uhlig, Manuel M Nietert, Tim Beißbarth, Annemarie Uhlig.

Investigation: Johannes Uhlig, Lorenz Biggemann, Joachim Lotz, Lutz Trojan, Annemarie Uhlig.

Methodology: Johannes Uhlig, Lorenz Biggemann, Tim Beißbarth, Joachim Lotz, Hyun S. Kim, Annemarie Uhlig.

Project administration: Johannes Uhlig, Joachim Lotz, Hyun S. Kim, Lutz Trojan, Annemarie Uhlig.

Resources: Johannes Uhlig, Tim Beißbarth, Joachim Lotz, Hyun S. Kim, Lutz Trojan, Annemarie Uhlig.

Software: Joachim Lotz, Hyun S. Kim, Lutz Trojan.

Supervision: Johannes Uhlig, Manuel M Nietert, Tim Beißbarth, Joachim Lotz, Hyun S. Kim, Lutz Trojan, Annemarie Uhlig.

Validation: Johannes Uhlig, Lorenz Biggemann, Annemarie Uhlig.

Visualization: Annemarie Uhlig.

Writing – original draft: Annemarie Uhlig.

Writing – review & editing: Johannes Uhlig, Lorenz Biggemann, Manuel M Nietert, Tim Beißbarth, Joachim Lotz, Hyun S. Kim, Lutz Trojan, Annemarie Uhlig.

Johannes Uhlig orcid: 0000-0003-3557-3194.