Performing Automatic Identification and Staging of Urothelial Carcinoma in Bladder Cancer Patients Using a Hybrid Deep-Machine Learning Approach

Abstract

Simple Summary

Early and accurate bladder cancer staging is important as it determines the mode of initial treatment. Non-muscle invasive bladder cancer (NMIBC) can be treated with transurethral resection whereas muscle invasive bladder cancer (MIBC) requires neoadjuvant chemotherapy with subsequent cystectomy as indicated. Our hybrid machine/deep learning model demonstrates improved accuracy of bladder cancer staging by CECT using a hybrid machine/deep learning model which will facilitate appropriate clinical management of the patients with bladder cancer, ultimately improving patient outcome.

Abstract

Accurate clinical staging of bladder cancer aids in optimizing the process of clinical decision-making, thereby tailoring the effective treatment and management of patients. While several radiomics approaches have been developed to facilitate the process of clinical diagnosis and staging of bladder cancer using grayscale computed tomography (CT) scans, the performances of these models have been low, with little validation and no clear consensus on specific imaging signatures. We propose a hybrid framework comprising pre-trained deep neural networks for feature extraction, in combination with statistical machine learning techniques for classification, which is capable of performing the following classification tasks: (1) bladder cancer tissue vs. normal tissue, (2) muscle-invasive bladder cancer (MIBC) vs. non-muscle-invasive bladder cancer (NMIBC), and (3) post-treatment changes (PTC) vs. MIBC.

1. Introduction

Bladder cancer imaging can be misleading. Findings such as perivesical fat stranding, hydronephrosis, focal bladder wall thickening, or a small bladder lesion may be wrongly perceived as a more advanced stage of bladder cancer. It is common to see small lymph nodes in the pelvis post transurethral resection of bladder tumor (TURBT) [1,2,3] or at the time of diagnosis [4]. These otherwise non-significant lymph nodes may harbor metastasis, which in turn might be difficult to decipher relying solely on visual inspection from CT scans [5,6]. Furthermore, bladder cancer is a heterogeneous disease with an extremely varied range of case-specific diagnoses [7,8]. From low-grade tumors such as $T_a$ (noninvasive papillary carcinoma) and $T_{is}$ (tumor in situ or “flat lesion”), which require TURBT or less aggressive endoscopic intervention [8], to high-grade muscle-invasive tumors, which require chemotherapy [9,10], bladder cancer diagnosis is highly dependent on the type and stage of the tumor [4,6,11].

Radiomics can provide tools to significantly improve the accuracy of clinical staging by analyzing multiple qualitative features, including but not restricted to texture analysis, raw digital data and deep model-generated embeddings.

Texture analysis helps capture local patterns in images from the intensity information contained within them. Such features are very effective in identifying tissue types from grayscale medical scans. The texture-based features generally used in medical imaging can be broadly categorized into two subdivisions, namely statistical approaches and transformation-based approaches.

First, we will review some of the most commonly used statistical texture features for bladder cancer detection and staging from medical scans. Ref. [12] utilized a set of functional, second-order statistical and morphological features to perform staging between T1 and T2 types of bladder cancer from MRI scans. The second-order statistical feature extraction approach included a total of 25 GLCM features and 16 gray level run length matrix (GLRLM) features obtained from 42 bladder MRIs (21 T1, 21 T2) post ROI segmentation, yielding an accuracy, sensitivity and specificity of 95.24% and an AUC of 98.64%. Furthermore, the authors also performed an extensive comparison between their proposed model and two state-of-the-art approaches, namely [13,14]. Ref. [15] makes use of LBP and GLCM features to perform primary tumor staging of bladder cancer into two groups: (1) tumor stage and primary tumor located completely within the bladder; (2) tumor stage and primary tumor extending outside the bladder. The SVM classifier, which was trained on T2-weighted MRI scans from 65 bladder cancer patients all with stage 1, reported an AUC of 80.60%. Ref. [16] performs prediction of recurrence and progression of urothelial carcinoma from a dataset of 42 patients—13 without recurrence, 14 with recurrence but not progression, and 15 with progression. Features extracted using LBP and local variance, after classification using the RUSBoost classifier, provided an accuracy of 70% and sensitivity of 84%. Ref. [17] utilizes LBP and GLCM features to classify the invasiveness of bladder cancer. The dataset comprised T2-weighted MRI scans from 65 preoperative bladder cancer patients followed by radical cystectomy. The proposed model reported a patient-level sensitivity of 74.20%, specificity of 82.40%, accuracy of 78.50% and AUC of 80.60%. Ref. [18] performed survival prediction of bladder urothelial carcinoma (BLCA) from CECT scans by utilizing LBP, wavelet and GLCM features. The dataset comprised scans from 62 bladder cancer patients with stages of urothelial carcinoma. The radiomics features extracted from the CECTs were used in combination with RNA-seq data for a complete radiogenomics signature, which in turn helped predict the survival of the patients. This study exhibits the applicability of radiomics and transcriptomics data in predicting BLCA survival. However, owing to the sheer small size of the dataset, the authors think that the model needs to be validated on a larger set of samples for a more foolproof analysis. The literature exhibits that statistical texture analysis is not only fast and easy to implement but also very effective in performing classification, staging and segmentation of bladder cancer from medical images.

Transformation-based approaches such as Fourier and Gabor wavelet transform are very effective in learning textural patterns from images and are therefore a popular option among medical imaging researchers. Ref. [19] utilized Gabor features to perform carcinoma cell classification from biopsy images associated with 14 distinct cancer types, scanned from 14 different patients. An SVM classifier with a radial basis function (RBF) kernel was subsequently trained on these features. The SVM classifier provides the highest cross-validation accuracy of 99.20% with a Gabor window size of 400 pixels and an image magnification of 10. Ref. [20] makes use of 2D Fourier-based features to identify cancer from 182 optical coherence tomography (OCT) scans obtained from 21 patients who were identified as high risk of having transitional cell carcinoma (TCC) and from 68 different areas of the bladder. The task was two-fold: (1) to perform classification between non-cancerous, dysplasia, carcinoma in situ (CIS), and papillary lesions; (2) to predict the invasiveness of the lesion. Other than 2D Fourier transform, the authors also extracted four different statistical feature extraction approaches, resulting in a total of 74 features obtained from the raw OCT images. A simple cross-correlation-based filter with a correlation threshold of 0.85 was employed for feature selection, which resulted in a final set of nine selected features. The decision tree classifier was utilized to finally perform classification on the selected feature set. The authors reported a non-cancerous versus cancerous classification sensitivity of 92.00% and specificity of 62.00%. Ref. [21], which has been summarized earlier, reports that GLCM and GLDM perform better than Fourier transform-based feature extractors in differentiating between tumors and peritumoral fat tissues.

From the literature reviewed above, it is evident that texture analysis is an effective approach when performing classification tasks on medical imaging, in general, and histologic analysis in particular.

Recently, deep learning-based models have emerged and gained popularity among researchers owing to their automatic feature extraction capabilities. Convolutional neural networks (CNNs) are the most commonly used type of deep models and a very popular framework when performing classification tasks on imaging-based applications, in general, and radiological data in particular. The authors of [22] designed a set of nine CNN-based models for the classification of MIBC and NMIBC from contrast-enhanced CT (CECT) images. The dataset comprised 1200 CT scans obtained from 369 patients undergoing radical cystectomy. A total of 249 out of these patients had NMIBC, while the remaining 120 had MIBC. The CNN model was pre-trained on the ImageNet dataset in order to improve classification performance. The model with the highest AUC on the test set was obtained using the VGG16 algorithm—with an AUC of 99.70%, accuracy of 93.90%, sensitivity of 88.90%, specificity of 98.90%, precision of 98.80% and negative predictive value of 89.90%. In contrast, the authors of [13] made use of Haralick features, which are a variant of GLCM, to identify muscular invasiveness in MRI scans containing a total of 118 volumes of interest (VOI) obtained from 68 patients—34 volumes labeled non-muscle-invasive bladder cancer (NMIBC), and 84 labeled muscle-invasive bladder cancer (MIBC). The final SVM classifier obtained an AUC of 86.10% and Youden index of 71.92%.

Ref. [21] performs classification between bladder cancers with and without response to chemotherapy from a set of CT scans obtained before and after treatment. The authors have reviewed three different models and their capabilities in performing classification: (1) a deep learning-convolutional neural network-based model (DL-CNN); (2) a more deterministic radiomics feature-based classifier (RF-SL); (3) an intermediate model that extracts radiomics features from image patterns (RF-ROI). The training dataset comprised 82 patients having 87 bladder cancers, scanned pre- and post-chemotherapy. The test set comprised 41 patients with 43 cancers. The radiomics feature-based model (RF-SL) performed the best with an AUC of 77.00%, while the two radiologists reported AUCs of 76.00% and 77.00%, respectively.

Ref. [23] proposed a CNN-based model for performing classification between low- and high-stage bladder cancer. The training dataset comprised 84 bladder cancer CR urography (CTU) images obtained from 76 patients (43 CTUs contained low-stage cancer, while 41 contained high-stage cancer). The test set consisted of 90 bladder CTUs obtained from 86 patients. The CNN classifier had a test set prediction accuracy of 91.00%, which the authors claim is higher as compared to texture-based classification using SVM on the same dataset (which had a prediction accuracy of 88.00%). In comparison, ref. [21] extracted GLCM- and histogram-based features from apparent diffusion coefficient (ADC) and diffusion-weighted images (DWI) to perform bladder cancer grading. A total of 61 patients were scanned for this study, 32 out of whom were in low-grade and the remaining 29 in high-grade classes. A combination of 102 GLCM and histogram features were initially extracted, out of which 47 were finally selected using the Mann–Whitney U-test, and an SVM classifier was used to perform classification between high- and low-grade bladder cancer with an accuracy of 82.90%. Ref. [21] extracted histogram and GLCM features to perform classification between high-grade and low-grade bladder cancer scans from a set of diffusion-based MRIs obtained from 61 bladder cancer patients (32 of them having low-grade and 29 having high-grade bladder cancer), yielding an accuracy of 82.90% and area under the curve (AUC) of 86.10%. Ref. [24] made use of GLCM, wavelet filter and Laplacian of Gaussian filter to extract features from a small dataset of 145 patients to perform grading on bladder cancer CT scans. Out of these 145 scans, 108 were used to train the model, and the remaining 37 were used to perform validation. The model provided an accuracy of 83.80% on the validation set.

From the literature reviewed above, it is evident that neural network-based classifiers are more effective than texture analysis when performing identification and staging of bladder cancer from medical imaging alone.

The authors of [25] claim that feature extraction when governed by domain knowledge performs better than CNN-based classifiers that are capable of automatic feature generation. The task in this case was to classify two early stages of bladder cancer that are histologically difficult to differentiate, namely Ta (non-invasive) and T1 (superficially invasive). The dataset comprised a total of 1177 bladder scans—460 non-invasive, 717 superficially invasive. CNN classifiers achieved the highest accuracy of 84.00%, performing considerably poorer than supervised machine learning classifiers that were trained on manually extracted features. The aforementioned literature on CNNs comprises end-to-end models, where both training and testing are performed on the same dataset. The problem with such an approach is that deep neural networks require large quantities of training data in order to avoid the pervasive issue of over-fitting. As a substitute to end-to-end deep models, researchers make use of a concept called “transfer learning”, where the neural network is first pre-trained on a large dataset such as ImageNet, and the learned weights are subsequently fine-tuned on the small target data. Since we were using a small dataset of 200 CT scans for this study, we decided to make use of transfer learning to improve classification results and alleviate overfitting. A 71-layer ResNet-18 model pre-trained on the publicly available ImageNet dataset was utilized to extract features from the bladder scans. The extracted features, after feature selection using a combination of supervised and unsupervised techniques, were finally used to perform classification by five different machine learning classifiers, namely k-nearest neighbor (KNN), support vector machine (SVM), linear discriminant analysis (LDA), decision tree (DT) and naive Bayes (NB).

2. Materials and Methods

Figure 1 provides a pictorial depiction of the entire workflow, starting from the raw bladder scans to the prediction labels obtained after classification. The proposed methodology comprises feature extraction, feature selection, and finally classification. In this hybrid approach, we extract feature vectors from the images using the trained model weights from the last pooling layer of the five most widely used pre-trained deep models, namely AlexNet, GoogleNet, InceptionV3, ResNet-50 and XceptionNet. We subsequently employ our feature selection algorithm on this feature vector. Finally, we use five machine learning classification algorithms on the selected feature set, namely k-nearest neighbor (KNN), naive Bayes (NB), support vector machine (SVM), linear discriminant analysis (LDA) and decision tree (DT).

Figure 1.

A schematic representation of the overall workflow.

2.1. Feature Extraction Using Pre-Trained Deep Models

Five popular neural network-based deep models, namely AlexNet, GoogleNet, InceptionV3, ResNet-50 and XceptionNet, all pre-trained on the ImageNet dataset [26], were trained using our bladder CT scan data to fine-tune the model parameters. The trained weights from the last pooling layer of each of these models were extracted and subsequently used as feature descriptors. Table 1 provides a comprehensive description of each of these models—including information regarding the pooling layer and the size of the extracted feature vector.

Table 1.

A description of the five pre-trained models used for feature extraction from the bladder scans, namely AlexNet, GoogleNet, InceptionV3, ResNet-50 and XceptionNet. The table contains the total number of layers per model; the last pooling layer from which features were being extracted; the layer number of the last pooling layer; and the length of the extracted feature vector.

Pre-Trained Model	Total No. of Layers in the Network	Last Pooling Layer	Layer No. of the Last Pooling Layer	Feature Vector Length
AlexNet	25	Max Pooling	16	9216
GoogleNet	144	Global Average Pooling	140	1024
InceptionV3	315	Global Average Pooling	312	2048
ResNet-50	177	Global Average Pooling	174	2048
XceptionNet	170	Global Average Pooling	167	2048

2.2. Feature Selection Mechanism

An ensemble feature selection technique was used to select the most important features from the originally extracted feature vector—which was obtained using five different pre-trained models, namely AlexNet, GoogleNet, Inception V3, ResNet-50 and XceptionNet, for performing the classification of normal vs. metastatic lymph nodes. First, we make use of a “sparsity filter” to remove the features that were not updated by the deep model. Through this step, we not only exclude features that have low variance but also automatically remove those that are less likely to have an impact on the classification process. Next, we utilize the “data imputer” to impute the unchanged values of the remaining columns by the mean value of the respective column. Subsequently, we made use of the “low coefficient of variation (CV) filter” to drop features with a CV value of less than 0.1. CV, or standard deviation normalized by mean, is a measure of information content; a lower CV indicates features with a lower normalized variance. Furthermore, in the next step, a correlation matrix was generated, which contains information regarding the cross-correlation values between every pair of features. For each pair with >95% correlation, the one with a lower correlation to the output is dropped. Finally, we employed random forest, a boosted decision tree algorithm, to train the remaining columns and target variables for feature importance score generation. The number of trees was set to a default value of 100, and the Gini index was used as the metric for calculating feature importance. The first four steps were unsupervised (statistical approaches used to perform selection on features alone, not labels); the last feature importance calculation step was supervised (relationship between dependent and independent variables critical in determining feature selection). Figure 2 provides a schematic depiction of the feature selection algorithm.

Figure 2.

A pictorial representation of the feature selection procedure.

2.3. Machine Learning-Based Classification

The important features obtained using the feature selection algorithm were used as inputs into the machine learning classifier. The three classification tasks that were performed were: (1) normal vs. bladder cancer, (2) NMIBC vs. MIBC, (3) post-treatment changes (PTC) vs. MIBC. A 10-fold cross-validation was utilized to evaluate the prediction performance of the proposed model. The samples were randomly re-arranged, then the dataset was split into 10 equal divisions. Nine of these divisions, at every iteration, were used for training; and the remaining was one used for testing. This process was repeated for 10 iterations, each time the test set being a different group. The prediction evaluation metrics calculated across the ten iterations were averaged and reported. Classification was performed using five different machine learning classifiers, namely k-nearest neighbor (KNN), naive Bayes (NB), support vector machine (SVM), linear discriminant analysis (LDA) and decision tree (DT).

2.4. Evaluation Metrics

In order to evaluate the efficacy of the classification model, the following five metrics were used:

$$Accuracy=\frac{TP+TN}{TP+TN+FP+FN}$$ (1)

$$Sensitivity=\frac{TP}{TP+FN}$$ (2)

$$Specificity=\frac{TN}{TN+FP}$$ (3)

$$Precision=\frac{TP}{TP+FP}$$ (4)

$$F1=\frac{2\times Precision\times Sensitivity}{Precision+Sensitivity}$$ (5)

where TP is the # of true positives, TN is the # of true negatives, FP is the # of false positives and FN is the # of false negatives.

Since our dataset is small and highly imbalanced, accuracy, precision and recall are not ideal in representing the classification performance of the models. Therefore, the F1-score was used to determine the overall effectiveness of the classifiers. While accuracy represents the overall percentage of correctly classified samples, precision represents the percentage of identified samples where the condition actually exists, and recall represents the proportion of samples with the condition that have been correctly diagnosed, none of them corrects for data imbalance. The F1-score, which is the harmonic mean of precision and recall, has been employed to rank classifiers in terms of diagnostic performance for this particular study.

3. Software and Tools

MATLAB version R2021b (developed by MathWorks, Massachussets USA) was used for the purpose of feature extraction and machine learning-based classification (The Deep Learning Toolbox was used to train the five pre-trained deep models, namely AlexNet, GoogleNet, InceptionV3, ResNet50 and XceptionNet. The Statistics and Machine Learning Toolbox was used to train the four machine learning-based classifiers, namely naive Bayes, support vector machine, linear discriminant analysis and decision tree).

ImageJ (developed by National Institutes of Health, Maryland USA) and RadiAnt Dicom Viewer (open source application) were used to analyze the CT images. BioRender was used to generate all the illustrations in the paper (Figures 1–4).

Python 3 (open source programming language) was used to program the feature selection workflow and generate the plots containing the F1-scores per model per classifier (Figures 5–7).

4. Dataset

A urothelial carcinoma dataset was provided by Mayo Clinic, Arizona. The dataset contained de-identified grayscale CT scans obtained from patients who were imaged before radical cystectomy and pelvic lymph node dissection as part of a trial. The location of each bladder mass was confirmed, and labels of the preoperative CT data were generated. The labels were “cancer” (meaning malignant cells) and “normal” (normal bladder wall).

There were a total of 100 CT scans of the pelvis with intravenous contrast images visualizing the bladder, obtained from 100 patients (one image captured per patient). Each scan had 2 masks (normal bladder wall and bladder cancer) manually annotated by encompassing both the entire region of biopsy-proven malignancy and the normal-appearing bladder by two radiologists familiar with bladder imaging, which were used to extract the respective regions of interest (ROI)—therefore resulting in 200 ROIs (100 normal tissue and 100 abnormal tissue). These ROIs were used as input for classification instead of the entire image. The patients were distributed across seven bladder cancer stages, namely $T_a$, $T_{is}$, $T_0$, $T_1$, $T_2$, $T_3$ and $T_4$. Figure 3 provides an axial CT scan with IV contrast, along with its corresponding region of interest (ROI) on the bladder wall, pertaining to each of the seven stages. $T_0$ represents a stage where the tissue of interest shows no evidence of malignancy, possibly but not necessarily following tumor resection and/or chemotherapy. This indicates carcinoma in situ, where the malignant cells only involves the innermost lining of the bladder wall. $T_1$ represents a stage where malignant cells involve the connective tissue beyond the innermost lining without involvement of the bladder muscle. $T_2$ represents the malignant spread of tumor involving the bladder muscle. $T_3$ represents a stage where malignant mass spreads outside the confines of bladder muscle with involvement of the perivesical fat. $T_4$ stage indicates the spread of tumor beyond the bladder with involvement of abdominal/pelvic wall and/or nearby organs. $T_a$, $T_{is}$ and $T_1$ stages are classified as non-muscle-invasive bladder cancer (NMIBC); $T_2$, $T_3$ and $T_4$ stages are classified as muscle-invasive bladder cancer (MIBC) [4,27]. Figure 4 is a pictorial depiction of the different stages of bladder cancer. As is evident from Figure 3, the stages are difficult to distinguish on visual inspection—thereby making it suitable for classification using AI-based models. Table 2 provides a summary of the number of patients per stage.

Figure 3.

One bladder CT scan per stage (along with the corresponding regions of interest that were used in the various classification tasks) have been provided. The 7 stages of urothelial carcinoma analyzed in the study are: $T_a$, $T_{is}$, $T_0$, $T_1$, $T_2$, $T_3$ and $T_4$ ($T_0$ has not been shown in the figure because $T_0$ represents a stage where the tissue of interest shows no evidence of malignancy).

Figure 4.

A pictorial representation of the 7 stages of urothelial carcinoma analyzed in the study ($T_0$ has not been shown in the figure because $T_0$ represents a stage where the tissue of interest shows no evidence of malignancy).

Table 2.

A summary of the number of patients per stage.

${\mathit{T}}_{\mathit{a}}$	${\mathit{T}}_{\mathbf{is}}$	${\mathit{T}}_0$	${\mathit{T}}_1$	${\mathit{T}}_2$	${\mathit{T}}_3$	${\mathit{T}}_4$
6	9	35	9	13	24	4

5. Results

The proposed model was used to perform three different classification tasks during the course of this study: (1) normal vs. bladder cancer; (2) NMIBC vs. MIBC; (3) post-treatment changes (PTC) vs. MIBC. In this section, for each of the three tasks, the best classification performances corresponding to each of the five pre-trained deep model-based features have been presented. The class-wise number of ROIs used for each of the individual tasks has also been summarized.

5.1. Normal vs. Cancer

Normal vs. cancer classification was performed with 10-fold cross-validation on a dataset of 165 ROIs (100 normal, 65 cancer)—35 T0 images were not relevant because they represent post-treatment changes (PTC) and not cancer. The LDA classifier on XceptionNet-based features provides the best performance with an accuracy of 86.07%, sensitivity of 96.75%, specificity of 69.65%, precision of 83.07% and F1-score of 89.39%. Table 3 summarizes the classification performances of the 10-fold machine learning classifiers on features extracted from each of the five pre-trained deep models (results visualized on the associated Figure 5 bar plot).

Table 3.

Classification performances of the 10-fold machine learning classifiers on features extracted from each of the five pre-trained deep models.

Feature Extractor	Classifier	Accuracy	Sensitivity	Specificity	Precision	F1-Score
AlexNet	NB	${\displaystyle 0.8053\pm 0.0029}$	${\displaystyle 0.8936\pm 0.004}$	${\displaystyle 0.6694\pm 0.0054}$	${\displaystyle 0.8062\pm 0.0025}$	${\displaystyle 0.8476\pm 0.0024}$
	SVM	${\displaystyle 0.7835\pm 0.0058}$	${\displaystyle 0.8944\pm 0.0056}$	${\displaystyle 0.6128\pm 0.0122}$	${\displaystyle 0.7805\pm 0.0055}$	${\displaystyle 0.8335\pm 0.0043}$
	LDA	${\displaystyle 0.7987\pm 0.0037}$	${\displaystyle 0.9307\pm 0.0027}$	${\displaystyle 0.5957\pm 0.0078}$	${\displaystyle 0.7799\pm 0.0034}$	${\displaystyle 0.8486\pm 0.0026}$
	DT	${\displaystyle 0.7373\pm 0.0113}$	${\displaystyle 0.8043\pm 0.0120}$	${\displaystyle 0.6342\pm 0.0231}$	${\displaystyle 0.7724\pm 0.0114}$	${\displaystyle 0.7877\pm 0.0089}$
GoogleNet	NB	${\displaystyle 0.7642\pm 0.0049}$	${\displaystyle 0.8674\pm 0.0058}$	${\displaystyle 0.6054\pm 0.0085}$	${\displaystyle 0.7718\pm 0.0040}$	${\displaystyle 0.8168\pm 0.0039}$
	SVM	${\displaystyle 0.7661\pm 0.0053}$	${\displaystyle 0.8921\pm 0.0068}$	${\displaystyle 0.5722\pm 0.0084}$	${\displaystyle 0.7624\pm 0.0038}$	${\displaystyle 0.8221\pm 0.0043}$
	LDA	${\displaystyle 0.7899\pm 0.0036}$	${\displaystyle 0.9368\pm 0.0038}$	${\displaystyle 0.5640\pm 0.0061}$	${\displaystyle 0.7678\pm 0.0027}$	${\displaystyle 0.8439\pm 0.0027}$
	DT	${\displaystyle 0.7562\pm 0.0136}$	${\displaystyle 0.8056\pm 0.0167}$	${\displaystyle 0.6803\pm 0.0231}$	${\displaystyle 0.7954\pm 0.0123}$	${\displaystyle 0.8001\pm 0.0116}$
InceptionV3	NB	${\displaystyle 0.7988\pm 0.0031}$	${\displaystyle 0.9016\pm 0.0042}$	${\displaystyle 0.6406\pm 0.0052}$	${\displaystyle 0.7942\pm 0.0024}$	${\displaystyle 0.8445\pm 0.0025}$
	SVM	${\displaystyle 0.7740\pm 0.0042}$	${\displaystyle 0.9036\pm 0.0041}$	${\displaystyle 0.5746\pm 0.0080}$	${\displaystyle 0.7657\pm 0.0036}$	${\displaystyle 0.8290\pm 0.0031}$
	LDA	${\displaystyle 0.7935\pm 0.0012}$	${\displaystyle 0.9096\pm 0.0010}$	${\displaystyle 0.6148\pm 0.0019}$	${\displaystyle 0.7841\pm 0.0009}$	${\displaystyle 0.8422\pm 0.0009}$
	DT	${\displaystyle 0.7295\pm 0.0127}$	${\displaystyle 0.7690\pm 0.0155}$	${\displaystyle 0.6686\pm 0.0211}$	${\displaystyle 0.7816\pm 0.0114}$	${\displaystyle 0.7749\pm 0.0110}$
ResNet50	NB	${\displaystyle 0.8248\pm 0.0039}$	${\displaystyle 0.9290\pm 0.0045}$	${\displaystyle 0.6646\pm 0.0073}$	${\displaystyle 0.8100\pm 0.0034}$	${\displaystyle 0.8654\pm 0.0030}$
	SVM	${\displaystyle 0.7771\pm 0.0051}$	${\displaystyle 0.9092\pm 0.0065}$	${\displaystyle 0.5738\pm 0.0081}$	${\displaystyle 0.7665\pm 0.0036}$	${\displaystyle 0.8317\pm 0.0040}$
	LDA	${\displaystyle 0.7862\pm 0.0028}$	${\displaystyle 0.9224\pm 0.0043}$	${\displaystyle 0.5766\pm 0.0056}$	${\displaystyle 0.7702\pm 0.0022}$	${\displaystyle 0.8395\pm 0.0022}$
	DT	${\displaystyle 0.8079\pm 0.0111}$	${\displaystyle 0.8418\pm 0.0135}$	${\displaystyle 0.7558\pm 0.0256}$	${\displaystyle 0.8424\pm 0.0135}$	${\displaystyle 0.8416\pm 0.0089}$
XceptionNet	NB	${\displaystyle 0.8395\pm 0.0035}$	${\displaystyle 0.8866\pm 0.0048}$	${\displaystyle 0.7671\pm 0.0058}$	${\displaystyle 0.8542\pm 0.0031}$	${\displaystyle 0.8701\pm 0.0030}$
	SVM	${\displaystyle 0.8322\pm 0.0047}$	${\displaystyle 0.9614\pm 0.0034}$	${\displaystyle 0.6335\pm 0.0100}$	${\displaystyle 0.8015\pm 0.0045}$	${\displaystyle 0.8742\pm 0.0033}$
	LDA	${\displaystyle 0.8607\pm 0.0038}$	${\displaystyle 0.9675\pm 0.0027}$	${\displaystyle 0.6965\pm 0.0094}$	${\displaystyle 0.8307\pm 0.0042}$	${\displaystyle 0.8939\pm 0.0026}$
	DT	${\displaystyle 0.8145\pm 0.0099}$	${\displaystyle 0.8453\pm 0.0134}$	${\displaystyle 0.7672\pm 0.0173}$	${\displaystyle 0.8486\pm 0.0095}$	${\displaystyle 0.8466\pm 0.0086}$

Figure 5.

F1-scores of the 10-fold machine learning classifiers on features extracted from each of the five pre-trained deep models.

5.2. NMIBC vs. MIBC

NMIBC vs. MIBC classification was performed with 10-fold cross-validation on a dataset of 65 ROIs (24 NMIBC, 41 MIBC). The LDA classifier on XceptionNet-based features provides the best performance with an accuracy of 79.72%, sensitivity of 66.62%, specificity of 87.39%, precision of 75.58% and F1-score of 70.81%. Table 4 summarizes the classification performances of the 10-fold machine learning classifiers on features extracted from each of the five pre-trained deep models (results visualized on the associated Figure 6 bar plot).

Table 4.

Classification performances of the 10-fold machine learning classifiers on features extracted from each of the five pre-trained deep models.

Feature Extractor	Classifier	Accuracy	Sensitivity	Specificity	Precision	F1-Score
AlexNet	NB	${\displaystyle 0.7768\pm 0.0067}$	${\displaystyle 0.6746\pm 0.0117}$	${\displaystyle 0.8366\pm 0.0089}$	${\displaystyle 0.7079\pm 0.0112}$	${\displaystyle 0.6905\pm 0.0090}$
	SVM	${\displaystyle 0.7709\pm 0.0143}$	${\displaystyle 0.6692\pm 0.0244}$	${\displaystyle 0.8305\pm 0.0147}$	${\displaystyle 0.6989\pm 0.0214}$	${\displaystyle 0.6830\pm 0.0203}$
	LDA	${\displaystyle 0.7963\pm 0.0065}$	${\displaystyle 0.6650\pm 0.0125}$	${\displaystyle 0.8732\pm 0.0076}$	${\displaystyle 0.7547\pm 0.0114}$	${\displaystyle 0.7067\pm 0.0097}$
	DT	${\displaystyle 0.7160\pm 0.0215}$	${\displaystyle 0.6008\pm 0.0445}$	${\displaystyle 0.7834\pm 0.0286}$	${\displaystyle 0.6223\pm 0.0329}$	${\displaystyle 0.6079\pm 0.0324}$
GoogleNet	NB	${\displaystyle 0.7480\pm 0.0108}$	${\displaystyle 0.5867\pm 0.0202}$	${\displaystyle 0.8424\pm 0.0124}$	${\displaystyle 0.6864\pm 0.0178}$	${\displaystyle 0.6319\pm 0.0166}$
	SVM	${\displaystyle 0.7238\pm 0.0088}$	${\displaystyle 0.4462\pm 0.0158}$	${\displaystyle 0.8863\pm 0.0107}$	${\displaystyle 0.6984\pm 0.0215}$	${\displaystyle 0.5438\pm 0.0154}$
	LDA	${\displaystyle 0.7758\pm 0.0069}$	${\displaystyle 0.6583\pm 0.0157}$	${\displaystyle 0.8446\pm 0.0064}$	${\displaystyle 0.7127\pm 0.0099}$	${\displaystyle 0.6841\pm 0.0120}$
	DT	${\displaystyle 0.7740\pm 0.0220}$	${\displaystyle 0.6925\pm 0.0408}$	${\displaystyle 0.8217\pm 0.0239}$	${\displaystyle 0.6968\pm 0.0312}$	${\displaystyle 0.6925\pm 0.0310}$
InceptionV3	NB	${\displaystyle 0.7575\pm 0.009}$	${\displaystyle 0.6137\pm 0.0194}$	${\displaystyle 0.8417\pm 0.0093}$	${\displaystyle 0.6945\pm 0.0139}$	${\displaystyle 0.6511\pm 0.0149}$
	SVM	${\displaystyle 0.7242\pm 0.0069}$	${\displaystyle 0.4163\pm 0.0179}$	${\displaystyle 0.9044\pm 0.0033}$	${\displaystyle 0.7174\pm 0.0114}$	${\displaystyle 0.5263\pm 0.0167}$
	LDA	${\displaystyle 0.7698\pm 0.0060}$	${\displaystyle 0.5854\pm 0.0134}$	${\displaystyle 0.8778\pm 0.0056}$	${\displaystyle 0.7373\pm 0.0098}$	${\displaystyle 0.6523\pm 0.0105}$
	DT	${\displaystyle 0.6909\pm 0.0210}$	${\displaystyle 0.5300\pm 0.0409}$	${\displaystyle 0.7851\pm 0.0306}$	${\displaystyle 0.5963\pm 0.0353}$	${\displaystyle 0.5572\pm 0.0306}$
ResNet50	NB	${\displaystyle 0.6878\pm 0.0103}$	${\displaystyle 0.8013\pm 0.0135}$	${\displaystyle 0.6215\pm 0.0149}$	${\displaystyle 0.5539\pm 0.0105}$	${\displaystyle 0.6547\pm 0.0098}$
	SVM	${\displaystyle 0.7240\pm 0.0065}$	${\displaystyle 0.3167\pm 0.0115}$	${\displaystyle 0.9624\pm 0.0074}$	${\displaystyle 0.8343\pm 0.0279}$	${\displaystyle 0.4584\pm 0.0138}$
	LDA	${\displaystyle 0.7748\pm 0.0098}$	${\displaystyle 0.5375\pm 0.0221}$	${\displaystyle 0.9137\pm 0.0130}$	${\displaystyle 0.7874\pm 0.0220}$	${\displaystyle 0.6373\pm 0.0173}$
	DT	${\displaystyle 0.7665\pm 0.0159}$	${\displaystyle 0.6088\pm 0.0300}$	${\displaystyle 0.8588\pm 0.0189}$	${\displaystyle 0.7190\pm 0.0287}$	${\displaystyle 0.6575\pm 0.0242}$
XceptionNet	NB	${\displaystyle 0.7240\pm 0.0094}$	${\displaystyle 0.6667\pm 0.0001}$	${\displaystyle 0.7576\pm 0.0149}$	${\displaystyle 0.6182\pm 0.0146}$	${\displaystyle 0.6411\pm 0.0079}$
	SVM	${\displaystyle 0.7597\pm 0.0094}$	${\displaystyle 0.5258\pm 0.0219}$	${\displaystyle 0.8966\pm 0.0085}$	${\displaystyle 0.7489\pm 0.0167}$	${\displaystyle 0.6170\pm 0.0182}$
	LDA	${\displaystyle 0.7972\pm 0.0058}$	${\displaystyle 0.6662\pm 0.0100}$	${\displaystyle 0.8739\pm 0.0058}$	${\displaystyle 0.7558\pm 0.0094}$	${\displaystyle 0.7081\pm 0.0086}$
	DT	${\displaystyle 0.7403\pm 0.0243}$	${\displaystyle 0.6733\pm 0.0379}$	${\displaystyle 0.7795\pm 0.0286}$	${\displaystyle 0.6446\pm 0.0344}$	${\displaystyle 0.6567\pm 0.0313}$

Figure 6.

F1-scores of the 10-fold machine learning classifiers on features extracted from each of the five pre-trained deep models.

5.3. Post-Treatment Changes (PTC) vs. MIBC

PTC vs. MIBC classification was performed with a 10-fold cross-validation on a dataset of 76 ROIs (35 PTC, 41 MIBC). LDA classifier on XceptionNet-based features provided the best performance with an accuracy of 74.96%, sensitivity of 80.51%, specificity of 70.22%, precision of 69.78% and F1-score of 74.73%. Table 5 summarizes the classification performances of the 10-fold machine learning classifiers, on features extracted from each of the five pre-trained deep models (results visualized on associated Figure 7 bar plot).

Table 5.

Classification performances of the 10-fold machine learning classifiers on features extracted from each of the five pre-trained deep models.

Feature Extractor	Classifier	Accuracy	Sensitivity	Specificity	Precision	F1-Score
AlexNet	NB	${\displaystyle 0.6695\pm 0.0097}$	${\displaystyle 0.5323\pm 0.0172}$	${\displaystyle 0.7866\pm 0.0114}$	${\displaystyle 0.6806\pm 0.0133}$	${\displaystyle 0.5969\pm 0.0139}$
	SVM	${\displaystyle 0.6880\pm 0.0139}$	${\displaystyle 0.6894\pm 0.0216}$	${\displaystyle 0.6868\pm 0.0168}$	${\displaystyle 0.6530\pm 0.0145}$	${\displaystyle 0.6702\pm 0.0157}$
	LDA	${\displaystyle 0.7100\pm 0.0070}$	${\displaystyle 0.6934\pm 0.0116}$	${\displaystyle 0.7241\pm 0.0121}$	${\displaystyle 0.6826\pm 0.0091}$	${\displaystyle 0.6877\pm 0.0076}$
	DT	${\displaystyle 0.6503\pm 0.0187}$	${\displaystyle 0.6160\pm 0.0290}$	${\displaystyle 0.6795\pm 0.0270}$	${\displaystyle 0.6226\pm 0.0215}$	${\displaystyle 0.6180\pm 0.0214}$
GoogleNet	NB	${\displaystyle 0.7176\pm 0.0101}$	${\displaystyle 0.5689\pm 0.0139}$	${\displaystyle 0.8446\pm 0.0149}$	${\displaystyle 0.7589\pm 0.0180}$	${\displaystyle 0.6497\pm 0.0124}$
	SVM	${\displaystyle 0.6592\pm 0.0085}$	${\displaystyle 0.5366\pm 0.0128}$	${\displaystyle 0.7639\pm 0.0111}$	${\displaystyle 0.6602\pm 0.0122}$	${\displaystyle 0.5917\pm 0.0110}$
	LDA	${\displaystyle 0.7276\pm 0.0072}$	${\displaystyle 0.6129\pm 0.0126}$	${\displaystyle 0.8256\pm 0.0082}$	${\displaystyle 0.7502\pm 0.0094}$	${\displaystyle 0.6743\pm 0.0097}$
	DT	${\displaystyle 0.6725\pm 0.0237}$	${\displaystyle 0.6471\pm 0.0321}$	${\displaystyle 0.6941\pm 0.0345}$	${\displaystyle 0.6464\pm 0.0288}$	${\displaystyle 0.6450\pm 0.0252}$
InceptionV3	NB	${\displaystyle 0.6495\pm 0.0150}$	${\displaystyle 0.5280\pm 0.0207}$	${\displaystyle 0.7532\pm 0.0205}$	${\displaystyle 0.6475\pm 0.0214}$	${\displaystyle 0.5808\pm 0.0182}$
	SVM	${\displaystyle 0.6907\pm 0.0100}$	${\displaystyle 0.6094\pm 0.0169}$	${\displaystyle 0.7600\pm 0.0121}$	${\displaystyle 0.6846\pm 0.0122}$	${\displaystyle 0.6444\pm 0.0129}$
	LDA	${\displaystyle 0.7083\pm 0.0068}$	${\displaystyle 0.6554\pm 0.0086}$	${\displaystyle 0.7534\pm 0.0109}$	${\displaystyle 0.6945\pm 0.0096}$	${\displaystyle 0.6742\pm 0.0070}$
	DT	${\displaystyle 0.6589\pm 0.0260}$	${\displaystyle 0.6583\pm 0.0383}$	${\displaystyle 0.6595\pm 0.0327}$	${\displaystyle 0.6238\pm 0.0270}$	${\displaystyle 0.6391\pm 0.0287}$
ResNet50	NB	${\displaystyle 0.6688\pm 0.0085}$	${\displaystyle 0.8689\pm 0.0138}$	${\displaystyle 0.4980\pm 0.0125}$	${\displaystyle 0.5965\pm 0.0066}$	${\displaystyle 0.7072\pm 0.0079}$
	SVM	${\displaystyle 0.6197\pm 0.0157}$	${\displaystyle 0.5600\pm 0.0251}$	${\displaystyle 0.6707\pm 0.0232}$	${\displaystyle 0.5930\pm 0.0186}$	${\displaystyle 0.5750\pm 0.0189}$
	LDA	${\displaystyle 0.6895\pm 0.0094}$	${\displaystyle 0.7903\pm 0.0306}$	${\displaystyle 0.6034\pm 0.0169}$	${\displaystyle 0.6298\pm 0.0069}$	${\displaystyle 0.7000\pm 0.0144}$
	DT	${\displaystyle 0.6161\pm 0.0207}$	${\displaystyle 0.5891\pm 0.0298}$	${\displaystyle 0.6390\pm 0.0313}$	${\displaystyle 0.5838\pm 0.0239}$	${\displaystyle 0.5851\pm 0.0229}$
XceptionNet	NB	${\displaystyle 0.6849\pm 0.0087}$	${\displaystyle 0.7726\pm 0.0142}$	${\displaystyle 0.6100\pm 0.0094}$	${\displaystyle 0.6284\pm 0.0075}$	${\displaystyle 0.6929\pm 0.0094}$
	SVM	${\displaystyle 0.6868\pm 0.0105}$	${\displaystyle 0.7931\pm 0.0170}$	${\displaystyle 0.5961\pm 0.0102}$	${\displaystyle 0.6263\pm 0.0085}$	${\displaystyle 0.6998\pm 0.0112}$
	LDA	${\displaystyle 0.7496\pm 0.0119}$	${\displaystyle 0.8051\pm 0.0193}$	${\displaystyle 0.7022\pm 0.0133}$	${\displaystyle 0.6978\pm 0.0113}$	${\displaystyle 0.7473\pm 0.0131}$
	DT	${\displaystyle 0.6253\pm 0.0219}$	${\displaystyle 0.5806\pm 0.0320}$	${\displaystyle 0.6634\pm 0.0298}$	${\displaystyle 0.5967\pm 0.0251}$	${\displaystyle 0.5872\pm 0.0254}$

Figure 7.

F1-scores of the 10-fold machine learning classifiers on features extracted from each of the five pre-trained deep models.

6. Discussion

Optimal management of bladder cancer requires a multidisciplinary approach, with tumor staging an important prognostic factor that determines the mode of initial treatment. Cystoscopy examination, together with a biopsy, remains the primary mode of tumor detection and clinical staging in patients with suspected bladder cancer. CT is often the first exam modality for bladder cancer detection due to its wide availability and minimal associated complication. However, the CT findings are nonspecific with limited accuracy. At least one article [28] reported the accuracy of CT evaluation for bladder cancer as low as 49%.

Differentiating non-muscle-invasive bladder cancer (NMIBC), consisting of Ta, Tis, and T1 stages, from muscle-invasive bladder cancer (MBIC), consisting of $T_2$-$T_4$4 stages, is important as MIBC is more likely to spread to lymph nodes and other organs requiring radical cystectomy with or without systemic chemotherapy [29]. In contrast, NMBIC has low risk of recurrent disease but can be effectively treated with intravesical chemotherapy, immunotherapy, and transurethral resection of bladder tumor (TURBT) [30]. Early differentiation of two bladder cancer stages is critical for the appropriate utilization of medical resources and optimization of targeted treatment. Therefore, accurate initial staging on the CT exam is crucial for therapeutic decision-making [31].

While histopathologic cancer detection using pre-trained neural network-based models is mainstream, in this project, we were faced with the additional issues of imbalanced data and limited sample size. In comparison to [32], where the number of image samples was 1350, and the number of classification categories was 2 (1200 bladder cancer tissues, 1150 normal tissues), we had a total of only 100 tissue scans distributed across 7 stages of urothelial carcinoma (6 $T_a$, 9 $T_{i}s$, 35 $T_0$, 9 $T_1$, 13 $T_2$, 24 $T_3$, 4 $T_4$). The issue of class-imbalanced samples could not be solved using SMOTE because the sheer lack of samples in some of the categories (especially $T_a$ and $T_4$) meant that the synthetically generated samples were very similar to the few original ones, and contributed towards decreased variance in the training dataset—therefore resulting in overfitting. Next, the issue of overfitting, which we faced while performing end-to-end classification using pre-trained networks, was tackled by: (1) making use of a combined deep learning-machine learning approach where the trained weights from the pre-trained neural network were then classified using statistical machine learning approaches. (2) An ensemble statistical- and supervised learning-based feature selection approach that helped remove features that were unimportant. Ref. [32], which focused on bladder cancer detection from cytoscopic images alone, reported an accuracy of 86.36% with deep learning-based classifiers and 84.09% with human surgical experts; however, no significant difference was found between the two (p-value greater than 0.05).

The LDA classifier on XceptionNet performed best in terms of the F1-score for all three experiments in our study—namely, normal vs. cancer, NMIBC vs. MIBC and PTC vs. MIBC. For normal vs. cancer classification, LDA on XceptionNet had an F1-score of (89.39 ± 0.26)%, which will facilitate clinicians in better detection of lesions because, for histopathologic images in general, and bladder scans in particular, flat and subtle lesions are often missed on visual inspection. For NMIBC vs. MIBC classification, LDA on XceptionNet had an F1-score of (70.81 ± 0.86). For PTC vs. MIBC classification, LDA on XceptionNet had an F1-score of (74.73 ± 1.31)%, which is especially encouraging because there is an unmet demand to develop new non-invasive techniques to assess accurate prediction of recurrence and response to chemotherapy. Currently, patients with bladder cancer require repeat cystoscopies and biopsies of the bladder to assess the response and recurrence of the disease. This procedure is very costly and invasive, with several associated complications, including bladder perforation.

Our study has certain limitations. It is a retrospective design based on a single-center small dataset that may overestimate the diagnostic performance of our model. Therefore, our next step is to extend the diagnostic model on prospective, multi-center datasets with external validation.

7. Conclusions

Our model showed a high F1-score, which means that our model indicated a high value for both recall and precision. We used the F1-score to compare our classifiers. We opted for the ResNet-50, whose F1-score was higher among others and ResNet-50 showed the best classification based on the F1-score for all three experiments.

Radiomics-assisted interpretation of CT by radiologists may help more accurately diagnose bladder cancer. This can allow the timely utilization of medical resources and consultation with oncologists and urologists, ultimately improving patients’ clinical outcomes.

Author Contributions

Conceptualization, S.S., K.M., W.I., A.C.S., P.S. and T.W.; methodology, S.S., T.W. and A.C.S.; software, S.S.; validation, S.S. and K.M.; formal analysis, S.S.; investigation, S.S.; resources, T.W., A.C.S. and P.S.; data curation, K.M., R.W.T., I.B.R., A.C.S., A.H.B., C.M., T.H.H., G.S., H.M.A.-M. and P.S.; writing—original draft preparation, S.S. and W.I.; writing—review and editing, S.S., K.M., W.I., T.W., A.C.S. and P.S.; visualization, S.S.; supervision, T.W., P.S. and A.C.S.; project administration, T.W., P.S. and A.C.S. All authors have read and agreed to the published version of the manuscript.

Institutional Review Board Statement

The study was conducted according to the guidelines of the Declaration of Helsinki, and approved by the Institutional Review Board (or Ethics Committee) of Mayo Clinic Arizona (protocol code 20-012296 and date of approval 15 December 2020).

Informed Consent Statement

Informed consent was obtained from all subjects involved in the study. Furthermore, all data used in the study have been de-identified of any personal information.

Data Availability Statement

Data will be made available under reasonable request.

Conflicts of Interest

The authors declare no conflict of interest.

Funding Statement

This research received no external funding.