Prostate cancer detection using modified transformer with optimal feature selection from MRI images

Abstract

Prostate cancer is one of the most prevalent malignancies among men, and early detection through Magnetic Resonance Imaging (MRI) plays a crucial role in improving patient outcomes. However, accurate identification of cancerous regions remains challenging due to image noise, variations in tissue structures, and subtle differences between benign and malignant lesions. To address these challenges, a novel automated detection framework named Prostate Cancer Detection Neural Network (ProstateNet) that enhances the precision of prostate cancer detection is proposed. The proposed model begins with MRI image preprocessing using a Bilateral Filter, which effectively reduces noise while preserving edge details. Next, feature extraction is performed using a Residual Recurrent Model (RRM), which captures spatial dependencies and fine-grained image representations. To refine the extracted features, we employ the Iterative Secrecy Bird Optimization (ISBO) algorithm, ensuring optimal feature selection and reducing computational complexity. Finally, disease detection is carried out using a Modified Transformer model, which considers CapsuleNet instead of Feed Forward Network to accurately classify cancerous and non-cancerous regions. The integration of these advanced techniques significantly enhances the robustness and reliability of prostate cancer detection, offering a promising outcome.

Introduction

Prostate cancer is a significant health concern, being the most common malignancy among men in several countries, including Australia, where approximately 25,000 men are diagnosed annually, leading to over 3,500 deaths each year¹. The exact causes of prostate cancer remain unclear; however, risk factors such as age, family history, and diets high in saturated fats have been identified². Early detection of prostate cancer is crucial, as it significantly improves survival rates and allows for a broader range of treatment options. Early-stage prostate cancer often presents with minimal or no symptoms, making proactive screening vital^3,4. Digital rectal examination (DRE) and the prostate-specific antigen (PSA) blood test are examples of conventional detection techniques. Increased PSA levels may be a sign of cancer. The PSA test quantifies the amount of PSA in the blood. However, because elevated PSA levels can also be caused by benign illnesses such prostatitis or benign prostatic hyperplasia, PSA testing has been criticized for its lack of specificity, which results in over-diagnosis and overtreatment⁵. The DRE has limits in terms of sensitivity and specificity, but it uses a physical examination of the prostate gland to identify abnormalities.

Based on pathological grading, PSA level, and tumor stage, prostate cancer is generally divided into clinically significant and clinically non-significant diseases in clinical practice. According to the TNM (Tumor, Node, Metastasis) staging system, clinically relevant prostate cancer usually refers to aggressive tumors that need to be treated definitively and are linked to Gleason Grade Group ≥ 2, PSA levels typically greater than 10 ng/mL, or clinical stage ≥ T2b. These tumors are linked to an increased risk of progression, extracapsular extension, or metastasis and frequently match PI-RADS 4–5 lesions on multiparametric MRI. On the other hand, Gleason Grade Group 1, PSA values usually less than 10 ng/mL, and clinical stage T1–T2a are widely used to describe indolent or low-risk prostate cancer. Active surveillance is often used to control these tumors instead of prompt intervention. In order to prevent overtreatment and ensure prompt management of aggressive illness, it is crucial to distinguish between clinically significant and non-significant prostate cancer. The proposed approach in this research carries out binary classification in accordance with this widely recognized risk stratification framework.

Traditional machine learning (ML) methods like support vector machines (SVMs), k-nearest neighbours (KNN), and decision trees have been employed to enhance the accuracy of prostate cancer detection⁶. These approaches often rely on handcrafted features extracted from medical images or clinical data, which can be time-consuming and may not capture the complex patterns associated with prostate cancer^7,8. Additionally, these models may struggle with generalization due to limited datasets and the inherent variability in medical imaging that leads to challenges in accurately distinguishing between malignant and benign cases^9,10. Deep learning (DL) techniques like convolutional neural networks (CNNs) have been developed to address these challenges by automatically learning hierarchical feature representations from raw data, thereby reducing the dependency on manual feature extraction¹¹. For instance, a study by¹² proposed a DL pipeline combining U-Net with long short-term memory (LSTM) networks for the segmentation and classification of multi-parametric MRI images, demonstrating improved diagnostic precision and consistency. Similarly,¹³ highlighted that DL models trained with digital pathologist labels achieved superior performance in prostate cancer detection across different cohorts, irrespective of the model architecture used¹⁴. These advancements suggest that DL methods can effectively overcome the limitations of traditional ML approaches by providing more accurate and reliable diagnostic tools for prostate cancer detection¹⁵.

For the identification of prostate cancer, current research has used transfer learning models like ResNet, EfficientNet, and ensemble CNN frameworks; nevertheless, these methods still have significant drawbacks^16,17. Local feature extraction is the main emphasis of pre-trained CNN-based models, which frequently fall short in capturing global contextual relationships across MRI slices a crucial component of accurate tumor classification^18,19. Furthermore, a lot of current approaches rely significantly on sizable annotated datasets, which makes them vulnerable to over-fitting when used with sparse medical data²⁰. Gaussian and Rician noise, two types of noise that are inherent in MRI imaging, further impair classification reliability because the majority of deep learning pipelines do not have specific edge-preserving noise suppression algorithms²¹. Furthermore, while ensemble techniques improve accuracy, they also increase inference time and decrease clinical practicality²². High-dimensional feature representations, in addition, increase computational burden and limit interpretability when improperly optimized. To overcome the challenges, this research introduces a novel deep learning model. The proposed ProstateNet framework introduces an integrated architecture that combines bilateral filtering for effective noise reduction while preserving anatomical structures, a Residual Recurrent Model to combined model deep spatial and contextual dependencies, an ISBO algorithm for selecting the most discriminative feature subsets to reduce redundancy and enhance generalization, and a Modified Transformer enhanced with Capsule Network integration to preserve hierarchical spatial relationships and improve discrimination between cancerous and non-cancerous tissues.

The proposed ProstateNet presents a fundamentally redefined learning architecture through three major innovations, compared to current transformer-based or CNN-based prostate cancer detection frameworks that mostly rely on architectural stacking or transfer learning. First, a dynamic routing-based Capsule Network is used in place of the traditional position-wise Feed Forward Network (FFN) in the transformer architecture. This allows for the preservation of hierarchical spatial relationships within MRI tokens, which is not possible with standard vision transformers used for prostate cancer detection. Second, the proposed ISBO uses iterative chaotic mapping within Secretary Bird Optimization in place of traditional feature pruning or attention-based selection. This introduces non-linear search dynamics that explicitly control exploration–exploitation balance during feature selection and prevent premature convergence. Additionally, by combining residual CNN feature hierarchies with SRU-based contextual modeling, the Residual Recurrent Model (RRM) creates a hybrid spatial–sequential representation mechanism that enables MRI slices to be understood as structured dependency sequences as opposed to separate spatial patches.

For sequential feature modeling, the proposed ProstateNet uses a Simple Recurrent Unit (SRU) rather than a traditional LSTM or GRU because SRU allows parallel computation and faster convergence while maintaining contextual dependencies in MRI features, ensuring an ideal balance between modeling capacity and computational efficiency. Additionally, the Capsule Network (CapsNet) replaces the conventional Transformer Feed Forward Network (FFN) in order to better retain hierarchical spatial linkages and part-whole anatomical structures that are crucial for prostate MRI analysis. In order to retain excellent diagnostic accuracy while keeping computing complexity within clinically practical bounds, the architectural depth and hyperparameters were established through empirical validation.

The major contributions of the research are:

• Design of RRM for feature extraction: The proposed Residual Recurrent Model (RRM) based feature extraction is efficient in extracting the spatial and temporal dependencies in MRI data. In this, ResNet is helpful in capturing deep spatial features and SRU in capturing sequential dependencies in the extracted features.

• Design of ISBO algorithm for feature selection: The ISBO algorithm is designed by incorporating the iterative chaotic mapping within the secretary bird optimization for selecting the appropriate features.

• Design of Modified Transformer for disease detection: A Modified Transformer (M-Trans) with CapsNet, wherein instead of FFN is proposed for prostate cancer detection that leverages the strengths of both architectures to enhance accuracy and robustness in MRI image analysis.

The research is structured as follows: Section "Related works" lists relevant literature together with the problem description, and Section "Proposed methodology" presents the comprehensive proposed technique. The findings, discussion, and conclusion are presented in Section "Result and discussion" and Section "Conclusion", respectively.

Related works

Prostate cancer detection using EfficientNet V2 was designed by²³ using tabular medical data. In this, patient’s tabular data was transformed into visual representation prior to the data augmentation. EfficientNet V2 with transfer learning was designed to analyze the generated images. Finally, stacked ensemble model was employed for the disease detection. The designed model demonstrated that the proposed method is a viable alternative to invasive prostate cancer diagnostic procedures. Transforming tabular data into images may reduce interpretability of the model.

A deep learning-based technique for the accurate detection and Gleason grading of prostate cancer using MRI images was designed by²⁴. In this, 3D CNN was employed to process the MRI volumes for epithelial detection and Gleason grading. Then, Faster R-CNN with Inception-ResNet-V2 Transfer Learning was employed for the segmentation of the lesion region. Improved prostate cancer classification based on Gleason grading was accomplished by the designed model along with precise localization and characterization of cancerous lesions. The consideration of ROIAlign layer may result in the loss of scale information that degrades the performance of the model.

Convolutional Neural Networks (CNNs) based on the ResNet was designed by²⁵ to classify prostate cancer. In this, digital pathology images were extracted at various patches at multiple magnification levels. The designed model capture varying tissue features from different magnification levels. The designed model demonstrated the effectiveness of ResNet models for multi-scale patch-level prostate cancer grading. The model achieved exceptional accuracy for improving the reliability of prostate cancer detection. Still, the efficiency and scalability was challenging due to the failure in optimizing the performance of ResNet model that requires careful tuning of hyper-parameters.

A modified deep learning architecture based on ResNet50 for better prostate cancer detection was designed by²⁶. The Prostate Cancer Detection Model (PCDM) model was suggested for the automatic detection of prostate cancer. In actual healthcare settings, it demonstrates its practical usefulness to support early prostate cancer identification and treatment. Faster R-CNN and dual optimizers were integrated into the PCDM model, a modified ResNet50-based architecture, to enhance detection performance. A sizable collection of annotated medical images was used to train the model.

Weakly supervised UNet (WSUNet) model was designed by²⁷ to develop a deep-learning model capable of detecting MRI-invisible prostate cancers. In this, utilizing a UNet-based convolutional neural network (WSUNet) the image segmentation and analysis were made. MRI-Ultrasound fusion was employed for comprehensive prostate biopsies. The designed model has the capability of detection cancers that are difficult to visualize on MRI. Still, weakly supervised learning may result in less precise predictions.

Problem statement

Traditional diagnostic methods of prostate cancer detection including biopsy and manual interpretation of MRI scans are often time-consuming, prone to human error and may lead to unnecessary invasive procedures. To address these challenges, deep learning methods were designed that provides the automated outcome without the need for manual processing. Existing deep learning-based methods for prostate cancer detection primarily include pre-trained convolutional neural networks (CNNs) like EfficientNet-v2 and ResNet, ResNet50 and ensemble learning approaches. Pre-trained models leverage transfer learning to extract features from MRI images, but they often struggle with domain-specific variations. ResNet, known for its deep architecture with residual connections, improves feature extraction but can suffer from over-fitting and high computational costs, making it less efficient for real-time applications. Ensemble learning, which combines multiple models for improved accuracy, often increases computational complexity and inference time, making it less suitable for clinical applications.

The proposed method, overcomes these challenges by leveraging a transformer-based architecture modified for medical image analysis. Unlike CNNs, transformers excel in capturing long-range dependencies and spatial relationships without being constrained by local receptive fields, leading to improved feature extraction from MRI images. The integration of an optimal feature selection technique further enhances the model’s performance by eliminating redundant and irrelevant features, reducing over-fitting and computational burden. Additionally, the proposed approach minimizes false positives and false negatives, improving diagnostic reliability. By combining the strengths of transformers with feature selection and feature extraction, the proposed method helps to achieve higher accuracy, better generalization to diverse datasets, and efficient processing, making it a more effective and practical solution for prostate cancer detection.

Proposed methodology

The proposed methodology for prostate cancer detection from MRI images integrates advanced image processing, feature extraction, optimization, and deep learning techniques to enhance accuracy and robustness. The pipeline begins with MRI image preprocessing using a Bilateral Filter, which effectively removes noise while preserving essential edge details, ensuring that critical anatomical structures are maintained. Following this, feature extraction is performed using a Residual Recurrent Model (RRM), which leverages residual connections and recurrent operations to capture both spatial dependencies and fine-grained image representations, enhancing the feature learning process. To optimize the extracted features and reduce computational complexity, the Iterative Secrecy Bird Optimization (ISBO) algorithm is applied, ensuring the selection of the most relevant and discriminative features for classification. Finally, the disease detection phase employs a Modified Transformer model, where the traditional Feed-Forward Network (FFN) is replaced with CapsNet to enhance spatial hierarchy representation and improve classification accuracy. CapsNet enables the model to better preserve spatial relationships between features, overcoming the limitations of traditional CNNs and FFNs, and improving robustness against variations in MRI images. The workflow of the proposed method is portrayed in Fig. 1.

Fig. 1.

Proposed Prostate Cancer Detection Workflow.

Data input

The dataset is collected from publically available source,²⁸ for processing the proposed prostate cancer detection model.

Bilateral filter based pre-processing

MRI images contain Gaussian noise, Rician noise, or speckle noise due to sensor imperfections and environmental factors. The Bilateral Filter (BF) is a non-linear, edge-preserving, and noise-reducing filter that smooths images while retaining important structural details. Bilateral filtering smooths noise while preserving edges and textures. The edge preservation ensures that pixels with similar intensities are given higher weight, preserving tissue boundaries and anatomical details. By reducing background noise, the contrast between different tissue types and improves the disease detection accuracy. Let, the input image at a pixel the bilateral filtered output is computed as:

1

where, is the intensity of pixel v in the local neighborhood γ, and is the spatial weight function (measures proximity). is the range weight function (measures intensity similarity) and is the normalization factor ensuring the weights sum to 1:

2

The resulting image has noise reduced while preserving edge and texture details of the input image.

Residual recurrent model based feature extraction

The proposed Residual Recurrent Model (RRM) based feature extraction is efficient in extracting the spatial and temporal dependencies in MRI data. Here, ResNet is helpful in capturing deep spatial features and SRU in capturing sequential dependencies in the extracted features. This hybrid approach enhances feature learning, ensuring that both spatial and temporal information are retained for better performance in medical image analysis. The structure of RRM model in feature extraction is presented in Fig. 2.

Fig. 2.

Structure of RRM based feature extraction.

The vanishing gradient problem in deep learning is resolved by ResNets, a class of deep convolutional neural networks (CNNs), which introduce skip connections known as residual connections. Training becomes more effective because to these connections, which enable the network to learn residual mappings. MRI images are high-dimensional and contain detailed spatial features. Deep CNN layers extract low-level like edges and textures characteristics and high-level like shapes, and regions of interest features. Residual connections help retain important features while avoiding degradation as depth increases. Batch normalization and activation functions improve generalization and training stability. Let q be the input feature map to a residual block, and be the transformation applied by convolutional layers. A residual block in ResNet is defined as:

3

where, m is the output feature map, q is the identity mapping that represents the original input and is the convolutional layers extracting features. While CNN-based ResNet effectively extracts spatial features, MRI data often requires modelling sequential or contextual dependencies. SRUs are designed for faster computation by introducing parallelizable operations. The SRU model takes the spatial feature maps from ResNet as input and captures temporal dependencies between time-series sequences. SRU has two main components:

Forget gate () controls how much past information is retained:

4

Memory cell update () keeps relevant information:

5

Final output () is computed as:

6

where, is an activation function, Θ represents element-wise multiplication and γ is the sigmoid activation. , are biases and , are weight matrices. By using RRM provides an efficient hybrid framework for MRI feature extraction for capturing spatial and temporal features.

Iterative secretary bird optimization algorithm

The proposed Iterative Secrecy Bird Optimization (ISBO) algorithm is an extension of the standard Secrecy Bird Optimization (SBOA) augmented with iterative chaotic mapping to improve search efficiency and reaching an optimal solution. Inspired by the distinctive behaviours of the secretary bird (Sagittarius serpentarius), the SBOA is a population-based metaheuristic technique. The positions of the secretary birds in the search space dictate the values of the decision variables, and each one of them offers a potential solution to the optimization issue. When hunting snakes, secretary birds go through three different phases of their hunting behavior: seeking prey, devouring prey, and attacking prey. These stages are modelled in the SBOA to mimic the bird’s natural hunting strategies for solving optimization problems effectively.

Initialization of candidate solutions: A colony of secretary birds is randomly initialized to begin the ISBO algorithm’s optimization phase. The location of each bird in the search space indicates a possible solution for the issue. Each secretary bird’s starting position is produced using:

7

where, is the position of the secretary bird in the dimension and is the lower bound of the variable. is the upper bound of the variable and g is the random number between [0,1] that ensures diversity in initial solutions. The population of secretary birds is represented as a matrix S of size B × E:

8

where, S is the secretary bird population indicates the set of all candidate solutions, is the secretary bird’s location indicates the candidate solution and is the variable of the secretary bird. B is the total number of secretary birds indicates the population size and E is the number of decision variables indicates the problem dimension. Each secretary bird is evaluated based on a predefined objective function , which determines the quality of the solution. The objective function values for all birds are stored in a vector K:

9

where, is the objective function value of the secretary bird.

Hunting behavior of secretary birds: The hunting behavior of secretary birds is divided into three main stages:

• Seeking prey: The condition considered is

• Devouring prey: The condition considered is

• Capturing Prey: The condition considered is

Each phase represents a different searching, decision-making, and attacking strategy, which are incorporated into the algorithm.

Seeking prey: Secretary birds use their keen vision to search the surroundings for snakes. Being aware of their surroundings, they move gently, sweeping the ground with their legs. They are able to keep a safe distance from possible dangers due to their long necks and legs. In this, Differential Evolution (DE) is used to introduce diversity, avoiding premature convergence. New positions are generated using differences between solutions, helping birds explore different regions.

10

Then, the solution’s feasibility is estimated as:

11

where, d is the current iteration number, D is the maximum number of iterations and , is the random candidate solutions. is the randomly generated array from [0,1], and is the fitness value of the new solution.

Devouring prey: Once a secretary bird finds a snake, it doesn’t attack immediately. Instead, it observes the snake’s movements while hovering, jumping, and provoking it. This gradual exhaustion technique allows the bird to tire the snake before attacking. Brownian motion for random movement is used to simulate the bird’s unpredictable movements. Brownian motion helps prevent premature convergence by adding randomness. The random movement is interpreted as:

12

where, the random movement is signified as RM, and the random number is indicated as Qb. The solution updation is defined as:

13

Then, the solution’s feasibility is estimated as:

14

where, is the best solution found so far and is the exponential function for adaptive movement.

Capturing Prey: The secretary bird quickly strikes when the snake is worn out. It frequently kicks the snake’s head with precision using its strong legs. The bird may drop the snake from a height if it is too big to kill right away. Lévy Flight is introduced to allow long jumps in search space, accelerating convergence. Large steps help explore the global search space, while small steps fine-tune accuracy. A nonlinear perturbation factor balances exploration and exploitation.

15

where, W is the weighted Lévy flight and is the nonlinear perturbation factor. The Iterative Chaotic Mapping is introduced into the exploration phase of SBO to enhance global search ability, improve convergence speed, and prevent the algorithm from getting trapped in local optima.

16

17

18

By integrating chaotic mapping, the exploration phase introduces a nonlinear and unpredictable search pattern, preventing birds from clustering too early around suboptimal solutions. where, F is the control parameter (typically set between 3.5 and 4.0 for full chaotic behavior). This ensures that the solution space is widely covered, thereby preventing early convergence to suboptimal solutions.

Evasion Strategies of Secretary Birds: Eagles, falcons, wolves, and leopards are among the natural adversaries of secretary birds, and they may attack them or try to take their food. These birds use two primary evasive techniques for self-defence:

• Flight or Rapid Running

• Camouflage in the Environment

A dynamic perturbation component DPF is added to strike a balance between exploitation (improving existing solutions) and exploration (looking for new ones):

19

where, d is the current iteration number, and D is the maximum number of iterations. The factor dynamically adjusts the algorithm’s behavior, enhancing either exploration or exploitation depending on the phase of optimization.

Camouflage in the Environment: Secretary birds can blend into their surroundings using their feather patterns and terrain structures. If they detect a predator nearby, they first attempt to hide within vegetation, rocks, or grasslands. If a suitable camouflage spot is not found, they switch to the flight or running strategy. If the secretary bird hides using camouflage, its position update is given by:

20

where, is the finest location, RM is the random value following a normal distribution, is the current location and is the perturbation factor that adjusts exploration and exploitation. The term introduces randomness to ensure variability in position updates.

Flight or rapid running: Secretary birds can sprint quickly because of their large legs. They are known as marching eagles because they can travel 20 to 30 km every day. In addition to running, they are skilled at taking flight to reach safer locations quickly. When faced with immediate threats and unable to hide, they resort to rapid running or flying away. If the secretary bird escapes by flying or running, its position is updated using:

21

where, is the randomly generated array from a normal distribution, is the arbitrarily captured position update and O is the randomly selected integer (1 or 2), calculated as:

22

where generates a arbitrary factor in the interval (0,1).

Termination: Proposed ISBO stops when predefined numbers of iterations have been completed or attainment of improvement in solution quality. The pseudo-code for the ISBO algorithm is presented in Algorithm 1.

Algorithm 1.

Pseudo-code for ISBO algorithm.

In high-dimensional medical feature spaces, the Iterative Secrecy Bird Optimization (ISBO) algorithm was chosen over traditional optimizers like Particle Swarm Optimization (PSO), Genetic Algorithm (GA), and Differential Evolution (DE) because of its better exploration–exploitation balance and capacity to prevent premature convergence. ISBO incorporates chaotic mapping and adaptive perturbation tactics that introduce nonlinear search dynamics and stronger population variety, in compared to PSO, which may quickly settle to local optima, and GA/DE, which frequently necessitate significant parameter adjustment and higher processing overhead. This improves convergence stability and generalization performance by making ISBO better suited for selecting discriminative MRI features in situations with intricate inter-feature correlations. Using the solution obtained by the ISBO algorithm, the appropriate feature selection is devised.

Modified Transformer for Prostate Cancer Detection

A Modified Transformer with CapsNet instead of FFN is proposed for prostate cancer detection that leverages the strengths of both architectures to enhance accuracy and robustness in MRI image analysis. In a traditional transformer, the Feed-Forward Network (FFN) processes token embeddings after self-attention, but it treats each feature independently and lacks spatial characteristics, which is crucial in medical imaging. By replacing FFN with CapsNet, the modified transformer improves feature representation by preserving hierarchical spatial relationships within MRI scans. CapsNet uses dynamic routing to model spatial hierarchies, ensuring that the detected patterns correspond to actual tumor structures rather than unrelated textures or noise. This integration helps differentiate between cancerous and non-cancerous regions more effectively. Additionally, CapsNet’s ability to encode pose information like scale, orientation, and spatial features assist to prevent misclassification due to variations in MRI scans. The modified transformer first applies self-attention mechanisms to capture long-range dependencies in MRI images, then processes the features through CapsNet layers, allowing the model to recognize complex prostate cancer patterns while reducing false positives and false negatives. This hybrid approach enhances detection accuracy, improves model interpretability, and makes the system more robust to variations in imaging conditions, offering a powerful solution for automated prostate cancer diagnosis. The structure of M-Trans model is presented in Fig. 3.

Fig. 3.

Structure of M-Trans model.

Input layer of the M-Trans model based prostate cancer detection model comprises of patch embedding and position embedding for converting the input feature patches into tokenized vectors. Here, after performing the patch tokenization, each patch is applied with the learnable linear transformation for obtaining the flattened vector. It is interpreted as:

23

where, is the embedded patch representation, v is the embedding dimension and is the bias. The patch size is symbolized as L, number of patches is indicated as G, a is the embedding dimension and input feature size is indicated as D. Positional encoding is added to retain spatial information:

24

where, is the positional encoding. To identify long-range dependencies in input features, the transformer employs self-attention. By giving varying degrees of relevance to various regions, it extracts important features. The multi-head self-attention layer’s query, key, and value matrices are interpreted as follows:

25

where, refers the learnable weights. Then, scaled dot-product attention is employed for assigning weights as:

26

where, is the dimension of the key vectors. Then, the outcome of the multi-head attention output :

27

where, is the attention output from each head and is the final projection matrix. After performing the layer normalization and residual connection stabilizes training by normalizing the features and preserves essential feature information through residual connections. The output from multi-head attention is normalized as:

28

where, the linear normalization is indicated as . Instead of the standard FFN, CapsNet is employed to improve feature representation and classification. The feature vectors from multi head self-attention are passed into convolutional capsules that extract spatial relationships. CapsNet enhances hierarchical spatial feature representation and captures spatial relationships between different image parts. The primary capsule layer converts the transformer’s attention output into capsule representations. Each capsule has a vector representation :

29

where, and are adjustable factors and χ is a non-linearity function. Dynamic routing between capsules ensures important spatial features are routed correctly. Here, coupling coefficient controls the information flow between capsules and is formulated as:

30

where, is an initialized agreement score. The transformation of the capsule vectors is:

31

Final capsule output aggregated through dynamic routing is defined as:

32

Then, squashing function ensures the capsule outputs remain within a unit vector range.

33

where, . Finally, classification layer converts the final capsule representations into prostate cancer detection using the softmax layer.

34

where, and are adjustable factors. Thus, the M-Trans model based on CapsNet significantly enhances MRI-based prostate cancer detection, improving classification accuracy compared to traditional deep learning models.

Result and discussion

The implementation of the proposed ProstateNet model is done on PYTHON programming language and is assessed based on various measures. Table 1 shows the implementation details and training configuration of ProstateNet. The dataset utilized for the processing of the proposed ProstateNet model is:

Table 1.

Implementation Details and Training Configuration of ProstateNet.

Component	Parameter	Value/Description
Input & Preprocessing	Image size	224 × 224
	Normalization	Pixel values scaled to [0, 1]
	Noise removal	Bilateral Filter
	Spatial kernel size	5
	Spatial standard deviation ()	2
	Intensity standard deviation ()	0.1
	Data augmentation	Rotation (± 15°), horizontal flip, zoom (0.1)
Residual Recurrent Model (RRM)	Backbone network	ResNet-50 (ImageNet pretrained)
	Residual stages used	4
	Recurrent unit	Simple Recurrent Unit (SRU)
	SRU hidden units	128
	Activation function	ReLU
	Dropout rate	0.5
	Output feature dimension	512
ISBO Feature Selection	Population size	30
	Maximum iterations	50
	Chaotic map	Logistic map
	Feature selection threshold	0.5
	Fitness function	Classification accuracy with feature reduction penalty
Modified Transformer (M-Trans)	Patch size	16 × 16
	Number of patches	196
	Embedding dimension	256
	Number of transformer layers	4
	Attention heads	8
	Normalization	Layer normalization with residual connections
CapsNet (FFN Replacement)	Primary capsules	32
	Capsule dimension	16
	Dynamic routing iterations	3
	Squashing function	Standard capsule squashing
Training Configuration	Optimizer	Adam
	Learning rate	0.0001
	Batch size	16
	Epochs	100
	Loss function	Cross-entropy loss
Implementation Setup	Framework	TensorFlow/Keras
	Programming language	Python
	Hardware	NVIDIA GPU (RTX 3080, 10 GB VRAM)

Dataset Description: The dataset,²⁸ is taken from Transverse Plane Prostate Dataset that comprises of 1528 images. The images are taken from 64 patients with training and validation images of 70% and 30% respectively. From the available images:

Train

• Clinically Significant: 536 images

• Clinically Non-Significant: 536 images

Validation

• Clinically Significant: 228 images

• Clinically Non-Significant: 228 images

Thus, the dataset is balanced, with an equal number of images in both classes. The categorization of prostate cancer in this research into clinically significant and clinically non-significant categories conforms to accepted clinical risk classification guidelines. Clinically significant cases are usually associated with suspicious lesions classified as PI-RADS 4–5 (Prostate Imaging Reporting and Data System) on multiparametric MRI, indicating a high probability of clinically relevant malignancy requiring intervention. These cases typically correspond to Gleason Grade Group ≥ 2 (Gleason score ≥ 3 + 4 = 7).In contrast, Gleason Grade Group 1 (Gleason score 3 + 3 = 6) and clinically non-significant cases are frequently linked to PI-RADS 1–3 lesions, which indicate low-risk, indolent disease that is frequently treated with active surveillance. Thus, the binary classification performed by the proposed model directly corresponds to clinically meaningful risk stratification used in routine urological practice.

The experimental outcome of the proposed ProstateNet model for the input, pre-processed and detected outcome is presented in Fig. 4.

Fig. 4.

Experimental outcome of proposed ProstateNet.

Accuracy: The overall correctness of the model’s predictions is measured by accuracy, which is shown in Table 2 and Fig. 5. In the ProstateNet model, the use of advanced image pre-processing, optimal feature extraction through RRM, and robust classification using M-Trans model that results in high accuracy by minimizing both over-fitting and under-fitting.

Table 2.

Accuracy Analysis.

K-Fold/methods	Faster RCNN	ResNet	EOML	WSUNet	Proposed
2	79.5689	77.1311	79.5805	82.2538	86.4819
4	82.293	80.9655	83.8589	85.866	87.8714
6	86.3496	85.6866	84.8032	88.5872	92.5934
8	88.5144	87.298	89.9705	94.4211	98.427
10	90.6816	90.6883	93.5826	95.8102	98.9601

Fig. 5.

Accuracy Analysis.

Precision: Fig. 6 and Table 3 show precision, which is a measure of the percentage of true positive predictions among all positive predictions. High accuracy in the context of ProstateNet-based prostate cancer diagnosis indicates that the model has a high probability of being right when it identifies a location as malignant. The proposed model’s use of CapsuleNet enhances spatial relationship understanding, which reduces misclassifications, thus increasing precision.

Fig. 6.

Precision Analysis.

Table 3.

Precision Analysis.

K-Fold/methods	Faster RCNN	ResNet	EOML	WSUNet	Proposed
2	78.6066	77.7149	79.6865	83.6389	88.6943
4	80.9185	81.7889	84.4197	85.8099	89.6855
6	84.7731	83.883	85.6328	87.8239	92.659
8	87.9653	89.2803	89.0485	92.3401	97.176
10	89.8408	88.7272	91.3596	94.4334	99.0466

Recall: As shown in Fig. 7 and Table 4, recall quantifies the percentage of true positives that the model correctly detects. High recall is critical in medical diagnosis to ensure that few cancerous regions go undetected. By combining the strengths of RRM and the M-Trans model, wherein the proposed ProstateNet model improves feature learning, enabling it to capture subtle patterns in MRI images, which in turn boosts the recall rate.

Fig. 7.

Recall Analysis.

Table 4.

Recall Analysis.

K-Fold/methods	Faster RCNN	ResNet	EOML	WSUNet	Proposed
2	79.0455	75.2928	80.7862	80.335	86.9297
4	82.6798	79.366	83.099	84.4099	87.9234
6	85.8744	83.6619	86.0733	88.045	91.999
8	87.5239	85.5325	88.825	91.6802	95.6342
10	91.381	89.8285	93.3403	94.6545	97.5056

Specificity: Specificity, which is shown in Fig. 8 and Table 5, is the percentage of actual negative predictions among the actual negatives. A high specificity for the detection ProstateNet model means that it can accurately identify non-cancerous areas. The refined features selected using ISBO help to filter out noise and irrelevant information, thereby reducing false alarms. Combined with the enhanced spatial discrimination provided by CapsuleNet, the model exhibits superior specificity, ensuring that benign cases are not mistakenly flagged as cancerous.

Fig. 8.

Specificity Analysis.

Table 5.

Specificity Analysis.

K-Fold/methods	Faster RCNN	ResNet	EOML	WSUNet	Proposed
2	78.4201	76.4705	79.9365	81.6696	86.2188
4	81.453	79.9365	82.7527	84.7024	87.5186
6	84.4858	84.0525	85.7856	88.8184	91.8512
8	86.8687	84.919	89.2516	92.5011	95.1007
10	90.5514	89.2516	94.4508	94.884	96.8337

F-Score: The F-Score, which is shown in Fig. 9 and Table 6, is the harmonic mean of precision and recall, providing a balance between the two. It offers a single statistic that takes false negatives and false positives into consideration. The improved feature selection through the ISBO algorithm, along with the spatial precision of CapsNet, results in a higher F-Score for ProstateNet model. It indicates that the model effectively minimizes both types of errors, achieving a robust balance between sensitivity and specificity.

Fig. 9.

F-Score Analysis.

Table 6.

F-Score Analysis.

K-Fold/methods	Faster RCNN	ResNet	EOML	WSUNet	Proposed
2	79.3687	75.9536	78.1447	80.7772	86.7119
4	79.3767	81.1273	83.7614	82.6486	88.1445
6	85.6542	82.7784	84.7501	88.2661	92.2185
8	87.0859	85.3147	88.3877	91.681	97.1736
10	89.6189	88.7296	92.0196	94.4318	97.725

MSE: As shown in Fig. 10 and Table 7, MSE is utilized to assess the difference between the predicted output and the ground truth, especially when training regression components or reconstructing features in specific network layers. Lower MSE values in the ProstateNet model show that the network is accurately identifying the underlying patterns in the data. Pre-processing with a bilateral filter and fine-grained feature extraction using RRM both help to reduce error, which lowers the MSE and improves model reliability.

Fig. 10.

MSE Analysis.

Table 7.

MSE Analysis.

K-Fold/methods	Faster RCNN	ResNet	EOML	WSUNet	Proposed
2	0.20431	0.22869	0.2042	0.17746	0.13518
4	0.17707	0.19034	0.16141	0.14134	0.12129
6	0.1365	0.14313	0.15197	0.11413	0.07407
8	0.11486	0.12702	0.10029	0.05579	0.01573
10	0.09318	0.09312	0.06417	0.0419	0.0104

Accuracy-Loss: The accuracy-loss curve helps in analysing over-fitting and under-fitting during training, which is presented in Fig. 11. The proposed ProstateNet model achieves faster convergence due to CapsNet’s spatial awareness and ISBO’s feature selection. A smaller accuracy-loss gap signifies better generalization and stability.

Fig. 11.

Accuracy-Loss Analysis.

Confusion Matrix: The confusion matrix provides a detailed performance of ProstateNet model by displaying the accuracy of the two various classes. For a binary classification task for distinguishing between significant (cancerous) and non-significant (non-cancerous) regions, the confusion matrix is presented in Fig. 12.

Fig. 12.

Confusion Matrix Analysis.

Convergence Analysis: Convergence analysis assesses how quickly and steadily the model reaches an optimal state during training is shown in Fig. 13. In ProstateNet model, the integration of the ISBO algorithm enhances the convergence of the proposed framework. In this, Iterative chaotic mapping helps escape local minima and enhances global search capabilities, leading to more optimal feature subsets.

Fig. 13.

Convergence Analysis.

Comparative Discussion: The comparative discussion of the proposed ProstateNet model with the best outcome is presented in Table 8. The best Accuracy, Precision, Recall, Specificity, F1-Score, and MSE derived by the proposed ProstateNet model is 98.9601%, 99.0466%, 97.5056%, 96.8337%, 97.725%, and 0.0104 respectively.

Table 8.

Comparative Discussion.

Metrics/methods	Faster RCNN	ResNet	EOML	WSUNet	Proposed
Accuracy	90.6816	90.6883	93.5826	95.8102	98.9601
Precision	89.8408	88.7272	91.3596	94.4334	99.0466
Recall	91.381	89.8285	93.3403	94.6545	97.5056
Specificity	90.5514	89.2516	94.4508	94.884	96.8337
F1-Score	89.6189	88.7296	92.0196	94.4318	97.725
MSE	0.09318	0.09312	0.06417	0.0419	0.0104

The high performance achieved by the proposed ProstateNet model can be attributed to several dataset-specific and architectural factors. With 536 clinically significant and 536 clinically non-significant images in the main folders, the dataset maintains a balanced class distribution, preventing bias toward a dominating class and fostering steady training dynamics. Effective pre-processing techniques, such as image normalization, scaling, and bilateral filtering, were used before to training in order to increase generalization by lowering noise and improving feature consistency across samples. Additionally, a CapsNet-based transformer classifier and Residual Networks with SRU are integrated in the proposed feature extraction technique, allowing the model to efficiently capture both spatial hierarchies and contextual relationships. Part-whole anatomical linkages are maintained through capsule networks, which is especially significant in medical imaging where structural integrity is essential for diagnosis. Furthermore, by optimizing feature selection and convergence behavior, the ISBO algorithm directs the model toward a more ideal solution space and strengthens classification boundaries. A 70%–30% training–validation split was used to further reduce overfitting, and regularization strategies like dropout, batch normalization, and controlled training were used. The stated high accuracy is supported by the near alignment between training and validation accuracy, which suggests controlled generalization rather than memory.

Here, the superior performance of the proposed ProstateNet model is achieved through a meticulously designed architecture that combines advanced pre-processing, robust feature extraction, optimal feature selection, and a modified classification framework. Each component contributes to enhanced performance metrics such as high accuracy, precision, recall, F-Score, and specificity, along with low MSE. The detailed analysis using a confusion matrix and convergence analysis further validates that the proposed methodology provides a reliable and efficient solution for prostate cancer detection from MRI images, making it highly effective for real-world clinical use.

The cross-dataset validation findings of the proposed ProstateNet model are shown in Table 9, indicating both its generalization capacity and resilience. The model performs well when tested and trained on the same dataset, with 98.96% accuracy and 0.0104 MSE for Dataset A²⁹ (PROSTATEx dataset) and 97.84% accuracy and 0.0142 MSE for Dataset B^30,31 (Microarray Gene Expression Dataset). When trained and evaluated on distinct datasets, performance significantly reduces but stays strong in cross-dataset evaluation, attaining 94.82% and 93.67% accuracy. Overall, the findings demonstrate that the model generalizes to different data distributions.

Table 9.

Cross-Dataset Validation Results.

Training Dataset	Testing dataset	Accuracy (%)	Precision (%)	Recall (%)	Specificity (%)	F-Score (%)	MSE
Dataset A	Dataset A (Internal)	98.96	99.04	97.51	96.83	97.72	0.0104
Dataset A	Dataset B (External)	94.82	95.36	93.91	92.74	94.62	0.0287
Dataset B	Dataset A	93.67	94.12	92.48	91.39	93.29	0.0315
Dataset B	Dataset B (Internal)	97.84	98.11	96.92	95.87	97.50	0.0142

Using a paired t-test, Table 10 shows the statistical significance analysis of the proposed ProstateNet model in comparison to other current techniques as Faster RCNN, ResNet, EOML, and WSUNet. The proposed model’s performance gains are statistically significant, as shown by the computed p-values of 0.0032, 0.0018, 0.0045, and 0.0121, all of which are below the significance threshold of 0.05. This demonstrates that the observed performance improvements are a result of the proposed architecture’s efficacy rather than random variance. Consequently, ProstateNet outperforms all evaluated baseline models in a statistically significant way.

Table 10.

Statistical Significance analysis (paired t-test results).

Comparison	p-value	Significant (p < 0.05)
Proposed vs Faster RCNN	0.0032	Yes
Proposed vs ResNet	0.0018	Yes
Proposed vs EOML	0.0045	Yes
Proposed vs WSUNet	0.0121	Yes

The ablation study of the proposed ProstateNet framework is shown in Table 11, highlighting the role that each module performs in overall performance. With an accuracy of 92.14% and an MSE of 0.0632, the Baseline Transformer shows a slightly poor capacity to represent features. The accuracy increases to 94.36% when SRU is removed, demonstrating that sequential modeling enhances performance. Accuracy rises to 95.48% when ISBO is excluded, demonstrating that the best feature selection improves discriminative power and lowers error. Although 96.87% accuracy is obtained when the traditional FFN is used in place of CapsNet, it is still less accurate than the whole architecture. With 98.96% accuracy, 99.04% precision, 97.51% recall, 97.73% F-Score, and the lowest MSE of 0.0104, the Full Proposed Model performs the best, proving that the combined integration of SRU, ISBO, and CapsNet within the modified transformer greatly enhances feature learning, spatial hierarchy preservation, and classification robustness.

Table 11.

Ablation study.

Model variant	Accuracy (%)	Precision (%)	Recall (%)	F-Score (%)	MSE
Baseline transformer	92.14	92.88	90.73	91.79	0.0632
Without SRU	94.36	94.92	93.11	93.99	0.0418
Without ISBO	95.48	96.02	94.27	95.14	0.0325
Without capsNet (FFN used)	96.87	97.33	95.94	96.63	0.0187
Full proposed model	98.96	99.04	97.51	97.73	0.0104

The cumulative ablation analysis of the suggested ProstateNet framework is shown in Table 12, showing the increasing contribution of each module to overall performance. With a typical FFN, the baseline Transformer achieves an accuracy of 92.14% and an MSE of 0.0632. Accuracy is increased to 93.87% by using the Bilateral Filter, which enhances edge preservation and noise suppression. By increasing accuracy to 95.94%, the Residual Recurrent Model (RRM) dramatically improves spatial-sequential feature learning. Including ISBO improves feature selection even more, lowering MSE to 0.0214 and attaining 97.08% accuracy. The efficacy of hierarchical spatial representation is finally confirmed by substituting CapsNet for FFN, which results in the maximum accuracy of 98.96% and the lowest error.

Table 12.

Independent component removal ablation analysis of prostateNet.

Model Variant	Accuracy (%)	Precision (%)	Recall (%)	F1-Score (%)	MSE
Baseline Transformer (FFN)	92.14	92.88	90.73	91.79	0.0632
+ Bilateral Filter	93.87	94.12	92.35	93.23	0.0527
+ RRM	95.94	96.22	94.68	95.44	0.0398
+ ISBO	97.08	97.61	95.92	96.75	0.0214
+ CapsNet (Full Model)	98.96	99.04	97.50	97.72	0.0104

Table 13 compares the efficiency and complexity of the models using various approaches. Faster RCNN uses more memory, has more parameters, and takes longer to compute than EOML. The proposed ProstateNet has competitive training and inference times while maintaining a moderate parameter size and GPU use. ProstateNet achieves a fair compromise between computing performance and model complexity. Figure 14 shows the Grad-CAM based attention visualization of ProstateNet for significant and non-significant MRI cases.

Table 13.

Comparative analysis of model complexity and efficiency.

Model	Parameters (Millions)	Training time/epoch (s)	Inference time/image (ms)	GPU memory usage (GB)
Faster RCNN	41.2 M	62	28	6.1
ResNet	25.6 M	38	12	4.3
EOML	18.9 M	35	10	3.8
WSUNet	29.4 M	44	15	4.9
Proposed (ProstateNet)	32.7 M	49	14	5.2

Fig. 14.

Grad-CAM visualization of prostatenet for significant and non-significant MRI Cases.

The comparative ROC-AUC values of various prostate cancer detection techniques are shown in Table 14. The capacity of each model to differentiate between clinically significant and non-significant cases across all classification levels is measured by the Area Under the Curve (AUC). Better sensitivity–specificity balance and discriminative performance are indicated by a higher AUC value. With an AUC value of 0.991, the proposed ProstateNet outperforms all other models in terms of classification performance and nearly flawless class separability. With an AUC of 0.912, ResNet (0.925), EOML (0.941), and WSUNet (0.963), it performs better than Faster R-CNN. ProstateNet offers better threshold-independent performance and more dependable diagnostic discrimination than the current techniques, as demonstrated by the significant increase in AUC.

Table 14.

ROC-AUC performance comparison.

Model	AUC
Faster RCNN	0.912
ResNet	0.925
EOML	0.941
WSUNet	0.963
Proposed (ProstateNet)	0.991

A comparison of the proposed CapsNet-based Modified Transformer (M-Trans) with current transformer-based and hybrid architectures is shown in Table 15. While the hierarchical Swin Transformer boosts performance to 96.11%, the Vision Transformer (ViT) attains 94.72% accuracy. The CNN–Transformer hybrid further improves accuracy to 97.03%, highlighting the advantages of integrating attention and convolutional methods. Nevertheless, with the lowest MSE of 0.0104, 98.96% accuracy, 99.04% precision, and 97.50% recall, the proposed M-Trans model performs better than any of the approaches that were examined. These results highlight the effectiveness of integrating CapsuleNet within the transformer framework for improved spatial hierarchy modeling and classification robustness.

Table 15.

Performance comparison with transformer-based models.

Model	Architecture type	Accuracy (%)	Precision (%)	Recall (%)	F1-Score (%)	MSE
Vision transformer (ViT)	Pure transformer	94.72	95.18	93.94	94.55	0.0368
Swin transformer	Hierarchical transformer	96.11	96.74	95.03	95.88	0.0285
CNN–transformer hybrid	Hybrid	97.03	97.22	96.41	96.81	0.0219
Proposed (CapsNet-based M-Trans)	Modified Transformer	98.96	99.04	97.50	97.72	0.0104

The computational complexity of the proposed ProstateNet and traditional CNN-based models is compared in Table 16. Although it uses the fewest parameters and resources, the Simple CNN’s representational capacity is constrained. Moderate amounts of GPU memory and training time are needed for ResNet and WSUNet. The proposed ProstateNet exhibits an efficient trade-off between computational cost and enhanced diagnostic performance, with slightly greater parameters (32.7 M) and training time (49 s/epoch) while maintaining competitive inference speed (14 ms) and appropriate GPU utilization (5.2 GB).

Table 16.

Computational Complexity Comparison of Proposed ProstateNet with CNN-Based Models.

Model	Parameters (Millions)	Training Time/Epoch (s)	Inference time/image (ms)	GPU memory (GB)
Simple CNN (5 Conv layers)	6.8 M	21	6	2.1
ResNet	25.6 M	38	12	4.3
WSUNet	29.4 M	44	15	4.9
Proposed (ISBO + CapsNet + SRU)	32.7 M	49	14	5.2

Conclusion

The proposed ProstateNet model for prostate cancer detection from MRI images integrates advanced deep learning techniques, feature optimization, and spatially aware classification to achieve superior diagnostic accuracy. By leveraging a Bilateral Filter for noise reduction, a RRM for feature extraction, Iterative Secrecy Bird Optimization (ISBO) for optimal feature selection, and a M-Trans, the model effectively enhances precision, recall, specificity, and F-score while reducing false positives and false negatives. The achievement of the proposed model is the ability to preserve spatial relationships within MRI images, which significantly improves classification performance compared to conventional CNN-based models. To ensure a balanced distribution of clinically significant and non-significant cases across folds, stratified k-fold cross-validation was used in the current investigation. Strict class balance was maintained in both training and validation sets, and data splitting was carried out at the image level. However, there was no explicit enforcement of patient-level separation, which might result in data leakage from several images of the same patient appearing in distinct folds. This restriction has been recognized, and in order to improve generalization reliability even further, future research will include stringent patient-wise cross-validation and multi-institutional external validation. Additionally, ability techniques such as attention visualization and interpretability models could be explored to make the predictions more transparent for clinical applications.

Author contributions

M.R.P.: Conceptualization, Methodology, Software, Data curation, Validation, Formal analysis, Writing–original draft. K.S.V.: Conceptualization, Methodology, Software, Resources, Validation, Visualization, Project administration, Supervision, Writing–original draft. C.K.: Formal analysis, Investigation. Abdulla Alwabli: Funding acquisition, Investigation, Writing–review & editing.

Funding

Research didn’t receive any fund.

Data availability

Publicly available in a repository https://www.kaggle.com/datasets/tgprostata/transverse-plane-prostate-dataset/data

Declarations

Competing interests

The authors declare no competing interests.

Informed Consent

Not applicable.