Reviewing 3D convolutional neural network approaches for medical image segmentation

Abstract

Background

Convolutional neural networks (CNNs) assume pivotal roles in aiding clinicians in diagnosis and treatment decisions. The rapid evolution of imaging technology has established three-dimensional (3D) CNNs as a formidable framework for delineating organs and anomalies in medical images. The prominence of 3D CNN frameworks is steadily growing within medical image segmentation and classification. Thus, our proposition entails a comprehensive review, encapsulating diverse 3D CNN algorithms for the segmentation of medical image anomalies and organs.

Methods

This study systematically presents an exhaustive review of recent 3D CNN methodologies. Rigorous screening of abstracts and titles were carried out to establish their relevance. Research papers disseminated across academic repositories were meticulously chosen, analyzed, and appraised against specific criteria. Insights into the realm of anomalies and organ segmentation were derived, encompassing details such as network architecture and achieved accuracies.

Results

This paper offers an all-encompassing analysis, unveiling the prevailing trends in 3D CNN segmentation. In-depth elucidations encompass essential insights, constraints, observations, and avenues for future exploration. A discerning examination indicates the preponderance of the encoder-decoder network in segmentation tasks. The encoder-decoder framework affords a coherent methodology for the segmentation of medical images.

Conclusion

The findings of this study are poised to find application in clinical diagnosis and therapeutic interventions. Despite inherent limitations, CNN algorithms showcase commendable accuracy levels, solidifying their potential in medical image segmentation and classification endeavors.

1. Introduction

The proliferation of medical imaging technologies has led to a significant increase in the generation of medical images within healthcare institutions. Daily, a substantial volume of medical images is generated across global medical facilities. Diverse forms of medical imagery, including microscopy, X-ray, ultrasound (US), computed tomography (CT), magnetic resonance imaging (MRI), and positron emission tomography (PET), find extensive applications in imaging systems. The foremost advantage of medical images lies in their utility for asymptomatic conditions. Furthermore, they serve to identify injuries, conditions, and diseases in their incipient stages. Among the various facets of medical image processing and analysis, medical image segmentation holds an impressive position. Image segmentation involves the recognition and demarcation of objects. Recognition pertains to the localization of anomalies and organs, while delineation encompasses outlining the procedure of recognition [1]. Accurate segmentation of medical images contributes to radiotherapy, disease quantification, and image-guided surgery. Put simply, segmentation tasks aid in the early detection, diagnosis, and treatment monitoring of ailments.

A diverse array of segmentation techniques exists, which encompasses traditional methods, graph-based approaches, semantic segmentation methods, classification and clustering methods, deformable methods, and other modalities [2,74]. Fig. 1(A–C), shows the diagram of different medical images.

Fig. 1.

Different medical images: a) Breast Ultrasound, b) MRI, 3) CT image.

Segmentation methods can be classified into three categories: manual, semi-automatic, and automatic procedures. Manual segmentation, deemed the gold standard, entails an expert radiologist's hand-drawn ground truth. Semi-automatic segmentation employs advanced algorithms to address manual segmentation issues, though some human intervention is still required. Fully automatic segmentation, on the other hand, doesn't rely on human involvement and utilizes both learning and non-learning-based techniques, representing the latest trend. Although manual segmentation is the benchmark in medical practice, it demands substantial time and resources, prompting the need for automated approaches [3]. Automatic segmentation leverages computer-aided diagnosis (CAD) systems to assist radiologists in clinical diagnosis, treatment planning, abnormal structure delineation, and region of interest identification.

The central objective of CAD is swift, accurate, automated segmentation to facilitate communication. Crucially, the segmentation procedure varies based on image modalities, body region localization, artifact types, and noise. There isn't a one-size-fits-all method for all medical images; hence, different algorithms must be considered [4,98]. The emergence of convolutional neural networks (CNNs) has revolutionized pattern recognition and computer vision, offering researchers a rapid, accurate segmentation solution. This study concentrates on the utilization of three-dimensional (3D) CNNs for the segmentation of abnormalities and organs in medical images. While two-dimensional (2D) CNNs were previously used for medical image segmentation, the recent trend involves the development of 3D CNNs tailored for effectively segmenting three-dimensional images or 2D slices. Notably, the key distinction lies in learning representations: 3D CNNs employ input volumes with filters to generate a 3D output mask. This review contributes to the body of knowledge by evaluating diverse methods employing 3D CNNs for MRI, CT, ultrasound, and other medical image segmentations.

The study's goal is to present the outcomes of various 3D CNN algorithms employed in investigating abnormalities and organ segmentation in medical images. This analysis delves into the latest trends, which include, assessment of the performance, preprocessing techniques, and validation metrics of 3D CNN methodologies. Furthermore, it discusses pertinent roadmaps and challenges for abnormalities and organ segmentation in MRI and ultrasound images.

The structure of this paper is as follows: Section 2 outlines the review's organization, search criteria, and methodologies. Section 3, discussed different databases and preprocessing techniques. Section 4 revealed statistical analyses of diverse algorithms, while Section 5 concludes the paper.

1.1. Review organization

This review's methodological framework is categorized into three main sections: (1) Encoder/Decoder, (2) Deep Convolutional Neural Networks (DCNNs) and Fully Connected Networks (FCNs), and (3) other. The comprehensive examination of CNN segmentation within these three domains will be presented. The "other" category encompasses attention models, combinations of various CNN, and similar cases. While 2D models have been employed for segmenting multiple medical images, this review will concentrate primarily on 3D images and Networks. A block diagram portraying the various 3D CNN approaches is illustrated in Fig. 2.

Fig. 2.

Categories of 3D CNN methods for medical images.

1.2. Search criteria

Relevant research publication databases, including Springer, IEEE, Pubmed, Sciencedirect, Researchgate, and Google Scholar, were meticulously searched for pertinent articles. These selected papers were subsequently screened to identify those specific to the study area of 3D CNN segmentation in medical images. The compiled list of medical images returned indicated that MRI, CT and ultrasound were the most prevalent. To conduct the paper search, pertinent keywords were employed. After completing the keyword search process, a total of 500 papers were chosen. Keywords like 3D CNN, Medical images, segmentation of organs, segmentation of tumors, and abnormalities were utilized. A subsequent step involves search based on paper titles, leading to the selection of an additional 100 papers. In total, 600 papers were acquired through a combination of keyword and title searches. To streamline the scope, papers published before 2018 were excluded, and only those utilizing 3D CNN were retained for review. Papers focussed on animal parts and review papers were omitted from consideration.

1.3. Review of methods

This section provides an introduction to the prominent methodologies involving 3D CNNs utilized for the segmentation of medical images. The chosen publications are organized according to distinct medical image categories. The subsequent discussion delves into various 3D CNN methods, focusing on their specific CNN network types.

1.3.1. Encoder/decoder network

1.3.1.1. U-net

Originally tailored for biomedical and medical image segmentation, the U-Net framework [[5], [34]] amalgamates low and high-level features to yield exceptional outcomes. Researchers have adapted the U-Net methodology to address abnormalities/organ segmentation within medical images. Comprising an encoder-decoder structure, the U-Net incorporates convolution layers, max-pooling (for downsampling reduction), deconvolution layers, and skip connections (refer to Ronneberger et al. [5] for comprehensive insights). However, due to the evolution of complex images, the 2D U-Net proved inadequate for 3D image segmentation, necessitating the creation of the 3D U-Net. Gcek et al. [6] improved the convention U-Net to a 3D model. The model consists of 4 3 × 3 convolutions, 2 deconvolutions, and 4 ReLU [7]. Several variants of 3D U-Net have been used for segmenting abnormalities/organs in medical images. For example, the research by Qamar et al. [8] combines the dense connection [79], residual connection [81], and the inception module [80] in a U-Net framework for segmentation. Yee et al. [19] proposed a U-Net variant that consists of 2 convolutions and 8 convolutions + dilations in the encoding stage. In the decoding stage, the network consists of 8 convolutions + dilations, 1 upsampling, and 1 convolution. Meyer et al. [21] proposed a variant of the U-Net that uses the anisotropic 3D multi-stream CNN architecture. The network consists of 27 convolutions + BN [78] + ReLU, 7 max-pooling, 12 upsampling, 4 dropouts, and 1 sigmoid.

Guo et al. [27] proposed the filterNet which is a variant of the U-Net for segmenting muscle in MRI. The network consists of 3 blocks with 7 convolutions, 4 BN + ReLU, 1 hardtanh, 1 element-wise addition [84], and short skip connections [83]. Recently, MRI has been used for detecting tumors in the breast. Qiao et al. [28] used the multi-label attention-guided joint phase network for segmenting tumors in the breast using MRI. This network consists of 5 encoders with 1 input intensity map and 4 post-contrast inputs. The input images are fed into each encoder and processed with the guided self-attention network [82] and concatenated for processing to the next layer in the decoder. Each encoder consists of 5 layers, with a convolution layer acting as a buffer at the end of each encoder. The decoder has 5 single layers with a self-attention module and a concatenation module. Jiang & Guo [30] used the improved 3D U-net for segmenting the brain in MRI. The network consists of 14 convolutions + BN + ReLU layers, 3 max-pooling, 3 upsampling layers and 3 concatenate and crop layers. The fully connected conditional random fields (CRF) [85] are used to produce the label maps (see Fig. 3).

Fig. 3.

3D U-net with CRF [30].

Zhang et al. [24] proposed the triple intersecting U-Net from segmenting brain tumors. The network consists of 3 U-Net intertwined to produce a segmentation mask. The first U-Net consists of 9 layers each in both the encoder and decoder. The output of the first U-Net is concatenated with the original image to act as the input for the next U-Net. The second U-Net consists of 15 layers each in both the encoder and decoder. The last U-Net involves compatibility with the second to produce the final output. Su et al. [35] apply the multi-pathway architecture using the convolution and transpose convolution to segment tumors in MRI. This network was inspired by the U-Net and consists of 4 dilated convolutions, 6 convolutions, and 10 max-pooling in the encoder. While the decoder consists of 4 transpose convolutions, 4 concatenations, and 4 convolutions. Tang et al. [36] proposed the Dual Attention-based Dense SU-Net for segmenting tumors in head and neck regions. The network consists of 3 convolutions, 10 dense layers, 5 max-pooling, 4 PAM blocks, 5 unpooling layers, and 4 channel attention modules. This network combines the attention and the dense network with the U-Net as the backbone.

Calisto & Lai-Yuen [38] used the encoder/decoder framework to search for the best architecture for the segmentation of MRI. This network utilized two components 1) Search Procedure, and 2) Macro search structure [86]. The macro-search consists of 5 max-poolings, 5 transpose convolutions, 5 summation layers, and 1 convolution + softmax. Zhou et al. [40] proposed an efficient residual neural network (ERV-Net) for the segmentation of brain tumors in MRI. The network was first made computationally efficient with a subnetwork in the encoder phase. Then, another subnetwork uses the residual network to avoid degradation at the decoder phase. The network uses an encoder/decoder medium that consists of 13 convolutions, 13 shuffle unit strides, 5 concatenation layers, and 5 skip networks. Zhou et al. [41] proposed the iterative localization refinement for segmentation of the prostate using MRI. The encoder consists of 3 convolutions, 4 residual IBN blocks, and one 2D convolution. Meanwhile, the decoder consists of 3 residual Instance Batch Normalization (IBN [87]) blocks, 1 IBN block, 4 upsampling layers, and 1 single convolution. Overall, there are 17 layers with different convolutions, factors, and strides.

The U-Net method has also been used directly for segmentation without much modification. For example, Huo et al. [43] used the U-Net with several other networks for segmentation of the brain in MRI. Similarly, Coupe et al. [44] used the U-Net for segmenting the whole brian in MRI images. Luo et al. [64] used the U-Net for segmenting multiple organs using multiple modalities. Herrera et al. [68] used the U-Net for segmenting the brain in volumetric MRI. Niyas et al. [48] used the Res-U-Net to segment lesions in MRI. The network replaced the conventional U-Net block with two consecutive residual blocks. The encoder/decoder network consists of 4 double consecutive blocks, and 4 max-pooling. The decoder part consists of 4 convolutional layers, and 4 3D transpose convolutional layers. The encoder and decoder are concatenated with the concatenation layer, the same way it was performed in the traditional U-Net.

Cao et al. [56] proposed the dilated densely connected U-Net for segmentation of mass in breast ultrasound images. The network consists of 4 dense blocks, 3 upsampling layers, 3 up-samplings, 4 convolution + BN + ReLu layers, and 1 max-pooling. The dense block consists of feature maps, 1 transition layer, and 5 BN + ReLU + convolution. Yang et al. [57] proposed the patch-of-interest FuseNet with the hybrid loss for segmentation of tumors in ultrasound images. The network use information in the 2D U-Net and 3D U-Net for the prediction of the final segmentation. The 3D U-Net is joined with the pyramidal input and output patches. The network consists of 14 convolution + Bn/ReLU layers, 2 max-pooling, 4 deconvolution layers, 4 concatenation layers, and 2 convolution + sigmoid layers (see Fig. 4).

Fig. 4.

PIO FuseNet [57].

Zhou et al. [58] proposed a lightweight multi-scale network for the segmentation of tumors in ultrasound images. The lightweight network is an encoder-decoder network that consists of 9 stages. The network contains 4 down-sampling layers, 4 up-sampling layers, 18 Conv + BN + ReLU, and 1 Conv + BN + ReLU. Yang et al. [59] proposed the contrastive rendering (C-Rend) framework to segment ovary and follicles in ultrasound images. The C-Rend is incorporated with a semi-supervised learning framework to prove reliable superior segmentation performance. The C-Rend is a modified U-Net that consists of an encoder-decoder, render [88], upsamplers, and identity layer [89]. Wang et al. [60] proposed the multi-scale context for lesion segmentation in breast ultrasound. The network used a mixed 2D and 3D convolution module for segmentation. The multi-scale block consists of 8 convolutions, 4 concatenation layers, 4 additional layers, and the U-Net network. Shirokikh et al. [65] propose the adaptive small-scale target localization for the segmentation of abnormalities in MRI. The network is a U-Net-based framework that consists of 3 average pooling [90] layers, 3 trilinear [91] blocks, and a single loss function calculation layer. Yang [71] proposed an improved U-Net framework for the segmentation of the brain in MRI. The traditional U-Net was improved, through the addition of the squeeze-and-excitation block [72], and a dilated convolution procedure [73]. Qin et al. [104] introduced a Node Growth Neural Architecture Search, termed NG-NAS, for the purpose of 3D medical image segmentation. This architecture adopts a U-shaped design, comprising a collection of nodes. Unlike the conventional approach of exploring within a restricted search space from a supernet, NG-NAS commences with a basic architecture containing just 5 nodes. It then progressively expands the optimal candidate node through a greedy procedure until satisfying the imposed constraint. The network was meticulously structured with two distinct generations of nodes, contributing to diverse network topologies, and it incorporates a selection of four candidate operators for each node. Liu et al. [106] present an approach, referred to as the "Prior-based 3D U-Net," which leverages prior knowledge for enhanced performance. The methodology unfolds in the following stages: 1) Initially, training data is utilized to train the 3D U-Net model and create an average shape model (ASM) of the knee joint, encompassing components like cartilage, distal femur, and proximal tibia. Subsequently, the trained 3D U-Net is employed to segment the distal femur and proximal tibia from new input knee MRI data. Then the newly input knee joint structures (distal femur and proximal tibia) are registered with the ASM via the construction of a registration transformation (RT). Next, the RT transformation then maps the ASM to the newly input MRI images, leading to the emergence of a predicted cartilage model. Finally, the predicted cartilage model, along with the distal femur and proximal tibia previously segmented by the initial U-Net step, serves as constraints for the subsequent 3D U-Net segmentation of the knee cartilage.

1.3.1.2. V-net

The V-Net is a 3D deformable structure proposed by Milletari et al. [9] and has been used to segment abnormalities and organs in medical images. Han et al. [10] cascaded the V-Net for whole heart segmentation of abnormal connections. This network consists of two V-Net frameworks joined and cascaded for heart segmentation. Jin et al. [14] used the improved V-Net (PBV-Net) to segment the prostate gland from MRI images. The network consists of 5 convolutions, 5 concatenations, 5 deconvolutions, and a single softmax and deconvolution layer.

1.3.1.3. Seg-Net

The Seg-Net [16] is a pixel-wise semantic segmentation framework that upsamples features in the decoder procedure with lower resolution input feature maps. Seg-Net provides a memory against accuracy tradeoff with good segmentation performance. Several modifications have been proposed for the segmentation of medical images. For example, Yamanakkanavar & Lee [15], proposed the M-Seg-Net that used the global attention mechanism [109] as a skip connection between the decoder and encoder. The network consists of 7 convolutions, 7 deconvolutions, 6 max-pooling, and 6 up-sampling layers. The global attention process the class localization details. Qin et al. [32] proposed the multi-scale discriminative network with the pyramid attention module (PAM [92]) and residual refinement [93] block (RRB) for segmentation of prostate in MRI. The Res layer [33] was used as the backbone encoder, while the PAM and RRB are used as the decoder. Each PAM consists of 2 summation layers, 1 multiplication and concatenation layer, and different convolutions. Meanwhile, each RRB consists of 3 convolutional layers, 1 passthrough, and 1 summation layer.

1.3.1.4. Alex-Net

The Alex-Net [22] is an 8-layer network with learnable parameters. The input format of the data source is in 3 dimensions with the network having 5 convolutions, 3 max-pooling, 1 dropout, and 5 ReLU layers. Chen et al. [23] proposed an improved Alex-Net to segment prostate tumors in MRI images. This network adds the BN and the global maximum pooling [94] algorithm to the conventional Alex-Net.

1.3.1.5. PSP-net

Pyramid Scene parsing network (PSPNet) [25] is a semantic segmentation model that learns and trains modules that can segment and classify pixels. Yan et al. [24] used the PSP-Net for segmenting prostate in MRI. The network is an encoder/decoder framework that consists of 9 convolutions, global average pooling, 3 upsampling, and 4 skip connections.

1.3.1.6. Others

Ke et al. [46] proposed the encoder/decoder dense network for segmentation of carcinoma in MRI. The network is a self-constrained method with an end-to-end pattern. It consists of 9 convolutions, 21 upsampling layers, 4 denseblock_down layers, 8 denseblock_down + transition block down layers, 5 concatenation layers, 1 multiplication layer, global average pooling, and 1 fully connected layer (See Fig. 5).

Fig. 5.

3D self-constrained DenseNet [46].

Celik & Talu [49] combine the generative adversarial network (GAN) [50] with the atrous convolution for segmentation of the brain in MRI. The network uses the collaborative effects of the generator and the discriminator for segmentation. The generator is an encoder/decoder U-shaped mechanism that consists of 11 convolutions, an atrous convolution feature pyramid (ACFP) with a position attention module (PAM) [52], and 2 concatenation layers. The ACFP and PAM act as the connection between the encoder and the decoder and consist of 3 convolutions, 2 upsampling, and 2 concatenation layers. The discriminator was used to classify the output of the generator. Cui et al. [53] proposed an end-to-end learning-based method (TSegNet) for the segmentation of teeth in dental models. The network consists of 18 fully connected layers, and 3 channel concatenation in the encoder. While the decoder consists of 9 fully connected layers and 2 channel concatenation. Organs segmented include, incisors, canines, premolars, and molars for the upper and lower teeth.

Dong et al. [62] proposed the deep atlas network for segmentation of ventricle in echocardiography. The network consists of a transformer and deformable networks. The transformer network contains 5 convolutions, 4 BN + ReLU, 4 max-pooling layers, and a single FC layer. The deformable network used trilinear sampling [63] to produce the final output. Meng et al. [75] proposed the 3D CNN with dual parts for the segmentation of the liver in MRI. The network has two parts 1) local pathway and 2) global pathways. Each pathway consists of 8 blocks and 3 concatenation layers. Each block contains 1 convolution, 1 BN, and 1 ReLU. At the end of the pathways, 2 fully connected dropouts with a softmax are used to produce the final output. The summary of methods in this section is available in Table 1.

Table 1.

Summary of Encoder/Decoder method.

CNN Type	Ref	Common image type	Popular Abnormalities/Organ	Coverage publication year	Average (DICE)	Average (Hausdorff)
U-Net	See Fig. 2	Brain, tumor, prostrate	MRI, CT, US	2019, 2020, 2021, 2022, and 2023	0.96	2.04
V-Net	See Fig. 2	Heart, prostate	MRI	2020, 2021	0.94	1.52
Seg-Net	See Fig. 2	Brain, prostate	MRI	2020, 2021	0.96	3.16
GAN	See Fig. 2	Ventricle, Brain	MRI, Echocardiography	2020, 2022	0.96	1.15
AlexNet and PSP-Net	See Fig. 2	Prostate	MRI	2021	0.97	1.02
Others	See Fig. 2	Liver, teeth, Carcinoma	MRI	2020, 2021	0.96	2.02

1.4. Summary of encoder/decoder methods

Researchers have explored the possibility of amalgamating two or more encoder/decoder frameworks. For instance, Martins et al. [55] harnessed insights from both U-Net [110] and V-Net architectures to segment cerebral ventricles in ultrasound images. This emerging trend yielded impressive accuracy, while also addressing associated intricacies common with medical image segmentation. Notably, the 3D U-Net emerged as the most widely employed network for segmenting ultrasound images. In breast ultrasound segmentation, detecting common abnormalities like tumors and lesions is crucial, prompting authors to tailor the U-Net framework accordingly. Intriguingly, a combination of 2D and 3D approaches holds promise for enhancing accuracy in medical image segmentation. Mlynarski et al. [75] skillfully merge both 2D and 3D techniques for brain segmentation in MRI scans. This synergy between distinct CNN types has the potential to mitigate redundancy and overfitting. Fig. 6 illustrates a graph detailing the frequency of segmentation utilization for each framework.

Fig. 6.

Number of the framework for encoder/decoder.

The encoder/decoder network exhibits certain common limitations, such as the potential loss of neighboring or local information due to successive pooling operations or convolution striding, resulting in decreased feature map resolution. Encoders serve as feature extraction units, while decoders function as fusion and processing components. A valuable takeaway from this section underscores the efficacy of encoder and decoder networks for medical image segmentation. In summary, the U-Net, AlexNet, PSPS-Net, Seg-Net, V-Net, and GAN emerge as dependable 3D segmentation frameworks for medical images.

1.5. DCNN and FCN methods

Deep Convolutional Neural Networks (DCNNs) are specifically crafted to acquire knowledge about filters and systematically amalgamate these acquired filters. Their predominant application lies within segmentation and classification endeavors, particularly when handling extensive datasets. Conversely, Fully Connected Networks (FCNs) encompass architectures wherein layers establish localized connections, thereby sidestepping the need for dense parameters. FCNs predominantly find application in semantic segmentation tasks, demonstrating remarkable efficiency in their training process.

1.5.1. DCNN

Charron et al. [11], used the 3D convolutional neural network (DeepMedic [12]) to segment and detect brain metastases on MRI images. The network uses a multi-scale CNN [109] with 2 convolutional pathways [54] and 11 layers (see Fig. 7).

Fig. 7.

DenseAFPNet [11].

Zhou et al. [13] proposed a DCNN framework that consists of a backbone, atrous convolution [95], and classification layers. Interestingly, the DeepMedic architecture is the inspiration for this research. Wei et al. [18] proposed the cascaded nested network (CasNet) for segmentation of brain MRI. The CasNet is a twin nested segmentation network that consists of 5 side blocks, 3 core blocks + pooling, 3 core blocks + deconvolution, 2 core blocks, and 1 convolution. Each core block consists of 2 convolutions, 2 ReLU, 2 BN, and a single dropout. Each side block consists of 1 core block, 1 convolution, and 1 deconvolution. Dolz et al. [26] proposed the ensemble 3D deep CNN for segmenting infant brain in MRI. This network was built on the DeepMedic and uses a combination of 2D on 3D models. The network consists of convolutional layers, fully connected layers, and classification layers.

Zhou et al. [31] used the atrous-convolution feature pyramid method as the backbone for feature learning to segment tumors in MRI. The CRF was used as the post-processing method to obtain structural segmentation. The network consists of 4 parts 1) backbone, 2) feature pyramid, 3) classification, and 4) processing. The backbone consists of 9 convolutions and 3 atrous convolution layers. While the feature pyramid has 9 atrous convolutional layers and 2 upsampling layers. The classification part consists of 6 convolution layers and 1 softmax. Finally, the processing part consists of a fully connected layer and CRF. Qui & Ren [61] proposed a deeply supervised network for automatic segmentation of Liver and heart volumetric images. The network conducts volume-to-volume learning and eliminates overfitting and gradient vanishing problems. This network consists of 6 convolutions, 2 max-pooling, 5 deconvolutions, and 3 softmax layers. The output from different softmax layers is fused to produce the final label.

1.5.2. FCN

Wu & Tang [17] proposed the multi-atlas 3D FCN ensemble model for the segmentation of the brain in MRI images. The network consists of 8 convolutions, 6 max-pooling, 3 convolutions + ReLU, 3 deconvolutions + ReLU, skip connections and 1 softmax layer. Hu et al. [29] proposed the improved FCN methods for segmenting ventricles in MRI. The network consists of 16 convolutions, 2 max-pooling, 3 upsampling, and 1 softmax layer (see Fig. 8).

Fig. 8.

3D-ASM [29].

Kong et al. [39] proposed the combination of the global attention mechanism (GAM) with the local attention mechanism (LAM) [96] for the segmentation of brain tumors. The GAM segments the discriminate learning with a weight-allocation loss function while the LAM reverses the effective feature guidance associated with the united loss function at different levels. The network consists of 4 LAM, 27 convolutions, 4 max-pooling layers, 1 sigmoid, residual network, 2 concatenations, and 4 skip connections. Li et al. [45] used the multi-scale modality dropout learning network to localize the intervertebral disc in MRI. The network involves three pathways which integrate spatial information into multiple-scale inputs. The first pathway consists of 7 convolutions, 1 max-pooling, and 1 deconvolution. The second pathway consists of 5 convolutions, while the third consists of 4 convolutions, 1 max-pooling, and 1 deconvolution. A merge operation concatenates the three pathways, then a final convolution and softmax are used to produce the final segmentation (See Fig. 9).

Fig. 9.

Multi-path network [45].

Dolz et al. [66] proposed an FCN that consists of 25 convolutions and Fc layers to segment the brain in MRI. The network is an improvement to the baseline FCN [[51], [67]].

1.5.3. Summary of deep CNN and fully CNN methods

A significant challenge associated with CNNs is the vanishing gradient issue. Nonetheless, certain researchers have harnessed the 3D learning and interference approach to mitigate this concern [61]. Employing techniques like cascaded networks, attention pooling, and the Siamese network has also proven to be effective in addressing the vanishing gradient challenge. In this context, the widely recognized DeepMedic [12] framework has been adopted for brain segmentation. This network, in conjunction with other methodologies, enhances segmentation effectiveness and elevates accuracy rates. While the FCN and DCNN architectures may not be inherently tailored for medical image segmentation, they nonetheless yield favorable outcomes when applied to this domain. The summary of methods in this section is available in Table 2.

Table 2.

Summary of DCNN and FCN method.

CNN Type	Ref	Common image type	Popular Abnormalities/Organ	Coverage publication year	Average (DICE)	Average (Hausdorff)
DCNN	See Fig. 2	Metastases, Tumor, Brain, Liver and Heart	MRI	2018, 2020, 2021	0.91	2.34
FCN	See Fig. 2	Tumor, Ventricles, Brain	MRI	2018, 2021, 2022	0.93	2.62

1.6. Other methods

There are other 3D CNN methods which are not part of the categories mentioned earlier, these methods are used for segmenting abnormalities and organs in medical images. The research by Ye et al. [20] proposed a parallel pathway dense neural network, a variant of the dense network. The network consists of 3 layers; input, convolution, and fully connected layer. The inputs are downsampled into two parts and the outputs of the parts are weighted and combined with the context pathway fusion method. Zhang et al. [37] combined the active learning, attention mechanism, and deep supervision mode to segment the brain in MRI. The network consists of 5 Resblocks, 2 poolings, 3 upsampling, and 2 hybrid attention mechanisms. Each Resblock consists of 2 convolutions, and 2 BN, ReLU layers. The hybrid attention mechanism consists of a channel attention module and a spatial attention module. It has 1 Convolution, BN, ReLU layer, 1 global average pooling, 1 convolution, ReLU, and 1 convolution, Sigmoid. Two concatenation modules are used to connect the layers with the output phase.

Chen et al. [42] proposed the voxelwise residual network (voxResNet) for segmentation of the brain in MRI. The network consists of convolution layers, 4 BN, ReLU layers, and 6 VoxRes layers. The network also includes 4 deconvolution layers, 4 classification layers, and a single fusion layer. Each voxRes module consists of 2 convolutions, 2 BN, ReLU, and 1 concatenation layer. Karayegen & Aksahin [47] proposed the combination of a simplified deep learning network with the semantic network for the segmentation of brain tumors in MRI. The network is composed of 2 convolutions + BN + ReLU, 2 max-pooling, 2 upsampling, 2 transpose convolution + ReLU, and a softmax layer. Semantic segmentation uses the neural network while deep learning uses upsampling and downsampling procedures. Zanjani et al. [52] proposed the masked new computation network (Mask-MCNet) for the segmentation of teeth in intra-oral scans. The network consists of 1) Backbone network 2) Monte Carlo ConvNet [97], 3) Detection Network 4) Localization network and 5) Mask generation network. The network consists of 11 X_convolutions, 9 fully connected layers, 13 BN, 4 Monte Carlo convolution layers, and 3 loss functions. Wang et al. [69] used the 3D P-Net [70] for the segmentation of multiple organs in MRI. The network contained 6 blocks, made up of 18 convolutions, 2 softmax layers, and 1 concatenation layer (see Fig. 10).

Fig. 10.

3D P-net [69].

Wang et al. (2023) Introduced the Tensorized Transformer Network designed for the segmentation of 3D medical images. This network encompasses three pivotal functions: 1) A multi-scale transformer, featuring layers-fusion, introduced to effectively capture intricate contextual interaction information. 2) The Cross Shared Attention (CSA) module, founded on pHash similarity fusion (pSF), is thoughtfully crafted to extract comprehensive multi-variate dependency features on a global scale. 3) The Tensorized Self-Attention (TSA) module is put forth to address the challenge of handling a substantial number of parameters. Furthermore, this module can be seamlessly integrated into other models. Li et al. [107] harnessed deep learning for the automated segmentation and detection of prostate cancer in MRI scans, introducing a novel framework named "3D Mask R–CNN." This innovative model integrates a 3D convolutional neural network within its Region Proposal Network (RPN) component to effectively extract intricate features from the MRI images. These extracted features play a pivotal role in generating region proposals, a critical step in object detection. The core of the methodology involves leveraging the 3D convolutional neural network within the RPN to extract informative features from the MRI data. These features are subsequently utilized to construct anchor boxes of varying scales and aspect ratios. This anchor box strategy enhances the accuracy of region proposal generation within the RPN, as it aligns the proposals more closely with the characteristics of the extracted features.

Guo et al. [108] present the causal knowledge fusion (CKF) framework for cardiac image segmentation. Initially, the CKF framework employs causal intervention techniques to acquire the anatomical factor while discarding the modality factor. Subsequently, the CKF framework introduces a 3D hierarchical attention mechanism to extract multi-scale information from 3D cardiac images. The summary of methods in this section is available in Table 3.

Table 3.

Summary of Other methods.

Author	Ref	Year	Image type	CNN Type	Abnormalities/Organ	Accuracies (DICE)
Ye et al.	[20]	2021	MRI	DenseNet	Tumor	0.88
Zhang et al.	[37]	2021	MRI	Quality-driven deep learning	Brain	0.98
Chen et al.	[42]	2018	MRI	Voxel residual network	Brain	0.91
Karayegen & Aksahin	[47]	2021	MRI	Semantic segmentation and deep learning	Tumor	0.99
Zanjani et al.	[52]	2021	Intraoral scan	Mask-MCNet	Tooth	0.99
Wang et al.	[69]	2018	MRI	P-Net	Multiple	0.90
Wang et al.	[105]	2023	MRI	Self-attention network	Heart, Brain, Uterine	0.93
Li et al.	[107]	2023	MRI	3D Mask R–CNN	Prostate	0.95
Guo et al.	[108]	2023	MRI, CT	Attention mechanism	Cardiac	0.97
Cao et al.	[111]	2023	MRI	Shuttle Attention mechanism	Brain	0.87

2. Datasets and image preprocessing

Among the scrutinized papers, the BRAT [[99], [100], [101]] and LITs2017 [102] datasets emerged as the most commonly employed datasets. Their widespread utilization can be attributed to their online availability and ease of access. Intriguingly, we noted that these same datasets found application in the segmentation of both abnormalities and organs within medical images. A condensed compilation of image preprocessing methodologies for the segmentation of abnormalities and organs is provided in Table 4.

Table 4.

Image preprocessing.

Reference	Method	Notes
[10,14,20,21,23,28,30,52,59]	Interpolation and rotation	This method interpolates, flips, and rotates the images
[11,19,28]	Anonymization, reorientation, and resampling	Methodic resampling and rescaling.
[17,24,35,42,47]	Histogram matching, equalization, and smoothing	Accurate identification of the region of interest size and center of the image patch
[27,44]	Bias field correction and inhomogeneity method	Reduces image intensity in-homogeneity by estimating high-frequency content.
[35,40,48,49,55,76]	Normalization, cropping, blurring, registration, and slicing	Shape and edge smoothing of images.
[45,65]	Mean subtraction and intensity clipping	Subtraction of the input data by the mean of intensity of the whole data

Interpolation, rotation, equalization, normalization, and filtering are among the preprocessing techniques that have exerted a notable influence on the successful segmentation of abnormalities and organs in medical images. Research findings underscore that an optimal dataset should possess relevance, usability, and high quality [77]. Consequently, it is our belief that the combination of suitable datasets and proficient preprocessing techniques holds the potential to enhance segmentation accuracy.

3. Summary of data

A total of 63 pertinent research papers have been investigated within this review. Widely adopted metrics such as the Jaccard similarity coefficient, Dice measure, F1 scores, mean surface distance, Hausdorff distance, and those pertaining to the Medical Image Computing and Computer-Assisted Intervention (MICCAI) [8,14,60] serve as prevalent performance indicators for the segmentation of medical images. Prominent organs subjected to segmentation include the brain, prostate, teeth, and ventricle (refer to Fig. 2). As for abnormalities, tumors, lesions, and metastases account for the most frequently segmented anomalies. It is intriguing to observe that the focal points of researchers' segmentation endeavors revolve around the brain and brain tumors. A comprehensive overview concerning the distribution of publications across different years is depicted in Fig. 11.

Fig. 11.

Number of papers published yearly.

We restricted our consideration to papers released within the preceding six years. The most substantial volume of publications was registered in the years 2020 and 2021. Across the spectrum, 42 papers were dedicated to encoder/decoder segmentation, while 21 papers were directed towards other segmentation. On the whole, the average dataset proportions employed for testing and training purposes were 175 and 75, respectively. Notably, the requisition of data for Encoder/Decoder segmentation surpassed that necessary for other segmentation. The information concerning the quantity of papers employed for abnormalities and organ segmentation can be found in Fig. 12. The figure illustrates that there is a greater count of papers utilized for organ segmentation compared to those employed for abnormalities segmentation. To be precise, there are 19 papers dedicated to abnormalities segmentation, whereas organ segmentation is represented by 44 papers. In the context of medical images, "abnormalities" pertain to images that deviate from a healthy depiction. Such images may reveal structural impairments, encompassing injuries, lesions, inflammation, swellings, bleeding, and tumors. Body regions under scrutiny encompass the chest, abdomen, pelvis, head, and neck. Organs residing within these regions include the heart, liver, biliary tract, kidneys, spleen, bowels, pancreas, adrenal glands, uterus, ovaries, and prostate gland (see Ref. [112] for details).

Fig. 12.

Number of papers published relating to organ and abnormalities.

4. Conclusions and future directions

Within the ambit of this investigation, we meticulously reviewed a total of 60 articles which harnessed the capabilities of 3D CNN methods for the precise segmentation of both organs and abnormalities within medical images. This classification process systematically categorized the articles into distinct groups, namely encoder/decoder, DCNN, FCN, and other methodologies. The commendable performance demonstrated by the 3D CNN frameworks has underscored their capacity to yield noteworthy segmentation and classification accuracies. Nonetheless, it's noteworthy that these networks remain relatively less utilized in comparison to their 2D CNN counterparts. Operating on the foundational tenets of the CNN framework, the 3D CNN harnesses components such as max-pooling, convolutional kernels, upsampling, and concatenation layers to achieve its segmentation objectives with efficacy.

4.1. Problems of the existing system, and future directions

This study aims to comprehensively review methods pertaining to 3D CNN segmentation of organs and abnormalities in medical images. The performance of various CNN algorithms was systematically evaluated, revealing a spectrum of outcomes ranging from excellent to very good accuracies. Medical images have become an increasingly valuable adjunct in detecting abnormalities, with the current progress of 3D CNN for diagnosis marking significant advancements. CNN algorithms have furnished clinicians with invaluable insights that aid in drawing conclusions regarding specific pathologies. Multiple 3D CNN techniques have effectively contributed to disease diagnosis and the prediction of ailments in patients.

However, certain limitations must be addressed to ensure effective abnormalities and organ segmentation. The first of these constraints is the paucity of available public volumetric databases for comprehensive analysis. The integration of traditional preprocessing methods with CNN models poses another challenge. Furthermore, the limited availability of annotated ground truth images represents an obstacle. The underutilization of 3D CNN frameworks for segmenting dental models stands as another limitation. Additionally, there's a need to develop a comprehensive 3D CNN capable of simultaneously segmenting multiple organs within a specific body region. Lastly, there's a concern regarding over-reliance on specific 3D CNN algorithms for medical image segmentation. These limitations are accompanied by potential solutions and future directions.

4.1.1. Insufficient volumetric databases and data availability

The accessibility of publicly available datasets for medical image segmentation remains limited. The utilization and distribution of datasets are often skewed, with a noticeable balance between private and publicly accessible datasets. This observation implies that datasets are often confined to specific institutions such as universities or hospitals, possibly due to privacy policies that safeguard patient data from experimental use. A potential resolution involves revising patient data protection rights to permit the use of anonymized patient data without compromising sensitive information. Another issue encountered is the inadequacy of data within the datasets themselves. Many datasets are found to have constrained quantities of data for both testing and training. It's postulated that datasets containing less than 400 volumetric data points are deemed insufficient. As a recommended course of action, analysts in medical image processing are encouraged to allocate time and resources for collecting data directly from clinicians, potentially involving the recruitment and training of graduate students under the guidance of principal investigators and mentors.

4.1.2. Effective integration of preprocessing and CNN models

Preprocessing methods are a vital component of medical image segmentation pipelines, enhancing both numerical outcomes and visual evaluations. However, effectively harmonizing these preprocessing techniques with 3D CNN models presents a notable challenge. To surmount this challenge, researchers are urged to develop efficient artificial intelligent preprocessing models that seamlessly integrate with CNN segmentation models. The shared domain between the preprocessing and segmentation models could facilitate this integration, streamlining the process.

4.1.3. Scarcity of annotated ground truth images

An integral element of medical image processing lies in the provision of meticulously annotated ground truth data. This serves as the benchmark for learning in CNN models, aiding their training. The production of such ground truth data is labor-intensive, necessitating the involvement of at least three experienced clinicians. The collective input of these clinicians results in a consensus-based ground truth. However, it has been observed that several datasets lack ground truth images due to the challenges involved in obtaining them [103]. To mitigate this issue, researchers are advised to employ a combination of different algorithms with minimal human intervention to generate ground truth markings. Alternatively, adopting deep learning mock-up strategies, such as weakly supervised methods, can be employed for unlabeled data.

4.1.4. Limited usage of 3D CNN frameworks for certain image modalities

Through this review, it becomes apparent that MRI and ultrasound modalities dominate the segmentation landscape. This could be attributed to the research focus on specific organs within the body. MRI demonstrates efficacy in brain tumor detection, while ultrasound excels in detecting breast tumors. Encouragement is extended to medical image analysts to explore the segmentation of organs and abnormalities across diverse modalities, potentially leading to improved accuracies.

4.1.5. Multi-organ and multi-abnormality segmentation using 3D CNNs

While effective segmentation of individual organs within specific body regions is established, further advancements could be realized by extending this capability to simultaneously segment multiple organs within the same region. To address this, the utilization of deep 3D CNN networks, such as the Siamese network, is recommended. Such networks are equipped to accommodate the distinctive characteristics of different organs within a given body region.

4.1.6. Diversification of 3D CNN methods for segmentation

The analysis of reviewed papers indicates a prevalence of 3D CNN methods built upon encoder/decoder networks (e.g., U-Net, attention, and adversarial networks). The performance of these networks, especially U-Net, in medical image tasks could account for this prevalence. While the utilization of U-Net is merited, a single-network reliance is impractical. Researchers are advised to explore a broader spectrum of CNN architectures, employing effective configurations to rival the U-Net model. Furthermore, the DCNN and FCN models also demonstrate the potential for accurate results, urging researchers to develop sophisticated models utilizing these frameworks.

4.1.7. Real-life performance of 3D CNN methods

The segmentation of organs and abnormalities using diverse 3D CNN methodologies is a significant endeavor. However, it's equally crucial to evaluate these algorithms in real-world scenarios. There remains uncertainty over whether certain algorithms will perform optimally when applied to practical applications. To mitigate this concern, analysts and computer vision engineers are encouraged to foster close collaborations with clinicians, ensuring the gradual integration of CNN algorithms into real-life applications.

4.1.8. Standardization of measuring metrics

The adoption of diverse evaluation metrics for quantitative result reporting is a prevalent practice among researchers. This diversity in metrics can lead to varied assessments, posing challenges in comparative analyses. While this review consolidates two common accuracies from the studies, further steps can be taken to minimize metric proliferation. A suggested approach is to designate MICCAI metrics as the standard for organ segmentation in CT images (as outlined in Section 4).

4.1.9. Addressing high computation space demands of CNNs

It's widely recognized that CNN algorithms demand significant computational resources. The availability of adequate computational space plays a pivotal role in achieving higher segmentation accuracies. To alleviate this concern, the proposal for creating additional cloud computing platforms (like Collab) with substantial computational capacity is made. Such platforms could facilitate researchers in running their applications without incurring substantial costs.

4.2. Conclusion

This study delved into the exploration of 3D CNN methodologies for the segmentation of organs within medical images. Various approaches to segmenting abnormalities within medical images were thoroughly examined. Insights into forthcoming trends have been projected, accompanied by a comprehensive evaluation of the merits and drawbacks of multiple 3D CNN techniques.

In summation, 3D CNNs exhibit promising capabilities in effectively segmenting both organs and abnormalities within medical images. The accrued advantages from employing 3D algorithms hold the potential for widespread adoption in the future. Significantly, collaborative efforts are essential to forge the development of novel and potent algorithms catering to the segmentation needs of abnormalities and organs within medical images.

Data availability statement

The data that support the findings of this study are available on request from the corresponding author.

CRediT authorship contribution statement

Ademola E. Ilesanmi: Writing – review & editing, Writing – original draft, Visualization, Validation, Methodology, Formal analysis, Data curation, Conceptualization. Taiwo O. Ilesanmi: Writing – original draft, Data curation. Babatunde O. Ajayi: Writing – original draft, Visualization, Formal analysis.

Declaration of competing interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.