Advanced 3D retinal lesion segmentation using channel-spatial attention-guided multi-scale feature aggregation

Abstract

Diabetic macular edema (DME) and age-related macular degeneration (AMD) have emerged as leading causes of vision impairment worldwide; optical coherence tomography (OCT) has proven to be a crucial diagnostic tool for these diseases, for its rapid, non-invasive high-resolution imaging of retinal structures. Further accurate assessment of lesions in retinal OCT images plays a pivotal role in the early diagnosis of DME and AMD. However, current diagnosing of AMD and DME is restricted to utilizations of OCT two-dimensional images, for crucial three-dimensional (3D) lesion information inherent in OCT 3D images cannot be effectively extracted and utilized without appropriate methods. Here, we proposed an innovative deep-learning network characterized by fusing multi-scale feature extraction-aggregation and channel-spatial joint attention for high-accuracy 3D lesion segmentations of DME and AMD. Extensive experiment results demonstrated that our proposed method has commendable 3D segmentation performances and robust generalization capabilities, probably helping to understand DME and AMD diseases better and providing great convenience for clinical diagnosis and treatment.

1. Introduction

Diabetic macular edema (DME), a leading cause of vision loss in the working-age population of most developed countries [1–3], is a condition characterized by thickening and swelling of the macula caused by microvascular leakage [4,5]. Age-related macular degeneration (AMD) is characterized by various pathological changes, including the deposition of metabolites between the retinal pigment epithelium (RPE) and Bruch’s membrane (BM) [6], resulting in a macular lesion with symptoms such as blurred or loss of central vision [7]. Intraretinal fluid (IRF) is considered one of the biomarkers for DME [8], while drusen are widely accepted as contributing factors to the pathogenesis of AMD [9]. To ensure effective management of retinal disease and prevent its progression to blindness, it is crucial to closely monitor IRF and drusen, which are recognized as critical clinical indicators and significant risk factors for the development of DME and AMD [10].

Optical coherence tomography (OCT) is a non-invasive three-dimensional (3D) imaging technology [11–13]and plays an important role in the diagnosis and monitoring of retinal diseases relying on its ability to provide high-resolution cross-sectional images for the retinal structures. With B-scans of OCT contained in OCT 3D scanning volume, it can accurately visualize the lesions of AMD and DME in the retina. Precise segmentation of corresponding lesion OCT images further holds substantial clinical significance, but manual segmentation by ophthalmologists suffers from inefficiency and subjectivity. Consequently, automatic lesion segmentation of retinal OCT images is crucial and progressively sought after. However, existing segmentations for multiple two-dimensional (2D) B-scans are inadequate in diagnosing definitive the development of retinal lesions. Previous automatic diagnosis techniques did not efficiently harness the 3D imaging characteristics of OCT to provide comprehensive 3D lesion information.

Over the past few years, deep learning has been successfully applied to the task of medical image processing [14–18], including the fluid region segmentation problem [19–21], Unet [22], Seg-Net [23], Deeplabv3+ [24], and Unet++ [25] are widely recognized and notable deep learning architectures developed for image segmentation and used in medical applications. Inspired by these networks, numerous efficient architectures have been proposed and show superior performance in retinal lesion 2D segmentation [26,27]. Particularly, A. Roy et al. proposed ReLayNet [19], which utilized a contracting path of convolution blocks to learn a hierarchy of contextual features, followed by an expansive path of convolutional blocks for semantic segmentation. X. Yu et al. [27] introduced a novel loss-balanced joint-loss function to handle imbalance in the roles of different branches. G. Xing et al. [28] adopted a novel curvature regularization term to incorporate shape-prior information for OCT fluid segmentation. Suchetha et al. [29] presented a deep learning-based predictive algorithm, which applied Region Convolutional Neural Network (R-CNN) and faster R-CNN to improve the accuracy of AMD lesion segmentation. M. Wang et al. [10] proposed a novel multi-scale transformer global attention network for drusen segmentation in retinal OCT images. W. Liu et al. [30] proposed an attention-based method for DME segmentation, and they integrated multi-scale inputs, multi-side-outputs, and attention methods into Unet++ for better performance. In a study by X. Liu et al., [31] an attention-based Unet model was developed for intraretinal cystoid fluid (IRC) segmentation in DME images with an attention structure that automatically considers suspected regions. R. Rasti et al. [32] proposed the RetiFluidNet model, a multi-attention and self-adaptive deep convolutional network based on an end-to-end training scheme for retinal OCT fluid segmentation, which has demonstrated high performance in fluid segmentation. Moradi et al. [33] introduced the ensemble mechanism to improve non-advanced AMD classification. M. Li et al. [34] employed multi-scale and cross-channel feature extraction and channel attention to segment wet AMD lesions.

Although these newly developed methods have achieved remarkable improvements in robustness and accuracy in 2D segmentations, the majority of them primarily consider the properties of OCT 2D images, therefore such approaches are limited in their ability to account for the complex and various retinal layers and lesions. Above all, as no 3D context is taken into account, 2D results suffer from a lack of 3D spatial consistency which leads to edge discontinuity in the long-axis direction, and false positive predictions at a slice containing other lesions similar to target lesions. Browsing through all 2D images within OCT 3D volume takes time, while reviewing only a few may pose the risk of missing lesions. Given the rapid advancement of technology and the increasing emphasis on precision, 3D results that can show more comprehensive and intuitive information associated with the lesions, are more favored by specialists. Recently, several 3D segmentation methods have been proposed for OCT image segmentation. For instance, Y. He et al. [35] introduced a Unet based on LSTM, which utilizes longitudinal information for OCT layer segmentation. S. Mukherjee et al. [36] proposed a 3D deep convolutional regression network capable of preserving semantic information through the utilization of spatial context, continuity, and anatomical relationships, thereby enabling retinal layer segmentation. Additionally, S. Mukherjee et al. [37] employed a 3D Unet to extract high-dimensional continuous features from OCT volume data, achieving promising performance in retinal layer segmentation tasks for AMD patients. Based on these advancements, accurate and automated 3D segmentation for IRF and drusen is critical and increasingly demanding.

To overcome this limitation, we proposed a novel multicategory, multi-scale, and attention-based network structure that can synergistically explore DME and AMD lesions from OCT 3D volume images and obtain accurate 3D morphology of IRF and drusen. Specifically, a channel and spatial joint attention (CSJA) module is proposed to adaptively identify and select the most discriminative semantic and spatial information to segment lesions from surrounding speckle noise. Further, we proposed an efficient multi-scale feature extraction (MsFE) module to exploit multi-scale features by employing multiple parallel filters with different rates. Meanwhile, a novel multi-scale feature aggregation (MsFA) module is designed to guide the model to fuse adjacent features in the encoding and decoding stages and improve 3D segmentation performance consequently.

The overall workflow of this paper is as follows: Section 2 provides detailed information about the dataset utilized in this study, as well as an exposition of the methodologies proposed and the training strategies employed. Section 3 outlines the experimental procedure, encompassing the data augmentation methods, the configuration environment in which the experiments were conducted, and the evaluation metrics utilized. The results of our proposed model are presented in section 4. In Section 5, we compare the proposed method with existing methods through comparative experiments. We also conduct ablation experiments to evaluate the effectiveness of our model. Finally, we discuss the potential clinical applications of our method. Section 6 is the conclusion related to the findings in this study.

2. Methods

2.1. Datasets

This study was approved by the Institutional Review Board (IRB) of Sichuan Provincial People’s Hospital (IRB-2022-258). The datasets comprising OCT volumes of patients with DME and AMD were obtained using the Swept Source OCT system (BM-400 K BMizar, TowardPi Medical Technology, Beijing, China). This system utilizes a vertically sweeping laser with a wavelength of 1060 nm and a scanning speed of 400,000 A-scans per second. Additionally, TowardPi’s OCT angiographic scanning mode can generate angiographic images covering a 6 mm × 6 mm area centered on the macula. Within this 6 mm × 6 mm scan range, there are 512 A-scans and 512 B-scans.

Considering that the limited training data annotated with a quality golden standard is also a significant factor restricting the development of 3D segmentation of DME and AMD, we constructed a new 3D dataset containing 120 cases (61,440 slices) of DME and AMD volumes and corresponding manual pixel-level annotation from specialized ophthalmologists. Firstly, three ophthalmologists manually annotated the IRF and drusen regions frame by frame in the dataset using ITK-SNAP software. After the annotations were completed, a senior ophthalmologist reviewed and rated them, and the final annotations were selected based on the highest ratings. All participants underwent a comprehensive ocular examination including refraction and best-corrected visual acuity, non-contact intraocular pressure (IOP), ocular axis, slit lamp, wide-angle fundus imaging, and OCT. This study was approved by the Ethics Committee of Sichuan Provincial People’s Hospital and adhered to the Declaration of Helsinki. All participants provided written informed consent after the nature and possible consequences of the study were explained. More detailed information can be seen in Table 1. In the study, we used a fixed split on both the DME and AMD datasets, with 45 samples for training and the remaining 15 samples for validation. And since data that simultaneously exhibits both IRF and drusen is rare, there is only one set in the dataset used as the validation set.

Table 1. Information of the employed datasets.

Dataset	DME	AMD	DME&AMD
Scanner device	BM-400 K Bmizar	BM-400 K Bmizar	BM-400 K Bmizar
Scan range	6 mm × 6 mm	6 mm × 6 mm	6 mm × 6 mm
Total	60 volumes × 512 slice	60 volumes × 512 slice	1 volumes × 512 slice
Size	512 × 1044	512 × 1044	512 × 1044
Resize	512 × 512	512 × 512	512 × 512
Dimension	3D	3D	3D
Training / Test	90/10	90/10	-

As shown in Fig. 1, (a1) represents the raw 3D data of the case related to DME, along with the corresponding 3D visualization of the labels. (a2) demonstrates the B-scan images that correspond to the blue slice of the raw data mentioned in (a1). (a3) exhibits the label images corresponding to the green dotted line observed in the 3D visualization of the labels. Additionally, (b) and (c) display cases of AMD, as well as a case where both DME and AMD lesions are present.

Fig. 1.

3D visualization and B-scans of examples in datasets. The first row shows the 3D OCT original image and the 3D lesion annotation, while the second row displays the blue cross-section from the OCT image and the corresponding 2D annotation with green dashed lines.

2.2. Overview architecture

To perform proper and robust 3D segmentation of DME and AMD, we proposed a novel network architecture, which adopts an encoder-decoder U-shaped network as the basic framework and integrates three core modules, as illustrated in Fig. 2, our main contributions are the synergistic effect among the three fused modules: CSJA, MsFE, and MsFA. The CSJA module is adopted in skip-connection to effectively extract multi-scale context information and enhance the proposed network’s ability to segment diverse retinal lesions. The MsFE module is proposed to adaptively identify and choose useful semantic information and spatial information for retinal multi-sort segmentation. In addition, the MsFA module is used to learn more semantic representations to refine the DME and AMD segmentation map.

Fig. 2.

The overall architecture of the proposed network is as follows: (a) shows the complete architecture of our model, which is capable of automatically performing 3D segmentation of OCT retinal lesions, and (b) illustrates the CSJA module, which leverages the attention mechanism to capture more accurate target information.

To dynamically determine the pertinent semantic and spatial information, we introduced the CSJA module (refer to Fig. 2(b)), aimed at facilitating the effective fusion of adjacent hierarchical features to more efficiently capture underlying semantic information. Initially, the features extracted from the $E_i$ layers, possessing identical dimensions, undergo a channel concatenation operation. Subsequently, to enhance the utilization of the most pertinent feature channels, channel attention is introduced to mitigate interference from irrelevant noise. Global max-pooling and global average pooling operations are employed to acquire the global information of each channel, and their outcomes are integrated. The resulting $M_c$ is processed in the spatial channel to obtain $M_s$ . Finally, we add the reweighted low-level features to the high-level features to yield the final result M.

$$M_c=f_{\sigma }\{f_{MLP}(f_{gap}(F))+f_{MLP}(f_{gmp}(F))\},$$ (1)

$$M_s=f_{\sigma }\{f_{7\times 7}\{\mathbb{C}(f_{avg}(M_c)+f_{max}(M_c))\}\},$$ (2)

$$M=M_c\otimes M_s+F,$$ (3)

where $f_{\sigma }$ refers to the sigmoid activation used for deriving the attention map at the present scale, $f_{MLP}$ denotes the Multi-Layer Perceptron (MLP) operator. Moreover, $f_{gap}$ represents the global max-pooling operation, $f_{gmp}$ stands for the global average pooling operation, and $f_{7\times 7}$ signifies the convolution with the 7 × 7 kernel. $\mathbb{C}$ represents the operation of concatenation, $f_{avg}$ and $f_{max}$ denote extract the average and maximum value in the channel, respectively. $\otimes$ indicates the multiplication. Additionally, F denotes the initial input feature.

At the bottom of our network, the MsFE module is devised to extract concealed background information and consolidate features across multiple scales. As shown in Fig. 3, the MSFE module contains multi-scale feature extraction (see Fig. 3. (a)) and adaptive feature selection (see Fig. 3. (b)). The multi-scale feature extraction component generates a more comprehensive feature map to effectively capture scale variations in images, enabling the model to better handle feature at different scales. First, the feature map generated by $E_5$ is equally divided into four parallel feature groups, each of which is subjected to a dilated convolution operation (dilated rate = 1, 6, 12 and 18, respectively) to derive a more abundant feature map with various receptive fields. Thus, features at different scales in DME and AMD lesions can be adequately perceived by this module. The operations for extracting parallel features are as follows:

$$f_i^{\mathrm{\prime }}=f_c^{d_i}(f_i),$$ (4)

$$F=f_1^{\mathrm{\prime }}\oplus f_2^{\mathrm{\prime }}\oplus f_3^{\mathrm{\prime }}\oplus f_4^{\mathrm{\prime }},$$ (5)

where $f_c^{d_i}$ represents the dilated convolution operation with dilated rate of $d_i$ , $\oplus$ denotes the element-wise addition. Finally, these multi-scale feature information F are merged by the element-level addition and delivered to the adaptive feature selection component to further obtain global contextual features and suppress the interference of irrelevant local features adaptively. After a layer normalized, the input feature $F\in {\mathbb{R}}^{H\times W\times C}$ (where H, W and C indicate the height, width and channel of F respectively) first produces query (Q), key (K), and value (V) projections, enriched with context. It is achieved by applying 1 × 1 convolutions to aggregate pixel-wise cross-channel context followed by 3 × 3 depth-wise convolution to encode channel-wise spatial features, yielding $Q=D_{1\times 1}^Q{D}_{3\times 3}^{Q}F$ , $K=D_{1\times 1}^K{D}_{3\times 3}^{K}F$ , $V=D_{1\times 1}^V{D}_{3\times 3}^{V}F$ . where $D_{1\times 1}^{(\cdot )}$ is the 1 × 1 point-wise convolution and $D_{3\times 3}^{(\cdot )}$ is the 3 × 3 depth-wise convolution. Next, we reshape query and key projections such that their dot-product interaction generates a transposed-attention map A of size C × C, instead of the huge regular attention map of size HW × HW. In addition, a residual connection is developed to aggregate the multi-scale feature maps. Overall, the adaptive feature selection process is defined as:

$$\hat{F}=D_{3\times 3}^{Q}Attention(Q,K,V)+F,$$ (6)

$$Attention(Q,K,V)=V\cdot Softmax(K\cdot Q/\alpha ),$$ (7)

Fig. 3.

Multi-scale Feature extraction Module. The MsFA module is designed to extract feature at multiple scales, enabling it to preserve more detailed global information. First, the input feature map is divided into four parallel branches. Then, dilated convolution operations with different dilation rates (1, 6, 12, and 18) are applied to these feature maps to extract a richer feature map with various receptive fields.

Here, $\alpha$ is a learnable scaling parameter to control the magnitude of the dot product of K and Q before applying the softmax function.

Several previous studies have shown that aggregating multi-scale features can effectively improve network performance [38,39]. Here we propose the MsFA module (see Fig. 4) to obtain more semantic details and refine the segmentation results. First, the input feature channel generated by the decoding layers is compressed to 8 using convolution.

$$F_i=f_c(D_i),i\in \{1,2,3,4\},$$ (8)

where $f_c$ is a convolution with 1 × 1 kernel. The process involves concatenating features from neighboring scales in a top-down manner. Subsequently, an attention mechanism is employed to refine the features at various scales, resulting in a more discriminative representation of features.

$$F_4^A=f_{c\mathrm{\prime }}\{f_{\sigma }(f_c(\mathbb{C}(F_4,f_{\mu }(F_5))))\otimes F_4\},$$ (9)

$$F_3^A=f_{c\mathrm{\prime }}\{f_{\sigma }(f_c(\mathbb{C}(F_3,f_{\mu }(F_4^A))))\otimes F_3\},$$ (10)

$$F_2^A=f_{c\mathrm{\prime }}\{f_{\sigma }(f_c(\mathbb{C}(F_2,f_{\mu }(F_3^A))))\otimes F_2\},$$ (11)

$$F_1^A=f_{c\mathrm{\prime }}\{f_{\sigma }(f_c(\mathbb{C}(F_1,f_{\mu }(F_2^A))))\otimes F_1\},$$ (12)

where $\otimes$ denotes the element-wise multiplication; $f_{\mu }$ represents the up-sampling operation; $f_{\sigma }$ denotes the sigmoid activation to obtain the attention map of the current scale; and $f_{c^{\mathrm{\prime }}}$ has a convolution with the 3 × 3 kernel, which is used to generate features with rich, detailed information and pass it to the next attention block until the final prediction of the category. Finally, the output of the MsFA module is obtained using the convolution of the two output channels followed by the sigmoid.

$$F_{MFA}=f_{\sigma }{f}_c(F_1^A),$$ (13)

Fig. 4.

Multi-scale Feature Aggregation Module. a convolution is applied to reduce the channel dimension of the input features from the decoding layers to 8. Next, features from adjacent scales are concatenated in a top-down fashion. This process enhances the feature representation, making it more discriminative.

Through the utilization of the MsFA module to steer the gradual aggregation of features across neighboring scales, our proposed model exhibits improved efficacy in lesion segmentation.

2.3. Loss function

The absence of distinct labels for the DME and AMD regions presents a significant challenge, as the background noise and blurred boundaries make up a substantial portion of the retinal image. This lack of labeling complicates the accurate identification and delineation of these specific regions within the images, thereby posing a significant obstacle to our analysis. The Dice loss function [40] is often used to mitigate the negative effects of the imbalance between foreground and background in a sample. As the Dice loss tends to apply to only one kind of segmentation task and may not exhibit optimal performance in multi-task segmentation. To tackle this problem, we choose the Tversky loss [41] as the loss function, which introduces two coefficients $\alpha$ and $\beta$ to better balance false negative and false positive. When both $\alpha$ and $\beta$ are equal to 0.5, the Tversky loss also becomes the Dice loss. The Tversky loss can be expressed by:

$$T(\alpha ,\beta )=\frac{\underset{i=1}{\overset{N}{\sum }}{p}_{0i}{g}_{0i}}{\underset{i=1}{\overset{N}{\sum }}{p}_{0i}{g}_{0i}+\alpha \underset{i=1}{\overset{N}{\sum }}{p}_{0i}{g}_{1i}+\beta \underset{i=1}{\overset{N}{\sum }}{p}_{1i}{g}_{0i}},$$ (14)

where the $p_{0i}$ is the probability of voxel i be a lesion and $p_{1i}$ is the probability of voxel i be a non-lesion. Also, $g_{0i}$ is 1 for a lesion voxel and 0 for a non-lesion voxel and vice versa for the $g_{1i}$ . To optimize the model, we further introduced binary cross entropy (BCE) to optimize the model, which is defined:

$$L_{BCE}=-(1-y)\log (1-x)-ylog(x),$$ (15)

where y is the annotated ground truth, x represents the predicted output by our model. Finally, we define the joint objective function of our network for DME and AMD segmentation as:

$$L_{joint}=\lambda T(\alpha ,\beta )+(1-\lambda )L_{BCE},$$ (16)

where $\lambda$ is a balance parameter that is set as 0.5 for all experiments.

3. Experiment setup

3.1. Data augmentation strategy

Our data was augmented with random rotations, shifts, rolls and resizes, resulting in seven augmented images for each image in the training datasets.

3.2. Implementation details

The proposed model was implemented in the TensorFlow v2.5.0 with NVIDIA RTX A5000 (24 G) GPUs, and Python 3.9. In all comparative experiments, we set the maximum number of epochs, batch size and initial learning rate as 50, 4 and 0.0001, respectively. The learning rate was multiplied by 0.8 every 5 epochs, and Adam serves as the optimizer. In addition, since all the B-scans contained a lot of background area without information, we cropped the original image to 512 × 512 to improve the training efficiency.

3.3. Performance measures

All comparison networks discussed in this paper were trained with consistent training process and utilized the same dataset to ensure accuracy and impartiality in the evaluation. To comprehensively evaluate the model's performance, we employed commonly used metrics to quantitatively evaluate the segmentation performance of various methods in this study. These metrics include the Dice similarity coefficient (DSC), balanced accuracy (ACC), precision, recall, specificity and intersection over union (IOU). These metrics yield values within the range of 0 to 1, with higher values indicating more accurate segmentation. Their calculation formula is as follows:

$$DSC=\frac{2|TP|}{2|TP|+|FN|+|FP|},$$ (17)

$$ACC=\frac{1}{2}\left(\frac{TP}{TP+FN}+\frac{TN}{TN+FP}\right),$$ (18)

$$Precision=\frac{TP}{TP+FP},$$ (19)

$$Recall=\frac{TP}{TP+FN},$$ (20)

$$Specificity=\frac{TN}{TN+FP},$$ (21)

$$IOU=\frac{TP}{TP++FP+FN},$$ (22)

where TP, FP, TN and FN denote true positive, false positive, true negative and false negative variables, respectively.

4. Result

In this section, we first present the 3D segmentation results for IRF and drusen using the proposed model, as shown in Fig. 5 and Fig. 6. Additionally, we display the segmentation results of our proposed network on new datasets, as illustrated in Fig. 7. Detailed files of 3D segmentation results and corresponding original 3D OCT images of Fig. 5–8 can be seen and downloaded from https://tianchi.aliyun.com/dataset/169157.

Fig. 5.

3D visualization of IRF segmentation results and some segmentation results in B-scans from freestanding individuals with DME in validation sets.

Fig. 6.

3D visualization of drusen segmentation results and some segmentation results in B-scans from freestanding individuals with AMD in validation sets.

Fig. 7.

3D segmentation results are obtained from a new dataset. The DME and AMD segmentations are shown in orange and yellow, respectively. (d), (e), and (f) display the 3D visualization of segmentation results associated with DME, AMD, and both conditions, respectively.

Fig. 8.

3D visualization of IRF segmentation results from different methods over the DME datasets. 3D images in the second and third rows correspond to the zoomed-in patches of the olive-green and aqua regions of the first row, respectively. The blue dashed circles in the second and third rows indicate that the comparison models may exhibit over-segmentation or under-segmentation.

Figure 5 and Fig. 6 show the 3D segmentation results of our network for patients with DME and AMD, respectively. Images in the first to third rows correspond to the OCT 3D images consisting of massive B-scans, the 3D segmentation results by our model and the corresponding manually annotated 3D images. As evidenced in Fig. 5 and Fig. 6, the visually intuitive results demonstrate that the 3D segmentation results of fluid lesions achieved by our proposed network closely align with the original labeled images, thereby furnishing more precise references for ophthalmologists to identify fluid lesions accurately. What's more, compared with conventional 2D segmentation methods, 3D segmentation results can provide a more intuitive and comprehensive representation of retinal disease progression. We can measure the central macular thickness (CMT) based on the volumetric visualization of fluid, which is the most widely employed among optical coherence tomography parameters to evaluate the efficacy of treatments against macular edema [42].

Our proposed model can also be utilized for segmenting images in which both DME and AMD lesions are present. Figure 7 illustrates the 3D segmentation results in OCT retinal images using our proposed model, obtained from randomly selected datasets (TowardPi, 512 × 1024 × 512), which are distinct from the training and validation sets. The IRF and drusen segmentation results are depicted in orange and yellow in the B-scan, respectively. As depicted in Fig. 7(f), our proposed network consistently delivers commendable 3D segmentation performance in the 3D images with two types of lesions. These findings substantiate the efficacy and resilience of our network for fluid segmentation in OCT retinal images, thus underscoring its potential applicability in clinical practice by furnishing precise lesion segmentation to facilitate ophthalmologists in diagnosis and alleviate their workload.

5. Discussion

In this section, a detailed comparison was conducted with a variety of classical medical image segmentation methods and the most advanced retinal fluid segmentation methods to offer a more thorough evaluation of the proposed approach’s performance. In order to demonstrate the efficacy of the proposed network, the average and standard values of DSC, precision, accuracy, recall, specificity, and IOU were utilized for a quantitative assessment of the DME and AMD 3D segmentation performance on our dataset. We tested all 15 samples in the validation set and performed statistical analysis on the obtained results. The standard deviation in the table represents the variability of the segmentation accuracy across the entire validation set.

5.1. Quantitative evaluation

Table 2 presents a quantitative metric results comparison among six metrics, our proposed model outperformed the comparative methods, with the average DSC, precision, specificity, and IOU values of 89.76%, 92.14%, 99.62%, and 78.57% respectively on DME dataset. Compared with those existing methods, the average DSC of our model outperforms Unet [22], ResUnet [43], AttentionUNet [44], Unet++ [25], CE-Net [45], RetifluidNet [32], 3D Unet [46] by 5.31%, 2.52%, 2.5%, 6.87%, 3.63%, and 3.42%, 1.69% respectively. In addition, the average precision, accuracy, specificity, and IOU of the proposed model are 3.64%, 0.04%, 0.1%, and 0.43% higher than RetifluidNet, which is state-of-art approaches for fluid segmentation on OCT images. The transformer-based model, Swin-Transformer [47], achieved an average DSC score of 85.22%, which is slightly lower than the performance of the model we proposed. The results consistently demonstrate that our method yields superior performance metrics.

Table 2. Quantitative results of different methods on the DME dataset.

Network	DSC	Precision	Accuracy	Specificity	IOU
Unet [22]	84.45 ± 6.39	75.83 ± 9.36	98.84 ± 0.56	98.92 ± 0.55	73.60 ± 9.14
Unet++ [25]	87.24 ± 5.52	84.74 ± 8.28	99.14 ± 0.42	99.45 ± 0.3	77.77 ± 8.17
AttentionUNet [44]	87.26 ± 5.21	82.47 ± 7.09	99.10 ± 0.45	99.26 ± 0.43	77.77 ± 7.85
ResUnet [43]	82.89 ± 6.75	82.87 ± 8.34	98.76 ± 0.77	99.41 ± 0.33	71.34 ± 9.77
CE-Net [45]	86.13 ± 5.99	80.54 ± 9.17	99.01 ± 0.50	99.23 ± 0.43	76.09 ± 8.62
RetiFluidNet [32]	86.34 ± 6.24	84.17 ± 7.94	99.04 ± 0.59	99.41 ± 0.36	76.47 ± 9.07
3D Unet [46]	88.07 ± 5.73	87.69 ± 5.31	98.87 ± 0.63	99.48 ± 0.25	77.55 ± 8.37
SwinTransformer [47]	85.22 ± 6.34	83.71 ± 7.54	98.93 ± 0.71	99.27 ± 0.39	74.85 ± 8.41
Ours	89.76 ± 5.13	92.14 ± 3.41	99.07 ± 0.52	99.62 ± 0.17	78.57 ± 7.94

We also conducted similar evaluations on the AMD 3D dataset, as Table 3 shows. The methods compared in this study include classical CNN-based medical image segmentation models, such as ResUnet [43], AttentionUNet [44], Unet++ [25], RetifluidNet [32], and 3D Unet [46] as well as transformer-based approaches, including Swin-Unet [48] and Swin-Transformer [47]. The average DSC, precision, accuracy, recall, and IoU of our method are 82.59%, 86.91%, 99.27%, 86.51%, and 70.36%, respectively, which convincingly demonstrates that our model is effective in drusen segmentation. In addition, the average DSC, precision, accuracy, recall, and IOU of the proposed model are 3.96%, 11%, 0.13%, 1.04% and 3.28% higher than RetifluidNet. The transformer-based models, Swin-Unet and Swin-Transformer, achieved average DSC scores of 65.76% and 71.82%, respectively, which are slightly lower than the performance of the model we proposed. On the whole, the proposed method demonstrates enhanced reliability in DME and AMD 3D segmentation across various metrics, indicating its significant superiority over other methods in terms of 3D segmentation performance.

Table 3. Quantitative results of different methods on the AMD dataset.

Network	DSC	Precision	Accuracy	Recall	IOU
Unet++ [25]	56.41 ± 23.44	82.69 ± 22.20	98.57 ± 0.71	44.66 ± 21.04	42.63 ± 20.35
AttentionUNet [44]	58.99 ± 23.25	73.10 ± 27.66	98.56 ± 0.69	50.23 ± 20.26	45.18 ± 20.22
ResUnet [43]	46.25 ± 26.21	54.70 ± 29.54	98.04 ± 0.79	40.76 ± 24.20	33.61 ± 20.61
Swin-Unet [48]	65.76 ± 18.33	83.68 ± 17.33	98.64 ± 0.77	55.72 ± 18.53	51.34 ± 17.15
RetiFluidNet [32]	78.63 ± 15.44	75.91 ± 20.59	99.14 ± 0.39	85.47 ± 8.28	67.08 ± 18.08
3D Unet [46]	77.58 ± 16.34	80.94 ± 21.63	99.16 ± 0.45	84.13 ± 9.82	68.91 ± 16.04
SwinTransformer [47]	71.82 ± 16.93	79.31 ± 18.57	98.87 ± 0.62	77.19 ± 13.20	65.45 ± 21.09
Ours	82.59 ± 11.35	86.91 ± 10.21	99.27 ± 0.31	86.51 ± 8.13	70.36 ± 15.43

5.2. Qualitative analysis

The qualitative experimental results of our method and other competitors on typical cases are illustrated in Fig. 8 and Fig. 9. We visualize results and compare qualitative results with zoomed-in patches. As shown in Fig. 8 and Fig. 9, most methods can generate the lesion of DME and AMD with high accuracy. However, in the case of fluid lesions, competing models generate fragments that do not belong to lesions of DME and AMD, particularly in the areas indicated by blue circles. Furthermore, they often fail to capture the finer details of the lesions, particularly in the regions highlighted by blue circles. Compared with other methods, our proposed method has achieved well-pleasing 3D results, as shown in the zoomed-in patches. As for the Unet++ method, it obtains more approving results than Unet but still has some fragments. AttentionUNet and RetifluidNet have achieved good results, but there is still a clear difference from the ground truth. Moreover, even speckle noise and blurry edges negatively affect the results, the proposed model achieves better segmentation performance than 3D UNet, AttentionUNet, Unet++, CE-Net, ResUnet, and Unet. Based on quantitative and qualitative analysis, we can see that the proposed model achieves a significant and consistent performance benefit, due to the synergistic effect among CSJA, MsFE, and MsFA modules by seamless integration. These results demonstrate the effectiveness and robustness of our proposed method when performing 3D segmentation of DME and AMD.

Fig. 9.

3D visualization of drusen segmentation results from different methods over the AMD datasets. 3D images in the second and third rows correspond to the zoomed-in patches of the olive-green and aqua regions of the first row, respectively. The blue dashed circles in the second and third rows indicate that the comparison models may exhibit over-segmentation or under-segmentation.

To further evaluate the performance of our model, we have conducted additional experiments using the publicly available RETOUCH dataset [49]. This dataset we used comprises 24 volumetric scans acquired with the Cirrus HD-OCT (Zeiss Meditec), each with dimensions of 512 × 1024 × 128. For preprocessing, all B-scan images across the various volumes were resized to 256 × 256 pixels to standardize the field of view. Subsequently, the dataset was then randomly split into training and validation sets in a 20:4 ratio to ensure a balanced approach for model training and evaluation, facilitating effective performance assessment. As shown in Table 4, our method achieved DSC, precision, accuracy, recall, and IOU scores of 86.47%, 83.20%, 98.65%, 88.26%, and 75.13%, respectively. These results demonstrate superior performance compared to AttentionUNet, 3D Unet, and Swin-Transformer, with our approach leading across all evaluation metrics. Figure 10 illustrates several B-scan segmentation results, where our method provides the segmentation outcomes most closely aligned with the ground truth. These findings highlight the significant potential and effectiveness of our approach in segmentation tasks.

Table 4. Quantitative results of different methods on the RETOUCH dataset.

Network	DSC	Precision	Accuracy	Recall	IOU
AttentionUNet [44]	78.71 ± 12.36	75.82 ± 16.04	97.25 ± 1.13	79.63 ± 10.39	70.82 ± 13.67
Swin-Transformer [47]	81.92 ± 10.04	76.18 ± 15.61	97.83 ± 1.06	84.19 ± 8.16	72.06 ± 12.23
3D Unet [46]	83.25 ± 9.67	81.13 ± 13.08	98.24 ± 0.91	85.27 ± 7.45	73.58 ± 10.78
Ours	86.47 ± 7.83	83.20 ± 9.16	98.65 ± 0.82	88.26 ± 5.73	75.13 ± 9.25

Fig. 10.

Comparison of different method IRF segmentation results on the RETOUCH dataset

In certain challenging scenarios, the performance of our model may be limited to some extent. For instance, OCT scans can suffer from jitter during the scanning process, which disrupts the three-dimensional continuity of the data. This disruption can significantly affect the model’s segmentation accuracy, especially when comparing frames before and after the jitter occurs. Additionally, in OCT images obtained from low-quality scans with low contrast, the continuity between different lesion regions becomes more blurred, which negatively impacts the model’s ability to accurately extract spatial information of distinct lesions. Future work will focus on developing specialized image registration and enhancement algorithms to address these issues.

5.3. Model parameters and inference times

In this section, we compare the parameter counts and inference times of different models to evaluate their computational efficiency. By analyzing both the number of parameters and the time required for predicting a single image, we aim to provide a comprehensive understanding of the trade-offs between model complexity and inference speed. As shown in Table 5, Unet, with its relatively simple structure, contains the fewest parameters. AttentionUNet achieves the fastest inference speed, with an average prediction time of 0.0299 seconds per image. However, it comes with a higher computational cost. The model we proposed requires a series of attention mechanism operations, which results in slightly higher parameter counts and inference times. In future work, optimizing for lightweight and more efficient segmentation models, with reduced computational overhead and faster inference times, will be a primary focus of our research.

Table 5. The comparison of model parameters and average inference time per image.

Network	Parameters (M)	Time (s)
Unet [22]	5.71	0.0336
Unet++ [25]	9.04	0.0363
AttentionUNet [44]	34.86	0.0299
ResUnet [43]	52.83	0.1348
CE-Net [45]	39.16	0.0807
RetifluidNet [32]	34.28	0.0797
Swin-UNet [48]	33.61	0.0931
Swin-Transformer [47]	88.40	0.1922
Ours	23.01	0.0514

5.4. Ablation experiment

Here we conducted a detailed ablation study on the proposed model’s architecture. Firstly, we took a U-shaped network consisting of an encoder with residual blocks [50] and a feature decoder as a baseline. Subsequently, we individually incorporated modules CSJA, MsFE, and MsFA into the baseline to obtain Model 1, Model 2, and Model 3. As shown in Table 6, all three models achieve commendable improvement over the base in terms of all four metrics. To further analyze the effectiveness of the proposed modules, Model 4 combined CSJA with MsFE, Model 5 combined CSJA with MsFA, and Model 6 combined MsFE with MsFA were compared. Finally, Model 7 was integrated with the three modules of CSJA, MsFE, and MsFA into the baseline. As shown in Table 6, compared with the baseline, DSC in Model 7 has improved by 5.31%, far greater than the performance of Model 1 (↑ 0.91%), Model 2 (↑ 1.44%), Model 3 (↑ 1.71%), Model 4 (↑ 2.33%), Model 5 (↑ 3.08%), and Model 6 (↑ 1.89%). The results of the ablation experiments conclusively demonstrated that the proposed modules significantly improve the 3D segmentation performance in OCT images.

Table 6. Ablation study results on the DME dataset.

Network	Module			Metrics
	CSJA	MsFE	MsFA	DSC	Precision	Accuracy	IOU
Baseline				84.45 ± 6.39	75.83 ± 9.36	98.84 ± 0.56	73.59 ± 9.14
Model 1	√			85.38 ± 6.39	80.48 ± 9.34	98.97 ± 0.54	74.97 ± 8.84
Model 2		√		85.89 ± 5.77	87.84 ± 6.09	99.0 ± 0.69	75.7 ± 8.46
Model 3			√	86.16 ± 5.93	91.72 ± 4.93	99.07 ± 0.52	76.13 ± 8.53
Model 4	√	√		86.78 ± 5.50	81.65 ± 7.60	99.07 ± 0.45	77.02 ± 7.91
Model 5	√		√	87.53 ± 4.12	85.00 ± 4.98	98.67 ± 0.55	78.06 ± 6.37
Model 6		√	√	86.34 ± 6.24	84.17 ± 7.94	99.04 ± 0.59	76.46 ± 9.07
Model 7	√	√	√	89.76 ± 5.13	92.14 ± 3.41	99.11 ± 0.53	78.56 ± 7.92

5.5. Clinical trial results

Figure 11 illustrates the progress of the DME and AMD patients over three months, during which time the patient received two intravitreal injections. These pictures were arranged by time from left to right, representing different stages of treatment. As shown in Fig. 11(a1) and 11(g1), the patient had significant abnormalities in central macular thickness (CMT) before initial treatment. At this point, the patient's CMT was measured to be 515 µm, and patients with CMT measured as ≥ 300 µm may undergo intravitreal anti-VEGF or steroid injections [51,52]. The second row of Fig. 10 shows the B-scan images corresponding to the same scanning positions. Figure 11(g1)-(i1) show the heatmap of CMT in the three stages. The green parts represent the thickness at the normal value and critical value, respectively. However, the red part indicates that the part is thicker than the normal range, and the redder the color, the thicker the thickness. One month after receiving the first treatment, the patient's CMT was reduced to 488 µm (see Fig. 11(h1)). A further diagnosis two months later showed the patient's CMT to have dropped significantly to 428 µm (see Fig. 11(i1)). Meanwhile, as shown in Fig. 11(a1)-(c1), the patient's DME lesions were significantly reduced after receiving the treatment. The above results further demonstrate the accuracy of our 3D segmentation results. The same follow-up findings are seen in patients with AMD (see Fig. 11(a2)-(i2)), The patient had significant CMT excess before treatment and this symptom improved after treatment, which is consistent with the segmentation results of our method.

Fig. 11.

Condition monitoring of a patient with DME (a1-i1) and AMD (a2-i2) at different stages of treatment. (a), (b) and (c) are the 3D visualization results of the DME lesions of this patient before, during, and after treatment, respectively. (d), (e) and (f) are the B-scan images corresponding to the same section. (g), (h) and (i) are the corresponding macular thickness heatmaps of (a), (b) and (c).

The above experiment demonstrates that the alterations in fundus conditions of patients with DME or AMD can be accurately and visually represented through the 3D segmentation results. Moreover, the 3D segmentation results align with the diagnostic findings obtained through existing clinical diagnostic methods. This approach can additionally offer comprehensive information regarding DME and AMD, thereby facilitating convenient diagnosis and monitoring of therapeutic efficacy. Furthermore, it enhances the acceptance and recognition of diagnostic outcomes among clinical patients. The 3D segmentation results of DME and AMD lesions also assist ophthalmologists in effectively communicating with patients by providing them with more visualized information.

6. Conclusion

In this paper, we proposed a novel multi-sort, attention-based, multi-scale network for high-accuracy 3D segmentation of IRF and drusen in OCT images. Our proposed network effectively leverages multi-attention mechanisms to select valuable information from multi-scale features. It fuses features at various levels to attain enhanced semantic representations for 3D segmentation. Three core modules are proposed, namely, CSJA, MsFE, and MsFA. The CSJA module is adopted in skip-connection to effectively extract multi-scale context information and enhance the proposed network’s ability to segment diverse retinal lesions. The MsFE module is proposed to adaptively identify and choose useful semantic information and spatial information for retinal multi-sort segmentation. In addition, the MsFA module is used to learn more semantic representations to refine the DME and AMD segmentation map. Compared with other state-of-the-art (SOTA) methods, the proposed method achieves promising performance in lesion segmentation on datasets acquired by the swept source OCT scanner. The proposed model achieves more accurate 3D prediction of fluid regions, and its DSC outperforms the SOTA methods by 5.31%, 2.52%, 2.5%, 6.87%, 3.63%, and 3.42% for DME, respectively. Likewise, our model demonstrates exceptional proficiency in the segmentation of AMD cases. Results demonstrate that our proposed method has commendable 3D segmentation performances and robust generalization capabilities, probably helping to understand DME and AMD diseases better and providing great convenience for clinical diagnosis and treatment.

In our future work, we plan to investigate domain adaptation algorithms to enhance both the performance and generalization of the model. This is motivated by the variability in image distributions captured by different OCT scanning devices. A model trained on a single dataset may struggle to maintain consistent high performance when applied to out-of-domain test data.

Funding

National Natural Science Foundation of China 10.13039/501100001809 ( 61905036); China Postdoctoral Science Foundation 10.13039/501100002858 ( 2021T140090, 2019M663465); Fundamental Research Funds for the Central Universities 10.13039/501100012226 ( ZYGX2021J012); Medico-Engineering Cooperation Funds from University of Electronic Science and Technology of China ( ZYGX2021YGCX019).

Disclosures

The authors declare no conflicts of interest.

Data availability

Data underlying the results presented in this paper are not publicly available at this time but may be obtained from the authors upon reasonable request.