DETECTION OF MORPHOLOGIC PATTERNS OF DIABETIC MACULAR EDEMA USING A DEEP LEARNING APPROACH BASED ON OPTICAL COHERENCE TOMOGRAPHY IMAGES

An image-based deep learning model was developed to detect different morphological patterns of diabetic macular edema based on optical coherence tomography images with high accuracy and transparency. This model could help ophthalmologists to make personalized therapeutic strategies for patients with diabetic macular edema.

Abstract

Purpose:

To develop a deep learning (DL) model to detect morphologic patterns of diabetic macular edema (DME) based on optical coherence tomography (OCT) images.

Methods:

In the training set, 12,365 OCT images were extracted from a public data set and an ophthalmic center. A total of 656 OCT images were extracted from another ophthalmic center for external validation. The presence or absence of three OCT patterns of DME, including diffused retinal thickening, cystoid macular edema, and serous retinal detachment, was labeled with 1 or 0, respectively. A DL model was trained to detect three OCT patterns of DME. The occlusion test was applied for the visualization of the DL model.

Results:

Applying 5-fold cross-validation method in internal validation, the area under the receiver operating characteristic curve for the detection of three OCT patterns (i.e., diffused retinal thickening, cystoid macular edema, and serous retinal detachment) was 0.971, 0.974, and 0.994, respectively, with an accuracy of 93.0%, 95.1%, and 98.8%, respectively, a sensitivity of 93.5%, 94.5%, and 96.7%, respectively, and a specificity of 92.3%, 95.6%, and 99.3%, respectively. In external validation, the area under the receiver operating characteristic curve was 0.970, 0.997, and 0.997, respectively, with an accuracy of 90.2%, 95.4%, and 95.9%, respectively, a sensitivity of 80.1%, 93.4%, and 94.9%, respectively, and a specificity of 97.6%, 97.2%, and 96.5%, respectively. The occlusion test showed that the DL model could successfully identify the pathologic regions most critical for detection.

Conclusion:

Our DL model demonstrated high accuracy and transparency in the detection of OCT patterns of DME. These results emphasized the potential of artificial intelligence in assisting clinical decision-making processes in patients with DME.

Diabetic macular edema (DME) is one of the major causes of vision loss among patients with diabetic retinopathy.¹ Fluorescein angiography is an important diagnostic tool for the investigation of microaneurysms and leakage in the retina. However, the precise structure of deep retinal layers cannot be observed in fluorescein angiography. Optical coherence tomography (OCT) can provide precise evaluation of retinal layers, quantification of retinal thickness and macular volume, and qualitative assessment of hyperreflective foci.² Therefore, OCT has been widely used in the diagnosis and evaluation of clinical outcomes of DME.² There are several patterns of DME on OCT examination, including diffused retinal thickening (DRT), cystoid macular edema (CME), and serous retinal detachment (SRD).^3,4 Different OCT patterns may be associated with different pathogenesis and treatment responses.^4–6

Previous studies have shown that certain OCT pattern may be accompanied by specific systemic risk factors (e.g., hypertension, hyperlipidemia, or renal dysfunction), indicating that the pathogenesis of different OCT patterns may be diverse.^7–9 The pathogenesis of DME mainly involves the breakdown of the inner and outer blood–retinal barriers (BRBs).^10–12 It has been shown that inner BRB breakdown was more responsible for DRT and CME, whereas SRD was more attributable to outer BRB breakdown.^13,14 Consistently, different OCT patterns have been shown to respond differently to treatments, suggesting that OCT pattern may be one of the key factors of determining the treatment modality of DME, along with other factors (e.g., age, lens status, and cost).^15–17

Deep learning (DL) has recently attracted tremendous interest in the field of ophthalmology.^18–20 Previously, several DL systems have been developed to detect DME with high sensitivity and specificity (>90%) based on OCT images.^18,19,21 However, no DL model to date has been developed to detect the morphologic patterns of DME based on OCT images. Such a model could help make personalized therapeutic strategies for patients with DME according to their OCT patterns. This study aimed to (1) develop a DL model to detect the OCT patterns of DME and (2) demonstrate the critical areas in OCT images highly correlated with accurate detection.

Methods

Composition of Image Data sets

The diagnosed DME patients were searched from local electronic medical record databases. Totally, 11,599 OCT images of diagnosed DME patients were extracted from a public data set (Mendeley).²² The Mendeley data set (https://data.mendeley.com/datasets/rscbjbr9sj/2) consists of the OCT images (Spectralis OCT; Heidelberg Engineering, Germany) selected from retrospective cohorts of adult patients from 4 clinical sites in California, Shanghai, and Beijing between July 1, 2013, and March 1, 2017, without exclusion criteria based on age, gender, or ethnicities.¹⁸ Meanwhile, 1,547 OCT images of 329 DME eyes at the Department of Ophthalmology, Guangdong Provincial People's Hospital between January 1, 2017, and May 1, 2019, were extracted from an OCT device (Spectralis OCT; Heidelberg Engineering, Heidelberg Eye Explorer version 6.9a, Germany). During each OCT scan, the 20° × 20° volume acquisition protocol was used to obtain a set of high-speed scans from each eye, with an axial resolution of 7 μm and a transverse resolution of 14 μm.²³ The high in-tissue resolution of the OCT device makes the accurate detection of the DME patterns possible. After excluding 781 low-quality images, 12,365 OCT images of DME eyes from the Mendeley data set and Department of Ophthalmology, Guangdong Provincial People's Hospital were included for training and cross-validating the DL model. Besides, 656 OCT images of 117 DME eyes from the Department of Ophthalmology, Zhujiang Hospital of Southern Medical University were extracted through the same process and used for external validation. All OCT sampling in Guangdong Provincial People's Hospital and Zhujiang Hospital of Southern Medical University were performed by trained technicians with more than 5-year experience in OCT imaging. Normally, the horizontal B-scan passing through the foveal reflex on the enface red-free macular image was extracted. In patients whose foveal reflex was invisible, the horizontal B-scan with the highest foveal bulge or the smallest tapering point of the inner nuclear layer was extracted. Of all OCT images included in our study, DRT, CME, and SRD were present, respectively, on 7,207, 5,284, and 2,472 images in the training set. In the external validation set, DRT, CME, and SRD were present, respectively, on 276, 301, and 256 images. The study was conducted according to the Declaration of Helsinki and was approved by the institutional review board of the Guangdong Provincial People's Hospital.

Classification of Optical Coherence Tomography Patterns of Diabetic Macular Edema

There were three OCT patterns of DME, which included the DRT as sponge-like retinal swelling of the macula with reduced intraretinal reflectivity, the CME as intraretinal cystoid spaces of low reflectivity and highly reflective septa separating cystoid-like cavities in the macular area, and the SRD as a shallow elevation of the retina and an optically clear space between the neurosensory retina and retinal pigment epithelium.^3,4 The OCT images of mixed DME pattern, defined as the mixture of two or three of these OCT patterns, were also included in our study (Figure 1). The OCT images of traction retinal detachment and posterior hyaloidal traction patterns of DME were excluded from this study to focus on identifying the OCT patterns primarily treated with intravitreal injection rather than pars plana vitrectomy.^24,25

Fig. 1.

Representative optical coherence tomography images: Top left. DRT with sponge-like retinal swelling (white arrowheads) of the macula and reduced intraretinal reflectivity. Top right. CME with intraretinal cystoid spaces (arrows) of low reflectivity and highly reflective septa separating cystoid-like cavities in the macular area. Bottom left. SRD with a shallow elevation of the retina and an optically clear space between the neurosensory layer retina and retinal pigment epithelium (*). Bottom right. Mixed DME represents the mixture of three OCT patterns (white arrowheads, arrows, and *).

Image Multilabeling

Because the mixed DME pattern is commonly seen in clinical practice, the multilabel approach is more similar to the clinical reality.^3,24 Before training, all OCT images went through a layered multilabel system consisting of multiple layers of trained graders with increasing expertise for verification and correction of image labels. The first layer of trained graders excluded images with low image quality. These images were taken of improper positioning during image acquisition or scans with strong motion artifacts, causing misalignment and blurring of sections. The second layer of graders consisted of two Chinese board-certified ophthalmologists (Q.P. and P.Z.) who independently multilabeled each image that had passed the first layer. The second layer of graders labeled the presence or absence of each OCT pattern with 1 or 0 (Table 1). Finally, the third layer graders consisting of two retinal specialists (Q.W. and B.L.) independently verified the true multilabel of all OCT images. We used the kappa statistic (κ) to evaluate the interobserver variation between these two retinal specialists. The κ was 0.762 for the detection of DRT, 0.856 for the detection of CME, and 0.825 for the detection of SRD. The results suggested excellent agreement of the two retinal specialists in OCT image labeling (κ = 0.61–0.80, substantial agreement; κ >0.80, almost perfect agreement). Moreover, if there was discordance between the two retinal specialists, arbitration was performed by a third senior retinal specialist (H.Y.) to generate the final multilabel. Thus, the effects of interobserver variation in OCT image labeling could be eliminated by the role of the third senior retinal specialist. These labels, representing the true presence of the three OCT patterns, were used as references both in the training process and assessing the performance of our DL model.

Table 1.

Multilabel of Different OCT Patterns of DME

OCT Patterns	DRT	CME	SRD	Multi-label
DRT	1	0	0	1/0/0
CME	0	1	0	0/1/0
SRD	0	0	1	0/0/1
DRT + CME	1	1	0	1/1/0
DRT + SRD	1	0	1	1/0/1
CME + SRD	0	1	1	0/1/1
DRT + CME + SRD	1	1	1	1/1/1

1 = presence; 0 = absence.

Images Preprocessing

Raw OCT images were preprocessed to normalize the input data. Saturated pixels were removed with an intensity value of 255 in the OCT images. The sparsity-based block-matching and 3D-filtering method was used to denoise and smooth the OCT images. Then, the retinal layer was cropped based on smooth pixel intensity to focus on the region of the retina that contains morphological structures with sufficient variation between different OCT patterns. Finally, the OCT images were resized into 224 × 224 pixel based on the requirements of Visual Geometry Group 16 (Department of Engineering Science, the University of Oxford).²⁶

Development of the Deep Learning Model

Figure 2 represents an abstraction of our DL model pipeline. The Visual Geometry Group 16 network was used as the benchmark DL model in our whole experiments.²⁶ Then, we replaced the last fully connected layer containing 1,000 neurons, the SoftMax layer, and the classification layer with a new fully connected layer containing 3 neurons and the regression layer. Finally, the OCT images were mapped into the OCT patterns using the regression Visual Geometry Group 16 model and adjusted the last fully connected layer to binary output classes for DRT, CME, and SRD, respectively. After training our DL model, we could obtain the probability of each OCT image with a 3-dimensional vector and divide each element in the vector to be 1 or 0 based on the threshold of 0.5 to obtain the three DME patterns information.

Fig. 2.

Abstraction of the proposed algorithmic pipeline: A deep learning model was developed to detect the three morphologic patterns of DME (i.e., DRT, CME, and SRD) using regression VGG-16 convolutional neural networks based on optical coherence tomography images. VGG-16, Visual Geometry Group 16 layers.

Model Validation

The correlation of predicted labels obtained from our DL model with the true labels was depicted as confusion matrices, which were used to calculate the accuracy, sensitivity, and specificity of the three binary classifications of OCT patterns (i.e., DRT, CME, and SRD). We also used the area under the receiver operating characteristic curve (AUC) to evaluate the accuracy of the DL model in detecting the three OCT patterns. For the internal validation, the popular 5-fold cross-validating scheme was used on the training data set that was divided into five independent portions randomly. In each run, four portions of the data set were used to train the DL model and the rest one was used to evaluate the performance. The experiments were conducted until each portion was tested. The average results after the five runs were recorded to measure the overall performance of the DL model. In addition to the cross-validating scheme, OCT images from Zhujiang Hospital of Southern Medical University were used to perform the external validation. The results were recorded to evaluate the performance of our well-constructed DL model in the training set.

Visualization Method of the Deep Learning Model

To visualize the critical areas in OCT images highly correlated with the DME patterns, the occlusion test was used to increase model transparency. Similar to the method described by Zeiler and Fergus,²⁷ a blank 51 × 51 pixel box was systematically moved across every possible position in the image, and the probabilities of the disease were recorded. The highest drop in the probability represents the part of the OCT image most critical for accurate detection. Furthermore, whether these regions identified by the occlusion test were the most clinically significant areas of pathology in DME eyes was verified by our retinal specialists.

Results

Internal Validation

In the internal validation, the binary classifier detecting the DRT pattern from the non-DRT pattern achieved a mean accuracy of 93.0%, with a mean sensitivity of 93.5%, a mean specificity of 92.3%, and a mean AUC of 0.971 (Figure 3, A and D). The binary classifier detecting the CME pattern from the non-CME pattern achieved a mean accuracy of 95.1%, with a mean sensitivity of 94.5%, a mean specificity of 95.6%, and a mean AUC of 0.974 (Figure 3, B and D). The binary classifier detecting the SRD pattern from the non-SRD pattern achieved a mean accuracy of 98.8%, with a mean sensitivity of 96.7%, a mean specificity of 99.3%, and a mean AUC of 0.994 (Figure 3, C and D).

Fig. 3.

Binary comparison evaluating the concordance between the DL model and retinal specialists in the internal validation: A. Confusion matrix of the binary classification for the DRT pattern. The row of matrix is the references verified by three independent retinal specialists. The column of matrix is the predicted labels obtained from the DL model. The mean accuracy for the detection of the DRT pattern was 93.0%, with a mean sensitivity of 93.5% and a mean specificity of 92.3%. B. Confusion matrix of the binary classification for the CME pattern. The row of matrix is the references verified by three independent retinal specialists. The column of matrix is the predicted labels obtained from the DL model. The mean accuracy for the detection of the CME pattern was 95.1%, with a mean sensitivity of 94.5% and a mean specificity of 95.6%. C. Confusion matrix of the binary classification for the SRD pattern. The row of matrix is the references verified by three independent retinal specialists. The column of matrix is the predicted labels obtained from the DL model. The mean accuracy for the detection of the SRD pattern was 98.8%, with a mean sensitivity of 96.7% and a mean specificity of 99.3%. D. ROC curve for DRT, CME, and SRD binary classifications: The mean area under the ROC curve was 0.971, 0.974, and 0.994, respectively. ROC, receiver operating characteristic.

External Validation

In the external validation, the binary classifier detecting the DRT pattern from the non-DRT pattern achieved an accuracy of 90.2%, with a sensitivity of 80.1%, a specificity of 97.6%, and an AUC of 0.970 (Figure 4, A and D). The binary classifier detecting the CME pattern from the non-CME pattern achieved an accuracy of 95.4%, with a sensitivity of 93.4%, a specificity of 97.2%, and an AUC of 0.997 (Figure 4, B and D). The binary classifier detecting the SRD pattern from the non-SRD pattern achieved an accuracy of 95.9%, with a sensitivity of 94.9%, a specificity of 96.5%, and an AUC of 0.997 (Figure 4, C and D). The retinal specialists required ∼60 minutes (range 49–65 minutes) to multilabel and assess all the OCT images (∼5.5 seconds per image) in the external validation set, which was much longer than the time required by the DL model (∼1.5 seconds per image).

Fig. 4.

Binary comparison evaluating the concordance between the DL model and retinal specialists in the external validation: A. Confusion matrix of the binary classification for the DRT pattern. The row of matrix is the references verified by three independent retinal specialists. The column of matrix is the predicted labels obtained from the DL model. The accuracy for the detection of the DRT pattern was 90.2%, with a sensitivity of 80.1% and a specificity of 97.6%. B. Confusion matrix of the binary classification for the CME pattern. The row of matrix is the references verified by three independent retinal specialists. The column of matrix is the predicted labels obtained from the DL model. The accuracy for the detection of the CME pattern was 95.4%, with a sensitivity of 93.4% and a specificity of 97.2%. C. Confusion matrix of the binary classification for the SRD pattern. The row of matrix is the references verified by three independent retinal specialists. The column of matrix is the predicted labels obtained from the DL model. The accuracy for the detection of the SRD pattern was 95.9%, with a sensitivity of 94.9% and a specificity of 96.5%. D. ROC curve for DRT, CME, and SRD binary classifications. The area under the ROC curve was 0.970, 0.997, and 0.997, respectively. ROC, receiver operating characteristic.

Occlusion Test

Results of the occlusion test showed that our DL model can successfully identify pathologic regions in the OCT images most critical for accurate detection. For example, in OCT images of the DRT pattern (Figure 5, A and B), the regions highlighted by the occlusion test were areas where sponge-like retinal swelling was the most prominent. Similarly, in OCT images of the CME and SRD patterns (Figure 5, C–F), the areas highlighted by the occlusion test were visually confirmed to be the intraretinal cystoid spaces and subretinal fluid. These highlighted areas represented the areas in each of the OCT image that were highly correlated with the accurate detection of the DME patterns.

Fig. 5.

Occlusion test successfully identified the pathologic regions in the OCT images of diffuse retinal thickening (A and B) pattern, cystoid macular edema (C and D) pattern, and serous retinal detachment (E and F) pattern. An occlusion map was generated by convolving an occluding kernel across the input image. The occlusion map is created after prediction by assigning the probability of the correct label to each occluded area. The occlusion map could then be superimposed on the input image to represent the critical areas in OCT images that were highly correlated with accurate detection of diabetic macular edema patterns.

Discussion

In this study, a DL model was developed to detect different morphologic patterns of DME based on OCT images. Our results showed that the accuracy of our DL model in the detection of the three OCT patterns was promising. Moreover, visualization of the DL model showed that the areas highlighted by the occlusion test were the most predominant pathologic regions of different DME patterns. These findings indicated that our DL model could successfully detect OCT patterns by recognizing the critical pathologic regions in OCT images.

Previous studies have suggested that the pathogenesis of each OCT pattern may be different from the others.^7–9,12 It has been shown that DRT and CME were mainly attributable to breakdown of the inner BRB, respectively, resulting in intracytoplasmic swelling and necrosis of Müller cells.^5,12–14 On the contrary, SRD is caused by dysfunction of the outer BRB, causing accumulation of subretinal fluid.^6,14 Consistently, different morphologic patterns of DME determined by OCT images may respond differently to anti-VEGF therapy because of their diverse pathogenesis.^4,5,13 In addition, the prognosis of patients with DME was also associated with integrity of the ellipsoid zone and external limiting membrane, the presence of the disorganization of retinal inner layers, and hyperreflective foci.^4,28,29 However, a much larger number of OCT images are required to train the DL model to make accurate detection of these subtle lesions. The labeling of these OCT images also needs tremendous workload. Therefore, our DL model can only be trained to predict the OCT patterns.

Nevertheless, the morphologic patterns of DME can indirectly reflect these lesions. It has been shown that the total number of hyperreflective foci and the damaging degree of the external limiting membrane and ellipsoid zone at baseline in the SRD pattern were significantly higher than those in the DRT and CME patterns.^4,28 In addition, visual acuity loss associated with the CME pattern may be primarily caused by the foveal disorganization of the retinal inner layer.²⁹ Hence, detecting the OCT patterns can indirectly or directly predict the prognosis of patients with DME, and personalized therapeutic strategies could be made according to the OCT patterns. However, current subjective detection and classification of OCT patterns is expertise-requiring and time-consuming.^21,30 Thus, it is particularly beneficial to develop an automated system for the detection of OCT patterns to assist the clinical decision-making processes in patients with DME.

DL systems have been shown to have expert-level performance in detecting various ocular diseases, including diabetic retinopathy,²⁰ DME,^18,19,21 and AMD.³⁰ For instance, a DL model developed based on 11,349 OCT images was shown to have high accuracy in detecting DME (accuracy = 98.2%).¹⁸ Another DL algorithm combined a three-dimensional segmentation network and a classification network to interpret three-dimensional OCT scans and identify different retinal diseases.¹⁹ After being trained with 14,884 OCT scan volumes, this algorithm achieved a mean AUC of 99.0% in DME detection.¹⁹ These findings indicated that DME diagnosis using DL models was promising. However, there was still no DL model for the detection of the OCT patterns.

In the current study, we trained a DL model to detect different OCT patterns. Except for effectively reducing the time required for image analysis, our DL model automatically processed features of OCT images and achieved robust performance. The mean accuracy for the detection of the three OCT patterns was 93.0% to 98.8%, with a mean AUC of 0.971 to 0.994 in internal validation (Figure 3). Moreover, the mean accuracy of detection of the DME patterns was 90.2% to 95.9%, with a mean AUC of 0.970 to 0.997 in external validation (Figure 4). These results were comparable with the previous studies using DL architectures to detect DME based on OCT images because the detection of the OCT patterns may be more complicated and challenging than distinguishing DME from normal eyes.^18,19,21 The difference in characteristics between different OCT patterns might be less obvious than that between DME and normal eyes. Therefore, the result achieved by our DL model is promising and encouraging.

In addition, our DL model could precisely recognize the critical areas in OCT images highly correlated with accurate detection. In this study, the occlusion test was performed to improve the transparency of our DL model and demonstrate the pathologic regions most critical to accurate detection of each OCT pattern. The occlusion test confirmed that the DL model made the detection based on accurately recognizing unique features of each OCT pattern (Figure 5). Deep learning models have often known as “black boxes” entities because of difficulties in understanding how the algorithms make their predictions.³⁰ These efforts to uncover the mystery of DL models may help convince ophthalmologists and patients to adopt DL models in clinical practice.

There are several limitations in the study. First, the detection ability of the DL model needs to be further validated by prospective multicenter trials. Second, another limitation is the sole use of OCT images obtained by the Heidelberg Spectralis imaging system in our study. Because the primary goal of our study was to develop a DL model for the detection of the morphologic patterns of DME, using the same OCT imaging system could help reduce confounding variables and increase accuracy of the DL model. Future researches could include images from different OCT platforms. Finally, the size of CME, SRD, and DRT lesions could not be estimated by a single B-scan. In the future, a three-dimensional model, used to quantify and derive measurements of the size of CME, SRD, and DRT lesions, can be incorporated into our DL model for more comprehensive detection and evaluation of the anatomical outcomes of different DME patterns.¹⁹

In conclusion, our DL model demonstrated high accuracy and transparency in the detection of the morphologic patterns of DME based on OCT images. After integrating the three morphologic patterns information, the final classification of OCT patterns could be obtained. This DME detection model could help make personalized therapeutic strategies for patients with DME according to the classification of OCT patterns. These results emphasized the potential of artificial intelligence in assisting clinical decision-making processes of DME.