Intelligent diagnosis of retinal vein occlusion based on color fundus photographs

Abstract

AIM

To develop an artificial intelligence (AI) diagnosis model based on deep learning (DL) algorithm to diagnose different types of retinal vein occlusion (RVO) by recognizing color fundus photographs (CFPs).

METHODS

Totally 914 CFPs of healthy people and patients with RVO were collected as experimental data sets, and used to train, verify and test the diagnostic model of RVO. All the images were divided into four categories [normal, central retinal vein occlusion (CRVO), branch retinal vein occlusion (BRVO), and macular retinal vein occlusion (MRVO)] by three fundus disease experts. Swin Transformer was used to build the RVO diagnosis model, and different types of RVO diagnosis experiments were conducted. The model's performance was compared to that of the experts.

RESULTS

The accuracy of the model in the diagnosis of normal, CRVO, BRVO, and MRVO reached 1.000, 0.978, 0.957, and 0.978; the specificity reached 1.000, 0.986, 0.982, and 0.976; the sensitivity reached 1.000, 0.955, 0.917, and 1.000; the F1-Sore reached 1.000, 0.955 0.943, and 0.887 respectively. In addition, the area under curve of normal, CRVO, BRVO, and MRVO diagnosed by the diagnostic model were 1.000, 0.900, 0.959 and 0.970, respectively. The diagnostic results were highly consistent with those of fundus disease experts, and the diagnostic performance was superior.

CONCLUSION

The diagnostic model developed in this study can well diagnose different types of RVO, effectively relieve the work pressure of clinicians, and provide help for the follow-up clinical diagnosis and treatment of RVO patients.

INTRODUCTION

As the second common retinal vascular disease, the incidence rate of retinal vein occlusion (RVO) is second only to diabetic retinopathy (DR)^[1]. According to the location of RVO, RVO is mainly divided into central retinal vein occlusion (CRVO), branch retinal vein occlusion (BRVO) and macular retinal vein occlusion (MRVO)^[2], of which BRVO is the most common. If RVO is not treated promptly and effectively, it is likely to cause serious complications, resulting in severe and irreversible visual impairment and even blindness^[3]–^[4].

The main fundus changes on color fundus photographs (CFPs) of patients with RVO include retinal hemorrhage, abnormal tortuous dilatation of retinal vessels, cotton velvet spots, and hard exudation^[5]–^[6]. The disease can also cause a variety of eye complications, such as retinal macular edema (RME), optic neuropathy, neovascular glaucoma, and traction retinal detachment^[3]–^[4],^[7]–^[9]. RME is the main cause of severe visual impairment in patients with RVO^[10]. Currently, the primary treatments for patients with RVO are vitreous injection, laser photocoagulation, and vitrectomy^[11]–^[13]. Intravitreal injection of anti-vascular endothelial growth factor (VEGF) drugs can significantly improve visual acuity loss caused by RME^[14], and laser photocoagulation is often performed in patients with intraocular neovascularization^[15]. For RVO patients, timely and effective treatment is particularly important to protect vision. Additionally, several studies have reported that the incidence of RVO increases with age. In light of the serious threat the disease poses for the vision of patients, it is very important for the screening and diagnosis of RVO. This requires us to find new methods to improve the efficiency of screening and diagnosis of RVO.

The concept of artificial intelligence (AI) was proposed in 1956, marking the birth of a new discipline. After entering the 21^st century, AI technology has developed rapidly, and many research results have been achieved in data developing, image processing and recognition^[16]. Deep learning (DL), which is a subfield of AI, is a neural network-based method to extract features from a large amount of labeled sample data and can complete complex tasks^[17]. In addition, a single DL network can carry out two classification tasks simultaneously by extracting the relevant features of a given classification task. After entering the 21^st century, DL algorithms and AI technology have been consistently developed to promote their application in medicine^[18], and many studies have demonstrated that AI technology can aid in the screening, diagnosis, and treatment of diseases. At present, in the field of ophthalmology, AI technology has made remarkable achievements in the study of ocular surface diseases such as dry eye, pterygium, keratitis^[19]–^[22], retinal vascular diseases such as DR, retinopathy of prematurity (ROP)^[23]–^[28], age-related macular degeneration (AMD)^[29]–^[31], and glaucoma^[32]–^[34]. However, few studies have investigated its application in the auxiliary diagnosis of RVO. Therefore, in this study, we used Swin Transformer to develop a diagnostic model based on CFPs to diagnose different types of RVO, and discuss the feasibility of the model application in the clinical diagnosis and treatment of RVO.

MATERIALS AND METHODS

Ethical Approval

In this study, all CFPs were obtained from the Affiliated Eye Hospital of Nanjing Medical University and the Shenzhen Eye Hospital of Jinan University. To prevent the leakage of patients' personal information, all photographs in this study were assessed anonymously and were devoid of information about the patient, except for the diagnosis.

Study Process

Figure 1 shows the general study process of this study, which was divided into five stages: image collection, image processing, database construction, model training and verification, and model testing. First, the CFPs of healthy individuals and patients with RVO were collected, the collected images were marked and classified by three fundus disease experts, and image processing and adjustment were applied; then, they were randomly divided into a training dataset, a verification dataset, and a testing dataset. The diagnostic model was built using the training and verification datasets, the parameters of the model were adjusted according to its output performance; final, the testing dataset was then used to assess the model's performance in diagnosing RVO.

Figure 1. General study process.

CRVO: Central retinal vein occlusion; BRVO: Branch retinal vein occlusion; MRVO: Macular retinal vein.

Image Collection and Processing

The initial data used in this study were obtained from the Affiliated Eye Hospital of Nanjing Medical University and the Shenzhen Eye Hospital of Jinan University and included 914 CFPs. All CFPs were selected and labeled by three fundus disease experts and marked according to Chinese fundus color photo annotations and quality control specifications. In this process, we exclude poor-quality CFPs, such as unclear photos and photos with incomplete fundus, and only keep high-quality CFPs. During the CFPs labeling process, all CFPs were labeled once by two fundus disease experts each. The two experts annotated the CFPs in a double-blind way. If the annotation results are consistent, it is recognized as the expert annotation results of the CFPs. If the results of the two experts are inconsistent, a third and more advanced fundus disease expert would label the inconsistent image. All the experimental data were processed anonymously before the study. The CFPs were divided into four categories by the three fundus disease experts: normal, CRVO, BRVO, and MRVO (Figure 2); the dataset included 259 normal, 215 CRVO, 356 BRVO, and 84 MRVO CFPs. The processing and adjustment methods for the fundus images were as follows: image standardization, unified image resolution, and image rotation.

Figure 2. Retinal vein occlusion color fundus photographs classification.

CRVO: Central retinal vein occlusion; BRVO: Branch retinal vein occlusion; MRVO: Macular retinal vein occlusion.

Dataset Construction

The experimental dataset was divided into three datasets (training dataset, verification dataset, and testing dataset). The training dataset was used to reduce the error of the intelligent diagnosis model, the verification dataset was used for the preliminary evaluation of its effectiveness, and the testing dataset was used for external verification. The training dataset contained 730 CFPs, the verification dataset 92 CFPs, and the testing dataset 92 CFPs. The four types of CFPs were randomly allocated to the three datasets and the three datasets all contain four types of CFPs (Table 1).

Table 1. Composition of the three datasets.

Parameters	Training dataset	Verification dataset	Testing dataset	Total
Normal	207	26	26	259
CRVO	171	22	22	215
BRVO	284	36	36	356
MRVO	68	8	8	84
Total	730	92	92	914

CRVO: Central retinal vein occlusion; BRVO: Branch retinal vein occlusion; MRVO: Macular retinal vein occlusion.

Model Training

We used a Swin Transformer^[35] to build the RVO diagnosis model. Swin Transformers use a hierarchical construction method similar to that used in convolutional neural networks (CNNs). As shown in Figure 3, its main structure consists of four stages (stage 1–4), and each stage consists of two parts: patch merging (stage 1 is linear embedding) and a Swin Transformer block. In addition, two structures, a Windows Multihead Self-Attention (W-MSA) and a Shifted Windows Multihead Self-Attention (SW-MSA) module, are introduced into this stage. The W-MSA can significantly reduce the computational complexity and amount of calculations, but it cannot transfer information between different windows, whereas the SW-MSA can do the latter, which solves a crucial problem. The general flow of the Swin Transformer is as follows: first, the input image is divided into blocks in the patch partition module; it is then flattened in the channel direction; and feature images of different sizes are constructed in Stages 1–4; finally, a classifier is used to classify the results.

Figure 3. Main frame structure of the Swin Transformer.

Model Evaluation

In order to evaluate the diagnostic performance of the diagnostic model^[36], we selected some performance indicators, including accuracy, sensitivity, specificity, precision, the F1-score and recall. Our calculation method is depicted in Figure 4.

Figure 4. Calculation method of each performance index.

TP: True positives; TN: True negatives; FP: False positives; FN: False negatives.

RESULTS

After training and verification, we assessed our model with the test dataset of 92 CFPs; Figure 5 depicts the confusion matrix of the results, with the horizontal axis representing the real label and the vertical axis representing the test label. The diagnostic results are showed in Table 2. The accuracy of the model for diagnosing normal, CRVO, BRVO, and MRVO reached 1.000, 0.978, 0.957, and 0.978, respectively, while the model sensitivity reached 1.000, 0.955, 0.917, and 1.000. Specificity reached 1.000, 0.986, 0.982, and 0.976, and precision stood at 1.000, 0.955, 0.971, and 0.800. The F1-score reached 1.000, 0.955, 0.943, and 0.887. In addition, we created the receiver operating characteristic (ROC) curve of the diagnostic model (Figure 6). The result showed that the area under the curve (AUC) values for normal, CRVO, BRVO, and MRVO diagnosed by the model were 1.000, 0.900, 0.959, and 0.970, respectively. These results show that the diagnostic performance of our diagnosis model was superior, the diagnostic effect of the different types of RVO was good, and the diagnostic results were highly consistent with those of the fundus disease expert. The model can thus complete the task of diagnosing different types of RVO and thereby help clinicians.

Figure 5. Diagnostic effect of the Swin Transformer model, illustrated by its confusion matrix.

CRVO: Central retinal vein occlusion; BRVO: Branch retinal vein occlusion; MRVO: Macular retinal vein occlusion.

Table 2. Diagnosis of different types of RVO using our Swin Transformer model.

Parameters	Accuracy	Sensitivity	Specificity	Precision	F1-Score
Normal	1.000	1.000	1.000	1.000	1.000
CRVO	0.978	0.955	0.986	0.955	0.955
BRVO	0.957	0.917	0.982	0.971	0.943
MRVO	0.978	1.000	0.976	0.800	0.887

RVO: Retinal vein occlusion; CRVO: Central retinal vein occlusion; BRVO: Branch retinal vein occlusion; MRVO: Macular retinal vein occlusion.

Figure 6. ROC curves for our Swin Transformer model diagnosing different types of RVO.

ROC: Receiver operating characteristic; RVO: Retinal vein occlusion; CRVO: Central retinal vein occlusion; BRVO: Branch retinal vein occlusion; MRVO: Macular retinal vein occlusion.

DISCUSSION

The main purpose of this study was to develop an RVO diagnosis model based on AI and to explore its applicability to the diagnosis of different types of RVO in the clinical practice. At present, the process of clinical diagnosis and treatment imposes considerable challenges and tremendous pressure on clinicians every day, which seriously affects their work efficiency. The development of an intelligent diagnostic model that can assist in the diagnosis of clinical diseases would greatly reduce the burden on clinicians, which improves not only their efficiency but also the best treatment provision to patients.

By combining CFPs with a DL algorithm, Chen et al^[37] constructed the Inception-v3 and DeepLab-v3 models and applied them to RVO screening and lesion segmentation. The Inception-v3 model had a sensitivity, specificity, F1-score, and AUC of 0.93, 0.99, 0.95, and 0.99, respectively, whereas the sensitivity, specificity, and AUC of the DeepLabure v3 model were 0.74, 0.97, and 0.83. Abitbol et al^[38], in order to distinguish healthy eyes from those with RVO and other fundus diseases, developed an AI model using a DL algorithm and ultra-wide CFPs. After many rounds of verification, the accuracy and AUC of their model for the diagnosis of RVO reached 0.884 and 0.912. Nagasato et al^[39] constructed a BRVO-detection model using VGG-16 networks and CFPs. Their experimental results showed good BRVO diagnosis performance. The retinal nonperfusion area (RNP) exhibits one of the characteristic changes in RVO that has an important impact on the visual acuity of patients. Tang et al^[40] developed a CNN model that can automatically segment the RNP on fluorescein angiography images to evaluate the ischemic state of RVO. The accuracy of their model stood at 0.883±0.166 after several rounds of verification. Kang et al^[41] likewise realized the intelligent diagnosis of RVO using a CNN. Their AI model had an AUC of 0.959 for diagnosing BRVO and one of 0.988 for diagnosing CRVO. These studies suggest that AI models based on DL algorithms have achieved many research results in the identification and diagnosis of RVO, demonstrating the great potential of AI models in clinical diagnosis and treatment in the future. In this study, we used a Swin Transformer to build an RVO diagnosis model that can distinguish between normal, CRVO, BRVO, and MRVO. Different from the previously published intelligent diagnosis study of RVO, our division of RVO into three subcategories and studied MRVO as a separate disease. This study therefore provides a more detailed classification of RVO, which will be helpful for clinicians to make accurate and effective treatment plans for their patients, according to the pathological characteristics and clinical manifestations of different RVO types. Finally, results show that the Swin Transformer model offers high accuracy in the diagnosis of different types of RVO, and that its diagnostic level is equivalent to that of ophthalmologists. Therefore, with the diagnostic advantages of the Swin Transformer model in different types of RVO, the model is very likely to be applied in clinical practice to assist clinicians in completing the clinical screening and diagnosis of RVO, providing great help to doctors, thereby reducing the work pressure of doctors , improve work efficiency and provide assistance in the treatment of RVO patients.

Although our diagnostic model shows good performance, this study has certain limitations. First, our datasets were relatively small, particularly the external testing dataset. Second, the quality of some CFPs in the dataset was poor, potentially due to the examination device, patients suffering from cataracts, vitreous hemorrhage, and other reasons resulting in some CFPs not being sufficiently clear. This might have affected our experimental results. Therefore, in future research, larger datasets should be used to improve image quality and thereby ensure experimental accuracy and effectiveness.

In conclusion, in this study, we built an RVO diagnosis model based on the Swin Transformer to realize the intelligent classification and diagnosis of normal, CRVO, BRVO, and MRVO CFPs, which can be used to diagnose RVO. The diagnostic performance of our model was highly consistent with that of expert ophthalmologists. Our model can thus effectively complete the task of diagnosing RVO, which can not only help solve the problem of shortages of medical resources in underdeveloped areas but also effectively alleviates the pressure imposed on clinicians to provide better treatment to patients. In addition, unlike most earlier research, we studies MRVO as a separate disease, in view of its characteristics and prognosis that differ from those of CRVO and BRVO, thereby achieving a more accurate diagnosis of RVO. Based on the experimental results of this study and current rapid developments in AI technology, we believe that AI can be fully applied in the process of clinical diagnosis and treatment in the near future to better assist clinicians.