Application and observation of artificial intelligence in clinical practice of fundus screening for diabetic retinopathy with non-mydriatic fundus photography: a retrospective observational study of T2DM patients in Tianjin, China

Abstract

Objective:

To observe the consistency of a preliminary report of artificial intelligence (AI) in the clinical practice of fundus screening for diabetic retinopathy (DR) using non-mydriatic fundus photography.

Methods:

Patients who underwent DR screening in the Metabolic Disease Management Center (MMC) of our hospital were selected as research participants. The degree of coincidence of the AI preliminary report and the ophthalmic diagnosis was compared and analyzed, and the kappa value was calculated. Fundus fluorescein angiography (FFA) was performed in patients referred to the out-of-hospital ophthalmology department, and the consistency between fluorescein angiography and AI diagnosis was evaluated.

Results:

In total, 6146 patients (12,263 eyes) completed the non-mydriasis fundus examination. The positive DR screening rate was 24.3%. When considering moderate nonproliferative retinopathy as the cut-off point, the kappa coefficient was 0.75 (p < 0.001), the sensitivity was 0.973, and the precision was 0.642, which was shown in the precision–recall curve. Fifty-nine patients referred to receive FFA were compared with non-mydriatic AI diagnoses. The kappa coefficient was 0.53, and the coincidence rate was 66.9%.

Conclusion:

Non-mydriasis fundus examination combined with AI has a medium-high consistency with ophthalmologists in DR diagnosis, conducive to early DR screening. Combining diagnosis and treatment modes with the Internet can promote the development of telemedicine, alleviate the shortage of ophthalmology resources, and promote the process of blindness prevention and treatment projects.

Introduction

Epidemiological surveys have shown that the incidence of diabetes in adults in China is as high as 11.6%, with the largest number of people with diabetes worldwide. ¹ Diabetic retinopathy (DR) is a common microvascular complication of diabetes mellitus. It is one of the most important causes of irreversible blindness in working-age individuals. ² However, if DR is screened and treated early, 90% of patients can avoid serious visual impairment. ³ Patients with diabetes seldom take the initiative to seek medical treatment before they have visual impairment. According to a survey, 50–60% of patients with diabetes have never undergone fundus examination. ⁴ The percentage of DR patients with an early diagnosis in China is less than 20%. ⁵ Early screening and timely referral can control the disease as soon as possible and effectively prevent the loss of visual function. However, compared to the high incidence rate of diabetes in China, the ability to screen for DR is insufficient. There are many reasons for this, including the lack of ophthalmic resources, lack of relevant awareness of patients, lack of effective screening methods, and the multidisciplinary process from image acquisition to DR diagnosis. Owing to the limited integration and utilization of ophthalmic medical resources, non-mydriatic fundus examination has been widely used in fundus screening. ⁶

Based on image recognition, in-depth learning technology, and artificial intelligence (AI)-assisted diagnosis, computers can obtain human-acquired knowledge in a short time to carry out reasoning, perception, learning, communication, classification, prediction, and judgment. AI analysis has several advantages. It has a large amount of data resources, precision, accuracy, and stability beyond the scope of human ability and saves human resources. AI has essential applications in various diseases such as pulmonary nodules, tumors, and skin diseases. ⁷ AI can automatically and accurately identify and diagnose DR in clinical trials through deep learning and presetting algorithms by learning many labeled DR information images and other data. ⁸ However, most studies have been conducted on high-quality image datasets, whereas few have been conducted on real-world implementations of DR screening. ⁸ Because this field involves multidisciplinary knowledge intersection and deep integration, there are still problems with data standardization and clinical evaluation unification.

AI has been widely applied for the screening of DR in developed countries. However, in China, its application is still in its infancy. If we need to improve the screening rate of DR, we need primary community hospitals with fundus screening capacity in China. The combination of non-mydriasis fundus examination and AI systems provides an opportunity for primary community hospitals. The Metabolic Disease Management Center (MMC) in our hospital, as early as October 2018, can provide non-mydriasis fundus examination for patients and is equipped with a DR AI diagnosis and treatment system. It significantly improved the screening rate of DR in the outpatient diagnosis and treatment process and issued a preliminary report of the AI before the ophthalmologist’s formal reports. ⁹ The purpose of this study was to observe the consistency between the preliminary report of AI and the report of ophthalmologists in a real-world clinical setting in China and to provide a reference for the application of AI in DR screening.

Patients and methods

Patients with type 2 diabetes mellitus (T2DM) who underwent DR screening at Tianjin 4th Central Hospital between 22 October 2018 and 16 June 2021 were selected as subjects. The clinical study protocol was approved by the Institutional Review Board (IRB) of Tianjin 4th Central Hospital. All steps were conducted in accordance with the principles of the World Medical Association Declaration of Helsinki (Trial registration code: ChiCTR1900027916). The IRB approved the collection and use of patient records according to the regulations for clinical trials in humans (IRB approval NO. 2017-SZXLL020). Written informed consent was obtained from all the patients.

The study inclusion criteria were as follows: (a) age ⩾ 18 years, (b) ability to communicate independently without mental disorder, (c) ability to complete non-mydriasis fundus examination (each patient can complete at least one fundus examination), and (d) no definite diagnosis of DR and no treatment for fundus diseases before this visit.

The exclusion criteria were as follows: (a) type 1 and other special types of diabetes mellitus, gestational diabetes mellitus, or diabetes mellitus complicated with pregnancy; (b) patients with serious mental illness and unclear consciousness; (c) it was impossible to spy on either side of the fundus for various reasons, and (d) patients with active tuberculosis and other infectious diseases.

Patients’ data from the Tianjin 4th Central Hospital were collected from the MMC doctor terminal, and patients meeting the previously mentioned conditions were screened. Full-time researchers used a unified form to collect information on demographic data and the duration of diabetes. At baseline, all data (including a standardized questionnaire and comprehensive clinical and laboratory examinations) were collected from each participant through an MMC-specialized electronic medical record system. ¹⁰ The diagnosis of type 2 diabetes was based on the World Health Organization (WHO) criteria, as fasting peripheral blood glucose (FPG) ⩾7.0 mmol/l and postprandial blood glucose ⩾11.1 mmol/l at 2 h. ¹¹

Fundus images: One standard, non-mydriatic, 45° field of view, macula-centered color, and non-stereoscopic retinal fundus images were acquired from each eye of each participant. The retina camera was a standard desktop retina camera (Reticam 3100, Chongqing Beaoxin Vision Medical Equipment Company). At least two photos per eye of the fundus (including one macula-centered and one disk-centered image) were required. The view angle of each photo is required to be ⩾45°. Image quality was confirmed using both the software quality control model and operator. The criteria for qualified images were as follows: (1) the image must cover at least 45° of the retinal area with the macula and optic disk visible; (2) at least 80% of the retinal area must be recognizable; and (3) no overexposure, underexposure, or blur caused by focusing failure and motion. The photos and related information were uploaded and sent simultaneously to the ophthalmologist. After imaging, the software sent the images and related information to a web server. The automatic DR grading results from the server by the AI system were returned in a few seconds as a temporary reference for the examiner. Patients with a positive result for referable DR were immediately referred to an ophthalmologist. Referable DR was defined by the updated International Council of Ophthalmology Guidelines for Diabetic Eye Care 2017, which includes moderate and severe diabetes phase nonproliferative retinopathy (NPDR) and proliferation stage of diabetic retinopathy (PDR). ¹² Trained technicians took only non-mydriatic images, and no pupillary dilation images were acquired. All participants’ images were anonymized before grading. Patients can immediately see fundus photographs and fundus results in real time after the ophthalmologist’s review by their mobile phone scanning the two-dimensional code. Diabetic macular edema (DME) was automatically assessed using AI. Description of DME severity grading: (1) grade 0, no apparent hard exudate(s); (2) grade 1, presence of hard exudate(s) outside the radius of one disk diameter from the macula center; and (3) grade 2, presence of hard exudate(s) within the radius of one disk diameter from the macula center. ¹³ Five ophthalmologists reviewed fundus images. All participants were retina-trained experts. Three ophthalmologists provided preliminary conclusions, and two deputy chief physicians or above were responsible for the final diagnosis.

In this study, referral patients in the initial diagnosis of AI were referred to the ophthalmology department. The Eye Hospital of Tianjin Medical University was responsible for providing further treatment services for patients with DR. With the patient’s consent, the MMC doctor completed the reservation service through WeChat. Patients were referred to the ophthalmology department for further diagnosis and treatment, including fundus fluorescein angiography (FFA). Patients were referred to the ophthalmology department for FFA, and the results were used as the gold standard for diagnosis. ¹⁴

The details about the applied AI algorithm

The automated DR grading software used in this study (VoxelCloud, China) was developed using deep-learning techniques. Two different networks were included in the software: the DR classification and quality control networks. The DR classification network, a crucial component of automated DR grading software, was trained on two datasets. The first dataset (the Eyepacs dataset) came from an extensive private retinal image database obtained between 2005 and 2015, containing 140,000 fundus photographs of approximately 37,000 patients, used to train the initial DR grading model. The images were assigned retinopathy severity levels based on the International Clinical Diabetic Retinopathy Severity (ICDRS) Scale, ¹⁵ developed by the International Council of Ophthalmology and adopted by the American Academy of Ophthalmology. The second dataset (domestic fundus dataset), ¹⁶ obtained from a public hospital in China (not from Shanghai General Hospital, which is different from the dataset obtained), contained approximately 1200 color fundus images, and the DR severity grade was assigned based on the consensus among three retinal specialists. These data were selected to improve the performance of the proposed model in complex situations.

The quality control network was trained on 6400 fundus photographs with different image qualities, a subset of the first dataset used to train the DR classification network.

All color fundus images were resized to a standard resolution of 800 × 800 pixels and normalized to pixel intensity values between 0 and 1 before being processed by the software. All the neural networks used the state-of-the-art Inception-ResNet v2 architecture. The system also included trained independent lesion models that detected the presence of lesions that contributed to DR grade, including fundus hemorrhage, hard exudates, and laser scars. ¹⁷ This algorithm was approved in China. ¹⁷

Observation indexes

This study aimed to further improve the evaluation of screening methods for DR in primary care units in China by investigating the consistency between initial reports of non-mydriatic fundus combined with AI fundus screening and final ophthalmologist reports in the real world. It is also possible to further optimize the AI algorithm. The main observation indices included (a) overall consistency between the preliminary AI and ophthalmic reports and (b) consistency between the AI and the preliminary report of ophthalmologists and the conclusion of FFA. Secondary observation: The positive rate of initial screening for DR. Research flow chart is shown in Figure 1.

Figure 1.

Research flow chart.

AI, artificial intelligence; FFA, fundus fluorescein angiography; MMC, Metabolic Disease Management Center; NPDR, nonproliferative diabetic retinopathy; PDR, proliferation stage of diabetic retinopathy; T2DM, type 2 diabetes mellitus.

Statistical analysis

The Statistical Program for Social Sciences software (version 20.0; SPSS, Inc., Chicago, IL, USA) was used for data collation and analysis. The Kolmogorov–Smirnov normal test was performed on the measurement data. Mean ± standard deviation was used for the statistical description of variables conforming to normal distribution, median (P50) was used for those not conforming to normal distribution, and percentage (%) was used for counting data. The PASS 2021 software was used to calculate the sample size [power value was 0.9, alpha = 0.05, and Kappa (H1) = 0.7, Kappa (H0) = 0.6]. p (frequencies) was set according to the results of this survey on DR in the Chinese diabetic population. ¹⁵

The consistency between the preliminary AI report and the diagnosis by ophthalmologists was compared, the kappa value was calculated, and a U-test on the obtained kappa value was performed. The MedCalc software (https://www.medcalc.org/manual/precision-recall.php) generated a precision–recall curve from the raw data. The area under the precision–recall curve was calculated using nonlinear interpolation. The F1 score measures a test’s accuracy and is the harmonic mean of the precision and recall. It was calculated at each measurement level, and F1max was the maximum F1 score for all measurement levels. FFA was used as a gold standard. The consistency between the preliminary report of AI and the preliminary reports of ophthalmologists and FFA were compared. FFA conclusion was taken as the classification variable, and AI and the preliminary report of ophthalmologists were taken as variables. The precision–recall curve was then calculated. All statistical tests were performed using bilateral tests with an alpha of 0.05.

Results

Demographic and clinical characteristics of subjects

In total, 6146 patients were eligible for inclusion in this study, including 3462 males (56.3%) and 2684 females (43.7%). The average age was 58.1 ± 11.4 years. The duration of T2DM was 4.3 ± 6.2 years (0.5–40 years). In total, 29 eyes failed to complete fundus photography. A total of 12,263 eyes were photographed, including 6125 left eyes and 6138 right eyes. There were 1438 patients (23.4%) with DR, 940 patients (15.3%) with mild NPDR, 388 patients (6.3%) with moderate NPDR, 46 patients (0.7%) with severe NPDR, and 64 patients (1%) with PDR. A total of 498 patients received ophthalmic referral, and 59 patients underwent FFA examination. The occurrence of DR in the patients’ demographic data and stratification of diabetes are shown in Table 1. The occurrences of DME are presented in Table 2.

Table 1.

Occurrence of DR in patients with different gender, age, and course of T2DM.

Items	N	DR	Referable DR
Gender
Male	3462	843 (24.4%)	296 (8.5%)
Female	2684	595 (22.2%)	202 (7.5%)
Age (years)
<30	109	24 (22.0%)	10 (9.2%)
30–50	1284	321 (25.0%)	129 (10.0%)
50–70	4122	960 (23.3%)	309 (7.5%)
⩾70	631	133 (21.1%)	50 (7.9%)
The course of T2DM (years)
<1	3317	524 (15.8%)	149 (4.5%)
1–5	1168	268 (22.9%)	73 (6.3%)
5–10	819	252 (30.8%)	87 (10.6%)
⩾10	842	394 (46.8%)	189 (22.4%)

DR, diabetic retinopathy; T2DM, type 2 diabetes mellitus.

Table 2.

The occurrence of DME.

DME grade	Grade 0	Grade 1	Grade 2
Left eyes (6125)	5844 (95.4%)	89 (1.5%)	192 (3.1%)
Right eyes (6138)	5846 (95.2%)	64 (1.0%)	228 (3.7%)
Total patients (6146)	5684 (92.5%)	114 (1.9%)	348 (5.7%)

DME, diabetic macular edema.

Consistency analysis between AI and ophthalmologist’s preliminary report

The kappa coefficient was 0.70 (p < 0.001). They were highly consistent, with an overall coincidence rate of 89.9%. When comparing the consistency with moderate NPDR as the referral cut-off point, the kappa coefficient was 0.75(p < 0.001), with a positive coincidence rate of 97.3% and a negative coincidence rate of 96.2%. The precision–recall curve is shown in Figure 2. The area under the curve (95% confidence interval) was 0.631 (range, 0.597–0.664). F1max was 0.774. The overall coincidence rate is 95.1%. The results are shown in Tables 3 and 4.

Figure 2.

Precision–recall curve with moderate NPDR as cut-off point.

NPDR, nonproliferative diabetic retinopathy.

Table 3.

Consistency between AI reports and ophthalmologist’s conclusions.

AI reports	Ophthalmologist’s reports
	0	1	2	3	4
0	9683	485	18	2	0
1	118	617	1	0	0
2	89	276	600	0	1
3	2	0	4	55	0
4	43	16	20	11	74

AI, artificial intelligence; NPDR, nonproliferative diabetic retinopathy; PDR, proliferation stage of diabetic retinopathy.

0, Diabetic retinopathy screening negative; 1, Mild NPDR; 2, Moderate NPDR; 3, Severe NPDR; 4, PDR.

Table 4.

Comparison of AI report and ophthalmologist’s conclusion with moderate NPDR as cut-off point.

AI report	Ophthalmologist’s report
	Positive	Negative
Positive	765	426
Negative	21	10,903

AI, artificial intelligence; NPDR, nonproliferative diabetic retinopathy; PDR, proliferation stage of diabetic retinopathy.

Negative included no diabetic retinopathy and mild NPDR; Positive included moderate and severe NPDR and PDR.

Forty-three patients classified as having no DR by the ophthalmologist were classified as grade 4 by AI. The 43 eyes included 19 left eyes and 24 right eyes. Thirty-eight patients were included in this study. A leopard pattern was found due to high myopia in 16 eyes; 8 eyes had old retinopathy (not caused by diabetes), 3 eyes had optic atrophy, and 2 eyes had vein obstruction. These diseases were observed in 26 eyes.

Comparison of AI and ophthalmologist’s preliminary report with FFA conclusion

Fifty-nine referred patients (118 eyes) underwent FFA, and 32 were recommended for further laser photocoagulation. Using FFA as the gold standard, the consistency between the AI and the ophthalmologist’s preliminary conclusion was further compared, as shown in Table 5. The kappa coefficients were 0.53 (p = 0.001) and 0.44 (p = 0.001), and the overall coincidence rates were 66.9% and 60.2%, respectively, which were moderately consistent with the results of the gold standard.

Table 5.

Consistency between AI, ophthalmologist’s preliminary report, and FFA conclusion.

AI reports	FFA conclusion
	0	1	2	3	4
0	3	6	0	0	0
1	0	1	0	0	0
2	0	10	41	2	5
3	0	2	2	7	6
4	0	3	2	1	27
Ophthalmologist’s preliminary reports
0	2	6	0	0	1
1	1	1	1	0	0
2	0	10	40	2	7
3	0	2	2	7	9
4	0	3	2	1	21

AI, artificial intelligence; FFA, fundus fluorescein angiography; NPDR, nonproliferative diabetic retinopathy; PDR, proliferation stage of diabetic retinopathy.

0, Diabetic retinopathy screening negative; 1, Mild NPDR; 2, Moderate NPDR; 3, Severe NPDR; 4, PDR.

If severe NPDR was used as the cut point value to compare the consistency of AI and ophthalmologist’s preliminary report with FFA conclusions, the kappa coefficient of the AI report was 0.72 (p = 0.001). Sensitivity, specificity, and accuracy were 85.4%, 87.1%, and 86.4%, respectively. Kappa coefficient of the ophthalmologist’s preliminary report was 0.67 (p = 0.001). Sensitivity, specificity, and accuracy were 79.2%, 87.1%, and 83.9%, respectively. The results are presented in Table 6. In the McNemar test, there was no significant difference between the AI and the ophthalmologist’s preliminary reports (χ² = 25.78, p = 0.250).

Table 6.

Consistency between AI, ophthalmologist’s preliminary report, and FFA with severe NPDR as cut-off point value.

AI reports	FFA		Ophthalmologist’s preliminary reports	FFA
	Positive	Negative		Positive	Negative
Positive	41	9	Positive	38	9
Negative	7	61	Negative	10	61

AI, artificial intelligence; FFA, fundus fluorescein angiography; NPDR, nonproliferative diabetic retinopathy.

The precision-recall curve is shown in Figure 3. The area under the curve of the AI reports was 0.782 (0.644–0.877) and F1max was 0.834. The areas under the curve of the ophthalmologist’s preliminary reports were 0.754 (0.614–0.856), and F1max was 0.800.

Figure 3.

Precision–recall curve with severe NPDR as cut-off point.

NPDR, nonproliferative diabetic retinopathy.

Discussion

Among the patients with T2DM, 70% developed systemic microvascular lesions, such as cataracts, glaucoma, eye movement disorders, and a series of eye diseases, among which DR is the most serious. In addition, the longer the course of diabetes, the greater the risk of retinopathy. ¹⁸ A total of 6146 patients with T2DM were enrolled in this study, and the positive rate of DR screening was 23.4%. The positive rate of DR was 15.8% when the course of the disease was less than 1 year and 46.8% when the disease course was more than 10 years. Fundus examination can detect early microaneurysms, patellar bleeding, and other early retinopathy, and active treatment can effectively delay the development of lesions. Under the conventional diagnosis and treatment mode in China, patients with diabetes need to go to the endocrinology department for initial diagnosis and the ophthalmology department to screen for eye complications. Screening for DR in daily clinical work has not yet been well established at diabetes centers in China owing to resource, infrastructure, and retinal specialist limitations. Without increasing the workload of ophthalmologists, non-mydriatic fundus examination combined with an AI diagnosis system can complete the screening process for DR in endocrine diagnosis. ⁶ This is conducive to early screening and treatment of DR. ³

This new DR diagnosis and treatment model is beginning in China. This study also showed that for the same fundus images without pupil dilation, the AI and ophthalmologist’s fundus diagnosis conclusions were highly consistent in a real-world clinical setting. When moderate NPDR was used as the referral cut-off point, the positive coincidence rate was 97.3%, the negative coincidence rate was 96.2%, the overall coincidence rate was 95.1%, and the kappa value was 0.75. The sensitivity and precision were 0.973 and 0.642, respectively, as shown by the precision–recall curve. Studies have shown that AI can not only quickly diagnose DR based on fundus color photos but also has a sensitivity and specificity as high as 96.1% and 93.9%, respectively. ⁸ This is because of the following factors: ⁸ algorithm optimization, primarily through applying deep learning principles such as neural networks; second, the DR diagnostic mode mainly depends on color fundus photos. Photography technology has made significant progress, and color fundus photographs provide essential information for AI diagnosis. Simultaneously, intelligent image recognition technology is especially suitable for deep learning. Third, the diagnostic criteria for DR grading and staging are straightforward and intuitive. Fourth, a large amount of data and experience is available for AI to learn and provide feedback. Studies have tested the sensitivity and specificity of DR diagnosis by AI based on HD color fundus photos. The results showed that AI is significantly higher than professional ophthalmologists, but this has not been verified in actual clinical practice. ¹⁹ Even when compared with FFA results, the preliminary diagnosis of AI still has a medium degree of consistency. The kappa value was 0.53, and the overall coincidence rate was 66.9%. Studies have shown that the accuracy of AI-assisted diagnosis in predicting whether DR needs treatment can be as high as 96%, significantly higher than that predicted by ophthalmologists. However, there is still a 12% false-negative rate and a 65% false-positive rate, resulting in patients missing treatment opportunities or receiving over-examination. ²⁰

Currently, patients with diabetes often contact endocrinologists from the beginning of diagnosis. While fundus screening is usually carried out in ophthalmology, active fundus screening is rarely performed, and 32% of patients with a high risk of visual impairment have never undergone fundus screening. ²¹ Non-mydriatic fundus examination can be performed in endocrinology clinics or general outpatient departments. The application of AI in DR is characterized by its high accuracy and fast analysis speed. The combination of the two is conducive to the early screening of DR, and some studies have shown that the combination can increase the screening rate of DR in outpatient departments to 70%. ²² There were only 32,000 ophthalmologists in China in 2016. There was only one ophthalmologist every 60,000 people, which is far from meeting the goal of the ‘Vision 2020’ initiative launched by the World Health Organization. ²³ There are almost no ophthalmic resources in many communities grassroots medical units, which has also become a major obstacle to the prevention and treatment of blindness in China as 70% of ophthalmologists are distributed in large- and medium-sized cities. In this study, 6146 patients were screened for fundus, and 498 cases were finally referred to ophthalmology. This diagnosis and treatment mode has not only completed the prevention and treatment of diabetic retinal fundus screening, but also greatly saved ophthalmic medical resources, so that patients needing treatment have been treated promptly. In this diagnosis and treatment mode, our referral process was implemented online. After obtaining informed consent from the patient, we made an appointment for ophthalmic treatment. The entire diagnosis and treatment process does not require ophthalmologists or patients to go back and forth, which is very convenient. Combining diagnosis and treatment modes with the Internet can promote the development of telemedicine, alleviate the shortage of ophthalmology resources at community grassroots medical units, and promote the process of blindness prevention and treatment projects. ²⁴

Conclusion and limitations

Non-mydriasis fundus examination combined with AI has a medium-high consistency with ophthalmologists in DR diagnosis, conducive to early DR screening. Combining diagnosis and treatment modes with the Internet can promote the development of telemedicine, alleviate the shortage of ophthalmology resources, and promote the process of blindness prevention and treatment projects. This study had several limitations. The research object is limited to the northern region of Tianjin. Although the ophthalmologist’s reports are issued by the deputy chief physician or above, the subjective difference cannot be ruled out. It is dishonest to show results with Kappa value alone, as the dataset is highly biased toward non-referable DR. In total, 43 patients classified as having no DR by the ophthalmologist were classified as grade 4 by the AI, and there might be some confounding factors, which were not fully balanced in this study. Moreover, the quality of the non-mydriasis fundus examination is affected by many factors such as light leakage, lens reflection, small unclear pupil, and refractive medium. The DR classifications were made only through retinal images in the study and not in classic 7 fields ETDRS (Early Treatment Diabetic Retinopathy Study) protocol or fundus examination.