Validation of Automated Screening for Referable Diabetic Retinopathy With an Autonomous Diagnostic Artificial Intelligence System in a Spanish Population

Abstract

Purpose:

The purpose of this study is to compare the diagnostic performance of an autonomous artificial intelligence (AI) system for the diagnosis of referable diabetic retinopathy (RDR) to manual grading by Spanish ophthalmologists.

Methods:

Subjects with type 1 and 2 diabetes participated in a diabetic retinopathy (DR) screening program in 2011 to 2012 in Valencia (Spain), and two images per eye were collected according to their standard protocol. Mydriatic drops were used in all patients. Retinal images—one disc and one fovea centered—were obtained under the Medical Research Ethics Committee approval and de-identified. Exams were graded by the autonomous AI system (IDx-DR, Coralville, Iowa, United States), and manually by masked ophthalmologists using adjudication. The outputs of the AI system and manual adjudicated grading were compared using sensitivity and specificity for diagnosis of both RDR and vision-threatening diabetic retinopathy (VTDR).

Results:

A total of 2680 subjects were included in the study. According to manual grading, prevalence of RDR was 111/2680 (4.14%) and of VTDR was 69/2680 (2.57%). Against manual grading, the AI system had a 100% (95% confidence interval [CI]: 97%-100%) sensitivity and 81.82% (95% CI: 80%-83%) specificity for RDR, and a 100% (95% CI: 95%-100%) sensitivity and 94.64% (95% CI: 94%-95%) specificity for VTDR.

Conclusion:

Compared to manual grading by ophthalmologists, the autonomous diagnostic AI system had high sensitivity (100%) and specificity (82%) for diagnosing RDR and macular edema in people with diabetes in a screening program. Because of its immediate, point of care diagnosis, autonomous diagnostic AI has the potential to increase the accessibility of RDR screening in primary care settings.

Introduction

Despite improvements in diabetic care, diabetic retinopathy (DR) remains one of the leading causes of blindness in Spain and the leading cause of vision loss in those of working age.¹ Prevalence of DR in people with diabetes ranges from 5% to 30%.^2-6 Screening for DR has proven to be effective in the prevention of blindness, and many professional organizations have adopted recommendations for regular DR screening.^7-9

Artificial intelligence (AI) systems for DR screening/grading have the potential to increase accessibility of DR screening for people with diabetes and increase diagnostic accuracy, efficiency, productivity, reproducibility, reduced time to access, and improved patient outcomes. Numerous academic AI algorithms have been developed for DR screening,^10-15 which have demonstrated a broad range of performance on validation sets of varying quality and reference standard. IDx-DR v2 (IDx Technologies Inc, Coralville, IA, United States) is a commercially available autonomous diagnostic AI system for identifying patients with referable diabetic retinopathy (RDR) by analyzing retinal images for signs of RDR and diabetic macular edema (DME) without physician or human oversight.¹⁶ The AI system has a class IIa CE-mark¹⁷ and is currently the only FDA-approved autonomous diagnostic AI system in the United States.¹⁸ The high diagnostic accuracy of this AI system has been previously demonstrated in publicly available datasets,¹⁶ retrospective data,^19,20 and in a prospective clinical trial against a patient outcome standard.²¹ Note that the AI system used in16 was IDx-DR vX2.1 wherein the specific biomarker detection classifiers were not finalized. IDx-DR v2 is essentially the same AI system as IDx-DR vX2.1 with finalized version of those biomarker detection classifiers.

The objective of this study is to compare the diagnostic accuracy of IDx-DR v2, to manual grading by Spanish ophthalmologists according to the International Clinical Diabetic Retinopathy severity scale International Clinical Diabetic Retinopathy (ICDR)²² in a real-life patient population in Spain.

Methods

Study Population

Retinal images were obtained from consecutive patients with diabetes, attending a regional primary care-based DR screening program in 2011 to 2012 in Valencia (Spain), according to a standardized imaging protocol. Patients were screened in 43 Health Centers from Health Departments (Alcoy, Requena, Vinaroz, and Xativa-Ontinyent).²³ According to criteria determined, patients were included when their general practitioner decided they needed a yearly ocular screening and were able to cooperate with the imaging protocol. Informed consent was obtained from all participants prior to study protocol, and the study was conducted in accordance with the 1983 Helsinki Declaration.²⁴ Ethical approval was obtained from the FOM Medical Research Ethics Committee for the use of anonymized images and grading data.

Imaging Protocol

For each patient visit, fundus photography was performed with a nonmydriatic retinal camera Topcon TRC-NW200 (Topcon, Tokyo, Japan) using a standardized protocol.⁶ The 45° retinal images of two fields (one field centered on the fovea and one field centered on the optic disc) in color were acquired for each eye by trained nurses. Mydriatic drops were used in all patients. The protocol did not include re-imaging of the patients.

The retinal images from each screening visit were uploaded to the FISABIO Oftalmología Medica (FOM), Valencia, Spain screening Patient Management System (PMS). For the purposes of this study, retrospective data from the PMS database in the form of 3531 randomly selected deidentified unique patient exams were obtained.

Reference Standard Grading

Three Spanish ophthalmologists graded the exams in masked fashion to both each other and the autonomous diagnostic AI system. Diabetic retinopathy severity and the presence of DME were graded using the ICDR scale.²² These grades were then mapped into grading the following categories:

1. Vision-threatening diabetic retinopathy (VTDR)—It is defined as Early Treatment Diabetic Retinopathy Study (ETDRS)²⁵ severity level 53 and up (or NPDR, PDR according to ICDR) and/or presence of clinically significant diabetic macular edema (CSDME).

2. Moderate diabetic retinopathy (MDR)—It is defined as ETDRS²⁵ severity level 35 to 47 (or moderate DR according to ICDR) and no CSDME.

3. Negative for more than mild DR—It is defined as ETDRS²⁵ severity 14 to 20 and 10 (or mild and no DR according to ICDR) and no CSDME. Mild is included into the AI system’s negative output based on the observed risk of a patient with mild disease going blind over a one-year screening interval as determined in the ETDRS study.²⁶

4. Laser scarring—Grader indicated the presence of pan-retinal or focal photocoagulation scars. Any patient who previously received laser treatment was excluded from analysis because they fall into the exclusion criteria for screening.

5. Poor exam quality—Subjective expert assessment that the images in the exam are of too poor of quality to grade safely. These exams were excluded from the study.

Referable diabetic retinopathy is defined as either MDR or VTDR, while negative for MDR and VTDR was considered as a non-RDR.

The final grade for the exam was adjudicated as follows. A grade was accepted when the three graders grades were in consensus at the nonreferable DR, MDR, and VTDR levels. For exams lacking consensus, an independent fellowship trained retinal specialist (FV) regraded the exams in masked fashion, and this grade was accepted if he agreed with one of the three original grades from the ophthalmologists. Otherwise, the exam was considered poor exam quality and excluded from the study.

Autonomous Diagnostic AI System

The AI system provides a specific ICDR-based DR-level as output by analyzing the retinal images in the exam through AI-based algorithms and learned filters which form a set of partially dependent detectors.¹⁶ Key features of the AI system include deep learning²⁷ based, high performing, redundant and independently validated lesion detectors, a final disease detection algorithm which evaluates the information provided by those lesion detectors, and a separate algorithm which evaluates whether the images provided are of sufficient quality and provide the necessary retinal coverage to enable the exclusion of presence of disease safely for the patient.

Specifically, the device outputs the following disease levels—VTDR, MDR, and negative, within one minute. Image quality analysis is also performed immediately by the autonomous diagnostic AI system. When the quality of image(s) is too low to rule out RDR, the exam is designated as insufficient quality and no disease output is available from the device. As the patient is still present at that time, in clinical practice, the camera-operator can retake (only) the failed images immediately and resubmit the exam after which a disease output should follow. Similarly, the imaging protocol is also evaluated by the AI system. The protocol analysis includes automated detection of the left and right eye images, and fovea and optic nerve head localization to ensure that both the fovea and the optic nerve head centered images from each eye of the patient are submitted. In cases where the AI system encounters a failed imaging protocol, the exam is designated as incorrect protocol and no disease output is available from the device. The camera-operator is then instructed to retake the exam.

The output of the autonomous diagnostic AI system is generated by combining and thresholding two different posterior probabilities: one reflects the likelihood of VTDR presence and the other of MDR presence.

Statistical Analysis

The duration of diabetes was self-reported. The AI system continuous index outputs were used to create receiver operating characteristic curves (ROC) for VTDR and MDR, and area under the ROC (AUC) was calculated.

We report the sensitivity and specificity, positive predictive value (PPV) and negative predictive value (NPV), and AUC of the AI system against the adjudicated manual grading, as well as their 95% confidence intervals. Additionally, any potential effects of sex, age, and duration of diabetes on sensitivity and specificity were tested. For each subcategory, we tested for significance in the difference of the observed metrics using Pearson’s chi-squared test, where a P-value of .05/3 (after Bonferroni correction) was considered significant. For further analysis, we also calculated the difference between the AUC obtained in this study to the AUC obtained from the French dataset¹⁶ using the same algorithm, wherein a P-value of .05 was considered significant.^28,29 Finally, we also compute the efficiency gain derived from the reduction of ophthalmologist workload by using the autonomous diagnostic AI system.

Results

From 3531 exams used in this study, 250 exams were excluded based on laser scarring; 195 exams were excluded because of insufficient image quality as judged by the graders only; 404 exams were excluded from the study based on insufficient image quality as determined by the AI system; and two exams were excluded based on insufficient image quality as determined both by the graders and the AI system.

In 3531 subjects, median age of the patients was 74 years and the median duration of diabetes was 6.9 years. In total, 55% of the patients were female and 89% had type 2 diabetes.

In 2680/3531 (76%) subjects, according to manual grading, prevalence of RDR was 111/2680 (4.14%) and prevalence of VTDR was 69/2680 (2.57%).

Table 1(a) and (b) tabulates the AI system outputs to the manual grading for RDR and VTDR. The AI system performance is summarized in Table 1(c). The AI system showed 0 false negatives for both RDR and VTDR, resulting in a sensitivity of 111/111 = 100% (95% confidence interval [CI]: 97%-100%) for RDR and a sensitivity of 69/69 = 100% (95% CI: 95%-100%) for VTDR. The corresponding specificities were 2102/2569 = 82% (95% CI: 80%-83%) for RDR and 2471/2611 = 95% (95% CI: 94%-95%) for VTDR. The corresponding PPV was 19.2% (95% CI: 18%-21%) for RDR and 33% (95% CI: 30%-37%) for VTDR, while the NPV was 100% for both RDR and VTDR. Consequently, the accuracy of the AI system for RDR was 82.57% and for VTDR was 94.78%. The F measure of the AI system for RDR was 0.322 and for VTDR was 0.496.

Table 1.

Performance Analysis of Artificial Intelligence System Compared to Gold Standard.

(a) Grading by the AI system compared to the gold standard for RDR
AI system
Gold standard	No RDR	RDR	Total
No RDR	2102	467	2569
RDR	0	111	111
Total	2102	578	2680
(b) Grading by the AI system compared to the gold standard for VTDR
AI system
Gold standard	No VTDR	VTDR	Total
No VTDR	2471	140	2611
VTDR	0	69	69
Total	2471	209	2680
(c) Sensitivity, specificity, negative predictive value (NPV), and positive predictive values (PPV) of the AI system for detection of RDR and VTDR
	RDR		VTDR
Sensitivity	$\frac{111}{111}=$100% (96.73%-100%)		$\frac{69}{69}=$100% (94.79%-100%)
Specificity	$\frac{2102}{2569}=$81.82% (80.27%-83.30%)		$\frac{2471}{2611}=$94.64% (93.7%-95.47%)
NPV	100%		100%
PPV	19.20% (17.96%-20.51%)		33.01% (29.55%-36.67%)

Abbreviations: AI, artificial intelligence; RDR, referable diabetic retinopathy; VTDR, vision-threatening diabetic retinopathy.

The AUC was 0.984 (95% CI: 0.97-0.99) when testing the ability of the AI system to detect RDR compared to manual grading and the ROC is illustrated in Figure 1. For the detection of VTDR by the AI system, the AUC was 0.998 (95% CI: 0.997-0.999) and the ROC is shown in Figure 1. For RDR, the measured AUC in the present study (0.984) was not significantly different (P > .70) than that of the AUC measured for the same AI system using the French dataset (0.980).¹⁶ Additionally, the 95% CI of the measured AUC for RDR of the AI system in the presented study (95% CI: 0.97-0.99) and the French dataset (95% CI: 0.968-0.992)¹⁶ both overlap with the theoretical maximum AUC of 0.99 measurable for a perfect detection system using multiple graders.²⁹

Figure 1.

Receiver operator curve: (left) Referable diabetic retinopathy; (right) vision-threatening diabetic retinopathy.

The AI system resulted in 111 true positives (TP) and 467 false positives (FP) for RDR out of 2680 processed exams (PE), thus resulting in an efficiency gain (EG) of 78.43% (EG = (1 – ((TP + FP) / PE) × 100).

We stratified sensitivity and specificity for the following subcategories: sex, age over 65, and duration of diabetes >10 years. There were no significant effects of any of these categories on sensitivity (P > .05/3). For specificity, there was no significant effect from sex (P > .655), but there was a significant effect of diabetes duration (P < .05/3) and age (P < .05/3): the AI system showed a higher specificity in subjects with a diabetes duration below ten years; 86%, whereas specificity for subjects with a diabetes duration over ten years was 71% (P < .0001). Similarly, the AI system showed higher specificity in younger subjects; the specificity for subjects younger than 65 was 89% compared to 79% for subjects older than 65 (P < .0001).

Discussion

This study compares the performance of an autonomous diagnostic AI system for RDR and VTDR detection against an adjudicated manual grading standard established by four ophthalmologists in a real-life patient population in Spain. The AI system studied in this paper has previously been studied using retrospective datasets in the French population,¹⁶ as well as prospectively in a preregistered clinical diverse US sample.²¹ This study on a Spanish population should be considered confirmatory in the context of prospective demonstration of the system’s performance, reflecting the performance of the AI system in a Spanish population with a reference standard established by four ophthalmologists. The breakdown of patient level disease grade based on diabetic age is shown in Table 2(a) and (b). As expected, the data from Table 2(a) and (b) shows that the percentage of patients with moderate DR and VTDR increases from 0.37% (moderate DR) and 1.51% (VTDR) for diabetic age 0 to 9 years to 3.79% (moderate DR) and 3.90% (VTDR) for diabetic age greater than 10 years.

Table 2.

Study Population Data for Diabetic Retinopathy Severity Based on Diabetic Age.

(a) All patients with disease grade
Diabetic age	None/mild	Moderate	VT	Total
0-9 years	1875	7	29	1911
≥10 years	827	34	35	896
Not available	290	3	5	298
Total	2992	44	69	3105
(b) Patients with disease grades excluding IDx-DR poor image quality exams
Diabetic age	None/mild	Moderate	VT	Total
0-9 years	1614	7	29	1650
≥10 years	691	32	35	758
Not available	264	3	5	272
Total	2569	42	69	2680

Similar retrospective studies have been conducted in a variety of populations using other AI algorithms and study protocols.^10-16 Variation in the performance between these studies is to be expected owing to different graders, disease distribution, and reference standards. As such, it is important that this study should be considered in the context of additional research further validating the performance of the AI system used in this study.

For example, when properly designed, the use of manual adjudicated grading by multiple ophthalmologists to determine AI performance can provide an excellent comparison to the standard of care in any given care setting or geography. At the same time, such studies do not necessarily provide a proxy for an AI’s relationship to patient outcomes, given the documented phenomenon of diagnostic drift—a term coined to describe how, over time, the standards by which health professionals diagnose disease “drift” from their original, outcome based, definitions.^30,31 In the recently published prospective pivotal trial,²¹ the highest possible level reference standard was used; a patient level clinical outcome-based reference standard or “gold standard,” where precise disease-level diagnoses are correlated to the actual probability of vision loss²⁵ For DR, this is determined through dilated four-widefield stereo digital color fundus photography as well as macular optical coherence tomography (for center-involved macular edema) performed by a certified and experienced ophthalmic photographer and read by a validated reading center (in this case the Wisconsin Reading Center), according to a patient outcome correlated severity scale, the ETDRS.²⁵ Under this protocol, the autonomous diagnostic AI system demonstrated a performance of 87% sensitivity and 91% specificity. The seemingly better performance of the identical AI system presented in the Spanish dataset is because it is being compared with the standard of care (manual adjudicated grading by clinical ophthalmologists) rather than the earlier described gold standard.^32-34

In the ICDR scale, MDR includes the ETDRS²⁵ severity levels 35 to 47. Included in the definition of an ETDRS level 35 is the presence of any lesion characteristic of DR (including hemorrhages, exudates, or cotton-wool spots) along with one or more microaneurysms. Not all alternative, nonpatient outcome based grading systems include the full ETDRS 35 severity into their definition of moderate DR. Such confusion results in diagnostic drift, depending upon the graders prior training and experience. Therefore, the diagnostic sensitivity of the system to RDR seems higher (100% vs 87%) in this study, but of course the underlying performance is the same. Figure 2 illustrates example of an exam which was graded as being negative for RDR by the ophthalmologists but were flagged as being positive for RDR by the AI system. Each exam clearly has lesions that warrant a moderate grade in the ICDR scale because there are hemorrhages and additionally other lesions, exudates, microaneurysms, or cotton wools.²² Specificity can be expected to go up in older people because they have less reflective internal limiting membrane, and the presence of this membrane can lead to FP.²¹ However, in this study, we found instead that specificity was estimated to be lower in older people. We hypothesize that this is because such exams that were graded as negative by the readers in this study but might well have been considered positive according to ETDRS (Figure 2).

Figure 2.

Example illustration of exam graded as being negative for referable diabetic retinopathy by human graders but graded as positive for moderate diabetic retinopathy by the artificial intelligence system.

The autonomous diagnostic AI system provides a patient-level result; not an image-level assessment. Therefore, the algorithm was designed to provide an “insufficient image quality” result, rather than a disease diagnosis in cases where the image data were insufficient to make a diagnosis. This is done for patient safety reasons and is an essential component to any fully automated diagnostic system. In this study, the AI system provided a diagnostic result for 86.8% of patients and provided a “poor quality” result for 13.2% of patients that were assigned a reference standard grade. Since this was a retrospective study, there was no ability to reimage the patient to attain a sufficient quality exam for grading by experts and/or the AI system. However, in the prospective clinical trial study,²¹ wherein the patients were reimaged/dilated as required, the imageability (% of screened patients receiving a diagnosis) for the same AI system as used in this presented retrospective study was 96.1% (95% CI, 94.6%-97.3%). It is at the discretion of licensed ophthalmologists to make a medical decision with regard to RDR in insufficient quality images. Figure 3 shows examples of image pairs flagged as poor image quality by the AI system (hence no DR grade was available), while the same image pairs were graded as good quality and received a final DR disease severity reference standard grade. Because this is a retrospective study, reimaging of the patients was not feasible. In a prospective clinical trial of the same AI system where reimaging of poor quality exams was allowed, and pharmacological dilation was used when necessary, the AI system provided a disease diagnosis in 96% of patients that also had a reference standard grade.²¹

Figure 3.

Fundus photography of several patients performed with a nonmydriatic retinal camera (Topcon TRC-NW200). Each column shows image pairs from the same exam. Image pairs in the illustration were designated as poor image quality by the artificial intelligence system, and hence, no diabetic retinopathy grade was provided. Contrastingly, the human readers found the images to be of sufficient image quality and provided a diabetic retinopathy grade.

Overall, in the context of the broader research on the autonomous diagnostic AI system studied here, this evaluation confirms the device’s robust performance across clinical populations, as originally observed in rigorous pivotal research.²¹ In summary, compared to manual grading by ophthalmologists, the AI system had a high sensitivity (100%) and specificity (82%) for diagnosing DR and macular edema in people with diabetes in a screening program. Because of its immediate, point of care diagnosis, autonomous AI has the potential to increase the accessibility for DR screening in primary care settings.