Diabetic Retinopathy Screening with Automated Retinal Image Analysis in a Primary Care Setting Improves Adherence to Ophthalmic Care

Abstract

Purpose:

Retinal screening examinations can prevent vision loss from diabetes, but are costly and highly underutilized. We hypothesized that artificial intelligence-assisted non-mydriatic point-of-care screening administered during primary care visits would increase the adherence to recommendations for follow-up eye care in patients with diabetes.

Design:

Prospective cohort study.

Participants:

Adults ages 18 or older with a clinical diagnosis of diabetes being cared for in a metropolitan primary care practice for low-income patients.

Methods:

All participants underwent non-mydriatic fundus photography followed by automated retinal image analysis with human supervision. Patients with positive or inconclusive screening results were referred for comprehensive ophthalmic evaluation. Adherence to referral recommendations was recorded and compared to the historical adherence rate from the same clinic.

Main outcome measure:

Rate of adherence to eye screening recommendations.

Results:

By automated screening, 8.3% of the 180 study participants had referable diabetic eye disease, 13.3% had vision-threatening disease, and 29.4% had an inconclusive result. The remaining 48.9% had negative screening results, confirmed by human over-read, and were not referred for follow-up ophthalmic evaluation. Overall, the automated platform showed a sensitivity of 100% (CI 92.29% to 100%) in detecting an abnormal screening result, while its specificity was 65.67% (CI 56.98% to 73.65%). Among patients referred for follow-up ophthalmic evaluation, the adherence rate was 55.4% at 1-year compared to the historical adherence rate of 18.7% (P < 0.0001, Fisher’s Exact Test).

Conclusions:

Implementation of an automated diabetic retinopathy screening system in a primary care clinic serving a low-income metropolitan patient population improved adherence to follow-up eye care recommendations while reducing referrals for patients with low-risk features.

INTRODUCTION

Diabetic retinopathy (DR) is an increasingly prevalent public health threat, but early screening and intervention can effectively prevent vision loss.^1–3 The American Academy of Ophthalmology (AAO) and the American Diabetes Association (ADA) recommends that most patients with diabetes receive annual screening eye examinations, with lower risk patients eligible for screening once every two years.^{4, 5} However only approximately 60% of American patients with diabetes adhere to these guidelines,⁶ with far lower rates among socioeconomically disadvantaged populations, the same populations at greatest risk for vision loss.^7–10 In a recent study of our institution’s primary care clinic, which serves a predominantly low-income, metropolitan patient population, the rate of adherence to screening examinations for DR was less than 20%.¹¹

Within the next decade rates of adherence to DR screening guidelines are expected to worsen, due to both an increasing prevalence of diabetes and a projected shortfall of medically-trained ophthalmic providers.^{1, 12} Therefore, alternatives to traditional in-office screening methods must be developed to meet the increasing demands of diabetes care. Increasingly, digital fundus photography has been used to detect DR in lieu of standard in-office dilated eye examinations.^13–16 The United Kingdom serves as a paradigm of this model, as the National Health Service (NHS) implemented a nationwide teleretinal screening program in 2003 with enormous success.¹⁷ An analysis of blindness certificates from 2009 to 2010 revealed that, for the first time in the last five decades, DR was no longer the leading cause of blindness in working-aged adults in the United Kingdom.¹⁸ From 2015 to 2016, over 80% of the 2.59 million people diagnosed with diabetes were screened under the NHS Diabetic Eye Screening Program.¹⁹

An even newer technology, automated retinal imaging analysis systems (ARIAS), is based on artificial intelligence and deep learning modalities and has repeatedly proven to be extremely sensitive and specific in detecting referable DR when compared to teleretinal reading centers.^20–24 Therefore, ARIAS offers the potential for more cost-effective, real-time screening that can be directly integrated into primary care settings. While there is considerable enthusiasm regarding the efficacy of ARIAS in detecting DR, its effects on patient behavior and outcomes are still poorly understood, especially when deployed as a non-mydriatic tool used by primary care givers.^{25, 26} In the current prospective cohort study, we sought to determine whether ARIAS, used as a point-of-care screening tool in a primary care setting, improved the rates of adherence to subsequent eye care recommendations.

METHODS

Patients with type 1 or type 2 diabetes mellitus over the age of 18 who had not attended an ophthalmology appointment within one year of the screening visit were eligible for participation in this prospective cohort study. From January 1, 2018 to August 31, 2018, patients were recruited during their primary care appointments at the Primary Care Medicine Clinic of Barnes Jewish Hospital in Saint Louis, MO. This internal medicine residency clinic provides care to uninsured and underinsured populations within the metropolitan area. Patients were excluded if they did not speak English or had a history of seizure disorders. The study was approved by the Human Research Protection Office/Institutional Review Board of the Washington University School of Medicine and adhered to the tenets of the Declaration of Helsinki and the Health Insurance Portability and Accountability Act. Informed consent was obtained from all participants.

Eligible participants were screened using non-mydriatic fundus photographs taken with a CR-2 retinal camera (Canon U.S.A, Inc., Melville, NY). The resultant images were subsequently analyzed using EyeArt 2.0 automated DR screening software (Eyenuk, Inc., Woodland Hills, CA). Software determination of DR grade has been previously described.²⁷ In brief, images with no apparent DR or mild DR on the International Classification of Diabetic Retinopathy (ICDR) scale²⁸ were classified as “nonreferable” whereas images with moderate DR or greater were classified as “referable.” Vision-threatening DR was defined as severe DR (or greater) or presence of center-involving macular edema. All participants were informed of their screening results and received standardized education from the examiners about the ophthalmic consequences of diabetes as well as an informational handout on DR. Patients with inconclusive screening results or those with evidence of DR requiring follow-up ophthalmic care had a referral order placed through the hospital electronic medical record system, and subsequently received a phone call within 2 weeks of their ARIAS screening to schedule an appointment at the resident ophthalmology clinic. Failed initial telephone contact resulted in 2 more phone calls over the following two business days. Patients who were unable to be contacted after 3 telephone calls received a letter regarding information about their testing results as well as instructions to schedule a follow-up examination.

Patients who were classified as having “vision-threatening diabetic eye disease” by the automated screen were instructed to attend an appointment at the hospital ophthalmology clinic within 1 month for comprehensive evaluation. Participants who were classified as having “referable diabetic eye disease” were told to receive a follow-up dilated eye examination within 3 months. Patients with inconclusive screening results were also referred to the ophthalmology clinic and advised to attend an appointment within 3 months.

All fundus images were independently manually reviewed and graded by fellowship-trained retina specialists (4 academic faculty members and 1 private practice retina specialist who received their fellowship training at Washington University School of Medicine) within 1 week of the screening visit at the primary care medicine clinic. Human graders utilized the ICDR grading system to classify the severity of diabetic retinopathy; based on their ICDR classification, patients were categorized as having nonreferable, referable diabetic retinopathy, or vision threatening diabetic retinopathy, utilizing the same categorization schema as EyeArt software. In cases of disagreement between automated screening and manual grading, the more conservative of the two follow-up interval recommendations was used. Participants with incidental findings as noted by the retina specialists (e.g. glaucoma, age-related macular degeneration, cataracts) were notified by telephone and advised to attend an ophthalmology appointment within the next 3 months. Any emergent ocular findings (such as retinal detachment, retinal vascular occlusion, etc.) merited a break from protocol and emergent evaluation as clinically indicated. Follow-up appointments were scheduled by telephone call after the initial primary care encounter.

For patients who required follow-up eye care, rates of attendance at ophthalmology appointments were calculated by reviewing eye clinic records at 3 months, 6 months, and 1 year after their reference clinic visit. As reference, we used the historical adherence rate of consecutive adult patients with diabetes seen between July 1, 2016 and March 31, 2017 from the same primary care clinic, as our group has previously reported.¹¹ We used comprehensive and inclusive criteria, with any adult patient with a diagnosis of diabetes seen in the clinic for a non-emergent reason during this time frame being included into the historical cohort analysis. In the final comparative analysis, patients in both the prospective cohort and the historical control were considered “adherent with eye care recommendations” if they attended an eye clinic appointment within 1 year after their reference primary care clinic visit. Fisher’s Exact Test was used to compare rates of adherence between the study participants and the historical controls at different time intervals, and Chi-square analysis was used for subgroup analyses. Prism 6.0 Software (GraphPad Inc., San Diego, CA) was used and all analyses were performed with significance at P < 0.05.

RESULTS

One hundred eighty-five adults with diabetes consented to participate in this prospective study. Five participants were excluded from the analysis: 2 had a history of seizures and 3 left the clinic before the screening exam could take place.

Demographic analysis of the remaining 180 patients revealed there were no statistically significant differences in age, gender, or ethnicity between the study cohort and the historical control cohort (Table 1). Mean glycated hemoglobin index tended to be higher in the prospective cohort (P = 0.028, 2-tailed t-test), although the exclusion of non-English speakers (who tend to have better diabetes control that English speakers in our healthcare system¹¹) and those who had attended an eye examination within 1 year in the prospective cohort may explain this difference.

Table 1: Patient Demographics.

Age and glycated hemoglobin index data were compared by two-tailed t-test, sex data were compared by Fisher’s Exact test, and race data were compared using Chi square analysis. Values represent mean (SD) unless otherwise specified, and values in bold indicate significant results.

Demographics	Historical Cohort n = 974	Prospective Cohort n = 180	P Value
Age, years	56.0 (13.8)	57.4 (11.4)	0.17
Sex, # (%)
Male	409 (42.0)	83 (46.1)	0.37
Female	565 (58.0)	97 (53.9)
Race, # (%)
White	161 (16.5)	27 (15.0)	0.25
Black or Afr Am	746 (76.6)	141 (78.3)
Other†	65 (6.7)	10 (5.6)
Unknown	2 (0.2)	2 (1.1)
Glycated hemoglobin index, % (n = 763 Historical; n = 140 Prospective)	8.0 (2.1)	8.4 (2.1)	0.028

Of the 180 included participants, 15 (8.3%) had a positive screening result for referable diabetic eye disease, 24 (13.3%) had a positive screening result for vision-threatening diabetic eye disease, 88 participants (48.9%) had a negative screening result, and 53 (29.4%) had an inconclusive screening result as determined by the software. No false negatives from ARIAS-screening were found, indicating 100% agreement between automated and manual-grading for patients without any apparent DR (Table 2). After pooling positive and inconclusive screening results together, we found that the sensitivity and specificity for an abnormal screening result was 100% (CI 92.29% to 100%) and 65.67% (CI 56.98% to 73.65%), respectively. The positive predictive value for the screening test was 50%, while the negative predictive value was 100%. Among 17 patients, additional pathology that required evaluation earlier than recommended by ARIAS was detected by human graders: 9 cases of grade 1-2 hypertensive retinopathy, 2 cases with age-related macular degeneration, 7 who were glaucoma suspects, and 1 with non-specific chorioretinal scarring (two patients had both hypertensive retinopathy and suspicion of glaucoma).

Table 2. Agreement between automated and manual human grading of retinal photographs.

ARIAS=Automated Retinal Image Analysis Software; rDED = referable diabetic eye disease; vtDED = vision threatening diabetic eye disease.

		Manual Grade
		(−) rDED	(+) rDED	(+) vtDED	Inconclusive	TOTAL	TOTAL Referred
ARIAS Screen	(−) rDED	88 (48.9%)	0 (0%)	0 (0%)	0 (0%)	88 (48.9%)	0
	(+) rDED	14 (7.8%)	1 (0.6%)	0 (0%)	0 (0%)	15 (8.3%)	15
	(+) vtDED	11 (6.1%)	7 (3.9%)	6 (3.3%)	0 (0%)	24 (13.3%)	24
	Inconclusive	21 (11.7%)	0 (0%)	3 (1.7%)	29 (16.1%)	53 (29.4%)	53
	TOTAL	134 (74.4%)	8 (4.4%)	9 (5.0%)	29 (16.1%)	180 (100%)	92

The overall rates of adherence to follow-up eye care recommendations for patients in the prospective cohort at 3 months (Table 3). Rates of adherence at 6 months and 1 year were also calculated for the purpose of comparison to the historical control cohort. At 12 months after the index primary care visit, 51 patients out of 92 total patients referred by ARIAS (55.4%) had attended an eye care visit. The majority of these patients attended the eye clinic within 3 months after referral from their primary care visit (30/51 patients, Table 3; Figure 1). In subgroup analyses, patients who received a screening result of vision threatening DR had similar rates of attendance at subsequent eye clinic visits as those with referable diabetic eye disease or an inconclusive screening result (at 12 months: 58.3%, 46.7%, and 56.6%, respectively, P = 0.75, Chi-square).

Table 3. Rates of completed eye examination among cohorts.

Rates of a completed eye examination were recorded at 3, 6, and 12 months after the index primary care visit. The reference cohort was a historic control, which represents attendance at a recommended eye clinic visit at the same time intervals. The proportion of total patients who attended or did not attend a recommended eye exam was compared between cohorts for each time interval using Fisher’s Exact test. Values in bold indicate significant results. rDED = referable diabetic eye disease; vtDED = vision threatening diabetic eye disease.

	Attended Recommended Visit	Did Not Attend Recommended Visit	P Value
At 3 months:
Historical Control:	92 (9.4%)	882 (90.6%)
Prospective Cohort:
rDED (n=15)	4 (26.7%)	11 (73.3%)
vtDED (n=24)	10 (41.7%)	14 (58.3%)
Inconclusive (n=53)	16 (30.2%)	37 (69.8%)
Total (n=92)	30 (32.6%)	62 (67.4%)	<0.0001
At 6 months:
Historical Control:	131 (13.4%)	843 (86.6%)
Prospective Cohort:
rDED (n=15)	7 (46.7%)	8 (53.3%)
vtDED (n=24)	13 (54.2%)	11 (45.8%)
Inconclusive (n=53)	23 (43.4%)	30 (56.6%)
Total (n=92)	43 (46.7%)	49 (53.3%)	<0.0001
At 12 months:
Historical Control:	182 (18.7%)	792 (81.3%)
Prospective Cohort:
rDED (n=15)	7 (46.7%)	8 (53.3%)
vtDED (n=24)	14 (58.3%)	10 (41.7%)
Inconclusive (n=53)	30 (56.6%)	23 (43.4%)
Total (n=92)	51 (55.4%)	41 (44.6%)	<0.0001

Figure 1: Non-Attendance Rate between Groups.

The rate of non-adherence was determined based on the number of uncompleted referral examinations as a measure of time from the reference primary care visit. The survival analysis for both the historical (“Prior to ARIAS”, n=974) and prospective cohorts (“With ARIAS”, n=92) are shown, with 95% confidence intervals around the curves (shaded areas) and P < 0.0001 by Log-rank (Mantel-Cox) test.

The retrospective analysis of historical control patients seen within the same primary care clinic included 974 adult patients with type 1 or type 2 diabetes.¹¹ Within this cohort, only 182 patients (18.7%) adhered to screening recommendations in the year following their reference primary care visit. Among patients who were due for a screening exam at the time of their primary care visit (644 patients), only 11.5% (74 patients) adhered to the screening recommendations and completed a subsequent screening exam within the following year. Compared to this historical cohort, patients screened by ARIAS were more likely to follow up with subsequent ophthalmic examination (Table 3).

Among patients who were adherent to follow-up eye care after ARIAS screening, the mean time period between the index primary care visit and the eye clinic visit was 98.0 days. A comparison of the final retinal diagnosis after dilated examinations and the initial ARIAS screening results is shown in Table 4. Of the 51 referred patients, 32 were ultimately found to have no DR or mild DR. Only 6 referred patients were found to have vision-threatening DR that necessitated further intervention. The time frame for attendance at follow-up appointments in these patients ranged from 0 days to 133 days, with 4 of the 6 patients attending an appointment within 5 weeks after their ARIAS screening. Of those patients, 2 received intravitreal injections, 1 received pars plana vitrectomy, 2 were lost to follow up, and 1 died before retinal intervention could be performed.

Table 4. Agreement between automated grading of retinal screening photography and clinic-based findings, among patients who underwent recommended dilated retinal examination.

ARIAS=Automated Retinal Image Analysis Software; rDED = referable diabetic eye disease; vtDED = vision threatening diabetic eye disease.

		Referral Exam Findings
		(−) rDED	(+) rDED	(+) vtDED	TOTAL
ARIAS Screen	(+) rDED	7	0	0	7
	(+) vtDED	4	5	5	14
	Inconclusive	21	8	1	30
	TOTAL	32	13	6	51

DISCUSSION

Though the treatment of vision loss in diabetes has evolved greatly over the past two decades, prevention of late stage disease through early screening remains an effective management strategy.²⁹ Current screening paradigms relying on dilated retinal examination or manual grading of teleretinal images will not be sufficient to meet the expected increases in the prevalence of DR in the near future. Emerging technologies such as non-mydriatic fundus photography and ARIAS offer potential solutions for current and anticipated deficits in DR screening.^20–24 These systems demonstrate remarkable efficacy, surpassing the accuracy of traditional reading centers.^{21, 24} Despite such enthusiasm, few studies to date have examined the impact of ARIAS on visual outcomes in patients with diabetes.^{25, 26}

The present study aimed to address this gap by studying the effect of a point-of-care retinal screening system using non-mydriatic retinal photography and EyeArt 2.0 ARIAS software on the behavior of patients with diabetes. Specifically, we asked whether ARIAS improves adherence to follow-up eye care, as recommended in the primary care setting. We found that the deployment of this system in a clinic serving a low-income, metropolitan patient population with a historically low rate of adherence to DR screening examination, vastly improved the likelihood of attendance at recommended follow-up eye care visits (Table 3). Furthermore, we found that ARIAS point-of-care screening reduced the number of eye clinic referrals by ~50%, thereby reducing unnecessary retinal examinations for low risk patients (Table 2). We observed a high rate of inconclusive screening results (29%, Table 2), due to small pupils, cataracts, or other media opacities causing suboptimal photograph quality. Since all patients who received an inconclusive screen were referred for examination, and since most of these patients ultimately received a diagnosis of “no apparent DR” or “mild DR” at their eye clinic visit, ARIAS is likely to perform even better as camera technology improves.

In a previous study, a subsidized screening program using nonmydriatic photography in primary clinics with manual grading of teleretinal images failed to show any improvement in patient adherence to follow-up eye care within the recommended period.²⁶ However, the delayed manner in which those patients received screening results and follow-up recommendations may have reduced the effectiveness of the screening program, as such delays limit opportunities for immediate, customized patient education and counseling. Furthermore, the immediate presentation of retinal screening results, including the images of the patient’s retinas, could have profound effects on follow-up behavior. Since this study utilized a cloud-based ARIAS capable of returning a rapid real-time diagnosis rather than a teleretinal manual grading system, we used the screen as a point-of-care test. Therefore, screening results could be provided immediately to patients at their primary care visit, along with a physical copy of their retinal images and standardized DR education. This key difference, along with differences in target populations, could account for the apparent discrepancy between our analysis and prior studies.

Our analysis found that non-mydriatic retinal photography and ARIAS screening performed during primary care visits dramatically increased patient adherence: 55.4% of patients who were advised to receive follow-up eye care attended an ophthalmology appointment, compared with 18.7% of patients in the historical control cohort (Table 3). Such improvement in patient adherence could have been influenced by DR education provided to the study participants. Limited healthcare knowledge among patients is one of the most significant barriers to diabetes care, and improving patient education can dramatically improve outcomes, such as DR screening adherence.^{7, 30, 31} Providing patients with their retinal photographs – as done in the present study – could be an important component of the education process, as it creates a visual representation of health concepts that would otherwise be entirely abstract.

A second notable finding from this study was that the ARIAS system was 100% accurate in identifying patients without DR. 48.9% of patients were determined to have no signs of disease, thus reducing the number of referrals by nearly half. ARIAS screening programs therefore have the ability to eliminate unnecessary healthcare spending. Other studies have similarly demonstrated potential cost-savings with ARIAS when compared to manual grading.^{15, 32, 33} One potential downfall of using ARIAS screening systems is that, in the absence of analysis by human graders, there is a potential that ophthalmic pathology may go unrecognized, therefore leading to delays in appropriate management and treatment. Following human re-grading of images, 17 patients were noted to have other pathology that required earlier evaluation compared to the automated screening recommendations, and were all non-urgent referrals (glaucoma suspect, mild hypertensive retinopathy, dry AMD). However, it is noteworthy that one inconclusive result involved a tractional retinal detachment requiring eventual surgery. Such a case illustrates the need for timely human interpretation of such inconclusive images or expeditious referral of patients with inconclusive screening results.

There are some notable caveats to our analysis. First, a subset of patients did not receive study results immediately, but rather within 2 weeks of the index visit. This delay in results reporting could have reduced adherence to follow-up eye examinations, as reported in a prior study.²⁶ In subgroup analysis, 42 participants received delayed results recommending further follow-up evaluation, 28 of which were adherent with this recommendation (66.7%). Though this potentially represents a higher rate of adherence than our overall observed average, the study was insufficiently powered to adequately test the difference between those who received immediate and delayed results.

Second, we encountered high rates of inconclusive screening results. In addition to pupillary miosis and media opacities, the environment in which retinal photographs were performed was not optimal. Photography was performed in a primary care clinic with ambient room lighting and by untrained personnel, to mimic the circumstances that we envision this screening tool to be deployed. Rates of successful image capture could have been improved with reduction in ambient lighting, training of camera operators, or administration of topical cycloplegia, which is safe for use in the primary care setting.

Third, a selection bias could have affected the interpretation of our results. Although all participants were adults with a diagnosis of diabetes recruited from the same clinic, the retrospective nature of the historical cohort inherently makes the direct comparison to the prospective cohort less than ideal. Whereas the historical cohort consisted of all patients consecutively seen at the primary care medicine clinic within a certain time period, patients in the prospective cohort were recruited for the study. Patients who consented to participate in the study may inherently be more likely to attend follow-up ophthalmic care appointments than patients who declined to receive ARIAS screenings as part of the study. A prospective, controlled study, in which participants were randomized to receive ARIAS point-of-care screening or referral for eye examination without a prescreening, would be better able to elucidate the difference in adherence rates after patients receive ARIAS screening. Furthermore, we did not assess baseline differences in tendency to be non-adherent with medical visits, as could have been estimated by non-attendance rate at diabetes clinic visits. Additionally, as shown in Table 1, the mean glycated hemoglobin index was higher in the ARIAS-screened group, compared to the control group, suggesting that there were demographic differences between the groups we studied. However, we expect that we underestimated the true effect of primary care ARIAS-based DR screening on follow-up eye care adherence since patients who were adherent with eye screening recommendations at the outset, and who therefore were more likely to have better glycemic control and continue to adhere to eye care screenings, were not included.

Lastly, although we determined that ARIAS-based screening improves the likelihood of patient adherence to follow-up eye care, we did not assess whether these screening systems improve visual outcomes in patients or change patterns of DR treatment among primary care clinic-referred patients.

Advances in non-mydriatic fundus photography and machine-learning based automated image analysis have the potential to provide accessible and accurate screening for DR. In our experience, the implementation of an automated DR screening system in a primary care clinic helped identify patients who needed urgent, in-office evaluation and reduced the number of unnecessary referrals in patients without retinal disease. Patients with positive or inconclusive ARIAS screening results were more likely to adhere to follow-up eye care recommendations compared to historical controls. These results suggest that automated screening for DR can be effectively implemented in the primary care setting, lead to improved access to ophthalmic care as well as eliminate unnecessary referrals and reduce the burden on the healthcare system. ARIAS programs may also lead to clinically and statistically significant changes in patient adherence, particularly in vulnerable patients who are at higher risk for DR. Longer-term studies are necessary to determine whether or not improved adherence to screening recommendations actually improves clinical outcomes.