Autonomous Artificial Intelligence in Diabetic Retinopathy Testing—Lessons Learned on Successful Health System Adoption

Abstract

Purpose

Artificial intelligence (AI)–aided diabetic retinopathy (DR) testing systems have been commercialized for 5 years, but adoption is still relatively limited. This article aims to summarize the evidence in clinical settings, describe the current state of adoption, and share themes of successful implementation.

Design

Evaluation of diagnostic test or technology.

Participants

Ophthalmologists.

Methods

We performed literature review and conducted interviews with ophthalmologists leading implementation of AI-aided DR testing programs at several academic health systems. The study focused on the 3 currently US Food and Drug Administration-cleared AI systems: LumineticsCore, EyeArt, and AEYE Diagnostic Screening (AEYE-DS), assessing their performance and strategies utilized by health systems to effectively implement this technology in clinics.

Main Outcome Measures

Diagnostic accuracy data, ophthalmologist feedback.

Results

The literature review found 6 publications reporting diagnostic accuracy data of autonomous AI DR testing in primary care office settings, including 5 for LumineticsCore and 1 for EyeArt. Additional articles, of which 18 were selected for detailed review, addressed impact on patient adherence, health equity, and carbon footprint, as well as cost-effectiveness and workflow efficiency analyses. There were no studies comparing the systems on the same patients. In aggregate, adopters of the AI systems reported average nonmydriatic gradability of 49% to 75% (n = 5), sensitivity 87% to 100% (n = 3), and specificity 60% to 91% (n = 4). Based on public records at the time of writing, both LumineticsCore and EyeArt have >5 academic adopters in the United States. Limited information is available on AEYE-DS given recency of regulatory clearance. Elements of successful implementation include proper site selection, aligning AI tools with primary care clinic workflows, streamlining patient engagement and referrals, and ongoing training of staff. Health systems utilizing AI reported improved Healthcare Effectiveness Data and Information Set measures, health equity, productivity, and patient adherence to follow-up with ophthalmology.

Conclusions

Artificial intelligence–aided diabetic eye examinations present a promising solution to facilitate early detection of DR, promote equitable access, and drive down system-level cost of care. Its successful implementation requires addressing technological, operational, and stakeholder engagement challenges. Our study underscores the potential of AI to revolutionize care delivery provided its adoption is strategically managed.

Financial Disclosure(s)

The author has no/the authors have no proprietary or commercial interest in any materials discussed in this article.

Artificial intelligence (AI) has emerged as a transformative technology across many industries, but its potential impact on health care is particularly significant. With its ability to process vast amounts of data, identify patterns, and make predictions, AI holds immense promise in reshaping the delivery of healthcare on multiple fronts, from improving patient access and care outcomes to boosting productivity and provider experience. While such potential has been increasingly recognized, adoption is still in its infancy with emerging but relatively limited evidence around efficacy in real clinical settings.

Imaging-based diagnostics is one of the earliest frontiers of AI development and has been cited by healthcare professionals as the most common application of AI solutions in health care today.¹ While the use of AI as a decision assistance tool is more commonly accepted across disciplines, ophthalmology was the first field in medicine to use autonomous AI in patient care to make diagnostic decisions independent of direct human input or supervision. In 2018, the US Food and Drug Administration (FDA) granted De Novo authorization to IDx-DR (now rebranded as LumineticsCore) (Digital Diagnostics Inc) as the first autonomous AI-based diagnostic system for diabetic retinopathy (DR) testing,² and the first ever FDA clearance for an autonomous AI algorithm. Two years later, another AI system, EyeArt (Eyenuk Inc), received 510(k) clearance on the LumineticsCore predicate for detecting more-than-mild DR (mtmDR) and vision-threatening DR.³ In late 2022, a third system, AEYE Diagnostic Screening (AEYE-DS), was also cleared through 510(k) using the same predicate. Today, DR remains the only indication in medicine where an AI system can provide an image analysis decision without clinician interpretation.

In the United States, DR is projected to affect 16 million people with diabetes by 2050.⁴ Between 12 000 and 24 000 individuals lose vision each year from DR,⁵ and >90% of vision loss can be avoided with early detection and treatment.⁶ The use of autonomous AI in DR testing offers several potential benefits over traditional methods, such as increasing access in underserved regions and underresourced clinics where eye care is not available, diagnosis without ophthalmology-trained medical personnel, real-time diagnosis while the patient is still with the provider, increased health equity,⁷ higher clinical productivity,⁸ and cost savings.⁸ While AI-aided testing systems hold promise for helping drive down the social and economic burden of DR, initial adoption of new technology is often limited by many challenges including lack of trust, resistance to change, training gaps, and uncertainty of the cost-benefit.⁹ The purpose of this article is to facilitate understanding of the evidence and experiences with AI-aided diagnostic systems in clinical settings in the United States, now >5 years after initial clearance. Specifically, we aim to:

• •Review published trial and evidence gathered from clinical implementations of the commercially available AI diagnostic systems for DR.
• •Describe the current state and scale of adoption, with emphasis on primary care clinics within academic health systems.
• •Share learnings from early adopters and synthesize themes of successful programs.

Method

Literature Review

A systematic search was conducted across electronic databases, including PubMed, Google Scholars, Scopus, and Web of Science. Keywords such as “artificial intelligence diabetic retinopathy evidence,” “autonomous eye exam,” “idx-dr,” “eyeart,” and “aeye-ds” were used to identify relevant articles. The search covered the period from January 2010 to October 2023. The initial search yielded 106 articles, from which duplicates were removed. Two independent reviewers (C.T. and S.P.) screened titles and abstracts for relevance, followed by full-text assessment. The inclusion was based on the presence of qualitative and quantitative data from adoption in the clinical setting, resulting in a final selection of 18 articles for detailed analysis.

Expert Interviews

To complement the findings from the literature, interviews were conducted with 6 ophthalmologists who lead implementation of AI DR testing programs at academic health systems. Semistructured interviews were conducted with the selected ophthalmologists. Our study adhered to the Declaration of Helsinki and informed consent was waived and was institutional review board exempt. The interviews, lasting approximately 60 min, cover topics such as experience with technology, implementation learnings and challenges, practice impact, and patient experience. All interviews were recorded and transcribed for further analysis. Thematic analysis was applied to the transcribed interviews to identify key insights and perspectives shared by the experts. The results were then compared with the findings from the literature review to provide a holistic understanding on the state of adoption in autonomous AI testing of DR.

FDA-Cleared AI Diagnostic Systems

Currently, there are 3 commercially available US FDA–cleared systems for DR testing: LumineticsCore (previously IDx-DR) by Digital Diagnostics, EyeArt by Eyenuk, Inc, and AEYE-DS by AEYE Health, Inc. While all these systems leverage AI to analyze digital fundus images for signs of DR, each has its unique features and advantages. To detect DR, some AI systems analyze digital fundus images to detect signs of DR such as microaneurysms, blot hemorrhages, hard exudates, cotton wool spots, intraretinal microvascular abnormalities, venous beading, neovascularization, vitreous or preretinal hemorrhages, fibrosis, and retinal detachments, while others utilize a deep convolutional neural network optimized for image classification trained using retrospective data sets.^10,11

LumineticsCore (IDx-DR)

Digital Diagnostics is the first autonomous AI company to receive the US FDA De Novo pathway clearance for DR testing. Their product, LumineticsCore (originally known as IDx-DR), is an AI diagnostic system designed to autonomously diagnose patients for mtmDR, central-involved diabetic macular edema traditionally diagnosed with OCT, and clinically significant diabetic macular edema diagnosed with fundus photographs, using images acquired with a Topcon TRC-NW400 camera (Topcon). Macula- and disc-centered color fundus images (2 images per eye), assisted by AI for quality control, are taken with a 45° field of view as input to the algorithm. LumineticsCore diagnoses according to the ETDRS and Diabetic Retinopathy Clinical Research network grading system to detect ETDRS level 35 or higher (mild or moderate DR depending on the classification system) or central-involved diabetic macular edema or clinically significant diabetic macular edema.¹² The system, which is directly connected to the camera, interfaces directly with all available electronic medical record (EMR) for ordering, billing, and resulting directly.¹³

In a study by Abramoff et al in 2016, an earlier version of LumineticsCore reported a referable DR sensitivity of 96.8% and specificity of 87.0% against the Messidor-2 database. However, this study remained limited given its retrospective nature and therefore could not demonstrate the system’s clinical potential.¹⁴ However, in 2018, Abramoff et al¹⁵ published a preregistered prospective clinical trial, a first of its kind in any field of medicine, with n = 900 fully screened patients in primary care by operators without previous experience reporting a sensitivity of 87.2%, specificity of 90.7%, and gradability of 96.1% for referrable DR, developed under a strict ethical framework.^16,17 Of note, the statistics are calculated against a prognostic level 1 reference standard, meaning that rather than compare the performance against clinicians, it was compared against diagnostic modalities that include OCT imaging for central-involved diabetic macular edema. This pivotal study illuminated the feasibility of implementing AI in clinical spaces and played a role in informing the FDA's decision to clear IDx-DR.

In April 2018, the US FDA cleared the first generation of IDx-DR through the De Novo premarket review pathway as a class II medical device.² Two years later, Digital Diagnostics filed a 510(k) application for the current version of LumineticsCore based on the previous IDx-DR as predicate device, which was subsequently cleared in May 2021, with the same De Novo special controls in place, which now apply to all 510(k)s.¹⁸ A few key technological updates that were made include option of Digital Imaging and Communications in Medicine image submission, incorporation of a guided workflow, local image retention, training mode, in-examination image quality feedback, and allows the user to resubmit images when applicable. Earlier in September 2020, the American Medical Association Current Procedural Terminology (CPT) Editorial Panel created code 92229 for retinal imaging with autonomous point-of-care interpretation, which allowed institutions that chose to adopt AI-guided DR testing to be billed for the procedure.¹⁹ This was the first CPT code for autonomous AI in any specialty.²⁰ Based on the claims database in 2023, DR testing has become one of the most widely adopted and fastest-growing AI-aided medical procedures in the United States, only second to coronary artery disease.²¹ LumineticsCore was also the first autonomous AI testing system to secure Medicare²² and Medicaid reimbursement,²³ and qualify for Healthcare Effectiveness Data and Information Set (HEDIS) care gap closure²⁴ and Merit-based Incentive Payment System care gap closure.²⁵ In 2022, Centers for Medicare and Medicaid Services announced that they would be reimbursing LumineticsCore procedures under the physician fee schedule, and private payers soon followed. In randomized clinical trials, this system has shown to improve health equity in minorities²⁶ and physician productivity,⁸ as well as demonstrated safety, efficacy, and equity in youth with diabetes.⁷ Today, LumineticsCore is utilized in >1000 US clinical sites spanning both academic health systems and community-based health organizations including Stanford Medicine, Johns Hopkins Health System, Cahaba Medical Care, and LabCorp, with expansion to sites outside the United States as well.

EyeArt

Eyenuk, Inc is a developer of clinically supported AI solutions aimed at identifying diseases through retinal image analysis using several camera models including the Canon CR-2 AF, Canon CR-2 Plus AF, and more recently, the Topcon NW400.²⁷ Of note, they also offer a “spellcheck”-like, assistive-AI function for ophthalmologists who read these images for other diseases such as glaucoma and age-related macular degeneration, which are not currently FDA-cleared for autonomous diagnosis. Input specs are identical to LumineticsCore, that is, macula- and disc-centered color fundus images with 45° field of view. In contrast, the EyeArt system grades images based on the International Clinical DR classification scale and also looks for evidence of macular edema. The International Clinical DR classification scale accounts for lesions associated with DR to categorize a patient into one of 5 categories: no DR, mild nonproliferative DR, moderate nonproliferative DR, severe nonproliferative DR, and proliferative DR.²⁸ The output from the EyeArt AI algorithm differentiates between “nonreferable” (no or mild retinopathy and no evidence of macular edema) and “referable” (mtmDR and vision-threatening DR). The purported benefit is to assign time-sensitivity to the patients with vision-threatening DR when managing referrals.

Eyenuk’s commercial launch of EyeArt began in the European Union in 2016 after receiving a Conformité Européenne Mark, when the system was only approved for investigational use in the United States.29, 30, 31 Institutions such as the Diabetes Center Mergentheim, Tübingen University Hospital, and >25 centers in Italy have adopted this technology. In 2018, Health Canada approved EyeArt, allowing its sale in Canada.³² More recently, in 2020, the system received FDA clearance under the 510(k) pathway using IDx-DR as the predicate device, and began its expansion into the United States. To date, EyeArt has been utilized in the United States by institutions such as Temple Health, University of Utah Health, and Cedars Sinai Health System.

Importantly, EyeArt is unique from their similar product, EyeScreen. While EyeArt autonomously generates a report on DR within 1 minute, the EyeScreen program integrates both an AI system and specialist human graders to assess retinal images. The process includes independent assessments by AI and human graders, followed by an International Classification of Diseases, Tenth Revision-compliant report to the physician. In cases where the AI and human assessments differ, adjudication is completed by a trained human expert. EyeScreen also enables the detection of other eye conditions such as macular degeneration and glaucoma through human grading.

A study in 2018 utilized the EyeArt testing software on retinal images captured with the Remidio Fundus on Phone system. This system, an FDA 510(k)–registered fundus camera used in a study by Rajalakshmi et al, combines a smartphone with patented optics to showcase the software's sensitivity in detecting both DR and sight-threatening DR. This indicated the potential of EyeArt in facilitating highly sensitive yet cost-effective mass retinal screening in the future.³³ A pivotal retrospective study from 2019 performed by Eyenuk with >100 000 patients tested by the software demonstrated a 91.3% sensitivity and 91.1% specificity. These findings, along with its use in the European Union and Canada, bolstered the credibility that EyeArt could be a clinically utilized AI-guided DR testing program in the United States.³⁴ A study in 2023 by Najac et al assessed the feasibility of telemedicine diabetic eye disease testing in a point-of-care general practice setting. Retinal images were acquired using a nonmydriatic camera. These images were then automatically graded using the commercially available AI algorithm, EyeArt version 2.1.0 by Eyenuk Inc. Their study found that the overall image quality achieved in this telemedical general practicioner-based diabetic eye disease testing was sufficient and would be accepted by both medical assistants and patients in most cases. However, the study also highlighted challenges in ensuring good image quality and integrating the system into the existing workflow.^35,36

AEYE-DS

AEYE Health's primary product for DR testing is the AEYE-DS which also has the ability to diagnose DR from retinal images using either the Topcon NW400 or Optomed Aurora IQ AEYE, a handheld product restricted to investigational-use only. Similar to the LumineticsCore system, they detect mtmDR, equating to an ETDRS level of ≥35.

AEYE reports sensitivity of 93.0% and a specificity of 91.4% on desktop cameras in their clinical trials. One of the differentiating features of the AEYE-DS system is its reduced workflow requirements. The system requires only 1 image per eye (ie, macula-centered color fundus images with ≥45° field of view) for diagnosis and requires dilation in <1% of patients.³⁷ Though the enhanced efficiency and potential pitfalls in clinical workflows of taking 1 image versus 2 images per eye have yet to be studied, the accuracy and low dilation requirement distinguish the AEYE-DS system from the other 2 commercially available systems.

In October 2022, AEYE-DS received FDA clearance under the 510(k) pathway using IDx-DR as the predicate device. To our best knowledge, AEYE-DS has been implemented in select community-based health organizations in the United States, such as the Bethesda Health Clinic. Given the recency of commercialization, limited literature to describe the experience of this system in practice, and lack of representative academic health system adopter at the time of this review, fewer implementation details are available for AEYE-DS compared with the other 2 systems in subsequent sections.

Emerging Themes from Early Adoption

Ophthalmology Departments Are Taking the Lead

Academic health systems have been the earliest adopters of autonomous AI testing. Based on publicly available information (from press release, academic publications), both LumineticsCore and EyeArt each have around 5 academic adopters in the United States today (Table 1, Table 2). Universally, leaders in ophthalmology departments have leaned into the effort to implement these systems. These departments also contribute numerous peer-reviewed articles and abstracts to describe their practical experience since 2018 (Table 3). On this front, leadership by ophthalmology has the additional benefit of breeding future expansion of this technology into other ocular diseases, such as age-related macular degeneration and glaucoma.

Table 1.

Evidence on Autonomous AI Testing Systems for DR from Clinical Implementations

Institution	AI System	Published Evidence		Gradability (mydriatic)	Sensitivity	Specificity	Positive Predictive Value	Negative Predictive Value	Ref.
		Sample size (n)	Gradability (nonmydriatic)
University of Iowa	LumineticsCore	892	73.4%	96.1%	87.2%	90.7%	76%	96%	15
Mayo Clinic	LumineticsCore	1052	55.1%	91.7%	100%	89.2%	27.5%	100%	40
Johns Hopkins University	LumineticsCore	241	49%	N/A	N/A	N/A	N/A	N/A	45
Stanford University	LumineticsCore	80	71%	N/A	96%	60%	48%	97%	46
Temple University	EyeArt	260	75%	N/A	100%	78%	19%	100%	41

AI = artificial intelligence; DR = diabetic retinopathy; N/A = not applicable.

Table 2.

Current Adopters of Autonomous AI Testing Systems for DR (Nonexhaustive)

Institution	Type	Location	AI System	Time of Adoption	Initial Scale of Adoption
Academic Adopters
University of Iowa	AMC	Coralville, Iowa	LumineticsCore	April, 2018	Diabetes and Endocrinology Center at UI Health Care–Iowa River Landing
Mayo Clinic	AMC	Minnesota	LumineticsCore	September, 2019	Mayo clinic downtown primary care center
Johns Hopkins University	AMC	Baltimore, Maryland	LumineticsCore	August, 2020	4 primary care sites
Temple University	AMC	Philadelphia, Pennsylvania	EyeArt	October, 2020	Full system adoption
Stanford University	AMC	Northern California	LumineticsCore	February, 2021	9 primary care sites
University of Illinois at Chicago	AMC	Chicago, Illinois	EyeArt	N/A	N/A
Doheny Eye Institute	AMC	Southern California	EyeArt	N/A	N/A
Cedars-Sinai Medical Center	AMC	Southern California	EyeArt	N/A	N/A
University of Pennsylvania	AMC	Philadelphia, PA	LumineticsCore	January 2024	N/A
Nonacademic adopters
Cahaba Medical Care	FQHC	Central Alabama	LumineticsCore	April, 2021	6 primary care clinics
ArchWell Health	Senior clinic	N/A	LumineticsCore	July, 2022	N/A
OSF Healthcare	IDN	Peoria, Illinois	LumineticsCore	July, 2022	8 primary care sites (with plan to expand to 24)
Tarzana Treatment Centers	FQHC	Southern California	LumineticsCore	September, 2022	6 primary care clinics
LabCorp	Lab service	Alabama	LumineticsCore	April, 2023	9 patient service centers

AI = artificial intelligence; AMC = Academic Medical Center; DR = diabetic retinopathy; FQHC = Federally Qualified Health Center; IDN = Integrated Delivery Network; N/A = not applicable.

Table 3.

Publications on Autonomous AI Testing of DR 2018–2024

Publication	Year	Author Affiliation	Key Results/Conclusion	Ref.
Diagnostic Accuracy of a Device for the Automated Detection of Diabetic Retinopathy in a Primary Care Setting	2019	University of Iowa	Among 1616 diabetic patients, hybrid deep learning–enhanced device’s sensitivity or specificity against the reference standard was, respectively, for vtDR 100%/97.8% and for mtmDR 79.4%/93.8%.	38
Diabetic Retinopathy Screening with Automated Retinal Image Analysis in a Primary Care Setting Improves Adherence to Ophthalmic Care	2020	Washington University in St. Louis	Among the 180 patients with diabetes, LumineticsCore demonstrated sensitivity of 100% and specificity of 65.7%. Among the patients referred for follow-up ophthalmic evaluation, the adherence rate was 55.4% at 1 year compared with the historical adherence rate of 18.7%.	39
The SEE Study: Safety, Efficacy, and Equity of Implementing Autonomous Artificial Intelligence for Diagnosing Diabetic Retinopathy in Youth Diabetes Care	2021	Johns Hopkins University	Among the 310 youth with diabetes aged 5-21, AI diagnosability was 97.5%, sensitivity 85.7%, and specificity 79.3% compared with the reference standard. Adherence improved from 49% to 95% after AI implementation.	7
Diabetic Retinopathy Telemedicine Outcomes With Artificial Intelligence-Based Image Analysis, Reflex Dilation, and Image Overread	2022	Mayo Clinic	Among the 1052 patients with diabetes, LumineticsCore demonstrated nonmydriatic gradability of 55.1%, overall gradability of 91.7%, sensitivity 100%, specificity 89.2%, PPV 27.5%, and NPV 100%. Image gradeability was inversely related to patient age.	40
A Comparison of Artificial Intelligence and Human Diabetic Retinal Image Interpretation in an Urban Health System	2022	Temple University	Among the 260 patients with diabetes, EyeArt demonstrated 100% sensitivity, 77.78% specificity, 19.15% PPV, and 100% NPV. Old age (>60) is associated with ungradable image.	41
Five-Year Cost-Effectiveness Modeling of Primary Care-Based, Nonmydriatic Automated Retinal Image Analysis Screening Among Low-Income Patients With Diabetes	2022	Washington University in St. Louis	Primary care-based autonomous AI DR screening is cost-effective when compared with standard of care screening methods. A Markov model of cost-effective analysis showed at 5 years, autonomous AI-based screening reduced costs by 23.3%, with an incremental cost-utility ratio (ICUR) of $258 721.81 comparing to current practice.	42
Potential reduction in healthcare carbon footprint by autonomous artificial intelligence	2022	Johns Hopkins University	The authors compared the marginal GHG contribution of an encounter performed by an autonomous AI to that of an in-person specialist encounter. Results show that an 80% reduction of health care GHG emissions may be achievable with autonomous AI.	43
Clinical Implementation of Autonomous Artificial Intelligence Systems for Diabetic Eye Exams: Considerations for Success	2023	Johns Hopkins University	The authors share experience and strategies for success from both the pediatric and adult care perspectives, from implementation experience at an integrated health care system, including tips on key stakeholders, camera setup, patient imaging, workflow, and billing.	44
Risk Factors for Nondiagnostic Imaging in a Real-World Deployment of Artificial Intelligence Diabetic Retinal Examinations in an Integrated Health care System: Maximizing Workflow Efficiency Through Predictive Dilation	2023	Johns Hopkins University	Type 1 diabetes, smoking, and age were associated with nondiagnostic results in a multivariable analysis of 241 patients. The authors created a predictive model using T1D, smoking, age, race, sex, and hypertension as inputs. The model showed an area under the receiver operating characteristic (ROC) curve of 0.76 in fivefold cross-validation, suggesting potential to implement a predicative dilation protocol.	45
AI-Human Hybrid Workflow Enhances Teleophthalmology for the Detection of Diabetic Retinopathy	2023	Stanford University	A 2-step AI-human hybrid workflow (i.e., AI algorithm initially rendered an assessment followed by overread by a retina specialist of mtmDR-positive encounters) lead to improvement of gradability (63.5%–95.6%) and specificity (60.3%–98.2%)	46
Artificial Intelligence Improves Patient Follow-Up in a Diabetic Retinopathy Screening Program	2023	Stanford University	In a sample of 2243 adult patients, those who screened positive for mtmDR under the AI workflow (results within 48 hrs) were 3 times more likely to follow up compared with those screened positive under the human workflow or the 2-step AI–human hybrid workflow (results in 7 days).	47
Autonomous artificial intelligence increases real-world specialist clinic productivity in a cluster-randomized trial	2023	University of Iowa	Cluster-randomized trial of 105 clinic days demonstrated that AI leads to 40% higher productivity (1.59 encounters/hour) than control (1.14 encounters/hour).	8
Autonomous artificial intelligence increases screening and follow-up for diabetic retinopathy in youth: the ACCESS randomized control trial	2024	Johns Hopkins University	Autonomous AI increases diabetic eye examination completion rates in youth with diabetes: completion rate was 100% in intervention group (autonomous AI examination at point of care) with 64% follow-through rate for abnormal result; vs in control group (scripted eye care provider referral and education), examination completion rate was 22%.	26
Abstracts
Autonomous artificial intelligence exams are associated with higher adherence to diabetic retinopathy testing in an integrated healthcare system	2023	Johns Hopkins University	Among the 22 263 patients, adherence was higher in patients in the AI group (64.0%) than in the non-AI group (46.4%). Adherent patients are more likely to be female (52.2% vs 50.6%), have higher mean age (59.8 vs 56.6 years), higher median adjusted clinical group (ACG) value (0.045 vs 0.035), and lower mean hemoglobin A1c value (7.1% vs 7.3%).	48
Identifying Causes of Ungradable Fundus Photos in an Artificial Intelligence Assisted Screening Program for Diabetic Retinopathy	2023	Temple University	Among the 60 patients deemed ungradable by the AI examination, follow-up slit lamp examination showed 97% were afflicted with ≥1 ocular pathology: 77% cataracts, 38% dry eye disease, 23% DR, 17% glaucoma, glaucoma suspect or ocular hypertension, 8% hypertensive retinopathy, 5% epiretinal membrane, and 3% lattice degeneration.	49
Evaluating Follow-Up Metrics in an AI-Assisted Telemedicine Screening Program for Diabetic Retinopathy in Primary Care Clinics After Hiring a Patient Care Navigator	2023	Temple University	Hiring a patient care navigator significantly improved the of rate of patients scheduling (57% vs 30%) and completing (35% vs 26%) a follow-up, as well as times to scheduling (8 vs 46 days) and completing the follow-up (67 vs 158 days).	50
Autonomous Artificial Intelligence (AI) Testing for Diabetic Eye Disease (DED) Closes Care Gap and Improves Health Equity on a Systems Level	2023	Johns Hopkins University	From 2019 to 2021, the overall adherence rate increased from 42.6% to 55.5% at AI sites (1949 patients) and increased from 38.0% to 41.0% at non-AI sites (5379 patients). The increase in overall adherence rate at AI sites was significantly greater than that at non-AI sites.	51
Autonomous artificial intelligence (AI) increases health equity for patients who are more at risk for poor visual outcomes due to diabetic eye disease (DED)	2023	Johns Hopkins University	Patients who underwent autonomous AI DED testing had higher systemic disease burden and corresponding higher risk for DED. Retrospective analysis was performed on 3745 patients (3352 in standard of care and 393 in AI group) The AI group was more likely to be Black (64.9% vs 44.4%), have higher Medicare coverage (39.4% vs 30.9%), and have higher systemic disease burden: hypertension (P = 0.001) and chronic kidney disease (P = 0.017)	52

AI = artificial intelligence; DED = diabetic eye disease; DR = diabetic retinopathy; GHG = greenhouse gas; mtmDR = more-than-mild diabetic retinopathy; NPV = negative predictive value; PPV = positive predictive value; T1D = type 1 diabetes; vtDR = vision-threatening diabetic retinopathy.

It is worth noting that AI is not a standalone, but rather a component in the overall design of a testing program for eye diseases. Hence, the decision to implement should be based upon an evaluation of the health system’s infrastructure, resources, and culture to ensure institutional readiness to benefit from the introduction of AI. In the following paragraphs, we aim to understand what makes an AI-guided testing program successful, with sustained and recorded improvement on disease capture followed by treatment initiation, and what challenges to anticipate as a prospective adopter.

Stakeholder Alignment Is Key, Most Effectively Driven from Top-Down

There are 3 main steps in a patient’s journey in an autonomous AI testing program: taking a fundus photo at primary care office (or endocrinology in select cases) with a point-of-care result, referral to ophthalmology if AI returns a positive result, and the follow-up with ophthalmology. Based on our interviews, the 6 academic ophthalmology programs agreed that buy-in from the primary care team is critical for initiating utilization of AI examinations. In order to roll out autonomous AI testing, the offices need to identify the right patients to apply testing, set up and maintain the camera, add photography to the workflow of a patient visit, allocate resources to perform relevant tasks, and provide appropriate training to employees.⁵³ Thoughtful planning, administrative oversight, and logistics support in each of these steps are critical to drive the initiative forward and prevent future frictions.

Based on our interviews, we have observed the most success with implementation of AI-aided DR detection when it is driven from top-down at the health system leadership level. For instance, 2 hospitals became interested in autonomous AI testing of DR as an effort to improve primary care HEDIS scores. With this objective articulated and priorities set, implementation followed as a coordinated effort between ophthalmology and primary care. In contrast, when a health system pursued implementation in a research context without top-down support, research staff visited primary care offices and tested patients without disrupting the regular visit workflow. Upon conclusion of the research study, the camera was returned to the manufacturer and the system was not implemented.

Adopters have found various levels of challenges in stakeholder alignment, largely depending on the historic interdepartmental relationship. Another potential solution here from a governance standpoint is an ophthalmology-primary care co-ownership model from the start, which could enforce collaboration and shared accountability.

There Are Clear Value Propositions for Health Systems

When executed well, an AI-aided DR testing program can create value and efficiency for the health system in the following ways:

Improve Quality Measure

Retinal eye examination is one of the 6 HEDIS measures for patients with diabetes.⁵⁴ Previously, this required patients to schedule and attend a second appointment with ophthalmology after receiving a referral from primary care. Historically, this is a process with <50% compliance rate.⁵⁵ With autonomous AI, family practitioners can now capture patients at their primary care appointments and perform retinal eye examinations before they leave the door. Johns Hopkins University found the percentage of diabetes patients who received annual eye examinations increased from 46.4% in nontesting clinics to 64% in AI-testing clinics in 2021.⁴⁸ Stanford University already had a telemedicine screening program before introducing AI but achieved an additional 8% increase in annual examination adherence rate (65.2%–72.8%) in AI-clinics.⁵⁶ One shall note that the increase in screening rates is a result of the evolution in screening process unlocked by the AI technology, rather than the software itself. It is critical for quality data to be consistently tracked to evaluate the performance of the program. Under pay-for-performance models such as pay-for-quality and value-based care, providers may receive additional revenue as HEDIS rates increase, which may be allocated to departments to drive aligned incentives.

Encourage Patient Adherence

The goal of early testing is to get patients with disease to ophthalmology clinics and start treatment as early as possible to improve outcomes. In the traditional model, one of the challenges is that a considerable proportion of patients fail to follow up after being referred for ophthalmology assessment. It has been proposed that AI testing could improve adherence: by providing results at the point-of-care, primary care physicians have the opportunity to explain the results to patients, who are likely to be more motivated to engage with greater sense of urgency. In the same study as previously mentioned, Stanford University found that those referred after an AI-based examination followed up at the eye institute at a rate nearly 3 times higher than those referred after teleophthalmology-only testing.⁴² Liu et al³⁹ found that among 180 study participants who underwent nonmydriatic fundus photography followed by automated retinal image analysis with human supervision, those referred for follow-up ophthalmic evaluation had an adherence rate of 55.4% at 1 year, compared with the historical adherence rate of 18.7%. If the scheduling system is fully integrated into the EMR, primary care can schedule the ophthalmology appointment before the patient leaves. This will not only drive internal referral and patient retention in the health system but also reduce the workload to call patients for scheduling on the back end. From the patients’ perspective, there is also improved access as this eliminates the need to call to schedule and allows the option to find earlier appointments for more urgent cases.

AI Examination Reimbursement

Providers are reimbursed for autonomous AI eye examinations under CPT code 92229 created in 2019 by Centers for Medicare and Medicaid Services: “Imaging of retina for detection or monitoring of disease; point-of-care automated analysis and report, unilateral or bilateral.” In 2021, Centers for Medicare and Medicaid Services established relative value units for this service and national payment rate.⁵⁷ Based on the Medicare Physician Fee Schedule look-up tool, the national payment amount was $47.06 in 2022, $45.74 in 2023, and $40.28 in 2024.⁵⁸ Following Medicare and Medicaid, a growing number of commercial payers established reimbursement. Interestingly, the department collecting the payment varies across health systems. Among the adopters we interviewed with, about half attribute payments to the primary care department, and the other half to ophthalmology. While not identified today, a model of payment split between the 2 departments in a health system could be considered.

Optimize Allocation of Resources

Prior to autonomous AI testing, all patients were referred to ophthalmology for an annual eye examination.⁵⁹ Particularly at health systems with large patient volume, this creates long wait times for an appointment that could lead to either the patient being lost to follow-up or seeking care elsewhere. When autonomous AI performs the first round of testing and refers only the patients with positive or ungradable examinations, this allows ophthalmologists to focus attention on those who are most likely to have disease and improve patient access to appointments and treatment. Compared to a human-interpretation model (Fig 1, Workflow 2), this also frees ophthalmologists from the time commitment of image batch reads and focuses on in-person patient care. Anecdotally, ophthalmologists have observed that patients referred to clinics are more likely to be disease-positive without change in overall volume after AI program roll-out. However, there has been limited data to prove this observation.

Figure 1.

Alternative workflow designs for DR testing with illustrative patient volume. AI = artificial intelligence; DR = diabetic retinopathy; EMR = electronic medical record.

Pilot before Scale

Among the adopters we interviewed with, the majority launched the autonomous AI program through a pilot in 4 to 6 primary care sites out of the 30+ locations in their university health network. An alternative single-pilot model was also reported to work well when there is one location that serves a significant number of patients with diabetes in the network, either an endocrinology office or a consolidated network of primary care offices. A key observation here is that the level of receptiveness varies from site to site depending on the preceding resource and culture. Hence, selectively partnering with the locations that are most setup for success is critical. A pilot also affords the opportunity to gather feedback, refine processes, and build confidence across the network to prepare for scaling. Among the early adopters, 2 that started with a pilot have increased the number of clinics using AI testing by 50% to 100% over a 2-year period despite the impact of coronavirus disease. While these expansions all took place reactively based on the success of the pilot, the authors believe that with sufficient proof-of-concept, a preplanned phased approach to full system adoption may work better to quickly achieve scale.

Redesigning Clinic and Referral Workflow

There are 2 main archetypes of adopters, those that had an existing human-read DR testing program in place at the time of introducing AI and those that did not. Without a screening program, all patients diagnosed with type 1 diabetic mellitus and type 2 diabetic mellitus are referred for an annual eye examination;⁴⁶ some require patients to call clinics to make appointments themselves, whereas some send a referral note on EMR to the ophthalmology department, which then calls patients for scheduling (Fig 1, Workflow 1). When a teleophthalmology screening program (without AI) is in place, staff at the primary care office, most often a medical assistant and sometimes a lab technician, takes fundus photos either before or at the end of the visit. Alternatively, some clinics treat the camera as a “provider” and schedule patients for the camera. The pictures are then uploaded to the patient’s record and sent to the on-point ophthalmologist or optometrist for batch reading. For patients who tested positive, the ophthalmology office would reach out to patients to schedule follow-up (Fig 1, Workflow 2).

When autonomous AI is introduced, the process of acquiring fundus photos remains the same as that for the teleophthalmology screening program. The difference is that the acquired image is analyzed by the AI algorithm, which returns the testing result within a few minutes. Prior to patient encounter, the clinic also needs to decide its specific workflow, such as who should explain the test result to the patient and how to approach scheduling follow-up for further ophthalmic care. Generally, the image and test report are uploaded to the patient's EMR (Fig 1, Workflow 3). Artificial intelligence algorithms have demonstrated a high sensitivity of 95% to 100% in clinical implementation, whereas specificity is relatively lower and with greater variation at 60% to 90%.^38,40,41,46 To improve the specificity of referral, Mayo Clinic and Stanford adopted an AI-human hybrid workflow (Fig 1, Workflow 4). Under this model, images with positive AI testing were sent to ophthalmologists to be overread; only the positive readings by the ophthalmologist would be referred for follow-up, whereas the others return to routine annual testing. Stanford’s experience with this hybrid system was reported by Dow et al, who found that the AI algorithm is more sensitive than remote image interpretation by a retinal specialist (95.5% vs 69.5%) but had lower specificity (60.3% vs 96.9%). Combining both the AI and human interpretation into a hybrid workflow as described earlier, this group found improvement of testing sensitivity to 95.5% and specificity to 98.2%.⁴⁸

On the back end, the workflow changes are enabled by the information technology team, which plays a crucial role in supporting camera setup and integration of the AI system with EMR. Connecting the local information technology team with the vendor information technology team was also found to be helpful. Wolf et al⁴⁴ shared additional pointers for successful implementation. In terms of staffing, 80% of the health systems did not add dedicated full-time equivalent to launch the AI program, although more than half of adopters we interviewed expressed “wish we could.” Nelson et al⁵⁰ found it helpful to add an additional camera supervisor and patient care navigator. The clinics should evaluate their need and ability to add resources and perform cost-benefit analysis.

Management of AI-Ungradable Images

Image quality in fundus photography is affected by a number of factors, such as pupil size, patient positioning and compliance, and media opacities such as cataracts. In the pivotal trial that led to the clearance of LumineticsCore, 76.4% of participants received results without pharmacologic dilation, and an additional 23.6% became gradable with dilation.¹⁵ In practice, the nonmydriatic gradability is found to range from 49% to 75% (Table 1). Some patient factors correlated to nondiagnostic image include type 1 diabetic mellitus, smoking, old age, and cataracts.^45,49 While multiple studies have shown that diagnostic rates can be increased by a reflexive dilation protocol,^60,61 utilization has been found to be challenging. Under this protocol, when AI is unable to grade the initial image, the operator performs pharmacologic dilation and takes a second image. This introduces another step in the primary care workflow in addition to a small patient risk,⁶² requiring the operator to be comfortable performing the procedure and understand relevant contraindications, as well as accommodation for added visit time and ensuring patients have transportation means. Mayo Clinic successfully implemented reflex dilation at a single site and did the following: when a patient receives an ungradable result, the technician sends an order for the patient to receive dilating eye drops from a registered nurse. The patient would then return to the photography room for a second image. Of note, one of the exclusion criteria for autonomous AI screening at this institution is that the patient must have had a prior dilated eye examination because once a nonmydriatic image has been taken, the patient automatically enters this protocol to receive pharmacologic dilation.

As an alternative to reflex dilation, Shou et al proposed a predictive dilation model using type 1 diabetes, smoking, age, race, sex, and hypertension as inputs in a retrospective review of 241 patients. The model demonstrated an area under the receiver operating characteristics curve of 0.76, with sensitivity of 77% and specificity of 68% at 50% cutoff value.⁴⁵ While its feasibility and efficacy in practice have not been proven in a prospective manner, this model could be a potential solution to simplify workflow and improve patient satisfaction. Nevertheless, it should first be optimized using a pooled database that informs a rigorous understanding of factors associated with ungradable results.

Of note, at Stanford, instead of reflexive dilation, a hybrid workflow is used where any AI-ungradable images are routed to a retina specialist at the Stanford Reading Center, where the images are subsequently read. With this process, the overall gradability of nonmydriatic images increases to 95.6% of encounters, as the human expert graders in this system are experienced retina specialists and have greater capacity for interpreting lower quality nonmydriatic images than the AI is currently programmed to accept.⁴⁶

Business Model Selection

According to current adopters, there are generally 2 business models offered by Digital Diagnostics and Eyenuk (with limited information on AEYE). First is a click fee model, where providers pay the company a fixed amount for each image interpreted. Second is a subscription model, where the company charges a monthly fee for all interpretations regardless of volume. If a health system already owns one of the compatible cameras, the AI interpretation service can be offered independently. If not, these companies also offer camera leasing options.

Multiple Routes to Funding

Every program launch and expansion require funding to install fundus cameras and pay for a click fee or monthly subscription. Three main sources of funding today are: (1) direct funding provided at the health system level; (2) industry grant applications, such as one offered by Regeneron; and (3) partnership with one of the AI testing companies.

Given the cost-effectiveness of autonomous AI testing and the potential to reduce health care disparities, we could envision a funding program from public payers in the future. In Singapore, it is estimated that autonomous testing costs $11-15 less per patient per year, leading to an annual national savings of $15 million by 2050.⁶³ Similar cost-effectiveness has been demonstrated in Europe.⁶⁴ In the United States, Fuller et al zoomed in on a local low-income population and built a 5-year cost-effective model, which demonstrated 23.3% cost savings using Automated Retinal Image Analysis System–based DR testing in a primary care medicine clinic. Future analysis on the national level may help generate awareness and accelerate adoption through alternative funding sources.

Other Considerations

Performance Management

Consistent data tracking and review are critically important for evaluating the program's effectiveness and strategizing for future implementations. The authors recommend that the following metrics be considered:

• -Number or percentage of patients with diabetes who received annual eye examination (HEDIS measure).
• -Gradability, sensitivity, and specificity of AI testing.
• -Number or percentage of patients with positive tests that followed up with ophthalmology.
• -Presence and classification of pathology among patients who followed up with ophthalmology.
• -Overall cost to serve diabetes population in network by ophthalmology.

System Upgrades and Data Rights

Under the FDA’s traditional framework for regulating hardware-based medical devices, AI algorithms are cleared as fixed products at the time of regulatory submission. Digital Diagnostics, Eyenuk, and AEYE Health all own a web service module that houses the diagnostic images, but it seems unclear who should be acquiring the consent to access these images and improve the AI algorithm with the growing database. A protocol is also required to mitigate the risk of algorithm changes leading to changes in the device’s technical specifications, and future algorithm improvements may trigger the requirement for a 510(k) premarket notification submission under guidelines for software changes published in 2017.⁶⁵ Meanwhile, the FDA recognizes the increasing use case of Software as Medical Device and that the traditional approach is not well-suited for the faster iterative design and type of validation used for software device functions. In September 2022, the FDA completed the Software Precertification Pilot Program, which was started in 2017, aimed to explore innovative approaches to regulatory oversight of medical device software.⁶⁶ A new regulatory paradigm could be on the horizon, but not without challenges, as it would require legislative changes. Health systems, software companies, and regulatory agencies need to work together to unlock faster development while protecting patient information and rights, with elucidation of the ownership of data, authorization to access, and purpose for use.

Adoption by Other Entities

Recently, the adoption of autonomous AI testing has expanded to provider organizations beyond academic health systems, the most notable being the Federally Qualified Health Centers (FQHCs). Federally Qualified Health Centers are federally funded nonprofit health centers that serve medically underserved areas and populations. They provide primary care services regardless of patients’ abilities to pay or health insurance status. Given the lack of access to specialist resources among the vulnerable population served by FQHCs, autonomous AI testing can deliver exponential value in reducing the disparity in access and improving outcomes. Whereas some large FQHCs like Cahaba Medical Care and Tarzana Treatment Center worked directly with Digital Diagnostics, small clinics can also participate. For instance, the University of Utah and Temple Health both partnered with a network of small FQHCs in the region to provide them with autonomous AI testing service, under a single subscription-based contract with Eyenuk to achieve economies of scale. Another benefit of this model is the potential to streamline referrals from FQHCs to ophthalmology offices for patients who tested positive.

Another notable adopter on the horizon is LabCorp, who launched DR examinations with LumineticsCore at 9 locations in Alabama in April 2023. While the success of the implementation has yet to be reported, this could signal an upcoming paradigm shift in the way DR testing is delivered, from a specialist visit to the ophthalmology office to a routine and noninvasive lab test.

Conclusion

Amidst rising DR prevalence, autonomous AI testing presents a promising solution to facilitate early detection, promote equitable access, and drive down system-level cost of care.^42,48,67 Among the 3 commercially available US FDA–cleared systems, both LumineticsCore and EyeArt have been implemented outside of trial conditions for 3 to 5 years with adequate levels of imageability and accuracy reported. Payment for the point-of-care diagnostic service is currently covered under CPT code 92229, and additional quality performance incentives may be addressable under select provider contexts. Future adopters should strategically manage critical drivers to implementation success, including stakeholder alignment across primary care and ophthalmology departments, workflow adaptation, site selection, and performance tracking to measure impact.

Manuscript no. XOPS-D-24-00284