Implementation of deep learning artificial intelligence in vision-threatening disease screenings for an underserved community during COVID-19

Abstract

Introduction

Age-related macular degeneration, diabetic retinopathy, and glaucoma are vision-threatening diseases that are leading causes of vision loss. Many studies have validated deep learning artificial intelligence for image-based diagnosis of vision-threatening diseases. Our study prospectively investigated deep learning artificial intelligence applications in student-run non-mydriatic screenings for an underserved, primarily Hispanic community during COVID-19.

Methods

Five supervised student-run community screenings were held in West New York, New Jersey. Participants underwent non-mydriatic 45-degree retinal imaging by medical students. Images were uploaded to a cloud-based deep learning artificial intelligence for vision-threatening disease referral. An on-site tele-ophthalmology grader and remote clinical ophthalmologist graded images, with adjudication by a senior ophthalmologist to establish the gold standard diagnosis, which was used to assess the performance of deep learning artificial intelligence.

Results

A total of 385 eyes from 195 screening participants were included (mean age 52.43  ±  14.5 years, 40.0% female). A total of 48 participants were referred for at least one vision-threatening disease. Deep learning artificial intelligence marked 150/385 (38.9%) eyes as ungradable, compared to 10/385 (2.6%) ungradable as per the human gold standard (p < 0.001). Deep learning artificial intelligence had 63.2% sensitivity, 94.5% specificity, 32.0% positive predictive value, and 98.4% negative predictive value in vision-threatening disease referrals. Deep learning artificial intelligence successfully referred all 4 eyes with multiple vision-threatening diseases. Deep learning artificial intelligence graded images (35.6  ±  13.3 s) faster than the tele-ophthalmology grader (129  ±  41.0) and clinical ophthalmologist (68  ±  21.9, p < 0.001).

Discussion

Deep learning artificial intelligence can increase the efficiency and accessibility of vision-threatening disease screenings, particularly in underserved communities. Deep learning artificial intelligence should be adaptable to different environments. Consideration should be given to how deep learning artificial intelligence can best be utilized in a real-world application, whether in computer-aided or autonomous diagnosis.

Introduction

Artificial intelligence (AI) is an umbrella term describing algorithms that can perform tasks typically reliant on human intelligence. In the model training stage, AI receives input annotated by experts and learns to recognize features or characteristics to make predictions and determine model parameters.^1,2 AI performance is then evaluated using a testing dataset, with the goal of maximizing the model's generalizability to real-world data.³ AI is especially useful in medical imaging. The use of AI in automated computer-aided diagnosis (CAD) in medicine began in the late 1900s, with its first FDA-approved use with a film-digitized mammography system in 1998.⁴ Since that time, CAD use has expanded into other fields of medicine.⁵ AI is particularly useful in ophthalmology, a field that heavily relies on imaging to diagnose pathology.^2,4,6,7 Deep learning AI (DLAI), a subset of AI that imitates the structure and function of the human brain in automated learning and prediction, has been widely utilized in algorithms for the image-based diagnosis of anterior segment pathology, for example, cataracts, glaucoma, and keratoconus; as well as posterior segment pathology, for example, retinopathy of prematurity, diabetic retinopathy, and age-related macular degeneration.^8,9 DLAI algorithms use multiple layers of extraction to recognize patterns from large unstructured data with high accuracy. Thus, DLAI has been applied to interpret multiple imaging modalities, including retinal fundus photographs, optical coherence tomography scans, and visual field tests.¹⁰

Age-related macular degeneration (AMD), diabetic retinopathy (DR), and glaucoma are vision-threatening diseases (VTDs) that are the leading causes of vision loss in the United States.¹¹ Early detection can limit the progression of vision loss, as all three VTDs progress asymptomatically in the early stages.^12–14 However, even though an estimated 93 million adults in the United States are at high risk for blindness, the Centers for Disease Control (CDC) reports only half of this high-risk population visited an eye doctor in the past year, leading to many VTDs going undetected.^11,15,16 This underutilization of eye care is exacerbated in populations that have historically been disconnected from care due to factors such as lack of insurance, transportation, or availability of ophthalmologists.^17,18 The COVID-19 pandemic compounded this problem, leading to fewer in-person visits and more missed appointments, especially in ethnic minorities and older patients.^9,19,20 Even with the rise of tele-ophthalmology use during the pandemic,^21,22 barriers such as broadband internet access and lack of comfort with technology disproportionately affect vulnerable populations.¹⁹

An automated tele-ophthalmology screening tool is one potential solution to reduce the VTD burden and identify individuals who need further ophthalmic care.²³ SELENA + , developed by EyRIS Pte Ltd in Singapore, is a DLAI that uses convolutional neural networks (CNNs), which differentiate between visual inputs by assigning varying weights to visual characteristics,²⁴ to detect VTDs in retinal fundus images. SELENA  +  can simultaneously detect three VTDs: AMD14, DR,^2,9,25 and glaucoma.^8,26 Previous research validating this DLAI has shown incredibly robust performance, with high sensitivities (90%−100%) and specificities (87.2%−91.6%) in VTD detection.⁸ Our study aimed to investigate real-time DLAI application in medical student-run screenings for an underserved, primarily Hispanic community in New Jersey. We assessed the ease of incorporation into an existing tele-ophthalmology protocol as well as performance compared to a human gold standard diagnosis.

Methods

Approval was obtained from the Rutgers Health Sciences Newark Campus Institutional Review Board, and the study was performed in accordance with the tenets of Helsinki. Informed consent was acquired from all study participants, and data was collected and analyzed in compliance with the Health Insurance Portability and Accountability Act.

From August 1 to November 2, 2021, participants were recruited from five vision screenings in an underserved, primarily Hispanic community in West New York, NJ (64.4% Hispanic; 19.7% poverty rate; $64,378 median household income).²⁷ Vision screenings were free and open to the public, advertised via social media and word of mouth. The screenings were staffed by medical student volunteers from Rutgers New Jersey Medical School and overseen by certified tele-ophthalmology grader BS (PhD) on-site and clinical ophthalmologist MH (MD) remotely. Prior to the screening, student volunteers underwent training on the use of ophthalmic imaging equipment. Each participant was assigned an anonymous ID, and demographic information including age, gender, and ethnicity were collected. 45-degree angle color retinal fundus images were captured by students using a non-mydriatic Canon CR-2 Plus AF (Tokyo, Japan) camera, which has been employed in previous screening protocols to image non-dilated participants.^28,29 Both macula- and optic nerve-centered posterior segment images were taken for each eye. In addition, anterior segment images were taken to evaluate for the presence of cataracts.

Fundus images were uploaded to Box^TM, a cloud-based sharing platform compliant with the Health Insurance Portability and Accountability Act, as part of the tele-ophthalmology protocol. Images were subsequently input into the cloud-based DLAI (SELENA + , EyRIS Pte Ltd), which was loaded onto a laptop using an encrypted and password-protected mobile Wi-Fi hotspot (Figure 1). There was no additional software or equipment required to use the DLAI. To ensure proper imaging technique, DLAI displayed a popup warning if the images appeared incorrectly placed (Figure 3). For each eye, DLAI reported referable, non-referable, or ungradable for each VTD. DLAI referred eyes with lesions consistent with DR, suspicious features of glaucoma on the optic disc, and/or at least intermediate dry AMD (Figure 2). Grading time for DLAI was documented as the time taken to submit images and generate the report.

Figure 1.

Medical student volunteers connect to the cloud-based deep learning artificial intelligence (DLAI) on a laptop via mobile Wi-Fi hotspot at the screening site.

Figure 3.

Sample report generated by DLAI. Possible results for each eye and VTD are non-referable, referable, and ungradable.

DLAI: deep learning artificial intelligence; VTD: vision-threatening disease.

Figure 2.

Popup message by deep learning artificial intelligence (DLAI) to prevent user error in input of retinal fundus images. Image upload fields include optic disc- and macula-centered for right and left eyes.

Three human graders were enlisted to establish a gold standard diagnosis, as per existing tele-ophthalmology screening protocols.³⁰ Images were graded by the certified tele-ophthalmology grader on site and clinical ophthalmologist remotely. A senior ophthalmologist adjudicated disputed findings between the tele-ophthalmology grader and clinical ophthalmologist when needed to establish the gold standard diagnosis. Image quality was graded on a 1-5 scale based on qualities such as gamma (exposure) and alignment as previously described in Kolomeyer Anton et al.³⁰ Graders also assessed the need for referral and graded fundus images according to the standardized Age-Related Eye Disease Studies (AREDS) categories for AMD, International Clinical Diabetic Retinopathy Disease Severity Scale (ICDRSS) for DR, and consideration of cup-to-disc ratio and nerve fiber layer for glaucoma suspect. Grading time, defined as time to evaluate images and fill out a grading form, was recorded. Documentation of gradings by the clinical ophthalmologist was performed by a scribe.

For further evaluation of patients with suspected pathology, optical coherence tomography (OCT-B, Topcon 3D Maestro-1, Tokyo, Japan) and virtual reality visual field (Palmscan VF2000 Analyzer) testing were performed. Visual acuity was also assessed using a Snellen eye chart. The onsite tele-ophthalmology expert discussed relevant findings with screening participants, and recommendations for further follow-up with an eyecare provider were given when appropriate.³¹

Referral of AMD, DR, or glaucoma suspect of each eye by DLAI was compared to the gold standard diagnosis. Overall referral of each eye, for any VTD, was also evaluated. Sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV) were calculated. Cohen's kappa was performed to assess intergrader variability. Independent sample t-test and Pearson chi-square were performed to compare grading times and proportion of images marked ungradable. SPSS Version 25 was used to perform statistical analysis, with significance set at p < 0.05.

Results

From five screenings, 385 eyes from 195 screening participants were included in the study (mean age 52.43  ±  14.5 years, 40.0% female). The ethnic makeup was 147/195 (75.4%) Hispanic, 30/195 (15.3%) African American, 13/195 (6.7%) White, and 5/195 (2.6%) Asian. 6/195 (3.1%) participants were referrable for AMD, 2/195 for DR (1.0%), and 40/195 (20.5%) for glaucoma.

DLAI was run with a mobile Wi-Fi hotspot with download and upload speeds of 33.7 and 7.9 Mbps, respectively. Grading times in seconds for the tele-ophthalmology grader (129  ±  41.0) and clinical ophthalmologist (68  ±  21.9) were longer than that of DLAI (35.6  ±  13.3, p < 0.001). DLAI marked 150/385 (38.9%) eyes ungradable, significantly more than the 10/385 (2.6%) truly ungradable as per the gold standard (p < 0.001). Of the eyes marked ungradable by DLAI, fundus image quality was ≤ 3, and 84/140 (56.2%) had at least 1  +  cataract severity.

Table 1 shows the sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV) of DLAI referral for each VTD individually and for any VTD. When referring eyes for any VTD, DLAI performed with high specificity and negative predictive value and relatively lower sensitivity and positive predictive value. For each VTD (AMD, DR, and glaucoma), specificity was similar. However, sensitivity varied. Although not significant, DLAI had higher sensitivity for DR and glaucoma compared to AMD. Four eyes had multiple VTDs, which were all correctly referred by DLAI.

Table 1.

Performance of DLAI in VTD referral compared to (A) gold standard diagnosis, (B) tele-ophthalmology expert grader, and (C) clinical ophthalmologist.

	Sensitivity	Specificity	Positive predictive value	Negative predictive value
(A) Compared to gold standard diagnosis
AMD	33.3%	97.9%	16.7%	99.1%
95% CI	0.0%−86.7%	96.0%−99.7%	0%−46.5%	97.9%−100.0%
DR	100.0%	95.9%	6.3%	100.0%
95% CI	100.0%−100.0%	93.8%−97.9%	0%−18.1%	100%−100%
Glaucoma	64.7%	90.5%	41.5%	96.1%
95% CI	48.6%−80.8%	87.4%−93.7%	28.2%−54.8%	94%−98.3%
All VTDs	63.2%	94.5%	32.0%	98.4%
95% CI	47.8%−78.5%	93.0%−95.9%	21.4%−42.6%	97.6%−99.2%
(B) Compared to certified tele-ophthalmology grader
AMD	16.7%	98.7%	25.0%	97.8%
95% CI	0.0%−46.5%	97.2%−100.0%	0.0%−67.4%	960.%−99.7%
DR	12.5%	99.7%	66.7%	96.1%
95% CI	0.0%−28.7%	99.1%−100.0%	13.3%−100.0%	94.1%−98.1%
Glaucoma	43.1%	96.5%	66.7%	91.2%
95% CI	29.5%−56.7%	94.4%−98.5%	50.6%−82.8%	88.1%−94.2%
All VTDs	62.5%	94.8%	34.2%	98.3%
95% CI	47.5%−77.5%	93.3%−96.2%	23.4%−45.1%	97.5%−99.2%
(C) Compared to clinical ophthalmologists
AMD	0.0%	97.4%	0.0%	99.6%
95% CI	0.0%−0.0%	95.4%−99.5%	0.0%−0.0%	98.7%−100.0%
DR	33.3%	95.8%	6.25%	99.4%
95% CI	0.0%−86.7%	93.8%−97.9%	0.0%−18.1%	98.6%−100.0%
Glaucoma	54.3%	91.1%	47.2%	93.2%
95% CI	40.0%−68.7%	88.0%−94.3%	33.7%−60.6%	90.4%−96.0%
All VTDs	52.0%	94.6%	34.7%	97.3%
95% CI	38.2%−65.8%	93.1%−96.1%	23.9%−45.4%	96.2%−98.4%

DLAI: deep learning artificial intelligence; AMD: age-related macular degeneration; DR: diabetic retinopathy; VTD: vision-threatening disease.

Intergrader agreement between human graders was almost perfect, with substantial agreement between DLAI and human readers, as demonstrated by Cohen's kappa values shown in Table 2. DLAI referred more eyes (202/385, 52.47%) for unreadability or referral of a VTD compared to the gold standard diagnosis (56/385, 14.5%) (p < 0.001).

Table 2.

Intergrader agreement as determined by Cohen's kappa.

	Cohen's kappa
Tele-ophthalmology grader/clinical ophthalmologist	0.946
Tele-ophthalmology grader/DLAI	0.755
Clinical ophthalmologist/DLAI	0.745
Gold-standard diagnosis/DLAI	0.754

DLAI: deep learning artificial intelligence. Cohen's kappa was calculated to assess agreement between human graders, DLAI, and the gold standard diagnosis.

Discussion

This study incorporated DLAI into a tele-ophthalmology protocol in medical student-run VTD screenings for an underserved, primarily Hispanic population. Several studies have retrospectively validated DLAI performance in detecting AMD,^14,24 DR,^32,33 and glaucoma⁸ using mydriatic retinal fundus image datasets. Given the extensive number of studies demonstrating the robust performance of DLAI algorithms, research focusing on the real-world application of DLAI in clinical deployment is important as well. While retrospective validation studies can demonstrate accuracy, prospective studies investigating real-time applications can evaluate utility. In addition, evaluating DLAI in different settings and roles can inform potential users of its strengths and limitations and encourage further exploration of how DLAI can be applied to current modalities of healthcare. The DLAI used in our study has previously been validated with image datasets from community and clinic-based populations from Asia, Australia, Africa, and North America.^{24, 25} The goal of this study was to prospectively evaluate real-time DLAI performance when incorporated into non-mydriatic community screenings.

The successful incorporation of DLAI into the screening protocol was facilitated by its ease of use. The only requirement for the cloud-based DLAI was internet connectivity, and our study successfully employed the DLAI in an environment without inherent Wi-Fi internet connectivity. Minimal training was needed to operate the user-friendly DLAI; all images were collected and input into the software by supervised medical students who underwent training beforehand. Further, DLAI's faster diagnostic speed allowed for the timely screening of a large cohort. The ability to simultaneously screen for multiple VTDs further improved efficiency and yield in a population who may not routinely utilize eye care,³⁴ as many can have concurrent ophthalmic diagnoses. For example, studies have demonstrated diabetes mellitus can be a risk factor for both diabetic retinopathy and primary glaucoma.^35,36

Overall, DLAI performed well with some limitations. There was substantial agreement between DLAI and the gold standard diagnosis, as well as DLAI and the two primary graders individually. Compared to previous validation studies, DLAI performed with similarly high specificity, albeit with lower sensitivity.^24,37 In our study, specificity was higher than sensitivity in all VTDs. Sensitivity was lower in AMD and glaucoma, while specificity was significantly lower in glaucoma. In comparison, Ting et al.'s validation study also reported lower specificity for glaucoma (87.2%) compared to AMD (88.7%) and DR (91.6%), although their study demonstrated lower sensitivity in DR referral (90.5%) compared to glaucoma (96.4%) and AMD (93.2%).²⁴ Potential differences in disease prevalence among the populations in these studies and that of our study should also be taken into consideration when evaluating these differences.

A significant limitation of DLAI was the high unreadability rate. Factors such as media opacity, small pupils, light artifacts, gamma (brightness), and image orientation can affect gradeability.³⁸ When implementing DLAI into a community screening setting, one must consider that screening environments are often unpredictable. Light artifacts, language barriers, and non-mydriatic imaging can be obstacles to perfect images. For example, in this study, we were challenged by non-adjustable lighting at the screening location; despite the use of a camera hood, light artifacts were unable to be avoided. Other studies have also reported this limitation and support that DLAI should be trained with and be generalizable to perform well with images taken in suboptimal environments, as many validation studies exclude ungradable images prior to analysis.^37,39

With only 5.7 ophthalmologists per 100,000 Americans and even fewer in regions with less developed healthcare infrastructure, automated algorithms can offer a solution to the problem of limited access to eye care.¹⁷ In areas with low ophthalmologist density, an accurate DLAI can provide an initial referral for patients in need of further ophthalmic evaluation, if not substitute for a human grader, minimizing dependency and burden on ophthalmologists. High specificity, of at least 90%, would be especially useful in this setting to accurately rule in those with vision-threatening pathology.⁴⁰ Several studies have supported DLAI's practicality. Previous investigations have shown that semi- or fully automated screenings using DLAI are more cost-effective than manual screenings.^18,25,26 Further, as our study demonstrates, DLAI's speed can increase a screening program's reach and volume.

Our study demonstrates successful DLAI incorporation into a mobile screening program that travels locally to underserved populations. DLAI can be combined with other tele-ophthalmology innovations such as smartphone fundus photography to further increase the ease of screening group mobility to regions with less developed health infrastructure.⁴¹ DLAI has been incorporated into other large-scale community screening programs worldwide.^25,42 DLAI was employed in a community-based DR screening program in Zambia, an under-resourced country with a shortage of ophthalmologists and no existing DR screening program.²⁵ This study used non-medically trained technicians to image over a thousand patients and reported both high DLAI sensitivity and specificity. However, consideration should be given to the costs and resources needed to develop and implement a tele-ophthalmology model, especially in populations without an existing telemedicine infrastructure.^43,44 Further, once screened, patients need to be appropriately referred for management and treatment, requiring partnership with eyecare providers.⁴⁵ Considering this potential bottleneck in resource-poor regions, one potential solution may be to use DLAI with a higher disease threshold to prevent excessive referrals.²⁵

DLAI can also be employed in clinical settings. DLAI can rapidly screen for VTDs in primary care settings and expand population-level eye screening.⁷ In addition, DLAI can be employed in more targeted screenings, such as in populations with genetic predispositions or comorbid risk factors. For example, DLAI could assess diabetic patients in an endocrinology clinic for diabetic retinopathy with a relatively higher pre-test probability, ensuring that those who most require ophthalmic examination receive the necessary care. A multicenter study performed by Lee et al. evaluated diabetic patients who were recruited by the Veterans Affairs teleretinal DR screening program. This study used seven algorithms, and similarly to our study, sensitivity and specificity ranged widely (50.98%−85.90% and 60.42%−83.69%, respectively).⁴⁶

Application in clinical settings may offer a significant advantage compared to community settings—a higher likelihood of follow-up care. Patients who are already established in a healthcare system, that is, have health insurance and/or follow-up with a provider for a medical diagnosis, are more likely to follow-up with an ophthalmic care provider for management.⁴⁵ Still, successful incorporation of tele-ophthalmology into clinical settings requires clinician acceptance and education, appropriate reimbursement, technical infrastructure, and establishment of regulations.^22,43 Nevertheless, there is space and opportunity for DLAI to enhance the administration of eye care in both community and clinical settings.

Another important consideration of DLAI use is autonomy. Currently, the FDA has approved two ophthalmology AIs for the autonomous detection of DR, that is, without the need for human review: IDx-DR (Digital Diagnostics, Coralville, Iowa) and EyeArt (EyeNuk, Los Angeles, CA).^2,6,7 In addition, in 2020, the Centers for Medicare and Medicaid Services (CMS) approved Current Procedural Terminology (CPT) code 9225X, allowing the use of autonomous AI in a reimbursable primary care setting.⁴⁷ Our study used DLAI in the context of computer-aided⁴ rather than autonomous diagnosis, with an overseeing human expert on-site and remotely to verify findings. Thus, the role of AI in real-world applications should be considered and further investigated. Recent studies in other specialties have sought to understand how AI and humans can best collaborate as a human-AI team.⁴⁸ Further research in ophthalmology should be performed to study the optimization of AI in different roles and scopes of practice.³⁴

DLAI has the potential to drastically increase efficiency, cost-effectiveness, and availability of care, especially in underserved communities with inadequate access to ophthalmic care. While innovations in DLAI continue to push boundaries and allow for rapid combined VTD detection, there is room for improvement in terms of readability and generalizability to different settings. In this study, DLAI's ease of use facilitated easy application in a real-world non-mydriatic community screening setting. Further research should evaluate how DLAI can best be utilized, whether in computer-aided or autonomous diagnosis, to serve healthcare providers and individuals with VTDs.