Deep Learning Detection of Sea Fan Neovascularization From Ultra-Widefield Color Fundus Photographs of Patients With Sickle Cell Hemoglobinopathy

Key Points

Question

Can deep learning automate detection of sea fan neovascularization from ultra-widefield color fundus photographs?

Findings

In a cross-sectional study using 1182 ultra-widefield color fundus photographs from 190 patients with sickle cell hemoglobinopathy, a convolutional neural network was trained to classify images for presence or absence of sea fan neovascularization. The sensitivity was 97.4% and specificity was 97.0% for detecting sea fans.

Meaning

Automated detection of sea fan neovascularization from ultra-widefield color fundus photographs may facilitate screening for vision-threatening proliferative sickle cell retinopathy.

Abstract

Importance

Adherence to screening for vision-threatening proliferative sickle cell retinopathy is limited among patients with sickle cell hemoglobinopathy despite guidelines recommending dilated fundus examinations beginning in childhood. An automated algorithm for detecting sea fan neovascularization from ultra-widefield color fundus photographs could expand access to rapid retinal evaluations to identify patients at risk of vision loss from proliferative sickle cell retinopathy.

Objective

To develop a deep learning system for detecting sea fan neovascularization from ultra-widefield color fundus photographs from patients with sickle cell hemoglobinopathy.

Design, Setting, and Participants

In a cross-sectional study conducted at a single-institution, tertiary academic referral center, deidentified, retrospectively collected, ultra-widefield color fundus photographs from 190 adults with sickle cell hemoglobinopathy were independently graded by 2 masked retinal specialists for presence or absence of sea fan neovascularization. A third masked retinal specialist regraded images with discordant or indeterminate grades. Consensus retinal specialist reference standard grades were used to train a convolutional neural network to classify images for presence or absence of sea fan neovascularization. Participants included nondiabetic adults with sickle cell hemoglobinopathy receiving care from a Wilmer Eye Institute retinal specialist; the patients had received no previous laser or surgical treatment for sickle cell retinopathy and underwent imaging with ultra-widefield color fundus photographs between January 1, 2012, and January 30, 2019.

Interventions

Deidentified ultra-widefield color fundus photographs were retrospectively collected.

Main Outcomes and Measures

Sensitivity, specificity, and area under the receiver operating characteristic curve of the convolutional neural network for sea fan detection.

Results

A total of 1182 images from 190 patients were included. Of the 190 patients, 101 were women (53.2%), and the mean (SD) age at baseline was 36.2 (12.3) years; 119 patients (62.6%) had hemoglobin SS disease and 46 (24.2%) had hemoglobin SC disease. One hundred seventy-nine patients (94.2%) were of Black or African descent. Images with sea fan neovascularization were obtained in 57 patients (30.0%). The convolutional neural network had an area under the curve of 0.988 (95% CI, 0.969-0.999), with sensitivity of 97.4% (95% CI, 86.5%-99.9%) and specificity of 97.0% (95% CI, 93.5%-98.9%) for detecting sea fan neovascularization from ultra-widefield color fundus photographs.

Conclusions and Relevance

This study reports an automated system with high sensitivity and specificity for detecting sea fan neovascularization from ultra-widefield color fundus photographs from patients with sickle cell hemoglobinopathy, with potential applications for improving screening for vision-threatening proliferative sickle cell retinopathy.

This cross-sectional study evaluates the development of a deep learning method to detect sea fan neovascularization in patients with sickle cell hemoglobinopathy.

Introduction

At least 1 in 1000 infants worldwide is born with sickle cell hemoglobinopathy (SCH), with a birth prevalence rate at least 10-fold higher in Africa compared with other parts of the world.¹ In addition to the morbidity and mortality caused by systemic vaso-occlusive disease, patients with SCH are at risk for vision loss from complications of proliferative sickle cell retinopathy (PSR).² In PSR, chronic retinal vaso-occlusion and associated retinal ischemia stimulate the development of hallmark pathologic sea fan neovascularization in the peripheral retina.² While spontaneous sea fan autoinfarction may occur in up to two-thirds of eyes with PSR,³ sea fans can be associated with recurrent vitreous hemorrhages causing prolonged and severe vision loss.^4,5 The most advanced stage of PSR involves the development of tractional retinal detachments,² which are often surgically challenging to repair.⁶

Early detection of potentially asymptomatic sea fan neovascularization affords the opportunity to offer prophylactic scatter laser photocoagulation, which a randomized clinical trial reported to approximately halve the rates of PSR-associated vision loss and vitreous hemorrhage compared with observation alone.^7,8 Untreated, 17% of a Jamaican cohort with PSR developed severe vision loss (best-corrected visual acuity of worse than 20/200 at visits separated by 3 months) over a mean 7 years of follow-up.⁵ The socioeconomic impact of such vision loss can be particularly significant for patients of school age or working age. By age 26 years, the prevalence of PSR has been reported to reach 43% in patients with hemoglobin SC disease and 14% in patients with hemoglobin SS disease.⁹

Because PSR has been identified in children as young as age 7 years in those with hemoglobin SC disease¹⁰ and 8 years in those with hemoglobin SS disease,¹¹ guidelines recommend screening dilated fundus examinations in patients with SCH beginning by age 10 years.^12,13,14 Adherence to these guidelines is low, however. Only 21% of Jamaican adults¹⁵ and 29% of Saudi Arabian patients¹⁶ with SCH reported ever having had eye examinations in recent publications. Children screened for sickle cell retinopathy in Toronto were lost to follow-up at rates of 15% for patients with hemoglobin SC disease and 6% for patients with hemoglobin SS disease.¹² Because many patients with PSR are visually asymptomatic,¹⁷ patients with SCH who have good vision may not prioritize attending screening retinal examinations, particularly in the setting of having multiple medical visits for systemic complications of SCH.¹⁸ In Africa, where SCH is most prevalent,¹ access to retinal specialists is also limited.¹⁹

An automated algorithm for grading ultra-widefield color fundus photographs (UWF-FPs) from patients with SCH may offer a complement or alternative to screening dilated fundus examinations for PSR, particularly when access to retinal specialist care is limited. Color UWF-FPs have been shown to have excellent sensitivity for capturing PSR when reviewed by retinal specialists or ophthalmology residents.²⁰ Deep learning systems have achieved comparable or superior accuracy to retinal specialists in classifying color UWF-FPs for a number of retinal conditions, including proliferative diabetic retinopathy, where retinal neovascularization typically affects the posterior fundus.²¹ However, to our knowledge, machine learning has not been previously applied to the problem of detecting peripheral retinal neovascularization—a critical feature of PSR. The objective of our study was to develop and evaluate a deep learning system for detecting sea fan neovascularization from color UWF-FPs of patients with SCH.

Methods

Study Population

In this cross-sectional study, we retrospectively identified adults (aged ≥18 years) who underwent color UWF-FPs (Optos 200Tx; Optos plc) under the care of a Wilmer Eye Institute retinal specialist between January 1, 2012, and January 30, 2019, with a visit diagnosis code of sickle cell retinopathy or sickle cell hemoglobinopathy (International Classification of Diseases, Ninth Revision codes 282.4 and 282.6 or International Statistical Classification of Diseases and Related Health Problems, Tenth Revision code D57, and/or identified from a preexisting institutional review board–approved database²²). Patient date of birth, sex, race/ethnicity, and type of SCH were obtained from the electronic medical records. Patients with sickle cell trait were included because PSR can be associated with sickle cell trait.^13,23,24,25 Exclusion criteria were diagnoses of diabetes and retinal vascular disease other than sickle cell retinopathy. Images from eyes with previous laser or surgery for PSR, macula-involving retinal detachment, severe media opacity, or vitreous hemorrhage obscuring more than 10% of fundus detail were excluded.

This study was approved by The Johns Hopkins Hospital Institutional Review Board and conducted in accordance with the Declaration of Helsinki.²⁶ Informed consent was not required by the institutional review board because the study was performed retrospectively with deidentified retinal images. This study followed the Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) reporting guideline.

Image Grading

Deidentified color UWF-FPs were exported in tagged image file format, along with corresponding early- and late-frame Optos ultra-widefield fluorescein angiography (FA) images when available. Two masked retinal specialist graders (M.G.U. and M.F.G.) independently reviewed each color UWF-FP together with corresponding FA images when available and graded the color UWF-FP for the presence, absence, or indeterminate presence of sea fan neovascularization. No distinction was made between active and regressed sea fans. Images receiving discordant or indeterminate grades were independently graded by a third masked retinal specialist (A.W.S.).

The consensus grades of the masked retinal specialists were taken as the final reference standard grades with 3 exclusion criteria. First, when consensus grades were inconsistent across images from the same eye, the involved images were excluded if a fourth, unmasked retina fellow grader (S.C.) could not identify a reason (eg, image artifact) for the discrepancy. Second, images receiving 2 indeterminate grades were excluded unless the third grader assigned a grade that matched the reference standard grades of other images from the same eye, in which case the third retinal specialist’s grade was accepted as the reference standard. Third, images with consensus grades of absence of sea fan neovascularization were excluded if independent review by the unmasked grader suggested a visible area of neovascularization on the color UWF-FP with corresponding leakage shown on FA imaging.

Architecture of the Deep Learning System

Given the previously reported good performance of the Inception V3 convolutional neural network (CNN) architecture for detection of diabetic retinopathy,²⁷ the updated Inception V4 CNN architecture²⁸ was used to train a deep learning model to distinguish between color UWF-FP images with and without sea fan neovascularization (Figure 1). This model was developed using PyTorch, version 1.3. Color UWF-FP images with assigned reference standard grades were resized to 1024 × 1024 pixels and divided into training, validation, and testing sets in a 70%:10%:20% ratio. No patient was represented in more than 1 set. The proportion of images with sea fan neovascularization was equal across sets.

Figure 1. Convolutional Neural Network Architecture.

Inception V4 convolutional neural network²⁸ architecture used for classification of ultra-widefield fundus photographs for presence (positive) or absence (negative) of sea fan neovascularization. A larger range of outputs was considered positive than negative to increase sensitivity. Numerical labels for each layer denote the following: the width dimension indicates the number of channels (number of values per pixel), and the depth dimension indicates the height and width of the image representation in pixels.

The CNN was trained to be robust against simple transformations to the input image using image augmentation, including horizontal and vertical shifting by up to 10%, scaling by up to 10%, rotation by up to 25°, and inclusion of random gaussian noise. The CNN was pretrained on the ImageNet data set²⁹ and then trained for 100 iterations of the SCH data set using binary cross entropy loss and the Adam optimizer with learning rate 0.0001, polynomial learning rate decay, and weight decay 0.00005.³⁰ A decision threshold was chosen to convert each raw CNN output value (ranging from 0 to 1, corresponding to the likelihood of an image having sea fan neovascularization) into a binary output of positive or negative for sea fan neovascularization. The decision threshold was selected by optimizing a weighted mean of the sensitivity and specificity of the CNN compared with reference standard grades in the validation set, weighting sensitivity 3 times as much as specificity.

Saliency Map Visualizations

Two saliency map visualization methods, guided gradient-weighted class activation mapping (Guided Grad-CAM)³¹ and SmoothGrad,³² were used to understand what information the CNN was using to classify each input color UWF-FP. Both methods generate an initial saliency map by taking the gradient of the output of the CNN with respect to the input color UWF-FP. This saliency map represents how much a small change to each pixel of the input image affects the output classification, which corresponds to how important each pixel is to the classification of the input image.

However, these visualizations can be noisy and sensitive to variation in the input color UWF-FP. SmoothGrad therefore repeats the process multiple times, each time adding random noise to the raw input image, and averages the results to yield a clearer and more accurate visualization.³² Guided Grad-CAM solves these problems by combining the initial saliency map with the saliency map of the image after it has been passed through all of the convolutional layers.³¹ At this point, the image retains some spatial features from the raw input image but also contains discriminative information relevant to the classification. The output of each method is colorized to produce heatmaps, with more red color indicating the pixels of the raw input images that are the most important for the final CNN output classifications.

Statistical Analysis

The performance of the deep learning model on the test set was assessed by computing the area under the receiver operating characteristic curve (AUC), accuracy, sensitivity, and specificity of the CNN in classifying color UWF-FPs from the test set for the presence or absence of sea fan neovascularization. To account for repeated images being allowed from each eye, performance metrics were also computed for eye encounters after clustering images from the same patient eye and imaging encounter. A cluster bootstrap was used to compute 95% CIs for the AUC, clustered by patient or eye encounter as applicable. Accuracy, sensitivity, and specificity were computed with exact Clopper-Pearson 95% CIs³³ (statsmodel package). The intergrader agreement between the 2 primary retinal specialist graders was evaluated using the weighted Cohen κ coefficient³⁴ (Stata, version 14.2; StataCorp LLC).

Results

Baseline Characteristics

From 1348 initially eligible color UWF-FPs obtained from 196 patients, 1182 images from 168 right eyes and 168 left eyes from 304 encounter dates from 190 patients were assigned reference standard grades and included for CNN training, validation, and testing. Corresponding FA images were available for 898 of included color UWF-FPs (76.0%). The Table summarizes baseline characteristics of the included patient population. Of the 190 patients, 119 patients (62.6%) had hemoglobin SS disease and 46 (24.2%) had hemoglobin SC disease; 179 patients (94.2%) were of Black or African descent. Eighty-nine patients (46.8%) were men and 101 patients (53.2%) were women.

Table. Baseline Characteristics of 190 Included Patients.

Characteristic	Patients, No. (%)
Age at initial encounter, mean (SD), y	36.2 (12.3)
Sickle cell classification
Hemoglobin SS	119 (62.6)
Hemoglobin SC	46 (24.2)
Hemoglobin S/β-thalassemia	18 (9.5)
Hemoglobin S/α-thalassemia	1 (0.5)
Sickle cell trait	5 (2.6)
Unknown	1 (0.5)
Race/ethnicity
Black or African descent	179 (94.2)
Hispanic	4 (2.1)
White	2 (1.1)
Other	5 (2.6)
Sex
Men	89 (46.8)
Women	101 (53.2)

Sea fan neovascularization was visible in 201 of the 1182 included images (17.0%) from 40 right eyes and 35 left eyes of 57 of the 190 patients (30.0%) included. Fifty-eight color UWF-FPs (6.5% of included color UWF-FPs that had corresponding FA images available) from 19 patients were assigned a reference standard grade of absence of sea fan neovascularization but had evidence of leakage shown on FA imaging in a region not adequately visualized on the color UWF-FP.

Intergrader Agreement

After assignment of ordinal grades of 0 for absence of sea fan, 1 for indeterminate presence of sea fan, and 2 for presence of sea fan, the weighted Cohen κ value between the 2 primary retinal specialist graders was 0.621 (SE, 0.024) in the initial cohort of eligible color UWF-FPs, indicating substantial intergrader agreement.³⁵ After applying the exclusion criteria described in the Methods section, the weighted Cohen κ value was 0.731 (SE, 0.027). By convention, a Cohen κ value greater than 0.80 defines excellent agreement.³⁵

Performance of Deep Learning Model

Compared with the consensus retinal specialist reference standard grades, the performance of the CNN for identifying sea fan neovascularization from individual color UWF-FPs in the test data set was AUC, 0.988 (95% CI, 0.969-0.999); accuracy, 97.0% (95% CI, 94.0%-98.8%); sensitivity, 97.4% (95% CI, 86.5%-99.9%); and specificity, 97.0% (95% CI, 93.5%-98.9%) (Figure 2A). Only 1 image received a false-negative classification from the CNN in the setting of a severe lid artifact obscuring most of the retinal vasculature.

Figure 2. Receiver Operating Characteristic Curves for Detection of Sea Fan Neovascularization From Ultra-Widefield Color Fundus Photographs.

A, Receiver operating characteristic (ROC) curve for convolutional neural network classification of individual ultra-widefield fundus photographs from the test set as positive or negative for sea fan neovascularization. B, ROC curve for eye encounters after clustering fundus photographs from the same patient eye and imaging encounter.

By comparison, for sea fan detection, 1 primary masked retinal specialist grader’s sensitivity was 100.0% and specificity was 98.0% for images in the test set. The other primary grader’s sensitivity was 94.9% and specificity was 97.0%. These statistics were computed after combining the grades of presence of sea fan neovascularization and indeterminate presence of sea fan neovascularization into a single category.

Because multiple color UWF-FPs could be included per patient eye, the performance of the CNN in identifying sea fan neovascularization was also analyzed for eye encounters after clustering color UWF-FPs from the same imaging encounter and patient eye. To maximize the sensitivity of the CNN as a screening tool, the reference standard grade for each eye encounter was considered positive for sea fan neovascularization if any color UWF-FP from the imaging encounter for the given eye had a positive reference standard grade. Similarly, the CNN output classification for an eye encounter was considered positive for sea fan neovascularization if any color UWF-FP from the imaging encounter for the given eye received a positive output classification. The performance of the CNN for identifying sea fan neovascularization from 107 distinct test set eye encounters was AUC, 0.994 (95% CI, 0.981-1.000); accuracy, 95.3% (95% CI, 89.4%-98.5%); sensitivity, 100.0% (95% CI, 82.4%-100.0%); and specificity, 94.3% (95% CI, 87.2%-98.1%) (Figure 2B).

Visual Explanations

The eFigure in the Supplement shows representative Guided Grad-CAM and SmoothGrad colorized heatmaps superimposed on the corresponding raw color UWF-FPs for representative test set images that the CNN accurately classified. On average, Guided Grad-CAM heatmaps highlighted the importance of the temporal and, to a lesser extent, nasal peripheral retina for determining CNN output classifications (Figure 3A and B). On average, SmoothGrad heatmaps highlighted the temporal more so than nasal peripheral retinal vasculature as important for CNN output classification determination (Figure 3C and D). At least 1 visualization method accurately highlighted 1 or more sea fans for each image from the test set that the CNN correctly classified as having sea fan neovascularization (eFigure, A-F in the Supplement).

Figure 3. Composite Heatmap Visualizations of Areas of Ultra-Widefield Color Fundus Photographs Most Important on Average for Convolutional Neural Network Image Classifications.

Composite colorized Guided Grad-CAM and SmoothGrad heatmaps highlighting in red or orange the areas of peripheral (especially temporal) retina most important, on average, for convolutional neural network (CNN) classifications of right (A and C) and left (B and D) eye test set images for the presence or absence of sea fan neovascularization.

Discussion

We developed a deep learning system to detect sea fan neovascularization from color UWF-FPs with high sensitivity and specificity compared with retinal specialist reference standard grades. Our model’s performance was similar to that of a previous CNN developed using color UWF-FPs for detection of proliferative diabetic retinopathy.²¹

A challenge of interpreting any deep learning system lies in its black box nature. We used 2 independent saliency map visualization methods to localize the regions of the input color UWF-FP images that most affected the CNN output image classifications. The relative importance of the temporal peripheral retina, highlighted by both the Guided Grad-CAM and SmoothGrad saliency maps, was consistent with the clinical observation that the temporal peripheral retina is the most susceptible to sea fan neovascularization in PSR.²

There is increasing interest in combining machine learning with telemedicine to enable rapid, reproducible, point-of-care screening for ocular pathologic conditions. Because patients with SCH are predominantly of Black or African and Hispanic descent,^36,37,38 PSR disproportionately affects medically underserved patients who may have less access to retinal specialist care. A robust automated system for detecting PSR from color UWF-FPs could increase access to regular fundus screenings for patients with SCH and help identify patients most in need of further evaluation and possible treatment by a retinal specialist. Integration of color UWF-FPs into nonophthalmic medical practices may help reduce the burden of separate in-person ophthalmology visits for asymptomatic patients with SCH. The fact that our CNN appeared to have superior sensitivity to one of the masked retinal specialist graders in detecting sea fan neovascularization also suggests a role for using our model to augment clinician interpretation of color UWF-FPs.

Strengths and Limitations

Our study has several strengths. We used what is, to our knowledge, the largest existing retrospective database in the published literature of color UWF-FPs from adults with SCH, with proportions of patients with hemoglobin SS and SC disease similar to the population distribution of SCH in the US.^36,37 Our study’s patient population was predominantly of Black or African descent, reflecting the population most at risk for SCH, as well as adding a contribution to the limited literature on deep learning classifications of color UWF-FPs for retinal pathologic conditions in ethnically diverse populations.^{21,39,40,41,42,43,44,45} Images from eyes with preexisting laser or surgical treatment for PSR were excluded to minimize confounding features associated with the presence of sea fan neovascularization. Reference standard grades were rigorously assigned, accounting for consistency of grading across images from the same eye. The retinal specialist graders were aided by FA imaging when available, enabling graders to retrospectively identify subtle sea fans on 19 color UWF-FPs from 11 patients after reviewing the corresponding FA images. In addition, our analysis of our CNN’s performance after clustering images by eye encounter showed that incorporating information from multiple images increased our model’s sensitivity for detection of sea fans, with an associated mild decrease in specificity. Further prospective work is needed to provide guidelines on the optimal number of repeat color UWF-FPs to be obtained per imaging encounter.

This study was limited by its single-institutional retrospective design and relatively small number of images. However, sea fan neovascularization is a relatively rare finding, and herein we used the largest existing database of color UWF-FPs from adults with SCH. In addition, Optos color UWF-FPs are limited by pseudocolor imaging, image distortion, lid artifact, and reduced peripheral resolution that can impair visualization of subtle peripheral pathologic characteristics,^46,47 as exemplified by the number of color UWF-FPs in our study with insufficient visualization of areas of leakage shown on FA imaging. Standardized criteria for grading color UWF-FP image quality will be important for developing guidelines for maximizing the likelihood of capturing peripheral sea fans on color UWF-FPs. Owing to the low representation of images with vitreous hemorrhage or retinal detachment in our study population, we trained our model to detect sea fans but not other complications of PSR. Our model is thus sensitive for detecting asymptomatic Goldberg stage 3 PSR, but further work is needed to assess and optimize its capacity to detect Goldberg stages 4 and 5 PSR.²

Before our CNN can be applied to clinical practice, it will be important to externally validate our model using additional prospective and retrospective data sets of color UWF-FPs obtained from both adult and pediatric patients, ideally using diverse imaging platforms.^48,49,50,51 In the future, integration of multiple deep learning models may also enable simultaneous screening for not only PSR but also other retinal, macular, and optic nerve diseases from color UWF-FPs.

Conclusions

We trained a CNN to detect sea fan neovascularization from color UWF-FPs with high sensitivity and specificity. With further validation, this deep learning system may facilitate automated screening of asymptomatic patients with SCH for PSR, helping to identify patients who most merit referral to a retinal specialist for evaluation and possible treatment of potentially vision-threatening retinopathy.

Supplement.

eFigure. Representative Heatmap Visualizations of Areas of Ultra-Widefield Color Fundus Photographs Most Important to Convolutional Neural Network Image Classifications