Comparison Between Graders in Detection of Diabetic Neovascularization with Swept Source OCT Angiography and Fluorescein Angiography

Abstract

Purpose:

To compare the ability of ophthalmologists to identify neovascularization (NV) in patients with proliferative diabetic retinopathy (PDR) using swept source optical coherence tomography angiography (SS-OCTA) and fluorescein angiography (FA).

Design:

Retrospective study comparing diagnostic instruments.

Methods:

Eyes with PDR or severe non-proliferative diabetic retinopathy and a high suspicion of NV based on clinical examination were imaged using SS-OCTA and FA at the same visit. Two separate grading sets consisting of scrambled, anonymized SS-OCTA and FA images were created. The ground truth for presence of NV was established by consensus of two graders with OCTA experience who did not participate in the subsequent assessment of NV in this study. The two anonymized image sets were graded for presence or absence of NV by 12 other graders that included two residents, six vitreoretinal fellows, and four vitreoretinal attending physicians. The percentage of correct grading of NV using SS-OCTA and FA was assessed for each grader and across grader training levels.

Results:

Forty-seven eyes from twenty-four patients were included in this study. Overall, the mean percentage of correct NV grading was 87.8% using SS-OCTA with B-scans and 86.2% using FA (p=0.92). Assessing each grader individually, there was no statistically significant asymmetry in correct grading using SS-OCTA and FA.

Conclusions:

Ophthalmologists across training levels were able to identify diabetic NV with equal accuracy using SS-OCTA and FA. Based on these results, SS-OCTA may be an appropriate stand-alone modality for diagnosing diabetic NV.

INTRODUCTION

Diabetic retinopathy (DR) is the leading cause of blindness in working-age adults in most developed countries.¹ Causes of vision loss in DR include macular edema and neovascularization (NV).² The early identification and treatment of NV is critical in preventing vision-threatening sequelae such as vitreous hemorrhage, tractional retinal detachments, and neovascular glaucoma.^3,4

For many years, fluorescein angiography (FA) has been considered the gold standard for the identification of NV.⁵ In proliferative diabetic retinopathy (PDR), FA demonstrates early hyperfluorescence with late leakage at sites of NV.⁵ However, FA is time-consuming, requires intravenous access, and can have adverse effects including nausea and more serious allergic reactions.⁶ Furthermore, one eye must be chosen as the initial transit eye causing an imbalance in the amount of information obtained for each eye.

Swept source optical coherence tomography angiography (SS-OCTA) has recently emerged as a noninvasive, fast, repeatable, and safe alternative to FA. Prior studies have demonstrated that SS-OCTA can be used to identify NV in diabetic retinopathy with high sensitivity.^7–9 These previous studies used imaging researchers who were extensively trained in OCTA as graders. In contrast, we sought to examine the ability of non-expert ophthalmologists across multiple training levels to identify neovascularization with widefield SS-OCTA compared with FA.

METHODS

This retrospective comparative case series was performed in accordance with both the Health Insurance Portability and Accountability Act of 1996 and the Declaration of Helsinki. Institutional Review Board approval was obtained from the University of Miami Miller School of Medicine. Informed consent for SS-OCTA imaging was obtained from all patients.

Patients with PDR or severe non-proliferative diabetic retinopathy (NPDR) and a high clinical suspicion for the presence of NV were imaged with both ultrawide-field FA (Optos, Inc, Marlborough, MA) and SS-OCTA (PLEX Elite 9000; Carl Zeiss Meditec, Inc, Dublin, CA) at the same visit. The exact time points for the early and late frames of the FA varied between cases depending on whether the eye included in the grading set was transited first or second. Generally, if available, the laminar phase images were used for the early frames while the late venous stage image was used for the late frame. The FA images were cropped to show the same area of the fundus as the 12×12mm SS-OCTA scans. The FA images were then collated into an FA grading set that contained an early and late frame image for each patient. The corresponding SS-OCTA grading set included a video scrolling through all 500 B-scans that constituted the en face total retinal and vitreoretinal interface (VRI) slabs (Supplementary Video). The FA and SS-OCTA grading sets consisted of images obtained from the same patients on the same day, but the images were presented in a randomly ordered sequence that differed between the two sets. Graders were masked to patient identity. To establish the ground truth for FA and SS-OCTA grading sets, two authors with OCTA experience (HA and JFR) independently graded images for the presence or absence of NV. Discrepancies between the two authors were adjudicated by a senior retina specialist and OCTA expert (PJR).

The image graders consisted of two ophthalmology residents, six vitreoretinal fellows, and four vitreoretinal attending physicians from three academic ophthalmology departments. Four of the twelve (33%) graders had previously graded OCTA images for research. Previous graders included 2 of the 4 attending retina specialists, 2 of the 6 fellows, and none of the residents. Meanwhile, 3 graders (25%) had served as lead authors on published research involving OCTA while none of the graders had been senior authors on OCTA-related manuscripts.

Graders were required to review a training slideshow explaining the characteristics of NV on SS-OCTA (Supplementary File). Graders were then required to pass an SS-OCTA training set consisting of five cases. A minimum score of 80% was required to pass training and proceed to the grading sets. In the FA and SS-OCTA grading sets, graders were asked the binary question of whether NV was present or absent in each image. For the SS-OCTA grading set, graders were first asked to grade using just the VRI and total retinal en face slabs. Six months later, the graders repeated grading of the same VRI and total retinal en face slabs with the addition of all corresponding SS-OCTA B-scans.

Student’s t-test was used to compare correct grading on FA and OCTA. McNemar’s exact test was used to compare individual ophthalmologists’ grades between FA and SS-OCTA images from the same patient. One-way ANOVA test was used to assess differences in mean percent correct grading of NV by training level. Chi-square test was used to compare the proportion of false positives and false negatives among incorrect answers on FA and OCTA. A p-value <0.05 was considered statistically significant. Analysis was performed using StataIC 15.1 (StataCorp LLC, College Station, TX).

RESULTS

A total of 47 paired SS-OCTA and FA images were collated from 24 patients with severe NPDR or PDR. Within the FA image set, NV was determined to be present by consensus-adjudication in 36 of 47 (76.6%) images. Within the matched SS-OCTA image set, NV was determined to be present by consensus-adjudication in 35 of 47 SS-OCTA images (74.5%). The consensus-adjudication process yielded concordant ground truth grades for FA and SS-OCTA images in all but one eye, which was judged to demonstrate NV on FA but not on corresponding SS-OCTA images.

Twelve ophthalmologists at various training levels (see Methods) passed the training set and completed the FA and SS-OCTA grading sets. Among all graders, there was no significant difference in the percentage of correct grading of NV using SS-OCTA with B-scans compared to FA (87.8% vs 86.2%, respectively, p=0.92). The mean percentage of correct grading of NV on SS-OCTA did not significantly increase with the inclusion of B-scans relative to the grading without corresponding B-scans (87.8% vs 86.7%, respectively, p=0.62) (Figure 1). Lastly, there were no statistically significant differences in the overall mean percentage of correct grading of NV when comparing residents, fellows, and attending retinal specialists using either SS-OCTA or FA (Table 1).

Figure 1 –

Swept Source OCT Angiography (SS-OCTA) of Retinal Neovascularization Demonstrates the Utility of Interpreting SS-OCTA B-Scans Alongside en face SS-OCTA Images.

(A,B) Fundus photographs show an area suspicious for retinal neovascularization (NV) near the superotemporal arcade (A, white arrow; magnified in B). Early (C) and late (D) fluorescein angiography (FA) images demonstrate an area suspicious for NV in the superotemporal region (red arrows). (E-H) Vitreoretinal interface (E) and total retinal en face SS-OCTA slabs (F) with corresponding SS-OCTA B-scans (G-H) of the same eye on the same day. The yellow arrows correspond to the same region of neovascularization seen in the FA. The B-scans (G,H) demonstrate the lesion extending into the vitreous with a robust SS-OCTA flow signal. Yellow dashed lines depict the locations of corresponding B-scans. Blue dotted lines depict segmentation for the SS-OCTA slabs.

Table 1 –

Mean Percentage of Overall Correct Grading of Neovascularization by Type of Grader using Fluorescein Angiography and Swept Source OCT Angiography with and without B-Scans

Grader Type	Mean % OCTA without B-scans Correct	Mean % OCTA with B-scans Correct	Mean % FA Correct
Resident (n=2)	91.5	87.2	79.8
Fellow (n=6)	86.9	88.3	88.7
Attending (n=4)	84.0	87.2	85.6
Total (n=12)	86.7	87.8	86.2
p-value *	0.29	0.96	0.21

^*oneway ANOVA test for difference in mean percent correct by type of grader (significance p<0.05)

Among incorrect answers on FA, 26% identified NV where none was present (false positive) and 74% missed NV when it was present (false negative). Comparatively, among the incorrect responses on OCTA, 23% were false positives and 77% were false negatives. There was no significant difference in the proportion of false positive and false negative grades on FA and OCTA (p=0.73).

Assessing all 12 graders individually, there was no statistically significant asymmetry in each grader’s correct grading of NV using FA compared to SS-OCTA (Table 2). When comparing between grader training levels, residents were statistically more likely as a group to grade the FA incorrectly but OCTA correctly compared with fellows and attendings (p=0.04), who did not show asymmetry in their grading on FA and OCTA (Table 3).

Table 2 –

Agreement of Neovascularization Gradings by Individual Graders using Fluorescein Angiography versus Swept Source OCT Angiography

Grader Type	Grader	% FA and OCTA Gradings Both Correct	% FA and OCTA Gradings Both Incorrect	% OCTA Correct and FA Incorrect	% FA Correct and OCTA incorrect	Exact McNemar’s Test (p-value)
Resident	R1	72	2	13	13	1
	R2	68	4	21	6	0.09
Fellow	F1	74	6	6	13	0.51
	F2	83	4	9	4	0.69
	F3	87	0	9	4	0.69
	F4	83	0	9	9	1
	F5	70	2	11	17	0.58
	F6	79	2	11	9	1
Attending	A1	66	15	11	9	1
	A2	81	2	13	4	0.29
	A3	83	4	6	6	1
	A4	87	4	2	6	0.63

Table 3 –

Neovascularization Grading Agreement on Fluorescein Angiography and Swept Source OCT Angiography by Grader Training Level

Grader Type	Mean % FA and OCTA Gradings Both Correct	Mean % FA and OCTA Gradings Both Incorrect	Mean % OCTA Correct and FA Incorrect	Mean % FA correct and OCTA incorrect
Resident (n=2)	70.2	3.2	17	9.6
Fellow (n=6)	79.4	2.5	8.9	9.2
Attending (n=4)	79.3	6.4	8	6.4
Total (n=12)	77.8	3.9	9.9	8.3
p-value *	0.3	0.32	0.04	0.53

^*oneway ANOVA test for difference in mean percent correct by type of grader (significance p<0.05)

Lastly, 6 of the 47 FA-OCTA image pairs demonstrated >33% discrepancy in percentage correct grading between FA and SS-OCTA. Of these six pairs, three cases demonstrated a greater percentage correct grading on SS-OCTA (e.g., Figure 2) and three had a greater percentage correct grading on FA (e.g., Figure 3).

Figure 2 –

An Example of Retinal Neovascularization (NV) Graded Correctly More Frequently with Swept Source OCT Angiography (SS-OCTA) Compared with Fluorescein Angiography (FA).

(A,B) Fundus photography demonstrates an area of NV near the superotemporal arcades (A, white arrows; magnified in B). (C,D) Early (C) and late (D) FA images demonstrate early hyperfluorescence with late leakage at the same area as the NV in A and B (red arrows). The fluorescence in D is partially blocked by a cataract. (E-H) Vitreoretinal interface (E) and total retinal en face SS-OCTA slabs (F) with corresponding SS-OCTA B-scans (G-H) of the same eye on the same day. The area of NV (yellow arrows) is highlighted in the vitreoretinal slab image (E). The B-scans clearly demonstrate the NV in the preretinal space with a robust SS-OCTA flow signal (yellow arrows). Yellow dashed lines depict the locations of corresponding B-scans. Blue dotted lines depict segmentation for the SS-OCTA slabs.

Figure 3 –

An Example of Neovascularization of the Disc (NVD) Graded Correctly More Frequently with Fluorescein Angiography (FA) Compared with Swept Source OCT Angiography (SS-OCTA).

(A,B) Fundus photography demonstrates NVD (white arrow). A network of fine vessels can be seen in the magnified image (white arrow) (B). (C,D) Early (C) and late (D) FA images show early hyperfluorescence with late leakage from NVD over the optic nerve (red arrows). (E-H) Vitreoretinal interface (E) and total retinal en face SS-OCTA slabs (F) with corresponding SS-OCTA B-scans (G-H) from the same eye on the same day. The vitreoretinal interface slab (E) highlights the area of NVD (yellow arrow). On the B-scans (G,H), a fibrovascular membrane with a flow signal is seen traversing the optic cup (yellow arrow). Yellow dashed lines depict the locations of corresponding B-scans. Blue dotted lines depict segmentation for the OCTA slabs.

DISCUSSION

Prior studies have compared the ability of expert graders to identify NV on OCTA compared with FA.^7,10–12 However, the generalizability of prior studies to the clinical setting is uncertain because there were only a few graders and these graders had received extensive training in interpretation of OCTA. Meanwhile, as of 2020 there is a wide variation in the use of OCTA by attending physicians, fellows, and residents. Experience with OCTA is limited by several factors including lack of access to OCTA machines, cost, reimbursement, lack of training in OCTA during residency and fellowship, and the relative novelty of the technology. Our study differed from prior studies by including 12 non-expert ophthalmologist graders across training levels. This allowed for a more clinically relevant study of whether SS-OCTA is as useful as FA in identifying NV in PDR in the clinical setting.

We found that ophthalmologists at all levels are able to identify NV in PDR using SS-OCTA just as well as with FA. The overall percentage of correct grading of NV did not differ significantly using OCTA (both with and without B-scans) and FA. When assessing each grader individually none were found to have a statistically significant asymmetry in their correct grading of NV using OCTA and FA.

The graders in our study evaluated the SS-OCTA images first using only the en face total retinal and VRI images, and then 6 months later using both types of en face images along with corresponding B-scans. The inclusion of the B-scans with en face SS-OCTA images did not lead to a statistically significant increase in the overall percentage of correct NV grading using OCTA. In many of the SS-OCTA grading set cases, the en face images were likely sufficient to identify the NV without the use of B-scans because there was extensive fibrovascular proliferation (Figure 4). Therefore, our study was likely underpowered to detect a statistically significant improvement in grading accuracy using B-scans alongside en face SS-OCTA images, if such a benefit exists. Future, larger studies using more cases with subtle foci of NV may validate our clinical impression that the interpretation of B-scans alongside en face SS-OCTA images can be helpful, as demonstrated in Figure 1.

Figure 4 –

An Example of Robust Neovascularization Easily Seen on en face Swept Source OCT Angiography (SS-OCTA)

(A) The total retinal en face SS-OCTA slab demonstrates several tufts of retinal neovascularization (NV) (yellow arrows). (B) These vessels are also seen on the en face vitreoretinal interface slab (yellow arrows) indicating they are growing into the vitreous. In such cases of robust NV, the diagnosis of NV can often be made without the B-scan.

The utility of layer-specific en face OCTA images is critically dependent on proper segmentation of retinal layers on corresponding B-scans. Segmentation errors are particularly problematic when the normal retinal anatomy is disrupted.¹³ Such segmentation errors can lead to misclassification of retinal vascular abnormalities. Other particularly relevant imaging errors that may affect OCTA image quality include motion artifact in patients with difficulty fixating and signal attenuation in the setting of cataracts and/or vitreous hemorrhage.¹⁴

The proportion of false positive and false negative NV grades did not differ between FA and OCTA. This finding is important because false positive and false negative gradings of NV may lead to different clinical consequences. A false positive identification of NV could lead to overtreatment with panretinal photocoagulation and/or anti-VEGF agents. Meanwhile a false negative could lead to inadvertent undertreatment or observation of active PDR.

Regarding false positives on OCTA, the most commonly misidentified structure was intraretinal microvascular abnormalities (IRMA). While these structures may appear to mimic neovascular fronds on the retinal OCTA slabs, analysis of the vitreoretinal interface slab shows that these vessels are confined to the retinal plane and do not extend into the vitreous cavity. As to the missed cases of NV on the OCTA, NVD was missed more frequently than NVE. This was likely due to the fact that in some cases NVD bridged the potential space of the optic cup and appeared to be in the retinal plane. Additionally, smaller, more subtle fronds of NVE that required careful examination were missed. Figures 2 and 3 highlight examples where NV lesions were incorrectly graded.

When comparing across grader training levels, residents were more likely to have an incorrect FA grade but correct SS-OCTA grade compared with vitreoretinal fellows and vitreoretinal attending physicians. One possible explanation for this finding is that contemporary residents are relatively inexperienced with FA. FA has been in use since the 1960s, but retinal OCT is now used much more frequently than FA in routine clinical care.^5,15–17 However, the resident cohort consisted of only two graders so further studies with more graders are needed to confirm this finding.

Our study also demonstrated that the logistical advantages of SS-OCTA over FA did not come at the cost of lower diagnostic accuracy. FA allows transit in only one eye, leading to less information in the second eye (Figure 2).¹⁸ For example, as seen in Figure 2C, the earliest available FA image of the left eye was already well into the venous phase because it was not the transit eye. In such cases where the eye in question is not the transit eye and there is no early frame, it can be challenging to interpret whether there has been progressive leakage in the late frames. This limitation of FA is more important in bilateral diseases such as PDR. SS-OCTA is readily repeatable, so it yields results despite poor patient cooperation (Figure 2D).

In some cases, graders performed better on FA than SS-OCTA. For example, the case shown in Figure 3 had five more correct grades on FA than OCTA. On the SS-OCTA en face images, careful examination revealed NV on the VRI slab over the optic nerve (Figure 3E). Additionally, a fibrovascular membrane with detectable flow bridges the potential space of the optic cup on the corresponding SS-OCTA B-scans (Figure 3G–H). However, despite the presence of NVD on the SS-OCTA images, graders performed better on the FA images for this particular case. We suspect that graders were more likely not to identify this NVD because the vascular proliferation bridged the optic cup in the plane of the retina rather than projecting into the preretinal space. Thus, we recommend careful attention to the disc when analyzing SS-OCTA images to ensure that NVD is not missed.

There are limitations to our study. Due to practical constraints, graders did not have access to the entire sequence of FA images; instead, representative early and late frame FA images were selected. Graders viewed images using personal computers, which may have had different levels of screen resolution. Also, there was not an equal number of graders for each training level. Finally, though the graders were of different training levels, none were experts nor did they routinely use OCTA in their clinical practices. All graders underwent a brief introductory training for detection of diabetic NV using OCTA to ensure they understood fundamental concepts of OCTA interpretation. This training set is available in the supplementary material as a resource to ophthalmologists who wish to utilize SS-OCTA for NV detection.

Another limitation in comparing the utility of SS-OCTA versus ultrawide-field FA is that the FA images in this study were cropped to the size of the corresponding 12×12mm OCTA image. In doing so, areas of NV outside a 12×12mm region of the posterior pole may have been excluded from the field of view. However, Russell et al, in prior work, demonstrated using a simulated OCTA widefield montage (about 22×22mm in size) that in naïve PDR eyes 99.4% had at least 1 NV site within the montage OCTA compared to ultrawide-field FA.⁹ Future studies comparing the ability of graders to identify diabetic NV on FA and OCTA should be performed with OCTA images encompassing a wider field of view.

Despite these limitations, the current study demonstrates the equivalent accuracy of non-expert ophthalmologists at various levels of training using SS-OCTA and FA to detect diabetic retinal NV. In total, combined with previous work demonstrating the advantages of SS-OCTA over FA in imaging PDR, our study provides further evidence for the adoption of SS-OCTA as an appropriate stand-alone imaging modality for diagnosing diabetic NV.^8,9

Highlights:

• OCTA is as effective as FA for the detection of neovascularization in diabetes

• The accuracy of grading of neovascularization did not differ between OCTA and FA

• Ophthalmologists across training levels are able to adopt OCTA with minimal training