Abstract
Purpose:
A training exercise was performed to study the ability of graders to reliably identify precursor lesions to geographic atrophy (GA), known as persistent choroidal hyper-transmission defects (hyperTDs), using en face optical coherence tomography (OCT) images from eyes with non-exudative age-related macular degeneration (AMD).
Design:
Intergrader agreement study
Participants:
Eleven graders participated in this exercise.
Methods:
Formal training on how to identify persistent hyperTDs on en face OCT images was provided to the graders. Persistent hyperTDs were defined as bright lesions having a greatest linear dimension (GLD) of at least 250 μm. Training consisted of a tutorial session followed by the grading of three pretest exercises, each consisting of three cases. After all the graders scored 100% on the pretest exercises, they performed a final exercise consisting of 30 en face OCT images from 29 eyes with non-exudative AMD containing 107 hyperTDs that each grader needed to evaluate. The cases contained a variety of AMD-related atrophic lesions.
Main Outcome Measures:
The sensitivity, positive predictive value (PPV), and modified accuracy were assessed for each grader.
Results:
A total of 1177 hyperTDs from 30 en face OCT images were reviewed by the graders. The mean sensitivity, PPV, and modified accuracy for all the graders were calculated to be 99.0%, 99.2%, and 98.2%, respectively. There was a 97% agreement observed between all the graders (AC1 = 0.97). Internal graders from the Bascom Palmer Eye Institute (BPEI) had a slightly higher agreement compared with the external graders (AC1 = 0.98 vs. 0.96). The hyperTDs most often incorrectly identified included the following features: (1) hyperTDs containing a hypo-transmission defect (hypoTD) core, (2) single hyperTDs that were incorrectly graded as two separate lesions, and (3) hyperTDs with a borderline GLD that was close to 250 μm.
Conclusion:
The accurate detection of persistent hyperTDs on en face OCT images by graders demonstrates the feasibility of using this OCT biomarker to identify disease progression in eyes with non-exudative AMD, especially when used as a clinical trial endpoint in studies designed to test new therapies that may slow disease progression from intermediate AMD (iAMD) to GA.
PRÉCIS
Consensus grading of precursor lesions to geographic atrophy, known as persistent hyper-transmission defects, using en face OCT images from eyes with non-exudative age-related macular degeneration achieved excellent sensitivity, accuracy, and positive predictive values.
INTRODUCTION
Geographic atrophy (GA) is considered the late-stage of non-exudative age-related macular degeneration (AMD) and is characterized by the progressive loss of photoreceptors, retinal pigmented epithelium (RPE), and choriocapillaris.1,2 Even though it was originally defined on color fundus and later on fundus autofluorescence imaging, there has been a major effort to characterize GA on optical coherence tomography (OCT) imaging.3–9 The Classification of Atrophy Meeting (CAM) group, which is an international team of AMD and retinal imaging experts whose goal is to develop multimodal definitions of atrophy in the setting of AMD, has accepted OCT as the reference method for defining the different stages of atrophy. Consequently, GA is also known as complete retinal pigment epithelium and outer retinal atrophy (cRORA) in AMD. However, not all lesions classified as cRORA meet the conventional definition of GA based on color fundus imaging.1,3,4,10
Due to the irreversible loss of visual function wherever GA occurs, it is important to identify precursor lesions that predict the future formation of GA as identified on fundus exam and color fundus imaging.3,4 This is particularly important when managing patient expectations and for designing clinical trials to test new therapies, most notably, treatments that slow the progression or prevent the formation of GA or cRORA.11–13 To date, three OCT precursor lesions have been shown to precede the onset of GA detected on color fundus imaging: (1) nascent GA (nGA),14 (2) incomplete RPE and outer retinal atrophy (iRORA),1,15 and (3) persistent choroidal hyper-transmission defects (hyperTDs).16,17
Both nGA and iRORA are defined by overlying photoreceptor degeneration while iRORA also incorporates the presence of choroidal hyper-transmission seen on B-scans.14,15,18,19 While these grading approaches are useful when examining individual OCT B-scans, the ability to detect these early precursor lesions depends on the horizontal B-scan, the inter B-scan spacing used to image an eye with AMD, and the geometry and orientation of the lesions. In particular, the definition of iRORA depends on a minimal threshold for the lateral extension of OCT features along B-scans. As a result, using horizontal B-scans may underestimate the true size of these early lesions, especially when they grow along a non-horizontal direction or the B-scan spacing is too large.17 Thus, some of the lesions may be miscalculated or missed when using definitions based on horizontal B-scans.
To overcome this limitation, Shi et al.16 proposed an alternative method of using hyperTDs to identify these precursor lesions, which are present in both iRORA and in many of the published examples of nGA.13,14,18,19 Also, instead of viewing each B-scan individually, they proposed using en face OCT imaging, which allows the reviewer to look at the whole volumetric scan at one time. By using this method, the reviewer can observe the hyperTDs as bright areas on the en face OCT image, which is derived from a sub-RPE slab positioned from 64 to 400 μm under Bruch’s membrane. This OCT slab was used in previous studies that employed en face images to identify GA.20–27 The hyperTD appears bright compared to the surrounding area on the en face image since there is increased light transmission into the choroid where the RPE is absent or attenuated. The reviewer can then position a B-scan on these areas and confirm that they represent an area of cRORA if the lesion is at least 250 μm in any en face dimension. These lesions measuring at least 250 μm in greatest linear dimension (GLD) were defined as persistent hyperTDs.16,17
To verify the use of persistent hyperTDs as a stand-alone precursor for the eventual formation of GA, Laiginhas et al.17 performed a natural history study using swept source OCT (SS-OCT) in patients with intermediate AMD (iAMD). From this study, they demonstrated that the presence of persistent hyperTDs, defined as having an en face GLD ≥ 250 μm, were highly predictive of the future formation of GA as defined by horizontal OCT B-scans that would be detectable using color fundus imaging. They also reported that graders could be easily trained to use this method to detect these early biomarkers of GA formation compared with previous strategies. As a result, they concluded that this en face imaging method could serve as a useful screening tool to identify early precursor lesions to help predict future disease progression and for enrolling patients into clinical trials designed to investigate new therapies to prevent disease progression in AMD.
To validate the ease and reliability of detecting persistent hyperTDs on en face OCT imaging, we performed a training and grading exercise to test the feasibility of using this method to accurately identify these precursor lesions. This report provides the results of the exercise and confirms the viability of this approach to assess persistent hyperTDs.
METHODS
A total of 11 graders participated in this exercise to validate a new grading system for identifying persistent hyperTDs on en face OCT images. Five graders (F.C., F.L.F., T.H.L., S.R.S., N.K.W.) were members of the CAM group and six graders (P.G.I., M.S., Y.S., O.T., L.W., G.G.) were members of the Bascom Palmer Eye Institute (BPEI) OCT research group who had more than 1-year of experience grading OCT images. Before performing the official grading exercise, each grader participated in formal training sessions on the definition of persistent hyperTDs and the identification of this feature on en face OCT images.
The imaging data in this exercise were obtained as part of an ongoing prospective SS-OCT imaging study at the BPEI. The institutional review board of the University of Miami Miller School of Medicine approved the study, and all patients signed an informed consent. The study was performed in accordance with the tenets of the Declaration of Helsinki and complied with the Health Insurance Portability and Accountability Act of 1996. The SS-OCT instrument (PLEX® Elite 9000, Carl Zeiss Meditec, Dublin, CA) utilized in this exercise had a central wavelength of 1050 nm and a scanning rate of 100,000 A-scans per second. In addition, only images with a 6 × 6 mm scan pattern centered on the fovea were used. This scan pattern consisted of 500 A-scans per B-scan, with each B-scan repeated twice at each position. There were 500 B-scan positions along the slow axis, resulting in a uniform spacing of 12 μm between A-scans and B-scans. The en face OCT structural images were created with a slab positioned from 64 to 400 μm under Bruch’s membrane. The automated Bruch’s membrane segmentation on the SS-OCT instrument is a robust and accurate algorithm. Thus, no manual adjustments to the segmentation were needed in any of the grading exercises. Only images from eyes with non-exudative AMD without evidence of treatment naive non-exudative macular neovascularization were included in this exercise. The images selected for this grading exercise were reviewed for quality and signal strength. Therefore, scans with a signal strength less than 7 based on the instrument’s output, as well as scans with significant motion artifacts, were excluded.
Grader Training
Graders were first required to attend a web-based tutorial session led by a senior author (P.J.R.). During the training session, the definition of a persistent hyperTD was established as a bright area on the en face OCT structural image with a greatest linear dimension (GLD) ≥ 250 μm. The hyperTDs would be secondarily confirmed by reviewing the corresponding B-scans to verify the presence of a choroidal hyper-transmission defect and attenuation of the outer nuclear layer (ONL). The graders were then instructed on how to measure the GLD of the hyperTD lesions on the two-dimensional en face image corresponding to the sub-RPE slab. After the instructions were provided, the graders reviewed and discussed seven training cases with the senior author. Finally, the graders were given the opportunity to ask questions about the definition of persistent hyperTDs, and the instructions for the exercise that they were about to perform were reviewed.
After the tutorial session, each grader was required to complete three separate pretests consisting of three cases each (available at https://www.ophthalmologyretina.org/). The cases were randomly selected by two authors (J.L. and R.L.) and approved by a senior author (P.J.R.) who also provided the ground truth answers for each case. For simplicity, the cases were given to the graders on PowerPoint slides. For each case, the graders were presented with a de-identified en face OCT structural image (Figure 1A). These images were exported as JPEG files and imported into PowerPoint. The dimensions of the en face structural images were kept constant for all the cases used in the exercise. On the next slide, the same en face OCT image was provided with B-scans color-coded by position to allow the graders to determine if a hyperTD on the en face image was present (Figure 1B, 1D–1I). Some of the B-scans involved lesions with true hyperTDs while other B-scans showed no evidence of hyperTDs. This prevented graders from assuming that all B-scans involved actual hyperTDs. Red circles with internal diameters equivalent to 250 μm relative to the size of the en face structural image were included next to the en face image. These circles allowed the graders to measure the GLD in any dimension. The graders then moved the red circles over areas with a hyperTD to determine if the GLD was at least 250 μm. If they decided that the lesion met the criteria to be considered a persistent hyperTD, then the graders were instructed to place the circle in the center of the bright lesion on the en face image (Figure 1C). The red circles were appropriately scaled by using the ruler tool on the OCT instrument, which allowed the reviewer to measure the GLD of the lesions on the en face structural image. This ruler was calibrated to 250 μm on a sample en face image on the OCT instrument and then exported as a JPEG file. The image was imported to PowerPoint and resized to have the same dimensions as the cases used in the exercise. The red circles were created to have the same internal diameter as the ruler on the sample en face image. Only three red circles were provided for each case and if the graders needed additional circles, they were instructed to use the copy and paste function in PowerPoint. For hyperTDs surrounding a hypo-transmission defect (hypoTD), also known as donut lesions, the graders were instructed to measure only the GLD of the bright rim surrounding the dark hypoTD (Figure 2). This strategy allowed for a uniform measurement of the hyperTD in any en face dimension and was consistent with the 250 μm definition of cRORA on horizontal B-scans.1,10
Figure 1.
Example of an en face swept-source OCT (SS-OCT) structural image from the pretest set showing persistent choroidal hyper-transmission defects (hyperTDs) that the graders needed to correctly identify before proceeding to the final exam. En face OCT structural images were created using a slab positioned from 64 to 400 μm under Bruch’s membrane. Color-coded B-scans were provided to the graders to help confirm that the bright area represented a true hyperTD on the en face structural image. Red circles were provided next to the en face structural image, and each circle had an internal diameter that was equivalent to 250 μm relative to the size of the en face image. A) SS-OCT en face structural image showing hyperTDs that the graders needed to identify. B, D–I) The same en face structural image shown in (A) with color-coded horizontal lines that correspond to the colored borders of the B-scans used by the graders to assess whether the region on the en face image corresponded to a true hyperTD. C) Red circles were used by the graders to measure the greatest linear dimension (GLD) of the hyperTD, and these circles were then placed in the center of the bright lesion on the en face image.
Figure 2.
Example of an en face OCT structural image from the test set showing a hyper-transmission defect (hyperTD) surrounding a hypo-transmission defect (hypoTD), also known as a donut lesion, that the graders needed to identify. En face OCT structural images were created using a slab positioned from 64 to 400 μm under Bruch’s membrane. Color-coded B-scans were provided to the graders to help confirm that the bright area represented a true hyperTD on the en face structural image. Red circles were provided next to the en face structural image, and each circle had an internal diameter that was equivalent to 250 μm relative to the size of the en face image. When grading the donut lesions, the graders were instructed to measure only the greatest linear dimension (GLD) of the bright rim surrounding the dark hypoTD. A) SS-OCT en face structural image showing hyperTDs that the graders needed to identify. B, D–I) The same en face structural image shown in (A) with color-coded horizontal lines that correspond to the colored borders of the B-scans used by the graders to assess whether the region on the en face image corresponded to a true hyperTD. C) Red circles were used by the graders to measure the GLD of the hyperTD, and these circles were then placed in the center of the bright lesion on the en face image.
After each pretest exercise was completed by the graders, a web-based meeting led by a senior author (P.J.R.) was held to review the answers and address questions about the exercise. To ensure that the graders were sufficiently trained, the final formal grading exercise was only performed once all the graders had achieved a 100% correct score on the final pretest. Overall, three pretests containing a total of nine cases were completed by the graders before they moved on to the official grading exercise.
Grading Exercise
After the training and pretest exercises were completed, all 11 graders received a formal grading exercise containing a total of 30 cases (available at https://www.ophthalmologyretina.org/). The cases were obtained from 29 eyes of 26 patients with one eye being included twice at different time points 4 years apart since the distribution and size of the hyperTDs had changed over this period. The format and instructions of the exercise were the same as the pretests, and each grader was required to identify the persistent hyperTDs in all the cases. A physical copy of the tutorial session with the instructions was given to the graders if they needed to refer to the original training exercise. The cases were randomly selected by two authors (J.L. and R.L.) and approved by a senior author (P.J.R.) who also provided the ground truth answers in the same manner as the pretests. They exemplified a variety of AMD-related atrophic changes, and all cases contained at least one hyperTD. It was decided not to include images without any hyperTDs as a control, since most of the scan area served as a negative control for each case.
Statistical Analysis
The sensitivity, positive predictive value (PPV), and modified accuracy in identifying hyperTDs for the grading exercise were obtained. The modified accuracy, which is calculated by dividing the true positives by the total lesions identified by the graders, was computed since the number of true negatives was undefined due to most of the scan area not containing a hyperTD.
The intergrader agreement was determined based on Gwet’s first-order agreement coefficient (AC1).28 The AC1 was employed in this study instead of the commonly used Cohen’s Kappa (κ) due to it demonstrating poor performance in situations with a high agreement and very asymmetric marginals.29 On the other hand, AC1 uses a chance-agreement probability that is calibrated to be consistent with the propensity of random rating.28 Thus, it is more stable when there are variations in the prevalence of a trait. The AC1 values for this study were interpreted the same as κ statistics: poor agreement (< 0), slight agreement (0–0.20), fair agreement (0.21–0.40), moderate agreement (0.41–0.60), substantial agreement (0.61–0.80), and almost perfect agreement (0.81–1.0).30 For the analysis, the results of each grader were paired with one another to find the percentage of agreement between the graders, and the AC1 was calculated to assess the intergrader agreement.
RESULTS
Grading Exercise Results
Thirty cases containing 107 hyperTDs were assessed by 11 graders for a total of 1177 hyperTDs. Table 1 shows the total number of true positive, false positive, and false negative lesions identified by all the graders. As previously mentioned, the specificity and the number of true negatives were undefined. The results for the individual graders are shown in Table 2. From these numbers, the mean sensitivity, PPV, and modified accuracy for all graders were calculated to be 99.0%, 99.2%, and 98.2%, respectively (Table 3).
Table 1.
Total Number of True Positive, False Positive, and False Negative Hyper-Transmission Defects Identified by All Participants in the Grading Exercise
| Lesion | No Lesion | Total | |
|---|---|---|---|
| Lesion | TP = 1165 | FP = 10 | 1175 |
| No Lesion | FN = 12 | / | 12 |
| Total | 1177 | 10 | 1187 |
Abbreviations: TP = true positives; FP = false positives; FN = false negatives
Table 2.
Grading Exercise Results for the Eleven Graders
| Grader | Grader Type | True Positives | False Negatives | False Positives | Total Lesions Detected | Sensitivity | Positive Predictive Value | Modified Accuracy |
|---|---|---|---|---|---|---|---|---|
| 1 | CAM | 106 | 1 | 0 | 107 | 99.1% | 100.0% | 99.1% |
| 2 | CAM | 106 | 1 | 1 | 108 | 99.1% | 99.1% | 98.1% |
| 3 | CAM | 107 | 0 | 3 | 110 | 100.0% | 97.3% | 97.3% |
| 4 | CAM | 105 | 2 | 3 | 110 | 98.1% | 97.2% | 95.5% |
| 5 | CAM | 105 | 2 | 1 | 108 | 98.1% | 99.1% | 97.2% |
| 6 | BPEI | 106 | 1 | 0 | 107 | 99.1% | 100.0% | 99.1% |
| 7 | BPEI | 107 | 0 | 0 | 107 | 100.0% | 100.0% | 100.0% |
| 8 | BPEI | 107 | 0 | 1 | 108 | 100.0% | 99.1% | 99.1% |
| 9 | BPEI | 105 | 2 | 0 | 107 | 98.1% | 100.0% | 98.1% |
| 10 | BPEI | 106 | 1 | 0 | 107 | 99.1% | 100.0% | 99.1% |
| 11 | BPEI | 105 | 2 | 1 | 108 | 98.1% | 99.1% | 97.2% |
Abbreviations: CAM = Classification of Atrophy Meeting; BPEI = Bascom Palmer Eye Institute
Table 3.
Predictive Values of the New Grading System in Detecting Persistent Hyper-Transmission Defects on En Face OCT Structural Images
| Predictive Values | Mean (SD) [min, max] |
|---|---|
| Sensitivity | 99.0% (0.7%) [98%, 100%] |
| PPV | 99.2% (1.0%) [97%, 100%] |
| Modified Accuracy | 98.2% (1.3%) [95%, 100%] |
Abbreviations: PPV = positive predictive value
Intergrader Agreement for Grading Exercise
Table 4 shows the intergrader agreement among the 11 graders for correctly identifying the hyperTDs in the formal grading exercise. Among all the graders, the results showed almost perfect agreement (AC1 = 0.97) with the average percent agreement being 97.0%. The BPEI graders had a slightly higher agreement (AC1 = 0.98) compared with the CAM graders. However, this result is marginal since the CAM group also showed almost perfect agreement (AC1 = 0.96). Similarly, the intergrader agreement between the CAM and BPEI graders was near perfect (AC1 = 0.97) with an average percent agreement of 96.6%. The highest agreement was between graders #1 and #6 (% agreement = 100.0%, AC1 = 1.00), while the lowest agreement was between graders #4 and #9 (% agreement = 93.6%, AC1 = 0.93).
Table 4.
Intergrader Agreement in Correctly Identifying Hyper-Transmission Defects Among the 11 Graders
| Grader Type | % Agreement Mean (SD) [Min, Max] | Gwet’s AC1 Mean (SD) [Min, Max] |
|---|---|---|
| CAM with CAM (n=10 pairs) | 96.6 (1.3) [94.6, 99.1] | 0.96 (0.01) [0.94, 0.99] |
| CAM with BPEI (n=30 pairs) | 96.6 (1.6) [93.6, 100.0] | 0.97 (0.02) [0.93, 1.00] |
| BPEI with BPEI (n=15 pairs) | 98.0 (1.0) [96.3, 99.1] | 0.98 (0.01) [0.96, 0.99] |
| Total (n=55 pairs) | 97.0 (1.5) [93.6, 100.0] | 0.97 (0.02) [0.93, 1.00] |
Abbreviations: AC1 = first-order agreement coefficient; CAM = Classification of Atrophy Meeting; BPEI = Bascom Palmer Eye Institute
Incorrectly Identified Cases
The hyperTDs that were most often incorrectly identified by graders included: (1) hyperTDs containing a hypoTD core, known as a donut lesion, (2) single hyperTDs that were incorrectly graded as two separate lesions, and (3) hyperTDs with a borderline GLD that was close to 250 μm.
Figures 3 and 4 demonstrate two cases containing donut lesions that are identified by the yellow arrows. These lesions were incorrectly identified by three and two graders, respectively. The missed hyperTD lesions were oriented along a diagonal and/or vertical meridian and may have contributed to the graders’ responses. Of note, the GLD of these hyperTDs can only be appreciated on the en face image since the horizontal B-scans detect hyperTDs with GLDs that are smaller than 250 μm.
Figure 3.
A case with a donut lesion that was missed by three graders. A) Swept-source OCT (SS-OCT) en face structural image containing persistent choroidal hyper-transmission defects (hyperTDs). B, D–I) The same en face structural image shown in (A) with color-coded horizontal lines that correspond to the colored borders of the B-scans used by the graders to assess whether the region on the en face image corresponded to a true hyperTD. C) Structural en face image with the red circles identifying the hyperTDs. The yellow arrow is pointing to a hyperTD that three graders failed to correctly identify. As shown in the en face structural image with the red circles, the donut lesion has a greatest linear dimension (GLD) > 250 μm along the diagonal dimension. However, on the respective B-scans, the GLD of the horizontal hyperTDs are all < 250 μm.
Figure 4.
A case with a donut lesion that was missed by two graders. A) Swept-source OCT (SS-OCT) en face structural image containing persistent choroidal hyper-transmission defects (hyperTDs). B, D–I) The same en face structural image shown in (A) with color-coded horizontal lines that correspond to the colored borders of the B-scans used by the graders to assess whether the region on the en face image corresponded to a true hyperTD. C) Structural en face image with the red circles identifying the hyperTDs. The lesion that the yellow arrow is pointing to represents the hyperTD that two graders failed to correctly identify. As shown in the en face structural image with the red circles, the donut lesion has a greatest linear dimension (GLD) > 250 μm along the diagonal dimension. However, on the respective B-scans, GLD of the horizontal hyperTDs are all < 250 μm.
In addition to errors associated with the donut lesions, the graders incorrectly identified some hyperTDs as being comprised of two lesions rather than one. As shown in Figure 5, the hyperTD with the blue and red circles was incorrectly identified as two lesions by three graders. However, based on the en face image, the hyperTD represented by the bright area is continuous between the red and blue circles. Thus, this lesion represents only one true persistent hyperTD.
Figure 5.
A case where three graders incorrectly identified a single persistent choroidal hyper-transmission defect (hyperTD) as two separate lesions. A) Swept-source OCT (SS-OCT) en face structural image containing hyperTDs. B, D–I) The same en face structural image shown in (A) with color-coded horizontal lines that correspond to the colored borders of the B-scans used by the graders to assess whether the region on the en face image corresponded to a true hyperTD. C) Structural en face image with the red circles identifying the hyperTDs. The lesion with the blue circle represents the hyperTD that three graders incorrectly identified as two lesions when in fact, it was a single hyperTD.
Finally, multiple graders failed to identify hyperTDs with a borderline GLD close to 250 μm. As shown in Figure 6, the hyperTD identified by the yellow arrow was missed by five graders. This hyperTD had a GLD of 255 μm, which was confirmed on the SS-OCT before and after the graders performed the exercise. In the exercise, graders were allowed to use the zoom feature in PowerPoint to magnify a lesion. As a result, whether the graders utilized this technique may have accounted for the differences in the detection rate in these borderline lesions.
Figure 6.
A case where five graders missed a persistent choroidal hyper-transmission defect (hyperTD) with a borderline greatest linear dimension (GLD) close to 250 μm. A) Swept-source OCT (SS-OCT) en face structural image containing hyperTDs. B, D–I) The same en face structural image shown in (A) with color-coded horizontal lines that correspond to the colored borders of the B-scans used by the graders to assess whether the region on the en face image corresponded to a true hyperTD. C) Structural en face image with the red circles identifying the hyperTDs. The lesion that the yellow arrow is pointing to represents the hyperTD that five graders failed to correctly identify due to its GLD being close to 250 μm. This hyperTD had a GLD of 255 μm, which was confirmed before and after the graders performed the exercise.
DISCUSSION
In this study, we assessed the ability of trained graders to use en face OCT images to identify persistent hyperTDs. The results show that after proper training, all 11 graders performed exceedingly well in identifying persistent hyperTDs. Consequently, the mean sensitivity and PPV for the graders were 99.0% and 99.2%, respectively. In addition, the intergrader agreement was almost perfect among all the graders (AC1 = 0.97). As a result, these findings demonstrate that evaluation of en face OCT images for the diagnosis of persistent hyperTDs shows high sensitivity, PPV, and intergrader agreement.
Before training, the capabilities of the graders to detect hyperTDs were largely dependent on their experience, background, and previous training in identifying precursor lesions for GA. After training, graders with formal experience in analyzing other lesions like nGA and iRORA did not appear to have any advantage over novice graders. This was noteworthy given the fact that CAM members had many years of experience compared with the novice graders from BPEI. We achieved this grading consensus by providing the graders with a formal tutorial session that defined persistent hyperTDs and gave instructions on how the graders should perform the exercises. We then provided intensive training to the graders through pretests, which were performed by the graders until all of them achieved a score of 100% on the final pretest exercise. This required all the graders to complete three pretest exercises, which included a total of nine cases. In addition, after each pretest was completed, a meeting was held to review the answers of the exercise and address questions from the graders.
After training, we observed this consensus performance by all the graders on the formal grading exercise, with the vast majority of hyperTDs being correctly identified (Table 1). As previously mentioned, three features made up the majority of incorrectly identified cases. However, these issues could easily be corrected by additional training. In addition, many of the incorrectly identified lesions in this exercise were borderline and would almost certainly develop into true persistent hyperTDs in a real-world situation. Also, in clinical trials, reading centers will measure the GLD more precisely rather than rely on the inner diameter of a circle. As a result, the issue of measuring borderline lesions that have a GLD close to 250 μm should be easily managed.
This study builds upon the previous efforts by Shi et al.16 and Laiginhas et al.17 to detect and measure hyperTDs using en face images obtained from a sub-RPE slab positioned from 64 to 400 μm under Bruch’s membrane. Compared with previous strategies that tediously examined every single B-scan to identify nGA and iRORA, the use of en face imaging provides an easier and faster strategy for identifying persistent hyperTDs. The hyperTDs could then be secondarily confirmed by the corresponding B-scans showing choroidal hyper-transmission and attenuation of the ONL. Another advantage of en face images is the ability to monitor the progression of hyperTDs in the horizontal, vertical, and diagonal dimensions compared to B-scans which only follow the horizontal dimension. This concept is especially well demonstrated with the donut lesions (Figure 3 and 4) where the GLD of these hyperTDs can only be fully appreciated on the en face image compared to the horizontal B-scans.
In the most recent CAM Report 6 publication, inter-reader agreement for iRORA was assessed by twelve graders from six reading centers. Even after intensive training, the report only showed a moderate agreement between the readers.31 In contrast, our new grading method using en face imaging showed almost perfect agreement between the graders (AC1 = 0.97). While some of our lesions may have represented iRORA on the horizontal B-scan, the ability to measure the lesion in two-dimensions using en face imaging would have most likely increased the consensus between graders in our exercise. However, this comparison has not yet been performed. Moreover, our new grading method has the advantage of identifying persistent hyperTDs using a less labor-intensive strategy compared with the inspection of individual B-scans. Also, our grading system can be easily taught to graders who do not possess a wealth of experience and could readily be adopted by experienced reading centers. This is due to the simplicity of our method in which graders just need to identify a bright area on the en face image and measure its GLD. If the GLD is ≥ 250 μm and the corresponding B-scans confirm the grading, then the lesion is considered a persistent hyperTD. Based on the high sensitivity and PPV results from our study, this new grading method would be especially useful as a screening tool to enroll patients with iAMD into clinical trials designed to prevent disease progression since iAMD is defined by the absence of these persistent hyperTDs. In addition, the growth of these hyperTDs can be studied, and the emergence of these lesions can be used as an endpoint in clinical trials looking at therapies to treat eyes with iAMD to either prevent the formation or growth of hyperTDs.
A limitation of this study includes the number of cases that were used in this grading exercise. As a result, we might not have covered all the configurations of hyperTDs due to atrophic changes seen in non-exudative AMD. In addition, the use of PowerPoint as a platform to administer the exercises may not reflect the real-world scenario on how graders will identify hyperTDs for a clinical trial in real time using a review station. In particular, the use of the inner diameter of a circle was a compromise measurement tool and measuring the GLD may be more accurate on the OCT review station. Since all graders did not have access to the same SS-OCT machine, the use of PowerPoint provided the most consistent platform for this grading exercise. While the inability to provide the entire OCT volume scan for review to the graders may have introduced a bias that could have impacted the sensitivity, accuracy, and PPV measurements, we conducted this grading exercise using a strategy that tried to closely approximate the real-world situation so that all graders from different geographic locations and with varying levels of experience could participate. Also, this study was based on SS-OCT images from a specific instrument, and dense volumetric OCT scans were performed. Thus, the results using other OCT instruments such as spectral domain OCT (SD-OCT) needs further investigation. Finally, the outcomes of this study could differ depending on the type of cases that were chosen. However, we made a concerted effort to use a representative sample of cases with varying disease severity to minimize this limitation.
In conclusion, this study demonstrated that the method of using en face OCT images to identify persistent hyperTDs, an early OCT biomarker for disease progression in eyes with iAMD, is an accurate and convenient way to document these lesions. With proper training, graders without extensive experience can be taught to reliably identify hyperTDs. In addition, the ability of en face images to monitor the progression of hyperTDs in two dimensions compared with B-scans is invaluable. With the emergence of novel therapies to prevent the formation and progression of GA, our new grading method is a dependable and straightforward strategy for screening subjects at risk of vision loss, for defining clinical trial endpoints, and for enrolling patients into clinical trials with the endpoints being the formation of persistent hyperTDs or the growth of these precursor lesions.
Financial Support (UM):
Research supported by grants from Carl Zeiss Meditec, Inc. (Dublin, CA), the Salah Foundation, the National Eye Institute Center Core Grant (P30EY014801) and Research to Prevent Blindness (unrestricted grant) to the Department of Ophthalmology, University of Miami Miller School of Medicine. The funding organization had no role in the design or conduct of this research.
Conflicts of Interest:
Giovanni Gregori and Philip J. Rosenfeld received research support from Carl Zeiss Meditec, Inc. Giovanni Gregori and the University of Miami co-own a patent that is licensed to Carl Zeiss Meditec, Inc. Dr. Rosenfeld also received research funding from Gyroscope Therapeutics and Stealth BioTherapeutics. He is also a consultant for Apellis, Boehringer-Ingelheim, Carl Zeiss Meditec, Chengdu Kanghong Biotech, Ocunexus Therapeutics, Ocudyne, Regeneron Pharmaceuticals, and Unity Biotechnology. He also has equity interest in Apellis, Valitor, Verana Health, and Ocudyne. Nadia K. Waheed is an officeholder at Gyroscope Therapeutics and receives research support from Carl Zeiss Meditec, Nidek and Heidelberg Engineering. She has received speaker fees from Nidek. She has an equity interest in Ocudyne. Srinivas R. Sadda serves as a consultant for Amgen, Apellis, Abbvie/Allergan, Iveric, Roche/Genentech, Novartis, Regeneron, 4DMT, Oxurion, Gyroscope, Nanoscope, Heidelberg, Optos, and Centervue. He has received speaker fees from Heidelberg, Carl Zeiss Meditec, Nidek, Topcon, Optos, and Novartis. He has received research instruments from Heidelberg, Carl Zeiss Meditec, Nidek, Topcon, Optos, and Centervue. Frederick L. Ferris 3rd serves on Data Monitoring Committees for: Norvo Nordisk, Apellis, Genentech, Roche, Eyevensys, Kodiak, 4D Molecular Therapeutics, Annexon, and Adverum. Tock Han Lim is a consultant of Novartis and has received speaker fees from Heidelberg Engineering. The remaining authors have no disclosures.
Abbreviations and Acronyms
- AC1
first-order agreement coefficient
- AMD
age-related macular degeneration
- BPEI
Bascom Palmer Eye Institute
- CAM
Classification of Atrophy Meeting
- cRORA
complete RPE and outer retinal atrophy
- FN
false negative
- FP
false positive
- GA
geographic atrophy
- GLD
greatest linear dimension
- HyperTD
hyper-transmission defect
- HypoTD
hypo-transmission defect
- iAMD
intermediate age-related macular degeneration
- iRORA
incomplete RPE and outer retinal atrophy
- κ
Cohen’s Kappa
- nGA
nascent GA
- OCT
optical coherence tomography
- ONL
outer nuclear layer
- PPV
positive predictive value
- RPE
retinal pigmented epithelium
- SD-OCT
spectral-domain optical coherence tomography
- SS-OCT
swept-source optical coherence tomography
- TP
true positive
Footnotes
Publisher's Disclaimer: This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting proof before it is published in its final form. Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.
This article contains additional online-only material. The following should appear online-only: Pretest Cases and Test Cases
REFERENCES
- 1.Sadda SR, Guymer R, Holz FG, et al. Consensus Definition for Atrophy Associated with Age-Related Macular Degeneration on OCT: Classification of Atrophy Report 3. Ophthalmology. 2018;125(4):537–548. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 2.Holz FG, Strauss EC, Schmitz-Valckenberg S, van Lookeren Campagne M. Geographic atrophy: clinical features and potential therapeutic approaches. Ophthalmology. 2014;121(5):1079–1091. [DOI] [PubMed] [Google Scholar]
- 3.Bird AC, Bressler NM, Bressler SB, et al. An international classification and grading system for age-related maculopathy and age-related macular degeneration. The International ARM Epidemiological Study Group. Surv Ophthalmol. 1995;39(5):367–374. [DOI] [PubMed] [Google Scholar]
- 4.The Age-Related Eye Disease Study system for classifying age-related macular degeneration from stereoscopic color fundus photographs: the Age-Related Eye Disease Study Report Number 6. Am J Ophthalmol. 2001;132(5):668–681. [DOI] [PubMed] [Google Scholar]
- 5.Veerappan M, El-Hage-Sleiman AM, Tai V, et al. Optical Coherence Tomography Reflective Drusen Substructures Predict Progression to Geographic Atrophy in Age-related Macular Degeneration. Ophthalmology. 2016;123(12):2554–2570. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 6.Thiele S, Nadal J, Pfau M, et al. Prognostic value of intermediate age-related macular degeneration phenotypes for geographic atrophy progression. Br J Ophthalmol. 2021;105(2):239–245. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 7.Thiele S, Pfau M, Larsen PP, Fleckenstein M, Holz FG, Schmitz-Valckenberg S. Multimodal Imaging Patterns for Development of Central Atrophy Secondary to Age-Related Macular Degeneration. Invest Ophthalmol Vis Sci. 2018;59(4):Amd1–amd11. [DOI] [PubMed] [Google Scholar]
- 8.Ouyang Y, Heussen FM, Hariri A, Keane PA, Sadda SR. Optical coherence tomography-based observation of the natural history of drusenoid lesion in eyes with dry age-related macular degeneration. Ophthalmology. 2013;120(12):2656–2665. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 9.Ferrara D, Silver RE, Louzada RN, Novais EA, Collins GK, Seddon JM. Optical Coherence Tomography Features Preceding the Onset of Advanced Age-Related Macular Degeneration. Invest Ophthalmol Vis Sci. 2017;58(9):3519–3529. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 10.Jaffe GJ, Chakravarthy U, Freund KB, et al. Imaging Features Associated with Progression to Geographic Atrophy in Age-Related Macular Degeneration: Classification of Atrophy Meeting Report 5. Ophthalmol Retina. 2021;5(9):855–867. [DOI] [PubMed] [Google Scholar]
- 11.Steinle NC, Pearce I, Monés J, et al. Impact of Baseline Characteristics on Geographic Atrophy Progression in the FILLY Trial Evaluating the Complement C3 Inhibitor Pegcetacoplan. Am J Ophthalmol. 2021;227:116–124. [DOI] [PubMed] [Google Scholar]
- 12.Jaffe GJ, Westby K, Csaky KG, et al. C5 Inhibitor Avacincaptad Pegol for Geographic Atrophy Due to Age-Related Macular Degeneration: A Randomized Pivotal Phase 2/3 Trial. Ophthalmology. 2021;128(4):576–586. [DOI] [PubMed] [Google Scholar]
- 13.Guymer RH, Wu Z, Hodgson LAB, et al. Subthreshold Nanosecond Laser Intervention in Age-Related Macular Degeneration: The LEAD Randomized Controlled Clinical Trial. Ophthalmology. 2019;126(6):829–838. [DOI] [PubMed] [Google Scholar]
- 14.Wu Z, Luu CD, Ayton LN, et al. Optical coherence tomography-defined changes preceding the development of drusen-associated atrophy in age-related macular degeneration. Ophthalmology. 2014;121(12):2415–2422. [DOI] [PubMed] [Google Scholar]
- 15.Guymer RH, Rosenfeld PJ, Curcio CA, et al. Incomplete Retinal Pigment Epithelial and Outer Retinal Atrophy in Age-Related Macular Degeneration: Classification of Atrophy Meeting Report 4. Ophthalmology. 2020;127(3):394–409. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 16.Shi Y, Yang J, Feuer W, Gregori G, Rosenfeld PJ. Persistent Hypertransmission Defects on En Face OCT Imaging as a Stand-Alone Precursor for the Future Formation of Geographic Atrophy. Ophthalmol Retina. 2021. [DOI] [PubMed] [Google Scholar]
- 17.Laiginhas R, Shi Y, Shen M, et al. Persistent Hyper-Transmission Defects Detected on En Face Swept Source OCT Images Predict the Formation of Geographic Atrophy in AMD. Am J Ophthalmol. 2021. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 18.Wu Z, Luu CD, Ayton LN, et al. Fundus autofluorescence characteristics of nascent geographic atrophy in age-related macular degeneration. Invest Ophthalmol Vis Sci. 2015;56(3):1546–1552. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 19.Wu Z, Luu CD, Hodgson LAB, et al. Prospective Longitudinal Evaluation of Nascent Geographic Atrophy in Age-Related Macular Degeneration. Ophthalmol Retina. 2020;4(6):568–575. [DOI] [PubMed] [Google Scholar]
- 20.Yehoshua Z, Rosenfeld PJ, Gregori G, et al. Progression of geographic atrophy in age-related macular degeneration imaged with spectral domain optical coherence tomography. Ophthalmology. 2011;118(4):679–686. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 21.Yehoshua Z, Garcia Filho CA, Penha FM, et al. Comparison of geographic atrophy measurements from the OCT fundus image and the sub-RPE slab image. Ophthalmic Surg Lasers Imaging Retina. 2013;44(2):127–132. [DOI] [PubMed] [Google Scholar]
- 22.Yehoshua Z, de Amorim Garcia Filho CA, Nunes RP, et al. Systemic complement inhibition with eculizumab for geographic atrophy in age-related macular degeneration: the COMPLETE study. Ophthalmology. 2014;121(3):693–701. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 23.Yehoshua Z, de Amorim Garcia Filho CA, Nunes RP, et al. Comparison of Geographic Atrophy Growth Rates Using Different Imaging Modalities in the COMPLETE Study. Ophthalmic Surg Lasers Imaging Retina. 2015;46(4):413–422. [DOI] [PubMed] [Google Scholar]
- 24.Yehoshua Z, de Amorim Garcia Filho CA, Nunes RP, et al. Association Between Growth of Geographic Atrophy and the Complement Factor I Locus. Ophthalmic Surg Lasers Imaging Retina. 2015;46(7):772–774. [DOI] [PubMed] [Google Scholar]
- 25.Schaal KB, Rosenfeld PJ, Gregori G, Yehoshua Z, Feuer WJ. Anatomic Clinical Trial Endpoints for Nonexudative Age-Related Macular Degeneration. Ophthalmology. 2016;123(5):1060–1079. [DOI] [PubMed] [Google Scholar]
- 26.Schaal KB, Gregori G, Rosenfeld PJ. En Face Optical Coherence Tomography Imaging for the Detection of Nascent Geographic Atrophy. Am J Ophthalmol. 2017;174:145–154. [DOI] [PubMed] [Google Scholar]
- 27.Shi Y, Zhang Q, Zhou H, et al. Correlations Between Choriocapillaris and Choroidal Measurements and the Growth of Geographic Atrophy Using Swept Source OCT Imaging. Am J Ophthalmol. 2021;224:321–331. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 28.Gwet KL. Computing inter-rater reliability and its variance in the presence of high agreement. Br J Math Stat Psychol. 2008;61(Pt 1):29–48. [DOI] [PubMed] [Google Scholar]
- 29.Feinstein AR, Cicchetti DV. High agreement but low kappa: I. The problems of two paradoxes. J Clin Epidemiol. 1990;43(6):543–549. [DOI] [PubMed] [Google Scholar]
- 30.Landis JR, Koch GG. The measurement of observer agreement for categorical data. Biometrics. 1977;33(1):159–174. [PubMed] [Google Scholar]
- 31.Wu Z, Pfau M, Blodi BA, et al. OCT Signs of Early Atrophy in Age-Related Macular Degeneration: Interreader Agreement: Classification of Atrophy Meetings Report 6. Ophthalmol Retina. 2021. [DOI] [PubMed] [Google Scholar]