Abstract
Purpose:
Fundus fluorescein angiography (FFA) is the standard modality to diagnose and manage choroidal neovascularization in AMD (CNV). However, FFA is costly, has considerable morbidity from allergic reactions, and mortality of 1 per 220000. Since the advent of anti-VEGF therapy for CNV, optical coherence tomography (OCT) used extensively to manage CNV, but FFA is also widely used. One recent study found the sensitivity and specificity of OCT compared to FFA in diagnosis of CNV to be 100 and 80.8%, respectively. We hypothesize that FFA does not affect the management of patients initially suspected of having CNV to a clinically significant degree.
Design:
Evaluation of diagnostic test using vignettes
Participants:
99 patients (99 eyes) who had an initial presentation of later-confirmed CNV.
Methods:
We retrospectively extracted in de-identified form the FFA, OCT, and clinical histories of the subjects. Vignettes were created with a standard narrative clinical history, posterior-pole color fundus image, central B-scan OCT of the initial visit, as well as early, mid, and late FFA of the affected eye. Four masked retinal specialists reviewed, in randomized order, these vignettes without FFA images (FFA− arm) and answered a forced choice management question: observation, three consecutive anti-VEGF injections, or other. After re-randomization, experts again reviewed the vignettes with the addition of the FFA images (FFA+ arm).
Main outcome measures:
intra- and inter-observer concordance and reliability statistics within and between specialists.
Results:
Among our retina specialists, intra-observer concordances were 89.7%, 88.7%, 88.7%, and 95.9% (average 90.7%, 95% CI 83.7-97.6%). The average inter-observer concordance for the FFA− arm was 84.0% (95% CI 72.6-95.4%) and for the FFA+ arm, 81.8% (95% CI 68.5-95.2%); paired t-testing demonstrated no significant difference between FFA− and FFA+ arms: t=0.6, p=0.55.
Conclusion:
Our data suggests a high degree of agreement in clinical decision-making whether FFA was utilized or not. There was a similar level of agreement among specialists in the FFA− and FFA+ groups, albeit at higher, not statistically significant, variability. We believe these findings further support deferring the use of FFA in the initial management of CNV in AMD, except in treatment failures and non-standard cases.
Background
In developed countries, choroidal neovascularization (CNV) in the context of age-related macular degeneration (AMD) is the leading cause of vision loss among those aged 55 or older.(1) Fundus fluorescein angiography (FFA) has long been the gold-standard for the diagnosis of CNV, as specifically stated in the American Academy of Ophthalmology (AAO) Preferred Practice Patterns (PPP):(2) Intravenous fundus fluorescein angiography Is indicated when the patient complains of new metamorphopsia or has unexplained blurred vision, and/or when clinical examination reveals elevation of the RPE or retina, macular edema, subretinal blood, hard exudates, or subretinal fibrosis, or the OCT shows evidence of fluid; all recent major AMD clinical trials also include FFA as part of their inclusion criteria.(3-5) However, FFA typically requires placement of an intravenous catheter, dedicated equipment, highly trained operators, additional time, and carries a spectrum of risk, ranging from mild (i.e., nausea in 1:20), moderate (i.e., urticaria, dyspnea in 1:60), and severe (i.e., anaphylaxis in 1:2000, death in 1:220000).(6) Historically, FFA was not only clinically useful for diagnosis, but was requisite for management of CNV with focal laser or photodynamic therapy.(7, 8) Laser-based modalities have since been largely replaced by intravitreal anti-vascular endothelial growth factor agents (anti-VEGF).(9) Various studies have demonstrated a reduction in the progression of vision loss, especially over a 2 year period after starting anti-VEGF treatment. (3, 9, 10) Indeed, both optical coherence tomography (OCT) and anti-VEGF therapies have altered practice patterns considerably with decreased FFA utilization along with increased OCT and anti-VEGF use.(11-13)
Several studies have shown that OCT has generally high sensitivity and moderate-to-high specificity to detect CNV as compared to FFA. Sandhu, et al, found that stereo color photos paired with OCT were 94% sensitive and 89.4% specific in identifying CNV when compared against an FFA reference standard by one expert.(14) Khurana and collaborators analyzed cases with retinal tomographical abnormalities (i.e., intra-retinal fluid, cystoid findings, sub-retinal fluid) in the context of CNV whether or not leakage was seen on FFA and found that spectral-domain OCT (SD-OCT) was 90% sensitive to leakage found on FFA but only 47% specific; that is, abnormalities were found on OCT in the absence of leakage, which suggests management decisions based on OCT alone might lead to overtreatment.(15) Wilde and his team retrospectively reviewed cases suspected of CNV in AMD in a tertiary referral-based retina clinic and compared SD-OCT alone to the FFA reference standard; the sensitivity and specificity of SD-OCT for detecting CNV was 100 and 80.8%, respectively. (16) A systematic review of eight studies comparing the sensitivity and specificity of active CNV detection of both time-domain OCT (TD-OCT) and SD-OCT against FFA; the pooled sensitivity was 85% (72-93% 95 CI), and specificity 48% (30-67% 95 CI); corresponding positive and negative predictive values are 70.66% and 60.26%, which suggests fair overall ability of OCT to demonstrate CNV, but considerable disagreement with FFA with regard to CNV activity.(17) De Salvo, et al, examined the characteristics of patients presenting with pigment epithelial detachments on SD-OCT suspected of having occult CNV or polypoidal choroidal vasculopathy (PCV) and found SD-OCT to have 94.6% sensitivity and specificity of 92.9% in detecting PCV lesions as compared to indocyanine green angiography (ICGA).(18)
Ultimately, the relevance of FFA in CNV management in AMD pertains directly to the treatment decision of patients suspected of having CNV, and thus our study evaluates the extent to which a baseline FFA affects management of CNV. We hypothesize that the availability of FFA does not alter the initial management decision of whether or not to institute anti-VEGF (or other) therapy to a clinically relevant degree.
Methods
We selected 200 consecutive patients that presented to the Retina Clinic at the University of Iowa, using billing diagnoses “exudative age related macular degeneration” and “choroidal neovascularization”. These patients always underwent posterior-pole color fundus images by a high-resolution fundus camera (TRC-50DX, Topcon inc.) SD-OCT images were obtained using the Spectralis (HRA+OCT, Heidelberg Engineering), using a volume scan of 20 × 20° cube at 1064x49x1024 voxels, FFA with intravenous injection of 5 ml of 20% fluorescein solution at baseline. Next we reviewed the color fundus photos, OCT images, and all FFA images of their first visit, and excluded those who had missing OCT, FFA or fundus photography, insufficient quality images or incorrect imaging protocol. This left 99 eyes of 99 patients that had originally presented as CNV in AMD with appropriate imaging quality. A central OCT B-scan, early, mid, and late phase FFA, as well as posterior pole fundus photograph were de-identified for each patient. Two standardized clinical vignettes were then created from each patient’s images, one including the FFA data (FA+) and the other without the FFA data (FA−), as well as a standard clinical history.(19) Due to the retrospective nature of the study, all patients were assumed to be symptomatic, only the images were patient specific, and we depended on the referring physician’s diagnosis of CNV in AMD. The study adhered to the tenets of the Declaration of Helsinki, and Institutional Review Board approval was obtained at the University of Iowa.
Four fellowship-trained retinal specialists with extensive clinical experience and participation in clinical trials FFA for CNV (‘experts’) were masked and reviewed, in randomized order, all the vignettes without FFA data (Figure 1). After review, they were asked to choose under a forced choice paradigm among three management options: a) Observation only, b) Intravitreal injection of an anti-VEGF agent, c) Other, please specify (See Figure 1). Vignettes were re-randomized and FFA data was added (FA+ arm). The experts then answered the same forced choice management question after a minimum two-week “forget-period” between the FFA− and FFA+ vignette review.
Statistical Analyses
The intra-observer concordance was calculated by the percentage of cases for which experts made the same decision for FFA− and FFA+ vignette groups utilizing the reliability intraclass correlation method.(20) We then determined average inter-observer concordances (i.e., the number of cases where two experts agreed in management decision-making divided by the total number of decisions) within the FFA− and FFA+ arms. We calculated Fleiss’ kappa to determine the pooled reliability for each group.(21) We further analyzed each subgroup to determine what effect, if any, FFA had when each specialist’s management decision changed between FFA− and FFA+. We also analyzed the cases with the highest and lowest concordances in each group (FFA− and FFA+) to determine the qualitative characteristics that correlate with agreement or disagreement among specialists. All numeric values were depicted as mean with the corresponding 95% confidence interval; the significance level for inferential statistics was set at alpha less than or equal to 0.05. Statistical analyses were conducted with Microsoft Excel.
Results
Among our retina specialists, intra-observer concordances were 89.7% (95% CI: 89.7-93.0%), 88.7% (95% CI: 83.6-92.3%), 88.7% (95% CI: 83.6-92.3%), and 95.9% (95% CI: 94.0-97.2%) with a pooled average of 90.7% (95% CI: 83.8-97.7%); these were not significantly different from each other. The average inter-observer concordance for the FFA− arm was 84.0% (95% CI: 72.6-95.4%) and the average inter-observer concordance for the FFA+ arm was 81.8% (95% CI: 68.5-95.2%). Paired t-testing of the FFA− and FFA+ arms could not reject the null hypothesis that the two groups were equivalent (t=0.6, p=0.55). Fleiss’ pooled kappa for the FFA− group was 0.569 (95% CI: 0.488-0.649) and 0.622 (95% CI: 0.542-0.703) for the FFA+ group. Out of 792 management decisions, there were a total of 36 disagreements (4.5%). When analyzing these intra-observer disagreements (i.e., number of cases the management changed between FFA− and FFA+), 59.4% of decisions changed from observation in FFA− to anti-VEGF treatment in FFA+ (p>0.1), and the remaining 40.6% was the opposite. When analyzing the cases with the highest and lower inter-observer concordances (i.e., when the specialists differed least and most, respectively), presence of intra-retinal or sub-retinal fluid correlated positively with high agreement; more subtle alterations of retinal lamination and lack of clear leakage on FFA were associated with low agreement. None of the 12 “split decision” cases (i.e., two specialists chose observation, two chose treatment) in the FFA− group were common to the 10 split decisions in the FFA+ group.
Discussion
The results of this study suggest that the availability of baseline FFA does not alter an examiner’s initial management decision for a patient with suspected CNV in AMD. However, the results do not show more than that and evidence for a recommendation that FFA need not be a routine part of CNV evaluation in AMD eyes needs to come from a prospective study, using a standardized treatment protocol, and using VA outcome as the basis for comparison, not inter-observer agreement on diagnosis as in the present study.
We simulated the actual patient encounter under standard conditions by providing color fundus photos (as a surrogate for clinical examination), OCT, and FFA images, and thereby were able to directly compare the efficacy of SD-OCT to FFA in the management of CNV in patients presumed to have neovascular AMD. In analyzing the imaging data carefully, our reviewers felt many cases possessed the characteristics of non-AMD diagnoses, such as central serous chorioretinopathy, polypoidal choroidal vasculopathy, pattern dystrophy, and adult vitelliform dystrophy. Indeed, there was a high level of intra-observer agreement in clinical decision-making whether FFA was used or not. Within the FFA− and FFA+ groups, management decisions among our retina specialists exhibited a lower, but still fairly high, level of agreement when compared to each specialist’s decision individually. Fleiss’ kappa, a standardized measure of statistical reliability, was not significantly different between the FFA− and FFA+ groups. Nevertheless, there was a higher trend in the FFA+ group, which may suggest that FFA interpretation is more standardized than OCT. When we analyzed cases where each specialist’s decision changed with the aid of adding FFA, there was a slight tendency toward treatment (versus observation); however, this was not statistically significant.
Interestingly, among the cases with the worst concordance in the FFA− and FFA+ groups, there were no cases in common. This suggests that in the case of disagreements in the FFA− group, the availability of FFA allowed for higher agreement among specialists, serving as a “tie-breaker,” per se. In contrast, cases with the most disagreements in the FFA+ group had ambiguous findings on FFA (see figure 9) leading to interpretative discord; this discord did not exist when only the color photo and OCT data were analyzed by each specialist.
As expected, there was higher variability between experts, compared to within experts, which is representative of real differences in practice patterns; we have demonstrated similar findings in retinal specialists deciding on the number and placement of focal laser spots for clinically significant macular edema when presented with identical objective data.(22) This is especially meaningful in light of the recent 5 year CATT data, which showed that presence of sub-retinal fluid was associated with better visual acuity levels, whereas intra-retinal fluid and retinal thinning were associated with worse visual outcomes);(23) these data have likely swayed some providers away from treating sub-retinal fluid with anti-VEGF therapy, while others might continue to treat with these agents.
Numerous studies have shown that SD-OCT has been shown to have high sensitivity and specificity in the diagnosis of all CNV subtypes compared to FFA;(24) however, there was considerable variability in number of graders and grading scheme used. For example, Khurana’s study had a single grader to determine leakage and OCT characteristics;(15) our study has four. Wilde’s cohort analyzed had two independent reviewers who then openly discussed the interpretation of OCT and FFA; if there was not greater than 90% grader confidence (not an objective measure), a third ophthalmologist served as an adjudicator – this has high group bias potential and is not generally reflective of real-life practice.(16)
Although not the focus of this study, we acknowledge that some clinicians will employ ICGA in conjunction with or as a separate modality in the diagnosis and management of certain retinal conditions (e.g., PCV, choroidal hemangioma, central serous retinopathy, recurrent CNV).(18) ICGA has long been used in the testing of cardiac and hepatic function; further, it was shown to have a lower adverse reaction rate (be it mild or severe) than FFA.(25) When comparing OCT to ICGA in the diagnosis of occult CNV or PCV, SD-OCT was found to have high levels of sensitivity and specificity;(18) as such, we would expect a similarly high concordance in treatment decisions in the appropriate clinical setting.
Like any retrospective study, there are inherent limitations within our study design, such as the ability to determine if the “correct” clinical decision was made based on subsequent visual function and structural measurements. Given the standard vignette, each expert could not gauge the evolution of the patient’s symptoms, which often affects decision-making in the real-world setting; thus, all patients were assumed to be symptomatic. Our institution is a large referral center with a wide breadth of pathology and a catchment area that includes a large part of the Midwest including metropolitan areas, and thus, certain diseases that can cause macular CNV are significantly are more prevalent (e.g., ocular histoplasmosis syndrome, various hereditary retinal disorders, etc.), while others are much less prevalent (e.g., PCV); this might limit the generalizability of our study compared to other regions in the U.S. We do feel that FFA can play a significant role in the diagnosis and management of macular CNV, particularly in treatment failures and non-standard cases (e.g., degenerative myopia, suspicion of posterior uveitis, history of trauma, inherited retinal disease, etc.)
There was also a non-zero probability of our specialists “remembering” their treatment decision as color fundus photos and OCT data were common to both FFA− and FFA+ groups; however, we minimized this by randomizing the images, creating a large sample (99 eyes), and requiring a hiatus of at least two weeks between review of FFA− and FFA+ groups. Certainly, clinicians who have a lower individual threshold toward treatment will likely have higher concordance whether FFA was available or not, which also serves as another potential source of bias; we minimized this by including four specialists of varied training backgrounds. The clinical experts in this study had at least 10 years of experience managing patients with CNV; the results of this study may not necessarily apply to clinicians with less experience.
In summary, the results of this study suggest that the availability of baseline FFA does not alter an examiner’s initial management decision for a patient with suspected CNV in AMD. Nevertheless, evidence sufficient to conclude that FFA need not be a routine part of CNV evaluation in AMD eyes will have to come from a prospective study, using a standardized treatment protocol, and using visual acuity based outcome as the basis for comparison, not inter-observer agreement on diagnosis as in the present study.
Acknowledgments
Financial support: Dr. Russell is supported by the Dina J Schrage Professorship for Macular Degeneration Research. Dr. Folk is supported by the Judith (Gardner) and Donald H. Beisner MD Professorship. Dr. Abramoff is supported by the Robert C. Watzke MD Professorship. This work was partially supported by NIH grants R01 EY019112, R01 EY018853 and R01 EB004640; and the Department of Veterans Affairs.
Footnotes
Meeting presentation: Presented at the Association for Vision and Research in Ophthalmology annual meeting, 2017
Conflict of interest: no conflicting relationship exists for any author
References
- 1.Age-Related Macular Degeneration Study Research, G. (2000) Risk factors associated with age-related macular degeneration. A case-control study in the age-related disease study: Age-Related Eye Disease Study: Report number 3. Ophthalmology 107, 2224–2232 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 2.American Academy of Ophthalmology. (2015) Age-Related Macular Degeneration Preferred Practice Pattern.
- 3.Comparison of Age-related Macular Degeneration Treatments Trials Research, G., Martin DF, Maguire MG, Fine SL, Ying GS, Jaffe GJ, Grunwald JE, Toth C, Redford M, and Ferris FL 3rd. (2012) Ranibizumab and bevacizumab for treatment of neovascular age-related macular degeneration: two-year results. Ophthalmology 119, 1388–139822555112 [Google Scholar]
- 4.Chakravarthy U, Harding SP, Rogers CA, Downes SM, Lotery AJ, Culliford LA, Reeves BC, and investigators, I. s. (2013) Alternative treatments to inhibit VEGF in age-related choroidal neovascularisation: 2-year findings of the IVAN randomised controlled trial. Lancet 382, 1258–1267 [DOI] [PubMed] [Google Scholar]
- 5.Heier JS, Brown DM, Chong V, Korobelnik JF, Kaiser PK, Nguyen QD, Kirchhof B, Ho A, Ogura Y, Yancopoulos GD, Stahl N, Vitti R, Berliner AJ, Soo Y, Anderesi M, Groetzbach G, Sommerauer B, Sandbrink R, Simader C, Schmidt-Erfurth U, View, and Groups, V. S. (2012) Intravitreal aflibercept (VEGF trap-eye) in wet age-related macular degeneration. Ophthalmology 119, 2537–2548 [DOI] [PubMed] [Google Scholar]
- 6.Gardner T, A J, Kirchhof B, and Ryan SJ (2007) Retinal Vascular Disease, Springer, Berlin [Google Scholar]
- 7.(1993) Laser photocoagulation of subfoveal neovascular lesions of age-related macular degeneration. Updated findings from two clinical trials. Macular Photocoagulation Study Group. Arch Ophthalmol 111, 1200–1209 [DOI] [PubMed] [Google Scholar]
- 8.Bressler NM (2001) Photodynamic therapy of subfoveal choroidal neovascularization in age-related macular degeneration with verteporfin: two-year results of 2 randomized clinical trials-tap report 2. Arch.Ophthalmol 119, 198–207 [PubMed] [Google Scholar]
- 9.Rosenfeld PJ, Brown DM, Heier JS, Boyer DS, Kaiser PK, Chung CY, Kim RY, and Group, M. S. (2006) Ranibizumab for neovascular age-related macular degeneration. N Engl J Med 355, 1419–1431 [DOI] [PubMed] [Google Scholar]
- 10.Ivan Study Investigators, Chakravarthy U, Harding SP, Rogers CA, Downes SM, Lotery AJ, Wordsworth S, and Reeves BC (2012) Ranibizumab versus bevacizumab to treat neovascular age-related macular degeneration: one-year findings from the IVAN randomized trial. Ophthalmology 119, 1399–1411 [DOI] [PubMed] [Google Scholar]
- 11.Day S, Acquah K, Lee PP, Mruthyunjaya P, and Sloan FA (2011) Medicare costs for neovascular age-related macular degeneration, 1994-2007. Am J Ophthalmol 152, 1014–1020 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 12.Holekamp NM, Liu Y, Yeh WS, Chia Y, Kiss S, Almony A, and Kowalski JW (2014) Clinical utilization of anti-VEGF agents and disease monitoring in neovascular age-related macular degeneration. Am J Ophthalmol 157, 825–833 e821 [DOI] [PubMed] [Google Scholar]
- 13.American Academy of Ophthalmology. (2016) AAO 2016 Membership Survey. In Presented at the 2016 AAO Retina Subspecialty Day, Chicago, IL [Google Scholar]
- 14.Sandhu SS, and Talks SJ (2005) Correlation of optical coherence tomography, with or without additional colour fundus photography, with stereo fundus fluorescein angiography in diagnosing choroidal neovascular membranes. Br J Ophthalmol 89, 967–970 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 15.Khurana RN, Dupas B, and Bressler NM (2010) Agreement of time-domain and spectral-domain optical coherence tomography with fluorescein leakage from choroidal neovascularization. Ophthalmology 117, 1376–1380 [DOI] [PubMed] [Google Scholar]
- 16.Wilde C, Patel M, Lakshmanan A, Amankwah R, Dhar-Munshi S, Amoaku W, and Medscape. (2015) The diagnostic accuracy of spectral-domain optical coherence tomography for neovascular age-related macular degeneration: a comparison with fundus fluorescein angiography. Eye (Lond) 29, 602–609; quiz 610 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 17.Castillo MM, Mowatt G, Elders A, Lois N, Fraser C, Hernandez R, Amoaku W, Burr JM, Lotery A, Ramsay CR, and Azuara-Blanco A (2015) Optical coherence tomography for the monitoring of neovascular age-related macular degeneration: a systematic review. Ophthalmology 122, 399–406 [DOI] [PubMed] [Google Scholar]
- 18.De Salvo G, Vaz-Pereira S, Keane PA, Tufail A, and Liew G (2014) Sensitivity and specificity of spectral-domain optical coherence tomography in detecting idiopathic polypoidal choroidal vasculopathy. Am J Ophthalmol 158, 1228–1238 e1221 [DOI] [PubMed] [Google Scholar]
- 19.Evans SC, Roberts MC, Keeley JW, Blossom JB, Amaro KS, Robles R, and Reed GM (2015) Vignette Metholodogies for Studying Clinicians’ Decision Making: validity, utility, and application in ICD-11 field studies. Int J Clinical Health Psychlogy 15, 160–170 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 20.Koo TK, and Li MY (2016) A Guideline of Selecting and Reporting Intraclass Correlation Coefficients for Reliability Research. J Chiropr Med 15, 155–163 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 21.Vanbelle S, and Albert A (2008) A bootstrap method for comparing corre;ated kappa coefficients. J Stat Comput Simul 78, 1009–1015 [Google Scholar]
- 22.van Dijk HW, Verbraak FD, Kok PH, Oberstein SY, Schlingemann RO, Russell SR, and Abramoff MD (2013) Variability in photocoagulation treatment of diabetic macular oedema. Acta Ophthalmol 91, 722–727 [DOI] [PubMed] [Google Scholar]
- 23.Comparison of Age-related Macular Degeneration Treatments Trials Research Group, Maguire MG, Martin DF, Ying GS, Jaffe GJ, Daniel E, Grunwald JE, Toth CA, Ferris FL 3rd, and Fine SL (2016) Five-Year Outcomes with Anti-Vascular Endothelial Growth Factor Treatment of Neovascular Age-Related Macular Degeneration: The Comparison of Age-Related Macular Degeneration Treatments Trials. Ophthalmology 123, 1751–1761 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 24.Mathew R, Pefkianaki M, Kopsachilis N, Brar M, Richardson M, and Sivaprasad S (2014) Correlation of fundus fluorescein angiography and spectral-domain optical coherence tomography in identification of membrane subtypes in neovascular age-related macular degeneration. Ophthalmologica 231, 153–159 [DOI] [PubMed] [Google Scholar]
- 25.Hope-Ross M, Yannuzzi LA, Gragoudas ES, Guyer DR, Slakter JS, Sorenson JA, Krupsky S, Orlock DA, and Puliafito CA (1994) Adverse reactions due to indocyanine green. Ophthalmology 101, 529–533 [DOI] [PubMed] [Google Scholar]