Evaluation of Vascular Disease Progression in Retinopathy of Prematurity using Static and Dynamic Retinal Images

Abstract

Purpose

To measure accuracy and speed for detection of vascular progression in retinopathy of prematurity (ROP) from serial images. Two strategies are compared: static side-by-side presentation vs. dynamic flickering of superimposed image pairs.

Design

Prospective comparative study.

Methods

Fifteen de-identified, wide-angle retinal image pairs were taken from infants who eventually developed plus disease. Image pairs representing vascular disease progression were taken ≥1 week apart, and control images without progression were taken on same day. Dynamic flickering pairs were created by digital image registration. Ten experts independently reviewed each image pair on a secure website using both strategies, and were asked to identify progression or state that images were identical. Accuracy and speed were measured, using examination date and ophthalmoscopic findings as a reference standard.

Results

Using static images, experts were accurate in a mean (%) ± standard deviation (SD) of 11.4/15 (76%) ± 1.7 image pairs. Using dynamic flickering images, experts were accurate in a mean (%) ± SD of 11.3/15 (75%) ± 1.7 image pairs. There was no significant difference in accuracy between these strategies (p=0.420). Diagnostic speed was faster using dynamic flickering (24.7±8.3 seconds) versus static side-by-side images (40.3±18.3 seconds) (p=0.002). Experts reported higher confidence when interpreting dynamic flickering images (p=0.001).

Conclusions

Retinal imaging provides objective documentation of vascular appearance, with potentially improved ability to recognize ROP progression compared to standard ophthalmoscopy. Speed of identifying vascular progression was faster by review of dynamic flickering image pairs than by static side-by-side images, although there was no difference in accuracy.

INTRODUCTION

Retinopathy of prematurity (ROP) is a leading cause of childhood blindness throughout the world.^1,2 In the United States, the incidence of ROP is over 65% in infants with birth weight <1,251 grams, resulting in an estimated 600 cases of infancy-acquired blindness annually.^3–6 Meanwhile, the number of infants at risk for ROP continues to rise as neonatal survival rates improve. Standard ROP management includes dilated ophthalmoscopic examination at the neonatal intensive care unit bedside, with hand-drawn sketches to document retinal findings.⁷

An international classification system has standardized the clinical diagnosis of ROP. One component of this classification system is plus disease, which is defined as arteriolar tortuosity and venous dilation within the posterior pole.^8,9 The multi-center Cryotherapy for ROP (CRYO-ROP) and Early Treatment for ROP (ETROP) studies have established that presence of plus disease warrants treatment with cryotherapy or laser photocoagulation.^9,10 Accurate assessment of plus disease is therefore essential for ROP management. Moreover, the 2005 revised international classification of ROP defined an intermediate “pre-plus” condition.¹¹ These facts underscore that ROP represents a spectrum of retinal vascular abnormalities, and that identification of vascular progression toward plus disease is critical for management.

However, vascular disease progression in ROP is often difficult to recognize with precision during serial ophthalmoscopic examinations. In particular, standard methods of documentation using retinal drawings are qualitative and imprecise regarding the appearance of retinal vessels, as well as the nature and location of peripheral disease. This essentially forces examiners to rely on memory to recognize changes in appearance, and creates problems when serial examinations are performed by different ophthalmologists. In addition, studies have shown that plus disease diagnosis may in itself be subjective, even by experts.^12–15

Digital retinal photography has potential to improve ROP management by objective documentation of retinal findings, by identification of disease progression toward plus disease through comparison with images from previous examinations, and by creating opportunities for remote telemedicine diagnosis and second opinions from experts throughout the world.^16–25 The ability to recognize vascular changes accurately and quickly would provide important advantages over traditional methods using serial ophthalmoscopy. The purpose of this study is to investigate the accuracy and speed of identifying progression in severity of retinal vascular changes from serial images of infants who eventually developed plus disease. Specifically, two strategies for comparing serial images are compared: (1) side-by-side static image pairs; and (2) dynamic flickering image pairs generated by digital image registration, in which the images are aligned to overlay one another under the premise that this may improve recognition of subtle retinal vascular changes.²⁶

METHODS

Image Selection and Processing

Images were captured during routine ROP ophthalmoscopic examinations from 2005–2010 at Columbia University and Weill Cornell Medical Center. Retinal images were obtained using a commercially-available camera (RetCam; Clarity Medical Systems, Pleasanton, CA) by a trained neonatal nurse, vitreoretinal fellow, or attending ophthalmologist on the same days that ophthalmoscopic examinations were performed. A set of 15 de-identified wide-angle retinal image pairs from infants who eventually developed plus disease as diagnosed by an expert ROP examiner (RVPC, MFC) were selected for the study. Image pairs representing plus disease progression were taken at least 1 week apart. Vascular disease progression did not require a change from “pre-plus” to “plus” (or “neither pre-plus nor plus” to “pre-plus”) by clinical diagnosis to be included in the study. In other words, “vascular disease progression” was defined for study purposes as any interval change in infants who were eventually diagnosed with plus disease during routine clinical ophthalmoscopic examination. Image pairs representing “controls” with no vascular disease progression were taken on the same day.

The same 15 image pairs were presented as two side-by-side static images and dynamic flickering images (Figure 1 and Supplemental Video 1 available at AJO.com). Dynamic flickering image pairs were created using digital registration software, which aligned serial photographs to the sub-pixel level using an algorithm to identify features such as the intersections of vessels and other corresponding peripheral retinal features (MatchedFlicker V1.2; EyeIC, Narberth, PA).^26,27 Dynamic images were displayed at a flicker rate of 2 Hertz.

Figure 1. Examples of identifying retinopathy of prematurity vascular disease progression from static side-by-side image pairs.

Serial image pairs were taken from infants who eventually developed plus disease, and the reference standard was based on defining the more severe image as the one taken on the later date. (Top left and top right) 10/10 (100%) experts responded correctly that top left was more severe. (Middle left and middle right) 5/10 (50%) experts responded correctly that middle right was more severe. (Bottom left and bottom right) 9/10 (90%) experts responded correctly that images showed no disease progression.

Image Interpretation by Study Participants

Eligible expert study participants were defined as practicing pediatric ophthalmologists or retina specialists who met at least one of three criteria: having been a study center principal investigator for the CRYO-ROP or ETROP studies, having been a certified investigator for either study, or having coauthored five or more peer-reviewed ROP manuscripts.

All images were uploaded to a secure web server. A web interface developed by the authors was used to display retinal images and collect responses, which were recorded to a secure database system (SQL Server 2005; Microsoft, Redmond, WA). Each participant who agreed to provide informed consent was given an electronic link to the study website, along with an individual secure login and password.

Participants were oriented to the study website with a 1-page instruction guide developed by the authors. ROP experts independently reviewed each image pair on the study website. The 15 image pairs were initially presented sequentially as side-by-side static images, and subsequently as dynamic flickering image pairs in the same sequence. After each image pair was displayed, participants were asked to provide one of three mutually exclusive responses with regard to ROP vascular progression toward plus disease (i.e., severity of vascular dilation and tortuosity): “image 1 is more severe,” “image 2 is more severe,” or “images 1 and 2 are from the same day and show exactly the same severity.” The reference standard for vascular disease progression was based on defining the more “severe” image as the one taken on the later date by both expert ROP examiners (RVPC, MFC), since all eyes included in the study were eventually diagnosed with plus disease during routine clinical ophthalmoscopic examination by one of two ROP examiners (RVPC, MFC).

To analyze speed, the participant response times for each static or dynamic image pair were recorded using a computer timestamp that was incorporated into the study website. Time measurement began when each new website page was loaded to display an image pair, and ended when the expert participant clicked “submit” to finalize his or her response. Finally, experts were asked to report their confidence (“confident,” “somewhat confident,” or “not confident”) for identifying the more severe image in each static or dynamic pair.

Data Analysis

Accuracy of detecting vascular progression of ROP disease severity, or recognizing that there was no progression because images were identical, was measured by comparing responses to the examination date sequence. The numbers of correctly graded image pairs for progression of ROP disease severity were compared between the static side-by-side image pairs and the dynamic flickering image pairs for each expert participant. Grading times and confidence scores for detecting vascular disease progression were compared for static side-by-side image pairs versus dynamic flickering image pairs.

Statistical software was used for data analysis (Excel 2008, Microsoft, Redmond, WA; SPSS 15.0, SPSS Inc, Chicago, IL). Comparisons between static side-by-side image pairs and dynamic flickering image pairs in evaluating ROP vascular progression were analyzed using the paired sample t-test, and confidence scores were analyzed using the Pearson chi-square test. Statistical significance was considered to be a two-sided P value <0.05.

RESULTS

Expert Participants and Overview of Responses

A total of 10 expert participants completed the web-based study. Seven (70%) out of the 10 experts were study center principal investigators for the CRYO-ROP or ETROP studies, 3/10 (30%) were certified investigators for either study, and 6/10 (60%) coauthored five or more peer-reviewed ROP papers. Examples of expert responses to interpreting vascular disease progression in static side-by-side and dynamic flickering image pairs are shown in Figure 1 and Supplemental Video 1 (available at AJO.com).

Accuracy

Table 1 summarizes the accuracy of determining ROP progression, or recognizing that there was no change in the images, among the 10 expert participants. For static side-by-side image pairs, experts made an accurate response in a mean (%) ± standard deviation (SD) of 11.4 (76%) ± 1.7 out of 15 image pairs. For dynamic flickering image pairs, experts made an accurate response in a mean (%) ± SD of 11.3 (75%) ± 1.7 out of 15 image pairs. There was no statistically significant difference in the accuracy of identifying progression of ROP disease severity between these two strategies (p=0.420, paired t-test).

Table 1.

Accuracy of responses by 10 expert participants for detecting retinopathy of prematurity (ROP) vascular disease progression using static side-by-side versus dynamic flickering image pairs.^*

Participant	Accuracy of Responses (%)
	Static side-by-side	Dynamic flickering
1	10/15 (67%)	11/15 (73%)
2	12/15 (80%)	13/15 (87%)
3	13/15 (87%)	13/15 (87%)
4	13/15 (87%)	13/15 (87%)
5	12/15 (80%)	10/15 (67%)
6	11/15 (73%)	9/15 (60%)
7	10/15 (67%)	9/15 (60%)
8	14/15 (93%)	13/15 (87%)
9	10/15 (67%)	10/15 (67%)
10	9/15 (60%)	12/15 (80%)
Mean (%) ± SD	11.4 (76%) ± 1.7	11.3 (75%) ± 1.7

^*Serial image pairs were taken from infants who eventually developed plus disease, and the reference standard was based on a diagnosis of more severity in retinal vascular changes in the image taken on the later date by both expert ROP examiners (MFC, RVPC). There was no statistically significant difference between static and dynamic image presentation strategies (p=0.420, paired t-test).

Examined another way, there were 150 corresponding pairs of 15 images that were interpreted by 10 expert participants. Among these corresponding image pair interpretations, 96/150 (64%) were accurately graded for progression of ROP disease severity by both the static side-by-side and dynamic flickering strategies, whereas 19/150 (13%) were not graded accurately by either strategy. Approximately equal numbers of image pairs were graded correctly by one strategy but incorrectly by the other strategy: only the static strategy was correct in 18/150 (12%), and only the dynamic strategy was correct in 17/150 (11%).

Speed and Confidence

Table 2 summarizes the speed of determining ROP progression, or recognizing that there was no change in the images, among the 10 expert participants. For static side-by-side image pairs, the mean ± SD (range) response time was 40.3 ± 18.3 (16–86) seconds. For dynamic flickering image pairs, the mean ± SD (range) response time was 24.7 ± 8.3 (15–44) seconds. Overall, diagnostic speed was significantly faster by interpretation of dynamic flickering image pairs than by interpretation of static side-by-side image pairs (p=0.002, paired t-test).

Table 2.

Speed of responses by 10 expert participants for detection retinopathy of prematurity vascular disease progression using static side-by-side versus dynamic flickering image pairs.^*

Participant	Mean response time (seconds)
	Static side-by-side	Dynamic flickering
1	52	28
2	43	24
3	26	19
4	46	22
5	36	21
6	36	28
7	16	15
8	39	28
9	86	44
10	26	18
Mean (SD)	40.3 (18.3)	24.7 (8.3)

^*Response times were measured by computer timestamps in a web-based image review system. Diagnostic speed was significantly faster by interpretation of dynamic flickering image pairs than by interpretation of static side-by-side image pairs (p=0.002, paired t-test).

With regard to confidence in determining ROP progression by static side-by-side compared to dynamic flickering images, there were 150 corresponding image pairs interpreted by the 10 expert participants. By interpretation of static side-by-side images, 87/150 (58%) experts were confident, 52/150 (35%) were somewhat confident, and 11/150 (7%) were not confident in recognizing disease progression. By interpretation of dynamic flickering image pairs, 116/150 (77%) experts were confident, 31/150 (21%) were somewhat confident, and 3/150 (2%) were not confident in recognizing disease progression. The difference in these confidence level distributions was statistically significant (p=0.001, chi-square test).

DISCUSSION

Retinal imaging allows objective documentation of examination findings, as well as comparison with old images to identify changes in ways that are not feasible using standard retinal drawings. Limited published research has systematically analyzed the identification of vascular disease progression using image interpretation. In this study, we examine the ability of experts to identify vascular disease progression toward plus disease, or to recognize absence of any vascular change, using static side-by-side image pairs compared to dynamic flickering image pairs created by image registration software. The key findings from this study are that: (1) The speed and confidence in identifying vascular disease progression were significantly better using dynamic flickering image pairs than using static side-by-side image pairs, and (2) There were no significant differences in accuracy of identifying disease progression using either strategy.

Recognizing the presence or absence of progression of vascular abnormality toward plus disease is critical for clinical ROP management. Based on the CRYO-ROP and ETROP studies, plus disease is sufficient to define severe ROP with high likelihood of retinal detachment and blindness without prompt treatment.^9,10 Furthermore, there is growing recognition that retinal vascular changes in ROP occur over a spectrum of abnormality, and that vascular changes insufficient for plus disease have utility in identifying infants at highest risk for progression to sight-threatening disease.^28–35 Faster rates of change in retinal vascular tortuosity and dilation have also been found to be prognostic indicators for progression to clinically-significant ROP.^28,30,33,35 Therefore, the ability to recognize earlier vascular changes (e.g. along the spectrum of pre-plus disease)³⁵ accurately and quickly using static or dynamic comparison of retinal images may provide advantages in clinical management compared to traditional care using serial indirect ophthalmoscopy with retinal drawings.

In this study, expert participants correctly identified vascular disease progression toward plus disease in 64% of image pair interpretations using both static side-by-side and dynamic flickering presentation of image pairs. In the remaining 36% of cases, experts incorrectly interpreted vascular disease progression in either or both static and dynamic presentations. Although this rate appears somewhat high, we suspect the ability of experts to identify progression of vascular changes using standard serial ophthalmoscopy would be even poorer because examiners would be forced to rely on retinal drawings and recollection of details from previous vascular appearance. This latter issue has never been systematically studied to our knowledge, and may warrant future research. Finally, we note that photographic documentation of ROP findings provides opportunities not only for comparison of serial images by experts, but also for objective quantification of retinal vascular parameters such as tortuosity and dilation using computer-based image analysis methods. Previous studies have suggested that semi-automated image analysis may identify plus disease with comparable or better accuracy than experts.^36–40 These quantitative techniques may provide a future method for reproducibly and precisely identifying disease progression from retinal photographs.

Interpretation of dynamic flickering images created by registration of superimposed serial images might be expected to improve recognition of vascular change by highlighting areas of disease progression. For example, it has been shown that humans fail or are delayed in detecting change between two similar images when the images are separated by blank screens, blinks, or saccades, a phenomenon known as “change blindness.” The ability to detect changes increases when the images are flickered back and forth without these disturbances.^41–45 However, the current study found no benefit in accuracy of dynamic flickering images over interpretation of side-by-side static images (Table 1). Previous studies have compared static side-by-side images and dynamic flickering images for detection of glaucomatous optic disc damage. In those previous studies, detection of disc hemorrhages and peripapillary atrophy showed increased sensitivity with flickering images, while detection of change in glaucomatous optic nerve head progression showed similar inter-grader agreement between techniques.^{26,27,46–49} However, with dynamic flickering images, assessments were driven more by blood vessel movements, while assessments of static side-by-side stereophotography were driven more by the appearance of disc hemorrhages.²⁶ It is possible that in the current study, experts may have been assessing different features depending on which image strategy was used. Future studies to investigate which features are being assessed between the two methods may help to improve accuracy for detection of change in ROP severity.

Response times with dynamic flickering image pairs in this study were significantly faster than with static side-by-side image pairs. Presumably this was due to the increased efficiency of assessing even subtle vascular changes with aligned serial images, although it is unclear whether this increased response time would have any practical significance. Additionally, experts were more confident about their responses for evaluating progression of ROP disease severity when viewing dynamic flickering images (confident on 77% of dynamic flickering image pairs vs. 58% of static side-by-side image pairs). There was no clear relationship among improved speed, confidence, and accuracy in this study, and the underlying factors relating these outcome measures may warrant further research. From a practical standpoint, a 2006 survey commissioned by the American Academy of Ophthalmology found that only half of pediatric ophthalmologists and retinal specialists are currently willing to manage ROP and over 20% of them plan to stop in the near future.⁵⁰ Many cited time, logistical difficulties and medicolegal liability as concerns. Image-based ROP diagnosis using telemedicine has been shown to be significantly faster than standard ophthalmoscopic bedside diagnosis.⁵¹ Digital registration with dynamic image flickering may provide a practical mechanism for improving the speed and efficiency of ROP management, while maintaining the accuracy and quality of diagnosis.^18,19,52

Several study limitations should be noted: (1) Several of the image pairs demonstrated very subtle vascular progression that may have been difficult to distinguish from control images showing no change (Figure 1 and Supplemental Video 1 available at AJO.com). This may have contributed toward lower accuracy of expert responses. All image pairs selected for this study to represent retinal vascular disease progression were diagnosed as eventually progressing to plus disease during serial clinical ophthalmoscopic examinations by the authors (RVPC, MFC). The amount of vascular disease progression between serial study images did not necessarily correspond to distinctions between “pre-plus” and “plus” (or “neither pre-plus nor plus” and “pre-plus). However, this decision to include subtle and potentially difficult comparisons was made in order to better distinguish between the discriminative abilities of static vs. dynamic image presentation strategies, and we do not believe that this would have systematically biased against either strategy. (2) This study examined only pairs of images. This decision was made to simplify the display and interpretation of both static and dynamic images. As a result, the findings may not be generalizable to real-world situations that might involve interpretation of multiple serial images to identify vascular disease progression. Future research examining disease progression of larger numbers of images over longer periods of time may be informative. (3) Study images displayed only the optic disc and posterior retinal vessels, without including any peripheral ROP features. This decision was made to focus the study on identification of change in vascular dilation and tortuosity, and to avoid confounding factors from recognition of peripheral ROP progression. Future studies examining progression of peripheral ROP using similar methods may highlight other benefits of static and dynamic image interpretation. (4) Study images were not paired with corroborative clinical diagnoses (e.g. “pre-plus”); instead vascular disease progression was defined based on the image taken later in infants who were eventually diagnosed with plus disease during clinical ophthalmoscopic examination. This decision was made because we believe that there is no standard method to define, quantify, and compare the amount of vascular abnormality in an image, and because incorporating clinical diagnoses such as “pre-plus” disease would complicate data analysis because of inter-observer variability. However, this raises the possibility that other features of digital retinal images could cause them to appear different in the absence of true “progression.” For example, image registration artifacts or differences in camera angle may influence interpretation of changes in vessel appearance.²⁶ Although these serial images may still be aligned using digital registration algorithms, dynamic viewing might enhance visualization of parallax vessel changes. These effects may have created difficulty in distinguishing between image pairs with subtle vascular changes and control images with no vascular changes. With more experience at evaluating progression of ROP disease with digital image registration, this problem may improve. (5) Capture of retinal images requires additional time and cost, which were not factored into the study analysis. Although studies have demonstrated that wide-angle retinal imaging for purposes of telemedicine management is faster and more cost-effective than standard ophthalmoscopic examination,^51,53 additional work in these areas may be warranted.

In summary, the ability to recognize vascular progression toward plus disease in ROP is important for clinical disease management, and may guide follow-up and management decisions. Retinal photography provides a mechanism for documentation of examination findings with greater objectivity and precision than with traditional retinal drawings. This study demonstrates that comparison of current images with previous ones can be an accurate way to identify vascular progression, and that interpretation of side-by-side vs. dynamic flickering image pairs had comparable accuracy. Expert study participants made faster diagnoses with greater confidence while reviewing dynamic flickering images, suggesting that future research involving digital image registration has potential to improve clinical ROP diagnosis and management.

Supplementary Material

01. Supplemental Video 1. Examples of identifying retinopathy of prematurity vascular disease progression from dynamic flickering image pairs.

Serial image pairs were taken from infants who eventually developed plus disease, and the reference standard was based on defining the more severe image as the one taken on the later date. (Top) 8/10 (80%) experts identified the more severe image (frame “A”) correctly. (Middle) 7/10 (70%) experts identified the more severe image (frame “A”) correctly. (Bottom) 10/10 (100%) experts responded correctly that images showed no disease progression.

Biographies

Jane S. Myung graduated from her vitreoretinal fellowship in June 2011 from Weill-Cornell Medical Center. She completed her residency training at New York University in June 2009, and her medical school training at the University of Iowa in May 2005. She is currently in private practice in New York City. She enjoys clinical teaching, and will continue to be involved in resident and fellow training.

Michael F. Chiang is Knowles Professor of Ophthalmology & Medical Informatics and Clinical Epidemiology at Oregon Health & Science University. His research involves telemedicine, image analysis, and electronic health record systems. Dr. Chiang received a B.S. in Electrical Engineering and Biology from Stanford University, an M.D. from Harvard Medical School and Harvard-MIT Division of Health Sciences and Technology, and an M.A. in Biomedical Informatics from Columbia University. He completed residency and pediatric ophthalmology fellowship training at the Johns Hopkins Wilmer Eye Institute.