Abstract
Importance
The present strategy to identify infants needing treatment for retinopathy of prematurity (ROP) requires repeated examinations of at risk infants by physicians. However, fewer than 10% ultimately require treatment. Retinal imaging by non-physicians with remote image interpretation by non-physicians may provide a more efficient strategy.
Objective
To evaluate the validity of a telemedicine system to identify infants who have sufficiently severe ROP to require evaluation by an ophthalmologist.
Design
An observational study of premature infants starting at 32 weeks postmenstrual age, was conducted from May 2011 through October 2013.
Setting
Neonatal intensive care units in 13 North American centers
Participants
Infants with birth weight (BW) <1251g
Interventions
Infants underwent regularly scheduled diagnostic examinations by an ophthalmologist and digital imaging by non-physician staff using a wide-field digital camera. Ophthalmologists documented whether an eye met criteria for RW-ROP, i.e. zone I ROP, stage 3 ROP, or plus disease. A standard 6-image set per eye was sent to a central server and graded by two trained, masked, non-physician readers. A Reading Supervisor adjudicated disagreements.
Main outcome
The validity of grading retinal image sets was based on the sensitivity and specificity for detecting RW-ROP compared to the criterion standard diagnostic exam.
Results
1257 infants (mean BW 864g, mean gestational age 27 weeks) underwent a median of 3 sessions of examinations and imaging. Diagnostic examination identified RW-ROP in 18.2% of eyes (19.4% of infants). Remote grading of images of an eye at a single session had sensitivity of 81.9% (95% confidence interval (CI): 77.4–85.6%) and specificity of 90.1% (95% CI: 87.9–91.8%). When both eyes are considered for the presence of RW-ROP, as would routinely be done in a screening, the sensitivity was 90.0% (95% CI 85.4–93.5%) with specificity of 87.0% (95% CI 84.0–89.5%), negative predictive value 97.3% and positive predictive value 62.5% at the observed RW-ROP rate of 19.4%.
Conclusion
When compared with the criterion standard diagnostic examination, these results provide strong support for the validity of remote evaluation by trained non-physician readers of digital retinal images taken by trained non-physician imagers from infants at risk for RW-ROP.
Introduction
Retinopathy of prematurity (ROP) is a leading cause of avoidable blindness in children worldwide1,2 and an increasing problem in those underserved areas of the US and Canada.3,4 In most settings, ophthalmologists travel regularly to neonatal units to perform serial eye examinations of infants at-risk, but fewer than 10% of infants examined require treatment.5 The interpretation of the ROP findings varies across examiners.6 A potential solution is to develop a telemedicine system using digital retinal imaging to detect sight-threatening disease. Such a solution must deal with key screening principles. Such evaluations differ fundamentally from clinical examinations by physicians and must provide compelling evidence for a high likelihood of altering the natural history of the disease in a significant proportion of individuals evaluated.7–9
In recent years, studies have evaluated the validity of retinal imaging to detect moderate to severe ROP.10–21 However, the sensitivities varied widely (33%11 to 100%10,12,14,18–21), as did the number of infants included (1010 -122218) and the primary outcome (plus disease10 to “suspect treatment requiring”18). Images for these studies were largely obtained by an ophthalmologist and were also graded by ophthalmologists. Given the current status of retinal imaging in ROP, the American Academy of Ophthalmology prioritized the need for “understanding …the place of digital wide-angle photography in the evaluation of at-risk infants.”22
Ells et al8 introduced the term “referral warranted ROP” (RW-ROP) in 2003 for use in telemedicine to describe eyes with ROP that had high risk characteristics defined as plus disease, ROP in zone I, or stage 3 ROP or above. Eyes with RW-ROP require careful ophthalmoscopic examination and many require treatment.
This large multi-center, National-Eye-Institute funded clinical study evaluated the validity of an ROP telemedicine system to detect eyes with RW-ROP.23 We compared remote evaluations of digital images to the findings of a criterion standard indirect ophthalmoscopic examination performed by experienced ophthalmologists.
Subjects and methods
Eligibility
Infants with birth weights (BW) <1251g meeting current ROP screening guidelines in twelve US centers and one Canadian center. Exclusion criteria were postmenstrual age (PMA) >39 weeks at first opportunity for imaging unless transferred in for ROP treatment, admission to a neonatal intensive care unit (NICU) with regressing or treated ROP, significant media opacity precluding visualization of the retina or major ocular or systemic congenital abnormality. The protocol and informed consent processes were approved by the institutional review boards of participating study centers. Informed consent was obtained for all participants.
Procedures
Infants underwent serial ROP imaging in both eyes using the RetCam Shuttle® (Clarity Medical Systems, Pleasanton, CA), in addition to a standard diagnostic examination by Study-certified ophthalmologists experienced in diagnosing ROP. The diagnostic examination results were classified as having clinical findings consistent with RW-ROP: zone I ROP, stage 3 ROP, or plus disease. The imagers were masked to results of the examination and the physicians were masked to images and subsequent grading. Timing of diagnostic examinations was determined by local Clinical Center criteria for usual clinical care and imaging sessions began at 32 weeks PMA. To prevent bias in terms of adverse events, the order of imaging and examinations alternated.
Imaging was conducted by 25 non-physician Study-certified personnel including NICU nurses (44%), neonatal nurse practitioners (24%), ophthalmic photographers (8%), ocular coherence tomography technician (4%), ophthalmic medical technologist (4%), and non-clinical backgrounds (16%). All participated in classroom and hands-on instruction in taking retinal images in infants, selecting and uploading images to the Inoveon ROP Data Center (IRDC) server in Oklahoma City. Imagers were certified after submission of quality pre-study image sets and passing a knowledge assessment test.
Images were obtained for each eye, using the video mode with a 130° wide-field imaging system. The imager selected still frames for a standard six-image set consisting of the pupil and five retinal fields, with optic disc central, nasal, temporal, superior, and inferior.
Demographic data and medical status of the infant before, during and after each session were collected. Surveillance for ocular and systemic complications and other adverse events was conducted.
The paired diagnostic examinations and imaging sessions continued as clinically indicated until the examining ophthalmologist noted any of the following: mature retinal vessels, immature Zone III on two occasions at least 7 days apart, ROP regressed or regressing on two occasions at least 7 days apart, treatment for severe ROP, or the infant reached 40 weeks PMA with no ROP or only stage 1 or 2 ROP.
Image grading
Image sets were uploaded as unmodified uncompressed files in.png format from the RetCam Shuttle to the IRDC server and used for remote grading by non-physician Trained Readers and by Expert Readers (ophthalmologists with ROP expertise). All Readers participated in joint didactic and image grading training sessions, underwent a certification process, and practiced with training image sets. All readers including three Trained Readers and three Expert Readers used the same software to access the images, used standardized workstations to grade images, and recorded findings on the same web-based data collection form. Each image set was graded independently by two Trained Readers with discrepancies adjudicated by the Reading Supervisor. All Readers were masked to results of diagnostic examinations, previous gradings for either eye of the infant, and demographic data.
All Readers determined the quality of each of the 5 retinal images in a set as good, acceptable, poor, or missing. They also determined, by quadrant, whether the posterior pole vessels were normal or sufficiently abnormal to be plus disease and determined zone of vascularization or the zone in which morphologic features consistent with ROP were present. ROP was determined by the presence of a demarcation line (stage 1), a ridge (stage 2), extraretinal neovascularization (stage 3), or retinal detachment (stage 4).
Image selection for grading
Trained Readers graded all images from 242 infants who developed RW-ROP based on the diagnostic examination. Approximately 80% of infants were not expected to develop RW-ROP; therefore a random sample of 613 infants (60.5%) who never developed RW-ROP was selected a priori. All image sets from this selected subsample of infants (n=242+613=855) were graded by the Trained Readers. A random sample of 200 of these 855 infants was also graded by Expert Readers (eFigure 1).
Statistical considerations
Sample Size
The sample size was determined by the need to have half width of the 95% Confidence Interval (95%CI) of sensitivity within 5%. Assuming sensitivity between 80–95%, approximately 250 infants with RW-ROP were needed for the study.
Sensitivity/specificity analysis
We compared, at the same session, RW-ROP finding (positive, negative, indeterminate) from evaluation of an image set to findings of the diagnostic examination consistent with RW-ROP (presence, absence, indeterminate). Sensitivity and specificity of image grading of detecting RW-ROP were calculated by using the results of the diagnostic examination as the criterion standard. Eyes with “indeterminate” status in the diagnostic examination were excluded from the sensitivity/specificity analysis. When the image set did not provide sufficient information for determining RW-ROP status, the eye was scored as RW-ROP positive in sensitivity/specificity analysis, since the primary aim of this study was to determine whether referral for a diagnostic examination was warranted.
We included only one session of digital image/diagnostic examinations from each eye in the primary analysis (single session per eye analysis). For sensitivity calculation, the session when the diagnostic examination first identified RW-ROP was used, while a random session was chosen for each eye that did not develop RW-ROP for the specificity calculation.
Sensitivity was calculated as the proportion of RW-ROP positive image gradings when examination indicated RW-ROP presence, and specificity was calculated as the proportion of RW-ROP negative image gradings when examination indicated RW-ROP absence. The 95% CI for sensitivity and specificity were calculated, with inter-eye correlation adjusted by generalized estimating equations24 using the sandwich robust estimate of variance.25 The pre-specified subgroup analyses of sensitivity/specificity were performed by birth weight, gestational age, and image quality.
Secondary analyses of sensitivity/specificity were conducted at the infant level by comparing the presence/absence of RW-ROP on examination versus RW-ROP positive/negative from image grading at one selected session (single session per infant analysis) and any sessions (any session per infant analysis). The sensitivity and specificity for detecting the infants undergoing ROP treatment were also calculated based on the last session before treatment (last session per infant analysis).
For per infant analysis, the negative predictive value (NPV) and the positive predictive value (PPV) were calculated based on their corresponding sensitivity, specificity, and the observed rate of RW-ROP. All the statistical analyses were performed using SAS v9.3 (SAS Institute Inc, Cary, NC).
Results
Population and ROP status
1284 infants with BW <1251g were enrolled from May 2011 through October 2013 (eFigure 1). 27 (2.1%) were discharged prior to a diagnostic examination. Table 1 provides characteristics of 1257 study infants who completed at least one diagnostic examination. Mean BW was 864g and mean GA 27 weeks. Most infants had BW ≤1000g including 34.1% of infants ≤750g and 36.2% from 751–1000g. About half had GA <27 weeks (51.9%) and few infants had GA ≥31 weeks (5.9%). More than half were non-Hispanic white, 29.3% black and 49.2% female, 63.0% were born at the enrolling Clinical Center and 29.8% were multiple births of which 84.3% were twin births.
Table 1.
Characteristics of study infants (n=1257)
| Characteristics at enrollment | # of infants | (%) |
|---|---|---|
| Birth Weight (grams) | ||
| Mean (standard deviation) | 864 (212) | |
| Median (1st quartile, 3rd quartile) | 860 (690, 1040) | |
| Gestational Age (weeks) | ||
| Mean (standard deviation) | 27 (2.2) | |
| Median (1st quartile, 3rd quartile) | 26 (25, 28) | |
| Ethnicity of infant | ||
| Hispanic or Latino | 122 | 9.7% |
| Not Hispanic or Latino | 1085 | 86.3% |
| Unable to answer | 50 | 4.0% |
| Race of infant | ||
| White only | 705 | 56.1% |
| Asian only | 18 | 1.4% |
| Black only | 368 | 29.3% |
| American Indian only | 21 | 1.7% |
| Native Hawaiian or Pacific Islander only | 4 | 0.3% |
| Mixed | 18 | 1.4% |
| Unable to answer | 123 | 9.8% |
| Any ROP | ||
| No | 456 | 36.3% |
| Yes | 801 | 63.7% |
| RW-ROP | ||
| No | 983 | 78.2% |
| Yes | 244 | 19.4% |
| Unknown | 30 | 2.4% |
| Laterality of RW-ROP* | ||
| Unilateral | 30 | 12.3% |
| Bilateral | 214 | 87.8% |
ROP = retinopathy of prematurity,
RW-ROP = referral warranted retinopathy of prematurity.
Among those infants with RW-ROP.
Among 1257 infants who had eye examinations, the mean number of examinations per infant was 3.4 (SD 2.1), median 3 (range: 1–12) and 40% had 4 or more examinations. The median interval between examinations was 9 days (range: 1–54), with 11% of exams within one week and 88% within 2 weeks.
At least one imaging session was conducted in 1241 infants. The mean number of imaging sessions per infant was 3.2 (SD 2.0), median 3 (range: 1–12), and 36% had 4 or more imaging sessions.
Among the 5520 image sets selected for Trained Reader grading, a total of 27,600 possible individual retinal images were evaluated for availability and image quality. Among these images, 3% were missing, 91% were adequate quality, 6% were poor quality.
ROP was noted on examination in 801 (63.7%) infants (Table 1). RW-ROP was noted in one or both eyes of 244 infants (19.4%) and was bilateral in 87.8% of infants. The presence of RW-ROP could not be determined for only 2.4% of infants. Clinical findings consistent with RW-ROP were detected in 458 eyes (18.2%), most frequently due to stage 3 or worse alone (48.5%) or in combination with zone I ROP and/or plus disease (33.6%, Table 2). All 3 components (plus disease, zone I ROP, and stage 3 ROP) were present rarely (3.3%).
Table 2.
Combination of RW-ROP* components among eyes with RW-ROP findings from diagnostic examinations (n=458 eyes)
| Combination of RW-ROP components in an eye | Number of eyes (%) |
|---|---|
| Plus and stage 3 or worse | 121 (26.4%) |
| Only stage 3 or worse | 222 (48.5%) |
| Only Zone I | 44 (9.6%) |
| Zone I and stage 3 or worse | 18 (3.9%) |
| Plus disease, Zone I and Stage 3 or worse | 15 (3.3%) |
| Only plus disease | 33 (7.2%) |
| Plus and Zone I | 5 (1.1%) |
RW-ROP = referral warranted retinopathy of prematurity.
Comparison of results from examination and imaging grading
Among the 5520 pairs of diagnostic examinations and image gradings, their RW-ROP status was in agreement in 78.6% (Table 3). In the 813 pairs (14.7%) with RW-ROP findings on diagnostic examination, image grading by Trained Readers detected one or more of the components of RW-ROP in 77.7% of image sets, while in 19.8% the image grading did not detect RW-ROP and 2.5% were indeterminate. In the 4648 pairs without RW-ROP on diagnostic examination, Trained Readers agreed in 3703 (79.7%) image gradings, while in 854 pairs (18.4%), Trained Readers detected findings consistent with RW-ROP. In 91 pairs, RW-ROP status was indeterminate.
Table 3.
Cross-tabulation of RW-ROP* findings from diagnostic examination vs. image grading of all sessions per eye
| RW-ROP from diagnostic examination | RW-ROP from image grading
|
|||
|---|---|---|---|---|
| No | Yes | Indeterminate | Total | |
| No | 3703 (67.1%) | 854 (15.5%) | 91 (1.7%) | 4648 (84.2%) |
| Yes | 161 (2.9%) | 632 (11.5%) | 20 (0.4%) | 813 (14.7%) |
| Indeterminate* | 47 (0.9%) | 9 (0.2%) | 3 (0.1%) | 59 (1.1%) |
| Total | 3911 (70.9%) | 1495 (27.1%) | 114 (2.1%) | 5520 (100.0%) |
RW-ROP = referral warranted retinopathy of prematurity defined as zone I ROP, stage 3 ROP or plus disease.
In 13 sessions, images were taken and graded, but no diagnostic examination was performed.
Sensitivity and specificity for RW-ROP
When image sets for a single session were graded by the Trained Readers and compared to the results of diagnostic examinations (1709 image sets in which the first session when RW-ROP findings were diagnosed and a random session for infants without RW-ROP)(Table 4), the sensitivity for detection of RW-ROP was 81.9% (95% CI 77.4–85.6%) with a specificity of 90.1% (95% CI 87.9–91.8%).
Table 4.
Sensitivity and specificity of image grading for detecting RW-ROP*
| Eyes/Infants with RW-ROP findings based on the diagnostic examination | Eyes/Infants without RW-ROP findings based on the diagnostic examination | ||||||
|---|---|---|---|---|---|---|---|
| Analysis Approach | # of eyes/infants for analysis | Positive from image grading | Negative from image grading | Sensitivity (95% CI) | Positive from image grading | Negative from image grading | Specificity (95% CI) |
| Single Session Per Eye**,§ | 1709 eyes | 366 | 81 | 81.9% (77.4–85.6%) | 125 | 1133 | 90.1% (87.9–91.8%) |
| Single Session Per Infant**,§ | 855 infants | 215 | 24 | 90.0% (85.4–93.5%) | 80 | 534 | 87.0% (84.0–89.5%) |
| Any session per infant† | 855 infants | 232 | 7 | 97.1% (94.0–98.6%) | 144 | 453 | 75.9% (72.2–79.1%) |
| Last session before treatment per infant‡ | 855 infants | 159 | 3 | 98.2% (94.4–99.4%) | 137 | 554 | 80.2% (77.0–83.0%) |
95% CI = 95% confidence interval
RW-ROP = referral warranted retinopathy of prematurity defined as the presence of zone I ROP, stage 3 ROP or plus disease.
For infants with RW-ROP, the images from the session when diagnostic examination first identified RW-ROP were used. For infants without RW-ROP findings on any diagnostic examination, a random image session was selected.
RW-ROP status could not be determined in 2 eyes based on the diagnostic exam and were excluded from analysis, and 2 eyes did not have images taken (while their fellow eye did have images) in the random selected session.
2 infants with RW-ROP status unknown in either eye at selected session were excluded.
Comparison of an infant ever had RW-ROP (yes/no) from results of all sessions vs. ever had RW-ROP (yes/no) from diagnostic examination results of all sessions, irrespective of whether the RW-ROP from image grading and diagnostic examination was at the same session or not. 19 infants with RW-ROP status unknown in either eye were excluded.
As shown in Table 4, when both eyes of the infant are considered as would routinely be done in a clinical setting, the sensitivity for RW-ROP was 90.0% (95% CI 85.4–93.5%) with specificity of 87.0% (95% CI 84.0–89.5%), negative predictive value (NPV) 97.3%, positive predictive value (PPV) 62.5% at the RW-ROP prevalence rate of the study (19.4%).
When RW-ROP results from any session of image grading and diagnostic examination are paired for the infant, the sensitivity increased to 97.1% (95% CI 94.0–98.6%). Specificity for this comparison was 75.9% (95% CI 72.2–79.1%), NPV 99.1%, PPV 49.2%.
We also calculated the sensitivity and specificity of this telemedicine system to detect RW-ROP in infants who underwent treatment in one or both eyes. When the last session before treatment was analyzed, sensitivity was 98.2% (95% CI 94.4–99.4%) with specificity 80.2 % (95% CI 77.0–83.0%), NPV 99.6% and PPV 44.3%, at a 13.8% treatment-requiring ROP rate. Only 3 of 162 infants treated by Clinical Center ophthalmologists did not have RW-ROP detected on the last image grading before treatment. On diagnostic examination, one infant had zone I stage 3 in both eyes and the other two infants had plus disease in both eyes.
Expert and Trained Readers independently graded a random sample of 1312 image sets from 100 infants with RW-ROP and 100 without RW-ROP. Using all 1312, Expert Readers had a lower sensitivity of 85.9% (95% CI: 80.8–89.8%) compared to 91.4% (95% CI: 86.1–94.8%) for Trained Readers and a lower specificity of 56.5% (95% CI: 51.9–61.0%) versus 73.3% (95% CI: 67.6–78.3%) for detecting an eye with RW-ROP.
Timing of RW-ROP detection on image grading
The timing of detection of RW-ROP was compared between the image grading and diagnostic examination. In 87.5% (391/447) of the cases, RW-ROP was detected on image grading before or at the same examination that documented RW-ROP findings. RW-ROP was detected on image grading an average of 15 days (SD 11) earlier than the examination in 42.7% (200/447) of cases. Image grading did not detect RW-ROP at any image grading in 7.1% (32/447) cases. In 5.4% (24/447) cases, RW-ROP was detected an average of 15 days (SD 13) after the examination documented RW-ROP. The results are very similar when analyzed by infant (not shown).
Sensitivity and specificity by BW, gestational age (GA), and image quality
In the single session per eye analysis, sensitivity of RW-ROP detection decreased with increasing BW (Table 5). This pattern was not observed for GA.
Table 5.
Subgroup analyses of sensitivity and specificity of RW-ROP* from Trained Readers Grading – Single Session Per Eye Analysis**
| # of eyes RW-ROP present from diagnostic examination | # of eyes RW-ROP positive from image grading | Sensitivity (95% CI) | # of eyes RW-ROP absent from diagnostic examination | # of eyes RW-ROP negative from image grading | Specificity (95% CI) | |
|---|---|---|---|---|---|---|
| Birth weight | ||||||
| ≤750 g (n=655) | 317 | 264 | 83.3% (77.9–87.6%) | 338 | 274 | 81.1% (75.7–85.5%) |
| >750, ≤900 g (n=353) | 74 | 62 | 83.8% (73.3–90.7%) | 279 | 243 | 87.1% (81.7–91.1%) |
| >900 g (n=697) | 56 | 40 | 71.4% (55.5–83.4%) | 641 | 616 | 96.1% (94.0–97.5%) |
| Gestation age | ||||||
| ≤ 24 weeks (n=186) | 128 | 106 | 82.8% (73.0–89.6%) | 58 | 47 | 81.0% (67.9–89.6%) |
| >24, ≤ 26 weeks (n=528) | 214 | 183 | 85.5% (79.9–89.8%) | 314 | 253 | 80.6% (74.9–85.2%) |
| >26 weeks (n=991) | 105 | 77 | 73.3% (62.1, 82.2%) | 886 | 833 | 94.0% (91.9–95.6%) |
| Good/adequate quality in all 5 retinal images | ||||||
| No (n=400) | 75 | 51 | 68.0% (56.0–78.0%) | 325 | 288 | 88.6% (84.2–91.9%) |
| Yes (n=1305) | 372 | 315 | 84.7% (80.0–88.4%) | 933 | 845 | 90.6% (88.2–92.5%) |
95% CI = 95% confidence interval
RW-ROP = referral warranted retinopathy of prematurity defined as the presence of zone I ROP, stage 3 ROP or plus disease.
For infants with RW-ROP, the images from the session when diagnostic examination first identified findings consistent with RW-ROP were used. For infants without RW-ROP, a random image session was selected.
Quality of the images submitted was important. When all 5 retinal images were judged to be of good or acceptable image quality by Trained Readers, the sensitivity was 84.7% compared with 68.0% when 4 or fewer images were of good or acceptable quality. Specificity for this comparison was similar.
Discussion
The e-ROP study results provide strong support for the validity of using a telemedicine system comprised of trained non-physician imagers and readers to detect RW-ROP in infants at risk. When image set grading of an eye was compared to the results of its paired diagnostic examination, the sensitivity for detection of RW-ROP was 81.9% with a specificity of 90.1%. When both eyes of an infant were considered, the sensitivity increased to 90.0% with specificity of 87.0%, NPV 97.3%, PPV 62.5% at the observed RW-ROP rate of 19.4%. Importantly, among infants treated for ROP, the sensitivity of image grading increased to 98.2%.
Among the strengths of the e-ROP study is the successful use of non-physician imagers. The imagers were trained to capture standard image sets for grading by non-physician Trained Readers using a standard grading protocol. The diagnostic examinations were performed by ophthalmologists with extensive ROP experience. Further, we elected to have the results of non-physician Trained Readers as the primary outcome measure since, as in other retinal disorders,26, 27 we confirmed in this study that dedicated and trained readers would perform at least as well as ophthalmologist experts and their availability was likely to be higher and sustained. Standardized imaging and grading protocols improves the generalizability of this ROP telemedicine system.
This study is limited by the difficulty of comparing image grading results to a criterion standard with known inherent variability.28, 29 Among eyes judged at treatment level by the first examiner in the Cryotherapy for ROP Trial, there was disagreement on a second “confirming” examination in 12% of the cases.6 Other studies have highlighted the variability in diagnosing plus disease and disease zone among ophthalmologist examiners.30–33 This variability across examiners may well help explain some of the false positive and false negative cases documented on image grading in the current study. In addition, the e-ROP study limited enrollment to infants at high risk i.e. <1251g BW, and may not be generalizable to all infants eligible for ROP examinations. Further, the e-ROP study was not designed to assess the time to reporting results to the clinical center or to provide rapid feedback requesting additional images when image quality was poor. Sensitivity improved substantially with better image quality so efforts to improve image quality are warranted.
One of the most important factors in evaluating telemedicine systems for ROP to consider is how many infants with serious ROP might be missed. In this study with a RW-ROP rate of 19.4%, we found the NPV was high (97.3%). Despite this high NPV value, if implementing such a system for clinical care, there will be a small number of infants in whom image grading does not detect RW-ROP and who are at risk for progression to blindness. What safeguards need to be in place for such infants? One might consider a more frequent imaging schedule than used in this study that mimicked current clinical care or also consider adding other algorithms such as WINROP34 or CHOP-ROP,35 which consider early weight patterns in predicting ROP risk, to determine those infants at highest risk.
Adopting a telemedicine system approach in ROP is influenced by factors other than validity including the availability of ROP specialists, the number of examinations required per week, and the prevalence of RW-ROP in a NICU. Licensing and liability issues must be dealt with, as well as establishing a consistent and reliable reading center. Procedures to address poor image quality are needed. Parental and NICU staff acceptance of a screening program that replaces a physician diagnostic examination with non-physician imaging needs further assessment. However, the results of this study provide important information about the validity of the e-ROP telemedicine system in managing ROP as we move forward to address the broader use of digital retinal imaging in NICUs in the US and other regions in the world.
Acknowledgments
We would like to acknowledge gratefully the contributions of Clare Gilbert, Alistair Fielder and Judy Alexander for their contributions to the early development of the e-ROP study.
Funded by National Eye Institute of the National Institutes of Health, Department of Health and Human Services. U10 EY017014, clinicaltrials.gov national registry number: NCT01264276
References
- 1.Gilbert C, Fielder A, Gordillo L, Quinn G, Semiglia R, Visintin P. Zin A on behalf of the International NO-ROP Group. Characteristics of infants with severe retinopathy of prematurity in countries with low, moderate and high levels of development: Implications for screening programs. Pediatrics. 2005;115:e518–e525. doi: 10.1542/peds.2004-1180. [DOI] [PubMed] [Google Scholar]
- 2.Gilbert C. Retinopathy of prematurity: A global perspective of the epidemics, population of infants at risk and implications for control. Early Human Development. 2008;84:77–82. doi: 10.1016/j.earlhumdev.2007.11.009. [DOI] [PubMed] [Google Scholar]
- 3.Kemper AR, Freedman SF, Wallace DK. Retinopathy of prematurity care: patterns of care and workforce analysis. J AAPOS. 2008;12(4):344–348. doi: 10.1016/j.jaapos.2008.02.012. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 4.Kemper AR, Wallace DK. Neonatologists’ practices and experiences in arranging retinopathy of prematurity screening services. Pediatrics. 2007;120(3):527–531. doi: 10.1542/peds.2007-0378. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 5.Early Treatment For Retinopathy Of Prematurity Cooperative Group. Revised indications for the treatment of retinopathy of prematurity: results of the early treatment for retinopathy of prematurity randomized trial. Arch Ophthalmol. 2003;121(12):1684–94. doi: 10.1001/archopht.121.12.1684. [DOI] [PubMed] [Google Scholar]
- 6.Reynolds JD, Dobson V, Quinn GE, et al. Evidence-based screening criteria for retinopathy of prematurity: natural history data from the CRYO-ROP and LIGHT-ROP studies. Arch Ophthalmol. 2002;120(11):1470–1476. doi: 10.1001/archopht.120.11.1470. [DOI] [PubMed] [Google Scholar]
- 7.Lee P. Telemedicine: Opportunities and challenges for the remote care of diabetic retinopathy. Arch Ophthalmol. 1999;117:1630–1. doi: 10.1001/archopht.117.12.1639. [DOI] [PubMed] [Google Scholar]
- 8.Adriansen WJ, Mathei C, Buntinx FJ, Arbyn M. A framework provided an outline toward the proper evaluation of potential screening strategies. J Clin Epidemiology. 2013;66:639–47. doi: 10.1016/j.jclinepi.2012.09.018. [DOI] [PubMed] [Google Scholar]
- 9.Holland W. Screening for disease – considerations for policy. Health Policy Bulletin for the European Observator on Health Systems and Policies. 2006;8(3):1–4. [Google Scholar]
- 10.Schwartz SD, Harrison SA, Ferrone PJ, Trese MT. Telemedical evaluation and management of retinopathy of prematurity using a fiberoptic digital fundus camera. Ophthalmology. 2000;107(1):25–8. doi: 10.1016/s0161-6420(99)00003-2. [DOI] [PubMed] [Google Scholar]
- 11.Yen KG, Hess D, Burke B, Johnson RA, Feuer WJ, Flynn JT. Telephotoscreening to detect retinopathy of prematurity: preliminary study of the optimum time to employ digital fundus camera imaging to detect ROP. J AAPOS. 2002;6(2):64–70. [PubMed] [Google Scholar]
- 12.Ells AL, Holmes JM, Astle WF, Williams G, Leske DA, Fielden M, Uphill B, Jennett P, Hebert M. Telemedicine approach to screening for severe retinopathy of prematurity: a pilot study. Ophthalmology. 2003;110(11):2113–7. doi: 10.1016/S0161-6420(03)00831-5. [DOI] [PubMed] [Google Scholar]
- 13.Chiang MF, Keenan JD, Starren J, Du YE, Schiff WM, Barile GR, Li J, Johnson RA, Hess DJ, Flynn JT. Accuracy and reliability of remote retinopathy of prematurity diagnosis. Arch Ophthalmol. 2006;124(3):322–7. doi: 10.1001/archopht.124.3.322. [DOI] [PubMed] [Google Scholar]
- 14.Wu C, Petersen RA, VanderVeen DK. RetCam imaging for retinopathy of prematurity screening. J Aapos. 2006 Apr;10(2):107–11. doi: 10.1016/j.jaapos.2005.11.019. [DOI] [PubMed] [Google Scholar]
- 15.Chiang MF, Wang L, Busuioc M, Du YE, Chan P, Kane SA, Lee TC, Weissgold DJ, Berrocal AM, Coki O, Flynn JT, Starren J. Telemedical retinopathy of prematurity diagnosis: accuracy, reliability, and image quality. Arch Ophthalmol. 2007;125(11):1531–8. doi: 10.1001/archopht.125.11.1531. [DOI] [PubMed] [Google Scholar]
- 16.The photographic screening for retinopathy of prematurity study (photo-ROP) Primary outcomes. Retina. 2008;28(3 Suppl):S47–54. doi: 10.1097/IAE.0b013e31815e987f. [DOI] [PubMed] [Google Scholar]
- 17.Dhaliwal C, Wright E, Graham C, McIntosh N, Fleck BW. Wide-field digital retinal imaging versus binocular indirect ophthalmoscopy for retinopathy of prematurity screening: a two-observer prospective, randomised comparison. Br J Ophthalmol. 2009;93(3):355–9. doi: 10.1136/bjo.2008.148908. [DOI] [PubMed] [Google Scholar]
- 18.Lorenz B, Spasovska K, Elflein H, Schneider N. Wide-field digital imaging based telemedicine for screening for acute retinopathy of prematurity (ROP). Six-year results of a multicentre field study. Graefes Arch Clin Exp Ophthalmol. 2009;247(9):1251–62. doi: 10.1007/s00417-009-1077-7. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 19.Silva RA, Murakami Y, Lad EM, Moshfeghi DM. Stanford University network for diagnosis of retinopathy of prematurity (SUNDROP): 36-month experience with telemedicine screening. Ophthalmic Surg Lasers Imaging. 2011;42(1):12–9. doi: 10.3928/15428877-20100929-08. [DOI] [PubMed] [Google Scholar]
- 20.Dai S, Chow K, Vincent A. Efficacy of wide-field digital retinal imaging for retinopathy of prematurity screening. Clinical & Experimental Ophthalmology. 2011;39(1):23–9. doi: 10.1111/j.1442-9071.2010.02399.x. [DOI] [PubMed] [Google Scholar]
- 21.Weaver DT, Murdock TJ. Telemedicine detection of type 1 ROP in a distant neonatal intensive care unit. J AAPOS. 2012;16:229–233. doi: 10.1016/j.jaapos.2012.01.007. [DOI] [PubMed] [Google Scholar]
- 22.Chiang MF, Melia M, Buffenn AN, Lambert SR, Recchia FM, Simpson JL, Yang MB. Detection of Clinically Significant Retinopathy of Prematurity Using Wide-angle Digital Retinal Photography: A Report by the American Academy of Ophthalmology. Ophthalmology. 2012;119(6):1272–80. doi: 10.1016/j.ophtha.2012.01.002. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 23.Telemedicine Approaches to Evaluating Acute-phase Retinopathy of Prematurity Cooperative Group. Telemedicine Approaches to Evaluating Acute-phase Retinopathy of Prematurity: Study Design. Ophthalmic Epidemiology. doi: 10.3109/09286586.2014.926940. in press. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 24.Liang K, Zeger S. Longitudinal data analysis using generalized linear models. Biometrika. 1986;73:13–22. [Google Scholar]
- 25.Smith PJ, Hadgu A. Sensitivity and specificity for correlated observations. Statistics in Medicine. 1992;11:1503–1509. doi: 10.1002/sim.4780111108. [DOI] [PubMed] [Google Scholar]
- 26.Feman SS, Leonard-Martin TC, Andrews JS, Armbruster CC, Burdge TL, Debalak JD, Lanier A, Fisher AG. A quantitative system to evaluate diabetic retinopathy from fundus photographs. Invest Ophthalmol Vis Sci. 1995;36:174–81. [PubMed] [Google Scholar]
- 27.Patra S, Gomm EMW, Macipe M, Bailey C. Interobserver agreement between primary graders and an expert grader in the Bristol and Weston diabetic retinopathy screening program: a quality assurance audit. Diabet Med. 2009;26:820–3. doi: 10.1111/j.1464-5491.2009.02767.x. [DOI] [PubMed] [Google Scholar]
- 28.Trese MJ. What is the real gold standard for ROP screening? Editorial. Retina. 2008;28:S1–2. doi: 10.1097/IAE.0b013e31816a5587. [DOI] [PubMed] [Google Scholar]
- 29.Phelps DL. It’s plus disease, isn’t it? Arch Ophthalmol. 2007;125:963. doi: 10.1001/archopht.125.7.963. [DOI] [PubMed] [Google Scholar]
- 30.Chiang MF, Jiang L, Gelman R, Du YE, Flynn JT. Interexpert agreement of plus disease diagnosis in retinopathy of prematurity. Arch Ophthalmol. 2007;125(7):875–80. doi: 10.1001/archopht.125.7.875. [DOI] [PubMed] [Google Scholar]
- 31.Chiang MF, Gelman R, Martinez-Perez ME, Du YE, Casper DS, Currie LM, Shah PD, Starren J, Flynn JT. Image analysis for retinopathy of prematurity Diagnosis. J AAPOS. 2009;13:438–445. doi: 10.1016/j.jaapos.2009.08.011. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 32.Darlow BA, Elder MJ, Horwood LJ, Donoghue DA, Henderson-Smart DJ. and on behalf of the Australian and New Zealand Neonatal Network. Does observer bias contribute to variations in the rate of retinopathy of prematurity between centres? Clinical and Experimental Ophthalmology. 2008;36:43–46. doi: 10.1111/j.1442-9071.2007.01652.x. [DOI] [PubMed] [Google Scholar]
- 33.Wallace DK, Quinn GE, Freedman SF, Chiang MF. Agreement among pediatric ophthalmologists in diagnosing plus and pre-plus disease in retinopathy of prematurity. Journal of AAPOS. 2008;12:352–6. doi: 10.1016/j.jaapos.2007.11.022. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 34.Löfqvist C, Hansen-Pupp I, Andersson E, Holm K, Smith LEH, Ley D, Hellström A. Validation of a New Retinopathy of Prematurity Screening Method Monitoring Longitudinal Postnatal Weight and Insulinlike Growth Factor I. Arch Ophthalmol. 2009;127(5):622–627. doi: 10.1001/archophthalmol.2009.69. [DOI] [PubMed] [Google Scholar]
- 35.Binenbaum G, Ying GS, Quinn GE, Dreiseitl S, Karp K, Roberts RS, Kirpalani H. A clinical prediction model to stratify retinopathy of prematurity risk using postnatal weight gain. Premature Infants in Need of Transfusion Study Group. Pediatrics. 2011;127(3):e607–14. doi: 10.1542/peds.2010-2240. [DOI] [PMC free article] [PubMed] [Google Scholar]