Skip to main content
NCBI home page
NIHPA Author Manuscripts logoLink to NIHPA Author Manuscripts
. Author manuscript; available in PMC: 2016 May 9.
Published in final edited form as: Ophthalmic Epidemiol. 2014 Jun 23;21(4):256–267. doi: 10.3109/09286586.2014.926940

Telemedicine Approaches to Evaluating Acute-phase Retinopathy of Prematurity: Study Design

e-ROP Cooperative Group
PMCID: PMC4861056  NIHMSID: NIHMS769644  PMID: 24955738

Abstract

Purpose

Detecting sight-threatening retinopathy of prematurity (ROP) relies on a diagnostic examination (DE) performed by an experienced ophthalmologist. An alternative may be a telemedicine system where retinal images of at-risk infants are graded by readers to determine features of ROP indicating the need for a DE.

Methods

The multicenter “Telemedicine Approaches to Evaluating Acute-phase ROP” (e-ROP) Study is a cohort study of 2,000 infants with birth weights <1251g. At each visit, ophthalmologists perform DEs and non-physician imagers obtain iris and five retinal images with the disc positioned in the center, right, left, up anddown. Images are uploaded to a secure server for grading by non-physician readers for the detection of plus disease, stage 3 ROP and/or zone I disease, any of which indicates “referral-warranted ROP (RW-ROP).” Images from all infants with RW-ROP and a random sample of infants without RW-ROP (based on DEs) are selected for grading. Gradings are compared to DEs to determine the validity and evaluate reliability, feasibility, safety, and cost-effectiveness of the telemedicine system.

Results

e-ROP is conducted in 12 Clinical Centers in the US and Canada with Study Headquarters, the Data Coordinating Center and the Image Reading Center in Philadelphia and the ROP Data Center in Oklahoma City. 27 Study Center Coordinators, 34 ophthalmologists; 26 imagers, and 4 readers have been certified. All study data are submitted using a secure web-based system.

Conclusion

The design and findings of this study will be useful to conduct other ROP studies or evaluate telemedicine for other diseases.

Keywords: retinopathy of prematurity, telemedicine, prematurity, childhood blindness

Introduction

ROP is a leading cause of avoidable childhood blindness and is becoming an increasing problem in underserved areas of the US and Canada as well as in countries with rapidly developing neonatal intensive care systems.1, 2 At present, premature infants requiring treatment are identified by ophthalmologists with ROP expertise, who visit neonatal units to examine at risk infants and often must return for multiple examinations. This is an inefficient use of skilled and costly personnel because fewer than 10% of infants examined require treatment.3–5 There are two additional challenges in providing the required examinations for infants at risk for ROP. First, there are too few experts in ROP worldwide to meet the growing demand for ROP evaluations. The limited number of providers with the specialized training required has dwindled in the US and Canada as some ophthalmologists have stopped examining infants because of workflow inconvenience, low reimbursement and/or the substantial legal liability associated with misdiagnosis or missed examinations.6–7 Second, there is an inherent variability in the examination results8 which may lead to missed opportunities to treat at an appropriate time. A potential solution to address both decreasing availability of clinicians who can evaluate ROP and variability in the quality of evaluations is to develop a telemedicine system that uses retinal imaging to detect sight-threatening disease in an efficient and cost-effective manner. Ells et al 9 introduced the term “referral-warranted ROP” (RW-ROP) in 2003 to describe eyes with ROP that had high risk characteristics such as stage 3 ROP, plus disease or any very posterior ROP (i.e., in zone I) (see Figure 1 for an example of RW-ROP). Eyes with RW-ROP require careful evaluation by an ophthalmologist with a substantial proportion requiring treatment.

Figure 1.

Figure 1

Figure 1

Figure 1

Figure 1

Figure 1

Figure 1

Open in a new tab

a–f: An example of images collected at each imaging session by the trained retinal imager. The standard image set was collected for each eye of the baby and consisted of an image of the anterior segment (Figure 1a) and five retinal fields where the image had the: (1) disc central (Figure 1b), (2) disc right (Figure 1c), (3) disc left (Figure 1d), (4) disc up (Figure 1e), and (5) disc down (Figure 1f). This image set demonstrates the presence of plus disease, stage 3 ROP, and zone I ROP – any one of which would qualify the eye as having RW-ROP by a reader.

Several approaches to capture still images in wide or narrow fields-of-view, as well as video sequences with indirect ophthalmoscopy, have been used to document retinal findings in ROP.9,10,11 Currently, the most commonly used digital camera for ROP is a wide-field digital camera (WF-DI), the RetCam™ (Clarity Medical Systems, Inc, Pleasanton, CA) that provides a 130-degree field of view. Harrison first described imaging the retina of premature infants using the RetCam system in 1998.12 To date, there have been a series of studies using various approaches and examining different components of an overall system for ROP telemedicine.13–27 These studies had widely different, sometimes contradictory results and variations in the reported sensitivity and specificity based on the criteria used to interpret the retinal image. In most previous studies both imaging and image reading had been done by ophthalmologists. A recent report by Chiang et al emphasized the strengths and weaknesses of reported studies that used a wide-angle camera to capture images of infants at risk for ROP.28

We report the design of e-ROP, a large, multicenter, adequately powered study, supported by the National Eye Institute of the National Institutes of Health. This study is undertaken to evaluate the validity, reliability, feasibility, safety and relative cost-effectiveness of a digital imaging system used by non-ophthalmologists to identify infants with RW-ROP.

Materials and methods

Study Organization and Personnel

The Study is directed by the Office of the Study Chair under the guidance of the Study Principal Investigator (PI) and the Project Director at the Children’s Hospital of Philadelphia. The PI also serves as the Chair of the Executive, Operations and Editorial Committees and acts as the Clinical Adjudicator for selected images. The Data Coordinating Center (DCC) and the Image Reading Center are located at the University of Pennsylvania. The DCC consists of a Director, systems analyst and biostatistician and the Image Reading Center consists of a Director who manages the operations of the four Trained Readers, a Lead Expert Reader, and two Expert Readers. The Director also acts as a Supervising Reader in the image grading adjudication process. The Inoveon ROP Data Center (IRDC) is managed by the PI at the University of Oklahoma Health Sciences Center and technical development and project support are directed by the Inoveon Management Team. The cost-effectiveness evaluation is directed by a PI at Duke University. Other e-ROP committees are listed in Figure 2. In addition, the National Eye Institute convened a Data Monitoring and Oversight Committee that provides valuable guidance and advice throughout the study. A complete listing of the e-ROP Study Group is provided at the end of this manuscript. The study is registered with the clinicaltrials.gov national registry (NCT01264276). The Institutional Review Boards of all participating entities and institutions granted approval for participation in the study.

Figure 2.

Figure 2

Open in a new tab

The components of the e-ROP Study.

Study population

Infants eligible for enrollment in the e-ROP study are those with birth weights (BW) of <1251g who qualify for the diagnostic examinations that are performed as part of participating Clinical Center protocols. Other criteria for e-ROP eligibility included likely survival to 28 days and likely to remain in the NICU for serial ROP examinations. Parents/guardians of eligible infants are approached for consent and infants are enrolled in the study if the parents/guardians consented.

Exclusion criteria for potential e-ROP subjects are: 1) PMA of >39 weeks at first opportunity for imaging unless transferred in for treatment, 2) admission to a Clinical Center NICU with ROP that is already regressing or treated, 3) the presence of a significant media opacity precluding visualization of the retina, or 4) major ocular or systemic congenital abnormality.

Study Procedures

Clinical Center personnel and activities

Figure 2 shows the 12 participating Clinical Centers in the US and Canada. These centers were selected based on their participation in previous multicenter clinical trials of ROP treatments (Cryotherapy for ROP,29 Early treatment for ROP30) and due to large number of infants at risk for ROP in their NICUs. Each Clinical Center research team consists of Study-certified personnel including the Principal Investigator, ophthalmologists who perform the diagnostic examinations, at least one Study Coordinator (SCC) responsible for enrolling infants and insuring data submission, and at least one certified ROP Imager (CRI) who undertakes digital retinal imaging of Study infants. e-ROP investigators at Clinical Centers attended a technical group meeting and completed general and role-specific knowledge assessments prior to study enrollment. All e-ROP study data are recorded either directly at the bedside using a tablet computer or recorded on paper and entered into the web-based data server from the Clinical Center.

Ophthalmological examinations

All study diagnostic examinations are conducted by board certified ophthalmologists experienced in either vitreo-retinal diseases or pediatric ophthalmology, who completed the e-ROP Study certification process. As part of the certification process, the Study Chair visited each participating Clinical Center and performed at least 4-paired examinations with the Clinical Center PI. The Clinical Center PI then performed at least 4-paired examinations with each of other Clinical Center ophthalmologists. In addition, the examining ophthalmologists underwent knowledge assessment examinations.

Enrolled infants undergo a series of ROP diagnostic examination of both eyes. In general, the initial diagnostic eye examination for ROP occurs in the NICU at four weeks after birth or at 32 weeks PMA. For infants with GA of less than 28 weeks, the protocol at the Clinical Center determines the timing of the initial diagnostic examination.

Follow-up diagnostic examinations are done at least as often as every other week unless medically contraindicated. The follow-up schedule or a decision to treat is determined by the examining ophthalmologist based solely on the findings of the examination. The results of the ROP examination are summarized using the 2005 International Classification.31 This classification, any other findings or the presence of any potential ocular or systemic adverse events such as corneal abrasion, apnea or bradycardia are recorded on the e-ROP Diagnostic Exam Form and uploaded to the e-ROP central server. The results of the diagnostic examination are recorded without knowledge of the imaging findings.

e-ROP Study examinations are continued as clinically indicated until the ophthalmologists noted one of the following: 1) mature retinal vessels, 2) immature Zone III on two occasions at least 7 days apart, 3) ROP regressed or regressing on two occasions at least 7 days apart, 4) treatment for severe ROP, or 5) 40 weeks PMA reached with No ROP or only stage 1 or 2 ROP noted.

ROP imaging

All e-ROP imaging sessions are conducted by a CRI, a non-physician who underwent extensive training in the use the RetCam Shuttle®, a wide field imaging camera system using a 130° lens. A range of individuals became CRIs including NICU nurses, neonatal nurse practitioners, ophthalmic photographers, ocular coherence tomography technician, and ophthalmic technologists. Certification consisted of onsite training by an experienced retinal imager from Clarity, submitting a series of 3 right eye and 3 left eye image sets of 6 images each to the e-ROP Reading Center for review, and feedback, and resubmission until satisfactory, a site visit from the Office of Study Chair to evaluate imaging, and completion of a series of knowledge assessments.

The enrolled infants undergo a series of image sessions in both eyes prior to or after the diagnostic examination. The two procedures were almost always consecutive as stipulated in the manual of procedures, and the interval could not exceed 24 hours. Due to poor image quality often encountered in infants <32 weeks PMA, and the very infrequent occurrence of severe ROP in such young infants, imaging is conducted at 32 weeks PMA or later, along with diagnostic examinations routinely indicated for the infant. At each image session, a standard image set is collected for each eye and include an image of the anterior segment and five retinal fields where the image had the disc: (1) central, (2) right, (3) left, (4) up, and (5) down. A RetCam video at 15 frames per second and up to 2 minutes in length is captured and the image set for each eye is selected, after stepping away from the infant’s bed, from the video for uploading after stepping away from the infant’s bed.

Noteworthy ocular or systemic events are recorded into the Retinal Imager Form. The reason for any interruption or termination of the imaging session is recorded. The imagers are unaware of the ocular findings of the diagnostic examination.

Data and image transmission from the Clinical Centers

As shown in Figure 3, all study data including subject demographic information, safety and diagnostic examination results, and image sets from each infant imaged at the Clinical Centers are electronically transmitted directly from the Clinical Center to a secure server at the IRDC in Oklahoma City. For each image set, the CRI selects an optimal image set from the RetCam, sorts and assigns them to the correct eye and specific field imaged. Images are transferred from the camera hard disk to a designated memory stick and in turn transferred via USB port to the dedicated IRDC Server using 256-bit encryption over secure Internet connections. All data are maintained on the IRDC Server with remote back up servers and full disaster recovery plan. Only anonymized study data with no protected health information is included in the data uploaded to the IRDC Server.

Figure 3.

Figure 3

Open in a new tab

The web-based grading form completed by all Trained and Expert Readers in the e-ROP study.

The DCC has secure access to the IRDC Server and downloads data regularly to monitor enrollment, data quality and safety, and select images for grading. Image sets were selected for grading by the DCC and assigned and queued for grading by e-ROP Readers. Image sets are accessed from the IRDC Server for grading by Readers at the Image Reading Center at University of Pennsylvania and for grading by Expert Readers (ophthalmologists) in Calgary, Alberta, Canada, Royal Oak, Michigan, and Atlanta, Georgia.

Selecting images for grading

Even though images of both eyes from all imaging sessions were submitted to the IRDC Server, only images from some infants were selected for grading, as a high percentage (>85%) of infants were not expected to develop RW-ROP. We randomly selected approximately 600 infants who did not develop RW-ROP at any examination to minimize the time in grading many normal images, but retain the ability to estimate precisely specificity. All image sets from the random sample of infants without RW-ROP and all image sets from infants who had developed RW-ROP based on diagnostic examination (anticipated n=250) are selected for grading by the Trained Readers. The purpose of including all RW-ROP-positive infants was to enrich the population of cases for the calculation of sensitivity. A subset (30%, 255 infants) of infants whose images were graded by Trained Readers are also graded by Expert Readers, for the purpose of comparing performance of Trained Readers and Expert Readers. This sample is enriched with half of the image sets from RW-ROP cases.

Certification of Trained Readers

Trained Readers (all non-medical personnel) attended Technical Group Meetings, as well as a training meeting with the Expert Readers, and completed General and Role Specific Knowledge Assessments. Trained Readers completed suggested readings recommended by the Expert Readers. In addition, they had several didactic sessions on identification of ROP using RetCam images, reviewed example images sets, and grading on non-e-ROP image sets. They then practiced grading sets of images of various levels of ROP severity along with an Expert Reader and followed with practice grading independently for 2 weeks, meeting with Expert Readers to review their gradings. Finally, the Trained Readers graded a set of Certification Images that contain varying levels of ROP, varying quality and completeness of the image set to assess agreement with other readers and the Expert Readers.

Image grading by Readers

Inoveon Corporation developed the image grading application according to the specifications defined by the research team. The System Analyst at the DCC assigns image sets and establishes reading queues for the Trained Readers, Reading Supervisor, Clinical Supervisor, and Expert Readers. Readers are masked from all diagnostic exam data and the demographic data. When any Reader logs into the web-based e-ROP viewer application, they are notified of image sets available for reading and are permitted to download them for grading and completion of the grading form. All image sets assigned to the Trained Reader queue are read independently by two of the Trained Readers. Once an image set is downloaded for grading by a Trained Reader, it is unavailable to the second Trained Reader until the first grading is completed. Grading is done in a systematic manner by all Readers using the same web-based e-ROP viewer (see Figure 4) following the evaluation system laid out in the grading protocol.

Figure 4.

Figure 4

Open in a new tab

The workflow in the e-ROP Study from identifying eligible babies to activities conducted at the clinical centers to uploading of Study data to the remote server and subsequent grading of images and analysis of the Study results.

Each Reader grades the quality of each image - good, adequate, or poor - based on the focus, clarity, and field definition. The Reader then determines the following: 1) whether the vessels of the posterior pole are sufficiently abnormal to be designated as pre-plus or plus disease, 2) the extent of retinal vascularization by zone, and 3) whether a demarcation line, ridge, extraretinal neovascularization, flat extraretinal neovascularization, or retinal detachment is present, and location (Zone I, II, III) if present. In addition, the Reader notes other findings and comments on the image set. Once finalized by the Reader, gradings cannot be changed. When an image set has been graded by two Trained Readers, gradings are compared by the image grading system and if agreement is within pre-determined adjudication rules, a final reading record is created for the Trained Reader. If the discrepancies are beyond the threshold of the adjudication rules, a final record is not created and the system places the image set with discrepant answers in a queue for the Reading Supervisor who grades only those fields with discrepancies resulting in a final adjudicated record. Rarely, the Reading Supervisor requests further review by the Clinical Adjudicator who then grades only the fields with discrepancies using the same web-based form. The system then generates a final adjudicated reading record using the final gradings by the Clinical Adjudicator.

Expert Readers

The three Expert Readers are retinal specialists with substantial experience in diagnosis and treatment of ROP, along with participation in ROP clinical trials. For e-ROP certification, Expert Readers were required to complete a role-specific knowledge assessment along with successful grading of pre-certification and certification image sets. The Clinical Supervisor and the Reading Supervisor also completed this certification process.

Using the same web-based e-ROP viewer system, each Expert Reader completes a subset of the image sets assigned to Trained Readers. Each image set assigned is graded by a single Expert Reader except for quality control image sets.

Primary outcome measure

The primary purpose of the e-ROP study is to determine the validity (as determined by the sensitivity and specificity) of detecting RW-ROP from gradings by Trained Readers of image sets obtained by CRIs, compared to the results of the diagnostic examination of that eye by an ophthalmologist experienced in ROP evaluation.

RW-ROP has three major components that are not mutually exclusive: a) any ROP present in zone I, b) any stage 3 or worse ROP, and/or c) plus disease defined as sufficient abnormalities of the posterior pole vessels (dilation and tortuosity) as defined by standard photographs, and noted in at least two quadrants. The results of the Early Treatment for Retinopathy of Prematurity trial30 provide further support for the criteria used for our definition of RW-ROP, which encompasses Type 1 or Type 2 ROP.

Secondary outcome measures

Four other important measures (reliability, feasibility, safety and cost-effectiveness) to evaluate an ROP telemedicine system are being undertaken. Intra-reader and inter-reader agreement is used to assess the reliability of image grading for RW-ROP. Feasibility is assessed by determining whether images sets could be obtained for each infant and whether the images obtained are of sufficient quality for grading. Safety of the system is evaluated by comparing ocular and systemic complications related to imaging versus the diagnostic examination. Finally, the costs and benefits of adopting a telemedicine imaging system for ROP is compared to the current cost of diagnostic examinations.

Sample size

The sample size was determined by the need to estimate sensitivity of the telemedicine system within +/− 0.05 (, i.e., the half width of 95% confidence interval (CI) is ≤ 0.050) in a population in which the overall prevalence of RW-ROP would be approximately 14%. This was assumed to be a sufficient sample to provide a precise estimate for specificity because the majority of infants would not develop RW-ROP.

For primary outcome sample size calculations, the following assumptions were made:

  1. The sensitivity and specificity for detecting eye-specific RW-ROP is between 0.80 and 0.95.

  2. The desired precision of the sensitivity and specificity estimate is within ± 0.050, The RW-ROP rate is 14%, based on the data obtained from the participating Clinical Centers.

  3. Among infants RW-ROP, RW-ROP is seen in both eyes in 80% of infants at first diagnosis, while in only one eye in 20% of infants.32

  4. All infants will undergo clinically indicated diagnostic examinations and thus eye examination results will be available. For image grading to determine the presence of RW-ROP, a random sample (600 infants) of infants without RW-ROP and all infants with RW-ROP will be selected. The unit of analysis is per-eye, and appropriate statistical analytic techniques will be used to account for the inter-eye correlation in RW-ROP status.

  5. In ~10% of enrolled infants, it will not be possible to include their data in sensitivity/specificity calculations due to absence of data from imaging evaluations and/or diagnostic examination, or due to the loss of follow-up before status of RW-ROP could be determined.

Sample size calculations indicated the need for 250 infants (450 eyes) with RW-ROP. Assuming the RW-ROP rate of 14%, we seek a total enrollment of 2000 infants in order to identify 250 infants with RW-ROP. Table 2 shows the precision of the estimate of sensitivity from these 250 infants with RW-ROP and specificity from 600 infants without RW-ROP for sensitivity/specificity values between 0.80 and 0.95. Such a sample size provides a precise estimate of both sensitivity and specificity (half width of 95% CI ranges from 0.026 to 0.048 for sensitivity, and 0.017 – 0.031 for specificity).

Table 2.

The precision of sensitivity and specificity estimates given the proposed sample size*

Total babies
(eyes) with
RW-ROP
observed** 		Sensitivity Half
width
of 95%
CI*** 		Total babies (eyes)
without RW-ROP
selected for image
evaluation		 Specificity Half width
of 95% CI
250 (450
eyes)		 0.95 0.026 600 babies (1200
eyes)		 0.95 0.017
0.925 0.031 0.925 0.020
0.90 0.036 0.90 0.023
0.85 0.043 0.85 0.027
0.80 0.048 0.80 0.031
Open in a new tab
*

Assuming 10% attrition rate due to absence of data on the presence vs. absence of RW-ROP on imaging evaluations and/or diagnostic examination, or due to losses to follow-up

**

Number of eyes with RW-ROP based on 2 eyes per baby in 80% of cases and 1 eye per baby in 20% of cases

***

CI – confidence interval

Statistical Analysis

The primary analysis of validity requires the calculation of sensitivity and specificity for the Trained Reader’s evaluation of each digital image set from an eye compared to the results of the diagnostic examination performed by the Clinical Center ophthalmologist at the same session. The outcome of the imaging evaluation for an eye is:

  1. RW-ROP present (test positive),

  2. RW-ROP absent (test negative), or

  3. Unable to determine if RW-ROP present or absent (indeterminant).

Eyes with “indeterminant” status are scored as test positive, since the primary aim of this study is to determine whether an expert opinion should be sought and, in the case of an indeterminant imaging evaluation, an expert opinion is warranted.

The result of the diagnostic examination constitutes the “gold standard” for sensitivity and specificity calculation. Diagnostic examination outcome is categorized as “RW-ROP present (disease positive)” or “RW-ROP absent (disease negative)” or “indeterminant” (i.e., the Clinical Center Ophthalmologist was unable to determine the presence or absence of RW-ROP). Eyes with “indeterminant” status based on the diagnostic examination were excluded from the analysis of sensitivity and specificity.

All enrolled infants undergo serial sessions of digital imaging and concurrent diagnostic examinations of both eyes. There are several ways to define the sensitivity and specificity for such data from multiple image evaluations and diagnostic examinations, depending on how many pairs and which pair(s) of digital image/diagnostic examinations are used. Due to its simplicity and ease of interpretation, we include only one pair of digital image/diagnostic examinations from each eye in the primary analysis. For sensitivity calculation, the pair when diagnostic examinations first identify RW-ROP is used, while a random pair is chosen for each eye that does not develop RW-ROP for the specificity calculation. Sensitivity is calculated as the proportion of positive image evaluations at the examination at which the RW-ROP is first identified by diagnostic examination. Specificity is calculated as the proportion of RW-ROP-negative diagnostic examinations that are imaging evaluation negative, using the randomly selected pair of digital image/diagnostic examinations for eyes that are RW-ROP negative in all sessions of diagnostic examinations.

The sensitivity calculated using this approach could be interpreted as probability of detecting RW-ROP by image evaluation system when RW-ROP is first noted by diagnostic examinations --the most important parameter in evaluating imaging systems. By using a randomly selected pair for specificity analysis, the impact of PMA at the time of the diagnostic examination on the specificity estimate is minimized.

95% confidence intervals (95% CI) are calculated for sensitivity and specificity. Because images from both eyes of an infant are evaluated, inter-eye correlations exist in these data. In such cases, point estimates of sensitivity and specificity remain valid without adjusting inter-eye correlation, but for confidence interval calculation the correlations from paired eyes need to be adjusted by generalized estimating equations33 where the sandwich robust estimate of variance is applied.34

Results

The study enrolled 1284 infants as of October 31, 2013 and the demographic characteristics of the infants are provided in Table 1. The mean birth weight was 865g and mean gestational age 27 weeks. The majority of the infants were white and non-Hispanic or Latino.

Table 1.

Overall Study Design for e-ROP

Objective: Evaluate the validity, reliability, feasibility, safety, and relative cost-effectiveness of a telemedicine system to detect eyes of at-risk babies in need of a diagnostic evaluation by an ophthalmologist experienced in ROP*.
Study Population: Premature babies (N=2,000) with birth weights of <1251g who meet current ROP screening guidelines in 12 NICUs** in US and Canada.
Procedures: Babies will undergo sequential ROP imaging in both eyes using the RetCam Shuttle, in addition to a standard diagnostic examination performed by a Study-certified ophthalmologist experienced in ROP. Digital images, obtained by a non-physician Certified ROP Imager (CRI), will be downloaded to the server for remote evaluation by Trained Readers and Expert Readers. The results of the Readers’ evaluations will be compared to the presence or absence of RW-ROP*** based on the diagnostic examination conducted at the same session and without knowledge of the digital image findings by the Clinical Center ophthalmologist.
Outcome measures:

  • Validity (sensitivity, specificity): Assessed by comparing the evaluations of digital images by trained readers to the findings of an examination performed by a Study-certified ophthalmologist, rigorously trained in ROP diagnostic examinations;

  • Reliability: Assessed by inter- and intra-reader agreement for deciding whether digital images indicate that an eye has RW-ROP;

  • Feasibility: Determine whether imaging evaluation can be achieved for each baby

  • Safety: Assessed by examining ocular and systemic complications of digital imaging compared with those associated with examinations performed by a Study-certified ophthalmologist using indirect ophthalmoscopy;

  • Relative cost-effectiveness: Assessed by comparing the cost and benefit of a telemedicine system to standard care, adjusted for the varying distribution of patient populations across each NICU and the local availability of eye care.

Open in a new tab
*

ROP – retinopathy of prematurity

**

NICU – neonatal intensive care unit

***

RW-ROP – referral –warranted ROP

The process of certification of the Trained Readers included, after extensive training, the grading of 15 image sets by all Readers (Trained Readers, Expert Readers and two adjudicator ophthalmologists) and comparing the agreement between each Trained Reader with the majority of ophthalmologist readers. There was excellent agreement for each Trained Reader, with percent of agreement ranging from 93% to 100% (93% indicates that one of the 15 images did not agree with the ophthalmologist readers).

Conclusion

The e-ROP study addresses the question of whether a telemedicine system using remote evaluation of digital images by trained readers can reliably detect, in at-risk infants, eyes that have clinical features of potentially serious ROP and are in need of a diagnostic evaluation by an ophthalmologist experienced in ROP. The large-scale, multicenter e-ROP study will determine the validity of comparing grading by trained non-physician readers of images obtained by non-physician imagers with the results of diagnostic examinations conducted at the same session by an e-ROP-certified ophthalmologist. Importantly, assessments of feasibility, safety, and relative cost-effectiveness will also be made.

To achieve the operational goals of the e-ROP study, we developed and used creative training techniques for retinal imagers and readers of digital retinal images. Further, all study data from the Clinical Centers are uploaded using web-based forms to a central server for remote grading by readers and analysis by the DCC. The study is powered to provide precise estimates of sensitivity and specificity, from which caregivers for at-risk infants can make informed decisions about whether such an approach is effective, safe, and feasible. In addition, the design and findings of the e-ROP study can be extended to support other ROP trials or evaluate telemedicine use for other diseases.

Table 3.

Demographic Characteristics of enrolled infants (n=1284)

Characteristics # of infants (%)
Birth Weight (grams)
    Mean (SD) 865 (212)
    Median (Q*1, Q3) 860 (690, 1040)
Gestational Age (weeks)
    Mean (SD) 27 (2.2)
    Median (Q1, Q3) 27 (25, 28)
Gender
    Female 633 49.3%
    Male 651 50.7%
Race of infant
    White only 717 55.8%
    Asian only 19 1.5%
    Black only 376 29.3%
    American Indian only 21 1.6%
    Native Hawaiian or Pacific Islander only 5 0.4%
    Mixed 19 1.5%
    Unable to answer 127 9.9%
Ethnicity of infant
    Hispanic or Latino 124 9.6%
    Not Hispanic or Latino 1109 86.4%
    Unable to answer 51 4.0%
Open in a new tab
*

quartile

Acknowledgments

Funded by National Eye Institute of the National Institutes of Health, Department of Health and Human Services. U10 EY017014

ANCILLARY TABLE OF ABBREVIATIONS USED

BW

Birth Weight

CI

Confidence interval

CRI

Certified Retinal Imager

DCC

Data Coordinating Center

DE

Diagnostic Examination

e-ROP

Telemedicine Approaches to Evaluating Acute-phase ROP

GA

Gestational Age

IRB

Institutional Review Board

IRDC

Inoveon ROP Data Center

NICU

Nursery Intensive Care Unit

PI

Principal Investigator

PMA

Post-menstrual age

ROP

Retinopathy of Prematurity

RW-ROP

Referral-warranted ROP

SCC

Study Center Coordinator

Appendix

e-ROP Study Group:

Office of Study Chair – The Children’s Hospital of Philadelphia

Graham E. Quinn, MD, MSCE – Chair

Kelly Wade, MD, PhD, MSCE – Chair of Safety Committee

Agnieshka Baumritter, MS - Project Director

Trang B. Duros - Research Business Manager

e-ROP Clinical Centers

01 Baltimore, MD - Johns Hopkins University

PI: Michael X. Repka, M.D.

SCC: Jennifer A. Shepard, CRNP

CRI: David Emmert, B.A

C. Mark Herring

02 Boston, MA – Boston Children’s Hospital

PI: Deborah VanderVeen, M.D.

Oph: Suzanne Johnston, MD

Oph: Carolyn Wu, MD

Oph: Jason Mantagos, MD

Oph: Danille Ledoux, MD

SCC: Tamar Winter RN, BSN, IBCLC

Frank Weng

CRI: Theresa (Terri) Mansfield, RN

03 Columbus, OH - Nationwide Children’s Hospital

PI: Don L. Bremer, M.D.

Oph: Mary Lou McGregor, M.D.

Oph: Catherine Olson Jordan, M.D

Oph: David L. Rogers, M.D.

SCC: Rae R. Fellows, M.Ed., CCRC

CRI: Suzanne Brandt, RNC, BSN

Brenda Mann, RNC, BSN

04 Durham, NC - Duke University

PI: David Wallace, M.D.

Oph: Sharon Freedman, MD

SCC: Sarah K Jones

CRI: Du Tran-Viet

Rhonda "Michelle" Young

05 Louisville, KY – University of Louisville

PI, Charles C. Barr, M.D.

Oph: Rahul Bhola, M.D.

Oph: Craig Douglas, MD

Oph: Peggy Fishman, MD

SCC: Michelle Bottorff

CRI: Brandi Hubbuch, RN, MSN, NNP-BC

Rachel Keith, PhD

06 Minneapolis, MN - University of Minnesota

PI: Eric D. Bothun, M.D.

Oph: Inge DeBecker, M.D.

Oph: Jill Anderson, M.D

SCC: Ann Marie Holleschau, BA, CCRP

CRI: Nichole E. Miller, MA, RN, NNP

Darla N. Nyquist, MA, RN, NNP

07 Oklahoma City, OK - University of Oklahoma

PI: R. Michael Siatkowski, M.D.

Oph: Lucas Trigler, M.D.

Oph: Marilyn Escobedo, MD,

SCC: Karen Corff, MS, ARNP, NNP-BC

CRI: Michelle Huynh, MS, ARNP

Kelli Satnes, MS, ARNP, NNP-BC

08 Philadelphia, PA – Children’s Hospital of Philadelphia

PI: Monte D. Mills, M.D.

Oph: William Anninger MD

Oph: Gil Binenbaum, MD MSCE

Oph: Graham Quinn, MD, MSCE

SCC: Karen A. Karp, BSN

CRI: Denise Pearson, COMT

09 San Antonio, TX - University of Texas

PI, Alice Gong, M.D.

Oph: John Stokes, M.D.

Oph: Clio Armitage Harper, MD

SCC: Laurie, Weaver

Carmen McHenry, BSN

CRI: Kathryn Conner

Rosalind Heemer

Elnora Cokley, RNC

10 Salt Lake City, UT - University of Utah

PI, Robert Hoffman, M.D.

Oph: David Dries, M.D.

SCC: Katie Jo Farnsworth

Deborah Harrison, M.S.

CRI: Bonnie Carlstrom

Cyrie Ann Frye, CRA, OCT-C

11 Nashville TN - Vanderbilt University

PI: David Morrison, M.D.

Oph: Sean Donahue, M.D.

Oph: Nancy Benegas, MD

SCC: Sandy Owings, COA, CCRP

CRI: Sandra Phillips, COT, CRI

Scott Ruark

12 Calgary, Alberta, Canada - Hospital of the Foothills Medical Center

PI: Anna Ells, M.D., FRCS

Oph: Patrick Mitchell, M.D.

SCC: April Ingram

CRI: Rosie Sorbie, RN

Data Coordinating Center– University of Pennsylvania School of Medicine

Gui-shuang Ying, Ph.D – PI, Director

Maureen Maguire, Ph.D. – Co-Investigator

Mary Brightwell-Arnold, BA, SCP – Systems Analyst

Max Pistilli, MS - Biostatistician

Kathleen McWilliams, CCRP – Monitor

Claressa Whearry – Data Assistant

Image Reading Center – University of Pennsylvania School of Medicine

Ebenezer Daniel, MBBS, MS, MPH - Director

E. Revell Martin, BA – Trained Reader

Candace R. Parker Ostroff – Trained Reader

Krista Sepielli – Trained Reader

Eli Smith – Trained Reader

Expert Readers

Antonio Capone, MD -The Vision Research Foundation, Royal Oak, Michigan

G. Baker Hubbard, MD - Emory University School of Medicine, Atlanta, Georgia

Anna Ells, MD, FRCS – University of Calgary Medical Center, Calgary, Alberta, Canada

Image Data Management Center – Inoveon Corporation

Peter Lloyd Hildebrand, M.D. – PI, Chairman of the Board

Kerry Davis – Senior Analyst

G. Carl Gibson, COO

Regina Hansen – Senior Analyst

Cost Effectiveness Component

Alex R. Kemper, MD, MPH, MS – PI, Director

Lisa Prosser, PhD - Consultant

Data Management and Oversight Committee (DMOC)

David C. Musch, Ph.D., M.P.H. - Chair

Stephen P. Christiansen, M.D.

Ditte J. Hess, CRA

Steven M. Kymes, Ph.D.

SriniVas R. Sadda, M.D.

Ryan Spaulding, Ph.D.

Editorial Committee

Graham E. Quinn, MD, MSCE – Chair

Ebenezer Daniel, MBBS, MS, MPH

Anna Ells, MD, FRCS

Peter Lloyd Hildebrand, M.D.

Alex R. Kemper, MD, MPH, MS

Kelly Wade, MD, Ph.D

Gui-shuang Ying, Ph.D

Eleanor B. Schron, PhD, RN, FAAN

Agnieshka Baumritter, MS

Executive Committee

Graham E. Quinn, MD, MSCE – Chair

Ebenezer Daniel, MBBS, MS, MPH

Anna Ells, MD, FRCS

Peter Lloyd Hildebrand, M.D.

Alex R. Kemper, MD, MPH, MS

Kelly Wade, MD, Ph.D

Gui-shuang Ying, Ph.D

Michael X. Repka, M.D, 2010-1

Vanessa Bergman, BS, COA, CCRC, 2010-1

Erick D. Bothun, M.D., 2011-2

Tamar Winter, RN, BSN, IBCLC, 2011-2

Alice Gong, M.D., 2012-3

Brandi Hubbuch, RN, MSN, NNP-BC, 2012-3

David Morrison, M.D., 2013-4

Rae Fellows, M.Ed., CCRC, 2013-4

Ex Officio Members:

Eleanor B. Schron, PhD, RN, FAAN – NIH/NEI Project Officer

Agnieshka Baumritter, MS – Office of the Study Chair

National Eye Institute

Eleanor B. Schron, PhD, RN, FAAN – Project Officer

Footnotes

clinicaltrials.gov national registry number: NCT01264276

None of the authors have any proprietary interests or conflicts of interest related to this submission.

This submission has not been published elsewhere and is not under consideration by any other publication.

References

  • 1.Gilbert C, Fielder A, Gordillo L, Quinn G, Semiglia R, Visintin P, Zin A on behalf of the International NO-ROP Group. Characteristics of infants with severe retinopathy of prematurity in countries with low, moderate and high levels of development: Implications for screening programs. Pediatrics. 2005;115:e518–e525. doi: 10.1542/peds.2004-1180. [DOI] [PubMed] [Google Scholar]
  • 2.Gilbert C. Retinopathy of prematurity: A global perspective of the epidemics, population of babies at risk and implications for control. Early Human Development. 2008;84:77–82. doi: 10.1016/j.earlhumdev.2007.11.009. [DOI] [PubMed] [Google Scholar]
  • 3.Kemper AR, Wallace DK. Neonatologists' practices and experiences in arranging retinopathy of prematurity screening services. Pediatrics. 2007;120(3):527–531. doi: 10.1542/peds.2007-0378. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 4.Kemper AR, Freedman SF, Wallace DK. Retinopathy of prematurity care: patterns of care and workforce analysis. J AAPOS. 2008;12(4):344–348. doi: 10.1016/j.jaapos.2008.02.012. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 5.Castillo-Riquelme MC, Lord J, Moseley MJ, Fielder AR, Haines L. Cost-effectiveness of digital photographic screening for retinopathy of prematurity in the United Kingdom. Int J Technol Assess Health Care. 2004;20(2):201–213. doi: 10.1017/s0266462304000984. [DOI] [PubMed] [Google Scholar]
  • 6.Reynolds J. Malpractice and the quality of care in Retinopathy of Prematurity. Trans Am Ophthalmol Soc. 2007;105:461–480. [PMC free article] [PubMed] [Google Scholar]
  • 7.Altersitz KPM. Survey: Physicians being driven away from ROP treatment. Ocular Surgery News. 2006 (August 15, 2006. http://www.osnsupersite.com/print.asp?rID=18018.
  • 8.Reynolds JD, Dobson V, Quinn GE, et al. Evidence-based screening criteria for retinopathy of prematurity: natural history data from the CRYO-ROP and LIGHT-ROP studies. Arch Ophthalmol. 2002;120(11):1470–1476. doi: 10.1001/archopht.120.11.1470. [DOI] [PubMed] [Google Scholar]
  • 9.Ells AL, Holmes JM, Astle WF, et al. Telemedicine approach to screening for severe retinopathy of prematurity: a pilot study. Ophthalmology. 2003;110(11):2113–2117. doi: 10.1016/S0161-6420(03)00831-5. [DOI] [PubMed] [Google Scholar]
  • 9.Skalet AH, Quinn GE, Ying GS, et al. Telemedicine screening for retinopathy of prematurity in developing countries using digital retinal images: a feasibility project. J AAPOS. 2008;12(3):252–258. doi: 10.1016/j.jaapos.2007.11.009. [DOI] [PubMed] [Google Scholar]
  • 10.Wallace DK, Jomier J, Aylward SR, Landers MB., 3rd Computer-automated quantification of plus disease in retinopathy of prematurity. J AAPOS. 2003;7(2):126–130. doi: 10.1016/mpa.2003.S1091853102000150. [DOI] [PubMed] [Google Scholar]
  • 11.Fielder A, Cocker K, Capone A, Trese M. Screening for retinopathy of prematurity using wide-field digital imaging: sensitivity and specificity. Arch Ophthalmol. 2002;120:1234. [PubMed] [Google Scholar]
  • 12.Harrison F. Published Abstract at ARVO. Invest Ophthalmol Vis Sci. 1998;39(supp):591. [Google Scholar]
  • 13.Schwartz SD, Harrison SA, Ferrone PJ, Trese MT. Telemedical evaluation and management of retinopathy of prematurity using a fiberoptic digital fundus camera. Ophthalmology. 2000;107(1):25–8. doi: 10.1016/s0161-6420(99)00003-2. [DOI] [PubMed] [Google Scholar]
  • 14.Yen KG, Hess D, Burke B, Johnson RA, Feuer WJ, Flynn JT. Telephotoscreening to detect retinopathy of prematurity: preliminary study of the optimum time to employ digital fundus camera imaging to detect ROP. J AAPOS. 2002;6(2):64–70. [PubMed] [Google Scholar]
  • 15.Chiang MF, Keenan JD, Starren J, Du YE, Schiff WM, Barile GR, Li J, Johnson RA, Hess DJ, Flynn JT. Accuracy and reliability of remote retinopathy of prematurity diagnosis. Arch Ophthalmol. 2006;124(3):322–7. doi: 10.1001/archopht.124.3.322. [DOI] [PubMed] [Google Scholar]
  • 16.Chiang MF, Wang L, Busuioc M, Du YE, Chan P, Kane SA, Lee TC, Weissgold DJ, Berrocal AM, Coki O, Flynn JT, Starren J. Telemedical retinopathy of prematurity diagnosis: accuracy, reliability, and image quality. Arch Ophthalmol. 2007;125(11):1531–8. doi: 10.1001/archopht.125.11.1531. [DOI] [PubMed] [Google Scholar]
  • 17.Wu C, Petersen RA, VanderVeen DK. RetCam imaging for retinopathy of prematurity screening. J AAPOS. 2006;10(2):107–11. doi: 10.1016/j.jaapos.2005.11.019. [DOI] [PubMed] [Google Scholar]
  • 18.Chiang MF, Jiang L, Gelman R, Du YE, Flynn JT. Interexpert agreement of plus disease diagnosis in retinopathy of prematurity. Arch Ophthalmol. 2007;125(7):875–80. doi: 10.1001/archopht.125.7.875. [DOI] [PubMed] [Google Scholar]
  • 19.The photographic screening for retinopathy of prematurity study (photo-ROP) Primary outcomes. Retina. 2008;28(3 Suppl):S47–S54. doi: 10.1097/IAE.0b013e31815e987f. [DOI] [PubMed] [Google Scholar]
  • 20.Scott KE, Kim DY, Wang L, Kane SA, Coki O, Starren J, Flynn JT, Chiang MF. Telemedical diagnosis of retinopathy of prematurity intraphysician agreement between ophthalmoscopic examination and image-based interpretation. Ophthalmology. 2008;115(7):1222–e83. doi: 10.1016/j.ophtha.2007.09.006. [DOI] [PubMed] [Google Scholar]
  • 21.Dhaliwal C, Wright E, Graham C, McIntosh N, Fleck BW. Wide-field digital retinal imaging versus binocular indirect ophthalmoscopy for retinopathy of prematurity screening: a two-observer prospective, randomized comparison. Br J Ophthalmol. 2009;93(3):355–9. doi: 10.1136/bjo.2008.148908. [DOI] [PubMed] [Google Scholar]
  • 22.Murakami Y, Jain A, Silva RA, Lad EM, Gandhi J, Moshfeghi DM. Stanford University Network for Diagnosis of Retinopathy of Prematurity (SUNDROP): 12-month experience with telemedicine screening. Br J Ophthalmol. 2008;92(11):1456–60. doi: 10.1136/bjo.2008.138867. [DOI] [PubMed] [Google Scholar]
  • 23.Murakami Y, Silva RA, Jain A, Lad EM, Gandhi J, Moshfeghi DM. Stanford University Network for Diagnosis of Retinopathy of Prematurity (SUNDROP): 24-month experience with telemedicine screening. Acta Ophthalmol. 2010;88(3):317–22. doi: 10.1111/j.1755-3768.2009.01715.x. [DOI] [PubMed] [Google Scholar]
  • 24.Silva RA, Murakami Y, Jain A, Gandhi J, Lad EM, Moshfeghi DM. Stanford University Network for Diagnosis of Retinopathy of Prematurity (SUNDROP): 18-month experience with telemedicine screening. Graefes Arch Clin Exp Ophthalmol. 2009;247(1):129–36. doi: 10.1007/s00417-008-0943-z. [DOI] [PubMed] [Google Scholar]
  • 25.Silva RA, Murakami Y, Lad EM, Moshfeghi DM. Stanford University network for diagnosis of retinopathy of prematurity (SUNDROP): 36-month experience with telemedicine screening. Ophthalmic Surg Lasers Imaging. 2011;42:12–9. doi: 10.3928/15428877-20100929-08. [DOI] [PubMed] [Google Scholar]
  • 26.Wallace DK, Freedman SF, Hartnett ME, Quinn GE. Predictive value of pre-plus disease in retinopathy of prematurity. Arch Ophthalmol. 2011 May;129(5):591–6. doi: 10.1001/archophthalmol.2011.63. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 27.Dai S, Chow K, Vincent A. Efficacy of wide-field digital retinal imaging for retinopathy of prematurity screening. Clinical & Experimental Ophthalmology. 2011;39(1):23–9. doi: 10.1111/j.1442-9071.2010.02399.x. [DOI] [PubMed] [Google Scholar]
  • 28.Chiang MF, Melia M, Buffenn AN, Lambert SR, Recchia FM, Simpson JL, Yang MB. Detection of clinically significant retinopathy of prematurity using wide-angle digital retinal photography. Ophthalmology. 2012;119:1272–1280. doi: 10.1016/j.ophtha.2012.01.002. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 29.Multicenter trial of cryotherapy for retinopathy of prematurity. Three-month outcome. Cryotherapy for Retinopathy of Prematurity Cooperative Group. Arch Ophthalmol. 1990;108(2):195–204. doi: 10.1001/archopht.1990.01070040047029. [DOI] [PubMed] [Google Scholar]
  • 30.Revised indications for the treatment of retinopathy of prematurity: results of the early treatment for retinopathy of prematurity randomized trial. Early Treatment For Retinopathy Of Prematurity Cooperative Group. Arch Ophthalmol. 2003;121(12):1684–94. doi: 10.1001/archopht.121.12.1684. [DOI] [PubMed] [Google Scholar]
  • 31.An International Committee for the Classification of Retinopathy of Prematurity (Co-Chair) The International Classification of Retinopathy of Prematurity- Revisited. Arch Ophthalmol. 2005;123:991–9. doi: 10.1001/archopht.123.7.991. [DOI] [PubMed] [Google Scholar]
  • 32.Quinn GE, Dobson V, Biglan A, Evans J, Plotsky D, Hardy RJ. Correlation of retinopathy of prematurity in fellow eyes in the CRYO-ROP Study. Arch Ophthalmol. 1995;113:469–73. doi: 10.1001/archopht.1995.01100040089032. [DOI] [PubMed] [Google Scholar]
  • 33.Liang K, Zeger S. Longitudinal data analysis using generalized linear models. Biometrika. 1986;73:13–22. [Google Scholar]
  • 34.Smith PJ, Hadgu A. Sensitivity and specificity for correlated observations. Statistics in Medicine. 1992;11:1503–1509. doi: 10.1002/sim.4780111108. [DOI] [PubMed] [Google Scholar]

RESOURCES