Abstract
Purpose:
To evaluate a tele-education system developed to improve diagnostic competency in retinopathy of prematurity (ROP) by ophthalmologists-in-training in Mexico.
Design:
Prospective, randomized cohort study.
Participants:
Fifty-eight ophthalmology residents and fellows from a training program in Mexico consented to participate. Twenty-nine of 58 trainees (50%) were randomized to the educational intervention (pretest, ROP tutorial, ROP educational chapters, and posttest), and 29 of 58 trainees (50%) were randomized to a control group (pretest and posttest only).
Methods:
A secure web-based educational system was created using clinical cases (20 pretest, 20 posttest, and 25 training chapter-based) developed from a repository of over 2500 unique image sets of ROP. For each image set used, a reference standard ROP diagnosis was established by combining the clinical diagnosis by indirect ophthalmoscope examination and image-based diagnosis by multiple experts. Trainees were presented with image-based clinical cases of ROP during a pretest, posttest, and training chapters.
Main Outcome Measures:
The accuracy of ROP diagnosis (e.g., plus disease, zone, stage, category) was determined using sensitivity and specificity calculations from the pretest and posttest results of the educational intervention group versus control group. The unweighted kappa statistic was used to analyze the intragrader agreement for ROP diagnosis by the ophthalmologists-in-training during the pretest and posttest for both groups.
Results:
Trainees completing the tele-education system had statistically significant improvements (P < 0.01) in the accuracy of ROP diagnosis for plus disease, zone, stage, category, and aggressive posterior ROP (AP-ROP). Compared with the control group, trainees who completed the ROP tele-education system performed better on the posttest for accurately diagnosing plus disease (67% vs. 48%; P = 0.04) and the presence of ROP (96% vs. 91%; P < 0.01). The specificity for diagnosing AP-ROP (94% vs. 78%; P < 0.01), type 2 ROP or worse (92% vs. 84%; P = 0.04), and ROP requiring treatment (89% vs. 79%; P < 0.01) was better for the trainees completing the tele-education system compared with the control group. Intragrader agreement improved for identification of plus disease, zone, stage, and category of ROP after completion of the educational intervention.
Conclusions:
A tele-education system for ROP education was effective in improving the diagnostic accuracy of ROP by ophthalmologists-in-training in Mexico. This system has the potential to increase competency in ROP diagnosis and management for ophthalmologists-in-training from middle-income nations.
Retinopathy of prematurity (ROP) is a vasoproliferative disease of the developing retina that is largely treatable with an appropriate and timely diagnosis.1–4 Although major advances in the management of ROP have occurred as a result of the classification criteria outlined by the Cryotherapy for ROP5 and Early Treatment for ROP6 studies, ROP remains a leading cause of childhood blindness throughout the world.
An increase in the incidence of ROP, termed the “third epidemic,” has uniquely occurred in middle-income countries.1,2,7 Middle-income countries like Mexico are in the unique conundrum of having sufficiently advanced medical facilities to support premature babies; however, they may lack the necessary resources to manage ROP appropriately.2 Specifically, previous reports have documented an inadequate number of ophthalmologists experienced in ROP diagnosis and management.1,2,7 The root cause of these shortages are partly based on workforce limitations and variable education on ROP diagnosis and management. Indeed, the previous literature has noted that there is a lack of standardization for ROP education within both high- and middle-income countries that has resulted in significant differences in the accuracy of ROP diagnosis among ophthalmologists-in-training.8–11
Web-based learning offers the unique opportunity to provide high-quality education to medical trainees in developing countries, particularly those with a critical shortage of medical faculty.12 The Global Education Network for ROP is a multi-institutional collaboration interested in developing innovative ways to educate and increase the workforce for ROP. In conjunction with the Imaging and Informatics in ROP Consortium, the Global Education Network for ROP has previously demonstrated the efficacy of web-based education for ROP among trainees in the United States and Canada,13 but there is limited work on web-based learning for ROP in developing and middle-income countries.
The purpose of this study is to evaluate whether a tele-education system can improve the diagnostic competency of ophthalmologists-in-training in a middle-income country.
Methods
The Weill Cornell Medical College Human Studies Committee approved this as a prospective study for the analysis of retinal images and approved the educational material used in this study. Administration of the tele-education system was also reviewed by the Weill Cornell Medical College Human Studies Committee and was granted an exemption because it was considered research in an established or commonly accepted educational setting involving normal educational practices such as research on the effectiveness of instructional techniques, curricula, and instructional strategies.
Image Acquisition
Images in the tele-education system were obtained utilizing a repository of over 2500 unique sets of ROP images. A total of 36 infants were used for the 65 clinical cases (20 pretest, 20 posttest, and 25 training chapter-based) in the system. Both eyes of each infant underwent funduscopic imaging, for a total of 72 eyes. All cases selected for the pretest, posttest, and training chapters were reviewed by the study authors (R.V.P.C., M.F.C.) to ensure that a spectrum of disease was represented. The relevant clinical characteristics of the cases are summarized in Table S1 (available at www.aaojournal.org).
Study Subjects
Ophthalmologists-in-training at the resident or fellow level were recruited by the co-authors (K.E.J., R.V.P.C., M.M.C.), from a single ophthalmology training program in Mexico. Trainees who participated in the tele-education system were provided access to a website where they could access the system. At initial recruitment, trainees were randomized to either the group who were to complete the tele-education system (referred to as the intervention group) or a control group who only took a pretest and posttest (Fig S1, available at www.aaojournal.org). The intervention group completed the pretest, training chapters, and posttest in sequential order. The control group completed the pretest and posttest and was not given access to any of the educational material available to the intervention group until completion of the study. The trainees in the intervention group were scheduled on a weekly schedule such that they completed 1 to 2 sections of the tele-education system per week, for a total of 8 sections.
Study Design
The specific study design of the pretest, ROP educational materials, and posttest used in the tele-education system has previously been described.13 Briefly, the tele-education system was created based on clinical cases applied in 3 different scenarios: pretest, chapters, and posttest. A clinical case was defined as providing a clinical diagnosis in both eyes of 1 patient. For each clinical case, demographic information including gestational age, birth weight, and postmenstrual age at time of imaging was provided to the trainee. A set of 5 retinal images (superior, inferior, posterior, nasal, temporal) was included for each eye and additional images were included if deemed to be clinically significant. Retinal images for both eyes were provided simultaneously; however, a clinical diagnosis of plus disease (no, pre-plus, plus), zone (I, II, III), ROP (yes, no), stage (1–5), category (none, mild, type 2 ROP, ROP requiring treatment), and aggressive posterior ROP (AP-ROP) (yes, no) was required for each individual eye. Participants were asked to provide a clinical follow-up time based on the diagnosis of both eyes for the patient.
After completion of the tele-education system, trainees were directed to complete a web-based survey regarding the effectiveness of the tele-education system. Items in existing psychometric instruments were adapted to measure trainees’ attitudes,14 and there were a total of 6 survey items, consisting of 2 items that assessed the trainees’ perception of their understanding of the diagnosis of ROP, 3 items that assessed the trainees’ attitudes toward preferred learning environment, and 1 item that assessed the trainees’ opinion of ease of use of the ROP tele-education system. Survey responses were recorded using a 5-point Likert-type scale (1 = strongly disagree, 2 = disagree, 3 = neutral, 4 = agree, 5 = strongly agree), and the average score for each question was calculated.
Data Analysis
All data were analyzed using statistical software (Stata/SE 12.0; StataCorp LP, College Station, TX). Statistical significance was considered to be represented by a 2-sided P value < 0.05. The diagnostic accuracy for ROP by the trainees was evaluated using sensitivity and specificity compared with the consensus reference standard diagnosis.15 Briefly, the reference standard diagnosis combined the clinical diagnosis as determined from indirect ophthalmoscopy by an ROP examiner with the image-based diagnosis from multiple experienced readers. For sensitivity and specificity calculations, the cutoff values that were investigated included stage 1 disease or worse, stage 2 disease or worse, stage 3 disease or worse, zone I or II disease, zone I disease, pre-plus or worse, plus disease, mild or worse disease, type 2 ROP or worse disease, disease requiring treatment, and the presence of AP-ROP.
Pretest and posttest sensitivities and specificities were compared within and between the intervention group and control group using the paired t test. Based on the 4 cases that were repeated in both the pretest and posttest, intragrader reliability was evaluated using the kappa statistic for chance-adjusted agreement in diagnosis. Specifically, pretest and posttest unweighted kappas for both the intervention and control groups were calculated for the diagnosis of plus disease, zone, stage, category, AP-ROP, and the presence of ROP of any severity. A well-known scale was used for interpretation of results: 0 to 0.20, slight agreement; 0.21 to 0.40, fair agreement; 0.41 to 0.60, moderate agreement; 0.61 to 0.80, substantial agreement; and 0.81 to 1.00, almost perfect agreement.8
Results
Table S2 (available at www.aaojournal.org) summarizes key characteristics of the 58 ophthalmologists-in-training who participated in the study. Twenty-nine of 58 trainees (50%) were randomized to the intervention group, and of the 26 trainees who completed the ROP tele-education system, 6 (23%) were postgraduate year (PGY)-1 residents, 8 (31%) were PGY-2 residents, 7 (27%) were PGY-3 residents, and 5 (19%) were retina fellows. Twenty-nine of 58 trainees (50%) were randomized to the control group, of whom 26 of 29 (90%) completed the pretest and posttest. Three of 29 participants (10%) in the control group did not complete the program (all 3 did not complete the posttest). Three of 29 participants (10%) in the intervention group did not complete the program (1 withdrew, 1 completed only the tutorial, and 1 did not complete the posttest).
Diagnostic Accuracy of the Control Group
When comparing the pretest and posttest results among the control group, there was no statistically significant difference in sensitivity or specificity for the diagnosis of stage, zone, plus, category, presence of AP-ROP, or presence of ROP. Specifically, there was no statistically significant difference between sensitivity and specificity of diagnosis for type 2 ROP or ROP requiring treatment in this control group. Among the control group, the mean (range) posttest sensitivity and specificity for detecting type 2 ROP or worse were 78% (10%–100%) and 84% (20%–100%), respectively. Furthermore, the mean (range) posttest sensitivity and specificity for detecting ROP requiring treatment were 64% (0%–100%) and 79% (34%–100%), respectively.
Diagnostic Accuracy of Trainees Participating in the Tele-education System (Intervention Group)
Comparing the pretest and posttest results of the intervention group revealed a statistically significant improvement in the sensitivity of ROP diagnosis for plus, zone, stage, category, and AP-ROP. In this group, there were significant improvements between the pretest and posttest examinations in the sensitivity for the detection of stage 3 or worse (49% vs. 62%; P < 0.01), zone I (32% vs. 54%; P < 0.01), plus disease (43% vs. 67%; P < 0.01), type 2 or worse ROP (67% vs. 81%; P < 0.01), and ROP requiring treatment (54% vs. 71%; P < 0.01). Similarly, there was a statistically significant improvement in specificity for the detection of stage 1 or worse disease (P = 0.01), stage 2 or worse disease (P = 0.02), stage 3 or worse disease (P = 0.04), mild or worse disease (P = 0.03), presence of AP-ROP (P = 0.04), and presence of ROP (P = 0.01). The mean (range) posttest sensitivity and specificity for detecting type 2 or worse ROP were 81% (30%–100%) and 92% (80%–100%), respectively. The mean (range) posttest sensitivity and specificity for detecting ROP requiring treatment were 71% (0%–100%) and 89% (63%–100%), respectively.
Comparison of Diagnostic Accuracy between the Control and Intervention Groups
Table 3 compares the sensitivity and Table 4 compares the specificity of ROP diagnosis by assessing the pretest and posttest results of trainees in the intervention group against trainees in the control group.
Table 3.
Sensitivity of Retinopathy of Prematurity Diagnosis by Trainees Participating in the Retinopathy of Prematurity Tele-education System Based on a Consensus Reference Diagnosis
| Sensitivity During Pretest, % (SE) | Sensitivity During Posttest, % (SE) | |||||
|---|---|---|---|---|---|---|
| Diagnosis | Intervention Group | Control Group | P Value* | Intervention Group | Control Group | P Value* |
| Stage | ||||||
| 1 or worse | 90 (1) | 91 (1) | 0.80 | 96 (1) | 91 (1) | <0.01 |
| 2 or worse | 83 (2) | 81 (3) | 0.75 | 93 (2) | 89 (3) | 0.20 |
| 3 or worse | 49 (4) | 55 (5) | 0.27 | 62 (4) | 63 (5) | 0.90 |
| Zone | ||||||
| Zone II or I | 59 (3) | 52 (3) | 0.13 | 85 (3) | 62 (3) | <0.01 |
| Zone I | 32 (6) | 22 (6) | 0.26 | 55 (6) | 39 (8) | 0.10 |
| Plus | ||||||
| Pre-plus or worse | 67 (5) | 68 (5) | 0.83 | 82 (3) | 78 (4) | 0.40 |
| Plus | 43 (6) | 43 (6) | 0.97 | 67 (6) | 48 (7) | 0.04 |
| Category | ||||||
| Mild or worse | 83 (3) | 82 (4) | 0.91 | 95 (1) | 84 (3) | <0.01 |
| Type 2 or worse | 67 (5) | 70 (5) | 0.69 | 81 (3) | 78 (4) | 0.63 |
| Requiring treatment | 54 (6) | 54 (7) | 0.97 | 71 (6) | 64 (7) | 0.43 |
| Presence of AP-ROP | 48 (10) | 38 (9) | 0.48 | 72 (8) | 60 (9) | 0.30 |
| Presence of ROP | 90 (1) | 91 (1) | 0.80 | 96 (1) | 91 (1) | <0.01 |
AP-ROP = aggressive posterior retinopathy of prematurity; ROP = retinopathy of prematurity; SE = standard error.
P value using paired t test. Boldface indicates statistical significance of P < 0.05.
Table 4.
Specificity of Retinopathy of Prematurity Diagnosis by Trainees Participating in the Retinopathy of Prematurity Tele-education System Based on a Consensus Reference Diagnosis
| Specificity During Pretest, % (SE) | Specificity During Posttest, % (SE) | |||||
|---|---|---|---|---|---|---|
| Diagnosis | Intervention Group | Control Group | P Value* | Intervention Group | Control Group | P Value* |
| Stage | ||||||
| 1 or worse | 79 (6) | 76 (6) | 0.71 | 94 (2) | 67 (7) | <0.01 |
| 2 or worse | 83 (4) | 83 (8) | 0.98 | 95 (1) | 87 (4) | 0.09 |
| 3 or worse | 89 (3) | 90 (3) | 0.84 | 96 (1) | 90 (3) | 0.08 |
| Zone | ||||||
| Zone II or I | –† | –† | –† | –† | ||
| Zone I | 86 (3) | 86 (3) | 0.86 | 89 (3) | 85 (3) | 0.30 |
| Plus | ||||||
| Pre-plus or worse | 87 (4) | 86 (5) | 0.92 | 96 (1) | 87 (4) | 0.05 |
| Plus | 90 (2) | 88 (3) | 0.58 | 92 (2) | 88 (2) | 0.21 |
| Category | ||||||
| Mild or worse | 81 (5) | 79 (5) | 0.75 | 94 (2) | 79 (5) | 0.01 |
| Type 2 or worse | 88 (3) | 85 (3) | 0.39 | 92 (1) | 84 (4) | 0.04 |
| Requiring treatment | 85 (3) | 80 (3) | 0.20 | 89 (2) | 79 (3) | <0.01 |
| Presence of AP-ROP | 82 (5) | 76 (6) | 0.47 | 94 (2) | 78 (5) | <0.01 |
| Presence of ROP | 79 (6) | 76 (6) | 0.71 | 94 (1) | 67 (7) | <0.01 |
AP-ROP = aggressive posterior retinopathy of prematurity; ROP = retinopathy of prematurity; SE = standard error.
P value using paired t test. Boldface indicates statistical significance of P < 0.05.
Specificity is undefined as all eyes contained disease in zone I and/or zone II.
Comparing the pretest results between the control group and intervention group revealed no statistically significant difference in sensitivity or specificity for the diagnosis of stage, zone, plus, category, presence of AP-ROP, or presence of ROP (Tables 3 and 4). In particular, when the intervention group was compared with the control group during the pretest, there was no significant difference in the sensitivity for the diagnosis of stage 3 or worse (49% vs. 55%; P = 0.27), zone I (32% vs. 22%; P = 0.26), plus disease (43% vs. 43%; P = 0.97), or ROP requiring treatment (54% vs. 54%; P = 0.97).
Comparing the posttest results revealed that trainees in the intervention group, compared with the control group, had statistically significant improvements in the sensitivity for detection of stage 1 or worse (P < 0.01), zone II or worse (P < 0.01), plus disease (P = 0.04), mild ROP or worse (P < 0.01), and presence of ROP (P < 0.01) (Table 3). Similarly, compared with the control group, trainees in the intervention group on the posttest had statistically significant improvements in the specificity for detection of stage 1 or worse (P < 0.01), mild ROP or worse (P < 0.01), type 2 ROP or worse (P = 0.04), ROP requiring treatment (P < 0.01), presence of AP-ROP (P < 0.01), and presence of ROP (P < 0.01) (Table 4).
Intragrader Agreement in Diagnosis of Retinopathy of Prematurity
Table 5 summarizes the intragrader agreement using the Cohen’s kappa statistic. Among the intervention group, there was an improvement from moderate to substantial agreement for the diagnosis of stage, plus, and presence of ROP. There was also an improvement from fair to substantial agreement for the diagnosis of zone. Among the control group, there was an improvement in agreement for the diagnosis of stage (moderate to substantial), zone (fair to moderate), and category (slight to moderate).
Table 5.
Intragrader Agreement of Retinopathy of Prematurity Diagnosis among Trainees Participating in the Retinopathy of Prematurity Tele-education System
| Cohen’s Unweighted Kappa | ||||
|---|---|---|---|---|
| Trainees in Intervention Group | Trainees in Control Group | |||
| Diagnosis | Pretest | Posttest | Pretest | Posttest |
| Stage | 0.62 | 0.92 | 0.74 | 0.84 |
| Zone | 0.32 | 0.75 | 0.39 | 0.65 |
| Plus | 0.44 | 0.75 | 0.60 | 0.68 |
| Category | 0.41 | 0.56 | 0.20 | 0.54 |
| Presence of AP-ROP | 0.43 | 0.52 | 0.65 | 0.66 |
| Presence of ROP | 0.64 | 0.86 | 0.68 | 0.75 |
AP-ROP = aggressive posterior retinopathy of prematurity; ROP = retinopathy of prematurity.
Post-Retinopathy of Prematurity Tele-education System Survey of Trainees
Twenty-four of 58 trainees (41%) who completed the ROP tele-education system also completed the posttraining survey (Fig 2). Using a 5-point Likert-type scale, the average score for question 1 was 2.63, as 12 of 24 respondents (50%) disagreed or strongly disagreed that they had an adequate understanding of the diagnosis of ROP before using the ROP tele-education system. The respondents’ score for question 4 showed a statistically significant self-perceived improved understanding of ROP, with an average score of 4.08, after completing the training (P < 0.01, signed rank test). Twenty-two of 24 respondents (92%) agreed or strongly agreed that they had an adequate understanding of the diagnosis of ROP after completing the ROP tele-education system.
Figure 2.
Survey results of trainees who completed the retinopathy of prematurity (ROP) tele-education system (N = 24). Survey questions after completion of the ROP tele-education system were rated using a Likert scale of “strongly agree” to “strongly disagree.” The questions included in the survey were (1) “I had an adequate understanding of the diagnosis of ROP before taking the pretest in the ROP Student Module”; (2) “The ROP Student Module was easy to use”; (3) “I learned from the feedback provided at the end of each case”; (4) “I had an adequate understanding of the diagnosis of ROP after completing the ROP Student Module”; (5) “I learn more effectively in a web-based environment compared to a traditional textbook format”; and (6) “I learned more effectively from ROP cases with automatic feedback compared to a traditional textbook format.”
Discussion
The key findings of this study are as follows: (1) a tele-education system for ROP can improve the diagnostic accuracy of ROP by ophthalmologists-in-training from a middle-income nation; (2) intragrader agreement, as determined by the kappa statistic, improved for identification of plus disease, zone, stage, and category of ROP after completion of the tele-education system; and (3) the trainees completing the tele-education system felt that their understanding of the diagnosis of ROP improved after participating in the training.
The first key finding is that a tele-education system for ROP can successfully improve ROP diagnosis by ophthalmologists-in-training from a middle-income nation. Trainees who completed the ROP tele-education system had statistically significant improvements in sensitivity for the diagnosis of plus disease, zone, stage, category, and AP-ROP. In our study, there was improvement in sensitivity for all categories of ROP diagnosis when analyzing the pretest and posttest results of trainees only in the educational intervention group. The results for the intervention group show that the mean (range) posttest sensitivity for detecting ROP requiring treatment was 71% (0%–100%). It should be noted, however, that only 1 of 26 participants had a posttest sensitivity of 0%.
To further determine the significance of the improvement in diagnostic accuracy for ROP in the group who participated in the tele-education system, we compared the results of the educational intervention group with the results from a group of trainees who were evaluated using the pretest and posttest but did not undergo the ROP tele-education system (control group). The posttest results of the 2 groups indicate that completion of the tele-education system resulted in a statistically significant improvement in the accuracy of ROP diagnosis. Notably, the educational intervention group, compared with the control group, had improvements in parameters of clinically significant disease, including the sensitivity of plus disease (67% vs. 48%; P = 0.04), specificity of AP-ROP (94% vs. 78%; P < 0.01), and specificity of ROP requiring treatment (89% vs. 79%; P < 0.01). The improvements in plus disease are critical, given that plus disease is a hallmark for the diagnosis of ROP requiring treatment and failure to recognize plus disease appropriately can lead to irreversible blindness secondary to progression of ROP.16–18 Currently, plus disease diagnosis has been subjective and varies among ROP experts, even when interpreting the same color fundus photographs.16,17,19 New paradigms in computer-facilitated image analysis along with tele-education initiatives may help with the diagnostic challenges associated with plus disease diagnosis.20–22
Furthermore, in a comparison of the posttest results between the intervention and control groups, there was no statistically significant difference between the groups for sensitivity of zone I disease (55% for the intervention group vs. 39% for the control group; P = 0.10), sensitivity of stage 3 or worse disease (62% for the intervention group vs. 63% for the control group; P = 0.90), or sensitivity of ROP requiring treatment (71% for the intervention group vs. 64% for the control group; P = 0.43). These nonsignificant differences when comparing the 2 groups are partly due to nonsignificant improvements within the control group from the pretest to posttest for the sensitivity of stage 3 or worse disease (55% for the pretest vs. 63% for the posttest), zone I disease (32% for the pretest vs. 39% for the posttest), and disease requiring treatment (54% for the pretest vs. 64% for the posttest). It is possible that these baseline improvements could be owing to the completion of additional clinical cases by the trainees in the control group or may be attributable to self-directed learning that was not part of the tele-education system. Indeed, previous studies by our group have shown that diagnostic accuracy may improve as trainees complete more cases, even in the absence of an explicit pedagogical model.8,9
In comparing the performance of the ophthalmologists-in-training from this study with that of ophthalmologists-in-training from high-income countries who completed the ROP tele-education system and were described in a previously published study by our group,13 we found that, at a minimum, there was no significant difference between the groups in the sensitivity or specificity of diagnosing clinically significant disease (type 2 or worse ROP or ROP requiring treatment) or the presence of ROP (Table 6). Notably, trainees from high-income countries, compared with trainees from middle-income countries, had improved sensitivity for the diagnosis of stage 3 or worse disease (P < 0.01) and zone I disease (P = 0.02). However, trainees from middle-income countries, compared with trainees from high-income countries, had improved specificity for the diagnosis of stage 3 or worse disease (P < 0.01).
Table 6.
Post Hoc Comparison of Accuracy of Retinopathy of Prematurity Diagnosis in the Posttest Examination Between Trainees from High-income Countries* and Trainees from Mexico
| Posttest Sensitivity, % (SE) | Posttest Specificity, % (SE) | |||||
|---|---|---|---|---|---|---|
| Diagnosis | Trainees from Mexico (N = 26) | Trainees from High-Income Countries1 (N = 31) | P Value† | Trainees from Mexico (N = 26) | Trainees from High-Income Countries1 (N = 31) | P Value† |
| Stage 3 or worse | 62 (4) | 78 (3) | <0.01 | 96 (1) | 92 (1) | <0.01 |
| Zone I | 55 (6) | 74 (5) | 0.02 | 89 (3) | 90 (2) | 0.77 |
| Plus | 67 (6) | 67 (5) | 1.00 | 92 (2) | 92 (1) | 1.00 |
| Type 2 or worse | 81 (3) | 84 (3) | 0.48 | 92 (1) | 89 (1) | 0.09 |
| Requiring treatment | 71 (6) | 78 (4) | 0.32 | 89 (2) | 90 (1) | 0.64 |
| Presence of AP-ROP | 72 (8) | 77 (6) | 0.61 | 94 (2) | 94 (1) | 1.00 |
| Presence of ROP | 96 (1) | 96 (1) | 1.00 | 94 (1) | 87 (3) | 0.07 |
AP-ROP = aggressive posterior retinopathy of prematurity; ROP = retinopathy of prematurity; SE = standard error.
Data adapted from Chan et al.13
P value using unpaired t test. Boldface indicates statistical significance of P < 0.05.
The second key finding is that intragrader agreement in the intervention group, as determined by the kappa statistic, improved from fair or moderate to substantial agreement for identification of plus disease, zone, stage, and presence of ROP (Table 5). There were also improvements in intragrader agreement among the control group for stage, zone, and category. This may suggest improvements in intragrader agreement by chance alone. However, it is possible that improvements may be secondary to self-directed learning between taking the pretest and posttest. Previous studies by our group have shown that intragrader reliability can vary for interpreting different aspects of ROP,17,23 and these current findings reinforce that there may be important subtleties in ROP diagnosis that could be overlooked by both trainees and experts in ROP.
The improvements in intragrader reliability and diagnostic accuracy as a result of using the ROP tele-education system have implications for improving international ROP education, clinical care, and telemedicine and increasing the workforce for ROP. The improvements in intragrader reliability seen in this study are significant for ROP classification because even though ROP classification has been derived through evidence-based guidelines and standardized according to the criteria outlined by the International Classification of ROP,18 the diagnosis of certain aspects of ROP can still vary significantly, even among ROP experts.17,19,21,24 As the paradigm for ROP management may more commonly include telemedicine for ROP screening, the reliability of the image graders, in addition to the accuracy of diagnosis, will be essential for successful telemedicine systems. Therefore, methods to improve diagnostic accuracy and intragrader reliability for ROP may be useful in training or certifying image graders for tele-medicine programs. In addition, an educational system that can improve image grading for ROP can potentially make an impact from a socioeconomic perspective, as ROP tele-medicine programs have repeatedly been shown to be cost-effective.25–28
From a treatment perspective, current guidelines recommend close follow-up of type 2 ROP within 1 week and treatment of type 1 ROP within 72 hours of diagnosis.3 However, in the international context, screening protocols may often be inconsistently implemented and some protocols may even be inappropriately designed, leading to children at risk for severe ROP not being evaluated.1 For example, a 2002 multicenter study of 11 neonatal intensive care units in Latin America noted a 42% rate of ROP in very low-birth weight infants, with only 68% of the babies receiving adequate screening.29 These limitations exacerbate the already time-sensitive nature of ROP evaluation and, if an examiner is inaccurate or inconsistent in diagnosis, ROP may progress without appropriate follow-up, leading to blinding disease.
Therefore, tele-education systems can be a part of the method to improve the quality of the workforce and address the shortage of skilled ROP examiners in middle-income countries where the current medical education system may not be able to train enough qualified providers to manage the emerging ROP epidemic.30 A common model for ROP training involves ROP experts taking part in live educational workshops to teach international clinicians on ROP diagnosis and management.31,32 Although these models may provide immediate assistance to areas in need, such interventions can be difficult to sustain given the administrative and travel costs for these workshops. Until recently, e-learning initiatives in middle-income countries were hampered by technological limitations; however, the telecommunications boom over the past decade has facilitated web-based learning models.33 Tele-education can provide high-quality education to medical trainees in developing countries and may even establish standardization of ROP education among developed and developing countries. Web-based educational initiatives are already being studied in other medical fields, including pediatrics,34 dermatology,35 and emergency medicine.36,37 In addition, the educational framework developed from these systems has the potential to be modified to create web-based training initiatives for other ophthalmic conditions (e.g., diabetic retinopathy).
The third key finding is that the trainees completing the tele-education system felt that their understanding of the diagnosis of ROP improved after they participated in the training (Fig 2). These findings are similar to the results of U.S. and Canadian trainees who had completed the tele-education system,13 and they reinforce previous studies of how trainees felt regarding their competency in ROP care.10,11 These results also indicate that trainees both subjectively felt that they improved their understanding of ROP and objectively demonstrated improvement in diagnostic accuracy. Furthermore, trainees indicated that they preferred to learn through automated feedback and case-based learning techniques, possibly because they retained information more effectively in a web-based environment compared with a traditional textbook format. These findings coincide with previous research that has examined web-based models for medical education.38,39
Several limitations of this study should be noted. (1) The tele-education system assessed the image-based diagnosis of ROP; however, this may not specifically translate into competency at the bedside using indirect ophthalmoscopy. Diagnosis by indirect ophthalmoscopy has traditionally been the standard method for ROP diagnosis, but studies have shown advantages of both image-based diagnosis and diagnosis by indirect ophthalmoscopy, and we believe that both modalities provide unique value for teaching techniques to diagnose ROP.40 (2) Although the ophthalmology residents (PGY-1 to PGY-3) had no exposure to clinical care for ROP, retina fellows in both the control and intervention groups had prior exposure to clinical care for ROP. Furthermore, trainees participating in the ROP tele-education system could have had access to other ROP educational materials, which could alter the diagnostic accuracy between the pretest and posttest. In particular, our results note nonsignificant improvements in the sensitivity of stage 3 or worse ROP, zone I disease, and ROP requiring treatment within the control group when comparing the pretest and posttest results. (3) The study population was based on 1 ophthalmology program from a middle-income nation, and such findings may not be generalizable to other programs. The study results, however, were consistent with improvements in ROP diagnostic accuracy for U.S. and Canadian ophthalmologists-in-training who participated in the tele-education system. (4) The participating program in this study was associated with an academic medical center with retina fellows who also screen for ROP, which may not be applicable for other international programs. However, studies utilizing this system in the Philippines and Brazil also demonstrated improvements in ROP diagnostic accuracy, and further testing would be warranted to understand the utility of the ROP tele-education system in other developing and middle-income countries.41 (5) Image-based ROP screening programs have historically focused on the prevention of retinal detachment by identifying disease that may soon progress to ROP requiring treatment, or disease that requires immediate treatment with laser therapy, cryotherapy, or anti-vascular endothelial growth factor therapy.42,43 Therefore, in our current design of the tele-education system, we did not include cases of ROP-related retinal detachment (stage 4 or stage 5 ROP). Future studies may include investigating the utility of tele-education for improving the diagnostic accuracy of stage 4 and stage 5 ROP. (6) The study subjects performed image-based interpretations of the clinical cases, but the consensus reference standard was determined using image-based diagnosis by 3 experienced readers in addition to the clinical diagnosis as determined by indirect ophthalmoscopy. Given the integration of the clinical diagnosis into the reference standard diagnosis, the diagnostic accuracy of these image interpretations may be more clinically relevant than a consensus reference standard that does not account for the clinical findings.
Overall, this study contributes to the body of ROP knowledge by showing that a web-based tele-education system for ROP diagnosis and management can improve the diagnostic competency of international ophthalmologists-in-training from a middle-income country. This system may provide significant utility in developing a standardized approach to educate international ophthalmologists for ROP care.
Supplementary Material
Financial Disclosure(s):
R.S., M.C.R., S.O., K.E.J., R.V.P.C., M.F.C.: Support - National Institutes of Health (Bethesda, MD) grant no. P30 EY010572.
J.P.C., S.O., M.F.C.: Support - National Institutes of Health (Bethesda, MD) grant no. R01 EY19474
M.F.C.: Support - National Institutes of Health (Bethesda, MD) grant no. R21 EY022387
M.F.C., R.S.: Support - grant no. T32 EY23211; Oregon Clinical and Translational Research Institute grant no. TL1TR000129 from the National Center for Advancing Translational Sciences at the National Institutes of Health, Bethesda, MD
S.N.P., J.P.C., R.S., M.C.R., S.O., K.E.J., R.V.P.C., M.F.C.: unrestricted departmental funding - Research to Prevent Blindness, New York, NY
R.V.P.C., K.E.J.: iNsight Foundation, New York, NY; The Bernadotte Foundation for Children’s Eyecare, New York, NY; Novartis Excellence in Ophthalmic Vision Award.
M.F.C.: Scientific advisory board - Clarity Medical Systems (Pleasanton, CA); Consultant - Novartis (Basel, Switzerland).
R.V.P.C.: Scientific advisory board - Visunex Medical Systems (Fremont, CA).
Abbreviations and Acronyms:
- AP-ROP
aggressive posterior retinopathy of prematurity
- PGY
postgraduate year
- ROP
retinopathy of prematurity
Footnotes
Supplemental material available at www.aaojournal.org.
Presented at: the Annual Meeting of the Association for Research in Vision and Ophthalmology; May 1–5, 2016; Seattle, WA.
The content is solely the responsibility of the authors and does not necessarily represent the official views of the NIH. No funding organizations had any role in the design or conduct of this research.
References
- 1.Gilbert C Retinopathy of prematurity: a global perspective of the epidemics, population of babies at risk and implications for control. Early Hum Dev. 2008;84(2):77–82. [DOI] [PubMed] [Google Scholar]
- 2.Gilbert C, Rahi J, Eckstein M, et al. Retinopathy of prematurity in middle-income countries. Lancet. 1997;350(9070): 12–14. [DOI] [PubMed] [Google Scholar]
- 3.American Academy of Pediatrics Section on Ophthalmology, American Academy of Ophthalmology, American Association for Pediatric Ophthalmology and Strabismus, American Association of Certified Orthoptists. Screening Examination of Premature Infants for Retinopathy of Prematurity. Pediatrics. 2013;131(1):189–195. [DOI] [PubMed] [Google Scholar]
- 4.National Eye Institute. Facts About Retinopathy of Prematurity (ROP). Available at: http://www.nei.nih.gov/health/rop/rop. Accessed November 14, 2014.
- 5.Cryotherapy for Retinopathy of Prematurity Cooperative Group. Multicenter trial of cryotherapy for retinopathy of prematurity: preliminary results. Pediatrics. 1988;81(5): 697–706. [PubMed] [Google Scholar]
- 6.Early Treatment For Retinopathy of Prematurity Cooperative Group. Revised indications for the treatment of retinopathy of prematurity: results of the Early Treatment for Retinopathy of Prematurity randomized trial. Arch Ophthalmol. 2003;121(12): 1684–1694. [DOI] [PubMed] [Google Scholar]
- 7.Gilbert C, Fielder A, Gordillo L, et al. Characteristics of infants with severe retinopathy of prematurity in countries with low, moderate, and high levels of development: implications for screening programs. Pediatrics. 2005;115(5):518–525. [DOI] [PubMed] [Google Scholar]
- 8.Chan RVP, Williams SL, Yonekawa Y, et al. Accuracy of retinopathy of prematurity diagnosis by retinal fellows. Retina. 2010;30(6):958–965. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 9.Myung JS, Chan RVP, Espiritu MJ, et al. Accuracy of retinopathy of prematurity image-based diagnosis by pediatric ophthalmology fellows: implications for training. J AAPOS. 2011;15(6):573–578. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 10.Nagiel A, Espiritu MJ, Wong RK, et al. Retinopathy of prematurity residency training. Ophthalmology. 2012;119(12): 2644–2645. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 11.Wong RK, Ventura CV, Espiritu MJ, et al. Training fellows for retinopathy of prematurity care: a Web-based survey. J AAPOS. 2012;16(2):177–181. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 12.Cook DA. Web-based learning: pros, cons and controversies. Clin Med. 2007;7(1):37–42. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 13.Chan RV, Patel SN, Ryan MC, et al. The Global Education Network for Retinopathy of Prematurity (Gen-Rop): development, implementation, and evaluation of a novel tele-education system (an American Ophthalmological Society thesis). Trans Am Ophthalmol Soc. 2015;113:T2. [PMC free article] [PubMed] [Google Scholar]
- 14.Yip MP, Chang AM, Chan J, MacKenzie AE. Development of the Telemedicine Satisfaction Questionnaire to evaluate patient satisfaction with telemedicine: a preliminary study. J Telemed Telecare. 2003;9(1):46–50. [DOI] [PubMed] [Google Scholar]
- 15.Ryan MC, Ostmo S, Jonas K, et al. Development and evaluation of reference standards for image-based telemedicine diagnosis and clinical research studies in ophthalmology. AMIA Annu Sym Proc. 2014:1902–1910. [PMC free article] [PubMed] [Google Scholar]
- 16.Hewing NJ, Kaufman DR, Chan RV, Chiang MF. Plus disease in retinopathy of prematurity: qualitative analysis of diagnostic process by experts. JAMA Ophthalmol. 2013;131(8):1026–1032. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 17.Chiang MF, Jiang L, Gelman R, et al. Interexpert agreement of plus disease diagnosis in retinopathy of prematurity. Arch Ophthalmol. 2007;125(7):875–880. [DOI] [PubMed] [Google Scholar]
- 18.International Committee for the Classification of Retinopathy of Prematurity. The International Classification of Retinopathy of Prematurity revisited. Arch Ophthalmol. 2005;123(7): 991–999. [DOI] [PubMed] [Google Scholar]
- 19.Wallace DK, Quinn GE, Freedman SF, Chiang MF. Agreement among pediatric ophthalmologists in diagnosing plus and pre-plus disease in retinopathy of prematurity. J AAPOS. 2008;12(4):352–356. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 20.Ataer-Cansizoglu E, Bolon-Canedo V, Campbell JP, et al. Computer-based image analysis for plus disease diagnosis in retinopathy of prematurity: performance of the “i-ROP” system and image features associated with expert diagnosis. Transl Vis Sci Technol. 2015;4(6):5. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 21.Campbell J, Ataer-Cansizoglu E, Bolon-Canedo V, et al. Expert diagnosis of plus disease in retinopathy of prematurity from computer-based image analysis. JAMA Ophthalmol. 2016;134(6):651–657. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 22.Abbey AM, Besirli CG, Musch DC, et al. Evaluation of screening for retinopathy of prematurity by ROPtool or a lay reader. Ophthalmology. 2016;123(2):385–390. [DOI] [PubMed] [Google Scholar]
- 23.Woo R, Chan RV, Vinekar A, Chiang MF. Aggressive posterior retinopathy of prematurity: a pilot study of quantitative analysis of vascular features. Graefes Arch Clin Exp Ophthalmol. 2015;253(2):181–187. [DOI] [PubMed] [Google Scholar]
- 24.Chiang MF, Thyparampil PJ, Rabinowitz D. Interexpert agreement in the identification of macular location in infants at risk for retinopathy of prematurity. Arch Ophthalmol. 2010;128(9):1153–1159. [DOI] [PubMed] [Google Scholar]
- 25.Kamholz KL, Cole CH, Gray JE, Zupancic JA. Cost-effectiveness of early treatment for retinopathy of prematurity. Pediatrics. 2009;123(1):262–269. [DOI] [PubMed] [Google Scholar]
- 26.Dunbar JA, Hsu V, Christensen M, et al. Cost-utility analysis of screening and laser treatment of retinopathy of prematurity. J AAPOS. 2009;13(2):186–190. [DOI] [PubMed] [Google Scholar]
- 27.Jackson KM, Scott KE, Graff Zivin J, et al. Cost-utility analysis of telemedicine and ophthalmoscopy for retinopathy of prematurity management. Arch Ophthalmol. 2008;126(4): 493–499. [DOI] [PubMed] [Google Scholar]
- 28.Dave HB, Gordillo L, Yang Z, et al. The societal burden of blindness secondary to retinopathy of prematurity in Lima, Peru. Am J Ophthalmol. 2012;154(4):750–755. [DOI] [PubMed] [Google Scholar]
- 29.Grupo Colaborativo Neocosur. Very-low-birth-weight infant outcomes in 11 South American NICUs. J Perinatol. 2002;22(1):2–7. [DOI] [PubMed] [Google Scholar]
- 30.Sommer A, Taylor HR, Ravilla TD, et al. Challenges of ophthalmic care in the developing world. JAMA Ophthalmol. 2014;132(5):640–644. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 31.Prakalapakorn SG, Smallwood LM, Helveston EM. ORBIS telemedicine users. Ophthalmology. 2012;119(4):880–881. [DOI] [PubMed] [Google Scholar]
- 32.Watts K, Taylor H, Taylor K. ORBIS—training nurses worldwide in ophthalmic care. AORN J. 1998;68(4):628–633. [DOI] [PubMed] [Google Scholar]
- 33.Graham M, De Sabbata S. Broadband Affordability. Oxford Internet Institute at University of Oxford; Available at: http://geography.oii.ox.ac.uk/?page=broadband-affordability. Accessed November 18, 2014. [Google Scholar]
- 34.Jain A, Agarwal R, Chawla D, et al. Tele-education vs classroom training of neonatal resuscitation: a randomized trial. J Perinatol. 2010;30(12):773–779. [DOI] [PubMed] [Google Scholar]
- 35.Shaikh N, Lehmann CU, Kaleida PH, Cohen BA. Efficacy and feasibility of teledermatology for paediatric medical education. J Telemed Telecare. 2008;14(4):204–207. [DOI] [PubMed] [Google Scholar]
- 36.Haney M, Silvestri S, Van Dillen C, et al. A comparison of tele-education versus conventional lectures in wound care knowledge and skill acquisition. J Telemed Telecare. 2012;18(2):79–81. [DOI] [PubMed] [Google Scholar]
- 37.Binks S, Benger J. Tele-education in emergency care. Emerg Med J. 2007;24(11):782–784. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 38.Sandars J, Haythornthwaite C. New horizons for e-learning in medical education: ecological and Web 2.0 perspectives. Med Teach. 2007;29(4):307–310. [DOI] [PubMed] [Google Scholar]
- 39.McKimm J, Jollie C, Cantillon P. ABC of learning and teaching: Web based learning. BMJ. 2003;326(7394):870–873. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 40.Scott KE, Kim DY, Wang L, et al. Telemedical diagnosis of retinopathy of prematurity: intraphysician agreement between ophthalmoscopic examination and image-based interpretation. Ophthalmology. 2008;115(7):1222–1228. [DOI] [PubMed] [Google Scholar]
- 41.Campbell JP, Swan R, Jonas K, et al. Implementation and evaluation of a tele-education system for the diagnosis of ophthalmic disease by international trainees. AMIA Annu Symp Proc. 2015;2015:366–375. [PMC free article] [PubMed] [Google Scholar]
- 42.Fijalkowski N, Zheng LL, Henderson MT, et al. Stanford University Network for Diagnosis of Retinopathy of Prematurity (SUNDROP): five years of screening with telemedicine. Ophthalmic Surg Lasers Imaging Retina. 2014;45(2): 106–113. [DOI] [PubMed] [Google Scholar]
- 43.Vinekar A, Gilbert C, Dogra M, et al. The KIDROP model of combining strategies for providing retinopathy of prematurity screening in underserved areas in India using wide-field imaging, tele-medicine, non-physician graders and smart phone reporting. Indian J Ophthalmol. 2014;62(1):41–49. [DOI] [PMC free article] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.