Skip to main content
PMC home page
AMIA Annual Symposium Proceedings logoLink to AMIA Annual Symposium Proceedings
. 2015 Nov 5;2015:366–375.

Implementation and evaluation of a tele-education system for the diagnosis of ophthalmic disease by international trainees

J Peter Campbell 1, Ryan Swan 1,2, Karyn Jonas 3, Susan Ostmo 1, Camila V Ventura 4, Maria A Martinez-Castellanos 5, Rachelle Go Ang Sam Anzures 6, Michael F Chiang 1,2, RV Paul Chan 3
PMCID: PMC4765571  PMID: 26958168

Abstract

Tele-education systems are increasingly being utilized in medical education worldwide. Due to limited human resources in healthcare in low and middle-income countries, developing online systems that are accessible to medical trainees in underserved areas potentially represents a highly efficient and effective method of improving the quantity and quality of the health care workforce. We developed, implemented, and evaluated an interactive web-based tele-education system (based on internationally accepted, image-based guidelines) for the diagnosis of retinopathy of prematurity among ophthalmologists-in-training in Brazil, Mexico, and the Philippines. We demonstrate that participation in this tele-education program improved diagnostic accuracy and reliability, and was preferred to standard pedagogical methods. This system may be employed not only in training, but also in international certification programs, and the process may be generalizable to other image-based specialties, such as dermatology and radiology.

Introduction

Medical education has undergone significant changes over the past decade with the integration of technology and online learning increasingly replacing physical lectures.1–5 In fact, so-called massive open online courses (MOOCs) are in some cases changing the entire paradigm of higher education away from on-site classroom learning.6–9 These online curricula have several advantages including standardization of content across large audiences, the ability to be utilized remotely, lower cost, and decreased dependence on local expert human resources for teaching.5,10 While in wealthier countries, this may change the form of knowledge delivery, in countries with less-developed healthcare and educational infrastructures, access to online tele-educational resources can dramatically change not only the form, but the quality and quantity of the educational content available to local trainees. Most importantly, tele-education systems allow for access to high quality education without an immediate need for an increase in local expert human resources, which may be non-existent or otherwise too busy to devote time to education.11,12 Therefore, the pedagogical argument for development of international tele-education systems is strongest for those specialties where the human resources are the most limited and/or the content is the most challenging to teach.

Retinopathy of prematurity (ROP) is a leading cause of childhood blindness worldwide.13 ROP care is clinically challenging and resource intensive. In the United States, despite a relative abundance of ophthalmologists and neonatologists compared to most of the world, there remain challenges to meeting the needs of ROP education as well as clinical care.14–16 Worldwide, this paucity of trained ROP providers is even more extreme in middle-income countries, where advancements in neonatal survival for preterm babies are resulting in increasing numbers of infants developing ROP.13 Moreover, accurate diagnosis of ROP can be challenging and many studies have demonstrated poor inter-examiner reliability, especially among providers with less experience, which raises serious concerns about the quality of care in medically underserved areas.14,16,17

Tele-education can improve the exposure to and standardization of ROP education, and has potential to improve diagnostic competency of ophthalmology trainees worldwide. We developed an interactive tele-education system for ROP education and implemented this system at ophthalmology training programs in Brazil, the Philippines and Mexico. The purpose of this project was to determine the efficacy and trainee satisfaction of a remotely implemented ROP tele-education system. This system has potential to improve the quality and widespread delivery of ROP education and therefore indirectly (but significantly) improve the quality of medical care particularly in medically underserved areas throughout the world. Furthermore, this study demonstrates validation of principles that may be generalizable to other medical specialties, particularly ones that rely heavily on image-based diagnosis such as dermatology and radiology.

Study Domain

ROP is an ocular disease that develops in low birth weight and low gestational age infants, affecting approximately 14,000–16000 children in the United States per year.18 Diagnosis requires accurate recognition of disease, which has been classified by international guidelines established to triage these infants into one of several categories: no ROP, mild ROP, moderate ROP, and severe (treatment-requiring) ROP.19 Approximately 90% of infants who develop ROP will have only mild or moderate disease and will require follow-up examinations every 1–2 weeks for the first few months of life.18,20 However, approximately 10% of patients will progress to severe ROP. For these latter infants, treatment with laser photocoagulation or intraocular injection of pharmacological agents has been shown to successfully prevent blindness in most cases. 18,20 Therefore, diagnostic accuracy of ROP that is high risk for requiring treatment is critical for preventing pediatric blindness.

Methods

The Weill Cornell Medical College (WCMC) and the Oregon Health & Science University (OHSU) Institutional Review Boards approved all aspects of the use and analysis of retinal images and educational material used in this study. Administration and analysis of the tele-education system was also reviewed by the WCMC Institutional Review Board, and was granted an exemption because it was considered research in an established or commonly accepted educational setting involving normal educational practices such as research on the effectiveness of instructional techniques, curricula, and instructional strategies. This study was conducted in adherence all federal and state laws and was in accordance with Health Insurance Portability and Accountability Act (HIPAA) guidelines.

Development of the Retinopathy of Prematurity Tele-education system

A secure web-based tele-education system was developed utilizing a repository of over 15,000 images captured from routine ROP exams. The system was coded in C# on the ASP.NET platform (Microsoft; Redmond, WA), and employed a SQL Server database. To facilitate software development and compatibility across different web browsers, the system was developed using an open source JavaScript wrapper (jQuery; Mozilla; Mountain View, CA). Modules were developed in both English and Portuguese.

Tele-education system content and procedures

Figure 1 demonstrates the procedure for the tele-education system, which consisted of four unique sections: (1) Pretest examination; (2) ROP tutorial; (3) ROP training chapters; (4) Post-test examination. Clinical cases were presented in the Pre-test examination, ROP training chapters, and the Post-test examination. Each case included relevant clinical demographics and retinal images of both eyes. (Figure 1A). Trainees were required to enter a diagnosis for all image-based characteristics of ROP as defined by an internationally standardized classification system: plus disease (no, pre-plus, plus), zone (I, II, III), stage (1–5), category (no ROP, mild ROP, moderate ROP, severe treatment-requiring ROP), and aggressive posterior-ROP (APROP) for each individual eye. For the purposes of this study, the primary outcome measure was the overall disease category: presence of mild-or-worse, moderate-or-worse, or severe (treatment requiring) ROP.

Figure 1.

Figure 1

Open in a new tab

ROP Tele-education system procedures. (A) demonstrates progression through the course (pre-test examination, ROP tutorial, training chapters, and post-test examination), and displays a sample clinical case. In the training chapters, following the selected diagnosis, trainees were provided immediate feedback (B) and targeted supplemental tutorial and review (C).

Sections 1 and 4: Pre-test / Post-test

At baseline, and following the completion of the ROP training chapters each trainee completed a set of 20 clinical cases with various degrees of ROP. 16/20 cases (80%) were unique and 4 (20%) were repeated to measure intra-grader reliability. Completed cases could be reviewed but not changed during the testing and no feedback was provided.

Section 2: ROP Tutorial

Following the pre-test examination, trainees completed a tutorial on ROP diagnosis and management designed by the co-authors (RVPC, KEJ, SO, MFC). The ROP tutorial included didactic information on the different classifications of ROP and pertinent management considerations (treatment, follow-up time). When trainees were in between different chapters, the ROP tutorial was available as a reference for clarification, but the tutorial was not available while trainees were in the pre-test or post-test examinations, or within a ROP training chapter.

Section 3: ROP Training Chapters

After completing the ROP tutorial, trainees were asked to complete five training chapters, each consisting of five clinical cases. Within each training chapter, trainees were given immediate feedback on their accuracy (Figure 1B) and targeted supplemental educational material was provided for any incorrect answers in four ways (Figure 1C): (1) Review of the image for which an incorrect response was provided; (2) Annotated images from the case that highlights the specific pathology; (3) Additional annotated images selected by the program from the database of validated images of other cases that highlight the specific pathology and; (4) Images that highlight the specific pathology corresponding to the (incorrect) chosen response. After reviewing the automated feedback based on their specific responses, trainees could then proceed to the next clinical case within the training chapter. After completing each chapter, trainees had the option to: (1) Proceed to the next chapter or section; (2) Review the ROP tutorial; (3) Review their responses and feedback from previous training chapters.

Consensus Reference Standard Diagnosis

To ensure accuracy in the educational content, particularly because of potential variability in diagnosis among physicians,21–23 a reference standard ROP diagnosis was determined for each case by consensus of three experts and the clinical diagnosis. This was done using previously-described methods: (1) The clinical diagnosis (based on complete ophthalmic exam by an experienced examiner) was recorded; (2) Each set of retinal images was interpreted by 3 experienced readers using a web-based system; (3) The image-based diagnosis that was selected by the majority of image readers was then compared to the clinical diagnosis.24 When these two diagnoses are the same, it was defined as the consensus reference standard diagnosis. When the diagnoses are different, all of the data were reviewed by two of the investigators (RVPC, MFC) along with two study coordinators (KEJ, SO), and a consensus reference standard was determined. This consensus reference standard diagnosis was then used for the purposes of this current study.

Study Population

Ophthalmologists in training were recruited by the co-authors (KEJ, RVPC, RA, CVV, MAMC) from training programs in Brazil, Mexico, and the Philippines. Ophthalmology trainees in all years were allowed to participate in the ROP tele-education system. At one program, a control group was provided the pre-test examination, followed by the post-test examination to be taken at least one month after the pre-test examination (to prevent immediate recall), but was not given access to the ROP tutorial or ROP training chapters. At this program, trainees were randomized 1:1 to participate in the program as part of the control group or take the tele-education system as indicated in Figure 1.

Survey

A survey was designed using a publicly available service (http://www.SurveyMonkey.com). Six questions were adapted from existing psychometric instruments to measure trainees’ attitudes using a 5-point Likert-type scale (1=strongly disagree, 2=disagree, 3=neutral, 4=agree, 5=strongly agree); one additional question allowed for free-text comments. Questions were reviewed by the authors for face validity and content validity and modified until all authors were satisfied with the survey instrument. Of the six questions, two assessed the trainees’ perception of their understanding of the diagnosis of ROP, three assessed the trainees’ attitudes toward preferred learning environment, and one assessed the trainees’ opinion of ease of use of the ROP tele-education system. All participants were invited to complete the survey.

Statistical Analysis

The main outcome measure was the post-ROP tele-education program sensitivity (compared to the pre-program sensitivity) of diagnosing any ROP (mild-or-worse ROP). The sensitivity and specificity of each disease category (mild-or-worse, moderate-or-worse, and severe) was measured before and after participation in the program. Sensitivities and specificities were compared using the paired t-test for each program and for all trainees as a single cohort, and by program, and statistical significance was considered to be a 2-sided P value <.05. For the training program with a control group, a secondary outcome measure included change in sensitivity of “active” education vs. control for each disease category. Data were analyzed using the R statistical software platform.25 Sensitivity and specificity values were calculated using the caret package.26 Based on the 4 cases that were repeated in both the pretest and post-test examination, intra-grader reliability was evaluated using the kappa (κ) statistic for chance-adjusted agreement in diagnosis. Cohen’s Kappa values were calculated using the irr package,27 and figures were generated using the ggplot2 package.28 A well-known scale was used for interpretation of results: 0 to 0.20, slight agreement; 0.21 to 0.40, fair agreement; 0.41 to 0.60, moderate agreement; 0.61 to 0.80, substantial agreement; and 0.81 to 1.00, almost-perfect agreement.22 Descriptive statistics were used to summarize the trainees’ responses to the post-ROP tele-education program web-based survey. The non-parametric Wilcoxon signed-rank test was used to compare the trainees’ pre- and post-ROP tele-education program perception of their understanding of the diagnosis of ROP.

Results

Characteristics of Study Population

Table 1 summarizes key characteristics of the three training programs and study participants. 92 ophthalmologists-in-training were provided access to the ROP tele-education program among the three programs, and 8 completed the program: 27 at program 1 (with 26 randomized to control), 43 at program 2, and 11 at program 3. None of the training programs involved have a formal ROP curriculum to assess clinical competency of graduates, but all programs had some exposure to ROP, and trainees were familiar with interpreting wide-angle retinal images such as those used in the program.

Table 1.

Study population

Program Characteristics Program 1 Program 2 Program 3
Number of trainees in program 58 54 11
ROP Curriculum (Y/N) N N N
Access to wide-angle camera (Y/N) Y Y N
Premature infants per year examined by faculty 94 350 58
Trainee Characteristics
Number completed program (%) 27* 43 11
Mean Age in Years (SD) 27 (1.2) 28 (2.5) 34 (2.8)
Gender
Male, n (%) 13 (48) 12 (28) 5 (45)
Female, n(%) 14 (52) 31 (72) 6 (55)
Training year
1st Year, n (%) 7 (26) 19 (44) 3 (27)
2nd Year, n (%) 8 (30) 15 (35) 2 (18)
3rd Year, n (%) 7 (26) 9 (21) 4 (36)
4th Year / Fellow, n (%) 5 (19) N/A 2 (18)
Open in a new tab
*

26 others participated as controls

Accuracy of ROP Diagnosis among Participants and Controls in the ROP Tele-education Program

Table 2 summarizes the pre- and post-ROP education program sensitivity and specificity for the diagnosis of ROP. The pre-test sensitivity of detecting mild-or-worse ROP was 81% (SE), which improved to 93% (SE) following the program, P < 0.001. There were statistically significant increases in sensitivity and specificity in all diagnostic categories. In subgroup analysis by program, these trends persisted with increases in sensitivity and specificity in each case.

Table 2.

Accuracy of retinopathy of prematurity (ROP) diagnosis in the participants in the ROP tele-education program.

ROP Disease Category Sensitivity, % (SE) Specificity, % (SE)
Pretest Posttest P value Pretest Posttest P value
All programs (N=81)
Mild or worse 81 (2) 93 (1) <0.001 69 (4) 88 (2) <0.001
Moderate or worse 71 (3) 79 (2) <0.002 79 (3) 90 (1) <0.001
Severe (treatment requiring) 55 (4) 66 (4) <0.022 79 (2) 89 (1) <0.001
Program 1 (N=27)
Mild or worse 83 (3) 95 (1) <.001 81 (5) 94 (2) <.001
Moderate or worse 67 (5) 81 (3) 0.003 88 (3) 92 (1) 0.18
Severe (treatment requiring) 54 (6) 71 (6) 0.012 85 (3) 89 (2) 0.27
Program 2 (N=43)
Mild or worse 80 (3) 92 (1) <.001 59 (6) 83 (3) <.001
Moderate or worse 73 (4) 81 (2) 0.111 71 (4) 88 (2) <.001
Severe (treatment requiring) 57 (6) 62 (5) 0.452 74 (3) 89 (2) <.001
Program 3 (N=11)
Mild or worse 78 (7) 90 (3) 0.176 81 (9) 95 (3) 0.176
Moderate or worse 73 (7) 82 (4) 0.377 84 (6) 89 (3) 0.182
Severe (treatment requiring) 48 (10) 66 (11) 0.343 81 (3) 89 (4) 0.11 2
Open in a new tab

ROP, retinopathy of prematurity; SE, standard error.

Table 3 demonstrates the change in sensitivity and specificity among the population of trainees randomized to participate in the program versus the control. In the control group there was no change in the sensitivity of detecting of mild-or-worse disease (P = 0.708), the main outcome measure.

Table 3.

Accuracy of retinopathy of prematurity (ROP) diagnosis in the trainees randomized to active tele-education and control at Program 1.

ROP Disease Category Sensitivity, % (SE) Specificity, % (SE)
Pretest Posttest P value Pretest Posttest P value
Tele-education group (N=27)
Mild or worse 83 (3) 95 (1) <.001 81 (5) 94 (2) <.001
Moderate or worse 67 (5) 81 (3) 0.003 88 (3) 92 (1) 0.18
Severe (treatment requiring) 54 (6) 71 (6) 0.012 85 (3) 89 (2) 0.27
Control (N=26)
Mild or worse 82 (4) 84 (3) 0.708 79 (5) 79 (5) 0.932
Moderate or worse 70 (5) 78 (4) 0.023 85 (3) 84 (4) 0.899
Severe (treatment requiring) 53 (7) 64 (7) 0.11 2 80 (3) 79 (3) 0.846
Open in a new tab

ROP, retinopathy of prematurity; SE, standard error.

Intra-Grader Agreement among Participants in the ROP Tele-education Program

Table 4 demonstrates the kappa statistics for intra-grader agreement among participants in the pre-test and post-test examinations of the ROP tele-education program. For the detection of mild-or-worse ROP among all participants (N=81), the kappa increased from moderate (0.57) to substantial agreement (0.8), with similar trends in every program and disease category. In the control group (N=26), the kappa also increased from 0.59 to 0.75, demonstrated improved intra-grader agreement with repeated testing over time, even without access to ROP tele-education program. 24 / 26 (92%) of control trainees waited the requested month between the pre-test and post-test examinations.

Table 4.

Kappa statistics for intra-grader agreement participants and controls in the pretest and posttest of the retinopathy of prematurity tele-education program

ROP Disease Category All Programs (N=81) Program 1 (N=27) Program 2 (N=43) Program 3 (N=11)
Pretest Posttest Pretest Posttest Pretest Posttest Pretest Posttest
Mild or worse 0.57 0.8 0.64 0.83 0.46 0.78 0.73 0.81
Moderate or worse 0.47 0.75 0.62 0.77 0.33 0.74 0.53 0.72
Severe (treatment requiring) 0.39 0.64 0.38 0.63 0.42 0.64 0.13 0.65
Open in a new tab

ROP, retinopathy of prematurity.

POST-RETINOPATHY OF PREMATURITY TELE-EDUCATION PROGRAM SURVEY OF TRAINEES

31 of the 81 trainees (38%) who completed the ROP tele-education program also completed the post-program survey.Figure 2 shows the results of this survey. With a maximum score of 5 for each question, the average score for each question was calculated. The average score for question 1 (adequate understanding of ROP before the tele-education system) was 2.4/5, as only 5 (16%) of the respondents agreed or strongly agreed that they had an adequate understanding of ROP diagnosis before taking the ROP tele-education program. Following the program, the respondents’ score averaged 4.3/5 suggesting that they felt their knowledge significant improved through the program (signed-rank test; P<.01). 31 / 31 (100%) of the respondents agreed or strongly agreed that they had an adequate understanding of the diagnosis of ROP after completing the ROP tele-education program. The average score for question 2 (ROP tele-education was easy to use) was 4.2/5 as 29 (94%) of the respondents agreed or strongly agreed that the system was easy to use. 30 (97%) of the respondents agreed that the feedback at the end of cases was helpful (question 3). For question 6 (preference for interactive system over textbook learning), the average score was 4.2/5 as 29 (94%) respondents agreed or strongly agreed that they learned more effectively with the automatic feedback opposed to textbook learning.

Figure 2.

Figure 2

Open in a new tab

Survey results of trainees who completed the retinopathy of prematurity (ROP) tele-education program (N = 31).

Discussion

Summary of Key Findings

We report the effective use of a tele-education system for teaching a clinically challenging ophthalmic disease in middle-income countries. We validate the concept that for image-based diagnosis, an internet-based tele-education system can be effective and preferred to more traditional educational models. The key findings from this study are: (1) This ROP tele-education system significantly improved the diagnostic accuracy and reliability of ophthalmology trainees. (2) Participants found the ROP tele-education system easy to use and they felt that they learned more effectively from ROP cases with automatic, targeted error-based feedback compared to a traditional textbook format.

Implications for Retinopathy of Prematurity Care and Telemedicine

These results demonstrate that a web-based tele-education system can be effective at improving the diagnostic accuracy for ROP for ophthalmology trainees in Brazil, Mexico, and the Philippines. Following completion of the course, trainees demonstrated 93% sensitivity at detecting mild-or-worse ROP, 71% sensitivity at detecting moderate disease, and 66% sensitivity at identifying severe (treatment-requiring) disease. To put this in perspective, these trainees outperformed a cohort of US pediatric ophthalmology fellows (85%, 53%, and 52% sensitivity for identifying mild, moderate, and severe disease, respectively) who completed ophthalmology training in the United States, but who did not complete this tele-education system.16 Additionally, participants in the tele-education program demonstrated improved intra-grader reliability of diagnosing ROP from “fair” or “moderate” to “substantial” agreement for all subtypes, including treatment-requiring ROP. These results are particularly significant in the diagnosis of ROP, in which despite the presence of international evidence-based, image-based guidelines, the literature consistently reports poor intra-rater and inter-rater reliability in ROP even among experts.21,22,29–31 As the routine care delivery model for ROP moves towards the use of telemedicine for screening and referral,32 the success of any telemedicine program will be dependent on the accuracy and reliability of the graders.33 Thus, though there remains room for improvement, particularly at recognizing moderate and severe ROP, as a pilot implementation of a tele-education system these results are extremely encouraging.

Implications for International Tele-Medical Education

The lack of human resources is perhaps the most critical factor limiting improvements in health care education and delivery in low and middle-income countries. In the US there is approximately one ophthalmologist for every 10,000 persons.34 In India, the number is closer to 1:100,000, and in most parts of Sub-Saharan Africa, it is less than 1:1,000,000.13,34,35 Compounding the human resource problem, among medical trainees in low and middle-income countries, access to quality medical education remains limited, and the ability to become well trained in the management of a complex disease such as ROP can be difficult. Even in the United States, ROP education has its challenges, as US trainees generally do not feel well qualified to perform ROP care independently without additional training.14–16 In middle-income countries where the number of cases of ROP is rising, the current medical education system has not been able to train enough qualified providers to manage emerging epidemic.13 Thus, a tele-education system that can be accessed from anywhere in the world that can demonstrate significant improvement in diagnostic accuracy and reliability has the potential to greatly impact the quality of ROP care, and mitigate the incipient rise in pediatric blindness in these countries. Moreover, these results suggest that a similar model could be used for other medical specialties that rely primarily on image-based diagnosis, such as dermatology,36,37 pathology,38 radiology,39 and cardiac electrophysiology and echography.40,41

In addition to being utilized to improve diagnostic performance among trainees and practitioners, a similar web-based interactive system could be used to standardize and certify medical specialists for any of these specialties. The development of this web-based training system utilized a novel method for validating the reference diagnosis for each image against which trainees could be evaluated.24 Compared to traditional examinations with a limited number of testable images, and no intra-test interaction with the test-taker, there is tremendous potential for an interactive web-based system with thousands of validated images not only to assess clinical competency but to improve the accuracy and reliability of participants, and thereby improve the quality of “certified” care providers around the world.

Limitations

There are several limitations that should be mentioned. First, due to small numbers limiting statistical power, we were only able to randomize participants into a control group at one of the three training programs, and we were not able to include a control group based on more traditional pedagogical methods, such as textbook chapters and classroom lectures. Presumably, traditional methods of learning can provide similar results. That said, none of these programs currently have an established curriculum (nor do, for that matter, most US programs),15 and we have demonstrated that providing remote access to this tele-education system can markedly improve diagnostic performance, exceeding that of US graduates, with limited local effort or resources.15,16 Second, in this pilot study, we did not attempt to determine the common diagnostic errors that led to incorrect responses. Third, we are unable to estimate whether a longer program or some modification of this program would perform better, or a shorter program perform equivalently. Fourth, though we did include a Portuguese system for the Brazil group, we relied on the common familiarity with medical English in Mexico and the Philippines. It is possible that these results may not easily translate when multiple translations are required for other countries and languages. We plan to investigate these questions in future studies.

Conclusions

In summary, we demonstrate that participation in this ROP tele-education system can increase diagnostic accuracy and reliability among trainees in Brazil, Mexico, and the Philippines.16 Program participants found the web-based, interactive system easy to use, and preferable to traditional learning methods. These results provide evidence that tele-education may be an effective modality to improve diagnostic performance in ophthalmologic disease, and particularly in middle-income countries, where there is an emerging epidemic, tele-education may represent a key strategic intervention to improve the quantity and quality of ROP care providers without added burden to existing local human and educational resources. As telemedicine and tele-education approaches to clinical care and education become more mainstream, we believe that these results may be generalizable to other medical specialties which rely on image-based detection of disease using international standards.

Acknowledgments

Supported by The St. Giles Foundation (RVPC); Grant R01 EY019474 from the National Institutes of Health (KEJ, MFC, RVPC, SO); The Bernadotte Foundation for Children’s Eyecare (KEJ, RVPC); The iNsight Foundation (KEJ, RVPC); Unrestricted departmental funding from Research to Prevent Blindness, Inc, New York (KEJ, MFC, RVPC, SO); Novartis Excellence in Ophthalmic Vision Award – XOVA (KEJ, RVPC).

References

  • 1.Kwon K, Saparova D, Hoffman K. Med Teach. Informa UK Ltd; London: 2014. Oct 15, Online lecture capturing system: Expected and actual effects of implementation in a problem-based learning medical curriculum; pp. 1–7. [DOI] [PubMed] [Google Scholar]
  • 2.George P, Dumenco L, Dollase R, Taylor JS, Wald HS, Reis SP. Introducing technology into medical education: two pilot studies. Patient Educ Couns. 2013 Dec;93(3):522–4. doi: 10.1016/j.pec.2013.04.018. [DOI] [PubMed] [Google Scholar]
  • 3.Chao SH, Brett B, Wiecha JM, Norton LE, Levine SA. Use of an online curriculum to teach delirium to fourth-year medical students: a comparison with lecture format. J Am Geriatr Soc. 2012 Jul;60(7):1328–32. doi: 10.1111/j.1532-5415.2012.04035.x. [DOI] [PubMed] [Google Scholar]
  • 4.Cardall S, Krupat E, Ulrich M. Live lecture versus video-recorded lecture: are students voting with their feet? Acad Med. 2008 Dec;83(12):1174–8. doi: 10.1097/ACM.0b013e31818c6902. [DOI] [PubMed] [Google Scholar]
  • 5.Liyanagunawardena TR, Williams SA. J Med Internet Res. 8. Vol. 16. JMIR Publications Inc; Toronto, Canada: 2014. Massive open online courses on health and medicine: review; p. e191. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 6.Hossain MS, Shofiqul Islam M, Glinsky JV, Lowe R, Lowe T, Harvey LA. A massive open online course (MOOC) can be used to teach physiotherapy students about spinal cord injuries: a randomised trial. J Physiother. 2015 Jan;61(1):21–7. doi: 10.1016/j.jphys.2014.09.008. [DOI] [PubMed] [Google Scholar]
  • 7.Stathakarou N, Zary N, Kononowicz AA. Beyond xMOOCs in healthcare education: study of the feasibility in integrating virtual patient systems and MOOC platforms. PeerJ PeerJ Inc. 2014;2(Suppl 2):e672. doi: 10.7717/peerj.672. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 8.Reich J. Education research. Rebooting MOOC research. Science. American Association for the Advancement of Science. 2015 Jan 2;347(6217):34–5. doi: 10.1126/science.1261627. [DOI] [PubMed] [Google Scholar]
  • 9.Heller RF. Learning by MOOC or by crook. Med J Aust. 2014 Mar 3;200(4):192–3. doi: 10.5694/mja14.00129. [DOI] [PubMed] [Google Scholar]
  • 10.Curran VR. J Telemed Telecare. 2. Vol. 12. SAGE Publications; 2006. Tele-education; pp. 57–63. [DOI] [PubMed] [Google Scholar]
  • 11.Mars M. Front Public Health. Vol. 2. Frontiers; 2014. Tele-education in South Africa; p. 173. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 12.Smith CE, Fontana-Chow K, Boateng BA, Azzie G, Pietrolungo L, Cheng-Tsallis A, et al. Pediatr Ann. 10. Vol. 38. SLACK Incorporated; 2009. Oct, Tele-education: linking educators with learners via distance technology; pp. 550–6. [DOI] [PubMed] [Google Scholar]
  • 13.Sommer A, Taylor HR, Ravilla TD, West S, Lietman TM, Keenan JD, et al. Challenges of ophthalmic care in the developing world. JAMA Ophthalmol. 2014 May;132(5):640–4. doi: 10.1001/jamaophthalmol.2014.84. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 14.Paul Chan RV, Williams SL, Yonekawa Y, Weissgold DJ, Lee TC, Chiang MF. Accuracy of retinopathy of prematurity diagnosis by retinal fellows. Retina (Philadelphia, Pa) 2010 Jun;30(6):958–65. doi: 10.1097/IAE.0b013e3181c9696a. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 15.Nagiel A, Espiritu MJ, Wong RK, Lee TC, Lauer AK, Chiang MF, et al. Retinopathy of prematurity residency training. Ophthalmology. 2012 Dec;119(12):2644–5. e1–2. doi: 10.1016/j.ophtha.2012.07.015. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 16.Myung JS, Paul Chan RV, Espiritu MJ, Williams SL, Granet DB, Lee TC, et al. Accuracy of retinopathy of prematurity image-based diagnosis by pediatric ophthalmology fellows: implications for training. J AAPOS. 2011 Dec;15(6):573–8. doi: 10.1016/j.jaapos.2011.06.011. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 17.Kang KB, Orlin A, Lee TC, Chiang MF, Chan RVP. The use of digital imaging in the identification of skip areas after laser treatment for retinopathy of prematurity and its implications for education and patient care. Retina (Philadelphia, Pa) 2013 Nov;33(10):2162–9. doi: 10.1097/IAE.0b013e31828e6969. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 18.Retinopathy of Prematurity Bethesda,MD:National Eye Institute [date unknown] Available from: https://www.nei.nih.gov/health/rop.
  • 19.International Committee for the Classification of Retinopathy of Prematurity . The International Classification of Retinopathy of Prematurity revisited. American Medical Association; 2005. pp. 991–9. [DOI] [PubMed] [Google Scholar]
  • 20.Good WV, Hardy RJ, Dobson V, Palmer EA, Phelps DL, Quintos M, et al. The incidence and course of retinopathy of prematurity: findings from the early treatment for retinopathy of prematurity study. Pediatrics. American Academy of Pediatrics. 2005 Jul;116(1):15–23. doi: 10.1542/peds.2004-1413. [DOI] [PubMed] [Google Scholar]
  • 21.Ataer-Cansizoglu E, Kalpathy-Cramer J, You S, Keck K, Erdogmus D, Chiang MF. Analysis of Underlying Causes of Inter-expert Disagreement in Retinopathy of Prematurity Diagnosis. Application of Machine Learning Principles. Methods Inf Med. 2015 Jan 12;54(1):93–102. doi: 10.3414/ME13-01-0081. [DOI] [PubMed] [Google Scholar]
  • 22.Chiang MF, Jiang L, Gelman R, Du YE, Flynn JT. Arch Ophthalmol. 7. Vol. 125. American Medical Association; 2007. Jul, Interexpert agreement of plus disease diagnosis in retinopathy of prematurity; pp. 875–80. [DOI] [PubMed] [Google Scholar]
  • 23.Williams SL, Wang L, Kane SA, Lee TC, Weissgold DJ, Berrocal AM, et al. Telemedical diagnosis of retinopathy of prematurity: accuracy of expert versus non-expert graders. Br J Ophthalmol. 2010 Mar;94(3):351–6. doi: 10.1136/bjo.2009.166348. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 24.Swan R, Ostmo S, Jonas K, Berrocal A, Drenser K, Horowitz J, Lee T, Simmons C, Martinez-Castellanos MA, Chan RVP, Chiang MF. AMIA Annu Symp Proc. 2014. Development and Evaluation of Reference Standards for Image-based Telemedicine Diagnosis and Clinical Research Studies in Ophthalmology. In Press. [PMC free article] [PubMed] [Google Scholar]
  • 25.R Core Team . R: A language and environment for statistical computing. R Foundation for Statistical Computing; Vienna, Austria: 2013. URL http://www.R-project.org/ [Google Scholar]
  • 26.Max Kuhn, Wing Jed, Weston Steve, Williams Andre, Keefer Chris, Engelhardt Allan, Cooper Tony, Mayer Zachary, Kenkel Brenton, the R Core Team. Benesty Michael. caret: Classification and Regression Training. R package version 6.0-37. 2014. http://CRAN.R-project.org/package=caret.
  • 27.Matthias Gamer, Jim Lemon, Ian Fellows, Puspendra Singh. irr: Various Coefficients of Interrater Reliability and Agreement. R package version 0.84. 2012. puspendra.pusp22@gmail.com. http://CRAN.R-project.org/package=irr.
  • 28.Wickham H. ggplot2: elegant graphics for data analysis. Springer; New York: 2009. [Google Scholar]
  • 29.Chiang MF, Thyparampil PJ, Rabinowitz D. Interexpert agreement in the identification of macular location in infants at risk for retinopathy of prematurity. Arch Ophthalmol. 2010 Sep;128(9):1153–9. doi: 10.1001/archophthalmol.2010.199. [DOI] [PubMed] [Google Scholar]
  • 30.Scott KE, Kim DY, Wang L, Kane SA, Coki O, Starren J, et al. Telemedical diagnosis of retinopathy of prematurity intraphysician agreement between ophthalmoscopic examination and image-based interpretation. Ophthalmology. 2008 Jul;115(7):1222–3. doi: 10.1016/j.ophtha.2007.09.006. [DOI] [PubMed] [Google Scholar]
  • 31.Chiang MF, Wang L, Busuioc M, YE Du, Chan P, Kane SA, et al. Arch Ophthalmol. 11. Vol. 125. American Medical Association; 2007. Nov, Telemedical retinopathy of prematurity diagnosis: accuracy, reliability, and image quality; pp. 1531–8. [DOI] [PubMed] [Google Scholar]
  • 32.Weaver DT. Telemedicine for retinopathy of prematurity. Current Opinion in Ophthalmology. 2013 Sep;24(5):425–31. doi: 10.1097/ICU.0b013e3283645b41. [DOI] [PubMed] [Google Scholar]
  • 33.Chiang MF, Starren J, YE Du, Keenan JD, Schiff WM, Barile GR, et al. British Journal of Ophthalmology. 10. Vol. 90. BMJ Publishing Group Ltd; 2006. Oct, Remote image based retinopathy of prematurity diagnosis: a receiver operating characteristic analysis of accuracy; pp. 1292–6. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 34.Spivey B. The opportunity for international ophthalmology in treating blindness. Trans Am Ophthalmol Soc. 2001;99:73–6. discussion76–7. [PMC free article] [PubMed] [Google Scholar]
  • 35.Palmer JJ, Chinanayi F, Gilbert A, Pillay D, Fox S, Jaggernath J, et al. Hum Resour Health. 1. Vol. 12. BioMed Central Ltd; 2014. Trends and implications for achieving VISION 2020 human resources for eye health targets in 16 countries of sub-Saharan Africa by the year 2020; p. 45. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 36.Ferrara G, Argenziano G, Piccolo D, Zalaudek I, De Rosa G. Tele-education in dermatopathology of pigmented lesions: is dermoscopy a valuable tool? J Telemed Telecare. 2004;10(3):183–6. doi: 10.1258/135763304323070869. [DOI] [PubMed] [Google Scholar]
  • 37.Nelson CA, Kovarik CL, Morssink CB. Tele-leprology: a literature review of applications of telemedicine and tele-education to leprosy. Lepr Rev. 2014 Dec;85(4):250–61. [PubMed] [Google Scholar]
  • 38.Pantanowitz L, Szymas J, Yagi Y, Wilbur D. Whole slide imaging for educational purposes. J Pathol Inform. 2012;3(1):46. doi: 10.4103/2153-3539.104908. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 39.Gotwald TF, Daniaux M, Stoeger A, Knapp R, Nedden zur D. The value of the World Wide Web for tele-education in radiology. J Telemed Telecare. 2000;6(1):27–30. doi: 10.1258/1357633001933907. [DOI] [PubMed] [Google Scholar]
  • 40.Bansal M, Singh S, Maheshwari P, Adams D, McCulloch ML, Dada T, et al. Value of interactive scanning for improving the outcome of new-learners in transcontinental tele-echocardiography (VISION-in-Tele-Echo) study. J Am Soc Echocardiogr. 2015 Jan;28(1):75–87. doi: 10.1016/j.echo.2014.09.001. [DOI] [PubMed] [Google Scholar]
  • 41.Finley JP, Beland MJ, Boutin C, Duncan WJ, Dyck JD, Hosking MC, et al. A national network for the tele-education of Canadian residents in pediatric cardiology. Cardiol Young. 2001 Sep;11(5):526–31. doi: 10.1017/s1047951101000762. [DOI] [PubMed] [Google Scholar]

Articles from AMIA Annual Symposium Proceedings are provided here courtesy of American Medical Informatics Association

RESOURCES