Fundus Photography vs. Ophthalmoscopy Outcomes in the Emergency Department (FOTO-ED) Phase III: Web-based, In-service Training of Emergency Providers

ABSTRACT

We evaluated a web-based training aimed at improving the review of fundus photography by emergency providers. 587 patients were included, 12.6% with relevant abnormalities. Emergency providers spent 31 minutes (median) training and evaluated 359 patients. Median post-test score improvement was 6 percentage points (IQR: 2–14; p = 0.06). Pre- vs. post-training, the emergency providers reviewed 45% vs. 43% of photographs; correctly identified abnormals in 67% vs. 57% of cases; and correctly identified normals in 80% vs. 84%. The Fundus photography vs. Ophthalmoscopy Trial Outcomes in the Emergency Department studies have demonstrated that emergency providers perform substantially better with fundus photography than direct ophthalmoscopy, but our web-based, in-service training did not result in further improvements at our institution.

Introduction

During the first phase of the Fundus photography vs. Ophthalmoscopy Trial Outcomes in the Emergency Department (FOTO-ED) study, we found that emergency providers (EPs) performed direct ophthalmoscopy on only 14% of patients who presented to our academic centre emergency department (ED) with complaints and conditions warranting ocular fundus examination.^1,2 Among the 350 patients enrolled, 13% had a relevant finding, such as papilloedema or grade III/IV hypertensive retinopathy, that should have altered the course of their ED management and disposition, and none of these funduscopic findings were identified by the EPs using ophthalmoscopy.^1,2 During the second phase, non-mydriatic ocular fundus photographs were provided to EPs during their routine clinical evaluations. EPs reviewed the photographs of 68% of the 355 patients enrolled and identified 46% of the relevant abnormalities during their routine evaluations without additional ophthalmology training.³

We anticipated that education on identifying abnormalities relevant to emergency care and on common artefacts that were frequently misidentified as abnormalities in prior FOTO-ED phases would improve EP performance. Thus, we undertook a quality improvement project to determine whether a web-based educational module would improve how often and how well EPs reviewed fundus photographs.

Materials and methods

Study setting and population

This project was evaluated and considered exempt by our Institutional Review Board as a quality improvement project. Adult patients who presented to our university hospital ED considered to be at high risk for ocular fundus findings based on chief complaints of headache, acute focal neurologic deficit, acute visual changes, or a triage diastolic blood pressure ≥120 mmHg were eligible for fundus photography. EPs who saw patients that had fundus photography obtained from March to December 2014 were included.

Photography protocol

Non-mydriatic photographs of the posterior pole of the ocular fundus (optic disc, macula, and major retinal vessels) were obtained from both eyes of included patients, placed in the medical record, and reviewed by EPs and neuro-ophthalmologists as previously described.³

Outcome measures

The primary outcome was the detection rate of relevant findings defined as optic disc oedema, optic disc pallor, retinal vascular occlusion, intraocular haemorrhages, and grade III/IV hypertensive retinopathy (Keith, Wagener, and Barker classification)⁴, findings that would have changed acute management in the ED or patient disposition. Abnormalities were considered mutually exclusive. Additional outcome measures of interest included pre- and post-test scores (each a 35-point test) and the frequency with which the EPs reviewed the fundus photographs.

Training protocol

The effectiveness of the training protocol at our institution was evaluated by comparing the outcome measures above before and after training for those who completed training and by comparing those who completed training to those who did not. Patients began to be included in the quality improvement project in March 2014. In May 2014 (month 2), a web-based educational in-service module was released. The web-based training module consisted of 35 pre-test questions, nine sections of training material (normal fundus, artefacts, isolated haemorrhage, severe hypertensive retinopathy, mild disc oedema, severe disc oedema, central retinal artery occlusion, optic disc pallor [subtle], optic disc pallor [severe]; see Figure 1), and 35 post-test questions. Core EPs who worked frequently in the Emory University ED were required by their clinical director to complete the module by Oct 2014 (month 7). Patient inclusion stopped 2 months later in December 2014 (month 9). Timing and length of training was recorded through digital timestamps.

Figure 1.

Examples of three training sections used in the web-based, in-service module.

Statistical methods

Statistical analysis was performed with R: A language and environment for statistical computing (R Foundation for Statistical Computing, http://www.R-project.org). Mean and standard deviation are reported for continuous, normally distributed data and median and interquartile range (IQR) are reported otherwise. Proportions were calculated and compared by the exact binomial method, and pre-/post-test scores by the Wilcoxon-Pratt signed-rank test. Statistical analysis was performed by BBB.

Results

Five hundred eighty-seven patients were included. Median age was 46 years (IQR: 34–61 years). Three hundred ninety-two patients (67%) were women and 333 patents (57%) were black, 206 (35%) white, and 48 (8%) were of other or unknown race.

Headache was a presenting complaint in 308 patients (52%), focal neurological deficit in 181 (31%), elevated diastolic blood pressure in 101 (17%), and visual changes in 65 (11%) [note that these sum to more than 100% because patients were allowed to meet more than one inclusion criteria]. Ocular fundus abnormalities were found in 74 (13%): 24 (32%) optic disc pallor, 20 (27%) optic disc oedema, 16 (22%) isolated retinal haemorrhages, 9 (12%) grade III/IV hypertensive retinopathy, and 5 (7%) retinal vascular occlusions.

Sixteen EPs were identified as core providers. Of these, 14 (88%) completed the training module between months 2 and 7 of the project as required. Three hundred fifty-nine (61%) of the included patients were evaluated by these 14 EPs.

EPs spent a median of 31 minutes on the training module (IQR: 24–42 minutes). Median test score was 65% pre-training (IQR: 56–72%) and 71% post-training (IQR: 66–80%; p = 0.06 comparing pre- vs. post-test score). The EPs reviewed 80 of 177 (45%) patients’ images prior to training vs. 78 of 182 (43%) after training (p = 0.73). They correctly identified 10 of 15 (67%) as abnormal prior to training and 8 of 14 (57%) after training (p = 0.88). They correctly identified 52 of 65 (80%) as normal before training and 54 of 64 (84%) after training (p = 0.67).

EPs who did not undergo training reviewed 79 of 228 patients’ images (35%), which was significantly less frequent than EPs who underwent training (45%, p = 0.03). They correctly identified 6 of 12 (50%) as abnormal and 53 of 67 (79%) as normal; neither was significantly different (p > 0.58) from the performance of the trained EPs.

Discussion

Consistent with previous phases of the FOTO-ED study, we found that about 13% of patients presenting to our ED with chief complaints of headache, acute focal neurologic deficit, acute visual changes, or a triage diastolic blood pressure ≥120 mmHg have important ocular fundus abnormalities.^2,3 The first two phases of the FOTO-ED study demonstrated that EPs perform significantly better with fundus photography than with direct ophthalmoscopy^2,3, and we hoped that providing additional training to EPs would lead to further improvements in their performance. However, while our web-based, in-service training showed a trend toward improved post-test scores, it did not result in any changes in provider performance in the clinical setting at our institution.

Instead, the only significant difference we found was that EPs who underwent training reviewed images about 25% more frequently than those who did not undergo training, but that difference was present before training. Thus, the difference observed most likely reflects the exposure of the majority of the core EPs to prior phases of the FOTO-ED study, which probably had several positive effects on their frequency of review (e.g., familiarity with the process, prior experience with the value of fundus photographs to their clinical care, higher interest).

So why did we fail to improve EP performance in reading fundus photographs? One reason could be the relatively short length of our training—30 minutes on average—and its self-led design; however, creating a longer, in-person course to provide additional depth and interaction was not considered practical for EPs. In fact, our intervention was designed based on the preferred learning method and timing parameters suggested to us by the EPs themselves. The failure of two core EPs to take the course, despite frequent reminders and the requirement of their clinical supervisor, demonstrates the significant time burdens EPs face that limit their capacity for continuing medical education, and perhaps also a relative lack of interest in learning more about ocular funduscopic examination.

In contrast to our experience, other brief educational interventions for EPs have been successful. For example, a five-minute educational intervention delivered by a clinical champion to EPs who prescribed opioid analgesics improved several measures of discharge prescribing, including a decrease in the dosage of opioids prescribed after the intervention.⁵ Likewise, a 1.5-hour course for EPs with no previous ultrasound experience for paediatric wrist fractures resulted in EPs missing only one minimal fracture.⁶

Alternatively, as in our case, other studies have shown that training does not guarantee improvement. For example, the implementation of a thrombolytic administration guideline in Australia, that included educational sessions conducted at three of the four treating venues and attended by the majority of the medical staff, had no impact on the proportion of eligible patients receiving thrombolysis or on door-to-needle time.⁷ It is also important to note that even when trainings result in short-term gains, they not infrequently fail to result in long-term improvements.^8–10

It is possible that more intense training could result in improvements in the EPs’ ability to read fundus photographs in both the short- and long-term, as was seen among medical students in the TOTeMS study^11,12: students correctly interpreted 85% of fundus photographs after initial training vs. 60% pre-training (p < 0.001), with sustained, significantly better performance over direct ophthalmoscopy on simulators at 1 year (72% vs. 65% correct, p = 0.004). However, the EPs in our current study were trained on identifying more subtle findings and on identifying abnormalities in the face of artefacts and other quality issues that we had previously noted degraded their performance.³ Tasks like these are likely more difficult to improve compared to identifying obvious findings on high-quality photographs as presented to the medical students. In addition, the EPs’ performance on interpreting fundus photography before the intervention was already quite good, particularly their frequency of correctly identifying normal photographs (86%), especially when compared to their performance using direct ophthalmoscopy.¹ It is even possible that we reached a ceiling of accuracy that would be difficult to overcome even with more extensive training.

Finally, because of the breadth of medical knowledge and practice required by EPs, of which ocular fundus examination is a small part, it would be unreasonable to expect EPs to perform fundus photography interpretations at the level of an ophthalmologist or neuro-ophthalmologist during routine clinical care. Instead, much like EPs rely on over-reads by attending radiologists to reduce diagnostic errors, it is likely best that fundus photographs taken in EDs be consistently over-read by ophthalmic providers, as was the case in this project and the prior phases of the FOTO-ED study. This is especially true given the ease with which digital photographs can be transferred in a secure fashion to other providers for telemedical evaluation, even over great distances.¹³ Overall, it does seem reasonable to expect EPs to review fundus photographs more frequently than they did in this study (about 45% of the time among the trained providers), since tele-ophthalmic review is unlikely to occur in real time and evaluation of the ocular fundus is generally recognized as a required part of the physical examination for patients presenting with the features included in this study.¹⁴ Thus, developing new tools to provide relevance to fundus findings in patient management scenarios may help to improve the frequency with which EPs review fundus photographs and ensure that important findings are not missed at the time of ED evaluation.

In conclusion, our brief web-based training did not improve provider performance at our institution. Future efforts directed at better demonstrating to EPs the value of fundus photography in emergency care and better integrating fundus photography reading and tele-ophthalmic consultation into routine practice may prove more useful than further attempts at improving training.

Funding Statement

This work was supported by the American Academy of Neurology; Knights Templar Eye Foundation; National Institutes of Health [K23-EY019341, KL2-RR025009, P30-EY006360, UL1-RR025008]; Research to Prevent Blindness.

Acknowledgments

This study was supported in part by an unrestricted departmental grant (Department of Ophthalmology) from Research to Prevent Blindness, Inc., New York, and by NIH/NEI core grant P30-EY006360 (Department of Ophthalmology).

Dr. Bruce received research support from the NIH/NEI (K23-EY019341) and the Knights Templar Eye Foundation, and received the American Academy of Neurology Practice Research Fellowship.

Dr. Wright received research support from NIH/PHS (KL2-RR025009).

Dr. Biousse received research support from NIH/PHS (UL1-RR025008).

Dr. Newman is a recipient of the Research to Prevent Blindness Lew R. Wasserman Merit Award.

No conflicting relationship exists for any author.