Abstract
Objective
To evaluate the diagnostic performance of emergency providers’ (EPs) interpretation of robotically acquired retinal optical coherence tomography (OCT) images for detecting posterior eye abnormalities in patients seen in the emergency department (ED).
Methods
Adult patients presenting to Duke University Hospital ED from November 2020 through October 2021 with acute visual changes, headache, or focal neurologic deficit(s) who received an ophthalmology consultation were enrolled in this pilot study. EPs provided standard clinical care, including direct ophthalmoscopy at their discretion. Retinal OCT images of these patients were obtained with a robotic, semi-autonomous OCT system. We compared detection of abnormalities in OCT images by EPs with a reference standard, a combination of ophthalmology consultation diagnosis and retina specialist OCT review.
Results
Nine EPs reviewed the OCT images of 72 eyes from 38 patients. Based on the reference standard, 33 (46%) eyes were normal, 16 (22%) had at least one urgent/emergent abnormality, and the remaining 23 (32%) had at least one non-urgent abnormality. EPs’ OCT interpretation had 69% (95% CI: 49%–89%) sensitivity for any abnormality, 100% (95% CI: 79%–100%) sensitivity for urgent/emergent abnormalities, 48% (95% CI: 28%–68%) sensitivity for non-urgent abnormalities, and 64% (95% CI: 44%–84%) overall specificity. In contrast, EPs providing standard clinical care did not detect any abnormality with direct ophthalmoscopy.
Conclusions
Robotic, semi-autonomous OCT enabled ocular imaging of ED patients with a broad range of posterior eye abnormalities. EP OCT interpretation was more sensitive than direct ophthalmoscopy for any abnormalities, urgent/emergent abnormalities, and non-urgent abnormalities in this pilot study with a small sample of patients and EPs.
Introduction
Background
The current standard of care technique for examining the posterior segment of the eye, direct ophthalmoscopy, is underutilized and inaccurate in the hands of emergency providers (EPs).1,2 Fundus photography, a more feasible clinical alternative, also is only 46% sensitive for findings relevant to acute patient care, including optic disc edema, optic disc pallor, retinal vascular occlusions, intraocular hemorrhages, and Grade III/IV hypertensive retinopathy, when used by EPs.3 Optical coherence tomography (OCT) is a non-invasive imaging modality widely used in ophthalmology to obtain high-resolution, three-dimensional cross-sectional images of the eye. OCT has been shown to outperform fundus photography in the diagnosis of many retinal and optic nerve conditions, but OCT image acquisition relies on trained ophthalmic photographers and mechanical head stabilization for the patient.4 Recently, our group developed a robotic, semi-autonomous OCT (RAOCT) system capable of provider-free imaging of freely sitting or standing individuals.5
Importance
In current clinical practice, patients with urgent/emergent eye conditions often experience delays in care and corresponding worse outcomes due to lack of ophthalmologists available for consultation.6 Conversely, many patients without ocular pathology may be unnecessarily referred to ophthalmology, straining the limited eye care workforce and prolonging emergency department (ED) lengths of stay without appreciable clinical benefits.6–8 If RAOCT could assist EPs in making more accurate ophthalmology referral decisions, patient triage and resource allocation for acute eye care would potentially improve.
Goal of This Investigation
The aim of this pilot study was to evaluate the diagnostic performance of EPs’ interpretation of OCT images obtained by RAOCT for detecting posterior eye abnormalities in a non-clinical testing environment.
Methods
Study Design and Setting
This study was conducted at Duke University Hospital in Durham, North Carolina, USA. Duke University Hospital ED is a Level I trauma center. Standard direct ophthalmoscopes (Welch Allyn, Skaneateles Falls, NY, USA) are available in the examination rooms. The study was conducted under a Duke University Institutional Review Board-approved protocol. Written informed consent was obtained from study participants. The STARD 2015 checklist was followed and reported in Table S1.
Selection of Participants
Adult patients presenting to the ED were evaluated for eligibility from November 2020 through October 2021. Inclusion criteria included patients presenting with acute visual changes, headache, or focal neurologic deficit(s) who received an ophthalmology consultation, needed to establish the reference standard diagnosis. Exclusion criteria included hemodynamic instability, penetrating eye trauma, inability to follow commands, and presence of a confirmed ophthalmology diagnosis prior to presentation. Eligible patients were approached for study participation when research staff were available. EPs were asked about the direct ophthalmoscopy examination results for enrolled patients, and if no direct ophthalmoscopy was performed, the reason why the examination was not performed. If EPs were not available to provide this information, their documentation in medical records was reviewed.
OCT Imaging
Our robotic, semi-autonomous, and contactless OCT system has been described previously5; patients sat freely in front of the system without chin or head rests, and the RAOCT’s face and pupil cameras automatically located the eye of interest and maintained OCT system alignment to the pupil (Figure 1 and Supplementary Video 1).5 An operator with minimal training (i.e. without any formal training in ophthalmic imaging) confirmed the presence of the retina in the OCT image and triggered the acquisition of the OCT scan either 6 feet away from the patient or in a separate room via a local network connection to the RAOCT system. Each scan resulted in an OCT volume comprised of a series of OCT images to provide three-dimensional information.
Figure 1:
Top panel: robotic, semi-autonomous optical coherence tomography (RAOCT) system imaging a mock study participant. Bottom panel: screenshot of OCT viewing software with representative OCT images. Emergency providers reviewed OCT volumes with both cross-sectional B-scans (left) and an en face volume projection (right) that shows the location of the B-scan on the fundus. In the en face projection, the optic nerve head is the circular structure in the right middle of the image where the vessels converge.
EP OCT Image Interpretation
Nine attending EPs participated in OCT image review as part of this study (independent from patient care). The nine EPs ranged in years-of-practice from 7 to 31 and had a broad range of educational backgrounds and clinical experiences. A 30-minute basic training session was provided to all participating EPs, focusing on how to identify abnormal OCT images. Two principles of OCT B-scan interpretation were emphasized: 1) the healthy retina contains layers of tissue with smooth, regular curvature; irregularity may indicate pathology; 2) the healthy fovea and optic nerve appear as depressions; other morphologies (e.g., elevations) may indicate pathology (Figure S1). Applications of these principles were demonstrated with publicly available OCT images of healthy retinas and optic nerves, retinal detachment, optic nerve edema, retinal vascular occlusion, age-related macular degeneration, and diabetic retinopathy.
The nine EPs were masked to all patient data and interpreted a subset of OCT volumes selected out of all OCT volumes acquired for this study 11 to 35 days from the training date. To do this in a randomized fashion, we assigned three randomly selected EPs to review each volume. Each volume was interpreted by three EPs independently. The EPs were provided with both OCT B-scans and an en face (front-on) volume projection in a custom OCT review interface (Figure 1). Each EP graded the OCT volumes as normal vs. abnormal vs. ungradable. The overall interpretation of each volume was determined by the simple majority method to account for variation between EPs in analysis.
Reference Standard
The patients’ clinical diagnoses and fundus examination findings documented by the consulting ophthalmologists were extracted from medical records in a standardized fashion prior to EP interpretation of OCT images. A retina specialist, prior to EP OCT interpretation and masked to all patient data including the assessment by the consulting ophthalmologists, reviewed the OCT volumes collected for this study for presence or absence of abnormalities on OCT imaging and provided descriptions of any abnormalities. If there was a disagreement between the consulting ophthalmologist’s clinical assessment and the retina specialist’s OCT assessment, a second senior retina specialist with extensive experience in OCT reading, masked to all patient data, determined whether an abnormality was present or absent.
Outcomes
The primary outcome measure was the EPs’ detection of abnormalities on OCT as defined by the reference standard. The secondary outcome measures were the EPs’ detection of urgent/emergent and non-urgent abnormalities on OCT. An abnormality is considered urgent or emergent if an urgent or emergent evaluation by an ophthalmologist is required to prevent permanent visual impairment. Abnormalities in this category include but are not limited to optic nerve edema, retinal detachment, and acute retinal vascular occlusions.3 Other abnormalities were classified as non-urgent.
Statistical Analysis
A sample size of 53 eyes was required for 90% power at α = 0.05 to detect a 30% difference in sensitivity between EPs’ OCT interpretation and direct ophthalmoscopy, assuming that direct ophthalmoscopy was 31% sensitive, the highest reported sensitivity for EP performance of the exam in the literature.2 Medians and interquartile ranges (IQR) were reported for continuous data, and percentages were reported for categorical data. Proportions were calculated with 95% confidence intervals determined by cluster bootstrapping to account for within-patient correlations between eyes. If a proportion was 0% or 100%, the exact binomial method was used to calculate the 95% confidence interval. Specificity and sensitivity calculations considered unperformed and indeterminate direct ophthalmoscopy as “normal,” as our outcome measures were EPs’ detection of abnormality, and abnormalities cannot be detected when no examination is performed. For OCT image interpretation, the EPs’ inter-rater reliability was described by percent agreement and Gwet’s AC2, as studies have demonstrated that other measures of inter-rater agreement are not robust to within-patient correlations, such as the correlations between the two eyes of a patient.9,10
Results
Among 50 patients approached for study participation, 41 patients were enrolled in the study. After exclusions for technical errors, OCT volumes of 72 eyes of 38 patients were provided to EPs for interpretation (Figure S2).
Thirty-six (95%) patients had presented to the ED with acute visual changes, 18 (47%) had headache, and 2 (5%) had focal neurologic deficits. 18 (47%) had more than one of these presenting complaints. Patient demographic and clinical characteristics, and reference standard eye-level diagnoses are reported in Table 1. The percent agreement between consulting ophthalmologists who made diagnoses based on fundus examinations and the retina specialist who reviewed OCT images was 75%; a senior retina specialist reviewed the OCT images of the remaining 25% to establish the final reference standard.
Table 1:
Demographic and clinical characteristics of patients included in analysis.
| Total number of patients | 38 |
| Total number of eyes | 72 |
| Median age, years (interquartile range) | 58 (37–66) |
| n/N (%) | |
| Sex | |
| Male | 20/38 (53) |
| Female | 18/38 (47) |
| Race | |
| Caucasian/White | 27/38 (71) |
| Black/African American | 7/38 (18) |
| Asian | 2/38 (5) |
| Caucasian/White, Black/African American | 1/38 (3) |
| Not reported/declined | 1/38 (3) |
| Presenting symptoms/signs* | |
| Acute visual changes | 36/38 (95) |
| Headache | 18/38 (47) |
| Focal neurologic deficits† | 2/38 (5) 1 |
| Normal per reference standard | 33/72 (46) |
| Abnormal per reference standard | 39/72 (54) |
| Urgent/emergent abnormality | 16/72 (22) |
| Optic nerve edema | 9/72 (13) |
| Retinal artery occlusion | 3/72 (4) |
| Retinal vein occlusion | 2/72 (3) |
| Retinal detachment | 1/72 (1) |
| Acute retinal necrosis | 1/72 (1) |
| Chorioretinal lesion of unknown significance in the setting of contralateral endogenous panophthalmitis | 1/72 (1) |
| Intraretinal fluid in the setting of giant cell arteritis | 1/72 (1) |
| Non-urgent abnormality | 23/72 (32) |
| Drusen | 11/72 (15) |
| Staphyloma or peripapillary atrophy | 10/72 (14) |
| Epiretinal membrane | 10/72 (14) |
| Non-proliferative diabetic retinopathy | 2/72 (3) |
| Grade 2 hypertensive retinopathy | 2/72 (3) |
| Geographic atrophy | 2/72 (3) |
| Hazy media | 1/72 (1) |
Note that these sum to more than 100% as some patients had more than one symptom/sign.
One patient had slight gait abnormality and slight facial droop; the other patient had esotropia.
Among the 72 eyes, EPs were able to assess substantially more eyes with OCT than with direct ophthalmoscopy (100% vs 24%). EPs’ self-reported reasons for not assessing or not being able to assess patients with direct ophthalmoscopy are described in Figure S3. The most common reasons were exam deferred to consulting ophthalmologist (59%), poor view on attempted exam (10%), and concern for patient photosensitivity/discomfort (10%).
The EPs interpreted 39 (54%) out of 72 OCT volumes as abnormal, compared with 0 using direct ophthalmoscopy (Figure 2). No OCT volumes were identified as ungradable by EPs. Compared with the reference standard, the EPs’ use of OCT had 69% (95% CI: 49%–89%) sensitivity for any abnormality, 100% (95% CI: 79%–100%) sensitivity for urgent/emergent abnormalities, 48% (95% CI: 28%–68%) sensitivity for non-urgent abnormalities, and 64% (95% CI: 44%–84%) overall specificity. Sensitivity of EPs’ direct ophthalmoscopy was 0% (95% CI: 0%–5%) for all abnormalities.
Figure 2:
Emergency providers’ evaluation based on A, direct ophthalmoscopy and B, OCT interpretation compared with the reference standard. Overall, direct ophthalmoscopy was either indeterminate or not performed for most eyes, whereas OCT had 69% (95% CI: 49%–89%) sensitivity for any abnormality and 64% (95% CI: 44%–84%) specificity. EP, emergency provider; OCT, optical coherence tomography.
On the individual EP level, each EP interpreted a subset of 24 OCT volumes randomly selected out of the 72 OCT volumes obtained (Table S2). The median EP sensitivity for any abnormality was 71% (IQR: 62%–80%), the median EP sensitivity for urgent/emergent abnormalities was 100% (IQR: 83%–100%), the median EP sensitivity for non-urgent abnormalities was 50% (IQR: 22%–80%), and the median EP overall specificity was 63% (IQR: 60%–70%). EPs had very good inter-rater reliability for urgent/emergent abnormalities (percent agreement=83%, Gwet’s AC2=0.80) but poor inter-rater reliability for non-urgent abnormalities (percent agreement=51%, Gwet’s AC2 =0.01).
Limitations
Our study was conducted in a well-resourced academic medical center with a full-time, in-house ophthalmology consultation service. EPs may have detected more abnormalities if they were able to perform direct ophthalmoscopy for every patient; however, prior studies do not appear to suggest this would occur.1,3 Importantly, the low rate (24%) of direct ophthalmoscopy utilization seen in this study is not unique to our institution as studies performed in other emergency departments have reported rates of direct ophthalmoscopy utilization at 14–20%.1,11 Direct ophthalmoscopy may be more useful in smaller community hospitals, but studies in such settings are lacking.
Additionally, some of the authors of this study have a pending patent application for the RAOCT system, posing a potential conflict of interest. However, the patent is currently unissued and has not been licensed, so there is no current financial conflict. Further, the authors on the patent application did not handle the data or perform the data analysis in this study.
Finally, the RAOCT is an investigational device, and in this pilot study, the OCT images were not shown to providers in real-time to ensure that the study did not pose undue risks to patient care as the diagnostic value of OCT when interpreted by emergency physicians had not been established.
Discussion
This pilot study demonstrated the use of both robotically acquired OCT and OCT images in general by EPs. Key findings of this study are: 1) robotically acquired OCT was an effective tool for imaging a broad range of posterior eye abnormalities in ED patients; and 2) EPs could more effectively assess the posterior eye using OCT than using direct ophthalmoscopy.
RAOCT technology has many advantages over current standard-of-care tools. RAOCT minimizes the requirement for expertise for image acquisition or exam performance, unlike direct ophthalmoscopy, fundus photography, and ocular ultrasonography, all of which are dependent on operator skill.3,12 Furthermore, RAOCT can overcome shortcomings of direct ophthalmoscopy. OCT systems utilize a near-infrared light source and therefore do not shine visible light into patients’ pupils, limiting concerns for photosensitivity. Since images can be acquired remotely with the press of a button, RAOCT limits the need for close contact near patients’ faces, limiting infectious exposure for healthcare workers and patients. Furthermore, EPs could more effectively assess the posterior eye using OCT than using direct ophthalmoscopy. Even with limited training, EPs were able to assess 100% of the eyes they were asked to assess using the OCT images, but only 24% with direct ophthalmoscopy. Point-of-care ocular ultrasonography is another imaging tool that EPs may use. A recent meta-analysis demonstrated that emergency practitioners can accurately diagnose retinal detachment by using point-of-care ocular ultrasonography (94% sensitivity).13 However, there is limited literature on the utility of ocular ultrasonography for other urgent/emergent ocular conditions, such as optic nerve edema and retinal vascular occlusions, which EPs were able to detect on OCT in this study.
Moreover, EPs’ interpretation of OCT images outperformed direct ophthalmoscopy in detecting any abnormality, urgent/emergent abnormalities, and non-urgent abnormalities. In particular, EPs’ use of OCT had 100% (95% CI: 79%–100%; IQR: 83%–100%) sensitivity for urgent/emergent abnormalities (vs 0% [95% CI: 0%–5%] using direct ophthalmoscopy). This sensitivity was substantially higher than previously reported sensitivity for urgent/emergent abnormalities based on fundus photography (46%).3 This contrast is impressive, as interpretation of fundus appearance is an integral part of medical education expected of all EPs. Additionally, even though nearly half of the eyes were normal per reference standard, EPs, in their own practice, consulted ophthalmology for all patients. Therefore, the improved diagnostic performance using OCT may help EPs to make more accurate ophthalmology referral decisions.
A strength of this study is our inclusion of a diverse range of both urgent and non-urgent posterior eye abnormalities, including abnormalities not included when training EPs. The fact that EPs recognized abnormalities even when they were not trained specifically on those abnormalities (e.g., acute retinal necrosis) suggests that OCT may be a versatile diagnostic with real-world utility for EPs.
In conclusion, robotic, semi-autonomous OCT enabled ocular imaging of ED patients with a broad range of posterior eye abnormalities. EPs’ OCT interpretation was more sensitive than their use of direct ophthalmoscopy for any abnormalities, urgent/emergent abnormalities, and non-urgent abnormalities. EPs’ high sensitivity for urgent/emergent abnormalities based on OCT is especially relevant to the timely management of acute eye disease. The use of RAOCT in the ED could help EPs to effectively identify patients with posterior eye abnormalities and obtain timely ophthalmology consultation. However, this was a pilot study; future research is needed to validate RAOCT in diverse ED settings, including those without institutional ophthalmology consultation services. While EPs did not perform direct ophthalmoscopy for most patients in this study, reflecting the real-world practice patterns of direct ophthalmoscopy in the ED, a clinical trial in which all patients receive both OCT and direct ophthalmoscopy is the logical next step.
Supplementary Material
Acknowledgments:
The authors would like to thank Drs. Joshua Broder, Daniel Buckland, Bruce Derrick, David Gordon, Sophie Galson, Karthik Rao, and Jeff Vista for participation in OCT image interpretation; Drs. James Tian, Lucy Hui, Sri Meghana Konda, Rami Gabriel, Eun Young (Alice) Choi, Katherine S. Peters, Andrew Gross, Yuxi Zheng, Nizar Abdelfattah, Alexander Snyder, Mr. Elmer Balajonda, and Ms. Teresa (Terry) Hawks for assistance in patient enrollment; and Dr. Shivram Chandramouli for assistance in study conceptualization.
Grant support:
This study was supported by NIH/NEI (R01-EY029302). Ailin Song received research support from NIH/NCATS (TL1-TR002555) and Research to Prevent Blindness Medical Student Eye Research Fellowship. The funders had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript, or decision to submit the manuscript for publication.
Conflicts of interest:
RPM and ANK receive grant support from Johnson & Johnson Vision (issued to institution). RPM, JAI, and ANK receive royalties from Leica Microsystems. ATL has had funded research contracts with Roche Diagnostics, Inc.; Abbott Laboratories; Hospital Quality Foundation; Bristol Meyers Squibb. Ischemia Care, LTD.; GE; AstraZeneca; Forest Devices, Inc.; Regeneron; Becton Dickinson; SENSE Neuro Diagnostics; Ophirex, Inc. All of the above are work unrelated to this study. MD, PO, JAI, RPM, and ANK have a pending patent application for the robotically aligned optical coherence tomography system.
Footnotes
Publisher's Disclaimer: This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting proof before it is published in its final form. Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.
Meetings: This work has not been presented previously.
References
- 1.Bruce BB, Lamirel C, Wright DW, et al. Nonmydriatic ocular fundus photography in the emergency department. N Engl J Med. 2011;364(4):387–389. doi: 10.1056/NEJMc1009733 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 2.Petrushkin H, Barsam A, Mavrakakis M, Parfitt A, Jaye P. Optic disc assessment in the emergency department: a comparative study between the PanOptic and direct ophthalmoscopes. Emerg Med J. 2012;29(12):1007–1008. doi: 10.1136/emermed-2011-200038 [DOI] [PubMed] [Google Scholar]
- 3.Bruce BB, Thulasi P, Fraser CL, et al. Diagnostic accuracy and use of non-mydriatic ocular fundus photography by emergency department physicians: Phase II of the FOTO-ED study. Ann Emerg Med. 2013;62(1):28–33.e1. doi: 10.1016/j.annemergmed.2013.01.010 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 4.Morgan JIW. The fundus photo has met its match: optical coherence tomography and adaptive optics ophthalmoscopy are here to stay. Ophthalmic Physiol Opt. 2016;36(3):218–239. doi: 10.1111/opo.12289 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 5.Draelos M, Ortiz P, Qian R, et al. Contactless optical coherence tomography of the eyes of freestanding individuals with a robotic scanner. Nat Biomed Eng. 2021;5(7):726–736. doi: 10.1038/s41551-021-00753-6 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 6.International Council of Ophthalmology. Data on Ophthalmologists Worldwide. Published December 14, 2021. Accessed September 6, 2022. https://icoph.org/advocacy/data-on-ophthalmologists-worldwide/
- 7.Nari J, Allen LH, Bursztyn LLCD. Accuracy of referral diagnosis to an emergency eye clinic. Canadian Journal of Ophthalmology. 2017;52(3):283–286. doi: 10.1016/j.jcjo.2016.12.011 [DOI] [PubMed] [Google Scholar]
- 8.U.S. Bureau of Labor Statistics. Ophthalmic Medical Technicians. Accessed September 6, 2022. https://www.bls.gov/oes/current/oes292057.htm
- 9.Gwet KL. Computing inter-rater reliability and its variance in the presence of high agreement. British Journal of Mathematical and Statistical Psychology. 2008;61(1):29–48. doi: 10.1348/000711006X126600 [DOI] [PubMed] [Google Scholar]
- 10.Wongpakaran N, Wongpakaran T, Wedding D, Gwet KL. A comparison of Cohen’s Kappa and Gwet’s AC1 when calculating inter-rater reliability coefficients: a study conducted with personality disorder samples. BMC Medical Research Methodology. 2013;13-(1):61. doi: 10.1186/1471-2288-1361 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 11.Golombievski E, Doerrler MW, Ruland SD, McCoyd MA, Biller J. Frequency of Direct Funduscopy Upon Initial Encounters for Patients with Headaches, Altered Mental Status, and Visual Changes: A Pilot Study. Front Neurol. 2015;6:233. doi: 10.3389/fneur.2015.00233 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 12.Whitson MR, Mayo PH. Ultrasonography in the emergency department. Crit Care. 2016;20:227. doi: 10.1186/s13054-016-1399-x [DOI] [PMC free article] [PubMed] [Google Scholar]
- 13.Propst SL, Kirschner JM, Strachan CC, et al. Ocular Point-of-Care Ultrasonography to Diagnose Posterior Chamber Abnormalities: A Systematic Review and Meta-analysis. JAMA Netw Open. 2020;3(2):e1921460. doi: 10.1001/jamanetworkopen.2019.21460 [DOI] [PMC free article] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.