Accuracy and Reliability of a Handheld, Nonmydriatic Fundus Camera for the Remote Detection of Optic Disc Edema

Abstract

Background: Optic disc edema can be an important indicator of serious neurological disease, but is poorly detected using the direct ophthalmoscope. Portable fundus photography may overcome this difficulty.

Introduction: The purpose of this study was to determine the sensitivity and specificity of a handheld, nonmydriatic fundus camera for the detection of optic disc edema.

Materials and Methods: Retrospective review of nonmydriatic optic disc photographs taken with a portable fundus camera (Pictor Plus; Volk Optical, Mentor, OH) from the University of Michigan Neuro-Ophthalmology Clinics. We included 103 consecutive eyes with optic disc edema and 103 consecutive eyes without optic disc edema of 109 patients. Four masked neuro-ophthalmologists graded a single photograph of each optic disc presented in randomized order and documented the presence of optic disc edema. Sensitivity and specificity of graders' photographic interpretation was compared with clinical examinations. Reliability of assessments within and between graders was determined using kappa statistics.

Results: The sensitivity and specificity for detection of optic disc edema were 71.8–92.2% and 81.6–95.2%, respectively. Photos were found to be ungradable in 0–8.3% of cases. The intergrader reliabilities ranged from 0.60 [95% confidence interval (CI): 0.52–0.67] to 0.72 (95% CI: 0.66–0.77). Intragrader reliability ranged from 0.76 (95% CI: 0.63–0.92) to 0.82 (95% CI: 0.69–0.95).

Discussion: Photographs taken with portable, nonmydriatic technology met threshold sensitivity and specificity for remote screening for optic disc edema when performed by most, but not all graders. Reliability between graders was moderate–strong and strong within individual providers.

Conclusions: Portable photography holds promise for use in remote screening of optic disc edema.

Introduction

Optic disc assessment by direct ophthalmoscopy to detect optic disc edema is considered a basic clinical skill for all graduating medical students by the International Council of Ophthalmology.¹ This standard exists because optic disc edema is the key diagnostic sign in a number of ophthalmic diseases (e.g., anterior ischemic optic neuropathy) and may alert the examining physician to urgent diagnoses such as obstructive hydrocephalus or giant cell arteritis rather than a benign headache. In many cases, rapid detection of optic disc edema may result in timely administration of both vision and lifesaving therapy. For these reasons, it is an expectation that all physicians, both eye care specialists and nonspecialists alike, are equipped with the skills to use a direct ophthalmoscope to detect this important eye finding.

Despite agreement among physicians that ophthalmoscopy is an important skill,² nonophthalmology physicians report lack of confidence in their ability to perform direct ophthalmoscopy³ and may not examine the fundus even in patients presenting with visual complaints.⁴ Physicians cite several reasons for not routinely performing direct ophthalmoscopy outside the ophthalmology clinic.⁵ First, the direct ophthalmoscope is difficult to use.^6,7 Physicians across all levels of training report the lowest self-confidence for the fundus examination compared with all other physical examination skills surveyed.³ Despite intensive efforts to educate students regarding the appropriate use of a direct ophthalmoscope, educational interventions do not appear to result in long-term skill retention.^6–8 Second, patient-level factors often make direct ophthalmoscopy more difficult, including miotic pupils, media opacities (corneal or lenticular), critical illness, poor lighting conditions, or lack of cooperation with the examiner. Third, even when providers are able to view the fundus with the direct ophthalmoscope, they may not have the expertise to correctly identify and interpret important findings.^2,5

Fundus photography is increasingly seen as a means of overcoming the challenges of direct ophthalmoscopy. When fundus photographs were provided to emergency room physicians for patients under their care, the physicians reviewed the fundus photographs in 68% of cases compared with performing direct ophthalmoscopy on 14% of patients when photographs were not available.⁹ Despite receiving no specific training in interpreting fundus photographs, they accurately detected 46% of ocular abnormalities in comparison to 0% with direct ophthalmoscopy.¹⁰

Even in the hands of ophthalmologists, direct ophthalmoscopy appears to be less sensitive than fundus photography in detecting important findings, such as glaucomatous optic disc changes or diabetic retinopathy.¹¹ Additionally, nonmydriatic, handheld cameras can address patient-level barriers, such as miosis, and can be easily used at the bedside in hospitalized or immobile patients. For example, one commercially available camera, Pictor Plus (Volk Optical, Mentor, OH), has a 45-degree field of view, which is nine times that of a standard direct ophthalmoscope. Therefore, employing the use of nonmydriatic, handheld cameras to obtain photographs of the fundus that could be rapidly interpreted by an eye care specialist has the potential not only to overcome these barriers, but also to directly influence patient care.

The purpose of this study was to determine the sensitivity and specificity of handheld nonmydriatic fundus photography performed by paraprofessionals, who are not trained ophthalmic photographers, for the detection of optic disc edema. We hypothesized that these photographs, interpreted by an eye care provider, can reliably detect optic disc edema when compared with the standard fundus examination by a neuro-ophthalmologist.

Materials and Methods

We retrospectively reviewed charts of all patients over the age of 10 years who had been evaluated in the University of Michigan Neuro-Ophthalmology Clinics of all four participating neuro-ophthalmologists. A consecutive series of patients who had undergone nonmydriatic optic disc photographs of one or both eyes as part of routine clinical care using a handheld ophthalmic camera (Pictor Plus; Volk Optical) until 103 eyes with disc edema and 103 eyes without disc edema were identified. No eyes with optic disc anomalies or pathology other than edema were included. The Pictor Plus camera has a 5-megapixel (1,536 × 1,152) resolution and a 45-degree field of view. A medical assistant, optometrist, or resident physician took photographs of the optic discs after a brief educational session, but without prior experience or training as an ophthalmic photographer. The photographers took photographs at their discretion until they felt they had captured at least one adequate photograph. The best photograph was selected for the study by an independent ophthalmologist (L.L.B.).

The standard for comparison was clinical examination performed by fellowship-trained neuro-ophthalmologists at the initial consult, which was the same encounter when the photograph was taken. Neuro-ophthalmologists are trained experts in the identification of optic disc edema. Clinicians could use any of the standard 5 mechanical or 5 vascular signs to determine the presence or absence of optic disc edema as per their usual practice.¹² An independent ophthalmologist (L.L.B.) abstracted data from the clinical examination record to determine whether optic disc edema was present or absent. Additional data abstracted included date of birth, sex, race/ethnicity, presence of cataract, and presence of corneal opacity. Pupil dilation, when deemed necessary by the examining physician, occurred after obtaining photographs. Pupil size and the number of photographs taken per eye to obtain the highest quality photograph were collected from each record.

All photographs were graded by four fellowship-trained practicing neuro-ophthalmologists (L.B.D., W.T.C., H.M.G., J.D.T.; “graders”). A mean of 10.9 months [standard deviation (SD) = 2.0, range = 7.4–14.1 months] had elapsed between collection of the last photograph and assessment by the graders to minimize the likelihood of recall of specific patients or their imaging. Photographs were assessed on computer monitors that met the minimum acceptable standards for diabetic retinopathy screening.¹³ Each photograph was viewed randomly, without being associated with the fellow eye and without any clinical information. A random sample of 20% of the photographs was graded a second time 6 months after the initial grading session to determine the intragrader reliability. The presence of edema was recorded as either present, likely present, uncertain, likely absent, or absent.

The graders assessed the overall photograph quality (excellent, acceptable, and not gradable) and quality of the portion of the photograph that captured the optic disc (excellent, acceptable, and not gradable) measured against a series of reference images (Fig. 1) using the following criteria: A photograph was considered excellent if it was in focus and the entire disc was visualized. A photograph was considered acceptable if it was overexposed, underexposed or the optic disc was out of focus, but adequate to determine the presence or absence of optic disc edema. An ungradable photograph was one where the optic disc was out of focus or obscured by glare or other artefact. This grading scheme was adapted from the National Health Service (United Kingdom) Diabetic Grading Forms, which has been used in prior studies of various eye diseases.^14–17 For photos that were rated as not gradable, presence of optic disc edema was still assessed by readers, as the full ordinal diagnosis scale provided an option of “uncertain.”

Fig. 1.

Optic disc photograph quality standards. (A) Excellent. The optic disc is in focus and the entire disc is visualized. (B) Acceptable. Blurring or overexposure of the optic disc, but adequate to make a determination about the presence or absence of optic disc edema. (C) Nongradable. The optic disc is out of focus or obscured by glare.

Statistical Analysis

Descriptive statistics of the sample were used to summarize patient and eye characteristics, including mean and SDs for continuous measures, and frequencies and percentages for categorical measures. Repeated measures analysis of variance (ANOVA) was used to test for difference between photographer type (medical assistant vs. optometrist) on the average number of photographs taken per eye. The sensitivity and specificity to detect optic disc edema from handheld, nonmydriatic fundus camera photographs compared with the standard clinical examination diagnosis were calculated for each grader. The presence of optic disc edema by graders was defined as a response of “present,” “likely present,” or “uncertain.” To determine optimum grading parameters, statistics were also recalculated when defining the presence of optic disc edema as a grade of “present,” “likely present,” “uncertain,” or “likely absent.” Intergrader agreement for the diagnosis of optic disc edema was assessed with weighted kappa statistics using the full ordinal diagnosis scale and included 95% confidence intervals (CIs).¹⁸ Intragrader agreement was also assessed with weighted kappa for the 20% random sample of photographs that were reassessed by each grader. Single kappa statistics were also computed using the definitions for presence of optic disc edema that combine responses (“present,” “likely present,” or “uncertain;” “present,” “likely present,” “uncertain,” or “likely absent”). Kappa values of 0–0.33 were considered weak agreement, 0.34–0.66 as moderate agreement, and 0.67–1.00 as strong agreement.^18,19 Repeated measures logistic regression with generalized estimating equations (to obtain robust standard errors in the presence of dependent data) was used to investigate factors associated with the probability of an image being ungradable. A photograph was defined as being ungradable if at least one of the four graders reported it to be ungradable. Factors investigated in models included presence of cataract, presence of corneal opacity, pupil size, and type of professional who took the photograph (medical assistant, optometrist, resident). Odds ratios (ORs) and 95% CIs are reported. Statistical analysis was performed using SAS version 9.4 (SAS Institute, Cary, NC).

Standard Protocol Approvals, Registrations, and Patient Consent

Approval was obtained from the University of Michigan Institutional Review Board and data were collected in accordance with Health Insurance Portability and Accountability Act (HIPPA) regulations.

Results

We identified 109 subjects contributing a total of 206 eyes (97 subjects contributed 2 eyes to the study, 12 contributed 1) presenting between September 30, 2014 and April 17, 2015. Edema was diagnosed in 103 eyes (50%) by clinical examination. Of the 97 subjects that contributed both eyes to the study, 42 had no edema in either eye, 37 had edema in both eyes, and 18 had one eye with edema and one eye without edema by standard clinical examination. The median age at the time of visit was 37.7 years (interquartile range = 23.8–49.3 years); 80 subjects were female (73.4%). Eighty-seven (79.8%) subjects were White, 11 were Black (10.1%), 1 was Asian (0.9%), and 10 were other, refused, or unknown (9.2%). A total of 31 eyes had observed cataracts (15.0%), and 7 eyes had a corneal opacity that may potentially affect photograph quality (3.4%; buphthalmos, large areas of superficial punctate keratitis, anterior basement membrane corneal dystrophy, corneal thinning, stromal scaring). On average, pupil size was 4.6 mm (SD = 1.3).

Of the 3 professionals who photographed eyes, the medical assistant photographed 105 eyes (51.0%), the optometrist photographed 99 eyes (48.1%), and a resident physician photographed 2 eyes (1.0%). There was no significant difference in the average number of photographs taken per eye by the medical assistant compared with the optometrist (3.3 vs. 3.2, respectively; p = 0.6120 repeated measures ANOVA). No data were missing from the clinical records or grading forms and no adverse events occurred as a result of this study.

The sensitivity and specificity for the detection of edema from optic disc photographs versus the clinical examination ranged from 71.8–92.2% and 81.6–95.2%, respectively, over all four graders (Table 1). These statistics are based on the definition for presence of disc edema as responses of “present,” “likely present,” and “uncertain.” When the presence of disc edema was redefined to include the response of “likely absent,” the sensitivities increased (range = 80.6–98.1%), but the specificities decreased (range = 59.2–92.2%).

Table 1.

Sensitivity and Specificity for Diagnosing Optic Disc Edema Using Fundus Photos Compared with the Standard Clinical Examination (n = 206 Eyes)

	STANDARD CLINICAL EXAMINATION
DIAGNOSIS WITH PHOTO	EDEMA	NO EDEMA	SENSITIVITY (95% CI)	SPECIFICITY (95% CI)
Grader 1
Present, likely present, uncertain	95	19	92.2 (95.4–96.0)	81.6 (73.0–87.9)
Likely absent, absent	8	84
Present, likely present, uncertain, likely absent	101	42	98.1 (93.2–99.5)	59.2 (49.6–68.2)
Absent	2	61
Grader 2
Present, likely present, uncertain	74	5	71.8 (62.5–79.6)	95.2 (89.1–98.9)
Likely absent, absent	29	98
Present, likely present, uncertain, likely absent	83	8	80.6 (71.9–87.1)	92.2 (85.4–96.0)
Absent	20	95
Grader 3
Present, likely present, uncertain	82	7	79.6 (70.8–86.3)	93.2 (86.6–96.7)
Likely absent, absent	21	96
Present, likely present, uncertain, likely absent	95	20	92.2 (85.4–96.0)	80.6 (71.9–87.1)
Absent	8	83
Grader 4
Present, likely present, uncertain	89	16	86.4 (78.5–91.7)	84.5 (76.3–90.2)
Likely absent, absent	14	87
Present, likely present, uncertain, likely absent	98	39	95.2 (89.1–97.9)	62.1 (52.5–70.9)
Absent	5	64

CI, confidence interval.

The intergrader reliabilities (weighted kappas) for agreement of disc edema diagnosis between pairs of readers ranged from 0.60 (95% CI = 0.52–0.67) to 0.72 (95% CI = 0.66–0.77), over all six unique pairs. Table 2 reports simple kappa statistics for intergrader agreement of disc edema diagnosis using the definitions for the presence of optic disc edema that combine responses. All kappa statistics showed agreement in diagnosis of edema between pairs of graders significantly better than zero, chance agreement (all p < .0001). Intragrader reliability (weighted kappas) for agreement of disc edema diagnosis upon repeat grading, using the full ordinal diagnosis scale, ranged from 0.76 (95% CI = 0.63–0.92) to 0.82 (95% CI = 0.69–0.95), over all four graders. Table 3 reports simple kappa statistics for intragrader agreement of disc edema diagnosis using the definitions for the presence of optic disc edema that combines responses.

Table 2.

Intergrader Reliability for the Diagnosis of Optic Disc Edema

	GROUPING 1a KAPPA (95% CI)	GROUPING 2b KAPPA (95% CI)
Graders 1 vs. 2	0.63 (0.53–0.73)	0.52 (0.42–0.62)
Graders 1 vs. 3	0.68 (0.59–0.78)	0.65 (0.55–0.76)
Graders 1 vs. 4	0.76 (0.67–0.85)	0.71 (0.61–0.81)
Graders 2 vs. 3	0.72 (0.62–0.82)	0.69 (0.60–0.79)
Graders 2 vs. 4	0.73 (0.64–0.82)	0.51 (0.41–0.62)
Graders 3 vs. 4	0.69 (0.59–0.79)	0.66 (0.55–0.76)

^aPresence of optic disc edema was defined as a response of “present,” “likely present,” or “uncertain” versus “likely absent” or “absent.”

^bPresence of optic disc edema was defined as a response of “present,” “likely present,” “uncertain,” “likely absent” versus “absent.”

Table 3.

Intragrader Reliability for the Diagnosis of Optic Disc Edema

	GROUPING 1a KAPPA (95% CI)	GROUPING 2b KAPPA (95% CI)
Grader 1	0.89 (0.75–1.00)	0.52 (0.23–0.81)
Grader 2	0.85 (0.65–1.00)	0.62 (0.35–0.90)
Grader 3	0.76 (0.51–1.00)	0.77 (0.54–1.00)
Grader 4	0.76 (0.52–1.00)	0.83 (0.60–1.00)

^aPresence of optic disc edema was defined as a response of “present,” “likely present,” or “uncertain” versus “likely absent” or “absent.”

^bPresence of optic disc edema was defined as a response of “present,” “likely present,” “uncertain,” “likely absent” versus “absent.”

Image quality overall was rated as not gradable in only 1.0%, 8.3%, 0.0%, and 4.4% of photographs by graders 1–4, respectively. Twenty-four photographs (11.7%) were rated as not gradable by at least one grader (20 by a single grader, 4 by two graders). For the optic disc portion of the photographs, image quality was rated as not gradable in only 2.9%, 8.3%, 5.8%, and 5.3% of photographs by graders 1–4, respectively. Thirty-four optic disc photographs (16.5%) were rated as not gradable by at least one grader (23 by only a single grader, 10 by two graders, and 1 by three graders). Presence of a cataract or corneal opacity were significantly associated with increased odds of obtaining a nongradable photograph, both overall [cataract: OR 3.99 (95% CI = 1.36–11.67), p = 0.01; corneal opacity: OR 6.42 (95% CI = 1.12–36.86), p = 0.04] and for the optic disc portion only [cataract: OR 2.60 (95% CI = 1.00–6.77), p = 0.05; corneal opacity: OR 3.42 (95% CI = 1.64–30.50), p = 0.01]. Pupil size and type of photographer (optometrist vs. medical assistant) were not significantly associated with probability of an image being nongradable, regardless of whether the full photograph or optic disc portion of the photograph was being assessed.

Discussion

The use of fundus photography to aid clinicians in making diagnoses at the point of care has been met with rising interest.^20,21 However, any new mode of care delivery must maintain quality if it is to be considered a reasonable adjunct to standard clinical care. Our study demonstrates that nonmydriatic, handheld fundus photography with interpretation by a neuro-ophthalmologist is ∼80.6–98.1% sensitive when we used a grading strategy that grouped all positive and uncertain responses together and compared this to a response of “absence of optic disc edema.” With the alternative grouping that included “likely absent” together with “absent,” 3 of the 4 graders were able to detect optic disc edema with a sensitivity of approximately 80% or greater. Because the detection of optic disc edema should trigger evaluation by an eye care provider to confirm the findings in the event that there is any uncertainty regarding the presence of optic disc edema, we would advocate a screening strategy that maximizes sensitivity to capture all patients in which there is any ambiguity.

The sensitivity of the grading strategy used in our study satisfies the suggested requirements for new technologies that facilitate remote consultation (>80%) and is in line with reliability indices found in other telescreening studies.²² For example, a pooled meta-analysis of telescreening for diabetic retinopathy showed overall sensitivity of 95%, with a subgroup analysis of nonmydriatic photography demonstrating sensitivity of 80%.²³ The evaluating acute-phase retinopathy of prematurity (e-ROP) study using telescreening for ROP found a sensitivity of 81.9%,²⁴ and a retrospective study of patients assessed through an established diabetic teleretinal imaging program showed 74% sensitivity in detecting optic disc abnormalities.²⁵

By adopting a screening strategy to maximize sensitivity for the detection of optic disc edema, we found a decline in specificity from 81.6–95.2% to 59.2–92.2%. Other studies using nonmydriatic, handheld cameras have found similar difficulties achieving high specificity. Strategies to improve accuracy have included adding a clinical vignette¹⁶ or analyzing paired instead of individual eyes.²⁶ Although both interventions further improved sensitivity, neither improved specificity. It is possible that these types of strategies may be more successful in the assessment of optic disc edema, which is unlike glaucoma in that it is rarely an isolated finding. Furthermore, in cases of unilateral disc edema, the normal contralateral disc can be helpful to emphasize the edema.

The intergrader reliability was moderate to strong between the four readers. By comparison, the e-ROP study demonstrated a strong intergrader agreement of 85% for the detection of referral-warranted ROP. In the e-ROP study, trained nonophthalmologist readers who underwent a standardized certification program interpreted the ROP images.²⁷ The readers in our study were practicing neuro-ophthalmologists with no formal training in interpreting isolated optic disc images. It is possible that reliability would be increased with prior practice on standardized images or reference images at the time of grading.

Image quality in this study, both overall and of the optic disc alone, was excellent. For 3 of 4 readers, nonmydriatic photography by the Pictor camera exceeded the recommended standards of <5% regarding technical failure.²² Although the percentage of ungradable photos ranged from 0% to 8.3% by grader, no photos were consistently rated as ungradable by all four graders, or even three graders. There were only four photos rated as ungradable with agreement between two of the graders. For the optic disc portion of the photograph, only one photograph was deemed ungradable by 3 of the 4 reviewers. The differing rates seem to be due to variability in interpretation of “ungradable,” despite the graders being provided a reference sheet to determine quality of photo (excellent, acceptable, and not gradable).

Image quality was excellent despite the fact that photographs were taken by members of the eye care clinic without prior ophthalmic photography training. This finding suggests that by delegating photography to a technician, no additional time burden is placed on the physician. The Fundus Photograph vs. Ophthalmoscopy Trial Outcomes in the Emergency Department (FOTO-ED) study showed similar high-quality results for photographs taken by nurse practitioners, with 85% of photographs graded as high quality and only 3% of no clinical value.²⁸ When quality was compromised, we found that cataracts and the presence of certain corneal opacities may be problematic as in other studies.²⁹ Although we did not find pupil size to be associated with an increased probability of having nongradable photographs, other studies have found that pupil size matters, with technical failure occurring less often in dilated patients undergoing diabetic retinopathy screening.^26,30

Our study has limitations. The mean age of patients was 34.6 years, which is relatively young. Nonmydriatic photography may be more difficult in an older population with more miotic pupils, increased frequency of cataract, or presence of corneal or other ocular opacities. However, this difficulty could likely be overcome by using mydriatic drops when necessary. Even in ideal conditions, two-dimensional (2D) photography may not detect cases of subtle optic disc edema, such as was found only with high-resolution optical coherence tomography. Stereoscopic photographs would better emphasize any optic disc elevation, but require specialized equipment to obtain and view images. The 2D imaging modality used in this study is more suited for detection of significant edema, which is more likely to be clinically relevant. In clinical practice, however, neuro-ophthalmologists are routinely referred cases that have milder edema because nonexpert providers are often uncomfortable making that determination themselves. Therefore, in the current study, we suspect that there are likely to be milder cases of disc edema represented than might be seen by the nonspecialty provider. Further studies are warranted to assess how sensitivity of detection by this imaging modality may be influenced by the severity of optic disc edema.

Another limitation is that the graders and clinical examiners were the same, which may have biased our results. However, a minimum of 7 months had passed between the clinical examination and the grading, which makes recall of specific cases unlikely. Furthermore, each eye had been examined by only one grader, so 3 of the 4 graders would not have previously seen the eye. Additionally, all graders were fellowship-trained neuro-ophthalmologists. If photographic screening for disc edema is broadly implemented, availability of graders and their qualifications would have to be evaluated for the program. Certainly, photographs could be securely transferred to trained or qualified graders. In this setting, a validated grading scale for image quality would be necessary to ensure consistency. In other telemedicine studies, nonspecialist graders successfully manage diabetic retinopathy and ROP screening interpretation.^26,27,31 Lastly, handheld nonmydriatic cameras are not widely available in clinical settings. However, these cameras are becoming more accessible as technology advances and competition between manufacturers increases, driving down costs (as little as $5,000) to less than half the cost of a standard tabletop fundus camera.

In summary, a handheld, nonmydriatic camera is particularly well suited to settings where portability must be maximized and pupil dilation occurs rarely or cannot occur. The camera used in this study is capable of taking high-quality photographs to detect optic disc edema with >80% sensitivity in patients whose pupils are not pharmacologically dilated. Training requirements are minimal, and a nonphysician can take photographs. Efforts to improve the detection of optic disc edema, particularly by noneye care providers, should focus on implementation of fundus photography and interventions to further improve the sensitivity and specificity of screening strategies.

Disclosure Statement

The camera used in this study was provided to the University of Michigan, Kellogg Eye Center, for research purposes by Volk Optical, Mentor, Ohio. Volk Optical had no role in the design or conduct of this research. M.A.W.: Dr. Woodward receives grant support from NIH K23 (K23EY023596). L.B.D.: Dr. De Lott receives grant support from NIH K12 (K12EY022299-04). For all other authors, no competing financial interests exist.