Quality and learning curve of handheld versus stand-alone non-mydriatic cameras

Abstract

Purpose

Nowadays, complex digital imaging systems allow detailed retinal imaging without dilating patients’ pupils. These so-called non-mydriatic cameras have advantages in common circumstances (eg, for screening or emergency purposes) but present limitations in terms of image quality and field of view. We compare the usefulness of two non-mydriatic camera systems (ie, a handheld versus a stand-alone device) for fundus imaging. The primary outcome was image quality. The secondary outcomes were learning effects and quality grade-influencing factors.

Methods

The imaging procedures followed standard protocol and were all performed by the same investigator. Camera 1 (DRS^®) was a stand-alone system, while Camera 2 (Smartscope^® PRO) was a mobile system. In order to evaluate possible learning effects, we selected an examiner with no prior training in the use of these systems. The images were graded separately by two experienced and “blinded” ophthalmologists following a defined protocol.

Results

In total, 211 people were enrolled. Quality grade comparisons showed significantly better grades for Camera 1. Both systems achieved better quality grades for macular images than for disc-centered images. No remarkable learning effects could be demonstrated.

Conclusions

Both camera systems are useful for fundus imaging. The greater mobility of Camera 2 was associated with lower image quality. For screening scenarios or telemedicine, it must be determined whether image quality or mobility is more important.

Introduction

In 1887, Lucien Howe described an experimental approach for fundus imaging in his publication “Photography of the Interior of the Eye”.¹ In the following decades, increased light sensitivity, optimized optics, new illumination sources and digital techniques led to a continuous increase in the image quality.² Today, state-of-the-art digital and complex imaging systems (cameras and scanners) allow detailed imaging of retinal structures without dilating patients’ pupils.³ Although non-mydriatic cameras provide partially reduced image quality and a reduced field of view compared to conventional cameras, the former have advantages in common circumstances (eg, for screening or emergency purposes).^4–7

The present study compares the usefulness of two non-mydriatic camera systems (ie, a handheld versus a stand-alone device) for fundus imaging needs. The primary outcome was image quality. The secondary outcomes were learning effects and quality grade-influencing factors.

Methods

People willing to be part of the study were randomly included after providing informed consent. The only other inclusion criteria were that participants must be at least 18 years of age. In addition to patients of the hospital, staff and patients’ companions could also be included. Each person underwent imaging in a completely dark room with both camera systems, in a randomly assigned chronology of the latter. Before taking the first images (right eye first, left eye second) with each camera system, a dark adaptation time of 2 minutes was allowed.

Camera 1 (DRS^®, software version: 2.2.11; CenterVue SpA, Padua, Italy) is a stand-alone system and uses a field of view of 45×40° with a 5-megapixel sensor with a resolution of 2,592×1,944 pixels. Camera 1 was mounted on a mobile lifting table; the patients sat on a vertically adjustable chair. Similar to the most diagnostic machinery in ophthalmology, Camera 1 had a chinrest and a forehead bar to allow for optimized positioning of the patient’s head. Thus, the camera lens faced toward the patient while the rear part of the camera, where the operating display was located, faced toward the physician. The pause time between each photograph was adjusted to 40 seconds with an integrated timer to allow for sufficient pupil relaxation after the camera flash, in accordance with comparable studies with non-mydriasis camera systems.⁴

Camera 2 (Smartscope^® PRO, software version: 3.2.6.3498; Optomed Oy, Oulu, Finland) is a handheld system that uses a field of view of 40×40° with a 5-megapixel sensor with a resolution of 1,536×1,152 pixels. The small on-device display can be used for aiming and quick quality control of the pictures taken. Camera 2 was used in a handheld setting with the patient and the physician stably positioned and sitting face to face, with the patient holding the camera system. The pause time between each photograph was adjusted to 40 seconds and measured with the help of a stopwatch.

With both camera systems, we aimed to capture, for each person, first an optic disc-centered image and then a macula-centered image (first of the right eye, then of the left eye). The image files of both systems were temporarily saved on internal storage and later copied for further evaluation using an external hard drive. In cases of bad image quality, one further attempt at increased quality was made by retaking the images.

Prior to this study, the examiner was an untrained medical doctor (LN) who read the manufacturer’s instructions for each imaging device and underwent a theoretical briefing from an experienced ophthalmologist (MAL) followed by a practical session with an examination of three patients for training purposes (six eyes).

The images were graded separately by two experienced and “blinded” ophthalmologists as subinvestigators (MG, CK) at the Tübingen Eye Reading Center using calibrated 27″ in-plane switching displays with full high-definition resolution and 32-bit color depth. The quality grading (Table 1; Figure 1) of an image was performed using a system, such as the 5-point scale of Lamirel et al.⁴

Table 1.

Quality grade classification

Grade	Description
1	Ideal quality
2	Subtle findings can be excluded; image quality not ideal
3	Obvious emergent findings can be excluded, no details
4	Excluding all emergency findings is not possible
5	Inadequate for any evaluation

Figure 1.

Image quality examples, beginning with grade 1 on the left side and ending with grade 5 on the right side.

Note: The upper row is from Camera 1 and the lower row is from Camera 2.

So that the results in the first step could be compared, all of the images taken were included. In the second step, only images from the participants’ right eyes were used, thus minimizing bias based on the reduction of pupil sizes after several photographs and the accompanying camera flashes. The right eye image selection is designated as “selection OD” (sOD) in the following text.

The collected data were descriptively analyzed using a statistical program (JMP 11.1; SAS Institute, Cary, NC, USA). All data were anonymously analyzed in accordance with the German directives on information security and data protection.

Written informed consent was obtained from all patients before the examination. The investigation followed the tenets of the Declaration of Helsinki. Approval from the responsible ethics committees (Ethics committee of the Medical Faculty, Eberhard Karls University, Tübingen; Ethics committee of the Medical Association Baden-Württemberg, Stuttgart) was obtained in advance. The study is registered at ClinicalTrials.gov (ID NCT02089009).

Results

In total, 211 (male: 35, female: 176) people were enrolled in the study. The median age was 35 years (minimum: 18 years, maximum: 66 years). The comparison of the achieved image quality grade for both camera systems was subdivided first for all images (Figure 2; Table 2; and for sOD only, Table 3). Fewer images were taken with Camera 2 than with Camera 1. This was due to a temporary mechanical defect within the period of data acquisition.

Figure 2.

Achieved quality grade (y-axis) for all images (upper panel) and sOD images (lower panel) divided into regions of interest, that is, macula versus disc (x-axis) and camera system (x-axis).

Note: Box and whisker diagrams show 5% and 95% quantiles (whiskers) and 25% and 75% quantiles (box).

Abbreviation: sOD, selection OD.

Table 2.

Descriptive overview of all evaluated images for both camera systems

Parameter	Camera 1	Camera 2
Images total	1,044	873
Macula-centered	541	433
Disc-centered	503	440
Quality of macula-centered images, mean (SD)	1.95 (0.93)	2.61 (0.68)	P<0.0001 (Wilcoxon/Kruskal–Wallis test)
Quality of disc-centered images, mean (SD)	2.21 (0.65)	2.69 (0.81)	P<0.0001 (Wilcoxon/Kruskal–Wallis test)
Macula-centered images with quality grade 1 or 2	484 (89%)	385 (83%)
Disc-centered images with quality grade 1 or 2	464 (92%)	343 (78%)

Table 3.

Descriptive overview of sOD images

Parameter	Camera 1	Camera 2
Macula-centered	220	190
Disc-centered	202	183
Quality macula-centered images, mean (SD)	1.94 (0.9)	2.57 (0.69)	P<0.0001 (Wilcoxon/Kruskal–Wallis test)
Quality disc-centered images, mean (SD)	2.15 (0.64)	2.73 (0.06)	P<0.0001 (Wilcoxon/Kruskal–Wallis test)
Macula-centered images with quality grade 1 or 2	222 (81%)	87 (39%)
Disc-centered images with quality grade 1 or 2	177 (71%)	86 (40%)

Abbreviation: sOD, selection OD.

Quality grade comparisons were performed using the Wilcoxon/Kruskal–Wallis test and showed significantly lower (better) quality grades for all images and for sOD for Camera 1. Both systems achieved better quality grades for macular images than for disc-centered images.

Furthermore, we evaluated the quality grading regarding learning effects of the used camera systems. Within the first 100 patients, we could not detect any significant learning effects. The highest increase in image quality could be seen as a trend for disc-centered images of Camera 2 (Figure 3).

Figure 3.

Achieved quality grade (y-axis) of images from the right eye of each patient (x-axis) divided into regions of interest, that is, macula versus disc (upper x-axis) and camera system (right y-axis).

Note: The gray fields mark the 90% density ellipse.

Logistic regression analyses were performed to evaluate the factors influencing the quality grade for sOD. The camera system resulted in a regression coefficient of 1.56 (standard error [SE]: 0.16), the image area in a regression coefficient of −0.28 (SE: 0.12), and the first-used camera device in a regression coefficient of 0.144 (SE: 0.20).

The internal measurement (Camera 1) of pupil diameter for sOD resulted in a mean diameter of 3.763 mm (SD: 0.73). A logistic regression analysis for Camera 1 sOD resulted in a regression coefficient for a pupil width of 2.55 (SE: 0.31). In cases in which the diameter of the pupil was ≥3.0 mm, we reached, in 82% of images, a quality better than 2.5. In cases with lower pupil width, only 25% of the images were of good quality (<2.5).

Discussion

Within our study, we compared two different camera devices that are optimized for non-mydriatic imaging. The systems were used by a previously inexperienced medical doctor. Image quality grades were superior for Camera 1 compared to Camera 2 and were better for macula-centered images than for disc-centered images. There were no significant learning curves to record. To eliminate technical bias (ie, pupil diameter and flash interval), we evaluated not only all images in total but also right eye images only (sOD).

In addition to scanning laser ophthalmoscopy⁸ and optical coherence tomography,⁵ non-mydriatic photography systems are used in telemedicine and screening programs. Therefore, evaluations of usability have been performed for several systems (eg, for diabetic retinopathy screening^9,10 or age-related macular degeneration screening).¹¹ Lamirel et al reported on important factors for increasing the quality of non-mydriatic photography operated by non-ophthalmically trained personnel in an emergency department.⁴ Despite the fact that groups of patients with, for example, age-related macular degeneration are older, we used a particularly young group of patients in order to create an optimal situation in which to compare the systems. We deemed this decision to be reasonable in this setting in order to eliminate the effects of pathology-caused influencing factors (eg, cataracts, corneal diseases, lack of concentration) on these two parameters. Still, further studies could investigate how great the effects of these factors really are on the results of each individual camera system.

We compared two independently developed camera devices that were designed for use outside an ophthalmological center. Beyond the common aim of achieving non-mydriatic photography, the systems are different with regard to their architecture. Camera 1 focuses on automation while Camera 2 aims for mobility. From our point of view, the price of increased mobility seems to be decreased stability and, therefore, reduced image quality. When the patient’s pupils are not dilated, it is a challenge to keep the patient’s eye and the camera aligned in terms of axis and distance (focus). Our expectation that there would be significant learning effects and an increase in image quality was not fulfilled. These results could change if a greater number of patients were studied. Still, factors such as anatomical limitations (eg, deep eye sockets) or physiological limitations (eg, tremors on the part of the physician or patient), which are likely to exist, especially when the patient group is not ideal, could lead to sustained image quality problems. So, the choice of which camera system is best might very much depend on the field in which the system is to be used; if the camera can be placed in a permanent position (eg, in the office of a general practitioner’s private practice, in a field hospital, or in a nursing home) Camera 1 will deliver better quality images, and the price for reduced mobility seems to be acceptable. If the screening is to be performed under very mobile conditions (eg, in rural areas in developing countries, or during home visits by a physician), the ability to take an image of relatively low quality might be preferable to taking no image or to losing time by setting up a heavy device for each examination.

In 2013, Ogunyemi et al investigated the use of autonomy versus the automation of non-mydriatic systems.¹² The study showed that the image quality of the automated system (which correlates to Camera 1) was obviously worse (in total, 61% of 142 images had a rating of excellent, good, or adequate) than our findings. These differences could have been caused by other imaging areas of the posterior pole and time delay protocols, but the results were not reported separately for the area subgroups.¹² In addition, Ogunyemi et al reported frustration and increased wait times with the automated system.¹² Within our setting, we could not confirm these findings, which could be based on differences in personnel.

The reduced quality of disc-centered images compared to macula-centered images is likely linked to the diameter of the pupil. The former is slightly tilted to image the disc, and therefore, the pass-through of light is reduced. We showed that reduced pupil diameters resulted in lower image quality for Camera 1. To get a good set of usable images, the width of the patient’s pupil should be 3.0 mm at a minimum. For Camera 2, such diameter values were not measured within the system. A limitation of the diameter measurements of Camera 1 is that we did not perform a crosscheck of these measurements with other devices.

Conclusion

Both camera systems studied here are useful for fundus imaging. The greater mobility of Camera 2 is associated with lower image quality. For screening scenarios or telemedicine, it must be determined whether image quality or mobility is more important. The limitations of our study include the relatively young patient group and the absence of crosschecked pupil measurements with Camera 1.