The Eye Phone Study: reliability and accuracy of assessing Snellen visual acuity using smartphone technology

Abstract

Purpose

Smartphone-based Snellen visual acuity charts has become popularized; however, their accuracy has not been established. This study aimed to evaluate the equivalence of a smartphone-based visual acuity chart with a standard 6-m Snellen visual acuity (6SVA) chart.

Methods

First, a review of available Snellen chart applications on iPhone was performed to determine the most accurate application based on optotype size. Subsequently, a prospective comparative study was performed by measuring conventional 6SVA and then iPhone visual acuity using the ‘Snellen' application on an Apple iPhone 4.

Results

Eleven applications were identified, with accuracy of optotype size ranging from 4.4–39.9%. Eighty-eight patients from general medical and surgical wards in a tertiary hospital took part in the second part of the study. The mean difference in logMAR visual acuity between the two charts was 0.02 logMAR (95% limit of agreement −0.332, 0.372 logMAR). The largest mean difference in logMAR acuity was noted in the subgroup of patients with 6SVA worse than 6/18 (n=5), who had a mean difference of two Snellen visual acuity lines between the charts (0.276 logMAR).

Conclusion

We did not identify a Snellen visual acuity app at the time of study, which could predict a patients standard Snellen visual acuity within one line. There was considerable variability in the optotype accuracy of apps. Further validation is required for assessment of acuity in patients with severe vision impairment.

Introduction

Incorporation of smartphone technology into daily modern medical practice is a rapidly growing trend. A survey published by the Manhattan Research Group demonstrated that 75% of physicians in the United States have purchased an Apple Inc. (Cupertino, CA, USA) mobile devices such as iPad, iPhone, or iPod.¹ Importantly, prior research conducted by the same group revealed up to 30% of doctors were using iPads to access patient records, radiology, and communicate information to their patients. With emerging acceptance of such technology by medical professionals, there is an increasing demand for medical programs and applications to be developed for such devices. Despite the rapid adoption of mobile technology, there is growing concern regarding the clinical validity and accuracy of these applications.² This has prompted national authorities to call for regulation of clinical software.³ Fundamental to clinical practice is the concept of evidence-based medicine, and as such clinical decisions need to be guided by the scientific literature.

The evolution of mobile technology in medical practice has been promulgated by the improvement in smartphone devices. Smartphones have a number of characteristics, which give them an edge over other technologies. These include portability, continuous internet connectivity, enough computing power to run complex applications and the simple fact that the majority of doctors have one in their pocket.⁴ The capacity for clinical use of smartphone technology is being increasingly documented in the medical literature. The assessment of wounds by picture messaging has become ubiquitous amongst plastic surgeons, and studies have found promising results.⁵ Communication between medical staff and hospitals has also been facilitated greatly with the use of ‘push email' and notifications; in addition to certain hospitals integrating paging systems with smartphone notifications.⁶ With the advent of custom designed applications, smartphone use has rapidly expanded and a number of specialties are producing innovative applications relevant to their own specialty, such as orthopedic decision support applications,⁷ offsite radiology access,^{8, 9} anesthetic techniques,¹⁰ or infectious disease physicians tracking epidemics,¹¹ to name a few. Smartphones have enabled volumes of medical literature and reference material to be accessible to students and clinicians in the palm of their hand.⁴

A recent survey amongst medical trainees found that >85% of respondents use a smart phone, and of these, the most popular was the iPhone.⁴ One of the reasons for the iPhone's popularity amongst medical professionals is the rapidly growing ‘App Store', which has the largest collection of applications amongst any device, and currently boasts >500 000 applications.¹² This translates into the most number of medical applications available for use in clinical practice. One application type that is quite popular amongst medical professionals are those to test the visual acuity of patients. The most common measurement of visual acuity is using the Snellen chart, which is engrained into physicians worldwide as the ‘standard' of measuring visual acuity. However, the Snellen chart has been criticized for optical inconsistencies.¹³ Although alternate charts such as logMAR charts are available for use in ophthalmic research,^{14, 15} the familiarity and availability of the 6-m Snellen chart in hospital and primary care settings have enabled the Snellen chart to remain the most popular and routinely used measure of visual acuity in clinical practice. As such, this study uses the Snellen acuity chart as a baseline comparative chart. The growing demand for fast and reliable measurement of visual acuity at the bedside or in an emergency department has prompted the development of ‘pocket' Snellen equivalent charts, including those on smartphones. Currently there is a paucity of literature supporting the clinical use of these smartphone Snellen visual acuity (iSVA) charts. We sought to identify the optical accuracy of Snellen eye chart smartphone applications, and to establish the clinical equivalence of these smartphone applications when compared with a standard 6-m Snellen visual acuity (6SVA).

Materials and methods

This study was conducted in two parts. First, a review of available Snellen chart applications on iPhone was performed. Second, a comparative study between the 6SVA and iSVA was conducted.

Application review and optotype height calculation

The Apple ‘App store' was searched using the keywords ‘eye test', ‘Snellen', ‘visual acuity', and ‘vision test' in 2012. This search was performed by two independent investigators to ensure all applications were identified. Applications were first screened to ensure they had a Snellen chart with clearly stated test distances. If a publisher had multiple similar applications with different names, only one was chosen for the purposes of this review. Each application was then downloaded, and the name, publisher, ratings data, and year of publication, and cost were recorded. Letter sizes displayed on screen for each application was measured with a ruler, and recorded in a Microsoft Excel spreadsheet.

A Snellen chart is designed such that each line would theoretically subtend an angle of 5° at the distance specified for that line. For example, letters on the 6/6 line of a Snellen chart should be sized such that it subtends 5 min of arc over 6 m. By using simple trigonometry, we were able to calculate the predicted optotype height for particular Snellen acuity line, using a chart held at any distance (Figure 1).

Figure 1.

Formula used to calculate theoretical optotype sizes. h—calculated optotype height (mm). L_m—unit calculated based on the line of the Snellen chart (eg 6/60, line modifier=10, 20/40=2). d—distance chart held (mm). c—denotes this value in degrees was converted to radian.

By comparing the calculated optotype height to the actual letter height, we were able to ascertain the percentage error for each line. The overall inaccuracy for the application was calculated by averaging the error for each line.

Subjects

The actual number recruited for the study was 88 patients. Adults were recruited for the study from inpatient wards at a university teaching hospital in Melbourne, Australia in 2012. We validated the sample size based on two methods. First, sample size was determined using a paired-sample t-test, which required a sample size of 28 to achieve a 90% power with significance level of 0.05 to detect a minimum difference of 0.17 logMAR. We chose to use 0.17 logMAR, as this approximates to the difference in one Snellen line, which we considered a clinically relevant distinction given the inherent discrepancies test–retest variation of Snellen charts.¹⁶ Second, sample size was also determined using a Pearson correlation coefficient r=0.59 derived from a pilot study which showed the minimum sample size required to reject the null hypothesis was 20. So our sample size for this study was large enough to detect a significant difference if it is was exist. Sample size calculation was performed using the software PS: Power and Sample Size, Dupoint, v3.0, 2009.

Participants were included if they were literate and able to comprehend and perform the test. Patients were randomly chosen from general medical and surgical wards that were medically fit enough to read the visual acuity charts from a sitting position in their hospital bed. Exclusion criteria for participants were applied to those under the age of 16 years, those of non English-speaking background, and patients too medically unstable to participate.

Charts used

The two charts used for this study were a conventional 6SVA chart, and a visual acuity chart application ‘Snellen' DrBloggs Ltd running on an Apple iPhone 4 Apple Inc., 2011. (See Figure 2) The iPhone 4 has screen dimensions of 77.5 × 51 mm, and has a pixel resolution of 960 × 640 pixels resulting in a resolution of 326 dpi. This application was chosen as it is the most accurate Snellen chart application which was free (Table 1).

Figure 2.

Screenshot taken of application ‘Snellen', by Dr Bloggs Ltd.

Table 1. iPhone Snellen chart applications. A review of all available Snellen chart applications in the Apple ‘App Store' comparing ratings, cost, test distances, and inaccuracy.

iPhone application name	Publisher	Rating (n)a	Cost ($)	Recommended distance chart held from patient (m)	Average line inaccuracy (%)
Eye Test	Bokan Technologies (2009)	2.5 (60)	0.99	1.50b	4.4
OptOK	Codev Ltd (2010)	N/A (0)	0.99	1.20	4.7
Snellen	Dr Bloggs Ltd (2011)	N/A (0)	Free	1.20	6.6
HOTV Acuity	KyberVision Consultingc (2009)	N/A (0)	0.99	1.20b	11.9
Eye Test Free	Claire Holmes (2011)	N/A (0)	Free	1.20	15.7
Check My Eyes	Scott Kehrberg (2011)	N/A (0)	0.99	1.20	17.2
iSnellen	Roninside (2011)	N/A (0)	Free	1.50	17.4
Eye Test Dr	Jose Barientos (2009)	2 (6)	0.99	1.50b	18.6
Eye Chart	DOK LLCc (2009)	2.5 (432)	Free	1.20	20.2
iExam	Corey Canfield (2010)	N/A (0)	0.99	0.60	35.6
Optician	Dimitro Gnyenko (2012)	N/A (0)	0.99	1.20	39.9

^aThis column represents the average rating, and number of ratings received for each application.

^bSome applications had variable testing distances. For purposes of consistency with other applications, we chose the closest distance to 1.2 m, where possible.

^cThese publishers had multiple similar applications under different names, only one was selected for this review.

Testing protocol

A single-examiner conducted data collection for consistency. No rehearsed reading of the eye chart was allowed before reading the charts for the study. For uniformity, acuity was measured only in the right eye, and patients wore their distance correction glasses, if used. A mirror was held at 3 m from the Snellen chart, which was held next to the patient, resulting in a simulated 6 m acuity measurement. This enabled patients who were bed bound to participate in the study. Thus allowing sampling of all hospital patients to ensure results would be applicable to general hospital practice. In order for accurate recording of Snellen acuity, charts should be read in a well-illuminated ambient environment. Accordingly, we held the Snellen chart beneath the room light, which was turned on for the examination, and provided adequate light.

Smartphone visual acuity was then recorded at a distance of 1.2 m (4 feet) from the participant as per the software instruction.

Letter scoring

A consistent single-letter scoring methodology was employed for both charts. The single-letter scoring system has been demonstrated in logMAR charts to provide a more accurate and consistent measurement of visual acuity compared to a simple line assignment method whereby acuity is the smallest line at which majority of letters are correctly identified.¹⁷ In accordance with published protocols, Snellen chart line acuity measurements were converted to a logMAR score in order to allow statistical analysis to be performed.¹⁸

Statistical analysis

Data were stratified based on the visual acuity from the 6SVA reading and logMAR scores were calculated for each of these groups. Paired sample t-test was performed to compare the mean difference between two groups with the null hypothesis that the mean of the difference between two charts was zero for the repeated subjects. Data analysis was performed using SPSS version 17, Chicago IL.

Ethics

The study was approved my the Human Research and Ethics Committee of the Royal Victorian Eye and Ear Hospital, Australia and was conducted in accordance with the Declaration of Helsinki.

Results

Application review

A total of 11 Snellen chart applications were identified (Table 1). Two developers were identified who had multiple applications (paid and free) with similar functions. Applications were developed between 2009 and 2012. Eight of the eleven applications had no documented rating, and only two applications had a significant number of ratings (>50). It was interesting to note that of the applications with reviews, the average rating was 2–2.5 stars on an arbitrary 5-star scoring system. Applications requiring payment for download were most common, with 64% of these requiring a payment of $AUD0.99. Five of the eleven applications were designed for the iPhone to be held at a distance of 1.2 m from the patients. We found this was a convenient distance, as it approximated to the end of the bed for a patient sitting with the head of the bed inclined at this particular hospital. Three applications included multiple Snellen charts, which could be used at variable distances.

Considerable variation was noted in accuracy of the applications when compared with the correct calculated optotype sizes, with error rates varying from 4.40 to 39.90%. The most accurate application available for free was ‘Snellen' by Dr Bloggs Ltd (2011) with an average line inaccuracy of 6.6%. Overall, Eye Test by Bokan Technologies (2009) was the most accurate application with an average line inaccuracy of 4.4%, but cost $AUD 0.99. Eight applications (72%) had an inaccuracy greater than 10%. Two applications were found to be grossly inaccurate with average line inaccuracies greater than 30%.; and only two applications had a significant number of reviews (greater than 50).

Six-metre Snellen chart vs iPhone Snellen chart

The mean logMAR acuity when using the 6SVA was 0.180 (SD=0.18), and the mean logMAR acuity using the iSVA was 0.160 (SD=0.18), both of which equate to approximately 6/9 Snellen acuity. There was no statistically significant difference between the mean logMAR acuity values of the two charts (P=0.293). Overall, the mean difference in logMAR acuities between the two charts was 0.02 logMAR (95% limits of agreement −0.332,0.372 logMAR). Figure 3 is a Bland–Altman plot that demonstrates an overview of the data.

Figure 3.

Bland–Altman plot of the differences in the logMAR visual acuity scores between the iPhone visual acuity chart and the 6-m visual acuity chart. The x axis represents the mean logMAR score between the two charts, and the y axis represents the mean difference between the two measurements. The mean difference in logMAR acuity (0.02 logMar) is shown by the central blue line. There appeared to be no association between the mean difference and the magnitude of the measure. A difference of one Snellen visual acuity line (0.17 logMar) is represented by the black solid lines, and the dashed lines represent the limits of agreement (mean difference±1.96 × SD).

The majority of patients tested had a visual acuity between 6/6 and 6/9 (31.8% for both 6SVA and iSVA). Only few patients (5.68% for both 6SVA and iSVA) had visual acuities worse than 6/18.

We further analyzed the data by stratifying patients into categorical groups according to their visual acuities as recorded by 6SVA. We then matched the recorded iSVA for the corresponding patients. Using these stratified groups, we then compared the visual acuities between the 6SVA and the iSVA (Table 2). In total, five groups of visual acuity were stratified. The largest mean discrepancy in logMAR acuities was noted in the group of patients with 6SVA >6/18 (n=5), who had an approximate difference of two Snellen visual acuity lines (0.276 logMAR).

Table 2. Frequency distribution of visual acuities amongst Standard 6-m Snellen visual acuity (6SVA) and iPhone Snellen visual acuity (iSVA).

Group	n	Mean 6SVA in LogMAR	Mean iSVA in LogMAR	Mean difference in LogMAR (Limits of Agreement) Limits of Agreement=mean difference ±1.96 × SD of the differences
≤6/6	22	−0.051	0.041	−0.10 (−0.22, 0.20)
>6/6 to ≤6/9	28	0.138	0.210	−0.071 (−0.37, 0.23)
>6/9 to ≤6/12	18	0.250	0.217	0.034 (−0.27, 0.34)
>6/12 to ≤6/18	15	0.386	0.253	0.132 (−0.25, 0.51)
>6/18	5	0.559	0.283	0.276 (−0.06, 0.61)
Overall	88	0.180	0.160	0.02 (−0.332, 0.372)

This column represents the average rating, and number of ratings received for each application.

Discussion

For patients with visual acuity equal to or worse than 6/18 we noted the largest difference in the mean recorded acuities. An obvious limitation of this group was the small sample of patients that fell into this acuity range, which limits the interpretation of this data in isolation. We compared mean logMAR difference between the two charts using equivalence in Snellen visual acuity lines rather than by logMAR units in order to keep the results clinically relevant. For example, it is more tangible for an examiner to interpret that the difference between the 6/6 and 6/9 is one line on a Snellen chart, rather than a 0.176 difference in logMAR units. The practical significance is that the margin of error between the two charts is less than one complete line, which would result in similar patient care decisions based on the visual acuity measurements from either chart.

Since 1862, visual acuity has been tested in the general clinical setting largely by Snellen visual acuity testing, and most physicians would consider the 6SVA chart to be the acceptable standard today. We recognize that the Snellen chart in itself has considerable flaws, such as inconsistent letter sizing from one line to another, large gaps in visual acuity at the lower end of the scale, unequal legibility of letters used and disjointed spacing between letters and rows, inconsistent repeatability.¹⁹ Current ophthalmic research literature tends to favor use of the ETDRS chart,²⁰ however, this brings its own limitations of an unfamiliar scoring system to nonopthalmic specialists, and increased time required to perform the test.¹⁷ Consequently, from the perspective of routine clinical practice, Snellen visual acuity testing is still the most popular and accessible measure of visual acuity.

The study sample was recruited from a general hospital sample of patients rather than those with ophthalmic pathology. This allows the results of this study to be relevant to general medical professionals who are most likely to use the iSVA in their daily practice. This is in contrast to ophthalmologists and ophthalmic care facilities that are more likely to have convenient access to a 6SVA in their clinical setting.

Through our Apple iPhone application review we demonstrated that there was varying quality and accuracy of Snellen charts developed for the iPhone. This has implications both for Snellen visual acuity testing, and highlights broader issues that need to be addressed for any mobile applications to be incorporated into the clinical setting. Only 3 of 11 applications reviewed had a measured optotype size within 10% of the necessary dimensions. There was no indication from reading the application descriptions that would accurately guide clinicians into the scientific validity of each application. This highlights a major barrier in order for such applications to be accepted into routine clinical practice. Potential methods to address this deficiency could include further studies of application validity, and involvement of advisory authorities to provide guidelines for medical professionals.³ Governance of application development is also critical. For example, prior to being available on the Apple ‘App Store' applications are reviewed by Apple staff to ensure the application satisfies basic functionality requirements, and does not harbor viruses.

Furthermore, the Apple iPhone was selected as the smartphone technology platform for this study as all models at the time of study had a uniform 3.5-inch diagonal screen size, and as such all Snellen applications would result in a similar optotype, no matter the model of the iPhone. Presently, the iPhone is the most commonly used smartphone amongst medical professionals,²¹ and as such it is likely that any smartphone visual acuity testing is likely to be done using an iPhone. One of the limitations of the 6SVA chart is the lack of standard illumination, however, when the iPhone is used for this task, one can guarantee that a standardised level of adequate illumination is provided.

We recognize that this study has certain limitations. First, visual acuities were only checked once on each patient, and as such test–retest variability (TRV) could not be evaluated. However, this study was primarily designed to establish whether the iSVA chart would give similar results to a 6SVA chart, and not to evaluate the TRV of the iSVA chart. This in itself would be an interesting area of study. Other studies have found overall the TRV of a Snellen chart to be approximately ±0.18 logMAR acuity.¹⁷

In addition, our study sample had relatively few (n=5) patients with low vision (worse than 6/18). We believe that this is representative of the visual acuities of general hospital patients, who are less likely to have severe visual impairment. It was also noted that the mean iSVA for this group was significantly lower than the 6SVA group, however, given the low number of patients in this group, inference of statistics is unreliable. We recognize that this would be an interesting area of future study, specifically investigating the value of smartphone-based visual acuity measures in those with low vision. This could be achieved by specifically targeting this group, or oversampling participants with low vision.

The impact of accommodation and testing distance must also be considered when interpreting the results measured from a hand-held (pocket) and iSVA chart. Most of the iSVA applications advised the chart to be held at a distance of 1.2 m away from the patient being tested. It is known that in most people with emmetropia where corrective error, accommodation takes begins at ~67 cm. However, patients who have certain refractive errors may need to accommodate in order to focus on the 1.2-m chart. This would disadvantage patients with presbyobia who are also hypermetropic, or have an unusual accommodative distance. Furthermore, when testing acuity at 6-m-small head movements toward the chart have a small impact on the angle subtended by the optotype. However, when the distance is reduced by using a smartphone to test acuity at a small distance, small head movements are expected to have a larger impact on the angle subtended by the optotypes. This observation may be more pronounced in patients with poor visual acuity, and may partially account for the larger discrepancies between the iSVA and 6SVA charts.

In our study, we chose to test the right eye of all patients for consistency, and this may have introduced selection bias, as the right eye is more frequently the dominant eye. This may affect testing if ocular dominance was an important factor in the reading of the chart. Also, in all instances, the Snellen visual acuity chart was first tested, followed by the iPhone Snellen acuity chart which may introduce a systematic bias from hidden factors such as anxiety in the patient during the first test, and becoming more accustomed to the testing process in the second instance. We would recommend future studies alternate or randomize which chart is shown first to eliminate such bias.

It is pertinent to acknowledge that Android and Windows Phone devices were not used for this study. Given the variability in the screen sizes and pixel densities of these devices, it would be extremely difficult for application developers to ensure accuracy between devices. As such, it is not possible to extrapolate findings of this study to these other devices. Future validation studies using other smartphones is therefore required to determine if other models can be of similar clinical use.

Conclusion

Through this study, we have identified smartphone applications that produce optically accurate Snellen visual acuity charts. Our data suggest that the overall statistical limits of agreement were greater than the cutoff required for clinical utility of the iSVA chart. Further validation is required for the assessment of acuity in patients with severe vision impairment. Given the growth of mobile technology in medical practice, we believe that all applications designed for clinical decision making must be rigorously validated before they are accepted into clinical practice.

CP and RC are chief editors of the Journal of Mobile Technology in Medicine. The remaining authors declare no conflict of intereset.