The Development and Validation of a Smartphone Visual Acuity Test (Peek Acuity) for Clinical Practice and Community-Based Fieldwork

Abstract

Importance

Visual acuity is the most frequently performed measure of visual function in clinical practice and the majority of people worldwide living with visual impairment are living in low and middle-income countries

Objective

To design and validate a smartphone-based visual acuity test that is not dependent on familiarity with symbols or letters commonly used in the English language.

Design

Validation study comparing results from smartphone Peek Acuity to Snellen Acuity (clinical normal) and the Early Treatment Diabetic Retinopathy Study (ETDRS) LogMAR chart (reference standard).

Setting

This study was nested within the six-year follow-up of the Nakuru Eye Disease Cohort in central Kenya.

Participants

Three hundred adults aged 55 years and above, recruited consecutively from the Nakuru Eye Disease Cohort Study..

Main Outcome(s) and Measure(s)

Outcome measures were monocular logarithm of the minimum angle of resolution (LogMAR) visual acuity scores for each test: ETDRS LogMAR, Snellen and Peek. Peek was compared, in terms of test-retest variability (TRV) and measurement time, with that of the Snellen and ETDRS LogMAR chart in participants’ homes and temporary clinic settings in rural Kenya in 2013/2014.

Results

The 95% confidence limits for TRV of smartphone acuity data were +/-0.033 LogMAR. The mean difference between smartphone and ETDRS and smartphone and Snellen acuity data was 0.07 (95%CI: 0.05-0.09) and 0.08 (95%CI: 0.06-0.10) LogMAR respectively indicating that smartphone acuities agreed well with those of the ETDRS chart and Snellen. The agreement of Peek and ETDRS was greater than Snellen with ETDRS, p=0.08 (95%CI 0.05 to 0.10). The local Kenyan community health care workers readily accepted the Peek Acuity smartphone test; it required minimal training and took no longer than Snellen; 77s vs. 82s (95%CI: 71 – 84s vs. 73 – 91s, p=0.13).

Conclusions

The study demonstrated that the Peek Acuity smartphone test is capable of accurate and repeatable acuity measurements consistent with published data on the TRV of acuities measured using five-letter-per-line retro-illuminated LogMAR charts.

Background

Visual acuity (VA) is the most frequently performed measure of visual function in clinical practice. VA measurements are used to establish the need for clinical investigation and quantify changes in central vision over time.

Four-percent of attenders to general practice in the UK do so with an eye problem¹ and a formal measure of VA should be part of each of these consultations.².

Globally, 285 million people have visual impairment, with 80% having diseases with known curative or preventative treatment. However, the majority live in low-income countries with minimal access to detection and subsequent treatment.³

The Snellen chart⁴ is the most common method for the measurement of VA in ophthalmic and general practice, but is limited by the non-geometric progression in letter sizing from line to line and the inconsistent number of letters per line.⁵ Different letters or optotypes (standardized symbols for testing vision) have varying legibility at the same size and secondary effects such as crowding are known to affect the ability of the patient to determine optotypes correctly and therefore could lead to measurement bias.

The limitations of the Snellen chart have largely been overcome with the development of LogMAR (Logarithm of the Minimum Angle of Resolution) acuity charts,⁶ which are now frequently employed in clinical research, such as the popular Early Treatment of Diabetic Retinopathy Study (ETDRS) charts. Despite this improvement, the Snellen chart remains the dominant method for acuity testing in clinical practice.⁷ This may be due to several factors including; familiarity, a well recognised scoring system, smaller chart size and the speed of performing the test relative to ETDRS.

Mobile phone technology has evolved rapidly in recent years. in 2013, an estimated 280 million (20%) of the 1.4 billion mobile phones sold were smartphones and that this proportion will increase, particularly in low-income settings.⁸ where fixed-line technology has been “leapfrogged” straight to mobile technology,⁹ providing the potential to access of health provision without the previously required infrastructure.¹⁰

The medical community is embracing mobile technologies with its potential in healthcare information delivery, real-time patient monitoring, research data collection and mobile telemedicine for the provision of expertise to remote locations.¹⁰

We hypothesise that a LogMAR-style smartphone vision test (Peek Acuity), with a fast-testing algorithm, would allow measurements to be made in a clinically acceptable period of time, with greater precision and reliability than is possible with Snellen charts. VA results can be displayed in familiar Snellen notation (imperial or metric) or LogMAR.

The Peek Acuity test was developed and compared, in terms of test-retest variability (TRV) and measurement time, with that of the Snellen and ETDRS-based tumbling E LogMAR chart (reference standard) in controlled and uncontrolled (“real world”) settings in rural Kenya.

Methods

Participants

This study was nested within the six-year follow-up of the Nakuru Eye Disease Cohort in central Kenya, a population-based study, which recruited 5000 individuals from 100 clusters in 2007, selected through probability proportionate to size of clusters, with individuals sampled within clusters through compact segment sampling.^11,12 Follow up of the participants was undertaken in 2013-2014.¹³ Three-hundred consecutive participants from the final twenty-one survey clusters who were undergoing reference measures of VA as part of the cohort follow-up were invited to enroll into this additional study of alternative VA measures. A temporary mobile eye clinic was set up in the centre of each cluster. All participants examined in the study were aged 55-years and above.

Ethics approval

The study adhered to the tenets of the Declaration of Helsinki, approved by the Ethics Committee of London School of Hygiene and Tropical Medicine and the African Medical and Research Foundation (AMREF), Kenya. Approval was sought from administrative heads in each cluster, usually the village chief who were given a copy of the consent form to read and pass on to those in the village.

Informed consent was obtained from all participants. The objectives of the study and examination process were explained in the local dialect, in the presence of a witness. All participants gave written (or thumbprint) consent to participate.

Peek Acuity Test

The Peek Acuity App was written in Android and for the purposes of this study used a Galaxy SIII GT-I9300 (Samsung C&T Corp., Seoul, Republic of Korea) running Android 4.0. The application was directly installed on to the test devices. Screen brightness was set to 100% within the app and all other options detailed below are built in.

Peek Acuity follows the standard ETDRS chart design with a 5x5 grid optotype letter “E” displayed in one of four orientations (90°,180°,270°and 0°). The participant points in the direction they perceive the “arms” of the E to be pointing and the tester uses the touch screen to swipe accordingly, translating the gestures from the patient. The tester is masked to the presented optotype and is unaware whether the participant is providing the correct response. This methodology reduces verbal or non-verbal clues which may bias the result. Single optotypes are shown to reduce confusion, however a bounding box is used to simulate the crowding effect of a standard ETDRS chart using a crowding bar, with thickness equivalent to the limb of the optotype, and spacing between optotype and crowding bar equal to that of half the total optotype size. This contour interaction format matches that employed by the reference standard ETDRS chart. A stair-casing algorithm is used to simulate clinical practice for time efficiency.

Peek Acuity offers standardized alternatives to “count fingers”, “hand movements” and “light perception. For “count fingers”, the app randomly presents between one and four bars and a correct or incorrect response is recorded on screen. For “hand movement”, a solid black box, half the width of the screen, moves backwards and forwards across the screen. For “perception of light”, Peek Acuity switches on the phone’s LED flashlight and the subject is asked to identify if and when they see the light come on and off, with the option to assess for perception of projection direction. Test completion is indicated by a sound and vibration alert.

VA results can be displayed in LogMAR, metric or imperial Snellen based on user preference. An additional option,“SightSim”, presents a live video feed with a Gaussian blur equivalent to the outcome of the vision test (eFigure 1) which is of value in sharing the information with those not familiar with acuity scoring.

Visual Acuity Measurement

Paired VA measures were made in both the participant’s home and in the central clinic on two consecutive days. For all tests the presenting acuity was measured, with habitual correction if worn. On Day-1 a health worker with basic eye-care training and a field-worker without formal healthcare training visited participants in their homes. The participants were tested using (a) Peek Acuity (LogMAR units) at two-meters and (b) a reduced three-meter “tumbling-E” Snellen chart (Sussex Vision) inside or close to the participant’s home (eFigure 2). The order of the test was determined randomly by coin toss. The detailed testing procedures are described in the Web Appendix.

On Day-2 the participants seen on Day-1 were re-assessed in the cluster’s central clinic. The same personnel re-tested the study participants using (a) Peek Acuity (LogMAR units) at two-meters and (b) a reduced three-meter “tumbling-E” Snellen chart, to allow for measures of TRV. The order of the test was determined randomly by coin toss. The ETDRS VA was measured using a back-illuminated four-meter ETDRS Chart (Precision Vision Inc) (eFigure 3) by an ophthalmic clinical-officer, which is the reference standard for this study. All testing (ETDRS, Snellen and Peek) at the different cluster clinic sites was standardized: conducted indoors, the test area was screened with “black-out” curtains and controlled ambient light levels within a range of 80-300lux (ISO-TECH: ILM1332A light meter), in accordance with British Standards for acuity assessment.¹⁴

Analysis

In total, eight comparisons of the various VA measures in the different settings were made (Figure 1).

Figure 1. Testing regime of Peek, Snellen and LogMAR in the participant’s homes and clinics.

For any pairwise comparison of methods, the TRV was estimated as 95% limits of agreement (mean paired difference between measures ±1.96SD). Histograms of the distribution of the test–retest and between-test method variability data suggested that the data were consistent with a normal distribution. Scatter plots of the observed TRV plotted against the average of the difference between the test and retest measurements suggested that there were no systematic association between TRV and the underlying bias relating to level of acuity. The Bland and Altman¹⁵ methods were therefore used for: (1) bias (mean and 95% confidence interval of the mean) between ETDRS (Reference test) and both Snellen and Peek Acuity scores; (2) TRV for the paired Snellen Acuity and Peek Acuity scores. Mean time scores between Snellen and Peek were compared using paired t-tests. Acuity scores were converted into a LogMAR for data analysis. eTable 1 outlines LogMAR scores used including where acuity was too poor to measure with optoytypes).¹⁶

Results

The Peek Acuity study took place between December 2013 and March 2014. Of the 300 participants selected, 293 enrolled (98%; 135 male, 158 female). In total, 272 people (91%; mean age 65 years, range 55 to 97) were examined and completed all three tests in the central clinic on Day-2. Of these, 233 (86%) were available and had also taken both VA tests at home on Day-1.

The median VA measured by ETDRS for all eyes tested (all levels of vision including those unable to read the ETDRS chart) was 0.23 LogMAR with a range of -0.2 to 4.0 LogMAR (Snellen equivalents: median 20/32, range 20/12.5 to NPL).

The results of the eight pair-wise comparisons of the right eye VA described above are presented in Table 1 and Figure 2 with results for the left eye available in the web appendix (no difference between right and left eyes was found, eTable 2). The comparisons of clinic based Snellen and clinic based Peek Acuity measures with EDTRS under the standardized clinic conditions indicates that Snellen tests show a high degree of correlation with ETDRS, but that this is higher still with Peek Acuity, p=0.08 (95%CI 0.05 to 0.10). The mean difference between the Peek Acuity in the clinic and EDTRS was 0.011 LogMAR units (95% CI: -0.014 to 0.035) and 0.032 LogMAR units (95% CI: 0.010 to 0.054) for the right and left eye, respectively. Equivalent to less than three letters on a line difference when taking the upper confidence limit of the mean difference. The correlation (scatter) plots and Bland-Altman difference plots for these comparisons in the right eye are shown in Figure 3a.

Table 1.

The results of the eight pair-wise comparisons of the right eye showing Bland-Altman and Pearson Correlation analysis

Comparison	N	Description	Difference of Average	95% Confidence Interval Mean Difference		95% Limits of Agreement		Pearson correlation coefficient (95%CI)
				Lower	Upper	Lower	Upper
1	272	ETDRS vs. Snellen in Clinic	0.088	0.063	0.114	-0.329	0.506	0.932 (0.914-0.946)
2	272	ETDRS vs. Peek in Clinic	0.011	-0.014	0.035	-0.396	0.417	0.936 (0.919-0.949)
3	272	Snellen Clinic vs. Peek Clinic	-0.078	-0.100	-0.056	-0.439	0.283	0.950 (0.937-0.960)
4	233	Peek Home vs. Snellen Home	0.029	-0.007	0.065	-0.517	0.575	0.902 (0.875-0.923)
5	233	ETDRS vs. Snellen at home	0.084	0.043	0.125	-0.541	0.709	0.865 (0.828-0.894)
6	233	Snellen Clinic vs. Snellen Home	-0.004	-0.038	0.030	-0.523	0.515	0.907 (0.881-0.927)
7	233	Peek Home vs. Peek Clinic	-0.054	-0.083	-0.025	-0.498	0.390	0.933 (0.914-0.948)
8	233	ETDRS vs. Peek at Home	0.055	0.023	0.088	-0.438	0.549	0.917 (0.893-0.935)

Figure 2. Graph showing eight outcomes (right eye) with difference of the average in LogMAR on the y-axis and comparisons on the x-axis.

Figure 3. Scatter plots and Bland and Altman plots for outcomes 2, 8 and 7 for the right eye.

Comparing Peek Acuity tested at home to ETDRS in the clinic, the mean difference between the Peek Acuity at home and EDTRS was 0.055 LogMAR (95%CI: 0.023-0.088) and 0.072 LogMAR (95%CI: 0.039-0.105) for the right and left eye, respectively, which is equivalent to five letters or one line of difference. (Table 1, Figure 3b).

The Peek Acuity TRV (7), performed by the same examiner, on Day-1 at home and on Day-2 in the clinic, had a high correlation and a small difference of averages (Table 1, Figure 3c). ¹⁷

Mean testing time for both eyes on 126 study participants in whom testing time was measured was 82seconds (95%CI:73–91seconds) with Snellen and 77seconds (95%CI: 71–84seconds) with Peek respectively, showing no difference (p=0.13).

Peek used at home by a community healthcare worker is 85% sensitive and 98% specific (eTable 3) at detecting eyes with severe visual impairment (deemed locally as the surgical cut-off point for “operable cataract”, Snellen equivalent of ≤6/60) when compared to ETDRS in controlled conditions.

No adverse events from performing any of the acuity tests were reported.

Discussion

The ubiquity of smartphones amongst healthcare professionals¹⁸ and increasing penetration, particularly in low and middle-income countries, provides potential for delivering high-quality, objective, repeatable and acceptable vision testing throughout the world.

With the majority of the world’s blind people living in low-income countries, the need for tools to increase early detection and appropriate referral are vital if the prevalence of blindness and visual impairment is to be reduced. In high-income settings, where primary care consultations are time pressured and confidence in diagnosing ophthalmic problems is low¹⁹, accessible tools to provide reliable measures to guide management are vital. The referral of patients with ophthalmic complaints from primary care, such as in general practice or accident and emergency, to specialist care, should include a measure of acuity that is reliable and accessible and further testing in these contexts is encouraged.

In this study we aimed to develop and validate a smartphone based VA test appropriate for use in challenging circumstances such as rural Africa as well as being reliable enough for use in routine clinical practice in well-established health systems. Overall Peek Acuity performed well, the testing time was no slower or less repeatable than Snellen whilst being comparable in accuracy to ETDRS. For clinical and population screening use, the TRV of acuity should be consistent across the acuity range and measurable in terms of lines or letters of change, measurement error obscures true clinical change and reduces the statistical power of clinical trials using acuity as a primary outcome measure,²⁰ Peek acuity testing proved to be repeatable and consistent. Our findings also indicate that the reduced Snellen chart is a repeatable and time efficient VA test that still has application in clinical and field settings.

In our study, the TRV of Snellen was higher than in comparable studies,^5,21 this may have been due to tightly defined end-points (no part scores were given for part completion of a line).

Although multiple applications for the testing of VA on smartphones are available, the majority have not been tested for repeatability or reliability against a reference standard.²² This study found Peek Acuity to be comparable with ETDRS style chart, with similar TRV to that previously reported for other tests.^23,24

Low Vision

Low vision in subjects whom have VA below the level that can be measured on a chart are subject to assessment of vision that lacks a standardized approach and is open to considerable variability. In standard practice, if no optotypes are visible at the reduced distance, “counting fingers” is performed, followed by “hand movements” and finally differentiating between “perception of light” and “no perception of light”. In practice this crucial measure of vision that may differentiate poor and good prognosis for treatment is often overlooked due to these non-standardized measures. Peek offers a standardized approach to testing such low levels of vision which could be also performed on a tablet but was not assessed formally in this study.

Limitations

The study population comprised older-aged Kenyan adults, who may not be representative of other populations and age groups, limiting the generalizability. Other studies are ongoing to determine the suitability of this tool in different contexts, across a range of different handsets and operating systems (this study only assessed the device on multiple handsets of the same phone model and operating system), including a school-aged population. Reflection from smartphone screens due to bright sunlight can be problematic, though antiglare screens have been shown to reduce this limitation on other platforms.²⁵ Smartphones are on the whole more expensive than a basic Snellen chart but less expensive than a retro-illuminated LogMAR or Snellen chart. With the increased availability of low-cost smartphones and tablets many health workers may already own a device suitable for downloading multiple apps. ²⁶

Concerns exist about data sharing and misuse with mobile Health platforms, which should be integrated with systems compliant with approved standards for data sharing.

Due to the size, weight and power requirements it was not possible to perform ETDRS in participants homes and therefore TRV of ETDRS was not assessed as with Snellen and Peek. We were therefore unable to assess ETDRS TRV in this environment.

Non-healthcare workers who received specific training in how to use Peek Acuity performed testing, further investigation of Peek Acuity’s usability with only in-built instructions is required.

Testing Distance

During the early development phase, Peek Acuity was performed at three-meters. However, in the study setting, it was often not possible to find an indoor space of three-meters to conveniently perform the test. In conditions where the ambient light measure on the phone was greater than 1000lux, measures of Peek Acuity did not correlate well with the reference standard. With a 4.8-inch screen, 720x1280 pixels and a viewing distance of two-meters it is possible to measure acuity of 1.0 LogMAR and 1.3 LogMAR when the testing distance is reduced to one meter. Therefore, the testing distance and software algorithm were changed to two-meters. Following this change over ninety percent of participants were tested indoors in their homes. The smartphone’s inbuilt ambient light detector (which was accessed in the Peek Acuity app to give a mean Lux reading per visual acuity test) provides a warning that test conditions are not suitable if >1000lux is detected.

Implications

The more widespread testing of VA in low and middle-income countries is likely to lead to greater awareness of treatable eye disease with an increased uptake of preventative and curative treatments. In non-ophthalmic departments, an easily accessible, easy to use, accurate and reliable vision test could lead to increased assessment of vision testing in routine practice. ²⁷

Conclusion

Additional applications to assess visual function and imaging of the eye make smartphones an attractive option for delivering ophthalmic assessment.^28,29 In settings where ophthalmic instrumentation or ophthalmic trained personnel are limited, the ability to reliably measure a change in vision, or detect abnormal vision, automation of stair-casing, masking of presented information, and generation of a jargon-free result greatly improve efficacy in the hands of minimally trained personnel. With the inherent connectivity and global positioning system (GPS) features of the device may ultimately lead to more people receiving timely and appropriate treatment.

Table 2. Key attributes and potential benefits of Peek Acuity.

Key Attributes	Potential Benefits
Use of “E” optotype widens accessibility to those unable to read letters	Increased objectivity of test
Use of “E” optotype rather than letters ensures acuity is resolution based rather than recognition
Random optotype direction prevents learning effect from one eye to the other
Automated visual acuity score calculation
End of test indicator (vibration and sound alert
Gesture based recording of responses making the test more objective by swiping in the direction indicated while not seeing the letter and shake to record not-seen
Standardized low vision measurement tools for counts fingers, hand movements and perception of light	Standardized testing and prompts for control of conditions
Ambient light sensor used for adjusting screen brightness and to detect thresholds ambient light levels above which acuity measurements decrease in accuracy
Use of ETDRS based optotype with result available in all the standard units: Decimal, LogMAR, Metric Snellen and Imperial Snellen	Easy interpretation of the results
Live video feed demonstrating appropriate level of Gaussian blur according to outcome of the vision test (eFigure 1) which is of value in sharing the information with those not familiar with acuity scoring
Downloadable from the Google Play Store	Accessible and validated
CE Marked (Class I)
Smartphone based	Potential to store data to an electronic patient record (EPR), increasing efficiency of data management and limiting potential recording error
	Data can be shared remotely with other healthcare providers for feedback
	The EPR can be geo-tagged which is of particular value in resource-limited settings where addresses may not be available and patient follow-up is challenging