Validation of Home Visual Acuity Tests for Telehealth in the COVID-19 Era

This comparative effectiveness research study compares at-home visual acuity tests with in-office acuity measurements to validate their use with telehealth appointments.

Key Points

Question

Are at-home tests valid for measurement of visual acuity during the COVID-19 pandemic?

Findings

In this randomized, comparative effectiveness research study, at-home acuity tests were performed by participants in their own homes and results were compared with standard-of-care visual acuity measurements during a clinic visit. Compared with in-office acuity measurements, all 3 at-home tests were within 1 line of Snellen acuity.

Meaning

In this study, 3 at-home visual acuity tests were validated through comparison with in-office visual acuity measurements, supporting their potential use in teleophthalmology care.

Abstract

Importance

Visual acuity (VA) is one of the most important clinical data points in ophthalmology. However, few options for validated at-home VA assessments are currently available.

Objective

To validate 3 at-home visual acuity tests in comparison with in-office visual acuity.

Design, Setting, and Participants

Between July 2020 and April 2021, eligible participants with VA of 20/200 or better were recruited from 4 university-based ophthalmology clinics (comprehensive, cornea, glaucoma, and retina clinics). Participants were prospectively randomized to self-administer 2 of 3 at-home VA tests (printed chart, mobile phone app, and website) within 3 days before their standard-of-care clinic visit. Participants completed a survey assessing usability of the at-home tests. At the clinic visit, best-corrected Snellen distance acuity was measured as the reference standard.

Main Outcomes and Measures

The at-home VA test results were compared with the in-office VA test results using paired and unpaired t tests, Pearson correlation coefficients, analysis of variance, χ² tests, and Cohen κ agreement. The sensitivity, specificity, positive predictive value, and negative predictive value of each at-home test were calculated to detect significant VA changes (≥0.2 logMAR) from the in-office baseline.

Results

A total of 121 participants with a mean (SD) age of 63.8 (13.0) years completed the study. The mean in-office VA was 0.11 logMAR (Snellen equivalent 20/25) with similar numbers of participants from the 4 clinics. Mean difference (logMAR) between the at-home test and in-office acuity was −0.07 (95% CI, −0.10 to −0.04) for the printed chart, −0.12 (95% CI, −0.15 to −0.09) for the mobile phone app, and −0.13 (95% CI, −0.16 to −0.10) for the website test. The Pearson correlation coefficient for the printed chart was 0.72 (95% CI, 0.62-0.79), mobile phone app was 0.58 (95% CI, 0.46-0.69), and website test was 0.64 (95% CI, 0.53-0.73).

Conclusions and Relevance

The 3 at-home VA test results (printed chart, mobile phone app, and website) appeared comparable within 1 line to in-office VA measurements. Older participants were more likely to have limited access to digital tools. Further development and validation of at-home VA testing modalities is needed with the expansion of teleophthalmology care.

Introduction

During the initial portion of the COVID-19 pandemic, eye care professionals needed to reduce the risk of exposure to patients while continuing to provide high-quality health care. The Casey Eye Institute at Oregon Health and Science University saw a decrease in clinic visits to 25% of prepandemic levels in April 2020 and did not return to baseline until March 2021.¹ With the need to limit clinic visits and an increase in the Centers for Medicare & Medicaid reimbursement for telehealth services,² teleophthalmology has played a critical role in responding to the COVID-19 pandemic.^1,3,4 Although published data are limited regarding teleophthalmology during the pandemic, a study looking at ophthalmology visits insured by Blue Cross Blue Shield of Michigan found a 4-fold increase in virtual visits compared with baseline data from the first 3 months of the pandemic.⁵ A study at Casey Eye Institute demonstrated positive clinician attitudes with telehealth implementation during the pandemic¹; however, access to an easy-to-use, validated at-home visual acuity (VA) test for adults limited telehealth care.

With increased use of telehealth technology, it is crucial to ensure that quality of care remains optimal. A key factor in evaluating and treating patients during eye care visits is the VA measurement. During a clinical encounter, this measurement is taken at specific distances using standardized charts, including Snellen, Early Treatment Diabetic Retinopathy Study (ETDRS), HOTV, and Tumbling E charts.⁶ Historically, VA testing has been done exclusively in a clinical setting under the guidance of a trained professional, but in recent years, various options have been created for testing outside of the clinic, including paper charts,^7,8,9,10 web-based tests,^11,12,13,14 and smartphone applications.^{15,16,17,18,19,20,21,22,23,24}

Many commercially available VA tests are not validated before public release, and many applications require users to pay a fee, making clinical application unrealistic for some patients.²⁵ A 2015 review of 11 Snellen chart iPhone applications found significant variability in application accuracy and validation methods.¹⁷ Some tests, while finding a positive correlation between acuity by smartphone apps and standard in-office testing, found discrepancies in the reported measurements themselves.^22,26 Results have also varied depending on a participant’s age and level of visual impairment.²⁶ However, there is a growing body of research examining new methods of VA measurement, such as the Peek Acuity app, which showed a mean difference of −0.07 logMAR, or less than 1 line of VA, between the app test and ETDRS in-office chart with a test-retest variability of 0.033 logMAR.¹⁶

A significant barrier to implementation of these new VA tests is translation to at-home use, as many of the studies were performed in the clinical or research setting^11,17,18 and used trained staff to administer the tests.^{7,12,15,16,19,20,21,22,26} These study designs pose significant constraints on the ability to extrapolate the findings to the real-world conditions of self-administered VA testing at home. A few recent studies have validated paper chart acuity tests^8,9,10 and a web-based test¹⁴ under home conditions. In a study by Crossland et al,⁸ the Home Acuity Test, a printed chart, had a mean difference of −0.10 logMAR compared with previous in-office ETDRS acuity. Two recent studies demonstrated at-home VA within 1 line of in-office acuity with the University of Arizona paper chart.^9,10

Beyond the few self-administered studies discussed above, there are few data about the accuracy and usability of different types of self-administered VA tests validated under home conditions. Our study aims to fill that knowledge gap because validation of at-home VA tests remains necessary to deliver reliable teleophthalmology care during the ongoing COVID-19 pandemic. This report compares 3 types of at-home VA measurement to standard in-office VA testing and seeks to validate the accuracy and usability of these tools.

Methods

Study Participants

Following approval by the Oregon Health & Science University institutional review board, eligible participants were identified from 4 university-based ophthalmology clinics (comprehensive, cornea, glaucoma, and retina) with the following inclusion criteria: age 18 years or older, VA of 20/200 or better in at least 1 eye, and access to an iOS mobile device or a computer with internet access or ability to provide a physical address for a mailed printed eye chart. Participants were excluded if they had ocular surgery within the past 3 months or if they reported a subjective change in VA between the at-home test and their clinic visit. Participants receiving intravitreal injections were not excluded. Between July 17, 2020, and March 9, 2021, 246 participants were enrolled. Eligible participants completed informed consent by telephone to limit COVID-19 exposure and were not offered compensation or incentives to participate. This study followed the International Society for Pharmacoeconomics and Outcomes Research (ISPOR) reporting guideline.

Participants with internet access or an iOS device were randomized to receive 2 of 3 at-home VA tests described below to form the randomized cohort (eFigure 1 in the Supplement). Participants without access to a computer with internet or an iOS device were assigned to a mail-only cohort and were mailed a printed University of Arizona chart (eFigure 2 in the Supplement).

Instructions for each test were provided by telephone and via email or mail, and the total training time was recorded (eTable 1 in the Supplement). Participants were asked to perform the at-home tests within 3 days before their standard-of-care in-office appointment. Participant instructions regarding screen brightness, glasses use, and testing distance are detailed in eFigure 2 in the Supplement.

Participants were asked to record the VA of the eligible eyes after completing the assigned tests and to return the results during their clinic appointment or via email. Participants were also asked to complete a feedback survey at the end of each test. This survey was not validated prior to its use in the study (eFigure 2 in the Supplement).

University of Arizona/Banner Eye Health Chart

This publicly accessible distance VA test used an ETDRS chart that could be printed or viewed on a computer screen. It had a picture of a quarter-sized circle for participants to use for correct scaling.

Verana Vision Test Mobile Phone App

This adaptable near VA test, available as a free iOS application, instructed participants to hold the device at a “comfortable and consistent reading distance.” It displayed 1 letter at a time and prompted participants to select the correct letter from 6 options before proceeding to the next letter.

Farsight.care Website

Farsight.care was a near VA test on a publicly accessible web page. It used a number chart and included a line for participants to measure against a credit card for correct scaling on their screen.

In-Office Visual Acuity Measurement

At the standard-of-care in-office clinic visit, distance VA with habitual correction was measured by an ophthalmic technician using an electronic Snellen eye chart (Smart System 2; M&S Technologies, Inc).

Charts were reviewed to identify an in-office baseline VA for each participant from the most recent clinic visit within 12 months prior to the clinic visit.

Statistical Analysis

Analysis was performed between March and July 2021. Visual acuity measurements were converted to logMAR. Descriptive analyses included the generation of mean (SD) for each variable of interest. Paired t tests were performed to compare mean acuity of the at-home tests with in-office and baseline acuities. The resultant P values were 2-sided and not adjusted for multiple analyses. The sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV) were calculated for each test. All analyses were conducted using Excel version 16.16.25 (Microsoft Corporation) and R version 4.1.0 (R Core Team).

Results

Randomized Cohort

Of 218 randomized participants, 112 (51.3%) completed 2 of the 3 at-home tests and submitted results. The mean (SD) age of those completing the tests was 63.8 (13.0) years (range, 18-78). There was a relatively equal number of participants from the 4 clinics and no difference in rates of participation (Table 1). There were no obvious differences between the participants who completed the tests vs those who did not based on age, sex, clinic, zip code, or training time (eTable 1 in the Supplement).

Table 1. Description of Study Participants.

	All participants	Comprehensive	Cornea	Glaucoma	Retina
No. of participants completing at-home test	121	31	29	34	27
No. of eyes	233	61	53	65	54
Age, mean (SD), y	63.8 (13.0)	62.7 (12.1)	59.5 (14.7)	67.6 (13.7)	64.9 (12.1)
Clinic acuity logMAR, mean (SD)	0.11 (0.17)	0.09 (0.19)	0.14 (0.18)	0.12 (0.20)	0.12 (0.16)
Snellen equivalent, No. of eyes
20/20-20/40	205	55	47	55	48
20/50-20/80	23	4	6	7	6
20/100-20/200	5	2	0	3	0

Mean in-office acuity for randomized participants completing the at-home tests was 0.11 logMAR, or slightly less than 20/25 (Table 1). There was no difference in mean in-office VA among the 4 clinics (range, 0.09-0.14 logMAR; 20/25 to 20/28 Snellen equivalent; P = .66) (eTable 2 in the Supplement) or among the participants taking each of the 3 tests (range, 0.10-0.14 logMAR; 20/25 to 20/28 Snellen equivalent; P = .59) (eTable 3 in the Supplement). Mean in-office acuity of all participants was not significantly different compared with baseline acuity (0.11 vs 0.10 logMAR; 20/26 vs 20/25 Snellen equivalent; P = .62).

To evaluate the validity of each at-home test, the at-home VA was compared with in-office VA. The mean difference of at-home compared with in-office VA was smallest with the printed chart (−0.07 logMAR, or <1 line of difference; 95% CI, −0.10 to −0.04) (Table 2). The mobile phone app and website test compared with the in-office test had a mean difference of −0.12 (95% CI, −0.15 to −0.09) and −0.13 logMAR (95% CI, −0.16 to −0.10), respectively. The Pearson correlation coefficient was greatest for the printed chart (0.72; 95% CI, 0.62-0.79), consistent with a high degree of correlation. The mobile phone app had a Pearson correlation coefficient of 0.58 (95% CI, 0.46-0.69) and the website test had a coefficient of 0.64 (95% CI, 0.53-0.73).

Table 2. Association Between Home Tests and Clinic Acuity^a.

	Printed chart	Mobile phone app	Website test	Mail-only cohort
No. of eyes tested	137	147	146	18
Mean difference in VA, logMAR (95% CI)	−0.07 (−0.10 to −0.04)	−0.12 (−0.15 to −0.09)	−0.13 (−0.16 to −0.10)	−0.09 (−0.15 to −0.03)
Limit of agreement (95% CI)
Lower	−0.39 (−0.44 to −0.34)	−0.50 (−0.55 to −0.44)	−0.53 (−0.58 to −0.46)	−0.32 (−0.42 to −0.22)
Upper	0.25 (0.20 to 0.30)	0.26 (0.21 to 0.32)	0.27 (0.21 to 0.33)	0.14 (0.04 to 0.25)
Pearson correlation coefficient (95% CI)	0.72 (0.62 to 0.79)	0.58 (0.46 to 0.69)	0.64 (0.53 to 0.73)	0.873 (0.69 to 0.95)
Cohen κ values
Unweighted	0.17 (0.09 to 0.26)	0.05 (−0.27 to 0.12)	−0.001 (−0.07 to 0.07)	−0.03 (−0.16 to 0.08)
Weighted	0.49 (0.40 to 0.59)	0.30 (0.19 to 0.41)	0.32 (0.22 to 0.43)	0.39 (0.14 to 0.63)

Abbreviation: VA, visual acuity.

^a All at-home test results were compared with Snellen distance acuity measurements from a clinic visit as the reference standard.

Weighted Cohen κ values, an assessment of relative agreement taking closeness of measurements into consideration, ranged from 0.30 to 0.49 (weak agreement) (Table 2). Bland Altman plots were created to compare the difference of the mean acuities and display the upper and lower limits of agreement (LOA) as well as the 95% CI for the LOA (Figure). The Bland Altman plots showed similar upper LOA for the at-home tests; however, the mobile phone app and website test had greater lower LOA and more instances in which the at-home test underestimated the in-clinic acuity.

Figure. Bland Altman Plots Comparing Differences in Mean Visual Acuity Between In-Office Measurements and Results From Each of the 3 At-home Tests.

Plots showed similar upper limit of agreement (LOA) for the at-home tests, but the mobile phone app and website test had greater lower LOA and more instances in which the at-home test underestimated the in-office acuity.

Although there was no difference between mean in-office acuity and mean baseline acuity (0.11 vs 0.10 logMAR; 20/26 vs 20/25 Snellen equivalent; P = .62), we sought to characterize whether the at-home tests could detect a clinically significant change in acuity for an individual participant. A clinically significant change was defined as 0.2 logMAR or greater (corresponding to ≥2 lines of acuity). For individual participants with a change of 0.2 logMAR or greater between baseline acuity measurement and the in-clinic measurement, the at-home acuity measurements were used to calculate sensitivity, specificity, PPV, and NPV for each of the at-home tests compared with in-clinic acuity (Table 3). The sensitivity of the tests ranged from 71.4% to 81.8%, specificity ranged from 67.2% to 75.0%, PPV ranged from 18.9% to 25.6% and NPV ranged from 94.1% to 97.9%. None of these metrics demonstrated a statistically significant difference.

Table 3. Ability of Home Tests to Detect 2 or More Lines of Acuity Change From Baseline.

	% (95% CI)
	Printed chart	Mobile phone app	Website test
Sensitivity	73.3 (50.9-95.7)	71.4 (47.8-95.1)	81.8 (59.0-100)
Specificity	70.8 (63.4-80.0)	67.2 (59.1-75.2)	75.0 (67.4-82.6)
Positive predictive value	25.6 (12.5-38.6)	18.9 (8.28-29.4)	22.5 (9.56-35.4)
Negative predictive power	94.1 (90.8-99.8)	95.7 (91.5-99.8)	97.9 (95.0-100)

Participants were asked to complete a 4-question survey for each of the tests completed at home (eFigure 2 in the Supplement). Overall, participants agreed that the tests were easy to view and understand with scores of 4 points or higher (scale: 1, disagree, to 5, agree). Regarding confidence in the results, participants were neutral or agreed (score range, 3.1-3.6). Participants were neutral in terms of interest in continuing with at-home acuity testing (score range, 2.9-3.4) (Table 4).

Table 4. Participant Survey Responses and Rate of Participation.

	Mean (95% CI)			P valuea
	Printed chart	Mobile phone app	Website test
Survey questionsb
“I was able to easily view/print the [home test]”	4.5 (4.3-4.7)	4.6 (4.4-4.8)	4.1 (3.7-4.4)	.15-.44
“I could understand the instructions for the [home test]”	4.4 (4.2-4.6)	4.1 (3.8-4.4)	4.0 (3.7-4.3)	.51-.92
“I am confident in the visual acuity results I got at home with the [home test]”	3.5 (3.2-3.8)	3.6 (3.3-3.9)	3.1 (2.8-3.4)	.05-.43
“I would like to continue to be able to conduct visual acuity testing at home with the [home test]”	3.1 (2.8-3.4)	3.4 (3.1-3.7)	2.9 (2.6-3.2)	.05-.29
Rate of participation after enrollment, No./total No. (%)	67/148 (45.3)	74/143 (51.7)	73/145 (50.3)

^a P values calculated by paired t test for individual comparisons of each test with the others.

^b Numeric response key: 5 indicates strongly agree; 4, agree; 3, neutral; 2, disagree; and 1, strongly disagree.

Mail-Only Cohort

Of the 28 participants enrolled in the mail-only cohort, 9 participants (32.1%) completed and submitted results, demonstrating a strong trend toward a lower rate of participation compared with the randomized cohort (range, 45.3%-51.7%; P > .05) (Table 4). Older participants were significantly more likely to be enrolled in the mail-only cohort compared with the randomized cohort (mean [SD] age, 75.0 [7.0] vs 63.8 [13.0] years; P = .02). However, participants in the mail-only cohort exhibited more positive responses on the survey (mean score, 3.7-4.6), with greater confidence in the test results compared with the website test and printed chart in the randomized cohort (4.4 vs 3.1; P = .007; and 4.4 vs 3.5; P = .04; respectively).

Discussion

Three types of at-home VA tests were compared in this university-based study completed during the COVID-19 pandemic. Because we wanted to avoid accessibility bias, we chose 3 tests that were readily available to the public at no cost. The tests differed in access method (printed, mobile phone app, or website), and the mobile phone app and website tests were near acuity tests while the printed chart measured distance acuity. This diverse group of tests was purposefully selected to expand the types of VA tests validated for self-administration under home conditions.

There were 4 key findings in this study. First, all 3 at-home self-administered VA tests were valid within 1 line of in-office Snellen acuity. These results were similar to other smartphone-based tests, including Peek Acuity, with a mean difference of −0.07 logMAR between the in-office ETDRS acuity and the Peek Acuity app administered in the clinic with staff guidance.¹⁶ The Home Acuity Test, an open-source VA screening test that was validated under home conditions, also performed similarly with a mean difference of −0.10 logMAR compared with the ETDRS logMAR chart in the clinic.⁸

Second, the printed chart had the smallest mean difference and greatest correlation compared with the in-office acuity measurement, although there was no statistically significant difference among the 3 tests. In this study, the mean difference between the University of Arizona printed chart and Snellen acuity was −0.07 logMAR, which was similar to the findings in Chen et al⁹ (mean difference of −0.02 logMAR) and Siktberg et al (mean adjusted VA letter difference of 4.1 letters).¹⁰ This study is one of the first to test a mobile phone app and web-based test under home conditions.

Third, older participants in the study were more likely to be enrolled in the mail-only cohort compared with the randomized cohort, suggesting older participants may not have had adequate access to internet or iOS devices to qualify for randomization in this study. As many older patients may benefit from at-home acuity testing, access to technology for this group appeared to be a barrier for some patients.

The final key finding was that participants found all 3 tests easy to use and expressed some interest in future at-home testing. Feedback from participants using all forms of at-home testing indicated that participants did not want their in-office acuity testing replaced by at-home acuity measurements because of concerns about accuracy.

Based on these data, it appeared that at-home VA tests may be best used to reassure patients and clinicians when there is no significant change in VA. However, given the lower PPV values, the at-home tests may not be as useful for identifying true change. For example, if the difference between the at-home test result and a previous clinic-based acuity is 0.2 logMAR or greater, it is more likely to be a false positive. Our intention for participants recruited for this study was that they had stable VA overall (excluding participants who had recent surgeries or procedures and who reported acute changes in VA between at-home and in-office testing); a higher NPV along with lower PPV is consistent with a population that was not expected to have significant changes in VA. Individual patient characteristics, such as age, ocular diagnosis, and time interval between visits, as well as acuity testing habits, such as history of reliability in the clinic, may need to be considered for future use of at-home tests.

Cohen κ agreement values demonstrating the greatest amount of agreement occurred for participants with 20/20 VA; those with poorer VA were less likely to have values that agreed, consistent with other studies.^9,26 It appeared that home acuity tests were more likely to report slightly decreased VA compared with clinical acuities, possibly due to the testing modality itself, including contrast and brightness, and the means of administering the test, including lack of technician oversight and possible distractions at home. Notably, this trend was seen in other studies in which a VA test was self-administered at home.^9,10,14,16 Overall, it seems most beneficial to have an at-home acuity test slightly underestimate rather than overestimate acuity for purposes of remotely detecting VA changes.

Limitations and Strengths

Limited availability of technology prevented some participants from being randomized in the study. However, the mail-only cohort, who received copies of the printed chart, had similar performance on the at-home test compared with in-office acuity with a mean difference of −0.09 logMAR (Table 2). Therefore, while access to technology was a barrier for some, there was evidence that providing testing materials by mail may overcome this barrier. Another technological limitation was that Android mobile phone users were unable to be randomized, but they were offered participation in the mail-only cohort. We were unable to identify a free, publicly available mobile phone app usable with both Android and iOS at the time of recruitment.

To facilitate timely implementation of this study during the COVID-19 pandemic, we chose 3 at-home tests that were publicly available at the start of enrollment in July 2020. These tests were only applicable to patients with VA of at least 20/200 because this was the largest optotype common to all 3 tests. This study did not include multiple measurements of VA using the same home test over time. Ideally, future longitudinal studies could examine the repeatability of home acuity measurements and their ability to detect clinically meaningful changes in VA.

Some aspects of at-home testing may have increased variability in the measurement of VA, including use of different device and screen types, resulting in variability of screen size and resolution. No data were collected on testing time for the at-home tests to minimize testing burden and therefore increase participant retention and limit bias related to study dropout. These study design choices were made to approximate real-world at-home testing conditions and maximize generalizability.

Conclusions

This study validated 3 publicly available, free, at-home VA tests that were self-administered under home conditions. Participants were of various ages from multiple clinics and represented urban and rural populations, which increases generalizability of the study findings.

The COVID-19 pandemic has created an opportunity for expansion of teleophthalmology services due to the necessity of limiting in-person exposures. The lessons learned in this context may have important implications for rural eye care, as well as eye care in other outreach settings or underresourced areas. Visual acuity remains a key clinical data point for eye care professionals when making diagnostic and treatment decisions. Validated at-home tests provide an important first step in the expansion of teleophthalmology.

Supplement.

eFigure 1. Enrollment and randomization flowchart

eFigure 2. At-home visual acuity tests, instructions, and survey for participants

eTable 1. Demographics of participants completing at-home tests

eTable 2. Comparison of mean visual acuity by clinic

eTable 3. Comparison of mean visual acuity of participants using each test