Validation of Electronic Visual Acuity (EVA) Measurement Against Standardised ETDRS Charts In Patients With Visual Field Loss From Inherited Retinal Degenerations

Abstract

Background

With the increase in clinical trials testing therapy for retinal disease there is a need to ensure that outcome measures are both accurate and standardized. The US Food and Drug Administration favours the use of visual acuity measured using ETDRS logMAR charts. The loss of visual field can interfere with visual tracking across the charts, leading to increased variability of measurements. Electronic visual acuity (EVA) presents the optotype on the centre of a screen, thereby removing the tracking element of the task, and may provide a more precise measurement.

Methods

Visual acuity was measured twice using ETDRS charts, EVA automated single letter (E-ETDRS) and EVA single line (EVA-SL) presentation (EMMES, USA). Patients underwent microperimetry (MAIA, Centervue, Italy) to determine visual field. We tested 65 patients with rod-cone dystrophies and 41 healthy volunteers.

Results

Both participant groups read 2-3 letters more on average on the electronic charts compared to ETDRS. Limits of agreement using a modified Bland-Altman analysis account for replicates were wider in eyes with foveal defects (-9 to 18) compared to eyes without foveal defects (-11 to 15). Electronic charts in the presence of foveal defects reduced the range (-11 to 13).

Conclusion

Electronic visual acuity may provide more accurate measures of visual acuity than traditional ETDRS charts in patients when the visual field loss encroached on the central vision. Electronic presentation with a single line of letters was the favoured style reported by patients and should be considered in future interventional clinical trials.

Introduction

Visual acuity (VA) is a key part of any ophthalmic assessment in both clinical practice and clinical research. It is a reflection of the ability to see fine detail and, as a basic measure, is well understood by clinicians. Measured in minutes of arc, VA is calculated as the reciprocal of the minimum angle of resolution (MAR), and is most commonly assessed with logMAR charts. Despite the popularity of these charts, clinical change can only be detected when VA alters more than 0.2 logMAR (2 lines on the chart), due to the test-retest variability.[1] Moreover, the variability in low vision patients is even greater.[2,3] New strategies are therefore required to reduce the variability in the low vision population to better quantify therapeutic effects.

When novel interventions are trialled for ophthalmic indications, VA is often defined as a key outcome measure.[4] It is a well-accepted measure by the Food and Drug Administration (FDA) and other regulatory authorities.[5] Standardised charts are highly regulated with control over the font type, spacing across and between lines, size progression and legibility of letters used.[6] Any new presentation or scoring methods must therefore meet the scientific rigours of the established ETDRS charts. New tests must also be validated against the ETDRS charts in both healthy controls and patient populations to ensure equivalency of the tests. This can be done using Bland-Altman comparison.

One possible reason for the reduced precision in the low vision population is the impact of visual field loss on the ability to track across the letters on the chart as demonstrated in Figure 1. In the presence of visual field loss, once patients lose their place on the chart it can be very difficult for them to pick up and continue to read. VA testing therefore becomes a test of localisation of the position of the letters on the chart, as well as acuity. A group of patients with choroideremia, a rodcone dystrophy, participating in a focus group to discuss visual testing, confirmed experiencing this difficulty. The aim of this study was to measure the impact of using an electronic VA testing system on VA measurement. This system reduces the requirement for letter localisation due to presentation of the letters on the centre of the screen. VA testing was performed in a group of healthy volunteers to investigate equivalence of the electronic systems and ETDRS charts. We also tested a group of patients with rod-cone dystrophies in order to determine the relationship between visual field loss, VA performance and type of test.

Figure 1.

Effect of visual field loss on reading ability on ETDRS charts. (A) Microperimetry of the remaining central visual field in one eye with advanced choroideremia. The colored areas indicate the seeing parts of the retina. (B) ETDRS chart for measuring visual acuity. (C) A simulation of the appearance of the chart in B to the eye in A, making it difficult to track across the letters and obtain an accurate measure of visual acuity. (D) E-ETDRS presentation with single letter on the center of the screen. (E) EVA-SL presentation with a single line of letters across the center of the screen. (F) Simulation of visual field loss on EVA, the patient can more easily search around the letter as it stays within a limited area.

An electronic VA measurement system (EVA) was devised to promote greater standardisation of ETDRS visual acuity measurement.[7] This has been commercialised by the EMMES Corporation (EMMES, Rockville, USA) and is FDA approved. The EVA is able to display single letters on the centre of a computer screen surrounded by crowding bars with automated scoring (E-ETDRS). The automated algorithm resembles standard ETDRS testing with 5 letters tested at each size. Testing continues until the smallest line with no letters seen is reached.[7] In addition, manual scoring can be conducted with presentation of single letters or single lines in the centre of the computer screen. All these presentation methods can be set to use Sloan letters in order to resemble ETDRS charts as closely as possible. The automated single letter presentation (E-ETDRS) and manual single line (EVA-SL) presentation methods were investigated here.

Methods

Ethical approval for this study was granted by the UK Health Regulatory Authority (reference 17/NS/0036) and it was conducted in accordance with the Declaration of Helsinki. All participants provided informed consent to take part. The study population consisted of two groups, non-visually impaired volunteers and patients with a confirmed diagnosis of rod-cone dystrophy from the outpatient clinics at the Oxford Eye Hospital, United Kingdom. Forty-one volunteers were recruited through accompanying persons in the eye clinics and age-matched to the patient group. VA is not significantly affected by gender or fully corrected refractive error in a clinically significant way so this was not controlled between groups.[8,9] Sixty-five consecutive patients, with a diagnosis of rod-cone dystrophy, were included in the study. Patients underwent VA testing with ETDRS charts, E-ETDRS automated paradigm and the EVA single line presentation (EMMES Corporation, USA) with optimal refractive error as determined by a subjective refraction in place. Each test was performed twice on the right eye of each participant. All tests were completed once and then repeated in a random order. The preferred test for each patient participant was recorded, with reasons for the preference. The patient group underwent visual field testing with microperimetry on the MAIA using the 10-2 test grid centred over the fovea (Centervue SpA, Padova, Italy).

ETDRS charts were presented at 4m in a backlit frame. Chart R was used for refraction, chart 1 was presented to the right eye and chart 2 presented for the second test. Both eyes were covered during changeover of the charts to prevent any letters being memorized. If less than 20 letters were read at 4m, the chart was moved to 1m to test the top 6 lines, and the final score was the number of letters read at both distances. If more than 20 letters were read at 4m, 30 letters were added to the letter to score as the assumption was made that all letters would be read at 1m. The EVA system was set at 3m. The E-ETDRS presented the examiner with instructions on the iPod used to control the test stimuli and followed scoring previously described by Beck et al.[7] Results were displayed on the IPod screen on completion and recorded on the case report form. The EVA single line (EVA-SL) presentation followed ETDRS parameters as closely as possible with the 20/200 line being presented in the first instance. The presentation of stimuli followed the procedure outlined in Figure 2. Room lights were switched off for the duration of all three tests.

Figure 2.

Presentation paradigm for single line method of measuring visual acuity using the EVA.

Based on previous work, the standard deviation of the differences between the visual acuity assessments methods was estimated to be 0.08logMAR, translating to 4 letters.[6] With a sample size of 50, this equates to a 95% CI for each of the Bland-Altman limits of agreement of +/- 1.9 letters (using 1.71SD/√n). A sample size of 50 is the minimum recommended sample size for method agreement studies.[10] The extended Bland-Altman method, appropriate for repeated measurements, was used to assess agreement between the charts for each group of participants by calculating 95% limits of agreement. This method is robust to non-normal data.[11] The coefficient of repeatability (CR), the difference within which repeat readings are expected to be for 95% of participants, was calculated as 1.96√2 times the within subject standard deviation using one-way ANOVA (within subject standard deviation calculated from the square root of the residual mean square) under the assumptions of i) true replicates (no systematic difference between the samples) and ii) no relationship between difference and mean, tested using plots.[12,13] The eyes were split into “seeing” and “non-seeing” foveas based on microperimetry measurements. Non-seeing does not relate to the visual function of the eye but to the central fovea only for the rest of the manuscript. In order to differentiate patients with advanced loss of peripheral visual field from those who had lost central fixation and were using eccentric viewing, eyes were classified as foveal seeing or non-seeing based on the proportion of central microperimetry points seen (Figure 4A and B). If two or more of the central four points had a threshold > 1dB, the eye was classed as seeing centrally (n=25), otherwise the eye was classified as non-seeing centrally (n=37). The distance to the edge of the nearest seeing point on the microperimetry plot was calculated in degrees (the minus sign denotes the point was on non-seeing retina). The pairwise difference between tests were correlated (using Spearman’s rank correlation coefficient) with the horizontal and vertical distance to the nearest seeing point in the non-seeing fovea group and with the central threshold and central island diameter in the seeing fovea group. Statistical analysis was conducted with SPSS (version 25.0, IBM Software, New York, USA).

Figure 4.

Representative microperimetry examples of (A) Non-seeing eye showing a maximum of 1 seeing point in the central visual field. (B) Seeing eye with all central points having a measurable threshold. (C) Correlation in non-seeing eyes between difference between charts and electronic test and distance to nearest seeing point. (D) Correlation in seeing eyes between difference between charts and electronic test and central microperimetry threshold. (E) Correlation in seeing eyes between difference between charts and electronic test and width of vision at the fovea.

Patient participants were asked which test they preferred, if any, first after the ETDRS charts and E-ETDRS had been presented, and again after all three tests had been completed. On both occasions, they were asked to provide supporting reasons for test preference. Preserving the participants own use of language, verbal responses were broken down into component ideas.[14] Ideas were then categorised as relating to one of three themes; psychological, visual field and visual acuity.[15] Qualitative analysis was conducted in NVIVO (Version 11, QSR International Ltd, Victoria, Australia). The results was presented as a network diagram, influenced by Chi and Koeske’s categoric mapping, with line thickness indicating the number of participants reporting the specific preference or idea.[16] Where more than one reason for preference was mentioned, all were included in the final analysis.

Results

Participant demographics and summary results for the right eye are presented in Table 1. A range of genetic diagnoses were represented within the patient group, with 51% of participants having choroideremia, 18% having RPGR, 6% with USH2A mutations, 18% without a confirmed genetic mutation and 7% having other mutations.

Table 1. Summary of participant demographics and summary of test results.

Factor	Healthy volunteers	Patient participants
N (people)	42	65
Age (years) mean [range]	46.1 [20.1 – 75.2]	51.2 [22.9 – 81.1]
Male / Female	26 / 16	54 / 12
ETDRS VA (letters) mean ± SD 1st attempt	87 ± 5	48 ± 29
ETDRS VA (letters) mean ± SD 2nd attempt	87 ± 5	51 ± 28
E-ETDRS (letters) mean ± SD 1st attempt	90 ± 6	51 ± 29
E-ETDRS (letters) mean ± SD 2nd attempt	90 ± 6	52 ± 30
EVA-SL (letters) mean ± SD 1st attempt	89 ± 6	52 ± 28
EVA-SL (letters) mean ± SD 2nd attempt	89 ± 6	53 ± 27

Visual acuity measurement in healthy volunteers

The boxplot in Figure 3A shows the spread of the data on each test. The coefficient of repeatability was 6 letters in the ETDRS charts, 5 letters on the E-ETDRS protocol and 5 letters on the EVA-SL presentation. On average, volunteers read 2-3 letters more on both electronic versions of the test compared to the ETDRS charts with 95% CIs from -4 (-4 to -5) to 10 (10 to 10) and -5 (-5 to -5) to 9 (9 to 9) (for ETDRS vs E-ETDRS and EVA-SL respectively (Figures 3B dashed lines on Bland-Altman plots). The agreement of the tests was assessed using a modified Bland-Altman analysis of the differences between tests using the right eye data incorporating repeat measures.[11] The results are shown in Table 2 and Figure 3B Bland-Altman plot which plot difference between tests against the mean of the measures. The dashed line showing the mean difference between the tests and the dotted lines showing the limits of agreement between tests. Supplementary Figure 1 shows this data in more detail by altering the X-axis scale to a minimum of 75 letters (equivalent to 6/9.5).

Figure 3.

(A) Box and whisker plot showing mean, standard deviation and range of data points in healthy volunteer participants with outliers identified. (B) Bland-Altman plot comparing ETDRS charts and the two EVA presentation paradigms of the automated E-ETDRS and single line presentation (EVA-SL) in healthy volunteer group. The dashed lines represent mean difference and the dotted lines show the limits of agreement. (C) Box and whisker plot showing mean, standard deviation and range of data points in patient participants with outliers identified in the rod-cone dystrophy patient group. (D) Bland-Altman plot comparing ETDRS charts and the E-ETDRS paradigm in the patient group. (E) Bland-Altman plot comparing ETDRS charts and single line presentation (EVA-SL) in the patient group. (F) Bland-Altman plot comparing E-ETDRS and EVA-SL. The dashed lines in the Bland-Altman plots represent mean difference and the dotted lines show the limits of agreement.

Table 2.

Summary of modified Bland-Altman results for participants comparing ETDRS charts (ETDRS), electronic ETDRS testing paradigm (E-ETDRS), and electronic single line presentation (EVA-SL). Final patient subgroup analysis is based on classification of how many of the central foveal points on microperimetry were seeing or non-seeing.

Volunteer participants
Letters	E-ETDRS vs ETDRS		EVA-SL vs ETDRS		E-ETDRS vs EVA-SL
Mean Difference (95% CI)	3 (2 to 4)		2 (1 to 3)		1 (0 to 2)
Lower limit of agreement (95% CI)	-4 (-5 to -4)		-5 (-5 to -5)		-6 (-6 to -5)
Upper limit of agreement (95% CI)	10 (10 to 10)		9 (9 to 9)		7 (7 to 8)
Patient participants complete group
Mean Difference (95% CI)	3 (2 to 5)		3 (1 to 5)		0 (-1 to 2)
Lower limit of agreement (95% CI)	-10 (-11 to -10)		-10 (-11 to -10)		-12 (-12 to -11)
Upper limit of agreement (95% CI)	17 (17 to 17)		16 (16 to 17)		13 (12 to 13)
Patient participant subgroup analysis
Subgroup	Non-seeing	Seeing	Non-seeing	Seeing	Non-seeing	Seeing
Mean Difference (95% CI)	5 (2 to 7)	2 (-1 to 4)	4 (3 to 10)	2 (0 to 5)	1 (-1 to 3)	-1 (-3 to 2)
Lower limit of agreement (95% CI)	-9 (-10 to -8)	-11 (12 to -10)	-12 (-12 to -11)	-9 (-10 to -8)	-11 (-12 to -11)	-11 (-12 to -11)
Upper limit of agreement (95% CI)	18 (17 to 19)	14 (14 to 15)	19 (18 to 19)	13 (13 to 14)	13 (13 to 14)	10 (9 to 11)

Visual acuity measurement in patient group

The boxplot in Figure 3C shows the spread of the data. The coefficient of repeatability was 7.5 letters in the ETDRS charts, 7.3 letters on the E-ETDRS protocol and 7.9 letters on the EVA-SL presentation. On average, patients read 3 letters more on both electronic versions of the test compared to the ETDRS charts with 95% CIs from -10 (-10 to -11) to 17 (17 to 17) and -10 (-10 to-11) to 16 (16 to 17) for ETDRS vs E-ETDRS and EVA-SL respectively (Figures 3D-F dashed lines on Bland-Altman plots). The comparability of the tests was assessed using a modified Bland-Altman analysis of the differences between tests using the right eye data incorporating repeat measures.[11] The results are shown in Table 2 with tighter agreement between the two electronic tests. The dashed line showing the mean difference between the tests and the dotted lines showing the limits of agreement between tests.

Relationship to visual field

Three patients were unable to perform reliable microperimetry. A modified Bland-Altman analysis was conducted on the non-seeing and seeing eye subgroups. The non-seeing eyes showed wider limits of agreement and larger coefficient of variation than the seeing eyes as shown in Table 2. The CR of repeatability in the non-seeing group was 8 letters for ETDRS charts, 7 letters for the E-ETDRS protocol and 8 for EVA-SL. In the seeing group, CR was 5, 7 and 8 letters respectively for the ETDRS, E-ETDRS and EVA-SL respectively. We explored the factors affecting the spread of the data to understand what role visual field played in the measurement of VA.

For eyes with a non-seeing fovea, Spearman correlation of the difference in visual acuity measurement between ETDRS charts and E-ETDRS score, and the distance to the edge of the nearest seeing point was not significant (r=0.09, P=0.31) (Figure 4C).

In the seeing eyes, the threshold measured at the central four points was averaged. Neither central threshold (r=-0.12, P=0.28), nor central island diameter at the fovea (r=0.09, P=0.32) predicted the difference in visual acuity measurement between ETDRS charts and E-ETDRS measurement (Figure 4D and E).

Qualitative analysis

Of the 65 patient participants who completed both ETDRS chart and E-ETDRS (single letter) testing, the majority (30 participants) preferred the ETDRS chart format (Figure 5A), with 27 patients preferring E-ETDRS and 8 expressing no preference. Reassessment of test preference following EVA-SL was completed by 63 patients and showed 32 participants (51%) to prefer EVA single line, followed in popularity by ETDRS charts (11 patients, 17%) with 10 participants (16%) each preferring E-ETDRS and expressing no preference (Figure 5B). Test preference was represented as a network diagram, with line thicknesses representing ten, five or one participants for each response type. At both response points, factors pertaining to visual field were most commonly cited as affecting test preference. Scanning across the charts was the most cited factor in the preference decision, mentioned by 27/65 and 23/63 participants at the first and second response points respectively (Figure 5C and D). E-ETDRS presentation was least popular due to the perceived lack of feedback of VA and the confusion caused by the crowding bars.

Figure 5.

A) Test preference of the patient participants after presentation of ETDRS chart and E-ETDRS only; B) test preference after completion of all three tests. Line thickness indicates number of participant responses. For both diagrams, the key represents how many patients contributed to each line based on thickness. C) The reason for test preference after completion of ETDRS Charts and E-ETDRS charts. Reasons were assigned to one of three broad categories, identified by the colored background. D) Test preference after completion of all three tests (ETDRS charts, E-ETDRS and EVA-SL), and reasons given to support test preference. Line thickness indicates number of participant responses. Line colour indicates preferred test. Where more than one reason was given, both are included. Reasons are grouped according to thematic content.

Longitudinal VA measurement

A further subset of 14 patients with a variety of visual defects underwent assessment with ETDRS and E-ETDRS at five visits over 6 months in order to confirm the stability of measurements. These are shown in Supplementary Figure 2. Visual analysis of the graph shows the straighter lines indicating more stable measurement with the E-ETDRS paradigm. This appears to be more marked as the VA reduces. However, the Levene’s test of equality of variances (one-way ANOVA) did not show any significant differences (F=1.18, P=0.28). The raw data are shown in table S1.

Discussion

The vision measured with the electronic charts is equivalent to traditional ETDRS charts in healthy participants and the tests can be used interchangeably. The limits of agreement in the patient population are larger than the 10 line cut off for variability in the rod-cone dystrophy patient population, particularly with the non-foveal seeing eyes, and therefore the charts should be used as appropriate for the study. Electronic presentation of optotypes on the centre of a computer screen may provide more precise measures of visual acuity than traditional ETDRS charts in patients with visual field loss affecting the central fovea. The ability to see with the central foveal area appears to be the only determinant of whether the electronic charts are beneficial as neither distance of the seeing points from the fovea of threshold in the centre were factors in explaining the results. The wider limits of agreement in the patient group is as expected, and the variability is largely explained by the non-seeing foveal eyes with the larger limits of agreement and larger CR as compared to the foveal seeing eyes. The two electronic display paradigms are closely related as shown by the comparison of the 95% confidence intervals for the means. Interestingly, the limits of agreement in the foveal non-seeing eyes are narrower and comparable to the foveal seeing eyes when comparing both electronic charts, indicating a higher degree of precision by the electronic charts when comparing the tests. The coefficient of repeatability was similar across test paradigms and remained within the 10 letters (2 lines) previously reported indicating any of the tests can be used for measurement of VA in a clinical trial setting. The largest CR was in the eyes where foveal vision was affected, indicating this group accounts for the bulk of the variability seen in patients. Single line presentation appeared to be the favoured style by patients and should be considered for future interventional clinical trials.

The electronic charts have two known advantages over the physical chart format. Firstly, the localisation element is removed. Secondly all VA sizes can be measured without adjustment of the screen. For VA measurement below 20/200, the ETDRS charts must be moved to 1m in order to continue testing which may introduce error due to change in the accommodative demand. Furthermore, in those with visual field loss, changing the distance of the charts will affect where the letters fall on the remaining visual field. By removing this requirement to move the chart, more consistent measurements can be obtained and may explain why the VA measurements appear to be more stable over time with the EVA system, particularly for those with low VA. With small visual fields, larger letters can be too big to fit within the area, causing discrepancies between VA measurements on large versus small letters. By projecting a single letter on the centre of the screen, the patient can more easily scan around the letter to identify it and not fear losing their place on the chart. Further testing on a larger sample size with a larger number of lower vision patients is required to confirm this trend.

Although tracking abilities are an important factor in day to day vision, clinical trials and clinical service measurement of vision will have different priorities. In clinical trials defining visual acuity as a key endpoint, the absolute acuity will be the critical measure. In this situation, removing the error introduced by tracking will be important. However, in paediatric clinics, the assessment of real world ability is a greater priority so this may not be the best approach to use. With low vision monitoring, the goals of the patient and practitioner will influence whether electronic VA or chart based VA measurement is most appropriate.

The psychological impact of undertaking research is becoming a critical factor, particularly for work funded by public funding bodies in Europe. The patient preference data provides a valuable indication of patient comfort and acceptability. Test preference did not impact on the variability reported in this single visit cohort. However, when patient comfort is increased, test reliability is also likely to be augmented in the longer term thus providing a more stable and reliable outcome measure for interventional trials with multiple repeated visits. [17]

We have validated two testing paradigms on the commercially available and FDA approved EVA system against ETDRS charts in a group of healthy participants. In patients with rod-cone dystrophy we have shown increased precision of VA measurement in the short and long term using electronic VA measurement, particularly when the visual field loss affects the fovea. We suggest larger studies comparing EVA to ETDRS in a range of disease types would be helpful to confirm these findings. We propose that future trial protocols should consider the use of electronic VA in patients with visual field loss.

Synopsis.

Measurement of visual acuity with an electronic system showing fewer letters increases precision in the measurement of visual acuity in patients with rod-cone dystrophies who have central visual field loss affecting the foveal area.