Abstract
Purpose:
Evaluate the differences between clinical visual acuity (VA) as recorded in medical records and electronic ETDRS (eETDRS) protocol VA measurements and factors affecting the size of the differences.
Design:
Retrospective chart review.
Participants:
Study and fellow eyes of participants enrolled in DRCR Retina Network Protocols AC and AE (diabetic macular edema), and W (non-proliferative diabetic retinopathy) with clinical VA recorded within 3 months before the protocol visit.
Methods:
Linear mixed models evaluated the differences and their association with patient and ocular factors in univariable and multivariable models, with random effects for correlations within sites and participants.
Main Outcome Measure:
Difference between VA letter scores measured by eETDRS during a study protocol visit versus measured by Snellen during a regular clinical visit (Snellen fraction converted to eETDRS).
Results:
Data from 1016 eyes (511 participants) across 74 sites were analyzed. The mean VA measurements were 68.6 letters (Snellen equivalent 20/50) at the clinical visit and 76.3 letters (Snellen equivalent 20/32) at the protocol visit, with a mean (standard deviation, SD) of 26 (21) days between visits. Mean (SD) protocol VA was better than clinical VA by 7.6 (9.6) letters overall, 10.7 (12.6) letters in eyes with clinical VA ≤20/50 (n = 376) and 5.8 (6.6) letters in eyes with clinical VA ≥20/40 (n = 640). On average, the difference between clinical and protocol VA was 1.3 letters smaller for every 1-line (5 letters) increase in clinical VA (p < 0.001). Mean (SD) differences by clinical correction of refractive error were 3.9 (9.0) letters with refraction, 6.9 (9.2) letters with glasses/contact lenses, 7.9 (11.5) letters with pinhole and 9.8 (9.3) letters without correction (p=0.06).
Conclusion
On average, clinical Snellen VA is likely to be 1–2 lines worse than eETDRS protocol refraction and VA testing, which may partly explain why clinical practice does not always replicate clinical trial results. Eyes with lower clinical measurements and eyes tested without clinical refraction tended to have larger differences. Considering the potential discrepancies between clinical and protocol VA measurements, refracting eyes in the clinic may benefit patients when determining treatment plans and study referrals based on vision.
Précis:
Visual acuity measurements from ETDRS protocol testing tend to be better than Snellen measurements recorded in clinical practice. Differences were greater when clinical measurements were lower and when they were taken without refraction.
Introduction
Current clinical trials involving retinal disease usually quantify and report outcomes for visual acuity (VA) with measurements made by certified technicians using modified ETDRS protocols for refraction and testing. Testing with either ETDRS visual acuity charts or an electronic ETDRS (eETDRS) is required for FDA registration trials. These tests provide accurate, repeatable measurements on an interval scale (logarithm of minimum angle of resolution [LogMAR]).1–3 In the opinion of the authors, most ophthalmologists are more familiar with Snellen charts and use them to obtain VA measurements in their offices. Often, measures of VA from clinical trials VA are reported in a Snellen equivalent for better understanding by clinicians who do not use ETDRS measurements.4–6 However, the direct conversion of ETDRS to Snellen VA does not completely account for the differences between the standardized ETDRS VA measurement obtained in a clinical trial setting and the routine Snellen VA obtained in clinical practice.7–9 Differences in lighting, testing distance, correction of refractive error, and testing technique may also affect the difference in measurements.
Previous reports of discrepancies between ETDRS and routine Snellen VA measurements have indicated that, on average, a better VA score is obtained with ETDRS testing.9–12 This finding was maintained when a standardized “best-corrected” refraction was used prior to testing with both methods.9 The disparity in VA measurements has been reported across multiple retinal diseases and appears to be greater in eyes with lower visual acuity and more macular edema.9–12 The DRCR Retina Network has published multi-center clinical trial results that provide guidance for drug choice and management strategies for the treatment of diabetic macular edema (DME).13,14 The guidance from some of these trials is dependent upon the visual acuity measurements at the onset of treatment and throughout the course of treatment.15,16 Application of this guidance to clinical practice requires a better understanding of the differences between standardized ETDRS vision testing in clinical trials and routine measurement of vision in clinical practice as recorded in the medical record. 9,10,17
Both choice of treatment modality based on results from previous clinical trials and referral for clinical trials may be influenced by the VA measurements made in routine clinical practice, where many of the conditions of VA testing (charts, lighting, refractive correction, and testing distance and technique) differ from those of ETDRS protocol testing. This study assessed the variability in the routine techniques of obtaining clinical VA and the differences between clinical Snellen and eETDRS protocol VA measurements performed for three different DRCR Retina Network clinical trials across 74 clinical centers. The centers followed varied practices to obtain routine clinical VA assessments before referring patients for clinical trial eligibility in which the visual acuity was remeasured using standard eETDRS protocols. Additionally, factors that might affect the size of the differences between the two methods of VA assessment were analyzed.
Methods
We evaluated the difference between protocol and clinical VA for participants in DRCR Protocols AC, AE, and W, which were active studies in May of 2020 when the retrospective data collection of clinical VA began. All participants were adults (≥ 18 years old) with Type 1 or Type 2 diabetes. The analysis included both study and non-study (fellow) eyes. Study eyes in Protocol AC had center-involved diabetic macular edema (CI-DME) and VA from 20/320 to 20/50 eETDRS. Study eyes in Protocol AE had CI-DME and VA at least 20/25 eETDRS. Study eyes in Protocol W had moderate to severe non-proliferative diabetic retinopathy (NPDR), no CI-DME and VA at least 20/25 eETDRS. Eligibility criteria for each clinical trial specified no treatment in study eyes within the 4-months before baseline; therefore, there was no treatment between the clinical and protocol visits. Fellow eyes needed to have an eETDRS protocol refraction at baseline and a clinical Snellen VA measurement within the preceding 3 months (≤93 days) but otherwise had no restrictions on the level of VA, eye pathologies, or treatment history. All participants signed consent forms that covered collection of ocular and medical history.
Clinical VA was collected retrospectively from site records of the last visit within 3 months of the protocol visit and recorded as a Snellen fraction without plus or minus letters. Protocol VA was collected on DRCR case report forms at the first study visit and was measured using eETDRS charts after refraction. Clinical Snellen fractions were converted to approximate ETDRS letter scores using the formula 85 + 50 × log10 (Snellen fraction), rounded to the nearest letter.18 Snellen fractions <20/800 were converted to a letter score of 0, the lowest testable letter score on the eETDRS testing system used in DRCR Retina Network clinical trials. The difference in VA was evaluated as the protocol eETDRS letter score minus clinical Snellen converted to a letter score (i.e., a positive difference indicates greater VA with eETDRS testing). The protocol eETDRS letter score was aggregated to a line-level letter score in a sensitivity analysis to assess the difference when clinical and protocol VA measurements had similar increments.
The VA differences were analyzed using linear mixed models with nested random intercept effects to model the potential correlation within sites and between the two eyes of a participant. An intercept-only model was used to estimate Bland-Altman 95% limits of agreement.19 Univariable models of candidate predictive factors were used to select factors with p <0.10 for inclusion in an initial full multivariable model. A final multivariable model was selected using a backward elimination procedure, iteratively dropping factors with the largest p-value. The factors examined were clinical VA, clinical VA correction method, protocol VA refractive error (spherical equivalent), days between the clinical and protocol visit, age, hemoglobin A1c (HbA1c), central subfield thickness (CST), lens status, diabetic retinopathy (DR) severity level, clinical trial (Protocol W, AC, or AE), and an indicator of whether the eye was a study eye. Unless specified as clinical, all factors were collected at the protocol visit. All analyses were conducted in SAS/STAT 15.1 (SAS Institute, Inc, Cary, NC).
Results
We analyzed data from 1016 eyes (511 participants) collected between February 2016 and February 2020 across 74 sites in the US and Canada. This included 70% (1016 of 1443) of eyes from the original studies that had an eETDRS protocol refraction, with at least one eye from 86% (74 of 86) of sites and 70% (511 of 733) of participants: 62% (84 of 135) from Protocol AE, 78% (211 of 270) from Protocol AC, and 66% (216 of 328) from Protocol W. The study excluded 269 (19%) of the 1443 eyes with protocol VA because clinical VA was not collected, 118 (8%) of eyes with clinical VA collected were excluded because the clinical VA was measured >3 months before the protocol visit (n = 98), or measured using methods other than Snellen charts (n = 14 with ETDRS, n = 4 with Colenbrander charts, and n = 2 with counting fingers), and the remaining 40 (3%) of eyes were excluded because the participant was enrolled at a non-participating site. Tables 1 and 2 summarize participant and eye characteristics for those included. The mean (standard deviation (SD)) age was 59 (10) years, 283 (55%) were male, 296 (58%) were White, 118 (23%) were Hispanic or Latino, and 73 (14%) were Black or African American. There were 582 (57%) study eyes and 434 (43%) fellow eyes. Sixteen percent of the eyes were of patients enrolled in Protocol AE, with the remainder split approximately equally between Protocols AC and W.
Table 1.
Participant characteristics
| Participant Characteristics | Excluded | Included |
|---|---|---|
|
| ||
| No. of participants | ||
| N | 222 | 511 |
| Protocol, N (%) | ||
| AC | 59 (27%) | 211 (41%) |
| AE | 51 (23%) | 84 (16%) |
| W | 112 (50%) | 216 (42%) |
| Age, years | ||
| Mean (SD) | 59 (11) | 59 (10) |
| Median (IQR) | 59 (52, 67) | 59 (53, 65) |
| Sex, N (%) | ||
| Female | 94 (42%) | 228 (45%) |
| Male | 128 (58%) | 283 (55%) |
| Race/Ethnicity, N (%) | ||
| White | 109 (50%) | 296 (58%) |
| Hispanic or Latino | 64 (29%) | 118 (23%) |
| Black/African American | 39 (18%) | 73 (14%) |
| Other | 8 (4%) | 20 (4%) |
| Diabetes type, N (%) | ||
| Type 1 | 20 (9%) | 36 (7%) |
| Type 2 | 202 (91%) | 475 (93%) |
| Duration of diabetes, years | ||
| Mean (SD) | 17.4 (10.3) | 16.7 (10.3) |
| Median (IQR) | 16.6 (10.2, 22.9) | 15.8 (9.4, 22.6) |
| Insulin used, N (%) | ||
| No | 81 (36%) | 167 (33%) |
| Yes | 141 (64%) | 344 (67%) |
| Hemoglobin A1c, % | ||
| Mean (SD) | 8.4 (1.8) | 8.5 (2.0) |
| Median (IQR) | 8.0 (7.2, 9.4) | 8.1 (7.1, 9.5) |
N missing: Race/Ethnicity (2 excluded and 4 included); Hemoglobin A1c (18 excluded and 17 included).
Table 2.
Eye characteristics
| Eye Characteristics | Excluded | Included | Included by Clinical VA |
|
|---|---|---|---|---|
| 20/40 or better | 20/50 or worse | |||
|
| ||||
| No. of eyes | ||||
| 427 | ||||
| N | (118 with Clinical VA) | 1016 | 640 | 376 |
| Time between clinical and protocol visita, days | ||||
| Mean (SD) | 204 (191) | 26 (21) | 29 (22) | 20 (18) |
| Median (IQR) | 145 (104, 240) | 21 (8, 38) | 25 (12, 42) | 14 (7, 28) |
| Clinical VA testina methodb. N (%) | ||||
| Colenbrander | 4 (3%) | |||
| Counting Fingers | 2 (2%) | |||
| ETDRS | 14 (12%) | |||
| Snellen | 98 (83%) | 1016 (100%) | 640 (100%) | 376 (100%) |
| Clinical VA correction method, N (%) | ||||
| Refracted | 32 (29%) | 63 (7%) | 46 (8%) | 17 (5%) |
| Glasses or contacts | 37 (34%) | 497 (53%) | 305 (52%) | 192 (56%) |
| Pinhole only | 10 (9%) | 117 (13%) | 72 (12%) | 45 (13%) |
| Uncorrected | 31 (28%) | 254 (27%) | 167 (28%) | 87 (26%) |
| Clinical VA, Snellen c | ||||
| Mean | 20/32 | 20/50 | 20/32 | 20/100 |
| Protocol VA, Snellen (converted from eETDRS) c | ||||
| Mean | 20/32 | 20/32 | 20/20 | 20/63 |
| Protocol, N (%) | ||||
| AC | 120 (28%) | 419 (41%) | 127 (20%) | 292 (78%) |
| AE | 102 (24%) | 168 (17%) | 149 (23%) | 19 (5%) |
| W | 205 (48%) | 429 (42%) | 364 (57%) | 65 (17%) |
| Eye enrollment, N (%) | ||||
| Fellow Eye | 177 (41%) | 434 (43%) | 302 (47%) | 132 (35%) |
| Study Eye | 250 (59%) | 582 (57%) | 338 (53%) | 244 (65%) |
| Protocol VA refractive error (spherical equivalent)d N (%) | ||||
| > +1.50 | 46 (11%) | 94 (9%) | 53 (8%) | 41 (11%) |
| +1.50 to +.625 | 92 (22%) | 193 (19%) | 115 (18%) | 78 (21%) |
| +.50 to −.50 | 152 (36%) | 371 (37%) | 254 (40%) | 117 (31%) |
| −.625 to −1.50 | 70 (16%) | 148 (15%) | 98 (15%) | 50 (13%) |
| < −1.50 | 67 (16%) | 210 (21%) | 120 (19%) | 90 (24%) |
| OCT Central subfield thickness (Spectralis)e, μm | ||||
| Mean (SD) | 358 (126) | 377 (131) | 312 (65) | 485 (141) |
| Median (IQR) | 314 (279, 377) | 329 (287, 429) | 297 (273, 335) | 457 (373, 574) |
| CI-DMEf, N (%) | ||||
| Yes | 142 (46%) | 368 (54%) | 140 (33%) | 228 (89%) |
| No | 168 (54%) | 317 (46%) | 289 (67%) | 28 (11%) |
| OCT Retinal volume (Stratus) g , mm3 | ||||
| Mean (SD) | 8.0 (1.6) | 8.3 (1.8) | 7.6 (1.0) | 9.7 (2.2) |
| Median (IQR) | 7.6 (7.1, 8.3) | 7.8 (7.2, 8.7) | 7.5 (7.0, 8.0) | 9.3 (8.0, 10.9) |
| Lens status, N (%) | ||||
| Aphakic | 1 (<1%) | |||
| IOL | 94 (22%) | 192 (19%) | 125 (20%) | 67 (18%) |
| Phakic | 332 (78%) | 824 (81%) | 515 (80%) | 309 (82%) |
| Diabetic retinopathy severity, N (%) | ||||
| No DR to moderate NPDR (< level 47) |
47 (23%) | 121 (24%) | 51 (18%) | 70 (30%) |
| Moderately Severe NPDR (level 47) |
84 (41%) | 225 (44%) | 164 (58%) | 61 (27%) |
| Severe NPDR (level 53) | 58 (28%) | 117 (23%) | 57 (20%) | 60 (26%) |
| PDR (> level 53) or prior PRP |
18 (9%) | 48 (9%) | 9 (3%) | 39 (17%) |
The time between the clinical and protocol visit was limited to 3 months (≤93 days) for included eyes.
The clinical VA testing method was limited to Snellen charts for included eyes because the other methods had <20 eyes each.
Letter score conversion to Snellen: 0–3 = <20/800; 4–8 = 20/800; 9–13 = 20/640; 14–18 = 20/500;19–23 = 20/400; 24–28 = 20/320; 29–33 = 20/250; 34–38 = 20/200; 39–43; 20/160; 44–48 = 20/125; 49–53 = 20/100;54–58 = 20/80; 59–63 = 20/63; 64–68 = 20/50; 69–73 = 20/40; 74–78 = 20/32; 79–83 = 20/25; 84–88 = 20/20; 89–93 = 20/16;94–98 = 20/12; 99–100 = 20/10.
Spherical equivalent refractive error = sphere power + 0.5 × cylinder power.
CST measurements on Cirrus were converted to Spectralis equivalents using the following formula: Spectralis = 40.78 + 0.95 × Cirrus.22
CI-DME was defined by CST on Cirrus ≥ 290 (females) or ≥ 305 (males); and CST on Spectralis ≥ 305 (females) or ≥ 320 (males).
All retinal volume measurements were converted to Stratus equivalents using the following formulas: Stratus = −1.21 + 1.01 × Cirrus; Stratus = −2.05 + 1.06 × Spectralis.23
N missing: Clinical VA (309 excluded); Time between protocol and clinical visit (309 excluded); Clinical VA testing method (309 excluded); Clinical VA correction method (317 excluded and 85 included); OCT Central subfield thickness (117 excluded and 331 included); CI-DME (117 excluded and 331); OCT Retinal volume (136 excluded and 381 included); Diabetic retinopathy severity (220 excluded and 505 included).
The mean (SD) time between the protocol and clinical visit was 26 (21) days, with 79% (806 of 1016) less than 6 weeks apart. The mean (SD) clinical VA (converted from Snellen fraction) was 68.6 (17.1) letters and the mean (SD) protocol VA (eETDRS) was 76.3 (15.6) letters. The differences in VA (protocol minus clinical) ranged from −35 to 76 letters. Potential outliers (n = 17) were truncated to within the 3 SDs of the mean of the raw data (−23.2 to 38.6) to prevent extreme values from disproportionately impacting the analysis. The differences in VA appeared to be approximately normally distributed with protocol VA a mean (SD) of 7.6 (9.6) letters better than clinical VA, with 95% limits of agreement of −11 to 26 letters (Table 3, Figure 1). In eyes with a clinical VA 20/50 or worse (n = 376), the mean (SD) difference in protocol minus clinical VA was 10.7 (12.6), approximately double the 5.8 (6.6) difference in eyes with a clinical VA 20/40 or better (n = 640). Protocol and clinical measurements were <5 letters (1-line) of each other for slightly less than a quarter (23%) of eyes in the 20/50 or worse group and slightly over a third (36%) of eyes in the 20/40 or better group. It was more common for eyes with clinical VA ≤ 20/50 to have protocol VA ≥15 letters better (38%) than 10–14 (13%) or 5–9 (16%) letters better. In contrast, more eyes with clinical VA ≥ 20/40 had protocol VA better by only 5–9 letters (31%) compared to 10–14 (19%) or ≥15 (9%) letters better. It was rare for protocol VA to be 10–14 (2%) or ≥15 (1%) letters worse than clinical VA. Only 7% of the eyes overall had protocol VA ≥5 letters worse than clinical VA (Table 3).
Table 3.
Visual acuity by each testing method and by clinical visual acuity subgroups
| Visual Acuity | Clinical VA |
||
|---|---|---|---|
| Overall | 20/40 or better | 20/50 or worse | |
|
| |||
| No. of eyes | |||
| N | 1016 | 640 | 376 |
| Clinical VA, letter score (converted from Snellen fraction) a | |||
| Mean (SD) | 68.6 (17.1) | 78.7 (5.8) | 51.3 (16.1) |
| Median (IQR) | 75 (60, 80) | 80 (75, 85) | 58 (45, 65) |
| Protocol VA, letter score (eETDRS) | |||
| Mean (SD) | 76.3 (15.6) | 84.5 (7.0) | 62.3 (16.3) |
| Median (IQR) | 82 (67, 87) | 86 (82, 89) | 64 (55, 71) |
| VA difference (protocol - clinical), letter score b | |||
| Mean (SD) | 7.6 (9.6) | 5.8 (6.6) | 10.7 (12.6) |
| Median (IQR) | 7 (2, 13) | 5 (2, 10) | 10 (2, 20) |
| VA difference (protocol - clinical), N (%) | |||
| Protocol ≥15 letters better | 204 (20%) | 60 (9%) | 144 (38%) |
| Protocol 10–14 letters better | 169 (17%) | 121 (19%) | 48 (13%) |
| Protocol 5–9 letters better | 256 (25%) | 197 (31%) | 59 (16%) |
| Same ± 4 letters | 316 (31%) | 229 (36%) | 87 (23%) |
| Protocol 5–9 letters worse | 40 (4%) | 19 (3%) | 21 (6%) |
| Protocol 10–14 letters worse | 17 (2%) | 11 (2%) | 6 (2%) |
| Protocol ≥15 letters worse | 14 (1%) | 3 (<1%) | 11 (3%) |
Converted letter score = 85 + 50 × log10(Snellen fraction), rounded to the nearest letter, or = 0 for Snellen fractions <20/800.
VA difference outliers were truncated at the raw overall mean ± 3 SD (7.7 ± 3 × 10.3).
Figure 1.
Scatter plots of (A) Protocol VA vs. Clinical VA; (B) VA Difference vs. VA Mean (Bland-Altman plot); (C) VA Difference vs. Clinical VA; D) VA Difference vs. Hemoglobin A1c. Scatter plots for (A) Protocol VA vs. Clinical VA; (B) VA Difference vs. VA Mean (Bland-Altman plot); (C) VA Difference vs. Clinical VA; and D) VA Difference vs. Hemoglobin A1c. Clinical visual acuity measurements of Snellen fractions were converted to approximate ETDRS letter scores using the formula: Converted letter score = 85 + 50 × log10(Snellen fraction), rounded to the nearest letter, or = 0 for Snellen fractions <20/800. VA differences were evaluated as protocol VA minus clinical VA, and outliers were truncated at the overall mean ± 3 SD (7.7 ± 3 × 10.3).
Among study eyes, 68% (396 of 582) of the clinical VA measurements were within the protocol VA eligibility criteria (at least 20/25 in protocols AE and W; and 20/320 to 20/50 in AC). This percentage was higher among clinical measurements tested with refraction [91% (31 of 34)] compared to those tested with glasses or contacts [73% (210 of 286)], pinhole [60% (39 of 65)], and uncorrected refractive error [53% (77 of 146)] or missing correction methods [76% (39 of 51)].
The mean (SD) of protocol VA, clinical VA, and the difference (protocol minus clinical) by each factor of interest are in Table 4 (continuous factors were dichotomized). The differences in protocol and clinical VA were not associated with the time between visits (p = 0.73), protocol (p = 0.37), CST (p = 0.78), DR severity (p = 0.95) or lens status (p = 0.82) based on the univariable models (eTable 5). However, clinical VA, clinical VA correction method, HbA1c, eye enrollment, age, and protocol VA refractive error were associated (p < 0.10) and were therefore selected for the initial multivariable model. Using backward elimination, age (p = 0.88) and protocol VA refractive error (p = 0.13) were dropped, and the other factors remained (eTable 6).
Table 4.
Visual acuity by each testing method and factors that may affect the difference in results between methods
| Factor | No. of eyes | Visual Acuity Letter Score | Mean (SD) | |
|---|---|---|---|---|
|
| ||||
| Clinical (Snellen conversion)a | Protocol (eETDRS) | Difference (Protocol - Clinical)b | ||
|
| ||||
| Hemoglobin A1c | ||||
| <7.5% | 348 | 68.1 (16.3) | 76.5 (14.4) | 8.3 (9.5) |
| ≥7.5% | 668 | 68.9 (17.5) | 76.2 (16.2) | 7.2 (9.6) |
| Age | ||||
| <60 years | 541 | 70.9 (16.8) | 77.9 (15.7) | 6.9 (9.7) |
| ≥60 years | 475 | 65.9 (17.1) | 74.5 (15.3) | 8.4 (9.4) |
| OCT Central subfield thickness (Spectralis) | ||||
| <330 μm | 344 | 78.2 (7.6) | 86.4 (6.1) | 8.2 (6.8) |
| ≥330 μm | 672 | 63.7 (18.5) | 71.1 (16.5) | 7.3 (10.7) |
| Time between protocol and clinical visit | ||||
| <6 weeks | 806 | 67.4 (17.6) | 75.0 (15.9) | 7.5 (9.9) |
| ≥6 weeks | 210 | 73.3 (14.1) | 81.3 (13.5) | 7.9 (8.0) |
| Clinical VA correction method | ||||
| Refracted | 63 | 73.9 (18.9) | 78.0 (17.5) | 3.9 (9.0) |
| Glasses or contacts | 497 | 68.5 (16.9) | 75.5 (15.2) | 6.9 (9.2) |
| Pinhole only | 117 | 66.1 (18.0) | 74.4 (16.9) | 7.9 (11.5) |
| Uncorrected | 254 | 68.6 (17.3) | 78.5 (15.1) | 9.8 (9.3) |
| Protocol VA refractive error (spherical equivalent) | ||||
| > +1.50 | 94 | 65.9 (16.7) | 74.4 (15.9) | 8.6 (8.8) |
| +1.50 to +.625 | 193 | 66.2 (18.5) | 76.0 (15.3) | 9.5 (9.7) |
| +.50 to −.50 | 371 | 70.4 (16.1) | 77.7 (15.8) | 7.3 (8.7) |
| −.625 to −1.50 | 148 | 70.1 (15.9) | 78.5 (13.9) | 8.2 (10.1) |
| <−1.50 | 210 | 67.8 (18.1) | 73.5 (16.2) | 5.5 (10.6) |
| Protocol | ||||
| AC | 419 | 57.7 (17.9) | 64.6 (15.5) | 6.8 (11.7) |
| AE | 168 | 77.2 (9.8) | 83.9 (8.2) | 6.8 (6.9) |
| W | 429 | 75.8 (12.1) | 84.7 (9.4) | 8.7 (8.0) |
| Eye enrollment | ||||
| Fellow Eye | 434 | 70.5 (16.1) | 78.8 (14.8) | 8.1 (9.4) |
| Study Eye | 582 | 67.2 (17.7) | 74.5 (16.0) | 7.2 (9.7) |
| Lens status | ||||
| IOL | 192 | 69.8 (15.5) | 76.9 (14.2) | 7.0 (8.0) |
| Phakic | 824 | 68.3 (17.4) | 76.2 (15.9) | 7.7 (9.9) |
| Diabetic retinopathy severity | ||||
| No DR to moderate NPDR (< level 47) |
121 | 63.1 (16.3) | 70.1 (14.6) | 6.9 (10.1) |
| Moderately Severe NPDR (level 47) |
225 | 72.6 (14.6) | 80.2 (14.0) | 7.7 (8.1) |
| Severe NPDR (level 53) | 117 | 63.9 (20.0) | 71.8 (18.0) | 7.8 (11.2) |
| PDR (> level 53) or prior PRP |
48 | 52.2 (19.6) | 59.0 (13.6) | 6.4 (12.5) |
Converted letter score = 85 + 50 × log10(Snellen fraction), rounded to the nearest letter, or = 0 for Snellen fractions <20/800.
VA difference outliers were truncated at the raw overall mean ± 3 SD (7.7 ± 3 × 10.3).
The final multivariable model included 901 eyes with non-missing HbA1c and clinical correction method (Table 5). On average, the difference between protocol and clinical VA was 1.3 letters smaller when the clinical VA letter score was 1 line (5 letters) better (p < 0.001) and 0.3 letters smaller when HbA1c was greater by 1% (p = 0.05). The mean (SD) differences between protocol and clinical VA measurements by the method of clinical refractive error correction were 3.9 (9.0) letters with refraction, 6.9 (9.2) with glasses or contacts, 7.9 (11.5) with pinhole, and 9.8 (9.3) without correction (Table 4). The adjusted mean differences were 3.0 letters smaller for refracted vs. uncorrected eyes (p = 0.04), and 1.8 letters smaller for eyes tested wearing glasses or contacts vs. uncorrected eyes (p = 0.02). Lasty, study eyes had an adjusted 1.9 letter smaller mean difference than fellow eyes (p < 0.001) (Table 5).
Table 5.
Final multivariable model estimates for VA difference (protocol – clinical)
| Final Multivariable Model Effectsa | Estimate | (95% CI) | P-value |
|---|---|---|---|
|
| |||
| Clinical VA, 1-line (5 letters converted from Snellen fraction) b | −1.3 | (−1.5, −1.1) | <0.001 |
| Hemoglobin A1c, % | −0.3 | (−0.6, −0.0) | 0.05 |
| Eye enrollment | <0.001 | ||
| Fellow Eye | 1.9 | (1.0, 2.8) | <0.001 |
| Study Eye (reference) | 0.0 | ||
| Clinical VA correction method | 0.06 | ||
| Refracted | −3.0 | (−5.9, −0.1) | 0.04 |
| Glasses or contacts | −1.8 | (−3.3, −0.3) | 0.02 |
| Pinhole only | −1.5 | (−3.5, 0.5) | 0.14 |
| Uncorrected (reference) | 0.0 | ||
The final multivariable model for VA difference (protocol - clinical) included N = 901 eyes with no missing covariate values in a linear mixed model with nested random effects on the intercept for correlation within sites and participants; VA difference outliers were truncated at the raw overall mean ± 3 SD (7.7 ± 3 × 10.3).
Converted letter score = 85 + 50 × log10(Snellen fraction), rounded to the nearest letter, or = 0 for Snellen fractions <20/800.
The intraclass correlation was 0.14 within sites and 0.38 within participants within sites. The conditional R2 (variance explained by fixed and random factors) was 0.52.
The distributions of VA differences were comparable across sites, with a 0.14 correlation among differences within sites. The final multivariable model explained 52% of the variation in the differences between protocol and clinical VA, meaning about half of the considerable variance (overall SD of differences = 9.6 letters ≈ 2 lines) was still unexplained. Similar estimates for the final model were obtained in three sensitivity analyses: (1) when clinical VA was included as a categorical instead of a continuous variable; (2) when an unknown category for the clinical correction method was included; and (3) when the protocol eETDRS letter score was aggregated to a line-level letter score before taking the difference with the clinical measurement (eTables 8–10).
Discussion
In this DRCR Retina Network assessment of differences between protocol eETDRS and Snellen VA recorded at a clinical visit within the preceding 3 months, protocol eETDRS vision averaged 7.6 letters (≈1.5 lines) better than clinical Snellen VA. The variability of VA differences was considerable, with 95% of differences expected to be within −11 to 26 letters (protocol VA approximately 2 lines worse to 5 lines better than clinical VA). It was rare for protocol VA to be substantially worse than clinical VA, with only 7% of protocol measurements lower than clinical measurements by 5 or more letters (1 line). As found in previous studies, the discrepancy between protocol and non-protocol VA measurements was greater among eyes with worse vision.6–9 Possible reasons for larger differences in eyes with worse clinical VA measurements are that Snellen charts have larger gaps in letter size between lines and fewer letters than the ETDRS charts at the lower VAs and test-retest reliability is worse at the lower VAs.2,3 Furthermore, in eyes with good clinical VA, the possible improvement by protocol assessment is limited by the upper limit of the letter score scale (100 letters, 20/10).
In addition to the chart used for testing, several factors may be responsible for differences in our observed VA measurements, including that not all clinical assessments tested the best corrected vison. In this study, methods of correcting refractive error for the clinical VA included no correction, habitual correction (current glasses or contact lenses), pin-hole correction, and refraction. As expected, refraction before clinical VA measurements resulted in less difference, reducing the average difference (7.6 letters) by about half (3 letters). Even with refraction, clinical VA still measured approximately one line worse than the eETDRS protocol VA. As such, it may be beneficial to refract patients when determining treatment plans and referrals to clinical trials based on VA. Based solely only on the most recent clinical Snellen VA recorded in site records prior to study entry, only about half (53%) of the eyes with clinical measurements tested without any refractive correction would have met study VA eligibility criteria, compared to 91% of those tested after refraction, even though 100% of the study eyes met the study VA eligibility criterion.
In the analysis of the relationship between VA differences and HbA1C, the decrease in mean VA difference as HbA1C increased (p = 0.05) was unexpected; however, the magnitude of the relationship was mild (−0.3 letters per 1% increase in HbA1c). The specific reasons why the mean difference in fellow eyes was about 2 letters larger than in study eyes (p < 0.001) was unclear. The fact that fellow eyes had no restrictions on the level of protocol VA, eye pathologies, or recent treatment may contribute to larger differences. Although we allowed clinical visits to be up to 3 months before the protocol visit, nearly 80% were within 6 weeks, and there was no apparent association between the VA differences and the time between visits (p = 0.73).The level of DR severity, CST, lens status, age, refractive error, and protocol also did not appear to influence the VA differences. We did not collect data on the physical environment of the testing for the clinical measurement, or on whether patients were required to provide reasons for uncertainty when attempting to identify a letter. Such “pushing” of the patient is an important feature of the ETDRS testing procedure and may contribute to better eETDRS letter scores. Also, the time of day for the measurements was not assessed and because some patients with DME report worse vision after a night in a recumbent position; differences in the time of testing may have contributed to the differences between measurements.20
The differences analyzed in this study do not reflect the differences between eETDRS and Snellen measurements tested under the same conditions (e.g., each tested after refraction). Instead, they show the discrepancies between protocol eETDRS and recent Snellen VA measured in routine practice at the retina clinic that recruited the participant for the protocol. Differences at follow-up protocol visits, where VA may influence treatment decisions, were not assessed. Still, the considerable variation in the recent clinical VA compared to the baseline protocol VA highlights the potential for follow-up VA assessments of patients outside of studies to have more variation than expected based on study results.
The findings of this study and previous reports on differences in VA measurements obtained using routine clinic VA testing versus measurements obtained using refracted ETDRS testing have implications for clinical decision making and setting patient expectations for results after treatment.9–12,21 For example, vision outcomes varied by baseline VA (ETDRS 20/40 or better vs. 20/50 or worse) in DRCR Protocol T for treatment of diabetic macular edema. In another example, the results from DRCR Protocol V support the initial observation of eyes with DME only to eyes with ETDRS 20/25. Using a routine clinical measurement of VA, especially one without refraction, may misclassify the patient and form a misdirected, potentially suboptimal treatment strategy. In addition, even when the optimal approach is applied, the vision results from these clinical trials apply to refracted-ETDRS measurements. An expected 2-year ETDRS VA of 20/20 may be measured in the clinic as 1 or 2 lines worse, disappointing both the patient and the ophthalmologist.
The ultimate purpose for VA measurements in a clinical setting can be different from that of protocol VA measurements in a study. While clinical VA measurements may help diagnose and assess treatment response, protocol VA measurements often determine a study’s inclusion/exclusion criteria and primary outcome. Change in VA over a study period is a commonly used metric to compare the efficacy of study treatments. Therefore, study VA measurements are standardized to maximize the precision of the VA measurement. As observed in this study, there is no such standardization in clinical practice, as shown broadly across these 74 clinical sites. This lack of standardization may contribute to imprecise or fluctuating VA measurements in clinical practice and a tendency for clinicians to depend less on VA measurements and more on anatomic measurements in decision-making for retreatment.
The retrospective nature of data collection limited the observations of this study. Participation in the collection of clinical VA on past medical records was not a part of the original protocol study designs, so there may be some ascertainment bias due to sites’ willingness to participate. This study did not include 14% (12 of 86) of sites from the original studies and did not obtain clinical VA for 30% (427 of 1443) of eyes with protocol VA. However, only 3% (40 of 1443) of eyes were excluded because they were at non-participating sites, and the mean (SD) of protocol VA in excluded eyes [78.3 (14.1) letters] resembled that of included eyes [76.3 (15.6) letters]. Additionally, the distributions of baseline participant and ocular characteristics between excluded and included eyes appeared mostly balanced, with the largest difference in time between the clinical and protocol visit, which was limited to 3 months in the included group (Table 2). Finally, when clinical visual acuity was reported as a Snellen acuity “+/− letters”, there was no accounting for this in the conversion to ETDRS letters, which may affect the difference of observed measurements.
When applying clinical trial-based guidelines in managing DR and DME, clinicians should be aware that Snellen VA, as measured in clinical care, is likely to be worse than that achieved after eETDRS protocol refraction and VA testing. We expect clinical Snellen measurements to be 1–2 lines lower than protocol eETDRS measurements on average. However, the difference between measurements for an individual patient may be substantial, particularly at poorer levels of vision. Selecting a treatment plan determined by baseline ETDRS should use a VA testing procedure that includes detailed refraction and mimics the ETDRS protocol testing approach whenever possible. The findings of this study may explain, at least in part, why clinical study results are not always reproduced in clinical practice. Lastly, these findings suggest that clinicians need better methods for accurate and reliable vision testing in standard clinical practice.
Supplementary Material
Table 7.
Final multivariable model estimates for VA difference (protocol – clinical)
| Final Multivariable Model Effectsa | Estimate | (95% CI) | P-value |
|---|---|---|---|
|
| |||
| Clinical VA, 1-line (5 letters converted from Snellen fraction) b | −1.3 | (−1.5, −1.1) | <0.001 |
| Hemoglobin A1c, % | −0.3 | (−0.6, −0.0) | 0.05 |
| Eye enrollment | <0.001 | ||
| Fellow Eye | 1.9 | (1.0, 2.8) | <0.001 |
| Study Eye (reference) | 0.0 | ||
| Clinical VA correction method | 0.06 | ||
| Refracted | −3.0 | (−5.9, −0.1) | 0.04 |
| Glasses or contacts | −1.8 | (−3.3, −0.3) | 0.02 |
| Pinhole only | −1.5 | (−3.5, 0.5) | 0.14 |
| Uncorrected (reference) | 0.0 | ||
The final multivariable model for VA difference (protocol - clinical) included N = 901 eyes with no missing covariate values in a linear mixed model with nested random effects on the intercept for correlation within sites and participants; VA difference outliers were truncated at the raw overall mean ± 3 SD (7.7 ± 3 × 10.3).
Converted letter score = 85 + 50 × log10(Snellen fraction), rounded to the nearest letter, or = 0 for Snellen fractions <20/800.
The intraclass correlation was 0.14 within sites and 0.38 within participants within sites. The conditional R2 (variance explained by fixed and random factors) was 0.52.
Funding Support:
Research reported in this publication was supported by the National Eye Institute and the National Institute of Diabetes and Digestive and Kidney Diseases of the National Institutes of Health under Award Number UG1EY014231. Additional funding For Protocol AE and W was received from the JDRF. PhotoOptx provided study devices for Protocol AE, Regeneron provided funding and study drug for Protocol W. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health. The DRCR Retina Network had complete control over the design of the protocol, ownership of the data, all editorial content of presentations and publications related to the protocol, and the decision to submit for publication
Role of the Sponsor:
The funding organization (National Institutes of Health) participated in oversight of the conduct of the study and review of the manuscript but not directly in the design or conduct of the study, nor in the collection, management, analysis, or interpretation of the data, or in the decision to submit for publication or the preparation of the manuscript.
Footnotes
Publisher's Disclaimer: This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting proof before it is published in its final form. Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.
Contributor Information
Carl W. Baker, Hilton Head Retina Institute.
Kristin Josic, Jaeb Center for Health Research, Northwestern University, Chicago, Illinois.
Maureen G. Maguire, Jaeb Center for Health Research, Northwestern University, Chicago, Illinois.
Lee M. Jampol, Feinberg School of Medicine, Northwestern University, Chicago, Illinois.
Daniel F. Martin, Cole Eye Institute, Cleveland Clinic, Cleveland, Ohio.
Soraya Rofagha, East Bay Retina Consultants, Oakland, California.
Jennifer K. Sun, Joslin Diabetes Center, Beetham Eye Institute, Harvard Department of Ophthalmology, Boston, Massachusetts.
References
- 1.Ferris FL 3rd, Kassoff A, Bresnick GH, Bailey I. New visual acuity charts for clinical research. Am J Ophthalmol. 1982;94:91–96. [PubMed] [Google Scholar]
- 2.Beck RW, Moke PS, Turpin AH, et al. A computerized method of visual acuity testing: adaptation of the Early Treatment of Diabetic Retinopathy Study testing protocol. Am J Ophthalmol. 2003;135(2):194–205. [DOI] [PubMed] [Google Scholar]
- 3.Blackhurst DW, Maguire MG. Reproducibility of refraction and visual acuity measurement under a standard protocol. The Macular Photocoagulation Study Group. Retina. 1989;9(3):163–169. [PubMed] [Google Scholar]
- 4.Comparison of Age-related Macular Degeration Trials (CATT) Group Research, Martin DF Maguire MG, et al. Ranibizumab and Bevacizumab for Treatment of Neovascular Age-related Macular Degeneration: Two-Year Results. Ophthalmology. 2012;119(7):1388–1398. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 5.Maturi RK, Glassman AR, Josic K, et al. Effect of Intravitreous Anti-Vascular Endothelial Growth Factor vs Sham Treatment for Prevention of Vision-Threatening Complications of Diabetic Retinopathy: The Protocol W Randomized Clinical Trial. JAMA Ophthalmol. 2021;139(7):701–712. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 6.Heier JS, Khanani AM, Quezada Ruiz C, et al. Efficacy, durability, and safety of intravitreal faricimab up to every 16 weeks for neovascular age-related macular degeneration (TENAYA and LUCERNE): two randomised, double-masked, phase 3, non-inferiority trials. Lancet. 2022;399(10326):729–740. [DOI] [PubMed] [Google Scholar]
- 7.Tsou BC, Bressler NM. Visual Acuity Reporting in Clinical Research Publications. JAMA Ophthalmol. 2017;135(6):651–653. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 8.Williams MA, Moutray TN, Jackson AJ. Uniformity of visual acuity measures in published studies. Invest Ophthalmol Vis Sci. 2008;49(10):4321–4327. [DOI] [PubMed] [Google Scholar]
- 9.Kaiser PK. Prospective evaluation of visual acuity assessment: a comparison of snellen versus ETDRS charts in clinical practice (An AOS Thesis). Trans Am Ophthalmol Soc. 2009;107:311–324. [PMC free article] [PubMed] [Google Scholar]
- 10.Yu HJ, Kaiser PK, Zamora D, et al. Visual Acuity Variability: Comparing Discrepancies between Snellen and ETDRS Measurements among Subjects Entering Prospective Trials. Ophthalmol Retina. 2021;5(3):224–233. [DOI] [PubMed] [Google Scholar]
- 11.Falkenstein IA, Cochran DE, Azen SP, et al. Comparison of visual acuity in macular degeneration patients measured with snellen and early treatment diabetic retinopathy study charts. Ophthalmology. 2008;115(2):319–323. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 12.Chen FK, Agelis LE, Peh KK, Teong J, Wong EN. Factors Contributing to Discrepancy Between Visual Acuity Fractions Derived From a Snellen Chart and Letter Scores on the Early Treatment Diabetic Retinopathy Study Chart. Asia Pac J Ophthalmol (Phila). 2014;3(5):277–285. [DOI] [PubMed] [Google Scholar]
- 13.Baker CW, Glassman AR, Beaulieu WT, et al. Effect of initial management with aflibercept vs laser photocoagulation vs observation on vision loss among patients with diabetic macular edema involving the center of the macula and good visual acuity: a randomized clinical trial. JAMA. 2019;321(19):1880–1894. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 14.Diabetic Retinopathy Clinical Research Network, Wells JA, Glassman AR, et al. Aflibercept, bevacizumab, or ranibizumab for diabetic macular edema. N Engl J Med. 2015;372(13):1193–1203. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 15.Glassman AR, Baker CW, Beaulieu WT, et al. Assessment of the DRCR Retina Network Approach to Management With Initial Observation for Eyes With Center-Involved Diabetic Macular Edema and Good Visual Acuity: A Secondary Analysis of a Randomized Clinical Trial. JAMA Ophthalmology. 2020;138(4):341–349. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 16.Wells JA, Glassman AR, Jampol LM, et al. Association of Baseline Visual Acuity and Retinal Thickness With 1-Year Efficacy of Aflibercept, Bevacizumab, and Ranibizumab for Diabetic Macular Edema. JAMA Ophthalmol. 2016;134(2):127–134. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 17.Heier JS, Bressler NM, Avery RL, et al. Comparison of Aflibercept, Bevacizumab, and Ranibizumab for Treatment of Diabetic Macular Edema: Extrapolation of Data to Clinical Practice. JAMA Ophthalmol. 2016;134(1):95–99. [DOI] [PubMed] [Google Scholar]
- 18.Gregori NZ, Feuer W, Rosenfeld PJ. Novel method for analyzing snellen visual acuity measurements. Retina. 2010;30(7):1046–1050. [DOI] [PubMed] [Google Scholar]
- 19.Bland JM, Altman DG. Measuring agreement in method comparison studies. Stat Methods Med Res. 1999;8(2):135–160. [DOI] [PubMed] [Google Scholar]
- 20.Larsen M, Wang M, Sander B. Overnight thickness variation in diabetic macular edema. Invest Ophthalmol Vis Sci. 2005;46(7):2313–2316. [DOI] [PubMed] [Google Scholar]
- 21.Schachat AP, Zarbin MA. Application of Clinical Trial Results to Clinical Practice: Some Reminders and Considerations. Ophthalmol Retina. 2021;5(3):221–223. [DOI] [PubMed] [Google Scholar]
- 22.Sun JK, Josic K, Melia M, et al. Conversion of Central Subfield Thickness Measurements of Diabetic Macular Edema Across Cirrus and Spectralis Optical Coherence Tomography Instruments. Transl Vis Sci Technol. 2021;10(14):34. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 23.Diabetic Retinopathy Clinical Research Network Writing Committee, Bressler SB, Edwards AR, et al. Reproducibility of spectral-domain optical coherence tomography retinal thickness measurements and conversion to equivalent time-domain metrics in diabetic macular edema. JAMA Ophthalmol. 2014;132(9):1113–1122. [DOI] [PMC free article] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.