Comparison of the Retinomax and Palm-AR Autorefractors: A Pilot Study

Abstract

Purpose

To compare the performance of two hand-held autorefractors, the Retinomax and the Palm-Automatic Refractometer (Palm-AR)^b, for detecting significant vision disorders in preschool children.

Methods

Children attending Philadelphia PreKindergarten Head Start were screened with the Retinomax and Palm-AR and underwent a gold standard eye examination. The results of cycloplegic retinoscopy, cover testing and visual acuity were used to classify children as having normal vision or one of four conditions: amblyopia, strabismus, significant refractive error, and reduced VA. Pass/fail criteria for each instrument were selected to maximize overall sensitivity (with specificity set at 90% and at 94%) for detecting targeted disorders. Comparisons of sensitivities between the autorefractors were performed using the exact McNemar test.

Results

Testability was greater than 99% for both instruments. Test time was similar for the two instruments (median 2 minutes; p-value = 0.10). At 90% specificity, the sensitivity for detection of one or more targeted conditions was 74% for the Palm-AR and 78% for the Retinomax. At 94% specificity, the sensitivity for detection of one or more targeted conditions was 66% for both the Palm-AR and the Retinomax. At 90% specificity, the sensitivity for detecting significant refractive error was 84% for both autorefractors, and at 94% specificity, the sensitivity was 76% for the Palm AR and 75% for the Retinomax. There were high correlations between the instruments for sphere (r = 0.85) and cylinder (r = 0.88) power. The mean difference between instruments was −.13 diopters (D) (95% limit of agreement: −2.28 to 2.02) for sphere, and −0.15 D (95% limit of agreement: −0.89 to 0.59) for cylinder.

Conclusions

In this pilot study, the Retinomax and Palm-AR appear comparable with respect to testability, sensitivity and specificity. There was strong agreement in readings of sphere and cylinder indicating that they may perform similarly in a screening setting.

The most prevalent childhood vision disorders are amblyopia, strabismus and significant refractive error including myopia, hyperopia, astigmatism and anisometropia.^1–5 Vision screening of children has been shown to be effective in detecting children at risk for having one or more of these disorders ^6–11. Hand-held autorefractors have been useful in a screening setting due to their portability, minimal space needed to operate, ease of use, high testability rates in young children and short administration time. ^12–29 An added advantage is the ability of non-eyecare professionals and lay screeners to use these instruments successfully. ^29,30 Comparisons of hand-held autorefractors with table mounted autorefractors and portable power refractors have also been reported in the literature.^31–36 The investigators of the Vision in Preschoolers Study and others have previously reported on comparisons of the hand-held Refractometer Retinomax (“Retinomax”; Nikon - Chiyoda-ku, Tokyo) and SureSight^™ Vision Screener (“SureSight”; WelchAllyn - Skaneateles Falls, NY) for detecting vision disorders in preschool children. ^32–34 A third autorefractor, the Palm-Automatic Refractometer (Palm-AR; Marco – Jacksonville, FL), is commercially available and offers many of the same features; however, there have been no reports published regarding its performance in preschool vision screening. Therefore, we undertook a study to provide initial data on the characteristics of the Palm-AR as a screening instrument in comparison with the Retinomax.

METHODS

Subjects

Children who were enrolled in Philadelphia area Pre-Kindergarten Head Start programs and participating in the Vision in Preschoolers Study Phase II were included in this pilot study. (n = 190). All children were between the ages of three and five years as of September 1^st of the academic year during which the children were tested. The Institutional Review Board of the Pennsylvania College of Optometry approved the study protocol. Written, informed consent was obtained from the parent or legal guardian of all eligible study participants.

Screening Procedure

Each participant received a vision screening using both the Retinomax and Palm-AR on one day followed by a standardized comprehensive eye examination on a subsequent day. Screenings took place at the Head Start Center attended by the child in a room designated for the vision screening. Both the Retinomax and the Palm-AR were administered with the order of screening randomly assigned for each child. The currently available Palm-ARK has the same properties as the Palm-AR that was used in this study. The actual instrument looks the same with the only difference being the Palm-ARK also provides keratometry readings.^a The currently available Retinomax has the same properties as the Retinomax used in this study. An alternative model of the Retinomax, the “Retinomax K plus” is also available that has the ability to record keratometry readings and has the same properties as the Retinomax that was used in this study.^b A trained and certified (non-eyecare professional) screener (ACarter) administered the screening tests to all children. Both autorefractions were completed at the same testing session.

Each time the Retinomax is administered, eight, automatic and consecutive readings are taken for each eye. These eight readings are averaged into one overall final reading along with a confidence level ranging from 1 to 10 for each eye. The manufacturer’s recommended confidence level is 8 to 10.³⁹ A maximum of three administrations were allowed for each eye to obtain an acceptable confidence level.^6,30,43

The Palm-AR provides up to five automatic consecutive readings for each eye with a confidence level for each of the readings. The confidence levels range from 5 to 9. One summary refractive error reading is provided by the instrument, without a summary confidence level. Summary readings for the Palm-AR were considered acceptable for the purposes of this study when all confidence levels associated with individual readings were 7 or greater and the majority of readings were 8 or greater (as recommended by the Marco representative at the time the study was conducted). A maximum of three administrations were allowed for each eye to obtain an acceptable summary reading.

Gold Standard Eye Examinations

Standardized Gold Standard Eye (GSE) examinations were performed by optometrists or ophthalmologists who were experienced working with children.^6,30 GSE examinations were conducted in a specially equipped vision van parked at each Head Start Center. The tests that were used to determine the presence of significant vision disorders included cycloplegic retinoscopy, distance and near cover test and monocular threshold visual acuity (VA) using crowded HOTV optotypes on the Electronic Vision Assessment (EVA) system.⁴⁰

Based upon the results of the GSE examinations, children were classified as having one or more of the four types of VIP targeted vision disorders (amblyopia, strabismus, significant refractive error (RE), and reduced VA) or normal vision. Significant refractive error included astigmatism, hyperopia, myopia and anisometropia. Children with more than one disorder (e.g., amblyopia and strabismus) would be included in each category. Vision disorders were also categorized by severity into Group 1: Very important to detect and treat early; Group 2: Important to detect early and Group 3: Detection clinically useful. Whenever children had more than one condition, they were assigned only to the Group corresponding to their most severe condition. A full description of the hierarchy of these targeted vision disorders has been previously reported, and their definitions are summarized in Table 1.⁶

Table 1.

Definition of VIP Targeted Disorders.

Group 1: Very important to detect and treat early

Amblyopia Presumed Unilateral: ≥ 3 line interocular difference, a unilateral amblyogenic factor, and worse eye VA ≤ 20/64 Suspected Bilateral: a bilateral amblyogenic factor, worse eye VA < 20/50 for 3 year olds or < 20/40 for 4 year olds, contralateral eye VA worse than 20/40 for 3 year olds or 20/30 for 4 year olds Strabismus: Constant in primary gaze Refractive Error Severe anisometropia (Interocular difference > 2D hyperopia, > 3D astigmatism, or > 6D myopia) Hyperopia ≥ 5.0 D Astigmatism ≥ 2.5 D Myopia ≥ 6.0 D

Group 2: Important to detect early Amblyopia Suspected Unilateral: 2-line interocular difference and a unilateral amblyogenic factor Presumed Unilateral: ≥ 3 line interocular difference, a unilateral amblyogenic factor, and worse eye VA > 20/64 Strabismus: Intermittent in primary gaze Refractive Error Anisometropia, (Interocular difference > 1D hyperopia, > 1.5D astigmatism, or > 3D myopia)† Hyperopia > 3.25 D and < 5.0 D AND interocular difference in SE ≥ 0.5 D Astigmatism > 1.5 D and < 2.5 D Myopia ≥ 4.0 D and < 6.0 D

Group 3: Detection clinically useful Unexplained Reduced VA Bilateral: no bilateral amblyogenic factor, worse eye VA < 20/50 for 3 year olds or < 20/40 for 4 year olds, contralateral eye VA worse than 20/40 for 3 year olds or 20/30 for 4 year olds Unilateral: no unilateral amblyogenic factor, worse eye VA < 20/50 for 3 year olds or < 20/40 for 4 year olds or ≥ 2 line difference between eyes (except 20/16 and 20/25) Refractive Error Hyperopia > 3.25 D and < 5.0 D AND interocular difference in SE < 0.5 D Myopia > 2.0 D and < 4.0 D

^*Modified from Tables 2 and 3 of VIP Study Group.⁶

^†Clarification of previous description of “Anisometropia, but not severe.” ^{6, 30}

Statistical Analysis

The testing characteristics of the Retinomax and Palm-AR were compared, including testability, testing time and the ability of each instrument to detect VIP targeted vision disorders. The Exact McNemar test was performed for comparison of the percentage of children testable, and the Wilcoxon signed-rank test was used for the comparison of testing time due to the skewed distribution of testing time. For comparison of testing accuracy (sensitivity at pre-determined specificity levels), the screening failure criteria that resulted in 90% and 94% specificity were determined for each autorefractor. Children who were not testable were considered as screening failures.⁴¹ The first reading from an eye with an acceptable confidence level was used for data analysis and the best reading was used for eyes with no acceptable confidence levels from three attempted consecutive tests. For each set of failure criteria, the sensitivities for detecting each of VIP-targeted conditions, any GSE condition, and the three hierarchical grouped conditions were calculated. The statistical comparison of sensitivities between the Retinomax and the Palm-AR at the same level of specificity (either 90% or 94%) was performed by the Exact McNemar test for correlated proportions, because the Retinomax and the Palm-AR were performed on the same child and the number of children with a VIP-targeted condition is small. Agreement on refractive error was assessed with Bland-Altman plots and 95% limits of agreement.⁴²

RESULTS

Of the 190 children who were initially enrolled and screened, 181 (95%) received a GSE examination. The remaining nine children were absent on the day of the GSE examination and were not able to be rescheduled for reasons such as illness or family relocation. Among the 181 with GSE examinations, 99 (55%) of children were 4 years old, with a mean age of 4.3 (range 3 to 5 years) and 94 (52%) were male.

Testability was greater than 99% for each instrument (Table 2; p=0.32). Acceptable confidence levels were obtained in 98.7% of eyes with the Retinomax and 89.4% of eyes with the Palm-AR (p< 0.0001). Overall, the Retinomax required less repeat testing and had a higher percentage of eyes with an acceptable confidence level after up to 3 tests (p<0.0001).

Table 2.

Testability of the Autorefractors.

	Palm-AR (N=380 eyes)	Retinomax (n=380 eyes)	P-value
Testability	n (%)	n (%)
Not testable*	3 (0.8)	1 (0.3)	0.32
Testable	377 (99.2)	379 (99.7)
Number of Tests Administered			<0.0001
0	3 (0.8)	1 (0.3)
1	270 (71.1)	320 (84.2)
2	46 (12.1)	46 (12.1)
3	61 (16.1)	13 (3.4)
Acceptable Confidence Level Obtained†			<0.0001
Yes	337 (89.4)	374 (98.7)
No	40 (10.6)	5 (1.3)

^*No reading obtained in 3 attempts.

^†Among the eyes with readings available

The time to complete testing for each child (Table 3) was also evaluated. Among the 145 children who had the testing time recorded for each autorefractor, the time ranged from less than one minute to ten minutes. There was no difference in the median testing time (2 minutes each) between the two instruments (p-value = 0.11, Wilcoxon signed-rank test).

Table 3.

Comparison of time to obtain readings.

Time (minutes)	Palm-AR (n=145)*	Retinomax (n=145)*	p-value†
	n (%)	n (%)
<1	9 (6.2)	1 (0.7)
1	55 (37.9)	70 (48.3
2	39 (26.9)	53 (36.6)
3	27 (18.6)	9 (6.2)
4	6 (4.1)	6 (4.1)
5	4 (2.8)	3 (2.1)
6	3 (2.1)	2 (1.4)
7–10	2 (1.4)	1 (0.7)
Median	2	2	0.11

^*The measurement time for 45 children was not recorded.

^†From the Wilcoxon signed-rank test

The screening failure criteria for each instrument (set to achieve maximum sensitivity to detect any of the targeted conditions when specificity was set at 90% or 94%) are provided in Table 4. Increasing the specificity from 90% to 94% for the Palm-AR required increasing the failure criterion for hyperopia by 0.50 diopter (D), from 1.00 D to 1.50 D. For the Retinomax, increasing the specificity resulted in increasing the criterion by 0.25 D for myopia, increasing the criterion by 1.25 D for astigmatism, and decreasing the criterion by 0.25 D for anisometropia.

Table 4.

Failure Criteria for the Palm-AR and Retinomax Autorefractors Screening Tests to Maximize Sensitivity with Specificity Set at 90% and 94%.

Instrument	Hyperopia	Myopia	Astigmatism	Anisometropia
Palm-AR
90% specificity	≥1.00 D	≥3.75D	≥2.00 D	≥2.50 D
94% specificity	≥1.50 D	≥3.75D	≥2.00 D	≥2.50 D
Retinomax
90% specificity	≥1.25 D	≥3.25D	≥1.25 D	≥1.75 D
94% specificity	≥1.25 D	≥3.50D	≥2.50 D	≥1.50 D

Sixty-five (35.9%) of the 181 children were determined to have one or more targeted vision conditions (Table 5) resulting in a total of 99 vision disorders among this subset of children. The sensitivities for detecting any of the targeted conditions were similar for the Palm-AR and the Retinomax (74% and 78%, respectively) at 90% specificity and were the same (66%) at 94% specificity. Sensitivities for detecting refractive error were very similar between the two auto-refractors at both levels of specificity (Table 5). Comparisons of sensitivity between the auto-refractors for the other subgroups of conditions are hampered because of the small sample sizes (<30) and resulting low precision.

Table 5.

Comparison of sensitivity of detecting hierarchical & four targeted conditions.

Sensitivity	90% Specificity				94% Specificity
	Palm-AR	Retino-max	Difference (95% C.I.)	p-value†	Palm-AR	Retino-max	Difference (95% C.I.)	p-value†
Conditions by Severity
Overall* (n=65)	0.74	0.78	0.05 (−0.07, 0.16)	0.58	0.66	0.66	0.00 (−0.11, 0.11)	1.00
Group 1 (n=28)	0.79	0.93	0.14 (−0.01, 0.31)	0.13	0.71	0.82	0.11 (−0.06, 0.28)	0.38
Group 2 (n=22)	0.77	0.64	−0.14 (−0.33, 0.07)	0.38	0.64	0.50	−0.14 (−0.32, 0.07)	0.38
Group 3 (n=15)	0.60	0.73	0.13 (−0.14, 0.38)	0.63	0.60	0.60	0.00 (−0.21, 0.21)	1.00
Type of Condition
Amblyopia (n=24)	0.75	0.88	0.13 (−0.04, 0.30)	0.25	0.67	0.83	0.17 (−0.01, 0.34)	0.13
Strabismus (n=10)	0.70	0.70	0.00 (−0.20, 0.20)	1.00	0.60	0.60	0.00 (−0.18, 0.18)	1.00
Refractive error (n=55)	0.84	0.84	0.00 (−0.12, 0.12)	1.00	0.76	0.75	−0.02 (−0.14, 0.11)	1.00
Reduced VA (n=10)	0.30	0.70	0.40 (−0.04, 0.68)	0.22	0.30	0.30	0.00 (−0.36, 0.36)	1.00

^*Includes the presence of any GSE targeted condition

^†From Exact McNemar test

When the 90% specificity failure criteria previously derived for lay screeners are applied to the Retinomax screening results from the children included in this study^30, the specificity is 88% and the sensitivity is 71% for any VIP targeted disorders, 86% for Group 1 conditions, 55% for Group 2, 67% for Group 3 conditions, 83% for amblyopia, 60% for strabismus, 78% for refractive error, and 50% for reduced VA.

The Bland-Altman plots in Figure 1 and 2 show the agreement between the two autorefractors for sphere and cylinder power values. Overall the correlation coefficient for both sphere and cylinder between the instruments was high (Pearson correlation coefficient = 0.85 for sphere and 0.88 for cylinder). The difference between the instruments is −0.13 D (95% limit of agreement: −2.28 to 2.02 D) for sphere, and −0.15 (95% limit of agreement: −0.89 to 0.59) for cylinder.

Figure 1.

Bland-Altman plot and 95% limit of agreement for sphere between Retinomax and Palm-AR. X-axis shows the mean sphere from Retinomax and Palm-AR for each eye. Y-axis shows the difference of sphere between Retinomax and Palm-AR (Retinomax – Palm-AR) for each eye. The middle dashed line shows the mean difference of sphere between Retinomax and Palm-AR, and the dashed lines above and below the middle dashed line are its 95% limits of agreement.

Figure 2.

Bland-Altman plot and 95% limit of agreement for cylinder between Retinomax and Palm-AR. X-axis shows the mean cylinder from Retinomax and Palm-AR for each eye. Y-axis shows the difference of cylinder between Retinomax and Palm-AR (Retinomax – PalmAR) for each eye. The middle dashed line shows the mean difference of cylinder between Retinomax and Palm-AR, and the dashed lines above and below the middle dashed line are its 95% limits of agreement.

DISCUSSION

The results of this study indicate that the Palm-AR and the Retinomax may both be useful instruments in a screening setting for preschool children. The two instruments appear comparable with respect to administration time, testability, sensitivity and specificity and provide similar measurements of sphere and cylinder power. The children in this study were a preselected group of preschool children at a high risk for vision disorders.

Vision screening with the Palm-AR required more repeat testing than with the Retinomax. Also, the Palm-AR did not yield a reading with a confidence level deemed acceptable by the manufacturer as often as the Retinomax. Nonetheless, the accuracy for detecting any targeted condition was similar between the two autorefractors. The refractive error components were highly correlated (correlation coefficient ≥ 0.85) between the two instruments, supporting the similarity in screening accuracy. However, the sample size for this pilot assessment of the PalmAR did not provide sufficient precision to rule out important differences between the two instruments, particularly for detecting children at risk for less prevalent vision disorders such as amblyopia. Further investigation with larger numbers of subjects is needed to precisely estimate the accuracy of the Palm-AR.

This study demonstrated that the Retinomax and Palm-AR autorefractors appear to be comparable with similar costs in a screening setting in this small sample size study. Future studies focusing on a more general population, larger sample size and older children can provide further insights into the advantages of each instrument.

1. The Palm-AR has been replaced by the Palm-ARK (http://www.marco.com/automated/4.html)

2. The Retinomax has been replaced by the Retinomax Series 3. (http://www.rightmedical.com/PRODUCT%20PDFs/RETINOMAX.pdf)

Appendix: The Vision in Preschoolers Study Group

VIP Writing Committee

Elise Ciner, OD (Chair), Ashanti Carter, Gui-Shuang Ying, PhD, Maureen Maguire, PhD, and Marjean Taylor Kulp, OD, MS.

Elise Ciner – The Pennsylvania College of Optometry at Salus University

Ashanti Carter – (she is no longer at Salus – but was a research intern and student here)

Gui-Shuang Ying, University of Pennsylvania, Department of Ophthalmology

Maureen Maguire, University of Pennsylvania, Department of Ophthalmology

Marjean Taylor Kulp, The Ohio State University College of Optometry

VIP Executive Committee

Paulette Schmidt, OD, MS (Chair), Agnieshka Baumritter, MA, Elise Ciner, OD, Lynn Cyert PhD, OD, Velma Dobson, PhD, Marjean Taylor Kulp, OD, MS, Maureen Maguire, PhD, Bruce Moore, OD, Deborah Orel-Bixler, PhD, OD, Graham Quinn, MD, Maryann Redford, DDS, MPH, Janet Schultz, MA, CPNP, and Gui-Shuang Ying, PhD

VIP Personnel at PCO

Principal Investigator: Elise Ciner, OD

Project Coordinator: Angela Duson

Asst. Project Coordinator: Lydia Parke, MS

Nurse Coordinator: Lynn Rosenbach, RN

Pediatric Eye Care Practitioners: Mark Boas, OD, Shannon Burgess, OD, Penelope Copenhaver, OD, Michael Gallaway, OD, Graham Quinn, MD, Janet Schwartz, OD, Brandy Scombordi-Raghu, OD, Edward Zikoski, OD, and Sheryl Menacker, MD

Research Intern: Ashanti Carter

Parent Liaison: Geneva Moss

Van Drivers: Jose Figueroa, David Harvey