Evaluation of the Spot Vision Screener in School-Aged Children

Abstract

Purpose:

To determine the accuracy of the Spot Vision Screener (Welch Allyn, Skaneateles Falls, NY) in children 6 years and older and recommend device thresholds to improve its accuracy for the detection of refractive error.

Methods:

The Spot Vision Screener results were compared with three gold standard conditions of increasingly narrow refractive error criterion. The sensitivity, specificity, positive predictive value, and negative predictive value of the Spot Vision Screener in detecting each gold standard criterion were calculated. The most accurate threshold setting for each parameter was identified by calculating the area under the curve receiver operating characteristic.

Results:

The Spot Vision Screener was able to successfully evaluate 313 of 330 children (95%). The sensitivity of the Spot Vision Screener to detect American Association for Pediatric Ophthalmology and Strabismus guidelines for amblyopia risk factors was 89.5% and the specificity was 76.7%. The sensitivity decreased to 80% and the specificity increased to 75.3% with narrower refractive criteria. The sensitivity in detecting refractive criteria improved with the proposed optimized device thresholds. Estimates for the general population indicate that the positive predictive value is reasonable at 52.3% to 61.8%, depending on the stringency of the criteria, with excellent negative predictive values.

Conclusions:

In school-aged children, the primary screening focus shifts from preventing amblyopia to detecting visual disturbances, including refractive error, that may interfere with academic performance. In this age group, the Spot Vision Screener was an acceptable method of detecting significant refractive error with improved sensitivity with threshold optimization.

INTRODUCTION

In 2016, the American Academy of Pediatrics and the National Center for Children’s Vision and Eye Health recommended instrument-based vision screening for children up to 6 years old and as an alternative when visual acuity cannot be obtained.^1–3 The devices provide quick screening results, with minimal effort on the part of the tester or the child. Multiple studies have reported good efficacy of screening devices in detecting amblyopia risk factors in children.^4–8 The National Center for Children’s Vision and Eye Health notes that “this age range may expand as high quality, peer reviewed, published research emerges.”⁹

The Spot Vision Screener (Welch Allyn, Skaneateles Falls, NY) is marketed for use in patients aged 6 months and older.¹⁰ In school-aged children, the focus of primary screening shifts from preventing amblyopia to detecting visual disturbances, including refractive errors that may interfere with academic performance. We sought to determine the accuracy of the Spot Vision Screener in children 6 years and older in our population of pediatric ophthalmology patients. Results of the Spot Vision Screener were compared with three gold standard conditions of increasingly narrow refractive error criteria, as found by comprehensive examination.

PATIENTS AND METHODS

The protocol was approved by the institutional review board at the Medical University of South Carolina for Human Research and informed consent was obtained from the patients’ parents and/or guardians prior to enrollment. The protocol for this continuing prospective study has been previously reported.^6,8,11,12 Patients between 6 and 16 years old who presented for a complete pediatric ophthalmological examination when study personnel were available between June 2012 and March 2017 were included in the study. Children were screened with the Spot Vision Screener by lay personnel prior to the ophthalmic examination. The Spot Vision Screener software v.2.1.4 with installed manufacturer “out-of-the-box” cutoffs was used.¹³

The Spot Vision Screener provides a report of pupillary diameter, ocular alignment, estimated binocular refraction, and referral recommendation: “all measurements within range” or “complete eye exam[ination] recommended.” Several attempts were made to obtain a successful reading. Patients for whom the screening was not able to be completed (ie, no refraction estimate or recommendation) were included in the study as automatic referrals. A comprehensive ophthalmic examination was performed by a pediatric ophthalmologist masked to the screener results. All patients had an evaluation of vision, stereopsis, motility, alignment, and the anterior segment. Patients underwent cycloplegic refraction and a fundus examination 30 to 40 minutes after instillation of proparacaine hydrochloride ophthalmic solution 0.5% USP followed by one or two drops of tropicamide 1%, phenylephrine 2.5%, and cyclopentolate 1%. The ages and demographics of the patients were collected. The following data were collected from the Spot Vision Screener: whether a screening examination was successfully completed, the estimated refraction, and the referral recommendation. The following data were collected from the physician examination: motility, alignment, cycloplegic refraction, and presence of systemic or ocular pathology.

The primary goal of this study was to evaluate the predictive ability of the Spot Vision Screener to correctly identify significant refractive errors and strabismus in our study population. Results of the Spot Vision Screener were compared with three “gold standard” criteria based on the results of the comprehensive examination (Table 1). Initial analysis (gold standard 1) was done using the 2013 guidelines of the American Association for Pediatric Ophthalmology and Strabismus (AAPOS) amblyopia risk factors as defined for the oldest age group (<48 months of age).¹⁴

TABLE 1.

Refractive Thresholds Used for the Three Gold Standards Used for Evaluation^a

Gold Standard	Hyperopia (D)	Myopia (D)	Astigmatism (D)	Anisometropia (D)
1: AAPOS > 48 months	> +3.50	> −1.50	> +1.50	> +1.50
2: Spot Vision Screener refractive criteria age (range: 72 to 240 months)	≥ +2.50	≥ −1.00	≥ +1.50	≥ +1.00
3: Stricter criteria	≥ +2.00	≥ −0.75	≥ +1.00	≥ +1.00

AAPOS = American Association for Pediatric Ophthalmology and Strabismus; D = diopters

^aNon-refractive errors of media opacity and manifest strabismus greater than 8 prism diopters were included for all three analyses.

The Spot Vision Screener is manufactured by Welch Allyn, Skaneateles Falls, NY.

Because the focus of screening school-aged children shifts from the detection of amblyopia risk factors to the detection of eye conditions such as refractive error that may impact academic performance,¹⁵ two additional analyses were performed with more narrow criteria. A second analysis (gold standard 2) was performed using the Spot Vision Screener manufacturer’s criteria for referral for the age group between 72 and 240 months as detected on the physician examination. A third analysis (gold standard 3) was performed using the Spot Vision Screener manufacturer’s criteria, except the threshold for astigmatism was narrowed to +1.00 diopters (D) or greater, myopia was changed to −0.75 D or greater, and hyperopia was changed to +2.00 D or greater. The amblyopia risk factors non-refractive criteria of significant media opacity and strabismus greater than 8 PD were included in analyses of the predictive ability of the Spot Vision Screener for the increasingly stringent gold standards.

Sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV) of the Spot Vision Screener in detecting each gold standard criterion in our population were calculated. Because PPV and NPV vary with disease prevalence,^16,17 these values were also estimated, assuming a fixed sensitivity and specificity, for a general population with an estimated disease prevalence of 24%.¹⁸

Calculation of the area under the receiver operating characteristic (ROC) curve provides a visual and statistical estimation of the predictive accuracy of a test. Because the Spot Vision Screener software can be manipulated by the user to adjust the thresholds for referral for each refractive parameter, we sought to identify the most accurate threshold settings for each parameter and for all four refractive parameters in this age group.

The associations between the gold standard hyperopia or myopia diagnoses and the Spot Vision Screener estimated spherical equivalent, the gold standard astigmatism diagnoses and the Spot Vision Screener estimated astigmatism, and the gold standard anisometropia diagnoses and anisometropia calculated from Spot Vision Screener estimates were evaluated using a logistic regression approach. This process was used for each of the three increasingly narrow gold standards. The prediction performance for the Spot Vision Screener refractive parameter used to predict each diagnosis was estimated using the area under the ROC curve. Additionally, we identified the best threshold for each refractive parameter to detect each diagnosis by determining the cutoff that maximized the chi-square statistic for the logistic regression model for each outcome regressed on the dichotomized Spot Vision Screener parameter.¹⁹ Sensitivity and specificity of the selected Spot Vision Screener parameter dichotomized at the selected threshold were evaluated for the different diagnoses. We also considered combinations of thresholds for the four parameters to identify the joint combination that provided the best cutoff for discriminating whether a child had a significant refractive error (ie, had at least one of the four refractive conditions) to determine if adjusting the Spot Vision Screener criterion improved performance on the Spot Vision Screener test. All analyses were conducted in R v.3.5.2. for Windows software (Microsoft Corporation, Redmond, WA).

Results

Results of the Spot Vision Screener and ophthalmologist examinations were collected for 330 sequential, eligible children between 72 and 221 months old (mean ± standard deviation [SD]: 103 ± 28 months) (Table 2). The Spot Vision Screener successfully evaluated 313 of 330 children (95%) in the current study. The Spot Vision Screener was unable to obtain a reading for 17 children: 3 were reported by the Spot Vision Screener as having “pupils [that were] too small,” 10 were reported as “machine could not read,” despite repeated attempts and adjustments, and 4 were deemed “uncooperative” by the tester.

TABLE 2.

Patient Demographics

Characteristic	Value
Age, mean ± SD (months)	103 ± 28 (range: 72 to 221)
Male gender, no. (%)	166 (50%)
Ethnicity, no.
White	122
African American	82
Hispanic	81
Other	45

SD = standard deviation

Eleven children were found to have constant strabismus of greater than 8 PD on the physician examination. Of these, 6 children had esotropia, 4 had exotropia, and 1 had hypertropia. All but one child with Duane syndrome and exotropia in primary gaze were referred by the Spot Vision Screener.

Prediction performance of the Spot Vision Screener for the three gold standards and estimated PPV and NPV are presented in Table 3. The prevalence of 2013 AAPOS 48-month guidelines of amblyopia risk factors (gold standard 1) was found to be 42% in this patient population. The sensitivity of the Spot Vision Screener to detect AAPOS amblyopia risk factors was 89.5% and the specificity was 76.7%. The PPV was 73.9% and the NPV was 90.8%. As disease prevalence decreases, assuming a fixed test sensitivity and specificity, the PPV is expected to decrease and the NPV is expected to increase.^16,17 The prevalence of amblyopia risk factors in our population of pediatric ophthalmology patients (42%) is higher than that estimated for the general population. For a general population with an estimated disease prevalence of 24%, the PPV is estimated to be 54.8% and the NPV is estimated to be 95.8%. Twelve children were not referred by the Spot Vision Screener but were found to have amblyopia risk factors on examination. Of the 12 children, 6 demonstrated hyperopia (+3.75 to 6.50 D), 3 demonstrated astigmatism (+1.75 to 2.25 D), 2 demonstrated anisometropia (1.80 to 2.00 D), and 1 demonstrated myopia (−2.25 D).

TABLE 3.

Spot Vision Screener Predictability Parameters for Gold Standard Criteria^a

Gold Standard	Sensitivity	Specificity	PPV	NPV	Est PPV	Est NPV	Prevalence
1	89.5 (84.3 to 94.7)	76.7 (70.5 to 82.9)	73.9 (67.1 to 80.7)	90.8 (86.2 to 95.4)	54.8	95.8	42%
2	80.0 (74.1 to 85.9)	84.8 (78.8 to 90.8)	87.0 (81.8 to 92.2)	77.0 (70.3 to 83.7)	62.4	93.1	56%
3	75.3 (69.2 to 81.4)	86.1 (80 to 92.2)	89.4 (84.6 to 94.2)	69.1 (61.8 to 76.4)	63.1	91.7	61%

PPV = positive predictive value; NPV = negative predictive value; Est = estimated

^aSensitivity, specificity, PPV, and NPV in the study population with 95% confidence intervals. PPV and NPV as estimated for the general population (as calculated values, confidence intervals are not available) and disease prevalence for each gold standard criterion are provided. Refractive criteria as described in Table 1. Media opacity and strabismus are included as referral criteria.

The Spot Vision Screener is manufactured by Welch Allyn, Skaneateles Falls, NY.

Sensitivity decreased with the narrower criteria to 80% with gold standard 2 and 75.3% with gold standard 3, with increasing specificity. For gold standards 2 and 3, 35 and 47 children, respectively, who met the criteria were passed by the Spot Vision Screener. Half of these children demonstrated hyperopia (+2.00 to +6.50 D). As estimated for the general population, the PPV remains slightly above half, suggesting a small majority of children referred will have the defined refractive error. The estimated NPV remains excellent, suggesting only a small percentage of children not referred will have significant refractive error.

The ROC curves for the appropriate refractive Spot Vision Screener parameters for each diagnosis (hyperopia, myopia, astigmatism, and anisometropia) as determined by the three different metrics (gold standards 1, 2, and 3) are shown in Figure 1. Figure 1 also shows that the ROC curve has at least one positive diagnosis for a multiple logistic regression model, including hyperopia, myopia, astigmatism, and anisometropia as predictors. The area under the curve is greatest when the outcome is set by gold standard 1 and AAPOS amblyopia risk factor criterion, although the areas under the curve are similar across all three standards with the only exception being myopia, for which the area under the curve is notably greater using the amblyopia risk factor gold standard 1.

Figure 1.

Receiver operating characteristic curves for detection of hyperopia, myopia, astigmatism, and anisometropia based on spherical equivalents or cylinder returned from the Spot Vision Screener (Welch Allyn, Skaneateles Falls, NY) relative to three gold standards. Gold standard 1 is the determination of these conditions according the 2013 amblyopia risk factors criterion. Gold standards 2 and 3 represent progressively more strict rules that determine the conditions (ie, more children will be categorized as having the condition). AUC = area under the curve

Table 4 shows the performance characteristics for the best individual cutoffs for the appropriate Spot Vision Screener parameter for each outcome as determined with the three gold standards. The recommended cutoffs for Spot Vision Screener detection of hyperopia for gold standards 1, 2, and 3 was a sphere of 0.50 D or greater for all three gold standards. The recommended cutoffs for Spot Vision Screener detection of myopia for gold standards 1, 2, and 3 were a sphere of −1.50 D or less, −1.00 D or less, and −0.75 D or less, respectively. The recommended cutoffs for the Spot Vision Screener detection of astigmatism for gold standards 1, 2, and 3 were a cylinder of 1.75 D or greater, 1.75 D or greater, and 1.25 D or greater, respectively. The recommended cutoffs for Spot Vision Screener detection of anisometropia for gold standards 1, 2, and 3 were a spherical equivalent of 1.375 D or greater, 0.75 D or greater, and 0.75 D or greater, respectively.

TABLE 4.

Performance Characteristics and 95% Confidence Intervals for the Spot Vision Screener^a

Characteristic	Hyperopia (D)	Myopia (D)	Astigmatism (D)	Anisometropia (D)	Combined
Gold standard 1	> 3.50	< −1.50	> 1.50	> 1.50	At least 1 criterion met
Recommended Spot Vision Screener cutoff	≥ 0.50	≤ −1.50	≥ 1.75	≥ 1.375
Sensitivity	75.7 (61.9 to 89.5)	90.4 (82.4 to 98.4)	86.1 (78.1 to 94.1)	79.5 (67.6 to 91.5)	90.3 (85.5 to 95.2)
Specificity	80.9 (76.4 to 85.4)	87.4 (83.5 to 91.3)	83.7 (79.2 to 88.2)	90.9 (87.6 to 94.2)	71.9 (65.4 to 78.4)
Gold standard 2	> 2.50	< −1.00	> 1.50	> 1.00	At least 1 criterion met
Recommended Spot Vision Screener cutoff	≥ 0.50	≤ −1.00	≥ 1.75	≥ 0.75
Sensitivity	63.1 (51.3 to 74.8)	84.3 (76.7 to 91.8)	82.0 (74.0 to 90.0)	76.9 (66.7 to 87.2)	86.7 (81.8 to 91.6)
Specificity	83.8 (79.3 to 88.2)	81.3 (76.4 to 86.2)	87.1 (82.9 to 91.4)	81.5 (76.8 to 86.2)	71.1 (63.7 to 78.6)
Gold standard 3	> 2.00	< −0.75	> 1.00	> 1.00	At least 1 criterion met
Recommended Spot Vision Screener cutoff	≥ 0.50	≤ −0.75	≥ 1.25	≥ 0.75
Sensitivity	60.5 (49.5 to 71.5)	90.7 (84.9 to 96.5)	84.7 (78.3 to 91.0)	76.9 (66.7 to 87.2)	91.2 (87.3 to 95.1)
Specificity	85.0 (80.7 to 89.4)	74.2 (68.6 to 79.9)	83.0 (77.9 to 88.1)	81.5 (76.8 to 86.2)	65.1 (56.8 to 73.4)

^aPerformance characteristics and 95% confidence intervals for each outcome of the Spot Vision Screener as defined by gold standards 1, 2, and 3, based on the cutoffs determined from the receiver operating characteristic curves.

^bThe children for whom a Spot Vision Screener reading was not obtained were treated as needing a referral for all receiver operating characteristic curves and estimated sensitivities and specificities. The combined test was based on meeting at least one or more of the criteria for the four conditions presented in the table.

The Spot Vision Screener is manufactured by Welch Allyn, Skaneateles Falls, NY.

We also identified the optimal joint combinations of cutoffs for the four parameters for discriminating whether a child needed further evaluation (ie, had at least one of the four conditions) to determine if adjusting the Spot Vision Screener criterion improved performance of the screening test. Here, gold standards 1, 2, and 3 were defined as being positive if a child had any one of the four conditions. For gold standard 1, the selected cutoffs were: a minimum sphere of −1.75 D or less, a maximum sphere of 0.75 D or greater, spherical equivalent of 0.75 D or greater, and a cylinder of 1.75 D or greater. For gold standard 2, the selected cutoffs were: a minimum sphere of −1.15 D or less, a maximum sphere of 0.75 D or greater, spherical equivalent of 0.75 D or greater, and a cylinder of 1.25 D or greater. For gold standard 3, the selected cutoffs were: a minimum sphere of −1.15 D or less, a maximum sphere of 1.00 D or greater, spherical equivalent of 0.625 D or greater, and a cylinder of 1.25 D or greater. In all cases, a child was classified as needing further evaluation regardless of whether one or more of the four criteria were met or whether the Spot Vision Screener was unable to obtain a reading (in keeping with current Spot Vision Screener manufacturer’s recommendations).

Table 5 shows the sensitivity and specificity for each gold standard based on the current Spot Vision Screener manufacturer’s “out-of-the-box” recommendations and the optimized cutoffs discussed previously. Gold standard refractive criteria are described in Table 1; however, media opacity and strabismus are not included as referral criteria in these calculations. There are small variations in these predictability parameters with adjustment of the device’s criteria to the jointly optimized cutoffs.

TABLE 5.

Estimated Sensitivity and Specificity for the Refractive Error Values of Each Gold Standard^a

Screener	Gold Standard 1 (CI)	Gold Standard 2 (CI)	Gold Standard 3 (CI)
Spot Vision Screener “out-of-the-box” manufacturer criteria
Sensitivity	91.7 (87.2 to 96.2)	82.4 (77.0 to 87.9)	77.9 (72.3 to 83.6)
Specificity	75.7 (69.5 to 81.9)	83.8 (77.7 to 89.9)	84.9 (78.7 to 91.2)
Spot Vision Screener recommendation based on jointly optimized cutoffs maximizing AUC
Sensitivity	89.0 (83.9 to 94.1)	88.3 (93.7 to 92.9)	85.8 (81.0 to 90.6)
Specificity	77.3 (71.3 to 83.3)	80.3 (73.7 to 86.8)	81.0 (74.1 to 87.8)

CI = confidence interval; AUC = area under the curve

^aThe values are based on the current Spot Vision Screener manufacturer’s”out-of-the-box” recommendations as used in this study and the jointly optimized cutoffs calculated as described in the text. Gold standard refractive criteria are as described in Table 1. Media opacity and strabismus are not included as referral criteria in these calculations.

The Spot Vision Screener is manufactured by Welch Allyn, Skaneateles Falls, NY.

DISCUSSION

The Spot Vision Screener demonstrated good sensitivity (89.5%) for detecting AAPOS amblyopia risk factors in older patients, which was slightly higher than some reports in younger children.^4-7 High sensitivity is especially desirable when screening older children.¹⁴ In school-aged children, the primary focus shifts from preventing amblyopia to detecting visual impairment, including refractive errors that may interfere with academic performance. The Spot Vision Screener demonstrated lower sensitivities (80 and 75.3%) with narrower criteria detecting smaller degrees of refractive error. As has been suggested in previous studies, hyperopia is the refractive parameter most commonly “missed” by the Spot Vision Screener.^12,20

We provide recommendations regarding the best cutoff threshold settings to maximize sensitivity and specificity for each of the three considered refractive gold standards. The proposed cutoffs have higher sensitivity for gold standards 2 and 3 but slightly lower specificity, whereas the current recommendation for gold standard 1 is similar to our optimized cutoffs. Although the confidence intervals overlap, thus preventing conclusive recommendations, users may consider modifying the current refractive cutoffs on the Spot Vision Screener in situations where the detection of smaller refractive errors is desirable.

A limitation of our study is the small and enriched population. Although the sensitivity and specificity are thought to be test-specific across populations, PPV and NPV vary with disease prevalence. We estimate that the PPV for the general population is between 52.3% and 61.8%, depending on the stringency of the criteria, with excellent NPVs. However, the estimates and results from our ophthalmology office population may not correlate with the results of a community or school screening. In addition, it may be helpful to further delineate the results by age, because visual needs change and growth is rapid during “school-age” years.

Currently, instrument-based screening is recommended when children are too young to cooperate with identifying letters (eg, preschool age), and optotype visual acuity screening is recommended for older children.^1,2 However, the legislative requirements and methods used for school vision screenings²¹ are inconsistent, and the efficacy of visual acuity screenings may vary with different testing situations. The current study found that the sensitivities for the Spot Vision Screener were not unlike the sensitivities reported for visual acuity testing in detecting significant refractive error in older children. In one report of larger scale vision screening, a maximum sensitivity of 75% and specificity of 85% were obtained from combining autorefraction and uncorrected visual acuity in the detection of myopia.^22–24

Automated screeners allow for rapid screening, approximately one child per minute, by minimally trained lay persons. Because of increased availability and ease of use, instrument-based screeners have been employed to screen school-aged children in areas with limited health care access^20,25,26 and in school systems.^25–27

Uncorrected refractive error is the primary cause of moderate to severe vision impairment in young people.²⁸ In some situations, such as cultural barriers or time limitation, automated screening may be more effective than acuity testing in the school-aged population. Expanding eye care to all students, communities, and institutions may benefit from a flexible approach, because automated screening may maximize vision screening for their situation. We found the Spot Vision Screener to be an acceptable method of detecting significant refractive error in the school-aged population.