Effectiveness of the Spot Vision Screener using updated 2021 AAPOS guidelines

Abstract

PURPOSE

To evaluate the Spot Vision Screener according to updated 2021 AAPOS Vision Screening Committee guidelines for instrument-based pediatric vision screen validation.

METHODS

As part of an IRB-approved ongoing prospective study, children were screened with the Spot prior to a complete examination.

RESULTS

Spot screening was successful in 1,036 of 1,090 children (95%). Forty-eight percent of participants were referred for further screening using the Spot manufacturer guidelines, and 40% of all children were found to have a 2021 amblyopia risk factor or visually significant refractive error by gold standard examination. The Spot recommendation compared reasonably well to the 2021 criteria, with an overall sensitivity of 0.88 and a specificity of 0.78. Applying updated guidelines to the Spot for hyperopia, anisometropia, and astigmatism yielded moderate-to-poor sensitivity (0.27–0.77) but excellent specificity (>0.9). The area under the curve of the receiver operating characteristic analysis demonstrates overall good prediction performance for the Spot for each diagnosis—myopia, hyperopia, astigmatism, anisometropia (range, 0.87–0.97). Results of our study suggest increasing the instrument referral criterion for astigmatism from 1.5 D (manufacturer thresholds of the screener used in this study) to 2 D in older children. Decreasing the anisometropia cut-off from 1 D to 0.75 D would improve sensitivity from 0.59 to >0.8.

CONCLUSIONS

In our study population, the overall predictive ability of the Spot is good, with a sensitivity of 0.88 and a specificity of 0.78. We recommend specific device refractive referral criteria to maximize screening effectiveness using the updated AAPOS guidelines.

Instrument-based pediatric vision screening is endorsed by the American Academy of Pediatrics as a valid vision screening method. It allows detection of amblyopia risk factors (ARFs) at an age when amblyopia can be prevented and effectively treated.^1–5 Since previous AAPOS guidelines for detection of ARFs were published in 2003 and 2013, screening devices have improved, and the prevalence and severity of risk factors has been better defined.⁶ In 2021, the Vision Screening Committee of the American Association for Pediatric Ophthalmology and Strabismus (AAPOS) updated guidelines for the detection of ARFs as well as visually significant refractive errors (VSREs).⁷ The updated “Uniform Guidelines” are used to evaluate screener effectiveness and to provide recommendations for device calibration to maximize efficacy in detecting children at risk of visual impairment.

Compared with the 2013 guidelines, the new guidelines raise the threshold for symmetric astigmatism, which has a high prevalence in many screening populations, and lowers the threshold for anisometropia, for which early treatment is important for preventing amblyopia. The new Uniform Guidelines (Table 1) are more specific for children <4 years of age and more sensitive for those ≥4 years of age. The new guidelines include detection of smaller amounts of astigmatism and myopia for children ≥4 to allow institution of myopia prevention and assure the best visual acuity for academic progress.^8–14

Table 1.

2021 AAPOS amblyopia risk factor and visually significant refractive error failure level definitions^a

Risk term	Age	Thresholdb
ARF (severity ranked)
Media opacity		>0.1 mm
Strabismus		>8 PD manifest
Anisometropia		>1.25 D
Hyperopia		>4.00 D
Astigmatism	<4 years	>3.00 D
Visually significant refractive errors
Astigmatism	≥4 years	>1.75 D
Myopia	<4 years	More than 3 D
Myopia	≥4 years	More than 2 D

ARF, amblyopia risk factor; D, diopters; PD, prism diopters.

^aARFs are stratified by amblyopia severity; combined cases could be listed by the more severe condition. Media opacity, manifest strabismus, anisometropia, and hyperopia are for all age levels. Reporting validation for each ARF and refractive error independently and also combined is recommended.

^bThresholds refer to meridional refractive power for hyperopia and myopia and the difference in the less hyperopic meridian for anisometropia.

The Spot Vision Screener (Welch Allyn, Skaneateles Falls, NY) is marketed and has been validated as a tool to detect ARF using previous guidelines.^4,12,15–19 We sought to further determine the accuracy of the Spot in our population of pediatric ophthalmology patients using the updated AAPOS 2021 guidelines and suggest device settings to maximize efficacy using these updated expert consensus guidelines.²⁰

Subjects and Methods

This study was approved by the Medical University of South Carolina Institutional Review Board and complied with the US Health Insurance Portability and Accountability Act of 1996. Informed consent was obtained from the patients’ guardians prior to enrollment. The protocol of this continuing prospective study has been previously reported.^21–23 This analysis included patients age 6 months to 13 years presenting for complete pediatric ophthalmological examination when study personnel were available between June 2013 and August 2021. The Spot software v.2.1.4 with installed manufacturer out-of-the-box cut-offs were used. Children were screened with the Spot Vision Screener by lay personnel prior to examination.

The Spot provides a report of pupillary diameter, ocular alignment, estimated binocular refraction, and referral recommendation—“all measurements within range” or “complete eye exam recommended.” Several attempts were made to obtain a successful reading. The Spot refractive measurement range is ±7.50 D. Patients who could not be screened (ie, no refraction estimate or recommendation) were included in this study as automatic referrals. A comprehensive examination was performed by a pediatric ophthalmologist masked to the screener results. In all patients, visual acuity, stereopsis, ocular motility, and ocular alignment were assessed, and the anterior segment was examined. Patients underwent cycloplegic refraction and a fundus examination 30–40 minutes after instillation of one drop of proparacaine hydrochloride ophthalmic solution USP 0.5% followed by 1–2 drops of tropicamide 1%/phenylephrine 2.5%/cyclopentolate 1%. Patient age and demographics were collected. The following data were collected from the Spot: whether the screening was successfully completed, estimated refraction, and referral recommendation. From the physician examination, the following data were collected: ocular motility, ocular alignment, cycloplegic refraction, diagnosis of amblyopia with history of treatment, and presence of systemic or ocular pathology.

Children who met one or more criteria of the new AAPOS ARF and VSRE guidelines (Table 1) were considered to have positive gold standard findings. These updated gold standard confirmatory comprehensive examination failure levels include anisometropia >1.25 D and hyperopia >4.0 D. Astigmatism >3.0 D in any meridian and myopia >3 D should be detected in children <48 months, whereas astigmatism >1.75 D and myopia 2 D or more should be detected after 48 months. Any media opacity >1 mm and manifest strabismus of more than 8^Δ should also be identified. An additional analysis was carried out including children with a diagnosis of amblyopia as a positive gold standard finding.

Statistical Analysis

Descriptive statistics were calculated and the percentage in whom the Spot obtained a result was noted. The primary goal of this study was to evaluate the predictive ability of the Spot to determine need for further evaluation according to the recently updated AAPOS criteria. Children were divided into age groups to determine gold standard results according to the 2021 AAPOS guidelines (Table 1). These guidelines specify that gold standard thresholds are determined using meridional refractive power for hyperopia and myopia and the difference in the lesser meridian for anisometropia.

Patients were considered to have ARFs or VSREs (henceforth, “ARFs”) based on the comprehensive examination and the physician’s diagnosis and 2021 AAPOS ARF guidelines (Table 1). Sensitivity, specificity, and positive and negative predictive values (PPV and NPV) of the Spot in detecting ARF were calculated. Children in whom the Spot was unable to obtain a result were included as automatic referrals. An additional analysis was carried out to include children with a diagnosis of amblyopia as a risk factor (“ARF+A”).

Because PPV and NPV are affected by the high disease prevalence of our study population, these values were estimated for the general population. Because the frequency of ARFs is reported by cycloplegic retinoscopy as 24% in normal preschool children, this prevalence value is used to estimate PPV and NPV in the general pediatric population.²⁴

The association between Spot recommended screening and meeting one or more ARF/ARF+A diagnosis was determined. The prediction performance for the Spot parameter used to predict each diagnosis (myopia, hyperopia, astigmatism, anisometropia) was estimated using the area under the curve (AUC) of receiver operating characteristics (ROC). All models were developed using refractive data from all patients for whom Spot screening was successful, divided into data from children less than 4 years of age, and data from children 4 years of age and older. In addition to the area under the ROC, we also calculated model sensitivity, specificity, PPV, and NPV for each outcome, determined by comparing the predicted disease status to the observed disease status. Confidence intervals for all performance parameters were calculated using the exact binomial confidence interval approach. Additionally, we examined different choices of cutoffs for Spot parameters to identify the optimal parameters to discriminate the different diagnoses. All analyses were conducted in R v. 4.1.0 (https://www.R-project.org/).

Results

Results from 1,090 children were analyzed (47.7% males; average age, 5.5 years; range, 6 months to 13 years). Patient characteristics are presented in Table 2. Spot screening was successful in 1,036 children (95%). Of the total, 369 children were <4 years of age, of whom 342 (93%) had a successful Spot screening. Forty-eight percent of participants were recommended for further examination based on Spot manufacturer guidelines, and 40% of children were found to have an ARF by gold standard examination according to the 2021 guidelines. Patient characteristics are reported in Table 2. With a prevalence of ARFs of 24%, the PPV and NPV were estimated to be 0.55 and 0.96.

Table 2.

Patient demographics

Characteristic	All subjects (n = 1090)
Sex, male, no. (%)	520 (47.7)
Age, months, mean ± SD	66.6 ± 34.9
Ethnicity, no. (%)
Hispanic or Latino	303 (27.8)
Not Hispanic or Latino	778 (71.4)
Not reported	9 (0.83)
Race, no. (%)
White	465 (42.6)
Black	273 (25.1)
Hispanic or Latino	303 (27.8)
Other or not reported	50 (4.59)
Spot Screening Successful, no. (%)	1036 (94.8)
Recommended for further evaluation, no. (%)
By Spot	527 (48.4)
By Spot excluding failed to read as automatic referrals	470 (43.1)
Meet updated AAPOS guidelines	432 (39.6)
Meet updated AAPOS guidelines + dx amblyopia	466 (42.8)

The difference between physician- and Spot-reported measures for each parameter are shown in the Bland-Altman plots in Figure 1. Points on the Bland-Altman plots represent the difference between the Spot and physician measures versus the mean of the two measures for each patient for the following: maximum meridional refractive power, minimum meridional refractive power, cylinder, and anisometropia (defined as the interocular difference in the lesser meridian). Maximum and minimum meridional powers both showed an overall negative bias (middle gray line below 0) toward lower values for the Spot screener, with greater underestimation of meridional power at higher levels of hyperopia. A small positive overestimation of astigmatism by the Spot is present compared with the physician measure. With regard to outliers on these plots, 89 of 1090 participants (8.2%) for whom the Spot could evaluate sphere had a Spot screener greater or lesser sphere that underestimated the physician lesser or greater meridional value by 3.5 D or more (less hyperopic, or more myopic), and a majority of these subjects were highly hyperopic. Fewer than 1% of participants (9 of 1,090) had a Spot screener greater or lesser sphere that overestimated the physician lesser or greater meridional value by ≥3.5 D (more hyperopic, or less myopic), and 6 of these subjects were highly myopic.

FIG 1.

Bland-Altman plots comparing Spot measurements to physician-reported measurements. The x-axis represents the mean of the Spot and physician measures for each child; the y-axis, the difference between the Spot and physician measures. The gray middle line represents the mean difference between the Spot and physician measures; the bottom and top gray lines represent the 95% agreement interval, that is the interval within which 95% of the differences between the first and second method fall.

The ROC curves for the appropriate Spot parameter for each diagnosis (hyperopia, myopia, astigmatism, and aniso-metropia) across all patients and stratified by patient age (<4 or ≥4 years) are shown in Figure 2. The AUC for the different diagnoses ranged from 0.88 to 0.97 across all children, from 0.82 to 0.99 when only children <4 years of age were considered and from 0.87 to 0.97 in children ≥4 years of age. The AUCs for myopia were larger than for the other three diagnoses and were similar between age groups. The AUCs for hyperopia, anisometropia, and astigmatism were similar (between 0.88 and 0.90 across all children) and were generally similar for older and younger children.

FIG 2.

Receiver operating characteristics (ROC) curves for the Spot criterion for physician diagnosed myopia, hyperopia, anisometropia, and astigmatism. The three ROC curves in each panel represent the ROC across all patients (light gray solid line), in children under 4 years of age (black dashed line), and in children 4 years or older (dotted-gray line).

Table 3 shows the performance characteristics for the overall Spot recommendation by the ARF and ARF or amblyopia (ARF+A) criteria and for each specific disease diagnosis across all patients and stratified by patient age. The overall Spot recommendation performed reasonably well in this clinic population compared to the ARF criterion and had an overall sensitivity of 0.88 and a specificity of 0.78. This performance was similar when considering the ARF+A criterion, with a sensitivity of 0.85 and specificity of 0.79. Overall Spot performance compared to the ARF or ARF+A guidelines was generally poorer in children under the age of 4 for both the ARF and ARF+A criteria. Applying updated AAPOS guidelines to the Spot measurements to assess myopia, hyperopia, anisometropia, and astigmatism generally yielded moderate to poor sensitivity but excellent specificity. Considering all age groups, sensitivities for the specific diagnoses ranged from 0.27 to 0.77 (similar results were observed when stratifying on <4 or ≥4 years of age). However, specificities were >0.9 for all four diagnoses across all children and within each age group.

Table 3.

Spot performance characteristics overall (by ARF and ARF + amblyopia criteria) and by specific diagnosis^a

Outcome	Metric	All patients (n = 1090/1033)	Patients <4 years (n = 369/342)	Patients ≥4 years (n = 721/690)
ARF	Sensitivity	0.88 (0.85, 0.91)	0.83 (0.75, 0.90)	0.89 (0.86, 0.93)
	Specificity	0.78 (0.74, 0.81)	0.73 (0.68, 0.78)	0.80 (0.77, 0.84)
	PPV	0.72 (0.68, 0.76)	0.56 (0.49, 0.64)	0.79 (0.75, 0.83)
	NPV	0.91 (0.88, 0.93)	0.91 (0.87, 0.95)	0.90 (0.87, 0.93)
ARF+A	Sensitivity	0.85 (0.82, 0.89)	0.80 (0.73, 0.87)	0.87 (0.84, 0.91)
	Specificity	0.79 (0.76, 0.83)	0.75 (0.70, 0.81)	0.82 (0.78, 0.86)
	PPV	0.76 (0.72, 0.79)	0.62 (0.54, 0.69)	0.81 (0.77, 0.85)
	NPV	0.88 (0.85, 0.91)	0.88 (0.84, 0.92)	0.88 (0.84, 0.91)
Myopia	Sensitivity	0.60 (0.50, 0.70)	0.62 (0.35, 0.88)	0.60 (0.49, 0.71)
	Specificity	0.99 (0.98, 1.00)	1.00 (0.99, 1.00)	0.99 (0.98, 1.00)
	PPV	0.86 (0.78, 0.95)	0.89 (0.68, 1.09)	0.86 (0.77, 0.95)
	NPV	0.96 (0.95, 0.97)	0.98 (0.97, 1.00)	0.95 (0.93, 0.97)
Hyperopia	Sensitivity	0.27 (0.20, 0.35)	0.24 (0.10, 0.38)	0.28 (0.19, 0.37)
	Specificity	1.00 (1.00, 1.00)	1.00 (1.00, 1.00)	1.00 (1.00, 1.00)
	PPV	0.97 (0.92, 1.00)	1.00 (1.00, 1.00)	0.96 (0.90, 1.03)
	NPV	0.90 (0.88, 0.92)	0.92 (0.89, 0.95)	0.90 (0.87, 0.92)
Anisometropia	Sensitivity	0.69 (0.59, 0.79)	0.69 (0.46, 0.91)	0.69 (0.58, 0.80)
	Specificity	0.95 (0.94, 0.97)	0.96 (0.93, 0.98)	0.95 (0.93, 0.97)
	PPV	0.56 (0.47, 0.66)	0.44 (0.25, 0.63)	0.60 (0.49, 0.71)
	NPV	0.97 (0.96, 0.98)	0.98 (0.97, 1.00)	0.97 (0.95, 0.98)
Astigmatism	Sensitivity	0.77 (0.70, 0.84)	0.25 (0.00, 0.55)	0.80 (0.73, 0.87)
	Specificity	0.93 (0.91, 0.94)	0.94 (0.91, 0.96)	0.92 (0.90, 0.94)
	PPV	0.61 (0.54, 0.69)	0.09 (0.03, 0.20)	0.70 (0.62, 0.77)
	NPV	0.96 (0.95, 0.98)	0.98 (0.97, 1.00)	0.95 (0.93, 0.97)

NPV, negative predictive value; PPV, positive predictive value; A, amblyopia; ARF, amblyopia risk factor or visually significant refractive error as defined in Table 1.

^aPPV and NPV is specific to study population.

Table 4A shows manufacturer refractive criteria for Spot screener, version 2.0.16. To identify Spot criteria that optimize sensitivity and specificity for the different diagnoses, we examined different cutoff values for the Spot measurements and evaluated their effect on sensitivity and specificity for each diagnosis. Table 4B shows the cutoff values for the Spot that provide the optimal sensitivity and specificity for myopia, hyperopia, anisometropia, and astigmatism. The optimal cutoff was selected as the cutoff that yielded the largest product of sensitivity and specificity and/or for which both sensitivity and specificity were >0.8. A majority of cutoffs provide both sensitivity and specificity >0.8, the only exceptions being the cutoffs for astigmatism in children <4 years of age (sensitivity = 0.88, specificity = 0.66) and hyperopia in children ≥4 years (sensitivity = 0.77, specificity = 0.82).

Table 4A.

Manufacturer refractive criteria in diopters for Spot screener, version 2.0.16

Condition	0.5–1 years	1–3 years	3–6 years	6–20 years
Myopia	−2	−2	−1.25	−1
Hyperopia	3.5	3	2.5	2.5
Anisometropia	1.5	1	1	1
Astigmatism	2.25	2	1.75	1.5

Table 4B.

Cutoff values in diopters for the Spot that provide the optimal sensitivity and specificity for meridional myopia, meridional hyperopia, anisometropia in the lesser meridian, and astigmatism

Condition	All	<4 years of age	≥4 years of age
Myopia	−0.75	−1.75	−0.75
Hyperopia	0.875	1.00	0.875
Anisometropia	0.75	0.75	0.75
Astigmatism	2.00	1.32	2.00

Discussion

Using updated AAPOS guidelines, the ability of the Spot to detect the composite outcome of “any ARF or VSRE” was good, with a sensitivity of 0.88 and a specificity of 0.78 in this large clinic-based cohort of patients up to 13 years of age. Our previous smaller study using 2013 guidelines found sensitivity of 0.88 and specificity of 0.76, suggesting maintained good sensitivity with a small improvement in specificity using the 2021 guidelines.¹⁶ Other studies also found similar efficacy using the older AAPOS guidelines among populations of ophthalmology practice patients.^4,15

The 2021 AAPOS guidelines update is designed to increase sensitivity for anisometropia in younger children while reducing false-positive referrals in preschoolers, especially those due to symmetric, moderate astigmatism. An additional goal is to detect lower magnitudes of astigmatism and myopia in older children as these conditions may influence school performance and be amenable to myopia prevention therapy. Our analysis suggests that the guidelines update will prove effective toward these goals.

The AUC analysis of ROC curves (Figure 2) demonstrates overall good prediction performance for the Spot parameter for each diagnosis (range, 0.88 to 0.97). The second highest predictive performance is that for anisometropia in younger children (AUC > 0.93), for whom therapy to treat or even prevent amblyopia is most effective. The highest predictive performance is that for myopia for all children (AUC > 0.97), for which therapy exists to reduce myopic progression.

The newer guidelines raise the recommended gold standard target for astigmatism detection from >2 D to >3 D for children <4 years old, and from >1.50 D to >1.75 in children ≥4 years of age. The Spot has been reported to overestimate astigmatism,²⁵ and our analysis of Bland-Altman curves comparing astigmatism measurements of the Spot and the examiner confirms a positive bias, suggesting a small overestimation of astigmatism. Within our population, the overall specificity for astigmatism is excellent at 0.93 while maintaining a good sensitivity for astigmatism (0.80) for children ≥4 years of age. The high specificity for astigmatism may reduce referrals for moderate symmetric astigmatism; the new guidelines still allow detection of significant astigmatism in children of school age.

High sensitivity for myopia is desirable in older children, for whom distance vision is important in the classroom. Using the new guidelines, the Spot demonstrates good sensitivity for myopia (0.89) in older children, with high predictive performance (AUC > 0.97).

Our findings confirm again that the Spot remains poorly sensitive (0.27) in detecting hyperopia.^19,23–25 Bland-Altman curves (Figure 1) demonstrate a tendency toward underestimation of hyperopia by the Spot screener, increasing with higher levels of hyperopia. Most outliers on the Bland-Altman plots are children with underestimated hyperopia. It is likely that accommodation on the Spot device plays a role in this finding. While there is a correlation between uncorrected hyperopia and poor academic performance, the causative effect of uncorrected hyperopia on learning and academic achievement is not well established.^26–28 It is not known to what degree underestimation of hyperopia affects the ability of the Spot to identify children having or at risk of developing accommodative esotropia.

We provide recommendations for Spot cut-off referral criteria to maximize sensitivity and specificity for the updated 2021 AAPOS guidelines. The out-of-the-box Spot manufacturer criteria can be adjusted by the user for desired sensitivity and specificity. Results of our study suggest increasing the instrument referral criterion for astigmatism from 1.5 D (manufacturer thresholds of the screener used in this study) to 2 D in older children.^16–17 An additional recommendation would decrease the anisometropia cut-off from 1 D to 0.75 D, which improves sensitivity from 0.59 to >0.8. The manufacturer may wish to consider these recommendations in future out-of-the box software versions.

Our study is limited by the fact that our population of ophthalmology patients have a relatively high disease prevalence, which may affect results, including testability. Although sensitivity and specificity estimates for individual refractive criteria should not vary according to disease prevalence in the population studied, the same is not true of a composite outcome such as “any ARF or VSRE,” for which sensitivity and specificity estimates may vary in different populations depending on the relative prevalence of the different components of the composite outcome. In addition, PPV and NPV do vary with disease prevalence, even for individual refractive criteria; to address this we also estimated PPV and NPV for the composite outcome for the general population based on published estimates of ARF or VSRE prevalence. Our estimates for the general population indicate that the PPV is reasonable at 0.55, so that 55% of children referred would be found to have a 2021 ARF or VSRE. Conversely, only 4% of children with a 2021 ARF or VSRE would be “missed” by the Spot screener. Although these estimates are useful, results from an ophthalmology office population may not correlate with results of community or school screening. We also included children up to the age of 13, and we found slightly lower sensitivity and specificity for composite outcomes in children <4 years of age compared to overall and much lower sensitivity for targeted levels of astigmatism in this subgroup than overall.

Further evaluation with recommendations from diverse and age-appropriate community screening populations will continue to improve efficacy of pediatric instrument-based vision.^19,24,29 We analyzed a large patient cohort to better define effectiveness and maximize referral criteria for the Spot vision screener.