The effectiveness of the Spot Vision Screener in detecting amblyopia risk factors

Abstract

Background

The purpose of this study was to evaluate the updated Spot Vision Screener (PediaVision, Welch Allyn, Skaneateles Falls, NY) in detecting amblyopia risk factors using the 2013 guidelines of the American Association for Pediatric Ophthalmology and Strabismus (AAPOS).

Methods

In this prospective study, patients seen from June 2012 to November 2013 were tested with the Spot prior to examination by a pediatric ophthalmologist who was masked to test results. The following data were analyzed: age, subject testability, examination findings, and systemic and ocular pathology. Children were divided into three age groups to determine gold standard results according to the AAPOS guidelines.

Results

A total of 444 children (average age, 72 months) were included. Compared to the ophthalmologist's examination, the Spot sensitivity was 87.7% and the specificity was 75.9% in detecting amblyopia risk factors. There were no significant differences in sensitivity between the age groups, although the positive predictive value improved in the older age groups.

Conclusions

In our study cohort, the Spot provided good specificity and sensitivity in detecting amblyopia risk factors according 2013 AAPOS criteria, with minor improvements with updated versions.

Amblyopia remains the most common cause of preventable visual loss in children,¹ and the American Academy of Pediatrics has recommended automated vision screeners as an acceptable alternative to traditional vision screening in children 3-5 years of age.² Although the Spot Vision Screener³ (Welch Allyn, Skaneateles Falls, NY) is marketed to schools,^4,5 there are few published reports evaluating its effectiveness.^6-8 Silbert and Matta⁹ recently reported their experience using the Spot with the original software (v. 1.0.3), noting that performance would be expected to improve with future criteria modifications and improved software. The purpose of this study was to evaluate the newer software versions of the Spot (v. 1.1.51 and v. 2.0.16) in detecting amblyopia risk factors according to the 2013 American Association of Pediatric Ophthalmology and Strabismus Vision Screening Committee guidelines for automated vision screeners.¹⁰

Methods

This prospective study was approved by the Medical University of South Carolina Institutional Review Board and adhered to the US Health Insurance Portability and Accountability Act of 1996. Written informed consent was obtained from parents or guardians. Patients aged 1 to 16 years presenting for complete pediatric ophthalmological examination to the Storm Eye Institute of the Medical University of South Carolina between June 2012 and November 2013, with appropriate personnel and guardianship available, were asked to participate. The study population included new patients as well as patients routinely followed.

Vision Screening

The Spot handheld photorefractor has been previously described.⁹ The device is supplied with out-of-the-box software referral criteria but allows user adjustment of referral criteria. The screener is held approximately 3 feet from the subject while the child looks at the display of twinkling lights and sounds. The screen reports whether the subject is too far or too close and shows a spinning circle and the child's face when data acquisition is occurring. Data acquisition is usually complete in approximately 2 seconds. A report of pupillary diameter, ocular alignment, estimated binocular refraction, and referral recommendation is displayed, stored, and available for printing. The Spot provides an interpretation—“all measurements within range” or “complete eye exam recommended.” When the device is unable to evaluate a subject, it will note “pupils too small” or “pupils not found,” “out of range,” or continue attempting to obtain a reading.

Spot software v.1.1.51 was employed. With the release of the 2.0.16 software, the Spot device and dataset were updated by the manufacturer. The software updates included modifications of refractive and strabismus referral criteria. The manufacturer's out-of-the-box referral criteria are given in Table 1. Screening was conducted by trained lay personnel on the same day as ophthalmological examination following manufacturer guidelines. Printouts of the results were collected. If the device was unable to obtain a reading after several minutes and multiple attempts, the tester noted “unable to obtain a reading.”

Table 1.

Manufacturer criteria for Spot screener, version 2.0.16

Age, months	Anisometropia, D	Astigmatism, D	Myopia, D	Hyperopia, D	Anisocoria, mm	Gaze, degrees
						Vertical	Nasal	Temporal	Asymmetry
6-12	1.5	2.25	2	3.5	1	8	5	8	8
12-36	1	2	2	3	1	8	5	8	8
36-72	1	1.75	1.25	2.5	1	8	5	8	8
72-240	1	1.5	1	2.5	1	8	5	8	8
240-1200	1	1.5	0.75	1.5	1	8	5	8	8

D, diopters.

Examination by Pediatric Ophthalmologist

A comprehensive examination was then performed, including visual acuity, stereopsis and motility evaluation, and examination of the anterior segment. Cycloplegic retinoscopy and fundus examination were performed by the examining pediatric ophthalmologist 30 to 40 minutes following the instillation of proparacaine hydrochloride ophthalmic solution USP 0.5% followed by 1-2 drops of a pediatric “combo drop” of tropicamide1%, phenylephrine2.5%, and cyclopentolate 1%. Four experienced pediatric ophthalmologists participated in the study and were masked to photoscreener results.

Data Collection

The following patient data was collected: age; whether or not a reading was obtained; pupillary size and distance; Spot screener recommendation (“complete eye exam recommended” [refer], and “all measurements in range” [pass]) according to manufacturer criteria and AAPOS referral criteria; examination of the pediatric ophthalmologist, including measured strabismus and cycloplegic refraction; and diagnoses of systemic and ocular pathology. Data was entered into a REDCap (Research Electronic Data Capture) database hosted at the Medical University of South Carolina for statistical analysis.¹¹

Statistical Methods

We included all patients, even those with no result. Descriptive statistics were calculated, and the percentage in whom the Spot obtained a result was noted.

Children were divided into age groups to determine gold standard results according to the AAPOS guidelines. Patients were considered to have amblyopia or amblyopia risk factors on the comprehensive examination on the basis of the physician's diagnosis and 2013 AAPOS amblyopia risk factor guidelines¹⁰ (Table 2).

Table 2.

Amblyopia risk factors targeted with automated preschool vision screening: recommended ARF criteria¹⁰

Refractive risk factor targetsa
Age, months	Astigmatism	Hyperopia	Anisometropia	Myopia
12-30	>2.0 D	>4.5 D	>2.5 D	> –3.5 D
31-48	>2.0 D	>4.0 D	>2.0 D	> –3.0 D
>48	>1.5 D	>3.5 D	>1.5 D	> –1.5 D
Nonrefractive amblyopia risk factor targetsa
All ages manifest strabismus >8 PD in primary position
Media opacity >1 mm

D, diopters; PD, prism diopter.

^aAdditional reporting of sensitivity to detect greater-magnitude refractive errors is encouraged.

^bFor all ages.

Patients were considered to have met the 2013 AAPOS guidelines for strabismus referral if a constant measurement of ≥8^Δ in primary position at distance or near was found at the time of examination. In addition, children with media opacities of >1 mm or a diagnosis of amblyopia were included as a positive for amblyopia risk factors.

Sensitivity, specificity, and positive and negative predictive values of the Spot in detecting amblyopia risk factors were calculated. Data was analyzed for both software updates (v. 1.1.51 and v. 2.0.16) using manufacturer's criteria. Results were also calculated with v. 2.0.16 using 2013 AAPOS guidelines in place of the manufacturer's criteria. Children in whom the Spot was unable to obtain a result were included as automatic referrals. Sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV) of the SPOT were also compared between three age groups (12-30 months, 31-46 month, >48 months) using the χ² or Fisher exact tests, as appropriate. Trends in each diagnositic variable across age groups were also evaluated using the Cochren-Armitage trend test. All analyses were conducted in SAS v. 9.3 (SAS Institute, Cary, NC).

PPV and NPV are affected by the prevalence of disease in the specific population being studied yet can be estimated for the general population. PPV and NPV are related to prevalence of disease in a population and to the sensitivity and specificity of a test according to the relationships shown below:

$$\begin{array}{cc}\hfill \mathrm{PPV}=& \frac{\text{senitivity}\times \text{prevalence}}{\text{sensitivity}\times \text{prevalence}+(1-\text{specificity})\times (1-\text{prevalence})}\hfill \\ \hfill \mathrm{NPV}=& \frac{\text{specificity}\times (1-\text{prevalence})}{\text{specificity}\times (1-\text{prevalence})+(1-\text{sensitivity})\times \text{prevalence}}\hfill \end{array}$$

Thus as prevalence decreases, the PPV is expected to decrease, but the NPV is expected to increase. Using these relationships, we estimated PPV and NPV of the Spot version 2.0.16 for the general preschool population.

Results

A total of 444 children (226 males [51%]) were included: 54% white, 35% African American/black, and 9% Hispanic. The average age was 72 months (range, 11-221 months). The prevalence of 2013 AAPOS amblyopia risk factors in our population was found to be 55%. Of those children found to have amblyopia risk factors, 93 (38%) were referred for strabismus and 127 (52%) for refractive error.

With version 1.151, the Spot referred 61% (272/444) of the patients. When compared with the ophthalmologist's examination, the sensitivity of the Spot to detect amblyopia risk factors was 88.1%; the specificity was 71.9% (Table 3).

Table 3.

Performance metrics for the updated Spot

	ARF+	ARF–	Sensitivity (95% CI)	Specificity (95% CI)	PPV (95% CI)	NPV (95% CI)
Version 1.151			88.1 (83.4-91.9)	71.9 (65.1-78.0)	79.3 (74.0-84.0)	83.1 (76.7-88.4)
Spot referral/positive	215	56
Spot pass/negative	29	143
Version 2.0.16			87.7 (82.9-91.5)	75.9 (69.3- 81.6)	81.7 (76.5-86.2)	83.4 (77.2-88.5)
Spot referral/positive	214	48
Spot pass/negative	30	151
Using 2013 AAPOS criteriaa			84.8 (79.7-89.1)	70.9 (64.0-77.1)	78.1 (72.6-82.9)	79.2 (72.5-84.9)
Spot referral/positive	207	58
Spot pass/negative	37	141

ARF, amblyopia risk factors; CI, confidence interval; NPV, negative predictive value; PPV: positive predictive value.

^aIn place of manufacturer criteria.

With version 2.0.16, 271 of 444 (61%) were referred. The sensitivity was 87.7%; the specificity, 75.9% (Table 3). Thus specificity improved.

We estimated the PPV and NPV in the general population for Spot version 2.0.16. The expected prevalence of positive amblyopia risk factors in the general preschool population is estimated to be approximately 20%,^10,12,13 which is lower than the observed prevalence in our population of 59%. For a population with a prevalence of 20% of children with amblyopia risk factors, the PPV is 48% and the NPV is 96%.

Results using the 2013 AAPOS guidelines instead of the manufacturer's recommended criteria are compared with these same guidelines as found on ophthalmological examination in Table 3. There was a decline in sensitivity to 84.8% and in specificity to 70.9%.

The Spot was unable to obtain a result on 9% (41 of 443) of the children and thus were automatically referred. Of these, 15 had pseudophakia, 10 glaucoma, 6 aphakia, 3 high myopia, 2 cataract, 2 nystagmus, and 1 ptosis. The Spot was unable to obtain a reading on 11 children who had a normal exam. The majority (73%) of children automatically referred by Spot had ocular pathology and/or +ARF.

Performance metrics for the Spot screener by age group are shown in Table 4. There were no significant differences in sensitivity or NPV between groups. However, the Spot had significantly higher specificity for children 31-48 months of age compared to children 12-30 months of age (P = 0.013). There was also a significant trend of increasing PPV with increasing age (P = 0.001).

Table 4.

Performance metrics for the Spot referral, version 2.0.16, by age group

	ARF +	ARF –	Sensitivity (95% CI)	Specificity (95% CI)	PPV (95% CI)	NPV (95% CI)
Patients 12-30 months
Spot referral/positive	27	13	81.8	56.7	67.5	73.9
Spot pass/negative	6	17	(68.1, 91.0)	(40.6, 71.7)	(53.5, 79.4)	(56.3, 86.8)
Patients 31-48 months
Spot referral/positive	23	10	82.1	76.2	69.7	86.5
Spot pass/negative	5	32	(67.3, 91.7)	(63.2, 86.1)	(54.5, 82.0)	(74.6, 93.8)
Patients >48 months
Spot referral/positive	164	25	89.6	80.3	86.8	84.3
Spot pass/negative	19	102	(84.9, 93.2)	(73.2, 86.2)	(81.7, 90.8)	(77.5, 89.6)
P value, age group compare			0.136	0.011	0.001	0.360

ARF, amblyopia risk factors; CI, confidence interval; NPV, negative predictive value; PPV, positive predictive value.

Discussion

This study compared the updated version of the Spot Vision Screener to ophthalmological examination in children seen at a pediatric ophthalmology practice. Using the manufacturer's referral criteria, we found reasonably good sensitivity and specificity with both recent software updates, as did Silbert and Matta⁹ using the original software and higher than did Arnold and colleagues.⁶ While a specificity in the low 70% range may be not be adequate for some screening situations, adjustments in referral criteria for the Spot may result in continued improvements in the future. We found improved specificity with the most recent Spot update (v. 2.0.16).

The 2013 AAPOS amblyopia risk factor criteria are intended to be used with the gold-standard examination to identify true positives and to serve as a standard for comparison of screening techniques. When we substituted these criteria for the manufacturer's out-of-the-box critera, we found that the sensitivity and specificity of the Spot decreased, supporting the recommendation that the 2013 AAPOS criteria not be used directly in photoscreening device criteria.

Ideally, photoscreening device criteria are chosen for high specificity for amblyopia detection in young children and high sensitivity in older children.¹⁰ We found both higher sensitivity and specificity in older children and a trend toward improved PPV with age with the revised Spot critera (v. 2.0.16). Differences in sensitivity between the three age groups were not significant. The targeted age for automated screening is the preschool age group. Although half of our subjects were beyond preschool age, photoscreening techniques have been shown to be as effective in children younger than 3 years of age as in older children.¹⁴ In the youngest age group, the sensitivity was 81.8% and specificity 56.7%. These children are usually too young to cooperate with traditional acuity and effective automated screening allows risk factors to be detected at this young age.

Changes in referral criteria alter sensitivity and PPV. Our estimated PPV of 48% for the general population indicates that almost half of the children referred by the Spot would be found to have an amblyopia risk factor. Identification of appropriate automated screening tools and criteria should consider the patient population, needs of the community, and the availability of follow-up care.

This study is limited by our testing of a high-risk population, which would be expected to decrease testability and alter the PPV and NPV. We provided the estimated PPV and NPV in the general population for reference. In addition, our sample size was only sufficient to detect larger differences between the two younger age groups and further analyses at these ages is recommended.

In conclusion, our prospective study found that the updated Spot offers good specificity and sensitivity, comparable to those reported by more validated automated screeners^15,6,16 and slightly improved from previous versions.