A Comparison of Referral Criteria used by the PlusoptiX Photoscreener

Abstract

Purpose

To evaluate the sensitivity, specificity, and predictive value of 7 different referral criteria used for the plusoptiX photoscreener on the same cohort of children.

Methods

Retrospective chart review of patients presenting to a pediatric ophthalmology clinic who underwent plusoptiX photoscreening as part of a comprehensive examination. We applied multiple referral criteria from previously published studies as well as the manufacturer’s criteria in order to calculate specificity, sensitivity, and predictive value differences between the various referral criteria. We compared all criteria to the results of a pediatric ophthalmology examination based upon the 2003 American Association for Pediatric Ophthalmology and Strabismus (AAPOS) criteria, as well as the newly accepted revision of the AAPOS referral criteria.

Results

109 children were examined with a thorough pediatric ophthalmic exam and with the plusoptiX photoscreener. Of these, 58 (53%) were confirmed to demonstrate amblyopia risk factors, according to 2003 AAPOS criteria. The plusoptiX referral criteria were adjusted to match 7 different published plusoptiX referral paradigms so that the differing referral paradigms could be analyzed for sensitivity and specificity. When comparing the differing plusoptiX referral paradigms to 2003 AAPOS criteria, the sensitivity/specificity of the 7 different paradigms were respectively: Matta/Silbert 98%/80%, Arthur (2) 67%/96%, Arnold 81%/96%, Arthur 81%/92%, PediaVision 80%/94%, plusoptiX 98%/41%, AAPOS 74%/86%. When comparing the 7 differing referral paradigms to the newly approved (2013) AAPOS criteria, the sensitivity/specificity were respectively: Matta/Silbert 98%/68%, Arthur (2) 73%/92%, Arnold 92%/90%, Arthur 86%/85%, PediaVision 90%/92%, plusoptiX 98%/35%, AAPOS 87%/87%.

Conclusion

There are multiple referral criteria available for the plusoptiX photoscreener. Screening programs need to evaluate their own requirements with respect to desired sensitivity and specificity and decide on the most appropriate referral criteria for their program. The “Arnold” criteria is the best at maximizing sensitivity and specificity utilizing the 2003 “AAPOS” criteria and the “Arnold” and “PediaVision” were best at maximizing sensitivity and specificity for the newly accepted AAPOS referral criteria. Screening programs will need to decide the level of sensitivity and specificity that they wish to obtain, but for most screening programs the “Arnold” criteria may be preferred.

PURPOSE

The plusoptiX S04 is an objective pediatric vision-screening device that functions not only as a photo-screener, but also as a non-cycloplegic autorefractor. It has the ability to evaluate a number of amblyogenic risk factors. Based on the autorefraction readings, the device can trigger a referral if the measurements exceed a user-definable set of values for anisometropia, hyperopia, astigmatism, myopia, and/or anisocoria. The device will not obtain a reading unless both pupils are within 10 degrees of primary gaze, and will therefore trigger a referral for a patient with a significant heterotropia. The device may not be able to obtain a reading on a patient with pupillary abnormalities, and will also refer patients with conditions such as an iris coloboma, or ptosis of such a degree that a portion of the pupil is covered. Referral criteria for the plusoptiX have been published by the manufacturer,¹ as well as by Pediavision,² (a former US distributor of the plusoptiX), Arnold,³ Arthur,⁴ Matta/Silbert,⁵ and Nathan & Donahue (“Arthur2”).⁶ We compared these various referral criteria on the same group of patients to determine the sensitivity, specificity, and predictive values of each. We also programmed the plusoptiX to use the 2013 AAPOS referral criteria (Table 1).⁷ Notably, AAPOS referral criteria that were proposed at the time this research was conducted was approved in 2013.⁸

TABLE 1.

Various referral criteria for the plusoptiX.

	Age	Aniso	Astig	Myopia	Hyper	Anisocoria
plusoptiX1	6–12	≥1	≥1	≥2	≥3	≥1
	12–36	≥1	≥0.75	≥2	≥1	≥1
	36–72	≥1	≥0.75	≥1	≥1	≥1
	72–240	≥0.75	≥0.75	≥0.75	≥0.75	≥1
Matta/Silbert5	6–12	≥1.25	≥1	≥2	≥3	≥1
	12–36	≥1.25	≥1	≥2	≥1.25	≥1
	36–72	≥1.25	≥1	≥1	≥1.25	≥1
	72–240	≥1.25	≥1.25	≥1	≥1	≥1
Arthur4		>1.5	>1.25	>3	>3.5	>1
AAPOS7		>1.5	>1.5 or	>3	>3.5
			>1 oblique axis
Arnold3	0–8	≥1.5	≥2	≥3	≥3	≥1
	9–72	≥1	≥2	≥2.25	≥2.5	≥1
	73–120	≥1.25	≥1.5	≥1.5	≥2	≥1
PediaVision2	6–12	≥1.5	≥2.25	≥2	≥3.5	≥1
	12–36	≥1	≥2	≥2	≥3	≥1
	36–72	≥1	≥1.5	≥1.5	≥2.5	≥1
	72–240	≥1	≥1.5	≥0.75	≥2.5	≥1
	240–1200	≥1	≥1.5	≥0.75	≥1.5	≥1
Arthur 26		≥1.5	≥2.5	≥3.0	≥3.5

Age = age in months; Aniso = anisometropia, Astig = astigmatism, Hyper = hyperopia.

MATERIALS AND METHODS

Prior to starting this research we received Institutional Review Board approval through the Lancaster General Hospital. We received a waiver of consent due to the low risk of this research and followed appropriate Health Insurance Portability and Accountability Act of 1996 guidelines.

A retrospective chart review was conducted on 109 consecutive pediatric patients presenting to one pediatric ophthalmologist (DS). All patients had undilated plusoptiX photoscreening testing performed as part of a comprehensive examination. All children had received a cycloplegic refraction the day of their photoscreening, or within the prior 6 months. A determination of amblyopia risk factors was based upon the 2003 AAPOS referral criteria⁷:

• anisometropia (spherical or cylindrical)>1.5D

• any manifest strabismus

• hyperopia>3.5D in any meridian

• myopia>3.0D in any meridian

• any media opacity>1mm in size

• astigmatism >1.5D at 90 degrees or 180 degrees; >1.0D in oblique axis (more than 10 degrees from 90 degrees or 180 degrees)

• ptosis ≤1mm margin reflex distance

We then analyzed the same cohort of patients with the same plusoptiX referral criteria against the recently approved (2013) AAPOS referral criteria⁸:

• Age 12–30 months

• ○>2.00D astigmatism
• ○>4.50D hyperopia
• ○>2.5D anisometropia
• ○>−3.50D myopia

• Age 31–48 months

• ○>2.00D astigmatism
• ○>4.00D hyperopia
• ○>2.00D ansiometropia
• ○>−3.00D myopia

• Age>48 months

• ○>1.50D astigmatism
• ○>3.50D hyperopia
• ○>1.50D anisometropia
• ○>−1.50D myopia

• Manifest strabismus>8 prism diopters in primary position

• Media opacity>1mm

RESULTS

One hundred and nine children were analyzed. Fiftyeight children were found to have amblyopia risk factors (53%) based on the 2003 AAPOS referral criteria. Seven referral paradigms were then analyzed based on the plusoptiX results including those of “Matta/Silbert,” “Arnold,” “Arthur,” the “plusoptiX manufacturer,” “PediaVision,” “Arthur2,” and “AAPOS.”

Results are shown in Table 2 with the “Matta/Silbert” and “plusoptiX” referral criteria having the highest sensitivity and the “Arnold” and “Arthur2” referral criteria having the highest specificity. Receiver operator characteristic curve analysis of the 7 criteria employed demonstrated that the criteria closest to ideal, ie, perfect sensitivity and specificity, is the “Arnold” criteria for the 2003 AAPOS referral criteria (Figure 1). The ROC curve allows screening programs to quickly look at various referral criteria and decide which criteria are most appropriate for their screening program. A program may choose to maximize sensitivity (thus reducing the number of normal children referred) or maximize specificity (thus decreasing the number of children with amblyopic risk factors who are missed).

TABLE 2.

Sensitivity and specificity for the plusoptiX photoscreening comparing the various referral criteria against the 2003 AAPOS guidelines for amblyopia risk factors.

	Sensitivity	Specificity
Matta/Silbert	98%	80%
Arnold	81%	96%
Arthur	81%	92%
PediaVision	80%	94%
PlusoptiX	98%	41%
AAPOS	74%	86%
Arthur2	67%	96%

FIGURE 1.

ROC curve for the plusoptiX photoscreening comparing the various referral criteria against the 2003 AAPOS guidelines for amblyopia risk factors.

We then compared the same cohort of children using the same referral criteria against the recently accepted revisions to the AAPOS referral criteria. These results are shown in Table 3 with the “Matta/Silbert” and “plusoptiX” referral criteria having the highest sensitivity and the “PediaVision” and “Arthur2” referral criteria having the highest specificity. Receiver operator characteristic curve analysis of the 7 criteria employed demonstrated that the criteria closest to ideal are the “Arnold” and “PediaVision” criteria for the newly revised AAPOS referral criteria (Figure 2).

TABLE 3.

Sensitivity and specificity for the plusoptiX photoscreening comparing the various referral criteria against the recently accepted revisions to the AAPOS guidelines for amblyopia risk factors.

	Sensitivity	Specificity
Matta/Silbert	98%	68%
Arnold	92%	90%
Arthur	86%	85%
PediaVision	90%	92%
PlusoptiX	98%	35%
AAPOS	87%	87%
Arthur2	73%	92%

FIGURE 2.

ROC curve for the plusoptiX photoscreening comparing the various referral criteria against the recently accepted revisions to the AAPOS guidelines for amblyopia risk factors.

CONCLUSION

The ideal screening device would have 100% sensitivity (the ability to detect all targeted disease) and 100% specificity (the ability to ignore all non-targeted disease). In the real world, no screener has reached such a level of perfection. There is a recognizable inverse relationship between sensitivity and specificity illustrated by the receiver-operator characteristic curve.⁹ Community screening programs and/or pediatrician offices should choose referral cut-offs that provide the best combination of sensitivity and specificity for their goals. For screening programs in rural regions with poor access to pediatric eye specialists, the cost to perform confirmatory examinations may be quite high due to travel expenses and time lost from employment. In these circumstances screening programs may decide to sacrifice sensitivity, and rather accept a higher number of false negative responses to avoid unnecessary costs to patients and their families. Alternatively, for programs with infrequent screening and poor access to medical care, high sensitivity criteria may be preferred.

We recognize that our study population is not representative of the general population; rather it is a population of patients referred to pediatric ophthalmology because of their medical history. Other studies have had a lower prevalence of disease and these referral criteria may act differently on different populations.^10,11 However, this should not alter the sensitivity and specificity of the various referral criteria employed by the plusoptiX screener. These numbers can be utilized to estimate the numbers of false positive or false negative results in different populations with different prevalence of disease.

It should be understood that the AAPOS referral criteria were not developed to be directly programmed into any objective screening device including the plusoptiX photoscreener. In fact, AAPOS referral criteria were created as a measure to judge the failures generated by vision screening devices and tests in order to be able to compare one device to another with the goal of standardizing detection of amblyopia risk factors in preschool children. The AAPOS criteria were generated through a consensus of experts utilizing the best studies available at the time and were meant to be applied to a patient’s cycloplegic refraction, rather than the non-cycloplegic autorefraction gleaned by screening devices. Utilizing the AAPOS referral criteria directly in a screening device uniformly reduces sensitivity. This result occurs because the non-cycloplegic autorefraction provided by a screening device would be expected to underestimate hyperopia and overestimate myopia in patients with normal accommodation.

Based on this study, we recommend that vision screening programs utilizing the plusoptiX photoscreener should evaluate their own referral criteria and adjust them as needed. Depending upon the patient population and the needs of the examiners, the sensitivity and specificity of referral criteria are expected to vary, although the prime directive should be to identify children in need of a comprehensive pediatric ophthalmology examination. The “Matta/Silbert” criteria might be chosen for a screening program seeking high sensitivity with fairly good specificity. The “Arnold” criteria might be chosen for programs seeking high specificity with reasonable sensitivity, and performed well with both the former and accepted revision to the AAPOS referral criteria. Knowing that sensitivity will not be 100%, programs should strive to repeat pediatric vision screenings at regular intervals to maximize detection of amblyopia risk factors.