Agreement and diagnostic accuracy of vision screening in preschool children between vision technicians and spot vision screener

Abstract

Purpose:

Screening preschool children for vision-related disorders poses a challenge. This study is designed to determine the agreement and diagnostic accuracy of the spot vision screener (SVS) in screening preschool children compared to screening procedure by vision technicians (VT).

Methods:

This study was conducted as a part of the ongoing study titled “Initiative for Screening Children for Refractive Errors and other Eye Health Needs (I-SCREEN).” Children from 33 Anganwadis (preschools) in two districts, Adilabad district of Telangana and Krishna district of Andhra Pradesh, in South India, underwent eye health screening by a VT and by a trained community eye health workers (CEHW) using the SVS. Findings were compared for agreement and diagnostic accuracy of assessment.

Results:

A total of 976 preschool children were screened by the VT and separately by the CEHW using the SVS in Adilabad (15 schools) and Krishna (18 schools) districts. The overall mean age of these children was 2.5 years (SD ± 1.3 years). There were 48 (4.9%) referrals by VT compared to 105 (10.8%) referrals by CEHW using SVS. The overall sensitivity of SVS was 91.7% (95% CI: 80%–97.7%) and the specificity was 93.4% (95% CI: 91.6%–94.9%). Positive predictive value was 41.9% (95% CI: 32.3%–51.9%) and negative predictive value was 99.5% (95% CI: 98.8%–99.9%) with a moderate agreement (0.54; 95% CI 0.49–0.64) between VT screening and screening with SVS.

Conclusion:

The SVS showed good diagnostic accuracy and agreement in screening for possible vision-related disorders in preschool children.

Vision screening in preschool children poses specific challenges due to the difficulty in measuring vision in these children. It has been shown that in order to increase awareness of vision problems/refractive error in this age group, it is essential to screen preschool children.[1] The US Preventative Services Task Force now actively recommends vision screening at least once for children between 36 months and 5 years of age and states explicitly that photo screening is an appropriate screening technology.[2] Early detection of a higher degree of refractive error may prevent amblyopia in children at risk and allow for simple refractive correction rather than active therapies at an age when amblyopia has not yet become entrenched. Therefore, major professional organizations concerned with children's vision and childhood blindness, including the American Academy of Ophthalmology, the American Association for Paediatrics Ophthalmology and Strabismus, and the American Association of Certified Orthoptists, have recommended that vision screening should begin in early childhood.[3] According to a study by Joish et al., the net benefit to society was the greatest when vision screening was performed in preschool children compared to school-age children.[4] Forcina et al. demonstrated good sensitivity and specificity of a photo-screener for detecting amblyopia risk factors in children aged 6 months to 35 months.[5] Children with amblyopia risk factors who underwent photo screening before 2 years of age have better visual outcomes than those screened later.[6] Early correction of refractive error is also known to improve visual acuity outcomes.[7]

Given the limitations of chart-based vision screening, several professional groups recommend replacing its use in young children with instrument-based screening using portable photo-screeners or autorefractors. In a 2016 policy statement by the American Academy of Pediatrics, experts indicated that instrument-based vision screening can be attempted beginning at 12 months of age in a community setting.[5] Numerous photo-screener devices have been used, and the current generation devices demonstrate high sensitivity and specificity relative to a gold standard comprehensive eye exam.[5,8,9,10,11,12,13] In a cluster randomized trial conducted in Boston to test the implementation of instrument-based screening compared to chart-based screening in children aged 3 to 5 years, completion of screening increased from 39% with chart-based screening to 87% with instrument screening. Instrument-based screening was also associated with 15% reduction in referrals in eye care.[8] Most of these devices have been used in a clinical setting and there is no data on its accuracy when used in population-based studies in developing countries in a community setting.

Our study was specifically designed to look at the diagnostic accuracy of screening with the spot vision screener (SVS) in preschool children as compared to screening by a vision technician (VT) in a community setting, i.e., compare two screening modalities. The Welch Allyn Spot Vision Screener; model VS100 is a photoscreener that is a noninvasive, portable instrument which captures vision data for both eyes in all age groups. It also measures pupil size, interpupillary distance, and gaze deviation. This instrument helps in identifying patients requiring further eye examinations as referrals for significant refractive error, anisometropia, anisocoria, and strabismus.[3,13,14,15] The advantage of using an instrument-based screening device is the ease of its applicability to the pediatric primary care setting without specialized training. It can also be used easily in preverbal children, children who are preliterate, and those with developmental delays.[8,16]

Methods

The study protocol was approved by the Institutional Review Board of L V Prasad Eye Institute and followed the tenets of the Declaration of Helsinki. This study is a part of the larger ongoing project on the “Initiative for Screening Children for Refractive Errors and other Eye Health Needs (I-SCREEN).” I-SCREEN is being carried out in four districts in two Indian states of Telangana and Andhra Pradesh [Fig. 1].[17] The L V Prasad Eye Institute pyramidal model for eye care delivery consists of a centre of excellence at the top catering to 50 million population followed by tertiary centres, each for 5 million population. At the next level, there are secondary centres covering 0.5–1 million population, followed by vision centres at primary level for 50,000 population, and vision guardians for 5,000 population. The functions at each level of the pyramid are clearly delineated and demarcated. The secondary centres are run by one or two ophthalmologists who are trained at tertiary centres or centre of excellence for a year. Patients from secondary centres are referred to tertiary centres or centre of excellence only for advanced care and management of complex problems. The vision centres are staffed by a VT.[18] VT are eye-care personnel who run primary vision centres. They are the local youth who have completed high school, and are trained for 2 years to provide primary eye care, including a basic eye examination, refraction, dispensing spectacles, and appropriate referrals. A VT helps in identifying common eye conditions that can cause vision loss in children. The VT training programme has been accredited by the National Skill Development Council of India.[19]

Figure 1.

Map showing study districts for I-SCREEN Andhra Pradesh and Telangana

This study involved screening of all children in preschools (Anganwadis) in Adilabad and Krishna districts. As a part of the Integrated Child Development Scheme, the Government of India started “The Anganwadi Program” in 1975. The program caters to children in the 0–6 years age group.[20] Written informed consent was obtained from the person in charge (Anganwadi teacher/worker) at each Anganwadi, after explaining the importance of the current study. A copy of the consent form was provided for their records.

In a comprehensive school eye health program (I-SCREEN), the key is to screen all preschoolers and school-going children with the help of trained VT.[17,21,22] In order to reach preschool and school-going children in remote rural parts of the country, a screening procedure is required that can be replicated across the country. In this study, we utilized community eye health workers (CEHW) for screening rural areas. However, they too have difficulty in screening preschool children. As it is difficult to have ophthalmologists or optometrists in rural areas, we identified VT for conducting screening these preschool children as initial step. As it was difficult to have VT at all Anganwadis as well as perform cycloplegic refraction for all children in community setting, there was a need for an alternative method. Hence, CEHW were provided with SVS and trained to use SVS for screening. This study compares the diagnostic accuracy of screening with SVS which compared to the screening by VT for the same preschool children.

For testing the accuracy of the SVS in our study, 15 Anganwadis in the districts of Adilabad and 18 Anganwadis in Krishna were randomly selected. All children in these preschools were screened by three VTs (two in Adilabad district and one in Krishna district). The same children were screened by a CEHW using SVS. There was a good agreement between these VTs for screening preschool children (overall kappa between VT1 and VT2 was 0.72, VT1 and VT3 was 0.68, and VT2 and VT3 was 0.76). The protocol for screening by VTs included collecting demographic information related to each child from the Anganwadi records. For vision testing, different vision tests were used depending on the age and response of the child. Initially, 6/12 tumbling E optotypes or an alphabet chart was used and if the child had difficulty, HOTV or a Lea symbol chart was used at a distance of 3 m, in ambient illumination. If a child could not understand any of these, finger puppets were used to assess the fixation pattern. In addition to this, distant direct ophthalmoscopy was done to rule out media opacities. The children with one or a combination of the factors, such as failure in vision screening and/or absence of red glow, were referred to the next level of care, i.e., secondary centre, where there is an ophthalmologist and cycloplegic refraction could be performed. Data was collected in prescribed forms. The study coordinator collected and verified the completed forms before forwarding them for data entry.

The same children were then screened by a CEHW using the SVS on the same day. To assess the child, the CEHW holds the SVS approximately three feet from the child, and twinkling lights and sounds provide a fixation target for the child. With a cooperative child in a dimly lit room, a reading is obtained in about 2 seconds. If the reading was not obtained in the initial attempt, several attempts were made until they succeeded. With a successful reading, the device immediately displays a report of the pupillary diameter, ocular alignment, binocular refraction, and a recommendation for a referral.[5] Depending on the result obtained on the SVS, the child was designated as “not referred” or “referred.” The VT and CEHW were masked to the results of each other during screening. The entire process was overseen by a local coordinator who also ensured that all of the examiners were masked to the findings of each other. Findings of the VTs were compared against findings of SVS for agreement and diagnostic accuracy.

Sensitivity was defined as the ability of the SVS to recommend referral for further eye examination among those who were referred by the VT; and specificity was defined as the ability of SVS to recommend no referral for those children confirmed as normal by the VT. Positive predictive value (PPV) was defined as the proportion of children who were recommended referral by SVS as compared those identified by the VT, and the negative predictive value (NPV) was defined as the proportion of children who were not recommended referral among those found to be normal by the VT.

Agreement (kappa statistic) and diagnostic accuracy (sensitivity, specificity, and predictive values) were calculated comparing the final SVS data with the reference standard of the VTs. Agreement was graded as poor (<0.2), fair (0.21–0.4), moderate (0.41–0.60), good (0.61–0.8), and excellent (0.81–1.0).

Results

A total of 976 preschool children were screened by both the VT and the CHEW using SVS from 33 Anganwadi schools in Adilabad (15) and Krishna (18) districts. In Adilabad district, 375 (38.4%) children were screened and in Krishna, 601 (61.6%) children were screened. The overall mean age of these children was 2.5 years (SD ± 1.3 years). There were 477 (48.9%) girls and 499 (51.1%) boys.

Table 1 shows the referrals by VT as well as the CEHW using SVS in both the districts and the total. Overall, there were 48 cases (4.9%) referred by VT and 105 cases (10.8%) referred by CEHWs using SVS.

Table 1.

Referrals by the vision technician and SVS in Adilabad and Krishna districts

Adilabad district
Vision Technician	Spot Vision Screener
		Yes	No	Total
	Referred	19	4	23
	Not referred	10	342	352
	Total	29	346	375
Krishna district
Vision Technician	Spot Vision Screener
		Yes	No
	Referred	25	0	25
	Not referred	51	525	576
	Total	76	525	601
Both districts
Vision Technician	Spot Vision Screener
		Yes	No
	Referred	44	4	48
	Not referred	61	867	928
	Total	105	871	976

Table 2 shows the diagnostic accuracy of the VTs with that of the SVS, in both districts. The overall sensitivity of the SVS in screening and referring a child was 91.7% (95% CI: 80%–97.7%) and the specificity was 93.4% (95% CI: 91.6%–94.9%). The overall PPV was found to be 41.9% (95% CI: 32.3%–51.9%) and the NPV was 99.5% (95% CI: 98.8%–99.9%). There was moderate agreement (0.54; 95% CI 0.49–0.64) between VT and SVS in both districts with moderate agreement in Krishna district (0.46; 95% CI 0.34–0.58) and good agreement in Adilabad district (0.71; 95% CI 0.57–0.85).

Table 2.

Referral accuracy of the spot vision screener compared with the vision technician

	Kappa (95% CI)	Sensitivity (%) (95% CI)	Specificity (%) (95% CI)	Positive predictive value (%) (95% CI)	Negative predictive value (%) (95% CI)
Adilabad	0.71 (0.57-0.85)	82.6 (61.2-95)	97.2 (94.8-98.6)	65.5 (45.7-82.1)	98.8 (97.1-99.7)
Krishna	0.46 (0.34-0.58)	100 (86.3-100)	91.1 (88.5-93.3)	32.9 (22.5-44.6)	100 (99.3-100)
Both district	0.54 (0.45-0.64)	91.7 (80-97.7)	93.4 (91.6-94.9)	41.9 (32.3-51.9)	99.5 (98.8-99.9)

Discussion

This is one of the first population-based studies in a developing country comparing the diagnostic accuracy of an instrument-based screening device in preschool children with screening by a VT in a community setting. The SVS showed high sensitivity and specificity in screening for vision-related abnormalities in preschool children. The overall sensitivity and specificity of SVS for screening were found to be good and comparable with other studies where pediatric ophthalmologist was used [Table 3].[5,9,11,13]

Table 3.

Studies showing results of diagnostic and referral accuracy of the spot vision screener

Author	Country/Region	Instrument used	Total children	Age (Mean and range)	Gold standard	Sen (%)	Spe (%)	PPV (%)	NPV (%)
Qian et al.[12]	China	1.Spot photoscreener (v2.1.4) 2. SW 800 vision screener (v1.0.1.0)	113	5.2 (4-6)	Optom	94 88.8	80 81.1	NA	NA
Forcina et al.[5]	South Carolina (USA)	Welch Allyn Spot Vision Screener (version 2.0.16)	184	23.3 months, (6-35 months)	PO	89.8	70.4	58.9	93.6
Arana Mendez et al.[9]	Costa Rica	Welch Allyn Spot Vision Screener (version 2.0.16)	219	60 months, (20-119 months)	PO	92.6	90.6	58.1	98.9
Peterseim et al.[11]	South Carolina (USA)	Spot Vision Screener 1.version 1.1.51 2.version 2.0.16	444	72 months, (11-221 months)	PO	88.1 87.7	71.9 75.9	79.3 81.7	83.1 83.4
Silbert and Matta[13]	U.S.A	Spot Photoscreener	151	NA	PO	87%	74%	NA	NA
Arnold and Armitage[10]	Anchorage Alaska	1. Plusoptix SPOT 2. iScreen 3. iScreen DCC 4. GoCheckKids	108	47 months (9 to 146 months)	Ortho and PO	83% 80% 75% 81%	88% 85% 88% 91%	87% 87% 89% 92%	NA

*PO=Pediatric Ophthalmologist; Sen=Sensitivity; Spe=Specificity; PPV=Positive Predictive Value; NPV=Negative Predictive Value; NA=Not available; Optom=Optometrist; Ortho=Orthoptist

In terms of predictive values, the overall PPV was 41.9%, and NPV was 99.5%. Very few studies have reported PPVs and NPVs in addition to sensitivity and specificity.[5,9,11] Forcina et al. from South Carolina reported a PPV of 58.9% and NPV of 93.6%. However, they had a small sample of 184 children, and the overall prevalence of an eye condition was 32.1%. Using a prevalence of 20% of the children with an amblyopia risk factor, the PPV and NPV were estimated to be 43.1% and 96.5%, respectively.[5] Similarly, Arana Mendez reported a PPV of 58.1% and NPV on 98.9% in a sample of 219 children, and the overall prevalence of any eye condition was more than 50%.[9] Peterseim et al. reported a PPV and NPV of more than 80% and the overall prevalence of any eye condition was 59%. However, with a population prevalence of 20%, the PPV was 48% and NPV was 96%.[11] In our population-based study, the overall prevalence of referral was 4.9%. As PPV is dependent on the prevalence of any eye condition, the low prevalence in our population can explain the difference in PPVs from other studies. Apart from this, all these studies the gold standard was cycloplegic refraction by an optometrist or pediatric ophthalmologist. In our study, it was comparing one screening method versus other as it was difficult to have optometrist screen in rural setting as well as cycloplegic refraction could not be performed on these children due to lack of consent from parents as well as associated complications with performing cycloplegia in the field.

This study concluded with a moderate agreement between the VTs and SVS. The reason for a moderate agreement could be related to the low prevalence of vision-related abnormality in our population as well no cycloplegic refraction being performed. However, we could not find an explanation for moderate agreement between VT and SVS in Krishna district and good agreement in Adilabad district and this need further exploration.

One of the limitations of this study was that screening was not done with cycloplegic refraction in these children and we might have missed number of children with hyperopia. While the VT could identify only 48 children with ocular morbidity, the SVS could identify 105. It is likely that some of them could have been hyperopic and missed by the VT. Hence, as a next step, it is proposed to do a compare the accuracy of SVS with a cycloplegic refraction by a pediatric optometrist or an ophthalmologist as the gold standard.

A positive outcome of our study was that screening a large number of preschool children with reasonable accuracy and without depending on trained personnel was shown to be a possibility, which would not have been conceivable in the past.

Conclusion

In conclusion, the importance of screening preschool children, to identify those at risk of refractive error and amblyopia, has been emphasized often. Most of time, screening at the primary care level is compromised due to inadequate number of trained vision technicians (VT) or availability of optometrist and a large percentage of children not attending school. The cost of detecting and treating amblyopia is quite reasonable, with a ratio of cost to quality-adjusted life years for amblyopia screening estimated at $6,000 which is significantly less than that of other screening (e.g., screening for diabetic retinopathy).[13] The SVS system is a useful alternative for remote locations to provide eye care services for children who have no access to or are unable to access care.[23] SVS has proven to be a useful tool, as it is portable, and a quick and easy to operate. A high sensitivity index makes it a reliable tool for replacing VT at least at the screening level. However, it should be understood that, at this point, it is just a screening modality and not a replacement of a cycloplegic refraction done by an optometrist or pediatric ophthalmologist and a further study is warranted to compare the accuracy of SVS with a cycloplegic refraction by a pediatric optometrist or an ophthalmologist as the gold standard.

Financial support and sponsorship

Lions Club International Foundation, USA, Lavelle Fund for the Blind, USA, Sun Pharma Corporate Social Responsibility grant and Hyderabad Eye Research Foundation.

Conflicts of interest

There are no conflicts of interest.