Validity of the Convergence Insufficiency Symptom Survey: A Confirmatory Study

Abstract

Purpose

The objectives of the present study were to evaluate whether investigator bias influenced the Convergence Insufficiency Symptom Survey (CISS) scores of children with normal binocular vision (NBV) in our original validation study, reevaluate the usefulness of the cut-off score of 16, and reexamine the validity of the CISS.

Methods

Six clinical sites participating in the Convergence Insufficiency Treatment Trial (CITT) enrolled 46 children 9 - <18 years with NBV. Examiners masked to the child’s binocular vision status administered the CISS. The mean CISS score was compared to that from the children with NBV in the original, unmasked CISS study and also to that of the 221 symptomatic CI children enrolled in the CITT.

Results

The mean (±SD) CISS score for 46 subjects with NBV was 10.4 (±8.1). This was comparable to that from our prior unmasked NBV study (mean = 8.1(± 6.2); p = 0.11), but was significantly different from that of the CITT CI group (mean = 29.8 ± 9.0; p < 0.001). Eighty-three percent of these NBV subjects scored less than 16 on the CISS, which is not statistically different from the 87.5% found in the original unmasked study (p = 0.49).

Conclusions

Examiner bias did not affect the CISS scores for subjects with NBV in our prior study. The CISS continues to be a valid instrument for quantifying symptoms in 9 to <18 year-old children and these results confirm the validity of a cut-point of ≥ 16 in distinguishing children with symptomatic CI from those with NBV.

The Convergence Insufficiency Symptom Survey (CISS) was designed to quantify the severity of symptoms associated with convergence insufficiency (CI). Initial studies indicated good construct validity and reliability (1, 2) and later studies confirmed the validity and reliability of the revised version of the survey. (3, 4) The latter version has since been used as the primary outcome measure for three Convergence Insufficiency Treatment Trial (CITT) pilot studies (5, 6, 7) and was used in the CITT, a large-scale randomized trial evaluating the effectiveness of active treatments for symptomatic CI in children. (8, 9)

The symptom score distributions of children with 3-sign symptomatic CI and children with NBV overlap. Obviously, any cutoff score is a compromise between incorrectly classifying one group or the other as symptomatic or asymptomatic. The symptom score of 16 or greater resulted in a few NBV children (12.5% original study) being classified as symptomatic, while the majority of the CI children were classified as symptomatic (87.5% original study).(3) However, the symptom score data used to determine the cut-point of 16 was collected by examiners who were not masked to the subjects’ visual status. Although the protocol for administering the CISS requires that each question be read verbatim by the examiner, it is possible that the unmasked examiners could have consciously or subconsciously influenced the child’s responses, thereby introducing interviewer bias. (12)

The objectives of the present study were to evaluate whether investigator bias influenced the CISS scores of children with NBV in our original validation study, reexamine the validity of the CISS, and reevaluate the usefulness of the cut-off score of 16 when administered by masked examiners.

METHODS

The study was supported through a cooperative agreement with the National Eye Institute of the National Institutes of Health and conducted by the CITT Group at 6 clinical sites (see appendix). The respective institutional review boards approved the protocol and HIPAA-compliant informed consent forms. The parent or legal guardian of each study subject gave written informed consent and written assent was obtained from each child. Study oversight was provided by an independent data and safety monitoring committee appointed by National Eye Institute.

Subject Selection

CITT-trained and certified optometrists or ophthalmologists using a previously described standardized protocol performed all testing (baseline and masked). An unmasked examiner performed eligibility testing, which included the following: best-corrected visual acuity at distance and near; cover testing at distance and near with objective prism neutralization; near point of convergence; positive and negative fusional vergence at near (fusional convergence and divergence amplitudes); near stereoacuity; monocular accommodative amplitude; and monocular accommodative facility (the ability to quickly achieve clear vision while alternately viewing 20/30 equivalent print through +2 D and −2 D lenses); cycloplegic refraction with 1% cyclopentolate; and an ocular health evaluation. All near testing with at 40cm. A masked examiner administered the CISS.

Major eligibility criteria for the study included best-corrected visual acuity at distance and near of 20/25 or better, no strabismus, heterophoria at near between 2Δ esophoria and 8Δ exophoria, near point of convergence closer than 6.0 cm break, negative fusional vergence at near greater than 7Δ BI-break and 5Δ BI-recovery, positive fusional vergence at near greater than 10Δ BO-break and 7Δ BO-recovery, monocular amplitude of accommodation in diopters greater than 15 minus 25% of the child’s age, and at least 500 seconds of arc of random dot stereopsis on the Randot® Stereotest (Stereo Optical Co, Chicago, IL). A refractive correction was required when the magnitude of uncorrected refractive error or change in refractive error (based on a cycloplegic refraction performed within 2 months) in either eye differed from the current prescription by 0.50 D or more in spherical equivalent of myopia, 1.50D or greater in spherical equivalent of hyperopia, or 0.75 D or greater of astigmatism. Table 1 has the complete listing of eligibility and exclusion criteria.

Table 1.

Eligibility Criteria for Normal Binocular Vision Subjects.

Inclusion Criteria

Age 9 to < 18 years Sex: either Ethnicity: any Best-corrected visual acuity ≥ 20/25 in both eyes at distance and near Appropriate refractive correction worn for at least 2 weeks (see below) Heterophoria at near between 2Δ esophoria and 8Δ exophoria Negative fusional vergence at near (greater than 7Δ BI-break/5Δ BI-recovery) Positive fusional vergence at near (greater than 10Δ BO-break/7Δ BO-recovery) NPC closer than 6.0 cm break Monocular amplitude of accommodation (greater than 15–0.25*age) Appreciation of random dot stereopsis using a 500 seconds of arc target Cycloplegic refraction within past 2 months Informed consent and willingness to participate in the study

Exclusion Criteria

Previous treatment for CI (Any office- or home-based vision therapy or more than 2 weeks of pencil push-up therapy) Previous use of plus add at near or base-in prism Amblyopia (≥ 2 line difference in best-corrected visual acuity between the two eyes) Constant strabismus History of strabismus or refractive surgery Anisometropia ≥ 2.0D spherical equivalent High refractive error: Myopia ≥ 6.00D sphere (in any meridian); Hyperopia ≥ 5.00D sphere (in any meridian); Astigmatism ≥ 4.00D Vertical heterophoria greater than 1Δ Diagnosed with multiple sclerosis, Grave’s thyroid disease, myasthenia gravis, diabetes, or Parkinson’s disease Manifest or latent nystagmus Developmental disability, mental retardation, ADHD or learning disability that, in the investigators opinion, would interfere with the child’s ability to respond to survey items

Procedures

To accomplish examiner masking, the subjects enrolled into this study were evaluated during the course of the main CITT study when potentially eligible children were undergoing eligibility examinations or subjects already enrolled into the CITT were returning for study-mandated follow-up visits after 4 weeks, 8 weeks, or 12 weeks of treatment or 6 months from the 12-week outcome examination. Although masked examiners were aware that children who presented for masked examinations could have been enrolled into the present study or the CITT, all scheduling procedures and data forms provided to the masked examiner were identical to prevent unmasking. Because all sites had more than one examiner, it did not matter if the examiner did not recall seeing the particular child in the past, as it was possible that the child had been seen previously by a different masked examiner.

Enrollment and Subject Inclusion

Between May 2006 and March 2007, 51 children were enrolled from six clinical teaching centers (see Appendix) that were currently participating in the CITT. NBV subjects either were at the center for their annual examination and were asked to participate or responded to advertisement to participate in the study that was sent to the college community. The number of subjects per site ranged from 8 to 9 with a median of 8.5. Based on a post-enrollment review, it was determined that two children did not meet the eligibility criteria for accommodative amplitude and therefore they were excluded from data analyses. Because our original unmasked study of children with NBV excluded those with Attention Deficit Hyperactivity Disorder (ADHD) and our preliminary work has shown that those with self-reported ADHD score higher on the CISS, (13) we also excluded three children with self-reported ADHD. Therefore, the data analyzed and reported herein are from the remaining 46 children.

Statistical Methods

The sample size for the study was based on constructing a confidence interval for the percentage (87.5%) of NBV subjects scoring less than 16 on the CISS in the original validation study. (3) If we assume percentage values from 75% to 95%, a sample size of 45 would allow us to calculate a 95% confidence interval with precision ranging from 0.13 (for 75%) to 0.05 (for 0.95%). Thus, with a sample of 45 subjects, the confidence interval for the specificity would be no more than 0.26 (2*0.13) units wide.

In addition, a post-hoc power analysis for non-inferiority was performed. The mean square error from the adjusted analysis was used as an estimate of the variability for comparing the two NBV groups. This study has 80% power to detect a difference of more than 3.5 points in the mean symptom score of the masked and unmasked groups.

A two-sample t-test was used to compare the mean CISS score obtained from the children with NBV in this study to those in our original unmasked study and also to the children with CI currently enrolled in the CITT. Analysis of covariance methods were used to compare the mean CISS scores while controlling for any demographic or clinical measure found to differ between the groups. In addition, Levene’s F-test was used to compare the variability in the group of subjects with NBV reported herein to both the original sample of subjects with NBV and the CI subjects in the CITT. Frequency tables were used to determine the percentage of subjects with NBV who exhibited CISS scores between 14 and 20. A chi-square test was used to compare the percentage of subjects with NBV scoring below 16 with the 87.5% found in our initial CISS study. All reported P values are 2 tailed. Analyses were conducted using SAS software version 9.1 (SAS Institute, Cary, North Carolina, USA).

RESULTS

Demographic and clinical characteristics for the NBV subjects are shown in Table 2 along with the corresponding data for the 56 NBV subjects enrolled in the original unmasked study (3) and the 221 subjects enrolled into the CITT (8, 9).

Table 2.

Summary statistics for clinical and demographic measures from the enrollment visit, by study.

	Subjects with NBV		Symptomatic
Characteristic	Masked (N=46)	Unmasked (n=56)	CI subjects (N=221)
Mean (SD) age in years	12.5 (2.4)	11.4 (2.2)	11.8 (2.3)
% Girls	60.9	45.5	59.3
Race
% American Indian/Alaskan Native	0.0	0.0	4.6
% Asian/Pacific Islander	2.2	4.3	1.8
% Black or African American	45.7	42.6	29.7
% White	32.6	51.1	54.8
% Other	19.6	2.1	9.1
% Hispanic or Latino	13.0	14.6	34.4
% self-reported ADHD	0.0	0.0	15.4
% visual acuity 20/20 equivalent or better at near	89.1	100.0	81.0
Mean (SD) Spherical Equivalent – Right Eye (D)	−0.75 (2.1)	−0.66 (1.6)	−0.08 (1.5)
Refractive error category – Right eye
% Myopic (more than −0.50D SPHEQ)	30.4	38.9	22.6
% Hyperopic (more than +1.00D SPHEQ)	0.0	3.7	9.1
% Emmetropic	69.6	57.4	68.3
% Glasses wearer	37.0	n/a	34.4
Mean (SD) Near Point of Convergence (cm)
Break	3.5 (1.2)	3.7 (1.1)	14.2 (7.5)
Recovery	5.2 (1.7)	5.4 (1.4)	17.9 (8.2)
Mean (SD) Positive Fusional Vergence (Δ)
Blur/Break	24.0 (10.2)	24.8 (8.4)	10.9 (3.9)
Recovery	22.1 (7.7)	20.2 (8.0)	8.8 (4.5)
Mean (SD) Phoria (Δ)
At Near	2.1 exo (2.3)	1.7 exo (2.3)	9.3 exo (4.4)
At Distance	0.6 exo (1.3)	0.6 exo (1.3)	1.9 exo (2.8)
% with Intermittent Tropia
At Near	0.0	0.0	10.4
At Distance	0.0	0.0	2.7
% failed Sheard’s criterion	2.2	0.0	82.3
Mean (SD) Negative Fusional Vergence (Δ)
Blur/Break	12.8 (4.4)	13.8 (6.2)	11.2 (4.7)
Recovery	11.0 (3.6)	11.0 (5.6)	9.9 (4.4)
Mean (SD) Monocular (O.D.) Accommodative Amplitude (D)	16.2 (4.1)	17.8 (6.5)	9.9 (3.8)
% with Accommodative Insufficiency	0.0	5.5	54.8
Mean (SD) Monocular (O.D.) Accommodative Facility (cycles/min)	8.9 (5.8)	11.8 (3.6)	6.5 (4.4)

Comparison of NBV Study Groups

NBV subjects enrolled in the present masked study were slightly older compared to those enrolled in the original unmasked study (p=0.018). Subjects in the current masked study were also slightly less likely to be White (p = 0.032) and monocular accommodative facility was approximately 3 cycles per minute less (p=0.004) compared to those in the original study. There were no discernable or statistically significant differences in the clinical measures of phoria (near and distance), near point of convergence (break and recovery), and positive fusional vergence (blur/break and recovery) between the two NBV groups.

The distribution of CISS scores for the NBV subjects in the present masked study and those from the original unmasked study are shown in Figure 1. The mean CISS score when examiners were masked at administration (10.4) was not significantly different from that found when the examiners were unmasked (8.1) (p = 0.11). Controlling for the observed differences in age, race, and accommodative facility, the adjusted mean CISS scores of 11.1 in the present study and 10.2 in the original study were also not different (p=0.59). Additional analyses were performed to compare the variability of the CISS scores obtained from the two NBV study samples. The standard deviation of scores when examiners were masked was 8.1 and in the original unmasked study was 6.2 (p = 0.058).

Figure 1.

Distribution of CI Symptom Survey score, by study sample.

Masked NBV Study Subjects Compared to CI Subjects

As shown in Table 2, the NBV subjects in this study were slightly more myopic when compared to the CITT patients (p=0.041); however, a similar percentage in both groups reported wearing glasses (p=0.75). Because the CITT and this masked NBV study had different inclusion criteria for near point of convergence, positive fusional vergence, phoria, and accommodative amplitude, therefore it is not surprising that there were large differences between the two groups in respect to these characteristics. Differences were also observed in negative fusional vergence blur/break (p=0.036) and accommodative facility (p=0.01).

While the CITT subjects were primarily White (54.8%), nearly half (45.7%) of the subjects in this NBV study were Black and a little less than one-third (32.6%) were White. In this study, subjects were less likely to be Hispanic (13.0% compared to 34.4%). The differences in distribution of both race (p=0.014) and ethnicity (p=0.004) are most likely because the Miami, FL site did not participate in this masked NBV study.

Subjects enrolled in the CITT scored, on average, 29.8 (SD=9.0) on the CISS at the eligibility examination, which was significantly higher than the mean score (10.4) among the masked NBV study subjects (p < 0.001). (3) The difference in mean CISS scores remained after controlling for the observed differences in age, spherical equivalent refractive error, race/ethnicity, and ADHD status among the CITT subjects. The adjusted mean CISS score for the NBV subjects was 13.0 compared to 32.3 for the CI subjects (p < 0.001). Caution, however, must be employed when examining the results of the comparison with the CI subjects because the distribution of CISS scores for CI subjects is truncated (i.e., one of the eligibility criteria for inclusion in the CITT study is a CISS score of 16 or greater).

Reassessment of the Current CISS Cut-off Point Value

As shown in Figure 2, 82.6% of the subjects enrolled in the masked study scored less than 16 on the CISS. This is not significantly different from the percentage found in the original unmasked NBV study in which 87.5% of the children scored less than 16 (p = 0.49).

Figure 2.

Distributions of CI Symptom Survey score for children with normal binocular vision (NBV) and children with convergence insufficiency (CI).

Table 3 provides the percentage of subjects falling below possible CISS cut-points ranging from 14 to 20 points for both the unmasked study and the masked study reported herein. For all 7 possible cut-points, the percentages falling below the cut-point obtained in the original NBV study with unmasked examiners are only slightly higher than those obtained in the current masked study.

Table 3.

Percentage of normal binocular vision study subjects scoring less than each cut-point of the CI Symptom Survey score, by study sample.

Cut-point	Masked study (n=46)	Unmasked study (n=56)
< 14	76.1	82.1
< 15	78.3	83.9
< 16	82.6	87.5
< 17	87.0	89.3
< 18	87.0	89.3
< 19	89.1	91.1
< 20	89.1	92.9

DISCUSSION

To learn whether investigator bias influenced the CISS scores of children with NBV in our original validation study (3), we evaluated whether the CISS scores were the same when the CISS was administered by examiners masked to the subjects’ visual status in this study as compared to when the CISS was administered by the unmasked examiners in our original study. The 2-point difference in mean scores from the original unmasked and the present masked study was not statistically significant; therefore, we found no evidence of investigator bias influencing the study outcomes in our original CISS study. Furthermore, the percentage of subjects in the present masked study who scored less than 16 points (82.6% or 38/46) on the CISS (and therefore were considered asymptomatic) was not statistically different from that found in our original unmasked study (87.5% or 49/56). The small difference in percentages was primarily due to a single highly symptomatic NBV subject (scored 46).

We can only speculate on why 8 subjects were symptomatic given our current data. Although it does appear four of the subjects had poor monocular accommodative facility (≤ 6 cpm), which has been associated with symptoms (14). The others met all inclusion criteria but may have had a binocular dysfunction that we did not assess (e.g., vergence facility), may have been interpreting normal physiological phenomenon as abnormal (15), may have had an undiagnosed reading or learning disorder, or the CISS may need further refinement. The one NBV subject with a very high symptom score could be an example of a subject’s conscious reaction when taking questionnaires known as “faking bad” in which a subject tries to appear sick to qualify for support. (12)

We were not surprised to find that examiner masking did not have a significant impact on the CISS mean score because the design and administration mode of the CISS allows little room for examiner influence. Common types of bias affecting questionnaires used in public health research arise from three basic sources:(12) 1) question design, 2) design of the questionnaire, and 3) administration of the questionnaire. The chance of bias in question design is limited for the CISS because it uses short, simple, unambiguous questions without technical jargon. The 5-point response scale provides a sensitive measure for detecting clinically significant change that avoids type II errors (i.e., the error of failing to observe a difference when in truth there is one) associated with measures with more limited categories. Because the questionnaire is administered in a face-to-face interview style problems associated with self-administered questionnaires are eliminated. (16) The questions are read aloud and in sequential order to the subject by the examiner as the subject holds a card containing the 5-response choices. If the subject does not understand the question or asks for further explanation, the examiner repeats the question verbatim without clarification and asks the subject to select one of the 5 response choices. This type of administration reduces the cognitive demand on the subject, which includes: comprehension of the question, recall of requested information from memory, evaluation of the link between the retrieved information and question, and communication of the response. (17) Because the 15 questions only take about 5 minutes to administer, problems of response fatigue that are often associated with face-to-face interviews are not present. All of these factors contribute to reducing the likelihood of interviewer bias.

The validity of the CISS is reinforced by the study results reported herein. Statistically significant differences were found in the mean CISS score between NBV and symptomatic CI children enrolled into the CITT, which were comparable to our previous study. (3) Nearly a 20-point difference or three fold increase in CISS scores of children with symptomatic CI was observed over CISS scores of children with NBV.

In conclusion, this study reinforces the validity and usefulness of the CISS as a primary outcome measure in clinical research and demonstrates that examiner masking did not have a significant effect on the CISS scores reported in our prior study. (3) The results of this study compared to our original unmasked study also suggest that the CISS may be used in a patient care setting, when evaluating children with the signs of CI, where masking is often not possible.

THE CONVERGENCE INSUFFICIENCY TREATMENT TRIAL INVESTIGATOR GROUP

Writing Committee

Michael W. Rouse, OD, MSEd, Annette Bade, OD, Carmen Barnhardt, OD, MSEd, Eric Borsting, OD, MS, Susan A. Cotter, OD, MS, Marjean Taylor Kulp, OD, MS, G. Lynn Mitchell, MAS, Mitchell M. Scheiman, OD, Tomohiko Yamada, OD

Clinical Sites

Sites are listed in order of the number of patients enrolled in the study with the number of NBV patients enrolled is listed in parentheses preceded by the site name and location. Personnel are listed as (PI) for principal investigator, (SC) for coordinator, (E) for examiner, and (VT) for therapist.

Study Center: Pennsylvania College of Optometry (9 NBV, 25 CI)

Michael Gallaway, OD (PI); Brandy Scombordi, OD (E), Mark Boas, OD (VT), Tomohiko Yamada, OD (VT), Ryan Langan (SC), Ruth Shoge, OD (E), Lily Zhu, OD (E)

Study Center: Southern California College of Optometry (9 NBV, 23 CI)

Susan Cotter, OD, MS (PI), Eric Borsting, OD, MS (E), Michael Rouse, OD, MSEd, (E), Carmen Barnhardt, OD, MS (VT), Raymond Chu, OD (VT), Susan Parker (SC), Rebecca Bridgeford (SC), Jamie Morris (SC), Javier Villalobos (SC)

Study Center: SUNY College of Optometry (8 NBV, 28 CI)

Jeffrey Cooper, OD (PI), Audra Steiner, OD (E, Co-PI), Marta Brunelli (VT), Stacy Friedman, OD (VT), Steven Ritter, OD (E), Lily Zhu, OD (E), Lyndon Wong, OD (E), Ida Chung, OD (E), Ashley Fazarry (SC)

Study Center: NOVA Southeastern University (8 NBV, 27 CI)

Rachel Coulter, OD (PI), Deborah Amster, OD (E), Gregory Fecho, OD (E), Tanya Mahaphon, OD (E), Jacqueline Rodena, OD (E), Mary Bartuccio, OD (VT), Yin Tea, OD (VT), Annette Bade, OD (SC)

Study Center - The Ohio State University College of Optometry (8 NBV, 24 CI)

Marjean Taylor Kulp, OD, MS (PI), Michelle Buckland, OD, MS (E), Michael Earley, OD, PhD (E), Gina Gabriel, OD, MS (E), Aaron Zimmerman, OD, MS (E), Kathleen Reuter, OD (VT), Andrew Toole, OD, PhD (VT), Molly Biddle, MEd (SC), Nancy Stevens, MS, RD, LD (SC)

Study Center: UAB School of Optometry (7 NBV, 28 CI)

Kristine Hopkins, OD (PI), Marcela Frazier, OD (E), Janene Sims, OD (E), Marsha Snow, OD (E), Katherine Weise, OD (E), Adrienne Broadfoot, MS, OTR/L (VT, SC), Michelle Anderson, OD (VT), Catherine Baldwin (SC), Leslie Simms (SC)

Study Center: Bascom Palmer Eye Institute (35 CI)

Susanna Tamkins, OD (PI), Hilda Capo, MD (E), Mark Dunbar, OD (E), Craig McKeown, MD (CO-PI), Arlanna Moshfeghi, MD (E), Kathryn Nelson, OD (E), Vicky Fischer, OD (VT), Adam Perlman, OD (VT), Ronda Singh, OD (VT), Eva Olivares (SC), Ana Rosa (SC), Nidia Rosado (SC), Elias Silverman (SC)

Study Center: University of CA San Diego: Ratner Children’s Eye Center (17 CI)

David Granet, MD (PI), Lara Hustana, OD (E), Shira Robbins, MD (E), Erica Castro (VT), Cintia Gomi, MD (SC)

Study Center: Mayo Clinic (14 CI)

Brian G. Mohney, MD (PI), Jonathan Holmes, MD (E), Melissa Rice, OD (VT), Virginia Karlsson, BS, CO (VT), Becky Nielsen (SC), Jan Sease, COMT/BS (SC), Tracee Shevlin (SC)

CITT Study Chair

Mitchell Scheiman, OD (Study Chair), Karen Pollack (Study Coordinator), Susan Cotter, OD, MS (Vice Chair), Richard Hertle, MD (Vice Chair), Michael Rouse, OD, MSEd (Consultant)

CITT Data Coordinating Center

Gladys Lynn Mitchell, MAS (PI), Tracy Kitts (Project Coordinator), Melanie Schray (Programmer), Linda Barrett (Data Entry), Loraine Sinnott, PhD (Biostatistician), Kelly Watson (student worker), Pam Wessel (Office Associate)

National Eye Institute, Bethesda, MD

Maryann Redford, DDS, MPH

CITT Executive Committee

Mitchell Scheiman, OD, G. Lynn Mitchell, MAS, Susan Cotter, OD, MS, Richard Hertle, MD, Marjean Kulp, OD, MS, Maryann Redford, DDS, MPH, Michael Rouse, OD, MSEd

Data and Safety Monitoring Committee

Marie Diener-West, PhD, Chair, Rev. Andrew Costello, CSsR, William V. Good, MD, Ron D. Hays, PhD, Argye Hillis, PhD (through March 2006), Ruth Manny, OD, PhD