The Reliability and Validity of the English and Spanish Strengths and Weaknesses of ADHD and Normal Behavior Rating Scales in a Preschool Sample: Continuum Measures of Hyperactivity and Inattention

Abstract

Objective

To evaluate the reliability and validity of the English and Spanish versions of the Strengths and Weaknesses of ADHD-symptom and Normal-behavior (SWAN) rating scale.

Method

Parents of preschoolers completed both a SWAN and the well-established Strengths and Difficulties Questionnaire (SDQ) on two separate occasions over a span of 3 months; instruments were in the primary language of the family (English or Spanish).

Results

Psychometric properties for the English and Spanish versions of the SWAN were adequate, with high internal consistency and moderate test–retest reliability. Skewness and kurtosis statistics for the SWAN were within the range expected for a normally distributed population. The SWAN also demonstrated adequate convergent and discriminant validity in correlations with the various subscales of the SDQ.

Conclusion

Psychometric properties of both the English and Spanish versions of the SWAN indicate that it is a reliable and valid instrument for measuring child attention and hyperactivity. The stability of ratings over time in this preschool sample was moderate, which may reflect the relative instability of these characteristics in preschool children.

There is a growing recognition that the symptoms of ADHD should be measured using dimensional approaches. For example, the summary statement of the National Institute for Mental Health Interdisciplinary Research on ADHD 2000 workshop included the following:

Researchers commented that inattention and hyperactivity/impulsivity are distributed in the population along a continuum with no clear threshold to define impairment. Additional participants argued that ADHD should not be conceptualized as a disorder that is qualitatively distinct from normal behavior. Rather, ADHD behaviors lie on a continuum, with many children at the high end of the continuum experiencing serious impairment. For instance, a participating geneticist noted that genetic links to ADHD do not represent mutations, but polymorphisms that create subtle differences in human behavior and are likely interacting with the environment to create symptoms and functional impairment. Consequently, an understanding of factors related to ADHD would be enhanced by additional focused research within both clinical and nonclinical populations.

Scores of many rating scales of ADHD and related behaviors are not normally distributed in the population. This is due to the definition of the items and rating categories of the rating scales. To address this, the Strengths and Weaknesses of ADHD-symptoms and Normal-behaviors (SWAN) rating scale was constructed to be dimensional at the item level. It differs from the typical parent rating scales of psychopathology, such as the Achenbach (1991), Conners, Parker, Sitarenios, and Epstein (1998), or Swanson, Nolan, and Pelham (SNAP; www.adhd.net) rating scales, which define responses to items with respect to presence of psychopathology.

For example, the items of the SNAP rating scales are the Diagnostic and Statistical Manual of Mental Disorders (4th ed.; DSM IV; American Psychiatric Association, 1994) symptoms of ADHD, which are defined as psychopathology due to greater severity or intensity than for other children of the same age. The presence of severity and intensity produces impairment. Raters make subjective judgments on each item regarding the severity of psychopathology manifested by a child (e.g., 0 = not present, 1 = just a little present, 2 = quite a bit present, and 3 = very much present). In contrast, the items of the SWAN are restated versions of each of the DSM IV symptoms designed to reflect the entire range of behavior in the population. The child is rated on each item compared with the rater's subjective assessment of what is average in the population (3 = far below average, 2 = below average, 1 = somewhat below average, 0 = average, –1 = somewhat above average, –2 = above average, and –3 = far above average). To be consistent with the SNAP rating scale, a scoring convention is used in which the positive ratings reflect presence of behavior that would define psychopathology in the extreme, and negative ratings reflect presence of behavior that would reflect absence of psychopathology. If the negative ratings (reflecting strengths) are all set to zero, then the modified SWAN is intended to be similar to the SNAP rating scale. Thus, the items of the SWAN and SNAP rating scale are based on the same content, but on the SWAN, each item is rated with respect to a dimension (above or below the population average), whereas on the SNAP each item is rated with respect to a category (i.e., as severity of psychopathology defined as rare in the population).

In this study, we compared the psychometric properties of this new dimensional rating scale of the underlying dimension of ADHD (the SWAN) with the psychometric properties of a subscale of an established standard that assesses the symptom domains of ADHD (the Hyperactivity/Inattention subscale of the Strengths and Difficulties Questionnaire [SDQ]). Although the SWAN has been used in previous published research studies (e.g., Hay, Bennett, Levy, Sergeant, & Swanson, 2007; Polderman et al., 2007; Ramtekkar, Reiersen, Todorov, & Todd, 2010; Volkow et al., in press; Young, Levy, Martin, & Hay, 2009) and has been translated into several languages, including French and Hungarian (Lakatos, Birkas, Toth, & Gervai, 2010; Robaey, Amre, Schachar, & Simard, 2007), to our knowledge, there is no published study addressing the reliability and validity of the original English version of the SWAN or the Spanish translation of the SWAN.

Method

Participants

We analyzed data from 90 preschool children (mean age of 3 years 4 months) referred to the CHOC/UCI Initiative for the Development and Readiness (CUIDAR) program, which provides community parent education (COPE; Cunningham, 1995) for parents of preschool children (ages 3-5 years) who are self-referred because of concern about their child's attention and behavior problems (see Lakes et al., 2009; Lakes, Vargas, Riggs, Schmidt, & Baird, 2010; Tamm et al., 2005). The CUIDAR programs serve children in Orange and San Bernardino counties in California. A total of 54% of the children were female. Ethnicity and race were diverse: 71% were Hispanic; 10% were White, non–Hispanic; 8% were African American; 8% were biracial; and 3% reported other racial/ethnic backgrounds. Parents were asked to indicate their primary language and instruments were provided in that language (52% Spanish and 48% English).

Procedures

This study was approved by a university human-subjects research review board, and we obtained written consent from all participants (parents). Questionnaires, English or Spanish, were mailed to parents, and parents either returned the questionnaires via mail or called the program office to complete them over the telephone. We obtained parent ratings on the SWAN and SDQ at two time points: after completing the 10-week COPE program (for evaluation of the psychometrics properties of the SWAN and the SDQ) and 3 months later (for evaluation of the test–retest reliability of the two scales).

Measures

SDQ

In this study, we used the parent-rated, English (United States) and Spanish versions of the SDQ (Goodman & Scott, 1999; www.sdqinfo.org). The SDQ has been widely used and has been normed for the United States (Bourden et al., 2005). The SDQ consists of five subscales: Emotional Symptoms, Conduct Problems, Hyperactivity/Inattention, Peer Problems, and Prosocial Behavior. Each subscale has five items and each item is stated as a strength or weakness but not both. The number stated as weaknesses varies across these subscales. For Emotional Symptoms, all five are stated as weaknesses; for Conduct Problems, four of the five; for Hyperactivity/Inattention, three of the five; for Peer Problems, two of the five; and for Prosocial, none of the five (i.e., all are stated as strengths). The score on each subscale of the SDQ is determined by the summation of items stated as strengths and weaknesses. Weaknesses are scored 0 for “not true,” 1 for “somewhat true,” and 2 for “certainly true.” Reverse scoring is used for strengths, which are scored 2 for “not true,” 1 for “somewhat true,” and 0 for “certainly true.”

The SDQ Hyperactivity-Inattention subscale provides a rating reflecting the symptom domains of ADHD (i.e., inattention, hyperactivity, and impulsivity). It consists of three items stated as difficulties (1 = restless, overactive, cannot stay still for long; 2 = constantly fidgeting or squirming; and 3 = easily distracted, concentration wanders) and two items as strengths (4 = thinks things out before acting and 5 = sees tasks through to the end, good attention span).

As directed by the SDQ manual, the items stated as difficulties were scored to reflect the degree of psychopathology (0 = not true, 1 = somewhat true, and 2 = certainly true). The items stated as strengths were reverse scored (2 = not true, 1 = somewhat true, and 0 = certainly true). Thus, the minimum on each subscale is 0 (representing lack of psychopathology) and the maximum score on each subscale is 10 (representing presence of psychopathology, except in the case of the Prosocial Behavior subscale).

SWAN

As directed by the SWAN manual (www.adhd.net), all items were scored on a 7-point scale reflecting degree of discrepancy from average behavior: 3 = far below average, 2 = below average, 1 = somewhat below average, 0 = average, –1 = somewhat above average, –2 = above average, and –3 = far above average.

Analyses

All analyses were completed using SPSS. Cronbach's alpha was used to evaluate internal consistency. Skewness and kurtosis statistics were calculated and evaluated using statistical significance and visual inspection of the distributions. Visual evaluation is recommended, especially with large sample sizes where minor deviations from normality can result in statistically significant skewness and kurtosis (Tabachnick & Fidell, 2006). Pearson's rs were used to assess test–retest reliability and evidence of convergent/divergent validity.

Results

Reliability

As shown in Table 1, Cronbach's alphas indicate strong internal consistency for the SWAN (.95) and moderate internal consistency (.70) for the SDQ Hyperactivity-Inattention subscale. Alphas were similar for English and Spanish versions of the SWAN (.95 and .96, respectively). For the SDQ, the alpha was stronger for the English version (.78 for English vs. .63 for Spanish).

Table 1.

Psychometric Properties of the SWAN and the SDQ Hyperactivity/Inattention Subscales in Both English and Spanish

Scale	Mean (SD)	Skewness (SE)	Kurtosis (SE)	Cronbach's α	Test/retest
SWAN Total Scorea	–0.54 (1.07)	–0.23 (0.25)	0.06 (0.50)	.95	.66
English	–0.31 (0.97)	–1.00 (0.38)	1.38 (0.75)	.95	.76
Spanish	–0.72 (1.11)	0.28 (0.33)	0.18 (0.66)	.96	.57
SWAN Attentiona	–0.60 (1.10)	–0.26 (0.25)	–0.35 (0.50)	.92	.61
English	–0.41 (1.02)	–0.95 (0.38)	0.79 (0.75)	.91	.72
Spanish	–0.77 (1.13)	0.19 (0.33)	–0.35 (0.66)	.92	.49
SWAN Hyperactivitya	–0.47 (1.18)	–0.16 (0.25)	0.15 (0.50)	.94	.66
English	–0.21 (1.04)	–0.86 (0.38)	1.09 (0.75)	.93	.71
Spanish	–0.68 (1.25)	0.27 (0.33)	0.30 (0.66)	.95	.61
SDQ Hyperactivity/Inattentiona	3.83 (2.34)	0.32 (0.25)	–0.50 (0.50)	.70	.61
English	3.97 (2.61)	0.18 (0.38)	–0.71 (0.75)	.78	.63
Spanish	3.69 (2.14)	0.48 (0.33)	–0.20 (0.66)	.63	.60

Note: SWAN = Strengths and Weaknesses of ADHD-symptom and Normal-behavior; SDQ = Strengths and Difficulties Questionnaire.

^aIncludes ratings from both English and Spanish versions of the same scale.

Test–Retest Reliability

Test–retest reliabilities also are reported in Table 1. A test–retest coefficient magnitude of 0.70 is generally considered acceptable (Salkind, 2006). The SWAN Total Score and SWAN Hyperactivity subscale approximated this standard (rs = .66 and .66, respectively). The SWAN Attention subscale and the SDQ fell slightly below this standard (rs = .61 and .61, respectively). In the English version, all SWAN scales met the standard of .70 (rs = .76, .72, and .71 for the total score, Attention subscale, and Hyperactivity subscale, respectively). Test–retest reliabilities trended lower for the SWAN Spanish versions (rs = .57, .49, and .61 for the total score, Attention subscale, and Hyperactivity subscale, respectively). For the SDQ, reliabilities were approximately the same for both English and Spanish versions (rs = .63 and .60, respectively).

Skewness

The values for mean, standard deviation, and skewness are shown in Table 1 for the parent ratings of the 18 ADHD items from the SWAN and the 5 Hyperactivity-Inattention items from the SDQ. In this at-risk sample, the value of the statistic for skewness is similar in size for the two scales, although opposite in direction (SWAN = –0.23 and SDQ = 0.32). The skewness statistics for both scales were tested to determine if they were significantly different from what would be expected in a normal population (S = 0); neither was significantly different (zs = –0.92 and 1.28, respectively). Visual inspections of the histograms with normal distributions superimposed also revealed no appreciable visual skew for either distribution.

Kurtosis

Kurtosis refers to the degree to which scores cluster around a central point. Kurtosis is zero in a normally distributed population; a positive statistic indicates that the scores cluster more than expected in a normally distributed population and a negative statistic indicates that the scores cluster less than expected in a normally distributed population. As shown in Table 1, the SWAN scores clustered slightly more than what would be expected in a normal population, and the SDQ Hyperactivity/Inattention scores clustered slightly less than in a normal population. The kurtosis statistics for both scales were tested to determine if they were significantly different from kurtosis in a normal distribution (K = 0); neither was significantly different (zs = 0.12 and –1.00, respectively). Again, visual inspection of the distributions against a superimposed normal did not indicate meaningful deviations from normality.

Convergent Validity

Table 2 summarizes results from validity analyses. We predicted that there would be a significant correlation between the SWAN Total Score and the SDQ Hyperactivity/Inattention subscale score, as both scales aim to measure attention and hyperactivity in children. The correlation between the SWAN and the SDQ Hyperactivity/Inattention subscale was large and significant (r = .54; p < .01). Although no absolute standard for expected magnitude can be set for a convergent validity coefficient, a minimal expectation would be that they are large (at least 0.50) and statistically significant (Anastasi & Urbina, 1997). We further predicted that the SWAN would have small to moderate correlations with Conduct Problems, Peer Problems, and Prosocial Behavior due to the high comorbidity of conduct and peer problems and ADHD. Results supported these predictions (see Table 2).

Table 2.

Convergent and Discriminant Validity: Correlations Between the SWAN and SDQ Scales

SDQ scale	SWAN Total Score	SWAN Hyperactivity	SWAN Attention
Emotional Symptoms	.11	.10	.11
Conduct Problems	.40**	.34**	.41**
Hyperactivity/Inattention	.54**	.49**	.52**
Peer Problems	.23*	.16	.28**
Prosocial Behavior	–.40**	–.34**	–.42**

Note: SWAN = Strengths and Weaknesses of ADHD-symptom and Normal-behavior; SDQ = Strengths and Difficulties Questionnaire.

^*p < .05.

^**p < .01.

Discriminant Validity

We predicted that the SWAN would have little to no correlation with the SDQ Emotional Symptoms subscale, and the results supported our hypothesis (see Table 2).

Discussion

The psychometric properties of the SWAN and the SDQ Hyperactivity/Inattention subscale are similar in this referred sample of preschool children, although some of the reliability statistics (i.e., internal consistency coefficients) were superior for the SWAN. This increase in reliability is to be expected given that the SWAN has 18 items and the SDQ Hyperactivity/Inattention subscale has only 5 items; it has been demonstrated elsewhere (e.g., Lakes & Hoyt, 2009) that increasing the number of items in a scale increases reliability coefficients.

Test–retest reliability coefficients were modest for both measures in this study, ranging between 0.61 and 0.71 overall, with lower test–retest reliability coefficients observed for the Spanish versions of both the SWAN and the SDQ (ranging from 0.49 to 0.61). The lower test–retest reliability coefficients are consistent with prior research (e.g., Loughran, 2003; Smith & Corkum, 2007) that has demonstrated that the stability of ratings of ADHD symptoms in preschool children is weaker than the stability of ratings in older children. Loughran (2003) hypothesized that this may be due to the lack of norms for preschool children as well as wide variations in behavior due to development. We also observed lower coefficients for the Spanish versions of both scales, which could be due to different cultural interpretations of scale items or cultural norms for behavior and attention; it is possible that direct translations of scale items do not convey the same meaning in other languages or cultural groups, scale formats may not be familiar, and that, as a result, uncertainty about responding could produce weaker reliability coefficients. This issue should be studied further with a larger sample in a study design that includes efforts (such as interviews with parents) to study respondent understanding of the scale format and purpose as well as interpretation of scale items. Thus, our recommendation that the SWAN be used as one tool in a multimodal assessment is particularly relevant for assessments of preschool children and children from families in which English is not the primary language, as a single rating scale may be limited for the reasons discussed above. For a review of additional measures used in the diagnosis of ADHD in preschool children that could be paired with the SWAN in a multimodal assessment, we refer readers to Smith and Corkum (2007).

The SWAN was significantly correlated with the SDQ Hyperactivity/Inattention subscale, and the large correlation we observed (r = .54) met the recommended standard for convergent validity (Anastasi & Urbina, 1997). As would be expected, significant correlations were found with the SDQ scales measuring behaviors known to have high comorbidity with ADHD (conduct and peer problems). The moderate negative correlation with the SDQ Prosocial Behavior subscale was also to be expected and indicates that children rated as more inattentive and hyperactive display fewer prosocial (e.g., helping) behaviors. The significant correlations we observed with the SDQ Conduct, Peer Problems, and Prosocial Behavior subscales were moderate to small; this is to be expected when rating scales measure different, but related, constructs and when they use different rating approaches (Cohen & Cohen, 1983; Lakes, Swanson, Riggs, Schuck, & Stehli, under review). Cohen and Cohen (1983) explained that correlations are largest between measures of variables that use similar measurement approaches. Finally, as we predicted, there was no significant correlation with the Emotional Symptoms subscale on the SDQ.

Limitations

Our study design was limited due to its inclusion of only two rating scale measures of ADHD; we did not include behavioral observations or lengthy ADHD assessments because the study was conducted in a volunteer community setting. We chose the SDQ for this study because it is a well-researched measure with an established subscale for attention and hyperactivity symptoms and because it contained other subscales that could be used for validity analyses. To avoid overburdening our community participants, we limited our measures to these two, but we acknowledge that the addition of other forms of measurement should be included in future research.

The SWAN shares an important limitation with all parent or teacher reports: the problem of rater bias. In the clinical or research application of parent or teacher rating scales, the impact of rater bias on scores should be acknowledged. When interpreting results, the clinician or researcher will not know how raters (i.e., parents or teachers) differed in their rating leniency or severity (in other words, we will not know the parent's or teacher's reference point for what is considered “average in the population” or “normal”); this is always a limitation when raters are nested within participants, as is the case when each child is rated by his parent or teacher and different parents and teachers rate different participants. This methodological issue was discussed by Lakes and Hoyt (2009), who demonstrated the advantages of using observer ratings in a fully crossed research design where all observers rate all children. This design may not always be feasible, particularly in clinical settings, and we expect that parent and teacher ratings will continue to be widely used in these settings. Therefore, we advise those who use the SWAN or other parent or teacher rating scales to consider the potential impact of rater bias on scores and to incorporate multiple methods of assessment in their evaluations of children. In future research, we encourage the use of the SWAN as one tool in a multimethod approach to rating ADHD symptoms.

Conclusion

The internal consistency and validity results provide evidence of strong reliability and validity for the English and Spanish versions of the SWAN. In this preschool sample, test–retest reliability coefficients were moderate for both the SWAN and SDQ, which suggests that given the lack of stability of symptoms in preschool children, assessments with preschool children should include ratings aggregated across multiple occasions and settings to improve reliability. As a continuum measure of hyperactivity and attention in children, the SWAN holds promise as a reliable and valid measure. The SWAN may be most advantageous in studies with nonclinical samples; for example, the SWAN has the potential to enhance epidemiologic or genetic studies of ADHD, where the full range of behavior and its impact on outcomes is of interest.

Biography

Kimberley D. Lakes, PhD, is an assistant professor in the Department of Pediatrics at the University of California (UC), Irvine. She was the principal investigator for this study. She is the codirector for Community Engagement of the Institute for Clinical and Translational Science in the UC Irvine School of Medicine and also is an investigator for the Orange County Vanguard Center for the National Children's Study.

James M. Swanson, PhD, is a retired professor of pediatrics at the University of California, Irvine. He is a senior fellow of the Sackler Institute at Cornell and lead investigator for the Orange County Vanguard Center and Southern and Central California Study Center for the National Children's Study. He consulted on the use of the SWAN in this study and contributed to the final manuscript.

Matt Riggs, PhD, is a statistician and professor in the Department of Psychology at California State University, San Bernardino, and he provided database management and statistical support for this study.