Brief measures of anxiety in non-treatment-seeking youth with autism spectrum disorder

Abstract

This study investigated the accuracy of brief anxiety scales for non-treatment-seeking youth with autism spectrum disorder. In all, 54 youth (7–17 years; IQ: 67–158) with autism spectrum disorder and their parents completed (a) an expanded version of the Anxiety Disorders Interview Schedule—Child/Parent designed to capture typical and atypical fears and (b) brief scales of anxiety symptoms (Behavior Assessment Schedule for Children, Second Edition; Screen for Child Anxiety and Related Emotional Disorders; Negative Affective Self-Statement Questionnaire; Pediatric Anxiety Rating Scale). The results indicate that measures lacked adequate sensitivity and specificity, and the detection of atypical fears was particularly poor. Revised cut scores are offered, but refined and/or revised instruments are likely needed for research on youth with autism spectrum disorder.

Many youth with autism spectrum disorders (ASDs) experience and struggle with co-occurring anxiety problems that further impair their functioning (De Bruin et al., 2007; Farrugia and Hudson, 2006; Kerns et al., 2014b; Van Steensel et al., 2011). Though there are studies on the prevalence and treatment of anxiety in ASD, it remains difficult to accurately identify anxiety symptoms and disorders in this population (Matson and Nebel-Schwalm, 2007). There is a lack of consensus regarding the best way to measure and differentiate anxiety and ASD and many inconsistencies regarding the distribution and characteristics of specific anxiety disorders in ASD (see Kerns and Kendall, 2012; White et al., 2009). Though inconsistencies may, in part, reflect differences in recruitment and sampling methods, they may also reflect the limitations of existing anxiety measures.

Differentially diagnosing anxiety and ASD are inherently difficult given the similarity of some anxiety and ASD symptoms and the potential for youth with ASD to express anxiety in an atypical manner. For example, social avoidance and isolation, poor communication, and repetitive and restricted behaviors are common and often difficult to disentangle from traditional symptoms of anxiety (Cath et al., 2008; Hartley and Sikora, 2009). Moreover, there is speculation as to whether anxiety presents in a similar or distinct manner across children with and without ASD (see Ollendick and White, 2012; Scahill, 2012; White et al., 2014; Wood and Gadow, 2010). In a sample of 1033 children with autism; Mayes et al. (2013) reported unusual fears, including fears of vacuum cleaners, elevators, mechanical toys, swings, and the wind in 41% of youth. Kerns et al. (2014a) found a prevalence of both Diagnostic Statistical Manual of Mental Disorders— Fourth Edition Revised (DSM-IV) consistent (48%) and inconsistent anxiety symptoms (46%), herein referred to as atypical anxiety, in youth with ASD using a semi-structured interview. Atypical anxiety symptoms included unusual specific phobia (e.g. fear of fans, radio jingles, bubbles); excessive worry about novelty and change; distressed ritualistic/compulsive behavior that could not be clearly connected to intrusive, unwanted thoughts, or feelings; and social fear despite limited awareness of social ridicule. Many other studies have described, but not formally assessed such atypical fears and worries in ASD (Hallett et al., 2013; Leyfer et al., 2006; Muris et al., 1998; Ozsivadjian et al., 2012; Trembath et al., 2012). Atypical symptoms and the overlap of many anxiety and ASD features raise questions about the performance of measures designed for typically developing youth in this population (see Scahill, 2012).

Studies of anxiety in youth with ASD have often relied on existing anxiety measures, particularly parent-and youth-report questionnaires. Questionnaires are appealing due to their brevity, yet this appeal is contingent on their reliability and agreement with a gold-standard criterion (i.e. diagnostic interview). Such agreement is largely unestablished for anxiety measures used for youth with ASD. To our knowledge, only two studies have examined the sensitivity and specificity of anxiety questionnaires in this population. Mazefsky et al. (2011) found the sensitivity and specificity of the Revised Children’s Manifest Anxiety Scale (Reynolds and Richmond, 1978) and Short Leyton Obsessional Inventory—Child Version (Bamber et al., 2002) to be lower in a community-based ASD sample than reported in the general literature. In anxiety disordered youth with and without ASD, Van Steensel et al. (2013) found relatively comparable levels of sensitivity and specificity for child-and parent-report versions of the Dutch SCARED-71 (Bodden et al., 2009). However, discriminant validity was poorer for youth with versus without ASD unless standard cut-off scores were raised.

Other studies have explored the factor structure, reliability, convergent, and divergent validity of additional anxiety scales in ASD. Storch et al. (2012b) found that the Pediatric Anxiety Rating Scale (PARS) (RUPP Anxiety Study Group, 2002) demonstrated acceptable convergent validity, inter-rater reliability (intraclass correlation coefficient (ICC) = 0.86), and retest reliability (ICC = 0.83) when administered after a comprehensive anxiety interview in a treatment-seeking sample of youth with ASD. By comparison, internal consistency (0.59) as well as parent-child agreement were limited (κ = 0.08–0.23). Similarly, White et al. (2012) reported poor parent and child agreement on the Child and Adolescent Symptom Inventory (CASI) (Sprafkin et al., 2002) anxiety subscale and Multidimensional Anxiety Scale for Children (MASC) (March, 1998) in a treatment-seeking sample of youth with ASD and impairing anxiety. In a second study, White et al. (2014) found evidence for a distinct factor structure for the MASC parent-report, but not child-report scales in youth with versus without ASD. In keeping with these results, Hallett et al. (2013) also found evidence of both DSM-IV consistent and more ambiguous anxiety factors when examining the CASI in youth with ASD in the United Kingdom. These findings support the notion that anxiety symptoms may manifest differently in ASD samples and require a modified assessment approach. They do not, however, speak to the accuracy with which the PARS, MASC, and CASI detect anxiety disorders in non-treatment-seeking samples of youth (i.e. as universal or low-risk screening measures). Even less is known about the psychometric properties of other anxiety measures employed to assess anxiety in ASD, including the Screen for Child Anxiety and Related Disorders (SCARED) (Birmaher et al., 1999), the anxiety subscales of the Behavior Assessment Schedule for Children (BASC) (Reynolds and Kamphaus, 2004), and the Negative Affective Self-Statements Questionnaire (NASSQ) ( Ronan et al., 1994; see Grondhuis and Aman, 2012 and Lecavalier et al., 2014 for a comprehensive review).

A few studies have adapted existing scales to assess anxiety in youth with ASD. Sukhodolsky et al. (2008) eliminated six items that might overlap with ASD symptoms (e.g. difficulty paying attention) from the CASI-4 anxiety scale to create an adapted, 20-item index with moderate internal consistency (Cronbach’s alpha = 0.85). Additionally, Kuusikko et al. (2008) created revised versions of the Child Behavior Checklist (CBCL), Social Anxiety Scale for Children-Revised, and Social Phobia and Anxiety Inventory for Children by excluding items that might overlap with the symptoms of ASD. These studies did not assess agreement between these adapted measures and a diagnostic interview; nonetheless, they are a step toward understanding the versatility of existing anxiety scales. Revising standard cut-off scores may be an additional avenue for adapting such tools for youth with ASD (Mazefsky et al., 2011; Van Steensel et al., 2011).

This study investigated the diagnostic accuracy of brief measures of anxiety as compared to a semi-structured diagnostic interview (i.e. the Anxiety Disorders Interview Schedule—Child and Parent (ADIS-C/P)), collected as part of an investigation of anxiety in a non-treatment-seeking sample of youth with ASD (Kerns et al., 2014a). Measures varied in type, including anxiety-specific instruments (SCARED, PARS) and anxiety subscales in broad measures of psychopathology (BASC, NASSQ). All but the PARS are measures with limited psychometric evaluation in ASD populations. The PARS was included because its ability to detect the presence of anxiety disorders in a low-risk population (e.g. not selected for anxiety problems) of youth with ASD is unstudied. In contrast to prior research, the ADIS-C/P was augmented with questions to aid in differential diagnosis of ASD and anxiety and to capture atypical fears and worries, which arise in ASD, but do not fit the DSM-IV categories. Although there is some evidence to support the conceptualization of DSM-IV-consistent anxiety symptoms as evidence of comorbidity in ASD, the correct conceptualization of atypical anxiety symptoms is unresolved (Kerns and Kendall, 2012). Accordingly, in this study, brief measures were compared first to DSM-IV-consistent anxiety diagnoses and then, as an exploratory aim, to atypical anxiety symptoms from the ADIS-C/P. As an additional exploratory aim, we examined whether the accuracy and positive predictive value (PPV) of the brief measures might be improved by altering established cut scores.

Method

Participants

The present data were gathered as part of a study of anxiety disorders in youth with ASD (Kerns et al., 2014a); however, the hypotheses and results evaluated in this study do not overlap with other reports. Fifty-nine children were initially recruited, but children presenting with solely specific phobias (n = 5) were excluded from the sample given that the anxiety measures targeted in this study were not designed to detect this diagnosis. Participants were 54 youth (M age = 10.54 years, standard deviation (SD) = 2.68 years) diagnosed with ASD and their parents. They were recruited by phone call or email to families affiliated with Kerns at al., 2014a or through an affiliated online research registry—AutismMATCH. All participants completed informed consent as approved by the Children’s Hospital of Philadelphia Institutional Review Board. To be included in the study, participants were required to be between the ages of 7 and 17 years with a diagnosis of ASD (as confirmed by standard assessments) and an IQ above 60. Exclusionary criteria included the following: (a) active psychosis or sensory-motor difficulties that would preclude valid use of diagnostic instruments; (b) a lack of proficiency in English in youth and primary caregivers; and (c) a history of traumatic brain injury or other significant medical or neurological abnormality. In all, 44% of youth were taking psychotropic medications, including antidepressants (17%), stimulants (15%), sympatholytics (4%), antipsy-chotics (5%), and buspirone (4%). Demographic data are provided in Table 1.

Table 1.

Demographic characteristics.

	Mean (SD)	Range
Age (in years)	10.54 (2.68)	7–17
IQa	105.17 (19.64)	67–158
	N (% of total)
Gender
Male	42 (78)
Female	l2 (22)
Race
Caucasian	5l (94)
African-American	2 (4)
Bi-racial	1 (2)
Family incomeb
<60,000	7 (14)
60,000–100,000	l4 (26)
>100,000	28 (52)
DSM-IV ASD diagnosis
Autistic disorder	38 (70.5)
Asperger’s disorder	5 (9)
PDD-NOS	11 (20.5)
Diagnostic profile of sample
No DSM-IV anxiety or atypical distress	22 (41)
DSM-IV anxiety disorders	20 (37)
DSM-IV anxiety disorders alone	6 (11)
DSM-IV anxiety disorder and impairing atypical distress	l4 (26)
Impairing atypical distress alone	l2 (22)
Depressive disorder	3 (6)
ADHD symptoms	2l (39)
ODD	4 (7)

SD: standard deviation; DSM-IV: Diagnostic and Statistical Manual of Mental Disorders—Fourth Edition Revised; ASD: autism spectrum disorder; PDD-NOS: pervasive developmental disorder, not otherwise specified; ADHD: attention deficit-hyperactivity disorder; ODD: oppositional defiant disorder.

^aCalculated from the Differential Abilities Scale—II (DAS-II).

^bMissing data on family income for five participants.

Characterization measures

The Differential Ability Scales, Second Edition (DAS-II) (Elliot, 2007), General Conceptual Ability (GCA) index was used as a full IQ estimate given its strong correlation with other IQ tests (Dumont et al., 1996). The Autism Diagnostic Interview-Revised (ADI-R) (Lord et al., 1994) and Autism Diagnostic Observational Schedule-Generic (ADOS-G) (Lord et al., 2000), gold-standard diagnostic instruments for ASD, were used with expert clinical judgment to confirm ASD diagnosis in all youth.

Anxiety measures

ADIS-C/P. The ADIS-C/P is a semi-structured interview for diagnosing DSM-IV anxiety disorders in youth ages 7–17 years (Silverman and Albano, 1996). It has favorable psychometric properties (Silverman et al., 2001), including convergent validity (Wood et al., 2002). The ADIS-C/P has demonstrated inter-rater reliability in youth with ASD (Ung et al., 2014). However, one study found child and parent agreement on the ADIS-C/P to be poor, underscoring the importance of using parent or composite diagnoses rather than child-report alone (Storch et al., 2012a).

For this study, the ADIS-C/P was augmented for youth with ASD (as in Kerns et al., 2014a). To maintain the integrity of the interview, questions and core administrative features of the ADIS-C/P were not altered; rather, follow-up questions to examine the differential diagnosis of ASD and anxiety were added, in keeping with the ADIS-C/P as a semi-structured interview. Diagnosticians assigned composite clinical severity ratings (CSRs; 0–8) for DSM-IV anxiety disorders (as part of the core interview) and for ambiguous or atypical symptoms of anxiety reported by children and parents (augmentation to core interview). That is, rather than excluding these unusual symptoms or subsuming them under ill-fitting DSM-IV diagnoses, atypical worries and fears were differentiated from DSM-IV symptoms and quantified. A detailed description of this expansion of the ADIS, including inter-rater and retest reliability and convergent and divergent validity, is reported elsewhere (see Kerns et al., 2014a). Per ADIS manual specifications, composite CSR ratings were derived from the highest CSR given per either parent or child (Silverman and Albano, 1996).

PARS.

The PARS is a clinician-rated scale for youth ages 7–17 years that assesses somatic, separation, specific, social, and generalized symptoms of anxiety disorders based on a joint parent and child interview (RUPP Anxiety Study Group, 2002). The PARS has demonstrated inter-rater reliability, test-retest reliability, and convergent validity in youth with ASD (Storch et al., 2012b). It demonstrated good internal consistency in this study (alpha=0.90). This study used the 5-item PARS Total Score, which has an optimal cut-off point of 11.5 to discriminate between anxious and non-anxious children (as determined in a non-ASD sample) (Ginsburg et al., 2011).

BASC-2.

The BASC-2 assesses adaptive and problem behaviors in youth, including anxiety (Reynolds and Kamphaus, 2004). It has good concurrent validity with other child behavior rating scales (e.g. CBCL) and has been studied in ASD (Volker et al., 2010). The BASC-2 Anxiety clinical scale was used in this study, with standardized T-scores ⩾70 indicating clinically significant anxiety. Across the different age-nor-med BASC-2 versions, internal consistency was acceptable, with alphas ranging from 0.80 to 0.92.

NASSQ.

The NASSQ is a 70-item self-report measure assessing negative and anxious self-statements (Ronan et al., 1994). Its 33-item anxiety subscale NASSQ-A has been found to differentiate anxious from non-anxious youth without ASD, with a cut-off score of 49 (Sood and Kendall, 2007) in youth 7–16 years of age (Lerner et al., 1999; Ronan et al., 1994). It has acceptable sensitivity (0.81), specificity (0.80), and 2-week retest stability (r=0.73– 0.96) as well as acceptability internal consistency in the current sample (0.80) (Sood and Kendall, 2007).

SCARED.

The SCARED is a 41-item child-and parent-report measure of DSM-IV symptoms of childhood separation, social, and generalized anxiety disorders for youth ages 4–18 years (Birmaher et al., 1997, 1999). It has demonstrated psychometric properties, including sensitivity and specificity for anxiety disorders with a total score ⩾25 (Birmaher et al., 1999) and internal consistency in this sample (0.94 and 0.91 for child- and parent-report, respectively).

Procedures

All interviews and assessments were completed by clinical psychologists and a clinical psychology doctoral student, all trained and reliable on the ADI-R (over 90% agreement), ADIS-C/P (over 80% agreement), and ADOS-G (over 80% agreement) according to standard procedures. Given the social, cognitive, and communication difficulties experienced by many children with ASD, the ADIS-C/P was administered to parents and youth concurrently in 38 of the 54 cases (70%) to encourage participant engagement and understanding. This approach is standard when administering the ADIS-C/P with ASD samples (White et al., 2013). The PARS was administered directly following the ADIS-C/P, whereas child and parent questionnaires were completed either before or after the diagnostic interview in an unsystematic fashion. Families were compensated for their time (US$10 per hour of assessment) and travel expenses, in addition to receiving a summary of results, recommendations, and appropriate referrals.

Reliability was established for the ADIS-C/P and PARS prior to and throughout the study via the use of videotapes and multiple-coder observation of live assessments. Specifically, in a random selection of 35% of cases, a second clinician observed the full ADIS-C/P administration either in person or via tape and then generated their scores independently from the primary clinician. The reliability of CSRs given for each DSM-IV anxiety disorder and atypical anxiety item was then calculated. Inter-rater reliability for traditional (ICC range 0.77–0.95) and atypical anxiety CSRs (ICC range 0.88–1.00) was acceptable. Percent exact agreement regarding traditional anxiety disorders and impairing atypical anxiety was consistent (95%−100%) (see Kerns et al., 2014a for more psychometric details).

Analytic plan

Primary analyses determined diagnostic agreement between anxiety measure cut-off scores and DSM-IV anxiety disorders as per the ADIS-C/P. “Caseness” was defined as youth with at least one DSM-IV diagnosis of generalized, social, or separation anxiety disorder per the core ADIS-C/P interview (CSR ⩾ 4; n = 20). Agreement for the primary analyses represented the agreement between the ADIS-C/P and the brief measures about the presence of only DSM-IV anxiety symptoms in a child (additional atypical anxiety symptoms on the ADIS-C/P were neither inclusionary nor exclusionary). Non-cases (n = 22) were individuals with ASD who demonstrated neither DSM-IV nor atypical anxiety per the ADIS-C/P. Youth with CSR ⩾4 for only atypical anxiety symptoms (i.e. with no DSM anxiety diagnosis; n = 12) were excluded from the primary analyses given that atypical symptoms may reflect altered manifestations of anxiety in ASD or symptoms of ASD, which the questionnaires were not designed to detect.

Secondary, exploratory analyses examined the agreement between anxiety measure cut-off scores and impairing atypical manifestations of anxiety (i.e. CSR ⩾4). For these analyses, a “case” referred to a child with impairing atypical anxiety only (i.e. no DSM anxiety disorder with the exception of specific phobia, n = 12); a “non-case” referred to a child with neither a DSM anxiety disorder nor atypical distress (n = 22).

For both primary and exploratory analyses, overall agreement between each measure and the reference standard (i.e. ADIS-C/P) was calculated, which represents the proportion of correct classifications (i.e. true positives + true negatives) to the total number of individuals. The receiver operator characteristic (ROC) method (Pintea and Moldovan, 2009) was then used to determine the optimal cut-off score of each measure to discriminate between ASD children with and without clinically significant anxiety, maximizing both sensitivity and specificity. The optimal cut-off score provides the most accurate classification of children as “anxious” or “non-anxious,” with both high sensitivity (i.e. true positive rate) and high specificity (i.e. true negative rate). The ROC method includes calculating and plotting the rates of true positives and false positives (i.e. 1—specificity) for different scores, which creates a curve. The area under the curve (AUC) represents the discriminatory accuracy of the measure; a perfect test produces an area of 1, whereas a test with no predictive power produces an area of 0.5. According to Streiner and Cairney (2007), an AUC <0.7 can be interpreted as low accuracy, between 0.7 and 0.9 as moderate accuracy and >0.9 as high accuracy. Based on maximum sensitivity and specificity, improved cut-off scores for each measure in an ASD sample were determined. Additionally, visual examination of positive and negative predictive value predictions, as well as sensitivity and specificity scores for a range of cut scores, informed the selection of cut scores that yielded optimal PPVs in this sample. PPV is distinct from sensitivity in that it describes the likelihood that a child identified as anxiety-disordered by a parent- or youth-report measure will meet full diagnostic criteria for an anxiety disorder on the ADIS-C/P. By comparison, sensitivity describes what proportion of anxiety-disordered youth in the sample are detected by the measure. Whereas sensitivity is essential for those wishing to screen for anxiety disorders in ASD youth where anxiety is not already suspected, high PPV may be useful for researchers interested in using a measure to increase the likelihood that children identified as anxiety-disordered actually present with clinical anxiety problems.

Results

Preliminary analyses

In all, 20 children (37% of the sample) presented with at least one DSM-IV anxiety disorder, including generalized anxiety disorder (24%), social phobia (19%), and separation anxiety disorder (11%). A total of 14 of the 20 children with DSM-IV anxiety disorders also had impairing atypical anxiety (i.e. 26% of sample overall or 70% of youth with DSM-IV anxiety disorders). Twelve youth (22%) presented with impairing atypical anxiety alone. Additionally, impairing attention deficit-hyperactivity disorder (ADHD) (39%) and depressive symptoms (6%) were apparent (see full descriptive of sample characteristics in Table 1). In all, 65% of youth with at least one DSM-IV anxiety disorder had another DSM-IV anxiety disorder diagnosis. Means and SDs for the anxiety questionnaires are as follows: ADIS-C/P DSM-IV Anxiety CSR (M = 3.03, SD = 2.21), ADIS-C/P Atypical Anxiety CSR (M = 2.75, SD = 2.27), BASC-2 Child (M = 48.61, SD = 11.56), BASC-2 Parent (M = 56.38, SD = 14.52), NASSQ-A (M = 58.84, SD = 21.55), PARS (M = 7.93 SD = 5.33), SCARED—Child (M= 23.71, SD = 16.57), and SCARED—Parent (M = 23.71, SD = 11.41). The distributions of the anxiety questionnaire scores were all approximately normal. Pearson correlations between the brief anxiety measures are reported in Table 2. Whereas the parent versions of the SCARED and BASC were correlated at 0.73, all other correlations ranged from weak to moderate (0.22–0.68).

Table 2.

Correlations between brief anxiety measures.

Measures—rater	1	2	3	4	5
1. SCARED—Child	1	0.50**	0.33*	0.22	0.46**
2. BASC-2—Child	0.50**	1	0.52**	0.68**	0.51**
3. SCARED—Parent	0.33*	0.52**	1	0.73**	0.60**
4. BASC-2—Parent	0.22	0.68**	0.73**	1	0.46**
5. PARS	0.46*	0.51**	0.60**	0.46**	1

SCARED: Screen for Anxiety and Related Emotional Disorders; BASC- 2: Behavior Assessment System for Children, Second Edition; PARS: Pediatric Anxiety Rating Scale.

^*p < 0.05

^**p < 0.01.

Missing data

Data were missing, at random, for a portion of the anxiety questionnaires. Sub-samples for each measure were as follows: PARS (n = 43), NASSQ-A (n = 52), BASC—Parent (n = 50), BASC—Child (n=42), SCARED—Parent (n = 52), and SCARED—Child (n = 38). There were no significant differences with regard to intellectual ability, child age, ASD diagnosis, anxiety severity (e.g. CSR for principal anxiety difficulty on ADIS-C/P), family income, race, or sex for youth who did versus did not complete questionnaires with the following exceptions. Youth who completed the child BASC (M IQ = 107.79, SD = 22.24) had significantly higher general cognitive ability (i.e. DAS-II GCA) than those who did not (M IQ = 98.94, SD = 9.20), t(41.77) = 3.25, p<0.01. Additionally, youth who completed the child SCARED (M age = 11.05 years, SD = 2.55 years; M IQ = 107.79, SD = 22.24) were significantly older, t(48) = 2.25, p = 0.03, and had significantly higher GCA scores, t(47.52) = 2.46, p = 0.0, than youth who did not (M age = 9.34 years, SD = 2.67 years; M IQ = 98.94, SD = 9.20).

Agreement between measures of DSM-IV anxiety disorders

Diagnostic agreement between the anxiety questionnaires and the ADIS-C/P using standard cut scores was less than optimal (Table 3). All measures either had low agreement on both positive and negative predictions or had islets of apparently high agreement among very small subsamples (e.g. only two children endorsed anxiety symptoms on the BASC-C, and both of those children had anxiety, but this high agreement is offset by the finding that 13 youth with anxiety did not endorse anxiety on the BASC-C). BASC-P, BASC-C, and PARS had few or no false positives, but identified true positives in less than 55% of diagnosed anxiety cases.

Table 3.

Agreement between the ADIS-C/P and anxiety measures for (a) DSM-IV anxiety disorders^a and (b) impairing atypical distress^b.

	n	Positive % agreement	Negative % agreement (n)	Overall % agreement
		(True/total positives)	(True/total negatives)
(a) DSM-IV anxiety disorders
BASC-2—Child	33	13% (2/15)	100% (18/18)	6l
BASC-2—Parent	38	47% (8/17)	95% (20/21)	74
NASSQ-A	40	78% (14/18)	59% (13/22)	68
PARS	36	53% (8/15)	95% (20/21)	78
SCARED—Child	30	54% (7/13)	65% (11/17)	60
SCARED—Parent	40	58% (11/19)	86% (18/21)	73
(b) Impairing atypical distress
BASC-2—Child	27	0% (0/9)	100% (21/21)	67
BASC-2—Parent	33	8% (1/12)	95% (20/21)	64
NASSQ-A	34	58% (7/12)	59% (13/22)	59
PARS	28	0% (0/7)	95% (20/21)	7l
SCARED—Child	25	25% (2/8)	65% (11/17)	52
SCARED—Parent	33	42% (5/12)	86%(l8/2l)	70

ADIS: Anxiety Disorders Interview Schedule—Child/Parent; BASC-2: Behavioral Assessment Schedule for Children—Second Edition, Anxiety Subscale; NASSQ-A: Negative Affective Self-statements Questionnaire, Anxiety Subscale; PARS: Pediatric Anxiety Rating Scale; SCARED: Screen for Child Anxiety and Related Emotional Disorders.

^aUsing a DSM-IV anxiety disorder diagnosis on ADIS-C/P and established cut-off scores on anxiety measures.

^bUsing atypical symptoms of distress classification from ADIS-C/P and established cut-off scores on anxiety measures.

We explored alternate cut scores given the parameters of a ROC analysis could improve the accuracy of each measure. The AUCs for each measure using the published cut scores ranged from 0.65 to 0.90, with the youth-report SCARED demonstrating low accuracy and the remaining measures demonstrating moderate accuracy within this sample (see Table 4). To achieve the maximum sensitivity and specificity, the optimal cut-off scores for each measure (with sensitivity, specificity) were generally lower than the standard cut-offs for each instrument: 47 for BASC-2— Child (0.93, 0.78), 55 for BASC-2—Parent (0.94, 0.76), 44 for NASSQ-A (0.89, 0.50), 7 for PARS (0.93, 0.71), 13 for SCARED—Child (0.85, 0.35), and 14 for SCARED—Parent (0.95, 0.62) (see Table 5). We further explored which cut scores would result in a high rate of true positives. Optimal cut scores (with PPV, sensitivity) for enhanced PPV with the smallest sacrifices in sensitivity are presented in Table 6.

Table 4.

ROC AUC analyses using published cut-off scores.

Measure	n	AUC (95% CI)	SE	Sensitivitya (95% CI)	Specificitya (95% CI)
BASC-2—Child	33	0.90** (0.80–0.99)	0.05	0.13 (0.02–0.40)	1.00 (0.81–1.00)
BASC-2—Parent	38	0.89** (0.78–0.99)	0.06	0.47 (0.23–0.72)	0.95 (0.76–0.99)
NASSQ-A	40	0.77* (0.62–0.92)	0.08	0.78 (0.52–0.93)	0.59 (0.36–0.79)
PARS	36	0.88** (0.75–0.99)	0.07	0.53 (0.27–0.79)	0.95 (0.76–0.99)
SCARED—Child	30	0.65 (0.45–0.85)	0.10	0.54 (0.25–0.81)	0.65 (0.38–0.86)
SCARED—Parent	40	0.86** (0.75–0.98)	0.06	0.58 (0.34–0.80)	0.86 (0.63–0.97)

ROC: receiver operator characteristic; AUC: area under the curve compared to the area under the non-informative line (AUC = 0.50); SE: standard error; CI: confidence interval; BASC-2: Behavioral Assessment Schedule for Children—Second Edition, Anxiety Subscale; NASSQ-A: Negative Affective Self-statements Questionnaire, Anxiety Subscale; PARS: Pediatric Anxiety Rating Scale; SCARED: Screen for Child Anxiety and Related Emotional Disorders.

^aBased on the established cut-off score for each measure and a DSM-IV anxiety disorder diagnosis on Anxiety Disorders Interview Schedule—Child/ Parent.

^*p < 0.05

^**p < 0.001.

Table 5.

Original and optimal cut-off scores for accurately identifying DSM-IV anxiety disorders per the ROC AUC analyses.

Measure	Original cut-off	Optimal cut-offa	Revised sensitivity (95% Cl)	Revised specificity (95% Cl)
BASC-2—Child	70	47	0.93 (0.68–0.99)	0.78 (0.52–0.93)
BASC-2—Parent	70	55	0.94 (0.71–0.99)	0.76 (0.53–0.92)
NASSQ-A	49	44	0.89 (0.65–0.98)	0.50 (0.28–0.71)
PARS	11.5	7	0.93 (0.68–0.99)	0.71 (0.48–0.89)
SCARED—Child	25	13	0.85 (0.55–0.98)	0.35 (0.14–0.62)
SCARED—Parent	25	14	0.95 (0.65–0.98)	0.62 (0.38–0.82)

DSM-IV: Diagnostic Statistical Manual of Mental Disorders—Fourth Edition Revised; ROC: receiver operator characteristic; AUC: area under the curve; CI: confidence interval; BASC-2: Behavioral Assessment Schedule for Children—Second Edition, Anxiety Subscale; NASSQ-A: Negative Affective Self-statements Questionnaire, Anxiety Subscale; PARS: Pediatric Anxiety Rating Scale; SCARED: Screen for Child Anxiety and Related Emotional Disorders.

^aThe modified cut-off score that maximizes the sensitivity and specificity (i.e. revised sensitivity and specificity) for each measure.

Table 6.

Original and optimal cut-off scores to reduce number of false positives.

Measure	Original cut-Off	Optimal cut-offa	PPV	Revised sensitivity (95% Cl)	Revised specificity (95% Cl)
BASC-2—Child	70	60	1.00	0.47 (0.21–0.73)	1.00 (0.81–1.00)
BASC-2—Parent	70	69	0.9	0.52 (0.28–0.77)	0.95 (0.76–0.99)
NASSQ-A	49	75	0.88	0.39 (0.17–0.64)	0.95 (0.77–0.99)
PARS	11.5	11.5	0.79	0.73 (0.45–0.92)	0.86 (0.63–0.97)
SCARED—Child	25	33	0.75	0.46 (0.19–0.75)	0.88 (0.63–0.98)
SCARED—Parent	25	29	0.89	0.42 (0.20–0.66)	0.95 (0.76–0.99)

PPV: positive predictive value; CI: confidence interval; BASC-2: Behavioral Assessment Schedule for Children—Second Edition, Anxiety Subscale; NASSQ-A: Negative Affective Self-statements Questionnaire, Anxiety Subscale; PARS: Pediatric Anxiety Rating Scale; SCARED: Screen for Child Anxiety and Related Emotional Disorders.

^aBased on visual examination of changes in PPV and sensitivity across ranging cut scores. Cut scores that provided high PPV (0.75 or above) with smallest loss of sensitivity were deemed optimal.

Agreement between measures on the presence of atypical distress

We examined the agreement between the anxiety questionnaires and impairing (CSR ⩾ 4) atypical manifestations of distress gathered from the expanded ADIS items. As shown in Table 3, negative percent agreement matched that of the primary analyses. However, the positive percent agreement was lower than observed for DSM-IV anxiety in all six measures. Measures showed low sensitivity for identifying potential atypical manifestations of distress often seen in ASD.

Discussion

To our knowledge, this study is the first to examine the sensitivity and specificity of the SCARED, BASC anxiety subscale, NASSQ anxiety scale, and PARS as compared to the ADIS-C/P in an ASD sample not seeking treatment for anxiety. In contrast to prior studies, DSM-IV anxiety disorders were systematically differentiated from ambiguous and atypical anxiety symptoms (e.g. unusual fears, fears of novelty/change) that also arise in ASD. Findings suggest that the accuracy of standard youth-and parent-report measures for anxiety may be limited. Diagnostic agreement between anxiety measures and the ADIS-C/P was less than optimal when standard cut scores were used and not easily improved by adjusting cut-off scores. Furthermore, detection of atypical, but impairing, anxiety symptoms was poor.

When using established cut-off scores, all measures lacked the combination of sensitivity and specificity required to justify their use as effectual DSM-IV anxiety disorder screeners within non-treatment-seeking individuals with ASD. Accuracy as determined by the AUCs from the ROC analysis ranged from low to moderate (0.65–0.90), with moderate ratings often bolstered by correct measurement in youth without anxiety (i.e. many true negatives, but few true positives). With the exception of the NASSQ-A, parent and child-report measures missed over 42% of youth with principal anxiety disorders (sensitivity ranged from 0.13 to 0.58). Though the NASSQ-A correctly identified a larger portion of youth with anxiety disorders, it did so at the cost of specificity, miscategorizing close to 41% of unaffected youth as having an anxiety disorder.

The propensity for these measures to both miss real cases of anxiety disorders in youth with ASD and falsely pathologize those without anxiety disorders is consistent with prior studies of self-report measures in youth with ASD (Mazefsky et al., 2011; White et al., 2012). Though Van Steensel et al. (2013) found the sensitivity of the SCARED-71 to be comparable in children with and without ASD, this may, in part, be attributable to their use of an entirely anxiety-disordered rather than non-treatment-seeking sample. The results support suggestions that anxiety prevalence may be overestimated or underestimated due to reliance on measures designed for youth without ASD (see Kerns and Kendall, 2012).

The inaccuracy of these measures may not be easily resolved. Based on results of the ROC curve, identifying “optimal” cut scores for each measure required a substantial reduction in thresholds for all measures to improve accuracy moderately. Optimal cut scores correctly identified up to 95% of youth with DSM-IV anxiety disorders, but reduced specificity, with 22%−29% of children being misidentified as anxiety-disordered in the best cases (BASC-2—Child, BASC-2—Parent, PARS) and 65% misidentified in the worst (SCARED—Child). Revised cut scores improved the sensitivity of the BASC-2 (child and parent) and PARS, supporting their potential use as screeners, but accuracy still fell short of ideal for a descriptive research measure.

For this study, a second set of cut scores was selected, based on visual examination of prediction values, to substantially reduce the risk of false-positive results for each measure. Most youth scoring above these cut scores presented with anxiety disorders (PPV range: 0.75–1.00), yet these youth typically represented half of the true anxiety cases or less, suggesting that many true anxiety cases will be missed with this approach. These secondary cut scores may be a helpful reference when the goal is confidence about the presence of anxiety problems in an individual child.

The brief measures used in this study did not appear to capture the atypical fears and worries that arise in youth with ASD. Atypical fears and worries were reported along with DSM-IV anxiety disorder in 26% of youth in this study and were the only impairing anxiety-like behaviors apparent for 22% of the participants. The child BASC and the PARS detected none of the children with impairing atypical fears; parent measures detected only slightly more (positive percent agreement ranged 8–58%). This result is not surprising given that atypical fears have historically been attributed to ASD alone. Nonetheless, such behaviors present as excessively fearful reactions to the environment that result in avoidance, somatic symptoms, or other arousal regulation behaviors—much like anxiety. Furthermore, they occurred in less than half of this sample (48%), suggesting that atypical fears are not universal or core features of ASD. In contrast to other measures, the NASSQ-A, a self-report measure of the negative cognitions that often underlie and maintain anxiety disorders, detected 58% of youth with atypical symptoms. It may be that cognitive processes associated with anxiety disorders, like negative cognitive style, better elucidate the range of anxiety symptoms evident in ASD than measures designed around specific DSM-IV criteria.

The presence of atypical symptoms and other social and behavioral difficulties in youth with ASD may influence a parent’s or child’s interpretation of anxiety symptoms and therein reduce agreement between brief measures and a clinical interview. For example, perhaps parents and children are less likely to report comorbid anxiety symptoms as anxiety when they co-occur with more unusual, ambiguous fears, or social deficits commonly associated with ASD. If some parents are accustomed to viewing their child’s difficulties through the lens of ASD, other explanations may not be pursued. It is worth noting that the measurement of atypical anxiety in this study may inform such hypotheses, but likely did not influence the poor agreement between measures. Brief anxiety measures were only moderately correlated suggesting limited agreement between measures even when atypical and ambiguous symptoms are not expressly assessed.

Given the small sample size, findings are preliminary and should be interpreted with caution. This is particularly true for cut-off scores derived from the visual examination of PPVs, a method susceptible to the ratio of cases versus non-cases in a sample. Nonetheless, the rate of DSM-IV anxiety disorders in the present sample (37%) was similar to that estimated for the ASD population in a recent meta-analysis (39.6%) (Van Steensel et al., 2011), supporting the appropriateness of this approach. In addition, smaller sample sizes are not unprecedented in initial ROC studies, particularly within ASD research (Grodberg et al., 2014; Mazefsky et al., 2011; Stone et al., 2008). Research is needed to examine the influence of child ethnic background, socioeconomic status, age, and IQ on these results. Anxiety symptoms may present or be conceptualized differently in youth from different socioeconomic and cultural backgrounds (Austin and Chorpita, 2004). Furthermore, slightly older and cognitively advanced children were more likely to complete the child SCARED and BASC in this study, though mean IQ was in the average range for both completers and non-completers.

In this study, the PARS was administered directly after the ADIS-C/P, a sequence that may have influenced parents’ understanding and awareness of anxiety symptoms. To differentiate DSM anxiety from other symptoms of distress in ASD, we expanded the ADIS-C/P to capture both DSM-IV consistent and inconsistent anxiety symptoms. This approach is relatively new (see Kerns et al., 2014a), and the psychometric properties of the expanded ADIS- C/P instrument need further examination. Finally, it is unknown whether administering the ADIS-C/P concurrently with the child and parent (as done with 70% of the current sample) influences their ratings of anxiety, compared to individual administrations.

Strengths of this study include the use of a non-treatment-seeking sample, a systematic method to differentiate anxiety from atypical symptoms of distress in ASD, and a multi-informant, multi-method approach. Future research should continue to investigate the use of anxiety measures designed for typically developing children in youth with ASD, with increased attention to possible atypical manifestations of anxiety in this population. Our results suggest that new or adapted measures are warranted and that measures that operate transdiagnostically to capture symptoms of negative affect, anxiety, arousal and fear in youth with various diagnostic presentations may be particularly helpful. In their current state, anxiety measures may miss many individuals with ASD who have anxiety that could be treated. This may be acceptable within certain research contexts if the potential for false negatives would not hamper the interpretation of results. Ultimately, however, scientific understanding and effective translation to clinical practice will require a more complete and accurate measurement of anxiety within ASD.