Preliminary evidence suggesting caution in the use of psychiatric self-report measures with adolescents with high-functioning autism spectrum disorders

Abstract

This study investigated the utility of self-report measures to screen for psychiatric comorbidities in autism spectrum disorders (ASDs). Thirty-eight 10–17 year olds with an ASD and without mental retardation completed: the Children’s Depression Inventory-Short version (CDI-S), Revised Children’s Manifest Anxiety Scale (RCMAS), Conners-Wells Adolescent Self-report Scale-Short edition (CASS-S), and Short Leyton Obsessional Inventory-Child Version (SLOI-CV). Their parents were interviewed with the Autism Comorbidity Interview-Present and Lifetime (ACI-PL) to establish psychiatric diagnoses. Sensitivity, specificity, positive and negative predictive values, and reliability coefficients were calculated for each self-report and compared to values from literature. The CDI-S and CASS-S yielded a high number of false negatives, with lower sensitivities and specificities in the sample than the literature. There was a nearly significant difference in total mean RCMAS scores between participants with and without anxiety, though again the means for both groups were below the threshold of concern. The SLOI-CV yielded a high false positive rate. All four instruments had reliability coefficients comparable to literature values. Results must be considered preliminary due to sample size. However, the findings suggest that although self-report instruments may provide useful information in the diagnosis of psychiatric comorbidities in ASD, caution must be exercised in their interpretation.

1. Introduction

Available evidence suggests that psychiatric comorbidity in autism spectrum disorders (ASDs) is a significant and prevalent problem (see Matson & Nebel-Schwalm, 2007 for review). For example, children with ASD have more psychiatric comorbidity than children with mental retardation (Brereton, Tonge, & Einfeld, 2008). A study using a population-based sampling method and a structured psychiatric interview (not specifically modified for ASDs) found that 70% of 10–14-year-old children with ASD met criteria for at least one comorbid psychiatric diagnosis and 41% had two or more. A study using an interview specifically modified for ASD, but not a population-based sample, found even higher rates; specifically they found that 72% met criteria for two or more psychiatric diagnoses (Leyfer et al., 2006).

Disorders that seem to be more commonly comorbid with ASD include depression (e.g. Stewart, Barnard, Pearson, Hasan, & O’Brien, 2006), anxiety (e.g. Gillott, Furniss, & Walter, 2001; White, Ollendick, Scahill, Oswald, & Albano, 2009), obsessive–compulsive disorder (OCD; e.g. Russell, Mataix-Cols, Anson, & Murphy, 2005), and attention-deficit/hyperactivity disorder (ADHD; e.g. Holtman, Bolte, & Poustka, 2007). However, specific rates for the various disorders are very unstable and true prevalence rates remain unknown. For example, the reported prevalence range for anxiety disorders in ASD has varied from 11% to 84% across studies (see White, Oswald, Ollendick, & Scahill, 2009 for review). One factor contributing to the wide ranges is the lack of large-scale epidemiological studies, but another is highly variable measurement and assessment of psychiatric comorbidity across studies (Matson & Nebel-Schwalm, 2007).

Differentiating impairment related to having an ASD versus impairment due to a separate psychiatric comorbid disorder is challenging. Recently, a few structured parent/caregiver interviews have been designed or modified specifically to assess comorbid psychiatric symptoms in ASD, such as the Autism Comorbidity Interview-Present and Lifetime Version (Leyfer et al., 2006) and the Baby and Infant Screen for Children with aUtism Traits (Matson et al., 2009; Matson, Fodstad, & Mahan, 2009). Although the development of such comprehensive interviews is clearly an important methodological advance, such measures are still in their infancy, can be time-intensive, and require highly trained personal to administer them who have experience both with ASD and general psychiatry. Often, questionnaires or other forms of screening are an important first step. Although again there are not many questionnaires developed specifically for ASD, there are a few with a small body of growing psychometric research supporting their use for individuals with ASD and comorbid intellectual disability, such as the Autism Spectrum Disorders-Comorbid for Children (Matson & Wilkins, 2008; Matson, LoVullo, Rivet, & Boisjoli, 2009) and the Autism Spectrum Disorders - Comorbid for Adults (Lovullo & Matson, 2009; Matson & Boisjoli, 2008).

Many Diagnostic and Statistical Manual of Mental Disorders-Fourth Edition, Text Revision (DSM-IV-TR; American Psychiatric Association, 2000) diagnostic criteria require a subjective description of experience, but even verbal, higher-functioning individuals with ASD tend to have limited self-awareness as a component of their disability. In addition, the reliability and validity of self-report measures in this population are unclear (Lainhart, 1999). None of the commonly used self-report measures have been validated specifically for individuals with ASD. Determining the utility of self-report through an examination of its concurrent validity with clinician interview and parent-report will be critical in establishing a methodology for diagnosing comorbidity in ASD.

1.1. The use of self-report screening questionnaires in typically developing populations

The self-report method of assessment for psychiatric symptoms has been widely used and studied with typically developing children and adolescents. Although traditionally kindergarten age was considered the earliest point at which children had enough self-awareness to report on their own symptoms, there is evidence to suggest that even preschool-aged children are able to provide valuable information on some core depressive and anxiety symptoms to aid in diagnostic decision-making if the format is modified so as not to require reading (e.g. Luby, Belden, Sullivan, & Spitznagel, 2007). Self-report questionnaires of anxiety and depressive symptoms have also been developed for use with children as young as 6 and 7 years old respectively (e.g. Kovacs, 1992; Reynolds & Richmond, 1985). Older children (11–17 years old) may provide reports that have slightly more agreement with parent-reports, but this effect is considered quite mild (Jensen et al., 1999).

Meta-analyses of studies suggest that self-reports of children and adolescents generally have a low correlation, in the range of 0.20, with their parents’ reports on companion measures (e.g. Achenbach, McConaughty, & Howell, 1987; Duhig, Renk, Epstein, & Phares, 2000; Renk & Phares, 2004). However, some studies have reported higher concordance rates. For example, correlations between the Conners-Wells’ Adolescent Self-Report Scale: Short Form and its parent version are 0.49 for boys and 0.41 for girls (Conners, 1997). It is difficult to explain why certain questionnaires or methods of assessment lead to higher concordance rates than others, as there do not seem to be any specific or consistent patterns (Jensen et al., 1999). Rather, there are many factors that influence concordance rates between self-report by children or adolescents and their parents, including the instruments used, populations studied, child’s age and gender, and disorders under investigation (Achenbach et al., 1987; Jensen et al., 1999). The assumption is that both sources provide unique and meaningful information, and thus using multiple reporters is the recommended best practice approach (Achenbach et al., 1987; Jensen et al., 1999). Specifically, Jensen et al. (1999, p. 1577) concluded that “for most conditions among 9–17-year-old children, both parent and child informants are necessary to obtain adequate diagnostic information, even though using only one informant may be appealing as more convenient or less costly.”

Perhaps an even more important testament to the utility of self-report questionnaires is how useful they are in diagnostic assignment. For example, adolescents’ self-report significantly increased the amount of variance explained in clinician diagnosis when added to parent-report (Becker, Hagenberg, Roessner, Woerner, & Rothenberger, 2004). In fact, Becker et al. (2004) found that self-report questionnaires were slightly superior to parent-report questionnaires in terms of predicting the presence of (any) psychiatric disorder in the child. Considering the sensitivity and specificity of certain self-report questionnaires further demonstrates their discriminative validity related to making specific diagnoses. For example, investigations of the Children’s DepressionInventoryself-reporthavefoundthatitcanbeusedtocorrectlyclassify80%ofdepressedchildrenasdepressedand84% of children who are not depressed as non-depressed (Kovacs & Staff, 2003). Overall there appears to be significant evidence in favor of the use of self-report instruments as part of the diagnostic process in typically developing populations.

1.2. The use of self-report screening questionnaires in ASD populations

Although the reliability and validity of self-report measures of psychiatric symptoms in ASD remains unknown, this method of assessment has been used in a variety of studies. For example, a few studies have utilized self-report measures of psychiatric symptoms as outcome measures for treatment studies in children and adolescents with ASD. For example, Chalfant, Rapee, and Carroll (2007) found that participants’ with ASD scores on self-report measures reflected a reduction in anxiety symptoms after a cognitive-behavioral treatment that were supported by parental reports. On the other hand, a pilot study of an anxiety treatment in ASD found that three of four participants demonstrated no change in their self-report of anxiety, despite significant changes in parent-reported measures and changes in their diagnostic status (White, Ollendick, et al., 2009). Another study showed a statistically non-significant decrease in self-reported depression scores after a social adjustment enhancement intervention, but the results were difficult to interpret due to small sample size and interactions of age and cognitive disability with the scores (Solomon, Goodlin-Jones, & Anders, 2004).

Some findings regarding the self-report of emotional responses in ASD raise questions about the validity of this method. For example, Shalom et al. (2006) presented high-functioning children with ASD and typically developing children with pleasant, unpleasant, and neutral pictures. Their physiological responses were measured and they were asked to self-report the degree to which they thought the pictures were pleasant or interesting. The results indicated that both groups had similar physiological responses to the stimuli, but different self-reports (Shalom et al., 2006). Although there are many potential interpretations for this finding, it could suggest that the children with ASD have difficulty identifying their own feelings. This notion was generally supported by a study of alexithymia (difficulty understanding one’s own emotions) in adults with ASD (Berthoz & Hill, 2005). Specifically, adults with ASD were found be more alexithymic overall as compared to normal adult controls, which was primarily due difficulty identifying, verbalizing, and analyzing their own emotions rather than experiencing emotions differently (Berthoz & Hill, 2005). Difficulty with these types of skills could suggest that the use of self-reports in ASD may not lead to an accurate portrayal of their psychiatric symptoms.

Yet, no studies, to our knowledge, have directly compared results from children and adolescents’ psychiatric self-report questionnaires with structured parent interview in ASD samples. Meyer, Mundy, Van Hecke, and Durocher (2006) administered both the parent- and child-report versions of a broad psychiatric screening instrument, the Behavioral Assessment System for Children, to children with Asperger’s syndrome and reported that “parent and child-reports of comorbidity were not significantly associated” but no further detail was provided. There has generally been very little systematic examination of the psychometric properties of self-report measures administered to children and adolescents with ASDs.

In sum, there is growing evidence to suggest significant psychiatric comorbidity in ASDs. Yet, there is a dearth of research in this area and the diagnosis of comorbid conditions remains difficult and variable across providers and studies. In typically developing populations, self-report measures are widely used and have successfully aided in the psychiatric diagnosis process. If self-report measures can be shown to be valid and reliable for children and adolescents with ASD, they can be valuable tools for clinicians to assess psychiatric comorbidities in their patients with ASD. Therefore, this study aimed to explore the utility and psychometric properties of four self-report measures that screen for depression, anxiety, attention-deficit/hyperactivity disorder, and obsessive–compulsive disorder, the Children’s Depression Inventory-Short version (CDI-S), Revised Children’s Manifest Anxiety Scale (RCMAS), Conners-Wells Adolescent Self-report Scale-Short edition (CASS-S), Short Leyton Obsessional Inventory-Child Version (SLOI-CV), respectively, in a sample of high-functioning children and adolescents with ASD. Sensitivity, specificity, positive and negative predictive values, and internal reliability were explored.

2. Method

2.1. Participants

The primary inclusion criteria included a diagnosis of either autistic disorder, Asperger’s disorder, or pervasive developmental disorder, not otherwise specified (PDD-NOS) and an age between 10 and 17 years old. Participants were excluded if they had comorbid mental retardation, as defined by an intellectual ability quotient (IQ) less than 70 on the Wechsler Abbreviated Scale of Intelligence (WASI; Psychological Corporation, 1999) and accompanying adaptive behavior impairments on the Adaptive Behavior Assessment System-Second Edition (ABAS; Harrison & Oakland, 2003).

Participants were recruited through word of mouth and fliers. Recruitment efforts were predominantly focused within a children’s hospital that has a diagnostic clinic for developmental disorders. Having a comorbid psychiatric disorder was not an inclusionary criterion and psychiatric comorbidity status was not incorporated into recruitment efforts. This method was intended to result in a sample with sufficient variability in comorbidity status to study the utility of self-report questionnaires in differential diagnosis.

The presence of an ASD was supported by Modules 3 or 4 of the Autism Diagnostic Observation Schedule-Generic (ADOS-G; Lord et al., 2000) and Autism Diagnostic Interview-Revised (ADI-R; Lord, Rutter, & Le Couteur, 1994), administered by the first author or by a research associate who has research-level reliability for administration of these measures. Most met the ASD subscale cutoffs on the ADOS-G: 34 (89.5%) met the communication cutoff, 36 (94.7%) met the social cutoff, and 35 (92.1%) met both the communication and social cutoffs. The majority also met the subscale cutoffs for a diagnosis of autistic disorder on the ADI-R: 30 (78.9%) met the social cutoff, 27 (71.1%) met the communication cutoff, and 35 (92.1%) met the developmental abnormality cutoff. Fewer subjects were expected to exceed the cutoffs on the ADI-R because it is designed to detect autistic disorder and our sample included Asperger’s Disorder and PDD-NOS as well. All diagnoses were confirmed by the expert opinion of a licensed clinical psychologist who specializes in ASD (CAM).

As shown in Table 1, participant characteristics were consistent with other studies of high-functioning ASD. The 38 participants were mostly male (82%), had a mean Full-Scale IQ in the average range, and mean adaptive behavior scale composite in the impaired range. Parents also completed a questionnaire that gathers information on the participant’s medical, educational, and psychiatric histories. Per parent-report, the majority of participants had received special education services (73.7%, n = 28). Nearly 87% (n = 33) had received outpatient psychiatric care (86.8%, n = 33) and 74% (n = 28) had been prescribed some type of psychotropic medication, but far fewer (21.1%; n = 8) had overnight stays in a psychiatric hospital. Less than half the sample was currently taking psychotropic medications (42.1%; n = 16). The specific type of medication markedly varied, though the most common were either some form of mood stabilizer or ADHD-related medication (31.6%; n = 12 for each).

Table 1.

Participant demographics.

Participant characteristics (n = 38)	Mean	SD	Range
Age	12	2	10–17
Percent male	82%
WASI Full-Scale IQ	105	17	71–144
ABAS II adaptive behavior scale composite	66	15	43–97
Research diagnosis
Autistic disorder	21.1% (8)
Asperger’s disorder	60.5% (23)
Race
Caucasian	89.5% (34)
African-American	2.6% (1)
Hispanic	2.6% (1)
Bi-racial	5.3% (2)

Group assignment as “comorbid” versus “non-comorbid” varied by diagnosis, and specific numbers are presented in Table 2. As described in further detail below in the Analyses section, group assignment for the four main analyses depended on if children did or did not meet subthreshold or greater current diagnostic criteria for: (1) any DSM-IV-TR depressive disorder; (2) any DSM-IV-TR anxiety disorder; (3) ADHD; or (4) OCD. There were no significant differences between groups (e.g. between the ADHD group and not ADHD group, between the OCD group and not OCD group, etc.) in Full-Scale IQ or age (p > 0.05, specific results available upon request).

Table 2.

Frequency of (current) psychiatric diagnoses based on parent-report on the Autism comorbidity interview—present and lifetime.

Disorder	Full DSM criteria % (n)	Subthreshold or greatera % (n)
Major depression	15.8 (6)	26.3 (10)
Depression NOS	2.6 (1)	2.6 (1)
Any depression	18.4 (7)	28.9 (11)
Generalized anxiety	7.9 (3)	15.8 (6)
Separation anxiety	7.9 (3)	7.9 (3)
Panic disorder	0 (0)	0 (0)
Social phobia	13.2 (5)	34.2 (13)
Specific phobia	13.2 (5)	31.6 (12)
OCD	2.6 (1)	10.5 (4)
Any DSM anxietyb	28.9 (11)	55.3 (21)
ADHD	36.8 (14)	57.9 (22)

^aThis number includes those participants who met full DSM-IV-TR criteria, subsyndromal criteria, or subthreshold criteria.

^bAny DSM-IV-TR anxiety disorder, excluding OCD.

2.2. Measures

2.2.1. Structured psychiatric interview

Both the participants and the participants’ parents were interviewed by the first author with the Autism Comorbidity Interview-Present and Lifetime version (ACI-PL; Leyfer et al., 2006). The ACI-PL was modified from the Kiddie Schedule for Affective Disorders and Schizophrenia, a structured-interview instrument designed to diagnose psychiatric disorders in children and adolescents (Ambrosini, 2000). The modifications included the addition of an introductory section that assesses the child’s emotions and behaviors at baseline, as well as additional screening questions at the beginning of each disorder section assessing additional observable features and applicability of symptoms to the child (e.g. an increased feeling of guilt is not an applicable symptom of depression in a child who does not understand guilt). The ACI-PL was piloted on 109 children (age 5–17) with a diagnosis of autistic disorder and was found to have good inter-rater reliability, test–retest reliability, and criterion and concurrent validities for the diagnosis of various psychiatric disorders (Leyfer et al., 2006). Diagnoses of comorbid psychiatric disorders in the current study were made based on the findings on parental interviews with the ACI-PL.

2.2.2. Self-report questionnaires

The participants completed four self-report measures designed to screen for various psychiatric disorders. Short versions were chosen when available, in hopes of identifying the most efficient screening tools that would be tolerable to children and adolescents with ASD. They included:

The Children’s Depression Inventory: Short version (CDI-S) is a self-report measure designed to screen for depression in children. The CDI-S was developed from the longer Children’s Depression Inventory, consists of 10 items, and yields results comparable to the original CDI (Kovacs, 1992). From the original normative sample, alpha was found to be 0.86 for CDI and 0.80 for CDI-S. Kappa ranged from 0.38 to 0.82 in different population samples (Kovacs, 1992). Craighead, Craighead, Smucker, & Ilardi, 1998 obtained a sensitivity of 0.69 and specificity of 0.91 for the CDI using score of 19 as cutoff point. Sensitivity and specificity of CDI-S were not found in literature but could be assumed to be similar to those of CDI because the two forms were found to yield comparable results in the normative samples.

The Revised Children’s Manifest Anxiety Scale (RCMAS) is a self-report measure designed to screen for anxiety disorders in children that has demonstrated good reliability and factorial and construct validity (Reynolds & Richmond, 1985). For the RCMAS, alpha is approximately 0.84, while kappa ranges from 0.68 to 0.98 in different population samples. Using a t-score > 60 as the cutoff, Hodges (1990) reported high specificity (92%) but low sensitivity (34%) when using RCMAS to differentiate between children with and without anxiety.

The Conners-Wells Adolescent Self-Report Scale-Short Edition (CASS-S) is a self-report measure designed to identify children with ADHD that has demonstrated excellent reliability and validity (Conners, 1997; Conners et al., 1997). The CASS-S manual cited alphas of 0.75–0.90, with kappa ranging from 0.60 to 0.90 (Conners, 1997). In terms of identifying children with ADHD using the CASS, Conners et al. (1997) reported a sensitivity of 0.81 and a specificity of 0.84 using an unreported t-score as cutoff. Sensitivity and specificity of CASS-S were not found in literature but should be comparable to the original CASS.

The Short Leyton Obsessional Inventory-Child Version (SLOI-CV) is a self-report measure designed to identify OCD symptoms in children. The SLOI-CV has demonstrated good internal reliability and validity (Bamber, Tamplin, Park, Kyte, & Goodyer, 2002). For SLOI-CV, alpha was calculated to be 0.90. Sensitivity was 0.78 and specificity was 0.70 with respect to differentiating children with and without OCD using a cutoff score of 5 (Bamber et al., 2002).

2.3. Procedures

After participants expressed an interest in the study, an appointment was scheduled where the ADOS-G, ADI-R, WASI, ABAS, and demographic forms were completed to verify that they met inclusion/exclusion criteria. The ADOS-G was always administered first, or simultaneously with the ADI-R. Given that the distance families travelled to complete the study varied, some parents chose to have their children complete the ACI-PL and self-report questionnaires during the first visit as well. Participants were provided with as many breaks as they wanted. Most families came back for a second visit to complete the ACI-PL. For a few participants, due to work schedule and travel demands, the parent version of the ACI-PL was completed over the phone. Nine families were able to have both parents/guardians complete the ACI-PL (with both parents present together during administration), but the rest were completed by just one parent (mothers in all cases but three). Both parents were not required to participant due to feasibility concerns and the intensive time-burden inherent in the ACI-PL and other measures.

Due to ethical concerns, we were unable to ask participants to suspend psychotropic medication use for the study. Furthermore, we did not with to exclude participants on psychotropic medications given that it would significantly truncate our sample. Per ACI-PL guidelines, parents of participants on psychotropic medications were asked to describe their child’s emotions and behavior before medication treatment was started, or during periods when the medication was not taken. Similar instructions were given to participants when they completed the self-report questionnaires. Scores on the CASS-S did not differ between those on ADHD-related medications and those not on ADHD-related medications, and the overall score on the CDI-S did not differ between those on anti-depressants and those not on anti-depressants, p > 0.05.

2.4. Analyses

The frequencies of participants meeting full DSM-IV-TR criteria for various psychiatric disorders based on parent-report on the ACI-PL were calculated. Frequencies were also calculated for participants meeting subthreshold or greater criteria (including subthreshold, subsyndromal, and full DSM-IV-TR criteria) based on ACI-PL parent-report. Subthreshold or greater diagnoses were used in analyses, as opposed to using those who met full DSM-IV-TR criteria only. This decision was based on our intent to evaluate the self-report questionnaires as screening tools, in which case higher sensitivity is desired; thus, we wished to identify those with diagnostically relevant concerns using a fairly liberal definition.

Given our large IQ range, Pearson’s correlations were conducted between the total scores for the four self-report measures and Full-Scale IQ. Summary statistics were also calculated for the self-report questionnaires. The mean and standard deviation (SD) for each subscale and total score on the four self-report measures were calculated separately for (a) the full sample regardless of comorbidity, (b) participants comorbid for a related disorder, and (c) participants not comorbid for a related disorder. The participants’ scores on each of the self-report measures were compared to the cutoff scores suggested by literature/manuals of the self-report instruments that signify a level of significant clinical concern regarding the possible presence of the disorder. If the participant’s score exceeded cutoff, they were considered a positive screen. The self-report results were then compared with the diagnosis based on the parent-reported subthreshold or greater symptoms via the ACI-PL using independent samples t-tests.

Sensitivity (probability of a positive test among participants with a disorder; e.g. probability of a true positive) and specificity (probability of a negative test among patients without a disorder; e.g. probability of a true negative) were calculated for each self-report instrument. For calculations of sensitivity and specificity, both traditional and alternate cutoffs indicating a “positive screen” were explored to determine if the alternate would improve utility for individuals with ASD. In particular, for the CDI-S, both t-scores > 61 corresponding to “above average” and t-scores > 56 corresponding to “slightly above average” (Kovacs, 1992) were tested. For CASS-S, both t-scores > 65 corresponding to “moderately atypical” and t-scores > 56 corresponding to “slightly atypical” (Conners, 1997) were tested. Similarly, a t-score of 56 on RCMAS was tested in addition to the t-score of 61 suggested by literature. A score of 5 on SLOI-CV was considered a positive screen as suggested in the literature. For each self-report measure, the sensitivity and specificity obtained in our sample were then compared to sensitivity and specificity reported in literature. For CDI-S and CASS-S, sensitivity and specificity values were not available in literature, so the sensitivity and specificity for CDI and CASS (the longer versions of these measures) were used instead in the data analysis.

In addition, the positive predictive value (ratio of true positives to combined true and false positives), and negative predictive value (ratio of true negatives to combined true and false negatives) were also calculated to consider the utility of the tests while also taking into account the number of children with these comorbid problems in the sample. The internal reliability of the various self-report measures was calculated (coefficient alpha internal consistency) and compared to the values found in the manuals.

3. Results

Table 2 summarizes the frequencies of current diagnoses based on parent-report on the ACI-PL, including both the frequencies of participants meeting full DSM-IV-TR criteria and the frequencies of participants meeting subthreshold or greater criteria (including subthreshold, subsyndromal, and full DSM-IV-TR criteria). Participants showed a high rate of current psychiatric comorbidity, with the total number of current full DSM-IV-TR criteria diagnoses ranging from 0 to 4 across the participants. The mean number of diagnoses was 1.06, with a standard deviation of 1.01. When the threshold was lowered to include those participants meeting subthreshold or greater criteria, 28.9% were found to meet criteria for current diagnoses of any form of depression, 55.3% with some form of DSM-IV-TR anxiety (excluding OCD), 10.5% with OCD, and 57.9% with ADHD.

Full-Scale IQ was not significantly correlated (p > 0.05) with the self-report questionnaire total scores, with the exception of the RCMAS, r (37) = −0.39, p = 0.015. To aid in interpretation of this finding, an additional analysis was conducted to determine if there was a relationship between IQ and anxiety disorder status based on the ACI-PL. Results of an independent samples t-test indicated that the anxious and non-anxious groups (based on current subthreshold or greater symptoms on the ACI-PL) did not differ based on Full-Scale IQ, t (34) = −0.22, p > 0.05. Further, Full-Scale IQ was not significantly correlated with the Lie Scale of the RCMAS, p > 0.05.

Table 3 summarizes the results of each self-report questionnaires compared to current comorbidity status based on parent-reported diagnoses of subthreshold or greater symptoms on ACI-PL. All means of the total scores, regardless of comorbidity status, were below the cutoff threshold for a “positive screen” on CDI-S, RCMAS, and CASS-S. On the CDI-S, CASS-S, and SLOI-CV, the means in the non-comorbid group were lower than the means in the comorbid groups by approximately half of a standard deviation, but none of the total scores significantly differed between groups (p > 0.05; specific results available upon request). Using Cohen’s d convention of effect size (Cohen, 1988), all effect sizes were in the small range. Taking the effect sizes and sample sizes into account, post hoc power analyses also revealed low power (less than 0.42 in all cases). However, for the RCMAS, there was a nearly significant difference in the mean total score between the comorbid and non-comorbid groups, t (35) = −1.96, p = 0.058. The means on SLOI-CV were elevated well above the threshold of concern for all participants, regardless of OCD status.

Table 3.

Self-report questionnaire results by current comorbidity status.

Self-report instrument	Threshold for a positive screen	Subscales	ACI parent-report diagnoses questionnaires were compared toa	Full sample		Participants NOT comorbid for disorder			Participant comorbid for disorder
				Mean	SD	Mean	SD	n	Mean	SD	n
CDI	61	Total score	All DSM-IV depressive disorders	51.0	10.6	49.5	9.4	26	55.4	12.6	11
RCMAS	61	Total anxiety	All DSM-IV anxiety disorders except obsessive–compulsive disorder	52.1	12.7	48.0	11.0	16	55.9	12.9	21
		Physiological anxiety		10.7	3.8	9.6	3.3	16	11.7	4.0	21
		Worry/over sensitivity		9.4	3.0	8.6	2.8	16	10.1	3.0	21
		Social concerns		10.4	3.1	9.3	3.0	16	11.4	2.8	21
		Lie		9.1	2.8	9.1	3.2	16	9.1	2.7	21
CASS-S	65	ADHD index	Attention-deficit/hyperactivity disorder	56.4	10.7	54.2	11.0	14	58.4	10.0	22
		Conduct problems		53.9	9.5	52.9	11.5	14	55.3	8.1	22
		Cognitive problems		56.3	10.4	54.3	11.5	14	58.0	9.0	22
		Hyperactivity		53.7	9.8	51.9	13.3	14	54.6	7.3	22
SLOI-CV	5	Total score	Obsessive–compulsive disorder	9.7	7.5	9.3	7.8	32	13.8	5.4	4
		Compulsions		2.7	3.3	2.5	3.4	32	3.8	3.0	4
		Obsessions		4.2	2.9	4.2	2.9	32	5.5	3.1	4
		Cleanliness		2.8	2.9	2.6	2.9	32	4.5	3.4	4

^aComparisons were based on parent-reported diagnoses on the Autism comorbidity interview-present and lifetime of subthreshold or greater current symptoms.

When subscales on RCMAS, CASS-S, and SLOI-CV were analyzed, the means on the subscales reflected the same pattern of lower scores in the non-comorbid than in the comorbid group as found in the total scores. However, the only subscale for which the mean scores showed a statistically significant difference between the comorbid and non-comorbid groups was the RCMAS social concern subscale, t (35) = −2.18, p = 0.036. As a follow-up to this finding, the means for the RCMAS total and subscale scores were calculated for those with subthreshold or greater social phobia diagnoses. The results were nearly identical to the full sample scores on the RCMAS (specific results available upon request). Further, there was no significant difference in the social concern subscale scores of the RCMAS between those with subthreshold or greater social phobia diagnoses and those without, t (35) = −0.37, p > 0.05. The lie subscale on the RCMAS yielded similar scores in both participants with and without subthreshold or greater levels of any anxiety (9.13 and 9.14), which are both well within the normal range (e.g. do not suggest concern regarding the child’s intention to provide false information or a generally inaccurate perception of self).

Table 4 summarizes the psychometric properties of the self-report questionnaires, including the performance of alternative cutoffs for positive screens. The sensitivity and specificity for CDI-S, RCMAS, and CASS-S were all significantly lower in the test sample than the values reported in literature in their standardization samples, even when alternative cutoffs were explored. When the threshold of the CDI-S was lowered from 61 to 56, sensitivity was improved at the expense of specificity, while positive and negative predictive values did not change substantially. Similarly, when the threshold for a positive screen was lowered from 65 to 56 on CASS-S, sensitivity was improved, specificity was lowered, and positive and negative predictive values did not change substantially. When the threshold of the RCMAS was lowered from 61 to 56, sensitivity, positive and negative predictive values improved, while specificity remained unchanged. For the SLOI-CV, the sensitivity was very high while positive predictive value was very low. The reliability of each self-report questionnaire was high in the test sample, with Cronbach’s alpha coefficients comparable to those reported in literature.

Table 4.

Psychometric properties of self-report questionnaires, including alternative cutoffs.

Self-report	Threshold	Sensitivity	Specificity	PPV	NPV	Reliability
CDI-S (test sample)	61	0.27	0.89	0.50	0.74	0.80
CDI-S (test sample)	56	0.45	0.77	0.45	0.77	0.80
CDI-S (literature)a	61	0.69	0.91			0.80
RCMAS (test sample)	61	0.33	0.88	0.78	0.50	0.88
RCMAS (test sample)	56	0.52	0.88	0.85	0.58	0.88
RCMAS (literature)	61	0.34	0.92			0.84
CASS-S (test sample)	65	0.23	0.79	0.63	0.39	0.86
CASS-S (test sample)	56	0.55	0.57	0.67	0.44	0.86
CASS-S (literature)b	NAc	0.81	0.84			0.75–0.90
SLOI-CV (test sample)	5	1.00	0.31	0.15	1.00	0.87
SLOI-CV (literature)	5	0.78	0.70			0.90

^aIn this row: sensitivity and specificity of CDI, reliability coefficient of CDI-S.

^bIn this row: sensitivity and specificity of CASS, reliability coefficient of CASS-S.

^cUnknown threshold.

4. Discussion

Psychiatric comorbidity in ASD is acknowledged as a rampant and significant problem, but it remains difficult to accurately identify psychiatric symptoms and disorders in individuals with ASD (Matson & Nebel-Schwalm, 2007). Thus far, there is no agreed upon method or set of measures for differentiating impairment related to having an ASD from that which is due to a separate psychiatric disorder. Although a structured diagnostic interview, i.e. the ACI-PL (Leyfer et al., 2006) has been recently developed for diagnosing psychiatric comorbidities in the ASD population, it is time-consuming and requires extensively trained personnel to administer. If valid and reliable for children with ASD, utilizing self-report screening questionnaires may be an efficient way to identify children at risk for psychiatric comorbidities and in need of more in-depth assessment. This study aimed to investigate the psychometric properties of four commonly used self-report measures as screening tools for psychiatric comorbidities in high-functioning children with ASD.

The need for screening measures for psychiatric disorders in ASD was confirmed by the high rates of psychiatric comorbidities found in our sample, which was not a sample specifically enriched for high rates of psychiatric concerns. Over 25% of participants presented with current subthreshold or greater depressive symptoms, and 50% or more presented with subthreshold or greater levels of current anxiety and ADHD symptoms based on parent-report on the ACI-PL. Unfortunately, results of the self-report measures did not correspond to these findings on the ACI-PL. All of the total scores, regardless of comorbidity status, were below the cutoff threshold for a “positive screen” on the CDI-S, RCMAS, and CASS-S. Accordingly, the sensitivity and specificity for CDI-S, RCMAS, and CASS-S were all significantly lower in the test samples than the values reported in literature, even when alternative cutoffs were explored. The means on SLOI-CV were elevated well above the threshold of concern for all participants, regardless of OCD status. Therefore, it resulted in the presence of many false positives.

It is difficult to make firm conclusions regarding the reason for the lack of concordance between the self-report results and parent-report on the ACI-PL. Many research studies have investigated the concordance rates between parent-report and child-report of symptoms of various psychiatric disorders in typically developing populations. These studies have yielded widely heterogeneous data suggesting that the concordance rates between parent-report and child-report vary depending on many factors, including the instruments used, the disorders under investigation, and the populations being studied, among other factors (Achenbach et al., 1987; Jensen et al., 1999).

It is possible that the lack of correspondence between self-report screeners and parent-reported psychiatric diagnoses in our sample is the result of children and adolescents with ASD having difficulty adequately identifying and reporting their own symptoms of depression, anxiety, and ADHD, consistent with findings of alexithymia in this population (e.g. Berthoz & Hill, 2005). However, it is somewhat encouraging that scores were higher in the comorbid than the non-comorbid groups, albeit only by an average of half of a standard deviation and despite lack of a significant difference. This may indicate that the participants do have some ability to report their own symptoms on self-report measures, though not at a level adequate for clinical diagnostic purposes. The high scores on the self-report of OCD, despite low levels of OCD in our sample, suggest that the SLOI-CV may be detecting some underlying repetitive thoughts and behaviors inherent to ASD. This finding is consistent with the body of research focused on symptom overlap between the two disorders and related phenomological and diagnostic debates (e.g. Cath, Ran, Smit, van Balkom, & Comijs, 2008; Matson & Nebel-Schwalm, 2007; Russell et al., 2005).

The results regarding internal reliability were encouraging, given that reliability scores were as good, or better, than the standardization samples. This suggests that the participants were able to read and understand the items, at least well enough to provide consistent answers. In addition, scores on the lie subscale on RCMAS were well within the normal range, suggesting that the participants were not providing false information due to either a generally inaccurate perception of self or the intention of providing socially desirable responses.

There were some unique findings regarding the RCMAS. Of all of the self-report measures, it showed the most promise. There was a nearly significant difference in total mean scores on the RCMAS between those with anxiety and those without, though again means for both groups were below the threshold of concern. The mean scores on the social concerns subscale were significantly different between the anxious group and the non-anxious group. This finding did not appear to be driven by participants with subthreshold or greater social phobia symptoms, given that the social phobia group’s mean RCMAS scores mirrored the full sample means regardless of comorbidity. In addition, the significant difference found between those with any subthreshold or greater anxiety and those without was not replicated when the comparison was between those with and without subthreshold or greater social phobia specifically. Therefore, these findings may suggest that anxiety experienced by the participants may be principally based on social concerns (regardless of the specific type of anxiety disorder the child may have or be at risk for). This hypothesis is consistent with conclusions based on a study by Gillott et al. (2001) that “children with ASD exhibit social worries both covertly (e.g., rumination) and overtly (e.g., actively avoiding social interaction with peers)” (White, Oswald, et al., 2009, p. 222) and that some of the anxiety in ASD may be driven by social skills deficits (Bellini, 2006). Alternatively, our findings could suggest that social worries are the type of anxiety that participants have the most self-awareness of or are better able to self-report. Finally, the RCMAS was the only self-report measure that was significantly correlated with IQ, such that participants with lower IQs had higher scores on this measure. The Lie Scale was not significantly correlated with IQ, so it unlikely that participants with lower IQs provided less consistent or accurate answers. This, together with the lack of relationship between IQ and anxiety disorders overall in this sample, could suggest the RCMAS is most sensitive for children with ASD in the lower end of the average or borderline range of IQ.

Interpretation of the results should be done with the following limitations in mind. First, results of the study must be considered preliminary due to the small size of the sample, which reduces the power of the study. Since this is a pilot study, further studies with larger samples may provide additional valuable information on the subject matter. In addition, although we attempted to interpret our findings in the context of how the measures fare in other populations, there was no typically developing control group or group with other delays or disorders. It would be helpful to explicitly compare findings to groups with other psychiatric disorders (but not ASD) matched on intellectual ability.

Rates of disorders should also be interpreted with caution. Although participants were not recruited based on psychiatric comorbidity status, it is possible that parents of children with behavioral and emotional concerns would be more interested in a study of this nature. Furthermore, we were unable to require both parents to participate and complete the ACI-PL due to feasibility. Therefore, the accuracy of parents’ descriptions should take into account possible under- or over-reporting that may occur with only one informant. Finally, it is possible that participants’ psychotropic medication use impacted both parent and child ratings given that just under half of the sample was on some type of medication at the time of the study. Unfortunately, this is a design concern that is fairly unavoidable for studies of psychiatric comorbidity, given that medications are widely used in ASD and other populations, and it would not be ethical to ask them to discontinue helpful medications. We attempted to address this concern by having parents and participants rate the symptoms under question for the time period when the child was not on medication. This introduces some unavoidable recall bias. However, our finding that overall CASS-S and CDI-S scores did not differ based on whether the child was or was not taking an ADHD-related medication or anti-depressants, respectively, suggests that medication use alone is not a full explanation for patterns of scores found in this study.

Despite the above-mentioned limitations, some preliminary clinical implications can be drawn. First, the high number of psychiatric comorbidities found using a structured diagnostic interview instrument specifically modified for individuals with ASD was consistent with previous research indicating a significant problem with psychiatric comorbidity in ASD. This highlights the importance of screening, diagnosing, and treating psychiatric comorbidities in this population. Second, the analysis of the self-report measures used in this study suggests that high-functioning individuals with ASD may be able to report their own psychiatric symptoms to a certain degree, but not sufficiently for clinical diagnostic purposes. Therefore, clinicians working with individuals with ASD should not rely too heavily on self-report instruments for detecting psychiatric comorbidities in their patients. Self-report measures administered to individuals with ASD may yield additional valuable information for clinical decision-making, but clinicians should interpret this information carefully based on their own clinical judgments. In particular, it will be important to conduct further assessment if other signs of a possible psychiatric disorder are present based on observation or parent or teacher report, even in the event of a negative screen on a self-report measure.