Concordance between a U.S. Educational Autism Classification and the Autism Diagnostic Observation Schedule

Abstract

Objective:

States in the United States differ in how they determine special education eligibility for autism services. Few states include an autism-specific diagnostic tool in their evaluation. In research, the Autism Diagnostic Observation Schedule (ADOS for first edition, ADOS-2 for second edition) is considered the gold-standard autism assessment. The purpose of this study was to estimate the proportion of children with an educational classification of autism who exceed the ADOS/ADOS-2 threshold for autism spectrum (concordance rate).

Method:

Data were drawn from four school-based studies across two sites (Philadelphia, Pennsylvania and San Diego, California). Participants comprised 627 children (ages 2-12 years; 83% male) with an autism educational classification. Analyses included 1) calculating the concordance rate between educational and ADOS/ADOS-2 classifications, and 2) estimating the associations between concordance and child’s cognitive ability, study site, and ADOS/ADOS-2 administration year using logistic regression.

Results:

More (97.5%) San Diego participants (all assessed with the ADOS-2) met ADOS/ADOS-2 classification than did Philadelphia participants assessed with the ADOS-2 (92.2%) or ADOS (82.9%). Children assessed more recently were assessed with the ADOS-2; this group was more likely to meet ADOS/ADOS-2 classification than the group assessed a longer time ago with the ADOS. Children with higher IQ were less likely to meet ADOS/ADOS-2 classification.

Conclusions:

Most children with an educational classification of autism meet ADOS/ADOS-2 criteria, but results differ by site and also by ADOS version and/or recency of assessment. Educational classification may be a reasonable but imperfect measure to include children in community-based trials.

During the 2014-2015 academic year, 575,879 children ages 3 to 21 in the United States received services under the special education classification of autism (Institute of Education Sciences, National Center for Education Statistics, 2016). While the Individuals with Disabilities Education Act (IDEA) of 1990 established autism as a separate category of children eligible for special education services, states vary in the criteria they use to identify these children (MacFarlane & Kanaya, 2009; Pennington, Cullinan, & Southern, 2014). Of note, the special education eligibility requirements for autism services are distinct from the psychiatric diagnostic criteria (APA, 2013), and a clinical diagnosis of autism is not required or sufficient to be eligible for autism services. Compared with the psychiatric diagnostic criteria for autism, the educational classification criteria as defined by IDEA are less specific about core symptoms and include the additional stipulation that a child’s autism-related behaviors must impair his or her educational functioning. While some states require that a clinician provide an autism diagnosis for special education eligibility, others simply require that a clinician be included on the evaluation team; most states do not require any clinician participation in the evaluation process (MacFarlane & Kanaya, 2009), although they may still be involved. To determine eligibility for autism services, states commonly include assessments of intelligence, speech-language, social-developmental history, and adaptive behavior, gathered through formal testing, parent interview, and direct observation (Pennington et al., 2014). Few states include an autism-specific assessment tool (Pennington et al., 2014); when an autism-specific tool is used, it is usually a screening measure, rather than a diagnostic instrument (Aiello, Ruble, & Esler, 2017).

This evaluation process in schools highlights a gap between educational practice and research standards. For example, in research, the Autism Diagnostic Observation Schedule, First Edition (ADOS; Lord et al., 2000) or Autism Diagnostic Observation Schedule, Second Edition (ADOS-2; Lord et al., 2012) is considered the gold-standard observational autism assessment and is generally required for clinical characterization in peer-reviewed autism studies. Although the ADOS/ADOS-2 was not designed to be a stand-alone autism assessment, it often is used this way by researchers as confirmation of diagnosis for study inclusion.

The ADOS/ADOS-2 is time-consuming and requires extensive training, making it relatively inaccessible in a school setting. A survey of school and clinical psychologists found that they viewed the ADOS as advantageous for capturing ASD-specific behaviors and standardizing the structure of behavioral observation, but burdensome in terms of required resources (Akshoomoff, Corsello, & Schmidt, 2006).

How many students with an educational classification of autism meet research criteria for autism on the ADOS/ADOS-2 is an important question for several reasons. First, many epidemiological studies rely on special education data to estimate the prevalence of autism (e.g., Gurney et al., 2003; Newschaffer, Falb, & Gurney, 2005). Notably, educational records form a major part of the Centers for Disease Control and Prevention’s (CDC) population-based surveillance system, which provides highly publicized autism prevalence estimates (Rice et al., 2007). In these studies, researchers do not directly assess children. Instead, conclusions are based on data provided in the educational records, sometimes augmented by records from other sources.

Second, the accuracy of the educational classification of autism offers insight into community diagnostic and school placement practices. If many children who do not meet ADOS/ADOS-2 criteria for autism receive the educational classification of autism, it suggests the need to re-evaluate these practices and consider the differences between a clinical diagnosis (which often is supported with ADOS/ADOS-2 results) and an educational classification of autism (e.g., a child’s autism-related behaviors must interfere with his or her educational functioning to be eligible for the educational classification). Disparities between educational and clinical classifications may mean that children are receiving autism services when they are not appropriate, or that university-based studies include children that are very different from the population of children receiving autism services in schools.

Third, as autism intervention research moves towards conducting larger pragmatic trials, validating autism diagnosis becomes cost prohibitive. If the educational classification of autism is highly concordant with a research diagnosis supported by the ADOS/ADOS-2, then researchers could have more confidence in the educational classification and not spend the time or money on in-depth diagnostic validation.

The purpose of this study was to estimate the proportion of children with an educational classification of autism who exceed the ADOS/ADOS-2 threshold for autism spectrum.

Methods

Procedures

Data were drawn from three school-based studies conducted at the University of Pennsylvania, in partnership with the School District of Philadelphia, and one school-based study conducted at the University of California at San Diego (investigators are now at University of California, Davis and San Diego State University), in partnership with 17 San Diego County school districts. Philadelphia is the largest city in the state of Pennsylvania, and San Diego is the second largest city in the state of California. All studies had Institutional Review Board approval. Two of the Philadelphia studies were school-based intervention trials. The first trial took place between 2008 and 2010 and used the ADOS (Mandell et al., 2013). The second took place between 2015 and 2016 and used the ADOS-2 (Pellecchia et al., 2016). Inclusion criteria for students in these two trials included (1) being enrolled full-time in an autism-support classroom; and (2) being in kindergarten, first or second grade. The third Philadelphia study comprised an observational study (2014-2016) of preschool children with autism and used the ADOS-2 (Nahmias, 2017). Participants in this study were between the ages of 35 and 59 months and receiving community-based preschool early intervention. The San Diego study was a school-intervention trial that took place between 2012 and 2015 and used the ADOS-2 (Suhrheinrich, Rieth, Dickson, Roesch, & Stahmer, 2018). Participating students were aged 3-12 years with an educational classification of autism in preschool or elementary school.

Participants

Participants comprised 627 children (518 males [83%]; ages 2-12 years). All children had an educational classification of autism and an ADOS (n = 287) or ADOS-2 (n = 340) completed by a trained clinician or researcher. Most participants were served in special education/autism support classrooms and not inclusion settings. Most participants (n = 390) were from Philadelphia; 237 were from San Diego. See Table 1 for participant characteristics.

Table 1.

Demographic Information by Site

	Pennsylvania sample (n = 390)	California sample (n = 237)
	n (%)	n (%)
Male	315 (80.8%)	203 (85.7%)
Race/ethnicitya**
Black or African American	186 (47.7%)	12 (5.1%)
White	111 (28.5%)	121 (51.1%)
Hispanic or Latino	56 (14.4%)	80 (33.8%)
Asian	20 (5.1%)	18 (7.6%)
American Indian	0 (0.0%)	4 (1.7%)
Multiracial	18 (4.6%)	18 (7.6%)
Other	1 (.3%)	13 (5.5%)
	M (SD); range	M (SD); range
Age (years)**	6.47 (1.21); 2.92-9.50	5.70 (2.06); 3.00-12.25
IQb*	61.81 (20.22); 30-113	65.92 (19.57); 28-128

^aRace/ethnicity data were available for a subset of the total sample (n = 336 of Pennsylvania sample and 186 of California sample).

^bIQ measured by the Differential Ability Scales, 2^nd Edition (DAS-II) General Conceptual Ability Standard Score (n = 357 of Pennsylvania sample and 123 of California sample) or the Mullen Scales of Early Learning (Mullen) Early Learning Composite Standard Score (n = 33 of Pennsylvania sample and 114 of California sample).

^*p<.05

^**p<.001 for between-group comparisons.

Measures/Variables

The dependent variable was concordance between educational classification and ADOS/ADOS-2 classification. The following ADOS or ADOS-2 modules were used to characterize the sample: Module 1 (for children without phrase speech), Module 2 (for children with phrase speech who are not verbally fluent), and Module 3 (for children with fluent language). The ADOS or ADOS-2 was coded by a research reliable ADOS clinician or researcher. The ADOS/ADOS-2 and cognitive assessment (described below) took place at either the child’s school or home, in the quietest, least distracting space available.

Independent variables included study site (Pennsylvania vs. California), ADOS/ADOS-2 module administered, year of ADOS/ADOS-2 administration, child’s gender, child’s race/ethnicity, child’s age, and child’s cognitive ability. Cognitive ability was measured using the Differential Ability Scales, Second Edition (DAS-II; Elliott, 2007; both Early Years and School-Age batteries) or the Mullen Scales of Early Learning (Mullen, 1995), depending on the child’s developmental level. Because some of the sample completed the DAS-II and some completed the Mullen, we used the DAS-II General Conceptual Ability standard score and Mullen Early Learning Composite standard score to estimate cognitive ability. These two measures are highly correlated in children with neurodevelopmental disorders (Farmer, Golden, & Thurm, 2016).

Data Analyses

All analyses were conducted using SPSS version 24. First, we calculated means and standard deviations or percentages as appropriate for each variable of interest. Between-site comparisons were conducted using independent samples t-tests or chi-square tests. Second, we calculated the proportion of children with an educational classification of autism who exceeded the validated threshold for “autism spectrum” or “autism” on the respective ADOS/ADOS-2 modules (Lord et al., 2000; Lord et al., 2012). Based on (1) differences in state-specific special education eligibility and (2) differences between the Pennsylvania and California samples on demographic variables (Table 1), we conducted this analysis separately for the two sites. In addition, given that the earliest Pennsylvania study used the first edition of the ADOS, we separated this study from the later Pennsylvania studies, which used the ADOS-2. Pairwise follow-up tests were conducted for this analysis with the Holm’s sequential Bonferroni method to correct for multiple comparisons (Holm, 1979). Third, we estimated the associations between covariates of interest – study site, ADOS/ADOS-2 module administered, year of ADOS/ADOS-2 administration, child’s gender, child’s race/ethnicity, child’s age, child’s cognitive ability – and concordance between educational classification and ADOS/ADOS-2 classification using logistic regression. We first estimated bivariate associations between each covariate and concordance. Variables significant at p < .2 were included in the adjusted logistic regression model (Hosmer & Lemeshow, 2000). Given that ADOS edition and study year are interdependent variables, we included only study year in this analysis.

Results

Table 2 presents the proportion of participants who exceeded the ADOS/ADOS-2 threshold for “autism spectrum” or “autism.” The chi-square test comparing the three groups (Pennsylvania participants with ADOS data, Pennsylvania participants with ADOS-2 data, California participants with ADOS-2 data) on concordance between educational classification and ADOS/ADOS-2 classification was statistically significant, χ²(2) = 31.08, p < .001. Post-hoc pairwise comparisons using Fisher’s exact tests and Holm-Bonferroni adjusted alphas revealed that a greater percentage (97.5%) of California participants (all with ADOS-2) were concordant, compared to 92.2% of Pennsylvania participants with ADOS-2 data (p = .04) and to 82.9% of Pennsylvania participants with ADOS data (p < .001). The Pennsylvania participants with ADOS-2 data showed greater concordance than the Pennsylvania participants with ADOS data (p = .02).

Table 2.

ADOS/ADOS-2 Data by Site and ADOS Edition

	PA ADOS (n = 287)	PA ADOS-2 (n = 103)	CA ADOS-2 (n = 237)
	M (SD); range	M (SD); range	M (SD); range
ADOS/ADOS-2 Total Score**	11.99 (4.76); 0-22	15.91 (5.37); 1-25	16.97 (4.94); 2-27
	n (%)	n (%)	n (%)
ADOS/ADOS-2
Module
Module 1	127 (44.3%)	57 (55.3%)	122 (51.5%)
Module 2	108 (37.6%)	34 (33.0%)	79 (33.3%)
Module 3	52 (18.1%)	12 (11.7%)	36 (15.2%)
ADOS/ADOS-2
Classification**
Non-spectrum	49 (17.1%)	8 (7.8%)	6 (2.5%)
Autism spectrum	105 (36.6%)	9 (8.7%)	21 (8.9%)
Autism	133 (46.3%)	86 (83.5%)	210 (88.6%)

^**p<.001 for between-group comparisons, with the Pennsylvania (PA) ADOS group having a significantly lower ADOS total score and significantly lower concordance rate than the Pennsylvania ADOS-2 group or California (CA) ADOS-2 group. There was no significant difference between the two ADOS-2 groups on ADOS-2 total score. The Pennsylvania ADOS-2 group had a significantly lower concordance rate than the California ADOS-2 group. Pairwise comparisons were corrected with the Holm’s sequential Bonferroni method (Holm, 1979).

Table 3 presents the logistic regression results. Three variables were statistically significantly associated with concordance between educational classification and ADOS/ADOS-2 classification. Children from California were more likely than children from Pennsylvania to meet ADOS/ADOS-2 criteria (OR = 5.79, 95% CI: 2.31-14.48, p < .001). Children with higher IQ were less likely to meet criteria (OR = .98 per IQ point, 95% CI: .96-.99, p = .02). Children assessed more recently were more likely to meet criteria (OR = 1.15 per year, 95% CI: 1.01-1.30, p = .03).

Table 3.

Logistic Regression Results

Variables	β (SE)	Wald	Odds Ratio	95% CI
State**	1.76 (.47)	14.09	5.79	2.31-14.48
IQ*	−.02 (.01)	5.32	.98	.96-.99
ADOS module		2.54
Module 2	−.47 (.39)	1.45	.62	.29-1.35
Module 3	−.80 (.50)	2.52	.45	.17-1.21
ADOS year*	.14 (.06)	4.69	1.15	1.01-1.30

Note. β = Standardized regression coefficient value; SE = Standard error; CI = Confidence interval; State = Pennsylvania (coded as 0) or California (coded as 1); IQ = Differential Ability Scales, 2^nd Edition (DAS-II) General Conceptual Ability Standard Score or Mullen Scales of Early Learning (Mullen) Early Learning Composite Standard Score; ADOS Module 1 = reference group; ADOS year = year of ADOS administration.

^*p<.05

^**p<.001 for between-group comparisons.

Discussion

In our sample of 627 children receiving special education services for autism, 90% met criteria for “autism spectrum” or “autism” on the ADOS/ADOS-2. Although this high percentage is encouraging, it is important to note that concordance significantly differed between the two sites. Close to 98% of the children in California met ADOS-2 criteria, compared with only 92.2% of Pennsylvania participants with ADOS-2 data and 82.9% of Pennsylvania participants with ADOS data. This difference could be explained by inter-state differences in how schools determine eligibility for autism services. For example, under Pennsylvania state regulations, a school psychologist is required to participate in the evaluation for autism; this requirement is not included in California state regulations. This finding also could be explained by the fact that California has a more developed system of autism services; California is home to the earliest behavioral treatment studies (Lovaas, 1987) and an established network of regional centers that connect individuals with autism and their families to services. Because our study focused only on positive predictive value, and because there usually is a tradeoff between sensitivity and specificity, it is possible that California has a higher false negative rate than Pennsylvania.

Another variable associated with concordance was the child’s cognitive ability. Children with higher IQ scores were less likely to meet criteria on the ADOS/ADOS-2. Special education evaluation teams may be more likely to misclassify children with average or above average IQ who present with other conditions (e.g., ADHD and anxiety) as having autism. The symptoms of conditions such as ADHD and anxiety can mirror some of those of autism (APA, 2013), and it may take more sophisticated diagnostic practices than those currently used in education settings to differentiate them. In addition, previous research has found that among students with low ADOS scores (indicating fewer or less severe autism symptoms), parents rated their children as more socially impaired than teachers rated them (Azad, Reisinger, Xie, & Mandell, 2016). Perhaps parents of less cognitively and socially impaired children, including children who fall below the ADOS threshold, are still concerned about receiving services for their children’s behavioral difficulties and therefore strongly advocate for a special education classification.

The third factor that significantly affected concordance was the year of ADOS/ADOS-2 administration. Children assessed more recently were more likely to meet criteria. Our sample was pulled from four studies spanning from 2008-2016. We have three hypotheses to explain improved concordance over time. First, this finding may be related to moving from the ADOS to the ADOS-2 in 2012. The ADOS-2 boasts improved sensitivity and specificity (Lord et al., 2012), and we found that children assessed with the ADOS had a significantly lower concordance than the children assessed with the ADOS-2. Second, as autism services become more sophisticated, special education evaluation practices may have improved. A competing hypothesis relates to the fact that educational services for children with autism cost three times those for typically developing students (Buescher, Cidav, Knapp, & Mandell, 2014). As the number of children diagnosed with autism increases, school systems may become more stringent in their classification practices in an attempt to control the number of children in this category.

Based on recent Department of Education statistics, the number of United States students ages 3 to 21 with an educational classification of autism has increased more than three-fold from the 2004-2005 to 2014-2015 school year (Institute of Education Sciences, National Center for Education Statistics, 2016). This increase means that more children are receiving special education services for autism in their schools, which is good news for many families. School is often the most feasible treatment setting these children. The link between an educational classification and school-based services can raise difficult ethical issues for clinicians and researchers when results from a clinical evaluation do not match a child’s educational classification of autism. For example, if a community clinician determines that a child does not meet diagnostic criteria for autism, but this child receives school-based autism services that the family perceives as beneficial, the clinician and family grapple with issues related to disclosure about the clinical evaluation results and possible loss of services. This tension is one reason why our research team does not exclude children with low ADOS scores from our school-based autism studies; we do not want to interfere with their current services. It is important to note that the reverse situation also can happen and cause stress for families, when a child carries a community diagnosis of autism, but results from the school evaluation do not support special education services.

Several study limitations deserve mention. First, we relied solely on the ADOS/ADOS-2 for diagnostic categorization. Children with low ADOS/ADOS-2 scores still may meet diagnostic criteria for autism based on a more thorough assessment and expert clinician opinion (Lord et al., 2000; Lord et al., 2012). Second, the ADOS/ADOS-2 provides a snapshot of a child’s current behaviors; it is possible that children who do not meet ADOS/ADOS-2 criteria at one point in time would have meet criteria previously (when assessed for educational classification). The ADOS/ADOS-2 is not a perfect tool, and it was not designed to be a stand-alone autism assessment; however, it is often used this way by researchers as confirmation of diagnosis for study inclusion. These two limitations may have biased results by reducing the concordance rate. Third, we were unable to fully disentangle the variables of study year, ADOS edition (first or second), and study site because all of the ADOS data came from one Philadelphia study that spanned from 2008-2010, whereas the other three studies took place in 2012 or later and used the ADOS-2. Thus, we are unable to determine whether the improvement in concordance in more recent years is a secular or test effect; however, the improvement is encouraging, regardless of the cause. Lastly, the ADOS/ADOS-2 was administered in the child’s school or home, which may be less controlled than a clinical setting. For the school administrations, the child’s parent was not usually present, which is part of standard administration of Modules 1 and 2. The ADOS/ADOS-2 clinician relied on a teacher or paraprofessional familiar to the child when possible, but the lack of a parent was less than ideal.

Despite these limitations, this study has key implications for autism research and services. For epidemiological researchers, estimates of autism prevalence based on one state or even several states may not generalize to the nation. Differences among states may be due in large part to different evaluation practices, and may contribute to the dramatic discrepancy in autism prevalence rates between states, based on CDC data (e.g., 1 in 34 children in New Jersey vs. 1 in 76 children in Arkansas; Baio et al., 2018). The between-site differences found here do not support broad reliance on educational classification for research purposes. However, based on project data, concordance may vary by state. In California, researchers may place a higher degree of confidence in the educational classification of autism. In Pennsylvania, the concordance rate is not as rigorous, but it is encouraging that this rate is improving over time.

This study also has important implications for service systems. The results suggest that on the whole, the large majority of children with an educational autism classification meet research criteria based on the ADOS-2. However, a subset of children who do not meet ADOS/ADOS-2 criteria for autism may be receiving special education services under this category.