Sensitivity and Specificity of Proposed DSM-5 Criteria for Autism Spectrum Disorder in Toddlers

Abstract

Autism spectrum disorder (ASD) diagnosis is based on behavioral presentation; changes in conceptual models or defining behaviors may significantly impact diagnosis and uptake of ASD-specific interventions. The literature examining impact of DSM-5 criteria is equivocal. Toddlers may be especially vulnerable to the stringent requirements of impairment in all three social-communication symptoms and two restricted/repetitive symptoms. Receiver operating characteristic (ROC) curves identified optimal cutoffs for sums of ADOS and ADI-R criteria mapped to each criterion for 422 toddlers. The optimal modification of DSM-5 criteria(sensitivity=.93, specificity=.74) required meeting the ROC-determined cutoffs for 2/3Domain A criteria and 1 point for 1/4 Domain B criteria. This modification will help insure that ASD is identified accurately in young children, facilitating ASD-specific early intervention.

Autism Spectrum Disorders (ASD) are one of the most common neurodevelopmental disorders, with an estimated prevalence rate of 1 in 88 children (CDC, 2012). ASD diagnosis is made solely on the basis of behavioral presentation, which means that changes in conceptual models of the disorder – or in the delineation of defining behaviors – are especially significant for diagnosis. Access to intensive, ASD-specific intervention services is typically contingent upon a diagnosis of an ASD, so the stakes dependent on clear behavioral descriptions and accurate diagnosis are very high.

The diagnostic standards outlined in the Diagnostic and Statistical Manual, Fourth Edition, Text Revision (DSM-IV-TR; American Psychiatric Association [APA], 2000) have recently been revised with DSM-5, scheduled for release in 2013. Proposed diagnostic criteria for ASD have been circulated (APA, 2011) and have generated considerable controversy (Ghaziuddin, 2010; Tsai, 2012). The revised criteria reflect three significant changes from DSM-IV-TR. First, autism is conceptualized as a broad spectrum of disorders and the specific diagnostic sub-categories contained under the rubric of Pervasive Developmental Disorders (PDD) in DSM-IV-TR have been eliminated. While this change has been criticized by some (McPartland, Reichow, & Volkmar, 2012; Tsai, 2012), it reflects growing consensus that it is not possible to identify subcategories of ASDs reliably and that evidence for the validity of sub-categories is quite limited (Happe, 2011; Frazier et al., 2012; Lord et al., 2012).

Second, ASDs have long been conceptualized as consisting of a triad of behavioral features including impaired social interaction, limited functional and social communication, and restricted or repetitive behaviors (Wing, 1981). In DSM-5, impaired social interaction and limited social communication have been integratedinto one category. Restricted and repetitive behaviors are retained as the second category of symptoms required for diagnosis of an ASD. Several groups have investigated the validity of this shift. Both factor analytic studies (Frazier et al., 2012) and construct validation studies (Mandy, Charman, & Skuse, 2012) have yielded support for the two factor model, and describe it as a better fit than the three factor or triadic model.

The third change imposed by DSM-5 has proven the most controversial. That is the delineation of more stringent criteria for the diagnosis of an ASD. DSM-5 requires that individuals meet all three of the criteria in the category of social-communication impairments and two of four criteria in the category of restricted and repetitive behaviors to receive a diagnosis of an ASD. The change was implemented in an effort to increase the specificity of diagnosis and reduce the incidence of false positives. However, in combination with the elimination of categories such as Pervasive Developmental Disorder, Not Otherwise Specified (PDD-NOS), which had much less stringent diagnostic criteria, it may mean that a significant number of children currently diagnosed with ASD – and in need of intervention services –will no longer meet criteria for the diagnosis.

This possibility has been investigated in several recent studies. McPartland and colleagues (2012) reanalyzed data from 933 individuals evaluated during DSM-IV field trials to evaluate the sensitivity and specificity of the DSM-5 criteria. They report high specificity but more limited sensitivity for the DSM-5 criteria. Approximately 39% of their sample of individuals diagnosed with an ASD using DSM-IV-TR criteria did not meet DSM-5 criteria for the disorder. Individuals with higher cognitive functioning and with diagnoses other than Autistic Disorder (e.g. PDD-NOS and Asperger's Disorder) were most likely to be excluded by the new criteria. The McPartland and colleagues (2012) study has been criticized for its reliance on archival data, which may not have contained sufficient information to evaluate the criteria proposed for DSM-5 (Swedo et al., 2012), but the data nonetheless raise significant concerns.

Mandy and colleagues (2012) reported on a similar analysis of 708 high functioning individuals aged 2-21 years, 488 of whom were diagnosed with an ASD. Mandy and colleagues used data from the Developmental, Dimensional, and Diagnostic Interview (3DI; Skuse et al., 2004), a computerized parent report tool that has previously shown high reliability with the ADI-R and clinical diagnosis (Skuse et al., 2004). They found that the proposed DSM-5 criteria did not exclude individuals diagnosed with Asperger's Disorder or those with higher cognitive function. However, the new criteria did exclude a majority of their participants who had been previously diagnosed with PDD-NOS. The fact that individuals diagnosed with Asperger's Disorder tend to be older at diagnosis than children diagnosed with PDD-NOS may suggest that younger children, who may not present the full syndrome of behaviors characteristic of ASD are at greatest risk of being excluded by the new criteria.

Matson and colleagues have published a series of studies which have addressed this question directly and compared DSM-IV-TR and DSM-5 criteria for the diagnosis of ASD in toddlers, children and adults with developmental disabilities. Worley and Matson (2012) used the Autism Spectrum Disorder – Diagnosis for Children (ASD-DC; Matson & Gonzalez, 2007), an informant based measure, to assess symptoms of autism in 208 children aged 3-16 years. The authors then classified children according to DSM-IV-TR criteria and DSM-5 criteria. They report a decrease of 32% in the number of children diagnosed with ASD using DSM-5 as compared to DSM-IV-TR. They also looked at symptom severity on the ASD-DC and noted that children diagnosed by DSM-IV-TR criteria (but not DSM-5) and those identified by DSM-5 criteria exhibited very similar levels of impairment and were both significantly different from controls, suggesting that the children who were not identified by DSM-5 criteria exhibit significant levels of impairment.

In a study of 330 developmentally disabled adults, Matson, Belva, Horowitz, Koslowski and Baumburg (2012) found that the number of adults diagnosed with ASD by DSM-5 criteria declined by 36% from the number identified by DSM-IV-TR criteria. The adults who met criteria on the DSM-5 standard exhibited higher levels of impairment, in both socialization and restricted/repetitive behaviors than those identified by DSM-IV-TR criteria, although the groups received similar scores in the area of communication. Notably, both the DSM-5 identified group and the DSM-IV-TR identified group exhibited highly significant levels of impairment, suggesting that some adults with serious impairments were not identified by the DSM-5 criteria.

Matson, Kozlowski, Hattier, Horovitz and Sipes (2012) looked at the same comparison in 2,721 toddlers at risk for developmental disability who were referred for assessment through a statewide early intervention program. Evaluation data included the Modified Checklist for Autism in Toddlers (M-CHAT; Robins, Fein, & Barton, 1999), the Battelle Developmental Inventory (Newborg et al., 1988), and the Baby and Infant Screen for Children with Autism Traits (Matson, Boisjoli, & Wilkins, 2007). Using those data, experienced clinicians assigned diagnoses according to DSM-IV-TR criteria, and DSM-5 criteria. Clinicians were blind to previous diagnoses and the two diagnoses were made months apart. That process yielded three comparison groups: children who met diagnostic criteria for an ASD on DSM-5, children who met diagnostic criteria on DSM-IV-TR (but not on DSM-5), and children who did not meet criteria for a diagnosis of ASD. The results reveal a 47% decrease in the diagnosis of an ASD in the DSM-5 group as compared to the DSM-IV-TR group. More striking, although 24% of toddlers who met criteria for a diagnosis of Autistic Disorder on DSM-IV-TR failed to meet criteria for an ASD diagnosis on DSM-5, 88% of toddlers diagnosed with PDD-NOS according to DSM-IV-TR failed to meet criteria for an ASD diagnosis on DSM-5, suggesting that the DSM-5 criteria are, in fact, likely to exclude toddlers who met DSM-IV-TR criteria for PDD-NOS.

In a second study, which used the same sample of toddlers, Matson, Hattier, and Williams (2012) compared two sets of modified criteria for the diagnosis of ASD. Clinicians relied upon the same data but here assigned diagnoses based on DSM-IV-TR criteria, DSM-5 criteria, and two sets of modified criteria. The first modification (Modified 1) included all DSM-5 symptoms except that it required only two of three symptoms in the social interaction and communication domain. The second modification (Modified 2) required children to meet two of three symptoms in social interaction and communication AND only one of four in restricted and repetitive behaviors. When Modified 1 criteria were compared to DSM-IV-TR, there was a 33% decrease in ASD diagnoses, and when Modified 2 criteria were compared to DSM-IV-TR, there was a 17.8% decrease. The authors note that while the children identified by DSM-5 criteria exhibited the highest levels of impairment, children in all three of the comparison groups also exhibited significant impairment. They suggest that the DSM-5 criteria be relaxed to insure that most individuals with an ASD diagnosis will be correctly identified.

In a smaller study designed to examine the same question, Gibbs, Aldridge, Chandler, Witslsperger, and Smith (2012) used the Autism Diagnostic Observation Schedule (ADOS; Lord et al., 1999) and the Autism Diagnostic Interview, Revised (ADI-R; Rutter, LeCouteur, & Lord, 2003), the most widely accepted tools for the diagnosis of ASDs to assess 132 children aged 2-16 years. They compared diagnostic outcomes using the DSM-IV-TR and the proposed DSM-5 criteria and found that 26 of 111 children (23%) who received a DSM-IV-TR diagnosis did not meet criteria on DSM-5. The majority of children who did not meet criteria on DSM-5 had received a diagnosis of PDD-NOS on DSM-IV-TR, and 54% of these children met only one of four criteria for restricted and repetitive behavior. Gibbs and colleagues (2012) suggest that modifying the restricted and repetitive behavior (RRB) criteria to require only one of the fourRRB symptoms listed, or adding an additional RRB criterion to separate behaviors currently grouped under one category, might reduce the number of children who do not meet DSM-5 criteria. For example, Gibbs and colleagues posit that separating the repetitive use of objects from stereotyped language might permit the identification of more children.

These data are compelling, and support the notion that young children may be at greatest risk of being under-identified by the DSM-5 criteria. Nonetheless, these studies share some of the concerns raised with the McPartland and colleagues (2012) study, in that they are reliant upon an original data set that may not have contained sufficient information to make a valid diagnosis using DSM-5 criteria.

In summary, a small but growing literature suggests that the proposed DSM-5 criteria for the diagnosis of ASDs may result in significant decreases in sensitivity with the resultant under-identification of children with significant impairments who could benefit from intervention. This appears most likely to affect children with milder forms of the disorder, including those previously diagnosed with PDD-NOS, those with higher cognitive function, and those who are younger and may be less likely to exhibit the full range of symptoms. This is of concern for several reasons. A series of studies (Ben-Itzchak, Lahat, Burgin, & Zachor, 2008; Dawson et al., 2009; Rogers & Vismara, 2008) have now demonstrated that early identification and intensive early intervention are associated with more favorable outcomes for all children with ASDs. In addition, several recent studies have revealed that a small portion of children with ASDs may be able to progress sufficiently to lose their ASD diagnosis and attain an “optimal outcome”, and these children are likely to be those with relatively milder symptoms and higher cognitive functioning. (Fein et al., 2013; see Helt et al., 2008 for review). If the proposed criteria are adopted without revision, some of these children who appear to benefit greatly from early intervention, may be those least likely to qualify for services. While increased specificity is important for any diagnostic system, the benefit of increased specificity must be weighed against the cost of diminished sensitivity.

The largest and the most recent study to address this question was described by Huerta, Bishop, Duncan, Hus, and Lord (2012) and those authors report very different results. Huerta and colleagues reported on data from 4,453 children aged 2-17 with DSM-IV-TR diagnoses of PDD and 690 children with non-PDD diagnoses. They report that the new DSM-5 criteria identified 91% of the children with DSM-IV-TR diagnoses of PDD with specificity estimated at .53. In addition, they report adequate sensitivity for children diagnosed with PDD-NOS and Asperger's Disorder, and for girls and children with nonverbal IQs below 70.Specificity estimates for those groups were highly variable and often unacceptably low. The authors note that they considered that a child demonstrated a DSM-5 symptom if one or more of the ADOS or ADI-R items thought to measure any aspect of that symptom was coded as a 1 or higher. This is a lower threshold than is typically used since a score of 1 on these instruments indicates mild impairment, which is not definitively autistic either in intensity or severity, and which might not be consistent with symptoms required for diagnosis of an ASD. The authors also calculated sensitivity and specificity using one symptom reported by parents, one observed by clinicians, and a requirement that symptoms be both reported and observed; they noted that while sensitivity decreased slightly, specificity improved to .63 when both observation and parental report were required. Nonetheless, these authors relied upon a relatively liberal symptom threshold that left specificity estimates quite low. Finally, the authors note that a small portion of their sample (approximately 20%) were children aged four years old or younger. They do not report how many of those children were under the age of three, nor report how DSM-5 functioned for these young children specifically.

The current study is designed to address the question of estimated sensitivity and specificity of the proposed DSM-5 criteria in a sample of 422 toddlers aged 16-40 months. All of the children received comprehensive developmental evaluations including the ADOS and ADI-R as part of a larger study, and all received DSM-IV-TR diagnoses based on the judgment of experienced clinicians. The present study will compare those diagnoses to DSM-5 criteria in an attempt to further elucidate the impact of proposed changes on young children suspected of ASD.

Method

Participants

Toddlers in the current study were recruited as part of an ongoing multi-site screening study. There were multiple recruitment sources for this study: toddlers were recruited for the low-risk sample if they attended a pediatric well-child visit between 16 and 30 months old with a participating provider (n=153). More than 30 pediatric sites recruited participants in the metropolitan Atlanta region, and more than 50 pediatric sites recruited participants in Connecticut and portions of bordering states. The Connecticut site also recruited several high-risk groups. One high-risk sample included toddlers already referred to the statewide early intervention (EI) program for a variety of developmental concerns (n=164), but not yet diagnosed. A second high-risk sample consisted of toddlers recruited because they had an older brother or sister already diagnosed with an ASD (Sib; n=62). Regardless of recruitment source, children who screened positive were contacted to complete the M-CHAT(-R) Follow-up Interview, and those toddlers who continued to show risk for ASD were invited for a diagnostic evaluation. Finally, toddlers were offered the evaluation for other reasons after screening negative on the M-CHAT and/or Follow-up Interview, such as physician or parent indicated concerns about ASD or the child failed an observational screening tool (n=25).

The resulting sample includes 422 toddlers (Mean age=25.76 mos, SD=4.44, range 16.79-39.36 mos); please see Tables 1 and 2 for demographics on the sample. Participants were excluded from the current sample if ASD diagnostic measures (ADOS and ADI(-R)) were not administered.

Table 1. Sample Characteristics.

	UConn (%)	GSU (%)	Total (%)
Total Sample	332	90	422
Low-risk	74(22.3%)	79(87.8%)	153(36.2%)
High-risk: EI	164 (49.4%)	0 (0%)	164 (38.9%)
High-risk: Sib	62 (18.7%)	0 (0%)	62 (14.7%)
Other1	14(4.2%)	11 (12.2%)	25(5.9%)
Unknown	18 (5.4%)	0 (0%)	18 (4.3%)
ASD	234 (70.5%)	50 (55.6%)	284 (67.3%)
NonASD	98 (29.5)	40 (44.4%)	138 (32.7%)
Male	256(77.1%)	65(72.2%)	321(76.1%)
Female	76 (22.9%)	25 (27.8%)	101 (23.9%)
White/Caucasian	251 (75.6%)	44 (48.9%)	295 69.9%)
Black/African-America	n 19 (5.7%)	25 (27.8%)	44 (10.4%)
Hispanic/Latino	31 (9.3%)	4 (4.4%)	35 (8.3%)
Bi- or Multiracial	8 (2.4%)	0 (0%)	8 (1.9%)
Asian/Pacific Islander	9 (2.7%)	4 (4.4%)	13 (3.1%)
Other/Not reported	14 (4.2%)	13 (14.4%)	27 (6.4%)

Table 2. Additional Demographic Information.

	ASD	nonASD	Total
Mean age (mos)	25.67	25.94	25.76
SD	4.53	4.25	4.44
range	16.92-39.36	16.79-34.66	16.79-39.36
ADI Versions
Toddler '91	225	95	230
Toddler '04	23	28	51
ADI	12	12	24
ADI-R	23	3	26
ADI-R Short	1	0	1

Measures

Diagnostic measures used in the current study include the individual items from the Autism Diagnostic Observation Schedule (ADOS; Lord et al., 1999), Module 1 and the Autism Diagnostic Interview. Over the course of the study, five different versions of the ADI were used: the original ADI (LeCouteur et al., 1989; n=24), ADI-R (Rutter et al., 2003; Lord, Rutter, &LeCouteur, 1994; n=26), a short form of ADI-R under development at the time (n=1), and two versions of the Toddler form, (ADI-R Toddler '91, Rutter, Le Couteur, & Lord, 1991, n=320; ADI-R Toddler '04, LeCouteur, Rutter, Lord, & DiLavore, 2004, n=51). All versions of the ADI used the same algorithm items, and are referred to collectively as ADI-R in this manuscript.

Procedures

Final diagnosis of ASD or nonASD was determined by clinical judgment, which took into account data from the ADOS and ADI-R, as well as additional information about cognitive, motor, language, and adaptive functioning derived from standardized tests, a history form completed by the parents, and behavioral observations of the child throughout the session. Clinicians were licensed psychologists or developmental-behavioral pediatricians, and were assisted by graduate students in psychology and research assistants. Legal guardians of participants provided informed consent to participate in the study. Institutional review boards at the academic institutions provided oversight for the study, which was conducted in accordance with the ethical standards in the 1964 Declaration of Helsinki and later amendments. Item-level data was double-entered into a database using FileMaker Pro, which allowed for the creation of new algorithms to examine the proposed DSM-5 criteria.

Consensus was reached among the authors about which ADI-R and ADOS items mapped on to each of the proposed DSM-5 ASD criteria, and no item contributed to more than one DSM criterion. After this work was underway, the Huerta and colleagues (2012) paper became available, and comparison determined that the mappings differed, especially for the social-communication symptoms. Therefore, analyses were repeated using the ADI-R and ADOS item mappings from the Huerta et al. group with permission from the first author.

Table 3 shows the mapping of ADI-R and ADOS items onto proposed DSM-5 criteria, both for the current authors' judgment and for the items used in the Huerta and colleagues (2012) paper.

Table 3. ADI-R and ADOS Item Mapping on Proposed DSM-5 Criteria.

	Barton et al. mapping		Huerta et al. (2012) mapping
DSM-5	ADI-R	ADOS Mod 1	ADI-R	ADOS
A1: reciprocity	51. Social smile1	B2. Social smile	51. Social smile	B2. Social smile
	52. Showing/directing attention	B9. Showing	52. Showing/directing attention	B9. Showing
	54. Seeking to share enjoyment	B10. Initiating joint Attention	54. Seeking to share enjoyment	B10. Initiating joint attention
		B12. Quality of social Overtures	31. Use other' s body to communicate	B12. Quality of social overtures
	42. Pointing (interest)	B11. Responding to join Attention	t 46. Attend to voice	A2. Frequency of vocalization directed to others
	53. Offering to share		55. Offering comfort	A6.Use other' s body to communicate
			61. Imitative social play	B5. Shared enjoyment in interaction
				B6. Response to name
				B8. Giving objects
A2: nonverbal communication	45. Conventional /instrumental gestures	A7. Pointing	45. Conventional /instrumental gestures	A7. Pointing
	50. Direct gaze	A8. Gestures	50. Direct gaze	A8. Gestures
	56. Quality of social Overtures	B1. Unusual eye contact	56. Quality of social overtures	B1. Unusual eye contact
	57. Facial expressions to communicate	B3. Facial expression directed to others	57. Facial expressions to communicate	B3. Facial expression directed to others
	31. Use another's body to communicate	B4. Integration of gaze & other behaviors	42. Pointing (interest)	B4. Integration of gaze & other behaviors
		B7. Requesting	43. Nodding for “yes”	B7. Requesting
		A6.Use of other' s body to Communicate	44. Shaking head for “no”	B11. Response to joint attention
		B5. Shared enjoyment in Interaction
A3: relationships	62. Interest in children		62. Interest in children
	63. Response to approaches of peers		63. Response to approaches of peers
			53. Offering to share
			58. Inappropriate facial expressions
			59. Appropriateness of social responses
B1: stereotyped speech, motor, or object use	69. Repetitive use of objects or interest in parts of objects	A4. Immediate Echolalia	69. Repetitive use of objects or interest in parts of objects	A4. Immediate echolalia
	77. Hand and finger mannerisms	A5. Stereotyped/ idiosyncratic use of words or phrases	77. Hand and finger mannerisms	A5. Stereotyped/ idiosyncratic use of words or phrases
	78. Complex mannerisms or stereotyped body movements	D2. Hand, finger, and other complex mannerisms	78. Complex mannerisms or stereotyped body movements	D2. Hand, finger, and other complex mannerisms
	33. Stereotyped utterances and delayed echolalia	D4. Unusually repetitive interests or stereotyped Behaviors		D4. Unusually repetitive interests or stereotyped behaviors
B2: routines/rituals	70. Compulsions/rituals		70. Compulsions/rituals
	74. Difficulties with minor changes in own routines or personal environment		74. Difficulties with minor changes in own routines or personal environment
	75. Resistance to trivial changes in environment		75. Resistance to trivial changes in environment
	39. Verbal rituals
B3: restricted interests	67. Unusual preoccupations		67. Unusual Preoccupations
	76. Unusual attachment		76. Unusual attachment
			68. Circumscribed interests
B4: sensory	71. Unusual sensory interests	D1. Unusual sensory Interests	71. Unusual sensory interests	D1. Unusual sensory interests
	73. Sensory aversions		73. Sensory aversions
			72. Sensitivity to noise

¹Note: Items common to both mappings are listed first for each DSM Criterion.

Data analysis was conducted using SPSS 18.0, and consisted of computing sums of ADI-R and ADOS item scores contributing to each proposed DSM-5 criterion. Item scores ranged from 0-2, using standard convention of converting 3′s to 2′s. Next, sums were examined for cutoffs that maximized both sensitivity and specificity using Receiver Operating Characteristic (ROC) curves. Final sensitivity and specificity of groups of symptoms according to the calculated cutoff scores was determined by examining frequencies for ASD and nonASD cases that were classified above or below each threshold.

Results

ROC Curves were generated using the sum of the ADI-R and ADOS items contributing to each DSM-5 criterion; these analyses were repeated using the mappings from Huerta and colleagues (2012) for direct comparison (see Tables 4 and 5 for ROC output and Table 6 for psychometrics of selected cut off scores). For the criteria in domain A (Social-Communication), cutoffs were selected by identifying the highest score that maintained sensitivity at or above .9. This high threshold for sensitivity was selected with the goal of minimizing cases that would no longer meet the new ASD criteria. A second threshold of 1 was considered for each domain, in order to compare results to the findings from Huerta and colleagues (2012), in which a score of ‘1’ on any item constituted an endorsement of that symptom. For domain B (Restricted and Repetitive Behaviors), fewer items were contributing to each criterion, thereby limiting the ability to select a cutoff with such high sensitivity. For criteria B1 (repetitiveness) and B4 (sensory), analyses were run examining cutoffs of 1 and cutoffs of 2; however, for criteria B2 (routines/rituals) and B3 (restricted interests) only a cutoff of 1 was considered, given the low frequency of endorsement of these items.

Table 4.

ROC Analyses to Examine Optimal Thresholds for the Social-Communication Criteria.

	A1 (10 ADI-R, ADOS)		A2 (12 ADI-R, ADOS)		A3 (3 ADI-R)
Cutoff	Sensitivity	Specificity	Sensitivity	Specificity	Sensitivity	Specificity
-1	1	0	1	0	1	0
0.5	1	0.029	1	0.014	0.951	0.348
1.5	1	0.087	1	0.123	0.873	0.558
2.5	0.996	0.196	0.996	0.225	0.725	0.746
3.5	0.993	0.304	0.986	0.377	0.577	0.891
4.5	0.989	0.42	0.979	0.507	0.363	0.971
5.5	0.982	0.5	0.958	0.609	0.092	0.978
6.5	0.965	0.601	0.944	0.652	0	1
7.5	0.947	0.674	0.908	0.761
8.5	0.919	0.783	0.88	0.804
9.5	0.884	0.862	0.852	0.891
10.5	0.835	0.899	0.782	0.92
11.5	0.764	0.942	0.711	0.949
12.5	0.68	0.957	0.637	0.971
13.5	0.623	0.957	0.56	0.986
14.5	0.539	0.964	0.511	0.993
15.5	0.426	0.978	0.384	0.993
16.5	0.313	0.978	0.296	0.993
17.5	0.229	0.986	0.246	0.993
18.5	0.144	1	0.162	1
19.5	0.046	1	0.106	1
20.5	0	1	0.056	1
21.5			0.028	1
22.5			0.011	1
23.5			0	1

Note. Shading indicates the optimal cutoff balancing sens and spec.

Table 5.

ROC Analyses to Examine Optimal Thresholds for the Restricted/Repetitive Criteria.

	B1 (8 ADI-R, ADOS)		B2 (4 ADI-R)		B3 (2 ADI-R)		B4 (3 ADI-R, ADOS)
Cutoff	Sensitivity	Specificity	Sensitivity	Specificity	Sensitivity	Specificity	Sensitivity	Specificity
-1	1	0	1	0	1	0	1	0
0.5	0.937	0.203	0.324	0.725	0.447	0.725	0.813	0.413
1.5	0.831	0.5	0.201	0.87	0.246	0.92	0.556	0.754
2.5	0.627	0.659	0.099	0.935	0.085	0.993	0.31	0.891
3.5	0.458	0.812	0.063	0.949	0.032	0.993	0.127	0.957
4.5	0.324	0.913	0.018	0.993	0	1	0.06	0.993
5.5	0.218	0.935	0	1			0.018	1
6.5	0.137	0.942					0	1
7.5	0.077	0.978
8.5	0.053	0.993
9.5	0.025	1
10.5	0.014	1
11.5	0.004	1
12.5	0	1

Note. Shading indicates the optimal cutoff balancing sens and spec.

Table 6.

Cutoffs Used for Each Criterion.

	Barton et al.			Huerta et al. (2012)
DSM-5 Criterion	Cutoff Score	Sens	Spec	Cutoff Score	Sens	Spec
A1. Reciprocity	9	.92	.78	12	.91	.83
	1	1.0	.03	1	1.0	.01
A2. Nonverbal Comm.	8	.91	.76	12	.90	.80
	1	1.0	.01	1	1.0	.02
A3. Relationships	1	.95	.35	3	.91	.46
				1	.99	.12
B1. Stereotypies	2	.83	.50	2	.83	.51
	1	.94	.20	1	.94	.22
B2. Routines/Rituals	1	.32	.73	1	.30	.73
B3. Restricted Interests	1	.45	.73	1	.45	.73
B4. Sensory	2	.56	.75	2	.58	.71
	1	.81	.41	1	.84	.36

After each child was classified as either meeting each cutoff threshold or not, participants were further classified based on the number of symptoms met in Domain A and in Domain B. For Domain A, two cutoffs were considered: meeting all three symptoms, as is proposed for DSM-5, and also a more “relaxed” threshold of meeting two out of the three Domain A symptoms. Similarly, for Domain B, two cutoffs were considered: meeting two out of the four Domain B symptoms, as is proposed for DSM-5, and also a more “relaxed” threshold of meeting only one Domain B symptom. Complete classification required that toddlers met thresholds for both Domain A and Domain B in order to be classified with ASD. Six possible solutions were considered, in an attempt to find possible solutions that maximized sensitivity without compromising specificity. See Table 7 for the psychometric properties of each of the possible solutions. Each solution has two defining factors: (a) the cutoff score within the sum of ADI-R and ADOS items that contributed to that criterion, (b) the number of criteria required in a domain. These values are presented separately for Domain A symptoms and Domain B symptoms.

Table 7.

Psychometric Properties of Six Possible Solutions.

		Mapping	Sens	Spec	PPV	NPV
Solution I		Barton:	.84	.55	.79	.63
		Huerta1:	.86	.41	.75	.59
A: Threshold: 12	A: 3/3 criteria
B: Threshold: 1	B: 2/4 criteria
Solution II		Barton:	.77	.94	.96	.66
		Huerta:	.72	.94	.96	.62
A: Threshold: ROC3	A: 3/3 criteria
B: Threshold: 1	B: 2/4 criteria
Solution III		Barton:	.85	.82	.91	.72
		Huerta:	.85	.83	.91	.72
A: Threshold: ROC	A: 2/3 criteria
B: Threshold: 1	B: 2/4 criteria
Solution IV		Barton:	.81	.91	.95	.70
		Huerta:	.76	.91	.95	.65
A: Threshold: ROC	A: 3/3 criteria
B: Threshold: ROC	B: 1/4 criteria
Solution V		Barton:	.89	.77	.89	.77
		Huerta:	.89	.79	.90	.77
A: Threshold: ROC	A: 2/3 criteria
B: Threshold: ROC	B: 1/4 criteria
Solution VI		Barton:	.93	.74	.88	.84
		Huerta:	.93	.78	.89	.84
A: Threshold: ROC	A: 2/3 criteria
B: Threshold: 1	B: 1/4 criteria

¹Huerta et al. (2012) mappings were contrasted with those determined by the current authors, in order to evaluate two sets of expert mappings of ADI-R and ADOS symptoms.

²Threshold of 1 point on any ADI-R or ADOS item that maps on to this symptom is sufficient for endorsement; Per communication with Cathy Lord.

³ROC-defined cutoffs were determined by examining the distribution of the sum of ADI-R and ADOS items that contributed to each DSM-5 criterion, and identifying the cut score that balanced sensitivity and specificity (Domain A: highest score that maintained a sensitivity of .90; Domain B: lowest score that maintained specificity of .50 or higher; see Tables 4-5).

First, the solution proposed by Huerta and colleagues (2012) was examined. Using a very liberal threshold of 1 point for each of the three A symptoms, and for any two of the four B symptoms led to high sensitivity, but unacceptably low specificity.Although a majority of the ASD cases were correctly classified, about half of the nonASD cases were classified with ASD as well. This is likely because a threshold of 1 meant that only one item across numerous ADI-R and ADOS items was scored in the “mild” or “possible” symptom range, which may not equate to a clinically significant deficit in that symptom.

Solutions II-VI considered different combinations of both symptom-level thresholds (i.e., meeting either the ROC-defined cutoff score, or meeting the cutoff score of 1), and domain-level thresholds (i.e., meeting the number of symptoms specified by the proposed DSM-5 criteria, or relaxing that by one item to capture the heterogeneity of ASD presentation in toddlers). All five of these solutions demonstrated adequate sensitivity and specificity, according to the expectation that these values should exceed .70 (see American Academy of Pediatrics et al., 2006); however, if one considers that a sensitivity of .70 means that nearly a third of cases that were classified with ASD under DSM-IV-TR criteria will no longer be classified with ASD under DSM-5, it seems reasonable to maximize sensitivity without decreasing specificity below .70. Therefore, Solution VI appears to be the best solution. It requires toddlers to exceed the ROC-determined cutoff score for two out of three Domain A items, and to exceed a score of 1 for any one of the Domain B items.

Discussion

The present study sought to replicate and extend the findings of previous studies regarding the sensitivity and specificity of the proposed DSM-5 criteria for diagnosis of ASD in young children. Initial data analysis revealed results similar to those of Huerta and colleagues (2012) using a liberal symptom level threshold of one point on any ADOS or ADI-R item that contributed to each DSM criterion, for each of the three symptoms in the domain of social communication and a similar threshold of one point for two of four symptoms in the domain of restricted interests and repetitive behaviors. Such a threshold resulted in relatively high sensitivity, but unacceptably low specificity. While more children with ASD were identified using this method than had been reported in earlier studies which used different thresholds, sensitivity estimates in this sample are not as high as those reported by Huerta and colleagues (2012). This may reflect the fact that the children in our study are all toddlers. Symptom presentation in young children is often less clear as symptoms may still be emerging when children are referred for early evaluation. Toddlers' behavior may also be more affected by situational variables, and their parents may have less experience than the parents of older children with developmental processes and age related expectations. In addition, about half of the children who did not have ASD were also identified using these cut-offs, suggesting that a mild endorsement of one aspect of a DSM-5 criterion may not meet the threshold for clinically significant impairment characteristic of autism.

Subsequent analyses explored changes to both the symptom-level thresholds (i.e., meeting the cutoff score of 1 as defined by Huerta and colleagues (2012) or meeting the ROC-defined cutoff score), and domain-level thresholds (i.e., meeting the number of symptoms specified by the proposed DSM-5 criteria, or relaxing that by one item). ROC-defined cutoffs were determined by examining the distribution of the sum of ADI-R and ADOS items that contributed to each DSM-5 criterion, and identifying the cut score that balanced sensitivity and specificity. Specifically,for the social symptoms, ROC-based cutoffs were chosen based on the highest score that maintained a sensitivity of .90. For restricted, repetitive behaviors, cutoffs of 1 and 2 were considered, unless the cutoff of 2 had sensitivity below .25, in which case only the cutoff of 1 was considered for that criterion. It is notable that for the two criteria in Domain B that had very low sensitivity, there were no ADOS items that mapped on to those criteria, for either the current authors' mapping or the Huerta and colleagues (2012) mapping. This may be specific to Module 1 of the ADOS, indicating that some Domain B symptoms might be difficult to identify in toddlers. These data reveal that a combination of modifications to both the symptom level thresholds and the domain level cutoffs may provide the highest level of sensitivity and specificity for young children. Specifically, increasing the symptom level threshold to that defined by the ROC analyses, and relaxing the domain level threshold to 2 of 3 symptoms in the social-communication domain, while retaining Huerta and colleagues' (2012) one point symptom-level threshold and relaxing the domain level threshold to 1 of 4 symptoms for the domain of restricted interests and repetitive behaviors resulted in the highest levels of sensitivity and adequate levels of specificity in this sample.

There are several reasons why the results from this sample may differ from those reported by Huerta and colleagues (2012). First, like the Matson, Hattier, and Williams (2012) study, and the Matson, Kozlowski et al. (2012) study, the current sample focused on children below the age of three, with a mean age of 25 months. Although Huerta and colleagues (2012) included a sample of children under the age of four in their large study,they do not report the mean age of that group or how many children in their sample were below the age of three. Some authors (e.g., Bishop, Richler, & Lord, 2006; Lord, 1995; Moore & Goodson, 2003; Stone et al.,1999; Wiggins, Robins, Adamson, Bakeman, & Henrich, 2012)have suggested that repetitive behaviors may emerge in some children in the fourth year of life and that few children exhibit restricted interests in the toddler years. Some proportion of children who receive a diagnosis of an autism spectrum disorder appear not to present the full range of symptoms before the age of four and therefore would not be identified using the new thresholds. In addition, autism is clearly a highly heterogeneous disorder. It may well be the case that heterogeneity is even more pronounced in the youngest children. Relaxing the symptom level threshold in the domain of restricted and repetitive behaviors from two of four to one of four has been suggested by other authors (Gibbs et al., 2012; Matson et al., 2012) and is supported by the data presented here. That modification appears to be especially important for children aged three years and younger.

In addition, many of children in the current sample were recruited from primary care settings as opposed to the specialized clinics and research projects from which the children in the Huerta and colleagues (2012) study were drawn. As those authors note, it may be that their participants included children with more complex presentations or children at greater biologic risk, than children recruited from the general population. The participants in the current study may present fewer or less severe symptoms which result in less diagnostic clarity.At the same time, they represent children from a broader cross section of the population, including some with limited access to diagnostic services and those most likely to be most affected by changes in the diagnostic threshold.

Finally, there is poor agreement among experts about which ADI-R and ADOS symptoms map onto which DSM-5 criteria or how those criteria are best defined. Our team, which has a great deal of collective research and clinical experience with autism in toddlers, did not find it straightforward to distinguish among DSM-5 symptoms or to map specific behavioral criteria onto these symptoms. We anticipate that other clinicians will find it similarly difficult and may adopt idiosyncratic understanding of how the symptoms map onto behavior in toddlers.This is especially true for symptoms in the domain of social communication, where there is some overlap between criterion A 1 and criterion A 3 (Huerta et al., 2012) and where some of the behavioral descriptors seemunclear.

In addition some of the behaviors listed in criteria A 3 (relationships) are rarely observed in very young children, especially those with developmental delays, including interest in and interaction with peers. It is notable that both mappings of ADOS and ADI-R items on to DSM-5 criteria (see Table 3) have no ADOS items that contribute to this third social-communication symptom from the Module 1 ADOS. As a result, toddlers may fail to meet this criterion unless the threshold of meeting all three social-communication criteria is modified. Adding more specific language and better operationalized descriptions of these criteria will enable clinicians who see a broad range of children to make diagnostic decisions with greater confidence and accuracy.

Similarly, some of the behaviors described in the domain of restricted interests and repetitive behaviors are difficult to observe in young children and may emerge slightly later in development. In support of this, only two of the four Domain B criteria have any ADOS Module 1 items that map onto them, indicating that some of these criteria may be difficult to observe in very young children. Gibbs and colleagues (2012) also suggest reducing the domain threshold from two to one symptom, and point out that several discrete behaviors are listed under one RRB criterion. Separating those behaviors into distinct criteria will help clarify diagnostic decisions and might permit additional children to meet diagnostic thresholds.Until we have reliable biomarkers of ASD, we must strive for as much precision as possible in our descriptions of behaviors relevant to diagnostic criteria. At the same time, we must make diagnostic criteria sensitive to developmental processes so that they capture the unique behavioral characteristics of children with ASD at all developmental levels.

Given the importance of early identification of young children with ASDs, sensitivity must be regarded as the criteria of greatest importance in designing diagnostic standards for young children.Increased sensitivity and adequate specificity in this sample was associated with reductions in the domain level threshold in both symptom domains and with increasing the symptom level threshold in the domain of social communication. Such modifications to the proposed criteria seem critical to protecting access to services for the youngest children.

There are several limitations to this study. While the sample includes a broad range of participants, a subset of children was recruited from early intervention sites and cannot be considered representative of the general population. Those children were likely at greater risk of receiving an ASD diagnosis and their presence may have inflated estimates of sensitivity.More significant, as with the other recent papers comparing DSM-IV-TR to DSM-5 classification, the present study is not a field study of the new diagnostic criteria and is therefore reliant upon data collected during diagnostic evaluations using DSM-IV-TR criteria. It is possible that clinicians did not ask for information relevant to the proposed criteria and that assessments tailored to the new criteria may result in different estimates of sensitivity and specificity. Replication of this study using historical data and field trials of the new criteria that focus specifically on the diagnosis of ASD in toddlers will be critical to ascertaining the impact of the revised criteria on the youngest children. In spite of these limitations, this study provides compelling evidence that the proposed DSM-5 criteria may be too stringent for children younger than three years old. Modifying the diagnostic criteria to reflect the variety of developmental patterns observed in early childhood will help insure that young children with ASDs will be identified and referred to intervention as early as possible.