Accurate Autism Screening at the 18-Month Well-Child Visit Requires Different Strategies Than at 24 Months

Abstract

Accuracy of autism screening using M-CHAT plus the Follow-Up Interview (M-CHAT/F) for children screened positive at 18-months was compared to screening at 24-months. Formal ASD testing was criterion for a community sample of M-CHAT positive children (n=98), Positive Predictive Value (PPV) was 0.40 for the M-CHAT and 0.58 for the M-CHAT/F. MCHAT/F PPV was 0.69 among children 20+ months compared to 0.36 for <20 months. Multivariate analyses incorporating data from the Ages and Stages Questionnaire, MacArthur-Bates Communicative Development Inventory, M-CHAT and M-CHAT/F results, and M-CHAT items suggest language variables carry greatest relative importance in contributing to an age-based algorithm with potential to improve PPV for toddlers <20 months to the same level as observed in older toddlers.

Current prevalence estimates suggest that autism spectrum disorder (ASD) now affects 1 in 68 children (Baio, 2014). Early evidence-based intervention for children with ASD is associated with improved long-term outcomes, making early detection vital (National Research Council, 2001; Howlin, Magiati, & Charman, 2009). The American Academy of Pediatrics (AAP) has issued guidelines for ASD screening at 18 and 24 months (American Academy of Pediatrics, 2006). Although earlier detection is improving, average age at first diagnosis remains over age three years (Fountain, King, Bearman, 2011). The U. S. Preventive Services Task Force concluded that the evidence for efficacy of screening in community settings is insufficient (Siu & U. S. Preventive Services Task Force, 2016), and specifically noted that “there is insufficient evidence to determine if certain risk factors modify the performance characteristics of ASD screening tests, such as the age at which screening is performed” (Siu & U. S. Preventive Services Task Force, 2016). However, the AAP continues its recommendation due to the prevalence and the importance of early detection. Therefore, much work remains to be done to improve early screening tools and practices to assure the adequacy at all recommended ages.

The Modified Checklist for Autism in Toddlers (M-CHAT) is an ASD screening instrument widely used in primary care (Robins, Fein, Barton & Green, 2001). The M-CHAT is among the tools recommended for ASD screening by the AAP (American Academy of Pediatrics, 2006), and major advocacy groups, such as Autism Speaks (Autism Speaks, 2016). The M-CHAT has replaced the original CHAT (Baron-Cohen, Allen & Giliberg, 1992) which was shown to have excellent specificity and positive predictive value (PPV) for early identification of autistic children but overlooked too many affected children (Baird et al., 2000). While studies of the M-CHAT suggested that the modifications of the test corrected the problem with sensitivity, these studies were conducted in samples heavily weighted with children already enrolled in clinical programs rather than only children presenting for screening in community primary care practices, which would tend to inflate estimates of sensitivity. While more recent M-CHAT studies did extend to large community pediatric samples (Chlebowski, Robins, Barton & Fein, 2013; Robins, Casagrande, Barton, Chen, Dumont-Mathieu & Fein, 2014) clarifying estimates of predictive validity that can be expected in primary care screening, accurate assessment of sensitivity has been limited for all autism screening tests, including the M-CHAT, because costs and feasibility issues preclude evaluating community samples of children passing screening tests. However, when a large community sample of children who had M-CHAT screening at 18 months was followed with additional questionnaire screens at 3, 5, and 7 years with evaluations occurring 2 ½ to 8 years later, 2/3’s of the children confirmed with ASD in the community had not been identified (Stenberg et al., 2014) indicating limited sensitivity. Another study of 18-month M-CHAT screening followed up at age three years showed a higher sensitivity (0.73) when the pass/fail cut point was decreased over the standard one but also causing a referral rate so high (17%) as to be of questionable feasibility (Kamio, Inada, Koyama, Inokuchi, Tsuchiya & Kuroda, 2014).

Since it is known that about a third of autism cases present with developmental regression (Barger, Campbell & McDonough, 2013) and these children may actually appear typically behaving at the 18-month visit, we can expect that a symptom report screening test may be less sensitive at that age than at two years, justifying recommendation of a second screen by the American Academy of Pediatrics. However, since earlier intervention has benefits, screening at the younger age is warranted if screening tools have an adequate proportion of positive screens confirmed by diagnostic testing, i.e., the positive predictive value of the test.

There is reason to believe that the M-CHAT may be less accurate when administered at 18 months than at age 2. M-CHAT authors found lower PPV of 0.28 in younger toddlers from a community sample compared to older toddlers (0.61) even when the recommended follow-up interview (M-CHAT/F) (Pandey et al., 2008) was used. A more recent study in a community sample suggested that younger toddlers may have a lower PPV, though that difference was not statistically significant (Robins, Casagrande, Barton, Chen, Dumont-Mathieu, Fein, 2014). However, it should be noted that in this study the required follow-up interview was completed an average of two months after the 18-month visit when more symptoms may have emerged or parent observation and concern may have been sharpened.

In the initial validation study of the M-CHAT, parents reviewed items they endorsed as positive with a research assistant over the telephone to determine if the parents would confirm their responses when probed with additional questions. Additional studies have used a formalized structured follow-up interview to improve PPV from 0.36 to 0.72 overall and from 0.11 to 0.68 in a community primary care sample (Kleinman et al., 2008). This follow-up interview (known now as M-CHAT/F) has been designated as a “highly recommended” critical part of a two-stage identification process (Chlebowski et al., 2013; Robins et al., 2014). Recently, it has been shown that the follow-up interview can be feasibly incorporated into routine check-up visits and is reliable when done with electronic decision support by Primary Care Providers (PCPs) (Sturner et al., 2016). Because use of a screen with low PPV increases unnecessary worries and expense for families with false positive results, M-CHAT screening without the follow-up interview for positives is presently not recommended in low risk populations such as routine primary care pediatrics (Robins et al., 2014).

With these weaknesses in screening at the more valuable early age, it is fortunate that progress in developing screening tools is being made. There is evidence that some children can be identified as early as one year of age via a parent questionnaire surveying autism symptoms (Reznick, Baranek, Reavis, Watson & Crais, 2007) although numbers of children being overlooked, PPV, and clinical utility are not yet defined. A standard language measure (MacArthur-Bates Communicative Development Inventory (CDI)) has shown promise to further enhance those predictions among one-year-olds (Veness, Prior, Veness, Eadie, Cini & Reilly, 2012), although this lengthy measure is not being advocated for use as an ASD screen. Further exploration of the value of standard language measures for enhancing prediction of autism outcomes in young children seems warranted. Recently, a widely used general developmental screen, the Ages & Stages Questionnaires- 3rd Ed. (ASQ) (Squires & Bricker, 2009) has been shown to be a possibly clinically valuable pre-screen for ASD with M-CHAT used as the second stage (Hardy, Haisley, Manning & Fein, 2015).

The objective of this study was to compare the accuracy of autism screening between younger and older toddlers using the results of the M-CHAT completed by parents and the follow-up interview completed by PCPs and to explore strategies to improve screening accuracy. We studied the M-CHAT along with the ASQ completed by parents online prior to routine check-up visits at 18 and 24 months in conjunction with automated presentation of M-CHAT/F items to the PCP and instant rescoring at the time of the visit. We then conducted multivariate analyses of screening data to explore the potential for enhancing the PPV of an autism screening method by incorporating individual M-CHAT item responses with elements of a broad band developmental screen and a standard language measure that was administered to parents of children screening positive on the M-CHAT who were either M-CHAT/F positive or negative.

Methods

Screening

PCPs screened children before routine 18- and 24-month visits with the M-CHAT and ASQ as per AAP recommendations. Both tools were completed by parents either at home or on a computer or touch pad in the waiting room via the online system called CHADIS (CHADIS.com, 2016). Data collection occurred between September, 2009 and February, 2013. Forty-seven self-selected pediatricians from 22 offices used the M-CHAT/F. These Maryland practices were mostly suburban (73%), with some rural (18%) and urban locations (9%). Practice estimation of office-level demographics showed that 31% of children were Medicaid insured (range 5–65 %) with 39% white; 33% African American; 16% Asian; and 8% Hispanic. Of the total of 5,071 children screened with the M-CHAT, 341 (6.7%) were positive. The follow-up interview was completed by the PCP with parents of children with a positive M-CHAT (see Figure 1 for sample flow). The M-CHAT was automatically scored, and for each positive screen, the PCP was prompted by the online system with the specific questions to follow up on failed items needed to complete the M-CHAT follow-up interview. The online system continually rescored the M-CHAT/F as items were asked and a positive screen was either confirmed or refuted by the scoring logic. The M-CHAT/F, administered online by a PCP has been shown to be as accurate as when administered by specially trained autism center staff (Sturner, et. al, 2016).

Figure 1.

Study Sample Flow

Diagnostic Testing and Assessment

All children with a positive M-CHAT, regardless of subsequent M-CHAT/F determination, were recruited for diagnostic testing at Kennedy Krieger Institute (KKI) Center for Autism and Related Disorders. Consent for participation was obtained using a protocol approved by The Johns Hopkins University School of Medicine Institutional Review Board (IRB). The IRB approved the initial phase of screening (M-CHAT) as a standard component of care requiring no study consent. There was no charge for the evaluation and the parents received a subject participation stipend. Parents were asked to complete a mailed copy of the age appropriate MacArthur-Bates Communicative Development Inventory (CDI) (Fenson et al., 2007) prior to the KKI evaluation. If the CDI was not completed by the time of the evaluation, the parent was asked to do so during the evaluation session. In addition, the M-CHAT/F interview was repeated by a KKI staff member over the phone at the time the appointment was made. Diagnostic testing was completed using the Autism Diagnostic Observation Schedule, 2nd Ed. (ADOS-2) (Luyster et al., 2009) administered by certified research reliable speech and language pathologists. The same evaluator administered the Mullen Scales of Early Learning (MSEL) (Mullen, 1995). KKI evaluators were aware of the study design with knowledge of M-CHAT positive results but were blind to PCP M-CHAT/F interview results and other test results. A total of 113 parents (18% of M-CHAT positive children) consented for this phase of the study and 98 (16% of M-CHAT positives) completed all parent-report measures.

Measures

Screening Instruments

The Modified Checklist for Autism in Toddlers (Robins et al., 2001) with Follow-Up Interview (Kleinman et al., 2008) (M-CHAT/F). The 23-item M-CHAT is a parent-report checklist with items presented in a yes/no format. The M-CHAT was derived from the CHecklist for Autism in Toddlers (CHAT) (Baron-Cohen et al., 1992). The original validation sample included 1,293 children 18–30 months old with 58 children receiving diagnostic evaluation and 38 diagnosed with ASD. In this initial study only cases where the M-CHAT result was confirmed by a “phone interview” were given a diagnostic evaluation. Only 7 of these were “low risk” by being selected from primary care pediatric settings with the remaining being selected from those already in an early intervention program (Robins et al., 2001). A discriminant function analysis of that sample assumed that non-risk children did not have autism and found that children could be best classified into the autistic or non-autistic groups based on a score of any 2 of 6 items considered ‘critical’, or any 3 items overall. M-CHAT together with a confirmatory phone interview yielded a sensitivity of 0.87 and specificity of 0.99, with a positive predictive value of 0.80. The critical items included those concerning joint attention (proto-declarative pointing, bringing to show, following a point), interest in other children, responding to name, and imitation.

The addition of a formalized post-questionnaire telephone interview (M-CHAT/F) increased sensitivity and specificity to 0.97 and 0.99 (Kleinman et al., 2008). The follow-up interview is described in a 26-page manual that was used to create the online version used in this study. A recently revised scoring (M-CHAT-R) follows an analysis in a larger sample (16,071) that eliminates three of the items, changed the item order, simplified the language of some of the items and requires any 3 or more failed items for an overall fail. The study reported here was completed prior to this revision of the M-CHAT and therefore used the original scoring.

Ages & Stages Questionnaires 3rd Edition (ASQ) (Squires et al., 2009): The ASQ is a parent-report screen for developmental problems among infants and toddlers, including 21 separate questionnaires at two-month age intervals between 2 and 60 months. Each questionnaire includes 6 questions with a “yes, sometimes, and not yet” response set in each of 5 domains: Communication, Gross Motor, Fine Motor, Problem Solving, and Personal-Social. Psychometrics, based on data from over 18,000 respondents, yielded agreement of 86% between ASQ and other measures of development (range: 83–88%). The ASQ is among those endorsed for general developmental screening by the American Academy of Pediatrics (American Academy of Pediatrics, 1994).

The MacArthur-Bates Communicative Development Inventory (CDI) (Fenson et al., 2007): The CDIs are parent-report measures designed to evaluate the communicative skills of young, typically developing children from their “early signs of comprehension, to their first nonverbal gestural signals, to the expansion of early vocabulary and the beginnings of grammar.” The CDI: Words and Gestures (Infant form) is designed for use with 8- to 16-month old children. The CDI: Words and Sentences (Toddler form) is designed for use with 16- to 30-month old children. Both were normed on a large population of monolingual English-speaking children, representing a socioeconomic status that was higher than the national average based on 1990 U.S. census information.

Diagnostic Assessments

Autism Diagnostic Observation Schedule, 2nd Ed. (ADOS-2) (Luyster et al., 2009): The ADOS-2, a semi-structured behavior observation assessment of social and communication skills, is the recognized standard test for diagnosis of ASD across age, developmental level, and language skills. The assessment relies on a series of ‘planned social occasions’ designed to elicit specific social and communicative situations in a standardized way. The ADOS includes 4 separate modules. Each module takes approximately 40–60 minutes to administer. Modules 1 through 4 provide cutoff scores for autism and autism spectrum classifications. Modules 1 through 3 also provide a Comparison Score indicating level of autism spectrum-related symptoms compared to children with ASD who are the same age and have similar language skills. Data collection for this study was completed before availability of the Autism Diagnostic Observation Schedule, Second Edition, Toddler Module and before the Diagnostic and Statistical Manual of Mental Disorders, Fifth Edition, redefinition of autism and social communication disorder. The dependent variable was the KKI clinical impression of ASD after performing the ADOS-2 and MSEL.

Mullen Scales of Early Learning (MSEL) (Mullen, 1995): The MSEL is a standardized developmental test for children between 3 and 69 months of age, with five subscales: Gross Motor, Fine Motor, Visual Reception, Receptive Language and Expressive Language. An Early Learning Composite (ELC) is generated using scores from the Fine Motor, Receptive Language, Expressive Language, and Visual Reception scales. The ELC provides a developmental quotient, which is used to determine the expected levels of communicative and social functioning which help to inform the clinical impression of autism along with the Autism Diagnostic Observation Schedule-2 (ADOS-2) results.

Analyses

All analyses were conducted using the statistical computing platform, R (Core Team, 2015), and Stata/IC 14.1 (StataCorp, 2015). Descriptive statistics were tabulated on demographic characteristics of the sample. Sensitivity was calculated based on the sample of children for which diagnostic data were available and therefore does not account for children in the larger sample who screened negative on the M-CHAT.

Applicable Receiver Operating Characteristics (ROC) and 95% Confidence Intervals were estimated for the M-CHAT screen alone, and the M-CHAT with Follow-Up Interview (M-CHAT/F). In both cases, the criterion variable was the KKI clinical ASD determination made after consideration of the results of the ADOS-2 and MSEL.

Comparisons of estimated ROCs were conducted between age groups and between M-CHAT and M-CHAT/F using z-tests and two one-sided tests of equivalence (TOST). Twenty months was determined to be the cutoff between age groups that maximized age related screening differences.

Since children with false positive screens for ASD may nevertheless have developmental difficulties benefitting from early intervention, we also examined screening performance against a combined criterion of scores >1.5 SD below the mean on two or more Mullen subscales or >2 SD on any one Mullen subscale (common state eligibility criteria e.g., per nectac.org, 2015), and/or receiving an ASD diagnosis.

Using the R package, rpart: Recursive Partitioning and Regression Trees (Therneau, Atkinson & Ripley, 2015) based on the concepts of Classification and Regression Tree (CART) analysis (Breiman, Friedman, Olshen & Stone, 1984), we explored alternative algorithms for determining positive ASD screens by including individual M-CHAT items, M-CHAT items corrected by the M-CHAT/F, ASQ scale scores, and CDI scale scores as independent variables. The dependent variable in the model was the same KKI ASD determination used as the criterion variable for the Screening ROCs. CART is a partitioning algorithm with potential utility in clinical practice due to the decision tree nature of the results. While the current sample is too small and narrow (i.e., no M-CHAT negatives are included in this study) to produce a CART model robust enough to form a basis for recommending changes to current screening protocols, its use as an exploratory tool may provide direction for further research as we seek better ways to identify younger toddlers at risk for autism.

The recursive partitioning algorithm used in this study is a non-parametric, nonlinear approach to modeling effects and interactions in the data without making assumptions about an underlying stochastic model of the data. In other words, there is no attempt to fit a function or estimate parameters. The algorithm utilizes the non-probabilistic Gini splitting criteria to determine which candidate predictor variables split the cases into the most homogeneous groups with regard to the dependent variable (in our case, KKI clinical ASD determination). It is an approach that assumes the relationships within the data are complex and unknown, and follows a stepwise process of creating locally optimal partitions within the context of parent partitions. As such, CART (as well as most other recursive partitioning algorithms) is well suited to exploring a small, highly dimensional dataset for insights into potential interactions among, and relative importance of, predictor variables.

The CART model was derived from a sample of all children from both age groups with non-missing data for criterion variable (n=98). Using resubstitution on the over-fitted model, we estimated three sets of ROCs based on the entire sample (n=98), younger toddlers (<20 months; n= 48), and older toddlers (20+ months; n=50). We then compared the estimated ROCs for the CART model between age groups and against the M-CHAT and M-CHAT/F. While the CART ROCs are overinflated and not useful for generalizing the model to a larger population, they do provide a means to understand potential differences between younger and older toddlers in what might be important to consider when screening for ASD.

A beneficial feature of CART is the use of surrogate predictor variables when a case is missing data for the primary splitter at any particular node. As such, a variable may appear in the tree many times either as a primary or surrogate splitter given its overall relative importance (Therneau et al., 2015). This provides an opportunity to consider the potential relative importance of the independent variables as applied to predicting a KKI clinical ASD determination. We calculated an overall measure of importance for each independent variable using goodness of split and adjusted agreement measures generated for each primary and surrogate splitter at each node.

Results

Demographic characteristics of the subjects completing diagnostic testing with complete data are presented in Table 1. Mean age is 22.9 months (s.d.=6.1) with a range of 14.7 months to 40.8 months. The M-CHAT was assigned for their “18 month” or “24 month” visit, but parents may complete the tool at a slightly older or younger age than the M-CHAT standardization age range. As typical for children who screen positive for ASD (Halladay et al., 2015), 74.5% are male. The demographics show a range of backgrounds roughly similar to U.S. national data except that Hispanics are underrepresented. The majority of the respondents were biological mothers (90%), and had a college degree (67.4%). With the exception of respondent education (χ²=12.7, p=.048), there were no significant differences in demographic characteristics between the two age groups.

Table 1.

Demographics

	Overall (n=98)	Under 20 (n=48)	20 Plus (n=50)	χ2	p-Value
Gender	-	-	-	0.333	.564
Female	25.5%	44.0%	56.0%	-	-
Male	74.5%	50.7%	49.3%	-	-
Race	-	-	-	2.718	.257
Non-White	34.7%	39.6%	28.0%	-	-
White	65.3%	58.3%	72.0%	-	-
Missing	1.0%	2.1%	0.0%	-	-
Hispanic	-	-	-	6.005	.050
Yes	5.1%	0.0%	10.0%	-	-
No	93.9%	97.9%	90.0%	-	-
Missing	1.0%	2.1%	0.0%
Insurance		-	-	7.216	.065
Private	75.5%	79.2%	72.0%	-	-
MA/CHIPS	20.4%	12.5%	28.0%	-	-
None	1.0%	2.1%	0.0%	-	-
Missing	3.1%	6.2%	0.0%	-	-
Respondent Relationship to Child		-	-	3.784	.151
Biological Mother	89.8%	93.8%	86.0%	-	-
Other	9.2%	4.2%	14.0%	-	-
Missing	1.0%	2.1%	0.0%	-	-
Respondent Education		-	-	12.702	.048
<= 12th Grade Education	5.1%	4.2%	6.0%	-	-
HS Diploma or GED	8.2%	2.1%	14.0%	-	-
Some College	16.3%	10.4%	22.0%	-	-
Associate Degree	3.1%	2.1%	4.0%	-	-
Bachelor Degree	30.6%	31.2%	30.0%	-	-
Graduate Degree	33.7%	43.8%	24.0%	-	-
Missing	3.1%	6.2%	0.0%	-	-

The PPV of the MCHAT/F on the entire sample (n=98) was significantly higher than the PPV of the M-CHAT alone (0.58 vs. 0.40, z=−2.48, p=.013; see Table 2). While the difference between M-CHAT and M-CHAT/F remained significant among the older toddlers (0.69 vs. 0.48, z=−2.16, p=.031), it was greatly diminished among younger toddlers (0.36 vs. 0.31, z=−0.46, p=.643).

Table 2.

Comparison of Positive Predictive Values Between M-CHAT and M-CHAT/F (Criterion: ASD Diagnosis)

			95% Confidence		Difference		Equivalence
	PPV	SE	Interval		Z	p	Z1	p1	Z2	p2
All Ages
M-CHAT	0.40	0.05	0.30	0.49	−2.48	.013	3.88	<.001	−1.08	.860
M-CHAT/F	0.58	0.05	0.48	0.67
< 20 Months
M-CHAT	0.31	0.07	0.18	0.44	−0.46	.643	1.50	.067	0.57	.283
M-CHAT/F	0.36	0.07	0.22	0.49
20+ Months
M-CHAT	0.48	0.07	0.34	0.62	−2.16	.031	3.17	<.001	−1.14	.873
M-CHAT/F	0.69	0.07	0.56	0.82
Difference
Equivalence	<20 Months n=48
Indeterminate	20+ Months n=50

We did not observe a significant difference nor equivalence of M-CHAT PPV (without follow-up interview) between younger and older toddlers (see Table 3). However, M-CHAT/F PPV (0.69 vs. 0.36, z=−3.32, p<.001) and sensitivity (0.75 vs. 0.33, z=−4.14, p<.001) were markedly higher among older toddlers as compared to younger toddlers, and the False Discovery Rate was half that of the younger age group (0.31 vs. 0.64, z=3.32, p<.001). The difference in specificity was clinically non-meaningful and statistically indeterminate (0.73 vs. 0.69).

Table 3.

Comparison of Positive Predictive Value, Sensitivity & Specificity Between Younger and Older Age Groups (Criterion: ASD Diagnosis)

			95% Confidence		Difference		Equivalence
	PPV	SE	Interval		Z	p	Z1	p1	Z2	p2
M-CHAT PPV
< 20 Months	0.31	0.07	0.18	0.44	−1.69	.090	2.70	.003	−6.8	.752
20+ Months	0.48	0.07	0.34	0.62
M-CHAT/F PPV
< 20 Months	0.36	0.07	0.22	0.49	−3.32	<.001	4.31	<.001	−2.33	.990
20+ Months	0.69	0.07	0.56	0.82
M-CHAT/F Sensitivity
< 20 Months	0.33	0.07	0.20	0.47	−4.14	<.001	5.14	<.001	−3.15	.999
20+ Months	0.75	0.06	0.63	0.87
M-CHAT/F Specificity
< 20 Months	0.73	0.06	0.60	0.85	0.38	.703	0.71	.239	1.47	.071
20+ Months	0.69	0.07	0.56	0.82
M-CHAT/F False Discovery Rate
< 20 Months	0.64	0.07	0.51	0.78	3.32	<.001	−2.33	.990	4.31	<.001
20+ Months	0.31	0.07	0.18	0.44
Difference
Equivalence	<20 Months n=48
Indeterminate	20+ Months n=50

Use of the combined criterion (ASD/Developmental Delay) increased the M-CHAT and M-CHAT/F PPVs to 0.68 and 0.88, respectively. The M-CHAT/F PPV for the younger age group increased to 0.71, and for the older age group, it increased to 0.96 (see Table 4).

Table 4.

Comparison of Positive Predictive Value, Sensitivity & Specificity Between Younger and Older Age Groups (Criterion: ASD/Developmental Delay Diagnosis)

			95% Confidence		Difference		Equivalence
	PPV	SE	Interval		Z	p	Z1	p1	Z2	p2
M-CHAT PPV
< 20 Months	0.65	0.07	0.51	0.78	−0.79	.430	1.85	.032	0.27	.392
20+ Months	0.72	0.06	0.60	0.84
M-CHAT/F PPV
< 20 Months	0.71	0.07	0.59	0.84	−3.34	<.001	4.69	<.001	−1.99	.977
20+ Months	0.96	0.03	0.91	1.01
M-CHAT/F Sensitivity
< 20 Months	0.32	0.07	0.19	0.45	−3.68	<.001	4.67	<.001	−2.69	.996
20+ Months	0.69	0.07	0.57	0.82
M-CHAT/F Specificity
< 20 Months	0.76	0.06	0.64	0.88	−2.26	.024	3.64	<.001	−0.88	.811
20+ Months	0.93	0.04	0.86	1.00
M-CHAT/F False Discovery Rate
< 20 Months	0.29	0.07	0.16	0.41	3.34	<.001	−1.99	.977	4.69	<.001
20+ Months	0.04	0.03	0.00	0.09
Difference
Equivalence	<20 Months n=48
Indeterminate	20+ Months n=50

CART analyses yielded a model with much higher but artificially inflated performance estimates due to the relatively small sample size, over fitting of the model and utilization of re-substitution (see Table 5). Direct comparisons of performance between the resulting model and the M-CHAT or M-CHAT/F are therefore neither meaningful nor valid. However, the estimates do provide insight into relative differences of model performance between age groups. Differences between age groups on PPV (0.72 vs. 0.88, younger vs. older, respectively), Sensitivity (0.87 vs. 0.92), and Specificity (0.85 vs 0.88) were all substantially smaller for the CART model as compared to M-CHAT/F, an improvement for the younger children. The difference in PPV between age groups was statistically significant (z=−1.96, p=.50). For sensitivity and specificity, the differences were not significant and statistical equivalence was indeterminate.

Table 5.

Comparison of Positive Predictive Value, Sensitivity & Specificity of CART Model Between Younger and Older Toddler Age Groups (Criterion: ASD Diagnosis) – Words Produced Included

Measure			95% Confidence		Difference		Equivalence
	Value	SE	Interval		Z	p	Z1	p1	Z2	p2
PPV
< 20 Months	0.72	0.06	0.60	0.85	−1.96	.050	3.206	<.001	−0.718	.076
20+ Months	0.88	0.05	0.79	0.97
Sensitivity
< 20 Months	0.87	0.05	0.77	0.96	−0.80	0.425	2.393	.008	0.798	.215
20+ Months	0.92	0.04	0.84	0.99
Specificity
< 20 Months	0.85	0.05	0.75	0.95	−0.53	.599	1.983	.024	0.93	.176
20+ Months	0.88	0.07	0.80	0.97
Difference
Equivalence	<20 Months n=48
Indeterminate	20+ Months n=50

This study provided interesting preliminary information on the relative importance of those predictor variables that contributed to generation of the CART model (Table 6). Language variables carried largest relative importance in contributing to the age-based algorithm described above for predicting autism diagnoses. Five of the top ten items in relative importance were from the CDI and 2 were language items from the M-CHAT (Responds to name; Understands what people say) with the Words Produced scale included. The CDI Words Produced subscale is likely to be too long to be included in a practical screening regimen. Therefore, the CART analyses were repeated excluding CDI Words Produced Score. While the relative importance of variables changes, the ROCs remained similar to the original model. Four of the top ten items were from the CDI when the Words Produced Scale was excluded (Table 7).

Table 6.

Relative Variable Importance - CART Model Between Younger and Older Toddler Age Groups (Criterion: ASD Diagnosis) – Words Produced Included

Variable	Description	Importance
mcdiWFNTile	CDI Word Forms Percentile	1.00
mchatAdjAge	Age at Time of M-CHAT, Adjusted for Prematurity	0.71
mcdiWENTile	CDI Word Endings Percentile	0.38
mfuFinal14PF	MCHAT/F Item 14: Responds to Name	0.38
mcdiWPNTile	CDI Words Produced Percentile	0.33
mfuFinal02PF	MCHAT/F Item 2: Interested in Others	0.33
mcdiCXNTile	CDI Sentence Complexity Percentile	0.29
mcdiM3LNtile	CDI Mean 3 Longest Sentences Percentile	0.24
mfuFinal21PF	MCHAT/F Item 21: Understands What People Say	0.19
mfuFinal22PF	MCHAT/F Item 22: Stares at Nothing/Wanders	0.19
mfuFinal18PF	MCHAT/F Item 18: Unusual Finger Movements	0.14
mfuFinal08PF	MCHAT/F Item 8: Plays Properly	0.10
mfuFinal20PF	MCHAT/F Item 20: Wonder if Deaf	0.10
mfuFinal12PF	MCHAT/F Item 12: Smiles in Response	0.05
mchatFUResult	M-CHAT/F Screening Result	0.05
mfuFinal03PF	MCHAT/F Item 3: Likes Climbing	0.05
mfuFinal15PF	MCHAT/F Item 15: Looks When You Point	0.05
mfuFinal17PF	MCHAT/F Item 17: Looks at Same Thing	0.05
mfuFinal05PF	MCHAT/F Item 5: Pretends	0.05
gender	Gender	0.05
mcdiTooOld50	Child Was Too Old for CDI	0.05

Table 7.

Relative Variable Importance - CART Model Between Younger and Older Toddler Age Groups (Criterion: ASD Diagnosis) – NO Words Produced

Variable	Description	Importance
mcdiWFNTile	CDI Word Forms Percentile	1.00
mchatAdjAge	Age at Time of M-CHAT, Adjusted for Prematurity	0.74
mcdiWENTile	CDI Word Endings Percentile	0.43
mfuFinal14PF	MCHAT/F Item 14: Responds to Name	0.39
mfuFinal02PF	MCHAT/F Item 2: Interested in Others	0.35
mcdiCXNTile	CDI Sentence Complexity Percentile	0.30
mfuFinal21PF	MCHAT/F Item 21: Understands What People Say	0.22
mfuFinal22PF	MCHAT/F Item 22: Stares at Nothing/Wanders	0.17
mcdiM3LNtile	CDI Mean 3 Longest Sentences Percentile	0.17
mfuFinal18PF	MCHAT/F Item 18: Unusual Finger Movements	0.17
mfuFinal08PF	MCHAT/F Item 8: Plays Properly	0.09
mfuFinal20PF	MCHAT/F Item 20: Wonder if Deaf	0.09
mfuFinal12PF	MCHAT/F Item 12: Smiles in Response	0.09
mfuFinal15PF	MCHAT/F Item 15: Looks When You Point	0.04
mfuFinal17PF	MCHAT/F Item 17: Looks at Same Thing	0.04
mfuFinal05PF	MCHAT/F Item 5: Pretends	0.04
mchatFUResult	M-CHAT/F Screening Result	0.04
mcdiTooOld50	Child Was Too Old for CDI	0.04
mfuFinal10PF	MCHAT/F Item 10: Looks You in the Eye	0.04

CART analyses using the combined criterion of ASD/Developmental Delay diagnosis again resulted in changes to relative variable importance and, as expected, higher ROCs. Using the combined criterion, the ASQ developmental screen rose in relative importance (Tables 8 & 9).

Table 8.

Relative Variable Importance - CART Model Between Younger and Older Toddler Age Groups (Criterion: ASD/Developmental Delay Diagnosis) – Words Produced Included

Variable	Description	Importance
mcdiM3LNtile	CDI Mean 3 Longest Sentences Percentile	1.00
mcdiWPNTile	CDI Words Produced Percentile	0.72
mcdiWFNTile	CDI Word Forms Percentile	0.68
mchatAdjAge	Age at Time of M-CHAT, Adjusted for Prematurity	0.40
mcdiCXNTile	CDI Sentence Complexity Percentile	0.36
mcdiTooOld50	Child Was Too Old for CDI	0.20
mcdiWENTile	CDI Word Endings Percentile	0.20
asq3Communication	ASQ3 Communication Scale	0.16
asq3FineMotor	ASQ3 Fine Motor Scale	0.08
asq3GrossMotor	ASQ3 Gross Motor Scale	0.08
mfuFinal01PF	MCHAT/F Item 1: Enjoys Being Swung	0.04
mfuFinal20PF	MCHAT/F Item 20: Wonder if Deaf	0.04

Table 9.

Relative Variable Importance - CART Model Between Younger and Older Toddler Age Groups (Criterion: ASD/Developmental Delay Diagnosis) – NO Words Produced

Variable	Description	Importance
mcdiM3LNtile	CDI Mean 3 Longest Sentences Percentile	1.00
mcdiWFNTile	CDI Word Forms Percentile	0.63
mchatAdjAge	Age at Time of M-CHAT, Adjusted for Prematurity	0.40
mcdiCXNTile	CDI Sentence Complexity Percentile	0.37
mcdiWENTile	CDI Word Endings Percentile	0.30
mcdiTooOld50	Child Was Too Old for CDI	0.20
asq3Communication	ASQ3 Communication Scale	0.17
asq3FineMotor	ASQ3 Fine Motor Scale	0.10
asq3GrossMotor	ASQ3 Gross Motor Scale	0.10
mfuFinal01PF	MCHAT/F Item 1: Enjoys Being Swung	0.03
mfuFinal20PF	MCHAT/F Item 20: Wonder if Deaf	0.03

Discussion

Consistent with prior M-CHAT research this study supports use of the M-CHAT/F to identify M-CHAT false positives among toddlers older than 20 months of age, thereby substantially reducing the number of unnecessary referrals for comprehensive autism evaluation (Kleinman et al., 2008; Robbins et. al., 2001). However, this study also shows that use of the M-CHAT/F does not substantially improve the relatively low PPV of the M-CHAT in low risk toddlers younger than 20 months, confirming the published M-CHAT study (Pandey et al., 2008) which compared screening results between younger and older toddlers, and included the use of the M-CHAT/F. Importantly, our results better represent the true performance of the M-CHAT/F for younger toddlers because, in contrast to these other studies, the M-CHAT/F was completed at nearly the same time as the 18-month check-up rather than via telephone interviews an average of three months later when tests for ASD all have higher PPV (Robins, 2008). The implication is that recommendations of methods for autism screening at the time of the standard 18-month visit may need to differ from those for the 24-month check-up. These issues are likely to be less relevant to populations of known at risk children, such as those already being followed by early intervention programs. Further exploration of these data suggest that including additional language and developmental data might provide a broader range of variables needed to differentiate these age groups.

Another study of screening 18-month-olds in Japan attempted to improve prediction by lowering the threshold for a positive M-CHAT (using one critical item instead of two) and completing the follow-up interview by phone at 19 to 20 months (Kaio et al, 2013). The lower threshold and somewhat different scoring of the M-CHAT resulted in a high referral rate (17%) but the PPV was still only 0.46. Strategies other than simply lowering the threshold are needed to make ASD screening as accurate for younger toddlers as it is for older ones, especially since referral rates are already higher for the younger toddlers as seen in this study and that of Pandey and colleagues (2008). The false discovery rate was twice as high in the younger group than the older one in this study making referral of all positives impractical.

It should be noted that while use of a follow-up interview may improve positive predictive value and conserve scarce evaluative resources, it will not do anything to improve sensitivity and may even overlook children who ultimately have an ASD diagnosis identified by the wider filter of the initial parent response items (Stenberg et al., 2014). These results underline the need to consider early identification as an iterative process over time rather than counting on a single screen. When the limitations of current screening tools are recognized, strategies may be considered for including the screening tools as part of a promising system of “developmental surveillance” using discrete observations by specially trained PCPs (Barbaro & Dissanayake, 2010). While there is already a general recommendation by the AAP for ongoing surveillance in addition to screening, it may be time to consider bolstering the surveillance component with this more targeted and enhanced approach, at least for the younger toddlers. In addition, clinicians need to be aware that prediction improves after 20 months. There may be some cases when re-screening the children who fail the M-CHAT/F at the 18-month visit two months later might be deemed appropriate. This would, of course, not include cases where there is reason for concern based on issues raised through a surveillance process or current parent or clinician concern.

The results of this study and that of Pandey and colleagues are quite similar, with PPV for predicting a clinical diagnosis of ASD in this study of 0.69 for the older toddlers and 0.36 for the younger ones, compared to 0.61 and 0.28 for the corresponding age groups of Pandey. These studies also point to differences in PPV when considering children with either developmental delays or ASD outcomes (ASD/DD) using a common criteria for offering developmental services in U. S. government sponsored early intervention services to define developmental difficulties. For the combined ASD/DD criterion PPV was higher for the older age group in both studies, 0.96 in this study and 0.95 in the Pandey study, compared to 0.71 and 0.72 in the younger age group. This suggests that, while most of the children being referred from M-CHAT screening at the 18 month visit may be found to have a developmental difficulty, the difficulty is less likely to be ASD than something else. This does not support the claim that the M-CHAT is an autism specific screen at this age. While the M-CHAT is one of the tools considered to be “autism specific” rather than a “broad band” developmental screen, our study confirms that children failing the M-CHAT and M-CHAT/F screen who are not determined to have an ASD diagnosis are still likely to have developmental difficulties severe enough to qualify for early intervention services. We may, therefore, want to think of the MCHAT as a broader clinical developmental screening tool that has the added benefit of picking up on autism – rather than counting on the tool to be the complete answer for autism identification.

There are a number of possible explanations regarding the difficulty in effectively detecting ASD in younger children. For example, the age of onset of symptoms varies, with 22–41% of cases regressing from apparently typical development or only mild delays to signs of ASD after 16 months (Barger et al., 2013). In those cases, most show clear signs by 24 months, warranting the second recommended screening age (Academy of Pediatrics, 2006). In addition, atypical odd and repetitive behaviors may not appear until the end of the second year (e.g., Veness et al., 2012). Prospective studies indicate that ASD symptoms may emerge gradually in the toddler age group (Landa, Holman & Garrett-Mayer, 2007; Landa, Gross, Stuart & Faherty, 2013; Ozonoff et al., 2008). Specifically children misclassified at 18 months in a prospective studies of siblings of a proband with ASD were higher functioning, and their autism symptoms increased between 18 and 36 months (Chawarska et al., 2014). Children who present for asymptomatic screening are more likely to be higher functioning than those who are identified because of specific parental concern simply because they have more reassuring developmental features as well as less autistic features. The yes/no format of the M-CHAT may also not capture the “sometimes” status of the behavior at that point in time. However, in addition to the children who are actually typical before regression, it also may not be reasonable to expect some bright children with mild ASD to be detectable as young toddlers.

Between 18 and 24 months is a time of great expressive language emergence. Evidence of adequate language development appeared to be an important differentiator between children who did not show evidence for ASD from those who did. This study provides evidence that beginning use of more advanced language forms as measured by a MacArthur-Bates Communicative Development Inventory has potential to be one of the major factors differentiating risk from non-risk for ASD in both age groups. For children who are positive on the M-CHAT screen for autism phenotype, additional items indicating that the child is beginning to exhibit language forms that are just emerging at that developmental age contribute to more accurate decision rules for autism evaluation referrals. However, use of language measures standardized by age have not been a part of autism specific screening tests such as M-CHAT, which uses the same items and same scoring across this entire age range. Some of the additional language items which improve prediction are relatively brief ones, but the CDI vocabulary inventory (Words Produced subscale) is probably too lengthy (680 words) to be considered practical for inclusion in a screening battery, even as a follow-up. However, existing short forms should be explored and, recently, an even more abbreviated computer adaptive version has been developed which is highly correlated with the longer version (Makransky, Dale, Havmose & Bleses, 2016). Issues of efficiency should be weighed in the development of screening tests suitable for younger toddlers. However, even without the use of this vocabulary subscale, other briefer CDI items still enhance prediction for the youngest group of toddlers almost as well. In addition, if the salience of language items in multivariate models predicting autism diagnoses in toddlers can be replicated, this may have implications for rethinking the centrality of language in the way autism is defined, such as when comparing the prominence of language functioning in DSM-4 criteria versus its lesser role in DSM-5.

While the ASQ developmental screening items did not significantly contribute beyond the CDI items to improve prediction of autism for young toddlers, this routine screen does make a significant contribution to a model predicting developmental delay as well as ASD for children who were positive on the M-CHAT screening test. It should be noted that the screening power of the language items found in this study may result from language being a better proxy for overall developmental level or IQ than the ASQ. This study also suggests that a different pattern of behaviors may characterize emerging ASD in the 18-month visit age group than seen as toddlers approach age two years. In addition, demonstration of more advanced language abilities at the younger toddler age may reassure that there is no ASD diagnosis obviating the need for additional assessments.

The decision trees developed through CART analyses models are too exploratory at this point for clinical application. If replicated, computer systems presenting screening items to parents could vary item presentation depending on the pattern of prior item responses to make a more accurate screening process. Some parents will have a short screening session and others somewhat longer with the variable pattern of item presentation generated through an algorithmic item selection process.

Conclusions

The known heterogeneous etiology of ASD (e.g., Scherer & Dawson, 2011) and its dynamic natural history present challenges that require different screening and assessment strategies at different ages. The two age groups described in this study are of practical relevance to child health care delivery because they correspond to two of the standard well-child visits that are ones at which autism screening is recommended by the American Academy of Pediatrics (18 and 24 months) (American Academy of Pediatrics, 2006). There is an understandable desire of screening test developers to employ simple scoring such as a total score for all ages of children. However, such ease of use comes at the cost of precision. While confirmation of the strategies suggested in this study is essential before clinical application, optimal scoring is likely to be complex, differ by age and gender and be difficult to remember and score unless aided by a computer. Computer-aided parent report screening technologies now allow for more sophisticated algorithms without placing undue burden on parents or clinicians.

Limitations

There are limitations to this study that must be considered when interpreting results. First, while a community pediatric practice population was sampled, there was no attempt to assess children passing the screening. Also, the subsample consenting and completing diagnostic testing was skewed by not being fully representative since many parents declined participation. While there was an attempt to identify missed cases by recruiting all children who screened positive before the follow-up interview, the estimate of sensitivity for the entire community sample remains artificially high because all the screened negative children were not evaluated. Fewer than half of the PCPs in participating offices joined the study and administered a M-CHAT/F when indicated, and only a third of M-CHAT/F positives were evaluated at the KKI Autism center. Additionally, the sample size is relatively small. However, the sampling procedures at the 18-month visit were the same as at 24 months with similar demographic characteristics suggesting that these sampling issues are unlikely to confound the issue of age differences, the central issue of this report. However, by using the office assignment to a “18 month or “24 month visit” rather than limiting subject enrollment by actual age, the sample included a few children who were a bit younger and a bit older than the standardization of the test. It should also be noted that the version of the M-CHAT used here was the 23-item earlier version, but we do not have a reason to believe that the slight item and wording modifications would affect the younger toddler age group differently than the older group.