Sensitivity and Specificity of the Modified Checklist for Autism in Toddlers (Original and Revised)

Key Points

Question

What factors are associated with the sensitivity and specificity of the Modified Checklist for Autism in Toddlers (M-CHAT) and the M-CHAT, Revised With Follow-up (M-CHAT-R/F)—henceforth referred to as M-CHAT(-R/F)?

Findings

In this systematic review and meta-analysis of 50 studies, the pooled sensitivity of M-CHAT-R/F was 0.83, and the pooled specificity was 0.94. Heterogeneity analyses revealed greater diagnostic accuracy for low- vs high-likelihood samples, a concurrent vs prospective case confirmation strategy, a large vs small sample size, use of M-CHAT(-R) Follow-up, and non-English vs primarily English.

Meaning

Findings suggest that M-CHAT(-R/F) shows strong performance as an autism spectrum disorder screener, but researchers and clinicians should be aware of the variability in the sensitivity and specificity based on multiple factors.

Abstract

Importance

The Modified Checklist for Autism in Toddlers (M-CHAT) and the M-CHAT, Revised With Follow-up (M-CHAT-R/F)—henceforth referred to as M-CHAT(-R/F)—are the most commonly used toddler screeners for autism spectrum disorder (ASD). Their use often differs from that in the original validation studies, resulting in a range of estimates of sensitivity and specificity. Also, given the variability in reports of the clinical utility of the M-CHAT(-R/F), researchers and practitioners lack guidance to inform autism screening protocols.

Objective

To synthesize variability in sensitivity and specificity of M-CHAT(-R/F) across multiple factors, including procedures for identifying missed cases, likelihood level, screening age, and single compared with repeated screenings.

Data Sources

A literature search was conducted with PubMed, Web of Science, and Scopus to identify studies published between January 1, 2001, and August 31, 2022.

Study Selection

Articles were included if the studies used the M-CHAT(-R/F) (ie, original or revised version) to identify new ASD cases, were published in English-language peer-reviewed journals, included at least 10 ASD cases, reported procedures for false-negative case identification, screened children by 48 months, and included information (or had information provided by authors when contacted) needed to conduct the meta-analysis.

Data Extraction and Synthesis

The systematic review and meta-analysis was conducted within the Preferred Reporting Items for Systematic Reviews and Meta-analyses (PRISMA) reporting guideline. The Quality Assessment of Diagnostic Accuracy Studies–2 tool evaluated bias in sample selection. Data extraction and quality assessment were performed by 2 authors independently. The overall diagnostic accuracy of the M-CHAT(-R/F) was assessed with the hierarchic summary receiver operating characteristic (HSROC) model.

Main Outcomes and Measures

Sensitivity, specificity, diagnostic odds ratios, and HSROC curves of M-CHAT(-R/F).

Results

The review included 50 studies with 51 samples. The pooled sensitivity of M-CHAT(-R/F) was 0.83 (95% CI, 0.77-0.88), and the pooled specificity was 0.94 (95% CI, 0.89-0.97). Heterogeneity analyses revealed greater diagnostic accuracy for low- vs high-likelihood samples, a concurrent vs prospective case confirmation strategy, a large vs small sample size, use of M-CHAT(-R/F) Follow-up, and non-English vs English only.

Conclusions and Relevance

Overall, results of this study suggest the utility of the M-CHAT(-R/F) as an ASD screener. The wide variability in psychometric properties of M-CHAT(-R/F) highlights differences in screener use that should be considered in research and practice.

This systematic review and meta-analysis assesses the sensitivity and specificity of the Modified Checklist for Autism in Toddlers (M-CHAT) and the M-CHAT, Revised With Follow-up as autism spectrum disorder screeners.

Introduction

Autism spectrum disorder (ASD) is a neurodevelopmental disorder marked by core deficits in social communication and restricted and repetitive behaviors. It can be accurately detected by 18 months of age, although the diagnosis usually occurs much later.^1,2,3 Early detection of ASD informs autism-specific early intervention, which improves outcomes.^4,5 The American Academy of Pediatrics (AAP) recommends all children undergo general developmental and autism-specific screening paired with developmental surveillance.⁶ Although most primary care professionals screen for autism with a standardized autism-specific tool (eg, 72%),⁷ implementation of screening is inconsistent, in part owing to the lack of a recommendation for universal screening from the US Preventive Services Task Force.⁵ Reports of low sensitivity of toddler screening^8,9 may also affect universal implementation.

The Modified Checklist for Autism in Toddlers (M-CHAT)¹⁰ and the M-CHAT, Revised With Follow-up (M-CHAT-R/F)¹¹—henceforth referred to as M-CHAT(-R/F)—are the most commonly used ASD-specific screening tools.¹² However, implementation often diverges from that in the original validation studies, including use of the Follow-up portion of the M-CHAT-R/F administration, which limits interpretation of sensitivity and specificity. Previous studies describing variability in the methods used to examine psychometric properties of ASD screening tools,^13,14 and the M-CHAT specifically,¹⁵ have been limited in their assessment of contextual factors, such as the rigor of false-negative case identification strategies. Subsequently, there is lack of consensus regarding M-CHAT(-R/F) screening properties. The present systematic literature review and meta-analysis examines screening properties of the M-CHAT(-R/F) to assess variations in reported sensitivity and specificity across multiple factors, including procedures for identifying missed cases, likelihood level, screening age, single or repeated screenings, sample size, version, language administered, and use of the structured Follow-up portion of the M-CHAT(-R/F).

Methods

Search Strategy

This systematic review and meta-analysis was conducted following the Preferred Reporting Items for Systematic Reviews and Meta-analyses (PRISMA) reporting guideline.¹⁶ We searched PubMed, Web of Science, and Scopus to identify peer-reviewed articles published between January 1, 2001, and August 31, 2022 (see eTable 1 in Supplement 1 for key words). This systematic review and meta-analysis has been registered with the International Prospective Register of Systematic Reviews (CRD42021232792).

Inclusion Criteria

Included articles used M-CHAT(-R/F) to screen for ASD and (1) were published in English-language peer-reviewed journals; (2) used M-CHAT(-R/F) to screen children younger than 48 months, before ASD diagnosis; (3) identified at least 10 ASD cases; (4) reported procedures for identifying false-negative cases (critical to estimate sensitivity); and (5) included information (or had information provided by authors when contacted) needed to conduct the meta-analysis (rates of true-positive, false-positive, false-negative, and true-negative cases). Reviews, commentaries, and conference articles were excluded. For studies with overlapping data sets, the study with the largest sample was included.

Data Extraction and Quality Assessment

Data in Table 1 were extracted independently by 2 authors (A.T.W. and L.N.W.).^{8,9,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,57,58,59,60,61,62,63,64} Screening metrics not provided in the studies were calculated from raw values. When possible, specificity was recalculated from the raw numbers so that true-negative cases included presumed true-negative ones (ie, screen-negative children who were not evaluated). Of 42 authors contacted for missing information, 27 either responded with the requested information or noted that it was not available.

Table 1. Study Characteristics and Screening Metrics for M-CHAT(-R/F).

Source	Total No.a	Screen age, mob	Evaluation age, mob	HL, LL, or mixed	M-CHAT language	FU	ASD detectionc	TP	FN	FP	TN	Sens	Specd
Baduel et al,17 2017	1250	22-26	24-34	LL	French	Y	C: FN strategy	12	6	8	1201	0.667	0.993
Beacham et al,18 2018	154	16-45	16-45	HL	English	N	C: all eval	105	19	14	16	0.847	0.533
Canal-Bedia et al,19 2011	2480	18-36	18-48	Mixede	Spanish	Y	C: weak	23	0	43	2414	1.000	0.980f
Carbone et al,8 2020	26 364	16-30	46.8 (17.7)g	LL	English	Y/Nh	P: record	125	253	579	25 407	0.331	0.978
Chang et al,20 2021	990	17-37	17-37	LL	English	Y	C: FN strategy	31	7	11	941	0.816	0.988
Charman et al,21 2016	543	18-56	32-73	HL	English	N	C: all eval	45	10	32.5	32.5	0.818	0.500
Chlebowski et al,22 2013	18 989	16-30	25.8 (4.5)	LL	English or Spanish	Y	C: FN strategy	92	6	79	18 289	0.939	0.996
Choueiri et al,23 2021	80	18-36	18-36	HL	English	N	C: all eval	53	3	0	24	0.946	1.000
Christopher et al,24 2021	290	18-48	18-48	HL	English	Y	C: all eval	170	48	48	24	0.780	0.333
Coelho-Medeiros et al,25 2019	120	16-30	16-30	Mixede	Spanish (Chile)	Y	C: FN strategy	18	0	4	97	1.000	0.960
Dereu et al,26 2012	199	16-31	13-51	HL	Dutch	N	C: FN strategy	10	4	22	163	0.714i	0.881
DiGuiseppi et al,27 2010	85	20-86	20-86	HL	English or Spanish	N	C: FN strategy	9	2	26	38	0.818	0.594
Dudova et al,28 2014	157	≈24	NA	HL	Czechj	N	C: FN strategy	9	4	9	112	0.692	0.926
Eaves et al,29 2006	84	17-48	22-53	HL	English	N	C: all eval	48	4	22	8	0.923	0.267
Guo et al,30 2019	7928	16-30	23 (4)	LL	Chinese	Y	C: FN strategy	72	10	103	7166	0.878	0.986
Guthrie et al,9 2019k	20 375	16-26	17-88	LL	English or Spanish	Y/Nh	P: record	225	229	1247	18 674	0.496	0.937
Harris et al,31 2021l	360	24-48	NA	LL	English or Spanish	Y	C: FN strategy	8	29	4	315	0.216	0.987
Hoang et al,32 2019	17 277	18-30	18-30	LL	Vietnamese	N	C: FN strategy	129	1	118	17 021	0.992	0.993
Inada et al,33 2011	1187	17-23	35-44	LL	Japanese	N	P: FN strategy	11	9	46	1121	0.550	0.961
Jonsdottir et al,34 2022	1586	31.7 (1.7)	NAm	LL	Icelandic	Y	P: record	18	11	7	1549	0.621	0.988
Kamio et al,35 2014	1851	17-26	33-73	LL	Japanese	Y	P: FN strategy	29n	22	24	1661	0.569	0.986
Kanne et al,36 2018	158	18-48	24-32	HL	English	Y	C: all eval	96	23	24	15	0.807	0.385
Kara et al,37 2014	618	18-36	24-42	Mixede	Turkish	Y	C: FN strategy	45	2	15	534	0.957	0.973
Keehn et al,38 2021	605	18-48	18-48	HL	English	Y	C: all eval	198	31	234	142	0.865	0.378
Kerub et al,39 2020	1591	18-36	NAo	LL	Hebrew	Y	P: record	7	3	43	1538	0.700	0.973
Kim et al,40 2016	827	14-43p	110-151	HL	English	N	P: all eval	30	28	123	646	0.517	0.840
Kleinman et al,41 2008q	1416	16-30	52.2 (8.0)g	Mixede	English	Y	P: FN strategy	73	7	51	1285	0.913	0.962
Koh et al,42 2014	580	17-48	18-69	HL	English or Chinese	N	C: all eval	158	41	123	258	0.794	0.677
Magán-Maganto et al,43 2020	6625	14-36	23-36	LL	Spanish	Y	C: FN strategy	15	4	24	6542	0.789	0.996
Matson et al,44 2013	552	16-30	16-30	HL	English	N	C: all eval	150	101	150	151	0.598	0.502
Oner and Munir,45 2020	6712	16-36	16-41	LL	Turkish	Y	C: weak	57	0	95	6388	1.000	0.985
Robins et al,46 2014	16 041	16-30	26.2 (5.5)	LL	English	Y	C: FN strategy	105	18	116	15 496	0.854	0.993
Salim et al,47 2020	143	18-48	18-48	HL	Indonesian	N	C: all eval	16	1	27	99	0.941	0.786
Salisbury et al,48 2018	485	16-48	16-48	HL	English	N	C: all eval	220	77	82	106	0.741	0.564
Samadi and McConkey,49 2015	2941	24-60	24-60	LL	Kurdish or Persian	N	C: FN strategy	25	3	45	2380	0.903r	0.981
Schjølberg et al,50 2022s	54 436	19.0 (1.2)	≈42	LL	Norwegian	N	P: record	105	232	4048	50 078	0.312	0.925
Smith et al,51 2013	217	18-48	18-48	HL	English	N	C: all eval	97	39	31	50	0.713	0.617
Snow and Lecavalier,52 2008	56	18-48	18-48	HL	English	N	C: all eval	38	5	8	5	0.884	0.385
Srisinghasongkram et al,53 2016 (sample 1)	109	18-48	18-48	HL	Thai	Y	C: all eval	40	5	1	63	0.889	0.984
Srisinghasongkram et al,53 2016 (sample 2)	732	18-48	18-48	LL	Thai	Y	C: FN strategy	9	0	1	722	1.000	0.999
Sturner et al,54 2016	5071	18-24	14-41	LL	English	Y	C: FN strategy	23	16	17	4772	0.590	0.996
Sturner et al,55 2022	408	16-20	20.5 (1.9)	LL	English	Y	C: FN strategy	46	17	118	227	0.730	0.658
Taylor et al,56 2014	145	28.1 (4.8)	28.1 (4.8)	HL	English	N	C: all eval	74	12	26	33	0.860	0.559
Toh et al,57 2018	19 297	15-36	NA	LL	Malay, Chinese, or English	N	P: record	18	32	20	19 227	0.360	0.999
Tsai et al,58 2019	317	16-32	36-37	Mixed	Mandarin Chinese	Y	P: all eval	22	3	19	273	0.860r	0.935
Thi Vui et al,59 2022	40 243	18-30	NA	LL	Vietnamese	N	C: FN strategy	302	3	193	39 726	0.990	0.995
Weitlauf et al,60 2015	74	16-21	18-43	HL	English	Y	C: FN strategy	21	6	7	29	0.778	0.806
Wieckowski et al,61 2021t	3052	17-22	18-60	LL	English or Spanish	Y	C: FN strategy	61	13	79	2729	0.824	0.972
Windiani et al,62 2016	110	18-48	18-48	HL	Indonesian	Y	C: all eval	16	2	5	87	0.889	0.946
Wong et al,63 2018	236	18-47	18-47	HL	Chinese	N	C: all eval	99	14	58	65	0.876	0.528
Zhang et al,64 2022	11 190	18-24	23.1 (4.6)	LL	Chinese	Y	C: FN strategy	33	15u	56u	11 056u	0.688	0.995

Abbreviations: all eval, all participants evaluated; ASD, autism spectrum disorder; C, concurrent; FN, false negative; FP, false positive; FU, M-CHAT(-R/F) Follow-up administration; HL, high likelihood for ASD; LL, low likelihood for ASD; M-CHAT(-R/F), Modified Checklist for Autism in Toddlers and Modified Checklist for Autism in Toddlers, Revised With Follow-up (original and revised versions combined); N, no; NA, not available; P, prospective; record, medical record review; Sens, sensitivity; Spec, specificity; TN, true negative; TP, true positive; Y, yes; Y/N, Y, but not consistently.

^a Refers to sample who received the M-CHAT(-R/F).

^b Range reported for entire sample who received M-CHAT(-R/F), when available. If not available, mean (SD) is reported. If mean (SD) was not available, an estimate from the article or from communication with authors is reported.

^c Strategy used to detect FN cases.

^d Specificity was recalculated from the raw numbers so that TN cases included presumed TN results (ie, including children who screened negative but were not further evaluated) for consistency across studies, unless noted otherwise. Negative screen results were presumed to be TN unless there was other presented evidence.

^e Reclassified as low risk for analyses because most participants were low risk.

^f True negative and specificity taken directly from article and not recalculated due to missing information.

^g Age of evaluation is for ASD sample only; age for non-ASD sample is unknown.

^h Yes/no was reclassified for analyses: Carbone et al⁸ was reclassified as N based on few practices using the Follow-up portion of the M-CHAT-R/F, and Guthrie et al⁹ was reclassified as Y even though the Follow-up portion of the M-CHAT-R/F was not always used, given that it was built into the medical record system and intended to be used when indicated.

ⁱ Sensitivity differs slightly from that reported in the article because of a focus on M-CHAT(-R/F) only.

^j Language presumed as not directly reported in the article.

^k Values were obtained from the main author for repeated screenings and do not match those reported in the article for single screening.

^l Subsample of children screened before 48 months of age only is reported. Values were obtained from communication with the main author.

^m Age of evaluation was up to 18 months after the age of screening.

ⁿ P value differs from the one reported in the article because of the addition of 9 nonresponders who needed the Follow-up portion of the M-CHAT-R/F but did not complete it and had confirmed ASD.

^o Age of evaluation was within 10 months of screenings.

^p Screen age reported is uncorrected for prematurity.

^q Study 2 sample only presented and analyzed because of overlap of sample 1 with Chlebowski et al.²²

^r Sensitivity calculation slightly differs from article’s reported sensitivity owing to rounding in calculation from raw numbers.

^s Information for M-CHAT²³ is reported.

^t Information is reported for 18-month screening start age only.

^u Numbers adjusted to include the 12 screen-negative children with a diagnosis of ASD during subsequent well-child visit and follow-up.

Two authors (A.T.W. and L.N.W.) used the Quality Assessment of Diagnostic Accuracy Studies–2 (QUADAS-2)⁶⁵ tool, adapting signaling questions (see Yuen et al¹⁵) to evaluate bias in sample selection, implementation of M-CHAT(-R/F), and diagnostic assessment procedures. QUADAS-2 assesses risk of bias across 4 domains (patient selection, index test, reference standard, and flow and timing) and applicability across 3 domains (patient selection, index test, and reference standard). Each study received ratings of low, high, or unclear risk of bias and applicability for each domain according to the signaling questions (eTables 2 and 3 in Supplement 1).

Data Analysis

Data analyses were conducted between March 11, 2022, and October 10, 2022. The overall diagnostic accuracy of the M-CHAT(-R/F) was assessed with the hierarchic summary receiver operating characteristic (HSROC)^66,67 model using the MetaDAS SAS version 1.3.0 (SAS Institute Inc) macro.⁶⁸ Models were run with and without covariates: likelihood level of sample, case confirmation strategy classification, sample size, M-CHAT version, use of the Follow-up, and language. The HSROC parameters output by each model were input into RevMan, version 5 (Cochrane) software to create the HSROC summary curves. Models were run with all included studies except those that were unable to be classified into main categories based on predominant data (eMethods in Supplement 1). The diagnostic odds ratio (DOR) is an estimate of overall diagnostic test accuracy, reflecting how many times higher the odds are of obtaining an M-CHAT(-R/F) score in the diagnostic range for a randomly selected person with ASD vs without ASD. The likelihood ratio test assesses the goodness of fit of 2 competing statistical models by testing whether the ratio of their likelihoods is significantly different from 1 and was used to assess whether the addition of each covariate to the model was significant (eMethods in Supplement 1).

Results

Search Results

Overall, 50 published studies met the criteria and were included in our systematic review (see eFigure 1 in Supplement 1 for exclusions). One study⁵³ included 2 distinct samples, resulting in 51 study samples described in the present review and meta-analysis.

Study Characteristics

Table 1 and eTable 4 in Supplement 1 summarize the characteristics of the included studies. Almost half of the samples were small (<500; 21 of 51 studies [41%]); 26 (51%) administered the M-CHAT(-R/F) primarily in English. Thirty-two studies (63%) used the original version of M-CHAT, whereas 19 (37%) used M-CHAT-R. Thirty studies (59%) used the structured Follow-up portion of the M-CHAT-R/F with the original or revised M-CHAT, and 21 (41%) used only the initial items.

Studies differed based on case confirmation strategies. Most of the 51 studies (n = 40 [78%]) used concurrent detection methods, defined as evaluation to confirm ASD status within 6 months of screening. These studies included (1) additional rigorous false-negative case detection approaches, such as asking pediatric physicians to note ASD concerns, using additional screening tools for all or a subset of the sample (n = 21), or both^{17,20,22,25,26,27,28,30,31,32,37,43,46,49,53,54,55,59,60,61,64}; or (2) evaluation of all children regardless of screening results (n = 17).^{18,21,23,24,29,36,38,42,44,47,48,51,52,53,56,62,63} In addition, 2 studies used strategies that were classified as weak concurrent false-negative case confirmation strategies, defined as strategies that resulted in few children who had not screened positive being invited for an evaluation.^19,45 The rest of the studies (n = 11 [22%]) used prospective approaches to identify children who received a diagnosis of ASD, defined as diagnostic confirmation occurring when children were older (>6 months from screening). These studies used strategies that included medical record reviews from both primary and specialty clinics (n = 6),^{8,9,34,39,50,57} additional rigorous strategies to identify false-negative cases through additional screening tools or physician-indicated concerns (n = 3),^33,35,41 and evaluation of all children regardless of screening result (n = 2).^40,58

Studies also differed significantly based on the characteristics of the children screened. Twenty-four of the 51 studies (47%) recruited low-likelihood (LL) children, which included screening at well-child visits, childcare centers, kindergarten, or preschool. Twenty-two studies (43%) used high-likelihood (HL) samples, including children referred to community health services or for ASD-specific evaluations because of parent or practitioner concern about development,^{18,21,24,29,36,38,42,47,48,51,52,53,56,62,63} those at elevated likelihood for ASD according to another screening result,²⁶ those receiving services through an early intervention system,^23,44 those with Down syndrome,²⁷ those with a history of prematurity or low birth weight,^28,40 and those with older siblings who had received a diagnosis of ASD.⁶⁰ In addition, 5 studies administered the M-CHAT(-R/F) to a mixed sample comprising both LL and HL children.^{19,25,37,41,58} Most of these studies included predominantly LL samples, however.

Methodological Quality of Included Studies

Figure 1 shows the results of the methodological quality for all published studies included in the review. There was generally low risk of bias across domains with the exception of the flow and timing, for which 36 of the 51 studies (71%) showed high risk of bias, although this was because many studies used LL samples that did not evaluate all children screened with the M-CHAT(-R/F) or because there was longer timing between screening and evaluation in prospective case confirmation studies. Regarding concern for applicability, all studies showed low concern for patient selection and reference standard, with only 1 rated to be of high risk and 1 unclear risk for concern of applicability for the index test domain. See eTable 3 in Supplement 1 for domain ratings for individual studies.

Figure 1. Methodological Quality Graph Depicting the Cumulative Findings of the Methodological Quality Analysis.

QUADAS-2 indicates Quality Assessment of Diagnostic Accuracy Studies–2.

Meta-analysis Results

Across all studies, M-CHAT-(R/F) sensitivity ranged from 0.22 to 1.00, with a pooled sensitivity of 0.83 (95% CI, 0.77-0.88). Specificity similarly ranged from 0.27 to 1.00, with a pooled specificity of 0.94 (95% CI, 0.89-0.97). See Table 1 for screening metrics for each identified study, Table 2 for HSROC model results, and Figure 2 for a forest plot of included studies, sorted by sensitivity.^{8,9,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,59,60,61,62,63,64} The SROC plot of the overall model is depicted in eFigure 2 in Supplement 1. Figure 3 displays a visual of sensitivity and specificity values grouped by several factors affecting the values, including strategy used for identifying missed cases, risk level, and size of the study.

Table 2. Sensitivity and Specificity of M-CHAT(-R/F) Overall and by Study Characteristics.

Characteristic	No.	Sens	Spec	DOR (95% CI)	−2LL	P value
Overall	49a	0.83	0.94	75.40 (35.56-159.87)	NA	NA
M-CHAT version
M-CHAT	31	0.83	0.94	63.47 (25.38-158.73)	1.35	.72
M-CHAT-R	18	0.83	0.95	102.69 (27.87-378.26)
Likelihood level
Low	27	0.83	0.99	334.78 (156.69-715.31)	58.28	<.001b
High	22	0.82	0.70	10.93 (4.45-26.83)
Case confirmation strategy
Concurrent	40	0.86	0.93	85.84 (37.56-196.16)	17.66	.001b
Prospective	9	0.57	0.97	38.71 (8.38-178.82)
Sample size
<500	20	0.83	0.78	17.31 (6.48-46.21)	38.26	<.001b
500-5000	17	0.80	0.96	88.94 (29.58-267.43)
>5000	12	0.85	0.99	553.41 (159.34-1922.06)
Follow-up
Initial only	22	0.82	0.84	24.61 (9.01-67.23)	11.29	.01b
Follow-up	27	0.83	0.97	182.87 (71.12-470.22)
Language
English or primarily English	26	0.78	0.85	20.19 (7.85-51.91)	23.96	<.001b
Other	23	0.89	0.98	361.76 (145.80-897.58)

Abbreviations: DOR, diagnostic odds ratio; M-CHAT, Modified Checklist for Autism in Toddlers; M-CHAT-R, Modified Checklist for Autism in Toddlers, Revised; NA, not applicable; Sens, sensitivity; Spec, specificity; −2LL, −2 log likelihood difference.

^a Studies classified as “mixed” or “unknown” in any category were excluded from the analysis (n = 2).^57,58 When the overall model was run with the full set of 51 studies, sensitivity was only slightly different, specificity was identical, and DOR was slightly higher (78.71 [95% CI, 38.11-162.57]). Therefore, for comparison across the subanalyses, the common set of 49 studies is reported.

^b P < .05.

Figure 2. Forest Plot of Modified Checklist for Autism in Toddlers and Modified Checklist for Autism in Toddlers, Revised With Follow-up (Original and Revised Versions Combined) Sorted by Sensitivity.

FN indicates false negative; FP, false positive; TN, true negative; and TP, true positive.

^aRefers to sample 2 of Srisinghasongkram et al.⁵³

Figure 3. Sensitivity and Specificity Estimates Grouped by Strategy Used for Identifying Missed Cases, Likelihood Level, and Size of Study Sample.

FN indicates false negative.

Analyses testing the association of covariates (likelihood level of sample, case confirmation strategy classification, sample size, M-CHAT version, Follow-up use, and language) suggested several significant covariates (overall: DOR, 75.40 [95% CI, 35.56-159.87]) (Table 2; eFigure 2 in Supplement 1). All covariates examined except for the M-CHAT(-R/F) version (original vs revised) were statistically significant (Table 2; eFigure 3 in Supplement 1). Mixed and unknown subgroups were excluded from the models presented. When included, sensitivity was not significantly different, specificity was identical, and DOR was slightly higher (78.71 [95% CI, 38.11-162.57]). The LL group reported a DOR more than 30 times greater than that of the HL group (334.78 [95% CI, 156.69-715.31] vs 10.93 [95% CI, 4.45-26.83]) (Table 2; eFigure 4 in Supplement 1). Sensitivity was similar between the 2 groups (0.83 vs 0.82), but specificity of the LL group was higher than that of the HL group (0.99 vs 0.70). Higher DOR was achieved by the concurrent rather than the prospective case confirmation strategy (85.84 [95% CI, 37.56-196.16] vs 38.71 [95% CI, 8.38-178.82]) (Table 2; eFigure 5 in Supplement 1). Although the concurrent group achieved a higher sensitivity than the prospective group (0.86 vs 0.57), there were no large differences in specificity based on the detection strategy (0.93 vs 0.97). Diagnostic accuracy increased with sample size; the small sample size group (<500) had a DOR 5 times smaller than that of the medium sample size group (500-5000), which was, in turn, approximately 6 times smaller than that of the large sample size group (>5000). The DOR was 17.31 (95% CI, 6.48-46.21), 88.94 (95% CI, 29.58-267.43), and 553.41 (95% CI, 159.34-1922.06), respectively (Table 2; eFigure 6 in Supplement 1). Specificity increased with increasing sample size (0.78, 0.96, and 0.99, respectively), but sensitivity did not (0.83, 0.80, and 0.85, respectively). Higher DOR, sensitivity, and specificity were observed for use of the Follow-up compared with initial screening only (DOR, 182.87 [95% CI, 71.12-470.22] vs 24.61 [95% CI, 9.01-67.23]; sensitivity, 0.83 vs 0.82; specificity, 0.97 vs 0.84) (Table 2; eFigure 7 in Supplement 1). Higher DOR, sensitivity, and specificity were observed for studies conducted in languages other than predominantly English (DOR, 361.76 [95% CI, 145.80-897.58] vs 20.19 [95% CI, 7.85-51.91]; sensitivity, 0.89 vs 0.78; specificity, 0.98 vs 0.85) (Table 2; eFigure 8 in Supplement 1).

Systematic Review Results

Two factors were examined in an insufficient number of studies to be assessed as covariates in meta-analyses: screening age and repeated screening. We therefore provide a descriptive review of these factors here. Although M-CHAT(-R/F) was initially validated for children between 16 and 30 months of age, it has been used with children up to 48 months of age. Only 5 included studies directly compared M-CHAT(-R/F) psychometric values for younger children within the validated age range (up to 30 months) vs children older than 30 months (eTable 5 in Supplement 1). Descriptively, all 5 studies found slightly lower sensitivity for children older than 30 months compared with younger than 30 months.^{18,24,36,42,48} However, specificity differed, with 1 study reporting lower specificity with children older than 30 months,¹⁸ whereas other studies showed higher specificity for the same age group.^24,36,42,48

Only 6 of the 51 studies (12%) reported repeated screenings at 18 and 24 months, as is recommended by the AAP. Three of the repeated screening studies used concurrent false-negative case detection strategies,^22,46,61 1 study used a prospective false-negative case detection strategy,⁴¹ and 2 studies used prospective record review.^8,9 Three studies did not report enough information to allow for comparison of sensitivity for single compared with repeated screenings.^22,41,46 The remaining 3 studies demonstrated 11% to 45% higher sensitivity across repeated screenings compared with sensitivity based on a single screening, without decreasing specificity (eTable 6 in Supplement 1).

Discussion

The AAP identifies sensitivity and specificity above 70% to be acceptable for screening measures.⁶⁹ Estimates of these properties for the M-CHAT(-R/F) vary widely according to study methods and sample characteristics, and this variability affects use in both clinical and research settings. For example, as a result of 2 large prospective studies’ finding of low sensitivity for M-CHAT(-R/F),^8,9 a recent study⁷⁰ used an alternative screener that not only lacked long-term outcome data but also had tested the screener with only small samples, insufficient for thorough validation. Overall, across the studies identified in this systematic review and meta-analysis, sensitivity and specificity of the M-CHAT(-R/F) were found to be strong, with pooled values of 0.83 and 0.94, respectively. The variability of the estimates of sensitivity and specificity of M-CHAT(-R/F), however, highlights a need to consider factors that influence screening performance.

Case confirmation strategies used to identify missed cases were closely associated with sensitivity estimates. Weak concurrent strategies used to detect false-negative results likely inflated sensitivity compared with rigorous false-negative strategies better equipped to identify missed cases. Prospective strategies, on the other hand, likely conflate missed children who may not have had measurable symptoms during toddlerhood (ie, children whose ASD symptoms emerged later in childhood) with children truly missed by screening (who should have been detectable), potentially because of parents’ inaccurate report or their not being willing or ready to endorse an increased likelihood of ASD behaviors during screening.⁷¹ Studies show that for some children with ASD, symptoms are subtle early in development or show a prolonged course of symptom development,⁷² consistent with the theory that although brain development may be different before birth, measurable symptoms of neurodevelopmental disorders may emerge gradually because children are expected to demonstrate more sophisticated behavior as they grow older.⁷³ Although an advantage of reviewing medical records or registries lies in obtaining information over a range of ages, this broad age range may also be associated with ascertainment differences, given expected differences in medical record content for preschoolers compared with older children. Furthermore, prospective record review often includes community diagnoses that may not be as rigorous as diagnostic procedures in research⁷⁴; therefore, some children identified through record review would not meet more stringent research classification.

Another factor associated with variability in M-CHAT(-R/F)’s specificity is the classification of study samples based on ASD likelihood. The M-CHAT(-R/F) casts a broad net for children in need of expert differential diagnosis. Therefore, it is not surprising that specificity is lower for HL groups compared with LL samples because HL samples include many children with other developmental delays or co-occurring conditions. However, sensitivity—the ability to detect ASD when it is truly present—is equally high for both groups. The lower performance of specificity—the ability to identify individuals without ASD—supports the recommendation for comprehensive evaluation to assess differential diagnoses when M-CHAT(-R/F) is used, particularly in HL samples. Similarly, because the M-CHAT(-R/F) was validated for children between 16 and 30 months of age, it is not surprising that studies found slightly lower sensitivity for those older than 30 months compared with 30 months of age or younger, although M-CHAT(-R/F) has utility for children up to 48 months of age.

Repeated screening is also an important factor in maximizing sensitivity. However, the results of this systematic review suggest that repeated screening is extremely underused because only 6 of the 51 studies reported systematically screening toddlers more than once. The 3 studies for which data allowed direct comparison suggested a large increase in sensitivity with repeated vs single screening, without decreasing specificity. Similarly, other studies have shown that repeated screenings for ASD at 18 and 24 months of age increase the likelihood of identifying children missed by earlier screening,^75,76 and rescreening after 18 months of age detects children with ASD who initially screened negative.⁷⁷ In addition to symptom emergence detected among children at later ages, parents’ limited knowledge of typical developmental milestones, or of ASD-specific symptoms, may be associated with negative screening results at 18 months. For example, some children demonstrate symptoms of ASD at 18 months, even though they screen negative for ASD at that age.⁷⁸ These findings support the AAP’s recommendation for repeated screening at 18- and 24-month well-child visits; however, most studies did not adopt this recommendation in their study designs.

The studies included in the systematic review and meta-analysis differed in many aspects, including sample size, version of M-CHAT (original vs revised), whether the structured Follow-up was used appropriately for children whose initial scores were in the medium likelihood of ASD range, and language of M-CHAT(-R/F). Larger sample sizes resulted in higher DORs, possibly because of improved statistical stability in larger samples or because more resources were available in larger studies. Use of the Follow-up portion of the M-CHAT(-R/F) significantly improved the tool’s performance, with greatly increased specificity and a more than 7-fold increase in DOR, consistent with findings from the original validation studies of the M-CHAT(-R/F).^22,46 Even though use of the Follow-up does not change sensitivity (nor would it be expected to), it greatly reduces false-positive rates, which in turn reduces the burden on tertiary care clinics that receive referrals. This finding highlights the importance and benefit of the use of the Follow-up portion of the M-CHAT(-R/F) during screening, even though only slightly more than half of the identified studies administered the Follow-up consistently. The structured Follow-up clarifies parents’ endorsements during initial screening, giving them the opportunity to explain behaviors beyond dichotomous response options, which improves accuracy. The improved performance of the M-CHAT in non-English languages is interesting. It is unclear whether other factors may be associated with this finding. For example, non-English administration of M-CHAT(-R/F) appears to occur more often in HL studies. These potential interactions between variables need to be explored in future studies. In addition to the factors explored in the present meta-analysis, future factors to explore include parents’ prior knowledge of ASD, which may be more common in HL compared with LL samples, or parent education, which may account for some of the findings.

Limitations

A limitation of this study is the methodological issues that were identified in a majority of the studies, which could have biased reported accuracy measures. In particular, many studies with LL samples did not evaluate every child who was screened with the M-CHAT(-R/F), likely because of obvious feasibility challenges. Children who screened negative and were not evaluated were therefore presumed to have true-negative results for analyses, but it is possible that some cases were missed. Similarly, variation in the report of ASD diagnostic criteria and the type of clinicians who performed assessments were associated with between-study heterogeneity. In addition, the present systematic review was limited to studies published in English-language peer-reviewed journals.

Conclusions

When the M-CHAT(-R/F)’s utility in detecting toddlers at greater likelihood for ASD is evaluated, it is critical to consider study strategies, age of the children, and ASD likelihood status of the children in addition to other factors that tend to vary widely between the existing studies. Although the AAP recommends screening at both 18- and 24-month visits and studies emphasize the added value of repeated screenings, very few studies consistently rescreened participants, highlighting the need for continued effort in dissemination of best practice screening protocols. The results of this systematic review and meta-analysis illuminate important clinical implications for pediatric physicians. Even though no single measure will identify all children at increased likelihood for ASD, M-CHAT(-R/F) shows strong overall sensitivity and specificity, and research with the tool indicates that screening early and at multiple time points is critical to identify children at increased likelihood for ASD who are in need of access to ASD-specific early intervention services.

Overall, the results of this systematic review and meta-analysis highlight strong sensitivity and specificity for the M-CHAT(-R/F) across the 51 study samples—critical information for clinicians, researchers, and policy makers alike. However, the wide variability, ranging from poor to excellent screening metrics, highlights the differences between screener use, which should be considered when studies are designed. Critically, although the version of M-CHAT (original vs revised) does not significantly affect sensitivity and specificity, use of the Follow-up portion of the M-CHAT(-R/F) significantly reduces false-positive rates, which in turn reduces the burden on diagnostic and intervention systems. Other guidance based on this study’s findings includes emphasizing the importance of referral for comprehensive evaluations to discern symptoms of autism from symptoms observed in other developmental disorders, particularly when M-CHAT(-R/F) is used with HL populations. In addition, the difference in M-CHAT(-R/F)’s sensitivity in concurrent vs prospective studies indicates the need to account for timing and diagnostic rigor when designing studies.

Supplement 1.

eMethods. Diagnostic Accuracy of the M-CHAT(-R/F)

eTable 1. Database Search Terms

eTable 2. QUADAS-2 Description and Adapted Signaling Questions

eTable 3. Quality Assessment of Studies Included in the Systematic Review

eTable 4. Additional Study Characteristics and Psychometric Properties for M-CHAT(-R/F)

eTable 5. Sensitivity and Specificity for Younger and Older Samples

eTable 6. Sensitivity and Specificity for Single and Repeated Screening

eFigure 1. Study Selection Flow Chart Following PRISMA Guidelines

eFigure 2. Overall SROC of M-CHAT(-R/F) (n = 49 Studies)

eFigure 3. SROC Plot of M-CHAT(-R/F) by M-CHAT Version (M-CHAT n = 31, M-CHAT-R n = 18)

eFigure 4. SROC Plot of M-CHAT(-R/F) by Likelihood Level of Sample (Low Likelihood n = 27, High Likelihood n = 22)

eFigure 5. SROC Plot of M-CHAT(-R/F) by Case Confirmation Strategy (Concurrent n = 40, Prospective n = 9)

eFigure 6. SROC Plot of M-CHAT(-R/F) by Sample Size (Small n = 20, Medium n = 17, Large n = 12)

eFigure 7. SROC Plot of M-CHAT(-R/F) With Follow-up vs Initial Only (Initial n = 22, Follow-up n = 27)

eFigure 8. SROC Plot of M-CHAT(-R/F) by Language (English/Primarily English n = 26, Other Language n = 23)

Supplement 2.

Data Sharing Statement