The Early Screening for Autism and Communication Disorders (ESAC): Field-Testing an Autism-Specific Screening Tool for Children 12 to 36 Months of Age

Abstract

There is a critical need for validated screening tools for autism spectrum disorder (ASD) in very young children so families can access tailored intervention services as early as possible. Few screeners exist for children between the recommended screening ages of 18–24 months. This study examined the utility of a new autism-specific parent-report screening tool, the Early Screening for Autism and Communication Disorders (ESAC) for children 12–36 months. Field-testing was conducted from five sites with 471 children screened for communication delays in primary care or referred for familial risk or concern for ASD. The ESAC was evaluated in three age groups: 12–17, 18–23, and 24–36 months. A best estimate diagnosis of ASD, developmental delay, or typical development was made. Receiver operating characteristic curves were examined for all 46 items and the 30 items that best discriminated ASD from the non-spectrum groups. Area under the curve estimates for the total were greater than .90 across age groups. Cutoffs were established for each age group with sensitivity between 0.86–0.92 and specificity between 0.74–0.85. Results provide preliminary support for the validity of the ESAC as an autism-specific screener in children 12–36 months with elevated risk of communication delay or ASD.

Lay Abstract

There is a critical need for accurate screening tools for autism spectrum disorder (ASD) in very young children so families can access tailored intervention services as early as possible. However, there are few screeners designed for children 18–24 months. Developing screeners that pick up on the signs of ASD in very young children has proved even more challenging. In this study, we examined a new autism-specific parent-report screening tool, the Early Screening for Autism and Communication Disorders (ESAC) for children between 12 and 36 months of age. Field-testing was done in five sites with 471 children screened for communication delays in primary care or referred for familial risk or concern for ASD. The ESAC was tested in three age groups: 12–17, 18–23, and 24–36 months. A best estimate diagnosis of ASD, developmental delay, or typical development was made. Analyses examined all 46 items and identified 30 items that best discriminated ASD from the non-spectrum groups. Cutoffs were established for each age group with good sensitivity and specificity. Results provide preliminary support for the accuracy of the ESAC as an autism-specific screener in children 12–36 months with elevated risk of communication delay or ASD.

Advances in research have documented reliable diagnoses of autism spectrum disorder (ASD) can be made between 14 and 24 months of age (Guthrie et al., 2013; Ozonoff et al., 2015; Pierce et al., 2019; Zwaigenbaum, Bauman, Stone et al., 2015). Since 2007, the American Academy of Pediatrics (AAP) has recommended screening all children for ASD at 18 and 24 months (Johnson & Myers, 2007), and recently reaffirmed these recommendations (Hyman et al., 2020). Yet, the median age of diagnosis in the U.S. has hovered at 4–5 years for more than a decade (Maenner et al., 2020). Mounting research has shed light on limitations of available autism screeners (Carbone et al., 2020; Guthrie et al., 2019; Stenberg et al., 2014; Sturner et al., 2017a; Yuen et al., 2018), one factor contributing to late diagnosis. There is a critical need for screeners to improve early detection. This paper reports on field-testing of a new autism screener for children 12–36 months with elevated risk of communication delay or ASD.

Limitations of Existing Screening Approaches and Tools

The Modified Checklist for Autism in Toddlers (M-CHAT) or M-CHAT-Revised (M-CHAT-R; Robins et al., 2001; 2014) is the most widely studied and utilized autism screener for toddlers. Reported sensitivity and specificity for this screener may be overestimated because, for many studies, only children receiving positive screens completed diagnostic evaluations (Zwaigenbaum, Bauman, Fein et al., 2015). Studies by Robins and colleagues have reported estimated sensitivities over 0.80 (Chlebowski et al., 2013; Kleinman et al., 2008; Robins et al., 2001, 2014) based on screening children referred for suspected ASD, receiving EI, or with familial risk as well as in primary care. However, more recent large studies conducted in primary care have shown limitations in the performance of the M-CHAT/M-CHAT-R (Carbone et al., 2020; Guthrie et al., 2019; Stenberg et al., 2014, 2020; Sturner et al., 2017a, 2017b).

Robins et al. (2014) concluded that the M-CHAT should be used with the Follow-Up interview (i.e., M-CHAT/F) to reduce false positives and avoid unnecessary referrals, as positive predictive value (PPV) increased from 0.36 to 0.74 with the Follow-Up interview. However, in a review of electronic medical records of universal screening using the M-CHAT/F between 16–24 months in a large pediatric practice (N=25,999), the M-CHAT/F showed substantially lower sensitivity (0.39; Guthrie et al., 2019), suggesting that this tool may miss more children with ASD than it detects. Similar findings were reported in a larger primary care sample using the M-CHAT (Carbone et al., 2020). The age between screening and diagnosis was much greater than in the Robins et al. 2014, which may partially explain poorer estimates of accuracy. Furthermore, in both studies, children with positive screens were diagnosed earlier than those with a negative screen or no screen, suggesting that screening continues to be important.

Research characterizing children accurately identified by the M-CHAT/M-CHAT/R has indicated that developmental level and age are important factors. Robins et al. (2014) reported that their ASD sample had below average developmental levels, with standard scores approximately two standard deviations below the mean. Results of a meta-analysis of 19 studies comparing outcomes of 1,658 children with ASD ascertained using the M-CHAT indicated that developmental level was significantly higher in infant sibling samples (M=81.25), than both community referral (M=62.71) and universal screening samples (M=60.91), providing evidence of a significant bias towards detecting children with lower developmental level (Micheletti et al., 2019).

Research on the M-CHAT/M-CHAT-R in primary care suggests it performs better in children 24–36 months compared to children under 24 months (Guthrie et al., 2019; Pandey et al., 2008; Sturner et al., 2017a; Toh et al., 2018; Yuen et al., 2018). For example, Guthrie et al. (2019) reported a sensitivity of 0.35 for children 16–20 months, compared to 0.49 for children 21–26 months. With regard to developmental level, in another population study, Stenberg et al. (2020) reported that at 18 months of age, the M-CHAT identified only 28.8% of children with ASD and identified 81.3% of children with intellectual disability without ASD. Collectively, these findings indicate that, in general population samples, the M-CHAT may be missing far more children with ASD than it is detecting at 18–24 months and identifying more children with developmental delays without ASD, highlighting the need for improved screeners at younger ages and for primary care settings.

The new AAP clinical report (Hyman et al., 2020) identified two promising autism screening tools for children 18–30 months of age, the Parents Observations of Social Interactions (POSI; Smith et al., 2012) and the Early Screening for Autism and Communication Disorders (ESAC). The POSI is a brief questionnaire been studied with preliminary community-referred samples, showing favorable comparison on sensitivity, but lower performance on specificity than the M-CHAT (Salisbury et al., 2018; Smith et al., 2012). Additional validation in larger samples is needed to understand psychometric properties and clinical utility of the POSI. The ESAC is a parent-report tool developed with an initial set of 46 items to capture the heterogeneity of delayed social communication skills and early signs of autism symptoms. Preliminary research on the development of ESAC items revealed a group difference with a large effect (g=1.15) in ESAC Total scores between children with and without ASD, who were administered the ESAC at 18–36 months (Stronach & Wetherby, 2017). Research studying the psychometric performance of the ESAC in preliminary community-referred and population samples is needed.

Accurately screening for ASD younger than 18 months is even more challenging. Research on the Early Screening of Autistic Traits Questionnaire (Dietz et al., 2006), designed to screen at 14–15 months in primary care, indicated a detection rate of 0.6/1,000. Similarly, the First Year Inventory for 12-month-olds (Reznick et al., 2007) demonstrated a sensitivity of 0.44 (Turner-Brown et al., 2013). While these screeners show promise, the low accuracy does not yet support clinical use.

Most children with ASD exhibit observable symptoms by 12–24 months (Bacon et al., 2018; Pierce et al., 2019; Zwaigenbaum, Bauman, Stone, et al., 2015), suggesting the potential of screening under 24 months. Existing tools may have limited utility capturing the heterogeneity of early behavioral manifestations of ASD, including symptoms in restricted, repetitive behaviors (RRBs) observed at or before 12–24 months (Dow et al., 2017; 2020; Elison et al., 2014; Kim & Lord, 2010; Richler et al., 2007; Watt et al., 2008; Wolff et al., 2014). The accuracy of autism screeners may be improved by incorporating questions capturing the range of ASD symptoms in young children.

Purpose of This Study

The ESAC addresses the need for a more accurate and comprehensive parent-report screening tool by incorporating questions about early social communication and the presence of RRBs, based on core diagnostic features of ASD (American Psychiatric Association, 2013). This study reports the performance of the ESAC during field-testing with children 12–36 months of age recruited from five sites using varied ascertainment sources. We recruited children with parental or professional concerns for ASD or developmental delay, familial-risk, or positive primary care screening for communication delay yielding a diverse sample of children with elevated risk for ASD. The first aim of this study was to examine item-level performance in order to identify the items that best differentiated these children with ASD and without ASD. The second aim was to examine the preliminary psychometric properties of the best set of ESAC items and performance across three age groups in this elevated-risk sample: 12–17, 18–23, and 24–36 months.

Methods

Procedures for Sample Selection and Recruitment

Participants were 471 toddlers 12–36 months of age, recruited from five sites: Florida State University (FSU; n=299), University of California, San Diego (UCSD; n=66), National Institute of Mental Health (NIMH; n=49), University of Michigan Autism and Communication Disorders Center (UM; n=45), and University of California, Davis (UCD; n=12). Children at FSU and UCSD were referred from primary care screening for communication delay using the Infant-Toddler Checklist (ITC; Wetherby & Prizant, 2002). Children with positive ITC screens, negative ITC screens with parental concern noted, and randomly selected negative ITC screens without parent concern (about 20% of FSU sample) were recruited for this study. Children at NIMH and UM were referred for diagnostic evaluations because of parental or professional concern about possible ASD. Children at UCD were referred because they had familial risk due to having an older sibling with ASD. All children participated in studies approved by institutional review boards at each site, and parents provided informed consent.

Parents were asked to complete the ESAC prior to or at the time of the developmental or diagnostic evaluation, before most families received the diagnosis. Only 9 ESACs were completed more than a month after the autism diagnostic evaluation. Some children were seen longitudinally, resulting in multiple ESACs. The earliest ESAC within each age group was chosen, yielding 596 ESACs (12–17 months: n=159; 18–23 months: n=187; 24–36 months: n=250). At each site, a clinical best-estimate diagnosis of ASD (n=183) or non-spectrum (NS; n=288) was made by experienced clinicians using all available information. The NS group included children with developmental delay (DD; n = 77) and typical development (TD; n = 211). All children showing signs of ASD completed the Autism Diagnostic Observation Schedule (ADOS; Lord, Luyster, et al., 2012; Lord, Rutter, et al., 2012; Lord et al., 2000) and Mullen Scales of Early Learning (MSEL; Mullen, 1995) for diagnosis or rule out. All children with non-spectrum DD completed the MSEL, and 53% completed the ADOS. Most children with DD had one or more scores on the MSEL below the 10^th percentile (i.e., T score ≤ 37). There were 30 children across all sites in the DD group who had significant developmental or psychological concerns in domains not captured by the Mullen, including highly unintelligible speech or high anxiety. In these cases, classification of DD was based on clinical judgement. For children with TD, 74% received the MSEL and 22% also received the ADOS. The remaining 26% of the TD group were from the FSU site and received the Communication and Symbolic Behavior Scales (CSBS; Wetherby & Prizant, 2002) Behavior Sample, a standardized norm-referenced communication evaluation. Children assigned to the TD group had standard scores above the 10^th percentile on the MSEL or CSBS and/or had no signs of delay based on clinical judgement.

Sample demographic and developmental characteristics are summarized in Table 1. Sex was reported for all but three children. Race and ethnicity were available for 73% (n=343) of the sample. Maternal education (72%; n=338) served as a proxy for socioeconomic status. The sample was primarily male and white, with most mothers having a college education.

Table 1.

Sample Characteristics by Diagnostic Group

	ASD (n = 183)		DD (n = 77)		TD (n = 211)		NS (n = 288)
Characteristic	M	(SD)	M	(SD)	M	(SD)	M	(SD)
Male, n (%)	156	(85)	55	(71)	121	(57)	176	(61)
Race, n (%)
White	97	(53)	35	(46)	145	(69)	180	(63)
Black	12	(7)	8	(10)	16	(8)	24	(8)
Asian	2	(1)	2	(3)	6	(3)	8	(3)
Pacific Islander	0	(0)	0	(0)	1	(<1)	1	(<1)
Multiracial	7	(4)	6	(8)	6	(3)	12	(4)
NR	65	(36)	26	(34)	37	(18)	62	(22)
Ethnicity, n (%)
Hispanic	11	(6)	5	(7)	14	(7)	19	(7)
Maternal education (years)	15.57	(2.28)	16.18	(2.27)	15.84	(2.20)	15.91	(2.21)
ADOS, n, (%)	183	(100)	41	(53)	47	(22)	88	(31)
Age (months)	28.79	(7.42)	29.19	(7.45)	26.20	(7.35)	27.59	(7.51)
SA CSS	6.24	(2.11)	2.39	(143)	1.79	(116)	2.07	(132)
RRB CSS	6.89	(2.32)	4.00	(2.32)	2.62	(2.20)	3.26	(2.35)
Total CSS	6.37	(2.07)	2.15	(122)	1.60	(0.93)	1.85	(110)
MSELa, n (%)	183	(100)	77	(100)	157	(74)	234	(81)
Age (months)	27.97	(7.34)	28.66	(6.74)	27.89	(6.77)	28.15	(6.74)
VR	40.12	(14.12)	46.29	(12.00)	60.46	(10.52)	55.76	(12.88)
FM	38.02	(13.11)	43.64	(11.83)	54.55	(10.73)	50.93	(12.22)
RL	32.63	(14.01)	42.53	(13.02)	57.10	(9.85)	52.27	(12.95)
EL	33.70	(13.17)	37.64	(1111)	54.88	(11.46)	49.16	(13.94)
ELC	75.10	(20.77)	86.19	(16.51)	113.36	(15.80)	104.34	(20.50)
Nonverbal DQ	87.11	(19.99)	97.04	(18.25)	116.81	(16.18)	110.31	(19.25)
Verbal DQ	69.40	(28.59)	85.34	(22.07)	116.63	(20.93)	106.33	(25.87)

ASD = autism spectrum disorder; DD = developmentally delayed; TD = typically developing; NS = non-spectrum; NR = not reported; ADOS = Autism Diagnostic Observation Schedule; SA= Social Affect; CSS = calibrated severity score; RRB = restrictive/repetitive behavior; MSEL = Mullen Scales of Early Learning; VR = Visual Recognition; FM = Fine Motor; RL = Receptive Language; EL = Expressive Language; ELC = Early Learning Composite; DQ = developmental quotient;

Note. NS groups includes DD and TD children. MSEL T-scores based on a M=50, SD=10; ELC based on M=100 and SD=15.

^aMSEL subscale and composite scores were not reported for n = 7, DQs were provided.

Measures

Early Screening for Autism and Communication Disorders

The ESAC was initially developed as an autism screener to follow up broadband screeners like the ITC, an AAP-recommended screener (Johnson et al., 2007; Hyman et al., 2020) for communication delays including ASD. The ESAC field-test version included 46 items generated by three authors (AMW, JW, and CL) covering ASD symptom domains (APA, 2013) to measure lack of typical social communication development and presence of RRBs. The majority of items (35) were rated on a three-point scale as occurring Often, Sometimes, or Not Yet/Rarely/Never. Higher scores indicated greater symptom severity (i.e., absence of typical skills or presence of unusual behaviors). Eight of the remaining 11 items comprised lists of specific behaviors that parents indicated as present/absent or rated on a three-point scale. Responses for these 8 items were summed to create an item-level score. Two items asked about number of words and word loss but were not included in the total score.

Mullen Scales of Early Learning

The MSEL (Mullen, 1995) is a standardized assessment of developmental level. Standard T-scores for Visual Reception, Fine Motor, Expressive and Receptive Language subscales were obtained. Because nearly 25% of children received one or more T-scores of 20 (i.e., the floor), developmental quotients (DQs) were calculated. DQ was derived by averaging Visual Reception and Fine Motor age equivalents for Nonverbal DQ and Receptive and Expressive Language age equivalents for Verbal DQ, and then dividing each by chronological age and multiplying by 100.

Autism Diagnostic Observation Schedule

The ADOS is a semi-structured, standardized assessment that measures symptoms of ASD and provides domain and total scores. Children received different modules based on age and verbal ability. To allow comparison of scores across modules, calibrated severity scores were derived (Esler et al., 2015; Gotham et al., 2009; Hus et al., 2014).

Statistical Analyses

Receiver operating characteristic (ROC) curves, based on logistic regressions, were estimated to generate area under the curve (AUC) values and examine performance of each item in discriminating between the ASD and NS groups.

For items with derived sums, ROC analysis results were used to determine item-level cutoffs that would convert scores into the same three-point rating scale. Criteria for assigning a “2” included a specificity value as close to 0.90 as possible and sensitivity ≥0.60. The cutoff for a “1” was based on optimal sensitivity and specificity (i.e., as close to 0.80 as possible). Cutoff values for these items were examined separately across age groups.

Using the best performing items, ROC curves were estimated with the sum of retained items creating an ESAC total to generate AUC and sensitivity/specificity values. Curves were estimated separately across age groups to compare optimal cutoffs and estimate PPVs and negative predictive values (NPVs). Internal consistency of the ESAC was estimated using coefficient alpha. Test-retest reliability was approximated for the subsample with two ESAC administrations (n = 111).

Community Involvement

There was no community involvement in this study.

Results

Site Comparisons of Children with ASD

The 121 children recruited from primary care samples (FSU, UCSD) were significantly younger than the 62 recruited from referred samples (NIMH, UM, UCD; p < .001) at the time of the first ESAC and the ADOS (p < .01) and MSEL (p < .05). However, comparisons across sites showed nonsignificant differences for ADOS Total CSS and MSEL scores (all p >.05).

Item-Level Analyses

Item-level results are presented in Table 2. Twenty items had AUC values less than 0.65. We chose to retain four of these items because they captured the heterogeneity of clinically relevant RRBs (items 32, 41, 44, 46; endorsing these items likely indicates clinical concern for ASD but frequency of endorsement is low). A total of 30 items were retained. Cutoffs, sensitivity, and specificity values for the seven continuous items are presented in Table 3. Because these items had different ranges (see Table 2), the values that met the criteria outlined above for a “1” and “2” varied across items and age. Different cutoff values were identified across age groups for three items (21, 31, 37); cutoffs did not vary by age for the remaining items.

Table 2.

Area Under the Curve and Item-level Information Across Diagnostic Groups

Item	Item Content	AUC	ASD M (SD)	Non-spectrum M (SD)	Score Range
Q1	Clearly requests	0.66	0.58 (0.70)	0.16 (0.44)	0–2
Q2	Express happiness	0.59	0.50 (0.92)	0.17 (0.57)	0–4
Q3	Enjoy social touches/affection	0.59	0.25 (0.49)	0.04 (0.22)	0–2
Q4	Directed smiling/laughing	0.62	0.33 (0.52)	0.05 (0.23)	0–2
Q5	Response to name	0.78	0.76 (0.55)	0.15 (0.36)	0–2
Q6	Response to voice	0.76	0.94 (0.62)	0.31 (0.49)	0–2
Q7	“Checking in” eye gaze	0.73	1.09 (0.65)	0.46 (0.53)	0–2
Q8	Eye gaze to get attention	0.73	0.87 (0.67)	0.29 (0.48)	0–2
Q9	Response to directions with gestures	0.75	0.90 (0.69)	0.25 (0.46)	0–2
Q10	Response to joint attention	0.76	0.98 (0.72)	0.25 (0.48)	0–2
Q11	Ease of interactions	0.72	0.62 (0.66)	0.09 (0.30)	0–2
Q12	Emotional dysregulation	0.64	1.65 (1.48)	0.94 (1.22)	0–6
Q13	Calming behavior	0.64	0.54 (0.58)	0.27 (0.51)	0–2
Q14	Initiation of communication	0.73	0.81 (0.75)	0.16 (0.40)	0–2
Q15	Response to ignore	0.54	1.11 (0.76)	1.01 (0.65)	0–2
Q16	Attention-getting behavior	0.59	0.69 (0.67)	0.47 (0.54)	0–2
Ql7	Initiation ofjoint attention for social purposes	0.75	1.30 (0.70)	0.57 (0.63)	0–2
Ql8	Drawing attention	0.79	1.40 (0.67)	0.53 (0.67)	0–2
Ql9	“Uh oh” response	0.74	1.23 (0.84)	0.40 (0.68)	0–2
Q20	Hand as tool	0.48	0.89 (0.86)	0.88 (0.80)	0–2
Q21	Gestures	0.84	7.53 (3.77)	2.54 (2.71)	0–14
Q22	Integration of gestures, sounds, and eye gaze	0.75	1.11 (0.83)	0.32 (0.59)	0–2
Q23	Unusual sounds	0.58	1.10 (0.80)	0.84 (0.83)	0–2
Q24	Imitation of gestures, sounds, and words	0.66	0.86 (0.68)	0.41 (0.54)	0–2
Q25	Unusual intonation	0.59	0.33 (0.58)	0.11 (0.38)	0–2
Q26	Number of Words	0.38	21.37 (33.46)	28.67 (35.80)	0–200
Q27	Loss of words	0.60	0.55 (0.90)	0.16 (0.54)	0–1
Q28	Stereotyped phrases	0.49	0.30 (0.64)	0.28 (0.53)	0–2
Q29	Unusual phrases	0.49	0.31 (0.65)	0.27 (0.52)	0–2
Q30	Back and forth conversation	0.68	1.68 (0.61)	1.09 (0.84)	0–2
Q31	Play with objects	0.75	3.47 (2.97)	1.23 (1.98)	0–12
Q32	Play with a variety of toys	0.63	0.41 (0.62)	0.08 (0.29)	0–2
Q33	Focus on objects rather than people	0.78	1.24 (0.72)	0.42 (0.59)	0–2
Q34	Intense focus on a certain toy	0.72	0.85 (0.78)	0.26 (0.53)	0–2
Q35	Sticky attention to a certain toy	0.75	0.91 (0.68)	0.28 (0.49)	0–2
Q36	Pretend play	0.70	1.30 (0.68)	0.76 (0.66)	0–2
Q37	Pretend play with toys	0.74	5.95 (2.06)	3.95 (2.32)	0–8
Q38	Pretend play with two sequenced actions	0.66	1.80 (0.48)	1.35 (0.74)	0–2
Q39	Attachment to unusual object	0.59	0.47 (0.78)	0.17 (0.46)	0–2
Q40	Interest in other children own age	0.76	1.14 (0.73)	0.39 (0.60)	0–2
Q41	Sensory responses	0.64	1.06 (1.29)	0.42 (0.80)	0–8
Q42	Repetitive behaviors/body mannerisms	0.74	1.61 (1.41)	0.52 (0.91)	0–6
Q43	Insistence on sameness	0.71	1.85 (1.50)	0.79 (1.09)	0–6
Q44	Rituals and routines	0.59	0.36 (0.61)	0.12 (0.35)	0–2
Q45	Unusual vocabulary	0.54	0.26 (0.67)	0.07 (0.37)	0–2
Q46	Intense interests	0.63	0.84 (1.33)	0.23 (0.66)	0–5

AUC= area under curve; ASD = autism spectrum disorder

Note. Bolded items were retained for the final item set

Table 3.

Cutoff Values for Retained Continuous Items based on Receiver Operating Characteristic Analysis

		Score of “2” Cutoff			Score of “1” Cutoff
Item	Agey Group (months)	Cutoff Value	Sensitivity	Specificity	Cutoff Value	Sensitivity	Specificity
	12–17	9	0.63	0.93	6	0.79	0.72
Q21	18–23	7	0.61	0.93	4	0.81	0.76
	24–36	5	0.70	0.91	3	0.85	0.82
	12–17	6	0.56	0.89	4	0.81	0.73
Q31	18–23	3	0.63	0.93	2	0.78	0.82
	24–36	2	0.49	0.90	1	0.70	0.84
	12–17	8	0.54	0.80	7	0.67	0.72
Q37	18–23	7	0.52	0.89	5	0.79	0.62
	24–36	6	0.52	0.86	4	0.82	0.63
	12–17	2	0.21	0.96	1	0.40	0.78
Q41	18–23	2	0.22	0.95	1	0.49	0.66
	24–36	2	0.35	0.88	1	0.65	0.66
	12–17	2	0.49	0.90	1	0.63	0.72
Q42	18–23	2	0.52	0.87	1	0.66	0.64
	24–36	2	0.47	0.89	1	0.79	0.63
	12–17	2	0.42	0.86	1	0.61	0.70
Q43	18–23	2	0.48	0.73	1	0.73	0.45
	24–36	2	0.65	0.70	1	0.88	0.54
	12–17	1	0.26	0.92	--	--	--
Q46	18–23	1	0.24	0.85	--	--	--
	24–36	1	0.54	0.80	--	--	--

Psychometric Properties of Final ESAC Item Set

ROC curves showed high AUC values, indicating the ESAC performed consistently well across age groups (Table 4); however, optimal cutoffs differed by age group. Within the 12–17-month subsample a cutoff of 20 yielded balanced sensitivity (0.86) and specificity (0.84). The cutoff of 18 yielded the strongest sensitivity (0.87–0.88) and specificity (0.84–0.85) estimates for both the 18–23- and 24–36-month subsamples. Estimated PPVs were highest for the oldest age group and NPVs were comparable across ages (Table 4).

Table 4.

ESAC Total Score Receiver Operating Characteristic Analysis Results

	Optimal Cutoff	AUC	[95% CI]	Sensitivity	Specificity	PPV	NPV
12–17 months (n = 159)	20	0.91	[0.86, 0.96]	0.86	0.82	0.64	0.94
18–23 months (n = 187)	18	0.93	[0.89, 0.97]	0.87	0.85	0.76	0.92
24–36 months (n = 249)	18	0.93	[0.89, 0.96]	0.88	0.84	0.81	0.90

AUC = area under curve; CI = confidence interval; PPV = positive predictive value; NPV = negative predictive value

Developmental Characteristics by Screening Status

ESAC Totals and developmental characteristics by screening status across age groups are reported in Table 5. A majority of children with ASD screened positive on the ESAC across age groups. Children with false negatives demonstrated a pattern of higher MSEL scores on average, compared to the children with true positives. Across age groups, there were only 28 false negatives of the 596 screens, and most of these were from the primary-care sample. Children with true negatives demonstrated a pattern of higher MSEL scores on average than children with false positives. Across age groups, 52%-77% of the children with false positives had a classification of DD. Furthermore, the average time between screening and diagnosis based on the ADOS decreased with age (12–17-months: n=70, M=8.85 months SD=8.73; 18–23-months: n=97, M=6.40 months SD=7.32; 24–36-months: n=160, M=1.68 months SD=3.76).

Table 5.

Developmental Characteristics by Screening Status

	True Positive		False Positive		False Negative		True Negative
	M	SD	M	SD	M	SD	M	SD
12–17 months	n = 37		n = 21		n = 6		n = 95
ESAC Score	33.62	(8.93)	26.00	(5.71)	15.00	(2.61)	11.19	(4.26)
ADOS, n (%)	37	(100)	11	(52)	6	(100)	16	(17)
SA CSS	6.57	(2.05)	2.82	(1.72)	4.50	(2.07)	1.94	(139)
RRB CSS	6.03	(2.54)	3.45	(2.42)	5.83	(2.99)	2.38	(2.13)
Total CSS	6.46	(2.28)	2.36	(112)	4.33	(2.07)	1.75	(118)
MSEL, n (%)	35	(95)	20	(95)	6	(100)	60	(63)
VR	43.06	(13.62)	50.75	(11.03)	56.00	(19.39)	60.07	(11.22)
FM	41.40	(12.30)	48.50	(10.92)	46.00	(16.08)	54.12	(10.74)
RL	33.34	(14.13)	49.10	(9.77)	50.83	(17.88)	56.65	(11.91)
EL	31.40	(10.44)	45.40	(10.55)	49.83	(20.70)	52.93	(13.43)
ELC	77.00	(18.29)	97.53	(15.45)	102.67	(30.45)	111.87	(18.23)
18–23 months	n = 58		n = 18		n = 9		n = 102
ESAC Score	33.83	(9.27)	24.00	(7.64)	13.67	(3.16)	9.71	(4.01)
ADOS, n (%)	58	(100)	9	(50)	9	(100)	21	(21)
SA CSS	6.67	(198)	2.89	(117)	5.67	(194)	2.33	(143)
RRB CSS	6.57	(2.39)	4.33	(194)	6.89	(145)	2.48	(2.18)
Total CSS	6.66	(188)	2.33	(100)	5.67	(2.00)	1.90	(100)
MSEL, n (%)	56	(97)	18	(100)	9	(100)	85	(83)
VR	39.55	(13.45)	48.44	(9.84)	48.56	(17.05)	58.93	(1157)
FM	39.98	(12.93)	46.06	(9.97)	42.78	(7.97)	53.60	(11.00)
RL	31.25	(12.93)	38.22	(15.06)	42.33	(13.26)	55.74	(10.29)
EL	33.32	(13.38)	38.83	(12.68)	41.11	(11.20)	51.19	(13.34)
ELC	74.91	(19.43)	86.94	(18.83)	88.22	(19.20)	109.59	(16.63)
24–36 months	n = 96		n = 22		n = 13		n = 118
ESAC Score	33.97	(9.30)	27.91	(7.75)	14.15	(3.05)	7.36	(4.71)
ADOS, n (%)	96	(100)	14	(64)	13	(100)	36	(31)
SA CSS	6.30	(2.17)	2.00	(104)	5.77	(174)	2.14	(146)
RRB CSS	7.54	(2.15)	4.43	(2.20)	8.15	(1.41)	3.81	(2.20)
Total CSS	6.65	(2.13)	2.07	(133)	6.38	(145)	1.89	(114)
MSEL, n (%)	93	(97)	22	(100)	13	(100)	94	(80)
VR	35.80	(12.93)	40.64	(15.23)	42.62	(16.46)	57.32	(12.51)
FM	34.06	(12.18)	38.55	(12.39)	38.69	(17.99)	51.72	(12.64)
RL	28.77	(11.89)	39.86	(15.04)	42.23	(15.36)	53.22	(11.46)
EL	30.30	(11.71)	37.64	(14.27)	43.69	(13.51)	50.20	(13.22)
ELC	68.41	(18.16)	81.05	(21.99)	84.62	(26.78)	106.15	(19.69)

ADOS = Autism Diagnostic Observation Schedule; SA= Social Affect; CSS = calibrated severity score; RRB = restrictive/repetitive behavior; MSEL = Mullen Scales of Early Learning; VR = Visual Recognition; FM = Fine Motor; RL = Receptive Language; EL = Expressive Language; ELC = Early Learning Composite; DQ = developmental quotient;

Note. MSEL T-scores based on a M=50, SD=10; ELC based on M=100 and SD=15.

Internal Consistency and Test-retest Reliability

Coefficient alphas for the final ESAC item set ranged from 0.92–0.95, indicating good internal consistency. For the subsample with ≥ two administrations (n=111), we observed a correlation of 0.82 between the first and second ESAC total with an average of 8.37 months (SD = 3.49) between administrations, indicating a strong association between scores over time.

Discussion

This study examined ESAC results from preliminary field-testing of children with elevated risk for ASD recruited from primary care and referred samples. ROC curves were examined to determine the items that best discriminated children with and without ASD. AUCs exceeded 0.90 for total scores of the best 30 items, and preliminary cutoffs were established for the 30-item ESAC across age groups with promising sensitivity and specificity. A cutoff score was recommended at 20 for 12–17 months and 18 for 18–36 months. Results provide initial support for validity of the ESAC as an autism screener for children 12–36 months with elevated risk, accurately identifying over 80% of children with ASD in this sample. The ESAC had strong psychometric properties in the AAP-recommended screening age range of 18–24 months, as well as adequate sensitivity down to 12 months, which is particularly promising given challenges of accurately screening for ASD under 24 months.

The ESAC was more accurate in a sample with a higher developmental level by 18 points on the MSEL Early Learning Composite (ELC), than children with ASD identified through community-referrals as well as universal screening with the M-CHAT (see Micheletti et al., 2019). Likewise, MSEL average T-scores of children with ASD were higher by at least 5 points than samples detected with the M-CHAT (Kleinman et al., 2008) and M-CHAT/F (Robins et al., 2014). The average MSEL ELC for children detected with ASD in this field test sample was 22 points higher than the average of the Bayley Mental Development Index (Bayley, 1993) for children with ASD in the initial study of the M-CHAT in primary care (Robins et al., 2001). Furthermore, Kleinman and colleagues found that their primary care and referred ASD groups had similar scores on the MSEL, suggesting that even in a primary care group, the M-CHAT is biased towards identifying children with lower developmental levels. The percentage of children with ASD with average or above-average intellectual abilities at 8 years of age has increased over time to more than half (Maenner et al., 2020), highlighting the importance of detecting children with higher developmental levels.

This study extends the findings of Wetherby et al. (2008) and Pierce et al. (2011) by documenting the accuracy of the ESAC as an autism-specific parent-report screener to follow-up a broadband screen. Over 75% of our sample was recruited using the ITC in primary care at two sites (FSU and UCSD) and were younger, on average, at age of screening at these sites. Sensitivity and specificity of the ESAC were better than psychometric properties reported in early validation studies of the M-CHAT (Kleinman et al., 2008; Robins et al., 2001), which included children screened in primary care and those referred through EI. Importantly, our sample included a large number of children screened under 24 months. Most M-CHATs in Robins and colleagues’ (2001) early study were administered to children at 18- or 24-month well visits, though they did not report the mean age at screening. Further, only children who were detected by their M-CHAT score or by the primary care provider concerns were invited for a diagnostic evaluation; therefore, the actual sensitivity is likely to be lower than what was reported, given that children who screened negative and did not have provider concerns were assumed to not have ASD in specificity calculations. Studies of wide-scale population use of the M-CHAT with more systematic follow-up of screen negatives have yielded poorer psychometric properties (Carbone et al., 2020; Guthrie et al., 2019). A large-scale study of the ESAC with an independent sample recruited from primary care is underway and is critical for further validation.

Our findings highlight the importance of examining performance across different age ranges from 12–36 months to retain adequate psychometric properties while still detecting children who present with “subtler” differences in social communication and RRBs (Sturner et al., 2017a; 2017b). Consistent with research on other screeners, the ESAC’s PPV increased with age. However, it was encouraging that the ESAC maintained strong psychometric properties across the age ranges considering the greater amount of time between screening and diagnosis in the youngest screening group. Interestingly, among the continuous items, some cutoff scores were observed to change with age and some did not. Different cutoffs were needed for early developing social communication skills, including gestures, play with objects, and pretend play with toys. Conversely, no adjustments to cutoff scores were needed for items that tapped into features specific to ASD like sensory responses, repetitive behaviors, insistence on sameness, and intense interests.

Finally, this study demonstrates the accuracy of parent report to screen young children. Parent concern, by itself, may be less accurate for children at younger ages (Wetherby et al., 2008). Retrospective and prospective studies show that about 75% of parents of children with ASD have concerns by 24 months of age, 50% at 18 months and 30% at 12 months (Chawarska, Paul et al., 2007; Wetherby et al., 2008). Results of this study support that parents are accurate at reporting what their child does and does not do, whether they are concerned or not. Furthermore, for the subsample with two ESACs, a strong association was observed between total scores, supporting the stability of ESAC scores over time.

Strengths, Limitations, and Future Directions

Strengths of this study include a large sample size of children recruited from five sites and prospectively followed and evaluated using gold-standard measures for ASD. Our sample was large enough to examine three separate age intervals, with ample sampling of positive and negative screens and a variety of ascertainment methods. Despite these strengths, our field-test trial has limitations. Because our sample had elevated risk for ASD, the reported accuracy of the ESAC may be higher than in a low-risk primary care sample. Group differences in developmental level may have contributed to the high discrimination among groups. The TD group had above average developmental skills; and, despite being higher than previous screening studies, the developmental level of the ASD group was lower than the DD and TD groups. Thus, it will be important for follow-up studies to examine contributions of developmental differences to the ESAC’s accuracy and specifically test the ESAC’s predictive utility in children with low and high developmental level.

Demographic information was not available for a substantial portion of children and this sample may be limited in racial, ethnic, and socioeconomic diversity. Additionally, the full diagnostic battery was not completed for all children in the NS group. Given the small sample sizes at some sites, the performance of the ESAC could not be examined separately across sites and ascertainment method. Finally, in order to increase sample size and resulting power, multiple ESACs (if available) from the same child were included across (but not within) age groups. While assumptions of independence were not violated, in the older age groups about a third of the ESACs were repeats, which may bias its accuracy. However, given AAP recommendation that children be screened for ASD at 18 and 24-month well-child visits, it is important that accuracy be established for initial and repeated administrations of screening tools.

Future directions include examining the final 30-item set of the ESAC across new independent samples of children at 12–36 months and the performance of the ESAC as an autism screener in a low-risk primary care sample. Further examination of psychometric features is needed, including replication with new samples, test-retest reliability, and a factor analysis to further establish validity. Research is needed to study parent-report screeners with follow-up observational measures (e.g., Dow et al., 2020) to accelerate referrals for diagnosis and eligibility for early intervention.

Conclusions

Results of this study provide preliminary support for the validity of the ESAC as an autism screener for children 12–36 months of age with elevated risk and add to research documenting the accuracy of parent report to screen young children. Use of a parent-report screening tool like the ESAC minimizes the time required of healthcare providers, maximizes the role of the family, and provides reasonably accurate information about whether to refer for further evaluation. These findings offer promise for this new screener to more accurately identify more children at younger ages and contribute to reducing the age of diagnosis and access to intervention.