Modifying a general social-emotional measure for early autism screening

Abstract

Objective: Broadband social-emotional screening tools are designed to evaluate a child’s social development and interactions. Such tools are expected to have reasonable sensitivity for identifying children at risk for autism spectrum disorder (ASD) but would also likely over-estimate risk for ASD since other conditions can also affect social development. In this study, a subset of ASD items from one general social-emotional screening measure, the Ages & Stages Questionnaires: Social Emotional, 2nd edition, was analyzed to determine if use of an ASD subscale might improve prediction of ASD risk for young children.

Methods: The ASD subscale was used with 60 families who had a child referred for an ASD evaluation. Social-emotional screening and ASD screening results were compared with the subsequent results from gold-standard diagnostic testing for ASD at a regional autism center, using contingency matrices.

Results: As expected, the social-emotional screening tool identified nearly all of the children in the high-risk clinical sample. Use of the ASD subscale increased specificity for ASD (from 4% to 52%) and demonstrated correct prediction of ASD diagnosis in 70% of ASD cases.

Conclusions: These preliminary results suggest that using a subset of ASD-specific items on a social-emotional screening tool can increase the tool’s specificity for ASD, by isolating ASD-specific concerns.

Introduction

Current prevalence rates for autism spectrum disorder (ASD) suggest that the behaviorally defined neurodevelopmental disorder affects 1 in 59 children (1 in 37 boys and 1 in 151 girls) (Baio et al. 2018). Due to a variety of factors, including long waitlists at diagnostic centers, the average age of diagnosis for children with ASD is about four years (Mandell et al. 2005). The diagnosis of ASD, however, can reliably be applied as early as two years of age (Lord et al. 2006), and there has been a resulting push for public awareness to support earlier identification with efforts to educate parents, pediatricians, and early childhood educators. In 2013, the Centers for Disease Control launched a campaign of social marketing to increase awareness and identification of ASD and developmental delay (Daniel et al. 2009). The resulting increase in public awareness of ASD and the growing body of research supporting early treatment and improved functional outcomes (Dawson et al. 2010) highlight a need for accurate and efficient early identification.

The American Academy of Pediatrics (AAP) recommends screening a child’s social development early in life to promote early diagnosis of ASD and access to services (Briggs et al. 2012, Weitzman and Wegner 2015). At present, the AAP recommendation is for pediatricians to provide developmental surveillance at all well-child visits and administer assessments at the 18- and 24-month checkups to screen for ASD specifically (Johnson and Myers 2007), though there is ongoing debate about the cost-effectiveness of universal screening versus screening of only high-risk children (Yuen et al. 2018). The purpose of either type of screening is not to diagnose ASD but to simply identify children who may require further evaluation. Additionally, the AAP and Pinto-Martin et al. (2008) suggest the use of ASD-specific screening tools to better identify these children. Indeed, a study examined the efficacy of surveillance versus standardized screening practices in pediatric offices and found that use of tools was necessary for accurate identification of social delay and to detect ASD risk (Gabrielsen et al. 2015). Furthermore, children who received both developmental surveillance and developmental screening were more likely to be involved in early intervention services than children who received one or the other (Barger et al. 2018). Many pediatricians, however, do not utilize ASD-specific tools, citing barriers such as lack of time, lack of familiarity with tools, and preference for specialist referral (Dosreis et al. 2006). General developmental screeners, as well as social-emotional screeners, are currently used in a variety of settings including primary care offices and childcare centers. Broadband screening tools may inform decisions regarding a child’s risk for ASD, and screening results may help providers and educators place referrals earlier and more appropriately thereby using specialty developmental clinics more effectively.

To date, few studies have addressed the use of general developmental screeners in the context of ASD screening. One study explored the utility of the Ages & Stages Questionnaires, Third Edition (ASQ-3) (Squires et al. 2009), a broadband developmental screener, to detect ASD (Hardy et al. 2015). Toddlers were given the ASQ-3 and the Modified Checklist for Autism in Toddlers-Revised (M-CHAT-R) (Robins et al. 2009). The ASQ-3 identified 95% of the ASD cases identified by M-CHAT-R (20/21) using a monitor and/or fail category on the ASQ-3. The communication domain of the ASQ-3 seemed particularly sensitive, identifying 20/21 cases independent of other ASQ-3 domains. The authors could not calculate specificity in the sample because children who ‘passed’ the ASQ-3 were not further evaluated, and the sensitivity is similarly an estimate because children may have been missed by the M-CHAT-R. Results from this study indicate that general screening tools may be capable of detecting ASD in toddlers and also suggest the need for further investigation.

In an effort to reduce over-referrals for ASD clinics from broadband screening tools, Glascoe (1997) conducted a post hoc review of parental concern on the Parents’ Evaluation of Developmental Status (PEDS). They found that specific domains of parental concern were predictive of M-CHAT results (Glascoe et al. 2007). Wiggins et al. (2014) also examined the M-CHAT and the PEDS, comparing the two with respect to ASD detection. Results suggested that ASD screening (i.e. M-CHAT) was superior to general developmental screening (i.e. PEDS) for ASD specifically, but that some children with ASD were only identified by the PEDS tool. In addition, many children with developmental delay were only identified by the PEDS tool, highlighting the need for general screening for both developmental delay as well as ASD.

Another recent study supports the use of broadband screening tools with regard to ASD detection. Using a large Norwegian cohort, Øien et al. (2018) identified ‘missed’ cases of ASD (i.e. false negative on M-CHAT at 18 months). The group analyzed the general screening results at 18 months and found significant differences in children with true-negative results and children with false-negative results, characterized by delays in the ASQ-3 domains of communication, social, and fine motor skills. Authors hypothesized that the ASQ-3 instrument may be particularly useful in eliciting parent concern due to the option for graded responses (sometimes/occasionally) and the use of age-defined intervals, which may allow for more specific milestone-based questioning. Additionally, the authors suggest a careful consideration of social-emotional differences at a young age, as girls in the study who had false-negative results on the M-CHAT were noted to have limited shyness or social inhibition.

While the ASQ-3 has been used in several studies related to ASD screening in toddlers, its social-emotional companion measure, the ASQ: Social-Emotional (Squires et al. 2002) has not. The ASQ:SE was not created to be used as a screening instrument specific to ASD; however, the behavioral characteristics of ASD are consistent with the social-emotional differences and delays measured by the ASQ:SE (Volkmar et al. 2005). During the revision of the ASQ:SE for its second edition (ASQ:SE-2) in 2015, an effort was made to include red flag items that may identify children at risk for ASD (Squires et al. 2015). These items were included in questionnaire intervals for children between 15 and 48 months based on both research-supported early indicators of ASD (Wetherby et al. 2004, Zwaigenbaum et al. 2013) and the age at which the targeted skills might be considered atypical or missing (Ozonoff et al. 2010). Little is known, however, about how well these items function for children who have ASD or how well the ASQ:SE-2 identifies children at risk for ASD.

The present study provides a preliminary examination of the validity of a social-emotional developmental screening questionnaire in identifying children at risk for ASD. It is hypothesized that children with ASD will have a screen-positive result on the ASQ:SE-2 and that higher scores on items specifically related to ASD (i.e. ‘red flag’ items) will predict ASD diagnosis and perhaps support differentiation of ASD from other social-emotional concerns.

Method

Participants

Sixty children with suspected ASD between 18 and 48 months of age and their parents were invited to participate. All participant families spoke English as a primary home language. The children were referred for an ASD evaluation due to concerns about their social, behavioral, and/or communicative development. In each case, the formal referral originated from the child’s primary care provider, although it is not known who had the original concern (family members, childcare providers, pediatricians, etc.). Children who met inclusion criteria but had previously been diagnosed with a medical condition that can affect development (e.g. cerebral palsy, vision loss, hearing loss, genetic syndromes) were not included in the study.

Procedures

Electronic medical health record system schedules were pre-screened for possible participants (reviewing age and primary language). Families meeting study inclusion criteria were invited to participate. They received a packet of information upon check-in to an ASD diagnostic clinic that included an introductory letter and an ASQ:SE-2. If parents agreed to complete the questionnaire and gave signed consent, forms were returned to the front desk staff in a closed envelope. The child and family then participated in regular clinical activities related to the diagnostic process.

Outcome measures

Social-emotional screening

The ASQ:SE-2 was completed by caregivers prior to the professional assessment. The ASQ:SE-2 is a broadband social-emotional screening instrument for children between one month and 6 years of age. Social-emotional domains include self-regulation, compliance, social communication, adaptive functioning, autonomy, affect, and interaction with people. The standardization sample included over 14,000 children with diversity that was reflective of U.S Census data.

ASQ:SE-2 test characteristics reflect an overall sensitivity of 81% across age intervals (with a range from 77% to 84%) and an overall specificity of 84% across age intervals (with a range of 76% to 98%) with high reliability (89% test–retest reliability) and high internal consistency (84%) (Squires et al. 2015). Validity statistics reported for the ASQ:SE-2 are based on convergent validity related to several evidence-based tools designed to measure general social-emotional competence in young children (Squires et al. 2015). Sample items from the ASQ:SE-2 (18-month interval) include the following: Does your child look at you when you talk to him?, When you point at something, does your child look in the direction you are pointing?, and Does your child make sounds, or use words or gestures, to let you know she wants something (for example, by reaching)? Parents have the option to choose ‘often or always,’ ‘sometimes,’ or ‘rarely or never.’ They are also given the opportunity to mark whether they have specific concerns about each particular item. Items are scored according to parent responses and result in zero to 15 points per item, with a total score range of zero to approximately 370 (varies by interval). Each interval (e.g. 6-month, 18-month, 48-month) has its own empirically derived cutoff score with higher scores more indicative of difficulties, and children above the cutoff may be referred for further evaluation. A monitoring zone identifies children who are close to the cutoff indicating higher risk, and should be monitored and rescreened. In this study, the ‘monitor’ and ‘fail’ categories on the ASQ:SE-2 were combined, as suggested in a study related to ASQ:3 (Hardy et al. 2015).

ASD subscale scores

Within each of the five ASQ:SE-2 intervals between 18 and 48 months, nine items were determined to be potentially representative of ASD at the corresponding ages based on the established characteristics and parent report of ASD in toddlers and young children (Wetherby et al. 2004, Ozonoff et al. 2009). The nine ASD items per interval were endorsed by a research reliable ADOS-2 examiner who was naive to the study, as being potential ASD indicators. This professional had a clinical doctorate in psychology, with eight years of team-based diagnostic experience with young children at risk for neurodevelopmental disorders. Examples of ASD items included questions regarding joint attention, eye contact, peer interaction, and conversational abilities (see Table 1). ASD subscale scores were calculated by summing the parent response (zero to 15) for these items, making the possible ASD subscale score range between zero and 135.

Table 1.

Questions for ASD items.

Item content	Interval(s)
Does your child respond to her name when you call her?	18, 24
When you point at something, does your child look in the direction you are pointing?	18, 24, 30
Does your child try to show you things (with point and check-in at later intervals)	18, 24, 30, 36
Does your child play with objects by pretending (symbolic at later intervals)?	18, 24, 30, 36
Does your child look at you when you talk to him?	18, 24, 30, 36, 48
Does your child do things over and over and get upset when you try to stop her?	18, 24, 30, 36, 48
Does your child let you know how she is feeling with gestures or words?	18, 24, 30, 36, 48
Does your child like to be around other children? (Also family members and friends for 18 month interval)	18, 24
Does your child greet or say hello to familiar adults?	24, 30
Does your child do what you ask him to do?	30
Does your child move from one activity to the next with little difficulty?	30, 36, 48
Can your child name a friend?	36, 48
Do other children like to play with your child?	36, 48
Does your child like to play with other children?	36, 48
Does your child show concern for other people’s feelings?	48
Does your child have simple conversations with you?	48

Social interaction testing and clinical diagnosis

Participants received a team-based ASD assessment that included the gold standard practice of reaching a best-estimate clinical consensus diagnosis. The evaluation included the Autism Diagnostic Observation Schedule-2^nd Edition (ADOS-2) (Lord et al. 2012) as well as a parent interview derived from the Autism Diagnostic Interview-Revised (Lord et al. 1994). Members of the ASD diagnostic team included audiology, speech-language pathology, psychology, occupational therapy, and developmental pediatrics. Providers who administered the assessments for this study included master’s and doctoral-level practitioners with administration reliability (i.e. 80% or better) as part of clinical and/or research ADOS-2 training. The average experience for these providers in using the ADOS was eight years.

This study was approved by the university’s Institutional Review Board, and consent for this minimal-risk study was obtained from the caregiver prior to the clinical participant’s entry into the study. Participants received a $10 coffee shop gift card.

Results

Sixty children between 18 and 48 months were evaluated for ASD by a multidisciplinary team at a regional center for ASD diagnostics (see Table 2 for demographic details). Eighty-five percent of pre-visit paperwork was completed by mothers. Insurance status in Table 2 is used as a proxy for socioeconomic status, and the percentage represents the number of children covered by public (versus private) insurance. ASQ:SE-2 questionnaires were selected based on the child’s age at the time of the visit: nearly half were in the 48-month interval, and only 1 was in the 18-month interval. Of the 60 children referred for ASD assessment, only one child screened negative on the ASQ:SE-2. Fifty-three screened positive on the ASQ:SE-2, and 6 had scores in the ‘monitoring zone.’ The scores followed a normal distribution, with one potential outlier. The caregivers of the outlier child indicated a very high level of concern, scoring the maximum point value on many items. Because the answers appeared to be an accurate representation of this parent’s concern, the score remained in the analyses. The cutoff scores for each interval vary, but the group average total score (153.44, indicating social-emotional risk) was above the cutoff for every interval. See Table 3 for screening and diagnostic results.

Table 2.

Participant information.

	n	M/Count	Minimum	Maximum	SD
Gender (male)	60	46 (77%)
Age (months)	60	37.65	19	48	8.54
Ethnicity (Hispanic)	60	7 (12%)
Race	60
White	45	75%
Black	3	5%
Multiracial	6	10%
Asian	2	3%
American Indian	1	2%
Declined	3	5%
Insurance status (public)	60	38 (63%)

Note: Count data are presented as n (%).

Table 3.

Screening and diagnostic results.

	n	M/Count	Minimum	Maximum	SD
ASD diagnosis given	60	37 (62%)
ASD diagnosis comorbid with:	37
Language impairment	12
Global developmental delay	25
ASQ:SE-2 total score	60	153.44	50	370	55.42
With ASD diagnosis	37	160.99	75	370	61.11
No ASD diagnosis	22	141.30	50	245	43.33
Subscale score	60	51.33	15	115	23.54
With ASD diagnosis	37	60.54	25	115	23.92
No ASD diagnosis	22	37.39	15	65	16.02

Note: Count data are presented as n (%).

ASD assessment

Of the 60 children referred for ASD evaluation, 37 (62%) received a diagnosis of ASD (Table 3). For girls, 64% of those referred were given a diagnosis of ASD, and 61% of referred boys were given a diagnosis of ASD. There was no significant difference in total ASQ:SE-2 scores for participants with and without ASD diagnoses. Of the 22 children who did not meet criteria for an ASD diagnosis, all of them received at least one diagnosis of some kind: 12 with Language Impairment, 8 with Global Developmental Delay, 4 with Attention Deficit Hyperactivity Disorder, 3 with Speech Articulation Disorder, and 2 with a behavioral disorder.

ASD subscale scores

A binomial logistic regression was performed to ascertain the effects of the ASD item scores on the likelihood that participants had ASD. Linearity of the continuous variables with respect to the logit of the dependent variable was assessed via the Box-Tidwell (Box and Tidwell 1962) procedure. Based on this assessment, completed in SPSS 23, the single continuous independent variable was found to be linearly related to the logit of the dependent variable. The logistic regression model was statistically significant, X²(1)=16.67, p<.0001. The model explained 33.0% (Nagelkerke R²) of the variance in ASD diagnosis and correctly classified 70.0% of cases. Sensitivity was 81.1%; specificity was 52.2%. The positive predictive value was 73.2%, and the negative predictive value was 63.2%; however, it is important to note that these values refer only to the current clinical sample and not the population as a whole. The odds ratio was 1.06 (with 95% confidence interval between 1.03 and 1.10), indicating that for every one-point increase in ASD subscale score, a diagnosis of ASD became 1.06 times as likely. Increasing ASD subscale score was associated with an increased likelihood of exhibiting ASD.

Creation of cutoff score for ASD subscale on ASQ:SE-2

A receiver operating characteristic (ROC) curve was produced to interpret sensitivity and specificity levels for the ASD subscale related to ASD diagnosis. The resulting area under the curve (AUC) for this analysis was .78, representing moderate accuracy and reliability, which may be appropriate given its use as a screening measure. This was a statistically significant finding. The 95% confidence interval of the AUC for this measure was between .67 and .90. Utilizing the coordinates of the ROC curve, a cut score of 40 was selected indicating that a score of 40 or above on the ASD subscale would indicate a positive autism screen. The cut score of 40 on these 9 items resulted in sensitivity and specificity levels that were equal to those created by the binomial logistic regression model: 81% sensitivity and 52% specificity. Contingency matrices for screener and subscale results compared to diagnostic results are presented in Tables 4 and 5.

Table 4.

Classification agreement between ASQ:SE-2 and ASD diagnosis.

		ASD Diagnosis
		ASD	Non-Spectrum	Total
ASQ:SE-2 Screen Result	Positive	37	22	59
	Negative	0	1	1
	Total	37	23	60

Table 5.

Classification agreement between subscale score and ASD diagnosis.

		ASD Diagnosis
		ASD	Non-Spectrum	Total
ASD subscale result	Positive	30	11	41
	Negative	7	12	19
	Total	37	23	60

Note. ASD subscale cut-off = 40.

Discussion

Early and accurate identification of children at risk for ASD remains a critically important component of pediatric health care. Recently the United States Preventative Services Task Force (USPSTF) issued a statement on ASD screening (Siu et al. 2016), concluding that there was insufficient evidence to assess the balance of benefits and harms of developmental screening for children not at risk for ASD (i.e. ‘universal screening’). Some researchers worried that this statement may undermine recent efforts to increase screening for ASD, but for many the statement served as an impetus to increase research in the areas of screening and ASD treatment efficacy (Coury 2015, Dawson 2016). Indeed, research has clearly shown benefits from early intervention for children with ASD, both in general developmental areas (Sheinkopf and Siegel 1998, Harris and Handleman 2000, Dawson et al. 2010) and with regard to core ASD symptomatology (Kasari et al. 2010, Landa et al. 2011). There is a current need for more research to address the efficacy of screening tools and available interventions to support developmental growth.

This study explored the use of a general social-emotional screening tool as an ASD specific screener. It is expected that children with ASD would screen positive on a social-emotional screening tool for young children, but certainly not all children who do will be diagnosed with ASD or will even require an evaluation to rule out ASD. Children with social-emotional differences may have language delays or disorders, behavior regulation problems, attention problems, developmental delays, or mental health disorders. As long waitlists for team assessments can contribute to the delay in diagnosis, adaptation of already-in-use screening tools may improve quality of referrals or efficacy of triage at intake.

As predicted, nearly all children referred for an ASD evaluation in this study had a positive screen on the social-emotional screening tool (59), reflecting a parental concern in the area of social-emotional development. Importantly, however, 22 of the 59 children with a screen-positive result did not have ASD (4% specificity). This finding supports the hypothesis that a general social-emotional screener has good sensitivity for ASD but is insufficient for differentiating ASD from other social-emotional delays or differences. However, when ASD-related items were considered as a subset, specificity for ASD increased to 52% in this population.

Implications and clinical relevance

While this is a preliminary study, there is potential for an ASD subscale on a social-emotional screening tool to be clinically useful. Providers may appreciate the ability to gather information related to a child’s risk for ASD from the use of an existing broadband social-emotional screener. Additionally, this study includes children between 18 and 48 months of age. This is clinically relevant because there are currently no screening tools directly intended for use in the 4th year of life (36 to 48 months), though a recent study investigated how tools for younger children function in this age range (Salisbury et al. 2018).

Use of an ASD subscale within a broadband social-emotional screener may also address USPSTF concerns regarding the possible ‘harm’ of conducting universal ASD screenings (Siu et al. 2016). Conducting an embedded ASD screening within the context of a broadband screening tool improves the universal ASD screening process in two ways. Use of the ASD subscale on the ASQ:SE-2 increases the specificity of the broadband screener for ASD, thereby reducing potential provider and parental concern for ASD in children who may have other social-emotional concerns. This may also reduce the number of children referred for specialty ASD diagnostic clinics that often have long waitlists. The ASD subscale may reduce the ASQ:SE-2’s overall sensitivity to ASD, but this is to be expected with a rise in specificity. This tool is also intended to be used in conjunction with clinical judgment, developmental surveillance, and discussion with family.

Limitations

One limitation of the study is its sample size of 60 participants. While the sample size is sufficient for the statistical analyses used, it was not large enough to evaluate other demographic variables, including race, gender, and socio-economic status of the participants. Recent studies have shown racial and ethnic disparities in identification of children with ASD; children who were Black, Hispanic, or of other race/ethnicity were less likely than White children to have documented ASD (Liptak et al. 2008, Mandell et al. 2009, Zuckerman et al. 2013). The present study included a diverse group of children, but the sample was limited to English-speaking families, and the numbers in each group were not adequate to allow for comparisons between groups of children.

Additionally, the current study did not consider the possibility that various subgroups within the group of participants may display ASD symptomatology differently. For example, Tek and Landa (2012) reported differences in ASD symptoms between minority and non-minority toddlers. Recent research has also explored the idea of how children of different genders may display ASD characteristics differently (Lai et al. 2015). With regard to general social-emotional development (as measured by the ASQ:SE-2), Chen et al. (2015) described both gender and cultural differences on some social-emotional competencies.

The sample is a clinical sample, because all children were referred for an ASD evaluation. This sampling affects the interpretation of the positive and negative predictive values because the prevalence of ASD in this study (62%) does not reflect the current known prevalence of ASD, which is one in 59 children (Baio et al. 2018). Using a clinical sample inflates the positive-predictive value and deflates the negative-predictive value (Parikh et al. 2008). The sampling method for this study was chosen intentionally to be able to evaluate the ability of a broadband social-emotional screener to differentiate between children with potential behavior or language problems and children with ASD. The ASQ:SE-2 has established reliability and validity, so its ability to differentiate between the general population and those with ASD is presumed. A more difficult task is to differentiate between children with a variety of social-emotional differences (e.g. ASD, Attention Deficit Hyperactivity Disorder). A larger sample size would allow for a more thorough examination into the various phenotypical presentations of ASD.

Finally, there is the obvious limitation that while the use of an ASD subset clearly improved specificity for ASD in a clinical sample, the resulting specificity for ASD was still low. This finding is consistent, however, with other research on ASD-specific screening tools. A recent study on the psychometric properties of two commonly used ASD screening tools used in similar populations found specificities in the range of 54%–59% (Salisbury et al. 2018). The present study provides preliminary support for similar results by using a modified version of an already in-use social-emotional tool. More research is needed on how this modification functions in the general population.

Conclusions

In the present study, a broadband social-emotional screening tool was used as an ASD specific screening tool by creating a subset of items that relate to core symptomatology of ASD. This was a necessary step, as the broadband social-emotional screening tool does not differentiate well between children with ASD and children with other social-emotional differences or delays. Indeed, in the present study that included children in a clinical or referred sample, scores on an ASD subscale provided a better prediction of diagnosis than broadband screening results. In addition, the ASD subscale was useful for children between 30 and 48 months of age, a period for which few level-two screening tools currently exist.

The findings from this study are important for several reasons. First, early ASD identification appears to be related to prognosis, since some of the best empirically supported ASD treatments are geared toward very young children (Dawson et al. 2010, Fernell et al. 2013). If children at risk for ASD are identified early and referred for comprehensive evaluation, they may have earlier access to treatments and family support. Clearly, more research is needed to support the usefulness of universal screening, but in the meantime, it may be useful for practitioners to be able to utilize a broadband screening tool to evaluate a child’s risk for ASD as an initial step.

Pediatricians and family practice physicians are pressed for time but have an obligation to observe a child’s physical, mental, and developmental growth in a short amount of time. Screening tools can support practitioners’ ability to reliably provide information in these areas. If some tools can provide information on multiple developmental areas at once, it may ameliorate some of the reported difficulty providers have in completing all necessary screenings and may support earlier access to and better referrals for comprehensive assessment and services. Findings from this preliminary study support the potential use of a broadband social-emotional screening tool as an ASD-specific screener and indicate a need for further research.

Disclosure statement

In accordance with Taylor & Francis policy and my ethical obligation as a researcher, I am reporting that the first and third authors receive royalty payments due to authorship (third author) and supporting authorship (first author) of the assessment tool studied in this manuscript. No other potential conflicts of interest exist for any of the authors.

Funding

This project has no federal funding.