Associations among referral concerns, screening results, and diagnostic outcomes of young children assessed in a statewide early autism evaluation network

Abstract

Objective(s):

To examine associations between referral concerns, screening results, and diagnostic outcomes for young children evaluated across a statewide primary care network for early screening and diagnosis of ASD.

Study Design:

The Early Autism Evaluation (EAE) Hub system was developed to increase developmental screening and improve access to timely ASD evaluations in local communities. In 2019, 858 children (ages 18–48 months; 40% diagnosed with ASD) received ASD evaluations across 12 EAE Hubs. Data on PCP- and caregiver-reported referral concerns, MCHAT-R/F and ASQ-3, and diagnostic outcome were collected.

Results:

Among children evaluated, there was low concordance between PCP and caregiver referral concern. While a positive MCHAT-R/F screen was associated with PCP but not caregiver-reported ASD referral concern, there was a significant linear relationship between MCHAT-R/F raw scores and both PCP and caregiver ASD referral concern. A different pattern of ASQ-3 delays was found to be associated with PCP- as compared to caregiver-reported ASD referral concern. Finally, PCP-reported ASD referral concern, positive MCHAT-R/F, and ASQ-3 Communication and Personal Social delays were associated with a significantly higher likelihood of subsequent ASD diagnosis.

Conclusion(s):

Understanding how community PCPs use surveillance and screening data, the extent to which PCPs and caregivers have shared understanding and engage in collaborative decision-making about evaluation referral, and how these factors relate to diagnostic outcome has the potential to impact educational efforts for both PCPs and caregivers of young children, as well as inform the development of more efficacious early identification approaches.

Autism spectrum disorder (ASD) is a complex neurodevelopmental disability with an estimated prevalence of one in 54 children (1) and yearly cost of $268 billion (2) in the United States. While ASD diagnosis becomes stable in the early toddler years (3), the average age of diagnosis is not until after 4 years of age (4). Delays in diagnosis lead to missed opportunities for interventions known to improve developmental outcomes (5, 6) and reduce lifetime costs (7, 8). As such, ensuring access to early, high quality ASD evaluation is of critical importance (9). Stakeholders, including the American Academy of Pediatrics (AAP; 10, 11), and researchers (12, 13) have highlighted routine developmental surveillance and screening during well-child primary care visits as critical strategies for reducing the age of ASD diagnosis.

The success of surveillance for early signs of ASD relies on both the pediatric primary care provider (PCP) as well as the child’s caregivers. While ASD practice guidelines (10, 11, 14) and resources (e.g., Center for Disease Control Prevention “Learn the Signs, Act Early” campaign, AAP’s Tools and Resources for Pediatricians) have been broadly disseminated, PCPs continue to report gaps in knowledge, comfort, and time and resources to adequately identify and care for children with ASD (15–17). Further, caregivers of children later diagnosed with ASD often perceive that their child’s PCP did not act early enough on their developmental concerns (18). Despite a strong association between caregivers early developmental concerns and age of ASD diagnosis (4, 19–21) a significant lag between first concern and diagnosis continues to be well documented (4, 22, 23).

Over the past decade, increasing numbers of pediatricians report adhering to recommended developmental screening procedures (24) despite controversy regarding the efficacy of universal ASD screening (25). Recently, research has documented mixed findings with regard to accuracy of both broadband developmental (26, 27) and ASD screening tools (26, 28) for early identification of children with ASD. Furthermore, recent research has revealed inconsistencies in how PCPs use screening results to guide referral for diagnostic evaluation (29, 30).

Substantial progress has been made in our understanding of developmental surveillance and screening practices with increasing attention being paid to the accuracy and limitations of screening tools for identifying young children with ASD. Yet, little research has examined how screening results may be used by PCPs and caregivers. In the present study, we leverage the Early Autism Evaluation (EAE) Hub system (31), a statewide infrastructure for early screening and diagnosis of young children at-risk for ASD, in order to better understand associations between referral concerns (i.e., the developmental and/or behavioral concerns of PCPs and/or caregivers that prompt referral for evaluation), screening results, and diagnostic outcomes for children evaluated in this community primary care-based network. Specifically, we sought to 1) describe PCP- and caregiver-reported primary referral concerns and determine the extent to which PCPs and caregivers agree on referral concerns, 2) examine associations between results of developmental and ASD screening tools and ASD concerns at referral, and 3) identify how ASD referral concerns, screening results, and child age may be associated with ASD diagnostic outcome.

Methods

Data presented in this report were collected as part of the Indiana University School of Medicine (IUSM) Early Autism Evaluation (EAE) Hub system quality improvement initiative. To minimize data collection burden and remain in compliance with IRB guidelines for quality improvement efforts, no protected health information or demographic data was collected. This study was reviewed by the Indiana University Institutional Review Board and a waiver for human subjects research was provided.

Early Autism Evaluation (EAE) Hub System

The EAE Hub system is an innovative statewide quality improvement initiative that aims to increase developmental screening and improve access to timely ASD evaluations in local communities, thereby lowering the age of ASD diagnosis and promoting earlier enrollment into evidence-based interventions. Comprehensive description of the development and scale-up of this system, as well as outcomes from six years of implementation can be found in McNally Keehn et al. (31). In summary, since 2012, a team of faculty within the Department of Pediatrics at IUSM has provided training and technical assistance to community PCPs in following AAP guidelines for developmental and ASD screening (11, 14). EAE Hubs were implemented in targeted geographic regions in collaboration with known pediatric champions. EAE Hub clinicians and their administrative teams were trained on a standard clinical pathway for evaluation of children 18–48 months referred by primary care providers. The EAE Hubs receive referrals from regional PCPs for evaluation of children determined to be at-risk for ASD based on developmental and/or ASD screening results or concerns elicited during routine surveillance. The EAE Hub evaluation protocol includes review and/or administration of standard screening tools, diagnostic interview including medical history and assessment of DSM-5 ASD criteria, physical examination, administration of the Screening Tool for Autism in Toddlers (STAT; 32), and integration of data to formulate a clinical diagnosis and report with intervention recommendations (including local community resources and supports). EAE Hub teams participate in collection and submission of quality indicator data as well as a monthly learning collaborative and annual meeting.

Participants

Twelve EAE Hubs with 24 clinicians (MD: n=15; NP: n=9) submitted evaluation data during the 2019 calendar year. EAE Hubs conducting less than 15 evaluations in 2019 (n=1) were excluded from the analysis. During 2019, 858 children ages 18 to 48 months (mean: 30.4 months; SD: 6.5) were evaluated in the EAE Hubs. MD clinicians conducted 73% (n=626) of all evaluations (NP: n=232; 27% evaluations). Referrals to the EAE Hubs were from 469 clinicians (n=463 practicing in Indiana; n=6 practicing in neighboring Midwest states) [pediatricians: n=312, 67%; family medicine: n=138, 29%; other: n=19, 4% (including developmental pediatrician (n=1), ENT (n=1), internist (n=2), neurologist (n=5), obstetrician (n=1), pulmonologist (n=1), unknown (n=8)]; ten percent (n=45) of referrals originated from within the EAE Hub practice. See Table 1 (Online only) for additional characteristics of Indiana referring providers and county-level demographics. Median latency from referral to evaluation was 90 days (mean: 94.9 days, SD: 56.6). Forty percent (n=344) of children evaluated in the EAE Hubs were diagnosed with ASD; 30.5% (n= 313) were diagnosed with global developmental delay (i.e., GDD; developmental delay in ≥ 2 domains of development without comorbid ASD), and 36.4% (n=176) were diagnosed with another developmental, behavioral, or medical disorder (i.e., speech language delay, motor delay, feeding disorder, social environment problems, history of prematurity, prenatal substance exposure, anxiety). Less than three percent of children (n=25) had no diagnostic concern identified during the evaluation.

Measures

Ages and Stages Questionnaire – Third Edition (ASQ-3; 33)

The ASQ-3 is a widely used 30-item Likert scale developmental screening tool designed to identify children 1 through 66 months of age who may be at risk for developmental delays. The ASQ-3 is completed by caregivers in 10–15 minutes. Scores across five developmental domains (i.e., Communication, Gross Motor, Fine Motor, Problem Solving, and Personal Social) are then compared to age-based cut-offs yielding categorical risk: On schedule; Monitor; Delay. Additional open-ended, unscored questions allow caregivers to provide more detail about developmental concerns. In the present study, On Schedule and Monitor results were collapsed and compared with Delay results due to small sample size for ASQ-3 scores in the On Schedule range (i.e., precluding comparison between On Schedule and Monitor + Delay). Further, although scores in the Monitor range may indicate incremental risk for ASD (as compared with On Schedule scores), the ASQ-3 guidance for clinical use specifies referral for further assessment when scores are in the Delay range.

Modified Checklist for Autism in Toddlers, Revised with Follow-Up (M-CHAT-R/F; 34)

The M-CHAT-R/F is a two-stage caregiver-report screening measure developed to assess risk for ASD in children ages 16–30 months. Twenty yes/no items can be scored in under two minutes. Total score ≤ 2 indicates low risk for ASD and no further follow-up is recommended. When a Total score between 3 and 7 is obtained, the second-stage Follow-Up questions should be administered to reduce false positive results; when a Total score ≥ 8 is obtained, it is permissible to bypass Follow-Up and refer immediately for diagnostic evaluation. For the present study, total scores ≥ 8 when bypassing Follow-Up or scores ≥ 2 with Follow-Up were categorized as positive risk (i.e., “At-Risk”) for ASD. Both continuous (i.e., raw) and categorical MCHAT-R/F risk scores were analyzed.

Procedures

The EAE Hubs enter de-identified data for each evaluation into a secure online research database (see 31 for description). As part of this effort, data on PCP- and caregiver-reported referral concerns, results of screening and evaluation measures, and diagnostic outcome are collected. Referral concerns of the PCP were extracted from standard medical referral documentation (i.e., referral from PCP to EAE Hub) by the EAE Hub team; caregiver concerns were gathered as part of the standard clinical interview during the evaluation and/or noted in the referral documentation. Referral concerns were documented by the EAE Hub team in a standardized data collection form (allowing for multiple concerns to be endorsed by each reporter) in the online database. For the present study, referral concerns were categorized into one of four primary mutually exclusive categories: ASD, global developmental delay (without ASD), speech language/delay, and behavior problems. For some analyses, referral concerns were dichotomized into ASD or non-ASD. ASQ-3 and M-CHAT-R/F screening results were collected from referring PCPs as part of the standard EAE Hub referral pathway (31). When screening results were obtained > 3 months prior to the EAE Hub evaluation, EAE Hubs re-administer and report updated ASQ-3 and MCHAT-R/F scores. Finally, diagnostic outcome for each EAE Hub evaluation was dichotomized (i.e., non-ASD [including GDD or another developmental, behavioral, or medical disorder] or ASD).

Statistical Analysis

Descriptive statistics are presented as absolute frequencies and percentages for categorical variables, and means and standard deviations for continuous variables. Cohen’s Kappa was calculated to obtain a measure of agreement on primary referral concern between PCPs and caregivers. Mantel-Haenszel Chi-Square tests were performed to test the association between ASD referral concerns of the PCP and caregiver and MCHAT-R/F and ASQ-3 categorical results, as well as the association between ASD diagnostic outcome and ASD referral concerns of the PCP and caregiver and MCHAT-R/F and ASQ-3 categorical results. Two sample t-tests were performed to compare continuous variables by ASD referral concern and diagnostic outcome. Multivariable logistic regression models were fitted to estimate the probability of ASD diagnostic outcome with covariates of ASD referral concerns of the PCP and caregiver, MCHAT-R/F and ASQ-3 categorical results, and child age. Data analysis was performed with SAS version 9.4 (SAS Institute, Inc., Cary, NC). p values < .05 were considered statistically significant.

Results

Primary Referral Concerns and Agreement among Reporters

There was a significant difference in the distribution of primary referral concerns among referring PCPs and caregivers (p<.001, see Table 2). Fifty percent (n=431) of children were identified by their referring PCP as having symptoms concerning for ASD. Ninety-two percent (n=785) of children evaluated had ≥ 1 PCP-reported referral concern (mean number concerns=1.24, SD=0.67) across the four primary categories. Other PCP-reported concerns not falling within these categories included growth and development, social development, and/or sensory processing. In contrast, the most frequent caregiver-reported referral concern was speech/language delay (40%). Nearly 98% of caregivers reported a concern within the four primary categories (mean number concerns=1.60, SD=0.73); additional concerns were in the domains of social development, sensory processing, feeding, sleep, and/or anxiety. There was slight to fair agreement (35) between PCP and caregiver-reported primary referral concerns (ASD: Κ=0.197; Developmental delay: Κ=0.188; Speech/language delay: Κ=0.188; Behavior problems: Κ=0.275).

Table 2.

PCP and caregiver reported primary referral concerns and agreement among reporters (N = 858)

	PCP	Caregiver	Both PCP & Caregiver
Primary referral concern	n (%)	n (%)	n (%)	Κ	p
Autism spectrum disorder	431 (50.20)	290 (33.80)	188 (21.91)	0.197	<.001
Developmental delay	203 (23.66)	136 (15.85)	58 (6.76)	0.188	<.001
Speech/language delay	116 (13.52)	339 (39.51)	80 (9.32)	0.188	<.001
Behavior problem	35 (4.08)	73 (8.51)	17 (1.98)	0.275	<.001

PCP = referring primary care provider. Referral concerns are not mutually exclusive for concern category or reporter (i.e., PCP, caregiver).

Associations between Screening Results and ASD Referral Concern

There was a significant association between PCP-reported ASD referral concern and categorical MCHAT-R/F screen positive results as well as MCHAT-R/F raw scores (ps<.001; see Table 3). PCPs reported a primary referral concern of ASD for 80% of children who screened positive on the MCHAT-R/F. While there was no significant association between caregiver-reported ASD referral concern and categorical MCHAT-R/F results (p=.090), there was a significant association with MCHAT-R/F raw scores (p=.005). There was a significant linear association between PCP-reported ASD referral concern and ASQ-3 Communication (p=.007), Problem Solving (p=.016), and Personal Social (p=.033) delays. A significant linear association between caregiver-reported ASD referral concern and ASQ-3 Gross Motor (p=0.022) and Problem Solving (p=0.023) delay was also found. Both PCPs and caregivers were more likely to have ASD concerns at referral as the total number of ASQ-3 delay domains increased (PCP: p=.014; Caregiver: p=.048).

Table 3.

MCHAT-R/F and ASQ-3 associations with PCP- and caregiver-report referral concern (n= 858)

	PCP			Caregiver
	Non-ASD Concern n=427	ASD Concern n=431	p	Non-ASD Concern n=568	ASD Concern n=290	p
Positive MCHAT-R/F,* n(%)	242 (66.1%)	210 (79.8%)	<.001	295 (69.7%)	157 (76.2%)	0.090
MCHAT-R/F score,* mean (SD)	5.6 (4.4)	7.1 (4.6)	<.001	5.9 (4.4)	6.9 (4.7)	0.005
ASQ-3 Domain Delay, n(%)
Communication	292 (70.5%)	329 (78.7%)	0.007	410 (73.9%)	211 (76.2%)	0.473
Gross Motor	144 (35.0%)	155 (37.2%)	0.523	184 (33.4%)	115 (41.5%)	0.022
Fine Motor	180 (43.4%)	195 (47.1%)	0.281	242 (44.0%)	133 (47.7%)	0.316
Problem Solving	248 (60.2%)	285 (68.2%)	0.016	339 (61.5%)	194 (69.5%)	0.023
Personal Social	261 (63.0%)	292 (70.0%)	0.033	366 (66.2%)	187 (67.3%)	0.755
Total ASQ-3 Delay, n(%)
0 Domains	52 (12.8%)	44 (10.7%)	0.014	65 (12.0%)	31 (11.3%)	0.048
1 Domain	60 (14.7%)	38 (9.3%)		68 (12.5%)	30 (10.9%)
2 Domains	66 (16.2%)	59 (14.4%)		88 (16.2%)	37 (13.5%)
3 Domains	77 (18.9%)	89 (21.7%)		118 (21.7%)	48 (17.5%)
4 Domains	82 (20.1%)	99 (24.1%)		116 (21.4%)	65 (23.7%)
5 Domains	70 (17.2%)	81 (19.8%)		88 (16.2%)	63 (23.0%)

Percentages are calculated based on those children for which data is available.

^*n = 605 due to sample restriction for those with MCHAT-R/F data. Positive MCHAT-R/F (categorical score); MCHAT-R/F score (continuous raw score).

p values represent significance of Chi-Square (categorical variables) and t-test (continuous variables) analyses.

MCHAT-R/F = Modified Checklist for Autism in Toddlers, Revised, with Follow-Up; PCP = referring primary care provider; ASQ-3 = Ages and Stages Questionnaire – Third Edition.

Associations between ASD Referral Concern, Screening Results, and Child Age and Diagnostic Outcome

First, bivariate analyses were performed to examine differences in PCP- and caregiver-reported ASD referral concern, MCHAT-R/F (i.e., raw and categorical scores) and ASQ-3 results, and child age across diagnostic outcome groups (i.e., non-ASD, ASD) (see Table 4 for subsample with MCHAT-R/F; see Table 6; online only for full sample). Next, associations between PCP- and caregiver-reported ASD referral concern, MCHAT-R/F categorical risk scores and ASQ-3 results, and child age by diagnostic outcome were examined using univariable (i.e., unadjusted) and multivariable (i.e., adjusted) logistic regression models (see Table 5). Given that PCPs are likely to use MCHAT-R/F categorical risk scores to identify children in need of referral for evaluation, this variable was employed in the regression models. All unadjusted variables were significant independent predictors of ASD diagnostic outcome (all p<.015) with the exception of child age (p=.647). In the adjusted multivariable regression model, PCP-reported ASD referral concern (p=.003), MCHAT-R/F screen positive result (p<.001), and ASQ-3 Communication (p<.001) and Personal Social (p=.025) delays remained significant predictors of ASD diagnostic outcome. More specifically, children who screened positive on the MCHAT-R/F were 2.41 (95% CI 1.45–4.02) times more likely to be diagnosed with ASD than those that did not screen positive. Those with ASQ-3 Communication and Personal Social delays were 2.86 (95% CI 1.65–4.97) and 1.76 (95% CI 1.07–2.88) times more likely to be diagnosed with ASD than those with without delays, respectively. Finally, children with a PCP-reported referral concern of ASD were 1.77 (95% CI 1.21–2.57) times more likely to be diagnosed with ASD than those with non-ASD PCP referral concerns. This analysis was replicated with the entire sample (n=817) with similar findings (see Table 7; online only). PCP-reported ASD referral concern (p<.001, OR: 1.92, 95% CI 1.40–2.63) and ASQ-3 Communication (p<.001, OR: 3.42, 95% CI 2.11–5.53), Fine Motor (p<.001, OR: 1.69, 95% CI 1.22–2.36), and Personal Social (p<.001, OR: 2.19, 95% CI 1.44–3.33) delays remained significant predictors of ASD diagnostic outcome.

Table 4.

PCP and caregiver-report ASD referral concern, MCHAT-R/F and ASQ-3 results, and child age by diagnostic outcome (n= 605)

	Non-ASD n= 376	ASD n=229	p
PCP ASD Concern, n (%)	132 (35.1%)	121 (52.8%)	<.001
Caregiver ASD Concern, n (%)	110 (29.3%)	89 (38.9%)	0.015
Positive MCHAT-R/F, n (%)	234 (62.2%)	198 (86.5%)	<.001
MCHAT-R/F score, mean (SD)	4.9 (4.0)	8.2 (4.4)	<.001
ASQ-3 Domain Delay, n (%)
Communication	238 (63.3%)	207 (90.4%)	<.001
Gross Motor	114 (30.3%)	99 (43.2%)	0.001
Fine Motor	134 (35.6%)	130 (56.8%)	<.001
Problem Solving	193 (51.3%)	180 (78.6%)	<.001
Personal Social	203 (54.0%)	189 (82.5%)	<.001
Child Age, mean (SD)	29.1 (6.4)	29.3(6.0)	0.6475

Positive MCHAT-R/F (categorical score); MCHAT-R/F score (continuous raw score).

p values represent significance of Chi-Square (categorical variables) and t-test (continuous variables) analyses.

MCHAT-R/F = Modified Checklist for Autism in Toddlers, Revised, with Follow-Up; PCP = referring primary care provider; ASQ-3 = Ages and Stages Questionnaire – Third Edition.

Table 5.

Associations between PCP and caregiver-report ASD referral concern, MCHAT-R/F and ASQ-3 results, and child age by diagnostic outcome (n= 605)

	Unadjusted model		Adjusted model
Predictor	OR (95% CI)	p a	AOR (95% CI)	p b
PCP ASD Concern	2.07 (1.48, 2.89)	<.001	1.77 (1.21, 2.57)	0.003
Caregiver ASD Concern	1.54 (1.09, 2.17)	0.015	1.23 (0.83, 1.83)	0.299
Positive MCHAT-R/F	3.88 (2.52, 5.97)	<.001	2.41 (1.45, 4.02)	<.001
ASQ-3 Domain Delay
Communication	5.45 (3.35, 8.87)	<.001	2.86 (1.65, 4.97)	<.001
Gross Motor	1.75 (1.24, 2.46)	0.001	0.92 (0.62, 1.36)	0.669
Fine Motor	2.37 (1.69, 3.32)	<.001	1.40 (0.94, 2.07)	0.096
Problem Solving	3.48 (2.39, 5.07)	<.001	1.41 (0.88, 2.24)	0.150
Personal Social	4.03 (2.71, 5.99)	<.001	1.76 (1.07, 2.88)	0.025
Child Age	1.01 (0.98, 1.03)	0.647	1.03 (1.00, 1.06)	0.099

p values represent significance of univariable

^aand multivariable

^blogistic regression analyses.

Discussion

Our goal in the present study was to examine associations between referral concerns, screening results, and diagnostic outcomes for young children evaluated across the Early Autism Evaluation (EAE) Hub system, a statewide primary care network for early screening and diagnosis of ASD. While recent research has focused on the accuracy and potential limitations of screening tools, to our knowledge this is the first study to examine how screening results may be used by PCPs and caregivers in a community sample of young children referred for ASD evaluation. Our findings revealed that referrals for ASD evaluation were made for a large number of children for which there was not a documented concern for ASD and there was low concordance between PCP and caregiver referral concern. While a positive MCHAT-R/F screen was associated with PCP- but not caregiver-reported ASD referral concern, there was a significant relationship between MCHAT-R/F raw scores and both PCP and caregiver ASD referral concern. A divergent pattern of ASQ-3 delays was found to be associated with PCP versus caregiver ASD referral concern. Finally, having a PCP-reported ASD concern at referral, positive MCHAT-R/F screen, and ASQ-3 Communication and Personal Social delays were associated with a significantly higher likelihood of subsequent ASD diagnosis. Below, we discuss each of these findings as well as implications for how these results may advance our approach in developing novel methods for early ASD diagnosis.

Referral Concerns Across a Statewide Early Autism Evaluation Network

In the present study, PCPs endorsed concern for ASD for only 50% of children referred for evaluation in the EAE Hub system. The remainder of PCP-reported referral concerns were in the domains of developmental delay (24%), speech/language delay (14%), and behavior problems (4%). Consistent with a large body of evidence suggesting that caregivers of young children later diagnosed with ASD report language and communication skills as among their earliest developmental concerns (19, 21), our data indicate that the most frequent caregiver-reported referral concern was speech/language delay (40%), followed by ASD (34%). The EAE Hub model specifies that community PCPs make a referral for evaluation at an EAE Hub when surveillance or screening indicates risk for ASD, yet McNally Keehn and colleagues (31) previously reported that referring PCPs use the system for evaluation of children with more broad neurodevelopmental concerns, potentially in order to access more timely local evaluations for their patients (31). An additional explanation may be that, given the highly variable baseline expertise of referring providers (including those in non-pediatric specialties), some may be less skilled in identifying-ASD specific concerns and/or more hesitant to clearly indicate concern for ASD despite identifying need for referral based on developmental surveillance or screening. Further, time constraints and poor communication between PCPs and specialists have been widely documented (36) and may also contribute to our findings. While other existing early evaluation models have similar referral criteria (37, 38), there has been no specific reports on adherence to stringent referral criteria; our findings highlight that improving referral communication and documentation may be a fruitful area of focus for future process improvements across early evaluation efforts. Although the vast majority (>97%) of children were likely to benefit from evaluation (i.e., due to receipt of an ASD or non-ASD neurodevelopmental diagnosis and subsequent intervention recommendations), the EAE Hub system has not yet been expanded to fulfill statewide capacity needs for ASD evaluation (31) and, as such, accepting broader referrals has the potential to reduce access (39) for those children at highest risk for ASD.

We found only slight to fair agreement between PCPs and caregivers regarding primary referral concerns. For example, both PCPs and caregivers endorsed ASD referral concern for only 22% percent of children receiving evaluations. These results align with Sacrey and colleagues (40) who found poor agreement between parent and clinician ratings of ASD behavior in 12–18 month old children at risk for ASD, but diverge from other studies (41) documenting similar ratings among parents and clinicians on standardized measures of ASD symptoms. Notably, moderating effects of child developmental factors (e.g., IQ, adaptive skills, and behavior problems) and race have been found to impact caregiver-clinician agreement on ASD symptoms (42). It is important to note that these studies have been conducted in the context of national consortium research programs, and thus comparison with findings from this statewide quality improvement network should be undertaken with caution. Regardless, understanding that referring PCPs and caregivers report discrepant referral concerns in children evaluated in community primary care settings raises important questions. Whether there is true disagreement about the nature of a child’s developmental concerns or, instead, the discrepancy is due to poor communication, time constraints and/or hesitancy of the PCP (15–17), deleterious consequences are possible. Miscommunication and lack of shared understanding about the reasons for ASD evaluation referral may lead to mistrust and increased stress during the evaluation process and ultimately impact how caregivers accept their child’s diagnosis and engage their child in interventions (43) known to improve developmental outcomes.

Associations between Screening Results and ASD Referral Concern

Our findings suggest significant associations between MCHAT-R/F raw scores and both PCP and caregiver ASD referral concern. That is, as caregivers endorsed an increasing number of ASD symptoms, both PCPs and caregivers were more likely to report an ASD specific referral concern. In contrast, a categorical screen positive result on the MCHAT-R/F was positively associated with PCP but not caregiver ASD referral concern. PCPs endorsed a non-ASD primary referral concern for 66% of children with a screen positive result on the MCHAT-R/F. These results bring into question how referring PCPs are using MCHAT-R/F results to inform their developmental concerns in children ultimately referred for evaluation, and whether they are communicating screening results to the child’s caregivers. Although recent research suggests that there are low rates of subsequent referral for diagnostic evaluation after positive ASD screening (30, 44, 45), there has been little attention paid to the qualitative process of how PCPs make decisions about screening and referral. While the critical outcome is that children in our EAE Hub system were ultimately referred for evaluation, as discussed above, there are likely to be downstream consequences if a reciprocal dialogue between the PCP and caregiver regarding the potential meaning of screening results does not occur at the time of referral.

While the ASQ-3 has not been identified as an ASD-specific screening tool, it is widely accepted and used in the context of primary care well-child visits to identify children needing early intervention services (11). Our data suggests that the total number of ASQ-3 delayed domains was significantly associated with both PCP- and caregiver-reported ASD referral concern; more specifically, as the number of delayed domains increase, so does the likelihood that the reporter will endorse an ASD referral concern. There was positive association between ASQ-3 Communication, Problem Solving, and Personal Social delays with PCP-reported ASD referral concern, whereas ASQ-3 Gross Motor and Problem Solving delays were positively associated with caregiver-reported ASD referral concern. While research has previously documented that total number of caregiver concerns (21), as well as caregiver concerns in the domains of communication and motor development (19, 21) predict later ASD diagnosis, to our knowledge there are no previous studies examining how clinicians and caregivers use broadband screening results to make meaning of ASD risk prior to diagnostic evaluation (i.e., at the stage of referral for evaluation).

Associations between ASD Referral Concern, Screening Results, and Child Age and Diagnostic Outcome

When examined independently, both PCP and caregiver ASD referral concern were significant predictors of ASD diagnostic outcome, though only PCP ASD referral concern remained a significant predictor when both variables were entered into the multivariable model. Children with a PCP-reported ASD concern at referral were 1.77 times more likely to have an ASD diagnostic outcome than those with PCPs reporting more broad non-ASD developmental concerns. In contrast, other studies have found concordance between caregiver ASD concern and diagnostic outcome (20, 21) and that caregivers’ ASD concerns may be better predictors of diagnostic outcome than those of PCPs (40). Our differential findings may be due in part to methodological differences, for example in the populations studied (i.e., research versus community samples) and measures used to quantify and define concern (i.e., specific standardized measures versus general endorsement of concern). Given that caregiver referral concerns were documented by the EAE Hub (i.e., not based on direct caregiver report), our findings may also reflect a more problematic phenomenon of failure to adequately solicit the caregiver’s perspective about their child’s developmental concerns. Such a phenomenon is too often reported by caregivers of children with ASD (17, 18, 46), and speaks to the ongoing need to support PCPs in developing and implementing these surveillance skills.

Both MCHAT-R/F raw scores and categorical risk were significantly associated with ASD diagnostic outcome when examined independently. Children with a positive MCHAT-R/F screen were 2.41 times more likely to have a diagnostic outcome of ASD. However, while 87% of children diagnosed with ASD in the EAE Hubs screened positive on the MCHAT-R/F, 62% of children with a non-ASD diagnostic outcome also screened positive. Our findings cannot be directly compared to large population-based studies, but do appear to align with several recent studies documenting limited accuracy of the MCHAT-R/F (28, 29, 47). While previous smaller studies (26, 27, 48) have shown that ASQ-3 Communication delay is associated with ASD diagnostic outcome, our data suggests that both Communication and Personal Social delays increase the likelihood (2.86 and 1.76 times, respectively) that younger toddlers in our sample (i.e., those with MCHAT-R/F data) will be diagnosed with ASD. For all children in our sample (i.e., 18–48 months regardless of MCHAT-R/F completion), ASQ-3 Fine Motor Delays (in addition to Communication and Personal Social delays) increased the likelihood of receiving a diagnosis of ASD 1.69 times. Together, these results provide further support for potential use of a combined screening approach (e.g., 26, 27) in which PCPs use both MCHAT-R/F and ASQ-3 results to inform risk for ASD and further elucidates a pathway to assist PCPs in their decision-making around diagnostic risk and subsequent referral for evaluation.

Strengths & Limitations

The present study has a number of strengths that extend the broad literature regarding referral concerns, use of screening data, and associations between these variables and diagnostic outcomes for young children at high risk for ASD. While previous work in this area has largely focused on young children recruited for research studies, we studied a large community-based sample within a novel statewide primary care network for screening and diagnosis of ASD. As such, our findings are likely to be highly generalizable to the real-world primary care setting where the majority of young children receive developmental surveillance and screening. However, with these strengths comes some limitations and considerations about how the findings of the current study may be interpreted. First, unlike laboratory-based research or clinical trials with strict adherence to standard protocols, our data was gathered from community clinical settings, and thus there is likely to be more variation in how each EAE Hub ascertained and reported data (despite use of a standard online data collection system), as well as how community practitioners adhere to guidelines for the administration of screening measures. Our findings cannot be extended to population-based screening as our sample consisted of only children referred for ASD evaluation. Although we provided comprehensive training to many referring PCPs and EAE Hub clinicians on use of the ASQ-3 and MCHAT-R/F, we cannot be sure that screening tools were administered accurately (i.e., use of MCHAT-R/F Follow-Up procedures and adherence to age-based criteria for administration), scored correctly, and were reliably collected from the referring PCP within the specified timeframe. For example, it is possible that screening measures were collected by the EAE Hub (versus the referring PCP) for children with a long (i.e., > 3 months) latency between referral and evaluation. Diagnostic outcome relied on EAE Hub clinicians and was not independently verified through expert evaluation; as such, we can only attest to the reported associations as they relate to diagnoses made by EAE Hub clinicians. Finally, given that this is a quality improvement initiative and our goal was to minimize burden on the EAE Hubs, we did not collect demographic information on the children evaluated. It is critical that research addresses factors of racial and socioeconomic diversity, equity, and healthcare access in light of studies demonstrating disparities in ASD screening, referral, and diagnosis (22, 30, 44). Future research regarding the EAE Hub system must attend to this call (and, as such, we began collecting demographic data in 2020).

Conclusion

Understanding how community PCPs use surveillance and screening data, the extent to and process by which PCPs and caregivers have shared understanding and engage in collaborative decision-making about referral for evaluation, and how these factors relate to diagnostic risk and outcome has the potential to impact how educational efforts may be tailored for both pediatric health professionals and caregivers of young children, as well as inform the development of more efficacious early identification approaches. While attention has been paid to improving screening implementation and measures, we must also address the significant research to real-world practice gap that exists through developing and disseminating educational opportunities to community primary care clinicians with front line responsibility for ASD risk detection. This type of education can not only equip pediatric health care clinicians with needed knowledge and skills to implement state-of-the-science early identification practices, but also foster collaborative communication and relationships between families and clinicians to support engagement in the diagnostic and intervention processes known to improve developmental outcomes for young children with ASD.

List of Abbreviations

ASQ-3

Ages and Stages Questionnaire, Third Edition

AAP

American Academy of Pediatrics

ASD

Autism spectrum disorder

EAE Hub

Early Autism Evaluation Hub

MCHAT-R/F

Modified Checklist for Autism in Toddlers, Revised with Follow-Up

PCP

Primary care provider