Validity of the Autism Mental Status Exam in Developmental Pediatrics and Primary Care Settings

Abstract

The Autism Mental Status Exam (AMSE) is a brief clinician-completed observational instrument that has shown promise in identifying autism spectrum disorder (ASD) in a referred sample. Our study explores the feasibility of the AMSE in both developmental pediatric and primary care samples. Fifty-three toddlers with ASD and other disabilities were scored using the AMSE and compared with 55 typically developing toddlers. AMSE scores differed significantly between ASD, non-ASD developmental disability, and neurotypical groups. A cutoff score on the AMSE of ≥5 for ASD maximized sensitivity (81.2%) and specificity (90.5%). Score differences between groups suggest that the AMSE may be useful in a clinical setting to help identify children with possible ASD.

Introduction

Early and intensive behavioral intervention yields optimal long-term developmental outcomes for young children with autism spectrum disorder (ASD).^1-3 Despite early parental concerns that often arise prior to 15 to 18 months of age,^4,5 some providers inappropriately reassure families about their child’s development, delaying the referral process.^6,7 Unfortunately, the average time delay between parents first expressing concern and the diagnosis of ASD is 2.7 years.⁷ Accurate screening by pediatric providers may facilitate early identification of ASD and timely access to early intervention.

Although pediatric providers are trained in physical examination skills, assessing a child’s development in a busy pediatric practice is often difficult. Lack of time and familiarity with screening tools are barriers that interfere with adequate developmental screening.⁸ Furthermore, many pediatric offices do not adhere to the American Academy of Pediatrics recommendations regarding 18- and 24-month screenings for ASD and referral to both an ASD specialist and early intervention program, with only 61% of children who failed initial screens being referred.⁹

Although there are screening tools available to identify young children with symptoms of ASD,¹⁰ most are based only on parent or caregiver report and do not incorporate clinical observations made by medical professionals. Even though screening instruments are readily accessible, they vary in their ease of scoring and interpretability and have the possibility of response bias with caregivers over- or underreporting of a child’s ASD symptomatology.¹¹

The Autism Mental Status Exam (AMSE) is a clinician-completed quick 8-item assessment that measures social, communicative, and behavioral functioning of an individual suspected of having ASD based on direct clinical observation and parent report.¹² The AMSE was developed to address the lack of a standardized observational assessment for ASD in underserved and underresourced clinical populations.¹³ Grodberg et al report that the AMSE does not add extra work to a provider encounter, but rather provides a clear and quantitative method of documenting passively observed behaviors.^13,14 Recently, a study applying the AMSE to a high-risk population of 45 toddlers aged 18 months to 5 years, showed that a cutoff score of 6 produced a sensitivity of 94% and a specificity of 100% in identifying children with ASD based on DSM-5 (Diagnostic and Statistical Manual of Mental Disorders, 5th edition) criteria.¹⁴ A limitation to date is the lack of data on AMSE scores in neurotypical controls outside of a subspecialty clinic.

The purpose of this study is to determine an AMSE cutoff that yields optimal sensitivity and specificity in differentiating young referred children with and without ASD and to determine the accuracy of AMSE scores in differentiating children with ASD, children with developmental delay (DD), and neurotypical (NT) controls.

Methods

Participants

A sample of 55 children without identified developmental disabilities (NT group) was recruited from a university-based general pediatric clinic. All children in the NT group had at least one negative Modified Checklist for Autism in Toddlers, no parental concern about their child’s development, and no history of DDs at the time of enrollment. A second group of 53 children were recruited from a developmental pediatric clinic and who were referred for concerns regarding speech, language, or ASD. The children in the referred group were individually administered developmental tests including the Battelle Developmental Inventory, Developmental Assessment of Young Children, and Stanford-Binet. All children in this group were ultimately diagnosed with ASD (ASD group) or other DD (DD group).

Autism Mental Status Exam

The AMSE requires an examiner to observe and document a patient’s social, communicative, and behavioral functioning in a clinical encounter.¹² For each of 8 items scores between 0 and 2 are documented with possible total scores ranging from 0 to 16. Higher scores reflect greater symptom severity. The items include (1) Eye Contact, (2) Interest in Others, (3) Pointing Skills, (4) Language, (5) Pragmatics, (6) Repetitive Behaviors, (7) Unusual or Encompassing Preoccupations, and (8) Unusual Sensitivities. When scoring the AMSE, social items (eye contact, interest in others, pointing, and language) must be observed, while other items (pragmatics, repetitive behaviors, unusual or encompassing preoccupations, and unusual sensitivities) can be reported by parents or observed. Additional details regarding AMSE use and scoring are available on the website, which provides a free online curriculum including video-simulated cases (http://autismmentalstatusexam.com/).¹⁵

Procedure

The comprehensive diagnostic clinic used a standardized approach, which included a semistructured clinical interview with the parents, a play-based structured observation session, speech and language testing, and the Checklist for Autism Spectrum Disorder (CASD)^16-21 completed by parents and teachers. Children in the referred group received a clinical diagnosis that was determined by a board-certified developmental pediatrician with input from a speech-language pathologist and pediatric neuropsychologist. If the diagnosis was unclear, the child returned for an Autism Diagnostic Observation Schedule administered by the neuropsychologist. Pediatric residents and medical students received training from the AMSE online training curriculum, which included expert explanation of the scoring instructions as well as 4 training cases.¹⁵ Trained pediatric residents completed the AMSE based on the standard scoring manual without knowing the child’s ultimate diagnosis. For NT controls, trained medical students observed children during their well or acute visits and completed the AMSE. Initially, in both the referred group and NT group, multiple providers would score the AMSE to ensure interrater reliability. Previous studies have demonstrated good to excellent interrater reliability.¹²

Data Analysis

Statistical analysis was performed in MATLAB R2016a (MathWorks, Inc, Natick, MA). As data could not be assumed to be normally distributed, more conservative nonparametric statistics were used when possible. A conventional significance threshold of P = .05 was used. An AMSE cutoff score was determined that maximized overall diagnostic accuracy (based on the clinical diagnosis and the CASD) and minimized the difference between sensitivity and specificity. Using this cutoff score, sensitivity (the number of children correctly identified as having ASD by the AMSE score divided by the total number of children with ASD) and specificity (the number of children correctly identified as not having ASD divided by the total number of children without ASD) were calculated. Distributions of AMSE scores from all 3 samples (ASD, DD, and NT) were compared using the Wilcoxon rank sum test.

Receiver operating curve (ROC) analysis was performed to measure discrimination within the referral population. Sensitivity and specificity of the AMSE using our best estimate clinical diagnosis were calculated at all integer thresholds. Overall ROC curves were plotted and areas under the curve calculated using the trapezoidal method.

Post hoc subgroup analysis to elicit the effects of gender and race on AMSE scores within each sample was performed. The Wilcoxon rank-sum test (and for race, its extension, the Kruskal-Wallace test) was used to determine the degree of significance. A linear model was used to determine the effects of cognitive scores on AMSE scores. This was extended to an ANCOVA (analysis of covariation) to estimate the relative size of effects of presence/absence of an ASD diagnosis versus the effects of cognitive scores.

Ethical Approval and Informed Consent

This study was approved by the Penn State College of Medicine Institutional Review Board (No. PRAMS039326EP). Informed consent was obtained from parents of all participants. The institutional review board granted a waiver of documentation of consent for typical controls recruited from the university-based general pediatric clinic (NT controls). Families were presented with a summary of research and the parent provided verbal consent at that time. Written informed consent was obtained for participants recruited from the developmental pediatric clinic (referred group). During the initial clinic visit, parents were provided with the summary of research and consent forms to take home to review. At a later date, a member of the research team called parents from the referred group to review the consent form in detail. If the parent/guardian decided to enroll their child, they signed the consent form and mailed it back to the principal investigator.

Results

The 55 children in the NT control group ranged in age from 18 to 68 months (mean [M] 37.9 months, SD 15.2), and 45% were male. The 32 children with ASD were 19 to 66 months of age (M 37.7 months, SD 10.8), 87.5% were male, and 71.9% were Caucasian. The mean cognitive standard score on developmental tests (eg, Battelle Developmental Inventory, Developmental Assessment of Young Children, and Stanford-Binet) was 71.2 (SD 14.9). The 21 children with other DD (DD group) were 17 to 66 months of age (M 42.1 months, SD 12.0), 67.0% were male, 76.2% were Caucasian, and the mean cognitive standard was 81.9 (SD 14.5).

AMSE scores did not differ significantly between males and females (P values for NT, DD, and ASD groups = .86, .20, and .23, respectively) or between racial groups (P values for DD and ASD groups = .88 and .88, respectively). Cognitive scores were significantly lower in the ASD group (median 72) than the DD group (median 82). Cognitive scores were significantly correlated with AMSE scores (Figure 1) in the ASD group (R² .29, P = .002) but not in the DD group (R² .05, P = .35). An ANCOVA comparing the effects of cognitive standard score versus ASD/DD diagnosis demonstrates greater contribution of diagnosis (F = 31.25, P < .001) than cognitive standard score (F = 7.27, P < .01).

Figure 1.

Autism spectrum disorder (ASD) group (squares) and developmental disability (DD) group (circles) shown. Linear regression lines are shown. Line of best fit in ASD predicts Autism Mental Status Exam (AMSE) = 16.1-0.13 * cognitive standard score.

The AMSE cutoff score that maximized agreement with the referred children’s clinical diagnosis and minimized the difference between sensitivity and specificity was ≥5, yielding a sensitivity of 81.2% and specificity of 90.5% (Figure 2). Agreement between the AMSE and CASD using a cutoff of ≥5 was 83.0% (true positive 77.1% and true negative 94.4%). AMSE scores differed significantly between the 3 groups (ASD vs DD, P < .001; ASD vs NT, P < .001; DD vs NT P < .011). For children with ASD, AMSE scores ranged from 0 to 13 (M 7.4, SD 3.5). Children in the DD group had AMSE scores ranging from 0 to 6 (M 1.9, SD 1.7), and AMSE scores for the NT group ranged from 0 to 2 (M 0.8, SD 0.8).

Figure 2.

Autism Mental Status Exam receiver operating curve (ROC). ROC analysis produced the curve shown above. Area under the curve by the trapezoidal method is 0.905.

Distributions of AMSE scores from all populations (ASD, DD, and NT controls) are summarized in Figure 3. In pairwise comparisons of the groups, AMSE scores of all 3 groups are significantly different from each other (rejecting the hypothesis of equal medians). The ASD versus DD referral patient groups showed a significant difference in AMSE scores at P = 9.7 × 10⁻⁷. The ASD referral versus control patient groups showed significant difference in AMSE scores at P = 2.4 × 10⁻¹². DD referral versus control patient groups showed significant difference in AMSE scores at P = .011.

Figure 3.

Autism Mental Status Exam (AMSE) score distributions. Boxplots show the AMSE score distributions in the neurotypical control general pediatric population, developmental disability (DD) referral population, and the autism spectrum disorder (ASF) referral population. In each box, the central mark indicates the median; bottom and top edges of the box indicate 25th and 75th percentiles, respectively. Whiskers extend to the most extreme data points within 1.5× the interquartile range of the end of the box; any outlier individuals outside this range are shown as “+.”

Discussion

Accurate identification of young children at risk for ASD and prompt referral provides the best opportunity for optimal developmental outcomes, and the well-child visit is and should remain the first assessment in detection and referral for neurodevelopmental disorders. Barriers to detection go beyond screening instruments and include lack of time and specific training.^22,23 Currently, a factor contributing to delayed diagnosis and referral is reassurance with a “wait and see” approach. This may be mitigated by improving providers’ ability to accurately observe development using a validated standardized assessment, such as the AMSE.

This study demonstrated that the AMSE scores differ significantly between children with ASD, other developmental disabilities, and neurotypical groups. We acknowledge that groups examined cannot be directly compared, but this study is among the first to compare the distributions of AMSE scores in these groups. Under these idealized circumstances, a cutoff of ≥5 identifies 81% of children with ASD and 90% of children with other developmental disabilities.

While the AMSE demonstrated high classification accuracy in this study, some limitations should be considered. Study design limitations prevented collection of certain demographic information in neurotypical controls (ie, race and socioeconomic status). Since residents scored the AMSE in the referred group and medical students administered the AMSE in the control group, interobserver agreement must be considered. Furthermore, sample sizes were small and may not be representative of a general population.

AMSE scores can be considered suggestive of a developmental disability but should not be considered adequate to make a diagnosis of ASD in isolation. Of note, AMSE scoring does not account for age, overall cognitive level, or language skill level. It also represents a short sampling of behavior in a single setting at a single point in time. It is well known that children with other developmental disabilities such as intellectual disability and language disorders can have behaviors similar to those with ASD, and careful longitudinal evaluation is often needed to fully understand a child’s phenotype and ultimate diagnoses. Of note, individuals with non-ASD developmental disabilities demonstrated intermediate AMSE scores between those of typical controls and those of individuals with ASD. This suggests that some AMSE scores below our suggested threshold perhaps should be considered “borderline” or “at risk” and closely followed. We recommend that AMSE be used in combination with other screening tools and parental questionnaires to improve a provider’s ability to suspect ASD and thus refer the child for a specialty evaluation and early intervention in a timely manner.

Implications

The AMSE may allow for earlier identification of ASD and other DD and timelier referral leading to improved access to critical interventions that are designed to maximize a child’s developmental potential. Future studies assessing the feasibility and outcomes of implementing the AMSE in primary care are needed. While we suspect that the AMSE would not place much additional burden on the pediatric provider, additional research is needed to determine if modifications in practice would be necessary.