A systematic review of observational, naturalistic, and neurophysiological outcome measures of nonpharmacological interventions for autism

Abstract

Objective:

Naturalistic and neurophysiological assessments are relevant as outcome measures in autism intervention trials because they provide, respectively, ecologically valid information about functioning and underlying neurocognitive mechanisms. We conducted a systematic review to highlight which specific neurophysiological techniques, experimental tasks, and naturalistic protocols have been used to assess neural and behavioral functioning in autism intervention studies.

Methods:

Studies were collected from four electronic databases between October 2019 and February 2020: MEDLINE (via PubMed), PsycINFO, LILACS, and Web of Science, and were included if they used structured observational, naturalistic, or neurophysiological measures to assess the efficacy of a nonpharmacological intervention for ASD.

Results:

Fourteen different measures were used by 64 studies, with the Autism Diagnostic Observation Schedule the most frequently used instrument. Thirty-seven different coding systems of naturalistic measures were used across 51 studies, most of which used different protocols. Twenty-four neurophysiological measures were used in 16 studies, with different experimental paradigms and neurophysiological components used across studies.

Conclusions:

Cross-study variability in assessing the outcomes of autism interventions may obscure comparisons and conclusions about how different behavioral interventions affect autistic social communication and underlying neurophysiological mechanisms.

Introduction

Autism spectrum disorder (ASD) is a developmental condition characterized by difficulties in social communication and interaction, restricted interests, and repetitive behaviors.1 These symptoms persist throughout life in most individuals.2 Many autistic individuals also have mood and anxiety disorders3 and impairments in cognitive4 and adaptive5 skills. Autistic symptoms and associated difficulties can result in reduced autonomy and impaired quality of life.6 The severity of autistic symptoms varies considerably across individuals.7 Correspondingly, adaptive functioning skills and the amount of daily life support required by autistic people vary from minimal to substantial.8 There is also a relationship between ASD severity and intellectual ability, with lower intelligence quotient (IQ) related to more severe autistic symptoms in many cases.8,9

Difficulties with social interaction include reduced ability to initiate social interactions, engage in relationships, and maintain social reciprocity.10 Communication problems include delayed or atypical speech and language development, difficulties in nonverbal communication, and low responsiveness in situations of shared attention.10,11 Restricted and repetitive behaviors (RRBs) include inflexibility and perseveration in interests and activities (also called insistence on sameness), motor stereotypies, and repetitive speech, routines, and rituals,10 as well as hyper- or hypo-reactivity to sensory input or unusual interests in sensory aspects of the environment.1 These symptoms are associated with difficulties in social perception and cognition, as well as atypical patterns of neurophysiological activity during social processing.12-16 Other neurocognitive alterations, such as reduced neural connectivity, have also been associated with ASD.17-20 These neurocognitive atypicalities seem to begin early in life, likely as a result of an altered trajectory of neural and behavioral development.21

ASD seems to emerge from an interaction between pre-existing neurodevelopmental vulnerabilities (e.g., genetics) and the prenatal environment.22,23 However, the postnatal environment is crucial in terms of managing ASD symptoms, coping with functional and neurocognitive difficulties, and ensuring quality of life.21,22 Based on this concept, several intervention programs have been designed to enhance the development of autistic children, some of them focusing on social communication and parent-child interaction24-27 and others focusing on sensory issues,28 RRBs,29 or co-occurring difficulties like anxiety.30,31 In autistic adolescents and adults, interventions based on social cognition training, cognitive-behavioral therapy, mindfulness, and applied behavior analysis have also been developed to target social communication skills and cognitive and emotional functioning.32

Given the range of intervention programs developed for autism, it is challenging for parents, clinicians, and public policy makers to select a particular intervention (or set of interventions) for real-world implementation outside of the research context. This problem is further compounded by the variability in response to treatment reported across intervention trials in autism.33,34 This variability likely reflects the heterogeneity of the ASD phenotype as well as differences in characteristics of intervention studies, such as the age of the child when the intervention was delivered and the type and intensity of treatment approach.33,34

Another factor that likely contributes to inconsistent findings concerning the efficacy of autism interventions is variability in the assessments used as outcome measures.35,36 For instance, to assess the effect of interventions on social communication skills, some studies have used observational instruments37,38 and others have used rating scales39-42 or naturalistic measures.43,44 This cross-study variability in outcome measure selection has been highlighted in previous narrative and systematic reviews.36,45-47 For example, Anagnostou45 found that 37 different outcome measures of social communication abilities had been used in clinical trials of autism interventions, but concluded that only six of them were appropriate. Likewise, McConachie46 reported that 188 different outcome measures had been used to assess progress and outcomes in core autism symptoms associated with interventions in children under 6 years of age. Analyzing variability in the selection of outcome measures in clinical trials, Provenzani36 found 327 different outcome measures of core ASD characteristics and co-occurring psychiatric symptoms, 69% of which were used only once.

However, those previous reviews focused on the use of observational instruments or parent-report rating scales as outcome measures. Although parent-report rating scales provide easy, quick, and efficient measurements of behaviors, they are also easily biased by factors such as the rater’s reading level or linguistic skills, learning disabilities, psychological maturity, emotional awareness, psychopathology, and other general contextual factors.48 Moreover, parents are rarely blinded to intervention group status. This is problematic because recent work has shown that parents tend to report reductions in their child’s autistic symptoms even in the absence of active intervention.49 Thus, rating scales may not be the most appropriate outcome measures for clinical trials testing interventions for autism.

Observational measures such as the Autism Diagnostic Observation Schedule (ADOS)10 provide more objective and standardized measures of autistic symptoms. However, there is evidence to suggest they may not be sufficiently sensitive to detect intervention-related changes in behavior.50 Furthermore, many observational measures, such as the ADOS, are expensive and require high levels of training in administration and coding, which limits the utility of these measures for low-resource settings where interventions are urgently needed.

A promising alternative method of evaluating outcomes of autism interventions is to assess the child’s functioning in everyday situations, using naturalistic measures. Naturalistic measures are ecologically valid behavioral observation methods used to assess behaviors of interest during a social interaction between the child and another person. The interaction is videotaped and coded offline using a predefined coding system.51 Naturalistic measures are particularly relevant to evaluating the outcomes of autism interventions because they provide a more realistic assessment of the individual’s social communication functioning.51,52

Another approach to assessing outcomes of autism interventions is to examine changes in neurophysiological activity associated with the neural systems underlying social-communicative functioning. Neurophysiological measures may be more objective than observer- or parent-evaluated symptom assessments, less susceptible to response bias, and are informative as to the underlying mechanisms of symptom improvements, thereby enhancing comprehension of intervention outcomes,53 although the clinical salience of neurophysiological measures is less clear. No prior systematic review has examined the extent to which naturalistic measures have been used as outcome measures in intervention studies, and only one review investigated neurophysiological outcome measures, such as electroencephalography (EEG).54

Therefore, we aimed to conduct a systematic review of naturalistic and neurophysiological outcome measures that have been used in studies of nonpharmacological interventions for ASD. We highlighted which specific neurophysiological techniques, experimental tasks, and naturalistic protocols have been used to assess neural and behavioral functioning, and the consistency with which those measures have been employed across intervention studies. For completeness and comparison, we also reviewed the specific observational outcome measures that have been used across intervention studies. We focused the review on nonpharmacological intervention studies, since pharmacological treatments usually target co-occurring difficulties, such as aggression, rather than autism itself. The information produced by this review will be useful for future intervention studies that aim to assess observational, naturalistic, and neurophysiological outcomes in ASD.

Methods

Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) statement and pre-registration

The protocol for this systematic review was pre-registered on PROSPERO (CRD42019137004). The methodology and reporting of outcomes are consistent with the PRISMA statement.

Search strategy and selection criteria

A systematic literature search was carried out to identify studies that assessed the efficacy of nonpharmacological interventions for ASD on structured observational, naturalistic, and neurophysiological outcome measures. Studies were collected from four electronic databases between October 2019 and February 2020: MEDLINE (via PubMed), PsycINFO, LILACS, and Web of Science. The searches included full-length articles written in Portuguese, English, or Spanish that were accepted for publication in peer-reviewed journal up to February 2020. Indexed terms were selected from the MeSH and DeCS databases, and the electronic search was conducted according to the PICO (Patient, Intervention, Comparison, and Outcome) methodology. The search query used in all four databases was: (“Autistic Disorder” OR “Autism Spectrum Disorders” OR “autism spectrum disorder” OR “ASD” OR “autism”) AND (“Social Communication Disorders, Therapy” OR “Autistic Disorder, Therapy” OR “Autism Spectrum Disorders, Therapy” OR “Communication, Therapy” OR “Social Skill, Therapy” OR “Social Behaviors, Therapy” OR “intervention” OR “treatment” OR “therapy” NOT “Drug Therapies” NOT “Therapy, Drug” NOT “Pharmacotherapy”) AND (“Social Skill” OR “Communication” OR “Verbal Behaviors” OR “Nonverbal Communications” OR “asd symptom severity” OR “social impairment” OR "communication impairment” OR “social communication skills” OR “social communication ability” OR “social cognition” OR “diagnostic techniques, neurologic” OR “neuroimaging, functional” OR “neuroimaging” OR “electrophysiology” OR “electroencephalography” OR “EEG” OR “nirs” OR “near-infrared spectroscopy” OR “fmri” OR “functional magnetic resonance imaging” OR “magnetic resonance imaging” OR “MRI”).

In the screening phase, the titles and abstracts of all studies retrieved by the electronic search were evaluated. Studies were selected for full-text reading if they reported the use of a nonpharmacological intervention for ASD in the title or in the abstract. Reporting of the use of an observational, naturalistic, or neurophysiological outcome measure in the title or abstract was not considered as an inclusion criterion in the screening stage because we believed that studies might include these measures but not report them in the abstract. During full-text reading, studies were included if they used structured observational, naturalistic, or neurophysiological measures to assess the efficacy of a nonpharmacological intervention for ASD. Studies were excluded if they were systematic reviews, case studies, studies that used interventions that involved ingestion of any substance, studies with interventions for conditions other than ASD or for children “at risk” for ASD, studies with interventions targeting symptoms not specifically related to ASD (such as sleep problems, depression, tone of voice, catatonia) or parental well-being, studies that did not use at least one structured observational, naturalistic, or neurophysiological measure, or studies with n<10.

Data extraction, coding, and synthesis

Study eligibility and data extraction were conducted independently by three investigators (PBGG, FMS, and LS), with each paper reviewed by two authors. Disagreements were resolved by discussion between the two investigators. Extracted information included: study population/design, participant characteristics (age, sex, socioeconomic status, race/ethnicity), clinical characteristics (IQ, language skills), study objectives, intervention description, measures used for clinical characterization (diagnostic measures), outcome measures at pre- and post-intervention assessments, and a descriptive summary of the results. A narrative synthesis of the findings was then created, with studies grouped according to the type of outcome measure used. The descriptive summary of results is not presented here because this review focused on outcome measures and assessment methods, rather than trial outcomes.

Results

After full-text reading, 92 non-duplicate studies met the inclusion criteria55-141 (Figure 1, and Table S1 ^{(444.7KB, pdf)} , available as online-only supplementary material.). A total of 75 different measures were used, consisting of 14 structured observational measures (Table 1), 37 naturalistic measures with coding systems grouped into seven categories (Table 2), and 24 neurophysiological measures (Table 3). Fifty-six percent (k = 43) of the instruments/methods were used only once to assess intervention outcomes. Few studies (k = 4) combined more than one type of these measures to assess intervention outcomes. One study used a structured observational measure only, 20 used naturalistic measures only, and three used neurophysiological measures only. Most studies (k = 69) combined one or more of these types of measure with rating scales or cognitive assessment. Demographic information for the samples of each study included can be found in Table S2 ^{(444.7KB, pdf)} , available as online only supplementary material.

Figure 1. Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) flowchart. ASD = autism spectrum disorder.

Table 1. Structured observational outcome measures used to assess autism intervention effects.

Structured observational measure	Behaviors of assessed with each measure	Procedure	First author (reference)
Autism Diagnostic Observation Schedule	Autistic traits	Series of structured and semi-structured tasks that involve social interaction between the examiner and the child under assessment. Over approximately 30 min, the examiner provides a series of social and communication opportunities for the subject and assigns these to predetermined observational categories	Aldred,55 Aldred,56 Arabi,59 Ben-Itzcha,61 Ben-Itzchak,62 Boyd,65 Byford,66 Estes,76 Fletcher-Watson,37 Green,83 Gutstein,84 Herbrecht,38 Howlin,87 Kasari,94 Lerna,104 Locke,105 Oosterling,110 Paul,114 Rogers,116 Solomon,124 Tsiouri,126 Ventola,130 Wetherby,132 Wong,134 Yoo,138 Yun139
Assessment of Basic Language and Learning Skills – Revised	Social interaction skills	Provides information about a child’s existing skills in 25 different and separated areas and the conditions under which the child uses these skills. The assessment is conducted via observation of the child’s behavior in each skill area. The instructor provides a stimulus to the child (verbal, hand-over-hand, non-verbal), and, depending on what the child does, determines their skill level	Zhao140
Brief Observation of Social Communication Change	Changes in social communication behaviors	Consists of a coding system to identify changes in social communication behaviors over relatively short periods of time by quantifying subtleties in both the frequency and the quality of the behaviors. The coding is based on videos of free-play interactions between a caregiver and the child	Fletcher-Watson37
Children’s Global Assessment Scale	Level of daily life functioning	Consists of a numeric scale (1-100) used by clinicians to rate the child/adolescents most impaired level of general functioning during the period rated by selecting the lowest level which describes his/her functioning on a hypothetical continuum of health-illness	Choque Olsson,71 Herbrecht,38 Jonsson,89 White133
Clinical Global Impression	Changes (how much symptoms have improved or worsened) in parent-child interaction and/or restricted and repetitive behaviors	Consists of a three-item observer-rated scale that measures illness severity, global improvement and therapeutic response. Comprises two companion one-item measures evaluating the following: a) severity of psychopathology from 1 to 7; and b) change from the initiation of treatment on a similar seven-point scale	Choque Olsson,71 Grahame,82 Hardan,85 Jonsson,89 Kouijzer,100 Oosterling,110 Ventola,130 White133
Contextual Assessment of Social Skills	Social interaction skills	Consists of two videotaped role play conversations between the participant and two different confederate. Prior to the role play, participant and confederate are instructed to act as if you had recently joined a new club or social group. Later, participants’ verbal and nonverbal behaviors are coded	Dolan75
Early Social Communication Scales	Social interaction skills	Consists of a videotaped structured observation instrument that provides measures of individual differences in nonverbal communication skills during 17 interaction tasks with the examiner	Almirall,57 Carter,68 Goods,80 Kaale,90 Kasari,91-93,96 Lawton,101 Siller120
Functional Emotional Assessment Scale	Social-emotional skills	Provides a systematic and observational assessment of the child and caregiver’s functional emotional capacities. These capacities include the child’s ability to organize play interactions with objects and persons, to self-regulate mood and organize attention, to form an attachment with the caregiver, to engage in reciprocal emotional interactions and communications, and to represent feelings and ideas and engage in emotional thinking through play interactions	Solomon124
Griffiths’ Mental Developmental Scale	Receptive and expressive language and personal-social skills	Provides an overall measure of a child’s development through direct observation of the child during interaction tasks with the examiner. Assesses five areas: foundations of learning, language and communication, eye and hand coordination, personal-social-emotional, and gross motor skills	Lerna,103,104
	Learning skills, language and communication, eye and hand coordination, personal-social skills, and gross motor.		Wong135
Preschool Language Scale	Receptive and expressive language skills	Consists of an interactive assessment of receptive and expressive language skills and behaviors considered to be language precursors during play tasks with the examiner	Byford,66 Gengoux,79 Green83
Symbolic Play and Language Comprehension in Autistic Children	Play behavior	The examiner sequentially introduces sets of toys in order to elicit child’s play behaviors. The session is videotaped and later coded for the number of different acts with toys within the same level of play	Kasari95
Symbolic Play Test	Play behavior	Assesses the early skills required for language development through play with miniature objects in four sets of toys that are presented to a child in a specific order	Parsons,113 Wong134,135

Table 2. Naturalistic outcome measures used to assess autism intervention effects.

Naturalistic measure/behaviors of interest coded	Procedure	First author (reference)
Video-recorded parent-child or teacher-child play interaction
Parental responsiveness to the child	Parent-child play interaction with a standardized set of toys – The percentage of parental acts that were responsive to the child’s focus was coded.	Paul114
	10-min PCI with a standard set of toys – The number of total parent communication acts (comments, statements, acknowledgements or social interaction maintaining the child’s responses) which were synchronous to the child’s play behaviors were coded.	Aldred56
	5-min parent-child play interaction in a child-led play situation according to the Dyadic Parent-Child Interaction Coding System – Two categories of parents’ communication skills were coded: positive following (the sum of verbal “do skills” [i.e., behavior descriptions, reflections, and praises]), and negative leading (the sum of verbal “don’t skills” [i.e., commands, questions, and critical statements])	Furukawa,78 Zlomke141
Child’s social communication skills	Structured Laboratory Observation – 10-min PCI with a standardized set of toys in which the parent was asked to elicit as much language from the child as possible to assess the total frequency of the child’s functional utterances. These utterances were classified as unintelligible, imitative, verbally prompted, non-verbally prompted, or spontaneous.	Gengoux79
	Natural Language Sample – 20-min caregiver-child interaction with a standardized set of toys to assess the child’s spontaneous expressive language ability, coded in terms of comments, requests, and verbal protests made by the child	Kasari,96 Almirall57
Child’s Attachment-related behaviors	Structured interaction between parent and child consisting of three parts. Part 1: 10-min parent-child play interaction; Part 2: a stranger entered the assessment room and mothers were asked to step outside. The stranger remained with the child, engaging him or her in play; Part 3: after about 2 min of separation, mothers re-entered the room, following three instructions: a) calling the child’s name loudly from outside the door; b) pausing momentarily after opening the door; and c) greeting the child naturally thereafter. Children’s behaviors during the reunion part were videotaped and coded for attachment-related behaviors using two scales: the Proximity and Contact Seeking Behaviors Scale (evaluates the intensity of a child’s effort to regain contact with, or proximity to, their mother) and Avoidant Behaviors Scale (evaluates the intensity and duration of the child’s avoidance toward their mother)	Siller121
Child’s play behavior and/or joint attention	Structured Play Assessment – The child is presented with four different set of toys at a table to assess functional play types, SP types and play level.15-min parent-child play interaction with a standardized set of toys to assess the child’s JA (frequency of JA initiations and responses) and play behaviors (the highest level of mastered play and SP types).	Kasari,91,92
	10-min parent-child play interaction using the Precursors of Joint Attention Measure observational protocol to assess the child’s focus on faces (defined as the child looking once or more at any part of the parent’s face), turn-taking (child performed one of at least two related actions in concert with a parent action within no more than two consecutive intervals), responding to JA (after the parent attempted to draw the child’s attention to an object the child alternated looks between the parent’s face and the object for the apparent purpose of sharing social interest), and initiating JA (child alternated looks between the parent’s face and an object for the apparent purpose of drawing the parent’s attention to the object)	Schertz43
Parent and child’s social communication skills	10-min parent-child play interaction – The number of occurrences of parent social communication behaviors (follow-in commenting, linguistic mapping, expansions, prompts, and redirects) and child social communication behaviors (prompted communication acts, spontaneous verbal communication acts, and spontaneous nonverbal communication acts) were coded.	Venker128
	7.5 min PCFP interaction – The child’s attention and initiation were coded using the Child Behavior Rating Scale. The parent’s behavior was coded for domains of responsiveness/child-oriented, affect/animation, achievement orientation, and directiveness using the Maternal Behavior Rating Scale.	Solomon124
	5 min PCI playing with a puzzle – Child’s social competence (autonomy, responsiveness, empathy, motor regulation, and emotional regulation) and the caregiver’s child rearing competence (respect for autonomy development, respect for responsiveness development, respect for empathy development, respect for cognitive development, and respect for social-emotional development) were coded using the Interaction Rating Scale.	Ichikawa88
	Parent-child play interaction with a standardized set of toys – The Dyadic Communication Measure for Autism was used to code child and parent communication acts (verbal and non-verbal behaviors that have communicative intent), child and parent shared attention (episodes in which the parent and child shared attentional focus), synchronous parental communication (comments, statements, acknowledgments, or social interaction which maintained the child’s responses), and asynchronous parental communication (responses aimed at redirecting, controlling or making demands on the child to respond).	Aldred,55 Green,83 Rahman115
	10-min mother-child play interaction with a standardized set of toys to assess parent verbal behavior (if synchronized or unsynchronized with the child’s attention and actions) and child toy-directed attention (the proportion of observation time children were attending to the target toys).	Siller120
	PCFP interaction to assess nonverbal parent responsivity (when a parent aided the child in their play, performed the same action as the child with a similar object, expanded on the child’s play, or responded to a child’s request), verbal parent responsivity (when a parent described or talked about the child’s current focus of attention, or verbally expanded upon a child’s communication act, without directing the child’s behavior), and frequency of child intentional communication (gestures or nonword vocalizations during which the child coordinated attention between the message recipient and an object or salient event; conventional gestures, as distal points, head nods, pantomime, with attention to an adult; spoken words or signs used in a non-imitative manner)	Carter68
Parent-child/teacher-child joint engagement	10-min parent-child play interaction with a standardized set of toys to assess child play diversity (different types of play) and joint engagement between caregiver and child (the child and caregiver engaged with the same activity and with both aware of the roles of the other).	Kasari96
	10-min parent-child and teacher-child play interaction with a standardized set of toys to assess joint engagement (when they were engaged for at least 3 s and both the adult and the child were visible on the screen) and child initiation of JA (frequency of alternating gaze, showing, pointing, and giving to share)	Kaale,90 Lawton101
Child’s RRBs	10-min PCI with a standardized set of toys to assess the frequency and duration of child RRBs (narrow repetitive interests, stereotyped behavior/non-functional interests, specific sensory interests, unusual or repetitive motor movement, repetitive words/sounds) and the parents’ response to these RRBs (non-intervening, preventing, engaging, and distracting/developing).	Grahame82
Video-recorded family-child interaction during dinnertime
Child’s affect during family interaction situation	5-min family-child interaction during dinnertime to assess the child’s level of happiness, interest, stress, and communication style	Koegel99
Video-recorded observation of the children during their classroom routines
Engagement states and play behavior	Observation of the child in their classroom setting to assess the frequency and type of spontaneous child-to-teacher communications with subsequent adult responses, and frequency and type of teacher-to-child communications with subsequent child responses.	Carr67
	15-20 min of observation of the child during normal classroom routines to assess engagement states (unengaged, on looking, object engagement, person engagement, supported joint engagement, and coordinated joint engagement) by the Structured Play Assessment: the child is presented with four different set of toys at a table to assess functioning play types, SP types and play level.	Boyd,65 Goods80
	At least two 10-to-15-min classroom observations to assess how the child: handles the unexpected, compromise, reciprocity, follows rules, transitions, gets stuck, negativity/overwhelm, and participates	Kenworthy98
Video-recorded examiner-child play interaction
Child’s focus of interest	15-min examiner-child unstructured free play interaction, with the examiner imitating the child’s play to assess the child’s focus of interest (the number of toys with which children used non-imitative, differentiated play actions)	Mcduffie109
Verbal expressive language	15-min examiner-child semistructured play interaction, in which the examiner imitated the child’s play to assess the child’s spoken communication (frequency of non-imitative spoken communication acts and the number of different non-imitative words spoken)	Yoder137
Child’s social communication skills	15-min examiner-child unstructured free play interaction to assess JA, verbal and nonverbal requests, initiation, cooperative play, and eye contact.	Lerna,103,104
	15-min interaction between the child and a speech-language pathologist to assess the Functional Communicative Profile (number of communicative acts produced per minute, percentage of interactive acts, and percentage of communicative space used) and the Social Cognitive Performance (gestural and vocal communication intention, use of the mediating object, gestural and vocal imitation, combinatorial play, and SP).	Martins,106 Sun44
	10-min examiner-child interaction to assess the child’s social initiations, classified as unprompted (i.e., spontaneous) question, and unprompted attempt at a question.	Verschuur131
	15-min videotaped observation of children during their class snack session with other children and class staff to asses frequency of child-initiated communication	Gordon81
Video-recorded observation of the autistic children with typically developing peers
Child’s social communication skills	5-to-15-min play interaction with peers during recess to assess social skills on three levels: positive social interaction (activities that exhibit verbal and nonverbal social behaviors that lead to an effective social process with peers), low-level social interaction (behaviors that indicate social intention but with minimal social enactment, such as close proximity to other children without initiating a positive social interaction), and negative social interaction (unpleasant social behaviors that operate to stop or decrease the likelihood of a positive social interaction).	Hopkins,86 Bauminger60
	15-min play interaction with peers during recess to assess social behavior. Behaviors were categorized into four types based on the child’s actions: solitary, initiation, response, and interaction. Peer responses were also assessed and classified as positive (when the action of the target child resulted in a clear positive response by the peer, such as an enthusiastic comment, a smile, a high-five, inclusion in an activity, or other behavior that had a distinctly positive quality), neutral (when the peer’s behavior lacked a specific positive or negative quality), or negative (ignoring, criticizing, aggression, or active exclusion).	Wood136
	30-min play interaction with peers during class break to assess social behavior, classified as positive responses (when the child answers a question, obeys an order, responds with a positive emotion, or imitates the actions of another child) and negative responses (when the child refuses to respond to a question or instruction from another child)	Arabi59
Child’s social initiations and duration of social interactions	10-min play interaction with peers (familiar, typically developing children that attended the same school as the target child) during class break in the school playground to assess the frequency of self-initiated social contact with peers and the duration of social interactions with peers.	Owens112
	15-min play interaction with peers during class snack session to assess the frequency of child communicative initiations, frequency of use of Picture Exchange Communication System symbols, and frequency of speech (including non-word vocalizations).	Howlin87
	Direct observation of the children during unstructured periods at school when they had free access to familiar peers to assess the frequency of self-initiated social contact and the duration of social interactions	LeGoff102
Child’s engagement	Playground Observation of Peer Engagement – 15-min observation of the child during play interaction with peers in the playground during class recess or lunch play period to assess percentage of intervals the target child spent in solitary play or jointly engaged with others (i.e., turn-taking in games with rules and engagement in conversations or joint activities).	Kasari,97 Shih119
	PIP – 20-min playground interaction with two novel peers to assess cooperative play (participant engaged in reciprocal play with at least one child) and cortisol levels in saliva. One peer was instructed to elicit play during prompted time intervals. The other child was a typically developing child with no specific directions.	Corbett72
	PIP – 20-min playground interaction with two novel peers to assess group play (the duration of activity when the participant is engaging with the group together in an activity by using the same types of equipment or toys as other members of the group) and equipment play (the duration of activity when the child is using equipment or toys on his/her own and not as part of a group). One peer was instructed to elicit play during prompted time intervals. The other child was a typically developing child and received no specific directions	Corbett73
Video-recorded observation of the autistic children with autistic peers
Social communication skills and quality of friendship	Friendship Observation Scale – Autistic dyad interaction while playing with 18 puzzle images projected onto a 32-inch-wide touch sensitive horizontal surface through a video projector. Social communication behaviors coded were: positive social interaction (goal-directed behaviors – e.g., offering a goal-oriented action; sharing behaviors – e.g., showing and directing attention; prosocial behavior – encouraging; conversation – e.g., negotiation; and nonverbal interaction – e.g., eye contact), negative social interaction (such as teasing and aggression), affect, play (parallel play, simple social play, collaborative play, and unoccupied play), and autistic behaviors (such as repetitive stereotypical motor and verbal behaviors).	Ben-Sasson63

JA = joint attention; PCFP = Parent-Child Free Play interaction; PCI = parent-child interaction; PIP = Peer Interaction Paradigm; SP = symbolic play.

Table 3. Neurophysiological outcome measures used to assess autism intervention effects.

Neurophysiological measure/procedure	Intervention	First author (reference)
Event-related electroencephalography
Amplitude and latency of the ERN and Pe ERP components measured on erroneous response trials during a visual oddball target detection task	TMS	Sokhadze123
Amplitude and latency of the ERN and Pe ERP components measured on erroneous response trials of a visual Go/No-go cognitive control task.	TMS	Sokhadze122
Amplitudes and latencies of the P50, N1, P2, N2 and P3 stimulus-locked ERP components measured at frontal and posterior scalp regions during Go and No-go trials of a Go/No-go cognitive control task
Oscillatory power in the theta (4-7.5Hz) frequency range source-localized to the anterior cingulate cortex during inhibition of motor responses on No-go trials of a Go/No-go cognitive control task	Mind-body exercises	Chan70
Amplitude and latency of the stimulus-locked P1, N170, and N250 ERP components at left and right hemisphere posterior scalp regions in response to face stimuli during a face processing task	Face processing training	Faja77
Oscillatory power in the delta (1-4 Hz), theta (4-8 Hz), alpha (8-12 Hz), SMR (12-15 Hz), low beta (15-18), and high beta (18-30) frequency bands measured across three task conditions: resting state, a two-back working memory task, and a self-referential task which involved rating how attributable participants believed different personality traits were to themselves and the Queen of the Netherlands	Biofeedback	Kouijzer100
Difference measure of electrophysiological activity at parietal scalp sites in the time-range of the P3 ERP component (300-500 ms) measured in response to single vs. repeated presentations of face stimuli	Theater intervention	Corbett73
Resting-state electroencephalography
Oscillatory power in the delta (1-3 Hz), theta (4-7 Hz), alpha (8-12 Hz), beta (15-20 Hz), and high beta (20-38 Hz) frequency bands measured during eyes-open resting state	Skin stimulation	Chan69
Interhemispheric coherence (a measure of oscillatory connectivity) at frontal, central, temporal, parietal, and occipital electrode sites in the delta (0.4-4 Hz), theta (4.5-8 Hz), alpha (frequency limits not specified), and beta (13.5-30 Hz) frequency bands during a resting-state condition	Assisted therapy with dolphins	Ortiz-Sánchez111
Peak frequency of power in the alpha (8-13 Hz) frequency range measured during a resting-state condition	Transcranial direct current stimulation	Amatachaya58
Event-related functional magnetic resonance imaging
BOLD activity in the fusiform gyrus in response to viewing human faces displaying emotional facial expressions vs. scrambled face stimuli	Computer-based program to teach the ability to identify basic facially expressed emotions	Bölte64
Functional connectivity in the Reading Network, consisting of nodes in the left hemisphere (inferior occipital gyrus, fusiform gyrus, superior temporal gyrus, precentral gyrus, superior parietal lobule, supplementary motor area, inferior frontal gyrus, middle frontal gyrus, and thalamus) derived from BOLD activity measured during a sentence-reading task	V/V intervention focused on comprehension and higher order thinking training	Murdaugh108
BOLD activity in superior temporal sulcus and ventral striatum in response to viewing biological motion videos vs. scrambled motion videos	Pivotal response treatment	Ventola129
Intrinsic functional connectivity in posterior regions of the Reading Network, consisting of nodes in the left hemisphere (inferior occipital gyrus, fusiform gyrus, superior temporal gyrus, precentral gyrus, superior parietal lobule, supplementary motor area, inferior frontal gyrus, middle frontal gyrus, and thalamus) derived from BOLD activity measured during a resting-state condition	V/V intervention focused on comprehension and higher-order thinking training	Maximo107
Resting-state functional magnetic resonance imaging
Intrinsic functional connectivity of fronto-temporal brain networks measured during a resting-state condition	Music therapy	Sharda118
Diffusion tensor imaging
Mean diffusivity, fractional anisotropy, axial diffusivity, and radial diffusivity measures of white matter tract microstructure in 21 white matter fiber bundles	Nursery school 5 days a week, 6-10 h of applied behavior analysis sessions in homes and/or in nurseries per week, in addition to 1 h each of speech, occupational and psychomotor therapy	Saaybi117

BOLD = blood oxygenation level-dependent; ERN = error-related negativity; ERP = event-related potential; Pe = error positivity; TMS = transcranial magnetic stimulation; V/V = Visualizing and Verbalizing for Language Comprehension and Thinking.

Structured observational measures

Structured observational measures are those in which an examiner uses a structured script and a standardized set of materials to interact with a child and classify a behavior or ability as present or absent. Fourteen different measures were used by 64 studies, with ADOS142,143 being the most frequently used instrument (k = 26), followed by the Early Social Communication Scales (ESCS)144 (k = 10). Together, both instruments represented 56% of the structured observational measures used.

Naturalistic measures

Fifty-one studies used naturalistic procedures as outcome measures of the effect of autism interventions. As can be seen in Table 2, video-recorded parent-child interaction was the most frequently used procedure (k = 24), but some studies also used teacher-child interactions (k = 2), examiner-child interaction (k = 8), family-child interactions during dinnertime (k = 1), child observation during normal classroom routines (k = 3), and child interactions with autistic or non-autistic peers (k = 13). A range of specific behaviors were coded from these interactions, including social communication skills, attachment-related behaviors, play behavior, joint attention, joint engagement, and RRBs. Thirty-seven different coding systems were used to quantify specific behaviors, and were rarely used in more than one study. Further, they were often not described in detail (Table 2).

Neurophysiological measures

Neurophysiological measures were used by 16 studies and could be grouped into the following broad categories (Table 3): event-related electrophysiological activity derived from EEG (event-related EEG, k = 6), resting-state EEG (k = 3), event-related blood oxygenation level-dependent (BOLD) neural activity measured with functional magnetic resonance imaging (event-related functional magnetic resonance imaging [fMRI], k = 3), resting-state fMRI (k = 2), and white matter microstructure measured with diffusion tensor imaging (DTI) (k = 1). Specific measures used within the event-related EEG category were event-related potential (ERP) components indexing error-monitoring (error-related negativity [ERN] and error positivity, k = 2), ERP components indexing early attentional and inhibitory processes (P50, N1, N2, P3) (k = 1), ERP components reflecting face processing (P1, N170, and N250 components in one study and the P3 in another study), and oscillatory power measured during working memory and self-referential processing (k = 1). Of note, none of the studies measuring these components used the same cognitive task to elicit event-related EEG activity (Table 3). Likewise, the three event-related fMRI studies examined BOLD activity in different regions or networks of the brain during three different cognitive tasks (reading network during sentence reading, fusiform gyrus during face processing, superior temporal sulcus during biological motion perception).

All four resting-state EEG studies used a different specific index of activity (inter-hemispheric coherence, oscillatory power, hemispheric asymmetry, peak alpha frequency), as did the two resting-state fMRI studies (functional connectivity in the reading network in one study, functional connectivity in frontotemporal networks in the other). Only one study examined structural brain metrics (using DTI) (Table 3). Most studies used neurophysiological measures combined with rating scales or observational or neurocognitive tools, but two studies used EEG only111,123 and two others used fMRI only107,108 No study using near infrared spectroscopy (NIRS) or functional NIRS was retrieved from the databases, although the search strategy included a term that covered this technique.

Discussion

To our knowledge, this is the first systematic review to synthesize structured observational, naturalistic, and neurophysiological assessments that have been used as outcome measures of interventions for ASD. Consistent with a recent review focused on the variability of outcome measures in autism intervention trials,36 our results revealed extensive variability in outcome measures employed to assess the efficacy of nonpharmacological interventions across studies. This was especially true for naturalistic and neurophysiological methods, with 61 different naturalistic and neurophysiological methods of assessment used across 66 studies. Since each assessment instrument has a different framework, content, and rationale for its construction, comparing the efficacy of interventions assessed with different outcome measures is a challenging task.35 Future intervention studies in autism should therefore consider including assessments that have been used in previous trials, perhaps as secondary outcome measures if a new primary outcome measure is needed, to facilitate the comparison of different interventions across studies.

In this regard, structured observational instruments may represent the best choice for outcome measures since they have most frequently been used in the literature, facilitating cross-study comparisons of the efficacy of autism interventions. In terms of specific structured observational instruments, the ADOS has most often been used in previous intervention studies. However, this assessment tool is expensive (both in terms of purchasing the materials and in training individuals in administration and coding) and may not be sensitive to intervention-related changes in behavior.50 On the other hand, the ADOS is one of few autism assessment tools that is available in a range of non-English languages, such as Brazilian Portuguese145 and Polish.146 The unavailability of assessment instruments in languages other than English is a major barrier to conducting autism intervention studies in non-English-speaking countries. However, it should be noted that instruments that measure reductions of autism symptoms, such as the ADOS, have been criticized as intervention outcome measures by the autistic community. This is because these are diagnostic assessment tools and there is the risk that, with reduced symptom severity on these measures, autistic individuals may no longer meet the diagnostic criteria for autism and, consequently, may no longer receive support that is dependent on a diagnosis.147 For this reason, instruments that assess autistic people acting and interacting in their own environment have been suggested as key to better understanding the real-life impact of interventions on autistic individuals.147

Naturalistic outcome measures may be helpful in this regard. Our results showed that 51 of the included studies used such measures, which consisted of observation and coding techniques to assess autistic individuals’ behavior while interacting with another person (caregiver, teacher, examiner or peers) or in their own real-life environment (e.g., in the classroom). However, the studies varied greatly in the specific procedures and coding systems adopted during naturalistic assessments. The most frequently used coding systems were the Dyadic Communication Measure for Autism (DCMA),55 which was used in three studies, and the Playground Observation of Peer Engagement (POPE),93 which was also used in three studies. A more standardized approach to the use of naturalistic outcome measures, as well as rigorous and widely available training resources, is needed. It will also be important for future work to establish the extent to which specific measures of social communication abilities are comparable across different naturalistic paradigms and coding procedures (e.g., DCMA vs. POPE).

In addition to the variation in the coding systems used in naturalistic outcome measures, the interaction partner varied considerably across studies. The majority of naturalistic assessments involved interactions with the autistic child’s parent or caregiver, or with the child’s teacher. However, it is arguably most relevant to assess the child’s social communication skills when interacting with peers, since difficulties in social interactions and communication with individuals of the same age are considered when determining level of the impairment associated with autism.1 The POPE93 is one measure that does assess children’s interaction behavior with peers in a playground setting. Since this was also one of the more frequently used measures across studies, it may be most appropriate for use in future autism intervention studies in terms of comparability to previous intervention work.

It is also important to consider recent findings indicating that autistic individuals do not show social communication impairments when interacting with other autistic people. These findings suggest that social communication difficulties in autism might actually reflect a mismatch of interaction styles between non-autistic and autistic people in the majority of naturalistic interaction assessments.148,149 Despite this evidence for the importance of autistic-autistic interactions, our results showed that only one study assessed social communication skills of autistic children while interacting with other autistic children as an outcome measure.63 This finding highlights the need for further studies using naturalistic measures of autistic-autistic interactions. Indeed, including such measures could highlight which specific aspects of social communication are most challenging for autistic people during interactions with both autistic and non-autistic individuals; this, in turn, may lead to new intervention targets.148,150

Concerning neurophysiological outcome measures, our findings showed that the use of these assessments in autism intervention trials has grown in the last 10 years. However, they are still a minority choice for evaluating the outcome of behavioral interventions for ASD. Furthermore, neurophysiological outcome measures vary widely in the specific techniques and methods of quantification used, which makes it difficult to understand how different interventions affect neurophysiological activity or how and whether different neurophysiological patterns influence intervention outcomes.151 Although there are limitations to the use of neurophysiological indices as outcome measures of interventions (e.g., it would be difficult to establish their clinical relevance if intervention-related changes were found in these but not in clinical assessments), neurophysiological measures still provide important information about neural mechanisms related to the autistic phenotype. For example, the N170 ERP component is a robust index of face processing (an important aspect of social cognition) and has been shown to be fairly reliably altered in at least some autistic individuals.152 Indeed, the Autism Biomarkers Consortium for Clinical Trials (ABC-CT) proposed the N170 component as a candidate biomarker for ASD.153 Yet, only one intervention study in the current review included the N170 as a neurophysiological outcome measure.77

The only neurophysiological measure that was used in more than one study was the ERN component, which was examined in two studies from the same group.122,123 While the ERN is a robust index of self-regulatory ability, specifically of error monitoring,154 there is little evidence to indicate it is reliably altered in autism or should be used as an intervention target.155 Further, the ERN was measured during different experimental paradigms across the two studies, which limits cross-study comparisons of intervention effects on error monitoring. The most commonly used experimental paradigm, employed in six studies, was the resting state. However, each of those studies used a different method to quantify resting-state EEG or fMRI activity. Furthermore, evidence indicates that the stability of resting-state measures over time is low, as indicated by poor test-retest reliability.156,157 It is therefore not clear whether resting-state activity is an appropriate method of detecting pre/post-intervention changes in neurophysiology. In contrast to resting-state functional activity, structural neural measures appear to have good test-retest reliability158 and be sensitive to the short-term changes in behavior that are tested in nonpharmacological psychosocial interventions.159 Yet, structural brain metrics (white matter microstructure assessed with DTI) were investigated in only one autism intervention study.117 Future research using neurophysiological outcome measures should select measures that have been extensively examined in ASD, such as the N170, which have been shown to be reliably associated with the condition and have good test-retest reliability.

Future directions for the selection of outcome measures in autism intervention trials should be in line with autistic individuals’ desires and needs.147 Along these lines, future trials could prioritize naturalistic assessments as outcome measures, particularly if they quantify autistic-autistic communication and interaction as well as autistic-non-autistic interactions. However, further work is needed to standardize naturalistic assessment protocols and coding systems. Qualitative interviews with autistic participants receiving an intervention and their caregivers should also be conducted to evaluate the acceptability and usefulness of the intervention for autistic people. For trials that include neurophysiological measures, paradigms and quantification procedures need to be standardized, and more ecologically valid situations (such as live social interaction) should be considered.

Some limitations of the current review should be noted. We decided not to include an analysis on risk of bias across studies because we did not focus on results related to the efficacy of different interventions. Therefore, although previous reviews have reported that the quality of autism trials is generally low,36 we did not examine how study quality may have influenced outcome measure selection (or vice versa) in the present review. We also did not analyze the quality and psychometric properties of individual outcome measures, nor the robustness of different naturalistic and neurophysiological methods. Magnetoencephalography (MEG) was not explicitly entered as a search term, though we believe this neurophysiological measure would have been covered by other terms in the search. We did not discuss which neurophysiological and naturalistic outcome measures may be most appropriate for use with different intervention protocols. Instead, we provide a broad overview of the pattern of utilization of these different outcome measures, which could be used to guide future outcome measure selection in terms of choosing measures that have most commonly been used in different studies, or with specific interventions.

Disclosure

The authors report no conflicts of interest.