Adverse Event Reporting In Intervention Research for Young Autistic Children

Abstract

Our team examined 150 reports on group design, non-pharmacological interventions for young autistic children, to determine the prevalence of adverse event reporting. We found that only 11 studies mentioned adverse events; one indicated adverse events occurred, and an additional three indicated adverse effects occurred (i.e., adverse events that could be attributed to the intervention). We also coded reasons for participant withdrawal, and found that of the 54 studies that reported reasons for withdrawal, ten studies reported reasons that could be categorized as an adverse event, eight reported reasons that could be categorized as an adverse effect, and an additional 12 studies reported reasons that were too vaguely described to determine adverse event status. We recommend that autism intervention researchers make concerted efforts to monitor, classify, and report adverse events so that practitioners, policy makers, and families are better equipped to weigh potential benefits of interventions against potential harms.

Professionals who support autistic children are bound by ethical codes stipulating their responsibility to do no harm (American Occupational Therapy Association, 2015; American Psychological Association, 2016; American Speech Hearing Association, 2015; Council for Exceptional Children, 2015). Despite this ethical imperative, little is known about the kinds of adverse effects or harms that could result from autistic children’s participation in clinical, school, or home based interventions. Indeed, autism intervention meta-analyses often neglect to analyze the extent to which adverse events are reported in primary literature, or the nature of these events (e.g., Eldevik et al., 2009; French & Kennedy, 2018; Reichow, 2012). Improving our understanding of this issue would allow researchers, practitioners, and policy-makers to make intervention recommendations that appropriately weigh potential benefits of participation in intervention programs against possible negative consequences.

According to the Cochrane Adverse Effects Methods group, an adverse event is “an unfavorable or harmful outcome that occurs during, or after, the use of a drug or other intervention, but is not necessarily caused by it,” and an adverse effect is an “adverse event for which the causal relation between the intervention and the event is at least a reasonable possibility” (Preyer et al., 2019, section 19–1-1). Duggen and colleagues (2014) further define ‘harm’ as a specific instance of adverse effect resulting in participants’ sustained deterioration during or beyond the intervention period. Even though it is often not possible to attribute a causal association between intervention receipt and the occurrence of adverse events, the Consolidated Standards of Reporting Trials (CONSORT) guidelines recommend that descriptive data on adverse events be collected and reported for all clinical trials, along with clear reporting of the methods used to collect this data (Ioannidis et al., 2004).

Adverse event monitoring and reporting is an especially pressing issue for autism intervention research, given that the first intervention programs designed for autistic children involved the intentional application of painful stimuli (i.e., ‘aversives’) such as electric shocks, unpleasant odors, sprays of water to the face, or slaps to the thigh (Favell et al., 1982; Lovaas, Schaeffer, & Simmons, 1965; Lovaas, 1987). At present, most intervention researchers and practitioners within and outside behavioral theoretical paradigms renounce the use of aversives as an intervention strategy. It is, however, informative to examine how adverse effects of aversive procedures were not sufficiently considered, which allowed these procedures to persist (we note that in the US, electric shocks were permitted as a behavioral treatment until 2020; United States Food and Drug Administration, 2020).

There are a small number of studies examining the adverse events that could accompany aversive procedures; we briefly touch on two of them here. First, Lichstein and Schreibman (1976) reviewed research on the provision of electric shock to autistic children. Based on their review of ten studies, they concluded that the reduction in children’s problem behavior and other positive ‘side effects’ that followed administration of electric shock outweighed reported adverse effects. However, the authors noted that most studies did not systematically define, monitor, or measure adverse events; instead they were reported in an ad hoc fashion and considered tangential to the studies. The review also did not indicate that any primary study included long-term follow-up procedures to determine if electric shock was associated with harms. Finally, the review authors concluded that positive ‘effects’ outweighed adverse effects by calculating the ratio of positive to negative effects. Notably, they removed fear reactions to the shock procedures in their calculation, as fear was considered a necessary response to the aversive. The authors’ conclusions are inappropriate for at least three reasons. First, the positive ‘effects’ tabulated in this review were not experimentally examined by the primary study authors and, therefore, cannot be attributed to the intervention. Second, the authors omitted pain and fear responses from their calculation, as they considered them necessary for behavior change. This is inappropriate, as the perceived ‘necessity’ of an event does not negate its occurrence. Third, the authors’ calculation incorrectly assumes that adverse events can simply be offset by the occurrence of positive events, without regard to the nature or severity of the events.

A second study systematically examined adverse events that could be associated with aversive procedures (e.g., Harris et al., 1991, summarized in Dawson, 2004). However, this study restricted their inquiry to determining if these procedures caused harm to the clinicians administering them to autistic children, and did not inquire as to whether there were potential adverse effects or harms for the young children who received these interventions.

Another useful illustration of an autism intervention that some considered beneficial, but that is now deemed harmful, is Facilitated Communication (FC). FC was designed to improve the communication abilities of autistic people who previously did not have a reliable means of communicating. The intervention involved a trained clinician supporting an autistic client’s arm, to facilitate their use of a computer keyboard to type out messages. While some researchers and practitioners were initially enthusiastic about the perceived benefits of this intervention, others pointed out that the research supporting its use did not meet rigorous quality standards that would allow for attributing participants’ improved communication abilities to the intervention (e.g., Jacoboson, Mulick, & Schwartz, 1995; Shane, 1993). Alarmingly, within five years following the introduction of FC to the United States, more than 60 allegations of child sexual abuse were made by children receiving the intervention against their parents, resulting in children being removed from their homes (Jacobson, Mulick, & Schwartz, 1995; Mostert, 2010). However, subsequent studies found evidence that these messages were not originating from the children, but were instead inadvertently produced by facilitators (Siegel, 1995). Because the potential harms of implementing FC are so great, and high quality evidence supporting the efficacy of this intervention is sorely lacking, most researchers and practitioners consider providing FC to autistic children to be unethical (American Speech-Hearing Association, 2018, but see Crane, 2018 for an alternative perspective). Importantly, the researchers that conducted the few available studies on FC did not track or report adverse events or harms themselves, which likely led to delays in understanding the potential negative consequences of participating in the intervention.

Monitoring and Reporting Adverse Events

There are two types of adverse event monitoring that can be used in intervention research. Active report monitoring involves collecting data using a predefined protocol, while spontaneous report monitoring involves recording all adverse events observed or reported by participants, regardless of whether they were predefined by the study team in advance of the intervention trial (Preyer et al., 2019). Both types of monitoring may be necessary to fully capture the range of adverse events that could occur. Active approaches ensure participants who would not otherwise report events have an opportunity to do so, and spontaneous approaches ensure that unexpected (and therefore not predefined) events are also recorded. Potential adverse effects should be assessed via subjective sources (e.g., self and other reports of well-being) and objective sources (e.g., psychometric measures of adjustment; Duggan et al., 2014). For autism intervention research, this means that participants themselves, as well as caregivers, should be queried to determine their perception of potential harms resulting from the intervention, in addition to undergoing examiner-administered measures of wellbeing. Adverse events and effects should be monitored during the intervention period, and via long term follow-up, as adverse effects may not manifest until long after the intervention has stopped.

In pharmacological research, where adverse event reporting is far more common than in non-pharmacological research, there are standardized definitions and examples of the kinds of adverse events that warrant reporting (e.g., Leape, 2002). Additionally, monitoring and reporting adverse events is a submission requirement for most journals that publish drug trials. However, even in this research, adverse events are not reliably categorized or documented (Pitrou, Boutron, Ahmad, & Ravaud, 2009; Schroll, Maund, & Gøtzsche, 2012). In psychosocial intervention research, on the other hand, adverse events are rarely systematically monitored at all. This is likely due to an assumption that, unlike pharmacological interventions where unintended side effects are considered ubiquitous, psychosocial interventions are (incorrectly) considered either helpful or benign (Lilienfeld, Lynn, & Lohr, 2003, and see Duggan et al., 2014 for a review). As Antshel and Barkley (2008) note, “psychosocial treatments of any power to influence behavior will produce AEs [adverse events] in some subset of the treated population” (p. 433). Therefore, systematic adverse event monitoring and reporting should be routine procedure for all non-pharmacological interventions, including for autism intervention research.

Most journals that publish autism intervention research require that studies involving human participants include a statement verifying the research team gained approval for study procedures from an institutional review board (IRB) prior to conducting the study. IRBs, in turn, require that researchers report to their office any serious adverse events that occurred during the conduct of research. However, we examined submission guidelines for nine journals (five of which are autism specific), and none presently require that researchers include information about adverse events, adverse effects, or harms when reporting on intervention efficacy. The sole journal that mentions adverse events in their publication guidelines (Pediatrics) limits the scope of events to those involving drugs or medical devices. This journal also does not require researchers to report adverse events in their submissions; their guidelines only indicate that these events should be reported to appropriate governmental agencies. For more detail on the journal submission guidelines related to adverse event reporting for each of the nine journals we examined, see Table 1.

Table 1.

Journal Guidelines Regarding Reporting of Adverse Events

Journal	Requirements Related to Adverse Events
Autism Research	Authors must include statements indicating that IRB approval was secured, and that recognized ethical guidelines were followed (e.g., Declaration of Helsinki, US Federal Policy for the Protection of Human Subjects).
Autism	Research ethics outlined in the Declaration of Helsinki must be followed, and submitted manuscripts should conform to International Committee of Medical Journal Editor recommendations for the conduct, reporting, editing and publication of scholarly work in medical journals (note that this resource does not specifically mention reporting adverse events in manuscripts). Requires statement of ethics committee or IRB approval, including full name and institution of the review committee and the approval number.
European Child and Adolescent Psychiatry	Author must include a statement that indicates IRB approval of the research, and that confirms research was conducted in accordance with the Declaration of Helsinki.
Journal of Autism and Developmental Disorders	Author must provide a statement that indicates IRB approval of the research study was secured (inclusive of the name and institution of the review committee) and that confirms ethical guidelines consistent with the Declaration of Helsinki were followed. If it is believed that the research may not have been aligned with the 1964 Helsinki Declaration, this information should be explained.
Journal of Child Psychology and Psychiatry	Author must provide a statement that indicates IRB approval of the research study was obtained (name and institution of the review committee), and that confirms legal requirements of the study country were followed.
Journal of Educational Psychology	Authors are required to state in writing that they have complied with APA ethical standards in the treatment of their sample, human or animal, or to describe the details of treatment.
Pediatrics	Author must provide a statement that indicates IRB approval of the research study was obtained. Any adverse drug or medical device events should be reported to the appropriate governmental agencies.
Research in Autism Spectrum Disorders	Author must provide a statement that indicates IRB approval of the research study was obtained (inclusive of name and institution of the review committee), and that study was conducted in compliance with the Declaration of Helsinki.
Molecular Autism	Author must provide a statement that indicates IRB approval of the research study was secured (inclusive of name and institution of the review committee), and confirm that study was conducted in compliance with the Declaration of Helsinki, or indicate appropriate exemptions.

Note. APA = American Psychological Association. IRB = institutional review board.

The Current Study

The current study is a secondary analysis of 150 reports that were gathered in the context of a scoping meta-analysis of all randomized controlled trials (RCTs) and quasi experimental studies for all non-pharmacological intervention types in young autistic children (aged 8 or younger; redacted for review). We were prompted to conduct this study following the publication of our main meta-analysis, after receiving useful feedback that we had failed to analyze the reporting and occurrence of adverse events as part of our review procedures (Dawson, 2019). We sought to determine:

1. The percentage of studies that reported the occurrence of adverse events or adverse effects. As journals generally do not require adverse events reporting, we hypothesized that such reporting would be rare in autism intervention research, and that most reports would not describe systematic means for monitoring adverse events.

2. Of those studies that monitored adverse events/effects, the percentage that indicated an adverse event/effect had occurred.

3. The percentage of studies providing reasons for withdrawal that contained at least one reason that could be considered an adverse event or adverse effect. We examined reasons for withdrawal in addition to direct reports of adverse events, under the assumption that this information would be more readily reported, and may contain reasons that could be classified as adverse events or effects.

4. The percentage of randomized controlled trials that contributed negative effect sizes for outcomes hypothesized to benefit the child.

Method

In the main meta-analysis from which adverse event data was gathered, all available non-pharmacological intervention studies involving young children with autism were considered for inclusion, and effect size data on all child outcomes were extracted.

Search

Nine databases were searched to gather relevant studies, including Academic Search Complete, Cumulative Index of Nursing and Allied Health Literature (CINAHL) Plus with Full Text, Educational Administration Abstracts, Education Resources Information Center (ERIC), Education Source, MEDLINE, Psychology and Behavioral Sciences Collection, PsycINFO, and SocINDEX with Full Text. Combinations of the following keywords were applied to the search boxes: (autis* OR ASD OR PDD OR Aspergers) AND (Intervention OR therapy OR teach* OR treat* OR program OR package) AND (Assign* OR “Control group” OR BAU OR “wait list” OR RCT OR Random* OR Quasi OR “treatment group” OR “intervention group” OR “group design” OR trial). Additionally, the National Database for Autism Research (NDAR), the National Institutes of Health (NIH) Matchmaker, and the Institute of Education Sciences (IES) databases were searched to identify unpublished or grey literature. Finally, we compiled a list of 106 investigators who received federal grants related to autism research, and sent emails to 90 of these researchers (contact info for 16 researchers could not be located) requesting that they share data meeting our inclusion criteria. No new data was acquired from this process. The aforementioned search yielded 12,933 manuscripts that were then subject to abstract screening and full text review.

Screening

Abstackr software (Wallace, Small, Brodley, Lau, & Thomas, 2012) was used to conduct a preliminary screening of each report based on titles and abstracts. After discarding irrelevant manuscripts, full texts were examined to determine if the following inclusion criteria were met: (a) published in English, (b) publication date between 1970-present, (c) participants were reported to have a diagnosis of autism, (d) the average age of participants ranged from 0–8 years, and (e) studies used group designs that included both a treatment and control or comparison group. Refer to (redacted for review) for the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) diagram depicting this process. A full list of included reports is available in the supporting information.

Text Search for Reporting of Adverse Events/Effects and Participant Withdrawal

PDF files of each of the 150 articles included in the meta-analysis were searched for the following terms: “adverse event”, “adverse effect”, “side effect”, complications, harm*, attrition, withdr*, and dropout. If a study did not include any of the search terms, coders scanned the methods section and appropriate figures to determine if any adverse events could be identified, or if reasons for withdrawal were provided.

Coding

Coding of adverse events, adverse effects, and reasons for withdrawal was restricted to treatment groups, active-treatment comparison groups, and sham groups that used procedures similar to the treatment group (e.g., a barometric chamber sham condition that was compared to a barometric chamber active condition).

Adverse Events

Studies were coded as either reporting an adverse event, reporting that no adverse events occurred, or not including this information. If an adverse event was reported and it was described as being caused by intervention procedures (or the assumption was reasonable), it was also categorized as an adverse effect. All studies were coded for adverse events by a primary coder, and a reliability coder randomly selected and coded 20% of the included studies. Kappa coefficients for inter-coder agreement were 0.86 and 1.00 for adverse events and effects, respectively.

Reasons for Withdrawal

Several steps were taken to extract and code reasons for participant withdrawal. First, the second author searched each study, and extracted verbatim reasons for withdrawal from studies that reported this information. A coding scheme was developed from this list to identify whether reasons for withdrawal were due to an adverse event, an adverse effect, neither an adverse event or effect, or if the reasons were too vague to determine adverse event/effect status. Based on guidance from the Cochrane Handbook (Preyer et al., 2019), adverse events were defined as any unfavorable or harmful outcomes (i.e., the physical or psychological distress of a child or parent interventionist) occurring during or after intervention. The primary and secondary coders both coded 100% of all extracted reasons for withdrawal. The kappa coefficient for inter-coder agreement was 0.80, and all discrepancies were resolved through consensus.

Locating Negative Effect Sizes

We sorted all effect sizes calculated for the main meta-analyses, and extracted effect sizes that were (a) negative (except in cases where negative effects indicated positive growth on the measure), (b) associated with confidence intervals that did not overlap with zero, and (c) were from RCTs. We restricted our selection of effect sizes to those from RCTs because these provide better evidence of causal associations between the intervention and the outcome (i.e., significant effects are less likely to be due to pre-treatment group differences, which are very likely to occur in quasi-experimental studies).

Analysis

Frequencies and percentages were calculated as appropriate to answer each research question.

Community Involvement Statement

An author on this paper is the parent of an autistic son. This author was involved in conceptualization of the study, reviewing and interpreting findings, and editing the final manuscript.

Results

Proportion of Studies Reporting Adverse Events and Adverse Effects

Out of 150 reports, 11 (~ 7%) included text about adverse events (see Table 2). Of these, four (~36%) reported either an adverse event or effect had occurred during the intervention period. One study reported a child health issue (adverse event), one study reported ear trauma associated with hyperbaric oxygen therapy, and two studies reported parent anxiety during parent-mediated intervention (adverse effects). None of the 11 studies that provided information on adverse events described systematic procedures for defining or monitoring the occurrence of adverse events. We surmised that it is therefore likely that spontaneous report monitoring, and not active monitoring, was used in most of the studies that reported on adverse events. Finally, none of the studies followed participants for any length of time beyond the intervention period to measure potential harms.

Table 2.

Adverse Events and Adverse Effects Reported in Treatment, Active Treatment Control, or Sham Groups of Included Studies

Study	Journal	Intervention	Design	Authors Reports of Adverse Events or Effects
Bieleninik et al., 2017	JAMA	Improvisation Music Therapy	RCT	Child health issue that resulted in hospitalization*
Dawson et al., 2010	Pediatrics	ESDM	RCT	No serious adverse events related to the intervention were reported during the 2-year period
Dawson et al., 2012	JAACAP	ESDM	RCT	There was no significant adverse events associated with the ESDM intervention
Hardan et al., 2015	JCPP	PRT	RCT	No adverse events were noted in either group
Ichikawa et al., 2013	Bio-Psycho-Social Medicine	TEACCH	RCT	No adverse events occurred during the program
Page, 2012	ProQuest Dissertations & Theses	Therapeutic Horseback Riding	Quasi	With regard to adverse effects reported as a result of therapeutic horseback riding, parents of participants all expressed that no harmful effects were observed
Rahman et al., 2016	The Lancet	PASS	RCT	No adverse events were reported in the PASS or treatment-as-usual groups
Sampanthavivat et al., 2012	Diving & Hyperbaric Medicine	Hyperbaric Oxygen Therapy	RCT	Minor-grade ear barotrauma events**
Silva et al., 2015	Autism Research & Treatment	Qigong Massage Treatment	RCT	One parent with severe wartime PTSD found that he was unable to give the massage due to excessive anxiety triggered by his child’s resistance to touch. Once he stopped giving the massage, he experienced no further anxiety relative to the intervention**
Smith et al., 2000	American Journal on Mental Retardation	EIBI	RCT	One parent in the parent training group reported that the treatment was stressful for her**
Weiner & Greene, 2014	EXPLORE	NMT	Quasi	No adverse reactions were reported

Note. EIBI = Early Intensive Behavioral Intervention, referred to as “Intensive Early Intervention” in the Smith et al., 2000 report. ESDM = Early Start Denver Model, JAACAP = Journal of the American Academy of Child and Adolescent Psychiatry, JAMA = The Journal of the American Medical Association, JCPP = Journal of Child Psychology and Psychiatry, NMT = NeuroModulation Technique, PASS = Parent-mediated intervention for autism spectrum disorder in South Asia, PRT = Pivotal Response Treatment, PTSD = Post traumatic stress disorder, Quasi = Quasi-experimental, RCT = Randomized Controlled Trial.

^*Reported occurrence of adverse event

^**Reported occurrence of adverse effect

Reasons for Participant Withdrawal

Fifty-four reports included reasons for participants’ withdrawal from the study. Of these, 24 (44%) provided reasons for withdrawal that were not adverse events, 10 (~19%) provided reasons for withdrawal that included at least one adverse event but the events appeared unrelated to the intervention, eight (~15%) provided reasons for withdrawal that included at least one adverse effect, and 12 (~22%) provided reasons for withdrawal that were too vague to determine if withdrawal was due to adverse events or effects (e.g., reasons such as “participants withdrew from the study because they stopped attending sessions”). See Table 3 for further details on these studies and the reasons for withdrawal that were categorized as adverse events or adverse effects. There was minimal overlap between studies that reported adverse events and had reasons for withdrawal that were coded as adverse events. Specifically, one study with a reason for withdrawal that was coded as an adverse event also reported this occurrence as an adverse event (Bieleninik et al., 2017). Two studies that reported reasons for withdrawal that were coded as adverse effects also reported other occurrences that were labeled as adverse effects (Sampanthavivat et al., 2012; Silva et al., 2015). Finally, one study that reported a reason for withdrawal (child’s fear) that was coded as an adverse effect reported that no adverse events occurred (Page, 2012). Recurring categories of adverse events that were reasons for withdrawal included children’s health issues (five studies), children’s distress during, or dislike of, the intervention (seven studies), and family crises (six studies). Similar to adverse event reporting, none of the studies described systematic procedures for determining participants’ reasons for withdrawal.

Table 3.

Reasons for Withdrawal Associated with Adverse Events/Effects

Authors, Year	Journal	Name of Intervention	Intervention Type	Design	Reasons for Participant Withdrawal
Bieleninik et al., 2017	JAMA	Improvisation Music Therapy	Sensory-based	RCT	Child health issue that resulted in hospitalization*
Clionsky, 2012	ProQuest Dissertations & Theses	PCIT or CDIT	Other	RCT	Only two families in the Immediate Treatment group dropped out of the study after initiating treatment. In one case, a primary caregiver was in the midst of a difficult divorce.*
Fletcher-Watson et al., 2016	Autism	FindMe iPad App	Technology-based	RCT	The two children who dropped out of the study citing lack of enjoyment of the app as their reason did play the app less than most others (105 and 159min, respectively)**
Fridenson-Hayo et al., 2017	European Child & Adolescent Psychiatry	Emotiplay Serious Game	Technology-based	RCT	The reasons for children’s failure to complete the intervention included children’s lack of interest in the Serious Game and medical reasons**
Ingersoll et al., 2016	JADD	ImPACT Online	NDBI	RCT	Family crisis*
LaGasse, 2014	Journal of Music Therapy	Music Therapy	Sensory-based	RCT	The Music Therapy group experienced 10% attrition, with one child taken off the study due to illness that led to excessive absences*
Malow et al., 2014	JADD	Individual Parent Sleep Education	Other	RCT	Six children could not tolerate the actigraphy device**
Page, 2012	ProQuest Dissertations & Theses	Therapeutic Horseback Riding	Animal-assisted	Quasi	The child became uninterested in and slightly fearful of the horses**
Roberts et al., 2011	RASD	Center-based Building Blocks Program	NDBI	Quasi	One participant withdrew as a result of a staff decision that he was too stressed in the small group setting**
Sallows & Graupner, 2005	American Journal on Mental Retardation	Intensive Behavioral Treatment	Behavioral	RCT	One girl was placed in foster care after 1 year of treatment, and the foster parents did not wish to continue treatment for her*
Salt et al., 2002	Autism	Scottish Early Intervention Program	Developmental	Quasi	Two twin boys discontinued attendance owing to being taken into social work care following a change in family circumstances*
Sampanthavivat et al., 2012	Diving & Hyperbaric Medicine	Hyperbaric Oxygen Therapy	Other	RCT	…another boy in the sham group dropped out following a febrile convulsion**
Solomon, 2014	JDBP	PLAY/DIR floortime	Developmental	RCT	Caregiver went to jail*
Silva et al., 2015	Autism Research & Treatment	Qigong (QST) Massage Treatment	Sensory-based	RCT	Parent’s poor health, family crisis*
Tellegen & Sanders, 2014	JCCP	Primary Care Stepping Stones Triple P	Other	RCT	Parental psychopathology*
Thompson et al., 2014	Child: Care, Health, & Development	Family-Centered Music Therapy	Sensory-based	RCT	Discontinued intervention due to illness*
Whitehouse et al., 2017	JCPP	TOBY App	Technology-based	RCT	Child did not enjoy TOBY app**
Williams et al., 2012	JCPP	Transporters Program for Children with Autism	Technology-based	RCT	Withdrew due to behavioral problems preventing child watching DVD**

Note. CDIT = Child-directed interaction therapy, DIR = Developmental, Individual-differences, & Relationship-based, E = Experimental group, JADD = Journal of Autism and Developmental Disorders, JAMA = Journal of American Medical Association, JCCP = Journal of Consulting and Clinical Psychology, JCPP = Journal of Child Psychology and Psychiatry, JDBP = Journal of Developmental & Behavioral Pediatrics, NDBI = Naturalistic Developmental Behavioral Interventions, PCIT = Parent-Child Interaction Therapy, PLAY = Play and Language for Autistic Youngsters, Quasi = Quasi-experimental, RASD = Research in Autism Spectrum Disorders, RCT = Randomized Controlled Trial, S= Sham Group, TOBY = Therapy Outcomes By You.

^*Adverse Event

^**Adverse Effect

Negative Effect Sizes

Of the 87 RCTs included in the analysis, we located 11 effect sizes from 10 studies (11% of RCTs) that were negative and significantly different from zero, indicating that the intervention was causally associated with participants’ regression on outcomes of interest. Domains included social communication (six outcomes), as well as language, play, restrictive and repetitive behavior, ‘challenging’ behavior, and socio-emotional outcomes (one outcome per domain). See Table 4 for additional details on the studies from which these outcomes were extracted. We also examined whether these negative effect sizes were discussed by study authors as being potentially adverse effects of the intervention. This was not the case for any study. Most studies simply noted the negative effects, and then attempted to explain why such negative effects could have been produced by an intervention that was otherwise apparently overall beneficial. Two studies made no mention of negative effect sizes in either the results or discussion sections (Smith, Groen, & Wynn, 2000; Srinivisan et al., 2015).

Table 4.

Randomized Controlled Trials with Negative Effect Sizes

Study Authors, Year	Journal	Intervention Type	Name of Intervention	Comparison Group	Outcome Measure	Outcome Domain	Effect Size
Chang et al., 2016	JADD	NDBI	JASPER	BAU	Simple play (number of unique play acts) from the SPA	Play	−0.53
Clionsky, 2012	ProQuest Dissertations & Theses	Other	PCIT or CDIT	BAU	Dyadic Parent Child Interaction Coding System total child verbalizations	Language	−0.80
Gulsrud et al., 2007	Autism	NDBI	JA Intervention	Symbolic Play Intervention	Non-verbal gestures	Social-communication	−0.77
Kasari et al., 2008	JCCP	NDBI	JA Intervention	Symbolic Play Intervention	Mother initiated JA at 12 months	Social-communication	−0.73
Murdock et al., 2014	Focus on Autism and Other Developmental Disabilities	Sensory-based	Vestibular Stimulation via a Platform Swing	BAU	Number of intervals self-stimulating	RRBs	−0.75
Rahman et al., 2016	The Lancet	Developmental	PASS	BAU	Proportion of time in shared mutual attention	Social-communication	−0.64
Smith et al., 2000	American Journal of Mental Retardation	Behavioral	Intensive Early Intervention	Parent Training	CBCL Withdrawal Score Teacher	Social Emotional	−1.15
		Behavioral	Intensive Early Intervention	Parent Training	CBCL Aggression Teacher	Challenging Behavior	−0.49
Srinivasan et al., 2015	RASD	Technology based	Robot Intervention	Comparison Intervention	Percent duration of social verbalization to trainer across sessions	Social-communication	−1.09
Srinivasan et al., 2015	Autism Research & Treatment	Technology based	Robot Intervention	Comparison Intervention	Training-specific test of imitation praxis	Social-communication	−0.95
Wong et al., 2010	JADD	Other	Autism 123	BAU	ADOS A1 Overall level of non-echoed language	Social-communication	−1.11

Note. ADOS = Autism Diagnostic Observation Schedule, BAU = business as usual, CARS = Childhood Autism Rating Scale, CBCL = Child Behavior Checklist, CDIT = Child-directed Interaction Therapy, CPEP-R = Chinese Psychoeducational Profile – Revised, DBC = Developmental Behavioral Checklist, EIBI = Early Intensive Behavioral Intervention, GMDS-ER = Griffiths Mental Development Scales – Extended Revised, HKBABS = Hong Kong Based Adaptive Behavior Scales, IQ = Intelligence Quotient, JA = Joint Attention, JADD = Journal of Autism and Developmental Disorders, JASPER = Joint Attention, Symbolic Play, and Engagement Regulation, JCCP = Journal of Consulting and Clinical Psychology, NDBI = Naturalistic Developmental Behavioral Intervention, PASS = Parent-mediated intervention for autism spectrum disorder in South Asia, PCIT = Parent-Child Interaction Therapy, PEP-R = Psychoeducational Profile - Revised, PEAC = Parent Education and Counseling, PRT = Pivotal Response Treatment, RASD = Research in Autism Spectrum Disorders, RCT = Randomized Controlled Trial, RDLS = Reynell Developmental Language Scales, RRBs = Restricted and Repetitive Behaviors, SARRB = Social Affect and Restricted Repetitive Behaviors, SPA = Structured Play Assessment, SRS = Social Responsiveness Scale, VABS = Vineland Adaptive Behavior Scales.

Discussion

In addition to the ethical guidelines to do no harm mentioned in the opening paragraph, researchers also have an ethical duty to seek out information about potential adverse effects and harms that could be caused by the interventions they study (Lilienfeld, 2016). In the context of a scoping meta-analysis of studies employing group designs to test the effects of treatment intended for children on the autism spectrum between birth and age 8, we searched for adverse events reported in primary studies through three sources: (a) direct reporting of adverse events, adverse effects, or harms, (b) reasons for withdrawal that could be classified as adverse events or adverse effects, and (c) negative effect sizes for outcomes designed to measure intervention benefits. Our results indicate that, in non-pharmacological intervention research for young autistic children, researchers are generally not actively seeking out adverse events that could occur alongside intervention benefits, or routinely labeling adverse events/effects as such when they do occur. In the main meta-analysis (redacted for review) we reported that several studies (29 total) included measures of child socio-emotional health, such as internalizing behavior, aggression, and caregiver stress, but these constructs were treated as pre-existing sample characteristics that could be alleviated by the intervention, and not as measures meant to track potentially adverse effects of participation in the intervention. The present finding of a relative lack of monitoring and reporting found in this analysis is similar to past findings for intervention research in other areas of psychology, such as psychosocial interventions for attention deficit/hyperactivity disorder (Antshell & Barkley, 2008).

The adverse events (i.e., occurrences noted for intervention participants that may or may not be causally related to the intervention) that we did identify include occurrences such as child health concerns and family crises. The adverse effects (i.e., occurrences noted for intervention participants that are likely or at least plausibly caused by the intervention) include major and minor physical trauma, child distress and dislike of the intervention, and caregiver anxiety and stress. Further, regression caused by the intervention was evident in a variety of domains such as social communication, restrictive and repetitive behavior, language, play, and socio-emotional outcomes.

Given that just over one-third of the studies that monitored adverse events/effects reported that one occurred, and that a similar proportion of studies that provided reasons for withdrawal described reasons that we categorized as adverse events or effects, it is likely that adverse events and effects are somewhat common at the study level. That is, a preliminary estimate given the available data suggests that adverse events occur in a small subset of participants for approximately a third of intervention studies, even though they are widely under-reported. More systematic monitoring and reporting in future research is necessary to refine this estimate. Additionally, about 10% of interventions may facilitate regression (rather than growth) on at least some important outcomes, even if researchers are not interpreting negative effect sizes in this way. When this occurs, researchers should consider whether these outcomes should be communicated as adverse effects of the intervention.

Shortcomings in addressing adverse events, adverse effects, and harms could be for multiple reasons, including: (a) failure to incorporate appropriate monitoring in the study design, (b) failure to collect appropriate data, (c) failure to report collected data, (d) failure to fully report collected data (‘restricted’ reporting), (e) distortedly reporting collected data, and (f) actively hiding collected data (Ioannidis, 2009). Poor or neglected attention to adverse event reporting is likely a snowball effect. Journals often do not require researchers to provide this information in submitted studies, researchers therefore rarely provide such information in their reports, and this increases the likelihood that subsequent studies will also omit information regarding adverse events. More dubious reasons for the invisibility or downplaying of adverse events in this literature (reasons d-f) could be due to researcher conflicts of interest (Ioannidis, 2009), which are prevalent (although also under-reported) in autism intervention literature (Bottema-Beutel et al., 2020).

Future Directions

Measurement of Adverse Events, Adverse Effects, and Harms

Overall, adverse event reporting (including documenting reasons for withdrawal) should be more widespread and more systematic in intervention research for young autistic children. Even in extant studies that have reported adverse events and reasons for withdrawal, much of the documentation was too scant to determine precisely what had occurred, or to readily classify the occurrence as an adverse event or adverse effect. More complete, transparent, and consistent reporting in future research would allow for a more accurate estimation of the likelihood of adverse events, the specific adverse events likely to occur, and the intervention types with which such events are most commonly associated. Routine monitoring and reporting across intervention studies would also permit more sophisticated analyses, such as the identification of subgroups of children who are more likely to experience adverse events. It is also important to note that our review is restricted to group-design research. In the future, single-case research (especially behavioral research involving aversives or withholding of children’s preferred activities) should also be examined to determine if there are shortcomings in monitoring and reporting harms.

To improve our understanding of whether and what types of adverse events are likely to occur, autism intervention researchers should devise methods of systematically monitoring and reporting adverse events and effects, both within and beyond the intervention period. This could include broadly defined, active monitoring procedures designed to be used across interventions, additional active monitoring procedures that are tailored to specific intervention types, and spontaneous report monitoring procedures that would capture any additional adverse events not anticipated or specified in advance. As improvements in one domain can cause deterioration in other domains, adverse events should be monitored across a wide range of constructs (Lilienfeld, 2007). Adverse event monitoring should be multi-faceted, and include systematic collection of child perspectives (where possible), parent perspectives, and objective measures of child and caregiver distress/well-being (Duggan, 2014). If adverse events are systematically measured in treatment and control groups, statistical analyses could determine if there is a causal association between the intervention and the adverse events (see Bearss et al., 2015 for an example).

However, Ioannidis (2009) warns that statistical differences between groups on any given adverse event may be difficult to detect, and cautions that combining different adverse events into a single ‘adverse event’ outcome category in order to improve power may make findings difficult to interpret. Given these difficulties in statistical testing at the primary study level, failure to find a significant group effect for adverse events should not be interpreted as evidence that any isolated events observed were not related to the intervention. Systematically collected, participant-level descriptive data is likely the most useful data to report in single studies. If primary studies sufficiently report adverse events, meta-analyses can then be employed to aggregate this data and provide more conclusive results regarding the causal link between interventions and adverse events (Ioannidis, 2009). Meta-analysts could then further categorize studies according to the extent of benefit offered by interventions in relation to the risk for harm. For example, Loke and colleagues (2007) suggest a taxonomy for evaluating interventions in this way, by considering the margin between benefit and harm, the availability of alternative treatments that have lower risks for adverse events, and whether risk of adverse events threaten intervention adherence.

Types of Adverse Effects and Harms Relevant to Young Autistic Children

Given that children’s dislike of the intervention was a common theme among reported reasons for withdrawal, child distress (including the intensity and duration) during intervention sessions, as well as in a variety of other settings over the course of the intervention period and over the long term, could be an important adverse event category to actively monitor. For children who are preverbal, this could be measured by operationalizing and quantifying instances where children are unwilling to calmly accompany the interventionist and participate in intervention activities or perhaps by employing physiologic measures of stress such as skin conductance or heart rate. Tests of the feasibility and validity of such candidate measures are much needed in future research. These measures should be only one facet in a battery of measures examining potential harms. Autistic children may come to an intervention study with a history of participation in interventions that deny them of choice, agency, and activities they enjoy. Paradoxically, this may result in autistic children appearing to ‘choose’ or willingly participate in interventions that include punishment (Hanley, 2010). However, this may not reflect children’s actual preference, but may be a manifestation of their overall lack of choice and access to desired activities outside of the intervention.

There are several other adverse effects that could conceivably occur in the intervention programs represented in this review. For example, interventions that require separating autistic children from other children for long periods of time should be especially careful to document potential harms that could occur as a result of extended segregation. For example, proponents of early intensive behavioral intervention recommend children participate in a form of predominantly adult-led therapy (that, at least in early stages, does not usually incorporate other children, but that is more regularly characterized by structured interactions with a trained adult) for 20–40 hours per week (Reichow et al., 2012). It is currently unclear if high intensity, one-to-one formats such as this may negatively influence children’s ability to build relationships with other children, or result in short- or long-term increases in children’s experiences of stigmatization. At least one study, however, suggests negative developmental consequences of exclusion (Strain, 2017). Monitoring and reporting potential adverse effects of extended segregation should occur in tandem with monitoring for adverse events that occur as a result of the intervention procedures themselves. Second, parent-mediated interventions that involve training caregivers to alter their interaction style and daily routines should include procedures for monitoring unintended consequences of these changes, especially when the research team does not share the cultural or linguistic background of the participating family (e.g., Divan et al., 2015; Yu, 2016). Although parental stress is a somewhat common measure in caregiver-mediated interventions, more specific measures tailored to the changes in routine required by the intervention should be developed and examined at the participant level (e.g., beyond comparisons of group means). Third, interventions that inherently involve substantial physical activity, such as equine or sensorimotor interventions, should take care to systematically monitor and report any physical injury or significant discomfort.

Conclusion

In the introduction, we noted two examples of intervention procedures (aversives and FC) that were once promoted for autistic children, but that are now widely considered to be associated with harms that outweigh any potential benefits. Our review highlights that potential adverse effects for most interventions for young autistic children are unknown. Because even seemingly innocuous procedures have the potential to inadvertently cause harm, it remains important to determine whether autism interventions of any type are associated with adverse effects or harms. Improved monitoring and reporting of adverse events will allow researchers to discern the prevalence and types of adverse events that are possible for different intervention procedures geared towards young autistic children. In turn, stakeholders will be better equipped to select interventions in light of potential harms that could mitigate potential benefits. Journal editors can facilitate this process by providing clear guidance on how adverse events, adverse effects, and harms should be monitored and reported, and by enforcing high standards to ensure such guidance is adhered to in future autism intervention studies.

We close by adding a final point of consideration on this topic. Ultimately, determining the utility of an intervention by weighing potential harms against potential benefits will require value judgments (Lilienfeld, 2007). What level of risk, and what types of harms, will stakeholders consider acceptable when choosing whether or not to adopt interventions for young autistic children? What level and types of outcomes will be considered beneficial enough to offset risks of harm? Once adverse events, adverse effects, and harms are more widely reported and understood, researchers, practitioners, policy-makers, and caregivers will need to grapple with these questions. Recent debate in the literature regarding ‘optimal outcomes’ for autistic people suggests that many autism researchers may not yet be adequately prepared for the ethical ramifications of these questions. In this literature, people who once met the criteria for an autism diagnosis, but no longer did, were considered to have achieved an ‘optimal outcome’ (e.g., Fein et al., 2013; Ornstein et al., 2015). This research was widely criticized by members of the autistic community, who noted that many autistic people had no desire to ‘pass’ as non-autistic, and that doing so came with significant socio-emotional and mental health costs (Kapp, 2014). While the optimal outcome literature is not associated with explicit examination of interventions, it betrays a value judgment that no longer meeting the criteria for an autism diagnosis is the ultimate benchmark for ‘success’, even when balanced against co-occurring negative quality of life outcomes such as depression and anxiety. Moving forward, the autistic community should be involved in ethical discussions regarding how potential harms of intervention programs should be weighed against potential benefits. Additionally, their input should be sought when determining what intervention outcomes, inclusive of both benefits and any possible harms, can truly be considered to constitute success (Lilienfeld, 2007).