Randomized Controlled Trial of a Video Gaming-Based Social Skills Program for Children on the Autism Spectrum

Abstract

Families often face financial and geographical barriers to services for children with autism. The current study explored the effectiveness of a parent-supported adaptation of the computer game-based social skills program Secret Agent Society (SAS). Seventy child-parent dyads were randomized to SAS (n = 35) or a caregiver-supported cognitive skills training game (CIA—control comparison; n = 35), both completed over 10 weeks. Child participants were on the autism spectrum and aged seven to 12 years (60 boys, 10 girls). SAS participants improved more than CIA participants on parent-rated social skills and problem behaviors and teacher-rated social skills. Findings suggest the intervention may be a convenient, cost-effective therapeutic approach, especially during times of restricted face-to-face service access, such as COVID-19.

Introduction

The social and emotional skill difficulties that challenge children on the autism spectrum can also predispose those children to a range of negative outcomes (Tantam 2003; White et al. 2007), including fewer age-appropriate friendships, higher rates of peer rejection, social isolation and loneliness (Chamberlain et al. 2007). Research also suggests that social skill deficits in youth on the spectrum may lead them to underachieve in academic and occupational domains (Howlin and Goode 1998) and increase their risk of depression, anxiety and other psychological disorders (Attwood 2006; Tantam 2003). Accessible and effective interventions that improve the social and emotion management skills of children on the spectrum are needed to offset these undesirable developmental trajectories.

The majority of psychological interventions targeting emotion management and social skills are underpinned by behavioral and social learning theories, which break down complex skills into explicit, logical steps and fall under the umbrella term of Social Skills Training (SST; Hadwin et al. 1996). The target skills in SST can include emotion recognition, introductions, choosing friends, conversation skills, play skills, handling disagreements, coping with bullying and self-regulation (Attwood 2006). Teaching techniques in SST usually include a combination of direct instruction, modeling, video-based instruction, role-playing, social stories and scripts, incidental teaching, shaping, feedback, and/or self-evaluation, which are coupled with reinforcement to increase motivation and attention to social cues (Cappadocia and Weiss 2011; Rao et al. 2008). More recent SST interventions have investigated the utility of additional elements such as cognitive behavior therapy to target self-regulation (Beaumont and Sofronoff 2008; White et al. 2013), parent training (Einfeld et al. 2018; Laugeson et al. 2015; Waugh and Peskin 2015), teacher training (Beaumont et al. 2015), involvement of a typically developing peer (Corbett et al. 2016), and computer-assisted methods (Cardon et al. 2019; Lee 2020). These additions are intended to increase program efficacy, skill generalization and child motivation. A recent review concluded that there is considerable evidence regarding the effectiveness of group-based SST for school-age children and youth on the autism spectrum without intellectual impairment (Moody and Laugeson 2020).

There appear to be three important limitations of most studies evaluating the effectiveness of SST for individuals with autism. First, with only a few exceptions (e.g., Lerner and Mikami 2012; Young and Posselt 2012), most controlled trials have compared a treatment group to a non-treatment group (i.e., a waitlist comparison, or a vaguely described “treatment-as-usual” comparison). Without placebo-control designs, it continues to be an empirical question as to whether improvements in outcomes are related to the specific intervention alone or factors not specific to the intervention, such as increased clinical or parental attention within a structured clinical situation. Second, most clinic-based randomized controlled trials of social skills training programs for children on the spectrum have shown non-significant improvements on teacher-reported social skills (e.g., Frankel et al. 2010; Laugeson et al. 2009), suggesting that the generalization of gains may be limited. Third, most research requires in-person participation, and there is a need to develop intervention methods that are less reliant on face-to-face service delivery (Radley et al. 2014; Sofronoff et al. 2015). The need for therapeutic services that can be successfully delivered remotely has become particularly apparent recently, with social distancing restrictions imposed by COVID-19.

Sofronoff et al. (2015) conducted a pilot trial of a parent-supported videogame-based social skills program (Secret Agent Society—SAS) for children on the spectrum with at least average cognitive abilities. Parents received online therapist coaching in how to deliver the intervention. The study included 38 parents and 41 child participants from metropolitan and regional/rural Queensland, Australia and used a pre-post, follow-up design with an 8-week baseline period (where children received no intervention) and 6-week follow-up period. Results suggested the parent-directed intervention was associated with significant improvements in child social skills as reported by parents, with gains maintained at 6-week follow-up and with large effect sizes (Sofronoff et al. 2015). Improvements in parent self-efficacy, child behavior and child anxiety were also noted. However, limitations of the trial included the lack of a control group and the absence of teacher-report data to examine skill generalization to schools. The present study extended this research by conducting a randomized controlled trial of a simplified parent-supported variant of SAS that was delivered at home with weekly online guided support, compared to an active cognitive skills training control condition. It was hypothesized that children in the SAS condition would improve more than active controls from pre-intervention to post-intervention and to 6-week follow-up, in the following domains:

Primary Outcome

1. Parent-reported social skills (including emotion-regulation skills).

Secondary Outcomes

1. Teacher-reported social skills (including emotion-regulation skills).

2. Parent-reported anxiety.

3. Parent-reported child behavior.

Methods

Design

This study was a randomized controlled trial with an active comparison condition. The outcome measures for the dependent variables were taken at three time-points (pre-intervention, post-intervention:10 weeks after the intervention began; and 6-weeks following post-intervention).

Participants

Inclusion criteria for the study were as follows: participants were aged between seven and 12 years at the time of trial entry, had a formal diagnosis of Autism Spectrum Disorder (DSM-5, American Psychiatric Association 2013) confirmed by a pediatrician, psychiatrist or clinical psychologist letter, had access to internet and a webcam, lived in Queensland, New South Wales or Victoria, Australia, had not played the SAS computer game before and were willing to refrain from engaging in any other therapies for social-emotional skills training for the duration of the study. At study entry, participants with Attention Deficit Hyperactivity Disorder (ADHD) or Oppositional Defiant Disorder (ODD) also needed to have a behavior management plan in place through parent skills training and/or medication management to minimize the potential of behavioral and attentional issues confounding study findings. This was confirmed through an initial conversation with caregivers. No children were excluded from the study due to this criteria not being met.

Eligible children were also required to have cognitive abilities in at least the average range, as defined by their Full Scale IQ or Verbal and Perceptual indices on standardized intelligence tests. If a child had never had a formal cognitive assessment, cognitive ability was inferred by the child’s teacher providing a written statement that the child’s academic performance was within average range relative to same-aged peers. Children were excluded from the trial if they were involved in another social-emotional skills program. Parents were required to be willing to dedicate two hours or more per week to trial activities.

An a priori G-Power Analysis (Faul et al. 2007) estimated that 70 participants allocated to two groups with measures taken at three time points would have at least an 80% chance of detecting effect sizes of η² ≥ .18 (p = 0.025 two tailed) between experimental conditions if they did exist. Participants were 70 children (boys = 60; girls = 10) with an age range of 7 years, 9 months, to 12 years, 11 months (M = 9.89, SD = 1.37) at trial entry. Demographic and clinical characteristics of the child participants are presented in Table 1. A significant proportion of children had a co-existing diagnosis, with the most common being ADHD (24.3%). Twenty-seven children (38.6%) presented reports from a previous cognitive assessment and had a Verbal IQ ranging from 83 to 131 (M = 104.64, SD = 13.60). Written statements from teachers indicated that the other 44 children who did not have a cognitive assessment were achieving at a satisfactory level relative to same-aged peers.

Table 1.

Characteristics of the child participants

Categorical variables	SAS-group (treatment)		CIA-group (control)		Total
	n	(%)	n	(%)	n	(%)
Sample size	35	(50.0)	35	(50.0)	70	(100.0)
Gender
Male	30	(86.0)	30	(86.0)	60	(86.0)
Female	5	(14.0)	5	(14.0)	10	(14.0)
Co-existing psychological diagnoses
ADHDa	13	(37.1)	4	(11.4)	17	(24.3)
ODD	1	(2.9)	1	(2.9)	2	(2.9)
OCD	0	(0.0)	3	(8.6)	3	(4.3)
Anxiety disorder	2	(5.7)	4	(11.4)	6	(8.7)
Tourette’s disorder	1	(2.9)	0	(0.0)	1	(1.4)
Major depressive disorder	3	(8.6)	1	(2.9)	0	(0.0)
Co-existing medical conditions	3	(8.6)	0	(0.0)	3	(4.4)
Home-schooled	4	(11.4)	1	(2.9)	5	(7.1)
Previous social/emotional intervention
No	5	(14.3)	6	(17.1)	11	(15.7)
Yes	30	(85.7)	29	(82.9)	59	(84.3)
Cognitive functioning
Inferred from teacher’s report	23	(65.7)	20	(57.1)	43	(61.4)
Confirmed by cognitive assessment	12	(34.3)	15	(42.9)	27	(38.6)
Number taking medication	18	(52.9)	17	(48.6)	35	(50.7)
Indication for medicationb
Sleep	3	(8.6)	5	(14.3)	8	(11.4)
Attention	13	(37.1)	10	(28.6)	23	(32.9)
Mood/anxiety	2	(5.7)	4	(11.4)	6	(8.6)
Challenging behavior	2	(5.7)	4	(11.4)	6	(8.6)

Continuous variables	M	SD	M	SD	M	SD
Age (years)	9.80	(1.57)	9.99	(1.16)	9.89	(1.37)
Cognitive functioning
Mean verbal (standard score)c	100.45	(13.34)	107.93	(13.36)	104.64	(13.60)
Verbal standard score range	83–128		87–131		83–131

^aSignificant difference between groups p < .05

^bMedications categorized by indication as follows: Sleep (Melatonin and Catapress); Attention (Methylphenidate, Concerta, Methylin, Medikinet, Ritalin, Dexamphetamine, Atoxomexetine); Mood/Anxiety (Fluoxetine and Sertraline); and Challenging Behavior (Risperidone)

^cCognitive assessments included Wechsler Intelligence Scale for Children 4th ed. (WISC-IV), Wechsler Primary & Preschool Intelligence Scale for Children—Third Edition (WPPSI-III), and the Stanford-Binet Intelligence Scales—Fifth Edition (SB-V). Further, if a standardized verbal score or percentile was not specifically reported, the FSIQ was used in its place. This was done for 2 cases, one in either group

Participating parents were primarily female (91.4%), married, and identified as Caucasian. Ages ranged between 31 and 56 years ($M=$40.79, SD = 4.92) and over 80% had completed at least a university Bachelor’s degree. Over half (61.4%) the families resided in cities, with 38.6% from inner regional, outer regional and remote areas. Demographic information provided by parents indicated that financial or geographical barriers had prevented access to SST interventions for 69% and 26% of families, respectively. Socio-economically, 17% of families reported they found it difficult to pay bills most of the time, and 38% lived in a postcode area that incurred significant restrictions to accessibility of goods, services and/or opportunities for social interaction.1 See Table 2 for demographic characteristics of parents.

Table 2.

Characteristics of the parent participants who primarily delivered the interventions

Variable	SAS-group (treatment)		CIA-group (control)		Total
	n	(%)	n	(%)	n	(%)
Sample	35	(50.0)	35	(50.0)	70	(100.0)
Relationship to child
Father	2	(2.9)	3	(4.3)	5	(7.2)
Mother	32	(45.7)	32	(45.7)	64	(91.4)
Foster-mother	1	(1.4)	0	(0.0)	1	(1.4)
Marital status
De facto	1	(1.4)	4	(5.7)	5	(7.1)
Married	27	(38.6)	27	(38.6)	54	(77.2)
Single parent, divorced	4	(5.7)	3	(4.3)	7	(10.0)
Single parent, never married	3	(4.3)	1	(1.4)	4	(5.7)
Ethnic background
Aboriginal or torres strait islander	0	(0.0)	1	(1.4)	1	(1.4)
European/caucasian	35	(50.0)	34	(48.6)	69	(98.6)
Highest level of education completed
Some high-school	1	(1.4)	3	(4.3)	4	(5.7)
Completed year 12	4	(5.7)	4	(5.7)	8	(11.5)
Apprenticeship/vocational course	10	(14.3)	8	(11.4)	18	(25.7)
University bachelor’s degree	14	(20.0)	11	(15.7)	25	(35.7)
University post-graduate degree	6	(8.6)	9	(12.9)	15	(21.4)
Another parent assisted in delivery	19	(27.1)	16	(22.9)	35	(50.0)

Intervention Condition: SAS

The SAS intervention was adapted from the published SAS program (Beaumont, 2010) and consisted of the SAS Computer Game, Visual Support Cards, parent training slides, and Program Delivery Guide (www.sst-institute.net). The original SAS program is a multimedia-based group intervention targeting social and emotional skills. The program was designed to encourage skill generalization in multiple environments by including parent and teacher handouts. The intention of the current SAS parent-supported program was to provide a therapeutic intervention for families who face geographical, temporal and/or financial barriers to accessing the original face-to-face program. A summary of the session activities included in the SAS intervention is provided in Table 3. Adaptations to the original SAS program delivery format are described in the Appendix.

Table 3.

Session activities included in the SAS and CIA (control) conditions

Session	SAS computer game activities	CIA website activities
1	Quiz 1 Level 1 facial expression and body posture recognition activities	Virtual jigsaw puzzles On-screen memory card game -basic (find the matching cards) Aerial analysis 1—deciphering the time of day it is from an aerial shot of a scene
2	Level 1 voice tone recognition activities Repeat Quiz 1	Word find—basic Aerial analysis 2—deciphering the time of day it is from an aerial shot of a different scene
3	Quiz 2 Level 2 activities that teach the players how to detect different strengths of emotions (happy, sad, angry and afraid) within themselves from body clues and thoughts	Word find—intermediate On-screen memory card game -hard (find the matching cards)
4	Level 2 activity that teaches players how to integrate face, voice, body and situational cues to detect how someone is feeling Repeat Quiz 2 Quiz 3	Photo analysis challenges 1 and 2—spotting the differences between photos CIA world fact book—Explore flags of the word. Player looks at the Australian flag and learns about its history. They select another flag of their choice
5	Level 3 animated missions where the player assumes the role of an avatar and learns about relaxation strategies like slow breathing and thinking helpful thoughts	Photo analysis challenge 3—spotting the differences between photos World exploration—3rd to 5th grade -answer geography questions about different countries through a secret agent mission
6	Level 3 animated mission where the player assumes the role of an avatar who has to engage in group work with peers and try a new competitive game with peers	Deciphering a secret message—Code 1 Photo analysis challenge 4—spotting the differences between photos
7	Level 3 animated mission where the player assumes the role of an avatar who has to start, continue and end a conversation with a group of new peers and manage bullying	Read about codes and ciphers Player makes a secret code and shows a friend how to read it using their cipher disk
8	Repeated of the animated mission from Session 7 Repeat of Quiz 3	Geography quizzes about Australia, it’s territories and capital cities
9	Review/consolidation of player’s favorite Level 3 animated mission	Deciphering a secret message—Code 2 Geography quizzes about continents and oceans of the world
10	Level 4 graduation ceremony	Repeat favorite activities

The intervention began with a therapist-led 150-minute parenting webinar involving parent introductions, goal setting and an overview and demonstration of the SAS Computer Game and other program elements. Behavior management tips were also provided to help parents facilitate home-sessions with their children. These included setting up a predictable routine, praising desirable behavior, use of rewards to increase motivation, incidental teaching, supporting children’s skill generalization and providing examples of how to manage child anger, anxiety and boredom through functional analysis of behavior. The webinar emphasized the importance of completing “home missions” (i.e. daily skills practice tasks) to help children apply the skills that were introduced in the SAS computer game in daily life.

The intervention also involved weekly online video coaching sessions to trouble-shoot technical or child behavioral issues that arose when parents supported their children through the program. Sessions were up to 30-minutes in duration, therapist-led, and involved up to three parent participants. The therapist used a guided participation approach to encourage parents to trouble-shoot difficulties by referring to the slide handouts forwarded to them at the initial training webinar.

Active Control “CIA” Condition

The control comparison condition, or “Central Intelligence Agency (CIA) condition” consisted of a similarly structured program that included a suite of engaging and interactive online cognitive activities with espionage themes but with no social or emotional skills training components. The games were produced by The Central Intelligence Agency (2007) (https://www.cia.gov/kids-page/games), the National Security Agency (2009) (www.nsa.gov/kids/home.shtml) and Lizard Point (https://lizardpoint.com/geography/). The games chosen were appropriate for children aged eight to twelve years. A summary of the activities completed by child participants in each session of the CIA program is shown in Table 3. Initial parent training and weekly online coaching sessions for CIA mirrored the content and format of the SAS parent sessions, except parents were informed about the CIA program instead of SAS.

Measures

Social Skills Questionnaire (SSQ; Spence 1995)

The SSQ is a broad measure of children’s social skills that has parent (SSQ-P) and teacher versions (SSQ-T). Parents and teachers are asked to rate how often a child displays 30 social skills on a 3-point Likert scale ranging from zero (never) to two (always). Total scores range from 0 to 60 and normative data are available. The present study obtained Cronbach’s alphas of .88 for the SSQ-P and .94 for the SSQ-T. In this study, the SSQ was used as a measure of ‘far’ social skill transfer, given that many items assess social skills that are not specifically taught in the SAS program (e.g. Expresses affection or positive feelings to others).

Emotion Regulation and Social Skills Questionnaire (ERSSQ; Beaumont and Sofronoff 2008)

The ERSSQ parent version (ERSSQ-P) and teacher version (ERSSQ-T) were both developed by Beaumont and Sofronoff (2008) as measures of child competence in the specific emotion recognition, emotion regulation and social skills taught in the SAS program. As such, the ERSSQ was used to assess ‘near’ social skill transfer in this study. The ERSSQ-P and ERSSQ-T have been validated by Butterworth et al. (2014), with norms yet to be determined. The ERSSQ-P has 27-items, however, due to a printing error on item 25, a 26-item scale was used allowing possible scores to range from 0 to 104. The present study obtained a Cronbach’s alpha of .95 for the ERSSQ-P. The ERSSQ-T contains 25 items, with scores ranging from 0 to 100 and a Cronbach’s alpha of .94.

Spence Children’s Anxiety Scale—Parent (SCAS-P; Nauta et al. 2004)

The SCAS-P is a 39-item parent-report measure indicating the type and severity of a child’s current anxiety. The measure covers a range of anxiety symptoms that form six subscales: separation anxiety, generalized anxiety, social phobia, panic/agoraphobia, obsessive–compulsive, and fear of physical injuries. The total score was used as an outcome measure and normative data exists for children aged from 6 to 18 years. The Cronbach’s alpha was .92 for the total score.

Eyberg Child Behavior Inventory—Parent (ECBI; Eyberg and Pincus 1999)

The ECBI is a 36-item questionnaire where parents report the current intensity of their child’s behavior using a seven-point Likert scale and also indicate whether each behavior is currently a problem for them. This allows intensity and problem scores to be calculated. The raw scores of the two scales can be converted to T-scores (M = 50, SD = 10) for children aged 2 to 16. In this trial, the intensity scale was used as an outcome measure and a Cronbach’s alpha of .90 was found for the ECBI intensity scale.

Program Satisfaction Ratings

Parents were asked to rate their program satisfaction at the end of the SAS and CIA interventions respectively by answering the following questions: “Overall, how enjoyable did you as the parent find the programme to deliver?” “Overall, how enjoyable did your child find the program?”. Ratings were on a 4 point Likert scale: 1-not enjoyable at all; 2-a little bit enjoyable; 3-very enjoyable; 4-extremely enjoyable.

Procedure

Ethical approval was obtained from the University of Queensland Ethical Review Committee in accordance with the standards required by the National Health and Medical Research Council of Australia. Participants were recruited through contact with autism professionals and clinics. Autism associations distributed flyers to rural and regional areas through outreach teams and advertised the study on their websites.

Figure 1 presents the CONSORT flow chart of participants through the trial. A total of 160 families contacted the primary investigator via email or phone expressing interest in participating in the trial. If they met initial eligibility criteria, they were provided with a weblink to complete consent forms and demographics questions. Of these families, 74 parent–child dyads were eligible to participate and were sent a survey link to complete the baseline and study outcome measures listed above, with the exception of the ECBI which was posted or completed over the phone due to copyright. Of the 74 families who were sent the baseline measures, 70 families returned their responses and paid the $60 AUD fee to cover the cost of program materials. These families were then randomly allocated to either the SAS condition (n = 35) or CIA condition (n = 35) using a computerized random number generator (Urbaniak and Plous 2013).

Fig. 1.

Flow of participants through each stage of the randomized controlled trial and completion of primary parent social and emotional regulation skills outcome measures

After ethical approval from the relevant educational body was granted and consent to contact the child’s teacher was approved by the school Principal, teacher information and consent forms for the trial were sent, following by a survey link for teachers to complete the SSQ-T and ERSSQ-T.

Condition Allocation

After all baseline parent, child and teacher measures had been completed, blocks of at least 10 child-parent dyads were randomly allocated to a condition as described above. This was conducted by a post-graduate researcher from the School of Psychology, University of Queensland, with no affiliation to the trial. The second author then informed participants of their allocation and sent them the appropriate materials.

Concealment of Allocation

The trial was ‘open-label’ in the sense that participants were not directly informed if they were in the control or treatment group. All CIA participants were offered the SAS intervention six weeks after the completion of their CIA program.

Parent Training and Weekly Support Sessions

After families were allocated to conditions, initial webinars and weekly parent coaching sessions were scheduled and conducted via online video chat with the second author.

Post-trial Assessment

Once families completed 10 weeks of their allocated program, they were contacted to complete the post-trial outcome measures. Six weeks after families completed the program, they were again contacted to complete the 6-week follow-up measures.

Program Fidelity

The online support session facilitator recorded the tasks that parents reported completing with their child each week on a checklist to assess fidelity of parent delivery of the programs and children’s completion of required activities. For example, a parent may be asked if they helped their child to complete the Spot the Suspect activity in the SAS computer game. An affirmative answer reflected both program fidelity and child program adherence.

Challenges to program fidelity (e.g. lack of child engagement, time management) were problem-solved during the weekly parent coaching sessions. In addition, in the post-trial and follow-up questionnaires, parents in both conditions reported how much of the program they completed with their child.

Data Analysis Plan

A series of mixed-model (group x time) repeated measures analyses of variance (ANOVA) was performed for primary and secondary outcome measures, with a two level between subjects factor (SAS or CIA condition), and a three level within-subjects factor (pre-trial, post-trial, and 6-week follow-up). Alpha was set at .05 for tests of the main and interaction effects. To control for elevated Type I error in planned follow-up pairwise comparisons, alpha was adjusted to a conservative p = 0.001.

For estimates of how much association there was between the independent and dependent variables, the effect size partial eta squared (η²) was used. Reliability change index (RCI) scores were also calculated for the SSQ-P (the primary outcome measure with published norms and minimal missing data) to determine the percentage of participants in the clinical range on the measure at baseline who improved to within the normal range by post-intervention and follow-up.

Results

Preliminary Analyses

A series of one-way ANOVAs was conducted on demographic and baseline dependent variables. Results suggested that the SAS and CIA conditions were similar at pre-treatment with respect to most variables except teacher-reported social and emotion-regulation skills as measured by the ERSSQ-T, F(1,45) = 5.49, p = .02, partial η² = .12, where the SAS group (M = 54.48, SD = 12.67) had higher baseline scores than the CIA condition (M = 44.79, SD = 14.40). In addition, children in the SAS condition were found to have a higher intensity of behavioral issues as measured by the ECBI-Intensity score (M = 64.34, SD 7.20), F(1,68) = 4.82, p = .03, partial η² = .07, when compared to the CIA condition (M = 60.31, SD = 8.13). There was also a higher proportion of children in the SAS condition with a diagnosis of ADHD (37.1%) relative to the CIA condition (11.4%), X² (1, n = 70) = 6.29, p = .01, phi =  − .30.

Missing Data

The self-directed and state-wide nature of the trial posed difficulties in ensuring questionnaire completion by families and teachers, particularly when data collection time points fell during school holiday periods. Information on the proportion of data missing from outcome measures is summarized in Table 4. Little’s Missing Completely at Random Test suggested that missing values were random, χ² = 335.04 (325), p = 0.12. Given that there was no clear systematic explanation for the missing data, additional analyses were conducted using the intention-to-treat principle via the ‘last observation carried forward’ (LOCF) method.

Table 4.

Comparisons of mean scores on outcome measures across time and experimental conditions

Measure	SAS condition				CIA condition				Difference between conditions
	n	%MD	M	SD	n	%MD	M	SD	MSAS–MCIA	t(df)	p
SSQ-P
Pre	35	0	29.74	(10.61)	35	0	28.97	(8.23)	0.77	0.34 (1.68)	.74
Post	31	11	42.54	(10.97)††	33	6	31.97	(9.04)	10.57	4.40 (1.68)	**
Follow-Up	25	29	41.97	(11.89)††	32	9	32.74	(10.28)††	9.23	3.47 (1.68)	**
ERSSQ-P
Pre	35	0	43.29	(10.66)	35	0	44.2	(9.58)	− 0.91	− 0.37 (1.68)	.71
Post	31	11	60.8	(13.77)††	33	6	47.71	(12.29)	13.09	4.19 (1.68)	**
Follow-up	25	29	60.43	(13.53)††	32	9	49.91	(11.89)†††	10.51	3.45 (1.68)	**
SSQ-T
Pre	23	34	40.09	(11.63)	25	29	34.12	(12.46)	5.97	1.71 (1.46)	.10
Post	17	51	43.78	(9.49)	14	60	34.20	(12.79)†	9.58	2.93 (1.46)	*
ERSSQ-T
Pre	23	34	54.48	(12.67)	24	32	44.79	(14.40)	9.69	2.44 (1.45)	.02
Post	17	51	57.65	(10.30)	14	60	46.58	(15.68)	11.07	2.85 (1.45)	*
SCAS-P
Pre	35	0	28.26	(15.65)	35	0	29.80	(18.05)	− 1.54	− .38 (1.68)	.70
Post	31	11	24.66	(13.41)	33	6	26.14	(16.05)	− 1.49	− .42 (1.68)	.68
Follow-up	25	29	24.03	(14.29)†††	32	9	24.69	(15.82) †††	− .66	− .18 (1.68)	.86
ECBI
Pre	35	0	64.34	(7.20)	35	0	60.31	(8.13)	4.03	2.20 (1.68)	.03
Post	31	11	59.89	(9.18)†††	33	6	58.89	(8.26)	1.00	.48 (1.68)	.63
Follow-up	25	29	55.14	(9.92)†††	32	9	56.57	(6.99)	− 1.43	− .70 (1.68)	.49

n indicates the number of returned responses at Time 1 and ^%MD indicates the proportion of missing data at the post-trial or follow-up periods compared to n at pre-trial. M indicates the mean score for a condition at that time-period using the method of Last Observation Carried Forward to account for missing data

*p < .01, **p ≤ .001, and indicate significant differences between conditions; and ^†p < .05, ^††p < .01, ^†††p ≤ .001, and indicate a significant difference within the specific condition from the mean at pre-trial

Primary Outcomes

Parent-Reported Social Skills

Group means and line graphs for primary outcome measures are displayed in Table 4 and Fig. 2 respectively. Results from the mixed-model ANOVA on the SSQ-P data showed a significant main effect for time, F(1.54, 104.66) = 54.61, p < .0005, for group, F(1, 68) = 9.47, p < 0.003 and a group x time interaction, F(1.54, 104.66) = 18.29, p < .0005. Planned comparisons within condition indicated a significant pre- to post-improvement in the SAS condition, t(1, 34) = 8.01, p < .0005, partial η² = .65, and the CIA condition t(1, 34) = 3.12, p = .004, partial η² = .22. Between-condition contrasts revealed that the mean SSQ-P scores for conditions were not different at pre-treatment, while at post-trial, the mean improvement in the SSQ-P was greater for the SAS condition than for the CIA condition t(1, 68) = 4.40, p = .0005, partial η² = .22; and this difference between the conditions remained significant at follow-up, t(1, 68) = 3.47, p < .0005, partial η² = .15.

Fig. 2.

Graphical displays of the primary outcome variables between conditions across time. a Mean SSQ-P scores between conditions across time. b Mean ERSSQ-P scores between conditions across time

At baseline, 18 children randomized to the SAS Condition and 17 children randomized to the CIA Condition were reported to have social skills difficulties in the clinical range on the SSQ-P (i.e. scores less than or equal to 28). With LOCF imputed for three cases (two CIA and one SAS), 77.8% of the SAS children (14 cases) and 41.2% of the CIA children (7 cases) were no longer in the clinical range by post-intervention. In addition, at follow-up, 72.2% of the SAS children (13 cases) and 35.3% of the CIA children (6 cases) were no longer in the clinical range.

The mixed-model ANOVA conducted on the ERSSQ-P data showed a similar pattern of findings: a significant main effect for time, F(1.58,107.42) = 71.91, p < 0.0005, for group, F(1,68) = 7.85, p = 0.007, and a group x time interaction, F(1.57,106.6) = 24.84, p < 0.0005. Planned comparisons revealed improvements in mean ERSSQ-P scores from pre- to post-trial for the SAS condition t(1, 34) = 8.61, p < .0005, partial η² = .69, but not the CIA condition t(1, 34) = 2.83, p = .01, partial η² = .19. At follow-up, the improvement from pre-trial was maintained for the SAS condition t(1, 34) = 8.61, p < .0005, partial η² = .69, and an improvement from pre-trial became apparent for the CIA condition t(1, 34) = 4.17, p < .0005, partial η² = .34. Between-condition contrasts revealed that the mean improvement in ERSSQ-P score was greater for the SAS condition at post-trial than for the CIA condition t(1, 68) = 4.20, p = .0005, partial η² = .20, which remained at follow-up, t(1, 68) = 3.46, p < .0005, partial η² = .15.

Secondary Outcomes

Teacher-Reported Social Skills

Only ten teachers completed the SSQ-T and ERSSQ-T at all three time-points, making imputing values as LOCF impractical. Twenty-nine teachers (41%) completed pre- and post-program questionnaires. Therefore, only the pre- and post-trial teacher data was analyzed. LOCF was used for an additional 19 cases who completed the pre-but not post-questionnaires, leading to a total sample of 48 and 47 out of 70 for the SSQ-T and ERSSQ-T, respectively.

ANOVA results for the SSQ-T showed a significant main effect for time, F(1, 46) = 4.73, p = 0.04, for group, F(1, 46) = 5.65, p < 0.02, and a group x time interaction, F(1, 27) = 4.34, p = 0.04, partial η² = .09. Improvements seen in the SAS Condition were significantly greater than for the CIA Condition. For the ERSSQ-T, ANOVA findings revealed a significant main effect for time, F(1, 45) = 5.49, p = .02, and for group, F(1, 45) = 7.54, p = .01, observed power = .77, but no significant group x time interaction, F(1, 45) = .42, p = 0.52, partial η² = .01. Follow-up analyses of the ERSSQ-T main effects indicated the improvement from pre to post-trial was not significant for the SAS condition t(1, 22) = 1.96, p = .06, partial η² = .15, observed power = .47, nor the CIA condition t(1, 23) = 1.30, p = .21, partial η² = .07, observed power = .24.

Parent-Reported Child Anxiety

Mixed-model ANOVA findings showed a significant main effect for time, F(1.70,115.64) = 18.41, p < 0.0005, but no significant main effect for group, F(1,68) = .12, p = .74, and no group x time interaction, F(1.70,115.64) = .18, p = 0.79.

Parent-Reported Child Behavior

Mixed-model ANOVA indicated a significant main effect for time, F(1.87,126.96) = 46.10, p < .0005 but not for group, F(1,68) = .43, p = .52, and there was a significant time by group interaction, F(1.87,126.96) = 8.21, p = .001. Within-condition contrasts showed that reductions of behavioral intensity from pre- to post-trial were significant for the SAS condition t(1, 34) = 5.39, p < .0005, partial η² = .46, but not the CIA condition t(1, 34) = 1.80, p = .06 partial η² = .10. Furthermore, the reduction for the SAS condition was maintained at follow-up, t(1, 34) = 8.28, p < .0005 partial η² = .67.

Program Satisfaction

Ratings indicated that parents in both the SAS and CIA conditions perceived both themselves (SAS: M = 2.84, SD = .74, N = 31; CIA: M = 2.55, SD = .62, N = 33) and their children (SAS: M = 2.97, SD = .84, N = 31; CIA: M = 2.67, SD = .78, N = 33) to find the interventions enjoyable. Independent t-tests suggested that both interventions were equally enjoyable for parents, t(1, 62) = .1.73, p = .09, and children, t(1, 62) = 1.49, p = .14.

Program Fidelity

A Chi-square test of independence revealed no association between the proportion of parents who completed all 10 weeks of the program and condition X² (1, n = 45) = .63, p = .43. Specifically, 74.2% of SAS parents and 78.8% of CIA parents reported completing the 10-week program. The mean number of online support sessions attended by parents for the SAS condition (M = 7.63, SD = 2.84) did not significantly differ from that of the CIA condition (M = 6.43, SD = 2.97), t(68) = 1.73, p = .09.

Discussion

The present study was conducted to examine the effectiveness of a parent-supported variant of the SAS social-emotional skills training program that could be accessed by families remotely. Consistent with the first hypothesis, child SAS participants showed significantly greater improvements in social-emotional functioning on parent-report measures compared with those in the control condition, and these were maintained at 6-week follow-up. While these findings are encouraging, caution is warranted in interpreting them, as parents were both the intervention-delivery agents and evaluators, making them more susceptible to responder bias. Indeed, this might also explain the significant improvements noted by parents of children in the CIA condition on the SSQ-P and ERSSQ-P over the course of the intervention (although they were not of the same magnitude as SAS intervention parents). Both interventions also provided the opportunity for more one-on-one quality time between parents and children, which was often noted by CIA parents, and could have influenced parents’ perceptions of their children’s social-emotional functioning over the course of the program.

Results provided partial support for the second hypothesis, which examined the generalization of treatment effects to the school environment. While improvements on the SSQ-T were greater for SAS than for CIA participants, no significant improvements were made by either group on the ERSSQ-T measure. This finding is surprising, given that the ERSSQ-T is a measure of ‘near’ skill transfer (i.e. questionnaire items map closely onto the skills taught in the SAS intervention), whereas the SSQ-T is a measure of ‘far’ skill transfer (more generic emotion regulation and social skills). As both questionnaires have a single factor structure and include both emotion regulation and social skill items, it is difficult to determine which social-emotional skills appeared to generalize better (or less well) to the school environment.

The weaker treatment effects shown on the teacher-report measures relative to the parent-report measures might suggest that greater engagement of school staff and specific skills practice in the school environment is required for emotion regulation and social skills to generalize and for changes to be noticed. Despite the low teacher questionnaire response rate (which brings the representativeness of the teacher data into question), the improvements on the SSQ-T shown by SAS participants were encouraging. This is one of few published trials providing evidence to suggest that social skills training programs delivered outside of a school setting might lead to improvements in the social functioning of children on the spectrum that generalize to the school environment.

The present findings did not support the third hypothesis, with parents of children in both conditions reporting similar significant reductions in children’s anxiety levels from pre-intervention to 6-week follow-up. This finding speaks to the importance of including an active control comparison condition in intervention studies, as factors non-specific to the SAS intervention (e.g. parent skills training, regular online therapist support, increased parent–child interaction) may have influenced parents’ perceptions and reporting of their children’s anxiety. The reduction in children’s anxiety levels at follow-up relative to pre-treatment might also have been due to the follow-up assessment time point coinciding with the school holidays, when social and academic demands were less.

Study findings supported the fourth hypothesis, with parents reporting a greater improvement in behavioral problems for SAS than for CIA participants. It is possible that the improvements found in social-emotional skills at home helped children to better communicate their needs to caregivers and manage their emotions, which contributed to a reduction in problem behavior. Further research is needed to explore whether improvements in social-emotional functioning associated with the intervention mediate behavioral improvements.

In summary, the current study findings build on the existing SST evidence-base (e.g. Frankel et al. 2010; Laugeson et al. 2009; Sofronoff et al. 2015) for children with ASD by demonstrating significant treatment effects for a parent-assisted computer game intervention relative to an active control condition (versus no-treatment comparison group), with some signs of social skill generalization to the school environment.

Limitations and Future Research

Despite this study’s strengths, including an active control comparison condition, recruitment of a moderately large geographically diverse sample and collection of follow-up data from multiple informants, it has limitations. Firstly, the parent participants were predominantly female, married, Caucasian and well educated. This significantly limits the generalizability of the findings to more heterogeneous populations. Secondly, the reliability and validity of assessing treatment fidelity based on parent self-report is questionable. Finally, the parent- and teacher-report data that was captured should be interpreted with caution. Parents were both the intervention-delivery agents and primary evaluators. Therefore, their perception of their children’s responsivity to the interventions may have been biased. The relatively low teacher-measure return rate, while comparable to other studies (e.g. Laugeson et al. 2012), also brings into question the representativeness of the teacher data collected. Nonetheless, it is encouraging that SAS participants made significantly greater improvements than participants in the CIA condition on the SSQ-T from pre- to post-treatment, given the conservative data analysis method used (intention to treat analysis).

Thirty-nine percent of children (27) had their cognitive ability confirmed as being at least average based on a previously completed standardized cognitive assessment, with the cognitive ability of 61% of child participants (44) being inferred based on a written teacher statement. This limitation was due to it being impractical to administer cognitive assessments to a large number of regional, remote and interstate families face-to-face, and the questionable validity of administering child cognitive assessments via videoconference at the time the study was conducted. Nonetheless, this methodological weakness brings into question the generalizability of the current findings.

To address the above limitations and determine whether the Secret Agent Society Computer Game meets evidence-based best practice standards, it is recommended that future randomized controlled trial evaluations of the software involve more demographically diverse participant samples and use more rigorous methods to capture treatment fidelity data. These may include auto-capture of computer game play data (since implemented by the program developers) and filming and coding of caregiver-supported computer-game play sessions and parent-coaching sessions. Where possible, best-practice assessments should be used to confirm children’s IQ. Cost effective, objective measures of children’s social cognition and social skills also need to be developed, perhaps using webcam and algorithm technology to analyze responses to virtual interactive avatars. Supplementing parent- and teacher-report data with blind clinician ratings of children’s social-emotional functioning would also reduce the potential impact of rater bias on trial findings.

Future research may also examine whether less structured interventions that involve lower dose, pre-recorded (versus live) online parent coaching or no parent coaching achieve comparable treatment effects to those shown in this study and if so, for what participant populations. A comparison of caregiver-supported versus unsupported SAS child computer-game play would also be of great value, together with an empirical investigation of the impact that factors such as child engagement and self-awareness of social-emotional challenges have on treatment outcome. Such approaches would help with optimizing program efficiency, cost-effectiveness and program fit.

There continues to be a pressing need for innovative social skills supports for children on the autism spectrum, and computer-based remoted delivery may be a viable option. Such interventions are even more critical during global crises such as COVID-19, where there has been a desperate need for evidence-based remote therapies that can be effectively provided without the risk of in-person contact. The use of supported videogame-based child interventions has the potential to significantly improve the affordability and accessibility of intervention services.

Parent-directed SAS program modifications

1. Reduction of resources To increase the ease with which parents could deliver the intervention and to improve program affordability, several of the resources in the SAS small group program were not included in the parent-directed variant evaluated in this paper. These included the helpful-thought game, walkie-talkies for practicing voice-tone recognition, tokens for in-session reinforcement, and the SAS Challenger role-play-based Board Game that reinforces skills. In addition, the Parent Workbook, Child Cadet Handbook and Teacher Tip Sheets were not used.

2. Program delivery agents Sessions were facilitated by parents who received brief online training in the program instead of trained educators and allied health professionals.

3. Structure and format Parents were given a program delivery guide to assist with structuring weekly activities for the program. Each week was divided into three segments: (i) The computer game activities to be completed, (ii) activities involving the visual support cards to be completed, and (iii) home missions (skills practice tasks) to be completed. There were 10 structured sessions in the program, with opportunities for ongoing sessions if desired.

4. Teacher-involvement Teachers were not formally involved in the delivery of the parent-directed program.

5. Training Parents were trained to deliver the SAS intervention via a 150-min live webinar. They were also provided with weekly online therapist support throughout the 10-week program via group video-chat sessions.

Author Contributions

RB, HW, JW and KS all contributed to the design of this study and the writing and editing of this paper. The study was executed by HW and KS. Data analyses were completed by HW and KS.

Compliance with Ethical Standards

Conflict of interest

Renae Beaumont is the Secret Agent Society Program developer, and receives royalty payments on all program materials and practitioner training courses sold. All other authors of this paper declare that they have no conflict of interest.