Pesky gNATs for children experiencing low mood and anxiety – A pragmatic randomised controlled trial of technology-assisted CBT in primary care

Abstract

Background

Compared to the evidence base for adult populations, less is known about the effectiveness of technology-assisted CBT interventions for children aged 12 and under in real-world settings.

Aims

This study aimed to evaluate the effectiveness of a technology-assisted CBT intervention called Pesky gNATs in primary care psychology services.

Methods

A 2 × 3 (group by time) waitlist randomised controlled trial design was applied (n = 122), using the total internalising score from the Child Behaviour Checklist (CBCL) as the primary outcome measure; and a range of secondary outcomes. Participants were recruited from primary care services across Ireland and were assessed at baseline (Time 1), post-intervention (Time 2), and a follow-up at 3 months (Time 3). Repeated measures ANOVAs were used to analyse the data; in addition to Cohen's d for effect sizes, and the reliable change index (RCI) for clinically meaningful change.

Results

Overall, no significant between-group differences were found on primary or secondary measures. Both intervention and waitlist control groups experienced significantly positive change across all timepoints. However, participant retention rate at Time 2 was 66%; and fell to 44% at Time 3. Intervention group feedback from children and parent at Time 2 indicated high levels of acceptability and satisfaction with Pesky gNATs. The RCI showed a statistically significant result at Time 2: more participants from the intervention group (n = 9) moved from clinical to non-clinical levels on the primary outcome than did waitlist control participants (n = 0); but no differences were found at Time 3.

Conclusion

This RCT did not demonstrate that Pesky gNATs is effective in providing clinically significant levels of change for low mood and anxiety. However, participant feedback showed high levels of acceptability and positive experiences However, given the high attrition rate and implementation challenges, further research is required to ascertain the effectiveness of technology-assisted CBT interventions such as Pesky gNATs.

Highlights

• •Some evidence suggests that technology-assisted CBT is an effective way to address psychological problems in a range of populations, but less is known about effectiveness with children.
• •This study used a waitlist control RCT to assess the effectiveness of a computerised CBT game (Pesky gNATs) in primary care psychology services in Ireland (n=122) for children aged 12 and under experiencing low mood and anxiety
• •No significant between-group differences were observed across primary and secondary measures
• •Both child and parent feedback demonstrated high levels of acceptance and satisfaction with the intervention
• •Results are discussed in the context of the high attrition recorded, implementation challenges, and future research pathways

1. Introduction

1.1. Background

Psychological difficulties in children can increase the risk of adverse outcomes in adult life (Kessler et al., 2007), and thus pose significant public health challenges. Internationally, many primary care psychology services are unable to fully meet the needs of children who are reporting growing levels of both low mood and anxiety (McGorry et al., 2013). Owing to the increased prevalence of childhood psychological difficulties reported across different regions (Baxter et al., 2014), primary care psychology resources have adapted their approaches to service provision by utilising technology-assisted interventions in what have become stepped-care models. Despite a limited evidence base, further adaptations to service provision have included the integration of staff with differing levels of psychology qualifications, such as assistant psychologists (APs) or primary care graduate mental health workers (psychology graduates trained in brief therapy techniques) (Bower, 2002). In response to the enduring undersupply of appropriately-qualified clinicians and inaccessible services, the use of novel technologies within newly-established primary care roles has been posited as one way to roll-out evidence-based interventions that are: cost-efficient, scalable, destigmatising; and potentially more engaging (especially for children) (Raney et al., 2017).

1.2. Technology-assisted cognitive behavioral therapy (CBT) for children – theoretical and evidential underpinnings

CBT is regarded as one of the most effective psychotherapeutic interventions for mild-to-moderate low mood and anxiety for a range of populations (Carr and McNulty, 2016; Cuijpers et al., 2013). The rich theoretical foundation to CBT – formulated by Beck (Beck, 1967) and – states that there is a complex relationship between psychological difficulties (such as anxiety) and one's thoughts, feelings, and behaviours. CBT targets negative automatic thoughts (NATs) and underlying problematic core beliefs to help individuals think about their thinking (meta-cognition), and collaboratively devise more helpful strategies thereafter (Beck and Haigh, 2014; Beck, 2011). Repeatedly, meta-analyses of CBT have demonstrated that it is effective for different types of psychological difficulties in adults, such as anxiety and low mood (Hofmann et al., 2012).

However, compared to research on both face-to-face and technology-assisted CBT for adults, the weight of evidence for technology-assisted CBT is considerably less for children, particularly in naturalistic settings for those aged 12 and under (Bachmann et al., 2010; Pennant et al., 2015). This may be attributable to several factors, including: transitional challenges from research-to-practice (Fleming et al., 2014; Grimshaw et al., 2012), variable stakeholder uptake of – and attitudes to – new technologies (Du et al., 2013), comparatively smaller overall numbers of randomised controlled trials (RCTs) involving children in real-world settings (Lau et al., 2016), poor understanding of comorbidity (Anckarsäter, 2010), and debates about the impact on traditional therapeutic alliance (Tremain et al., 2020). Additionally, there are concerns regarding the extent to which many interventions are genuinely CBT-based (Stawarz et al., 2018; Wolpert et al., 2019); and poor implementation strategies are reported across settings (Blandford et al., 2018; Mohr et al., 2017).

A further key distinction is the unique challenges of using technology-assisted CBT with children aged 12 and under. As this age group presents with different developmental needs (Carr, 2015), traditional CBT requires appropriate tailoring for such children to offset the complexity of meta-cognitive demands within CBT tasks (Stallard, 2019). Technology-assisted CBT such as computer games and mobile apps can offer unique mechanisms for therapists to adapt dense traditional CBT materials into child-friendly content through a variety of avatar-led metaphors, story-telling and gamification (O'Reilly, 2018). Although these approaches have been effectively applied for adolescent low mood and anxiety - such as SPARX (Merry et al., 2012) - interventions specifically for children aged 12 and under are limited. Of the available RCTs on technology-assisted CBT exclusively for this population, there is evidence of clinical effectiveness for low mood and anxiety, but sample sizes are problematically low: Camp Cope-A-Lot (n = 49) (Khanna and Kendall, 2010); Think, Feel, Do (n = 20) (Stallard et al., 2011); and BRAVE for Children (n = 73) (March et al., 2009). There is also emerging evidence for technology-assisted interventions for children experiencing other issues, such as autism spectrum disorder (Tanaka et al., 2010); and the effective use of newer technologies such as virtual reality for phobias is building apace (Scozzari and Gamberini, 2011). From the point of view of children engaging in technology-assisted CBT, qualitative evidence synthesis suggests that they find it helpful, therapeutic, transferable, playful, but also with some limitations regarding broadness of content and some negative experiences (McCashin et al., 2019).

Recent meta-analyses have concluded that existing technology-assisted CBT interventions hold promise for young people (Vigerland et al., 2016). However, they have also clearly demonstrated significantly smaller effect sizes from what are lower quality studies involving children aged 12 and under (Arnberg and Ost, 2014; Ebert et al., 2015; Podina et al., 2016). Moreover, there is poor evidence for long-term positive outcomes (James et al., 2013). Nonetheless, in an updated review of the available evidence, the National Institute for Health and Care Excellence has recommended CBT as the first-response to mild-to-moderate depression and anxiety in children, with the latest recommendation identifying digital CBT as a first-response (Wise, 2019).

1.3. Pesky gNATs

Pesky gNATs is a seven-level 3D CBT computer game that utilises both theory-driven and evidence-based content to educate young people from the ages of 7 and above about core CBT concepts and skills (O'Reilly and Coyle, 2015). Facilitated by an appropriately qualified professional, the game is played by children experiencing low mood or anxiety. Pesky gNATs blends models from both clinical psychology and human-computer interaction (HCI) (van der Meulen et al., 2018), and is currently used by a range of therapists in different settings internationally. Preliminary evaluations and naturalistic deployment studies indicate that therapists highly rate the intervention for clinical use, and that it supports the CBT process (Coyle et al., 2011; van der Meulen et al., 2019). Furthermore, small scale feasibility and qualitative studies for Pesky gNATs and Mindful gNATs (a mobile app containing mindfulness-based content from Pesky gNATs) have demonstrated acceptability for school-aged children (Chapman et al., 2016; Tunney et al., 2017). In a recent RCT, an adapted version of Pesky gNATs for adults with intellectual disability (ID) found medium-sized effects on anxiety post-intervention, large-sized effects at 3-month follow-up, in addition to 40% of the sample showing clinically meaningful change at follow-up (Cooney et al., 2017). However, to date, there has been no formal RCT evaluation of Pesky gNATs for children in primary care settings.

1.4. Current study

To address the limitations of the extant literature, this study aimed to evaluate the effectiveness of Pesky gNATs in a naturalistic setting. Of note is that this study took place within an Irish service setting that has recently developed stepped care models, similar to other international models of primary care psychology where non-psychologist staff offer intervention services (Haas, 2004). Additionally, to improve accessibility and address growing waitlists, the systematic integration of assistant psychologists (APs) for the provision of technology-assisted CBT interventions has occurred for the first time in primary care in Ireland (Health Service Executive, 2016). Using a RCT design within primary care child psychology services in Ireland, this study investigated if Pesky gNATs is effective in reducing clinically significant levels of low mood and anxiety in children aged 8–12 in primary care as delivered by APs.

2. Method

2.1. Design

This study used a two-armed randomised controlled trial (RCT) parallel-design across three timepoints. Participants were randomly assigned to either the Pesky gNATs intervention group or the waitlist control group. Primary and secondary measures were collected pre-intervention (Time 1), post-intervention (Time 2), and at a three month follow-up (Time 3). The RCT registration is freely available at ISRCTN (60159987).

2.2. Participants

122 children were recruited from primary care psychology services within the Health Service Executive (HSE) in Ireland (see Section 3.1 for baseline characteristics). The HSE comprises nine community healthcare organisations (CHO) across Ireland that provide local areas with a wide range of services outside of acute hospital settings. These services also include child psychology therapies, parents courses, and monthly drop-in advice clinics. To participate in the RCT, participants had to meet the following inclusion criteria: aged between 8 and 12 years of age (inclusive), clinically significant levels of internalising problems according to the cut-offs established in the Child Behaviour Checklist (CBCL) (see Section 2.4), willingness and capacity to commit to the research timeline, informed consent by parent or guardian, and assent by the child. The exclusion criteria was: formal diagnosis of ID, interpersonal difficulties that could adversely impact participation (active psychosis, significant cognitive difficulties, and English language difficulties), or in formal care.

2.3. Intervention

The 7 scaffolded levels within Pesky gNATs each cover a central CBT concept: thoughts, feelings and behaviours (TFBs), cognitive monitoring (across 2 levels), cognitive restructuring, negative core belief identification, negative core belief reappraisal, and prelapse prevention. The guiding metaphor is one of negative automatic thoughts (NATs) in the form of gNATs or little flies that can sting us thereby impacting our TFBs. Applying a social narrative to align with these child-friendly metaphors, participants meet the explorer David gNATtenborogh who helps them identify, trap, and swat gNATs; as well as hunting them back to their hives and splatting them. Each level is designed to last approximately 45 min and is played by the child alongside their therapist to allow for the maintenance of a traditional therapeutic relationship. To offset any privacy and ethical concerns, in addition to optimising child autonomy, no usage data is collected from Pesky gNATs. A full breakdown of Pesky gNATs theory and content is provided in the supplementary materials.

To participate in this study, incoming APs from the HSE attended a 3-day training session in Pesky gNATs, following approval by their supervising clinical psychologist. Six training events were hosted between June 2018 and March 2019 in Dublin, Limerick, Galway or Kilkenny; where a total of 89 APs were trained. Facilitated by both the third and first author, training consisted of: the theoretical knowledge underpinning Pesky gNATs; practical technology skills; the clinical skills necessary for using the intervention in primary care; and an outline of the RCT protocol. Both passive (presentations and game demonstrations) and active training components (feedback, questions-and-answers, full gameplay experience, and role-plays) were used. APs were provided with an overall RCT materials folder, containing all paper-based outcomes, participant information sheets, consent and assent forms, and a checklist summary for the RCT timeline.

2.4. Outcomes

2.4.1. Primary outcome

Child Behaviour Checklist (CBCL) (Achenbach and Rescorla, 2001) The total internalising subscale of the Child Behaviour Checklist (CBCL) Age 6–18 was chosen as the primary inclusion and outcome variable to measure participant low mood and anxiety. The CBCL is a 118-item scale that measures parent perceptions of the competencies and difficulties of their child within a 6-month period. Reported directly by parents, the scale takes approximately 15 min to complete. Respondents are asked to rate their child on various behaviours using a three-point Likert scale ranging from 0 (not true) to 2 (often true). The CBCL produces an overall symptom index that comprises two broader dimensions of internalising (total of Anxious/depressed, Withdrawn-depressed, and Somatic complaints) and externalising problems (Rule-breaking and Aggressive behaviour). The internalising total scores are classified within the following ranges for children aged 6–11: normal (total score < 9), borderline (total score between 9 and 11) or clinical (total score ≥ 12). The CBCL has high test-retest reliability (range: 0.95–1.00), and strong inter-rater reliability (range: 0.93–0.96). Similar findings are reported for internal consistency (range: 0.78–0.97), and good validity has been established by various international studies (Ivanova et al., 2007). The CBCL is widely regarded as a cross-culturally valid and generalisable measure for psychological difficulties in young people.

2.4.2. Secondary outcomes

The remaining subscales of the CBCL were chosen as the secondary measures. This includes the total score for externalising problems, other problems, and an overall total problems score that subsumes all other scores. The CBCL also produces eight syndrome subscales: Withdrawn/Depressed, Anxious/Depressed, Somatic Complaints, Social Problems, Thought Problems, Attention Problems, Rule-Breaking Behaviour, and Aggressive Behaviour. Additionally, the CBCL calculates six DSM-oriented scales: Depressive Problems, Anxiety Problems, Somatic Problems, Attention Deficit Problems, Oppositional Defiant Problems and Conduct Problems. All raw scores are classified into normal, borderline, and clinical ranges relative to the general population. The externalising scale has excellent internal consistency (0.94) and inter-rater reliability (0.85).

Youth Self Report (YSR) Age 11–18 (Achenbach and Rescorla, 2001). The YSR serves as a complementary measure to the CBCL. It is a 112-item scale providing self-ratings on a range of problem items by the young person during the preceding 6 months. Aligning with the CBCL, the child is invited to rate various behaviours on a three-point Likert scale ranging from 0 (not true) to 2 (often true). As above, the YSR also contains the same eight syndrome subscales and six DSM-aligned subscales. The YSR has a mean test-retest reliability of 0.88 for the competence scales and 0.82 for the empirically-based problem scales, with good validity and cross-cultural reliability reported by several international studies (Ivanova et al., 2007). Note, the YSR is only applicable to participants aged 11 and over.

Within-Game Outcomes. Pesky gNATs was designed with the following standardised measures built within the software, and is routinely completed by participants during the game:

Child Outcome Rating Scale (CORS) (B. L. Duncan et al., 2003). The CORS is a 4-item visual analogue self-report outcome developed to measure a young person's personal wellbeing, interpersonal relationships, social relations and overall sense of wellbeing. Its authors report internal consistency of 0.84, with a test-retest reliability of 0.60 (Sparks et al., 2006). In Pesky gNATs, the CORS is presented at the beginning of each level.

Child Session Rating Scale (CSRS) (Barry L. Duncan et al., 2003). As per the CORS, the CSRS is also a 4-item visual analogue self-report outcome designed to measure the young person's perception of therapeutic alliance. The following dimensions are covered within the CSRS: the participant's view of the therapeutic relationship, the importance of goals, session engagement, overall session rating, and listening. High reliability has been reported (0.96), alongside a test-retest reliability of 0.50. In Pesky gNATs, the CORS is presented at the end of each level.

Revised Child Anxiety and Depression Scale (RCADS) (Chorpita et al., 2000). The RCADS is a 47-item, youth self-report questionnaire with subscales that measure: separation anxiety disorder (SAD), social phobia (SP), generalised anxiety disorder (GAD), panic disorder (PD), obsessive compulsive disorder (OCD), and major depressive disorder (MDD). It also produces a total anxiety scale and a total internalising scale. Using a 4-point Likert-scale from 0 (never) to 3 (always), participants are asked to rate how often each item applies to them. It has demonstrated high reliability across all subscales (SAD = 0.78; SOC = 0.87; OCD = 0.82; PD = 0.88; GAD = 0.84; MDD = 0.87). The authors report good test-retest reliability over a one-week period ranging from 0.65 (OCD) to 0.80 (SP). In Pesky gNATs, the RCADS is presented at the beginning of level 1, and at the end of the game.

Participant Feedback. Child and Parent Feedback forms. Finally, to gather personalised feedback relating to game-specific clinical experiences, a short questionnaire using both open and closed questions was implemented at Time 2 for intervention group participants and their parents. Containing 13 paper-based questions, participants were invited to rate the helpfulness of different features within Pesky gNATs on a Likert-scale from 1 (not at all helpful) to 5 (extremely helpful). Seven open-ended questions provided participants with the opportunity to give insights into their experiences (for example, ‘do you think you changed from playing Pesky gNATs? Please explain below’).

2.5. Sample size

According to Cohen, in order to detect a medium effect size using two-group ANOVAs, 64 participants per group would be necessary with statistical power of 80% and an α of 0.05 (Cohen, 1992). Using Cohen's criteria, the magnitude of effect size is as follows: 0.20 is small, 0.50 is medium, and 0.80 is large.

2.6. Randomisation and blinding

The simple randomisation procedure was implemented (Altman and Bland, 1999) by the first author. A random sequence of numbers was computed in Microsoft Excel (2016), alongside a separate column containing group allocation values (1 = intervention; 2 = control). Using the sort function in Excel, a random list of allocations was then generated. This Excel procedure was conducted by a researcher separate to the RCT to minimise bias. As this was a naturalistic study design involving a behavioral intervention, no formal blinding procedure was implemented.

2.7. Procedures

Following the completion of Pesky gNATs training, each CHO area identified potential participants using standard primary care protocols alongside the RCT inclusion and exclusion criteria. Where potential participants were identified, APs provided information sheets about the RCT to parents and children for consideration. Where potential participants were interested, they completed the CBCL, consent, and assent forms which were then returned by the AP to the first author. CBCLs were scored using the ASEBA Software, with the subsequent report used to determine eligibility. Eligible participants and their AP were then provided with a random allocation to either the intervention group or the waitlist control group by the primary researcher via email. To ensure pseudo-anonymity, participants received a uniquely generated code for use with all data collection and communications. Owing to the variability of psychology primary care services in Ireland, APs and supervisors had autonomy as regards their optimal recruitment and data transfer procedures for their area – this pragmatism was essential to ensuring sufficient flexibility within the parameters of the RCT protocol. Regular communications were maintained between APs and the research team using phone and email throughout the RCT timeline. All data was stored securely in IBM SPSS version 26.

2.8. Ethics

This study was fully approved by the Human Research Ethics Committee in University College Dublin, Ireland (ref. HS-18-76-McCashin-O'Reilly).

2.9. Statistical analysis

Following the comparison of baseline outcomes across groups, repeated measures ANOVAs (RM-ANOVAs) were performed across all timepoints. For the in-game measures, a summary descriptive analysis was performed to provide engagement statistics (but no group comparisons were applicable). As statistical significance does not provide sufficient insight into potentially clinically meaningful change, the Reliable Clinical Indices (RCI) was performed for both groups following the Jacobson and Truax guidelines (1991). In addition, a descriptive analysis on the parent and child feedback forms was used to further gauge any clinical insights regarding the intervention group progress between pre and post Pesky gNATs.

For primary outcome (CBCL-Internalising), and all subscales of the CBCL, there was 34.4% missing data at T2 (80/122 participant available); and 44.3% at T3 (54/122 participant data available). For all CBCL items, Little's ‘Missing Completely At Random’ (MCAR) demonstrated that data across both groups was missing completely at random at T1 (χ2(5134) = 4923.90, p = .98), T2 (χ2(3984) = 1615.90, p = 1.00), and T3 (χ2(2593) = 212.31, p = 1.0). In keeping with best practice for handling missing data in RCTs (Dong and Peng, 2013; Jakobsen et al., 2017), an intention-to-treat (ITT) approach was implemented if 10% or less of individual case data was missing at the respective time point. Following an intention-to-treat approach, at each timepoint, multiple imputation using the median replacement was performed on cases that had 10% or less missing data using the ‘Replace Missing Values’ function within IBM SPSS version 26. To ascertain the degree to which group differences were present, effect sizes were calculated using the Cohen criteria (1998).

All in-game measures (RCADs, CORS, CSRS) were analysed using a Per Protocol (PP) analysis, as no between-group comparisons were possible. PP analysis was also applied to the YSR due to its limited applicability for this sample that was predominantly under 11 (mean age = 9.9) (see Supplementary section).

3. Results

The outcomes from this research comprise four parts: baseline participant comparisons and CONSORT; repeated measures ANOVAs (RM-ANOVAs) and effect sizes; clinically meaningful change using the RCI; and participant feedback.

3.1. Baseline participant comparisons and CONSORT

To assess any potential baseline group differences and to ensure comparability, a series of independent t-tests and Pearson's chi-squared tests were performed on all applicable primary and secondary outcomes. As summarised in Table 1, there were no significant group differences on any of the demographic variables or clinical characteristics.

Table 1.

Baseline demographics and group differences.

	Intervention group (n = 58)	Waitlist control group (n = 64)	Test statistic
Age, Mean (SD)	9.90 (1.21)	9.97 (1.32)	t(120) = −0.31, p = .75
Female, n (%)	24 (41%)	27 (42%)	χ2(1) = 0.01, p = .93
CBCL total scores, Mean (SD): Internalising	25 (8.77)	26.53 (8.94)	t(120) = −0.95, p = .34
Externalising	12.26 (7.33)	13.28 (9.22)	t(120) = −0.67, p = .50
Total problems	63.50 (23.03)	68.95 (24.75)	t(120) = −1.27, p = .21
Depressive problems Anxiety problems Somatic problems Attention/deficit problems Oppositional defiant problems Conduct problems	7.93 (4.18) 10.34 (3.38) 3.60 (2.85) 5.41 (3.43) 4.10 (2.41) 2.55 (2.98)	8.56 (3.86) 10.89 (3.43) 4.17 (3.24) 6.36 (3.47) 4.31 (2.65) 3.39 (4.07)	t(120) = −0.87, p = .39 t(120) = −0.88, p = .38 t(120) = −1.02, p = .31 t(120) = −1.51, p = .13 t(120) = −0.45, p = .65 t(120) = −1.29, p = .20

In Fig. 1, the CONSORT diagram illustrates the 140 participants who were screened for the RCT, from which 122 were eligible and provided consent and assent (Time 1). The simple randomisation procedure allocated 58 participants to the intervention group to receive Pesky gNATs and 64 to the waitlist control group. At Time 2, the participant retention rate was 66%; with an even split of 40 participants per group. However, at the follow-up (Time 3), the participant retention rate was 44%, with 54 participants remaining (23 in intervention; 31 in control).

Fig. 1.

CONSORT flowchart of participants throughout the RCT.

3.2. RM-ANOVAs and effect sizes

To examine any statistically significant group differences between T1 and T2 timepoints, a series of RM-ANOVAs were conducted on the primary outcome (CBCL-Internalising total score), and all available secondary measures. A statistically significant main effect of time was observed F(1, 78) = 37.64, p < .001, but no significant group × time interactions were found. No significant group × time interactions were seen across the secondary measures, except for the DSM-oriented Somatic Problems scale (F(1, 78) = 4.12, p ≤ 0.05) and the Attention/Deficit scale (F(1, 78) = 4.71, p < .05). Using the Bonferroni adjustment for multiple comparisons, post hoc pairwise comparisons for the somatic problems scale demonstrated that the intervention group had a statistically significant mean difference of −1.11 (p < .05). For the Attention/Deficit scale, post hoc comparisons showed a statistically significant mean difference of −1.61 (p < .05) between the groups. The descriptive statistics and all group × time interactions are presented in Table 2.

Table 2.

Time 1 – Time 2 Repeated-measures ANOVA results and descriptive statistics for intervention and control groups.

Outcome	Intervention (n = 40)		Control (n = 40)		Group × time RM-ANOVAs
	T1 M (SD)	T2 M (SD)	T1 M (SD)	T2 M (SD)
CBCL Internalising	25 (9.28)	18.80 (9.18)	25.83 (8.09)	20.76 (8.81)	F(1, 78) = 0.65, p = .42
Externalising	11.56 (7.73)	9.45 (7.97)	12.10 (9.11)	12.48 (10.96)	F(1, 78) = 0.86, p = .36
Total problems	61.60 (23.42)	47.18 (26.42)	64.93 (23.87)	57.63 (29.52)	F(1, 78) = 1.63, p = .21
DSM-problem subscales:
Depressive	7.98 (4.49)	5.68 (4.08)	8.00 (3.80)	6.95 (4.35)	F(1, 78) = 0.57, p = .45
Anxiety	10.30 (3.56)	8 (3.93)	10.60 (3.30)	8.95 (3.63)	F(1, 78) = 0.76, p = .39
Somatic	3.43 (2.84)	2.40 (2.16)	4.60 (3.13)	3.45 (2.64)	F(1, 78) = 4.12, p = .046*
Attention/deficit	4.85 (3.35)	4.03 (2.92)	6.18 (3.69)	5.93 (4.02)	F(1, 78) = 4.71, p = .033*
Oppositional defiant	3.65 (2.36)	3.23 (2.26)	4.05 (2.67)	3.83 (2.80)	F(1, 78) = 0.89, p = .35
Conduct	2.53 (3.20)	2.35 (3.03)	3.03 (4.26)	3.63 (5.18)	F(1, 78) = 1.09, p = .30

To examine any statistically significant group differences throughout the RCT, a series of RM-ANOVAs between all timepoints were performed for the primary and secondary outcomes. A statistically significant main effect of time was observed F(2, 104) = 20.91, p < .001, but no significant group or group × time interactions were found across either the primary or secondary outcomes. A summary of the descriptive statistics and all group × time interactions are presented in Table 3. A visual representation of the significant effect of time on both groups is provided in Fig. 2.

Table 3.

Repeated-measures ANOVA results and descriptive statistics for intervention and control groups across time.

Outcome	Intervention (n = 23)			Control (n = 31)			Group × time RM-ANOVAs
	T1 M (SD)	T2 M (SD)	T3 M (SD)	T1 M (SD)	T2 M (SD)	T3 M (SD)
CBCL Internalising	25.17 (9.59)	19.44 (10.43)	17.83 (10.34)	25.57 (8.72)	19.80 (9.47)	18.90 (9.70)	F(1, 52) = 0.07, p = .79
Externalising	11.88 (8.44)	10.69 (8.88)	9.83 (7.43)	11.10 (8.73)	11.03 (9.46)	9.77 (8.71)	F(1, 52) = 0.06, p = .94
Total problems	62.83 (25.53)	50.15 (30.63)	47.33 (27.16)	64.00 (24.13)	54.52 (29.02)	49.87 (27.77)	F(1, 52) = 0.16, p = .70
DSM-problem subscales:
Depressive	8.13 (4.55)	6.21 (4.93)	5.33 (4.30)	7.77 (4.12)	6.33 (4.49)	5.80 (4.35)	F(1, 52) = 0.01, p = .94
Anxiety	10.63 (3.55)	8.38 (4.17)	7.88 (4.93)	10.77 (3.43)	8.87 (4.09)	8.33 (4.05)	F(1, 52) = 0.14, p = .71
Somatic	3.50 (2.98)	2.52 (2.30)	2.29 (2.05)	4.67 (2.99)	3.63 (2.59)	3 (2.57)	F(1, 52) = 2.59, p = .11
Attention/deficit	4.50 (3.01)	4.04 (3.16)	4.04 (3.28)	6.13 (3.60)	5.78 (4.03)	5.33 (3.84)	F(1, 52) = 2.99, p = .09
Oppositional defiant	3.79 (2.28)	3.54 (2.38)	3.38 (2.24)	3.87 (2.49)	3.67 (2.55)	3.50 (2.37)	F(1, 52) = 0.03, p = .86
Conduct	2.83 (3.84)	2.71 (3.48)	2.21 (2.34)	2.70 (4.29)	3 (4.59)	2.43 (3.88)	F(1, 52) = 0.02, p = .90

Fig. 2.

Changes in primary outcome across time in both groups (using error bars with 95% confidence intervals).

Using Cohen's d, a summary table of effect sizes is provided in Table 4. Large effect sizes were observed within both groups across time on the primary outcome measure. However, between group effect sizes were small at Time 2, and negligible at Time 3.

Table 4.

Effect sizes (d) between and within groups across time for primary and secondary outcomes [95% confidence intervals].

Outcome	Intervention within-group ES	Intervention within-group ES	Control within-group ES	Control within-group ES	Between-group ES	Between-group ES
	T1 v. T2	T1 v. T3	T1 v. T2	T1 v. T3	T2	T3
CBCL Internalising	1.34 [0.18, 0.46]	1.45 [0.06, 0.56]	1.49 [0.12, 0.53]	1.53 [0.21, 0.52]	0.22 [0.01, 0.10]	0.11 [0.01, 0.09]
Externalising	0.79 [0.01, 0.33]	0.75 [0.01, 0.36]	0.14 [0.01, 0.12]	0.52 [0.01, 0.27]	0.32 [0.01, 0.13]	0.01 [0.01, 0.01]
Total problems	1.32 [0.08, 0.48]	1.29 [0.10, 0.61]	0.96 [0.02, 0.39]	1.52 [0.10, 0.56]	0.38 [0.01, 0.14]	0.09 [0.01, 0.06]
DSM-problem subscales:
Depressive	1.27 [0.07, 0.47]	1.28 [0.03, 0.51]	0.74 [0.01, 0.32]	1.25 [0.05, 0.49]	0.31 [0.01, 0.01]	0.11 [0.01, 0.09]
Anxiety	1.24 [0.07, 0.47]	1.28 [0.03, 0.52]	1.20 [0.05, 0.45]	1.32 [0.06, 0.51]	0.26 [0.01, 0.11]	0.11 [0.01, 0.08]
Somatic	0.78 [0.01, 0.33]	0.98 [0.00, 0.03]	1.19 [0.06, 0.45]	1.36 [0.01, 0.02]	0.44 [0.01, 0.16]	0.31 [0.01, 0.15]
Attention/deficit	0.64 [0.01, 0.29]	0.31 [0.01, 0.22]	0.26 [0.01, 0.17]	0.83 [0.01, 0.37]	0.55 [0.01, 0.19]	0.36 [0.01, 0.17]
Oppositional defiant	0.47 [0.01, 0.22]	0.33 [0.01, 0.29]	0.28 [0.01, 0.16]	0.91 [0.01, 0.24]	0.24[0.01, 0.10]	0.06 [0.01, 0.01]
Conduct	0.17 [0.01, 0.13]	0.42 [0.01, 0.26]	0.41 [0.01, 0.21]	0.24 [0.01, 0.18]	0.31 [0.01, 0.12]	0.06 [0.01, 0.05]

3.3. Clinically meaningful change

To ascertain the extent to which Pesky gNATs was associated with clinically significant change in participant's lives, the Jacobson-Truax method was applied to compute the reliable change index (RCI) at both post-intervention and follow-up (Jacobson and Truax, 1991). Participants were defined as in recovery if they moved from the clinical range of total internalising problems (CBCL) to the non-clinical range, as defined by Achenbach and Rescorla (2001), in addition to holding an RCI value larger than 1.96. As summarised in Table 5, at Time 2, recovery rates between the intervention and control groups were significantly different.

Table 5.

Chi-squared tests on recovery rates at Time 2 and Time 3 for both groups.

	Intervention	Control	Chi-squared	P	Interpretation
Time 2, n (%)
In recovery	9 (22.5%)	0 (0%)	7.53	0.006⁎	Intervention > control
Non-recovery	31 (77.5%)	40 (100%)
Time 3, n (%)
In recovery	3 (13%)	1 (3.2%)	1.86	0.17	No group differences
Non-recovery	20 (87%)	30 (96.8%)

^⁎Statistically significant at p < .05.

3.4. Participant feedback at T2

At Time 2, after intervention group participants completed Pesky gNATs, both the child and parent completed a feedback questionnaire. A full summary of feedback regarding the helpfulness of the intervention across a range of areas is summarised in Fig. 3 (parent data), and Fig. 4 for (child data).

Fig. 3.

Parent rating of the helpfulness of Pesky gNATs across feedback questionnaire.

Fig. 4.

Child rating of the helpfulness of Pesky gNATs across feedback questionnaire.

When asked if Pesky gNATs would be helpful into the future for managing psychological difficulties, 88% of children said yes; and 83% of parents said yes. Additionally, when asked if they would recommend Pesky gNATs to other children or parents with children experiencing similar psychological difficulties, 90% of children said yes; and 88% of parents said yes respectively.

3.5. Safety

No adverse outcomes were reported throughout the RCT.

4. Discussion

This study used a pragmatic real-world RCT to assess if the technology-assisted CBT game Pesky gNATs – as delivered by APs – was effective in reducing clinically significant levels of anxiety and low mood in children in primary care. Overall, both intervention and waitlist control groups experienced statistically significant reductions in internalising problems over time. There was no between-group differences on the primary outcome, and only small effect sizes were observed. When examining clinically meaningful change using the RCI, there was a was a statistically significant result demonstrating that more participants from the intervention group moved from clinical to non-clinical levels of internalising difficulties at Time 2. This effect was not maintained at follow-up. Participant feedback data from both children and parents showed high levels of acceptability, helpfulness and positive change.

In contrast to the literature concerning similar interventions, this study did not yield results suggesting that technology-assisted CBT is effective in primary care using staff with different levels of psychology qualifications, such as assistant psychologists (Khanna and Kendall, 2010; Stallard et al., 2011). In the context of the broader meta-analytic literature for the effectiveness of technology-assisted CBT, these findings are in keeping with the small overall effect sizes and limited evidence of effectiveness synthesised elsewhere (Pennant et al., 2015; Bachmann et al., 2010; Ebert et al., 2015). However, unlike this literature, the present study did not find any statistically significant effects to demonstrate the clinical effectiveness of technology-assisted CBT for children.

The potential explanation for these contrasting findings is likely attributable to a number of interlinked factors, including: regression to the mean, ineffectual intervention, study quality, the naturalistic setting of primary care, and the ongoing debate about the appropriateness of using APs for CBT. Prior systematic reviews (such as Pennant et al., 2015) on technology-assisted CBT for children found that the existing RCTs were of low quality, thus this current RCT may have been more reflective of the true effects due to its larger size within an applied setting, and thus potential for higher quality. With respect to the naturalistic setting of primary care, this RCT may have been impacted by immeasurable confounding variables that impeded the ability of the intervention to provide clinical benefit. The ongoing debate about the appropriateness of using staff other than professionally-qualified psychologists in the provision of a full CBT intervention for children is also a potential explanation for the findings. Although no comparison was made between professional psychologists and APs with respect to clinical competencies and proficiency with Pesky gNATs, it is possible that the use of fully-qualified clinical psychologists may have produced clinically-significant results.

Based on the quantitative findings, this study does not suggest that Pesky gNATs provided added benefit and value to participants – despite this, the qualitative findings suggest positive experiences. According to participants, Pesky gNATs is highly acceptable and experienced as helpful. A separate qualitative paper from this RCT found that APs also mirrored this feedback (McCashin et al., 2020), as does prior preliminary research on the use of Pesky gNATs in real-world settings (Coyle et al., 2011; Van der Meulen et al., 2019; Chapman et al., 2016; Tunney et al., 2017).

4.1. Strengths and limitations

This was the first pragmatic RCT to evaluate Pesky gNATs in an externally valid naturalistic service setting. The controlled design ensured that two comparable groups were randomly recruited and measured across time, including a 3-month follow-up. The use of the RCI and feedback questionnaires provided insights into the clinically meaningful process of change for intervention group participants at Time 2, and provided context for the other outcomes. Taken together, although no significant group differences were observed over time, there is tentative evidence that Pesky gNATs can still provide clinically meaningful change; and is well-regarded by parents, APs, and children – itself evidenced by the majority of stakeholders recommending its continued use in primary care.

Similar to other RCTs (Vigerland et al., 2016), this study experienced significant attrition challenges (44% retention at T3) which limited its ability to detect effects due to lower statistical power (Field, 2018). Furthermore, this RCT only occurred within primary care which brings with it a number of complexities – the potential impact of which were unfeasible to measure, such as: the effects of different waitlist durations for participants across sites, the role of parents and concurrent use of other primary care services, and variable service delivery. Qualitative data from APs in this RCT and from a systematic review suggest that the optimal role of parental involvement throughout a child's intervention is very important, but remains unclear (McCashin et al., 2019; McCashin et al., 2020). However, the potential effects of too much or too little parental involvement were beyond the scope of this RCT. Moreover, this study did not measure any potential comorbidity in the sample; and no data was available from participant's use of the supporting mobile app or workbook. Finally, due to logistical delays in setting up the RCT nationally, only a small proportion of those trained in Pesky gNATs (n = 89) were eventually able to offer it (n = 23).

4.2. Future implications

Future research will require larger sample sizes to definitively test the effectiveness of Pesky gNATs over time, and in different service settings. The extent to which there are differences in clinical outcomes between qualified CBT practitioners versus APs who provide the intervention also merits further investigation. Clinically, although this RCT demonstrates that Pesky gNATs is both functional and acceptable; but further research is required to ascertain its clinical effectiveness, and the extent to which it adds value to service provision.

Declaration of competing interest

The authors declare the following financial interests/personal relationships which may be considered as potential competing interests: this research is part of TEAM, an Innovative Training Network funded by the European Union's Horizon 2020 research and innovation program under the Marie Skłodowska-Curie grant agreement number 722561.

GOR and DC are the creators of Pesky gNATs, which is owned by Handaxe Community Interest Company (CIC), a not-for-profit company. GOR and DC are directors of Handaxe CIC but receive no financial reward for this role.

Appendix A. Supplementary data

Supplementary material