Using a Smartphone App and Clinician Portal to Enhance Brief Cognitive Behavioral Therapy for Childhood Anxiety Disorders

Abstract

Cognitive behavioral therapy (CBT) is an efficacious treatment for child anxiety disorders, but 40–50% of youth do not respond fully to treatment, and time commitments for standard CBT can be prohibitive for some families and lead to long waiting lists for trained CBT therapists in the community. SmartCAT 2.0 is an adjunctive mobile health program designed to improve and shorten CBT treatment for anxiety disorders in youth by providing them with the opportunity to practice CBT skills outside of session using an interactive and gamified interface. It consists of an app and an integrated clinician portal connected to the app for secure 2-way communication with the therapist. The goal of the present study was to evaluate SmartCAT 2.0 in an open trial to establish usability, feasibility, acceptability, and preliminary efficacy of brief (8 sessions) CBT combined with SmartCAT. We also explored changes in CBT skills targeted by the app. Participants were 34 youth (ages 9–14) who met DSM-5 criteria for generalized, separation, and/or social anxiety disorder. Results demonstrated strong feasibility and usability of the app/portal and high satisfaction with the intervention. Youth used the app an average of 12 times between each therapy session (M = 5.8 mins per day). At post-treatment, 67% of youth no longer met diagnostic criteria for an anxiety disorder, with this percentage increasing to 86% at two-month follow-up. Youth showed reduced symptom severity over time across raters and also improved from pre- to post-treatment in CBT skills targeted by the app, demonstrating better emotion identification and thought challenging and reductions in avoidance. Findings support the feasibility of combining brief CBT with SmartCAT. Although not a controlled trial, when benchmarked against the literature, the current findings suggest that SmartCAT may enhance the utility of brief CBT for childhood anxiety disorders.

Anxiety disorders are the most common class of disorders among children and adolescents worldwide, affecting about 6.5% of youth at any time (Polanczyk, Salum, Sugaya, Caye, & Rohde, 2015). Generalized anxiety disorder (GAD), social anxiety disorder (SocAD), and separation anxiety disorder (SepAD), in particular, tend to be highly comorbid with each other and prevalent in children (Costello, Mustillo, Erkanli, Keeler, & Angold, 2003; Lahey et al., 2008). Anxiety in youth can be chronic and impairing (Costello, Copeland, & Angold, 2011; Costello et al., 2003), contributing to social and academic problems and the development of depression and substance abuse (Costello et al., 2003; Cummings, Caporino, & Kendall, 2014).

Multiple independent randomized trials support the efficacy of CBT in treating childhood anxiety disorders (Kendall et al., 1997; Silverman et al., 1999; Walkup et al., 2008), yet obstacles prevent many anxious youth from receiving CBT treatment in the community. First, most protocols require a commitment of 12 to 20 weekly sessions with a trained CBT therapist (A. C. James, A. Soler, & R. Weatherall, 2005). For example, “Coping Cat” (Kendall & Hedtke, 2006a), one of the most widely disseminated CBT interventions for childhood anxiety, includes 14–16 sessions. Some families may be unable to commit to a 4–5 month treatment due to work schedules, family commitments, and transportation difficulties (Collins, Westra, Dozois, & Burns, 2004), and some practitioners find the number of sessions to be a challenge in managed care settings (Nelson, Steele, & Mize, 2006). Second, there is insufficient availability in many communities of CBT-trained therapists (Collins et al., 2004), leading to lengthy waitlists for the few available trained clinicians. Third, costs associated with treatment can present challenges to healthcare consumers and providers (Nelson et al., 2006).

Briefer CBT for childhood anxiety may help to overcome these obstacles. A briefer approach would be more feasible for busy families, more affordable for consumers and healthcare systems, and enable in-demand CBT therapists to make treatment accessible to a larger number of youth. Briefer CBT may also be more feasible to implement in busy community settings, such as primary care offices (Weersing, Rozenman, Maher-Bridge, & Campo, 2012) and schools (Langley, Nadeem, Kataoka, Stein, & Jaycox, 2010). Effective brief treatments could also facilitate faster amelioration of symptoms and impairment. Kendall and colleagues developed a brief (8-session) cognitive-behavioral therapy (BCBT) for anxiety disorders adapted from 16 session Coping Cat (Kendall & Hedtke, 2006a, 2006b). The BCBT program, which focuses heavily on exposure, was evaluated with 26 youth with SAD, GAD, and/or SP (Crawley et al., 2013). Satisfaction with treatment was favorable, and recruitment, retention, and treatment fidelity ratings indicated that the program was feasible to implement. At post-treatment, 42% of youth were free of their primary anxiety diagnosis, but this percentage decreased to 33% at two-month follow-up. By comparison, a meta-analysis of childhood anxiety trials revealed an average response rate of 56% across lengthier (typically 16 session) CBT trials (A. James, A. Soler, & R. Weatherall, 2005), and the large-scale multi-site CAMS trial yielded a similar response rate of 60% for Coping Cat (Walkup et al., 2008). These data are consistent with meta-analytic findings from 48 CBT trials for child anxiety (Reynolds, Wilson, Austin, & Hooper, 2012), which indicated that effect sizes for brief treatments (≤ 8 sessions) were favorable but smaller (d=.35) than effect sizes for lengthier (13–16 session) treatments (d=.75). Thus, although brief CBT for childhood anxiety is acceptable to families and feasible to implement, and outcomes are favorable, the recovery rates are not as impressive as those from full-length CBT trials.

Approaches that engage youth out-of-session could be particularly helpful in making up for “lost sessions” in BCBT. Youth may be less successful in brief treatment because they don’t understand CBT concepts and skills well enough or they don’t obtain sufficient practice actually using the skills in anxiety provoking situations. Research shows that learning skills is not sufficient for reducing anxiety; these skills must be applied during exposure to anxiety eliciting stimuli (Hudson, 2005; Kendall et al., 2005). “Homework” is routinely assigned in CBT as a way to consolidate skill acquisition and encourage skill use in the real world. A recent meta-analysis provides evidence that the use and completion of homework enhances therapy outcomes in adults (Kazantzis, Whittington, & Dattilio, 2010), especially when the homework incorporates exposure (Cammin-Nowak et al., 2013). The role of homework in treatment response in child CBT has received little attention, but one study showed that among youth who received CBT for anxiety, those who were found to be treatment responders were more likely to have been assigned home-based exposures (Tiwari, Kendall, Hoff, Harrison, & Fizur, 2013). Yet, not surprisingly, youth often fail to complete therapy homework. This non-completion can occur for a variety of reasons, including (1) not remembering to complete homework, (2) not wanting to complete written and/or exposure-based homework (both of which can be aversive and/or anxiety-provoking to youth), and (3) not understanding the skills needed to complete the homework.

Recent advances in electronic health technologies offer potential to overcome barriers to home-based skills practice for youth, thus providing extra out-of-session learning opportunities (Berry & Lai, 2014). There are several computer-based CBT interventions for child anxiety that have shown excellent promise, such as BRAVE (March, Spence, & Donovan, 2009), Camp-Cope-a-Lot (Khanna & Kendall, 2010), and Cool Kids/Cool Little Kids/Cool Teens Online (Morgan et al., 2017; Rapee et al., 2006; Wuthrich et al., 2012). These programs provide CBT skill instruction using a sequence of “sessions” delivered online or via DVD, and typically also include contact with a therapist or mental health “coach.”

These programs focus on skill acquisition, but do not necessarily increase skill use in daily life as they are delivered in discrete chunks of time, similar to therapy sessions. In contrast, Ecological Momentary Interventions (EMI’s) deliver therapeutic content in real-time using portable devices (Heron & Smyth, 2010). Initial studies used PDA’s and text messaging, which showed promise in enhancing the efficacy of behavioral therapies for child disorders (Thomas, Ellis, Konrad, Holzer, & Morrissey, 2009). However, the rapid spread of smartphone use among youth offers unprecedented opportunities that could be leveraged by mental health practitioners. Smartphones can be used to provide opportunities for more frequent but briefer periods of skill learning and rehearsal, prompts to use skills during daily activities, as well as motivational tools such as digital rewards and e-communications from therapists. Because today’s youth are constantly processing and communicating information via smartphone, they may find this mode of engaging with therapy content more appealing than traditional paper and pencil approaches.

In recognition of the potential for smartphone apps to help youth with emotional problems, there has been a recent proliferation of industry-developed apps targeting child and teen mental health (Grist, Porter, & Stallard, 2017; Hollis et al., 2017). In contrast, apps developed in academia that are guided by evidence-based research have been slower to reach the market (Bry, Chou, Miguel, & Comer, 2018) and the evidence base for the efficacy of apps currently available on the marketplace remains unclear (Grist et al., 2017). The majority of mental health apps geared toward youth do not cite any evidence of efficacy (Radovic et al., 2016). This is consistent with a broader trend among mental health apps. For example, a recent systematic review of CBT apps in the marketplace revealed that only 9 out of 500 available apps (< 1%) had been evaluated in published research (Torous, Levin, Ahern, & Oser, 2017).

There is a similar lack of evidence for consumer apps that focus specifically on anxiety among youth. Bry et al. (2018) reviewed the 121 apps that were marketed toward youth with anxiety on the Google play and Apple store as of 2016. They found that none of the apps covered the full range of evidence-based components considered central to CBT for anxiety (e.g. exposure, psychoeducation, self-monitoring, contingency management, problem-solving, thought challenging). Less than half of the apps included any of these evidence-based components, and only 23% incorporated even two or more evidence-based components. Non-evidence-based components primarily consisted of games, coloring, and visual activities that could be used for distraction from anxiety. Bry et al. (2017) also noted that very few apps leveraged advanced functionalities of smartphones, such as GPS, cameras, sensors, or microphones. They also commented that few apps took advantage of the opportunity to collect real-time data about the youths’ experience of anxiety in daily life that could facilitate personalized treatment. For example, only 4% of apps collected real-time data on emotional state or behavior and only 2.5% contained personalized or customizable content. None of the apps offered mechanisms for integration with clinical care. Based on these results, Bry et al. concluded that there was a pressing need for more comprehensive anxiety apps that incorporated multiple evidence-based components and could be personalized to the individual needs of anxious youth.

While industry-based apps have been faster to reach the marketplace, several academic research teams have designed mHealth apps that are based on scientific evidence. For example, Anxiety Coach was developed at the Mayo Clinic to help patients with anxiety conduct exposures in the home environment (Whiteside, 2016). Although the content is not tailored towards children, it is considered appropriate for adolescents (Carper, 2017). However, Anxiety Coach has not yet been empirically validated in terms of usability or efficacy (Whiteside, 2016). REACH (Stoll, Pina, Gary, & Amresh, 2017) is a smartphone app designed for prevention and early intervention of anxiety in children. It incorporates multiple evidence-based treatment activities, including exposure, thought challenging, and contingency management, uses a child friendly interface, and has the capability to export a data file that can be shared with providers. REACH has not yet been evaluated for efficacy, but has demonstrated adequate usability.

The literature indicates that, while there is potential for smartphone apps to be used to improve the efficiency and efficacy of CBT interventions for child anxiety, existing programs have not yet been effectively designed, leveraged, and evaluated for this purpose. SmartCAT (Smartphone-Enhanced Child Anxiety Treatment) is an mHealth platform, including a patient app and integrated clinician portal, as well as real-time communication connecting the two, that was designed to enhance CBT for child and adolescent anxiety disorders. We developed SmartCAT as an adjunct to treatment rather than a standalone intervention based on Mohr, Cuijpers, and Lehman’s (2011) model of “Supportive Accountability,” which demonstrates that mHealth interventions tend to be more effective when paired with some level of human support and accountability. Consistent with this model, existing computerized CBT interventions for anxiety in youth incorporate regular contact with a therapist or mental health coach (Khanna & Kendall, 2010; March et al., 2009; Wuthrich et al., 2012). We similarly reasoned that this need for supportive accountability from a therapist would be important in the context of an exposure-based CBT protocol, which requires considerable encouragement, motivation, and support.

As noted by Bry et al. (2017), SmartCAT incorporates all 7 evidence-based treatment components indicated in the literature to be central to the treatment of child anxiety. SmartCAT incorporates real-time anxiety monitoring, integrates with clinical care, and takes advantage of advanced features of mobile technology such as context-sensitive prompting via GPS, camera, and AV capabilities. Preliminary feasibility and usability for an earlier version of SmartCAT (SmartCAT 1.0) were previously demonstrated in a small pilot study (Pramana, Parmanto, Kendall, & Silk, 2014). Based on participant feedback, a new version of the app, SmartCAT 2.0, was designed to be more child friendly, engaging, and interactive. SmartCAT 2.0 leverages current mobile technology capabilities to accomplish several goals, including: (1) automatically cueing youth to practice skills at home at prespecified times and places, so that youth have the opportunity to practice even when they forget to initiate skills practice on their own; (2) increasing youths’ desire to practice skills by providing more fun and interactive ways to learn these skills (i.e. games), and using digital rewards to reinforce practice, (3) creating opportunities and reminders for personalized home-based exposures, and (4) providing a safe and convenient platform for encouragement and communication between therapists and patients using secure communication for messaging and exchanging audiovisual materials (i.e. photos, videos). We recently demonstrated that the new gamified version, SmartCAT 2.0 was more engaging to patients compared to SmartCAT 1.0, based on frequency and duration of app use (Pramana et al., 2018). The present study reports on the usability of the SmartCAT 2.0 app (as assessed by youth) and clinician portal (as assessed by therapists) and on the feasibility and acceptability to children and families of integrating SmartCAT 2.0 with 8 sessions of brief CBT. We also aimed to establish preliminary efficacy of the combined intervention as a first step to support a future randomized controlled trial as a more rigorous test of efficacy.

An additional exploratory goal of the present study was to generate evidence of target engagement by demonstrating that the intervention was associated with improvements in the specific CBT skills targeted in the app’s activities. This goal is consistent with the NIMH’s experimental therapeutics paradigm in which the first stage of intervention evaluation is to demonstrate that the intervention exerts an observable effect on hypothesized targets or mechanism of action (Insel, 2012). Although the development and use of CBT skills is presumed to be a key mechanism underlying CBT interventions for children, skills are rarely formally assessed (Lindhiem, Higa, Trentacosta, Herschell, & Kolko, 2014). Existing anxiety trials have mostly relied on reports of the frequency of negative automatic thoughts (Kendall & Treadwell, 2007; Muris, Mayer, den Adel, Roos, & van Wamelen, 2009) or homework completion (Kazantzis, Deane, & Ronan, 2000) as the indices of skill use or knowledge. In contrast, performance-based assessments, in which individuals are asked to demonstrate use of a skill and their performance is scored using a rubric, are recognized as the gold standard for measuring skill acquisition (Johnson, Penny, & Gordon, 2009). We included a battery of performance-based evaluations to assess changes in skill acquisition with treatment, focusing on four skills targeted in the app: emotion identification, thought challenging, problem solving, and decreasing avoidance/approaching fears. This goal was considered exploratory because performance-based measures of CBT skills in youth have not yet been widely validated for use in measuring change with treatment, thus we had to adapt existing measures for this purpose.

In summary, as described in Pramana et al. (2018), we developed and refined a more interactive and personalized version of SmartCAT (Version 2.0) that would incorporate existing features of SmartCAT 1.0, such as a skills coach, media library, therapist messaging, and digital rewards, with novel interactive features including: (a) a series of “skillbuilder” interactive activities that include animation and gaming, (2) context-sensitive location- and time-based activation of activities in settings and/or times that frequently trigger anxiety, and (3) a therapist-customized home challenge menu to facilitate completion of home-based exposures. The first goal of the present study was to demonstrate that the SmartCAT 2.0 system is usable and acceptable to children, families, and therapists and feasible to deliver in conjunction with brief CBT. The second goal was to establish preliminary efficacy by examining rates of remission of anxiety diagnosis and reductions in anxious symptomatology in an open trial. As an exploratory goal, we also examined whether the use of brief CBT + SmartCAT was associated with improvement in skills targeted by the mHealth platform.

Method

Participants

Participants were 34 anxious youth ages 9 to 14 (M = 11.40 [SD = 1.62]) who met DSM-5 (American Psychiatric Association, 2013) criteria for current GAD (n = 22), SAD (n = 15), and/or SocAD (n = 11). Participants (50% female) were 85% Caucasian, 3% Hispanic, and 15% Biracial (Table 1). Participants were recruited from a metropolitan American city through (1) referrals from local pediatricians; (2) letters sent through a University-sponsored research registry to families interested in participating in behavioral health research studies; and (3) community advertising via flyers, internet, and print publications. Exclusion criteria included (1) neuromuscular or neurological disorder, (2) current comorbid psychiatric diagnosis that would require alternative treatment or interfere with treatment [i.e. major depressive disorder, obsessive-compulsive disorder, post-traumatic stress disorder, conduct disorder, substance abuse or dependence, or ADHD combined type or predominantly hyperactive-impulsive type], (3) a lifetime diagnosis of autism spectrum disorder, bipolar disorder, or psychotic disorder, (4) a prior trial of ≥ 6 sessions of CBT, (5) IQ below 70 as assessed by the Wechsler Abbreviated Scale of Intelligence ([WASI] Wechsler, 1999) or reading level below 80 on the Wide Range Achievement Test-4 ([WRAT-4] Wilkinson & Robertson, 2006), (6) concurrent psychological therapy or treatment with anxiolytic or antidepressant medication (could be on medication for ADHD if dose had been stable for at least 4 weeks), and (7) acute suicidality or risk for harm to self or others.

Table 1.

Baseline Characteristics of Participants

	N (%)	Mean (SD)	Range
Demographic Characteristics
Age (years)	-	11.40 (1.62)	9.07 – 14.67
Sex
Female	17 (50%)	-	-
Race
White	29 (85.3%)	-	-
Hispanic	1 (2.9%)	-	-
Biracial	5 (14.7%)	-	-
Family socioeconomic status
Total Family Income*	-	7.84	2 – 10
Clinical Characteristics
Primary diagnosis of anxiety disorder
Generalized anxiety disorder only	11 (32.4%)	-	-
Social anxiety disorder only	5 (14.7%)	-	-
Separation anxiety disorder only	5 (14.7%)	-	-
Generalized and separation anxiety disorders	7 (20.6%)	-	-
Generalized and social anxiety disorders	3 (8.8%)	-	-
Social and separation anxiety disorders	2 (5.9%)	-	-
Generalized, social, and separation anxiety	1 (2.9%)	-	-
Anxiety Severity at Baseline
PARS 6-item Total Score	-	13.97 (3.75)	8 – 25
SCARED: Child Report	-	36.24 (9.63)	19 – 56
SCARED: Parent Report	-	36.41 (8.41)	17 – 53

Note. SCARED = Screen for Child Anxiety Related Emotional Disorders; PARS = Pediatric Anxiety Rating Scale

^*Total family income measured in increments of $10,000 from 1 (< 10,000) to 10 (100,000+). A mean of 7.84 indicates an annual income between $70,001 and $90,000. A range of 2 to 10 indicates annual incomes between $20,001 and $100,000+.

Procedure

This project had 2 phases. Phase I involved the development and usability testing of the Smart 2.0 app and portal (see Pramana et al., 2018). In Phase II, reported here, the feasibility, acceptability, and preliminary efficacy of implementing SmartCAT as an adjunct to brief CBT for child anxiety was evaluated. All study procedures were approved by the University’s Institutional Review Board. The study involved 5 assessments and 8 treatment sessions, all conducted in a University clinic. Following a brief phone screen, participants were scheduled for an intake assessment during which a clinical interviewer administered a structured diagnostic interview and rating scales/questionnaires to the child and his/her primary caregiver. Active, signed primary caregiver consent and youth assent were obtained for all participants after a detailed study explanation. Participants meeting study criteria were scheduled for their 2^nd visit, a CBT skills pre-test, during which participants completed a series of performance-based skills assessments which were used to test participants’ knowledge and use of relevant CBT skills prior to starting therapy. Youth and parents completed measures of treatment satisfaction and app usability at post-treatment and clinicians also rated portal usability. All clinical interviews, questionnaires, and performance-based skills assessments were repeated at post-treatment and 2 month follow-up.

Measures

Diagnostic and clinical assessments.

Parents and youth were interviewed separately using the Schedule for Affective Disorders and Schizophrenia in School-Age Children—Present and Lifetime version (K-SADS-PL, Kaufman, Birmaher, Brent, & Rao, 1997), which was updated by the authors for DSM-5. The K-SADS-PL is a semi structured interview with the clinical interviewer integrating data from both informants to determine the final diagnosis. All interviews were carried out by trained MA and Ph.D. level clinicians. Inter-rater reliability for anxiety diagnoses was calculated based on 15% of interviews (kappa = .77). Anxiety severity was also rated by the interviewer on the Pediatric Anxiety Rating Scale (PARS; RUPP Study Group, 2002). A total score was computed by summing six items assessing anxiety severity, frequency, distress, avoidance, and interference during the previous week (ICC = .91). Child- and parent-report of anxious symptomatology was obtained using the Screen for Childhood Anxiety and Related Disorders (SCARED; Birmaher et al., 1997), a 41-item questionnaire that assesses symptoms of separation anxiety, social phobia, general anxiety disorders, panic, and school refusal. Internal consistency was high for parent (α = .91) and child report (α = .93).

Skill acquisition measures.

A 45-minute performance-based evaluation battery was used to assess the acquisition of four key CBT skills targeted by the app: (a) emotion identification (b) thought challenging (c) problem-solving and (d) reducing avoidance. The battery included three measures. First, the Penn Emotion Recognition Task (ER40; Gur et al., 2002; Pinkham et al., 2008) was used to assess emotion identification. The ER40 requires participants to identify the emotion displayed in each of 40 color photographs of faces expressing varying intensities of happiness, sadness, anger, or fear, or a neutral expression. The ER40 has been shown to be a reliable test of emotion identification in healthy individuals (Gur et al., 2002). Second, we used a modified version of the Ambiguous Situations Questionnaire (ASQ; Barrett, Rapee, Dadds, & Ryan, 1996) to assess threat interpretations (reverse scored; ICC = .97) and avoidance (ICC = .82) in response to potentially-threatening scenarios, as coded by trained undergraduate raters. A modified version of the ASQ, similar to the version used in the current study, has been shown to correlate with trait anxiety in youth (Chorpita, Albano, & Barlow, 1996). Third, we used the child version of the Skill Acquisition Measure (SAM-C; Lindhiem, Kolko, & Higa, 2012). The SAM-C is an online assessment system designed to measure acquisition of child CBT knowledge and skills that was developed through an iterative process that combined a review of the “common elements” literature, expert feedback, clinician feedback, and pilot testing. For the present study, we focused on child skills targeted in the present intervention, specifically problem-solving and thought challenging (i.e. identifying/reframing automatic thoughts). A parent-report version of the SAM was evaluated in a nationally representative sample of parents of youth ages 5–12 (Lindhiem et al., in press). Open-ended responses to video-recorded vignettes with actors were coded by trained undergraduate coders to assess the quality of problem solving (ICC = .82) and cognitive reframing of automatic thoughts (ICC = .87).

Usability and acceptability.

Youth rated the usability of the smart phone app and clinicians rated the usability of the clinician portal, both using the 10-item System Usability Scale (SUS; Brooke, 1995). The SUS assesses perceived ease-of-use and satisfaction with computer systems and apps, assessed on a 0–100 scale, which can be converted to a letter grade derived from analysis of 446 surveys/usability studies (Sauro, 2011). The SUS has been evaluated in a number of prior studies as a reliable measure of usability across a range of sample sizes (Tullis & Stetson, 2004; Bangor, Kortum, & Miller, 2008, 2009). Amount of time spent on the portal and number of text messages sent and received per patient per week were also calculated as indices of therapist burden. Youth and parents rated their overall satisfaction with the combined Brief CBT + SmartCAT intervention at post-treatment using the eight item Client Evaluation of Services Questionnaire (CSQ-8; Nguyen, Attkisson, & Stegner, 1983). In support of validity, the CSQ-8 correlates highly with therapy-related service utilization variables, including treatment status (i.e., continuing therapy versus termination of therapy) at one month after intake (r = .57) and number of therapy sessions in one month (r = .56) (Attkisson & Zwick, 1982).

Brief Coping Cat Treatment

Treatment was implemented by one clinical psychologist and one licensed professional counselor. Both therapists had completed previous training in the 16 session Coping Cat protocol (Kendall & Hedtke, 2006a) and had five years of experience treating youth with anxiety disorders. Therapists participated in ongoing weekly group supervision with the first author. CBT was delivered in 8 one-hour sessions using the Brief Coping Cat manual (Crawley et al., 2013). The treatment included two key components: (1) CBT skills training, including emotion identification and labeling, cognitive reframing, and problem-solving, and (2) CBT skills practice through graded exposure to feared stimuli. As in the 16-session manual, children are taught to develop and implement a coping plan using the acronym “FEAR,” which stands for Feeling frightened? Expecting bad things to happen? Attitudes and actions that can help; Results and rewards. It should be noted that breathing and muscle relaxation are not formally taught in the brief version, but were included as supplemental video and audio material in the app.

SmartCAT 2.0.

SmartCAT 2.0 was developed and pilot tested based on user-centered design principles, as described in Pramana et al. (2018). SmartCAT utilized a new mHealth platform called iMobile Health and Rehabilitation (iMHere; Parmanto et al., 2013) consisting of smartphone app, a clinician portal, and a secure two-way communication channel connecting the two using the Extensible Messaging Presence Protocol (XMPP). This technology runs continuously in the background and “wakes up” with communication from either the app or portal, thus keeping the app and portal in connection while using minimal resources. The patient can still use the app when no connection is available, and the data are automatically pushed to the server the next time the patient goes into an area with Wi-Fi or cellular service. The 2-way communication allows the therapist to use the server to customize app material based on patient progress and provide updated materials to support treatment without having to physically manipulate the phone. At session 1, participants were given an Android smartphone with the app pre-loaded to use for the duration of treatment. A research assistant provided a brief orientation to the phone and app at the end of session 1. As part of the treatment, the children were asked to complete Skillbuilder Activities using the app at home, with specific activities assigned each week depending on content covered to date in therapy (see Supplemental Table 1). The app was programmed to launch one Skillbuilder Activity each day at a time selected by the participant. Participants earned one digital point each time they completed the scheduled activity. Participants could voluntarily complete activities at additional times for 2 points per activity.

Skills Coach.

SmartCAT 2.0 includes a new Skills Coach (see Figure 1a) that leads the participant through a series of coping and problem-solving steps relevant to current or recent anxiety. This was designed to be consistent with the FEAR Plan framework in the Coping Cat manual (Kendall & Hedtke, 2006a). Children were provided with pre-populated examples of anxious thoughts, coping thoughts, and problem-solving solutions during the early sessions while they were still learning relevant skills and were then asked to type in their own self-generated responses at later sessions (once the therapist felt they were ready). The Skills Coach was programmed to initiate at scheduled times or places using alarm and geo-fencing features, but could also be activated additional times by the participant during in-vivo anxiety-provoking situations. All participants were given the option to set up location-aware reminders to complete the skills coach when entering a location where he or she tends to experience anxiety, as determined by child or parent report. This was accomplished using geofencing capabilities on the phone. Geofencing enables automatic detection of mobile objects as they enter or exit a geofence, which is a virtual boundary for a real-world area. The participants’ responses to Skills Coach questions were visually summarized as a “Fear Plan”, which was automatically transferred to a “Fear Files” library within the app so that the participant could revisit previous plans when facing a similar situation in the future. Participants could also mark which skills coach entries they wanted to discuss with the therapist. These entries were highlighted on the clinician portal to streamline the therapists’ time in reviewing entries and ensure that the most useful information from the patients’ daily experience was brought to the therapists’ attention.

Figure 1a.

Skills Coach Screens

Interactive Mini-Games.

SmartCAT 2.0 features 4 interactive mini-games to reinforce skill acquisition: (a) Thought-buster (b) Thought-swapper (c) Problem Solver; and (d) What’s the Feeling. These are brief 2–5 minute interactive activities with child friendly graphical interfaces (see Figure 1b) that support the acquisition and utilization of skills in emotion identification, thought challenging, and problem solving (see Pramana et al. 2018 for additional details). Each game contains content for multiple scenarios and levels so that it could be completed by the participant numerous times. All content was developed by the first author and 2 study clinicians based on their experience working with anxious youth. Briefly, “What’s the feeling?” was designed to reinforce skills in emotion identification by asking children to identify emotions associated with various scenarios using somatic and contextual cues. Once the correct emotion is identified, the participant can “relieve” the somatic symptoms using their finger on the touchscreen. “Thought-buster” reinforces thought challenging skills by providing practice in identifying anxious self-talk and coping talk by popping corresponding though-bubble balloons. “Thought-swapper” provides more advanced practice in thought challenging by guiding the child in reframing an anxious thought for a fictional character. For each hypothetical situation, an anxious thought appears in a thought bubble on top of the character and the player is instructed to choose a new thought (from a selection of 4) to swap into the character’s thought bubble in order to decrease or increase the character’s anxiety. An anxiety meter visually displays increases and decreases in the character’s anxiety based on the thought that the child choses to swap into the character’s mind. This helps to foster an understanding that thoughts can influence emotions and provides exposure to typical anxious thoughts and coping thoughts. “Problem-solver” provides an interactive way for the child to learn and practice the four steps of problem-solving: define the problem, come up with as many solutions as you can think of, evaluate all of the options, and pick one or two best solutions. This game imitates a text message conversation between the child and his/her virtual friend who is experiencing a hypothetical problem (e.g., performing at the talent show after school, going to a friend’s sleepover). The player is asked to help his/her virtual friend solve the problem generated each time the module is initiated by choosing from potential adaptive and maladaptive solutions.

Figure 1b.

Interactive Skill-Builder Games

Chillax.

We also developed a “Chillax” module consisting of video and audio files to support progressive muscle relaxation and deep breathing. Relaxation skills were removed from the Brief Coping Cat protocol based on limited therapist time and previous evidence that CBT remains effective without incorporating relaxation training (Hudson, 2005), however we included them in the app as a potentially useful supplement to treatment for children to learn and practice on their own.

Challenger.

Participants were asked to complete the “Challenger” activity following session 4 to facilitate home-based exposure practice and decrease avoidance tendencies. Using the clinician portal, the therapist assigned a home-based exposure from a list of common home exposures (or manually entered a customized exposure assignment). The app notified participants to complete the exposure at a time (or times) agreed upon by the child/family and therapist. The child was prompted to send the therapist a message about how the exposure went and also had the option to share a related photograph or video from the phone.

Show That I Can tasks (STIC).

We also incorporated into the app session-specific homework assignments from the Brief Coping Cat Workbook (Crawley et al., 2013), which were adapted from paper to digital format. These included specific reinforcement of skills covered that session using an open-ended format and hypothetical scenarios. One STIC per session was completed per week and therapists were able to view the responses and discuss in session.

Rewards.

Points earned by completing activities were displayed as stars in a digital reward bank on the home screen. Therapists redeemed points for small rewards chosen by the participant (i.e. pens, accessories, balls, stickers, small gift certificates) using the clinician portal whenever the participant earned the requisite number of points. Participants also earned silver and gold digital trophies when they completed all of the required activities for each session.

Messages.

To support therapist-patient interaction beyond office visits, a HIPAA-compliant messaging system was incorporated. Using this feature, a participant could compose a message on his or her phone, and the message would be sent to a web-based portal rather than the therapist’s private phone. This protects the private space of the clinician and allows the communication to be part of the treatment record. Incoming/outgoing messages from/to the therapist were encrypted and stored in the phone’s local storage using AES with a 256-bit key. During transmission, these messages were encrypted using RSA algorithm with a 2048-bit key to prevent man-in-the-middle attacks. The portal was securely protected by a corporate firewall.

Clinician Portal

The clinician portal is a secure website that can be accessed from a computer, tablet, or smartphone. Therapists used the portal to view activities completed on the app, send and receive messages, and schedule upcoming activities. Therapists were encouraged to spend the beginning of each session reviewing skills coach entries and STIC tasks on the portal with the patient, similar to the review of paper and pencil homework in a standard CBT session. At the end of each session, therapists were asked to select relevant activities and challenges for the participant to complete the following week. Therapists also used the portal periodically to manage the reward system (i.e. set # of points required for a desired prize and redeem points when prize was obtained) and to send, read and reply to messages. If desired by the family, the therapist could also activate the location-aware feature of the app by entering the address of the anxiety-provoking location after discussing it with the patient. In this case, the address was geocoded into a latitude/longitude format by the portal and then sent to the app.

Analytic Plan

The primary categorical indicator of treatment outcome was absence of a diagnosis of any qualifying anxiety disorders (GAD, SAD, or SocAD) on the K-SADS at post-treatment. The primary continuous measure of symptom severity was PARS total score, as assigned by the IE. Child and parent report of symptom severity on the SCARED-C and P were used as secondary measures. Changes in primary and secondary severity measures from pre- to post-treatment and post-treatment to two-month follow-up were analyzed using paired-samples t-tests, with the within-subjects Cohen’s d calculated as a measure of effect size (.2 = small, .5 = moderate, .8 = large). A parallel analysis was conducted to examine changes in CBT skill use across the three time points.

Results

Feasibility and Acceptability

Enrollment and attrition.

A total of 230 potential participants were phone-screened, with 71 invited to complete an intake assessment with an IE (see Supplemental Figure 1). Of these, 35 met eligibility criteria and 34 (97%) enrolled, with 30 (88%) completing treatment. Two participants withdrew after the first session, one due to an unexpected relocation and one due to time constraints. Two participants were referred out for alternative treatment of symptoms that became apparent during the course of therapy (OCD; thyroid disorder). Follow-up data were not available for the 4 participants who withdrew/were withdrawn from therapy, but all 30 participants who completed therapy participated in the follow-up assessment and 29 completed a two-month follow-up.

Treatment satisfaction and SmartCAT usability.

Satisfaction with the combined BCBT + SmartCAT treatment was very high: 97% of youth and 97% of parents reported that they were “mostly satisfied” or “very satisfied” with treatment. Mean satisfaction across all CSQ-8 items was 3.58 out of 4 (SD = .35) for parents and 3.46 (SD = .50) for youth. Participants rated the app as highly usable, with an average usability rating of 84.50 (SD=12.70), corresponding to an A+ usability grade (Sauro & Lewis, 2012). The therapist usability rating of 91.17 (SD = 10.98) for the clinician portal also corresponds to an A+ grade.

SmartCAT Usage.

Participants used the app approximately 12 times between each session (M = 12.22, SD = 9.81) suggesting a high level of engagement with notable variability between youth. They spent a mean of 5.8 minutes per day (SD = 8.44) using the app. Data also indicate that the therapist burden was not excessive. Therapists spent an average of 13.72 minutes per patient/per session using the SmartCAT portal. Therapists reported that the majority of this time was spent reviewing skills coach entries together with the child at the beginning of session and assigning tasks for the following week at the end of session. Therapists responded to an average of 1.21 messages from patients per session (SD =1.11; maximum of 6).

Treatment Outcome

Post-treatment diagnosis.

At post-treatment, 20 out of 30 participants (66.67%) no longer met DSM 5 criteria based on the K-SADS interview for any of the qualifying anxiety diagnosis (GAD, SocAD, or SAD). This positive response rate increased to 25 out of 29 participants (86.2%) free of all qualifying anxiety diagnoses at two-month follow-up.

Post-treatment anxiety severity.

Results of paired-samples t-tests examining symptom change from pre-to post-treatment and post-treatment to follow-up are reported in Table 2. IE-rated symptom severity on the PARS improved, with large reductions in severity from pre-to post-treatment (Cohen’s d=1.05), and moderate further reductions from post-treatment to two-month follow-up (Cohen’s d=.55). Secondary analyses revealed that there were also large reductions in child- and parent-reported symptom severity from pre-to post-treatment (Cohen’s d_child = 1.09; Cohen’s d_parent =1.15), with moderate further reductions from post-treatment to two-month follow-up (Cohen’s d_child =.56; Cohen’s d_parent = .47).

Table 2.

Change in Anxiety and CBT Skill Acquisition from Pre- to Post-Treatment and Post-Treatment to 2 Month Follow-up

	Pre-treatment	Post-treatment	2 month follow-up
Diagnosis
% free of GAD, SAD, & SocAD	0.00%	66.67%	86.20%
				∆ Pre-Post			∆ Post-2 mo follow-up
	M (SD)	M (SD)	M (SD)	t-value	p-value	d	t-value	p-value	d
Severity
PARS 6-item	13.97 (3.75)	9.23 (4.57)	6.69 (4.61)	5.74	<.001	1.05	2.98	.006	.55
SCARED-Child	36.24 (9.63)	19.17 (13.47)	15.03 (10.89)	5.96	<.001	1.09	3.00	.006	.56
SCARED-Parent	36.41 (8.41)	21.17 (11.84)	17.41 (11.07)	6.29	<.001	1.15	2.55	.017	.47
Skill Acquisition
Emotion Identification
ER40 (Overall)	.76 (.07)	.84 (.06)	.82 (.07)	−5.40	<.001	.99	1.24	.227	.23
Thought Challenging
ASQ-C Threat	4.91 (2.31)	2.41 (1.27)	2.08 (1.65)	5.40	<.001	1.00	.69	.495	.14
ASQ-PC Threat	6.45 (2.00)	5.59 (1.96)	4.42 (2.48)	3.69	.001	.68	2.38	.027	.50
SAM-C Automatic Thoughts	.38 (.65)	1.07 (1.09)	1.61 (1.16)	−2.59	.015	.49	−1.63	.118	.34
Problem Solving
SAM-C Problem Solving	1.97 (.97)	1.96 (.96)	1.96 (.83)	.30	.764	.06	−.17	.870	.03
Approach/Avoidance
ASQ-C Avoidance	1.29 (1.19)	1.83 (2.25)	2.31 (2.81)	−2.05	.050	.38	−1.42	.167	.28
ASQ-PC Avoidance	5.70 (2.47)	4.32 (2.26)	3.74 (1.91)	2.24	.033	.42	.49	.628	.10

Note. Reported p-values are uncorrected; d = Cohen’s d; GAD= generalized anxiety disorder; SAD = social anxiety disorder; SCARED = Screen for Childhood Anxiety and Related Disorders; ER40 = Penn Emotion Recognition Task, SAM = Skill Acquisition Measure; ASQ = Ambiguous Situations Questionnaire; C=child report, PC = parent report on child.

Skill Acquisition

Paired sample t-tests examining changes in performance-based measures of skill acquisition are also reported in Table 2. Youth showed large improvements in accurately identifying emotional facial expressions on the ER-40 from pre- to post-treatment (Cohen’s d = .99). Youth also showed moderate-to-large changes in thought challenging skills from pre-to post-treatment, as indicated by reductions in threatening interpretations of ambiguous scenarios on the ASQ by both child-report (Cohen’s d = 1.00) and parent-report (Cohen’s d = .68) and decreased negative automatic thoughts generated in response to a hypothetical anxiety-provoking situation on the SAM (Cohen’s d = .49). Youth also showed small-to-moderate reductions in avoidance as assessed by endorsement of avoidance responses to ambiguous scenarios on the ASQ by both child-report (Cohen’s d = .38) and parent-report (Cohen’s d = .42). There were no improvements in problem solving, as assessed on the SAM (Cohen’s d = .06). As shown in Table 2, there were no further improvements in skill acquisition from post-treatment to two-month follow-up with the exception of parent report of anxious self-talk on the ASQ, which revealed continued decreases in parents’ expected threat interpretations for their child (Cohen’s d = .50).

Discussion

Although the potential of technology to improve behavioral health interventions for youth has been widely touted, most existing child anxiety apps were not developed based on empirical research, and the evidence supporting their efficacy is limited (Bry et al., 2018; Radovic et al., 2016). This study reports on the feasibility, acceptability, and preliminary efficacy of a smart-phone app and integrated clinician portal that was developed to target the full spectrum of evidence-based treatment components that are considered integral to the treatment of anxiety in children and adolescents. The program is novel in its comprehensive evidence-base, integration with clinical care, and incorporation of gamification and advanced cell phone capabilities (i.e. GPS, camera). Findings demonstrate that combining the SmartCAT app and portal with a brief 8-session CBT protocol for anxious youth was feasible, acceptable to youth, families, and clinicians, and was associated with significant improvements in CBT skills targeted by the app as well as high rates of recovery among treated youth.

In terms of feasibility, families and youth were open to the idea of incorporating technology into treatment (97% of eligible families chose to enroll). Although many of the children had their own phones and had sufficient experience with smartphone technology, for some children this was their first experience independently carrying a cell phone. A few parents expressed concerns that their children might come across inappropriate material on the Internet or use the phones inappropriately. We were able to address these concerns by installing Mobile Fence, a parental control app that allowed us to block usage of unsuitable apps and websites, on all phones. We found that the children--even those who did not own their own phones--easily learned how to use the phone and the app, as reflected in the A+ usability rating. Most youth who enrolled completed the entire 8 sessions of treatment (88%), suggesting that using the app did not comprise a burden. Of the four youth who did not complete treatment, 2 were referred out for medical reasons and 2 withdrew for logistical reasons. At the end of treatment, 97% of both youth and parents reported that they were satisfied with the treatment that they received. Although we did not directly assess which components of the treatment contributed to satisfaction ratings, it is likely that components of the app and portal, such as engaging games and ongoing monitoring and communication from the therapist, contributed to families’ high satisfaction.

Therapists also found the integrated program to be acceptable and feasible and rated the clinician portal as easy to use (A+). Time burden associated with the portal was acceptable, with therapists spending the majority of time on the portal during session, corresponding to a similar amount of time spent assigning and reviewing homework in standard therapy. The number of messages sent by participants ranged from 0 to 6 (average 1.2) and could therefore be easily managed by therapists. These data support feasibility from the perspective of the clinician, but it will be important to test this in settings where therapists have larger patient loads and busier schedules. Relatedly, it is also worth exploring whether clinicians can receive reimbursement from insurance companies for time spent in electronic communication with patients.

The high degree of engagement with the app exhibited by participants is noteworthy. In contrast to one homework assignment per session in typical CBT, SmartCAT prompted youth to complete an activity on the app once per day (i.e. ~7 times between sessions). We found that youth actually exceeded this expectation, using the app an average of 12 times between sessions. Not only did the participants complete the app when prompted, but they regularly initiated the app on their own. Pramana et al. (2018) demonstrated that the addition of gamification into SmartCAT appears to account for increased engagement in SmartCAT 2.0, tested here, compared to a preliminary version of the app that did not include games. We also suspect that the rewarding of double points for self-initiating the app played a role in maintaining engagement. Points and rewards are used in CBT, but the integration of a digital point system and reward bank within the app allows participants to accrue points regularly throughout the week in a way that was not previously possible without therapist monitoring.

In addition to high levels of engagement with the app, we found preliminary evidence of efficacy to support further testing in a randomized controlled trial with an active comparison condition. A full two-thirds of youth (66.67%) no longer met diagnostic criteria for any of the anxiety disorders treated in the study (generalized, separation, and/or social anxiety) at the conclusion of the 8-week intervention. This exceeded the 42% diagnostic recovery rate reported in the Crawley et al. (2013) trial of Brief Coping Cat administered without SmartCAT. This discrepancy suggests that adding daily opportunities to learn and practice CBT skills via the app may be an effective approach to boosting the efficacy of brief treatments. Interestingly, the present treatment response rates were commensurate with those typically reported in trials testing a full course of CBT for child anxiety. For example, a Cochrane Review reported an average response rate for remission of anxiety diagnosis of 56% (A. C. James et al., 2005). Given the present remission rate of 67%, it appears that technological adjuncts to treatment may be beneficial even in the context of a full course of CBT. These conclusions, though not the result of a randomized clinical trial, are supported by of the data on symptom change. A meta-analysis of 48 CBT trials for child anxiety (Reynolds et al., 2012) reported that effect sizes for symptom change in brief treatments (≤ 8 sessions) were typically small (d=.35), whereas effect sizes for lengthier (13–16 session) treatments were large (d=.75). In the present study, Cohen’s d effect sizes for change in symptom severity from pre-to post-treatment were large, exceeding 1.05 across IE, parent, and child-report.

One mechanism through which the app may enhance the efficacy of treatment is via increased opportunities to learn and practice skills in daily life. As an exploratory goal, we used a performance-based skills assessment battery to test whether participants improved in their ability to understand and implement skills that were targeted by the app. We found that youth improved in 3 of the 4 skills targeted by the app from pre-to post-treatment. First, they showed an increased ability to identify emotions accurately. Second, they showed improvements in thought challenging skills, as indicated by reductions in threatening interpretations and automatic negative thoughts generated by hypothetical anxiety-provoking scenarios. Third, they also showed a shift in approach/avoidance tendencies, generating fewer avoidant responses to the hypothetical anxiety-provoking scenarios reductions. Although emotion identification was only assessed via child performance, reductions in anxious self-talk and avoidance were corroborated by parent report. Surprisingly, no changes were found in problem-solving from pre-to post-treatment. Participants already showed moderate skill in problem-solving at pre-treatment (mean of 1.97 on a 0 to 3 scale), therefore there may have been less room to grow in this area. Nevertheless, this finding could suggest that additional focus on problem solving in Brief Coping Cat and/or SmartCAT might be warranted. Although skills generally improved with treatment, we did not have the statistical power to examine whether skill acquisition mediated the link between app use and treatment response. Testing this mechanistic model in a larger, more fully powered randomized trial will be an important next step in this line of research.

Interestingly, we found that symptoms continued to improve over the two months following termination of treatment, with moderate effect sizes for post- to two-month follow-up symptom severity reductions as rated by IE, parent, and child. Diagnostic remission rates also continued to increase. This increase took place in the absence of continued improvements in skill acquisition from post-treatment to two-month follow-up and suggests that although skill change occurred primarily during acute treatment, participants were continuing to accrue benefits from their improved skills after treatment ended. This finding is consistent with several other studies that reported continued reductions in anxiety over time following a full course of treatment (i.e. Silk et al., 2016), but stands in contrast with results from the brief Coping Cat trial (Crawley et al., 2013), in which diagnostic remission declined following brief CBT in the absence of SmartCAT. This finding is consistent with the notion that use of the app may help with retention of skills accrued during brief treatment, perhaps because skills became more ingrained and automatic with more frequent repetition and practice. In future research, it would be interesting to test whether skills and/or symptoms would continue to improve further following treatment if participants continued to use the app after treatment ended.

Study limitations merit mentioning. This was an open trial with no comparison condition for SmartCAT. Although we drew comparisons with published outcomes that used the same treatment, there could be other factors contributing to differences in outcomes across trials. We also cannot rule out other sources of internal invalidity, such as the passage of time or the effects of repeating assessments. Demonstrating specificity of treatment outcomes to app use in a randomized controlled trial comparing brief CBT alone to brief CBT plus SmartCAT will be a critical next step in the treatment evaluation process. Furthermore, although we included participants with common comorbid mental health disorders such as other anxiety disorders, ADHD inattentive only type, and oppositional defiant disorder, we also excluded other diagnoses that could interfere with treatment such as major depressive disorder, obsessive-compulsive disorder, post-traumatic stress disorder, and conduct disorder. This may limit the generalizability of our findings. Additionally, to our knowledge, no widely-used, validated performance-based measures of CBT skill use in anxious youth exist, thus to address our exploratory hypothesis we adapted existing measures to assess change in skill use for the current study (i.e, ER40, ASQ, SAM-C). More work is needed to demonstrate that these measures are sensitive to change with intervention. An additional issue is that the measure of treatment satisfaction did not separate out satisfaction with the CBT protocol and the app. Thus, although children and families were highly satisfied, the relative contributions of the app and in person therapy sessions to their overall satisfaction remains unclear and needs to be addressed in future research. The sample size was also insufficient to examine mediators of treatment response as well as potential moderators. Future research can investigate whether use of technology is more fruitful for participants with different demographic and/or clinical profiles. Recent data indicate that clinicians in routine practice can implement Coping Cat and achieve outcomes consistent with those reported in the literature (Villabø, Narayanan, Compton, Kendall, & Neumer, 2018)—it would be interesting to see if the SmartCAT app enhances outcomes in community settings. It will also be important to demonstrate that SmartCAT can be integrated into usual care in busy community settings such as schools or pediatrics practices. The app was only available on the Android platform and with more recent operating systems, therefore we chose to standardize phone use by providing all youth with a study-purchased mobile phone. It is possible that results may have differed if participants had been permitted to use their own phones. Finally, the present study design did not allow us to compare the efficacy of different features of the app. In the future it would be useful to investigate which components of the app and clinical portal are most strongly associated with patient engagement as well as skill change and treatment response.

Despite these limitations, the SmartCAT app integrated seamlessly with clinical care and provided therapists with access to real-time and contextualized data about how patients experience and cope with anxiety. Integrating technology with brief anxiety treatments may improve dissemination of evidence-based approaches in community settings, such as schools and primary care practices Integrating SmartCAT with computer-based CBT interventions, such as Camp-Cope-a-Lot (Khanna & Kendall, 2010) may be an additional path to improving dissemination.

• mHealth SmartCAT app was used in conjunction with brief CBT for child anxiety

• App and portal demonstrated strong feasibility, usability, and acceptability

• Brief CBT + app associated with high rates of recovery and improved CBT skills