Pilot Evaluation of a Tablet-Based Application to Improve Quality of Care in Child Mental Health Treatment

Abstract

Mental health systems need scalable solutions that can reduce the efficacy–effectiveness gap and improve mental health outcomes in community mental health service settings. Two major challenges to delivery of high-quality care are providers’ fidelity to evidence-based treatment models and children’s and caregivers’ engagement in the treatment process. We developed a novel, tablet-based application designed to enhance via technology the quality of delivery of trauma-focused cognitive-behavioral therapy (TF-CBT). We piloted its use in four community mental health service organizations using a blocked randomized controlled trial to examine the feasibility of implementing tablet-facilitated TF-CBT versus standard TF-CBT with 13 providers and 27 families. Provider fidelity and child engagement in treatment were observationally measured via session audio recording. Parent and child perceptions of the tablet application were assessed using structured interviews and mixed-method analyses. Providers actively and appropriately used tablet TF-CBT to facilitate treatment activities. Providers and families expressed high satisfaction with its use, demonstrating acceptability of this approach. Youth and caregivers in both conditions reported high alliance with their providers. Overall, we found that tablet-facilitated treatment is accepted by providers and families and may be integrated into mental health treatment with minimal training. Further study is needed to examine the extent to which technology-based applications may enhance the reach, quality, and clinical outcomes of mental health treatment delivered to children and families.

EFFICACIOUS PSYCHOSOCIAL TREATMENTS for children and adolescents are available for a wide variety of mental health disorders (Chorpita et al., 2011). However, child outcomes are generally inferior in real-world practice settings versus controlled research settings (Weiss, Guidi, & Fava, 2009). Treatment fidelity and child engagement are associated with mental health outcomes (e.g., symptom severity, attrition) and serve as key correlates of successful translation of treatments from randomized controlled trial (RCTs) to “real-world” settings and prime targets for intervention (Bond, Becker, & Drake, 2011; Chu & Kendall, 2004; Gopalan et al., 2010; McCabe, 2004; Novins, Green, Legha, & Aarons, 2013; Schoen-wald, Sheidow, & Letourneau, 2004). Treatment fidelity, defined as the degree to which providers adhere to treatment protocols and deliver them competently, is highly variable in mental health service settings, even among well-trained providers (Drake, Torrey, & McHugo, 2003; McHugo et al., 2007; Saunders, 2009). Engagement, defined as children’s general level of emotional and behavioral involvement in the treatment process, is also a key quality-of-care indicator (Drake et al., 2003; Garland et al., 2010; Kazak et al., 2010). Scalable interventions that improve fidelity and engagement may help to close the gap between what is known and what is practiced in community service settings. The aim of this study was to develop a scalable, sustainable technology-based approach to improve quality of care in child mental health treatment.

The Potential Role of Mobile Applications in Community Practice Settings

Mobile applications may have the potential to support effective transport of evidence-based treatments (EBTs) and transform the way EBTs are delivered in community mental health settings. They may allow the opportunity for personalized learning in which multiple learning methods (e.g., provider guided, child guided) and procedures (e.g., in-session content, at-home practice) can be applied to promote skill acquisition (Cucciare, Weingardt, & Villafranca, 2008). Moreover, they may assist in delivery of treatment components that are difficult for providers to navigate. Touch-screen learning, interactive games, and video demonstrations are examples of activities that may enhance children’s engagement in the treatment process. Moreover, technology-based decision support tools have been developed in the broader health care field, and initial data suggest that this approach improves clinical decision making and adherence to best-practice and treatment protocols (Kubben et al., 2011; Roy et al., 2009). Whereas a large body of research has examined self-help resources and technological adjuncts to mental health care, far less is known about the use and possible benefits of technology-based applications that are designed to improve patient–provider interactions and quality of care. Research is needed to test the feasibility and acceptability of these approaches, and to ensure that they do not have adverse effects on therapeutic alliance (Hanson et al., 2014).

development of tablet trauma-focused cognitive-behavioral therapy (tf-cbt)

The purpose of this project was to develop technology-based tools to aid in the delivery of key treatment components and techniques of TF-CBT (Cohen, Mannarino, & Deblinger, 2017) and test their feasibility in community mental health service settings by frontline community providers. TF-CBT is a short-term treatment model featuring eight treatment components that comprise the acronym PPRACTICE (Psychoeducation, Parenting skills; Relaxation, Affective expression, and modulation; Cognitive coping, development, and processing of a Trauma narrative; In vivo exposure; Conjoint sessions between the child and a supportive caregiver; and strategies to Enhance future safety and development). We selected TF-CBT as a treatment model because it has been evaluated in numerous RCTs, is a well-established and widely disseminated mental health treatment, and addresses multiple symptom domains (e.g., posttraumatic stress disorder, depression, disruptive behavior), thereby potentially enhancing applicability of our findings to a broad range of child mental health treatments (Cary & McMillen, 2012; Cohen et al., 2017; de Arellano et al., 2014; Deblinger, Mannarino, Cohen, Runyon, & Steer, 2011; Dorsey et al., 2017; Wethington et al., 2008). We hypothesized that use of these tools would be acceptable to families and providers and feasible to implement with minimal training in community practice settings. We measured two key quality-of-care indicators in this context—provider fidelity and child engagement—as well as patient satisfaction and therapeutic alliance.

Development of tablet TF-CBT was informed by qualitative interviews with nationally certified trainers (Hanson et al., 2014) and local mental health providers and families (Ruggiero et al., 2015). It was designed to be used by providers in session with children and caregivers throughout the course of treatment to support treatment delivery and patient engagement. The development process for tablet TF-CBT is described in detail elsewhere (Ruggiero et al., 2015). Briefly, tablet TF-CBT consists of 11 activities or “chapters” that are designed to help providers navigate and improve engagement of children in important and potentially challenging patient–provider interactions. Many activities are broadly relevant to a wide range of child mental health treatments, including breathing retraining, muscle relaxation, affective regulation, and cognitive coping; other activities are specific to the TF-CBT model, including psychoeducation relating to traumatic events, safety education, and gradual exposure. These activities were developed and informed via an iterative process guided by experts, providers, and families (Ruggiero et al., 2015). All major components of TF-CBT were addressed in the tablet TF-CBT platform with the exception of the trauma narrative, which we did not develop due to budgetary restrictions.

Method

procedure

TF-CBT trained providers with active child case-loads from two community-based mental health clinics and two children’s advocacy centers that were recruited. Providers were randomized within each site to either the tablet-facilitated TF-CBT (standard TF-CBT + tablet) or standard TF-CBT conditions. Alternatives to this design were considered. For example, randomization could have occurred at the patient level to address the potential influence of provider factors on outcomes, but a weakness is that use of the tablet could affect performance with subsequent standard-care cases. Randomization at the site level is ideally suited for future implementation studies. Therefore, it was concluded that randomization at the level of provider was a preferable design strategy. A provider was not randomized to condition until its first eligible patient referral was enrolled in the study. All providers were oriented to study objectives and procedures. Providers randomized to the experimental condition received training (~ 45 minutes) in the use of tablet TF-CBT and were encouraged to use the tablet activities flexibly—that is, they were advised to only use chapters that they determined would be appropriate and helpful to each family based on factors such as provider preferences and patient traumatic event histories, styles of learning, and symptoms. This approach was consistent with stakeholders’ focus group recommendations. Providers referred patients to the study following standard clinic intake assessments. Eligible children were ages 5–16 years with clinically elevated symptoms related to one or more traumatic events (e.g., sexual abuse, physical assault, witnessed violence, disaster, serious accident), including at least one symptom within each DSM-IV PTSD symptom cluster (reexperiencing, avoidance, hyperarousal) based on a diagnostic interview. No families referred to the study were excluded. A trained evaluator masked to study condition contacted referred patients, obtained informed consent, and conducted the baseline and posttreatment assessments over telephone or in person according to participant preference. Assessments included a diagnostic interview and self-report measures. Treatment sessions were audio recorded and coded for provider fidelity and child engagement by independent coders masked to study hypotheses. Providers also completed a poststudy qualitative interview. All study procedures were approved by the Institutional Review Board of the Medical University of South Carolina.

participants

Providers

We consented 18 providers with active child caseloads from four partnering sites—two child advocacy clinics and two community mental health centers—all of whom had been trained in TF-CBT via standard statewide community-based learning collaboratives within the past decade. Thirteen (65%) actively participated by referring patients to the study: 7 were randomized to tablet TF-CBT, 6 to standard TF-CBT. Two of the five providers who did not refer families to the study were lost after the consent process despite multiple attempts to contact by study staff. The remaining three providers who did not refer families indicated that they did not have eligible new patients during the brief (i.e., 4-month) study recruitment period.

Patients

Thirty-two youth and caregivers were referred by providers to the study over a 4-month recruitment period and assessed for eligibility (see Figure 1); 27 (84.3%) were successfully contacted and consented. Tablet TF-CBT providers referred families more actively (n = 18 families [11 postrandomization]; M = 2.6, Mdn = 3.0) than standard TF-CBT providers (n = 9 families [3 postrandomization]; M = 1.5, Mdn = 1.5) despite consistent communication and engagement procedures across conditions. Average child age was 11.5 years (range = 5–16); the median age range for caregivers was 30–44 years. There were more girls (n = 17) than boys (n = 10) for youth and caregivers (women n = 23, men n = 4). Eighteen youth reported White race (67%), 30% African American, and 4% Native American. Similarly, 18 caregivers reported White race (67%), 27% African American, 3% Native American, and 3% other.

FIGURE 1.

CONSORT diagram.

observational measures

Fidelity

Independent, trained coders rated fidelity to the TF-CBT model as well as child engagement via coding of audio-recorded sessions. Although it was not possible to mask raters to the use or nonuse of the tablet, the study team blinded trained coders to study purpose and hypotheses. Moreover, we aimed to minimize recognition of the tablet as the experimental manipulation by contextualizing training of the coders in a way that reinforces recognition of the wide range of tools (e.g., online tools, apps), approaches, and styles that providers may introduce in their sessions. Raters used the TF-CBT version of the Therapy Process Observational Coding System for Child Psychotherapy (TPOCS-S; Deblinger, Dorsey, Cooper, McLeod, & Garland, 2013). For the TF-CBT TPOCS-S, coders recorded clinicians’ use of 25 different item codes that correspond to the elements of TF-CBT, other content items, and therapeutic techniques (e.g., establishing an agenda, Socratic questioning, role play) during each session. After a session was coded, coders provided an extensiveness rating (i.e., 6-point rating to reflect both frequency and thoroughness) for each item coded (by session type) and these were averaged in the present study. Frequency was defined as the number of instances a provider used a specific therapeutic technique and is associated with the amount of time the provider invested in a particular skill or activity during the session. Thoroughness relates to how intensively (e.g., concentration of effort, detail in describing rationale) a provider pursues a specific therapeutic intervention or technique. Each trained coder completed a standard 65- to 70-hour training process. Mean intraclass correlation coefficient (ICC) across coders was .97.

Child Engagement

The Child Involvement Ratings Scale (CIRS; Chu & Kendall, 2004) was used to code child engagement. Ten child involvement items—six positive and four negative—were rated for each session on a 6-point scale (i.e., not at all to a great deal present). The positive-involvement items emphasized the extent to which children initiated discussions, demonstrated enthusiasm, self-disclosed, and demonstrated understanding of session content. Items for negative involvement indicated withdrawal or avoidance in treatment. The four negative involvement items are reverse scored, and their sum is added to the sum of positive-involvement items to compute a CIRS involvement score. Coders provided separate ratings for the first and second halves of each session, permitting observation of shifts in engagement within and across sessions. Initial reliability training consisted of a 2-day in-person didactic that reviewed the CIRS manual and illustrated individual items and scoring using gold-standard recordings of TF-CBT sessions that were obtained for training purposes. Coders were considered reliable when they achieved an ICC ≥ .60 on all 10 CIRS items compared to gold-standard ratings. Study raters were successfully trained to reliability (ICC M = 0.81, range: 0.60–1.00). Biweekly reliability checks were used to prevent coder drift.

clinical outcome measures

Caregivers and children completed a clinical interview and self-report measures. We completed posttreatment assessments with 21 of the 27 participating families (77.8%).

Clinical Interview

The PTSD module of the Kiddie Schedule for Affective Disorders and Schizophrenia for School Age Children—Present and Lifetime version (K-SADS-PL PTSD module; Kaufman et al., 1997) is a well-established and widely used semistructured interview of DSM-IV PTSD symptoms and diagnosis, as well as functional impairment in school, social, and family life. Caregivers and children completed this interview.

Child Self-Report Measures

The UCLA PTSD Reaction Index for DSM-IV, Child Version (UCLA-PTSD-RI-C; Pynoos, Rodriguez, Steinberg, Stuber, & Frederick, 1998a; Rodriguez, Steinberg, & Pynoos, 2001) assesses exposure to traumatic events and PTSD symptoms. The extent to which symptoms have been experienced during the prior month are rated on a 5-point Likert scale from 0 (none) to 4 (most of the time) with total scores calculated as the sum of all items. Total scores range from 0 to 68 and a score ≥ 38 indicates significant PTSD symptom severity (Rodriguez et al., 2001).

The Center for Epidemiological Studies Depression Scale for Children (CES-DC) assesses the severity of depressive symptoms in children. The CES-DC is a 20-item measure for which items are rated on a 4-point Likert scale from 0 (not at all) to 3 (a lot of the time) and summed total scores range from 0 to 60. Scores over 15 are indicative of significant depressive symptoms (Faulstich, Carey, Ruggiero, Enyart, & Gresham, 1986; Weissman, Orvaschel, & Padian, 1980).

The Working Alliance Inventory—Short Form (WAI; Hatcher & Gillaspy, 2006; Horvath & Greenberg, 1989) is a 12-item measure of patient–therapist alliance with subscales that assess the extent to which patients and their therapists agree about the goals of therapy (goal), agreement about the tasks of therapy (task), and the bond between the patient and therapist (bond) using a 7-point Likert scale from 1 (never) to 7 (always). Scores for each subscale are calculated as the mean rating for the four items that make up each subscale, and scores ≥ 5 indicate agreement between the patient and provider from often to always.

The Child/Adolescent Satisfaction Questionnaire (CASQ; Lang & Franks, 2007) uses 12 items to assess children’s satisfaction with mental health treatment using a 5-point Likert scale to rate the extent to which patients agree with statements about their satisfaction with treatment from 1 (very much false) to 5 (very much true). Total scores on the CASQ were averaged for the present study.

caregiver report measures

The UCLA PTSD Reaction Index for DSM-IV, Parent Version (UCLA-PTSD-RI-P; Pynoos, Rodriguez, Steinberg, Stuber, & Frederick, 1998b) was used to assess caregiver report of the severity of the child’s PTSD symptoms.

The Child Behavior Checklist (CBCL; Achenbach, 1991) is a widely used measure of emotional (i.e., internalizing subscale) and behavioral (i.e., externalizing subscale) functioning in children. Raw scores were used for the present study, with higher scores indicating higher levels of emotional and behavioral difficulty.

The Caregiver Satisfaction Questionnaire (CSQ; Lang & Franks, 2007) is a 12-item measure to assess caregiver satisfaction with mental health treatment that uses the same 5-point Likert scale and scoring procedures as the CASQ.

The Working Alliance Inventory—Short Form (WAI-S; Hatcher & Gillaspy, 2006; Horvath & Greenberg, 1989) is a 12-item measure of patient–therapist alliance with subscales that assess the extent to which caregivers and their child’s therapists agree about the goals of therapy (goal), agreement about the tasks of therapy (task), and the bond between the patient and therapist (bond) using a 7-point Likert scale from 1 (never) to 7 (always). Scores for each subscale are calculated as the mean rating for the four items that make up each subscale, and scores ≥ 5 indicate agreement between the caregiver and provider from often to always.

data analysis

Because this was a feasibility study, it was underpowered for comparisons across provider fidelity, child engagement, and clinical outcomes, and thus, only descriptive statistics are reported.

qualitative interviews

Interview Procedures

Individual thematic interviews were administered in person to families and providers in the experimental condition to obtain direct input on the content, functionality, and perceived value of the tablet toolkit. Trained staff conducted a 30- to 45-minute semistructured interview and interviews were audio recorded. For the provider interviews, the interviewer then asked three to five open-ended questions with follow-up probes. Relevant follow-up questions were asked as needed to clarify participant responses. For the provider interviews, probes queried for (a) provider use of the different toolkit components during sessions; (b) reactions to look and feel, navigation, and interactive components; and (c) impressions about the relevance and usefulness of the toolkit. Probes in the family interviews queried for (a) youth and caregiver use of the various toolkit activities during session, and (b) impressions about the relevance and usefulness of the toolkit.

Data Analytic Plan

The qualitative approach chosen for this analysis is derived from constructivist grounded theory (Charmaz, 2006) for coding the data. This is an approach that acknowledges the researcher’s prior knowledge and influence in the process, and supports and provides guidelines for building a conceptual framework to understand the interrelations (e.g., the what and how) between constructs. First, a content analysis of the responses was conducted through multiple close readings of the transcriptions by two independent coders. Each coder generated an independent list of thematic categories and subcategories based on their review of the data. These themes were then further developed and ordered by the first author and reviewed. The authors then met in a consensus conference to discuss the categories, resolve questions, and refine the thematic categories prior to developing the final thematic categories. Themes that were reported by at least four participants were considered salient and were interpreted.

Results

study acceptability

Use of Tablet TF-CBT

The seven providers randomized to the experimental condition used all 11 chapters of tablet TF-CBT. Chapters addressing psychoeducation, relaxation, affective regulation, and cognitive coping were used with between 77 and 89% of families. Chapters featuring components of TF-CBT that are less commonly used by providers in practice, such as behavior management, in vivo exposure, and safety education, were used with between 28 and 50% of families. An average of 5.9 (Mdn = 5.0, SD = 3.0) different tablet TF-CBT chapters were used per family. The most frequently used chapter was “What Do You Know?”, a chapter designed to facilitate psychoeducation, which was launched a total of 79 times with the 18 families served by providers in the tablet TF-CBT condition (M = 4.4 times per family).

Working Alliance and Treatment Satisfaction

Youth and caregivers generally reported high alliance with their providers. Mean scores indicated that both groups were mostly to very satisfied with treatment (see Table 1).

Table 1.

Child and Caregiver Therapeutic Alliance and Satisfaction Posttreatment

	Tablet TF-CBT	TF-CBT
	M (SD)	M (SD)	t	df	sig.	d
WAI-Child
Goal	5.3 (1.2)	5.2 (1.0)	−.046	15	.964	.02
Task	5.2 (1.0)	5.7 (1.0)	.897	15	.384	.47
Bond	5.6 (1.2)	5.8 (1.3)	.211	15	.836	.11
WAI-Caregiver
Goal	6.2 (0.8)	6.4 (0.7)	.628	12	.542	.37
Task	6.1 (0.9)	6.1 (1.2)	.056	12	.956	.03
Bond	6.5 (0.6)	6.3 (1.4)	−.415	12	.686	.24
CASQ	4.0 (0.7)	4.3 (0.4)	1.063	18	.302	.53
CSQ	4.1 (1.0)	4.6 (0.4)	1.187	19	.250	.58

Note. TF-CBT = trauma-focused cognitive-behavioral therapy; WAI = Working Alliance Inventory; CASQ = Child/Adolescent Satisfaction Questionnaire; CSQ = Caregiver Satisfaction Questionnaire.

provider fidelity, child engagement, and clinical outcomes

As illustrated by Figure 2, fidelity estimates across the primary TF-CBT components were variable across conditions. Figure 3 illustrates child engagement estimates by condition across the PRAC (psychoeducation, relaxation, affective regulation, cognitive coping) and ICE (in vivo exposure, enhancing safety) components of TF-CBT.

FIGURE 2.

Fidelity estimates across primary TF-CBT components.Note. TF-CBT = trauma-focused cognitive-behavioral therapy; estimates based on data from the TF-CBT version of the Therapy Process Observational Coding System for Child Psychotherapy; Tablet-Facilitated TF-CBT (n = 18); control (n = 9).

FIGURE 3.

Child engagement estimates by condition.Note. PRAC = psychoeducation, relaxation, affective regulation, cognitive coping; ICE = in vivo exposure, enhancing safety; TF-CBT = trauma-focused cognitive-behavioral therapy; estimates based on data from the Child Involvement Rating Scale; tablet-facilitated TF-CBT (n = 18); control (n = 9).

Most children met diagnostic criteria for PTSD at baseline. The percentage of children who met criteria for PTSD, as measured by the PTSD module of the K-SADS-PL, decreased by 32 percentage points (i.e., 77.8 to 46.2%) in the tablet TF-CBT condition versus 21 percentage points (i.e., 77.8 to 57.1%) in the standard TF-CBT condition. Means and standard deviations for clinical outcome measures at baseline and posttreatment are presented in Table 2. Data for the UCLA-PTSD-RI were not available for three children (i.e., one child from the tablet TF-CBT condition and two children from the control condition), and data for the CES-DC and CBCL were not available for two children (i.e., one child from each group) because the children fell outside of the recommended age ranges for the measures. Reductions in symptom severity from baseline to posttreatment were observed in both conditions.

Table 2.

Diagnosis and Symptom Severity at Baseline and Posttreatment

	Baseline		Posttreatment
	Tablet TF-CBT	TF-CBT	Tablet TF-CBT	TF-CBT
	n (%)	n (%)	n (%)	n (%)	X2	df	p	Φ
Met K-SADS-PL PTSD criteria	14 (77.8)	6 (75.0)	6 (46.2)	4 (57.1)	.220	1	.649	−.105
	M (SD)	M (SD)	M (SD)	M (SD)	F	df	p	η2
UCLA-PTSD-RI
Child	33.1 (14.3)	35.0 (16.6)	20.8 (14.7)	29.8 (18.2)	1.01	15	.332	.063
Parent	38.1 (18.0)	35.3 (19.7)	28.5 (14.9)	29.8 (18.1)	0.03	16	.857	.002
CESD-C	20.3 (12.7)	26.1 (8.3)	19.5 (11.6)	19.6 (11.8)	0.05	15	.823	.003
CBCL
Internalizing	18.5 (10.6)	16.9 (10.8)	10.4 (9.0)	13.8 (11.8)	1.82	17	.195	.097
Externalizing	22.6 (15.4)	17.4 (11.3)	15.3 (12.8)	16.4 (12.0)	2.72	17	.117	.138

Note. TF-CBT = trauma-focused cognitive-behavioral therapy; K-SADS-PL = Kiddie Schedule for Affective Disorders and Schizophrenia for School Age Children–Present and Lifetime version–PTSD module; UCLA PTSD RI = UCLA PTSD Reaction Index for DSM-IV; CESD-C = Center for Epidemiological Studies Depression Scale for Children; CBCL = Child Behavior Checklist; values are presented as raw scores.

posttreatment qualitative interviews

Provider Feedback

Semistructured thematic interviews were conducted with all seven providers assigned to the experimental condition to assess overall reactions to the toolkit and gather suggestions for improvement. All providers expressed using all of the toolkit chapters at least once. The final thematic categories (see Table 3) fell across two domains: (a) appeal and usability (i.e., usefulness of tools during session), and (b) relevance and helpfulness (i.e., satisfaction with toolkit).

Table 3.

Provider Reactions to Tablet-Based TF-CBT

Core theme	Subtheme	Number of participants raising theme
Appeal and usability	Activities were easy to use	7 (100%)
	Visuals and interactivity helped child understand material	6 (86%)
	Some content was geared toward younger children	4 (57%)
Relevance and helpfulness	Helpful in keeping on track with TF-CBT model	4 (57%)
	Made session content more manageable for child	6 (86%)
	Increased child engagement in session	7 (100%)
	Would recommend to colleagues	7 (100%)
	Toolkit would be helpful for wide range of patients	6 (86%)

Note. TF-CBT = trauma-focused cognitive-behavioral therapy.

Appeal and Usability.

Most providers reported that the toolkit was easy to navigate and use, and reported that children were generally highly engaged in the tablet activities. One provider expressed, “I absolutely loved using the iPad. [It] helped the kids with understanding far better than they ever would without it or with just plain worksheets.”

Relevance and Usefulness.

All of the providers reported that the toolkit activities increased child engagement during session. For example, one provider stated, “The kids are connecting with the iPad resource and look forward to being able to use it in the therapy sessions.” Another provider noted, “The kids LOVE using the iPad. They have been very engaged in the activities and report having fun!” Moreover, six providers (86%) expressed that they made the session content more manageable for the child. One provider reported, “The videos of the child in cognitive coping has helped children see the concept in action, which has helped many of them get a better understanding of how their thoughts influence their feelings and behavior.” Over half of the providers also expressed that the toolkit helped keep them on task. For example, one provider stated, “I love using the iPad. It has made facilitating session easier. The app goes along beautifully with the TF-CBT model. The kids love it. They have been very engaged in the activities.” All providers reported that they would recommend the tablet toolkit to other providers and six felt that the toolkit could be used with a wide range of patients. For example, one provider reported, “I love that I can use these tools with other patients because it teaches CBT skills. I have used them with all of my patients!”

Child and Caregiver Feedback

Semistructured thematic interviews were conducted with seven patients in the experimental condition, and their caregivers (n = 6), to assess reactions to tablet TF-CBT. The final thematic categories (see Table 4) for youth fell across two domains: (a) use of toolkit activities (i.e., how tools were used in session), and (b) relevance and usefulness (i.e., satisfaction with toolkit). The main theme for caregivers was focused on relevance and usefulness of the toolkit.

Table 4.

Patient and Caregiver Reactions to Tablet-Based TF-CBT

Core theme	Subthemes	Number of participants raising theme
Use of toolkit Activities	Liked interactivity of activities (e.g., WDYK, wheel)	7 (100%)
	Activities helped with expressing emotions better	4 (57%)
Relevance and helpfulness—youth	Increased engagement in sessions	6 (86%)
	Made session content fun	7 (100%)
	Helps understand session content	6 (86%)
	Would prefer iPad in all treatment sessions	6 (86%)
Relevance and helpfulness—caregiver	Increased child engagement in session	6 (100%)
	Made session content more manageable for child	4 (67%)
	Made session content fun for child	6 (100%)

Note. TF-CBT = trauma-focused cognitive-behavioral therapy; WDYK = What Do You Know?

Use of Toolkit Activities.

Several youth also stated that they felt that using the iPad in session was “fun” and allowed them to express themselves well. One child said, “I liked this one [‘Affective Modulation’ chapter] because I got to spin the wheel and act the feeling … I really liked that one! I also liked the card game [‘What Do You Know?’] because they helped me get my feelings out.” Another child expressed, “I liked the wheel the most! Every time we spin the wheel whatever I would choose I would have to act it out—I really liked that game because my auntie got to act it out too.”

Relevance and Usefulness.

Youth and caregivers both reported liking the use of the iPad in treatment sessions because it helped increase engagement and assist with skills-based learning. One child noted, “I played the games and they helped me understand better—it was very useful and fun.” Another child reported, “Just in general I focus more on things digital than on paper … I learn it better.” Moreover, most caregivers reported feeling that using the iPad in the session was “natural” for their children and felt that their children were highly engaged with the activities because they were “gamelike.” For example, one caregiver stated, “I think it made her get more into the sessions. I liked it because it made it fun for her and it didn’t seem like she was coming to see a doctor every time, it made it fun for her—it felt more normal. I think all of the activities her therapist did with [child’s name] were good for her—she responds better to doing games. The games helped her stay on task better and she got to have fun, too.” Families also offered several recommendations to improve the activities (see Table 4).

Discussion

Results supported the acceptability and feasibility of implementing tablet-facilitated treatment in community mental health service agencies. First, providers actively used each of the 11 tablet TF-CBT chapters. Provider and family reactions and engagement with these activities were favorable. Several modules were used extensively. Children were actively engaged, and providers actively used the tablet tools with children throughout the course of their treatment. This was encouraging because we had instructed providers to use only chapters that they felt would be most helpful and relevant to their patients, and therefore did not make any assumptions about the extent to which the tablet toolkit would be used.

Because this was a pilot feasibility study, findings relating to child engagement, provider fidelity, and clinical outcomes were intended to be preliminary and to inform plans for a large-scale trial. With regard to child engagement, we found that children in both conditions were relatively engaged in the therapeutic process. Further study is important to draw interpretations and estimates of impact across conditions. Provider fidelity across conditions was more variable. Treatment fidelity was scored with the use of extensiveness ratings that were operationalized using a combination of the frequency and the thoroughness of the therapeutic intervention and techniques used. It is possible that providers who achieved a particular treatment objective efficiently (e.g., teaching relaxation) may have received a lower extensiveness rating as a result of the frequency score being lower, than one who completed the same objective with less efficiency. Additionally, the components on which standard TF-CBT providers were slightly higher in fidelity than tablet TF-CBT providers are typically used in a broad range of child mental health treatments (e.g., relaxation, affective modulation). Therefore, these components may be less affected by a system like tablet TF-CBT than treatment components that are more unique and specific to a particular protocol (e.g., in vivo exposure, enhancing safety). With regard to patient outcomes, we found that the percentage of children who met criteria for PTSD decreased in the tablet TF-CBT condition (46.1% at postbaseline) compared to the standard TF-CBT condition (57.1% at postbaseline). While preliminary, our rates of PTSD from pre- to posttreatment are comparable to that of other community-based trials (e.g., Jaycox et al., 2010). Future research directions will include a larger sample for statistical comparisons.

Providers, patients, and caregivers indicated, through qualitative interviews, that using the tablet in treatment improved patient engagement, teaching, and learning of content, and that it helped children feel more comfortable in session. Providers also reported that it improved their comfort in delivery of certain TF-CBT components (e.g., cognitive coping). Further study is warranted to examine with adequate statistical power the extent to which technology-based applications in treatment may enhance children’s engagement, provider fidelity, and clinical outcomes.

clinical implications

Whereas many studies have shown that the use of technology does not adversely affect therapeutic alliance in mental health treatment (Sucala et al., 2012), including exposure-based therapies (Germain, Marchand, Bouchard, Guay, & Drouin, 2010; Wrzesien et al., 2013), providers might still harbor concerns about how this may hinder the therapeutic relationship. Our results indicated that the two conditions were very similar with regard to treatment alliance and satisfaction. Moreover, qualitative data suggested that the toolkit helped patients stay engaged in session and assisted with learning the content. Indeed, use of the iPad in session as a learning tool was seen as “natural” by patients, caregivers, and providers, likely because many youth routinely use tablets for learning activities in classrooms and other settings. Youth and caregivers both reported that the tablet toolkit made the sessions “fun,” helped them stay on task, learn the content, and in some cases, gave them the option to express themselves through the games (e.g., writing tasks, emotions wheel) when feeling that they were not ready to verbalize their emotions. The majority of children also reported that they preferred using the iPad in treatment. Several providers noted that the toolkit helped them stay on track when delivering TF-CBT, suggesting that this approach could reduce provider drift by providing tangible reminders of key treatment components.

We chose TF-CBT as our treatment model because it addresses several mental health domains and shares components with other evidence-based, child-focused treatments and involves significant interaction with caregivers. This may heighten applicability of the content to other child mental health treatments. The majority of providers noted in qualitative interviews that they would use this toolkit for children experiencing other mental health disorders, such as depression, anxiety disorders, and disruptive behavior disorders. This approach therefore may have high potential for adaptation to other treatment models.

limitations

One challenge we encountered was that, after providers were randomized, providers in the tablet TF-CBT condition were more likely to refer additional cases to the study than those in the control group (i.e., 79% of families referred postrandomization were in the tablet TF-CBT condition), despite comparable communication and engagement procedures across study conditions. This was consistent with feedback we received from providers at the time of randomization, where many voiced disappointment upon notification of their assignment to the control condition. Adherence to other study procedures (e.g., session audio recording) by standard TF-CBT providers was nevertheless similar to that of their counterparts. Second, we did not measure caregiver engagement, which will be addressed in future work. Third, we were not able to include a chapter to support delivery of the trauma narrative due to budgetary limitations. Future work will include development of this chapter featuring strategies for creating the narrative and for trauma processing. Finally, whereas audio recording of treatment sessions was optimal for coding fidelity, it missed opportunities to assess nonverbal behavior indicators of child engagement or nonengagement in various treatment activities (e.g., body posture or facial expressions to show distraction, avoidance, withdrawal). Future work should include video-recorded sessions coded for fidelity and engagement.

future directions

Data support the feasibility and acceptability of tablet-based applications in community practice settings. Patients, caregivers, and providers readily accepted this approach and were enthusiastic in their expression of perceptions of its value in increasing engagement, supporting learning, and maintaining treatment fidelity. This enthusiasm likely reflects our iterative design approach in which we gathered input from expert trainers/providers (Hanson et al., 2014) and local providers and families (Ruggiero et al., 2015) at multiple stages of development prior to this test of feasibility. Next, we plan to conduct a rigorous, hybrid implementation-effectiveness trial to examine the impact of tablet-facilitated TF-CBT on provider fidelity, child and caregiver engagement, and child mental health outcomes while observing and gathering information on implementation. If data support the effectiveness and incremental value of this approach in community practice settings, we will lead efforts to adapt this approach to other evidence-based treatments and examine optimal approaches to dissemination and implementation in child advocacy centers and other community service settings.