Technology-Enhanced Program for Child Disruptive Behavior Disorders: Development and Pilot Randomized Control Trial

Abstract

Objective

Early onset Disruptive Behavior Disorders (DBDs) are overrepresented in low-income families; yet, these families are less likely to engage in Behavioral Parent Training (BPT) than other groups. This project aimed to develop and pilot test a technology-enhanced version of one evidence-based BPT program, Helping the Noncompliant Child (HNC). The aim was to increase engagement of low-income families and, in turn, child behavior outcomes, with potential cost-savings associated with greater treatment efficiency.

Method

Low-income families of 3-to-8 year old children with clinically-significant disruptive behaviors were randomized to and completed standard HNC (n =8) or technology-enhanced HNC (TE-HNC) (n = 7). On average, caregivers were 37 years old, female (87%), and most (80%) worked at least part-time. Half (53%) of the youth were boys, average age of the sample was 5.67 years. All families received the standard HNC program; however, TE-HNC also included the following smartphone-enhancements: (1). Skills video series; (2). Brief daily surveys; (3). Text message reminders; (4). Video recording home practice; and (5). Mid-week video calls.

Results

TE-HNC yielded larger effect sizes than HNC for all engagement outcomes. Both groups yielded clinically significant improvements in disruptive behavior; however, findings suggest that the greater program engagement associated with TE-HNC boosted child treatment outcome. Further evidence for the boost afforded by the technology is revealed in family responses to post-assessment interviews. Finally, cost analysis suggests that TE-HNC families also required fewer sessions than HNC families to complete the program, an efficiency that did not compromise family satisfaction.

Conclusions

TE-HNC shows promise as an innovative approach to engaging low-income families in BPT with potential cost-savings and, therefore, merits further investigation on a larger scale.

Disruptive behaviors (e.g., noncompliance, defiance, aggression) are among the most common reasons children are referred to mental health care, with the 12-month prevalence of Disruptive Behavior Disorders (DBDs) worldwide second only to anxiety disorders in childhood and adolescence (see Merikangas, Nakamura, & Kessler, 2009, for a review). Although several models have evolved to explain the etiology of the early onset of DBDs in particular, at its core each model highlights the central role of early parenting and parent-child interactions [e.g., Bridging Model (Shaw & Bell, 1993); Cascade Model (Dodge et al., 2009); Early Onset Type (Moffitt et al., 2008)]. Both theoretical and empirical work suggest that there is an escalation of coercive processes characterized by parents initially trying more controlling parenting techniques in response to the child’s disruptive behavior; however, when the child’s noncompliance and oppositionality escalate in response, parents of youth vulnerable to DBDs tend to acquiesce, unintentionally reinforcing and, subsequently, exacerbating the disruptive behavior (see McMahon & Forehand, 2003, for a review). With the aim of early intervention, the field of Behavioral Parent Training (BPT) evolved via the development of programs targeting the coercive cycle and, in turn, decreasing vulnerability for protracted disturbances of behavior in adolescence and adulthood (see Forehand, Jones, & Parent, 2013; Garland, Hawley, Brookman-Frazee, & Hurlburt, 2008; Jones et al., 2013; Reitman & McMahon, 2012, for reviews).

BPT programs have a rich history and robust evidence-base (see Chorpita et al., 2011, for a representative review); however, consistent with children’s mental health more broadly, “parenting interventions in general are less successful at engaging the most distressed and disadvantaged families” (Gardner et al., 2009; p. 545). Family Stress Theory highlights the indirect impact of financial strain on children through parental stress and associated compromises in parenting (see Conger & Donnellan, 2007, for a review). Accordingly, it is not surprising that low-income families are more vulnerable to the coercive cycle of parent-child interaction implicated in the development and exacerbation of DBDs and, in turn, more likely to have a child with an early onset DBD than relatively higher income families (see Dekovic et al., 2011; Jones et al., 2013; Lundahl, Risser, & Lovejoy, 2006; Piquero, Farrington, Welsh, Tremblay, & Jennings, 2009, for reviews). Data suggest that if we successfully engage low-income families in BPT services, they benefit as much, if not more, than relatively higher income families, particularly at post-treatment and when the problem behaviors are in the clinical range (Fernandez, Butler, Eyberg, 2011; also see Dekovic et al., 2011; Leijten, Raaijmakers, de Castro, & Matthys, 2013; Reyno & McGrath, 2006, for reviews). Financial strain and associated difficulties (e.g., poor mental/physical health, un/underemployment, lack of health insurance/underinsured), however, decrease the probability that low-income families will engage at a level necessary for BPT to be efficacious (e.g., 12 to 28 session hours, mid-week telephone check-ins, daily home practice; see Eyberg, Nelson, & Boggs, 2008; McMahon & Forehand, 2003; Reyno & Mcgrath, 2006; Thomas &-Gembeck, 2007, for reviews). Accordingly, a number of approaches have been tested to improve the engagement of families in BPT, including home-based and group-focused programs, as well as monetary incentives for attendance (Dumas, Begle, French, & Pearl, 2010; Gross et al., 2011; MacKenzie, Fite, & Bates, 2004; McGilloway et al., 2012). These approaches, however, have yielded similarly or more disappointing engagement rates than traditional clinic-based, individual family-focused BPT programs (also see Jones et al., 2013; Lundahl et al., 2006, for reviews), highlighting the critical public health importance of testing innovative approaches to engage low-income families.

As highlighted elsewhere (Aguilera & Muench, 2012; Kazdin & Blasé, 2011), it is time for a paradigm shift in the delivery of clinical services and technology is at the forefront of this shift. Technology, albeit in its most basic forms (i.e., videotape modeling), is firmly rooted in the origins and history of BPT (e.g., Flanagan, Adams, & Forehand, 1979; Nay, 1976; O’Dell et al., 1982); however, the potential exists for technology to advance the field further (see Jones et al., 2013, for a review). Of greatest relevance to our purposes, there is a relatively untapped potential for technology to better situate BPT programs at the forefront of low-income families’ daily lives, particularly between sessions and inspite of both acute and chronic family stressors associated with financial strain. Although some may worry that the utilization of technology to increase engagement may widen the digital divide, low-income homes are actually the most likely to forego landline service in favor of mobile phones, with rates of smartphone ownership in particular on the rise due to the growing accessibility of a range of affordable smartphone options and providers, as well as the range of accessible platforms afforded by one handheld and portable device (e.g., internet, video calls and conferencing, videotaping, email, chat rooms, & social networking) (see Anderson & Subramanyam, 2011; Davies, 2011; Snider, 2011, for reviews). It is precisely this accessibility, portability, and varied functionality of smartphones that have led some to refer to them as “therapeutic gold”, highlighting the relatively untapped potential to utilize smartphones to better connect clients with evidence-based services (Aguilera & Muench, 2012; p. 70). Accordingly, this study describes a line of research in which the authors developed and pilot tested the feasibility and outcomes of the Technology-Enhanced Helping the Noncompliant Child (TE-HNC) program, a smartphone-enhanced version of one evidence-based BPT program, Helping the Noncompliant Child (HNC; McMahon & Forehand, 2003).

HNC is one example of a clinic-based, individual family-focused, BPT program that evolved from the Hanf Model of BPT, a BPT framework that is synonymous with a collection of efforts to promote improved behavior and healthier family functioning among young children with behavior problems (see McMahon & Forehand, 2003; Reitman & McMahon, 2012, for reviews). The HNC program in particular, however, may be exceptionally well suited to serve as a comparison group when evaluating the effects of smartphone-enhancement with low-income families for two primary reasons. First, HNC is a criterion-driven program; that is, progression to each new skill is dependent upon meeting criterion (i.e., mastery) on the prior skill. As highlighted elsewhere, BPT programs requiring mastery before advancement are associated with better outcomes for low-income caregivers (Rogers, Forehand, Griest, Wells, & McMahon, 1981; also see Reyno & McGrath, 2006, for a review). Second, HNC materials are written at a 6^th grade reading level, making it ideal for low-income families with potential literacy issues (see McMahon & Forehand, 2003, for a review).

Building upon the relevance of HNC in particular for low-income families, we expected that the smartphone-enhancements to the HNC program (i.e., skills video series, daily surveys of skill practice and progress, video recording home practice for review and feedback, text message reminders and alarms regarding sessions, calls, practice; and mid-week video call check-ins) would provide increased opportunities for feedback, support, and skill modeling to families both between and within HNC sessions, increasing the likelihood that families would remain engaged in the HNC program (i.e., attend weekly sessions and mid-week phone calls) and practice the HNC skills outside of session (i.e., daily home practice) (see Jones et al., 2013; Williams, Lynch, & Glasgow, 2007, for discussion of how technology has the potential to increase intrinsic motivation for behavior change). With regard to daily home practice in particular, we were interested in the extent to which the smartphone-enhancements had the potential to not only increase the participating caregiver’s practice, but non-participating caregiver practice as well. Regardless of income, only one caregiver typically participates in BPT services (Cowan, Cowan, & Berry, 2011; also see McMahon & Forehand, 2003, for a review); however, this trend is even more pronounced and, in turn, may have more pronounced implications for treatment outcome in low-income families. That is, low-income families who are navigating challenges with balancing child care, shift or hourly employment, and transportation may simply not be able to afford or manage the participation of more than one caregiver, a circumstance that increases the potential for inconsistency between parents, and decreases the likelihood that the program will lead to improvements in child behavior (see McMahon & Forehand, 2003, for a review). As such, we hypothesized that the smartphone-enhancements may facilitate coparent involvement in home practice by providing participating caregivers a mechanism (e.g., skills videos series) and support (e.g., text message reminders & smartphone-assessment items regarding the importance of coparent involvement and practice) for involving the non-participating coparent in the BPT program.

HNC is an established, evidence-based BPT program; therefore, we expected both groups to evidence clinically significant improvement in child disruptive behaviors. However, we hypothesized that the smartphone-enhancements would boost HNC’s impact on child behavior via enhanced engagement and skill generalization, as well as increased opportunity for skill modeling, practice, and feedback. Finally, by collecting both start-up and implementation costs, the study design allowed us to assess whether enhanced engagement and skill generalization resulted in greater efficiency (i.e., cost-effectiveness) of services for TE-HNC without compromising family satisfaction with the program.

Method

Overview

The line of research presented here has proceeded consistent with the recently proposed Web-Based Treatment Research Cycle (Web-TRaC), which aims to move the field beyond the development of novel, mobile mental health intervention approaches (Stage 1) to theoretically-informed RCTs (Stage 2), which are largely lacking in the literature on mobile and other technology-enhanced interventions (see Enock & McNally, 2013, for a review).

Participants

Families were included in the project if they met criteria for “low-income” (i.e., adjusted gross income did not exceed 150% of the federal poverty limit, which takes into account both income and number of residents in the home); they had a child in the 3-to-8 year old age range (age range for which HNC was developed and tested); and the child exhibited disruptive behaviors in the clinical range as evidenced by meeting or exceeding clinical-cutoffs on the caregiver-report of the Eyberg Child Behavior Inventory (ECBI; Eyberg & Pincus, 1999) Severity or Intensity Subscales.

Exclusion criteria were: (1). Child developmental or physical disability that precluded use of HNC skills; (2). Caregiver current diagnosis of substance abuse/dependence, mood, or psychotic disorder; and/or (3). Family involvement with Department of Social Services related to abuse/neglect.

Smartphone ownership was not an inclusion/exclusion criterion as the project provided identical smartphone models to all participating families in order to ensure that the human-interface (i.e., design, usability, interactivity) of the smartphone enhancements were identical for all families; however, the generalizability of this work rests on the premise that the technology is accessible and available to low-income families. Forty-seven of the 48 families who completed a clinic-based eligibility interview (see Figure 1) owned a mobile phone and, of those, 30% (n = 14) owned a smartphone.

Figure 1.

Participant-flow from baseline to trial completion.

Procedure

Low-income families in north central North Carolina (NC) were recruited via: (1). Advertisements targeting areas, work places, and retail outlets with an overrepresentation of low-income parents (48% of pilot families); (2). Healthcare, social service, and other agencies that serve low-income families (28% of pilot families); (3). Local schools (19% of pilot families); and (4). Word-of-mouth (5% of pilot families). Of note, NC has the 10^th worst child poverty rate in the country, with 46% of children living in low-income families (National Center for Children in Poverty, 2010). As such, this area afforded a relevant recruitment infrastructure for our pilot research.

A brief (20-minute) phone screen was conducted to determine interest and initial eligibility, which was then confirmed by a clinic-based interview, which also included consent and a more extensive caregiver assessment battery. During treatment, therapists collected data on engagement and skill generalization, as well as costs (e.g., number of sessions). Within two weeks of treatment termination, the caregiver assessment battery was re-administered (post-assessment). All assessments and sessions were completed at a community-based university training clinic in north central NC. Families were typically seen in the late afternoon to evening (school-aged children) or morning to early afternoon (preschool children), with appointments chosen by families to enhance convenience. Caregivers were compensated $50 per assessment and TE-HNC children received a “safe phone return bonus” ($100). All smartphones were returned.

Shared Intervention Features

All families received the standard, two-phase HNC program. HNC (McMahon & Forehand, 2003). Following an orientation session, caregiver-child dyads start Phase I, Differential Attention, in which caregivers learn to increase the frequency and range of social attention to the child and to reduce the frequency of competing verbal behavior. A primary goal is to reduce the coercive cycle of parent-child interaction by (re)establishing a positive and mutually reinforcing parent-child relationship. In the context of “Child’s Game” (i.e., child-directed play), the caregiver is taught to: (1). Increase the frequency and range of positive attention; (2). Eliminate instructions, questions, and criticisms; and (3). Ignore minor inappropriate behavior. Parents are instructed to practice Child’s Game for at least 15 minutes per day and told that coparents also should practice. Parents progress to Phase II when the criteria for Phase I skills are met (McMahon & Forehand, 2003).

In Phase II, Compliance Training, caregivers are taught the difference between unclear and clear instructions, to give the “Clear Instruction” sequence, and to use a nonphysical discipline procedure, “Time-Out”, for occasions of noncompliance and other inappropriate behavior that cannot be ignored. Phase II skills are taught within the context of “Parent’s Game” (i.e., parent-directed activities, such as a clean up task), although caregivers are instructed to continue to practice Child’s Game at home to maintain mastery of Phase I skills. Program completion is based on caregivers meeting criteria for all Phase I and II skills, which requires an average of 8 to 12 sessions (McMahon & Forehand, 2003).

Development of the TE-HNC Program

Prior health services research, as well as research on BPT in particular, suggests that some level of therapist involvement may be optimal for the most distressed and disadvantaged families (e.g., Webster-Stratton, 1990; Webster-Stratton, 1992; Webster-Stratton, Kolpakoff, & Hollinsworth, 1988; also see Mohr, Cujpers, & Lehman, 2011; Tate & Zabinski, 2004, for reviews). Accordingly, TE-HNC includes the HNC program, as well as smartphone-components that were developed to enhance, rather than replace, clinic-based, therapist-guided services. Smartphone-enhancements were developed via an interdisciplinary partnership including: 1) Researchers with expertise in BPT with underserved families; 2) A Clinician Advisory Panel (20% male; 20% ethnic minority) who practice at least one BPT program; 3) An industry partner with experience developing sustainable technological applications; and 4) Health economists with expertise in health care efficiency, efficacy, and value.

Specifically, iterative feedback and modifications led to these smartphone-enhancements (see Jones, Forehand, Mckee, Cuellar, & Kincaid, 2010, for a review): (1). A 3-minute skills video for each of the HNC skills, including psychoeducation, as well as modeling of the skill by parent-child dyads; (2). Daily surveys of skill practice and progress that are used to guide mid-week calls and weekly sessions (e.g., problem-solving more suitable home practice times if a parent indicates a failure to practice on a daily survey); (3). Mid-week video-calls during which therapists reinforce caregivers for progress and problem-solve obstacles to practice (e.g., helping the family to pick a time-out location in the home); (4). Weekly videotaped home practice, which provided a “window” for therapists to use during the session to provide feedback regarding skill development; and (5). Text reminders regarding the relevance of home practice, the mid-week call, and session attendance, as well as reinforcing messages regarding progress. As such, TE-HNC capitalizes on the capacity for smartphones to push HNC content to the caregiver, rather than relying on the caregiver to access the content, a proven strategy with other low-income clients (Aguilera & Muench, 2012; Aquilera & Munoz, 2011).

Therapist Training & Supervision

Therapists were M.A. level graduate students. Training included didactic presentations and practice, one practice case per therapist, and clinical emergency protocols. Therapists participated in weekly supervision, which included reviewing and discussing videotaped sessions. Therapist fidelity to program materials for both programs was coded using the following procedures: 1) The critical material (e.g., rationale for program, explanation of a skill, practice of the skill with the child) to be covered in each session was delineated; 2) Naïve, trained coders watched video recordings of sessions; and 3) Coders indicated whether or not each of the critical points was covered in the session. Over 50% of sessions were coded for fidelity by two coders, who achieved over 90% reliability, yielding an average fidelity rating 90%. Finally, therapists treated families in both arms of study.

Measures

Engagement & skill generalization

Given the pilot nature of the study, engagement and skill generalization were assessed utilizing both between- and within-group measures. The between-group measures of engagement were operationalized as the following: 1). Session attendance (i.e., family showed for scheduled weekly appointment) and 2). Mid-week call availability (i.e., family available for the scheduled mid-week call). In addition, the between-group measure of skill generalization was assessed utilizing a measure of caregiver report to the therapist regarding whether s/he completed their daily skill practice, as well as whether the caregiver completed assigned worksheets.

Several measures of engagement and skill generalization within the TE-HNC group were assessed. Caregivers randomized to the TE-HNC program were asked to do the following utilizing their smartphones: 1). Complete brief daily surveys regarding their skill practice and 2). Videotape at least one home practice. As noted earlier, five versions of the daily surveys were created, one version per HNC skill (e.g., caregivers were asked to complete the Attends Survey each day between sessions until they met behavioral criteria for Attends). Daily smartphone surveys were intended to be brief, with the surveys increasing slightly in length as the family progressed through the program and skills (e.g., Rewards Survey included questions about Attends plus questions about Rewards). Each item was also intended to be brief, with one question (e.g., “Did you practice Child’s Game today”) and the associated response options (e.g., “Yes” or “No”) appearing on the smartphone screen at a time. Caregiver response to each item determined the sequence of items (e.g., if the caregiver indicated that s/he practiced Child’s Game, the next item would ask “How long did you practice Child’s Game?” with response options ranging from less than 15 minutes, 15 minutes, or more than 15 minutes). Responses were automatically uploaded to the therapist’s secure computer via identification code and were utilized by therapists to guide the mid-week check-in and subsequent session.

TE-HNC families were also asked to utilize their smartphones and tripods to video record at least one home practice (i.e., Child’s Game) during the week to share with the therapist. During the session, therapists allowed caregivers to suggest parts of the video recorded home practice that they thought went particularly well or parts where they may have had more difficulty, and therapists provided reinforcing and corrective feedback accordingly. If caregivers video recorded more than one home practice, they picked the one for which they had the most difficulty or the most questions.

Finally, we were interested in the extent to which smartphone-enhancements facilitated non-participating coparent involvement in home practice. TE-HNC families were asked the following open-ended question at post-assessment only: “Did you think sharing the parenting skills videos with your coparent was useful in terms of letting him/her know what you were learning and getting him/her to practice the skills too?” At the post-assessment, HNC families were provided an overview of the smartphone-enhancements and asked: “Now that you know all of the things the families in the other group used the smartphones to do, do you think having coparents watch the skills videos everyday would have been helpful with getting coparents like yours involved in the parenting program?”

Child disruptive behavior

Intensity and Problem subscales on the 36-item ECBI (Eyberg & Pincus, 1999) served as the dependent measure of behavior change due to the availability of normative data sensitive to age (2 to 16 years old; Burns & Patterson, 1991; Burns, Patterson, Nussbaum, & Parker, 1991) and established psychometrics with low-income samples (e.g., Fernandez et al., 2011). For each item, caregivers rate the intensity of the behavior (0 = never to 7 = always) and whether each behavior is a problem (0 = no; 1 = yes). Clinically-significant symptoms are defined by scores more than 2 standard deviations above the normed mean for Intensity (clinical cut-off = 127) and/or Problem (clinical cut-off = 11) Scales. Alphas in this study were 0.86 (Intensity) and 0.72 (Problem).

Program costs and consumer satisfaction

Program start-up and implementation costs were collected and analyzed separately. Start-up costs include non-labor costs for therapist manuals, toys, and handouts for both groups and, for the TE-HNC group only, smartphone-related costs (i.e., purchase of smartphones, service plans, enhancements etc.). Implementation costs include the value of labor resources required to deliver the program (including value of therapist time). Therapists reported time spent both in and out of face-to-face sessions (including time spent on phone calls, review of session notes, supervision, and other activities). Therapist time was valued using 2010 median national hourly wage rates for M.A.-level mental health counselors from the Bureau of Labor Statistics.

In order to ensure that any cost savings of the TE-HNC program (e.g., through reductions in the number of intervention sessions) were not obtained at the expense of family satisfaction, HNC and TE-HNC families completed the HNC Consumer Satisfaction Scale (McMahon & Forehand, 2003). The Consumer Satisfaction Scale is a 42-item measure that assesses caregiver satisfaction with the overall HNC program, the difficulty and usefulness of the HNC program format and skills, and the efficacy of the therapist. A total satisfaction score was calculated (alpha = 0.82).

Data Analytic Approach

Enrollment and baseline characteristics of the sample are examined. Then, owing to the pilot nature of the study, pre-post analysis of primary study variables (engagement, skill generalization, and child behavior outcomes) are conducted with only those families who completed both the pre- and post-assessment. First, between-group effect sizes (0.20 – 0.49 small, 0.50 – 0.79 medium, and ≥ .80 large) were used to compare TE-HNC to HNC on engagement and skill generalization to the home (Cohen, 1988). Whereas significance testing conveys the likelihood that study results differ from chance expectations, effect-size calculations convey the relative magnitude of the experimental effect and, in turn, provide the opportunity to compare the magnitude of treatment effects within and across studies (see Thalheimer & Cook, 2002, for a review). Consistent with Cohen’s d and more recent modifications (D’Amico et al., 2001), the difference between group means (e.g., EngagementMean_TE-HNC – EngagementMean_HNC) was divided by the average of each mean’s standard deviation (e.g., (SD_TE-HNC + SD_HNC)/2). In addition, as one per week video recordings and daily surveys of Child’s Game home practice were part of the TE-HNC program, we also utilized within-group statistics only to examine whether TE-HNC caregivers completed these assignments.

Second, we compared TE-HNC and HNC groups on child behavior outcomes (ECBI Intensity and Problem Subscales) utilizing both within- and between-group effect sizes and measures of clinical significance. Within-group effect sizes were calculated utilizing Morris and DeShon’s (2002) Equation 8 to control for the correlation between pre- and post- assessment means for each group. Given that small sample sizes can differ on pre-assessment scores, between-group effect sizes were calculated by first creating change scores for each family (e.g., pre-assessment to post-assessment). The between-group effect size then is calculated by subtracting the average HNC group change score from the average TE-HNC change score, and dividing by the standard deviation as described above.

In order to test clinical significance of treatment effects (see Kendall, 1999, for an introduction to a special section on clinical significance), we utilized: 1) Reliable Change Index (RCI; Jacobson & Truax, 1991), which indicates change attributable to treatment is most likely not due to chance (i.e., RCI ≥ 1.96; also see Abramowitz, 1988, for modifications for use with samples); and 2) Normative Comparisons (Kendall & Grove, 1988), which assess if scores at post-treatment are distinguishable from individuals in the normative range [i.e., is the group mean and upper limit (M + 1 SD)].

Finally, we examined program costs and consumer satisfaction. Costs were quantified using a payer perspective to value the time and resources required to deliver the program. Costs associated with program start-up (i.e., non-labor start-up and development costs) and implementation (i.e., therapist time) were analyzed separately. Start-up costs are considered investments that are spread over all program participants, so we assigned an equal proportion of these costs to each program participant, regardless of how long they participated in HNC or TE-HNC (i.e., these analyses include both completers and non-completers). In order to avoid inadvertently and inaccurately biasing implementation costs downward by including labor costs for participants who dropped out before mastering a skill, implementation costs were collected at the participant level and include completers only. Consumer satisfaction was assessed using between group effect sizes described above.

Results

Enrollment and Baseline Characteristics

Of the low-income families who completed a clinic-based eligibility interview (n = 48), 24 were ineligible for the pilot RCT (e.g., pending DSS investigation) and 2 eligible families chose not to enroll. Using restricted random assignment to force equal sample sizes, Master’s-level therapists randomly assigned the remaining 22 eligible low-income families to the HNC or TE-HNC group. Of these 22 families, 96% (n = 21) owned a mobile phone and, of these, 33% (n = 7) owned a smartphone; however, as noted earlier all families were provided a project smartphone to ensure a common interface and access to materials among participants.

The initial three families randomized served as practice cases for each of the three project therapists, and the remaining 19 families HNC (n = 10) and TE-HNC (n = 9) were considered for use in the current analyses.

Attrition and Missing Data

Overall rate of attrition (21%) was substantially lower than what has been reported in prior BPT work with low-income samples (i.e. as high as 56%; Fernandez et al., 2011; also see Lundahl et al., 2006; Reyno & McGrath, 2006, for reviews). The four families (HNC = 2; TE-HNC = 2) who dropped out of the study notified project staff prior to discontinuing participation and each cited a major health (e.g., organ transplant) or family (e.g., divorce) stressor necessitating dropout. Given the pilot nature of the project, we considered only complete data (i.e., data from participants available at pre- and post- treatment). On average (see Table 1), caregivers who completed both the pre- and post-assessments were 37 years old, most were female (87%), and 80% worked at least part-time. Half (53%) of the children were boys, with an average age of 5.67 years (range 3 to 8 years old).

Table 1.

Demographic characteristics of the low-income sample at pre-assessment (Total n =15; TE-HNC n =7; HNC n =8).

		Treatment Groups
		Total Sample			TE-HNC			HNC
Measure	Possible Range	%	Mean	SD	%	Mean	SD	%	Mean	SD
Child Demographics
Gender (% Male)		53			57			50
Age (yrs)	3–8		5.67	1.72		5.57	1.27		5.75	2.12
Ethnicity/Race (% minority)					57			63
Caregiver Demographics
Gender (% Female)		87			71			100
Age (yrs)			36.73	8.81		35	5.92		38.25	10.95
Ethnicity/Race (% minority)					29			50
Marital status
Married		33			43			38
Single		67			57			62
Employment status
Unemployed		20			29			13
Part-time		33			29			38
Full-time		47			42			50
Child Behavior
Eyberg Child Behavior Inventory
Intensity	0–252		139.6	24.13		148.86	22.51		131.5	23.87
Problem	0–36		21.47	4.93		22.57	5.19		20.50	4.81

Analysis of TE-HNC vs. HNC Condition at Baseline

As shown in Table 1, there were no differences between families randomized to TE-HNC or HNC on demographics; however, randomization failed to yield equivalent groups on an established correlate of BPT drop out, baseline disruptive behaviors. Caregivers randomized to TE-HNC were more likely to report higher levels of child disruptive behaviors on the Intensity Subscale in the TE-HNC group at baseline (M_{ECBI Intensity}: TE-HNC = 148.86; HNC = 131.50); however, both groups evidenced problem behaviors in the clinical range on both the Problem and Intensity Scale scores.

Engagement & Skill Generalization

As shown in Table 2, findings demonstrated medium to large between group effect sizes for engagement and skill generalization favoring TE-HNC relative to HNC. Effect sizes demonstrated that families in TE-HNC were more likely to attend weekly sessions (d = 0.88), participate in mid-week calls (d = 2.59), and complete their Child’s Game home practice (d = 0.63) than families in HNC.

Table 2.

Means, standard deviations, effect sizes, and clinically significant change for primary study variables (TE-HNC n = 7; HNC n = 8).

	Assessment
Measure	Pre % M(SD)	Post % M(SD)	ESa Pre-Post (d)	CIb 95%	ESc TE-HNC vs. HNC	CId 95%	RCIe	Normativef Comparison
Engagement					0.88	−0.29; 1.83
Session Attendance
TE-HNC		97 (5)
HNC		90 (11)
Mid-week Check-In					2.59	0.97; 3.92
TE-HNC		93 (8)
HNC		58 (19)
Skill Generalization
Home Skill Practice					0.63	−0.42; 1.68
TE-HNC		91 (13)
HNC		77 (23)
Child Behavior
ECBI
Intensity					0.99	−0.13; 2.05
TE-HNC	148.86 (22.51)	83.00 (15.34)	3.71	2.00; 7.80			7.40*	Yes
HNC	131.50 (23.87)	91.63 (21.25)	1.20	0.36; 2.98			4.48*	Yes
Problem					0.54	−0.51; 1.56
TE-HNC	22.57 (5.19)	6.14 (5.67)	2.00	1.59; 4.73			6.65*	Yes
HNC	20.50 (4.81)	8.88 (8.17)	1.24	0.87; 4.38			4.71*	No

^aWithin group effect size (ES) at post-assessment controlling for pre-assessment;

^b95% Confidence interval (CI) for within-group ES calculations;

^cBetween-group ES comparing TE-HNC vs. HNC at post-assessment controlling for pre-assessment;

^d95% CI for between group ES calculations;

^eReliable Change Index (RCI) = change attributable to treatment is clinically significant (*RCI ≥ 1.96);

^fNormative Comparison = at post-treatment group mean and upper-limit (M + 1 SD) is within normative range.

TE-HNC families also provided information regarding Child’s Game practice by completing scheduled daily smartphone surveys and a video recording of one home practice per week. Regarding the surveys, across each of the five HNC skills, on average, TE-HNC families completed on average a survey on 60% of possible between session days. The median time for survey completion across families and skills was 2 to 3 minutes. During Phase I, TE-HNC families reported the following: (a). They practiced Child’s Game on the majority (69%) of the days for which they completed the surveys; (b). On 86% of those days, families reported that they practiced Child’s Game the recommended minimum of 15 minutes per day; and (c). The majority (95%) reported that their child enjoyed Child’s Game. Findings were similar for Phase II: (a). Caregivers reported practicing Child’s Game on 76% of the days for which they completed a survey; (b). On the majority (83%) of those days, caregivers reported practicing the minimum of 15 minutes per day; and (c). Almost all (98%) reported that their child enjoyed Child’s Game. Regarding video recording, the data suggest this pattern: 2 families video recorded at least one home practice per week between each HNC session, whereas the remaining families recorded at least one practice between 71% (n = 1), 50% (n = 3), and 33% (n = 1) of sessions.

Finally, representative feedback from TE-HNC families regarding how smartphone-enhancements facilitated coparent involvement in practice included the following: “It was nice to be able to not only tell him what I learned, but to show him [with the skills videos]”; “[without the smartphone enhancements], I don’t think he would have understood everything as well”; and “If my husband [did not have the smartphone enhancements], I don’t think any of this would have made sense to him and he would have lost interest”. Representative feedback from caregivers in the HNC group regarding how the smartphone-enhancements would have been helpful include: “It would have allowed her to have a fuller sense [of the program] without being affected by my wording”, “having the [smartphone-enhancements] would have made the program more real [for him], [he] would have taken the program more seriously”, and “he would have gotten the information on his own terms and in his own time [with smartphone enhancements, which] would have eased the tension”.

Child Disruptive Behavior

Findings reported in Table 2 reveal that HNC yields large effect sizes (Intensity d = 1.20; Problem d = 1.24) for change in child behavior from pre-to-post treatment, as well as clinically significant improvement (Intensity RCI = 4.48; Problem RCI = 4.71) that is within the normative range at post-treatment for the Intensity but not the Problem Subscale. However, all the indicators suggest TE-HNC may boost child behavior outcomes: the between-group effect size comparing the two interventions favored TE-HNC (Intensity d = 0.99; Problem d = 0.54;); the within-group TE-HNC effect size (Intensity d = 3.71; Problem d = 2.00) was larger than the HNC effect size; the RCIs for TE-HNC (Intensity RCI = 7.40; Severity RCI = 6.65) were greater than the RCIs for HNC; and normative comparisons resulted in TE-HNC being within normative range for both ECBI Subscales).

Program Costs & Consumer Satisfaction

As described in the Data Analytic Approach, start-up costs were calculated for all enrolled families, while implementation costs were calculated for families who completed the programs only. Both interventions incurred start-up costs for therapist manuals, toys, and handouts, resulting in program start-up costs of an average of $10 per enrolled family. The TE-HNC intervention had additional start-up costs for smartphone-related services and materials, yielding total program start-up costs for the TE-HNC intervention of $671 per enrolled family.

Consistent with our interests in the implementation costs of TE-HNC compared to HNC (i.e., therapist time cost savings, if any, associated with utilizing the smartphone-enhancements, Table 3 reports mean implementation costs only (i.e., excludes program start-up costs) for each of the program components among program completers only for HNC and TE-HNC families. Skill 5 (Time Out) required the most therapist time (an average of 2.3 sessions among HNC families and 2.1 sessions among TE-HNC families) and, thus, was the most expensive skill, costing an average of $127 per family in HNC and $119 per family in TE-HNC to master. The average implementation cost to master a skill was similar for each intervention approach: $82 for HNC and $80 for TE-HNC. Total implementation costs were also similar for the two interventions, although HNC costs were slightly higher than costs for TE-HNC ($491 for HNC and $478 for TE-HNC) because of the additional number of sessions required for HNC families to complete the program. HNC families had an average of 10 therapist sessions (with a range of 7 to 12 sessions) to complete the program, while TE-HNC families averaged 8 therapy sessions to completion (range of 7 to 10 sessions).

Table 3.

Mean therapist costs, sessions, and time per program completer (TE-HNC n = 7; HNC n = 8)

	HNC			TE-HNC
	Cost	Number of Sessions	Totala Time (minutes)	Cost	Number of Sessions	Totala Time (minutes)
Program Components
Orientation	$50	1.0	138	$61	1.0	158
Attends	$64	1.2	171	$83	1.4	228
Rewards	$123	2.4	330	$72	1.3	200
Ignoring	$57	1.1	157	$61	1.0	164
Clear Instruction	$70	1.3	194	$81	1.4	224
Time Out	$127	2.3	328	$119	2.1	328
Mean Per Component	$82	1.6		$80	1.4
Total Per Completer	$491	10.0	1318	$478	8.3	1302

^aIncludes sessions, mid-week calls, session notes, supervision, review of daily surveys (TE-HNC only), watching home practice video (TE-HNC only), and other program related activities.

Our cost data analyses also allowed us to examine therapist time per completer family in order to consider whether enhanced engagement, skill generalization, and child behavior outcomes were simply a function of TE-HNC therapists spending more time in contact with families than HNC therapists. As demonstrated in Table 3, TE-HNC required slightly less, rather than more, total therapist time on average (M = 1302 total minutes per family) than HNC (M = 1318 total minutes per family).

Finally, consumer satisfaction analyses revealed that the increased efficiency (i.e., fewer sessions) did not compromise family satisfaction with the TE-HNC (M = 59.83, SD = 1.94) relative to HNC (M = 56.56, SD = 2.47) program. Rather, between-group effect size analysis favored TE-HNC (d = 1.48).

Discussion

Low-income youth are more likely to have DBDs than relatively higher income youth, yet less likely to engage in BPT (see Jones et al., 2013; Lundahl et al., 2006; Reyno & McGrath, 2006; for reviews). The objective of this project was to develop and pilot a technology-enhanced version of one evidence-based BPT program, HNC (McMahon & Forehand, 2003), with the aim of increasing engagement and, in turn, BPT outcomes, among low-income families of children with DBD. Findings suggest that TE-HNC shows promise as an innovative and efficacious approach to engaging low-income families in BPT with potential cost-savings.

The current pilot data suggested medium (mid-week call participation) to large (session attendance, home practice) effect sizes favoring TE-HNC relative to HNC on measures of engagement and skill generalization. The weekly demands of BPT are time-intensive and low-income families are less likely to effectively engage in BPT services due to financial strain and associated stressors (see Eyberg et al., 2008; McMahon & Forehand, 2003; Reyno & Mcgrath, 2006, for reviews). Our pilot findings suggest that smartphone-enhancements have the potential to increase the therapist and program’s connection to and support for the family between sessions and, in turn, increase family’s autonomy with implementing the skills in the home setting.

As HNC is an evidence-based BPT program, it was expected, and did occur, that HNC would yield clinically significant change in child disruptive behavior; however, effect size and clinical significance findings suggest that TE-HNC may boost child treatment outcome. Although testing the mechanisms that account for the boost in child behavior outcomes is not possible due to limitations in statistical power, we hypothesize that enhanced engagement and skill generalization boost the impact of the program on child behavior outcomes.

Although limited by within-group examination, trends in the daily smartphone surveys and video recording of home practice may further help to explain the mechanisms by which smartphone-enhancements boost child outcomes. First, daily survey data revealed that TE-HNC caregivers reportedly practiced Child’s Game for at least the recommended 15 minutes per day on the majority of days between sessions, suggesting that the smartphone reminders about skill practice is associated with skill generalization from the clinic to the home and, in turn, enhanced child behavior outcomes. Second, the majority of HNC families video recorded at least one home practice between the majority of their sessions, suggesting that families benefited from increased opportunity for therapist review of and feedback on their skill practice with subsequent impact on child behavior. Third, caregiver responses to post-treatment interviews about the programs suggest that smartphone-enhancements may have enhanced coparent practice of the skills at home by providing caregivers a mechanism (i.e., skills videos series) and support (e.g., text message reminders & smartphone-assessment items regarding the importance of coparent involvement and practice) for involving the non-participating coparent in the BPT program. This can increase the consistency of skill utilization between caregivers, and, in turn, lead to greater improvements in child behavior (see McMahon & Forehand, 2003).

Tate, Finkelstein, Khavjou, and Gustufson (2009) recently highlighted that cost-effectiveness is a primary rationale for utilizing technology to enhance service delivery; nevertheless they point out that few studies actually report data on economic indicators. Fortunately, some BPT programs have begun to report cost data (e.g., Foster, Prinz, Sanders, & Shapiro, 2008; O’Neil, McGilloway, Donnelly, Bywater, &Kelly, 2011; Sanders, 2008). Our pilot study was designed in part to move beyond these studies and address the question of whether technology-enhanced delivery of BPT had the potential to yield cost savings.

Not unexpectedly, the development of TE-HNC was costly; however, these start-up costs are viewed as one-time expenses for a service delivery agency. Furthermore, as smartphones continue to increase in use, it will not be necessary (as we did) for agencies to purchase smartphones and service plans for their clients. In essence, as the cost of smartphones continue to decline and the availability of contract-free carriers (e.g., $19.99 smartphone models with no credit checks and unlimited data at no extra charge), increases the cost of facilitating smartphone-enhanced programs like TE-HNC will decline.

Our primary interest was not in program development (i.e., start-up) costs but rather implementation. The data from this study suggest that the number of sessions required to complete treatment (i.e., achieve HNC criteria for skills) was fewer and overall therapist time from orientation through Time-Out was slightly less for TE-HNC than for HNC. We believe that as therapists became increasingly proficient with the technology, the therapist time to implement TE-HNC will decrease, boosting the minutes per client differential between TE-HNC and HNC.

Of course, this was a pilot study and the limitations must be noted. First, as noted earlier, limitations in sample size and methods preclude significance testing, analysis of the mechanisms by which smartphone-enhancements impact BPT outcomes, as well as quantitative comparison of groups on some study variables (i.e., TE-HNC only variables). In addition, as noted earlier, randomization did not yield equivalent groups on children’s baseline level of disruptive behaviors, a variable typically associated with treatment outcome; however, it is notable that while TE-HNC children had higher levels of disruptive behaviors than HNC children at baseline, the former group had lower levels at post-treatment. Finally, there is much discussion in the broader treatment outcome literature regarding tension between internal and external validity and our pilot study is no exception. Our exclusion criteria have the potential to limit the generalizability of our findings to real world practice settings, which deal with complex families and constellations of presenting issues (e.g., families with maltreatment histories/DSS involvement). That said, BPT programs may require adaptations to effectively deal with specific populations, including those with maltreatment histories and others (McMahon & Forehand, 2003). Given the pilot nature of our study, it was a necessity that we focus more narrowly on DBDs as the primary presenting issue and the standard HNC program.

This pilot study also has several strengths. First, this is the first study to our knowledge to examine the use of smartphone-enhancements to improve the engagement and treatment outcomes of low-income families in BPT. Given that low-income families are among the most difficult to engage in BPT, smartphone-enhancements may show promise across the broader socioeconomic spectrum as well. In addition, this study utilized the gold standard for evaluating treatment-efficacy, a randomized control trial design. As such, we now have the research infrastructure and promising findings to guide a future, sufficiently powered trial. Third, in contrast to typical practice, the smartphone-enhancements developed in this study represent the collaboration of researchers with expertise in BPT and underserved families, practicing clinicians with training in BPT, economists with expertise in tracking and calculating intervention costs, and an industry partner with experience in developing sustainable technological applications. Even a cursory review of BPT-relevant “applications” (e.g., a time-out timer) suggests that typical developers of service-related application likely have little understanding of BPT principles or the real challenges facing families (i.e., tracking elapsed time is unlikely to be the most challenging issue).

Fourth, although the integration of technology into services research has proceeded largely without theory (see Riley et al., 2011; Ritterband, Thorndike, Cox, Kovatchev, & Gonder-Frederick, 2009, for reviews), our aims and hypotheses were firmly grounded in a theoretical framework. Specifically, we posited, and findings supported, that the portability, availability, and range of applications bundled into the smartphone had the potential to boost the reach of BPT program materials and skills to families between sessions, even in the face of the financial strain and stressors that compromise engagement in low-income famiies (see Aguilera et al., 2012; Anderson & Subramanyam, 2011; Davies, 2011; Jones et al., 2013; Snider, 2011; Williams et al., 2007, for reviews). Importantly, however, the functions (i.e., skill modeling, assessment of practice, session reminders etc.) of the smartphone-enhancements developed for this project could be delivered via a range of technologies available to therapists and families and can also be adapted as technology innovations evolve (see Riley et al., 2011, for a review of the importance of focusing on the function of technology as the field evolves).

Finally, HNC is but one of several theoretically and practically similar BPT programs evolving from the Hanf Model (Reitman & McMahon, 2012). Practicality and cost-effectiveness of technology depends in large part on the generalizability of approaches tested from one BPT program for DBDs to another. As such, we believe that the technology-enhancements tested with HNC in this study could be used similarly with other Hanf-based programs with potential implications for the broader field of children’s mental health as well (see Jones et al., 2013, for review).

Before concluding, it is important to acknowledge the potential ethical issues associated with the intersection of technology and BPT, issues that are relevant for the broader field of children’s mental health as well. As with other aspects of the use of technology in services research, the field is progressing far more quickly than advances in relevant ethical guidelines (Novotney, 2011; also see Jones et al., 2013; Reed, McLaughlin, & Milholland, 2000; Richardson et al., 2009). In fact, leaders in the field highlight that “the tail is wagging the dog in some ways on this issue” (Novotney, 2011, p. 40), as advances in technology far outpace the rate at which practice guidelines are updated to deal with new challenges. Some of the potential ethical issues related to any telehealth approach include standard-of-care (e.g., emergency protocols when a client is not physically in the same room as the provider or there is no “provider”), privacy and security (e.g., use of secure networks), and feasibility (e.g., training both therapists and clients in the use of technology). The emerging field of technologyenhanced BPT is certainly not immune from any of these ethical issues (see Jones et al., 2013, for a review). For example, in our study we had standards in place to ensure that access to smartphones was protected by programming only strong passwords, that survey data was de-identified and linked only to a numerical code associated with the family, and that, if lost or misplaced, smartphones could be remotely “bricked” or deactivated to ensure protection of all family data, including identifiable information (e.g.,, home practice videos). Of note, none of our families reported their smartphones lost or misplaced during the course of the project, even temporarily, and all families returned their smartphones to the project staff at the end of the project. That said, we as a field must continue to give careful attention and discussion to how such issues will be handled as we consider deploying smartphone-enhanced and other technology-enhanced interventions beyond resource-intensive and carefully controlled university training clinics to real-world practice settings.

In conclusion, TE-HNC shows promise as an innovative approach to boosting BPT gains for low-income families. Future research with sufficient power is planned to examine whether the trends in this study are replicated and, if so, whether they reach statistical significance. Given our reliance on industry statistics for data regarding smartphone ownership, our goal is also to continue to collect data on emergent trends among low-income consumers in particular to ensure that our approach is one that remains viable. In the meantime, it is our hope that our pilot work will motivate collective efforts across the field to capitalize on advances in technology to best meet the needs of families.