Optimizing Engagement in Behavioral Parent Training: Progress Toward a Technology-Enhanced Treatment Model

Abstract

Low-income families are more likely to have a child with an early-onset Behavior Disorder (BD); yet, socioeconomic strain challenges engagement in Behavioral Parent Training (BPT). This study follows a promising pilot to further examine the potential to cost-effectively improve low-income families’ engagement in and the efficiency of BPT. Low-income families were randomized to (a) Helping the Noncompliant Child (HNC; McMahon & Forehand, 2003), a weekly, mastery-based BPT program that includes both the parent and child or (b) Technology-Enhanced HNC (TE-HNC), which includes all of the standard HNC components plus a parent mobile application and therapist web portal that provide between-session monitoring, modeling, and coaching of parent skill use with the goal of improved engagement in the context of financial strain. Relative to HNC, TE-HNC families had greater homework compliance and mid-week call participation. TE-HNC completers also required fewer weeks to achieve skill mastery and, in turn, to complete treatment than those in HNC without compromising parent satisfaction with treatment; yet, session attendance and completion were not different between groups. Future directions and clinical implications are discussed.

Engagement, generally considered reflective of an individual’s or family’s commitment to treatment, is a challenge in the delivery of evidence-based mental health care to children (see Becker et al., 2018, for a review). Defined by a range of constructs (e.g., session attendance, homework compliance, treatment completion), engagement has consistently been linked to treatment outcome; therefore, optimizing engagement is critical if we are to realize the full clinical potential of our evidence base (Becker et al., 2018; Chacko et al., 2016). One prime example of such engagement challenges is evident in the treatment for young children (3 to 8 years old) with early-onset behavior disorders (BDs), including attention-deficit/hyperactivity disorder, oppositional-defiant disorder, and conduct disorder (see Jones et al., 2013, for a review). BDs are the second-leading cause of child mental health referrals worldwide (Merikangas, Nakamura, & Kessler, 2009), affecting an estimated 113 million youths (Polanczyk, Salum, Sugaya, Caye, & Rohde, 2015). If left untreated, early-onset BDs predict later antisocial behavior, substance use, underachievement, employment instability, and chronic illness (e.g., Fergusson, John Horwood, & Ridder, 2005; Odgers et al., 2007; Piquero et al., 2016). Early identification and intervention can thus save $2.6 to $4.4 million per high-risk child (Cohen & Piquero, 2009); yet, poor engagement in evidence-based early-intervention programming limits effectiveness and outcomes (e.g., Gardner et al., 2009; Jones et al., 2013; Shaw & Taraban, 2016). Importantly, the evidence base for Behavioral Parent Training (BPT), the standard of care for early-onset BDs, has been called robust and the efficacy seems to be equivalent regardless of socioeconomic status, at least at posttreatment and particularly when the behavior is clinically significant (see Leijten, Raaijmakers, deCastro, & Matthys, 2013, for a meta-analysis). Attrition rates hovering at 50%, however, suggest that engagement in and completion of BPT is difficult for all families and even more so for low-income families (see Chacko, Jensen, et al., 2016; Gardner et al., 2009; Lundahl, Risser, & Lovejoy, 2006; Reyno & McGrath, 2006; Shaw & Taraban, 2016, for reviews).

Consistent with family stress theory (see Conger & Donnellan, 2007, for a reviews) and the poverty-related stress model (Wadsworth, Raviv, Santiago, & Etter, 2011), chronic socioeconomic strain increases low-income families’ vulnerability to the coercive cycle of parent–child interaction implicated in the development and exacerbation of early-onset BDs (e.g., Jones et al., 2013; McMahon & Forehand, 2003; Shaw & Taraban, 2016). Financial strain and associated stressors (e.g., illness, shift work, lack of child care) also make it difficult for low-income families to effectively engage in time-intensive BPT services (e.g., 12 to 28 session hours, mid-week check-ins, daily home practice; Eyberg, Nelson, & Boggs, 2008; McMahon & Forehand, 2003; Reyno & McGrath, 2006). Consideration of promising advances in the engagement literature more generally suggests that improving BPT engagement among low-income families in particular likely requires support for parental engagement both in- and out-of-session and in the context of those stressors (Chacko, Wymbs, Chimiklis, Wymbs, & Pelham, 2012; Miller & Prinz, 2003; Nock & Kazdin, 2005; Williams, Lynch, & Glasgow, 2007). One possible strategy to accomplish this goal capitalizes on the potential for technology-enhanced treatment models, particularly those that rely on mobile phones, to increase the reach and impact of services to low-income families (e.g., Georgeson, Highlander, Loiselle, Zachary, & Jones, 2020; Jones et al., 2013; Lindhiem, Bennett, Rosen, & Silk, 2015).

Although terminology continues to evolve, telemental health broadly refers to the use of technology to increase the reach, use, and impact of evidence-based mental health services. To this end, technology has been firmly rooted in the long history of BPT in particular, with efforts ranging from the relatively early and basic (e.g., videotape modeling) to those more recent and sophisticated (e.g., online therapist training, remote coaching, internet-delivered sessions) (e.g., Chacko, Isham, Cleek, et al., 2016; Comer et al., 2015; Ortiz, Vidair, Acri, Chacko, & Kobak, 2020; also see Jones et al., 2013, for a review). While proposing a technology-enhanced treatment model for low-income families may seem somewhat counterintuitive in light of the challenges linked to the digital divide more generally, low-income homes are equally if not more likely to “cut-the-cord” on landlines than other SES groups, relying instead on mobile phones alone as the primary and often only digital device in the home (e.g., Blumberg & Luke, 2018; Pew Research Center, 2019; Vangeepuram et al., 2018). This trend makes practical sense, given that mobile phones can cost-effectively increase low-income parents’ ability to access and use a range of features and functionality (e.g., text/email, internet/web, electronic/shared calendars) essential to parenting young children in the 21^st century. Building upon these trends, a prototype of a technology-enhanced treatment model was developed using one example, Helping the Noncompliant Child (HNC; McMahon & Forehand, 2003), of the Hanf Model or a family of BPT programs collectively characterized by common theory and practice element targeting the coercive cycle of parent-child interaction implicated in the etiology and maintenance of early-onset BDs (see Kaehler, Jacobs, & Jones, 2016; McMahon & Forehand, 2003; Reitman & McMahon, 2012, for reviews). Although there are also group-based Hanf Model programs, HNC is one example of an individual, family-focused program characterized by weekly sessions, mid-week telephone check-ins to assess skill practice and progress, and the assignment of daily skill practice and use. Although mixed, some data suggest that individual family-focused BPT programs may be a better fit when the goal is improved engagement in particular (Chacko, Jensen, et al., 2016), including for low-income families who may evidence better outcomes with family-focused rather than group-based programs (see Lundahl et al., 2006 for a meta-analysis). Hypotheses for the potential benefit of family-focused BPT programs include the opportunity for in vivo skill practice with the parent and child and thus providing treatment tailored to the particular presenting needs of the family (see Kaminski, Valle, Filene, & Boyle, 2008 for a meta-analysis). For example, HNC is a mastery-based BPT program for which progression with skills (i.e., Attends, Rewards, Ignoring, Clear Instructions, Time-Out), Phases (I. Differential Attention, II. Compliance Training) and, ultimately, program completion is dependent upon clinician coding of parent use and mastery of skills, as well as the child’s response to those skills (e.g., Time-Out mastery criterion includes ratio of parent Clear Instructions and child compliance). HNC materials are also written at a sixth-grade reading level, making it accessible for low-income families who are more likely than relatively higher income parents to have literacy challenges (McMahon & Forehand, 2003).

Results from a preliminary randomized control trial suggested that the Technology-Enhanced HNC (TE-HNC) prototype held promise as a cost-effective approach to increasing engagement in (i.e., session attendance, mid-week call participation, homework compliance) and efficiency of (i.e., fewer sessions to meet mastery criteria) services relative to HNC (Jones et al., 2014). TE-HNC did not increase the likelihood that low-income families would complete treatment in the pilot; however, the relatively low dropout rate in both groups (n = 2 TE-HNC; n = 2 HNC) suggested the need for further inquiry. This study aimed to replicate and extend that pilot work by designing and testing an integrated TE-HNC parent mobile application and therapist web-portal in a larger RCT. It was predicted that the TE-HNC group would have higher levels of engagement relative to HNC and that those gains would be achieved with greater efficiency of service delivery without significantly increased implementation costs. Given documented clinicians’ concerns regarding technology compromising treatment process (Anton and Jones, 2017, Anton and Jones, 2019), we also compared parent satisfaction between the two groups.

Methods

participants

English-speaking, low-income [< 250% of Federal Poverty Guidelines (FPL)] legal guardians and their 3- to 8-year-old children with clinically significant problem behavior (Eyberg Child Behavior Inventory Problem > 15 or Intensity > 131; Eyberg & Pincus, 1999) participated (see Figure 1 ). Although various measures of socioeconomic status have strengths and limitations (see Jones et al., 2016, for a review), FPL (i.e., number of people in home or family/annual income) is generally used to determine qualification for federal, state, and local benefits, programs, and subsidies. Forty-nine states cover children with incomes up to at least 200% of the FPL (3 people in home/annual income) through Medicaid and the Children’s Health Insurance Program (CHIP) and 19 of these (including North Carolina) cover children with incomes < 300% FPL. Given that our focus was reaching families in our area who otherwise have difficulty accessing and/or engaging in mental health services, we used incomes < 250% of the FPL. Families were recruited via (a) advertisements targeting areas, work places, retail outlets, and social media with an overrepresentation of low-income parents; (b) healthcare, social service, and other agencies that serve low-income families; (c) local schools; and (d) word-of-mouth (see Khavjou, Turner, & Jones, 2018; Khavjou et al., in press, for details).

Figure 1.

CONSORT diagram describing participant flow in RCT. HNC = Helping the Noncompliant Child, TE-HNC = Technology-Enhanced Helping the Noncompliant Child

Given clinical concerns inherent in a treatment approach in which a child’s symptoms may worsen before they improve (i.e., extinction burst), families were excluded if the parent had a current mood, psychotic, and/or alcohol or drug abuse/dependence (excludes marijuana use) disorder that should be the primary or more imminent focus of services. In addition, parents who had a current pending and/or prior substantiation of child abuse/neglect were excluded in order to optimize the safety of the child without substantively modifying the treatment manual in the context of research. Finally, families were excluded if the child had significant developmental and/or physical impairment that prohibited use of HNC (e.g., unable to hear parent’s verbal use of skills or physically unable to do Time-Out).

procedures

Interested families contacted study staff who conducted a brief phone screen to determine initial eligibility (e.g., 3- to 8-year-old child, low-income, clinically significant behavior). Phone-screen eligible families were then scheduled for a more extensive baseline assessment at a community-based clinic to obtain parent consent for self and child, to confirm eligibility criteria, and to gather more detailed demographic and psychosocial information. Baseline-eligible families were then randomized to HNC (McMahon & Forehand, 2003) or TE-HNC (Jones, Forehand, McKee, Cuellar, & Kincaid, 2010; Jones et al., 2014) and, thus, considered enrolled at the first session. Postassessment procedures were similar to baseline. Families were paid $50 per assessment. All procedures were approved by the university’s institutional review board.

intervention

All families received HNC (McMahon & Forehand, 2003), which is a therapist-delivered, mastery-based (i.e., weekly observation and coding of skill use determines progression through skills and completion of program) program that includes weekly sessions and brief mid-week telephone check-ins to assess progress and problem solve obstacles to home-based skill practice. In Phase I, Differential Attention, the parent is taught to: (a) Increase the frequency and range of positive attention (i.e., Attends, Rewards); (b) Eliminate instructions, questions, and criticism; and (c) Ignore minor inappropriate behavior (i.e., Ignoring). Parents practice these skills in the context of child- directed play (i.e., Child’s Game), which they are also instructed to do at home for at least 15 minutes per day, as well as to use the skills throughout the day. In Phase II, Compliance Training, parents are taught the Clear Instruction sequence in order to maximize child compliance, as well as a nonphysical consequence (i.e., Time-Out) for noncompliance, as well as safety-related behaviors (e.g., aggression). Phase II skills are taught in the context of Parent’s Game (i.e., clean-up task); however, parents are instructed to continue to practice Child’s Game at home to maintain mastery of Phase I skills.

In addition to the standard HNC format and content, TE-HNC families also had access to a HIPAA-compliant, interactive system that allowed therapists (via web-portal) to monitor and tailor parent activity on the mobile application (Tantrum Tamers©), as well as the focus and pace of subsequent mid-week calls and sessions. Building upon the prototype functionality and content tested in the pilot study (Jones et al., 2010, Jones et al., 2014), TE-HNC components included: (a) daily surveys of skills practice, which guided mid-week calls and sessions; (b) weekly video-recorded home practice, which afforded an opportunity for personalized feedback regarding skill development; (c) daily text reminders (e.g., skill practice, appointments), as well as reinforcing messages regarding progress; (d) video calls with the family midweek to problem solve obstacles to skill practice and progress; and (e) skills videos series to model new parenting skills and share with other parents and/or coparents. In addition, based on recommendations of pilot families, a homework checklist was added to remind parents of daily and weekly assignments (e.g., assigned handouts, skill practice, days/times sessions).

therapist training, fidelity, and competence

Master’s-level therapists were trained in and treated families in both the HNC and TE-HNC. Training included establishing knowledge of the relevant manuals, reliability with the HNC mastery coding criteria, role-play and session observation and discussion, as well as weekly observation, supervision, and feedback by two licensed clinical psychologists. Approximately one-quarter (24%) of sessions were coded for therapist fidelity to the treatment manual by one M.A.-level coder (97% fidelity) and 72% of those double-coded for reliability between the two coders (90% reliability). In addition, 35% of the sessions were coded for competence by at least one Ph.D.-level coder and of those 22% by a second Ph.D.-level coder, yielding an average competence rating of 97%.

measures

Session Attendance

Therapists recorded family session attendance weekly. Session attendance was calculated by the ratio of sessions that families attended to the total number of sessions as initially scheduled. For example, if a family attended 12 of 15 weekly sessions as initially scheduled due to cancellations, no shows, or reschedules, they would have 80% session attendance. Session attendance was averaged for families in each group.

Mid-Week Call Participation

Therapists recorded family mid-week call participation weekly. Mid-week call participation was calculated by the ratio of mid-week calls in which families participated to the total number of mid-week calls as initially scheduled. For example, if a family participated in five of the 10 mid-week calls as initially scheduled, again due to missed calls or rescheduling, they would have 50% mid-week call participation. Mid-week call participation was averaged for families in each group.

Homework Compliance

Homework is a common practice element in children’s mental health generally; yet, there are well-documented challenges to relying on therapist-report of client’s homework completion (e.g., Busch, Ubelacker, Kalibatseva, & Miller, 2010; Detweiler & Whisman, 1999; Primakoff, Epstein, & Covi, 1986). Therefore, we developed an observer-based coding system to characterize parent-responses to therapist prompts, explicit in the therapist guide, regarding if and how often parents reported: (a) daily practice of Child’s Game and (b) daily use of skills throughout the day. Building on prior work demonstrating that coding a subset of sessions affords a good proxy for overall homework completion (Busch et al., 2010), we selected 4 sessions per family for observers to watch and code: (a) first Phase I skills-focused session (i.e., Attends); (b) first Rewards session (2^nd skill families learn in HNC program); (c) first Phase II session (i.e., Path A, Clear Instructions, the first phase in the Clear Instruction Sequence); and (d) the last session for which homework is assigned [i.e., Paths A (Clear Instruction), B (Warning), and C (Time-Out) of Clear Instruction Sequence]. The coding team, which was trained and supervised by a master’s-level graduate student, used a 3-point scale: 0 = parent did not do homework, 1 = parent did homework, but less than half of the possible days, and 2 = parent did homework at least half of the possible days. Sessions were initially reviewed as a group to allow coders to practice identifying where in the session the discussion of homework typically occurred (i.e., first 15 minutes), as well as to practice using the coding system in response to therapist check-in regarding home practice (e.g., “Were you able to practice Child’s Game this week?” If yes, “how often?”) until reliability with the master coder was consistently achieved. Then, selected sessions were coded by a single coder. Twenty-five percent of coded sessions were coded by second person to assess inter-rater reliability (83% observer agreement). Any code deviations were reviewed and reconciled by a third coder and/or consensus of the group. Homework completion was averaged for families by group.

Efficiency and Program Costs

Service efficiency was measured using the number of weeks and sessions required for each family to complete the program. Of note, only weeks in which no session occurred due to client reasons (e.g., cancellation, no show, unavailable), rather than therapist reasons (e.g., therapist sick) were included in analyses. Program costs were quantified using a payer approach. Data on program costs were collected using Excel-based cost instruments that therapists and program staff used to report nonlabor resources and time spent on various program activities (Honeycutt, Khavjou, Jones, Cuellar, & Forehand, 2015). Costs were estimated separately for (a) research-specific programmatic activities, including participant recruitment, phone screens, and assessments, as well as therapist supervision, and (b) delivery of the program to participants. Our decision regarding therapist supervision stems from acknowledgement that the clinical supervision offered in efficacy trials such as this one (e.g., weekly, observation and discussion of session, administration) can differ widely from that offered in community practices settings (see Bearman et al., 2013, for a review). For example, results from Accurso, Taylor, and Garland’ (2011) study of clinics serving youth with behavior disorders suggest that supervisees report discussing evidence-based practice elements in supervision thoroughly only rarely (4.5% of sessions) and not at all in nearly half (42.5%) of sessions. Therapist and supervisor time were valued using the 2017 median national hourly wage for marriage and family therapists and psychologists, respectively, from the Bureau of Labor Statistics. Nonlabor costs were valued based on the payments made for program materials and supplies and were expressed in 2017 dollars. Research-specific programmatic costs were estimated per family enrolled as total costs divided by the number of enrolled families. In the case of supervision costs, for example, cost was estimated based on the 1 hour per week that each therapist was delivering the intervention and divided by the total number of families enrolled in order to estimate a per family supervision cost rather than actual cost per family or by group. Program delivery costs were estimated per family who completed the program. Therapist time spent on delivering the program captured the time spent both during and outside of face-to-face sessions, including both typical clinical related activities (e.g., phone calls, paperwork), as well as technology-specific costs (e.g., watching home practice videos, reviewing survey responses). Phone-related costs, such as phone cases, screen protectors, and monthly phone bills, were also tracked for TE-HNC participants because these were provided to families that participated in the study. Even though research-specific and phone-related costs are not expected to be incurred during the implementation of this intervention in community-based practice settings, these costs are important to consider to better understand the resources required to deliver a program in various settings.

Consumer Satisfaction

Both TE-HNC and HNC families completed the HNC Consumer Satisfaction Scale (McMahon & Forehand, 2003) to measure acceptability of the intervention and investigate differences between treatment groups. A total satisfaction score for the overall program, reported difficulty of each skill and overall program, and reported usefulness of each skill and overall program was calculated for each group (α = .86). The TE-HNC group was given an additional satisfaction questionnaire that asked them to rate (7-point scale) how easy to use and useful each smartphone component (e.g., daily surveys, skills videos, reminders) was in their treatment. Higher scores on each item indicate the component was considered relatively easier or more useful.

data analytic approach

Enrollment and baseline characteristics of the sample were examined to assess the integrity of randomization and ensure balanced characteristics in each group using categorical chi-square and parametric t-tests. We report dropouts by group; however, it made most conceptual sense to us to examine engagement as a function of families receiving TE-HNC or HNC (i.e., per protocol approach), rather than engagement as a function of assignment to group (i.e., intent-to-treat analyses) (Shrier, Verhagen, & Stovitz, 2017). Accordingly, we excluded anyone who did not or could not adhere to the treatment protocol (i.e., families with phone or software issues for at least half of treatment) or failed to complete treatment (i.e., dropout) after randomization in the protocol approach.1

Next, between-group effect sizes (0.20 – 0.49 small, 0.50 – 0.79 medium, and > 0.80 large; Cohen, 1988), as well as more traditional significance testing using independent samples t-tests to compare group mean differences between TE-HNC and HNC on various markers of engagement (Cohen, 1988), were utilized. Whereas significance testing conveys the likelihood that study results differ from chance expectations, effect-size calculations convey the relative magnitude of the experimental effect and, in turn, provide the opportunity to compare the magnitude of treatment effects within and across studies (see Thalheimer & Cook, 2002 for a review). Consistent with Cohen’s d and more recent modifications (D’Amico, Neilands, & Zambarano, 2001), the difference between group means (e.g., Engagement Mean TE-HNC – Engagement Mean HNC) was divided by the average of each mean’s standard deviation (e.g., (SD TE-HNC + SD HNC/2)). We assessed statistical significance of differences in costs of delivering HNC and TE-HNC using t-tests.

Results

enrollment and baseline characteristics

Three hundred sixty-three families completed a phone screen (see Figure 1). Of the 155 families who were eligible to continue (i.e., primary reason for ineligibility was low-income criterion), 133 completed the baseline, 120 met full eligibility criteria, and 101 enrolled and were randomized to HNC (n = 54) or TE-HNC (n = 47). Sixteen families in the TE-HNC group experienced significant disruptions with the smartphone application that prevented consistent and/or sustained use for at least half of their time in treatment (e.g., iOS or other Apple updates that were more difficult to predict and/or more disruptive given that for research purposes the app was not released through the traditional App store mechanism); therefore, these families were excluded from these analyses (see Discussion for implications and future considerations). These technology disruptions occurred at random within the sample and were independent of participant activity or demographic characteristics. There were no pretreatment group differences on demographic characteristics or child behavior severity.

dropout and missing data

The overall dropout rate (28%), as well as dropout by group (HNC: 26%; TE-HNC: 32% χ² = 0.039, n.s.), yielded a completer sample of n = 62 (n = 40 HNC; n = 22 TE-HNC). With the exception of two families who dropped out in Phase II of treatment, most families dropped out in Phase I, with no significant difference in number of sessions to dropout between HNC (M = 3.8) and TE-HNC (M = 3.0). The most common reasons given for dropout were stressors related to health (25.9%), family (11.1%), work/transportation (22.2%), and/or housing (22.2%). Furthermore, there were no differences between dropouts and completers on any demographic variables or ECBI scores.

On average (see Table 1 ), parents who completed both the pre- and postassessments were 33.24 years old, most were female (96.8%), a little less than half (42.9%) were ethnic or racial minority, and 50% worked at least part-time. Children were 4.18 years old on average, approximately one-third (38.7%) were girls, and half (54.1%) were ethnic or racial minorities.

Table 1.

Demographic Characteristics of Completers at Pre-Assessment

					Treatment Groups
		Total Sample (N = 62)			TE-HNC (n = 22)			HNC (n = 40)
Measure	Possible Range	%	M	SD	%	M	SD	%	M	SD
Child Demographics
Age (Years)	3-8		4.18	1.03		3.95	0.79		4.30	1.14
Gender (% Female)		38.7			36.4			40.0
Race
White		69.4			77.3			65.0
African American		16.1			9.1			20.0
2 or More Races		12.9			13.6			12.5
Not Reported		1.6			-			2.5
Hispanic/Latino		14.5			9.1			17.5
Parent Demographics
Age (Years)			33.24	5.68		31.86	5.28		34.00	5.81
Gender (% Female)		96.8			95.5			97.5
Race
White		72.6			81.8			67.5
African American		17.7			9.1			22.5
2 or More Races		8.1			9.1			7.5
Not Reported		1.6			-			2.5
Hispanic/Latino		6.5			4.5			7.5
Marital Status
Single		19.4			9.1			25.0
Married/ living as married		62.9			72.7			57.5
Divorced/separated		17.7			18.1			17.5
Employed in Any Capacity		50.0			40.9			55.0

HNC = Helping the Noncompliant Child

TE-HNC = Technology-Enhanced Helping the Noncompliant Child

session attendance, mid-week call participation, and homework compliance

As shown in Table 2 , per protocol analysis of average session attendance revealed that it was relatively high and similar (ES = 0.30) for both HNC (84.7%) and TE-HNC (88.8%). In contrast, TE-HNC families had significantly better mid-week call participation (ES = 1.06; TE-HNC 85.6%; HNC 59.7%) and homework compliance (ES = 0.56; TE-HNC 88.5%; HNC 59.7%) than HNC families.¹

Table 2.

Between-Group Differences on Engagement, Efficiency, and Consumer Satisfaction

Variables	M (SD)	CI (95%)	p	HNC vs. TE-HNC (d)
Engagement
Attendance (%)			n.s.	0.30
HNC	84.7 (16.3)	79.5-89.9
TE-HNC	88.8 (10.7)	84.1-93.5
Mid-Week Call (%)			***	1.06
HNC	59.7 (29.2)	50.4-69.0
TE-HNC	85.6 (18.7)	77.4-93.9
Homework Compliance (%)			*	0.56
HNC	80.0 (16.5)	74.5-85.5
TE-HNC	88.5 (11.5)	83.5-93.5
Efficiency
Number of weeks			*	0.47
HNC	14.15 (6.9)	11.9-16.4
TE-HNC	11.64 (2.9)	10.3-12.9
Number of sessions			n.s.	0.24
HNC	10.03 (2.0)	9.4-10.7
TE-HNC	9.55 (2.0)	8.7-10.4
Consumer Satisfaction
Program Satisfaction			n.s	.043
HNC	62.44 (4.17)	61.03-63.86
TE-HNC	62.65 (5.36)	60.33-64.97
Ease of Skill Use			n.s.	.050
HNC	33.97 (4.03)	32.61-35.34
TE-HNC	34.17 (4.52)	32.22-36.13
Usefulness of Skills			**	.522
HNC	37.06 (2.87)	36.09-38.03
TE-HNC	38.57 (2.92)	37.30-39.68

*p< .05; **p < .01; ***p < .001

HNC = Helping the Noncompliant Child

TE-HNC = Technology-Enhanced Helping the Noncompliant Child

efficiency and program costs

As shown in the Table 2, families in TE-HNC completed the program in fewer weeks (M = 11.63) than HNC (M = 14.15, p ≤ .05). TE-HNC families also completed the program in fewer sessions; however, the difference was not statistically significant. Total research-specific program costs were $1,152 per enrolled family, including $601/family for supervision, and an additional $585 for phone-related costs/family (see Table 3 ). Over and above these research costs, program delivery cost was approximately $300 per family that completed the program and most of that cost was for therapist time. Program delivery costs were similar between HNC and TE-HNC groups.

Table 3.

Research-Specific and Program Delivery Costs

	Cost Per Family (2017)
	HNC	TE-HNC
Research-specific programmatic costs	$1,152
Recruitment	$145
Phone Screens	$87
Assessments	$319
Supervision	$601
Phone-related costs	$0	$585
Program delivery costs	$293	$312
Non-labor	$18
Labor	$275	$294
Orientation	$26	$33
Skill 1 -Attends	$57	$65
Skill 2 - Rewards	$35	$36
Skill 3 -Ignoring	$38	$39
Skill 4 -Clear Instructions	$68	$67
Skill 5 - Time Out	$51	$54

HNC = Helping the Noncompliant Child

TE-HNC = Technology-Enhanced Helping the Noncompliant Child

consumer satisfaction

There were no significant differences in program satisfaction or ease of use of skills (see Table 2). Patterns suggest that parents in TE-HNC rated the usefulness of the overall program higher than HNC (ES = .052). Components of TE-HNC were also all rated as “easy/very easy to use” and “useful/very useful,” except daily reminders, which were rated as “neutral” to “somewhat useful” (see Table 4 ).

Table 4.

Mobile Application Component Ease of Use and Usefulness

Smartphone Application Component	Median Response	Qualitative Description
Daily Surveys
Ease of use	7.0	Very Easy
Usefulness	6.0	Useful
Skills Videos
Ease of use	7.0	Very Easy
Usefulness	6.0	Useful
Sharing Skills Videos with Coparent
Ease of use	7.0	Very Easy
Usefulness	7.0	Very Useful
Daily Reminders
Ease of use	6.0	Easy
Usefulness	4.5	Neutral to Somewhat Useful
Recording Home Practice
Ease of use	7.0	Very Easy
Usefulness	6.0	Useful
Convenience of App	7.0	Very Convenient

Range of response 1-7 with 7 representing easier and more useful. Given that scores were positively skewed, medians are presented rather than means.

Discussion

This study aimed to improve BPT engagement and efficiency without increasing intervention costs or compromising consumer satisfaction. Similar to the pilot, families in TE-HNC were more likely to do their daily practice of Child’s Game, as well as to use their skills throughout the day, than those in HNC. Research on homework process and implementation in general (Kazantzis, Whittington, & Dattilio, 2010), as well as well as on BPT homework in particular (e.g., Chacko et al., 2009; Chacko, Anderson, Wymbs, & Wymbs, 2013; Nock & Kazdin, 2005), suggests that the primary reasons parents fail to complete homework are less specific to the therapist or the session and more related to remembering to do it and then finding the time and overcoming obstacles to doing it. Thus, consistent with recommendations by others, the technology-enhancements available to TE-HNC parents (e.g., homework reminders, daily surveys of skill use and practice, skills modeling videos) may have increased the salience of and support for parent homework completion in the context of families ‘everyday lives (Chacko et al., 2013).

Also consistent with the pilot, TE-HNC improved participation in the mid-week call. While less standard across BPT programs (Kaehler et al., 2016), HNC’s mid-week call assesses and problem solves obstacles to home practice in order to maximize parent confidence and competence between sessions toward better skill mastery and progress. Indeed, HNC’s mid-week call serves a similar function as “coaching calls,” which, for example, are now a standard part of other evidence-based approaches, including Dialectical Behavior Therapy (DBT; Oliveira & Rizvi, 2018). Thus, increasing the probability that families complete mid-week calls allows for the possibility of more personalized weekly sessions and overall treatment model (McMahon & Forehand, 2003).

TE-HNC did not, however, improve session attendance or prevent dropout. Although our dropout was substantially lower than in prior BPT work with low-income samples (e.g., 56%; Fernandez, Butler, & Eyberg, 2011), it was consistent with the broader literature that estimates that a quarter (26%) of families begin, but dropout of BPT services (see Chacko, Jensen, et al., 2016, for a review). This is perhaps particularly important given changes in the significance of our findings when we used raw counts to examine engagement in analyses that included noncompleters (i.e., ITT) instead of the primary analyses of percentages with completers only (i.e., per protocol).¹ Although we detail our rationale for the per protocol approach, the pattern of findings between the two sets of analyses suggest that the added boost for TE-HNC relative to HNC for mid-week calls and homework may be more pronounced in analyses that capture the variable number of sessions inherent in a mastery-based program like HNC and/or when focusing only on those who complete treatment (i.e., TE-HNC helps those who do not complete less). The latter is important to consider as it reaffirms the point that we need to continue to better understand the precipitants of dropout in BPT in general and if and how technology can better prevent dropout for low-income families in particular (e.g., Chacko, Wymbs, Rajwan, Wymbs, & Feirsen, 2017; Kazdin & Whitley, 2003; Nock & Kazdin, 2005). One possibility to further consider on this front is that while TE-HNC was designed to motivate and support skill practice and progress in the context of stressors so common among low-income families, neither functionality nor content explicitly addresses those stressors or associated parental distress. This seems particularly important given that the majority of dropout families identified at least one stressor as the precipitant for their decision, which is consistent with work by Chorpita and colleagues (e.g., Chorpita, Korathu-Larson, Knowles, & Guan, 2014; Guan et al., 2017; Guan, Park, & Chorpita, 2019) highlighting the correlation between unexpected stressors more likely to occur among low-income families and disruption of treatment process and outcome. In addition, findings by Wadsworth et al. (2011; i.e., poverty-related stress model) and Matthews and colleagues (2011; i.e., reserve capacity model) suggest that chronic socioeconomic strain not only increases the severity and frequency of stressors with which low-income families must cope, but also depletes reserves necessary to cope with those stressors. This study was completed prior to the COVID-19 pandemic; however, COVID-19 provides a context for precisely this point, given that lower-income families have been disproportionality affected by both the health and economic impacts of the virus. Thus, future work targeting service completion must continue to explore how technology can play a more central and perhaps tailored role as low-income parents continue to navigate how best to seek and engage in mental health services for their children while also coping the emergence of often acute, unpredictable, and even devastating stressors.

Although TE-HNC did not improve rates of completion relative to HNC, TE-HNC completers mastered program skills in fewer weeks and sessions, although the latter was not statistically significant. One way to conceptualize this is that TE-HNC did not facilitate skill mastery per se (i.e., number of sessions per skill determined by parent skill mastery) but instead reduced the overall amount of time (i.e., weeks) required for families to progress through the skills/sessions. This suggests that if we can keep families coming consistently each week then TE-HNC may allow them to complete services more efficiently. Moreover, TE-HNC efficiency did not come at the cost of consumer satisfaction, which we know is a worry for therapists as technology increasingly intersects with mental health delivery (Anton and Jones, 2017, Anton and Jones, 2019). Efficiency comes without significant added implementation costs as well, suggesting the promise of a cost-effective approach.

As with all research this project has limitations. First, we provided all families with phones and service plans, given data to suggest that 44% of low-income users let service plans lapse due to finances (Pew Research Center, 2015). While families with a mobile phone may have found carrying two phones inconvenient, it also did not prevent a sizeable portion of families from having technical issues. This affected our sample size, analyses, and interpretations, but also reflects the reality of consumer uptake of technology in general, and telemental health will not be an exception. Second, costs limited us to choosing one platform and we chose the iPhone as at the time iOS allowed us to develop more functionality; however, price points suggest that low-income families are more likely to choose an Android phone. Third, mobile technologies have the potential to boost a range of psychotherapy outcomes whether as a technology-enhancement or standalone approach (see Lindhiem et al., 2015 for a review); however, there continues to be discussion regarding the potential need for a technology-enhanced model in particular with more vulnerable populations and/or for clinical intervention rather than prevention (e.g., Mohr, Burns, Schueller, Clarke, & Klinkman, 2013; Tate & Zabinski, 2004). That said, technology-enhanced models like the one tested here still require weekly, clinic-based session, which our findings further highlight are difficult for low-income families to continue. Fourth, our findings suggest that TE-HNC requires minimal additional time outside of session for therapists relative to HNC; yet, further work must continue to examine uptake of and satisfaction with technology-enhanced treatment models, particularly in publicly funded clinics where low-income families are more likely to seek children’s mental health services (Anton and Jones, 2017, Anton and Jones, 2019). Fifth, we conceptualized supervision as a research cost and estimated time per family, rather than actual time by family or group. We did this given discussion in the literature regarding the lack of consistency between the more intensive nature and clarified focus of supervision in clinical trials relative to what happens or may be feasible in community mental health settings. That said, that decision may be less relevant for front-line service settings that do provide more intensive supervision and would be interested in actual costs per family and by group. A final limitation, unrelated to technology, is our decision to exclude parents with current psychopathology, as well as to rely on study-specific rather than community-based therapists, which may limit generalizability (see Weisz, Krumholz, Santucci, Thomassin, & Ng, 2015, for a review).

This study also has strengths. First, there has been significant investment in technology-enhanced service delivery models. For example, the National Institute of Mental Health reports having awarded 404 grants totaling $445 million for technology-enhanced mental health interventions between fiscal years 2009 and 2015 (National Institute of Mental Health, 2017). While improving engagement has been cited as a primary rationale, few studies in child or adolescent mental health in general or BPT in particular assess or report whether engagement improves as a function of a technology-enhanced delivery model (Georgeson et al., 2020). Second, we focused on low-income families who are more likely to have a child with an early-onset BD and less likely to engage in mental health services; yet, it is important to note that we do not have a theory-driven reason to expect TE-HNC to work differently for higher income families. Third, like most (if not all) evidence-based treatments for children, homework compliance is critical to skill generalization in BPT; yet, rates of homework compliance (and other out-of-session markers) are less commonly reported in the literature than other engagement measures (e.g., Chacko, Jensen, et al., 2016; Chorpita, Daleiden, & Weisz, 2005; McMahon & Forehand, 2003). We look at both in- and out-of-session indicators of engagement in this study. Finally, given that HNC is one example of a family of evidence-based BPT programs with common history, theory, and practice elements, findings may be generalizable to other programs as well (see Kaehler et al., 2016; Reitman & McMahon, 2012; Southam-Gerow & Prinstein, 2014, for reviews).

In summary, TE-HNC contributes to a growing literature exploring the potential for technology to increase engagement in children’s mental health (see Georgeson et al., 2020; Jones et al., 2013, for review; also see Chacko, Isham, et al., 2016, for promising pilot work in this area). Additional work is needed to further understand how to continue to optimize session attendance and program completion in particular, which is critical if all children, including low-income children, are to benefit fully. In addition, our work and that of others must continue to explore if, how, and when improved engagement is linked to improved clinical outcomes. In the case of mastery-based programs like HNC, we have increased confidence that parents’ skill use is improved and children’s problem behavior reduced as a function of reliance on the mastery-criterion to determine progress in and completion of the program. Yet, prior pre-to-post pilot analyses has shown technology-enhanced approaches can bolster outcomes even relative to standard BPT (Jones et al., 2014). It will also be important to examine variability in the extent to which TE-HNC parents were motivated to engage with the technology-enhancements in order to determine if greater or more consistent use is linked to improved outcomes (Anton et al., 2016; Dewar, Bull, Malvey, & Szalma, 2017). Importantly, these latter two directions, symptom reduction and technology use, can most richly be examined using time-intensive data analyses, which is increasingly a hallmark of the technology-enhanced treatment literature (see Baraldi, Wurpts, MacKinnon, & Lockhart, 2014, for a review). Finally, others have highlighted the difficulties inherent in inconsistent definitions of engagement throughout the literature (Becker et al., 2018; Chacko, Jensen, et al., 2016). We chose to focus this examination of engagement on families who enroll in treatment, rather than those who are eligible but never enroll (Chacko, Jensen, et al., 2016). We believe that both directions are critically important, albeit slightly different in terms of the feasibility of clinician’s time and resources in community settings. We developed our technology-enhancements with the goal of not substantively increasing therapist out-of-session activity for current cases. It is more difficult to consider how best to do this for cases that have perhaps had a general intake with an agency, but never initially met with the therapist or started BPT. That said, data on the implications of children failing to receive treatment highlights the importance of increasing families’ access to and initiation of BPT as well.

Conflict of Interest Statement

The authors declare that there are no conflicts of interest.