A study protocol testing the implementation, efficacy, and cost effectiveness of the ezParent program in pediatric primary care

Abstract

Introduction

Up to 20% of children demonstrate behavior problems that interfere with relationship development and academic achievement. Parent participation in behavioral parent training programs has been shown to decrease child problem behaviors and promote positive parent-child relationships. However, attendance and parent involvement in face-to-face parent training remain low. Testing the implementation, efficacy, and cost of alternative delivery models is needed to (a) increase the reach and sustainability of parent training interventions and (b) address the barriers to parent participation and implementation of such programs, specifically in primary health care settings. The purpose of this paper is to describe the study protocol evaluating the implementation, efficacy, and cost-effectiveness of delivering the tablet-based ezParent program in pediatric primary care sites.

Methods

The implementation of the ezParent in four pediatric primary care sites will be evaluated using a descriptive design and cost-effectiveness analysis. The efficacy of the ezParent will be tested using a randomized controlled trial design with 312 parents of 2 to 5 year old children from pediatric primary care settings. Data on parenting and child behavior outcomes will be obtained from all participants at baseline, and 3, 6, and 12 months post baseline.

Discussion

Integrating and evaluating the implementation of the ezParent in pediatric primary care is an innovative opportunity to promote positive parenting with potential for universal access to the preschool population and for low cost by building on existing infrastructure in pediatric primary care.

1. Introduction

Fifteen to 20% of children demonstrate behavior problems that interfere with relationship development and academic achievement [1–3]. Children who experience an early onset of behavior problems may be at higher risk for persistent behavior problems [4]. Parent participation in parent training (PT) has been shown to decrease child problem behaviors and promote positive parent-child relationships [5,6]. One program, the Chicago Parent Program delivered in a group setting, has demonstrated these positive outcomes [7,8]. However, attendance and parent involvement in face to face PT remain low. Barriers to participation may be one of the most common obstacles to effective treatment.

Parents who face barriers to participation don't receive the full dosage of PT when group sessions are missed, or skill acquisition is less effective when later skills build on earlier skills [9–11]. Indeed, face to face PT studies have reported attendance rates ranging from35%–50% of sessions for those who enroll, and up to one-third who sign up attend no sessions [7,12,13]. Attendance at group sessions is often impeded by tangible challenges such as access to transportation, group meeting times that are difficult for families who work non-traditional work hours, childcare issues, and the additional time for travel to and from group sessions [12,14,15]. In an attempt to increase access to and completion of PT by low income families the Chicago Parent Program group curriculum was adapted to be delivered digitally as a tablet-based application [16]. The ezParent program uses a portable and self-directed tablet-based delivery method to provide families with 24/7 access. The digital and portable delivery allows parents to complete the PT modules at their own pace, in their home, without additional cost of childcare and travel. In preliminary testing with a sample of primarily low income African American and Hispanic mothers of children age 2 to 5 years old (n = 40) recruited from a pediatric primary care clinic, we found that the average ezParent module completion was 85.4%, indicating that parents completed approximately five of the six modules. Further, 65% of parents completed all six modules, the full curriculum [17].

In addition to addressing barriers to parent access to PT, digital delivery has the potential for rapid translation and implementation in usual care settings. Therefore, it is prudent to build in implementation and cost evaluation to efficacy trials to inform efforts for translation into practice. The RE-AIM (Reach, Efficacy, Adoption, Implementation, Maintenance) framework offers a comprehensive structure to evaluate individual and setting level implementation processes [18,19]. The five elements of the RE-AIM framework provide important information that can facilitate our understanding of factors that can facilitate implementation and translation of the ezParent program into practice.

In this paper, we describe the study rationale, design, methods, and protocol for the evaluation of the implementation and cost effectiveness of the ezParent Program in primary care and a randomized controlled trial (RCT) testing the programs efficacy.

2. Materials and methods

2.1. Intervention and control conditions

2.1.1. ezParent program

The ezParent Program is a 6-module, self-administered, tablet-based, parent training application that uses multiple strategies to promote learning in parents (see Table 1 for module topics and for a full description of the development of the program, see Breitenstein et al., [16]). The program is a delivery adaptation of the group-based Chicago Parent Program [16,20] and was developed in collaboration with and managed by Klein Buendel, Inc. a technology company based in Denver, Colorado. Each module includes (a) didactic teaching of parenting skills using a video narrator describing parenting strategies, (b) video vignettes of parents and children as examples of how parenting strategies work, (c) questions following each vignette for parents to reflect upon, (d) interactive activities for parents to complete, and (e)multiple choice and true/false questions to assess parent understanding of the strategies [16] (see Appendix A for screen shots depicting various aspects of the program). At the end of each module, parents receive a practice assignment designed to help parents apply the parenting skills that they learned with their children. Examples of practice assignments that parents are asked to complete include (a) spending 15 min of “child-centered time” with their child, (b) identifying a problem behavior in their child that they are going to ignore, and (c) using a logical consequence for their child's misbehavior. Practice assignments are linked to the content of the module.

Table 1.

ezParent module topics.

Unit 1: The value of your attention

Module 1: Child-Centered Time, Routines and Traditions

Module 2: Using Praise, Encouragement, and Rewards

Unit 2: Using your authority wisely

Module 3: Say What You Mean and Mean What You Say

Module 4: Threats, Consequences, Ignore and Distract

Module 5: Discipline strategies and using Time-Outs

Unit 3: Managing your stress

Module 6: Reducing Stress and Problem Solving

Reference Materials: Summary materials: digital handouts, bag of tricks, and ability to return to any module content

The program is completely self-directed, and parents can opt to complete each module in one or multiple sessions. In our previous work, the average time participants spent using the modules was 37.5 min (SD = 22.3 min) [21]. Parents are prompted to complete a practice checklist for each module to assess what practice they completed. Parents are given access to the next module after they review the practice assignment for the previous module. To encourage and support program completion, parents receive virtual badges when they complete modules, practice assignments, and activities. When parents earn a badge, they are alerted in the program with a pop-up message congratulating them on their accomplishment; all earned badges are housed in the “my badges” page.

2.1.1.1. Health-e Kids control condition

The Health-e Kids digital application was developed for parents in the control condition. Health-e Kids includes pdf information sheets, websites, and resources that are regularly provided to parents at the pediatric primary care practices during well child visits for children age 2–5 years old (see Appendix B for screen shots of the Healthy-e Kids application). Health-e Kids includes information regarding child development, common childhood illnesses, vaccinations, health and safety, and nutrition and fitness that align with anticipatory guidance guidelines [22]. Health-e Kids was developed in collaboration with the four primary care practice implementation sites and aligns with commonly used handouts and information that are provided to parents at well-child visits.

2.2. Study aims

The purpose of this study is to evaluate the implementation, efficacy, and cost-effectiveness of the ezParent Program in pediatric primary care practices serving low-income, urban families. The specific aims are to: (1) quantify the levels of program implementation of ezParent in primary care using the RE-AIM framework; (2) test the direct effects of the ezParent Program on parenting outcomes (parenting behavior, self-efficacy, and stress) and child outcomes (child problem and prosocial behavior) compared to the enhanced usual care control condition (Health-e Kids); and (3) compare the cost-effectiveness of ezParent relative to the Health-e Kids control condition for parenting and child outcomes (see Table 2 for Aims, Research Questions, and Hypotheses).

Table 2.

Study aims, research questions, and hypotheses.

Study aim	Research question/hypothesis
1. Quantify the levels of program   implementation of the ezParent in   primary care using the RE-AIM   framework.	Research question: What is the degree of Reach, Efficacy, Adoption, Implementation, and Maintenance of the ezParent Program in pediatric primary care?
2. Test the direct effects of the ezParent   Program on parenting outcomes   (parenting behavior, self-efficacy, and   stress) and child outcomes (child   problem and prosocial behavior)   compared to the enhanced usual care   control condition (Health-e Kids)   among low-income parents with   young children seen in primary care.	Hypothesis 1: Relative to parents in the control condition, parents in the ezParent condition will report greater improvements in parenting and child outcomes at the 3-month assessment. Hypothesis 2: The improvements in parenting outcomes and child outcomes the ezParent condition relative to the control condition will be maintained through the 6- and 12-month assessments.
3. Compare the cost-effectiveness of the ezParent relative to control for parenting and child outcomes.	Hypothesis: The ezParent is cost-effective relative to the control for parenting and child outcomes.

2.3. Study design

This study uses a type 1 hybrid effectiveness – implementation design that will rigorously test the intervention while gathering information and data to evaluate the potential for intervention implementation in “real world” settings [23]. Four pediatric primary care practices serving predominantly low-income, Medicaid recipients are collaborating as implementation and recruitment sites in this study. Well-child visits (e.g., office visits for health prevention and promotion of children and families [24]) are provided by attending and resident physicians or nurse practitioners in the primary care practices. To approximate the process of delivering the ezParent Program to parents during well child visits, the providers introduce the study to parents. Using a descriptive design, the RE-AIM framework guides the evaluation of ezParent implementation in the pediatric primary care sites [19].

To test the efficacy of the ezParent Program, this study uses an experimental design with person-level randomization for parents of 2 to 5 year-olds from the four primary care practices. Parents (n = 312) are randomized to the intervention (ezParent) or control condition (Health-e Kids). Randomization occurs using a random number generator integrated into the baseline survey. To ensure balance between study conditions over time, randomization is completed in blocks of 10 cases (5 cases randomized to each condition). The research assistants and participants are blind to the study condition until it is revealed at the end of the baseline assessment. Data on parenting and child outcomes is obtained from all participants at baseline, 3 months (end of intervention phase), and 6- and 12-months (end of reference phase) post baseline. The reference phase is the time that parents have continued access to the ezParent Program. The Health-e Kids application is designed as a digital resource for usual care at pediatric primary care sites. Therefore, there is no designated intervention period and parents in the control condition have access to Health-e Kids throughout the 12-month study. A descriptive design will be used to evaluate the cost of implementation of the ezParent from the perspective of primary care practice sites, providers, and parents. The study is approved by IRBs at the two institutions where the study is being conducted and recruitment is occurring.

2.4. Sample

2.4.1. Sample size

Power for this study was calculated using Optimal Design software [25]. Our power calculation is based on an effect size of d = 0.31, α_2-tailed=0.05, an intracluster correlation (ICC) of 0.50 across repeated assessments. The effect size was established from analysis of parent outcomes from our pilot study of the ezParent program [17]. Using data from the pilot study, ICCs for outcome variables ranged from 0.65–0.78 over three assessments. We selected ICC = 0.50 as a conservative estimate for this study. Under these assumptions, a sample size of 68 parents (34 interventions; 34 controls) in each of four sites for a total sample size of 272 will provide power of 0.80 to detect differences in the analysis of intervention effects on parent and child outcomes. A previous study of the group-based Chicago Parent Program had a reported attrition rate of 13% over 12 months [7] and our pilot study had an attrition rate of 4% over 6 months [17]. Although attrition in the pilot study was quite low, we conservatively estimated attrition to be 13% over 12 months; therefore, we will recruit a total sample size of 312 at baseline. Pre-implementation activities (2 months) and recruitment (10 months) will be staggered across the four practice sites, with a newsite added every four months. We will recruit approximately 8 parents per month at each site for a total of 78 (39 in each condition) per site for parents over the 10-month recruitment phase in order to achieve the total sample size of 312.

2.4.2. Inclusion criteria

The four implementation practice sites serve between 600 and 4000 children aged two to five years old annually and 50% and 70% of the practice population is designated low-income. Inclusion criteria for parent participants are: (a) the parent, legal guardian, or primary caregiver for the target child, (b) the target child is 2 to 5 years old, (c) the child receives care at the primary care practice site, and (d) the parent can speak and read English. Only one parent per family will be eligible to participate in the study, and parents who have previously used the ezParent program will be excluded.

2.5. Implementation procedures

Parents are introduced to the study during a scheduled well-child visit by their primary care provider at one of the four primary care sites. The purpose of introducing the study at a well-child visit is to integrate positive parenting into the primary care setting and to approximate the process that would occur for providers to offer the ezParent Program to families. In addition, the introduction of the program by trusted primary care providers reassures parents that the ezParent program is a worthwhile pursuit. As part of the pre-implementation plan, the study team has met with the practice site leadership and primary care providers to develop the procedures for introduction of the study to parents at the specific site. These procedures include: (1) a method for reminding providers to introduce the study at all well-child visits for children aged 2 to 5 years, (2) a tracking method for providers to identify who was provided information regarding the study program, and (3) orientation of the providers on how to introduce the study to families.

At the well-child visit, parents receive an information card and an interest form, then, providers briefly (<2 min) introduce parents to the opportunity to participate in the study. The information card is for parents to keep and provides a description of the study and options for learning more about the study and expressing interest in participating. The interest form is a paper form that includes the parent name, contact information, and indication that the parent is interested in being contacted about the study. If interested, parents complete the interest form and return it to the front desk. If parents do not wish to complete the interest format that time, they can contact the study team by one of the other methods included on the information card (e.g., a Quick Response (QR) code and an internet link connecting parents to an online interest form or contact information for the study staff). Providing multiple options allows parents to express their interest in a way most convenient for them. Also, using the interest form (paper or online) allows parents to passively opt out without having to decline participation directly to their provider.

Recruitment into the study begins once the parent expresses their interest in the study using one of these methods. Interested parents are contacted by telephone by the study staff. During the first phone contact, the study staff confirms the eligibility of prospective parents for the study and sets up an initial data collection appointment.

2.6. Measures and data collection

2.6.1. Implementation measures

The evaluation of ezParent implementation in pediatric primary practices is guided by the RE-AIM framework. The RE-AIM framework was designed to provide a model to enhance the quality, speed, and public health impact of efforts to translate research into practice using five steps: REACHING your intended target population; EFFICACY of the intervention and implementation strategies; ADOPTION by target providers and settings; IMPLEMENTATION including consistency and cost of delivery; and MAINTENANCE of intervention effects in individuals and implementation in settings over time [18,19]. Estabrook and colleagues hypothesized that employing the RE-AIM elements in planning and evaluating interventions should translate to an intervention and implementation plan that is sustainable across different settings and personnel [26]. We will use the RE-AIM framework for evaluating the implementation processes and laying the groundwork for scaling up the implementation and dissemination of the ezParent Program in primary care. Measures for evaluating of the implementation of the ezParent Program will be collected from multiple sources including the practice sites electronic health records (EHR), study records, parent report, and provider report. Table 3 outlines the specific data and sources for each of the RE-AIM components.

Table 3.

Data and source for implementation evaluation.

RE-AIM component	Data and source
REACH Proportion and characteristics of parents from the practices that were introduced to and enrolled in the study.	Proportion of parents introduced to the study during the well child visit to the number of monthly well-child visits for children aged 2–5 over the course of study implementation (source: EHRa) Proportion of parents who enroll in the study to the number introduced to the study during the well child study visit (source: EHR and study records) Characteristics of parents enrolled in the study (source: Demographic and Technology Inventories)
EFFICACY Evaluation of intervention efficacy.	Efficacy of the ezParent intervention is assessed by changes in parenting and child behavior (see Parent and Child Outcome Measures).
ADOPTION Characteristics of providers and practices oriented and agreeing to deliver the intervention (e.g., introduce the study to eligible parents).	Provider characteristics (e.g., specialty, years in practice, and current practices of managing parenting and child behavior concerns) (source: Primary Clinician Survey) Primary care site characteristics (# of patients aged 2–5, average time of well-child visits) (source: EHR) Total number of orientation sessions per practice site and proportion of providers that attend the orientation (source: study records)
IMPLEMENTATION Assessment of the delivery of the introduction of the study by providers and associated costs.	Proportion of providers that introduce the study to parents of patients aged 2–5 at well-child visits (source: EHR) Tracking and monitoring of implementation adaptations by the site and providers (source: study records) Evaluation of implementation-related cost will occur as part of data collection and analysis of resource and cost measures (see Resource and cost measures)
MAINTENANCE Evaluation of the practice site maintenance of implementation procedures and participant level maintenance of behavior change.	Evaluation of maintenance of procedures across the study period at the practice sites by monitoring recruitment rates and the proportion of parents introduced to the study monthly (source: study records and EHR) The total number of follow-up orientation sessions for providers (source: study records) Assessment of implementation challenges and facilitators, and sustainability of activities in clinical practice (source: study records and post-implementation interviews with providers) Maintenance of parenting and child behavior change over the 12-month study period (see Parent and Child Outcome Measures)

^aEHR = Electronic Health Record.

All data from the EHR and study records is aggregated, de-identified, and collected monthly from the primary care practices. The Technology Inventory – an 18-itemsurvey of parents use and ownership of technology – is completed by parents as part of the baseline data assessment. The Primary Clinician Survey, a 25-item measure developed to assess pediatric primary care providers' current practices in helping parents with child behavior problems and parenting concerns [27], will be completed prior to the start of study implementation by the providers. There is no identifying data collected as part of the Primary Clinician Survey and a consent waiver was obtained for collecting the data. All providers in the practice sites are invited to complete the survey, by not completing the survey, providers are able to opt out of participation in the study. At the end of the recruitment period in each practice site, we will conduct post-implementation interviews with the providers and site administrators who participated in introducing the study. Providers will be invited to share their feedback regarding implementation processes. The interviews individually or in groups will be conducted using a structured interview script to assess facilitators and barriers in implementing the study and the ezParent program in practice sites and to explore the potential for maintenance and sustainability of implementation efforts.

2.6.2. Parent and child measures

Parent and child outcomes measures are used to measure the efficacy of the ezParent program. All parent and child measures are obtained by parent self-report and collected at the four data collection time points (baseline, 3-, 6-, and 12-months post baseline).

We are collecting information on social determinants of health that may affect parenting and child behavior outcomes. The social determinants of health include general demographics, income and economic hardship, and neighborhood and community characteristics [28]. Demographics (e.g., age, race/ethnicity, household structure) are collected using a 22-item demographic inventory. Income and economic hardship is assessed with a 12-item income inventory and an economic hardship scale (7 items). [20,29] In our previous study, Cronbach's alpha for the economic hardship scale was 0.68 [17]. Neighborhood and community characteristics will be collected using: (a) 4 items from the National Survey of Children's Health on perceived neighborhood social support [30], (b) 10-items from the Extent of Neighborhood Problems Scale [31] and (c) 4-items from the Fear of Crime Scale [31]. Cronbach alpha reliabilities for mother reports in a study of family function and child delinquency on the Extent of Neighborhood Problems Scale and Fear of Crime Scale were 0.84 and 0.76, respectively [31].

Primary parenting outcome variables include parenting behavior, self-efficacy, and stress. The primary child outcomes are child behavior problems and prosocial behavior. Parenting behavior includes parenting discipline strategies and positive parenting behavior. Parenting discipline strategies will be measured using the 40-item Parent Questionnaire (PQ). Three scales – warmth, follow-through on discipline, and corporal punishment – measure how parents discipline their children. In a RCT of the group based Chicago Parent Program, Cronbach's alpha for the three subscales were 0.84, 0.80, and 0.68, respectively [8]. The PQ has been validated in samples of African American and Latino families [8,32]. Positive parenting behavior will be assessed using the 21-item Parenting Young Children (PARYC) measure. The PARYC includes three subscales: supporting positive behavior, setting limits, and proactive parenting. In a study with 579 high risk families, Cronbach's alpha for the three scales of the PARYC were 0.78, 0.79, and 0.85, respectively [33].

Parenting self-efficacy is evaluated using the 17-item Parenting Sense of Competence Scale (PSOC). The PSOC has two subscales: satisfaction (person's liking of the parenting role) and efficacy (person's perceived competence in the parenting role). In a study with 110 mothers of children age 5–12 years old, internal consistencies for both the satisfaction and efficacy scales was 0.80 [34]. In addition, the PSOC is correlated with other measures of family life and child behavior, and the satisfaction subscale is strongly correlated with measures of child behavior, parent well-being, and parenting style [34,35].

Parenting stress is assessed with the Parenting Stress Index-Short Form (PSI-SF). The PSI-SF is a 36-itemsurveywith three scales: parental distress, parent-child dysfunction, and difficult child [36]. In their study of 103 parents of children in Head Start, Roggman and colleagues reported internal consistency reliabilities of 0.90 for the total scale, 0.79 for parental distress, 0.80 for parent-child dysfunction, and 0.78 for difficult child [37]. The PSI-SF is a valid measure of parenting stress in multicultural samples and for parents from lower socioeconomic groups [38,39].

Child behavior problems are assessed using the Eyberg Child Behavior Inventory (ECBI) [40]. The ECBI is a 36-item scale designed to measure the presence and intensity of problem behavior. Each item is measured on two scales, the Problem Scale and Intensity Scale. The ECBI is a valid measure of child behavior problems, with established convergent validity across racial and ethnic populations and economically and linguistically diverse samples [40–42]. In our previous work, Cronbach alpha reliabilities of the Intensity and Problem Scale scores were 0.92 and 0.93 [17]. The Strengths and Difficulties Questionnaire (SDQ) provides a measure of child behavior problems and prosocial behavior. The 25-itemSDQ has a five-factor structure (emotional, conduct, hyperactivity-inattention, peer, and prosocial). Mean Cronbach's alpha across multiple studies is 0.73 [43]. The SDQ has been validated in multiple cultures and socioeconomic levels [44].

2.6.3. Intervention dose and satisfaction

The ezParent program and Health-e Kids control usage data will be collected digitally via the program applications. Dose will be measured by digital tracking of parent usage of the ezParent and Health-e Kids applications. Three primary usage data points will be collected, they are: (1) hit rate (a hit reflects each time the parent opens a page in the application); (2) proportion of material completed (ezParent: measure of the percent of the ezParent content that is opened in each module; Health-e Kids control: proportion of material opened in the Control application); and (3) intensity of dose (amount of time parents use the application). Usage data will be collected throughout the 12-month study period, stored in parents' Android tablet, and automatically uploaded to secure servers when parents are connected to the internet.

At the end of the 3-month intervention period, parents will complete an end-of-program survey corresponding to their study condition (ezParent or Health-e Kids control). The end-of-program survey assesses satisfaction, usefulness, and facilitators and barriers to the delivery methods and intervention.

2.6.4. Resource and cost measures

We will collect detailed information on the intervention delivery and participant resources and costs associated with the ezParent Program. All cost data will be estimated in US dollars. Resources used for the intervention include the ezParent delivery costs (tablet, tech support, and maintenance costs), modification of the EHR for reminders and tracking in the primary care sites, orientation of primary care providers for introducing the study, and time for the provider to introduce the study to parents during a well-child visit (Table 4). Participant costs are the parents' opportunity costs, representing the value of their time spent in the ezParent Program. Resource use and cost data will be drawn from invoices, staff and provider report, study records, EHR, and time stamp records of parent time spent using the application. Resources used will be translated into costs by multiplying the number of units with the cost per unit for each cost category.

Table 4.

Description of the resources used and costs.^*

Cost category	Source	Unit of measure	Translation to costs
ezParent costs
ezParent delivery costs	Invoice	Participant	Average cost per participant
Orientation delivery costs
EHR modification	Programmer report	Minutes per site	Programmer average hourly wage
Provider time receiving study orientation	Study records	Number of providers Minutes per provider	Provider average hourly wage
Time to lead orientation	Study records	Minutes	Study staff average hourly wage
Provider costs
Time during well-child visit to introduce study	Study records	Minutes per participant	Provider average hourly wage
Parent costs
Time spent learning about the study	Study records	Minutes per participant	Parent average hourly wage
Time spent completing ezParent modules	Digital time logs	Minutes per participant	Participant average hourly wage

^*Note: All cost data will be estimated in US dollars.

ezParent delivery costs related to technology support and maintenance will be collected from our multimedia partners to determine the amount of technology support provided and fixed costs of hosting and maintaining the ezParent program. At each practice site, we will collect the amount of time required by the programmer to make modifications to the EHR for including a reminder to providers to introduce the study to all parents of children aged 2 to 5 during well-child visits. Time spent on the HER modification will be collected from the programmers log (start and stop times for each episode of programming work) of minutes spent making the modifications. The provider time receiving the study orientation and study staff time to lead the orientation is collected by the trainer tracking the start and stop time of each study orientation session, and the average duration will be used to calculate the minutes per provider. Finally, we will estimate the time the providers report during post implementation interviews that it took them to introduce the study to the parent during a well-child visit and average the time across providers and by site. All time estimates will be in minutes.

2.7. Data collection and intervention procedures

2.7.1. Procedures for data collection of parent and child measures

Following informed consent, parents complete baseline survey measures on parent and child behavior. The surveys take approximately 45 min to complete. The data is collected on the Android tablet that is assigned to them to use during the study. A research assistant is present to offer any clarification of survey questions. At the end of the data collection, parents are informed of their random study assignment (e.g., ezParent or Health-e Kids control condition). Once the randomized condition is revealed to the parent, the research assistant downloads the corresponding application to the Android tablet. Continuous internet access is not required for using either application. By providing the control condition with the Android tablet and usual care parent information materials, we control for the potential novelty of the technology and device. Research assistants provide standardized training on the use of the tablet (e.g., turning on/off, charging, how to begin using the application, how to upload information) and brief training on how the application works for the intervention or control condition. A detailed hardcopy manual will be provided to all parents on how to use the tablet and contact information (toll-free phone number and email) for technical issues or questions. The research assistant will ask the parents to demonstrate using the tablet to assure parent understanding and their ability to use it.

Parents in both conditions have access to the corresponding program throughout the 12 month study period. Parents in the ezParent group are instructed to complete the six modules over the first 3 months of the study (intervention phase), or approximately one module every 2 weeks and to practice the parenting skills for at least one week before moving on to the next module. At the end of the intervention phase, parents will receive a completion certificate that includes their badges earned in the ezParent program. Parents in the Health-e Kids control condition are instructed to access that application as often as they require resources throughout the 12-month study period. There is email and toll-free telephone technology support to assist parents with any technology issues they experience during the study. The technology support plan was developed with our technology partners at Klein Buendel.

Post-intervention assessments occur at 3-, 6-, and 12-months post baseline. Two weeks prior to the scheduled assessments, parents will be contacted by study staff to confirm their contact information and remind them that they will be receiving an email and text message to instruct them to complete the survey measures and a link to the survey measures. To ensure that confidentiality is strictly maintained, a random ID number will be assigned to the parent, this number will be used to identify respondents within the dataset. Parents will log in to the survey using their study identification number and complete the surveys.

2.8. Parent participant retention

Retention strategies include web-based data collection, multiple methods for contacting participants, and study incentives. All data collection appointments after the first occur via an internet-based data collection protocol. Using internet-based data collection allows parents to complete the surveys at times and locations convenient to them and provides a secure and confidential way to complete the surveys [45]. Two weeks prior to each assessment time point, all participants will receive a reminder. The mode of contact will be that of parent preference for communication (e.g., phone, email, or text). Allowing parents to identify their primary and preferred mode for communication contributed to high rates of retention in our preliminary study. [17] To help track each contact, we will use an access data base as a tracking system. Parents will be given a dedicated phone number and email to contact with any questions or concerns regarding the study. At each data collection time, participants will receive a $10 gift card upon completion of the surveys. The $10 gift card will be delivered to them digitally providing an immediate incentive to complete the surveys. Throughout the study period, the tablet (cost $100) that the parent uses will be considered the property of the study. As an additional incentive, at the completion of the study, we will sign over the ownership of the tablet to the parent.

2.9. Data analytic plan

2.9.1. Specific aim 1: quantify the levels of program implementation of ezParent in primary care using the RE-AIM framework

Descriptive statistics will be calculated to identify achievements across each level of RE-AIM for each of the four primary care practices. We will develop a demographic, income, and technology use profile for the populations reached (i.e., parents enrolled in the study). We will use contingency table analyses and independent t-tests to compare the demographic and income profiles to the characteristics of the population of the primary care sites service catchment communities (i.e., zip codes) using data from the U.S. Census. This analysis will allow us to assess the extent to which we are either over- or under-representing our priority populations (low-income families) from the local catchment area in this study. Technology ownership and use will be compared to data from national data sets from the Pew Research Center's Internet and American Life Project [46].

To determine the effectiveness of the implementation strategy and inform dissemination of the ezParent, a cascaded probability model will be constructed [47,48]. This will be developed using the following proportions: (a) the likelihood of provider adoption at the site (proportion 1 [pr 1]), (b) the likelihood of implementation (study introduction) by providers at the site (pr 2), (c) the likelihood of parent enrollment in the study (pr 3), and (d) the likelihood of maintenance of behavior change by the ezParent participants (pr 4), defined as the proportion of participants whose parenting and child behavior outcomes at the 12-month data collection compared to the 6-month data collection are either the same (i.e., 12-month scores are within ±0.25 sd of 6- month scores) or improved (i.e., 12-month scores are more than 0.25 sd above 6-month scores). The estimated likelihood of parents being introduced to the program will be the product of pr1 and pr2. Second, the compound likelihood of parent enrollment in the study will be the product of pr1, pr2, and pr3. Finally, we will estimate the compound likelihood of maintaining behavior change by examining the product of pr1, pr2, pr3, and pr4. This analysis will provide us with two important components of implementation. First, the probable benefits of the program in terms of how likely it is for parents to achieve maintenance of behavior change. The second component is the phase of the implementation process (as operationalized by pr1, pr2, pr3, and pr4) that may interfere with the dissemination of the ezParent.

We will conduct a content analysis of the post-implementation interviews with the providers. We will use a coding-based system of content obtained from the interviews using the notes taken during the interview using Atlas.ti software [49]. Atlas.ti allows for the identification, organization, and systematizing of codes that can then be assessed in the context of the provider interviews. Themes will be summarized by site and used to inform the development of processes and procedures to support the implementation and dissemination of the ezParent in pediatric primary care practice sites.

2.9.2. Specific aim 2: test the direct effects of the 6-module ezParent program on parenting outcomes (parenting behavior, self-efficacy, and stress) and child outcomes (child problem and prosocial behavior) compared to an enhanced usual care control condition among low-income parents with young children seen in primary care

All tracking of participants will follow CONSORT guidelines and be entered into a Microsoft Access database [50]. SPSS for Windows (v. 23) and SAS (v. 9.3) will be used for data management and statistical analysis. Missing data will be imputed using SAS Proc MI software and the multiple imputation strategy described by Rubin [51]. A two-tailed 0.05 significance level will be used for all tests. From previous research, we expect all outcome measures to be close to normally distributed; however, we will use Tukey's ladder of transformation to achieve normality, if needed [52]. Outcome measures that cannot be successfully transformed to achieve normality will be analyzed separately using the rank-ordered multilevel analysis available in SAS. Analyses of baseline measures of demographic characteristics, parenting, and child outcomes will be conducted to verify that intervention and control groups are comparable.

All analyses for parent and child outcomes will be performed on an intent-to-treat basis. We will test intervention efficacy using multilevel growth models [53,54] with three levels (assessments [Level 1] within parent [Level 2] within primary care practice site [Level 3]). Intervention condition will be coded (ezParent vs. control) at Level 2, and covariates at site-level (e.g., clinic size) or parent-level (e.g., age, race/ethnicity) will be included at the appropriate level of analysis.

Hypothesis 1. Relative to parents in the control condition, parents in the ezParent condition will report greater improvements in parenting and child outcomes at the 3-month assessment. By including baseline measures of the outcome variable being tested as a covariate at Level 2 and setting the intercept at the 3-month assessment, we will be able to test Hypothesis 1 (intervention effects on changes from baseline to the 3-month assessment) as the main effect of intervention condition (i.e., controlling for baseline level of the outcome being tested).

Hypothesis 2. The improvements in parenting outcomes and child outcomes the ezParent condition relative to the control condition will be maintained through the 6- and12-month assessments. In the same growth model used to test Hypothesis 1, the condition × time interaction will then serve to test differential change in outcome measures following the 3 months of active intervention (i.e., maintenance). In addition to comparing time slopes from months 3–12, planned, and planned contrasts will be used to evaluate intervention effects on the change from baseline to 6 and 12 months.

2.9.3. Specific aim 3: compare the cost-effectiveness of the ezParent program relative to control for parenting and child outcomes

We will assess cost-effectiveness of ezParent from the societal perspective, including costs borne by the program, health care system, and parent. We will conduct additional analyses from the program, health care system, and parent perspectives separately. For the cost measurement, quantities of resources used and their associated prices will be collected for the ezParent (either prices paid or value of the provider time) and participant. Program costs for the control condition are assumed to be $0, because there is not a “standard of care” parenting program for comparison. Program and parent costs will be calculated for the 12-month study duration and summed to calculate total cost per parent. All costs will be valued in 2016 US dollars.

For the cost effectiveness measurement, effectiveness will be measured using the parenting and child outcomes. Cost-effectiveness will be evaluated by combining the mean total cost per parent with effectiveness (parenting and child behavior outcomes). We will calculate the incremental cost-effectiveness ratio (ICER) for ezParent compared with the health-e Kids control condition, such that ICER = (C₁ − C₀)/(E₁ − E₀), where C is cost and E is effectiveness. Subscript 1 denotes ezParent and subscript 0 denotes the control condition; 95% confidence intervals for the ICERs will be calculated to evaluate the uncertainty in these results [55–57]. We will conduct one-way and multi-way sensitivity analyses for the key parameters to evaluate whether the ICERs are sensitive to plausible changes in their values. The sensitivity analysis is a check on the robustness and will determine the key parameters impacting the ICERs. We will also plot acceptability curves based on varying threshold (willingness to pay) values for adherence and change in parenting and child behavior outcomes.

3. Discussion

3.1. Limitations and challenges

Although this study is innovative with high dissemination potential there are some anticipated limitations and challenges. First, because we are relying on technology, it is possible that technical issues may present barriers to intervention content delivery and dose. To address this limitation we developed a strong technical development and support team. There will be 24-hour email and toll-free telephone technology support to assist parents with any technology issues throughout the intervention period. In addition, both the intervention and control applications are accessible offline. Second, because ezParent is a self-directed intervention, it is possible that individuals other than the enrolled parent can complete the program. In an attempt to protect against this, each parent will have a unique password for the ezParent Program. Therefore, the only way that someone else could complete the program would be if the parent shared the password or his or her tablet. During the orientation to the use of the tablet and program, parents will be instructed to keep their password confidential. Third, although we are approximating delivery of the ezParent program by pediatric primary care providers this is not a true test of the implementation of the program. The next step in research would be to conduct a trial where initial delivery of the intervention occurred at the point of care where the primary care provider would enroll patients in the ezParent Program.

Finally, although multiple method and informant data sources are ideal for a test of efficacy, all of our parent and child outcome data will be obtained by parent self-report. Previous findings from RCTs of the group-based Chicago Parent Program have found significant results across parent self-report, teacher report and parent-child observation [7,8]. In these studies, the parent self-report findings have been consistent with other methods of measuring parent and child outcomes.

3.2. Strengths and opportunities

It is estimated that translating and disseminating research into practice can take up to 17 years [58–60]. Dissemination challenges of behavioral interventions include factors related to implementing in target settings and characteristics of the interventions [61]. This study addresses these barriers by using tablet-based technology as a delivery method, which is an under-investigated and novel method for delivering preventive interventions that promote mental health in primary care. This is important because using this delivery strategy places minimal time demands on providers and staff, is highly standardized, and is designed to be self-sustaining. Tablet-based delivery has five key advantages over traditional face-to-face delivery models. First, the parent controls the intervention delivery to provide an efficient, flexible, and convenient way to receive the ezParent program. Second, there is consistency of intervention delivery or high fidelity. Third, there is a low time burden on providers implementing the intervention. Fourth, monetary and time costs associated with travel and in-person implementation are minimized. Finally, tablet-based delivery provides the ability to monitor and assess intervention usage and dose via real-time digital analytics [62–64]. These advantages eliminate major contributing factors to poor attendance that results in low intervention dosage at face-to-face PT programs.

Pediatric primary care practices are ideal settings for implementing and providing programs for supporting effective parenting behavior and preventing behavioral problems in young children. Primary care settings are readily accessible and used by children and families, given that well-child visits are recommended annually for children ages 2 to 5. Further, parents generally view primary providers as trusted sources of information and care and may feel greater comfort in discussing parenting issues and prevention of behavioral problems with these providers [65,66]. Pediatric providers are in a unique position to introduce interventions and provide guidance to parents to promote positive parenting practices and prevent child behavior problems [67].

A strength of this study is embedding and evaluating ezParent program implementation in primary care to inform dissemination efforts with potential for rapid translation into existing models of care. Integrating implementation research strategies with a RCT and cost effective analysis will allow for movement towards rapid dissemination. Evaluating the cost, reach, and setting adoption and implementation will provide important information for identifying facilitators and barriers for delivery of the ezParent Program in primary care. These data will be used to build an implementation toolkit to support dissemination of the intervention once efficacy is established. The implementation toolkit will include processes and procedures for primary care sites to use to guide successful implementation of the ezParent in practice. Our plans are to provide this toolkit as a free resource to sites and providers interested in using the ezParent program.

4. Conclusions

Taken together, this study integrates an efficacy trial and implementation and cost evaluation that will provide essential information and data for ezParent dissemination in pediatric primary care. In addition, this study will inform the broader study of eHealth behavioral interventions through evaluation of the processes and patterns related to parent use and provider implementation of a tablet based behavior change intervention.

Abbreviation

PT

Parent Training