Implementation Findings from an Effectiveness-Implementation Trial of Tablet-Based Parent Training in Pediatric Primary Care

Abstract

Introduction:

The use of mobile delivery to deliver parent training can address barriers to access and improve the translation of interventions in existing settings like pediatric primary care. Studying implementation provides critical information to identify and address barriers and facilitators and inform sustainability efforts.

Methods:

This study was a descriptive evaluation using the RE-AIM framework as part of a Hybrid Type I Effectiveness-Implementation trial of the ezParent Program within four pediatric primary care clinics. We collected data before, during, and after implementation to evaluate provider implementation and their perspectives on program benefits and barriers to implementation.

Results:

On average 14% of eligible parents were introduced to the study. Of these parents, 78% expressed interest in participating and 37% enrolled in the study. Seventy-eight percent of staff providers (n =36) implemented the procedures at least once and among those who implemented three or more times (n = 24), 79% maintained implementation for six months. Barriers to implementation include limited time, lack of information, and full practice buy-in and engagement.

Discussion:

Implementation fidelity may improve with additional education and training of the interdisciplinary team, clear messaging regarding the purpose and content of the ezParent program, defining roles within the care team, identifying practice champions, and use of the EHR. Findings from this evaluation, data from the RCT and literature to support intervention effectiveness and implementation, will be used to develop an implementation toolkit to include specific strategies for implementation and ideas for local adaptations.

Parent training (PT) is the gold standard for helping parents develop skills to effectively manage problem behaviors, promote positive behaviors, and prevent maltreatment (Chen & Chan, 2016; SAMHSA, 2017). Unfortunately, system and logistic barriers make PT largely unavailable to families (Forgatch, Patterson, & Gewirtz, 2013). Mobile delivery approaches can increase access and availability to parents, and improve the quality and translation of PT in pediatric primary care (PPC; Breitenstein, Brager, Ocampo, & Fogg, 2017; Leslie et al., 2016). This paper reports implementation findings from a Hybrid Type I Effectiveness-Implementation trial evaluating a tablet-based PT intervention in PPC.

ezParent Program

The ezParent program is a 6-module, tablet-based PT program adapted from the group-based Chicago Parent Program (Breitenstein, Fogg, Ocampo, Acosta, & Gross, 2016). ezParent was developed to be culturally and contextually relevant for low-income, ethnic minority families of children age 2–5 years old. ezParent helps parents develop positive and effective parenting skills and decrease physical punishment through use of behavioral strategies (e.g., routines, labeled praise), videos of parents using the strategies, activities, quizzes, and assignments. In a previous RCT (n = 79 parents), 85% completed all six program modules, 88% reported that ezParent was very helpful, and 82% would highly recommend the program (Breitenstein et al., 2016). Modest improvements in parenting and child outcomes (Cohen’s d = .14 −.31) are consistent with universal primary prevention program effects (Tanner-Smith, Durlak, & Marx, 2018).

PT in pediatric primary care

PPC is an ideal setting for providing PT because it offers a consistent and supportive context (Perrin et al., 2016). PPC is accessible to parents, has an existing infrastructure for disseminating information, and parents view PPC providers as trusted sources of information (McLearn et al., 2004). PPC providers are often the first professionals that families approach regarding parenting concerns or child behavior problems (Berkout & Gross, 2013). Recent AAP guidelines for decreasing corporal punishment highlight the need for accessibility of resources and programs like ezParent in PPC (Sege et al., 2018).

Expanding PT in PPC can maximize the public health impact of these interventions by improving access and reach. However, interventions need to be easily implemented to avoid adding to an already burdened system (Leslie et al., 2016; Perrin et al., 2016). There is a need identify implementation processes to create organizational change (Brown, Bignall, & Ammerman, 2018) and studying PT implementation in PPC provides critical information to identify and address barriers and facilitators and promote sustainability efforts.

Hybrid Type 1 Effectiveness-Implementation design

Three types of hybrid effectiveness-implementation designs vary on the emphasis placed on effectiveness testing and implementation evaluation (Curran, Bauer, Mittman, Pyne, & Stetler, 2012). This Type 1 study design employs a rigorous test of intervention effectiveness while collecting implementation data for feasibility and acceptability of implementation (Curran et al., 2012). In this paper we report on the implementation processes in PPC using RE-AIM (Reach, Effectiveness, Adoption, Implementation, and Maintenance; Glasgow & Estabrooks, 2018). We report encountered barriers and facilitators to implementation. This implementation evaluation will inform future efforts to add PT to standard practice in primary care.

Methods

Study Design and Context

A descriptive design using RE-AIM guided the evaluation of implementation in PPC. We evaluated provider practice of introducing the study to parents of children ages 2–5 years during a well-child visit (WCV). This introduction served as proxy for recommending the intervention. The implementation preceded individual-level randomization to ezParent or the control program. We report on reach, adoption, implementation and maintenance components that were assessed using quantitative measures.

Implementation occurred between April 2016 and April 2018 in four urban PPC sites that served predominantly low-income Chicago communities. All sites report a diverse and a large Medicaid/underserved population base. PPC1 is a medicine-pediatrics practice, PPC3 a family medicine practice, and PPC2 and PPC4 are general pediatric clinics.

Implementation Approach and Procedures

The implementation approach for introduction to the study was tailored to each of the PPC sites. As part of implementation agreements with each site, we identified key stakeholders (providers, staff, and administration). We worked closely with the stakeholders to understand individual practice environments and workflows and to develop tailored implementation plans. Across all sites, PPC providers introduced the study to parents as part of a WCV, allowing integration into the workflow and approximate the process that would occur in practice. The Consolidate Framework for Implementation Research (CFIR; Keith, Crosson, O’Malley, Cromp, & Taylor, 2017) and a review of strategies for implementation in health care (Powell et al., 2015) guided our implementation approach.

Once the implementation approach was finalized (Table 1), we developed scripts and training materials. All sites provided parents with written information describing the study and an interest form with contact information for parents to complete. Two sites (PPC1 and PPC4) included the study material in the packets given to parents at their WCV. In the two other sites (PPC2 and PPC3), the staff who brought the family to the exam room provided the study information sheet. Therefore, all parents should have the study information when the provider entered the room. The providers briefly (< 2 minutes) presented the opportunity to receive parent support via tablet-based apps and to participate in the study. After the visit, the providers completed tracking logs (paper or electronic health record (EHR)) to report compliance with the implementation procedures. Implementation was planned for 10 months at each site and start times were staggered by four months.

Table 1.

Implementation Approach in PPC

Implementation strategy	Description	Implementation Approach
Plan	Assess readiness, identify barriers, build relationships	• Developed relationships of study team and PPC team (written communications and face-to-face meetings) • Reviewed workflow and organizational climate • Developed tailored implementation strategy based on agreement from study and PPC teams • Identified potential barriers to implementation and tailored strategies to overcome barriers and honor PPC site preferences • Identified and prepared champions (1 provider per site) • Develop staff and provider training materials
Educate	Informing stakeholders of implementation initiative	• Conducted educational sessions for staff, providers, and administration on implementation activities
Finance	Incentives	• Provided snacks at educational sessions and periodically through implementation period as thank you for participation.
Restructure	Alterations in roles, structures, and data systems	• One site adapted the electronic Health Record (EHR) where providers could select if they implemented the study introduction, the remaining sites used a paper tracking system to monitor implementation.
Quality Management	Audit and provide feedback	• Collected monthly performance data (see RE-AIM measures) and provided reports to PPC providers and staff via email updates. Used this feedback to provide reminders and clarify messaging and adjust implementation procedures as needed.

Study procedures and protocols were approved by IRBs at the two primary institutions of the four PPC sites.

Measures and data sources

We collected data at three points: pre-implementation, during implementation, and postimplementation.

Reach.

To evaluate that we were reaching our intended population (i.e., parents of children age 2–5 years old) we measured the characteristics of parents from the practices as compared to those who enrolled in the study. All parents who presented with their 2–5-year-old child for a WCV were eligible. All data collected from the EHR were stripped of identifiers and aggregated.

Adoption.

To evaluate adoption of the implementation procedures we examined the characteristics of the PPC sites. We tracked the number of educational sessions to orient providers and staff to the project and implementation procedures. We assessed providers’ current practices for responding to parent and child behavior concerns. Prior to the start of implementation, providers responded to the Primary Provider Survey (Metzler et al., 2014), demographic questions and 32 items assessing their current practice and attitudes in helping parents with managing child behavior problems. Item responses were on a 5-point scale assessing level of difficulty in addressing children’s behavior problems, parenting concerns, confidence in managing these problems, and rating of the practice’s capacity for dealing with parenting difficulties or child behavior problems.

Implementation.

To track provider delivery, three sites used paper tracking logs and one site used the EHR to record whether the provider discussed and gave the study information to parents. If not discussed, providers recorded the reason. Variations in tracking procedures (e.g., paper tracking logs or EHR) occurred due to preference and administrative support to amend the EHR. We also tracked and monitored any adaptations made during delivery. Given variations in resident rotations, we were unable to identify the proportion of residents that implemented or maintained procedures; therefore, proportions are presented for staff providers.

Maintenance.

To evaluate the maintenance of procedures, we developed a postimplementation survey to evaluate providers’ knowledge and implementation of the procedures and assess barriers and facilitators to implementation. Providers responded to seven questions assessing their role and ease or difficulty of implementation of the procedures and rated their perception of implementing the study introduction into their day-to-day activities on a 5-point scale - not at all, only slightly, somewhat, quite, or very. A final open-ended question queried respondents if they had other thoughts or suggestions. We invited providers to complete the survey after the end of implementation.

Analysis

For all analyses, descriptive statistics were calculated by clinic and overall. Given inherent differences between clinics, we did not assess between-clinic differences. However, we provide descriptive data by clinic to illustrate variability across sites. To evaluate reach, chi-square tests were used to compare the demographic characteristics of the study participants to the patient population.

For maintenance, we assessed only those providers who had five or more WCVs in at least one month during the implementation period. This approach was taken to ensure adequate exposure to the study and opportunities for implementation of the study protocol. All providers with fewer than five WCVs were either part-time staff or medical residents.

Results

Reach

Across sites, the monthly number of WCVs for children ages 2–5 years ranged from 20 to 269. On average, providers introduced 14% (18/124) of eligible parents to the study each month. Of the parents introduced to the study, 78% (14/18) expressed interest in participating (Table 2). Seven hundred and fifty nine parents signed forms indicating their interest in the study; however, the research staff never reached 29% (217/759) to screen for eligibility. Of those screened for eligibility, 3% (14/542) were ineligible, 44% (241/542) did not show up for their baseline appointment, and 53% (287/542) enrolled.

Table 2.

REACH Metrics for Implementation in Pediatric Primary Care

	Recruitment period	Well-child visits for children aged 2–5	Parents introduced to the study	Parents expressing interest in study	Parents enrolled in the study	Total parents enrolled
Site	Months	M(Range) per month	M(Range) per month	M(Range) per month	M(Range) per month	n
1	18	19.9 (7–54)	10.3 (2–29)	7.9 (3–16)	3.1 (1–7)	56
2	11	269.3 (209–348)	39.2 (22–52)	19.6 (12–33)	8.8 (2–14)	100
3	14	81.7 (37–153)	12.8 (5–29)	12.6 (5–20)	3.6 (1–6)	52
4	11	200.5 (137–315)	14.2 (4–23)	19.0 (7–32)	7.2 (2–16)	79
Total		123.9 (7–348)	17.7 (2–52)	13.8 (3–33)	5.2 (1–16)	287

Although race/ethnicity of enrolled parents did not differ between sites, in two sites the race distribution of those who enrolled differed from the race distribution of the practice. Specifically, in PPC2, enrolled parents were less likely to be White (12% vs. 23% in the practice) and more likely to be Black/African American (57% vs. 47%) or other (31% vs. 24%; p = 0.008). In PPC4, participants were more likely to be White (17% vs. 8%) and Black/African American (61% vs. 48%) and less likely to be other (23% vs. 37%; p = .002). There were no significant child gender differences between sites, however, PPC3 had fewer enrolled females (39% vs. 52%; p = 0.05).

Adoption

All sites adopted their individualized implementation plan (Table 3). We report responses from the staff MDs and APNs as they were primarily responsible for implementation. Across the four sites, 24 (67%) providers responded to the Primary Provider Survey. They reported an average of 14.94 (SD = 10.07) years of practice, 83% reported that it was slightly (58%) to very (25%) difficult to address parents’ concerns about their children’s behavior problems during WCVs, 79% reported that it was very difficult to make an appropriate referral for children’s behavior problems, and 92% reported that they felt inadequately trained to effectively help parents to address behavioral or emotional problems. Providers rated the practices capacity for dealing effectively with parenting difficulties or children’s behavior problems as poor (12%), fair (68%), and good (20%).

Table 3.

Description of Primary Care Clinics Sites that Adopted the Implementation Plan

Practice Site	Site Description	Patient Population	Providers	Allotted time for well child visit
PPC 1	Combined internal medicine and pediatrics	Newborn through geriatric	4 attending MDs; 1 NP; 16 resident MDs	30 min (MDs) 40 min (NP)
PPC 2	General pediatrics	Newborn through young adult	15 attending MDs; 33 resident MDs	20 min
PPC 3	Federally Qualified Health Center	Newborn through adults	5 family medicine MDs; 2 Pediatric MDs; 24 family medicine residents	15 min
PPC 4	General and specialty pediatrics	Newborn through young adult	8 attending MDs; 1 NP; 39 resident MDs	30 min (1st year residents only) 20 min (all other)

Across the four sites we held an average of two sessions (Range = 1 – 3) to orient providers and staff to the project and implementation procedures. In PPC2, we conducted one training session for the staff and nurses; however, we were unable to conduct direct training to the providers. Instead, our site liaison (author LP) introduced the study and process to the providers at a standing meeting and during practice times. In addition to the in-person trainings, we sent reminder emails to all providers and staff the day of the implementation launch, and in PPC2 and PPC3 we attended brief morning huddles to remind providers and staff of the start of the program.

Implementation

Implementation feasibility was high across providers: 67% - 100% of staff providers reported that they implemented the procedures (Table 4). Monthly implementation tracking was used to identify adaptations and methods to improve implementation. We provided emails to all providers and staff containing implementation rates, procedure reminders, encouragement, and positive reinforcement. Overall, adaptations in all of the sites were minor (e.g., placing forms in areas more visible, changing location of form collection boxes, and posting reminders in break rooms). In addition, our project staff were on site at the PPCs at least twice weekly to collect parent interest forms and support implementation.

Table 4.

Provider Implementation and Maintenance of Procedures

Site	Total Staff	Times Implemented			Maintained 6+
	Providers	Never	1–2 Times	3+ Times	Months1
PPC 1	5	0 (0)	0 (0)	5(100)	5(100)
PPC 2	15	4 (27)	0 (0)	11 (73)	7 (64)
PPC 3	7	1 (14)	3 (43)	3 (43)	3 (50)
PPC 4	9	3 (33)	1 (11)	5 (56)	4 (67)
Total	36	8 (22)	4 (11)	24 (67)	19 (68)

¹The denominator for the maintenance percentages is the number of staff providers who implemented one or more times

Maintenance

We examined staff providers implementing at least one time (n = 28) to evaluate maintenance over time, of these, 68% maintained implementation for at least six months. Among those who implemented three or more times, 79% maintained implementation for six months (Table 4). In addition to monthly tracking, at the end of implementation, providers responded to a post implementation survey. Sixty-eight providers (32 staff MDs/NPs and 36 resident MDs) responded to the survey. Most (77%) respondents reported that they referred at least one parent to the project. One provider noted that they believed “this was an excellent program and helpful for patients” and another that the program contained “very useful information for the patients.” One provider noted that they were motivated by parent report that “the modules helped them deal with their children better.” Over half (59%) of respondents reported that they provided the parents with program materials, 57% reported that they explained the opportunity to parents, and 27% reported that they answered parents’ questions regarding the program. Providers commented that barriers to implementation related to inconsistencies in maintaining protocols (“materials were inconsistently part of the well-child packets” and “patients did not bring the forms into the exam room”). One provider commented “more information about the interventions themselves would be helpful” Another recommended “MDs do not have reasonable time to perform necessary introduction and stay efficient.” Finally, several residents commented that it was easier to implement when their supervisor promoted the program.

Despite the majority of providers (69%) endorsing the importance and appropriateness of providing parenting resources in PPC, nearly 25% reported they did not participate in any implementation activities. This is consistent with daily tracking (53% of providers implemented the procedures more than once). Of the providers (23%) who reported that they did not participate in the implementation activities, the reasons (multiple endorsements allowed) included: unaware of the program (19%); inadequate time (31%); did not understand procedures (50%); forgot (38%); the parent did not receive materials (63%); and not a priority (13%). As one provider noted “...within a 20-minute time period...I am to take a history, do an exam... fill out WIC and school forms, log into the computer, enter orders, explain vaccines and make an attempt to be civil and engaging. On many days, we are running behind, so I don’t even really have my full 20 minutes. Therefore, I had no time to add more to my work.”

Discussion

In this section, we review relevant findings from the RE-AIM evaluation and discuss factors that contributed to implementation success, challenges, and recommendations for future efforts.

A compelling finding was that we did not adequately reach the target parents for this program. On average, only 4% of potentially eligible parents enrolled in the study. One factor to this low level of reach is related to implementation breakdowns and that providers introduced only 14% of eligible parents to the program. Despite careful planning, we encountered challenges consistent with other studies of brief intervention in PPC, including time, competing priorities, and lack of information/communication about the program (King, Muzaffar, & George, 2009; Rahm et al., 2015).

Practices (PPC2, PPC3, and PPC4) that conducted more monthly WCVs for children aged 2–5 years old had lower rates of program introduction than PPC1 (M = 20 visits/month). One explanation is that smaller practices have fewer moving parts and greater ability to integrate new procedures. Relative to the other sites, PPC1 had fewer residents in the practice, the RN director provided support to providers with verbal reminders prior to a WCV, and the providers documented adherence to implementation in the EHR. Successful implementation requires this type of holistic approach to alter provider habits and increase implementation (Johnson & May, 2015). As identified by a resident, if the attending physician supervising them did not promote the program they were unlikely to do so. This further complicates implementation maintenance as residents rotate frequently through practice settings. Furthermore, 60% of providers indicated that materials were not in the packet or provided to the patient. Thus, even if providers wanted to introduce the program, the lack of materials may have prohibited it. Reminders to include this information for the providers could increase reach.

PPC1 was the only practice that was able to use the EHR to document implementation. We were not able to track whether providers in the other PPCs forgot to fill out the paper tracking logs, making it possible that we have underestimated provider implementation (e.g., providers introduced the study but failed to report that they did). This highlights the benefit of having the cue to integrate the introduction to the program as part of anticipatory guidance procedures in the EHR. Offering universal PT as part of anticipatory guidance aligns with AAP recommendations for promoting family support and child development (Duncan et al., 2015). It is also possible that administrative support to amend the EHR for this project signals a higher level of support and buy in for interventions.

Another implementation challenge was the context of a RCT. Because providers introduced the study prior to randomization, we did not provide comprehensive information to the providers regarding ezParent and there were multiple steps required for parents to enroll (e.g., complete interest form, set up baseline appointment) that may have created provider and parent barriers. During training sessions, staff and providers received minimal information about ezParent and the control condition (e.g., parents would receive either tablet-based parenting support or health promotion content). Therefore, with limited information and uncertainty about which program would be delivered, providers may have been less motivated to provide information to families. In practice, with specific information regarding content and outcomes of the PT, providers may be more likely to endorse the program and maintain implementation of the procedures. Further, with multiple competing QI and research projects, it is possible that providers were less motivated/able to integrate another new activity into their practice. If ezParent delivery was a permanent initiative integrated into the practice, it would become a normative feature and may have higher provider engagement and allow parents immediate access to the intervention. Despite these limitations, we believe that collecting implementation data while conducting a RCT informs the external validity of the program, uptake, and how to support practice site implementation.

Implementation was low despite providers’ survey responses that highlight acceptability and appropriateness of the intervention in PPC (Proctor et al., 2011). One potential explanation is that ezParent was delivered as a universal prevention program and was appropriate for all parents in the practice with 2–5-year-old children and not focused on specific behavior challenges. We intentionally designed the delivery to have few criteria to lessen stigma and increase ease of identification and delivery. However, given the limited time and providers’ responsibility to address requirements for the WCV and parent concerns, providers may be less likely to promote programs or studies that are not indicated for a given patient. For example, if parents are asking questions about their child’s behavior, the provider may be apt to use the time to explain a program relevant to their concerns. While always relevant during the WCVs, parenting is not always one of the main concerns addressed during a WCV. Successful implementation may require a stepped-care approach in which the providers direct parents to a universal parenting program for all parents and a more targeted intervention or referral for parents who have a specific behavioral concern. This may be a more feasible and acceptable method for providers to use existing infrastructure to reach all parents in the practice. This aligns with findings that primary care may be a critical point of entry for prevention interventions but not the sole intervention site (Rojas et al., 2019).

Our findings are consistent with the literature that identifies barriers to implementation, including time, lack of information, and full practice buy-in and engagement. Therefore, to support successful implementation we will use these findings and the Agency for Healthcare Research and Quality toolkit guidelines (2013) to develop an implementation toolkit to guide pediatric practices to deliver ezParent. The toolkit will include education and training of the interdisciplinary team, clear messaging regarding ezParent purpose and content, defining roles within the team, development of practice champions, parent education and advertising, use of the EHR, data from the RCT and literature to support intervention effectiveness and implementation, strategies for implementation, and ideas for local adaptations.

Using pre-implementation findings from a Hybrid Type I Effectiveness-Implementation trial provides important information to identify challenges in delivering PT in primary care. Application of these findings and the existing literature will inform our ability to leverage existing infrastructure to integrate mental and behavioral health prevention in PPC.