Parent And Caregiver Active Participation Toolkit (PACT): Adaptation for a Home Visitation Program

Abstract

Objectives

Parent engagement poses a persistent challenge to home visitation (HV) programs. Previous work on parent engagement in HV has focused primarily on enrollment, attendance, and retention, with less attention on participation. The purpose of this study was to adapt an engagement toolkit originally developed for child mental health treatment settings, the Parent And Caregiver Active Participation Toolkit (PACT), and test the adapted toolkit in a HV program, SafeCare® (SC), with a focus on parent participation.

Methods

Toolkit adaptation was informed by interviews/focus groups with parents and home visitors. Next, home visitors (n = 6) were trained to use adapted PACT for SC as part of SC delivery to 18 parents. A comparison group included 24 parents who received SC one year prior to this study. Analyses compared PACT for SC participants to the comparison group on parent participation and home visitor fidelity to assignment of homework. Qualitative and quantitative data from parents, home visitors, and supervisors (n = 4) assessed the acceptability, utility, appropriateness, and feasibility of PACT for SC.

Results

Parents receiving PACT for SC had higher participation and reported greater home visitor fidelity to homework assignment than comparison parents. Parents found PACT for SC acceptable and useful as part of SC. Home visitors and supervisors identified some limitations in PACT for SC’s utility but generally found it to be a positive, feasible addition to HV services.

Conclusions

Results suggest that enhancing HV programs with an engagement toolkit may improve parents’ participation in services and providers’ assignment of homework between sessions.

Engaging parents or other caregivers (hereafter referred to as parents) to participate in home-based services is a critical component to service delivery in early childhood home visiting programs (Korfmacher, Kitzman, & Olds, 1998; Raikes et al., 2006). Yet, full parental engagement has been difficult to accomplish and continues to be one a major barrier to the effectiveness of home visitation programs (Sweet & Applebaum, 2004). Engagement is a broad term that encompasses enrollment, attendance/retention, in-session participation, effort between sessions, and maintenance of new skills after service completion (Littell, Alexander, & Reynolds, 2001; Wagner, Spiker, Inman Linn, & Hernandez, 2003). Two understudied parts of engagement are in-session and between-session participation (Haine-Schlagel & Walsh, 2015). In-session participation includes attending to information delivered, sharing questions and concerns, observing the home visitor modeling skills, in-session practice, and contributing to discussions about homework (Lefever, Bigelow, Carta, & Borkowski, 2013; Martinez & Haine-Schlagel, 2018). Between-session participation includes efforts between sessions, such as homework follow-through and generalized skills practice (Gomby, 2005; Lefever et al., 2013).

Home visitation programs are designed to reduce many practical barriers to the enrollment and attendance/retention components of engagement that families can face, such as transportation, childcare, and time off work (Sweet & Appelbaum, 2004). Home visiting programs also address attitudinal barriers to engaging in services, such as increasing parenting efficacy and motivation (Iannos & Antcliff, 2013; Lewis et al., 2018). A lack of participation both during sessions and effort between sessions can reduce the effectiveness of home visitation programs (Carta, Lefever, Bigelow, Borkowski, & Warren, 2013; Gomby, Culross, & Behrman, 1999). In-session participation is important to increase learning and the ability to utilize new skills (Nock & Ferriter, 2005), while effort between sessions increases the likelihood that behavior changes can be generalized to achieve a program’s desired outcomes (Karver, Handelsman, Fields, & Bickman, 2006; Kazantzis, Whittington, & Dattilio, 2010).

Funding for home visitation programs is shifting to a determination based on whether programs demonstrate a minimum threshold designation of “promising” evidence and follow a structured or semi-structured protocol (Gellatly et al., 2019). Within this context, an additional challenge to program effectiveness is the role of fidelity – delivery of the program as intended (Gellatly et al., 2019). If home visitation programs are not being delivered as intended, there is little confidence in whether a sufficient dose of the program is being provided to families. The completion of homework to facilitate learning and skill practice is an important component of many evidence-based (EB) parenting programs (Clarke et al., 2015; Garland, Hawley, Brookman-Frazee, & Hurlburt, 2008; Kling, Forster, Sundell, & Melin, 2010), including home visitation programs. Practicing skills outside of session is often an important facet of structured EB interventions and if fidelity to homework assignment is poor, parents are receiving a lower dose of the intervention than intended. The manner in which homework is assigned is also likely important for homework adherence in parenting programs. Detweiler-Bedell and Whisman (2005) found that for adult depression treatment, reduction in depressive symptoms was associated with factors such as therapist-client collaboration on assignment of homework, development of written homework reminders, and discussion of barriers to homework completion.

Given home visitation programs’ primary focus is on changing parent behavior, both in-session and between-session participation are potentially potent targets to improve program effectiveness (Sweet & Appelbaum, 2004). Only a small handful of studies have examined parent participation as an outcome of engagement interventions for home visitation programs. One is Planned Activities Training (PAT; Carta et al., 2013; Lefever et al., 2013). In a study of 371 low-income mothers and their 3.5- to 5.5- year-old children recruited from community health, early education, and social service agencies, researchers enhanced PAT with text messaging and found positive effects on parent- and child-level outcomes (Carta et al., 2013). In another study of PAT with 255 mother-child dyads, researchers also found that early engagement, which included participation in sessions and practice of new skills during sessions, predicted intervention completion (Lefever et al., 2013). It is notable that the odds of completing PAT were more than 14 times higher for parents with higher early participation than for those with lower early participation (Lefever et al., 2013). In addition, Thompson, Bender, Windsor, and Flynn (2009) developed an “Engagement Activities” program in part to enhance parent participation in home-based family therapy programs. The engagement enhancement did not demonstrate positive effects on participation.

The need for interventions to increase parent participation in home visitation programs is heightened in the child welfare context. Child welfare service providers report one of their greatest challenges is difficulty promoting parent behavior change (Forrester, Westlake, & Glyn, 2012; Pecora, 1989). Child welfare service providers are expected to balance parent partnership with a focus on child needs and protection and often do not receive adequate training in promoting parent behavior change (Forrester, McCambridge, Waissbein, & Rollnick, 2008).

While limited research on parent participation strategies has been conducted to date in home visitation programs, researchers have been tackling the challenge of developing strategies to promote parent participation in child mental health services and have met with some success. For example, a review of strategies to promote engagement tested across treatments delivered in the child mental health services context found that assigning homework was a frequent and effective strategy to increase parents’ in-session and between-session participation (Becker et al., 2015). Although optimally integrating participation and other engagement strategies into existing structured EB protocols is a unique challenge, some interventions to enhance engagement in specific EB child mental health treatments have yielded promising results (Chaffin et al., 2009; Dorsey et al., 2014).

In contrast to engagement strategies designed for specific interventions, the Parent And Caregiver Active Participation Toolkit (PACT) is a set of parent participation strategies for standard community-based child mental health treatment, rather than for a specific, structured EB program (Haine-Schlagel et al., , 2018). PACT is a services toolkit designed with community stakeholder input to complement standard child mental health treatment (Haine-Schlagel, Martinez, Roesch, Bustos, & Janicki, 2018). The original toolkit includes: 1) a set of evidence-informed engagement strategies for therapists (referred to as the “ACEs” to represent the focus on alliance, collaboration, and empowerment); 2) a DVD and accompanying workbook focused on promoting parents’ participation and collaborative partnership with the therapist; 3) a worksheet to facilitate collaborative homework planning (referred to as the “Action Sheet”); and 4) motivational messages delivered to parents between sessions. Training includes an in-person workshop, group webinar consultations, an individual consultation, and weekly training tips.

A randomized pilot study of PACT was recently conducted with families of children (aged 4 to 13 years) with behavior problems, who received standard mental health services in child outpatient mental health clinics (Haine-Schlagel et al., , 2018). This study yielded promising results on several engagement outcomes that are important for the delivery of home visitation programs (Haine-Schlagel et al., , 2018). For example, both therapists who used PACT as part of their usual service delivery and parents who received PACT as part of their child’s standard care demonstrated a significant increase in working collaboratively to plan for homework between sessions. Further, therapists in the PACT condition demonstrated more extensive use of strategies to promote the parent’s strengths and effort in sessions than therapists in the control condition without PACT.

PACT’s positive results suggest that the toolkit may show promise for use in home visitation programs given the focus of many home visitation programs is on building parenting skills, thus necessitating active parent participation both in and between sessions. In addition, PACT’s focus on the process of delivering a service rather than a specific curriculum suggests it is well-suited to be adapted for structured programs. However, to date no efforts have been made to adapt PACT or any other participation-enhancing strategies from the child mental health service system context for structured, EB home visitation programs.

To answer the question of whether PACT can be adapted for a structured, EB home visitation program and can be implemented effectively in community settings, the current study adapted and tested PACT for use in SafeCare® (SC), an EB home visitation program. The following hypotheses or research questions were examined. In terms of hypotheses, parent in-session participation and home visitor assignment of homework as designated in the home visitation curriculum is hypothesized to be higher for cases in which PACT is used compared to cases in which PACT is not used. Further, home visitors will report greater parent between-session homework completion for cases in which they used PACT compared to their anecdotal reporting regarding cases in which they did not use PACT. In terms of research questions, this study will examine the degree to which parents, home visitors, and/or supervisors report both quantitatively and qualitatively that PACT was: 1) acceptable and useful to increase parent participation; 2) an appropriate fit for the agency itself and the population it serves; and 3) feasible to use. Third, this study will examine the degree to which both home visitors and supervisors perceive the PACT training protocol as acceptable, useful, and feasible. Fourth, this study will examine the degree to which home visitors and supervisors report plans to sustain the use of the PACT toolkit.

Method

Participants

This mixed-methods study was primarily embedded within a large-scale, National Institute of Mental Health-funded study, the Interagency Collaborative Teams to Scale-Up Evidence Based Practice study (ICT; Hurlburt, Aarons, Fettes, Willging, Gunderson, & Chaffin, 2014). The majority of study activities took place in San Diego, California. In the child welfare service system in San Diego, SC is delivered to parents with open cases who have been referred to child welfare services with child neglect as the primary referral reason. Parent participant eligibility criteria for the ICT study were: 1) at least 18 years of age; 2) were referred by the San Diego County Child Welfare Service for child neglect; and 3) had at least one child in the family under age 12. Home visitor participant eligibility criteria for the ICT study were: 1) employed by community partner agencies contracted to deliver home visitation services; and 2) providing SC through a county contract. Parent participants in the ICT study participated at a rate of over 90%. Participants are described below for the adapting and examining PACT for SC study phases separately.

Phase I: Adapting PACT for SC: Parents

Parents were recruited from the ICT study for phase one of this study via criterion sampling. Recruitment flyers were distributed to home visitors in all partner agencies in San Diego delivering SC. Home visitors were asked to share the flyers with parents who had been receiving SC for at least six weeks or one third of the curriculum, thereby inviting parents with experience in the program to provide feedback. Parents who were interested in participating then directly contacted the research team via the information on the flyer. Parent recruitment concluded when saturation was reached regarding the content of the adapted PACT toolkit (Francis et al., 2010). All parents who contacted the researchers were invited to participate in an interview. Of those who contacted the researchers, 79% (11/14) participated. No parents were systematically excluded. Of the 11 parents who participated, eight were female and six were Hispanic/Latinx. Their average age was 29 and the majority had a high school diploma or less.

Phase I: Adapting PACT for SC: Home visitors

Six focus groups were conducted consisting of home visitors across San Diego (n = 4) and Oklahoma (n = 2), two of the largest service systems implementing SC for over a decade. The research team had an initial target of four focus groups, comprising the full universe of home visitors in the San Diego service system. After completing the four focus groups, the research team concluded it would be prudent to the adaptation to include home visitors outside the local context, but with commensurate expertise in SC. As such, two additional focus groups were conducted in a comparable SC serving child welfare service system in Oklahoma. The emergent ideas and feedback on the adaptation of PACT were convergent across the settings. All home visitors within the teams from the six unique community-based organizations contracted to deliver SC were invited to participate, with a participation rate of over 80% (48 of 56 total home visitors, across 6 focus groups). The overwhelming majority of participants were female (95%) and most held a Bachelor’s degree (56%). In San Diego, home visitors were majority Hispanic/Latinx (70%) and in Oklahoma the majority were Non-Hispanic White (75%). San Diego home visitors were also participants in the ICT study.

Phase II: Examining PACT for SC: Home visitors, supervisors, and parents

One partner agency in San Diego was selected to examine PACT for SC delivery given the diverse range of parents served and the agency’s interest in obtaining PACT for SC training for its home visitors. Home visitor participants comprised all six provider staff at the partnering community agency who deliver SC to child welfare-involved parents. In addition, four supervisors and SC coaches who oversaw the delivery of SC in the partnering agency participated in this study. For this phase of the study, parent participants were 18 parent clients of the six home visitors (3 parents per home visitor). A comparison group of 24 SC parents was extracted from the ICT study to examine the study hypotheses. This comparison group was comprised of all participating parents who initiated services with the participating home visitors during the same timeframe as current study recruitment, one year prior to the current study’s implementation. See Table 1 for demographic information.

Table 1.

Study Participant Demographics

Demographic Variables	Supervisors/Coaches (N=4)	Home visitors (N = 6)	Parents PACT for SC (N = 18)	Parents Comparison (N = 24)
Female	100%	100%	72.2%	66.7%
Race/Ethnicity
Hispanic	75%	83.3%	72.2%	58.3%
Non-Hispanic White	25%		5.6%	16.7%
Non-Hispanic Black		16.7%	11.1%	16.7%
Non-Hispanic Asian			5.6%	4.2%
Non-Hispanic Other Race			5.6%	4.2%
Education
Did not graduate high school			22.2%	25.0%
HS diploma/GED			38.9%%	29.2%
Some college		16.7%	27.8%%	29.2%
College Degree	25%	66.7%	5.6%	16.7%
Graduate Degree	75%	16.7%	5.6%
Age	M=40.3 (SD=7.8)	M = 38.3 (SD = 6.4)	M = 28.9 (SD = 6.0)	M = 32.4 (SD = 9.5)
Job Tenure	M=9.9 (SD=4.3)	M = 9.4 (SD = 1.6)
Number of children in the home*			M = 1.5 (SD = 1.0)	M = 2.8 (SD = 1.5)
Household size*			M = 3.8 (SD = 2.0)	M = 4.8 (SD = 2.5)

Note.

^*p<.05

Procedures

A multiple stakeholder team that included the authors and a SC supervisor completed both phases of this study. All data collection procedures were approved by the University of California, San Diego Institutional Review Board.

Overview of SC

an EB program, SC is structured and behaviorally prescriptive on how its curriculum is to be delivered to the targeted families by home visitors. Although services are expected to be individualized for each family’s needs, SC home visitors teach parents to problem-solve using the its three curriculum modules. SC program completion typically takes around 18 to 20 home visiting sessions over six months. A number of studies attest to SC effectiveness and provide evidence on improved parental outcomes in parental use of planned activities, better management of their child’s health, increased home safety, and more positive and sensitive parent-child interaction (Bigelow & Lutzker, 1998; Gershater-Molko, Lutzker, & Wesch, 2003; Lutzker, Bigelow, Doctor, & Kessler, 1998). In the child welfare context, SC has shown to significantly reduce maltreatment recidivism relative to standard family preservation/family reunification services (Chaffin, Hecht, Bard, Silovsky, & Beasley, 2012).

Phase I: Adapting PACT for SC

Parent participants were invited to participate in a 30–60 minute interview at their home. Research staff invited home visitor participants in San Diego to participate in a 60 to 90-minute focus group during a team meeting at each partnering community agency. Recruitment for the two Oklahoma focus groups was conducted by the research team via email invitation. Two doctoral level researchers (one the second author) with extensive experience collecting qualitative data from various stakeholder groups completed the interviews and focus groups, which were all audio-recorded.

Phase II: Examining PACT for SC: Training

Participating home visitors first attended a two-hour training to learn how to use PACT for SC. After that initial training, participating home visitors were asked to utilize PACT for SC with their next three new parents who had consented to participate in the larger study. Seven monthly one-hour in-person consultation meetings were held to reinforce the use of PACT for SC tools and problem solve implementation challenges. Weekly training tips were emailed to all participating home visitors during the training period. All training took place at the participating community agency, typically during arranged staff meeting times. Supervisors were welcome to attend all training sessions, which were facilitated by the PACT for SC developers (first and second authors). Fidelity forms submitted by home visitors (described below) were reviewed on a regular basis and feedback regarding PACT for SC delivery was provided at each consultation. The initial training and consultations were video-recorded, and any home visitor who missed a meeting was asked to view the recording and complete knowledge questions about the recording. Consultations included didactic information sharing, discussion and feedback regarding PACT, and in vivo demonstrations of PACT.

Phase II: Examining PACT for SC: Implementation

In order to examine the efficacy and feasibility of PACT, several data collection tools were put into place. First, as part of the ICT study parents completed a questionnaire at the end of each home visit that asked about fidelity to the SC model and a second questionnaire at the end of each completed SC module asking about their participation in the SC program. Second, a checklist was created to measure home visitor fidelity to the PACT for SC procedures. The checklist included 11 close-ended questions that assessed the use of each PACT for SC strategy or tool, with space for comments. Following the end of each session with parents for whom they were using PACT for SC, home visitors completed a fidelity form and submitted the form and any other completed PACT for SC worksheets to the research team. In addition, as part of their coaching role, supervisors attended some sessions in which PACT for SC was being used. At the end of these sessions, the supervisors completed and submitted a fidelity form regarding the home visitor’s fidelity to the PACT for SC protocol. The checklist was scored by calculating a frequency count of each PACT for SC tool utilized across all submitted forms. Separate counts were calculated for each rater type (home visitor or supervisor). In addition, as parents receiving PACT for SC were approaching completion of SC, all were invited to take part in a 15–30 minute feedback interview. Finally, at the conclusion of the seven-month training period, participating home visitors and their supervisors were invited to participate in separate 60-minute feedback focus groups. At the time of their invitation to provide qualitative feedback, parents, home visitors, and supervisors were also asked to complete an accompanying survey regarding perceptions of PACT-enhanced SC.

Measures

Phase I: Adapting PACT for SC

The two measures for this phase are described below. They include a parent interview guide and a home visitor focus group guide.

Parent interview guide

This guide was divided into three sections: 1) questions about the parent’s relationship with their home visitor; 2) general impressions of SC, including challenges; and 3) impressions of their home visitor’s use of various engagement strategies within the original version of PACT. The first and second sections were included to collect parents’ perspectives on the parent-home visitor relationship and the delivery of SC for use in the adapted PACT for SC training for home visitors. The third section was included to provide the research team with key information to inform the adaptation of the original PACT tools for use in SC.

Home visitor focus group guide

The home visitor focus group questions were divided into four sections: 1) perceptions of the role of the home visitor in services; 2) perceptions of the goals of home visiting; 3) perceived parent engagement challenges and strategies used to address those challenges; and 4) review of the original PACT tools. The first two sections were created in collaboration with the study team’s SC supervisor partner to facilitate rapport building with participants prior to obtaining feedback about the PACT tools. The third and fourth sections were included to generate key information to inform the adaptation of the original PACT tools for use in SC.

Phase II: Examining PACT for SC

The four measures for this phase are described below. They include three quantitative measures and one qualitative measure.

Parent Participation Engagement Measure (PPEM; Haine-Schlagel et al., 2016)

This five-item self-report measure assesses parent participation during the current session and was administered at the end of the first module of SC delivery (about 6–8 weeks after enrollment). The Likert scale responses, scored on a 5-point scale, ranged from “Not at all” to “Very much.” Content of items included assessment of the degree to which the parent asked their home visitor questions, shared their opinion or point of view with the home visitor, took part in home visit activities, took part in planning for homework, and agreed with the homework plan. The Cronbach’s alpha in the current sample was 0.88. In a sample of 1374 parents whose children were receiving publicly-funded mental health services, model fit indices and factor loadings supported a one-factor model and internal consistency reliability was strong across English and Spanish versions (.86 and 0.88, respectively; Haine-Schlagel et al., 2016). In addition, the measure has demonstrated some convergent validity across a range of factors associated with parent participation and discriminant validity across service type (Haine-Schlagel et al., 2016).

SC homework fidelity

The SC protocol includes assignment of homework at the end of the second through fifth sessions in each module (modules consist of six sessions). As noted earlier, parents completed an ICT study fidelity survey at the end of each session. As part of this survey, parents reported whether the home visitor had assigned homework. The fidelity survey was adapted from the National SafeCare Training and Research Center training checklists and the entire fidelity measure has been validated over the course of SC implementation (Chaffin et al., 2016). Homework fidelity was averaged across sessions in which homework was intended to be assigned. The question reads “My home visitor gave me homework to do before the next home visit.” Response options were “not at all,” “some,” and “a lot.” The “some” and “a lot” responses were combined to indicate that homework was assigned at that session.

Qualitative feedback guides

The parent interview questions inquired about the following separately for each PACT for SC tool (copies of each tool were provided as prompts): 1) general perception of the tool; 2) use of the tool; and 3) ideas for improving the tool. The home visitor and supervisor focus group questions asked about: 1) acceptability, appropriateness, and perceived effectiveness of the toolkit and ideas for improvements; 2) acceptability and utility of the PACT for SC training and barriers to completing the training; and 3) sustainment plans. Both guides were based on feedback guides from the original PACT study (Haine-Schlagel, Mechammil, & Brookman-Frazee, 2017).

Quantitative feedback surveys

The parent feedback survey included 15 questions that addressed the home visitor’s use of PACT for SC and perceived effectiveness of the tools. Four items that assessed perceptions of home visitors’ use of four specific engagement strategies that are part of PACT for SC (see Table 2) were analyzed as an average composite, with an alpha of 0.87. Similarly, four items that assessed the degree to which parents recommended keeping each PACT for SC tool in SC were analyzed as an average composite, with an alpha of 0.97. Two items each assessed perceptions of three PACT for SC tools (Organizational Tools, End of Visit Worksheet, and Strengths Worksheet; see Table 2). Each pair was combined into an average composite based on correlations of 0.63, 0.74, and 0.86, respectively. The remaining item was analyzed independently. The home visitor and supervisor feedback survey consisted of 47 items and assessed perceptions of: 1) PACT for SC training acceptability, appropriateness, utility, and effectiveness; and 2) toolkit acceptability, appropriateness, utility, effectiveness, and sustainment plans. After the 47 items were analyzed, 21 items were retained because of their strong internal consistency and salience. Some were transformed into composites: toolkit appropriateness (3 items; alpha of 0.74); training acceptability (six items; alpha of 0.87); and plans for future use (3 items; alpha of 0.86). Similar to the interview guides, the feedback surveys were based on the original PACT study measures (Haine-Schlagel et al., 2017).

Table 2.

PACT for SC Adaptation

Original Tool	Adapted Tool	Description of Adapted Tool	Challenge(s) Addressed/Adaptations Made (from qualitative feedback)
Alliance, Collaboration, and Empowerment (ACEs) Engagement Strategies	Partnership, Empowerment, and Collaboration (PECs) Engagement Strategies	One Partnership strategy: Talk about being partners in learning SafeCare One Empowerment strategy: Comment on strengths and effort Two Collaboration strategies: Ask for input and create opportunities to talk about challenges learning SafeCare	-Too many sub-strategies in original; reduced number to four. -ACE acronym has different meaning in child welfare context; changed to PECs.
Workbook (overall)	Organizational Tools	Folder, notepad, pen, magnet, and tip sheet for asking questions	-For home visiting context a folder kept at home was more appropriate than original binder kept by provider. -Too many informational tip sheets in previous version; kept just one about asking questions and reduced content.
Workbook (I Have Strengths as a Parent Activity)	My Strengths as a Parent Worksheet	Worksheet completed by parent and home visitor together to highlight parents’ strengths and effort to promote empowerment and activation	-Too much content and not all appropriate for context; reduced questions. -Administered at beginning and end of parenting module to help assess changes in perceived strengths.
Workbook (My Point of View Activity)	Parent Perspectives Worksheet	Worksheet to learn about parent and tailor interactions to parent’s beliefs and concerns	-Too much content and not all appropriate for context; reduced questions.
Action Sheet	End of Visit Worksheet	Worksheet for end of each session to summarize session in parent’s own words and collaboratively plan for completing homework	-Too much content and not all appropriate for child welfare context; reduced questions.
Training Package	PACT for SC Training Package	1) one 2-hour in-person training with training vignettes; 2) seven 1-hour in-person consultations held monthly; 3) weekly email tips; 4) manual	-Changed observed feedback to self/supervisor-reported fidelity because recording not feasible for population. -Shortened workshop length and reduced frequency of consultations to fit home visitors’ schedules. -Updated content to reflect tool adaptations -Integrated tools into existing SC curriculum and created training materials to communicate integration.

Data Analyses

Phase I: Adapting PACT for SC

Given the need to utilize the feedback collected in a timely manner to inform the adaptation of PACT for SC, interviews and focus groups were analyzed using elements of the Rapid Assessment Procedures framework (RAP; Beebe, 2001; Palinkas & Zatzick, 2019). The first step was to form a multidisciplinary team that included a clinical psychologist (first author), a sociologist (second author), and an educational psychologist, with consultation from the study team’s SC supervisor partner. This team then created a summary template from the main topics or domains in the guides. Two independent coders [sociologist or educational psychologist who conducted the interview/focus group and a research assistant (third author)] independently reviewed each recording. The educational psychologist, who has expertise in implementing RAP, provided training in coding procedures. Coders then independently completed summary templates for each recording. The templates were compared and any inconsistencies were discussed with a third member of the research team (first author). Finalized summary templates for each stakeholder were entered into a matrix to allow results to be compared and contrasted to generate primary adaptation themes. The results were distilled into actionable items and then triangulated through consultation with SC developers. Results then informed the PACT for SC adaptation. This method allows for rapid, real time modifications to be made to interventions and implementation efforts (Curran, Bauer, Mittman, Pyne, & Stetler, 2012) and has been demonstrated to yield reliable results when compared to more in-depth qualitative coding (Gale et al., 2019).

Phase II: Examining PACT for SC

First, inferential statistical analyses were conducted to compare the PACT for SC group of parents to the comparison group of parents on the PPEM and homework fidelity measures. Stata v.13 was used to conduct independent samples t-tests for this purpose. Second, mixed method analyses were conducted to examine the study’s research questions about the implementation of PACT for SC. More specifically, the complementarity function was employed, such that the quantitative data were used to provide breadth of understanding while the qualitative data provided depth of understanding (Palinkas et al., 2011). The specific constructs examined across the mixed method feedback data included acceptability, utility, appropriateness, feasibility, fidelity, and plans for sustained use of PACT for SC (Proctor et al., 2011). The same RAP method was utilized to analyze these qualitative data as described earlier (Beebe, 2001; Palinkas & Zatzick, 2019).

Results

Phase I: Adapting PACT for SC

Overall perceptions about the PACT tools and their relevance for SC implementation were positive across stakeholders. See Table 2 for a description of each original tool, PACT for SC tool, and comments on the primary adaptations based on the qualitative feedback from parents and home visitors. The final PACT for SC included a set of engagement strategies (Partnership, Empowerment, and Collaboration or PECs), a set of Organizational Tools, an End of Visit Worksheet, a My Strengths as a Parent Worksheet, and a Parent Perspectives About the Program Worksheet. In addition, adaptations were made to the training package based on home visitor feedback and consultation with the team’s community member and SC developers. These changes are also indicated in Table 2.

Phase II: Examining PACT for SC

Impact on parent in-session participation and homework fidelity

Compared to parents who did not receive PACT for SC, parents who did receive PACT for SC reported significantly higher levels of participation in sessions with the same home visitor (PACT for SC M = 4.27, SD = 0.66; comparison M = 3.53, SD = 0.94; p = .036; d = .91). Further, home visitor fidelity to assigning homework was higher for cases in which the home visitor utilized PACT for SC compared to cases in which the home visitor did not utilize PACT for SC (PACT for SC M = 0.87, SD = 0.23; comparison M = 0.65, SD = 0.40; p = .047; d = .67).

Impact on parent between-session homework completion

Unfortunately, this study was unable to empirically compare parents’ between-session homework completion because the ICT study did not collect that information. However, at the first PACT for SC consultation session, participating home visitors were asked as a group the percentage of sessions overall in which their parents completed homework outside of sessions and their collective answer was 50% of the time. Homework completion outside of sessions was then tracked through responses to a question on the PACT for SC fidelity form. Results showed that parents completed homework 72.3% of the time when assigned, which was much higher than the baseline 50% estimate.

Parent perceptions of PACT for SC tools

Thirteen parents or 72% of the sample participated in feedback data collection. Overall, parents indicated that the PACT for SC tools were acceptable and useful (see Table 3). Parents reported overall acceptability (demonstrated by their recommendation that the tools be kept as part of SC), and indicated some challenges related to only one specific tool (parent perspectives worksheet). Parents were consistently positive about the toolkit’s utility, finding the worksheets to be helpful and demonstrating particular enthusiasm for the End of Visit worksheet, which helped parents maintain a connection with the curriculum between sessions as well as demonstrate the progress they were making in the program to their caseworker.

Table 3.

Study Mixed Method Feedback Results from Parents, Home Visitors, and Supervisors

Stakeholder	Theme	Quantitative Results	Qualitative Subtheme
PACT for SC Tools
Parents	Toolkit Acceptability	-Parent recommend keeping tools in SC composite M = 4.20 (SD = 1.11); 90% agree or strongly agree	-Overall acceptable but some challenges with specific tools (e.g., Parent Perspectives questions difficult because asked too early in program).
Parents	Toolkit Utility	-Parent End of Visit Worksheet helpful composite M = 4.35 (SD = 1.01); 84.6% agree or strongly agree	-Most enthusiastic about End of Visit Worksheet helping remember what to do between sessions and was useful to document service participation for caseworker.
		-Parent Strengths Worksheet helpful composite M = 4.25 (SD = 1.03); 80% agree or strongly agree
		-Parent Perspectives Worksheet helpful item M = 4.42 (SD = 0.67); 91.7% agree or strongly agree.
		-Parent Organizational Tools helpful composite M = 3.79 (SD = 1.27); 53.8% agree or strongly agree.
Home Visitors	Toolkit Acceptability	-HV satisfaction with PACT tools item M = 3.8 (SD = 0.98) with 3 out of 6 agreeing or strongly agreeing.
Home Visitors	Toolkit Utility	-HV PACT helps SC be more effective item M = 3.00 (SD = 0.89); 33.3% agree or strongly agree	-Tools were useful; in particular recommended using Strengths Worksheet more frequently
		-HV PACT useful for parents receiving SC item M = 3.67 (SD = 0.82); 50% agree or strongly agree	-Some parents usedOrganizational Tools and End of Visit Worksheet between sessions.
		-HV PACT useful for parents receiving other HV services item M = 4.00 (SD = 0.89); 66.7% agree or strongly agree	-Fidelity form a helpful reminder to support parents and use tools.
		-HV PACT increased my skills to help parents learn SC item M = 2.50 (SD = 0.84); 16.7% agree or strongly agree	-PACT would be more helpful for -HVs just learning SC.
		-HV PACT increased my skills in working with parents item M = 2.83 (SD = 0.98); 33.3% agree or strongly agree	-PACT may be more useful for less structured home visiting interventions.
		-HV PACT helps parents learn SC M item = 2.50 (SD = 1.64); 33.3% agree or strongly agree
		-HV PACT helps parents participate item M = 2.83 (SD = 1.33); 33.3% agree or strongly agree
Home Visitors	Toolkit Appropriateness	-HV appropriateness composite M = 3.56 (SD = 0.50); 50% agree or strongly agree	-Sessions took longer.
		-HV compatibility with program mission and values item M = 3.17 (SD = 1.17); 50% agree or strongly agree	-Some repetition with the existing SC curriculum.
Home Visitors	Toolkit Feasibility		-Toolkit more feasible if better integrated into SC curriculum.
Supervisors	Toolkit Acceptability	-Supervisor satisfaction with PACT tools item M = 3.67 (SD = 0.58); 66.7% agree or strongly agree
Supervisors	Toolkit Utility		-Tools were useful; in particular recommended using Strengths Worksheet more frequently.
			-Fidelity form a helpful reminder to support parents and use tools.
			-End of Visit Worksheet resulted in more homework completion outside of sessions.
Supervisors	Toolkit Appropriateness	-Supervisor appropriateness composite M = 3.89 (SD = 0.51); 66.7% agree or strongly agree	-Perceived some repetition with the existing SC curriculum.
		-Supervisor compatibility with program mission and values item M = 3.67 (SD = 1.16); 33.3% agree or strongly agree	-Highly supportive of PACT fitting SC, population they serve, and their program. However, recommended PACT be used as needed (e.g., for quiet parents) rather than universally.
Supervisors	Toolkit Feasibility		-Would be more feasible if better integrated into SC curriculum.
			-Feasibility would have been enhanced if HVs were using toolkit with all parents during.
PACT for SC Training
Home Visitors	Training Acceptability	-Home visitor (HV) training acceptability composite M = 3.86 (SD = 0.65); 50% agree or strongly agree	-Training useful overall but no perceived need for ongoing consultations; not enthusiastic about role-play practices.
Home Visitors	Training Utility		-Useful for trainers to be open to feedback throughout training period about ways to enhance materials and training.
Supervisors	Training Acceptability	-Supervisor training acceptability composite M = 3.91 (SD = 0.12); 50% agree or strongly agree	-Consultations helpful for ongoing training and buy-in; wanted more modeling and practice.
Supervisors	Training Utility		-Useful for trainers to be open to feedback throughout training period about ways to enhance materials and training.
PACT for SC Sustainment Plans
Home Visitors	Toolkit Sustainment Plans	-HV plans for future use composite M = 3.0 (SD = 1.15); 33.3% agree or strongly agree	-Plans to continue to use Strengths Worksheet.
Supervisors	Toolkit Sustainment Plans		-Interest in incorporating questions from worksheets into services, primarily intake assessment.

Note: HV = Home Visitor.

Home visitor perceptions of PACT for SC tools

Home visitors indicated that the tools themselves were moderately acceptable, useful, appropriate for their practice, and feasible to use (see Table 3). Home visitors were somewhat satisfied with the toolkit, found some of the specific tools useful to enhance parent participation, and reported that parents used the Organizational Tools and End of Visit Worksheet between sessions. However, home visitors perceived that they already worked effectively with parents and that, overall, PACT for SC did not enhance their skills to increase parents’ participation or follow the SC protocol. Home visitors instead perceived PACT for SC to be potentially helpful for home visitors just learning to deliver SC and/or for less structured home visiting programs. Regarding the appropriateness of the PACT for SC tools, home visitors indicated they were neutral to positive about the fit of PACT for SC within their agency. Identified concerns were that sessions took longer and there was some repetition with the existing SC protocol. Home visitors recommended the need for further integration of PACT into the SC curriculum and training materials to increase its feasibility.

Supervisor perceptions of PACT for SC tools

Similar to home visitors, supervisors indicated that the tools themselves were moderately acceptable, useful, appropriate for their setting, and feasible to use (see Table 3). Supervisors were somewhat satisfied with the toolkit and perceived the toolkit to be useful in enhancing parent participation. Supervisors also mirrored the anecdotal evidence provided at baseline regarding an increase in parents’ completion of assigned homework between sessions for parents who received PACT for SC. Supervisors indicated they believed that PACT for SC fit within the SC protocol, the population served, and the agency mission. However, supervisors recommended that PACT for SC be used as needed rather than universally. Supervisors shared home visitors’ concerns about conceptual overlap between PACT for SC and the existing SC curriculum and agreed that PACT for SC was not optimally integrated into the curriculum. Supervisors felt feasibility would have been enhanced if home visitors were using the toolkit with all their parents during the current study.

Perceptions of PACT for SC training

Overall, home visitors demonstrated that the training protocol was feasible to implement and reported perceiving the PACT for SC training protocol as acceptable and useful. Regarding training feasibility, home visitors did attend the in-person meetings, with five home visitors (83%) attending the initial in-person training and three home visitors (50%) attending all seven in-person consultation sessions (average in-person consultation attendance = 5.1). Further, home visitors participated in PACT for SC delivery, submitting 182 session fidelity forms across the 18 parent participants during the seven-month training period. As Table 3 indicates, home visitors perceived the training protocol as moderately acceptable. They reported that the most helpful component of the training was the openness of the trainers to making small adjustments to the materials and training protocol based on their feedback to enhance training effectiveness. However, home visitors did not see the need for the consultations and did not feel comfortable engaging in role-play practice as part of the consultation sessions.

Similar to the home visitors, supervisors perceived the PACT for SC training protocol as acceptable and useful, although in contrast to the home visitors, supervisors wished that the consultation sessions included more role-plays (see Table 3). Supervisors also reported that the most helpful component of the training was the openness of the trainers to making small requested adjustments to the materials and training protocol.

Home visitor fidelity to the PACT for SC delivery protocol

Per fidelity forms, in 94.4% of cases home visitors reported giving out the Organizational Tools, and in 94.4% and 100% of cases, respectively, the home visitor submitted a Parent Perspectives Worksheet or Strengths Worksheet. Home visitors submitted 178 End of Visit Worksheets during the training period. In terms of self-reported PECs usage (see Table 2), home visitors reported average ratings of 1.78, 1.86, 1.83, and 1.79 for Partnership, Empowerment, Collaboration open-ended questions, and Collaboration challenges, respectively, on a 0 to 2-point scale. Supervisors submitted 15 fidelity forms. Supervisor and home visitor responses submitted for the same session (n = 9 sessions) yielded almost perfect agreement (98.1%). In addition, parents reported in their feedback survey that home visitors frequently used the PECs strategies (composite M = 4.04; SD = 1.05; 10 out of 13 parents agreeing or strongly agreeing).

Home visitor and supervisor plans for PACT for SC sustainment

As shown in Table 3, home visitors reported some plans to continue using some or all of PACT for SC, with a particular interest in continuing to use the Strengths Worksheet. Supervisors also indicated an interest in incorporating many of the questions from the PACT for SC worksheets into broader service delivery.

Discussion

The results of this study indicate that both parent in-session and between-session participation were higher in cases in which PACT for SC was used compared to cases in which PACT for SC was not used. Specifically, parents who received PACT for SC reported that they were more actively engaged in sessions by asking questions, sharing their opinion, and participating in homework planning, among other behaviors, compared to parents who did not receive PACT for SC. Further, home visitor assignment of homework as designated in the SC curriculum was significantly higher, perhaps a testament to the use of the End of Visit worksheet that was designed to facilitate between-session participation. In addition, as hypothesized, home visitors reported more between-session homework completion for PACT for SC parents than anecdotally reported for their previous parents who did not receive PACT for SC. Supervisors, who perceived that PACT for SC parents completed more homework between sessions than other SC parents, supported this finding. Additionally, home visitors indicated parent use of PACT for SC tools between sessions, such as the End of Visit Worksheet.

Parents perceived PACT for SC to be acceptable and useful to increase their participation in SC. Home visitors and supervisors reported moderate PACT for SC acceptability and some conditions under which PACT for SC would be useful and effective. Supervisors perceived PACT for SC to be an appropriate fit for their agency and the population it serves. Findings also indicate that home visitors found PACT for SC training acceptable, useful, and feasible. Home visitors and supervisors reported some planned sustainment of use of PACT for SC tools. Study strengths include utilization of parent and home visitor feedback to inform the PACT adaptation process and the use of a measure of in-session participation with strong psychometric properties, which is not common in the parent participation literature (Haine-Schlagel & Walsh, 2015).

The findings regarding the positive impact of PACT for SC on parent in-session participation are consistent with the previous PACT randomized study, which found increases in observed parent participation for families receiving PACT as part of standard community-based child mental health care compared to families receiving standard care only (Haine-Schlagel et al.,, 2018). The current study’s focus on a structured EB home visitation program suggests that similar yet adapted tools can impact participation across services and contexts. These results are particularly relevant for home visitation services given parent participation has been hypothesized as one factor that suppresses the positive effects of home visitation programs (Carta et al., 2013; Gomby et al., 1999). Thus, these results demonstrate the potential promise of participation enhancements such as PACT for SC to improve effect sizes for home visitation programs (Sweet & Appelbaum, 2004). Further, as Hoagwood et al. (2014) have indicated, parents who are successful participants in services are more likely to utilize needed services in the future. The results of this study suggest the possibility that participation enhancements such as PACT for SC can have long-term impacts for families and for service systems. It is possible that the opportunities the PACT for SC training provided for ongoing professional development and peer support may serve as potential alternative explanations for the positive results. However, the evidence of fidelity to the delivery of PACT for SC as intended lends support for the notion that the toolkit itself contributed to these improvements beyond the more process elements of the PACT for SC training protocol.

The significant increase in home visitors’ assignment of homework as delineated in the SC curriculum preliminarily suggests that participation enhancements, which include a focus on supporting home visitors’ assigning of homework, may improve fidelity to that common component of EB programs. In addition, a recent study demonstrated that parents’ in-session participation can contribute to home visitors’ perceived ability to maintain program fidelity, suggesting the possibility that the observed improvement in parent in-session participation may have contributed to the increase in fidelity to homework assignment (Gellatly et al., 2019). As noted earlier, homework assignment is critical to increase the likelihood that behavior changes that may occur as a result of the program are internalized and generalized. Thus, participation-enhancement strategies like PACT for SC may potentially result in a higher dose of intervention being delivered and thus better outcomes.

The positive parent perceptions of PACT for SC are consistent with parents’ perceptions of the original PACT intervention provided in the standard community-based child mental health treatment context (Haine-Schlagel et al., 2017), and suggest that, in general, parents are amenable to the use of tools to encourage and support their participation in services. More specifically, PACT for SC may have addressed a communicated need by parents for home visitors to tailor SC delivery (Gallitto, Romano, & Drolet, 2018).

The home visitors’ more equivocal perceptions of PACT for SC were neither consistent with the parents’ perceptions, nor entirely consistent with the original PACT study. Original PACT study providers were all licensed mental health professionals or advanced trainees who were providing services as usual, while the home visitors in the current study were unlicensed home visitors delivering a very structured program that does not allow for a great amount of flexibility in delivery. It is possible that the home visitors’ perceptions of PACT for SC in part reflected equivocal perceptions about SC in general. Previous research has suggested that providers who have less positive attitudes towards EB programs in general are more likely to perceive parent engagement challenges (Lewis & Simons, 2011). Perhaps participation enhancements to structured, EB programs may require additional attention to providers’ experience of the program. In addition, both home visitors and supervisors shared concerns about the conceptual overlap between PACT for SC and SC itself, as well as the need for greater integration of PACT for SC strategies and tools into the SC curriculum. These results highlight the need to integrate engagement enhancements such as those focused on parent participation and homework assignment fidelity into existing intervention materials. Results also suggest the need to attend separately to training home visitors just learning an intervention versus home visitors already delivering an intervention. Further, observed increases in parent in-session participation and fidelity to homework assignment when using PACT for SC may be considered more meaningful given the home visitors’ somewhat ambivalent perspectives about the toolkit.

The process of completing this study generated some notable overall learnings consistent with inner and outer context factors that can impact implementation of EB programs (Aarons, Hurlburt, & Horwitz, 2011). For example, other system demands, such as increased paperwork requirements that were implemented simultaneously with this study, affected receptivity to the practice changes associated with PACT for SC. However, it appeared that the lack of receptivity to PACT for SC on the part of the home visitors was mitigated to an extent by the high degree of supervisor buy-in (e.g., supervisor completion of fidelity forms, attendance at consultation meetings), as evidenced by home visitor attendance at consultations and submission of fidelity forms.

Future directions include conducting a larger scale study to examine the impact of PACT for SC that improves upon the initial set of materials and training protocols, in particular the integration between PACT for SC and the SC curriculum. Utilizing a randomized controlled trial design within multiple agencies will be key to expand understanding of how PACT for SC can improve participation in services. Further, a larger study can include measurement of desired program outcomes to evaluate whether increases in parent in-session participation and/or homework assignment fidelity translate into more positive outcomes and what degree of participation may be necessary to achieve such outcomes. Additional future directions include adapting PACT for SC for other home visitation programs and integrating additional EB engagement strategies such as text messaging (Carta et al., 2013; Lefever et al., 2013).

Limitations

Several important limitations should be noted. A primary limitation is the small sample size, which mitigated statistical power, the generalizability of the findings, and the ability to examine mechanisms of change to inform future efforts to promote parent participation in home visitation services. In addition, the data were nested and, due to power limitations, the statistical analyses employed were unable to account for that nesting. Further, the lack of random assignment of families introduces the potential for the two groups to be different in systematic ways. Given the scope of this study allowed for implementation in just one agency, the ability to generalize to other agencies that provide home visitation services is limited, in particular given the documented contributions of agency- and provider-level factors to variability in the delivery of home visitation services (Latimore et al., 2017). Another limitation was the inability to collect audio or video recordings of SC sessions that included PACT for SC to both inform the ongoing consultations and evaluation of toolkit fidelity. Further, no measure of homework completion was completed as part of the larger study so changes in homework completion following implementation of PACT for SC could not be empirically evaluated. In addition, parent participation in home visitation services is a dynamic construct that can change over time (Lefever et al., 2013); the measurement of participation at one time point rather than over time limits the knowledge that can be gained regarding the potential impact of PACT for SC.