Skip to main content
NCBI home page
NIHPA Author Manuscripts logoLink to NIHPA Author Manuscripts
. Author manuscript; available in PMC: 2026 Apr 22.
Published in final edited form as: J Educ Psychol Consult. 2025 Sep 16;36(1):130–153. doi: 10.1080/10474412.2025.2556756

Adaptation of a fidelity monitoring tool for online delivery of the Family Check-Up

Katherine A Hails a, Amanda Trujillo b, Audrey CB Sileci b, Elizabeth JS Bates b, Anne Marie Mauricio a,b, Elizabeth A Stormshak a,b
PMCID: PMC13099231  NIHMSID: NIHMS2149501  PMID: 42023134

Abstract

Motivational interviewing (MI) strategies have been incorporated into several parenting interventions designed for in-person delivery, with evidence that such strategies are critical for enhancing treatment engagement and outcomes. An online adaptation of the Family Check-Up (FCU), an evidence-based parenting intervention grounded in MI, was developed to increase families’ access to evidence-based parenting support. The FCU Online is a digital health intervention that includes evidence-based parenting content delivered via a mobile app and tailored telehealth coaching based in MI principles. In the current study, our goal was to adapt the COACH, an observational tool designed to assess fidelity for the in-person version of the FCU, to fit the online coaching model. Like the original, the adapted COACH assesses different domains of fidelity to the FCU model, including competence in MI skill utilization and adherence to structural aspects of the model. We used the adapted COACH to code a sub-sample of audio-recorded FCU Online coaching sessions (n=134) from a recent randomized controlled trial of the FCU Online for early childhood. Our second aim was to test whether COACH domains assessing MI skills versus other aspects of fidelity were more consistently associated with caregiver intervention engagement. We found that COACH fidelity ratings were associated with caregiver intervention engagement in both the app and coaching, with COACH domains assessing MI skills more consistently associated with engagement. Overall, our findings provide support for the utility of a fidelity coding tool for a flexible, MI-based digital health intervention that can be delivered in educational settings.

Keywords: digital health, motivational interviewing, fidelity

Online behavioral health interventions have potential to fill critical gaps in service delivery for children, youth, and families (Brian et al., 2022; Tan-MacNeill et al., 2021), with many online behavioral parenting interventions associated with significant improvements in child behavior, parenting skills, and parenting self-efficacy (McAloon & Armstrong, 2024). Broadly speaking, an online parenting intervention is any form of parenting support that is delivered online, including through websites, videoconferencing, and apps (Canário et al., 2024). There is increasing interest in online parenting interventions that offer self-directed engagement in asynchronously delivered digital content, often via mobile app, as they may increase the reach and accessibility of evidence-based parenting supports (McAloon & de la Poer Beresford, 2023).

Consistent with the supportive accountability theoretical model (Mohr et al., 2011), there is mounting evidence that online parenting interventions offering self-directed treatment are most effective when they offer live coaching support, delivered via telehealth (Connell et al., 2024; Day & Sanders, 2018; McAloon & Armstrong, 2024). Theoretically, motivational interviewing (MI) is an excellent fit for online parenting programs that include self-directed treatment paired with coaching support. In such a delivery model, parents receive evidence-based instruction on parenting skills and child behavior via a digital program, with MI-based coaching to enhance parent motivation to engage in digital content, practice skills at home, and set goals for themselves and their families (Connell & Stormshak, 2023). Briefly, MI is an approach to communication that involves promoting clients’ individualized desires to pursue their own goals and make change (Miller & Rollnick, 2013) (see Introduction to the Special Issue for additional background and details on MI). MI strategies have been incorporated into several parenting interventions designed for in-person delivery, with a plethora of evidence that such strategies enhance treatment engagement, as measured by session attendance (Berkel et al., 2021; Chaffin et al., 2009; Nock & Kazdin, 2005; Sterrett et al., 2010), treatment adherence (Nock & Kazdin, 2005) and parent satisfaction (Sterrett et al., 2010), and improve outcomes for families (e.g., reducing child maltreatment (Chaffin et al., 2011). However, despite MI’s potential impact for engaging parents in online parenting interventions, there is significantly less research exploring how MI is used in these intervention models. Understanding the extent to which MI facilitates parents’ engagement in online parenting interventions is important as digital programming proliferates, particularly given research suggesting that parent participation in online programs without coaching is somewhat limited (Connell et al., 2024; Day & Sanders, 2018).

The Family Check-Up (FCU), and its digital adaptation, the FCU Online, are evidence-based family-centered interventions that focus on parenting skills and are grounded in the use of MI (Dishion et al., 2008), an essential mechanism of change in the model. Specifically, FCU clinicians’ use of MI strategies has been found to be associated with greater client engagement, which is subsequently linked with greater use of positive parenting skills and reductions in child problem behavior (Chiapa et al., 2015; Smith et al., 2013). There are several other online parenting interventions that offer telehealth or phone coaching support to encourage families to engage in digital parenting content modules, assist parents in applying new skills, and respond to questions (e.g., Sourander et al., 2016). However, to our knowledge, the FCU Online is the only online parenting intervention that formally incorporates MI into its coaching model. Specifically, coaches use motivational interviewing skills and strategies to engage families, set goals, and highlight strengths (Hails et al., 2025).

Intervention Model: The FCU Online

The FCU Online was adapted from the FCU in-person model to enhance the intervention’s accessibility, particularly for communities and settings with limited resources and staffing. The original FCU is a brief, family-centered, parent-directed, MI intervention (Dishion & Stormshak, 2007). Caregivers participate in an ecological assessment, which typically consists of a strengths-based intake interview, caregiver completion of standardized questionnaires about parenting, child behavior, and contextual factors (e.g., caregiver stress, mental health), and videotaped parent-child interaction tasks. Following the assessment, the clinician engages the caregiver in a feedback session where they collaboratively review the results of the assessment, focusing on family strengths and the parent’s goals and motivation for change. After the feedback, caregivers can elect to engage in follow-up sessions focused on the specific parenting skills and family goals that are a priority for their family, guided by the Everyday Parenting Curriculum (Stormshak, Mauricio, et al., 2024). MI strategies are used to engage caregivers at every step of the FCU.

The FCU Online program consists of a web-based application with online, self-paced, asynchronous evidence-based parenting content paired with telehealth coaching sessions. There are two versions of the FCU Online, with the early childhood version intended for caregivers of toddler to preschool-age children and the school-age version intended for parents of school-age and early adolescent youth (see Supplemental Material for a comparison of the early childhood and school-age versions). The FCU Online is intended as a preventative intervention for families at risk of experiencing challenges with child behavior or parenting due to factors such as parent mental health problems and family stress. The early childhood version was specifically designed to support parents of young children with a history of substance use and mental health challenges (Stormshak et al., 2021), as research suggests that their children are at greater risk of experiencing behavioral problems (Hser et al., 2015) and parents have greater difficulty accessing behavioral healthcare for their families (Feder et al., 2018). Furthermore, direct contact with an interventionist through telehealth coaching may be particularly important for families experiencing such social disadvantages that function as barriers to intervention engagement (Harris et al., 2020).

The online content for both versions draws from the Everyday Parenting Curriculum (Stormshak, Mauricio, et al., 2024) and is organized into five distinct modules. Each of the five modules incorporates assessment, feedback, and parenting skills content. At the beginning of each module, caregivers respond to a brief series of questions regarding their use of the skills that are targeted in that module, and immediately after they receive an auto-generated visual feedback summary. The intent of the feedback summary is to help caregivers identify areas of strength and skills they may want to focus on during their participation in the FCU Online. Telehealth coaching sessions are intended to flexibly correspond with caregivers’ completion of each of the five app modules. The goals of coaching sessions are to motivate continued engagement with the online content, help caregivers understand how skills presented in the online content connect to their individual goals for their child and family, and support and tailor their use of these skills. The FCU Online is available in English and Spanish and includes an easy-to-use integrated online administration website that includes a portal designed specifically for coaches to view the extent to which families engage with the online program. Both the early childhood and school-age versions of the FCU Online programs (i.e., app plus telehealth coaching) have demonstrated efficacy in improving parenting skills (Connell & Stormshak, 2023; Hails et al., 2025), with the school-age version also associated with significant improvements to child behavior (Stormshak, Connell, et al., 2024); child outcomes have not yet been tested for early childhood version.

COACH Fidelity Monitoring Tool

The COACH is an observational tool designed to assess fidelity to the FCU model (Dishion et al., 2010). COACH is an acronym for the five fidelity domains assessed: 1) Conceptually accurate and adherent to the model, 2) Observant and responsive to the family’s needs, 3) Active in structuring the session, 4) Careful when teaching and providing feedback, and 5) Helpful in building hope and motivation. The COACH assesses fidelity to the FCU across two overarching dimensions: Adherence (i.e., the extent to which the intervention protocol is followed) and Competence (i.e., how skillfully the intervention is delivered to promote participant engagement and behavior change) (Cross & West, 2011), with the “competence” domain captured primarily in COACH domains 2 (Observant and responsive to the family’s needs) and 5 (Helpful in building hope and motivation). Although all five COACH domains assess the extent to which MI strategies are effectively incorporated into sessions, Observant and responsive and Helpful in building hope and motivation most explicitly capture the provider’s use of MI OARS skills (open questions, affirmations, reflective listening, and summary reflections; Miller & Rollnick, 2013) and Helpful in building hope and motivation assesses the extent to which the provider elicits motivation for change (i.e., change talk). In the original COACH used to assess fidelity of the in-person FCU, each domain is scored separately using a 9-point scale. In addition to behavioral markers of fidelity in each category (e.g., “tailors feedback to caregiver’s education, emotional needs, and cultural background” for Observant and responsive), the COACH also includes a list of behavioral markers characterized as “barriers to effective practice” (e.g., “lectures, steam rolls; disproportionate therapist/client talk ratio” as a barrier for Observant and responsive) incorporated into each domain’s score (see Supplemental Materials for FCU Feedback session COACH rating form).

The COACH is empirically validated, with research suggesting that overall intervention fidelity (i.e., including both competence and adherence domains) improves client engagement, which in turn is positively associated with improvements to parenting and child behavior (Smith et al., 2013). However, because the COACH was designed to assess fidelity to the full FCU model and not solely the MI components, it has not been validated against established MI fidelity measures such as the Motivational Interviewing Treatment Integrity Coding Manual (MITI; Moyers et al., 2014) or the Motivational Interviewing Skill Code (MISC; Miller et al., 2003). When the MITI has been used to assess MI fidelity in the FCU, providers met or exceeded the “fair” benchmark of fidelity for the global scores (DeVargas & Stormshak, 2020; Flack, 2024. Thus, MI is strongly emphasized and embedded within the FCU model, even though the COACH lacks formal validation as a specific measure of MI skills.

There is a great deal of overlap in item content across the behavioral markers for COACH domains Observant and responsive and Helpful in building hope and motivation and established MI measures. Within the Helpful in building hope and motivation domain (see Supplemental Materials for COACH rating form), one of four behavioral markers includes “prompts, evokes, and supports change talk,” similar to one of the four MITI global scores, “Cultivating Change Talk” in which the highest anchor indicates that “clinician shows a marked and consistent effort to increase the depth, strength, or momentum of the client’s language in favor of change” (Moyers et al., 2014). Within Observant and responsive, two of four behavior behavioral markers includes “establishes a collaborative set using reflective listening and empathy,” and “uses language and examples that are those of the caregiver and reflect the family storyline and social context,” both of which combine aspects of MITI global scores “Partnership” (“clinician actively fosters and encourages power sharing in the interaction in such a way that client’s contributions substantially influence the nature of the session”) and “Empathy” (“clinician shows evidence of deep understanding of client’s point of view, not just for what has been explicitly stated but what the client means but has not yet said”), as well as the “Empathy” global score from the MISC (“counselors…show an active interest in making sure they understand what the client is saying, including the client’s perceptions, situation, meaning, and feelings”) (Miller et al., 2003).

Research suggests that providers’ MI-specific skills in the FCU, as assessed by the COACH and MI-specific instruments, are significantly associated with both client engagement and intervention outcomes. Berkel and colleagues (2021) found that providers’ use of MI skills as measured by the COACH predicted caregivers’ in-session engagement, follow-up session attendance, and improvements in motivation to address treatment goals. In other research, FCU providers’ MITI-rated MI skills were associated with decreases in adolescents’ health risk behaviors (DeVargas & Stormshak, 2020) and improvements in parenting skills (Flack, 2024).

Intervention Fidelity and Client Engagement

Findings on the relationship between MI skills implemented in the context of the FCU and client engagement are in line with theory suggesting that providers’ competence in intervention delivery, including both interpersonal process skills and proficient use of MI skills, are associated with in-session client engagement through increasing the therapeutic working alliance (Moyers et al., 2005). There is limited research specifically testing different aspects of fidelity in relation to client engagement, with more work focusing on fidelity and intervention outcomes. Theoretically, it could be that while adherence to intervention ingredients is important for promoting positive intervention outcomes (as long as providers are not overly rigid in their delivery) (Breitenstein et al., 2010), providers’ competence is more predictive of client engagement. A recent systematic review tested relationships between both adherence and competence and participant engagement in parenting interventions, with authors concluding that adherence was not consistently associated with engagement, but competence was positively related to parental satisfaction in interventions (Basha et al., 2024).

Current Study

Need for FCU Online COACH Adaptations

As existing evidence-based parenting interventions transition to an online format with coaching support for greater accessibility, dissemination, and consumer acceptability (Werntz et al., 2023), it is essential to ensure that fidelity assessment tools are appropriately adapted for this novel context, particularly given that the parenting coaches providing support in community settings often have minimal formal mental health training (Rosenberg et al., 2022). In the current study, we adapted the COACH fidelity tool, originally developed for the in-person version of the FCU, to map onto the delivery and content of the digital health model. Given the significant differences between in-person FCU sessions and the telehealth coaching sessions offered with the FCU Online app, it was necessary to adapt the COACH to enhance its applicability as a tool to support provider fidelity to the coaching model. For example, as described above, the in-person FCU has one single feedback session, with subsequent follow-up parenting sessions, and there are two versions of the COACH each specific to the feedback or follow-up sessions. In contrast, the FCU Online coaching sessions typically incorporate both feedback and parent skills teaching into every session. In addition, while the FCU feedback session is typically highly structured to include a number of required components, coaching sessions are shorter (typically 20–30 minutes) and highly variable in content, as interventionists are encouraged to flexibly follow the caregiver’s lead in guiding the session. When coaches engage with caregivers for any given coaching session, a caregiver may have recently completed one parenting skills module in the app, completed all five modules, or have not engaged with the app at all. Moreover, because much of the parenting skills content is embedded in the app, this facilitates Conceptual adherence to content (domain one of the original COACH). Finally, given the anticipated variation in training, background, and skills among FCU Online coaches, there was a need to adapt the COACH to fit the breadth of intended providers delivering the FCU Online, including simplifying the language in the behavioral descriptors, reducing the number of points on the rating scale, and incorporating a more motivating and strengths-based frame into the scoring system.

Overview and Study Aims

Using audio-recorded sessions from a recently completed randomized controlled trial (RCT) of the early childhood version of the FCU Online program, we utilized the adapted COACH to code a sub-sample of sessions (Aim 1) and explore associations between different domains of FCU Online fidelity and caregiver intervention engagement (Aim 2). We used sessions from the RCT of the early childhood program, as opposed to other recently completed trials of the school-age program, because the early childhood sample was more diverse, particularly in terms of socioeconomic and geographic (i.e., urban or rural) status. Although the adapted COACH has thus far only been applied to the early childhood version of the FCU Online, it was developed with the intention for use with the school-age version, as well as for future possible iterations of the FCU Online model, without need for further adaptation. Therefore, the current study is the first to assess fidelity to the FCU Online coaching model.

In addition to describing the process for adapting the COACH for the online version of the FCU, the current study sought to test whether providers’ fidelity to the FCU Online for each of the COACH domains was associated with caregivers’ engagement in the intervention. Consistent with research indicating that therapists’ use of MI skills is associated with greater client engagement (Berkel et al., 2021; Boardman et al., 2006; Smith et al., 2013), we predicted that higher COACH ratings would be significantly associated with time spent in coaching sessions, participation in app modules, and coach-rated engagement during coaching sessions, and that associations with engagement would be strongest for the MI domains (Observant and responsive and Helping in build hope and motivation),.

Method

Participants

Caregivers

Participants in the RCT of the early childhood version of the FCU Online were primary caregivers of children aged 1.5–5 years old (i.e., parents or legal guardians who lived with the child at least 50% of the time) who were randomly assigned to the intervention group (N = 180). To meet eligibility for the study, caregivers had to endorse any one of the following criteria: binge drinking or any recreational drug use (including cannabis) in the past year, lifetime history of opioid misuse, or endorsement of depressive symptoms on the Patient Health Questionnaire-2 (PHQ-2; Löwe et al., 2005). Caregivers were recruited through community partnerships, social media, and flyers. Interested caregivers contacted the study team via phone call, email, or a form on the project’s website. A staff member contacted the caregiver to explain the study, assess eligibility, and review informed consent if eligibility criteria were met. Study procedures for the RCT were approved by the University of Oregon institutional review board office, Research Compliance Services, approval #10212019.029.

Caregivers in the intervention group were predominantly female (94%) and White, Non-Hispanic (73%). The median annual family income bracket was $35,000-$49,999. The average age of caregivers’ target child was approximately 3.5 years old. Approximately one third of caregivers scored above the clinical cutoff on anxiety and/or depressive symptoms; approximately one third endorsed a lifetime history of opioid misuse. Nearly half of caregivers (41%) resided in a rural area, according to whether their reported zip code was designated as rural by the state’s Office of Rural Health’s geographic definitions (Oregon Office of Rural Health, 2024). Nearly 10 percent (9.4%) of caregivers elected to complete research procedures in Spanish.

Coaches

A total of 10 research clinicians were FCU Online coaches in the RCT. Coaches were paired with caregivers upon enrollment. The majority of participants in the intervention group (60%) were paired with one of two licensed clinical social workers who were research study staff and the only full-time coaches on the team. The remaining 40% of participants were paired with coaches who were trainees on the research team (two counseling psychology Ph.D. students, one social work intern, three postdoctoral fellows), as well as one licensed psychologist who was a co-investigator and one licensed professional counselor who was a consultant.

Procedure

Training

All coaches except one had been previously trained and had expertise in the delivery of the in-person FCU model. The current standard model of training for the in-person FCU model is approximately 8–10 hours of online training followed by 9–12 hours of webinar focused on practice delivering the model. All coaches with previous training in the in-person FCU model were trained in the FCU Online using a variety of strategies. The standard FCU Online training model is a 4-hour virtual training. Trainings for both the FCU and FCU Online include didactic instruction and interactive practice in MI skills, including OARS skills (Miller & Rollnick, 2013), with specific practice in open-ended questions, affirmations, reflections, and creating a partnership that honors a client’s values, including compassion, empathy, and acceptance. Coaches met for biweekly peer consultation meetings throughout the duration of the study. The peer consultation meetings served as opportunities for coaches to confidentially share their experiences providing the intervention, addressing coaching challenges and successes, and receive feedback on their coaching. Introduction to and applied review of the adapted COACH was also incorporated into several early peer consultation meetings. Specifically, each coach had at least one opportunity to play one of their recorded sessions for the consultation group. Then, group members coded the tape individually, followed by a group discussion of the scores.

Coaching Sessions

After completing the baseline research assessment, caregivers who were randomly assigned to the intervention group were contacted by their coach. During the initial contact, the coach helped the caregiver to set up the app on their mobile device and completed a brief “get to know you” interview, which was a MI-guided exploration of caregiver perceptions of family strengths and challenge areas to promote reflection on program goals. Subsequent coaching sessions were designed to support and individualize caregivers’ practice of skills learned using the app, with approximately one session devoted to each of the five app modules. Coaches sometimes elected to combine the “get to know you” interview with the coaching session on the first module. The early childhood version of the FCU Online consists of two modules devoted to caregivers’ own mental health and self-care (Wellbeing & Self-Care and Parenting & Substance Use) and three modules covering parenting skills and strategies (Positive Parenting, Proactive Parenting, and Rules & Consequences) (Stormshak et al., 2021).

All coaching sessions were delivered via telehealth and the intended length was approximately 20–30 minutes, with sessions occurring weekly or biweekly for approximately six sessions (i.e., “get-to-know-you” and one session after each module). While sessions could take place over phone or video conferencing, the vast majority of sessions (99%) were conducted over the phone. Coaches were encouraged to deliver the intervention flexibly and in line with caregivers’ needs and preferences. Most caregivers (92%) engaged with coaching sessions beyond the introductory session. Caregivers completed an average of 5.99 (SD = 4.7) coaching sessions. Across all documented sessions, average session length was 29.44 minutes (SD = 15.5). It should be noted that these descriptive statistics are based on coaching engagement for the full sample of participants randomly assigned to the intervention group (N = 180). For more details on intervention delivery for the FCU Online, please refer to Hails et al. (2025).

FCU Online COACH Adaptation Process

Following an initial non-randomized study to pilot the intervention protocol with 10 participants prior to beginning the RCT, a COACH adaptation team was convened to create an initial draft of the COACH for FCU Online coaching, then iteratively adapt the draft while listening to the recorded sessions from the pilot study. The goal of the adaptation was to create a version of the COACH that would be appropriate for assessing fidelity for the FCU Online coaching sessions, but also to develop a more user-friendly version that would be easily integrated into community settings (e.g., schools). As part of the adaptation process, all coaches were invited to provide feedback on item inclusion, scoring, and the wording of COACH items, which contributed to the adaptations described in the section below.

Major Changes to the FCU Online Adapted COACH

Many of the changes to the adapted COACH focused on simplifying wording. For example, “establishes a collaborative set using reflective listening and empathy” from the original COACH was revised to, “uses active listening skills (e.g., summarizing, reflecting) and empathy.” To further streamline and simplify the adapted COACH, and to focus on reinforcing clinicians’ strengths, we also removed the portion of the original COACH containing “barriers to effective practice” for each of the domains. In addition, in the final adapted version of the FCU Online COACH, the first domain, Conceptually accurate and adherent to the model, was changed to Connects conversation to the app, given that much of the “conceptual accuracy to the model” was delivered through the content embedded in the app.

The final major change was using a 3-point rather than a 9-point scale to score each domain, with a score of 3 coded as “great work,” 2 as “good work,” and 1 as “almost there” to be more consistent with the strengths-based framework of the FCU intervention and implementation models. Domains that were rated as “3 - Great work” included consistent use of key elements in each domain. For example, in Section 2, Observant and responsive, a 3 demonstrated the coach’s consistent use of active listening skills, encouragement of caregiver reflections, contributions, and responses, congruency with caregiver’s immediate concerns, and consideration of parent’s cultural values. Domains that were rated as “2 - Good work” involved partial use of elements outlined in each domain. While some domain specific elements were present, there may have been specific elements that were missing or required further development to be consistent with the FCU model. Examples of sessions that earned a 2 in Section 2 include those that demonstrated a need for improvement in attentiveness to the caregiver’s immediate concern, or initial identification of a concern with a lack of follow-up questions about the topic. Domains scored as “1 - Almost there” included inconsistent or lack of multiple key elements in each domain. An example of a session that earned a 1 in Section 2 included one in which the session content was not congruent with the caregiver’s immediate concerns and the coach did not encourage caregiver reflection. An additional marker specific to this domain was the ratio of coach to parent talk. When the coach talked for a majority of the session and did not offer space for the parent to contribute or reflect, Section 2 was scored “1 - almost there”. Moreover, for the domains Connects conversation to the app, Active in structuring the session, Careful when teaching and providing feedback) an “N/A-unable to be scored” option was added to the response scale, with the intended use for when adhering to elements in these COACH sections (e.g., “uses elicitation and active learning” in Careful when teaching and providing feedback) would be counter to the caregiver’s immediate needs during the session (e.g., caregiver wanted support and resources to address a crisis). Refer to the Appendix to review the final adapted FCU Online COACH.

FCU Online COACH coding

External coding.

A COACH coding team was created to code 20% of the audio-recorded sessions from the randomized controlled trial. The coding team was led by a postdoctoral fellow with a Ph.D. in clinical psychology with training in the original FCU in-person model, who was also a member of the clinical and research team in the study. The coding team included three additional student members, including a Ph.D. student in counseling psychology and two Ph.D. students in prevention science; none of whom had prior experience with the FCU. Thus, three out of the four codes were not coaches in the clinical trial. The COACH coding team met on a weekly basis. First, the coding team leader presented an overview of the FCU Online intervention model and team members independently read the coaching intervention manual and reviewed all app content. Next, the team leader trained team members in COACH coding by listening to session recordings together and engaging the group in a discussion about the behavioral markers observed and how they corresponded to the COACH domains. The team continued listening to and coding session recordings together, resolving coding differences through discussion. Next, each team member independently listened to and coded the same recordings to establish inter-rater reliability. After inter-rater reliability was established between the coding team leader and each of the three coders (criterion was 80% agreement or higher on three consecutive recordings), team members coded recordings independently, with every fourth recording coded for reliability by the team leader.

Audio recordings and selection of sample

The protocol for the RCT included audio recording of all FCU Online coaching sessions; however, some recordings were not usable (e.g., were inaudible) and some sessions were not recorded (e.g., coach forgot to record). We selected a sample of sessions to code using the following criteria: 1) recording was available and usable, 2) session recording was at least 15 minutes long, 3) “purpose for contact” was coded by coach as “deliver session” (other reasons for contact included, but were not limited to, providing resources and helping caregivers manage crisis), 4) the session targeted at least one of the five FCU Online modules (e.g., not the “get-to-know-you” coaching session), and 5) the session recording was the 10th or earlier coaching session. Regarding this final inclusion criterion, it was determined that sessions after the 10th would not be eligible for coding because caregivers were likely to have completed the app modules already, rendering some of the COACH domains, particularly Connects conversation to the app, irrelevant.

Of those tapes that met the inclusion criteria (N = 648), 20% were randomly selected, with some constraints. For example, within the sample, all 10 coaches were included with at least one session. The number of recordings per coach included in the coding sample was selected to be roughly proportional to the number of overall sessions each coach completed. Multiple sessions from the same caregiver-coach dyad were eligible for inclusion.

Caregiver Engagement Measures

Four variables were used to capture caregivers’ intervention engagement. Two of the four variables captured app engagement: 1) total minutes engaging with app content and 2) number of modules that the caregiver engaged with throughout the intervention, both of which are automatically collected via the app. The third variable used the total minutes of coach contact to represent caregivers’ engagement with the telehealth coaching component, which was recorded by coaches following each session. The fourth and final measure of caregivers’ intervention engagement is the mean of coach-rated caregiver engagement across all available caregiver sessions. After each session, coaches completed a brief series of questions, which included rating the caregiver’s engagement in the session on a 3-point scale (1 = low; 3 = high). An example of low caregiver engagement was lack of or minimal response to questions posed by the coaches. An example of high caregiver engagement was active participation and reflection to responses. The variable for observed caregiver engagement was calculated as a mean for all of the caregiver’s sessions, not just the sessions for which COACH ratings were available.

Data Analytic Plan

We used percent agreement to calculate inter-rater reliability (Feinstein & Cicchetti, 1990; Zhao et al., 2022). To explore associations between the adapted COACH domain ratings and measures of caregiver engagement, we used bivariate correlations. We used Steiger’s z tests to compare the strength of correlation coefficients (Steiger, 1980). For this analysis, we averaged COACH ratings across caregiver sessions if there was more than one COACH-coded session for any given caregiver because three out of four of the engagement variables captured engagement throughout the intervention, as opposed to engagement in the specific session. We used bivariate correlations to explore intercorrelations among COACH domains.

Results

Descriptive Statistics

There were a total of 1105 coaching sessions logged by the 10 coaches. Of those, 648 met the aforementioned inclusion criteria. Of those that did not meet inclusion criteria, the majority were excluded due to the “purpose for contact” being for something other than intervention delivery, (e.g., providing resources to caregiver). A sample of 20% of all eligible recordings (N = 134) was randomly selected. Of the 134 recordings in the sample, 113 (84%) were from the first through fifth coaching sessions (1st session count = 24, 2nd session = 27, 3rd session = 33, 4th session = 17, 5th session = 12), with the remaining 16% from the sixth through tenth coaching sessions (6th session = 11, 7th session = 6, 8th session = 2, 9th session = 1, 10th session = 1). The number of recordings per coach included in the coding sample ranged from 2–52.

Of the 180 caregivers in the intervention group, 164 caregivers had at least one session that was eligible for inclusion (of the 16 caregivers without an eligible session, 15 did not engage with the intervention and one did not have a session that was at least 15 minutes). Of those 164, 90 caregivers had recordings that were randomly selected for coding (see above for more details on eligibility for inclusion and random selection). Approximately 9% of the selected sessions were delivered in Spanish by two of the coaches. Independent samples t tests and chi-square tests were used to test for differences on key outcomes (e.g., parenting skills, caregiver mental health) and demographic variables (e.g., caregiver race, ethnicity, family income) assessed at baseline between caregivers with and without sessions randomly selected for COACH-coding; no significant differences were found.

Descriptive statistics for COACH coding scores by section are as follows (see Table 1), with possible scores ranging from 1–3 for all domains: Section 1: Connects conversation to the app: M = 2.88, SD = .38 ; Section 2: Observant and responsive: M = 2.71, SD = .56; Section 3: Active in structuring session: M = 2.77, SD = .53; Section 4: Careful when teaching and providing feedback: M = 2.72, SD = .58; Section 5: Helpful in building hope and motivation: M = 2.72, SD = .59. Mean scores across COACH domains indicated acceptable coach fidelity to the FCU Online, with all sections scoring above the benchmark for competent implementation of the FCU Online model (i.e., a score of 2 or greater). Inter-rater reliability, calculated as percent agreement, ranged from 91% (Connects conversation to the app) to 74% (Careful when teaching and providing feedback). See Table 1 for more details.

Table 1.

Descriptives statistics, inter-rater reliability, and intercorrelations between Family Check-Up Online COACH fidelity variables

Family Check-Up Online COACH domain 1 2 3 4 Mean (SD), range
N = 134		 Coder inter-rater reliability
(% agreement)
1. C-Connects conversation to the app -- 2.88 (.38), 1–3 91%
2. O-Observant & responsive .37* -- 2.71 (.56), 1–3 83%
3. A-Active in structuring session .68* .59* -- 2.77 (.53), 1–3 83%
4. C-Careful when teaching & providing feedback .47* .79* .63* -- 2.72 (.58), 1–3 74%
5. H-Helpful in building hope & motivation .43* .87* .57* .85* 2.72 (.59), 1–3 87%
Open in a new tab
*

p < .05

Note: Range of possible scores for each COACH domain was 1–3.

Bivariate correlations between sections ranged from moderate (r = .37 for correlation between Connects conversation to the app and Observant and responsive) to high (r = .87 between Observant and responsive and Helpful in building hope and motivation, see Table 1).

Finally, we tested whether session number (1–10) was associated with scores on any of the COACH domains. Session number was significantly associated with only Connects conversation to the app (r = −.21, p = .02), where later sessions were rated as having significantly lower fidelity in this domain.

Associations between Fidelity and Caregiver Engagement

COACH scores for Observant and responsive were significantly correlated with the number of coaching support sessions the caregiver completed (r = .22, p < .05) (see Table 2, which also includes descriptive statistics for all caregiver engagement measures). This correlation was significantly stronger than those for two of the other COACH categories (Connects conversation to the app and Active in structuring session) and number of sessions and did not significantly differ from the other two (Careful when teaching & providing feedback and Helpful in building hope & motivation). COACH scores for both Observant and responsive and Helpful in building hope and motivation were significantly correlated with the number of app modules the caregiver completed (r’s = .24 and .25, respectively), however, only the correlation between number of app modules and Helpful in building hope and motivation was significantly different from Careful when teaching and providing feedback and was not significantly different from the other three COACH categories and number of app modules. COACH scores for Connects conversation to the app were significantly correlated with time caregivers spent using the app (r = .21, p < .05), although the strength of this correlation was not significantly different from that of the other COACH categories and time spent in the app. In addition, all COACH categories, as well as a COACH composite representing the average score across all five categories, were significantly correlated with coach-reported caregiver engagement (r’s ranging from .37 - .65, p < .001), with Connects Conversation to the app having a significantly weaker correlation with caregiver engagement than the other four COACH variables.

Table 2.

Correlations between COACH scores and caregiver engagement (N = 90 participants)

Variable Observed caregiver engagement (average across all sessions; possible range 1–3) # of coaching support sessions Minutes engaged in app over intervention period # of app modules completed (possible range 0–5)
C-Connects conversation to the app .37*1 −.132 .21*3 .14
O-Observant & responsive .57* .22* .13 .24*
A-Active in structuring session .60* −.10 .09 .19
C-Careful when teaching & providing feedback .60* .16 .13 .12
H-Helpful in building hope & motivation .55* .15 .15 .25*
COACH composite .65* .13 .16 .23*
Descriptive statistics for caregiver engagement variables
Mean (SD), range 2.90 (.3), 1–3 8.10 (5.5), 1–35 100.72 (52.9), 0–339 4.77 (.87), 0–5
Open in a new tab
*

p < .05

1

Strength of correlation between observed engagement and Connects conversation to the app was significantly smaller than correlations between observed engagement and all other COACH variables except Helpful in building hope and motivation (z scores ranging from -3.22 - -2.0). None of the other correlations significantly differed from each other.

2

Strength of correlation between # of coaching sessions and Observant & responsive was significantly larger than correlations between # of coaching sessions and “connects conversation to the app” and “active in structuring session” (z-scores ranging from -3.37 - -2.97). None of the other correlations significantly differed from each other.

3

Strength of correlation between minutes engaged in app and “Connects conversation to the app” was not significantly different from correlations between minutes engaged in app and other COACH variables.

4

Strength of correlation between # of app modules completed and Helpful in building hope & motivation was significantly greater than correlation between # of app modules completed and Careful when teaching & providing feedback (z = 2.27). None of the other correlations significantly differed from each other.

Discussion

In the current study, we adapted an existing fidelity tool (COACH) for an in-person parenting intervention that relies heavily on MI skills (Family Check-Up; FCU) for an online implementation of the intervention model (FCU Online). We used the adapted COACH to code a sub-sample of sessions from a trial of the FCU Online and test associations between different domains of FCU Online fidelity, including three domains assessing model adherence and two domains assessing MI skills utilization, and caregivers’ engagement in the intervention.

Mean scores on COACH domains indicated acceptable fidelity to the FCU Online, with ratings on all COACH domains above the benchmark for competent implementation of the FCU Online model (i.e., a score of 2 or greater on a scale from 1–3). The very high bivariate correlations between the two domains assessing MI skills utilization (Observant and responsive and Helpful in building hope and motivation) indicated that clinicians who demonstrated high fidelity to one of the MI skills domains were also highly likely to receive high scores on the other MI skills domain. This suggests that these two domains underlie a common construct (e.g., competence or quality of intervention delivery; Cross & West, 2011). Correlations between COACH scores on Connects conversation to the app, a domain assessing coach adherence to structural aspects of the FCU Online, and COACH scores on the two MI skills utilization domains were moderate and statistically significant; however, these correlations were not as strong as the correlations between the two MI skills domains.

Fidelity to the FCU Online model was associated with variables capturing caregiver intervention engagement, including coach-rated caregiver engagement, time spent using the app, number of app modules completed, and number of coaching sessions. Aligned with hypotheses and prior research supporting the role of MI in facilitating participant engagement (Boardman et al., 2006; Smith et al., 2013), high scores on the domain Observant and responsive—one of two domains capturing coaches’ effective use of MI skills—were correlated with both the number of coaching sessions caregivers attended and the number of app modules completed. This finding supports the argument that providers’ use of MI skills may be a strong predictor of engagement. Helpful in building hope and motivation, the second MI skills domain, was associated with the number of app modules completed, though it should be noted that the strength of the correlation was not significantly different from those of the other COACH categories and number of app modules completed. Perhaps, when coaches were able to effectively elicit caregiver perceptions that change was possible, caregivers were more likely to engage with online content in an attempt to make that change. Scores in Connections conversation to the app were correlated with minutes spent engaging with the FCU app, suggesting that coaches who effectively incorporate app content into coaching sessions facilitate further caregiver engagement in the app. However, again, it should be noted that the correlation was not significantly different from that of the other COACH variables. While there is a great deal of prior research linking the use of MI to improvements in participant intervention engagement with in-person interventions (e.g., Berkel et al., 2021; Chaffin et al., 2009; Sterrett et al., 2010), to our knowledge this is the first study demonstrating that clinicians’ use of MI skills was also associated with their engagement in an online program.

Several important insights also emerged through the process of adapting and implementing the FCU Online COACH in the current study. It is notable that of the over 1,000 interactions with participants that coaches logged in the current study, approximately 41% were considered outside of the scope of “typical” intervention delivery (i.e., parent coaching specifically related to app content) and were thus excluded from possible selection for coding. Most of these contacts were dedicated to activities such as resource provision or crisis management. This finding is important to consider for incorporation into future cost analysis of the FCU Online and for understanding optimal intervention dosage. In the current study, we trialed a 3-point scale for scoring each COACH domain, with prior versions of the COACH using a 9-point scale (Smith et al., 2013). Prior work on the original COACH with the in-person FCU, also in the context of a clinical research trial, indicated mean COACH scores close to the middle range of the scale (5–6 out of 9; Smith et al., 2013); whereas in the current study, fidelity scores were closer to the upper end of the scale (i.e., 2.7 or higher out of 3). It could be that a simpler rating scale resulted in scoring skewed toward the upper end of the scale, perhaps because coders were reluctant to utilize the bottom of the scale. Prior research also suggests that simpler fidelity scales (e.g., 3- as compared with 5-point scales) may be less reliable or accurate (Suhrheinrich et al, 2020). However, a simpler scale provides advantages of simplicity and feasibility, particularly considering future implementation in community settings, and may outweigh these potential costs. It is encouraging that significant correlations were observed between COACH ratings and caregiver intervention engagement, providing evidence that the FCU Online COACH’s 3-point fidelity scale is sufficiently sensitive to capture meaningful differences related to key outcomes.

Limitations and Next Steps

One limitation is that we used mean levels of caregiver engagement (in the app and coaching) across the intervention period rather than engagement in individual sessions. Another related limitation is that we were unable to code every coaching session for our sample of participants. While this was unavoidable in the current study, this approach limited our ability to draw specific conclusions about the relationship between model fidelity and caregiver engagement within individual sessions. Future research should explore relationships between session-specific fidelity and session-specific engagement.

Another limitation in the interpretation of our findings is the wide range of coded audio files selected across the ten coaches (ranging from 2–52), which introduces potential bias due to between-coach variability and limits the generalizability of these findings across other providers and implementation settings. However, because the number of tapes included per coach is roughly proportional to their total number of recorded sessions in the full dataset, concerns about the validity of the fidelity-related conclusions drawn in the current sample are somewhat mitigated.

An important additional limitation is that coaches in the current study were highly trained clinicians (licensed clinical social workers with extensive experience implementing the FCU in-person model). As the FCU Online was developed with low-cost and accessible implementation in community and educational settings in mind, next steps for the adapted COACH are to evaluate its use with providers in community contexts who may have less clinical experience and training compared to the coaches in the RCT. In the context of community implementation, the adapted COACH tool can be incorporated in the coach portal available in the administrative back end of the FCU Online app to facilitate ease of self-ratings. Future studies should assess overall scores on the COACH across sessions as correlates of caregiver engagement and intervention benefits when implemented in community settings.

Though the adapted COACH strongly emphasizes the use of MI skills, it was not designed explicitly as a measure of MI as it incorporates fidelity to other aspects of the FCU model. Therefore, the COACH has not been validated against established MI fidelity measures such as the MITI (Moyers et al., 2014) or the MISC (Miller et al., 2003). Existing research using the MITI to rate providers’ use of MI skills in delivering the FCU Online found that providers exceeded the “fair” benchmark of acceptability and that providers’ MI use predicted improvements in parenting skills (Flack, 2024) and reductions in adolescent health risk behaviors (DeVargas & Stormshak, 2020). Future studies should incorporate ratings using the adapted COACH to assess alignment of the COACH with established measures of MI (e.g., MITI) and to examine whether MI skills and adherence to structural components of the FCU Online model represent distinct pathways to caregiver engagement.

Conclusions

In the current study, we described the steps taken to adapt the in-person FCU COACH fidelity tool for an online implementation of the FCU, an MI-based intervention model. In comparison to the original COACH, the adapted COACH for the FCU Online is simple, brief, and designed to be accessible to a range of prospective providers. Results suggest that fidelity to the FCU Online model as assessed by the adapted COACH, particularly those domains most related to MI skills utilization, was associated with higher levels of caregiver engagement in both coaching and the app. Overall, our study provides initial support for the utility of this fidelity coding tool for a flexible, MI-based digital health intervention that can be delivered within educational settings.

Supplementary Material

Supplemental material

Funding:

The research reported here was supported by the National Institutes of Health, National Institute on Drug Abuse, through Grant P50DA048756 to the University of Oregon. The opinions expressed are those of the authors and do not represent views of the National Institutes of Health.

Appendix. Family Check-Up Online COACH

Family Check-Up Online COACH

Section 1 Overall: Connects conversation to the app

  1. Almost there

  2. Good work

  3. Great work

    N/A – Unable to be scored; “connects conversation to the app” was determined to be counter to client’s immediate needs during the session; clinician used judgment to adjust session to meet client’s needs

  • Session content links to evidence-based principles and content in the online program

Section 2 Overall: Observant and responsive to family’s needs

  1. Almost there

  2. Good work

  3. Great work

  • Uses active listening skills (e.g., summarizing, reflecting) and empathy

  • Asks for caregiver’s perspective and invites contributions and responses

  • Congruent with caregiver’s immediate concerns and motivation

  • Considers parent’s cultural values and beliefs

Section 3 Overall: Active in structuring the session

  1. Almost there

  2. Good work

  3. Great work

    N/A – Unable to be scored; “active in structuring the session” was determined to be counter to client’s immediate needs during the session; clinician used judgment to adjust session to meet client’s needs

  • Sets agenda (how are you doing, check in on home practice goal, check in on module)

  • Checks in on work between sessions and discusses barriers

  • Identifies skill to focus on in session

  • Identifies home practice

Section 4 Overall: Careful when teaching and providing feedback

  1. Almost there

  2. Good work

  3. Great work

    N/A – Unable to be scored; “careful when teaching and providing feedback” was determined to be counter to client’s immediate needs during the session; clinician used judgment to adjust session to meet client’s needs

  • Scaffolds skill teaching to caregiver’s skills, motivation, and preferences

  • Offers constructive feedback and reinforces parent’s positive use of skill

  • Uses elicitation and active learning rather than solely relying on instruction (e.g., role-playing, encouraging parent to generate ideas for specific applications of skills)

  • Encourages parent self-reflection (e.g., asking parent to reflect on what went well with home skills practice; what stuck out to them in the app)

Section 5 Overall: Helpful in building hope and motivation

  1. Almost there

  2. Good work

  3. Great work

  • Validates parent’s use of skills

  • Identifies and builds on existing strengths

  • Prompts, evokes, and supports change talk

  • Focuses on realistic goals (in session and for home practice) with clear, specific, and achievable steps

  • Uses a strength-based approach

Notes

Footnotes

Conflict of Interest: Elizabeth Stormshak has an ownership interest in Northwest Prevention Science, Inc. (NPS), and serves as CEO of that entity. Anne Marie Mauricio serves as a consultant with NPS. The content of this manuscript overlaps with activities at NPS, and this conflict of interest has been managed by the University of Oregon.

References

  1. Basha SA, Kim JJ, Cai Q, Kuckertz M, & Gewirtz AH (2025). What is fidelity? A systematic review of provider fidelity and its associations with engagement and outcomes in parenting programs. Clinical Psychology Review, 115, 102531. 10.1016/j.cpr.2024.102531 [DOI] [PMC free article] [PubMed] [Google Scholar]
  2. Begeny JC, Wang J, Levy RA, Sanetti LM, Loehman J, & Rodriguez K (2023). Considering the implementation research-to-practice gap: An experimental evaluation of intervention-general methods for assessing and supporting intervention fidelity through coaching. Journal of School Psychology, 97, 152–170. 10.1016/j.jsp.2023.01.003 [DOI] [PubMed] [Google Scholar]
  3. Bellg AJ, Borrelli B, Resnick B, Hecht J, Minicucci DS, Ory M, ... & Czajkowski S. (2004). Enhancing treatment fidelity in health behavior change studies: best practices and recommendations from the NIH Behavior Change Consortium. Health psychology, 23(5), 443–451. 10.1037/0278-6133.23.5.443 [DOI] [PubMed] [Google Scholar]
  4. Berkel C, Mauricio AM, Rudo-Stern J, Dishion TJ, & Smith JD (2021). Motivational interviewing and caregiver engagement in the Family Check-Up 4 Health. Prevention Science, 22(6), 737–746. 10.1007/s11121-020-01112-8 [DOI] [PMC free article] [PubMed] [Google Scholar]
  5. Boardman T, Catley D, Grobe JE, Little TD, & Ahluwalia JS (2006). Using motivational interviewing with smokers: Do therapist behaviors relate to engagement and therapeutic alliance? Journal of substance abuse treatment, 31(4), 329–339. 10.1016/j.jsat.2006.05.006 [DOI] [PubMed] [Google Scholar]
  6. Bond L, Simmons E, & Sabbath EL (2022). Measurement and assessment of fidelity and competence in nonspecialist-delivered, evidence-based behavioral and mental health interventions: a systematic review. SSM-Population Health, 19, 101249. 10.1016/j.ssmph.2022.101249 [DOI] [PMC free article] [PubMed] [Google Scholar]
  7. Borrelli B (2011). The assessment, monitoring, and enhancement of treatment fidelity in public health clinical trials. Journal of public health dentistry, 71(s1), S52–S63. 10.1111/j.1752-7325.2011.00233.x [DOI] [Google Scholar]
  8. Breitenstein SM, Gross D, Garvey CA, Hill C, Fogg L, & Resnick B (2010). Implementation fidelity in community‐based interventions. Research in nursing & health, 33(2), 164–173. 10.1002/nur.20373 [DOI] [PMC free article] [PubMed] [Google Scholar]
  9. Brian J, Solish A, Dowds E, Roth I, Bernardi K, Perry K, ... & Bryson S. (2022). “Going Mobile”-increasing the reach of parent-mediated intervention for toddlers with ASD via group-based and virtual delivery. Journal of Autism and Developmental Disorders, 52(12), 5207–5220. 10.1007/s10803-022-05657-1 [DOI] [PMC free article] [PubMed] [Google Scholar]
  10. Canário AC, Pinto R, Silva-Martins M, Rienks K, Akik BK, Stanke KM, ... & Leijten P. (2025). Online parenting programs for children’s behavioral and emotional problems: a network meta-analysis. Prevention Science, 26(4), 592–609. 10.1007/s11121-024-01735-1 [DOI] [PMC free article] [PubMed] [Google Scholar]
  11. Chaffin M, Funderburk B, Bard D, Valle LA, & Gurwitch R (2011). A combined motivation and parent–child interaction therapy package reduces child welfare recidivism in a randomized dismantling field trial. Journal of consulting and clinical psychology, 79(1), 84–95. 10.1037/a0021227 [DOI] [PubMed] [Google Scholar]
  12. Chaffin M, Valle LA, Funderburk B, Gurwitch R, Silovsky J, Bard D, ... & Kees M (2009). A motivational intervention can improve retention in PCIT for low-motivation child welfare clients. Child maltreatment, 14(4), 356–368. 10.1177/10775595093322 [DOI] [PubMed] [Google Scholar]
  13. Chiapa A, Smith JD, Kim H, Dishion TJ, Shaw DS, & Wilson MN (2015). The trajectory of fidelity in a multiyear trial of the family check-up predicts change in child problem behavior. Journal of Consulting and Clinical Psychology, 83(5), 1006–1011. 10.1037/ccp0000034 [DOI] [PMC free article] [PubMed] [Google Scholar]
  14. Connell AM, & Stormshak EA (2023). Evaluating the efficacy of the family check-up online to improve parent mental health and family functioning in response to the COVID-19 pandemic: A randomized clinical trial. Journal of Prevention, 44(3), 341–357. 10.1007/s10935-023-00727-1 [DOI] [PMC free article] [PubMed] [Google Scholar]
  15. Connell AM, Stormshak EA, Mauricio AM, Hails KA, Ramirez-Miranda J, & Inyangson JI (2024). A digital health model for school-based implementation to improve parent and child outcomes: Comparison of active versus light-touch coaching effects. Journal of Prevention, 45, 927–945. 10.1007/s10935-024-00806-x [DOI] [PubMed] [Google Scholar]
  16. Cross W, & West J (2011). Examining implementer fidelity: Conceptualising and measuring adherence and competence. Journal of Children’s Services, 6(1), 18–33. 10.5042/jcs.2011.0123 [DOI] [Google Scholar]
  17. Day JJ, & Sanders MR (2018). Do parents benefit from help when completing a self-guided parenting program online? A randomized controlled trial comparing Triple P Online with and without telephone support. Behavior therapy, 49(6), 1020–1038. 10.1016/j.beth.2018.03.002 [DOI] [PubMed] [Google Scholar]
  18. DeVargas EC, & Stormshak EA (2020). Motivational interviewing skills as predictors of change in emerging adult risk behavior. Professional Psychology: Research and Practice, 51(1), 16–24. 10.1037/pro0000270 [DOI] [PMC free article] [PubMed] [Google Scholar]
  19. Dishion TJ, Knutson N, Brauer L, Gill A, & Risso J (2010). Family check-up: COACH ratings manual. Unpublished coding manual. Available from Child and Family Center. [Google Scholar]
  20. Dishion TJ, Shaw D, Connell A, Gardner F, Weaver C, & Wilson M (2008). The family check‐up with high‐risk indigent families: Preventing problem behavior by increasing parents’ positive behavior support in early childhood. Child development, 79(5), 1395–1414. 10.1111/j.1467-8624.2008.01195.x [DOI] [PMC free article] [PubMed] [Google Scholar]
  21. Dishion TJ, & Stormshak EA (2007). Intervening in children’s lives: An ecological, family-centered approach to mental health care. American Psychological Association. [Google Scholar]
  22. Flack C (2024). Mechanisms of motivational interviewing in the Family Check-Up [Unpublished doctoral dissertation]. University of Wisconsin-Madison. [Google Scholar]
  23. Feder KA, Mojtabai R, Musci RJ, & Letourneau EJ (2018). US adults with opioid use disorder living with children: Treatment use and barriers to care. Journal of Substance Abuse Treatment, 93, 31–37. 10.1016/j.jsat.2018.07.011 [DOI] [PubMed] [Google Scholar]
  24. Feinstein AR, & Cicchetti DV (1990). High agreement but low kappa: I. The problems of two paradoxes. Journal of Clinical Epidemiology, 43(6), 543–549. 10.1016/0895-4356(90)90158-L [DOI] [PubMed] [Google Scholar]
  25. Hails KA, McWhirter AC, Sileci AC, & Stormshak EA (2025). Family Check-Up Online effects on parenting and parent wellbeing in families of toddler to preschool-age children. Early Childhood Research Quarterly, 70, 144–153. 10.1016/j.ecresq.2024.09.009 [DOI] [PMC free article] [PubMed] [Google Scholar]
  26. Harris M, Andrews K, Gonzalez A, Prime H, & Atkinson L (2020). Technology-assisted parenting interventions for families experiencing social disadvantage: A meta-analysis. Prevention Science, 21(5), 714–727. 10.1007/s11121-020-01128-0 [DOI] [PubMed] [Google Scholar]
  27. Hogue A, Henderson CE, Dauber S, Barajas PC, Fried A, & Liddle HA (2008). Treatment adherence, competence, and outcome in individual and family therapy for adolescent behavior problems. Journal of consulting and clinical psychology, 76(4), 544–555. 10.1037/0022-006X.76.4.544 [DOI] [PMC free article] [PubMed] [Google Scholar]
  28. Hser YI, Lanza HI, Li L, Kahn E, Evans E, & Schulte M (2015). Maternal mental health and children’s internalizing and externalizing behaviors: beyond maternal substance use disorders. Journal of Child and Family Studies, 24, 638–648. 10.1007/s10826-013-9874-3 [DOI] [PMC free article] [PubMed] [Google Scholar]
  29. Kuehn BM (2022). Clinician shortage exacerbates pandemic-fueled “mental health crisis”. JAMA, 327(22), 2179–2181. doi: 10.1001/jama.2022.8661 [DOI] [PubMed] [Google Scholar]
  30. Löwe B, Kroenke K, & Gräfe K (2005). Detecting and monitoring depression with a two-item questionnaire (PHQ-2). Journal of Psychosomatic Research, 58(2), 163–171. 10.1016/j.jpsychores.2004.09.006 [DOI] [PubMed] [Google Scholar]
  31. McAloon J, & Armstrong SM (2024). The effects of online behavioral parenting interventions on child outcomes, parenting ability and parent outcomes: A systematic review and meta-analysis. Clinical Child and Family Psychology Review, 27, 523–549. 10.1007/s10567-024-00477-4 [DOI] [PMC free article] [PubMed] [Google Scholar]
  32. McAloon J, & de la Poer Beresford K (2023). Online behavioral parenting interventions for disruptive behavioral disorders: A PRISMA based systematic review of clinical trials. Child Psychiatry & Human Development, 54(2), 379–396. 10.1007/s10578-021-01253-z [DOI] [PubMed] [Google Scholar]
  33. McLeod BD, Porter N, Hogue A, Becker-Haimes EM, & Jensen-Doss A (2023). What is the status of multi-informant treatment fidelity research?. Journal of Clinical Child & Adolescent Psychology, 52(1), 74–94. 10.1080/15374416.2022.2151713 [DOI] [PubMed] [Google Scholar]
  34. Miller WR, Moyers TB, Ernst D, & Amrhein P (2003). Manual for the motivational interviewing skill code (MISC). Unpublished manuscript. Albuquerque: Center on Alcoholism, Substance Abuse and Addictions, University of New Mexico. [Google Scholar]
  35. Miller WR, & Rollnick S (2013). Motivational interviewing: Helping people change (3rd ed.). Guilford Press. [Google Scholar]
  36. Mohr D, Cuijpers P, & Lehman K (2011). Supportive accountability: a model for providing human support to enhance adherence to eHealth interventions. Journal of Medical Internet Research, 13(1), e30. 10.1080/15374416.2022.2151713 [DOI] [PMC free article] [PubMed] [Google Scholar]
  37. Moyers TB, Manuel JK, Ernst D, Moyers T, Manuel J, Ernst D, & Fortini C (2014). Motivational interviewing treatment integrity coding manual 4.1 (MITI 4.1). Unpublished manual. [Google Scholar]
  38. Moyers TB, Miller WR, & Hendrickson SM (2005). How does motivational interviewing work? Therapist interpersonal skill predicts client involvement within motivational interviewing sessions. Journal of consulting and clinical psychology, 73(4), 590–598. 10.1037/0022-006X.73.4.590 [DOI] [PubMed] [Google Scholar]
  39. Nock MK, & Kazdin AE (2005). Randomized controlled trial of a brief intervention for increasing participation in parent management training. Journal of Consulting and Clinical Psychology, 73(5), 872–879. 10.1037/0022-006X.73.5.872 [DOI] [PubMed] [Google Scholar]
  40. Oregon Office of Rural Health. (2024). About rural and frontier data. Oregon Health & Science University. https://www.ohsu.edu/oregon-office-of-rural-health/about-rural-and-frontier-data
  41. Rosenberg BM, Kodish T, Cohen ZD, Gong-Guy E, & Craske MG (2022). A novel peer-to-peer coaching program to support digital mental health: design and implementation. JMIR mental health, 9(1), e32430. 10.2196/32430 [DOI] [PMC free article] [PubMed] [Google Scholar]
  42. Smith JD, Dishion TJ, Shaw DS, & Wilson MN (2013). Indirect effects of fidelity to the Family Check-Up on changes in parenting and early childhood problem behaviors. Journal of Consulting and Clinical Psychology, 81(6), 962–974. 10.1037/a0033950 [DOI] [PMC free article] [PubMed] [Google Scholar]
  43. Steiger JH (1980). Tests for comparing elements of a correlation matrix. Psychological Bulletin, 87, 245–251. 10.1037/0033-2909.87.2.245 [DOI] [Google Scholar]
  44. Sterrett E, Jones DJ, Zalot A, & Shook S (2010). A pilot study of a brief motivational intervention to enhance parental engagement: A brief report. Journal of Child and family Studies, 19(6), 697–701. 10.1007/s10826-010-9356-9 [DOI] [Google Scholar]
  45. Stormshak E, Connell A, Mauricio AM, McLaughlin M, & Caruthers A (2025). Digital Health Delivery of Parenting Skills to Improve Conduct Problems in Middle School Youth Across Two Distinct Randomized Trials. Prevention Science, 26, 582–591. 10.1007/s11121-024-01750-2 [DOI] [PMC free article] [PubMed] [Google Scholar]
  46. Stormshak EA, Matulis JM, Nash W, & Cheng Y (2021). The Family Check-Up Online: A telehealth model for delivery of parenting skills to high-risk families with opioid use histories. Frontiers in Psychology, 12, 695967. 10.3389/fpsyg.2021.695967 [DOI] [PMC free article] [PubMed] [Google Scholar]
  47. Stormshak EA, Mauricio AM, & Gill AM (2024). Everyday parenting: A professional’s guide to building family management skills, 2nd edition. Research Press Publishers. [Google Scholar]
  48. Stormshak E, Mauricio AM, McWhirter AC, & Reiter L (2025). A model of implementation for the family check-up in community mental health settings. Pedagogia Social Revista Interuniversitaria, (46), 59–73. 10.7179/PSRI_2025.46.03 [DOI] [Google Scholar]
  49. Sourander A, McGrath PJ, Ristkari T, Cunningham C, Huttunen J, Lingley-Pottie P, ... & Unruh A. (2016). Internet-assisted parent training intervention for disruptive behavior in 4-year-old children: a randomized clinical trial. JAMA psychiatry, 73(4), 378–387. 10.1001/jamapsychiatry.2015.3411 [DOI] [PubMed] [Google Scholar]
  50. Suhrheinrich J, Dickson KS, Chan N, Chan JC, Wang T, & Stahmer AC (2020). Fidelity assessment in community programs: An approach to validating simplified methodology. Behavior Analysis in Practice, 13(1), 29–39. 10.1007/s40617-019-00337-6 [DOI] [PMC free article] [PubMed] [Google Scholar]
  51. Tan-MacNeill KM, Smith IM, Johnson SA, Chorney J, & Corkum P (2021). A systematic review of online parent-implemented interventions for children with neurodevelopmental disorders. Children’s Health Care, 50(3), 239–277. 10.1080/02739615.2021.1886934 [DOI] [Google Scholar]
  52. Werntz A, Amado S, Jasman M, Ervin A, & Rhodes JE (2023). Providing human support for the use of digital mental health interventions: systematic meta-review. Journal of Medical Internet Research, 25, e42864. 10.2196/42864 [DOI] [PMC free article] [PubMed] [Google Scholar]
  53. Zhao X, Feng GC, Ao SH, & Liu PL (2022). Interrater reliability estimators tested against true interrater reliabilities. BMC medical research methodology, 22(1), 232. 10.1186/s12874-022-01707-5 [DOI] [PMC free article] [PubMed] [Google Scholar]

Associated Data

This section collects any data citations, data availability statements, or supplementary materials included in this article.

Supplementary Materials

Supplemental material
NIHMS2149501-supplement-Supplemental_material.docx (54KB, docx)

RESOURCES