Transitioning to telehealth due to COVID‐19: Maintaining model fidelity in a home visiting program for parents of vulnerable infants

Abstract

Maintaining treatment fidelity when implementing evidence‐based interventions is a significant challenge. The inability to deliver in‐person services due to the COVID‐19 pandemic critically challenged the foundation of implementation fidelity for home visiting programs across the globe. The Attachment and Biobehavioral Catch‐Up (ABC) program is an evidence‐based home visiting intervention designed to increase sensitivity in parents of infants who have experienced early adversity. ABC's community effectiveness is due to rigorous fidelity monitoring and supervision. Fidelity is measured by microanalytic coding of parenting opportunities and “in‐the‐moment” commenting, the active ingredient of ABC. In this study, we examined intervention fidelity among parent coaches implementing ABC through telehealth. Random 5‐min clips from 510 telehealth ABC session videos conducted by 91 parent coaches at 48 agencies were coded for their frequency and quality of in‐the‐moment comments. On average, parent coaches were able to exceed in‐person commenting fidelity standards when implementing ABC through the telehealth format. The active fidelity monitoring and supervision inherent to ABC's dissemination afforded a smooth transition to implementing ABC through telehealth while adhering to fidelity standards. Procedural and clinical challenges to telehealth implementation are discussed, along with future directions for telehealth program effectiveness.

1. INTRODUCTION

As the crisis of COVID‐19 escalated across the globe in the first few months of 2020, home visiting programs grappled with how best to respond to the needs of children, staff, and communities, while also prioritizing safety and avoiding face‐to‐face interaction. This crisis was especially significant given that early home visiting is a critical service for vulnerable families (Avellar & Supplee, 2013; Filene et al., 2013; Garner, 2013; Minkovitz et al., 2016). Infants, toddlers, and their families rely on home visiting interventions to provide a range of services, including those meeting basic needs as well as those focused on skill building and education. While the first few years of life are characterized by remarkable development and plasticity, young children are also particularly vulnerable to the effects of stress and trauma (Hertzman, 1999; National Research Council & Institute of Medicine, 2000; Shonkoff & Garner, 2012). Early parenting programs aim to promote sensitive and nurturing relationships, which can in turn promote healthy development and mitigate the effects of early stress (Sweet & Appelbaum, 2004). The cascading effects of COVID‐19 both increased levels of stress and trauma for families facing unemployment and financial strain, but also threatened home visiting as a means for receiving support and intervention.

1.1. The importance of implementation fidelity

Home visiting models are, as their names suggest, implemented within family homes. Meeting with families within their own homes accomplishes two important goals: 1) Families learn new skills in the environment in which they will be used, thus expanding the likelihood of generalizability (Howard & Brooks‐Gunn, 2009) and 2) families can receive services without the cost and practical challenge of traveling to clinics or hospitals. These goals, though important, are perhaps orthogonal to treatment fidelity. The growing field of research in home visiting implementation science has established that fidelity (i.e., whether a model is implemented as it is was designed and studied) is critical to treatment effectiveness (Allen et al., 2012; Casillas et al., 2015; Durlak & Dupre, 2008; Meyers et al., 2012; Paulsell et al., 2014; Schoenwald et al., 2011). Measures of implementation fidelity should capture carefully tested and delineated active ingredients, the key portions of an intervention that makes the program effective (Durlak & DuPre, 2008; Home Visiting Applied Research Collaborative, HARC, 2020). The advent of COVID‐19 prompted home visiting leaders to evaluate program active ingredients and whether these ingredients could be implemented successfully across a telehealth modality.

Few home visiting models have carefully specified active ingredients for program effectiveness (Home Visiting Applied Research Collaborative, 2020). Indeed, infant home‐visiting program effectiveness could be enhanced if active ingredients are identified and tested within a consistent theory of change among specific target populations (Segal et al., 2012). HARC highlights the Attachment and Biobehavioral Catch‐up (Dozier & Bernard, 2019; ABC) program as an evidence‐based home visiting model that specifies active ingredients and objectively measures those active ingredients as part of the implementation.

THREE KEY FINDINGS/PRACTITIONER POINTS

•  Attachment and Biobehavioral Catch‐up, an evidence‐based home visiting program, successfully transitioned to telehealth because of a rigorous commitment to measuring and maintaining fidelity.

•  Weekly supervision meetings to assess and support fidelity were key in transitioning to telehealth.

•  Parent coaches performed at certifiable levels in making in vivo comments about program‐relevant caregiving interactions.

STATEMENT OF RELEVANCE TO THE FIELD

The global COVID‐19 pandemic was challenging for providers of all in‐person services. This study addressed whether an evidence‐based parenting program, Attachment and Biobehavioral Catch‐up (ABC), could be implemented as intended through a telehealth modality. Evaluation of fidelity data from parent coaches in training during the COVID‐19 crisis demonstrated that ABC's active ingredient, in‐the‐moment commenting, could be carried out through telehealth services. Implications beyond the pandemic are discussed.

1.2. Attachment and Biobehavioral Catch‐up

1.2.1. Description of ABC

Attachment and Biobehavioral Catchup (ABC) is a ten‐session, manualized home‐visiting intervention designed to increase sensitive and nurturing caregiving in parents of infants or toddlers who have experienced early adversity. Trained home visitors, referred to as “parent coaches,” visit the home for 1 h with the parent, child, and all other caregivers and children in the home who would like to participate. Most of the session is unstructured, while parent coaches promote intervention target behaviors through the use of guided discussion, video examples, a few semi‐structured practice activities to facilitate practice of the target behaviors, and most importantly, the frequent use of “in‐the‐moment” comments. In‐the‐moment comments are immediate parent coach responses to intervention‐relevant behaviors that occur. Frequent and high quality in‐the‐moment comments are associated with change in parental sensitivity, and in‐the‐moment commenting has been identified as the “active ingredient” of ABC (Caron et al., 2016).

ABC targets three key parenting behaviors in order to increase sensitive care. First, parents are coached to respond to their children's distress in a way that is consistent, reassuring, and empathetic. This first target behavior is called nurturance, which is critical in preventing a disorganized attachment in children who have experienced adversity (Dozier et al., 2001). The second target behavior, following the lead, focuses on parent‐child interactions when the child is not distressed. By promoting following the lead, parent coaches help parents to engage in child‐directed play when the child is not distressed. Following the lead is an important building block for the child's development of self‐regulation (Bornstein & Tamis‐LeMonda, 1997; Feldman et al., 1999), and vulnerable infants with parents who followed the lead more often showed signs of enhanced physiological regulation in a randomized clinical trial (Bernard et al., 2010). The third target of ABC is to help parents eliminate frightening and overwhelming behavior, as frightening parental behavior is linked with physiological dysregulation and disorganized attachment (van IJzendoorn et al., 1999).

1.2.2. ABC efficacy

ABC has demonstrated efficacy in improving both child outcomes and parental sensitivity in several randomized clinical trials. In families referred by Child Protective Services, children whose parents received ABC had a significantly lower rate of disorganized attachment and a higher rate of secure attachment than children whose parents received a control intervention (Bernard et al., 2012). In the same sample, children whose parents received ABC also showed a steeper (i.e., more typical and adaptive) diurnal slope of cortisol production than those who received a control intervention when assessed immediately post‐intervention (Bernard et al., 2015) and at a 3‐year follow‐up (Bernard et al., 2015). Additional child outcomes include greater social‐emotional competence (Lind et al., 2021) and executive function (Lind et al., 2017). Further research has revealed that intervention effects such as attachment security (Zajac et al., 2020), autonomic nervous system regulation (Tabachnick et al., 2019), and inhibitory control (Korom et al., 2020) last into middle childhood, and current efforts are investigating adolescent outcomes. The increase in parental sensitivity observed in parents who received ABC across multiple populations (Bick & Dozier, 2013; Yarger et al., 2016) is a critical outcome of the intervention, as parental sensitivity change is the mechanism through which many of the child outcomes are achieved (Garnett et al., 2020; Raby et al., 2019).

1.2.3. In‐the‐moment commenting

In‐the‐moment commenting, a form of in vivo feedback in which parent coaches bring attention to and celebrate target behaviors when they occur, is an essential component of ABC. Results by Caron et al. (2016) showed a direct association between high frequency and quality in‐the‐moment comments and parental change in sensitivity. Parental sensitivity change is the mechanism of change leading to improved child outcomes (Garnett et al., 2020; Raby et al., 2019). The direct link between in‐the‐moment comments and child outcomes has not been examined. In‐the‐moment comments can include one or more of the following three components: describing the observed parent behavior, labeling the behavior as an ABC target, and linking the behavior to a long‐term child outcome. An example comment that contains all three of these components could be, “She cried, and you picked her right up, rubbed her back, and said ‘I know, mommy's here.’ (description) Beautiful nurturance! (labeling behavior target) This helps her develop trust in you and others. (linking to long‐term outcome)”

1.2.4. Training and supervision

All ABC parent coaches first complete an initial intensive training. Prior to COVID‐19 precautions, the initial training was structured as an in‐person group event for two full days. After the initial training, parent coaches then receive both clinical and fidelity supervision through videoconferencing weekly throughout their first year of implementing ABC. During this time, they videorecord their ABC sessions with families. Video sharing is central to ABC supervision and is used in both weekly supervision meetings to monitor adherence and fidelity. Specifically, fidelity supervision occurs individually for 30 min, with supervision efforts aimed at addressing the in‐the‐moment commenting that was delivered during ABC sessions. To measure fidelity, ABC fidelity supervisors use a quantitative coding system to code for intervention‐relevant parenting behaviors (e.g., nurturance, following the lead with delight, avoiding overwhelming/frightening behaviors) and parent coaches' in‐the‐moment responses to them. As part of the supervision process, supervisors and parent coaches code the same segment of the video. This practice was found to increase coach frequency and quality of comments (Caron & Dozier, 2019). Clinical supervision occurs for one hour weekly, in small group meetings where parent coaches consult on manual adherence, case conceptualization, and logistics.

ABC has demonstrated effectiveness in increasing parental sensitivity in community settings across the United States, with large effect sizes that are comparable to effect sizes observed in randomized clinical trials (Caron et al., 2016; Roben et al., 2017). ABC's effectiveness in community settings has been largely attributed to the strong process of fidelity supervision.

1.3. Evidence for telehealth as an intervention modality

Although the pandemic led to an unprecedented uptake of telehealth services across early intervention, behavioral health, and medicine, the potential benefits of offering internet‐based services to families had been of interest to the mental health field for years. The use of telehealth for adult psychotherapy has been more widely studied than for child and family therapies. Reviews across multiple diagnoses and treatments found roughly equivalent outcomes between in‐person and telehealth modalities (Gros et al., 2013; Norwood et al., 2018), equivalent process variables such as attendance and satisfaction with treatment (Gros et al., 2013), but mixed findings regarding working alliance, which while still adequate, could be lower in telehealth individual than in‐person therapies (Norwood et al., 2018).

Family therapies add complexities to the dynamics of a session, most notably the inclusion of young children to the room, many of whom are not likely to stay seated in one place or may be interested in playing with the device used for telehealth (McLean et al., 2021). Many of the families most in need also have inequitable access to technology and internet (e.g., Hall & Bierman, 2015; Racine et al., 2020). However, despite the challenges and potential barriers to telehealth, many researchers and clinicians have expressed optimism and have promoted the excellent opportunity telehealth provides for reaching families who need intervention (Cluxton‐Keller et al., 2018; Doss et al., 2017; Gurwitch et al., 2020; Hall & Bierman, 2015; Racine et al., 2020; Wrape & McGinn, 2019).

One evidence‐based program that serves young children with an in vivo commenting component similar to ABC, Parent‐Child Interaction Therapy (PCIT; McNeil & Hembree‐Kigin, 2010) compared their in‐person program delivered in the clinic with an adapted remote program delivered in the home through videoconferencing (Comer et al., 2017). The randomized trial enrolled children between 3 and 5 years of age with a diagnosed disruptive behavior disorder. Notably, all Internet‐delivered PCIT (I‐PCIT) families were supplied with webcams and a bug‐in‐the‐ear device, similar to how feedback is provided to families receiving live comments during PCIT sessions in the clinic. Positive child outcomes of I‐PCIT were comparable to or better than PCIT, and parents perceived fewer barriers to treatment with I‐PCIT than PCIT. Such results provide precedence for implementing in‐the‐moment comments to families over videoconferencing. However, PCIT is traditionally implemented as a clinic‐based program, and thus it is not surprising that families found it easier to access treatment from home than in the clinic. The authors also noted challenges that could arise when the intervention was implemented outside of a randomized controlled trial, and access to the internet and devices like the bug‐in‐the‐ear may not be possible for all families. Indeed, these were all challenges that were challenges for PCIT therapists at the beginning of COVID‐19 (Gurwitch et al., 2020).

Until 2020, the home visiting as a field as a whole had not embraced telehealth as a mode of intervention. Some models had used phone calls or telehealth visits as a potential method of connection when necessary or preferred (e.g., Nurse Family Partnership; McConnell et al., 2020), but, to our knowledge, evidence‐based home visiting models had not been rigorously tested for efficacy when services were administered via videoconferencing or other remote connections. As in‐person visits were restricted due to safety concerns, home visiting model leadership across the United States quickly developed guidance and best practices for maintaining relationships with families as home visits were disrupted (National Alliance of Home Visiting Models, 2020; Rapid Response Virtual Home Visiting, 2020). For ABC, a precise, targeted model with an active ingredient based on observing and commenting on live parent‐child interaction, it was evident that in order to provide a continuity of service, home visitors would need to pivot immediately to using video conferencing as a mode of intervention. Thus, the ABC dissemination team at the University of Delaware rapidly developed a manual for what they called “TeleABC.”

1.4. The transition to TeleABC

As described above, ABC parent coaches in training were already video recording their ABC sessions and engaging in weekly remote clinical and fidelity supervision via videoconferencing. Given these supervision processes were established prior to COVID‐19, parent coaches entered the transition to telehealth already familiar with video conferencing platforms and secure, HIPAA‐compliant video sharing software. Early conversations with parent coaches and local implementing agencies included selecting a secure, easy‐to‐use video platform and updating consent forms and processes to reflect the use of telehealth. Additionally, the ABC team talked with coaches and agencies about supporting both families and parent coaches in having consistent and reliable access to high‐speed internet. For example, to increase internet access equity, some public health and education initiatives, as well as private internet providers, sought to improve internet quality or offer free services to those who needed them (e.g., Federal Communications Commission, 2020). Within ABC, some local implementing agencies were also able to use funds that would have been spent on parent coach transportation to improve parent coach home internet connections or offer families digital devices or connectivity solutions such as hotspots.

Another important consideration for TeleABC was navigating videorecording sessions to review in supervision and to provide video feedback for families. Some secure platforms have recording enabled, whereas others do not have that option. If agencies preferred or were required to use a system without embedded recording, parent coaches impressed us with their creativity. Some parent coaches were not able to access secure platforms with embedded recording, nor were they able to record their computer screens using software such as QuickTime. In these cases, parent coaches resorted to recording their computer screens using their handheld digital video recorders that they had previously used during in‐home visits. They set up their handheld video recorders on tripods behind them as they conducted the session through the secure video‐based platform. Supervisors and parent coaches used supervision meetings to learn how to optimize session delivery, such as setting up quality audio for the session recording or having the videoconferencing program only record the view of the family, and not of the parent coach. Parent coaches shared solutions and options for setting up home devices in ways that would best capture the parent‐child interaction, while also not serving as a distraction for the child. For example, one parent coach noticed that turning off her own video resulted in a child being less focused on her mother's phone compared to when the coach's video was turned on. Another coach discovered that asking a parent to put her phone in an empty drinking glass helped to amplify the audio and did not require the parent to have to purchase any additional equipment to be able to hear the coach well during sessions. These types of simple solutions allowed parent coaches to attempt implement ABC as it was intended, with frequent in‐the‐moment comments and discussion of manual content.

The content of clinical and fidelity supervisions was also influenced by considerations specific to COVID‐19. For example, clinical supervisors and parent coaches discussed the increased stress families were facing related to COVID‐19 and how this acute and often intense experience of stress could affect parents responding with nurturance and sensitivity to their children. Fidelity supervisors quickly developed strategies for making in‐the‐moment comments even when parent coaches could not see the entire interaction. For instance, a parent coach might say, “I couldn't quite see what made him so upset, but I can hear his crying and see that you went right over to him to see if he was ok. Wonderful nurturance!”

Alongside the individual work that was being done in the supervision meetings, the ABC dissemination team problem‐solved challenges and shared clinical solutions through weekly supervision consultation meetings and a group messaging channel. The team noted that, compared to delivering services in person, some parent coaches reported that it was easier to give more direct feedback to families in telehealth sessions. Some parent coaches also reported that families remained engaged with their children more actively while the in‐the‐moment feedback was being given over the telehealth platform, as compared to in‐person services. Other parent coaches indicated that commenting was more difficult in telehealth than through in‐person sessions because they did not always have a clear view of the entirety of the parent‐child interaction. This was especially true in sessions where there were long spans of parent inattention toward the child, many of which occurred because parents felt they needed to hold their phone to have a “FaceTime‐style” conversation with the parent coach. Overall, the ABC team felt thankful to the ingenuity of the ABC parent coaches in training, who not only helped brainstorm creative solutions, but also assisted in writing instructions and sharing their solutions with others in training. One parent coach even presented her innovative solutions on a Rapid Response Virtual Home Visiting webinar, which reached an audience of over 2000 home visitors (Rapid Response ‐ Virtual Home Visiting Collaborative, 2020).

Despite the many dramatic changes to procedures and set‐up of ABC, in‐the‐moment commenting, the core and active ingredient of ABC, did not change in the shift from ABC to TeleABC. Given the in‐the‐moment coding and commenting system that delineates parent behavior targets (e.g., nurturance, following the lead with delight, avoiding overwhelming/frightening behaviors) and associated coach comments, ABC parent coaches and supervisors continued to have a clear sense of intervention fidelity even as ABC moved from in‐person to telehealth service delivery. The use of the in‐the‐moment coding system during fidelity supervision also allowed parent coaches to monitor their progress in delivering high quality and frequent in‐the‐moment comments during TeleABC sessions. To assess how parent coaches maintained fidelity to ABC via in‐the‐moment commenting during the pivot to TeleABC, this study examined the supervision fidelity reports from the first 7 months of the COVID‐19 pandemic across all ABC community dissemination sites implementing TeleABC during this time.

2. METHODS

2.1. Participants

ABC fidelity data were collected from 510 TeleABC session videos, conducted by 91 parent coaches from 48 agencies throughout the United States. Parent coaches implemented between 1 and 10 TeleABC sessions per week. All parent coaches implementing ABC in this evaluation were currently in training. For some parent coaches, implementing ABC is their primary responsibility, whereas other coaches implement ABC as one of many programs or roles within their broader community practice. The types of community agencies varied widely, but all provide prevention or intervention to families experiencing early adversity. Agencies include state or county early intervention services for children between 0 and 3 years of age, organizations involved in the child welfare system, and private organizations serving the local community.

A voluntary demographic survey was sent out to the 91 parent coaches included in the current evaluation. Coaches were asked to report age, gender, ethnicity, race, education level, and years of experience in their given field. Seventy‐six coaches (83.6%) completed at least some of the survey, and 69 coaches (75.8%) completed the entire survey. The missingness was random across agencies.

Of the 76 participants who provided demographic data, 72 (94.7%) clinicians identified as female, and 4 (5.2%) identified as male. Parent coaches’ ages ranged from 22 to 63 (M = 36.54, SD = 10.42). The majority of parent coaches identified as White (69.7%, n = 53), with significant percentages identifying as Black/African‐American (15.8%, n = 12) and Latino/Latina (15.8%, n = 12), and others identifying as multiracial (3.9%, n = 3), American Indian (2.6%, n = 2), and four parent coaches indicating Asian Ancestry, Colombia Native, Moor, or Uruguayan.

Most parent coaches held master's degrees (55.3%, n = 42), but parent coaches’ highest level of education ranged from having completed high school (n = 1, 1.3%) to having completed a doctoral degree (n = 4, 5.3%). Seven parent coaches had some college experience (9.2%), 15 had completed bachelor's degrees (19.7%), one was enrolled in a master's program (1.3%), and one was enrolled in a Ph.D. program (1.3%). Parent coaches’ experience in the field ranged from 0–29 years (M = 7.84, SD = 7.45). Parent coaches’ initial training dates ranged from 3/5/2017 to 5/28/2020. Some coaches had re‐enrolled in supervision in order to gain certification in the ABC‐Toddler model. Parent coaches must be certified in ABC‐Infant before training in ABC‐Toddler. Approximately half of the sessions were ABC‐Infant sessions (53.14%). Enrollment in ABC‐Infant or ABC‐Toddler was not associated with whether or not fidelity averages were above certification criteria as described further below.

The data used in this report to measure parent coach fidelity and demographics were originally collected for program evaluation purposes. The University's institutional review board considered the research exempt because we used archived and de‐identified data that had been collected and coded for the purposes of program evaluation.

2.1.1. Procedure

All parent coaches in the current study had completed the 2‐day intensive training and were receiving weekly supervision via videoconferencing throughout their training year of implementing ABC. During the period of data collection, parent coaches received clinical supervision from an expert ABC clinical supervisor and fidelity supervision from an expert in‐the‐moment coder. Supervisors included those employed by the ABC development team, as well as those employed by local implementing agencies and certified as ABC supervisors. All sessions used in the current study came from parent coaches who had not yet been certified and were currently engaged in their year of ABC training.

Thirty‐one total in‐the‐moment supervisors coded the clips used in this study. All in‐the‐moment supervisors first underwent a 3‐month training period which consisted of weekly hour‐long coding practice and homework. At the end of the training, trainees coded a set of 10 challenging videos on their own, and to become reliable in in‐the‐moment coding, the trainee's 10 coding sheets had to meet 70% reliability with the master coder. After becoming reliable in in‐the‐moment coding, coders began providing fidelity supervision to parent coaches. In‐the‐moment supervisors attended a weekly group supervision to consult about coding and strategies to improve their parent coaches’ commenting.

Parent coaches videorecorded every ABC session they conducted, and their in‐the‐moment supervisor then randomly selected one 5‐min clip from one session per week to code, using a random number generator (www.random.org). Five‐minute clips capture a variety of parent behaviors but can still be coded in 30–45 min. Longer clips would cause undue burden on home visitors for weekly review. Clips were randomly selected in order to ensure that parent coaches were meeting fidelity standards throughout the entire session, across cases. Only one session was coded per week, even if a coach had completed multiple sessions. Generally, if a coach was seeing multiple families simultaneously, in‐the‐moment supervisors were careful to select different cases for coding for each supervision session. If the parent coach expressed difficulty with a particular case, the in‐the‐moment supervisor would collaborate with the coach to decide which case should be coded each week to ensure challenging cases were being supervised.

After an in‐the‐moment supervisor randomly selected a clip to be coded, they informed the parent coach of which 5 minutes to code, and both the supervisor and the parent coach came to each supervision meeting with the clip already coded. Each in‐the‐moment supervision meeting was 30 min long, and supervisors discussed both the ABC‐targeted parent behaviors and the coach's commenting in the random clip, as well as worked on ways to improve the coach's in‐the‐moment commenting skills through worksheets or live commenting practice via video review.

In the current study, TeleABC sessions completed between 3/27/2020 to 11/5/2020 were examined. Every TeleABC session coded during weekly in‐the‐moment supervisions conducted within this date range was included in the dataset.

2.2. Measure

2.2.1. Fidelity

Fidelity was measured using the in‐the‐moment coding system. Fidelity data were taken from in‐the‐moment supervisors’ coding sheets. In‐the‐moment supervisors coded each occurrence of an ABC‐targeted parent behavior (e.g., nurturance, following the lead, overwhelming/intrusive behavior), whether or not the coach made a comment, and the accuracy and quality of each comment. For example, if the child shakes a toy truck up and down and the caregiver copies the child, the behavior is described and gets a numerical code that represents following the lead. The coach's comment is transcribed and coded as either an on‐target or off‐target comment. Finally, the comment would be coded for the number of information components from zero to three. The coding sheet automatically calculates summary statistics which report coaches’ commenting rate per minute, percentage of missed opportunities, percentage of on‐target comments, and the average number of information components of each in‐the‐moment comment.

2.3. ABC certification standards

On average, by the end of the year of training and supervision, parent coaches are expected to meet fidelity certification standards for in‐the‐moment commenting. For 7 out of the 10 most recent ABC sessions, parent coaches are expected to make comments that are 80% or above on target (i.e., the parent behavior commented upon aligns with the coded parent behavior), that have at least one information component on average, and that are made with a frequency of at least 1.0 comments per minute or have fewer than 50% missed opportunities. We examine commenting rate using both frequency of commenting and missed opportunities in order to account for the variation in the number of parent behaviors upon which a coach could comment. For example, if a parent coach only makes four comments within a 5‐min segment (0.8 comments per minute), but there were only five opportunities for a comment (only 20% missed opportunities), we would consider this an adequate rate of commenting. Notably, we do not expect, nor would we want, a parent coach to comment on 100% of opportunities. A five‐minute clip can sometimes include 20 or more opportunities for comments. If a coach were to comment on every opportunity available, the frequency of comments could feel too high to both parents and coaches, even though most comments are positive and supportive. In a five‐minute clip with 20 opportunities, coaches could comment on only half of the observed parent behaviors (50% missed opportunities), yet still have a high rate of commenting at two comments per minute.

The standards described here were arrived at iteratively as the program was developed. Skilled parent coaches, whose parent coaching resulted in the most observed parent behavior change, and who built strong rapport with families and appeared the most comfortable in sessions, commented at or above the minimum standards reported above. Additional research with an early community dissemination group confirmed the association between comments and parent behavior change (Caron et al., 2016), while also showing that those who completed ABC training reached these standards (N = 9 parent coaches; rate = 1.05 comments per minute, percent missed opportunities = 59.91, percent on‐target comments = 83.06, average number of components = 1.05), while those who dropped out of training did not reach fidelity standards. Community implementation has confirmed that these standards are difficult, yet possible to achieve with rigorous supervision (Costello et al., 2019), and have resulted in community effectiveness (Roben et al., 2017; Roben et al., 2021). Fidelity measures of a program's identified and measurable active ingredient, which are measured using observation instead of provider report, are not typically used in the home visiting field. Ratings of amount of session time devoted to model content and procedures are commonly used as measures of fidelity in most well‐known home visiting models (see Daro et al., 2014).

3. RESULTS

3.1. Observed ABC commenting in telehealth sessions

Across the 510 ABC sessions examined for this study, ABC parent coach fidelity standards were exceeded when implementing ABC through the telehealth format. An average of 89.61% (SD = 18.36) comments were on‐target. Comments were provided at a frequency of 1.39 (SD = 0.80) comments per minute, with 44.90% (SD = 21.61) missed opportunities. Comments contained an average of 1.38 (SD = 0.56) information components (see Table 1).

TABLE 1.

In‐the‐moment commenting certification standards and TeleABC metrics

	Certification Goals (7/10 Sessions)	TeleABC (510 sessions)	% Sessions Above Certified Levels
Frequency (comments/minute)	1.0	1.39	79.22% a
% Missed Opportunities	≤ 50%	44.90%
% On‐target Behaviors	≥ 80%	89.61%	83.33%
Number of Components	1.0	1.38	87.25%

Note. This table compares averages of weekly fidelity scores to ABC certification standards. Certification goals are examined at end of the year of training, for seven out of the ten most recent sessions. The second column lists the averages of all 510 TeleABC sessions, and the third column represents percentage of the 510 sessions that were at or above the certification goals.

^a The certification goals for rate of comments are calculated using a combination of the frequency of comments and the percent of missed opportunities. 79.22% of the TeleABC sessions met either the frequency or percent of missed opportunities standard.

We also examined the percentage of sessions that met each certification standard. Parent coaches are expected to be able to reach each certification standard for 7 out of 10 sessions by the end of the certification year. Across this study's sample of telehealth sessions, 425 out of 510 sessions (83.33%) had comments that were 80% or above on‐target. Out of 510 session, 352 (69.02%) had a frequency of 1.0 comments per minutes or above. Out of 510 sessions, 287 (56.27%) had comments that had fewer than 50% missed opportunities. When examining the commenting rate criteria (frequency and missed opportunities) together as intended, 404 out of 510 sessions (79.22%) had a frequency of 1.0 comments per minute or above or fewer than or equal to 50% missed opportunities. Out of 510 sessions, 445 (87.25%) had an average of 1.0 information components per session or above.

4. DISCUSSION

The global COVID‐19 pandemic created an incredible challenge for providers of all in‐person services. The home‐visiting field, which previously had put a priority on in‐person connections and relationships, pivoted quickly to provide services to families, along with supervision, training, and implementation support to the home visiting workforce (National Alliance of Home Visiting Models, 2021). However, it was unknown how well the substituted supports were reflective of the originally designed in‐person programs. ABC's precision approach, with an identified active ingredient of change, positioned the model well to ensure the implementation of key ingredients through telehealth. The evaluation of fidelity data from parent coaches in training during the COVID‐19 crisis demonstrated a robust implementation through telehealth of ABC's active ingredient, in‐the‐moment commenting.

Examination of fidelity data across 91 coaches, in various stages of training, demonstrated that parent coaches were able to make high quality and frequent comments when implementing Attachment and Biobehavioral Catch‐up through telehealth. Specifically, the majority of randomly selected 5‐minute segments of TeleABC sessions for parent coaches in training were above criteria for certification in ABC. The TeleABC fidelity averages reported here are comparable to previous ABC program evaluation data from thousands of in‐person sessions of coaches during the training year across many dissemination sites and years of implementation, where the average commenting frequency was 1.37, average percent missed opportunities was 51.80, average percent of on‐target comments was 86.06, and average number of components per comment was 1.34 (Milberg et al., unpublished manuscript). We partially credit this success to the ways in which supervisors and parent coaches worked together to develop strategies for commenting through the video, which were then shared through the ABC supervision team to the wider field of ABC parent coaches. The community‐focused and supportive atmosphere was shared across the field as a whole, as home visitors, agencies, state leaders, and models shared resources and strategies to bring relief and support to highly stressed families. It should be noted that these fidelity data come from parent coaches who were experiencing their own stressors during the pandemic. Like so many of the families that they served, parent coaches faced illness and job insecurity, worried and cared for sick or isolated family members, and managed changing jobs and technology while watching and home schooling their own children. We find it remarkable that ABC parent coaches not only continued to carry out visits at all, but that they also did them so very well.

ABC's identified active ingredient and detailed, weekly coding system made it possible to prioritize implementation fidelity in the pivot to telehealth. This capacity to focus on what exactly is core and necessary for program success is similar to what has helped ABC implement with fidelity in different cultures with sensitivity (Aparicio et al., 2016; Costello et al., 2021). Certain components of the program can shift to support implementation in different contexts, while also maintaining strict fidelity to the central tenets of ABC. Whereas previous research has supported in‐the‐moment commenting as the active ingredient that leads to change in parent behavior (Caron et al, 2016), it will be important to monitor and evaluate fidelity in new contexts and to measure the relative impact on resulting program effectiveness.

4.1. Limitations

The program evaluation data for this study were collected during the pandemic‐driven need to pivot to a telehealth mode of intervention. Because of these circumstances, we do not have a direct and randomized comparison sample for these data. While these data are comparable to past evaluations of fidelity in ABC, a future randomized trial is critical. Furthermore, these data do not include measures of family engagement or family outcomes. As emphasized by Duggan (2021), in order to understand what works best for which families, in which contexts, and how, it is critical to include measures of engagement and context as potential moderators of an intervention's mechanism of change. Studies that include measurement of the family perspective would better elucidate the potential of the effectiveness, reach, and sustainability of this mode of intervention.

4.2. Future directions for ABC

The global pandemic led to this unplanned transition to telehealth, yet these preliminary fidelity data suggest TeleABC should be considered for use in the future. ABC community partners report many benefits to telehealth, including saved time for parent coaches, reduced transportation costs, and resulting expansion in served geographical areas. Because parent coaches were able to exceed in‐person commenting fidelity standards when implementing ABC through the telehealth format, we feel optimistic about the ability to support continued use of telehealth. However, there is a need for prospective and randomized evaluation of telehealth adaptations of in‐person programs (Richardson et al., 2009). Prospective, randomized trials can not only compare TeleABC with in‐person ABC outside of the context of a global pandemic and evaluate short and long‐term family outcomes, but it can also assess for whom the telehealth modality works, and in what contexts it is and is not a suitable course of intervention. A randomized trial could examine differences in our indices of fidelity and change, such as whether there are more challenges to capturing and commenting upon observable parent behaviors for commenting (i.e., do we find fewer opportunities in telehealth sessions than in‐person sessions). This research can guide best practices for decision making regarding TeleABC versus in‐person ABC for individual families and communities.

4.3. Future directions for home visiting

Telehealth sessions will remain a critical option for home visiting programs for at least the near future. While community transmission of the coronavirus remains high, telehealth sessions provide a safe alternative to in‐person sessions for many vulnerable parents and children, family members residing in multigenerational homes, home visitors, and family members of home visitors. In order for the telehealth option to be feasible in both the short‐ and long‐term, structural policies will need to support this modality of service. Temporary adjustments to federal and local policies, reporting requirements, and billing systems made in the immediate wake of the pandemic will need to be considered and adapted for long‐term use.

Most importantly, program evaluation of telehealth during the pandemic and future research must consider the relative accessibility of in‐person versus telehealth services. The pandemic has highlighted how internet accessibility is an essential need for families, and reducing the digital divide for those in rural and underserved communities could be a gateway to consistent and high‐quality services. If research finds telehealth to be less effective than in‐person services, policy makers and community leaders will need to weigh both accessibility and effectiveness when creating regulations and reimbursement procedures.

Roben, C. K. P. , Kipp, E. , Schein, S. S. , Costello, A. H. , & Dozier, M. (2022). Transitioning to telehealth due to COVID‐19: Maintaining model fidelity in a home visiting program for parents of vulnerable infants. Infant Mental Health Journal, 43, 173–184. 10.1002/imhj.21963