Factors associated with the implementation of the Familias Unidas intervention in a type 3 translational trial

Abstract

This study highlights how Familias Unidas, a Hispanic-specific, evidence-based, family centered preventive intervention, progressed from intervention development (type 1 translation; T1) through rigorous evaluation (T2) and examines the role of intervention fidelity—adherence and competence—in a T3 trial. Effects of participant, provider, and organizational variables on direct (observational) and indirect (self-reported) fidelity were examined as were effects of fidelity. Two structural equation models were estimated using data from 367 Hispanic parent-adolescent dyads randomized to Familias Unidas. Facilitator perceptions of parental involvement in schools, school performance grade, and school socioeconomic status predicted indirect adherence ratings, which were positively related to adolescent substance use. Facilitator openness to evidence-based practices was associated with indirect competence ratings, school performance grade and size were associated with direct competence ratings, and direct competence ratings were negatively associated with substance use. Findings highlight unique contributions of direct and indirect fidelity ratings in the implementation of Familias Unidas.

Family based preventive interventions have been shown to be efficacious and effective in preventing and reducing adolescent problem behaviors, including substance use [1]. For evidence-based interventions (EBIs) to have a positive impact at the population level, however, they must be translated into real-world settings. Translation, or the process through which EBIs are brought to scale, has been associated with numerous challenges, many of which are related to ensuring an intervention is delivered as intended, or with fidelity to its core components [2]. As interventions move along the research to practice continuum from efficacy and effectiveness to implementation trials, drift from the original protocol can often occur, making the need to monitor intervention fidelity increasingly important [3]. Despite the need to better understand intervention fidelity for the continued process of translation, research on what factors impact fidelity and the extent to which fidelity impacts the effectiveness of family based preventive interventions is not completely understood. This study thus moves the field of translation forward by examining “real world” factors that affect fidelity, and in turn, how fidelity is related to the effectiveness of Familias Unidas, a family based preventive intervention for Hispanics.

Familias Unidas is a culture-specific, parent-centered intervention, shown to be both efficacious and effective in preventing or reducing a number of adverse health behaviors (e.g., substance use, sexual risk behaviors) in Hispanic adolescents [4–6]. It has been cited by the Institute of Medicine as an efficacious intervention ready to move through more advanced stages in the translational research process [7]. This process, from basic biopsychosocial research to translation into global communities, has been described as occurring across several stages [8] (Table 1). As a program of research, Familias Unidas has moved from initial intervention development (“type 1 (T1) translation”) through rigorous testing with well-defined populations (“T2 translation”) and most recently, to a trial examining its effectiveness (“T3 translation”) [5]. Familias Unidas is continuing to move across the stages of translation (“T4–T5 translation”) and is currently being adapted for both online and international use in South America.

Table 1.

Translational research stages

Type	Type 0 translation (T0)	Type 1 translation (T1)	Type 2 translation (T2)
Definition	The fundamental process of translating findings and discoveries from social, behavioral and biomedical sciences into research applied to prevention intervention.	Moving from bench to bedside. Translation of applied theory to methods and program development.	Moving from bedside to practice and involves translation of program development to implementation.
Type	Type 3 translation (T3)	Type 4 translation (T4)	Type 5 translation (T5)
Definition	Determining whether efficacy and effectiveness trial outcomes can be replicated under real-world settings.	Wide-scale implementation, adoption and institutionalization of new guidelines, practices, and policies.	Translation to global communities. Involves fundamental, universal change in attitudes, policies, and social systems.

Because a more detailed summary of the Familias Unidas program of research from T1–T3 has been previously published [4], the translational process is briefly described below. During T1, community stakeholders identified conduct problems and substance use as key concerns among local parents. Given risk and protective factors associated with adolescent risky behaviors are present at multiple levels of the social environment, the ecodevelopmental framework [9] was used to guide intervention development. This framework organizes risk and protective factors from the macrosystem (i.e., broad societal factors, such as Hispanic cultural values) to the microsystem (i.e., contexts in which youth participate directly, such as family, peers, and school). Focus groups with Hispanic parents additionally highlighted the role of acculturation as contributing to youth problem behaviors. As such, problem-posing and participatory learning exercises were built into each session to address a number of relevant cultural issues raised by participants, such as those related to having to raise children in a foreign country (e.g., immigration stress, acculturation, lack of social support). This process, whereby participants are encouraged to raise personally relevant cultural issues, has facilitated the translation of the intervention across diverse contexts and Hispanic populations, including those living outside the USA. During T2, Familias Unidas was rigorously tested across three randomized controlled trials and found to be efficacious in preventing and reducing substance use and sexual risk-taking in universal, selected, and indicated samples. The present study is characteristic of T3 in that parameters associated with the implementation of the Familias Unidas effectiveness trial, conducted in a real-world practice setting by school counselors, are examined to inform the continued process of “scaling up.”

The models that provided the conceptual basis for this study (Figs. 1 and 2) suggest (a) a range of participant, provider, and organizational variables at baseline impact intervention fidelity, and (b) intervention fidelity impacts participant attendance, family functioning, and adolescent substance use, measured up to two and a half years post baseline.

Fig. 1.

Model for adherence. The reported values are unstandardized path coefficients rounded to two decimal places. Dotted lines represent non-significant hypothesized pathways. Model fit without the pathway from school performance grade to family functioning at T1 was unacceptable (CFI = 0.85). This pathway was thus added in accordance with modification indices to improve overall model fit. A-US adolescent US born, D direct ratings of competence, HSI Hispanic Stress Inventory, I indirect ratings of competence, P-US parent US born, SES socioeconomic status, School performance grade was dummy coded with “D” schools as the reference group ^aA vs. D schools, ^bB vs. D schools, ^cC vs. D schools; *p < 0.05

Fig. 2.

Model for competence. The reported values are unstandardized path coefficients rounded to two decimal places. Dotted lines represent non-significant hypothesized pathways. Model fit without the pathway from school performance grade to family functioning at T1 was unacceptable (CFI = 0.85). This pathway was thus added in accordance with modification indices to improve overall model fit. A-US adolescent US born, D direct ratings of competence, HSI Hispanic Stress Inventory, I indirect ratings of competence, P-US parent US born, SES socioeconomic status. School performance grade was dummy coded with “D” schools as the reference group ^aA vs. D schools, ^bB vs. D schools, ^cC vs. D schools; *p < 0.05

THE INFLUENCE OF INTERVENTION FIDELITY ON STUDY MEDIATORS AND OUTCOMES

Intervention fidelity represents a critical aspect of implementation that includes both adherence and competence [10, 11]. Adherence reflects the degree to which an intervention is implemented in accordance with the overarching framework, and competence is related to the level of clinical, or facilitator, skill in promoting behavior change. These indicators of fidelity may be assessed using “direct,” observational measures completed by independent raters and/or “indirect,” self-reported measures completed by interventionists [12]. EBIs for reducing child behavior problems and substance abuse have used fidelity rating systems that jointly account for adherence and competence [10, 11, 13]. For both the Parent Management Training Oregon (PMTO) and Family Check-Up programs, higher direct ratings of competent adherence have predicted increases in effective parenting practices [10, 11, 13], a key intervention mediator, as well as improvements in the primary outcome, child problem behavior [14]. A similar multiple observational framework has been shown to predict outcomes in a school-focused prevention program [15]. More commonly, effects of adherence and competence on intervention mediators and outcomes are evaluated separately. Direct ratings of leader competence have predicted greater changes in observed positive parenting following the Incredible Years Intervention, which in turn, resulted in improvements in child behavior [16]. Contrastingly, higher direct ratings of adherence, but not competence, have been associated with greater declines in adolescent substance use and parent-reported externalizing behaviors in individual and family therapy [17]. Given the importance of maintaining fidelity to the content and process of Familias Unidas, this study examines the separate effects of adherence and competence on family functioning (mediator) and adolescent substance use (outcome). Furthermore, although direct assessments of fidelity currently reflect the gold standard, assessing more cost-effective methods for monitoring fidelity are necessary for translation into practice settings [18]. As such, the present study assesses adherence and competence based on direct and indirect methods.

PARTICIPANT, PROVIDER, AND ORGANIZATIONAL PREDICTORS OF INTERVENTION FIDELITY

Given the importance of implementation for study outcomes, a thorough understanding of factors that affect real-world implementation are critical to moving along translational stages. A systems-contextual approach to understanding implementation and dissemination highlights the role of variables at the participant, provider, and organizational levels [19].

Participant variables

At the participant level, some previous parenting interventions have found that higher fidelity scores are associated with lower baseline parenting scores [11], and Cross et al. [15] tested whether child behavior affected fidelity, finding null results. As noted by Forgatch et al. [11], some family characteristics may make it more or less difficult for facilitators to competently adhere to intervention protocols. Because preventive interventions are likely to have greater variability in participants’ need for assistance than are treatment interventions, more distressed, lower functioning families at baseline (e.g., those with greater levels of stress and problem behaviors, lower skills) may require more structure from a given intervention. As such, facilitators may need to maximize their use of clinical skills and adherent delivery of intervention content based on individual participant characteristics. For Hispanic families, unique risk factors associated with immigration (e.g., adapting to a new culture, language barriers) may compromise family functioning, an important buffer against negative adolescent behavioral outcomes [20].

Characteristics of the family may also impact intervention attendance, which has been positively associated with ratings of fidelity [21]. In previous Familias Unidas efficacy trials, family characteristics (e.g., stress, income, and adolescent behavior problems) were significantly associated with initial engagement to the intervention [22]. Initial engagement and within-group cohesion at the first group session were subsequently related to retention to the intervention [22].

Provider variables

Characteristics of those delivering interventions, including positive attitudes towards EBIs, have also been associated with high-quality implementation [23]. Similar to participants making behavioral changes within preventive interventions, community providers adopting EBIs may differ in their readiness to engage in new practices [24]. While community mental health providers have indicated concerns regarding the relevance of highly controlled (efficacy) studies, they have expressed greater openness to effectiveness research due to its increased external validity [25]. Overall, those more open towards EBIs are more likely to report using them [26].

In the context of the school setting, provider perceptions of school culture, including levels of parental involvement in school, may also be related to how well an EBI is implemented [27]. For example, teacher perceptions of parental involvement in school have been associated with student achievement [28]. Because school counselors were delivering the present intervention, their perceptions of parental involvement were assessed as a predictor of their own practices for delivering the intervention.

Organizational variables

Unique school contextual variables may also impact implementation quality [29]. School size and climate are characteristics linked to fidelity. Providers working in larger schools may have a more difficult time implementing an intervention with fidelity due to competing demands [30, 31]. Because Miami Dade County public schools is the fourth largest school district in the nation and serves a high proportion of low-income ethnic minority families, school counselors may have increased demands placed on their time, which affect their ability to deliver EBIs.

THIS STUDY

This study informs the continued process of translating Familias Unidas by examining how participant, facilitator, and organizational variables impact intervention fidelity (adherence, competence), and in turn, how fidelity impacts family functioning and adolescent substance use within an effectiveness trial. The following was hypothesized: (a) baseline family functioning will have a negative relationship with direct and indirect fidelity ratings; (b) facilitator variables (e.g., openness to EBIs) will be positively related to direct and indirect fidelity ratings; (c) school socioeconomic status (SES) and size will be negatively related, while school performance grade will be positively related to direct and indirect fidelity ratings; and (d) direct and indirect fidelity ratings will be positively associated with attendance and family functioning and negatively associated with substance use, with direct ratings more strongly related to substance use.

METHOD

Participants

Participants were 367 adolescents (52 % males, 13.9 ± 0.66 years old, 54 % US born) and their parents (49 % college/graduate school, 34 % income ≥US$30,000/year) recruited from 18 middle schools and randomly assigned at the family or individual level to the intervention condition of the Familias Unidas effectiveness trial [5]. Families were eligible to participate if they had an adolescent in the 8th grade, there was one Hispanic (self-identified) primary caregiver living in the same household as the adolescent willing to participate, they lived within the catchment area of the participating middle schools, and planned to live in South Florida for the duration of the study.

Procedures

The study was approved by the University of Miami Miller School of Medicine’s Institutional Review Board and the Miami-Dade County Public Schools Research Ethics Board. Participants completed assessments at 6, 18, and 30 months post baseline.

Intervention description

A detailed description of Familias Unidas can be found elsewhere [4]. Briefly, Familias Unidas is delivered in 12 weekly sessions comprised of eight parent group sessions (where only parents participate) and four family visits (where both parents and adolescents participate). The intervention aims to involve parents in adolescent peer and school activities, facilitate parent-adolescent cohesion, build supportive relationships among parents, and reduce/prevent problem behavior by improving family functioning. School counselors served as facilitators in this trial (52 % delivered the intervention in their school; 48 % at a different school).

Measures

Participant variables

Demographic data included country of birth for parents and youth, parents’ educational level, and household income. Reliable and valid measures were used to assess parent stress (Hispanic Stress Inventory [32]) and adolescent conduct disorder (Revised Behavior Problem Checklist [33]). Finally, parental self-report of three indicators were used to create a latent construct of family functioning, assessed at baseline and the 6-month follow-up: (1) peer monitoring (Parent Relationship with Peer Group Scale [34]), (2) positive parenting (Parenting Practices Scale [35]), and (3) family communication (Family Relations Scale [36]).

Provider variables

Openness towards adopting EBIs was measured at baseline with the openness subscale of the Evidence-Based Practice Attitude Scale [37]. Facilitator perceptions of parental involvement in their schools were assessed at baseline using a 20-item measure [28].

Organizational variables

School SES. School SES was assessed using the percentage of students receiving free or reduced school lunch. School performance grades (ranging from “A” (best performing) to “F” (poorest performing) based on academic performance and learning gains) and size were obtained from a publicly available database (www.oada.dadeschools.net). Performance grades among schools in the present study were: “A” (55.6 %), “B” (16.7 %), “C” (22.2 %), and “D” (5.6 %). The average school size was 1020 students (SD = 293).

Attendance

Attendance at the first group session was coded as 0 = no and 1 = yes based on whether or not the participant parent(s) were present. Attendance at group sessions 2–8 was measured by summing across the total number of those sessions attended. Average attendance across the full intervention, including group sessions and family visits was 6.4 ± 4.2 sessions.

Fidelity

Ratings of adherence and competence were conducted using structured, session-specific forms. These forms were completed by facilitators following each group session as well as by trained, independent raters coding videotaped sessions. Facilitators used a 7-point Likert scale to rate each session, with higher scores indicating greater fidelity. Observers used the same 7-point Likert scale to rate sessions in four 30-min segments during each 2-h group session. Twenty percent of all sessions were also coded by a second rater (Kappa > 0.80). Separate scores for direct and indirect measures of adherence and competence were calculated by averaging across group sessions. Adherence. Adherence was measured using 2–7 session-specific items related to how well the facilitator presented content in accordance with the facilitators’ manual. Sample items from the first group session included, “To what extent did the facilitator: process the homework assignment, inform parents about the importance of effective parent-adolescent communication, and coach parent(s) to use communication skills.” Competence. Facilitator competence was measured by the same four items each session: “To what extent did the facilitator: join all members of the group, establish group member alliances, utilize problem-posing and participatory learning methods, and act as a switchboard and/or speak for long periods of the activities.” Item four (switchboard) was reverse scored.

Adolescent substance use

Substance use at baseline and 30 months was assessed using three Y/N items. A binary variable was created to indicate whether adolescents had smoked tobacco, drank alcohol, or used an illicit drug in the past 90 days.

Analytic plan

To test the hypothesized models, a latent family functioning variable at baseline and 6 months was created using peer monitoring, positive parenting, and family communication. The factor loadings and intercepts of family functioning at baseline and 6-month assessments were freely estimated because a repeated measures analysis on family functioning was not conducted, which requires measurement invariance across time points. Second, the hypothesized models were estimated using SEM. Model fit was ascertained using the chi-square index (χ2), the comparative fit index (CFI), and the root mean square error of approximation (RMSEA). Study hypotheses were tested using Mplus (7.2). The default estimator, the mean and variance adjusted weighted least square estimator was used to estimate parameters and standard errors.

RESULTS

Overall fidelity ratings

The average overall fidelity ratings were 3.61 (SD = 0.56) for direct observations and 5.10 (SD = 0.58) for indirect facilitator ratings. Across group sessions, average overall fidelity ratings ranged from 2.04 to 4.30 (M = 3.67, SD = 0.86) for direct observations and 4.86 to 5.25 (M = 5.06, SD = 0.13) for indirect facilitator ratings. Across interventionists, average overall fidelity ratings ranged from 2.40 to 4.75 (M = 3.70, SD = 0.53) for direct observations and 3.80 to 6.0 (M = 5.00, SD = 0.60) for indirect facilitator ratings.

Measurement model

Family functioning

Results from the CFA suggested that the three indicators of family functioning loaded significantly onto a single construct at baseline and 6-month assessments. Standardized loadings at baseline and 6 months, respectively, were as follows: family communication, λ = .50, .59; peer monitoring, λ = .52, .55; and positive parenting, λ = .42, .39.

Correlations between direct and indirect ratings of adherence and competence

Indirect ratings of adherence were negatively associated with direct ratings of competence (r = −.42, p < .05). Indirect ratings of adherence were positively related to indirect ratings of competence (r = .74, p < .05). No other correlations were significant.

Structural equation model: hypothesized “adherence” model

The model for adherence (Fig. 1) provided an adequate fit to the data (χ² (164) = 189.75, p = .08; CFI = 0.95, RMSEA = 0.02). Results for the hypothesized paths of interest are discussed in the following sections.

Predictors of adherence

Participant variables: Family functioning at baseline and attendance at the first group session were not significantly related to direct or indirect ratings of adherence. Provider variables: Facilitator perceptions of parental involvement in school were positively associated with indirect ratings of adherence (b = .008, p < .05, 95 % confidence interval (CI) [.001, .015]). Facilitator openness to EBIs was not significantly associated with direct or indirect ratings of adherence. Organizational variables: School size was positively associated with direct ratings of adherence (b = 0.023, p < .05, 95 % CI [.015, .030]). School SES (b = −0.051, p < .05, 95 % CI [−.096, −.007]) and school performance grade (b = −0.791, p < .05, 95 % CI [−.1.565, −.017]) were negatively associated with indirect ratings of adherence. For school performance grade, facilitators delivering the intervention in “C” schools had lower indirect ratings of competence than did those delivering the intervention in “D” schools.

Attendance

Participant demographic characteristics, Hispanic stress, and conduct disorder were not significantly associated with attendance at group session 1. Attendance at group session 1 was positively associated with attendance at group sessions 2–8 (b = 2.821, p < .05, 95 % CI [2.197, 3.446]). Attendance at group sessions 2–8 was not significantly related to family functioning at 6 months.

Effects of adherence

Attendance: Direct ratings of adherence were positively associated with participant attendance at group sessions 2–8 (b = 1.653, p < .05, 95 % CI [.594, 2.712]). Family functioning: Neither direct nor indirect ratings of adherence were related to family functioning at the 6-month assessment. Substance use: Indirect ratings of adherence were positively associated with adolescent substance use at 30 months (b = 0.531, p < 0.05, CI [.153, .910]). The direct measure of adherence was not related to substance use (b = −0.28, p = .50).

Structural equation model: hypothesized “competence” model

The model for competence (Fig. 2) provided an adequate fit to the data (χ² (164) = 199.14, p < .05; CFI = 0.93, RMSEA = 0.02). Results for the paths of interest are discussed in the following sections.

Predictors of competence

Participant variables: Family functioning at baseline and attendance at the first group session were not significantly related to direct or indirect ratings of competence. Provider variables: Facilitator openness to EBIs was positively associated with indirect ratings of competence (b = .249, p < .05, 95 % CI [.182, .316]). Facilitator perceptions of parental involvement in school were not significantly associated with direct or indirect ratings of competence. Organizational variables: School performance grade was positively associated with direct ratings of competence (b = .710, p < .05, 95 % CI [.157, 1.263]), such that facilitators delivering the intervention in “C” schools had higher direct ratings of competence than did those delivering the intervention in “D” schools. School size was negatively associated with direct ratings of competence (b = −0.014, p < .05, 95 % CI [−.026, −.003]). School SES was not significantly associated with direct or indirect ratings of competence.

Attendance

Hispanic stress and conduct disorder were not significantly associated with attendance at group session 1. Attendance at group session 1 was positively associated with attendance at group sessions 2–8 (b = 2.787, p < .05, 95 % CI [2.163, 3.310]). Attendance at group sessions 2–8 was not significantly related to family functioning at 6 months.

Effects of competence

Attendance: Neither direct nor indirect ratings of competence were associated with participant attendance at group sessions 2–8. Family functioning: Neither direct nor indirect ratings of competence were significantly related to family functioning at 6 months. Substance use: Direct ratings of competence were negatively associated with adolescent substance use at 30 months (b = −.999, p < 0.05, 95 % CI [−1.682, −.315]).

DISCUSSION

Following the progression of the Familias Unidas program of research from T1 to T3 translation, this study examined the role of fidelity, including adherence and competence, in the implementation of the Familias Unidas effectiveness trial. Participant, provider, and organizational predictors of intervention fidelity (measured using direct, observational ratings and indirect, self-reported ratings) were examined as was the impact of fidelity on family functioning, the intervention’s mediator, and adolescent substance use. Separate models for adherence and competence each provided an adequate fit to the data, with approximately half of the pathways of primary interest (i.e., those to and from fidelity) found to be significant. These models each demonstrated different associations among variables. For adherence, results indicated that perceptions of parental involvement in schools and school SES predicted indirect adherence ratings in the expected directions. Contrary to hypotheses, school size was positively associated with direct adherence ratings, whereas indirect adherence ratings were negatively related to school performance grade and positively related to substance use. The relationship between direct adherence ratings and substance use, though not significant, was in the expected direction (i.e., negative). For competence, facilitator openness to EBIs was associated with indirect competence ratings, whereas school performance grade and school size were associated with direct competence ratings, all in the expected directions. Direct competence ratings, in turn, were negatively associated with substance use. These results have important implications for monitoring intervention fidelity as interventions like Familias Unidas move beyond T3 translation.

The most noteworthy findings in this study are related to differences observed between direct and indirect ratings of adherence and competence and their respective associations with adolescent substance use. Whereas direct ratings of competence negatively predicted substance use, indirect ratings of adherence were positively associated with this outcome. These findings speak to (1) the advantages and disadvantages of using observational vs. self-reported measures for assessing fidelity, (2) differences inherent to the measures of adherence and competence used in this study, and (3) the importance of competent intervention delivery.

Observational data are generally more accurate than self-report data [38]. As interventions are brought to scale, however, it may not be feasible to use traditional observational methods for monitoring fidelity as they are more costly and time intensive [18, 39]. Researchers have reported taking ~40 h to appropriately train observers to monitor intervention fidelity [11]. Hogue and colleagues [18] argue that reliable, resource-efficient therapist self-report methods are urgently needed to monitor implementation. As part of a family based preventive intervention delivered to inner-city adolescents at risk for substance use, these authors demonstrated high reliability between self-reported therapist ratings of fidelity and those of nonparticipant coders when using a rating system based on the number of minutes dedicated to predefined intervention targets and foci [18]. Substituting self-report measures of intervention extensiveness for those that more closely approximate “billable clinical hours” used in practice settings may thus be a potential strategy for increasing the overall utility of self-reported measures. This more objective, self-report strategy may also decrease social desirability responding, which could have accounted for an overestimation of indirect ratings of adherence in the present study. In addition to developing more reliable/valid self-report measures, future research should examine other aspects of facilitator self-report (e.g., openness to EBIs assessed at multiple time points) that might be useful for monitoring implementation.

Findings from this study also emphasize the need for more practical observational methods. One possibility for moving forward is the use of computational linguistics methods for fidelity assessment, which would automate ratings and increase the ability to monitor and provide timely supervision [40]. Indeed, a proof of concept for the use of computational linguistics has been conducted for Familias Unidas [41]. Specifically, one of the indicators of competence (“joining”) was examined in a third of family visits in this trial using a specified decision tree algorithm that categorized transcribed facilitators’ Spanish and/or English-language utterances (e.g., use of open-ended questions) as characteristic of low or high joining with families [41]. This algorithm, developed with input from Familias Unidas developers and the lead clinical supervisor, was designed to rate “what/why” questions as more favorable for joining than other types of questions and/or utterances and demonstrated high reliability (kappa = 0.83) with human observers. Use of these or similar innovative technology-based methods may reduce high costs, improve the speed with which feedback is provided to facilitators, and strengthen associations between fidelity and outcomes.

Careful consideration must also be given to the items used to measure fidelity. Current measures of fidelity are either specific to a given intervention or more global ones used across multiple interventions [39]. The measure of adherence in this study was not only intervention-specific, but also session-specific. In other words, the number and content of items was contingent on material covered during a given session. As such, adherence at one group session may not have corresponded to adherence at another group session. Cross et al. [15] showed that multiple session fidelity ratings could correct for “difficult” sessions and provide a better overall measure of delivered fidelity. Contrastingly, competence was assessed using the same four items each week. This is similar to the fidelity rating system used in the PMTO intervention, which rates the same five dimensions (i.e., knowledge of PMTO, structure, teaching, clinical process, and overall quality) across sessions [11]. As Familias Unidas continues to move along the translation stages, it may be beneficial to revise the measure of adherence such that intervention-specific content is still captured while using items that remain consistent across sessions. For example, items may be related more broadly to how well facilitators maintained session structure by following the agenda and engaging in outlined activities. This revision of items may enhance the measure’s utility and improve its predictive validity, especially for lay populations more likely to implement and monitor interventions as they are brought to scale.

Discrepant findings between direct and indirect measures of adherence and competence also highlight the important role of competent intervention delivery in producing desired study outcomes. Competent intervention delivery in this intervention is characterized by facilitators’ use of joining, group cohesion-building, and participatory learning techniques, all of which represent core process elements of Familias Unidas [20]. Beyond Familias Unidas, these findings have implications for other group-based interventions for which similar competencies are central to the success of the intervention. Specifically, if limited resources are available for monitoring fidelity, it may be important to prioritize direct ratings of competence over adherence. With regard to indirect ratings of adherence, on the other hand, findings suggest facilitator perceptions of rigid adherence to session content may actually be counterproductive to study outcomes. Barber and colleagues [42] found that moderate adherence to drug counseling was more predictive of positive outcomes than perfect adherence. Finding ways for facilitators to have flexibility within the structure of EBIs may improve the relationship between self-reported adherence and outcomes. Furthermore, achieving a better understanding of how direct and indirect ratings of adherence and competence interact in predicting outcomes is warranted. In the present study, follow-up analyses examining indirect ratings of adherence as a moderator of the relationship between direct ratings of competence and substance use were conducted. Though not found to be statistically significant, these types of analyses may provide information regarding the most and least optimal combinations of direct vs. indirect ratings of adherence and competence in future studies. Overall, special supervisory attention should be given to those facilitators who consider themselves highly competent and/or adherent but whose direct measures indicate otherwise.

Other key findings from this study are related to systems-contextual factors that impact intervention adherence and competence. While parent and adolescent nativity status and parental stress were related to family functioning at baseline, neither family functioning nor attendance at the first session were related to fidelity. Several variables at the facilitator and organizational levels, however, were found to impact direct and indirect adherence and competence ratings. These findings are generally consistent with previous literature indicating that positive provider attitudes and perceptions are positively related to intervention fidelity [23]. School size and other indicators of the school climate (i.e., SES, school performance grade), also impacted either adherence or competence [30, 31]. Contrary to hypotheses, school performance grade and size were negatively and positively related to direct ratings of adherence, respectively. These findings may be explained by the fact that 47 % of facilitators conducted sessions outside of the schools employing them as counselors and may thus not accurately reflect their work demands. Overall, knowledge of these relationships may aid in the use of more proactive technical assistance provided throughout the translation process [43]. For example, knowing in advance that facilitator openness positively impacts competence may indicate the need for tailored technical assistance provided to those scoring low on this indicator. Similarly, prioritizing technical assistance to providers at larger, lower-resourced schools may also be beneficial.

These results should be considered in light of study limitations. Substance use was assessed with self-reported vs. an objective indicator. In addition, no data were collected on other potentially important and more wide-ranging provider (e.g., personality traits, attitudes about healthcare and prevention programs, beliefs about help seeking, attributions regarding the causes of adolescent risk taking) and systems-contextual factors associated with the implementation of EBIs (e.g., organizational support for program delivery) as has been done in previous studies [44]. Finally, the predominant Hispanic subcultural group in the USA (i.e., Mexicans) was not well-represented in the current sample which limits generalizability of findings beyond Miami, FL, to the larger US Hispanic population. Despite these limitations, this study has numerous notable strengths, including the examination of factors impacting implementation of an effectiveness trial using a large sample of Hispanic parent-adolescent dyads, the use of SEM to understand complex relations among variables, and assessment of the primary outcome at 30 months post baseline.

As family centered, evidence-based preventive interventions are scaled both up and out to diverse settings, ensuring they are implemented with fidelity in the real world may maximize positive outcomes. Understanding factors associated with intervention fidelity and how fidelity relates back to key study mediators and outcomes is critical for the process of translation. This study highlights how Familias Unidas moved from T1 to T3 translation and contributes more specifically to the literature on T3 translation by examining parameters associated with the implementation of a highly replicated program with consistent positive effects across time and context. As this program of research moves to more advanced stages in the translational process, ongoing next steps involve testing both international and online adaptations of the intervention. The online adaptation, in particular, will likely be more transportable and may thus be integrated into a wider range of contexts, including primary care [45]. In addition to having important implications for researchers, these findings also offer important insights for communities and organizations looking to adopt EBIs. Beyond obtaining an understanding of the actual intervention, learning more about factors found to facilitate or impede the translation of the EBI under consideration (e.g., characteristics of facilitators, available resources, and size of the organization) as well as resource-efficient methods for monitoring fidelity may be equally as important to the success of the intervention and may aid in prioritizing resources. Overall, given many community practitioners adapt evidence-based programs to fit with their cultures, orientations, and funding, more needs to be learned about the promotion of intervention fidelity and its association with long-term outcomes.