Evaluating the Impact of Implementation Factors on Family-Based Prevention Programming: Methods for Strengthening Causal Inference

Abstract

Despite growing recognition of the important role implementation plays in successful prevention efforts, relatively little work has sought to demonstrate a causal relationship between implementation factors and participant outcomes. In turn, failure to explore the implementation-to-outcome link limits our understanding of the mechanisms essential to successful programming. This gap is partially due to the inability of current methodological procedures within prevention science to account for the multitude of confounders responsible for variation in implementation factors (i.e., selection bias). The current paper illustrates how propensity and marginal structural models can be used to improve causal inferences involving implementation factors not easily randomized (e.g., participant attendance). We first present analytic steps for simultaneously evaluating the impact of multiple implementation factors on prevention program outcome. Then we demonstrate this approach for evaluating the impact of enrollment and attendance in a family program, over and above the impact of a school-based program, within PROSPER, a large scale real-world prevention trial. Findings illustrate the capacity of this approach to successfully account for confounders that influence enrollment and attendance, thereby more accurately representing true causal relations. For instance, after accounting for selection bias, we observed a 5% reduction in the prevalence of 11^th grade underage drinking for those who chose to receive a family program and school program compared to those who received only the school program. Further, we detected a 7% reduction in underage drinking for those with high attendance in the family program.

Prevention scientists are increasingly concerned with evaluating the impact of implementation on the effectiveness of prevention efforts (Elliott & Mihalic, 2004; Domitrovich, Gest, Jones, Gill, & Derousie, 2010; Kam, Greenberg, & Walls, 2003). In particular, implementation broadly appears to be related to the effectiveness of programs delivered in school and family contexts (August, Bloomquist, Lee, Realmuto, & Hektner; 2006; Carroll et al., 2007)¹. Implementation is comprised of many factors that include a prevention effort's reach (e.g., enrollment and attendance rates), delivery quality (e.g., fidelity), and capacity to plan for program delivery (Berkel, Maurcio, Schoenfelder, & Sandler, 2010 Glasgow, Vogt, & Boles, 1999; Glasgow, Mckay, Piette & Reynolds, 2001; Spoth, Redmond & Shin, 2000a).

Despite growing recognition of the important role implementation plays in successful prevention efforts, relatively little work has sought to demonstrate a causal link between implementation factors and participant outcomes (c.f., Hill, Brooks-Gunn, & Waldfogel, 2003; Imbens, 2000; Foster, 2003). In turn, failure to explore the implementation-to-outcome relationship limits our understanding of the mechanisms essential to successful programming. This gap is partially due to the inability of current methodological procedures within prevention science to account for the multitude of confounders responsible for variation in implementation factors (i.e., selection bias). The current paper illustrates a novel approach for improving causal inferences involving different implementation factors not easily randomized (e.g., participant attendance, program fidelity). Specifically, we seek to demonstrate the capacity of this approach to account for selection bias resulting from voluntary enrollment and participation in a family prevention program. Participant enrollment and attendance rates are implementation factors often associated with the reach of a prevention effort (Durlak & DuPre, 2008; Glasgow et al., 1999; Graczyk, Domitrovich, & Zins, 2003). For this demonstration we selected factors related to reach in an attempt to both build on related efforts to improve causal inference in behavioral intervention (Angrist & Hahn, 2006; Connell, Dishion, Yasui & Kavanagh, 2007; Imbens, 2000) and to illustrate this methodological approach with data that is often available to researchers (e.g., attendance records as opposed to fidelity observations).

In this context, we illustrate that propensity and marginal structural models (Holland, 1986; Robins, Hernán, & Brumback, 2000; Rubin, 2005) can be used to simultaneously estimate the effect of multiple implementation factors. We first present background on the role of implementation within prevention science, and then describe analytic steps for evaluating multiple implementation factors using propensity and marginal structural models. We demonstrate the utility of this approach in an outcome analysis of a real-world prevention trial known as PROSPER.

Accounting for Participant Selection Effects

Prevention researchers have drawn few causal inferences about how implementation factors impact program outcomes (c.f., McGowan, Nix, Murphy, Bierman, & CPPRG, 2010; Stuart, Perry, Le, & Ialongo, 2008). This is largely because current methodological approaches have a limited capacity to successfully account for selection biases that influence variation in implementation (Taylor, Graham, Cumsille, & Hansen, 2000). With regard to a prevention effort's reach, a variety of factors may influence participants' enrollment and attendance in family-focused prevention programs, including demographic factors (youth gender, race, SES; Dumas, Moreland, Gitter, Pearl, & Nordstrom, 2008; Winslow, Bonds, Wolchik, Sandler, & Braver, 2009), youth factors (past delinquency and parent perceptions; Brody, Murray, Chen, Kogan, & Brown, 2006; Dumas, Nissley-Tsiopinis, & Moreland, 2006) and logistic factors (e.g., access to programming; Rohrbach et al., 1994; Spoth, Redmond & Shin, 2000a—see Table 1). For example, pre-existing characteristics of enrolled participants (e.g., parenting style, education level, youth risk-level) may influence an outcome more than the intervention itself. Consequently, we demonstrate the value of using propensity models to account for the effects of these factors (i.e., confounders) on program enrollment and attendance. Specifically, we first examine the impact of program enrollment on underage drinking during middle and high school. Second, we simultaneously examine the impact of different levels of program attendance (high and low) on underage drinking. To accomplish this we model data from the PROSPER prevention trial.

Table 1.

Confounders of Program Use and Attendance in the Literature

Confounder	Variable Included in Model	References
• Family Ethnicity	• Youth Ethnicity	• Albin, Lee, Dumas, Slater, & Witmer, 1985; Holden, Lavigne, & Cameron, 1990; Prinz & Miller, 1994
• Parent & Child Age	• Child Age
• Marital Status	• Marital Status
• Family SES	• Participation in School Lunch Program	• Toomey et al, 1996; Williams et al., 1995; Dumas, Moreland, Gitter, Pearl & Nordstrom, 2008; Winslow, Bonds, Wolchik, Sandler & Braver, 2009
• Educational Attainment	• Youth Grades, Youth School Attitude, Bonding and Absences	• Rohrbach et al., 1994
• Parent Gender	• Parent Gender	• Dumas, Nissley-Tsiopinis & Moreland, 2006; Spoth & Redmond, 1995; Spoth, Redmond, Kahn, & Shin, 1997
• Perceived Child Susceptibility to Teen Problem Behaviors	• Youth Engagement in Problem Behaviors, Parent-Child Affective Quality
• Perceived Severity of Teen Problem Behaviors	• Youth Engagement in Problem Behaviors, Family Cohesion Scale
• Perceptions of Intervention Benefits	• Child Management Techniques Employed: Monitoring, Reasoning, Discipline
• Time Demands & Scheduling Issues	• Distance from Program, Frequency of Youth-Parent Activities

Traditionally, prevention researchers include confounders, unaccounted for by experimental designs, as covariates within their models (e.g., ANCOVA; Coffman, 2011; Schafer & Kang, 2008). However, this approach can account for only a limited number of confounders, especially if those confounders span across multiple levels (e.g., individual, family, and community factors). One method to account for a large number of confounders is to employ propensity scores to estimate the probability of a given level of a factor (e.g., enrolling in or attending a program; Rosenbaum & Rubin, 1983). Propensity scores (π_i) are traditionally the probability that an individual i receives a program (A_i) given measured confounders (X_i) (D'Agostino, 1998; Harder, Stuart, & Anthony, 2008, 2010; Rosenbaum & Rubin, 1983):

$${\pi }_i=P\left(A_i\right)=1\mid X_i$$

Propensity scores may be used within a marginal structural model to weight individuals in order to create balanced groups of participants based upon the confounders included in the propensity model. Marginal structural models are a class of causal models that may be used to define and estimate, from observational data, the causal effect of an intervention in the presence of time-varying exposures and confounders (Robins et al., 2000). Thus, assuming that no confounders are unaccounted for, this procedure results in groups that approximate those that would be achieved through randomization (Rosenbaum & Rubin, 1983, 1985; Sampson, Sharkey, & Raudenbush, 2008). Using propensity scores to balance different groups allow researchers to estimate the mean differences between the groups. This difference is commonly known as the `average causal effect' (Schafer & Kang, 2008). Interpreting this effect, under the assumption that all confounders are accounted for in the estimation of the propensity scores, allows researchers to draw causal inferences about group differences with a greater degree of confidence that would be possible otherwise (Coffman, 2011; Luellen, Shadish, & Clark, 2005; Schafer & Kang, 2008). The potential outcomes framework shares this assumption with traditional regression approaches (Rubin, 2005). A key distinction between common approaches for handling selection bias (e.g., ANCOVA) and marginal structural models is that the latter controls for confounding by changing the data to mimic what would be obtained in a randomized experiment whereas ANCOVA controls for confounding by including covariates in the model for the outcome. Next we describe the analytic steps for applying the potential outcomes framework and marginal structural models to outcome evaluation of the PROSPER prevention trial.

Illustrative Example: The PROSPER Delivery System

The PROSPER (PROmoting School-university-community Partnerships to Enhance Resilience) study is a community-randomized trial of an innovative approach for delivering substance use prevention programs (see Spoth, Greenberg, Bierman, & Redmond, 2004). There were two program components implemented in all intervention communities. All families were offered a family-based prevention program in the evenings when their child was in 6th grade (Strengthening Families Program 10–14) and 17% percent of families enrolled. The following year, all youth in intervention communities received evidence-based preventive interventions (EBIs) during the regular school day in the 7th grade. Evaluations of the PROSPER trial (28 communities, 17,701 adolescent participants) have demonstrated the system's effectiveness in promoting EBI adoption and implementation (Spoth, Guyll, Lillehoj, Redmond, & Greenberg, 2007; Spoth, Guyll, Redmond, Greenberg, & Feinberg, 2011; Spoth & Greenberg, 2011) as well as significantly reducing rates of adolescent alcohol and substance abuse (Spoth et al., 2007; 2011).

Our goal is to assess the specific impact of family program enrollment and attendance—in addition to the school program—on participant outcomes. In previous reports, 10^th grade youth in PROSPER intervention communities—compared to the control communities—were significantly less likely to report ever consuming alcohol—(Spoth et al., 2011). This evaluation differentiated between those that received only the school-based program and those that received both the school- and family-based programs. In addition, this evaluation assessed the potential differences between those enrolled in the family program with low versus high program attendance. Consequently, we seek to tease apart the effects of these two aspects of implementation on rates of underage drinking during middle and high school to demonstrate the value of the potential outcomes framework and marginal structural models for the study of implementation.

Methods

To evaluate the impact of program enrollment and attendance within the PROSPER trial, it is necessary to account for two types of selection bias: (1) the possible bias that impacts whether a family decides to enroll in the family program, and (2) the possible bias that influences enrolled families' rate of attendance. The family program delivered within PROSPER was a seven-session EBI known as the Strengthening Families Program: For Parents and Youth 10–14 (Spoth, Redmond, & Shin, 2000b, 2001). Thus, families who were enrolled (N = 859) could attend one to seven sessions. Figure 1 shows the distribution of families' attendance across all sessions within PROSPER. Most families in the trial attended five or more sessions (N = 671), but some families attended only one to four sessions (N=188). There appears to be a natural break in the distribution between those attending four sessions or less and those attending five sessions or more. Although differential participant attendance may be aggregated in a variety of ways, for the purpose of this demonstration we considered those who attended one to four sessions as having `low attendance' and those attending five or more sessions as having `high attendance'.

Figure 1.

Distribution of Family Attendance in SFP 10–14 in PROSPER Trial

Measures

Confounders

A wide variety of available measures are included in the propensity model to account for factors that may contribute to family program enrollment and attendance (Table 1). These include youth demographics (gender, ethnicity, age, enrollment in school lunch programs), functioning (stress management, problem solving, assertiveness, frequency of risk activities, frequency of self-oriented activities, academic achievement, frequency of school absences, attitude towards school, school bonding and achievement, deviant behaviors, antisocial peer behavior), proclivity towards alcohol and substance abuse (substance refusal intentions and efficacy, positive expectations, perceptions of and attitudes toward substance abuse, as well as cumulative alcohol and substance initiation and use indices), as well as family environment (parent marital status, youth residence with biological parents, affective quality between parents and youth, parental involvement, parent child management, family cohesion) and each family's geographic distance from the program site (see Chilenski & Greenberg, 2009; Redmond et al., 2009).

Underage Drinking

Each youth participant's initiation of alcohol consumption was measured annually from 6^th through 11^th grade. A dichotomous (Yes/No) self-report measure was used to evaluate the initiation of underage alcohol consumption (“Ever drunk more than a few sips of alcohol”).

Procedure

Applying the potential outcomes framework to evaluate the impact of multiple implementation factors entails five steps. These include (1) defining the causal effects, (2) estimating individuals' propensity scores, (3) calculating inverse propensity weights (IPWs), (4) evaluating balance, and (5) estimating the marginal structural models. We employed a multiple imputation approach to account for any missing data (20 imputations; D'Agostino & Rubin, 2000; Graham, Olchowski, & Gilreath, 2007; Little & Rubin, 2002, Schafer & Graham, 2002).

Defining Causal Effects

The first step is to define the causal effects that will be estimated in order to isolate each effect within the model. For the purpose of this analysis, let e denote program enrollment and a denote attendance. The potential outcomes for individual i at time t in community c are Y_itc(e,a). Thus, Y_itc (0,0) is the potential outcome if the individual does not enroll and does not attend, Y_itc(1,0) is the potential outcome if the individual enrolls but does not attend, Y_itc(1,1) is the potential outcome if the individual enrolls and attends at least 1 but fewer than 5 sessions, and Y_itc(1,2) is the potential outcome if the individual enrolls and attends 5 or more sessions. Note that there is a monotonic function in that if an individual does not enroll, then they cannot have a = 1 or a = 2; thus the potential outcomes Y_itc(0,1) and Y_itc(0,2) are equal to Y_itc(0,0). By employing the potential outcomes framework, we may estimate (0.1) the effect of enrolling in the family program, but not attending any sessions vs. not enrolling or attending,; (0.2) the effect of enrolling and attending at least 1 but fewer than 5 sessions vs. enrolling but attending no sessions; (0.3) the effect of enrolling and attending at least 1 but fewer than 5 sessions vs. not enrolling; (0.4) the effect of enrolling and attending 5 or more sessions vs. enrolling but attending no sessions; and (0.5) the effect of enrolling and attending 5 or more sessions vs. not enrolling at all. The effects are expressed as follows:

$$E\left[Y_{itc}\right(1,0)-Y_{itc}(0,0\left)\right]$$ (0.1)

$$E\left[Y_{itc}\right(1,1)-Y_{itc}(1,0\left)\right]$$ (0.2)

$$E\left[Y_{itc}\right(1,1)-Y_{itc}(0,0\left)\right]$$ (0.3)

$$E\left[Y_{itc}\right(1,2)-Y_{itc}(1,0\left)\right]$$ (0.4)

$$E\left[Y_{itc}\right(1,2)-Y_{itc}(0,0\left)\right]$$ (0.5)

Our notation for the potential outcomes implicitly makes the stable unit treatment value assumption at the community level (i.e., SUTVA; VanderWeele, 2008). This implies that for individual j living in Community A, enrollment or attendance is not influenced by the enrollment or attendance of an individual i in Community B although it may be influenced by other individuals in Community A. For the remainder of the article we will drop the subscripts for individual, time, and community. Additionally, we will not estimate any random effects for the community level although we account for the nested structure in the models.

The marginal structural model is given as follows:

$$E\left[Y\right(e,a\left)\right]={\beta }_0+{\beta }_{1}e+{\beta }_{2}ea$$

Thus, the causal contrasts given in Equations 1–5 are equal to:

$$({\beta }_0+{\beta }_1)-{\beta }_0={\beta }_1,$$

$$({\beta }_0+{\beta }_1+{\beta }_2)-({\beta }_0+{\beta }_1)={\beta }_2,$$

$$({\beta }_0+{\beta }_1+{\beta }_2)-{\beta }_0={\beta }_1+{\beta }_2,$$

$$({\beta }_0+{\beta }_1+2{\beta }_2)-({\beta }_0+{\beta }_1)=2{\beta }_2,\text{and}$$

$$({\beta }_0+{\beta }_1+2{\beta }_1)-{\beta }_0={\beta }_1+2{\beta }_2.$$

Note that by leaving out the main effect term for attendance, we have a monotonic function because

$$E\left[Y\right(1,1\left)\right]={\beta }_0+{\beta }_1+{\beta }_2,$$

$$E\left[Y\right(0,0\left)\right]={\beta }_0,$$

$$E\left[Y\right(1,0\left)\right]={\beta }_0+{\beta }_1,$$

$$\text{and}E\left[Y\right(0,1\left)\right]=E\left[Y\right(0,2\left)\right]=E\left[Y\right(0,0\left)\right]={\beta }_0.$$

Propensity Score Estimation

We first estimated propensity scores (π_e) for each youth who received either the school program or the school and family program together using a logistic regression model (i.e., the propensity model). Then a multinomial probit model was used to estimate the propensity scores of intervention participants having no, low, or high attendance (π_a) in the family program. We evaluated the `overlap', which indicates if there are individuals with similar propensity scores in each of the groups by considering the distributions of the propensities for being in either the school only or school and family program groups (Rosenbaum & Rubin, 1983). Without overlap casual comparisons between the two groups are generally not warranted. Here we observed substantial overlap between the family program enrollment and no-enrollment distributions (see Figure 2). Similarly, we considered the overlap of the low and high attendance groups and again observed substantial overlap of the propensities between the levels of attendance (see Figure 2).

Figure 2.

Overlap Diagnostics for Family Program Enrollment and Program Attendance Levels

Inverse Propensity Weight Calculation

Next the IPWs for the level of program enrollment e and attendance a actually received by each participant were calculated. The IPWs are similar to survey weights and allow us to make adjustments to the sample data to account for selection effects influencing both participant enrollment and attendance by up-weighting those that are underrepresented and down-weighting those that are overrepresented (Hirano & Imbens, 2001). When modeling multiple independent variables—such as the two implementation factors considered here—the product of the weights may be used (IPW_e*IPW_a –Robins et al., 2000; Spreewenberg et al., 2010).

$$\mathrm{If}\mathrm{e}={\text{School only, then IPW}}_e=\frac{1}{{\widehat{\pi }}_s}$$

$$\mathrm{If}\mathrm{e}=\text{School}+{\text{Family, then IPW}}_e=\frac{1}{{\widehat{\pi }}_{s+f}}$$

$$\mathrm{If}\mathrm{a}={\text{No Family Attendance, then IPW}}_a=\frac{1}{{\widehat{\pi }}_{a\left(0\right)}}$$

$$\mathrm{If}\mathrm{a}={\text{Low Attendance, then IPW}}_a=\frac{1}{{\widehat{\pi }}_{a\left(L\right)}}$$

$$\mathrm{If}\mathrm{a}={\text{High Attendance, then IPW}}_a=\frac{1}{{\widehat{\pi }}_{a\left(H\right)}}$$

where ${\widehat{\pi }}_s$ is the estimated propensity that a participant did not enroll in the family program, ${\widehat{\pi }}_{s+f}$ is the estimated propensity that an individual enrolled in the family program, ${\widehat{\pi }}_{a\left(0\right)}$ is the estimated propensity that a participant attended no family sessions, ${\widehat{\pi }}_{a\left(L\right)}$ is the estimated propensity that a participant attended between 1 and 4 sessions, and ${\widehat{\pi }}_{a\left(H\right)}$ is the estimated propensity that a participant attended between 5 and 7 family sessions.

Balance Diagnostics

We evaluated the balance of the different groups—before and after weighting—to ascertain whether the adjustment using the IPWs successfully resulted in group equivalence on the modeled confounders (Harder et al., 2010).This evaluation included a comparison of the standardized mean differences (SMDs) of the unweighted sample to the weighted sample for each of the confounders included in the propensity model, allowing us to ascertain whether the SMDs decreased after weighting (i.e., effect size comparison). It is recommended that these differences be less than .2 (in absolute value), which is considered a “small” effect size (Cohen, 1992; Harder et al., 2010). Figure 3 illustrates balance for both program enrollment and attendance within the PROSPER trial, as all weighted SMDs are less than .2.

Figure 3.

SMDs of Confounders Predicting Enrollment in School or School and Family Program and Attendance of Family Program

Outcome Analysis

The fifth step was to evaluate how differential implementation impacted underage drinking using the IPWs within marginal structural models to account for selection bias. In this case we modeled whether youth reports of underage drinking varied across different levels of program enrollment and attendance. To evaluate the effect of implementation, we constructed logistic generalized estimating equations for the marginal models to examine differences between implementation groups across time (categorical) using the IPW estimation method. The model is given as

$$\mathit{logit}\left(PR\right(Y=1\mid E=e,A=a\left)\right)={\beta }_0+{\beta }_{1}e+{\beta }_{2}ea+{\beta }_3\mathit{Time},$$

where β₁ is the effect of enrolling in the family program, β₂ is the effect of enrolling and having low attendance in the family program versus not attending any family sessions, and β₁+β₂ is the effect of enrolling in and having low attendance versus not enrolling and not attending any family sessions. Additionally, 2β₂ is the effect of enrolling in and having high attendance and β₁+2β₂ is the effect of enrolling and having high attendance versus not enrolling and not attending any family sessions. β₃ is the effect of time, which was coded to capture the unequal interval between baseline, 6 month follow-up and subsequent annual assessments of underage alcohol use (e.g., 0, .5, 1, 2 etc.). We estimated the model using PROC GLIMMIX in SAS 9.1 (SAS Institute, 2004), in which the outcome was the binary alcohol measure of whether youth had ever consumed alcohol. PROC GLIMMIX allows the logit link function for dichotomous outcome data, and error terms for non-normally distributed dependent variables (in this case binomial). Additionally, PROC GLIMMIX allows a weighting function that may be employed to include IPWs in the model. This procedure allowed us to analyze the three-level, nested design of the model, with measurement occasions nested within individuals and individuals nested within communities (Littell, Milliken, Stroup, Wolfinger, & Schabenberger, 2006).

Results

Utilizing the potential outcomes framework and marginal structural models described above, we evaluated the impact of enrolling in the family program in addition to the school program on reduction of underage drinking. We found a significant difference in the probability of alcohol use between those that received the school program and those that received both the family and school program (F = 2.66, p = .01). Results are plotted in Figure 4. As illustrated by the plot, the effect of receiving the family program in addition to the school program vs. only the school program translated into about a 5% decrease in the prevalence of youth who have ever consumed alcohol in the 11th grade (see Figure 4).

Figure 4.

Weighted Prevalence of Underage Drinking by Program and Attendance Level

Next, we assessed the impact of differential program attendance in the context of program enrollment. We did not find significant differences between youth who enrolled in the school and family program and had low attendance compared to those who did not enroll in the family program at all (F = 1.15, p = .32). However, we found a significant difference between those who did not enroll in the family program and those that enrolled and had high attendance in the family program (F = 3.03, p < .001). Attendance in 5 or more sessions of the family program—compared to not enrolling in the family program—translated into about a 7% decrease in the prevalence rate of underage drinking (see Figure 4).

Discussion

These results illustrate the value of employing IPWs within marginal structural models for disentangling the effects of implementation factors in real-world contexts. Through the application of the potential outcomes framework, this approach provides a method to account for selection effects that lead to differential program enrollment and attendance. This method permits researchers to make stronger causal inferences about the impact of differential implementation. Here these methods allowed us to evaluate the impact of enrolling in a family program in addition to a school-based EBI to reduce underage alcohol use. Below we discuss three major implications of this application for the field of prevention science.

The first implication pertains to the development of increasingly efficient prevention programs. As researchers seek to optimize prevention efforts, they will need to assess the value of each program, session, and component and trim away any material that is ineffective or counterproductive in order to optimize prevention programs. While increasingly sophisticated experimental designs are being used to conduct such work (e.g., Collins, Murphy, Nair, & Strecher, 2007), researchers may also employ the approach demonstrated above to existing data when undertaking this optimization process. For instance, here we demonstrated the added value of receiving a family-based EBI in addition to a school-based EBI and emphasized the importance of program attendance. If participants' attendance in different components or sessions is tracked this method could be applied to program optimization in a similar fashion—allowing researchers to test the added value of specific curriculum components as opposed to the program as a whole.

A second implication of this work is the value that information from evaluations such as the one presented here may have when scaling-up EBIs. Specifically, because evaluations that employ propensity scores may be used in the context of real-world prevention trials (e.g., PROSPER), the inferences are likely to be more generalizable than those drawn from smaller efficacy trials. This is illustrated by the fact that program implementation is likely to be much more comparable in dissemination trials to that of real-world settings than evaluations that use more restrictive designs (Flay et al., 2005).

A third implication pertains to ongoing efforts to enhance and maintain implementation factors—in particular enrollment and attendance rates. Using these statistical approaches we were able to better represent the differential influence of family program attendance. This information may in turn be persuasive to local implementers about the importance of program attendance. Possibly findings such as these could even encourage community efforts to allocate additional resources to increasing participants' attendance in family-based EBIs.

Conclusions

This work demonstrates the value of using potential outcomes and marginal structural models for the study of implementation factors within prevention science. Studies that collect large amounts of data on a variety of domains, including on implementation factors and the dynamics that may influence implementation factors, are especially suited for this type of analysis. One limitation of this work is that, while it is possible to include multiple implementation factors within a marginal structural model, we included only two within this evaluation. Further multivariate analyses of how implementation factors influence participant outcomes could provide greater insight into best practices for facilitating EBI implementation.