Skip to main content
NCBI home page
NIHPA Author Manuscripts logoLink to NIHPA Author Manuscripts
. Author manuscript; available in PMC: 2019 Jun 1.
Published in final edited form as: Eval Health Prof. 2018 Feb 20;41(2):270–289. doi: 10.1177/0163278718759397

Effects of exposure to the Communities That Care prevention system on youth problem behaviors in a community-randomized trial: Employing an inverse probability weighting approach

Isaac C Rhew a,*, Sabrina Oesterle b, Donna Coffman c, J David Hawkins b
PMCID: PMC5943175  NIHMSID: NIHMS949654  PMID: 29463119

Abstract

Earlier intention-to-treat (ITT) findings from a community-randomized trial demonstrated effects of the Communities That Care (CTC) prevention system on reducing problem behaviors among youth. In ITT analyses, youth were analyzed according to their original study community’s randomized condition even if they moved away from the community over the course of follow-up and received little to no exposure to intervention activities. Using inverse probability weights (IPWs), this study estimated effects of CTC in the same randomized trial among youth who remained in their original study communities throughout follow-up. Data were from the Community Youth Development Study, a community-randomized trial of 24 small towns in the United States. A cohort of 4407 youth was followed from fifth grade (prior to CTC implementation) to eighth grade. IPWs for one’s own moving status were calculated using fifth and sixth grade covariates. Results from inverse probability weighted multilevel models indicated larger effects for youth who remained in their study community for the first two years of CTC intervention implementation compared to ITT estimates. These effects included reduced likelihood of alcohol use, binge drinking, smokeless tobacco use and delinquent behavior. These findings strengthen support for CTC as an efficacious system for preventing youth problem behaviors.

Keywords: community-randomized trial, community prevention system, inverse probability weighting, substance use, delinquency, adolescents

Introduction

Although there is evidence from randomized controlled trials demonstrating effectiveness of interventions to prevent youth substance use and other problem behaviors (Mihalic et al, 2004; O’Connell, Boat & Warner, 2009), few tested and effective programs have been widely implemented in communities and many service agencies and schools use prevention strategies with little or no evidence of effectiveness (Ennett et al, 2003; Kumpfer and Alvarado, 2003). Strategies that rely on community coalitions have the potential to translate findings from prevention research into everyday practice (Fagan, Hawkins, & Catalano, 2011; Foster-Fishman, Nowell, & Yang, 2007; Mzarek & Haggerty, 1994). The Communities That Care (CTC) prevention system is one such promising coalition-based approach (Fagan et al, 2009). CTC is a manualized system that mobilizes and empowers local coalitions consisting of diverse stakeholders to develop and implement a science-based approach as a basis for community prevention services (Hawkins, Catalano & Arthur, 2002). With training and technical assistance from CTC staff, CTC coalitions learn how to collect epidemiologic data on risk and protective factors, prioritize risk and protective factors to address in their community, select from a menu of tested and effective prevention programs, and implement and monitor the programs with fidelity.

CTC’s model is is based on principles of prevention science (Coie et al., 1993; Mzarek & Haggerty, 1994) and theories of community competence (Eng & Parker, 1990), public health promotion (Bracht & Kingsbury, 1990; Butterfoss, Goodman, & Wandersman, 1993), and the social development model (Catalano & Hawkins, 1996). According to CTC’s theory of change, provision of CTC training and technical assistance will mobilize a community coalition of diverse stakeholders, increase the coalition members’ commitment to using a science-based approach to prevention, stimulate broader adoption of science-based prevention approaches among the community’s prevention service providers, and ultimately increase the use of tested and effective preventive interventions that address risk and protective factors prioritized by the community. These prevention system changes are hypothesized to produce intended changes in these prioritized risk and protective factors, which should in turn lead to decreases in the prevalence of youth problem behaviors.

Evidence from a large-scale community-randomized controlled trial (RCT), the Community Youth Development Study (CYDS), shows protective effects of CTC. In a cohort of 4407 youth followed from fifth to eighth grade, youth from communities randomly assigned to receive training in and technical assistance with the CTC system compared to youth from control communities were less likely to report alcohol and smokeless tobacco use in the past month, binge drinking in the past two weeks, and engaging in delinquent behaviors in the past year (Hawkins et al., 2009). These original multi-level analyses were based on the intention-to-treat (ITT) design such that all youth were retained even if they moved away from the community during the active intervention phase, and they were analyzed according to their original study community’s randomized condition. In this community-randomized trial, 15% of the participants moved away from their original study community between 6th grade and the 8th grade follow-up assessment. Because participants who moved away from communities assigned to CTC received less exposure to the CTC intervention activities, findings from the ITT design may have underestimated the magnitude of the effect of CTC for those who remained in their original experimental communities during the first two years of intervention implementation.

Challenges of estimating effects of exposure in a RCT

Estimating an effect of an intervention among those actually exposed to intervention in a community-randomized trial is not straightforward. On the surface, it may seem reasonable to conduct sub-group analyses among the stayers to estimate effects of CTC among those who actually received full “exposure” to CTC. However, by examining CTC effects only among the stayers, the analyses would be conditioned on moving status, which is assessed post-randomization and thus not subject to experimental protections afforded by randomization. It is plausible that the likelihood of remaining in one’s original study community may differ by study condition. If this were the case, failing to account for factors that may be associated with moving status and the outcome could bias estimates of the intervention effects (Toh & Hernán, 2008). For example, changes in a youth’s level of substance use post-baseline may be associated with likelihood of moving (Gasper, DeLuca, & Estacion, 2010) as well as the substance use outcome at the follow-up assessment. Not accounting for earlier levels of substance use would bias the association between CTC and substance use among those who remained in the study community.

Inverse Probability Weighting

Different methods to estimate effects of an intervention according to level of exposure or adherence in randomized trials have been developed. The instrumental variable (IV) and its extensions (Jo, Asparouhov, Muthen, Ialongo, & Brown, 2008; Stuart, Perry, Le, & Ialongo, 2008; Sussman & Hayward, 2010) are one class of approaches. IV methods examine how randomized assignment, the instrument, is related to exposure to the intervention and the outcome and then using this information estimate the effects of exposure to the intervention on the outcome. Another analytic approach to estimate unbiased effects in youth exposed to an intervention in a RCT uses inverse probability of treatment weight (IPW) estimators. Based on a counterfactual framework, the IPW approach represents an extension of propensity score methods (Rosenbaum & Rubin, 1983) and has been most commonly used to strengthen causal inference in observational studies (Austin, 2011; Robins, Hernan, & Brumback, 2000). In observational studies that utilize IPW methods to estimate effects of an exposure, participants are weighted by the inverse of one’s model-predicted probability, or propensity, of his/her given exposure status according to covariate history. Applying these weights to the study sample yields a “pseudo-population,” where distribution of putative confounders is similar between the exposed and unexposed, and thus estimates of effects of the exposure should be unbiased (i.e., plausible differences between exposure groups are removed statistically through the propensity weight).

The IPW approach has also been applied in the context of RCTs to estimate unbiased effects of an intervention in those who were exposed to the intervention that they were randomly assigned to receive (Toh, Hernández-Díaz, Logan, Rossouw, & Hernán, 2010). In this scenario, participants are weighted by the inverse of the model-predicted likelihood of observed adherence to their assigned treatment condition (e.g., active treatment or placebo control). Under the assumption that all relevant covariates are measured, often referred to as the exchangeability assumption, the application of IPWs to the sample removes potential bias by making the weighted covariate distribution among those who adhered to their assigned treatment similar to the full sample. This allows an investigator to make inferences regarding the effects of the intervention as if all participants adhered to the study condition as randomly assigned (Toh & Hernán, 2008). In the CYDS community-randomized trial, youth who remain in their original communities and receive greater “exposure” to their assigned intervention are analogous to those who adhered to an assigned treatment. A similar IPW scheme could be used to estimate unbiased effects in those who stayed in their communities, irrespective of condition, where individual participants are weighted by the inverse probability of staying in the study according to covariates. This would allow inferences regarding the effects of CTC had all participants remained in their original study communities.

Focus of the Current Study

In the current study, we examined effects of CTC on past month drug use and past year delinquency in the 8th grade, 3 years after baseline (pre-intervention), among youth who remained in their original study communities during the first two years of CTC implementation. We applied an IPW methodology to reduce potential bias associated with conditioning on remaining in the study community, a post-randomization variable. We expected effects of CTC to be larger in magnitude among those who remained in their assigned study community through eighth grade compared to the original ITT effect estimates. We based this expected outcome on the premise that those who lived in study communities for a longer period of time were more likely to be exposed to the community-level prevention system and benefit from tested and effective preventive programs chosen using the CTC process.

Method

Data for this study were obtained from the Community Youth Development Study (CYDS), a community-RCT of CTC conducted in 24 relatively small towns located in Colorado, Illinois, Kansas, Maine, Oregon, Utah, and Washington. The towns were selected from a larger sample of 41 communities participating in a previous observational study examining diffusion of science-based prevention strategies (Arthur, Glaser, & Hawkins, 2005). These 24 towns were deemed eligible for the CYDS because they did not show evidence for adopting a science-based approach to prevention (i.e., they did not select and use a tested and effective prevention intervention to addressed prioritized risk factors; Arthur, Ayers, Graham, & Hawkins, 2003; Hawkins et al., 2008). For the RCT, the participating towns were matched within each state according to population size, racial/ethnic composition, crime rate, and socioeconomic indicators. In 2003, one town from each pair was randomly assigned to the CTC condition and the other to the control condition.

CTC Implementation

CTC training and implementation were initiated between April and August of 2003 in those communities assigned to the CTC intervention condition. Certified trainers administered six training sessions in CTC over the course of six to 12 months. As part of this training, community leaders were oriented to the CTC system and created a coalition of diverse community stakeholders to implement CTC. Coalition members were trained in using epidemiologic data collected from youth residing in the community. The coalition learned how to prioritize risk factors that should be targeted for prevention activities, to select appropriate tested and effective intervention programs, to implement programs with fidelity, and to monitor implementation and outcomes of the selected evidence-based programs. In addition to training, CTC staff provided regular technical assistance through weekly phone calls, e-mails, and annual site visits. CTC communities also received study funding to support hiring a local community coordinator and sponsor prevention program activities, which were implemented beginning in the 2004 – 2005 school year. Proactive technical assistance and funding for programs and a community coordinator were provided to CTC communities until the spring of 2008. Control communities also collected behavioral surveillance data on students residing in their catchment area; however, they did not receive any training in or funding for CTC activities from the CYDS project during the course of the study.

Sampling Procedures

A longitudinal panel of youth was recruited in the 24 study communities from all students attending fifth grade in public schools during the 2003–2004 school year. Students were surveyed annually each spring through the eighth grade. A total of 4407 students whose parents provided written informed consent were enrolled in the longitudinal study (76.2% of those eligible in CTC and 76.7% of those eligible in control communities).

At each study assessment participants in the longitudinal panel were administered the Youth Development Survey, a paper-and-pencil questionnaire used to collect self-report behavioral data from youth in the community. The survey was designed to be administered in schools and completed during a 50-minute classroom period. To ensure confidentiality, a study identification number used to track individual students, but no names or other identifying information, was included on the survey. Prior to taking the survey, participants read and signed assent statements indicating that they were informed of their rights as study participants and agreed to participate. For their participation, students received small gifts valued from about $5 to $8. Of those enrolled in the longitudinal study, 96% completed the survey in eighth grade whether they remained in the original study community or not.

Survey Measures

Drug use

Students reported past 30-day frequency of use for alcohol, cigarettes, marijuana, smokeless tobacco, inhalants, cocaine, methamphetamines, psychedelics, and prescription drugs. Consistent with the Monitoring the Future (MTF) national survey of adolescents (Johnston et al, 2016), the response format for these substance use items ranged from 0 (“0 occasions”) to 6 (“40 or more occasions”). Responses were dichotomized to indicate any (1) or no (0) past month use. The low prevalence rates for the younger cohort (Brown et al., 2009) necessitated using a binary form of drug use. This choice is consistent with the primary goals of prevention, which are to delay drug onset as a means of preventing later substance abuse and dependence (Grant & Dawson, 1997; Wagenaar et al, 2000). In keeping with a strategy used in the MTF study (Johnston et al, 2016), we also created an index of illicit drug use (cocaine, methamphetamines, psychedelics and prescription drugs) additively indicating “any use” of these substances.

Delinquent behavior

In sixth grade, participants indicated the frequency of engaging in four delinquent acts over the past year (stealing, property damage, shoplifting, and attacking someone). In eighth grade, five additional items were asked (carrying a gun to school, beating up someone, stealing a vehicle, selling drugs, and being arrested). We created a dichotomous variable to indicate whether students had engaged in any of the nine acts in eighth grade, which is consistent with earlier CYDS analyses.

Covariates

A number of covariates were used to calculate the IPWs and as auxiliary variables for imputation of missing data. Demographic characteristics included gender, race/ethnicity, and parents’ level of education as well as whether the student lived with both biological parents. Additional risk and protective factor covariates included student’s perception of community attitudes towards drugs, family history of antisocial behavior, poor family management, favorable attitudes towards drugs, peers’ belief in the moral order, friends’ use of other drugs, intentions to use drugs as an adult, interaction with antisocial peers, and frequency of religious service attendance. These measures have shown both strong internal reliability and predictive validity for problem behaviors (Arthur, Hawkins, Pollard, Catalano, & Baglioni, 2002). These risk and protective factor measures also showed good psychometric properties in this sample with Cronbach’s alpha at the fifth grade assessment ranging from .66 for interaction with antisocial peers to .79 for favorable attitudes towards drugs.

Moving status

Prior to each data collection period, members of the investigative staff contacted participating schools in study communities to assess whether youth participating in CYDS remained enrolled at the school. For youth no longer attending the school, study staff verified their new location by contacting the youths’ new school as indicated by transfer records provided by the prior school or by contacting parents or other family members directly. Movers were defined as those who were not attending a school in the study community anymore or who were living at least 30 minutes away by car from their original study community prior to the seventh or eighth grade study assessment. Stayers were defined as those students who were present in the original assigned study community at both seventh and eighth grade study waves.

Analytic Plan

To account for potential bias associated with estimating effects of CTC among those who remained in their original study communities, we employed the use of IPWs. The IPW strategy involves two steps: 1) calculating IPWs for an individual’s observed moving status, and 2) applying the IPWs to multilevel models in order to estimate program effects of CTC on problem behaviors among stayers.

Calculating IPWs

For the denominator of the IPWs, the likelihood of each student’s observed moving status (1: moved; 0: stayed) was estimated using logistic regression, based on fifth and sixth grade past month drug use, risk and protective factors, living in a two-parent household, demographic characteristics, and study condition. In this study with individuals nested within 24 distinct communities, there may be unmeasured factors that make students residing within a community more similar to each other than students from a different community (Murray, 1998). To account for clustering within these communities, we also modeled the study community using dummy variables in the regression model. Schuler and colleagues conducted simulation studies showing that a community fixed-effect approach to account for clustering in propensity score calculations with multilevel data yielded little bias and good 95% confidence interval coverage (Schuler, Chu, & Coffman, 2016). To calculate the model-predicted probability of staying for participants who remained in the community through eighth grade, we subtracted the predicted probability of moving from 1. Depending on the strength of correlation between covariates and moving status, the weights can take on extreme values, which could lead to inflated standard errors (Cole & Hernan, 2008). To account for this, we used a stabilized form of the IPW to improve model precision (Austin & Stuart, 2015). The stabilized IPWs were calculated by adding a numerator to the IPW described above that was the model-predicted probability of a student’s moving status according only to study condition and community (Schuler et al., 2016). The resulting mean of the stabilized IPWs was 1.00 (SD = .16; range: .49, 1.71).

Multilevel models

To account for the nesting of participants within communities, which were nested within matched pairs, multilevel models with random intercepts for community and matched pair were conducted to test effects of CTC on problem behaviors. Because outcomes were dichotomous, we specified a logistic form of the model to estimate odds ratios (ORs). The model included individual-level covariates for baseline age, gender, race (White vs. non-White), Hispanic ethnicity, parents’ highest level of education, religious service attendance, rebelliousness, and baseline measure of the corresponding outcome; and the community-level indicator for study condition (CTC [1] vs. control [0]) and community-level covariates for student population size and percentage of student population receiving free or reduced price lunch, which serves as an indicator of community-level socioeconomic disadvantage.

We estimated the multilevel models with IPWs only among the stayers. We compared the results to non-weighted models among stayers to understand the potential bias associated with not accounting for covariates that may influence the likelihood of staying vs. moving. We also compared the results to ITT effects by estimating non-weighted multilevel models in the full sample. Although ITT effects were estimated earlier (Hawkins et al., 2009), they were re-estimated for the comparison with the IPW analyses because we used a different analysis software package to accommodate the weights.

Missing data

Multiple imputation was used to account for missing data (Graham, 2009). In order to match the methods used in the original ITT analyses (Hawkins et al, 2008) we created 40 imputed datasets (Brown et al., 2009; Graham, Olchowski, & Gilreath, 2007) using a multivariate normal approach based on the full covariance matrix. This approach uses an expectation maximization (EM) algorithm and generates random draws using an iterative Markov Chain Monte Carlo (MCMC) procedure. The variables in the imputation model included all predictors used to estimate the IPWs and those covariates used in the final multilevel models. IPWs and weighted multilevel models were estimated within each of the 40 datasets. Parameter estimates and standard errors were combined across data sets using Rubin’s rules to account for missing data uncertainty between and within the imputed datasets (Rubin, 1987). All statistical analyses, including the multiple imputation, were conducted using Stata 14.0 (StataCorp, College Station, TX).

Results

Table 1 shows selected fifth grade (baseline) and sixth grade characteristics by whether the student stayed in the community or left the study community prior to the eighth grade data collection. Compared to the movers, stayers were more likely to have been from CTC as opposed to control communities. Furthermore, students who remained in their assigned community condition as opposed to those who moved away from their experimental communities were more likely to be Hispanic, have more educated parents, and report lower risk in the sixth grade (e.g. lower levels of perceived drug availability, alcohol use, and antisocial behavior).

Table 1.

Selected participant characteristics according to follow-up moving status

Characteristics Stayers
N = 3765
Mean (SD) or %		 Movers
N = 640
Mean (SD) or %
CTC study condition 57.05 39.84
Female sex 49.32 52.34
White race 67.46 66.95
Hispanic ethnicity 20.80 17.50
Parent completed some college or more 72.29 67.81
Perceived availability of drugs, sixth grade −.16 (.70) −.03 (.80)
Perceived community laws and norms favorable to drug use, sixth grade −.12 (.69) −.03 (.75)
Peers’ favorable attitudes towards drugs, sixth grade −.10 (.73) −.03 (.77)
Past 30 day alcohol use, sixth grade 6.37 8.91
Any past year delinquent behavior, sixth grade 22.02 29.85
Open in a new tab

Table 2 shows the unadjusted prevalence of eighth grade problem behaviors by study experimental condition among only the 3765 participants who remained in their study community through eighth grade. Among these stayers, the unadjusted prevalence of all problem behaviors was higher in youth from control compared to CTC communities.

Table 2.

Prevalence of problem behaviors in eighth grade by study condition among youth who remained in their original study community through eighth grade.

Problem Behavior Outcome CTC
N = 2148
%		 Control
N = 1617
%
Past month alcohol use 16.37 23.29
Past two week binge drinking 5.22 9.47
Past month marijuana use 3.94 5.66
Past month cigarette use 5.28 7.55
Past month smokeless tobacco use 2.42 4.71
Past month inhalant use 4.52 5.29
Past month other drug use 2.84 4.27
Past year delinquent behavior 31.80 41.88
Open in a new tab

Table 3 displays the estimates for CTC effects in the full sample from non-weighted models (the ITT estimate), inverse probability weighted effect estimates for stayers, and unweighted effect estimates for stayers. Consistent with the original ITT findings, these ITT analyses showed that youth from communities randomized to CTC intervention versus control had a significantly lower likelihood of engaging in alcohol use, binge drinking, smokeless tobacco use, and delinquent behavior in eighth grade. When comparing the inverse probability weighted results to the ITT results, effects of CTC were stronger (i.e., departed more from a value of 1) in the IPW case for all problem behaviors among those who remained in their community through eighth grade.

Table 3.

Intent-to-treat (ITT) odds ratios (ORs) for CTC effects in the full sample and weighted and unweighted ORs among those who remained in their original communities

Eighth grade outcome Full sample, ITT
N = 4407		 Stayers, weighted
N = 3765		 Stayers, unweighted
N = 3765
OR 95% CI OR 95% CI OR 95% CI
Past month alcohol use .81 .67, .97 .74 .63, .88 .74 .62, .90
Past two week binge drinking .74 .56, .97 .67 .55, .83 .68 .51, .91
Past month marijuana use .88 .65, 1.21 .77 .53, 1.14 .80 .57. 1.14
Past month cigarette use .80 .62, 1.03 .74 .52, 1.06 .77 .55, 1.07
Past month smokeless tobacco use .61 .42, .89 .56 .41, .77 .57 .38, .85
Past month inhalant use .90 .61, 1.33 .88 .54, 1.43 .88 .60, 1.30
Past month other drug use .83 .58, 1.20 .80 .60, 1.05 .81 .55, 1.21
Past year delinquent behavior .73 .62, .86 .72 .59, .88 .73 .61, .87
Open in a new tab

Note: Both weighted and unweighted models included individual-level covariates for baseline age, gender, race (White vs. non-White), Hispanic ethnicity, parents’ highest level of education, religious service attendance, rebelliousness, and baseline measure of the corresponding outcome; and community-level covariates for student population size and percentage of student population receiving free or reduced price lunch

Consistent with the previous ITT findings, statistically significant effects of CTC from the IPW analyses were observed on past month alcohol and smokeless tobacco use, past two week binge drinking, and past year delinquent behavior among those who remained in their original communities. Analyses were also performed using standard unweighted models among the stayers and movers. For some outcomes (marijuana, cigarette, and other drug use), the unweighted ORs were somewhat closer to a value of 1 compared to the weighted ORs. For others, the unweighted estimates were similar to the weighted.

Discussion

In this study, we used an IPW approach to estimate effects of the CTC prevention system on youth problem behaviors among youth who were most likely to be exposed to the CTC intervention because they remained in their assigned study community compared to individuals who moved during the course of CTC implementation. By staying in the CTC assigned community youth were more likely to have benefited from the prevention activities that CTC offers. Study findings indicated that weighted CTC effects were generally stronger than estimates previously obtained using ITT analyses. Thus, the results from this study further reinforce the efficacy of CTC in preventing youth problem behaviors by showing that effects were stronger among those with the greatest amount of exposure to intervention activities.

The IPW approach minimizes bias associated with conditioning on moving status, a post-randomization variable. The methodology can account for a number of factors associated with moving status as well as later problem behaviors that are often not considered in traditional subgroup analyses that would only analyze effects among those who remained in the community (the “treated”). Causal inference for estimates of CTC effects among the stayers is thereby strengthened by accounting for multiple covariates that could bias the association between CTC and outcomes in the stayers. When comparing weighted to unweighted models for stayers, the ORs were generally fairly similar. This suggests that the set of covariates included in the unweighted multilevel model may have been sufficient to account for any potential confounding between the study condition and outcomes when conditioned on stayer status.

As described earlier, other methods, namely IV approaches and their extensions, have been used to address similar questions of limited exposure to the intervention in randomized. IV approaches compare the occurrence of the outcome in those who were randomly assigned and actually exposed to the intervention to the full control group. In this study, youth from both CTC and control communities could have moved from their study communities during the course of follow-up. Thus, the comparison between the “stayers” in CTC communities and “stayers” in control communities was the most relevant when estimating CTC intervention effects. This scenario is similar to a placebo blind randomized drug trial where participants could adhere to either the intervention treatment medication or to the control placebo. If students from control communities did not move, then the application of instrumental variable approaches would have been more appropriate.

The present findings suggest that greater exposure to the CTC intervention activities is associated with greater reduction in problem behaviors, particularly alcohol, smokeless tobacco use, and delinquent behavior. These findings are consistent with prior analyses using the same RCT data that considered other aspects of CTC exposure. Using data from a repeated cross-sectional survey spanning two waves of data collected four years apart, post-hoc analyses showed that CTC effects were somewhat larger in magnitude in youth from CTC communities that had greater intervention saturation, characterized by the proportion of youth in the community reached by prevention programs, compared to those from CTC communities with lower saturation (Rhew et al., 2016).

These results build on existing research supporting the effectiveness of CTC on reducing problem behaviors as well as literature regarding its successful implementation in accordance with its theory of change. For example, additional ITT results showed sustained effects of CTC into later grades after study funding for intervention activities was removed (Hawkins, Oesterle, Brown, Abbott, & Catalano, 2013; Hawkins et al., 2012). CTC communities in the CYDS implemented prevention programs with high fidelity (Fagan, Hanson, Hawkins, & Arthur, 2008) and implemented more evidence-based programs than control communities. There was also greater reach of these programs in the CTC communities during the course of the study (Fagan, Hanson, Briney, & Hawkins, 2012). Studies also indicate that communities randomized to the CTC condition compared to control communities had higher levels of adoption of a science-based approach to prevention and other prevention system characteristics during the active implementation phase of the randomized trial (Brown, Hawkins, Arthur, Abbott, & Van Horn, 2008; Brown, Hawkins, Arthur, Briney, & Abbott, 2007) and after funding for intervention activities ceased (Gloppen et al., 2016; Rhew, Brown, Hawkins, & Briney, 2013).

Study Limitations

There were several limitations to this study that are worth noting. The communities participating in this study were all relatively small towns. This maintained the rigors of the study design and implementation of CTC. However, it is unclear whether the results will be generalizable to larger and more urban communities. We based estimates of program effects using self-report measures and did not collect any other form of drug surveillance or third-party reports (i.e., parents or teachers). However, there is considerable evidence that self-reports are accurate and valid measures of behavior even with relatively young students (Brown et al, 1998; Johnston et al, 2016). In the current study, the survey from which the problem behavior items were obtained has been used extensively in research and practice settings. Further, we would expect any measurement error from unreliable measures to be non-differential with respect to experimental condition and thus lead to conservative estimates of effects. It is possible that not all relevant covariates that are associated with moving status and the problem behavior outcomes were measured, which would violate the exchangeability assumption of the IPW approach (Hernán & Hernández-Díaz, 2012). Although this assumption is untestable, it is not likely that it was violated because we incorporated numerous covariates into models to calculate the IPWs. It is possible that the IPWs can be unstable when model-predicted probabilities are very small or large, which could lead to inflation of standard errors of effect estimates. However, we used a stabilized form of the IPWs, which reduces the impact of extreme values of the IPWs.

Conclusions and Future Directions

This study found that students who remained in their original study community benefited from their exposure to the CTC prevention activities and were less likely to report engaging in problem behaviors. These effects in the stayers were stronger compared to the ITT effect. Potential biases associated with conditioning on moving status, a post-randomization variable, were minimized through the use of an IPW approach. For investigators interested in moving beyond ITT to estimate unbiased effects of exposure to an intervention as part of a RCT, the IPW approach may be a useful tool. In future work, this IPW approach could be extended to more complex scenarios such as the estimation of CTC effects on the initiation of problem behaviors using a survival analysis approach. Further, this approach may be warranted for other community RCTs where, for a variety of reasons, there is noticeable out-migration from study communities.

Acknowledgments

This work was supported by the National Institute on Drug Abuse (R01 DA015183), with co-funding from the National Cancer Institute, the National Institute of Child Health and Human Development, the National Institute of Mental Health, the Center for Substance Abuse Prevention, and the National Institute on Alcohol Abuse and Alcoholism. The content of this paper is solely the responsibility of the authors and does not necessarily represent the official views of the funding agencies.

Footnotes

Declaration of Conflicting Interests

The authors declare that there is no conflict of interest.

References

  1. Arthur MW, Ayers CD, Graham KA, Hawkins JD. Mobilizing communities to reduce risks for drug abuse: a comparison of two strategies. In: Bukowski WJ, Sloboda Z, editors. Handbook of drug abuse prevention: Theory, science and practice. New York, NY: Kluwer Academic/Plenum Publishers; 2003. pp. 194–144. [Google Scholar]
  2. Arthur MW, Glaser RR, Hawkins JD. Steps towards community-level resilience: Community adoption of science-based prevention programming. In: Peters RDV, Leadbeater B, McMahon RJ, editors. Resilience in children, families, and communities: Linking context to practice and policy. New York, NY: Kluwer Academic/Plenum; 2005. pp. 177–194. [Google Scholar]
  3. Arthur MW, Hawkins JD, Pollard JA, Catalano RF, Baglioni AJ., Jr Measuring risk and protective factors for substance use, delinquency, and other adolescent problem behaviors. The Communities That Care Youth Survey. Evaluation Review. 2002;26(6):575–601. doi: 10.1177/0193841X0202600601. [DOI] [PubMed] [Google Scholar]
  4. Austin PC. An introduction to propensity score methods for reducing the effects of confounding in observational studies. Multivariate Behavioral Research. 2011;46(3):399–424. doi: 10.1080/00273171.2011.568786. [DOI] [PMC free article] [PubMed] [Google Scholar]
  5. Austin PC, Stuart EA. Moving towards best practice when using inverse probability of treatment weighting (IPTW) using the propensity score to estimate causal treatment effects in observational studies. Statistics in Medicine. 2015;34(28):3661–3679. doi: 10.1002/sim.6607. [DOI] [PMC free article] [PubMed] [Google Scholar]
  6. Bracht N, Kingsbury L. Community organization principles in health promotion. In: Bracht N, editor. Health promotion at the community level. Thousand Oaks, CA: Sage; 1990. pp. 66–88. [Google Scholar]
  7. Brown EC, Graham JW, Hawkins JD, Arthur MW, Baldwin MM, Oesterle S, … Abbott RD. Design and analysis of the Community Youth Development Study longitudinal cohort sample. Evaluation Review. 2009;33(4):311–334. doi: 10.1177/0193841X09337356. [DOI] [PMC free article] [PubMed] [Google Scholar]
  8. Brown EC, Hawkins JD, Arthur MW, Abbott RD, Van Horn ML. Multilevel analysis of a measure of community prevention collaboration. American Journal of Community Psychol. 2008;41(1–2):115–126. doi: 10.1007/s10464-007-9154-8. [DOI] [PubMed] [Google Scholar]
  9. Brown EC, Hawkins JD, Arthur MW, Briney JS, Abbott RD. Effects of Communities That Care on prevention services systems: Findings from the Community Youth Development Study at 1.5 years. Prevention Science. 2007;8(3):180–191. doi: 10.1007/s11121-007-0068-3. [DOI] [PubMed] [Google Scholar]
  10. Brown SA, Myers MG, Lippke L, Tapert SF, Stewart DG, Vik PW. Psychometric evaluation of the Customary Drinking and Drug Use Record (CDDR): a measure of adolescent alcohol and drug involvement. Journal of studies on alcohol. 1998;59(4):427–438. doi: 10.15288/jsa.1998.59.427. [DOI] [PubMed] [Google Scholar]
  11. Butterfoss FD, Goodman RM, Wandersman A. Community coalitions for prevention and health promotion. Health Education and Research. 1993;8(3):315–330. doi: 10.1093/her/8.3.315. [DOI] [PubMed] [Google Scholar]
  12. Catalano RF, Hawkins JD. The social development model: A theory of antisocial behavior. In: Hawkins JD, editor. Delinquency and crime: Current theories. New York, NY: Cambridge University Press; 1996. pp. 149–197. [Google Scholar]
  13. Coie JD, Watt NF, West SG, Hawkins JD, Asarnow JR, Markman HJ, … Long B. The science of prevention. A conceptual framework and some directions for a national research program. American Psychologist. 1993;48(10):1013–1022. doi: 10.1037//0003-066x.48.10.1013. [DOI] [PubMed] [Google Scholar]
  14. Cole SR, Hernan MA. Constructing inverse probability weights for marginal structural models. Amercian Journal of Epidemiology. 2008;168(6):656–664. doi: 10.1093/aje/kwn164. [DOI] [PMC free article] [PubMed] [Google Scholar]
  15. Eng E, Parker E. Community competence and health: Definitional, conceptual, and measurement issues. Paper presented at the Annual Meeting of the American Public Health Association, Health Promotion and Health Education Section; New York, NY. 1990. [Google Scholar]
  16. Ennett ST, Ringwalt CL, Thorne J, Rohrbach LA, Vincus A, Simons-Rudolph A, Jones S. A comparison of current practice in school-based substance use prevention programs with meta-analysis findings. Prevention Science. 2003;4(1):1–14. doi: 10.1023/a:1021777109369. [DOI] [PubMed] [Google Scholar]
  17. Fagan AA, Hanson K, Briney JS, David Hawkins J. Sustaining the utilization and high quality implementation of tested and effective prevention programs using the communities that care prevention system. American Journal of Community Psychology. 2012;49(3–4):365–377. doi: 10.1007/s10464-011-9463-9. [DOI] [PMC free article] [PubMed] [Google Scholar]
  18. Fagan AA, Hanson K, Hawkins JD, Arthur MW. Bridging science to practice: Achieving prevention program implementation fidelity in the community youth development study. American Journal of Community Psychology. 2008;41(3–4):235–249. doi: 10.1007/s10464-008-9176-x. [DOI] [PubMed] [Google Scholar]
  19. Fagan AA, Hanson K, Hawkins JD, Arthur MW. Translational research in action: Implementation of the Communities That Care prevention system in 12 communities. Journal of Community Psychology. 2009;37(7):809–829. doi: 10.1002/jcop.20332. [DOI] [PMC free article] [PubMed] [Google Scholar]
  20. Fagan AA, Hawkins JD, Catalano RF. Engaging communities to prevent underage drinking. Alcohol Research & Health. 2011;34(2):167–174. [PMC free article] [PubMed] [Google Scholar]
  21. Foster-Fishman PG, Nowell B, Yang H. Putting the system back into systems change: A framework for understanding and changing organizational and community systems. American Journal of Community Psychology. 2007;39(3–4):197–215. doi: 10.1007/s10464-007-9109-0. [DOI] [PubMed] [Google Scholar]
  22. Gasper J, DeLuca S, Estacion A. Coming and going: Explaining the effects of residential and school mobility on adolescent delinquency. Social Science Research. 2010;39(3):459–476. doi: 10.1016/j.ssresearch.2009.08.009. [DOI] [Google Scholar]
  23. Gloppen KM, Brown EC, Wagenaar BH, Hawkins JD, Rhew IC, Oesterle S. Sustaining adoption of science-based prevention through Communities That Care. Journal of Community Psychology. 2016;44(1):78–89. doi: 10.1002/jcop.21743. [DOI] [PMC free article] [PubMed] [Google Scholar]
  24. Graham JW. Missing data analysis: Making it work in the real world. Annual Review of Psychology. 2009;60:549–576. doi: 10.1146/annurev.psych.58.110405.085530. [DOI] [PubMed] [Google Scholar]
  25. Graham JW, Olchowski AE, Gilreath TD. How many imputations are really needed? Some practical clarifications of multiple imputation theory. Prevention Science. 2007;8(3):206–213. doi: 10.1007/s11121-007-0070-9. [DOI] [PubMed] [Google Scholar]
  26. Grant BF, Dawson DA. Age at onset of alcohol use and its association with DSM-IV alcohol abuse and dependence: Results from the National Longitudinal Alcohol Epidemiologic Survey. Journal of Substance Abuse. 1997;9:103–110. doi: 10.1016/s0899-3289(97)90009-2. [DOI] [PubMed] [Google Scholar]
  27. Hawkins JD, Catalano RF, Arthur MW, Egan E, Brown EC, Abbott RD, Murray DM. Testing Communities That Care: The rationale, design and behavioral baseline equivalence of the Community Youth Development Study. Prevention Science. 2008;9(3):178–190. doi: 10.1007/s11121-008-0092-y. [DOI] [PMC free article] [PubMed] [Google Scholar]
  28. Hawkins JD, Graham JW, Maguin E, Abbott RD, Hill KG, Catalano RF. Exploring the effects of age of alcohol use initiation and psychosocial risk factors on subsequent alcohol misuse. Journal of Studies on Alcohol. 1997;58:280–290. doi: 10.15288/jsa.1997.58.280. [DOI] [PMC free article] [PubMed] [Google Scholar]
  29. Hawkins JD, Oesterle S, Brown EC, Abbott RD, Catalano RF. Youth problem behaviors 8 years after implementing the Communities That Care prevention system: A community-randomized trial. JAMA Pediatrics. 2013;168(2):122–129. doi: 10.1001/jamapediatrics.2013.4009. [DOI] [PMC free article] [PubMed] [Google Scholar]
  30. Hawkins JD, Oesterle S, Brown EC, Arthur MW, Abbott RD, Fagan AA, Catalano RF. Results of a type 2 translational research trial to prevent adolescent drug use and delinquency: A test of Communities That Care. Archives of Pediatric and Adolescent Medicine. 2009;163(9):789–798. doi: 10.1001/archpediatrics.2009.141. [DOI] [PMC free article] [PubMed] [Google Scholar]
  31. Hawkins JD, Oesterle S, Brown EC, Monahan KC, Abbott RD, Arthur MW, Catalano RF. Sustained decreases in risk exposure and youth problem behaviors after installation of the Communities That Care prevention system in a randomized trial. Archives of Pediatric & Adolescent Medicine. 2012;166(2):141–148. doi: 10.1001/archpediatrics.2011.183. [DOI] [PMC free article] [PubMed] [Google Scholar]
  32. Hernán MA, Hernández-Díaz S. Beyond the intention to treat in comparative effectiveness research. Clinical Trials (London, England) 2012;9(1):48–55. doi: 10.1177/1740774511420743. http://doi.org/10.1177/1740774511420743. [DOI] [PMC free article] [PubMed] [Google Scholar]
  33. Jo B, Asparouhov T, Muthen BO, Ialongo NS, Brown CH. Cluster randomized trials with treatment noncompliance. Psychological Methods. 2008;13(1):1–18. doi: 10.1037/1082-989X.13.1.1. [DOI] [PMC free article] [PubMed] [Google Scholar]
  34. Johnston LD, O’Malley PM, Miech RA, Bachman JG, Schulenberg JE. Monitoring the Fture national survey results on drug use, 1975–2015: Overview, key findings on adolescent drug use. Ann Arbor: Institute for Social Research, The University of Michigan; 2016. [Google Scholar]
  35. Kumpfer KL, Alvarado R. Family-strengthening approaches for the prevention of youth problem behaviors. American Psychologist. 2003;58(6–7):457. doi: 10.1037/0003-066X.58.6-7.457. [DOI] [PubMed] [Google Scholar]
  36. MacCallum RC, Zhang S, Preacher KJ, Rucker DD. On the practice of dichotomization of quantitative variables. Psychological Methods. 2002;7(1):19–40. doi: 10.1037/1082-989x.7.1.19. [DOI] [PubMed] [Google Scholar]
  37. Mihalic S, Fagan AA, Irwin K, Ballard D, Elliott D. Blueprints for Violence Prevention. Boulder, CO: National Youth Violence Prevention Resource Center, University of Colorado; 2004. [Google Scholar]
  38. Murray DM. Design and analysis of group-randomized trials. New York, NY: Oxford University Press; 1998. [Google Scholar]
  39. Mzarek PJ, Haggerty RJ, editors. Reducing risks for mental disorders: Frontiers for prevention intervention research. Washington, DC: National Academy Press; 1994. [PubMed] [Google Scholar]
  40. O’Connell ME, Boat T, Warner KE. Preventing Mental, Emotional, and Behavioral Disorders Among Young People. Washington, D. C: National Academies Press; 2009. [PubMed] [Google Scholar]
  41. Rhew IC, Brown EC, Hawkins JD, Briney JS. Sustained effects of the Communities That Care system on prevention service system transformation. American Journal of Public Health. 2013;103(3):529–535. doi: 10.2105/AJPH.2011.300567. [DOI] [PMC free article] [PubMed] [Google Scholar]
  42. Rhew IC, Hawkins JD, Murray DM, Fagan AA, Oesterle S, Abbott RD, Catalano RF. Evaluation of community-level effects of Communities That Care on adolescent drug use and delinquency using a repeated cross-sectional design. Prevention Science. 2016;17(2):177–187. doi: 10.1007/s11121-015-0613-4. [DOI] [PMC free article] [PubMed] [Google Scholar]
  43. Robins JM, Hernan MA, Brumback B. Marginal structural models and causal inference in epidemiology. Epidemiology. 2000;11(5):550–560. doi: 10.1097/00001648-200009000-00011. [DOI] [PubMed] [Google Scholar]
  44. Rosenbaum PR, Rubin DB. The central role of the propensity score in observational studies for causal effects. Biometrika. 1983;70:41–55. [Google Scholar]
  45. Rubin R. Multiple imputation for nonresponse in surveys. New York, NY: John Wiley & Sons; 1987. [Google Scholar]
  46. Schuler MS, Chu W, Coffman D. Propensity score weighting for a continuous exposure with multilevel data. Health Services Outcomes Research & Methodology. 2016;16(4):271–292. doi: 10.1007/s10742-016-0157-5. [DOI] [PMC free article] [PubMed] [Google Scholar]
  47. Stuart EA, Perry DF, Le HN, Ialongo NS. Estimating intervention effects of prevention programs: Accounting for noncompliance. Prevention Science. 2008;9(4):288–298. doi: 10.1007/s11121-008-0104-y. [DOI] [PMC free article] [PubMed] [Google Scholar]
  48. Sussman JB, Hayward RA. An IV for the RCT: Using instrumental variables to adjust for treatment contamination in randomised controlled trials. British Medical Journal. 2010;340:c2073. doi: 10.1136/bmj.c2073. [DOI] [PMC free article] [PubMed] [Google Scholar]
  49. Toh S, Hernán MA. Causal inference from longitudinal studies with baseline randomization. The International Journal of Biostatistics. 2008;4(1) doi: 10.2202/1557-4679.1117. [DOI] [PMC free article] [PubMed] [Google Scholar]
  50. Toh S, Hernández-Díaz S, Logan R, Rossouw JE, Hernán MA. Coronary heart disease in postmenopausal recipients of estrogen plus progestin therapy: Does the increased risk ever disappear? A randomized trial. Annals of Internal Medicine. 2010;152(4):211–217. doi: 10.7326/0003-4819-152-4-201002160-00005. [DOI] [PMC free article] [PubMed] [Google Scholar]
  51. Wagenaar AC, Murray DM, Toomey TL. Communities mobilizing for change on alcohol (CMCA): Effects of a randomized trial on arrests and traffic crashes. Addiction. 2000;95(2):209–217. doi: 10.1046/j.1360-0443.2000.9522097.x. [DOI] [PubMed] [Google Scholar]

RESOURCES