Evaluation of Community-level Effects of Communities That Care on Adolescent Drug Use and Delinquency Using a Repeated Cross-sectional Design

Abstract

The Communities That Care (CTC) prevention system has shown effects on reducing incidence and prevalence of problem behaviors among a panel of youth followed from 5th through 12th grade. The present report examines whether similar intervention effects could be observed using a repeated cross-sectional design in the same study. Data were from a community-randomized trial of 24 U.S. towns. Cross-sectional samples of 6th, 8th, and 10th graders were surveyed at four waves. Two-stage ANCOVA analyses estimated differences between CTC and control communities in community-level prevalence of problem behaviors for each grade, adjusting for baseline prevalence. No statistically significant reductions in prevalence of problem behaviors were observed at any grade in CTC compared to control communities. Secondary analyses examined intervention effects within a “pseudo cohort” where cross-sectional data were used from 6th graders at baseline and 10th graders four years later. When examining effects within the pseudo cohort, CTC compared to control communities showed a significantly slower increase from 6th to 10th grade in lifetime smokeless tobacco use, but not for other outcomes. Exploratory analyses showed significantly slower increases in lifetime problem behaviors within the pseudo cohort for CTC communities with high, but not low, prevention program saturation compared to control communities. Although CTC demonstrated effects in a longitudinal panel from the same community-randomized trial, we did not find similar effects on problem behaviors using a repeated cross-sectional design. These differences may be due to a reduced ability to detect effects because of potential cohort effects, accretion of those who were not exposed, and attrition of those who were exposed to CTC programming in the repeated cross-sectional sample.

Communities That Care (CTC) is a coalition-based prevention system that mobilizes community leaders and stakeholders to adopt a science-based approach to prevention that involves the local collection of epidemiologic data on risk and protective factors experienced by youth to inform selection and implementation of tested and effective programs to reduce the initiation and prevalence of problem behaviors (Hawkins & Catalano, 2002). A community-randomized trial, the Community Youth Development Study (CYDS) (Hawkins et al., 2008), has examined the efficacy of CTC on youth outcomes and prevention system characteristics. Multiple sources of data and designs were utilized as part of this community-randomized trial. One key design was a longitudinal panel of youth from communities randomized to CTC or control conditions. In this panel, youth were assessed repeatedly starting in 5^th grade in 2004, prior to implementation of CTC. A number of positive effects of CTC have been found using this longitudinal panel design. For example, by end of eighth grade, statistically significant omnibus tests for overall effects of CTC were observed for incidence of substance use initiation (p = .03) and prevalence of substance use behaviors (p = .03). Among the specific effects, control compared to CTC community students showed an elevated risk for initiation of cigarette use (Odds Ratio [OR] = 1.79), alcohol use (OR = 1.60), and delinquent behavior (OR = 1.41); reported higher likelihood of past-month alcohol (OR = 1.25) and past two week binge drinking (OR = 1.40); and committed more past-year delinquent acts (Count Ratio [CR] = 1.34) (Hawkins et al., 2009). Sustained effects of CTC were also observed in this longitudinal panel in 10th and 12th grades (1 and 3 years after study support for CTC implementation ended) (Hawkins, Oesterle, Brown, Abbott, & Catalano, 2013; Hawkins et al., 2012).

These longitudinal panel findings accounted for nesting of observations within individuals within communities, allowing inference about effects on individuals over time. This present study examines whether intervention effects could be observed using a repeated cross-sectional design. Rather than test effects on the same individuals over time, repeated cross-sectional designs test intervention effects on changes in outcomes at a specific grade level over time (Murray, 1998). Repeated cross-sectional designs present different opportunities and challenges than longitudinal panel studies. For example, given good participation rates, repeated cross-sectional samples are representative of the community at each study wave, reflecting the net effect of in- and out-migration patterns, whereas a longitudinal panel usually does not account for in-migration (Murray, 1998). However, because those in repeated cross-sectional samples who enter the community later will receive less exposure to intervention, this could lead to attenuated intervention effect estimates. Further, cohort effects (i.e., important differences in the composition of individuals participating in the cross-sectional samples over time) could create a form of bias that also obscures estimation of effects (Cook & Campbell, 1979).

Studies that include longitudinal panel and repeated cross-sectional designs (e.g., Sorensen et al., 2002) allow examination of both grade-level effects in the repeated cross-sectional data and individual-level effects in the longitudinal panel. Where those effects differ, alternate analyses of the repeated cross-sectional data that attempt to mimic a longitudinal panel may minimize selection bias and provide a useful comparison to the longitudinal panel data.

This study examines effects of CTC on prevalence of youth problem behaviors using a repeated cross-sectional design. Only one previous study has utilized the repeated cross-sectional data from the CYDS. That study using multilevel mixture models found that 10^th graders from CTC compared to control communities had a lower likelihood of being in a latent subgroup of alcohol users vs. being in a subgroup of abstainers, but there was no significant difference in likelihood of experimenting with substances nor engaging in both substance use and delinquency (Van Horn, Fagan, Hawkins, & Oesterle, 2014). This present study uses analytic methods originally proposed by Murray et al. (Murray, Van Horn, Hawkins, & Arthur, 2006) to analyze the repeated cross-sectional CYDS data to compare prevalence of 6th-, 8th-, and 10th-grade student outcomes in follow-up years in CTC and control communities, adjusting for baseline prevalence at corresponding grades. As a secondary aim, repeated cross-sectional data are used to assess effects of CTC on changes in prevalence within a “pseudo cohort” (i.e., 6th graders in 2004 who are 10th graders in 2008). We also explore whether effects within this pseudo cohort differ depending on degree of saturation of prevention programs (i.e., number of participants served) implemented through the CTC system.

Materials and Methods

Community Selection

Data for this study were from the CYDS, a community-randomized controlled trial of 24 small towns in Colorado, Illinois, Kansas, Maine, Oregon, Utah, and Washington. The towns were selected from a larger sample of 41 communities participating in an observational study of the diffusion of science-based prevention strategies (the Diffusion Project) (Arthur, Glaser, & Hawkins, 2005). Data from surveys of community key leaders indicated that 26 of these communities would be eligible for participation in the current study, as they did not show evidence of advancing in the use of science-based prevention; that is, they did not routinely select and use tested and effective prevention interventions to address targeted prioritized risk factors (Arthur, Ayers, Graham, & Hawkins, 2003; Hawkins et al., 2008). Twenty-four of these towns agreed to participate and were matched within each state according to population size, racial/ethnic composition, crime rate, and socioeconomic indicators. In 2003, one town from each pair was randomly assigned to the CTC condition and the other to the control condition.

CTC Implementation

For those communities assigned to the CTC condition, CTC training and implementation was initiated between April and August of 2003. Certified CTC trainers administered six training sessions in CTC over the course of 6 to 12 months. During this time, community leaders were oriented to the CTC system and identified or created a coalition of diverse stakeholders within the community to implement CTC. The coalition members were trained in using epidemiologic data to prioritize risk factors to be targeted for prevention activities, to select appropriate tested and effective intervention programs, to implement programs with fidelity, and to monitor implementation and outcomes of the installed programs. CTC communities were asked to focus their prevention programming for youths in Grades 5 to 9 and their families and schools to correspond to the ages of students in the longitudinal panel. In addition to training, CTC staff provided technical assistance through weekly phone calls, e-mails, and annual site visits. CTC communities also received study funding for a local community coordinator and prevention programs, which were implemented beginning in the 2004 - 2005 school year. Funding and proactive technical assistance were provided to CTC communities until the spring of 2008. During this time, CTC communities were required to select and implement programs from a menu of programs that showed strong evidence of efficacy in well-controlled trials in preventing alcohol, tobacco, or other drug use or delinquent behavior among adolescents (Hawkins & Catalano, 2002). Control communities did not receive any training in or funding for CTC activities from the CYDS project during the course of the study.

Student Sample

As part of the Diffusion Project, a cross-sectional survey was administered in public middle and high schools to 6th, 8th, and 10th graders in all participating communities biennially starting in 1998. Although it was originally proposed that the pre-CTC baseline prevalence would be established pooling data from 1998, 2000, and 2002 (Murray et al., 2006); we used data only from 2000 and 2002 to establish the baseline prevalence of outcomes for the current analyses. Data from 1998 were not used because they were further removed temporally from the initiation of the CYDS. The biennial repeated cross-sectional surveys continued during the CYDS (in 2004, 2006, and 2008). For primary analyses, we used data from the 2006 and 2008 waves for assessment of post-implementation outcomes because prevention programs were not fully implemented in the intervention communities in spring 2004 when surveys were administered. Cross-sectional sample sizes over survey years ranged from 4,647 to 5,077 for 6th graders (79% to 87% response rate), 4,491 to 4,984 for 8th graders (72% to 84% response rate), and 3,854 to 4,726 for 10th graders (60% to 78% response rate).

Surveys were administered in schools during a classroom period. Teachers were given specific instructions to ensure the anonymity of student responses and were required to sign an agreement to adhere to these instructions. Passive consent was obtained from parents and informed assent was obtained from students. The study protocol was reviewed and approved by the University of Washington's Human Subjects Review Committee.

Measures

Measures of drug use, delinquency, risk and protective factors, and demographic characteristics were assessed using the Communities That Care Youth Survey (CTCYS). The CTCYS is an anonymous paper and pencil survey that is designed to be completed within a 50-minute classroom period and is appropriate for youth aged 11 to 18 years (Arthur et al., 2007).

Substance use

Participants were asked to report their use of the following drugs: alcohol, cigarettes, smokeless tobacco, and marijuana. Students indicated frequency of use (e.g., 0 occasions, 1 - 2 occasions, 3 - 5 occasions, etc.) for each substance during their lifetime and in the past 30 days. For analyses, each substance use variable was dichotomized to indicate any (1 - 2 occasions or more) versus no use (0 occasions). A separate item asked students about frequency of binge drinking (5 or more alcoholic drinks in a row) during the past 2 weeks. This item was also re-categorized into a dichotomous variable (0 occasions vs. 1 or more occasions). We used the dichotomous version of these variables in analyses instead of the original ordinal metric because of the public health importance of reducing risk of initiation during adolescence (Substance Abuse and Mental Health Services Administration--U.S. Department of Health and Human Services, 2007). Further, these dichotomous forms of the measures are used for epidemiologic monitoring by communities using CTC.

Delinquency

In 2000 and 2002, students were asked to report how often in the past year they sold illegal drugs, stole or tried to steal a motor vehicle, attacked someone, brought a handgun to school, or were arrested. In the post-baseline surveys, three additional items were included (stole something worth $5, purposely damaged or destroyed someone else's property, and shoplifted). Each of the delinquent behavior items was recoded into never or at least once. Using these dichotomous variables, two summary delinquency measures were created. The first is a delinquent behavior count reflecting the number of delinquent behaviors committed in the past year. The maximum value was 5 in pre-intervention years and 8 in post-intervention years. Second, a dichotomous variable was created to indicate whether any delinquent act was committed in the past year (1) or not (0).

Targeted risk factors

Thirty-two risk and protective factors across community (e.g., laws and norms favorable toward drug and alcohol use), school (e.g., low commitment to school), family (e.g., attachment to parents), peer (e.g., friends involved in problem behaviors), and individual (e.g., prosocial involvement) domains were assessed in the CTCYS. Each of the risk factors was assessed as a scale based on a composite of between two and eight items. The scales displayed strong internal reliability and validity and measurement invariance across racial/ethnic groups and gender (Arthur, Hawkins, Pollard, Catalano, & Baglioni, 2002; Glaser, Van Horn, Arthur, Hawkins, & Catalano, 2005). As part of the CTC implementation process, coalition members in CTC communities collectively prioritized between two and five risk factors that were elevated in their community according to pre-intervention data from the biennial cross-sectional surveys. An average targeted risk factor score was calculated by standardizing the community-specific targeted risk factors as a z-score for each grade separately across the multiple waves from 2000 to 2008 and then averaging them. Because control communities did not prioritize and target risk factors, the set of targeted risk factors for each control community's matched CTC community was used.

Missing Data

The percentage of missing data across outcome variables used in analyses was 8.2% for 6th graders, 5.1% for 8th graders, and 4.3% for 10th graders. To account for missingness, multiple imputation was used (Graham, 2009). Imputation models included all covariates and outcomes used in analyses as well as other individual and community characteristics and dummy-coded community indicators to account for the clustering of students in communities. Data were imputed separately for each grade and year combination as well as separately for CTC and control communities. We created 40 imputed datasets using ‘proc impute’ in SAS (SAS Institute, Cary, NC). Statistical models were performed across each of the 40 imputed datasets and summary parameter estimates and their associated standard errors were calculated to account for the uncertainty of imputed values (Rubin, 1987).

Data Analytic Plan

To examine the effects of CTC on youth outcomes, we utilized a two-stage ANCOVA approach (Murray, 1998). Based on power calculations conducted earlier, the two-stage ANCOVA approach was expected to have greater statistical power compared to a random coefficients time-by-condition modeling approach (Murray et al., 2006). At the first stage, for each grade separately, we estimated the adjusted prevalence of each problem behavior (or mean for the antisocial behavior count and level of targeted risk factors) for a community in a given year. We performed a linear regression model where the student-level outcome (e.g., past-30-day alcohol use) was regressed on dummy codes for community, dummy codes for survey year, community-by-year interactions, age, sex, race, religious service attendance, and highest level of parent's education. Intervention condition was not included in this initial model. After determining the model-predicted community prevalence estimate or mean for each year, the 2000 and 2002 data were averaged to create a single pooled baseline level of the outcome for each community. Similarly, the estimates for the 2006 and 2008 years were averaged for a single pooled estimate of the post-intervention prevalence for each community.

In the second stage, the unit of analysis was the community (i.e., n = 24). A mixed effects model was estimated where a random intercept was specified for matched community pair in order to account for clustering of communities within pair. The adjusted post-intervention pooled 2006 - 2008 community prevalence or mean estimated from the first stage was regressed on study condition (1: CTC, 0: Control), the pooled 2000 - 2002 baseline community level of the outcome for the same grade, percent of students in the community who received free or reduced-fee lunch, and number of students in the community. Because of variation in the sample size across communities, analyses were weighted according to the number of participants in the community. Degrees of freedom for the intervention effect were equal to the number of matched pairs (12) minus the number of covariates (3) minus 1 (i.e., df = 8). Using the example of 10th-grade outcomes, given adjustment for baseline prevalence of the outcome at the corresponding grade, the coefficient for study condition can be interpreted as the difference between CTC and control communities in the adjusted change in 10th-grade community-level prevalence or mean from baseline to post-intervention. As indicated by Murray et al. (2006), omnibus test statistics were conducted to assess whether there were overall CTC effects across all youth outcomes (Feng & Thompson, 2002). For the omnibus test statistic, individual t-values for the CTC parameters for each outcome were averaged and then adjusted to account for the community-level correlation of outcomes, number of communities, and number of outcomes. Within each grade, separate omnibus tests were calculated for past-30-day and lifetime outcomes.

As secondary analyses, we also examined whether CTC had effects on changes in prevalence over time within a specific pseudo cohort followed from 6th grade in 2004 to 10th grade in 2008 using the repeated cross-sectional data. We use the “pseudo cohort” term because we assume that a majority of students who participated in the cross-sectional survey when they were in 6th grade in 2004 were among the 10th-grade respondents in 2008, but the cross-sectional surveys were conducted anonymously and individual students could not be linked across time. Further, the 10th-grade respondents included some students who were not in the CYDS community in their 6th-grade year (accretion) and did not include some students who moved away from the community since 6th grade (attrition). We selected this particular grade pseudo cohort because it was not exposed to any prevention programs implemented as part of CTC prior to the 2004 survey. This pseudo cohort was also most interesting for the purposes of this study because it was expected to have received the most exposure to prevention programming during the active intervention years, 2005 - 2008. Furthermore, this 2004 sixth-grade cohort provided a good comparison to the cohort followed in the longitudinal panel, which is only a year younger, as they were in Grade 5 in 2004.

Within-pseudo cohort analyses employed a similar two-stage ANCOVA model for comparisons of CTC and control communities. In the first stage, community-specific prevalence estimates or means of outcomes were obtained for 6th graders in 2004 and 10th graders in 2008, adjusted for individual-level covariates. Unlike the analyses above, age was not included as a covariate in this Stage 1 regression model because 6th- and 10th-grade students differed appreciably in age, and adjustment would lead to predicted prevalence estimates calculated at the average age, which would be inappropriate. In the second stage, we again used a mixed effects model with a random intercept for matched community pair where the 10th-grade community prevalence or mean of the outcome was regressed on study condition, 6th-grade community prevalence or mean of the outcome in 2004, percent free or reduced-fee lunch, and student population size. Because we adjusted for 6th grade in 2004 as the baseline, the coefficient for CTC represents the adjusted difference between CTC and control communities in the change in community-level prevalence or mean from 6th grade in 2004 to 10th grade in 2008.

As post hoc exploratory analyses, we examined whether there were differential effects of CTC on within-pseudo cohort changes in outcomes from 6th grade in 2004 to 10th grade in 2008 according to community-level intervention program “saturation” or reach. Program saturation reflects the proportion of youth in this particular grade cohort reached with preventive interventions in each community, considering all the prevention programs implemented in the community during the intervention period using study funding. As previously stated, to receive study funding, CTC communities were expected to implement tested, effective programs with students in Grades 5 - 9. However, they could designate the particular grade or age groups of youth to target for services within this range. Findings from the study process evaluation indicated that during the 4 years in which services were funded, the age and number of participants served by programs varied by year and across communities (Fagan, Hanson, Hawkins, & Arthur, 2009). The result was that some communities served a much larger percentage of this specific pseudo cohort than other communities, resulting in significant variation across the 12 CTC communities in the amount of programming that students in this cohort would have received. For example, some communities implemented school-based prevention curricula in all sixth-grade classrooms each year of the study, meaning that most of the sixth-grade cohort in 2004 would have received that service, but other communities delivered classroom curricula to other grades or implemented family or afterschool programs which targeted students in other cohorts (or reached a much smaller number of youth in the sixth-grade cohort). The post hoc analyses assessed the degree to which program saturation was associated with stronger intervention effects. For ease of interpretation, we evenly divided the CTC communities into “high saturation” vs. “low saturation” where the six CTC communities offering the largest number of programs to the greatest number of participants were defined as having high saturation and the other six CTC communities were defined as low in program saturation. Dummy variables were created to define low saturation or high saturation, with control communities being the reference group. As before, two-stage ANCOVA models were conducted where linear regression models were used in the first stage to estimate community-level means according to individual-level covariates. In the second stage, 10th-grade 2008 prevalence or means of outcomes were regressed on dummy indicators for low saturation and high saturation, 6th-grade in 2004 level of the outcome, and community covariates. Again, community pair was included as a random effect.

To aid in clarification of the analyses performed, Table 1 summarizes the timing of the outcome and baseline adjustment variables and the interpretation of the coefficient for study condition. All statistical analyses were conducted in SAS version 9.3 (SAS Institute, Cary, NC).

Table 1.

Summary of the Timing of Outcome and Baseline Adjustment for the Three Sets of Two-stage ANCOVA Analyses

Analysis	Grade(s) and year(s) of outcomes	Grade(s) and year(s) of baseline level of outcome covariates	Interpretation of coefficient(s) for study condition
Primary	a. 6th grade, pooled 2006-2008	a. 6th grade, pooled 2000-2002	a. Difference between CTC and control in change in 6th-grade prevalence at baseline to 6th-grade prevalence at follow-up
	b. 8th grade, pooled 2006-2008	b. 8th grade, pooled 2000-2002
	c. 10th grade, pooled 2006-2008	c. 10th grade, pooled 2000-2002
			b. Similar to above but for 8th grade
			c. Similar to above but for 10th grade
Secondary	10th grade, 2008	6th grade, 2004	Difference between CTC and control communities in change in prevalence from 6th grade in 2004 to 10th grade in 2008
Post-hoc	10th grade, 2008	6th grade, 2004	Difference between CTC high saturation and control or between CTC low saturation and control in change in prevalence from 6th grade in 2004 to 10th grade in 2008

Results

For most student characteristics at post-intervention, CTC and control youth were similar; however, youth from CTC compared to those from control communities were more likely to report White race and less likely to report Hispanic ethnicity (Table 2).

Table 2.

Demographic Characteristics of Students and Communities in the Pooled 2006 and 2008 Follow-up Waves by Study Condition and Grade

Characteristic	6th grade		8th grade		10th grade
	CTC	Control	CTC	Control	CTC	Control
	N = 5391	N = 4736	N = 5317	N = 4284	N = 4617	N = 4530
	Mean (SD) or %	Mean (SD) or %	Mean (SD) or %	Mean (SD) or %	Mean (SD) or %	Mean (SD) or %
Male sex	49.7	49.9	49.7	49.5	47.7	48.2
Age	11.6 (.6)	11.6 (.6)	13.6 (.6)	13.6 (.6)	15.6 (.6)	15.6 (.6)
White race	72.2	64.1	72.2	64.4	77.2	70.4
Hispanic ethnicity	15.3	26.6	16.5	25.9	14.3	21.5
Highest level of parent education high school or less	23.9	28.6	27.0	31.6	26.4	30.9
Attends religious services weekly	36.6	32.6	36.1	30.1	33.4	30.4

A supplemental table (available online) shows prevalence estimates of problem behaviors at the pooled 2006 and 2008 follow-up waves in CTC and control study conditions for each grade. Table 3 shows second-stage results of the two-stage ANCOVA models for the differences between CTC and control communities in the prevalence or mean of outcomes in 6th, 8th, and 10th grades in the follow-up years, adjusting for community covariates, including pooled 2000 and 2002 levels of the outcome in the corresponding grades. Among sixth graders, there was only one statistically significant difference between CTC and control communities in youth outcomes, but it was not in the expected direction. The prevalence of any antisocial behavior was higher in CTC compared to control communities (β = .037; 95% CI: .002, .072). There were no other statistically significant differences in 6th grade and no statistically significant differences between CTC and control conditions for any of the outcomes among 8th or 10th graders. Further, neither the omnibus test for an overall difference across past-30-day substance use outcomes nor for lifetime outcomes were statistically significant for any of the grades.

Table 3.

Adjusted^a Coefficients From Two-stage ANCOVA Models for Differences in Community-level Prevalence or Mean of Outcomes Between CTC and Control Communities by Grade

	6th grade			8th grade			10th grade
Outcome	β	95% CI	p-value	β	95% CI	p-value	β	95% CI	p-value
Past-30-day alcohol	0.008	−.010, .027	0.301	0.015	−.031, .060	0.475	0.014	−.055, .082	0.652
Lifetime alcohol	−0.003	−.054, .048	0.897	0.009	−.045, .062	0.711	0.008	−.057, .072	0.789
Past-2-week binge drinking	0.006	−.005, .017	0.241	0.010	−.022, .043	0.475	−0.001	−.049, .048	0.966
Past-30-day cigarette	0.006	−.004, .017	0.172	0.020	−.017, .080	0.075	−0.007	−.056, .041	0.722
Lifetime cigarette	0.012	−.010, .034	0.236	0.031	−.017, .080	0.166	0.010	−.060, .080	0.741
Past-30-day marijuana use	0.006	−.004, .015	0.194	0.007	−.013, .028	0.427	−0.001	−.045, .042	0.946
Lifetime marijuana use	0.011	−.002, .024	0.088	0.030	−.011, .071	0.126	0.005	−.049, .059	0.823
Past-30-day smokeless tobacco	0.011	−.006, .027	0.169	0.003	−.017, .024	0.694	−0.008	−.035, .019	0.505
Lifetime smokeless tobacco	0.022	−.008, .053	0.124	0.004	−.034, .043	0.796	−0.011	−.050, .028	0.512
Any past-year antisocial behavior	0.037	.002, .072	0.043	0.004	−.050, .058	0.873	−0.014	−.078, .049	0.601
Antisocial behavior index	0.045	−.090, .181	0.449	0.035	−.155, .226	0.669	0.013	−.202, .228	0.887
Targeted risk factors	0.005	−.043, .054	0.794	0.059	−.004, .121	0.061	0.011	−.046, .067	0.669

^aAdjusted for percentage in community receiving free and reduced-fee lunch and number of enrolled students

Results of the within-pseudo cohort analyses (Table 4) showed that 10 of the 12 outcomes in this analysis had a lower prevalence in CTC compared to control communities. However, the global test statistic was not statistically significant for either the past-30-day outcomes (p = .82) or the lifetime outcomes (p = .13). Only the difference in prevalence of lifetime smokeless tobacco use was significantly lower in CTC compared to control communities (β = -.047; 95% CI: -.081, -.012). There were no other statistically significant differences between CTC and control communities.

Table 4.

Coefficients for Difference Between CTC and Control Communities in Community-level Prevalence or Mean in 10th-grade 2008 Adjusted for 6th-grade 2004 Outcomes and Other Covariates

Outcome	Estimate	95% CI	p-value
Past-30-day alcohol	−0.0004	−.072, .071	0.991
Lifetime alcohol	−0.005	−.074, .065	0.873
Past-2-week binge drinking	−0.005	−.051, .042	0.824
Past-30-day cigarette	0.006	−.023, .035	0.631
Lifetime cigarette	−0.015	−.073, .043	0.553
Past-30-day marijuana use	−0.001	−.043, .042	0.972
Lifetime marijuana use	−0.025	−.089, .040	0.394
Past-30-day smokeless tobacco	−0.009	−.032, .014	0.374
Lifetime smokeless tobacco	−0.047	−.081, −.012	0.017
Any past-year antisocial behavior	−0.068	−.223, .088	0.329
Antisocial behavior index	−0.046	−.097, .006	0.074
Targeted risk factor	0.027	−.045, .099	0.407

As post hoc exploratory analyses, we explored possible differences in outcomes associated with the level of prevention program saturation within communities in order to better understand differences between findings from the longitudinal panel and repeated cross-sectional designs. Figure 1 shows the regression coefficients and their 95% confidence intervals from the second stage of the two-stage ANCOVA models for the difference in community-level prevalence estimates of outcomes in high- and low-program-exposure CTC communities compared to control communities. The omnibus test comparing overall differences across outcomes in high-exposure CTC to control communities was statistically significant for lifetime outcomes (p = .017), indicating overall reductions in lifetime problem behaviors in high-program-exposure CTC communities. The omnibus test was not significant for past-30-day outcomes (p = .278). With respect to specific outcomes, the prevalence of lifetime marijuana use in 10th grade was significantly lower in high-program-exposure CTC communities compared to control communities (β = -.107; 95% CI: -.213, -.001; p = .050). The prevalence of past-30-day marijuana use also was lower in high-exposure CTC compared to control communities, but not at the .05 significance level (β = -.054; 95% CI: -.117, .010; p = .082). Lifetime cigarette use also showed a trend in the expected direction, but was not statistically significant (β = -.070; 95% CI: -.153, .014; p = .086). It is notable that CTC high-exposure communities tended to have a lower prevalence than control communities for all other outcomes except past-30-day smokeless tobacco use. In contrast, low-exposure CTC communities tended to be more similar to control communities across outcomes, with the exception of lifetime smokeless tobacco, where there was a significantly lower community-level prevalence in low-exposure CTC communities compared to control communities (β = -.052; 95% CI: -.100, -.005; p = .037). The omnibus test for overall differences between low-exposure CTC and control communities was not statistically significant for either lifetime (p = .514) or past-30-day (p = .757) outcomes.

Figure 1.

Coefficients for differences in community-level prevalence or mean of outcomes in high- and low-program exposure CTC communities compared to control communities in 10th-grade 2008 adjusted for 6th-grade 2004 outcomes and other covariates.

Conclusions

In prior studies, CTC was observed to reduce students’ risk for initiation and likelihood of currently engaging in problem behaviors in a panel followed from 5th grade through 12th grade (Hawkins et al., 2013; Hawkins et al., 2009; Hawkins et al., 2012). In this paper, we assessed whether effects on youth outcomes could be detected using repeated cross-sectional samples over time. Given the positive findings observed in the panel, we hypothesized that similar effects would be observed using the repeated cross-sectional data. However, in this study we did not observe beneficial effects of CTC in outcomes assessed in repeated cross-sectional surveys of students in 6th, 8th, or 10th grades.

There are several possible explanations for the contrasting findings between the repeated cross-sectional design and the longitudinal panel. One possibility is that the repeated cross-sectional findings represented a true lack of effect of CTC, and findings showing positive effects in the longitudinal panel may have been biased in some way. However, we believe that this is unlikely. As reported elsewhere, there were no statistically significant differences in level of risk or protective factors or prevalence of substance use or delinquent behaviors between CTC and control community youth in the longitudinal panel at the baseline Grade 5 visit (Brown et al., 2009). This supports the baseline equivalence of study condition groups. Further, beyond the longitudinal panel results, other evidence from the CYDS indicates that CTC was functioning as would be expected. According to the CTC theory of change, implementation of the CTC system is expected to lead to prevention system changes in the community as a whole. These changes include greater adoption of a science-based approach to prevention, increased community support for prevention, increased collaboration around prevention across service sectors, and increased community norms against adolescent substance use. These improvements in prevention system characteristics should lead to increased use and spread of tested and effective prevention programs, reduced levels of risk factors and elevated levels of protective factors, and reduced levels of youth problem behaviors. Prior evidence from the CYDS communities was consistent with the underlying theory of change. For example, findings suggest that CTC communities implemented prevention programs with high fidelity (Fagan, Hanson, Hawkins, & Arthur, 2008) and implemented a greater number of tested and effective programs than control communities, and these tested and effective programs reached more youths in CTC communities during the course of the study (Fagan, Arthur, Hanson, Briney, & Hawkins, 2011; Fagan, Hanson, Briney, & David Hawkins, 2012). Further, key leaders from communities randomized to the CTC condition compared to those from the control communities showed higher levels of adoption of a science-based approach to prevention and other prevention system characteristics during the active implementation phase of the randomized trial (Brown, Hawkins, Arthur, & Briney, 2011; Brown, Hawkins, Arthur, Briney, & Abbott, 2007) as well as 1.5 years after study funding for intervention activities ceased (Rhew, Brown, Hawkins, & Briney, 2013). Reduced levels of targeted risk factors and problem behaviors were observed in the longitudinal panel during the active intervention phase as well as after study funding for intervention activities ceased (Hawkins et al., 2013; 2009; 2012). Finally, mediation analyses showed that effects of CTC on eighth-grade youth outcomes in the longitudinal panel were mediated through increasing communities’ adoption of a science-based approach to prevention (Brown et al., 2013).

Given that bias in the longitudinal panel was unlikely, other factors related to the repeated cross-sectional design may have contributed to the contrasting findings. One explanation may be cohort effects. In the panel design, the same students were followed over time. In the primary analyses for the repeated cross-sectional study, there was a different population of students in the baseline sample compared to the follow-up sample at each grade level. Although measured demographic characteristics were comparable across time, there may have been unmeasured characteristics (e.g., specific socioeconomic indicators, changes to school or community policies that affected a particular grade, unique cultural factors, etc.) that biased the findings. Descriptive analyses of prevalence estimates of problem behaviors for a specific grade over time within communities are revealing (data not shown, but are available from the first author upon request). For a number of communities in this study, prevalence estimates of outcomes for a given grade did not show an obvious pattern within a period of 8 years, but instead increased and decreased sharply from time point to time point. However, when aggregating data to all communities within a given study condition, trends were clearer (e.g., a decrease in use over time) and tended to parallel trends in national data. This would suggest that within the same community the prevalence of problem behaviors can be quite variable over a relatively short duration, and cohort differences could account for some of this variability.

To mitigate possible cohort effects, we used the repeated cross-sectional data to examine a pseudo cohort over time (6th grade to 10th grade), but without linking individuals over time because surveys were anonymous. We observed stronger differences in outcomes between CTC and control communities in those analyses than in the primary analyses. Many of the measured problem behaviors showed a reduced prevalence in 10th grade in CTC compared to control communities, adjusting for 6th-grade prevalence 4 years earlier. However, the global test statistic was not statistically significant and only prevalence of lifetime smokeless tobacco showed a statistically significant difference between conditions, with lower prevalence in CTC compared to control communities. A quasi-experimental study of over 90 school districts in Pennsylvania utilized a repeated cross-sectional design to examine pseudo cohorts over time and also found positive effects of CTC on youth outcomes, although not in a randomized trial (Feinberg, Jones, Greenberg, Osgood, & Bontempo, 2010). That study found that pseudo cohorts of youth in CTC compared to control districts showed significantly slower increases in delinquency, slower growth in risk factors, and slower decreases in protective factors and academic performance.

Another possible explanation may be the reduced power available for the analysis of the repeated cross-sectional design compared to the panel design. Where correlation over time within participants is fairly strong, the panel design has an advantage over the repeated cross-sectional design in analyses that take the longitudinal data into account (Murray, 1998). Further, prior estimations showed that the CYDS was likely powered to detect only strong effects using the repeated cross-sectional design (Murray, Van Horn, Hawkins, & Arthur, 2006). Simulation-based post-hoc power analyses (see supplemental Appendix) using observed levels of covariates and baseline prevalence of outcomes generally confirmed the original estimations presented in Murray et al., (2006), such that similar large effects could be observed with .80 power. However, it is unlikely that low statistical power is a major explanation for the inconsistency between the repeated cross-sectional and longitudinal panel findings given the almost negligible (< 1%) differences in prevalence across outcomes observed between CTC and control communities (Table 3). The potential lack of power may be more relevant for the pseudo-cohort analyses. Our inability to link data from individuals over time in the repeated cross-sectional design, and thus take the longitudinal nature of the data into account, could have contributed to a loss of power to detect effects in the pseudo cohort. In the quasi-experimental study conducted in Pennsylvania, sample size was much larger than that of the current study with data available from over 90 school districts compared to only 24 communities in this study (Feinberg, Jones, Greenberg, Osgood & Bontempo, 2010).

The possibility of differential accretion and attrition between the longitudinal panel and the repeated cross-sectional samples may also explain differences in findings (Murray, 1998). Because the same students were followed over time, the longitudinal panel includes some students who have left the community, but not any new students who moved to the community after CTC implementation. In the repeated cross-sectional sample, the student population assessed at follow-up was composed of those who were currently living in the community and students who had moved to the community since baseline, but not any students who moved away from the community. Thus, the loss of these students who were exposed to the CTC system and then moved away could attenuate effects because they are not in the sample at follow-up. Further, students who moved into CTC communities in the cross-sectional samples may have had limited exposure to the CTC system which would have led to an attenuated effect of CTC in these analyses. Unfortunately, there were no student data on length of time residing in the community in the anonymous cross-sectional surveys to explore this possibility.

Another consideration is the timing and targets of the prevention programs implemented by CTC communities. Most CTC community prevention program efforts targeted middle school youth. However, there was variation in the number of programs implemented and the proportion of youth at specific grades and times reached by these programs. As a result, the main analyses represent a conservative test of the power of CTC to affect changes in antisocial behaviors, as not all the intervention community youth included in these analyses would have received tested and effective programs, which is a main component of the CTC theory of change. Post hoc analyses supported this possibility. Significant effects of CTC were found in the omnibus test in the pseudo cohort when comparing CTC communities with high prevention program saturation to control communities. Low-saturation CTC communities tended to have prevalences of problem behaviors similar to control communities for most outcomes. The suggestion that the effects of CTC may be restricted to students who were more likely to receive interventions is consistent with the CTC emphasis on using tested, effective programs that target local needs in order to have a community-wide impact on youth behaviors. It should be noted again that these analyses were exploratory and may be subject to bias because stratifying on a post-intervention variable, program saturation, breaks the intent-to-treat design and may introduce bias.

Although CTC has shown effects of reducing the initiation and prevalence of problem behaviors in a longitudinal panel of youth, we did not observe effects of CTC when using a repeated cross-sectional design. There may be important research and practical implications of these findings. These results, along with the findings from the longitudinal panel, suggest that studies of outcomes assessed prospectively within individuals using a longitudinal panel design may be more sensitive to effects of CTC and other community-level interventions than studies using repeated cross-sectional samples. When a longitudinal panel design is not feasible and a repeated cross-sectional design is instead utilized, our findings would suggest the importance of examining the data as a pseudo-cohort in order to assess changes over time within a specific grade cohort. Results based on a pseudo cohort using the repeated cross-sectional data were more suggestive of positive effects of CTC perhaps because this design may reduce potential cohort effects and increase the likelihood of detecting a true intervention effect. There may also be implications for communities that implement CTC. As part of the CTC system, communities use repeated cross-sectional data to monitor the prevalence of problem behaviors over time. Because findings from the current study suggest that repeated cross-sectional data of a specific grade may not detect true effects of CTC, it may be important for communities to compare their epidemiologic data to other data such as Monitoring the Future in order to assess whether trends in their community suggest effects of CTC beyond secular trends in adolescent health risking behaviors. Further, given potential cohort effects and the variability of prevalence from one grade cohort to another, it may be important to collect and evaluate the repeated cross-sectional data over a longer duration to monitor community-level progress in problem behavior reduction.