A Meta-Analysis of the Effectiveness of Interactive Middle School Cannabis Prevention Programs

Abstract

This meta-analysis examines the effectiveness of interactive middle school-based drug prevention programs on adolescent cannabis use in North America, as well as program characteristics that could moderate these effects. Interactive programs, compared to more didactic, lecture style programs, involve participants in skill-building activities and focus on interaction among participants. A systematic literature search was conducted for English-language studies from January 1998 to March 2014. Studies included evaluations using random assignment or a quasi-experimental design of interactive school-based substance use prevention programs delivered to adolescents (aged 12–14) in North American middle schools (grades 6–8). Data were extracted using a coding protocol. The outcomes of interest were post-treatment cannabis use, intent to use, and refusal skills compared across intervention and control groups. Effect sizes (Cohen’s d) were calculated from continuous measures, and dichotomous measures were converted to the d index. A total of 30 studies yielding 23 independent samples were included. The random effects pooled effect size for cannabis use (k=21) was small (d̄=−0.07, p<0.01) and favorable for the prevention programs. The pooled effect sizes for intention to use (k=3) and refusal skills (k=3) were not significant. Moderator analyses indicated significant differences in program effectiveness between instructor types, with teachers found to be most effective (d̄ =−0.08, p=0.02). The findings provide further support for the use of interactive school-based programs to prevent cannabis use among middle school students in North America.

Adolescent substance abuse is a critical public health priority in both the United States (US) and Canada. In particular, cannabis use is gaining increasing attention as several US states have passed marijuana legalization laws and there are concerns that prevalence rates of abuse among adolescents may increase (Johnston, O’Malley, Miech, Bachman, & Schulenberg, 2015). According to the 2014 Monitoring the Future survey, the lifetime cannabis use rate for US youth in Grades 8 through 12 was 30.5%, with past-month use continuing to exceed cigarette use among high school seniors, and—for the first time—daily use being higher than cigarette use (Johnston et al., 2015). In Canada, the lifetime cannabis use rate among youth age 15 to 19 years was 25.8%, with the average age of initiation at 15.1 years (Government of Canada, 2015).

Research demonstrates both short and long-term risks associated with cannabis use during the adolescent developmental period. Early cannabis use in adolescence has been linked to advanced drug use and addiction in adulthood (Chen, Storr, & Anthony, 2009; Lopez-Quintero et al., 2011). Moreover, students who use marijuana have been found more likely to have lower grades, less class participation, and poorer attendance rates as compared to non-users (Cox, Zhang, Johnson, & Bender, 2007; Finn, 2012). Marijuana use has also been linked to impairments in cognitive functioning in adolescents through its impacts on working memory performance (Harvey, Sellman, Porter, & Frampton, 2007; Wright, 2015). Some studies also have demonstrated increased risk for developing depression and anxiety later in adulthood after early cannabis use (Patton et al., 2002; Wright, 2015). Therefore, targeting adolescents in middle school may be critical for school prevention efforts that aim to modify expectations, beliefs, and behaviors related to cannabis use (Government of Canada, 2015; Substance Abuse and Mental Health Services Administration, 2014).

Systematic reviews and meta-analyses also support the importance of implementing substance abuse prevention programs during the middle school years (Gottfredson & Wilson, 2003; Norberg, Kezelman, & Lim-Howe, 2013). In particular, the findings of previous meta-analyses have identified that interactive programs, or those programs that involve participants in skill-building and engagement with other participants, are more effective at preventing adolescent substance abuse than didactic programs, or those programs using a lecture-style of program delivery (Cuipers, 2002; Faggiano et al., 2008; Porath-Waller, Beasley, & Beirness, 2010; Tobler et al., 2000). In addition, previous findings have indicated that substance abuse prevention programs delivered by clinicians and peers are also more effective (Faggiano et al., 2008; Gottfredson & Wilson, 2003; Norberg et al., 2013; Porath-Waller et al., 2010). To date, however, only two meta-analyses and two systematic reviews have focused specifically on cannabis use (Norberg et al., 2013; Porath-Waller et al., 2010; Tobler, Lessard, Marshall, Ochshorn, & Roona, 1999; White & Pitts, 1998). None of these cannabis-specific reviews discuss treatment of effects from studies that include more than one treatment condition with a shared control group, which present statistical dependence between effect sizes (Card, 2012; Scammacca, Roberts, & Stuebing, 2014). And, though studies of school-based programs tend to have clustering effects with samples drawn from more than one school, none of the four reviews for cannabis outcomes report adjustments for clustering. Moreover, age categories are mixed in other substance abuse prevention program reviews, making it hard to discern effects of programs delivered specifically during middle school (Norberg et al., 2013; Porath-Waller et al., 2010; Tobler et al., 1999). Only one meta-analysis of school-based substance abuse prevention programs distinguished the specific effects of programs on elementary, middle, and high school students (Gottfredson & Wilson, 2003). Given the limitations of these prior reviews, the current body of rigorous and homogenous studies focused on interactive, skills-based program designs, and the recent cultural shifts and passage of laws related to marijuana use and related prevention efforts, the purpose of this meta-analysis was to evaluate the effectiveness of interactive, school-based North American middle school drug prevention programs on adolescent cannabis use, as well as understand program characteristics that could moderate these effects.

Methods

Methods and findings are reported following guidelines outlined in the —Preferred Reporting Items for Systematic Reviews and Meta-analysis statement (Moher, Liberati, Telzlaff, Altman, & the PRISMA Group, 2009). The protocol for this review is available by request from the authors.

Eligibility Criteria

For inclusion in this meta-analysis, studies had to meet the following criteria: 1) evaluation of a middle school-based substance use prevention program; 2) report measures of cannabis use; 3) use random assignment or a quasi-experimental design; 4) conducted in North America; 5) published in English; 6) conducted during 1998 or later; and 7) present sufficient statistical information to calculate effect sizes indexing the magnitude and direction of effects; 8) the program had to include interactive components, such as skill building and student engagement; and 9) the program had to be delivered to adolescents (aged 12–14) in middle school (grades 6–8). Additional details on inclusion criteria definitions are available online.

Studies were excluded from the meta-analysis for six reasons, including 1) the primary program components were delivered mainly in the home or in a community setting; 2) the authors described program components as —extracurricular activities, such as social groups, athletics, arts, and academic tutoring; 3) the primary program components were delivered in elementary or high school; 4) the only comparison group was another prevention program; 5) the study assessed only program completers rather than using an intent-to-treat strategy; or 6) attrition differed causing the sample groups to be non-equivalent.

Information Sources and Search Strategy

The systematic search strategy followed the recommendations of Card (2012) and Littell and Maynard (2014) to identify relevant published and unpublished studies. The following databases were identified in consultation with a university librarian and searched February 28 through March 6, 2014: Social Work Abstracts (ProQuest); Social Services Abstracts (ProQuest); Sociological Abstracts (ProQuest); PsycINFO (EBSCO); ERIC (EBSCO); Academic Search Complete (EBSCO); PubMed (Medline); Cochrane Library. Eight other databases and websites were queried to identify unpublished, non-peer-reviewed reports: Dissertations and Theses (ProQuest); ERIC (EBSCO); Google Scholar; National Criminal Justice Reference Service Abstracts; National Institute for Health and Clinical Excellence; Office of Juvenile Justice and Delinquency Prevention; RAND Corporation; and Washington State Institute for Public Policy. Query terms were: ((adolescent* OR middle school OR teen*) AND (school*) AND (*drug OR alcohol OR substance) AND (prevention)). Additional studies were identified from citations in prior reviews and eligible studies through April 27, 2015.

Data Collection and Coding

Data were extracted using a coding protocol and form (Card, 2012; Littell & Maynard, 2014). The form included items on study citation, research design, sample characteristics, outcome measures, statistical results, and program characteristics. Program characteristics included dosage (number of sessions); booster sessions delivered in the next semester or grade level (yes/no); delivery setting (during regular school hours, after regular school hours, or both); and instructor type (i.e., teacher, clinician, police officer, and peer with adult). The clinician category included instructors described as persons with clinical, counseling, or substance abuse prevention training who worked as staff or a volunteer with a university or social service organization. The geographical setting (urban, rural, or mixed); percent male (program group); percent non-White (program group); and low SES, indicated as program participant qualified for free or reduced price lunch or from a household with median income below poverty level (yes/no) also were coded. Treatment and control sample sizes were recorded at baseline and at all follow-ups. Statistical information to compute effect sizes included treatment and control sample sizes at baseline and follow-up, number of cluster locations (if clustered design), ICC values, and outcome measure results. To test the risk of bias in individual studies, study characteristics included: research design (RCT, quasi-experimental); fidelity adherence (reporting how implementation followed program design, yes/no); follow-up time (months); control/comparison condition (no programming, treatment as usual with minimal instruction, or alternative health instruction); and studies that lacked clarity on how they addressed non-completers (yes/no, explained below). Some researchers have found that the program developer’s involvement in the evaluation as a researcher significantly increased the effect size (Petrosino & Soydan, 2005; Washington State Institute for Public Policy, 2015). Accordingly, studies were coded as developer-involved when the report named the developer as an author. Information not available from the reports was retrieved from other reports, websites for the program, or requested from authors. During the coding process, the coding team identified five studies that appeared to use an intent-to-treat design, but lacked clear details to confirm this. In these cases, the lead authors were contacted, three of whom confirmed this. For the studies whose authors did not respond, this uncertainty was treated as a variable to explore risk of bias.

A university faculty member and two graduate students coded the studies. First, training involved all researchers coding six studies. Afterwards, two graduate students coded remaining studies, and the lead author reviewed these for accuracy and consistency. The coding team met weekly and resolved discrepancies by group consensus in order to attain full agreement on all coding. The coding team entered the information from each form into an Excel database. The lead researcher reviewed all data entered into the database to ensure accuracy.

Synthesis of Results

Computing effect sizes

Cannabis use outcome measures included initiation (ever used cannabis or used in the past year); recent use (amount of cannabis used or any used in the past month or week); intent to use (likelihood of using during next week or month); and refusal skills (ability to resist offers to use). Program effects were measured by computing the standardized mean difference effect size (d), weighted by the pooled standard deviations of the treatment and comparison groups (Card, 2012; Lipsey & Wilson, 2001). Wherever possible, the sample sizes reported from the outcome statistics were used rather than the initial counts at baseline. Equations reported in Card (2012) and Lipsey and Wilson (2001) were used for computing effect sizes from a variety of continuous measures. The Cox transformation was used with dichotomous outcomes to approximate the standardized mean effect size (Sánchez-Meca, Marín-Martinez, & Chacón-Moscoso, 2003), computed using procedures discussed in Washington State Institute for Public Policy (2015). An effect size was used as reported in a study only if sufficient information was provided to confirm that the method was the same as that used in the present meta-analysis. Effect sizes from studies that reported no significant effects (without any other statistical information) were coded by assigning a value of 0.0 with a one tailed p-value of 0.5, as recommended by Rosenthal (1995). Negative effect sizes were specified by adding a negative sign to denote the treatment group had lower usage and intent to use outcomes than the control group. A positive effect size for refusal skill outcomes denotes treatment groups scoring higher on refusal skills than the control group.

Statistical analyses

Meta-analyses and moderator and risk of bias analyses were performed using Comprehensive Meta-Analysis (CMA) Version 2 (Borenstein, Hedges, Higgins, & Rothstein, 2005). Effect size calculations and descriptive analyses were performed using Microsoft Excel. Effect sizes from subgroups and multiple follow-up points were averaged into single effect sizes for each sample using fixed effects models with inverse variance weights to account for differences in subgroup sizes. Additional details about multiple studies contributing to effect sizes from the same samples are available online. To ensure statistical independence of effect sizes in analyses, the initiation measure was selected when studies reported both initiation and recent use. Individual effect sizes were combined in CMA using random effects models employing inverse variance weights with each effect size (Card, 2012; Lipsey & Wilson, 2001). Random effects models were used to account for possible variation other than sampling error among effect sizes as well as heterogeneity in programs and participants (Card, 2012; Raudenbush, 2009). Homogeneity was tested using the Q statistic and the I-squared (I²) statistic was computed to describe the proportion of variation across studies due to heterogeneity (Higgins & Thompson, 2002).

Treating sample sizes for clustered studies

Cluster or nested designs are commonly used in school-based studies, where individual students were the units of analysis but their assignment to treatment or comparison conditions were by school or district. This clustering has the effect of limiting the standard error or confidence interval in the estimated program effect because of similarities between students within schools or districts. To account for this, an effective sample size was estimated for each clustered sample to avoid inflating the effect size based on clustering (McKenzie, Ryan, & Di Tanna, 2014). Many studies reported an intracluster correlation coefficient (ICC), which was used to estimate the effective sample size. However, some studies did not report an ICC. For these, the average ICC values for each outcome were estimated from a sample of ICC scores that the authors compiled from the reported ICCs in the included studies, as well as those reported in separate analyses (Murray & Hannan, 1990; Schheier, Griffin, Doyle, & Botvin, 2002).

Moderator analysis

Moderator analyses were conducted on the pooled effect size for any cannabis use with the program and study characteristic variables described above. Mixed effects models were used on categorical variables and meta-regression analyses with method of moments models were used on continuous moderators (Borenstein, et al., 2005). The between-group heterogeneity (Q_Between) was estimated to assess the reliability of moderation on effect sizes (Card, 2012). The Tau-squared (τ²) statistic was also computed to show the amount of random effects distribution variance. For categorical variables with more than two categories, separate analyses were conducted on each category to assess the degree of variation between each type. Bonferroni-adjusted alpha criterion was used to control for Type 1 error when interpreting statistical significance of the Q statistic (Holm, 1979). Risk of bias in individual studies was assessed with bi-variate moderator analysis using random effects models on four study characteristics: study design (whether RCT or quasi-experimental); fidelity adherence (yes/no); follow-up length (number of months); comparison group condition; and where intent-to-treat design could not be confirmed.

Results

The search of eight electronic databases yielded 8,303 records for the broader review of substance use prevention programs (Figure 1). A total of 7,056 records were left for screening after removing duplicates and adding 56 records identified from other sources. After review of titles and abstracts, 6,412 records were excluded. Another 424 records were excluded after screening full-text articles, and two articles could not be obtained. The remaining 218 studies were coded. After coding, 47 records were identified for study inclusion, of which 30 studies yielding 23 independent samples had cannabis outcomes and were included in the present meta-analysis. Of the 188 excluded studies, four were not in English or from North America, 66 were not impact evaluations, 57 did not include predominantly middle-school youth, 29 had no cannabis or other drug or alcohol use outcomes, 20 had studies conducted before 1998, and 12 had methodological problems (such as differential attrition or insufficient statistical data).

Figure 1.

Flow Diagram for Studies Included in the Meta-Analysis

Study Characteristics

Study design and participants

Table 1 presents descriptive information on all studies included in the meta-analysis. All but one study was from the US, with the other from Canada (DeWit et al., 2000). One study (D’Amico & Edelen, 2007) did not use random assignment of participants. The authors used matched samples based on demographic variables and baseline substance use measures to generate a weighted sample of students from a control school that was comparable with the sample of participants in the program school (D’Amico & Edelen, 2007). The follow-up times used in studies ranged from immediate post-test in three studies to 72 months in one study, M=16 months (SD=17). Two studies reported only intent and refusal skills outcomes (Clark, Ringwalt, Hanley, & Shamblen, 2010; Longshore, Ellickson, McCaffrey, & Clair, 2007). All but two studies included clear descriptions of how the program was implemented with fidelity to the design. All but two studies used a confirmed intent-to-treat design. Demographic information about participants is reported in Table 1 for the sample sizes used at the time of program delivery. Sample sizes varied from 42 to 5,756 (M =1,613, SD=1,672). The gender distribution was relatively equal (M=49%, SD=5%). The racial composition varied in studies with the percentage of participants identifying as non-White ranging from 3% to 99% (M =50%; SD=31%). Additionally, 46% of studies reported some percentage of the participants as to be from low socio-economic status backgrounds.

Table 1.

Description of Study Characteristics and Effect Sizes Included in the Meta-Analysis

Study	Tx n	RC T	Fidelity	ITT Design	Developer	Comparison	Follow up (months)	Initiation d (SE)	Recent Use d (SE)	Intent to Used (SE)	Refusal Skills d (SE)
Apsler et al. (2006)	87	Yes	Yes	Yes	No	Health info	18		0.00 (0.45 )
Aseltine et al. (2000)	198	Ye	No	Yes	No	No program	6	−0.23 (0.15)
Bacon et al. (2013)	5066	Yes	Yes	Yes	No	No program	6	−0.04 (0.01)	−0.25 (0.01 )
Clark et al. (2010)	2421	Yes	Yes	Yes	No	No program	12			− 0.02 (0.04)	0.00 (0.04)
D'Amico et al. (2007)	64	No	Yes	Yes	Yes	No program	1.5		−0.36 (0.19)
DeWit et al. (2000)	87	Yes	Yes	No	No	No program	6		−0.40 (0.16)
Eisen et al. (2003)	2732	Yes	Yes	Yes	No	TAU	12	−0.12 (0.05)	−0.14 (0.06)	− 0.01 (0.05)
Ellickson et al. (2003)	2553	Yes	Yes	Yes	Yes	TAU	18	−0.25 (0.05)	−0.03 (0.03)
Fosco et al. (2013)	332	Yes	Yes	Yes	Yes	TAU	24		−0.18 (0.10)
Gottfredson et al. (2010)	191	Yes	Yes	Yes	No	TAU	0	−0.07 (0.17)	−0.01 (0.14)
Griffin et al. (2003)	379	Yes	Yes	Yes	Yes	TAU	12		−0.07 (0.05)
Griffin et al. (2009)	92	Yes	Yes	Yes	No	TAU	12		−0.30 (0.16)
Hecht et al. (2006)	2143	Yes	Yes	Yes	Yes	TAU	14		−0.02 (0.02)		0.01 (0.03)
Longshore et al. (2007)	827	Yes	Yes	Yes	Yes	TAU	30			−0.09 (0.04)	0.02 (0.03)
McNeal et al. (2004)	342	Yes	Yes	Yes	No	TAU	6		0.01 (0.18)
Parent (2010)	67	Yes	Yes	Yes	No	TAU	0	0.62 (0.22)
Ringwalt et al (2010).	2470	Yes	Yes	Yes	No	No program	12	−0.04 (0.08)	0.00 (0.07 )
Slater et al. (2006)	2108	Yes	Yes	Yes	No	TAU	0	−0.01 (0.06)
Sloboda et al. (2009)	5756	Yes	Yes	Yes	Yes	No program	48	0.02 (0.04)	−0.03 (0.05)
Spoth et al. (2008)	428	Yes	Yes	Yes	No	Health info	72	−0.15 (0.09)
St. Pierre et al. (2005)	597	Yes	Yes	Yes	No	No program	12	0.11 (0.10)	0.04 (0.10)
Turner-Musa et al. (2008)	42	Yes	No	No	Yes	No program	6		0.00 (0.36)
Vicary et al. (2006)	531	Yes	Yes	Yes	Yes	No program	36	0.00 (0.07)	−0.17 (0.11)

Note: Tx n, treatment/intervention group size; RCT, randomized control trial; ITT, intent-to-treat; TAU, treatment as usual

Program characteristics

Table 2 presents the program characteristics from each study. Nine studies were conducted with students in predominantly urban locations, four were in rural locations, and ten were in mixed settings. A variety of curricula were represented, with five studies implementing Project ALERT, three implementing Life Skills Training, three implementing All Stars, and the remaining twelve implementing different program curricula. The delivery setting was a combination of during and after school activities in four studies, after school in two studies, and during school in the remaining 17 studies. For instructors, a clinician delivered the program in nine studies, three studies had either a police officer, a trained adult with a peer leader, or a volunteer from the community led the program, and teachers delivered the curricula in the remaining ten studies. The number of sessions ranged from 3 to 40 (M=17, SD=9). Booster sessions given six months to one year or later occurred in 10 studies. Participants were compared to students who received no programming in 10 studies, health information in two studies, or treatment as usual in the remaining 11 studies.

Table 2.

Description of Program Characteristics from Studies Included in the Meta-Analysis

Study	Location	Program	Instructor	School Setting	Sessions	Booster	Male	Non- White	Low SES
Apsler et al. (2006)	Urban	CASPAR	Clinician	During	30	No	52%	33%	Yes
Aseltine et al. (2000)	Mixed	Across the Ages	Volunteer	During	27	No	50%	72%	Yes
Bacon et al. (2013)	Mixed	TGFD	Clinician	During	10	No	52%	52%	Yes
Clark et al. (2010)	Mixed	Project ALERT	Teacher	During	14	Yes	49%	43%	Yes
D'Amico et al. (2007)	Mixed	CHOICE	Clinician	After	5	No	46%	39%	No
DeWit et al. (2000)	Mixed	Opening Doors	Clinician	During	17	No	44%	NR	Yes
Eisen et al. (2003)	Mixed	Lions Quest	Teacher	During	40	No	48%	74%	No
Ellickson et al. (2003)	Rural	Project ALERT	Teacher	During	14	Yes	50%	14%	No
Fosco et al. (2013)	Urban	Family Check-Up	Clinician	Mixed	3	No	51%	43%	No
Gottfredson et al. (2010)	Urban	All Stars	Clinician	After	13	Yes	54%	72%	Yes
Griffin et al. (2003)	Urban	LST	Teacher	During	25	Yes	49%	87%	Yes
Griffin et al. (2009)	Urban	BRAVE	Clinician	During	30	No	37%	99%	No
Hecht et al. (2006)	Urban	Keepin' it REAL	Teacher	During	10	Yes	53%	76%	Yes
Longshore et al. (2007)	Rural	Project ALERT	Teacher	During	19	Yes	56%	20%	No
McNeal et al. (2004)	Mixed	All Stars	Clinician	Mixed	13	No	46%	23%	No
Parent (2010)	Urban	PPP	Adult & Peer	Mixed	16	No	52%	85%	Yes
Ringwalt et al. (2010)	Mixed	Project ALERT	Teacher	During	14	Yes	49%	43%	Yes
Slater et al. (2006)	Mixed	All Stars	Teacher	Mixed	22	Yes	48%	13%	No
Sloboda et al. (2009)	Urban	TCYL	Police	During	17	Yes	45%	56%	No
Spoth et al. (2008)	Rural	LST	Teacher	During	15	No	53%	4%	No
St. Pierre et al. (2005)	Mixed	Project ALERT	Clinician	During	14	Yes	51%	19%	No
Turner-Musa et al. (2008)	Urban	H2P (Hip-Hop)	Teacher	During	10	No	34%	95%	No
Vicary et al. (2006)	Rural	LST	Teacher	During	15	No	54%	3%	No

Note: PPP, Peer Prevention Project; TCYL, Take Charge of Your Life; NR, not reported; SES, socio-economic status

Risk of Bias within Studies

Categorical moderator analyses assessed clarity of intent-to-treat design, comparison condition, evidence of fidelity, and developer involvement in evaluation. Results indicated that the effect sizes differed, though these were not significantly associated with effect size moderation as indicated by the between-group heterogeneity values. A meta-regression analysis of the number of follow-up months on effect size was also not significant. The results suggest no evidence of bias due to study quality or characteristics.

Individual Study Results and Synthesis

The effect sizes for individual studies are summarized in the forest plots shown in Figures 2 through 4 (for cannabis use, intent to use, and resistance skills, respectively), showing each study in the meta-analysis with follow-up length in months, standardized mean differences with confidence intervals, and pooled result calculated with random effects weights. Two studies had medium effect sizes, and the rest had small effects. Two studies had no effect and four studies had negative effects favoring the control groups. The overall program effect on adolescent cannabis use was small and significant, favoring the programs (Figure 2). The random-effects weighted mean effect size for any cannabis use (k=21) was d̄ =−0.07, 95% CI [−0.12; −0.02]; p<0.01). The homogeneity test result suggested evidence of marginal heterogeneity (τ²=0.004, Q=29.78, df=20, p=0.07, I²=32.83%). For the one study that used a quasi-experimental design rather than random assignment (D’Amico & Edelen, 2007), the effect size for cannabis use (d̄ =−0.36, 95% CI [−0.73, 0.01], p=0.07) exceeded the pooled effect size from studies using random assignment (d̄ =−0.07, 95% CI [−0.11, −0.02], p<.01). The pooled results for intention to use cannabis (k=3) was small and did not achieve statistical significance (d̄ =−0.046, 95% CI [−0.10, 0.01]; p=0.09) (Figure 3). The pooled result for refusal skill outcomes (k=3) was also not statistically significant (d̄ =0.01; 95% CI [−0.02, 0.05]; p=0.49) (Figure 4). The homogeneity tests were null and the I² statistic was 0% for the last two outcomes, indicating no study heterogeneity.

Figure 2.

Forest Plot Showing Effect Size (Standardized Mean Differences) and 95% Confidence Interval with Random Effects Mean for Cannabis Use Outcomes

Figure 4.

Forest Plot Showing Effect Size (Standardized Mean Differences) and 95% Confidence Interval with Random Effects Mean for Refusal Skills to Resist Cannabis Use Outcomes

Figure 3.

Forest Plot Showing Effect Size (Standardized Mean Differences) and 95% Confidence Interval with Random Effects Mean for Intention to Use Cannabis Outcomes

The possibility of publication bias was assessed first by visually inspecting funnel plots of standard error by standard difference in means for the three outcome measures. The funnel plot for any cannabis use outcome was symmetrical, suggesting no indication of study bias. A regression test of association between study effect sizes and sample sizes was not statically significant (b=−0.45; SE=0.47, p=0.35). However, a trim and fill analysis (Card, 2012; Duvall, 2005) yielded one trimmed study to the right of the mean, but the effect was trivial (d̄ =−0.06, 95% CI [−0.11, −0.02]). Publication bias test results for intention to use and refusal skills outcomes were omitted with too few cases for assessment.

Results of the moderator analysis indicated significant differences only for the program instructor type (Q_Between=11.40, df=4, p=0.02). The within-group weighted mean effect sizes for the program instructor categories for teachers (d̄ =−0.08, 95% CI [−0.15, −0.01], p=0.02) and clinicians (d̄ =−0.10, 95% CI [−0.20, −0.01], p=0.04) significantly favored reduced cannabis use. The clinician category included the one study that used a quasi-experimental design rather than random assignment (D’Amico & Edelen, 2007). When this single study was removed, the effect size became smaller and non-significant for clinicians (d̄ =−0.08, p=0.10), while the effect for teachers remained significant (d̄ =−0.08, p=0.02). The one program delivered by a youth peer with an adult significantly favored increased cannabis use (d̄ =0.62, 95% CI [0.18, 1.06], p=0.01) with significant differences between categories (Q_Between=9.53, df=1, p<0.01). The instructor categories, however, explain a trivial proportion of variation with I²=13.73% in the clinician category and I²=0.0% in the other categories. All the other moderator variables were not statically significant.

Discussion and Conclusions

This meta-analysis is the first to examine the effectiveness of interactive North American school-based substance use prevention programs delivered exclusively during middle school on cannabis outcomes. The overall effect size suggests that interactive middle school-based programs potentially delay or prevent cannabis use among North American adolescents. This finding is consistent with findings from earlier reviews (Norberg et al., 2013; Porath-Waller et al., 2010). As such, interactive cannabis use prevention programs offered during middle school present an effective strategy to consider, particularly as rates of cannabis use have been found to increase with age. The small overall effect size, however, may signal the need for future studies to further explore the linkage between program theory, program content, and program exposure to help identify what may be contributing to these limited program effects. While the effect size is small, however, it is still important to consider the practical significance of even a small effect size (McCartney & Rosenthal, 2000), especially if these more effective substance use programs were implemented on a large-scale

The type of program instructor appears to moderate program effectiveness in reducing cannabis use. Delivery of these programs by teachers was found significantly more effective and there was a large negative effect for delivery by youth peers with adults. Both of these findings contradict the findings of prior reviews. However, the finding related to youth peers should not be generalized since it was based on a single study with an effect size in the peer category. This signals the need for studies that further explore how instructor type may influence differences in program effectiveness, especially given considerations around scalability and cost-effectiveness for schools who implement prevention programming. No other moderating factors were found significant. One explanation is that the interactive programs included in this study had very similar theoretical and delivery design characteristics. The marginal differences in effects might be explained by other factors that were either not coded (e.g., single vs. multiple modality with family or community-based components) or external to the programs (e.g., exposure to other drug prevention or health information).

Some study limitations should be considered. Non-significant moderator analysis results may be due to the small number of studies, which reduced statistical power to detect effects (Card, 2012). There are many ways to assess research quality, and the present meta-analysis did so by testing individual research components for risk of bias assessment rather than by scoring items in a scale. In addition, all of the studies utilized self-report measures of outcomes. Self-report accuracy could have differed across conditions in these studies and thus influenced the results of those studies, and ultimately, the inferences made in this meta-analysis. This study also did not code programs based on whether they were universal, selective, or indicated. Therefore, differences based on this typology could not be explored. Finally, the present review focused on programs implemented within the school setting.

As Norberg et al. (2013) show, effects of school-based cannabis prevention programs may differ by modality and external components, such as family and community-based features that potentially enhance prevention. As major cultural shifts continue related to acceptance of marijuana use, adolescent substance abuse prevention still remains a critical priority in the US and Canada. School-based programs can be a cost-effective component of wider prevention strategies (Lemon, Pennucci, Hanley, & Aos, 2014). While more rigorous studies continue to be needed that evaluate the impact of these programs, the present meta-analysis provides further support for the use of interactive programs in middle schools to prevent cannabis use by adolescents in North America.