The Making of Evidence-based Practice: The Case of Project ALERT

Abstract

Evidence-based practice has been enthusiastically embraced within the field of drug prevention during the past decade. Project ALERT, a school-based universal prevention program, is among the most widely advocated evidence-based interventions. We examined the results of three large-scale evaluations of Project ALERT, and concluded that assessment of data from the total samples shows that the program has little effect on drug use. Despite this, Project ALERT is included on evidence-based drug prevention lists because the criteria for inclusion are extremely weak. We discuss the implications of this for drug prevention evaluation research and the creation of evidence-based practice lists.

1. Introduction

Like many areas of social policy, the field of drug prevention has enthusiastically embraced the idea of evidence-based practice in the last decade (Hallfors et al., 2007; Petrosino, 2003; Weiss et al., 2008). Project ALERT, a school-based prevention program targeted at middle-school children, is among the most widely advocated evidence-based interventions. For example, it was rated an “exemplary program” by the US Department of Education (US Department of Education Safe, Disciplined, and Drug-Free Schools Expert Panel, 2002) and classified as a “research-based” program by the National Institute on Drug Abuse (2003). Most important, given the list’s high profile in the drug prevention field (Hallfors et al., 2007), Project ALERT is included in both the original and revised versions of the Substance Abuse and Mental Health Service’s Administration’s (SAMHSA) registry of evidence-based interventions.¹ Each of these lists of best practices claims that the inclusion of Project ALERT is based on the research evidence pertaining to the program’s efficacy in reducing drug use (effects on non-behavioral variables such as attitudes and knowledge are insufficient for inclusion). Likewise, the marketing of the program is firmly grounded in the idea that the program’s efficacy has been established through rigorous evaluations. As the promotional material posted on the ALERT webpage states: “Project ALERT works. That’s not just marketing hype. Empirical research tells us so” (Best Foundation for a Drug-Free Tomorrow, no date).

Project ALERT has been the subject of two large-scale evaluations conducted by its developers at the RAND Corporation and one independent evaluation. The first of the developer-led evaluations commenced in the mid-1980s and was conducted with middle and junior high school students in Oregon and California (referred to herein as the “West Coast evaluation”) (Bell, Ellickson and Harrison, 1993; Ellickson & Bell, 1990a; Ellickson et al., 1993). The second developer-led evaluation was conducted with students from middle schools in South Dakota and began in the late-1990s (Ellickson et al., 2003; Longshore et al., 2007). The independent evaluation of Project ALERT was conducted in eight middle schools in Pennsylvania in the early-2000s (St. Pierre et al., 2005).

Given the timing of the evaluations, it was the results of the West Coast study that were the basis for Project ALERT’s inclusion on the various best practice lists that appeared in the early-2000s (i.e., the US Department of Education’s list of exemplary programs and SAMHSA’s original National Registry of Effective Prevention Programs). However, the findings of the two more recent evaluations were available for consideration when Project ALERT was reviewed for inclusion in SAMHSA’s revised National Registry of Evidence-based Programs and Practices (NREPP) in December of 2006 (National Registry of Evidence-based Programs and Practices, 2006).

Here we will examine the results of each of the three evaluations.² This is followed by a discussion of how the findings from these evaluations are used in the evidence-based practices literature, especially that of the revised NREPP. We conclude with a discussion of the implications of our findings for the manner in which evaluation research is conducted in the field of drug prevention and the procedures to generate lists of evidence-based practices.

2. West Coast Evaluation of Project ALERT

The version of the ALERT program that was evaluated in the West Coast study comprised 11 sessions, eight delivered in 7^th grade and three in 8^th grade. Two versions of the program were assessed, one delivered just by health educators and one delivered by health educators and teen peers. Outcome data pertaining to actual drug use (as opposed to mediator variables such as knowledge and attitudes) from this evaluation of ALERT are presented in three main publications. The first of these contains a set of findings regarding its effects at 3-, 12-, and 15-month follow-up when subjects were in 7^th and 8^th grade (Ellickson and Bell, 1990a). The second publication presents results from a 9^th grade follow-up (Bell et al., 1993), while the third contains data collected three and five years after the intervention was administered, when subjects were in 10^th and 12^th grade (Ellickson et al., 1993a). It is not necessary to spend much time on the 9^th through 12^th grade evaluations since the data from these revealed essentially no differences between ALERT subjects and controls on measures of drug use. The number of statistically significant differences between the two ALERT and control groups at 9^th grade was just two out of 78 comparisons, a difference that the investigators acknowledge “is easily explained by chance” (Bell et al., 1993: 475). Also no effects were found for alcohol, cigarettes or marijuana use at 12^th grade, leading the investigators to observe that: “Once the lessons stopped, the program’s effects on drug use stopped” (Ellickson et al., 1993a, p. 856). The evidence of the program’s efficacy must therefore be found in the 1990 publication in Science (Ellickson & Bell, 1990a) that reported outcomes at 3-, 12- and 15-month follow-ups.³

The results of Ellickson and Bell’s (1990a) evaluation of ALERT have been discussed in detail in a number of earlier publications (Gandhi et al., 2006; Gorman, 1994; 1998), including a re-analysis of the data by a committee of the National Research Council (Gerstein & Green, 1993). The main issue raised in these earlier critiques is that Ellickson and Bell’s (1990a) publication presents no results describing how the total ALERT groups compared to the total control group at follow-up; rather, the sample was broken down into three subgroups according to drug use reported at baseline. In addition, the effects of the program were assessed for multiple outcome variables at each follow-up point (e.g., “ever” used the drug and “weekly” use of the drug). The details of these subgroups and outcome variables are presented in the footnote to Table 1.

Table 1.

Summary of Results from the 7^th and 8^th Grade Follow-ups of the West Coast Project ALERT Evaluation (Ellickson & Bell, 1990a)^a

Variables	Logically Possible Comparisons	Comparisons Reported by Ellickson & Bell (1990a)	Statistically Significant Positive Results	Statistically Significant Negative Results
Alcohol Use	68	54	2	1
Cigarette Use	86	64	6	4
Marijuana Use	70	46	6	0

Notes

^aThe total subgroup analyses for each drug was a function of the number of follow-up points, study conditions, risk groups, and outcome variables. For each drug, there were 3 follow-up points (3-, 12- and 15-month) and 3 study conditions (ALERT-teacher taught; ALERT-teacher and peer taught; control). There were also 3 risk subgroups for each drug, although the definitions of these differed. For alcohol and cigarettes, the subgroups were “nonusers”, “experimenters” and “users”. The former were those who had never used the drug, experimenters those who had used the drug <3 times and not in the previous month, and users those who had used ≥3 times in the previous year or at any point during the previous month. In the case of marijuana, the 3 subgroups were based on prior use of marijuana and prior use of cigarettes (since use of marijuana was relatively infrequent). Those who had used neither drug at baseline were defined as non-users, marijuana nonusers/cigarette as experimenters, and users of both drugs as users. Finally, there were 5 outcome variables assessed at follow-up for alcohol and marijuana, and 6 for cigarettes.

The combination of these risk groups and outcome variables, along with the three follow-up periods (3-, 12- and 15-month) and two study conditions (teacher taught; teacher plus peer taught) resulted in a large number of comparisons being made between the ALERT and control groups for each drug. The number of such logically possible comparisons between study conditions is shown in column 2 of Table 1. Ellickson and Bell (1994), however, contend that it is unfair to judge the effectiveness of ALERT in terms of all of the logically possible comparisons that their analysis strategy produced. Specifically, they argue that 42 of the comparisons that were logically possible involved subgroups of individuals that reported too little drug use to allow for meaningful statistical analysis to be conducted. In line with this argument, the actual number of comparisons for each drug as reported by Ellickson and Bell (1990a) is shown in column 3 of Table 1. The two remaining columns of the table show the number of statistically significant (at the conventional level of p ≤0.05) positive and negative results that emerged from the analysis. It can be seen that even when judged in terms of the findings reported by Ellickson and Bell (1990a), the effects of ALERT that emerged from the subgroup analyses were limited. For alcohol use, just two of the follow-up comparisons made between ALERT participants and controls were favorable to the intervention and statistically significant (one other showed the program was detrimental). Similarly, while six comparisons indicated that there were proportionally fewer cigarette users in the ALERT groups, four others showed that a higher proportion of program participants than controls were smoking at follow-up assessments. In the case of marijuana, six of the 70 comparisons between the ALERT and control groups were statistically significant and favorable to the program (Gorman, 1994).

3. South Dakota Evaluation of Project ALERT

The revisions to the ALERT curriculum that occurred in the late-1990s involved additional lessons on smoking cessation and alcohol use and the inclusion of “home-learning opportunities” intended to encourage parental involvement in drug prevention. The revised curriculum comprised 14 sessions, 11 delivered in 7^th grade (compared to eight in the previous version) and three in 8^th grade (as in the previous version) (Ellickson et al., 2003, p. 1831). In addition, to these revisions, Ellickson and colleagues also developed a version of the program that included 9^th and 10^th grade booster sessions as well as the 7^th and 8^th grade program sessions. This extended version of the program they called “ALERT Plus”.

The revised 7^th and 8^th grade ALERT curriculum (henceforth called “ALERT Revised”) and ALERT Plus were evaluated in a randomized trial conducted in 55 South Dakota schools starting in the Fall of 1997. A total of 4,689 students were assessed at baseline – 87% of the 5,412 7^th graders enrolled in the eligible schools. Like the West Coast evaluation, the immediate effects of ALERT through middle school were first assessed followed by the long-term effects as subjects entered 9^th and 10^th grade.

3.1 Effects of the Revised ALERT Curriculum at 8^th Grade

In the first report to contain outcome information about the effects of the revised ALERT curriculum on cigarette, alcohol and marijuana use, Ellickson and colleagues present data collected at the 18-month follow-up when the students had received the 8^th grade as well as 7^th grade classes (Ellickson et al., 2003). Since this assessment was conducted before implementation of the 9^th and 10^th grade ALERT Plus booster sessions, the data from the ALERT Revised and ALERT Plus intervention groups were combined in the analysis. The total sample for this comprised 4,276 students who completed both the baseline and 18-month assessments – 91% of the baseline sample and 79% of those enrolled in the eligible schools.

Results for the same three risk groups used in the West Coast evaluation (and defined in the same way) were again reported, that is, nonusers, experimenters and users. Similar outcome variables were also used in the analysis, specifically post-baseline initiation, use in the past month, and weekly use (measures from the West Coast evaluation not used included quitting and frequency of monthly use). Three additional variables (not used in the West Coast evaluation) were also reported for alcohol use: alcohol-related consequences (5 dichotomous items including fighting and getting into trouble at school), high-risk drinking (3 dichotomous items pertaining to heavy alcohol use and mixing alcohol use with marijuana use), and overall misuse (all 8 items combined). Since the analysis reported by Ellickson et al. (2003) involved just one experimental condition, only one follow-up point, and fewer outcome variables, there was not the same multitude of analyses reported as in the initial publication from the West Coast evaluation (Ellickson & Bell, 1990a). In addition, unlike the West Coast evaluation, effects for the total sample as well as subgroups were reported.

Table 2 summarizes the statistically significant findings (at the conventional p ≤0.05) from the 8^th grade follow-up reported by Ellickson et al. (2003). The effects of the program were most noticeable for cigarette smoking, for which eight of the ten comparisons presented were statistically significant. Specifically, there were statistically significant positive program effects observed for all three outcomes among the total sample, and the subgroup analyses showed significantly less regular use among both baseline non-users and experimenters and significantly less current use among baseline experimenters and users. There were, however, fewer statistically significant differences between the ALERT group and the control group on the marijuana and alcohol outcome measures (three of 11 comparisons and five of 22, respectively). In the case of the former, the effects were limited to initiation: 5% of ALERT non-users initiated use by the 8^th grade follow-up compared to 8% of the control group. The figures for the ALERT and control marijuana experimenter subgroup were 27% and 37%, respectively. For alcohol, program effects were observed for two of the three misuse measures in the total sample (consequences and overall misuse) and on all three misuse measures among the 16% of the subjects categorized as baseline users. There were no effects observed on any of the three measures of actual alcohol use.

Table 2.

Summary of Results from the 8^th Grade Follow-up of the South Dakota Project ALERT Evaluation (Ellickson et al., 2003): Presence of Statistically Significant Difference between ALERT Group and Control Group (Yes or No)^a

Drug	Variable	Total Sample	Non-Users	Subgroupsb Experimenters	Users
Cigarettes	Regular (Weekly)	Yes	Yes	Yes	No
	Current (Past Month)	Yes	No	Yes	Yes
	Ever (Initiation)c	Yes	Yes
Marijuana	Regular (Weekly)	No	No	No	No
	Current (Past Month)	No	No	No	No
	Ever (Initiation)d	Yes	Yes	Yes
Alcohol	Regular (Weekly)	No	No	No	No
	Current (Past Month)	No	No	No	No
	Ever (Initiation) c	No	No
	Consequences	Yes	No	No	Yes
	High-Risk	No	No	No	Yes
	Overall Misusee	Yes	No	No	Yes

Notes:

^aNot all of these findings are presented in the results section of the paper of Ellickson et al. (2003). However, each of these is clearly a possible outcome variable. Indeed, Ellickson and colleagues note that the items they used to assess alcohol, cigarette and marijuana use “…allowed us to evaluate program effects on ‘ever’, ‘past month’, and ‘weekly’ use” (Ellickson et al., 2003, p. 1832).

^bThe subgroups were defined in the same way as in the West Coast evaluation (see Note^a of Table 1 for details).

^cFor cigarettes and alcohol, the total sample and non-user subgroup are the same, as only baseline non-users can initiate use of these substances after the baseline assessment.

^dFor marijuana, the experimenter subgroup is not defined by baseline use of the drug and therefore both non-users and experimenters can initiate marijuana use over the course of the study.

^eThe 8-item overall misuse measure was created by combining the 5-item consequences measure and the 3-item high-risk measure.

The analysis conducted by Ellickson and colleagues (2003) of the 8^th grade follow-up of the Revised ALERT curriculum is an improvement over that found in the first evaluation of the original program (Ellickson & Bell, 1990a), if only because there were far fewer subgroup analyses presented. In addition, the results are stronger, at least in the case of cigarette smoking. However, there are at least two aspects of the results presented that might temper one’s optimism about the program’s efficacy. First, the number of independent positive results reported (and summarized in Table 2) is not quite as high as it initially appears. In the case of cigarette use, for example, the positive result found for initiation among the total sample and that found for the baseline non-user subgroup are the same result since, by definition, the only subgroup of the total sample that can initiate use after baseline is the baseline nonuser subgroup. In the case of alcohol, it is the three misuse variables (consequences, high-risk and overall misuse) that are not independent measures: specifically, the outcome overall misuse was simply created by combining the 5 items that were used to assess alcohol-related consequences with the 3 used to assess high-risk drinking. Second, the statistically significant differences in the misuse measures between the groups at follow-up are of questionable practical significance since they are very small. Among the baseline user subgroup, for example, the statistically significant differences between the ALERT group and control group were of the magnitude of 0.45 on the 8-item overall misuse scale, 0.25 on the 5-item consequences scale, and 0.18 on the 3-item high-risk use scale. And, as shown in Figure 1, the differences were even smaller for the total sample, just 0.2 (i.e., one-fifth of a problem) on the 8-item overall misuse scale and 0.1 (i.e., one-tenth of a problem) on the 5-item consequences scale. It is unlikely that such minute differences have any practical implications for subjects’ immediate or future risk of experiencing alcohol-related problems.

Figure 1.

Follow-up Scores for the Total Sample on Alcohol Misuse Scale (range 0-8) and Overall Consequences Misuse Scale (range 0-5) Reported in Ellickson et al. (2003)

Ellickson and colleagues state that the findings of the South Dakota 8^th grade follow-up are “new and significant” and “suggest that programs such as Project ALERT can generate a broad range of public health benefits” (Ellickson et al., 2003, pp. 1844 & 1835). Overall, the results of the evaluation are said to provide support for the social influence approach to prevention and for the implementation of evidence-based programs, and, when considered together with the earlier evaluation, to “indicate that Project ALERT works in the Midwest as well as the West Coast” (Ellickson et al., 2003, p. 1835, emphasis added). While Ellickson and colleagues might have had reason to be encouraged by the results pertaining to cigarette smoking, the findings for marijuana and alcohol use hardly suggest that a broad range of public health benefits will follow from widespread dissemination of the Revised ALERT curriculum or that this is a program that unequivocally “works” to prevent a broad range of drug use behaviors. Moreover, since this was just the first stage of their evaluation, with additional follow-ups to come in 9^th and 10^th grade, a more caution interpretation of these results might have been in order especially given that the effects observed at 8^th grade in the initial evaluation of the original ALERT curriculum disappeared by the 9^th grade follow-up (Ellickson et al., 1993a).

3.2 Effects of the Revised ALERT and ALERT Plus Curricula at 9^th and 10^th Grade

No detailed account of the effects of the two revised ALERT curricula on the full South Dakota sample has been published to date. However, in their analysis of a high-risk subgroup Longshore et al. (2007) note that the effects on drug use and cognitive factors observed in the 8^th grade among the full sample “did not persist at ninth grade in either program condition” (Longshore et al., 2007, page 655).⁴ This finding essentially replicates that of the West Coast evaluation of the first ALERT curriculum in that any immediate reductions in drug use dissipated by high school.

The high-risk subgroup analysis presented by Longshore et al. (2007) reports 9^th grade data from just 1,383 (30%) of the 4,689 subjects who were assessed at baseline. “High-risk” individuals were those who reported life-time use of either tobacco or marijuana at the 7^th grade baseline assessment. In the analysis presented, the effects of ALERT on three dichotomous measures of drug use were assessed – weekly use of alcohol, tobacco and marijuana. In addition, the alcohol consequences and high-risk drinking measures used in the 8^th grade evaluation were used in the analysis along with a 4-item measure of consequences of marijuana use. Thus, a number of the variables reported in the previous studies (e.g., monthly use, daily use) were not reported in this analysis. The results of the analyses, which are summarized in Table 3, were presented separately for the Revised ALERT curriculum and the ALERT Plus curriculum, and separately for boys and girls. It can be seen that statistically significant effects were only found for the high-risk females in the ALERT Plus group. This group was comprised of just 163 subjects (of more than 2,500 who comprised the original data analysis sample in the ALERT condition). Moreover, the authors of the report note that attrition in the study “was neither negligible nor random” (Longshore et al., 2007, p. 666). It is therefore questionable whether these results even generalize to the other high-risk students in the original study sample, let alone beyond these to “high risk” students in general.

Table 3.

Summary of Results from the 9^th Grade Follow-up of High-Risk Sub-sample of the South Dakota Project ALERT Evaluation (Longshore et al., 2007): Presence of Statistically Significant Difference between ALERT Groups and the Control Group (Yes or No)

	Females		Males
Outcome Variable	ALERT Plus	Revised ALERT	ALERT Plus	Revised ALERT
Weekly alcohol use	Yes	No	No	No
Weekly tobacco use	No	No	No	No
Weekly marijuana use	Yes	No	No	No
High-risk alcohol use	Yes	No	No	No
Alcohol consequences	Yes	No	No	No
Marijuana consequences	No	No	No	No

4. The Pennsylvania Independent Replication Study

The revised version of ALERT, with 11 classes in 7^th grade and three 8^th grade booster sessions, was evaluated in the independent evaluation conducted by St. Pierre and colleagues (2005). As in the original West Coast evaluation, two versions of the program were assessed, one taught solely by an adult and the other taught by both an adult and a teen. However, unlike the program used in the Ellickson et al. evaluations, the adults in this study were trained program leaders from the community and not middle school teachers.

The experimental design involved two consecutive cohorts of 7^th grade students (n = 1,649 at baseline) followed up at posttest, before and after the 8^th grade booster sessions, and one year after the boosters when students were in 9^th grade. Seventy-three percent of students completed all five assessments and 88% completed four of the five. The main outcomes assessed were past year, past month, and lifetime use of alcohol, cigarettes and marijuana (the two latter measures having also been employed to assess use of each drug in the evaluations conducted by Ellickson and colleagues). In addition, last month heavy alcohol use (number of days consuming three or more drinks) was assessed. St. Pierre and colleagues conducted three types of statistical analyses all of which took account of the nested quality of the study design (that is, analyzing data from students randomly assigned to study conditions through their classrooms), and one of which directly replicated the analysis strategy used by Ellickson et al. (2003) in the South Dakota study. This ensured that any discrepancy in results between the two studies could not be attributed to differences in the analyses performed.

Of the 62 comparisons made between the ALERT groups and control group using data from the entire sample, none showed a statistically significant effect on drug use favoring the program. There were three statistically significant negative effects, but these were probably just chance occurrences given the large number of comparisons made. The only statistically significant positive effect found was for past month marijuana use in one of the 18 risk-factor subgroup analyses that were conducted, a result that could also simply be due to chance. The gender subgroup analyses showed no differences between the groups. St. Pierre and colleagues seemed somewhat surprised by their findings, and since the program appears to have been properly implemented they were forced to look elsewhere for an explanation of why their implementation of ALERT yielded nothing. The two reasons they offered were the difference in program implementers (non-teachers versus teachers) and the context in which the program was delivered (essentially, Pennsylvania is not South Dakota). Both points are, of course, highly relevant to the question of the dissemination of universal evidence-based programs, for having an intervention that only works if delivered by teachers in South Dakota would severely limit potential utility and value.

5. Summary of the Results of the Project ALERT Evaluations

Before discussing how the findings from the ALERT evaluations have been interpreted in the evidence-based literature, a brief summary of the main findings from these will be presented. Since ALERT is considered to be both a universal intervention and a selective intervention (National Registry of Evidence-based Programs and Practices 2006), the results from the analyses of the total sample and those from the various subgroups analyses will be discussed separately.

5.1 Analyses Based on the Full Sample

Table 4 shows that the main findings regarding alcohol, cigarette and marijuana use among the full sample in each of the three evaluations of ALERT are essentially consistent with one another. The immediate program effects refer to those that occurred in 7^th grade and 8^th grade, and the long-term effects to those that occurred in 9^th grade and beyond.⁵

Table 4.

Summary of Results Pertaining to Alcohol, Cigarette and Marijuana Use in the Full Samples from the Three Project ALERT Evaluations

	Effects on Alcohol Use		Effects on Cigarette Use		Effects on Marijuana Use
Evaluation	Immediate	Long-term	Immediate	Long-term	Immediate	Long-term
West Coast a	---	None	---	None	---	None
South Dakota b	None	None	3 out of 3c	None	1 out of 3d	None
Pennsylvania e	None	None	None	None	None	None

Notes:

^aThe immediate effects of the West Coast evaluation on the total sample were not reported by Ellickson and Bell (1990a). Hence, no entry appears in the table (for a summary of the results of a re-analysis of the published data conducted by the National Research Council Committee see Footnote⁴). The long-term follow-ups refer to those conducted in 9^th, 10^th and 12^th grade (Bell et al., 1993; Ellickson et al., 1993a).

^bImmediate effects of the South Dakota evaluation refer to the 8^th grade follow-up (Ellickson et al., 2003) and long-term effects to the 9^th and 10^th grade follow-ups (Longshore et al., 2007).

^cStatistically significant positive effects were found for all 3 of the outcome variables measured (initiation, weekly use and monthly use).

^dA statistically significant positive effect was found for 1 of the outcome variables measured (initiation) but not for the other 2 (weekly use and monthly use).

^eThe immediate effects of the Pennsylvania evaluation refer to the 7^th and 8^th grade follow-ups and the long-term effects to the 9^th grade follow-up (St. Pierre et al., 2005).

The table indicates that there is no evidence that the ALERT program in either the original or revised forms has any beneficial effects on alcohol use and consumption (it should also be recalled that while the revised version showed some immediate effects on measures of misuse, these were of questionable practical significance). There were also no long-term effects among the full sample on cigarette use found in any of the three evaluations. One of the two studies that reported results at 8^th grade found no immediate effects, while the other reported positive program effects on initiation of use and both weekly and monthly cigarette use. Finally, with regard to marijuana use, there was again no evidence of any long-term effect of the program. As for immediate effects, the South Dakota evaluation found a difference between the two groups in terms of initiation, but not weekly use or monthly use.

5.2 Subgroup Analyses

Despite its limited potential as a universal intervention the evaluations may still indicate that ALERT reduces certain types of drug use with certain types of adolescents, thereby allowing the program to be effectively used as a selective (i.e., targeted at individuals at increased risk of developing a drug problem) or indicated intervention (i.e., targeted at individuals already experimenting with drugs) (National Institute on Drug Abuse, 2003). In order to use the findings from the ALERT evaluations to build a potentially effective targeted intervention these results would have to show some consistency across studies in terms of the high-risk subgroups that the program affected. It would not be of much practical use for the subsets of individuals for whom the program was effective to vary from one analysis to the next, since this would create uncertainty as to whether or not the program would be effective with a particular subgroup in the future. In contrast, if the program was found to be effective with some subgroups across multiple evaluations, then one would have more confidence that it would produce similar effects in the future (assuming such conditions as delivery of the intervention with fidelity and similar baseline levels of risk in the target population).

Table 5 summarizes the statistically significant results that have been reported from the subgroup analyses of the West Coast and South Dakota ALERT evaluations (as noted above, no such effects were found in the Pennsylvania evaluation). As stated earlier, the same subgroups were used in the analysis of the South Dakota 8^th grade follow-up data as in the analysis of data from the 3-, 12- and 15-month follow-ups in the West Coast evaluation, thereby making comparison of the two relatively easy. The findings that are consistent (in terms of the subgroup and type of drug use affected) across the two sets of analyses appear in bold in Table 5. It can be seen that these are limited to initiation of marijuana use among baseline non-users and weekly and past-month use of cigarettes among baseline experimenters. The alcohol misuse and consequences measures for which there were statistically significant differences in the 8^th grade South Dakota evaluation where not included in the West Coast evaluation, so it is not possible to assess consistency across the two studies on these measures. However, the two subgroups for which statistically significant effects on alcohol measures of any kind were found were not the same in the two studies (i.e., baseline non-users in the West Coast evaluation and baseline users in the South Dakota 8^th grade evaluation).

Table 5.

Summary of Results of the Subgroup Analyses from the West Coast Project ALERT Evaluation (Ellickson & Bell, 1990a) and the South Dakota Project ALERT Evaluation (Ellickson et al., 2003; Longshore et al., 2007)^a,^b

	Subgroup and Variable for which a Statistically Significant Difference between the ALERT and Control Conditions was Found
Drug	West Coast Evaluationc,d (3-, 12- and 15-month Follow-ups)	South Dakota Evaluationd (8th Grade Follow-up)	South Dakota Evaluatione (9th Grade Follow-up)
Alcohol	Non-Users Initiation (at 3-month follow-up) Past-month (at 3-month follow-up)	Users Consequences High-risk use Overall misuse	High-risk females Weekly use Consequences High-risk use
Marijuana	Non-users Initiation (at 12-month follow-up) Initiation (at 15-month follow-up)	Non-users Initiation	High-risk females Weekly use
	Experimenters Monthly (at 12-month follow-up)	Experimenters Initiation
	Users Weekly (at 3-month follow-up)
Cigarettes	Experimenters Quit (at 12-month follow-up) Quit (at 15-month follow-up) Daily (at 15-month follow-up) Weekly (at 15-month follow-up) Monthly (at 15-month follow-up) Past-month (at 15-month follow-up)	Non-user Initiation Weekly	High-risk females Weekly use
		Experimenters Weekly Past-month
		Users Past-month

Notes

^aNeither the long-term follow-up of the West Coast evaluation (Bell et al., 1993; Ellickson et al., 1993a) or the Pennsylvania evaluation (St. Pierre et al., 2005) produced any statistically significant effects in the subgroup analyses that could not be attributed to chance occurrence (see text for details).

^bThe results that are consistent across studies in terms of the high-risk subgroup and type of drug use that are found to be affected by ALERT appear in bold. Since the 9^th grade follow-up of the South Dakota subjects used a different definition of “high-risk” to that used in the two earlier analyses, these findings could not be examined for their consistency with the West Coast evaluation or the 8^th grade follow-up of the South Dakota evaluation.

^cThe table does not differentiate between the version of the ALERT program taught by teachers plus teen peers and the version of the program taught by teachers only. With just one exception, all of the results listed for 3-, 12- and 15-month follow-ups of the West Coast evaluation pertain to just one version of the program. The exception was initiation of marijuana use at 15-months among baseline non-users. Both versions of the program differed significantly from the control condition on this variable.

^dFor details of how the non-users, experimenters and users of each drug were defined see Note ^a of Table 1.

^e“High-risk” was defined as life-time use of either tobacco or marijuana at the 7^th grade baseline assessment.

Comparison of the 9^th grade South Dakota subgroup analysis with the two earlier analyses is also problematic since the criteria used in the former to identify “high-risk” individuals did not match those used in the latter to create the non-user, experimenter, and user subgroups. It is unclear why different criteria were used and whether other analyses were performed that used different definitions of “high-risk”. Whether or not the latter were performed, it would be helpful to know how robust the findings concerning high-risk females were to modifications of the criteria used. For example, would the subgroup differences remain if “high-risk” was defined according to some threshold level of baseline use for either cigarettes or marijuana (≥3 times in the previous month), or if baseline alcohol use was also included in the definition of “high-risk”?

Overall, the results presented in Table 5 suggest that the subgroup analyses that have been conducted to date in the ALERT evaluations provide little guidance as to exactly who the intervention should be targeted at if it is used as a selective as suggested on the NREPP website. The most one can say is that it might have some immediate effects on initiation of marijuana use among non-users and reduce smoking among those who have experimented with tobacco. However, even these findings should be interpreted with caution since the application of multiple statistical tests in subgroup analyses can lead to misleading results by, among other problems, inflating the number of false positive tests (Lagakos, 2006; Wang et al., 2007). As noted above, in the West Coast ALERT study, the positive results emerged from a subgroup analysis that involved a large number of comparisons among study conditions. In addition, randomization of subjects to study conditions is only designed to bring about comparability of the entire randomized groups in a study, not subsets of these groups (Gail, 1985). Thus, while the three main groups in each of the ALERT evaluations may have been comparable following randomization, this does not mean that the subsets created for the subgroup analyses remain comparable across study condition. So, for example, the ALERT “high-risk” female subgroup created for the analysis of the 9^th grade follow-up data of the South Dakota study might not be comparable to the control group “high-risk” female subgroup created for this analysis.

6. Why is Project ALERT Considered an Evidence-based Practice?

The question that arises from the above review is: How is it that the ALERT program ends up appearing on so many evidence-based practice lists of drug prevention? As noted earlier, the lists that appeared in the early-2000s focused almost exclusively on the West Coast evaluation. While this produced few, if any, short-or long-term effects on drug use when data from the entire sample were analyzed, some statistically significant positive effects were found when multiple subgroup analyses were performed on the middle school data. Such isolated statistically significant effects provide sufficient evidence for inclusion on the major lists of evidence-based drug prevention programs and practices, since their primary outcome criterion is typically just a single effect on one behavioral outcome from one evaluation (Gandhi et al., 2006; Gorman, 2002). Whether or nor this single effect emerged as a result of multiple subgroup analyses is irrelevant. Indeed, in the materials disseminated by agencies describing the evidence-based programs they have selected, these isolated effects are emphasized and the plethora of null findings ignored.

The 2002 Department of Education exemplary and promising programs report provides a good example of this process, describing the results of the West Coast ALERT evaluation as follows: “Statistically significant and consistent differences were found between the treatment and control students on both their use and beliefs about use for marijuana, alcohol, and cigarettes” (Safe, Disciplined, and Drug-Free Schools Expert Panel, 2002: 27). Along similar lines, the summary of the West Coast ALERT evaluation that was produced as part of the original NREPP review states that the program reduced initiation of marijuana by 30%, current use by 60% in the teacher-led version of the program, and current and occasional cigarette smoking by up to 25% (Schinke et al., 2002, p. 158). Both of these descriptions are accurate – as shown in Table 1, the West Coast subgroup analyses did produce a few statistically significant differences between the ALERT and control groups in terms of drug use at follow-up, and some of these were in the order of 25%, 30% and 60%. However, neither of these summaries can seriously be considered to contain a complete and candid description of the results of the first ALERT evaluation. Specifically, the fact that the later follow-ups of the program found absolutely no effects on drug use is not mentioned in either the Department of Education or NREPP documents. Nor is the fact that the majority of comparisons showed no difference between the study conditions.

The revised NREPP procedure for designating a program as “evidence-based” is very similar to its predecessor. It simply requires that: “The intervention demonstrates one or more positive outcomes (p ≤ .05) in mental health and/or substance use behavior among individuals, communities, or populations” (Department of Health and Human Services, 2007, p. 30815). Evidence of this positive outcome needs to have been published in a peer-reviewed journal or documented in a comprehensive evaluation report. In addition, the scoring system used in the review process assigns an additional “priority point” to interventions that were evaluated using a quasi-experimental or experimental study design (Department of Health and Human Services, 2007).

The NREPP website contains an overview of each program selected, with a brief summary of its effects on targeted outcomes (called “key-findings”), a rating of the quality of the research, a description of the samples used in the evaluations, and a list of the research reports that were reviewed by NREPP in making its determination (National Registry of Evidence-based Programs and Practices, 2006). In the case of the West Coast ALERT evaluation, two of the three publications that report data on behavioral outcomes were reviewed (Ellickson & Bell, 1990a; 2003), along with a publication describing effects on knowledge and attitudes (Ellickson et al., 1993b) and a RAND report describing the design of the evaluation (Ellickson et al., 1988). For the South Dakota evaluation, just two publications are cited in the material reviewed: one is the journal article describing 8^th grade follow-up results (Ellickson et al., 2003) and the other a publication that reported effects on risk factors but not actual drug use (Ghosh-Dastidar et al., 2004). Although the re-review was conducted in December of 2006, there is no mention in the NREPP discussion of replications of the independent Pennsylvania evaluation, the results of which were published the previous year (St. Pierre et al., 2005).

The summary of the “key findings” from the evaluations of ALERT that appears on the NREPP website states that: “In multiple randomized control group studies, Project ALERT produced lasting outcomes for participants from a variety of ethnic and economic backgrounds who were at low, moderate, or high risk of alcohol, tobacco, or marijuana use” (National Registry of Evidence-based Programs and Practices, 2006, emphasis added). Results from only two evaluations are in fact presented in the summary (the West Coast and South Dakota studies), although the impression of multiple studies is presented by describing pulications that report data from a single evaluation project as separate “studies”. And while it is true that there were some effects on alcohol, tobaco and marijuanan use found in the various subgroup analyses that were conducted in the two studies, only those among a group of 163 high-risk females were found to be “lasting”.

The NREPP key findings summary of ALERT simply focuses on a handful of positive effects that emerged from the multiple subgroup analyses used in the studies. For the West Coast evaluation this focus is on the the marijuana non-user subgroup. ALERT participants in this subgroup are said to be nearly 50% less likely than controls to become current marijuana users by 8th grade, a figure that increased to more than 60% following the 8th-grade booster sessions. The former presumably refers to the difference at the 12-month follow-up of 1.6% versus 3.2% between the ALERT Health Educator group and the control group on the past-month measure of use. By the 15-month follow-up, the difference was 1.4% versus 3.7% (that is, about 60% lower in the ALERT Health Educator group than the control group). It is also noted in the summary on the NREPP website that Project ALERT participants were 30% less likely than other students to begin using marijuana (National Registry of Evidence-based Programs and Practices, 2006). This presumably refers to the difference at the 15-month follow-up of 8.3% versus 12.1% between both ALERT groups and the control group on the measure of initiation (i.e., “ever” used the drug). No mention is made of the fact that these few positive results emerged from a set of subgroup analyses that involved multiple comparisons between the two ALERT conditions and the control group. Nor is any mention made of the fact that even these few differences between the conditions were gone by the ninth grade follow-up.

The summary of the South Dakota evaluation also focuses on a few specific positive results from the earlier part of the evaluation, in this case those found for the alcohol misuse measures and for those who reported use of drugs at baseline. Thus, it is stated that participants in the revised ALERT curriculum were less likely to report alcohol-related consequences, and that the program “also had preventive effects on alcohol, cigarettes, and marijuana use among students who had tried these substances by 7^th grade” (National Registry of Evidence-based Programs and Practices, 2006). It is also stated that alcohol use was reduced by 24% in the ALERT group, although the published account of the evaluation states that “the program did not curb alcohol initiation or current use in the combined sample” (Ellickson et al., 2003, page 1833).

As noted above, while such very precise descriptions of isolated findings from the two ALERT evaluations may be factually correct, they cannot be considered as a thorough, or even a representative, summary of the results of the evaluations. However, presenting a positive result pertaining to the use of one drug from one set of analyses and another result pertaining to a different drug from another set of analyses, while making no menton of the fact that each result emerged from a process that involved making numerous compaisons between the study conditions, easily creates the impression that the program can produce a wide range of enduring differences in drug use between ALERT particpants and controls.

7. Conclusions

Project ALERT is among the most widely advocated evidence-based drug prevention programs in the United States. The review of the available evaluation studies presented herein indicates, however, that it is largely ineffective as a universal program in preventing alcohol, cigarette and marijuana use. Isolated statistically significant results have been found in the evaluations of the program, but the preponderance of evidence pertaining to all of those who receive ALERT (as opposed to subgroups of the total sample) indicates that it has only a few short-term effects on drug use behaviors and no long-term effects at all. As for the subgroup analyses, even if one ignores the analytic problems that these give rise to, there is little consistency in the results across studies, making it difficult to use these to target the intervention at specific subsets of individuals.

Beyond what it tells us about the evaluation of Project ALERT, this paper adds to the emerging literature in two areas of scholarship that have begun to raise concerns about the impact that the evidence-based movement is having on research and policy development in the drug prevention field. First, it is among a growing number of case studies that indicate that program evaluation research in the area of drug prevention is frequently not informed by a critical-rational and objective orientation (Brown, 2001; Gorman, 2002; 2005; Gorman, Conde & Huber, 2007; Littell, 2005; Midford, 2008). Rather such research functions to prove the hypothesis that the intervention program works. If the achievement of this goal requires multiple subgroup analyses or the use of other questionable data analysis and presentation practices, then so be it (Gorman, 2003). Second, it indicates that, like its predecessor and many other drug prevention evidence-based practice lists, the review criteria and procedures used in the revised NREPP place far too much emphasis on isolated statistically significant findings (however, they are produced) while ignoring the preponderance of evidence indicating that a program has no effect (Gandhi et al., 2006; Gorman, 2002; Weiss et al., 2008). Setting the minimal requirement for “evidence-based” at just one positive behavioral outcome will result in the inclusion in the registry of largely ineffective programs, such as Project ALERT. And, cherry-picking isolated positive results to include in program summaries is hardly consistent with SAMHSA’s claim that the NREPP system is “designed to provide the public with reliable information on the scientific basis and practicality of interventions” (Department of Health and Human Services, 2007, p. 30815).

If the NREPP process truly is going to provide potential users with reliable and unbiased information about the scientific basis of the interventions that it recommends, it must make available all of the relevant evidence pertaining to these, not just a selection of research reports provided by program developers. It must also revise its review criteria such that some consideration of the manner in which statistically significant results are produced by evaluators is included in the assessment of the quality of the science. Specifically, findings that emerge from multiple subgroup analyses (especially when these involve no statistical adjustment for multiple comparisons between study conditions) should be interpreted with caution by NREPP reviewers. The reviewers should also attend more closely to the variables that are reported in published accounts of an evaluation – specifically, is there consistency in the way that outcomes are operationalized or do these change from one report to another, and are truly independent variables being reported or is the same finding being presented in different ways through the use of composite measures or multiple variable labels. Also, if NREPP is to continue to designate programs as “universal” or “selective” it must be clear what it means by these terms. For example, if an intervention is said to be effective as a selective program, then it must be stated exactly which subgroups it is effective for and this recommendation must be grounded in empirical evidence from evaluations demonstrating that the intervention produces this effect with some consistency. The same applies to use of the term “universal program”. If this is to be applied to an intervention, then there should be a consistent body of research demonstrating that the program has widespread effects within a broad target population.