WHO BENEFITS MOST FROM A BROADLY TARGETED PREVENTION PROGRAM? DIFFERENTIAL EFFICACY ACROSS POPULATIONS IN THE TEEN OUTREACH PROGRAM

Abstract

This study examined a highly successful, well-documented, national program to prevent teenage pregnancy and school failure—the Teen Outreach program—to address a fundamental question: How well can a developmentally focused, broadly targeted prevention program address the needs of those students within the program who are at the highest risk of problematic behavior. The hypothesis that the developmental focus of a broadly targeted intervention would lead it to have greater program efficacy among those young people who began the program at greatest risk was examined with multisite data collected on more than 3,300 Teen Outreach and comparison group students. Results confirmed prior findings regarding the overall efficacy of the Teen Outreach program, and indicated that the program appeared most effective for those students at greatest initial risk of the problem behaviors being targeted. Implications for the targeting of the Teen Outreach program specifically and of similar primary prevention programs more generally are discussed.

INTRODUCTION

Although tremendous effort has been expended in advancing the state of the art of program evaluation, it is now widely recognized that understanding the conditions under which a program works best may be as important as determining whether a program works at all (Allen, Philliber, & Hoggson, 1990; Allen, Kuperminc, Philliber, & Herre, 1994; Gray & Braddy, 1988). Even successful preventive intervention programs are unlikely to work equally well with all individuals they serve. As a field, however, we have been remiss in studying the conditions under which successful programs do work. This is in part because we lack data that can inform the question of whether broadly targeted successful preventive interventions work equally well with all of the individuals they serve.

This report addresses this question using national evaluation data from a highly successful preventive intervention targeted toward reducing pregnancy and school failure rates among high school-aged youth. It both assesses how well one national prevention program works with different populations and considers the specific hypothesis that even a broadly targeted developmentally focused prevention program may well have its greatest impact on youths who are struggling most in their development.

One of the most difficult transitions in the life course of large-scale preventive interventions is moving from the piloting/program development phase into large-scale dissemination (Blakely et al., 1987; Zigler & Muenchow, 1992). One obstacle faced by programs is the frequent need to work with populations that may differ from those that the program originally served. In efforts to prevent teen pregnancy and school failure, the question of differential program efficacy across different sociodemographic groups has received attention in several studies. Unfortunately, analyses of such Program × Recipient interaction effects have typically been done almost as an afterthought, following an absence of findings of program efficacy across an entire sample, in an effort to find some positive evidence of program efficacy within subgroups of those served (see review by Kirby, 1997). Although such analyses may be useful in finding value in the midst of otherwise disappointing evaluation results, we may actually have the most to learn about how programs work with different populations when we examine the differential efficacy of a clearly successful program across different populations.

The Teen Outreach program is a national pregnancy and school failure prevention program serving over 6,000 young people annually in sites in dozens of states, which has recently received both scholarly attention (Allen, Kuperminc, & Moore, 1997a) and popular attention as representing “the best evidence we have that social programs can prevent teen pregnancy” (Kirby, 1997). Initially established in 1978, Teen Outreach is a school-based program that involves young people in volunteer service in their communities. The program links this volunteer work to classroom-based, curriculum-guided group discussions on a wide range of issues, from family conflict to human growth and development. This combination places students in a help-giving (as opposed to help-receiving) role, and research to date on the program suggests that it works from an empowerment perspective by enhancing students’ sense of autonomy while helping them maintain a sense of relatedness with peers, facilitators, and other adults (Allen et al., 1990; Chinman & Linney, 1998; Rappaport, 1987; Riessman, 1965). Most importantly, the program has been shown to be highly effective in reducing rates of teen pregnancy, and school failure and suspension when evaluated nationally using a rigorous experimental design (Allen, Philliber, Herrling, & Kuperminc, 1997b). Consistently over a 10-year period, evaluation data including both random and nonrandom designs have shown reductions in pregnancy and failure rates of 30–50% among high school students who participated in the Teen Outreach program (Allen et al., 1990; 1997a).

The need for successful programs addressing teen pregnancy and school difficulties is tremendous. Teenage pregnancy, school failure, and school dropout result in enormous costs each year, both to individual adolescents and to the larger society (Center for Population Options, 1992; Furstenberg, 1991; Maynard, 1996; Wm. T. Grant Commission, 1988). Efforts have increased to prevent these problems, often via large-scale, school-based interventions (Dryfoos, 1990). Yet, recent evidence suggests mixed results to date, with some programs showing signs of real success and others with less clear outcomes (Allen et al., 1997; Conduct Problems Prevention Research Group, 1992; Dryfoos, 1991; Frost & Forrest, 1995; Kirby, 1997; Kirby et al., 1994).

There is an equally pressing need to understand the range and limits of competence-enhancing prevention programs such as Teen Outreach in working with youths of different backgrounds. In part, such information is needed to assure that promising programs are implemented with populations for whom they are most likely to be effective. A broader question, however, is whether nonclinical interventions that are broadly targeted can actually reach high-risk youth. For example, does Teen Outreach, which focuses on volunteer activities and community service, actually work for at-risk youths who might seem to have little interest in volunteering to help others when their own lives were foundering? It could be argued that a broadly targeted intervention, particularly one that often functions as a primary prevention program, is unlikely to be equipped to deal with the needs of very high-risk youth.

In contrast, a developmental perspective suggests a far more positive answer to the question of whether broadly targeted developmental interventions can address the needs of high-risk youth. To the extent that prevention programs meet developmental needs that are not met elsewhere, such programs may even have their greatest impact with high risk youth. Research on the multiplicative nature of risk factors suggests that eliminating even a single risk factor in childhood and adolescence can in some cases bring about large reductions in problem behaviors (Allen et al., 1997a; Rutter, 1979). Findings from one of the few studies to consider the moderating effect of family status on young mothers’ gains in an early childhood home visitation program further support the notion that the most demographically at-risk adolescents may be poised to make the greatest gains from preventive interventions (Cole, Kitzman, Olds, & Sidora, 1998). From this perspective, youths who were already demonstrating problems might be most able to profit from having their developmental needs met by a program such as Teen Outreach. These youths might still struggle under multiple risk factors, but the gains they may make in their development may be enough to bring about significant behavioral changes. To date, the question of how well broadly targeted developmentally focused prevention programs will work with high-risk vs. low-risk populations has not been subjected to thorough empirical scrutiny.

The implications of the answers to this basic question are enormous. For Teen Outreach, which has been one of the most successful and carefully evaluated broad-based pregnancy prevention programs, we need to know how likely it is to be successfully replicated with high-risk populations. Understanding whether the program is equally effective with different demographic groups of students represents an important challenge in learning to appropriately target this large-scale national program. For successful competence-enhancing prevention programs in general, there is a broader question as to whether it may make theoretical sense, or even be optimal under some conditions, to expand into the domain of early intervention and begin to target at-risk children and adolescents, as Cowen (1996, 2000) notes that primary prevention programs are often pushed to do. Because of the scale of the Teen Outreach program, spanning multiple sites and multiple cities, it provides an ideal case to begin to examine the degree to which the effectiveness of a prevention program is likely to vary when offered to different groups of students under different conditions beyond the pilot program phase of program development and dissemination.

This study thus utilized data collected over a 4-year period across over 60 sites nationwide to examine: (a) whether this broad-based competence-enhancing intervention may be most efficacious when serving higher risk adolescents, assessed in terms of both familial risk factors and behavioral risk factors; and (b) whether the program appears equally effective across different sociodemographic groups of young people.

METHOD

Settings

The Teen Outreach Program

Teen Outreach is a program offered to high school age students that consists of three interrelated elements: supervised community volunteer service, classroom-based discussions of service experiences, and classroom-based discussions and activities related to key social-developmental tasks of adolescence.

Teen Outreach participants, who were in grades 9 through 12, engaged in a range of volunteer activities supervised by trained staff often working in conjunction with staff and volunteers of local community organizations. Volunteer activities were selected by participants under the supervision of trained staff and adult volunteers to be sensitive to the needs and capacities of both participants and of local communities. These experiences varied substantially in their nature and in the amount of commitment required of participants. Volunteer activities included work as aides in hospitals and nursing homes, participation in walkathons, peer tutoring, and a wide range of other types of work. Teen Outreach sites were expected to provide a minimum of 20 hours/year of volunteer experience to participants. Participants in the program actually averaged 37.4 hours of volunteer service over the course of the program (SD = 54.9), with the median participant performing 25 hours of service over the course of the program.

In addition to volunteer service, students also participated in ongoing classroom-based discussions that occurred at least once weekly throughout an academic year. All classroom discussions and activities were based upon the Teen Outreach Curriculum (Edwards, Bell, & Hunter-Geboy, 1996), which is designed to engage students via structured discussions, group exercises, role plays, guest speakers, and informational presentations. Classroom discussions and activities focused either upon maximizing learning from the service experiences, or upon helping teens cope with important developmental tasks they faced. Service-learning discussions focused upon helping students prepare for and make plans about their service experiences (including dealing with a lack of self-confidence, social skills, assertiveness, self-discipline, etc.). They are encouraged to think about what they have experienced while volunteering, and listen to others do the same.

In developmentally oriented classroom discussions and activities, a trained facilitator lead small groups of students in activities and topics of particular interest and relevance to young people. Topic areas included: understanding yourself and your values, life skills, dealing with family stress, human growth and development, and issues related to the social and emotional transitions from adolescence to adulthood. In covering these topics, facilitators encouraged students to discuss their feelings and attitudes about important developmental issues (e.g., managing family relationships, new academic and employment challenges, handling close friendships and romantic relationships, etc.). Facilitators were given considerable latitude in covering topics in the curriculum, and for each topic, the curriculum contains a wide array of activities and materials for discussion, so that facilitators may choose those that appear most appropriate for their group. Most strikingly, and in keeping with its broad developmental focus, the program placed very little direct emphasis upon the two target behaviors to be prevented. Specifically, material about sexuality comprises less than 15% of the written curriculum, and was often not used (discretion is given to individual sites) when it overlapped with other material being offered in school or conflicted with prevailing community values.

Classroom discussions were led by trained facilitators, who were often school teachers or guidance personnel. Classroom sessions met at least once weekly during a full academic year, beginning in September and running through May or June (depending upon the school schedule of individual sites). Most sites held once weekly meetings, although a few met two to three times/week. Prior research has found no evidence of differential program impact for sites varying in numbers of class sessions within this range (Allen et al., 1990).

Cost analyses of the program indicate that it can be offered for a full academic year to a class of 18 to 25 students for approximately $500 to $700 per student. These figures include costs for facilitator and site-level coordinator time; when this time is provided as an in-kind contribution by schools and community volunteer service organizations, the direct costs of the program drop to under $100 per student.

Comparison students were selected in one of three ways. For a subset of approximately 20% of students (N = 660), random assignment to treatment and control groups was used. In other cases, either Teen Outreach students nominated other students whom they guessed “would fill out the entry questionnaire about the same way [they] did,” or facilitators or guidance counselors responsible for selecting Teen Outreach students also selected students from similar sociodemographic backgrounds and with similar levels of problems as Teen Outreach participants. We included both randomly and nonrandomly assigned students because: (a) given the low base rate of teen pregnancy and our interest in examining moderators (i.e., interaction effects) the random sample would be far too small to provide adequate power; (b) we were not primarily interested in comparing treatment vs. control/comparison groups, thus reducing the centrality of randomization; (c) both randomly and nonrandomly assigned groups had similar effect sizes. Nonetheless, we present analyses below that consider the possible impact of this sampling variable on all substantive outcomes of interest.

Participants

Participants in this evaluation of Teen Outreach included 1,673 students who participated in the Teen Outreach Program and 1,604 comparison students. All students were in the ninth through 12th grades (mean grade level = 10.0, SD = 1.1). Students sought to enter the program through a variety of means: some as part of their “health” curricula; some as an academic elective; some via teacher/guidance counselor suggestion, and some as an after-school activity. A small number of participants (5.9%) had been previously involved with the Teen Outreach Program.

Attrition over the course of the study, as a result of student dropout from the program or from school, or from failure to fully complete exit questionnaires was 8.9% among Teen Outreach students and 7.8% among comparison students. Attrition analyses revealed that students who did not complete the study were more likely to have had or caused a prior pregnancy, been suspended and failed courses previously. They were also more likely to be younger, male, members of a racial/ethnic minority group, and living in a single parent household. Although dropouts from the study differed from those who completed it, analyses revealed no evidence that these differences were in any way linked to treatment vs. comparison group membership (i.e., there was no interaction of attrition and group membership). For example, previously pregnant students were less likely to complete the study, but the previously pregnant Teen Outreach participants were no less likely to complete the study than were previously pregnant comparison group members).

Information on the demographic characteristics of both Teen Outreach and comparison students for whom both entry and exit data were available is presented in Table 1. These data indicate that the treatment and comparison groups were well matched demographically at entry, with very slight differences in the racial and gender composition of the groups, and no differences on any measures of demographic risk or of prior problem behaviors. These data also indicate that the program sampled an ethnically and socio-economically diverse group of students.

Table 1.

Sociodemographic Characteristics of Teen Outreach and Comparison Students at Entry

	Teen Outreach N = 1673 Mean (SD)	Control N = 1604 Mean (SD)
Age (years)	15.9 (1.20)	16.0 (1.24)
Grade in School
9th	39.6%	39.3%
10th	28.0%	28.2%
11th	18.2%	18.6%
12th	14.2%	13.9%
Gender
Females	24.6%	29.1%**
Males	75.4%	70.9%
Race/Ethnicity
Black	44.3%	46.1%*
White	38.1%	35.4%
Hispanic	12.6%	12.9%
Other	5.0%	5.6%
Mother’s Education Level (1—not a h.s. graduate; 4—college graduate)	2.29 (0.95)	2.31 (0.93)
Live in two-parent household	54.1%	56.4%
Prior suspensions	18.7%	19.1%
Prior course failures	31.7%	31.9%
Prior pregnancies	7.8%	8.6%
Teenage parenthood	3.1%	3.6%

Note: **p < 1; *p <.05.

Measures

Demographic Characteristics

Students filled out a brief self-report questionnaire indicating their age, grade level in school, race, predominant household composition (one vs. two parent) and parents’ education levels (1—not a high school graduate; 2—H.S. graduate; 3—some college; 4—college graduate).

Problem Behaviors

Self-report questionnaires were used to assess students’ problem behaviors. When sensitively collected, anonymous self-report instruments have been found to be among the least biased means of assessing adolescent problem behaviors such as teenage pregnancy, with substantial evidence available to support their overall reliability and validity (Elliott, Huizinga, & Menard, 1989; Farrington, 1973; Patterson & Stouthamer-Loeber, 1984). At entry, we asked students: (1) whether they had ever been pregnant (females) or caused a pregnancy (males); (2) whether they had failed any courses during the prior year at school; and, (3) whether they had been suspended in the prior year at school. At exit, we asked the same questions of students (except that the pregnancy question was modified to refer only to the academic year of the program). Prior research suggests that information provided by student reports in this program is generally accurate, and that any inaccuracies are unbiased with respect to student’s participation in the program vs. comparison groups (Allen et al., 1997b).

Procedure

Both the Teen Outreach Program and its evaluation typically occurred as part of the regular school curriculum over a single school year, with participation usually occurring as part of a class (typically health or social studies) taken for credit. Students were assessed during the first several weeks of the school year (which typically began in late August or early September), and then again at program exit at the end of the school year (i.e., in May or June). Questionnaires were administered by Teen Outreach facilitators during an early Teen Outreach class, or in study halls and other school settings for comparison students. Students were told that the information they provided would be kept strictly confidential, and that only identity codes would be included on their questionnaires, and they were specifically reassured that none of their answers would be available to program facilitators or to other school officials, and that no data which in any way identified them would be reported.

RESULTS

Preliminary Analyses

Effects of Program Participation

The effect of participation in the Teen Outreach program was assessed for each of the three targeted problem behaviors in three logistic regression equations. Each equation assessed the extent to which the incidence of a target behavior could be predicted (after accounting for student background factors, entry levels of problem behavior, and demographic characteristics) from a dummy variable that coded program participation vs. control group membership. These analyses, outlined in detail in the sections below, indicated that participation in the Teen Outreach program was related to significant reductions in levels of pregnancy, course failure, and school suspension.

Further analyses assessed whether the magnitude of the observed effects would differ depending upon whether students were randomly vs. nonrandomly assigned into comparison groups. Given that the sample size of the random assignment sample by itself was too small to provide adequate power to assess possible interactions of program participation with student background factors in predicting dichotomous outcomes of interest, comparison of outcomes for students who were vs. were not randomly assigned provides a basis for assessing the impact of including both randomly assigned and comparison group students in analyses below. These analyses revealed no effects of membership in the randomly assigned vs. nonrandomly assigned groups for any of the outcomes observed.

Further analyses examined whether random assignment interacted with any of the effects of interest in this study in predicting outcomes. For these analyses, we assessed the interaction of a random assignment dummy variable (for whether an individual was or was not from a randomly assigned group) with all of the primary effects of interest in the study (e.g., program participation and all two-way interaction variables). This approach directly tests the proposition that the random assignment factor may have altered the results presented in primary analyses below. These analyses revealed that these blocks of interactions of random assignment with other effects of interest added very little to the explained variance in outcomes [changes in 0-square for the addition of 8 degrees of freedom (program participation, plus Steps II–IV in the models shown in Tables 2 through 4 below) were 4.3, 7.0, and 12.0, for pregnancy, course failure, and suspension, respectively, all nonsignificant). These findings indicate that the results presented below do not significantly differ across the two subsamples.

Table 2.

Predicting Pregnancies at Program Exit from Background Factors, Program Participation, and the Interaction of Program Participation × Background Factors

	Pregnancies at Program Exit
Predictors	β	Odds ratio	χ2	Δχ2
Step I. main effects
Demographic background
Grade	.01
Sex	.08
Minority	.17***	1.88
Proximal behavioral risks
Prior pregnancy	.48***	25.01
Teenage parenthood	−.05
Demographic risks:
Parents education level	−.11
Single-parent household	.05
Program participation	−.17***	0.53
Summary statistics for step I			375.0***	375.0***
Step II. Interactions with demographic background
Prog. participation × grade	−.02
Prog. participation × gender	.05
Prog. participation × minority	−.02
Summary statistics for step II			376.3***	1.3
Step III. Interactions with behavioral risks
Prog. participation × prior pregnancy	.03
Prog. participation × teen parenthood	−.09**
Stratified odds ratios for prog. partic. for:
Nonteenage parents		0.66*
Teenage parents		0.18***
Summary statistics for step III			386.4***	11.1**
Step IV. Interactions with familial risks
Prog. participation × par. educ. level	−.01
Prog. participation × single-parent household	.02
Summary statistics for step IV			386.6***	0.2

Note: ***p < .001; **p < .01; *p < .05. β weights and odds ratios are from variables entry into model. Odds ratios are presented only for significant βs.

Table 4.

Predicting Academic Suspension at Program Exit from Background Factors, Program Participation, and the Interaction of Program Participation × Background Factors

	Academic Suspensions by Program Exit
Predictors	β	Odds ratio	χ2	Δχ2
Step I. Main effects
Demographic background
Grade	−.11***	0.82
Sex	−.13***	0.59
Minority	.08**	1.34
Proximal behavioral risks
Prior course failures	.13***	1.63
Prior school suspensions	.39***	6.19
Demographic risks:
Parents’ education level	.00
Single-parent household	.11***	1.50
Program participation	−.18***	0.52
Summary statistics for Step I			533.1***	533.1***
Step II. Interactions with demographic background
Prog. participation × grade	.05
Prog. participation × gender	.01
Prog. participation × minority	−.02
Summary statistics for step II			535.9***	2.8
Step III. Interactions with behavioral risks
Prog. participation × prior course failures	.02
Prog. participation × prior suspensions	.04
Summary statistics for step III			538.6***	2.7
Step IV. Interactions with familial risks
Prog. participation × par. educ. level	.01
Prog. participation × single-parent household	−.02
Summary statistics for step IV			539.1***	0.5

Note: ***p > .001; **p > .01. β weights and odds ratios are from variables entry into model. Odds ratios are presented only for significant βs.

Primary Analyses

Primary analyses for this study assessed whether the effects of program participation interacted with any of students’ demographic, behavioral, or familial characteristics examined in predicting behavioral outcomes. These analyses address the question: Did the Teen Outreach Program lead to better or worse results for students with different background characteristics? Each problem behavior was assessed separately using hierarchical logistical regression analysis. In the first block of the equation, main effects of student background characteristics and program participation were entered. These were followed by three blocks of interaction factors to consider interactions of program participation with: (a) student demographic characteristics, (b) prior proximal behavioral risk factors; and (c) prior familial risk factors. Finally, given the nested nature of the data (i.e., students with classes), multilevel modeling procedures were employed to assess whether these nesting effects altered any of the results obtained from hierarchical logistic analyses. These results are presented separately for each problem behavior below.

Teen Pregnancy

Initial examination of the main effects predictors of pregnancies by program exit revealed that membership in a racial/ethnic minority group and prior history of pregnancy were significant background factors related to pregnancy by program exit. After accounting for these factors, program participation was also a significant predictor, with students participating in Teen Outreach demonstrating only 53% the risk of pregnancy as demonstrated by students in the Comparison group. This main effect of program participation is quite similar in magnitude to the previously reported effects obtained in analyses of just the randomly assigned students, who were a subset of the current sample (Allen et al., 1997). These analyses are presented in Table 2.

Examination of the interaction of participation with student entry characteristics revealed that the program was not significantly more or less effective in preventing pregnancies for students of different grades in school, different genders (either becoming pregnant or causing a pregnancy), or for students from racial/ethnic minority vs. majority groups.

Primary analyses of treatment × student characteristic interactions revealed a significant effect for students who were vs. were not teen parents. Examination of the two groups separately revealed that the program continued to have significant positive effects within both the teen parent and nonteen parent groups, but Teen Outreach program effects were found to be significantly stronger for the teen parents in the program. Nonteenage parents in the program had odds ratios of pregnancy of approximately two-thirds the ratios of comparison group students (reflecting the program’s overall efficacy). In contrast, teenage parents in the program had only approximately one-fifth the risk of pregnancy (odds ratio = .18) relative to teenage parents in the comparison group. These results are depicted in Figure 1.

Figure 1.

Pregnancy × teen parenthood interaction: nonteenage parents (left), and teenage parents (right).

No interactions of program participation with familial risk factors were observed, indicating that the program was not significantly more or less effective in preventing pregnancies for students from families with more vs. less educated parents or from one- vs. two-parent families.

Course Failure

Initial examination of the main effects predictors of course failure by program exit revealed that course failure was significantly related to being in younger grades in high school, history of prior course failures and academic suspensions, lower levels of parents’ education, and residence in a single-parent family. After accounting for these factors, program participation was also a significant predictor, with students participating in Teen Outreach demonstrating only 60% the risk of course failure as demonstrated by students in the Comparison group, a finding quite similar to that previously reported based just on the randomly assigned samples (Allen et al., 1997b). These analyses are presented in Table 3.

Table 3.

Predicting Course Failures at Program Exit from Background Factors, Program Participation, and the Interaction of Program Participation × Background Factors

	Pregnancies at Program Exit
Predictors	β	Odds ratio	χ2	Δχ2
Step I. main effects
Demographic background
Grade	−.15***	0.77
Sex	−.04
Minority	.07**	1.30
Proximal behavioral risks
Prior course failures	.44***	5.46
Prior school suspensions	.12***	1.76
Demographic risks
Parents education level	−.08***	0.85
Single-parent household	.08***	1.35
Program participation	−.14***	0.60
Summary statistics for step I			671.5***	671.5***
Step II. Interactions with demographic background
Prog. participation × grade	−.02
Prog. participation × gender	.06*
Stratified odds ratios for prog. partic. for:
Females		0.52***
Males		0.83
Prog. participation × minority	.05*
Stratified odds ratios for prog. partic. for:
Racial/Ethnic Majority Students		0.74*
Racial/Ethnic Minority Students		0.52***
Summary statistics for step II			683.2***	11.7*
Step III. Interactions with behavioral risks
Prog. participation × prior course failures	.03
Prog. participation × prior suspensions	−.05*
Stratified odds ratios for prog. partic. for students:
with no prior suspensions		0.66*
with prior suspensions		0.43***
Summary statistics for step III			690.2***	6.9**
Step IV. Interactions with familial risks
Prog. participation × par. educ. level	−.05*
Prog. participation × single-parent household	−.00
Summary statistics for step IV			694.7	4.5

Note: ***p < .001; **p < .01; *p < .05. β weights and odds ratios are from variables entry into model. Odds ratios are presented only for significant βs within significant blocks of variables.

Examination of the interaction of participation with student entry characteristics revealed that Teen Outreach had significantly higher levels of efficacy in preventing course failure for females and for members of racial/ethnic minority groups. These results are depicted in Figure 3.

Figure 3.

Course failure × prior suspensions interaction: students with no suspension history (left), and students with prior suspensions (right).

Primary analyses of treatment × student characteristic interactions revealed a significant effect for students who did vs. did not have a history of prior suspensions. Although examination of the two groups separately revealed that the program continued to have significant positive effects within both the previously suspended and nonsuspended groups, program effects were significantly stronger for the previously suspended students. Teen Outreach students with no suspension history had about two-thirds the risk of suspensions of similar students in the comparison group, whereas Teen Outreach students with prior suspension histories had slightly less than half the risk of suspension when compared to students with similar histories in the comparison group. These results are also depicted in Figure 3.

The block of familial risk factors did not significantly contribute to program outcomes; thus, no interpretations were made of marginally significant effects within this block.

Academic Suspension

Initial examination of the main effects predictors of academic suspension by program exit revealed that academic suspension was significantly related to being in younger grades in high school, history of prior course failure and academic suspension, lower levels of parents’ education, and living in a single-parent family. After accounting for these factors, program participation was also a significant predictor, with students participating in Teen Outreach demonstrating only 52% the risk of academic suspension as demonstrated by students in the comparison group, a finding quite similar to that previously reported based just on the randomly assigned samples (Allen et al., 1997a). These analyses are presented in Table 4.

No interactions of program participation with blocks of student demographic characteristics, proximal behavioral risk factors, or familial risk factors significantly added to prediction of outcomes, indicating that the program was not significantly more or less powerful in preventing academic suspensions when offered to students with different background characteristics.

Alternative Attrition Assumptions

We next examined what might happen to our results if we made assumptions that those individuals lost to attrition had each of several different levels of problem behaviors, and included these imputed values in analyses. This is in contrast to the data presented above which analyzes only actual data on those individuals available at pre- and posttest. We produced several sets of hypothetical imputed data, using various assumptions about the level of problem behaviors in the attriting groups. This is a simplified version of the “modeling our ignorance” approach to understanding effects of attrition proposed by Delucchi (1994). First, an alternative data set was created in which the attriting groups were included in the analyses and assumed to have the same base rates of problem behavior as those actually followed at posttest (actual assignments of presence vs. absence of problem behaviors were made using a random number generator). A second data set was created next in which attriting groups were assumed to have twice the base rate of problem behaviors of those remaining in the study. This is plausible, given that problematic behaviors, such as school failure, could have been one source of attrition. Finally, data sets were created in which the attriting group was assumed to have had four times and eight times the rates of problem behavior of those remaining in the study, up to a 100% rate. Together, these analyses yield one source for estimating the likely sensitivity of results presented above to attrition within the data set.

Analyses of each of these data sets indicated that for pregnancy, all of the participation and interaction effects described as significant above remained significant. For course failure, all effects remained significant when attriters were assumed to have twice the failure rate (64%), and all but one (the interaction of program participation and parents’ education) remained significant when attriters were assumed to have had a 100% failure rate. No new effects were detected as significant for any of the analyses conducted. Overall, these findings provide evidence that the results of primary analyses above were only slightly sensitive to effects of attrition within the study.

Multilevel Analyses Accounting for Nested Data

Because students were nested within classes in this study, the final set of analyses were performed to determine whether accounting for this nesting would substantially alter the findings reported above. Multilevel procedures were used for these analyses. This approach could not be used for the primary analyses presented, because, with low-frequency dichotomous outcome data, the full models described above did not yield converging solutions even when intensive computational resources (e.g., supercomputer facilities) were applied. However, it was possible to obtain converging solutions for slightly simplified models in which both fixed and random effects were estimated for the primary outcomes of interest in the study (e.g., participation in Teen Outreach, all significant interactions of participation with other factors, and the main effects of the specific factors from those interactions) and just fixed effects were estimated for all other variables. In essence, this approach examines whether taking nesting of students into account (i.e., as random effects in multilevel models) for the significant findings presented above alters those findings (i.e., fixed effects). Each of these effects remained at the same level of significance when considered together with their parallel random effects in models, with the exception of the interaction of participation × minority status in predicting course failure, which slipped to the trend level, and participation × student gender in predicting course failure, which rose to the .01 level of significance. This indicates that multilevel models that take into account the variation of the significant parameters described in primary analyses above across and within schools (albeit in imperfect models) yield highly similar results to those presented in the primary analyses.

DISCUSSION

These findings make clear that even a broadly successful program such as Teen Outreach may have substantially different levels of effectiveness when serving different populations of young people. Further, these differences were not random in nature, but suggested that even a competence-enhancing program such as Teen Outreach that is targeted at broad populations of young people may have its greatest effectiveness with those young people who are at greatest risk for the problems the program seeks to prevent. These findings have implications both for this highly promising approach to preventing teen pregnancies and school failures, and for a developing technology of preventive interventions.

The most striking finding was that Teen Outreach appeared most effective as a prevention program with youths who were most at-risk of the specific type of problem behaviors being assessed. The program had the greatest impact in reducing future pregnancies among the group at highest risk of such pregnancies (those who have already given birth to a child) (Kirby, 1997). For this group, the likelihood of an additional pregnancy was less than one-fifth as large in the Teen Outreach group as in the comparison group, even after accounting for other background factors that may have also affected pregnancy rates. For academic failure, Teen Outreach demonstrated greater efficacy for youths who had been previously suspended than for those who had not. The program was also found to be more effective for members of racial ethnic minority groups, who were also at greater risk for academic difficulty in this study.

Although these findings are consistent with existing theory, it certainly could have been the case that a program that was oriented toward primary prevention and that sought to build competence rather than addressing specific deficits might have most benefitted those who were already best off to begin with—i.e., the rich getting richer. The actual findings, in contrast, are more in keeping with a nutritional metaphor of prevention. Just as additional food provides the greatest gains in growth to children who are below their normal growth trajectory due to prior malnutrition, Teen Outreach appeared to provide the greatest gains toward a normal developmental trajectory for youths who were initially the farthest off from this trajectory.

One explanation is that Teen Outreach is enhancing development by providing opportunities for adolescents to establish their autonomy and competence in a context that also allows them to remain engaged (and receive encouragement from) their schools and communities (Allen et al., 1994). Disengagement from school is a robust correlate of risk of both academic problems and pregnancy in adolescence, and the failure of schools to provide opportunities for adolescents to establish autonomy within them has been identified as a likely cause of this disengagement (Eccles & Midgley, 1991; Eccles et al., 1993; Manlove, 1988). It may simply be that there is more work to be done, and more to be gained, in populations of at-risk youth where levels of disengagement are greatest, an explanation that would also apply to the findings of other prevention programs that have been shown to be more effective with higher risk subgroups (e.g., Price, Cowen, Lorion, & Ramos-McKay, 1989; Wolchik, West, Westover, & Sandler, 1993).

These findings raise the question of whether in some cases the greatest cost effectiveness might be achieved when programming targets clearly at-risk youths (some of whom may have already experienced a first instance of the problem to be prevented). This cost effectiveness argument would apply even if Teen Outreach were only equally effective across different groups—because a given percentage reduction in a behavior would translate into more instances of the behavior being prevented in high-risk groups. Yet, Teen Outreach is more effective within high-risk groups, further bolstering the cost effectiveness argument for targeted intervention. In this sense, these findings actually suggest that the best results from implementing the Teen Outreach program might occur when it is implemented as a targeted intervention, rather than as a primary prevention program. However, this statement must be interpreted in the context of recognizing that the program remained effective even in the low-risk groups examined, suggesting that both approaches to implementing the program (broad vs. narrow targeting of youth) are of value, and that the program is helpful to both at-risk and not-at-risk youths.

The findings of this study can also be taken as answer to a potential criticism that is made against primary prevention programs—that in their effort to attain breadth, they may not truly address the needs of those youths who are most at-risk of the problems they seek to prevent. A program such as Teen Outreach that encourages young people to perform volunteer work and to reflect upon their lives and values could be considered too genteel in nature to address the needs of high risk youths. The present findings strongly refute this notion. Teen Outreach worked equally well with youths from one- and two-parent families, and from those with more vs. less educated parents, and in some cases appeared more effective with those groups that might have seemed least likely to be receptive to its volunteer emphasis.

Several caveats must be considered in interpreting these findings. Most importantly, the relative paucity of data on differential effectiveness of large-scale, successful preventive interventions targeting adolescents does not give license to treat one instance of such data as reflecting the whole of such programs. The present findings clearly speak to the Teen Outreach program, and given the size of the program and attention it has received, are important in their own right. To the extent they say more, it is in demonstrating the plausibility of an important possible outcome of large-scale preventive interventions. Further, analyses suggested that biases in findings due to attrition or to nesting of data could have slightly altered several of the findings reported. From this perspective, it makes most sense to consider the overall pattern of findings—that the program works better for those at greatest risk upon entering it—which appeared in very similar form across alternative analytic methods. From this perspective, these data can be taken as a case study of one program (albeit one serving thousands of individuals scattered across dozens of communities nationally) that may demonstrate important points about the potential of preventive interventions, but that should not be taken as characterizing all such interventions.

Similarly, although some of the students in this study were randomly assigned between treatment and control group conditions (and program effects were similar regardless of whether random vs. nonrandom matching procedures were used), youths obviously could not be randomly assigned to different risk statuses. Thus, this study can only show that Teen Outreach achieved better results with higher risk youth, but cannot further clarify the nature or the mechanisms of any possible causal link between risk status and program efficacy. Nonetheless, these findings did not represent simple regression to the mean among at-risk youth. Although high-risk Teen Outreach students demonstrated relatively reduced risk of problem behaviors, high-risk students in the comparison group went on to have higher levels of problem behaviors by program exit. An additional limitation of these data is their reliance upon student self-reports of problem behaviors. Although prior research with a subsample of youth in this study has provided evidence that these reports are both accurate and unbiased with regard to program participation (Allen et al., 1997b), logistical considerations prevented this issue from being addressed with the sample as a whole.

Finally, although quite sizable differences in estimates of program efficacy across groups were often obtained, these differences should not obscure findings about the broad general effectiveness of the Teen Outreach Program. Overall, young people in the program had about one-half the rate of problem behaviors as comparison youth—a rate consistent with findings from controlled-experimental studies of this program (Allen et al., 1997b). For academic suspension, no differences across groups were found in program efficacy. For the other two problem behaviors, differences across groups were found, but even in the subgroups for which the program was least effective, the program always appeared to be operating in the direction of reducing problem behaviors, an effect that was statistically reliable in all but one case. Thus, although these data suggest there is much to learn about how best to target this broad-scale preventive intervention, they also suggest that even without targeting the program is quite effective in reaching a diverse array of youth.

Figure 2.

Course failure × gender interaction (top), and course failure × ethnicity interaction (bottom).