Evaluating Youth Development Programs: Progress and Promise

Abstract

Advances in theories of adolescent development and positive youth development have greatly increased our understanding of how programs and practices with adolescents can impede or enhance their development. In this paper the authors reflect on the progress in research on youth development programs in the last two decades, since possibly the first review of empirical evaluations by Roth, Brooks-Gunn, Murray, and Foster (1998). The authors use the terms Version 1.0, 2.0 and 3.0 to refer to changes in youth development research and programs over time. They argue that advances in theory and descriptive accounts of youth development programs (Version 2.0) need to be coupled with progress in definitions of youth development programs, measurement of inputs and outputs that incorporate an understanding of programs as contexts for development, and stronger design and evaluation of programs (Version 3.0). The authors also advocate for an integration of prevention and promotion research, and for use of the term youth development rather than positive youth development.

The efforts of researchers, practitioners, and policy makers concerned with improving the lives of youth coalesced in the 1990s around the benefits of using positive youth development as a guiding principle to understand adolescent development and improve programs and activities for youth (Roth, Brooks-Gunn, Murray, & Foster, 1998). This approach, rooted in psychological theories stressing the plasticity of human development, dovetailed with a growing awareness of the importance of the afterschool hours, and led to an explosion in the number of youth development programs. The quantity and quality of theoretical and empirical research on the positive youth development framework similarly increased. Thus, as a field we have learned a great deal in the past 20 years about how programs and practices with youth can impede or enhance adolescent development (Eccles & Gootman, 2002; Vandell, Larson, Mahoney, & Watts, 2015)

Given the tremendous strides, the field is poised to initiate the next iteration of research, program development, and evaluation. Perhaps we might call this next phase Version 3.0, with Version 1.0 being the work done through the mid 1990s (Roth et al., 1998) and Version 2.0 being what was done through the mid 2010s ((reviewed in Lerner, Bowers, Minor, Boyd, Mueller, Schmid, et al., 2013; Mahoney, Vandell, Simpkins, & Zarrett, 2009; Roth & Brooks-Gunn, 2003; Vandell et al., 2015). In Version 3.0, we envision several emerging themes, including attention to definitions and measurement, advancements in the design and evaluation of programs, focus on youth programs as a context for development, and integration of promotion and prevention programmatic approaches. As part of our envisioned Version 3.0, we also argue for a blending of prevention and promotion approaches to research, theory, and programs (Brooks-Gunn & Roth, 2013). Thus in this paper we incorporate the lessons learned from both the promotion and prevention fields to argue that youth development programs are best understood, and thus studied, as a context for youth development. Although researchers refer to youth development programs as a context for development, typical evaluation efforts often fail to measure or investigate contextual aspects of programs, or their interrelation with other contexts in the participants’ lives. Throughout this paper, we use the term ‘youth development’ rather than ‘positive youth development’. We believe that the lessons of Versions 1.0 and 2.0 have been so thoroughly incorporated into the zeitgeist that the descriptor ‘positive’ is no longer needed. In the early childhood field, the term is not modified by positive or negative, or promotion or prevention. The amazing growth of the youth development field and the more nuanced work being conducted suggests the same should be true for the field of youth development.

Defining Youth Development Programs

Although organized youth activities and youth-serving organizations have a long history in the United States, the concept of programs geared specifically for positive youth development briefly entered the public policy arena in the early 1960s, but did not begin to gain prominence until the late 1980s. The rise of the positive youth development framework, informed by developmental theories of resilience, plasticity, and competency building, replaced the deficit view of youth as “problems to be managed” with a more positive image of youth as “resources to be developed” (Lerner et al., 2013; Roth & Brooks-Gunn, 2003a; 2003b). This paradigm shift from a primary focus on the prevention of problem behaviors (e.g., drug use, teenage pregnancy), captured by the phrase problem free is not fully prepared, recognized that preventing problem behaviors is not all that is needed to prepare youth for their future (Pittman, 1991). Youth require opportunities and supports for positive growth, including positive relationships with caring adults, challenging experiences, and skill-building opportunities. This shift was central to Version 1.0 theorizing and program development. And, it was necessary given the scarcity of research on positive aspects of youth development.

Struck by the potential of this mindset, we set out to review the evidence for the effectiveness of this “new” approach to programs for adolescents (Roth et al., 1998). At that time, however, no single definition of exactly what constituted a youth development program existed to guide us in selecting studies. Thus we crafted the following vague definition from research on adolescent development and lessons learned from the failures of traditional prevention and intervention programs:

Youth development programs are developmentally appropriate programs designed to prepare adolescents for productive adulthood by providing opportunities and supports to help them gain the competencies and knowledge needed to meet the increasing challenges they will face as they mature (p. 427).

We concluded from our review of the extant experimental and quasi-experimental evaluation literature that work was needed to operationalize the principles of a youth development program as well as to specify the boundaries. That is, without knowing which components, elements, or characteristics are necessary for a program to be considered a youth development program, researchers could not answer questions about the utility of this approach to programming for youth.

The popularity of the positive youth development approach to youth programming has spurred discussion of, and research on, the elements of a youth development program. The field, however, still lacks a clear definition of a youth development program (Walker, Gambone, & Walker, 2011). Instead, most researchers focus their investigations on organized activities generally or extracurricular activities or afterschool programs specifically. The term “youth development program” or “positive youth development program” is less commonly used to describe the program(s) being studied, at least in the quantitative research literature. This lack of clarity contributes to the often conflicting findings about the effectiveness of youth development programs. As revealed from the wealth of research on extracurricular activities, activities vary with regard to the developmental experiences provided, such as experiences of teamwork, initiative, and adult networking (Larson, Hansen, & Moneta, 2006) and the outcomes associated with participation (Eccles & Barber, 1999; Marsh & Kleitman, 2002). The search for the essential elements of a youth development program coupled with recent advances in measuring program quality, discussed below, demonstrate the importance of considering the program context when investigating outcomes. In addition, it is virtually impossible to compare the effectiveness of different programs without contextual information. Such comparisons are central to cost-effectiveness analyses in the education and social services fields (Levin & Belfield, 2015), and ought to be part of the agenda for youth programs Version 3.0.

Three main lists of “required elements” appear repeatedly in the research literature and in guidelines for program staff and developers striving to create effective youth development programs. Eccles and Gootman (2002) list eight features of positive developmental settings, including physical and psychological safety, appropriate structure, supportive relationships, opportunities for belonging, positive social norms, support for efficacy and mattering, opportunities for skills building, and the integration of family, school, and community efforts. Roth and Brooks-Gunn (2003a, 2003b) and Lerner (2004) both reduce the list down to three critical features. For Roth and Brooks-Gunn, the defining aspects of a youth development program include program goals that seek to promote positive development, even when striving to prevent problem behaviors; a program atmosphere that supports positive relationships with adults and peers, empowers youth, communicates expectations for positive behavior, and provides opportunities for recognition; and program activities that allow participants to build skills, engage in real and challenging activities, and broaden their horizons. Lerner’s list of the “big three” fundamental characteristics similarly includes positive and sustained adult-youth relations and life-skill building activities, but adds opportunities for youth participation in and leadership of valued family, school, and community activities. A review of instruments commonly used to assess program quality reveals that all include measures of relationships, environment, engagement, social norms, skills-building opportunities, and routine/structure (Yohalem & Wilson-Ahlstrom, 2010).

Despite the similarities and overlap in these oft-cited lists, as a field we still have not agreed on the fundamental defining characteristics of an effective youth development program (Lerner et al., 2013). In our review of the state of the research almost two decades ago, we concluded that although research points to a number of elements likely to be critical to the success of youth development programs, no evaluation efforts had systematically varied program design to determine which, or what mix, of these elements were necessary. In the intervening years, researchers have made strides in operationalizing the elements of a youth development program, but not in systematically studying them. The lack of specificity allows a myriad of programs to profess to be youth development programs, and hinders our ability as researchers to better understand the role youth development programs play in improving the lives of adolescents. As the present funding climate increasingly calls for evidence-based practice, it is even more imperative that in Version 3.0, the research on youth development programs improves to meet this challenge. Knowing what is and what is not a youth development program is a critical first step in accumulating this type of evidence.

In fact, even general estimates of what percentage of youth participates in a youth development program – as opposed to any type of organized activity – are lacking in large part due to the absence of an agreed upon definition of a youth development program. Large survey studies from which such information could be drawn do not ask about youth development programs per se. Instead, they gather information on participation in any type of organized activity, or in different types of organized activities, either differentiated by content-type (e.g., sports, arts, after-school program) or sponsorship (e.g., school-based, community-based).

Two recent studies, however, did seek to provide estimates on the number of youth participating in youth development programs. In one, the 4-H Study of Positive Youth Development, students were asked specifically about their attendance at 4-H Clubs, Boys Scouts and Girl Scouts, YMCAs and YWCAs, Big Brothers/Big Sisters, and Boys and Girls Clubs. These large youth-serving organizations were classified as youth development programs because their mission statements specifically emphasize a positive youth development perspective (Lerner et al., 2005). Results from this study showed that 92% of sixth graders participated in some type of organized activity outside of school. However, only 36% participated in a youth development program, compared to 72% who participated in sports and 64% in arts (e.g., band, drama, dance) activities (Balsano, Phelps, Theokas, Lerner, & Lerner, 2009). Although this is a longitudinal study with data collected through 12^th grade, subsequent articles do not report the percentage of the sample participating in youth development programs as adolescents, when participation in these programs typically declines.

This percentage is higher than that found in an on-line survey of over 1800 15 year olds, likely due to both the age of the sample, the sampling frame (the 4-H study included youth drawn from schools as well as community youth programs), and the method for classifying participation in a youth development program. Here the researchers used the adolescents’ reports of their experiences in organized activities as the basis for determining involvement in a youth development program. Activities deemed to be youth development programs were those where the adolescent reported developing warm and trusting relationships within the program; involvement in pursuing his/her sparks, or passions (defined as a self-identified interest or skill that provides energy, joy, purpose and direction for the adolescent); and opportunities for feeling empowered through learning teamwork or leadership skills and being allowed to make decisions. Only 23% of the 15 year olds reported attending youth development programs, compared with 68% who reported participating in some type of organized afterschool activity that did not provide these experiences (Scales, Benson, & Roehlkepartain, 2011).

This method for determining participation in a youth development program uses the critical elements of a youth development described, but filtered through the adolescents’ experiences at the program. This approach is rare, but lies at the heart of capturing essential elements of a youth development program, the program experience. We return to this point below in our discussion of the importance of program quality in studies assessing the outcomes associated with participation in youth development programs. But first, we need to briefly discuss how advances in positive youth developmental theory can enlighten our understanding of youth development programs.

Promotion and Prevention

Another classification issue in need of clarification to move the field forward in Version 3.0 pertains to the current division between the fields of promotion and prevention. Although interest in positive youth development programs arose out of the limitations and failures of traditional prevention programs, theory-building research suggests that the risk avoidance and competency promotion views of development might be better understood as coexisting, rather than as competing ideas (Lerner et al., 2013; Schwartz, Pantin, Coatsworth, & Szapocznik, 2007). Findings from studies suggest that problem behaviors often coexist with positive ones within individuals; the two are not simply inversely related. That is, increases in positive outcomes do not always lead to decreases in negative outcomes. Three recent studies using data from the 4-H Study of Positive Youth Development depict this more complex relation. In one person-oriented analysis of positive and problem behavior trajectories, the researchers found that only one-sixth of the sample of early adolescents showed increases in positive behavior and decreases in problem behaviors, the pattern predicted by positive youth development theory (Phelps et al., 2007). In another study extending this research to middle adolescence, the researchers found that while many youth with increasing positive youth development scores showed low levels of risk behaviors, a substantial subset engaged in risky behaviors (Lewin-Bizan et al., 2010). The third study employed survival analysis to examine the association of positive youth development and the likelihood of initiation of sex and tobacco, alcohol, and marijuana use. Again, the association was complex, and varied by gender and risk outcome (Schwartz et al., 2010).

Further support for an integrative model that bridges deficit-based prevention science and the positive youth development perspective arises from the recognition that many of the same general concepts, albeit with different terminology, exist in both theories. Most notably, both approaches highlight similar mechanisms in the family, school, peers, and neighborhoods as well as individual beliefs and attitudes that shape both problem and positive outcomes. Prevention researchers refer to protective factors in these contexts while promotion researchers refer to developmental assets. The extent of convergence between developmental assets and protective factors, both concurrently and over time, or how they may be similarly responsible for inhibiting problem behaviors and promoting thriving, however, requires further empirical study (Schwartz et al., 2007). Although researchers tend to work within these separate theoretical models—promoting youth development or preventing risk-taking behaviors—practitioners often recognize the importance of both approaches. The mission of most youth development programs includes both enhancing competencies and skills and preventing health-compromising behaviors (Roth & Brooks-Gunn, 2003a, 2003b). Version 3.0 research on youth development programs needs to similarly integrate the two approaches to adolescent development.

Programs as a Context of Development

Research in the past two decades, what we are calling Version 2.0, has greatly improved our understanding of the impact of youth development programs. The intersection of the current state of research on youth development programs and federal and private funders increasing emphasis on evidence-based practices produces seemingly conflicting agendas however. The widespread acceptance of the benefits of youth development programs has propelled researchers to declare that the field is ready to move forward from the basic question of “does it work” to more complex questions, such as “why” and “what works best for whom” (Granger, 2010). Yet currently no youth development program meets the requirements set out by the Congressional Top Tier evidence standard (see www.coalition4evidence.org). Despite abundant evidence from surveys linking participation in organized activities to positive developmental outcomes, the few extant studies employing experimental design typically fail to find effects. We argue that one way researchers can reconcile these differences is by more fully investigating youth development programs as a context for development when designing experimental studies.

Contemporary theoretical frameworks for understanding adolescent development share an emphasis on the bidirectional relations between a developing individual and the changing environmental contexts of which he/she is a part. These nested proximal and distal contexts include the family, peers, school, neighborhood, community, region, and country. Increasingly, evidence suggests that youth development programs, vaguely defined, should be regarded as a distinct context influencing development (Larson, 2011; Mahoney et al., 2009). And, as in other contexts, such as schools and families, a developmentally appropriate match between individual characteristics and features of the program, called a “stage-environment fit” promotes development (Eccles et al., 1993). The characteristics of a good stage-environment fit change as youth mature.

Mahoney and colleagues (2009) outlined four implications for studying the effects of out-of-school activities stemming from conceptualizing activities as developmental contexts that pertain to youth development programs as well. First, youth select to attend based on a combination of individual and contextual factors, such as characteristics and interests of the adolescent, encouragement from peers and families, financial resources, parent work schedules, and cost and types of programs in the community. That is, involvement will vary for adolescents based on the intersection of these factors; some youth will have many options for participation while other may not. Second, the outcomes from participation will depend on the proximal processes within the program context, including the characteristics of the individual (e.g., competencies, cultural background, and needs) and the program (e.g., content, quality, peer relations, and adult-youth interactions), as well as the “fit” between the two. Third, these proximal process both affect and are affected by adolescent experiences in other contexts (e.g., their school or families), as well as other program contexts. A complete understanding of the role of the program context requires investigating the interrelations between multiple contexts. And fourth, the changing nature of both adolescents and programs necessitates longitudinal studies. Bringing together these four implications, Mahoney and colleagues call for studying the “dynamics between persons, contexts, processes, and time occurring within and across different types of out-of-school activity settings, for youth from different backgrounds and developmental levels, all in relation to other developmental contexts of adolescence” (p. 233). In the following sections we use these four implications stemming from this view of programs as a developmental context to frame our discussion. We focus on areas needing greater attention in program evaluation research with the hope of spurring program evaluation research Version 3.0.

Participation Differences

The first implication of understanding youth development programs as one of the many contexts influencing development pertains to the question of who participates. A contextual view of youth development programs requires a better understanding of the individual and contextual factors, and their interaction, that influence participation and the specific type of activity chosen. Research findings on activity participation in the past 20 years have identified how demographic factors such as age, socioeconomic status, and race/ethnicity shape the overall availability of activities as well as the types of programs chosen. For example, ample evidence documents differences in participation in organized activities by age, gender, race/ethnicity, and neighborhood, such that youth from more disadvantaged families or communities participate in fewer organized activities, and for less time (Theokas & Bloch, 2006; Wimer et al., 2006). Research has also shown how individual competencies, interests, and motivations are important determinants of participation. Youth with higher academic or social competence are more likely to participate in organized activities after controlling for family background (Mahoney, Cairns, & Farmer, 2003; Marsh & Kleitman, 2002). Youth typically mention their interests, encouragement from friends, and opportunities provided by the activity as reasons for participation (Gambone & Arbreton, 1997; Perkins et al., 2007). Less frequent are studies investigating how these demographic and individual factors interact with the other contexts in adolescents’ lives, including their peer groups, families, schools, neighborhoods, and the activities themselves to shape participation (see Mahoney et al., 2009 for a full review).

Research on these interactions offers the next step in our understanding of factors influencing participation. For example, researchers have recently sought to disentangle income from race/ethnicity by investigating differences in participation within ethnic groups, instead of between groups, to better understand how ethnicity contributes to participation. Simpkins and colleagues (2011) found large variability in participation within each of the four largest Latino ethnic groups (Mexican, Cuban, Puerto Rican, Center/South American) based on markers of cultural orientation. Markers of cultural orientation, such as nativity, were more important predictors of participation in school-based activities than family SES for these groups of Latino adolescents. In another study, the researchers focused on how individual, family, school, and neighborhood risk factors interact to predict activity participation, and how these interactions may vary with youths’ age. Findings indicate that adolescents with greater academic and social problems (child-level risks) and those from less supportive, involved, and engaged families (family-level risks) were less likely to participate in organized activities. The findings for school and neighborhood risks were less straightforward; they varied by age and activity type (Wimer et al., 2008).

Thus although research in the past two decades has shed a great deal of light onto participation differences among youth from different backgrounds, we know less about differences in youth development program participants, largely because we have no agreed upon definition of a youth development program. Many youth development programs arose to address the need for more opportunities and supports for youth in low-income, minority communities and thus participants tend to be more racially or socioecomically homogenous. Studies of programs for youth, rather than extracurricular activities, report conflicting findings about the demographic characteristics of the youth they serve. Examining the demographic characteristics of the sample in most evaluations of youth programs leads to the conclusion that although minority and low income youth are less likely to participate in organized activities in general, they are overrepresented in youth programs. Reports on the composition of youth participating in the programs of major youth development organizations (such as 4-H, Boys and Girl Scouts), however, indicate that ethnic minority youth are underrepresented, and more likely to drop out compared to White youth (Russell & Heck, 2008; Villarruel, Perkins, Borden, & Keith, 2003).

Effects of Participation

Much of our knowledge about the positive effects of participation in youth development programs relies heavily on research demonstrating the favorable influence of participation in organized activities in general, rather than specifically in youth development programs (Feldman & Matjasko, 2005; Mahoney, Harris, & Eccles, 2006). The lack of a clear definition of a youth development program likely explains why, despite the acceptance of the positive youth development approach, the term “youth development program” is rarely, and inconsistently, used to describe programs in research linking participation to outcomes. Researchers using data from the 4-H Study of Positive Youth Development to investigate the outcomes associated with participation in national youth serving organizations with a positive developmental mission failed to find a direct association between participation and a combined measure of positive youth development (Lerner et al., 2005; Mueller et al., 2011). Participation in these youth development programs did however predict concurrent reports of civic engagement and youth contribution (Bobek, Zaff, Li, & Lerner, 2009; Lerner et al., 2005). In contrast, researchers defining youth development programs by their essential programmatic elements do find positive effects from participation when compared to programs not incorporating these elements (Catalano, Berglund, Ryan, Lonczak, & Hawkins, 1999; Durlak & Weissberg, 2007; Roth & Brooks-Gunn, 2003b).

Program Processes

The essential elements used to compare programs in these studies varied however. As noted above, as a field we still have not agreed on the essential elements of a youth development program. Yet this is critical to understanding how programs can change young people’s lives. The second implication of viewing youth development programs as a context for development, the importance of capturing characteristics of the individual and the program when investigating outcomes, lies at the heart of investigations of the effects of participation in youth development programs. Identifying the important proximal processes within programs has been a main focus of research on activity settings version 2.0. This research illustrates the importance of program quality. Features of a quality program mirror those of a youth development program. That is, there is general agreement that programs are most effective when they create positive relationships for youth and engage them in challenging, authentic activities. Empirical evidence that links specific program features to youth outcomes, however, is rare (Yohalem & Wilson-Aslstrom, 2010).

The importance of positive, supportive, and sustained adult-youth relationships appears in every discussion of why programs impact participants, leading some researchers to refer to them as the “critical ingredient” in successful programs (Rhodes, 2004). The quality of these relationships has been linked to youth attendance, engagement, and positive outcomes. Much of this research comes from theory-building and qualitative studies. These studies find participants place great importance on these relationships. For example, during interviews many adolescents report they attend programs because they “like the staff” or the adults at the program “care” about them (Fredricks, Hackett, & Bregman, 2010; Gambone & Arbreton, 1997; Perkins et al., 2007). The qualitative research into the workings of successful youth programs by Larson and colleagues finds that adults play an important role in facilitating positive development. Successful adult leaders engage youth by creating a welcoming atmosphere (Pearce & Larson, 2006). They use a variety of strategies to facilitate growth, such as balancing youths’ needs for ownership with keeping them moving towards a goal (Larson, Hansen, & Walker, 2002).

Few researchers empirically test the association between adult-youth relationships and youth outcomes. Researchers in one recent study of high quality programs report that adolescents’ perceptions of how caring and competent program staff were predicted their engagement, and was more important than demographic characteristics (Greene, Lee, Constance, & Hynes, 2013). In another, the researchers found that adolescents, particularly those with poor relationships with their parents, reported less depressed mood when they attended programs where they perceived high support from the activity leader (Mahoney, Schweder, & Stattin, 2002). Neither of these studies, however, utilized longitudinal designs. Further evidence for the importance of a supportive atmosphere comes from a study comparing the features of youth development programs that showed improved adolescent sexual and reproductive health outcomes at the end of the program and beyond with the characteristics of programs that did not. Effective programs were significantly more likely to deliver activities in a supportive environment (Gavin, Catalano, David-Ferdon, Gloppen, & Markham, 2010). This characteristic was one of the few to significantly differentiate successful programs from unsuccessful ones.

The second essential element of successful youth development programs pertains to engaging youth in challenging and authentic activities. Engagement refers to the cognitive, behavioral, and emotional attributes necessary to connect to the people and activities at a program. Apathy, boredom, inattentiveness, and passivity characterize low engagement while relatively high attention, interest, enjoyment, and effort to master new skills depict high engagement (Larson, 2000; Weiss, Little, & Bouffard, 2005). Numerous studies, using a variety of methodological techniques, find greater levels of engagement in structured activities when compared to school or unstructured activities (Bohnert, Richards, Kolmodin, & Lakin, 2008; Larson, 2000). Few studies, however, examine how youth engagement while at a program relates to outcomes (Bohnert, Fredricks, & Randall, 2010; Roth, Malone, & Brooks-Gunn, 2010). The few extant studies find that greater engagement predicts longitudinally higher academic outcomes for participants in afterschool programs (Mahoney, Lord, & Carryl, 2005; Shernoff, 2010).

In our 1998 review we noted that although we had uncovered a number of elements that seemed critical to program success, no evaluation efforts had systematically varied program design to determine which, or what mix, of elements are necessary (Roth et al., 1998). This conclusion is still true today. Research on identifying and operationalizing key programmatic elements is progressing, but still leaves many unanswered questions. As one research team noted, “we know little about how to reliably create these conditions across programs, different groups of staff, and youth from diverse backgrounds” (Larson, Kang, Perry, & Walker, 2011). Program designers are left to decide for themselves which specific program features are most important in achieving outcomes, and thus necessary to include in their programs (Arnold & Cater, 2011).

We are hopeful, however, that the current emphasis on evaluating program quality combined with the increasing understanding of the importance of program implementation has laid the groundwork for these types of evaluation efforts in Version 3.0. The realization that programs are only as good as their implementation has gone hand-in-hand with the growth of the program quality movement (Arnold & Cater, 2011; Hirsch, Mekinda, & Stawicki, 2010). Capitalizing on natural variation in program implementation provides one way to examine the relative impacts of different program features. To do so, researchers must measure point-of-service differences in quality, such as differences in staff practices and youth experiences in addition to outcomes, at multiple sites utilizing the same program model. Then researchers can examine the extent to which variation in these factors predicted variation in program effects. After this type of exploratory research, the next step is to rigorously confirm the hypothesis about the influence of specific program features by randomizing planned variation of these features (Weiss, Bloom, & Brock, 2013).

To date, only a few researchers have reported on such studies. Their findings point to the difficulty of this type of research. In one, a randomized trial of an afterschool program for middle school students, tests for interaction by program implementation quality were not significant for any of the outcomes studied (Gottfredson, Cross, Wilson, Rorie, & Connell, 2010). Although there was variability across the sites in quality, it was not related to program effectiveness. The authors attribute the lack of findings to the overall low levels of attendance at any of the programs; the program, as designed, was not engaging enough for middle school students. In another randomized study designed to compare three programmatic elements (exposure to prosocial adults, provision of afterschool activities, and in-school cognitive behavioral therapy) of a program to prevent violence in high-crime neighborhoods, the researchers could not analyze the data as planned due to inadequate implementation of the program elements and high crossover between conditions (Heller, Pollack, Ander, & Ludwig, 2013, May).

Interactions between program and individual contexts

We now turn in our discussion to the second part of the second implication of studying youth development programs as contexts, the importance of investigating the interaction of individual and programmatic characteristics. From the start, leaders in the field of youth development programs have questioned “what works for whom”. At the most basic level, the answer to this question has implications for the universal approach espoused for youth development programs—that all youth benefit from participation in a program employing the features of successful youth development programs. Yet naturally youth vary, both in their individual characteristics and in the features of the other contexts of their lives (e.g., families, peers, and neighborhood). These differences raise the question of whether or not youth development programs are better for some participants’ than others, and whether certain features of youth development programs have a greater impact for some participants than others. In our initial review of the program evaluations, we surmised that some groups of youth may benefit from youth development programs more than others, but at the time, this question had not been adequately addressed. Researchers have begun to make progress in this area, but still cannot fully answer this question.

Research to date suggests that in general, youth at greater risk reap more benefits from participation. This conclusion emerges from studies of different individual characteristics, such as race/ethnicity and income, as well as prior behavior problems or low achievement levels with regard to extracurricular activities (Marsh, 1992), organized activities (Mahoney & Cairns, 1997; Mahoney & Sattin, 2000), and afterschool programs (Black, Doolittle, Zhu, Unterman, & Grossman, 2008; Policy Studies Associates, 2002). It is important to note, however, that the scant research on outcomes from participation in organized activities for minority youth typically confound issues of race and income by equating low socioeconomic status with ethnicity (Fredricks & Simpkins, 2012). Research that untangles these suggests future research take precautions against describing minority youth as a homogeneous group. For example, using data from a sample of high achieving minority youth attending a selective urban high school, researchers found that high levels of participation in organized activities (i.e., overscheduling) had a negative effect on low-income, but not high-income youth (Randall & Bohnert, 2012). Differences within ethnic groups, such as cultural orientation and immigration status among Latino ethnic groups, or variations in SES within an ethnic group are emerging as important contextual factors to consider when studying ethnic groups (Fredricks & Simpkins, 2012).

To our knowledge no extant empirical research addresses the question of whether certain features of youth development programs have a greater impact for some participants than others. A contextual view of programs suggests that one size does not fit all with regard to youth development programs. Practitioners recognize this, and typically adapt program models to fit their population. These adaptations lead to implementation differences. Whether or not they are meaningful, however, remains an open question. Researchers need to distinguish between core features that can be applied broadly to programs of various types, and the specific features required for programs that target specific populations, such as those of a particular developmental stage (i.e., age), cultural background, or risk status (Granger, 2010). For example, evidence from a study of an afterschool program for low income Latino adolescents suggests the importance of aligning program qualities with the specific needs of the target population. In this study, participation in a high quality program was associated with increases in participants’ self-worth, but only programs with an emphasis on ethnic socialization were associated with ethnic identity development, an important developmental process for minority youth (Riggs, Guzman, & Davidson, 2010).

The notion that some youth may benefit more from participation than others, known as differential effectiveness, can help explain the null results from evaluations employing random assignment designs (Weiss et al., 2013). In addition, understanding differential effects can help researchers and practitioners refine a program’s theory of change, make program improvements, and tailor programs or program components to specific populations or contexts (Greenberg & Lippold, 2013). Researchers in the field of prevention science are further along methodologically in their tests of whether the effects of interventions are universal, or stronger among some subpopulations than others, likely due to a greater reliance on random assignment designs. Findings from subgroup analyses underscore the importance of moving beyond analyses of main effects to identify participants who are most or least likely to respond positively to a program. For example, research on universal violence prevention programs has found that the programs were only effective for the highest risk youth (Conduct Problems Prevention Research Group, 2011; Jones, Brown, & Aber, 2011). Other studies, however, find programs to be the most effective with youth showing moderate risk levels (Schwartz, Rhodes, Chan, & Herrera, 2011; Spilt, Koot, & Van Lier, 2013). Still others show that positive main effects can mask the ineffectiveness or even adverse effects for certain youth. For example, research on another violence prevention program demonstrated beneficial effects for high-risk adolescents but also increases in aggression for low-risk adolescents (The Multisite Violence Prevention Project, 2009). Yet, as highlighted in a recent special issue of the journal Prevention Science, many of the studies employing and reporting subgroup analyses suffer from methodological flaws, such as the lack of careful planning to ensure appropriate power to detect effects, clear specification of exploratory versus confirmatory subgroups, and reporting results appropriately (Supplee, Kelly, MacKinnon, & Barofsky, 2013).

Interactions among Contexts

The third implication of studying youth development programs as contexts, that proximal processes within a program context both effect and are effected by adolescent experiences in other contexts (e.g., their school or families), as well as other program contexts, underscores two areas of much needed research for version 3.0. First, few researchers empirically study the influence of other contexts, outside of the individual and family contexts, on program outcomes, and ever fewer investigate interactions and reciprocal relations between contexts. Second, few empirical studies collect adequate information about other program contexts for program participants and the comparison group youth. We end this section with an example of a promising approach to improving the lives of youth that utilizes our knowledge of the interactions among contexts.

Research on the associations between other contexts and developmental outcomes from program participation tends to focus on risks in these contexts, such as risks in peer relationships, schools, or neighborhoods. In line with the above discussion of risk, these more limited findings also show greater benefits for youth at greater risk. For example, in a longitudinal study of older adolescents, those with lower friendship quality and more loneliness at baseline benefited more from activity involvement (Bohnert, Aikins, & Edidin, 2007). In another longitudinal study, researchers found that neighborhood-level variation contributed to the valence of the association between participation in extracurricular activities and youth’s behavior, although the effects were small (Fauth, Roth, & Brooks-Gunn, 2007). For example, perceived neighborhood violence moderated the unfavorable association between participation in community-based clubs and youth’s anxiety/depression such that participation in community-based clubs was only predictive of increases in anxiety/depression in violent neighborhoods. Similarly, participation in church groups was protective against substance use, but only for youth in low-violence neighborhoods. Other researcher investigating neighborhood effects have found an interaction between neighborhood and individual characteristics. Following up on findings that extracurricular activity involvement had its greatest impact on youth living in more disadvantaged settings, defined as lower asset neighborhoods, the researchers found that person-level factors moderated this association (Urban, Lewin-Bizan, & Lerner, 2009). Among adolescents living in low asset neighborhoods those with higher intentional self regulation skills, especially girls, benefited the most from participation in extracurricular activities (Urban, Lewin-Bizan, & Lerner, 2010). It is important to note, however, that all of this research pertains to extracurricular activities, not youth development programs per se.

Another way contexts can interact is through reciprocal relations, such as when participation improves relationships with peers or youth’s family, which in turn facilitates more participation in organized activities. For example, researchers have found that more activity involvement in 10^th grade predicted fewer internalizing symptoms in 11^th grade, which then predicted more activity involvement in 12^th grade after controlling for prior symptoms and risk (i.e., maternal depression history). No reciprocal relations were found for externalizing behaviors (Bohnert, Kane, & Garber, 2008). Yet few researchers examine reciprocal relations.

A contextual view of youth development programs also demands more detailed data collection with regard to the full array of activity participation (i.e., breadth) for both participants and control or comparison group youth. Researchers operationalize breadth of participation differently; in some studies of activity involvement, researchers measure breadth as the total number of activities in which youth participate while in others they measure it by the array of activity types youth attend in a week or year. Youth typically participate in more than one activity, and different types of activities carry different benefits and risks to development. For example, a more complete picture of activity participation can explain the mix of positive and negative outcomes associated with sports participation. Findings from person-centered analysis show that adolescents who participated only in sports activities had more positive outcomes compared with those who had little or no involvement in organized activities, but less positive outcomes compared with those who participated in sports plus other types of activities (Linver, Roth, & Brooks-Gunn, 2009). In another study of sports participation, the researchers found that the conclusions about the riskiness of participation in sports differed by referent group. That is, the odds of nonviolent delinquency were higher among boys who participated in sports compared to boys who participated only in nonathletic activities, but not when compared to boys who did not participate in any organized activity (Gardner, Roth, & Brooks-Gunn, 2009).

Within a specific program, breadth refers to the mix of activities youth participate in while at the program. Few program evaluations capture this type of breadth among program participants (Roth et al., 2010). Similarly, most of the extant research fails to measure the activities of the comparison group. By definition, the students in the comparison group are not participating in the program being studied, but too often, researchers do not know if they are participating in a similar program, or any other type of afterschool activity. For example, in a recent meta-analysis of 69 after school programs, only five of the evaluations reported if youth in the control group were participating in other potentially beneficial programs (Durlak, Weissberg, & Pachan, 2010). As has been shown with studies of breadth of organized activity participation, knowing what program participation is being compared to can alter the interpretations of the consequences of participation. Although there are some notable exceptions, the vast majority of program evaluations are vague about the types of alternative arrangements that serve as the reference comparison (Durlak, Mahoney, Bohnert, & Parente, 2010).

Almost two decades ago we noted that better designs to capture adolescents’ movements in and out of different organizations were needed. At the time, most evaluation designs viewed movement between programs as a contaminating factor. However, youth’s ability to move voluntarily in and out of programs and to attend more than one program are very important features of youth development. It appears that there has been little improvement in this area.

We end this section with a promising approach to youth development from the prevention field that builds on the growing awareness of the importance of the interactions between contexts, the Communities that Care (CTC) model (Lerner et al., 2013). This program focuses on preventing delinquency and substance use by building youth connections to other contexts, such as family, school, and community. After an initial survey of elevated risk factors and depressed protective factors in the community, the diverse and representative prevention coalition chooses from a list of evidence-based programs. The programs on the list are quite varied, and include programs aimed at improving family interactions starting at birth, reading skills, and/or youth development as well as those targeted at preventing problem behaviors. The coalition then implements the programs with fidelity, and evaluates their effectiveness. A national community-randomized trial of the CTC model followed 4407 fifth grade students annually through 10^th grade. The most recent results, gathered one year after the end of technical assistance, show the success of this model. Risky behaviors, including alcohol use, cigarette use, and delinquency were lower by grade 10 among students in the CTC communities than in the control communities. Although protective factors were not measured in this study, evidence from a quasi-experimental study of CTC in Pennsylvania suggest increases in protective factors, such as community and family cohesion, and school prosocial support (Feinberg, Jones, Greenberg, Osgood, & Bontempo, 2010).

Longitudinal studies

The final implication stemming from viewing programs as contexts for development, that the changing nature of both adolescents and programs necessitates longitudinal studies, has been often repeated, but not often implemented. The field still is dominated by cross sectional studies. In our past review there were so few quality evaluations that we did not even raise the issue of the need for longitudinal studies. The growing collection of well-conducted studies in Version 2.0 now allows us to draw attention to the need for longitudinal research. Thus, it is a start, but not sufficient, for researchers to collect data on program participants at some point after the end of the program. To fully understand how program participation can have lasting impacts on participants it is important to also collect information on their activities in the intervening years.

Further, a contextual view suggests the value of studies that capture the changing nature of youth and programs over time. Youth’s interests and needs change as they age, and programs that once provided appropriate opportunities and supports may no longer offer a good fit. Similarly, programs change over time as they mature and funding priorities shift (Mahoney et al., 2009).

Conclusions and Future Directions

Almost two decades ago we conducted an exhaustive review of the literature on youth development programs. Writing about that review, we concluded that the field of youth development was poised to begin its second generation of programmatic efforts. At that time (Version 1.0), we observed a mismatch between the enthusiasm for these programmatic efforts and the empirical evidence. In the intervening years (Version 2.0), researchers have made great strides in furthering our understanding of positive youth development, contexts of development, and the elements of a quality program. Yet challenges still exist in the research literature empirically testing portions of these theoretical advances, most notably the critical elements of a youth development program and the influence of other contexts on the association between program participation and outcomes for youth. In this last section of the paper, suggestions are provided for incorporating this lens in future research of youth development programs (Version 3.0).

Research on youth development programs falls within the general area of research on organized activities. Despite ample research showing variations in both experiences and outcomes for youth participating in different types of organized activities, too often conclusions from one category of activities are assumed to apply to all types of programs or activities. We have argued for the necessity of a clear definition of a youth development program (or at least of the features of any given program, using the same metric across evaluations). Mounting evidence of the importance of program quality, largely from research on afterschool programs, highlights the vast differences between programs following similar models of programming. Building on the research gains in these areas lays the foundation for the next steps for youth development programs. The characteristics of a quality program include the essential elements of a youth development program, and can be used to distinguish between organized activities in general and youth development programs specifically. Any evaluation of a youth development program needs to collect data on the program processes, including data on program quality. Researcher can improve the data available from large-scale surveys by including, at a minimum, questions that capture the context of the activity, not just the content (i.e., sports, arts) or sponsorship (i.e., school- or community-based). The survey questions used by Scales and colleagues (2011) described in this paper offer a good starting point for assessing youth’s experiences in any type of program.

In addition to the need for more information on the program context, researchers need to collect more data on the other contexts of youths’ lives. With regard to the individual context, researchers need to move beyond analyses that differentiate youth based simply on their demographic characteristics such as race/ethnicity, SES, and age to include a fuller picture of the individual context. Prior research suggests the usefulness of collecting data on individual characteristics such as interest, personality characteristics, health, and academic and social competence. Similarly, influences from other contexts need to be included to understand how they influence who participates in youth development programs as well as how these characteristics interact with the program context to influence outcomes for youth. The field is more than ready to move beyond studies that fail to take these other contexts, such as peers and neighborhoods, into account. The same holds true for time; the context of development requires that studies be longitudinal. At a minimum, studies need to collect data at two time points, preferably at the start and end of the program or school year.

Every discussion of the state of the program evaluation literature ends with a similar call for more research. We concur with our colleagues in concluding that we need more research on “who does what to whom, in what ways, in what types of settings, within what broader context; and what level of participation or engagement is needed by which populations to achieve what types of outcomes” (Durlak, Mahoney, et al., 2010). Youth development program evaluation is a complex task, and evaluation methods and measures need to fully consider the social and developmental contexts of the programs. This complexity should not discourage the use of randomized experiments to reach causal conclusions and rule out selection bias however. The false assumption of the difficulties of conducting randomized experiments results in too little high-quality research on youth development programs. Different research techniques, such as group-level experimental designs, offer one approach to overcoming the problems associated with randomization on the individual level, such as treatment diffusion, and concerns about withholding services from a population in need (Durlak, Mahoney, et al., 2010). This approach to randomization can also provide valuable information in line with the contextual perspective. For example, group-level randomization designs can generate information on the outcomes for youth from attending similar programs located in different neighborhood contexts or that serve different populations. Another approach is a regression-discontinuity design, which may be applied if programs are implemented differentially by locale or by age (Cook, 2013).

While disagreement exists among our colleagues about the necessity of experimental or quasi-experimental designs for studying youth development programs, we believe that findings about program effectiveness from non-experimental designs are much less likely to be accepted as valid by researchers in other fields or by policy makers. Increasing the number of programs and youth being served is a goal of everyone involved in the field. Policy makers and funders want evidence that investments matter. Demonstrating effectiveness to these stakeholders requires more than descriptive data, no matter how compelling such data might be. As stated earlier, Congress and OMB are requiring evidence for funding of programs, more so today than even five years ago (see Haskins, Paxson & Brooks-Gunn, 2009 and Haskins, 2014, for a discussion of evidence-based programing in early childhood).

Research on youth development programs has much to gain by applying a contextual lens to the study of program impacts. A more complete understanding of the interaction between individual characteristics, the program context, specifically the program processes related to quality, and the peer, family, school, and neighborhood contexts can propel the field forward by demonstrating the essential programmatic elements, and if they vary depending on the context.