Fidelity of Afterschool Program Implementation Targeting Adolescent Youth: Identifying Successful Curricular and Programmatic Characteristics

Abstract

BACKGROUND

This study consisted of a formative evaluation of an after-school health education program designed for adolescent females, entitled Girls on Track. Evidence-based after-school programs have potential to supplement the traditional school day, encourage social and emotional skill development, improve the quality of student health, and contribute to a healthier school environment. Implementing comprehensive programs outside of the classroom, however, is challenging and gaps exist in the literature regarding implementation fidelity.

METHODS

The national program was in a cohort of adolescent females from 11 Girls on Track locations in fall 2008 and 10 locations in spring 2009. Mixed-method analyses evaluated fidelity of lesson implementation, described curricular and programmatic characteristics associated with implementation fidelity, and assessed coach perceptions toward program quality and feasibility.

RESULTS

Specific lesson characteristics associated with higher rates of implementation fidelity and favorable coach perceptions toward program feasibility included improved lesson clarity, more time to process health topics, teaching and learning objectives that were well-aligned with lesson activities, incorporation of alternative activities for less-motivated participants, and provision of resources on key health topics for coaches.

CONCLUSION

Girls on Track is currently being implemented across the United States and Canada, reaching over 5,000 adolescent females annually. Identifying and incorporating specific curricular and programmatic characteristics associated with high levels of implementation fidelity can enhance the quality and benefits of after-school programs.

Comprehensive after-school health education programs have potential to provide an enriching supplement to the conventional school day.¹ For adolescent females, the need to address relevant health and behavior issues is particularly important during this period of physical and emotional development. Adolescents are susceptible to developing poor habits and making unhealthy decisions that could lead to a range of problem behaviors.^2,3 It is a confusing and complex time, and adolescents are more likely to engage in high-risk activities during this period.⁴

Education policies in the past 2 decades have placed a greater emphasis on standardized testing as a means of assessing academic achievement;⁵ therefore, minimizing time during the school day for adolescents to focus on the development of key behavioral skills. Research indicates, however, that intellectual development needs to be supplemented with social and emotional skill development,⁶ and studies suggest that evidence-based after-school health education programs can help adolescents be more physically active, acquire new health knowledge, develop important social and emotional coping skills, minimize juvenile crime, and foster the development of healthy friendships.^1,7–11 Research has further shown that well developed and carefully implemented after-school programs may contribute to narrowing the achievement gap among disadvantaged youth.⁸

Whereas there is a relative paucity of research on after-school programs, considerable research has focused on the pedagogy and curriculum development of in-classroom teaching and learning initiatives.^12–14 Studies have been conducted to improve understanding about classroom dynamics, promote learning among students of different abilities, and improve classroom assessment methods.^12,13

In contrast to their in-classroom counterparts, volunteer-based after-school programs are more likely to experience inconsistencies in both staff and implementation.¹⁵ Despite there being a great deal of research devoted exclusively to the development of after-school programs, implementing after-school health education programs as intended has traditionally proven to be challenging and a range of barriers to program implementation can affect the effectiveness and fidelity of curricula.¹⁵ Little is known about how to develop after-school curricula and organize a program’s structure in such a way that fidelity of implementation is maximized. By identifying determinants of implementation fidelity from both a curricular and programmatic standpoint, the long-term effectiveness of adolescent health education programs that take place outside of the classroom can ultimately be maximized.

Girls on the Run, International (GOTRI) is an example of a leading non-profit organization dedicated to improving the lives of young girls across the US and Canada via a comprehensive after-school health education program. Currently over 60,000 girls in over 160 communities are a part of this organization. The purpose of GOTRI is to promote a balanced and healthy lifestyle, teach young girls how to make healthy decisions, and emphasize the importance of being kind to others, while simultaneously training them to cultivate a desire for physical activity. GOTRI has 2 after-school components: a 2-semester sequence for girls in grades 3–5 (entitled “Girls on the Run” (GOTR)) and a 1-semester sequence for girls in grades 6–8 (entitled “Girls on Track” (GOT)). Each city or county that GOTRI takes place in is called a “council”, comprising of a council director, staff, and volunteers who together oversee the implementation of the programs within their specified geographic region. Each GOTR or GOT site within a council typically has two volunteer coaches, one of whom is CPR-certified and designated the “head coach.” The organization is funded primarily through corporate sponsors, private foundations and program fees. GOT is a 12-week program, consisting of 24 bi-weekly lessons. In 2008, the study authors collaborated with GOTRI to revise the GOT program based on the results of an evaluation conducted between 2002 and 2006.¹⁶ The implementation and formative evaluation of this new version of the GOT program subsequently served as the basis for this study.

The purpose of this study was to identify specific characteristics, from both a curricular and programmatic perspective, that maximize the fidelity of after-school program implementation for after-school health education initiatives such as GOT. The study specifically addressed 2 research questions: (1) What are the curricular and programmatic barriers hindering fidelity of after-school health education program implementation? (2) If these barriers are directly addressed, does implementation fidelity improve?

METHODS

This study had a mixed method design: both qualitative and quantitative data were collected. The new GOT curriculum was implemented and formatively evaluated among eleven GOT locations with a total of 176 participants in fall 2008. The results of this evaluation guided revisions to the curriculum, which was then implemented in spring 2009 among ten comparable GOT locations with a total of 113 participants. Parental consent, the adolescents’ assent, and consent from the coaches and council directors at each location were received at the beginning of the GOT seasons.

Participants

There were 289 program participants distributed among 21 GOT sites, with an average of 13.8 girls participating per GOT location. The mean self-reported age for the fall 2008 and spring 2009 cohorts was 11.6 years (±0.67) and 12.6 years (±1.01), respectively. Of the 21 locations, the majority (80.8%) took place at public schools, with the remaining locations taking place at private schools (4.8%), parochial schools (4.8%), community centers (4.8%), and independent schools (4.8%). One third of the participating GOT sites were located in suburban areas, 28.6% were in urban areas, 28.6% were in rural areas, and the remaining sites (9.5%) classified their geographic region as “other.”

There were 53 coaches distributed over the 21 locations. Over half of the coaches (54.7%) indicated that it was their first time coaching for GOTRI and nearly 80% of the coaches indicated that they did not have previous coaching experience with adolescents outside of GOTRI.

Instruments

“Implementation” was operationally defined as the extent to which each of the program’s lesson activities were implemented as intended, without modifications. Specifically, the proportion of lesson activities implemented as intended was used as a measure of fidelity. Fidelity of program implementation was measured, coach perceptions toward program quality and feasibility were assessed, and demographic data on the participants, coaches, and site locations were collected via 7 evaluation instruments; (1) implementation monitoring survey, (2) formative evaluation measure, (3) site-level demographic survey, (4) program participant demographic survey, (5) program participant opinion survey, (6) coach demographic survey, and (7) telephone survey. Each instrument is described with respect to purpose, respondent, number of items, and response format in Table 1. These measurement tools collected both qualitative and quantitative data.

Table 1.

Evaluation Instruments

Title	Purpose	Evaluation Completed By	Number of Items	Item Example	Answer Response Format
Implementation Monitoring Survey	To measure fidelity of lesson activity implementation and collect data on any modifications made to the lesson activities.	Head Coaches (with input from Assistant Coaches as deemed necessary)	7–8 Items per Lesson	For Lesson 3 (“Learning to Deal”), Activity 1 (“Getting on Board”, please circle 1:	Multiple Choice and Open Answer Questions
				Y = Yes, I used this activity M = I modified this activity and used it N = I did not use this activity
Formative Evaluation Measure	To assess coach perceptions toward program quality and feasibility	Head Coaches (with input from Assistant Coaches as deemed necessary)	17 Items	With regard to Lesson 3 “Learning to Deal”, the objectives were stated clearly (Please circle one)	Likert-Scale Response Format (Multiple Choice)
				1 = Agree Strongly 2 = Agree 3 = Disagree 4 = Disagree Strongly
Site-Level Demographic Survey	To collect demographic data on each GOT site.	Council Directors	4 Items	What type of setting best describes this Girls on Track locations? (Please circle one).	Multiple Choice and Fill-in-the-Blank
				1 = Public School 2 = Private School 3 = Parochial School 4 = Community Center 4 = Other (Please Specify)
Program Participant Demographic Survey	To collect demographic data on each of the program participants.	Program Participants	5 Items	What grade are you in? (Please circle one).	Multiple Choice and Fill-in-the-Blank
				1 = 5th Grade 2 = 6th Grade 3 = 7th Grade 4 = 8th Grade 5 = 9th Grade
Program Participant Opinion Survey	To collect opinions on the lessons as a whole from the program participants.	Program Participants	74 Items	“I really liked Lesson 3.” (Please circle 1):	Likert-Scale Response Format (Multiple Choice) and Open Answer Question
				1 = I strongly agree with this 2 = I agree with this 3 = I disagree with this 4 = I strongly disagree with this
Coach Demographic Survey	To collect demographic data on each of the coaches.	Head Coaches and Assistant Coaches	12 Items	Is this your first time coaching for Girls on the Run, International? (Please circle one).	Multiple Choice and Fill-in-the-Blank
				1 = Yes 2 = No
Telephone Survey	To measure opinions on the overall quality of the curriculum, health topics, and implementation.	Head Coaches	3 Items	Was there anything about the lessons or program as a whole that you or your assistant coaches found problematic?	Open Answer Questions

Procedure

The study’s evaluation process included 7 different instruments and the data collection procedure remained similar across these different tools. During fall 2008, the written surveys, a standardized administration procedure, and stamped addressed envelopes were mailed to the home address of each head coach 2 weeks prior to the administration of each evaluation measure. An email reminder was also sent to the head coach during this time. If after 2 weeks the completed surveys were not returned, a reminder was emailed directly to the head coach. In the case the email did not prompt a response, the head coaches were called directly.

Based on results from the fall 2008 program evaluation, the efficiency of the evaluation process was enhanced by administering several of the measures online. The instruments themselves remained the same with the exception of the implementation monitoring survey. Slight changes in content to the implementation monitoring survey were made to reflect changes resulting from revisions to the curriculum following the fall 2008 implementation.

The response rates for the implementation monitoring survey, which was conducted on a bi-weekly basis, were 96.3% in fall 2008 and 91.2% in spring 2009. The response rates for the formative evaluation measure were 94.3% in fall 2008 and 91.3% in spring 2009. All the council directors completed the site-level demographic questionnaire at the beginning of the 2008 and 2009 seasons (N = 13). The administration of the coach demographic survey was overseen by the head coach at each site and the response rates were 96.7% (N = 30) in fall 2008 and 81.8% (N = 18) in spring 2009. Response rates for the telephone survey conducted during the final week of the season were 90.1% (N = 10) in fall 2008 and 90% (N = 10) in spring 2009. Lastly, the head coach at each site also oversaw the administration of the participant demographic and opinion measures. The response rates for the participant demographic survey and participant opinion measure were 39.7% (N = 70) in fall 2008 and 40.7% (N = 46) in spring 2009.

Data Analysis

Mixed methods were used to analyze the study data. Due to time constraints and logistical issues associated with weather and space availability, several of the GOT sites (N = 16) implemented a 10-week (versus 12-week) version of the program. This distinction was taken into consideration during the analysis process.

The qualitative data from the implementation monitoring survey, program participant opinion survey, and telephone survey were transcribed, thematically coded, and subsequently summarized. In reviewing these data, recurring themes and key words were highlighted, and as more data were collected, additional exemplars were added under each theme. The data were then all reviewed once more and summaries for each theme were collated to help further abstract the findings.

The quantitative data were analyzed using the statistical package SPSS (version 17.0). Descriptive statistics were used for demographics and program fidelity. To calculate fidelity, the results of the implementation monitoring survey were used to determine the overall proportion of lesson activities implemented as intended for each participating GOT site. This value was calculated by averaging the percentage of lesson activities fully implemented as intended for each site over all the lessons. If a site did not respond to or complete an evaluation for a particular lesson, it was assumed that the lesson and corresponding activities were not implemented. In the 10-week curriculum, there were 100 different lesson activities evaluated and in the 12-week curriculum there were 107 different lesson activities.

RESULTS

Descriptive statistics are presented regarding overall levels of program implementation. Qualitative results concerning curricular and program characteristics are outlined. Comparisons between the 2008 and 2009 GOT seasons are summarized.

Fidelity of Program Implementation

The proportion of lesson activities implemented as intended was calculated for each GOT site and served as a quantitative measure of implementation fidelity. These values ranged from 0.47 to 0.87 (mean = 0.68 ± 0.12) during fall 2008 and from 0.56 to 0.88 (mean = 0.76 ± 0.11) during spring 2009. The overall implementation rates in spring 2009 improved by 10.6% from fall 2008.

The authors used the qualitative data from the implementation monitoring survey, the telephone survey, and the participants’ opinion survey, to identify specific characteristics from both a curriculum development and programmatic perspective that were associated with higher rates of implementation fidelity. A number of these findings were supported by the study’s quantitative data.

Curricular Characteristics

Several themes emerged from the qualitative data; the coaches and program participants indicated that several key elements were needed: simpler lessons with fewer games, more time to process the health topics as a group, and fewer sub-topics addressed within each health issue. Furthermore, the coaches reported finding lessons with these characteristics more realistic to implement. Following the fall 2008 implementation, the curriculum was revised to incorporate the suggestions. Upon implementing the revised curriculum in spring 2009, data from the formative evaluation measure revealed that fewer coaches had difficulty implementing the lessons (23.6% versus 33.1%). GOT coaches also indicated that lesson activities which incorporated clear and specific directions and learning objectives were easier to implement. As a result of incorporating these elements into the curriculum in 2009, the percentage of head coaches who agreed that the lessons were feasible to implement, that the lesson directions and objectives were clear, and that there was consistency between the teaching and learning activities and objectives, was higher during the spring 2009 implementation (85.0% versus 63.7%). Although the small sample size precludes having sufficient power to conduct tests of significance across GOT sites, these data suggest that incorporating specific characteristics into the GOT curriculum contributed to improving program implementation.

The addition of alternative activities and suggestions for working with less motivated participants also helped the coaches to implement the lessons as intended. In fall 2008, coaches described the need for lesson content to be on par with the developmental stage of the participants. After changes to the curriculum were made to reflect this need, nearly all of the head coaches (93.3%) agreed that, overall, the lesson content was appropriate for the developmental level of the participants.

The data also indicated that, in general, the coaches were more likely to implement lesson activities that allowed time for the participants to share their personal experiences surrounding a health issue. The lessons that reportedly allowed more time for quiet and individual self-reflection helped the participants focus on the lesson’s health learning components. The curriculum’s eighth lesson (“Centering: The Importance of Slowing Down”) addressed deep breathing as a coping mechanism and was the simplest with regard to the number of different activities and allowed time for a prolonged period of quiet self-reflection. Nearly all of the sites during both the fall 2008 and spring 2009 seasons (89.7%) implemented this entire lesson as intended, and this lesson also warranted the fewest revisions post-evaluation.

Programmatic Characteristics

Specific programmatic characteristics needed to successfully implement GOT were also identified and described. One of the main themes that emerged from the analysis was the desire from the coaches to be equipped with resources on each of the health topics. For example, during the telephone surveys, the coaches asked for more information on effectively and safely training adolescent girls for a 5-kilometer race. In this way the coaches felt they would be able to better ensure the participants were prepared for the end-of-season race. Similarly, the coaches also requested that GOTRI provide them with specific information on Internet use, common websites that adolescents use, and information on privacy issues typically associated with social networking websites. Specifically, these coaches stated that the addition of such resources would improve their ability to answer the participants’ questions and ensure that the information being provided to the participants is both current and accurate.

The qualitative data also revealed specific coach training suggestions made by the head coaches (N = 9). Several felt that having opportunities to role-play or practice the lesson activities prior to the beginning of the season would be enormously helpful; particularly for those coaches with less experience working with adolescents. Similarly, some coaches felt that the provision of specific information on the nature of emotional and physical developmental changes that typically take place among adolescents would be useful for coaches learning how to most effectively connect with such a young and vulnerable population.

From a logistical perspective, several coaches and participants stated that GOT’s effectiveness would increase and simultaneously help the participants focus if the program could take place in a space free from other distractions. Many GOT locations share their meeting space with other clubs and sports teams; the qualitative data revealed that this was a source of disruption for both the participants and coaches. The results also underscored the need for a large-enough indoor space as a safe and dry alternative during inclement weather. Lastly, many coaches commented on the need for greater parental involvement within the program. This was highlighted as particularly important, not only to help sustain program attendance rates, but also so that parents could then be equipped to answer questions and serve as an additional resource for the participants.

DISCUSSION

After-school programs can improve the quality of health among youth and contribute to a healthier school environment, but only if they are implemented. Addressing program implementation and circumventing barriers to implementation fidelity can help ensure that evidence-based programs are translated accurately into practice. Overall, the fidelity of program implementation for GOT increased from 2008 to 2009 suggesting that the changes made to address implementation barriers were useful for improving implementation feasibility. Identifying curricular and programmatic characteristics that can be incorporated into after-school programs can help maximize the effectiveness of such initiatives.

From a curricular perspective, the results demonstrated that simpler lessons with fewer games, more time to process the health topics as a group, and lessons that addressed fewer sub-topics within each health issue, were well received by both the coaches and the program participants. In addition to being more engaging for the participants, these lessons were also more realistic to implement. Similarly, lesson activities that incorporated clear directions and comprised teaching and learning objectives that aligned with the lesson activities, allowed the coaches to implement the activities with greater ease. The addition of alternative activities and suggestions for working with less motivated participants also helped to improve the implementation of certain lesson activities. Other changes included improving lesson clarity, adjusting the wording of instructions, modifying the pacing of the activities, and editing lesson content. These were all important features to note because the GOT program, as with many afterschool programs, must remain accessible to the individuals delivering it and incorporate features that encourage the likelihood of successful implementation. This will also promote enthusiasm for the program and encourage volunteers to stay with the organization over multiple seasons.

From a programmatic perspective, the formative evaluation process provided insight as to what types of resources and training materials would be most useful for the coaches and what logistic needs would help improve program implementation. These problems are particularly important to address since, unlike classroom teachers, those implementing afterschool programs traditionally rely on volunteers with limited professional teaching experience. Indeed, this may be one of the most challenging aspects of implementing effective after-school programs. It should be noted that GOTRI as an organization intends to address the barriers to implementation fidelity as a result of the findings from this study.

Limitations

This study relied on self-report data, which are subject to human error and differences in interpretation. In addition, whereas the coaches, council directors, and participants were all provided with clear and consistent directions for completing each of the measures, the authors were not physically present to oversee the data collection process. The authors, therefore, cannot verify that each respondent completed the measures under similar conditions. In addition, a spike in absences among the program participants at the end of each GOT season was observed due to an unforeseen increase in school field trips and limited indoor space availabilities during inclement weather conditions. Thus, the response rates for the program participant surveys were low. It should also be noted that the study took place in the United States; however, GOTRI has GOT locations in Canada as well and intends to expand their international cohort. The conclusions from this study should not be generalized outside of the United States, nor should they be generalized to other after-school programs.

Recommendations for Future Research

Additional research is needed to improve our understanding about ways that after-school programs can effectively contribute to improving health and academic outcomes among youth. Unlike in-school programs, after-school initiatives typically rely on staff and volunteers with comparatively little professional preparation, thus research to identify successful approaches to cultivate professional development is needed. Further formative research is needed to identify and define the kinds of teaching and learning activities that are most and least likely to be implemented as planned, and similarly identify factors that facilitate and hinder fidelity of program implementation. Given the relative paucity of research on the efficacy of such programs, evaluative research is needed to assess the effects of alternative program approaches on public health needs and academic outcomes. Additional research is sorely needed to identify why after school program are successful and, conversely, why they fail to be more effective.

Conclusion

Implementing after-school programs are challenging. Factors that facilitate and hinder successful implementation need to be identified and addressed if after-school programs are going to play a significant role in helping to improve school health outcomes. Though far more research is needed, this study identified specific aspects of GOT that may contribute to improving implementation fidelity, and warrant consideration by those designing and implementing after-school programs for adolescents.

IMPLICATIONS FOR SCHOOL HEALTH

Implementing comprehensive school health programs within the traditional school day is challenging because of academic course requirements and testing needs currently consuming the bulk of the time available. Proper implementation of well-developed after-school programs, however, can effectively supplement academic learning components. Specifically, the time spent participating in after-school programs that are delivered as intended, has the potential to cultivate social and emotional coping skill development among youth, encourage additional physical activity, provide an enriching alternative to high-risk behaviors, and foster the development of healthy social support networks among students. Moreover, research suggests that the development of these behavioral skills will enhance the students’ focus during the school day, minimize the onset of aggressive behaviors, and potentially improve academic performance. Formatively evaluating after-school programs can further help ensure fidelity of program implementation.

Human Subjects Approval Statement

The study was approved by the Institutional Review Board at Columbia University, Teachers College.