The researchers have left the building: what contributes to sustaining school-based interventions following the conclusion of formal research support?

Abstract

BACKGROUND

This study examined the sustainability of New Moves, a school-based program aimed at decreasing weight-related problems in adolescent girls. The National Cancer Institute recognizes New Moves as a research-tested intervention program that produced positive behavioral and psychosocial outcomes.

METHODS

Ten schools participated in the sustainability study. Teachers completed a survey and interview, and research staff observed one physical education (PE) class within 2 years of the study’s completion. Qualitative data were grouped by themes. Frequencies were calculated using quantitative data.

RESULTS

All schools continued all-girls PE classes using New Moves components following the study period. Fewer schools continued the nutrition and social support classroom modules and individual coaching sessions while no schools continued lunch get-togethers. Program components were sustained in both New Moves intervention schools and control schools.

CONCLUSIONS

Programs are most likely to be sustained if they: (1) fit into the current school structure; (2) receive buy-in by teachers; and (3) require minimal additional funds or staff time. Providing control schools with minimal training and intervention resources was sufficient to continue program components if staff perceived the program was important for students’ health and compatible within the school’s existing infrastructure.

Schools are a popular setting for delivering health promotion interventions. They offer many benefits such as a built-in infrastructure for the promotion of health behaviors, including facilities for classroom interventions, personnel capable of being involved in such efforts, and a captive audience of youth. Each year, researchers develop and test various school-based health promotion programs.^1,2 However, questions exist as to what happens when the research study ends. The continuation or sustainability of programs after the research study has ended has emerged as a growing area of emphasis for prevention researchers in a variety of settings.^3–5

Despite the well-accepted premise that sustainability should be an essential step to prevention and intervention program efforts,^6,7 there are few examples in the literature of programs evaluated within research studies that have been continued successfully.^8–13 The sustainability of 3 research-tested programs aiming to increase physical activity in a school-based setting have been examined. All concluded that programs continued to be sustained, however, not at the same intensity as during the research study.^11–13 Key factors that contributed to program continuation included support from school staff¹¹ and teachers who had received training on the program.¹³ Understanding which intervention components are important to a program’s long-term success may enhance the likelihood that a program will continue. Similiarly, understanding the challenges to continued implementation, and proactively developing strategies to address these barriers prior to the end of a research study may increase the likelihood that the program remains.¹⁴

This paper builds on the extant, but limited, research by examining whether a specific school-based program, New Moves, could be sustained following the research study. New Moves was developed to address the needs of adolescent girls who are at risk for weight-related problems. Through an all-girls PE class, New Moves aimed to provide an environment in which girls felt comfortable being physically active and discussing weight-related issues. The program was evaluated with a group–randomized controlled design, with 6 intervention and 6 control high schools in the Minneapolis/St. Paul metropolitan area of Minnesota during 2007–2009. High schools were randomized into intervention and control conditions. The program led to positive changes in weight-related attitudes and behaviors, including improved self-efficacy, body image and self-worth, and decreased sedentary activity, and the use of unhealthy weight-control behaviors; however, the program did not significantly impact body mass index or percent body fat.¹⁵ Survey data collected at the end of the class indicated very high satisfaction among girls, parents, and teachers. The New Moves program is now listed as a research-tested intervention program by the National Cancer Institute, a site that aims to disseminate intervention programs that have been found to be effective using strong evaluation designs.¹⁶

This paper reports on the findings from class observations, teacher surveys, and teacher interviews conducted the year after the research study ended. The aim was to explore and understand the process better, including identification of outcome and potential facilitators and barriers to sustaining a school-based intervention successfully. Data addressed the following research questions: (1) Can a school-based health promotion program with multiple components developed as part of a research study be sustained after the research team leaves? (2) What factors served as facilitators to sustainability? and (3) What underlying challenges did teachers face to continue the program and how were the challenges addressed?

METHODS

Overview of the New Moves intervention

New Moves included 3 main components: (1) one semester of an all-girls PE class taught 4 days/week for one semester; (2) classroom sessions that focused on nutrition and social support modules taught one day/week; and (3) maintenance activities outside of class including periodic individual counseling sessions and weekly lunch get-togethers in the semester following the completion of the New Moves PE class.¹⁵ School PE teachers taught the PE component of the class, while New Moves study staff taught the nutrition and social support components and continued to meet with the girls throughout the remaining school year at weekly lunch get-togethers called lunch bunches and individual coaching sessions. Details on the program and all intervention materials are available at www.newmovesonline.com.

PE teachers in intervention schools were trained on the study protocol and program philosophy. Training included a full-day session prior to beginning the program and a follow-up half-day session mid-way into the semester. Teachers received intervention materials, including a curriculum for girls called Girl Pages, and a Teacher Guidebook. Teachers in control schools were instructed to facilitate an all-girls PE class and did not receive any training on the program. Once the study year ended, teachers from the control schools participated in a one-day training session similar to the initial intervention teacher training and received all study materials.

Participants

The New Moves evaluation study was originally implemented in 6 intervention and 6 control schools during the 2007–2008 and 2008–2009 school years, 3 intervention and 3 control schools per year. The study sample for the current analysis includes teachers from 10 of the original 12 schools. One school closed and the other school did not continue to offer the New Moves program due to budget constraints that led to decreased PE staff and fewer PE classes being offered. Of the 10 teachers, 2 were men and 8 were women. Five taught at the intervention schools and 5 at the control schools. Four of the teachers were new to the school and were not at the schools during the New Moves study. Whereas one of the new teachers was able to attend the training, 3 were not trained on New Moves. The current study was implemented in 2009 and 2010, one to 2 years after the New Moves study ended at each school.

Instruments and Procedures

All data were collected from the PE teacher who currently was teaching the New Moves class and included: (1) a brief written survey; (2) a follow-up interview conducted by a member of the New Moves research team; and (3) one class observation in which a member of the New Moves research team observed one PE class session. The survey, interview questions, and observation checklists were developed by the New Moves research team to address the philosophies and behavioral objectives of the intervention. Survey questions (Table 1) asked about class components currently being implemented. Open-ended interview questions addressed program successes and barriers to continuing the class (Table 2). The interviews were tape-recorded and transcribed. The survey and interviews took about 45 minutes to complete. The classroom observations occurred on the same day as the interview and were arranged to occur during physical activity sessions rather than during times when either nutrition or social support were being taught. During the classroom observations, a member of the New Moves research team completed a checklist to assess the types of activities being conducted and both the degree of participation and activity level of the students. The observation checklist is included in Table 3. During classroom observations, a New Moves team member assessed whether the majority of the students spent at least half of the class time engaged in activities that were of moderate to vigorous intensity. Activities were defined as moderate or vigorous based on a list of general physical activities and their intensities provided by the Centers for Disease Control and Prevention.¹⁷ Teachers provided consent and received a $50 Target gift card in appreciation of their time.

Table 1.

Questions Included in Teacher Survey

How many years has the class been offered for?
Approximately how many girls have gone through the program?
In what grades are the girls allowed to take the class?	9 10 11 12 (circle all that apply)
How many girls are in the class?
Is the class offered as an elective?	Y N
Is your class an all-girls PE class	Y N
Is your class targeted toward inactive girls?	Y N
Do the girls have any choice in the activities which you undertake in the class?	Y N
Do you conduct any field trips with the class?	Y N
Do you do strength training in the class?	Y N (if yes how many times per week)
Do you do any circuit training in the class?	Y N (if yes how many times per week)
Do you even bring in any guest instructors from the community into your class?	Y N
Does the class offer non-competitive activities?	Y N
Do you or anyone else teach any of the ‘Be Fueled’ classroom sessions?	Y N (if yes how many times per week)
Do you or anyone else teach any nutrition in the classroom sessions?	Y N (if yes how many times per week)
Do you or anyone else teach any of the ‘Be Fabulous’ classroom sessions?	Y N (if yes how many times per week)
Do you or anyone else teach any social support sessions?	Y N (if yes how many times per week)
Do you hold lunch bunches?	Y N
Do you hold any individual counseling sessions with the girls?	Y N
Were you trained by the University of Minnesota staff to teach New Moves?	Y N

Table 2.

Guide for Interviews with Teachers

Why did you continue the program?

Do you use the New Moves curriculum?

What do the girls say about the program?

What has been easy about implementing New Moves?

How do you decide what components of New Moves to teach?

How have you sustained the program?

How do you recruit girls for the class?

Who has been instrumental in ensuring that New Moves continues to be offered?

How is New Moves different from other PE classes at your school?

Has being involved with the New Moves program changed your teaching style at all?

What are some of the barriers or obstacles you encountered when implementing the New Moves class?

What are some of the barriers or obstacles in continuing to offer the New Moves class?

What is your ideal vision for the New Moves program?

What type of assistance is necessary in order for New Moves to be successful in a school?

Do you have any additional thoughts or ideas about New Moves that you would like to share?

Table 3.

New Moves Classroom Observation Checklist Completed by New Moves Study Staff during Classroom Observation Following Completion of the Research Study

Is this a girls-only class?			Yes	No
What activities were being conducted in class today?
The girls were supportive of each other	Not at All	A little bit	Most of the time	All the time	N/A
The students were engaged in the class	Not at All	A little bit	Most of the time	All the time	N/A
Students were provided with choices	Not at All	A little bit	Most of the time	All the time	N/A
Most students appeared to enjoy PE class	Not at All	A little bit	Most of the time	All the time	N/A
The teachers used language that is consistent with the New Moves philosophy	Not at All	A little bit	Most of the time	All the time	N/A
The teachers used strategies to keep girls active	Not at All	A little bit	Most of the time	All the time	N/A
Students were encouraged to be active regardless of ability	Not at All	A little bit	Most of the time	All the time	N/A
Students are participating in non-competitive activities	Not at All	A little bit	Most of the time	All the time	N/A
Students are participating in lifestyle activities	Not at All	A little bit	Most of the time	All the time	N/A
Active attendance is taken	Not at All	A little bit	Most of the time	All the time	N/A
50% of class time was spend doing MVPA	Not at All	A little bit	Most of the time	All the time	N/A

Data Analysis

Data from the surveys and classroom observations were used to determine the extent to which the New Moves components were continued at each school. Frequencies of responses were calculated from the teacher surveys and classroom observations using Microsoft Excel version 2007. In addition to examining responses for all 10 schools, the control and intervention schools were compared. The answers to the survey questions and classroom observations were tabulated and compared for the intervention and control schools separately. Responses from the surveys and classroom observations revealed similar findings across conditions. For example, 2 of the 5 control schools and 3 of the 5 intervention schools continued to teach the nutrition and social support classes after the study ended. Therefore, intervention and control schools were combined for all analyses presented here. To determine the facilitators and barriers to continuing to offer New Moves, research staff analyzed qualitative data from the interview transcripts using the thematic analysis technique recommended by Miles and Huberman.¹⁸ The research team created broad content areas and grouped emerging themes into subthemes. Transcripts were then reviewed and coded independently by 2 members of the research team. Discrepancies were discussed until 100% agreement between the members was reached.

RESULTS

Sustainability of a School-based Program

Survey responses and classroom observations indicated that in all 10 schools, an all-girls PE class was offered, including components of the New Moves curriculum shown in Table 4. All teachers indicated they continued to offer strength training as part of the class. Strategies such as incorporating non-competitive physical activities including yoga, dance, or swimming and giving girls choices in their activities, which are important to the New Moves philosophy and taught in training, were evident in classroom observations and reported on surveys. New Moves research staff reported in the majority of classes (9 out of 10) they observed most girls being moderately or vigorously active for at least 50% of the class, a goal of the program. The classroom components, involving teaching nutrition and social support sessions, continued to be implemented in half of the schools; schools that delivered the nutrition sessions also delivered the social support sessions. Maintenance components were sustained less frequently: 3 teachers reported meeting individually with the girls to help set goals although they met less frequently than the 7 meetings that were part of New Moves; and none of the schools continued with lunch bunches.

Table 4.

Continuation of Intervention Components in the 5 Intervention and 5 Control Schools in the 1–2 Years Following Completion of the New Moves Study (N=10)^*

	Component	Description	Frequency	Continued	Did not continue
Physical Education Class	Be Fit/Physical Activity	All-girls’ class Non competitive, lifestyle activities Strength training Taught by physical education teacher	4 times a week throughout 1 semester	10	0
Classroom Component	Be Fueled/Nutrition class	Healthy eating, avoiding dieting, taste testing Taught by study staff	1 time every other week throughout 1 semester	5	5
	Be Fabulous/Social Support	Goal setting, healthy body image, stress reduction Taught by study staff	1 time every other week 1 semester	5	5
Maintenance Phase	Individual Sessions	Individual meeting with study staff (New Moves coach) Set personalized health goals	7 individual meetings throughout 2 semesters	3**	7
	Lunch Bunch	Weekly healthy lunch Informal discussion topic related to healthy eating, social support or physical activity Lead by study staff	1 time a week throughout 1 semester (following completion of the New Moves class)	0	10

^*Program sustainability did not differ by intervention or control conditions so results are combined across condition.

^**The 3 teachers did not do all 7 individual sessions but they did meet individually with students to focus on goal setting.

Teachers provided additional details as to why they continued to offer some components of New Moves and not others in follow-up interviews. The interviews allowed researchers to probe deeper into the facilitators and barriers to sustaining the New Moves class; major themes are described here.

Facilitators to Sustainability

Three major themes emerged regarding facilitators to program implementation: (1) teachers believing in the importance of the program (N=7); (2) feelings of support for the class by school administrators (N=9); and (3) receiving teacher training on the New Moves program from the research team (N=6).

Teachers reported being motivated to continue the class because they liked teaching an all-girls class and they saw benefits for their students. Teachers mentioned they enjoyed working with girls and introducing alternative ways to be physically active, like yoga that were not part of traditional co-educational classes. Teachers felt the girls showed increased self-confidence and greater participation in an all-girls environment leading to high satisfaction from both teachers and students.

Teachers said support from school administrators also contributed to sustainment of the class. School administrators make the decisions about what classes to offer so teachers mentioned administrator’s support as crucial for ensuring the class could continue. Many teachers stated the reason they were able to continue the class was their principal was supportive of the class philosophy, and therefore, willing to make scheduling and staffing plans that included this class. Many also mentioned receiving support to continue to offer the class from department heads or other PE teachers at their school.

Teachers who had attended the New Moves training identified it as a determining factor for continuing to offer the program. Teachers expressed increased confidence and a clear understanding of the program philosophy. Two of the teachers who did not attend the training said they thought receiving training on New Moves would have helped them to deliver the class more effectively.

Barriers to Sustaining the Class and How Teachers Addressed Them

The main themes that emerged as barriers or challenges to program sustainability included: (1) low levels of motivation among girls (N=4); (2) class size (N=4); (3) lack of adequate time and finances (N=9); and (4) teacher turnover (N=4).

The low levels of motivation among girls may have been due to the fact that the program purposefully targeted inactive girls, many of whom did not enjoy physical activity or felt uncomfortable being active. Thus, the teachers talked about not having leaders within the class to set an example of being active for the other girls. To address this challenge, teachers used many of the techniques taught in the New Moves training such as playing music, not keeping score, and offering alternatives to traditional team sports such as soccer played with 4 balls, volleyball with beach balls, or games of tag.

Class sizes that were either too large or too small were discussed as challenges to program sustainability. Large class sizes led to difficulties in class management, whereas class sizes that were too small were not sustainable due to school policies for minimal class sizes. Teachers who had taken an active role in determining their class list were more successful at overcoming this barrier. Teachers reported working with counselors to limit class size. At several schools, teachers identified girls who they had taught in previous classes who might benefit and personally invited them to enroll.

Lack of money and time were commonly mentioned as challenges to offering components other than the PE class. Lack of finances was mentioned as a reason that teachers did not offer guest instructors or hold weekly lunch bunches. Whereas some teachers asked for volunteers to teach yoga or dance, others used videos or asked students to pay a $5 activities fee at the beginning of the class to use for guest instructors’ fees. All teachers felt they did not have time to meet with students outside of class for individual counseling meetings, but some teachers incorporated individual meetings into the class and met with girls a few times over the semester to set goals together. Continuing to offer lunch bunches was not feasible due to a lack of money to buy food as well as lack of time. In addition, schools often had multiple lunch periods and teachers had other responsibilities they were expected to do over the lunch hour.

Teacher turnover was common in the schools. At 4 of the 10 schools, 2 intervention and 2 control schools, a new PE teacher was teaching the class. Three of these 4 teachers had not participated in New Moves training. Two of the new teachers had not received any of the New Moves materials and were not aware of the additional classroom or maintenance components of the program. These teachers were doing a good job teaching non-competitive physical activity to the girls but they had little to work with beyond the course description that talked about an all-girls class offering alternative activities.

DISCUSSION

Careful consideration must be given to understanding and promoting sustainment of successful school-based intervention programs following time- and resource-intensive evaluation studies. This paper sheds light on the types of intervention components that are most likely to be sustained and important variables that may enhance or encumber program sustainability. In all of the 10 schools participating in the sustainability study, New Moves was continued at some level, although not in the full capacity that had existed during the research study. A number of important variables that facilitated or served as challenges to program sustainability in the year following the completion of a randomized-controlled trial were identified. Facilitators to program sustainability included: (1) teachers believing in the importance of the program; (2) feeling supported by school administrators; and (3) receiving training on the program from the research program team. Challenges included: (1) low levels of motivation among girls; (2) class sizes that were either too big to manage or too small to justify offering the class; (3) lack of adequate time and finances; and (4) teacher turnover.

The PE component of the New Moves program was the one most likely to be sustained, followed by the classroom nutrition, and social support components. The maintenance components were less likely to be delivered; individual sessions were offered in some of the schools at lower intensity than during the study and the lunch get-togethers were not offered in any of the schools. This finding suggests the importance of developing program components that can be easily integrated into the existing school structure such as part of PE classes to ease their continuation following completion of the research study. The PE component was the only component continued by all of the schools and also the only component taught by the PE teachers in the original study as New Moves staff taught the classroom and maintenance components. Although 7 of the 10 teachers were trained to deliver the classroom nutrition and social support sessions and the maintenance components, they may not have felt comfortable delivering components outside of the area of PE. During the interviews, one teacher stated that she did not feel confident answering questions about nutrition. Teacher self-efficacy to deliver a program has been shown to be an important factor in whether programs are sustained.⁸ Overall, the further away program components were from the classroom setting and the existing school infrastructure, the more likely modifications and changes in implementation occurred. Other studies have similarly reported that programs that do continue after a research study contain fewer sessions or a reduction in components compared to the original research program.^4,19

The primary reason the schools continued to offer the program was the teacher’s belief that the program successfully met an important need for the students within their school. Others have also found teacher buy-in to be necessary for continuation of programs.^4,10 Han⁷ identified teacher satisfaction with a program and the perceived behavior change by students as 2 of the main factors in program continuation. As a result, getting teachers to support the idea of the class even before the research study begins and listening to their needs is important to sustain the program after the researchers leave. The teacher interviews revealed motivated teachers who witnessed positive results for their students from the class were able to come up with creative solutions to many of the barriers they faced in implementing the program after the research study ended.

An interesting finding from this study was that one year following the research study control schools were just as likely as intervention schools to implement New Moves. Teacher training has been cited as a requirement for the continuation of school-based interventions¹⁰ and teachers from both intervention and control conditions received training by New Moves intervention staff and all intervention materials. Additionally, intervention teachers had the benefit of weekly contact at their school with the study staff during the research study and access to watching the New Moves study staff deliver the classroom and maintenance components. This finding has important implications for researchers who need to make decisions about how much time and money they should devote to making the intervention available to control conditions after the intervention is over. In this case, the relatively limited training and course materials were sufficient for teachers in the control schools to deliver the PE class and classroom activities at the same level as teachers from intervention schools who had received much greater study staff support. The CATCH sustainability study showed similar results in that teachers who received training reported higher levels of program sustainment compared to those who were not trained despite their school’s original assignment to intervention or control condition.¹³ Therefore, we strongly recommend that research studies include both a training component and intervention materials for control conditions following study completion.

This study found a high level of teacher turnover, which may have contributed to limited program implementation. Other sustainability studies^11,20 have identified staff turnover as a barrier to program sustainability. In the current study, at 4 schools the New Moves teacher was new to the school and 3 of the new teachers had not received the New Moves training. For these teachers, the PE class continued to be offered but the teachers were often unaware of the additional components including the classroom lessons in nutrition and social support or maintenance activities. This finding indicates a need for ongoing teacher training opportunities, even after the study funding ends. Alternatives need to be developed to make training more feasible such as online training modules that teachers can access without support from the research team.

Limitations

This contributes to the literature by illustrating how a school-based intervention program is sustained following a research study in both intervention and control schools. The combination of teacher surveys, interviews with teachers, and direct observation of classes allowed for the identification of program components most likely to be sustained, an assessment of factors that facilitated program execution, and the recognition of barriers to the implementation of program components and strategies used to overcome the challenges. However, data are limited in that they were collected only from teachers; students were not interviewed, and administrators and other staff were not included, although they play a key role in continuation of a program. The surveys and interviews were designed to capture information about sustainability without burdening teachers but could be expanded in future studies. Additionally, data were collected at only one point in time and only from a small number of schools, given that only schools that were initially part of the New Moves study were included. With increased attention being directed toward the sustainability of programs implemented and evaluated within research studies,²¹ it is important to address these limitations by gathering data from persons in different roles and by continuing the follow-up assessment over time.

Conclusion

In designing health promotion interventions, careful consideration must be given to factors likely to enhance program sustainability and program effectiveness. Findings from this study suggest that programs most likely to be sustained are those that: (1) fit easily into the current school structure and curriculum; (2) receive the support and buy-in by teachers and administrators; and (3) require minimal additional money or staff time outside of the classroom to implement. Providing high quality training and materials to both intervention and control groups will also promote program sustainment after the study is completed.

IMPLICATIONS FOR SCHOOL HEALTH

Study findings indicate that teachers are interested in teaching innovative, high-quality programs, especially if they observe health benefits for their students. Designing a program that fits into the existing infrastructure of the school or modifying the infrastructure to include the program such as getting an all-girls class included in the course guide increases the likelihood that a program will continue after a research study ends. High teacher turnover rates and the importance of training suggest developing online or video trainings that teachers can access without the research team may increase sustainability of programs. To enhance the likelihood of sustainment over time, researchers should consider building maintenance activities into the classroom component or retooling them so they do not require extra teacher time and money such as using student led support groups or individual goal setting during class. Lessons can be learned from existing research-tested school-based programs that have been successfully sustained and disseminated.²² Appropriate planning, assessment and evaluation by researchers, in collaboration with school administrators and teachers, would result in a better understanding of why and how school-based intervention programs last and significantly impact the health of school-aged children and youth.

Human Subjects’ Approval Statement

This research was approved by the University of Minnesota’s Institutional Review Board and by participating school districts.