Making healthy eating and physical activity policy practice: process evaluation of a group randomized controlled intervention in afterschool programs

Abstract

This study describes the link between level of implementation and outcomes from an intervention to increase afterschool programs’ (ASPs) achievement of healthy eating and physical activity (HE-PA) Standards. Ten intervention ASPs implemented the Strategies-To-Enhance-Practice (STEPs), a multi-component, adaptive intervention framework identifying factors essential to meeting HE-PA Standards, while 10 control ASPs continued routine practice. All programs, intervention and control, were assigned a STEPs for HE-PA index score based on implementation. Mixed-effects linear regressions showed high implementation ASPs had the greatest percentage of boys and girls achieving 30 min of moderate-to-vigorous physical activity (47.3 and 29.3%), followed by low implementation ASPs (41.3 and 25.0%), and control ASPs (34.8 and 18.5%). For healthy eating, high/low implementation programs served fruits and vegetables an equivalent number of days, but more days than control programs (74.0 and 79.1% of days versus 14.2%). A similar pattern emerged for the percent of days sugar-sweetened foods and beverages were served, with high and low implementation programs serving sugar-sweetened foods (8.0 and 8.4% of days versus 52.2%), and beverages (8.7 and 2.9% of days versus 34.7%) equivalently, but less often than control programs. Differences in characteristics and implementation of STEPs for HE-PA between high/low implementers were also identified.

Introduction

Recently, ASPs have been recognized as a setting capable of enhancing children’s health. In response, a variety of healthy eating and physical activity (HE-PA) standards have been adopted by ASPs across the country [1]. Two of the most promising standards are the National Afterschool Association Healthy Eating Standards [2] and the Physical Activity Guidelines created by the California After School Resource Center and California Department of Education [3], hereafter referred to as the HE-PA Standards. These standards call for ASPs to serve a fruit or vegetable daily, and eliminate sugar-sweetened foods and beverages, as well as provide all children 30 min of moderate-to-vigorous physical activity (MVPA) while in attendance at the program. These standards are promising because they are written using clear, measureable language and are tied to key public health recommendations for youth, specifically the accumulation of at least 60 min of MVPA [4] and increasing the consumption of fruits and vegetables daily [5].

One effort to help ASPs achieve these HE-PA Standards is Making HE-PA Policy Practice, a 3-year multi-component group randomized controlled trial including 10 delayed intervention and 10 immediate intervention ASPs [6]. The intervention included professional development training, technical assistance and monthly feedback. Delivery of the intervention was founded in the Strategies-To-Enhance-Practice (STEPS) for HE-PA conceptual framework [6]. The STEPs conceptual framework aims to identify essential building blocks necessary for creating HE-PA friendly environments.

In the summer of 2013 program leaders in immediate intervention ASPs worked with research staff to determine their location on the STEPs for HE-PA continuum, set goals for the upcoming ASP year and plan action steps for meeting those goals. Technical assistance related to the achievement of these goals was provided by intervention staff and included developing detailed ASP schedules and snack menus, identifying low-cost healthy snack items and outlets from which to purchase those snacks, creating a staff expectations document and identifying pre-existing healthy eating education materials for delivery. Program leaders were contacted monthly through phone to provide additional technical assistance. In addition, all intervention program leaders and staff received five professional development trainings, including one initial training and four follow-up booster trainings, from August 2013 to February 2014. Trainings were based on the 5Ms conceptual framework—mission, motivate, manage, monitor and maximize [7]. Trainings focused on communicating the goals and action steps identified by the program leader and providing staff with the skills necessary to reach those goals.

The approach adopted by Making HE-PA Policy Practice diverges from previous efforts to increase children’s HE-PA in ASPs. Previous interventions have focused on the adoption of fixed components (e.g. curriculums) and in turn have produced limited and mixed results [8–17]. Little information is available to explain why these interventions have not produced greater outcomes. What little evidence does exist suggests that staff are hesitant to deliver physical activity curriculum because they do not understand curriculum content, and children do not like the games included [13, 18]. Staff have also indicated that serving healthy snacks is not a priority of ASPs and that the cost of healthier snacks is a major barrier [9, 18]. While these studies provide information on potential barriers to implementation, no ASP studies have directly linked implementation of an intervention to study outcomes.

One way to prevent this ‘black box’ evaluation (i.e. the outcomes of interventions are measured but there is limited understanding of how the intervention achieved those outcomes) is to complete detailed process evaluation [19, 20]. Because there is limited information on why previous interventions have produced limited results and process evaluation can illuminate what is causing/hindering desired changes, detailed process evaluation is an essential component of the Making HE-PA Policy Practice intervention. Further, the multi-component adaptable nature of Making HE-PA Policy Practice makes process evaluation necessary for understanding outcomes and how they relate to implementation [21–23]. The purpose of this evaluation, therefore, was 3-fold. The first objective was to describe the implementation level of physical activity and healthy eating promotion strategies based on STEPs for HE-PA [6], a complex multi-component HE-PA promotion intervention for ASPs. The second objective was to examine the relationship of the level of STEPs implementation to the main outcomes (i.e. children’s engagement in MVPA and snacks served). The third objective was to explore differences between high/low implementation intervention programs and control programs in order to identify characteristics that may be related to implementation level.

Methods

Participants

In partnership with the South Carolina Afterschool Alliance, the University of South Carolina recruited 20 ASPs to participate in an intervention to increase the quality of snacks served and children’s time spent in MVPA during the ASP. The participating programs represent a diverse sample of ASPs from 12 different organizations. For details on participants see Beets et al. [6, 24].

Procedures

The implementation and outcomes described herein represent the first year of a 3-year study and compare intervention programs (i.e. immediate) to control (i.e. delayed). Baseline and post-assesment evaluation observations occurred on eight (four at baseline and four at first year post-assessment evaluation) unannounced non-consecutive weekdays, Monday through Thursday (February to April in 2013 and 2014). During each observation day children’s activity levels, the snack served and the HE-PA environment of the ASP were assessed. Randomization to intervention versus control group was performed following baseline data collection. Detailed information on randomization can be found elsewhere [6].

Process evaluation measures

Process evaluation measures were chosen to measure ASP components deemed critical for increasing children’s HE-PA and based on the STEPs framework. A list of the measures included in this study is detailed below.

Review of records and direct observation

On each observation day, a schedule of daily activities was collected from the program leader. If the program leader did not have a schedule of activities they were asked to describe the schedule for that day. Detailed notes were kept on schedules, including what activities (i.e. physical activities, enrichment, snack, homework) were offered, the times activities began and ended and which staff led these activities. Data collectors also noted on an exit checklist whether or not all staff were wearing clothing appropriate for physical activity (i.e. tennis shoes, clothes that do not restrict movement), a schedule of activities was posted for parents to see, and a snack menu was posted.

Staff behaviors and structure of physical activity opportunities

The System for Observing Staff Promotion of Physical Activity and Nutrition (SOSPAN), a systematic observation instrument [25], was used to track the structure of physical activity opportunities (e.g. presence of lines, elimination games), and staff behaviors (e.g. encouragement, engaged). Observers systematically rotated through areas that were occupied by children and staff from the beginning to the end of each observation day. Five SOSPAN scans were completed consecutively with no break between scans in each area occupied by program participants, prior to the observer moving to the next occupied area. This procedure produced a representative sample of all activities occurring over the course of one ASP day.

Program HE-PA policy

At baseline all programs’ HE-PA policy environments were assessed using the Healthy Afterschool Activity and Nutrition Documentation instrument [26]. Details of the measure are presented elsewhere [6, 26]. In brief, the HAAND assesses policy characteristics of ASPs via document review, interview and observation. The instrument consists of two subscales, the Healthy After-school Program Index for Physical Activity and the Healthy After-school Program Index for Nutrition scale. Within these scales are seven separate domains, written policy, child involvement, screen times or access to vending machine, schedule of physical activity or snack quality, training, curriculum and evaluation. Each domain is rated and scored by a trained HAAND observer. Scores on each domain are then summed to create an overall HAAND score. Higher scores indicate a stronger HE-PA policy environment.

Outcome evaluation measures

Physical activity levels of children

Physical activity levels of children were measured using previously established protocols [24, 27, 28]. On each observation day, all children attending the ASP had equal opportunity to wear an ActiGraph GT3X+ (Shalimar, FL) accelerometer. Accelerometer data were distilled using 5-s epochs [29–31]. Children were fitted with an accelerometer upon arrival to the program and the time was recorded (monitor ‘time on’). Children then participated in the normally scheduled ASP activities. Before a child left the ASP, research staff removed the belt and recorded the time of departure (monitor ‘time off’). Cut point thresholds associated with moderate and vigorous activity were used to distill physical activity intensity levels [32].

Snacks served

Snacks severed were recorded through direct observation using previously established protocols [9, 33, 34]. Prior to the start of snack on each measurement day, a trained observer recorded the foods and beverages served. Food and beverage items served as snacks were classified according to existing valid and reliable categories for snacks and beverages [33, 34]: sugar-sweetened beverages (e.g. soda, powered drink mixed, sport drinks, chocolate milk), sugar-sweetened foods (e.g. Trix® yogurt, cookies, Pop-Tarts, chocolate, frozen treats) and fruits and vegetables (e.g. fresh, frozen, dried).

Assigning STEPs index scores

Foundational and quality of HE-PA index scores, as outlined in Tables I and II, were assigned to all ASPs at both baseline and post-assessment. The foundational index scores target the program leader and are conceptualized as foundational building blocks that integrate HE-PA Standards into routine practice (e.g. program/snack schedule, budget for snacks, scheduling PA opportunities) [6]. The quality index scores focus on quality of the environment and target staff that interact with children daily (e.g. staff model healthy behaviors). All foundational and quality index scores were created by summing the baseline index score, post-assessment index score and change in index score (change =post-assessment − baseline). Each ASP received a total of four index scores across baseline and post-assessment:

1. STEPs physical activity foundational index score (PA-F);

2. STEPs physical activity quality index score (PA-Q);

3. STEPs healthy eating foundational index score (HE-F) and

4. STEPs healthy eating quality index score (HE-Q).

This process for creating index scores was adopted so that performance at baseline and post-assessment along with change from baseline to post-assessment were taken into account. Therefore, the index score was unbiased toward programs that had high or low STEPs index scores at baseline. Further, a combined index score for both HE-PA was not created because the outcomes are unrelated to each other (i.e. a program could be serving excellent snacks and have low levels of PA or vice versa). After index scores were assigned, programs were ranked from highest (20) to lowest (1) on each index score separately. When programs received the same score they were assigned the same rank. A final total index score, the STEPs physical activity (PA-T) or healthy eating total index score (HE-T), was then created by summing the ranks of the foundation and quality index scores. This resulted in a total index score that could range from 2 to 40.

Table I.

Data source, definition, and index coding procedures for the STEPs for Physical Activity promotion variables

Variable	Data source	Definition	Index score coding procedure
			Adequate implementation = 2	Partial implementation = 1	No implementation = 0
STEPs physical activity foundational index score
Schedule level of detail	Document review of schedule	Schedules coded as none, limited detail, or detailed. Detailed schedules clearly define type of activity [snack, homework, physical activity], location, staff roles and materials. Limited detail schedules define at least one of the items above and none indicates the site has no schedule	Detailed schedule	Limited detail	None
Scheduled activity time	Document review of schedule	Total number of minutes dedicated to physical activity opportunities on the ASPs’ schedule	>60 min	60 min	<60 min
Physical activity is the only option	Document review of schedule	Total number of minutes dedicated exclusively to physical activity (i.e. children must choose a physical activity)	>60 min	60 min	<60 min
Number of days all staff wore physically active clothing	Direct observation	Active clothing defined as clothes that would not restrict staff from participating in activity with children (e.g. tennis shoes).	Every day	Some days	No days
Days a schedule was posted	Direct observation	Schedule posted in plain sight of parents at the program	Every day	Some days	No days
STEPs physical activity quality index score
Children wait for their turn	SOSPANa	Children waiting their turn to play/participate	Index scores for the quality of physical activity opportunities were created by summing the percent of SOSPANa scans the six variables were observed and then dividing by 6 to derive the mean amount of time these quality variables were observed separately for both baseline (Spring 2013) and post-assessment (Spring 2014). A change score was then created by subtracting the baseline SOSPAN score from the post-assessment SOSPAN score. The baseline SOSPAN score, post-assessment SOSPAN score and the SOSPAN change score were then summed. All 20 programs were then ranked from the highest to the lowest with the highest score receiving a 20 and the lowest receiving a 1. Programs’ rank was considered their index score
Elimination games	SOSPANa	Physical activity eliminates children from physical activity opportunities as it progresses
Small team games	SOSPANa	Children are divided into several small games (no more than five per team) instead of one large game
Staff playing with children	SOSPANa	Staff member is participating in physical activity with the children
Staff verbally encouraging children	SOSPANa	Staff member verbally promotes physical activity (e.g. keep going, awesome job, good effort)
Girls-only physical activity option	SOSPANa	There is a physical activity opportunity in which only girls participate

^aSystem for Observing Staff Promotion of Physical Activity and Nutrition

Table II.

Data source, definition and index coding procedures for the STEPs for healthy eating promotion variables

Variable	Data source	Definition	Index Score Coding Procedure
			2	1	0
STEPs healthy eating foundational index score
Schedule of daily snack offerings (snack menu)	Direct observation	The site has a snack menu for the week that the observation took place	Everyday	Some days	No days
Following schedule of daily snack offerings	Direct observation	The site served what was on the menu the day of the observation	Everyday	Some days	No days
Snack cost	HAANDa	The program leader is aware of the snack budget	N/A	Clear snack budget	Unclear/no snack budget
Location of snack purchases	HAANDa	The program purchases snacks from one or several retailers on a regular basis	N/A	One or several retailers	No consistent retailer
STEPs healthy eating quality index score
Staff eat snacks with children	SOSPANb	Staff eat fruit or vegetable served for the day with the children during snack time	Everyday	Some days	No days
Staff refrain from eating inappropriate foods in front of children	SOSPANb	Staff do not eat other foods in front of children during program time	Everyday	Some days	No days
Staff refrain from drinking inappropriate drinks in front of children	SOSPANb	Staff drink only water in front of children during program time	Everyday	Some days	No days
Staff deliver healthy eating education	SOSPANb	Staff deliver healthy eating education (e.g. healthy eating coloring sheets/crosswords/word searches, trivia, etc.) to children	More than one observation day	One observation day	No days
Staff encourage children to eat the snack served	SOSPANb	Staff verbally encourage children to take or eat fruit vegetable or water	Everyday	Some days	No days

^aHealth Afterschool Activity and Nutrition Documentation instrument

^bSystem for Observing Staff Promotion of Physical Activity and Nutrition

Analytical plan

Differences in STEPs for HE-PA

Mann–Whitney U, non-parametric ANOVAs were used to identify differences between intervention and control programs on STEPs for HE-PA foundation, quality and total index scores, separately.

Defining high/low implementers

Intervention ASPs were divided into high/low implementers for HE-PA, separately. To determine which programs were high/low implementers, the HE-T and PA-T were visually inspected to identify natural breaks between high/low implementation programs. This procedure is consistent with previous research when there is no criterion cutoff for high implementation [21, 35]. The high implementation group consisted of four programs with HE-T at or above 11 and six programs with HE-T at or below 9. A high implementation group for physical activity was evident with seven programs with PA-T at or above 20 and three programs with HE-T at or below 16.

Linking implementation level and Outcomes

The relationship between implementation as a categorical low/high variable and outcomes (i.e. percent of girls and boys accumulating 30 min of MVPA [24]; days a program served a fruit or vegetable, sugar-sweetened food or sugar-sweetened beverage [33]) were evaluated using random effects general linear models. Models for physical activity accounted for days nested within children nested within ASPs and were estimated separately for boys and girls. Enrollment at the program, total time children attended, child age and ethnicity, total time scheduled for physical activity, percent of population in poverty in the surrounding community (determined by 2010 census data) and setting (i.e. community and faith versus school) were included as covariates in the model. Healthy eating models accounted for observation days nested within programs and were estimated separately for days a program served a fruit or vegetable, sugar-sweetened food, or sugar-sweetened beverage. Covariates in healthy eating models included enrollment, percent of population in poverty in the surrounding community and setting. All analyses were completed using Stata (v.13.0, College Station, TX). Effect sizes were also calculated for the differences between groups, and were corrected to account for the small sample size using the equation:

$$\frac{M_{\text{E}}-M_{\text{C}}}{\text{Sample SD pooled}}x\left(\frac{N-3}{N-2.25}\right)x\sqrt{\frac{N-2}{N}}\hspace{0.5em},$$

where M_E represents the intervention mean, M_C represents the control mean, and N equals the total sample size of 20 programs [36].

Identifying differences between high/low implementers

To identify differences in the characteristics of high/low implementers, descriptive statistics were estimated. To identify differences in both foundation or quality index score strategies that high/low implementers adopted, individual variables composing the four STEPs scores (foundation and quality) were converted into T-scores. If T-scores’ differed by one or more standard deviation (10 points) the variable was considered to differentiate between groups.

Results

Differences in STEPS for HE-PA

Index scores for intervention and control programs are presented in Table III. Overall, intervention programs had statistically significantly higher implementation scores for the HE-T (8.5 versus 6.0) and HE-Q (5.5 versus 2.0) indices than control programs. There were also statistically significant differences between the control and intervention programs on implementation scores for the PA-T (21.5 versus 9.5), PA-F (6.0 versus. 4.0), PA-Q (15.5 versus 5.5).

Table III.

Comparison of intervention and control programs; and high/low implementing and control programs on STEPs for physical activity

	Intervention							Control
	All intervention programs	Range	P value (I v. C)	Implementation level
				High	Range	Low	Range	All controls	Range
Healthy eatinga
STEPs healthy eating total index median scoreb	8.5	(5.0, 12.0)	0.011	12.0	(11.0, 12.0)	7.5	(5.0, 9.0)	6.0	(2.0, 7.0)
STEPs healthy eating foundational index median score	3.5	(2.0, 6.0)	0.616	6.0	(5.0, 5.0)	2.5	(2.0, 4.0)	4.5	(1.0, 5.0)
STEPs healthy eating quality index median scoreb	5.5	(2.0, 7.0)	<0.001	6.0	(5.0, 7.0)	4.5	(2.0, 6.0)	2.0	(1.0, 4.0)
Physical activityc
STEPs physical activity total index median scoreb	21.5	(12.0, 29.0)	<0.001	23.0	(20.0, 29.0)	15.0	(12.0, 16.0)	9.5	(5.0, 17.0)
STEPs physical activity foundational index median scoreb	6.0	(1.0, 9.0)	0.007	6.0	(5.0, 9.0)	5.0	(1.0, 8.0)	4.0	(2.0, 6.0)
STEPs physical activity quality index median scoreb	15.5	(8.0, 20.0)	0.002	17.0	(13.0, 20.0)	10.0	(8.0, 11.0)	5.5	(1.0, 14.0)

Abbreviations: Intervention (I), Control (C), High Implementation (H), Low Implementation (L)

Statistically differences are bolded.

^aPrograms considered high implementers if STEPs healthy eating total index score was ≥11 (n = 4)

^bIntervention is different than control at P ≤ 0.05; based on Mann–Whitney U non-parametric ANOVA

^cPrograms considered high implementers if STEPs physical activity total index score was above ≥ 20 (n = 7)

Implementation level and outcomes

The comparison of activity levels and snacks served in high/low implementation programs and control programs is presented in Table IV. For healthy eating, both high/low implementers served fruits and vegetables on more days (74.0 and 79.1% versus 14.2%) and sugar-sweetened foods (8.0 and 8.4% versus 52.2%) and beverages (8.7 and 2.9% versus 34.7%) on fewer days than control programs. However, differences between high/low implementation programs on healthy eating outcomes were minimal. For physical activity, high implementing programs had the greatest percentage of boys achieving 30 min/day of MVPA at post-assessment (47.3%), followed by boys attending low implementation ASPs (41.3%) and control ASPs (34.8%). This pattern held for girls as well, with 29.3% of girls in high implementing programs achieving 30 min/day of MVPA, 25.0% of girls in low implementing programs and 18.5% of girls in control programs.

Table IV.

Outcome variables relationship to high, low implementing and control program

	Intervention														Control
					Implementation Level										Spring 2013	Spring 2014
	All Intervention programs				High						Low
	Spring 2013	Spring 2014	P value (I v. C)	Cohen’s d (I v. C)	Spring 2013	Spring 2014	P value (H v. C)	P value (H v. L)	Cohen’s d (H v. C)	Cohen’s d (H v. L)	Spring 2013	Spring 2014	P value (L v.C)	Cohen’s d (L v.C)
Healthy eatinga
Percent of days a program serves a fruit or vegetable	7.2	74.5	<0.001	2.47	10.6	74.0	<0.001	0.895	2.29	−0.15	9.1	79.1	<0.001	2.8	18.2	14.2
Percent of days a program serves a sugar-sweetened food	71.2	18.7	<0.001	−1.29	59.5	8.0	<0.001	0.901	−1.2	−0.01	61.4	8.4	<0.001	−1.33	48.7	52.2
Percent of days a program serves a sugar-sweetened beverage	52.8	8.0	<0.001	−0.86	37.3	8.7	0.008	0.176	−0.72	0.13	31.8	2.9	<0.001	−0.99	25.8	34.7
Physical activityb
Percent of girls accumulating 30 min of MVPA	23.1	28.0	0.004	0.65	24.8	29.3	0.009	0.299	0.74	0.28	18.8	25.0	0.055	0.35	21.7	18.5
Percent of boys accumulating 30 min of MVPA	37.1	45.4	0.002	0.86	39.5	47.3	0.006	0.328	0.95	0.26	30.6	41.3	0.032	0.54	36.5	34.8

Abbreviations: Intervention (I), Control (C), High Implementation (H), Low Implementation (L)

Statistically significant differences are bolded

^aPrograms considered high implementers if STEPs healthy eating total index score was ≥11 (n = 4)

^bPrograms considered high implementers if STEPs physical activity total index score was above ≥20 (n = 7)

Differences between high/low implementers

Characteristics of intervention and control, and high/low implementing ASPs are presented in Table V. For both HE-PA, high implementing programs had higher enrollment, were located in communities with lower population poverty rates, provided more annual professional development training that was not related to healthy eating or physical activity for site leaders and had more turnover of site leaders.

Table V.

Characteristics of high & low STEPs for HE-PA implementers in the treatment group and control programs

	All intervention programs	All control programs	Physical activity		Healthy eating
			High Implementation (n = 7)	Low implementation (n = 3)	High implementation (n = 4)	Low implementation (n = 6)
Child characteristics
Race/Ethnicity (%)
White non-Hispanic	64.6	48.4	65.0	67.0	53.0	74.8
African American	29.7	44.7	30. 2	30.9	41.4	21.6
Other	5.7	6.9	4.8	2.1	5.6	3.6
Age (years, M, SD)	7.9 (±1.8)	8.1 (±1.8)	7.8 (±1.7)	8.0 (±1.8)	7.8 (±1.7)	7.8 (±1.7)
ASP characteristics
Average number of children enrolled baseline (M, SD)	89.5 (±52.0)	87.0 (±47.8)	112.9 (±44.8)	39.0 (±16.5)	99.0 (±49.6)	85.2 (±56.9)
Average number of children enrolled post-assessment (M, SD)	76.3 (±45.9)	86.4 (±39.5)	94.7 (±41.7)	33.3 (±18.2)	77.5 (±44.5)	75.5 (±51.1)
Average percent of population in poverty (M, SD)a	13.3 (±15.6)	17.5 (±10.2)	8.3 (±10.2)	25.2 (±21.9)	10.1 (±13.8)	15.5 (±17.6)
Average available indoor space (M, SD)b	13,043 (±12,788)	9,544 (±7,642)	17,567 (±12,860)	2,487 (±971)	N/A	N/A
Average available outdoor space (M, SD)b	199,075 (±229,785)	218,273 (±320,146)	259,096 (±250,652)	59,027 (±84,241)	N/A	N/A
Average used indoor space baseline (M, SD)b	9,400 (±10674)	4,241 (±5,545)	12,567 (±11,439)	2,010 (±1762)	N/A	N/A
Average used outdoor space baseline (M, SD)b	40,488 (±49,690)	67,009 (±56,616)	51,425 (±55,993)	14,967 (±17,595)	N/A	N/A
Average used indoor space post-assessment (M, SD)b	4,422 (±5,249)	2,735 (±3,305)	6,159 (±5,427)	371 (±642)	N/A	N/A
Average used outdoor space post-assessment (M, SD)b	57,521 (±65,496)	65,496 (±37,801)	64,268 (±67662)	41,781 (±63,447)	N/A	N/A
Faith based programs	3	1	3	0	2	1
(located in churches, synagogues, or places of worship)
School based programs	3	6	2	1	1	2
(located at schools)
Community based programs	4	3	2	2	1	3
(located at boys and girls clubs, recreational parks, etc.)
Average healthy eating policy environment (M, SD)d	6.4 (±3.9)	9.1 (±4.2)	N/A	N/A	7.0 (±3.6)	6 (±4.3)
Average physical activity policy environment (M, SD)d	8.1 (±3.5)	9.1 (±2.4)	9.0 (±3.6)	6.0 (±2.6)	N/A	N/A
Program leader and staff characteristics
Average number of program leaders from baseline to post-assessment (M, SD)	1.6 (±1.3)	1.6 (±0.8)	2.3 (±1.0)	1.0 (±0.0)	2.3 (±1.0)	1.7 (±1.0)
Program leader educational attainment
High school diploma	0	2	0	0	0	0
Some college	6	2	4	2	3	3
Bachelor’s degree	2	5	2	0	0	2
Graduate degree	2	1	1	1	1	1
Average number of trainings for program leader per year (M, SD)c	1.5 (±1.6)	2.1 (±1.8)	2.0 (±1.7)	0.3 (±0.2)	2.3 (±2.1)	0.9 (±1.1)
Average number of staff employed (M, SD)	4.5 (±2.6)	2.9 (±1.4)	5.6 (±2.4)	2.0 (±1.0)	4.8 (±2.8)	4.3 (±2.8)
Number of trainings for staff per year (M, SD)c	2.4 (±2.7)	2.0 (±1.9)	1.6 (±1.5)	4.2 (±4.3)	1.5 (±1.2)	2.9 (±3.3)

^aBased on 2010 census data

^bIn square feet

^cAll trainings including HE-PA and other content

^dTaken from the Healthy Afterschool Activity and Nutrition Documentation instrument [26]. Physical activity policy environment range of possible scores 0–20. Healthy eating range of possible scores 0–34.

Based on T-scores high implementing programs differentiated themselves from low implementing programs on seven of the 20 STEPs process markers (Table VI). For physical activity, high implementers had higher scores for schedule level of detail, staff playing with children, staff verbally encouraging children and providing a girls-only physical activity opportunity. For healthy eating, high implementers had higher scores for the number of days following schedule of daily snack offerings, following the schedule, snack cost and staff deliver healthy eating education.

Table VI.

Average rank of programs on variables composing index scores by treatment and implementation group

	Intervention						Control programs	SD
	All intervention programs	SD	High implementation	SD	Low implementation	SD
STEPs physical activity foundational index score
Schedule level of detail c, d	50.0	(14.5)	54.4	(11.6)	39.7	(17.8)	50.0	(0.0)
Scheduled activity time b	54.9	(7.4)	56.3	(9.5)	51.9	(14.8)	45.1	(10.1)
Physical activity is the only option	53.7	(8.9)	53.9	(12.4)	49.4	(7.6)	46.3	(10.0)
Number of days all staff wore physically active clothing	53.3	(8.6)	55.2	(6.6)	50.4	(8.4)	46.7	(10.6)
Days a schedule was posted	51.4	(9.5)	52.6	(9.6)	47.2	(10.1)	48.6	(10.8)
STEPs physical activity quality index score
Children wait for their turn	52.3	(11.7)	51.3	(10.5)	54.6	(16.5)	47.7	(8.0)
Elimination games	52.8	(5.2)	51.6	(5.6)	55.7	(3.2)	47.2	(12.9)
Small team games	52.2	(11.4)	52.5	(8.8)	51.4	(18.7)	47.8	(8.4)
Staff playing with children a, b, d	55.7	(10.1)	60.1	(8.1)	45.3	(5.9)	44.3	(6.1)
Staff verbally encouraging children a, b, d	55.3	(9.8)	58.3	(8.4)	48.2	(10.7)	44.7	(7.3)
Girls-only physical activity option b, d	54.1	(10.9)	58.3	(8.6)	44.5	(10.9)	45.9	(7.3)
STEPs healthy eating foundational index score
Schedule of daily snack offerings (snack menu) b, d	53.2	(0.0)	58.7	(0.0)	44.2	(9.4)	46.8	(13.7)
Following schedule of daily snack offerings b, d	50.7	(11.8)	62.1	(6.7)	43.1	(6.9)	49.3	(8.5)
Snack cost d	50.0	(10.3)	56.4	(0.0)	45.7	(11.6)	50.0	(10.3)
Location of snack purchases	53.2	(0.0)	53.2	(0.0)	53.2	(0.0)	46.8	(13.7)
STEPs healthy eating quality index score
Staff eat snacks with children a, b, c	56.7	(8.5)	61.5	(0.0)	53.4	(9.9)	43.3	(6.4)
Staff refrain from eating inappropriate foods in front of children a, b, c	56.0	(6.3)	58.0	(0.0)	54.6	(8.1)	44.0	(9.6)
Staff refrain from drinking inappropriate drinks in front of children a, b, c	56.0	(9.6)	52.0	(11.5)	58.6	(8.1)	44.0	(6.3)
Staff deliver healthy eating education a, b, d	55.2	(12.3)	63.4	(14.3)	49.8	(7.7)	44.8	(0.0)
Staff encourage children to eat the snack served b	54.4	(10.8)	59.6	(7.3)	51.0	(12.0)	45.6	(7.1)

Scores represent the mean T-scores of programs within each group. Higher scores represent higher implementation

^aDifference between intervention and control programs of more than one standard deviation

^bDifference between high implementation and control programs of more than one standard deviation

^cDifference between low implementation and control programs of more than one standard deviation

^dDifference between high and low implementation programs of more than one standard deviation

High implementing programs differentiated themselves from control programs on 10 STEPs strategies. For physical activity, high implementers had higher scores for scheduled activity time, staff playing with children, verbally encouraging and girls-only opportunities. For healthy eating, high implementers had higher scores on having a daily snack menu, following the daily snack menu, staff eating the snack with children, staff refraining from eating unhealthy foods in front of children, staff delivering healthy eating education and staff encouraging children to eat the provided snack. Low implementing programs were differentiated from control programs on four STEPs strategies. Three HE-Q strategies favored the low implementing programs (staff eat snacks with children, staff refrain from eating inappropriate foods in front of children and staff refrain from drinking inappropriate drinks in front of children), while one PA-F strategy favored the control programs (schedule level of detail).

Discussion

This article describes the implementation of STEPs for HE-PA, core strategies of the Making HE-PA Policy Practice intervention. It also explores the relationship between implementation and study outcomes, and differences between high/low implementing programs. Both high/low implementing programs had a positive impact on the percent of children accumulating 30 min/day of MVPA and the frequency that fruits and vegetables, sugar-sweetened foods, and beverages were served. However, it appears that implementation level impacted physical activity outcomes more so than healthy eating outcomes with high implementing programs having greater improvements in children’s physical activity, whereas low implementing programs experienced similar healthy eating outcomes as high implementing programs. Further, based on this study’s findings, we were able to identify STEPs strategies implemented more frequently and characteristics that differentiated high implementing from low implementing programs and high/low implementing programs from control programs.

Implementation level and outcomes

For this study, there was a clear stair-stepping pattern for both boys’ and girls’ physical activity from low to high implementation programs and compared with control programs. Therefore, both low and high levels of implementation increased boys and girls accumulation of MVPA during ASP time, but higher implementation of the intervention resulted in greater improvements, which is consistent with the literature on implementation [21–23]. However, this was not the case for healthy eating, as both high/low implementation programs were similar on all three healthy eating outcomes, though markedly better compared with control programs. Low levels of implementation of STEPs for HE-PA can produce desired improvements in the foods and beverages served. This is particularly encouraging since program leaders and staff often believe that changing the quality of snacks served in a program will be difficult [13, 18]. This finding suggests that, contrary to staff beliefs, changing the snacks served to healthier options is easier than increasing physical activity promotion in ASPs. This may be because changes to snacks served can be accomplished by working with the single person who purchases snacks, whereas changes to physical activity promotion requires the site leader and all staff to work collaboratively to create a more physical activity friendly environment (i.e. play active games during the program, encourage children to be active during play).

Differences between high/low implementers

When comparing characteristics of high/low implementers, several differences emerged. Some of the most interesting findings are that high implementers operated programs with higher enrollment, were located in communities with lower population poverty rates, had site leaders who received more annual training not related to HE-PA and experienced more site leader turnover. It is unclear why larger programs were more successful in the implementation of STEPs. One explanation may stem from the complexity inherent in running larger programs, with such programs having a large number of staff that require clear directions (i.e. daily program schedules) to ensure the children attending are occupied over the 3 h of operation. Similarly, given the large number of children attending, having a snack menu that informs staff of what will be served each day is important to ensure appropriate snack servings are available. Thus, for a given enrollment size, there appears to be inherent benefits to developing, implementing and adhering to a snack menu and program schedule on most, if not all, of the days.

Programs in communities with higher poverty rates were also disproportionally represented in the low implementation group. This finding is consistent with school-based health promotion findings, where teachers in areas of low socioeconomic-status have been shown to have higher rates of emotional exhaustion and teacher burnout compared with schools in areas of high socioeconomic-status [37, 38]. In turn, high levels of exhaustion or burnout are related to low implementation of new programming [38, 40]. If ASP staff in areas with high poverty rates experience the same phenomenon, this may explain why they were less apt to implement changes to routine practice. However, this appears to only have affected the outcomes associated with physical activity and not healthy eating.

Another characteristic that differentiated high implementing programs from low implementing programs was the amount of training program leaders received. Leaders in high implementing programs received two or more trainings per year while low implementing program site leaders received less than one training per year. It is important to note that these trainings covered content unrelated to healthy eating or physical activity such as policies and procedures, child safety and child development. While these trainings were unrelated to healthy eating or physical activity, regular training is an indicator of organizational capacity [41], which in turn is linked to program implementation [19, 42].

Interestingly, while staff turnover is high in all ASPs because many staff are transitional, part-time employees (high school and college students) [24], high implementing physical activity and healthy eating programs experienced more turnover at the program leader level than low implementing programs (2.3 versus 1.0 and 2.3 versus 1.7, respectively). This finding suggests that STEPs for HE-PA might be robust to program leader turnover. Previous interventions targeting children’s physical activity in ASPs have also cited staff turnover as a challenge [15–17]. Similar to this study, these studies trained staff. However, the focus of these trainings was on delivering a specific curriculum. Conversely, rather than training staff to deliver prepackaged curricula, this study worked with program leaders to establish more fixed components essential to running a quality program (e.g. creating a detailed schedule of daily activities, creating a snack menu) and to infuse HE-PA within those components. This strategy may be more resistant to site leader or staff turnover because the infrastructure of quality programming is put into place, allowing the new site leader and/or staff to pick up where the previous site leader and/or staff left off. Still it is possible that additional trainings increased site leader and staff burden and may have contributed to site leader turnover. However, if ASPs are to meet policy goals, ongoing training is essential to introduce new program leaders to STEPs for HE-PA.

Several strategies differentiated high implementers from low implementers and control programs as defined in this study. These were staff playing with children, verbally encouraging children to be active and offering girls-only activity opportunities. These components, which were embedded within the STEPs framework, appear to be primary drivers of the improvements in activity levels and should be emphasized in future studies. Further, identifying programs that are likely to be low implementers (e.g. smaller programs in high poverty areas) and working with these programs to develop a detailed schedule and daily snack menu, encourage staff to play with children and helping staff to encourage children to eat the healthy snacks, may enhance the effectiveness of STEPs for HE-PA in the future.

This study has several strengths including a group randomized controlled study design, using statistical modeling to depict implementation level’s effect on the study’s main outcomes, the use of objective measures to document process and main outcomes of the intervention and the diversity of the sample. A limitation of this study is the small number of programs that participated (n = 20). Further, while program visits occurred on non-consecutive, unannounced program days, staff may have reacted to observation during program time. However, reactivity to the observation would have been equally distributed across intervention and control programs, and the unannounced nature of the observations minimized this threat to internal validity. Another limitation of this study is the use of unweighted index score. Using an index score that weights all the included variables equally does not take into account the possibility that some practices influence children’s HE-PA more than other practices. However, the finding that both the HE-T and PA-T index scores were related to study outcomes lends credibility to using these index scores. Finally, this study does not include child level consumption data. Therefore, it is unknown if serving healthier snacks led to child consumption of those snacks, or just increased snack waste. However, studies have shown that when children are provided healthy options, in the absence of unhealthy options, the majority of children will consume the healthier foods [43].

Lessons learned

Key findings of this study include:

• Changing the snacks served in ASPs can be accomplished with relatively little change to routine practice (i.e. low implementation group experienced equivalent increases in quality of snacks served as high implementation group), however, it is often perceived as a challenging process by staff. Conversely, changes to children’s physical activity levels may take more concerted and coordinated effort by all staff (i.e. apparent stair stepping effect in outcomes as programs implemented more STEPs strategies).

• Programs that are smaller and in lower SES areas are likely to have more difficulty implementing strategies to increase HE-PA. Therefore, these programs may need additional support throughout the intervention process. Specifically, these programs may need help-implementing foundational STEPs components such as developing a detailed schedule of program activities and/or a snack menu.

• High implementing programs differentiated themselves from low implementers by implementing the following STEPs components: staff playing with children, verbally encouraging children to be active and offering girls-only activity opportunities. These strategies appear to be related to changes in children’s physical activity and should be emphasized in future interventions.

• Strategies for increasing HE-PA that are easily infused into routine practice are successful. STEPs for HE-PA was designed to help practitioners easily identify modifiable levers for program change that required little to no cost and could be easily integrated into routine practice. Results from this study indicate that programs were able to implement STEPs for HE-PA and in turn increase children’s physical activity levels and the quality of snacks served.

• ASPs experience high amounts of staff and site leader turnover. Therefore, ASPs require ongoing support for implementing HEPA standards.

This information is critical as Making HE-PA Policy Practice continues into a second year of intervention. Further, future interventions that utilize similar processes as STEPs can incorporate these lessons learned into their design.

Conclusion

The results of this study further illuminate how implementation of STEPs for HE-PA are related to children’s physical activity and the quality of snacks served. By conducting this evaluation, characteristics of low implementation programs were identified along with strategies that they struggled to implement. As programs across the country begin to work toward achieving healthy eating and/or physical activity standards, the identification of strategies for meeting policy goals, like those presented herein, will be essential.

Funding

This work was supported by National Institutes of Health [Grant: HL11278702] from the NHLBI.

Conflict of interest statement

None declared.