Impact of implementation factors on children’s water consumption in the Out-of-School Nutrition and Physical Activity group-randomized trial

Abstract

The Out-of-School Nutrition and Physical Activity afterschool intervention substantially increased water intake during snack time with stronger effects for programs with kitchens, low child-to-staff ratios, experienced site directors, and improved support from schools, highlighting the importance of contextual factors in planning, implementing, and disseminating obesity prevention efforts.

As childhood obesity has emerged as a major chronic health condition, translating nutrition and physical activity evidence into effective strategies for change in real-world settings has become a top priority. Our study aims to identify actionable implementation factors that affect children’s water consumption in the Out-of-School Nutrition and Physical Activity (OSNAP) initiative—a group-randomized trial designed to create nutrition and physical activity policies and practices that promote child health.

In this chapter, we investigate the key implementation determinates of practices that promote water intake. National data suggest that children are not consuming enough water.¹ Experimental evidence has linked increased water consumption to obesity prevention, and the National AfterSchool Association named serving water as one of its nutrition standards.² Also, water access has been documented as an area of inequity in the city of Boston.³ Significant changes in water consumption were found in the main effects of the OSNAP trial.⁴

This study responds to a call to action put forth by the Early Assessment of Programs and Policies to Prevent Childhood Obesity to use practice-based evidence to determine how to improve nutrition outcomes in afterschool settings. Durlak and DuPre’s multilevel ecological Framework for Effective Implementation, adapted for afterschool in Figure 5.1, serves as the conceptual framework for this investigation.⁵ Like other implementation and dissemination theories, this framework asserts that innovation characteristics, organizational capacity, provider characteristics, and community context each contribute to effective implementation of interventions.⁶ Innovation characteristics in this framework are aspects of the program or intervention that are being newly introduced. Organization capacity refers to the structural characteristics of the setting, in this case the afterschool site, that are responsible for guiding the implementation of a new program. Provider characteristics are traits of the nonresearch staff who implement the intervention—in our study, the afterschool directors and staff. Finally, community context refers to the local environment, including school and neighborhood demographics, into which the intervention is being introduced.⁷

Figure 5.1. Multilevel conceptual framework of afterschool implementation predictors adapted from the Framework for Effective Implementation.

Source: Durlak, J. A., & DuPre, E. P. (2008). Implementation matters: A review of research on the influence of implementation on program outcomes and the factors affecting implementation. American Journal of Community Psychology, 41(3–4), 327–350.

Although the domains investigated in this study have been applied to studies of implementation in schools, we know of only one study that has looked at the influence of these factors on implementation of an afterschool physical activity and nutrition intervention.⁸

The objective of this study is to investigate the influence of implementation factors on an intervention to increase children’s water consumption using data from the Boston-based OSNAP trial. Given that this was an effectiveness trial conducted in a real-world setting and delivered by existing sites and staff, implementation differences emerged. We modeled interaction effects to test the impact of a number of implementation factors on the effectiveness of the intervention. We hypothesized that the organizational capacity of the afterschool programs, the characteristics of the providers, and the community context would affect changes in children’s water consumption, as shown in Figure 5.1.

Methods

Data are from the OSNAP group-randomized controlled trial. The study was approved by the Harvard School of Public Health Office of Regulatory Affairs and Research Compliance.

Research design

Twenty Boston afterschool sites—ten intervention sites and ten matched controls—participated in the study from fall 2010 (baseline) to spring 2011 (follow-up). At baseline, parents or guardians provided written informed consent, and trained research assistants obtained verbal child assent for a plate waste protocol to evaluate the effect of the intervention on water consumption. We conducted direct observations of water consumption during snack time on two days at baseline and two days at follow-up. Site director questionnaires were also collected at baseline and follow-up. Further details on the design of the study, including a randomized controlled trial flow chart, are available in our earlier publication.⁹

Intervention

The OSNAP intervention was designed to follow the social ecological model.¹⁰ Activities targeted change at the school community, organizational, interpersonal, and individual levels. Increasing water consumption was one of ten health goals of the intervention. At the community (school district) level, we worked with the Boston Public Schools (BPS) Food and Nutrition Services department to change the foods and beverages served for after-school snack. To promote water consumption, food service staff filled insulated jugs with tap water or pitchers with bottled water from coolers, depending on facility infrastructure. A series of three three-hour learning collaborative trainings with afterschool staff was conducted over the course of the school year to create changes at the organizational afterschool site level.¹¹ The training included education on the health effects of drinking water and sugar-sweetened beverages. It also included skill-building activities on how to sanitize water pitchers, write policies, and model healthy behaviors. Materials including tips and success stories (available at osnap.org) were used to help staff create targeted action plans with specific practice, policy, and community strategies for change. Intervention educational activities on water consumption, available at foodandfun.org, were delivered to families and children by existing afterschool staff.¹²

Measures

Water intake, the primary outcome of this study, was measured by direct observation. Implementation measures at the program level were collected to determine the impact of the implementation context on the effectiveness on the intervention. Individual demographic data were measured to adjust for potential confounding.

Water intake

Every day, trained observers recorded whether water was served at snack time along with the size of cup used. Child water consumption was measured by direct observation on two days at each site at baseline and at follow-up. Data collectors assessed the portion of water children consumed from cups during the snack period. Ratings of none, some, most, or all were converted to approximate percentages: 0 percent, 33 percent, 66 percent, and 100 percent. This direct observation was moderately correlated with a weighing criterion method in which cups with water were measured before and after consumption.¹³

Implementation measures

Data on organizational capacity, provider characteristics, and community context were collected using administrative records, census data, and site director surveys. Since this was a group-randomized controlled trial, all intervention sites were assumed to have the same intervention and accompanying training and technical assistance. Therefore, variation in the innovation—one of Durlak and DuPre’s core implementation factors—could not be investigated.

Aspects of organizational capacity captured on the survey included child enrollment; number of staff; nutrition as a program priority (on a five-point scale); and the degree to which agency support, funding, time, and space were barriers to nutrition change. Site directors rated all perceived barriers on a three-point scale. Sites were classified as having an on-site kitchen foodservice operation if administrative records indicated snacks were prepared in a kitchen at the school where the program was situated.

Provider characteristics on the survey included the site director’s education and years of experience. Site directors were asked how many staff members left the site between baseline and follow-up and the degree to which they perceived turnover as a barrier to nutrition change. Shared commitment was determined by the number of staff whose names appeared on attendance lists at the learning communities and who were assigned tasks on action plans.

Community context was measured primarily by BPS administrative records and census data. School data included the percentage of students who qualified for free or reduced-price lunch and the racial and ethnic demographics of the student body during 2010–2011. 2010 census data on race and ethnicity and on homeownership in the program’s census block were obtained from the Boston Redevelopment Authority’s MyNeighborhood Census Viewer (http://hubmaps.cityofboston.gov/MyNeighborhood). On their survey, site directors reported the degree to which lack of school support was a barrier to nutrition change.

Sociodemographic measures

At baseline, parents reported child age, gender, and race and ethnicity on consent forms. Parents were given an open response for race and ethnicity; investigators subsequently categorized responses into seven categories: White, Hispanic/Latino, Black/African American, Asian, Cape Verdean, Black Hispanic, and multiracial.

Analysis

We conducted descriptive analyses of the distribution of each implementation factor and estimated correlations among the factors. We found that the perception of “authority to make decisions” as a barrier to nutrition change was highly correlated with four more concrete perceived barriers: funding, time, school support, and space. Therefore, this variable was omitted from analyses.

We conducted multivariable regression analyses to assess the impact of the afterschool intervention on increases in water consumption, constructing person–period data sets that took into account the clustering of repeated dietary intake observations within each child over time within afterschool sites.¹⁴ The longitudinal sample consisted of data from children who had at least one day of baseline data and one day at follow-up.

We accounted for the clustering of observations within children (repeated measure over time) and programs (children within afterschool sites) using SAS PROC MIXED (SAS Institute, Cary, NC).¹⁵ Our random intercept models assumed compound symmetry and used the repeated function to account for repeated observations nested within children. We included terms for the period (baseline or follow-up), intervention status, and a period by intervention interaction. This period by intervention interaction was the parameter for main effect of the intervention. All models controlled for the following potentially confounding variables: age in years, gender, six of the seven race and ethnicity categories (Black/African American, Hispanic/Latino, Asian, multiracial, Cape Verdean, and Black Hispanic), and day of data collection (first or second day).

To investigate the impact of each implementation factor on the intervention effect, first we fit a series of independent models. We added the implementation factor and an interaction effect to the main effects model. The effect estimate and p value for the interaction term indicates whether that implementation factor altered the intervention main effect. We used p =.01 as the level of significance in this analysis due to multiple comparisons. The following factors were included as continuous variables: percentage of white children at the school, percentage of white people on the census block, percentage of children eligible for free or reduced-price lunch, percentage of homeowners on the census block, number of staff who left the site (turnover), and the number of staff who participated in intervention activities—for example, they attended learning community meetings and were listed on action plans. Binary variables included the presence of an on-site kitchen, site director education (college versus high school), site director experience of two years or more, child-to-staff ratio of greater than 10:1, and enrollment size of less than fifty children. All barrier variables reported on the site director survey were coded 0 for “never” a barrier, 1 for “sometimes” a barrier, and 2 for “often” a barrier. Barrier scores were converted to change scores, so that a site director who reported funding as a barrier “sometimes” at both baseline and follow-up would have a funding change score of 0, while a site director who reported funding as a barrier “sometimes” at baseline but “never” at follow-up would have a funding change score of –1. Nutrition as a program priority was rated on a five-point scale, with 1 being the top priority and 5 being the bottom priority.

The final model included a main effect and interaction for on-site kitchen, as this was the strongest interaction factor influencing water consumption results in the independent regressions. Next, one at a time, we added main effects and interactions for all implementation predictors that were significant (p <.01) in the independent regressions. Factors with implementation data from all twenty sites were added before any factors with incomplete data. The final model includes any factors from the independent models that remained significant (p <.01) when tested along with other implementation factors. All analyses were conducted using an intention-to-treat protocol with participants analyzed in their original condition.¹⁶

Results

At baseline, six hundred children, or 52 percent of those attending the twenty afterschool sites, agreed to participate. These analyses use a longitudinal sample of four hundred children who had complete data on age and gender and at least one day of water consumption data at each time point. Participants were 5–12 years old, averaging age 8. About half of the study participants were girls, and most parents identified their children as Black, African American, Hispanic, or Latino. There were no age or gender differences by intervention status. Intervention sites had fewer White and Black children than did control sites (p <.05). In fall 2010, children were consuming very little water at snack, averaging 0.27 (SD 0.82) ounces in intervention sites and 0.17 (SD 0.80) ounces in control sites.

Data on the implementation predictors under investigation are presented in Table 5.1.

Table 5.1.

Community, provider, and organizational predictors of water intervention implementation in twenty Boston afterschool programs

Construct	Measure	N	Intervention mean (SD)	Control mean (SD)	p value
Community context
School race/ethnicity	% White in school student body	20	7% (6%)	11% (10%)	0.14
Neighborhood race/ethnicity	% White on census block	20	28% (34%)	38% (32%)	0.80
	% eligible for free/reduced lunch	20	84% (7%)	79% (14%)	0.04
School poverty level	% households inhabited by homeowners on census block	20	30% (21%)	55% (26%)	0.56
Neighborhood homeownership	School support as a barrier to nutrition change, baselinea	20	0.60 (0.70)	0.20 (0.42)	0.15
Supportive school environment	School support as a barrier to nutrition change, follow-up	20	0.50 (0.53)	0.10 (0.32)	0.06
Provider characteristics
Education	Proportion of site directors with a college degree or higher	20	90%	70%	0.22
Experience	Average years as site director	20	3.83 (3.67)	3.21 (4.97)	0.76
Turnover	Staff turnover between baseline and follow-up data collection	19	0.30 (0.67)	0.89 (1.05)	0.21
	Turnover as a barrier to nutrition change, baseline	19	0.22 (0.44)	0.20 (0.42)	0.89
	Turnover as a barrier to nutrition change, follow-up	20	0.50 (0.53)	0.10 (0.32)	0.14
Shared commitment	Average number of staff at learning community meetings	20	1.83 (0.63)	0.00 (0.00)	< 0.01
	Average number of staff assigned tasks on action plans	20	2.30 (1.83)	0.00 (0.00)	< 0.01
Organizational capacity
Kitchen facilities	Proportion of programs with an on-site kitchen	20	40%	30%	0.85
Change valence	Nutrition as a program priority, baselineb	18	2.50 (0.71)	2.50 (0.76)	0.83
	Nutrition as a program priority, follow-up	19	2.50 (0.33)	2.76 (0.67)	0.30
Organizational support	Agency support as a barrier to nutrition change, baseline	19	0.44 (0.53)	0.00 (0.00)	< 0.01
	Agency support as a barrier to nutrition change, follow-up	19	0.44 (0.53)	0.00 (0.00)	< 0.01
Funding	Funding as a barrier to nutrition change, baseline	20	1.00 (0.67)	0.70 (0.82)	0.54
	Funding as a barrier to nutrition change, follow-up	20	1.20 (0.92)	0.70 (0.48)	0.07
Program facilities	Space as a barrier to nutrition change, baseline	20	0.70 (0.95)	0.50 (0.85)	0.75
	Space as a barrier to nutrition change, follow-up	20	0.40 (0.70)	1.00 (0.81)	0.65
Program size	Average number of children enrolled	20	62.1 (36.7)	82.7 (85.9)	0.02
	Average child-to-staff ratio	20	9.45 (2.42)	8.59 (2.09)	0.66
Time	Time as a barrier to nutrition change, baseline	20	0.70 (0.67)	0.30 (0.48)	0.33
	Time as a barrier to nutrition change, follow-up	19	0.78 (0.44)	0.60 (0.52)	0.67

^aAll barriers to nutrition change rated on a three-point scale: 0 = never, 1 = sometimes, 2 = always.

^bNutrition as a program priority rated on a five-point scale with 1 being the top priority and 5 being the bottom priority.

Organizational capacity

Funding was perceived as the most frequent organizational barrier to nutrition program change at baseline. The average child-to-staff ratio at baseline was approximately 9:1. Control sites had a higher enrollment than intervention sites (p =.02). Thirty-five percent of schools had an on-site kitchen.

Provider characteristics

In terms of provider characteristics, 80 percent of site directors had a college degree. They averaged 3.5 years at their site. Thirty-seven percent of site directors reported that staff members had left the site between baseline and follow-up, while 21% reported turnover as a barrier to nutrition change at baseline.

Community context

The average percentage of white residents in a site’s neighborhood was 33 percent, and the average percentage of households inhabited by homeowners was 42 percent. The percentage of children eligible for free or reduced-price lunch was slightly higher at intervention (84 percent) compared to control (79 percent) schools (p = .04). Thirty-five percent of site directors reported school support as a barrier to nutrition change at baseline.

Main intervention effects on water consumption

There was a strong main effect of the OSNAP intervention on children’s water consumption. After controlling for the individual-level factors of age, gender, race and ethnicity, and day of data collection, children in intervention sites had greater increases in water consumption with snacks (1.49 ounces, 95% CI 1.21–1.78, p <.0001) than children in control sites.

Effect of implementation predictors on water consumption change

Table 5.2 shows the results of the series of independent regression models that demonstrate the impact of organizational capacity, practitioner characteristics, and community context implementation factors on the intervention results. Results of the independent regression models we conducted indicate that four implementation predictors per snack had a significant impact on the intervention results: on-site school kitchen (1.73 ounces, 95% CI 1.30–2.15), increase in the perception that school support was a barrier to nutrition change (−0.48 ounces, 95% CI −0.73 to −0.22), site directors having two or more years of experience (1.08 ounces, 95% CI 0.69–1.47), and child-to-staff ratio of more than 10:1 (−1.19 ounces, 95% CI −1.62 to −0.76). p values of interaction terms were all <0.001. For instance, children who attended sites led by a director with two or more years of experience consumed an average of 1.08 more ounces of water apiece at each snack than those who attended a site with a less experienced site director, independent of the other predictors. Children who attended a site with a child-to-staff ratio greater than 10:1 consumed 1.19 fewer ounces of water than children who attended a site with a lower child-to-staff ratio, independent of the other predictors.

Table 5.2.

Estimated main and interaction effects of community, provider, and organizational predictors on ounces of water consumed per day among 400 children in intervention (n = 10) and control sites (n = 10)^a

	Main effectb	Independent regressionsb	Final regression model
Intervention	1.49 (1.21,1.78) p < .0001		0.43 (0.02, 0.84) p < .04
Community context
Intervention		1.70 (1.31, 2.10) p < .001
School % White (N = 20)		-1.19 (−5.65, 3.26) p = 0.60
School % White*intervention		−2.98 (−6.72, 0.77) p = .12
Intervention		1.67 (1.33, 2.00) p < .001
Block % White (N = 20)		−0.20 (−1.29, 0.89) p = .72
Block % White*intervention		−0.68 (−1.35, −0.01) p = .05
Intervention		4.16 (1.31, 7.01) p < .01
School income (N = 20)		−0.20 (−3.72, −3.32) p = .91
School income*intervention		−3.16 (−6.52, 0.20) p = .07
Intervention		1.61 (1.20, 2.02) p < .001
Block homeownership (N = 20)c		0.78 (−0.71, 2.26) p = .30
Block homeownership* intervention		−0.41 (−1.45, 0.63) p = .44
Intervention		1.10 (0.83, 1.38) p < .001	–
School barrier change (N = 20)c		−0.29 (−0.68, 0.11) p = .15	−0.36 (−0.74, 0.02) p = .06
School barrier change*intervention		−1.35 (−1.56, −1.15) p < .01	−0.48 (−0.73,−0.22) p < .001
Provider characteristics
Intervention		1.20 (0.42, 1.98) p < .001
Education (N = 20)		0.19 (−0.69, 1.07) p = .67
Education*intervention		0.31 (−0.47, 1.10) p = .43
Intervention		0.42 (0.05, 0.79) p = .03	–
2+ years as site director (N = 20)		−0.41 (−1.14, 0.31) p = .26	−0.34 (−0.80, 0.13) p = .15
2+ years as site director*intervention		1.79 (1.38, 2.20) p < .001	1.08 (0.69, 1.47) p < .001
Intervention		1.78 (1.48, 2.08) p < .001
Turnover (N = 19)		0.11 (−0.31, 0.53) p = .61
Turnover*intervention		−1.21 (−1.56,−0.86) p < .01
Intervention		1.16 (0.87, 1.44) p < .001
Turnover barrier baseline (N = 19)		0.29 (−0.48, 1.05) p = .46
Turnover barrier baseline*intervention		−1.38 (−1.85, −0.91) p < .01
Intervention		1.83 (1.08, 2.59) p < .001
Staff at learning communities (N = 20)		−0.08 (−0.90, 0.73) p = .83
Staff at LCs*intervention		−0.18 (−0.55, 0.19) p = .34
Intervention		1.20 (0.78, 1.62) p < .001
Staff on action plans (N = 20)		−0.21 (−0.49, 0.07) p = .14
Staff on action plans*intervention		0.12 (−0.01, 0.24) p = .06
Organizational capacity
Intervention		0.12 (−0.20, 0.44) p = .46	–
On-site kitchen (N = 20)		−0.35 (−1.02, 0.33) p = .31	−0.36 (−0.84, 0.11) p = .13
On−site kitchen*intervention		2.76 (2.38, 3.14) p < .001	1.73 (1.30, 2.15) p < .001
Intervention		1.44 (1.12, 1.76) p < .001
Nutrition priority change (N = 18)d		0.13 (−0.36, 0.62) p = .60
Nutrition priority*intervention		−0.49 (−0.78, −0.20) p < .01
Intervention		0.86 (0.60, 1.13) p < .001
Agency support change (N = 19)		−0.06 (−1.10, 0.98) p = .92
Agency support change*intervention		0.30 (−0.17, 0.78) p = .21
Intervention		1.58 (1.29, 1.87) p < .001
Funding change (N = 20)		0.09 (−0.36, 0.54) p = .70
Funding change*intervention		−0.60 (−0.88, −0.32) p < .01
Intervention		0.80 (0.51, 1.10) p < .001
Space change (N = 20)		0.19 (−0.32, 0.70) p = .46
Space change*intervention		−1.44 (−1.68, −1.19) p < .01
Intervention		2.48 (2.14, 2.82) p < .001	–
Child-to-staff ratio >10 (N = 20)		−0.28 (−0.85, 0.29) p = .33	−0.18 (−0.71, 0.34) p = .49
Child-to-staff ratio >10*intervention		−2.01 (−2.41, −1.62) p < .01	−1.19 (−1.62, −0.76) p < .001
Intervention		2.15 (1.83–2.47) p < .001
<50 children enrolled (N = 20)		0.30 (−0.39–0.99) p = .39
<50 children enrolled*intervention		−1.67 (−2.08–1.26) p < .01
Intervention		1.50 (1.20, 1.80) p < .001
Time change (N = 19)		0.36 (−0.29, 1.01) p = .29
Time change*intervention		−1.18 (−1.52, −0.85) p < .01

^aRestricted to children with at least one day of data at baseline and one day of data at follow-up. Seven hundred and thirty-eight person days among 212 children in control programs and 639 person days among 188 children in intervention programs.

^bAll regression models adjusted for baseline continuous age, binary gender (0 =female, 1 = male), indicator variables for six race/ethnicity categories, and binary indicator for day data collected (1 = first day, 0 = any other day). Regression estimates account for the clustered sampling design at the child and program level.

^cAll barriers to nutrition change rated on a three-point scale: 0 =never, 1 =sometimes, 2 = always.

^dNutrition as a program priority rated on a five-point scale with 1 being the top priority and 5 being the bottom priority.

Discussion

The results of this study establish the effectiveness of the OS-NAP intervention on increasing children’s water consumption and demonstrate the influence community, provider, and organizational factors can have on the impact of an obesity prevention intervention. They highlight the importance of understanding how interventions are delivered in real-world settings. Each domain in the Framework for Effective Implementation in Figure 5.1 had a significant impact on water consumption change in the intervention. Children who attended sites with on-site kitchens and low child-to-staff ratios had greater increases in water consumption than those with satellite kitchens and high child-to-staff ratios. Site director characteristics also influenced the effectiveness of the intervention: children who attended a site led by a director with two or more years of experience had greater increases in water consumption than children at sites with less experienced directors. No factors related to the neighborhood community context significantly infiuenced the intervention’s effectiveness. However, children at sites whose directors reported improvements in school support for nutrition efforts from baseline to follow-up had greater increases in water consumption than those at other sites. The intervention effects represent promising change in light of the facts that only 15 percent of boys and 26 percent of girls aged 9–13 consume the Institute of Medicine’s recommended adequate intake of water and that baseline water consumption estimates in our study were very low.¹⁷ Health benefits also result when children shift from sugary drinks to water, a practice change OSNAP successfully accomplished.¹⁸ Furthermore, the interaction effects are relatively large, compared to the main effects, indicating the importance of implementation factors in understanding real-world public health intervention results.

Our findings are supported by a number of implementation and dissemination theories and by studies of interventions in school and afterschool settings. To our knowledge, no nutrition intervention research has reported the impact of child-to-staff ratios on intervention effectiveness, but this factor has been studied extensively in early childcare settings.¹⁹ Results align with the organizational capacity domain in the Framework for Effective Implementation and with Greenhalgh’s system antecedents in health service delivery organizations.²⁰ Provider or adopter characteristics like level of experience were first theorized to affect adoption of innovations by Rogers.²¹ They continue to be a major focus in school-based studies, where teacher tenure has been investigated as a predictor of implementation.²² Previous school-based research and a qualitative afterschool physical activity and snack intervention study also found that school administrative support was an important factor influencing implementation.²³

The implementation predictors identified here are actionable. Our findings on the importance of site director experience suggest that training and staff development should be employed to help retain leaders in the afterschool field. With more years of experience, staff members are able to lead more confidently and implement health-related changes more effectively. Developing practices and policies to keep child-to-staff ratios low should also be a priority for afterschool programming. With more staff on hand, these sites were likely able to fill water pitchers and coolers, offer second servings of water, and model healthy behavior more easily and frequently than sites that were understaffed. An on-site kitchen means programs have access to refrigeration and running water. On-site kitchen cafeteria managers have more local autonomy to make changes in beverages served than do managers at satellite sites where contracted, prepackaged snacks are served. Still, a number of low-cost strategies could be employed at afterschool sites without on-site kitchens to improve water consumption. These include ensuring children’s access to pitchers of water and cups, installing bottle-filler fixtures on existing fountains, or purchasing refrigerated coolers from which to serve water during snack time.²⁴

Our results also demonstrate the benefits that can be gained from embedding the investigation of intervention implementation into traditional randomized controlled trials. While most implementation science research to date has focused on short-term implementation outcomes such as intervention adoption, this study was able to assess the impact of implementation factors on behavior change in a longitudinal sample of children. Future research should seize the opportunities group-randomized controlled trials present for understanding how the interventions they are testing work in real-world settings.

This study has a number of limitations. With a small sample of twenty sites in one urban school system, we were limited in our ability to investigate all hypothesized implementation predictors. For instance, there was minimal variability in agency support and in aspects of the community context such as school racial and ethnic demographics. Similarly, eighteen of the twenty programs participated in the national Afterschool Snack Program (part of the National School Lunch Program) and were served by the same food service department; these factors represent two likely predictors of menu choices that we could not explore. This limited variation in community context limits the generalizability of our findings. Also, we cannot be certain these results generalize even to the population of children who attend the twenty OSNAP sites because only about half of children attending these sites agreed to participate in data collection. However, the study design did intentionally enroll sites that were sponsored by a variety of agencies in disparate neighborhoods. Although the site director survey used to quantify the implementation predictors drew upon previous afterschool intervention work, the psychometric properties of this measure have not been assessed. Survey results were subject to self-report bias. Additionally, the survey was limited in scope: questions about barriers to implementation did not ask about water specifically, and important staff characteristics such as attitudes and self-efficacy were not investigated. Furthermore, the study sample experienced some attrition; however, our comparison of baseline and longitudinal samples indicates that selection bias did not compromise the study’s internal validity.

Conclusion

Our results demonstrate the effectiveness of an afterschool intervention on increases in children’s water consumption. They identify four key factors that are important for implementing successful afterschool interventions to increase water consumption: an experienced site director, a low child-to-staff ratio, strong school support, and an on-site kitchen. The findings also point to the importance of considering organizational capacity, provider characteristics, and community context more generally as public health interventions are evaluated and applied in real-world settings.