Validation of the Out-of-School-Time Snacks, Beverages, and Physical Activity Questionnaire

Abstract

Background: Recent research has highlighted the potential for out-of-school-time (OST) environments to impact children's energy balance. Expanding upon this evidence requires validated measures to assess nutrition and physical activity (PA) environments across large samples of OST programs. The aim of this study was to assess the criterion validity of the Out-of-School-Time Snacks, Beverages, and Physical Activity Questionnaire (OST-SBPA), an online questionnaire designed to measure OST nutrition and PA environments by program leaders' reports.

Methods: The study consisted of two samples of OST programs (total n=65). Objective data across program meetings were compared to program leaders' reports of usual snack, beverage, and PA offerings. All 65 programs provided snack and beverage data; PA data were available from 31 programs. Validity was assessed using percent agreement, Cohen's kappa, and Spearman's correlations.

Results: All OST-SBPA items demonstrated agreement above 60% when dichotomized (e.g., none/some versus most/all of the time for frequency items). Most OST-SBPA items were significantly correlated with objective data, including how often fresh and processed fruits and vegetables, salty snacks, sweet snacks, protein snacks, milk, juice, and sweetened beverages were offered and four PA measures (r=0.43–0.78; p<0.01), whereas one item, the frequency that water was offered, was not (r=0.13; p=0.37). The water item demonstrated validity once water from water fountains was recoded and not considered a program-provided beverage.

Conclusions: These findings fill a gap in the literature through the validation of a brief questionnaire that can be used to assess OST nutrition and PA environments. The current results support the use of the OST-SBPA in studies aiming to assess and impact these environments.

Introduction

Although efforts to reverse the childhood obesity epidemic have led to an improved school wellness environment, less effort has been directed toward the out-of-school-time (OST) environment as a setting for obesity prevention. Broadly defined, OST programs include organized activities for children that take place before or after school, on weekends, and during school vacations.¹ Tens of millions of youth participate regularly in structured OST programs, including traditional afterschool programs, as well as enrichment and sports activities, providing an opportunity to reach and impact the energy balance of a large, diverse population of children.^2–6

Within the emergent literature on obesity prevention during OST, rigorous measures have been used, such as accelerometry,³ direct observation of program sessions,^4,6 and detailed coding of snack menus.⁵ Assessment tools, such as direct observation, can provide high-quality, in-depth assessments of OST nutrition and physical activity (PA) environments, but administration of such procedures is not always feasible. As researchers move from pilot studies and efficacy trials to evaluations of larger-scale interventions aiming to promote healthy eating and PA during OST, brief and valid questionnaires are needed to facilitate repeated assessments of usual practices across large samples of OST programs and to increase confidence in the accuracy and credibility of the information obtained. Such a measure would contribute significantly to the field, facilitating evaluations of OST programs' potential for large-scale impact in promoting healthy environments for children, and offering possibilities to measure and communicate positive changes in OST nutrition and PA environments to broader audiences. Therefore, the aim of this study was to test the criterion validity of the Out-of-School-Time Snacks, Beverages, and Physical Activity Questionnaire (OST-SBPA), in which OST program leaders (staff and volunteers working directly with children) report on snack and beverage offerings and PA opportunities available to children during their programs.

Methods

Participants

The unit of analysis for this study was the OST program. We recruited programs that served children within the age range of 5–12 years in two waves. The waves were conceptualized as two substudies, with the two samples recruited 1 year apart. During winter 2013, 32 OST programs in Massachusetts and New Hampshire were recruited with a primary objective of assessing the criterion validity of the OST-SBPA; at the end of the study, 31 of these programs provided data on both the criterion measure and OST-SBPA (Sample 1). During winter 2014, we recruited 37 additional OST programs from Massachusetts, New Hampshire, and Maine. These programs were recruited for a study of snacks during OST, with a secondary aim of examining the criterion validity of the OST-SBPA; 34 of these programs provided data on both the criterion measure and the OST-SBPA (Sample 2). Because PA data were not available through the criterion measure used with Sample 2, combining across the two samples yields a final sample of 65 OST programs with snack and beverage data and 31 programs with PA data. Further descriptive information about each sample can be found in Table 1. All procedures involving human subjects were reviewed and approved by the Tufts University Institutional Review Board.

Table 1.

Characteristics of Programs and Participating Children^a

Sample 1	Frequency or mean
Program type	48% traditional afterschool; 23% enrichment (e.g., scouting); 29% sports
Number of childrenb	M=29 children (SD=17; range=4–71)
Child sexb	M=55% boys
Child age	M=∼50% 8–12 years, 33% 5–7 years, 10% 13 years+, <5% <5 years
Child race/ethnicity	M=∼50% White, 10% African American, 10% Hispanic, 10% Asian, <5% other

Sample 2	Frequency or mean
Program type	91% enrichment; 9% sports
Number of children	M=11 children (SD=6; range=2–28)
Child sex	M=68% boys
Child age	M=∼77% 8–12 years, 16% 5–7 years, 7% 13 years+, <1% <5 years
Child race/ethnicity	M=∼95% White, 1% African American, 1% Asian, 3% other

^aChild variables were leader-reported, unless otherwise noted. Leader-reported child demographics are averages of reported percentages (e.g., leaders reported on the percentage of children in their program in each race/ethnicity category) and thus may not sum to 100%.

^bIn these cases, demographic data were available through observations, so this data source was used instead of leader reports. Values were averaged across site visits before calculating means.

M, mean; SD, standard deviation.

Procedure

Overview

Our objective was to validate the OST-SBPA by testing whether program leaders' responses on the questionnaire aligned with criterion measures of snacks, beverages, and PA during OST program meetings. Programs in Sample 1 were observed during two site visits in January and February 2013, with an average of 1 week between visits (standard deviation [SD]=5.5 days). After the second visit, a link to the OST-SBPA was sent to a staff member or volunteer who led the program. The program leaders completed the questionnaire within 3 weeks of the second site visit (mean=4 days after the visit). In Sample 2, OST program leaders were asked to submit photographs of all snack and beverage items served during each of four program meetings between January and June 2014. Digital photography is a reliable, valid measure of dietary intake^7,8 and has been used to evaluate foods brought from home to school^9,10 and to study snacks in OST settings.¹¹ Photographs were submitted electronically, with an average of 6.6 weeks between submission of the first and last study photo (SD=2.4 weeks). After submission of the last photo, a link to the OST-SBPA was sent, and leaders were asked to complete it.

Questionnaire

A team of nutrition researchers (a subgroup of the authors) developed the OST-SBPA questionnaire items to detect incremental differences or changes in nutrition and PA environments across diverse OST programs. The questionnaire was designed to require little nutrition knowledge and minimize respondent burden and social desirability. Snacks and beverages were grouped into categories (Table 2), which were generated after reviewing categories used in national surveillance datasets,^12,13 snacks observed in field studies of OST and childcare programs,^14,15 and evidence linking snacks and beverages with energy intake and weight status.^16,17

Table 2.

Description of Key Snack, Beverage, and PA Items

	Described in questionnaire as:
Snack categories
Fresh FV	Snack category 1: Fresh fruits and vegetables such as apples, oranges, celery, and carrots
Processed FV	Snack category 2: Fruits and vegetables with dips or sauces, canned fruit or vegetables, fruit cups or applesauce, and dried fruit such as raisins
Salty snacks	Snack category 3: Snacks such as Doritos or Sun Chips, pretzels, popcorn, crackers such as Cheez-Its, Goldfish, or Wheat Thins, or tortilla chips
Sweet snacks	Snack category 4: Snacks such as ice cream, pudding, cookies, donuts, candy, granola or cereal bars, graham crackers, and breakfast cereals
Protein snacks	Snack category 5: Snacks such as nuts and nut butters, yogurt, cheese, cottage cheese, beans, and hummus
Beverage categories
Water	Beverage category 1: Water
Milk	Beverage category 2: Milk
Juice	Beverage category 3: 100% juice such as apple juice or orange juice
Sweetened beverages	Beverage category 4: Sweetened beverages such as regular or diet soda, flavored iced teas, sports drinks, and juice drinks such as Capri Sun or Kool Aid
Aspects of PA
Frequency	How often opportunities for PA are offered
Percent time	Proportion of program time children are active
Duration	How many minutes of PA during program session
Reach	How many children participate when opportunities for PA are offered

Respondents were asked about the frequency that the different categories of snacks and beverages were served across typical OST program sessions, as well as information about four different aspects of OST physical activity.

FV, fruits and vegetables; PA, physical activity; OST, out-of-school time.

Preceding the current study, 15 questionnaire items were piloted in a separate group of 19 program leaders across five OST organizations (Boy Scouts, Girl Scouts, 4-H, US Youth Soccer, and Pop Warner) to ensure that questions were understood by program leaders and applicable to the OST environment. After the pilot study, all 15 items were retained, with one change in framing: specifying that respondents should consider typical program sessions in their responses. (Previously, no session type had been specified.) Thus, the final OST-SBPA consists of 15 items and focuses on program-provided snacks, beverages, and PA opportunities during typical OST program sessions. The questionnaire uses 13 items with continuous response scales, which assess specific snacks and beverages served and aspects of PA, and two categorical items about who provides the snacks and beverages (see Tables 2 and 3). The aim of the current study was to validate the 13 key items about specific snacks, beverages, and PA opportunities provided. The OST-SBPA was administered to program leaders using Qualtrics software (Qualtrics, Provo, UT) as part of a larger battery of questions that included program-level demographics.

Table 3.

OST-SBPA Items Verbatim

Question	Response options	Skip patterns
1. Who provides the snacks that children eat at typical program sessions? Select all that apply.	A. Staff, volunteers, parents, and/or the program provides snack(s) for the group B. Children bring their own individual snacks C. Children do not eat snacks at typical program sessions D. Don't know	If response=A, then respondents complete 2a–2e below. If response=B–D, then respondents proceed to question 3.
2a–e. Thinking about typical program sessions, how often are the following snacks provided for the group? This may include snacks provided by staff, volunteers, and/or parents. (Respondents complete this question for each snack category; Table 2.)	A. Every time we meet B. Most of the times we meet C. Only some of the times we meet D. None of the times we meet E. Don't know	N/A
3. Who provides the beverages that children drink at typical program sessions? This may include tap or bottled water, juice, milk, or other beverages. Select all that apply.	A. Staff, volunteers, parents, and/or the program provides beverage(s) for the group B. Children bring their own individual beverages C. Children drink from a water fountain, water cooler or bubbler where [program] meetsa D. Children do not drink beverages at typical program sessions E. Don't know	If response=A, then respondents complete 4a–4d below. If response=B–E, then respondents proceed to question 5.
4a–d. Thinking about typical program sessions, how often are the following beverages provided for the group? This may include beverages provided by staff, volunteers, and/or parents. (Respondents complete this question for each beverage category; Table 2.)	A. Every time we meet B. Most of the times we meet C. Only some of the times we meet D. None of the times we meet E. Don't know	N/A
5. How often are opportunities for physical activity provided during typical program sessions?	A. Every time we meet B. Most of the times we meet C. Only some of the times we meet D. None of the times we meet E. Don't know	If response=A–C, then respondents complete questions 6–8 below. If response=D or E, questionnaire ends here.
6. On average, what proportion of time are children physically active during typical program sessions?	A. They are active or moving just some or only a little of the time B. They are active or moving most of the time C. They are active or moving all or nearly all of the time D. Don't know	N/A
7. On average, how many minutes of physical activity do children typically get during typical program sessions?	A. None B. 1–15 minutes C. 16–30 minutes D. 31–45 minutes E. 46–60 minutes F. More than 60 minutes G. Don't know	N/A
8. When opportunities for physical activity are provided during typical program sessions, how many children participate?	A. All children B. Most children C. Only some children D. No children E. Don't know	N/A

Respondents were instructed to focus on typical program sessions, which were defined as follows. For sports programs: Typical program sessions include practices only. Please do NOT include games, tournaments, camps or other special events. For all other programs: Typical program sessions include regularly scheduled program activities or meetings. Please do NOT include field trips, camps, award ceremonies or other special events.

For Sample 1, questions were administered as written above. For Sample 2, very minor changes were made to incorporate customized language corresponding to the particular OST organization (e.g., “during typical troop or den meetings” instead of “during typical program sessions” for Scouts).

^aThis response option was added after completing the study with Sample 1.

OST-SBPA, the Out-of-School-Time Snacks, Beverages, and Physical Activity Questionnaire; N/A, not applicable.

Criterion measure: Sample 1

A team of three trainers and eight research assistants conducted OST program observations. Before visiting program sites, the research assistants received training materials, including background information and descriptions of key variables, and attended an in-person training. A multistep approach was taken to ensure reliability between observers: Two of the trainers helped develop the observation protocols and demonstrated 100% agreement with one another across observational data corresponding to key items (Table 2) at the first two independent site visits. The third trainer had previous experience in nutrition research and demonstrated high agreement with other trainers across two visits (κ=0.89; κ=1.0). Each research assistant was initially accompanied by a trainer and was required to demonstrate adequate agreement with the trainer (κ ≥0.80). Once reliability was established (range=1–3 visits), research assistants conducted observations without the accompaniment of a trainer. For site visits at which there were two observers (trainer and research assistant; one third of visits), the trainer's data were used in analyses. During each program's two site visits, the entire program session was observed (mean=2.1 hours; range=47 minutes–4 hours).

Observers recorded whether the program provided snacks in each of the five categories and beverages in each of the four categories of interest. For each category, either “yes” or “no” was recorded, with supplemental, open-ended documentation of specific items offered. Water from water fountains was considered program-provided water. Additionally, multiple aspects of PA, or “any body movement that works muscles and requires more energy than resting” (e.g., walking briskly or running),¹⁸ were recorded, including: whether any PA opportunities were offered (frequency) and the start and end times of periods in which any children were engaged in PA, as well as the start and end times of the program overall. Differences between PA end and start times were used to indicate the duration of PA in minutes; this value divided by total program time was used to indicate the percentage of the meeting that children spent in PA. For example, if children in a 60-minute OST program session were physically active from 3:32 pm to 4:05 pm, then PA duration would be 33 minutes and the percentage of the meeting that was PA would be 55%. These variables represent the time that children spent being physically active, as opposed to scheduled PA time. Observers also recorded reach, or the proportion of children who participated in the observed PA time (Some: at least one child but fewer than 75% of children; Most: at least 75% but not all children; All: 100% of children), as well as the numbers of boys, girls, and program leaders present.

Criterion measure: Sample 2

The following guidelines were shared with program leaders, who were asked to submit photographs of all snacks and beverages served at each of four program meetings: (1) Include all foods and beverages (even water) that you serve at the meeting; (2) photograph snacks in their original packaging; (3) take the photo before snacks are served; (4) take the photo from above (aerial view); and (5) please do not include any images of children or other individuals. Program leaders were also provided with a sample photograph. Two trained research assistants coded the submitted photographs, each of which showed all snacks/beverages served during the corresponding meeting. The research assistants recorded the category (Table 2) corresponding to each snack/beverage item depicted in each photograph using a coding scheme developed by the first author. The team met after coding an initial 10 photographs to discuss and resolve any issues. All photographs were coded by both research assistants, and the two data sets were compared, with discrepancies resolved by the first author. Programs with photographs from 3–4 typical meetings were retained in the final analytic sample, resulting in the exclusion of four programs. For parsimony, criterion measures from Sample 1 and 2 will be referred to as observational data herein.

Statistical Analysis

Data were analyzed using SAS software version 9.2 (SAS Institute Inc., Cary, NC). Responses of “don't know” were coded as missing (n=1 “don't know” response, to item about sweet snacks). Observational data from Sample 1 were aggregated across the two site visits. Nondichotomous observations (e.g., PA duration) were averaged across site visits. Similarly, dichotomous observations (e.g., whether or not each snack/beverage category was provided, 1 or 0) were summed and divided by 2, resulting in a variable indicating the percentage of observations at which the item was provided. Observational data from Sample 2 were aggregated across photographs from the 3–4 program sessions by summing whether or not each snack/beverage category was provided and dividing by the number of photographs submitted (3–4), thus also indicating the percentage of observations in which each snack/beverage type was provided. Leader-reported OST-SBPA data were combined with these observational data, by program. The questionnaire includes skip patterns (see Table 3); to retain the full sample, zeroes were entered where items did not apply. For example, if respondents indicated that no snack was served, a “0” (never) was entered in the case of each specific snack category. Analytic samples ranged from n=63 to 65 for snacks and n=48 to 52 for beverages, owing to missing data and “don't know” responses. All 31 programs from Sample 1 were included in analyses of PA items.

The frequencies that snacks and beverages in each of the categories were observed were calculated in each sample. Additionally, the following frequencies were calculated using observational data from Sample 1: whether the program provided any snacks and beverages at zero, one, or two site visits and the frequency, percent time, duration, and reach of PA. These frequencies were not calculated in Sample 2, in which all programs served snacks owing to the aim of the study, and observational PA data were not collected. Validity was assessed by percent agreement, Cohen's kappa, and Spearman's correlations.¹⁹ Observational and questionnaire data were dichotomized for calculation of percent agreement and kappa values. For most items, observational data were dichotomized based on whether the item was observed during ≤50% of observations versus >50%, and survey data were split at the middle of the response scale (none/some of the time vs. most/all of the time). Exceptions were as follows: PA percent time was dichotomized as ≤25% versus >25% of observed sessions, PA duration was dichotomized as ≤30 versus >30 minutes, and PA reach was dichotomized as none/some vs. most/all children. Spearman's rank-order correlations were conducted with the continuous (nondichotomized) data, testing whether program leaders' reports of the 13 key items on the OST-SBPA (Table 2) were significantly and positively associated with corresponding observational data, across the combined samples as well as separately for each sample. Spearman's correlations were selected because the questionnaire scales were ordinal, and some variables were skewed. Additionally, Fisher's z-transformations and 95% confidence intervals (CIs) were calculated.

Results

Descriptive Information: Sample 1

Observations of whether any snacks, beverages, or PA opportunities were present during program sessions revealed that each of these was present consistently in the majority of programs. Fifty-five percent of programs had snacks at both site visits, 16% had snacks at one visit, and 29% of programs did not offer snacks at either visit. In 81% of programs, children drank beverages at both site visits, and in 19% of programs, children did not drink beverages at either site visit. Seventy-one percent of programs offered PA opportunities at both site visits, 10% offered PA opportunities at one visit, and 19% of programs did not offer PA at either visit. A variety of PAs were observed during program sessions, including dodgeball, jumping rope, dancing, basketball, swimming, hockey, soccer, tag, gymnastics, football, relay races, and four square. Descriptive statistics based on specific aspects of snacks, beverages, and PA observed in both samples are presented in Table 4.

Table 4.

Specific Snacks, Beverages, and Aspects of PA Observed

Sample 1. Data from 2 program site visits
Percentage of programsa with each snack category observed at:
	No visits	1 of 2 visits	Both visits
Fresh FV	55	23	23
Processed FV	73	23	5
Salty snacks	32	27	41
Sweet snacks	41	36	23
Protein snacks	59	32	9

Percentage of programsa with each beverage category observed at:
	No visits	1 of 2 visits	Both visits
Water	12	20	68
Milk	84	8	8
Juice	64	20	16
Sweetened beverages	92	8	0

Percentage of programsa with each aspect of PA in indicated amount:
Frequency	No visits	1 of 2 visits	Both visits
	—a	12	88

Percent time	Up to 1/3	1/3–2/3	2/3–all of session
	48	20	32

Duration, minutes	1–30	31–60	>60
	24	64	12

Reach (how many children)	Some	Most	All
	12	32	56

Sample 2. Data from photographs of snacks/beverages during 3–4 program meetings
Percentage of programsa serving each snack category in:
	0% of photos	1–25% of photos	26–50% of photos	51–75% of photos	76–100% of photos
Fresh FV	26	18	12	26	18
Processed FV	38	38	12	6	6
Salty snacks	44	24	15	9	9
Sweet snacks	47	21	18	9	6
Protein snacks	50	29	15	0	6

Percentage of programsa serving each beverage category in:
	0% of photos	1–25% of photos	26–50% of photos	51–75% of photos	76–100% of photos
Water	26	17	22	13	22
Milk	91	4	4	0	0
Juice	83	9	9	0	0
Sweetened beverages	52	26	13	4	4

Programs that never offered snacks, beverages, or any PA were excluded from the respective sections. Thus, these descriptive statistics depict specific snacks, beverages, and PA among programs that offered at least some type of snack (n=22 of 31 in Sample 1; n=33 of 33 in Sample 2), beverage (n=25 of 31 in Sample 1; n=23 of 33 in Sample 2), or PA (n=25 of 31 in Sample 1). All programs offered a snack in Sample 2 per the focus of the study. PA was not observed in Sample 2. In some cases, rows total slightly less or more than 100% owing to rounding.

FV, fruits and vegetables; PA, physical activity.

Validation of Key OST-SBPA Items

Percent agreement

When dichotomizing items, all items demonstrated agreement more than 60% of the time (Table 5), and most kappa values demonstrated fair-to-moderate agreement, with exceptions being sweet snacks, water, and PA reach. When separating response scales into three categories, as opposed to two (i.e., none, some, and most/all), observed and reported data were in agreement the majority of the time for all but one item (water item; 46% agreement), with percent agreement for the remaining items ranging from 55% to 82% (data not shown).

Table 5.

Percent Agreement between Observations and OST-SBPA Items (Dichotomous^a)

Item	Percent agreement	Kappa values
Snacks
Fresh FV	69.2	0.39
Processed FV	78.1	0.28
Salty snacks	73.8	0.42
Sweet snacks	65.1	0.08
Protein snacks	78.1	0.29
Beverages
Water	61.5	0.19
Milk	91.7	0.62
Juice	76.0	0.39
Sweetened beverages	93.9	0.54
PAb
Frequency	80.6	0.42
Percent time	67.7	0.36
Duration	74.2	0.38
Reach	61.3	0.09

^aFor all frequency items (the first 10 items listed), observational data were dichotomized based on whether the item was observed during ≤50% of observations versus >50%. Survey data were split at the middle of the response scale (none/some of the time vs. most/all of the time). Other items were dichotomized as follows: for PA percent time, ≤25% versus >25% of observed sessions and survey responses of none/just some of the time versus most/all of the time; for PA duration, observed and survey data were recoded as ≤30 minutes versus >30 minutes of the session; and for PA reach, observed and survey data were recoded as none/some versus most/all of the children.

^bPA data are from Sample 1 only. Snack and beverage data are from Samples 1 and 2.

OST-SBPA, the Out-of-School-Time Snacks, Beverages, and Physical Activity Questionnaire; FV, fruits and vegetables; PA, physical activity.

Spearman's correlations

Across the two samples, 12 of the 13 items demonstrated statistically significant, positive relationships between program leaders' reports and observational data (Table 6). These items included how frequently all five snack categories were offered, how frequently milk, juice, and sweetened beverages were offered, and all four facets of PA. In contrast, the correlation between leaders' reports and observations of water was nonsignificant. These overall results were consistent when excluding cases in which no snacks and no beverages were served. The results were also fairly consistent when analyzing the two samples separately, based on the magnitude and direction of the correlation coefficients, although some of the correlations did not reach statistical significance in these separated analyses with smaller samples (Table 7). Further, one result that deviated from the overall pattern was the correlation coefficient for sweetened beverages in Sample 1, which was close to zero, although it should be noted that there was a restricted range in this case, with 94% of sites serving sweetened beverages at neither of the two site visits.

Table 6.

Relationships between Reported and Observed Data for Key Validation Items

	Spearman's correlation (r)	Fisher's z	95% CIa	n
Snacks served
Fresh FV	0.53	0.60	0.33–0.69****	65
Processed FV	0.45	0.48	0.23–0.62***	64
Salty snacks	0.56	0.63	0.36–0.70****	65
Sweet snacks	0.46	0.49	0.23–0.63***	63
Protein snacks	0.43	0.46	0.20–0.61***	64
Beverages served
Waterb	0.13	0.13	−0.15–0.38	52
Milk	0.46	0.50	0.20–0.65***	48
Juice	0.55	0.62	0.32–0.72****	50
Sweetened beverages	0.55	0.62	0.31–0.72****	49
PAc
Frequency	0.78	1.05	0.58–0.89****	31
Percent time	0.64	0.76	0.36–0.81****	31
Duration	0.70	0.87	0.45–0.84****	31
Reach	0.52	0.57	0.19–0.73**	31

^*p<0.05; ^**p<0.01; ^***p<0.001; ^****p<0.0001.

^a95% CIs around Fisher-transformed estimates.

^bWhen excluding water fountains, the relationship between observations of program-provided water and responses to this item on the OST-SBPA became significant: r₍₄₉₎=0.50; p<0.001.

^cObserved PA data were available in Sample 1 only. Snack and beverage data are from Samples 1 and 2.

FV, fruits and vegetables; PA, physical activity; CI, confidence interval; OST-SBPA, the Out-of-School-Time Snacks, Beverages, and Physical Activity Questionnaire.

Table 7.

Relationships between Reported and Observed Data, by Sample

	Sample 1: Spearman's correlation (r)	Sample 1: sample size (n)	Sample 2: Spearman's correlation (r)	Sample 2: sample size (n)
Snacks served
Fresh FV	0.69****	31	0.22	34
Processed FV	0.46**	31	0.50**	33
Salty snacks	0.52**	31	0.63****	34
Sweet snacks	0.45*	30	0.49**	33
Protein snacks	0.37*	31	0.49**	33
Beverages served
Water	0.25	31	0.20	21
Milk	0.57***	31	0.22	17
Juice	0.75****	31	0.17	19
Sweetened beverages	−0.05	31	0.61**	18
PAa
Frequency	0.78****	31	—	—
Percent time	0.64***	31	—	—
Duration	0.70****	31	—	—
Reach	0.52**	31	—	—

^*p<0.05; ^**p<0.01; ^***p<0.001; ^****p<0.0001.

^aObserved PA data were available in Sample 1 only.

FV, fruits and vegetables; PA, physical activity.

We conducted post-hoc examinations to probe the lack of validity of the water item. In Sample 1, observers were trained to code children drinking from water fountains as “program-provided water”; subsequent to the nonsignificant finding, we hypothesized that program leaders did not consider water fountains as a program-provided beverage. We examined whether observations of “program-provided water, excluding water fountains” were correlated with leaders' responses on the OST-SBPA. In Sample 1, we coded open-ended observational data to identify cases where there was program-provided water that was not from water fountains. In Sample 2, we excluded the four cases in which leaders indicated that children drink water from water fountains. After these adjustments, observations of program-provided water were significantly correlated with leaders' reports of the frequency that water was provided (r=0.50; p<0.001), providing evidence suggesting that program leaders were valid reporters of program-provided water, but that this did not include water from fountains.

Discussion

This study provides initial support for the criterion validity of a new questionnaire intended to assess usual snack, beverage, and PA offerings during OST programs serving school-age children. Leader-reported OST-SBPA data were significantly and positively associated with observational data; similarly, calculations of agreement showed that the majority of OST-SBPA responses were aligned with observational data, with lower levels of agreement found for sweet snacks, water, and PA reach, compared to the other snack, beverage, and PA items. Overall, the results support the use of the OST-SBPA in studies aiming to assess the PA opportunities available and the quality of snacks and beverages served in OST programs and highlight avenues for additional research.

Observational data demonstrated some healthy attributes of OST environments across programs, as well as room for improvement. The overall quality of snacks served was mixed (e.g., both fruits and vegetables (FV) and salty snacks were common offerings), and observations of beverages showed a low prevalence of sweetened beverages and a high prevalence of water, a beverage with positive implications for hydration,²⁰ energy balance, and obesity prevention.^17,21 Whereas the majority of sites (Sample 1) offered PA opportunities that reached most or all children at both visits, most sites offered PA for less than 1 hour and for less than one third of the program session. Increasing opportunities for high-quality PA during OST could be an effective avenue for health promotion, as program priorities allow.

It is likely that efforts promoting healthy eating and PA during OST will continue to grow, given that OST programs are promising environments for health promotion and obesity prevention.^{3–6,22–26} With expanded research comes the need for validated questionnaires, given that methodologies such as direct observation are not always feasible in larger-scale effectiveness or dissemination research. Accordingly, the aim of the current study was to test the criterion validity of a brief questionnaire through which program leaders report usual snack and beverage offerings and PA opportunities during OST. Twelve of the 13 OST-SBPA items were significantly correlated with corresponding observational data, with correlation coefficient magnitudes greater than those reported in previous validation studies of self-report items assessing behaviors linked to obesity risk (e.g.,^27,28). Leaders' reports about the remaining item (water) were also significantly associated with observational data when excluding water from water fountains. Thus, leaders' responses to this item on the OST-SBPA should be interpreted as including program-provided water except from water fountains. This is consistent with recent health policies intended for OST programs, such as the Healthy Eating and Physical Activity guidelines,²⁹ which specify that water should be served on the table at snack time. Separate indicators could be explored to assess the availability of water fountains (e.g., see Table 3, question 3).

This study fills a gap in the literature through the initial validation of a questionnaire that measures usual snack, beverage, and PA offerings during OST programs. In particular, the results suggest that this measure can be used to rank-order programs in terms of the different snacks, beverages, and PA opportunities provided. The OST-SBPA may be less precise in measuring absolute levels of snacks, beverages, and PA, although analyses of agreement suggest that it is a reasonable indicator when response scales are collapsed into three categories (e.g., none, some, and most/all of the time for frequency items) and a better indicator when response scales are dichotomized (e.g., none/some vs. most/all), with fair-to-moderate agreement found for most items. Exceptions in which lower agreement was found were sweet snacks, water, and PA reach. Future research could explore whether similar findings emerge in other studies, and if so, modifications, such as collapsing salty and sweet snacks together and focusing on the other facets of PA, may be warranted.

Strengths of this research include representation of different types of OST programs across three states and the use of repeated observations. Limitations include less demographic diversity in Sample 2, compared to Sample 1 (68% boys, 95% white, and 91% enrichment programs), and the lack of random sampling in assessing criterion measures. Program leaders knew when researchers would be conducting site visits (Sample 1) and chose when to take and submit their photographs (Sample 2). A more rigorous design would have entailed unannounced visits and photograph requests at randomly selected times; these methods were infeasible in the context of this initial validation study. Our engagement with this population of busy staff members and volunteers suggests that it would be implausible for them to make significant modifications to snacks, beverages, and PA on observation days; yet, additional measurement work is needed to confirm and extend the current results.

Future research should seek to examine the performance of the OST-SBPA in larger, more diverse samples that include additional afterschool, enrichment, and youth sports programs and should explore test-retest reliability and factor analyses of the questionnaire items. We cannot generalize beyond the types of programs included in the present analysis, nor can we draw conclusions about snacks and beverages brought by youth, owing to the fact that the OST-SBPA focuses on those that are provided by the program during typical program sessions. Yet, we believe that program-provided snacks and beverages offer the greatest potential for impact through policies and interventions.

Conclusions

In sum, the OST-SBPA shows promise as a valid measure for assessing usual snack, beverage, and PA offerings during OST. The initial validation of this measure is timely given that OST programs have been highlighted as promising settings for obesity prevention efforts, and items in this questionnaire include evidence-based contributors to energy balance and thus obesity risk. The OST-SBPA can facilitate the assessment of nutrition and PA environments during OST, providing opportunities to expand current obesity prevention research in these settings, which impact tens of millions of children.

Acknowledgments

This study was funded with support from the Harvard Pilgrim Health Care Foundation and Newman's Own Foundation. The authors thank Heather Angstrom, Molly Newman, Sarah Sliwa, and Shanti Sharma for their contributions to the Out-of-School-Time Snacks, Beverages, and Physical Activity Questionnaire; Clarissa Brown, Jacklyn Emilo, and Anna Marie Finley for their assistance in collecting and coding data; Peter Bakun for assistance with data analysis; and the other data collectors and out-of-school-time program staff and volunteers who were involved with the study.

Author Disclosure Statement

No competing financial interests exist.