Validation of a Brief Questionnaire against Direct Observation to Assess Adolescents’ School Lunchtime Beverage Consumption

Abstract

Objective

Beverage consumption is an important determinant of youth health outcomes. Beverage interventions often occur in schools, yet no brief validated questionnaires exist to assess whether these efforts improve in-school beverage consumption. This study validated a brief questionnaire to assess beverage consumption during school lunch.

Methods

Researchers observed middle school students’ (n = 25) beverage consumption during school lunchtime using a standardized tool. After lunch, students completed questionnaires regarding their lunchtime beverage consumption. Kappa statistics compared self-reported to observed beverage consumption across 15 beverage categories.

Results

Eight beverages showed at least fair agreement (κ > 0.20) for both type and amount consumed, with most showing substantial agreement (κ > 0.60). One beverage had high raw agreement but κ < 0.20. Six beverages had too few ratings to compute κ’s.

Conclusions and Implications

This brief questionnaire was useful for assessing school lunchtime consumption of many beverages, and provides a low-cost tool for evaluating school-based beverage interventions.

INTRODUCTION

Beverage consumption is an important determinant of youth health outcomes such as obesity.¹ Because youth spend a large portion of their waking hours in schools, many healthy beverage interventions take place at school.² To evaluate whether these strategies effectively change students’ in-school beverage consumption habits, validated measures are needed. While dietary recalls, plate waste measures, and direct observation are considered gold standards for assessing beverage consumption, these resource-intensive techniques are not feasible in many research or practice settings, and measurement tools are needed to assess beverage consumption in a rapid, low-cost manner.

Several brief questionnaires exist to assess beverage consumption, but each has limitations for assessing youth’s in-school beverage consumption. The BEVQ15 and BEVQ19 were developed for adult, not youth, populations.^3,4 Paxton et al.⁵ developed a school lunch recall for fourth-graders, but the tool does not assess beverages from non-cafeteria sources, which may comprise a large portion of beverages consumed at school.⁶ The Beverage and Snack Questionnaire⁷ was developed for use with adolescents, but does not capture intake of several beverage categories of interest to public health practitioners and policymakers, such as water.

To address these gaps in the literature, this study aimed to describe the development of a brief, self-administered questionnaire to assess adolescents’ beverage consumption during school lunchtime and to examine the initial validation of the questionnaire against direct observations of students’ beverage consumption.

METHODS

Participants and Recruitment

Data collection took place between December 2012 and February 2013. A convenience sample of three standard (i.e., non-charter, non-magnet) public schools in the San Francisco Bay Area region of California was recruited. Eligible schools served students in grades 6 to 8. Because low-income and minority children tend to have less healthy beverage consumption habits than higher income and white children,⁸ eligible schools had at least 50% of students eligible for free or reduced price meals through the National School Lunch Program (NSLP) and had at least 50% of students of Latino or African American race/ethnicity. Schools were selected to represent a range of on-site beverage options, including milk and juice served as part of NSLP, a variety of a la carte beverage choices, and different options for free water (traditional water fountain, water dispenser with cups, and water bottle filling station). To recruit schools, research staff contacted school food service directors to assess interest and eligibility, mailed an informational letter to interested administrators, and made phone calls to explain study procedures and schedule a time for data collection.

At each study school, school staff recruited a convenience sample of five to ten English-speaking students (total n = 25). Students’ parents received an informational letter and provided written consent, and students gave written assent. All procedures were approved by the University of California, San Francisco Committee on Human Research.

Measures and Procedures

Based on a review of existing measures,^4,5,7 a brief, self-administered questionnaire was developed. The questionnaire was developed to evaluate a school-based cafeteria intervention,⁹ so focused on assessing students’ beverage consumption during school lunchtime. This focus may also increase accuracy, as youth can more accurately recall their consumption at a single meal compared to an entire 24-hour period.¹⁰ Additionally, because youth report their dietary intake more accurately soon after consumption,¹¹ the questionnaire was designed for administration immediately after lunchtime.

To ensure face validity, several experts in dietary assessment as well as staff at California Food Policy Advocates, a public health organization with expertise in nutrition policy, provided input on the questionnaire (e.g., whether appropriate beverages and portion sizes were used). Initial drafts were pre-tested with three middle-school students and revised based on their feedback, including adding instructions and reformatting so that each beverage appeared on a separate page. Next, a pilot of the validation procedures (see below) was conducted with 11 students at two eligible schools not included in the main validation study. Further revisions to the questionnaire were then made, including adding items asking for the name, flavor and brand of each beverage item consumed.

The revised instrument was validated in a convenience sample of students (n = 25) from the three study schools. The instrument (see Supplementary Material Exhibit 1) included 14 closed-ended questions asking students to report whether they drank (yes/no) the following specific beverages during lunchtime: tap water from the cafeteria; tap water from outside of the cafeteria; tap water from home; plain bottled water; flavored bottled water; plain milk; flavored milk; diet drinks; regular sodas; regular sports drinks; 100% fruit juice; other sugary or sweetened drinks (e.g., fruit-flavored drinks, sweetened coffee/tea); energy drinks; and any other beverages (write in the beverage type). For each beverage consumed, students indicated the amount they consumed (a few sips, less than 1 glass or half a bottle, 1 glass or half bottle, 2 glasses or 1 bottle, or more than 2 glasses or one bottle). Each item included images of the beverage type (e.g., image of a milk carton) and of portion sizes (e.g., image of a half-full glass).

Students also reported their demographic characteristics. Students completed questionnaires immediately after lunch in the cafeteria or another quiet location (e.g., library). Questionnaires took approximately 5–10 minutes to complete. Students received a $5 movie theater gift card for participating.

Trained research staff unobtrusively observed students’ beverage consumption during lunchtime using a standardized tool (see Supplementary Material Exhibit 2). Research staff were paired to students one-to-one; thus, the inter-rater reliability among observers was not assessed. Researchers recorded each beverage the student consumed and the estimated number of ounces the student consumed (based on the observed starting and ending amounts in the container and/or the number of sips observed). For comparison with the questionnaire, the observer translated these estimates into questionnaire response options using the following conversions: <3 ounces as response option 1 (“a few sips”); 3 to <8 ounces as response option 2 (“less than 1 glass or half a bottle”), and 8 ounces as response option 3 (“1 glass or half bottle”). No students were observed consuming more than 8 ounces of a given beverage (i.e., there were no observations that corresponded with the two highest response options).

Data Analysis

Research staff double-entered all data using REDCap data entry system.¹² For analyses, three new beverage categories were created: water from a free source at school (combination of all free water sources at school); all plain water (combination of water from a free source at school, tap water brought from home, and plain bottled water); and any SSB (combination of flavored water, soda, energy drinks, sports drinks, and other sugary or sweetened beverages). All other beverages were assessed separately.

Validity was assessed by comparing researcher observations of students’ beverage consumption to students’ self-reported consumption. Raw percent agreement and kappa statistics (κ) were calculated to examine the agreement between observations and questionnaire data on the type of beverages consumed (yes/no for each beverage). Next, linear-weighted agreement and kappa statistics were calculated to examine agreement between the observed and reported amount consumed for each beverage.^13,14 Following generally accepted interpretations,¹⁵ kappa scores between 0.21 and 0.40 were considered as indicating “fair agreement,” 0.41 to 0.60 as “moderate agreement,” 0.61 and 0.80 as “substantial agreement,” and 0.81 to 1.0 as “almost perfect agreement.” Analyses were completed using Stata Version 13.1 (StataCorp LP, College Station, Texas).

RESULTS

Participants were predominantly minority, with about half (48%) identified as Hispanic/Latino, 20% as Black, 20% as White, 16% as Asian, and 16% as American Indian or Alaskan Native (Table 1). Most (83%) were born in the U.S., and 36% reported that they primarily spoke a language other than English at home.

Table 1.

Demographic Characteristics of Middle-School Students (n = 25) Participating in Observation and Questionnaire Data Collection about School Lunchtime Beverage Consumption

Characteristics	%	(na)
Age in years, mean (SD)	12.5	(0.9)
Female	68%	(17)
Race/ethnicityb
Hispanic/Latino	48%	(12)
Black	20%	(5)
White	20%	(5)
Asian	16%	(4)
American Indian/Alaskan Native	16%	(4)
Born in the U.S.	83%	(20)
Language most often spoken at home
English	64%	(14)
Other languagec	36%	(8)

^aTotal n across categories may not sum to 25 due to missing data.

^bCategories sum to greater than 100% because students could select all race/ethnicity categories that applied.

^cIncludes Tagalog, Cantonese, and “Other.”

Of the 15 beverage categories assessed, four (all plain water, water from free school sources, flavored milk, and other sugary or sweetened beverages) demonstrated almost perfect agreement on whether the beverage was consumed (range of κ’s = 0.82 – 1.00), and one (plain milk) demonstrated substantial agreement (κ = 0.78) (Table 2). Agreement regarding consumption of flavored water and of any SSBs was moderate (κ’s = 0.47 and 0.52, respectively). Agreement for whether “other beverages” were consumed was fair (κ = 0.24), and agreement for whether soda was consumed was low (κ = 0.00) despite high raw agreement. The six remaining beverages had too few ratings to compute accurate κ’s, as few or no students consumed them during school lunchtime.

Table 2.

Agreement on Type and Amount of Beverages Consumed Between Middle-School Students’ (n = 25) Self-Report and Researchers’ Direct Observation

Beverage	% (N) Consumed in Observation	% (N) Consumed in Self Report	Kappa for Whether Beverage Consumed	Raw Agreement for Whether Beverage Consumed	Weighted Kappae for Amount of Beverage Consumed	Weighted Raw Agreemente for Amount of Beverage Consumed
All plain watera	28% (7)	36% (9)	0.82	92.0%	0.81	96.0%
Water from a free source at schoolb	28% (7)	36% (9)	0.82	92.0%	0.67	92.0%
Flavored water	4% (1)	12% (3)	0.47	92.0%	0.31	94.7%
Plain milk	20% (5)	28% (7)	0.78	92.0%	0.75	92.0%
Flavored milk	16% (4)	12% (3)	0.83	96.0%	0.75	94.7%
Soda	0% (0)	4% (1)	0.00	96.0%	0.00	96.0%
Other sugary or sweetened beveragesc	4% (1)	4% (1)	1.00	100.0%	0.79	98.7%
Other beverages	12% (3)	12% (3)	0.24	84.0%	0.45	92.0%
All sugar-sweetened beveragesd	8% (2)	20% (5)	0.52	88.0%	0.45	92.0%
Tap water from home	0% (0)	0% (0)	Incalculable*	100.0%	Incalculable*	100.0%
Plain bottled water	0% (0)	0% (0)	Incalculable*	100.0%	Incalculable*	100.0%
100% fruit juice	0% (0)	0% (0)	Incalculable*	100.0%	Incalculable*	100.0%
Sports drinks	0% (0)	0% (0)	Incalculable*	100.0%	Incalculable*	100.0%
Energy drinks	0% (0)	0% (0)	Incalculable*	100.0%	Incalculable*	100.0%
Diet drinks	0% (0)	0% (0)	Incalculable*	100.0%	Incalculable*	100.0%

^*Incalculable due to perfect agreement and no variation in responses.

^aIncludes water from a free source at school (fountain, dispenser, or water bottle filling station), tap water brought from home, and non-flavored bottled water.

^bIncludes water from a water fountain, dispenser, or water bottle filling station at school.

^cIncludes sweetened drinks such as fruit-flavored drinks (Capri Sun, Sunny Delight), sweetened coffee/tea, aguas frescas.

^dIncludes flavored water, sports drinks, energy drinks, soda, and other sugary or sweetened beverages.

^eWeighted kappa and weighted agreement use a linear weighting function.

Most beverages showed high levels of agreement on amount consumed (Table 2). Of the nine beverages with enough ratings to calculate a kappa statistic, all but two had at least moderate agreement (range of weighted κ’s = 0.45 – 0.81). The remaining beverages showed fair agreement (flavored water: weighted κ = 0.31) or low agreement (soda: weighted κ = 0.00), despite high raw agreement.

DISCUSSION

This brief, self-administered questionnaire shows promise for assessing beverage intake during school lunchtime in a diverse sample of middle school students. The majority of beverage types demonstrated substantial or almost perfect agreement between observations and self-report for the type and amount of the beverage the student consumed during lunchtime. To the authors’ knowledge, there are no other validated brief questionnaires to assess adolescents’ beverage consumption during school lunchtime. While other questionnaires are available for rapid assessment of beverage consumption in adults,⁴ usual beverage and snack consumption in adolsecents,⁷ and certain food items eaten during school lunchtime among elementary school students,⁵ this questionnaire is unique in providing a low-cost, expeditious way to collect information specifically on adolescents’ school lunchtime beverage consumption across a range of beverage types. As such, this tool provides a new evaluation option for assessing the impact of school-based interventions to promote consumption of healthy beverages.

The questionnaire’s validity might be enhanced with slight modifications to the instrument. For example, providing more detailed definitions of the beverage categories might improve accuracy. It is also possible that the nonspecific nature of the ‘other beverage’ category made it more difficult for students to respond accurately,¹⁶ and future iterations of this questionnaire could include additional beverage categories as appropriate to the population of interest to avoid many responses in the ‘other beverage’ category.

This study had several limitations. The tool is specific to students’ beverage intake during school lunch, and may not be valid for assessing consumption in other settings. One-to-one direct observation of students necessitated a small sample size, and precluded calculating the inter-rater reliability among observers. In part due to the small sample size, several beverages assessed in the questionnaire were consumed by few or no students in the study. This lack of variation likely contributed to the low kappa scores observed for items related to consumption of soda, flavored water, and ‘other beverages,’ despite achieving high raw agreement. Additionally, kappa statistics could not be computed for all beverage categories, as some beverages were not consumed by students in the sample. Future research is needed to assess whether this instrument is valid for these beverages. Finally, while researchers attempted to observe students unobtrusively (e.g., stood to the side of the cafeteria, did not speak with students), the students knew they were being observed, and may have paid more attention to their beverage consumption than they would otherwise.

This study also has several strengths. For example, the use of direct observation reduces the threat of common-method bias.¹⁷ The questionnaire included images of the beverage items and of portion sizes, aiding comprehension and accuracy. The questionnaire also assessed, and demonstrated good agreement for, a range of beverage categories important for health, including water, sugary drinks, and plain and flavored milk. To the authors’ knowledge, no other brief beverage consumption questionnaires cover this range of beverage types.

IMPLICATIONS FOR RESEARCH AND PRACTICE

In this sample, a brief questionnaire was a useful tool for assessing the type and amounts of some of the beverages that students commonly consume during school lunchtime, including plain and flavored water, plain and flavored milk, and sugar-sweetened beverages. This tool is a promising first step toward developing low-cost means for evaluating the effectiveness of school-based strategies to improve beverage consumption, including efforts to promote water consumption and reduce sugar-sweetened beverage consumption. Future research should validate the tool for beverages that were not consumed by students in this sample, in larger samples, and in other populations.