Abstract
Sweetened beverage and fast food intake have been identified as important targets for obesity prevention. However, there are few brief dietary assessment tools available to evaluate these behaviors among adolescents. The objective of this research was to examine reliability and validity of a 22-item dietary screener assessing adolescent consumption of specific caloric and non-caloric beverages (9 items) and fast food (13 items). The screener was administered to adolescents (ages 11–18 years), recruited from the Minneapolis/St. Paul metro region. One sample of adolescents completed test-retest reliability of the screener (n=33, primarily Caucasian). Another adolescent sample completed the screener along with 3 24-hour dietary recalls to assess criterion validity (n=59 Caucasian). Test-retest assessments were completed approximately 7–14 days apart, and agreement between the two administrations of the screener was substantial, with most items yielding Spearman correlations and Kappa statistics that were >0.60. When compared to the “gold standard” dietary recall data, findings indicate that the validity of the screener items assessing adolescents’ intake of regular soda, sports drinks, milk and water was fair. However, the differential assessment periods captured by the two methods (i.e., one month for the screener vs. 3 days for the recalls) posed challenges in analysis and made it impossible to assess the validity of some screener items. Overall while these screener items largely represent reliable measures with fair validity, our findings highlight the challenges inherent in the validation of brief dietary assessment tools.
Keywords: Dietary factors, nutrition assessment
INTRODUCTION
Traditional dietary assessment methods have largely focused on quantifying aggregate nutrient intakes (1), rather than food groups. Though diet-disease relationships are often quantified via nutrient-specific analyses, there is growing recognition that public health messaging around foods and eating patterns may be important in health promotion (2, 3).
In assessing dietary intake, “gold standard” methods (e.g., 24-hour recalls) are often cost prohibitive, requiring trained staff, sophisticated nutrient databases and standardized data collection protocols. Depending on number of recall days needed to accurately assess nutrients of interest (1), recalls may substantially increase participant burden. Food frequency questionnaires have been developed for use among adolescent populations, though are intended to provide a comprehensive assessment of intake and have not been validated to assess individual food groups (4, 5). Specific foods (e.g., sweetened beverages, fast foods) have been implicated in the obesity epidemic, and though not all scholars agree upon the extent of their role in obesity (6), these foods are currently targeted by on-going obesity prevention interventions; thus, dietary assessment tools are needed to assess intake of these foods (7, 8).
The development of brief, validated assessment tools that allow for the characterization of food-related risk or protective factors in various populations is warranted. While previous epidemiologic work has utilized single-item questions to assess sweetened beverage and fast food intake, in many cases the criterion validity of such questions has not been examined. Additionally, the food industry has recently undergone dramatic diversification, now offering newer non-soda beverages (e.g., sports drinks) and non-traditional fast food restaurants (e.g., quick-service sandwich shops and ethnic restaurants) that are important to quantify.
The objective of this research was to develop a brief assessment tool (i.e., a dietary screener) specifically targeting beverage and fast food intake within the last month and assess the reliability and validity of this tool in an adolescent population.
METHODS
These data were collected as part a larger study whose sample was recruited from the Minneapolis/St. Paul, Minnesota region (Lytle, 2008, unpublished manuscript). The University of Minnesota Institutional Review Board approved this study.
Screener development/piloting
Investigators reviewed survey items from existing research. To meet the needs of the study and better characterize adolescent sweetened beverage and fast food intake, previously developed survey items were adapted, re-worded and expanded (9). Items included on the screener were selected based on: (a) research suggesting frequent consumption by this age group and contributions to energy balance (10), (b) expert panel discussion and recommendation, and (c) literature (both scientific and lay literature) suggesting emerging trends in the consumption of specific items by this age group (i.e., sports drinks, coffee drinks) (11, 12). Specific beverages were included in the screener instrument to understand what types of beverages were commonly consumed and the consumption frequency of these specific beverages. The number of times that respondents purchased any food at fast food restaurants in the past month and the types of fast food restaurants frequented was also assessed.
In pilot testing, study staff conducted interviews with adolescents, discussing the revised items as participants completed the screener to provide insight into item comprehension. Interviews (n=10) were conducted until saturation was achieved. Findings were used to further alter the structure and wording of screener items. Figure 1 details the adapted screener.
Figure 1.
Dietary screener items and response options used to assess beverage and fast food intake.
Reliability testing
A adolescent convenience sample was recruited from community recreation facilities. Thirty-three adolescents (15 males), aged 11–18, completed the self-administered screener at two time points. Most participants (85%) completed the screener 7–14 days apart, with outliers ranging from 2–21 days. Given that the time frame of the screener included the past month, investigators did not expect to observe real change in dietary intake during this period. Rather, discrepancies in test-retest reports would likely be due to the lack of reliability of the survey items and/or on-going intra-individual dietary variation.
Data were entered by the University of Minnesota Health Survey Research Center and reviewed by study staff to ensure accuracy. Analysis of reliability data included: (a) Spearman correlation coefficients (for items with ≥5 response categories) and (b) kappa coefficients (for items with <5 response categories). All analyses were item-specific (i.e., examining screener items individually, rather than in aggregate).
Criterion validity
Validity analyses were conducted among 59 Caucasian adolescents (26 males) 11–18 years old, recruited as part of the larger cohort study through (a) an existing cohort of youth, (2) Department of Motor Vehicle lists, and (3) a convenience sample (Lytle, 2008, unpublished manuscript). Socioeconomic status (SES) was high with 59% of participants’ parents reporting being a college graduate (5% with missing data). In addition to completing the dietary screener, participants completed 3 telephone-administered 24-hour dietary recalls (2 weekdays and 1 weekend day). On average, participants completed all 3 recalls within a 15-day time period. The screener and the first recall were collected, on average, within 10 days of each other. Trained staff administered the recalls, using the Nutrition Data System Research (NDS-R) with an interactive, multiple-pass interview (13–16) format with direct data entry linked to a nutrient database (17). NDS-R data provided food/beverage group information, which served as the criterion or “gold standard” for the beverage screener. To assess fast food frequency in the 3-day recalls, an additional recall prompt was included: “Were any of the foods in this meal purchased from a restaurant, fast food restaurant, or bagel/coffee shop? If so, what is the restaurant/shop name?” Restaurant/shop names were categorized by type (e.g., “burger-and-fries,” Mexican, fried chicken) to compare with screener categories. Data were reviewed by interviewers as well as University of Minnesota Nutrition Coordinating Center Quality Assurance staff to ensure that outliers and any possible data errors were identified and resolved.
Comparing the 24-hour recalls with the screener is challenging due to the response periods captured by each instrument (i.e., recalls assessed intake over 3 days, whereas the screener assessed past month intake). Ideally, a month or more of recalls would be used to validate a screener assessing past month intake; however, this was not feasible. Since estimating actual intake was not possible, beverage and fast food intake frequencies from both the recalls and screener were transformed into tertiles, characterizing individuals as “high,” “moderate” and “low” consumers (1). Due to a skewed distribution of sports drink consumption in the recalls, participants were coded as “low” (n=48, no reported sports drinks), “moderate” (n=7, one drink during the three recall days) and “high” (n=4, ≥2 drinks). The proportion of individuals correctly classified and/or misclassified using the two methods was also assessed.
Because analyses comparing beverage serving sizes and fast food restaurant types were not subject to the problems of differential time frames, tertiles were not computed. Since all variables examined in the validity analyses had 3–4 response categories, kappa coefficients were used. Statistical analyses were completed in SAS v.9.1 (Release 9.1. Cary, NC: SAS Institute Inc.) and STATA v.9.0 (STATA Corporation, College Station, TX, 2005).
RESULTS AND DISCUSSION
These findings indicate that item reliability was generally high, with substantial test-retest agreement. Most items yielded Spearman correlations and Kappa statistics >0.60. Agreement was particularly strong for frequency and amounts of beverages consumed. Reliability was also significant for overall fast food frequency and reported purchasing from several restaurant types (e.g., traditional “burger-and-fries,” Mexican, fried chicken, bagel shops, ice cream and burger restaurants). Reliability results suggest that the screener questions were sufficiently clear and behaviors were stable enough that responses to the screener were consistent over time.
When compared to the “gold standard” dietary recall data, findings indicate that screener items assessing adolescents’ frequency of intake of regular soda, sports drinks, and milk had acceptable, though modest, criterion validity. Kappas assessing the quantity of regular soda and water consumed were also statistically significant. Magnitude of agreement could generally be considered fair (18) with regard to ranking individuals by tertile (Kappas ranging from 0.19–0.38), but statistically significant (p<0.002). The magnitude of agreement between these screener items and the 24-hour recalls is only moderately lower than that reported in other dietary questionnaire validity studies (4, 5, 19).
Agreement between the screener and the recalls was not significant for other sweetened beverages and overall fast food frequency. For many youth, it may be difficult to quantify the sources of “other” sweetened beverages on the screener, as well as to tally a wide range of fast food sources that are likely consumed with varying frequency. Youth may have difficulty estimating behavior over the past month and/or may not identify “sweetened beverages” in the same way as health professionals.
There was no significant differential missclassification in tertile classifications (i.e., a greater proportion of misclassification due to over- versus under-reporting on the screener) for soda, other sweetened beverages, milk or fast food, though there was differential misclassification for sports drinks (p<0.001). The screener over-estimated sports drink intake compared to recalls; while 63% of participants were classified consistently between the two methods, the screener overestimated consumption relative to the recalls in 32% of participants and under-estimated consumption in only 5%.
Future research should focus on adapting item phrasing to reduce ambiguity and/or test alternative formatting of these questions to facilitate participants in quantifying complex behaviors. In addition, 3 days of recalls may not adequately represent past month intake, particularly for items that have high intra-individual variation (i.e., are consumed sporadically). Future research may consider more efficient means of accurately estimating intra-individual variance with regard to intake of food groups (as has been done with nutrients) in order to determine how many days of recalls are needed to capture “usual intake” of specific foods (1).
Overall, while the items included on our dietary screener largely represent reliable measures, with some items meeting criteria for “fair” validity (18) (i.e., items related to beverage intake), these findings highlight the challenges inherent in validating such brief dietary assessment tools. The differential assessment periods captured by the two methods used here (i.e., one month for the screener vs. 3 days for recalls) posed analytic challenges, making it impossible to assess the validity of certain screener items. Due to the infrequent consumption of diet soda and coffee drinks in the sample, insufficient responses were available in the recalls to generate stable comparison estimates. Furthermore, though overall “fast food” (assessed as a collective category) was consumed somewhat regularly, there was not a single fast food type that was consumed with sufficient frequency to be accurately captured by only 3 recall days. Thus, despite growing interest in food intake at varying types of quick-service restaurants, capturing these specific adolescent food behaviors may require many days of dietary recalls, well over the standard 2–4 days that is widely accepted as a practical means for validating diet questionnaires (9, 19–21).
This study has several limitations. Study participants spanned a relatively wide range of age, and there are likely developmental differences across these ages that influence the accuracy and reliability of the dietary data. In addition, limited response options on the screener may influence the accuracy of these self-reports; for example, though respondents were asked to report usual soda intake in terms of ‘cans,’ many youth today now consume soda in 20-ounce bottles. Finally, given that our sample represents a Caucasian, high SES population, literacy and reading comprehension was likely high; reliability and validity estimates may be inflated. Valid and reliable dietary assessment tools are needed for minority and low SES groups. In addition, there is a widespread interest in developing common measures that could be used across a variety of settings and populations and allow for direct comparison of study findings. However, it is unclear whether it is possible to develop such common measurement tools, given the various factors influencing diets across different populations.
CONCLUSIONS
The financial costs and response burden of highly accepted, “gold standard” dietary assessment methods, such as 24-hour recalls, often prohibit their use in research and other settings. Given that assessing the intake of specific food items may require weeks or months of recall data to accurately capture usual intake and account for intra-individual variation in intake (1), there is a striking need for brief assessment tools. The findings of this study suggest that the screener presented here represents a reliable tool, with fair validity in assessing adolescents’ intake of regular soda, sports drinks, milk and water. In the absence of superior tools that demonstrate a greater degree of validity, this screener may be useful to researchers and practitioners who are attempting to rank individuals on these behaviors.
Table 1.
Test-retest reliability assessment of the items included on the dietary screener, and validity assessment of the dietary screener compared to the “gold standard” of 3 days of 24-hour dietary recalls.
| Item | Reliability testing | Validity testing | ||||
|---|---|---|---|---|---|---|
| Spearman correlation | Simple kappa | P-value | Simple kappa (comparing tertiles of intake) | P-value | Reported consumption in recalls, n | |
| Beverage frequency | ||||||
| Regular soda | 0.82 (n=33) | -- | <0.001 | 0.38 (n=59) | <0.001 | 27 |
| Diet soda | 0.63 (n=33) | -- | <0.001 | -- | -- | 5 |
| Sports drinks | 0.75 (n=33) | -- | <0.001 | 0.25 (n=59) | 0.001 | 11 |
| Other sweetened beverages | 0.66 (n=33) | -- | <0.001 | 0.11 (n=59) | 0.11 | 27 |
| Milk | 0.84 (n=32) | -- | <0.001 | 0.27 (n=59) | 0.002 | 51 |
| Coffee drinks | 0.79 (n=33) | -- | <0.001 | -- | -- | 0 |
| Beverage amount | ||||||
| Regular soda | -- | 0.59 (n=31) | <0.001 | 0.19 (n=59) | <0.001 | 27 |
| Diet soda | -- | 0.73 (n=17) | <0.001 | -- | -- | 5 |
| Water | -- | 0.68 (n=33) | <0.001 | 0.20 (n=59) | <0.001 | 51 |
| Fast food | ||||||
| Overall frequency | 0.67 (n=33) | -- | <0.001 | 0.03 (n=59) | 0.38 | 35 |
| Fast food type | ||||||
| Traditional “burger-and- fries,” | -- | 0.60 (n=33) | <0.001 | -- | -- | 5 |
| Mexican | -- | 0.76 (n=33) | <0.001 | -- | -- | 6 |
| Fried chicken | -- | 0.80 (n=33) | <0.001 | -- | -- | 0 |
| Sandwich/subs | -- | 0.51 (n=33) | 0.003 | -- | -- | 6 |
| Pizza | -- | 0.34 (n=33) | 0.05 | -- | -- | 8 |
| Asian | -- | 0.20 (n=33) | 0.20 | -- | -- | 7 |
| Bakery/donut shop | -- | 0.10 (n=33) | 0.56 | -- | -- | 2 |
| Bagel shop | -- | 0.78 (n=33) | <0.001 | -- | -- | 1 |
| Coffee shop | -- | 0.51 (n=33) | 0.002 | -- | -- | 3 |
| Ice cream and burgers | -- | 0.63 (n=33) | <0.001 | -- | -- | 8 |
| Snack bar | -- | 0.39 (n=33) | 0.03 | -- | -- | 2 |
Footnotes
Publisher's Disclaimer: This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting proof before it is published in its final citable form. Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.
References
- 1.Willett W. Nutritional Epidemiology. 2. New York: Oxford University Press; 1998. [Google Scholar]
- 2.U.S. Department of Health and Human Services and U.S. Department of Agriculture. Dietary Guidelines for Americans. 6. Washington, DC: U.S. Government Printing Office; 2005. [Google Scholar]
- 3.USDA. [Accessed Jan 28, 2008];MyPyramid. www.mypyramid.gov.
- 4.Rockett HR, Wolf AM, Colditz GA. Development and reproducibility of a food frequency questionnaire to assess diets of older children and adolescents. J Am Diet Assoc. 1995;95:336–340. doi: 10.1016/S0002-8223(95)00086-0. [DOI] [PubMed] [Google Scholar]
- 5.Rockett HR, Breitenbach M, Frazier AL, Witschi J, Wolf AM, Field AE, Colditz GA. Validation of a youth/adolescent food frequency questionnaire. Prev Med. 1997;26:808–816. doi: 10.1006/pmed.1997.0200. [DOI] [PubMed] [Google Scholar]
- 6.Keith SW, Redden DT, Katzmarzyk PT, Boggiano MM, Hanlon EC, Benca RM, Ruden D, Pietrobelli A, Barger JL, Fontaine KR, Wang C, Aronne LJ, Wright SM, Baskin M, Dhurandhar NV, Lijoi MC, Grilo CM, DeLuca M, Westfall AO, Allison DB. Putative contributors to the secular increase in obesity: exploring the roads less traveled. Int J Obes (Lond) 2006;30:1585–1594. doi: 10.1038/sj.ijo.0803326. [DOI] [PubMed] [Google Scholar]
- 7.Koplan J, Liverman C, Kraak V, editors. Food and Nutrition Board. Institute of Medicine, Preventing Childhood Obesity: Health in the Balance. National Academy of Sciences; 2005. [PubMed] [Google Scholar]
- 8.Barlow SE. Expert committee recommendations regarding the prevention, assessment, and treatment of child and adolescent overweight and obesity: summary report. Pediatrics. 2007;120 (Suppl 4):S164–192. doi: 10.1542/peds.2007-2329C. [DOI] [PubMed] [Google Scholar]
- 9.Harnack LJ, Lytle LA, Story M, Galuska DA, Schmitz K, Jacobs DR, Jr, Gao S. Reliability and validity of a brief questionnaire to assess calcium intake of middle-school-aged children. J Am Diet Assoc. 2006;106:1790–1795. doi: 10.1016/j.jada.2006.08.014. [DOI] [PubMed] [Google Scholar]
- 10.Nielsen SJ, Popkin BM. Changes in beverage intake between 1977 and 2001. Am J Prev Med. 2004;27:205–210. doi: 10.1016/j.amepre.2004.05.005. [DOI] [PubMed] [Google Scholar]
- 11.O’Dea JA. Consumption of nutritional supplements among adolescents: usage and perceived benefits. Health Educ Res. 2003;18:98–107. doi: 10.1093/her/18.1.98. [DOI] [PubMed] [Google Scholar]
- 12.Shields DH, Corrales KM, Metallinos-Katsaras E. Gourmet coffee beverage consumption among college women. J Am Diet Assoc. 2004;104:650–653. doi: 10.1016/j.jada.2004.01.015. [DOI] [PubMed] [Google Scholar]
- 13.Conway JM, Ingwersen LA, Vinyard BT, Moshfegh AJ. Effectiveness of the US Department of Agriculture 5-step multiple-pass method in assessing food intake in obese and nonobese women. Am J Clin Nutr. 2003;77:1171–1178. doi: 10.1093/ajcn/77.5.1171. [DOI] [PubMed] [Google Scholar]
- 14.Jonnalagadda SS, Mitchell DC, Smiciklas-Wright H, Meaker KB, Van Heel N, Karmally W, Ershow AG, Kris-Etherton PM. Accuracy of energy intake data estimated by a multiple-pass, 24-hour dietary recall technique. J Am Diet Assoc. 2000;100:303–308. doi: 10.1016/s0002-8223(00)00095-x. quiz 309–311. [DOI] [PubMed] [Google Scholar]
- 15.Johnson RK, Soultanakis RP, Matthews DE. Literacy and body fatness are associated with underreporting of energy intake in US low-income women using the multiple-pass 24-hour recall: a doubly labeled water study. J Am Diet Assoc. 1998;98:1136–1140. doi: 10.1016/S0002-8223(98)00263-6. [DOI] [PubMed] [Google Scholar]
- 16.Johnson RK, Driscoll P, Goran MI. Comparison of multiple-pass 24-hour recall estimates of energy intake with total energy expenditure determined by the doubly labeled water method in young children. J Am Diet Assoc. 1996;96:1140–1144. doi: 10.1016/S0002-8223(96)00293-3. [DOI] [PubMed] [Google Scholar]
- 17.Schakel S, Sievert Y, Buzzard I. Sources of data for developing and maintaining a nutrient database. J Am Diet Assoc. 1988;88:1268–1271. [PubMed] [Google Scholar]
- 18.McGinn T, Wyer PC, Newman TB, Keitz S, Leipzig R, For GG. Tips for learners of evidence-based medicine: 3. Measures of observer variability (kappa statistic) CMAJ. 2004;171:1369–1373. doi: 10.1503/cmaj.1031981. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 19.Marshall T, Eichenberger Gilmore J, Broffitt B, Levy S, Stumbo P. Relative validation of a beverage fequency questionnaire in children ages 6 months through 5 years using 3-day food and beverage diaries. J Am Diet Assoc. 2003;103:714–720. doi: 10.1053/jada.2003.50137. [DOI] [PubMed] [Google Scholar]
- 20.Thompson FE, Subar AF, Smith AF, Midthune D, Radimer KL, Kahle LL, Kipnis V. Fruit and vegetable assessment: performance of 2 new short instruments and a food frequency questionnaire. J Am Diet Assoc. 2002;102:1764–1772. doi: 10.1016/s0002-8223(02)90379-2. [DOI] [PubMed] [Google Scholar]
- 21.Boucher B, Cotterchio M, Kreiger N, Nadalin V, Block T, Block G. Validity and reliability of the Block98 food-frequency questionnaire in a sample of Canadian women. Public Health Nutr. 2006;9:84–93. doi: 10.1079/phn2005763. [DOI] [PubMed] [Google Scholar]