Abstract
To assess whether a brief dietary screener designed to assess fast food and beverage consumption in a primarily Caucasian, adolescent population, is also valid and reliable in an overweight, adolescent, Latina population. This screener was developed by University of Minnesota to assess beverage consumption (9 items) and fast food consumption (13 items) in normal weight, primarily Caucasian, adolescents (ages 11–18). Thirty-five at risk for overweight (BMI ≥85th percentile), adolescent (ages 14–17), Latina females were recruited from East Los Angeles and completed the screener twice, approximately seven to 14 days apart, during the fall of 2007. Dietary intake was also assessed by three-day diet records. Spearman correlation and simple Kappas were employed for test-retest assessment and comparisons between the screener and the records. Test-retest assessment yielded a mean Spearman or Kappa statistic of 0.49 with 17 of the 21 responses being significant (P<0.05). Validity was much lower and yielded a Kappa statistic of only 0.08 and no responses were significant. Although this screener appeared to be a valid and reliable measure to assess beverage and fast food consumption in a primarily Caucasian, adolescent population, it does not appear to be appropriate for an overweight Latina female adolescent population.
Keywords: Validation and Reliability, Brief Dietary Screener, Overweight Latino Adolescents
INTRODUCTION
The prevalence of obesity continues to rise in children and adolescents. According to NHANES 2003–04 data, in the last three decades the prevalence of overweight among adolescents has more than tripled going from five to 17% (1, 2). This epidemic is even more pronounced in ethnic populations and pre-diabetes and type two diabetes have emerged as significant health issues in overweight adolescents, especially among Latina adolescents(3). We have previously shown that 32% of overweight Latino children in the Los Angeles area have pre-diabetes (4, 5). We have also previously shown that in this overweight Latino pediatric cohort, total sugar intake, specifically, sugar sweetened beverage intake, was the only dietary variable inversely related to insulin secretion (6) and positively related to adiposity (7).
There is strong evidence to support the negative effects of fast food consumption and sugar sweetened beverage intake on pediatric obesity (8, 9) and obesity-related metabolic disorders, such as type two diabetes (10, 11). To date, there are several subjective tools available to assess dietary intake, with food frequency questionnaires, diet records and 24-hour recalls being among the most common. Given that fast food and sugar sweetened beverage consumption are emerging as the leading culprits for childhood obesity and related diseases, particularly with Latino youth, it is surprising that there is not a culturally appropriate, screener that specifically assesses these foods and beverages. Thus, the purpose of this study is to assess the reliability and validity of a brief screener, which was developed and deemed reliable and valid in a normal weight, primarily Caucasian, pediatric population from the metropolitan area of Minneapolis/St. Paul, Minnesota (2008, unpublished data) to accurately capture fast food and beverage consumption in an overweight, Latina, female, adolescent population from Los Angeles.
METHODS
Screener development and initial pilot testing
The brief screener was adapted, reworded and expanded upon from previous research conducted in Minnesota in the IDEA cohort study and complete details of the development and pilot testing are outlined in the paper by Nelson & Lytle (2008, unpublished data). In short, the screener included questions on how often beverages and overall fast food intake are consumed over the past month with eight response options ranging from never/rarely, to three or more times per day. In addition, the screener further probes on the amount consumed of soda and diet soda. Then yes/no questions for specific types of restaurants are included, i.e., traditional hamburger, Mexican, fried chicken, etc.
The reliability and validity of the adapted 22-item screener was established with primarily Caucasian adolescents (11–18 years) from the Minneapolis/St. Paul metro region. One sample of adolescents completed test re-test of the screener instrument, approximately seven to 14 days apart (n=33, primarily Caucasian) and another sample of adolescents completed the screener questions along with three 24-hour dietary recalls (n=59, Caucasian). Test-retest assessments were high, with Spearman correlations and Kappa statistics that were > 0.60, between the two screener administrations. When compared to the dietary recall data, the screener items assessing adolescents’ intake of regular soda, sports drinks, milk and water yielded acceptable validity estimates (2008, unpublished data).
Participants
Participants were a sub-sample of the Strength and Nutrition Outcomes for Latino Adolescents (SANO LA) study, which is part of the Transdisciplinary Research on Energetics and Cancer (TREC) initiative, i.e., a multi-center effort to study and prevent obesity and cancer. SANO LA was a 16-week nutrition and strength training intervention study to reduce obesity and type two diabetes risk factors in overweight Latino adolescents from East Los Angeles. Study participants were required to meet the following inclusion criteria: 1) grade 9th thru 12th (14–18 years); 2) BMI ≥85th percentile for age and gender based on the Centers for Disease Control and Prevention guidelines (12); and 3) Latino ancestry (all four grandparents of Latino origin as determined by parental self-report). Participants were ineligible if they were taking medications known to affect dietary intake, body composition or insulin dynamics, had syndromes or diseases known to affect body composition or fat distribution, had any major illness since birth, or were pregnant. Prior to any testing procedure, informed written consent from parents and assent from the children were obtained. The Institutional Review Board of the University of Southern California, Health Sciences Campus, approved this study.
Analyses
Participants completed the screener while at the GCRC. Participants returned to the GCRC approximately one week later (mean 7±3.7 days) and completed the screener for a second time in order to assess test-retest reliability. In subsequent data analyses, test-retest reliability of screener items was evaluated using a combination of: (a) Spearman correlation coefficients (for categorical-response survey items with ≥5 response categories) and (b) simple kappa coefficients (for items with <5 response categories).
At the baseline visit participants were also given three-day diet records (2 weekdays and 1 weekend) to complete at home before their next GCRC visit. Dietary technicians, who were trained and supervised by a Registered Dietitian, instructed participants on how to accurately complete diet records. At the second GCRC visit the technicians clarified the dietary records. Data were analyzed using the Nutrition Data System for Research (NDS-R version 5.0_35, University of Minnesota, Minneapolis, MN). Beverage consumption was obtained using NDS-R-specified beverage sub-group classifications, while study investigators had to average restaurant/shop names by fast food type (e.g., “burger-and-fries,” Mexican, fried chicken), in order to correspond with possible options provided in the screener items.
To assess the validity of the screener, which reflects intake over the month, to that of the diet records, which reflects intake over three days, we divided beverage and fast food intake frequencies into tertiles. Given that all variables examined in the validity analyses had a small number of response categories (i.e., 3–4), validity was evaluated using only simple kappa coefficients. By assessing relatively “high,” “moderate” and “low” consumers via each tool, a more appropriate comparison was possible. Post hoc analyses were used to explore possible differential misclassification of tertile rankings (e.g, where the screener systematically overestimates tertile ranking compared to the recalls, or vice versa).
RESULTS AND DISCUSSION
Reliability and Validity Testing
Spearman and Kappa statistics ranged from 0.08 to 0.73, with the mean of 0.49 ± 1.7 (Table 1) indicating a moderate level of agreement according to Landis and Koch (13). Seventeen of the 21 test-retest comparisons were statistically significant (P<0.05).
Table 1.
Test retest reliability assessment of the items included on the dietary screener (n=35)
| Item | Spearman correlation | Simple kappa | P-value |
|---|---|---|---|
| Beverage frequency | |||
| Regular soda | 0.53 | -- | 0.001 |
| Diet soda | 0.69 | -- | <0.0001 |
| Sports drinks | 0.65 | -- | <0.0001 |
| Sweetened beverages | 0.37 | -- | 0.027 |
| Milk | 0.70 | -- | <0.0001 |
| Coffee drinks | 0.71 | -- | <0.0001 |
| Beverage amount | |||
| Regular soda | -- | 0.44 | 0.0001 |
| Diet soda | -- | 0.45 | <0.0001 |
| Water | -- | 0.61 | <0.0001 |
| Fast food | |||
| Overall frequency | 0.49 | -- | 0.003 |
| Fast food type | |||
| Traditional “burger-and-fries,” | -- | 0.45 | 0.005 |
| Mexican | -- | 0.58 | 0.0006 |
| Fried chicken | -- | 0.49 | 0.004 |
| Sandwich/subs | -- | 0.31 | 0.073 |
| Pizza | -- | 0.25 | 0.138 |
| Asian | -- | 0.51 | 0.002 |
| Bakery/donut shop | -- | 0.73 | <0.0001 |
| Bagel shop | -- | 0.35 | 0.038 |
| Coffee shop | -- | 0.08 | 0.632 |
| Ice cream and burgers | -- | 0.57 | 0.001 |
| Snack bar | -- | 0.28 | 0.112 |
Spearman correlations were used for categorical-response survey items with ≥ 5 response categories and simple Kappas were used for items with <5 response categories.
The overall mean Kappa statistic for validity testing was only 0.08 ±0.06 and none of the item responses were significant (see Table 2). This level of agreement is considered poor (13). It was not possible to generate stable estimates for validity for diet soda, sports drinks, and coffee and all of the specific fast food types since less than 10 participants reported consuming foods at any of these types of restaurants at least once a day within the three-day period.
Table 2.
Validity assessment of the dietary screener, as compared to three-day diet records (n=35)
| Item | Simple kappa | P-value | Number of subjects who reported consumption in records |
|---|---|---|---|
| Beverage frequency (tertiles) | |||
| Regular soda | 0.07 | 0.55 | 23 |
| Diet soda | -- | -- | 2 |
| Sports drinks | -- | -- | 4 |
| Sweetened beverages | 0.08 | 0.50 | 16 |
| Milk | 0.18 | 0.14 | 26 |
| Coffee drinks | -- | -- | 0 |
| Beverage amount | |||
| Regular soda | 0.01 | 0.89 | 23 |
| Diet soda | -- | -- | 2 |
| Water | 0.08 | 0.35 | 33 |
| Fast food (tertiles) | |||
| Overall frequency | 0.08 | 0.53 | 24 |
| Fast food type | |||
| Traditional “burger-and-fries” | -- | -- | 7 |
| Mexican | -- | -- | 2 |
| Fried chicken | -- | -- | 1 |
| Sandwich/subs | -- | -- | 3 |
| Pizza | -- | -- | 2 |
| Asian | -- | -- | 2 |
| Bakery/donut shop | -- | -- | 1 |
| Bagel shop | -- | -- | 0 |
| Coffee shop | -- | -- | 0 |
| Ice cream and burgers | -- | -- | 1 |
| Snack bar | -- | -- | 0 |
Beverage and fast food intake frequencies from both the screener and food record items were divided into tertiles.
Simple Kappas were used to assess validity between screener and food record items. Stable estimates for validity were not possible for diet soda, sports drinks, and coffee and all of the specific fast food types since less than 10 participants reported consuming foods at any of these types of restaurants at least once a day within the three-day period.
There were no statistically significant misclassifications when assessing the agreement in tertile rankings between reported intakes on the three-day diet records versus the screener for soda, milk, other sweetened beverages and fast food (data not shown). Post hoc analyses examining differential misclassification of tertile rankings by the screener versus three-day records shows that the range of congruent classification was from 37% (for soda) to 49% (for milk). Although not statistically significant, the screener did appear to overestimate compared to the records for soda (40.0% vs. 22.9%) and sweetened beverage intake (35.3% vs. 23.5).
We hypothesized that our testing would reveal a reliable and valid screener, which could be used to capture problematic dietary variables and rank study participants across TREC centers and populations. The brief dietary screener, developed by University of Minnesota, was found to be reliable and valid in a primarily Caucasian adolescent population (2008, unpublished data). However, when we tested the screener, using similar methods and statistics, but in an overweight Latina adolescent population, we found that the screener had moderate reliability for test-retest and poor validity when compared to three-day diet records.
There are several possible explanations for why the screener was shown to be valid and reliable in the Caucasian population but did not fare as well in our Latina population. Cultural differences with regard to food consumed and options on the screener could have contributed to the inconsistent validity results. Numerous adult studies have found that FFQs are less valid in minority populations compared to Caucasian populations (14–16). Fewer studies have assessed the validity across ethnicities in pediatric populations. One study by Jensen et al (17) showed that a FFQ to measure calcium intake was much less valid in Hispanic youth (10 to 18 years) compared to white youth. Another study found much lower validation coefficients in Hispanic and African American youth (7th and 8th graders) compared to their white counterparts (18). For our study, it is possible that the beverage and fast food options included on the screener were not culturally relevant.
Another possible explanation to consider is the lower socioeconomic status (SES) of the Latina population compared to the high SES of the primarily Caucasian population from Minneapolis. It is unclear whether SES has an effect on dietary reporting, especially in children. One study by Price et al. found that lower SES levels during childhood are associated with underreporting energy intake in adulthood (19), however the effect of under-reporting during childhood was not examined. In contrast, a study by Bandini et al. that compared energy intake from seven day diet records to doubly labeled water techniques in 109 preadolescent girls showed that income level was not significantly related to the accuracy of reporting (20).
The educational or literacy level of our sample should also be considered. Although the adolescents from both sites were in similar grade levels, high schools in the Los Angeles School District have historically poor academic performance levels and low literacy levels (21). Several adult studies have shown that lower levels of education are associated with under-reporting (22–24). This finding is not surprising given that most methods of recording food intake are dependent on literacy level. Maintaining day-to-day records of dietary intake may be easier for low literacy youth, particularly in comparison to having to estimate usual dietary intake over the past month (as is required by the screener).
In addition, there were some differences in the criterion measure used in the two studies. In our study we used three-day diet records for the validity analysis, whereas three 24-hour dietary recalls were used in the Minnesota sample. A review study by McPherson et al. on dietary assessment methods among school-aged children, showed that food records underestimated energy intake compared to doubly labeled water (DLW), whereas 24-hour recalls had higher agreement compared to direct observation (25). The completion of diet records may be more dependent on the literacy level of the child compared to the completion of 24-hour recalls. Interactive 24-hour recalls do not require a child to write down what they eat and they include standardized prompting with regard to how foods are prepared, portion size, and where foods were consumed. Given the low literacy level of our Latina population, the use of three, 24-hour recalls may have improved the validity.
There are a couple limitations that need to be addressed. First this study includes a relatively small sample size (n=35) and only includes females. This validation study was a secondary analysis and began after the initiation of the SANO LA intervention study. Therefore, the reliability and validity of this screener was only tested in a female sub-sample. But the homogeneity of this sample, all overweight Latina female teens, could also be considered a strength because it allows the screener to be tested in a very specific, high-risk population. However, given that only 56% of the Minnesota sample was female, this could, in part, explain the differences in validity findings. Numerous studies have shown that females underreport to a greater extent compared to males (22, 23) and especially so for foods that are perceived as socially unacceptable, e.g., sweets and junk food (26, 27), which were the items specifically probed for on the screener. Thus, the gender differences could have contributed to the inconsistent validity results between the two samples.
CONCLUSIONS
Our results suggest that nutritional assessment tools need to be developed by starting with formative assessment, determining the most culturally relevant foods for the diet concern and target population. More research should be conducted to make sure that the foods included and the format of the screener are acceptable for the population of interest. Other ways to improve validity include: 1) using more dietary record days to establish criterion validity, especially given the sporadic nature of dietary behaviors; 2) using a larger sample with both genders; and 3) using a criterion test that is not dependent on reading level. Additional research is needed to develop and test brief dietary screeners in all types of populations, particularly low-income, high-risk minority youth.
Footnotes
Publisher's Disclaimer: This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting proof before it is published in its final citable form. Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.
Contributor Information
Jaimie N. Davis, Assistant Professor, Department of Preventive Medicine, Keck School of Medicine, University of Southern California, 2250 Alcazar Street, Bldg CSC, Suite 200, Los Angeles CA 90033, 323-442-3066 (work); 323-442-4103 (fax), jaimieda@usc.edu.
Melissa C. Nelson, Assistant Professor, Division of Epidemiology & Community Health, University of Minnesota, 1300 S. 2nd Street, WBOB Suite 300; Minneapolis, MN 555454-1015, 612-624-8832 (work); 612-624-0315 (fax), nels5024@umn.edu.
Emily E. Ventura, Research Assistant, Department of Preventive Medicine, Keck School of Medicine, University of Southern California, 2250 Alcazar Street, Bldg CSC, Suite 200, Los Angeles CA 90033, 323-442-3198 (work); 323-442-4103 (fax), eventura@usc.edu.
Leslie A. Lytle, Associate Professor, Division of Epidemiology & Community Health, University of Minnesota, 1300 S. 2nd Street, WBOB Suite 300; Minneapolis, MN 555454-1015, 612-624-3518 (work); 612-624-0315 (fax), lytle@epi.umn.edu.
Michael I Goran, Associate Director and Professor, Department of Preventive Medicine, Keck School of Medicine, University of Southern California, 2250 Alcazar Street, Bldg CSC, Suite 200, Los Angeles CA 90033, 323-442-3027 (work); 323-442-4103 (fax), goran@usc.edu.
References
- 1.Ogden CL, Carroll MD, Curtin LR, McDowell MA, Tabak CJ, Flegal KM. Prevalence of overweight and obesity in the United States, 1999–2004. JAMA. 2006;295:1549–55. doi: 10.1001/jama.295.13.1549. [DOI] [PubMed] [Google Scholar]
- 2.Ogden C, Flegal K, Carroll M, Johnson C. Prevalence and trends in overweight among US children and adolescents, 1999–2000. JAMA. 2002;288:1728–1732. doi: 10.1001/jama.288.14.1728. [DOI] [PubMed] [Google Scholar]
- 3.Rosenbloom AL, Joe JR, Young RS, Winter WE. Emerging epidemic of type 2 diabetes in youth. Diabetes Care. 1999;22:345–54. doi: 10.2337/diacare.22.2.345. [DOI] [PubMed] [Google Scholar]
- 4.Cruz ML, Weigensberg MJ, Huang T, Ball GDC, Shaibi GQ, Goran MI. The metabolic syndrome in overweight Hispanic youth and the role of insulin sensitivity. JCEM. 2004;89:108–113. doi: 10.1210/jc.2003-031188. [DOI] [PubMed] [Google Scholar]
- 5.Weigensberg MJ, Ball GD, Shaibi GQ, Cruz ML, Goran MI. Decreased beta-cell function in overweight Latino children with impaired fasting glucose. Diabetes Care. 2005;28:2519–24. doi: 10.2337/diacare.28.10.2519. [DOI] [PubMed] [Google Scholar]
- 6.Davis J, Ventura E, Weigensberg M, et al. The relation of sugar intake to beta-cell function in overweight Latino children. Am J Clin Nutr. 2005;82:1004–1010. doi: 10.1093/ajcn/82.5.1004. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 7.Davis JN, Alexander KE, Ventura EE, et al. Associations of dietary sugar and glycemic index with adiposity and insulin dynamics in overweight Latino youth. Am J Clin Nutr. 2007;86:1331–8. doi: 10.1093/ajcn/86.5.1331. [DOI] [PubMed] [Google Scholar]
- 8.Ludwig DS, Peterson KE, Gortmaker SL. Relation between consumption of sugar-sweetened drinks and childhood obesity: a prospective, observational analysis. Lancet. 2001;357:505–8. doi: 10.1016/S0140-6736(00)04041-1. [DOI] [PubMed] [Google Scholar]
- 9.Giammattei J, Blix G, Marshak HH, Wollitzer AO, Pettitt DJ. Television watching and soft drink consumption: associations with obesity in 11- to 13-year-old schoolchildren. Arch Pediatr Adolesc Med. 2003;157:882–6. doi: 10.1001/archpedi.157.9.882. [DOI] [PubMed] [Google Scholar]
- 10.Schulze MB, Manson JE, Ludwig DS, et al. Sugar-sweetened beverages, weight gain, and incidence of type 2 diabetes in young and middle-aged women. JAMA. 2004;292:927–34. doi: 10.1001/jama.292.8.927. [DOI] [PubMed] [Google Scholar]
- 11.Schulze MB, Liu S, Rimm EB, Manson JE, Willette WC, Hu FB. Glycemic index, glycemic load, and dietary fiber intake and incidence of type 2 diabetes in younger and middle-aged women. Am J Clin Nutr. 2004;80:348–356. doi: 10.1093/ajcn/80.2.348. [DOI] [PubMed] [Google Scholar]
- 12.Centers for Disease Control and Prevention. CDC growth Charts. Atlanta, GA: U.S. Department of Health and Human Services, Centers for Disease Control and Prevention, National Center for Health Statistics. (U.S. Publ. no. 314); 2000. [Google Scholar]
- 13.Landis JR, Koch GG. The measurement of observer agreement for categorical data. Biometrics. 1977;33:159–74. [PubMed] [Google Scholar]
- 14.Stram DO, Hankin JH, Wilkens LR, et al. Calibration of the dietary questionnaire for a multiethnic cohort in Hawaii and Los Angeles. Am J Epidemiol. 2000;151:358–70. doi: 10.1093/oxfordjournals.aje.a010214. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 15.Liu K, Slattery M, Jacobs D, Jr, et al. A study of the reliability and comparative validity of the cardia dietary history. Ethn Dis. 1994;4:15–27. [PubMed] [Google Scholar]
- 16.Coates RJ, Monteilh CP. Assessments of food-frequency questionnaires in minority populations. Am J Clin Nutr. 1997;65:1108S–1115S. doi: 10.1093/ajcn/65.4.1108S. [DOI] [PubMed] [Google Scholar]
- 17.Jensen JK, Gustafson D, Boushey CJ, et al. Development of a food frequency questionnaire to estimate calcium intake of Asian, Hispanic, and white youth. J Am Diet Assoc. 2004;104:762–9. doi: 10.1016/j.jada.2004.02.031. [DOI] [PubMed] [Google Scholar]
- 18.Cullen KW, Zakeri I. The youth/adolescent questionnaire has low validity and modest reliability among low-income African-American and Hispanic seventh-and eighth-grade youth. J Am Diet Assoc. 2004;104:1415–9. doi: 10.1016/j.jada.2004.06.022. [DOI] [PubMed] [Google Scholar]
- 19.Price GM, Paul AA, Cole TJ, Wadsworth ME. Characteristics of the low-energy reporters in a longitudinal national dietary survey. Br J Nutr. 1997;77:833–51. doi: 10.1079/bjn19970083. [DOI] [PubMed] [Google Scholar]
- 20.Bandini LG, Cyr H, Must A, Dietz WH. Validity of reported energy intake in preadolescent girls. Am J Clin Nutr. 1997;65:1138S–1141S. doi: 10.1093/ajcn/65.4.1138S. [DOI] [PubMed] [Google Scholar]
- 21.Buckley J, Schneider M, Y S. LAUSD School Facilities and Academic Performance. 2008 [Google Scholar]
- 22.Klesges RC, Eck LH, Ray JW. Who underreports dietary intake in a dietary recall? Evidence from the Second National Health and Nutrition Examination Survey. J Consult Clin Psychol. 1995;63:438–44. doi: 10.1037//0022-006x.63.3.438. [DOI] [PubMed] [Google Scholar]
- 23.Briefel RR, Sempos CT, McDowell MA, Chien S, Alaimo K. Dietary methods research in the third National Health and Nutrition Examination Survey: underreporting of energy intake. Am J Clin Nutr. 1997;65:1203S–1209S. doi: 10.1093/ajcn/65.4.1203S. [DOI] [PubMed] [Google Scholar]
- 24.Ballard-Barbash R, Graubard I, Krebs-Smith SM, Schatzkin A, Thompson FE. Contribution of dieting to the inverse association between energy intake and body mass index. Eur J Clin Nutr. 1996;50:98–106. [PubMed] [Google Scholar]
- 25.McPherson R, Hoelscher D, Alexander M, Scanlon K, Serdula M. Dietary assessment methods among school-aged children: Validity and reliability. Prev Med. 2000;31:S11–S33. [Google Scholar]
- 26.Grogan SC, Bell R, Conner M. Eating sweet snacks: gender differences in attitudes and behaviour. Appetite. 1997;28:19–31. doi: 10.1006/appe.1996.0067. [DOI] [PubMed] [Google Scholar]
- 27.Schoeller DA. How accurate is self-reported dietary energy intake? Nutr Rev. 1990;48:373–9. doi: 10.1111/j.1753-4887.1990.tb02882.x. [DOI] [PubMed] [Google Scholar]