Dietary Assessment in Food Environment Research

Abstract

Context

The existing evidence on food environments and diet is inconsistent, potentially due in part to heterogeneity in measures used to assess diet. The objective of this review, conducted in 2012–2013, was to examine measures of dietary intake utilized in food environment research.

Evidence acquisition

Included studies were published from January 2007 through June 2012 and assessed relationships between at least one food environment exposure and at least one dietary outcome. Fifty-one articles were identified using PubMed, Scopus, Web of Knowledge, and PsycINFO; references listed in the papers reviewed and relevant review articles; and the National Cancer Institute's Measures of the Food Environment website. The frequency of the use of dietary intake measures and assessment of specific dietary outcomes was examined, as were patterns of results among studies using different dietary measures.

Evidence synthesis

The majority of studies used brief instruments, such as screeners or one or two questions, to assess intake. Food frequency questionnaires were used in about a third of studies, one in ten used 24-hour recalls, and fewer than one in twenty used diaries. Little consideration of dietary measurement error was evident. Associations between the food environment and diet were more consistently in the expected direction in studies using less error-prone measures.

Conclusions

There is a tendency toward the use of brief dietary assessment instruments with low cost and burden rather than more detailed instruments that capture intake with less bias. Use of error-prone dietary measures may lead to spurious findings and reduced power to detect associations.

Introduction

In the search for effective strategies to improve diet and fight the obesity epidemic—and given the intuitively appealing notion that the food environment is an important determinant of diet—interventions aimed at creating more supportive food environments are underway in jurisdictions in the U.S. and around the world. Examples include initiatives to improve food retail access in underserved areas, calorie labeling on restaurant menus, and restrictions on the availability of particular products, such as sugar-sweetened beverages, in certain settings.^1–5 While such initiatives may hold promise in contributing to an improvement in the dietary intakes and health of the population, we have much to learn about the extent to which the food environment influences health and the mechanisms by which it exerts this influence.^6–8

Understanding how the food environment affects dietary intakes is particularly salient to informing policy. Diet represents the most logical pathway between food environment features and health outcomes, such as overweight and obesity, cardiovascular disease, and type 2 diabetes.⁶ As a result, diet has been a common focus of studies aimed at elucidating the effects of the food environment on health. Like other areas of inquiry within the broader field of food environment research,^7,9–11 the existing evidence on the food environment and diet is inconsistent, with some studies showing associations in the hypothesized direction, others showing null effects, and still others finding relationships opposite to those expected.⁶ Although it is possible that these inconsistencies are due to true differences in the influence of the food environment among different contexts¹² and populations and in relation to specific dietary outcomes, these discrepancies may also reflect methodologic heterogeneity across studies. A number of review articles have focused on measures used to capture environmental exposures.^{6,9,11,13–22} For example, in their recent review, Caspi et al.⁶ evaluated the literature on the food environment and diet, finding that despite a large number of studies on the topic, reproducibility was limited due to a lack of standards for measuring environmental exposures. Although not their primary focus, Caspi and colleagues also drew attention to measures used to characterize dietary outcomes, finding no comprehensible association between the quality of methods used to assess exposures and those used to assess outcomes among the studies reviewed.^6,23

Accurately measuring diet is a challenge. Interest typically lies in how characteristics of food environments influence usual or long-run average dietary intakes, which cannot be directly observed among free-living individuals. Therefore, there is a reliance on self-report measures, such as 24-hour recalls (24HR), food records or diaries, food frequency questionnaires (FFQ), and screeners.²⁴ However, biomarker-based validation studies have documented serious underreporting and other reporting errors in self-reported dietary intake data.²⁵ Error can be reduced through the selection of the most appropriate instrument for the dietary outcome of interest. It is therefore important to consider dietary measurement error in study design, as well as in the interpretation of study findings.

The objective of this article is to review the food environment literature with a focus on the methods used to characterize diet. This effort adds to the insights provided by previous reviews by illuminating the need for heightened attention to the robustness of measures used to assess outcomes in food environment research.

Evidence Acquisition

Articles published in English in peer-reviewed journals from January 2007 through June 2012 were considered; this 5-year period was selected to capture recent trends in food environment research, and because few articles reporting on studies examining relationships between food environment features and diet were published prior to 2007.⁶ Included studies assessed relationships between at least one food environment exposure (i.e., access to or characteristics of food stores, restaurants, schools, public facilities) and at least one dietary outcome. (Although the literature currently consists of predominantly cross-sectional research, the terms exposure and outcome are used in this paper since these reflect the presumed directionality of the relationships examined; i.e., diet is modeled as the dependent variable, with environmental exposures as independent variables.) Studies that reported on the effects of an intervention rather than an environmental exposure per se, that considered only the food environment within the home, or that were conducted within laboratory settings were excluded. Studies focusing only on indirect indicators of diet, such as food purchasing or frequency of visits to a food outlet, rather than intakes were also excluded. If more than one article was available for a single study, only one was considered unless different articles reported on unique exposures and/or outcomes.

Articles were identified using the keywords (food OR nutrition OR diet) AND (environment OR community OR neighborhood OR neighbourhood) AND (measure) AND (assess) OR food environment and the search engines PsycINFO, PubMed, SCOPUS, and Web of Science. The use of these multiple search engines was intended to provide full coverage of the literature given that they represent major repositories of peer-reviewed health research. A total of 2450 unique references were identified, 149 of which were selected for full-text review based on an assessment of the titles and abstracts and 38 of which met the inclusion criteria (Figure 1). Articles selected for full-text review were reviewed by at least two members of the review team; in a small number of cases in which there was not initial consensus, the review team met to discuss the article and make a decision based on the inclusion criteria.

Figure 1.

Article identification and selection process

An additional 13 articles meeting the inclusion criteria were identified through review of (1) the reference lists of each article identified and relevant review articles and (2) citations included in the National Cancer Institute's Measures of the Food Environment website (www.riskfactor.cancer.gov/mfe), an online repository of food environment research that is updated on an ongoing basis. The final pool of articles were reviewed from 2012 to 2013. Pertinent methodologic details of each article, including the setting, study sample, the food environment exposure(s), and the dietary outcome(s) and outcome measure(s), were examined.

Exposure methods and measures include geographic analyses to characterize the macro or community food environment (e.g., density of or physical access to food outlets); surveys, inventories, or checklists used to assess the characteristics of a given environment (e.g., availability, quality and/or price of certain foods and beverages in food stores or schools); and surveys used to assess perceptions of the food environment. More detailed examinations of exposure methods and measures are available elsewhere.^{6,9,11,13–22}

Outcome measures are self-report dietary assessment instruments including 24HR, food records or diaries, FFQ, and screeners (Table 1). A few studies made use of a focused or targeted 24HR, which uses techniques similar to those of 24HR, but collects information on particular foods or food groups (e.g., fruits and vegetables).²⁶ Instruments that captured frequency information on only one or a finite number of foods or food groups rather than the total diet were categorized as screeners, even if described as FFQ within the original article. Studies that used only two items or a single-item question to measure intake of a particular food or food group (e.g., how many fruits and vegetables do you consume in a typical day?) were differentiated from more extensive screeners. The outcome measures used were sometimes not clearly described, with authors using various names to identify instruments or referring only in passing to dietary data collected as part of a larger survey. Decisions about how to categorize instruments were based on the reviewers’ judgment of the description included in the original article, cited references, and if available, review of other documentation on the data source. In addition to examining the frequency of the use of each instrument, the prevalence of the assessment of specific dietary components, including fruits and/or vegetables, sugar-sweetened beverages, and fast food, as well as diet quality, was examined. These components were selected since preliminary analysis showed that they were the most frequently studied outcomes.

Table 1.

Overview of self-report dietary assessment instruments^a

Instrument	Description	Main sources of error
24-hour recall (24HR)	Captures detailed information regarding all foods and beverages consumed by the respondent the previous day. Typically uses multiple passes to collect detailed information about foods consumed and to enhance accuracy and completeness. Traditionally interviewer-administered but self-administered systems are now available.	Random error, primarily driven by day-to-day variation in intakes.
Food record/diary	A self-reported, real-time accounting of all foods and beverages consumed by the respondent on one or more days. Portion size may be estimated or weighed.	Random error, primarily driven by day-to-day variation in intakes Reactivity bias, in which the respondent changes his/her intake behavior or reported intake in response to the act of recording.
Food frequency questionnaire (FFQ)	Consists of a finite list of foods and beverages with response categories to indicate usual frequency of consumption (and possibly usual portion size) over the time period queried (e.g., past month or past year). For assessment of the total diet, the number of foods typically ranges from 80 to 120. Typically self-administered.	Systematic error (bias), driven by lack of detail and error inherent in cognitive task of estimating usual intake over a period of time.
Screener	A short food frequency type of questionnaire, usually without portion size questions or a short behavioral type of questionnaire that asks about general dietary practices, for example, do you generally butter your bread? Typically self-administered.	Systematic error (bias), driven by lack of detail and error inherent in cognitive task of estimating usual intake over a period of time.

^aSingle-item questions and pairs of questions are also employed to assess intake; for example, by asking about typical amount or frequency of consumption of fruit or vegetables, or frequency of consumption of fast food. The resulting data are likely to be biased because of a lack of detail and the difficulty inherent in estimating usual consumption.

A brief summary of the key findings of each study is provided for the purpose of helping to inform future research and, to the extent possible, policies and programs. It should be noted that this summary is not exhaustive and does not include the results of every test conducted within a given study, particularly those with null findings. Finally, the overall direction of the effects observed in the reviewed studies was examined.

Evidence Synthesis

Appendix A provides summaries of the 51 articles,^27–77 organized by dietary outcome measure (available online at www.ajpmonline.org). Researchers used a cross-sectional study design, with the exception of Boone-Heinonen et al.,²⁷ who used a longitudinal design. Thirty studies (58.8%) included youth in their sample, but young children were not frequently studied; the youngest age specified was 5 years. Aspects of the food environments studied were varied and assessed using geographic analysis, checklists, inventories, menu analysis, and interviews.

In over two thirds of studies, a brief instrument, such as a screener or one or two items, was used to assess dietary outcomes (Figure 2). The most common approach was single-item questions (used in 35% of studies), while 6% used two items. A quarter of the studies (27%) used a screener and another quarter (27%) included an FFQ. About one in 10 (10%) used one or more 24HR and only two (4%) used food diaries (note that the total exceeds 100% because some studies included multiple instruments). Studies that relied on more detailed measures (24HR and diaries) typically appeared to be secondary analyses of survey datasets or analyses of data collected as part of a larger study in which the question of how the food environment relates to diet was potentially one of many lines of inquiry.

Figure 2.

Dietary assessment instruments used in studies (n=51) examining associations between food environments and dietary outcomes

^a One study included both a two-item measure and a single-item measure; thus, the total number of studies using either 1 or 2 items is 20, as indicated in the Appendix.

^b The outcomes included are those most commonly examined among the studies reviewed. Note that a single study may include multiple outcomes and so the sum of studies examining unique outcomes exceeds the total number of studies reviewed.

^c The count of instruments exceeds the number of studies reviewed because one or more studies used multiple instruments.

The most commonly studied dietary components were fruit and/or vegetables (35 studies, 69%); sugar-sweetened beverages (described as sugar-sweetened beverages, sugared beverages, soda, soft drinks, pop, fruit drinks, fizzy drinks [14 studies, 27%]); and fast food (9 studies, 17%) (Figure 2). Studies that assessed fruit and vegetable consumption used a variety of measures, but a brief instrument, such as a screener or single-item question, was most common. The use of a single-item question was also common for the assessment of fast food consumption, with five of nine studies using this approach. Over half of the studies that assessed the consumption of sugar-sweetened beverages used a brief instrument. Nine studies (17%) included a measure of diet quality, with an FFQ most commonly used for this purpose.

A review of the methods and discussion sections of the articles suggests little attention to the issue of error in dietary intake data and most papers did not include a consideration of the potential impact of such error on the findings. However, in several papers, the authors indicated that the dietary assessment measure(s) used had been validated.^{27,29,34,36,37,39–41,43,45,47,51,52,55,59,62,63,65,68–70,72,76} Validity was often assessed by comparing the instrument in question to another more detailed self-report instrument (e.g., multiple 24HR or a food record). An examination of the key findings of all studies reviewed shows results in the expected direction (although often of small effect size), null effects, and effects opposite to those expected (although it should be noted that many studies had mixed results).

When the studies were limited to those that did not use brief instruments (screeners or one or two items) to assess diet, a more consistent pattern of effects in the expected direction is apparent. Among the 21 studies that included 24HR, diaries, or FFQ to assess diet, 16 (76%) showed overall effects in the expected direction and the remaining 5 had null, mixed or unexpected results (Appendix A, available online at www.ajpmonline.org). Among the 33 unique studies using brief instruments (Arcan et al.²⁹ used both screeners and a single-item question, resulting in 33 rather than 34 studies), 18 (55%) showed overall effects in the expected direction, four had overall effects in an unexpected direction, and the remaining 11 were mixed or null (Appendix A, available online at www.ajpmonline.org).

Discussion

Although the measurement of features of local food environments has been reviewed in several articles,^{6,9,11,13–22} attention is also needed to approaches to measuring diet. Importantly, the selection of dietary measures used in assessments of the food environment on dietary behavior appears to influence the consistency of findings. This review shows a tendency toward the use of brief dietary assessment instruments as opposed to more detailed measures. The choice of brief instruments, which tend to impose lower cost and respondent burden than more intensive methods, may be a consequence of limited resources, a limited scope of interest in terms of dietary outcomes, and/or potentially a higher prioritization of the measurement of environmental exposures versus dietary outcomes. However, depending on the outcome of interest, the use of brief instruments may be problematic from the perspective of measurement error. The current review also suggests little attention to the issue of dietary measurement error, regardless of the assessment instrument used.

All data collected using self-report dietary assessment instruments contain measurement error, but the types and extent of the error depend on the specific instruments used and the dietary outcomes of interest (Table 1). Insights into measurement error in intake data have come from validation studies using recovery biomarkers for energy and protein.²⁵ The Observing Protein and Energy Nutrition study, which used recovery biomarkers to estimate true intakes of energy and protein, indicated that data collected using 24HR data are affected to a lesser extent by systematic error (a type of error in which measurements consistently depart from the true value in the same direction) as compared to FFQ data.²⁵ In addition to other factors, this is likely related to the complexity of the task involved; recalling intake for the previous day using a 24HR is less complex cognitively than is estimating usual intake over a longer period, such as the past month or year, using an FFQ. Further, the detailed information collected using 24HR methodology enables more accurate estimates of nutrient and food group intakes, whereas with a food frequency questionnaire, respondents are limited to a finite list of foods that may not exactly represent their food consumption.²⁴

Recovery biomarkers are currently known for only a few dietary components, making it impossible to assess how well reported intake using these instruments reflects true intake of other components, such as fat, sugar, or fruits and vegetables. Nonetheless, the existing evidence favors recalls over FFQ in terms of collecting intake data less affected by systematic error. Given that food records or diaries also capture detailed information about food intake for a short period of time, the resulting data are likely to be less affected by systematic error than FFQ data, although reactivity bias (i.e., changes in eating behavior or reporting in response to the act of recording) can be a problem with records and diaries because they capture data in real time.²⁴

Less is known about error in brief tools for measuring diet. The premise of such instruments is that the factors to be estimated should be concentrated in a small number of food sources. Since they do not capture total energy or protein intake, it is not possible nor relevant to assess their performance relative to the known recovery biomarkers. However, based on the task involved (e.g., averaging intake of foods over some period of time, with relatively little detail recorded), data collected using such tools are likely to contain considerable systematic error when used to assess foods and food groups that could be commonly consumed in many foods and beverages, such as fruits and vegetables. One study that compared estimates of fruit and vegetable intake from two sets of two items and a 16-item screener to those from multiple 24HR concluded that, although the screeners were more cost effective and less burdensome than the 24HR, they were not appropriate for assessing precise intake levels.⁷⁸ That study also found that a longer screener performed better than did the two-item screeners. This is not surprising given the substantial cognitive challenge posed to a respondent by one or two questions that require estimating usual intake of a broad category of foods, such as fruits and vegetables, over some period of time.

As noted above, brief instruments were commonly used for the capture of various dietary outcomes, leading to concerns about error in estimated intakes. This error can lead to biased estimates of relationships between food environments and diet, as well as reduced statistical power, making it difficult to detect critical relationships that truly exist. Thus, error in dietary data used in the food environment literature could explain, at least in part, null effects and effects in the direction opposite to that hypothesized within this body of research. Error may also contribute to the small effects observed in a number of studies, even when in the expected direction. Interestingly, when the pool of studies examined in the current review was limited by excluding those that used brief instruments to assess dietary intake, a more consistent pattern of effects in the expected direction is apparent, highlighting the potential role that error in dietary measures may play in the inconsistency in the literature overall.

It should be noted however that concerns about error in data from brief instruments may not apply to studies in which interest is in capturing intake of a specific type of food or drink. For example, a brief instrument may be appropriate in studies focused on fast food, depending on whether the intent is to capture the behavior of consuming fast food (e.g., how often it is consumed) or to examine the nutritional quality of fast food meals eaten; in the latter case, a more comprehensive measure of intake is likely needed. Similarly, in studies in which the aim is to assess intake of sweetened carbonated soft drinks (i.e., soda), one question may be appropriate. However, if interest is in the broader class of sugar-sweetened beverages, one or two items are unlikely to be sufficient to accurately capture usual intake of the various items in this category, including soft drinks, fruit drinks, sports drinks, and flavored milks.

Over the past several years, there has been much work undertaken to improve approaches for accounting for measurement error in dietary intake data. However, this work has largely focused on surveillance and epidemiology,⁷⁹ and the strategies developed, such as regression calibration, are not applicable to situations in which diet is the outcome rather than the exposure of interest. Further research is needed on approaches for mitigating error in such cases, but in the meantime, the use of instruments with less systematic error is encouraged. Based on our current understanding, 24HR and food records or diaries are among the preferred modalities of measurement because they are less affected by systematic error, as noted above. The collection of detailed intake data using these tools also has the advantage of allowing researchers to examine many aspects of diet. Further, the use of 24HR and records is conducive to comparability across studies given that they are not based on food lists that may vary, as is the case with FFQ and screeners. Recall and food record data collected are affected to a greater degree than other tools by random error, driven largely by day-to-day variation in intake, which may result in a loss of power to detect effects. However, this power loss can be offset by averaging repeat administrations of 24HR or records. Repeat measures are recommended for the full sample (not for a subset) to maximize the ameliorative effects.

Use of 24HR methodology has traditionally been cost prohibitive for many studies due to the need for trained interviewers and replicate measures. Accordingly, among the studies reviewed, 24HR methodology appeared to be somewhat limited to those that reported on secondary analyses of datasets. However, new tools such as the Automated Self-Administered 24-hour recall (ASA24) system⁸⁰ have the potential to make it feasible to collect high-quality recalls from even large samples at low cost.⁸¹ Technologic innovations are also underway for food records and may reduce the cost and burden associated with collecting and coding food record data, although other potential sources of error, such as reactivity bias (in which respondents change their intakes or reporting of intakes in response to the act of recording) must be considered,^24,26 as noted above. Other factors such as respondent burden and challenges associated with low literacy that could affect response rates, attrition and data quality and thus, study findings, should also be considered with the use of more intensive assessment methods, such as 24HR and records. Further research is needed to identify strategies for the effective use of innovative methods of dietary assessment to maximize response rates and data quality and minimize researcher and respondent burden.

In situations in which it is not feasible to use more detailed measures of dietary assessment due to resource or time constraints, researchers may continue to turn to brief instruments. However, the outcome of interest should be carefully considered to assess whether it is likely to be accurately reported using a limited number of items. This consideration may also apply to FFQ, which may include a small number of items on any given food or food group. For example, in the case of components like fruits and vegetables that are widely dispersed throughout the food supply, multiple items designed to capture cognitively distinct sources (e.g., fruits and vegetables consumed alone as well as in mixtures) are recommended. In contrast, a single item may be acceptable for measuring intake of sweetened carbonated soft drinks, as noted above. In studies using a brief instrument, one potential approach to reduce error is to calibrate the data to a more accurate instrument. For example, scoring algorithms have been developed for the calibration of the National Cancer Institute's Dietary Screening Questionnaire, based on 24HR data from the National Health and Nutrition Examination Survey.⁸² Work is underway to determine the utility of this approach in cases in which diet is the outcome.

In addition to using measures thought to capture the dietary component of interest with the least systematic error possible, the use of objective measures is another potential means of lending support to findings of studies on food environments and diet. The use of concentration biomarkers (which unlike recovery biomarkers cannot be used to ascertain true intake, but do nonetheless have some relationship to intake) such as serum carotenoids, in conjunction with intake data has been suggested as a means of reducing the effects of measurement error in epidemiologic studies.^83,84 The use of objective measures, including biomarkers but also less invasive and perhaps more feasible markers such as sales or purchasing data, should be explored as a means of corroborating findings and building a stronger evidence base on relationships between the food environment and diet.

Attention to the validity of the instruments used for the population of interest is also warranted. As noted, a number of authors indicated that the instruments were validated; however, in most cases it appeared that validation was conducted against other self-report instruments that are also affected by error. This validation approach is limited but probably the most feasible, especially for instruments focused on dietary components for which no recovery biomarkers are known (e.g., fruits and vegetables, sugar-sweetened beverages). Other options could include intensive measures, such as direct observation or duplicate portions.

Studying the effect of the food environment on diet is challenging. From the perspective of characterizing environmental exposures, there are many considerations, such as the specific food environment to focus on, what features of that environment are most salient, and what tool to use to capture those features.^6,9,10,13,14 To contribute to a more robust evidence base upon which to base policy recommendations for creating food environments conducive to healthy diets, the capture of dietary outcomes requires equal attention. Considering measurement error in the selection of dietary assessment instruments and interpretation of findings may improve our understanding of how the food environment affects diet and health.