Considering the Value of Dietary Assessment Data in Informing Nutrition-Related Health Policy^,

Abstract

Dietary assessment has long been known to be challenged by measurement error. A substantial amount of literature on methods for determining the effects of error on causal inference has accumulated over the past decades. These methods have unrealized potential for improving the validity of data collected for research studies and national nutritional surveillance, primarily through the NHANES. Recently, the validity of dietary data has been called into question. Arguments against using dietary data to assess diet–health relations or to inform the nutrition policy debate are subject to flaws that fall into 2 broad areas: 1) ignorance or misunderstanding of methodologic issues; and 2) faulty logic in drawing inferences. Nine specific issues are identified in these arguments, indicating insufficient grasp of the methods used for assessing diet and designing nutritional epidemiologic studies. These include a narrow operationalization of validity, failure to properly account for sources of error, and large, unsubstantiated jumps to policy implications. Recent attacks on the inadequacy of 24-h recall–derived data from the NHANES are uninformative regarding effects on estimating risk of health outcomes and on inferences to inform the diet-related health policy debate. Despite errors, for many purposes and in many contexts, these dietary data have proven to be useful in addressing important research and policy questions. Similarly, structured instruments, such as the food frequency questionnaire, which is the mainstay of epidemiologic literature, can provide useful data when errors are measured and considered in analyses.

Introduction

Over the past 40 years, there has been a proliferation of research aimed at understanding the role of diet in health, with 93% of articles with a MeSH heading including the word “diet” or diet as a text word, published from 1973 to late May 2014. The bulk of this work in humans involved epidemiologic studies assessing the influence of diet and nutrition on disease risks. The vast majority of these studies are observational, with only limited experimental-trial representation. In addition, there is a smaller literature based on ongoing nutrition surveys designed to assess and monitor the content and quality of diet in populations.

Reflecting the growing interest in diet and health and building on national health surveys conducted primarily in the 1960s, the U.S. government began long-term monitoring of the food and nutrient intake and nutritional status of the U.S. population through the first NHANES performed from 1971 to 1973 (1). Several other waves of the NHANES were conducted, including one focused on Hispanics. In each of these, representative samples of the U.S. population were surveyed (2). In 1999, the NHANES operations were converted into a continuous, ongoing survey activity, now overseen by the Centers for Disease Control and Prevention (3). The data derived, including estimates of food and nutrient intake and measurements of nutritional biomarkers among numerous other health indicators, formed a basis for examining secular trends and to provide data to help inform federal food and nutrition policy.

Although NHANES and other national survey data help inform federal nutrition policy, it has long been recognized that self-reported dietary data are challenged by systematic and random measurement error. This is true not only for survey-derived measures but for all data derived from standard interview- or questionnaire-based dietary assessment methods (4). These assessment methods include the 24-h dietary recall interview (24HR)¹⁴, which forms the backbone of the NHANES dietary assessment but is used much less frequently in nutritional epidemiology because of its high costs. Reporting errors also exist in the food frequency questionnaire (FFQ), a structured questionnaire that is a mainstay of analytic epidemiologic studies (5, 6).

A recent article by Archer et al. (7) underscored this known limitation by stating that “across the 39-y history of the NHANES, [energy intake] data on the majority of respondents … was not physiologically plausible.” That article and 2 published since (8, 9) attempt to discredit a long history of nutritional research that has provided a robust literature consisting of >1 million articles published in the broad area of diet and health since 1946 and that formed the basis of numerous health recommendations and guided the nutrition-related public policy debate. Recommendations by scientific advisory bodies charged with addressing the role of diet in health parallel and reinforce some of the concerns raised about errors in dietary self-report (10). Therefore, it is important that the criticisms be clarified so that they may be addressed. The purpose of this article is to identify specific issues raised by these authors with respect to putative flaws in dietary assessment and their role in both epidemiologic studies and nutritional surveys.

Issues Drawn from Recent Literature

Nine issues were identified from the Advisory Committee on the Dietary Guidelines for Americans, 2010, which recently published criticisms of nutritional survey data that relate to measuring diet, identifying and controlling for errors in self-reports, designing studies, conducting surveys, and drawing inferences to inform public policy (7–10). Comprehending both the nature and consequences of measurement error is necessary to allow for continuing improvement in dietary assessment methodologies and to make informed inferences from existing sources of information.

First, it is well known that surveys based on dietary self-report underestimate total energy intake (EI) compared with estimates of metabolic need (11–14). Recognition of a 20–25% underestimate in mean EI observed in large-scale population-based surveys, such as the NHANES, spawned extensive research that has deepened the understanding of self-report errors (5, 6, 15–38) and led to improved measurement methods (6, 37, 39–44) and analytic techniques (5, 21, 24, 26, 45–48) to mitigate their effects (Table 1). This culminated in practical applications for estimating the effect of diet on health outcomes of free-living individuals (49, 50). In contrast, definition of validity using “disparity values,” e.g., testing if measured EI falls within a 95% CI for predicted EI (7), is limited because it fails to provide a measure of the signal strength relative to the underlying error structure of the data.

TABLE 1.

Key advances in assessment and interpretation of dietary data and suggestions for continued improvement

Key advances	Suggestions for continued improvement
1. Increased understanding of self-report errors with improved measurement methods and analytic techniques to mitigate their effects (5, 6, 15–40, 42–48, 127, 128).	Explore sources of reporting biases and develop means to measure and reduce the bias or mitigate its effect. Explore construct validators.
2. Single days of the 24-h dietary recall interview have large intra-individual variation that can be reduced with multiple days of recall (51–53).	When practical, increase the number of 24-h dietary recall interviews in those circumstances in which that method is appropriate.
3. Better understanding of intra-individual variability in energy intake led to improvements in accounting for this source of variability (55–60).	Conduct research in disparate populations to increase understanding of the sources of this variability.
4. Improved understanding of the many factors that play a role in underestimation of energy intake, including portion-size estimation errors, omissions of foods consumed, inclusion of foods not consumed, and self-report biases (e.g., social desirability or overweight/obesity status) (30, 32–38, 51, 52, 63–65, 70).	Total energy intake is an important factor in energy balance and therefore may be an important determinant of health outcomes, so future work should focus on improving its estimation. This could include designing small, focused studies that use the method of triads (129, 130) using a variety of criterion validators, such as energy expenditure from doubly-labeled water. Design studies to test whether or not the new technology-based methods for dietary assessment, such as the National Cancer Institute Automated Self-Administered 24-H Recall, collect better data than the traditional non-technology–driven method. Also, use of multiple recalls/records as main instrument combined with FFQ data could provide better energy intake estimates.
5. Rigorous studies of identified sources of bias showed them to be of sufficient magnitude to explain observed errors in measurement of energy intake (5, 32, 59, 127, 128).	Identify and quantify self-report biases in measuring diet, including total energy intake.
6. Conversion of foods to nutrients introduces error that may bias temporal comparisons but is not a major source of systematic bias in either group comparisons or analytic epidemiologic studies of point estimates of dietary intake and health outcomes (84–86).	Improve nutrient databases and conduct research to deepen understanding regarding the conversion of foods to nutrients and other bioactive compounds.
7. Research showing that errors in nutrient measurement do not necessarily lead to misclassification and bias; careful collection of additional data are needed to understand the extent to which bias may occur with systematic errors in measurement (37, 63, 70, 88).	Use existing databases and conduct future research to understand the effect of measurement error in misclassification on estimating health effects in epidemiologic studies.
8. Nutrition-related policy is based on evidence from multiple sources; inferences drawn from nutrition surveys alone rarely drive policy (90, 92, 93, 131).	Monitor how policy-oriented decisions are made in relation to the scientific basis for decision-making. Seek and exploit opportunities for better-quality observational studies and trials to inform key diet and health questions. This would include establishing cohorts under favorable circumstances with some combination of greater exposure variability, reduced bias, and improved measurement methods.
9. Evidence of diet effects on health come from a variety of research designs and sources, with studies using national nutrition surveys providing only a small fraction of the total evidence base (123, 132–133).	Educate the scientific community on the use of existing databases, including national and other surveys, to estimate the effect of diet on health.

Second, the recent literature critiquing dietary data focuses heavily on the use of the 24HR in surveillance, especially the NHANES. This method was selected to characterize population or group intake and not for use as a measure of individual intake. When this survey-design decision was made, it was well known that self-reported EI based on a single 24HR is not necessarily indicative of usual or habitual intake, on either an individual or group basis. Any particular day represents the potential for extremes which, were they to represent a long-term average, might be biologically implausible. Multiple days of 24HR, including weekend days, are needed to account for day-to-day variation, and a minimum number of days is necessary to estimate EI with adequate precision (51, 52). Moreover, the number of days of data needed varies by weight category (53). The use of a single day of data to characterize an individual’s usual diet (see reference 7) is now considered insufficient to generate reliable estimates of intake. As a result, we have seen over time an evolution in the methodology in which multiple days, including weekends, are randomly sampled, and the multi-pass interview protocol is used, which demonstrated better agreement (54).

Third, examining extreme values in survey data requires an understanding of statistical and related issues. Large-scale survey data based on a single-day 24HR, such as those from the NHANES, reflect both interperson and intraperson sources of variability (55). Although this may have little or no effect on the estimated group mean value, failing to take into account the effect of adding this relatively large intraperson error component, which typically accounts for at least half of the total variability for a variable such as EI, overestimates the variance and inflates the proportion of the population in the extreme portions of the distribution (56–60). Indeed, when we reported NHANES data to compare homogeneity of within-U.S. population nutrient intake to international norms, we were careful to adjust for intra-individual variability (55, 61). Failing to do so would result in inflating the number of individuals who provide either lower-than-average or higher-than-average estimates of dietary intake, i.e., the very group on which some authors tend to base their conclusions. Furthermore, using a reported EI/basal metabolic rate (BMR) cut point of 1.35 (as in reference 7) would inflate the number of implausibly low values if adjustments are not made for intra-individual variability. Bingham (45) cautioned that values <1.20 (especially if predicted rather than measured) should be excluded from analyses with certainty as erroneous EI. Goldberg et al. (62), who developed EI/BMR cut points for assessing underreporting of EI, advised using a cut point of 1.35 only if BMR was actually measured; however, when an estimate of habitual dietary intake is attempted, for example, by use of a single 24HR, then a more liberal cut point of 0.92 is warranted.

Fourth, a core group of factors play a role in underestimation, including the following: 1) portion-size estimation errors; 2) omissions of foods consumed; and 3) self-report biases (e.g., social desirability or overweight/obesity status) (38, 63). The first 24HR in a sequence underestimates EI to a greater extent than subsequent 24HR. This is likely due in part to more food omissions (51, 52, 64) and portion estimation errors (64, 65) that diminish over time, possibly as the result of increasing familiarity with the method that comes from repeated interviews or changes in the demand characteristics that could modify response bias. These are systematic biases that lead to underestimation in intake estimates but are not necessarily differential in nature and may not interfere with the ability to differentiate or rank individuals or groups in a population. Although the older NHANES datasets are limited to a single day of 24HR, this limitation does not apply to other survey data or to most of the research studies that use 24HR (38, 51, 66, 67). Using NHANES methodology (i.e., the 5-step automated multiple-pass method and 2 24HRs) and modifying it to include a third 24HR has shown greater agreement with estimates of energy expenditure using doubly-labeled water and resulted in less underreporting (54). In addition, under-eating as a conscious effort to lose weight is highly prevalent among Americans, particularly those who are overweight and obese, and this is yet another reason why 1-d estimates of EI could be lower than expected (68, 69).

Fifth, literature accumulated over the past 20 y identifies specific sources of bias associated with response sets, such as social desirability and social approval (30, 32–37, 70). This research required the rigorous design and implementation of a variety of studies, including the use of criterion validators, to quantify potential biases that, in turn, required understanding cognitive issues in formulating self-reports. Using these data in predictive models, biologic constructs, such as serum lipid concentrations, can be predicted using self-report data (49, 71) with accuracy and precision similar to results produced using data collected in metabolic wards (72–74). Use of model systems that rely on biologic constructs, such as serum lipids, that respond predictably to produce average changes in populations provides important validation when criterion measures are unavailable, as usually is the case in free-living populations (75). Likewise, changes in body mass can be predicted with similar adjustments for error (50). With regard to error specification, errors due to social desirability are in the range of what was reported for the NHANES data (7) [e.g., 375 kcal/d across the full range of measured social desirability scores for 24HR compared with total energy expenditure from doubly-labeled water for women in the Energy Study (5)]. A difference of this magnitude from a single potential bias could explain a substantial portion of the crude differences noted. It also should be noted that these errors are not limited to the self-report of diet. For example, we have observed biases in reports of physical activity (76) that are similar to those observed in dietary self-reports (5, 30, 32, 33, 36, 37, 77) and are consistent with observations made by others (78, 79). These developments aimed at improving assessment methods are consistent with the recommendation of Webb et al. (4) in their 2013 article “Strategies to Optimize the Impact of Nutritional Surveys and Epidemiological Studies.”

Sixth, conversion of foods to nutrients is not a major source of systematic bias in group comparisons. These errors are not differential, and they are specific to the underlying food composition database. There is no evidence in the references provided by Archer et al. (7) that would indicate that errors introduced in this stage of preparing data for analyses would either exacerbate biases in self-report or influence the ability to estimate health outcomes. Indeed, numerous enhancements in food/nutrient databases occurred over the past couple of decades that improve nutrient intake estimates based on reported food intake data (80–83). The dynamic nature of the food supply and the rapid discovery of new bioactive substances related to health outcomes are other factors that influence changes in food composition databases. Understanding the evolution of these developments in the conversion of foods to nutrient intake is important, because changes may bias comparisons of intake over time if they are not taken into account (84–86). However, such changes should not bias single time-point estimation of diet in relation to health outcomes, which is nearly always the estimation of effect that is performed in nutritional epidemiology.

Seventh, to appreciate the consequence of measurement error, it is essential to understand the exact nature, and not just the crude overall magnitude, of the errors. Ultimately, the aim is to account for or control for identified errors to use data collected under “real-world” conditions to adjust estimates of health effects. Estimating risk in epidemiologic studies almost always requires comparison across categories of exposure (e.g., to obtain RR estimates). Therefore, it is essential to know how errors affect classification into these categories to know whether there is any distortion in risk estimation. Random error may attenuate observable risk, but it should not result in spurious risk estimates (87). Recent publications citing this as an issue provide no evidence regarding how errors in these self-report measures are distributed or how they relate to potential confounders and effect modifiers (7, 8).

Epidemiologic studies typically control for potential confounders and consider effect modifiers in the analysis. As we showed previously with social desirability, some of these errors are associated with psychological predispositions (e.g., acquiescent personality type), sex, and education, factors that are known to be related to many health outcomes. As we demonstrated nearly 2 decades ago in our original article on the subject of response set biases (37) and in correspondence published in its aftermath (77, 88), the modeling of effect modification and confounding is a complicated business about which relevant data must be collected to estimate their effects on predicting health outcomes. Without such information, there is no way of knowing whether misclassification occurs because of these errors or how they are related, either organically or statistically, to known or suspected effect modifiers or confounders. Therefore, the results presented in the recent literature (7, 8) are uninformative regarding their effect on risk estimation and, by logical extension, on inferences that might lead to informing public policy.

Eighth, there is a large, unsubstantiated jump from detecting potential problems with measurement error to policy implications. Virtually never do dietary data alone—or data on any exposure, for that matter—result directly in policy recommendations. It is impossible to make meaningful inferential assertions about the effect of errors without knowing whether they influence the prediction of health outcomes. Making policy recommendations requires access to results based on relevant health outcomes. For example, large-scale global investment to prevent malnutrition in young children did not begin garnering its current high global priority until after demonstration that approximately half of young-child deaths are caused by the synergistic effect of malnutrition with infection (89). Attempts to limit tobacco exposure did not occur without first understanding the effects of tobacco on health outcomes (90, 91). Furthermore, when recommendations were promulgated and laws implemented, they were not based on market or use surveys but rather on estimates of health effects derived primarily from epidemiologic studies (90, 92, 93).

On their own, the data from the NHANES will rarely provide sufficient evidence to inform inferences regarding diet–disease relations. Recognizing impediments imposed by studies conducted within populations having limited variability in dietary exposures, several of us wrote on problems with nutritional homogeneity (55, 94, 95) and proposed a variety of solutions, including international studies (61) and intra-country studies with large contrasts (59, 95). For example, the Multiethnic Cohort Study (96, 97) and the EPIC (European Prospective Investigation into Cancer and Nutrition) study (98, 99) were designed with this purpose in mind. These “natural” contrasts may be much more desirable from a methodologic perspective than trying to create them within the context of randomized controlled trials (RCTs). Once the diet or nutrient–disease relation has been firmly established from relevant research studies, then data from the NHANES can help to estimate the potential attributable risk in the population and recommend potential avenues most amenable to intervention.

The complaint made about measurement validity undermining the support of health effects due to diet (7) are reminiscent of protests by the tobacco industry and its allies that occurred over the many decades during which they challenged the nature and quality of the epidemiologic evidence linking tobacco to health (100, 101). This industry challenged the validity of epidemiologic evidence and made demands, unreasonable on both ethical and pragmatic grounds, to accept evidence only from RCTs. Based on Bradford Hill’s Criteria for Judging Causality (102, 103), which remain hallmarks for assessing whether or not putative risk factors constitute “cause,” the expert panel convened by the Surgeon General of the United States in 1964 concluded that RCTs were not needed to assert that tobacco “caused” a variety of health outcomes, including lung cancer (90).

Similar arguments have been made for nutritional research, citing errors in diet assessment as a reason for dismissing observational studies of diet and health and calling for RCTs as the only answer (8). This argument would leave us with little additional evidence on diet and health for many years and with uncertain promise of evidence on relevant questions on diet and health in the future. Advocates of this argument often cite examples of successful trials of diet, such as the recently published trial showing benefits of the Mediterranean diet in heart disease prevention (104). However, this is the exception, and there are other examples of expensive and lengthy trials that failed to provide definitive answers to the questions that provided their rationale. For many dietary issues, trials are neither feasible nor ethical and may be limited in the generalizability of their findings (105–107).

Trials are not immune to the challenges of diet measurement. They are susceptible to errors in measurement of diet in relation to implementing the intervention and monitoring compliance (108). For example, we showed that individuals in the Women’s Health Initiative (WHI) who were eligible for the diet modification arm overestimated their self-report dietary intake by ~ 169 kcal/d relative to women who were ineligible (32). Even for relatively simple interventions, it will be necessary to measure diet, and this example underlines that biased estimates of intake need to be understood and, as has been done for observational studies, estimated and controlled. Despite the enormous expense and time it required, the diet modification arm of the WHI provided only ambiguous, uncertain results for the benefits of diet, and the primary question tested (total dietary fat reduction) was considered outdated (supplanted by alterations in type of fat) by the time the results went to press (109, 110). This problem is certainly not unique to the WHI and will likely apply to other large-scale, long-term trials of dietary effects on chronic disease risks.

The reality is that conventional agent-oriented RCTs may focus only on 1 or 2 exposures at a time, potentially limiting the relevance of their findings for the effects of diet in population health. For example, assuming that distilling complex dietary patterns into a single agent (e.g., a vitamin supplement) that is characteristic of dietary pattern contrasts can be misleading. Single-agent dietary trials also may turn out to be of limited value because they inadvertently studied the wrong population or the wrong type of exposure at the wrong time in the disease process. For example, the ATBC (Alpha-Tocopherol, Beta-Carotene Cancer Prevention) study and the CARET (β-Carotene and Retinol Efficacy Trial) (111, 112) unexpectedly found evidence for a detrimental effect of β-carotene supplements on risk of lung cancer in older smoking men, thus contributing ambiguous and inconsistent evidence on the role of these agents in reducing cancer risks for the larger population. The results contradicted a belief based on hundreds of studies showing salubrious effects of whole food diets rich in antioxidant and anti-inflammatory micronutrients on cancers of various sites (113–116). The reasons for these paradoxical results are only partly understood but include design decisions made for efficiency and cost, such as studying only high-risk populations (e.g., older smokers) and exposures relatively late in terms of cancer latency. These problems were not foreseen at the time these trials were initiated.

RCTs that study diet, and therefore choices of free-living study participants, face a host of problems in attempting to create large contrasts in free-living populations, as seen in the WHI and other trials (117, 118). Changing behaviors is challenging, and these trials may require extreme commitment to make and sustain large changes. Furthermore, it is unlikely that either someone who is willing to accept randomization would have the motivation to persevere if randomly assigned to an intensive intervention or would not seek out other means for achieving change if randomly assigned to a “no-treatment” control. For example, the PPT (Polyp Prevention Trial) found no effect of a low-fat, high-fiber, high-fruit and vegetable intervention on adenoma recurrence (117, 118). However, analyses of carotenoid biomarker data and FFQ data in the PPT revealed that participants consuming diets rich in dry beans, vegetables, and fruits (as sources of carotenoids and flavonoids) were at reduced risk of adenoma or advanced adenoma recurrence, regardless of intervention arm assignment. Perhaps this result is partially explained by the larger contrast in exposure able to be obtained in the observational compared with the experimental (intervention vs. control) analyses (119–121).

For the foreseeable future, trials may answer only a few limited questions, and observational studies will remain the primary means for evaluating relations between diet and health outcomes. Such studies constitute a major portion of the evidence that underlies food and nutrition guidelines such as those of the Dietary Guidelines for Americans (10) or the diet and physical activity guidelines for cancer prevention of the World Cancer Research Fund and the American Institute for Cancer Research (122). Future observational studies should attempt to improve on methods in design, analysis, and presentation (123, 124), with better consideration of errors in measurement and potential biases, and by establishing cohorts under favorable circumstances with greater exposure variability, reduced bias, and/or better diet information.

Ninth, although there are some studies linking diet to health outcomes using data from national and other surveys such as the NHANES, these studies represent only a tiny fraction of the literature linking diet to health. It is also well recognized that diet and health studies using NHANES data are among the least informative of such analytic epidemiologic studies, based in part on the use of a single 24HR to represent individuals' food or nutrient intake, for the reasons outlined above. Despite this recognized limitation, the results from these NHANES studies are broadly consistent with those obtained from many other studies using more robust measures of individual dietary exposures.

Conclusions

Nothing is measured without error. Virtually everything we measure represents a combination of truth and error—usually both from random sources, such as from use of a single day to represent “usual” intake, and systematic biases, such as may result from social desirability. What is important, and what validity implies, is whether a method is suitable for providing useful analytical measurement for a given purpose and context (125, 126). For many purposes and in many contexts, 24HR data from surveys such as the NHANES proved to be useful in helping to address important research and policy questions, despite their known errors. Likewise, despite their well-acknowledged flaws, FFQ data produced results across a wide variety of studies and in many different populations and cultural contexts that are broadly consistent with one another and form the mainstay of what we know about diet and health.