Opportunities and challenges in conducting systematic reviews to support the development of nutrient reference values: vitamin A as an example^,

Abstract

Nutrient reference values have significant public health and policy implications. Given the importance of defining reliable nutrient reference values, there is a need for an explicit, objective, and transparent process to set these values. The Tufts Medical Center Evidence-based Practice Center assembled a group of nutrition experts from academic institutions and federal government agencies, led participants in discussions, conducted exercises in formulating questions and evidence review criteria that would be amenable to systematic reviews of the scientific literature, performed a literature search on the questions to identify potentially relevant publications, and identified challenges and limitations of applying this method to support the development of nutrient reference values using vitamin A as an example. The workgroup concluded that the systematic review approach could be productively used to inform the development of reference values. Challenges identified in this exercise include prioritizing and defining research questions when the volume of literature is large, relying on intermediate (surrogate) outcomes when few or no studies directly linking nutrient intake with clinical outcomes are available, and determining reliable nutrient biomarkers. Ultimately, an objective, unbiased systematic review of a defined question could be useful, not only in helping to set nutrient reference values, but also for increasing the transparency of the decision making process.

BACKGROUND

Nutrient reference values have significant public health and policy implications. These values are needed for planning diets, assessing the adequacy of diets in individuals and populations, developing nutrition education, and setting reference values for nutrition labeling (1). An example of nutrient reference values is the Institute of Medicine (IOM) Dietary Reference Intakes (DRIs) developed for healthy US and Canadian populations (2). The DRI values are established by ad hoc study committees convened by the Food and Nutrition Board of the IOM. The DRIs for each nutrient are a set of nutrient reference values that typically include an Estimated Average Requirement (EAR), a Recommended Dietary Allowance (RDA), and a Tolerable Upper Intake Level (UL). Six DRI reports have been published by the IOM, and they are organized around functionally related groups of nutrients (3–8).

However, the process of establishing nutrient reference values generally has been variable and has evolved as experience accrued from study committees. Concern has been expressed that in some cases the methods used to determine reference values by various groups have suffered from a lack of transparency and consistency (2, 9). For example, differences in the reference values derived by various groups of nutrition experts worldwide have been noted for the same nutrient. This has occurred even though all presumably have used the same body of available evidence. For example, 3 different organizations from the European Union [Scientific Committee on Food (SCF)], United Kingdom [Expert Group on Vitamins and Minerals (EVM)], and the United States (IOM) proposed different upper levels of intake for vitamin A ranging from 1500 to 3000 μg/d for adult women (9).

Given the importance of defining reliable nutrient reference values, there is a need for an explicit, objective, and transparent process to set these values. Evaluating evidence is a major component of informing the process. Over the past 15 y, the concept of evidence-based medicine, building on the foundation of systematic reviews, meta-analyses, and related methods as important tools for systematic evidence-based practice, has gained widespread acceptance in the evaluation of medical evidence for health care decision-making (10). A systematic review is an approach to comprehensively identify relevant literature meeting predefined eligibility criteria and synthesize their results according to an a priori formulated protocol, focusing on a topic or on related key questions. The application of this approach to evaluating the nutrition literature would provide a transparent, comprehensive and objective evaluation of scientific evidence and could provide support for a consistent approach to establishing nutrient reference values for all dietary components.

Whereas the basic methodologies used in systematic reviews of medical topics can be adapted to nutrition topics, there are challenges and unique problems when incorporating the process for establishing nutrient reference values. For example, nutrient reference values are needed for specific categories of people (based on age, sex, and pregnancy status) of generally healthy populations. However, to put these methodologies into operation in reviewing studies, it is necessary to make judgments about what health conditions and nutritional status are of specific interest, are acceptable, or should be excluded. These decisions can be quite complex and need to be made in light of what is known about how the nutrient affects or is affected by conditions such as cancer, infections, and malnutrition. Furthermore, in contrast with pharmaceuticals, for nutrients it is common that there is a range of substances (that may not yet be fully elucidated) that are active and that interact with each other. Examples include the different forms of vitamins A and D, or multiple minerals altering gut bioavailability of the other (eg, high dietary phosphorous decreases the bioavailability of calcium). Thus, decisions are needed regarding which forms of the nutrient are of interest when establishing nutrient reference values and what methods for measuring their intake are appropriate, taking into account such factors as the strengths and limitations of the dietary assessment methods and of the quality of the food composition database.

The Office of Dietary Supplements (ODS) of the National Institutes of Health, through the Agency for Health Care and Quality (AHRQ), requested the Tufts Medical Center Evidence-based Practice Center (Tufts-EPC) to conduct an exercise to identify the issues and challenges of including systematic review methods as a component of the process used to support the development of nutrient reference values. The Tufts-EPC assembled a group of nutrition experts from academic institutions and federal government agencies, led participants in teleconferences and meetings, conducted exercises in formulating questions that would be amenable to systematic reviews of the scientific literature, and identified challenges and limitations of applying this method to establishing nutrient reference values. By leading a workgroup through the systematic review process to formulate questions that a nutrient reference values panel might consider, the intent was to identify unanticipated issues and challenges.

In this article, we first briefly describe systematic review methods that were tailored to support the development of nutrient reference values, followed by a report of the mock workgroup in describing their experiences in the development of nutrient reference values for vitamin A and discussions about the opportunities and challenges learned from this exercise. Of note, though, the intent of this exercise was neither to make specific recommendations for reference values nor to propose how an expert panel should integrate the evidence into its decision-making process.

OVERVIEW OF SYSTEMATIC REVIEW METHODS

The systematic review process involves many steps: formulating specific key questions, developing a protocol, refining the questions of interest, conducting a literature search for evidence, selecting studies that meet eligibility criteria, critically appraising the studies, and synthesizing and interpreting the results (11). Systematic reviews optimally should be carried out by a team composed of methodologists in collaboration with domain experts. In this workgroup exercise, we focused on the feasibility of formulating the types of questions that would likely be encountered by an actual workgroup tasked with developing nutrient reference values and on identifying the potential challenges of reviewing the evidence for these questions. It is worth noting that reviewing evidence is only one aspect of setting nutrient reference values. Because we did not actually review the evidence identified in the literature searches, the methods for critical appraisal of individual studies (12–14) and for synthesizing and interpreting the results (15–17) were only presented to the workgroup briefly. As they were not implemented, these methods are not described here.

The first step to conducting a systematic review is to formulate specific key questions. Key questions are analogous to the hypotheses of primary research studies; they should be focused and explicitly defined because they describe the scope of research the systematic review will address. Key questions are commonly formulated according to the PICO method, which defines the Population, the Intervention (or exposure in the case of observational studies), the appropriate Control or Comparator, and the Outcomes of interest (18). Describing the inclusion criteria for each of the PICO elements clearly provides the investigators with the opportunity to understand the specific questions being asked, and, equally important, what questions are not being asked in the systematic review. This step ensures reproducibility and transparency. It also guides study selection, data extraction, analysis, and interpretation of results.

To guide assessment of studies that address the topic of interest, it is helpful to develop an analytic framework (also called an evidence model) that visually depicts the specific linkages associating the populations of interest, exposures, modifying factors, and outcomes of interest (19). We developed a generic analytic framework for nutrient reference values (Figure 1). The framework describes the relations between “exposure” (nutrient intake) and outcomes of interest. The components of the generic framework are described in more detail below.

FIGURE 1.

Generic analytic framework applicable to assessment of nutrients. The representation includes putative associations between an exposure (eg, a nutrient) and indicator markers of intake (eg, serum or tissue nutrient concentrations), nonvalidated intermediate outcomes, valid surrogate markers, and clinical outcomes. The solid arrows represent established associations among factors. Line thickness represents the relative directness of an association, and the strength of the relation with clinical outcomes. Dotted arrows represent associations to surrogate markers for which there is no direct evidence of an association with clinical outcomes. I. Association of exposure with clinical outcomes of interest. II. Association of exposure with surrogate outcomes for which there is “good” evidence of a linkage with clinical outcomes. IIa. Association between surrogate outcomes and clinical outcomes (“good” evidence for linkage) III. Association of exposure with surrogate markers for which the linkage with clinical outcome is uncertain. IIIa. Association between surrogate markers and clinical outcomes (uncertain linkage). IV. Association of indicator markers to clinical outcomes. IVa. Association between exposure and indicator markers. V. Association of indicator markers to surrogate outcomes (with good or possible evidence for linkage with clinical outcomes). Indicator markers (of nutrient intakes) are measures correlated with dietary intakes of a nutrient, such as biomarkers of intake, nutritional status, or markers of nutritional status. Biomarkers of intake are measurements of the nutrient itself or a metabolite of the substance in biological samples (eg, serum selenium) that have been validated to confirm that they reflect the intake of that nutrient. Nutritional status is the state of a person's health in terms of the nutrients in his or her diet and is a global term that encompasses a number of specific components from nutritional assessments. Nutritional assessment is a comprehensive approach to define nutritional status on the basis of physical examination, anthropometric measurements, and laboratory data, and medical, nutrition, and medication histories. Marker of nutritional status is a laboratory measurement or a physical sign that is thought to reflect the early stage of an abnormality (either deficiency or toxicity) of a nutrient intake. Changes induced by a nutritional intervention on the marker of nutritional status are expected to reflect the correction of the abnormality. Surrogate outcome marker is a laboratory measurement or a physical sign used as a substitute for a clinical outcome. Changes induced by a therapy on a surrogate outcome marker are expected to reflect changes in a clinical outcome. Clinical outcome is a measure of how a patient (or study subject) feels, functions, or survives or a clinical measurement of the incidence or severity of a disease (eg, diagnosis of disease).

Notably, not all components in a framework would necessarily be considered for all nutrients or all questions for a specific nutrient; therefore, individual analytic frameworks need to be developed for each nutrient of interest. The generic analytic framework is thus a template that must be tailored to specific examples. For any specific nutrient, it should be modified to reflect the underlying biological factors associated with that nutrient. Furthermore, different analytic frameworks will be needed to describe the exposures and clinical outcomes, including the intermediate factors, for the different reference values because the criteria for exposure and clinical outcomes will be different when developing reference values for adequate compared with safe intakes.

MOCK WORKGROUP EXPERIENCE: USING SYSTEMATIC REVIEW TO SUPPORT THE DEVELOPMENT OF REFERENCE VALUES FOR VITAMIN A

The workgroup held two 1-d meetings in 2007 to consider whether the systematic review process could be used to support the development of adequate and upper intake reference values for vitamin A. The workgroup was composed of 6 experts from academic institutions with expertise in vitamin A research or familiarity with the process of deriving nutrient reference values, 8 representatives from various US federal government agencies, a representative from IOM, and members of Tufts-EPC. In addition to the workgroup meetings, teleconferences were held before, between, and after the meetings to discuss the agenda and format of the meetings, initiate discussion of the issues, and to conduct a postworkshop exercise in abstract screening.

In the first meeting, the workgroup was briefed on the use of the evidence-based approach and the methodologies used in systematic reviews and reviewed key aspects of vitamin A metabolism and the process that was used by the IOM's Micronutrient DRI Panel to set the EAR and UL for vitamin A in 2000 (6). The Micronutrient Panel had used a factorial method to set the EAR for vitamin A, which included the percentage of vitamin A stores lost per day, the minimal acceptable liver vitamin A reserve, the ratio of liver weight to body weight, reference weight for a specific age or sex, the ratio of body to liver vitamin A reserves, and the efficacy of storage of ingested vitamin A (20). Of these 6 factors, the variance was known only for the percentage of body vitamin A stores lost per day. This variance was used for deriving an RDA from the EAR. The final EARs set for vitamin A were 625 μg/d for men and 500 μg/d for women.

For setting the UL, the lowest adverse effect level of 14,000 μg/d was derived based on liver toxicity and by using an arbitrary uncertainty factor of 5.0. The UL for vitamin A for adults (other than women of reproductive age) was set at 3000 μg/d. For women of reproductive age, a No Adverse Effect Level was set at 4500 μg/d on the basis of teratogenicity and by using an uncertainty factor of 1.5. The UL was established at 3000 μg/d.

The workgroup noted that 3 different expert vitamin A panels (ie, IOM, EVM, and SCF panels) had access to the same information and literature, but used different pieces of evidence to establish their reference values (6, 21, 22). It was clear that the international vitamin A expert panels had not used predefined analytic frameworks to assess the evidence, and they lacked a common set of review criteria standards. It is likely that different types of studies (in vitro, animal, and human) were evaluated by the 3 panels and that they placed different emphases on the evidence, thus leading to the different interpretations of the data for setting reference values for adequate and safe intakes. The workgroup concluded that a systematic review process could mitigate or avoid these problems because of its structured and transparent approach.

The principal charge of the workgroup at the first meeting was to derive a key question (among several that might need to be addressed) that is important to consider in the process of setting a reference value of adequacy for vitamin A as well as to specify the literature review criteria using the PICO method. At the start of the exercise of choosing a key question, the workgroup members agreed on the following: 1) serum retinol values are held fairly constant over a wide range of vitamin A nutritional status (ie, the serum concentration of retinol does not fall until liver levels go below 20 μg/g liver) (20), 2) the liver is the primary storage organ for vitamin A, 3) vitamin A is stored in the liver as retinyl esters, and 4) when the liver is overloaded with vitamin A (>300 μg/g), retinyl esters begin to spill out into the circulation (23). The workgroup agreed that the most common indicator of vitamin A status, which has been most frequently used in studies across all life stages and populations, is serum or plasma retinol concentrations. The workgroup also agreed that plasma vitamin A corresponds with liver vitamin A concentrations in a lower range of the distribution (eg, liver levels <20 μg/g), which would be relevant for use in establishing a reference value of adequacy.

The key question chosen as an example of a vital question to answer before using serum retinol as the indicator for establishing a vitamin A reference value of adequacy was, “What levels of plasma retinol are associated with increased risks of xerophthalmia, immune dysfunction or compromise, infectious morbidities (known to respond to vitamin A), mortality and poor growth (in infants and children)?” Notwithstanding concerns about homeostatic control of this particular nutrient, if a critical level of plasma retinol could be established, the relation between dietary intake and that concentration of retinol could then be examined. Furthermore, if the within-population variance of the vitamin A intake was found to be constant across populations that varied by vitamin A status and other factors, the mean of the distribution could be relocated over the cutoff for a specified biomarker or clinical outcome and used to estimate the upper end of the distribution of requirements. The analytic framework for setting an Adequate Intake for vitamin A is shown in Figure 2. The eligibility criteria that the vitamin A workgroup established are listed in Table 1.

FIGURE 2.

Analytic framework for adequacy of vitamin A intake. “Immune dysfunction/compromise” serves as a placeholder for a clinical outcome that was not defined by the workgroup. I. Association of exposure with clinical outcomes of interest. IV. Association of indicator markers to clinical outcomes. IVa. Association between exposure and indicator markers.

TABLE 1.

Inclusion and exclusion criteria for vitamin A mock Dietary Reference Intake exercise

Inclusion criteria

Population: all human studies

For growth outcome: infants and children (6 mo to 6 y) only

Exposure: blood vitamin A measurements (eg, plasma retinol), blood β-carotene, vitamin A or β-carotene intervention, presence of vitamin A deficiency, or time at which vitamin A status was measured

Comparator: different levels or concentrations in individuals or population

Outcomes: as mentioned above according to the exposure

Study designs: any design

For mother-infant pair study: the exposure and outcome can be measured in either mother or infant or both

Exclusion criteria

Nonhuman

Study designs: in vitro and ex vivo (or cell-level outcomes only)

For outcomes of immune dysfunction or compromise outcome (including cancer outcomes)—cross-sectional design

Exposure: all-trans retinoic acid therapy

Disease or conditions at baseline (in the whole study population):

Kidney diseases, liver disease, kidney or liver transplantation, HIV-AIDS, diseases or conditions requiring medications that may interfere with lipid or protein metabolism, acute infectious diseases (eg, malaria, meningococcal meningitis, and trichuriasis)

For plasma retinyl ester only—hyperlipidemia

Tufts-EPC subsequently conducted a literature search and screened abstracts according to the agreed-on eligibility criteria. The Medline search yielded a total of 3170 abstracts, of which 363 (11%) met criteria: 55 relevant to mortality, 89 to vision, 129 to immune function, and 54 to growth.

During the second meeting of the vitamin A workgroup, the abstract screening was reviewed and a discussion was held on 20 preselected abstracts, so that all members of the workgroup could understand the complexity of making inclusion or exclusion judgments in specific cases. The workgroup experienced first hand the importance of the iterative or consultative process as a part of the evidence-based review process for establishing nutrient reference values. Furthermore, in the second meeting, the analytic framework for establishing an upper level was established by the group (Figure 3).

FIGURE 3.

Analytic framework for excessive intakes of vitamin A. II. Association of exposure with surrogate outcomes for which there is “good” evidence of a linkage with clinical outcomes. IIa. Association between surrogate outcomes and clinical outcomes (“good” evidence for linkage). IV. Association of indicator markers to clinical outcomes. IVa. Association between exposure and indicator markers. V. Association of indicator markers to surrogate outcomes (with good or possible evidence for linkage with clinical outcomes).

CONCLUSIONS

The workgroup took this exercise no further, but came away with the conclusion that the systematic review approach could be productively used to help set reference values for safe and adequate intakes. Ultimately, an objective, unbiased systematic review of a defined question could be useful, not only in helping to set a reference value but also for increasing the transparency of how and why decisions are made.

The published literature on vitamin A is large, and its extent is likely similar to that of many other nutrients for which reference values need to be established. Rational and well-defined eligibility criteria must be applied when conducting a systematic review to manage the workload. Appropriate questions must be formulated so that the answers to those questions can be used to inform the derivation of a set of reference values and to help ensure transparency and reproducibility and form the foundation for future updates as new data emerge. This implies that, without unlimited resources, only a limited set of questions could be addressed by systematic reviews. An expert panel tasked with deriving reference values must prioritize the topics according to availability of resources.

In the past, the process of deriving a set of reference values for a particular nutrient has, to a certain degree, been dependent on the makeup of the expert panel. The systematic review process, in contrast, holds the promise of implementing a standardized methodology that is less susceptible to the opinions of individual panel members. Thus, it is likely that reference values will be more reproducible or at least that discrepancies between different expert groups could be made more transparent. To implement the systematic review approach, it would not only be important that members of the expert panels represent a balanced range of scientific views but also that they be familiarized with the process of conducting a systematic review and interpreting its results. Individual systematic reviews are labor-intensive activities that might require the evaluation of hundreds of articles before a handful of studies meeting the predetermined eligibility criteria are identified.

Because there are few or no studies that directly link specific nutrient intakes with clinical outcomes, intermediate (surrogate) outcomes need to be considered. Typically, one would consider only validated surrogates of the clinical outcome. These are outcomes that are strongly correlated with the clinical outcome (eg, bone mineral density is a valid surrogate for fractures), and changes in their status reflect corresponding changes in the risk of the clinical outcome (eg, changes in bone mineral density generally reflect changes in fracture risk). However, in the absence of validated surrogate outcomes, one might consider such intermediates as the next best available evidence. An example would be the absence of anemia as an intermediate marker for the absence of disease (well-being) or serum osteocalcin (a bone turnover index) as an intermediate marker for fractures. When considering nonvalidated intermediate markers, the implicit assumption is that they would have the properties of a validated surrogate outcome. Not only should this assumption be made explicit, but the uncertainties involved in applying this assumption should also be identified and discussed.

The ultimate goal is to provide dietary reference intakes for all essential nutrients; however, it may be preferable to associate clinical outcomes with an indicator (or biomarker) for the corresponding nutrient when such a reliable indicator (or biomarker) is available. A reliable indicator (or biomarker) fulfills the classic assessment model (ie, linear dose-response model) with an error that is independent of that for (self-reported) nutrient consumption estimates. The use of a reliable indicator of dietary intake (eg, serum concentrations of vitamin C) can enhance objectivity because estimates of intake are not affected by the subject's memory or ability to accurately report food intake or by the accuracy of the database used to estimate nutrient intake from food intake data. However, readily available indicators of nutrient exposure, more frequently blood and urine, are sometimes under homeostatic control and do not represent good markers of nutrient status. Moreover, like the error and variation associated with dietary intake measures, the magnitude and impact of both biological (preanalytic) and laboratory (analytic) variability need to be considered when biomarkers are used (24). Using a biomarker to evaluate the strength of downstream associations requires that the biomarker levels be back-translated into levels of nutrient intake. Thus, if an association is found between a given biomarker concentration and risk of a clinical outcome, an estimate of the nutrient intake that corresponded with the clinical outcome would likewise be necessary. This step has proven to be challenging.

It became apparent from this exercise that it would be desirable if nutrient reference values could be linked to specific health outcomes. Although this issue presents challenges, it should be considered for the future. The process for selecting specific health outcomes and intermediate outcomes (eg, biomarkers and indicators) when specific health outcomes are not available for deriving of a reference value should be defined before systematic reviews begin.

Finally, some of the issues in the existing literature that will need to be addressed directly when applying systematic reviews to the process of establishing nutrient reference values include the following: relevance of studied populations with respect to nutrient distributions and health risks to those for which reference values are being established, generalizability of well-controlled experiments with few subjects (eg, <10) or studies of subjects having narrow eligibility criteria, applicability of findings of animal studies to humans, generalizability of early studies that used methodologies that are not state of the art or directly comparable with contemporary data, and appropriate interpretation and integration of scientific evidence from observational studies. Contemporary issues, such as the role of genomics, and nutrient fortification will also need to be factored in when undertaking this process.