Systematic review to support the development of nutrient reference intake values: challenges and solutions^,

Abstract

Workshops sponsored by the Institute of Medicine (IOM) and the World Health Organization suggested that incorporating systematic (evidence) reviews into the process of updating nutrient reference values would provide a comprehensive and distilled evidence document to decision makers and enhance the transparency of the decision-making process the IOM used in recommending the Dietary Reference Intake values (DRIs) for US and Canadian populations. At the request of the US and Canadian government sponsors of the ongoing review of the 1997 vitamin D and calcium DRI values, the Tufts Evidence-based Practice Center performed a systematic review for the current DRI Committee to use early in its deliberations. We described the approach used to include systematic review into the IOM process for updating nutrient reference values and highlighted major challenges encountered along with the solutions used. The challenges stemmed from the need to review and synthesize a large number of primary studies covering a broad range of outcomes. We resolved these challenges by 1) working with a technical expert panel to prioritize and select outcomes of interest, 2) developing methods to use existing systematic reviews and documenting the limitations by doing so, 3) translating results from studies not designed to address issues of interest by using a transparent process, and 4) establishing tailored quality-assessment tools to assist in decision making. The experiences described in this article can serve as a basis for future improvements in systematic reviews of nutrients and to better integrate systematic review into development of future nutrient reference values.

INTRODUCTION

Dietary Reference Intake values (DRIs) are the nutrient reference values issued by the Institute of Medicine (IOM) of the National Academy of Sciences. These values are used as standards for a variety of purposes, such as planning and assessing diets by schools, food manufacturers, and nutritionists, and for determining the criteria for nutrient labeling on packaged foods. The IOM and others have proposed that systematic reviews of the evidence should be used when revising the DRIs (1, 2). The rationale for incorporating systematic reviews into nutrient reference value development such as the DRI process is to promote a transparent and rigorous evidence review process for the expert committees and to facilitate the update of reference values as new data become available. With this framework in mind, the Office of Dietary Supplements of the National Institutes of Health, through the Agency for Healthcare Research and Quality, contracted with the Tufts Medical Center Evidence-based Practice Center to conduct a systematic review of the literature for these nutrients as related to a wide range of health outcomes specified by a technical expert panel before conducting the literature search (3). The purpose of this article is to describe the basic process used in the first example of including a systematic review in the process of updating nutrient reference values, the major issues and challenges encountered summarizing large bodies of evidence for 2 nutrients with multiple health outcomes, and the solutions used to best support the IOM panel.

EVIDENCE REPORT PROCESSES AND METHODS

The methods used to produce the evidence report generally followed those outlined in the draft Agency for Healthcare Research and Quality Methods Guide (4), with additional considerations particularly germane to nutrition (5). These include data on background nutrition status or nutrient exposure, quality appraisal items for observational studies that use dietary assessments or biomarkers to measure nutrient exposure, and sources of nutrient exposure (eg, food or supplement). Full details of this approach as applied to vitamin D and calcium have been documented previously (3).

MAJOR CHALLENGES AND SOLUTIONS

Selection of critical health outcomes for establishing nutrient reference values

In the mid-1990s, a new approach to establish nutrient reference values to include chronic disease and long-term outcomes (eg, diabetes, hear disease, cancer) to determine Adequate Intake or Upper Intake levels was put in place by the IOM (2). This expanded the range of outcomes of interest beyond nutrient essentiality, which was identified when dietary deficiency of a nutrient led to the development of a well-defined disease or growth failure (6). However, in the new approach, selection of appropriate chronic disease endpoints can be challenging because there is generally a lack of established data on the precise association between dietary intake levels, nutrient status, and chronic disease outcomes. In addition, etiologies of chronic diseases are generally not nutrient-specific but multifactorial, reflecting a combination of genetics, environmental exposure, and lifestyle patterns. The systematic review approach can highlight these challenges by making transparent where there is and is not evidence addressing populations and outcomes of interest and the quality and applicability of the evidence. Systematic review documents the criteria and decisions for selecting relevant chronic disease endpoints by conducting a comprehensive search of all available studies that report relevant data, critically appraising the quality and applicability of the identified studies, and synthesizing the findings for each key question (eg, relations between vitamin D status and cancer risks).

In contrast with most medical interventions, most nutrients have direct and indirect effects on a wide range of health outcomes and could potentially prevent or ameliorate acute and chronic diseases. The breadth of outcomes, and thus research, that needs to be assessed for a given nutrient is consequently much broader than what is typically covered by a systematic review of a medical intervention. It is therefore especially important that methodologists (those conducting the systematic review) and technical experts (and other interested parties) closely collaborate to balance the breadth of the topics to be reviewed with available time and resources. In the systematic review of vitamin D and calcium, we found that the number of potentially relevant vitamin D studies indexed in MEDLINE was large (≈15,000) and the number of calcium studies was 7-fold larger (≈110,000). Prioritizing and formulating the key questions clearly were vital. The questions must meet the intended purpose of the systematic review but should avoid covering topics that will be of limited value to the DRI Committee. Ideally, this should be an iterative process involving the sponsors, methodologists, technical experts, and targeted end-users (in the case of DRIs, the DRI panel members). In particular, careful consideration is needed to determine which outcomes and topics (and similarly, which populations, outcomes, and study designs) will not be of interest in the committee's decision-making process.

We found that selecting the relevant health outcomes was challenging in the absence of definitive data. The technical expert panel played a critical role in decision making. The focus of the technical expert panel deliberations appeared to hinge on the expertise of the particular members of the panel. Thus, it is possible that a different technical expert panel comprised of members with different expertise could have recommended a different set of health outcomes for inclusion. To minimize this variability in the future, a set of instructions to weigh each potential outcome (taking into account such factors as population-attributable risk, morbidity, severity, and resource utilization) for possible inclusion should be developed by the Tufts Medical Center Evidence-based Practice Center and technical expert panel before the selection of potential health outcomes.

An alternative method to achieve informed decision making on the outcomes of interest (which was not used for the systematic review of vitamin D and calcium) is to gather data to create an overview of the literature pertinent to the topic at hand. The overview “evidence map” allows assessment of the variety of outcomes and the amount of potentially useful literature available for each outcome. Investigators would index all studies that are potentially relevant to the nutrient of interest according to predefined characteristics (such as study design and sample size). A “map” of the available evidence is thus created that quantifies the number of potentially relevant studies for each potential outcome of interest. Decisions on which outcomes to include could then be made on the basis of both the importance of the outcome from a public health perspective and the number of potentially relevant studies available for that outcome. This “evidence mapping,” however, requires additional time and resources from both the technical experts and the methodologists. The use of a combination of an expert-defined approach and evidence mapping may be a reasonable compromise.

Problems with having to rely on existing systematic reviews

Given the dilemma of the need for a DRI panel to review a large array of populations, outcomes of interest, and types of studies and the need to complete the task within a limited timeframe and with limited resources, it would be advantageous to make use of existing systematic reviews where possible. The use of existing systematic reviews frees up resources to conduct de novo systematic reviews on additional outcomes or topics that might otherwise need to be omitted. However, relying on previous systematic reviews has several drawbacks, including the risk of propagating deficiencies and errors introduced in those reviews (7) and the inability to reanalyze data used in existing reviews. In addition, the use of existing systematic reviews could preclude covering some questions of interest to the DRI panel not adequately covered by the original systematic reviews (eg, different effects or associations in different life stages) because of differences in study eligibility criteria, and because the original reviews may not be up-to-date.

To ensure that the overall systematic review was performed on the basis of only relevant evidence, only systematic reviews that meet criteria (ie, similar population, intervention, comparator, and outcome criteria) should be included. The limitations of the included existing systematic reviews should also be clearly delineated. We implemented the methodology for using existing systematic reviews to replace de novo processes in conducting comparative effectiveness reviews (8) so that the DRI panel would better understand the sources of the evidence and potential caveats of previous systematic review findings. In the future, a solution to overcome the limitations of using existing systematic reviews would be to develop a repository of data extracted from primary studies used in systematic reviews. Such an archive would facilitate update of existing systematic reviews and minimize redundant efforts in data extraction of study results and maximize the use of the extracted data in different systematic reviews to address similar or different research questions.

Translating results from studies not designed to address issues relevant to establishing DRIs

None of the studies in the systematic review of vitamin D and calcium were designed to specifically address issues relevant to establishing nutrient reference values, based on the predefined analytic framework (3). This is in contrast to trials of medical interventions, which generally are relevant to clinical practice guidelines. In general, studies on vitamin D or calcium did not enroll subjects with ages that could be easily mapped to specific life stages as defined within the DRI framework (with the exception of postmenopausal women and pregnant or lactating women) and did not evaluate health outcomes on the basis of what levels of intake will lower risk of a particular disease. Generalization of the studies was often limited with respect to establishing nutrient reference values for the full range of the general population. There were also inadequate data to link different doses studied in trials with actual dietary intake levels. These links are essential for establishing nutrient reference values. Likewise, most observational studies based their analyses on quantile categories of nutrient intakes, and the distributions of these categories often varied across studies. This made comparisons across studies difficult because each study used different thresholds for low and high nutrient status. These challenges are likely to be encountered in future systematic reviews that support development of DRIs for any nutrient.

Although the systematic review method could not eliminate the above-mentioned deficiencies, making these deficiencies transparent clarifies what analyses would be required to best address the needs of future DRI panels. Time and resource limitations did not allow us to perform quantitative syntheses (ie, meta-analyses) of vitamin D and calcium studies. Whereas meta-analysis would not overcome the deficiencies in the evidence, it could allow for more objective assessments of the association between nutrient intake and outcomes quantitatively. Methods beyond basic meta-analysis models may be useful in synthesizing evidence from different types of studies and in considering uncertainties around the effect measures for deriving nutrient reference values. Possible methods include teleoanalysis (9), generalized or multiparameter evidence synthesis (10), and confidence profile meta-analysis (11). Developing new meta-analytic methods that help account for the uncertainties around measurement errors common in observational studies would also be useful.

Assessment of study quality and nutrient reference value decision making

In any systematic review, quality assessment of the primary studies is essential to determine the validity of the study findings. Quality appraisal is important because the strength of a body of evidence for the answers to a particular research question is primarily determined by the validity of the primary studies relevant to that research question. However, there is currently a lack of consensus on which additional study characteristics or nutrition-specific factors should be considered.

The causal chain between nutrient exposure and disease outcomes is more complex compared with that of pharmacologic treatment and disease outcomes. Thus, quality assessment of primary studies used for DRI purposes needs to consider factors unique to nutrition studies. In the systematic review of vitamin D and calcium, we not only assessed the standard quality items for validity of randomized controlled trials (eg, blinding and allocation concealment), but we also considered cross-contamination (eg, participants assigned to the placebo group crossed over to active treatment) and adherence. For observational studies, in addition to standard quality items for validity of prospective cohort studies, we considered validity of dietary assessment methodology and assay methodology of biomarkers of intake in the quality appraisal. Many of these quality items are designed to assist the assessments of the uncertainty in measuring nutrient exposures in primary studies. These issues are generally content-specific and require input from technical experts. Continuous efforts on improving quality assessments in systematic reviews tailored to nutrition application are needed.

CONCLUSIONS

DRIs allow for the informed development of nutrition policy in the United States and Canada and serve as a yardstick to evaluate population status. An unbiased review of the scientific literature to support a panel's deliberations is vital in the decision-making process and will facilitate future updates to the DRIs and other nutrient reference values as new data emerges. The first example of including an independent and comprehensive systematic review of the scientific literature into revisions of nutrient reference values showed this approach to be feasible. Although we encountered some major challenges to applying systematic review methodology to the complex literature, these difficulties were mitigated by working closely with our technical experts and focusing on the issues relevant to the DRI determination process. Many of the challenges we identified are not specific to systematic reviews in the field of nutrition, per se, but are problems intrinsic to primary studies assessing nutrient-disease associations. We believe that the solutions to these challenges presented in our article will be beneficial in making the best use and interpretation of the nutrition literature in general. We also believe our experiences could serve as a basis for future advances in methodology when integrating systematic review into the deliberations for development of DRI and other nutrient reference values.