Reporting of systematic reviews of micronutrients and health: a critical appraisal^,

Abstract

Background: The quality of nutrition-related systematic reviews (SRs) is an unstudied but important factor affecting their usefulness.

Objectives: The objectives were to evaluate the reporting quality of published SRs and to identify areas of improvement.

Design: Descriptive and exploratory analyses of the reporting quality (7 nutrition items and 28 SR reporting items) of all English-language SRs published through July 2007 linking micronutrients and health outcomes in humans were conducted. Factors that may be associated with reporting quality were also evaluated.

Results: We identified 141 eligible SRs of 21 micronutrients. Ninety SRs that included only interventional studies met a higher proportion of our reporting criteria (median: 62%; interquartile range: 51%, 72%) than did 31 SRs with only observational studies (median: 53%; interquartile range: 47%, 60%) or 20 SRs with both study designs (median: 47%; interquartile range: 39%, 52%) (P < 0.001). SRs published after consensus reporting standards (since 2003) met a higher proportion of the reporting criteria than did earlier SRs (median: 59% compared with 50%; P = 0.01); however, the reporting of nutrition variables remained unchanged (median: 38% compared with 33%; P = 0.7). The least-reported nutrition criteria were baseline nutrient exposures (28%) and effects of measurement errors from nutrition exposures (24%). Only 58 SRs (41%) used quality scales or checklists to assess the methodologic quality of the primary studies included.

Conclusions: The reporting quality of SRs has improved 3 y after publication of SR reporting standards, but the reporting of nutrition variables has not. Improved adherence to consensus methods and reporting standards should improve the utility of nutrition SRs.

INTRODUCTION

Leading nutrition organizations are using systematic reviews (SRs) to develop evidence-based nutrition and research agendas, revise dietary guidelines, formulate public health policies, and support dietetic practice guidelines with the goal of improving patient outcomes and practitioner effectiveness (1). The Office of Dietary Supplements in collaboration with other institutes and centers of the National Institutes of Health use SRs to identify research needs and set research priorities (2, 3). In 2001, the American Dietetic Association began carrying out SRs on a wide range of nutrition-related diseases (Evidence Analysis Library, http://adaevidencelibrary.com/). Evidence-based guidelines are being developed to provide an additional tool for food and nutrition professionals to apply the best research results to their practice, with the goal of improving patient outcomes and practitioner effectiveness (4, 5). In addition, the Food and Drug Administration developed a draft guidance document of an evidence-based review system to evaluate publicly available scientific evidence for health claims on food and supplement products (6). The US Preventive Services Task uses SRs to develop clinical practice recommendations on preventive and counseling interventions, including recommendations on nutrition topics (http://ahrq.gov/clinic/USpstfix.htm).

The complexity of relations between nutrition and health and the lack of widely accepted guidance on how to address nutrition issues have impeded the transfer of evidence-based methodologies from medicine to the field of nutrition. Whereas the concepts and methods of evidence-based medicine can be applied to nutrition questions, there are important differences between evaluations of drug therapies and nutrient-related health outcomes (7, 8). For SRs of medical interventions, there exist checklists to improve SR reporting quality (ie, clarity and transparent reporting of SR methods and results) such as MOOSE (Meta-analysis of Observational Studies in Epidemiology) (9) and QUOROM (Quality Of Reporting Of Meta-analyses) (10). Whereas these checklists represent consensus guidelines to improve the quality of SRs in general, they do not provide guidance for reporting or analyses of variables unique to the field of nutrition. Standardized guidance for researchers conducting SRs on nutrition-related topics could benefit the users of these reviews (11, 12).

Our aim was to examine the reporting quality of existing SRs linking micronutrients and health outcomes and to identify areas for improvement. We also performed exploratory analyses to evaluate factors that may be associated with reporting quality, such as the designs of primary studies (interventional compared with observational studies), years of publications, methods of evidence syntheses (meta-analyses or qualitative synthesis), and impact factors of journals that published these reviews.

METHODS

Literature search

We searched Medline from its inception through July 2007 using keywords for micronutrients, multivitamins, and antioxidants. We also searched for SRs, evidence-based reviews, and meta-analyses (see supplementary Table under “Supplemental data” in the online issue). Citations of SRs were reviewed for additional relevant articles. The essential micronutrients included in the analysis were fat-soluble vitamins A, D, E, and K; water-soluble B vitamins (thiamine, riboflavin, niacin, pantothenic acid, pyridoxine, biotin, folate, and B-12); vitamin C; macrominerals (calcium, chloride, magnesium, phosphorous, potassium, sodium, and sulfur); and trace minerals (chromium, copper, fluoride, iodide, iron, manganese, molybdenum, selenium, and zinc). Multivitamins or minerals and antioxidant supplements were also included. Potentially relevant reviews included those whose abstracts described searches or eligibility criteria for study identification or included terms such as “systematic,” “evidence,” “evidence-based,” “meta-analysis,” or “pooled analysis.”

Eligibility criteria

Full-text articles of screened-in abstracts were retrieved and examined to confirm their eligibility according to predetermined criteria. For the purpose of this study, we defined an SR as a study that contained 3 components: a statement of the research questions (aims or objectives), a description of the literature search, and a listing of the study eligibility criteria. A review that lacked any of these components was excluded. We did not attempt to contact authors for clarification. The following types of reviews were excluded: reviews of foods or diets that did not quantify micronutrient intake, reviews including nonoral routes of nutrient delivery, reviews that did not relate nutrients to health outcomes, reviews of nonhuman data, and pooled analyses of primary databases (ie, secondary database analyses of multiple cohorts) that did not include an SR.

Data abstraction and collection

The unit of analysis was the SR article. We did not analyze the primary studies within the SRs. The following data were collected from the full-text articles of eligible SRs: topics covered (exposures and outcomes), whether meta-analyses were performed, specific journal, publication date, and number of citations per SR. We categorized the outcomes examined as either clinical outcomes or intermediate outcomes. A clinical outcome was defined as a measurement of how a person feels, functions, or survives; the severity of an existing disease; or the incidence of a new diagnosis. Intermediate disease outcomes included laboratory measurements or physical signs used as surrogates for a clinical endpoint (eg, plasma cholesterol concentrations or blood pressure for cardiovascular disease or dark adaptation for night blindness).

A standardized form was used for data collection. From published guidelines for reporting of the meta-analyses, such as MOOSE (9) and QUOROM (10), we collected and evaluated 28 reporting items regarding the search and study selection criteria; methods for assessing methodologic quality of the included primary studies, methods for quantitative syntheses, and protocols for reporting of results. The primary goal of guidelines for SR reporting is to encourage authors to provide clear and transparent reporting of the factors relating to the literature review and evidence syntheses they carried out. Most widely recognized reporting guidelines reflect consensus opinion of groups of experts in a particular field, including research methodologists and journal editors (13). Because there is no widely accepted guidance for reporting or analyses of variables unique to the field of nutrition in SRs, we included 7 items in addition to those identified in MOOSE and QUOROM specific to nutrition or diet variables based on the concern that failure to adhere to the items could lead to biased syntheses and/or interpretation of results in nutrition-related SRs. The definitions and the reasons for selecting these 35 reporting items are described in Table 1.

TABLE 1.

Reporting items for nutrition-related systematic reviews¹

Reporting item	Definition for adequate reporting	Rationale for inclusion
With or without meta-analyses
Search terms	Keywords for identifying relevant studies for the research questions [ie, PI(E)COS], or complete search strategy (eg, keywords and medical subject headings) were described or referred to elsewhere.	In QUOROM and MOOSE
Searches in multiple databases	Search was conducted in more than one electronic database.	In QUOROM and MOOSE
Search years	Time period of the articles searched and included was described.	In QUOROM and MOOSE
Searches in multiple languages	Search was conducted in English and other languages.	In QUOROM and MOOSE
Searching for unpublished data	Authors explicitly stated the efforts to include unpublished data (eg, contact with authors, meeting abstracts or conference preceding, dissertations, or gray literature search).	In QUOROM and MOOSE
Inclusion or exclusion criteria	Definitions of at least 2 of the PI(E)COS criteria (eg, randomized controlled trials of vitamin E were included) were reported.	In QUOROM and MOOSE
Baseline nutrition status of the population	Nutrition status of the population at baseline (ie, malnutrition, normal, or mixed). Acceptable data include data from nutrition assessments, explicit interpretations or discussions of the nutrition status of the locations where the study were conducted, and inclusion/exclusion criterion for the nutrition status of the study population.	Malnutrition is associated with vitamins and/or mineral deficiencies. Under- or over-nutrition is associated with mechanisms that affect health outcomes (14). Therefore, baseline nutrition status is an important covariate in any studies concerning the associations between micronutrients and health.
Types of interventions/exposures	Nutrient interventions or exposures were described (must include dose/level and type).	In QUOROM and MOOSE
Types of comparators	Comparators were described (must include dose/level and type).	In QUOROM and MOOSE
Types of outcomes	Outcomes or endpoints were defined.	In QUOROM and MOOSE
Types of study designs	Design of the included studies was described.	In QUOROM
Number of included and excluded studies	Number of eligible and ineligible studies identified from the search was reported.	In QUOROM
Reasons for exclusion	Reasons for exclusions were described.	In QUOROM and MOOSE
Use of specific checklist for quality items	The list of quality items for the validity (or quality) assessment of studies were applied and reported for each included study.	In QUOROM and MOOSE
Overall rating of the study given	An overall rating of study quality was assessed (eg, A, B, C or good, fair, poor).	In QUOROM and MOOSE
Models for meta-analyses2	The methods of combining estimates (eg, fixed- and random-effects models) were reported.	In QUOROM and MOOSE
Assessment for heterogeneity	Heterogeneity across studies was assessed (ie, statistical methods) or discussed (ie, qualitative analyses).	In QUOROM and MOOSE
Dose-response relation of the nutrient-outcome associations/effects	Dose-response relations were examined by using dose-response statistical models, meta-regression, or subgroup analyses by different doses (ie, quantitative assessments), or examined qualitatively (ie, discussions).	In MOOSE
Assessment of publication bias	Quantitative assessment of publication bias (eg, funnel plot and Begg and Egger tests) were used.	In QUOROM and MOOSE
Discussion of publication bias	Issue of publication bias was raised in Discussion.	In MOOSE
Data sufficient to calculate the effect size2	Data needed to calculate the effect size (eg, 2 × 2 table or mean change within group) for each study were presented in the tables or figures.	In QUOROM and MOOSE
Flow diagram for the number of included and excluded studies	A flow diagram showing the progress of study selection was presented.	In QUOROM
The total number of primary studies included in the systematic review/meta-analysis	The total number of studies that met inclusion criteria was reported in the text, tables, or figures.	In QUOROM and MOOSE
Graphical presentation of the results	Graphics summarizing individual study estimates and overall estimates were presented.	In MOOSE
Strength (eg, effect size) of nutrient-outcome associations/effects	The principal measures of effect (eg, relative risk, odds ratio, risk difference, or absolute difference) were reported.	In QUOROM and MOOSE
Uncertainty of nutrient-outcome associations/effects	Indication of statistical uncertainty of findings (eg, CI), and/or description on the ranges of estimates (eg, SD) was reported.	In QUOROM and MOOSE
Analysis (qualitatively or quantitatively) for potential confounding or interactions of the nutrient-outcome association	Assessment of confounding and/or interactions (eg, comparability of study groups) was reported, or analyzing crude and adjusted effect sizes separately.	In MOOSE
Specific future research recommendations	Specific suggestions for future research agenda (ie, other than “more future research is needed”).	In QUOROM and MOOSE
Including intervention studies
Sources of the nutrient interventions	Brand names or components (or formulation) of the nutrient supplements, or foods (or recipes) in the nutrition interventions were reported.	Different forms of nutrients (eg, all-rac-α-tocopherol (chemically synthesized form), RRR-α-tocopherol (naturally occurring form), or γ-tocopherol) may have different health benefits and/or bioavailability in the body.
Doses of the nutrient interventions	The amount of nutrients (or the doses) in the interventions and intervention regimens (eg, the number of times per day) were reported.	High dose of nutrient supplementations may have harmful health effect (15). Also, the dose is necessary to understand what the intervention was.
Baseline nutrient exposures in the study population	Baseline nutrient exposures or the background diet (ie, baseline dietary intake levels or the levels of the biomarker of intakes) in the study population were reported.	Data suggest differential effects of nutrient supplementations on the prevention of chronic diseases depending on the background nutrient exposures (16–19).
Including observational studies
Methods/instruments for assessing intakes of nutrient exposures	Methods or instruments for assessing intakes of nutrient exposures [ie, dietary assessments (FFQ, 24-h recall, diet record, or diet recall) and/or biomarkers of intakes] were reported.	There are known errors associated with different methods or instruments for assessing dietary intakes. The ideal method or instrument for assessing intakes of nutrient exposures depends on the research question being asked.
Ranges or distributions of the nutrient exposures	Ranges or distributions of the nutrient exposures (ie, quartiles, mean and SD, or ranges) in the study population were reported.	Ranges or distributions of the nutrient exposures represent the ranges of doses of the nutrients in relation to the health outcomes.
Errors in assessing nutrient exposures	Measurement errors of the dietary assessments or biomarkers of intakes were reported or discussed.	Dietary intake cannot be estimated without errors. Some of these errors can be dealt with by analytic techniques (20). Some of these errors can introduce bias.
Potential impacts of the errors from assessing the nutrient exposures on the nutrient-outcome association	Potential impacts of the errors from assessing the nutrient exposures on the nutrient-outcome association were reported or discussed.	The impact of particular type of errors in measuring the nutrient exposures depends on the research question being asked and the analytic methodology used to address it (21).

¹PI(E)COS, Population, Intervention (Exposure), Comparator, Outcome, and Study design; QUOROM, Quality Of Reporting Of Meta-analyses; MOOSE, Meta-analysis of Observational Studies in Epidemiology; FFQ, food-frequency questionnaire.

²Data were collected for systematic reviews with meta-analyses only.

Additional data elements collected included the number of primary studies, instruments or methods used to assess the quality of the primary studies, and the types of primary studies (interventional or observational studies). An interventional study was defined as a study with an active intervention, such as randomized or nonrandomized controlled trials, crossover trials, quasi-interventional studies (or community trials), and before-and-after studies. Observational studies included cohort, case-control, cross-sectional and ecological studies, case series, and case reports, where the intervention was not dictated by the investigator.

For each SR, we also collected citation counts of the SRs and impact factors of the journals that published these reviews from the Science Citation Index and the Institute for Scientific Information Journal Citation Reports edition 2006. The impact factor of a journal is calculated based on a 3-y period and can be considered to be the average number of times articles published in the journal are cited up to 2 years after publication. The citation count is the number of times an article was cited by other articles published in journals indexed in Journal Citation Reports. Citation counts were collected in February 2008. The mean yearly citation number for each SR was calculated [citation count of SR/(2008-publication year of SR)].

Statistical analyses

Descriptive analyses and summary statistics were performed on the reporting characteristics of SRs, including whether the reporting followed published standards such as MOOSE (9) and QUOROM (10), reporting of nutrition variables, number and types of primary studies analyzed, whether quality assessment of primary studies were performed, and what instruments were used to assess quality or susceptibility to biases. Fisher's exact test was used to examine differences in the proportion of SRs reporting each item and to compare the SRs that included observational studies with those that included interventional studies.

We used the Mann-Whitney U test to examine differences in the proportion of reporting criteria met by SRs of different study types (interventional studies, observational studies, or both designs), to compare SRs published before with those published 3 y after QUOROM and MOOSE standards, and to compare SRs with those without meta-analyses. A correlation analysis was conducted to examine the association between journal impact factors and citation numbers and the proportion of reporting criteria met among SRs. The maximal number of reporting criteria is 29 (26 SR-reporting factors and 3 nutrition variables) for SRs of interventional studies alone, 30 (26 + 4) for SRs of observational studies alone, and 33 (26 + 7) for SRs of both designs. Two reporting items for SRs containing meta-analyses (reporting of models for meta-analyses and data needed to calculate the effect size) were excluded from these calculations. Medians and interquartile ranges (IQRs) are reported when the distributions were skewed. All P values are 2-tailed and were considered significant when P < 0.05.

RESULTS

The Medline search identified 3796 citations, of which 259 full-text articles were retrieved and examined to confirm their eligibility. Three additional articles were identified from citations in the retrieved SRs. A total of 141 SRs (105 with and 36 without meta-analyses) were eligible (15, 22–161). Of these, 90 included interventional studies alone, 31 included observational studies alone, and 20 included both types of study designs (Figure 1). Of the reviews that did not meet eligibility criteria, 9 publications stated they were an SR and/or meta-analysis or an evidence-based review but they did not meet the criteria of our predetermined definition, mostly because the authors did not state the eligibility criteria for primary studies reviewed (162–170). Among the eligible reviews, alternative names used for SRs included evidence-based review, evidence review, critical review, qualitative overview, overview, in-depth review of the evidence, and review.

FIGURE 1.

Selection process and the number of the included and excluded systematic reviews.

The earliest SR identified was published in 1989 (51). Half of the SRs were published since 2003. There has been a steady increase in the number of SRs published annually; the number of published SRs tripled from 1999 to 2006 (Figure 2). The number of primary studies included in each SR ranged from 1 to 264; 60% of the SRs included <20 primary studies. A wide variety of potential relations between micronutrients and health outcomes were examined (Table 2). Of 141 SRs, 88 (62%) evaluated clinical outcomes, 35 (25%) intermediate outcomes, and 18 (13%) both types of outcomes. Cardiovascular disease and cancers were the most common outcomes reported.

FIGURE 2.

Annual publication of systematic reviews of micronutrients and health (search ended week 2, July 2007).

TABLE 2.

Topics covered in the 141 qualifying systematic reviews linking micronutrients and health outcomes¹

	No. of systematic reviews				Health outcomes
Micronutrient (references)		Clinical outcomes	Intermediate outcomes	Both	Age-related/neurologic2	Bone3	Cancer4	CVD5	Death6	DM7	Eye8	Infection9	Pregnancy/birth10	Other11
		n (%)	n (%)	n (%)
Calcium (15, 22–51)	30	14 (46)	11 (37)	5 (17)	0	15	5	7	2	0	0	0	2	3
Vitamin A and β-carotene (15, 26, 32, 52–76)	28	23 (82)	3 (11)	2 (7>)	1	0	8	6	9	0	0	5	0	7
Vitamin E (15, 32, 52–56, 59–61, 66, 70–72, 76–89)	28	22 (79)	2 (7)	4 (14)	3	0	6	11	12	1	2	1	2	5
Vitamin C (15, 32, 52–56, 60, 66, 67, 69–72, 76–78, 85, 87, 90–93)	23	16 (70)	4 (17)	2 (9)	0	0	5	8	9	0	0	4	1	2
Folic acid (26, 54, 67, 85, 94–110)	21	14 (67)	4 (19)	3 (14)	2	0	7	7	2	0	0	0	4	2
Vitamin D (22, 29, 30, 32–34, 36, 38, 39, 41, 111–114)	14	10 (71)	1 (7)	3 (21)	0	9	4	0	0	0	0	0	0	1
Vitamin B-12 (54, 67, 85, 97, 101, 104, 105, 107, 108, 115–117)	12	5 (42)	4 (33)	3 (25)	2	0	1	6	1	1	0	0	1	2
Selenium (15, 32, 52, 53, 56, 84, 118–121)	10	10 (100)	0	0	0	0	6	1	5	0	0	0	0	1
Sodium (122–131)	10	0	8 (80)	0	0	0	0	9	1	0	0	0	0	2
Vitamin B-6 (26, 54, 85, 97, 101, 105, 108, 115, 132)	9	5 (56)	3 (33)	1 (11)	1	0	2	4	1	1	0	0	0	1
Iron (57, 67, 133–139)	9	4 (44)	3 (33)	2 (22)	1	0	0	0	0	0	0	1	2	7
Zinc (15, 56, 84, 85, 87, 140, 141)	7	5 (71)	0	2 (29)	0	0	1	1	2	0	0	0	0	3
Magnesium (86, 142–145)	5	2 (40)	3 (60)	0	0	0	0	4	1	2	0	0	0	0
Chromium (86, 146–148)	4	2 (50)	2 (50)	0	0	0	0	0	0	2	0	0	0	2
Multivitamin and/or multimineral supplements (149–153)	5	5 (100)	0	0	0	0	2	1	2	0	1	2	1	0
Potassium (124, 154, 155)	3	0	3 (100)	0	0	0	0	3	0	0	0	0	0	0
Vitamin K (156, 157)	2	1 (50)	0	1 (50)	0	1	1	0	0	0	0	0	0	1
Fluoride (38, 158)	2	2 (100)	0	0	0	1	0	0	0	0	0	0	0	1
Iodine (159, 160)	2	1 (50)	0	1 (50)	0	0	0	0	0	0	0	0	0	2
Thiamin (115)	1	1 (100)	0	0	0	0	0	0	0	1	0	0	0	0
Riboflavin (56)	1	1 (100)	0	0	0	0	1	0	1	0	0	0	0	0
Manganese (161)	1	1 (100)	0	0	1	0	0	0	0	0	0	0	0	0
Total	—	88 (62)	35 (22)	18 (13)

¹One systematic review may have more than one micronutrient and health outcome. CVD, cardiovascular disease; DM, diabetes mellitus.

²Age-related or neurologic outcomes include Alzheimer disease, Parkinson disease, tardive dyskinesia, cognitive function testing, and epilepsy.

³Bone outcomes include the prevalence or incidence of fracture, osteoporosis, and bone mineral density or content.

⁴Cancer outcomes include the prevalence, incidence, or recurrence of cancers or malignant tumors, precursors of malignant tumors (eg, cervical squamous neoplasia and colorectal adenoma), and cancer mortality.

⁵Cardiovascular disease outcomes include the prevalence or incidence of cardiovascular diseases (eg, heart diseases, vascular disease, and cerebrovascular disease), blood pressure, lipid profiles, and homocysteine concentrations, intima media thickness, arrhythmia, and cardiovascular disease mortality.

⁶Death outcomes include all-cause or total mortality, infant mortality, and fetal neural tube defects.

⁷Diabetes outcomes include the prevalence or incidence of diabetes, glycemic control, diabetic neuropathy, and glucose or insulin concentrations.

⁸Eye outcomes include cataracts, infant eye outcomes, and age-related macular disease.

⁹Infection outcomes include infectious diseases, common cold or respiratory infections, pneumococcal colonization, immune markers, and pneumonia-specific mortality.

¹⁰Pregnancy or birth outcomes include preeclampsia, preterm delivery or prematurity, infant growth retardation, low birth weight, retinopathy of prematurity, small-for-gestational age, oral cleft birth, placental abruption or infarction, congenital anomalies, and spontaneous abortion.

¹¹Other outcomes include falls, diarrhea, hemoglobin concentration, any morbidity, growth, healing of chronic wound, toxicity, twinning, strength or physical performance, body weight, depressive symptoms, symptoms of vitamin B-12 deficiency, environment-associated health disorders, premenstrual syndrome, anemia, loss of renal function, hormone concentrations (eg, renin, aldosterone, and catecholamines), hemorrhagic disease of newborns, dental fluorosis, goiter, thyroid-stimulating hormone, and endothelial dysfunction.

Reporting characteristics of the 141 SRs linking micronutrients and health outcomes are summarized in Table 3. Items that SRs commonly did not report or include were as follows: whether literature searches in multiple languages (30% of SRs), whether unpublished data were included (28%), descriptions of the nutrition status of the population at baseline (32%), use of quality scales or items to assess validity (29%), dose-response relations of the nutrient-outcome association (35%), assessments or discussions of publication bias (40%), use of a flow diagram for the number of studies included and excluded (26%), evaluations of potential confounding or interactions of the nutrient-outcome association (49%), specific future research recommendations (35%), sources of the nutrient interventions (ie, brand names, components or formulation of the nutrient supplements, or foods or recipes) (46%), baseline nutrient exposures in the study population (28%), ranges of the nutrient exposures (33%), errors from assessing nutrient exposures (ie, errors from dietary assessments or biomarker assays) (31%), and potential impacts of the errors from assessing the nutrient exposures on the findings (24%). The definitions of adequate reporting of the 35 reporting items are described in Table 1.

TABLE 3.

Reporting characteristics in systematic reviews (with or without meta-analyses) of micronutrients and health outcomes¹

				Systematic reviews of study types
Topic	Reporting item	QUOROM	MOOSE	Intervention (n = 90)	Observational (n = 31)	Both (n = 20)	Total (n = 141)
				n (%)			n (%)
Search	Search terms were described or referred to elsewhere	√	√	67 (74)	24 (77)	13 (65)	104 (74)
	Multiple databases were searched	√	√	58 (64)	16 (52)	11 (55)	85 (60)
	Years searched were described	√	√	76 (84)	27 (87)	15 (75)	118 (84)
	Multiple languages were included in search	√	√	27 (30)	10 (32)	5 (25)	42 (30)
	Authors explicitly stated searching for unpublished data	√	√	36 (40)2	1 (3)2	2 (10)	39 (28)
Selection	Inclusion or exclusion criteria were stated3	√	√	90 (100)	31 (100)	20 (100)	141 (100)
	Nutrition status of the population at baseline was reported			29 (32)	7 (23)	9 (45)	45 (32)
	Interventions or exposures were described	√	√	88 (98)	30 (97)	19 (95)	137 (97)
	Comparators were described	√	√	73 (81)	25 (83)	15 (75)	113 (81)
	Outcomes were described	√	√	87 (97)	31 (100)	20 (100)	138 (98)
	Types of studies included were reported	√		90 (100)	31 (100)	20 (100)	141 (150)
	Number of studies included and excluded were reported	√		62 (69)	19 (61)	9 (45)	90 (64)
	Reasons for exclusion were described	√	√	58 (64)4	13 (42) 4	10 (50)	81 (57)
Validity	Quality rating was used (eg, A, B, C or good, fair, poor)	√	√	31 (342	0 (0)2	6 (30)	37 (26)
	Quality items or checklists were applied and reported	√	√	35 (39)2	1 (3)2	5 (25)	41 (29)
Quantitative or qualitative synthesis	Models for meta-analyses were reported5	√	√	66 (89)	18 (86)	7 (70)	91 (87)
	Heterogeneity was assessed or discussed?	√	√	71 (79)	27 (87)	13 (65)	111 (79)
	Dose-response relation of the nutrient-outcome association/effect was examined		√	28 (31)	14 (45)	7 (35)	49 (35)
	Publication bias was assessed	√	√	32 (36)	13 (42)	3 (15)	48 (34)
	Publication bias was discussed		√	33 (37)	16 (52)	8 (40)	57 (40)
	Data needed to calculate the effect size were given5	√	√	54 (73)	16 (73)	7 (70)	77 (73)
Results	A flow diagram for the number of studies included and excluded was used	√		33 (37)4	2 (6)4	1 (5)	36 (26)
	The total number of primary studies included in the systematic review/meta-analysis was reported	√	√	89 (99)	31 (100)	20 (100)	140 (99)
	Results were presented graphically		√	61 (68)	18 (58)	8 (40)	87 (62)
	Strength (eg, effect size) of nutrient-outcome associations/effects were described	√	√	81 (90)	30 (97)	19 (95)	130 (92)
	Uncertainty of nutrient-outcome associations/effects was described	√	√	77 (86)	27 (87)	15 (75)	119 (84)
	Potential confounding or interactions of the nutrient-outcome association/effect were analyzed (qualitatively or quantitatively)		√	33 (37)4	22 (71)4	14 (70)	69 (49)
	Specific future research recommendations were made	√	√	26 (29)4	16 (52)4	7 (35)	49 (35)
Nutrition variables (interventional studies)	Sources of the nutrient interventions were described			46 (51)	NA	5 (25)	NA
	Doses of the nutrient interventions were described			84 (93)	NA	16 (80)	NA
	Baseline nutrient exposures in the study population were described			24 (27)	NA	7 (35)	NA
Nutrition variables (observational studies)	Methods/instruments for assessing intakes of nutrient exposures were reported			NA	24 (77)	10 (50)	NA
	Ranges or distributions of the nutrient exposures were described			NA	14 (45)	3 (15)	NA
	Errors from assessing nutrient exposures were described or discussed			NA	11 (35)	5 (25)	NA
	Potential impacts of the errors from assessing the nutrient exposures on the nutrient-outcome association were described or discussed			NA	9 (29)	3 (15)	NA

¹QUOROM, Quality Of Reporting Of Meta-analyses; MOOSE, Meta-analysis of Observational Studies in Epidemiology; NA, not available.

²P < 0.001, Fisher's exact test for the difference between intervention and observational studies.

³Inclusion or exclusion criteria must be stated to be included in the analyses.

⁴P < 0.05, Fisher's exact test for the difference between intervention and observational studies.

⁵Data were collected for systematic reviews with meta-analyses only (n = 104).

Factors associated with the reporting quality

On average, SRs that linked micronutrients and health outcomes met 57% (IQR: 48%, 66%) of our reporting criteria. SRs that included only interventional studies met a higher proportion of reporting criteria (median: 62%; IQR: 51%, 72%) than those with only observational studies (median: 53%; IQR: 47%, 60%) or both study designs (median: 47%; IQR: 39%, 52%) (P < 0.001) (Figure 3). There were significantly more SRs of interventional than observational studies that reported a search for unpublished studies (40% compared with 3%), described the reasons for study exclusions (64% compared with 42%), used quality scales or items to assess validity (39% compared with 3%), and included a flow diagram of the number of studies included and excluded (37% compared with 6%). There were significantly fewer SRs of interventional than observational studies that analyzed the potential confounding or interactions of the nutrient-outcome associations (37% compared with 71%) and that made specific future research recommendations (29% compared with 52%).

FIGURE 3.

Proportion of reporting criteria met among 141 systematic reviews of micronutrients and health.

We examined the association between the reporting quality and publication of the MOOSE and QUOROM reporting standards for SRs by testing the difference in reporting quality comparing those published before publication of these standards and SRs published 3 y after. One-hundred fifteen SRs were published before 1999 (n = 31) or since 2003 (n = 84); articles published between 1999 and 2002 were excluded for being conducted too close in time to the publication of the reporting standards. Before the reporting standards, SRs met a lower proportion of our reporting criteria than after the publication of the standards (median: 50% compared with 59%; P = 0.01), which suggests that the overall reporting quality of SRs linking micronutrients and health outcomes has improved since publication of the reporting standards. In contrast, the reporting of nutrition variables remained unchanged (median: 33% compared with 38%; P = 0.7) (Figure 4).

FIGURE 4.

Proportion of reporting criteria met comparing systematic reviews published before 1999 and 3 y after publication of QUOROM (Quality Of Reporting Of Meta-analyses) and MOOSE (Meta-analysis of Observational Studies in Epidemiology).

Of the 141 SRs, 128 were published in 84 journals with impact factors that ranged from 0.3 to 25.8; 13 SRs (8 with meta-analyses) were published in journals not indexed in the Journal Citation Reports; therefore, they were excluded from the relevant analyses. There was a positive correlation between the proportion of reporting criteria met and the journals' impact factors (r = 0.35, P < 0.001), which indicates that SRs published in higher impact journals were more likely to have met a high proportion of our reporting criteria. The median yearly number of citations attributable to the SRs was 4, ranging from 0 to 100 [excluding an outlier (109) that has had 2128 citations since 1995]. The proportion of reporting criteria met was not significantly correlated with the yearly number of citations (r = 0.11, P = 0.18), but both the correlation coefficient and statistical significance improved after the outlier SR was excluded (r = 0.26, P = 0.003).

SRs containing meta-analyses (n = 105) met a higher proportion of our reporting criteria compared with the 31 SRs without meta-analyses (median: 62% compared with 48%; P < 0.001). SRs containing meta-analyses were also published in journals with higher impact factors (median: 4.3 compared with 2.8; P = 0.001) and received more yearly citations (median: 16 compared with 6; P = 0.001).

Quality assessment of the primary studies

There were 58 SRs (49 of interventional studies, 1 of observational studies, and 8 of both designs) that used quality scales or checklists to assess the methodologic quality of the primary studies. The most commonly used were Jadad (171) and Schulz (172) quality scores or checklists, which were designed to assess the adequacy of the RCTs. The one SR of observational studies used a modified quality checklist, which was originally developed to evaluate the quality of interventional studies (an unpublished thesis). Of the 8 SRs of both interventional and observational studies, 8 different quality scales or checklists were used. Seven of the 8 SRs used single quality scales (eg, good, fair, or poor) for both intervention and observational studies. The definitions (or the quality items considered) of these quality scales varied. One SR used separate quality checklists for intervention (Jadad) and observational studies.

DISCUSSION

The number of SRs relating micronutrient intake to health outcomes has grown rapidly in recent years. These reviews have been published in a broad range of journals, many with relatively high citation impacts. These trends suggest an increasing acceptance of SRs as a useful way to summarize the data by the nutrition community. SRs of the literature serve as the core of evidence-based guideline development. Dietary guidance issued without prespecified and transparent evidentiary support may be more prone to errors (173) because of their greater reliance on expert opinion and the potential for omitting important data unknown to the experts. Because of the complex nature of how nutrients are handled and function in the human body, a large number of linked questions are often required for the development of nutrition guidelines. Incorporating currently existing SRs into a new SR can be a cost-effective use of resources but also has potential risks associated with doing so (174). To ensure that future nutrition-related SRs will be of maximal value, the highest standards in their conduct and reporting must be used. Good-quality SRs should minimize the likelihood of bias or misinterpretation. SRs are also helpful in identifying knowledge gaps for which specific research agenda or recommendations are needed.

Because of deficiencies in the conducting and reporting of SRs in the medical literature, expert panels convened to develop guidelines for SRs. The resulting QUOROM and MOOSE lists have been adopted by SR methodologists and medical journals as standards (13). However, several factors are important to interpreting nutrition research, and thus nutrition SRs, that are not included in the SR quality checklists designed for the medical literature. Thus, we developed a list of 35 items that included the potentially relevant items from QUOROM and MOOSE, along with new nutrition-specific items following the rationale described in Table 1.

Our analysis of a large cohort of nutrition SRs found that 14 of the 35 items commonly were not reported or considered in the SRs; of these, 6 concerned variables that are unique to the field of nutrition. Moreover, we identified deficiencies in reporting of 8 (of 28) items on the clarity or transparency of methods and results (Table 3). Whereas there is currently no consensus on nutrition quality rating issues, the reporting items used in this analysis were selected because of the likelihood that they would have generic utility across SRs conducted for different purposes. It is, however, also recognized that exceptions to generic reporting standards for nutrition SRs may be needed for specific SR applications (eg, regulatory applications). In these cases, justification for the exceptions could be noted in the design and reporting of the SR. This standardization and transparency would clarify the applicability of an SR for purposes other than those for which it was designed and enhance comparisons of results across SRs on similar topics.

Some generic quality issues are applicable to all SRs. For example, a comprehensive and transparent search strategy, with adequate justifications for inclusion or exclusion of specific studies, is needed to ensure an unbiased selection of studies for SRs and to improve understanding of how the SR was conducted. Furthermore, searching for unpublished data and comparing them with published data could shed some insights on the potential impact of publication bias (175). There is an underlying suspicion of publication bias against studies having either null or negative outcomes (176). It is important to note that there are no reliable methods to measure publication bias. Studies have shown that the most frequently used method to assess publication bias (funnel plots) can be misleading (177–179). Quality assessment of the primary studies is essential for the evaluation of validity and the overall strength of the conclusions in an SR.

The strength of SRs and meta-analyses relies not only on the validity of the included primary studies, but also on the clarity of the reporting of the SR itself. Although good reporting does not necessarily equate valid results, good reporting provides useful information for evaluating the validity of the findings. Our analyses showed that more SRs of interventional studies than those of observational studies (54% compared with 3%, respectively) used quality scales or checklists to assess the methodologic quality of the primary studies included. Without quality assessments, the validity of the included primary studies is unclear and the impact of the potential biases in the primary studies on the conclusions of an SR cannot be assessed. Furthermore, SRs of interventional studies met more quality criteria than did SRs of observational studies. This finding could be explained in part by the lack of reporting standards for observational studies. This is in contrast with RCTs, many of which have adopted the CONSORT reporting standards (180, 181). Recently, the Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) checklist (182) was developed to improve the reporting quality of observational studies. It is important to note that CONSORT and STROBE are aimed at guiding authors to report the findings of the primary studies; they were not designed as tools to assess the quality of the primary studies included in the SRs or meta-analyses. Our analyses also showed that the proportion of reporting criteria met was significantly, positively correlated with both the journals' impact factors and yearly citation numbers. This suggests that SRs of higher reporting quality are more likely to be published on higher impact journals and had wider research dissemination.

In summary, our findings suggest that the reporting quality of SRs has improved since publication of the reporting standards, but the reporting of nutrition or diet variables has not. This limits their potential value to help in formulating nutrition-related guidelines, recommendations, or research agendas. Reporting standards of SRs should be tailored for specific types of research to help the users of these SRs interpret the results. An improvement in the reporting quality of meta-analyses of RCTs in the critical care literature was documented after the publication of QUOROM (183). Our analysis documents the lack of consistent standards in conducting and reporting SRs of nutrition-related topics. It also provides useful insights on key reporting items for nutrition SRs. In addition to study design features that are important in reducing bias in all studies, for nutrition-related interventional studies it is critical to report the source and dose of the intervention, such as brand names or components (or formulation) of the nutrient supplements, or foods (or recipe) in the nutrition interventions, and the amount of nutrients (or the doses) in the interventions and intervention regimens (eg, the number of times per day). It is also important to report the baseline nutrient exposures or the background diet (ie, baseline dietary intake levels or the levels of the biomarker of intakes) in the study population, because the background diet could be one source of heterogeneity (ie, differential effects of nutrient supplementations on health outcomes) in an SR or meta-analysis. For the nutrition epidemiologic studies, it is important to report the methods or instruments for assessing intakes of nutrient exposures, ranges or distributions of the nutrient exposures, measurement errors of the diet or nutrient variables, and the potential impact of the errors from assessing the nutrient exposures on the nutrient-outcome association.

Improving the methodologic and reporting quality of nutrition SRs ought to produce more accurate, less biased summaries of the evidence and will allow users of the SRs—general readers, guideline developers, policy makers, and others—to have a better understanding of what evidence the SRs summarize and what biases may exist. Whereas there is room for revision of the quality items for nutrition SRs based on expert consensus, better adherence to the quality items analyzed here is likely to improve the usefulness and acceptance of nutrition SRs.

Note Added in Proof: When this article was in press, we found one qualified SR (184) that was not included in our original analyses. Adding this article, the number of SRs of interventional studies alone changed from 90 to 91, and the number of SRs with meta-analyses changed from 105 to 106. However, our findings and conclusions did not change.