Impact of quality of research on patient outcomes in the Institute of Medicine 2013 report on dietary sodium

Abstract

The 2013 Institute of Medicine report entitled “Sodium Intake in Populations: Assessment of Evidence” found inconsistent evidence of health benefit with dietary sodium intake <2300 mg/d. Different studies reported benefit and harm of population dietary intake <2300 mg/d. The Institute of Medicine committee, however, did not assess whether the methodology used in each of the studies was appropriate to examine dietary sodium and health outcomes. This review investigates the association of methodological rigor and outcomes of studies in the Institute of Medicine report. For the 13 studies that met all methodological criteria, nine found a detrimental impact of high sodium consumption on health, one found a health benefit, and in three the effect was unclear (P = .068). For the 22 studies that failed to meet all criteria, 11 showed a detrimental impact, four a health benefit, and seven had unclear effects from increasing dietary sodium (P = .42).

1. INTRODUCTION

High dietary sodium is indicated to be a leading risk for death and disability globally according to the World Health Organization (WHO) and Global Burden of Disease Study.1 Reviews of the dietary sodium evidence by governmental and nongovernmental scientific bodies have found evidence to support recommendations to reduce dietary sodium to <2400 mg/d and most recommend <2000 mg/d.1 However, in 2013, the Institute of Medicine (IOM) released a report titled “Sodium Intake in Populations: Assessment of Evidence,” which examined the relationship between sodium intake and various health outcomes based on evidence between 2004 and 2013 focusing on intake of sodium <2300 mg/d.2 The report was unable to identify a relationship between sodium intake and health outcomes at a dietary sodium intake <2300 mg/d and continues to be heavily cited by a small number of scientists who disagree with lowering dietary sodium.

The IOM report found that much of the research was of low quality but did not identify a relationship between study quality and outcomes.2 Further, the IOM committee assessed study quality using traditional risk of bias assessment and did not systematically assess the quality of methodology critical to studies on dietary sodium, such as used in the WHO review of evidence and by some other international health and scientific organizations.2, 3, 4

Many prominent international health and scientific organizations have expressed concern that low‐quality research methods on dietary sodium are generating controversy about reducing dietary sodium.1, 5, 6 Since 2013, ongoing systematic reviews of clinical and epidemiological evidence on dietary sodium found that when quality indicators are applied to dietary sodium studies, most studies report health harms of high dietary sodium and few report health benefits.3, 4 The objective of this study was to apply quality criteria to the studies included in the IOM report to identify whether studies with methodological flaws had a different distribution of outcomes compared with those that met all quality criteria. We hypothesized that more studies that met all the quality criteria would report health harms from higher sodium intakes. Secondarily, we hypothesized that studies that failed to meet all of the quality criteria have a mixture of health benefits and harms from high sodium intake.

2. METHODS

All studies in the IOM report were identified and included in this analysis.2 The studies were assessed using quality assessment criteria developed for the bimonthly review of the Science of Sodium (SOS), which, in turn, were adapted from the systematic review used to develop the WHO sodium guidelines.1, 4 Different quality criteria were given for studies assessing the relationship between dietary sodium and (1) blood pressure (BP) and hypertension as outcomes, and (2) “hard” health outcomes (eg, fatal and nonfatal major health events such as myocardial infarction or stroke). Quality criteria for BP/hypertension outcomes were as follows: (1) the study design is a randomized control trial with at least two groups at differing levels of sodium intake; (2) the study had a duration of ≥4 weeks; (3) the study had a difference of sodium intake of at least 40 mmol/d between the intervention and control groups; (4) the study measured sodium intake with 24‐hour urinary sodium excretion; and (5) the study did not have any concomitant interventions in the intervention group that were not also applied to the control groups. Quality criteria for hard health outcome studies were as follows: (1) the study design was either a randomized control trial or a prospective cohort trial; (2) cohort studies excluded acutely ill patients, or those with type 1 diabetes mellitus or heart failure, to reduce the risk of reverse causality (where sicker people could be expected to both have more events and to eat less food, and hence sodium); (3) the study had a duration of ≥1 year; (4) the study measured sodium intake for a minimum of 24 hours using 24‐hour urine samples, dietary records, and or dietary surveys; and (5) the study did not have any concomitant interventions in the intervention group that were not also applied to the control groups. In addition, all studies, regardless of health outcome of interest, had additional quality criteria applied to them that was not adapted from the SOS review and (6) none of the authors had any conflicts of interest pertaining to the commercial interests of the salt industry. The total number of quality criteria met by each study was totaled.

The health outcomes and association with sodium intake were also classified in each study. Studies were classified as demonstrating a health benefit, health detriment, or unclear or conflicting effects on health outcomes as a result of increased sodium intake. For studies that reported multiple outcomes, the outcome that was most broadly relevant from the hierarchy below was taken as the primary outcome of the study.4 The hierarchy was: (1) total mortality and disease‐specific mortality; (2) disease morbidity, such as stroke, infection, or cardiovascular disease; (3) changes in symptoms, quality of life, or functional status; (4) clinical surrogate outcomes (BP or hypertension); (5) other clinical surrogate outcomes, such as obesity, diagnosis of chronic conditions, and bone mineral density; and (6) physiologic/biomarker surrogate outcomes, such as heart rate, plasma or urinary norepinephrine, or carotid artery thickness. Studies were then categorized as either meeting all quality criteria or not meeting all quality criteria. The determined outcome of each study was verified by comparing them with those reported in the IOM report.

Fisher exact test was performed to identify differences in the number of health outcomes (health benefit, health detriment, or unclear) in studies that met all criteria, and the number of health outcomes in studies that did not meet all criteria. This was done both in the studies with BP/hypertension outcomes and in the studies with hard outcomes. In a post hoc analysis, to add additional statistical power, studies identified as meeting all criteria (except criteria relating to conflicts of interest) by the SOS annual reviews3, 4 were totaled according to their outcomes and added to the studies identified as meeting all criteria (except criteria relating to conflicts of interest). The conflict of interest criteria was excluded as the SOS reviews did not account for it in their own inclusion criteria. Fisher exact test was performed to identify differences in the number of studies from the IOM and SOS reviews that met all quality criteria (excluding criteria 6) across the different health outcomes. All statistics were performed using IBM SPSS Statistics version 20.

3. RESULTS

A total of 38 studies were identified from the IOM report. Three studies on heart failure were excluded because they were part of a series of five studies from a single center and some data from the different trials were found to be identical in a meta‐analysis.7 The investigator was not able to provide verification to support the duplicate data, indicating that the trial data were lost as a result of a computer failure. The meta‐analysis has since been withdrawn. A final pool of 35 studies remained for the analysis.

The Table details the studies and the number of quality criteria they met. Overall, 13 of 35 studies met all of the quality criteria. Nine of the 13 studies that met all of the quality criteria showed a health detriment from increasing dietary sodium, three studies found an unclear impact of increasing sodium, and one study demonstrated a health benefit. The distribution of studies with health benefits, unclear effects, or health detriments that met all criteria for the IOM studies was not significantly different than predicted by chance (Figure A, P = .068). For studies examining hard patient outcomes, 10 met all quality criteria. Of those, increasing dietary sodium resulted in a health detriment in six studies,8, 9, 10, 11, 12, 13 had unclear effects in three studies,14, 15, 16 and had a health benefit in one study.17 For the three BP outcome studies that met all criteria, all showed a health detriment from high dietary sodium.18, 19, 20

Table 1.

Number of quality criteria met by outcome type

No. of criteria met	BP outcome			No. of criteria met	Non‐BP outcomes
	Health benefit	Health detriment	Unclear effects		Health benefit	Health detriment	Unclear effects
5	0	3	0	5	3	6	3
4	0	1	2	4	1	2	3
3	0	2	0	3	1	2	0
2	0	1	1	2	0	0	0
1	0	3	1	1	0	0	0

No. of criteria met + conflict of interest criteria	BP outcomes			No. of criteria met + conflict of interest criteria	Non‐BP outcomes
	Health benefit	Health detriment	Unclear effects		Health benefit	Health detriment	Unclear effects
6	0	3	0	6	1	6	3
5	0	1	2	5	2	2	2
4	0	2	0	4	2	1	1
3	0	1	1	3	0	1	0
2	0	3	1	2	0	0	0
1	0	0	0	1	0	0	0

Abbreviation: BP, blood pressure. Tables are displayed both before and after the conflict of interest quality criteria were added.

Figure 1.

(A) The observed distribution of Institute of Medicine (IOM) studies that met all quality criteria across health outcomes as a result of increased sodium intake vs the expected distribution across health outcomes (P = .068). (B) The observed distribution of IOM studies that did not meet all quality criteria across health outcomes as a result of increased sodium intake vs the expected distribution across health outcomes (P = .42). (C) The observed distribution of IOM and Science of Sodium studies that met all quality criteria (excluding criteria on conflicts of interest) across health outcomes as a result of increased sodium intake vs the expected distribution across health outcomes (P = .035)

Overall, 22 studies did not meet all of the methodological criteria. Of those, 11 showed a health detriment,^21–31 four found a health benefit,32, 33, 34, 35 and seven had unclear effects of increasing dietary sodium.36, 37, 38, 39, 40, 41, 42 There were no significant differences in the distribution of health outcomes for studies that did not meet all criteria compared with what would have been expected by chance (Figure B, P = .42).

Eleven studies that examined hard outcomes did not meet all quality criteria: four were associated with a health benefit,32, 33, 34, 35 four were associated with a health detriment,28, 29, 30, 31 and three had unclear effects with increasing dietary sodium.36, 37, 38 Two studies that were associated with a health benefit from increasing dietary sodium met all except conflict of interest quality criteria.34, 35 For BP studies that did not met all criteria, seven studies found a health detriment21, 22, 23, 24, 25, 26, 27 and four studies had an unclear impact39, 40, 41, 42 on BP outcomes with increasing dietary sodium.

4. DISCUSSION

One study in the IOM report that met all of the quality criteria showed a health benefit from increasing dietary sodium and in two studies the impact on the major outcome was unclear, but the other nine studies supported the position that there is harm from increasing dietary sodium. This is consistent with results of ongoing systematic reviews of the impact of dietary sodium on health outcomes after 2013. In 20 studies reviewed by the SOS since 2013 that had either BP outcomes or hard clinical outcomes and that met quality criteria (excluding the conflict of interest criteria), 13 showed harms from increasing dietary sodium, three showed health benefits, and three showed unclear effects on health outcomes.3, 4 In a post hoc analysis, we added the SOS trials that met the quality criteria to those in the IOM report that met the criteria. Results from chi‐square test indicated that the distribution of studies with health benefits, unclear effects, or health detriments that met all criteria for the IOM and SOS studies combined was significantly different than predicted by chance (Figure C, P = .035), with the majority showing health detriments. In contrast, when no quality criteria are applied, a systematic review found that approximately a third of studies suggest health benefits of increasing dietary sodium, while just over 50% show harms.43

In four of the five studies in the IOM report that did not meet the quality criteria and showed a health benefit, there was an author with a potential conflict of interest33, 34, 35 or the study examined populations with chronic illness.32, 33 In the two studies with participants who had chronic illness, it would be expected that the sicker persons would eat less (including sodium) and have more events relating to their more advanced disease (reverse causality).44 One of the studies that had a senior author with a potential conflict of interest showed health benefits of increasing dietary sodium, but those results were not reproduced.35 When the same database (NHANES III [Third National Health and Nutrition Examination Survey]) was later analyzed by independent investigators, harm not benefit was associated with higher dietary sodium.35 The same senior investigator with the potential conflict of interest has coauthored three other studies showing health benefits or unclear effects from increasing dietary sodium.34, 45, 46 Two of these additional studies have also had results that were not reproducible on later reanalysis,47, 48 while the remaining study has not been independently reanalyzed.34 The association between conflict of interest and outcomes favoring the food industry in nutritional research is substantive and there is increasing evidence of interference in science and public health policy by food industries.49, 50 In a meta‐analysis of systematic reviews on the impact of sugar on obesity, over 80% of studies in which there were financial interests concluded that there was a lack of evidence to support an association, while over 80% of the studies without a financial interest concluded that there was an association.50 Financial interests may bias selection and adjustment of confounding factors in cohort and cross‐sectional studies, altering associations between sodium intake and outcomes. Although a conflict of interest does not necessarily make the results of a study invalid, it represents a bias that, similar to other methodological biases, makes the results less likely to be valid and reproducible. Hence, we elected to include conflicts of interest with the other methodological criteria in this review. However, we have also provided the results of the IOM review without excluding studies that had authors with conflicts of interest as well.

A single major flaw in a study can invalidate a study's findings. We noted that nearly all studies that reported a health benefit from increasing dietary sodium had at least one methodological flaw. The studies that meet all standards for quality should be focused on when identifying implications and making conclusions. If only the highest‐quality studies were included in the IOM analysis then primarily health detriment outcomes would have been reported from high dietary sodium, with few studies reporting health benefits or unclear effects. Since several of the studies that contributed to the conclusions of the IOM analysis were flawed showing health benefits, as well as flawed studies showing health detriments and unclear effects of increasing dietary sodium, the validity of the conclusions of the analysis can be called into question. Analysis of the impact of dietary sodium on health outcomes should exclude studies with a flawed research design.

5. CONCLUSIONS

With the controversy resulting from the 2013 IOM report on sodium intake, effective quality criteria should be applied in conducting, funding, and publishing research on dietary sodium. Low‐quality research is a threat to scientific integrity and public health. Such criteria should likely include assessing conflicts of interest as a bias with the potential to skew results in favor of health benefits from increasing sodium consumption.

CONFLICTS OF INTEREST

AL and CD have no conflicts of interest to declare. NRCC is a paid consultant to the Novartis Foundation to support their program to improve hypertension control in low‐ to middle‐income countries, which includes travel support for site visits and a contract to develop a survey. NRCC has also agreed to provide paid consultative advice on accurate blood pressure assessment to Midway Corporation and is an unpaid member of World Action on Salt and Health.

Lucko A, Doktorchik CT, Campbell NR. Impact of quality of research on patient outcomes in the Institute of Medicine 2013 report on dietary sodium. J Clin Hypertens. 2018;20:345–350. 10.1111/jch.13168