Errors in Meta-Analysis Should Be Corrected: Comment on “Effects of a Paleolithic Diet on Cardiovascular Disease Risk Factors: A Systematic Review and Meta-Analysis of Randomized Controlled Trials”

Dear Editor:

We appreciated reading the work of Ghaedi et al. (1), “Effects of a Paleolithic Diet on Cardiovascular Disease Risk Factors: A Systematic Review and Meta-Analysis of Randomized Controlled Trials,” which is a meta-analysis of the Paleolithic diet. When reading through the work and the articles it is based on, we had some questions which can be divided into 3 categories: discrepancies in effect sizes reported, inconsistencies and unexplained selection of effect sizes included, and CIs that we were unable to reproduce.

1. Discrepancies in effect sizes reported

   1. For the effect size for highly sensitive C-reactive protein (CRP) from Irish et al. (2) we calculate an effect of −1.63 based on the means reported in the text, whereas Ghaedi et al. (1) reported −1.69.

   2. For triglycerides (TGs) from Masharani et al. (3), we calculate an effect of −18 mg/dL from the mean changes reported, whereas Ghaedi et al. reported −0.46 mg/dL. However, we believe that Ghaedi et al. were actually reporting the TGs and total cholesterol (TC) in units of mmol/L from all of the articles, even though they were labelled as “mg/dL,” because the other source articles report “mmol/L” for the same units. When we convert Masharani et al.’s mg/dL to mmol/L to be consistent with the other articles [1 mmol/L × 88.57 = 1 mg/dL (4)], we obtain −0.20 mmol/L, which is still different from Ghaedi et al.’s −0.46.

   3. In Genoni et al. (5) the treatment effect on TG is reported as −0.14, whereas Ghaedi et al. have TG's effect size listed as −0.41 in Table 1.

   4. In the study of Mellberg et al. (6), the effect size for BMI was −1.8, which is different from the −0.8 in Ghaedi et al.’s article.

   5. In the study of Boers et al. (7), the effect size for BMI was −0.8, whereas Ghaedi et al. gave a result of −1.8.

2. Selection of effect sizes included

   1. In Stomby et al. (8), the period of the study was 2 y (720 d), and the measurements were made at baseline and after 6 and 24 mo. However, in Ghaedi et al.’s article, only the mean differences at month 6 were used, but not the results from the end of the study, with no further explanation.

   2. Also in the study of Mellberg et al. (6), Ghaedi et al. only used the mean differences at month 6, whereas in the original article, the period of the study was 2 y (720 d), and the measurements were made at baseline and after 6, 12, 18, and 24 mo. There is no explanation of why only the results from 6 mo are reported.

   3. From Stomby et al. (8), Ghaedi et al. only included body fat in the meta-analysis, whereas in the original article, the authors reported BMI, weight, waist, systolic blood pressure (SBP), diastolic blood pressure (DBP), TC, TG, HDL, and LDL.

   4. In the study of Lindeberg et al. (9), the authors only reported weight, waist circumference (WC), and body fat in the article, yet Ghaedi et al. also included SBP, DBP, TC, TG, and HDL in the meta-analysis. Outcomes for SBP, DBP, TC, TG, and HDL were not found in the original article.

3. CIs that could not be reproduced

   1. For treatment effects from Boers et al. (7), we were unable to replicate the CIs reported in Ghaedi et al. for weight, WC, BMI, SBP, DBP, CRP, TC, TG, HDL, and LDL. We used standard formulas [95% CI = Diff ± 1.96 × SE, where SE = SD/sqrt(n); SD(Diff) = sqrt(Var(post − pre)); Var(post − pre) = Var(pre) + Var(post) − (2 × Cov(pre, post)); Cov(pre, post) = Cor(pre, post) × SD(pre) × SD(post)] for the 95% CIs, with a correlation of 0.8 within groups and independence between groups. For example, for weight (kg), we calculate an effect of −1.00 (95% CI: −7.82, 5.82), whereas Ghaedi et al. reported −1.00 (95% CI: −3.15, 1.15).

   2. For treatment effects from Jönsson et al. (10), we were also unable to calculate the CIs from Ghaedi et al. Here we used a correlation of 0.8 within group and between the 2 conditions in the crossover study. For example, for weight (kg), we calculate an effect of −3.00 (95% CI: −6.53, 0.53), whereas Ghaedi et al. reported −3.00 (95% CI: −3.55, −2.45).

We ask that Ghaedi et al. address our questions and revise the meta-analyses accordingly.

Notes

XC and SLD: reviewed the original article, recalculated effects, and noted discrepancies; and all authors: discussed the errors identified and how to present them, and read and approved the final manuscript.

Supported in part by NIH grant R25HL124208 (to DBA).

The opinions expressed are those of the authors and not necessarily those of the NIH or any other organization.

Author disclosures: DBA has received personal payments or promises for same from the American Statistical Association; ASN; Biofortis; California Walnut Commission; Columbia University; Fish & Richardson, PC; Frontiers Publishing; Henry Stewart Talks; IKEA; Indiana University; Johns Hopkins University; Laura and John Arnold Foundation; Law Offices of Ronald Marron; MD Anderson Cancer Center; Medical College of Wisconsin; Nestlé; NIH; The Obesity Society; Sage Publishing; Tomasik, Kotin & Kasserman LLC; University of Alabama at Birmingham; University of Miami; and WW (formerly Weight Watchers International, LLC). Donations to a foundation have been made on his behalf by the Northarvest Bean Growers Association. DBA is an unpaid member of the International Life Sciences Institute North America Board of Trustees. The institution of DBA, XC, and SLD, Indiana University, has received funds to support their research or educational activities from the NIH; Alliance for Potato Research and Education; American Federation for Aging Research; Dairy Management Inc.; Herbalife; Laura and John Arnold Foundation; National Cattlemen's Beef Association; Oxford University Press; The Gordan and Betty Moore Foundation; the Sloan Foundation; and numerous other for-profit and nonprofit organizations to support the work of the School of Public Health and the university more broadly. DBA's prior institution, the University of Alabama at Birmingham, received gifts, contracts, and grants from other organizations including the Coca-Cola Company, Pepsi, and Dr. Pepper/Snapple.

Abbreviations used: CRP, C-reactive protein; DBP, diastolic blood pressure; SBP, systolic blood pressure; TC, total cholesterol; TG, triglyceride; WC, waist circumference.