Dear Editor,
In reading the work of Ahn et al [1], we found inconsistencies, discrepancies, and errors in forest plots that were only partially discussed in prior correspondence by Bueno et al [2] and the response by Kim & Jun [3].
First, we found that the hepatic fat reduction for the low-fat diet for the Browning et al. study in Table 2 and Figure 6 should be 28% instead of 8%. Also, Figure 6 is missing results from Bian et al for the low-fat diet after a duration of 7 months. Correcting these two mistakes makes study duration less impactful for low-fat diets compared to how it was described by Ahn et al.
For intrahepatic fat reduction (Figure 3), the Standard Errors of the Mean (SEM) for Kirk et al. were mistakenly reported as standard deviations (SD). We re-analyzed the data (random effects model with DerSimonian-Laird method in Stata) with the corrected SD and obtained a standardized mean difference (SMD, Hedges’ g) of 0.73 [95% Confidence Interval (CI): −0.08, 1.55, p=0.08] instead of 1.39 (95% CI: −0.25, 3.02, p=0.10). The magnitude of the effect estimate reduces to half the size.
For ALT (Figure 4), the sample size for the low-fat diet in the de Luis et al study should be 13 instead of 15. Correcting this leads to a slightly reduced Hedges’ g for this study of 1.86 (95% CI: 0.99, 2.74) instead of 1.91. The change in the meta-analysis result is negligible, changing from 0.50 (95% CI: −0.13, 1.14) to 0.49 (95% CI: −0.13; 1.11).
Although the sole use of SMD in meta-analyses is not incorrect, and Kim & Jun describe why they selected SMD in this case [3], some authors recommend to additionally report the mean difference (MD) on the original scale for enhanced interpretability [4]. Contrary to the SMD approach, when MD is applied here the overall effect estimate for hepatic fat reduction is significant, MD=14.13 (95% CI: 2.25, 26.02) p=0.02, favoring the low-carbohydrate diet. Using MD for ALT and AST reduction between low-carbohydrate and low-fat diets did not change the significance of results, compared to SMD.
In all forest plots, the axis labels seem to be reversed where effects greater than zero correspond with data favoring the “low carbohydrate diet”, while negative values favor the “low fat diet”, but they are labeled conversely. In Kim and Jun’s [3] response to Bueno et al [2], they agree that the axis labels “seem confusing” but did not acknowledge the reversal.
Also, the title of Figure 3 should be changed to “Forest plot of % decrement,” instead of “% change”, like some of the other plots, given that positive numbers represent reductions in hepatic fat.
Results from re-analysis are presented in Table 1. Data and code from our re-analysis are readily available at doi.org/10.5967/v2c3-3r26.
Table 1.
Results from re-analysis after corrected data extraction and including unstandardized results.
| Outcome | Re-analysis | Reported in Ahn et al. | ||||
|---|---|---|---|---|---|---|
| Mean Difference (MD) | Standardized Mean Difference (SMD) | |||||
| Mean Difference (95% CI) | p-value | Hedges’ g (95% CI) | p-value | Hedges’ ga (95% CI) | p-value | |
| IHL reduction (Figure 3) | 14.13 (2.25, 26.02) | 0.02 | 0.73 (−0.08, 1.55) | 0.08 | 1.39 (−0.25, 3.02) | 0.10 |
| ALT reduction (Figure 4) | 7.84 (−1.42, 17.10) | 0.10 | 0.49 (−0.13, 1.11) | 0.12 | 0.50 (−0.13, 1.14) | 0.12 |
| AST reduction (Figure 5) | 6.50 (−1.38, 14.37) | 0.11 | 0.61 (−0.19, 1.41) | 0.13 | 0.61 (−0.18, 1.40) | 0.13 |
Note: mean differences are outcome reduction for low carbohydrate diet minus low fat diet. Positive numbers for mean differences indicate that the low carbohydrate diet was favored.
While Ahn et al. did not specify which SMD was used, we inferred it to be Hedges’ g as the default in the reported software package, RevMan.
Our concerns, combined with those of Bueno et al., require formal correction of the analysis by Ahn et al.
Funding statement:
This work was funded in part by the National Institutes of Health (NIH): R25HL124208 and R25DK099080. The opinions expressed are those of the authors and not necessarily of the NIH or any other organization.
Conflict of interest:
In the last thirty-six months, Dr. Allison has received personal payments or promises for same from: American Society for Nutrition; Alkermes, Inc.; American Statistical Association; Big Sky Health; Biofortis; California Walnut Commission; Clark Hill PLC; Columbia University; Dynamic AQS; Fish & Richardson, P.C.; Frontiers Publishing; Gelesis; Henry Stewart Talks; IKEA; Indiana University; Arnold Ventures (formerly the Laura and John Arnold Foundation); Johns Hopkins University; Kaleido Biosciences; Law Offices of Ronald Marron; MD Anderson Cancer Center; Medical College of Wisconsin; National Institutes of Health (NIH); Medpace; National Academies of Science; Sage Publishing; The Obesity Society; Sports Research Corp.; The Elements Agency, LLC; Tomasik, Kotin & Kasserman LLC; University of Alabama at Birmingham; University of Miami; Nestle; WW (formerly Weight Watchers International, LLC); Whistle Labs, Inc. Donations to a foundation have been made on his behalf by the Northarvest Bean Growers Association. Dr. Allison was previously an unpaid member of the International Life Sciences Institute North America Board of Trustees.
The institution of the authors, Indiana University, and the Indiana University Foundation have received funds or donations to support their research or educational activities from: NIH; USDA; Soleno Therapeutics; American Egg Board; California Walnut Commission; Almond Board; Peanut Institute; Mondelez; Whistle Labs, Inc; National Cattlemen’s Beef Association; Eli Lilly and Co.; Reckitt Benckiser Group PLC; Alliance for Potato Research and Education; American Federation for Aging Research; Dairy Management Inc; Arnold Ventures LLC; the Gordon and Betty Moore Foundation; the Alfred P. Sloan Foundation; Center for Open Science; and numerous other for-profit and non-profit organizations to support the work of the School of Public Health and the university more broadly.
References:
- [1].Ahn J, Jun DW, Lee HY, & Moon JH (2019). Critical appraisal for low-carbohydrate diet in nonalcoholic fatty liver disease: review and meta-analyses. Clinical Nutrition, 38(5), 2023–2030. [DOI] [PubMed] [Google Scholar]
- [2].Bueno NB, dos Santos Silva MM, & de Melo ISV (2020). Low-carbohydrate diets and intrahepatic lipid content in individuals with non-alcoholic fatty liver disease: Evidence from a meta-analysis of randomized trials. Clinical Nutrition, 39(1), 310–311. [DOI] [PubMed] [Google Scholar]
- [3].Kim M, & Jun DW (2020). Reply-Letter to the Editor: Critical appraisal for low-carbohydrate diet in nonalcoholic fatty liver disease. Clinical Nutrition, 39(1), 306. [DOI] [PubMed] [Google Scholar]
- [4].Takeshima N, Sozu T, Tajika A, Ogawa Y, Hayasaka Y, & Furukawa TA (2014). Which is more generalizable, powerful and interpretable in meta-analyses, mean difference or standardized mean difference? BMC Medical Research Methodology, 14(1), 30. [DOI] [PMC free article] [PubMed] [Google Scholar]