Potential Errors and Omissions Related to the Analysis and Conclusions Reported in Cuspidi C, et al., AJH 2014; 27(2):146–156

To the Editor: We have observed what we believe to be errors and omissions in the paper “Effects of bariatric surgery on cardiac structure and function: a systematic review and meta-analysis” by Cuspidi et al.,¹which may be of sufficient magnitude to meaningfully alter estimates or conclusions if corrected. We write to summarize the communications with the contact author and the journal staff to elicit corrections to the published paper. We first sent an e-mail to the contact author to request clarifications on 22 November 2014. When this went unanswered, we sent a letter directly to the journal editor on 8 September 2015. This was forwarded to the contact author (Dr Cuspidi), who replied on 8 November 2015. While some errors were acknowledged in Dr Cuspidi’s reply, other points remained unaddressed. We then received a request from the editor to submit this summary.

We remain concerned that (i) the methods reported in the paper have not been reported in sufficient detail and clarity to determine exactly what was done and whether the methods were valid and, consequently, (ii) if the methods are not valid, the conclusions may be affected.

The greatest issues relate to the definition and calculation of the effect sizes and associated variances. Our main concerns are as follows:

1. We noted that many effect sizes were reported by Cuspidi et al. as being exactly 0 from source papers that did not report exact statistics. The paper should describe the exact methods used to impute or derive variance estimates based on the variety of data given in the source papers or what was done when needed data were not provided (more details on this issue have been discussed below).

2. The paper states use of standardized mean differences (SMDs), but the authors do not define the statistic. There are multiple ways to calculate SMDs in pre–post designs.² These different ways are neither mathematically equivalent nor conceptually interchangeable, and the authors have not made clear which one they employed. The exact method for determining effect sizes for the types of studies included (pre–post change studies) was likely chosen from the following 2 options (where ${\overline{X}}_{\text{post}}$ and ${\overline{X}}_{\text{pre}}$ are the pre- and posttreatment means, respectively, and SD is a certain standard deviation):

   $$\frac{{\overline{X}}_{\text{post}}-{\overline{X}}_{\text{pre}}}{{\text{Pooled SD}}_{\text{pre,post}}},$$ (1)

   where one uses the pooled SD of pre- and postintervention observations or

   $$\frac{{\overline{X}}_{\text{post}}-{\overline{X}}_{\text{pre}}}{{\text{SD}}_{\text{post}-\text{pre}}},$$ (2)

   where one uses the SD of the observed pre–post change scores.

   These are 2 different statistics with 2 different meanings, the SEs of these statistics need to be calculated differently, and the 2 different statistics should not be combined in analyses as though they are the same. Based on the information in the source papers and the incomplete descriptions received from Dr Cuspidi, we wonder whether one metric may have been calculated in some cases, and another metric calculated in others, or the 2 inadvertently (and incorrectly) combined. Whichever method is chosen, it should be clearly defined in mathematical terms, justified, and consistently applied.

   Other concerns to be addressed include:

3. The paper reports an SMD value of “0” for a study³ for which the SMD is not 0. The source paper provides data to calculate the exact value, which depending on the method used (see previous point) would be either $\text{SMD}=\frac{0.03}{0.01}=3$ or $\text{SMD}=\frac{0.03}{0.03}=1$.

4. The source of the data represented in the figures and accompanying descriptions should be consistent with the stated methods. It is possible to calculate the appropriate effect from the data as reported¹ for Leichman et al. (Table 3)⁴ using the data as reported if the 0–3 months and 3–9 months data were calculated separately. Simply adding these 2 intervals together, as it appears to be what Cuspidi et al. did in Table 2 that reports “−9.3,”¹ does not allow for correct variance calculations to accompany this estimate. Without precise specification of the exact SMD, one cannot evaluate whether the variance statistic is calculated properly for.⁴ Further, there appears to be some amount of dropout during this study although the authors do not report exact numbers for the analysis of each phase. Whatever the data in the tables represent, this should be appropriately noted within the table footnotes so that the tables stand alone and are correct.

5. We also note an apparent transcription or unit conversion error in the data⁵ listed in Table 3¹ for the left atrial diameter outcome. The source paper⁵ values indicate SDs should be reported as 0.43 and 0.40, respectively, not 4.3 and 4.0. This error affects study weighting and thus the summary SMD of the group of studies in Figure 4a.¹

Our aim is to help assure the integrity of the scientific record. We suggest consideration of the following:

Because the authors have not communicated what was done in terms that would allow a scientist with statistical training to understand and attempt to reproduce the results, and the authors have not, to date, fully addressed some of our points that note apparent factual errors in the paper, we respectfully suggest that the authors make all raw (meta-analytic) data and calculations used in their paper public. This will allow the meta-analysis to be verified by statistical experts. Following this, if the conclusions remain largely unchanged, the authors should submit an erratum or revised paper, with appropriately modified sections of abstract, results, and conclusions. Otherwise, the authors should retract the paper.

We finally note that a third-party expert reviewer commissioned by the journal concurred with our concerns and recommended actions regarding the performance of a reanalysis while providing outputs to a statistical reviewer to ensure validity.

DISCLOSURE

The authors have no potential sources of conflicts of interest to disclose related to this letter.