Double-counting of effect sizes and inappropriate exclusion of studies in “The influence of vitamin D supplementation on IGF-1 levels in humans: A systematic review and meta-analysis”

Dear Editor,

We read with interest “The influence of vitamin D supplementation on IGF-1 levels in humans: A systematic review and meta-analysis”, written by Kord-Varkaneh et al. (1). While we applaud this team for their efforts, we have identified errors in the inclusion of effect sizes from several of the studies, and erroneous exclusion of studies from the meta-analysis.

Our first concern is that in studies with more than two arms, the authors included the control group more than once. Based on the Cochrane Handbook, including multiple comparisons from one study with a shared intervention group to the meta-analysis should not be done (2). This “‘double counts’ the participants in the ‘shared’ intervention group(s), and creates a unit-of-analysis error due to the unaddressed correlation between the estimated intervention effects from multiple comparisons” (2). In the current meta-analysis, this error occurred three times with studies with two intervention groups and a control group (Kamycheva et al. (3), Mortensen et al. (4), and Ameri et al. (5)). This calculation error can affect the reported meta-analysis and meta-regression estimates, which may lead to biased results and conclusions.

Our second concern is that the inadequate inclusion strategy excluded a set of studies. Our replication of their search terms revealed at least 3 relevant papers that should have been included in the current review (6–8). Upon brief review, the only criterion these papers fail to meet is “Studies which provided circulating IGF-1 serum in the form of mean differences (MD) with the 95% confidence intervals (95% CI)” (1). This criterion is not justified by Kord-Varkeneh et al. There are statistically robust methods for including papers that do not meet that criterion, outlined in the Cochrane Handbook and elsewhere, including simply reaching out to the studies’ authors for these estimates. For these three excluded papers, one included median and interquartile range (6); one included pre/post means and standard deviations and a non-significant change in IGF-1 (8); and one included pre and post means with a between-group p-value for reconstructing the between-group effect estimate (7). This latter study could therefore be included without making any additional assumptions, while the former two would require some additional assumptions. Conversely, for at least one of the studies the authors originally included (5), group results were reported as means ± standard deviations, indicating that the 95% CI inclusion criterion was not universally applied. It is possible that the meta-analytic conclusions would change if the authors had included the missed papers.

We again commend the authors for addressing this topic. However, because both the unit-of-analysis error and exclusion of relevant literature can lead to biased and unreliable results, we request that these issues be corrected. We offer our assistance to the authors on the corrections.