Comment on: ‘Vitamin D supplements and cancer incidence and mortality: a meta-analysis'

Sir,

A letter by Jiang et al (2015) raised important issues for conducting meta-analyses. Although Jiang et al acknowledge that these issues are unlikely to alter our findings, these points are worth discussing as they do illustrate gaps between meta-analysis in principle and meta-analysis in practice.

It is important to clarify that the choice of fixed vs random effects model should not be guided by I² value, but rather by a priori scientific belief about the nature of the exposure–disease relationship under study. Because a fixed-effect model requires a strong assumption that variation in effect estimates across studies is solely due to the play of chance, the random-effects model based on the DerSimonian–Laird estimator is generally considered as the standard weighting scheme in the current practice of meta-analysis. Conceivably, the effect of vitamin D supplements on cancer endpoints may vary depending on various factors including study population, dose and duration of intervention, baseline vitamin D status, etc., which varied across the studies included, so the choice of the random effects model is justified. In fact, Jian et al stated the importance of accounting for methodological qualities of RCTs and of providing information on gender ratio of population (which was provided in Table 1 as % males (Keum and Giovannucci, 2014)), all of which are potential contributors to heterogeneity across the trials. Furthermore, as I² value was 0%, both the DerSimonian–Laird random effects model and the inverse-variance fixed effect model lead to the same summary estimates. Even if a fixed-effect model should be used, our data are not rare enough for Mantel–Haenszel fixed effect to be preferred over the inverse-variance fixed-effect model among other options.

When publishing a meta-analysis under limits on word count and on the number of tables and figures, priority should be given to highlight key findings rather than to state the obvious. Most of the points Jiang et al argue we ‘should' have stated relate to this priority issue. First, restriction to English only articles and small number of trials included are self-evident limitations of our meta-analysis. Limited word count could be better used to discuss more fundamental limitations (e.g., our findings based on trials on short-term duration of vitamin D supplementation are silent about a potential long-term benefit vitamin D supplementation). Second, as meta-analysis is only as valid as studies included, it is undeniably important to appraise study quality. Yet, given only four trials available, which precludes a meaningful subgroup analysis by score, calculating an arbitrary score based on ‘randomisation, allocation concealment, blinding of participants and personnel, blinding of outcome assessment, withdrawals and dropouts' does little to enhance the credibility of the findings. Furthermore, while the criteria suggested by Jian et al are important methodological aspects to consider, more fundamental factors that affect the validity of findings concern underlying vitamin D status of the population, increment of 25(OH)D or attained vitamin D status due to intervention, and duration of treatment. These points cannot be meaningfully graded because, for instance, we do not know the appropriate level of 25(OH)D. Thus, we opted for qualitative appraisal of study quality through discussion section.

Third, heterogeneity is an important issue in meta-analyses. Thus, statements such as ‘I²-values of 25%, 50% and 75% are used to classify low, moderate and high heterogeneity, respectively' or ‘Test of heterogeneity is low-powered, and thus, a P-value of less than 0.1 rather than 0.05 was used to determine statistical significance of heterogeneity' are helpful information for potential readers. Yet, given our results that I²-value is 0 and P-value for heterogeneity is far from 0.1, those cut-off points are rather irrelevant here. Likewise, publication bias deserves a critical evaluation in meta-analysis. While funnel plot is a possibility, it is limited to subjective evaluation of publication bias. Results from Eggers' test and Begg's test we provided are statistical analogues of funnel plot and allow for an objective evaluation of publication bias albeit low powered when the number of included studies is small.

Meta-analysis is a process of putting together scattered pieces of puzzles available to infer about the complete picture. While suggested standard guidelines for conducting meta-analysis should not be ignored, it is at researchers' discretion to identify and highlight important aspects of the findings from the incomplete status of knowledge. Thus, rather than mechanically following so-called the standard protocols, researchers may be able to conduct a more meaningful meta-analysis by factoring in the underlying biology behind the research question at hand.