Reply to Hua Liu, HaiCun Shi and PingLei Pan: Coordinate based meta-analyses in a medium sized literature: Considerations, limitations and road ahead

We thank Liu and colleagues for their interest and thoughtful comments on our neuroimaging meta-analysis on in somnia disorder [1]. In their commentary, the authors highlight the conceptual and technical diversity among current neuroimaging approaches, including those that were integrated in our recent analysis. We completely concur with their summary on these differences, as indeed different modalities and analysis methods reveal different neurobiological features [2,3]. Therefore, we would like to focus on what we perceive as the critical question arising from this commentary, namely whether the findings obtained by the different methods can be combined in a useful and interpretable way by meta-analyses.

From the technical perspective, the results from any imaging modality that provides stereotaxic coordinates in a standard reference space for the peak locations of those clusters that became significant in a voxel-wise whole-brain analysis are readily includable in a coordinate-based meta-analysis (CBMA) [4]. This flexibility is a key advantage of CBMA and can be attributed to the fundamental principle of assessing spatial convergence in the reported locations. The admittedly sparse representation of published findings by spatial locations of the reported peaks not only avoids any influence of modality-immanent differences in signal characteristics, e.g., scaling, heteroscedastic, and smoothness. Rather, it also negates any influence of statistical choices such as thresholding procedure and significance levels. Put it simply, CBMA addresses the question, where in the brain previously reported effects for a particular topic show a higher spatial convergence than expected by a random spatial association — independently of the modality, methods and analytical choices of the original papers [5].

Such approach focusing on the spatial aspect not only acknowledges the fact that the primary objective of most neuroimaging studies is spatial inference, but also raises an important conceptual distinction between CBMA and classical effect-size meta-analyses in behavioral and clinical sciences [6]. In contrast to the latter, CBMA should not be seen as the search for an absolute ground truth as it cannot establish the presence or absence of an effect at any given location due to the sparse representation and absence of effect-size measures. Rather, from the very outset, neuroimaging meta-analyses have been developed and enjoy great success as a method to consolidate a rich but heterogeneous literature by distilling spatial convergence. As rightfully pointed out by Liu and colleagues, there is an immense experimental and analytical flexibility of neuroimaging studies. In addition, worries about p-hacking, selective reporting and publication bias towards positive findings are also well documented. This leads to a situation, where the literature may consist of many spurious, maybe even false positive effects, reflecting peculiarities of each study. CBMA then serve the critical role of consolidating the reported locations of significant effects into robust evidence by the analysis of spatial convergence [5].

Thus, indeed, CBMA has provided useful insights into the brain regions involved in various neuropsychiatric disorders [7–11]. Our finding that, given the current literature, the same method has not pinpointed similar consistent findings for insomnia disorder may have several causes as discussed in the paper. In particular, the lack of findings could be attributable to, e.g., an insufficient power to find very small or noisy effects among a plethora of experimental and analytical choices resulting in spatial noise; to associations of insomnia with more subtle or distributed brain structural and functional deviations; or the presence of distinct subtypes among insomnia patients [12].

Whereas the suggested alternative approach of a qualitative review has a value if done with the aim of presenting a falsifiable model or hypothesis, such an approach has its own limitations. Even the most careful and systematic approach will be susceptible to subjective biases. Moreover, CBMA has the distinct advantage of providing an objective spatial inference, something which is hard to achieve in a qualitative review. Nevertheless, reviewing as well as CBMA is only a starting point into understanding the etiology of brain disorders. Either provides, with different aims, approaches and limitations, a consolidation of the current knowledge, but not in itself a model of pathophysiology. They may then, however, provide critical information and constraints on falsifiable models or hypothesis on why insomnia would be associated with a particular location as well as why their location differs across subjects and studies.

Liu and colleagues make an important suggestion in accordance to the best-practice guidelines published earlier by Muller and colleagues to include reasonable amount of homogeneous experiments [5]. Put differently, they suggest to first aggregate findings within each modality and at a next level integrate the information across modalities. This would indeed represent an optimal approach, but obviously requires breaking down the available literature into smaller, more homogeneous groups of experiments. Such approach thus requires dealing with a trade-off that is inherent to any meta-analytic procedure and also well-known in CBMA. On one hand, studies included into a meta-analysis should evidently be as homogeneous as possible. On the other hand, a larger number of included studies not only increases power to detect smaller effects but also increase robustness and provide superior evidence for the generalization across experimental and analytical procedures. Importantly, this weighting between inclusiveness and focus is fundamentally unre-solvable as there is no generally optimal balance between both opposing attractors. Rather, there are guidelines establishing lower bounds for either end. For inclusiveness, the minimal number of studies needed in ALE analyses to avoid spurious effects driven by a single study is an important and clearly established lower bound [13]. It also represents the reason why the analysis strategy suggested by Liu and colleagues to first conduct separate CBMA of studies with the same imaging modality was not viable in our study [1], given that there were not enough studies per modality to perform a valid neuroimaging meta-analysis over the single modalities [13].

But can we formulate a similar lower bound on focus, i.e., how diverse should we allow studies to be? Here the answer is at the same time simpler and more complex than one may intuitively think, as this floor indeed exists but is specific to each meta-analysis as it reflects the scientific question. If one wants to investigate, where in the brain task activations are most likely to occur, one would include the entire task-based literature to the extent that is technically possible [14]. Likewise, combination of structural and functional studies [15–18], various tasks [19–21], or various resting-state methods [11,22–24] has been also well-documented previously. Constructing a toy example for the opposite, i.e., very tight approach, if the question is to find the representation of reflexive pronouns in sentences containing German action verbs, a study that uses the exact same task and material in English may already be considered too divergent. That is, the homogeneity required for a CBMA is solely dependent on the scientific question of interest. In our recent meta-analysis on insomnia disorder [1], we tried to address the question, whether there are spatially consistent abnormalities in the brains of patients relative to healthy control subjects and hence pooled the entire available literature.

As a final thought, does this exclude the emergence of convergent abnormalities in any individual modality, any particular type of analyses? Evidently it does not. Rather, it is well conceivable, that once more primary neuroimaging studies in patients with insomnia are conducted and published, subsequent meta-analyses following the same stringent methodological approach as the current but addressing a more focused question may reveal convergence within that particular part of the literature on insomnia disorder. Our meta-analysis, however, highlights the lack of spatial convergence within the hitherto existing limited neuroimaging literature. Taken together, we agree with Liu and colleagues that when more upcoming well-designed imaging studies in insomnia disorder are available, this CBMA could be revisited in the future and might then reveal consistent regional alterations. Another promising avenue would be to use a more rich representation of the original data, e.g., by moving towards image-based meta-analysis as pioneered by ENIGMA consortium approaches. These may provide an alternative view into structural and functional regional disturbance in insomnia disorder in the future.