Epidemiological challenges in systematic reviews

We agree with Brugha and colleagues that the field of psychiatric epidemiology poses particular challenges to systematic reviewers.¹ Heterogeneity between studies may arise from differences in outcomes and it is certainly true that psychiatry still lacks ‘biologically based gold standards’ regarding their definition. However, we disagree that these are necessarily linked. For the purpose of systematic reviews and meta-analyses, the issue is not to what extent an outcome is definable, with biological tests or otherwise; rather, how comparable individual studies are in their measurement of whatever outcome they use. For example, studies of schizophrenia defined by standard diagnostic tools such as the ICD-10, and applied using common operationalised criteria, should be looking at the same construct to a large extent. Definitions of physical health conditions also vary, even when specific tests are available for diagnosis. For example, definition of hypertension is not the same across national guidelines used in the USA and Europe.^2,3 We acknowledge that differences exist in psychiatry between diagnostic tools which attempt to define the same or similar conditions, such as schizophrenia in ICD-10 v. DSM-IV. Often studies include outcomes such as psychotic, depressive or other symptoms instead of a diagnostic category, which can make comparison harder. Therefore, we recommend systematic reviews pay close attention to how outcome is defined in individual studies so that they are comparable. This should be considered as part of mandatory reporting of individual study quality in systematic reviews, as we have recently done,⁴ and as Brugha et al rightly encourage. Biologically based outcomes may help in due course but, currently, attention needs to be focused on the principle of comparability of outcomes we have now.

Another important contributor to heterogeneity is variation in exposure measurement which we think needs to be emphasised. In our systematic review and meta-analysis of premorbid IQ in schizophrenia, we found that the effect size varied as a result of differences in IQ testing methods and age at testing.⁵ Therefore, as well as ensuring that measurement of exposure is similar across included studies, differences should be explored further by subgroup and sensitivity analysis.

With regard to meta-analysis, combining methodologically incomparable studies will have serious implications for the validity and generalisability of findings. For example, a pooled odds ratio of 1.34 was reported for schizophrenia for exposure to herpes simplex virus type 2 (HSV-2) in a recent meta-analysis.⁶ Unfortunately, this tells us very little because the reviewers conflated studies which considered HSV-2 infection in early life and subsequent schizophrenia (i.e. prospective designs) with those which considered the prevalence of infection in people with established schizophrenia (i.e. a cross-sectional design). Such differences may not be picked up by tests for heterogeneity. The responsibility for establishing that individual studies are sufficiently comparable in design and other aspects in order to justify combining their results in a meta-analysis lies with researchers conducting systematic reviews, as well as with the reader.

It was not clear from the meta-review how many original reviews followed some kind of guidelines. Guidelines for reporting of systematic reviews, including those of observational studies, already exist, such as Preferred reporting Items for Systematic reviews and Meta-Analyses (PRISMA) and Meta-analyses Of Observational Studies in Epidemiology (MOOSE). They include comprehensive checklists for the assessment, for example, of outcome, exposure, effects of bias and confounding in individual studies. We believe more widespread use of these guidelines, something that can be mandated by journal editors and peer reviewers, should greatly increase comparability of individual studies, and overall, lead to an improvement in the quality of systematic reviews and meta-analyses.