Abstract
The explosion of biomedical publishing makes keeping up with the primary studies an impossible task. The often disparate, confusing and contradicting findings of individual studies makes healthcare professionals turn to review articles where knowledge has been collated and published in summaries. Narrative reviews lack rigorous, systematic and reproducible synthesis. In contrast, systematic reviews are conducted using systematic and explicit methods to identify, select and critically appraise relevant research, and to collect and analyse data from the studies that are included in the review. The final pathway for systematic review is a statistical summary of the results of primary studies, or meta-analysis. This article provides some guidelines to health care providers in understanding the key aspects of systematic review and meta-analysis. Steps involved in systematic review are discussed. The potential pitfall of meta-analysis was also explored.
THE NEED FOR REVIEWS
The huge amount of medical information available and its exponential growth have become common problem in the literature of biomedical information. Health care practitioners face the explosion in biomedical knowledge which makes keeping up with the primary research an impossible feat. There are approximately 17,000 new biomedical books published every year along with 30,000 biomedical journals, with an annual increase of 7%. For instance, MEDLINE alone contains more than eleven million citations, and more than 400,000 articles are added to the file each year.1 Majority health care providers noted that the current volume of scientific literature is unmanageable2 and often do not have sufficient time for reading medical journals as the information explosion continues.3 Further, some of these studies could be unclear, confusing or may also have contradicting results.
THE NARRATIVE REVIEWS
To make this task easier and manageable for health care providers as well as decision makers, reviews are often among their information resources. Reviews have always been a part of the medical literature. Traditionally, medical research has been integrated in the narrative or nonsystematic form. An expert in a particular field will review studies, decided on the relevance, and highlight the findings, both in terms of results and, to a lesser degree the methodology.4 Such narrative reviews tend to be unsystematic and susceptible to many biases. Firstly, no systematic approach is prescribed to obtain the primary data and to integrate the data. Often, subjective judgment of the reviewer was used. There were often no explicit standards exist to assess the quality of review. Moreover, narrative reviewer also does not synthesize data quantitatively.
A CALL FOR SYSTEMATIC REVIEWS
A systematic review is defined by the Cochrane Handbook as ‘A review of a clearly formulated question that uses systematic and explicit methods to identify, select and critically appraise relevant research, and to collect and analyse data from the studies that are included in the review’. In contrast to narrative review, systematic review allows readers to appraise how the review was conducted and synthesized. It is of particular value in bringing together a number of separately conducted studies, sometimes with conflicting findings, and synthesizing their results. Systematic reviews have been proven to be able to yield valid, precise, and widely applicable answers to clinical questions.5 In short, systematic reviews summarise large amounts of information and are more likely than individual trials to describe the true clinical effect of an intervention. Thus, systematic reviews have come to play a central role in informing clinical decisions and guidelines. A systematic review is also often called an ‘overview’.
A meta-analysis takes a systematic review one step further by statistically pooling the results of combinable studies. Since its introduction, meta-analysis has established itself as an influential branch of clinical epidemiology and health services research, with hundreds of meta-analyses published in the medical literature each year.6
THE PROCESS OF SYSTEMATIC REVIEW
Systematic review should be carefully planned with a detailed written protocol prepared in advance as any other search project. Systematic review involves several discrete steps and the steps are summarised below.
Step 1: Formulate review question. This requires the formulation of a clear statement of relevant patient groups, intervention of interest, as well as outcomes. The details are used to select studies for inclusion in the review.
Step 2: Locate studies. Systematic review must be undertaken in accordance with a predefined search strategy that would allow the completeness of the search to be assessed. Search strategies should consider the following sources: The Cochrane Controlled Trials Register (CCTR), other electronic databases and trials registered not covered by CCTR, checking reference lists, hand searching of key journals and personal communication with experts in the fields. The selection of primary studies is governed by inclusion and exclusion criteria that are initially specified when the protocol is defined.
Step 3: Appraising the quality of studies. After an exhaustive search, all possible primary studies that have been identified need to be assessed for eligibility for inclusion. Application of stringent inclusion/exclusion criteria should be addressed for example types of participants, interventions, outcomes, study designs and methodological quality. Independent assessment by more than one observer is desirable.
Step 4: Combining the results. The findings from combinable individual primary studies are then pooled to produce an ‘overall estimate’ on the clinical effectiveness of the intervention. The aggregation can be qualitative, or more appropriate, by statistically combining the data produced by individual studies into a single summary estimate. The statistical pooling of data is termed meta-analysis (Figure 1). In meta-analysis, results from studies are combined using ‘inverse variance method’, whereby larger studies and studies with less random variation are given greater weight than smaller studies.
Figure 1.
Meta-analysis: Data from several different studies are combined and produce a single estimate
Meta-analysis can only be undertaken when studies address the same question, administer the intervention in a similar manner or measure the same outcomes. When studies differ in one or more of these components, meta-analysis is not appropriate. Therefore, systematic review may or may not include meta-analysis.
In meta-analysis, for outcomes measured on a continuous scale, the weighted mean difference is commonly used. For outcomes measured on a dichotomous scale, common approaches include the use of odds ratio or relative risk. There are two approaches for combining the data: fixed-effects model assume that an intervention has a single true effect whereas random-effects models assume that an effect may vary across studies.7 The results of meta-analysis can be displayed graphically (Forest plot) to allow a visual comparison of findings of individual studies (Figure 2).
Figure 2.
Forest plot of meta-analysis
Systematic review should continue with an investigation of the reasons for heterogeneity. Subgroup analysis, sensitivity analysis and meta-regression are frequently used to investigate heterogeneity of individual studies in meta-analysis. One of the major drawbacks to using meta-analysis is the possibility of publication bias. One way to investigate whether a review is subject to publication bias is to prepare a ‘funnel plot’ (Figure 3) and examine this for signs of asymmetry.8
Figure 3.
Funnel plot showing evidence of publication bias
Step 5: Interpret results. The findings from systematic review and statistical pooling of the studies then need to be interpreted, discussed and set out the implications for practice or further research. Issues such as the quality and heterogeneity of the included studies plus the possible impact of bias need to be discussed.
PITFALLS AND PROBLEMS
Meta-analyses have received mixed receptions. Some see meta-analysis as an exercise of ‘mega-silliness’,9 ‘a tool has become a weapon’10 and a number of statisticians think that meta-analysis represents the unacceptable face of statisticism.11 There are also those that still prefer the conventional narrative review article.12 The mixed receptions were due to opposite conclusions observed in some systematic reviews that address the same issue.13,14 Also, meta-analyses of small trials were discovered to contradict by a single large randomized trial.15
Publication bias could be a serious problem for meta-analyses, secondly, studies may be of varying quality. Clearly the quality of trials included in a systematic review and meta-analysis is of crucial importance and should be of high methodological quality as well as free from biases. Meta-analysis should therefore be considered only within the framework of systematic review that has been prepared using a systematic approach to mitigate all kinds of biases and explicitly address the issue of the completeness of the evidence identified, the quality of component studies and the combinability of studies.15 The Cochrane Collaboration has been established to overcome this problem by providing high quality and authoritative systematic reviews and meta-analyses.16,17 The collaboration not only ensures that high-quality reviews are conducted but also update reviews when new evidence becomes available.
CONCLUSION
Systematic review is an invaluable resource for both clinicians and researchers. However, not all reviews are systematic and even those that are described as systematic may be methodologically flawed. Nonetheless, a high quality systematic review provides the best available evidence. The usefulness of a systematic review can further be enhanced by statistical summary of the results by meta-analytical technique. Pooling individual studies may reduce the risk of random error, increase statistical power and allow for a more accurate estimate of effect size.
SOME TERMINOLOGIES
Bias (synonym: systematic error): the distortion of the outcome, as a result of a known or unknown variable other than intervention (i.e. the tendency to produce results that depart from the “true” result).
Cochrane Collaboration: The Cochrane Collaboration is an international organization that aims to help people make well-informed decisions about healthcare by preparing, maintaining & promoting the accessibility of systematic reviews of the effects of healthcare interventions.
Cochrane Controlled Trials Register (CCTR): CCTR is a database of references to controlled trials in health care.
Critical appraisal: systematically finding, appraising and interpreting evidence of effectiveness. It is aimed to examine research evidence to assess its validity, results and relevance before using it to inform a decision.
Cumulative meta-analysis: the repeated performance of meta-analysis whenever a new trial becomes available for inclusion. In cumulative meta-analysis studies are added one at a time in a specified order.
Effect size: refers to the size of a relationship between an expose and an outcome. The term is applied to measurement of the differences in the outcome between the study groups. Relative risk, odds ratio, and risk differences can be defined as effect sizes for dichotomous scale. Effect size of continuous variable is the standardized mean differences.
Fixed-effect model: a mathematical model that combines the results of studies that assume the effect of the intervention is constant in all subject population studied. Only within study variation is included when assessing the uncertainty of results.
Forest plot: a forest plot presents the means and variance for the difference for each pooled primary study. The line represents the standard error of the difference, the box represents the mean difference and its size proportional to the number of subjects in the study. The bottom entry in a forest plot is the summary estimate of the treatment difference and confidence interval for the summary difference (Figure 2).
Funnel plot: a graphical method of assessing bias; the effect size of each study is plotted against some measure of study information. If the shape of the plot resembles an inverted funnel, it can be stated that there is no evidence of publication bias within the systematic review (Figure 3).
Heterogeneity: the variability between studies in terms of key characteristics (i.e. ecological variables) quality (i.e. methodology) or effect (i.e. results). Statistical tests of heterogeneity may be used to assess whether the observed variability in effect size (i.e. study results) is greater than that expected to occur purely by chance.
Meta-regression: a multivariable model investigating effect size from individual studies, generally weighted by sample size, as a function of various study characteristics (i.e. to investigate whether study characteristics are influencing effect size).
Outlier : an outlier study in meta-analysis is study that results very different from the rest of the studies. Outlier could alter the conclusions of a meta-analysis.
Overall estimate: is the pooled estimate from a meta-analysis. The overall estimate from a meta-analysis is always displayed with its confidence interval.
Primary studies: Individual studies contributing to a systematic review are called primary studies whereas a systematic review is a form of a secondary study.
Publication bias: publication bias refers to the problem that positive results are more likely to be published than negative results and this may therefore give a misleading assessment of the impact of an intervention. Publication bias can be examined via a funnel plot.
Random-effects model: a mathematical model for combining the results of studies that allow for variation in the effect of the intervention amongst the subject populations studied. Both within-study variation and between-study variation is included when assessing the uncertainty of results.
Review: article that summarizes a number of primary studies and discusses the effectiveness of a particular intervention. It may not be a systematic review.
Search strategy: a description of the methodology used to locate and identify research articles pertinent to a systematic review, as specified within the relevant protocol. It includes a list of search terms, based on the subject, intervention and outcome of the review, to be used when searching electronic databases, websites, reference lists and when engaging with personal contacts. If required, the strategy may be modified once the search has commenced.
Sensitivity analysis: repetition of the analysis using different sets of assumptions in order to determine the impact of variation arising from these assumptions, or uncertain decisions, on the results of a systematic review.
Subgroup analysis: used to determine if the effects of an intervention vary between subgroups in the systematic review.
Weighted mean difference: a method used to combine measures on continuous scales (where the mean, standard deviation and sample size in each group are known) and the weight given to each study is determined by the precision of its estimate of effect.
Cochrane Systematic Review and SEA-ORCHID Project
Cochrane systematic reviews combine the results of the best medical research using rigorous methods, and are regarded as the gold standard of reference for health care professionals. Malaysia has relatively minor involvement in Cochrane Collaboration despite its economic growth and the fast improving standard of medical care. It is likely that clinical questions with high relevance to Malaysia are therefore not being addressed in Cochrane reviews.
The SEA-ORCHID project, which stands for South East Asia Optimising Reproductive and Child Health Outcomes in Developing Countries Project, is a five-year project (2003 to 2008) aiming to promote the synthesis and application of high level clinical evidence on issues relevant to this region, focusing on maternal and child health but also involving other related disciplines. Jointly funded by the Wellcome Trust and the Australian National Health and Medical Research Council and supported by the Cochrane Australasian Centre, the project activities include regular Cochrane Systematic Review Workshop and work-in sessions throughout the country. This is a good opportunity for the pool of clinical and research talents in our country to contribute in synthesizing the best clinical evidence and making a significant impact on evidence-based health care.
If you are interested in authoring or co-authoring a Cochrane review, you will be guided at every step by experienced reviewers leading to its publication in the Cochrane Library. In this workshop, you will also hear the experiences of people who are in the process of developing a protocol or review.
For further information please contact:
Professor Jackie Ho jackie@pmc.edu.my
Dr Lai Nai Ming naiming_lai@imu.edu.my
References
- 1.MEDLINE fact sheet. [Web Document]. Bethesda, MD: National Library of Medicine. http://www.nlm.nih.gov/ pubs/factsheets/medline.html. 2007 Mar 12 [Google Scholar]
- 2.Williamson JW, German PS, Weiss R, Skinner EA, Bowes F. Health science information management and continuing education of physicians. A survey of U.S. primary care practitioners and their opinion leaders. Ann Intern Med. 1989;110(2):151–60. doi: 10.7326/0003-4819-110-2-151. [DOI] [PubMed] [Google Scholar]
- 3.Hunt RE, Newman RG. Medical knowledge overload: a disturbing trend for physicians. Health Care Manage Rev. 1997;22(1):70–5. [PubMed] [Google Scholar]
- 4.Thacker SB. Meta-analysis: a qualitative approach to research integration. JAMA. 1988;259(11):1685–9. doi: 10.1001/jama.259.11.1685. [DOI] [PubMed] [Google Scholar]
- 5.Oxman A, Guyatt GH, Cook D, Montori V, Guyatt GH, Rennie D. Chicago, Ill: AMA Press; 2005. User’s Guides to the Medical Literature. A Manual for Evidence-based Clinical Practice; pp. 155–73. [Google Scholar]
- 6.Egger M, Davey Smith G, O’Rourke K. In: Systematic Reviews in Health Care: Meta-Analysis in Context. Egger M, Davy Smith G, Altman DG, editors. London: BMJ Books; 2001. Rational, potentials and promise of systematic reviews; pp. 23–42. [Google Scholar]
- 7.Sutton AJ, Abrams KR, Jones DR, Sheldon TA, Song F. Systematic reviews of trials and other studies. Health Technol Assess. 1998;2(19):1–272. [PubMed] [Google Scholar]
- 8.Ferrer RL. Graphical methods for detecting bias in meta-analysis. Fam Med. 1998;30(8):579–83. [PubMed] [Google Scholar]
- 9.Eysenck HJ. An exercise in mega-silliness. Am Psychol. 1978;33:517. [Google Scholar]
- 10.Boden WE. Meta-analysis in clinical trials reporting: Has a tool become a weapon? Am J Cardiol. 1992;69(6):681–6. doi: 10.1016/0002-9149(92)90164-t. [DOI] [PubMed] [Google Scholar]
- 11.Oakes M. Chichester: John Wiley & Sons,; 1986. Statistical inference: a commentary for the social and behavioral science. [Google Scholar]
- 12.Bailar JC. The promise and problems of meta-analysis. N Engl J Med. 1997;337(8):559–61. doi: 10.1056/NEJM199708213370810. [DOI] [PubMed] [Google Scholar]
- 13.Nurmohamed MT, Rosendaal FR, Bueller HR. et al. Low molecular-weight heparin versus standard heparin in general and orthopaedic surgery: a meta-analysis. Lancet. 1992;340:152–6. doi: 10.1016/0140-6736(92)93223-a. [DOI] [PubMed] [Google Scholar]
- 14.Leizorovicz A, Haugh MC, Chapuis FR, Samama MM, Boissel JP. Low molecular-weight heparin in prevention of perioperative thrombosis. BMJ. 1992;305:913–20. doi: 10.1136/bmj.305.6859.913. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 15.Egger M, Davey Smith G, Sterne JAC. Uses and abuses of meta-analysis. Clin Med. 2001;1(6):478–84. doi: 10.7861/clinmedicine.1-6-478. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 16.Chalmers I, Altman DG. Systematic reviews. London: BMJ Press; 1995. [Google Scholar]
- 17.Mulrow CD. Rationale for systematic reviews. BMJ. 1994;309:597–9. doi: 10.1136/bmj.309.6954.597. [DOI] [PMC free article] [PubMed] [Google Scholar]