Systematic Reviews: What Do You Need to Know to Get Started?

INTRODUCTION

The cornerstone of clinical research on interventions is generally considered to be the randomized controlled trial (RCT).¹ However, in topic areas where the number of patients is limited or the evidence is conflicting, systematic reviews offer the benefit of collating evidence from a variety of sources.^1,2 A systematic review attempts to bring together all available evidence on a specific, clearly defined topic. Moreover, in areas where a number of large-scale trials have had similar results, a systematic review that includes meta-analysis of the data can help researchers to find a population estimate for the overall effect of the intervention.

In the hierarchy of evidence, systematic reviews of randomized trials offer the highest level of evidence.¹ The strongest inferences can be drawn if the systematic review is well conducted and includes methodologically sound RCTs with consistent results. In making treatment decisions, the highest quality of evidence should be sought, but well-conducted systematic reviews may not always be available.

This article provides a concise overview of the steps in a systematic review, with a focus on systematic reviews of RCTs. For more in-depth discussion of issues that go beyond the scope of this article, readers are directed to other resources.^2,3

DISTINCTION BETWEEN SYSTEMATIC REVIEWS AND META-ANALYSES

One of the most common errors in understanding systematic reviews is the perception that they are interchangeable with meta-analyses. “Systematic review” is the overarching term for studies that collate available evidence related to a directed clinical question.^2,4 A meta-analysis is a review in which statistical methods are employed to collate the numeric data from the primary studies. For various reasons, not all systematic reviews can combine the available data to generate summary numeric results; however, all systematic reviews should employ stringent methods to summarize the available research.

STUDY QUESTION

As in any type of research, a clear clinical question is needed for a systematic review. The question should state the patient group of interest, the intervention being investigated, the control or comparator group, and the outcomes of interest.^1,2 For example, if you were interested in conducting a systematic review of the effectiveness of new agents (e.g., gliptins) in the treatment of diabetes mellitus, you would need to clearly define your target population and specify whether you are considering studies that compare gliptins with other antidiabetic agents or studies that compare gliptins just with placebo. If you neglect to define each of these parameters a priori, you will have problems in determining which studies should be included and which should be excluded from your systematic review. Table 1 provides an example of how to define the question.

Table 1.

Example of Components in Posing a Relevant Clinical Question

Component of Question	Example
Patient population	Patients with poorly controlled type 2 diabetes mellitus
Intervention	Gliptins (all agents)
Control	Placebo
Outcomes	Change in A1C, hypoglycemic episodes

A1C = glycated hemoglobin.

INCLUSION AND EXCLUSION CRITERIA

Just as an RCT requires that the investigator specify criteria for deciding which patients will be included in the study, a systematic review requires that the investigator define, at the proposal-writing stage, clear criteria for deciding which studies will be included. For example, will you limit your review to English-language studies? Will you include only RCTs, or will you accept other study designs? What outcomes will meet your inclusion criteria—will you include only hard clinical end points (e.g., admissions to hospital, deaths), or will you also consider surrogate end points (e.g., laboratory values)? The decisions about inclusion and exclusion criteria should relate directly to the defined research question. In the example given in Table 1, you might want to consider what is meant by “poorly controlled” (e.g., defined in terms of glycated hemoglobin [A1C] or other clinical markers) and whether there is a specific age range you want to investigate (e.g., between 18 and 75 years). You might also decide whether studies must have a minimal methodologic quality to be considered for inclusion, particularly in terms of the study designs that you will consider acceptable, according to your area of study. RCTs represent the optimal study design; however, not all clinical questions can be answered by an RCT, so you may also need to include observational study designs, depending on the topic.

LITERATURE SEARCH

A systematic review is only as good as the data on which it is based, that is, the primary studies. To ensure that the widest scope of primary research is identified, a thorough and complete search of the literature is needed. The best way to accomplish this is to have help from a librarian with expertise in the area of systematic reviews in defining the search terms, search strategies, and databases to be used. The rule of thumb for a systematic review literature search is that more than 2 databases should be used. For the casual reader, the databases that should be used for a particular systematic review can be difficult to judge, but generally databases beyond just MEDLINE should be searched. Ideally, articles in languages other than English should be included, and there should be an attempt to find unpublished research and research that has not been formally published in a journal. This “grey literature” is the most difficult to find. Grey literature has been defined as “that which is produced on all levels of government, academics, business and industry in print and electronic formats, but which is not controlled by commercial publishers”.⁵ It can include reports, theses, conference proceedings, technical specifications and standards, noncommercial translations, bibliographies, technical and commercial documentation, and official documents not published commercially (such as government reports).⁵ The instantaneous nature of the Internet has led to a proliferation of this type of literature, but the challenge lies in finding it. Searching the websites of agencies and organizations that may be involved in the area of interest is a reasonable way to start. Grey literature repositories and gateways have evolved and offer another method for researchers to find primary evidence not available in the peer-reviewed literature.⁵ Again, searching for grey literature can be much easier with the help of a librarian.

In addition to a formal bibliographic search and a grey literature search, hand-searching of reference lists or key journals should be performed, as well as searching of controlled trial registers. This type of search can also be guided by a librarian. The difficulty with the search stage of a systematic review is keeping all of your potential studies organized. Online or shareable citation software is an easy way to manage the potentially large number of studies being considered. By using an online version, you can allow your co-researchers to have access to the studies. Given that at least 2 researchers will be needed to review the potentially included studies, online citation management allows for easy access by the multiple researchers.

STUDY SELECTION

At this stage, the eligibility of potential studies (based on the predefined inclusion and exclusion criteria) must be checked independently by at least 2 researchers. The lists of included studies prepared by the 2 reviewers are then compared and disagreements resolved, either through discussion or the involvement of a third researcher. The numbers of studies searched and included, and the reasons for study exclusion, should be recorded and presented as a flow diagram in the final manuscript.⁶ Each and every potentially relevant study found through the search should be accounted for in this diagram.

DATA EXTRACTION AND QUALITY ASSESSMENT

As with study selection, data extraction should be conducted independently by 2 members of the research team. This helps to reduce or eliminate errors in the data compiled for analysis. Bias can also be reduced if the data extractors are blinded to study authors and journals of publication. To eliminate discrepancies, the data extraction forms should be standardized, so that the 2 reviewers are collecting the same information from each study. Similar to the process for study selection, a method for dispute resolution should be discussed a priori.

Determination of the quality of included studies should be based on whether the included studies have minimized bias in their study design (internal validity). The Cochrane Collaboration has moved away from using the term “quality” and now uses “bias” instead.³ One reason for this shift is that focusing on bias allows you to concentrate on the quality of the underlying research, not just the reporting of that research. Meta-epidemiologic methods have been used to study the biases associated with study characteristics.⁷ This type of work has led researchers to focus bias assessment on areas such as allocation concealment, randomization, blinding, and incompleteness of data and reporting. The Cochrane Collaboration has developed a tool to record whether these aspects of the study were done appropriately, not simply whether they were reported (as is commonly seen with checklist approaches to quality assessment).³ A full description of the use and applicability of these tools is beyond the scope of this article; for more detailed instructions, readers are referred to the Cochrane Handbook for Systematic Reviews of Interventions (see Chapter 8: Assessing risk of bias in included studies).³ The risk-of-bias tool is deemed more difficult and time-consuming to complete than previous methods of assessing bias; however, it is also viewed as providing more valuable information.⁸ The main area of difficulty for researchers is how to incorporate risk-of-bias information into the data analysis. The Cochrane Collaboration is currently working on advanced instructions to assist in this domain.⁸

Another step at this stage of the process is to contact the corresponding authors of the included studies for answers to any questions that have arisen during data extraction and for clarification of any areas of uncertainty in the methods and results. You can also inquire about any unpublished data that may contribute to your outcomes of interest. Authors are generally happy to provide this information, which helps to strengthen the results that you will generate.

DATA ANALYSIS

At this point in the review process, the hardest work has been completed, as you have already sifted through hundreds of citations and extracted data from each of the included studies. Now you need to determine if meta-analysis of the results will be appropriate. This decision will be driven mostly by whether or not the outcomes of the included studies were measured in a similar fashion and reported consistently. For example, if all of your included studies reported change in A1C at the end of the follow-up period, those data can easily be combined. However, if the studies report a variety of outcomes that are not directly comparable (e.g., A1C, fasting glucose, quality of life), the data cannot be combined. Other issues, such as similarity of patient populations, interventions, and methods used in the primary studies, will also need to be considered before data are combined; this is known as a qualitative assessment of heterogeneity. Although your study protocol should include your plan for data analysis based on your research question and outcomes of interest, you will not know the exact types of data you will collect until data extraction is complete.

The Cochrane Collaboration offers free software called Review Manager (RevMan)⁹ that is useful for analyzing results and generating forest plots and risk-of-bias tables. Although this software is relatively straightforward and easy to use, you will need a basic understanding of statistical methods to make decisions on how the data should be analyzed. In addition, your primary outcome of interest and how it is measured (dichotomous versus continuous data) will affect the type of analysis you can do. For studies with dichotomous outcomes (e.g., dead or alive, admitted to hospital or not admitted), RevMan can calculate odds ratios and risk ratios from the included data. The generation of combined odds ratios (and confidence intervals) will also lead to an I² statistic to help in interpretation of heterogeneity (described in more detail below). How the software calculates the odds ratios will depend on your decision of what type of statistical model you want: fixed or random effects. A random-effects model assumes that the different studies are estimating different, yet related, intervention effects.³ Generally, it is considered the most conservative method for meta-analyses. A complete discussion about determining what model should be used is beyond the scope of this article; however, the Cochrane Handbook for Systematic Reviews of Interventions³ offers useful background information to help in answering some of these statistical questions. In addition, consultation with a statistician will help at this point of the research, if you have not already consulted one during proposal and study design. A statistician can also assist in selecting further statistical methods for subgroup analysis, sensitivity analysis, and investigation of sources of heterogeneity.

HETEROGENEITY

Heterogeneity is defined as any variability seen across the included studies.³ For example, heterogeneity may derive from differences in study designs, populations, and outcomes.¹⁰ If significant heterogeneity exists between the included studies, meta-analysis should not be considered. Even if the data can be combined, heterogeneity throughout the combined data should be measured. The most useful measure of heterogeneity is the I² statistic, which quantifies the level of heterogeneity present in each outcome and also helps in determining, post hoc, whether the data are interpretable.¹⁰ A full discussion of heterogeneity is beyond the scope of this article, but readers can find more information in specific references on this topic.^3,10 These references also discuss methods that can be used to explore the sources of heterogeneity based on meta-regression and subgroup analyses.

INTERPRETATION AND REPORTING OF RESULTS

Now the work of the systematic review is done, and you want everyone to know what you found! However, putting the results together in a succinct and readable form can be difficult, especially if you have included a large volume of evidence.

The Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) statement offers an outline of what should be included in a published systematic review.⁶ The PRISMA checklist covers everything from what should be mentioned in the title, to what methods should be reported, to how the results should be presented and discussed. It is an excellent template for formatting your report and determining what may be missing (the PRISMA checklist can also be used as an aid during design of your study protocol, as it will help to frame some of the design decisions you need to make along the way). In general, the first figure in your article should be a clear flow diagram of all the studies found, the various exclusion steps, and the studies actually included in the systematic review. The first table in the systematic review should list all of the included studies, with details about each one, such as the patient population, intervention, control group, outcomes, and some basic results. Table 2 in the current article summarizes each step of the process of conducting a systematic review and will help you in drafting your final manuscript.

Table 2.

Key Steps in Conducting a Systematic Review

Step	Components
Define the clinical question	Use the PICO format (population, intervention, control, outcome).
Define inclusion and exclusion criteria	Consider study design, time of follow-up, clinical setting, comordities, demographic characteristics (e.g., age groups, sex), characteristics of the control group, outcome importance, language of publication, date of publication (as a range of years).
Conduct literature search	Consider appropriate databases, number of databases, publication date range, grey literature.
Select studies	Use at least 2 independent reviewers. Have the reviewers use the inclusion and exclusion criteria as the basis for their selections. Specify a method to resolve discrepancies (e.g., consensus, third reviewer).
Extract the data	Develop a data extraction form based on PICO. Have 2 researchers extract the data independently.
Determine level of bias (previously termed “study quality”) of included studies	The risk-of-bias tool is currently the preferred method. A different tool may be needed if non–randomized controlled trials are included.
Determine heterogeneity (qualitatively)	Determine whether there are major differences between studies in terms of patients, interventions, outcomes, study design (quality).
If appropriate, conduct meta-analysis	Using appropriate statistical software and the I2 statistic, combine the data. Reassess outcomes for statistical heterogeneity. Generate summary estimates and confidence intervals.
Evaluate and interpret results	Consider clinical significance and practice context.
Publish	Follow the PRISMA guidelines6 in drafting your manuscript.

PRISMA = Preferred Reporting Items for Systematic Reviews and Meta-Analyses.

CONCLUSION

To ensure you are producing high-level evidence, begin by partnering with researchers who have experience with systematic reviews and ensuring accessibility to a librarian and a statistician. Using available resources, such as the Cochrane Collaboration and the PRISMA guidelines, develop a strong question and project proposal, with clearly defined inclusion and exclusion criteria, literature search strategy, and data analysis plan. Do not be discouraged if the data cannot be analyzed using meta-analytic methods, as the systematic review can still be highly relevant and useful without pooled data. By following these steps, you should be able to develop a plan and assemble a strong team to move your planned review forward.

This article is the ninth in the CJHP Research Primer Series, an initiative of the CJHP Editorial Board and the CSHP Research Committee. The planned 2-year series is intended to appeal to relatively inexperienced researchers, with the goal of building research capacity among practising pharmacists. The articles, presenting simple but rigorous guidance to encourage and support novice researchers, are being solicited from authors with appropriate expertise.

Previous article in this series:

Bond CM. The research jigsaw: how to get started. Can J Hosp Pharm. 2014;67(1):28–30.

Tully MP. Research: articulating questions, generating hypotheses, and choosing study designs. Can J Hosp Pharm. 2014;67(1):31–4.

Loewen P. Ethical issues in pharmacy practice research: an introductory guide. Can J Hosp Pharm. 2014;67(2):133–7.

Tsuyuki RT. Designing pharmacy practice research trials. Can J Hosp Pharm. 2014;67(3):226–9.

Bresee LC. An introduction to developing surveys for pharmacy practice research. Can J Hosp Pharm. 2014;67(4):286–91.

Gamble JM. An introduction to the fundamentals of cohort and case–control studies. Can J Hosp Pharm. 2014;67(5):366–72.

Austin Z, Sutton J. Qualitative research: getting started. Can J Hosp Pharm. 2014;67(6):436–40.

Houle S. An introduction to the fundamentals of randomized controlled trials in pharmacy research. Can J Hosp Pharm. 2014; 68(1):28–32.