Systematic Reviews in Hematopoietic Cell Transplantation and Cellular Therapy: Considerations and Guidance from the American Society for Transplantation and Cellular Therapy, European Society for Blood and Marrow Transplantation, and the Center for International Blood and Marrow Transplant Research Late Effects and Quality of Life Working Committee

Abstract

Systematic reviews apply rigorous methodologies to address a pre-specified, clearly formulated clinical research question. The conclusion that results is often cited to more robustly inform decision-making by clinicians, third-party payers and managed care organizations about the clinical question of interest. While systematic reviews provide a rigorous standard, they may be unfeasible when the task is to create general disease-focused guidelines comprised of multiple clinical practice questions versus a single major clinical practice question. Collaborating transplantation and cellular therapy societal committees also recognize that the quantity and or quality of reference sources may be insufficient for a meaningful systematic review. As the conduct of systematic reviews has evolved over time in terms of grading systems, reporting requirements and use of technology, here we provide current guidance in methodologies, resources for reviewers, and approaches to overcome challenges in conducting systematic reviews in transplantation and cellular therapy.

Introduction

Reviews based on a systematic assessment of evidence from the literature help providers make informed clinical decisions and educate third-party payers and managed care organizations about best practices.^1,2 These evidence-based, or systematic reviews, include an a priori stated research question, a well-described comprehensive literature search, a justification for the inclusion or exclusion of the available evidence based on independent reviewers, a risk of bias assessment, an appraisal of the quality of collected data, and a systematic presentation and synthesis of the findings of included studies.^3–8 If the quality of evidence permits, a quantitative analysis (meta-analysis) may also be performed.⁹ This systematic approach minimizes individual bias and increases the reliability and quality of the conclusions drawn,⁸ making systematic reviews the gold standard in evidence-based medicine.¹⁰

In 1999, the American Society for Blood and Marrow Transplantation (ASBMT, now the American Society for Transplantation and Cellular Therapy [ASTCT]) launched an initiative to conduct reviews of the role of transplantation in select diseases.¹ In 2000, the Evidence-Based Review Steering Committee published the methodology to be followed, with inclusion criteria to be determined by the expert panel, the quality of the body of evidence to be graded based per Shipp et al,¹¹ and the strength of recommendations to be graded per Chalmers et al.¹² The methodology was revised in 2005 to incorporate four standard inclusion/exclusion criteria. Studies had to be published in 1990 or later, have a minimum of 70% of subjects with the disease under review or results stratified by the disease, have 25 or more subjects unless exclusion of such studies would profoundly affect treatment recommendations, and meeting abstracts and data from non-peer reviewed journals were to be excluded.¹³ The Scottish Intercollegiate Guidelines Network (SIGN) guidelines were to be used to assess the levels of evidence and to grade the strength of treatment recommendations; methodology for conducting a meta-analysis was included.^13,14 As of 2009, seven reviews had been published,^15–21 at which time the Steering Committee recommended that reviews be updated at 5-year intervals.²²

Many systematic reviews have been performed by ASTCT, the European Society for Blood and Marrow Transplantation (EBMT), the Center for International Blood and Marrow Transplantation Research (CIBMTR), the National Institutes of Health (NIH), other professional societies, and investigators throughout the world on transplant and cellular therapy (TCT)-related topics using various methodologies. For example, over 50 systematic reviews and meta-analyses were published from January 1, 2018 to July 31, 2020 (MeSH terms: “hematopoietic stem cell transplantation” AND “systematic review” AND “meta-analysis”; MEDLINE search and review independently conducted by AS and LJB).

Since the 2005 ASBMT statement, the conduct of systematic reviews has evolved with the use of new grading systems, adoption of uniform reporting requirements, and development of software programs that facilitate international collaborative reviews. Here we review how the current standards for conducting systematic reviews apply to the field of TCT, provide useful resources for professionals undertaking such reviews, and give guidance on how to overcome common pitfalls along with some practical considerations.

Conduct of a Systematic Review

Suggested steps for conducting the review are shown in Figure 1 with details provided in the following sections. Examples of software and websites that can assist throughout these steps are provided in Table 1.

Figure 1.

A flow diagram of the various steps involved in developing a systematic review.

Table 1.

Examples of software and websites that can assist in developing and managing a systematic review.

Product	Cost	Functionality
		Citation manager	Deduplication	Screening studies	Documentation of review	Data extraction	Statistical analysis
Abstrackr http://abstrackr.cebm.brown.edu	Free			×
CADIMA https://www.cadima.info	Free		×	×	×
Colandr https://www.colandrapp.com	Free			×	×	×
Covidence https://www.covidence.org	Free limited version. Full version requires subscription.			×	×	×
DistillerSR https://www.evidencepartners.com/products/distillersr-systematic-reviewsoftware/	Subscription required		×	×	×	×
EndNote https://endnote.com	Free online version lacks some functionality. Desktop version requires subscription.	×	×	×
EPPI-Reviewer https://eppi.ioe.ac.uk/CMS/Default.aspx?alias=eppi.ioe.ac.uk/cms/er4&	Subscription required	×		×	×	×	×
Rayyan https://rayyan.qcri.org	Free			×	×
RevMan https://training.cochrane.org/resource/introduction-revman	Free	×		×	×		×
Systematic Review Data Repository (SRDR) https://ser.ahrq.gov	Free			×	×	×
System for the Unified Management, Assessment and Review of Information (SUMARI) http://www.jbisumari.og	Subscription required	×		×	×	×	×
Zotero https://www.zotero.org	Free	×	×	×

Form a multidisciplinary team

Prior to beginning the review process, a multidisciplinary study team should be assembled based on the intended purpose and target audience. Expertise in data search and information retrieval, the clinical topic, and knowledge synthesis methods should be represented in such a team. A library information sciences specialist is an asset to facilitate design of a comprehensive search strategy and enable retrieval of all relevant studies across different databases. Specific expertise in biostatistics should be included if quantitative data synthesis is planned. Ideally, stakeholders linked to the research topic (such as clinicians, patients, caregivers, donors, and policy makers), should be included to enhance usage of review findings^{6, 23–25} plus their inclusion may be mandated by some funding bodies.²⁶ All team members should disclose relevant conflicts of interest and professional or intellectual biases in order to critically appraise their potential contribution to the team.

Determine the question and inclusion/exclusion criteria

A clear, concise, specific, and answerable research question is critical in developing a high-quality review. In preparation to question development, the team should first evaluate what research has been conducted previously or is ongoing, noting what findings will be relevant to the target audience.^{27, 28} It may be optimal to update a previous review if there have been considerable advances in the literature or methodological advances.^29–34

A common acronym used to refine research questions is PICOS: P stands for population or problem of interest; I for intervention, exposure, or indicator to be evaluated; C for comparison or control group; O for the outcome measure being assessed; and S for study design.²⁵ All five components may not be applicable but considering each will enable the team to refine the question. In TCT, one can define the study population based on standard characteristics such as: (1) demographics including participants’ age, sex, race, and ethnicity; (2) disease-related factors, such as disease status (e.g., in remission or with relapsed or refractory disease) and disease risk (e.g., early, intermediate, high, very high);^35–37 and (3) transplant and cell therapy-related factors, including type (e.g., allogeneic or autologous), donor source (e.g., human leukocyte antigen [HLA]-matched related/unrelated donor or haploidentical donor), graft source (e.g., bone marrow, peripheral blood–derived stem cells, or cord blood). Similarly, comparison groups may be broadly (e.g., HLA-matched related graft recipients versus HLA-matched unrelated graft recipients) or more specifically defined (e.g., transplant recipients receiving calcineurin inhibitor–based graft-versus-host disease [GVHD] prophylaxis versus others).

Study inclusion/exclusion criteria should be defined before initiating the literature search in order to be unbiased by the literature available. Any necessary revisions based on practical aspects of the data available following the initial search should be applied consistently throughout the review process.^28,38 Criteria should state years of study publication, language restrictions (if applicable) and specific items linked to the research question. Accepted study designs that may best answer the research question should also be predefined.³⁹ Studies may be of varying designs but should be studying the same outcome. Generally, when conducting a systematic review measuring an outcome of an intervention, data from randomized controlled trials (RCT) and quasi-RCT are preferred. However, high-quality non-randomized studies and observational studies may be included as they provide evidence of “real world” data. Case series and case reports are usually excluded because they are associated with a high potential for bias²⁷ but may be included if the research question is related to a rare event or outcome. Review articles, abstracts, meeting presentations, editorials, viewpoints and other data that have not been peer reviewed are typically excluded.

Write a plan for the review (protocol)

The plan for the review should be summarized into a protocol guided by consensus recommendations in the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines.⁴⁰ The protocol should include relevant background, rationale for conducting the review, the research question, data collection methods (literature search plan, inclusion and exclusion criteria and data extraction methods), risk of bias assessment methods, and planned data synthesis and/or analysis methods. The protocol should be published to maintain transparency, prevent duplicative efforts by other investigators, ensure reproducibility, and reduce bias.⁶ This can be done through online registration in databases such as PROSPERO,⁴¹ Cochrane Library,⁴² F1000 Research,⁴³ Joanna Briggs Institute,⁴⁴ Open Science Framework registries,⁴⁵ or Zenodo⁴⁶ or published in a peer-reviewed journal. Any protocol amendments should be recorded. An example of such a protocol recently initiated as a collaborative effort between the CIBMTR and the EBMT is available in the PROSPERO database (registration number: CRD42020147640).⁴⁷

Search for studies

As bibliographic databases differ with respect to their coverage of scholarly disciplines, publication types, dates of publication, journals and books indexed, geographic locations, languages, and search syntaxes, it is a best practice to use more than one database.⁴⁸ In addition to the commonly used MEDLINE, other medical databases include Embase, Web of Science, Scopus, Cumulative Index to Nursing and Allied Health Literature (CINAHL), PsycINFO and the Cochrane Library. Most databases require subscriptions; access may be available through academic libraries.

The search strategies should be constructed in collaboration with a library information sciences specialist to ensure that they are well-constructed, thorough and reproducible.⁴⁹ The team should identify keywords, name variations, synonyms, and subject headings for the key concepts of the research question. Researchers from different institutions, fields of discipline, or geographic locations may use different terminology to describe the same idea; therefore, the search strategy should be as inclusive as possible. For instance, hematopoietic stem cell transplantation may be referred to as HSCT, hematopoietic cell transplantation, bone marrow transplantation, stem cell transplantation, cord blood transplantation, or peripheral blood transplantation. Terms are joined using Boolean operators (AND, OR, NOT). An example of a search strategy in TCT is included as Supplementary Table 1. Search strategies should also be adjusted by using the advanced search features specific to the database searched and peer-reviewed to ensure quality.⁵⁰ The search strategy should be pilot tested to ensure that the major articles considered relevant are being captured. If key articles are missing, the search string should be modified and retested prior to finalization.

Screen and select studies

Specialized software applications have simplified the process of screening, removing duplications, and selecting studies (Table 1). All references should undergo screening in two stages by at least two independent members of the study team.^6,25 The first stage involves screening the title and abstract to exclude articles that clearly do not meet the inclusion criteria. In the second stage, the full text of all potentially eligible articles is reviewed to ensure that the studies meet all inclusion and exclusion criteria. If there are unresolved disagreements, a discussion with a third reviewer may help reach a consensus. For studies chosen for inclusion, the reference list of each article and/or any publications referring to that study can be hand-searched for further relevant studies (the ‘snowball method’). The study flow and the reasons for excluding full-text papers should be documented in an adapted PRISMA flow diagram (a hypothetical example diagram is shown in Figure 2).^{51, 52}

Figure 2.

Hypothetical PRISMA flow diagram.

Source: Moher D, Liberati A, Tetzlaff J, Altman DG, PRISMA Group. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. BMJ 2009;339:b2535.

Extract data from the studies

The use of a standardized data collection form ensures the uniformity of data acquisition. Piloting the form will ensure that all relevant data are collected, and adjustments made to avoid unnecessary details. Free-text fields should be limited, and data coded in pre-defined numerical, fixed-text (yes/no), or drop-down menus or lists to avoid discrepancies and ambiguity at the time of data synthesis. Software applications are available that support the creation of custom data-collection forms (Table 1), although word-processing or spreadsheet software programs may also be used. A minimum of two independent reviewers should collect data from all included studies. The level of inter-rater agreement may be calculated using the kappa statistic and reported.⁵³ Any disagreements should be resolved by discussion with a third reviewer.

Assess the quality of the studies

The quality of evidence and risk of bias of the included studies should be evaluated. Table 2 provides the quality ratings and their definitions of several frameworks that have been used in TCT reviews depending on the scope and research question. Over 100 global organizations have adopted the Grading of Recommendations Assessment, Development and Evaluation (GRADE)^54–56 framework for evaluating the quality of a body of evidence.⁵⁷ The GRADE framework has four levels of quality; several factors may impact the quality levels.⁵⁸ Factors decreasing the quality of a review include inadequate design of included studies, lack of direct evidence, unexplained heterogeneity or inconsistency of results, lack of precision of results or publication bias (i.e., when the outcome influences the decision to publish). Inclusion of several large studies or studies reporting a large magnitude of effect or dose-response gradient in a review increase quality of the evidence. The quality rating may decrease or increase by one level for each factor up to a maximum of three levels. Therefore, the GRADE framework addresses a common challenge reviewers face when assessing the quality of evidence from a wide range of studies with heterogeneity in design, populations, interventions and/or outcomes. While quality of evidence from case reports and case series is very low, they may be incorporated in decision making based on the GRADE approach when no other higher level of evidence is available.^{58, 59}

Table 2.

Frameworks for assessing quality of evidence.

Framework	Quality rating	Definition
AHRQ74	Good	Design and conduct of study addresses risk of bias with appropriate measurement of outcomes and analytic methods
	Fair	Do not meet criteria for good quality, no flaw likely to cause major bias, missing information often drives rating
	Poor	Inappropriate design, conduct, analysis, or reporting
GRADE54–56	High	Randomized trials; or double-upgraded observational studies.
	Moderate	Downgraded randomized trials; or upgraded observational studies.
	Low	Double-downgraded randomized trials; or observational studies.
	Very low	Triple-downgraded randomized trials; or downgraded observational studies; or case series/case reports.
NCCN75	High	Based upon factors of quality (e.g., trial design and how the results/observations were derives), quantity of data (e.g., number of trials, size of trials, clinical observations only), and consistency of data (e.g., similar or conflicting results across available studies of observations),
	Lower
	Any
SIGN14	1++	High-quality meta-analyses, systematic reviews of RCTs, or RCTs with a very low risk of bias
	1+	Well-conducted meta-analyses, systematic reviews of RCTs, or RCTs with a low risk of bias
	2++	High-quality systematic reviews of case-control or cohort studies; high-quality case-control or cohort studies with a very low risk of confound, bias, or chance and a high probability that the relationship is causal.
	2+	Well-conducted case-control or cohort studies with a low risk of confounding, bias, or chance and a moderate probability that the relationship is causal.
	2−	Case-control or cohort studies with a high risk of confounding, bias, or chance and a significant risk that the relationship is not causal
	3	Nonanalytic studies, e.g., case reports of case series
	4	Expert opinion
USPHS/IDSA76	I	Evidence from ≥1 properly randomized, controlled trial.
	II	Evidence for ≥1 well-designed clinical trial without randomization, from cohort or case-controlled analytic studies (preferable from >1 center), or from multiple time series or dramatic results from uncontrolled experiments.
	III	Evidence from opinions of respected authorities based on clinical experience, descriptive.

Abbreviations: AHRQ, Agency for Healthcare Research and Quality; GRADE, Grading of Recommendations Assessment, Development and Evaluation; NCCN, National Comprehensive Cancer Network; RCTs, randomized controlled trials; SIGN, Scottish Intercollegiate Guidelines Network; USPHS/IDSA, United States Public Health Service and Infectious Disease Society of America

Several tools are available for evaluating the risk of bias within a study and across studies. The Cochrane risk-of-bias tool may be used for RCTs,^{60, 61} ROBINS-I for non-randomized studies of interventions,⁶² and the Newcastle-Ottawa Scale for observational studies.^63–65 Case reports and small case series cannot be assessed for risk of bias.

Synthesize the data

All systematic reviews include a systematic presentation and synthesis of the findings from the studies. Software programs that facilitate the synthesis of data are included in Table 1. Developing a table with similar information for each study may identify patterns and lead to further thematic clustering of studies and tabulation of data.⁶⁶ Calculating the frequency of different types of results and/or outcomes is another way to create an initial description of the pattern of the included studies, but does not take into account the sample size, the precision level, or the size of the effect.^{67, 68}

If the quality of evidence permits, a quantitative synthesis of the data (meta-analysis) may be performed to evaluate effect-size more accurately.⁹ A meta-analysis combines results of separate RCTs or other high-quality studies using statistical methods.⁶⁹ Weighted study effect sizes are used to give greater weight to studies with higher precision.⁷⁰ Results are usually presented in the form of a forest plot (example shown in Supplementary Figure 1), which not only gives an overview of individual studies but also shows the degree to which they are statistically heterogeneous.⁷¹ If there is a statistical heterogeneity among the included studies, a subgroup analysis may be explored.⁷² Funnel plots are frequently used to detect potential publication bias where studies with negative results or no statistically significant findings are less likely to be published in peer reviewed journals (example shown in Supplementary Figure 2).

Including only randomized or quasi-randomized studies in a systematic review may be possible for a well-studied topic.⁷³ However, it still may be appropriate to perform a meta-analysis with few RCT⁶¹ and/or retrospective cohort studies if the quality of the studies is high.⁶⁵ However, pooling data from poor quality studies into a meta-analysis is not recommended because errors and biases from individual studies will be compounded, yielding misleading results and conclusions.

Discuss and conclude overall findings

Team member discussion is critical to reach valid conclusions and, if relevant, establish the strength of recommendation for an intervention. Multiple frameworks have been used in TCT reviews for assessing strength of a recommendation (Table 3). While GRADE is frequently used,^56,57 other frameworks for assessing the strength ofrecommendations may be considered based on the focus and scope of the review.^{14, 74–76}

Table 3.

Frameworks for assessing strength of recommendations.

Framework	Strength	Definition
AHRQ74	High	We are very confident that the estimate of effect lies close to the true effect for this outcome. The body of evidence has few or no deficiencies.
	Moderate	We are moderately confident that the estimate of effect lies close to the true effect for this outcome. The body of evidence has some deficiencies.
	Low	We have limited confidence that the estimate of effect lies close to the true effect for this outcome. The body of evidence has major or numerous deficiencies (or both).
	Insufficient	We have no evidence, we are unable to estimate an effect, or we have not confidence in the estimate of effect for this outcome. No evidence is available, or the body of evidence has unacceptable deficiencies, precluding reaching a conclusion.
GRADE54–56	Strong	When the desirable effects of an intervention clearly outweigh the undesirable effects, or clearly do not
	Weak	When the trade-offs are less certain - either because of low quality evidence or because evidence suggests that desirable and undesirable effects are closely balanced
NCCN75	1	Based on high-level of evidence, there is uniform NCCN consensus that the intervention is appropriate
	2A	Based on lower level of evidence, there is uniform NCCN consensus that the intervention is appropriate
	2B	Based on lower level of evidence, there is NCCN consensus that the intervention is appropriate
	C	Based on high-level of evidence, there is major NCCN disagreement that the intervention is appropriate
SIGN14	A	At least 1 meta-analysis, systematic review, or RCT rated as 1++ and directly applicable to the target population or a systematic review of RCTs or a body of evidence consisting principally of studies rated as 1+, directly applicable to the target population, and demonstrating overall consistency of results
	B	A body of evidence including studies rates as 2++, directly applicable to the target population, and demonstrating overall consistency of results of extrapolated evidence from studies rated as 1++ or 1+
	C	Body of evidence including studies rated as 2+, directly applicable to the target populations, and demonstrating overall consistence of results or extrapolated evidence from studies rated as 2++
	D	Evidence level 3 or 4 or extrapolated evidence from studies rated as 2+.
USPHS/IDSA76	A	Should always be offered.
	B	Should generally be offered.
	C	Evidence for efficacy is insufficient to support a recommendation for or against, or evidence for efficacy might not outweigh adverse consequences, or cost of the approach. Optional
	D	Moderate evidence for lack of efficacy or for adverse outcome supports a recommendation against use. Should generally not be offered.
	E	Good evidence for lack of efficacy or for adverse outcome supports a recommendation against use. Should never be offered.

Abbreviations: AHRQ, Agency for Healthcare Research and Quality; GRADE, Grading of Recommendations Assessment, Development and Evaluation; NCCN, National Comprehensive Cancer Network; SIGN, Scottish Intercollegiate Guidelines Network; USPHS/IDSA, United States Public Health Service and Infectious Disease Society of America

Factors that affect the strength of a recommendation in GRADE include the quality of the evidence, uncertainty about the balance between desirable and undesirable effects, uncertainty or variability in patient values and preferences, and uncertainty about whether the intervention represents a wide use of resources. Strong recommendations suggest that all or almost all individuals would choose that intervention, and therefore, it may not be necessary to present several therapeutic options to patients. Weak recommendations imply that there is likely to be important variation in the decision that informed individuals are likely to make, and that engaging patients and families in a shared decision-making process is essential. Recommendations are more likely to be weak rather than strong when the quality of evidence is low, when evidence suggests that desirable and undesirable consequences are closely balanced,⁵⁸ when there is substantial variation or uncertainty in patient values and preferences, and when interventions require considerable resources.

Prepare the manuscript

The PRISMA statement includes a 27-item checklist that covers the steps of a systematic review and meta-analysis that should be included with reviews submitted for peer review.⁷⁷ Other specialized guidelines and checklists that may be utilized depending on the review include ENhancing Transparency in REporting the synthesis of Qualitative research (ENTREQ),⁷⁸ Meta-Analysis Reporting Standards (MARS)⁷⁹ and Meta-analysis Of Observational Studies in Epidemiology (MOOSE).⁸⁰

Disseminate the review to enhance uptake of findings

The effective dissemination of study findings has a significant impact on how evidence-based findings are incorporated into clinical practice.^81,82 The study team should develop a plan early in the process for dissemination of their findings, including but not limited to, publication in a peer-reviewed journal. Once published, additional aspects of the dissemination plan should be executed, including presentations at conferences, promotion on social media, and distributing the findings in an appropriate format to all other relevant stakeholders, such as policy makers, payers, professional organizations, patients/caregivers and advocates, and hospital administrators. Dissemination of findings can be enhanced by ensuring that multidisciplinary stakeholders are involved from the beginning.

Approaches to Systematic Reviews when there is a Lack of High-Quality Evidence

The study team may consider alternative approaches when the amount or quality of evidence is too low to form conclusions. Scoping reviews are a valid method of summarizing the literature when the literature on a particular topic is very scarce or when reviewers want to address a broad research question.^{83, 84} This type of review entails the same searching process as a systematic review but does not require a formal evaluation of the quality of studies and risk of bias assessment.

Clinical questions regarding recently introduced therapeutic approaches may be addressed (e.g., clinical practice recommendations for chimeric antigen receptor T cell therapy ^85,86 or the literature summarized to serve as a basis for future research (e.g., conditioning intensity in obese patients).⁸⁷ Another option is to avoid a full formal systematic review but use the modified Delphi method for determining the degree of consensus among experts,⁸⁸ which may then be combined with a grading system for assigning a strength of the recommendation.^89–94

Conclusion

Systematic reviews are critical pieces of medical literature that summarize the totality of evidence to enable sound clinical and policy decisions and identify avenues for future research when applied to a single clinical research question of interest. Transparency of methods throughout their conduct enables reproducibility and a critical assessment of the results by and for all stakeholders. The best practices in methodologic frameworks reviewed here serve as a basis for the conduct of high-quality systematic reviews and meta-analyses in TCT and should be updated as they evolve. The use of evidence-based recommendations, informed by rigorous systematic reviews, can be a powerful approach to improve patients’ health outcomes.