Various biases in systematic review and meta-analysis and their assessment

Abstract

Systematic reviews and meta-analyses synthesise existing evidence for clinical and scientific decision-making. Clinicians and researchers need to understand how to interpret the evidence with the bias involved, as well as the various available tools and how to use them when performing a systematic review. The validity of systematic reviews and meta-analyses depends on comprehensive assessment and mitigation of biases. Biases have the potential to compromise the validity and reliability of results. By being aware of and addressing these different biases, researchers and clinicians can more confidently interpret findings and increase the impact and dependability of their conclusions. The article discusses the biases involved in systematic reviews and meta-analyses and various ways to assess them.

INTRODUCTION

Systematic reviews and meta-analyses (SRMA) synthesise existing evidence to inform clinical and scientific decision-making. Biases have the potential to compromise the validity and reliability of results. Accurately interpreting systematic reviews and meta-analyses requires understanding these biases and how to evaluate them.[1] The present article describes the commonly encountered biases in SRMA and the methods utilised for risk-of-bias (RoB) assessment.

SELECTION BIAS

Randomisation is an essential aspect of a randomised controlled trial (RCT). A fundamental difference between the patients included in a study’s treatment arms as a result of the way the patients were assigned to the treatment groups is the best way to characterise selection bias. The random sequence generation techniques and allocation concealment of the RCT must be considered to evaluate selection bias.[2] A truly random sequence for treatment allocation means that the baseline characteristics of both groups will be naturally balanced. Sequence generation refers to how patients were randomised to the treatment groups. However, a bias in how the allocation is performed could result in differences between the comparison groups. These methods may allow clinicians to choose which treatment patients will receive within the study based on their experience and expertise (i.e. a non-random factor). Meticulous allocation concealment could prevent anyone from being able to predict or deduce patient allocations.

ATTRITION BIAS

There is always a chance that patients who have been enrolled in an RCT will leave the study before finishing their follow-ups. Attrition bias occurs when the subjects withdraw from the study, thus affecting the study’s results.[3] The withdrawal imbalance may impact the study’s findings and conclusions if a reason for withdrawal is present or more common in the comparison groups. Results would be affected if a particular patient group within one of the comparison groups had a greater probability of discontinuing the study.

DETECTION BIAS

Detection bias refers to the possibility of differences between the comparison groups regarding how the outcomes are measured or assessed.[4] To reduce detection bias, the outcome assessor should be masked. This ensures that the methods used to measure an outcome are the same for all study participants, meaning that the outcome measurement is consistent across patients assigned to the comparison groups.

REPORTING BIAS

When reservations arise about the findings presented in a study’s results, reporting bias may arise.[5] This usually occurs when a study reports significant outcome findings while leaving out non-significant outcome findings. This emphasises the significance of a predetermined study protocol that specifies all outcomes that will be evaluated, even though it can be challenging to identify.

PUBLICATION BIAS

Publication bias occurs when the published literature, which frequently favours studies with positive or noteworthy findings, does not represent all research on a topic.[6] Positive results are more likely to be published in journals, which causes the effects of interventions to be inflated. Statistical tests such as Egger’s testplots the effect estimates from individual studies against their standard errors, and can identify asymmetry in study effects using funnel plots to evaluate publication bias.

TOOLS FOR ASSESSING RISK OF BIAS (ROB)

The most commonly used RoB tool for RCTs is the Cochrane RoB tool using the RevMan programme. The Rob is represented graphically as a RoB graph (traffic-light plot) or a RoB summary (summary plot).[7] Table 1 mentions the various domains included in the Cochrane RoB tool. The RoB-2 tool was developed later to address a few concerns with the Cochrane RoB tool.[8] Table 1 mentions the various domains included in the RoB-2 tool.

Table 1.

Various domains for different RoB tools

Cochrane RoB tool	Revised RoB tool (RoB2)	ROBINS-I tool	ROBINS-E tool
Sequence generation (selection bias)	Bias arising from the randomisation process	Bias due to confounding and selection of participants (pre-intervention stage)	Bias due to confounding
Allocation sequence concealment (selection bias)	Bias due to deviations from intended interventions	Bias in the classification of interventions (intervention stage)	Bias due to selection
Blinding of personnel and participants (performance bias)	Bias due to missing outcome data	Bias due to deviations from the intended intervention (post-intervention stage)	Bias due to classification
Blinding of outcome assessment (detection bias)	Bias in the measurement of the outcome	Bias due to missing data (post-intervention stage)	Bias due to departures
Incomplete outcome data (attrition bias)	Bias in the selection of the reported result	Bias due to measurement of outcomes (post-intervention stage)	Bias due to missing data
Selective outcome reporting (reporting bias)	Overall bias	Bias due to the selection of the reported result (post-intervention stage)	Bias due to measurement
Other potential sources of bias			Bias due to reporting

RoB=risk of bias, ROBINS-I=Risk of Bias In Non-randomised Studies of Interventions, ROBINS-E=Risk of Bias In Non-randomised Studies - of Exposures

A study is deemed to have high RoB if any domain suggests high risk. If the study is found to have some concerns across many domains, it significantly reduces the confidence in the findings. A study is deemed to have a low RoB if the bias is low across all domains. Even though there is no high RoB in any domain, the study is thought to raise concerns in at least one of them for this result. The results of RoB are declared low risk, high risk, and no information (if the methodology is unclear).

ROBINS-I (Risk of Bias In Non-randomised Studies of Interventions) is a tool used to assess the RoB in estimates of an intervention’s safety (benefit or harm) or effectiveness from research that did not assign interventions using randomisation.[9] The various domains included in the ROBINS-I tool are mentioned in Table 1.

ROBINS-E (Risk Of Bias In Non-randomised Studies - of Exposures) is a tool for assessing the RoB in observational epidemiological studies.[10] It is based on ROBINS-I. ROBINS-E evaluates seven domains of bias: confounding, selection, classification, departures, missing data, measurement, and reporting [Table 1].

For the assessment of bias, two authors perform the RoB assessment for all the studies that fulfil the inclusion criteria. Suppose there are discrepancies (e.g. a difference of opinion, such as high bias by one author and no information or unclear bias/low bias by another author). In that case, a third author must take a final call to declare the category of bias involved.

A traffic light plot is a visual representation of RoB in the results of the systematic review. It uses the colour scheme of traffic lights (red, green, yellow, and blue) to display each result’s domain-level judgments. The distribution of RoB assessments within each bias domain is displayed in a summary plot (weighted bar plot). The height of the bars represents the proportion of studies with low, unclear, or high RoB.

A web application called ‘robvis’ can be used to visualise RoB analyses carried out as a component of a systematic review. The tool generates weighted bar plots of the distribution of RoB judgments within each bias domain and ‘traffic light’ plots of the domain-level judgments for each result.[11] The final RoB for each domain is entered in an Excel sheet, which is then uploaded to the web app to generate the plots. Figure 1a and b represents a hypothetical RoB assessment using a robvis tool. From the robvis tool web app, the required RoB plot can be selected based on the type of review. Figure 1a and b shows hypothetical RoB diagrams (traffic light plot and summary plot).

Figure 1.

(a) RoB for studies ABC, DEF, GHI, MON, and PQR is high (depicted with ‘X’ in red circle), and RoB is low for the study JKL (depicted with ‘+’ in green circle). (b) Summary of risk for bias for various domains. As seen in red for the overall RoB (occupying ~ 80% of the area), the RoB is very high). RoB = Risk of Bias, ABC/DEF/GHI/MON/PQR = hypothetical studies for the purpose of explaining bias

Many other tools have been used and validated for RoB assessment and quality of study assessment, with some variations in the domains and scoring. The choice of the RoB tool depends on the planned, systematic review and is also at the researchers’ discretion. These tools are summarised in Table 2.

Table 2.

Summary of Jadad Score, NOS, and QUADAD-2 tool

Jadad score	NOS		QUADAS-2
	Cohort studies	Case-control studies
a. Was the study described as randomised? (Yes - 1, No - 0) b. Was the method of randomisation appropriate? (Yes - 1, No - 1, Not described - 0) c. Was the study described as blinding? (Yes - 1, No - 0) d. Was the method of blinding appropriate? (Yes - 1, No - 1, Not described - 0) e. Was there a description of withdrawals and dropouts? (Yes - 1, No - 0) f. Was there a clear description of the inclusion/exclusion criteria? (Yes - 1, No - 0) g. Was the method used to assess adverse effects described? (Yes - 1, No - 0) h. Was the methods of statistical analysis described? (Yes - 1, No - 0)	Selection: • Representative of the cohort • Selection of non-intervention cohort • Ascertainment of Intervention • Demonstration of the presence of outcome before the beginning of the study (yes/no) Comparability: Comparability of cohorts Outcome: • Assessment of outcome • Duration of outcome (Ex, more or less than 6 months) • Adequate follow-up	Selection: • Case definition • Representation of cases • Selection of control • Definition of control Comparability: Comparability of cases and control Exposure: • Ascertainment of exposure • Same method of exposure for cases and control • Non-response rate	Domains: • Patient selection • Index test • Reference standard • Fow and timing (RoB assessment is done for each domain)

NOS=Newcastle-Ottawa scale, QUADAS=Quality assessment of diagnostic accuracy studies, RoB=Risk of bias

Jadad score is a system that uses eight criteria to assess the methodological quality of a clinical trial.[12] High Jadad scores indicate better studies. Scores of 3–5 usually indicate a high-quality trial.

Newcastle-Ottawa scale (NOS) is an eight-item instrument used to assess the quality of studies in clinical trials. NOS is divided into three categories: selection, comparability, and exposure or outcome for cohort or case-control studies.[13] An outcome is investigated in a cohort study, and an exposure is conducted in a case-control study. It provides a rating system using star (*) ranging from 0 to 9, with scores equal to or greater than seven considered high quality and less than seven regarded as low quality. A study may receive up to one star for every item in the selection and exposure categories. For comparability, a maximum of two stars may be awarded. Studies are categorised as good, fair, or poor based on the stars in the three dimensions.

Quality Assessment of Diagnostic Accuracy Studies (QUADAS) tool was developed for systematic reviews of studies on diagnostic accuracy, which was later revised into the QUADAS-2 tool.[14] The four domains of QUADAS-2 are patient selection, index test, reference standard, flow, and timing. The RoB is evaluated for each domain, and signalling questions help judge RoB. The QUADAS-2 tool is used in four stages: summarise the review question, tailor the tool and produce review-specific guidance, construct a flow diagram, and evaluate bias and applicability.

CONCLUSION

SRMAs are effective instruments in evidence-based practice. However, their validity depends upon the meticulous evaluation and reduction of biases. By being aware of and addressing these different biases, researchers and clinicians can more confidently interpret findings and increase the impact and dependability of their conclusions.

Conflicts of interest

There are no conflicts of interest.

Funding Statement

Nil.