Skip to main content
. 2018 Mar 23;2018(3):CD010840. doi: 10.1002/14651858.CD010840.pub2

1. Criteria for 'Risk of bias' judgement.

Criteria Description
Random sequence generation (selection bias ‐ biased allocation to interventions ‐ due to inadequate generation of a randomised sequence)
  1. Criteria for a judgement of low risk of bias: the investigators describe a random component in the sequence generation process such as:

    1. referring to a random number table;

    2. using a computerised random number generator;

    3. coin tossing;

    4. shuffling cards or envelopes;

    5. throwing dice; or

    6. drawing of lots and minimisation.

  2. Criteria for a judgement of high risk of bias: the investigators describe a non‐random component in the sequence generation process. Usually the description would involve some systematic, non‐random approach, for example:

    1. sequence generated by odd or even date of birth;

    2. sequence generated by some rule based on date (or day) of admission;

    3. sequence generated by some rule based on hospital or clinic record number. Other non‐random approaches happen much less frequently than the systematic approaches mentioned above and tend to be obvious, for example: allocation by judgement of the clinician; allocation by preference of the participant; allocation based on the results of a laboratory test or a series of tests and allocation by availability of the intervention.

  3. Criteria for a judgement of unclear risk of bias: insufficient information about the sequence generation process to permit a judgement of low or high risk of bias.

Allocation concealment (selection bias ‐ biased allocation to interventions ‐ due to inadequate concealment of allocations prior to assignment)
  1. Criteria for a judgement of low risk of bias: participants and investigators enrolling participants could not foresee assignment because one of the following, or an equivalent method, was used to conceal allocation:

    1. central allocation;

    2. sequentially numbered drug containers of identical appearance; or

    3. sequentially numbered, opaque, sealed envelopes.

  2. Criteria for a judgement of high risk of bias: participants or investigators enrolling participants could possibly foresee assignments and thus introduce selection bias such as allocation based on:

    1. an open random allocation schedule (for example, a list of random numbers);

    2. assignment of envelopes without appropriate safeguards (for example, if envelopes were unsealed or non­opaque or not sequentially numbered);

    3. alternation or rotation;

    4. date of birth;

    5. case record number; or

    6. any other explicitly unconcealed procedure.

  3. Criteria for a judgement of unclear risk of bias: insufficient information to permit a judgement of low or high risk of bias. This is usually the case if the method of concealment is not described or not described in sufficient detail to allow a definite judgement; for example, if the use of assignment envelopes is described but it is unclear whether envelopes were sequentially numbered, opaque and sealed.

Blinding of participants and personnel (performance bias due to knowledge of the allocated interventions by participants and personnel during the study)
  1. Criteria for a judgement of low risk of bias may involve any one of the following:

    1. no blinding or incomplete blinding, but the review authors judge that the results are unlikely to be influenced by lack of blinding; or

    2. blinding of participants and key study personnel ensured, and it is unlikely that the blinding could have been broken.

  2. Criteria for a judgement of high risk of bias may involve any one of the following:

    1. no blinding or incomplete blinding, and the results are likely to be influenced by lack of blinding; or

    2. blinding of key study participants and personnel attempted, but it is likely that the blinding could have been broken, and the results are likely to be influenced by lack of blinding.

  3. Criteria for a judgement of unclear risk of bias may involve any one of the following:

    1. insufficient information to permit a judgement of low or high risk of bias; or

    2. the study did not address this outcome.

Blinding of outcome assessment (detection bias due to knowledge of the allocated interventions by outcome assessors)
  1. Criteria for a judgement of low risk of bias may involve any one of the following:

    1. no blinding of outcome or outcome assessment (or both), but the review authors judge that the outcome and its measurement are unlikely to be influenced by lack of blinding; or

    2. blinding of outcome and outcome assessment ensured, and it is unlikely that the blinding could have been broken.

  2. Criteria for a judgement of high risk of bias may involve any one of the following:

    1. no blinding of outcome or outcome assessment (or both), and the outcome and its measurement are likely to be influenced by lack of blinding; or

    2. blinding of outcome or assessment (or both), but it is likely that the blinding could have been broken, and the outcome and its measurement are likely to be influenced by lack of blinding.

  3. Criteria for a judgement of unclear risk of bias may involve any one of the following:

    1. insufficient information to permit a judgement of low or high risk of bias; or

    2. the study did not address this outcome.

Incomplete outcome data (attrition bias due to the amount, nature or handling of incomplete outcome data)
  1. Criteria for a judgement of low risk of bias may involve any one of the following:

    1. no missing outcome data;

    2. reasons for missing outcome data are unlikely to be related to true outcome (for survival data, censoring unlikely to be introducing bias);

    3. missing outcome data balanced in numbers across intervention groups, with similar reasons for missing data across groups;

    4. for dichotomous outcome data, the proportion of missing outcomes compared with the observed event risk is not enough to have a clinically relevant impact on the intervention effect estimate;

    5. for continuous outcome data, the plausible effect size (difference in means or standardised difference in means) among missing outcomes is not enough to have a clinically relevant impact on the observed effect size; or

    6. missing data have been imputed using appropriate methods.

  2. Criteria for a judgement of high risk of bias may involve any one of the following:

    1. the reason for missing outcome data is likely to be related to true outcome, with either imbalance in numbers or reasons for missing data across intervention groups;

    2. for dichotomous outcome data, the proportion of missing outcomes compared with the observed event risk is enough to induce clinically relevant bias in the intervention effect estimate;

    3. for continuous outcome data, the plausible effect size (difference in means or standardised difference in means) among missing outcomes is enough to induce clinically relevant bias in the observed effect size;

    4. 'as‐treated' analysis done with substantial departure of the intervention received from that assigned at randomisation; or

    5. potentially inappropriate application of simple imputation.

  3. Criteria for a judgement of unclear risk of bias may involve any one of the following:

    1. insufficient reporting of attrition or exclusions (or both) to permit a judgement of low or high risk of bias (for example, number randomised not stated, no reasons for missing data provided); or

    2. the study did not address this outcome.

Selective reporting (reporting bias due to selective outcome reporting)
  1. Criteria for a judgement of low risk of bias may involve any of the following:

    1. the study protocol is available and all of the study's pre‐specified (primary and secondary) outcomes that are of interest in the review have been reported in the pre‐specified way; or

    2. the study protocol is not available but it is clear that the published reports include all expected outcomes, including those that were pre‐specified (convincing text of this nature may be uncommon).

  2. Criteria for a judgement of high risk of bias may involve any one of the following:

    1. not all of the study's pre‐specified primary outcomes have been reported;

    2. one or more primary outcomes is reported using measurements, analysis methods or subsets of the data (for example, subscales) that were not pre‐specified;

    3. one or more reported primary outcomes were not pre‐specified (unless clear justification for their reporting is provided, such as an unexpected adverse effect);

    4. one or more outcomes of interest in the review were reported incompletely so that they could not be entered in a meta‐analysis; or

    5. the study report failed to include results for a key outcome that was expected to have been reported for such a study.

  3. Criteria for a judgement of unclear risk of bias: insufficient information to permit a judgement of low or high risk of bias. It was likely that most studies would fall into this category.

Other bias (other sources of bias not captured by the other domains)
  1. Criteria for a judgement of low risk of bias: the study appears to be free of other sources of bias.

  2. Criteria for a judgement of high risk of bias: at least one important risk of bias exists. For example, the study:

    1. had a potential source of bias related to the specific study design used; or

    2. has been accused of bring fraudulent; or

    3. had some other problem.

  3. Criteria for a judgement of unclear risk of bias: there is a risk of bias, but there is either:

    1. insufficient information to assess whether an important risk of bias exists; or

    2. insufficient rationale or evidence that an identified problem introduced bias.