Table 2.
How risk of bias assessments using ROBINS-I were conducted and reported in the included systematic reviews
| AMSTAR 2 confidence rating |
|||||
|---|---|---|---|---|---|
| Characteristic | Overall, N = 124 | High, N = 7 | Moderate, N = 21 | Low, N = 41 | Critically low, N = 55 |
| ROBINS-I judgments reported | |||||
| Overall and domain-specific RoB judgments for all studies | 82 (66%) | 7 (100%) | 15 (71%) | 27 (66%) | 33 (60%) |
| Domain-specific RoB judgments for all studies | 19 (15%) | 0 (0%) | 3 (14%) | 5 (12%) | 11 (20%) |
| Overall RoB judgments for all studies | 15 (12%) | 0 (0%) | 2 (10%) | 7 (17%) | 6 (11%) |
| Incomplete or aggregated RoB judgments only | 2 (2%) | 0 (0%) | 0 (0%) | 0 (0%) | 2 (4%) |
| No RoB judgments | 6 (5%) | 0 (0%) | 1 (5%) | 2 (5%) | 3 (5%) |
| Reported justifications for individual ROBINS-I judgments | |||||
| Overall and domain-specific RoB judgments for all studies | 6 (5%) | 2 (29%) | 1 (5%) | 3 (7%) | 0 (0%) |
| Domain-specific RoB judgments for all studies | 5 (4%) | 0 (0%) | 0 (0%) | 2 (5%) | 3 (5%) |
| Overall RoB judgments for all studies | 3 (2%) | 0 (0%) | 0 (0%) | 3 (7%) | 0 (0%) |
| No justifications reported | 110 (89%) | 5 (71%) | 20 (95%) | 33 (80%) | 52 (95%) |
| Reporting of ROBINS-I pre-assessment stage | |||||
| Confounding domains listed | 5 (4%) | 0 (0%) | 4 (19%) | 1 (2%) | 0 (0%) |
| Confounding domains and co-interventions listed | 3 (2%) | 1 (14%) | 1 (5%) | 1 (2%) | 0 (0%) |
| No reporting of pre-assessment stage | 116 (94%) | 6 (86%) | 16 (76%) | 39 (95%) | 55 (100%) |
| Scale used to rate RoB | |||||
| Standard scale (low, moderate, serious, critical) | 93 (75%) | 5 (71%) | 16 (76%) | 31 (76%) | 41 (75%) |
| Non-standard: 3 levels (e.g., low, moderate, high) | 14 (11%) | 1 (14%) | 2 (10%) | 6 (15%) | 5 (9%) |
| Non-standard: 2 levels (e.g., low, high) | 9 (7%) | 1 (14%) | 2 (10%) | 2 (5%) | 4 (7%) |
| Non-standard: other (e.g., yes, probably yes, probably no, no) | 2 (2%) | 0 (0%) | 0 (0%) | 0 (0%) | 2 (4%) |
| None | 6 (5%) | 0 (0%) | 1 (5%) | 2 (5%) | 3 (5%) |
| Explicitly incorporated RoB into evidence synthesis | |||||
| Yes | 38 (31%) | 6 (86%) | 13 (62%) | 13 (32%) | 6 (11%) |
| No | 86 (69%) | 1 (14%) | 8 (38%) | 28 (68%) | 49 (89%) |
| How RoB was incorporated into evidence synthesis (N = 38) | |||||
| Discussed RoB in narrative synthesis | 28 (74%) | 5 (83%) | 11 (85%) | 9 (69%) | 3 (50%) |
| Conducted sensitivity analysis or subgroup analysis | 10 (26%) | 3 (50%) | 3 (23%) | 3 (23%) | 1 (17%) |
| Excluded studies at high RoB | 7 (18%) | 1 (17%) | 2 (15%) | 2 (15%) | 2 (33%) |
| Deviations from ROBINS-I guidance | |||||
| Modified the rating scale | 25 (20%) | 2 (29%) | 4 (19%) | 8 (20%) | 11 (20%) |
| Modified the bias domains | 8 (6%) | 0 (0%) | 1 (5%) | 2 (5%) | 5 (9%) |
| Assigned an overall RoB judgment lower than the highest-rated bias domaina | 18 (20%) | 0 (0%) | 5 (31%) | 7 (23%) | 6 (17%) |
| Included critical-RoB studies any synthesis | 24 (19%) | 2 (29%) | 7 (33%) | 10 (24%) | 5 (9%) |
| Included critical-RoB studies in meta-analysis | 9 (7%) | 1 (14%) | 2 (10%) | 2 (5%) | 4 (7%) |
| Applied ROBINS-I to a non-interventional research question | 14 (11%) | 2 (29%) | 3 (14%) | 1 (2%) | 8 (15%) |
RoB, risk of bias.
a
Excluding studies that did not report both overall and per-domain RoB judgements. N=90