Table 3.
Assessment of the quality of included network meta-analyses using the ISPOR checklist
| Question | Woolacott 2006 [35] | Reich 2008 [32] | Bansback 2009 [31] | Reich 2012 [39] | Lin 2012 [43] | Gupta 2014 [19] | Messori 2015 [18] | Signorovitch 2015 [42] | Gomez Garcia 2016 [23] | Jabbar-Lopez 2017 [17] | Sbidian 2017 [34] | Geng 2018 [22] | Loos 2018 [24] |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Did the researchers attempt to identify and include all relevant randomized controlled trials? (RCTs) | ✓ | ✓ | ✓ | ✓ | ✓ | ✗ | ✗ | ✓ | ✗ | ✓ | ✓ | ✗ | ✗ |
| Do trials for interventions of interest form one connected network of RCTs? | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ |
| Is it apparent that poor quality studies were included, thereby leading to bias? | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | ✓ | ✗ |
| Is it likely that bias was induced by selective reporting of outcomes in the studies? | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ |
| Are there systematic differences in treatment effect modifiers across the different treatment comparisons in the network? | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | ? | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ |
| Were imbalances in effect modifiers across the different treatment comparisons identified before reviewing individual study results? | ✗ | ✗ | ✗ | ✗ | ✓ | ✓ | ✗ | ✓ | ✗ | ✗ | ✓ | ✗ | ✓ |
| Were statistical methods used that preserve within-study randomization? | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ |
| If both direct and indirect comparisons are available for pairwise contrasts was agreement evaluated or discussed? | N/A | N/A | ✗ | ✗ | ✓ | ✓ | ✗ | ✗ | ✓ | ✓ | ✓ | ✓ | ✗ |
| In the presence of consistency, were both direct and indirect evidence included in the network meta-analysis? | NI | ✓ | NI | NI | ✓ | ✓ | NI | NI | ✓ | ✓ | ✓ | ✓ | NI |
| With an imbalance in the distribution of treatment effect modifiers across the different types of comparisons in the network of trials, did researchers attempt to minimize this bias with the analysis? | N/A | N/A | N/A | N/A | ✓ | N/A | NI | ✓ | NI | NI | ✓ | NI | ✓ |
| Was a valid rationale provided for the use of RE or FE models? | ✓ | ✓ | ✗ | ✗ | ✓ | ✗ | ✓ | ✗ | ✗ | ✗ | ✓ | ✗ | ✓ |
| If a RE model was used, were assumptions about heterogeneity explored or discussed? | ✗ | ✗ | ✗ | ✗ | ✓ | ✓ | N/A | ✓ | ✗ | ✗ | ✓ | ✗ | ✓ |
| Is a graphical or tabular representation of the evidence network provided with information on the number of RCTs per direct comparison? | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ |
| Are the individual study results reported? | ✓ | ✓ | ✓ | ✓ | P | ✗ | ✓ | ✓ | ✗ | ✓ | ✓ | ✓ | ✓ |
| Are direct results reported separately from results of indirect comparisons or network meta-analyses? | P | ✓ | ✗ | ✗ | P | P | ✗ | ✗ | ✓ | ✓ | ✓ | P | ✗ |
| Are all pairwise contrasts between interventions as obtained with NMA reported along with measure of uncertainty? | ✗ | ✗ | ✗ | ✗ | P | ✓ | ✓ | P | ✓ | ✓ | ✓ | ✓ | ✓ |
| Is a ranking of interventions provided given the reported treatment effects and its uncertainty outcome? | ✗ | ✗ | ✗ | ✓ | P* | P* | ✓ | ✗ | ✓ | ✓ | ✓ | P* | ✗ |
| Is the impact of important patient characteristics on treatment effects reported? | ✗ | ✗ | ✗ | ✓ | ✓ | ✗ | ✗ | ✓ | ✗ | ✗ | ✓ | ✗ | ✗ |
| Question | Sawyer 2018 (i) [36] | Sawyer 2018 (m) [40] | Cameron 2018 [38] | Sawyer 2019 [37] | Bai 2019 [21] | Xu 2019a [25] | Xu 2019b [26] | Warren 2019 [41] | Wu 2020 [44] | Sbidian 2020 [33] | Warren 2020 [20] | Armstrong 2020 [30] |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Did the researchers attempt to identify and include all relevant randomized controlled trials? (RCTs) | ✓ | ✓ | ✓ | ✓ | ✗ | ✗ | ✗ | ✓ | ✓ | ✓ | ? | ✓ |
| Do trials for interventions of interest form one connected network of RCTs? | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | P | ✓ | ✓ | ✓ |
| Is it apparent that poor quality studies were included, thereby leading to bias? | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | ✓ | ✗ | ✗ | ✗ |
| Is it likely that bias was induced by selective reporting of outcomes in the studies? | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ |
| Are there systematic differences in treatment effect modifiers across the different treatment comparisons in the network? | ✓ | ✓ | ✓ | ✓ | ✗ | ✓ | ✗ | ✓ | ? | ✓ | ? | ✓ |
| Were imbalances in effect modifiers across the different treatment comparisons identified before reviewing individual study results? | ✓ | ✓ | ✓ | ✓ | ✗ | ✗ | ✗ | ✗ | ✗ | ✓ | ✗ | ✓ |
| Were statistical methods used that preserve within-study randomization? | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | N/A | ✓ | ✓ | ✓ |
| If both direct and indirect comparisons are available for pairwise contrasts was agreement evaluated or discussed? | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✗ | ✓ | ✓ | ✗ | ✗ |
| In the presence of consistency, were both direct and indirect evidence included in the network meta-analysis? | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | NI | ✓ | ✓ | NI | NI |
| With an imbalance in the distribution of treatment effect modifiers across the different types of comparisons in the network of trials, did researchers attempt to minimize this bias with the analysis? | ✓ | ✓ | ✓ | ✓ | NI | NI | NI | NI | NI | ✓ | NI | ✓ |
| Was a valid rationale provided for the use of RE or FE models? | ✓ | ✓ | ✓ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ |
| If a RE model was used, were assumptions about heterogeneity explored or discussed? | ✓ | ✓ | ✓ | ✓ | ✗ | ✗ | NR | N/A | ✗ | ✓ | NR | ✓ |
| Is a graphical or tabular representation of the evidence network provided with information on the number of RCTs per direct comparison? | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✗ | ✓ | ✓ | ✓ |
| Are the individual study results reported? | ✓ | ✓ | ✗ | ✓ | ✓ | ✗ | ✓ | ✓ | ✓ | ✓ | ✗ | ✓ |
| Are direct results reported separately from results of indirect comparisons or network meta-analyses? | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | ✓ | ✗ | ✓ | ✓ | ✗ | ✗ |
| Are all pairwise contrasts between interventions as obtained with NMA reported along with measure of uncertainty? | ✗ | ✓ | ✓ | ✓ | ✓ | ✓ | P | ✗ | ✓ | ✓ | ✗ | ✗ |
| Is a ranking of interventions provided given the reported treatment effects and its uncertainty outcome? | ✗ | ✓ | ✓ | ✗ | ✓ | ✓ | ✓ | ✗ | ✓ | ✓ | P* | ✗ |
| Is the impact of important patient characteristics on treatment effects reported? | ✓ | ✓ | ✓ | ✓ | ✗ | ✗ | ✓ | ✗ | ✗ | ✓ | ✗ | ✓ |
N/A not applicable, NI not investigated, NR not reported, P partly, ? unclear; * no uncertainty reported, ✗ no; ✓ yes