Table 4.
Risk of bias/ quality assessment tools for primary studies used by included systematic reviews
| Tool used | Number of reviews using approach | Number of reviews that used modifications | Originally developed for | Exposure assessment (original version) | Co-exposures (original version) | Confounding (original version) |
|---|---|---|---|---|---|---|
| Newcastle–Ottawa Scale (NOS) [86] | 9 [71, 77–84] | 2 [71, 81] | Evaluating non-randomized studies in systematic reviews | Yes (general)a | No | Yes |
| Mustafic et al. (modified NOS) [87] | 2 [79, 81] | 0 | Time-series and case-crossover studies of air pollution exposure | Yes | Possible (“long-term trends”) | Yes |
| Herzog et al. (modified NOS) [88] | 1 [84] | 0 | Cross-sectional studies (developed for vaccine-related knowledge, attitude, and behavior) | Yes (general)a | No | Yes |
| Modesti et al. (modified NOS) [89] | 1 [81] | 0 | Cross-sectional studies (developed for studies of blood pressure) | Yes (general)a | No | Yes |
| Office of Health Assessment and Translation (OHAT) [15, 66, 90] | 4 [73, 75, 79, 81] | 3 [73, 75, 81] | Systematic reviews and evidence integrations of environmental health research | Yes | Yes | Yes |
| Navigation Guide [5, 64] | 2 [76, 81] | 0 | Systematic reviews of human studies in environmental health | Yes | No§ | Yes |
| ACROBAT-NRSI [91] (later ROBINS-I) (97) | 1 [72] | 1 [72] | Non-randomized studies of interventions | Yes (“measurement of intervention”) | Yes (“co-interventions”) | Yes |
| Agency for Healthcare Research and Quality (AHRQ) [62] | 1 [78] | 0 | Systematic reviews of studies of healthcare interventions | Yes (general)a | Yes (“co-interventions”) | Yes |
| Cochrane tool for RCTs (ROB 1) [92] | 1 [78] | 0 | Systematic reviews of individual RCTs | No | No | Yes (randomization) |
| Hoy et al. [93] | 1 [74] | 0 | Prevalence studies of low back and neck pain | No | No | No |
| ROBINS-E (preliminary version) [94] | 1 [74] | 0 | Non-randomized studies of exposures | Yes | Yes | Yes |
| Centre for Evidence Based-Medicine (CEBM) [95, 96] | 1 [70] | Unclear (insufficiently reported) | Clinical decision-making | Yes (general)a | No | Yes |
| Scottish Intercollegiate Guidelines Network (SIGN) [97] | 1 [69] | 1 [69] | Guideline development of clinical care (all study designs) | Yes (general)a | No | Yes |
| Own criteria [5, 68] | 2 [5, 68] | N/A | N/A | Yes | No | Yes |
Abbreviations: ACROBAT-NRSI A Cochrane Risk Of Bias Assessment Tool: for Non-Randomized Studies of Interventions, AHRQ Agency for Healthcare Research and Quality, CEBM Centre for Evidence Based-Medicine, NOS Newcastle Ottawa Scale, OHAT Office of Health Assessment and Translation, ROB Risk of bias, ROBINS-E Risk of bias in non-randomized studies of exposure, ROBINS-I Risk of bias in non-randomized studies of interventions, SIGN Scottish Intercollegiate Guidelines Network
aYes (general) refers to tools that include a criterion relating to the validity of the exposure assessment method, without clear relevance to environmental exposures
§In other case studies (e.g., on flame retardant exposure) other pollutants were considered under the confounding domain, but not in the version considered herein