Table 1.
Items, response choices, and inter-rater reliability estimates for the Evidence Project risk of bias tool
| Risk of bias tool domains | Items | Response choices | Kappa (κ)1 | |
|---|---|---|---|---|
| Dichotomous | Categorical | |||
| Study design | 1: Cohort | Yes, No | 0.48 | |
| 2: Control or comparison group | Yes, No | 0.80 | ||
| 3: Pre/post intervention data | Yes, No | 0.74 | ||
| Participant representativeness | 4: Random assignment of participants to the intervention | Yes, No, NA | 0.78 | 0.56 |
| 5: Random selection of participants for assessment | Yes, No | 0.41 | ||
| 6: Follow-up rate of 80% or more | Yes, No, NA, NR | 0.67 | 0.55 | |
| Equivalence of comparison groups | 7: Comparison groups equivalent on sociodemographics | Yes, No, NA, NR | 0.65 | 0.56 |
| 8: Comparison groups equivalent at baseline on outcome measures | Yes, No, NA, NR | 0.59 | 0.50 | |
| Median kappa score (κ) across individual items 1–8 | 0.66 | |||
| Weighted kappa (κw) of the total count of items (sum of individual item dichotomous responses) | 0.66 | |||
1Kappa estimates are reported for dichotomous (Yes, No) and categorical ratings when appropriate. Categorical response sets further classify binary No ratings as: No (reported), NA (not applicable), NR (not reported). Agreement was categorized as poor (0.00), slight (0.01–0.20), fair (0.21–0.40), moderate (0.41–0.60), substantial (0.61–0.80), or almost perfect (0.81–1.00)