Table 2. Comparison of raters’ judgments for one rater using SAQAT and one rater using the standard GRADE approach (before and after SAQAT amendment).
| Quality Rating domain | Weighted Kappa (95% CI) | Magnitude of Agreement | ||
|---|---|---|---|---|
| Before amendment | After amendment | Before amendment | After amendment | |
| Risk of bias | 0.13 (-0.22 to 0.48) | 0.48 (0.01 to 0.94) | Slight | Moderate |
| Inconsistency | 0.08 (-0.01 to 0.42) | 0.30 (0 to 0.84) | Slight | Fair |
| Imprecision | 0.47 (0.05 to 0.90) | 0.47 (0.05 to 0.90) | Moderate | Moderate |
| Indirectness | Not estimable 1 | Not estimable 1 | N/A | N/A |
| Publication bias | Not estimable 2 | Not estimable 2 | N/A | N/A |
| Overall judgment | 0.11 (-0.34 to 0.47) | 0.35 (-0.09 to 0.87) | Slight | Fair |
1 One discrepancy out of 15 judgments (93% agreement)
2 No discrepancies but only one review was rated as publication bias suspected (100% agreement)