Table 1.
Multi-rater reliability for reviewers rating of 30 studies of diagnostic reliability using QAREL
|
Item |
Item description (abbreviated) |
Subsequent evaluation |
|
|---|---|---|---|
| κ | 95% CI | ||
| 1 |
Was the sample of subjects representative? |
0.27 |
(0.11, 0.42) |
| 2 |
Was the sample of raters representative? |
0.59 |
(0.43, 0.74) |
| 3 |
Were raters blinded to the findings of other raters? |
0.92 |
(0.76, 1.00) |
| 4 |
Were raters blinded to their own prior findings? |
0.78 |
(0.62, 0.94) |
| 5 |
Were raters blinded to the accepted reference standard? |
0.66 |
(0.49, 0.82) |
| 6 |
Were raters blinded to clinical information not part of test |
0.51 |
(0.37, 0.64) |
| 7 |
Were raters blinded to additional non-clinical cues? |
0.59 |
(0.39, 0.78) |
| 8 |
Was the order of examination varied? |
0.71 |
(0.58, 0.84) |
| 9 |
Was the time interval between repeated measures appropriate? |
0.69 |
(0.50, 0.88) |
| 10 |
Was the test applied correctly and interpreted appropriately? |
0.35 |
(0.18, 0.51) |
| 11 | Were appropriate statistical measures of agreement used? | 0.73 | (0.54, 0.92) |
κ = multi-rater kappa. 95% CI = 95% confidence interval.