Table 6.
Effect of guidelines on interrater reliability. Agreement metrics (mean and SD) before and after introducing structured guidelines for 3 tasks. Δ indicates the change. All improvements were statistically significant (P<.001).
| Task/method | QWKa | Agreement rate (%) | ICCb (95% CI) | ||
|
|
|
Pairwise | Complete |
|
|
| Finger-tapping | |||||
|
|
Before guideline | 0.46 (0.10) | 62.5 (5.1) | 47.8 (6.8) | 0.51 (0.30-0.69) |
|
|
After guideline | 0.89 (0.05) | 92.7 (3.1) | 89.9 (4.1) | 0.90 (0.78-0.98) |
|
|
Δ | 0.43 (0.10) | 30.2 (5.1) | 42.1 (6.9) | 0.38 |
| Smile | |||||
|
|
Before guideline | 0.61 (0.08) | 70.5 (4.9) | 57.7 (7.0) | 0.68 (0.53-0.80) |
|
|
After guideline | 0.84 (0.04) | 78.5 (4.4) | 67.8 (6.6) | 0.84 (0.76-0.91) |
|
|
Δ | 0.23 (0.10) | 8.1 (6.7) | 10.1 (9.6) | 0.16 |
| Speech | |||||
|
|
Before guideline | 0.64 (0.11) | 81.9 (4.3) | 73.9 (6.2) | 0.65 (0.32-0.84) |
|
|
After guideline | 0.90 (0.05) | 91.9 (3.0) | 87.9 (4.5) | 0.91 (0.80-0.97) |
|
|
Δ | 0.25 (0.12) | 10.0 (5.4) | 14.0 (7.9) | 0.26 |
aQWK: quadratic weighted Cohen κ.
bICC: intraclass correlation coefficient.