Table 5. Inter-rater reliability for each pair of judges at second scoring.
This scoring was completed two weeks after the first scoring. The average Cohen’s kappa is lower, but still acceptable at 0.66 with a percentage of agreement of 74.2%. As before, most scores that were not in agreement differed by one score.
| Judge pair | 1–2 | 1–3 | 1–4 | 1–5 | 2–3 | 2–4 | 2–5 | 3–4 | 3–5 | 4–5 | average |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Two scores agree | |||||||||||
| n |
66 | 69 | 68 | 71 | 62 | 63 | 61 | 78 | 65 | 65 | 66.8 |
| % | 73.3 | 76.7 | 75.6 | 78.9 | 68.9 | 70.0 | 67.8 | 86.7 | 72.2 | 72.2 | 74.2 |
| Number of scores that differ by |
|||||||||||
| 1 |
20 | 19 | 22 | 14 | 22 | 24 | 24 | 11 | 20 | 21 | 19.7 |
| 2 |
4 | 2 | 0 | 3 | 5 | 3 | 3 | 1 | 4 | 3 | 2.8 |
| 3 |
0 | 0 | 0 | 2 | 1 | 0 | 2 | 0 | 1 | 1 | 0.7 |
| >3 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0.0 |
| Cohen's Kappa | 0.65 | 0.70 | 0.68 | 0.72 | 0.60 | 0.61 | 0.58 | 0.83 | 0.64 | 0.64 | 0.66 |