Table 1.
Interrater reliability figures between SentiStrength and human raters.
| Reliability figuresa | SentiStrength | Rater 1 | Rater 2 | Rater 3 | |
| Positive scale |
|
|
|
|
|
|
|
Mean (95% CI) | 2.16 (2.10, 2.22) | 2.11 (2.05, 2.18) | 2.09 (2.03, 2.15) | 2.74 (2.65, 2.82) |
|
|
Full agreement, % |
|
53.60 | 55.60 | 31.90 |
|
|
Close agreement, % |
|
92.50 | 93.20 | 75.90 |
|
|
Cohen’s κ |
|
.349 | .375 | .139 |
|
|
Spearman ρ |
|
.635 | .647 | .596 |
| Negative scale |
|
|
|
|
|
|
|
Mean (95% CI) | –1.93 (–2.00, –1.85) | –1.75 (–1.82, –1.69) | –1.72 (–1.79, –1.66) | –1.94 (–2.01, –1.86) |
|
|
Full agreement, % |
|
64.70 | 67.60 | 58.80 |
|
|
Close agreement, % |
|
89.70 | 91.00 | 84.10 |
|
|
Cohen’s κ |
|
.430 | .470 | .342 |
|
|
Spearman ρ |
|
.719 | .728 | .630 |
a A random sample of 1000 comments were reviewed. Full agreement means that the human rater and SentiStrength had exactly the same rating. Close agreement means that the difference was maximum 1 point on the 5-point scale.