Table 3.
Previous studies reporting weighted kappa (κw) values for intra- and inter-rater agreements together with the test set agreement between L.C. and AVRA (in bold text), and L.C and VGG16 as a reference. The interval given refers to the minimum and maximum κw value reported in the referenced study. The N column refers to the number of images used for the intra- and inter-rater assessment (if two values are given the number of images rated were different for the intra- and the inter-rater analysis). ∗ denotes if L.C. (whose ratings was used for training in this study) was one of the raters in the reported agreements.
| Study | Scale | N | Intra-rater agreement (κw) | Inter-rater agreement (κw) |
|---|---|---|---|---|
| Cavallin et al. (2012) | MTA | 100 | 0.83–0.94∗ | 0.72–0.84∗ |
| Cavallin et al. (2012b) | MTA | 100 | 0.84–0.85∗ | – |
| Westman et al. (2011) | MTA | 100 | 0.93∗ | – |
| Velickaite et al. (2017) | MTA | 20/50 | 0.79–0.84 | 0.6–0.65∗ |
| Ferreira et al. (2017) | MTA | 120 | 0.89–0.94∗ | 0.70–0.71∗ |
| Koedam et al. (2011) | MTA | 29/118 | 0.91–0.95 | 0.82–0.90 |
| VGG16 | MTA | 464 | 1 | 0.58–0.59∗ |
| AVRA | MTA | 464 | 1 | 0.72–0.74∗ |
| Koedam et al. (2011) | PA | 29/118 | 0.93–0.95 | 0.65–0.84 |
| Ferreira et al. (2017) | PA | 120 | 0.88∗ | 0.88∗ |
| VGG16 | PA | 464 | 1 | 0.63∗ |
| AVRA | PA | 464 | 1 | 0.74∗ |
| Ferreira et al. (2016) | GCA-F | 100 | 0.70∗ | 0.59∗ |
| Ferreira et al. (2017) | GCA-F | 120 | 0.83∗ | 0.79∗ |
| VGG16 | GCA-F | 464 | 1 | 0.56∗ |
| AVRA | GCA-F | 464 | 1 | 0.62∗ |