Statistical Methods for the Analysis of Inter-Reader Agreement Among Three or More Readers

The inter-reader agreement is a key imaging interpretation-related outcome parameter in radiological research. The interpretation of medical images can be affected by the subjective assessment of readers. Therefore, the measure of inter-reader agreement is crucial. A substantial degree of reliability is required in clinical research involving image interpretation.

To evaluate the extent of inter-reader agreement, measures of agreement such as kappa, intraclass correlation coefficient (ICC), and concordance correlation coefficient (CCC) are commonly employed [1]. In a study with only two readers, statistical methods for analyzing inter-reader agreement can be easily applied and interpreted. Nevertheless, research studies may necessitate the involvement of three or more readers to enhance the generalizability of results across diverse clinical practices [2]. Researchers maybe less acquainted with statistical methods for analyzing inter-reader agreements involving three or more readers compared to methods for two readers. Therefore, this article provides a brief guide on statistical methods for analyzing inter-reader agreement among three or more readers. The recommended methods are listed in Table 1. Statistical methods were classified according to the scale of the readers’ interpretations: binary (e.g., presence vs. absence of a finding/disease) or nominal scale (e.g., category of imaging findings), ordinal scale (e.g., a 5-point Likert scale for image quality from 1 for poor quality to 5 for good quality), and continuous scale (e.g., size measurement of a lesion).

Table 1. Recommended statistical methods for analysis of interreader agreement among three or more readers.

Type of ratings	Statistical method
Binary or nominal	Brennan-Prediger’s BP
	Conger’s Kappa
	Fleiss’ Kappa
	Gwet’s AC1
	Krippendorff’s α
	Light’s Kappa
Ordinal	Generalized weighted Kappa
	Gwet’s AC2
	Intraclass correlation coefficient
	Light’s Kappa
Continuous	Concordance correlation coefficient
	Intraclass correlation coefficient

The methods are presented in alphabetical order rather than by frequency of use or order of recommendation.

AC = agreement coefficient

Binary or Nominal Scale

Cohen’s kappa [3] was used by only two readers. However, for three or more readers, statistical analysis is more complex because all possible combinations between multiple readers are included in calculating the statistics of agreement. Fleiss’s kappa [4] and Conger’s kappa [5] are well-known alternatives to Cohen’s kappa for three or more readers. Light’s kappa [6] represents the average Cohen’s kappa value calculated for all two-reader combinations. Additionally, to overcome some drawbacks of these chance-corrected measures such as prevalence paradoxes [7], other measures such as Gwet’s agreement coefficient 1 (AC1) [8], Brennan–Prediger’s BP [9], and Krippendorff’s α [10] are used.

Ordinal Scale

Statistical methods for evaluating the agreement of ordinal scales among three or more readers have not been firmly established, and these methods are generally not available in user-friendly statistical software programs. Therefore, published radiological studies often inadequately use the Fleiss Kappa statistic, either neglecting the ordinal nature of the data or applying Cohen’s weighted kappa to all possible reader pairs. Weighted Kappa [11] is another version of Cohen’s kappa that incorporates the weights of pairs of categories in cases with ordinal ratings. The weights can be assigned differently when calculating the kappa to account for varying degrees of agreement between the ratings. However, they can only be applied to data from two readers because they are based on the cross-tabulation of the ratings between the two readers.

Improved methods are available for this purpose (Table 1). Light’s kappa [6], which uses the average of the weighted kappa values obtained for all possible reader pairs, maybe a potentially useful strategy; however, the method does not consider the agreement among all readers. Generalized weighted Kappa including Gwet’s AC2 (Table 1) that incorporate different types of weights to the Kappa for binary or nominal scale is suggested in the literature and implemented in R package ‘irrCAC’ [12]. Although the statistical properties of these approaches have not been fully demonstrated, researchers have attempted to demonstrate them.

Continuous Scale

Inter-reader agreement is generally assessed based on reliability statistics such as ICC [13] or CCC (Table 1) [14]. Additionally, relevant statistical methods for quantitative imaging parameters, which are measured on a continuous scale, are present in the Radiological Society of North America-Quantitative Imaging Biomarkers Alliance (RSNA-QIBA), and COnsensus-based Standards for the selection of health Measurement INstruments (COSMIN) initiative [15,16]. In addition, a graphical presentation through the Bland-Altman plot accommodated by multiple readers [17] can be created by calculating the points of differences and averages of multiple measurements on the x- and y-axes, respectively, for each reader with different symbols/colors. Although the modified Bland-Altman plot cannot provide a measure of inter-reader agreement, the limits of agreement between the two methods can be estimated by considering multiple ratings by multiple readers.

Further Consideration

To evaluate inter-reader agreement on an ordinal scale, various statistics have been proposed and comparative studies [18] have been published. In particular, statistical methods for ordinal scales among three or more raters are not well known, and most of the statistics are based on nominal ratings with some weights. However, whether authors used weighted statistics often remains unclear. Chance-corrected measures of agreement are limited by the prevalence effects, imbalances among categories, and missing values. To overcome these issues, presenting the analytical results alongside the proportion of observed agreements may help readers understand inter-reader rating data. Researchers should present the statistical methods and software used to analyze the inter-reader agreement data to reflect the rating nature and promote high quality and transparency of the reporting.