Simulating and estimating agreement in the presence of multiple raters and covariates

Abstract

Cohen’s and Fleiss’s kappa are popular estimators for assessing agreement among two and multiple raters, respectively, for a binary response. While additional methods have been developed to account for multiple raters and covariates, they are not always applicable, rarely used, and none simplify to Cohen’s kappa. Furthermore, there are no methods to simulate Bernoulli observations under the kappa agreement structure such that the developed methods could be adequately assessed. This manuscript overcomes these shortfalls. First, we developed a model-based estimator for kappa that accommodates multiple raters and covariates through a generalized linear mixed model and encompasses Cohen’s kappa as a special case. Second, we created a framework to simulate dependent Bernoulli observations that upholds all 2-tuple pair of rater’s kappa agreement structure and includes covariates. We used this framework to assess our method when kappa was nonzero. Simulations showed that Cohen’s and Fleiss’s kappa estimates were inflated unlike our model-based kappa. We analyzed an Alzheimer’s disease neuroimaging study and the classic cervical cancer pathology study. The proposed model-based kappa and advancement in simulation methodology demonstrates that the popular approaches of Cohen’s and Fleiss’s kappa are poised to yield invalid conclusions while our work overcomes shortfalls, leading to improved inferences.

1 |. INTRODUCTION

Many fields, such as medicine and psychology, utilize agreement studies.^1–3 Estimators for agreement are also used in other fields, such as a performance metric in machine learning.^4,5 Agreement studies are useful when the outcome is unknown or subjective (eg, Alzheimer’s disease diagnosis based on neuroimaging biomarkers or the optimal machine learning model). Multiple raters may be included in agreement studies. One of the first proposed estimators of agreement among multiple (>2) raters was Fleiss’s kappa, which remains popular today.⁶ Although developed explicitly for two raters, some use Cohen’s kappa to assess agreement across all pairs of two raters.⁷

There are limitations to Cohen’s kappa and Fleiss’s kappa. It is well known that Cohen’s kappa depends on the marginal probability of a positive response (frequently termed sample disease prevalence).⁸ Both Cohen’s and Fleiss’s kappa can have paradoxical results, where there is high observed agreement, but low values of kappa.⁹ Neither approach can directly control for explanatory factors.

There are other statistical methods that can accommodate multiple raters and tend to provide population based inference. These include Landis and Koch’s hierarchical kappa-type statistic,¹⁰ Tanner and Young’s log-linear modeling approach,¹¹ Klar’s generalized estimating equations (GEE) approach,¹² Kraemer’s intraclass kappa,¹³ and Nelson’s generalized linear mixed model (GLMM) kappa-like statistics.^14,15 These methods perform best with a larger number of raters.

Fleiss’s and Cohen’s kappa are generally favored in practice over these newer methods. This may be due to the ease of calculation and widespread acceptance of Fleiss’s and Cohen’s kappa as estimators for agreement. For instance, the GEE approach of Klar, Lipsitz and Ibrahim requires solving two sets of equations by iterative procedures.¹² None of these newer methods provide estimates of agreement for each pair of raters and across all pairs of raters after accounting for explanatory factors. Additionally, these newer methods may not be applied to all situations. For example, the kappa-like statistic proposed by Nelson and Edwards^14,15 cannot be used when the GLMM does not converge (ie, the variance of the random effects is not estimable). These methods do not simplify to Fleiss’s or Cohen’s kappa, thus while comparisons could technically be made, the interpretations of their estimates are not exchangeable.

Aside from these statistical estimators, there are no frameworks available to stimulate Bernoulli observations under a particular agreement dependency structure. Several methods have been proposed to generate correlated binary variables; however, correlation does not imply agreement.^16–22

The purpose of this work is 2-fold. First, propose an estimator of the kappa agreement structure that accounts for multiple raters and explanatory factors when the outcome is binary. This estimator includes Cohen’s kappa as a special case. Second, provide a framework for simulating Bernoulli observations under the kappa agreement structure while adjusting for covariates. This framework is used to generate Bernoulli observations for multiple raters under the kappa agreement structure for each 2-tuple pair of raters. The primary context considered is the assessment of pairwise agreement among multiple raters evaluating each subject once.

In Section 2, we describe the kappa agreement structure and common estimators: Cohen’s and Fleiss’s kappa. We define our estimator and show it is a function of each 2-tuple pair of raters’ kappa agreement structure. In Section 3, the novel simulation framework is described. In Section 4, simulation results show that Cohen’s and Fleiss’s kappa estimators provide inflated estimates while our estimator does not. In Section 5, we analyze an Alzheimer’s neuroimaging study and a classic pathology agreement study. These analyses demonstrate when our method can be used despite lack of convergence of the GLMM. Lastly, in Section 6, we provide a discussion.

2 |. METHODS

Kappa is a popular agreement structure and is defined as a function of observed $\left({\pi }_o\right)$ and expected $\left({\pi }_e\right)$ agreement:

$$\kappa =\frac{{\pi }_o-{\pi }_e}{1-{\pi }_e}$$ (1)

Many estimators for kappa have been proposed.^6,23,24 These works define ${\pi }_o$ as the observed joint probability of agreement between raters, and ${\pi }_e$ as the chance-expected agreement.²⁵ A common estimate for ${\pi }_o$ is the proportion of subjects the raters provided the same assessment while ${\pi }_e$ is estimated under the assumption of rater independence. These estimators define ${\pi }_e$ under different functions of each rater’s marginal probabilities for each outcome.

In 1960, Cohen proposed a parameterization of kappa (termed Cohen’s kappa) that is widely used today. For a binary outcome (say category 1 = positive and 2 = negative) and two raters, Cohen’s kappa $\left({\kappa }_C\right)$²³ can be written as: ${\kappa }_C=\frac{\left({\pi }_{11}+{\pi }_{22}\right)-\left({\pi }_{1+}{\pi }_{+1}+{\pi }_{2+}{\pi }_{+2}\right)}{1-\left({\pi }_{1+}{\pi }_{+1}+{\pi }_{2+}{\pi }_{+2}\right)}$, where ${\pi }_{1+}$ and ${\pi }_{+1}\left({\pi }_{2+}\right.\mathrm{a}\mathrm{n}\mathrm{d}\left.{\pi }_{+2}\right)$ are the marginal probabilities the raters categorized the subjects “1” (“2”) and ${\pi }_{11}\left({\pi }_{22}\right)$ is the unconstrained probability both raters classified the same subjects as “1” (“2”). In this case, ${\pi }_o={\pi }_{11}+{\pi }_{22}$ and ${\pi }_e={\pi }_{1+}{\pi }_{+1}+{\pi }_{2+}{\pi }_{+2}$.

Fleiss’s kappa $\left({\kappa }_F\right)$ was proposed as an estimator of kappa when there are multiple raters. A variety of study designs can be analyzed using ${\kappa }_F$. Examples include when all raters evaluate all subjects and when each rater evaluates a different subset of subjects. Notably ${\kappa }_F$ requires the raters to evaluate the same number of subjects.⁶ In this estimator, ${\pi }_e$ is a function of the average probability for each category across all raters while ${\pi }_0$ is the sum of the proportion of rater-pairs in agreement for each category.

While ${\kappa }_C$ was developed exclusively for two raters, there are cases where it is used when there are more than two raters.^7,26 In these situations, an estimate of ${\kappa }_C$ is generally provided for each pair of raters, while ${\kappa }_F$ provides a single estimate across all raters.

2. 1 |. Formulating kappa using generalized linear mixed models

Cohen’s kappa was first developed as a data-driven measure of agreement.²³ Subsequent works showed that ${\pi }_e$ in ${\kappa }_C$ has maximum likelihood estimator properties.²⁵ The purpose of this subsection is to demonstrate that ${\pi }_e$ can be formulated using GLMMs. This is valuable as we can model the complex data structure that is mis-specified under the widely used ${\kappa }_C$ and ${\kappa }_F$.

Consider the marginal probability of a positive (or negative) evaluation used in the definitions of ${\pi }_e$ for ${\kappa }_C$ and ${\kappa }_F$ $\left(\mathrm{e}\mathrm{g},{\pi }_{1+}\right)$. This is simply the proportion of subjects the given rater evaluated as positive (or negative). It is known that the marginal probability of a positive (or negative) evaluation can be equivalently defined under a logistic regression model that includes only raters as a fixed effect in the linear predictor term.²⁷ While ${\kappa }_C$ and ${\kappa }_F$ are currently classified in the literature as simple “chance” adjusted measures of agreement,¹⁵ ${\kappa }_C$ and ${\kappa }_F$ are indeed model-based, since ${\pi }_e$ is a function of a very simple assumed logistic regression model.

Including raters as the only fixed effect in this logistic regression model has three limitations. First, we assume that the included raters encompass the entire population of raters, and that the scope of inference is limited to these specific raters. Second, the probability of a positive response is assumed to be constant for all subjects evaluated by a given rater, regardless of each individual subject’s characteristics. Third, as the number of raters becomes larger, more parameters must be estimated; this could limit the number of raters included in each analysis or require more subjects obtain the maximum likelihood estimate.

As further described below in Section 2.2, this work overcomes the limitations of defining ${\pi }_e$ as a function of the highly restricted logistic regression model that includes only raters as a fixed effect. This is accomplished by including raters as a random effect and explanatory measures as fixed effects in the model used to define ${\pi }_e$.

2. 2 |. Model-based estimator of kappa

The primary study design considered in this manuscript is when the outcome is binary, there are more than two raters, and each rater evaluates each subject once. Additionally, the goal of the study is to assess agreement under a pairwise structure, or when a pair of two raters reaches the same conclusion. Other definitions of agreement are not considered.^28,29

We defined observed agreement (${\pi }_o$ in Equation 1) as the joint probability of agreement among all pairs of raters, and expected agreement (${\pi }_e$ in Equation 1) as the probability of agreement for each pair of raters under the assumption of rater conditional independence. Simple, unadjusted proportions in the data (ie, a saturated model) are used to estimate ${\pi }_0$. Thus, we describe ${\pi }_o$ as unconstrained. A GLMM is used to estimate ${\pi }_e$, instead of the minimal logistic regression model described in Section 2.1. We use the term model-based to describe ${\pi }_e$, to emphasize the fact that ${\pi }_e$ is always a function of a GLMM.

$$\text{ Consider the following notation. Let }{Y}_{ij}=\left\{\begin{array}{ll}1,& \text{ rater }j\text{ evaluates subject }i\text{ as positive }\\ 0,& \text{ rater }j\text{ evaluates subject }i\text{ as negative }\end{array}\right.$$

where $i=1,\dots ,N$ and $j=1,\dots ,R$. Then, $Y_{ij}~\mathit{\text{Bern}}\left({\theta }_{ij}\right)$, where ${\theta }_{ij}=E\left[Y_{ij}\right]=P\left(Y_{ij}=1\right)$.

In a population of $N$ subjects, $R$ raters, and $T=\left(\begin{array}{l}R\\ 2\end{array}\right)$ pairs of two raters, the unconstrained probability of agreement is defined as

$${\pi }_o=\frac{1}{N}\frac{1}{T}\sum _{i=1}^N\sum _{j=1}^{R-1}\sum _{j^{\prime }=j+1}^R\left[P\left(Y_{ij}=1\cap Y_{i{j}^{\prime }}=1\right)+P\left(Y_{ij}=0\cap Y_{i{j}^{\prime }}=0\right)\right]$$

and the model-based probability of agreement, under the assumption of rater independence, is

$${\pi }_e=\frac{1}{N}\frac{1}{T}\sum _{i=1}^N\sum _{j=1}^{R-1}\sum _{j^{\prime }=j+1}^R\left[P\left(Y_{ij}=1\right)P\left(Y_{i{j}^{\prime }}=1\right)+\left(1-P\left(Y_{ij}=1\right)\right)\left(1-P\left(Y_{i{j}^{\prime }}=1\right)\right)\right]$$

The probability rater $j$ evaluates subject $i$ as positive can be modeled using a GLMM, where raters are included as a random effect. This allows for rater-specific tendencies to be modeled and accounts for non-independent observations since each rater evaluates more than one subject.

Consider model $M:M\equiv \mathit{\text{logit}}\left({\theta }_{ij}\right)={\mathit{x}}_{ij}^T\mathit{\beta }+{\mathit{z}}_j^T\mathit{b}$, where $i=1,\dots ,N$, $j=1,\dots ,R$, $\mathit{\beta }$ is a vector of parameters for subject and rater characteristics, $\mathit{b}~N\left(0,{\sigma }_R^2\right)$ is a vector of random effects for the raters, and ${\theta }_{ij}=E\left(Y_{ij}\mid {\mathit{x}}_{ij},b_j\right)=P\left(Y_{ij}=1\mid {\mathit{x}}_{ij},b_j\right)$. Then, the model-based probability of agreement under model $M$ is: ${\pi }_e\mid M=\frac{1}{N}\frac{1}{T}{\sum }_{i=1}^N{\sum }_{j=1}^{R-1}{\sum }_{j^{\prime }=j+1}^R\left[{\theta }_{ij}{\theta }_{i{j}^{\prime }}+\left(1-{\theta }_{ij}\right)\left(1-{\theta }_{i{j}^{\prime }}\right)\right]$, which includes random and fixed effects.

The proposed model-based estimator of kappa, under model $M$, is then:

$${\kappa }_M=\frac{\frac{1}{N}\frac{1}{T}\sum _{i=1}^N\sum _{j=1}^{R-1}\sum _{j^{\prime }=j+1}^R\left[P\left(Y_{ij}=Y_{i{j}^{\prime }}\right)\right]-\frac{1}{N}\frac{1}{T}\sum _{i=1}^N\sum _{j=1}^{R-1}\sum _{j^{\prime }=j+1}^R\left[{\theta }_{ij}{\theta }_{i{j}^{\prime }}+\left(1-{\theta }_{ij}\right)\left(1-{\theta }_{i{j}^{\prime }}\right)\right]}{1-\frac{1}{N}\frac{1}{T}\sum _{i=1}^N\sum _{j=1}^{R-1}\sum _{j^{\prime }=j+1}^R\left[{\theta }_{ij}{\theta }_{i{j}^{\prime }}+\left(1-{\theta }_{ij}\right)\left(1-{\theta }_{i{j}^{\prime }}\right)\right]}$$

Cohen’s kappa $\left({\kappa }_C\right)$ and Fleiss’s kappa $\left({\kappa }_F\right)$ can also be written under this notation (Supplemental Appendix A). Using algebra, ${\kappa }_M$ can be re-written as:

$${\kappa }_M=\frac{{\pi }_o-{\pi }_e\mid M}{1-{\pi }_e\mid M}=\sum _{l=1}^T{\kappa }_l\frac{\left(1-{\pi }_{e,l}\mid M\right)}{c}$$ (2)

where $l$ is the index for each of the $T$ 2-tuple pair of raters, $c=T-{\sum }_{l=1}^T{\pi }_{e,l}\mid M$ and is a constant, and ${\pi }_{e,l}\mid M=\frac{1}{N}{\sum }_i\left[{\theta }_{ij}{\theta }_{i{j}^{\prime }}+\left(1-{\theta }_{ij}\right)\left(1-{\theta }_{i{j}^{\prime }}\right)\right]$ is the model-based probability of agreement for the 2-tuple rater pair $l$. In words, ${\kappa }_M$ is a weighted average of each rater-pair’s kappa, where the weights are the proportion of model-based disagreement for rater-pair $l$ compared to the total model-based disagreement among all rater-pairs.

We estimate ${\pi }_o$, in a sample of $n$ subjects and $r$ raters, as the average of the proportion of subjects upon which the evaluations agreed within each rater-pair:

$${\widehat{\pi }}_o=\frac{1}{n}\frac{1}{t}\sum _{i=1}^n\sum _{j=1}^{r-1}\sum _{j^{\prime }=j+1}^r\left[\left(y_{ij}{y}_{i{j}^{\prime }}\right)+\left(1-y_{ij}\right)\left(1-y_{i{j}^{\prime }}\right)\right]$$

where $t=\left(\begin{array}{l}r\\ 2\end{array}\right)$, $i=1,\dots ,n\le N$, and $j=1,\dots ,r\le R$.

Under a specified (assumed) model $M$, we obtain predicted probabilities of a positive response for each subject-rater observation, ${\hat{\theta }}_{ij}$. These model-based predicted probabilities, $\mathit{\text{logit}}\left({\hat{\theta }}_{ij}\right)={\mathit{x}}_{ij}^T\widehat{\mathit{\beta }}+b_j$, are then used to calculate ${\widehat{\pi }}_e$ assuming rater conditional independence:

$${\widehat{\pi }}_e\mid M=\frac{1}{n}\frac{1}{t}\sum _{i=1}^n\sum _{j=1}^{r-1}\sum _{j^{\prime }=j+1}^r\left[{\widehat{\theta }}_{ij}{\widehat{\theta }}_{i{j}^{\prime }}+\left(1-{\widehat{\theta }}_{ij}\right)\left(1-{\widehat{\theta }}_{i{j}^{\prime }}\right)\right]$$

In order to determine which fixed effects to include in estimating the probability of a positive response, usual GLMM building methods can be used.³⁰ While the estimates of kappa are known to be approximately normally distributed when the number of subjects is large,²³ we bootstrap with subject-level resampling to estimate the variances of $\widehat{\kappa }$ across all raters as well as for ${\widehat{\kappa }}_l$, which estimates agreement for the 2-tuple rater pair $l.$³¹ Empirical confidence intervals were calculated using the bootstrap samples. The estimates, empirical standard errors, and empirical 95% confidence intervals (using the 0.025 and 0.975 quantiles of the bootstrapped data) are reported for all results (simulations and real-data analyses). The mean squared error is reported for simulation results.

3 |. SIMULATING BERNOULLI VARIABLES UNDER THE KAPPA AGREEMENT STRUCTURE

There are two general simulation situations of interest. First, when there is no agreement beyond rater independence (and what assumed model $M$ would predict): $\kappa =0$. It is straightforward to generate data when $\kappa =0$ (and ${\kappa }_l=0\forall l$), as joint distributions are simply the products of marginal univariate Bernoulli random variables.

Second, when agreement is greater than an independence structure and what model $M$ would predict: $\kappa >0$. Generating Bernoulli observations is not trivial when $\kappa >0$ (and ${\kappa }_l=c_l\forall l$, where $c_l\in (0,1]\forall l)$. All 2-tuple pairs of raters have a specific kappa dependency structure, and each rater contributes to $R-1$ values of ${\kappa }_l$. For instance, suppose $R=4$ and we are simulating Rater 4’s observation vector. Rater 4’s observations (drawn once) must adhere to the three, potentially distinct, kappa dependency structures with Raters 1, 2 and 3; in addition to previously drawn bivariate dependency structures of Raters 1 and 2, Raters 1 and 3, and Raters 2 and 3.

Methods are available to simulate correlated binary observations. These methods use the marginal distribution of the observations (ie, after integrating out any random effects) and may include covariates.^16–21 However, we know of no simulation method that enforces a specific kappa agreement dependency structure; i.e., for each 2-tuple rater pair $\left({\kappa }_l\right)$ and across all raters $\left(\kappa \right)$ that simultaneously allows for the probability of a positive response to vary for each subject-rater observation. Further, our development allows for rater-specific inference, rather than population-based inference, through including raters as random effects.

The main idea of our approach is to use the multiplication rule for conditional probabilities to draw observations for each rater sequentially.³² This is necessary to maintain the kappa agreement dependency structure for each 2-tuple rater pair and allow for the probability of a positive response for each subject-rater observation to vary. For example, Rater 1 will either have a positive $\left(Y_{i1}=1\right)$ or negative $\left(Y_{i1}=0\right)$ evaluation for subject $i$ based on Rater 1’s probability of a positive evaluation, $P\left(Y_{i1}=1\right)$. Next, we calculate Rater 2’s probability of a positive response based on Rater 1’s outcome: $P\left(Y_{i2}=1\mid Y_{i1}=1\right)=\frac{P\left(Y_{i1}=1,Y_{i2}=1\right)}{P\left(Y_{i1}=1\right)}$ and $P\left(Y_{i2}=1\mid Y_{i1}=0\right)=\frac{P\left(Y_{i1}=0,Y_{i2}=1\right)}{P\left(Y_{i1}=0\right)}$. This is completed sequentially for each rater. Figure 1 is a tree diagram that depicts drawing sequential observations for three raters.

FIGURE 1.

Tree diagram that depicts drawing sequential observations for three raters.

More details are provided. Suppose there are $R$ raters and an assumed model $M$. The values of ${\kappa }_l$ are selected based on model $M$. Note that raters with similar tendencies may attain a larger ${\kappa }_l$. The value of $\kappa$ is calculated via Equation (2). The theoretical probabilities of the usual two-way table of two raters for each subject (eg, $P\left(Y_{ij}=1,Y_{i{j}^{\prime }}=1\right)$; see Supplemental Table S1 for complete details) are calculated using the corresponding value of ${\kappa }_l$, $P\left(Y_{ij}=1\right)$ and $P\left(Y_{i{j}^{\prime }}=1\right)$ where $l$ represents the rater pair of Raters $j$ and $j^{\prime }$ (details in Supplemental Appendix B).

Since each rater’s observations are drawn sequentially, the joint probabilities for each set of raters, $P\left(Y_{i1}=y_{i1},Y_{i2}=y_{i2},Y_{i3}=y_{i3}\right)$, $P\left(Y_{i1}=y_{i1},Y_{i2}=y_{i2},Y_{i3}=y_{i3},Y_{i4}=y_{i4}\right),\dots ,P\left(Y_{i1}=y_{i1},\dots ,Y_{iR}=y_{iR}\right)$ for all possible combinations of each of the $y_{ij}=1$ or $0$ must be solved. Each joint probability is a numerator for a conditional probability used to draw the next rater’s observation. For example, $P\left(Y_{i1}=y_{i1},\dots ,Y_{iR}=y_{iR}\right)$ is a numerator for drawing the last rater’s (Rater $R$) probability of a positive response.

First, the probabilities involving Rater $R$ (a total of $2^R$ probabilities) are found. For instance, if $R=4$, we solve for the $16\left(=2^4\right)$ probabilities: $P\left(Y_{i1}=1,Y_{i2}=1,Y_{i3}=1,Y_{i4}=1\right)$, $P\left(Y_{i1}=1,Y_{i2}=1,Y_{i3}=1,Y_{i4}=0\right),\dots ,P\left(Y_{i1}=0,Y_{i2}=0,\left.Y_{i3}=0,Y_{i4}=0\right)\right.$. We use non-negative least squares to solve for these probabilities: $\mathit{A}{\mathit{u}}_i={\mathit{q}}_i$ where ${\mathit{u}}_i\ge 0.$³³ Here, ${\mathit{q}}_i$ is a vector containing the known probabilities of agreement for all 2-tuple rater pairs, $P\left(Y_{ij}=1,Y_{i{j}^{\prime }}=1\right)$ and $P\left(Y_{ij}=0,Y_{i{j}^{\prime }}=0\right)$, and each rater’s probability of a positive response, $P\left(Y_{ij}=1\right)$. An example ${\mathit{q}}_i$, ${\mathit{u}}_i$, and $\mathit{A}$ is provided in Supplemental Appendix C.

The probabilities in ${\mathit{u}}_i$ form a partition of the joint probabilities for each set of raters needed to compute the conditional probabilities of a positive response when drawing the raters sequentially.³² Hence, algebra is used to solve for the required sets of joint probabilities. Recall that when $R=4$, ${\mathit{u}}_i=\left[P\left(Y_{i1}=1,Y_{i2}=1,Y_{i3}=1,Y_{i4}=1\right),\dots ,\left.P\left(Y_{i1}=0,Y_{i2}=0,Y_{i3}=0,Y_{i4}=0\right)\right].\right.$ To solve for the probabilities involving Raters 1, 2 and 3 $\left(P\left(Y_{i1}=1,Y_{i2}=1\right.\right.,\left.\left.Y_{i3}=1\right),\dots ,P\left(Y_{i1}=0,Y_{i2}=0,Y_{i3}=0\right)\right)$, the probabilities in the respective partition in ${\mathit{u}}_i$ are summed. For example, $P\left(Y_{i1}=1,Y_{i2}=1,Y_{i3}=1\right)=P\left(Y_{i1}=1,Y_{i2}=1,Y_{i3}=1,Y_{i4}=1\right)+P\left(Y_{i1}=1,Y_{i2}=1,Y_{i3}=1,Y_{i4}=0\right)$.

Each subject is considered separately throughout all calculations. This is because the true probabilities of agreement and disagreement are known for each subject within each pair of raters. There may be some cases where a calculated probability is larger than 1. This may happen when either $P\left(Y_{ij}=1,Y_{i{j}^{\prime }}=1\right)<0$ or $P\left(Y_{ij}=0,Y_{i{j}^{\prime }}=0\right)<0$, which indicates that the selected ${\kappa }_l$ is not possible given the marginal probabilities of the raters (ie, given model $M$).

There are several packages available in R that can implement non-negative least squares (eg, “limSolve”, “glmnet” or “bvls”).^34–36 We used the “limSolve” package due to ease of implementation. Their approach utilizes FORTRAN from Linpack.³³

4 |. SIMULATION RESULTS

Extensive simulations were completed. While data were simulated under a variety of conditions, four representative results are presented. All reported simulations had the same number of raters $(r=6)$, number of subjects $(n=200)$, and model $M$ :

$$M:\mathit{\text{logit}}\left({\theta }_{ij}\right)=1.5x_{i,\mathit{\text{ case }}}+1x_{i,\mathit{\text{ age }}}-0.75x_{j,\mathit{\text{ unexp }}}+b_j$$

The outcome in these simulations can be thought of as a medical diagnostic test evaluation. The explanatory factors included the disease status of each subject ($x_{i,\mathit{\text{case}}}=1$ if subject $i$ was a case), the standardized age of each subject ($x_{i,\mathit{\text{age}}}$, continuous), and the (binary) training level of the rater ($x_{j,\mathit{\text{unexp}}}=1$ if the rater was unexperienced). The proportion of cases among subjects was set as 0.5, as well as the proportion of unexperienced raters. Each rater’s random effect was drawn independently from a $N\left(0,{\sigma }_r^2\right)$. The simulations differed in the SD of the rater random effect (${\sigma }_r=0.5$ or $1$) and the value of kappa ($\kappa =0$ or $\kappa >0$).

We estimated each rater-pair’s ${\kappa }_l$ using Cohen’s kappa $\left({\kappa }_{l,C}\right)$ and the proposed method $\left({\kappa }_{l,M}\right)$. Kappa was estimated among all raters using Fleiss’s kappa $\left({\kappa }_F\right)$ and the proposed method $\left({\kappa }_M\right)$. The mean squared error (MSE) was calculated. Numerical simulation results are provided in Supplemental Tables S2 and S3, with key findings presented for ${\sigma }_r=0.5$. The provided results are representative of all conducted simulations.

Figure 2 is a plot of densities for ${\widehat{\kappa }}_l$ across all simulations. The nominal kappa values $\left({\kappa }_l\right)$ are represented by color-coded, vertical reference lines. The rows refer to each simulation: (1) ${\sigma }_r=0.5$ and $\kappa =0$ (and ${\kappa }_l=0\forall l$); (2) ${\sigma }_r=0.5$ and $\kappa >0$ (and ${\kappa }_l>0\forall l$); (3) ${\sigma }_r=1$ and $\kappa =0$ (and ${\kappa }_l=0\forall l$); (4) ${\sigma }_r=1$ and $\kappa >0$ (and ${\kappa }_l>0\forall l$). The distance between the nominal kappa values and the corresponding densities of ${\widehat{\kappa }}_{l,M}$ and ${\widehat{\kappa }}_{l,C}$ represents the bias of the proposed method and Cohen’s kappa, respectively. Rater random effects were consistent for a given value of ${\sigma }_r$, allowing for comparison of Simulations 1 and 2, and Simulations 3 and 4.

FIGURE 2.

Simulation densities of estimates for each pair of two raters. Estimates were calculated using Cohen’s kappa (left panel) and the GLMM-based kappa (right panel). Nominal kappa values are represented by color-coded, vertical reference lines.

There are several key findings. First, ${\widehat{\kappa }}_{l,C}$ tended to be larger (biased) than the nominal values, whereas ${\widehat{\kappa }}_{l,M}$ was distributed around the nominal values (unbiased). Under the hypothesis of no agreement beyond what model $M$ would predict under the assumption of independence (ie, $\kappa =0$), ${\widehat{\kappa }}_{l,C}$ was not centered around 0, while ${\widehat{\kappa }}_{l,M}$ was centered around 0. For example, ${\stackrel{-}{\stackrel{ˆ}{\kappa }}}_{1,C}=0.261(\mathrm{S}\mathrm{D}=0.057)$ while ${\stackrel{-}{\stackrel{ˆ}{\kappa }}}_{1,M}=-0.02(\mathrm{S}\mathrm{D}=0.071)$, where ${\stackrel{-}{\stackrel{ˆ}{\kappa }}}_{1,C}$ and ${\stackrel{-}{\stackrel{ˆ}{\kappa }}}_{1,M}$ are the averages of ${\hat{\kappa }}_{1,C}$ and ${\widehat{\kappa }}_{1,M}$, respectively, over all simulations (see also Supplemental Table S2). Likewise, when $\kappa >0$, ${\widehat{\kappa }}_{l,C}$ was generally biased larger than the nominal values whereas ${\widehat{\kappa }}_{l,M}$ was centered/unbiased. Comparing the estimates for Pair 1 (when ${\sigma }_r=0.5$; ${\kappa }_1=0.321$), ${\stackrel{-}{\stackrel{ˆ}{\kappa }}}_{1,C}=0.51(\mathrm{S}\mathrm{D}=0.051)$ while ${\stackrel{-}{\stackrel{ˆ}{\kappa }}}_{1,M}=0.303(\mathrm{S}\mathrm{D}=0.061)$ (Supplemental Table S2). Notably, as ${\kappa }_l$ became closer to its upper bound of 1, Cohen’s kappa estimates became less inflated. For example, when ${\kappa }_4=0.928$, ${\stackrel{-}{\stackrel{ˆ}{\kappa }}}_{4,C}=0.953(\mathrm{S}\mathrm{D}=0.022)$ and ${\stackrel{-}{\stackrel{ˆ}{\kappa }}}_{4,M}=0.928(\mathrm{S}\mathrm{D}=0.034)$ (Supplemental Table S2).

While ${\widehat{\kappa }}_{l,C}$ was inflated compared to ${\widehat{\kappa }}_{l,M}$, the SD of ${\widehat{\kappa }}_{l,C}$ was slightly less than that of ${\widehat{\kappa }}_{l,M}$ (Supplemental Tables S2 and S3). This indicates that ${\widehat{\kappa }}_{l,C}$ was more precise around a biased estimate. Across both estimation methods, the SD of ${\widehat{\kappa }}_l$ decreased as the value of ${\kappa }_l$ increased. This is consistent with properties of the binomial distribution, where the variance of a binomial random variable is smallest when the probability of a positive response is close to 0 or 1. The MSE was smaller under the proposed method; however, when ${\kappa }_l$ was closer to its upper bound of 1, the MSE under both methods were comparable. Lastly, the SD of the random effects did not impact the ability to estimate ${\kappa }_l$.

Similar results were found when examining estimates of $\kappa$ (Figure 3 and Supplemental Table S4). Estimates of ${\kappa }_F$ were inflated (biased upward) and produced similar values while estimates of ${\kappa }_M$ were centered around the nominal values (unbiased). The MSE for ${\widehat{\kappa }}_M$ was always smaller that of ${\widehat{\kappa }}_F$ (Supplemental Table S4). The estimates were not impacted by ${\sigma }_r$.

FIGURE 3.

Simulation density of estimates among all raters. Estimates were calculated using Fleiss’s kappa (red) and the GLMM-based kappa (green). Nominal kappa values are represented by color-coded, vertical reference lines.

5 |. APPLICATIONS TO BIOMEDICAL STUDIES

Two study datasets were analyzed. The first study dataset contained interpretations of 54 Florbetapir positron emission tomography (PET) scans by three experienced raters for evidence of elevated amyloid-$\mathrm{\beta }$ plaque burden (a pathological hallmark of Alzheimer’s disease).⁷ Clinical and image-specific information was provided. The second study dataset contained seven rater’s interpretations on the cervical cancer status (present or absent) of 118 pathology samples.^10,27,37 This dataset is considered to be a classical example of assessing agreement among multiple raters, but does not contain covariate data. Like simulations, ${\kappa }_l$ was estimated using ${\kappa }_{l,M}$ and ${\kappa }_{l,C}$ while $\kappa$ was estimated using ${\kappa }_F$ and ${\kappa }_M$.

We include both sets of analyses as they highlight different features of our approach. The agreement among raters of the Florbetapir PET scans demonstrates the impact of ignoring covariates. The agreement among raters of pathology samples shows that including a random effect for rater accounts for the dependence structure induced when each rater is paired with all others and results in estimates of agreement consistent with other unadjusted analyses.

5.1 |. Agreement of Florbetapir PET scans

Alzheimer’s disease is common, and its prevalence is expected to increase as our population ages.³⁸ It is important to identify biomarkers of Alzheimer’s disease, as historically diagnosis was determined at autopsy. A pathological hallmark of Alzheimer’s disease is amyloid-$\mathrm{\beta }$ plaques in the brain.³⁸ Imaging technologies, such as Florbetapir PET scans, have advanced such that these plaques can be visualized pre-autopsy.³⁹ The degree of amyloid-$\mathrm{\beta }$ deposition in the brain is not linearly correlated with clinical features of Alzheimer’s disease, and deposition may begin before clinical symptoms appear.⁴⁰ As such, agreement studies have been conducted to assess the utility of Florbetapir PET scans in identifying patients with high amyloid-$\mathrm{\beta }$ deposition.⁷

One such study comes from the Alzheimer’s Prevention Program (APP) from the Alzheimer’s Disease Center (ADC) at the University of Kansas Medical Center (KUMC).⁷ In the APP study, 54 Florbetapir PET scans were evaluated for the presence of high amyloid-$\mathrm{\beta }$ deposition by three experienced raters. Information on study participants was known, such as age and sex. Additionally, the standard uptake value ratio (SUVR), a quantitative (computer algorithm based) measure of the amyloid-$\mathrm{\beta }$ burden status (elevated or not-elevated), was available. Raters were not provided the SUVR status and evaluated all scans independently.

The outcome $\left(Y_{ij}\right)$ was binary for the presence of elevated amyloid-$\mathrm{\beta }$ deposition, where $Y_{ij}=1$ when rater $j$ interpreted scan $i$ as elevated and $Y_{ij}=0$ otherwise. The final model selected included SUVR status (elevated or non-elevated) as a fixed effect, as other participant features were not indicative of describing variation in a positive test interpretation $(P>0.10)$. The model with rater as a random effect did not converge, as there were only three raters. Thus, rater was reverted to a fixed effect and model $M$ was: $\mathit{\text{logit}}\left({\theta }_{ij}\right)={\beta }_0+{\beta }_j+{\beta }_3{x}_{i,\mathit{\text{SUVR}}+}$, where ${\theta }_{ij}=P\left(Y_{ij}=1\right)$, ${\beta }_0$ was the intercept, ${\beta }_j$ was the effect of rater $j$($j=1,2$, with rater 3 as baseline), and ${\beta }_3$ was the effect of an elevated SUVR status. Note that since the GLMM did not converge, other methods, such as that of Nelson and Edwards, cannot be used to assess agreement.

Supplemental Table S5 provides the logistic regression results and the values of ${\widehat{\kappa }}_l$ and $\widehat{\kappa }$. Figure 4 shows the empirical densities of ${\widehat{\kappa }}_l$ (Figure 4A) and $\widehat{\kappa }$ (Figure 4B). The observed proportion of a positive evaluation was 0.259 for Rater 2 and 0.296 for Raters 1 and 3. The estimated rater effects were not detectibly different $(P>0.55)$. Out of the 54 images, 24 were elevated based on SUVR status. The odds of a positive evaluation for scans with an elevated SUVR status were 43.467 times the odds of a scan with a non-elevated SUVR status $(P<0.01)$.

FIGURE 4.

APP study results among all raters. Observed estimates are denoted by vertical lines. (A) Plots for all pairs of two raters under Cohen’s kappa (yellow) and the model-based kappa method (green). (B) Plots for all raters under Fleiss’s kappa (red) and the model-based kappa (green).

All estimates of ${\kappa }_l$ were larger for ${\kappa }_{l,C}$ as compared to ${\kappa }_{l,M}$ (Figure 4A; Supplemental Table S5). Kappa was largest for Pair 1: ${\widehat{\kappa }}_{1,C}=0.816(95\mathrm{\%}\mathrm{C}\mathrm{I}:\mathrm{0.611,1})$ and ${\widehat{\kappa }}_{1,M}=0.705(95\mathrm{\%}\mathrm{C}\mathrm{I}:0.376,1)$. The estimates for Pairs 2 and 3 were nearly identical within each estimation method. The standard errors were smaller for ${\widehat{\kappa }}_{l,C}$ as compared to ${\widehat{\kappa }}_{l,M}$. As discussed before, the SD of Bernoulli random variables is smallest when the probability of a positive response is close to 0 or 1. The estimate of $\kappa$ encompassing all raters was larger for ${\widehat{\kappa }}_F$ than for ${\widehat{\kappa }}_M:{\widehat{\kappa }}_F=0.757(95\mathrm{\%}\mathrm{C}\mathrm{I}:0.575,0.912)$ and ${\widehat{\kappa }}_M=0.605(95\mathrm{\%}\mathrm{C}\mathrm{I}:0.345,0.816)$ (Supplemental Table S5). This is from adjusting for SUVR status.

5.2 |. Analysis of cervical cancer data

A classic example for assessing agreement among multiple raters is the interpretation of 118 samples for the presence/absence of cervical cancer from seven pathologists.^10,27,41 This dataset does not contain any additional information related to the samples or the pathologists.

In this analysis, the outcome $\left(Y_{ij}\right)$ was binary for the presence of cancer, where $Y_{ij}=1$ when pathologist $j$ interpreted sample $i$ as cancerous. Since there were no additional covariates, model $M$ was $\mathit{\text{logit}}\left({\theta }_{ij}\right)={\beta }_0+b_j$, where ${\theta }_{ij}=P\left(Y_{ij}=1\right)$, ${\beta }_0$ was the intercept, and $b_j$ was the pathologist random effect assumed to be identically and independently normally distributed for all seven pathologists (ie, $\left.b_j~N\left(0,{\sigma }_R^2\right),j=1,\dots ,7\right)$. Numerical results are provided in Supplemental Table S6 and S7 for ${\widehat{\kappa }}_l$ and $\widehat{\kappa }$, respectively.

Figure 5A shows empirical sampling distributions of ${\widehat{\kappa }}_l$. Estimates of ${\kappa }_l$ ranged from 0.215 (Pair 10) to 0.810 (Pair 20) but did not greatly change by computation method (${\kappa }_{l,M}$ vs ${\kappa }_{l,C}$; Supplemental Table S6). The greatest difference was for Pair 10, where ${\widehat{\kappa }}_{10,M}=0.215(95\mathrm{\%}CI:0.123,0.324)$ and ${\widehat{\kappa }}_{10,C}=0.234(95\mathrm{\%}CI:0.144,0.339)$, and the smallest difference was for Pair 6, where ${\widehat{\kappa }}_{6,M}=0.7943(95\mathrm{\%}CI:\mathrm{0.6750,0.9105})$ and ${\widehat{\kappa }}_{6,C}=0.7937(95\mathrm{\%}CI:0.6742,0.9098)$.

FIGURE 5.

Cervical cancer pathology study results. Observed estimates are denoted by vertical lines. (A) Results for each pair of two raters under Cohen’s kappa and the GLMM-based kappa. (B) Results among all raters under Fleiss’s kappa (red) and the GLMM-based kappa (green).

The estimates of $\kappa$, and corresponding empirical 95% confidence intervals, were similar for both methods (Supplemental Table S7): ${\widehat{\kappa }}_F=0.512(95\mathrm{\%}\mathrm{C}\mathrm{I}:0.423,0.594)$ and ${\widehat{\kappa }}_M=0.519(95\mathrm{\%}\mathrm{C}\mathrm{I}:\mathrm{0.437,0.598})$. Figure 5B shows the empirical sampling distributions of $\kappa$. Compared to the previous example, the standard errors of ${\widehat{\kappa }}_l$ and $\widehat{\kappa }$ were smaller. This is likely due to the larger number of samples and raters.¹³

The similarity in values of ${\widehat{\kappa }}_l$ and $\widehat{\kappa }$ regardless of estimation method suggests that ${\kappa }_M$ produces similar results to ${\kappa }_C$ and ${\kappa }_F$ when there is no adjustment for covariates. However, the proposed method correctly accounts for the agreement dependency structure among all pairs of two raters and provides theoretical justification for considering all pairs of raters.

6 |. DISCUSSION

An estimator for the agreement structure kappa that accounts for multiple raters and covariates was proposed. Predicted probabilities under a GLMM, including covariates as fixed effects and raters as random effects, were used to estimate ${\pi }_e$. This new approach does not require the assumption of equality of any probabilities, which is a limitation for methods commonly used. Through simulations and the amyloid-$\mathrm{\beta }$ PET analysis, we show that ignoring necessary factors leads to inflated estimates. The amyloid-$\mathrm{\beta }$ PET analysis also demonstrated that our approach can still be used even if the GLMM does not converge, while other approaches cannot. Our estimate was similar to Fleiss’s kappa in the analysis of the cervical cancer pathology data, which included no covariates.

A simulation framework for generating multiple Bernoulli observations under an agreement structure was presented. This required accounting for multivariate dependence structures across each 2-tuple pair of raters, not just a marginal dependence overall. Our framework is flexible and allows the probability of a positive response to depend on subject and rater specific factors. To our knowledge, this is the first simulation framework that uses the conditional (rather than marginal) distribution of Bernoulli observations to account for all dependent structures across and between observations. Together, the proposed method and simulation framework provide a structure for assessing more general agreement study designs, where multiple raters and covariates can be included in the analysis.

An important discussion point of this work is the understanding that Cohen’s and Fleiss’s kappa are a function of an assumed model (via ${\pi }_e$). This model is a logistic regression model including only raters as a fixed effect. Through this understanding, a model is always assumed when calculating Cohen’s and Fleiss’s kappa, whether or not that model is explicitly stated. Under the framework presented in this manuscript, a more flexible (and arguably more plausible) model for data generated from a given study can be used to estimate agreement.

The proposed method and simulation framework is different than previously developed methods. Some methods produce a distinct kappa value for each covariate pattern.¹⁵ For example, if there is a binary explanatory factor, their measure of agreement will include at least 2 kappa-like statistics. Additionally, the definitions of observed and expected agreement are not always consistent. For instance, Nelson & Edwards define expected agreement as a function of two distinct raters and subjects: $p_c=1-2P\left\{\left(y_{ij}=1\right)\cap \left(y_{i^{\prime }{j}^{\prime }}=0\right)\right\}$. Additionally, these previously developed methods may not always be applicable as they utilize the estimated variance of the raters random effects and therefore rely on the convergence of the GLMM. Their method could not be used to assess agreement in the APP study.

Many of those works utilize GEEs to estimate the marginal distribution of the binary observations. We prefer the use of a GLMM to that of a GEE for several reasons. First, the GEE framework restricts us to the population-averaged marginal inference space. Second, GLMMs provide greater flexibility when determining the covariance matrix of the random effects.³⁰ Lastly, under a conditional GLMM framework the probability of a positive response, and not the mean of the marginal distribution, is explicitly estimated. We refer the interested reader to³⁰ for further theoretical differences between GEEs and GLMMs.

There are limitations with the proposed method. Convergence of the GLMM may be challenging with a smaller number of raters. If this occurs, raters may be included as fixed effects, just as one would do for modeling in general. Another limitation is that if the model perfectly predicts the response (eg, the model is saturated), ${\widehat{\kappa }}_l$ and $\widehat{\kappa }$ will be 0. This implies two interpretations. First, reproducibility may be suspect, as saturated model results often do not generalize or perform well with other datasets. Second, if the included covariates account for all variation in the data (and the model is not saturated), then those covariates are important factors for agreement. Lastly, while the simulation framework may theoretically include any number of raters, we are limited to $R<31$, as the maximum vector length is $2^{31}-1$ in R version 4.1.1 on Windows 64-bit operating system with 16 GB RAM.

Future work includes expanding the proposed estimator and simulation framework so each rater may evaluate each subject more than once. It is clear how to generalize the estimation of ${\pi }_e$ and the simulation framework; it is less clear how to estimate ${\pi }_0$. By including replicates of all subject-rater observations, the proposed method could be compared to Nelson’s kappa-like statistic and may be used to assess intra-rater agreement (ie, rater consistency). Additionally, it is not clear how to choose the optimal number of raters and subjects. The simulation framework outlined in this manuscript may be modified to answer such questions.

In conclusion, we have developed an estimator for the agreement structure kappa that allows for multiple raters by accounting for the dependency among all pairs of two raters and adjusts for covariates. We provided a framework for simulating Bernoulli variables under the agreement structure kappa. This work demonstrates that usual methods are poised to yield invalid conclusions while our work overcomes shortfalls, leading to improved inferences.

DATA AVAILABILITY STATEMENT

Supplemental appendices and Tables are available online. The Alzheimer’s disease data that support the findings of this study may be made available from, and with the approval of, the University of Kansas Medical Center (KUMC) Alzheimer’s Disease Center (ADC). R code detailing simulations can be found on GitHub at https://github.com/CRISsupport/-Simulating-multi-rater-kappa-with-covariates.