Improving the reliability of diagnostic tests in population-based agreement studies

Abstract

Many large-scale studies have recently been carried out to assess the reliability of diagnostic procedures, such as mammography for the detection of breast cancer. The large numbers of raters and subjects involved raise new challenges in how to measure agreement in these types of studies. An important motivator of these studies is the identification of factors that contribute to the often wide discrepancies observed between raters’ classifications, such as a rater’s experience, in order to improve the reliability of the diagnostic process of interest. Incorporating covariate information into the agreement model is a key component in addressing these questions. Few agreement models are currently available that jointly model larger numbers of raters and subjects and incorporate covariate information. In this paper, we extend a recently developed population-based model and measure of agreement for binary ratings to incorporate covariate information using the class of generalized linear mixed models with a probit link function. Important information on factors related to the subjects and raters can be included as fixed and/or random effects in the model. We demonstrate how agreement can be assessed between subgroups of the raters and/or subjects, for example, comparing agreement between experienced and less experienced raters. Simulation studies are carried out to test the performance of the proposed models and measures of agreement. Application to a large-scale breast cancer study is presented.

1. Introduction

The reliability of common medical procedures for the diagnosis of cancer and other diseases has become an important issue over the last few decades [1–4]. Subjective classifications of tests including mammograms, x-rays and biopsies are routinely carried out by physicians and other biomedical professionals, yet wide discrepancies have been observed between experts [1, 4–6]. Many studies have recently been carried out to measure the levels of agreement between experts in such settings, and to attempt to identify factors that may contribute to the observed discrepancies. Identification of influential factors provides valuable insight into how the reliability of diagnostic procedures might be improved [7–9]. These studies include, among others, a nationwide study carried out by Beam et al. [9] involving the classification of 148 mammograms by over 100 experts. Their study concluded that individual radiologists’ current reading volume was not statistically associated with the accuracy in reading screening mammograms, but several other factors were associated. Miglioretti et al. [7] examined the variability between raters in their classification of over 35 000 mammograms. They found considerable variation in the interpretative performance of diagnostic mammography that was not explained by the characteristics of the patients whose mammograms were interpreted. Thirty-eight prostate cancer biopsies were classified by 41 pathologists in a Gleason grading study [10], where barely moderate agreement between raters was found. The assessment of agreement in these studies is challenging due to the large numbers of experts and subjects under study. Further, the modeling of additional information that may lead to the identification of influential factors raises a further challenge.

Currently available methods for modeling agreement and reliability among raters can be broadly categorized as summary measures and model-based approaches. Summary statistics include the very popular Cohen’s kappa [3] and its extensions, which are often applied in biomedical studies to describe the overall level of chance-corrected agreement between two or more raters. The inclusion of covariate information when using Cohen’s kappa has also been described [11–13]. However, it is well acknowledged that Cohen’s kappa has a number of weaknesses in its usage [14, 15] and can lead to an inaccurate assessment of agreement, sometimes severely so. Cohen’s kappa is applicable when several subjects are classified by two raters. However, when multiple raters are classifying several subjects, the multirater intraclass kappa described in Kraemer et al. [16] is the appropriate summary measure of reliability to use. In general, summary statistics provide a useful overall measure of agreement, but sustain a large loss of information due to outputting a single value about the agreement within a process.

Model-based approaches provide a more complete and broader framework for assessing factors influencing agreement, and thus perhaps improving reliability. These include log-linear models [17–21], latent class and trait models [20–24], and logistic regression models [25, 26]. Several of these approaches include the raters as fixed effects, and provide inference to the specific raters and items under study, not extending to agreement involving entire populations of such raters. Such methods work well when a small number of raters is involved, but become increasingly complex, frequently involving a large number of parameters when the classifications of more than two or three raters are included. Log-linear models and latent models can incorporate covariate information; however, few methods currently available are able to both incorporate covariate information and yield inference regarding the underlying diagnostic procedure.

For common medical procedures, it is desirable to make inference regarding the levels of agreement in the underlying populations of raters and subjects who are typically involved in the diagnostic procedure of interest, and to examine the influence of factors on the agreement process in such a setting. A number of approaches have been developed with this purpose in mind, including a population definition for the intraclass kappa [27], a flexible latent variable model proposed by Williamson and Manatunga [23] for ordinal classifications, and more recently, a generalized linear mixed model for modeling binary classifications [28], which is designed to examine agreement in a population-based setting. While Williamson and Manatunga’s approach can potentially include any number of raters, a fixed effect term is included for each individual rater in the model and is thus best-suited to a small to a moderate number of raters.

Clinical factors are often associated with the prevalence of a disease and may influence the reliability of diagnostic procedures. In the breast cancer setting, a mammogram presenting an advanced stage breast cancer is easier to distinguish than many less advanced forms of breast cancer, leading to a better agreement between the raters. Prior knowledge regarding a subject’s clinical history could also influence a rater’s perception of the mammogram and consequently their classification. For example, in the classification of mammograms, the age of the woman is an important factor as the prevalence of breast cancer increases with a woman’s age [29]. Other factors such as a rater’s level of experience and type of training could affect the levels of agreement present.

The earlier paper by Nelson and Edwards [28] proposed a simple population-based agreement model and summary statistic that focuses on inference regarding agreement in a diagnostic procedure for large numbers of raters and subjects (assumed randomly sampled from their populations), where each rater in the sample of raters is assumed to rate each subject independently of the other raters. This paper focused on defining the general concepts of measuring reliability, and also how the class of generalized linear mixed models could be used as a suitable framework in such a setting. Attention was focused on the more theoretical aspects underlying the simplest model proposed through the use of probit and logit link functions.

In this paper we extend the simple population-based agreement model and summary measure of agreement proposed by Nelson and Edwards [28] to the measurement of agreement between subgroups of raters and/or subjects, for example, how agreement between experienced raters compares with that of less experienced raters, by the incorporation of important covariate information into the model. Such models can lead to the identification of factors that influence agreement and the reliability of the diagnostic procedure, where consistency between raters is an important prerequisite for a reliable procedure. In agreement models, the inclusion of covariate information requires careful consideration due to its influence on both the prevalence of the disease under study and on the agreement process, and yields valuable information regarding factors that influence both how a rater classifies a subject and the agreement between the raters.

The remainder of the paper is as follows: Section 2 introduces the basic population-based agreement model described in Nelson and Edwards [28] and presents the extensions to the model. In Section 3, the associated population-based summary measure of agreement and an extended version are described. Simulation studies are carried out in Section 4 to examine the effects of including important factors into the proposed model and the agreement summary statistic described in Sections 2 and 3. Application to a breast cancer study involving the classifications of a large set of mammograms is presented in Section 5. In Section 6 concluding remarks and discussion are made.

2. Models and measures of agreement

A natural choice for modeling agreement data when the underlying populations of raters and subjects and the diagnostic procedure are of interest is the class of generalized linear mixed models with a crossed random effect structure. We restrict our attention here to classifications made on a binary scale, for example, y_ij = 1 if the ith subject is rated as positive (for example, diseased) by the jth rater, and y_ij = 0 otherwise. It is assumed that the raters and the subjects are randomly selected from their respective populations, and that each of the J raters classify all of the subjects under study (independently from the other J − 1 raters). Although binary outcomes can be modeled using a generalized linear mixed model with either a probit or logit link function, among others, we will focus here on the probit link function for ease of mathematics. Nearly identical results are achieved for the logistic link function [28]. The basic form of the agreement model is

$${\mathrm{\Phi }}^{-1}(p_{ij})=\eta +u_i+v_j,$$ (1)

for subject i = 1, …, I and rater j = 1, …, J. The quantity p_ij = pr(y_ij = 1) is the probability of the ith item being classified as positive by the jth rater, and the constant η refers to the intercept term of the model and can also be regarded as a measure of the prevalence of positives in the data. When η is large, the overall frequency of positives in the data is high. The terms u_i and v_j represent random effects for the ith subject and jth rater respectively, with assumed independent Normal $(0,{\sigma }_u^2)$ and Normal $(0,{\sigma }_v^2)$ distributions. A subject with a positive (negative) random effect is more (less) likely than other such subjects to be classified as positive over many raters. A large value of ${\sigma }_u^2$ is indicative of subjects that are easy to distinguish from one another. A rater with a positive (negative) random effect is more liberal (cautious) in classifying a subject as positive over many subjects. A large value of ${\sigma }_v^2$ suggests more variability between the raters within the population in how they classify the subjects.

2.1. Inclusion of covariates

Many agreement models, including log-linear and logistic regression models, utilize an agreement index as the response, where 0 = two raters disagree, 1 = two raters agree, in their classification of a single item [17, 25, 26, 30]. In the generalized linear mixed model considered here, and in Williamson and Manatunga’s agreement model [23], the response variable y_ij, instead, is the binary classification made on the ith subject by the jth rater. Then when a covariate is included into the model, careful consideration has to be given as to whether it is likely to directly impact the prevalence so that it is included as a fixed effect, or more likely to influence the agreement between the raters and is then included as a random effect term. Covariates that influence both the prevalence and the agreement can be included as both fixed and random effects.

Earlier studies [5, 7, 9, 10] demonstrated that certain factors including a radiologist’s average volume of mammogram interpretations, a woman’s age and time since previous mammogram, a physician’s length of practice and type of training may influence how a rater classifies an item, and may affect the agreement between the raters. For example, raters who carry out more classifications on a routine basis may develop more experience and accuracy in reading mammograms, leading to a higher rate of agreement between the experienced raters. Factors related to subject (for example, stage of cancer) would lead to using a test in some clinical populations and not in others. Factors related to raters would change the form of the test in the same population. The basic agreement model in equation (1) is extended to

$${\mathrm{\Phi }}^{-1}(p_{ij})=\eta +{\mathit{\beta }}_{\mathit{1}}^{\prime }{\mathit{x}}_i+{\mathit{\beta }}_{\mathit{2}}^{\prime }{\mathit{x}}_j+{\mathit{z}}_1^{\prime }{\mathit{u}}_i+{\mathit{z}}_2^{\prime }{\mathit{v}}_j,\hspace{1em}i=1,\dots ,I,j=1,\dots ,J,$$ (2)

where x_i (r × 1) is the vector of factors associated with the ith subject, and x_j (s × 1) the vector of fixed effects associated with the jth rater. The associated vectors of parameters are β₁ (r × 1) and β₂ (s × 1), respectively. The vectors z₁ (p × 1) and z₂ (q × 1) represent the design vectors of the random effects for the random effect vectors u_i (p × 1) and v_j (q × 1), respectively, where u_i and v_j contain the associated random effects for the subjects and the raters. The random effects are assumed to follow multivariate normal distributions

$$\mathit{u}~\mathrm{M}\mathrm{V}\mathrm{N}(\mathbf{0},{\mathbf{\Sigma }}_{\mathit{u}})\text{ and }\mathit{v}~\mathrm{M}\mathrm{V}\mathrm{N}(\mathbf{0},{\mathbf{\Sigma }}_{\mathit{v}}),$$

where the covariance matrices Σ_u and Σ_v are of dimensions p × p and q × q, respectively. More complex random effect structures can be employed if required and if sufficient data are available. The random effect vectors u_i and v_j can be predicted to describe the unique effects of each rater and subject included in the study.

3. A population-based measure of agreement

Cohen’s kappa is a very popular summary measure of agreement due to its appealing simplicity of both calculation and interpretation. In this section, we describe and extend a simple population-based summary measure of agreement introduced in Nelson and Edwards [28] to include covariate information associated with the raters and subjects to try and improve the reliability of a diagnostic test, and to determine their influence on the prevalence and agreement. It is demonstrated how the extended summary statistic can be used to compare the agreement between the subgroups of raters and/or subjects, for example, the amount of agreement between the experienced and the inexperienced raters. The summary statistic is comparable to Cohen’s kappa in style and interpretation, avoiding many of the weaknesses observed in Cohen’s kappa use, while allowing for population-based inference. An advantage of the model-based kappa statistic is that all available data is utilized to estimate the parameters, even when the agreement between the sub-groups are of interest. A general overall measure of agreement based upon all the raters and subjects included in a study can be obtained by fitting the simplest form of the generalized linear mixed model presented in equation (1).

The simplest form of the model-based kappa measure of agreement as introduced in Nelson and Edwards [28] is

$${\kappa }_{\mathrm{m}}=1-4{\int }_{-\infty }^{\infty }\mathrm{\Phi }\left(\frac{z\sqrt{\rho }}{\sqrt{1-\rho }}\right)[1-\mathrm{\Phi }(\frac{z\sqrt{\rho }}{\sqrt{1-\rho }}\left)\right]\varphi (z)\mathrm{d}z,$$ (3)

where $\rho ={\sigma }_u^2/({\sigma }_u^2+{\sigma }_v^2+1)$ and 0≤κ_m≤1; the maximum likelihood estimates of the variances ${\sigma }_u^2$ and ${\sigma }_v^2$ are obtained from the corresponding generalized linear mixed model in equation (1). We can interpret the numerical value of our summary statistic in a similar manner to Cohen’s kappa, where a value close to 1 is suggestive of strong agreement, such that as given in Landis and Koch [31]. The approximate asymptotic variance of κ̂_M is derived using the multivariate delta theorem, and is estimated as

$$\widehat{\text{var}}({\hat{\kappa }}_{\mathrm{m}})\approx 16[{\left\{{\int }_{-\infty }^{\infty }\right(\frac{1}{2\hat{\rho }(1-\hat{\rho })}\left(\frac{z\sqrt{\hat{\rho }}}{\sqrt{1-\hat{\rho }}}\right)\varphi \left(\frac{z\sqrt{\hat{\rho }}}{\sqrt{1-\hat{\rho }}}\right)[1-2\mathrm{\Phi }(\frac{z\sqrt{\hat{\rho }}}{\sqrt{1-\hat{\rho }}}\left)\right])\varphi (z)\mathrm{d}z\}}^2\times [\left(\frac{{\hat{\sigma }}_v^2-{\hat{\sigma }}_w^2}{{({\hat{\sigma }}_T^2)}^2}\right)\left(\frac{2{\hat{\sigma }}_u^4}{I}\right)+{\left(\frac{{\hat{\sigma }}_u^2}{{({\hat{\sigma }}_T^2)}^2}\right)}^2\left(\frac{2{\hat{\sigma }}_v^4}{J}\right)\left]\right].$$

3.1. Inclusion of covariates

In equation (4), we present a measure of agreement (extended from the original κ_m in equation (3)) between two raters j and j′ each classifying the ith subject, which accounts for the fixed effects vector x_j, where the two raters may have different values for at least one of the covariates of interest contained in x_j. For example, κ_m in equation (4) provides a summary measure of agreement between two randomly chosen raters, who have different amounts of experience in reading mammograms.

$${\kappa }_{\mathrm{m}}=1-4{\int }_{-\infty }^{\infty }\mathrm{\Phi }\left(\frac{z\sqrt{\rho }+\frac{{\beta }_1^{\prime }{x}_i+{\beta }_2^{\prime }{x}_j}{\sqrt{{\sigma }_T^2}}}{{(1-\rho )}^{1/2}}\right)[1-\mathrm{\Phi }(\frac{z\sqrt{\rho }+\frac{{\beta }_1^{\prime }{x}_i+{\beta }_2^{\prime }{x}_{j\prime }}{\sqrt{{\sigma }_{T\prime }^2}}}{{(1-\rho \prime )}^{1/2}}\left)\right]\varphi (z)\mathrm{d}z$$ (4)

where

$$\rho ={\sigma }_1^2/({\sigma }_1^2+{\sigma }_2^2+1)\text{ and }\rho \prime ={\sigma }_1^2/({\sigma }_1^2+{\sigma }_{2\prime }^2+1).$$ (5)

The terms ${\sigma }_1^2$ and ${\sigma }_2^2$ represent the variances of the sums of the random effect components ${\mathit{z}}_1^{\prime }{\mathit{u}}_i$ and ${\mathit{z}}_2^{\prime }{\mathit{v}}_j$ for the ith subject and jth rater, respectively, that is, the variability associated with the rater and the subject random effects such that ${\sigma }_1^2=\text{var}(z_1^{\prime }{u}_i)=z_1^{\prime }{\mathrm{\Sigma }}_u{z}_1$ and ${\sigma }_2^2=\text{var}(z_2^{\prime }{v}_j)=z_2^{\prime }{\mathrm{\Sigma }}_v{z}_2$. Similarly, ${\sigma }_{2\prime }^2=\text{var}(z_2^{\prime }{v}_j^{\prime })$ for the j’th rater. The quantity ${\sigma }_T^2={\sigma }_1^2+{\sigma }_2^2+1$ is a measure of the total variability present in the model (T for total), given the covariate values of the ith subject and the jth rater. Similarly, ${\sigma }_T^2$ is the total variability in the model given the covariate values of the ith subject and the j’th rater. In equation (5) an extended version of ρ is presented. It is defined as a measure of subject distinguishability relative to the variability between two raters with the same covariate information, given the ith subject’s covariate information. Similarly, the term ρ′ specifies the value of ρ for another rater j′ who may have a different set of covariate values from rater j, thus leading to a different value of ${\sigma }_2^2$, and consequently ρ. The formula for ρ′ is presented in equation (5).

With the inclusion of covariates, the exact form of the variance of κ_m is dependent upon the random effect design vectors z₁ and z₂ and the assumed correlation structures of the random effects. The model-based kappa statistic κ_m is a function of the variance components contained in Σ_u and Σ_v, and the regression coefficient vectors β₁ and β₂, where vector θ contains all of the individual components. The asymptotic variance of κ_m can then be obtained using the multivariate delta theorem, and estimated as

$$\widehat{\text{var}}({\hat{\kappa }}_{\mathrm{m}})=16\text{var}\left({\int }_{-\infty }^{\infty }\mathrm{\Phi }\right(\frac{z\sqrt{\hat{\rho }}+\frac{{\hat{\beta }}_1^{\prime }{x}_i+{\hat{\beta }}_2{x}_i}{\sqrt{{\hat{\sigma }}_T^2}}}{{(1-\hat{\rho })}^{1/2}}\left)\right[1-\mathrm{\Phi }\left(\frac{z\sqrt{\hat{\rho }}+\frac{{\hat{\beta }}_1^{\prime }{x}_i+{\hat{\beta }}_2{x}_{j\prime }}{\sqrt{{\hat{\sigma }}_{T\prime }^2}}}{{(1-\hat{\rho }\prime )}^{1/2}}\right)]\varphi (z)\mathrm{d}z)$$ (6)

$$=\frac{16}{IJ}(\mathit{h}{\mathbf{\Sigma }}^{-1}\mathit{h}\prime )\text{ where }\mathit{h}=(\frac{\delta {\kappa }_{\mathrm{m}}}{\delta {\theta }_1},\dots ,\frac{\delta {\kappa }_{\mathrm{m}}}{\delta {\theta }_l})$$ (7)

is the vector of derivatives of κ_m with respect to the variance components contained in the Σ_u and Σ_v matrices and β₁ and β₂, and Σ (l × l) is the variance–covariance matrix of all the variance components and β contained in θ. An application of this equation is presented in the Appendix.

4. Simulation studies

Simulation studies were carried out to examine the effects of including factors of interest on the estimation in the models and measures of agreement described. The simulations were based upon three probit generalized linear mixed models of increasing complexity as follows:

$$\text{Model }(\mathrm{a}):{\mathrm{\Phi }}^{-1}(p_{ij})=\eta +u_i+v_j,$$

$$\text{Model }(\mathrm{b}):{\mathrm{\Phi }}^{-1}(p_{ij})=\eta +\beta x_j+u_i+v_j,$$

$$\text{Model }(\mathrm{c}):{\mathrm{\Phi }}^{-1}(p_{ij})=\eta +\beta x_j+u_i+v_j+z_2{v}_{1j},$$

where i = 1, …, I, j = 1, …, J with I and J each set at 50. The term η is the intercept and the response variable y_ij represents the classification made by the jth rater on the ith subject, equaling 0 for a subject classified as not diseased and 1 otherwise. The random effects u_i and v_j in models (a) and (b) are assumed to be normally distributed as $\mathrm{N}(0,{\sigma }_u^2)$ and $\mathrm{N}(0,{\sigma }_v^2)$, respectively. The additional random effect v_1j in model (c) is associated with a factor z₂ likely to influence the agreement between the raters, such as a rater’s level of experience (z₂ = 1 for an experienced rater, and 0 otherwise). For simulation purposes, z_1j is randomly generated from a Bin(n = 1, p = 0.5) distribution, j = 1, …, J, and the random effect term v_1j is also assumed to be normally distributed with variance ${\sigma }_{v1}^2$ and correlated with the other rater random effect v_j. The covariance matrix for model (c) thus takes the form

$$\left(\begin{array}{c}\hfill u_i\hfill \\ \hfill v_j\hfill \\ \hfill v_{1j}\hfill \end{array}\right)~\mathrm{M}\mathrm{V}\mathrm{N}(\mathbf{0},[\begin{array}{ccc}\hfill {\sigma }_u^2\hfill & \hfill 0\hfill & \hfill 0\hfill \\ \hfill 0\hfill & \hfill {\sigma }_v^2\hfill & \hfill {\rho }_v{\sigma }_v{\sigma }_{v_1}\hfill \\ \hfill 0\hfill & \hfill {\rho }_v{\sigma }_v{\sigma }_{v_1}\hfill & \hfill {\sigma }_{v_1}^2\hfill \end{array}\left]\right).$$ (8)

In models (b) and (c), x_j represents a fixed covariate value for the jth rater, (for example, the experience level of the jth rater (1 = high, 0 = low). In the simulations, x_j is randomly generated from a Bin(1,0.5) distribution. Two different values for the intercept, η = 1 and 3, were included in the simulations. The regression coefficient β was set at 0.5. Different values of the variance components were also included to assess the effects of increasing variability: for all three models, ${\sigma }_u^2,{\sigma }_v^2$ and ${\sigma }_{v_1}^2$ were set at (1,1,1) and (5,5,5), and for model (c), the correlation ρ_v between random effects v_j and v_1j was set at 0.25. Due to the computational intensity involved, the number of simulations was restricted to the fitting of 100 randomly generated data sets for each simulation scenario.

Each data set was fitted using a Monte-Carlo expectation–maximization algorithm (MCEM) developed by McCulloch [32] (see also Kuk and Cheng [33]) to obtain almost-exact maximum likelihood estimates of the parameters in the generalized linear mixed model. A description of this algorithm is provided in Nelson and Edwards [28].

Starting values of the parameters in the vector θ were set at η = 0.5, β = 0.05 and the random effect parameters, ${\sigma }_u^2,{\sigma }_v^2,{\sigma }_{v1}^2$ and ρ_v were all set at 0.5 for simplicity. Other sets of starting values were also tested, and resulted in the same estimated parameters in each case. Convergence criteria within the algorithm was set at max(|θ_k − θ_k−1|<0.0001, where k = 1, …, K is the number of parameters to be estimated, and θ contains the parameters to be estimated.

The model-based kappa statistic was calculated for each individual data set, based upon the estimated values of the parameters from the corresponding generalized linear mixed model, and for model (a), a version of Cohen’s kappa for multiple raters [31] was calculated.

Tables I and II display the mean estimates of the parameters and their associated standard errors from the simulation studies are described. We observe that the parameters η and β are estimated on average with little bias in each of the 12 models examined. Similarly, the almost-exact maximum likelihood mean estimates of the variance components ${\sigma }_u^2$ and ${\sigma }_v^2$ are nearly unbiased, although there is a slight increase in the level of bias observed for the variance components for model (c). The mean estimates of the correlation coefficient ρ_v are underestimated for each simulation scenario for model (c), sometimes severely so. In the simplest model (a), the mean estimated Cohen’s kappa is only slightly lower in value than the corresponding model-based kappa statistic when the prevalence term η = 1; however, the mean estimated Cohen’s kappa is noticeably smaller than the mean estimated model-based kappa for η = 3 (Cohen’s κ̂ = 0.09, κ̂_m = 0.21) and the associated standard errors are large enough to suggest that there may be no significant difference between these two estimated measures of agreement.

Table I.

Simulation results for the probit generalized linear mixed model and κ_m based upon 100 data sets for I = 50 and J = 50 for two different sets of parameter values $\mathit{\theta }=(\eta ,\beta ,{\sigma }_u^2,{\sigma }_v^2{\sigma }_{v_1}^1,{\rho }_v)$.

Parameter	True value	Model (a)	Model (b)	Model (c)
(i) η = 1, ${\sigma }_{u_0}^2={\sigma }_v^2={\sigma }_{v_1}^2=1$
η	1	0.95 (0.17)	0.99 (0.24)	1.02 (0.36)
β	0.5	—	0.46 (0.25)	0.48 (0.41)
${\sigma }_u^2$	1	0.97 (0.20)	0.94 (0.26)	0.94 (0.21)
${\sigma }_v^2$	1	1.02 (0.23)	1.09 (0.31)	0.97 (0.32)
${\sigma }_{v_1}^2$	1	—	—	1.37 (0.46)
ρv01	0.25	—	—	−0.0004 (0.001)
κ		0.19 (0.04)
κm		0.21 (0.03)
(i) η = 1, ${\sigma }_{u_0}^2={\sigma }_{v_0}^2={\sigma }_{v_1}^2=5$
η	1	0.89 (0.29)	0.71 (0.48)	1.049 (0.50)
β	0.5	—	0.38 (0.67)	0.43 (0.66)
${\sigma }_u^2$	5	4.90 (0.84)	4.65 (0.98)	4.66 (0.92)
${\sigma }_v^2$	5	5.26 (1.26)	5.09 (1.36)	4.28 (0.82)
${\sigma }_{v_1}^2$	5	—	—	6.64 (2.24)
ρv	0.25	—	—	0.0004 (0.005)
κ		0.28 (0.05)
κm		0.29 (0.04)

The three models fitted are: (a) Φ⁻¹(p_ij) = η+u_i+v_j, (b) Φ⁻¹(p_ij) = η+βx_j+u_i+v_j and (c) Φ⁻¹(p_ij) = η+βx_j+u_i+v_j+z₂v_1j; x_i ~ Bin(1,0.5), z₂ ~ Bin(1,0.5). Cohen’s kappa = κ and model-based kappa = κ_m. Mean parameter estimates are presented with associated standard errors in parentheses.

Table II.

Simulation results for the probit generalized linear mixed model and model-based kappa statistic κ_m based upon 100 data sets for I = 50 and J = 50 using two different sets of parameter values $\mathit{\theta }=(\eta ,\beta ,{\sigma }_u^2,{\sigma }_v^2{\sigma }_{v_1}^1,{\rho }_v)$.

Parameter	True value	Model (a)	Model (b)	Model (c)
(ii) η =3, ${\sigma }_{u_0}^2={\sigma }_{v_0}^2={\sigma }_{v_1}^2=1$
η	3	2.95 (0.25)	3.05 (0.41)	3.02 (0.41)
β	0.5	—	0.51 (0.40)	0.42 (0.08)
${\sigma }_u^2$	1	1.00 (0.37)	1.05 (0.30)	0.76 (0.30)
${\sigma }_v^2$	1	1.09 (0.38)	1.14 (0.36)	1.3 (0.52)
${\sigma }_{v_1}^2$	1	—	—	1.25 (0.94)
ρv	0.25	—	—	−0.004 (0.002)
κ		0.09 (0.04)
κm		0.21 (0.05)
(ii) η = 3, ${\sigma }_{u_0}^2={\sigma }_{v_0}^2={\sigma }_{v_1}^2=5$
η	3	2.93 (0.33)	2.87 (0.49)	2.71 (0.29)
β	0.5	—	0.45 (0.56)	0.54 (0.31)
${\sigma }_u^2$	5	5.00 (1.12)	4.44 (1.04)	3.99 (0.56)
${\sigma }_v^2$	5	5.39 (1.51)	4.34 (0.51)	4.44 (1.59)
${\sigma }_{v_1}^2$	5	—	—	5.21 (3.25)
ρv	0.25	—	—	0.003 (0.01)
κ		0.24 (0.06)
κm		0.29 (0.05)

The three models fitted are: (a) Φ⁻¹(p_ij) = η+u_i+v_j, (b) Φ⁻¹(p_ij) = η+βx_j+u_i+v_j and (c) Φ⁻¹(p_ij) = η+βx_j+u_i+v_j+z₂v_1j; x_i ~ Bin(1,0.5), z₂ ~ Bin(1,0.5). Cohen’s kappa = κ and model-based kappa = κ_m. Mean parameter estimates are presented with associated standard errors in parentheses.

5. Application to a breast cancer study

An agreement study was carried out by Beam et al. [9] where 148 randomly selected mammograms were classified by a large number of physicians randomly selected from a group of 294 physicians from the USA. The mammograms included both diseased and non-diseased cases. Data on a number of covariates, including the subject’s age, the number of mammograms read in the previous year by each rater and the number of years of experience of the raters, were collected. The subjects’ ages ranged from 40 to 85 years. The classifications were made using the BIRADS scale, and have been dichotomized here so that an outcome of 0 represents a mammogram classified as non-diseased, and 1 as diseased. Full details on the data collection can be found in Beam et al. [9]. Data on 104 randomly sampled physicians were analyzed using the models and measure of agreement described in Sections 2 and 3 and a summary of the pairwise agreement is presented in Table III. The age of each woman on whom the mammograms was taken was included as an indicator variable x_i, where x_i = 1 for a subject less or equal to 60 years of age. The level of experience of a physician was included as an indicator variable z₂ = 1 if a physician had 10 or more years experience of rating mammograms and 0 otherwise. These are two examples of models that can be fit to this data set; there are other models that could also be feasible.

Table III.

Summary of the pairwise agreement between 104 randomly selected physicians each independently classifying 148 slides for the presence (y_ij = 1) or absence (y_ij = 0) of breast cancer [9].

	Physician B
	Category	Non-diseased	Diseased	Total
Physician A	Non-diseased	460 951	64 531	525 482
	Diseased	74 467	192 739	267 206
	Total	535 418	257 270	792 688

Two generalized linear mixed models with a probit link function were fitted to this data set using McCulloch’s MCEM algorithm; one model (i) to assess the overall agreement present between all the raters and subjects included in the study, and a second model (ii) to assess agreement after accounting for a subject’s age and each rater’s level of experience. These models are

$$\text{Model }(\mathrm{i}):{\mathrm{\Phi }}^{-1}(p_{ij})=\eta +u_i+v_j,$$

$$\text{Model }(\mathrm{i}\mathrm{i}):{\mathrm{\Phi }}^{-1}(p_{ij})=\eta +\beta x_i+u_i+v_j+z_2{v}_{1j}.$$

Details on the forms of ρ_v, var(κ̂_m) and the likelihood function L(θ, y) for model (ii) are presented in the Appendix.

Table IV presents the parameter estimation and model-based and Cohen’s kappa statistics for these two models. We observe a negative intercept for both models, reflecting the fact that over half of the mammograms were classified as not having cancer present. The negative regression coefficient for subject’s age suggests that the odds in favor of a younger patient (less than 60 years) being classified as having a diseased mammogram is approximately 45 per cent of that of an older patient (over 60 years old). The variability observed between the subjects $({\sigma }_u^2)$ is larger than the variability observed between the raters $({\sigma }_v^2)$. Cohen’s kappa for overall agreement between all raters was estimated at κ̂ = 0.60, whereas the model-based kappa κ̂_m = 0.53 (se = 0.08). The agreement between highly experienced raters classifying mammograms of younger women (z₂ = 1 and x_i = 1) is κ̂_m = 0.53 (se = 0.07), whereas chance-corrected agreement between less experienced raters is estimated as κ̂_m = 0.50 (se = 0.08), suggesting little difference between these two estimated measures of agreement.

Table IV.

Results for the breast cancer data set.

Parameter	Model (a)	Model (b)
η	−0.83 (0.15)	−0.13 (0.02)
β	—	−0.37 (0.03)
${\sigma }_u^2$	3.54 (0.45)	3.45 (0.40)
${\sigma }_v^2$	0.25 (0.04)	0.25 (0.03)
${\sigma }_{v_1}^2$	—	0.24 (0.003)
ρv	—	0.001 (0.01)
Cohen’s kappa κ	0.60
Model-based kappa κm	0.53 (0.08)
κm(z2 = 0, xi = 1)	—	0.50 (0.08)
κm(z2 = 1, xi = 1)	—	0.53 (0.07)

The two models fitted are: (a) Φ⁻¹(p_ij) = η+u_i+v_j and (b) Φ⁻¹(p_ij) = η+βx_i+ u_i+v_j+z₂v_1j; x_i is an indicator variable for ith subject’s age (1 for subjects less than or equal to 60 years of age, 0 for subjects greater than 60 years of age). The term z₂ = 0 for an inexperienced rater, and 1 for an experienced rater. Parameter estimates are presented with standard errors in parentheses.

6. Discussion

In this paper, we have extended a model-based approach and measure of agreement, which are appropriate for use in large studies with the goal of improving diagnostic tests that wish to examine the reliability between many raters each classifying the same set of many subjects. This population-based approach, which is based upon the class of generalized linear mixed models allows for inference regarding the underlying diagnostic procedure, and for conclusions to be made regarding the populations of raters and subjects who are typically involved in the medical testing procedure of interest.

Extending the basic model and measure of agreement presented by Nelson and Edwards [28] to include covariate information that may affect both the prevalence of the disease and the agreement process yields valuable information about the roles of raters and subjects. For example, the significance of the level of experience of raters when classifying test items, such as mammograms, can be more closely examined. Other information can also be easily incorporated in this modeling approach, such as factors related to the subject’s clinical history. The use of generalized linear mixed models allows for missing observations and unbalanced data, and easily incorporates large numbers of raters and subjects, unlike most other methods for assessing agreement. Specific details regarding the performance of individual raters included in the study can also be obtained easily by estimating the random effects.

Obtaining almost-exact maximum likelihood estimates of the parameters in the generalized linear mixed models requires the use of a computationally intensive algorithm, such as McCulloch’s MCEM algorithm [32] or an equivalent [33]. At present, software packages do not have the capacity to obtain almost-exact maximum likelihood estimates for generalized linear mixed models with crossed random effects structures. The model-based kappa statistic can also be calculated using standard software such as R and SAS; a computer function written in the freely available software package R included in Table V.

Table V.

Function in R for calculating the two-rater model-based kappa statistic and its variance.

twokappafn = function(sigma2u, beta1)

{

integrand1 = function(z)

{

term1=pnorm(sqrt(sigma2u)*z − beta1)

term2= pnorm(sqrt(sigma2u)*z + beta1)

fullintegrand= term1*term2*dnorm(z)

}

result1 = integrate (integrand1, lower=−100, upper=100)

integrand2=function(z)

{

term3=(1-pnorm(sqrt(sigma2u)*z − beta1))

term4=(1-pnorm(sqrt(sigma2u)*z + beta1))

fullintegrand2=term3*term4*dnorm(z)

}

result2= integrate(integrand2, lower=−100, upper=100)

res2=result2 $value

# two-rater model-based kappa

2*(res1 + res2)−1

}

# Calculation of variance: need to enter values for varbeta1, varsigmasqu, sigmasqu and beta1.

integrand1a = function(z)

{term=(0.5*(1/sqrt(sigmasqu))*dnorm(sqrt(sigmasqu)*z + beta1/2)*pnorm(sqrt(sigmasqu)*z − beta1/2)

+ 0.5*(1/sqrt (sigmasqu))*pnorm(sqrt(sigmasqu)*z + beta1/2)*dnorm(sqrt(sigmasqu)*z − beta1/2))

*dnorm(z)*z }

result1a=integrate(integrand1a, lower=−100, upper=100)

integrand1b = function(z)

{term=(−0.5*(1/sqrt(sigmasqu))*dnorm(sqrt(sigmasqu)*z + beta1/2)*(1-pnorm(sqrt(sigmasqu)*z − beta1/2))*z

−0.5*(1/sqrt(sigmasqu))*pnorm(sqrt(sigmasqu)*z + beta1/2)*dnorm(sqrt(sigmasqu)*z − beta1/2))

*dnorm(z)*z}

result1b=integrate(integrand1b, lower=−100, upper=100)

vectorh1=result1a$value+result1b$value

integrand1b = function(z)

{term=(0.5*dnorm(sqrt(sigmasqu)*z + beta1/2)*pnorm(sqrt(sigmasqu)*z − beta1/2) − 0.5*pnorm(sqrt(sigmasqu)*z

+ beta1/2)*dnorm(sqrt(sigmasqu)*z − beta1/2))*dnorm(z)}

result2a=integrate(integrand2a, lower=−100, upper=100)

integrand2b = function(z)

{term=(−0.5*dnorm(sqrt(sigmasqu)*z + beta1/2)*(1-pnorm(sqrt(sigmasqu)*z − beta1/2))

+ 0.5*(1-pnorm(sqrt(sigmasqu)*z + beta1/2))*dnorm(sqrt(sigmasqu)*z − beta1/2))*dnorm(z)}

result2b=integrate(integrand2b, lower=−100, upper=100)

vectorh2=result2a$value+result2b$value

covarmat = matrix(c(varsigmasqu,0,0,varbeta1), ncol=2)

vectorh = c(vectorh1, vectorh2)

varkappam = 4*(vectorh %*% covarmat %*% vectorh)

}

As is the case for any class of models where the outcomes of interest are binary, larger data sets are required when more factors are to be included into the model, since more parameters are required to be estimated. This also ensures successful convergence of the MCEM algorithm in the parameter estimation process.

It is assumed in the current model setting that the random effects of the subjects and raters are normally distributed. Previous research [34] has shown that for a generalized linear mixed model with a Poisson link function, fixed effect parameters are estimated with little or no bias, whereas variance components are estimated with more bias and variability when non-normal random effects are modeled assuming normality.

Further extensions to the model include multiple independent classifications of each item by the individual raters. This can be useful when we wish to examine the variability of ratings made by any rater for a particular item, also known as ‘intra-rater’ variability. This can be flexibly included in the framework of the generalized linear mixed model and summary statistic proposed, and is a topic for future research. Other future directions for extending this class of models in the assessment of agreement include development of a model to allow for ordinal classification scales, and to investigate the inclusion of non-normal random effect distributions for the raters and items.

APPENDIX A

DETAILS ON MODEL FITTING FOR APPLICATION

For model (i), the random effects u_i and v_j, i = 1, …, I and j = 1, …, J, are assumed to be normally distributed with zero means and variances ${\sigma }_u^2$ and ${\sigma }_v^2$, respectively. The random effects in model (ii) are assumed to be distributed as in Section 5. For model (ii), the term ρ takes the form

$$\rho =\frac{{\sigma }_1^2}{({\sigma }_2^2+{\sigma }_1^2+1)}=\frac{{\sigma }_u^2}{{\sigma }_u^2+{\sigma }_v^2+z_2^2{\sigma }_{v_1}^2+2z_2{\rho }_v{\sigma }_v{\sigma }_{v1}+1}$$

and ${\sigma }_T^2={\sigma }_u^2+{\sigma }_v^2+z_1^2{\sigma }_{v_{ij}}^2+2z_1{\rho }_v{\sigma }_v{\sigma }_{v1}+1$. The statistic κ̂_m is calculated using the estimated quantities, ${\hat{\sigma }}_u^2,{\hat{\sigma }}_v^2,{\hat{\sigma }}_{v1}^2$ and ρ̂ from the fitted model. To calculate the variance for κ_m, let $\mathit{\theta }=({\sigma }_u^2,{\sigma }_v^2,{\sigma }_{v1}^2,{\rho }_v,\beta )$. Since κ_m is a function of θ, the multivariate delta theorem can be applied directly. The variance of κ̂_m, var(κ̂_m) can be obtained using equation (7), where vector h takes the following form:

$$\mathit{h}=(\frac{\delta {\kappa }_{\mathrm{m}}}{\delta {\sigma }_u^2},\frac{\delta {\kappa }_{\mathrm{m}}}{\delta {\sigma }_v^2},\frac{\delta {\kappa }_{\mathrm{m}}}{\delta {\sigma }_{v1}^2},\frac{\delta {\kappa }_{\mathrm{m}}}{\delta {\rho }_v},\frac{\delta {\kappa }_{\mathrm{m}}}{\delta \beta }).$$

The matrix Σ is of dimension 5 × 5 since the vector θ contains five elements, and takes the form

$$\mathbf{\Sigma }=-{\left(E\right[\frac{{\delta }^2\log L}{\delta \mathit{\theta }\delta \mathit{\theta }}\left]\right)}^{-1},$$

where the likelihood function of interest here is

$$L(\mathit{\theta };\mathit{y})=\underset{i=1}{\overset{I}{\mathrm{\Pi }}}\underset{j=1}{\overset{J}{\mathrm{\Pi }}}{p}_{ij}^{y_{ij}}{(1-p_{ij})}^{1-y_{ij}}\frac{1}{2\pi {\sigma }_v{\sigma }_{v_1}\sqrt{1-{\rho }^2}}\text{exp }\left\{\frac{-1}{2(1-{\rho }_v^2)}\right(\frac{v_j^2}{{\sigma }_v^2}+\frac{v_{1j}^2}{{\sigma }_{v1}^2}-\frac{2{\rho }_v{v}_j{v}_{1j}}{{\sigma }_v{\sigma }_{v1}}\left)\right\}\times \frac{1}{\sqrt{2\pi {\sigma }_u^2}}\text{exp }\left\{\frac{-u_i^2}{2{\sigma }_u^2}\right\}$$

with p_ij = Φ(η+βx_i+u_i+v_j+z₂v_1j).