Measuring Rater Bias in Diagnostic Tests with Ordinal Ratings

Summary

Diagnostic tests are frequently reliant upon the interpretation of images by skilled raters. In many clinical settings, however, the variability observed between experts’ ratings plays a detrimental role in the degree of confidence in these interpretations, leading to uncertainty in the diagnostic process. For example, in breast cancer testing, radiologists interpret mammographic images, while breast biopsy results are examined by pathologists. Each of these procedures involves elements of subjectivity. We propose here a flexible two-stage Bayesian latent variable model to investigate how the skills of individual raters impact the diagnostic accuracy of image-related testing in large-scale medical testing studies. A strength of the proposed model is that the true disease status of a patient within a reasonable time frame may or may not be known. In these studies, many raters each contribute classifications on a large sample of patients using a defined ordinal grading scale, leading to a complex correlation structure between ratings. Our modeling approach considers the different sources of variability contributed by experts and patients while accounting for correlations present between ratings and patients, in contrast to currently available methods. We propose a novel measure of a rater’s ability (magnifier) that, in contrast to conventional measures of sensitivity and specificity, is robust to the underlying prevalence of disease in the population, providing an alternative measure of diagnostic accuracy across patient populations. Extensive simulation studies demonstrate lower bias in estimation of parameters and measures of accuracy, and illustrate outperformance of the proposed model when compared to existing models. Receiver operator characteristic (ROC) curves are derived to assess the diagnostic accuracy of individual experts and their overall performance. Our proposed modeling approach is applied to a large breast imaging study for known disease status and a uterine cancer dataset for unknown disease status.

1 ∣. INTRODUCTION

Assessing diagnostic accuracy is an important component of evaluating the performance of a medical diagnostic test. When the medical procedure produces binary results, diagnostic accuracy can be relatively easy to assess through estimating sensitivity and specificity of the test and receiver operating characteristics (ROC) analysis^1,2,3,4,5. However, in many medical diagnostic procedures, an ordered categorical scale is used to meausre the severity of disease. For example, the Breast Imaging Reporting and Data System (BI-RADS) scale⁶ is used by radiologists for classifying breast cancer status from mammograms, and the Gustilo and Anderson scale⁷ is used for grading severity of open fractures. Simply dichotomizing the ordinal scale and applying the methods for binary data is a possible way to analyze ordinal data. However, doing this decreases a lot of data information and therefore loses accuracy when assessing the performance of the diagnostic tests.

There have been numerous methods proposed to analyze the accuracy of multirater ordinal data⁸. Such data arises in clinical studies assessing the diagnostic accuracy of an existing or new diagnostic procedure, where many experts each interpret the test results of a large sample of patients. Subsequently, variability between the multiple experts’ ratings needs to be accounted for in the modeling process. Many such methods focus on the calculation of omnibus indices, such as the kappa coefficient and all types of kappa-like coefficients^{9,10,11,12,13,14,15}, to measure agreement between raters. Obuchowski et al. (2004)¹⁶ examined several different methods for multireader (multirater) ROC studies. In the past decades, Bayesian models have also been developed for the analysis of multirater ordinal data^17,18,19 where a gold standard diagnosis is not required in any of these approaches. Cao et al. (2010)¹⁹ proposed a Bayesian hierarchical model for ordinal score data with no gold standard outcome (e.g., diseased/non-diseased) using rater bias and discrimination terms, however they focus on estimating the ranks of the latent traits rather than their values (which are our focus). Thus, they do not discuss non-identifiability of the model parameters as it is not a salient issue in estimation of the rank of the parameters. Recently, Bayesian nonparametric techniques have also been applied to analyze ordinal data^20,21,22 where no such outcome as a gold standard is considered.

In this paper, we are interested in large scale studies, where many raters contribute ordinal classifications on a large sample of items. Our planned modeling approach flexibly accommodates research studies where the true disease status of patients is known or unknown. The motivating data set was collected by Beam et al. (2003)²³ who studied variability in the interpretation of mammograms by radiologists in the United States. Each of a random sample of one hundred and seven radiologists interpreted a large random sample of mammograms using a modified ordinal five-point BI-RADS scale to record the rating results. Although the motivating data is a balanced design, our proposed method can flexibly accommodate unbalanced data with missing ratings.

For this multirater ordinal data structure, a two-stage latent trait model is proposed. Specifically, we employ a patient-level latent trait variable to link radiologist ordinal ratings with patient true disease-development binary outcomes. The first component of our modeling approach adopts a simple probit model to model the probability of being diseased, based upon a patient’s unique latent trait, with the patient’s true binary disease outcome (diseased/not diseased). For the ordinal rating component, our model builds upon Lin et al. (2018)⁵‘s binary model, where a rater’s overall diagnostic ability is modeled as diagnostic bias and magnifier. While the earlier latent trait models of Uebersax and Grove (1993)²⁴ and Johnson (1996)²⁵ assume additive effects of patient latent trait and rater effects, our model assumes that raters, based upon their ability, can magnify or shrink the impact of observed patient latent trait and then adjust with their innate diagnostic bias to evaluate the patient disease severity. Qu et al. (1996)¹ propose a latent class model with random effects in which patient latent trait is multiplied by a rater-specific coefficient; however, this model is developed for binary data.

Although the motivating mammogram dataset has the true patient disease outcomes available, our proposed method is applicable even when the patient true disease outcomes are unknown. This can be done by just fitting the model for radiologist ordinal ratings conditional on the patient latent trait variable which is drawn based on the data augmentation algorithm²⁶. The Holmquist et al. (1968)²⁷ dataset is a relevant application where seven pathologists each independently evaluated and classified lesions on 118 slides with potential carcinoma in situ of the uterine cervix into a five-category ordinal scale: from 1 (negative) to 5 (invasive carcinoma). In the simulation study, we also test our model performance under the setting when the true disease status is not known. In practice it is recommended that model robustness is carefully examined without the presence of the true disease outcomes²⁸. Our proposed two-stage latent trait model conveniently assesses diagnostic accuracy through estimating sensitivities and specificities and ROC analysis. Sensitivities and specificities can be calculated for each cut-point of the ordinal ratings. Accordingly, individual rater and an overall ROC curve for the medical test can be generated, facilitating the evaluation of the abilities of individual experts by comparison with the overall ROC curve. A Bayesian framework is adopted for estimation of the proposed ordinal latent model. For a large scale study, each rater contributes classifications on the same large sample of patients’ test results, therefore leading to a complex correlation structure between ratings. Several approaches^29,30,31 have been proposed for studies when there is strong dependence between ratings. Here, we adopt a Bayesian framework for estimation, where a Bayesian hierarchical structure and Markov chain Monte Carlo (MCMC)^32,33 computation naturally accommodate the complex correlation between ratings and patients. Through borrowing information from each other, these methods can provide robust estimation even when there are missing ratings. Not only can individual diagnostic ability (bias and magnifier) be estimated, but the overall performance of raters can also be can be naturally assessed with regard to diagnostic accuracy (e.g., ROC analysis).

The remainder of the article proceeds as follows. Section 2 proposes our Bayesian hierarchical model and discusses the conditions for identifiability. Section 3 provides details on posterior sampling for two different cases: when true disease status is known and when true disease status is unknown. In Section 4, a rigorous simulation study for estimation of the parameters is presented. In Section 5, the ROC curve based upon the proposed model is presented. Section 6 presents extensive simulation studies to compare the proposed model to existing methods. Section 7 illustrates how to check goodness-of-fit of the proposed model, while Section 7 illustrates the proposed approach using a large-scale breast screening study²³ for known disease status and a uterine cancer dataset²⁷ for unknown disease status. The paper concludes with a discussion in Section 8.

2 ∣. PROPOSED BAYESIAN HIERARCHICAL MODEL

We base our modeling on a study setting where n raters (such as radiologists) each independently classify the same set of m patient test results (such as mammograms) according to an ordinal classification scale with K categories. This results in m × n classifications of the form W_ij = k (i = 1, ⋯ , m; j = 1, ⋯ , n; k = 1, ⋯ , K). Our proposed model is a two-stage Bayesian hierarchical model that flexibly incorporates the true disease status of patients and the classifications of the raters as presented in Eq. (1) and (2), respectively:

$$P(D_i\mid u_i)=\Phi (u_i)$$ (1)

$$P(W_{ij}\le k\mid u_i)=\Phi ({\alpha }_k-(a_j+b_j{u}_i)),$$ (2)

for k = 1, ⋯ , K; i = 1, ⋯ , m; j = 1, ⋯ , n, where u_i is an unobserved continuous latent variable of the underlying latent disease severity of patient i. When the true disease status of each patient D_i is known, we define D_i = 0 if the patient does not have the disease, while D_i = 1 if the patient has the disease. The disease status may be confirmed or determined within a reasonable time interval after the patient’s image was taken. The latent variable u_i links the true disease outcome (D_i = 0 or 1) in Eq (1) with the classification assigned by the jth rater, W_ij = k in Eq (2). The rater decision W_ij is also affected by the a_j and b_j parameters. The term a_j (j = 1, ⋯ , n) refers to the intrinsic bias of a rater to assign a higher (more severe) disease score to a patient whose latent trait u_i value is 0 (i.e., an equal chance of disease or no disease). The term b_j (j = 1, ⋯, n) reflects the clinician’s diagnostic ability and skill to magnify the patient latent disease severity in the same direction (when b_j > 0).

The proposed model introduces a salient challenge in parameter identifiability by dealing with ordinal data, while building upon an earlier model for binary classification processes in Lin et al. (2018)⁵.

2.1 ∣. Identifiability of Proposed Model

The issue of identifiability commonly arises in the modeling of ordinal data^34,35. Without any restrictions on the model parameters in the proposed Bayesian latent model (e.g.a_j’s, b_j’s and u_i’s), the parameters in Eq. (1) and (2) generally cannot be point-identified as multiple parametrizations are equivalent for some of the parameters (e.g., b_j × u_i = (c × b_j) × (u_i/c) for all i = 1, ⋯ , m and j = 1, ⋯, n)³⁶. However, by fixing the posterior mean of u_i based upon the prevalence of disease in the study population via Eq. (1), we are satisfactorily able to identify all parameters in the model. Additionally, the assumption that the distribution of the rater bias terms a_j has a zero mean provides a simple and reasonable interpretation - a rater with a large positive a_j suggests this rater has a tendency to assign larger scores to patients relative to other raters. It is worth noting that we do not shift a_j’s by the mean of a_j’s in each MCMC iteration (i.e., to guarantee the mean of a_j’s is 0), but we shift the posterior samples of a_j’s after all MCMC iterations (post-processing) and shift all other posterior distributions of the relevant parameters accordingly. In this way, the joint posterior distribution of all parameters can be explored more freely without additional restrictions during the MCMC iterations. In Section 4, we demonstrate these properties through the use of simulation studies under various different scenarios.

3 ∣. POSTERIOR SAMPLING OF THE PROPOSED MODEL

3.1 ∣. When true disease status is known

In this section we develop a Bayesian algorithm for posterior sampling to fit the proposed two-stage model in Eq. (1) and (2). For efficient Bayesian computation in probit models, we introduce normal latent variables. For the disease status D_i of each patient, a normal latent variable

$$Z_{i0}\mid u_i\sim N(u_i,1)$$

is introduced. For each score by the jth rater on the ith patient’s test result W_ij, we define a normal latent variable Z_ij

$$Z_{ij}\mid u_i\sim N(a_j+b_j{u}_i,1).$$

For each pair of W_ij and Z_ij, for i = 1, ⋯ , m; j = 1, ⋯ , n, we have

$$\begin{gathered} P(Z_{ij}<{\alpha }_1\mid u_i)=P(W_{ij}=1\mid u_i) \\ P({\alpha }_1\le Z_{ij}<{\alpha }_2\mid u_i)=P(W_{ij}=2\mid u_i) \\ ⋮⋮ \\ P({\alpha }_{K-1}\le Z_{ij}\mid u_i)=P(W_{ij}=K\mid u_i). \end{gathered}$$

Then, the augmented joint density function of d = (d₁, d₂, ⋯, d_m), w = (w₁₁, w₁₂, ⋯, w_mn) and z = (z₁₀, z₁₁, z₁₂, ⋯ , z_mn) is

$$\begin{gathered} p(\mathit{d},\mathit{w},\mathit{z}\mid \mathit{a},\mathit{b},\mathit{u},\alpha )= \\ \prod _{i=1}^m\left[\varphi (z_{i0}-u_i)[I_{(z_{i0}>0)}{]}^{d_i}[I_{(z_{i0}\le 0)}{]}^{1-d_i}\prod _{j=1}^n\left(\varphi (z_{ij}-a_j-b_j{u}_i)[I_{(z_{ij}<{\alpha }_1)}{]}^{I_{(w_{ij}=1)}}[I_{({\alpha }_1\le z_{ij}<{\alpha }_2)}{]}^{I_{(w_{ij}=2)}}\cdots [I_{({\alpha }_{K-1}\le z_{ij})}{]}^{I_{(w_{ij}=K)}}\right)\right] \end{gathered}$$ (3)

We adopt Gibbs sampling³⁷, a special case of the Metropolis-Hastings algorithm, for our posterior sampling with the following prior specifications: u_i ~ N(μ_u, τ_u), a_j ~ N(0, τ_a), b_j ~ N(μ_b, τ_b), μ_b ~ N(0, τ), τ_a ~ Ga(γ, γ), τ_b ~ Ga(γ, γ), α₁ ~ Unif (−δ, δ), and α_k∣α_k−1 ~ Unif (α_k−1, α_k−1 + δ) for k = 2, ⋯ , K − 1 where τ_a and τ_b are the precision parameters of the prior distributions of a_j and b_j, respectively. Here, we assume μ_u = 0 and τ_u = τ = 1. Then, the following steps are iterated to obtain the posterior samples of the latent variables (z_i0, z_ij; i = 1, … , m, j = 1, … ,n) and the parameters a_j, b_j, u_i, μ_b, τ_a, τ_b, α_k; i = 1, ⋯ , m, j = 1, ⋯ , n, k = 1, ⋯, K):

1. Sample z_i0, for i = 1, ⋯, m,

   $$z_{i0}=\{\begin{array}{cc}N(u_i,1)I_{(z_{i0}>0)},\hfill & \text{for}{d}_i=1\hfill \\ N(u_i,1)I_{(z_{i0}\le 0)},\hfill & \text{for}{d}_i=0\hfill \end{array}\phantom{\}}$$
2. Sample z_ij, for i = 1, ⋯ , m; j = 1, ⋯, n,

   $$z_{ij}=\{\begin{array}{cc}N(a_j+b_j{u}_i,1)I_{(z_{ij}<{\alpha }_1)},\hfill & \text{for}{w}_{ij}=1\hfill \\ N(a_j+b_j{u}_i,1)I_{({\alpha }_1\le z_{ij}<{\alpha }_2)},\hfill & \text{for}{w}_{ij}=2\hfill \\ ⋮\hfill & \hfill \\ N(a_j+b_j{u}_i,1)I_{({\alpha }_{K-1}\le z_{ij})},\hfill & \text{for}{w}_{ij}=K\hfill \end{array}\phantom{\}}$$
3. Sample u_i, for i = 1, ⋯ , m,

   $$u_i^{\prime }\sim N\left(\frac{{\sum }_{j=1}^n{b}_j(z_{ij}-a_j)+z_{i0}+{\mu }_u{\tau }_u}{{\sum }_{j=1}^n{b}_j^2+1+{\tau }_u},\frac{1}{{\sum }_{j=1}^n{b}_j^2+1+{\tau }_u}\right)$$

   and set $u_i=(u_i^{\prime }+C)$ where a constant C is solved in each iteration to satisfy $P(D=1)=E(\Phi (u))\approx 1∕m{\sum }_{i=1}^m\Phi (u_i)$. See below for more details.
4. Sample a_j, for j = 1, ⋯, n,

   $$a_j\sim N\left(\frac{{\sum }_{i=1}^m(z_{ij}-b_j{u}_i)}{m+{\tau }_a},\frac{1}{m+{\tau }_a}\right)$$
5. Sample b_j, for j = 1, ⋯ , n,

   $$b_j\sim N\left(\frac{{\sum }_{i=1}^m{u}_i(z_{ij}-a_j)+{\mu }_b{\tau }_b}{{\sum }_{i=1}^m{u}_i^2+{\tau }_b},\frac{1}{{\sum }_{i=1}^m{u}_i^2+{\tau }_b}\right)$$
6. Sample μ_b,

   $${\mu }_b\sim N\left(\frac{{\tau }_b{\sum }_{j=1}^n{b}_j}{n{\tau }_b+\tau },\frac{1}{n{\tau }_b+\tau }\right)$$
7. Sample τ_a,

   $${\tau }_a\sim Ga\left(\left(\gamma +\frac{n}{2}\right),\left(\gamma +\frac{1}{2}\sum _{j=1}^n(a_j{)}^2\right)\right)$$
8. Sample τ_b,

   $${\tau }_b\sim Ga\left(\left(\gamma +\frac{n}{2}\right),\left(\gamma +\frac{1}{2}\sum _{j=1}^n(b_j-{\mu }_b{)}^2\right)\right)$$
9. Sample α_k, for k = 1, K − 1,

   $${\alpha }_k\sim Unif(L_k,U_k),$$

   where

   $$\begin{gathered} L_k=\{\begin{array}{cc}\max (-\delta ,{\alpha }_{k+1}-\delta ,\underset{i,j:w_{ij}=k}{\max }{z}_{ij}),\hfill & \text{for}k=1\hfill \\ \max ({\alpha }_{k-1},{\alpha }_{k+1}-\delta ,\underset{i,j:w_{ij}=k}{\max }{z}_{ij}),\hfill & \text{for}k=2,\cdots ,K-2\hfill \\ \max ({\alpha }_{k-1},\underset{i,j:w_{ij}=k}{\max }{z}_{ij}),\hfill & \text{for}k=K-1\hfill \end{array}\phantom{\}} \\ {U}_k=\{\begin{array}{cc}\text{min}(\delta ,{\alpha }_{k+1},\underset{i,j:w_{ij}=k+1}{\text{min}}{z}_{ij}),\hfill & \text{for}k=1\hfill \\ \text{min}({\alpha }_{k-1}+\delta ,{\alpha }_{k+1},\underset{i,j:w_{ij}=k+1}{\text{min}}{z}_{ij}),\hfill & \text{for}k=2,\cdots ,K-1\hfill \end{array}\phantom{\}} \end{gathered}$$

In Step 3, we shift the posterior samples of u_i’s to meet the condition for our model identifiability in Section 2.1: P(D) = E(Φ(u)). In each iteration, we solve the equation $P(D=1)=1∕m{\sum }_{i=1}^m\Phi (u_i^{\prime }+C)$ for C using the rootSolve R package and set $u_i=u_i^{\prime }+C$. Note that since the latent variable in our model (u_i) is continuous, a label switching issue may occur up to alternating signs (e.g., b_j × u_i = (−b_j) × (−u_i)). However, we fix the posterior mean of u_i based upon the prevalence of disease in the study population via Eq. (1) for the issue of identifiability and this label switching issue can be avoided.

For effective dissemination of our method, we developed an R package, BayesODT (available at https://lit777.github.io/BayesODT.html), that runs the MCMC algorithm described above and provides other useful features such as visualization and receiver operating characteristic curves.

3.2 ∣. When true disease status is not known

We note that this Gibbs sampling algorithm can also be implemented in a study setting where the true disease status of patients is unknown. In that case, we only assume Model (2) and omit Step 1 in the sampling procedure in Section 3.1 and Step 3 is replaced by

$$u_i^{\prime }\sim N\left(\frac{{\sum }_{j=1}^n{b}_j(z_{ij}-a_j)+{\mu }_u{\tau }_u}{{\sum }_{j=1}^n{b}_j^2+{\tau }_u},\frac{1}{{\sum }_{j=1}^n{b}_j^2+{\tau }_u}\right)$$

and set $u_i=(u_i^{\prime }-1∕m{\sum }_{i=1}^m{u}_i^{\prime })$. Note that when the true disease status is not known, the first component of the two-stage model (Eq. (1)) is not implemented and, therefore, the posterior mean of u_i is not adjusted based upon the prevalence of disease. We present examples in the simulation studies and the performance (identifiability) of our proposed approach in Section 4 for both scenarios (when the true disease status of patients is known and unknown). We also demonstrate the performance of our proposed approach with comparison to existing approaches in a cancer dataset where the true disease status of patients is unknown.

4 ∣. SIMULATION STUDY

4.1 ∣. When true disease status is known

We conduct an extensive simulation study to assess the performance of the proposed Bayesian latent variable model under a diverse range of disease prevalence rates. In this section, we focus on evaluation of identifiability and parameter estimation of our proposed model; that is, we examine estimates of a_j, b_j and u_i parameters for all i = 1, … , m and j = 1, … , n. In Section 6, we use the same simulation settings with a diverse range of additional settings to compare our model performance to other existing methods in estimation of average AUC. Table 1 presents the prevalence for each set of simulations as follows: Scenario (1) P(D = 1) = 0.05; (2) P(D = 1) = 0.25; (3) P(D = 1) = 0.5. Further, we examine both scenarios, where the true disease status of each patient is ‘known’ and ‘unknown’. In each situation we have m = 90 patients and n = 50 raters. We also test another scenario (4) with m = 148 patients and n = 100 raters for the moderate prevalence rate P(D = 1) = 0.25 that reflects the imaging application in Section 7. We set K = 4 where the ordinal classification scale has 4 categories and the vector of true cutpoints is set to (α₁ = −1, α₂ = 1, α₃ = 2). True values of a_j and b_j for each rater j = 1, ⋯ , n are independently sampled from N(0, 0.5) and N(1.7, 0.5), respectively. Also, patient’s unobserved disease severity terms (i.e., latent traits) u_i, i = 1, ⋯ , m, are randomly sampled from mixture distributions: Scenario (1) 0.95N(−1.5, 0.5²) + 0.05N(1.5, 0.5²) for P(D = 1) = 0.05; Scenario (2) 0.9N(−1, 0.5²) + 0.1N(1.5, 0.5²) for P(D = 1) = 0.25; and Scenario (3) 0.9N(0, 0.5²) + 0.1N(1.5, 0.5²) for P(D = 1) = 0.5. We also consider a non-normal mixture for u_i: (4) 0.9N(−1, 0.5²) + 0.1 Exp(1). Based on these values, true disease statuses D_i, i = 1, ⋯ , m are sampled using the model in Eq. (1). With the simulated a_j, b_j and u_i, in each dataset, the ratings W_ij, i = 1, ⋯, m; j = 1, ⋯ , n, are generated based on Eq. (2). In this simulation study, we evaluate the model performance in terms of the estimation of the quantities α_k, a_j, b_j and u_i for k = 1, 2, 3; j = 1, ⋯ , n; i = 1, ⋯ , m when we shift the mean of the a_j parameters to be zero after model fitting for easy interpretation. The simulation data are openly available in github at https://lit777.github.io/.

TABLE 1.

Various scenarios in the simulation study with m = 90 patients and n = 50 raters.

		True disease prevalence
		P(D = 1) = 0.05	P(D = 1) = 0.25	P(D = 1) = 0.50
Disease status (D)	Known	(1)	(2), (4)*†, (6)*	(3)
	Unknown		(5)*

^*indicates scenarios with larger sets of patients (m = 148) and raters (n = 100).

^†indicates scenario with a non-normal mixture model for u_i.

In the Supplementary Materials, we present Figures S1,S2 and S3 for simulation scenarios 1,2 and 3 (i.e., P(D = 1) = 0.05, 0.25 and 0.5), respectively. Figures S4 and S6 display results for simulation scenarios 4 and 6 (i.e., larger dataset), which demonstrate similar excellent parameter estimation. The results illustrate that the Bayesian estimation procedure is very successful, leading to estimates of α_k, a_j and b_j which are approximately unbiased except for a few b_j parameters which are uniformly shrunk; however those biases are only small to moderate. Also, the proposed model almost fully captures the true latent variables u_i’s supporting the robustness of our method for estimating a non-normal distribution of patient disease severity u_i’s.

4.2 ∣. Simulation scenario when true disease status is unknown

In Figure S5 in the Supplementary Materials, we examine another scenario when true disease status is not known (Scenario (5)) with setting m = 148, n = 100 and P(D = 1) = 0.25. Using the modified Gibbs sampling algorithm described in Section 3.2, we obtain the estimates of the target parameters. Compared to the estimates from scenario (4), the estimates of α_k and a_j’s are slightly more biased but the estimates of b_j’s are more consistent with the true parameter values. Since scenario (5) does not impose any restriction on u_i through the model P(D_i) = Φ(u_i), b_j parameters that form an interaction term with u_i in Eq. (2) have more freedom to explore the parameter space so that the estimates of b_j parameters in scenario (5) have higher possibility to capture the true parameter values. The simulation data that support the findings of this study are available from the corresponding author upon request.

5 ∣. ROC CURVE

5.1 ∣. Estimation of ROC Curves for Each Individual Rater

In addition to estimating the model parameters of direct interest for each rater and patient (i.e., a_j, b_j, u_i), an important goal of the proposed modeling approach is the generation of Receiver Operating Characteristic (ROC) curves³⁸ for each individual rater and the overall diagnostic procedure. ROC curves provide a graphical assessment of diagnostic accuracy by plotting the true-positive rate vs the false-positive rate. An ROC curve with a steeper curve above the 45 degree line indicates a more accurate diagnostic test while a curve closer to the 45 degree line suggests ratings that are less accurate. The estimated ROC curve of each individual rater provides valuable insight into their performance in classifying the disease status of patients based upon their personal skill and bias. Posterior estimates of ROC curves for individual raters can be estimated through simulated values from MCMC iterations. Denoting the rth sampled value from the posterior distribution for (α_k, a_j, b_j, u_i) by (${\alpha }_k^{(r)}$, $a_j^{(r)}$, $b_j^{(r)}$, $u_i^{(r)}$) (i.e., posterior samples from the rth MCMC iteration; r = 1, ⋯ , R), the conditional distribution function of W_ij

$$P(W_{ij}\le k\mid {\alpha }_k^{(r)},a_j^{(r)},b_j^{(r)},u_i^{(r)}),\text{for}k=1,\cdots ,K,$$ (4)

averaged over these values can be used to construct a Bayesian estimate for the estimated ROC curve. Specifically, for each rater j, the ROC curve is created by plotting the following points at each of the k cutpoints (k = 1, ⋯ , k)

$$\{P(W_{\cdot j}>k\mid D_{\cdot }=0),P(W_{\cdot j}>k\mid D_{\cdot }=1)\}$$

where W_·j is the average of the ratings of the j-th rater over all the patients in the study. The cumulative probability of the average rating for the j-th rater, P(W_·j ≤ k ∣ D_· = d), given the true disease status d can be estimated by

$$\begin{gathered} P(W_{\cdot j}\le k\mid D_{\cdot }=d) \\ =\frac{P(W_{\cdot j}\le k\cap D_{\cdot }=d)}{P(D_{\cdot }=d)} \\ =\frac{\int \Phi ({\alpha }_k-(a_j+b_{j}u))\Phi (u{)}^d(1-\Phi (u){)}^{(1-d)}dF(u)}{\int \Phi (u{)}^d(1-\Phi (u){)}^{(1-d)}dF(u)} \\ \approx \frac{1}{R}\sum _{r=1}^R\frac{\frac{1}{m}{\sum }_{i=1}^m\Phi ({\alpha }_k^{(r)}-(a_j^{(r)}+b_j^{(r)}{u}_i^{(r)}))\Phi (u_i^{(r)}{)}^d(1-\Phi (u_i^{(r)}){)}^{(1-d)}}{\frac{1}{m}{\sum }_{i=1}^m\Phi (u_i^{(r)}{)}^d(1-\Phi (u_i^{(r)}){)}^{(1-d)}} \end{gathered}$$ (5)

for k = 1, ⋯ , K, and d denotes true disease status (1 = disease, 0 = normal) and F(u) denotes the true distribution of the latent disease severity u. The final line in Eq (5) follows by inserting the r-th posterior samples into the models in Eq (1) and Eq (2) and averaging over all R sets of the posterior samples. Some selected estimated ROC curves for individual raters are presented in Figure 2 (a). Figure 2 (a) depicts estimated ROC curves for a selection of raters (rater ID: 32, 42, 17 and 40) along with the estimated ROC curve for all 50 raters from one simulated dataset based on scenario 2. These raters exhibited high bias a_j and high magnifier b_j (rater ID 32); high bias a_j and low magnifier b_j (rater ID 42); low bias a_j and high magnifier b_j (rater ID 17); and low bias a_j and low magnifier b_j (rater ID 40). As expected, when diagnostic magnifier b_j is high (rater ID 32 and 17), the area under the ROC curve (AUC), which is a measure of how much a more highly skilled rater (with a higher b_j value) is able to distinguish between categories in the ordinal scale, is high. The AUC for the j-th rater, AUC_j, can be expressed as³⁰

$${\text{AUC}}_j=\sum _{k=1}^{K-1}\left\{P(W_{\cdot j}=k\mid D_{\cdot }=0)\sum _{c=k+1}^{K}P(W_{\cdot j}=c\mid D_{\cdot }=1)\right\}+0.5\sum _{k=1}^{K}P(W_{\cdot j}=k\mid D_{\cdot }=0)P(W_{\cdot j}=k\mid D_{\cdot }=1),$$

where P(W_·j = k∣D_· = d) is estimated by

$$\begin{gathered} \widehat{P}(W_{\cdot j}=k\mid D_{\cdot }=d) \\ =\frac{1}{R}\sum _{r=1}^R\frac{\frac{1}{m}{\sum }_{i=1}^m\left\{\Phi ({\alpha }_k^{(r)}-(a_j^{(r)}+b_j^{(r)}{u}_i^{(r)}))-\Phi ({\alpha }_{k-1}^{(r)}-(a_j^{(r)}+b_j^{(r)}{u}_i^{(r)}))\right\}\Phi (u_i^{(r)}{)}^d(1-\Phi (u_i^{(r)}){)}^{(1-d)}}{\frac{1}{m}{\sum }_{i=1}^m\Phi (u_i^{(r)}{)}^d(1-\Phi (u_i^{(r)}){)}^{(1-d)}}. \end{gathered}$$

FIGURE 2.

(a) Estimated ROC curves and AUC values for our proposed model for four selected raters (j = 17, 32, 40, 42) and overall from one simulated dataset in Scenario 2 (see Section 5.1). (b) Estimates of the smoothed ROC curve (red) based on Eq. (7), the ROC curve based on Eq. (6) (blue), the ROC curve based on Eq. (8) (cyan) from the same simulated dataset in Scenario 2.

The AUC over all raters can be estimated similarly.

5.2 ∣. Estimation of ROC curve over all raters

A Bayesian estimate for the ROC curve over all raters can be constructed by averaging over the group of n raters (j index). This estimated ROC curve provides a valuable overall assessment of the accuracy of the underlying population of raters in the study setting after accounting for the variability between raters. The curve is generated based upon the following points at each k cutpoint (k = 1, … , K)

$$\{P(W_{\cdot \cdot }>k\mid D_{\cdot }=0)),(P(W_{\cdot \cdot }>k\mid D_{\cdot }=1)\}$$

where W_·· is the average rating over all the patients and the raters in the study. The cumulative probability of the average rating for all the raters, P(W_·· ≤ k ∣ D_· = d), given the true disease status d can be estimated by

$$\begin{gathered} P(W_{\cdot \cdot }\le k\mid D_{\cdot }=d) \\ =\frac{P(W_{\cdot \cdot }\le k\cap D_{\cdot }=d)}{P(D=d)} \\ \approx \frac{1}{R}\sum _{r=1}^R\frac{\frac{1}{n}{\sum }_{j=1}^n\frac{1}{m}{\sum }_{i=1}^m\Phi ({\alpha }_k^{(r)}-(a_j^{(r)}+b_j^{(r)}{u}_i^{(r)}))\Phi (u_i^{(r)}{)}^d(1-\Phi (u_i^{(r)}){)}^{(1-d)}}{\frac{1}{m}{\sum }_{i=1}^m\Phi (u_i^{(r)}{)}^d(1-\Phi (u_i^{(r)}){)}^{(1-d)}}. \end{gathered}$$ (6)

The overall estimated ROC curve for one simulated dataset is presented in Figure 2 (b). Figure 1 depicts three rater-level ROC curve plots to show the impact of increasing b_j for a fixed a_j value. True disease statuses (i = 1, ⋯ , 50) are simulated based on disease prevalence rate P(D = 1) = 0.25 (Scenario 2) and patient ordinal test results (k = 1, ⋯ , 4) are generated for each a_j = (−1.5, 0, 1.5) and b_j = (0.5, 1, 1.5, 2). In general, the resulting ROC curves are increasing in the b_j parameter in the sense that when b₁ > b₂ the corresponding AUC(b₁) > AUC(b₂) (except the (a_j = 0, b_j = 1) case).

FIGURE 1.

Rater-level ROC curve plots for our proposed ordinal latent variable model in Eqs (1) and (2) for different a_j and b_j values. (a) a_j = −1.5 and b_j = {0.5, 1, 1.5, 2}. (b) a_j = 0 and b_j = {0.5, 1, 1.5, 2}. (c) a_j = 1.5 and b_j = {0.5, 1, 1.5, 2}. Overall indicates the estimated ROC curve over all 4 raters for our proposed model for each a_j value.

5.3 ∣. Smoothed ROC curve

To construct a smoothed overall ROC curve based upon all raters, we can use a posterior estimate ($\widehat{a}$, $\widehat{b}$) for the means of (a_j, b_j) where $\widehat{a}.=\frac{1}{R}\frac{1}{n}{\sum }_{r=1}^R{\sum }_{j=1}^n{a}_j^{(r)}$ and $\widehat{b}.=\frac{1}{R}\frac{1}{n}{\sum }_{r=1}^R{\sum }_{j=1}^n{b}_j^{(r)}$ obtained from the Bayesian modeling of the dataset. We then evaluate the following formula which calculates $P(W_{..}\le s\mid D_{.}=d)$ for arbitrarily many categories s = 1, ⋯ , S with the corresponding cutpoints t = {t₁, ⋯ , t_S} (−∞ < t₁ < ⋯ < t_S < ∞)

$$\frac{1}{R}\sum _{r=1}^R\frac{\frac{1}{m}{\sum }_i^m\Phi (t_s-({\widehat{a}}_{\cdot }+{\widehat{b}}_{\cdot }{u}_i))\Phi (u_i{)}^d(1-\Phi (u_i){)}^{(1-d)}}{\frac{1}{m}{\sum }_i^m\Phi (u_i{)}^d(1-\Phi (u_i){)}^{(1-d)}}$$ (7)

over a fine grid of t values ranging between two bounds that ensure the above formula is approximately 0 for the lower bound of t (t₁) and 1 for the upper bound of t (t_S) (e.g., a sequence of numbers from −10 to 10 equally spaced by 0.1). This is a similar approach to that described in³⁹ in that we use a posterior estimate (the posterior mean value) of each parameter instead of inserting in the r-th sample, and evaluate the function over these posterior values. Figure 2(a) displays the estimated ROC curves for four raters and the overall ROC for a simulated dataset from Scenario 2. Figure 2 (b) displays an estimate for the smoothed ROC curve defined by Eq. (7) and an estimate for the ROC curve over all raters on the basis of Eq. (6) based upon the same simulated dataset as in Figure 2(a) based upon Scenario 2. The gaps between the different ROC curves in Figure 2 (b) are mainly induced by different ways of averaging over the posterior samples. Since the cumulative distribution function of the standard normal distribution, Φ(), is not a linear function, two approaches, (a) averaging over the posterior samples in Φ() function (smoothed ROC, Eq. (7)) vs (b) averaging over the posterior samples outside of Φ() function (ROC 1, Eq. (6)), potentially create discrepancy between the resulting estimated ROC curves. If we replace Eq. (6) with Eq. (8), which an alternative estimate for P(W_·· ≤ k ∣ D_· = d) that Eq. (6) is averaged over the posterior samples inside the cumulative distribution function, a new estimate for the ROC (ROC 2 in Figure 2) approaches the smoothed ROC

$$\begin{gathered} P(W_{\cdot \cdot }\le k\mid D_{\cdot }=d) \\ \approx \frac{1}{R}\sum _{r=1}^R\frac{\frac{1}{m}{\sum }_{i=1}^m\Phi ({\alpha }_k^{(r)}-(\frac{1}{n}{\sum }_{j=1}^n{a}_j^{(r)}+\frac{1}{n}{\sum }_{j=1}^n{b}_j^{(r)}{u}_i^{(r)}))\Phi (u_i^{(r)}{)}^d(1-\Phi (u_i^{(r)}){)}^{(1-d)}}{\frac{1}{m}{\sum }_{i=1}^m\Phi (u_i^{(r)}{)}^d(1-\Phi (u_i^{(r)}){)}^{(1-d)}}. \end{gathered}$$ (8)

6 ∣. COMPARISON TO EXISTING METHODS

In this section, we evaluate our model performance and compare results to two existing models proposed in Albert (2007)³⁰, namely a Gaussian random effects (GRE) model and a finite mixture (FM) of normals. Albert’s GRE model is a latent variable model for ordinal ratings which does not assume a known gold standard measurement is available, with the following form:

$$P(W_{ij}\le k\mid D_i,b_{D,i})=\Phi (C_{D_i,k}+b_{D_i,i}),$$

where cutpoints C_Di,k are non-decreasing for each unknown disease status D_i ∈ {0, 1} and b_Di,i is a random effect to capture the dependence in the joint conditional distribution of the vector of rater classifications, P(W_i ∣ D_i) for W_i = (W_i1, W_i2, ⋯ , W_in). Albert’s FM model is a possible alternative approach for incorporating conditional dependence between tests:

$$P({\mathit{W}}_i\mid D_i)=\{\begin{array}{cc}{\eta }_{D_i}+(1-{\eta }_{D_i}){\prod }_{j=1}^{\mathit{J}}\Phi (C_{D_i,1})\hfill & \text{if}{\mathit{W}}_i+(1,1,\cdots ,1{)}^{\prime }\text{and}{D}_i=0\hfill \\ {\eta }_{D_i}+(1-{\eta }_{D_i}){\prod }_{j=1}^{\mathit{J}}\{1-\Phi (C_{D_i,K-1})\}\hfill & \text{if}{\mathit{W}}_i=(K,K,\cdots ,K{)}^{\prime }\text{and}{D}_i=1\hfill \\ (1-{\eta }_{D_i}){\prod }_{j=1}^{\mathit{J}}\{\Phi (C_{D_i,W_{ij}})-\Phi (C_{D_i,W_{ij}-1})\}\hfill & \text{otherwise}\hfill \end{array}\phantom{\}}$$

where fractions η₀ and η₁ fall between [0, 1]. We compare the performance of our model to Albert’s GRE and FM models in terms of estimation of average AUC based upon extensive simulation scenarios: (1) four scenarios from the GRE model; (2) four scenarios from the FM model; (3) six scenarios from our simulation setup in Table 1; and finally, (4) one scenario (the simulation example in Ishwaran and Gatsonis (2000)³⁹; hereafter I&G), where I&G describes an ordinal latent variable modeling approach which allows for additional location and scale covariates for each patient i. There are nine scenarios (eight from Albert (2007) and one from our simulation scenario) for unknown disease status and six scenarios (five from our simulation scenarios and one from Ishwaran and Gatsonis (2000)) for known disease status. We generate 500 simulated dataset per scenario. For the GRE-based scenario, in cases 1 and 2, we use (C_0,1, C_0,2, C_0,3, C_0,4) = (1.0, 2.9, 4.8, 6.7) and (C_1,1, C_1,2, C_1,3, C_1,4) = (−1.5, −1.1, −0.6, −0.1). In cases 3 and 4, we use (C_0,1, C_0,2, C_0,3, C_0,4) = (0.5, 1.0, 2.0, 2.5) and (C_1,1, C_1,2, C_1,3, C_1,4) = (−1.8, −1.4, 1.2, 4.0) where σ₀ = σ₁ = 1 in all four cases. For the FM-based scenario, in cases 1 and 2, we use (C_0,1, C_0,2, C_0,3, C_0,4) = (0, 0.5, 1.0, 1.5) and (C_1,1, C_1,2, C_1,3, C_1,4) = (−1.5, −1.1, −0.6, −0.1). In cases 3 and 4, we use (C_0,1, C_0,2, C_0,3, C_0,4) = (−0.75, −0.25, 0.25, 0.75) and (C_1,1, C_1,2, C_1,3, C_1,4) = (−1.5, −1.1, −0.6, −0.1) where η₀ = η₁ = 0.2 in all four cases. For the GRE and FM-based scenarios, we assume P(D_i = 1) = 0.2. For the scenario based on I & G, two patient-specific covariates are simulated. The first binary covariate x₁ is randomly generated from (−0.5, 0.5) and the second covariate x₂ is generated from the discrete uniform distribution (−2, −1, 0, 1, 2). Additionally, two dimensional covariates u_i,k are simulated from the discrete uniform distribution (0, 1, 2) and normal distribution N(0, 0, .3²), respectively, for each i and k.

Table 3 shows results of the simulation study. As we expected, the GRE and FM models perform well under their respective scenarios (i.e., correct model specification) in terms of biases, mean squared errors (MSEs) and coverage probabilities. However, when the models are misspecified (i.e., the GRE is assumed for FM/I&G/our proposed model, or the FM model is assumed for GRE/I&G/our proposed model), estimates of the AUC for the FM, GRE and I& models are biased substantially by 0.033 to 0.222 in magnitude (gray cells indicate absolute biases larger than 0.1) and coverage probabilities are less than the nominal level of 0.95. On the contrary, our proposed model performs well over all scenarios with respect to biases, MSEs and interval widths. Especially, for the GRE-based scenarios, the biases and MSEs of our proposed model are similar to those of the GRE model, the correct model, in the first two scenarios and in the next two scenarios, our model produces much lower biases than other competing models. Our model produces coverage probabilities less than 0.95 in some scenarios, but it shows decent performance even compared to the corrected specified models (e.g., the GRE model for the GRE-based scenarios 3 and 4, the FM model for the FM-based scenarios 3). Although the GRE model is correctly specified for the 3rd and 4th GRE-based cases, the GRE shows the worst performance because of a "relabeling" of the latent variable in its likelihood as mentioned in Albert (2007) whereas our proposed model generates much smaller biases and MSEs.

TABLE 3.

Simulation results based upon 500 datasets for 8 scenarios based on Albert (2007), 1 scenario based on Ishwaran and Gatsonis [I & G] (2000), and 6 scenarios from our simulation setting in Section 4. Our proposed model is examined along with Gaussian random effect [GRE], finite mixture [FM] and Dorfman-Berbaum-Metz [DBM] methods in terms of the following metrics: averages of estimated AUCs, standard errors, 95% confidence intervals (credible intervals for our Bayesian methods), biases, MSEs and coverages (nominal level of 0.95). Gray cells indicate biases larger than 0.1 (in magnitude).

True Model Scenario	Case	Known Disease	Num. of items (m)	Num. of raters (n)	Truth		Specified Model
							GRE (Albert 2007)				FM (Albert 2007)
							Avg. of estimated AUCs (SE)	95% C.I.	Bias (MSE)	Coverage	Avg. of estimated AUCs (SE)	95% C.I.	Bias (MSE)	Coverage
GRE	(1)	No	150	7	0.90		0.90 (0.02)	0.87-0.93	0.002 (0.001)	0.98	0.94 (0.02)	0.89-0.97	0.035 (0.002)	0.52
	(2)	No	500	7	0.90		0.90 (0.03)	0.85-0.95	0.004 (0.001)	0.99	0.93 (0.01)	0.91-0.96	0.033 (0.001)	0.13
	(3)	No	50	7	0.80		0.63 (0.13)	0.44-0.85	−0.170 (0.045)	0.05	0.91 (0.05)	0.84-0.99	0.108 (0.014)	0.04
	(4)	No	500	7	0.80		0.59 (0.12)	0.46-0.82	−0.208 (0.056)	0	0.89 (0.03)	0.86-0.98	0.085 (0.008)	0.01
FM	(1)	No	150	7	0.90		0.83 (0.03)	0.77-0.88	−0.073 (0.006)	0.87	0.89 (0.02)	0.86-0.93	0.004 (0.001)	0.98
	(2)	No	500	7	0.90		0.83 (0.02)	0.79-0.86	−0.069 (0.005)	0.35	0.90 (0.01)	0.88-0.92	0.001 (0.001)	0.96
	(3)	No	150	7	0.80		0.76 (0.03)	0.69-0.79	−0.044 (0.003)	0.96	0.82 (0.04)	0.75-0.89	0.023 (0.002)	0.92
	(4)	No	500	7	0.80		0.76 (0.01)	0.73-0.78	−0.039 (0.002)	0.87	0.81 (0.03)	0.77-0.88	0.007 (0.001)	0.96
Our Scenarios	(1)	Yes	90	50	0.78		0.64 (0.06)	0.51-0.74	−0.136 (0.022)	0.14	0.96 (0.02)	0.93-0.98	0.184 (0.034)	0
	(2)	Yes	90	50	0.71		0.59 (0.06)	0.46-0.71	−0.111 (0.016)	0.27	0.93 (0.02)	0.87-0.97	0.222 (0.049)	0
	(3)	Yes	90	50	0.64		0.57 (0.09)	0.41-0.72	−0.071 (0.013)	0.53	0.80 (0.01)	0.75-0.85	0.158 (0.025)	0
	(4)	Yes	148	100	0.67		0.58 (0.03)	0.52-0.64	−0.085 (0.008)	0.35	0.86 (0.04)	0.78-0.93	0.192 (0.038)	0
	(5)	No	148	100	0.71		0.57 (0.03)	0.51-0.64	−0.135 (0.019)	0.19	0.93 (0.02)	0.88-0.96	0.222 (0.049)	0
	(6)	Yes	148	100	0.71		0.57 (0.03)	0.51-0.64	−0.135 (0.019)	0.19	0.93 (0.02)	0.88-0.96	0.222 (0.049)	0
I & G	(1)	Yes	250	3	0.79		0.84 (0.05)	0.71-0.92	0.045 (0.005)	0.76	0.96 (0.01)	0.95-0.98	0.171 (0.029)	0
True Model Scenario	Case	Known Disease	Num. of items (m)	Num. of raters (n)	Truth		Specified Model
							DBM				Our Proposed Method
							Avg. of estimated AUCs (SE)	95% C.I.	Bias (MSE)	Coverage	Avg. of estimated AUCs (SE)	95% C.I.	Bias (MSE)	Coverage
GRE	(1)	No	150	7	0.90		-	-	-	-	0.90 (0.05)	0.85-0.94	0.002 (0.003)	0.84
	(2)	No	500	7	0.90		-	-	-	-	0.91 (0.05)	0.88-0.93	0.007 (0.002)	0.74
	(3)	No	50	7	0.80		-	-	-	-	0.82 (0.04)	0.79-0.85	0.025 (0.002)	0.95
	(4)	No	500	7	0.80		-	-	-	-	0.79 (0.06)	0.78-0.81	−0.008 (0.004)	0.97
FM	(1)	No	150	7	0.90		-	-	-	-	0.86 (0.07)	0.82-0.89	−0.046 (0.007)	0.81
	(2)	No	500	7	0.90		-	-	-	-	0.86 (0.06)	0.84-0.88	−0.035 (0.001)	0.25
	(3)	No	150	7	0.80		-	-	-	-	0.78 (0.05)	0.75-0.81	−0.025 (0.003)	0.96
	(4)	No	500	7	0.80		-	-	-	-	0.80 (0.06)	0.78-0.82	−0.004 (0.004)	0.97
Our Scenarios	(1)	yes	90				0.77 (0.05)	0.65-0.86	−0.003 (0.003)	0.99	0.80 (0.05)	0.69-0.90	0.023 (0.004)	0.97
	(2)		90				0.71 (0.04)	0.62-0.79	0.001 (0.002)	0.99	0.71 (0.04)	0.65-0.79	<0.001 (0.001)	0.94
	(3)		90				0.64 (0.03)	0.58-0.70	0.007 (0.001)	0.79	0.67 (0.02)	0.64-0.70	0.035 (0.002)	0.96
	(4)		148				0.67 (0.03)	0.61-0.74	0.003 (0.001)	0.95	0.67 (0.02)	0.63-0.72	0.002 (0.001)	0.98
	(5)										0.72 (0.03)	0.67-0.78	0.009 (0.001)	0.99
	(6)	Yes	148	100	0.71		0.71 (0.03)	0.64-0.77	0.001 (0.001)	0.99	0.70 (0.03)	0.65-0.77	−0.004 (0.001)	0.93
I & G	(1)	Yes	250		0.79		0.79 (0.03)	0.74-0.84	0.002 (0.001)	0.97	0.79 (0.01)	0.79-0.81	0.007 (0.001)	0.94

In contrast to our model performing relatively well in both of Albert’s GRE and FM-based scenarios, both of Albert’s competing GRE and FM models perform poorly in our simulation scenarios illustrated in Section 4. Most of biases and MSEs for the GRE and FM models are larger than 0.1 (in magnitude) and 0.03, respectively. Even under the situation where the true disease is not known (case 5), the two models do not capture the true AUC in their 95% C.I.s while our proposed model is not affected by the absence of the true disease status. This discrepancy is mainly driven by the fact that the proposed model employs rater-specific parameters (a_j, b_j), patient-specific parameters (u_i) and their interaction b_ju_i, which create random effects for both rater and patient while Albert’s competing models are limited to only capturing the dependence structure between raters’ classifications.

For our simulation scenarios of known disease status, we additionally compare our model with the Dorfman-Berbaum-Metz Method (DBM) (1992)⁴⁰. The DBM approach performs a mixed-effects ANOVA model on jack-knife pseudovalues based on accuracy for multirater scenarios. This method is restricted to assessing accuracy when the true disease status of patients is known. Overall, we observe that the DBM method exhibits similar levels of bias to our proposed method, and lower bias than Albert’s GRE and FM methods in the scenarios examined in the simulation studies. The coverage probabilities of the DBM approach tend to be generally larger than our proposed approach.

In the last scenario from Ishwaran and Gatsonis (2000) (the simulation example in Section 6), a vector of patient and rater specific covariates is introduced into the modeling. In fact, our model has patient-specific parameters (u_i) that can incorporate a patient-specific covariate if available, rater-specific parameters (a_j, b_j) that can play a role as the vector of rater-specific covariates, and their interactions b_ju_i which allow more complex structures driven by patient-level and rater-level covariates. It can be seen numerically in the last scenario in that our model (and the DBM method) captures the true AUC with the smallest MSE while the GRE and FM models generate estimates that substantially differ from true AUC. From these extensive simulation scenarios, our method is demonstrated to be more flexible model to handle many different multirater ordinal data structure with or without the true disease status.

In conclusion, we have demonstrated through these extensive simulation studies that our method provides a more flexible modeling approach which can accommodate many different multirater ordinal data structures when the true disease status is known or unknown.

7 ∣. GOODNESS-OF-FIT TESTS

One way to assess the goodness-of-fit of the proposed Bayesian latent variable model is to use a posterior predictive p-value^41,42, the Bayesian version of evaluating the tail-area probability based on posterior simulations. Formally, the posterior predictive p-value, p_B, is defined as follows:

$$p_B=P(T({\mathit{W}}^{\text{rep}},\theta )\ge T(\mathit{W},\theta )\mid \mathit{W}),$$

where W = (W₁₁, W₁₂, ⋯ , W_mn). θ = (a, b, u, α, z) and the replicated data, ${\mathit{W}}^{\text{rep}}=(W_{11}^{\text{rep}},W_{12}^{\text{rep}},\cdots ,W_{mn}^{\text{rep}})$. Technical details are described in Gelman et al. (2013)⁴². If the model assumptions are not well-reflected in the observed data, the posterior predictive p-value has a value near 0 or 1.

To compute the posterior predictive p-value, we evaluate the following integral

$$p_B=\int \int {\mathit{I}}_{T({\mathit{W}}^{rep},\theta )\ge T(\mathit{W},\theta )}p({\mathit{W}}^{\text{rep}}\mid \theta )p(\theta \mid \mathit{W})d{\mathit{W}}^{\text{rep}}d\theta ,$$

where I is the indicator function. From the posterior samples of θ, we draw one replicated data W^rep,r for the r-th posterior sample (θ^(r), r = 1, ⋯ , R), and evaluate test quantities T(W^rep,r, θ^(r)) and T(W, θ^(r)). The posterior predictive p-value is estimated as the proportion of the total R simulations for which the predictive test quantity is larger than or equal to the realized test quantity.

In our application, we assess whether our proposed model is adequate based on the following test quantity for ordinal data

$$T(\mathit{W},\theta )=\sum _{k=1}^K\frac{(N_k-p_k\ast N{)}^2}{p_k\ast N},$$

where N_k is the number of observed W_i,j = k for i = 1, ⋯ , m; j = 1, ⋯ , n, k = 1, ⋯ = K and $N={\sum }_{k=1}^K{N}_k$. The probability p_k = P(W_i,j = k) for k = 1, ⋯ , K is estimated based on Eq. (2) and the posterior samples of the parameters. We apply this goodness-of-fit approach to two applications below in Section 8.

8 ∣. APPLICATIONS

8.1 ∣. Breast cancer study with known disease status

Despite the widespread use of mammography as a screening tool for breast cancer, issues still surround its use due to variability between radiologists’ ratings. In view of this, Beam et al. (2003)²³ undertook a large-scale breast screening study to assess the accuracy and agreement between radiologists. One hundred and forty eight mammograms were randomly obtained from a large breast cancer screening program (affiliated with the University of Pennsylvania), among those, 64 (43%) were from women with cancer. Radiologists from randomly sampled mammography facilities accredited by the U.S. Food and Drug Administration as of January 1, 1998 were invited to participate and 104 radiologists were recruited. Each rater was asked to grade each mammogram based on a modified version of the Breast Imaging Reporting and Data System (BI-RADS)⁶ ordinal classification scale: 1=normal, return to normal screening; 2=benign, return to normal screening; 3=probably benign; 4=possibly malignant, biopsy recommended; 5=probably malignant, biopsy strongly recommended. We apply our proposed model to the Beam et al. (2003)²³ dataset with the goal of assessing the accuracy of the radiologists’ ratings while considering the variability between radiologists.

First, to fit the model, we ran the Gibbs sampler with 25,000 iterations. After a burn-in of 15,000 iterations, we take every 5th draw of the sampler (thining to decrease autocorrelation) for a total of R = 2,000 iterations. The visual assessment of trace plots⁴² supports convergence of the MCMC chain. For the hyper-parameters in Section 3.1, we set γ = τ = 0.1 and δ = 15. For the condition of identifiability, we shift the sampled u_i in each MCMC iteration to have E(Φ(u_i)) = 0.43 (the prevalence disease rate in the study) by solving an equation $1∕m{\sum }_{i=1}^m\Phi (u_i^{\prime }+C)=0.43$ for C and setting $u_i=u_i^{\prime }+C$ for all i = 1, ⋯ , m in each iteration. In practice, we use the multiroot function in the R package, rootSolve, to solve the equation.

Table 2 contains the posterior summaries of the overall (hyper) parameters. The posterior mean of the mean bias for the overall raters ${\overline{a}}_j$ is −1.41 (95% C.I. [−1.46, −1.36]), however, for simpler interpretability, we shift this to 0. It is worth noting that we are not shifting ${\overline{a}}_j$’s to 0 in each MCMC iteration, but we shift it to 0 after all MCMC iterations. The posterior mean of the rater overall mean diagnostic magnifier of the group ${\overline{b}}_j$ is 0.55 (95% C.I. 0.49–0.62). The posterior means (and 95% C.I.) of four cutpoints are ${\widehat{\alpha }}_1=-0.96(-1.03,-0.90)$, ${\widehat{\alpha }}_2=-0.24(-0.29,-0.19)$, ${\widehat{\alpha }}_3=0.75(0.72,0.78)$ and ${\widehat{\alpha }}_4=2.65(2.58,2.71)$ under the post-processing step of ${\overline{a}}_j=0$. The posterior means of τ_a and τ_b are 0.46 and 80.70, implying that a_j’s are estimated with a large uncertainty because the raters have a large variation in their diagnostic biases. It is also worth noting as presented in Figure 3 that most of the raters have estimated diagnostic magnifiers, ${\widehat{b}}_j$, greater than 0.4 indicating their level of skill. We observe that rater 51’s estimated magnifier is ${\widehat{b}}_{32}=0.79$, indicating a very skilled rater who can successfully distinguish disease status according to a patient’s true disease traits (as reflected by u_i). Estimated rater biases ${\widehat{a}}_j$ vary between −1.11 and 0.96. For example, rater 45 has an ${\widehat{a}}_{45}=0.96$. This indicates that rater 45 tends to assign higher ratings on average relative to other raters in the study. Furthermore, the majority of the patients with breast cancer have estimated latent severity values greater than 0 (the black solid line in Figure 4), which suggests higher ratings are assigned to them given the fact all of the estimates for b_j are positive.

TABLE 2.

Posterior summary of (hyper-) parameters for our proposed model for the Beam et al. (2003)²³ mammogram dataset with 104 radiologists’ classifications of 148 mammograms.

	Parameter	Posterior Mean	95% Credible Interval	Note
	α1	−2.37	(−2.44, −2.31)	Ordinal cut-points
	α2	−1.65	(−1.69, −1.60)
	α3	−0.66	(−0.69, −0.63)
	α4	1.24	(1.17, 1.30)
	τa	0.46	(0.34, 0.59)	Rater bias - precision
	μb	0.55	(0.49, 0.62)	Rater magnifier - mean
	τb	80.70	(55.70, 112.01)	Rater magnifier - precision

FIGURE 3.

The Beam et al. (2003)²³ mammogram data with 104 radiologists classifications of 148 mammograms: the posterior means (95% posterior intervals) of individual a_j, b_j and u_i parameters based on our proposed model. The red stars (black circles) in the plot for u_i are for subjects with disease (no disease).

FIGURE 4.

The Beam et al. (2003)²³ mammogram data with 104 radiologists’ classifications of 148 mammograms: estimation of the distribution of patient latent disease severity u. The estimated distributions are depicted separately for the patients with D = 0 (no disease) and for the patients with D = 1 (disease) along with the estimated distribution for all patients.

In Table 4, our estimates of the rater-specific AUCs are compared with the estimates from the DBM method⁴⁰. Among 104 rater-specific AUCs, the minimum, 1st quartile, 2nd quartile, 3rd quartile and maximum values that correspond to each method are presented. The interquartile ranges of the estimated rater-specific AUCs from two methods coincide considerably. Figure 5 illustrates the posterior estimates of ROC curves for individual raters and the overall group of all 104 raters. The left plot depicts the estimated ROC curves and AUCs for rater IDs 41, 45 and 101, representing a rater with low departure a_j from the overall mean of 0 and low magnifier (low skill level) b_j (ID 41); a rater with high departure (strong bias) a_j from the overall mean of 0 and high magnifier (strong skill level) b_j (ID 45); and a rater with high departure a_j from the overall mean of 0 and low magnifier b_j (ID 101). As we discussed in the simulation study, when diagnostic magnifier is high (ID 45), the area under the ROC curve is high. In addition to the estimated ROC for rater IDs 41, 45 and 101, the empirical ROC curves for raters corresponding to the minimum, 1st quartile, 3rd quartile and maximum of the set of all 104 AUCs are presented. Figure 5(b) depicts the estimated overall summary ROC curves for the group of 104 raters derived from different methods. The estimated smoothed, non-smoothed ROC curves based upon our methods (Eq. 6-8) listed in Section 5 and the empirical overall ROC curve coincide each other. The estimated overall ROC curve based on Eq. 6 (ROC 1 in the figure) has the estimated AUC of 0.88 (SD: 01), the estimated overall ROC curve based on Eq. 8 (ROC 2 in the figure) gives the estimated AUC of 0.90 (SD: 0.01), and the empirical overall ROC curve gives the estimated AUC of 0.90 (SE: 0.04).

TABLE 4.

Estimates (standard errors) of the rater-specific AUC are obtained under different models: Gaussian random effect (GRE) and finite mixture (FM) models from Albert (2007), Dorfman-Berbaum-Metz (DBM) method from Dorfman et al. (1992), and our proposed model.

		Estimated rater-specific AUC
	Method	Min.	1st Quartile	2nd Quartile	3rd Quartile	Max.
Beam Data (Known disease status)	DBM	0.73 (0.03)	0.88 (0.02)	0.91 (0.02)	0.93 (0.02)	0.97 (0.01)
	Our Model	0.79 (0.01)	0.88 (0.01)	0.89 (0.01)	0.91 (0.01)	0.93 (0.01)

		Estimated rater-specific AUC
	Method	1	2	3	4	5	6	7
Holmquist Data (Unknown disease status)	GRE	0.87 (0.04)	0.83 (0.05)	0.89 (0.04)	0.99 (0.01)	0.86 (0.05)	0.79 (0.05)	0.95 (0.04)
	FM	0.94 (0.02)	0.89 (0.03)	0.90 (0.04)	0.94 (0.03)	0.93 (0.02)	0.85 (0.03)	1.00 (0.01)
	Our Model	0.93 (0.02)	0.91 (0.01)	0.89 (0.01)	0.89 (0.01)	0.89 (0.01)	0.85 (0.01)	0.91 (0.01)

FIGURE 5.

Different ROC curves from the Beam et al. (2003)²³ mammogram dataset. The left plot displays estimated ROC curves and corresponding AUCs for a selection of individual raters based upon our model along with empirical ROC curves for 4 different individual raters. The right plot displays the smoothed overall summary ROC curves based upon the group of 104 raters. Red curve is for the smoothed ROC based on Eq. (7). Pink curve is for the ROC based on Eq. (8). Blue curve is for the ROC based on Eq. (6).

We check the goodness-of-fit of the model in the breast imaging study data using the posterior predictive p-value method described in Section 7. From a comparison between the predictive test quantity T(w^rep, θ) and the realized test quantity T(w, θ), the estimated posterior predictive p-value is 0.22 which lies within a reasonable range (between 0.05 and 0.95⁴²), implying the model provides a reasonable fit overall.

8.2 ∣. Uterine cancer study with unknown disease status

We also consider the uterine cancer data from Holmquist et al. (1968)²⁷ where seven raters were asked to classify each of 118 slides (items) with potential carcinoma in situ of the uterine cervix based on a five category ordinal scale: 1=Negative; 2=Atypical squamous hyperplasia; 3=Carcinoma in situ; 4=Squamous carcinoma with early stromal invasion; and 5 invasive carcinoma. Within the setting of the unknown disease status, we apply our proposed model to the the Holmquist et al. (1968) dataset and compare our model performance with that of other available competing models, the FM and GRE models from Albert (2007). Table S1 in the supplementary material contains the posterior summaries of the overall (hyper) parameters. The posterior means (and 95% C.I.) of four cutpoints from our model are ${\widehat{\alpha }}_1=-1.18(-2.74,-0.35)$, ${\widehat{\alpha }}_2=0.59(-0.89,1.37)$, ${\widehat{\alpha }}_3=3.09(1.58,3.93)$ and ${\widehat{\alpha }}_4=4.57(2.98,5.53)$.

Table 4 includes our estimates of the rater-specific AUCs for all seven raters along with the estimates from the GRE and FM models. In general, they produce similar estimates of the rater-specific AUCs. Figure 6 depicts posterior estimates of ROC curves for individual raters and the estimated overall summary ROC curves for group of 7 raters derived from different methods. The estimated overall ROC curve based on Eq. 6 (ROC1 in the figure) has the estimated AUC of 0.88 (SD: 0.01) and the estimated overall ROC curve based on Eq. 8 (ROC2 in the figure) gives the estimated AUC of 0.90 (SD: 0.01). Based on the estimated posterior predictive p-value of 0.49, the proposed model has a reasonable fit for the Holmquist et al. (1968) dataset.

FIGURE 6.

Estimated ROC curves from the Holmquist et al. (1968)²⁷ uterine cancer dataset based upon our proposed model. The left plot displays estimated ROC curves and corresponding AUCs for all 7 raters. The right plot displays the smoothed overall summary ROC curves based upon the group of all 7 raters. Red curve is for the smoothed ROC based on Eq. (7). Red dotted curve is for the ROC based on Eq. (8). Black dotted curve is for the ROC based on Eq. (6).

9 ∣. SUMMARY

Assessing the diagnostic accuracy of a test procedure is challenging in large multirater studies where many raters contribute ratings on the same sample of patients’ test results. In this paper, we propose a Bayesian hierarchical two-stage latent variable model to analyze multirater ordinal rating data. The model provides novel measures for the robust estimation of rater diagnostic ability (bias and magnifier) parameters and patient latent trait values with the true disease development outcomes known or unknown. Such information can provide valuable training information for individual raters and clearly defines the degree of variability between raters in a diagnostic setting. Diagnostic accuracy is not a fixed property of a screening test and can vary between patient subgroups with their spectrum of disease, the clinical setting, and the test raters^43,44. It is essential to consider these elements when evaluating the diagnostic accuracy of a test. Our proposed novel measure of rater ability (magnifier) is providing an alternative measure of diagnostic accuracy across patient populations. An important strength of our proposed modeling approach is that, in contrast to existing models³⁰, the measures of rater ability (magnifier) and bias are robust to model misspecifications of the latent trait distribution of patients’ true disease status, and our model flexibly captures the dependencies for both patients raters and their interactions. Here, the latent trait implies an unobserved continuous variable of the underlying latent disease severity of each patient.

The idea of modeling the probability of the true binary disease status using the patients’ latent traits which provides much convenience to assess diagnostic accuracy through estimating sensitivities and specificities and ROC analysis. A further advantage of our proposed method is that a strict distribution specification for the latent trait variable is not required, in contrast to frequentist methods²⁴. Instead, the data themselves can correctly inform the distribution of the latent trait through Bayesian posterior computation. A further strength of the proposed approach is that each rater is not required to rate all patients as long as each patient has several ratings. The Gibbs sampler can be slightly modified to accommodate missing ratings. Specifically, in step 1 of the Gibbs sampler, we only need to sample z_ij’s from untruncated normal distributions for those missing ratings.

The proposed method assumes common ordinal thresholds for all the raters, which can be problematic. However, the rater-specific bias parameters (a_j’s) contribute to some degree in alleviating this limitation. Further, the proposed model’s common thresholds assumption can be relaxed to allow rater specific thresholds, although in that situation we would require a large dataset to ensure having sufficient data to estimate these many additional parameters. Overall, our proposed approach is well-suited to large-scale studies and can be applied very efficiently using our R package in a real-life study setting. Our model can also be extended to investigate the effects of significant factors on rater diagnostic skills and on patient latent traits which can be a topic for future research.

For computational convenience in our Gibbs sampling algorithm, we assign independent normal priors for rater bias and magnifier. To investigate the correlation between rater bias and rater magnifier, it is reasonable to instead assign a bivariate normal prior on them. Furthermore, for a large-scale study, some nonparametric techniques, such as assigning Dirichlet process mixture priors^19,20 on rater parameters, may be adopted to help with clustering of raters.