TABLE 3.
Simulation results based upon 500 datasets for 8 scenarios based on Albert (2007), 1 scenario based on Ishwaran and Gatsonis [I & G] (2000), and 6 scenarios from our simulation setting in Section 4. Our proposed model is examined along with Gaussian random effect [GRE], finite mixture [FM] and Dorfman-Berbaum-Metz [DBM] methods in terms of the following metrics: averages of estimated AUCs, standard errors, 95% confidence intervals (credible intervals for our Bayesian methods), biases, MSEs and coverages (nominal level of 0.95). Gray cells indicate biases larger than 0.1 (in magnitude).
| True Model Scenario |
Case | Known Disease |
Num. of items (m) |
Num. of raters (n) |
Truth | Specified Model | ||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| GRE (Albert 2007) | FM (Albert 2007) | |||||||||||||
| Avg. of estimated AUCs (SE) | 95% C.I. | Bias (MSE) | Coverage | Avg. of estimated AUCs (SE) | 95% C.I. | Bias (MSE) | Coverage | |||||||
| GRE | (1) | No | 150 | 7 | 0.90 | 0.90 (0.02) | 0.87-0.93 | 0.002 (0.001) | 0.98 | 0.94 (0.02) | 0.89-0.97 | 0.035 (0.002) | 0.52 | |
| (2) | No | 500 | 7 | 0.90 | 0.90 (0.03) | 0.85-0.95 | 0.004 (0.001) | 0.99 | 0.93 (0.01) | 0.91-0.96 | 0.033 (0.001) | 0.13 | ||
| (3) | No | 50 | 7 | 0.80 | 0.63 (0.13) | 0.44-0.85 | −0.170 (0.045) | 0.05 | 0.91 (0.05) | 0.84-0.99 | 0.108 (0.014) | 0.04 | ||
| (4) | No | 500 | 7 | 0.80 | 0.59 (0.12) | 0.46-0.82 | −0.208 (0.056) | 0 | 0.89 (0.03) | 0.86-0.98 | 0.085 (0.008) | 0.01 | ||
| FM | (1) | No | 150 | 7 | 0.90 | 0.83 (0.03) | 0.77-0.88 | −0.073 (0.006) | 0.87 | 0.89 (0.02) | 0.86-0.93 | 0.004 (0.001) | 0.98 | |
| (2) | No | 500 | 7 | 0.90 | 0.83 (0.02) | 0.79-0.86 | −0.069 (0.005) | 0.35 | 0.90 (0.01) | 0.88-0.92 | 0.001 (0.001) | 0.96 | ||
| (3) | No | 150 | 7 | 0.80 | 0.76 (0.03) | 0.69-0.79 | −0.044 (0.003) | 0.96 | 0.82 (0.04) | 0.75-0.89 | 0.023 (0.002) | 0.92 | ||
| (4) | No | 500 | 7 | 0.80 | 0.76 (0.01) | 0.73-0.78 | −0.039 (0.002) | 0.87 | 0.81 (0.03) | 0.77-0.88 | 0.007 (0.001) | 0.96 | ||
| Our Scenarios | (1) | Yes | 90 | 50 | 0.78 | 0.64 (0.06) | 0.51-0.74 | −0.136 (0.022) | 0.14 | 0.96 (0.02) | 0.93-0.98 | 0.184 (0.034) | 0 | |
| (2) | Yes | 90 | 50 | 0.71 | 0.59 (0.06) | 0.46-0.71 | −0.111 (0.016) | 0.27 | 0.93 (0.02) | 0.87-0.97 | 0.222 (0.049) | 0 | ||
| (3) | Yes | 90 | 50 | 0.64 | 0.57 (0.09) | 0.41-0.72 | −0.071 (0.013) | 0.53 | 0.80 (0.01) | 0.75-0.85 | 0.158 (0.025) | 0 | ||
| (4) | Yes | 148 | 100 | 0.67 | 0.58 (0.03) | 0.52-0.64 | −0.085 (0.008) | 0.35 | 0.86 (0.04) | 0.78-0.93 | 0.192 (0.038) | 0 | ||
| (5) | No | 148 | 100 | 0.71 | 0.57 (0.03) | 0.51-0.64 | −0.135 (0.019) | 0.19 | 0.93 (0.02) | 0.88-0.96 | 0.222 (0.049) | 0 | ||
| (6) | Yes | 148 | 100 | 0.71 | 0.57 (0.03) | 0.51-0.64 | −0.135 (0.019) | 0.19 | 0.93 (0.02) | 0.88-0.96 | 0.222 (0.049) | 0 | ||
| I & G | (1) | Yes | 250 | 3 | 0.79 | 0.84 (0.05) | 0.71-0.92 | 0.045 (0.005) | 0.76 | 0.96 (0.01) | 0.95-0.98 | 0.171 (0.029) | 0 | |
| True Model Scenario |
Case | Known Disease |
Num. of items (m) |
Num. of raters (n) |
Truth | Specified Model | ||||||||
| DBM | Our Proposed Method | |||||||||||||
| Avg. of estimated AUCs (SE) | 95% C.I. | Bias (MSE) | Coverage | Avg. of estimated AUCs (SE) | 95% C.I. | Bias (MSE) | Coverage | |||||||
| GRE | (1) | No | 150 | 7 | 0.90 | - | - | - | - | 0.90 (0.05) | 0.85-0.94 | 0.002 (0.003) | 0.84 | |
| (2) | No | 500 | 7 | 0.90 | - | - | - | - | 0.91 (0.05) | 0.88-0.93 | 0.007 (0.002) | 0.74 | ||
| (3) | No | 50 | 7 | 0.80 | - | - | - | - | 0.82 (0.04) | 0.79-0.85 | 0.025 (0.002) | 0.95 | ||
| (4) | No | 500 | 7 | 0.80 | - | - | - | - | 0.79 (0.06) | 0.78-0.81 | −0.008 (0.004) | 0.97 | ||
| FM | (1) | No | 150 | 7 | 0.90 | - | - | - | - | 0.86 (0.07) | 0.82-0.89 | −0.046 (0.007) | 0.81 | |
| (2) | No | 500 | 7 | 0.90 | - | - | - | - | 0.86 (0.06) | 0.84-0.88 | −0.035 (0.001) | 0.25 | ||
| (3) | No | 150 | 7 | 0.80 | - | - | - | - | 0.78 (0.05) | 0.75-0.81 | −0.025 (0.003) | 0.96 | ||
| (4) | No | 500 | 7 | 0.80 | - | - | - | - | 0.80 (0.06) | 0.78-0.82 | −0.004 (0.004) | 0.97 | ||
| Our Scenarios | (1) | yes | 90 | 0.77 (0.05) | 0.65-0.86 | −0.003 (0.003) | 0.99 | 0.80 (0.05) | 0.69-0.90 | 0.023 (0.004) | 0.97 | |||
| (2) | 90 | 0.71 (0.04) | 0.62-0.79 | 0.001 (0.002) | 0.99 | 0.71 (0.04) | 0.65-0.79 | <0.001 (0.001) | 0.94 | |||||
| (3) | 90 | 0.64 (0.03) | 0.58-0.70 | 0.007 (0.001) | 0.79 | 0.67 (0.02) | 0.64-0.70 | 0.035 (0.002) | 0.96 | |||||
| (4) | 148 | 0.67 (0.03) | 0.61-0.74 | 0.003 (0.001) | 0.95 | 0.67 (0.02) | 0.63-0.72 | 0.002 (0.001) | 0.98 | |||||
| (5) | 0.72 (0.03) | 0.67-0.78 | 0.009 (0.001) | 0.99 | ||||||||||
| (6) | Yes | 148 | 100 | 0.71 | 0.71 (0.03) | 0.64-0.77 | 0.001 (0.001) | 0.99 | 0.70 (0.03) | 0.65-0.77 | −0.004 (0.001) | 0.93 | ||
| I & G | (1) | Yes | 250 | 0.79 | 0.79 (0.03) | 0.74-0.84 | 0.002 (0.001) | 0.97 | 0.79 (0.01) | 0.79-0.81 | 0.007 (0.001) | 0.94 | ||