A Comparison of Denominator Degrees of Freedom Methods for Multiple Observer ROC Analysis

Abstract

There are several methods available for analyzing multireader ROC studies that generalize results to both the reader and case populations. Two of these methods - the Dorfman-Berbaum-Metz (DBM) method and the Obuchowki-Rockette (OR) method - appear to be quite different in their original formulations. However, recently it has been shown that the DBM and OR procedures yield the same test statistic when based on the same accuracy measure and covariance estimation method, but inferences can vary depending on which denominator degrees of freedom (ddf) method, DBM or OR, is used. I show in simulations that there are problems with both ddf methods: OR is ultraconservative with significance levels considerably below the nominal level, and DBM can result in extremely wide confidence intervals because the ddf can be close to zero. I propose a new ddf method that overcomes both of these problems and can be used with either the OR or DBM procedure.

1 INTRODUCTION

Receiver operating characteristic (ROC) curve analysis is a well established method for evaluating and comparing the performance of diagnostic tests [1-5]. There are several methods available for analyzing multireader ROC studies that generalize results to both the reader and case populations. Five such methods have recently been described and compared in an analysis of three data sets by Obuchowski et al [6] The study shows that the methods can lead to different conclusions, illustrating the need for more theoretical and empirical investigation of the methods to provide insight and guidance regarding the appropriateness of each procedure in a given situation.

Two of the these methods - the Dorfman-Berbaum-Metz (DBM) method [7, 8] and the Obuchowski-Rockette (OR) method [9, 10] - appear to be quite different in their original formulations. However, recently it has been shown by Hillis et al [11] that the DBM and OR procedures yield the same test statistic when based on the same accuracy measure and covariance estimation method, but inferences depend on which denominator degrees of freedom (ddf) method, DBM or OR, is used.

In simulations I find that there are problems with both ddf methods: OR is ultraconservative with significance levels considerably below the nominal level while DBM, though much closer to the nominal significance level, sometimes results in extremely wide confidence intervals because the ddf can be close to zero. I propose using a new ddf estimator that overcomes these problems and can be used with either the DBM or OR procedure.

The outline of the paper is as follows. I describe the OR method in Section 2. In Section 3 I derive the new ddf from the OR model, discuss the rationale for the original OR ddf, and compare the new ddf with the DBM ddf. In Section 4 I show how the new ddf can be derived from the DBM model and discuss the possibility that the DBM ddf may be based on an inadequate Satterthwaite approximation. In Section 5 I discuss confidence intervals for test differences, inferences for single tests, and fixed readers analysis. In Section 6 I compare the performance of the new ddf method with the OR and DBM ddf methods in a simulation study. I illustrate and compare the three ddf methods using a previously published data set in Section 7 and make concluding remarks in the final section.

2 THE OBUCHOWSKI-ROCKETTE (OR) METHOD

2.1 Design and notation

A commonly used study design in radiology that allows generalization to both the reader and case populations is the test reader case factorial design, where each case (i.e., patient) undergoes each diagnostic test (e.g., CT scan, MRI, or X-ray) and the resulting images are evaluated once by each reader (usually a radiologist). Typically the number of cases is 25-200 while the number of readers is 3-15. Throughout I assume that the data have been collected using this factorial design.

Let Z_ijk denote the rating assigned to the kth case by the jth reader using the ith modality. For example, a five-level ordinal integer scale is commonly used with higher rating values indicating a higher level of confidence that the case is diseased; thus a rating value of 1 might correspond to “definitely not diseased,” 5 to “definitely diseased,” with other values corresponding to intermediate assessments. Alternatively, a quasi continuous 0% to 100% confidence scale is often used. The observed data consists of the Z_ijk, with i = 1,..., t, j = 1,..., r, k = 1,..., c, where t is the number of tests (or modalities), r the number of readers, and c the number of cases. In addition, each case is classified as diseased or nondiseased according to an available reference standard.

2.2 Model and test statistic

The corrected F test was proposed by Obuchowski and Rockette [9]. Let ${\widehat{\theta }}_{ij}$ denote the AUC estimate (or other accuracy estimate) for the ith test and jth reader. b Their approach is to use a test reader ANOVA model for the AUCs, but unlike a conventional ANOVA model they allow the errors to be correlated to account for correlation in the AUCs due to each reader evaluating the same cases. Thus their two-way ANOVA model corresponds to the three-way study design. For the factorial design with only one replication, the corrected F test model, which I refer as the OR model, can be written as

$${\widehat{\theta }}_{ij}=\mu +{\tau }_i+R_j+{\left(\tau R\right)}_{ij}+{∊}_{ij}$$ (1)

i = 1,..., t, j = 1,..., r, where τ_i denotes the fixed effect of test i; R_j denotes the random effect of reader j, (τR)_ij denotes the random test reader interaction, and ∊_ij is the error term. The R_j and (τR)_ij are assumed to be mutually independent and normally distributed with zero means and variances ${\sigma }_R^2$, reflecting differences in reader ability, and ${\sigma }_{TR}^2$, reflecting test-by-reader interaction. The ∊_ij are assumed to be normally distributed with zero mean and constant variance ${\sigma }_{∊}^2$, which represents variability attributable to cases and within-reader variability that describes how a reader interprets the same image in different ways on different occasions. The ∊_ij are independent of the R_j and (τR)_ij. However, since the same cases are read by each reader using each test, the ∊_ij are not assumed to be independent. Instead, equi-covariance of the errors between readers and tests is assumed, resulting in three possible covariances given by

$$\mathrm{Cov}({∊}_{ij},{∊}_{i^{\prime }{j}^{\prime }})=\{\begin{array}{cc}{\mathrm{Cov}}_1\hfill & i\ne i^{\prime },j=j^{\prime }(\text{different test},\text{same reader})\hfill \\ {\mathrm{Cov}}_2\hfill & i=i^{\prime },j\ne j^{\prime }(\text{same test},\text{different reader})\hfill \\ {\mathrm{Cov}}_3\hfill & i\ne i^{\prime },j\ne j^{\prime }(\text{different test},\text{different reader})\hfill \end{array}\phantom{\}}$$

It follows from model (1) that ${\sigma }_{∊}^2$, Cov₁, Cov₂, and Cov₃ are also the variance and corresponding covariances of the AUC estimates, conditional on the reader and test×reader effects. Obuchowski and Rockette [9] suggest the following ordering for the covariances:

$${\mathrm{Cov}}_1\ge {\mathrm{Cov}}_2\ge {\mathrm{Cov}}_3\ge 0.$$ (2)

I only consider the factorial design with one replication; however, my results will also apply when there are multiple replications.

The corrected F statistic for testing the null hypothesis of no test effect (H₀: τ₁ = τ₂ = ... = τ_t) is given by

$$F^{\ast }=\frac{\mathrm{MS}\left(\mathrm{T}\right)}{\mathrm{MS}\left({\mathrm{T}}^{\ast }\mathrm{R}\right)+r({\mathrm{Cov}}_2-{\mathrm{Cov}}_3)}$$ (3)

where $\mathrm{MS}\left(\mathrm{T}\right)=\frac{r}{t-1}{\sum }_{i=1}^t{({\widehat{\theta }}_{i\cdot }-{\widehat{\theta }}_{\cdot \cdot })}^2$ and $\mathrm{MS}\left({\mathrm{T}}^{\ast }\mathrm{R}\right)=\frac{1}{(t-1)(r-1)}{\sum }_{i=1}^t{\sum }_{j=1}^r{({\widehat{\theta }}_{ij}-{\widehat{\theta }}_{i\cdot }-{\widehat{\theta }}_{\cdot j}+{\widehat{\theta }}_{\cdot \cdot })}^2$. A subscript replaced by a dot indicates that values are averaged across the missing subscript; for example, ${\widehat{\theta }}_{i\cdot }=\frac{1}{r}{\sum }_{j=1}^r{\widehat{\theta }}_{ij}$ and ${\widehat{\theta }}_{\cdot \cdot }=\frac{1}{tr}{\sum }_{i=1}^t{\sum }_{j=1}^r{\widehat{\theta }}_{ij}$. Obuchowski and Rockette [9] state that F ^* has an approximate null $F_{t-1;{\mathrm{ddf}}_{\mathrm{o}}}$ distribution, where

$${\mathrm{ddf}}_{\mathrm{o}}=(t-1)(r-1)$$ (4)

Throughout, the ddf subscript denotes the first author of the paper that proposed the ddf method: O = Obuchowski, D = Dorfman, H = Hillis.

In practice we do not know Cov₂ or Cov₃ and have to estimate them from the data or use estimates from previous studies. Thus the statistic actually used is

$$F_{\mathrm{OR}}=\frac{\mathrm{MS}\left(\mathrm{T}\right)}{\mathrm{MS}\left({\mathrm{T}}^{\ast }\mathrm{R}\right)+\max [r({\widehat{\mathrm{Cov}}}_2-{\widehat{\mathrm{Cov}}}_3),0]}$$ (5)

where ${\widehat{\mathrm{Cov}}}_2$ and ${\widehat{\mathrm{Cov}}}_3$ denote estimates for Cov₂ and Cov₃, respectively. Note that equation (5) incorporates constraint (2) by setting ${\widehat{\mathrm{Cov}}}_2-{\widehat{\mathrm{Cov}}}_3$ to zero if it is negative. Since Cov₂ and Cov₃ are also the corresponding covariances of the AUC estimates conditional on the reader and test×reader effects, they can be estimated using ROC analysis methods that treat cases as random but readers as fixed. For example, for trapezoidal-rule AUC estimates [5], Obuchowski and Rockette [9] suggest estimating ${\sigma }_{∊}^2$, Cov₁, Cov₂ and Cov₃ by averaging corresponding variances and covariances computed using the method of DeLong et al [12], which provides covariance and variance estimates treating only cases as random. More generally, any acceptable method for estimating the AUC variance and covariances that treats readers as fixed can be used, such as jackkni-ng or bootstrapping. The OR estimates obtained from averaging the corresponding fixed-reader AUC variances and covariances are denoted by ${\widehat{\sigma }}_{∊}^2,{\widehat{\mathrm{Cov}}}_1,{\widehat{\mathrm{Cov}}}_2$, and ${\widehat{\mathrm{Cov}}}_3$.

3 THE PROPOSED DDF

In this section I show that F ^* has an approximate F _{( t -1),df2} null distribution, where

$${\mathrm{df}}_2=\frac{{[E\left(\mathrm{MS}\left({\mathrm{T}}^{\ast }\mathrm{R}\right)\right)+r({\mathrm{Cov}}_2-{\mathrm{Cov}}_3)]}^2}{\frac{{\left[E\left(\mathrm{MS}\left({\mathrm{T}}^{\ast }\mathrm{R}\right)\right)\right]}^2}{(t-1)(r-1)}}$$ (6)

In deriving (6) I treat Cov₂ and Cov₃ as known. An estimate of df₂ that incorporates constraint (2) is given by

$${\mathrm{ddf}}_{\mathrm{H}}=\frac{{\{\mathrm{MS}\left({\mathrm{T}}^{\ast }\mathrm{R}\right)+\max [r({\widehat{\mathrm{Cov}}}_2-{\widehat{\mathrm{Cov}}}_3),0]\}}^2}{\frac{{\left[\mathrm{MS}\left({\mathrm{T}}^{\ast }\mathrm{R}\right)\right]}^2}{(t-1)(r-1)}}$$ (7)

Thus I propose using ddf_H as the ddf for F _OR, since F _OR approximates F ^*. I also discuss why Obuchowski and Rockette suggested (t-1) (r-1) as the ddf.

3.1 Derivation of ddf_H

The expected values for MS(T), MS(R), and MS(T^*R) are derived in the Appendix and given in Table 1, where $\mathrm{MS}\left(\mathrm{R}\right)=\frac{t}{r-1}{\sum }_{j=1}^r{({\widehat{\theta }}_{\cdot j}-{\widehat{\theta }}_{\cdot \cdot })}^2$. It follows from Table 1 that

$$E\left[\mathrm{MS}\left(\mathrm{T}\right)\right]=\{\begin{array}{cc}E\left[\mathrm{MS}\left({\mathrm{T}}^{\ast }\mathrm{R}\right)\right]+r({\mathrm{Cov}}_2-{\mathrm{Cov}}_3),\hfill & \text{if}{\mathrm{H}}_0\hfill \\ \frac{r}{t-1}\sum _{i=1}^t{({\tau }_i-{\tau }_{\cdot })}^2+E\left[\mathrm{MS}\left({\mathrm{T}}^{\ast }\mathrm{R}\right)\right]+r({\mathrm{Cov}}_2-{\mathrm{Cov}}_3),\hfill & \text{if}{\mathrm{H}}_1\hfill \end{array}\phantom{\}}$$ (8)

where H₀ is the null hypothesis of no test effect (H₀:τ₁ = ... = τ_t = 0). Thus the numerator and denominator of F ^* in equation (3) have the same expected value under H₀, while the numerator has a larger expected value if H₀ does not hold; hence F ^* is an appropriate ANOVA test statistic. [ Table 1 ]

Table 1.

Expected mean squares for the Obuchowski-Rockette model.

Mean square	Expected mean square
MS(T)	$\frac{r}{t-1}{\sum }_{i=1}^t{({\tau }_i-{\tau }_{\cdot })}^2+{\sigma }_{\tau R}^2+{\sigma }_{∊}^2-{\mathrm{Cov}}_1+(r-1)({\mathrm{Cov}}_2-{\mathrm{Cov}}_3)$
MS(R)	$t{\sigma }_R^2+{\sigma }_{\tau R}^2+{\sigma }_{∊}^2-{\mathrm{Cov}}_2+(t-1)({\mathrm{Cov}}_1-{\mathrm{Cov}}_3)$
MS(T*R)	${\sigma }_{\tau R}^2+{\sigma }_{∊}^2-{\mathrm{Cov}}_1-{\mathrm{Cov}}_2+{\mathrm{Cov}}_3$

To show that F ^* has an approximate F _{( t -1),df2} null distribution with df₂ defined by equation (6), I first consider an approximation for the distribution of its denominator, MS(T^*R)+r (Cov₂-Cov₃). Let X_m denote a random variable with a ${\chi }_m^2∕m$ distribution that is independent of MS(T^*R). Then E (X_m) = 1 and X_m converges to 1 in probability as m → ∞. By Slutsky’s theorem the limiting distribution of MS(T^*R)+r (Cov₂-Cov₃) X_m as m → ∞ is equal to the distribution of MS(T^*R)+r (Cov₂-Cov₃). My approach is to approximate the distribution of MS(T^*R)+r (Cov₂-Cov₃) by the limiting distribution of an approximate distribution for MS(T^*R)+r (Cov₂-Cov₃) X_m as m → ∞.

I show in the Appendix that MS(T) and MS(T^*R) are independently distributed, $\mathrm{MS}\left({\mathrm{T}}^{\ast }\mathrm{R}\right)\sim {\chi }_{(t-1)(r-1)}^2\frac{E\left[\mathrm{MS}\left({\mathrm{T}}^{\ast }\mathrm{R}\right)\right]}{(t-1)(r-1)}$, and under ${\mathrm{H}}_0\mathrm{MS}\left(\mathrm{T}\right)\sim {\chi }_{t-1}^2\frac{E\left[\mathrm{MS}\left(\mathrm{T}\right)\right]}{t-1}$. Satterthwaite [13, 14] has shown that a linear function of independent random variables $\mathrm{MS}={\sum }_i^{}{a}_i{\mathrm{MS}}_i$, where ${\mathrm{MS}}_i\sim {\chi }_{{\mathrm{df}}_i}^{2}E\left({\mathrm{MS}}_i\right)∕{\mathrm{df}}_i$, is approximately distributed as ${\chi }_{\mathrm{df}}^{2}E\left(\mathrm{MS}\right)∕\mathrm{df}$, where

$$\mathrm{df}=\frac{{\left[\sum _i{a}_{i}E\left({\mathrm{MS}}_i\right)\right]}^2}{\sum _i{\left[a_{i}E\left({\mathrm{MS}}_i\right)\right]}^2∕{\mathrm{df}}_i}$$ (9)

Using Satterthwaite’s procedure with a ₁ = 1; MS₁ =MS(T^*R), a ₂ = r (Cov₂-Cov₃), and MS₂ = X_m, it follows that MS(T^*R)+r (Cov₂-Cov₃) X_m is approximately distributed as

$${\chi }_{\mathrm{df}\_\mathrm{S}\left(m\right)}^2\frac{E\left[\mathrm{MS}\left({\mathrm{T}}^{\ast }\mathrm{R}\right)\right]+r({\mathrm{Cov}}_2-{\mathrm{Cov}}_3)}{\mathrm{df}\_\mathrm{S}\left(m\right)}$$ (10)

where df_S(m) is given by

$$\mathrm{df}\_\mathrm{S}\left(m\right)=\frac{{[E\left(\mathrm{MS}\left({\mathrm{T}}^{\ast }\mathrm{R}\right)\right)+r({\mathrm{Cov}}_2-{\mathrm{Cov}}_3)]}^2}{\frac{{\left[E\left(\mathrm{MS}\left({\mathrm{T}}^{\ast }\mathrm{R}\right)\right)\right]}^2}{(t-1)(r-1)}+\frac{{\left[r({\mathrm{Cov}}_2-{\mathrm{Cov}}_3)\right]}^2}{m}}$$

Since lim_{m →∞} df_S(m) = df₂, where df₂ is defined by equation (6), it follows that the limiting distribution of distribution (10) as m → ∞ is

$${\chi }_{{\mathrm{df}}_2}^2\frac{E\left[\mathrm{MS}\left({\mathrm{T}}^{\ast }\mathrm{R}\right)\right]+r({\mathrm{Cov}}_2-{\mathrm{Cov}}_3)}{{\mathrm{df}}_2}$$ (11)

Thus the distribution of the denominator of F ^*, MS(T^*R)+r (Cov₂-Cov₃), is approximated by distribution (11).

Since MS(T) and MS(T^*R)+r (Cov₂-Cov₃) are independent, $\mathrm{MS}\left(\mathrm{T}\right)\sim {\chi }_{t-1}^{2}E\left[\mathrm{MS}\left(\mathrm{T}\right)\right]∕(t-1)$ under H₀, MS(T^*R)+r (Cov₂-Cov₃) is approximately ${\chi }_{{\mathrm{df}}_2}^2\{E\left[\mathrm{MS}\left({\mathrm{T}}^{\ast }\mathrm{R}\right)\right]+r({\mathrm{Cov}}_2-{\mathrm{Cov}}_3)\}∕{\mathrm{df}}_2$, and E[MS(T)] = E[MS(T^*R)]+r (Cov₂-Cov₃) under H₀, it follows that the null distribution of F ^* = MS(T)/[MS(T^*R)+r (Cov₂-Cov₃)] is approximated by

$$\frac{U∕(t-1)}{V∕{\mathrm{df}}_2}$$

where $U\sim {\chi }_{t-1}^2$, $V\sim {\chi }_{{\mathrm{df}}_2}^2$, and U and V are independent; that is, F ^* has an approximate F _{( t -1),df2} null distribution. Since F _OR approximates F ^* and ddf_H (7) estimates df₂, I propose using ddf_H as the ddf for F _OR.

In summary, approximating the distribution of F _OR involves three distinct approximations. First, F _OR is approximated by F ^*. The adequacy of this approximation depends on how well ${\widehat{\mathrm{Cov}}}_2-{\widehat{\mathrm{Cov}}}_3$ approximates Cov₂-Cov₃. Throughout, I assume that the number of cases is moderate or large, as is typical for ROC studies; thus it is reasonable to expect that this approximation will be adequate since the precision of the covariance estimates increases with the number of cases. Second, the distribution of MS(T^*R)+r (Cov₂-Cov₃) is approximated by the distribution of MS(T^*R)+r (Cov₂-Cov₃) X_m for large m, where $X_m{}^{\sim }\chi _m^2∕m$, so that the Satterthwaite approximation can be applied to the denominator of F _OR; this approximation is justified by Slutsky’s theorem and the law of large numbers. Finally, the distribution of MS(T^*R)+r (Cov₂-Cov₃) X_m for large m is approximated using the Satterthwaite procedure to give distribution (11), with df₂ then estimated by ddf_H. This approximation depends on the adequacy of the Satterthwaite approximation, with Cov₂-Cov₃ approximated by ${\widehat{\mathrm{Cov}}}_2-{\widehat{\mathrm{Cov}}}_3$ and E[MS(T^*R)] approximated by MS(T^*R). I expect this approximation to be adequate since the Satterthwaite approximation has been shown to perform acceptably for sums of mean squares, with expected mean squares estimated by observed means squares when estimating the degrees of freedom, even when the mean square degrees of freedom are low [13, 14]. In the simulation study I empirically investigate the performance of ddf_H.

3.2 Rationale for ddf_O

To understand why Obuchowski and Rockette [9] suggested using (t - 1) (r - 1) as the ddf, I now reexamine their argument. Based upon the work of Pavur and Nath [15], they note that under H₀

$$\frac{1-\rho }{1+(r-1)\rho }F$$ (12)

is distributed as F _{( t -1),( t -1)( r -1)}, where F =MS(T)/MS(T^*R) and

$$\rho =\frac{2({\mathrm{Cov}}_2-{\mathrm{Cov}}_3)}{\mathrm{Var}({\widehat{\theta }}_{ij}-{\widehat{\theta }}_{i^{\prime }j})},i\ne i^{\prime }$$

They then replace ρ in equation (12) by the estimate

$$\widehat{\rho }=\frac{2({\widehat{\mathrm{Cov}}}_2-{\widehat{\mathrm{Cov}}}_3)}{\widehat{\mathrm{Var}}({\widehat{\theta }}_{ij}-{\widehat{\theta }}_{i^{\prime }j})}$$

where

$$\widehat{\mathrm{Var}}({\widehat{\theta }}_{ij}-{\widehat{\theta }}_{i^{\prime }j})=2(\mathrm{MS}\left({\mathrm{T}}^{\ast }\mathrm{R}\right)+{\widehat{\mathrm{Cov}}}_2-{\widehat{\mathrm{Cov}}}_3)$$ (13)

to derive the OR F statistic (5).

Although replacing ρ by a high precision estimate should not appreciably alter the distribution of (12), the variance estimate (13) lacks precision for the typical study with only a few tests and readers, since MS(T^*R) has degrees of freedom (t-1) (r-1). Furthermore, the estimate of ρ is correlated with F, and this correlation needs to be taken into account because of the lack of precision in the estimate. Thus ddf_O does not adequately describe the distribution of the F _OR statistic (5) for typical studies because it does not account for either the lack of precision in estimating ρ or the correlation between the estimate of ρ and F.

4 DDF_H AND THE DBM PROCEDURE

4.1 The DBM procedure

The DBM method proposed by Dorfman et al [7] for analyzing multireader ROC studies also generalizes results to both the reader and case populations. For this method AUC pseudovalues are computed using the Quenouille-Tukey jackknife [16-18] separately for each reader-test combination. Let Y_ijk denote the AUC pseudovalue for test i, reader j, and case k; by definition $Y_{ijk}=c{\widehat{\theta }}_{ij}-(c-1){\widehat{\theta }}_{ij\left(k\right)}$, where ${\widehat{\theta }}_{ij}$ denotes the AUC estimate based on all of the data for the ith test and jth reader band _{ij ( k )} denotes the AUC estimate when data for the kth case are omitted. Using the Y_ijk as the responses, the DBM procedure specifies testing for a test effect using a fully crossed three-factor ANOVA, with test treated as a fixed factor and reader and case as random factors. The DBM estimate of ${\widehat{\theta }}_{ij}$ is Y_ij., which is the jackknife accuracy estimate corresponding to ${\widehat{\theta }}_{ij}$.

Recently Hillis et al [11] generalize the DBM method by showing how it can be used with normalized pseudovalues and quasi pseudovalues: normalized pseudovalues $Y_{ijk}^{\ast }$ are defined by $Y_{ijk}^{\ast }=Y_{ijk}+({\widehat{\theta }}_{ij}-Y_{ij\cdot })$, and quasi pseudovalues are defined as any values such that the resulting test-reader sample means, variances, and covariances are identical to ${\widehat{\theta }}_{ij}$ and $c\widehat{\Sigma }$, where $\widehat{\Sigma }$ is the estimated fixed-reader covariance matrix used to compute the OR procedure quantities ${\widehat{\sigma }}^2,{\widehat{\mathrm{Cov}}}_1,{\widehat{\mathrm{Cov}}}_2$, and ${\widehat{\mathrm{Cov}}}_3$. For normalized and quasi pseudovalues the DBM accuracy estimate, given by Y_dij, is equal to ${\widehat{\theta }}_{ij}$. They show that using DBM with normalized pseudovalues yields the same test statistic b for testing for a modality effect as using OR with jackknife covariance estimates, and more generally, using DBM with quasi pseudovalues yields the same test statistic as OR for other covariance estimation methods. However, even when DBM and OR yield the same test statistic, inferences depend on which ddf method, DBM or OR, is used. From hereon I assume, when comparing the DBM and OR procedures, that the pseudovalues are the normalized pseudovalues if ${\widehat{\mathrm{Cov}}}_2$ and ${\widehat{\mathrm{Cov}}}_3$ are based on jackknife covariance estimates, and are quasi pseudovalues if ${\widehat{\mathrm{Cov}}}_2$ and ${\widehat{\mathrm{Cov}}}_3$ are based on another fixed-reader covariance estimation method such as DeLong et al [12].

Let MS(T)_pseudo, MS(T^*R)_pseudo, MS(T^*C)_pseudo, and MS(T^*R^*C)_pseudo denote the test, test×reader, test×case, and test×reader case mean squares for the DBM three-way ANOVA of the pseudovalues. The DBM F statistic for testing the null hypothesis of no test effect is

$$F_{\mathrm{DBM}}=\frac{\mathrm{MS}{\left(\mathrm{T}\right)}_{\text{pseudo}}}{\mathrm{MS}{\left({\mathrm{T}}^{\ast }\mathrm{R}\right)}_{\text{pseudo}}+\max [\mathrm{MS}{\left({\mathrm{T}}^{\ast }\mathrm{C}\right)}_{\text{pseudo}}-\mathrm{MS}{\left({\mathrm{T}}^{\ast }{\mathrm{R}}^{\ast }\mathrm{C}\right)}_{\text{pseudo}},0]}$$ (14)

Hillis et al [11] show that F _DBM = F _OR by showing that

$$\mathrm{MS}\left(\mathrm{R}\right)=\frac{1}{c}\mathrm{MS}{\left(\mathrm{R}\right)}_{\text{pseudo}}$$$$\mathrm{MS}\left({\mathrm{T}}^{\ast }\mathrm{R}\right)=\frac{1}{c}\mathrm{MS}{\left({\mathrm{T}}^{\ast }\mathrm{R}\right)}_{\text{pseudo}}$$ (15)

and

$${\widehat{\mathrm{Cov}}}_2-{\widehat{\mathrm{Cov}}}_3=\frac{1}{rc}[\mathrm{MS}{\left({\mathrm{T}}^{\ast }\mathrm{C}\right)}_{\text{pseudo}}-\mathrm{MS}{\left({\mathrm{T}}^{\ast }{\mathrm{R}}^{\ast }\mathrm{C}\right)}_{\text{pseudo}}]$$ (16)

The result follows by substitution in (14). They also show that the DBM model implies the OR model, and they give the one-to-one mapping between the OR and DBM model parameters.

This form (14) of the DBM F statistic utilizes the same constraints as F _OR (5), since equation (16) implies that the constraint MS(T^*C)_pseudo-MS(T^*R^*C)_pseudo ≥ 0, utilized in the denominator of F _DBM (14), is equivalent to the constraint ${\widehat{\mathrm{Cov}}}_2-{\widehat{\mathrm{Cov}}}_3\ge 0$ employed in F _OR. Although Dorfman et al [7, 8] suggest also constraining d MS(T^*R)_pseudo-MS(T^*R^*C)_pseudo to be nonnegative, References [11, 19] discuss conceptual reasons for not using this constraint; furthermore, simulations [20] show that performance is better when only MS(T^*C)_pseudo-MS(T^*R^*C)_pseudo is constrained to be nonnegative. Throughout this paper I use F _DBM as specified by equation (14).

4.2 Comparison of ddf_H, ddf_O, and ddf_D

The DBM numerator and denominator degrees of freedom for the null distribution of F _DBM (14) are t-1 and ddf_D, respectively, where

$${\mathrm{ddf}}_{\mathrm{D}}=\{\begin{array}{cc}\frac{{[\mathrm{MS}{\left({\mathrm{T}}^{\ast }\mathrm{R}\right)}_{\text{pseudo}}+\mathrm{MS}{\left({\mathrm{T}}^{\ast }\mathrm{C}\right)}_{\text{pseudo}}-\mathrm{MS}{\left({\mathrm{T}}^{\ast }{\mathrm{R}}^{\ast }\mathrm{C}\right)}_{\text{pseudo}}]}^2}{\frac{\mathrm{MS}{\left({\mathrm{T}}^{\ast }\mathrm{R}\right)}_{\text{pseudo}}^2}{(t-1)(r-1)}+\frac{\mathrm{MS}{\left({\mathrm{T}}^{\ast }\mathrm{C}\right)}_{\text{pseudo}}^2}{(t-1)(c-1)}+\frac{\mathrm{MS}{\left({\mathrm{T}}^{\ast }{\mathrm{R}}^{\ast }\mathrm{C}\right)}_{\text{pseudo}}^2}{(t-1)(r-1)(c-1)}},\hfill & \mathrm{MS}{\left({\mathrm{T}}^{\ast }\mathrm{C}\right)}_{\text{pseudo}}-\mathrm{MS}{\left({\mathrm{T}}^{\ast }{\mathrm{R}}^{\ast }\mathrm{C}\right)}_{\text{pseudo}}>0\hfill \\ (t-1)(r-1),\hfill & \mathrm{MS}{\left({\mathrm{T}}^{\ast }\right)\mathrm{C}}_{\text{pseudo}}-\mathrm{MS}{\left({\mathrm{T}}^{\ast }{\mathrm{R}}^{\ast }\mathrm{C}\right)}_{\mathrm{pseudo}}\le 0\hfill \end{array}\phantom{\}}$$ (17)

Hillis et al [11] show that ddf_D has the following form when expressed in terms of the OR mean squares and covariance estimates:

$${\mathrm{ddf}}_{\mathrm{D}}=\{\begin{array}{cc}\frac{{[\mathrm{MS}\left({\mathrm{T}}^{\ast }\mathrm{R}\right)+r({\widehat{\mathrm{Cov}}}_2-{\widehat{\mathrm{Cov}}}_3)]}^2}{\frac{\mathrm{MS}{\left({\mathrm{T}}^{\ast }\mathrm{R}\right)}^2}{(t-1)(r-1)}+\frac{{[{\widehat{\sigma }}_{∊}^2-{\widehat{\mathrm{Cov}}}_1+(r-1)({\widehat{\mathrm{Cov}}}_2-{\widehat{\mathrm{Cov}}}_3)]}^2}{(t-1)(c-1)}+\frac{{[{\widehat{\sigma }}_{∊}^2-{\widehat{\mathrm{Cov}}}_1-{\widehat{\mathrm{Cov}}}_2+{\widehat{\mathrm{Cov}}}_3]}^2}{(t-1)(r-1)(c-1)}},\hfill & {\widehat{\mathrm{Cov}}}_2-{\widehat{\mathrm{Cov}}}_3>0\hfill \\ (t-1)(r-1),\hfill & {\widehat{\mathrm{Cov}}}_2-{\widehat{\mathrm{Cov}}}_3\le 0\hfill \end{array}\phantom{\}}$$ (18)

Equations (17) and (18) yield the same value for ddf_D.

Comparing equations (7) and (18) we see that

$${\mathrm{ddf}}_{\mathrm{D}}\le {\mathrm{ddf}}_{\mathrm{H}}$$ (19)

This relationship is intuitive, since in deriving ddf_H I treat σ², Cov₁, Cov₂, and Cov₃ as known while DBM treats them (or equivalently, the corresponding DBM model variance components) as unknown; thus the uncertainty from estimating them is manifested in the lower DBM degrees of freedom.

It follows from equation (7) that

$${\mathrm{ddf}}_{\mathrm{o}}=(t-1)(r-1)\le {\mathrm{ddf}}_{\mathrm{H}}$$

with equality attained if and only if ${\widehat{\mathrm{Cov}}}_2-{\widehat{\mathrm{Cov}}}_3$ < 0 , showing that the ddf_O method is more conservative than the ddf_H method. There is not a similar relationship between ddf_O and ddf_D: ddf_D can be larger or smaller that ddf_O. Table 2 presents the ddf formulas in terms of the OR model in part (a) and the ddf relationships in part (b). [ Table 2 ]

Table 2.

Denominator degrees of freedom summary.

a) In terms of the OR mean squares and covariances estimates:

ddfO = (t-1)(r-1)

$${\mathrm{ddf}}_{\mathrm{D}}=\{\begin{array}{cc}\frac{{[\mathrm{MS}\left({\mathrm{T}}^{\ast }\mathrm{R}\right)+r({\widehat{\mathrm{Cov}}}_2-{\widehat{\mathrm{Cov}}}_3)]}^2}{\frac{\mathrm{MS}{\left({\mathrm{T}}^{\ast }\mathrm{R}\right)}^2}{(t-1)(r-1)}+\frac{{[{\widehat{\sigma }}_{∊}^2-{\widehat{\mathrm{Cov}}}_1+(r-1)({\widehat{\mathrm{Cov}}}_2-{\widehat{\mathrm{Cov}}}_3)]}^2}{(t-1)(c-1)}+\frac{{[{\widehat{\sigma }}_{∊}^2-{\widehat{\mathrm{Cov}}}_1-{\widehat{\mathrm{Cov}}}_2+{\widehat{\mathrm{Cov}}}_3]}^2}{(t-1)(r-1)(c-1)}},\hfill & {\widehat{\mathrm{Cov}}}_2-{\widehat{\mathrm{Cov}}}_3>0\hfill \\ (t-1)(r-1),\hfill & {\widehat{\mathrm{Cov}}}_2-{\widehat{\mathrm{Cov}}}_3\le 0\hfill \end{array}\phantom{\}}$$

$${\mathrm{ddf}}_{\mathrm{H}}=\frac{{[\mathrm{MS}\left({\mathrm{T}}^{\ast }\mathrm{R}\right)+\max [r({\widehat{\mathrm{Cov}}}_2-{\widehat{\mathrm{Cov}}}_3),0]]}^2}{\frac{{\left[\mathrm{MS}\left({\mathrm{T}}^{\ast }\mathrm{R}\right)\right]}^2}{(t-1)(r-1)}}$$

b) Relationships:

ddfD ≤ddfH

$${\mathrm{ddf}}_{\mathrm{o}}=(t-1)(r-1)\{\begin{array}{cc}\le {\mathrm{ddf}}_{\mathrm{H}},\hfill & {\widehat{\mathrm{Cov}}}_2-{\widehat{\mathrm{Cov}}}_3>0\hfill \\ ={\mathrm{ddf}}_{\mathrm{H}},\hfill & {\widehat{\mathrm{Cov}}}_2-{\widehat{\mathrm{Cov}}}_3=0\hfill \end{array}\phantom{\}}$$

ddfD can be larger or smaller than ddfO

c) In terms of the DBM mean squares:

ddfO = (t-1)(r-1)

$${\mathrm{ddf}}_{\mathrm{D}}=\{\begin{array}{cc}\frac{{[\mathrm{MS}{\left({\mathrm{T}}^{\ast }\mathrm{R}\right)}_{\text{pseudo}}+\mathrm{MS}{\left({\mathrm{T}}^{\ast }\mathrm{C}\right)}_{\text{pseudo}}-\mathrm{MS}{\left({\mathrm{T}}^{\ast }{\mathrm{R}}^{\ast }\mathrm{C}\right)}_{\text{pseudo}}]}^2}{\frac{\mathrm{MS}{\left({\mathrm{T}}^{\ast }\mathrm{R}\right)}_{\text{pseudo}}^2}{(t-1)(r-1)}+\frac{\mathrm{MS}{\left({\mathrm{T}}^{\ast }\mathrm{C}\right)}_{\text{pseudo}}^2}{(t-1)(c-1)}+\frac{\mathrm{MS}{\left({\mathrm{T}}^{\ast }{\mathrm{R}}^{\ast }\mathrm{C}\right)}_{\text{pseudo}}^2}{(t-1)(r-1)(c-1)}},\hfill & \mathrm{MS}{\left({\mathrm{T}}^{\ast }\mathrm{C}\right)}_{\text{pseudo}}-\mathrm{MS}{\left({\mathrm{T}}^{\ast }{\mathrm{R}}^{\ast }\mathrm{C}\right)}_{\text{pseudo}}>0\hfill \\ (t-1)(r-1),\hfill & \mathrm{MS}{\left({\mathrm{T}}^{\ast }\mathrm{C}\right)}_{\text{pseudo}}-\mathrm{MS}{\left({\mathrm{T}}^{\ast }{\mathrm{R}}^{\ast }\mathrm{C}\right)}_{\text{pseudo}}\le 0\hfill \end{array}\phantom{\}}$$

$${\mathrm{ddf}}_{\mathrm{H}}=\frac{{\{\mathrm{MS}{\left({\mathrm{T}}^{\ast }\mathrm{R}\right)}_{\text{pseudo}}+\max [\mathrm{MS}{\left({\mathrm{T}}^{\ast }\mathrm{C}\right)}_{\text{pseudo}}-\mathrm{MS}{\left({\mathrm{T}}^{\ast }{\mathrm{R}}^{\ast }\mathrm{C}\right)}_{\text{pseudo}},0]\}}^2}{\mathrm{MS}{\left({\mathrm{T}}^{\ast }\mathrm{R}\right)}_{\text{pseudo}}^2∕\left[(t-1)(r-1)\right]}$$

4.3 Ddf_H in terms of the DBM mean squares

I can also express ddf_H in terms of the DBM analysis mean squares. It follows from equations (7,15-16) that

$${\mathrm{ddf}}_{\mathrm{H}}=\frac{{\{\mathrm{MS}{\left({\mathrm{T}}^{\ast }\mathrm{R}\right)}_{\text{pseudo}}+\max [\mathrm{MS}{\left({\mathrm{T}}^{\ast }\mathrm{C}\right)}_{\text{pseudo}}-\mathrm{MS}{\left({\mathrm{T}}^{\ast }{\mathrm{R}}^{\ast }\mathrm{C}\right)}_{\text{pseudo}},0]\}}^2}{\mathrm{MS}{\left({\mathrm{T}}^{\ast }\mathrm{R}\right)}_{\text{pseudo}}^2∕\left[(t-1)(r-1)\right]}$$ (20)

Part (c) of Table 2 presents the ddf formulas expressed in terms of the DBM model. Note that the relationships in part (b), which we previously derived from the part (a) formulas, can also be derived from the equivalent part (c) formulas.

Alternatively, I can derive ddf_H in form (20) directly from the DBM model in the following way. It is shown by Hillis et al [11] that, under the assumptions of the DBM model,

$$E[\mathrm{MS}{\left({\mathrm{T}}^{\ast }\mathrm{C}\right)}_{\text{pseudo}}-\mathrm{MS}{\left({\mathrm{T}}^{\ast }{\mathrm{R}}^{\ast }\mathrm{C}\right)}_{\text{pseudo}}]=r{\sigma }_{\tau C}^2$$ (21)

where ${\sigma }_{\tau C}^2$ is the test case variance component for the DBM model. Define

$$F_{\mathrm{DBM}}^{\ast }=\frac{\mathrm{MS}{\left(\mathrm{T}\right)}_{\text{pseudo}}}{\mathrm{MS}{\left({\mathrm{T}}^{\ast }\mathrm{R}\right)}_{\text{pseudo}}+r{\sigma }_{\tau C}^2}$$

For typical ROC studies (c > 25) the degrees of freedom will be at least 25 for each mean square in equation (21); thus MS(T^*C)_pseudo-MS(T^*R^*C)_pseudo should approximate $r{\sigma }_{\tau C}^2$ reasonably well, implying

$$F_{\mathrm{DBM}}\approx F_{\mathrm{DBM}}^{\ast }$$

My approach is to show that an approximate ddf for $F_{\mathrm{DBM}}^{\ast }$, and hence for F _DBM, is given by equation (20).

Under the DBM model assumptions (conventional three-way ANOVA model assumptions for the pseudovalues) the numerator and denominator of $F_{\mathrm{DBM}}^{\ast }$ have the same null expected values, MS(T)_pseudo and MS(T^*R)_pseudo are independently distributed,

$$\mathrm{MS}{\left({\mathrm{T}}^{\ast }\mathrm{R}\right)}_{\text{pseudo}}\sim {\chi }_{(t-1)(r-1)}^{2}E\left[\mathrm{MS}{\left({\mathrm{T}}^{\ast }\mathrm{R}\right)}_{\text{pseudo}}\right]∕\left[(t-1)(r-1)\right]$$

and

$$\mathrm{MS}{\left(\mathrm{T}\right)}_{\text{pseudo}}\sim {\chi }_{t-1}^{2}E\left[\mathrm{MS}{\left(\mathrm{T}\right)}_{\text{pseudo}}\right]∕(t-1)\text{if}{\mathrm{H}}_0\text{is true}.$$

Using the same argument given in Section 3.1 but with MS(T) replaced by MS(T)_pseudo, MS(T^*R) replaced by MS(T^*R)_pseudo, and r (Cov₂-Cov₃) replaced by $r{\sigma }_{\tau C}^2$, it follows that $F_{\mathrm{DBM}}^{\ast }$ has an approximate F_{t -1,df2} null distribution, with

$${\mathrm{df}}_2=\frac{{[E\left(\mathrm{MS}{\left({\mathrm{T}}^{\ast }\mathrm{R}\right)}_{\text{pseudo}}\right)+r{\sigma }_{\tau C}^2]}^2}{\frac{{\left[E\left(\mathrm{MS}{\left({\mathrm{T}}^{\ast }\mathrm{R}\right)}_{\text{pseudo}}\right)\right]}^2}{(t-1)(r-1)}}$$

Replacing $E\left[\mathrm{MS}{\left({\mathrm{T}}^{\ast }\mathrm{R}\right)}_{\text{pseudo}}\right]$ and $r{\sigma }_{\tau C}^2$ by their estimates, MS(T^*R)_pseudo and $\max [\mathrm{MS}{\left({\mathrm{T}}^{\ast }\mathrm{C}\right)}_{\text{pseudo}}-\mathrm{MS}{\left({\mathrm{T}}^{\ast }{\mathrm{R}}^{\ast }\mathrm{C}\right)}_{\text{pseudo}},0]$, respectively, yields equation (20).

4.4 Inadequate DBM Satterthwaite approximation

When MS(T^*C)_pseudo-MS(T^*R^*C)_pseudo > 0, ddf_D in equation (17) is the Satterthwaite approximation (9) for MS(T^*R) + MS(T^*C) MS(T^*R^*C). Satterthwaite [13] remarks that caution should be used when applying formula (9) to a linear function of mean squares when some of the coefficients are negative, and Gaylor and Hopper [21] provide guidelines for determining if the Satterthwaite approximate is valid for this situation. We will see in the simulation study that an inadequate Satterthwaite approximation can cause ddf_D to approach zero, resulting in an extremely wide confidence interval.

5 SPECIAL CASES

5.1 Confidence interval for the difference of two tests

Define θ_i to be the expected accuracy estimate for test i; that is, ${\theta }_i=E\left({\widehat{\theta }}_{ij}\right)$. I have shown that F _OR, and hence also F _DBM, has an approximate F _{( t -1);ddfH} distribution; it follows that for t = 2 an approximate (1-α)100% confidence interval for θ_i-θ_j is given by

$$({\widehat{\theta }}_{i\cdot }-{\widehat{\theta }}_{j\cdot })\pm t_{\frac{\alpha }{2};{\mathrm{ddf}}_{\mathrm{H}}}\sqrt{\frac{2}{r}{\mathrm{MSden}}_{\mathrm{OR}}}$$ (22)

where MSden_OR is the denominator of F _OR (5), or equivalently by

$$({\widehat{\theta }}_{i\cdot }-{\widehat{\theta }}_{j\cdot })\pm t_{\frac{\alpha }{2};{\mathrm{ddf}}_{\mathrm{H}}}\sqrt{\frac{2}{rc}{\mathrm{MSden}}_{\mathrm{DBM}}}$$ (23)

where MSden_DBM is the denominator of F _DBM (14). It can be shown, using that approach of Section 3.1, that equations (22-23) are also valid when t > 2.

5.2 Single test inference using all of the data

The results in this and the following section are presented without proof, but they can be derived using an approach similar to that used in Section 3.1. Define

$${\mathrm{MSden}}_{\mathrm{OR}\_\text{single}}=\frac{1}{t}\{\mathrm{MS}\left(\mathrm{R}\right)+(t-1)\mathrm{MS}\left({\mathrm{T}}^{\ast }\mathrm{R}\right)+tr\max ({\widehat{\mathrm{Cov}}}_2,0)\}$$$${\mathrm{ddf}}_{\mathrm{H}\_\text{single}}=\frac{{\{\mathrm{MS}\left(\mathrm{R}\right)+(t-1)\mathrm{MS}\left({\mathrm{T}}^{\ast }\mathrm{R}\right)+tr\max ({\widehat{\mathrm{Cov}}}_2,0)\}}^2}{\frac{{\left\{\mathrm{MS}\left(\mathrm{R}\right)\right\}}^2}{r-1}+\frac{{\left\{(t-1)\mathrm{MS}\left({\mathrm{T}}^{\ast }\mathrm{R}\right)\right\}}^2}{(t-1)(r-1)}}$$

An approximate (1-α)100% confidence interval for θ_i is given by

$${\widehat{\theta }}_{i\cdot }\pm t_{\frac{\alpha }{2};{\mathrm{ddf}}_{\mathrm{H}\_\text{single}}}\sqrt{\frac{1}{r}{\mathrm{MSden}}_{\mathrm{OR}\_\text{single}}}$$ (24)

and a test for H₀>:θ_i = θ₀ can be made by comparing $t=({\widehat{\theta }}_i-{\theta }_0)∕\sqrt{\frac{1}{r}{\mathrm{MSden}}_{\mathrm{OR}\_\text{single}}}$ to a t distribution with degrees of freedom ddf_{H_single}. Equivalent expressions can be derived for the DBM procedure, but I omit those since I recommend instead the more robust approach described in the next section.

5.3 Single test inference using only corresponding data

A confidence interval and hypotheses test for θ_i can alternatively be based only on the data for the ith test. When we consider the data for test i, the OR model (1) reduces to a one-factor random effects ANOVA with the covariance for each pair of errors equal to Cov₂. Since this single-test model makes no assumptions about the variances or covariances of the error terms corresponding to other tests, the resulting confidence interval should be more robust than confidence interval (24). Let ${\widehat{\mathrm{Cov}}}_2^{\left(i\right)}$ denote the average of the fixed reader covariance estimates for the ith test and define

$${\mathrm{MSR}}^{\left(i\right)}=\frac{\sum _{j=1}^r{({\widehat{\theta }}_{ij}-{\widehat{\theta }}_{i\cdot })}^2}{r-1}$$

$${\mathrm{MSden}}_{\mathrm{OR}\_\text{single}}^{\left(i\right)}={\mathrm{MSR}}^{\left(i\right)}+\max [r{\widehat{\mathrm{Cov}}}_2^{\left(i\right)},0]$$

and

$${\mathrm{ddf}}_{\mathrm{H}\_\text{single}}^{\left(i\right)}=\frac{{\{{\mathrm{MSR}}^{\left(i\right)}+\max (r{\widehat{\mathrm{Cov}}}_2^{\left(i\right)},0)\}}^2}{\frac{{\left({\mathrm{MSR}}^{\left(i\right)}\right)}^2}{r-1}}$$

An approximate (1-α)100% confidence interval for θ_i is given by

$${\widehat{\theta }}_{i\cdot }\pm t_{\frac{\alpha }{2};{\mathrm{ddf}}_{\mathrm{H}\_\text{single}}^{\left(i\right)}}\sqrt{\frac{1}{r}{\mathrm{MSden}}_{\mathrm{OR}\_\text{single}}^{\left(i\right)}}$$ (25)

A similar result is also given by Obuchowski and Rockette [9], but instead of ${\mathrm{ddf}}_{\mathrm{H}\_\text{single}}^{\left(i\right)}$ they use r-1; thus we see that ${\mathrm{ddf}}_{\mathrm{H}\_\text{single}}^{\left(i\right)}$ yields a less conservative result, since ${\mathrm{ddf}}_{\mathrm{H}\_\text{single}}^{\left(i\right)}\ge r-1$.

Similarly, the DBM procedure reduces to a conventional reader×case ANOVA of pseudovalues when we consider only data for a single test. Let $\mathrm{MS}{\left(\mathrm{R}\right)}_{\text{pseudo}}^{\left(i\right)},\mathrm{MS}{\left(\mathrm{C}\right)}_{\text{pseudo}}^{\left(i\right)}$, and $\mathrm{MS}{\left({\mathrm{R}}^{\ast }\mathrm{C}\right)}_{\text{pseudo}}^{\left(i\right)}$ denote mean squares using only pseudovalues corresponding to the ith test. An equivalent expression for confidence interval (25) is given by

$${\widehat{\theta }}_{i\cdot }\pm t_{\frac{\alpha }{2};{\mathrm{ddf}}_{\mathrm{H}\_\text{single}}^{\left(i\right)}}\sqrt{\frac{1}{rc}{\mathrm{MSden}}_{\mathrm{DBM}\_\text{single}}^{\left(i\right)}}$$

where

$${\mathrm{MSden}}_{\mathrm{DBM}\_\text{single}}^{\left(i\right)}=\mathrm{MS}{\left(\mathrm{R}\right)}_{\text{pseudo}}^{\left(i\right)}+\max [\mathrm{MS}{\left(\mathrm{C}\right)}_{\text{pseudo}}^{\left(i\right)}-\mathrm{MS}{\left({\mathrm{R}}^{\ast }\mathrm{C}\right)}_{\text{pseudo}}^{\left(i\right)},0]$$

and

$${\mathrm{ddf}}_{\mathrm{H}\_\text{single}}^{\left(i\right)}=\frac{{\{\mathrm{MS}{\left(\mathrm{R}\right)}_{\text{pseudo}}^{\left(i\right)}+\max [\mathrm{MS}{\left(\mathrm{C}\right)}_{\text{pseudo}}^{\left(i\right)}-\mathrm{MS}{\left({\mathrm{R}}^{\ast }\mathrm{C}\right)}_{\text{pseudo}}^{\left(i\right)},0]\}}^2}{\frac{{\left(\mathrm{MS}{\left(\mathrm{R}\right)}_{\text{pseudo}}^{\left(i\right)}\right)}^2}{r-1}}$$

A test for H₀: θ_i = θ₀ can be made by comparing either $t=({\widehat{\theta }}_i-{\theta }_0)∕\sqrt{\frac{1}{r}{\mathrm{MSden}}_{\mathrm{OR}\_\text{single}}^{\left(i\right)}}$ or $t=({\widehat{\theta }}_i-{\theta }_0)∕\sqrt{\frac{1}{rc}{\mathrm{MSden}}_{\mathrm{DBM}\_\text{single}}^{\left(i\right)}}$ to a t distribution with degrees of freedom ${\mathrm{ddf}}_{\mathrm{H}\_\text{single}}^{\left(i\right)}$.

5.4 Fixed readers

If readers are treated as fixed for the OR model (1) then the only random effects are the error terms. Treating the variance and covariances of the error terms as known, the method of generalized least squares can be used, as discussed by Obuchowski and Rockette [9]. The test statistic, given by

$$\frac{(t-1)\mathrm{MS}{\left(\mathrm{T}\right)}_{{\widehat{\theta }}_{ij}}}{[{\widehat{\sigma }}_{∊}^2-{\widehat{\mathrm{Cov}}}_1+(r-1)({\widehat{\mathrm{Cov}}}_2-{\widehat{\mathrm{Cov}}}_3)]}$$

has an approximate chi-squared distribution with t-1 degrees of freedom. Hillis et al [11] show that the DBM analysis, treating readers as fixed, will give the same test statistic when suitably normalized and approximately the same p-value for typical ROC studies. I see no problems with these methods in terms of the degrees of freedom.

6 SIMULATION STUDY

In a simulation study I compare ddf_H, ddf_O, and ddf_D with respect to the empirical significance level for testing the null hypothesis of no test effect; and with respect to the width of a 95% confidence interval for the difference of the test AUCs. The simulation model of Roe and Metz [22] provides continuous decision-variable outcomes generated from a binormal model that treats both cases and readers as random. I use this simulation model for simulating rating data, taking integer values from one to five, by transforming the continuous outcomes to discrete ratings using the same cutpoints as Dorfman et al [8]; the combinations of reader and case sample sizes, AUC values, and variance components are the same as used in Roe and Metz [22] and Dorfman et al [8]. Briefly, rating data are simulated for 144 combinations of three reader-sample sizes (readers = 3, 5, and 10), four case sample sizes (10+/90-, 25+/25-, 50+/50-, and 100+/100-, where “+” indicates a diseased case and “-” indicates a normal case), three AUC values (AUC = .702, .855, and .961), and four combinations of reader and case variance components. Two thousand samples are generated for each of the 144 combinations; within each simulation, all Monte Carlo readers read the same cases for each of two tests. Since these are null simulations the test effect in the model is set to zero. The simulation design is summarized in Table 3. [ Table 3 ]

Table 3.

Simulation study design.

Factor	Number of levels	Description of levels
reader-sample size	3	readers = 3, 5, 10
case-sample size	4	cases = 10+/90-, 25+/25-, 50+/50-, 100+/100-
AUC	3	AUC = .702, .855, .961
variance components	4	HH, HL, LH, LL

Notes: For each factor combination n = 2000 samples are simulated for t = 2 tests, with each reader reading the same cases using each test. “+”= diseased case, “-”= normal case. See Roe and Metz [22] for the definitions of HH, HL, LH, and LL.

In terms of the OR procedure, each sample is analyzed using two different methods for estimating the AUC and ${\sigma }_{∊}^2,{\mathrm{Cov}}_1,{\mathrm{Cov}}_2$ and ${\mathrm{Cov}}_3$. For one method I estimate the AUC using maximum likelihood estimation assuming a binormal model [23,24] and obtain ${\widehat{\sigma }}_{∊}^2,{\widehat{\mathrm{Cov}}}_1,{\widehat{\mathrm{Cov}}}_2$, and ${\widehat{\mathrm{Cov}}}_3$ using the jackknife method; for the other method I estimate the AUC using the trapezoidal-rule (trapezoid) method and obtain ${\widehat{\sigma }}_{∊}^2,{\widehat{\mathrm{Cov}}}_1,{\widehat{\mathrm{Cov}}}_2$, and ${\widehat{\mathrm{Cov}}}_3$ using the DeLong method. These estimation methods are referred to as the MLE-jack and trapezoid-DeLong methods, respectively. For each method I compute F _OR, ddf_H, ddf_O, and ddf_D using equations (5,7,4,18). The null hypothesis of no test effect is rejected if F _OR > F _.05;1,ddf. For each of the 144 combinations the empirical significance level is the proportion of samples for which the null hypothesis is rejected. For each simulation I also compute the width of the 95% confidence interval for the difference of the two test AUCs using equation (22) and report the mean width for the 144·2000 simulations.

Identical results can also be obtained using the DBM procedure, as discussed in Hillis et al [11]. MLE-jack results can be obtained by computing the normalized pseudovalues corresponding to the MLE AUC estimates, computing MS(T)_pseudo, MS(T^*R)_pseudo, MS(T^*C)_pseudo, and MS(T^*R^*C)_pseudo, computing F _DBM (14) which will be the same as F _OR, and then computing ddf_O, ddf_H, and ddf_D using equations (4,20,17). A 95% confidence for the difference of the two test AUCs is given by equation (23). Trapezoid-DeLong results can similarly be obtained using the DBM procedure with quasi pseudovalues as described in Hillis et al [11]. I exploit these OR-DBM equivalent relationships by using the DBM procedure for the MLE-jack simulations and the OR procedure for the Trapezoid-DeLong simulations. Simulations are performed using the IML procedure in SAS 9.1 [25] running under Windows XP. The MLE AUC pseudovalues are computed using a dynamic linked library (DLL), written in Fortran 90 by Don Dorfman and Kevin Schartz, that is accessed from within the IML procedure; this DLL is available on request.

The empirical significance levels from the simulation study are described in Table 4 and displayed in dot plots in Figure 1. The mean significance levels, with ranges indicated in parentheses, for ddf_O, ddf_H, and ddf_D, respectively, are 0.018 (0.041), 0.051 (0.052), and 0.043 (0.050) for MLE-jackknife estimation, and 0.019 (0.047), 0.054 (0.051), and 0.047 (0.058) for trapezoid-DeLong estimation. Thus we see that the average significance levels for ddf_H and ddf_O are much closer to the nominal .05 level than for ddf_O, with ddf_D slightly more conservative than ddf_H in accord with relationship (19). The dot plots show the ultraconservative performance of ddf_O: several significance levels are 0.00 and all of the significance levels are less than .05. For ddf_H and ddf_D the dot plots show bell-shaped distributions without outliers. For ddf_H, 93% (134/144) of the significance levels are within the interval [0.03, 0.07] for both estimation methods and the standard deviations of the significance levels are 0.011 (MLE-jackknife) and 0.010 (Trapezoid-DeLong) computed across the 144 combination of design factors; for ddf_D, 86% (124/144) and 92% (133/144) of the significance levels are within the interval [0.03, 0.07] for MLE-jackknife and trapezoid-DeLong estimation, respectively, and the standard deviations of the significance levels are 0.011 for both estimation methods. I conclude from the closeness of the mean significance level to the nominal level, the relatively small standard deviation, the high proportion of significance levels within .02 of the nominal level, and the absence of outliers that both ddf_H and ddf_DBM perform satisfactorily with respect to significance levels. [ Table 4 ] [ Figure 1 ]

Table 4.

Results of the simulation study for the 144 combinations of reader-sample size, case-sample size, AUC, and variance components.

		Significance levels
Estimation method	ddf method	N	Mean	Min	Max	Range	SD	CI width mean
MLE-jackknife	O	144	0.018	0.000	0.041	0.041	0.0144	0.231
	H	144	0.051	0.025	0.077	0.052	0.0105	0.173
	D	144	0.043	0.017	0.067	0.050	0.0109	2.36E+121
Trapezoid-Delong	O	144	0.019	0.000	0.047	0.047	0.0124	0.245
	H	144	0.054	0.032	0.082	0.051	0.0100	0.184
	D	144	0.047	0.023	0.080	0.058	0.0105	2.74E+121

Notes: Min: minimum; Max: maximum; SD: standard deviation; CI width: width of a 95% confidence interval for the difference of the AUC estimates; MLE-jackknife: binormal maximum likelihood AUC estimation and jackknife covariance estimation; Trapezoid-DeLong: trapezoid AUC estimation and DeLong covariance estimation.

Figure 1.

Dot plots of the 144 empirical significance levels using MLE-jackknife and trapezoid-DeLong estimation with each ddf method. The nominal significance level is .05.

The extremely large mean confidence interval widths corresponding to ddf_D in Table 4 (mean CI width: ddf_D = 2.36E+121 vs. ddf_H = 0.173 for MLE-jackknife estimation; ddf_D = 2.74E+121 vs. ddf_H = 0.184 for Trapezoid-DeLong estimation) can be attributed to a small proportion of samples for which ddf_D approaches zero. For example, ddf_D ≤ 1 in 0.002 of the samples; when I set ddf_D = 1 for these samples, then the resulting mean confidence interval widths are comparable (although somewhat larger) than those for ddf_H. Each ddf_D value less than one corresponds to an inadequate Satterthwaite ddf approximation, using the adequacy guidelines provided by Gaylor and Hopper [21]. I conclude that although the ddf_H and ddf_D methods are comparable with respect to significance levels, ddf_H performs better when confidence interval width is also considered.

Table 5 shows the mean empirical significance levels for ddf_H by factor level for each of the four simulation study factors. The results suggest that high AUC (.961) is associated with mild conservatism, while low AUC (.701) and low number of readers (3) are associated with mild liberalism. Accordingly, 17 of the 18 lowest empirical ddf_H significance levels in Figure 1(a) and 18 of the lowest 19 empirical ddf_H significance levels in Figure 1(b) correspond to AUC = .961; in contrast, 22 of the highest 23 values in Figure 1(a) and 21 of the 22 highest values in Figure 1(b) corresponding either to AUC = .701 or readers = 3. [ Table 5 ]

Table 5.

Simulation study mean significance levels for ddf_H by factor level.

			Estimation method
factor	factor level	N	MLE-jackknife (mean = 0.051)	Trapezoid-Delong (mean = 0.054)
case-sample size	10+/90-	36	0.047	0.053
	25+/25-	36	0.049	0.053
	50+/50-	36	0.052	0.055
	100+/100-	36	0.054	0.056
reader-sample size	3	48	0.056	0.060
	5	48	0.048	0.052
	10	48	0.048	0.051
AUC	0.702	48	0.057	0.060
	0.855	48	0.052	0.056
	0.961	48	0.043	0.047
variance components	HH	36	0.051	0.054
	HL	36	0.055	0.057
	LH	36	0.046	0.051
	LL	36	0.050	0.054

7 EXAMPLE

My example comes courtesy of Carolyn Van Dyke, MD. The study [26] compared the relative performance of single spin-echo magnetic resonance imaging (SE MRI) to cinematic presentation of MRI (CINE MRI) for the detection of thoracic aortic dissection. There were 45 patients with an aortic dissection and 69 patients without a dissection imaged with both SE MRI and CINE MRI. Five radiologists independently interpreted all of the images using a five-point ordinal scale: 1 = definitely no aortic dissection, 2 = probably no aortic dissection, 3 = unsure about aortic dissection, 4 = probably aortic dissection, and 5 = definitely aortic dissection.

The analysis of this study using trapezoid AUC estimates and DeLong covariance estimates is displayed in Table 6 and the empirical ROC curves are displayed in Figure 2. Although Table 6 only displays the analysis using the OR procedure, identical results can be obtained with the DBM procedure using quasi pseudovalues. For testing H₀: θ₁ = θ₂ we have F _OR = 4.485, ddf_O = 4 (p = .102), ddf_H = 15.07 (p = .051), and ddf_D = 13.81 (p = .053). Thus ddf_O and ddf_H give somewhat different results, while ddf_H and ddf_D give similar results, with ddf_H > ddf_D in accord with relationship (19). [ Table 6 ] [ Figure 2 ]

Table 6.

Analysis of Van Dyke et al [26] data using trapezoid AUC estimation and DeLong covariance estimation for t = 2 tests and r = 5 readers.

a) Trapezoid AUCs:
	test
	1 (CINE)	2 (Spin Echo)
reader (j)	${\widehat{\theta }}_{1j}$	${\widehat{\theta }}_{2jq}$
1	0.9196	0.9478
2	0.8588	0.9053
3	0.9039	0.9217
4	0.9731	0.9994
5	0.8298	0.9300
	${\widehat{\theta }}_{1\cdot }$ = .8970	${\widehat{\theta }}_{2\cdot }$ =.9408

b) ANOVA table:
Source	df	Sum of squares	Mean square
T	1	0.00479617	0.00479617
R	4	0.01534480	0.00383620
T*R	4	0.00220412	0.00055103

c) Covariance estimates computed from DeLong covariance matrix:

${\widehat{\sigma }}_{∊}^2$= .000792133, ${\widehat{\mathrm{Cov}}}_1$ = .000342009, ${\widehat{\mathrm{Cov}}}_2$ = .000339526, ${\widehat{\mathrm{Cov}}}_3$ = .000235850

d) FOR =MS(T)/[MS(T*R) + $\max [r({\widehat{\mathrm{Cov}}}_2-{\widehat{\mathrm{Cov}}}_3),0]$ = 4.4849

e) Denominator degrees of freedom:

ddfO = (t - 1)(r - 1) = 4

$${\mathrm{ddf}}_{\mathrm{H}}=\frac{{[\mathrm{MS}\left({\mathrm{T}}^{\ast }\mathrm{R}\right)+\max [r({\widehat{\mathrm{Cov}}}_2-{\widehat{\mathrm{Cov}}}_3),0]]}^2}{\frac{{\left[\mathrm{MS}\left({\mathrm{T}}^{\ast }\mathrm{R}\right)\right]}^2}{(t-1)(r-1)}}=15.0661$$

$${\mathrm{ddf}}_{\mathrm{DBM}}=\frac{{[\mathrm{MS}\left({\mathrm{T}}^{\ast }\mathrm{R}\right)+r({\widehat{\mathrm{Cov}}}_2-{\widehat{\mathrm{Cov}}}_3)]}^2}{\frac{\mathrm{MS}{\left({\mathrm{T}}^{\ast }\mathrm{R}\right)}^2}{(t-1)(r-1)}+\frac{{[{\widehat{\sigma }}_{∊}^2-{\widehat{\mathrm{Cov}}}_1+(r-1)({\widehat{\mathrm{Cov}}}_2-{\widehat{\mathrm{Cov}}}_3)]}^2}{(t-1)(c-1)}+\frac{{[{\widehat{\sigma }}_{∊}^2-{\widehat{\mathrm{Cov}}}_1-{\widehat{\mathrm{Cov}}}_2+{\widehat{\mathrm{Cov}}}_3]}^2}{(t-1)(r-1)(c-1)}}=13.8133$$

f) P-values for H0: θ1 = θ2:
ddf method	ddf	p = Pr = (F ( t -1), ddf ≥ F OR)
O	4	0.1016
H	15.07	0.0512
D	13.81	0.0528

g) Single test 95% confidence intervals using only corresponding data:
i	${\widehat{\theta }}_i$	${\widehat{\mathrm{Cov}}}_2^{\left(i\right)}$	MSR(i)	${\mathrm{MSDen}}_{\mathrm{OR}}^{\left(i\right)}$	${\mathrm{ddf}}_{\mathrm{H}}^{\left(i\right)}$	95% CI for θi
1 (Cine)	.8970	.0004785	.003083	.005470	12.60	.8970±.07169
2 (Spin Echo)	.9408	.0002015	.001305	.002312	12.57	.9408±.0466

Figure 2.

Trapezoid ROC curves of the five readers for SE MRI and CINE MRI in the detection of aortic dissection. The average areas under the empirical curves are .941 (SE) and .897 (CINE).

Single test 95% confidence intervals using only the corresponding data, as given by equation (25), are given in part (f) of Table 5. We see that the confidence interval for CINE (.897±0717) is roughly 50% wider that the interval for SE MRI (.941±0.0467). When based upon the combined data using equation (24), the single test confidence intervals are given by ${\widehat{\theta }}_i$±0.0591, i = 1, 2. I prefer the first approach since it does not assume that the AUCs have the same variances and covariances for each test, as does the combined data approach.

8 DISCUSSION

The motivation for this paper was the recent finding by Hillis et al [11] that the DBM and OR procedures yield identical test statistics when based on the same accuracy measure and covariance estimation methods, but their different ddf methods, ddf_D and ddf_O, can result in considerably different inferences. I proposed a new ddf estimator, ddf_H, that overcomes problems with the ddf_D and ddf_O methods. I derived ddf_H by showing that the null distribution of the OR test statistic can be approximated by an F_{t -1,ddfH} distribution. I showed how ddf_H can be used with both the OR and DBM procedures and how it can also be derived from the DBM model. The p-value corresponding to ddf_H will always be less than or equal to that corresponding to ddf_D or ddf_O.

Although ddf_H can be derived from the DBM model assumptions, the derivation in terms of the OR model is more important because the OR model provides an acceptable conceptual model. In contrast, the DBM model does not provide an acceptable conceptual model because pseudovalues have no intrinsic meaning; hence Hillis et al [11] characterize the DBM model as a “working model” that allows one to fit the OR model using conventional ANOVA software.

In simulations ddf_H performed better than ddf_O with respective to significance level, with the ddf_H mean significance level much closer to the nominal level. The shape and spread of the distributions of the significance levels were similar for the ddf_H and ddf_D methods, with ddf_H closer to the nominal level for binormal AUC estimation than ddf_D (mean significance levels: ddf_H = .051, ddf_D= .043) but deviating approximately the same amount from the nominal level for trapezoid AUC estimation (mean significance levels: ddf_H = .054, ddf_D = .047). However, a drawback of ddf_D was that confidence intervals were sometimes extremely wide due to ddf_D being close to zero. For these reasons I concluded that ddf_H performs better than either ddf_D or ddf_O. A SAS macro for implementing the OR-DBM procedure using ddf_H is available on request.

A limitation of this study is that I used only typical ROC sample sizes where the number of readers is low (≤ 10) and the number of cases is moderate (≥ 50) in the simulation study. Although the derivation of ddf_H involved several approximations, I discussed in Section 3.1 why ddf_H should perform adequately for the typical ROC diagnostic study where the number of cases is moderate to large, regardless of the number of readers. However, the performance of the new method may not be acceptable when the number of cases is less; certainly, further research is required before using it in this situation. Although the simulations only looked at small reader sample sizes, I see no reason why the new method should not work satisfactorily when the number of readers is moderate or large, but again I caution that our simulations have only been for typical ROC study sample sizes.

A topic to consider for future research is the robustness of the OR and DBM procedures to violations of the model assumptions. Since the procedures can be viewed as equivalent, only the less restrictive OR model assumptions need to be considered. The OR model assumes that the variances and covariances for the accuracy estimates do not vary by test. I conjecture that, similar to conventional ANOVA models, the test of the null hypothesis of no test effect will be fairly robust to departures from this assumption, but further investigation is required to support this conjecture. Although I would not expect single test confidence intervals based on this model to be robust to violations of this assumption, this problem can be circumvented by basing the confidence interval only on data for the corresponding test, as discussed in Section 5.3.

A Appendix

In this section I show that the OR mean squares, MS(T), MS(R), and MS(T^*R), have expectations as given in Table 1 and are independently distributed as follows: $\mathrm{MS}\left(\mathrm{T}\right)\sim \frac{c_1}{t-1}{\chi }_{t-1,\lambda }^2$, with

$c_1={\sigma }_{\tau R}^2+{\sigma }_{\epsilon }^2-{\mathrm{Cov}}_1+(r-1)({\mathrm{Cov}}_2-{\mathrm{Cov}}_3)$ (A1)

and noncentrality parameter $\lambda =\frac{r}{c_1}{\sum }_{i=1}^t{({\tau }_i-\tau )}^2;\mathrm{MS}\left(\mathrm{R}\right)\sim \frac{E\left[\mathrm{MS}\left(\mathrm{R}\right)\right]}{(r-1)}{\chi }_{(r-1)}^2$; and $\mathrm{MS}\left({\mathrm{T}}^{\ast }\mathrm{R}\right)\sim \frac{E\left[\mathrm{MS}\left({\mathrm{T}}^{\ast }\mathrm{R}\right)\right]}{(t-1)(r-1)}{\chi }_{(t-1)(r-1)}^2$. Furthermore, from Table 2 we see that E[MS(T) | H₀] = c ₁; hence if H₀: τ₁ = ... = τ_t is true, then $\mathrm{MS}\left(\mathrm{T}\right)\sim \frac{E\left[\mathrm{MS}\left(\mathrm{T}\right)\right]}{t-1}{\chi }_{t-1}^2$.

Let $\widehat{\theta }={({\widehat{\theta }}_{11},\dots ,{\widehat{\theta }}_{1r},\dots ,{\widehat{\theta }}_{t1},\dots ,{\widehat{\theta }}_{tr})}^{\prime }$ denote the vector of outcomes for the OR model (1) and let $\theta ={({\theta }_{11},\dots ,{\theta }_{1r},\dots ,{\theta }_{t1},\dots ,{\theta }_{tr})}^{\prime }$ denote the corresponding mean vector, where θ_ij = μ+τ_i. Then θ ∼ N (θ,Σ) where Σ=Cov(θ). I assume that Σ is positive definite.

I first show that Σ has the following form:

$$\Sigma ={\mathbf{I}}_t\otimes (x_1{\mathbf{I}}_r+x_2{\mathbf{M}}_r)+{\mathbf{M}}_t\otimes (x_3{\mathbf{I}}_r+{\mathrm{Cov}}_3{\mathbf{M}}_r)$$ (A2)

where ⊗ denotes the Kronecker product operator, I_p denotes the p × p identity matrix, M_p denotes a p × p matrix of ones,

$$x_1={\sigma }_{\tau R}^2+{\sigma }_{∊}^2-{\mathrm{Cov}}_1-{\mathrm{Cov}}_2+{\mathrm{Cov}}_3$$ (A3)

$$x_2={\mathrm{Cov}}_2-{\mathrm{Cov}}_3$$ (A4)

and

$$x_3={\sigma }_R^2+{\mathrm{Cov}}_1-{\mathrm{Cov}}_3$$ (A5)

It follows from the OR model (1) that $\mathrm{Var}\left({\widehat{\theta }}_{ij}\right)={\sigma }_R^2+{\sigma }_{\tau R}^2+{\sigma }_{\epsilon }^2$ and

$$\mathrm{Cov}({\widehat{\theta }}_{ij},{\widehat{\theta }}_{i^{\prime }{j}^{\prime }})=\{\begin{array}{cc}{\mathrm{Cov}}_1+{\sigma }_R^2\hfill & i\ne i^{\prime },j=j^{\prime }(\text{different test},\text{same reader})\hfill \\ {\mathrm{Cov}}_2\hfill & i=i^{\prime },j\ne j^{\prime }(\text{same test},\text{different reader})\hfill \\ {\mathrm{Cov}}_3\hfill & i\ne i^{\prime },j\ne j^{\prime }(\text{different test},\text{different reader})\hfill \end{array}\phantom{\}}$$

Define ${\sigma }_{\widehat{\theta }}^2={\sigma }_R^2+{\sigma }_{\tau R}^2+{\sigma }_{\epsilon }^2$. Then, for example, for t = 2 and r = 3 the covariance matrix of $\widehat{\theta }={({\widehat{\theta }}_{11},{\widehat{\theta }}_{12},{\widehat{\theta }}_{13},{\widehat{\theta }}_{21},{\widehat{\theta }}_{22},{\widehat{\theta }}_{23})}^{\prime }$ is given by

$$\Sigma =\left[\begin{array}{cc}\hfill {\Sigma }_1\hfill & \hfill {\Sigma }_2\hfill \\ \hfill {\Sigma }_2\hfill & \hfill {\Sigma }_1\hfill \end{array}\right]$$

where

$${\Sigma }_1=\left[\begin{array}{ccc}\hfill {\sigma }_{\widehat{\theta }}^2\hfill & \hfill {\mathrm{Cov}}_2\hfill & \hfill {\mathrm{Cov}}_2\hfill \\ \hfill {\mathrm{Cov}}_2\hfill & \hfill {\sigma }_{\widehat{\theta }}^2\hfill & \hfill {\mathrm{Cov}}_2\hfill \\ \hfill {\mathrm{Cov}}_2\hfill & \hfill {\mathrm{Cov}}_2\hfill & \hfill {\sigma }_{\widehat{\theta }}^2\hfill \end{array}\right],{\Sigma }_2=\left[\begin{array}{ccc}\hfill {\mathrm{Cov}}_1+{\sigma }_R^2\hfill & \hfill {\mathrm{Cov}}_3\hfill & \hfill {\mathrm{Cov}}_3\hfill \\ \hfill {\mathrm{Cov}}_3\hfill & \hfill {\mathrm{Cov}}_1+{\sigma }_R^2\hfill & \hfill {\mathrm{Cov}}_3\hfill \\ \hfill {\mathrm{Cov}}_3\hfill & \hfill {\mathrm{Cov}}_3\hfill & \hfill {\mathrm{Cov}}_1+{\sigma }_R^2\hfill \end{array}\right]$$

We can write

$$\Sigma =\left[\begin{array}{cc}\hfill {\Sigma }_1-{\Sigma }_2\hfill & \hfill 0\hfill \\ \hfill 0\hfill & \hfill {\Sigma }_1-{\Sigma }_2\hfill \end{array}\right]+\left[\begin{array}{cc}\hfill {\Sigma }_2\hfill & \hfill {\Sigma }_2\hfill \\ \hfill {\Sigma }_2\hfill & \hfill {\Sigma }_2\hfill \end{array}\right]={\mathbf{I}}_2\otimes ({\Sigma }_1-{\Sigma }_2)+{\mathbf{M}}_2\otimes {\Sigma }_2$$ (A6)

where 0 denotes a matrix of zeros. Since

$${\Sigma }_1-{\Sigma }_2=\left[\begin{array}{ccc}\hfill {\sigma }_{\tau R}^2+{\sigma }_{\epsilon }^2-{\mathrm{Cov}}_1\hfill & \hfill {\mathrm{Cov}}_2-{\mathrm{Cov}}_3\hfill & \hfill {\mathrm{Cov}}_2-{\mathrm{Cov}}_3\hfill \\ \hfill {\mathrm{Cov}}_2-{\mathrm{Cov}}_3\hfill & \hfill {\sigma }_{\tau R}^2+{\sigma }_{\epsilon }^2-{\mathrm{Cov}}_1\hfill & \hfill {\mathrm{Cov}}_2-{\mathrm{Cov}}_3\hfill \\ \hfill {\mathrm{Cov}}_2-{\mathrm{Cov}}_3\hfill & \hfill {\mathrm{Cov}}_2-{\mathrm{Cov}}_3\hfill & \hfill {\sigma }_{\tau R}^2+{\sigma }_{\epsilon }^2-{\mathrm{Cov}}_1\hfill \end{array}\right]=x_1{\mathbf{I}}_3+x_2{\mathbf{M}}_3$$

and Σ₂ = x ₃I₃+Cov₃M₃, where x ₁, x ₂, and x ₃ are defined by equations (A3-A5), then substituting in (A6) gives

$$\Sigma ={\mathbf{I}}_2\otimes (x_1{\mathbf{I}}_3+x_2{\mathbf{M}}_3)+{\mathbf{M}}_2\otimes (x_3{\mathbf{I}}_3+{\mathrm{Cov}}_3{\mathbf{M}}_3)$$

which is equation (A2) with t = 2; r = 3. It can similarly be shown for arbitrary t and r that the covariance matrix for $\widehat{\theta }={({\widehat{\theta }}_{11},\dots ,{\widehat{\theta }}_{1r},\dots ,{\widehat{\theta }}_{t1},\dots ,{\widehat{\theta }}_{tr})}^{\prime }$ is given by equation (A2).

Define ${\mathbf{C}}_p={\mathbf{I}}_p-\frac{1}{p}{\mathbf{M}}_p$. Smith and Lewis [27] give matrix expressions for balanced ANOVA sums of squares; from their results we have $\mathrm{MS}\left(\mathrm{T}\right)={\widehat{\theta }}^{\prime }{\mathbf{A}}_1\widehat{\theta }∕(t-1)$, $\mathrm{MS}\left({\mathrm{T}}^{\ast }\mathrm{R}\right)=\widehat{\theta }{\mathbf{A}}_2\widehat{\theta }∕\left[(t-1)(r-1)\right]$, and $\mathrm{MS}\left(\mathrm{R}\right)={\widehat{\theta }}^{\prime }{\mathbf{A}}_3\widehat{\theta }∕(r-1)$, where ${\mathbf{A}}_1=\frac{1}{r}{\mathbf{C}}_t\otimes {\mathbf{M}}_r$, A₂ = C_t ⊗ C_r, and ${\mathbf{A}}_3={}_t{}^1\mathbf{M}_t\otimes {\mathbf{C}}_r$.

Let A and B be any two matrices and let tr(·) denote the trace function. Results that I utilize for the mean square distribution derivations are given below. For properties (a-c) I also assume that A and B are tr × tr symmetric matrices.

1. $E\left({\widehat{\theta }}^{\prime }\mathbf{A}\widehat{\theta }\right)=\mathrm{tr}\left(\mathbf{A}\Sigma \right)+{\theta }^{\prime }\mathbf{A}\theta$

2. ${\widehat{\theta }}^{\prime }\mathbf{A}\widehat{\theta }\sim {\chi }_{m,\lambda }^2$ where m =rank(AΣ) and λ = θ′Aθ if and only if AΣ is idempotent

3. ${\widehat{\theta }}^{\prime }\mathbf{A}\widehat{\theta }$ and ${\widehat{\theta }}^{\prime }\mathbf{B}\widehat{\theta }$ are distributed independently if and only if AΣB = 0

4. 0 ⊗ A = A ⊗ 0 = 0

5. a A ⊗ b B = ab A ⊗ B

6. (A ⊗ C) (B ⊗ D) = (AB) ⊗ (CD)

7. (A + B) ⊗ C = A ⊗ C + B ⊗ C; C ⊗ (A + B) = C ⊗ A + C ⊗ B

8. rank(A ⊗ B) =rank(A) rank(B)

9. if B is nonsingular then rank(AB) =rank(A)

10. if A is idempotent then rank (A) =tr(A)

11. C_pC_p = C_p

12. M_pM_p = p M_p

13. M_pC_p = C_pM_p = 0

14. A ₁, A ₂, and A ₃ are idempotent

15. tr(A ₁) = t-1; tr(A ₂) = (t-1) (r-1); tr(A ₃) = (r-1)

16. A ₁ A ₂ = A ₁ A ₃ = A ₂ A ₃ = 0

Properties (a-c) are well known [28], (d-j) are standard matrix results [29], and properties (k-p) are easily derived.

To derive the distribution of $\mathrm{MS}\left(\mathrm{T}\right)={\widehat{\theta }}^{\prime }{\mathbf{A}}_1\widehat{\theta }∕(t-1)$, I show below that A ₁Σ = c ₁ A ₁, where c ₁ is defined by equation (A1). Using property (a) we then have

$$E\left[\mathrm{MS}\left(\mathrm{T}\right)\right]=\frac{1}{t-1}E\left({\widehat{\theta }}^{\prime }{\mathbf{A}}_1\widehat{\theta }\right)=\frac{1}{t-1}[\mathrm{tr}\left({\mathbf{A}}_1\Sigma \right)+{\theta }^{\prime }{\mathbf{A}}_1\theta ]=\frac{1}{t-1}[\mathrm{tr}\left(c_1{\mathbf{A}}_1\right)+{\theta }^{\prime }{\mathbf{A}}_1\theta ]$$

Since θ_ij = μ, then ${\theta }^{\prime }{\mathbf{A}}_1\theta =r{\sum }_{i=1}^t{({\theta }_{i\cdot }-{\theta }_{\cdot \cdot })}^2=r{\sum }_{i=1}^t{({\tau }_i-{\tau }_{\cdot })}^2$; also, from property (o) we have tr(c ₁ A ₁) = c ₁(t - 1). Thus

$$E\left[\mathrm{MS}\left(\mathrm{T}\right)\right]=\frac{1}{t-1}[\mathrm{tr}\left(c_1{\mathbf{A}}_1\right)+{\theta }^{\prime }{\mathbf{A}}_1\theta ]=\frac{1}{t-1}\left[c_1(t-1)+r\sum _{i=1}^t{({\tau }_i-{\tau }_{\cdot })}^2\right]={\sigma }_{\tau R}^2+{\sigma }_{\epsilon }^2-{\mathrm{Cov}}_1+(r-1)({\mathrm{Cov}}_2-{\mathrm{Cov}}_3)+\frac{r}{t-1}\sum _{i=1}^t{({\tau }_i-{\tau }_{\cdot })}^2$$

as given in Table 1. Since A ₁Σ = c ₁ A ₁, then from property (n) it follows that $\frac{1}{c_1}{\mathbf{A}}_1\Sigma$ is idempotent, and hence properties (b,i,j,o) imply $\frac{1}{c_1}{\widehat{\theta }}^{\prime }{\mathbf{A}}_1\widehat{\theta }\sim {\chi }_{t-1,\lambda }^2$ where the noncentrality parameter is $\lambda =\frac{1}{c_1}{\theta }^{\prime }{\mathbf{A}}_1\theta =\frac{r}{c_1}{\sum }_{i=1}^t{({\tau }_i-{\tau }_{\cdot })}^2$. It follows that $\mathrm{MS}\left(\mathrm{T}\right)\sim \frac{c_1}{t-1}{\chi }_{t-1,\lambda }^2$.

To show A ₁ = c ₁ A ₁, using (A2) we have

$${\mathbf{A}}_1\Sigma =\frac{1}{r}{\mathbf{C}}_t\otimes {\mathbf{M}}_r[{\mathbf{I}}_t\otimes (x_1{\mathbf{I}}_r+x_2{\mathbf{M}}_r)+{\mathbf{M}}_t\otimes (x_3{\mathbf{I}}_r-{\mathrm{Cov}}_3{\mathbf{M}}_r)]=\frac{1}{r}{\mathbf{C}}_t\otimes {\mathbf{M}}_r[{\mathbf{I}}_t\otimes (x_1{\mathbf{I}}_r+x_2{\mathbf{M}}_r)]+\frac{1}{r}{\mathbf{C}}_t\otimes {\mathbf{M}}_r[{\mathbf{M}}_t\otimes (x_3{\mathbf{I}}_r+{\mathrm{Cov}}_3{\mathbf{M}}_r)]$$

Using property (f) it follows that

$${\mathbf{A}}_1\Sigma =\frac{1}{r}\left({\mathbf{C}}_t{\mathbf{I}}_t\right)\otimes \left[{\mathbf{M}}_r(x_1{\mathbf{I}}_r+x_2{\mathbf{M}}_r)\right]+\frac{1}{r}\left({\mathbf{C}}_t{\mathbf{M}}_t\right)\otimes \left[{\mathbf{M}}_r(x_3{\mathbf{I}}_r+{\mathrm{Cov}}_3{\mathbf{M}}_r)\right]$$

Using properties (d,e,g,l,m) we have

$${\mathbf{A}}_1\Sigma =\frac{x_1}{r}{\mathbf{C}}_t\otimes {\mathbf{M}}_r+\frac{x_2}{r}{\mathbf{C}}_t\otimes {\mathbf{M}}_r{\mathbf{M}}_r+0=\frac{(x_1+r{x}_2)}{r}{\mathbf{C}}_t\otimes {\mathbf{M}}_r=(x_1+r{x}_2){\mathbf{A}}_1=c_1\mathbf{A}$$

I similarly derive the distribution of $\mathrm{MS}\left({\mathrm{T}}^{\ast }\mathrm{R}\right)={\widehat{\theta }}^{\prime }{\mathbf{A}}_2\widehat{\theta }∕\left[(t-1)(r-1)\right]$ by showing below that A ₂Σ = c ₂ A ₂, where

$$c_2={\sigma }_{\tau R}^2+{\sigma }_{\epsilon }^2-{\mathrm{Cov}}_1-{\mathrm{Cov}}_2+{\mathrm{Cov}}_3$$

Since ${\theta }^{\prime }{\mathbf{A}}_2\theta ={\sum }_{i=1}^t{\sum }_{j=1}^r{({\theta }_{ij}-{\theta }_{i\cdot }-{\theta }_{\cdot j}+{\theta }_{\cdot \cdot })}^2=0$, then similar to the derivation for MS(T) distribution it follows that

$$E\left[\mathrm{MS}\left({\mathrm{T}}^{\ast }\mathrm{R}\right)\right]=\frac{1}{(t-1)(r-1)}[\mathrm{tr}\left(c_2{\mathbf{A}}_2\right)+{\theta }^{\prime }{\mathbf{A}}_2\theta ]=c_2={\sigma }_{\tau R}^2+{\sigma }_{\epsilon }^2-{\mathrm{Cov}}_1-{\mathrm{Cov}}_2+{\mathrm{Cov}}_3$$

as given in Table 1, and $\frac{1}{c_2}{\widehat{\theta }}^{\prime }{\mathbf{A}}_2\widehat{\theta }\sim {\chi }_{(t-1)(r-1)}^2$. Since E [MS(T^*R)] = c ₂, then it follows that $\mathrm{MS}\left({\mathrm{T}}^{\ast }\mathrm{R}\right)\sim \frac{E\left[\mathrm{MS}\left({\mathrm{T}}^{\ast }\mathrm{R}\right)\right]}{(t-1)(r-1)}{\chi }_{(t-1)(r-1)}^2$.

To show A ₂Σ = c ₂ A ₂, using (A2) we have

$${\mathbf{A}}_2\Sigma ={\mathbf{C}}_t\otimes {\mathbf{C}}_r[{\mathbf{I}}_t\otimes (x_1{\mathbf{I}}_r+x_t{\mathbf{M}}_r)+{\mathbf{M}}_t\otimes (x_3{\mathbf{I}}_r+{\mathrm{Cov}}_3{\mathbf{M}}_r)]={\mathbf{C}}_t\otimes {\mathbf{C}}_r[{\mathbf{I}}_t\otimes (x_1{\mathbf{I}}_r+x_t{\mathbf{M}}_r)]+{\mathbf{C}}_t\otimes {\mathbf{C}}_r[{\mathbf{M}}_t\otimes (x_3{\mathbf{I}}_r+{\mathrm{Cov}}_3{\mathbf{M}}_r)]$$

Using property (f) it follows that

$${\mathbf{A}}_2\Sigma =\left({\mathbf{C}}_t{\mathbf{I}}_t\right)\otimes \left[{\mathbf{C}}_r(x_1{\mathbf{I}}_r+x_2{\mathbf{M}}_r)\right]+\left({\mathbf{C}}_t{\mathbf{M}}_t\right)\otimes \left[{\mathbf{C}}_r(x_3{\mathbf{I}}_r+{\mathrm{Cov}}_3{\mathbf{M}}_r)\right]$$

Using properties (d,e,g,m) we have

$${\mathbf{A}}_2\Sigma ={\mathbf{C}}_t\otimes (x_1{\mathbf{C}}_r+0)+0=x_1{\mathbf{C}}_t\otimes {\mathbf{C}}_r=x_1{\mathbf{A}}_2=c_2\mathbf{A}$$

From property (c), ${\widehat{\theta }}^{\prime }{\mathbf{A}}_1\widehat{\theta }$ and ${\widehat{\theta }}^{\prime }{\mathbf{A}}_2\widehat{\theta }$ are independent if A ₁ΣA ₂ = 0. To show A ₁ΣA ₂ = 0, note that since A ₁Σ = c ₁ A ₁, then A ₁ΣA ₂ = c ₁ A ₁ A ₂ = 0 using property (p). It follows that MS(T) and MS(T^*R) are independent. Similarly, the distribution of MS(R) can be derived and it can be shown that MS(R) is independent of MS(T) and MS(T^*R).