Table 5.

Algorithm for deriving mm-ANOVA formulas

Derive the mm-ANOVA model Define the conventional ANOVA model that corresponds to the study design as if each reader-performance measure was the mean of case-level outcomes. (Note: Since reader-performance measures are measures of discrimination between diseased and nondiseased cases, disease status should not be included as a factor.) From the conventional ANOVA model defined in step 1a, derive the mm-ANOVA model by averaging across cases. Define the mm-ANOVA model error term equal to the mean, across cases, of the sum of the conventional ANOVA model error term and random effects involving case. Express the mm-ANOVA model error variance and covariances in terms of the conventional ANOVA model variance components. Determine the mm-ANOVA model covariance constraints implied by step 1c. Derive the mm-ANOVA model test statistic and its null distribution for a hypothesis express in terms of test accuracies (i.e., expected reader-performance measures) State the hypothesis of interest in terms of the mm-ANOVA model. Express the hypotheses from step 2a in terms of the conventional ANOVA model. Create the expected-mean-square table for the conventional ANOVA model Determine the conventional ANOVA F statistic corresponding to the step 2b hypotheses. Express mm-ANOVA mean squares in terms of conventional ANOVA mean squares. Express F from step 2d in terms of the mm-ANOVA model mean squares and U, where U is a linear function of conventional ANOVA model mean squares that involve case. Express E (U) in terms of conventional ANOVA model variance components, and then in terms of mm-ANOVA model error covariance parameters using the relationships from step 1c. Modify F from step 2f to produce the mm-ANOVA statistic $F_{\mathrm{O}\mathrm{R}}^{*}$ by replacing U by E (U), expressed as a linear function of mm-ANOVA covariance parameters. Derive FOR by replacing covariance parameters in $F_{\mathrm{O}\mathrm{R}}^{*}$ by estimates that take into account the constraints from step 1d. Determine the approximate null distribution of FOR in the following way: Write the denominator of FOR in the form $b({\sum }_i{a}_i\widetilde{\mathrm{M}{\mathrm{S}}_i}+\hat{d})$ where the $\widetilde{\mathrm{M}{\mathrm{S}}_i}$ are mm-ANOVA model mean squares, d̂ is a function of the covariance parameter estimates, and the ai and b are constants. Then FOR will have an approximate Fdf1,df2 null distribution, where df1 is the numerator degrees of freedom for the conventional ANOVA model test statistic in step 2d and df2 is given by $$\mathrm{d}{\mathrm{f}}_2=\frac{{[{\sum }_i{a}_i\widetilde{\mathrm{M}{\mathrm{S}}_i}+\hat{d}]}^2}{{\sum }_i\frac{{[a_i\widetilde{\mathrm{M}{\mathrm{S}}_i}]}^2}{\mathrm{d}\mathrm{f}(\widetilde{\mathrm{M}{\mathrm{S}}_i})}}$$ where $\mathrm{d}\mathrm{f}\left(\widetilde{\mathrm{M}{\mathrm{S}}_i}\right)$ is the degrees of freedom for $\widetilde{\mathrm{M}{\mathrm{S}}_i}$, and hence also for MSi. Derive confidence intervals for a linear function g (θ) of test accuracy parameters. Write the test accuracy parameter vector θ in terms of the mm-ANOVA model. Write θ in terms of the conventional ANOVA model. Determine the conventional ANOVA estimate for θ, denoted by θ̂. Determine the variance V of g (θ̂) in terms of conventional ANOVA parameters. Write V from step 3d in the form V = bE (∑ aiMSi) for constants b and ai. Write V from step 3e in the form $V=\tilde{b}E(\sum {ã}_i\widetilde{\mathrm{M}{\mathrm{S}}_i}+U)$ where b̃ and ãi are constants and U is a linear function of conventional ANOVA mean squares that involve case. Express E (U) in terms of conventional ANOVA model variance components and then in terms of mm-ANOVA model error covariance parameters, using the relationships from step 1c; then rewrite V using this expression for E (U). Derive the variance estimate V̂ from V by replacing expected mean squares by mean squares and replacing covariances by estimates that take into account the constraints from step 1d. Derive the degrees of freedom df2 for V̂ using the general formula for df2 given in step 2j. Write θ̂ from step 3c in terms of the mm-ANOVA model. An approximate (1 − α) 100% confidence interval for g (θ) is given by $g\left(\hat{\mathit{\theta }}\right)\pm t_{\alpha /2;\mathrm{d}{\mathrm{f}}_2}\sqrt{\hat{V}}$, where V̂ is determined in step 3h, df2 in step 3i and θ̂ in step 3j. Derive the non-null distribution of FOR from step 2i Compute the noncentrality parameter in terms of the conventional ANOVA model: $\lambda =\frac{\mathrm{d}\mathrm{f}(\mathrm{M}{\mathrm{S}}_{\text{num}})\mathrm{M}{\mathrm{S}}_{\text{num}}{|}_{\mathbf{Y}=E(\mathbf{Y})}}{E(\mathrm{M}{\mathrm{S}}_{\text{num}}|{\mathrm{H}}_0)}$ where MSnum is the numerator mean square from the conventional ANOVA F statistic given in step 2d. Express λ in terms of mm-ANOVA parameters by replacing variance components involving case by mm-ANOVA covariances. Determine the denominator degrees of freedom in terms of mm-ANOVA parameters using $\mathrm{d}{\mathrm{f}}_2=\frac{{[\sum _i{a}_{i}E(\widetilde{\mathrm{M}{\mathrm{S}}_i})+d]}^2}{\sum _i{[a_{i}E(\widetilde{\mathrm{M}{\mathrm{S}}_i})]}^2/\mathrm{d}\mathrm{f}(\widetilde{\mathrm{M}{\mathrm{S}}_i})}$ where $b({\sum }_i{a}_i\widetilde{\mathrm{M}{\mathrm{S}}_i}+d)$ is the denominator of $F_{\mathrm{O}\mathrm{R}}^{*}$ from step 2h The non-null distribution is given by Fdf1,df2;λ, where df1 = df (MSnum), df2 is determined in step 4c and λ in step 4b.