Table 6.

Mm-ANOVA approach for typical test×reader×case factorial study design

Derive the mm-ANOVA model Conventional ANOVA model: Yijk = μ + τi + Rj + Ck + (τR)ij + (τC)ik + (RC)jk + (τRC)ijk + εijk, i = 1, …, t; j = 1, …, r; k = 1, …, c, with variance components ${\sigma }_R^2,{\sigma }_C^2,{\sigma }_{TR}^2,{\sigma }_{TC}^2,{\sigma }_{RC}^2,{\sigma }_{\tau RC}^2$, and ${\sigma }_{\epsilon }^2$ and constraint ${\sum }_{i=1}^t{\tau }_i=0$. Define ${\sigma }^2={\sigma }_{TRC}^2+{\sigma }_{\epsilon }^2$. Mm-ANOVA model (note: Ỹij = Yij•): ${Ỹ}_{ij}=\mu +{\tau }_i+R_j+{(\tau R)}_{ij}+{\tilde{\epsilon }}_{ij}$ where ${\tilde{\epsilon }}_{ij}=C_{•}+{(\tau C)}_{i•}+{(RC)}_{j•}+{(\tau RC)}_{ij•}+{\epsilon }_{ij•}$ and ${\sum }_{i=1}^t{\tau }_i=0$ Mm-ANOVA error variance and covariances expressed in terms of conventional ANOVA variance components: ${\tilde{\sigma }}_{\epsilon }^2=\frac{1}{c}({\sigma }_C^2+{\sigma }_{TC}^2+{\sigma }_{RC}^2+{\sigma }^2)$, ${\text{Cov}}_1\equiv \text{cov}({\tilde{\epsilon }}_{ij},{\tilde{\epsilon }}_{i\prime j})=\frac{1}{c}({\sigma }_C^2+{\sigma }_{RC}^2)$, ${\text{Cov}}_2\equiv \text{cov}({\tilde{\epsilon }}_{ij},{\tilde{\epsilon }}_{ij\prime })=\frac{1}{c}({\sigma }_C^2+{\sigma }_{\tau C}^2)$, ${\text{Cov}}_3\equiv \text{cov}({\tilde{\epsilon }}_{ij},{\tilde{\epsilon }}_{i\prime j\prime })=\frac{1}{c}{\sigma }_C^2$, where i ≠ i′, j ≠ j′ Covariance constraints: Cov1 ≥ Cov3; Cov2 ≥ Cov3; Cov3 ≥ 0 Derive the mm-ANOVA test statistic and its null distribution Mm-ANOVA model hypothesis of equal test accuracies: H0 : θ1 = ⋯ = θt where θi = E (Ỹi•) Conventional ANOVA model hypothesis: θi = E (Yi••) = μ + τi ⇒ H0 : τ1 = ⋯ = τt Conventional ANOVA expected mean squares Mean square Expected mean square MS(T) $\frac{rc}{(t-1)}{\sum }_{i=1}^t{\tau }_i^2+c{\sigma }_{TR}^2+r{\sigma }_{TC}^2+{\sigma }^2$ MS(R) $tc{\sigma }_R^2+c{\sigma }_{TR}^2+t{\sigma }_{RC}^2+{\sigma }^2$ MS(C) $tr{\sigma }_C^2+r{\sigma }_{TC}^2+t{\sigma }_{RC}^2+{\sigma }^2$ MS(T * R) $c{\sigma }_{TR}^2+{\sigma }^2$ MS(T * C) $r{\sigma }_{TC}^2+{\sigma }^2$ MS(R * C) $t{\sigma }_{RC}^2+{\sigma }^2$ MS(T * R * C) ${\sigma }^2\equiv {\sigma }_{TRC}^2+{\sigma }_{\epsilon }^2$ Conventional ANOVA test statistic: $F=\frac{\mathrm{M}\mathrm{S}(T)}{\mathrm{M}\mathrm{S}(T*R)+\mathrm{M}\mathrm{S}(T*C)-\mathrm{M}\mathrm{S}(T*R*C)}$ $$\widetilde{\mathrm{M}\mathrm{S}}(T)=\frac{1}{c}\mathrm{M}\mathrm{S}(T),\widetilde{\mathrm{M}\mathrm{S}}(T*R)=\frac{1}{c}\mathrm{M}\mathrm{S}(T*R),\widetilde{\mathrm{M}\mathrm{S}}(R)=\frac{1}{c}\mathrm{M}\mathrm{S}(R)$$ $F=\frac{\widetilde{\mathrm{M}\mathrm{S}}(T)}{\widetilde{\mathrm{M}\mathrm{S}}(T*R)+U}$ where $U=\frac{1}{c}\{\mathrm{M}\mathrm{S}(T*C)-\mathrm{M}\mathrm{S}(T*R*C)\}$ $$E\{\mathrm{M}\mathrm{S}(T*C)\}=r{\sigma }_{TC}^2+{\sigma }^2,E\{\mathrm{M}\mathrm{S}[T*R*C]\}={\sigma }^2\Rightarrow E(U)=\frac{1}{c}(r{\sigma }_{TC}^2)=r({\text{Cov}}_2-{\text{Cov}}_3).$$ $$F_{\mathrm{O}\mathrm{R}}^{*}=\frac{\widetilde{\mathrm{M}\mathrm{S}}(T)}{\widetilde{\mathrm{M}\mathrm{S}}(T*R)+r({\text{Cov}}_2-{\text{Cov}}_3)}$$ $$F_{\mathrm{O}\mathrm{R}}=\frac{\widetilde{\mathrm{M}\mathrm{S}}(T)}{\widetilde{\mathrm{M}\mathrm{S}}(T*R)+r\text{max}({\widehat{\text{Cov}}}_2-{\widehat{\text{Cov}}}_3,0)}$$ Under H0, FOR ≈ Ft−1,df2 where $\mathrm{d}{\mathrm{f}}_2=\frac{{[\widetilde{\mathrm{M}\mathrm{S}}[T*R]+r\text{max}({\widehat{\text{Cov}}}_2-{\widehat{\text{Cov}}}_3,0)]}^2}{{[\widetilde{\mathrm{M}\mathrm{S}}[T*R]]}^2/[(t-1)(r-1)]}$ Derive confidence intervals (a) Mm-ANOVA test accuracy parameters: θ = (θ1, …, θt)′, with θi = E (Ỹi•), i = 1, …, t (b) Corresponding conventional ANOVA parameters: θi = E (Yi••) = μ + τi (c) Conventional ANOVA estimate: θ̂i = Yi•• CI for l′ (θ) with l = (l1, …, lt)′, ${\sum }_{i=1}^t{l}_i=0$: (d) $$\mathit{l}\prime (\hat{\mathit{\theta }})={\sum }_{i=1}^t{l}_i{\hat{\mathit{\theta }}}_i={\sum }_{i=1}^t{l}_i{Y}_{i••}={\sum }_{i=1}^t{l}_i{\tau }_i+{\sum }_{i=1}^t{l}_i[{(\tau R)}_{i•}+{(\tau C)}_{i•}+{(\tau RC)}_{i••}+{\epsilon }_{i••}]\Rightarrow V={\sum }_{i=1}^t{l}_i^2[\frac{{\sigma }_{TR}^2}{r}+\frac{{\sigma }_{TC}^2}{c}+\frac{{\sigma }^2}{rc}]=\frac{1}{rc}{\sum }_{i=1}^t{l}_i^2[c{\sigma }_{TR}^2+r{\sigma }_{TC}^2+{\sigma }^2]$$ (e) $$V=\frac{1}{rc}{\sum }_{i=1}^t{l}_i^{2}E[\mathrm{M}\mathrm{S}(T*R)+\mathrm{M}\mathrm{S}(T*C)-\mathrm{M}\mathrm{S}(T*R*C)]$$ (f) $V=\frac{1}{r}{\sum }_{i=1}^t{l}_i^{2}E[\widetilde{\mathrm{M}\mathrm{S}}(T*R)+U]$ where $U=\frac{1}{c}\{\mathrm{M}\mathrm{S}(T*C)-\mathrm{M}\mathrm{S}(T*R*C)\}$ (g) $$E(U)=\frac{r{\sigma }_{TC}^2}{c}=r({\text{Cov}}_2-{\text{Cov}}_3)\Rightarrow V=\frac{1}{r}{\sum }_{i=1}^t{l}_i^2\left\{E\right[\widetilde{\mathrm{M}\mathrm{S}}(T*R)]+r({\text{Cov}}_2-{\text{Cov}}_3)\}$$ (h) $$\hat{V}=\frac{1}{r}{\sum }_{i=1}^t{l}_i^2\{\widetilde{\mathrm{M}\mathrm{S}}(T*R)+\text{max}[r({\widehat{\text{Cov}}}_2-{\widehat{\text{Cov}}}_3),0\left]\right\}$$ (i) $\mathrm{d}{\mathrm{f}}_2=\frac{{[\widetilde{\mathrm{M}\mathrm{S}}(T*R)+r\text{max}({\widehat{\text{Cov}}}_2-{\widehat{\text{Cov}}}_3,0)]}^2}{{[\widetilde{\mathrm{M}\mathrm{S}}(T*R)]}^2/[(t-1)(r-1)]}$ (same as df2 in step 2j) (j) θ̂i = Ỹi• (k) $$\mathrm{C}\mathrm{I}:{\sum }_{i=1}^t{l}_i{Ỹ}_{i•}\pm t_{\alpha /2;\mathrm{d}{\mathrm{f}}_2}\sqrt{\frac{1}{r}{\sum }_{i=1}^t{l}_i^2\{\widetilde{\mathrm{M}\mathrm{S}}(T*R)+\text{max}[r({\widehat{\text{Cov}}}_2-{\widehat{\text{Cov}}}_3),0\left]\right\}}$$ CI for θi (d) $${\hat{\theta }}_i=Y_{i••}={\tau }_i+R_{•}+C_{•}+{(\tau R)}_{i•}+{(\tau C)}_{i•}+{(RC)}_{••}+{(\tau RC)}_{i••}+{\epsilon }_{i••}\Rightarrow V=\frac{{\sigma }_R^2}{r}+\frac{{\sigma }_C^2}{c}+\frac{{\sigma }_{TR}^2}{r}+\frac{{\sigma }_{TC}^2}{c}+\frac{{\sigma }_{RC}^2}{rc}+\frac{{\sigma }^2}{rc}=\frac{1}{rc}(c{\sigma }_R^2+r{\sigma }_C^2+c{\sigma }_{TR}^2+r{\sigma }_{TC}^2+{\sigma }_{RC}^2+{\sigma }^2)$$ (e) $$V=\frac{1}{trc}E[\mathrm{M}\mathrm{S}(R)+(t-1)\mathrm{M}\mathrm{S}(T*R)+\mathrm{M}\mathrm{S}(C)-\mathrm{M}\mathrm{S}(R*C)+(t-1)\mathrm{M}\mathrm{S}(T*C)-(t-1)\mathrm{M}\mathrm{S}(T*R*C)]$$ (f) $$V=\frac{1}{tr}E[\widetilde{\mathrm{M}\mathrm{S}}(R)+(t-1)\widetilde{\mathrm{M}\mathrm{S}}(T*R)+U]$$ where $$U=\frac{1}{c}\{\mathrm{M}\mathrm{S}(C)-\mathrm{M}\mathrm{S}(R*C)+(t-1)\mathrm{M}\mathrm{S}(T*C)-(t-1)\mathrm{M}\mathrm{S}(T*R*C)\}$$ (g) $$E(U)=\frac{tr}{c}({\sigma }_C^2+{\sigma }_{TC}^2)=tr{\text{Cov}}_2\Rightarrow V=\frac{1}{tr}\left\{E\right[\widetilde{\mathrm{M}\mathrm{S}}(R)+(t-1)\widetilde{\mathrm{M}\mathrm{S}}(T*R)]+tr{\text{Cov}}_2\}$$ (h) $$\hat{V}=\frac{1}{tr}[\widetilde{\mathrm{M}\mathrm{S}}(R)+(t-1)\widetilde{\mathrm{M}\mathrm{S}}(T*R)+tr\text{max}({\widehat{\text{Cov}}}_2,0\left)\right]$$ (i) $$\mathrm{d}{\mathrm{f}}_2=\frac{{[\widetilde{\mathrm{M}\mathrm{S}}(R)+(t-1)\widetilde{\mathrm{M}\mathrm{S}}(T*R)+tr\text{max}({\widehat{\text{Cov}}}_2,0\left)\right]}^2}{\frac{{[\widetilde{\mathrm{M}\mathrm{S}}(R)]}^2}{r-1}+\frac{{[(t-1)\widetilde{\mathrm{M}\mathrm{S}}(T*R)]}^2}{(t-1)(r-1)}}$$ (j) θ̂i = Ỹi• (k) $$\mathrm{C}\mathrm{I}:{Ỹ}_{i•}\pm t_{\alpha /2;\mathrm{d}{\mathrm{f}}_2}\sqrt{\frac{1}{tr}[\widetilde{\mathrm{M}\mathrm{S}}(R)+(t-1)\widetilde{\mathrm{M}\mathrm{S}}(T*R)+tr\text{max}({\widehat{\text{Cov}}}_2,0\left)\right]}$$ Derive the non-null distribution Fdf1,df2;λ of the step-2 F statistic Step 2d F numerator: MSnum = MS(T), $E[\mathrm{M}\mathrm{S}(T)]=\frac{rc}{(t-1)}{\sum }_{i=1}^t{\tau }_i^2+c{\sigma }_{TR}^2+r{\sigma }_{TC}^2+{\sigma }^2$, df (MS (T)) = t − 1, $E(Y_{ijk})=\mu +{\tau }_i\Rightarrow \lambda =\frac{\mathrm{d}\mathrm{f}(\mathrm{M}{\mathrm{S}}_{\text{num}}){\mathrm{M}{\mathrm{S}}_{\text{num}}|}_{\mathbf{Y}=E(\mathbf{Y})}}{E(\mathrm{M}{\mathrm{S}}_{\text{num}}|{\mathrm{H}}_0)}=\frac{rc{\sum }_{i=1}^t{\tau }_i^2}{c{\sigma }_{TR}^2+r{\sigma }_{TC}^2+{\sigma }^2}$ $$r{\sigma }_{TC}^2+{\sigma }^2=c[{\sigma }_{\tilde{\epsilon }}^2-{\text{Cov}}_1+(r-1)({\text{Cov}}_2-{\text{Cov}}_3)]\Rightarrow \lambda =\frac{r{\sum }_{i=1}^t{\tau }_i^2}{{\sigma }_{TR}^2+{\sigma }_{\tilde{\epsilon }}^2-{\text{Cov}}_1+(r-1)({\text{Cov}}_2-{\text{Cov}}_3)}$$ Step 2h $F_{\mathrm{O}\mathrm{R}}^{*}\text{denominator}=\widetilde{\mathrm{M}\mathrm{S}}(T*R)+r({\text{Cov}}_2-{\text{Cov}}_3)$, $$\begin{gathered} E(\widetilde{\mathrm{M}\mathrm{S}}(T*R))=\frac{1}{c}E(\widetilde{\mathrm{M}\mathrm{S}}(T*R))=\frac{1}{c}(c{\sigma }_{TR}^2+{\sigma }^2)=({\sigma }_{TR}^2+{\sigma }_{\tilde{\epsilon }}^2-{\text{Cov}}_1-{\text{Cov}}_2+{\text{Cov}}_3)\Rightarrow \mathrm{d}{\mathrm{f}}_2 \\ =\frac{{[{\sigma }_{TR}^2+{\sigma }_{\tilde{\epsilon }}^2-{\text{Cov}}_1+(r-1)({\text{Cov}}_2-{\text{Cov}}_3)]}^2}{\frac{{[{\sigma }_{TR}^2+{\sigma }_{\tilde{\epsilon }}^2-{\text{Cov}}_1-{\text{Cov}}_2+{\text{Cov}}_3]}^2}{(t-1)(r-1)}} \end{gathered}$$ FOR ~˙ Ft−1,df2;λ