Evaluating the Improvement in Diagnostic Utility from Adding New Predictors

Summary

Multiple diagnostic tests and risk factors are commonly available for many diseases. This information can be either redundant or complimentary. Combinations of these tests and risk factors may improve the diagnostic/predictive accuracy but also may unnecessarily increase complexity, risks, and/or costs. The improved accuracy gained by including additional variables can be evaluated by the increment of the area under (AUC) the receiver operating characteristic (ROC) curves with and without the new variable(s). In this paper, we derive a new test statistic to accurately and efficiently determine the statistical significance of this incremental AUC under a multivariate normality assumption. Our test links the difference in AUC to a quadratic form of a standardized mean shift in a unit of the inverse covariance matrix through a properly linear transformation of all diagnostic variables. The distribution of the estimator of the quadratic form is related to the multivariate Behrens-Fisher problem. We provide explicit mathematical solutions of the estimator and its approximate noncentral F-distribution, type I error rate, and sample size formula. We use simulation studies to prove that our new test maintains prespecified type I error rates as well as reasonable statistical power under practical sample sizes. We use data from the Study of Osteoporotic Fractures (SOF) as an application example to illustrate our method.

1 Introduction

The utility of a diagnostic test is measured in sensitivity and specificity. For ordinal diagnostic variables, a receiver operating characteristic (ROC) curve plots the false positive rate (FPR, or 1-specificity) and the true positive rate (TPR, or sensitivity) over the diagnostic decision range. The area under the ROC curve (AUC), ranging from 0.5 to 1, is a summary measure of overall diagnostic utility (Pepe et al., 2004; Wang et al., 2006; Ware, 2006). For many diseases, more than one diagnostic test is available, and combining them may improve the diagnostic accuracy. While the optimal method to summarize multiple variables is the likelihood ratio score (McIntosh and Pepe, 2002; Neyman and Pearson, 1933), the most common medical applications use simple linear combinations. Jin and Lu proposed a bootstrap-based non-parametric test (Jin and Lu, 2008) and a permutation test (Jin and Lu, 2009a) to confirm the non-inferiority of linear combinations to the optimal likelihood ratio score. In the remaining paper, we will focus only on linear combinations of diagnostic variables. The ROC curve corresponding to linear combinations has been called the generalized ROC (GROC) curve by Reiser and Faraggi (Reiser and Faraggi, 1997).

Under an assumption of multivariate normally distributed diagnostic variables for both diseased and non-diseased populations, Su and Liu (Su and Liu, 1993) proved that the Fisher’s discriminant function is the optimal linear combination (OLC) that maximizes the AUC of the ROC curves. Jin and Lu (Jin and Lu, 2009b) extend Su and Liu to the generalized linear models. Pepe and colleagues (Pepe, Cai and Longton, 2006; Pepe and Thompson, 2000) relaxed the multivariate normality assumption and found linear forms which maximize the non-parametric estimate of the AUC. Schisterman, Schisterman, Faraggi and Reiser (Schisterman, Faraggi and Reiser, 2004) discussed linear combinations of multivariate normally distributed diagnostic variables with an adjustment for covariates. Liu, Schisterman and Zhu (Liu, Schisterman and Zhu, 2005) proposed the OLC of multivariate normal variables for a given range of specificity. Cai and Cheng (Cai and Cheng, 2008) proposed robust combination of multiple diagnostic tests for censored data. While these papers constructed the OLCs, they did not evaluate the incremental AUC by adding new variables to a panel of possibly prior multiple diagnostic variables.

The OLCs with and without new variables are different. Adding new variables to a panel of multiple diagnostic variables always increases the AUC of the ROC curve. However, the increment in the apparent AUC may be too small to justify the increase in cost, risk, or logistic complications. This paper addresses statistical estimation and inference for the incremental AUC. We focus our comparisons on linear combinations of multivariate normal diagnostic variables based on linear discriminant function. Our test compares two AUCs from two combinations based on an incremental standardized quadratic form. Explicit analytic estimator and test statistics are derived. For continuous diagnostic variables that fail the normal distributional assumption, a Box-Cox type power transformation can be used prior to analysis.

Our paper is organized as follows. In Section 2, we describe the ROC model for linear combinations and define the index of incremental utility. In Section 3, we discuss the estimator and its approximate distribution. In Section 4, we propose a hypothesis testing procedure and sample size formula. Section 5 applies the proposed methods to the Study of Osteoporotic Fractures (SOF). Section 6 presents simulation experiments, and Section 7 offers discussion and conclusions.

2 Index of Diagnostic Utility Improvement

We use bold letters for vectors and matrices. Let ${\mathit{X}}^{(k)}={(X_1^{(k)},X_2^{(k)},\cdots ,X_p^{(k)})}^{\prime }$ be p-dimensional normally distributed diagnostic variables. A superscript indicates observations from diseased (k=1) or non-diseased (k=0) populations; thus X⁽⁰⁾ is independent of X⁽¹⁾ (i.e., X⁽⁰⁾ ⊥ X⁽¹⁾). Let ${\mathit{X}}^{(k)}:\mathit{N}({\mathit{\mu }}_{\mathit{X}}^{(k)},{\mathit{\sum }}_{\mathit{XX}}^{(k)})$ For a p-dimensional vector a, the AUC of the ROC curve constructed by the linear combination a′X is

$$\mathrm{AUC}({\mathit{a}}^{\prime }\mathit{X})=\mathrm{\Phi }(\frac{{\mathit{a}}^{\prime }{\mathit{\mu }}_{\mathit{X}}}{\sqrt{{\mathit{a}}^{\prime }{\mathit{\sum }}_{\mathit{XX}}\mathit{a}}}),$$

where ${\mathit{\mu }}_{\mathit{X}}={\mathit{\mu }}_{\mathit{X}}^{(1)}-{\mathit{\mu }}_{\mathit{X}}^{(0)}$, ${\mathit{\sum }}_{\mathit{XX}}={\mathit{\sum }}_{\mathit{XX}}^{(1)}+{\mathit{\sum }}_{\mathit{XX}}^{(0)}$, and Φ is the cumulative distribution function (CDF) of the standard normal distribution. As Su and Liu (Su and Liu, 1993) pointed out, the linear discriminant function

$$L(\mathit{X})={({\mathit{\mu }}_{\mathit{X}})}^{\prime }{\mathit{\sum }}_{\mathit{XX}}^{-1}\mathit{X}$$ (1)

(or $\mathit{a}={\mathit{\sum }}_{\mathit{XX}}^{-1}{\mathit{\mu }}_{\mathit{X}}$) maximizes the AUC as

$${\theta }_{L(\mathit{X})}=\underset{\mathit{a}}{\max }\mathrm{AUC}\left({\mathit{a}}^{\prime }\mathit{X}\right)=\mathrm{AUC}\left({\mathit{\mu }}_{\mathit{X}}^{\prime }{\mathit{\sum }}_{\mathit{XX}}^{-1}\mathit{X}\right)=\mathrm{\Phi }\left(\sqrt{{\mathit{\mu }}_{\mathit{X}}^{\prime }{\mathit{\sum }}_{\mathit{XX}}^{-1}{\mathit{\mu }}_{\mathit{X}}}\right)$$ (2)

Thus the AUC of the OLC is a monotonic function of the quadratic form of a standardized mean shift in a unit of the inverse covariance matrix, ${\mathit{\mu }}_{\mathit{X}}^{\prime }{\mathit{\sum }}_{\mathit{XX}}^{-1}{\mathit{\mu }}_{\mathit{X}}$. The ROC curve of the OLC is

$$\begin{gathered} \mathrm{ROC}(t)=\mathrm{\Phi }\left(\frac{{\mathit{\mu }}_{\mathit{X}}^{\prime }{\mathit{\sum }}_{\mathit{XX}}^{-1}{\mathit{\mu }}_{\mathit{X}}}{\sqrt{{\mathit{\mu }}_{\mathit{X}}^{\prime }{\mathit{\sum }}_{\mathit{XX}}^{-1}{\mathit{\sum }}_{\mathit{XX}}^{(1)}{\mathit{\sum }}_{\mathit{XX}}^{-1}{\mathit{\mu }}_{\mathit{X}}}} \\ +\frac{\sqrt{{\mathit{\mu }}_{\mathit{X}}^{\prime }{\mathit{\sum }}_{\mathit{XX}}^{-1}{\mathit{\sum }}_{\mathit{XX}}^{(0)}{\mathit{\sum }}_{\mathit{XX}}^{-1}{\mathit{\mu }}_{\mathit{X}}}}{\sqrt{{\mathit{\mu }}_X^{\prime }{\mathit{\sum }}_{\mathit{XX}}^{-1}{\mathit{\sum }}_{\mathit{XX}}^{(1)}{\mathit{\sum }}_{\mathit{XX}}^{-1}{\mathit{\mu }}_{\mathit{X}}}}{\mathrm{\Phi }}^{-1}(t)\right)t\in \left[0,1\right] \end{gathered}$$ (3)

Adding a combination of q new variables Y = (Y₁, Y₂, ⋯, Y_q)′ to X will increase the AUC of the ROC curve. To determine the significance of the difference in the AUCs for the OLC of X versus OLC of (X′, Y′)′, we first give the following theorem that characterizes the amount of AUC increment by decomposing the quadratic form ${\mathit{\mu }}_{\left(\mathit{X}\prime ,\mathit{Y}\prime \right)\prime }^{\prime }{\mathit{\sum }}_{{\left({\mathit{X}}^{\prime },{\mathit{Y}}^{\prime }\right)}^{\prime }{\left({\mathit{X}}^{\prime },{\mathit{Y}}^{\prime }\right)}^{\prime }}^{-1}{\mathit{\mu }}_{{\left({\mathit{X}}^{\prime },{\mathit{Y}}^{\prime }\right)}^{\prime }}$ of the standardize mean shift of L(X,Y) in (2) in comparison to that of L(X).

Theorem 2.1

Let θ_L(X) and θ_L(X,Y) be the maximum AUC of the OLCs of X and (X′, Y′)′, denoted as L(X) and L(X, Y), respectively. Suppose (X′^(k), Y′^(k))′ ~ N (μ^(k), Σ^(k)) and the OLC residuals of Y on X, ${\mathit{Z}}^{\left(k\right)}={\mathit{Y}}^{\left(k\right)}-{\mathit{\sum }}_{\mathit{YX}}{\mathit{\sum }}_{\mathit{XX}}^{-1}{\mathit{X}}^{\left(k\right)}$. Then

$${\theta }_{L(\mathit{X},\mathit{Y})}={\theta }_{L(\mathit{X},\mathit{Z})}=\mathrm{\Phi }[{({\mathit{\mu }}_{\mathit{X}}^{\prime }{\mathit{\sum }}_{\mathit{XX}}^{-1}{\mathit{\mu }}_{\mathit{X}}+{\mathit{\mu }}_{\mathit{Z}}^{\prime }{\mathit{\sum }}_{\mathit{ZZ}}^{-1}{\mathit{\mu }}_{\mathit{Z}})}^{1/2}]$$ (4)

where ${\mathit{\mu }}_{\mathit{Z}}={\mathit{\mu }}_{\mathit{Z}}^{(1)}-{\mathit{\mu }}_{\mathit{Z}}^{(0)}={\mathit{\mu }}_{\mathit{Y}}-{\mathit{\sum }}_{\mathit{YX}}{\mathit{\sum }}_{\mathit{XX}}^{-1}{\mathit{\mu }}_{\mathit{X}}$ and ${\mathit{\sum }}_{\mathit{ZZ}}={\mathit{\sum }}_{\mathit{ZZ}}^{(1)}+{\mathit{\sum }}_{\mathit{ZZ}}^{(0)}={\mathit{\sum }}_{\mathit{YY}}-{\mathit{\sum }}_{\mathit{YX}}{\mathit{\sum }}_{\mathit{XX}}^{-1}{\mathit{\sum }}_{\mathit{XY}}$.

Therefore the incremental utility of adding Y to X in the AUC can be determined by

$$\lambda (\mathit{Y}\mid \mathit{X})={\left[{\mathrm{\Phi }}^{-1}\left({\theta }_{L(\mathit{X},\mathit{Y})}\right)\right]}^2-{\left[{\mathrm{\Phi }}^{-1}\left({\theta }_{L(\mathit{X})}\right)\right]}^2={\mathit{\mu }}_{{({\mathit{X}}^{\prime },{\mathit{Y}}^{\prime })}^{\prime }}^{\prime }{\mathit{\sum }}_{{({\mathit{X}}^{\prime },{\mathit{Y}}^{\prime })}^{\prime }{({\mathit{X}}^{\prime },{\mathit{Y}}^{\prime })}^{\prime }}^{-1}{\mathit{\mu }}_{{({\mathit{X}}^{\prime },{\mathit{Y}}^{\prime })}^{\prime }}-{\mathit{\mu }}_{\mathit{X}}^{\prime }{\mathit{\sum }}_{\mathit{XX}}^{-1}{\mathit{\mu }}_{\mathit{X}}={\mathit{\mu }}_{\mathit{Z}}^{\prime }{\mathit{\sum }}_{\mathit{ZZ}}^{-1}{\mathit{\mu }}_{\mathit{Z}}$$ (5)

in a quadratic form ${\mathit{\mu }}_{\mathit{Z}}^{\prime }{\mathit{\sum }}_{\mathit{ZZ}}^{-1}{\mathit{\mu }}_{\mathit{Z}}$ of the residuals $\mathit{Z}=\mathit{Y}-{\mathit{\sum }}_{\mathit{YX}}{\mathit{\sum }}_{\mathit{XX}}^{-1}\mathit{X}$, which is a monotone function of difference between squared roots of AUCs and represents the additionally standardized mean shift for OLC due to Y after removing all contributions of X. Note that the regression coefficient $\mathit{\beta }={\mathit{\sum }}_{\mathit{XX}}^{-1}{\mathit{\sum }}_{\mathit{XY}}$ is different from the ordinal least squared estimator of the regression coefficient of Y on X.

From (5), we have the following theorem.

Theorem 2.2

Suppose X,^(k), ${\mathit{Y}}_1^{\left(\mathit{k}\right)}$, and ${\mathit{Y}}_2^{\left(\mathit{k}\right)}$ be the p, q, and r-dimensional normally distributed diagnostic variables (k=0,1 for non-diseased and diseased subjects, respectively), then the incremental utility of (X, Y₁, Y₂) over X is additive, i.e.,

$$\lambda \left(({\mathit{Y}}_1,{\mathit{Y}}_2)\mid \mathit{X}\right)=\lambda \left({\mathit{Y}}_1\mid \mathit{X}\right)+\lambda \left({\mathit{Y}}_2\mid (\mathit{X},{\mathit{Y}}_1)\right)=\lambda \left({\mathit{Y}}_2\mid \mathit{X}\right)+\lambda \left({\mathit{Y}}_1\mid (\mathit{X},{\mathit{Y}}_2)\right)$$ (6)

Proof of Theorems is in Appendix A.

3 Estimation

Under the multivariate normality assumption, the incremental AUC of (X, Y) over X alone is uniquely determined by (5). Suppose that we have independent and identically-distributed (i.i.d) random samples ${\left({\mathit{X}}_{\mathit{i}}^{{(k)}^{\prime }},{\mathit{Y}}_{\mathit{i}}^{{(k)}^{\prime }}\right)}^{\prime }={\left(X_{1i}^{(k)},X_{2i}^{(k)},\mathrm{\dots },X_{\mathit{pi}}^{(k)},Y_{1i}^{(k)},Y_{2i}^{(k)},\mathrm{\dots },Y_{\mathit{qi}}^{(k)}\right)}^{\prime }$, i = 1,2,…, n_k from N (μ^(k), Σ^(k)), where the subscripts indicate vector components from the ith subject in the kth group. Denoting sample means and covariance matrices for each group k as, X̄^(k), Ȳ^(k), ${\mathit{S}}_{\mathit{XX}}^{(k)}$, ${\mathit{S}}_{\mathit{XY}}^{(k)}$, and ${\mathit{S}}_{\mathit{YY}}^{(k)}$, $\widehat{\mathit{\beta }}={\mathit{S}}_{\mathit{XX}}^{-1}{\mathit{S}}_{\mathit{XY}}$ is the consistent estimator of regression coefficient β, with ${\mathit{S}}_{\mathit{XX}}={\mathit{S}}_{\mathit{XX}}^{(0)}+{\mathit{S}}_{\mathit{XX}}^{(1)}$, ${\mathit{S}}_{\mathit{XY}}={\mathit{S}}_{\mathit{XY}}^{(0)}+{\mathit{S}}_{\mathit{XY}}^{(1)}$, and

$${\widehat{\mathit{Z}}}_i^{(k)}={\mathit{Y}}_i^{(k)}-{\widehat{\mathit{\beta }}}^{\prime }{\mathit{X}}_i^{(k)},i=1,2,\cdots ,n_k$$ (7)

be the residuals. We define the sample mean $\overline{\widehat{\mathit{Z}}}$ and covariance ${\mathit{S}}_{\mathit{ZZ}}^{(k)}$, of $\{{\widehat{\mathit{Z}}}_i^{(k)}\}$ as

$${\overline{\widehat{\mathit{Z}}}}^{(k)}={\overline{\mathit{Y}}}^{(k)}­{\mathit{S}}_{\mathit{XX}}^{­1}{\mathit{S}}_{\mathit{XY}}{\overline{\mathit{X}}}^{(k)},k=0,1,$$ (8)

$${\mathit{S}}_{\mathit{ZZ}}={\mathit{S}}_{\mathit{ZZ}}^{(0)}+{\mathit{S}}_{\mathit{ZZ}}^{(1)}={\mathit{S}}_{\mathit{YY}}-{\mathit{S}}_{\mathit{YX}}{\mathit{S}}_{\mathit{XX}}^{-1}{\mathit{S}}_{\mathit{XY}}$$ (9)

We estimate λ (5) by

$$\widehat{\mathrm{\lambda }}={\left({\overline{\widehat{\mathit{Z}}}}^{(1)}-{\overline{\widehat{\mathit{Z}}}}^{(0)}\right)}^{\prime }{\mathit{S}}_{\mathit{ZZ}}^{­1}\left({\overline{\widehat{\mathit{Z}}}}^{(1)}-{\overline{\widehat{\mathit{Z}}}}^{(0)}\right).$$ (10)

Conditioning on β̂, $\{{\widehat{\mathit{Z}}}_i^{(k)},i=1,2,..n_k\}$ can be regarded as i.i.d samples. Therefore the distribution of statistic λ̂ in (10) is related to a noncentral F- distribution, according to Reiser and Faraggi (Reiser and Faraggi, 1997). However, the common estimator β̂ introduces additional correlations among $\{{\widehat{\mathit{Z}}}_i^{(k)},i=1,2,..n_k\}$, and thus $\mathrm{var}({\overline{\widehat{\mathit{Z}}}}^{(k)})\ne {\mathit{\sum }}_{\mathit{ZZ}}^{(k)}/n_k$. In the Appendix B, we prove

$${\overline{\widehat{\mathit{Z}}}}^{(1)}-{\overline{\widehat{\mathit{Z}}}}^{(0)}\dot{~}N({\mathit{\mu }}_{\mathit{Z}},{\mathit{\sum }^{\sim }}_{\mathit{ZZ}}),$$ (11)

where ${\mathit{\sum }^{\sim }}_{\mathit{ZZ}}=\frac{{\mathit{\sum }}_{\mathit{ZZ}}^{(0)}}{n_0}+\frac{{\mathit{\sum }}_{\mathit{ZZ}}^{(1)}}{n_1}+{\mathit{V}}_1+{\mathit{V}}_2$, with

$${\mathit{V}}_1=\sum _{k=0}^1\frac{1}{n_k-1}\left[{\mathit{\mu }}_{\mathit{X}}^{\prime }{\mathit{\sum }}_{\mathit{XX}}^{-1}{\mathit{\sum }}_{\mathit{XX}}^{(k)}{\mathit{\sum }}_{\mathit{XX}}^{-1}{\mathit{\mu }}_{\mathit{X}}\right]\otimes \mathrm{Var}({\mathit{e}}^{(k)}),$$ (12)

$$\begin{gathered} {\mathit{V}}_2=\left[{\mathit{\mu }}_{\mathit{X}}^{\prime }\otimes {\left({\mathit{\beta }}^{(1)}-{\mathit{\beta }}^{(0)}\right)}^{\prime }\right]\mathrm{Var}\left[\mathrm{vec} \\ \left\{{\mathit{S}}_{\mathit{XX}}^{(0)}{\mathit{S}}_{\mathit{XX}}^{-1}\right\}\right]\left[{\mathit{\mu }}_{\mathit{X}}\otimes \left({\mathit{\beta }}^{(1)}-{\mathit{\beta }}^{(0)}\right)\right] \end{gathered}$$ (13)

and

$$\begin{array}{l}\mathrm{Var}\left[\mathrm{vec}\left({\mathit{S}}_{\mathit{XX}}^{(0)}{\mathit{S}}_{\mathit{XX}}^{-1}\right)\right]\\ \approx \sum _{k=0}^1\frac{n_k}{{(n_k-1)}^2}\left[{\mathit{\sum }}_{\mathit{XX}}^{-1}\otimes \left({\mathit{\sum }}_{\mathit{XX}}^{(1-k)}{\mathit{\sum }}_{\mathit{XX}}^{-1}\right)\right]\left[\left({\mathit{I}}_{p^2}+{\mathit{K}}_{\mathit{pp}}\right)\left({\mathit{\sum }}_{\mathit{XX}}^{(k)}\otimes {\mathit{\sum }}_{\mathit{XX}}^{(k)}\right)\right]\left[{\mathit{\sum }}_{\mathit{XX}}^{-1}\otimes \left({\mathit{\sum }}_{\mathit{XX}}^{-1}{\mathit{\sum }}_{\mathit{XX}}^{(1-k)}\right)\right],\end{array}$$ (14)

where vec means vector operator, ⊗ is the Kronecker product, I_p² is a p² × p² identity matrix, K_pp is a p² × p² commutation matrix. We can estimate V₁, V₂ by substituting sample means, sample variances, mean squared residual, and β̂^(k) for μ’s, Σ’s, var(e) and β^(k) in (13) and (14), and denote them by V̂₁, V̂₂. Let

$${\widehat{\mathit{\sum }^{\sim }}}_{\mathit{ZZ}}=\frac{{\widehat{\mathit{\sum }}}_{\mathit{ZZ}}^{(0)}}{n_0}+\frac{{\widehat{\mathit{\sum }}}_{\mathit{ZZ}}^{(1)}}{n_1}+{\widehat{\mathit{V}}}_1+{\widehat{\mathit{V}}}_2,$$ (15)

where ${\widehat{\mathit{\sum }}}_{\mathit{ZZ}}^{(k)}={\mathit{MSE}}^{(k)}+{\left({\widehat{\mathit{\beta }}}^{(k)}-\widehat{\mathit{\beta }}\right)}^{\prime }{\mathit{S}}_{\mathit{XX}}^{(k)}\left({\widehat{\mathit{\beta }}}^{(k)}-\widehat{\mathit{\beta }}\right)$, and MSE^(k) is the mean squared error of observed residuals. In addition, from the Appendix C, we can derive the distribution of λ̂ as

$$\frac{\widehat{f}-p-q+1}{(\widehat{f}-p)\widehat{\upsilon }}\widehat{c}\widehat{\lambda }\dot{~}F\left(\widehat{\upsilon },\widehat{f}-p-q+1,\frac{\widehat{f}}{\widehat{f}-p}\widehat{c}\lambda \right),$$ (16)

where

$$\begin{array}{l}\widehat{f}=\frac{\left[\mathrm{tr}\left\{{\left({\mathit{S}}^{(0)}+{\mathit{S}}^{(1)}\right)}^2\right\}+{\left\{\mathrm{tr}\left({\mathit{S}}^{(0)}+{\mathit{S}}^{(1)})\right)\right\}}^2\right]}{\left[\mathrm{tr}\left\{{\left({\mathit{S}}^{(0)}\right)}^2\right\}+{\left\{\mathrm{tr}\left({\mathit{S}}^{(0)}\right)\right\}}^2\right]/(n_0-1)+\left[\mathrm{tr}\left\{{\left({\mathit{S}}^{(1)}\right)}^2\right\}+{\left\{\mathrm{tr}\left({\mathit{S}}^{(1)}\right)\right\}}^2\right]/(n_1-1)}\end{array},$$

$$\widehat{c}=\frac{\mathrm{tr}({\mathit{S}}_{\mathit{ZZ}}^{-1}{\widehat{\mathit{\sum }^{\sim }}}_{\mathit{ZZ}})+2{\left({\overline{\widehat{\mathit{Z}}}}^{(1)}­{\overline{\widehat{\mathit{Z}}}}^{(0)}\right)}^{\prime }{\mathit{S}}_{\mathit{ZZ}}^{­1}\left({\overline{\widehat{\mathit{Z}}}}^{(1)}­{\overline{\widehat{\mathit{Z}}}}^{(0)}\right)}{\mathrm{tr}[{({\mathit{S}}_{\mathit{ZZ}}^{-1}{\widehat{\mathit{\sum }^{\sim }}}_{\mathit{ZZ}})}^2]+2{\left({\overline{\widehat{\mathit{Z}}}}^{(1)}­{\overline{\widehat{\mathit{Z}}}}^{(0)}\right)}^{\prime }{\mathit{S}}_{\mathit{ZZ}}^{­1}{\widehat{\mathit{\sum }^{\sim }}}_{\mathit{ZZ}}{\mathit{S}}_{\mathit{ZZ}}^{-1}\left({\overline{\widehat{\mathit{Z}}}}^{(1)}­{\overline{\widehat{\mathit{Z}}}}^{(0)}\right)}.$$

$$\widehat{\upsilon }=\widehat{c}·\mathrm{tr}({\mathit{S}}_{\mathit{ZZ}}^{-1}{\widehat{\mathit{\sum }^{\sim }}}_{\mathit{ZZ}})$$

It’s easy to get υ̂ = 1 when only adding one variable Y is added over X.

In the special case when λ=0, i.e, there is no improvement from Y beyond X, (f̂ − p − q + 1)/[(f̂ − p) υ̂]ĉλ̂ follows central F distribution with degrees of freedom υ̂ and f̂ − p − q + 1.

4 Hypothesis Test

Statistical inference of whether adding new variables (diagnostic test 1) is superior to existing variables (test 2) is based on testing a one-sided hypothesis about the difference in their AUCs, i.e., H₀ : θ₁ − θ₂ ≤ Δ versus H₁ : θ₁ − θ₂ > Δ, where Δ ≥ 0 is a prespecified superiority margin and θ_i’s are the corresponding AUCs. A conventional test statistic is

$$\mathit{st}=\frac{{\widehat{\theta }}_1-{\widehat{\theta }}_2-\mathrm{\Delta }}{\sqrt{\mathrm{var}({\widehat{\theta }}_1-{\widehat{\theta }}_2)}},$$ (17)

which asymptotically follows a standard normal distribution. The key of this conventional approach is to derive the unbiased estimates of AUCs θ̂₁ and θ̂₂ respectively for the best combinations of vectors X and (X′, Y′)′. A naïve approach to use test statistic (17) is to use the nonparametric formula by DeLong, DeLong and Clarke-Pearson (DeLong, DeLong and Clarke-Pearson, 1988) to estimate difference in AUCs and its variance based on the two linear discriminant function scores for combinations of markers. When the diagnostic procedures follow bivariate normal distributions, there is also a parametric approach based on the parameters from normal distributions (Metz, 1986). Both approaches ignore uncertainties introduced in estimating linear discriminant functions, and the overly optimistic estimates of AUCs because of the same data being used for fitting and evaluating models. Although Janes, Longton and Pepe (Janes, Longton and Pepe, 2009) proposed use of cross-validation (CV) to correct for the over optimism, they used naïve estimates from Stata in their examples to compare the ROC curves for the two linear prediction models for all subjects that still utilized same data for fitting and testing.

Alternative to direct comparisons of AUCs, Liu et al (Liu, et al, 2006) proposed inverse normal transformation of AUC and evaluate difference Φ⁻¹ (θ̂₁) − Φ⁻¹ (θ̂₂) to assess the equivalence or non-inferiority of two diagnostic variables. The difference reflects the standardized mean shift between two ROC models. Liu and colleagues demonstrated improvement in statistical consistency and efficiency by using the standardized difference in comparison to AUCs. Our λ in (5) is a natural generalization from their standardized difference to multivariate linear combinations of L(X) and L(X,Y). The hypotheses to be tested are

$$H_0:\lambda \le \delta \mathrm{versus}{H}_1:\lambda >\delta$$ (18)

where the margin δ can be determined from AUC difference Δ by the relationship of

$$\delta ={[{\mathrm{\Phi }}^{-1}({\theta }_{L(X)}+\mathrm{\Delta })]}^2-{[{\mathrm{\Phi }}^{-1}({\theta }_{L(X)})]}^2.$$

Using λ̂ defined in (10) and distribution property of (16), we reject the null hypothesis for a too large (f̂ − p − q + 1)/[(f̂ − p) υ̂]ĉλ̂. That is to reject null H₀ : λ ≤ δ at significance level α if

$$\begin{gathered} (\widehat{f}-p-q+1)/[(\widehat{f}-p)\widehat{\upsilon }]\widehat{c}\widehat{\lambda }>F_{1-\alpha }^{\left(-1\right)} \\ \left(\widehat{\upsilon },\widehat{f}-p-q+1,\widehat{f}/(\widehat{f}-p)\widehat{c}\delta \right), \end{gathered}$$ (19)

where $F_{\alpha }^{\left(-1\right)}({\mathit{df}}_1,{\mathit{df}}_2,\mathit{ncp})$ denotes the 100α quintile of the F distribution F(.; df₁, df₂, ncp) and the p-value is 1 − F ((f̂ − p − q + 1)/((f̂ − p)υ̂)ĉλ̂; υ̂, f̂ − p − q + 1, f̂/(f̂ − p)ĉδ). The statistical power is

$$\begin{gathered} \begin{array}{l}P\left\{\frac{\widehat{f}-p-q+1}{(\widehat{f}-p)\widehat{\upsilon }}\widehat{c}\widehat{\lambda }>F_{1-\alpha }^{\left(-1\right)}\left(\widehat{\upsilon },\widehat{f}-p-q+1,\frac{\widehat{f}}{\widehat{f}-p}\widehat{c}\delta \right)\mid \lambda >\delta )\right\}\\ =1-F\left\{F_{1-\alpha }^{\left(-1\right)}\left(\widehat{\upsilon },\widehat{f}-p-q+1,\frac{\widehat{f}}{\widehat{f}-p}\widehat{c}\delta \right); \\ \widehat{\upsilon },\widehat{f}-p-q+1,\frac{\widehat{f}}{\widehat{f}-p}\widehat{c}\lambda \mid \lambda >\delta \right\}\end{array} \end{gathered}$$ (20)

It is easy to see that the power depends on the degrees of freedom or sample size, superiority margin δ, and the true amount of improvement λ. Thus, for a given type I error rate α, expected power 1-β, λ under the alternative hypothesis δ, and covariance matrices, we can get the degrees of freedom f̂ and corresponding required sample size.

When superiority margin δ → 0, the superiority test is simplified as the conventional hypothesis test for H₀ : λ = 0 versus H₁ : λ >0. Test statistics (f̂ − p − q + 1)/[(f̂ − p)υ̂]ĉλ̂ is central F(.; υ̂, f̂ − p − q + 1, 0) distributed under H₀ : λ = 0. Therefore combining Y with X presents an improvement beyound X alone when $(\widehat{f}-p-q+1)/[(\widehat{f}-p)\widehat{\upsilon }]\widehat{c}\widehat{\lambda }>F_{1-\alpha }^{\left(-1\right)}(\widehat{\upsilon },\widehat{f}-p-q+1,0)$.

5 An Example of Application

Between September 1986 and October 1988, 9,704 ambulatory white women aged 65 years or older without bilateral hip replacements were recruited from population-based listings in four United States cities in the Study of Osteoporotic Fractures (SOF, http://sof.ucsf.edu/Interface/). At the second visit (1988-89), 7,784 women (82% of the survivors at that time) had bone mineral density (BMD) measurements of the posterior-anterior (PA) spine (SBMD) and proximal femoral neck (NBMD), trochanteric (TRBMD), intertrochanteric (INTRBMD), and total hip (TBMD) using dual x-ray absorptiometry (DXA) scanners, and at the calcaneus, distal radius (DBMD), and proximal radius (PBMD) using single photon absorptiometry (SPA) scanners. Besides BMD, additional information, including age, weight, height, and body mass index (BMI), was also collected at all visits. The women were followed for hip fracture by letter or telephone every 4 months for an average of 4.1 years with 99 percent completion. Reported hip fractures were confirmed by a radiologist from preoperative radiographs. Details of the study design and summary statistics of SOF have been published previously (Cummings et al., 1993; Cummings et al., 1995) and available online.

To apply our method to the SOF data, we examined whether TBMD measured by a hip DXA scan or PBMD measured by a SXA forearm scan or their combination can provide incremental diagnostic information for 10-year hip fracture risk beyond BMI and height. The case group consists of women who developed hip fracture during 10 years follow-up. The control group consists of women who had been followed for more than 10 years without hip fracture. To illustrate our method, we select a random subset of 1275 women with a 1:2 age-matched cases and controls. Table 1 summarizes our data samples.

Table 1.

Summary statistics of BMI, height, TBMD and PBMD (Mean ± SD)

	Controls (n0=850)	Fractured Subjects (n1=425)
BMI	26.269 ± 4.232	25.431 ± 4.345
Height	158.612 ± 5.765	158.604 ± 6.491
TBMD	0.755 ± 0.127	0.668 ± 0.106
PBMD	0.629 ± 0.103	0.602 ± 0.103

To improve the normality fit, we made a natural log transformation of BMI (lnBMI). Quantile-quantile plots for TBMD, PBMD, lnBMI, and height fit normal distributions well. Based on (Su and Liu, 1993), the OLC is the linear discriminant functions (LDF). For X = (lnBMI, Height), LDF is −2.139 × lnBMI −0.0008 × height, with AUC=0.560. For (X,Y) = (lnBMI, height, TBMD), LDF is 1.580×lnBMI+0.016×height −3.600×TBMD, with AUC=0.706. For (X,Y)=(lnBMI, height, PBMD) LDF is −1.519×lnBMI −0.004×height −1.105×PBMD, with AUC=0.584. The ROC curves for the OLCs are shown in Figure 1.

Fig. 1.

ROC curves for OLCs for (1) lnBMI, height (dotted line); (2) lnBMI, height, TBMD (solid line); (3) lnBMI, height, PBMD (dashed line).

Suppose the superiority margin Δ=0.05 for AUC difference, the corresponding margin δ for λ is

$$\delta ={[{\mathrm{\Phi }}^{-1}(0.560+\mathrm{\Delta })]}^2-{[{\mathrm{\Phi }}^{-1}(0.560)]}^2=0.055.$$

Table 2 gives the estimates λ̂, ĉ, f̂ for adding TBMD or PBMD. The degree of freedom from numerator may be different from the number of added diagnostic variables, although it is always 1 from adding only 1 variable. For the given margin δ=0.055, the one-sided p-values are <0.0001 for λ̂ (TBMD∣lnBMI, height), 0.978 for λ̂ (PBMD∣lnBMI, height), and <0.0001 for λ̂ (TBMD,PBMD∣lnBMI, height) based on equation (16). So we reject H₀ : λ ≤ δ for adding TBMD but accept H₀ for PBMD. Therefore, we conclude that at the 0.05 significance level (1) the addition of TBMD by hip DXA scan significantly increase diagnostic accuracy from using only lnBMI and height; (2) the addition of PBMD by SXA scan doesn’t significantly improve diagnostic accuracy from using only lnBMI and height. Furthermore, by testing λ̂(TBMD∣PBMD,lnBMD,height) directly, we compared λ̂(TBMD,PBMD∣lnBMI,height) =0.2882 versus λ̂(PBMD ∣ lnBMI,height) according to Theorem 2.2 and showed significant benefit of including TBMD on top of lnBMI, height, and PBMD.

Table 2.

The estimations λ̂, ĉ, f̂, and p values for the SOF Example for X=(lnBMI,height)

	TBMD∣X	PBMD∣X	(TBMD,PBMD)∣X	TBMD∣(PBMD, X)
λ̂	0.2710	0.0224	0.2882	0.2658
ĉ	597.110	560.732	594.803	578.422
υ̂	1	1	2.076	1
f̂	1035.129	1035.096	1035.144	1035.144
P(Δ=0.05)a	0.0000	0.9780	0.0000	0.0000
P(Δ=0.10)	0.0007	1.0000	0.0002	0.0012

^aSuperiority margin Δ=0.05 (δ=0.055) or Δ=0.10 (δ=0.147) for superiority test

Using formula (20), we can calculate sample size needed in planning a study. For example, to use a superiority margin δ=0.147, α=0.05, and λ=0.271, we will need 248 cases and 496 controls (in 1:2 ratio) or 347 cases and 347 controls (in 1:1 ratio) to assure 80% statistical power of the study.

6 Simulation Studies

In simulation studies under Gaussian assumption, we specify mean vectors and covariance matrices based on the values in the SOF sample. Let X = (lnBMI, height)′; k = 1 for hip fracture within 10 years and k = 0 otherwise; and (Y₁, Y₂)′ =(TBMD, PBMD)′. Accordingly, the means and covariance matrices for (lnBMI, height, TBMD, PBMD)′ are μ⁽⁰⁾ =(1.179, 158.612, 0.755, 0.629)′, μ⁽¹⁾ =(1.169, 158.604, 0.668, 0.602)′, and

$${\mathit{C}}^{(0)}=\left(\begin{array}{cccc}0\mathrm{.002}& -\mathrm{0.029}& \mathrm{0.003}& \mathrm{0.001}\\ -\mathrm{0.029}& \mathrm{33.236}& \mathrm{0.157}& \mathrm{0.141}\\ \mathrm{0.003}& \mathrm{0.157}& \mathrm{0.016}& \mathrm{0.008}\\ \mathrm{0.001}& \mathrm{0.141}& \mathrm{0.008}& \mathrm{0.011}\end{array}\right),{\mathit{C}}^{(1)}=\left(\begin{array}{cccc}\mathrm{0.003}& \mathrm{0.003}& \mathrm{0.003}& \mathrm{0.002}\\ \mathrm{0.003}& \mathrm{42.135}& \mathrm{0.160}& \mathrm{0.164}\\ \mathrm{0.003}& \mathrm{0.160}& \mathrm{0.011}& \mathrm{0.006}\\ \mathrm{0.002}& \mathrm{0.164}& \mathrm{0.006}& \mathrm{0.011}\end{array}\right)$$

for controls and cases, respectively. In addition, we add a Y₃ = lnBMI+ε, where ε is Gaussian noise N(0, 4 Var(l nBMI)). So Y₃ is conditional independent with the fracture outcome for given X. Therefore there is no benefit adding Y₃ over X. Let Y = (Y₁, Y₂, Y₃)′. Because the equality of population covariance matrices affects the distribution of the estimator, we simulate two scenarios:

• Experiment 6.1: Σ⁽⁰⁾ = C⁽⁰⁾, Σ⁽¹⁾ = C⁽¹⁾ and ${\mathit{\sum }}_{\mathit{ZZ}}^{(0)}\ne {\mathit{\sum }}_{\mathit{ZZ}}^{(1)}$ and

• Experiment 6.2: Σ⁽⁰⁾ = Σ⁽¹⁾ = (C⁽⁰⁾ + C⁽¹⁾)/2 and ${\mathit{\sum }}_{\mathit{ZZ}}^{(0)}={\mathit{\sum }}_{\mathit{ZZ}}^{(1)}$.

Simulations are run by generating observations from a multivariate normal distribution. For each scenario, we reduplicated 10,000 times. Each replication generates n=n₀ + n₁ observations of cases (n₁) and controls (n₀). The proportion of replications rejecting H₀ is used to estimate the probability of type I error or power when α=0.05 and superiority margin δ are specified. When we specify δ=0.1λ(Y₁∣X₁, X₂) and n₀ : n₁ = 1 in Experiment 6.1, to achieve 80% power, n=104 subjects are needed to test the additive utility Y₁ over X₁, X₂, for our proposed method. Thus we choose n=120 as the largest sample size in simulation studies.

Firstly, we evaluate possible estimation bias of effect size for our quadratic index and estimation of AUC difference using non-parametric AUC estimators (DeLong et al., 1988). One of non-parametric estimations is derived from naïve procedure where we fit OLCs and then assess the non-parametric AUCs from the the predicted values for all subjects in the same dataset. Another non-parametric estimation is derived via cross-validation (CV) technique (Janes, Longton and Pepe, 2009), where different datasets are used in OLCs constructing and OLCs predicting. In a K-fold cross-validation, the original subset is partitioned into K subsets with almost equal sample size. Leave out one subset by turn, we construct OLCs from the remaining K − 1 subsets and applied the fitted linear functions OLCs to the leave-out subset and then get the predicted values for the subjects in the leave-out subset. Repeat the process K times, we’ll get the predicted values for all subjects, and then derive the corresponding non-parametric AUC estimations. Tables 3 reports the bias of the estimations of the three approaches for utility of (Y₁,Y₂) over (X₁,X₂) when (n₀, n₁) = (60,60) under Experiment 6.1. Here, for CV procedure, we applied 60-fold CV to the dataset with one case and one control in each subset. The bias is defined as the difference between sample mean estimation versus the theoretical mean value in 10,000 runs. According to F distribution, the bias for λ̂ is calculated by

$$\overline{\widehat{\lambda }}-\mathrm{E}(\widehat{\lambda })=\overline{\widehat{\lambda }}-\frac{(\widehat{f}-p)\widehat{\upsilon }+\widehat{f}\widehat{c}\lambda }{\widehat{c}(\widehat{f}-p-q-1)}.$$

The bias in AUC difference was asympotically normal and estimated by $\overline{\widehat{\theta }}-\theta$. The relative bias is defined as bias relative to the corresponding true value.

Table 3.

Bias and standard error (SE) of estimations for utility of (Y₁,Y₂) over (X₁,X₂) in Experiment 6.1 when (n₀, n₁) = (60,60) in 10,000 Runs

	Naive AUC Bias±SE (relative bias)	CV-based AUC Bias±SE (relative bias)	Quadratic-form-based λ Bias±SE (relative bias)
(X1,X2) (θ=0.5599)	0.0255±0.000427 (4.55%)*	-0.0312±0.000512 (-5.58%)
(X1,X2,Y1,Y2) (θ=0.7114)	0.0180±0.000442 (2.53%)	-0.0646±0.000563 (-9.09%)
(Y1,Y2) over (X1,X2) (Δθ=0.1516, λ=0.2882)	-0.0075±0.000515 (-4.93%)	-0.0334±0.000564 (-22.05%)	0.0007±0.001292 (0.25%)

^*Relative bias is bias divided by the true value expressed in percentage.

We find the non-parametric estimations of the AUCs from both naïve procedure and CV procedure to be biased. The naïve procedure over-estimates, whereas 60-fold CV procedure under-estimates the AUCs of the OLCs, i.e. 60-fold CV procedure over corrects the overoptimism from naïve procedure. Both under-estimates the additive performance of (Y₁,Y₂) over (X₁,X₂). For example, when n₀=n₁=60 in Experiment 6.1, the mean AUC difference by naïve estimation from 10,000 runs is 0.1441, whereas the true difference is 0.1516. The agreement in λ is substantially better. Figure 2 shows histograms for the λ-based and naïve non-parametric θ-based test statistics for direct comparison of AUC difference (17) (DeLong et al., 1988) from 10,000 runs to test the additive utility of (Y₁,Y₂) over (X₁,X₂) when n₀=n₁=60 in Experiment 6.1. Even after bias correction, the empirical type I error rate is still about half of 0.05. Our simulations show the naïve non-parametric statistic is positive skewed from a standard normal distribution when sample size is not big enough. Because of the bias or departure from null distribution, the AUC-based method from naïve procedure or CV procedure will not maintain proper the I error rate, and will not get correct power either. So next, we only present empirical type I error rate and power for the newly proposed method.

Fig. 2.

Histograms of λ-based test statistic and naïve non-parametric θ-based test statistic from 10,000 runs to test additive utility of (Y₁,Y₂) over (X₁,X₂) when n₀=n₁=60 in Experiment 6.1.

(1) Histograms of quadratic-form-based test statistic.

Noncentral distribution under null hypothesis (solid curve); 95% quantile of the null distribution(dashed line).

(2) Histograms of classic non-parametric AUC-based test statistic.

Standard normal distribution under null hypothesis (solid curve); 5% and 95% quantiles of standard normal distribution (dashed line); Bias corrected normal distribution under null hypothesis (dotted curve); 5% and 95% quantiles of bias corrected normal distribution (dotted line).*

Tables 4 and 5 report the empirical type I error rate and power to test the incremental utilities of adding Y₁, Y₂, Y₃, (Y₁, Y₂)′, (Y₁, Y₂, Y₃)′, (Y₁, Y₂, Y₃)′ and (Y₁, Y₂, Y₃)′ over X = (X₁, X₂)′ under Gaussian assumption with experimental conditions 6.1 and 6.2 with sample sizes (n₀, n₁) = (30,30), (40,20), (60,60), and (80,40). They demonstrate accurate type I error rates as well as practically acceptable statistical power of the newly proposed method for assessing the added diagnostic utility. The simulations show when the variable Y₃ with no benefit over X is added to Y₁ or Y₂, the power decreased as expected.

Table 4.

Empirical type I error rates and power for Experiment 6.1 under Gaussian distribution (10,000 Runs)

Margin δ	Y3∣X λ=0	Y1∣X λ=0.271	(Y1, Y3)∣X λ=0.271	Y2 ∣X λ=0.022	(Y2,Y3)∣X λ=0.022	(Y1,Y2)∣X λ=0.288	(Y1,Y2,Y3)∣X λ=0.288
(n0,n1)=(30,30)
δ=λ	0.0448	0.0436	0.0460	0.0430	0.0453	0.0437	0.0458
δ=0.5λ	---	0.1769	0.1726	0.0683	0.0621	0.1753	0.1684
δ=0.1λ	---	0.5638	0.5153	0.1023	0.0845	0.5222	0.4873
(n0,n1)=(40,20)
δ=λ	0.0481	0.0430	0.0433	0.0439	0.0445	0.0443	0.0415
δ=0.5λ	---	0.1734	0.1538	0.0662	0.0595	0.1647	0.1495
δ=0.1λ	---	0.5362	0.4579	0.0958	0.0769	0.4817	0.4245
(n0,n1)=(60,60)
δ=λ	0.0473	0.0493	0.0472	0.0483	0.0492	0.0493	0.0475
δ=0.5λ	---	0.2897	0.2846	0.0918	0.0853	0.2966	0.2882
δ=0.1λ	---	0.8446	0.8145	0.1695	0.1296	0.8307	0.8069
(n0,n1)=(80,40)
δ=λ	0.0497	0.0442	0.0472	0.0479	0.0510	0.0458	0.0460
δ=0.5λ	---	0.2765	0.2701	0.0836	0.0834	0.2751	0.2710
δ=0.1λ	---	0.8182	0.7798	0.1533	0.1241	0.8034	0.7665

Table 5.

Empirical type I error rates and power for Experiment 6.2 under Gaussian distribution (10,000 Runs)

Margin δ	Y3∣X λ=0	Y1∣X λ=0.271	(Y1,Y3)∣X λ=0.271	Y2 ∣X λ=0.022	(Y2,Y3)∣X λ=0.022	(Y1,Y2)∣X λ=0.288	(Y1,Y2,Y3)∣X λ=0.288
(n0,n1)=(30,30)
δ=λ	0.0461	0.0460	0.0456	0.0445	0.0463	0.0457	0.0447
δ=0.5λ	---	0.1817	0.1719	0.0701	0.0625	0.1763	0.1694
δ=0.1λ	---	0.5732	0.5044	0.1091	0.0826	0.5291	0.4780
(n0,n1)=(40,20)
δ=λ	0.0444	0.0485	0.0434	0.0470	0.0431	0.0461	0.0456
δ=0.5λ	---	0.1704	0.1584	0.0702	0.0615	0.1656	0.1535
δ=0.1λ	---	0.5110	0.4591	0.0999	0.0777	0.4709	0.4277
(n0,n1)=(60,60)
δ=λ	0.0474	0.0479	0.0425	0.0511	0.0469	0.0466	0.0445
δ=0.5λ	---	0.2893	0.2772	0.0935	0.0806	0.2966	0.2780
δ=0.1λ	---	0.8367	0.8113	0.1746	0.1312	0.8299	0.8100
(n0,n1)=(80,40)
δ=λ	0.0497	0.0444	0.0493	0.0488	0.0488	0.0454	0.0489
δ=0.5λ	---	0.2643	0.2642	0.0856	0.0779	0.2626	0.2650
δ=0.1λ	---	0.7997	0.7555	0.1561	0.1267	0.7845	0.7459

7 Discussion

The AUC under an ROC curve measures diagnostic accuracy for discriminating diseased from nondiseased subjects. In a binormal ROC model, the difference in AUC can be more accurately and efficiently compared via the mean shift relative to the variance (Liu et al., 2006). The incremental diagnostic accuracy by adding new variables has to be evaluated based on the increase in the AUC of ROC curves (Pepe et al., 2004; Wang et al., 2006; Ware, 2006). Parametric approaches used the binormal distribution models (Metz, 1986) or non-parametric estimation by DeLong et al (DeLong et al., 1988) can evaluate differences in AUCs from randomized clinical trials if one group diagnosed with X only and another group uses the combination of X and Y. However, in a paired design commonly used in diagnostic trials, we need to use the data first to fit OLCs based on the linear discriminant functions of X and the combination of X and Y. Theorem 2.1 proves that the gain in AUCs is a function of regression residuals of Y on X, and the regression coefficient estimator β̂ introduces non-linear correlated in AUC differences based on OLCs. If we use the predicted OLC values directly in evaluation of AUC difference, we will result in biased estimation of AUC gain and variance in practical sample sizes. The resulting statistical tests will be biased. Fitting and evaluating models on the same data is known to produce overly optimistic estimates of model performance. CV procedure will correct the overoptimism but also have a risk of underestimation the true difference under practical sample sizes. Our simulations show 60-fold CV procedure over corrects the overoptimism for the specific simulation data. Our proposed index is a function of transformed AUC differences and has desired statistical consistency and efficiency demonstrated in our simulation studies. Pencina et al. (Pencina et al., 2008) introduced two alternative measures to assess the clinical benefits of adding new diagnostic variables, one based on integrated sensitivity and specificity, and the other on event-specific reclassification tables. Further research is needed to evaluate these metrics.

We formulated the incremental diagnostic utility of adding new variables Y = (Y₁, Y₂, ⋯, Y_q)′ to the existing diagnostic variables X = (X₁, X₂, ⋯, X_p)′ under a multivariate normal distributional assumption by quadratic form $\lambda ={({\mathit{\mu }}_{\mathit{Z}}^{(1)}-{\mathit{\mu }}_{\mathit{Z}}^{(0)})}^{\prime }{({\mathit{\sum }}_{\mathit{ZZ}}^{(0)}+{\mathit{\sum }}_{\mathit{ZZ}}^{(1)})}^{-1}{({\mathit{\mu }}_{\mathit{Z}}^{(1)}-{\mathit{\mu }}_{\mathit{Z}}^{(0)})}^{\prime }$ with $\mathit{Z}=\mathit{Y}-{\mathit{\sum }}_{\mathit{YX}}{\mathit{\sum }}_{\mathit{XX}}^{-1}\mathit{X}$, where λ reflects a quadratic form of mean shift relative to the variance from Z. We account the uncertainties introduced in estimating linear discriminant functions into the distribution of Z. The inference based on λ̂ estimates fewer parameters than non-parametric approaches to compare AUCs of L(X), L(X,Y), which require estimates of variance and covariance of the AUCs. In addition, it has no ceiling (≤1) nor floor (≥0.5) effect in this index. As with the one-dimensional case (Liu et al., 2006), the λ-based method maintains the correct type I error rate and is more efficient than AUC-based methods.

The distribution estimator λ̂ is related to the multivariate Behrens–Fisher problem. In deriving the distribution, we did not limit ourselves to special conditions, such as ${\mathit{\sum }}_{\mathit{ZZ}}^{(0)}={\mathit{\sum }}_{\mathit{ZZ}}^{(1)}$. For special conditions of homogenous covariance matrixes $({\mathit{\sum }}_{\mathit{ZZ}}^{(0)}={\mathit{\sum }}_{\mathit{ZZ}}^{(1)})$, there are more efficient estimators, such as the conventional Hotelling T² test. Our simulations cover both conditions of with and without homogeneous covariance, and the results show that the test statistics (19) maintain the proper type I error rate under both conditions. It can also be used to evaluate sample size needed for such studies.

One major limitation of current approach is the assumption of multivariate normal distributions of diagnostic variables. Because the most important derivations relate to the residual Z, it should be possible to relax the multivariate normal assumptions by requiring multivariate normal distribution assumption only to the residual Z. Further studies are necessary to confirm this conjecture. If we fail to make proper variable transformation to fit a multivariate normal distribution, we can always use the bootstrap method to derive the empirical p-value. The bootstrap method, however, will not help the sample size calculation when we plan a study.

In summary, we derived a new test statistic to efficiently determine the statistical significance of an incremental AUC under a multivariate normal assumption. Our test links the difference in AUC to a quadratic form of standardized mean shift relative to the covariance after a properly linear transformation of all diagnostic variables. The distribution of the estimator of the quadratic form is related to the multivariate Behrens-Fisher problem. We provide an explicit mathematical solution of the estimator and its approximate noncentral F-distribution, type I error rate, and sample size formula. Our simulation studies demonstrate that our new test maintains the prespecified type I error rates as well as satisfactory statistical power under practical sample sizes. In addition, our simulation study suggests that the proposed estimator and test statistics maintain the correct type I. We use an example of predicting osteoporotic hip fracture to illustrate our method.

Appendix A

Proof of Theorem 2.1

We give two lemmas first.

Lemma 1

If X̃ is a nonsingular linear transformation of X, the maximum AUC for the linear combination of X̃ is the same as that of X, i.e. θ_L(X̃) = θ_L(X).

The lemma is obvious because X̃ and X are linear combinations of each other. Let ${\mathit{\mu }}^{(k)}=\left(\begin{array}{c}{\mathit{\mu }}_{\mathit{X}}^{(k)}\\ {\mathit{\mu }}_{\mathit{Y}}^{(k)}\end{array}\right),{\mathit{\sum }}^{(k)}=\left(\begin{array}{cc}{\mathit{\sum }}_{\mathit{XX}}^{(k)}& {\mathit{\sum }}_{\mathit{XY}}^{(k)}\\ {\mathit{\sum }}_{\mathit{YX}}^{(k)}& {\mathit{\sum }}_{\mathit{YY}}^{(k)}\end{array}\right)$, and μ = μ⁽¹⁾ − μ⁽⁰⁾, Σ = Σ⁽¹⁾ + Σ⁽⁰⁾.

Lemma 2

If (X′^(k), Y′^(k))′ : N(μ^(k), Σ^(k)), ${\mathit{Z}}^{(k)}={\mathit{Y}}^{(k)}-{\mathit{\sum }}_{\mathit{YX}}{\mathit{\sum }}_{\mathit{XX}}^{-1}{\mathit{X}}^{(k)}$ with ${\mathit{\sum }}_{\mathit{YX}}={\mathit{\sum }}_{\mathit{YX}}^{(1)}+{\mathit{\sum }}_{\mathit{YX}}^{(0)}$ and ${\mathit{\sum }}_{\mathit{XX}}={\mathit{\sum }}_{\mathit{XX}}^{(1)}+{\mathit{\sum }}_{\mathit{XX}}^{(0)}$, then

$$\left(\begin{array}{c}{\mathit{X}}^{(k)}\\ {\mathit{Z}}^{(k)}\end{array}\right)~N\left(\left(\begin{array}{c}{\mathit{\mu }}_{\mathit{X}}^{(k)}\\ {\mathit{\mu }}_{\mathit{Z}}^{(k)}\end{array}\right),\left(\begin{array}{cc}{\mathit{\sum }}_{\mathit{XX}}^{(k)}& {\mathit{\sum }}_{\mathit{XZ}}^{(k)}\\ {\mathit{\sum }}_{\mathit{ZX}}^{(k)}& {\mathit{\sum }}_{\mathit{ZZ}}^{(k)}\end{array}\right)\right)$$

with ${\mathit{\sum }}_{\mathit{XZ}}={\mathit{\sum }}_{\mathit{XZ}}^{(0)}+{\mathit{\sum }}_{\mathit{XZ}}^{(1)}=0$, and ${\mathit{\sum }}_{\mathit{ZZ}}={\mathit{\sum }}_{\mathit{ZZ}}^{(0)}+{\mathit{\sum }}_{\mathit{ZZ}}^{(1)}={\mathit{\sum }}_{\mathit{YY}}-{\mathit{\sum }}_{\mathit{YX}}{\mathit{\sum }}_{\mathit{XX}}^{-1}{\mathit{\sum }}_{\mathit{XY}}$. Hence

$${\left(\begin{array}{cc}{\mathit{\sum }}_{\mathit{XX}}& {\mathit{\sum }}_{\mathit{XZ}}\\ {\mathit{\sum }}_{\mathit{ZX}}& {\mathit{\sum }}_{\mathit{ZZ}}\end{array}\right)}^{-1}=\left(\begin{array}{cc}{\mathit{\sum }}_{\mathit{XX}}^{-1}& 0\\ 0& {\mathit{\sum }}_{\mathit{ZZ}}^{-1}\end{array}\right).$$

Because (X′, Z′)′ is a linear transformation of (X′, Y′)′, according to the above Lemmas 1 and 2, we proved Theorem 2.1.

Appendix B

Moments of ${\overline{\widehat{\mathit{Z}}}}^{(1)}-{\overline{\widehat{\mathit{Z}}}}^{(0)}$

Because

$${\overline{\widehat{\mathit{Z}}}}^{(1)}-{\overline{\widehat{\mathit{Z}}}}^{(0)}={\overline{\mathit{Y}}}^{(1)}-{\overline{\mathit{Y}}}^{(0)}-{\widehat{\beta }}^{\prime }({\overline{\mathit{X}}}^{(1)}-{\overline{\mathit{X}}}^{(0)}),$$

with

$$\widehat{\mathit{\beta }}={\mathit{S}}_{\mathit{XX}}^{-1}{\mathit{S}}_{\mathit{XY}}={\mathit{S}}_{\mathit{XX}}^{-1}\left[{\mathit{S}}_{\mathit{XX}}^{(0)}{\widehat{\mathit{\beta }}}^{(0)}+{\mathit{S}}_{\mathit{XX}}^{(1)}{\widehat{\mathit{\beta }}}^{(1)}\right],$$

$$\mathrm{E}\left(\widehat{\mathit{\beta }}\mid \mathit{X}\right)={\mathit{S}}_{\mathit{XX}}^{-1}\left[{\mathit{S}}_{\mathit{XX}}^{(0)}{\mathit{\beta }}^{(0)}+{\mathit{S}}_{\mathit{XX}}^{(1)}{\mathit{\beta }}^{(1)}\right].$$

β̂ is an asymptotic unbiased estimator of β since S is an unbiased estimator of Σ, and S⁻¹ is an asymptotic unbiased estimator of Σ⁻¹. So ${\overline{\widehat{\mathit{Z}}}}^{(1)}-{\overline{\widehat{\mathit{Z}}}}^{(0)}$ is a consistent estimator of ${\mathit{\mu }}_{\mathit{Z}}^{(1)}-{\mathit{\mu }}_{\mathit{Z}}^{(0)}$.

Note that ${\mathit{\beta }}^{(k)}={\left({\mathit{\sum }}_{\mathit{XX}}^{(k)}\right)}^{-1}{\mathit{\sum }}_{\mathit{XY}}^{(k)}$ is the regression coefficient for ${\mathit{Y}}^{(k)}={\mathit{\beta }}_0^{(k)}+{({\mathit{\beta }}^{(k)})}^{\prime }{\mathit{X}}^{(k)}+{\mathit{e}}^{(k)}$, where ${\mathit{\beta }}_0^{(k)}={\mathit{\mu }}_{\mathit{Y}}^{(0)}-{({\mathit{\beta }}^{(k)})}^{\prime }{\mathit{\mu }}_X^{(0)}$, and the error term e^(k) follows a q-dimensional normal distribution with mean vector 0 and e^(k) ⊥ X^(k).

$$\mathrm{Var}[\mathrm{vec}({\widehat{\mathit{\beta }}}^{\prime })]=\mathrm{E}\left[\mathrm{Var}\left(\mathrm{vec}({\widehat{\mathit{\beta }}}^{\prime })\mid \mathit{X}\right)\right]+\mathrm{Var}\left[\mathrm{E}\left(\mathrm{vec}({\widehat{\mathit{\beta }}}^{\prime })\mid \mathit{X}\right)\right],$$ (A.1)

where, vec means vector operator, and

$$\begin{array}{l}\mathrm{Var}[\mathrm{vec}({\widehat{\mathit{\beta }}}^{\prime })\mid \mathit{X}]\\ =\sum _{k=0}^1\left({\mathit{S}}_{\mathit{XX}}^{-1}{\mathit{S}}_{\mathit{YX}}^{(k)}\otimes {\mathit{I}}_q\right)\left\{\mathrm{Var}\left[\mathrm{vec}({\widehat{\mathit{\beta }}}^{\prime (k)})\mid \mathit{X}\right]\right\}\left({\mathit{S}}_{\mathit{YX}}^{(k)}{\mathit{S}}_{\mathit{XX}}^{-1}\otimes {\mathit{I}}_q\right)\\ =\sum _{k=0}^1\frac{1}{n_k-1}\left({\mathit{S}}_{\mathit{XX}}^{­1}{\mathit{S}}_{\mathit{YX}}^{(k)}\otimes {\mathit{I}}_q\right)\left\{{[{\mathit{S}}_{\mathit{XX}}^{(k)}]}^{-1}\otimes \mathrm{Var}({\mathit{e}}^{(k)})\right\}\left({\mathit{S}}_{\mathit{YX}}^{(k)}{\mathit{S}}_{\mathit{XX}}^{-1}\otimes {\mathit{I}}_q\right)\\ =\sum _{k=0}^1\frac{1}{n_k-1}\left({\mathit{S}}_{\mathit{XX}}^{-1}{\mathit{S}}_{\mathit{XX}}^{(k)}{\mathit{S}}_{\mathit{XX}}^{-1}\right)\otimes \mathrm{Var}({\mathit{e}}^{(k)}),\end{array}$$ (A.2)

$$\begin{array}{l}\mathrm{Var}\left[E\left\{\mathrm{vec}\left({\widehat{\mathit{\beta }}}^{\prime }\mid \mathit{X}\right)\right\}\right]\\ =\mathrm{Var}\left[E\left\{\mathrm{vec}\left({\mathit{\beta }}^{\prime (k)}-{\widehat{\mathit{\beta }}}^{\prime }\mid \mathit{X}\right)\right\}\right]\\ =\mathrm{Var}\left[\mathrm{vec}\left\{{\mathit{\beta }}^{\prime (k)}-\left({\mathit{\beta }}^{\prime (0)}{\mathit{S}}_{\mathit{XX}}^{(0)}+{\mathit{\beta }}^{\prime (1)}{\mathit{S}}_{\mathit{XX}}^{(1)}\right){\mathit{S}}_{\mathit{XX}}^{-1}\right\}\right]\\ =\mathrm{Var}\left[\mathrm{vec}\left\{{\left({\mathit{\beta }}^{(k)}-{\mathit{\beta }}^{(1-k)}\right)}^{\prime }{\mathit{S}}_{\mathit{XX}}^{(1-k)}{\mathit{S}}_{\mathit{XX}}^{-1}\right\}\right]\\ =\left[{\mathit{I}}_p\otimes {\left({\mathit{\beta }}^{(1)}-{\mathit{\beta }}^{(0)}\right)}^{\prime }\right]\mathrm{Var}\left[\mathrm{vec}\left({\mathit{S}}_{\mathit{XX}}^{(1-k)}{\mathit{S}}_{\mathit{XX}}^{-1}\right)\right]\left[{\mathit{I}}_p\otimes \left({\mathit{\beta }}^{(1)}-{\mathit{\beta }}^{(0)}\right)\right].\end{array}$$ (A.3)

$\mathrm{Var}\left[\mathrm{vec}\left({\mathit{S}}_{\mathit{XX}}^{(1-k)}{\mathit{S}}_{\mathit{XX}}^{-1}\right)\right]$ can be derived from the delta method. Note that

$$\begin{array}{lll}\frac{\partial [{\mathit{S}}_{\mathit{XX}}^{(k)}{\mathit{S}}_{\mathit{XX}}^{-1}]}{\partial {\mathit{S}}_{\mathit{XX}}^{(k)}}& =& {\mathit{S}}_{\mathit{XX}}^{-1}\otimes I_p-{\mathit{S}}_{\mathit{XX}}^{-1}\otimes ({\mathit{S}}_{\mathit{XX}}^{(k)}{\mathit{S}}_{\mathit{XX}}^{-1})={\mathit{S}}_{\mathit{XX}}^{-1}\otimes ({\mathit{S}}_{\mathit{XX}}^{(1-k)}{\mathit{S}}_{\mathit{XX}}^{-1}),\\ \frac{\partial [{\mathit{S}}_{\mathit{XX}}^{(k)}{\mathit{S}}_{\mathit{XX}}^{-1}]}{\partial {\mathit{S}}_{\mathit{XX}}^{(1-k)}}& =& -{\mathit{S}}_{\mathit{XX}}^{-1}\otimes ({\mathit{S}}_{\mathit{XX}}^{(k)}{\mathit{S}}_{\mathit{XX}}^{-1}).\end{array}$$

Adopting the first-order Taylor approximation,

$$\begin{array}{l}\mathrm{Var}\left[\mathrm{vec}\left({\mathit{S}}_{\mathit{XX}}^{(k)}{\mathit{S}}_{\mathit{XX}}^{-1}\right)\right]=\mathrm{Var}\left[\mathrm{vec}\left\{{\mathit{S}}_{\mathit{XX}}^{(k)}{\left({\mathit{S}}_{\mathit{XX}}^{(0)}+{\mathit{S}}_{\mathit{XX}}^{(1)}\right)}^{-1}\right\}\right]\\ \approx \frac{\partial \left[{\mathit{S}}_{\mathit{XX}}^{(k)}{\mathit{S}}_{\mathit{XX}}^{-1}\right]}{\partial {\mathit{S}}_{\mathit{XX}}^{(k)}}{\mid }_{\mathit{S}=\mathrm{\sum }}\mathrm{Var}\left[\mathrm{vec}({\mathit{S}}_{\mathit{XX}}^{(k)}\right]{\left[\frac{\partial \left({\mathit{S}}_{\mathit{XX}}^{(k)}{\mathit{S}}_{\mathit{XX}}^{-1}\right)}{\partial {\mathit{S}}_{\mathit{XX}}^{(k)}}{\mid }_{\mathit{S}=\mathrm{\sum }}\right]}^{\prime }+\\ \frac{\partial \left[{\mathit{S}}_{\mathit{XX}}^{(k)}{\mathit{S}}_{\mathit{XX}}^{-1}\right]}{\partial {\mathit{S}}_{\mathit{XX}}^{(1-k)}}{\mid }_{\mathit{S}=\mathrm{\sum }}\mathrm{Var}\left[\mathrm{vec}({\mathit{S}}_{\mathit{XX}}^{(1-k)}\right]{\left[\frac{\partial \left({\mathit{S}}_{\mathit{XX}}^{(k)}{\mathit{S}}_{\mathit{XX}}^{-1}\right)}{\partial {\mathit{S}}_{\mathit{XX}}^{(1-k)}}{\mid }_{\mathit{S}=\mathrm{\sum }}\right]}^{\prime }\\ =\sum _{l=0}^1\frac{n_l}{{(n_l-1)}^2}\left[{\mathit{\sum }}_{\mathit{XX}}^{-1}\otimes \left({\mathit{\sum }}_{\mathit{XX}}^{(1-l)}{\mathit{\sum }}_{\mathit{XX}}^{-1}\right)\right]\left[\left({\mathit{I}}_{p^2}+{\mathit{K}}_{\mathit{pp}}\right)\left({\mathit{\sum }}_{\mathit{XX}}^{(l)}\otimes {\mathit{\sum }}_{\mathit{XX}}^{(l)}\right)\right]\left[{\mathit{\sum }}_{\mathit{XX}}^{-1}\otimes \left({\mathit{\sum }}_{\mathit{XX}}^{-1}{\mathit{\sum }}_{\mathit{XX}}^{(1-l)}\right)\right],\end{array}$$

where K_pp is denoted the p² × p² commutation matrix, defined as

$${\mathit{K}}_{\mathit{pp}}=\sum _{i,j=1}^p({\mathit{H}}_{\mathit{ij}}\otimes {\mathit{H}}_{\mathit{ij}}^{\prime }),$$

where H_ij denote the p × p matrix with the element h_ij =1 and all other elements equal to 0. Note that β̂ ⊥ (Ȳ⁽¹⁾, Ȳ⁽⁰⁾, X̄⁽¹⁾, X̄⁽⁰⁾), hence

$$\begin{array}{l}\mathrm{Var}\left({\overline{\widehat{\mathit{Z}}}}^{(1)}-{\overline{\widehat{\mathit{Z}}}}^{(0)}\right)\\ =\mathrm{Var}\left[\mathrm{E}\left({\overline{\mathit{Y}}}^{(1)}-{\overline{\mathit{Y}}}^{(0)}-{\widehat{\beta }}^{\prime }({\overline{\mathit{X}}}^{(1)}-{\overline{\mathit{X}}}^{(0)})\mid \widehat{\beta }\right)\right]+\mathrm{E}\left[\mathrm{Var}\left({\overline{\mathit{Y}}}^{(1)}-{\overline{\mathit{Y}}}^{(0)}-{\widehat{\beta }}^{\prime }({\overline{\mathit{X}}}^{(1)}-{\overline{\mathit{X}}}^{(0)})\mid {\widehat{\beta }}_c\right)\right]\\ =\mathrm{Var}({\overline{\mathit{Z}}}^{(1)}-{\overline{\mathit{Z}}}^{(0)})+\mathrm{E}\left[\left\{{\left({\overline{\mathit{X}}}^{(1)}-{\overline{\mathit{X}}}^{(0)}\right)}^{\prime }\otimes I_q\right\}\mathrm{Var}\left\{\mathrm{vec}({\widehat{\mathit{\beta }}}^{\prime })\right\}\left({\overline{\mathit{X}}}^{(1)}-{\overline{\mathit{X}}}^{(0)}\right)\otimes I_q]\right]\\ \approx \frac{{\mathit{\sum }}_{\mathit{ZZ}}^{(0)}}{n_0}+\frac{{\mathit{\sum }}_{\mathit{ZZ}}^{(1)}}{n_1}+\left\{{\left({\mathit{\mu }}_{\mathit{X}}^{(1)}-{\mathit{\mu }}_{\mathit{X}}^{(0)}\right)}^{\prime }\otimes I_q\right\}\mathrm{Var}\left[\mathrm{vec}({\widehat{\mathit{\beta }}}^{\prime })\right]\left\{\left({\mathit{\mu }}_{\mathit{X}}^{(1)}-{\mathit{\mu }}_{\mathit{X}}^{(0)}\right)\otimes I_q\right\}.\end{array}$$

Here, we only omit a term O(min(n₀,n₁)⁻²) in the above approximation. Plugging in Var[vec(β̂′)] from (A.1)-(A.3), we get

$$\mathrm{Var}\left({\overline{\widehat{\mathit{Z}}}}^{(1)}-{\overline{\widehat{\mathit{Z}}}}^{(0)}\right)=\frac{{\mathit{\sum }}_{\mathit{ZZ}}^{(0)}}{n_0}+\frac{{\mathit{\sum }}_{\mathit{ZZ}}^{(1)}}{n_1}+V_1+V_2,$$ (A.4)

where

$${\mathit{V}}_1=\sum _{k=0}^1\frac{1}{n_k-1}\left[{\mathit{\mu }}_{\mathit{X}}^{\prime }{\mathit{\sum }}_{\mathit{XX}}^{-1}{\mathit{\sum }}_{\mathit{XX}}^{(k)}{\mathit{\sum }}_{\mathit{XX}}^{-1}{\mathit{\mu }}_{\mathit{X}}\right]\otimes \mathrm{Var}({\mathit{e}}^{(k)}),$$

$${\mathit{V}}_2=\left[{\mathit{\mu }}_{\mathit{X}}^{\prime }\otimes {\left({\mathit{\beta }}^{(1)}-{\mathit{\beta }}^{(0)}\right)}^{\prime }\right]\mathrm{Var}\left[\mathrm{vec}\left\{{\mathit{S}}_{\mathit{XX}}^{(0)}{\mathit{S}}_{\mathit{XX}}^{-1}\right\}\right]\left[{\mathit{\mu }}_{\mathit{X}}\otimes \left({\mathit{\beta }}^{(1)}-{\mathit{\beta }}^{(0)}\right)\right].$$

Appendix C

Distribution of Estimator λ̂

Since

$${\mathit{S}}_{\mathit{ZZ}}={\mathit{S}}_{\mathit{ZZ}}^{(0)}+{\mathit{S}}_{\mathit{ZZ}}^{(1)}={\mathit{S}}_{\mathit{YY}}-{\mathit{S}}_{\mathit{YX}}{\mathit{S}}_{\mathit{XX}}^{-1}{\mathit{S}}_{\mathit{XY}}.$$

Let

$$\mathit{S}=\left(\begin{array}{cc}{\mathit{S}}_{\mathit{XX}}& {\mathit{S}}_{\mathit{XY}}\\ {\mathit{S}}_{\mathit{YX}}& {\mathit{S}}_{\mathit{YY}}\end{array}\right)={\mathit{S}}^{(0)}+{\mathit{S}}^{(1)},\mathit{\sum }=\left(\begin{array}{cc}{\mathit{\sum }}_{\mathit{XX}}& {\mathit{\sum }}_{\mathit{XY}}\\ {\mathit{\sum }}_{\mathit{YX}}& {\mathit{\sum }}_{\mathit{YY}}\end{array}\right)={\mathit{\sum }}^{(0)}+{\mathit{\sum }}^{(1)},$$

where

$${\mathit{S}}^{(k)}=\left(\begin{array}{cc}{\mathit{S}}_{\mathit{XX}}^{(k)}& {\mathit{S}}_{\mathit{XY}}^{(k)}\\ {\mathit{S}}_{\mathit{YX}}^{(k)}& {\mathit{S}}_{\mathit{YY}}^{(k)}\end{array}\right),{\mathit{\sum }}^{(k)}=\left(\begin{array}{cc}{\mathit{\sum }}_{\mathit{XX}}^{(k)}& {\mathit{\sum }}_{\mathit{XY}}^{(k)}\\ {\mathit{\sum }}_{\mathit{YX}}^{(k)}& {\mathit{\sum }}_{\mathit{YY}}^{(k)}\end{array}\right)k=0,1,$$

The distribution of S is related to the multivariate Behrens-Fisher problem.

$$\mathit{S}={\mathit{S}}^{(0)}+{\mathit{S}}^{(1)}\sim \mathrm{Wishart}\left(n_0-1,\frac{{\mathit{\sum }}^{(0)}}{n_0-1}\right)+\mathrm{Wishart}\left(n_1-1,\frac{{\mathit{\sum }}^{(1)}}{n_1-1}\right),$$

Following (Nel and Merwe, 1986), approximately,

$$\mathit{S}\dot{~}\mathrm{Wishart}\left(f,\frac{\mathit{\sum }}{f}\right),$$

where

$$f=\frac{\mathrm{tr}\left\{{\left(\mathit{\sum }\right)}^2\right\}+{\left\{\mathrm{tr}\left(\mathit{\sum }\right)\right\}}^2}{\left[\mathrm{tr}\left\{{\left({\mathit{\sum }}^{(0)}\right)}^2\right\}+{\left\{\mathrm{tr}\left({\mathit{\sum }}^{(0)}\right)\right\}}^2\right]/(n_0-1)+\left[\mathrm{tr}\left\{{\left({\mathit{\sum }}^{(1)}\right)}^2\right\}+{\{\mathrm{tr}\left({\mathit{\sum }}^{(1)}\right)\}}^2\right]/(n_1-1)},$$

where “tr” means trace. According to the theorem from Muirhead p. 93-97 (Muirhead, 1982),

$${\mathit{S}}_{\mathit{ZZ}}={\mathit{S}}_{\mathit{YY}}-{\mathit{S}}_{\mathit{YX}}{\mathit{S}}_{\mathit{XX}}^{-1}{\mathit{S}}_{\mathit{XY}}~\mathrm{Wishart}\left(f-p,\frac{{\mathit{\sum }}_{\mathit{ZZ}}}{f}\right).$$

So E(S_ZZ) = (f − p) Σ_ZZ / f. The variance of ${\mathit{S}}_{\mathit{ZZ}}^{(k)}$ tends to 0 as n^(k) approaches infinity. So S_ZZ converges in the probability to the value of (f − p)Σ_ZZ / f. On the basis of the convergence theorem, the asymptotic distribution of λ̂ is the same as the quadratic statistics f / (f − p)Λ₁ for large sample size, where

$${\mathrm{\Lambda }}_1={\left({\overline{\widehat{\mathit{Z}}}}^{(1)}-{\overline{\widehat{\mathit{Z}}}}^{(0)}\right)}^{\prime }{\mathit{\sum }}_{\mathit{ZZ}}^{-1}\left({\overline{\widehat{\mathit{Z}}}}^{(1)}-{\overline{\widehat{\mathit{Z}}}}^{(0)}\right).$$

Many exact and approximate methods are available to calculate the distribution of quadratic forms in normal variables, including Patnaik’s two-moment central χ²-approximation (Box, 1954; Patnaik, 1949), Pearson’s three-moment central χ² approximation etc. (Imhof, 1961; Pearson, 1959). All these estimations were derived either based on an assumption of equal means for Z⁽¹⁾ and Z⁽⁰⁾ (thus zero differences) or simply approximate noncentral χ² by a central χ² approximation. They are inappropriate to our problem because we want to test the quadratic statistics above a positive non-zero constant. Instead, we use a two-moment noncentral χ² approximation to calculate the distribution of Λ₁. We approximate

$$c^{\ast }{\mathrm{\Lambda }}_1\approx {\chi }^2(\upsilon ,c^{\ast }\lambda )$$

with degrees of freedom υ and a noncentral parameter c* λ, where c*, υ are derived from the two-moment equations

$$c^{\ast }\left[\mathrm{tr}({\mathit{\sum }}_{\mathit{ZZ}}^{-1}{\mathit{\sum }^{\sim }}_{\mathit{ZZ}})+{\mathit{\mu }}_{\mathit{Z}}^{\prime }{\mathit{\sum }}_{\mathit{ZZ}}^{-1}\mathit{\mu }\right]=\left(\upsilon +c^{\ast }{\mathit{\mu }}_{\mathit{Z}}^{\prime }{\mathit{\sum }}_{\mathit{ZZ}}^{-1}{\mathit{\mu }}_{\mathit{Z}}\right)$$

and

$$2\mathrm{tr}[{({\mathit{\sum }}_{\mathit{ZZ}}^{-1}{\mathit{\sum }^{\sim }}_{\mathit{ZZ}})}^2]+4{\mathit{\mu }}_{\mathit{Z}}^{\prime }{\mathit{\sum }}_{\mathit{ZZ}}^{-1}{\mathit{\sum }^{\sim }}_{\mathit{ZZ}}{\mathit{\sum }}_{\mathit{ZZ}}^{-1}{\mathit{\mu }}_{\mathit{Z}}=\frac{1}{{(c^{\ast })}^2}\left(2\upsilon +4c{\mathit{\mu }}_{\mathit{Z}}^{\prime }{\mathit{\sum }}_{\mathit{ZZ}}^{-1}{\mathit{\mu }}_{\mathit{Z}}\right),$$

where ${\mathit{\sum }^{\sim }}_{\mathit{ZZ}}=\mathrm{Var}({\overline{\widehat{\mathit{Z}}}}^{(1)}-{\overline{\widehat{\mathit{Z}}}}^{(0)})=\frac{{\mathit{\sum }}_{\mathit{ZZ}}^{(0)}}{n_0}+\frac{{\mathit{\sum }}_{\mathit{ZZ}}^{(1)}}{n_1}+{\mathit{V}}_1+{\mathit{V}}_2$, Therefore we get

$$c^{\ast }=\frac{\mathrm{tr}({\mathit{\sum }}_{\mathit{ZZ}}^{-1}{\mathit{\sum }^{\sim }}_{\mathit{ZZ}})+2{\mathit{\mu }}_{\mathit{Z}}^{\prime }{\mathit{\sum }}_{\mathit{ZZ}}^{-1}\mathit{\mu }}{\mathrm{tr}[{({\mathit{\sum }}_{\mathit{ZZ}}^{-1}{\mathit{\sum }^{\sim }}_{\mathit{ZZ}})}^2]+2{\mathit{\mu }}_{\mathit{Z}}^{\prime }{\mathit{\sum }}_{\mathit{ZZ}}^{-1}{\mathit{\sum }^{\sim }}_{\mathit{ZZ}}{\mathit{\sum }}_{\mathit{ZZ}}^{-1}\mathit{\mu }},$$

and

$$\upsilon =c^{\ast }\cdot \mathrm{tr}({\mathit{\sum }}_{\mathit{ZZ}}^{-1}{\mathit{\sum }^{\sim }}_{\mathit{ZZ}})$$

In the special case of Σ_ZZ = lΣ̃_ZZ for a positive constant l, which is always true for one-dimensional q = 1, c* = l, υ = q, lΛ₁ = χ²(q, l λ).

So, when n₀, n₁ are large enough, f / (f − p)Λ₁ can directly estimate λ with approximate distribution

$$c^{\ast }{\mathrm{\Lambda }}_1\dot{~}{\chi }^2(\upsilon ,c^{\ast }\lambda ).$$

For small sample size, however, we cannot simply replace Σ_ZZ by f(f − p)S_ZZ. Let

$${\mathrm{\Lambda }}_2=\frac{{\left({\overline{\widehat{\mathit{Z}}}}^{(1)}-{\overline{\widehat{\mathit{Z}}}}^{(0)}\right)}^{\prime }{\mathit{\sum }}_{\mathit{ZZ}}^{-1}\left({\overline{\widehat{\mathit{Z}}}}^{(1)}-{\overline{\widehat{\mathit{Z}}}}^{(0)}\right)}{{\left({\overline{\widehat{\mathit{Z}}}}^{(1)}-{\overline{\widehat{\mathit{Z}}}}^{(0)}\right)}^{\prime }{\mathit{S}}_{\mathit{ZZ}}^{-1}\left({\overline{\widehat{\mathit{Z}}}}^{(1)}-{\overline{\widehat{\mathit{Z}}}}^{(0)}\right)}.$$

Then λ̂ = Λ₁/Λ₂ is a Hotelling-T² type statistic. Although ${\overline{\widehat{\mathit{Z}}}}^{(1)}-{\overline{\widehat{\mathit{Z}}}}^{(0)}$ and S_ZZ may correlate, the correlation between is small as O(min(n₀,n₁)⁻¹) and we can ignore it. According to the theorem from (Muirhead, 1982), f Λ₂ follows a central χ²-distribution with degrees of freedom f − p − q +1. Therefore

$$\frac{f-p-q+1}{f\upsilon }{c}^{\ast }\widehat{\lambda }=\frac{c^{\ast }{\mathrm{\Lambda }}_1/\upsilon }{f{\mathrm{\Lambda }}_2/(f-p-q+1)}\dot{~}F(\upsilon ,f-p-q+1,c^{\ast }\lambda ),$$ (A.5)

with degrees of freedom υ, f − p − q + 1 and noncentrality parameter c* λ.

We can estimate f, c*, and υ by substituting S^(k), f(f − p)S_ZZ, and ${\widehat{\mathit{\sum }^{\sim }}}_{\mathit{ZZ}}$ for Σ^(k), Σ_ZZ and Σ_ZZ,

$$\widehat{f}=\frac{\left[\mathrm{tr}\left\{{\left({\mathit{S}}^{(0)}+{\mathit{S}}^{(1)}\right)}^2\right\}+{\left\{\mathrm{tr}\left({\mathit{S}}^{(0)}+{\mathit{S}}^{(1)})\right)\right\}}^2\right]}{\left[\mathrm{tr}\left\{{\left({\mathit{S}}^{(0)}\right)}^2\right\}+{\left\{\mathrm{tr}\left({\mathit{S}}^{(0)}\right)\right\}}^2\right]/(n_0-1)+\left[\mathrm{tr}\left\{{\left({\mathit{S}}^{(1)}\right)}^2\right\}+{\left\{\mathrm{tr}\left({\mathit{S}}^{(1)}\right)\right\}}^2\right]/(n_1-1)}$$

$${\widehat{c}}^{\ast }=\frac{\widehat{f}}{\widehat{f}-p}\widehat{c},$$

$$\widehat{\upsilon }=\widehat{c}\cdot \mathrm{tr}({\mathit{S}}_{\mathit{ZZ}}^{-1}{\widehat{\mathit{\sum }^{\sim }}}_{\mathit{ZZ}}),$$

with

$$\widehat{c}=\frac{\mathrm{tr}({\mathit{S}}_{\mathit{ZZ}}^{-1}{\widehat{\mathit{\sum }^{\sim }}}_{\mathit{ZZ}})+2{\left({\overline{\widehat{\mathit{Z}}}}^{(1)}­{\overline{\widehat{\mathit{Z}}}}^{(0)}\right)}^{\prime }{\mathit{S}}_{\mathit{ZZ}}^{-1}\left({\overline{\widehat{\mathit{Z}}}}^{(1)}­{\overline{\widehat{\mathit{Z}}}}^{(0)}\right)}{\mathrm{tr}[{({\mathit{S}}_{\mathit{ZZ}}^{-1}{\widehat{\mathit{\sum }^{\sim }}}_{\mathit{ZZ}})}^2]+2{\left({\overline{\widehat{\mathit{Z}}}}^{(1)}­{\overline{\widehat{\mathit{Z}}}}^{(0)}\right)}^{\prime }{\mathit{S}}_{\mathit{ZZ}}^{-1}{\widehat{\mathit{\sum }^{\sim }}}_{\mathit{ZZ}}{\mathit{S}}_{\mathit{ZZ}}^{-1}\left({\overline{\widehat{\mathit{Z}}}}^{(1)}­{\overline{\widehat{\mathit{Z}}}}^{(0)}\right)}.$$

Then (A.5) can be written as

$$\frac{\widehat{f}-p-q+1}{(\widehat{f}-p)\widehat{\upsilon }}\widehat{c}\widehat{\lambda }\dot{~}F\left(\widehat{\upsilon },\widehat{f}-p-q+1,\frac{\widehat{f}}{\widehat{f}-p}\widehat{c}\lambda \right).$$ (A.6)