Significance tests for covariates in the diagnostic accuracy index of a biomarker against a continuous gold standard

Summary

Receiver operating characteristic (ROC) curve is a popular tool to describe and compare the diagnostic accuracy of biomarkers when a binary-scale gold standard is available. However, there are many examples of diagnostic tests whose gold standards are continuous. Hence, Several extensions of receiver operating characteristic (ROC) curve are proposed to evaluate the diagnostic potential of biomarkers when the gold standard is continuous-scale. Moreover, in evaluating these biomarkers, it is often necessary to consider the effects of covariates on the diagnostic accuracy of the biomarker of interest. Covariates may include subject characteristics, expertise of the test operator, test procedures or aspects of specimen handling. Applying the covariate adjustment to the case that the gold standard is continuous is challenging and has not been addressed in the literature. To fill the gap, we propose two general testing frameworks to account for the covariates effect on diagnostic accuracy. Simulation studies are conducted to compare the proposed tests. Data from a study that assessed three types of imaging modalities with the purpose of detecting neoplastic colon polyps and cancers are used to illustrate the proposed methods.

1 |. INTRODUCTION

In medical diagnostics, the receiver operating characteristic (ROC) curve is a useful tool to assess a biomarker’s accuracy and compare accuracies of competing biomarkers. In traditional ROC analysis, a subject is diagnosed as diseased and nondiseased based on the subject’s level of the biomarker. The diagnostic accuracy of the biomarker depends on the existence of a binary-scale gold standard, usually obtained from clinical follow-up, surgical verification, autopsy, and so on. Sensitivity and Specificity at various cutpoints of the biomarker’s test results summarize the diagnostic accuracies of the biomarker.

In practice however, there are many examples of gold standards whose outcomes are not on a binary-scale, but rather on a continuous-scale. For example, glycosylated hemoglobin is usually used as a primary index of diabetic control, and is originally measured as a continuous-valued variable. Setting an inappropriate threshold to dichotomize a continuous gold standard results in imprecision of the diagnostic accuracy of the biomarker and conceals an important relationship between the biomarker and the gold standard. Obuchowski^1,2 extended the analysis of the diagnostic accuracy of a binary gold standard to the case of a continuous gold standard, and proposed an index measure and presented a nonparametric estimator of the index for a continuous biomarker. Chang³ discussed linearly combining multiple biomarkers to maximize this accuracy index.

It is well known that the performance of a diagnostic test/biomarker may depend on certain characteristics of patients, the environmental factors of diagnostic tests, setting and types of diagnostic tests, and so on. When the gold standard is binary-scale, various methods have been developed for regression analysis of ROC curves and the summary measures of accuracy derived from them to account for the effect of covariates. Methods found in the literature for modeling the effect of covariates on ROC curves can be loosely divided into two categories, the indirect and the direct approach; see, among many others.^4,5 The indirect approach is to model the test results as a function of the binary gold standard and covariates and then derive the ROC curve or its summary measures. This method was first proposed by Tosteson and Begg,⁶ who fitted an ordinal regression model with location and scale parameters on the result of the diagnostic test that is ordered categories. O’Malley et al⁷ proposed a Bayesian regression method for continuous test results and then evaluated the ROC curve. Faraggi⁸ described the biomarker values as functions of covariates with the assumption that there is no covariate dependence in the diseased population and then obtained the estimates for the area under the ROC curve and the Youden index. The direct approach is to model the ROC curves or summary measures on covariates and the binary gold standard. Regression models for various ROC summary measures have been proposed, such as the area under the ROC curve based on a conventional analysis of variance (ANOVA).^9,10 Pepe¹¹ proposed a regression model for ROC curves directly and provided the parameter estimation using GLM procedures. Kim and Zeng⁵ generalized an accelerated failure time model from survival analysis to ROC analysis, assuming a relationship between covariates and the baseline ROC curve. Nonparametric and semi-parametric estimation of the effect of covariates was also discussed, in which regression models for ROC summary indices or ROC curves were available; see References 12–15, among others. Regression analysis has also been developed when there is an ignorable verification bias and missing or partially missing gold standard.^14,16–18

To the best of our knowledge, however, there is no specific methodology developed in the literature to account for the effect of covariates in assessing the diagnostic accuracy when the gold standard is continuous. To fill this gap, we develop methodology to directly evaluate the effect of covariates on the accuracy index concerning a biomarker and a gold standard, both measured on a continuous scale. A motivating example is to assess the accuracy in detecting colon cancer of two imaging modalities, barium enema (BE) and computed tomographic colonography (CTC), as compared to colonoscopy (CO) which is considered the gold standard. Due to the potential biological differences in race and family history, our interest is to assess the how the diagnostic accuracy varies according to the family history and race of the patient.

The paper is arranged as follows. In Section 2, we propose two test methods to assess the effect of a common covariate on the diagnostic accuracy of a biomarker against a continuous gold standard. In Section 3, we first present simulation results on type I error and power of the tests for various sample sizes, and then demonstrate the methods using imaging modalities data. Some discussion is provided in Section 4. The proofs of the asymptotic results are relegated to the appendix.

2 |. METHODOLOGY

Let $Y_i$ be the gold standard, $X_i$ be the biomarker, and $Z_i$ be the covariate, for the $i$th subject $(i=1,2,\dots ,N)$. We assume that $X$ and $Y$ are continuous and without loss of generality, $Z$ is discrete, taking $K$ possible values $z_1,\dots ,z_K$. We want to build a model of the diagnostic accuracy on the covariate. Obuchowski^1,2 defined the accuracy as the probability that a randomly selected patient with a higher gold standard outcome has a higher biomarker test result than a randomly selected patient with a lower gold standard outcome. Extending this definition, for each level $k(=1,2,\dots ,K)$, we define the diagnostic accuracy index as

$${\theta }_k=P\left(\left(Y_i-Y_j\right)\left(X_i-X_j\right)>0\mid Z_i=Z_j=Z_k\right).$$ (1)

We estimate ${\theta }_k$ with Obuchowski’s^1,2 nonparamatric estimator ${\hat{\theta }}_k$, which is calculated by summing the indicators of all possible pairs of subjects. Thus, at the $k$th level ($k=1,2,\dots K$), we have an unbiased estimator for ${\theta }_k$:

$${\hat{\theta }}_k={\left(\genfrac{}{}{0pt}{}{n_k}{2}\right)}^{-1}\sum _{1\le i<j\le n_k}\mathrm{\Psi }\left(\left(X_i-X_j\right)\left(Y_i-Y_j\right)\mid Z_i=Z_j=z_k\right),$$ (2)

where $n_k$ is the number of observations at the covariate level $k$ and the summations are taken over the corresponding sets of observations with the same covariate level, the function $\mathrm{\Psi }(\cdot \mid z)$ given $Z=z$ satisfies that

$$\mathrm{\Psi }(t\mid z)=\left\{\begin{array}{ll}1,& \text{if}t>0,\\ 0.5,& \text{if}t=0,\\ 0,& \text{otherwise.}\end{array}\right.$$ (3)

For simplicity, we denote

$${\mathrm{\Psi }}_{i,j\mid z_k}=\mathrm{\Psi }\left(\left(X_i-X_j\right)\left(Y_i-Y_j\right)\mid Z_i=Z_j=z_k\right).$$

Our main aim is to investigate whether the accuracy $\theta$ depends on the covariate $Z$.

2.1 |. Regression analysis

Assume that the link model $G\left({\theta }_k\right)$ is a linear function of the covariate $Z$ at each level $k$, that is,

$$G\left({\theta }_k\right)=b_0+b_1{z}_k,$$ (4)

where $G(\cdot )$ is a strictly monotone function with domain ($-\mathrm{\infty },+\mathrm{\infty }$) and range (0, 1). In practice, $G\left({\theta }_k\right)$ is usually be adopted as the probit-link function ${\mathrm{\Phi }}^{-1}\left({\theta }_k\right)$ or the logit-link function $\mathrm{l}\mathrm{o}\mathrm{g}\left({\theta }_k/\left(1+{\theta }_k\right)\right)$, where $\mathrm{\Phi }(\cdot )$ is the standard normal distribution function. The regression coefficients $\mathit{b}={\left(b_0,b_1\right)}^T$ can be estimated using the weighted linear regression technique on the data $\left\{G\left({\theta }_k\right),z_k\right\}$.

The exact variance of ${\hat{\theta }}_k$ given below is derived using the theory developed for general $U$-statistics,

$$\begin{gathered} \mathrm{V}\mathrm{a}\mathrm{r}({\hat{\theta }}_k)=E{({\hat{\theta }}_k-{\theta }_k)}^2={\left(\genfrac{}{}{0pt}{}{n_k}{2}\right)}^{-2}E{\left(\sum _{1\le i<j\le n_k}\left({\mathrm{\Psi }}_{i,j\mid z_k}-{\theta }_k\right)\right)}^2 \\ ={\left(\genfrac{}{}{0pt}{}{n_k}{2}\right)}^{-2}\left(\left(\genfrac{}{}{0pt}{}{n_k}{2}\right)\mathrm{V}\mathrm{a}\mathrm{r}\left({\mathrm{\Psi }}_{i,j\mid z_k}\right)+6\left(\genfrac{}{}{0pt}{}{n_k}{3}\right)\mathrm{C}\mathrm{o}\mathrm{v}\left({\mathrm{\Psi }}_{i,j\mid z_k},{\mathrm{\Psi }}_{i,l\mid z_k}\right)\right) \end{gathered}$$

where $\mathrm{V}\mathrm{a}\mathrm{r}({\mathrm{\Psi }}_{i,j\mid z_k})=E{({\mathrm{\Psi }}_{i,j\mid z_k}-{\theta }_k)}^2={\theta }_k-{\theta }_k^2$, and

$$\begin{gathered} \mathrm{C}\mathrm{o}\mathrm{v}({\mathrm{\Psi }}_{i,j\mid z_k},{\mathrm{\Psi }}_{i,l\mid z_k})=\mathrm{C}\mathrm{o}\mathrm{v}({\mathrm{\Psi }}_{i,j\mid z_k},{\mathrm{\Psi }}_{m,j\mid z_k})=\mathrm{C}\mathrm{o}\mathrm{v}({\mathrm{\Psi }}_{i,j\mid z_k},{\mathrm{\Psi }}_{j,l\mid z_k})=\mathrm{C}\mathrm{o}\mathrm{v}({\mathrm{\Psi }}_{i,j\mid z_k},{\mathrm{\Psi }}_{m,i\mid z_k}) \\ =E({\mathrm{\Psi }}_{i,j\mid z_k}-{\theta }_k)({\mathrm{\Psi }}_{i,l\mid z_k}-{\theta }_k)=E({\mathrm{\Psi }}_{i,j\mid z_k}{\mathrm{\Psi }}_{i,l\mid z_k})-{\theta }_k^2. \end{gathered}$$

Denote $p_k=E({\mathrm{\Psi }}_{i,j\mid z_k}{\mathrm{\Psi }}_{i,l\mid z_k})$. Thus the variance of ${\hat{\theta }}_k$ can also be written in terms of $p_k$ and ${\theta }_k$ as

$${\sigma }_k^2=\mathrm{V}\mathrm{a}\mathrm{r}({\hat{\theta }}_k)=\frac{2}{n_k\left(n_k-1\right)}{\theta }_k-\frac{2\left(2n_k-3\right)}{n_k\left(n_k-2\right)}{\theta }_k^2+\frac{4\left(n_k-2\right)}{n_k\left(n_k-1\right)}{p}_k.$$ (5)

We use a nonparametric unbiased estimator ${\hat{p}}_k$ to estimate $p_k$, where

$${\hat{p}}_k=\frac{1}{3}{\left(\genfrac{}{}{0pt}{}{n_k}{3}\right)}^{-1}\sum _{1\le i<j<l\le n_k}\{{\mathrm{\Psi }}_{i,j\mid z_k}{\mathrm{\Psi }}_{i,l\mid z_k}+{\mathrm{\Psi }}_{i,j\mid z_k}{\mathrm{\Psi }}_{j,l\mid z_k}+{\mathrm{\Psi }}_{i,l\mid z_k}{\mathrm{\Psi }}_{j,l\mid z_k}\}.$$

Substituting ${\theta }_k$ and $p_k$ for ${\hat{\theta }}_k$ and ${\hat{p}}_k$, we obtain an estimator of the variance of ${\hat{\theta }}_k$:

$${\hat{\sigma }}_k^2=\widehat{\mathrm{V}\mathrm{a}\mathrm{r}}({\hat{\theta }}_k)=\frac{2}{n_k\left(n_k-1\right)}{\hat{\theta }}_k-\frac{2\left(n_k-3\right)}{n_k\left(n_k-1\right)}{\hat{\theta }}_k^2+\frac{4\left(n_k-2\right)}{n_k\left(n_k-1\right)}{\hat{p}}_k.$$ (6)

Alternatively, the variance of ${\hat{\theta }}_k$ can be estimated via bootstrapping with replacement from $\left\{\left(\left(X_i,Y_i\right)\mid Z_i=Z_k\right),i=\right.1,2,\dots ,N\}$, denoted as ${\hat{\sigma }}_{\mathit{bootstrap},k}^2$. Later we will present simulation results to compare these two variance estimation approaches in controlling error rates.

Using the delta-method, the variance of $G({\hat{\theta }}_k)$ is found to be

$$w_k=\mathrm{V}\mathrm{a}\mathrm{r}(G({\hat{\theta }}_k))={\left[g\left({\theta }_k\right)\right]}^2{\sigma }_k^2$$ (7)

and its estimator

$${\hat{w}}_k=\widehat{\mathrm{V}\mathrm{a}\mathrm{r}}(G({\hat{\theta }}_k))={\left[g({\hat{\theta }}_k)\right]}^2{\hat{\sigma }}_k^2\left(\text{or}{\left[g({\hat{\theta }}_k)\right]}^2{\hat{\sigma }}_{\mathit{bootstrap},k}^2\right),$$ (8)

where $g(\cdot )$ is the derivative of $G(\cdot )$. Especially, $g\left({\theta }_k\right)=1/\varphi \left({\mathrm{\Phi }}^{-1}\left({\theta }_k\right)\right)$ when $G\left({\theta }_k\right)={\mathrm{\Phi }}^{-1}\left({\theta }_k\right)$, and $g\left({\theta }_k\right)=1/\left({\theta }_k\left(1+{\theta }_k\right)\right)$ when $G\left({\theta }_k\right)=\mathrm{l}\mathrm{o}\mathrm{g}\left({\theta }_k/1+{\theta }_k\right)$, where $\varphi$ is the standard normal density.

To investigate whether the accuracy $\theta$ depends on the covariate $Z$, we test the null hypothesis $H_0:b_1=0$. To this end, we rewrite our model $G({\hat{\theta }}_k)=b_0+b_1{z}_k$ as

$$\mathit{R}=\mathbf{Z}\mathit{b}+\mathit{e},$$ (9)

where $\mathit{R}={(G({\hat{\theta }}_1),G({\hat{\theta }}_2),\dots ,G({\hat{\theta }}_K))}^T,\mathbf{Z}={((1,1,\dots ,1{)}^T,{(z_1,z_2,\dots ,z_K)}^T)}^T,\mathit{b}={(b_0,b_1)}^T$, and $\mathit{e}\sim N_K(\mathbf{0},\mathbf{W})$.

Assume that samples at different covariate levels are independent. Then it has

$$\mathrm{C}\mathrm{o}\mathrm{v}({\hat{\theta }}_k,{\hat{\theta }}_m)=0,\mathrm{C}\mathrm{o}\mathrm{v}(G({\hat{\theta }}_k),G({\hat{\theta }}_m))=0,1\le k\ne m\le K.$$ (10)

Furthermore, we have that the variance-covariance matrix $W$ of the response variable $\mathit{R}$ is a diagonal matrix with elements $w_1,w_2,\dots ,w_K$. Thus the regression coefficients $\mathit{b}$ can be estimated by

$$\stackrel{\mathit{ˆ}}{\mathit{b}}={\left({\mathbf{Z}}^T{\mathbf{W}}^{-1}\mathbf{Z}\right)}^{-1}{\mathbf{Z}}^T{\mathbf{W}}^{-1}\mathit{R},$$ (11)

with $\mathrm{C}\mathrm{o}\mathrm{v}(\stackrel{\mathit{ˆ}}{\mathit{b}})={({\mathbf{Z}}^T{\mathbf{W}}^{-1}\mathbf{Z})}^{-1}$. Replacing $\mathbf{W}$ with its estimate $\widehat{\mathbf{W}}=\mathrm{D}\mathrm{i}\mathrm{a}\mathrm{g}({\hat{w}}_1,{\hat{w}}_2,\dots ,{\hat{w}}_K)$ obtains the feasible estimator of $\mathit{b}$, still denoted by $\stackrel{\mathit{ˆ}}{\mathit{b}}$ in the following. Note that ${\hat{b}}_1=(0,1)\stackrel{\mathit{ˆ}}{\mathit{b}}$. Therefore, the variance of ${\hat{b}}_1$ can be approximately estimated by $\widehat{\mathrm{V}\mathrm{a}\mathrm{r}}({\hat{b}}_1)=(0,1){({\mathbf{Z}}^T{\widehat{\mathbf{W}}}^{-1}\mathbf{Z})}^{-1}(0,1{)}^T$.

A test statistic can then be defined as

$$T_0=\frac{{\hat{b}}_1}{\sqrt{\widehat{\mathrm{V}\mathrm{a}\mathrm{r}}({\hat{b}}_1)}},$$ (12)

which asymptotically follows a standard normal distribution when $b_1=0$. The null hypothesis is rejected if $\left|T_0\right|>Z_{\alpha /2}$, where $Z_{\alpha /2}$ is the upper $\alpha /2$ percentile of the standard normal distribution.

2.2 |. The Hotelling’s ${\mathit{T}}^2$ approach

The regression approach in general assumes that the accuracy index and the covariate satisfy certain known functional relationship. In the regression model developed in the previous section, we assume that the transformed accuracy under a known monotone link is a linear function of the covariate values, and obtain estimates of the regression coefficients via weighted least squares method. In simulation we adopt the normal probit link. When such a functional relationship is not clear we may test a more general hypothesis $H_0:{\theta }_1={\theta }_2=\cdots ={\theta }_K$, with the alternative hypothesis $H_1$: not all ${\theta }_k$s are equal. Equivalently, the test can be written in the form of a union-intersection test,

$$H_0:\bigcap _{m<K}\left\{{\theta }_m-{\theta }_K=0\right\}\leftrightarrow H_1:\bigcup _{m<K}\left\{{\theta }_m-{\theta }_K\ne 0\right\}.$$ (13)

We propose a test statistic analogous to the Hotelling’s $T^2$ to test the above null hypothesis. Since all the ${\hat{\theta }}_k$s are $U$-statistics, they have asymptotically normal distributions and are mutually independent. Therefore,

$$\begin{array}{l}E({\hat{\theta }}_m-{\hat{\theta }}_K)={\theta }_m-{\theta }_K,\\ \mathrm{V}\mathrm{a}\mathrm{r}({\hat{\theta }}_m-{\hat{\theta }}_K)=\mathrm{V}\mathrm{a}\mathrm{r}({\hat{\theta }}_m)+\mathrm{V}\mathrm{a}\mathrm{r}({\hat{\theta }}_K)={\sigma }_m^2+{\sigma }_K^2,m=1,2,\dots ,K-1,\\ \mathrm{C}\mathrm{o}\mathrm{v}({\hat{\theta }}_m-{\hat{\theta }}_K,{\hat{\theta }}_l-{\hat{\theta }}_K)=\mathrm{V}\mathrm{a}\mathrm{r}({\hat{\theta }}_K)={\sigma }_K^2,m\ne l.\end{array}$$

Or equivalently, $\stackrel{\mathit{ˆ}}{\mathit{\theta }}$ is asymptotically $N(\mathit{\mu },\mathbf{\Sigma })$ where where

$$\begin{array}{c}\stackrel{\mathit{ˆ}}{\mathit{\theta }}={({\hat{\theta }}_1-{\hat{\theta }}_K,{\hat{\theta }}_2-{\hat{\theta }}_K,\dots ,{\hat{\theta }}_{K-1}-{\hat{\theta }}_K)}^T,\mathit{\mu }={\left({\theta }_1-{\theta }_K,{\theta }_2-{\theta }_K,\dots ,{\theta }_{K-1}-{\theta }_K\right)}^T,\\ \mathbf{\Sigma }=\left(\begin{array}{cccc}{\sigma }_1^2+{\sigma }_K^2& {\sigma }_K^2& \cdots & {\sigma }_K^2\\ {\sigma }_K^2& {\sigma }_2^2+{\sigma }_K^2& \cdots & {\sigma }_K^2\\ ⋮& ⋮& \ddots & ⋮\\ {\sigma }_K^2& {\sigma }_K^2& \cdots & {\sigma }_{K-1}^2+{\sigma }_K^2\end{array}\right).\end{array}$$

Then our Hotelling’s $T^2$ statistic is

$$T^2={\stackrel{\mathit{ˆ}}{\mathit{\theta }}}^T{\stackrel{\mathbf{ˆ}}{\mathbf{\Sigma }}}^{-1}\stackrel{\mathit{ˆ}}{\mathit{\theta }},$$ (14)

where the estimate $\stackrel{\mathbf{ˆ}}{\mathbf{\Sigma }}$ of $\mathbf{\Sigma }$ is obtained with ${\sigma }_k^2$ replacing ${\hat{\sigma }}_k^2$ (or ${\hat{\sigma }}_{\mathit{bootstrap},k}^2),k=1,\dots ,K$. Under the null hypothesis, $T^2$ asymptotically has a ${\chi }^2$ distribution with $K-1$ degrees of freedom. If the observed $T^2$ is greater than ${\chi }_{K-1}^2(\alpha )$, we reject the null hypothesis $H_0$ at the $\alpha$ significance level, where ${\chi }_{K-1}^2(\alpha )$ is the upper $\alpha$ percentile of the ${\chi }^2$ distribution with $K-1$ degrees of freedom. Technical details on the asymptotic distribution of $\stackrel{\mathit{ˆ}}{\mathit{\theta }}$ are provided in the appendix.

Remark 1. When $K=2$, the test statistic $T_0$ based on the regression method and the Hotelling’s $T^2$ are respectively

$$T_0=\frac{G({\hat{\theta }}_1)-G({\hat{\theta }}_2)}{\sqrt{{\hat{w}}_1+{\hat{w}}_2}},T^2=\frac{{({\hat{\theta }}_1-{\hat{\theta }}_2)}^2}{{\hat{\sigma }}_1^2+{\hat{\sigma }}_2^2}.$$

Due to the monotonicity of $G(\cdot )$, it follows from the delta-method that the two methods perform similarly well, especially when $n_1$ and $n_2$ are large, see simulation results in the next section.

Remark 2. The test based on regression analysis usually has higher power if the accuracy index $\theta$ increases as the value of $Z$ increases (or decreases), even if the link function is misspecified, otherwise Hotelling’s $T^2$ test performs better.

Remark 3. For the original sample of the continuous random variables $(X,Y),\mathrm{\Psi }\left(\left(Y_i-Y_j\right)\left(X_i-X_j\right)\mid Z_i=Z_j=\right.\left.Z_k\right)$ in (2) can be replaced by the indicator function $I\left(\left(Y_i-Y_j\right)\left(X_i-X_j\right)>0\mid Z_i=Z_j=Z_k\right)$, but the latter will lead to an underestimation of ${\theta }_k$ under bootstrap sampling, due to the repetition of some subjects in each bootstrap sample.

3 |. SIMULATION STUDY AND A REAL EXAMPLE

We conducted simulation studies to examine the finite sample performance of the two methods proposed in the previous section, the $Z$-test based on the regression analysis and the Hotelling $T^2$ test. For each test, we also compared the two estimates of the variance of ${\theta }_k$: one is the approximate variance (6) given in Section 2.1, and the other is the bootstrap variance with 1000 repetitions. We compared the type I error rates and powers of the two tests under various alternatives. This section summarizes the simulation results and demonstrates the proposed methods with a real example.

3.1 |. Comparison of regression analysis and Hotelling’s ${\mathit{T}}^2$ test

We assume that given the covariate $Z,(X,Y)$ follows a bivariate normal distribution. Then it follows from Reference 19 that

$${\theta }_k=P\left(\left(X_i-X_j\right)\left(Y_i-Y_j\right)>0\mid Z_i=Z_j=Z_k\right)=\frac{1}{2}+\frac{1}{\pi }\mathrm{a}\mathrm{r}\mathrm{c}\mathrm{s}\mathrm{i}\mathrm{n}\left({\rho }_k\right),$$ (15)

where ($X_i,Y_i$) and ($X_j,Y_j$) are random samples from the distribution of ($X,Y$) given $Z=z_k$,

$${\rho }_k=\mathrm{C}\mathrm{o}\mathrm{r}\mathrm{r}\left(X,Y\mid Z=z_k\right)=\frac{\mathrm{C}\mathrm{o}\mathrm{v}\left(X,Y\mid Z=z_k\right)}{\sqrt{\mathrm{V}\mathrm{a}\mathrm{r}\left(X\mid Z=z_k\right)\mathrm{V}\mathrm{a}\mathrm{r}\left(Y\mid Z=z_k\right)}}.$$

Without the loss of generality, we assume

$$(X,Y)|Z=z_k\sim N_2\left(\left(\genfrac{}{}{0pt}{}{0}{0}\right),\left(\begin{array}{cc}{\sigma }_X^2& {\sigma }_X{\sigma }_Y{\rho }_k\\ {\sigma }_X{\sigma }_Y{\rho }_k& {\sigma }_Y^2\end{array}\right)\right),$$

and ${\theta }_k$ is a function of ${\rho }_k$, which is the correlation coefficient between the biomarker $X$ and the gold standard $Y$ when the covariate is at the kth level. In our simulations, $G\left({\theta }_k\right)$ is taken as the probit link function ${\mathrm{\Phi }}^{-1}\left({\theta }_k\right)$, the covariate $Z$ was set to have $K=2,3,4$ or 5 levels, and variances of $X$ and $Y$ are equal (${\sigma }_Y={\sigma }_X=1$) or unequal (${\sigma }_Y=1\ne {\sigma }_X=1.2$). For each level, we assumed that we could obtain $n_1=\cdots =n_K=n=30,60,100$, or 150 observations. We generated 1000 data sets to obtain the empirical type I error rates and power under various alternative hypotheses.

3.1.1 |. The type I error

Under the null hypothesis $H_0:{\theta }_1=\cdots ={\theta }_K$, which is equivalent to $H_0:{\rho }_1=\cdots ={\rho }_K=\rho$, we fix $\rho =0.6,0.8$ for each level of $Z$. Table 1 summarizes the type I error rates of the regression-based test and the Hotelling $T^2$ test under various configurations of variance estimation, number of levels of the covariate, and sample sizes. As expected, for relatively large sample sizes, the two test approaches control the type I error quite satisfactorily at the nominal level of 0.05 using either the approximate or bootstrap variance, that is, ${\hat{\sigma }}_k^2$ or ${\hat{\sigma }}_{\mathit{bootstrap},k}^2$. For a relatively small sample size (such as $n=30$), however, using the approximate variance yield inflated type I error rates in which case the bootstrap variance is preferred, especially when $K$ is relatively large. Partly because the two proposed test statistics, $T_0$ and $T^2$, involve the estimation of the $2K$ unknown parameters, $\left({\theta }_k,{\sigma }_k^2\right)$ s, the smaller $n$ and the larger $K$, the more their distributions of $T_0$ and $T^2$ deviate from their asymptotic distributions under the null hypothesis $H_0:{\theta }_1=\cdots ={\theta }_K$. Furthermore, the type I error rates of the two tests seem to be less affected by the value of $\rho$.

TABLE 1.

Comparison of type I error rates of the regression-based test and the Hotelling T² test (runs:1000).

n	Test	Variance	The number of levels of Z
			K = 2		K = 3		K = 4		K = 5
			${\sigma }_X^2={\sigma }_Y^2$	${\sigma }_X^2\ne {\sigma }_Y^2$	${\sigma }_X^2={\sigma }_Y^2$	${\sigma }_X^2\ne {\sigma }_Y^2$	${\sigma }_X^2={\sigma }_Y^2$	${\sigma }_X^2\ne {\sigma }_Y^2$	${\sigma }_X^2={\sigma }_Y^2$	${\sigma }_X^2\ne {\sigma }_Y^2$
${\rho }_1=\cdots ={\rho }_K=\rho =0.6$
30	Regression	Approximate	0.072	0.089	0.081	0.084	0.084	0.105	0.092	0.124
		Bootstrapping	0.042	0.054	0.046	0.046	0.049	0.058	0.055	0.066
	Hotelling	Approximate	0.070	0.083	0.091	0.094	0.111	0.120	0.149	0.137
		Bootstrapping	0.040	0.053	0.042	0.045	0.041	0.059	0.065	0.064
60	Regression	Approximate	0.070	0.062	0.081	0.078	0.078	0.051	0.059	0.061
		Bootstrapping	0.057	0.040	0.063	0.057	0.057	0.039	0.042	0.048
	Hotelling	Approximate	0.072	0.061	0.067	0.075	0.092	0.062	0.078	0.077
		Bootstrapping	0.054	0.040	0.039	0.058	0.065	0.042	0.051	0.047
100	Regression	Approximate	0.056	0.067	0.051	0.042	0.063	0.067	0.075	0.064
		Bootstrapping	0.048	0.056	0.046	0.036	0.050	0.054	0.060	0.054
	Hotelling	Approximate	0.057	0.066	0.066	0.048	0.054	0.052	0.077	0.065
		Bootstrapping	0.048	0.053	0.052	0.032	0.036	0.039	0.057	0.050
150	Regression	Approximate	0.049	0.057	0.062	0.057	0.056	0.052	0.052	0.059
		Bootstrapping	0.045	0.050	0.058	0.049	0.048	0.045	0.047	0.056
	Hotelling	Approximate	0.049	0.055	0.058	0.046	0.054	0.052	0.066	0.072
		Bootstrapping	0.044	0.048	0.050	0.040	0.047	0.041	0.055	0.055
${\rho }_1=\cdots ={\rho }_K=\rho =0.8$
30	Regression	Approximate	0.079	0.091	0.083	0.086	0.106	0.111	0.109	0.127
		Bootstrapping	0.039	0.050	0.044	0.038	0.047	0.053	0.045	0.062
	Hotelling	Approximate	0.071	0.077	0.098	0.087	0.105	0.124	0.138	0.138
		Bootstrapping	0.033	0.039	0.030	0.035	0.030	0.042	0.037	0.042
60	Regression	Approximate	0.071	0.061	0.081	0.089	0.071	0.060	0.070	0.067
		Bootstrapping	0.048	0.046	0.059	0.062	0.049	0.036	0.046	0.042
	Hotelling	Approximate	0.068	0.058	0.060	0.072	0.082	0.067	0.088	0.070
		Bootstrapping	0.042	0.038	0.033	0.040	0.044	0.029	0.042	0.038
100	Regression	Approximate	0.048	0.049	0.053	0.064	0.062	0.053	0.062	0.082
		Bootstrapping	0.043	0.045	0.042	0.049	0.039	0.045	0.048	0.042
	Hotelling	Approximate	0.045	0.045	0.049	0.064	0.056	0.052	0.064	0.069
		Bootstrapping	0.035	0.037	0.040	0.044	0.046	0.046	0.034	0.030
150	Regression	Approximate	0.049	0.049	0.050	0.059	0.070	0.054	0.056	0.058
		Bootstrapping	0.045	0.056	0.061	0.060	0.055	0.051	0.037	0.047
	Hotelling	Approximate	0.047	0.048	0.056	0.065	0.070	0.061	0.067	0.074
		Bootstrapping	0.040	0.052	0.047	0.049	0.043	0.044	0.037	0.043

Note: n is the sample size in each level of Z.

3.1.2 |. Power analysis

Simulations were conducted to estimate the empirical power of the tests under various configurations of the alternative hypothesis $H_1$: not all ${\theta }_k$s are equal. Under bivariate normal assumption, we considered three settings for generating the covariate dependent correlation coefficient ${\rho }_k$. The first setting is to exactly meet the assumptions of the regression analysis, in which ${\mathrm{\Phi }}^{-1}\left({\theta }_k\right)$ increases by the same amount as the level of $Z$ is increased by one unit. To generate data for this setting, we fixed $\mathit{b}={\left(b_0,b_1\right)}^T$ and solved for ${\rho }_k$s from equations (15) and ${\mathrm{\Phi }}^{-1}\left({\theta }_k\right)=b_0+b_1{z}_k$, that is,

$${\rho }_k=\mathrm{s}\mathrm{i}\mathrm{n}\left(\left(\mathrm{\Phi }\left(b_0+b_1{z}_k\right)-0.5\right)/\pi \right),$$

where we take $\mathit{b}=(0.6,0.2{)}^T$ and $z_k=k$ for $k=1,\dots ,K$, then $\left({\rho }_1,\dots ,{\rho }_5\right)=(0.786,0.878,0.935,0.968,0.985{)}^T$. The second setting departs slightly from the first setting in that the logit link function $\mathrm{l}\mathrm{o}\mathrm{g}(\theta /(1+\theta ))$ was used instead of the probit link function ${\mathrm{\Phi }}^{-1}\left({\theta }_k\right)$ to generate the data. From the Equations (15) and $\mathrm{l}\mathrm{o}\mathrm{g}\left({\theta }_k/\left(1+{\theta }_k\right)\right)=b_0+b_1{z}_k$, we can derive ${\rho }_k=\mathrm{s}\mathrm{i}\mathrm{n}\left(\left({\theta }_k-0.5\right)/\pi \right)$ with ${\theta }_k=\mathrm{e}\mathrm{x}\mathrm{p}\left\{b_0+b_1{z}_k\right\}/\left(1-\mathrm{e}\mathrm{x}\mathrm{p}\left\{b_0+b_1{z}_k\right\}\right)$, so we obtain $\left({\rho }_1,\dots ,{\rho }_5\right)=(0.562,0.664,0.747,0.813,0.864{)}^T$ when $\mathit{b}=(0.6,0.2{)}^T$ and $z_k=k$ for $k=1,\dots ,K$. The third setting is more general in that the ${\theta }_k$s are different but do not follow any functional trend as the covariate varies, where ${\theta }_k$ is computed from Equations (15) in which we take $\left({\rho }_1,\dots ,{\rho }_5\right)=(0.80,0.90,0.85,0.70,0.75{)}^T$. For all the three settings, the number of levels of the covariate and the sample sizes were the same as the set-up for type I error simulations. Both equal and unequal variances of $X$ and $Y$ were considered. The equal variances case occurs more likely when $X$ and $Y$ measure the same variable, for example, when $Y$ is the gold standard and $X$ is an imperfect instrument; the unequal variances case occurs more likely otherwise.

The simulated powers for the three settings are presented in Tables 2,3, and 4, respectively. As expected, power increases with sample size. The power obtained with the approximate variance appears to be larger than the power obtained with the bootstrap variance. Combining Table 1, we suggest that the approximate variance is reliable for the relatively large sample size. For small sample sizes, however, the bootstrap variance should be used due to the inflation of type I error rates with the approximate variance. It appears that the regression method is more powerful than the Hotelling’s test in monotone settings 1 and 2, even when the regression model assumptions are violated (in settings 2). However, In setting 3, the Hotelling’s $T^2$ test performs significantly better than regression method except the case $K=2$, just as shown in Table 4. Tables 1,2,3, and 4 all show that the Hotelling’s $T^2$ test performs nearly as well as the regression method may when $K=2$ and $n$ is relatively large, which confirms Remark 1 in Section 2. Thus the regression method is preferred if the ${\theta }_k$ s do follow some monotone functional trend as the covariate varies, otherwise, Hotelling’s $T^2$ is chosen. For a given sample size $n$, we can also find from Tables 2 and 3 that the powers of the two tests increase with $K$ in the monotone settings 1 and 2. The reason for this is that a larger $K$ results in the more significant difference between ${\theta }_k$s. For a relatively small $n$, a larger $K$ also results in an inflated type I error rate, which will increase the powers of the two tests based on the approximate variances.

TABLE 2.

Comparison of powers of the regression-based test and the Hotelling T² test under the first setting (runs:1000).

n	Test	Variance	The number of levels of Z
			K = 2		K = 3		K = 4		K = 5
			${\sigma }_X^2={\sigma }_Y^2$	${\sigma }_X^2\ne {\sigma }_Y^2$	${\sigma }_X^2={\sigma }_Y^2$	${\sigma }_X^2\ne {\sigma }_Y^2$	${\sigma }_X^2={\sigma }_Y^2$	${\sigma }_X^2\ne {\sigma }_Y^2$	${\sigma }_X^2={\sigma }_Y^2$	${\sigma }_X^2\ne {\sigma }_Y^2$
30	Regression	Approximate	0.223	0.223	0.566	0.590	0.908	0.919	0.998	0.995
		Bootstrapping	0.139	0.131	0.432	0.454	0.854	0.863	0.996	0.988
	Hotelling	Approximate	0.207	0.200	0.460	0.479	0.823	0.837	0.985	0.982
		Bootstrapping	0.118	0.105	0.252	0.272	0.601	0.625	0.897	0.884
60	Regression	Approximate	0.348	0.342	0.864	0.890	0.999	1.000	1.000	1.000
		Bootstrapping	0.290	0.297	0.819	0.845	0.998	1.000	1.000	1.000
	Hotelling	Approximate	0.339	0.335	0.789	0.795	0.991	0.998	1.000	1.000
		Bootstrapping	0.266	0.267	0.672	0.688	0.970	0.987	1.000	1.000
100	Regression	Approximate	0.527	0.521	0.974	0.981	1.000	1.000	1.000	1.000
		Bootstrapping	0.489	0.468	0.972	0.975	1.000	1.000	1.000	1.000
	Hotelling	Approximate	0.524	0.514	0.956	0.960	1.000	1.000	1.000	1.000
		Bootstrapping	0.470	0.448	0.929	0.937	1.000	1.000	1.000	1.000
150	Regression	Approximate	0.716	0.720	1.000	1.000	1.000	1.000	1.000	1.000
		Bootstrapping	0.678	0.690	1.000	1.000	1.000	1.000	1.000	1.000
	Hotelling	Approximate	0.715	0.709	0.990	1.000	1.000	1.000	1.000	1.000
		Bootstrapping	0.664	0.658	0.985	0.988	1.000	1.000	1.000	1.000

Note: The data were generated under the assumptions of regression analysis with a probit link function, in which ${\mathrm{\Phi }}^{-1}\left({\theta }_k\right)$ increases by the same amount as the level of Z is increased by one unit.

TABLE 3.

Comparison of powers of the regression-based test and the Hotelling T² test under the second setting (runs:1000).

n	Test	Variance	The number of levels of Z
			K = 2		K = 3		K = 4		K = 5
			${\sigma }_X^2={\sigma }_Y^2$	${\sigma }_X^2\ne {\sigma }_Y^2$	${\sigma }_X^2={\sigma }_Y^2$	${\sigma }_X^2\ne {\sigma }_Y^2$	${\sigma }_X^2={\sigma }_Y^2$	${\sigma }_X^2\ne {\sigma }_Y^2$	${\sigma }_X^2={\sigma }_Y^2$	${\sigma }_X^2\ne {\sigma }_Y^2$
30	Regression	Approximate	0.114	0.114	0.288	0.256	0.510	0.511	0.765	0.773
		Bootstrapping	0.074	0.075	0.199	0.188	0.402	0.415	0.679	0.687
	Hotelling	Approximate	0.112	0.109	0.231	0.240	0.390	0.415	0.608	0.636
		Bootstrapping	0.069	0.072	0.146	0.144	0.228	0.240	0.401	0.428
60	Regression	Approximate	0.151	0.144	0.383	0.415	0.761	0.796	0.959	0.956
		Bootstrapping	0.119	0.120	0.330	0.363	0.721	0.743	0.948	0.938
	Hotelling	Approximate	0.149	0.141	0.325	0.336	0.620	0.639	0.886	0.859
		Bootstrapping	0.120	0.117	0.257	0.274	0.538	0.559	0.821	0.784
100	Regression	Approximate	0.210	0.224	0.598	0.614	0.955	0.950	1.000	1.000
		Bootstrapping	0.183	0.191	0.565	0.569	0.935	0.940	1.000	1.000
	Hotelling	Approximate	0.208	0.221	0.534	0.518	0.850	0.880	1.000	1.000
		Bootstrapping	0.176	0.186	0.463	0.452	0.790	0.810	0.985	1.000
150	Regression	Approximate	0.225	0.235	0.770	0.782	0.980	0.982	1.000	1.000
		Bootstrapping	0.190	0.200	0.760	0.768	0.980	0.983	1.000	1.000
	Hotelling	Approximate	0.225	0.230	0.705	0.712	0.965	0.965	1.000	1.000
		Bootstrapping	0.190	0.200	0.650	0.635	0.950	0.960	1.000	1.000

Note: The data were generated under the assumptions of regression analysis with a logit link function, in which $\log \left({\theta }_k=\left(1+{\theta }_k\right)\right)$ increases by the same amount as the level of Z is increased by one unit.

TABLE 4.

Comparison of powers of the regression-based test and the Hotelling T² test under the third setting (runs:1000).

n	Test	Variance	The number of levels of Z
			K = 2		K = 3		K = 4		K = 5
			${\sigma }_X^2={\sigma }_Y^2$	${\sigma }_X^2\ne {\sigma }_Y^2$	${\sigma }_X^2={\sigma }_Y^2$	${\sigma }_X^2\ne {\sigma }_Y^2$	${\sigma }_X^2={\sigma }_Y^2$	${\sigma }_X^2\ne {\sigma }_Y^2$	${\sigma }_X^2={\sigma }_Y^2$	${\sigma }_X^2\ne {\sigma }_Y^2$
30	Regression	Approximate	0.300	0.272	0.121	0.103	0.230	0.197	0.326	0.342
		Bootstrapping	0.186	0.188	0.059	0.054	0.140	0.126	0.205	0.223
	Hotelling	Approximate	0.266	0.257	0.234	0.248	0.497	0.484	0.534	0.539
		Bootstrapping	0.156	0.155	0.110	0.104	0.279	0.279	0.273	0.302
60	Regression	Approximate	0.457	0.460	0.126	0.113	0.305	0.303	0.516	0.510
		Bootstrapping	0.381	0.387	0.091	0.094	0.238	0.234	0.452	0.436
	Hotelling	Approximate	0.443	0.449	0.386	0.382	0.741	0.729	0.780	0.785
		Bootstrapping	0.347	0.360	0.274	0.296	0.630	0.621	0.668	0.687
100	Regression	Approximate	0.688	0.644	0.162	0.172	0.458	0.451	0.737	0.705
		Bootstrapping	0.642	0.601	0.139	0.148	0.417	0.410	0.687	0.673
	Hotelling	Approximate	0.683	0.634	0.587	0.578	0.934	0.943	0.959	0.961
		Bootstrapping	0.626	0.586	0.516	0.514	0.902	0.909	0.934	0.939
150	Regression	Approximate	0.855	0.839	0.213	0.198	0.600	0.611	0.870	0.888
		Bootstrapping	0.831	0.822	0.204	0.174	0.585	0.595	0.848	0.866
	Hotelling	Approximate	0.852	0.836	0.759	0.745	0.986	0.993	0.998	0.996
		Bootstrapping	0.824	0.816	0.715	0.706	0.986	0.990	0.998	0.992

Note: The data were generated more generally, in which ${\theta }_k$s are different but do not follow any functional trend as the covariate Z varies.

3.2 |. A real example

We used data from a study that assessed three types of imaging modalities: barium enema (BE), computed tomographic colonography (CTC) and colonoscopy (CO) to detect neoplastic colon polyps and cancers. Colonography ($Y$) is considered the gold standard for detecting neoplastic polyps or colon cancer. In this study, 614 patients completed all the three modalities. For the purpose of our example, we used data from the computed tomographic colonography $(X)$ and colonography modalities. There were 273 patients who had neoplastic lesions detected by either computed tomographic colonography and colonography. The median size of the largest lesion detected by colonography was 8 mm (range = 6–70), whereas the size of the lesion detected by computed tomographic colonography in the same area of the colon was 7 mm (range = 3–70). Due to the potential biological differences in race and family history, our interest was to assess the diagnostic accuracies of the sizes of the lesion sizes detected by the computed tomographic colonography and colonography according to the patient’s family history and race. The null hypothesis to be tested was that the diagnostic accuracy was the same across the four combination of family history and race, that is, ${\theta }_1={\theta }_2={\theta }_3={\theta }_4$. However, because there were only five patients with family history of colon cancer and were from other racial groups, this group was deleted from the analysis due to the small numbers.

Let the class variable $Z=1,3,4$ indicate that the patient is of ‘White/no family history’, ‘Other race /no family history’ and ‘White/family history’. And in the regression analysis, we adopt the probit-link model

$${\mathrm{\Phi }}^{-1}\left({\theta }_k\right)=b_0+b_1{z}_k,z_1=1,z_2=3,z_3=4.$$

Then, the null hypothesis was tested by both the regression statistic $T_0$ given by (11) and the Hotelling $T^2$ statistic given by (12) with approximate (bootstrapping) variance, where the approximate variance of ${\hat{\theta }}_k$ was defined by (6), and its bootstrapping variance was estimated by bootstrapping with replacement from $\left\{\left(X_i,Y_i\right)\mid Z_i=Z_k\right\}$s. The diagnostic accuracy of the size detected by the computed tomographic colonography $(X)$ and colonography $(Y)$ by family history and race $(Z)$ is presented in Table 5. Table 5 shows that there were no statistical differences in accuracies across the different groups of family history and racial groups using either the regression analysis approach or the Hotelling $T^2$. We conclude that there were no differences in the diagnostic accuracies of computed tomographic colonography and colonography among the three groups of family history and race.

TABLE 5.

Comparison of accuracies among races by family history.

Race by family history	$\hat{\theta }$	$\sqrt{\widehat{\mathrm{V}\mathrm{a}\mathrm{r}(\hat{\theta })}}$	Regression analysis	Hotelling T2 Test
White/No family history	0.877	0.017(0.017)	${\hat{b}}_1=-0.042(-0.043)$	χ2 = 1.517(1.382)
Other race/No family history	0.887	0.013(0.015)	$\sqrt{\widehat{\mathrm{V}\mathrm{a}\mathrm{r}}({\hat{b}}_1)}=0.066(0.066)$	P-value=0.468(0.501)
White/Family history	0.853	0.0245(0.0247)	P-value=0.525(0.509)

Note: The numbers before and in (⋅) mean the corresponding values calculated by approximate method and bootstrapping.

4 |. DISCUSSION

In this article, we developed two methods, regression analysis and Hotelling’s $T^2$, to investigate the effect of covariates on the diagnostic accuracy of a continuous biomarker when the gold standard is also measured on a continuous scale. Approximate variance estimates were derived for inference on the accuracy index and were compared with variance estimates using bootstrap. Simulation studies indicate that the two methods with either variance estimates control type error rates when sample sizes are relatively large. For small sample sizes, the bootstrap variance estimates are preferred. The regression analysis, even if the model is slightly misspecified, performs more powerful in monotone setting, otherwise, the Hoteling’s $T^2$ is preferred.

The regression analysis approach focuses on a discrete covariate but the method can be readily extended to deal with a continuous covariate. Alternatively, one can discretize the continuous covariate by binning, just like Metz et al²⁰ and Zhou and Lin²¹ categorized continuous data for ROC curve, and use the regression analysis directly or the Hoteling’s $T^2$ developed in the article to test the null hypothesis of no difference in accuracy. Caution needs to be given if the test fails to reject the null hypothesis since the p-value from the test may depend on how the continuous covariate is discretized. However, it is still an challenging issue to how to find better category boundaries for the continuous covariate in Equation (1) and needs further study.

APPENDIX A. ASYMPTOTICS

Since $Z$ is a categorical covariate and samples at different covariate levels are assumed to be independent, the derivation of the asymptotic distribution of ${\hat{\theta }}_k$ is the same with the case with no covariates.

We will derive the asymptotic distribution of the empirical estimator of the accuracy index, which is a $U$-statistic for continuous probability distribution in 2-dimension with a symmetric kernel

$$h\left(\left(X_1,Y_1\right),\left(X_2,Y_2\right)\right)=\mathrm{\Psi }\left(\left(X_1-X_2\right)\left(Y_1-Y_2\right)\right),$$ (A1)

where $\mathrm{\Psi }(t)=1$ if $\left(X_1-X_2\right)\left(Y_1-Y_2\right)>0,0.5$ if $X_1=X_2$ and $Y_1=Y_2$, otherwise 0. It is easy to prove that

$$E\left(h\left(\left(X_1,Y_1\right),\left(X_2,Y_2\right)\right)\right)=\theta ,\mathrm{V}\mathrm{a}\mathrm{r}\left(h\left(\left(X_1,Y_1\right),\left(X_2,Y_2\right)\right)\right)=\theta -{\theta }^2.$$ (A2)

We first introduce the theoretical result on $U$-statistic from Hoeffding.²²

Lemma 1.

Let $v$ be an even integer. If $E\left(H^v\right)<\mathrm{\infty }$ (implied by $E\left(h^v\right)<\mathrm{\infty }$), then

$$E{(U_n-{\hat{U}}_n)}^v=O\left(n^{-v}\right),n\to \mathrm{\infty },$$

where $U_n$ is a $U$-statistic with kernel $h,{\hat{U}}_n$ is the corresponding projection of $U_n$, and $H$ is the symmetric kernel of their difference.

The observations $\left(X_i,Y_i\right)$ s are i.i.d. with a common distribution $F(x,y)$. Define ${\tilde{h}}_1\left(x_1,y_1\right)=h_1\left(x_1,y_1\right)-\theta$ with

$$\begin{array}{ll}{h}_1\left(\left(x_1,y_1\right)\right)& =E\left(h\left(\left(x_1,y_1\right),\left(X_2,Y_2\right)\right)\right)=P\left(\left(x_1-X_2\right)\left(y_1-Y_2\right)>0\right)\\ & =P\left(x_1<X_2,y_1<Y_2\right)+P\left(x_1>X_2,y_1>Y_2\right)\\ & =1-F_X\left(x_1\right)-F_Y\left(y_1\right)+2F\left(x_1,y_1\right),\end{array}$$

and denote $p=P\left\{\left(Y_1-Y_2\right)\left(X_1-X_2\right)>0,\left(Y_1-Y_3\right)\left(X_1-X_3\right)>0\right\}$.

Note that

$$E\left({\tilde{h}}_1\left(X_1,Y_1\right)\right)=E\left(h\left(\left(X_1,Y_1\right),\left(X_2,Y_2\right)\right)\right)-\theta =0,$$ (A3)

$$\begin{array}{ll}\mathrm{V}\mathrm{a}\mathrm{r}\left({\tilde{h}}_1\left(X_1,Y_1\right)\right)& =\mathrm{V}\mathrm{a}\mathrm{r}\left(h_1\left(\left(X_1,Y_1\right)\right)\right)\\ & =\mathrm{C}\mathrm{o}\mathrm{v}\left(\mathrm{\Psi }\left(\left(Y_1-Y_2\right)\left(X_1-X_2\right)\right),\mathrm{\Psi }\left(\left(Y_1-Y_3\right)\left(X_1-X_3\right)\right)\right)\\ & =P\left(\left(Y_1-Y_2\right)\left(X_1-X_2\right)>0,\left(Y_1-Y_3\right)\left(X_1-X_3\right)>0\right)-{\theta }^2\\ & =p-{\theta }^2=:\zeta .\end{array}$$ (A4)

We define a projection of ${\hat{\theta }}^{\mathrm{*}}$ of $\hat{\theta }$,

$${\hat{\theta }}^{\mathrm{*}}=\sum _{i=1}^{n}E\left(\hat{\theta }\mid \left(X_i,Y_i\right)\right)-\left(n-1\right)\theta .$$ (A5)

In terms of the function ${\tilde{h}}_1$, we have

$${\hat{\theta }}^{\mathrm{*}}-\theta =\frac{2}{n}\left(\sum _{i=1}^n{\tilde{h}}_1\left(X_i,Y_i\right)\right).$$

Since $2{\tilde{h}}_1\left(X_i,Y_i\right)$s are i.i.d with mean 0 and variance $4\zeta$, it follows that $\sqrt{n}({\hat{\theta }}^{\mathrm{*}}-\theta )\stackrel{ℒ}{\to }N(0,4\zeta )$ by the central limit theorem, where $\stackrel{ℒ}{\to }$ means ‘converge in distribution’. To prove that the asymptotic distribution of $\hat{\theta }$ are the same as ${\hat{\theta }}^{\mathrm{*}}$, we need to show that $\sqrt{n}(\hat{\theta }-\theta )$ and $\sqrt{n}({\hat{\theta }}^{\mathrm{*}}-\theta )$ are asymptotically equivalent. It is sufficient to show that $nE{(\hat{\theta }-{\hat{\theta }}^{\mathrm{*}})}^2=0$.

Note that

$$nE{(\hat{\theta }-{\hat{\theta }}^{\mathrm{*}})}^2=n\mathrm{V}\mathrm{a}\mathrm{r}(\hat{\theta }-{\hat{\theta }}^{\mathrm{*}})$$

The difference $\hat{\theta }-{\hat{\theta }}^{\mathrm{*}}$ may itself be expressed as a $U$-statistic,

$$\hat{\theta }-{\hat{\theta }}^{\mathrm{*}}=\frac{2}{n(n-1)}\sum _{1\le i<j\le n}H\left(\left(X_i,Y_i\right),\left(X_j,Y_j\right)\right)$$

based on the symmetric kernel

$$H\left(\left(x_i,y_i\right),\left(x_j,y_j\right)\right)=h\left(\left(x_i,y_i\right),\left(x_j,y_j\right)\right)-{\tilde{h}}_1\left(x_i,y_i\right)-{\tilde{h}}_1\left(x_j,y_j\right)-\theta .$$

Note that $E\left[h\left(\left(X_i,Y_i\right),\left(X_j,Y_j\right)\right)\right]=\theta ,E\left({\tilde{h}}_1\left(X_i,Y_i\right)\right)=E\left({\tilde{h}}_1\left(X_j,Y_j\right)\right)=0$,

$$\begin{array}{ll}E\left[\left(h\left(\left(X_i,Y_i\right),\left(X_j,Y_j\right)\right)-\theta \right){\tilde{h}}_1\left(X_i,Y_i\right)\right]& \left.=E\left[E\left(\left(h\left(\left(X_i,Y_i\right),\left(X_j,Y_j\right)\right)-\theta \right)\right){\tilde{h}}_1\left(X_i,Y_i\right)\right)\mid \left(X_i,Y_i\right)\right]\\ & =E[{\tilde{h}}_1^2\left(X_i,Y_i\right)]=\mathrm{V}\mathrm{a}\mathrm{r}\left({\tilde{h}}_1\left(X_i,Y_i\right)\right)=\zeta ,\end{array}$$

${\tilde{h}}_1\left(X_i,Y_i\right)$ and ${\tilde{h}}_1\left(X_i,Y_i\right)$ are independent and identically distributed. We have $E(H)=0$ and

$$\begin{array}{ll}E\left(H^2\right)& =\mathrm{V}\mathrm{a}\mathrm{r}\left(h\left(\left(X_i,Y_i\right),\left(X_j,Y_j\right)\right)\right)+E{\left({\tilde{h}}_1\left(X_i,Y_i\right)+{\tilde{h}}_1\left(X_j,Y_j\right)\right)}^2\\ & -2E\left(h\left(\left(X_i,Y_i\right),\left(X_j,Y_j\right)\right)-\theta \right)\left({\tilde{h}}_1\left(X_i,Y_i\right)+{\tilde{h}}_1\left(X_j,Y_j\right)\right)\\ & =\mathrm{V}\mathrm{a}\mathrm{r}\left(h\left(\left(X_i,Y_i\right),\left(X_j,Y_j\right)\right)\right)-\mathrm{V}\mathrm{a}\mathrm{r}\left({\tilde{h}}_1\left(X_i,Y_i\right)\right)-\mathrm{V}\mathrm{a}\mathrm{r}\left({\tilde{h}}_1\left(X_j,Y_j\right)\right)\\ & =\theta +{\theta }^2-2p<\mathrm{\infty }.\end{array}$$

According to Lemma 1,

$$nE{({\hat{\theta }}^{\mathrm{*}}-\hat{\theta })}^2=nO\left(n^{-2}\right)=O\left(n^{-1}\right),n\to \mathrm{\infty },$$

Therefore,

$$\sqrt{n}(\hat{\theta }-\theta )\stackrel{ℒ}{\to }N(0,4\zeta ),$$

Note that samples at different covariate levels are assumed to be independent. Thus, ${\hat{\theta }}_1,\dots ,{\hat{\theta }}_K$ are mutually independent, each is asymptotically normally distributed, that is,

$$\sqrt{n_k}({\hat{\theta }}_k-{\theta }_k)\stackrel{ℒ}{\to }N\left(\mathrm{0,4}{\zeta }_k\right),$$ (A6)

where ${\zeta }_k=p_k-{\theta }_k^2$. According to the delta-method, $G({\hat{\theta }}_k)$ are also mutually independent, the approximate distribution of each is

$$\sqrt{n_k}\left(G({\hat{\theta }}_k)-G\left({\theta }_k\right)\right)\stackrel{ℒ}{\to }N\left(0,4g^2\left({\theta }_k\right){\zeta }_k\right).$$ (A7)

Note that the true variance of ${\hat{\theta }}_k$ is ${\sigma }_k^2$ given in (5) and ${\mathrm{l}\mathrm{i}\mathrm{m}}_{n_k\to \mathrm{\infty }}{n}_k{\sigma }_k^2=4{\zeta }_k$. Therefore, when $n_k$ is not enough large, replacing $4{\zeta }_k$ with $n_k{\sigma }_k^2$ in (A6) and (A7) obtains better approximate distributions of ${\hat{\theta }}_k$ and $G({\hat{\theta }}_k)$.