Optimal sampling ratios in comparative diagnostic trials

Summary

A subjective sampling ratio between the case and the control groups is not always an efficient choice to maximize the power or to minimize the total required sample size in comparative diagnostic trials.We derive explicit expressions for an optimal sampling ratio based on a common variance structure shared by several existing summary statistics of the receiver operating characteristic curve. We propose a two-stage procedure to estimate adaptively the optimal ratio without pilot data. We investigate the properties of the proposed method through theoretical proofs, extensive simulation studies and a real example in cancer diagnostic studies.

1. Introduction

Diagnostic trials evaluate the diagnostic accuracy of a marker or compare the diagnostic accuracy of two markers. For example, in a diagnostic trial by Hendrick et al. (2008), investigators compared the accuracy of digital mammography with screen film mammography. Pepe et al. (2001) referred to these trials as phase III diagnostic trials. In these trials, the true disease status of subjects is known. To evaluate the diagnostic accuracy of a binary marker, sensitivity and specificity are used. Sensitivity is the probability of having a positive test result for a case subject. Specificity is the probability of having a negative test result for a control subject. The false positive rate (FPR) is 1– specificity. For continuous markers, we obtain the sensitivity and FPR on the basis of a threshold that distinguishes the test result as being positive or negative. Varying thresholds allow a number of sensitivities and FPRs to be computed simultaneously. The receiver operating characteristic (ROC) curve plots sensitivities against FPRs for all thresholds (Zhou et al., 1998, 2002).

Typically the ratio between the number of cases versus the number of controls is fixed in advance. A lung cancer prevention trial used an equal ratio with 71 prostate cancer cases and 71 age-matched controls (Etzioni et al., 1999). Some studies use other case–control ratios; for example, the controls who were enrolled for prostate cancer screening were four times as many as the cases in the Physicians Health Study (Etzioni et al., 2003). In a cancer diagnostic trial from Goddard and Hinberg (1990), 135 cancer patients and 218 non-cancer patients were recruited. A traditional biomarker A and newly developed diagnostic biomarkers were used to test blood samples from each subject. The power for comparing biomarkers A and D is below 45% by using the original sampling ratio of 0.62. Thus, these ratios may not be the best choice to maximize the test power, and optimal sampling ratios need to be derived and utilized to improve the power.

Janes and Pepe (2006) provided the optimal ratio for evaluating a continuous marker to maximize the power for a fixed total sample size (SS). Their method is the first attempt to address the optimal sampling ratio in diagnostic trials. However, pilot data are required to estimate the optimal ratio. Without pilot data, some distributions must be assumed to perform the calculations. An optimal ratio from an incorrect distributional assumption may lead to an underpowered study. It is desirable to recalculate the optimal ratio when data become available during the trial. In addition, optimal ratios for comparative diagnostic trials have not been discussed in the literature.

We make the following methodological contributions in this paper:

1. a design to update the optimal ratio for evaluating a single marker with questionable parametric assumptions;

2. extension of Janes and Pepe (2006) to two continuous markers and ordinal markers;

3. a design to update the optimal ratio for comparing two markers and rigorous proof of its properties.

In this paper we first derive a general expression for the optimal sampling ratio of cases to controls in diagnostic trials. The ratio proposed is based on a common variance structure that is shared among existing ROC summary statistics. Special cases of these statistics include the non-parametric area under the ROC curve statistic AUC that was proposed by DeLong et al. (1988) and the weighted AUC-statistic by Wieand et al. (1989). The method proposed can be used in evaluating one marker or comparing two markers. The rest of the paper is organized as follows. In Section 2, we start with the optimal ratios for diagnostic trials. In Section 3, we propose a two-stage method to incorporate the idea of internal pilot data to estimate adaptively the optimal sampling ratio. The method can be applied in trials that evaluate one marker or compare two markers. We show that, although the optimal ratio is updated during a diagnostic trial, the analysis at the end of the trial can be carried out in the same fashion as in the traditional trial without affecting the nominal type I error rate. Section 4 illustrates the increase in power and the savings on the overall required SS by using the proposed method through a cancer example. Section 5 investigates benefits of the proposed procedure through extensive simulation studies. Some discussion is provided in Section 6.

The data that are analysed in the paper can be obtained from http://wileyonlinelibrary.com/journal/rss-datasets

2. Optimal sampling ratio

Suppose that we have N subjects with m cases and n controls. Each subject is measured by diagnostic test l (l = 1, 2). We define the ith case as X_li, where i = 1, . . . , m, and the jth control as Y_lj, where j = 1, . . . , n. The joint cumulative survival functions for cases are (X_1i, X_2i) ~ S_d(x₁, x₂) and the joint cumulative survival functions for controls are $(Y_{1j},Y_{2j})\sim S_{\stackrel{‒}{d}}(y_1,y_2)$. Their marginal survival distributions are X_li ~ S_d,l and $Y_{lj}\sim S_{\stackrel{‒}{d},l}\left(y\right)$ respectively. For the threshold c varying in (−∞, ∞), the sensitivity is S_d,l and the FPR is $S_{\stackrel{‒}{d},l}\left(c\right)=\mathrm{Pr}(Y_{lj}>c)$. Consequently, the ROC curve for test l is defined as $R_l\left(u\right)=S_{d,l}\left\{S_{\stackrel{‒}{d},l}^{-1}\left(u\right)\right\}$, where the FPR u falls within [0,1].

Summary measures for a single ROC curve include the area under the ROC curve, AUC, the partial AUC, pAUC, and the weighted AUC, wAUC. wAUC for marker l, ${\Omega }_l={\int }_0^1{R}_l\left(u\right)\mathrm{d}W\left(u\right)$, was given by Wieand et al. (1989) where W(u) is a probability measure. We let W(u) be a point u₀, an FPR, to calculate the sensitivity of a test, or W(u) = u, where u ∈(0, 1), to obtain AUC. When W(u) = (u – u₀)/u₁ – u₀), where u ∈ (u₀, u₁), Ω_l gives the partial AUC.

The statistics for comparing markers might be parametric (Mazumdar and Liu, 2003), or non-parametric (DeLong et al., 1988; Wieand et al., 1989). Let θ be the parameter in the ROC comparison, and $\widehat{\theta }$ be the estimator. On the basis of the variance expressions for these ROC statistics, we identify the following common structure for the variance of the aforementioned ROC statistics when the sample sizes become large:

$$\mathrm{var}\left(\widehat{\theta }\right)=\frac{v_x}{m}+\frac{v_y}{n},$$ (1)

where v_x is the variance associated with measurements of case patients and v_y is the variance related to control patients. In this paper we use the non-parametric statistics of DeLong et al. (1988) and Wieand et al. (1989). We present the variance expressions for these statistics in Sections 2.1.2 and 2.2. A similar variance structure for a conventional binormal ROC statistic of Mazumdar and Liu (2003) is presented in Appendix A.

Given the variance structure in equation (1), the total required SS in a diagnostic trial can be minimized by using an optimal sampling ratio when the variance is fixed. In other words, the power for comparing two markers can be maximized by using this optimal sampling ratio. Suppose that the total required SS in the diagnostic trial is N = m + n; the sampling ratio is r = m/n. Let the variance of $\widehat{\theta }$ be a fixed constant, a. Since m = rn = Nr(1 + r), it follows that

$$\frac{v_x}{m}+\frac{v_y}{n}=\frac{(1+r)(v_x∕r+v_y)}{N}=a.$$

The total required SS can then be expressed as

$$N=\frac{(1+r)(v_x∕r+v_y)}{a}.$$

To minimize N, we take the first derivative with respect to r and equate it to 0. We obtain v_y/a – v_x/ar^–2=0.By solving this equation, the optimal sampling ratio is obtained as

$$r^{\ast }=\surd (v_x∕v_y).$$ (2)

It is worth noting that the optimal sampling ratio is analogous to the Neyman allocation ratio for clinical trials which has been widely used to reduce the overall SS for a fixed power. However, as will be seen from Sections 2.1 and 2.2, v_x and v_y in diagnostic trials take more complicated forms than those used in clinical trials which are commonly the variances of response variables in treatment and control groups. Interested readers are refered to Jennison and Turnbull (2000) and Rosenberger and Lachin (2002).

2.1. Optimal sampling ratio for continuous markers

The difference Δ=Ω₁ –Ω₂ was used in Wieand et al. (1989) to compare the wAUCs for continuous data. Here the estimator ${\widehat{\Omega }}_{l,m,n}$ of Ω_l, for l =1, 2, is obtained by substituting the empirical function estimators Ŝ_d,l for S_d,l and Ŝ _d̄,l for S_d̄,l in Ω_l:

$${\widehat{\Omega }}_{l,m,n}={\int }_0^1{\widehat{S}}_{d,l}\left\{{\widehat{S}}_{\stackrel{‒}{d},l}^{-1}\left(u\right)\right\}\mathrm{d}W\left(u\right).$$

The resulting Δ-statistic is given by ${\widehat{\Delta }}_{m,n}={\widehat{\Omega }}_{l,m,n}-{\widehat{\Omega }}_{2,m,n}$. Hereafter the subscripts m and n in ${\widehat{\Delta }}_{m,n}$ will be omitted unless necessary and the notation $\widehat{\Delta }$ will be used. We shall need differentiability of the ROC functions for our main theorem. The following assumption guarantees this property.

Assumption 1. S_d,l and S_d̄,l have continuous positive derivatives on $\mathbb{R}$. Let $S_{d,l}^{\prime }$ and $S_{\stackrel{‒}{d},l}^{\prime }$ denote their derivatives.

Let

$$w_{i,l}={\int }_0^1[R_l\left(u\right)-I\{X_{li}\le S_{\stackrel{‒}{d},l}^{-1}\left(u\right)\}]\mathrm{d}W\left(u\right)$$

and

$$v_{j,l}={\int }_0^1\left(R_l^{\prime }\left(u\right)[I\{Y_{lj}\le S_{\stackrel{‒}{d},l}^{-1}\left(u\right)\}-u]\right)\mathrm{d}W\left(u\right),$$

for l=1,2, where $R_l^{\prime }\left(u\right)=S_{d,l}^{\prime }\left\{S_{\stackrel{‒}{d},l}^{-1}\left(u\right)\right\}∕S_{\stackrel{‒}{d},l}^{\prime }\left\{S_{\stackrel{‒}{d},l}^{-1}\left(u\right)\right\}$. The variances of w_i,j and v_j,l are

$$\mathrm{var}\left(w_{i,l}\right)={\int }_0^1{\int }_0^1{R}_l(s\wedge t)\mathrm{d}W\left(s\right)\mathrm{d}W\left(t\right)-{\left\{{\int }_0^1{R}_l\left(s\right)\mathrm{d}W\left(s\right)\right\}}^2$$

and

$$\mathrm{var}\left(v_{j,l}\right)={\int }_0^1{\int }_0^1{R}_l^{\prime }\left(s\right)R_l^{\prime }\left(t\right)(s\wedge t)\mathrm{d}W\left(s\right)\mathrm{d}W\left(t\right)-{\left\{{\int }_0^1{R}_l^{\prime }\left(s\right)s\mathrm{d}W\left(s\right)\right\}}^2.$$

Let w_i = w_i,1 – w_i,2 and v_j = v_j,1 – v_j,2. Further denote ${\stackrel{‒}{w}}_m={\sum }_{i=1}^m{w}_i∕m$ and ${\stackrel{‒}{v}}_n={\sum }_{j=1}^n{v}_j∕n$. Wieand et al. (1989) and Tang et al. (2008) studied the Δ-statistic and showed that, under assumption 1, $\widehat{\Delta }$ is ${\stackrel{‒}{w}}_m+{\stackrel{‒}{v}}_n+{\Omega }_1-{\Omega }_2+T_{m,n}^{\ast }$, where $T_{m,n}^{\ast }$ is a small order term with $T_{m,n}^{\ast }{(m+n)}^{-1∕2}$ converging to 0 in probability, as m, n → ∞ (Wieand et al. (1989), page 591). They also showed that

$$\mathrm{var}\left(w_i\right)=\sum _{l=1}^2\mathrm{var}\left(w_{i,l}\right)-2{\int }_0^1{\int }_0^1[S_d\{S_{\stackrel{‒}{d},1}^{-1}\left(s\right),S_{\stackrel{‒}{d},2}^{-1}\left(t\right)\}-R_1\left(s\right)R_2\left(t\right)]\mathrm{d}W\left(s\right)\mathrm{d}W\left(t\right),$$ (3)

and

$$\mathrm{var}\left(v_j\right)=\sum _{l=1}^2\mathrm{Var}\left(v_{j,l}\right)-2{\int }_0^1{\int }_0^1{R}_1^{\prime }\left(s\right)R_2^{\prime }\left(t\right)[S_{\stackrel{‒}{d}}\{S_{\stackrel{‒}{d},1}^{-1}\left(s\right),S_{\stackrel{‒}{d},2}^{-1}\left(t\right)\}-st]\mathrm{d}W\left(s\right)\mathrm{d}W\left(t\right).$$ (4)

2.1.1. Optimal sampling ratio for evaluating one continuous marker

We start with one marker, say marker 1. It follows from the approximation of $\widehat{\Delta }$ that ${\widehat{\Omega }}_1$ is asymptotically equivalent to ${\sum }_{i=1}^m{w}_{i,1}∕m+{\sum }_{j=1}^n{v}_{j,1}∕n+{\Omega }_1$. The w_i,1s are independent, identically distributed random corresponding to measurements of cases and the v_j,1s are independent, identically distributed random variables related to measurements of controls. Following the general expression (2), we can see that the optimal sampling ratio for evaluating marker 1 on the basis of wAUC is given by $r_1^{\ast }\surd \{\mathrm{var}\left(w_{i,1}\right)∕\mathrm{var}\left(v_{j,1}\right)\}$. This ratio includes existing results for AUC by Hanley and Hajian-Tilaki (1997) and for the sensitivity by Janes and Pepe (2006). wAUC ${\widehat{\Omega }}_1$ estimates AUC when W(u) = u for 0 < u₀ < 1. Consequently, the optimal ratio becomes

$$\surd \left[\frac{E\left\{I(X_{1i}>Y_{1j})I(X_{1i}>Y_{1l})\right\}-E{\left\{I(X_{1i}>Y_{1j})\right\}}^2}{E\left\{I(X_{1i}>Y_{1j})I(X_{1k}>Y_{1j})\right\}-E{\left\{I(X_{1i}>Y_{1j})\right\}}^2}\right],$$

i, k = 1, . . . , m, and j, l = 1, . . . , n. This can be written in terms of placement values, $\surd [\mathrm{var}\left\{S_{\stackrel{‒}{d},l}\left(Y_{1j}\right)\right\}∕\mathrm{var}\left\{S_{d,1}\left(X_{1i}\right)\right\}]$, as shown in Janes and Pepe (2006). When W(u)=I{u=u₀} for 0<u₀<1,${\widehat{\Omega }}_1$, estimates the sensitivity at the FPR u₀ and the optimal ratio can be shown reduce to

$$r_{s,1}^{\ast }=\surd \left\{\frac{R_1\left(u_0\right)-R_1^2\left(u_0\right)}{R_1^{\prime }{\left(u_0\right)}^2{u}_0-R_1^{\prime }{\left(u_0\right)}^2{u}_0^2}\right\},$$

or

$$\frac{\surd [R_1\left(u_0\right)\{1-R_1\left(u_0\right)\}∕\left\{u_0(1-u_0)\right\}]}{R_1^{\prime }\left(u_0\right)}.$$

This has been derived in Janes and Pepe (2006).

2.1.2. Optimal sampling ratio for comparing two continuous markers

Since the w_is are random variables corresponding to measurements of case patients and the v_js are also random variables related to measurements of control subjects, expression (2) gives the optimal ratio for comparing the difference between wAUCs:

$$r^{\ast }=\surd \{\mathrm{var}\left(w_i\right)∕\mathrm{var}\left(v_j\right)\},$$ (5)

where the variances are given in equations (3) and (4).

Since $\widehat{\Delta }$ compares AUCs, partial AUCs or sensitivities at a particular FPR, we discuss the optimal ratios for these special cases by specifying corresponding weight functions. When we let the weight function be W(u) = u, for 0 < u < 1, $\widehat{\Delta }$ compares the AUCs. The optimal ratio in equation (5) implies that the following ratio between the case and the control maximizes the power for comparing the AUCs A: $r_A^{\ast }=\surd (v_x^A∕v_y^A)$, where

$$v_x^A=\sum _{l=1}^2[E\left\{I(X_{li}>Y_{lj})I(X_{li}>Y_{ll})\right\}-E{\left\{I(X_{li}>Y_{lj})\right\}}^2]-2[E\left\{I(X_{1i}>Y_{1j})I(X_{2i}>Y_{2l})\right\}-E\left\{I(X_{1i}>Y_{1j})\right\}E\left\{I(X_{2i}>Y_{2l})\right\}]$$ (6)

and

$$v_y^A=\sum _{l=1}^2[E\left\{I(X_{li}>Y_{lj})I(X_{lk}>Y_{lj})\right\}-E{\left\{I(X_{li}>Y_{lj})\right\}}^2]-2[E\{I(X_{li}>Y_{lj})I(X_{2k}>Y_{2j})-E\left\{I(X_{1i}>Y_{1j})\right\}E\left\{I(X_{2k}>Y_{2j})\right\}],$$ (7)

as shown in Appendix A. When W(u)= I{u = u₀} for 0 < u₀ < 1, $\widehat{\Delta }$ compares the sensitivities of two markers at the FPR u₀. The optimal ratio in equation (5) reduces to

$$r_s^{\ast }=\surd \left(\frac{\sum _{l=1}^2\{R_l\left(u_0\right)-R_l^2\left(u_0\right)\}-2A}{\sum _{l=1}^2[R_l^{\prime }{\left(u_0\right)}^2-{\left\{R_l^{\prime }\left(u_0\right)u_0\right\}}^2]-2B}\right).$$

where $A=\mathrm{Pr}\{X_{1i}>S_{\stackrel{‒}{d},l}^{-1}\left(u_0\right),X_{2i}>S_{\stackrel{‒}{d},2}^{-1}\left(u_0\right)\}-R_1\left(u_0\right)R_2\left(u_0\right)$ and $B=R_1^{\prime }\left(u_0\right)R_2^{\prime }\left(u_0\right)[\mathrm{Pr}\{X_{1i}>S_{\stackrel{‒}{d},1}^{-1}\left(u_0\right),X_{2i}>S_{\stackrel{‒}{d},2}^{-1}\left(u_0\right)\}-u_0^2]$

2.2. Optimal sampling ratio for ordinal markers

The variance of the $\widehat{\Delta }$-statistic involves the first derivatives of the ROC curves. The optimal ratio in equation (5) cannot be readily applied to ordinal data which often occur in radiology. We thus consider the non-parametric statistic by DeLong et al. (1988) to obtain the optimal ratio for comparing two ordinal markers which are usually two imaging modalities in radiology. Let ${\Omega }_l^A=P(X_{li}>Y_{lj})+P(X_{li}=Y_{lj})∕2$ for marker l, and ${\widehat{\Omega }}_l^A$ be its estimator. DeLong's statistic estimates ${\widehat{\Delta }}^{\mathrm{D}}={\Omega }_1^A-{\Omega }_2^A$ and is given as

$${\widehat{\Delta }}^{\mathrm{D}}={\widehat{\Omega }}_1^A-{\widehat{\Omega }}_2^A=\frac{1}{mn}\sum _{j=1}^n\sum _{i=1}^m({\psi }_{ijj}^{\left(1\right)}-{\psi }_{ij}^{\left(2\right)}),$$

where ${\psi }_{ij}^{\left(l\right)}$ equals 1, for Y_lj < X_li, $\frac{1}{2}$ for Y_lj > X_li and 0 for Y_lj > X_li, for marker l, l=1,2.

DeLong et al. (1988) showed that the large sample variance of ${\widehat{\Delta }}^{\mathrm{D}}$ has the form $\mathrm{var}\left({\widehat{\Delta }}^{\mathrm{D}}\right)=v_x^{\mathrm{D}}∕m+v_y^{\mathrm{D}}∕n$, with

$$v_x^{\mathrm{D}}=\sum _{l=1}^2\{E\left({\psi }_{ij}^{\left(l\right)}{\psi }_{il}^{\left(l\right)}\right)-E{\left({\psi }_{ij}^{\left(l\right)}\right)}^2\}-2\{E\left({\psi }_{ij}^{\left(1\right)}{\psi }_{ij}^{\left(2\right)}\right)-E\left({\psi }_{ij}^{\left(1\right)}\right)E\left({\psi }_{ij}^{\left(2\right)}\right)\}$$

from the cases, and

$$v_y^{\mathrm{D}}=\sum _{l=1}^2\{E\left({\psi }_{ij}^{\left(l\right)}{\psi }_{kj}^{\left(l\right)}\right)-E{\left({\psi }_{ij}^{\left(l\right)}\right)}^2\}-2\{E\left({\psi }_{ij}^{\left(1\right)}{\psi }_{kj}^{\left(2\right)}\right)-E\left({\psi }_{ij}^{\left(1\right)}\right)E\left({\psi }_{kj}^{\left(2\right)}\right)\}$$

from the controls. Therefore, it follows from equation (2) that the ratio $r_{\mathrm{D}}^{\ast }=\surd (v_x^{\mathrm{D}}∕v_y^{\mathrm{D}})$ maximizes the power for comparing two ordinal markers. For the problem of evaluating a single ordinal marker, the optimal ratio is reduced to

$$\surd \left\{\frac{E\left({\psi }_{ij}^{\left(1\right)}{\psi }_{il}^{\left(1\right)}\right)-E{\left({\psi }_{ij}^{\left(1\right)}\right)}^2}{E\left({\psi }_{ij}^{\left(1\right)}{\psi }_{kj}^{\left(1\right)}\right)-E{\left({\psi }_{ij}^{\left(1\right)}\right)}^2}\right\}.$$

3. Two-stage procedure to obtain the optimal ratio

We may assume a parametric model to obtain the variances and resulting optimal ratios derived in the preceding section. When a parametric model is correctly specified, the optimal ratio can be calculated from equation (2) for comparing ROC summary measures, and the SS to obtain a specified power can be subsequently derived. However, if the parametric model is misspecified, the SS calculated may not give the appropriate power. Fig. 1 shows the optimal ratios for comparing the AUCs and pAUCs with the case and control having different variances. The case and control observations are from the bivariate normal distributions with (X₁, X₂)~ N{(2, 2), Σ_x}, and (Y₁, Y₂) ~ N{(0, 0), Σ_y) , where Σ_x has diagonal elements 1 and off-diagonal elements 0.1, and Σ_y has diagonal elements σ_Y² and off-diagonal elements of 0.1σ_Y².We see that the optimal ratio decreases as αY increases from 0.8 to 1.3. This indicates that the variances of the case and the control play an important role in the optimal ratio. When the variance for the control is larger than the case, the optimal ratio becomes larger than 1, indicating that sampling more controls than cases yields a better power to detect a difference between markers. Thus, the misspecification of parametric models at the planning stage may lead to an incorrect optimal ratio.

Fig. 1.

Optimal sampling ratio for comparing (a) the AUCs or (b) pAUCs: the case and control observations are from the bivariate normal distributions with (X₁, X₂)~N{(2, 2), Σ_x} and (Y₁, Y₂) ~ N{(0,0), Σ_y}, where Σ_x has diagonal elements and off-diagonal elements 0.1, and Σ_y has diagonal elements σ_Y² and off-diagonal elements 0.1σ_Y²; the pAUCs are obtained over the FPR between 0 and 0.6

For a fixed sample two-sided hypothesis test, to detect the difference between ROC summary measures, the required SSs m and n with power 1 – β and the significance level of α are given by

$$m=rn=\frac{{(z_{1-\alpha ∕2}+z_{\beta })}^2(v_x+v_y)}{{\Delta }_1^2},$$ (8)

where Δ₁ is the diffference between ROC summary measures under the alternative hypothesis to be detected. The total required SS is N = m + n.

Proschan (2005) introduced the concept of internal pilot data which often refers to accumulated data after a trial has been carried out for a certain period of time. To correct for the model misspecification at the beginning of the trial, we propose a two-stage procedure to use internal pilot data after some observations are available during the trial. Suppose that the total required SS N is fixed. Without loss of generality, we use a two-sided test in the procedure proposed. The procedure is given in the following steps.

• Step 1: specify a parametric model to obtain v_x,0 and v_y,0, and the resulting initial optimal ratio $r_0^{\ast }=\surd (v_{x,0}∕v_{y,0})$.

• Step 2: use the ratio $r_0^{\ast }$ together with v_x,0 and v_y,0 in the following SS formula to calculate initial SS m₀ and n₀ with power 1 – β and significance level α, $m_0={(z_{\alpha ∕2}+z_{\beta })}^2(v_x+r_0^{\ast }{v}_y)∕{\Delta }_1^2$, and n₀ = N – m₀, where Δ₁ is the difference between ROC summary measures under the alternative hypothesis.

• Step 3: after sufficient marker measurements are available on m₁ cases and n₁ controls at the first stage, the variance expressions of either the Δ-statistic or DeLong's statistic are recalculated by using available data. These variance estimators, ${\widehat{v}}_{x,1}$ and ${\widehat{y}}_{y,1}$, are applied in equation (2) to recalculate the optimal ratio ${\widehat{r}}^{\ast }=\surd ({\widehat{v}}_{x,1}∕{\widehat{v}}_{y,1})$.

• Step 4: continue the trial by recruiting M₂ cases and N₂ controls, where M₂ and N₂ are given by

  $$\begin{array}{cc}\hfill M_2=& \frac{N{\widehat{r}}^{\ast }}{1+{\widehat{r}}^{\ast }}-m_1,\hfill \\ \hfill N_2=& \frac{N}{1+{\widehat{r}}^{\ast }}-n_1.\hfill \end{array}$$ (9)

It was showed in Proschan (2005) that using the internal pilot data for comparing population means in clinical trials maintains the nominal type I error rate. The reason is that the sample variance that is obtained at the end of the first stage does not give information for the sample mean at the end of the trial. We show that it is also true in our case as m,n → ∞. Suppose that ${\widehat{\Delta }}_{m,n}$ is the estimated Δ at the end of the stage with m cases and n controls. The variance estimators at the first stage are ${\widehat{v}}_{x,1}={\Sigma }_{i=1}^{m_1}{(w_i-{\stackrel{‒}{w}}_{m_1})}^2∕m_1$ and ${\widehat{v}}_{y,1}={\Sigma }_{j=1}^{n_1}{(v_j-{\stackrel{‒}{v}}_{n_1})}^2∕n_1$, where w_i and v_j are given in Section 2.2. We first state the results for w̄_m and v̄_n in the following theorem, and then state the result for the Δ-statistic in the consequent corollary. The proof is provided in Appendix A.

Theorem 1. Let H₀ :Ω₁=Ω₂. Assume that m, n → ∞, m₁/m → λ₁, n₁/n → λ₂ and m/n →, where 0 < λ < ∞ and 0 < λ₁, λ₂ < 1. Then,

$${({\stackrel{‒}{w}}_m,{\widehat{v}}_{x,1})}^{\mathrm{T}}\surd m\stackrel{\mathrm{d}}{\to }N\{(0,{\sigma }_{v,x}^2),{\Sigma }_w\},$$ (10)

where ${\sigma }_{v,x}^2=\mathrm{var}({\widehat{v}}_{x,1}\surd m)$ and ${\Sigma }_v=\mathrm{diag}\{\mathrm{var}({\stackrel{‒}{v}}_n\surd m),\mathrm{var}({\widehat{v}}_{y,1}\surd m)\}$. Also, under assumption 1,

$${({\stackrel{‒}{v}}_n,{\widehat{v}}_{y,1})}^{\mathrm{T}}\surd m\stackrel{\mathrm{d}}{\to }N\{(0,{\sigma }_{v,y}^2),{\Sigma }_v\},$$ (11)

Theorem 1 implies that w̄_m is asymptotically independent of v̂_x,1, and v̄_n is asymptotically independent of v̂_{y, 1}. We also observe that both w̄_m and v̂_x,1 are obtained on different subjects from v̄_n and v̂_y,1. Thus, we can obtain the following corollary for ${\widehat{\Delta }}_{m,n}$ by ignoring the small order term $T_{m,n}^{\ast }$ in the approximation of $\widehat{\Delta }$.

Corollary 1. Under the regularity conditions in theorem 1, ${\widehat{\Delta }}_{m,n}$ is asymptotically independent of v̂_x,1 and v̂_y,1 as m, n → ∞.

Therefore, the variance estimated at the first stage does not give information for the Δ-statistic at the end for large SSs. Thus, the resulting optimal ratio by using data from the first stage does not reveal information about the estimated difference between two ROC statistics obtained at the end of the second stage. Consequently, although the optimal ratio is updated during the trial, the analysis at the end of the trial can be carried out in the same fashion as in the trial without updating the optimal ratio. This is important in maintaining the proper type I error rate.

4. Application to the cancer diagnostic trial

We applied our method to the cancer diagnostic trial from Goddard and Hinberg (1990). Measurements from the blood samples are highly skewed for all biomarkers. We compared a new biomarker D and the reference biomarker A to illustrate the increment in power and the SS savings by using the procedure proposed. We assumed a contrast of Δ₁ =0.05 between AUCs and the type I error rate 0.05 for calculating power and SS from a two-sided alternative. The overall SS N is 353 by summing the numbers of cases and controls. At the first stage, we accrued data on m₁ = 60 cancer and n₁ = 60 non-cancer patients and obtained the variance estimates v̂_x,1 = 0.082 and v̂_y,1 = 0.035, which resulted in the optimal case–control ratio r̂* = 1.53, from equation (2). Using this optimal ratio in the expression (9) in step 4 of the procedure proposed, the numbers of the cases and controls to be recruited in the second stage were calculated to be 153 and 80 respectively. The power by using the optimal ratio was then 50.9% from the equation

$$1-\beta =\Phi \left[{\Delta }_1\surd \left\{\frac{N{\widehat{r}}^{\ast }}{(1+{\widehat{r}}^{\ast })({\widehat{v}}_{x,1}+{\widehat{v}}_{y,1}{\widehat{r}}^{\ast })}\right\}-z_{\alpha ∕2}\right].$$ (12)

This power offers a 7% increment over the power 43.8% calculated by using equation (12) by replacing r̂* with the original ratio of 0.62. We also investigated the savings on the overall SS by using the procedure proposed. Using the original power 43.8% with the estimated optimal ratio r̂* = 1.53, the overall SS was calculated to be 292 with 177 cancer patients and 115 non-cancer patients. This offers savings of 61 patients over the original ratio.

5. Simulation studies

In this section, we demonstrate the performance of our method for maximizing power or minimizing total SSs when comparing summary statistics of diagnostic tests in extensive simulation studies. We consider both continuous data and ordinal data.

5.1. Simulation studies based on continuous data

The biomarker results in the example used in Section 4 are highly skewed, and a log-normal distribution was used by Goddard and Hinberg (1990) as a possible approximation to the distribution of results. Thus, we consider bivariate log-normal distributions in the simulation studies. In addition, we simulate data from both bivariate normal distributions which are commonly used for symmetrically distributed marker results and bivariate exponential distributions which can approximate survival biomarker results. The bivariate normal models have the forms (X₁, X₂/ ~ N{(μ₁, μ₂, Σ_X} and (Y₁, Y₂/ ~ N{(0, 0), Σ_Y}, where the diagonal elements of Σ_X and Σ_Y are 1 and 9 respectively, and the correlation parameter ρ is the same for two matrices. We choose ρ = 0.1 and ρ = 0.25 in our simulations. AUC is set to be 0.70 for marker 1, and 0.75 or 0.80 for marker 2. pAUC with the FPR in the range (0, 0.6) is set to be 0.30 for marker 1, and 0.35 or 0.40 for marker 2. The bivariate log-normal models have the forms exp(X₁, X₂) and exp(Y₁, Y₂) for cases and controls respectively. The AUCs and pAUCs remain the same as in the normal models. The log-normal distribution may also demonstrate the robustness of the aforementioned non-parametric methods. The performance of the methods is expected to be similar for the normal and log-normal distributions because the non-parametric estimators should remain invariant under monotone transformations.

According to the algorithm in Gumbel (1960), the bivariate exponential random variables take the form

$$H(x,y)=H_1\left(x\right)H_2\left(y\right)[1+4\rho \{1-H_1\left(x\right)\}\{1-H_2\left(y\right)\}],$$

where H_l, l = 1, 2, are univariate exponential functions, and ρ is in [–0.25, 0.25]. We set ρ to 0.1 or 0.25 here. The marginal survival functions are exp(–β_l1x) and exp(–β_l2y), so we could generate data from these two distributions. In the simulation, AUC is set to 0.70 for marker 1, and 0.75 or 0.80 for marker 2. pAUC with the FPR in the range (0, 0.6) is set to 0.30 for marker 1, and 0.35 or 0.45 for marker 2.

We compare the proposed two-step procedure with the equal case–control ratio and the optimal ratio. We use DeLong's statistic for comparing the AUCs and the Δ-statistic for comparing the pAUCs. In our simulation, we first assume that our samples were from bivariate normal distributions; then we use equation (8) to calculate the initial total required SS. With the type I error rate 0.05 and power 80%, the initial total required SSs are N = 1421, or N = 326 to detect the difference of two pairs of AUCs of (0.70, 0.75) and (0.70, 0.80) respectively, with ρ=0.1. When ρ=0.25, the total required SSs, N =1207, or N = 278, are needed to detect the difference in these pairs. There are three different sampling ratios:

1. the proposed two-stage optimal ratio;

2. the optimal ratio of 0.5 for the normal and log-normal distributions and the optimal ratio of 1.5 for the exponential distributions;

3. the equal sampling ratio.

To implement the method proposed, we let m₁ = n₁ = N/4. By substituting non-parametric variance estimates v̂_x,1 and v̂_y,1, the resulting optimal ratio is estimated by ${\widehat{r}}^{\ast }=\surd ({\widehat{v}}_{x,1}∕{\widehat{v}}_{y,1})$, and M₂ and N₂ are calculated by using equation (9). We then generate M₂ new observations for cases and N₂ observations for controls. Consequently, the null hypothesis of equal AUCs or pAUCs is rejected in favour of the alternative if the absolute value of the calculated Z-statistic is greater than or equal to z_0.025. The simulated power is then calculated as the percentage of times out of 5000 simulation runs that the null hypothesis is rejected. The simulated powers are presented in Table 1, which illustrates that the simulated powers of the two-stage method proposed are close to those of the optimal ratio and are greater than those of the equal sampling ratio in the normal settings. Since the optimal ratio for the exponential distribution specified is close to 1.5, we see that most of the powers of the method proposed are greater than those of fixed ratios.

Table 1.

Simulated power for comparing AUCs or pAUCs by using the two-stage method proposed and fixed ratios, over 5000 simulations^†

ρ	Distribution	Powers (%) for comparing AUCs				Powers (%) for comparing pAUCs
		AUC for marker 2	Two-stage	Fixed ratio		pAUC for marker 2	Two-stage	Fixed ratio
				Equal	Optimal			Equal	Optimal
0.10	BN	0.75	80.0	77.0	79.5	0.35	33.3	31.4	32.8
		0.80	80.4	74.6	80.2	0.45	88.1	86.2	88.9
	LN	0.75	79.1	74.5	78.3	0.35	34.0	31.9	32.1
		0.80	80.5	74.8	79.4	0.45	89.1	85.3	88.0
	BE	0.75	81.0	80.4	82.2	0.35	84.0	83.4	84.6
		0.80	81.6	80.0	82.7	0.45	85.0	84.0	84.4
0.25	BN	0.75	82.2	78.0	81.7	0.35	37.3	34.8	37.5
		0.80	81.0	77.6	80.1	0.45	91.8	89.4	92.1
	LN	0.75	82.0	78.7	81.5	0.35	37.0	34.2	36.7
		0.80	83.5	78.1	82.6	0.45	92.7	89.8	92.3
	BE	0.75	83.7	82.6	83.3	0.35	91.2	90.3	91.0
		0.80	83.6	82.8	84.4	0.45	90.9	90.7	90.8

^†AUC for marker 1 is 0.70, and pAUC for marker 1 is 0.30. BN, bivariate normal; LN, bivariate log-normal; BE, bivariate exponential. ρ is the correlation coefficient of two markers. The optimal ratios for the bivariate normal and log-normal distributions are close to 0.5, and the optimal ratios for the bivariate exponential distribution are close to 1.5.

We also conduct simulation studies to illustrate that the method proposed reduces the total SS compared with the equal ratio. The aforementioned bivariate normal distribution is applied to simulate test results. We first calculate the initial total SS N with the equal ratio, type I error rate 0.05 and power 80%. At the end of stage I, with m₁ =n₁ simulated test results from two groups, we update the case/control ratio with the estimated optimal ratio from the interim data, and recalculate the total SS that is needed to achieve 80% power on the basis of the estimated ratio. Additional test results are then generated according to the updated SS in two groups, and the Z-statistic is estimated. The null hypothesis of equal AUCs is rejected in favour of the alternative if the absolute value of the calculated Z-statistic is greater than or equal to z_0.025. The simulated power is given by the percentage of times out of 5000 simulation runs that the null hypothesis is rejected. The simulated power and the average updated total SS with m₁ = n₁ (N/5, N/7) are presented in Table 2, which illustrates that the two-stage method proposed reduces the total SS compared with the equal ratio. The simulated power of the two-stage method proposed is close to the nominal power for all parameterizations. In addition, the simulated power and updated SS vary little with different sizes at stage I.

Table 2.

Average updated total SS and simulated power for comparing AUCs by using the proposed two-stage method over 5000 simulations^†

ρ	Results for m1 = n1 = N/5				Results for m1 = n1 = N/7
	AUC for marker 2	Initial SS	Updated SS	Power (%)	AUC for marker 2	Initial SS	Updated SS	Power (%)
0.10	0.75	1744	1333	80.9	0.75	1744	1335	80.4
	0.80	405	311	80.6	0.80	405	313	79.1
0.25	0.75	1527	1160	80.3	0.75	1527	1161	80.2
	0.80	357	273	81.5	0.80	357	275	79.8

^†The AUC for marker 1 is 0.70. ρ is the correlation coefficient of two markers.

We also evaluate the performance of the two-stage procedure to see whether the procedure maintains the nominal type I error rate. We use N = 200, 400, 500. We consider the parametric distributions and the three different sampling ratios that were used in the previous simulation. We assume equal AUCs or pAUCs with the AUCs being (0.70, 0.75, 0.80), and the pAUCs being (0.30, 0.35, 0.40). The nominal type I error rate is 0.05 in our simulation. The simulated type I error rates with 10000 simulation runs are shown in Table 3. All these rates are close to the nominal level when the sample size goes to 500.

Table 3.

Type I error rates for comparing the AUCs or pAUCs by using the two-stage method proposed, over 10000 simulations^†

ρ	Distribution	Error rates (%) for comparing the AUCs				Error rates (%) for comparing the pAUCs
		AUCs	N = 200	N = 400	N = 500	pAUCs	N = 200	N = 400	N = 500
0.1	BN	0.70	4.5	5.0	5.0	0.30	4.8	5.1	5.0
		0.75	5.1	5.0	4.9	0.35	5.0	4.9	5.0
		0.80	4.9	5.1	4.9	0.40	5.2	5.1	5.5
	LN	0.70	4.9	5.0	5.0	0.30	4.6	5.2	5.1
		0 75	5.1	4.9	5.1	0.35	4.6	5.1	5.0
		0.80	5.0	4.4	5.0	0.40	5.0	5.1	4.9
	BE	0.70	5.0	5.1	5.0	0.30	5.2	4.9	5.0
		0.75	5.0	4.9	4.9	0.35	5.3	5.0	5.1
		0.80	5.2	5.1	4.9	0.40	4.7	4.9	5.1
0.25	BN	0.70	4.9	4.8	4.7	0.30	5.1	5.0	4.9
		0.75	5.1	5.0	5.0	0.35	5.2	5.3	5.2
		0.80	5.2	5.1	5.0	0.40	4.9	5.1	5.1
	LN	0.70	5.0	5.1	5.0	0.30	5.1	5.3	5.2
		0.75	4.9	4.7	4.9	0.35	4.5	5.0	4.8
		0.80	4.8	3.9	5.0	0.40	4.6	4.8	5.0
	BE	0.70	4.2	5.0	5.2	0.30	5.0	5.0	4.8
		0.75	5.3	5.0	4.9	0.35	5.1	4.7	5.0
		0.80	4.2	5.1	4.9	0.40	4.9	4.7	5.0

^†BN, bivariate normal distribution; LN, bivariate log-normal distribution; BE, bivariate exponential distribution.

N is the total required SS and ρ is the correlation coefficient of two markers.

5.2. Simulation studies based on ordinal data

We also conduct simulation studies to evaluate the simulated power of the proposed method on ordinal test results. We first use the aforementioned bivariate log-normal distributions and bivariate exponential distributions to simulate continuous results. We then use the 20th, 40th, 60th and 80th percentiles of the distributions to categorize the simulated continuous data as follows. A test result is recoded as 1 if it is less than the 20th percentile, 2 if it is between the 20th and 40th percentiles, 3 if it is between the 40th and 60th percentiles, 4 if it is between the 60th and 80th percentiles, and 5 if it is greater than the 80th percentile. The rest of the simulated settings are identical to those in the previous section on evaluating the power for continuous data. The results in Table 4 indicate that the simulated power by using the method proposed is similar to that of the optimal ratios and is higher than for those parameterizations using the equal ratio.

Table 4.

Simulated power for ordinal data for comparing AUCs by using the two-stage method proposed and fixed ratios, over 5000 simulations^†

ρ	Distribution	AUC for marker 2	Two-stage power (%)	Fixed ratio power (%)
				Equal	Optimal
0.10	LN	0.75	81.2	75.3	79.7
		0.80	82.0	77.3	83.2
	BE	0.75	87.1	84.7	88.6
		0.80	86.6	84.5	86.4
0.25	LN	0.75	80.9	77 6	80.0
		0.80	80.0	78.6	80.2
	BE	0.75	89.0	87.1	88.6
		0.80	88.1	87.2	88.9

^†The AUC for marker 1 is 0.70. LN, bivariate log-normal distribution; BE, bivariate exponential distribution. ρ is the correlation coefficient of two markers.

6. Discussion

The optimal sampling ratio in diagnostic trials can maximize the test power or minimize the overall SS. The optimal sampling ratio that is discussed in this paper is analogous to the optimal allocation ratio in assigning treatments to patients in clinical trials. The optimal allocation ratio has been used in clinical trials for decades, but the importance of the optimal ratio in diagnostic trials has not been widely recognized. Implementation requires the calculation of complicated variances of frequently used ROC statistics. This paper discusses a common variance structure for ROC statistics and thereby introduces optimal sampling ratios in comparative diagnostic trials based on these statistics. Two popular non-parametric ROC statistics are used to illustrate the explicit forms of the optimal ratios because their variance expressions can be written as the sum of separate terms; one relates to the cases, and the other relates to the controls.

If preliminary studies are available before carrying out a comparative diagnostic trial, the variance can be estimated by using pilot data to obtain the optimal ratio for comparing specified ROC summary measures. The ratio can then be used to recruit patients in the trial, and recalculating the ratio may not be necessary during the trial. However, when medical practitioners do not have preliminary data for the markers and are not certain about the distributions of the marker results, the distribution assumption that is used for obtaining the optimal ratio may be far from the true underlying distributions for the marker results. This may result in less power or larger overall SSs than using the true optimal ratio. The two-stage procedure proposed is then particularly useful to ensure that the optimal ratio can be recalculated by using internal pilot data during the trial. The procedure proposed performs well in a large-scale simulation study. We also demonstrate that the procedure proposed maintains the nominal type I error rate in the simulation. We use an example in cancer diagnostic studies to illustrate the application of our method on maximizing the test power and saving overall SSs. The results indicate that, compared with the original sampling ratio, using the proposed two-stage procedure for a fixed overall sample size increased the test power. Alternatively, for the fixed test power, the procedure proposed reduces the overall SS by nearly 25%.

In some rare diseases, it may not be possible to recruit the required number of the cases. Suppose that only 135 cancer patients can be recruited in the aforementioned cancer diagnostic trial. If the calculated optimal ratio of 1.53 is maintained, then 89 non-cancer patients should be in the trial. This leads to the total SS of N =224. Using the power calculation formula (12) gives a power of 35.2%, which sacrifices 8% power while reducing the SS by 129. This indicates that, for a fixed number of cases, recruiting more controls may increase the power if the budget of a trial allows. This can be seen from the variance expression (1) since, when m is fixed in equation (1), the variance decreases as n increases. Thus, with the constraint of total 353 subjects and 135 cases, the original sampling ratio of 1.14 (135/118) gives the maximum power.

The characteristics of subjects are often matched in case–control studies to minimize the confounding effects. Janes and Pepe (2008) illustrated that a ROC summary estimate without adjusting for covariates may be biased. If covariate information is available, matching should be considered for deriving the optimal sampling ratio. Future research on this topic is warranted.

Appendix A: Variance expressions of receiver operating characteristic statistics, variance deviation and proof of proposition 1

A.1. Variance expressions for a parametric receiver operating characteristic statistic

When measurements of markers have bivariate normal distributions, Mazumdar and Liu (2003) provided expressions for the variances, v_x and v_y. Suppose that (X₁, X₂) ~ N{(μ_1d, μ_2d), Σ_d} for i=1, . . . , m and (Y₁,Y₂) ~ N{(μ_1d̄, μ_2d̄} for j = 1, . . . , n, where

$${\Sigma }_d=\left(\begin{array}{cc}\hfill {\sigma }_{1d}^2\hfill & \hfill \rho {\sigma }_{1d}{\sigma }_{2d}\hfill \\ \hfill \rho {\sigma }_1{\sigma }_{2d}\hfill & \hfill {\sigma }_{2d}^2\hfill \end{array}\right)$$

and

$${\Sigma }_{\stackrel{‒}{d}}=\left(\begin{array}{cc}\hfill {\sigma }_{1\stackrel{‒}{d}}^2\hfill & \hfill \rho {\sigma }_{1\stackrel{‒}{d}}{\sigma }_{2\stackrel{‒}{d}}\hfill \\ \hfill \rho {\sigma }_{1\stackrel{‒}{d}}{\sigma }_{2\stackrel{‒}{d}}\hfill & \hfill {\sigma }_{2\stackrel{‒}{d}}^2\hfill \end{array}\right).$$

The statistic considered in Mazumdar and Liu (2003) is the partial AUC estimator given by

$${\widehat{\theta }}_p^{bn}={\int }_{c_1}^{c_2}\Phi ({\alpha }_1-{\beta }_{1}x)\varphi \left(x\right)\mathrm{d}x-{\int }_{c_1}^{c_2}\Phi ({\alpha }_2-{\beta }_{2}x)\varphi \left(x\right)\mathrm{d}x,$$

where α₁ = (μ_1d – μ_{1 d̄})/σ_1d, β₁ = σ_1d̄/σ_1d and α₂ = (μ_2d – μ_2d̄)/σ_2d and β₂ = σ_2d̄/σ_2d. Let

$${\Gamma }_1(r,s;c_1,c_2)={\phantom{\mid }\frac{1}{\surd (s^2+1)}\varphi \left\{\frac{r}{\surd (s^2+1)}\right\}\Phi \left(x\right)\mid }_{c_1^{\prime }}^{c_2^{\prime }}$$

and

$${\Gamma }_2(r,s;c_1,c_2)={\phantom{\mid }-\frac{1}{s^2+1}\varphi \left\{\frac{r}{\surd (s^2+1)}\right\}\left\{\varphi \left(x\right)+\frac{rs}{\surd (s^2+1)}\Phi \left(x\right)\right\}\mid }_{c_1^{\prime }}^{c_2^{\prime }},$$

with $c_1^{\prime }=c_1\surd (s^2+1)+rs∕\surd (s^2+1)$ and $c_2^{\prime }=c_2\surd (s^2+1)+rs∕\surd (s^2+1)$.

Let

$$f_{l1}=\frac{1}{{\sigma }_{ld}}{\Gamma }_1\left(\frac{{\mu }_{ld}-{\mu }_{l\stackrel{‒}{d}}}{{\sigma }_{ld}},-\frac{{\sigma }_{l\stackrel{‒}{d}}}{{\sigma }_{ld}};c_1,c_2\right)$$

and f_l2 = –f_l1, for l = 1, 2. In addition, let

$$f_{l3}=\frac{2}{2{\sigma }_{ld}^3}\left\{({\mu }_{l\stackrel{‒}{d}}-{\mu }_{ld}){\Gamma }_1\left(\frac{{\mu }_{ld}-{\mu }_{l\stackrel{‒}{d}}}{{\sigma }_{ld}},-\frac{{\sigma }_{l\stackrel{‒}{d}}}{{\sigma }_{ld}};c_1,c_2\right)+{\sigma }_{l\stackrel{‒}{d}}{\Gamma }_2\left(\frac{{\mu }_{ld}-{\mu }_{l\stackrel{‒}{d}}}{{\sigma }_{ld}},-\frac{{\sigma }_{l\stackrel{‒}{d}}}{{\sigma }_{ld}};c_1,c_2\right)\right\},$$

and

$$f_{l4}=-\frac{1}{2{\sigma }_{ld}{\sigma }_{l\stackrel{‒}{d}}}{\Gamma }_2\left(\frac{{\mu }_{ld}-{\mu }_{l\stackrel{‒}{d}}}{{\sigma }_{ld}},-\frac{{\sigma }_{l\stackrel{‒}{d}}}{{\sigma }_{ld}};c_1,c_2\right).$$

The variances v_x and v_y for ${\widehat{\theta }}_p^{bn}$ can be written as

$$v_x={\sigma }_{1d}^2(f_{11}^2+2f_{13}^2{\sigma }_{1d}^2)+{\sigma }_{2d}^2(f_{21}^2+2f_{23}^2{\sigma }_{2d}^2)-2\rho {\sigma }_{ld}{\sigma }_{2d}(f_{11}{f}_{21}+f_{13}{f}_{23})$$

and

$$v_y={\sigma }_{1\stackrel{‒}{d}}^2(f_{12}^2+2f_{14}^2{\sigma }_{1\stackrel{‒}{d}}^2)+{\sigma }_{2\stackrel{‒}{d}}^2(f_{22}^2+2f_{24}^2{\sigma }_{2\stackrel{‒}{d}}^2)-2\rho {\sigma }_{1\stackrel{‒}{d}}{\sigma }_{2\stackrel{‒}{d}}(f_{12}{f}_{22}+f_{14}{f}_{24}).$$

A.2. Derivation of $v_x^A$ and $v_y^A$

We can show that

$${\int }_0^1{\int }_0^1{S}_d\{S_{\stackrel{‒}{d},1}^{-1}\left(s\right),S_{\stackrel{‒}{d},2}^{-1}\left(t\right)\}\mathrm{d}s\mathrm{d}t$$

can be expressed as

$${\int }_{-\infty }^{\infty }{\int }_{-\infty }^{\infty }{S}_d(y_1,y_2)\mathrm{d}{S}_{\stackrel{‒}{d},1}\left(y_1\right)\mathrm{d}{S}_{\stackrel{‒}{d},2}\left(y_2\right).$$

Let $S_{\stackrel{‒}{d},1}^{-1}\left(s\right)=y_1$ and $S_{\stackrel{‒}{d},2}^{-1}\left(t\right)=y_2$; then we have

$${\int }_0^1{\int }_0^1{S}_d\{S_{\stackrel{‒}{d},1}^{-1}\left(s\right),S_{\stackrel{‒}{d},2}^{-1}\left(t\right)\}\mathrm{d}s\mathrm{d}t=E\left\{I(X_{1i}>Y_{1j})I(X_{2i}>Y_{2l})\right\}.$$

Similarly, v_y becomes

$$v_y=\sum _{l=1}^2\left[{\int }_0^1{\int }_0^1{R}_l^{\prime }\left(s\right)R_l^{\prime }\left(r\right)(s\wedge t)\mathrm{d}s\mathrm{d}t-{\left\{{\int }_0^1{R}_l^{\prime }\left(s\right)s\mathrm{d}s\right\}}^2\right]-2{\int }_0^1{\int }_0^1{R}_1^{\prime }\left(s\right)R_2^{\prime }\left(t\right)[S_{\stackrel{‒}{d}}\{S_{\stackrel{‒}{d},l}^{-1}\left(s\right),S_{\stackrel{‒}{d},2}^{-1}\left(t\right)\}-st]\mathrm{d}s\mathrm{d}t.$$

It follows that

$${\int }_0^1{\int }_0^1{R}_1^{\prime }\left(s\right)R_2^{\prime }\left(t\right)S_{\stackrel{‒}{d}}\{S_{\stackrel{‒}{d},1}^{-1}\left(s\right),S_{\stackrel{‒}{d},2}^{-1}\left(t\right)\}={\int }_0^1{\int }_0^1\frac{S_{d,1}^{\prime }\left\{S_{\stackrel{‒}{d},1}^{-1}\left(s\right)\right\}}{S_{\stackrel{‒}{d},1}^{\prime }\left\{S_{\stackrel{‒}{d},1}^{-1}\left(s\right)\right\}}\frac{S_{d,2}^{\prime }\left\{S_{\stackrel{‒}{d},2}^{-1}\left(t\right)\right\}}{S_{\stackrel{‒}{d},2}^{\prime }\left\{S_{\stackrel{‒}{d},2}^{-1}\left(t\right)\right\}}{S}_{\stackrel{‒}{d}}\{S_{\stackrel{‒}{d},1}^{-1}\left(s\right),S_{\stackrel{‒}{d},2}^{-1}\left(t\right)\}\mathrm{d}s\mathrm{d}t.$$

Let $S_{\stackrel{‒}{d},1}^{-1}\left(s\right)=y_1$ and $S_{\stackrel{‒}{d},2}^{-1}\left(t\right)=y_2$; then it follows that

$$\begin{array}{cc}\hfill \iint S_{d,1}^{\prime }\left(y_1\right)S_{d,2}^{\prime }\left(y_2\right)S_{\stackrel{‒}{d}}(y_1,y_2)\mathrm{d}{y}_1\mathrm{d}{y}_2& =E\left\{I(X_{1i}<Y_{1j})I(X_{2k}<Y_{2j})\right\}\hfill \\ \hfill & =E\left[\{1-I(X_{1i}>Y_{1j})\}\{1-I(X_{2k}>Y_{2j})\}\right]\hfill \\ \hfill & =1-E\left\{I(X_{1i}>Y_{1j})\right\}-E\left\{I(X_{2k}>Y_{2j})\right\}+E\left\{I(X_{1i}>Y_{1j})I(X_{2k}>Y_{2j})\right\}.\hfill \end{array}$$

Because

$${\int }_0^1{\int }_0^1{R}_1^{\prime }\left(s\right)R_2^{\prime }\left(t\right)st\mathrm{d}s\mathrm{d}t$$

can also be written as

$$1-\mathrm{Pr}(X_{1i}>Y_{1j})-\mathrm{Pr}(X_{2k}>Y_{2j})+E\left\{I(X_{1i}>Y_{1j})\right\}E\left\{I(X_{2k}>Y_{2j})\right\},$$

the expressions for v_x and v_y are simplified to equations (6) and (7) respectively.

A.3. Proof of theorem 1

Recall that ${\stackrel{‒}{w}}_m={\Sigma }_{i=1}^m{w}_i∕m$, and ${\stackrel{‒}{v}}_n={\Sigma }_{j=1}^m{v}_j∕n$ are the sample means at the end of the trial and E(w̄_m) = E(v̄_n) = 0, where m and n are the SSs at the end of the trial. The variance estimators at the end of the first stage are ${\widehat{v}}_{x,1}={\Sigma }_{i=1}^{m_1}{(w_i-{\stackrel{‒}{w}}_{m_1})}^2∕m_1$ and ${\widehat{v}}_{y,1}={\Sigma }_{j=1}^{n_1}{(v_j-{\stackrel{‒}{v}}_{n_1})}^2∕n_1$, where m₁ and n₁ are SSs at the end of the first stage. Let $A_m={\stackrel{‒}{w}}_m\surd m$ and $B_m={\widehat{v}}_{x,1}\surd m$. We shall show that

$${\mathbf{X}}_m≔\left(\begin{array}{c}\hfill {\mathit{A}}_m\hfill \\ \hfill {\mathit{B}}_m-{\sigma }_{v,x}^2\hfill \end{array}\right)\stackrel{\mathrm{d}}{\to }{N}_2(0,{\Sigma }_w),$$ (13)

where Σ_w is a diagonal matrix. For this, using the Cramer–Wold device, consider l^TX_m, where l = (l₁, l₂)^T. Since the B_m can be expressed as

$${\mathit{B}}_m=\surd m\left(\sum _{i=1}^{m_1}{w}_i^2∕m_1-m_1{\stackrel{‒}{w}}_{m_1}^2\right),$$

it follows that

$$\begin{array}{cc}\hfill {\mathbf{I}}^{\mathrm{T}}{\mathbf{X}}_m=& \left(\sum _{i=1}^m\frac{l_1{w}_i}{m}+\sum _{i=1}^{m_1}\frac{l_2{w}_i^2}{m_1}-l_2{\stackrel{‒}{w}}_{m_1}^{-2}\right)\surd \mathit{m}-{\sigma }_{v,x}^2\hfill \\ \hfill =& \frac{\surd m}{m_1}\sum _{i=1}^{m_1}\left(\frac{m_1}{m}{l}_1{w}_i+l_2{w}_i^2-\frac{{\sigma }_{v,x}^2}{\surd m}\right)+\frac{\surd m}{m-m_1}\sum _{i=m_1+1}\frac{m-m_1}{m}{l}_1{w}_i-l_2{\stackrel{‒}{w}}_{m_1}^2\surd m\hfill \\ \hfill =& T_m\left(1\right)+T_m\left(2\right)-T_m\left(3\right).\hfill \end{array}$$ (14)

Since the w_is are bounded random variables, T_m(1) and T_m(2) have finite second moments. Also, T_m(1) and T_m(2) are independent since T_m(1) is based on the random variables {w_i : i = 1, . . . , m₁} and T_m(2) is based on the random variables {w_i : i = m₁ + 1, . . . , m}. Hence, by the central theorem and Slutsky's theorem (Serfling, 1980), it follows that, as m → ∞,

$$T_m\left(1\right)+T_m\left(2\right)\stackrel{\mathrm{d}}{\to }N[0,\mathrm{var}({\lambda }_1{l}_1{w}_1+l_2{w}_1^2)+\mathrm{var}\left\{(1-{\lambda }_1)l_1{w}_1\right\}].$$

Also, under hypothesis H₀, since $E\left(w_1^3\right)=0$, the limiting variance can be shown to reduce to $l_1^2{\sigma }_1^2+l_2^2{\sigma }_2^2$, where ${\sigma }_1^2=\{{(1-{\lambda }_1)}^2+1\}$, and ${\sigma }_2^2=\mathrm{var}\left(w_1^2\right)$. Now, returning to the last term on the right-hand side of equation (14), note that $T_m\left(3\right)=m^{-1∕2}{l}_2{({\stackrel{‒}{w}}_{m_1}\surd m)}^2$ converges to 0 in probability, as m → ∞, by the central limit theorem. This completes the proof of expression (13) and hence expression (10).

We now turn to the proof of expression (11). Now, under assumption 1, R_l(u) is continuously differentiable, and it follows that

$$v_j={\int }_0^1(R_1^{\prime }\left(u\right)[I\{Y_{1j}\le S_{\stackrel{‒}{d},1}^{-1}\left(u\right)\}-u]-R_2^{\prime }\left(u\right)[I\{Y_{2j}\le S_{\stackrel{‒}{d},2}^{-1}\left(u\right)\}-u])\mathrm{d}W\left(u\right).$$

Now the proof can be completed along the lines of the proof of expression (10). This completes the proof of theorem 1.