Exact confidence interval estimation for the difference in diagnostic accuracy with three ordinal diagnostic groups

Abstract

In the cases with three ordinal diagnostic groups, the important measures of diagnostic accuracy are the volume under surface (VUS) and the partial volume under surface (PVUS) which are the extended forms of the area under curve (AUC) and the partial area under curve (PAUC). This article addresses confidence interval estimation of the difference in paired VUS s and the difference in paired PVUS s. To focus especially on studies with small to moderate sample sizes, we propose an approach based on the concepts of generalized inference. A Monte Carlo study demonstrates that the proposed approach generally can provide confidence intervals with reasonable coverage probabilities even at small sample sizes. The proposed approach is compared to a parametric bootstrap approach and a large sample approach through simulation. Finally, the proposed approach is illustrated via an application to a data set of blood test results of anemia patients.

1. Introduction

Receiver-operating characteristic (ROC) curves, which can be constructed by plotting the false-positive rate (i.e. 1-specificity) against the true-positive rate (i.e. sensitivity), have been common tools for evaluating the performance of diagnostic tests. The area under the ROC curve (AUC) has been widely used as a quantitative index of discriminating ability of a biomarker, measured on a continuous scale, between two states of a disease; e.g., Shapiro (1999), Zhou et al. (2002) and Pepe (2003). In practice, investigators often need to compare the diagnostic accuracies between two biomarkers or diagnostic tests. The comparison of the overall diagnostic accuracy between two biomarkers measured simultaneously on an individual is frequently addressed by comparing the resulting paired AUC s. For example, Delong et al. (1988) presented a non-parametric approach to the analysis of areas under correlated ROC curves; Obuchowski (1997) and Zhou et al. (2002) applied two one-sided tests to evaluate the two-sided equivalence of two diagnostic procedures; Wieand et al. (1989) proposed non-parametric and parametric tests for the same problem; Molodianovitch et al. (2006) extended Wieand et al.’s approach to non-normal data by using Box–Cox transformation; Vexler et al. (2008) applied the maximum likelihood technique to compare AUC s for data with limits of detection; Liu et al. (2006) proposed to use the standardized difference for assessing equivalence of paired AUC s; and Li et al. (2008) show that the generalized variable approach is very appropriate to make inference about paired AUC s.

Under many circumstances, the research interest only lies in lower part of range of false-positive rates, that is, a minimum acceptable specificity is imposed. Therefore, the partial area under ROC curve (PAUC) over a practicably relevant range of false positive rate can be considered as a reasonable summary measure of diagnostic accuracy. There exist a few methods for estimating and comparing PAUC s. For example, McClish (1989, 1990) proposed a method for comparing PAUC s under the assumption of binormal model; Thompson and Zucchini (1989) proposed an ANOVA model to compare PAUC s; Jiang et al. (1996) extended Mclish’s work to highly sensitive diagnostic tests; Zhang et al. (2002) presented a non-parametric method for comparing PAUC s; Dodd and Pepe (2003) discussed a parsimonious regression model of PAUC; recently, Li et al. (2007) proposed a generalized variable approach for comparing paired PAUC s for normally distributed data.

In practice, there exist many disease processes with three ordinal disease classes. For example, mild cognitive impairment (MCI) and/or early stage Alzheimer’s disease is a transitional stage between the cognitive changes of normal aging and the more serious problems caused by Alzheimer’s disease (AD) stated in Xiong et al. (2006). As another example, in a study of iron deficiency related anemia by Wians et al. (2001), non-pregnant women with anemia and a ferritin concentration less than 20 μg/l were considered to have iron deficiency anemia (IDA) and the ones with anemia and a ferritin concentration greater than 240 μg/l were considered to have anemia of chronic disease (ACD), while the ones with a ferritin concentration between 20 and 240 μg/l were considered to belong to the intermediate group. Since patients at different disease states require different treatments, it is important to have good diagnostic tests which can discriminate among these three ordinal diagnostic groups. Thereafter, we refer the state between “diseased” and “healthy” as “intermediate”, in other words, transitional or early/mild diseased. To be specific, denote Y₁, Y₂ and Y₃ the scores of a biomarker or results of a diagnostic test and let F₁, F₂ and F₃ be the corresponding cumulative distribution functions for non-diseased, intermediate and diseased groups, respectively. Assume the results of a diagnostic test are measured on continuous scale and higher values indicate greater severity of the disease. Let p₁ = F₁(c₁), p₃ = 1–F₃(c₃), where c₁ and c₃ are threshold values (c₁ < c₃) for non-diseased group and diseased group, respectively, be true classification rates for non-diseased and diseased groups, respectively. Then the probability that a randomly selected subject from intermediate group has a score between c₁ and c₃ is

$$p_2=F_2\left(c_3\right)-F_2\left(c_2\right)=F_2\left[F_3^{-1}\right(1-p_3\left)\right]-F_2\left[F_1^{-1}\right(p_1\left)\right].$$ (1)

As a function of (p₁, p₃), p₂= p₂(p₁, p₃) defines a surface in the three-dimensional space (p₁, p₃, p₂), called ROC surface. The point (p₁, p₃, p₂)=(1, 1, 1) indicate perfect discrimination ability of the marker between three ordinal disease groups. The volume under surface (VUS) can be used as a summary measure of the diagnostic accuracy. One can prove that VUS, analogous measure to the AUC for binary classification, equals to the probability that a random selected triple with one individual from each diagnostic group have the correct ordering (i.e. Y₁ < Y₂ < Y₃). More details were given by Nakas and Yiannoutsos (2004) and Xiong et al. (2006). Similar to PAUC for the cases with two diagnostic groups, the partial volume under surface (PVUS) has been proposed to denote the diagnostic accuracy with pre-specified minimum classification rates for the cases with three ordinal diagnostic groups. Recently, the traditional ROC analysis based on a binary gold-standard for the true disease status has been extended to three diagnostic groups; e.g., Mossman (1999), Dreiseitl et al. (2000) and Heckerling (2001). Furthermore, Nakas and Yiannoutsos (2004) proposed distribution-free approaches for hypothesis testing for a single VUS and paired VUS s; Xiong et al. (2006) developed an asymptotic approach for confidence interval estimation of VUS and PVUS for normally distributed data; Nakas and Alonzo (2007) and Alonzo and Nakas (2007) proposed non-parametric inference procedures for diagnostic accuracy with three disease classes under umbrella ordering; and Xiong et al. (2007) developed a large sample approach for comparing several VUS s for normally distributed data. Most recently, Li and Fine (2008) proposed a method for ROC analysis with multiple classes.

The aim of this paper is to develop an approach for confidence interval estimation of the difference in paired VUS s and paired PVUS s based on the concepts of generalized inference. The generalized variables (GV) and generalized pivots were introduced by Tsui and Weerahandi (1989) and Weerahandi (1993); see the book by Weerahandi (2003)) or a detailed discussion. A brief summary of the concepts is included in Appendix. The concepts of generalized confidence interval and generalized P-value have been successfully applied to a variety of practical settings where standard exact solutions do not exist for confidence intervals and hypothesis testing. It has been shown that generalized variable approaches typically have good performance at small sample sizes; e.g. Weerahandi (1995), Weerahandi and Berger (1999), Krishnamoorthy and Lu (2003), Tian and Cappelleri (2004), Iyer et al. (2004), and Krishnamoorthy et al. (2009). Especially, as mentioned aforehand, GV approaches were proposed to construct an exact test for equivalence of diagnostic accuracy based on paired PAUC s by Li et al. (2007) and to estimate confidence interval of the difference in paired AUC s by Li et al. (2008) for normally distributed data.

This paper is organized as follows. Section 2 presents the preliminary knowledge about VUS and PVUS. In Section 3, the GV approaches for confidence interval estimation of the difference in paired VUS s and paired PVUS s are proposed. In Section 4, simulation results are presented to evaluate the performance of the proposed approach. In Section 5, the proposed approach is applied to a data set from a study of anemia subjects. Section 6 presents summary and discussion. Appendix contains a brief review of the basic concepts of generalized inference.

2. Preliminaries

In the following, we will briefly review the definitions of VUS and PVUS for which more details can be found in Xiong et al. (2006).

Denote Y₁, Y₂ and Y₃ to be the scores of a biomarker or the results by a diagnostic test and let F₁, F₂ and F₃ be the corresponding cumulative distribution functions for non-diseased, intermediate and diseased groups, respectively. Assume the responses are measured on a continuous scale and higher values indicate greater severity of the disease. Let c₁ and c₃ be threshold values (c₁ < c₃) for the non-diseased group and the diseased group, respectively. Following Eq. (1), the volume under the ROC surface can be proved to be equal to the probability that Y₁, Y₂ and Y₃ are in correct order, that is

$$\mathit{VUS}=P(Y_1<Y_2<Y_3)={\int }_0^1{\int }_0^{1-F_3\left[F_1^{-1}\right(p_1\left)\right]}\left\{\right[F_2\left[F_3^{-1}\right(1-p_3\left)\right]-F_2\left[F_1^{-1}\right(p_1\left)\right]\}d{p}_{3}d{p}_1.$$ (2)

The partial volume under surface is defined by

$$\mathit{PVUS}={\iint }_{D_{p_{10}{p}_{30}}}\left\{F_2\right[F_3^{-1}(1-p_3)]-F_2[F_1^{-1}\left(p_1\right)\left]\right\}d{p}_{3}d{p}_1,$$ (3)

where $D_{p_{10}{p}_{30}}=\left\{\right(p_1,p_3)\mid p_{10}\le p_1\le 1,p_{30}\le p_3\le 1-F_3[F_1^{-1}\left(p_1\right)\left]\right\}$ with p₁₀ and p₃₀ as the minimum desired rates for non-diseased and diseased groups, respectively. In other words, p₁₀ and p₃₀ represent the minimum desired specificity and the sensitivity between non-diseased and diseased groups. When non-diseased, intermediate and diseased groups can be discriminated perfectly, PVUS reach its maximum value as PVUS_max=(1 – p₁₀) (1 – p₃₀). A value of PVUS closer to PVUS_max indicate a better discrimination ability of this biomarker among three ordinal diagnostic groups. As p₁₀ = p₃₀ = 0, PVUS = VUS.

Assume $Y_i\sim \mathrm{N}({\mu }_i,{\sigma }_i^2)$ where i = 1,2,3. Define ratios a = σ₂/σ₁, b = (μ₁ – μ₂)/σ₁, c = σ₂/σ₃ and d = (μ₃ – μ₂)/σ₃. The volume under the ROC surface and the partial volume under surface as stated in Xiong et al. (2006) are

$$\mathit{VUS}={\int }_{-\infty }^{\infty }\Phi (as-b)\Phi (-cs+d)\varphi \left(s\right)ds,$$ (4)

$$\mathit{PVUS}={\int }_{\left[{\Phi }^{-1}\right(p_{10})+b]∕a}^{[d-{\Phi }^{-1}\left(p_{30}\right)]∕c}\left[\Phi \right(as-b\left)\Phi \right(-cs+d)-p\Phi (-cs+d)-q\Phi (as-b)+pq]\varphi \left(s\right)ds.$$ (5)

3. The generalized variable approach

3.1. Differences in paired VUS s and PVUS s

In this paper, we focus on the inference about the differences in paired VUS s and PVUS s for the cases with paired data. Let Y₁, Y₂ and Y₃ be two-dimensional vectors denoting the scores for markers A and B measured simultaneously for the non-diseased, intermediate, and diseased groups, respectively. Specifically, assume

$${\mathbf{Y}}_{\mathbf{i}}=\left(\begin{array}{c}\hfill Y_{iA}\hfill \\ \hfill Y_{iB}\hfill \end{array}\right)\sim {\mathrm{N}}_2({\mu }_i,{\Sigma }_i)\text{for}i=1,2,3,$$ (6)

where

$${\mu }_i=\left(\begin{array}{c}\hfill {\mu }_{iA}\hfill \\ \hfill {\mu }_{iB}\hfill \end{array}\right)\text{and}{\Sigma }_i=\left(\begin{array}{cc}\hfill {\sigma }_{iA}^2\hfill & \hfill {\sigma }_{iAB}^2\hfill \\ \hfill {\sigma }_{iAB}^2\hfill & \hfill {\sigma }_{iB}^2\hfill \end{array}\right).$$

For the marker A, we denote the ratios a_A = σ_2A/σ_1A,b_A = (μ_1A – μ_2A)/σ_1A, c_A = σ_2A/σ_3A and d_A = (μ_3A – μ_2A/σ_3A. The volume under the surface and the partial volume under the surface for marker A are

$${\mathit{VUS}}_A=P(Y_{1A}<Y_{2A}<Y_{3A})={\int }_{-\infty }^{\infty }\Phi (a_{A}s-b_A)\Phi (-C_{A}s+d_A)\varphi \left(s\right)ds,$$ (7)

$${\mathit{PVUS}}_A={\int }_{\left[{\Phi }^{-1}\right(p_{10})+b_A]∕a_A}^{[d_A-{\Phi }^{-1}(p_{30}\left)\right]∕c_A}\left[\Phi \right(a_{A}s-b_A\left)\Phi \right(-c_{A}s+d_A)-p\Phi (-c_{A}s+d_A)-q\Phi (a_{A}s-b_A)+pq]\varphi \left(s\right)ds,$$ (8)

respectively. It is clear that, for the marker B, the volume under surface VUS_B and partial volume under surface PVUS_B can be obtained by replacing A with B in Eqs. (7) and (8).

In this article, we propose methods to estimate the confidence intervals of the difference between the paired VUS and PVUS, i.e.

$$\Delta \mathit{VUS}={\mathit{VUS}}_A-{\mathit{VUS}}_B,$$ (9)

$$\Delta \mathit{PVUS}={\mathit{PVUS}}_A-{\mathit{PVUS}}_B.$$ (10)

3.2. Generalized pivots for ΔVUS and ΔPVUS

Denote N_p(μ_i,Σ_i) to be a p-variate normal distribution functions with mean vector μ_i and covariate matrix Σ_i. Assume that Y_1,1, Y_2,1, …, Y_n1,1 consist a sample from N₂(μ₁,Σ₁) for non-diseased group; Y_1,2, Y_2,2, …, Y_n2,2 consist a sample from N₂(μ₂,σ₂) for intermediate group; and Y_1,3, Y_2,3, …, Y_n3,3 consist a sample from N₂(μ₃,σ₃) for diseased group. For the ith population, let ${\overline{\mathbf{Y}}}_i$ and S_i be the sample mean vector and sample covariance matrix, respectively. It is well-known that ${\overline{\mathbf{Y}}}_i$ and S_i are mutually independent as well as

$${\stackrel{‒}{\mathbf{Y}}}_i\sim {\mathrm{N}}_2({\mu }_i,{\Sigma }_i∕n_i)\text{and}{\mathbf{U}}_i=\frac{n_i-1}{n_i}{\mathbf{S}}_i\sim W_2(n_i-1,{\Sigma }_i∕n_i),i=1,2,3,$$ (11)

where W_p(m,Σ) denotes a p-dimensional Wishart distribution with degrees of freedom m and scale matrix Σ.

The generalized pivots for μ_i can be given as (Lin et al., 2007)

$$R_{{\mu }_i}={\stackrel{‒}{\mathbf{y}}}_i-{\left({\mathbf{u}}_i^{1∕2}{\mathbf{W}}_i^{-1}{\mathbf{u}}_i^{1∕2}\right)}^{1∕2}{\mathbf{Z}}_i\text{for}i=1,2,3,$$ (12)

where Z_i ~ N₂(0,I₂) with I₂ as a 2 by 2 identity matrix and the generalized pivot for Σ_i can be given as

$$R_{{\Sigma }_i}=n_i{\mathbf{u}}_i^{1∕2}{\mathbf{W}}_i^{-1}{\mathbf{u}}_i^{1∕2}\text{for}i=1,2,3,$$ (13)

where Wi ~ W₂(n_i–1,I2). Note that

$$R_{{\mu }_i}=\left(\begin{array}{c}\hfill R_{{\mu }_{iA}}\hfill \\ \hfill R_{{\mu }_{iB}}\hfill \end{array}\right)\text{and}{R}_{{\Sigma }_i}=\left(\begin{array}{cc}\hfill R_{{\sigma }_{iA}^2}\hfill & \hfill R_{{\sigma }_{i\mathit{AB}}^2}\hfill \\ \hfill R_{{\sigma }_{i\mathit{AB}}^2}\hfill & \hfill R_{{\sigma }_{iB}^2}\hfill \end{array}\right)$$

for i = 1, 2, 3. Hence we can obtain generalized pivots R_aA,R_bA,R_cA,R_dA for a_A, b_A, c_A, d_A in the following forms:

$$R_{a_A}=\frac{R_{{\sigma }_{2A}}}{R_{{\sigma }_{1A}}},$$ (14)

$$R_{b_A}=\frac{R_{{\mu }_{1A}}-R_{{\mu }_{2A}}}{R_{{\sigma }_{1A}}},$$ (15)

$$R_{c_A}=\frac{R_{{\sigma }_{2A}}}{R_{{\sigma }_{3A}},}$$ (16)

$$R_{d_A}=\frac{R_{{\mu }_{3A}}-R_{{\mu }_{2A}}}{R_{{\sigma }_{3A}}}.$$ (17)

The generalized pivots R_VUSA and R_PVUSA for VUS and PVUS for marker A can be derived by substituting a_A, b_A, c_A, d_A with their corresponding generalized pivots R_aA,R_bA,R_cA,R_dA as follows:

$$R_{{\mathit{VUS}}_A}={\int }_{-\infty }^{+\infty }\Phi (R_{a_A}s-R_{b_A})\Phi (-R_{c_A}s+R_{d_A})\varphi \left(s\right)\mathit{ds},$$ (18)

$$R_{{\mathit{PVUS}}_A}={\int }_{\left({\Phi }^{-1}\right(p_{10})+R_{b_A})∕R_{a_A}}^{(R_{d_A}-{\Phi }^{-1}(p_{30}\left)\right)∕R_{c_A}}\left[\Phi \right(R_{a_A}s-R_{b_A}\left)\Phi \right(-R_{c_A}s+R_{d_A})-p\Phi (-R_{c_A}s+R_{d_A})-q\Phi (R_{a_A}s-R_{b_A})+\mathit{pq}]\varphi \left(s\right)\mathit{ds}.$$ (19)

For marker B, we can obtain generalized pivots R_aB,R_bB,R_cB,R_dB for a_B, b_B, c_B, d_B similarly to (14)–(17). Using R_aB,R_bB,R_cB, and R_dB, one can calculate R_PVUSB and R_VUSB in the similar way as R_VUSA and R_PVUSA.

One can easily check that R_VUSA,R_PVUSA,R_VUSB and R_PVUSB are bona fide generalized pivots as follows. For given ${\overline{\mathbf{y}}}_i$ and u_i (i = 1,2,3), the following holds: (1) the distributions of R_VUSA,R_PVUSA,R_VUSB and R_PVUSB are independent of any unknown parameters, and (2) the value of R_VUSA,R_PVUSA,R_VUSB and R_PVUSB are VUS_A, PVUS_A, VUS_B and PVUS_B as ${\overline{\mathbf{Y}}}_i={\overline{\mathbf{y}}}_i$,U_i = u_i, for i = 1,2,3.

Remark 3.1

Note that R_μi and R_Σi are defined using bivariate mean vector ${\overline{\mathbf{Y}}}_i$ and the corresponding scaled sample variance matrix U_i which incorporate markers A and B simultaneously. Therefore, the facts that markers A and B are paired and hence VUS_A and VUS_B (or PVUS_A and PVUS_B) are not independent are taken care of automatically in this proposed approach.

Furthermore, the generalized pivots for ΔPVUS and ΔVUS can be defined as

$$R_{\Delta \mathit{VUS}}=R_{{\mathit{VUS}}_A}-R_{{\mathit{VUS}}_B},$$ (20)

$$R_{\Delta \mathit{PVUS}}=R_{{\mathit{PVUS}}_A}-R_{{\mathit{PVUS}}_B}.$$ (21)

For testing the hypothesis H₀ : ΔVUS = ΔVUS₀ vs. H₁ : ΔVUS > ΔVUS₀, the generalized test variable is defined as

$$T_{\Delta \mathit{VUS}}=R_{\Delta \mathit{VUS}}-\Delta {\mathit{VUS}}_o.$$ (22)

It is clear that T_ΔVUS is a bona fide generalized test variable. In a similar manner, the generalized variable

$$T_{\Delta \mathit{PVUS}}=R_{\Delta \mathit{PVUS}}-\Delta {\mathit{PVUS}}_o.$$ (23)

for testing the hypothesis H₀ : ΔPVUS = ΔPVUS₀ vs. H₁ : ΔPVUS > ΔPVUS₀.

3.3. Computing algorithm

For a given data set containing outcomes of markers A and B which are measured simultaneously on non-diseased, intermediate and diseased groups, respectively, the generalized confidence intervals for ΔVUS and ΔPVUS can be obtained following the simulation steps:

1. Compute the sample mean vector ${\overline{\mathbf{y}}}_i$ and sample covariance matrix s_i for i = 1,2,3.

2. Generate Z_i ~ N₂(0,I₂) and W_i ~ W₂(n_i–1,I₂). Calculate R_μi and R_Σi for i = 1, 2, 3, following (12) and (13).

3. Compute R_aA,R_bA,R_cA,R_dA and R_aB,R_bB,R_cB,R_dB following (14)–(17).

4. Compute R_ΔVUS and R_ΔPVUS following (20) and (21).

5. Repeat Steps 2–5 a total m times (in general, m is set as ≥ 2000) and obtain an array of R_ΔVUS’s values and an array of R_ΔPVUS’s values.

6. Rank the array of R_ΔVUS’s and the array of R_ΔPVUS’s from small to large.

Denote R_ΔVUS(α) as the 100αth percentile of R_ΔVUS ’s. Then (R_ΔVUS(α/2),RΔVUS(1–α/2)) is a two-sided 100(1–α)% confidence interval of ΔVUS. The percentage that R_ΔVUS’s are less than or equal to ΔVUS₀ is a Monte Carlo estimate of the generalized P-value for testing ΔVUS = ΔVUS₀ vs. ΔVUS > ΔVUS₀. Similarly, the generalized P-values for testing ΔVUS = ΔVUS₀ vs. ΔVUS ≠ ΔVUS₀ can be obtained. The confidence interval estimation and hypothesis testing about ΔPVUS can be done similarly.

4. A simulation study

Simulation studies were performed to assess the coverage probabilities of the proposed confidence interval estimations for ΔVUS and ΔPVUS. We generated data from the bivariate normal distributions N₂(μ₁,σ₁,N₂(μ₂,σ₂), N₂(μ₃, σ₃) for non-diseased, intermediate and diseased groups, respectively, with μ₁ = (0,0)′,μ₂ = (2,3)′, μ₃ = (5,4)′, or μ₁ = (0,0)′, μ₂ = (0.5,0.3)′, μ₃ = (1.0,0.6)′ and with three different configurations of (Σ₁,Σ₂,Σ₃) as follw:

$$\text{config}.1:\left(\left(\begin{array}{cc}\hfill 1\hfill & \hfill 0.5\hfill \\ \hfill 0.5\hfill & \hfill 1\hfill \end{array}\right),\left(\begin{array}{cc}\hfill 1\hfill & \hfill 0.5\hfill \\ \hfill 0.5\hfill & \hfill 1\hfill \end{array}\right),\left(\begin{array}{cc}\hfill 1\hfill & \hfill 0.5\hfill \\ \hfill 0.5\hfill & \hfill 1\hfill \end{array}\right)\right),$$

$$\text{config}.2:\left(\left(\begin{array}{cc}\hfill 1\hfill & \hfill 0.5\hfill \\ \hfill 0.5\hfill & \hfill 1\hfill \end{array}\right),\left(\begin{array}{cc}\hfill 4\hfill & \hfill 2\hfill \\ \hfill 2\hfill & \hfill 4\hfill \end{array}\right),\left(\begin{array}{cc}\hfill 16\hfill & \hfill 8\hfill \\ \hfill 8\hfill & \hfill 16\hfill \end{array}\right)\right),$$

$$\text{config}.2:\left(\left(\begin{array}{cc}\hfill 1\hfill & \hfill 1\hfill \\ \hfill 1\hfill & \hfill 4\hfill \end{array}\right),\left(\begin{array}{cc}\hfill 1\hfill & \hfill 1\hfill \\ \hfill 1\hfill & \hfill 4\hfill \end{array}\right),\left(\begin{array}{cc}\hfill 1\hfill & \hfill 1\hfill \\ \hfill 1\hfill & \hfill 4\hfill \end{array}\right)\right).$$

The different configurations of (Σ₁,Σ₂,Σ₃) were chosen to represent three possible scenarios of covariance structures of the three disease groups. Config. 1 stands for the scenario with equal variances for markers A and B across three disease groups; Config. 2 stands for the scenario with equal variances for markers A and B but with increasing variances across three disease groups from non-diseased group to diseased group; Config. 3 stands for the scenario with unequal variances for markers A and B.

Table 1 presents the coverage probabilities of proposed confidence intervals for ΔVUS at nominal level 0.95 based on 2000 random samples, in comparison with those of the large sample approach (Xiong et al., 2007) and a parametric bootstrap approach. To estimate the confidence intervals by the proposed generalized variable approach, within each of the 2000 random samples, 2000 R_ΔVUS’s were calculated using the algorithms presented in Section 3.3. The large sample approach proposed by Xiong et al. (2007) for parametric comparison of VUS s can provide confidence intervals for the differences in paired VUS s. Additionally, we also consider percentile intervals by parametric bootstrap. Overall speaking, the proposed exact confidence intervals provide reasonable coverage whether or not the sample sizes are imbalanced across diagnostic groups except that it tends to be slightly conservative as sample sizes are small. When the sample sizes are the same across diagnostic groups, the large sample approach works reasonably well even as the sample sizes are quite small; however, the imbalance in sample sizes also seems to be associated with poor coverage probabilities in comparing paired VUS s, regardless of whether the covariance matrices are the same across diagnostic groups. This similar phenomenon was also observed in simulation studies regarding the coverage probability of Student t-confidence intervals when two means are compared from two normal distributions with unequal variances (Milliken and Johnson, 1992). Note that for the setting μ₁ = (0,0)′,μ₂ = (2,3)′,μ₃ = (5,4)′, some of the true VUS is large, and the Fisher z-transformation was then used in calculating the CI for the difference on VUS, as recommended by Xiong et al. (2007). The parametric bootstrap approach generally performs well; however, for scenarios with small sample sizes, it also can underestimate the coverage probabilities regardless of settings for the covariance matrices. For example, as the sample sizes equal (10, 10, 5), the coverage probabilities by the bootstrap approach can be as low as 0.90.

Table 1.

Empirical coverage probabilities (94–96% considered satisfactory) of approximate 95% two-sided confidence bounds for ΔVUS (based on 2000 simulations).

Sample sizes	Config. of Σ1,Σ2,Σ3a
	1			2			3
	Proposedb	Largec	Bootd	Proposed	Large	Boot	Proposed	Large	Boot
μ1 = (0,0)′,μ2 = (2,3)′,μ3 = (5,4)′
(5, 5, 5)	97	96	92	95	96	91	96	95	92
(10, 10, 10)	96	95	93	96	96	93	96	95	94
(10, 10, 5)	96	92	91	94	94	90	96	93	92
(20, 20, 20)	96	95	94	96	95	95	96	94	94
(20, 10, 10)	97	92	93	96	91	92	96	93	93
(30, 20, 10)	96	94	94	94	94	93	95	94	95
(50, 50, 50)	96	95	95	95	96	96	97	95	96
(50, 20, 10)	96	92	94	95	92	93	95	93	92
μ1 = (0,0)′,μ2 = (0.5,0.3)′,μ3 = (1,0.6)′
(5,5, 5)	95	93	93	95	95	91	95	91	93
(10, 10, 10)	96	95	94	95	96	93	96	95	95
(10, 10, 5)	96	95	91	94	96	90	95	94	93
(20, 20, 20)	96	96	95	94	95	95	96	93	95
(20, 10, 10)	96	87	94	96	90	93	95	89	93
(30, 20, 10)	95	91	93	95	93	92	95	92	93
(50, 50, 50)	96	94	95	95	94	95	96	95	95
(50, 20, 10)	95	87	93	95	87	93	95	88	94

^aConfig. of Σ₁,Σ₂,Σ₃: 1 :$\left(\begin{array}{cc}\hfill 1\hfill & \hfill 0.5\hfill \\ \hfill 0.5\hfill & \hfill 1\hfill \end{array}\right),\left(\begin{array}{cc}\hfill 1\hfill & \hfill 0.5\hfill \\ \hfill 0.5\hfill & \hfill 1\hfill \end{array}\right),\left(\begin{array}{cc}\hfill 1\hfill & \hfill 0.5\hfill \\ \hfill 0.5\hfill & \hfill 1\hfill \end{array}\right)$ 2 : $\left(\begin{array}{cc}\hfill 1\hfill & \hfill 0.5\hfill \\ \hfill 0.5\hfill & \hfill 1\hfill \end{array}\right),\left(\begin{array}{cc}\hfill 4\hfill & \hfill 2\hfill \\ \hfill 2\hfill & \hfill 4\hfill \end{array}\right),\left(\begin{array}{cc}\hfill 16\hfill & \hfill 8\hfill \\ \hfill 8\hfill & \hfill 16\hfill \end{array}\right)$ 3 : $\left(\begin{array}{cc}\hfill 1\hfill & \hfill 1\hfill \\ \hfill 1\hfill & \hfill 4\hfill \end{array}\right),\left(\begin{array}{cc}\hfill 1\hfill & \hfill 1\hfill \\ \hfill 1\hfill & \hfill 4\hfill \end{array}\right),\left(\begin{array}{cc}\hfill 1\hfill & \hfill 1\hfill \\ \hfill 1\hfill & \hfill 4\hfill \end{array}\right)$

^bProposed: The proposed generalized approach.

^cLarge: The large sample approach by Xiong et al. (2007).

^dBoot: The parametric bootstrap approach.

Table 2 presents simulation results for coverage probabilities of confidence intervals for ΔPVUS by the proposed approach and the parametric bootstrap approach. Note that the large sample approach for comparing paired PVUS s has not been developed, and therefore it is not included in the comparison. The desired minimum classification rates for non-diseased and diseased groups were set as p₁₀ = p₃₀ = 0:5, i.e. ΔPAUC is obtained for the region with both the minimum desired specificity and sensitivity for diseased and non-diseased groups as 0.5, similar to the setting in the asymptotic approach by Xiong et al. (2006). In general, the proposed approach works well except that for a few scenarios it tends to be slightly liberal. The parametric bootstrap approach works well for balance cases, but can slightly underestimate the coverage probabilities, e.g., as the sample sizes equal (50, 20, 10), the coverage probabilities can be as low as 0.92.

Table 2.

Empirical coverage probabilities (94–96% considered satisfactory) of approximate 95% two-sided confidence bounds for DPVUS (based on 2000 simulations).

Sample sizes	Config. of Σ1, Σ2, Σ3a
	1		2		3
	Proposedb	Bootd	Proposed	Boot	Proposed	Boot
μ1 = (0,0)′,μ2 = (2,3)′,μ3 = (5,4)′
(10, 10, 10)	96	93	94	93	94	93
(20, 20, 20)	95	94	95	94	95	94
(20, 10, 10)	96	93	94	92	94	92
(30, 20, 10)	96	93	94	93	94	93
(50, 50, 50)	95	94	95	94	95	94
(50, 20, 10)	96	93	94	93	94	92
μ1 = (0,0)′,μ2 = (0.5,0.3)′,μ3 = (1,0.6)′
(10, 10, 10)	96	94	95	93	96	95
(20, 20, 20)	96	95	96	95	96	95
(20, 10, 10)	95	94	95	93	95	94
(30, 20, 10)	94	93	95	93	95	94
(50, 50, 50)	95	95	95	95	95	95
(50, 20, 10)	95	94	95	94	95	94

Note: See bottom of Table 1 for footnotes a, b, and d.

To investigate the robustness of the proposed test, a simulation study was conducted for the mixture of normal data. The results are shown in Table 3. For each parameter setting presented, the mixture of normal data was generated as a mixture of two normal distributions; i.e.,

$$Y_i=\left(\begin{array}{c}\hfill Y_{iA}\hfill \\ \hfill Y_{iB}\hfill \end{array}\right)\sim (1-\alpha ){\mathrm{N}}_2\left({\mu }_i,\frac{1}{1.1}{\Sigma }_i\right)+\alpha {\mathrm{N}}_2\left({\mu }_i,\frac{2}{1.1}{\Sigma }_i\right),$$

where α = 0:1 for i = 1, 2, 3. Note that Y_i from such a normal mixture distribution has mean μ_i and variance Σ_i. The results presented in Table 3 show that the proposed approach generally has satisfactory coverage probabilities for normal mixture data.

Table 3.

Empirical coverage probabilities (94–96% considered satisfactory) of approximate 95% two-sided confidence bounds for ΔAUS and ΔPAUS with minimum classification rate 0.5 for both non-diseased and diseased groups for mixture of normal data: a robust study (based on 2000 simulations).

Config.a of Σ1,Σ2,Σ3		Sample sizes
		(10, 10, 10)	(20, 20, 20)	(50, 50, 50)	(50, 20, 20)	(50, 20, 10)
μ1 = (0,0)′,μ2 = (2,3)′,μ3 = (5,4)′
1	Δ VUS	96	96	96	96	97
	Δ PVUS	96	95	94	95	96
2	Δ VUS	95	95	96	96	95
	Δ PVUS	93	94	93	95	94
3	Δ VUS	96	97	97	96	95
	Δ PVUS	92	95	94	94	94
μ1 = (0,0)′,μ2 = (0.5,0.3)′,μ3 = (1.0,0.6)′
1	Δ VUS	96	96	95	96	95
	Δ PVUS	96	95	95	95	95
2	Δ VUS	95	95	96	96	95
	Δ PVUS	95	95	96	96	94
3	Δ VUS	95	96	96	95	95
	Δ PVUS	96	96	96	95	95

Note: See bottom of Table 1 for Configurations of Σ₁,Σ₂,Σ₃.

The mixture of bivariate normal is defined as ${\mathbf{Y}}_{\mathbf{i}}=\left(\begin{array}{c}\hfill Y_{iA}\hfill \\ \hfill Y_{iB}\hfill \end{array}\right)\sim (1-\alpha ){\mathrm{N}}_2({\mu }_i,(1∕1.1\left){\Sigma }_i\right)+\alpha {\mathrm{N}}_2({\mu }_i,(2∕1.1\left){\Sigma }_i\right)$ where α = 0:1 for i = 1, 2, 3. Note that Y_i from such a normal mixture distribution has mean μ_i and variance Σ_i.

Remark 4.1

As stated in Section 3, the proposed approach can easily provide P-values for hypothesis testing. An additional Simulation study shows that the proposed test has satisfying type-I error control. These simulation results are not presented in this article, but they are available upon requests.

5. An example

In this section, the proposed approach for confidence interval estimations of ΔVUS and ΔPVUS is illustrated via a data set of blood test results for patients with anemia (Wians et al., 2001). These data were also used by Obuchowski (2006). A total of 134 patients with anemia underwent a series of blood tests. To eliminate the bias which might cause by gender, we will limit our analysis to the 55 female study patients. Ferritin concentration provides a useful screening test for iron deficiency anemia (IDA). Non-pregnant women with anemia and a ferritin concentration less than 20 μg/l were assigned to the IDA group, while those with anemia and a ferritin concentration greater than 240 μg/l were assigned to be the anemia of chronic disease (ACD) group. The intermediate group consists of the women with ferritin concentration between 20 and 240 μg/l. There were 12, 14, 29 female study subjects in ACD, intermediate and IDA groups, respectively. We are interested in comparison of the diagnostic accuracy between two rapid blood tests, i.e. total iron binding capacity (TIBC) and per cent transferrin saturation (%TS), for discriminating between the ACD, intermediate and IDA groups. The multivariate normality for each group was tested by Henze–Zirkler (1990) test and not rejected with P-values as 0.14, 0.26 and 0.46 for ACD, intermediate and IDA groups, respectively.

The sample mean for ACD group is ${\overline{\mathbf{x}}}_1$ = (TIBC,%TS)′ = (214:00,6:42)′ and the sample covariance is

$${\mathbf{s}}_1=\left(\begin{array}{cc}\hfill 2803.82\hfill & \hfill -54.45\hfill \\ \hfill -54.45\hfill & \hfill 9.17\hfill \end{array}\right),$$

the sample mean for intermediate group is ${\overline{\mathbf{x}}}_2$ = (282:64,5:07)′ and the sample covariance is

$${\mathbf{s}}_2=\left(\begin{array}{cc}\hfill 3881.79\hfill & \hfill -16.90\hfill \\ \hfill -16.90\hfill & \hfill 6.69\hfill \end{array}\right),$$

and the sample mean for intermediate group is ${\overline{\mathbf{x}}}_3$ = (430:14,3:53)′ and the sample covariance is

$${\mathbf{s}}_3=\left(\begin{array}{cc}\hfill 8107.34\hfill & \hfill -56.57\hfill \\ \hfill -56.57\hfill & \hfill 3.54\hfill \end{array}\right),$$

The point estimates of VUS and PVUS are 0.7036 and 0.1218 for TIBC, and 0.3607 and 0.0114 for %TS, respectively. Note that TIBC increases from the ACD group to the IDA group while %TS decreases from ACD group to IDA group. The 95% confidence intervals by the proposed approach are (0.1103, 0.5139) for ΔVUS and (0.038, 0.1515) for ΔPVUS respectively. Both results show that TIBC has better diagnostic ability than %TS in discriminating subjects with anemia among ACD group, intermediate group and IDA group.

6. Summary and discussion

This article focuses on the confidence interval estimation of the differences in paired volumes under surfaces (VUS) and paired partial volumes (PVUS) under surfaces based on the generalized inference theory. In addition to confidence interval estimation, the proposed approach can easily provide P-values for hypothesis testing. The proposed approach is a numeric method which involves generating multivariate normal data and can be performed using a few straightforward simulation steps presented in Section 3.3. Considering the facts that (1) the large sample approach for comparing two VUS s can be very liberal; (2) the large sample approach for comparing two PVUS s has not been developed and it is expected to be not so straightforward; and (3) the parametric bootstrap approach also can underestimate the coverage probabilities, the proposed approach based on generalized inference can serve as a good candidate for confidence interval estimate of the difference of paired VUS s and paired PVUS s, especially in cases with small to medium sample sizes.

All the inference procedures based on the generalized variable theory requires parametric assumptions. For example, when there are two diagnostic groups, the generalized variable approaches proposed by Li et al. (2007, 2008) for inferences about ΔAUC and ΔPAUC are based on normality assumptions of the data distributions. In parallel, when there are three diagnostic groups, the proposed generalized variable approach for ΔVUS and ΔPVUS also utilize normality assumptions. It is well-known that AUC and PAUC are invariant measures of diagnostic accuracy under any monotonic transformation. Similarly, VUS and PVUS are also invariant measures under any monotonic transformations. Therefore, the proposed approach is expected to have wide practical applications due to the fact that it not only accommodates multivariate normal data, also any data from distributions which can be transformed into normal by monotonic transformation. In their paper (Xiong et al., 2006) on asymptotic approach based on normality for confidence interval estimation for the VUS and PVUS, Xiong et al.’s stated that it is important to check the model assumption of the original and transformed data before applying their asymptotic approach. Similarly, we recommend to check the model assumption of the original and transformed data before applying the proposed generalized variable approach. To investigate the robustness of the proposed approach, a simulation study based on mixture of multivariate normal distribution is performed and simulation results show that the proposed approach give satisfactory results.

Appendix A. Generalized pivots and generalized test variables

In the following, the basic concepts for generalized inference developed by Tsui and Weerahandi (1989) and Weerahandi (1993) are described.

Suppose that Y = (Y₁, Y₂,…,Y_n)′ form a random sample from a distribution which depends on the parameters θ = (ψν) where ψ is the parameter of interest and ν’ is a vector of nuisance parameters. A generalized pivot R(Y;y,ψ,ν), where y is an observed value of Y, for interval estimation defined in Weerahandi (1993), has the following two properties:

1. R(Y;y,ψ,ν) has a distribution free of unknown parameters.

2. The value of R(y;y,ψ,ν) is ψ.

Let that R_α be the 100_αth percentile of R. Then R_α becomes the 100(1 – α)% lower bound for ψ and (R_α/2,R_1–α/2 becomes a 100(1–α)% two-sided generalized confidence interval for ψ.

Now consider testing H₀ : ψ = ψ₀ vs. H₁ : ψ > ψ₀ where ψ₀ is a specified quantity. A generalized test variable of the form T(Y;y,ψ,ν) where y is an observed value of Y, is chosen to satisfy the following three conditions (Tsui and Weerahandi,1989):

1. For fixed y, the distribution of T(Y;y,ψ,ν) is free of the vector of nuisance parameters ν.

2. The value of T(Y;y,ψ,ν) at Y = y is free of any unknown parameters.

3. For fixed y and ν, and for all t, Pr[T(Y;y,ψ,ν) > t] is a monotonic function of ψ.

A generalized extreme region is defined as C = [Y : T(Y;y,ψ,ν) ≥ T(y;y,ψ,ν) if T(Y;y,ψ,ν) is stochastically increasing in ψ (i.e. Pr[T(Y;y,ψ,ν) > t] is a non-decreasing function of ψ). If T(Y;y,ψ,ν) is stochastically decreasing in ψ (i.e. Pr[T(Y;y,ψ,ν) > t] is a non-increasing function of ψ), a generalized extreme region is defined as C = [Y : T(Y;y,ψ,ν) ≤ T(y;y,ψ,ν). Then the generalized P-value is defined as P(Cψ).