Confidence Interval Estimation of the Youden index and corresponding cut-point for a combination of biomarkers under normality

Abstract

In prognostic/diagnostic medical research, it is often the goal to identify a biomarker that differentiates between patients with and without a condition, or patients that will have good or poor response to a given treatment. The statistical literature is abundant with methods for evaluating single biomarkers for these purposes. However, in practice, a single biomarker rarely captures all aspects of a disease process; therefore, it is often the case that using a combination of biomarkers will improve discriminatory ability. A variety of methods have been developed for combining biomarkers based on the maximization of some global measure or cost-function. These methods usually create a score based on a linear combination of the biomarkers, upon which the standard single biomarker methodologies (such as the Youden’s index) are applied. However, these single biomarker methodologies do not account for the multivariable nature of the combined biomarker score. In this article we present generalized inference and bootstrap approaches to estimating confidence intervals for the Youden’s index and corresponding cut-point for a combined biomarker. These methods account for inherent dependencies and provide accurate and efficient estimates. A simulation study and real-world example utilize data from a Duchene Muscular Dystrophy study are also presented.

1. Introduction

In practice the ROC curve is often used to evaluate the diagnostic or prognostic utility of a selected biomarker. The ROC curve plots the sensitivity versus (1-specificity) for every possible decision threshold (c) of the biomarker, and is often summarized using the area under the curve (AUC) (Pepe 2004, Obuchowski 2005). In clinical application, finding the optimal decision threshold (i.e. for differentiating between a health and disease population) from this curve is of importance. One such approach for identifying this “optimal” decision threshold is to select the value of c that maximizes the Youden index (Youden 1950), defined as:

$$J=sensitivity(c)+specificity(c)-1,$$ (1)

where −∞ < c < ∞. and can also serve as a global measure of diagnostic accuracy. The statistic J ranges from 0 to 1, with a value of 1 indicating perfect discrimination; while a value of 0 indicates the biomarker is no better than chance. With respect to the ROC curve, this index is the maximum vertical distance between the ROC curve and the diagonal chance-line, representing a global measure of diagnostic utility (Perkins and Schisterman 2006). As such, there exist a variety of approaches for estimating the Youden index and its corresponding optimal threshold in the single biomarker setting (Fluss et al. 2005; Schisterman et al. 2007; Lai et al. 2010).

In practice, however, a single biomarker often characterizes only a one aspect of a disease process and thus decisions based on just a single biomarker can be less than optimal. Therefore, it is advantageous to use a combination of biomarkers when evaluating a patient’s disease status. If the individual biomarkers are sensitive to differing aspects of the disease process, then their combination may drastically improve the diagnostic utility. Several approaches have been described for combining biomarkers within an ROC framework, where the markers are combined such that the AUC (Su & Liu 1993; Pepe & Thompson 2000; Esteban et al. 2011; Yan et al. 2015), partial-AUC (Gao et al. 2008; Wang & Chang 2001), correct classification rate (Yin and Tian 2014), or some other cost-benefit type function (Kang et al. 2016) is maximized. Perkins et al. (2013) provide a methodology when biomarkers are subject to a limit of detection. In this paper, we focus on biomarkers under normality (or for which a suitable normalizing transformation exists), for which a closed for solution exists that maximizes the AUC of the combined biomarkers (Su & Liu’s 1993).

Despite the rich literature on optimally combining biomarkers, there is very little discussion on the resulting risk score in terms of estimating the “optimal” decision threshold. Often, the standard single biomarker methods are applied to the combined biomarker; however, while this may provide an unbiased estimate for the decision threshold, it will not accurately capture the uncertainty or variability associated with that estimate. Hence, the purpose of this paper is to provide an approach for estimation of the optimal diagnostic threshold corresponding to the Youden index (c_J) for a linear combination of normally distributed biomarkers. Confidence intervals are developed using generalized inference (Weerahandi 1993) and bootstrap (Efron & Tibshirani 1986) approaches. Generalized inference has been extremely helpful in obtaining confidence intervals for “non-standard” parameters and have been applied in a variety of settings: log-normal distribution (Krishnamoorthy & Mathew 2003), repeated measures (Weerahandi 2004), intra-class correlation coefficients (Tian & Cappelleri 2004), common mean vectors (Lin et al. 2007), paired AUCs (Li et al. 2008), common correlation coefficients (Tian & Wilding 2008), the Weibull distribution (Krishnamoorthy et al. 2009), the Youden index (Lai et al. 2010), and the Bi-normal ROC curve (Yin and Tian 2016).

This paper is organized as follows. In Section 2, background is provided on the Youden index and its optimal decision threshold for the univariate and multivariate settings. In Section 3, the generalized inference and bootstrap confidence intervals for the Youden index and decision threshold are proposed. In Section 4, a simulation study evaluates the proposed approaches under normal and non-normal settings. In Section 5, the proposed approaches are applied to a Duchenne muscular dystrophy (DMD) dataset. A summary and discussion are presented in Section 6.

2. Preliminaries

In the following, we briefly review point estimation of J and c_J for a normally distributed biomarker (further details in Fluss et al. 2005 and Schisterman et al. 2007) and a combination of normally distributed biomarkers.

2.1. The Youden index for normal biomarkers

Let X₁ and X₂ denote the biomarker values for the healthy and disease populations, respectively. Assume that X₁ ~ N(μ₁, σ₁) and X₂ ~ N(μ₂, σ₂), and that they are independent. Without loss of generality we assume that μ₁ < μ₂. Under the assumption of normality, Fluss et al. (2005) and Schisterman et al. (2007) describe a closed form solution for c and J:

$$c_J=\frac{{\mu }_1(b^2-1)-a+b\sqrt{a^2+(b^2-1){\sigma }_1^{2}ln(b^2)}}{(b^2-1)},$$ (2)

$$J=\Phi \left(\frac{{\mu }_2-c_J}{{\sigma }_2}\right)+\Phi \left(\frac{c_J-{\mu }_1}{{\sigma }_1}\right)-1,$$ (3)

where a =μ₂ - μ₁, b = σ₂/σ₁, and Φ(·) denotes the standard normal cumulative distribution. When the standard deviations are equal, σ₁ = σ₂, then the optimal threshold is the midpoint between μ₁ and μ₂:

$$c_J=\frac{{\mu }_1+{\mu }_2}{2}.$$ (4)

The value of J invariant to a monotonic normalizing transformation, a trait inherited through its relationship with the ROC curve (Su & Liu 1993). The estimated c_J and J are obtained by replacing μ_i and σ_i (i=1, 2) with their maximum likelihood estimates.

2.2. The Youden index for a combination of normal biomarkers

Consider now a set of p candidate biomarkers, and let X_i ~ MVN_p(μ_i,Σ_i) be the random variables that correspond to the values of the biomarkers from the healthy (i=1) and disease (i=2) populations. Then Su & Liu (1993) proposed a form for the risk coefficients that maximize the AUC:

$$\lambda =({\Sigma }_2+{\Sigma }_1{)}^{-1}({\mu }_2-{\mu }_1).$$ (5)

The ROC curve corresponding to the combined biomarkers, or linear risk scores, Y_i = λ′X_i (i=1,2) produces an AUC that is greater than the set of AUC’s corresponding to the set all other possible linear risk scores. The estimated risk coefficients, $\widehat{\lambda }$, are obtained by replacing μ_i and Σ_i (i=1,2) with their maximum likelihood estimates:

$$\widehat{\lambda }=({\widehat{\Sigma }}_2+{\widehat{\Sigma }}_1{)}^{-1}({\widehat{\mu }}_2-{\widehat{\mu }}_1).$$ (6)

This combination method proposed by Su & Liu (1993) works reasonably well with skewed non-normal distributions, as long as the skew is not too significant.

The linear risk scores Y_i = λ′X_i are linear combinations of normal random variables. Therefore the random variables Y_i (i =1,2) are normally distributed mean μ_Yi = λ′μ_i and standard deviation ${\sigma }_{Y_i}=\sqrt{{\lambda }^{\prime }{\Sigma }_i\lambda }$. The Youden index and its corresponding optimal threshold for the combined biomarkers are then given by:

$$J=\Phi \left(\frac{{\lambda }^{\prime }{\mu }_2-c_J}{\sqrt{{\lambda }^{\prime }{\Sigma }_2\lambda }}\right)+\Phi \left(\frac{c_J-{\lambda }^{\prime }{\mu }_1}{\sqrt{{\lambda }^{\prime }{\Sigma }_1\lambda }}\right)-1\text{and}$$ (7)

$$c_J=\frac{{\lambda }^{\prime }{\mu }_1(b^2-1)-a+b\sqrt{a^2+(b^2-1){\sigma }_{Y_1}^{2}ln(b^2)}}{(b^2-1)},$$ (8)

where a = (μ_Y2 − μ_Y1) = (λ′μ₂ − λ′μ₁) and $b={\sigma }_{Y_2}∕{\sigma }_{Y_1}=\sqrt{{\lambda }^{\prime }{\Sigma }_2\lambda }∕\sqrt{{\lambda }^{\prime }{\Sigma }_1\lambda }$. Confidence intervals for J and c_J could be obtained by applying the large sample method of Schisterman et al. (2007) on the combined markers. However, the resulting confidence intervals will be too narrow and provide less than nominal coverage because this approach does not account for the uncertainty in the coefficients used to generate the linear risk scores.

3. Confidence Intervals for the Youden index

In this section we propose generalized and bootstrap confidence intervals of the Youden index and its corresponding optimal decision threshold for a linear combination of normal biomarkers.

3.1. A generalized confidence interval

Let ${\widehat{\mu }}_{ı}$ and ${\widehat{\Sigma }}_{ı}$ be, respectively, the observed values of μ_i and Σ_i (i=1,2). The generalized pivotal quantity for estimating μ_i can be expressed by:

$${\mathit{R}}_{{\mu }_i}={\overline{\mathit{x}}}_i-{\left({\mathit{u}}_i^{1∕2}{\mathit{W}}_i^{-1}{\mathit{u}}_i^{1∕2}\right)}^{1∕2}{\mathit{Z}}_i,$$ (9)

where W_i and Z_i are realizations from a Wishart(n_i − 1, I_p) and p-variate standard normal distributions, respectively, and

$${\mathit{u}}_i=\frac{n_i-1}{n_i}{\mathit{s}}_i,$$ (10)

where s_i is the sample covariance matrix (i=1,2). The generalized pivotal quantity for estimating Σ_i can be expressed as:

$${\mathit{R}}_{{\Sigma }_i}=n_i{\mathit{u}}_i^{1∕2}{\mathit{W}}_i^{-1}{\mathit{u}}_i^{1∕2},$$ (11)

where the W_i can be the same realization from equation (9) or a new realization from a Wishart(n_i − 1,I_p) (Lin et al. 2007).

The generalized pivotal quantity R_λ for λ is obtained by substituting μ_i and Σ_i (i=1,2) in equation (5) with corresponding generalized pivotal quantities (Tian 2010):

$${\mathit{R}}_{\lambda }=({\mathbf{R}}_{{\Sigma }_2}+{\mathbf{R}}_{{\Sigma }_1}{)}^{-1}({\mathbf{R}}_{{\mu }_2}-{\mathbf{R}}_{{\mu }_1}).$$ (12)

The generalized pivotal quantities R_a and R_b for a and b can be obtained by substituting λ, μ_i and Σ_i (i=1,2) with corresponding generalized pivotal quantities:

$$R_a={\mathit{R}}_{\lambda }^{\prime }({\mathit{R}}_{{\mu }_2}-{\mathit{R}}_{{\mu }_1}),$$ (13)

$$R_b=\sqrt{\frac{{\mathit{R}}_{\lambda }^{\prime }\cdot {\mathit{R}}_{{\Sigma }_2}\cdot {\mathit{R}}_{\lambda }}{{\mathit{R}}_{\lambda }^{\prime }\cdot {\mathit{R}}_{{\Sigma }_1}\cdot {\mathit{R}}_{\lambda }}.}$$ (14)

The generalized pivotal quantities R_c and R_J for c_J and J can be obtained by substituting a, b, λ, μ_i and Σ_i (i=1,2) in equations (7) and (8) with their corresponding generalized pivotal quantities. Consequently, the generalized pivotal quantity for the optimal cut-point, Rc, with unequal variances, is given by:

$$R_c=\frac{{\mathit{R}}_{\lambda }^{\prime }\cdot {\mathit{R}}_{{\mu }_1}(R_b^2-1)+R_a+R_b\sqrt{R_a^2+(R_b^2-1)\cdot ({\mathit{R}}_{\lambda }^{\prime }\cdot {\mathit{R}}_{{\Sigma }_1}\cdot {\mathit{R}}_{\lambda })\cdot \text{ln}(R_b^2)}}{R_b^2-1}.$$ (15)

When the variances are equal, equation (15) is undefined and it can be replaced by:

$$R_c=\frac{{\mathit{R}}_{\lambda }^{\prime }({\mathit{R}}_{{\mu }_1}+R_{{\mu }_2})}{2}.$$ (16)

After substituting λ, μ_i, Σ_i (i=1,2) and c_J with their generalized pivotal quantities in equations (12), (9), (11) and (15), respectively; the generalized pivotal quantity for the Youden index can be derived as:

$$R_J=\Phi \left(\frac{R_c-{\mathit{R}}_{\lambda }^{\prime }\cdot {\mathit{R}}_{{\mu }_1}}{\sqrt{{\mathit{R}}_{\lambda }^{\prime }\cdot {\mathit{R}}_{{\Sigma }_1}\cdot {\mathit{R}}_{\lambda }}}\right)-\Phi \left(\frac{R_c-{\mathit{R}}_{\lambda }^{\prime }\cdot {\mathit{R}}_{{\mu }_2}}{\sqrt{{\mathit{R}}_{\lambda }^{\prime }\cdot {\mathit{R}}_{{\Sigma }_2}\cdot {\mathit{R}}_{\lambda }}}\right).$$ (17)

It is easy to check that R_J and R_c satisfy the two conditions necessary for them to be the generalized pivotal quantities, as described in Lai et al. (2010). For given ${\widehat{\mu }}_{ı}$ and ${\widehat{\Sigma }}_{ı}$ (i = 1, 2): (1) the distributions of R_J and R_c are independent of any unknown parameters; and (2) the values of R_J and R_c are J and c_J, respectively, as ${\mu }_i={\widehat{\mu }}_{ı}$ and ${\Sigma }_i={\widehat{\Sigma }}_{ı}$ for i = 1, 2.

Let R_J(p) and R_c(p) denote the p^th quantiles of the distributions of R_J and R_c, respectively. Then the 100 (1-α) % generalized confidence intervals for J and c_J based on R_J and R_c are given by: (R_J(α/2), R_J(1−α/2)) and (R_c(α/2), R_c(1−α/2)), respectively. The distributions of R_J and R_c are estimated by simulation as described below.

Computing Algorithm

For a given dataset x_i1, … , x_ini ( i=1,2), the generalized confidence intervals are computed using the following algorithm.

1. Compute the sample statistics ${\overline{\mathit{x}}}_i$ and u_i, i=1,2

2. For k=1,…K

   • Generate Z_i ~ MVN(0, I_p) and W_i ~ W_p(n_i − 1, I_p), i=1,2.
   • Compute R_μi and R_Σi, i=1,2, following (9) and (11).
   • Compute R_λ following (12).
   • Compute: R_c,k and R_J,k following (15) – (17).
     (end k loop)
3. Let R_J(p) and R_c(p) denote the p^th percentiles of the R_J,k′s and R_c,k′s, for k=1 to K, respectively:

   • The (1-α)·100% generalized confidence interval for J is given by: (R_J(α/2), R_J(1−α/2)).
   • The (1-α)·100% generalized confidence interval for c_J is given by: (R_c(α/2), R_c(1−α/2)).

3.2. A bootstrap confidence interval

Parametric bootstrap confidence intervals are proposed for J and c_J using the normal approximation (BSN), percentile (BSP) and bias corrected (BSBC) methods described in Efron & Tibshirani (1986).

For a given dataset x_i1, … , x_ini (i=1,2), the algorithm for obtaining 100(1-α)% bootstrap confidence intervals about J and c for combined normal biomarkers is given by the following:

1. Compute the sample statistics ${\overline{\mathit{x}}}_i$ and s_i, i=1,2.

2. For k = 1 to K:

   • Generate: ${\mathit{y}}_{ij}\sim MVN(\mu ={\overline{\mathit{x}}}_i,\Sigma ={\mathit{s}}_i)$, i=1,2 and j=1,…,n_i
   • Compute ${\widehat{\mu }}_i=\frac{1}{n_i}{\sum }_{j=1}^{n_i}{\mathit{y}}_{ij}$ and ${\widehat{\Sigma }}_i=\frac{1}{n_i-1}{\sum }_{j=1}^{n_i}({\mathit{y}}_{ij}-{\widehat{\mu }}_i)({\mathit{y}}_{ij}-{\widehat{\mu }}_i{)}^{\prime }$, for i=1,2.
   • Compute $\widehat{\lambda }={\left({\widehat{\Sigma }}_1+{\widehat{\Sigma }}_2\right)}^{-1}({\widehat{\mu }}_2-{\widehat{\mu }}_1)$
   • Compute $c_k=\frac{\widehat{\lambda }({\widehat{\mu }}_1+{\widehat{\mu }}_2)}{2}\text{if}{\widehat{\lambda }}^{\prime }{\widehat{\Sigma }}_1\widehat{\lambda }={\widehat{\lambda }}^{\prime }{\widehat{\Sigma }}_2\widehat{\lambda }$,

     • Else compute $c_k=\frac{{\widehat{\lambda }}^{\prime }{\widehat{\mu }}_1({\widehat{b}}^2-1)+\widehat{a}+\widehat{b}\sqrt{{\widehat{a}}^2+({\widehat{b}}^2-1)\cdot ({\widehat{\lambda }}^{\prime }\cdot {\widehat{\Sigma }}_1\cdot \widehat{\lambda })\cdot \text{ln}({\widehat{b}}^2)}}{{\widehat{b}}^2-1}$.
     • Where $\widehat{a}={\widehat{\lambda }}^{\prime }({\widehat{\mu }}_2-{\widehat{\mu }}_1)$ and $\widehat{b}=\sqrt{\frac{{\widehat{\lambda }}^{\prime }{\widehat{\Sigma }}_2\widehat{\lambda }}{{\widehat{\lambda }}^{\prime }{\widehat{\Sigma }}_1\widehat{\lambda }}}$.
   • Compute $J_k=\Phi \left(\frac{c_b-{\widehat{\lambda }}^{\prime }\cdot {\widehat{\mu }}_1}{\sqrt{{\widehat{\lambda }}^{\prime }{\widehat{\Sigma }}_1\widehat{\lambda }}}\right)-\Phi \left(\frac{c_b-{\widehat{\lambda }}^{\prime }\cdot {\widehat{\mu }}_2}{\sqrt{{\widehat{\lambda }}^{\prime }{\widehat{\Sigma }}_2\widehat{\lambda }}}\right)$.
     (end k loop)
3. BSN Interval: Let ${\tilde{\sigma }}_J$ and ${\tilde{\sigma }}_c$ denote the standard deviation of the J_k′s and c_k′s, for k =1 to K, respectively:

   • The (1-α)·100% BS large sample confidence interval for J is given by: ${\overline{J}}_K\pm Z_{(1-\alpha ∕2)}\cdot {\tilde{\sigma }}_J$.
   • The (1-α)·100% BS large sample confidence interval for c_J is given by: ${\overline{c}}_K\pm Z_{(1-\alpha ∕2)}\cdot {\tilde{\sigma }}_c$.
4. BSP Interval: Let J_(p) and c_(p) denote the p^th quantiles of the J_k′s and c_k′s, for k =1 to K, respectively:

   • The (1-α)·100% BS percentile confidence interval for J is given by: (J_(α/2), J_(1-α/2)).
   • The (1-α)·100% BS percentile confidence interval for c_J is given by: (c_(α/2), c_(1−α/2)).
5. BSC Interval: Let $z_{0,J}={\Phi }^{-1}\left({\widehat{G}}_J(\widehat{J})\right)$ and $z_{0,c}={\Phi }^{-1}\left({\widehat{G}}_c(\widehat{c})\right)$, where $\widehat{J}$ and $\widehat{c}$ are the sample J and c_J, and ${\widehat{G}}_J(\cdot )$ and ${\widehat{G}}_c(\cdot )$ are the bootstrap CDFs for J and c_J.

   • The (1-α)·100% BS bias corrected confidence interval for J is given by: $\left({\widehat{G}}_J^{-1}\left(\Phi (2z_{0,J}+z_{\alpha ∕2})\right),{\widehat{G}}_J^{-1}\left(\Phi (2z_{0,J}+z_{1-\alpha ∕2})\right)\right)$.
   • The (1-α)·100% BS bias corrected confidence interval for c_J is given by: $\left({\widehat{G}}_c^{-1}\left(\Phi (2z_{0,c}+z_{\alpha ∕2})\right),{\widehat{G}}_c^{-1}\left(\Phi (2z_{0,c}+z_{1-\alpha ∕2})\right)\right)$.

The R code for these algorithms can be found in Research Gate under DOI: 10.13140/RG.2.2.11740.33926.

4. Simulation Study

In this section a simulation study is preformed to evaluate the coverage probabilities and mean lengths of the confidence intervals based on the generalized inference (GI) as compared to those based on the parametric bootstrap methods (BSN, BPS and BSC). A range of sample sizes, and both balanced and unbalanced designs are considered. Twelve different scenarios for the joint distributions consisting of two or three candidate biomarkers (p=2, 3) are presented.

Scenarios 1–5 consider moderate discriminatory ability for two biomarkers, with the mean vectors given by: ${\mu }_1=\left(\begin{array}{c}\hfill 0.1\hfill \\ \hfill 0.1\hfill \end{array}\right)$ and ${\mu }_2=\left(\begin{array}{c}\hfill 1.1\hfill \\ \hfill 0.7\hfill \end{array}\right)$. Scenarios 1–3 examine performance under equal variability (Σ₁ = Σ₂ = Σ), with $\Sigma =\left(\begin{array}{cc}\hfill 0.3\hfill & \hfill 1\hfill \\ \hfill 1\hfill & \hfill 0.3\hfill \end{array}\right)$, $\Sigma =\left(\begin{array}{cc}\hfill 0.5\hfill & \hfill 1\hfill \\ \hfill 1\hfill & \hfill 0.5\hfill \end{array}\right)$ and $\Sigma =\left(\begin{array}{cc}\hfill 0.7\hfill & \hfill 1\hfill \\ \hfill 1\hfill & \hfill 0.7\hfill \end{array}\right)$ for scenarios 1, 2 and 3, respectively. Scenarios 4 and 5 examine performance with unequal variances such that ${\Sigma }_1=\left(\begin{array}{cc}\hfill 0.3\hfill & \hfill 1\hfill \\ \hfill 1\hfill & \hfill 0.3\hfill \end{array}\right)$, and ${\Sigma }_2=\left(\begin{array}{cc}\hfill 0.5\hfill & \hfill 1\hfill \\ \hfill 1\hfill & \hfill 0.5\hfill \end{array}\right)$ for scenario 4 and ${\Sigma }_2=\left(\begin{array}{cc}\hfill 0.7\hfill & \hfill 1\hfill \\ \hfill 1\hfill & \hfill 0.7\hfill \end{array}\right)$ for scenario 5.

Scenarios 6 and 7 consider unequal variances, using the same covariance matrices in scenarios 4 and 5, but examine performance when only one biomarker has significant discriminatory ability. That is, the mean vectors are given by: ${\mu }_1=\left(\begin{array}{c}\hfill 0.1\hfill \\ \hfill 0.1\hfill \end{array}\right)$ and ${\mu }_2=\left(\begin{array}{c}\hfill 0.3\hfill \\ \hfill 1.7\hfill \end{array}\right)$.

Scenarios 8-12 consider models with three candidate biomarkers, with the mean vectors given by: ${\mu }_1=\left(\begin{array}{c}\hfill 0.1\hfill \\ \hfill 0.1\hfill \\ \hfill 0.1\hfill \end{array}\right)$ and ${\mu }_2=\left(\begin{array}{c}\hfill 1.1\hfill \\ \hfill 1.7\hfill \\ \hfill 0.3\hfill \end{array}\right)$. Scenarios 8-10 examine performance under equal variability, where $\Sigma =\left(\begin{array}{ccc}\hfill 1\hfill & \hfill 0.3\hfill & \hfill 0.3\hfill \\ \hfill 0.3\hfill & \hfill 1\hfill & \hfill 0.3\hfill \\ \hfill 0.3\hfill & \hfill 0.3\hfill & \hfill 1\hfill \end{array}\right)$, $\Sigma =\left(\begin{array}{ccc}\hfill 1\hfill & \hfill 0.5\hfill & \hfill 0.5\hfill \\ \hfill 0.5\hfill & \hfill 1\hfill & \hfill 0.5\hfill \\ \hfill 0.5\hfill & \hfill 0.5\hfill & \hfill 1\hfill \end{array}\right)$ and $\Sigma =\left(\begin{array}{ccc}\hfill 1\hfill & \hfill 0.7\hfill & \hfill 0.7\hfill \\ \hfill 0.7\hfill & \hfill 1\hfill & \hfill 0.7\hfill \\ \hfill 0.7\hfill & \hfill 0.7\hfill & \hfill 1\hfill \end{array}\right)$ for scenarios 8, 9 and 10, respectively. Scenarios 11 and 12 examine performance with unequal variances such that ${\Sigma }_1=\left(\begin{array}{ccc}\hfill 1\hfill & \hfill 0.3\hfill & \hfill 0.3\hfill \\ \hfill 0.3\hfill & \hfill 1\hfill & \hfill 0.3\hfill \\ \hfill 0.3\hfill & \hfill 0.3\hfill & \hfill 1\hfill \end{array}\right)$, and ${\Sigma }_2=\left(\begin{array}{ccc}\hfill 1\hfill & \hfill 0.5\hfill & \hfill 0.5\hfill \\ \hfill 0.5\hfill & \hfill 1\hfill & \hfill 0.5\hfill \\ \hfill 0.5\hfill & \hfill 0.5\hfill & \hfill 1\hfill \end{array}\right)$ for scenario 11 and ${\Sigma }_2=\left(\begin{array}{ccc}\hfill 1\hfill & \hfill 0.7\hfill & \hfill 0.7\hfill \\ \hfill 0.7\hfill & \hfill 1\hfill & \hfill 0.7\hfill \\ \hfill 0.7\hfill & \hfill 0.7\hfill & \hfill 1\hfill \end{array}\right)$ for scenario 12.

For each scenario, we generate multivariate observations from the underlying distributions using the R statistical software program; where a total of 2000 iterations, with K=2000 random numbers and bootstraps were generated in each iteration. For each iteration, 95% confidence interval estimates are obtained using the methods described in Section 3. The estimated coverage rate and mean interval length are then calculated for each simulation scenario.

4.1. Results for the Youden index

The simulations results for J are presented in Tables 1 and 2. The GI method consistently provided nominal coverage regardless of the sample size or design for the equal variance scenarios 1–3 and 8–10. The bootstrap methods provided poor coverage in small samples sizes and the BSC method provided nominal coverage in the large sample case, but the BSN and BSP methods performed poorly within the sample sizes studied. Although not presented, these methods do achieve nominal coverage as the sample sizes increase significantly. The GI method generally produced confidence intervals that were wider, but the difference in interval widths erodes as the sample sizes increased.

Table 1:

Simulation results for the Youden index (1-6)

		Interval Coverage				Interval Width					Interval Coverage				Interval Width
Scenario	Sample Size	GI	BSC	BSN	BSP	GI	BSC	BSN	BSP	Scenario	GI	BSC	BSN	BSP	GI	BSC	BSN	BSP
1	N=M=10	0.966	0.902	0.828	0.860	0.544	0.525	0.522	0.513	4	0.967	0.903	0.832	0.859	0.550	0.530	0.533	0.525
	N=M=20	0.957	0.928	0.887	0.904	0.406	0.402	0.398	0.396		0.956	0.927	0.893	0.905	0.412	0.407	0.405	0.402
	N=M=30	0.952	0.933	0.909	0.919	0.337	0.335	0.333	0.331		0.953	0.937	0.906	0.914	0.342	0.340	0.338	0.337
	N=M=40	0.954	0.946	0.919	0.929	0.292	0.291	0.290	0.289		0.946	0.932	0.926	0.927	0.298	0.297	0.297	0.295
	N=M=50	0.958	0.950	0.936	0.941	0.263	0.262	0.261	0.261		0.956	0.941	0.927	0.927	0.268	0.267	0.267	0.266
	N=M=100	0.949	0.945	0.939	0.940	0.188	0.187	0.187	0.187		0.952	0.950	0.938	0.942	0.191	0.191	0.191	0.190
	N=50, M=20	0.952	0.934	0.904	0.914	0.345	0.341	0.339	0.338		0.954	0.937	0.904	0.912	0.350	0.346	0.345	0.344
	N=50, M=30	0.948	0.934	0.910	0.916	0.302	0.301	0.299	0.298		0.952	0.942	0.923	0.927	0.308	0.307	0.306	0.304
	N=50, M=40	0.952	0.937	0.925	0.933	0.279	0.278	0.277	0.276		0.945	0.933	0.916	0.922	0.283	0.283	0.282	0.281
2	N=M=10	0.965	0.899	0.825	0.857	0.547	0.527	0.530	0.521	5	0.966	0.899	0.832	0.859	0.548	0.528	0.529	0.521
	N=M=20	0.953	0.921	0.886	0.900	0.411	0.407	0.404	0.402		0.953	0.924	0.888	0.902	0.411	0.406	0.404	0.401
	N=M=30	0.950	0.937	0.908	0.916	0.341	0.339	0.338	0.336		0.953	0.934	0.909	0.915	0.341	0.339	0.338	0.336
	N=M=40	0.951	0.936	0.914	0.921	0.297	0.296	0.295	0.294		0.955	0.940	0.932	0.934	0.297	0.296	0.295	0.294
	N=M=50	0.947	0.938	0.928	0.931	0.267	0.266	0.266	0.265		0.957	0.950	0.940	0.946	0.267	0.266	0.266	0.265
	N=M=100	0.958	0.947	0.937	0.941	0.190	0.190	0.190	0.189		0.953	0.947	0.946	0.948	0.191	0.190	0.190	0.189
	N=50, M=20	0.961	0.934	0.907	0.915	0.349	0.344	0.343	0.342		0.942	0.919	0.885	0.888	0.344	0.342	0.340	0.339
	N=50, M=30	0.949	0.933	0.915	0.920	0.307	0.305	0.304	0.303		0.953	0.938	0.920	0.923	0.304	0.304	0.302	0.301
	N=50, M=40	0.953	0.939	0.923	0.928	0.283	0.282	0.281	0.280		0.948	0.934	0.916	0.921	0.281	0.280	0.279	0.278
3	N=M=10	0.965	0.890	0.821	0.853	0.548	0.528	0.531	0.522	6	0.964	0.900	0.829	0.859	0.545	0.527	0.526	0.517
	N=M=20	0.960	0.929	0.895	0.907	0.413	0.408	0.406	0.404		0.958	0.934	0.889	0.905	0.409	0.405	0.402	0.399
	N=M=30	0.950	0.934	0.908	0.918	0.342	0.340	0.339	0.337		0.946	0.932	0.907	0.915	0.339	0.337	0.335	0.333
	N=M=40	0.952	0.936	0.919	0.926	0.298	0.297	0.297	0.295		0.958	0.944	0.921	0.924	0.295	0.294	0.293	0.292
	N=M=50	0.955	0.941	0.936	0.940	0.268	0.267	0.266	0.265		0.945	0.931	0.922	0.927	0.265	0.264	0.264	0.263
	N=M=100	0.952	0.946	0.943	0.943	0.191	0.190	0.191	0.190		0.954	0.950	0.937	0.938	0.189	0.188	0.189	0.188
	N=50, M=20	0.951	0.932	0.904	0.911	0.351	0.347	0.345	0.344		0.954	0.939	0.906	0.913	0.344	0.342	0.340	0.338
	N=50, M=30	0.959	0.947	0.926	0.932	0.309	0.307	0.306	0.305		0.950	0.930	0.912	0.920	0.304	0.302	0.301	0.300
	N=50, M=40	0.954	0.939	0.921	0.924	0.284	0.283	0.282	0.281		0.944	0.932	0.910	0.918	0.280	0.279	0.278	0.277

GI = generalized inference method, BSN = bootstrap normal approximation, BSP = bootstrap percentile method and BSC = bootstrap bias corrected method.

Table 2:

Simulation results for the Youden index (7-12)

		Interval Coverage				Interval Width					Interval Coverage				Interval Width
Scenario	Sample Size	GI	BSC	BSN	BSP	GI	BSC	BSN	BSP	Scenario	GI	BSC	BSN	BSP	GI	BSC	BSN	BSP
7	N=M=10	0.958	0.886	0.719	0.766	0.514	0.505	0.494	0.483	10	0.960	0.886	0.710	0.760	0.513	0.503	0.486	0.474
	N=M=20	0.953	0.919	0.830	0.849	0.392	0.394	0.388	0.385		0.956	0.923	0.832	0.855	0.390	0.391	0.383	0.380
	N=M=30	0.948	0.925	0.874	0.884	0.329	0.330	0.327	0.325		0.955	0.936	0.875	0.889	0.327	0.328	0.323	0.322
	N=M=40	0.958	0.943	0.895	0.902	0.288	0.289	0.287	0.286		0.947	0.934	0.894	0.902	0.286	0.286	0.284	0.283
	N=M=50	0.943	0.929	0.899	0.907	0.259	0.260	0.259	0.258		0.952	0.941	0.905	0.911	0.257	0.258	0.256	0.255
	N=M=100	0.947	0.941	0.927	0.930	0.186	0.187	0.186	0.186		0.952	0.946	0.938	0.940	0.185	0.184	0.184	0.183
	N=50, M=20	0.955	0.931	0.871	0.884	0.336	0.336	0.333	0.331		0.953	0.930	0.858	0.870	0.337	0.337	0.333	0.331
	N=50, M=30	0.951	0.934	0.888	0.897	0.297	0.298	0.295	0.294		0.951	0.931	0.893	0.901	0.298	0.298	0.295	0.294
	N=50, M=40	0.951	0.934	0.897	0.905	0.274	0.275	0.273	0.272		0.955	0.941	0.904	0.914	0.274	0.274	0.272	0.271
8	N=M=10	0.960	0.885	0.713	0.761	0.514	0.505	0.494	0.483	11	0.956	0.881	0.703	0.755	0.515	0.504	0.492	0.480
	N=M=20	0.952	0.921	0.837	0.858	0.393	0.395	0.389	0.386		0.951	0.916	0.834	0.856	0.394	0.394	0.388	0.385
	N=M=30	0.952	0.930	0.883	0.889	0.330	0.331	0.328	0.326		0.950	0.929	0.864	0.877	0.329	0.330	0.326	0.325
	N=M=40	0.952	0.939	0.897	0.906	0.288	0.289	0.287	0.286		0.940	0.926	0.892	0.901	0.288	0.289	0.287	0.285
	N=M=50	0.953	0.945	0.911	0.917	0.260	0.261	0.260	0.259		0.950	0.939	0.913	0.918	0.260	0.260	0.259	0.258
	N=M=100	0.946	0.940	0.923	0.924	0.186	0.187	0.187	0.186		0.953	0.950	0.929	0.931	0.186	0.187	0.186	0.186
	N=50, M=20	0.953	0.932	0.876	0.889	0.337	0.337	0.334	0.332		0.947	0.922	0.866	0.873	0.339	0.340	0.337	0.335
	N=50, M=30	0.947	0.932	0.884	0.890	0.297	0.298	0.296	0.295		0.950	0.933	0.887	0.893	0.299	0.300	0.298	0.297
	N=50, M=40	0.950	0.940	0.898	0.904	0.275	0.276	0.274	0.273		0.949	0.945	0.894	0.901	0.276	0.276	0.275	0.274
9	N=M=10	0.961	0.888	0.717	0.763	0.505	0.496	0.470	0.458	12	0.960	0.887	0.712	0.763	0.508	0.501	0.481	0.470
	N=M=20	0.957	0.921	0.841	0.863	0.382	0.383	0.374	0.370		0.954	0.927	0.840	0.858	0.387	0.390	0.382	0.379
	N=M=30	0.952	0.930	0.878	0.892	0.319	0.321	0.316	0.314		0.953	0.932	0.876	0.888	0.324	0.326	0.322	0.320
	N=M=40	0.950	0.935	0.900	0.908	0.279	0.280	0.277	0.276		0.951	0.936	0.885	0.893	0.283	0.285	0.282	0.281
	N=M=50	0.946	0.940	0.897	0.905	0.251	0.251	0.250	0.249		0.947	0.935	0.904	0.912	0.256	0.256	0.255	0.254
	N=M=100	0.947	0.943	0.922	0.927	0.179	0.180	0.179	0.179		0.950	0.938	0.928	0.931	0.183	0.183	0.183	0.182
	N=50, M=20	0.953	0.936	0.868	0.882	0.326	0.327	0.321	0.320		0.940	0.920	0.852	0.868	0.323	0.324	0.320	0.318
	N=50, M=30	0.947	0.936	0.886	0.894	0.287	0.288	0.285	0.284		0.948	0.930	0.883	0.896	0.289	0.290	0.288	0.286
	N=50, M=40	0.958	0.949	0.912	0.919	0.266	0.267	0.265	0.264		0.942	0.931	0.900	0.904	0.269	0.270	0.268	0.267

GI = generalized inference method, BSN = bootstrap normal approximation, BSP = bootstrap percentile method and BSC = bootstrap bias corrected method.

In the unequal variances scenarios (4–7, 11 and 12) similar results are observed. The GI method produces slightly wider interval estimates that provide nominal coverage regardless of the sample size or design. The bootstrap methods generally perform poorly, producing narrower interval estimates at the sacrifice of interval coverage. In the large sample size cases the BSC method tends to provide nominal coverage, but no benefit in interval width over the GI method.

4.2. Results for the optimal threshold

The results pertaining to c_J are presented in Tables 3 and 4. In the equal variance scenarios (1–3 and 8–10) the GI method provided nominal coverage for the balanced sample sizes, but was slightly liberal when the sample sizes were unbalanced. The BSC method produced interval estimates with nominal coverage with relatively small sample sizes, and its coverage was unaffected by unbalanced designs. The BSN and BSP methods were too conservative and liberal, respectively, in small sample size settings; but tended towards nominal coverage as the sample sizes increases. The GI method consistently produced significantly narrow interval estimates as compared to the bootstrap methods, at times sacrificing coverage. The BSC method generally produced narrower interval estimates as compared to the other bootstrap methods, BSN and BSP.

Table 3:

Simulation results for the optimal threshold (1-6)

		Interval Coverage				Interval Width					Interval Coverage				Interval Width
Scenario	Sample Size	GI	BSC	BSN	BSP	GI	BSC	BSN	BSP	Scenario	GI	BSC	BSN	BSP	GI	BSC	BSN	BSP
1	N=M=10	0.945	0.950	0.990	0.935	1.442	2.015	2.293	2.227	4	0.959	0.945	0.995	0.926	1.329	1.835	2.115	2.053
	N=M=20	0.942	0.946	0.973	0.940	0.956	1.118	1.178	1.170		0.954	0.937	0.977	0.937	0.879	1.018	1.086	1.078
	N=M=30	0.945	0.946	0.965	0.942	0.765	0.851	0.879	0.876		0.953	0.940	0.970	0.943	0.701	0.778	0.810	0.808
	N=M=40	0.953	0.952	0.966	0.949	0.661	0.718	0.736	0.734		0.945	0.943	0.964	0.946	0.594	0.648	0.668	0.666
	N=M=50	0.950	0.949	0.964	0.949	0.586	0.628	0.639	0.638		0.947	0.937	0.960	0.941	0.528	0.571	0.584	0.583
	N=M=100	0.952	0.954	0.960	0.954	0.408	0.427	0.431	0.430		0.949	0.942	0.956	0.942	0.372	0.395	0.399	0.399
	N=50, M=20	0.949	0.948	0.966	0.936	0.856	0.951	1.011	1.005		0.959	0.943	0.976	0.938	0.786	0.870	0.933	0.928
	N=50, M=30	0.947	0.941	0.963	0.934	0.719	0.783	0.812	0.810		0.957	0.941	0.967	0.939	0.655	0.711	0.742	0.740
	N=50, M=40	0.952	0.952	0.962	0.948	0.637	0.684	0.701	0.700		0.948	0.941	0.960	0.941	0.583	0.628	0.648	0.646
2	N=M=10	0.942	0.947	0.989	0.933	1.358	1.913	2.151	2.094	5	0.961	0.934	0.996	0.921	1.359	1.833	2.173	2.106
	N=M=20	0.946	0.944	0.975	0.941	0.897	1.054	1.106	1.099		0.959	0.937	0.979	0.934	0.881	1.016	1.101	1.093
	N=M=30	0.951	0.946	0.969	0.943	0.717	0.803	0.827	0.825		0.949	0.938	0.969	0.939	0.700	0.782	0.825	0.822
	N=M=40	0.946	0.947	0.967	0.940	0.612	0.670	0.686	0.684		0.955	0.940	0.965	0.944	0.599	0.659	0.686	0.684
	N=M=50	0.957	0.951	0.969	0.945	0.546	0.589	0.599	0.598		0.962	0.951	0.967	0.949	0.535	0.585	0.604	0.602
	N=M=100	0.950	0.949	0.955	0.951	0.381	0.402	0.406	0.405		0.950	0.942	0.956	0.943	0.372	0.404	0.410	0.409
	N=50, M=20	0.953	0.941	0.967	0.933	0.802	0.897	0.952	0.947		0.954	0.926	0.971	0.919	0.789	0.875	0.952	0.947
	N=50, M=30	0.954	0.949	0.965	0.938	0.674	0.733	0.757	0.755		0.957	0.933	0.967	0.936	0.659	0.721	0.761	0.759
	N=50, M=40	0.953	0.948	0.962	0.944	0.598	0.646	0.661	0.660		0.958	0.940	0.964	0.939	0.586	0.640	0.666	0.664
3	N=M=10	0.945	0.943	0.990	0.931	1.338	1.890	2.119	2.063	6	0.953	0.943	0.992	0.926	1.397	1.924	2.229	2.161
	N=M=20	0.950	0.946	0.976	0.941	0.878	1.033	1.084	1.077		0.954	0.946	0.978	0.937	0.912	1.063	1.135	1.127
	N=M=30	0.947	0.948	0.970	0.948	0.698	0.784	0.806	0.804		0.954	0.946	0.972	0.943	0.730	0.816	0.852	0.849
	N=M=40	0.951	0.947	0.965	0.947	0.598	0.658	0.672	0.671		0.961	0.950	0.971	0.941	0.628	0.690	0.712	0.710
	N=M=50	0.954	0.951	0.970	0.953	0.531	0.574	0.584	0.583		0.945	0.935	0.957	0.940	0.551	0.601	0.617	0.615
	N=M=100	0.951	0.946	0.957	0.945	0.372	0.394	0.398	0.397		0.949	0.948	0.956	0.945	0.389	0.416	0.422	0.421
	N=50, M=20	0.952	0.943	0.962	0.934	0.788	0.880	0.932	0.927		0.955	0.933	0.967	0.922	0.806	0.900	0.966	0.961
	N=50, M=30	0.959	0.958	0.975	0.954	0.656	0.716	0.739	0.737		0.945	0.936	0.954	0.938	0.679	0.743	0.778	0.775
	N=50, M=40	0.953	0.949	0.967	0.946	0.585	0.632	0.646	0.645		0.955	0.940	0.965	0.938	0.608	0.661	0.683	0.681

GI = generalized inference method, BSN = bootstrap normal approximation, BSP = bootstrap percentile method and BSC = bootstrap bias corrected method.

Table 4:

Simulation results for the optimal threshold (7-12)

		Interval Coverage				Interval Width					Interval Coverage				Interval Width
Scenario	Sample Size	GI	BSC	BSN	BSP	GI	BSC	BSN	BSP	Scenario	GI	BSC	BSN	BSP	GI	BSC	BSN	BSP
7	N=M=10	0.947	0.959	0.997	0.919	1.547	2.298	2.820	2.744	10	0.951	0.957	0.997	0.909	1.579	2.277	2.906	2.811
	N=M=20	0.953	0.952	0.981	0.933	0.989	1.177	1.291	1.283		0.950	0.950	0.980	0.931	1.014	1.170	1.301	1.290
	N=M=30	0.952	0.953	0.976	0.941	0.786	0.882	0.934	0.931		0.947	0.950	0.972	0.939	0.801	0.872	0.930	0.927
	N=M=40	0.957	0.959	0.974	0.941	0.669	0.732	0.765	0.762		0.946	0.958	0.978	0.941	0.681	0.720	0.754	0.752
	N=M=50	0.952	0.949	0.961	0.943	0.591	0.638	0.660	0.658		0.944	0.948	0.962	0.939	0.608	0.631	0.655	0.653
	N=M=100	0.945	0.948	0.957	0.944	0.411	0.431	0.438	0.437		0.947	0.951	0.961	0.946	0.423	0.423	0.432	0.430
	N=50, M=20	0.958	0.952	0.974	0.925	0.871	0.972	1.082	1.076		0.952	0.943	0.974	0.909	0.913	0.988	1.118	1.109
	N=50, M=30	0.961	0.950	0.968	0.936	0.735	0.803	0.857	0.854		0.947	0.950	0.971	0.943	0.751	0.793	0.854	0.850
	N=50, M=40	0.948	0.946	0.962	0.933	0.651	0.703	0.735	0.734		0.951	0.948	0.963	0.937	0.668	0.696	0.732	0.730
8	N=M=10	0.951	0.962	0.998	0.914	1.527	2.282	2.783	2.708	11	0.953	0.946	0.998	0.901	1.546	2.180	2.855	2.758
	N=M=20	0.954	0.949	0.981	0.933	0.970	1.154	1.262	1.255		0.946	0.944	0.982	0.924	0.974	1.114	1.255	1.245
	N=M=30	0.948	0.951	0.970	0.936	0.762	0.858	0.906	0.904		0.946	0.942	0.970	0.929	0.774	0.840	0.905	0.902
	N=M=40	0.951	0.950	0.967	0.942	0.655	0.716	0.747	0.745		0.940	0.949	0.971	0.938	0.658	0.699	0.738	0.736
	N=M=50	0.955	0.949	0.968	0.940	0.581	0.625	0.647	0.645		0.949	0.941	0.959	0.946	0.585	0.612	0.638	0.637
	N=M=100	0.942	0.945	0.952	0.946	0.402	0.422	0.429	0.428		0.949	0.949	0.957	0.948	0.405	0.413	0.421	0.421
	N=50, M=20	0.959	0.945	0.973	0.927	0.856	0.956	1.062	1.056		0.954	0.932	0.977	0.910	0.882	0.951	1.087	1.080
	N=50, M=30	0.955	0.947	0.967	0.937	0.715	0.782	0.834	0.831		0.947	0.938	0.965	0.930	0.730	0.773	0.837	0.834
	N=50, M=40	0.954	0.950	0.969	0.936	0.637	0.688	0.719	0.718		0.944	0.941	0.965	0.934	0.642	0.674	0.712	0.711
9	N=M=10	0.945	0.959	0.997	0.913	1.681	2.464	3.091	2.995	12	0.957	0.955	0.997	0.907	1.625	2.399	2.979	2.895
	N=M=20	0.953	0.956	0.980	0.929	1.074	1.265	1.396	1.385		0.957	0.949	0.979	0.926	1.016	1.215	1.347	1.337
	N=M=30	0.949	0.948	0.972	0.938	0.849	0.943	1.004	1.000		0.957	0.946	0.971	0.932	0.806	0.917	0.982	0.978
	N=M=40	0.952	0.956	0.968	0.948	0.726	0.785	0.823	0.820		0.957	0.949	0.966	0.937	0.682	0.762	0.803	0.800
	N=M=50	0.943	0.944	0.963	0.940	0.643	0.685	0.712	0.709		0.948	0.945	0.954	0.932	0.602	0.666	0.695	0.692
	N=M=100	0.948	0.941	0.952	0.941	0.449	0.465	0.474	0.473		0.952	0.947	0.955	0.941	0.419	0.456	0.466	0.465
	N=50, M=20	0.956	0.946	0.966	0.920	0.939	1.048	1.174	1.165		0.955	0.939	0.964	0.908	0.878	1.001	1.121	1.113
	N=50, M=30	0.958	0.948	0.968	0.931	0.795	0.864	0.927	0.923		0.954	0.939	0.961	0.916	0.734	0.823	0.885	0.881
	N=50, M=40	0.945	0.951	0.964	0.943	0.700	0.752	0.789	0.787		0.952	0.942	0.959	0.931	0.655	0.727	0.768	0.765

GI = generalized inference method, BSN = bootstrap normal approximation, BSP = bootstrap percentile method and BSC = bootstrap bias corrected method.

In the unequal variances scenarios (4-7, 11 and 12) similar results are observed. In the balanced sample size scenarios, the GI method produces interval estimates with nominal coverage that significantly narrower than those of the bootstrap methods. However, when the sample sizes are unbalanced, the interval estimates produced by the GI method can be slightly liberal. In these cases the BSC method provides reasonable coverage with interval estimates that are significantly wider than those of the GI method. The BSN and BSP methods require larger sample sizes before they achieve nominal coverage, and their corresponding interval estimates tend to be wider than those of the BSC method.

4.3. Robustness Evaluation

The robustness of the proposed procedures when the underlying distributions violate the normality assumption was investigated via simulation. Two additional scenarios were considered: in scenario 13 the biomarkers follow a bivariate T-distribution (df=20) with mean vectors ${\mu }_1=\left(\begin{array}{c}\hfill 0\hfill \\ \hfill 0.2\hfill \end{array}\right)$ and ${\mu }_2=\left(\begin{array}{c}\hfill 0.8\hfill \\ \hfill 1.0\hfill \end{array}\right)$, and covariance matrix $\Sigma =\left(\begin{array}{cc}\hfill 1.0\hfill & \hfill 0.2\hfill \\ \hfill 0.2\hfill & \hfill 1.1\hfill \end{array}\right)$. In scenarios 14 the biomarkers follow multivariate skewed normal distribution, where mean vectors and covariance matrix are as described in Scenario 13 and the skewness vector is given by $\tau =\left(\begin{array}{c}\hfill 3\hfill \\ \hfill 0\hfill \end{array}\right)$. The random samples are generated using the “rmvst” and “rmvsnorm” functions available in the fMultivar package for R (Wuertz et al. 2009).

The results for the interval coverage and width corresponding to J and c_J are presented in Table 5. When distributions of the biomarkers have heavier tails as compared to that of the normal distribution (scenario 13), the GI and BSC methods tend to provide below nominal coverage for J and c_J. The BSN method provides conservative coverage for J and c_J; while the BSP method is conservative for J and fails to provide nominal coverage for c_J. In the presence of a single skewed biomarker (scenario 14) the coverage probability for J is below the nominal level for all approaches, but appears to improve with increased sample sizes. For c_J, the methods appear to be converging to the nominal level as the sample sizes increase; where the BSN and BSP methods tend to be conservative, while the GI and BSC methods are below the nominal level.

Table 5:

Simulation results for the non-normal scenarios (13 & 14)

		Interval Coverage for J				Interval Width for J				Interval Coverage for c				Interval Width for c
Scenario	Sample Size	GI	BSC	BSN	BSP	GI	BSC	BSN	BSP	GI	BSC	BSN	BSP	GI	BSC	BSN	BSP
13	N=M=10	0.925	0.864	0.982	0.969	0.558	0.483	0.573	0.565	0.932	0.869	0.996	0.916	0.697	1.342	1.098	1.086
	N=M=20	0.905	0.897	0.974	0.975	0.396	0.371	0.425	0.421	0.919	0.877	0.995	0.877	0.370	0.557	0.537	0.536
	N=M=30	0.930	0.875	0.961	0.974	0.332	0.322	0.360	0.356	0.923	0.883	0.995	0.885	0.267	0.396	0.407	0.403
	N=M=40	0.938	0.909	0.977	0.964	0.288	0.290	0.320	0.316	0.899	0.881	0.993	0.894	0.211	0.319	0.336	0.331
	N=M=50	0.928	0.905	0.969	0.988	0.257	0.265	0.291	0.288	0.896	0.896	0.981	0.886	0.176	0.278	0.297	0.291
	N=M=100	0.918	0.930	0.954	0.974	0.183	0.207	0.219	0.217	0.909	0.905	0.963	0.888	0.108	0.190	0.208	0.203
	N=50, M=20	0.902	0.886	0.965	0.974	0.311	0.319	0.359	0.356	0.898	0.874	0.988	0.880	0.229	0.337	0.356	0.350
	N=50, M=30	0.907	0.893	0.932	0.947	0.288	0.297	0.326	0.322	0.889	0.895	0.987	0.905	0.230	0.338	0.354	0.349
	N=50, M=40	0.913	0.901	0.903	0.920	0.268	0.279	0.305	0.302	0.886	0.897	0.988	0.899	0.198	0.300	0.320	0.315
14	N=M=10	0.974	0.897	0.840	0.865	0.736	0.530	0.534	0.525	0.936	0.836	0.986	0.968	2.604	4.796	4.033	3.926
	N=M=20	0.940	0.923	0.903	0.913	0.653	0.409	0.408	0.405	0.924	0.872	0.981	0.972	1.727	2.181	2.085	2.070
	N=M=30	0.929	0.933	0.927	0.931	0.602	0.341	0.340	0.339	0.919	0.898	0.962	0.957	1.389	1.598	1.571	1.565
	N=M=40	0.911	0.925	0.925	0.931	0.568	0.299	0.298	0.297	0.921	0.915	0.964	0.957	1.197	1.316	1.309	1.304
	N=M=50	0.912	0.940	0.938	0.941	0.538	0.268	0.268	0.267	0.924	0.928	0.955	0.948	1.062	1.145	1.144	1.141
	N=M=100	0.920	0.925	0.939	0.938	0.419	0.192	0.192	0.192	0.928	0.941	0.959	0.942	0.742	0.772	0.774	0.772
	N=50, M=20	0.916	0.944	0.939	0.941	0.604	0.350	0.349	0.347	0.919	0.914	0.964	0.956	1.283	1.419	1.411	1.404
	N=50, M=30	0.911	0.935	0.940	0.946	0.571	0.308	0.307	0.306	0.924	0.910	0.968	0.957	1.278	1.415	1.406	1.400
	N=50, M=40	0.909	0.939	0.944	0.949	0.553	0.284	0.284	0.283	0.935	0.933	0.966	0.957	1.146	1.246	1.241	1.237

J = Youden index and c = optimal threshold corresponding to the Youden index.

GI = generalized inference method, BSN = bootstrap normal approximation, BSP = bootstrap percentile method and BSC = bootstrap bias corrected method.

5. Duchene Muscular Dystrophy Example

The proposed methods were applied to a dataset involving Duchene Muscular Dystrophy (DMD). The DMD dataset is currently available from the Carnegie Mellon University Statlib Datasets Archive (http://lib.stat.cmu.edu/datasets/biomed.desc). The data was originally discussed by Cox et al. (1982), and analyzed in the context of ROC with measurement errors by Reiser (2000), partial area under the ROC curve (PAUC) by Tian (2010) and the Youden index by Fluss et al. (2005). It is known that DMD patients generally have little physical symptoms, but tend to have increased levels of certain serum enzymes.

The data comes from a program aimed at developing an effective method for screening potential female DMD carriers. The sample contains 38 DMD carriers and 87 healthy or normal patients, from which blood samples were obtained and four different serum enzyme levels measured. A log-transformation was applied to improve normality of the four enzyme measurements (Anderson-Darling p-values > 0.1 for all enzymes, except for M2 in the healthy patients [p<0.01]). The log serum enzymes were combined using the method proposed by Su & Liu (1993) and 95% confidence intervals for J and c_J were obtained using the proposed methods. Results are presented in Table 6.

Table 6:

DMD Example

Method	Cut-point cJ	CI width	Youden index J	CI width
GI	31.5 (18.9 – 42.5)	23.6	0.75 (0.59 – 0.84)	0.25
BSN	33.2 (20.4 – 46.0)	25.6	0.76 (0.65 – 0.87)	0.22
BSP	31.5 (21.9 – 48.3)	26.4	0.75 (0.65 – 0.87)	0.22
BSC	31.5 (22.3 – 48.0)	25.7	0.75 (0.63 – 0.86)	0.23

GI = generalized inference method, BSN = bootstrap normal approximation, BSP = bootstrap percentile method and BSC = bootstrap bias corrected method.

The confidence intervals associated with J are narrower for the BSN, BSP and BSC as compared to GI. In the simulation study it was also observed that, under normality, the narrower bootstrap confidence intervals came with a sacrifice to interval coverage. The confidence intervals for the optimal threshold are all similar, where the GI has the smallest interval width and BSP has the largest. These results are consistent with observations made in the simulation study.

6. Discussion

The rapid pace and relative ease at which biomarkers are becoming available to researchers and clinicians is creating important statistical issues in diagnostic testing. While extensive literature exists for combining biomarkers, there has been little discussion on how this impacts the selection of a diagnostic threshold. In this manuscript we addressed one such issue by proposing generalized inference and bootstrap confidence interval estimates for the Youden index and corresponding cut-point in the presence of combined normal biomarkers. The proposed methods were based on the assumption that the biomarkers can be combined in a linear fashion using the coefficients proposed by Su and Liu (1993). Utilizing this method for combination has the added benefits that the resulting ROC curve of the combined biomarkers is invariant to monotonic transformations or standardization of the individual biomarkers (Su & Liu 1993). However, there is a limitation in efficiency and robustness when the number of potential biomarkers becomes large relative to the sample size; i.e. n is not a magnitude greater than p (Yan et al. 2015).

A simulation study evaluated the proposed confidence intervals with respect to interval coverage and width. The generalized inference approach provided reasonable interval coverage for both the Youden index and its decision threshold; approaching the nominal level even with small sample sizes. The intervals based on the generalized inference approach tended to have narrower widths for the decision threshold, but wider widths for the Youden index as compared to those of the bootstrap approaches. The interval coverage for the generalized inference method was consistently around the nominal level, as was the case for the bias corrected bootstrap method. The bootstrap percentile and normal approximation methods had, at times, issues with under and over coverage, respectively. Additional scenarios considered biomarkers with heavier tailed or slightly skewed distributions, and may better reflect the real-world application of these methods. In general, these methods became conservative or failed to achieve the specified confidence level. In the presence of biomarkers with a skewed distribution, the methods appear to converge towards the specified confidence level; while this was not the case for biomarkers with heavier tailed distributions. This may indicate an implicit bias in these methods, especially in small sample sizes.

When confronted with heavy tailed or highly skewed biomarkers, investigators may be inclined to combine biomarkers using logistic regression. In this setting, confidence interval estimates for J and c_J may be obtained using large sample theory or bootstrap methods. However, these methods may not perform as well when a normalizing transformation exists, as the approach by Su & Liu (1993) provides the combination with a dominating ROC curve.

The proposed methods were applied to a DMD dataset in which serum enzymes were being considered as markers for identifying DMD carriers. The enzymes were combined in a linear fashion and the proposed methods were then used to estimate the Youden index and corresponding optimal threshold. The resulting interval estimates further supported the results observed in the simulation study.

In this paper we have proposed generalized inference and bootstrap confidence interval estimates for the Youden index and its optimal decision threshold in the presence of combined normal biomarkers. While the generalized inference approach is based on assumed normality, it provided interval estimates with nominal coverage and was generally robust against minor deviations from normality. Therefore, in the applied setting, these methods are reasonable to consider when examining diagnostic scores based on a linear combination of biomarkers.