A test for comparing conditional ROC curves with multidimensional covariates

Abstract

The comparison of Receiver Operating Characteristic (ROC) curves is frequently used in the literature to compare the discriminatory capability of different classification procedures based on diagnostic variables. The performance of these variables can be sometimes influenced by the presence of other covariates, and thus they should be taken into account when making the comparison. A new non-parametric test is proposed here for testing the equality of two or more dependent ROC curves conditioned to the value of a multidimensional covariate. Projections are used for transforming the problem into a one-dimensional approach easier to handle. Simulations are carried out to study the practical performance of the new methodology. The procedure is then used to analyse a real data set of patients with Pleural Effusion to compare the diagnostic capability of different markers.

1. Introduction

In any classification problem such as a diagnostic method – in which the aim is to discriminate between two populations, usually identified as the healthy population and the diseased population – the main concern is to minimize the number of subjects that are misclassified. Receiver Operating Characteristic (ROC) curves are commonly used in this context for studying the behaviour of the classification variables [see, for example, the monograph of 19, as an introduction to the topic]. They combine the notions of sensitivity (the ability of classifying a diseased patient as diseased) and specificity (the ability of classifying a healthy individual as healthy), two measurements that can be expressed in terms of the cumulative distribution functions of the diagnostic variables of the diseased and the healthy populations.

When there is more than one variable for diagnosing a certain disease one can compare their respective ROC curves in order to decide whether their discriminatory capability is different or not. This is what happens in the medical example that we will be analysing in this paper, a real data set containing the information of patients with pleural effusion. In this data set there are two variables (the carbohydrate antigen 152 and the cytokeratin fragment 21-1) that can be used for deciding whether that pleural effusion is due to the presence of a malignant tumour or not. The objective of the analysis will be to compare the diagnostic capability of those markers.

There are several methodologies discussed in the literature for making that sort of comparisons [for a review of such methodologies, see 6], although most of them do not consider the possible effect that the presence of covariates can have in the performance of the test. In the example provided, apart from the diagnostic variables there are other covariates such as the age or the neuron-specific enolase of the patients. It is important to take this information into account, because the diagnostic capability of a marker may change with the value of a covariate [17].

In this paper the aim is to propose a test to compare ROC curves that includes the presence of a multidimensional covariate in the analysis. With this methodology, given a new patient waiting for a diagnosis on his or her pleural effusion, we could compare the different diagnosis mechanisms taking into account the covariate values of this particular individual and see which one would be the most appropriate.

One way of introducing the effect of the covariates into the study is by using the conditional ROC curve. If we consider $Y^F$ and $Y^G$ as the continuous diagnostic markers in the diseased and healthy populations, respectively, ${\mathit{X}}^F={(X_1^F,\dots ,X_d^F)}^{\mathrm{\prime }}$ as the continuous $d-$dimensional covariate of the diseased population and ${\mathit{X}}^G={(X_1^G,\dots ,X_d^G)}^{\mathrm{\prime }}$ as the continuous $d-$dimensional covariate of the healthy population, then, given a fixed value $\mathit{x}={(x_1,\dots ,x_d)}^{\mathrm{\prime }}\in {\mathit{R}}_{\mathit{X}}$ (where ${\mathit{R}}_{\mathit{X}}$ is the intersection of ${\mathit{R}}_{{\mathit{X}}^{\mathit{F}}}$ and ${\mathit{R}}_{{\mathit{X}}^{\mathit{G}}}$, the supports of ${\mathit{X}}^F$ and ${\mathit{X}}^G$, and is assumed to be non-empty), the conditional ROC curve is defined as

$$RO{C}^{\mathit{x}}(p)=1-F(G^{-1}(1-p|\mathit{x})|\mathit{x}),p\in (0,1),$$ (1)

where $F(y|\mathit{x})=P(Y^F⩽y|{\mathit{X}}^F=\mathit{x})$, and $G(y|\mathit{x})=P(Y^G⩽y|{\mathit{X}}^G=\mathit{x})$.

By comparing these conditional ROC curves instead of the standard ROC curves it is possible to incorporate the potential effect of the covariates in the analysis of the equivalence of two or more methods of diagnosis. A test for performing this comparison is proposed in [7] for the case of a continuous one-dimensional covariate. The objective here is to extend that methodology to the case in which we have a multidimensional covariate. Thus, the aim is to test, given a certain $\mathit{x}\in {\mathit{R}}_{\mathit{X}}$,

$$H_0:RO{C}_1^{\mathit{x}}(p)=\cdots =RO{C}_K^{\mathit{x}}(p)\text{for all}p\in (0,1),$$ (2)

where K is the number of diagnostic markers (and thus, ROC curves) that are being compared. In this context we would have K diagnostic variables and one $d-$dimensional covariate in the healthy population, $({\mathit{X}}^F,Y_1^F,\dots ,Y_K^F)$, and similar variables in the diseased population, $({\mathit{X}}^G,Y_1^G,\dots ,Y_K^G)$. In practice this kind of test could help to design a more personalized diagnostic method based on the covariate values of each patient. With this methodology, in the medical example at hand we could determine whether the carbohydrate antigen 152 and the cytokeratin fragment 21-1 are equally suitable for the diagnosis of a patient with a certain age and a certain enolase value.

In order to be able to make this comparison, we are going to rely on the estimation of the corresponding conditional ROC curves. There is a wide range of estimation methods in the literature: some of them estimate the conditional distribution functions involved in the definition of the conditional ROC curve, others use regression functions to include the effect of the covariates (following direct or indirect approaches). See [17] for a further review of this topic.

In [7] the estimation of the conditional ROC curve that is used is based on the indirect (or induced) regression methodology, which incorporates the covariate information through regression models by considering the effect of those covariates in the diagnostic marker in each population of healthy or diseased separately. However, this method was originally designed for one single covariate. One could think of extending that methodology by changing the estimator of the conditional ROC curve for another capable of handling multidimensional covariates. Nevertheless, there are not many methods in the literature capable of considering more than one covariate when estimating the conditional ROC curve, and most of them have some parametric assumptions that we would like to avoid making. See [11] as an example of a non-parametric Bayesian model to estimate the conditional distribution functions involved in the ROC curves, [13], [21] or [22] as examples of a direct ROC regression model (where generalized additive models are used to directly regress the ROC curve) or [20] as an example of induced methodology (framed in a Bayesian setting). In our case we will be following a frequentist approach.

The tests related to multidimensional data tend to become less powerful when the dimension of the problem increases. This is why, in this paper, the problem of comparing conditional ROC curves is first transformed using projections in such a way that the multidimensional problem becomes a unidimensional problem easier to handle. This idea has been applied several times in the literature for reducing the dimension in goodness-of-fit problems [see, for example, 5,8,18], but, to the best of our knowledge, it is the first time that it is applied on an ROC curve setting. In the last few years random projections are increasingly being used as a way to overcome the curse of dimensionality. The characterization of the multidimensional distribution of the original data by the distribution of the randomly projected unidimensional data is what allows for the reduction of the dimension.

To that end, in Section 2, which is dedicated to the exposition of the methodology introduced in this paper, we begin by showing how (2) can be transformed in a test with one-dimensional covariates using projections. In Section  2.2 we show how to compare ROC curves in that simplified setting, with a unidimensional covariate and in Section  2.3, which contains the major contribution of this paper, we propose a methodology for testing that equivalent hypothesis with a multi-dimensional covariate. This includes the proposal of a bootstrap algorithm to approximate the distribution of the statistic under the null hypothesis. In Section 3 the results from a simulation study show the practical performance of the test in terms of level approximation and power. The procedure is illustrated in Section 4 by analysing the real data set containing information of patients with pleural effusion.

2. Methodology

This section is divided in three subsections. In the first one, Section  2.1, we present a result that allows us to transform the problem discussed in (2) into an equivalent one, easier to handle, by using projections to reduce the multidimensional role of the covariate to a unidimensional one.

In Section 2.2 we show a methodology to test the equality of conditional ROC curves on a unidimensional problem [based on the one proposed in 7]. Finally, in Section  2.3, we combine that methodology with the result obtained in Section  2.1 to solve our original problem with multidimensional covariates. Both Sections 2.2 and 2.3 include the statistic proposed to perform the test and a bootstrap algorithm to approximate its distribution.

2.1. An equivalent problem

In order to present the transformation of the problem, first we need to introduce the definition of the ROC curve conditioned to a pair $(x^F,x^G)\in R_{X^F}\times R_{X^G}$:

$$RO{C}^{x^F,x^G}(p)=1-F(G^{-1}(1-p|x^G)|x^F),p\in (0,1).$$ (3)

This concept is very similar to the conditional ROC curve (1): the only difference is that this new definition allows us to condition on different values for the diseased and healthy populations. In this case $x^F$ and $x^G$ are unidimensional, but the definition could be applied on a multidimensional case. Even if the interpretability of this new ROC curve is not very clear in practice, theoretically it does not present any problems (as it will not do its estimation), as the population of healthy and diseased are always considered to be independent.

The following result is the base for developing the test for comparing ROC curves with multidimensional covariates. It borrows the ideas in [5] of using projections for reducing the dimension of the covariate in a regression context. Since here we are dealing with ROC curves, the dimension reduction is less straightforward and some adjustments are required, as each ROC depends on two cumulative distribution functions. To the best of our knowledge, the idea of using projections has not been considered in the context of ROC curves.

Given $\mathit{x},\mathit{\beta }\in {\mathbb{R}}^d$, ${\mathit{x}}^{\mathbf{\prime }}\mathit{\beta }$ denotes the scalar product of the vectors $\mathit{x}$ and $\mathit{\beta }$. For now on, all the vectors representing the projections will be considered to be contained in the d-dimensional unit sphere ${\mathbb{S}}^{d-1}=\{\mathit{\beta }\in {\mathbb{R}}^d:||\mathit{\beta }||=1\}$. This way we ensure that all possible directions are equally important.

Lemma 2.1

Assume $\mathbb{E}|Y_k^F|<\mathrm{\infty }$ and $\mathbb{E}|Y_k^G|<\mathrm{\infty }$ for every $k\in \{1,\dots ,K\}$. Then, given a certain $\mathit{x}\in {\mathit{R}}_{\mathit{X}}$, and assuming dependence among the ROC curves (meaning the covariate is common for all the K curves considered), then

$$RO{C}_1^{\mathit{x}}(p)=\cdots =RO{C}_K^{\mathit{x}}(p)\text{for all }p\in (0,1)a.s.$$

if and only if

$$RO{C}_1^{({\mathit{\beta }}^F{)}^{\mathrm{\prime }}\mathit{x},({\mathit{\beta }}^G{)}^{\mathrm{\prime }}\mathit{x}}(p)=\cdots =RO{C}_K^{({\mathit{\beta }}^F{)}^{\mathrm{\prime }}\mathit{x},({\mathit{\beta }}^G{)}^{\mathrm{\prime }}\mathit{x}}(p)\text{for all }p\in (0,1)a.s.\text{for any}{\mathit{\beta }}^F,{\mathit{\beta }}^G,$$

where ${\mathit{\beta }}^F$ and ${\mathit{\beta }}^G$ are $d-$dimensional coordinates in ${\mathbb{S}}^{d-1}$ that represent the directions of the projections.

The proof of this Lemma can be found in the Appendix. Note that ${({\mathit{\beta }}^F)}^{\mathrm{\prime }}\mathit{x}$ and ${({\mathit{\beta }}^G)}^{\mathrm{\prime }}\mathit{x}$ are one-dimensional values. By using these ROC curves conditioned to a pair of projected covariates (as defined in 3), the problem is reduced to a one-dimensional covariate conditional ROC curve comparison test for each possible direction ${\mathit{\beta }}^F$ and ${\mathit{\beta }}^G$.

Thus, taking advantage of the result in Lemma 2.1, instead of testing for the null hypothesis (2), we may use this equivalent formulation to develop a methodology that, given a certain $\mathit{x}\in R_{\mathit{X}}$, tests

$$H_0:RO{C}_1^{({\mathit{\beta }}^F{)}^{\mathrm{\prime }}\mathit{x},({\mathit{\beta }}^G{)}^{\mathrm{\prime }}\mathit{x}}(p)=\cdots =RO{C}_K^{({\mathit{\beta }}^F{)}^{\mathrm{\prime }}\mathit{x},({\mathit{\beta }}^G{)}^{\mathrm{\prime }}\mathit{x}}(p)\text{for all }p\in (0,1)\mathrm{\forall }{\mathit{\beta }}^F,{\mathit{\beta }}^G$$ (4)

against the general alternative $H_1:$ $H_0$ is not true. The notation ∀ will be used instead of ‘for any’ to shorten the expression (this applies mainly in the proofs found in the Appendix).

Note that values of the projections do not have a meaning on their own: there is not optimal direction of the coefficients to be found, they are all equally important. This methodology should not be confused with the search for the best linear combination of markers for developing new diagnostic tests, like the ones proposed in [23] or in [12]. Here we are combining the components of a multidimensional covariate to perform a test about the performance of two or more markers.

In a first step, a statistic for testing the equivalence of these ROC curves is presented for a certain pair of fixed projections, and then that statistic is adapted to include all possible directions.

2.2. Test for a one-dimensional covariate

The objective in this section is to develop a test for the equivalent problem presented in Lemma 2.1 for a fixed pair of projections ${\mathit{\beta }}^F$ and ${\mathit{\beta }}^G$. Here a test is presented for comparing two or more dependent ROC curves conditioned to two one-dimensional values. Given the pair $(x^F,x^G)\in R_{X^F}\times R_{X^G}$, the aim is then to test

$$H_0:RO{C}_1^{x^F,x^G}(p)=\cdots =RO{C}_K^{x^F,x^G}(p)\text{for all }p\in (0,1)$$ (5)

against the general alternative $H_1:H_0$ is not true.

The samples available in this context are:

• $\{(X_i^F,Y_{1,i}^F,\dots ,Y_{K,i}^F){\}}_{i=1}^{n^F}$ an i.i.d. sample from the distribution of $(X^F,Y_1^F,\dots ,Y_K^F)$,

• $\{(X_i^G,Y_{1,i}^G,\dots ,Y_{K,i}^G){\}}_{i=1}^{n^G}$ an i.i.d. sample from the distribution of $(X^G,Y_1^G,\dots ,Y_K^G)$,

with $n^F$ and $n^G$ the sample sizes of the diseased and healthy populations, respectively. Define $n=n^F+n^G$ as the total sample size used for the estimation of each conditional ROC curve (that will be the same for all $k\in \{1,\dots ,K\}$). Note that both $X^F$ and $X^G$ are here one-dimensional covariates.

The method used for the estimation of the conditional ROC curves is based on the one proposed in [9], which relies on non-parametric location-scale regression models. To be more precise, for each $k=1,\dots ,K$, assume that

$$\begin{array}{rl}{Y}_k^F& ={\mu }_k^F(X^F)+{\sigma }_k^F(X^F){\epsilon }_k^F\end{array}$$ (6)

$$\begin{array}{rl}{Y}_k^G& ={\mu }_k^G(X^G)+{\sigma }_k^G(X^G){\epsilon }_k^G\end{array}$$ (7)

where, for $D\in \{F,G\}$, ${\mu }_k^D(\cdot )=E(Y_k^D|X^D=\cdot )$ and $({\sigma }_k^D{)}^2(\cdot )=Var(Y_k^D|X^D=\cdot )$ are the conditional mean and the conditional variance functions (both of them unknown smooth functions), and the error ${\epsilon }_k^D$ is independent of $X^D$. The dependence structure between the K diagnostic variables is modelled by introducing a dependence structure between the errors: $({\epsilon }_1^D,\dots ,{\epsilon }_K^D)$ will follow a multivariate distribution function with zero mean and a covariance matrix with ones in the diagonal.

Given this location-scale regression model structure for the diagnostic variables, the k-th ROC curve conditioned to a pair of values $(x^F,x^G)\in R_{X^F}\times R_{X^G}$ can be expressed in terms of the marginal cumulative distribution functions of the errors, $H_k^F$ and $H_k^G$:

$$RO{C}_k^{x^F,x^G}(p)=1-H_k^F({\left(H_k^G\right)}^{-1}(1-p)b_k(x^F,x^G)-a_k(x^F,x^G)),$$ (8)

where

$$a_k(x^F,x^G)=\frac{{\mu }_k^F(x^F)-{\mu }_k^G(x^G)}{{\sigma }_k^F(x^F)}\mathrm{and}{b}_k(x^F,x^G)=\frac{{\sigma }_k^G(x^G)}{{\sigma }_k^F(x^F)}.$$

Thus, this k-th conditional ROC curve can be estimated by

$${\widehat{ROC}}_k^{x^F,x^G}(p)=1-\int {\hat{H}}_k^F({\left({\hat{H}}_k^G\right)}^{-1}(1-p+h_{k}u){\hat{b}}_k(x^F,x^G)-{\hat{a}}_k(x^F,x^G))\kappa (u)\mathrm{d}u,$$ (9)

where, for $D\in \{F,G\}$,

• ${\hat{H}}_k^D(y)=(n^D{)}^{-1}\sum _{i=1}^{n^D}I({\hat{\epsilon }}_{k,i}^D⩽y)$,

• ${\hat{\epsilon }}_{k,i}^D={\displaystyle \frac{Y_{k,i}^D-{\hat{\mu }}_k^D(X_i^D)}{{\hat{\sigma }}_k^D(X_i^D)}}$, with $i\in \{1,\dots ,n^D\}$,

• ${\hat{\mu }}_k^D(x)=\sum _{i=1}^{n^D}{W}_{k,i}^D(x,g_k^D)Y_{k,i}^D$ is a non-parametric estimator of ${\mu }_k^D(x)$ based on local weights $W_{k,i}^D(x,g_k^D)$ depending on a bandwidth parameter $g_k^D$,

• $({\hat{\sigma }}_k^D{)}^2(x)=\sum _{i=1}^{n^D}{W}_{k,i}^D(x,g_k^D)[Y_{k,i}^D-{\hat{\mu }}_k^D(X_i^D){]}^2$ is a non-parametric estimator of $({\sigma }_k^D{)}^2(x)$. For simplicity we take the same bandwidth parameter $g_k^D$ that is used for the estimation of the regression function ${\hat{\mu }}_k^D(x)$,

• $W_{k,i}^D(x,g_k^D)={\displaystyle \frac{{\kappa }_{g_k^D}(x-X_i^D)}{\sum _{l=1}^{n^D}{\kappa }_{g_k^D}(x-X_l^D)}}$ are Nadaraya–Watson-type weights, where ${\kappa }_{g_k^D}(\cdot )=\kappa (\cdot /g_k^D)/g_k^D$ and κ is a probability density function symmetric around zero,

• ${\hat{a}}_k(x^F,x^G)=({\hat{\mu }}_k^F(x^F)-{\hat{\mu }}_k^G(x^G))/{\hat{\sigma }}_k^F(x^F)$ and ${\hat{b}}_k(x^F,x^G)={\hat{\sigma }}_k^G(x^G)/{\hat{\sigma }}_k^F(x^F)$,

• $h_k$ is a bandwidth parameter responsible for the smoothness of the estimator. Its value does not seem to have a significant effect on the conditional ROC curve estimation.

This way of estimating the conditional ROC curve is similar to the one proposed in [9], with the difference that they condition the ROC curve on a single value x and here we have a pair of values $x^F$ and $x^G$, each one of them related to the diseased and the healthy population, respectively. As both populations are independent, the adaptation of the methodology of [9] to this case is straightforward.

Once we know how to estimate this doubly conditional ROC curve we can propose a test statistic for the test (5):

$$S^x=\sum _{k=1}^K\psi (\sqrt{n{g}_k}\{{\widehat{RO{C}_k}}^{x^F,x^G}(p)-{\widehat{ROC}}_{\bullet }^{x^F,x^G}(p)\}),$$ (10)

where:

• for $k\in \{1,\dots ,K\}$, $g_k=\frac{n^F{g}_k^F+n^G{g}_k^G}{n}$, where $g_k^F$ and $g_k^G$ are bandwidth parameters involved in the estimation of the kth conditional ROC curve.

• for $k\in \{1,\dots ,K\}$, ${\widehat{ROC}}_k^{x^F,x^G}(p)$ is the estimated conditional ROC curve given $(x^F,x^G)$, as seen in (9),

• ${\widehat{ROC}}_{\bullet }^{x^F,x^G}(p)=(\sum _{k=1}^K{g}_k{)}^{-1}\sum _{k=1}^K{g}_k{\widehat{RO{C}_k}}^{x^F,x^G}(p)$ is a sort of weighted average of the K conditional ROC curves.

• ψ is a real-valued function that measures the difference between each estimated conditional ROC curve and the weighted average of all of them. This function may be similar to the ones used for the comparison of cumulative distribution functions (after all, a ROC curve can be viewed as a cumulative distribution function). For example, if one considers the $L_2$-measure, then the resulting test statistic is

  $$S_{L2}^x=\sum _{k=1}^{K}n{g}_k\int {({\widehat{RO{C}_k}}^{x^F,x^G}(p)-{\widehat{ROC}}_{\bullet }^{x^F,x^G}(p))}^2\mathrm{d}p.$$

  On the other hand, when using the Kolmogorov–Smirnov criteria the resulting test statistic is

  $$S_{KS}^x=\sum _{k=1}^K\sqrt{n{g}_k}\underset{p}{sup}|{\widehat{RO{C}_k}}^{x^F,x^G}(p)-{\widehat{ROC}}_{\bullet }^{x^F,x^G}(p)|.$$

The null hypothesis will be rejected for large values of $S^x$. In order to obtain the distribution of this statistic, a bootstrap algorithm is proposed. This bootstrap algorithm is adapted from the procedure proposed in [14] and has been already used by Martínez-Camblor et al. [15] and by Fanjul-Hevia et al. [7] in the context of ROC curves. The key of this algorithm is that

$$\begin{array}{rl}{T}^x& =\sum _{k=1}^K\psi (\sqrt{n{g}_k}\{({\widehat{ROC}}_k^{x^F,x^G}(p)-{\widehat{ROC}}_{\bullet }^{x^F,x^G}(p))\\ & -(RO{C}_k^{x^F,x^G}(p)-RO{C}_{\bullet }^{x^F,x^G}(p))\}),\end{array}$$

coincides with the statistic $S^x$ as long as the null hypothesis holds, where

$$RO{C}_{\bullet }^{x^F,x^G}(p)={\left(\sum _{k=1}^K{g}_k\right)}^{-1}\sum _{k=1}^K{g}_{k}RO{C}_k^{x^F,x^G}(p),0<p<1.$$

The quantity $T^x$ can be rewritten as

$$T^x=\sum _{k=1}^K\psi (\sum _{j=1}^K\sqrt{n{g}_j}{\alpha }_{kj}\{{\widehat{ROC}}_j^{x^F,x^G}(p)-RO{C}_j^{x^F,x^G}(p)\}),$$ (11)

where ${\alpha }_{kj}=I(k=j)-\sqrt{g_k}\sqrt{g_j}(\sum _{i=1}^K{g}_i{)}^{-1}$. Note that, in general, $T^x$ cannot be computed from the data, as it depends on the unknown theoretical conditional ROC curves, but it is useful when applying the bootstrap algorithm.

The bootstrap algorithm suggested to approximate a p-value for this test is the following:

• A.1From the original samples, $\{(X_i^F,Y_{1,i}^F,\dots ,Y_{K,i}^F){\}}_{i=1}^{n^F}$ and $\{(X_i^G,Y_{1,i}^G,\dots ,Y_{K,i}^G){\}}_{i=1}^{n^G}$, compute the test statistic value (10), that we will denote by $s^x$.
• A.2
  For $b=1,\dots ,B$, generate the bootstrap samples $\{(X_i^F,Y_{1,i}^{F,b\ast },\dots ,Y_{K,i}^{F,b\ast }){\}}_{i=1}^{n^F}$ and $\{(X_i^G,Y_{1,i}^{G,b\ast },\dots ,Y_{K,i}^{G,b\ast }){\}}_{i=1}^{n^G}$ as follows:

  1. For each $D\in \{F,G\}$, let $\{({\epsilon }_{1,i}^{D,b\ast },\dots ,{\epsilon }_{K,i}^{D,b\ast }){\}}_{i=1}^{n^D}$ be an i.i.d. sample from the empirical cumulative multivariate distribution function of the original residuals.
  2. Reconstruct the bootstrap samples $\{(X_i^D,Y_{1,i}^{D,b\ast },\dots ,Y_{K,i}^{D,b\ast }){\}}_{i=1}^{n^D}$ for each $D\in \{F,G\}$, where $Y_{k,i}^{D,b\ast }={\hat{\mu }}_k^D(X_{k,i}^D)+{\hat{\sigma }}_k^D(X_{k,i}^D){\epsilon }_{k,i}^{D,b\ast }$.
• A.3
  Compute the test statistic based on the bootstrap samples, for $b=1,\dots ,B$ using (11) as

  $$\begin{array}{r}{t}^{x,b\ast }=\sum _{k=1}^K\psi (\sum _{j=1}^K\sqrt{n{g}_j}{\alpha }_{kj}\{{\widehat{ROC}}_j^{x^F,x^G,b\ast }(p)-{\widehat{ROC}}_j^{x^F,x^G}(p)\}),\end{array}$$

  where ${\widehat{ROC}}_j^{x^F,x^G,b\ast }$ is the estimated j-th conditional ROC curve of the b-th bootstrap sample.
• A.4
  The distribution of $S^x$ under the null hypothesis (and thus, the distribution of $T^x$) is approximated by the empirical distribution of the values $\{t^{x,1\ast },\dots ,t^{x,B\ast }\}$ and the p-value is approximated by

  $$p-value=\frac{1}{B}\sum _{b=1}^{B}I(s^x⩽t^{x,b\ast }).$$

In contrast with the usual bootstrap algorithms in testing setups, in this case the null hypothesis is not employed when generating of the bootstrap samples (Step A.2), because replicating the null hypothesis of equal ROC curves is not a straightforward problem. Instead, it is used in the computation of the bootstrap statistic (Step A.3) by using $T^x$ instead of $S^x$, that are equal under the null hypothesis. This particularity also appears in the bootstrap algorithm of the next section.

There are two kind of bandwidth parameters that appear in the estimation of the $k-$th conditional ROC curve (9), with $k\in \{1,\dots K\}$. The first one, $h_k$, is taken as $1/\sqrt{n}$, and the second ones, $g_k^F$ and $g_k^G$, are selected by least-squares cross-validation. Note that, for each bootstrap iteration, the bandwidth parameters could change, as their selection depends on the sample. However, $h_k$ remains constant, as we are choosing it in terms of the sample size, and that is the same for each bootstrap iteration. As for $g_k^F$ and $g_k^G$, for computational issues we have decided to compute them on step A.1 using the original sample, and then apply the same bandwidths for all the bootstrap estimations. The cross-validation method can be very time-consuming, and this simplification prevents the simulations to become infeasible.

2.3. Test for a multidimensional covariate

In the previous subsections we have shown how to transform our original multidimensional problem into a one-dimensional one by using projections and how to test the equality of ROC curves conditioned to a given pair $(x^F,x^G)$ (and more particularly, to a fixed pair of directions). It is time now to resume our main objective, which was to compare ROC curves conditioned to a multidimensional covariate.

Once having seen a strategy for testing (4) for only one pair of fixed directions, the idea now is to modify the previous procedure so the new statistic takes into account all the possible directions that ${\mathit{\beta }}^F$ and ${\mathit{\beta }}^G$ can take. For that purpose, consider the test statistic

$$D_S^{\mathit{x}}={\int }_{{\mathbb{S}}^{d-1}}{\int }_{{\mathbb{S}}^{d-1}}{S}^{({\mathit{\beta }}^F{)}^{\mathrm{\prime }}\mathit{x},({\mathit{\beta }}^G{)}^{\mathrm{\prime }}\mathit{x}}\mathrm{d}{\mathit{\beta }}^F\mathrm{d}{\mathit{\beta }}^G,$$ (12)

where $\mathrm{d}{\mathit{\beta }}^F$ and $\mathrm{d}{\mathit{\beta }}^G$ represent the uniform density on the sphere of dimension d, ${\mathbb{S}}^{d-1}$. This ensures that all directions are equally important.

The expression $S^{({\mathit{\beta }}^F{)}^{\mathrm{\prime }}\mathit{x},({\mathit{\beta }}^G{)}^{\mathrm{\prime }}\mathit{x}}$ is equal to the statistic used in (10) for testing the equality of K ROC curves when conditioned to the value of the pair $(({\mathit{\beta }}^F{)}^{\mathrm{\prime }}\mathit{x},({\mathit{\beta }}^G{)}^{\mathrm{\prime }}\mathit{x})$, that is,

$$S^{({\mathit{\beta }}^F{)}^{\mathrm{\prime }}\mathit{x},({\mathit{\beta }}^G{)}^{\mathrm{\prime }}\mathit{x}}=\sum _{k=1}^K\psi (\sqrt{n{g}_k}\{{\widehat{RO{C}_k}}^{({\mathit{\beta }}^F{)}^{\mathrm{\prime }}\mathit{x},({\mathit{\beta }}^G{)}^{\mathrm{\prime }}\mathit{x}}(p)-{\widehat{ROC}}_{\bullet }^{({\mathit{\beta }}^F{)}^{\mathrm{\prime }}\mathit{x},({\mathit{\beta }}^G{)}^{\mathrm{\prime }}\mathit{x}}(p)\}).$$

Note that, in this context with $d-$dimensional covariates, the samples are $\{({\mathit{X}}_i^F,Y_{1,i}^F,\dots ,Y_{K,i}^F){\}}_{i=1}^{n^F}$ and $\{({\mathit{X}}_i^G,Y_{1,i}^G,\dots ,Y_{K,i}^G){\}}_{i=1}^{n^G}$, with ${\mathit{X}}_i^F=(X_{1,i}^F,\dots ,X_{d,i}^F{)}^{\mathrm{\prime }}$ and ${\mathit{X}}_i^G=(X_{1,i}^G,\dots ,X_{d,i}^G{)}^{\mathrm{\prime }}$.

In practice, as it is done in [1], to compute the test statistic $D_S^{\mathit{x}}$ random directions ${\mathit{\beta }}_1^F,\dots ,{\mathit{\beta }}_{n_{\beta }}^F$ and ${\mathit{\beta }}_1^G,\dots ,{\mathit{\beta }}_{n_{\beta }}^G$ are drawn uniformly from ${\mathbb{S}}^{d-1}$, where $n_{\mathit{\beta }}$ is the number of random directions considered (the same number of directions is taken for ${\mathit{\beta }}^F$ and for ${\mathit{\beta }}^G$). With them, the approximated statistic is

$${\tilde{D}}_S^{\mathit{x}}=\frac{1}{n_{\mathit{\beta }}^2}\sum _{r=1}^{n_{\mathit{\beta }}}\sum _{l=1}^{n_{\mathit{\beta }}}{S}^{({\mathit{\beta }}_r^F{)}^{\mathrm{\prime }}\mathit{x},({\mathit{\beta }}_l^G{)}^{\mathrm{\prime }}\mathit{x}}.$$ (13)

In order to obtain the distribution of the statistic, a bootstrap algorithm (similar to the one described in the previous section) is proposed. To do so, the following expression is introduced:

$$D_T^{\mathit{x}}={\int }_{{\mathbb{S}}^{d-1}}{\int }_{{\mathbb{S}}^{d-1}}{T}^{({\mathit{\beta }}^F{)}^{\mathrm{\prime }}\mathit{x},({\mathit{\beta }}^G{)}^{\mathrm{\prime }}\mathit{x}}\mathrm{d}{\mathit{\beta }}^F\mathrm{d}{\mathit{\beta }}^G,$$ (14)

where $T^{({\mathit{\beta }}^F{)}^{\mathrm{\prime }}\mathit{x},({\mathit{\beta }}^G{)}^{\mathrm{\prime }}\mathit{x}}$ is the same as in (11), but for the conditioning values of $(({\mathit{\beta }}^F{)}^{\mathrm{\prime }}\mathit{x},({\mathit{\beta }}^G{)}^{\mathrm{\prime }}\mathit{x})$:

$$T^{({\mathit{\beta }}^F{)}^{\mathrm{\prime }}\mathit{x},({\mathit{\beta }}^G{)}^{\mathrm{\prime }}\mathit{x}}=\sum _{k=1}^K\psi (\sum _{j=1}^K\sqrt{n{g}_j}{\alpha }_{kj}\{{\widehat{ROC}}_j^{({\mathit{\beta }}^F{)}^{\mathrm{\prime }}\mathit{x},({\mathit{\beta }}^G{)}^{\mathrm{\prime }}\mathit{x}}(p)-RO{C}_j^{({\mathit{\beta }}^F{)}^{\mathrm{\prime }}\mathit{x},({\mathit{\beta }}^G{)}^{\mathrm{\prime }}\mathit{x}}(p)\}).$$

As it happened in (11), $T^{({\mathit{\beta }}^F{)}^{\mathrm{\prime }}\mathit{x},({\mathit{\beta }}^G{)}^{\mathrm{\prime }}\mathit{x}}$ cannot be computed without knowing the true distribution of the diagnostic markers. However, it can be computed in the bootstrap algorithm below, and there $D_T^{\mathit{x}}$ is approximated by

$${\tilde{D}}_T^{\mathit{x}}=\frac{1}{n_{\mathit{\beta }}^2}\sum _{r=1}^{n_{\mathit{\beta }}}\sum _{l=1}^{n_{\mathit{\beta }}}{T}^{({\mathit{\beta }}_r^F{)}^{\mathrm{\prime }}\mathit{x},({\mathit{\beta }}_l^G{)}^{\mathrm{\prime }}\mathit{x}}.$$ (15)

As happened before, for two given projections ${\mathit{\beta }}^F$ and ${\mathit{\beta }}^G$, $S^{({\mathit{\beta }}^F{)}^{\mathrm{\prime }}\mathit{x},({\mathit{\beta }}^G{)}^{\mathrm{\prime }}\mathit{x}}$ and $T^{({\mathit{\beta }}^F{)}^{\mathrm{\prime }}\mathit{x},({\mathit{\beta }}^G{)}^{\mathrm{\prime }}\mathit{x}}$ coincide as long as the null hypothesis holds, and thus the same happens with $D_S^{\mathit{x}}$ and $D_T^{\mathit{x}}$.

Taking into account these approximations, the resulting bootstrap algorithm goes as follows:

• B.1
  Draw $n_{\mathit{\beta }}$ random directions ${\mathit{\beta }}_1^F,\dots ,{\mathit{\beta }}_{n_{\beta }}^F$ and ${\mathit{\beta }}_1^G,\dots ,{\mathit{\beta }}_{n_{\beta }}^G$ uniformly from ${\mathbb{S}}^{d-1}$. This can be done by using the method proposed by Muller [16] to generate points in a multidimensional sphere. For each random direction ${\mathit{\beta }}_{\mathit{s}}^{\mathit{D}}$, with $D\in \{F,G\}$ and $s\in \{1,\dots ,n_{\beta }\}$:

  1. Generate d values independently from a normal standard distribution: $u_1,\dots ,u_d$.
  2. Consider the projection obtained by normalizing the vector $(u_1,\dots ,u_d)$:

     $${\mathit{\beta }}_{\mathit{s}}^{\mathit{D}}=(\frac{u_1}{\sqrt{u_1^2+\cdots +u_d^2}},\dots ,\frac{u_d}{\sqrt{u_1^2+\cdots +u_d^2}}).$$
• B.2For each random directions ${\mathit{\beta }}_r^F$ and ${\mathit{\beta }}_l^G$ (with $r,l\in \{1,\dots ,n_{\mathit{\beta }}\}$), consider the sample $\{(({\mathit{\beta }}_r^F{)}^{\mathrm{\prime }}{\mathit{X}}_i^F,Y_{1,i}^F,\dots ,Y_{K,i}^F){\}}_{i=1}^{n^F}$ and $\{(({\mathit{\beta }}_l^G{)}^{\mathrm{\prime }}{\mathit{X}}_i^G,Y_{1,i}^G,\dots ,Y_{K,i}^G){\}}_{i=1}^{n^G}$ and the conditioning values $(({\mathit{\beta }}_r^F{)}^{\mathrm{\prime }}\mathit{x},({\mathit{\beta }}_l^G{)}^{\mathrm{\prime }}\mathit{x})$. With them, following steps A.1–A.3 of the bootstrap algorithm of the previous subsection, compute the value of $s^{({\mathit{\beta }}_r^F{)}^{\mathrm{\prime }}\mathit{x},({\mathit{\beta }}_l^G{)}^{\mathrm{\prime }}\mathit{x}}$ and the B corresponding $t^{({\mathit{\beta }}_r^F{)}^{\mathrm{\prime }}\mathit{x},({\mathit{\beta }}_l^G{)}^{\mathrm{\prime }}\mathit{x},b\ast }$.
• B.3Compute ${\tilde{d}}_S^{\mathit{x}}=\frac{1}{n_{\mathit{\beta }}^2}\sum _{r=1}^{n_{\beta }}\sum _{l=1}^{n_{\beta }}{s}^{({\mathit{\beta }}_r^F{)}^{\mathrm{\prime }}\mathit{x},({\mathit{\beta }}_l^G{)}^{\mathrm{\prime }}\mathit{x}}$ and ${\tilde{d}}_T^{\mathit{x},b\ast }=\frac{1}{n_{\mathit{\beta }}^2}\sum _{r=1}^{n_{\mathit{\beta }}}\sum _{l=1}^{n_{\mathit{\beta }}}{t}^{({\mathit{\beta }}_r^F{)}^{\mathrm{\prime }}\mathit{x},({\mathit{\beta }}_l^G{)}^{\mathrm{\prime }}\mathit{x},b\ast }$ as in (13) and (15).
• B.4
  Approximate the p-value of the test by:

  $$p-value=\frac{1}{B}\sum _{b=1}^{B}I({\tilde{d}}_S^{\mathit{x}}⩽{\tilde{d}}_T^{\mathit{x},b\ast }).$$

Remark 2.1

Note that $n_{\mathit{\beta }}$ represents the number of random directions drawn from ${\mathbb{S}}^{d-1}$ considered for the approximation of (13) and (15), but that, in fact, we are using $n_{\mathit{\beta }}^2$ different combination of pairs $({\mathit{\beta }}^F,{\mathit{\beta }}^G)\in {\mathbb{S}}^{d-1}\times {\mathbb{S}}^{d-1}$ to make that approximation. This could become a problem from the computational point of view, as the complexity of the problem increases very fast when increasing the value of $n_{\mathit{\beta }}$.

As an alternative, we could consider using

$$D_S^{\mathit{x}}={\int }_{{\mathbb{S}}^{d-1}\times {\mathbb{S}}^{d-1}}{S}^{({\mathit{\beta }}^F{)}^{\mathrm{\prime }}\mathit{x},({\mathit{\beta }}^G{)}^{\mathrm{\prime }}\mathit{x}}\mathrm{d}{\mathit{\beta }}^F{\mathit{\beta }}^G,$$

instead of statistic (12), where $\mathrm{d}{\mathit{\beta }}^F{\mathit{\beta }}^G$ represents the uniform density on the torus of dimension d, ${\mathbb{S}}^{d-1}\times {\mathbb{S}}^{d-1}$. This ensures, as before, that all pairs of directions are equally important. Thus, in practice, instead of using the approximation (13) we could consider

$${\hat{D}}_S^{\mathit{x}}=\frac{1}{m_{\mathit{\beta }}}\sum _{r=1}^{m_{\mathit{\beta }}}{S}^{({\mathit{\beta }}_r^F{)}^{\mathrm{\prime }}\mathit{x},({\mathit{\beta }}_r^G{)}^{\mathrm{\prime }}\mathit{x}},$$

where $({\mathit{\beta }}_1^F,{\mathit{\beta }}_1^G),\dots ,({\mathit{\beta }}_{m_{\beta }}^F,{\mathit{\beta }}_{m_{\beta }}^G)$ are pairs of random directions drawn uniformly from ${\mathbb{S}}^{d-1}\times {\mathbb{S}}^{d-1}$, and where $m_{\mathit{\beta }}$ would represent here the same as $n_{\mathit{\beta }}^2$ before, with the advantage that it allows for more flexibility because it can assume non-squared values. A similar adaptation could be applied for the approximation of $D_T^{\mathit{x}}$ in (14).

Remark 2.2

In the literature we can find papers, like for example [2] or [3], that use only one random projection. The main idea is to perform the test at hand for a randomly selected projection instead of for all possible projections. The use of projections results in a dimension reduction (as desired), and, despite being a procedure that may produce less powerful tests, the use of one single projection results in a reduction of the computational cost.

Following that idea, instead of testing the equality of covariate-projected ROC curves for all possible projections, we could test the equality of covariate-projected ROC curves for some random pair of projections given a certain $\mathit{x}\in R_{\mathit{X}}$, meaning:

$$H_0:RO{C}_1^{({\mathit{\beta }}^F{)}^{\mathrm{\prime }}\mathit{x},({\mathit{\beta }}^G{)}^{\mathrm{\prime }}\mathit{x}}=\cdots =RO{C}_K^{({\mathit{\beta }}^F{)}^{\mathrm{\prime }}\mathit{x},({\mathit{\beta }}^G{)}^{\mathrm{\prime }}\mathit{x}}\text{for some}{\mathit{\beta }}^F,{\mathit{\beta }}^G.$$ (16)

The equivalence between this hypothesis and the one of interest in this paper given in (2) still needs theoretical justification. However, it is a possibility worth studying, if only for computational reasons. A way of perform this approach could be to consider the proposed methodology for $n_{\mathit{\beta }}=1$.

Additionally, in a practical situation we should take into account that the different magnitudes of the covariates that we are projecting may obscure the effect of some of them. In order to prevent this from happening we suggest standardizing the multidimensional covariate $\mathit{X}$ before we start the analysis. Note that an ROC curve conditioned to a certain value $\mathit{x}$ is the same as the ROC curve in which the covariate is modified by a one-to-one transformation and the ROC curve is conditioned to the corresponding transformed $\mathit{x}$ value.

Given a non-degenerate multidimensional covariate $\mathit{X}$ the standardization proposed here is to consider the multidimensional covariate ${\mathit{X}}_s={\mathit{B}}^{-1}(\mathit{X}-\mathit{a})$, with $\mathit{B}$ a diagonal matrix with $(\sqrt{Var(X_1)},\dots ,\sqrt{Var(X_d)})$ in the diagonal and $\mathit{a}=(E(X_1),\dots ,E(X_d){)}^{\mathrm{\prime }}$. Then, for a given variable Y, a given $y\in \mathbb{R}$ and a certain value of the covariate $\mathit{x}$,

$$P(Y⩽y|\mathit{X}=\mathit{x})=P(Y⩽y|{\mathit{B}}^{-1}(\mathit{X}-\mathit{a})={\mathit{B}}^{-1}(\mathit{x}-\mathit{a}))=P(Y⩽y|{\mathit{X}}_s={\mathit{x}}_s)),$$

with ${\mathit{x}}_s={\mathit{B}}^{-1}(\mathit{x}-\mathit{a})$ and, thus,

$$RO{C}^{\mathit{x}}(p)=1-F(G^{-1}(1-p|\mathit{x})|\mathit{x})=1-F(G^{-1}(1-p|{\mathit{x}}_s)|{\mathit{x}}_s)=RO{C}^{{\mathit{x}}_s}(p).$$

The standardization that takes place here does not care for the covariance between the covariates that conform $\mathit{X}$, as we are only interested on obtaining covariates with similar magnitudes. Also, in practice the standardization is made considering the sample mean and the sample standard deviation of the covariates at hand.

3. Simulations

In order to analyse the performance of the proposed methodology, simulations were run for the comparison of several dependent conditional ROC curves. On a first stage, these simulations were focused on analysing the behaviour of the unidimensional test described in Section 2.2, but we do not display them here, as they are very similar to the ones that can be found in [7]. Instead, we show the results for several scenarios (first under the null hypothesis and then under the alternative) in which we compare K ROC curves (with $K\in \{2,3\}$) conditioned to a $d-$dimensional covariate (with $d\in \{2,3\}$).

All the curves used in the simulation study were drawn from location-scale regression models similar to the ones presented in (6) and (7), only that, in this case, the regression and the conditional standard deviation functions are for d-dimensional covariates. The construction of those curves is summarized in Table 1, were all the different conditional mean and conditional standard deviation functions are displayed.

Table 1.

Conditional mean and conditional standard deviation functions of the conditional ROC curves considered in the simulation study.

Covariate	ROC curves	Regression functions	Conditional standard deviation functions
	$RO{C}_1^{\mathit{x}}$	${\mu }_1^F(\mathit{x})=\sin (0.5\pi x_1)+0.1x_2$ ${\mu }_1^G(\mathit{x})=0.5x_1{x}_2$	${\sigma }_1^F(\mathit{x})=0.5+0.5x_1$ ${\sigma }_1^G(\mathit{x})=0.5+0.5x_1$
$\mathit{x}=\left(\begin{array}{c}{x}_1\\ x_2\end{array}\right)$	$RO{C}_2^{\mathit{x}}$	${\mu }_2^F(\mathit{x})=0.3+\sin (0.5\pi x_1)+0.1x_2$ ${\mu }_2^G(\mathit{x})=0.5x_1{x}_2$	${\sigma }_2^F(\mathit{x})=0.5+0.5x_1$ ${\sigma }_2^G(\mathit{x})=0.5+0.5x_1$
	$RO{C}_3^{\mathit{x}}$	${\mu }_3^F(\mathit{x})=\sin (0.5\pi x_1)+0.1x_2$ ${\mu }_3^G(\mathit{x})=-0.3+0.4x_2+0.5x_1{x}_2$	${\sigma }_3^F(\mathit{x})=0.5+0.5x_1$ ${\sigma }_3^G(\mathit{x})=0.5+0.5x_1$
	$RO{C}_4^{\mathit{x}}$	${\mu }_4^F(\mathit{x})=\sin (0.5\pi x_1)+0.1x_2+0.5x_3,$ ${\mu }_4^G(\mathit{x})=0.5x_1{x}_2+x_3$	${\sigma }_4^F(\mathit{x})=0.5+0.1x_3,$ ${\sigma }_4^G(\mathit{x})=0.5+0.1x_3$
$\mathit{x}=\left(\begin{array}{c}{x}_1\\ x_2\\ x_3\end{array}\right)$	$RO{C}_5^{\mathit{x}}$	${\mu }_5^F(\mathit{x})=\sin (0.5\pi x_1)+0.1x_2+0.5x_3,$ ${\mu }_5^G(\mathit{x})=x_1{x}_2+x_3$	${\sigma }_5^F(\mathit{x})=0.5+0.1x_3$ ${\sigma }_5^G(\mathit{x})=0.5+0.1x_3$
	$RO{C}_6^{\mathit{x}}$	${\mu }_6^F(\mathit{x})=\sin (0.5\pi x_1)+0.1x_2+0.5x_3,$ ${\mu }_6^G(\mathit{x})=-0.3+0.5x_1{x}_2+x_3$	${\sigma }_6^F(\mathit{x})=0.5+0.2x_2+0.3x_3$ ${\sigma }_6^G(\mathit{x})=0.5+0.1x_3$

The regression errors were considered to have multivariate normal distribution with zero mean, variance one and correlation ρ for all the models.

In all scenarios the covariates $X_1^F$, $X_1^G$, $X_2^F$, $X_2^G$, $X_3^F$ and $X_3^G$ are uniformly distributed in the unit interval. Thus, the value of the multidimensional covariate $\mathit{x}$ at which the conditional ROC curves should be compared is contained in $[0,1{]}^d$. Particularly, the comparisons are made for $\mathit{x}=(0.5,0.6{)}^{\mathrm{\prime }}$ and for $\mathit{x}=(0.5,0.6,0.5{)}^{\mathrm{\prime }}$, for d = 2 and d = 3, respectively.

The study contains simulations for different sample sizes $(n^F,n^G)\in \{(100,100)$, $(250,150)$, $(250,350)\}$ and different values of ρ that represent different possible degrees of correlation between the diagnostic variables under comparison ( $\rho \in \{-0.5,0,0.5\}$).

Moreover, two different functions ψ were considered for the construction of $S^{({\mathit{\beta }}^F{)}^{\mathrm{\prime }}\mathit{x},({\mathit{\beta }}^G{)}^{\mathrm{\prime }}\mathit{x}}$: one based on the $L_2-$measure and the other one based on the Kolmogorov-Smirnov criterion (from now on denoted by $L_2$ and KS, respectively). The number of iterations used in the bootstrap algorithm was 200, and 500 data sets were simulated to compute the proportion of rejection in each scenario.

Furthermore, the number of directions that was used for approximating the test statistic $D_S^{\mathit{x}}$ was taken as $n_{\mathit{\beta }}=5$ (as mentioned in Remark 2.1, notice that this means that $n_{\mathit{\beta }}^2=25$ different pairs of directions were considered).

3.1. Level of the test

The scenarios that were considered for calibrating the level of the test (by comparing the same conditional ROC curves) are represented in Table 2. The results of the simulations obtained for $n_{\mathit{\beta }}=5$ are summarized in Figures 1 (for d = 2) and 2 (for $d=3$). Each subfigure represents the test of one scenario for a particular sample size. The nominal level considered is 0.05. The estimated proportion of rejections over 500 replications of the data sets is represented along with the rejection region of such nominal level. For the test to be well calibrated the estimated proportions should fall between the gray lines.

Table 2.

Scenarios under the null hypothesis considered for calibrating the level of the test.

Figure 1.

Estimated proportion of rejection under the null hypothesis and the corresponding limits of the critical region (in gray) for the level 0.05 (dotted black line) with d = 2 and $n_{\mathit{\beta }}=5$ for different sample sizes and different ρ.

Figure 2.

Estimated proportion of rejection under the null hypothesis and the corresponding limits of the critical region (in gray) for the level 0.05 (dotted black line) with d = 3 and $n_{\mathit{\beta }}=5$ for different sample sizes and different ρ.

In general it can be said that the expected nominal level is reached, as most of the estimated proportions are close to the corresponding nominal level. The $L_2$ statistic seems to overestimate the level in a few scenarios, but its behaviour improves when increasing the sample size. The KS statistic is a little more conservative.

3.2. Power of the test

On the other hand, the scenarios that were considered for studying the power of the test (by comparing different conditional ROC curves) are represented in Table 3. The results of the simulations are summarized in Figure 3 (for $n_{\mathit{\beta }}=5$). In those figures the first and second row represent the simulation results for the scenarios with K = 2 and K = 3, respectively, and the first and the second column represent the simulation results for d = 2 and for d = 3, respectively. In this case, only $\alpha =0.05$ was considered.

Table 3.

Scenarios under the alternative hypothesis considered for calibrating the power of the test.

Note: $RO{C}_1^x$ and $RO{C}_4^x$ are represented in purple, $RO{C}_2^x$ and $RO{C}_5^x$ in green, and $RO{C}_3^x$ and $RO{C}_6^x$ in yellow.

Figure 3.

Estimated proportion of rejection under the alternative hypothesis for different sample sizes and different ρ, for $n_{\mathit{\beta }}=5$ ( $\alpha =0.05$).

It can be seen that the power of the test grows with the considered sample sizes. The $L_2$ statistic yields higher power than the KS statistic, which is consistent with KS being more conservative. Moreover, the difference between the conditional ROC curves considered for the case of d = 2 is bigger than the difference between the ROC curves in the scenarios with d = 3, which translates in higher power for the cases in which d = 2.

We can also observe that for each scenario, the highest power is always obtained for the cases in which the correlation of the diagnostic variables is $\rho =0.5$, and the lowest for $\rho =-0.5$.

Note that for the scenario with d = 3 and $\rho =-0.5$ the power of the test does not increase significantly from the first sample size to the second (in fact, for K = 3 it even decreases a little), but this can be due to the fact that the lower sample size has balanced data, ( $n^F$, $n^G$) being (100,100). whereas for the second sample size considered ( $n^F$, $n^G$) take the value (250,150). The highest sample size is also unbalanced, but not so much.

Remark 3.1

In order to evaluate the modification of the method proposed in Remarks 2.1 and 2.2 we have run simulations for the same scenarios previously described. We show here the results for the scenarios with K = 2 and d = 2 under the null and the alternative hypotheses for assessing the level and the power of the test, respectively. Similar conclusions were obtained with the rest of the scenarios. The parameters that are used here are the same as before, with the exception that now 1000 data-sets were simulated instead of 500.

Figure 4 shows the results of the simulations when considering the modification of Remark 2.1 for $m_{\mathit{\beta }}=50$ (first row) and $m_{\mathit{\beta }}=25$ (second row), and the results for considering only one random projection (Remark 2.2), i.e. $m_{\mathit{\beta }}=n_{\mathit{\beta }}=1$ (third row). Note that taking $m_{\mathit{\beta }}=25$ is comparable with $n_{\mathit{\beta }}=5$ used in the previous simulations (see first row of Figure 1), and that the results are very similar: the estimated proportion or rejections is a little overestimated for the $L_2$ statistic for the smaller sample size and otherwise close to the nominal level, and the KS statistic is always more conservative. Increasing $m_{\mathit{\beta }}$ from 25 to 50 does not seem to affect the results significantly, and neither does reducing it to a single random projection ( $m_{\mathit{\beta }}=1$).

In Figure 5 we can observe the results for the simulations under the alternative hypothesis, once again for $m_{\mathit{\beta }}=50$, $m_{\mathit{\beta }}=25$ and $m_{\mathit{\beta }}=1$. The firs two graphics are very similar to the one obtained for $n_{\mathit{\beta }}=5$ (see the first graphic of Figure 3), but from the last graphic it is obvious that by using only one random projection the power of the test decreases considerably (as it was expected).

In the light of these results it seems that the alternative methodology proposed in Remark 2.1 yields similar conclusions than the first proposal, with no noticeable gain when increasing the number $m_{\mathit{\beta }}$ used to approximate the value of the statistic from 25 to 50. It remains an open problem to determine an optimal value for that parameter.

As for the idea mentioned in Remark 2.2, using only one random projection seems to produce a well calibrated test, despite having considerably lower power.

Figure 4.

Estimated proportion of rejection under the null hypothesis and the corresponding limits of the critical region (in gray) for the level 0.05 (dotted black line) with K = 2, d = 2 and $m_{\mathit{\beta }}=50,25,1$ for different sample sizes and different ρ.

Figure 5.

Estimated proportion of rejection under the alternative hypothesis for different sample sizes and different ρ, for $n_{\mathit{\beta }}=50,25,1$ and for the scenarios with K = 2 and d = 2.

3.3. Some extra simulations: changing the distribution of the covariates

The previous simulations were all obtained for scenarios in which the covariates are uniformly distributed in the unit interval. The point $\mathit{x}$ at which the comparisons where made was selected ensuring that it has enough data around. However, in practice the covariates may not follow a uniform distribution and the may even behave differently in the diseased and the helathy populations. This may result in scenarios where the point $\mathit{x}$ is closer to the boundaries of the supports of the covariates or in scenarios where it would be advisable to standardize the covariates with different magnitudes.

In this section we repeat the simulations for two of the scenarios considered previously, changing just the distribution of the covariates. The first one is the scenario under the null hypothesis where we compare two ROC curves equal to $RO{C}_1^{\mathit{x}}$ (defined in Table 1) for d = 2, with $\rho =0$ and $(n^F,n^G)=(250,150)$, at $\mathit{x}=(0.5,0.6)$. The second one is the scenario under the alternative hypothesis where we compare two curves, $RO{C}_1^{\mathit{x}}$ and $RO{C}_2^{\mathit{x}}$, also for d = 2, with $\rho =0$ and $(n^F,n^G)=(250,150)$ and at $\mathit{x}=(0.5,0.6)$. Both scenarios are represented in Tables 2 and 3. Note that it does not matter if we change the distribution of the covariates: the conditional ROC curve remains the same. The simulations were run in similar conditions as before, with $m_{\beta }=25$, 200 bootstrap iterations and 500 replications of the data sets.

We considered a total of 16 new models for the covariates, with different combinations of the distributions of $X_1^F$, $X_1^G$, $X_2^F$ and $X_2^G$. Models A.1–A.4 follow uniform distributions; the difference between them is that the point $\mathit{x}$ is almost at the centre of the support of the first model A.1, but as we move from A.1 to A.4 it gets closer and closer to the boundary of the support. The same applies for models B.1–B.4, but now with normal distributions. In models C.1–C.4 we add the feature that the the four variables have different distributions, and that the healthy and the diseased populations grow apart for each model. The distributions of the covariates in each case are described in Table 4. We also consider the models D.1–D.4, that used the same distributions than in C.1–C.4 but for which we applied the standardization recommended at the end of Section 2.

Table 4.

Distributions of the covariates considered for the simulations.

	$X_1^F$	$X_1^G$	$X_2^F$	$X_2^F$
A.1	$\mathcal{U}(0,1)$	$\mathcal{U}(0,1)$	$\mathcal{U}(0,1)$	$\mathcal{U}(0,1)$
A.2	$\mathcal{U}(0.1,1.1)$	$\mathcal{U}(0.1,1.1)$	$\mathcal{U}(0.1,1.1)$	$\mathcal{U}(0.1,1.1)$
A.3	$\mathcal{U}(0.2,1.2)$	$\mathcal{U}(0.2,1.2)$	$\mathcal{U}(0.2,1.2)$	$\mathcal{U}(0.2,1.2)$
A.4	$\mathcal{U}(0.3,1.3)$	$\mathcal{U}(0.3,1.3)$	$\mathcal{U}(0.3,1.3)$	$\mathcal{U}(0.3,1.3)$
B.1	$\mathcal{N}(0.5,1/\sqrt{12})$	$\mathcal{N}(0.5,1/\sqrt{12})$	$\mathcal{N}(0.5,1/\sqrt{12})$	$\mathcal{N}(0.5,1/\sqrt{12})$
B.2	$\mathcal{N}(0.6,1/\sqrt{12})$	$\mathcal{N}(0.6,1/\sqrt{12})$	$\mathcal{N}(0.6,1/\sqrt{12})$	$\mathcal{N}(0.6,1/\sqrt{12})$
B.3	$\mathcal{N}(0.7,1/\sqrt{12})$	$\mathcal{N}(0.7,1/\sqrt{12})$	$\mathcal{N}(0.7,1/\sqrt{12})$	$\mathcal{N}(0.7,1/\sqrt{12})$
B.4	$\mathcal{N}(0.8,1/\sqrt{12})$	$\mathcal{N}(0.8,1/\sqrt{12})$	$\mathcal{N}(0.8,1/\sqrt{12})$	$\mathcal{N}(0.8,1/\sqrt{12})$
C.1	$\mathcal{N}(0.55,2/\sqrt{12})$	$\mathcal{N}(0.45,2/\sqrt{12})$	$\mathcal{N}(0.55,1/\sqrt{12})$	$\mathcal{N}(0.45,1/\sqrt{12})$
C.2	$\mathcal{N}(0.6,2/\sqrt{12})$	$\mathcal{N}(0.4,2/\sqrt{12})$	$\mathcal{N}(0.6,1/\sqrt{12})$	$\mathcal{N}(0.4,1/\sqrt{12})$
C.3	$\mathcal{N}(0.7,2/\sqrt{12})$	$\mathcal{N}(0.3,2/\sqrt{12})$	$\mathcal{N}(0.7,1/\sqrt{12})$	$\mathcal{N}(0.3,1/\sqrt{12})$
C.4	$\mathcal{N}(0.8,2/\sqrt{12})$	$\mathcal{N}(0.2,2/\sqrt{12})$	$\mathcal{N}(0.8,1/\sqrt{12})$	$\mathcal{N}(0.2,1/\sqrt{12})$

Note: Models D.1–D.4 are not included as they are a standardization of models C.1–C.4.

The results for the scenario under the null hypothesis are summarized in Figure 6, considering the 16 different combinations of covariate distribution for both the $L_2$ and the KS type of statistic. The estimated proportion of rejections and its rejection region is presented as before for nominal level 0.05.

Figure 6.

Estimated proportion of rejection under the null hypothesis and the corresponding limits of the critical region (in gray) for the level 0.05 (dotted black line) for different combinations of distribution for the bidimensional covariates (models A.1–A.4–D.1–D4) for both the $L_2$ and the KS statistics.

The simulations show that the test is well calibrated regardless of the distribution of the covariates. It shows some problems when we take the point $\mathit{x}$ too close to the boundaries or in a place where there is limited data (models .4), but it is also expected considering that the sample size that we are using is not too large, $(n^F,n^G)=(250,150)$. Models .1 and .2 are comparable with the ones studied in previous sections, and only A.3 of the .3 models overestimates the level of the test for the $L_2$ statistic. Also, the good behaviour of the test for models D.1–D.4 shows that the standardization proposed in this methodology does not affect the result of the test.

The results of the simulations for the scenario under the alternative hypothesis are collected in Table 5. Leaving aside the differences between the results for the $L_2$ and the KS statistic, that were observed also in the previous sections, the power of the test does not change significantly when the covariate distribution changes, and it is comparable to the power obtained when we compared the same conditional ROC curves in the prevoius section. As it could be expected, it decreases when we move from the models with more data (.1) to the models with less data (.4) around the point $\mathit{x}$. Moreover, the power experiments some increasing from models in C to models in D for the $L_2$ statistic, which means that the standardization does not worsen the behaviour of the test. Note that in this case the models in C do not need the standardization for the test to work, as the difference of magnitude of the covariates involved is minimal.

Table 5.

Estimated proportion of rejection under the alternative hypothesis for different combinations of distribution for the bidimensional covariates (models A.1–A.4–D.1–D4) for both the $L_2$ and the KS statistics.

	$L_2$				KS
Models	$.1$	$.2$	$.3$	$.4$	$.1$	$.2$	$.3$	$.4$
A	0.670	0.636	0.590	0.530	0.520	0.454	0.426	0.386
B	0.650	0.614	0.580	0.538	0.470	0.472	0.420	0.386
C	0.642	0.626	0.602	0.588	0.464	0.458	0.440	0.424
D	0.652	0.638	0.614	0.600	0.452	0.458	0.438	0.424

4. Application

An illustration of the proposed test is displayed in this section through the analysis of the previously mentioned data set concerning 463 patients with pleural effusion. This data set has been provided by Dr. F. Gude, from the Unidade de Epidemioloxía Clínica of the Hospital Clínico Universitario de Santiago (CHUS), and it has been used for a previous study in [24].

From a medical perspective, the goal is to find a way to discriminate the patients in which the pleural effusion (PE) has a malignant origin (MPE) from those in which the PE is due to other non-cancer-related causes. 200 individuals form the sample had MPE (the diseased population in this context), against 263 who did not (healthy population). For that matter, two diagnostic markers were considered, the carbohydrate antigen 152 (ca125) and the cytokeratin fragment 21-1 (cyfra). Moreover, the information of two different covariates is also available: the age and the neuron-specific enolase (nse). Due to the characteristics of the data (positive values, most of them close to zero, with some extreme high values), logarithms of those variables – excluding the variable age – were considered for the study. Being the logarithm a monotone transformation, its use does not have an effect on the estimation of the common ROC curve. However, it does affect the estimation of the conditional ROC curves, as it reduces the effect of the more extreme values of the variables. A representation of the relationship of each one of those biomarkers with the two covariates is depicted in Figure 7, for both MPE (green) and the non-MPE (blue) patients.

Figure 7.

Scatterplots of the two different diagnostic biomarkers in function of the two covariates considered: age and $\log (nse)$. Contour plots were added in the bidimensional marginal scatterplots. The healthy subjects are represented in blue and the diseased ones in green (in the printed version, the healthy subjects appear in a darker colour than the diseased subjects).

It can be observed that the shape of the point clouds of the two populations changes with the values of the covariates, specially in the case of the diseased population.

In order to evaluate whether the discriminatory capability of those markers ( $Y_1^F$ and $Y_1^G$ as the variables containing the information of $\log (ca125)$, and $Y_2^F$ and $Y_2^G$ as the variables containing the information of $\log (cyfra)$) is the same when the covariates age and $\log (nse)$ are taken into account, the methodology explained in previous sections is applied, comparing their respective ROC curves conditioned to different values of the bidimensional covariate $\mathit{X}=(X_1,X_2)$ with $X_1=$ age and $X_2=\log (nse)$. In order to explore the advantages of using this method over the ones that do not consider multidimensional covariates, we also test the equivalence of the ROC curves of those diagnostic markers for the case in which no covariates are taken into account and for the case in which only one of the covariates is included in the analysis. Figure 8 shows how those two covariates are distributed in the diseased and healthy populations. The scatterplot included there highlights also the pair of values of the bidimensional covariate at which we are going compare the ROC curves in this analysis. Note that the covariates have different magnitudes: the values that the variable age takes are always going to be bigger than the values of $\log (nse)$. Thus, if we were to use the procedure directly over these variables, when projecting the multidimensional covariate $\mathit{X}$ on any direction, the effect of the second component will be overshadowed by the first component's. To prevent this from happening we decided to use the standardized variables of $X_1$ and $X_2$ instead of the originals. This also affects the value $\mathit{x}$ at which the conditional ROC curves are being compared. So in this case, instead of the multidimensional covariate $\mathit{X}$ and the conditional value $\mathit{x}$, we consider the standardized version ${\mathit{X}}_{\mathit{s}}=(\frac{X_1-\overline{X_1}}{S_{X_1}},\frac{X_2-\overline{X_1}}{S_{X_2}}{)}^{\mathrm{\prime }}$ and ${\mathit{x}}_{\mathit{x}}=(\frac{x_1-\overline{X_1}}{S_{X_1}},\frac{x_2-\overline{X_1}}{S_{X_2}}{)}^{\mathrm{\prime }}$, with $S_{X_1}$ and $S_{X_2}$ the sample standard deviation of the marginal covariates.

Figure 8.

Histograms, boxplots and scatterplots (with the corresponding contour plots) of the two covariates considered (age and $\log (nse)$). The healthy subjects are represented in blue and the diseased ones in green (in the printed version, the healthy subjects appear in a darker colour than the diseased subjects). The black histogram lines and the white boxplot correspond to the two populations of the healthy and the diseased patients combined. The red points in the scatterplot correspond to the values of $\mathit{x}$ at which the ROC curves are compared.

We start the analysis of the performance of the two diagnostic markers by comparing their respective ROC curves without taking into account any covariate information. For that matter we use the method proposed by DeLong et al. [4]. The estimated ROC curves for both markers are depicted in Figure 9. The p-value obtain for that comparison was 0.138. Similar results were obtained when using other ways of comparing ROC without covariates (like [15] or [25]). Thus, we do not find significant differences between the two diagnostic variables in terms of diagnostic accuracy.

Figure 9.

ROC curve estimation for both diagnostic variables (log(ca125) and log(cyfra), represented by the solid and the dashed line, respectively) without covariates and conditioned to different values of the covariates age and $\log (nse)$.

Next, we compare the two diagnostic markers taking into account a unidimensional covariate using the test proposed in [7] for dependent diagnostic markers. We consider the covariates age and $\log (nse)$, each one at a time. We test the equality of the ROC curves conditioned to the values of $\{51,67,83\}$ in the case of age and the values of $\{-0.92,1.14,3.27\}$ in the case of $\log (nse)$. The corresponding ROC curve for every case is estimated in Figure 9. For each considered covariate and each value of the covariate we obtain a p-value of the test, summarized in Table 6. The test is made considering two types of statistics, one based on the $L_2$-measure and the other in the Kolmogorov-Smirnov criteria, although both of them yield similar results. When comparing the ROC curves conditioned on different values of the age, the results are in line with the obtained for the previous case, in which no covariates where taken into account: the equality of the two curves is not rejected. However, when considering the covariate $\log (nse)$, we see that for a certain value (1.14) the null hypothesis is rejected (for a significance level of 5 $\mathrm{\%}$). This matches the representation of the conditional ROC curves depicted in Figure 9.

Table 6.

Results for the comparison of the ROC curves of the diagnostic markers $\log (ca125)$ and $\log (cyfra)$ when considering a unidimensional covariate, that covariate being the age or the $\log (nse)$.

age	51	67	83
p-values ( $L2$)	0.454	0.218	0.936

age	51	67	83
p-values (KS)	0.512	0.202	0.762

$\log (nse)$	-0.92	1.14	3.20
p-values ( $L2$)	0.844	0.012	0.470

$\log (nse)$	-0.92	1.14	3.20
p-values (KS)	0.900	0.008	0.412

Finally, we compare the performance of the two diagnostic variables considering the effect of both the age and the $\log (nse)$ at the same time. This is where we use the methodology proposed in this paper. We test the equality of their respective ROC curves conditioned to nine pairs of values of the two covariates: the ones obtained by making all the possible combinations of $\{51,67,83\}$ and $\{-0.92,1.14,3.27\}$. As before, two different type of statistics were considered: $L_2$ and KS (and once again, the results are similar in both cases). The results obtained are summarized in Table 7.

Table 7.

Results for the comparison of the ROC curves of the diagnostic markers log(ca125) and log(cyfra) when considering the multidimensional covariate (age, $\log (nse)$) for the $L_2$ and the KS statistics (to the left and to the right, respectively).

$\log (nse)$
age	51	67	83
3.20	0.026	0.056	0.010
1.14	0.152	0.070	0.004
-0.92	0.000	0.030	0.258

$\log (nse)$
age	51	67	83
3.20	0.066	0.196	0.032
1.14	0.212	0.050	0.016
-0.92	0.004	0.048	0.424

Note that in this case we did not represent the estimated ROC curves conditioned to the bidimensional covariate $(age,\log (nse))$. This is to stress the fact that, with this methodology, ${\widehat{ROC}}^{\mathit{x}}$ (with $\mathit{x}$ bidimensional) does not need to be computed at all.

The obtained p-values show that, depending on the pair of values of the covariate considered, we can find significative differences between the ROC curves of the $\log (ca125)$ and the $\log (cyfra)$ markers, including pairs of values that when considered separately in the previous test did not rejected the null hypothesis. Likewise, finding differences between the ROC curves conditioned to marginal covariates at certain values does not mean that those differences will be significant when considering the multidimensional covariates (for example, when we conditioned the ROC curves marginally to the value of 1.14 $\log (nse)$ we find differences, but when considering both covariates this difference between the ROC curves only remains significant for the age of 83).

This means that if two patients with pleural effusion entered the doctors office, patient A with 1.14 level of $\log (nse)$ and 67 years old, and patient B also with 1.14 level of $\log (nse)$ but 83 years old, we could apply this methodology and take into account those covariate values to personalize their diagnostic. Without including the covariates in the analysis both diagnostic methods (based on the $\log (ca125)$ and the $\log (cyfra)$) would seem equally effective to detect MPE. However, when we take into account that multidimensional covariate information, we would see that for patient B (and not for patient A) there are significant differences between the two diagnostic markers.

5. Discussion

In this work a new non-parametric methodology has been presented for comparing two or more dependent ROC curves conditioned to the value of a continuous multidimensional covariate. This method combines existing techniques for reducing the dimension in goodness-of-fit tests and for estimating and comparing ROC curves conditioned to a one-dimensional covariate. Although in this paper we have used a induced regression models to include the covariate effect in the ROC curves, we believe that this test could be adapted and extended to other estimation techniques. This opens the door to future research that could include longitudinal or functional covariates, using, for example, the approach presented in [10] for extending the induced ROC methodology to functional covariates.

A simulation study was carried out in order to analyse the practical performance of the test. Two different functions were proposed for the construction of the statistic, the $L_2$ and the KS, the second one being a little more conservative. Different correlations between the diagnostic variables and different sample sizes have been considered, including uneven ones without any appreciable effect on the test performance.

The behaviour of the test was also studied for different distributions of the covariates. It showed that, even if those distribution do not seem to affect the test, the lack of data around the point $\mathit{x}$ at which we perform the test does reduce its power, so the points taken too close to the boundary of the sample range should be analysed with precaution.

Finally, the methodology was illustrated by means of an application to a data set: with this new test it was possible to detect differences on the discriminatory ability of two diagnostic variables conditioned to two different covariates without the need of an estimator of an ROC curve conditioned to a multidimensional covariate. With this application it becomes clear the importance of being able to include the effect of multidimensional covariates to the ROC curves analysis, as different conclusions could be drawn of the comparison of those curves when considering a multidimensional covariate, when considering unidimensional covariates or when excluding the covariates from the study. It can also be an important tool for the personalized medicine, as it allows us to compare different diagnostic methodologies using the personal information of each patient.

Appendix A. Proofs

The proofs needed for Lemma 2.1 are presented below.

Lemma A.1

[5] or [3]: Given a random variable Y such that $\mathbb{E}|Y|<\mathrm{\infty }$,

$$\mathbb{E}[Y|\mathit{X}]=0a.s.\iff \mathbb{E}[Y|{\mathit{\beta }}^{\mathbf{\prime }}\mathit{X}]=0a.s.\text{ for any vector }\mathit{\beta }\in {\mathbb{S}}^{d-1}.$$

From now on it will be assumed that all directions $\mathit{\beta }$ considered satisfy $\mathit{\beta }\in {\mathbb{S}}^{d-1}$.

Lemma A.2

Let $Y_1,\dots ,Y_K$ be K dependent random variables with cumulative distribution functions $F_1,\dots ,F_K$, respectively, such that $\mathbb{E}|Y_k|<\mathrm{\infty }$ for every $k\in \{1,\dots ,K\}$. Let $\mathit{X}$ be a multidimensional covariate. Then, given $c_1,\dots ,c_K$,

$$F_1(c_1|\mathit{X})=\cdots =F_K(c_K|\mathit{X})a.s.\iff F_1^{\mathit{\beta }}(c_1|{\mathit{\beta }}^{\mathbf{\prime }}\mathit{X})=\cdots =F_K^{\mathit{\beta }}(c_K|{\mathit{\beta }}^{\mathbf{\prime }}\mathit{X})a.s.\mathrm{\forall }\mathit{\beta },$$ (A1)

with $\mathit{\beta }\in {\mathbb{S}}^{d-1}$ and where $F_i^{\mathit{\beta }}(c_i|{\mathit{\beta }}^{\mathbf{\prime }}\mathit{X})=P(Y_i⩽c_i|{\mathit{\beta }}^{\mathbf{\prime }}\mathit{X}={\mathit{\beta }}^{\mathbf{\prime }}\mathit{X})$ for $i\in \{1,\dots ,K\}$.

Proof.

It is proven for K = 2.

Seeing that $F_1(c_1|\mathit{X})=F_2(c_2|\mathit{X})a.s.$ is the same as proving that $\mathbb{E}[I(Y_1⩽c_1)|\mathit{X}]=\mathbb{E}[I(Y_2⩽c_2)|\mathit{X}]a.s.$, which, given that the random variables are dependent in the sense that they are conditioned to the same covariate $\mathit{X}$, is equivalent to $\mathbb{E}[I(Y_1⩽c_1)-I(Y_2⩽c_2)|\mathit{X}]=0a.s.$ Now, because of Lemma A.1, this is the same as saying that

$$\mathbb{E}[I(Y_1⩽c_1)-I(Y_2⩽c_2)|{\mathit{\beta }}^{\mathbf{\prime }}\mathit{X}]=0a.s.\mathrm{\forall }\mathit{\beta }.$$

Using again the dependence between the random variables, that is equivalent to $\mathbb{E}[I(Y_1⩽c_1)|{\mathit{\beta }}^{\mathbf{\prime }}\mathit{X}]=\mathbb{E}[I(Y_2⩽c_2)|{\mathit{\beta }}^{\mathbf{\prime }}\mathit{X}]a.s.\mathrm{\forall }\mathit{\beta }$, which in turn is equivalent to $F_1^{\mathit{\beta }}(c_1|{\mathit{\beta }}^{\mathbf{\prime }}\mathit{X})=F_2^{\mathit{\beta }}(c_2|{\mathit{\beta }}^{\mathbf{\prime }}\mathit{X})a.s.\mathrm{\forall }\mathit{\beta }$.

Definition A.3

The inverted conditional ROC curve $($ IROC $)$ is defined as:

$$IROC(p)=1-G(F^{-1}(1-q)),q\in (0,1).$$

Related to the previous definition, the inverted conditional ROC curve ( $IRO{C}^x$), given the pair $(x^F,x^G)\in R_{X^F}\times R_{X^G}$, is defined as:

$$IRO{C}^{x^G,x^F}(q)=1-G(F^{-1}(1-q|x^F)|x^G),q\in (0,1).$$

Lemma A.4

The equality of ROC curves is equivalent to the equality of the inverted ROC curves, i.e.

$$RO{C}_1(p)=\cdots =RO{C}_K(p)\mathrm{\forall }p\in (0,1)\iff IRO{C}_1(q)=\cdots =IRO{C}_K(q)\mathrm{\forall }q\in (0,1).$$

Moreover, the same property holds when talking about conditional ROC curves. Given the pair $(x^F,x^G)\in R_{X^F}\times R_{X^G}$, $RO{C}_1^{x^F,x^G}(p)=\cdots =RO{C}_K^{x^F,x^G}(p)\mathrm{\forall }p\in (0,1)$ holds if and only if $IRO{C}_1^{x^G,x^F}(q)=\cdots =IRO{C}_K^{x^G,x^F}(q)\mathrm{\forall }q\in (0,1)$.

Proof.

It is proven for the unconditional case, and for K = 2. The conditional case is similar.

$$RO{C}_1(p)=RO{C}_2(p)\mathrm{\forall }p\in (0,1)\iff 1-F_1(G_1^{-1}(1-p))=1-F_2(G_2^{-1}(1-p))\mathrm{\forall }p\in (0,1).$$

Take $q=1-F_2(G_2^{-1}(1-p))$ (and hence, $q=RO{C}_2(p)$). q will take all the values in $(0,1)$, and thus, $p=1-G_2(F_2^{-1}(1-q))=IRO{C}_2(q)$. Then,

$$\begin{array}{rl}RO{C}_1(p)& =RO{C}_2(p)\mathrm{\forall }p\in (0,1)\\ & \iff 1-F_1(G_1^{-1}(1-(1-G_2(F_2^{-1}(1-q)))))\\ & =q\mathrm{\forall }q\in (0,1)\\ & \iff 1-G_2(F_2^{-1}(1-q)=1-G_1(F_1^{-1}(1-q))\mathrm{\forall }q\in (0,1)\\ & \iff IRO{C}_2(q)=IRO{C}_1(q)\mathrm{\forall }q\in (0,1).\text{■}\end{array}$$

Proof Proof of Lemma 2.1 —

It is proven for K = 2. For $p\in (0,1)$,

$$RO{C}_1^{\mathit{x}}(p)=RO{C}_2^{\mathit{x}}(p)a.s.\iff F_1(G_1^{-1}(1-p|\mathit{x})|\mathit{x})=F_2(G_2^{-1}(1-p|\mathit{x})|\mathit{x})a.s.$$

Using Lemma A.2, that is equivalent to

$$F_1^{{\mathit{\beta }}^F}(G_1^{-1}(1-p|\mathit{x})|({\mathit{\beta }}^F{)}^{\mathrm{\prime }}\mathit{x})=F_2^{{\mathit{\beta }}^F}(G_2^{-1}(1-p|\mathit{x})|({\mathit{\beta }}^F{)}^{\mathrm{\prime }}\mathit{x})a.s.\mathrm{\forall }{\mathit{\beta }}^F,$$

which in turn is the same as saying that $RO{C}_1^{({\mathit{\beta }}^F{)}^{\mathrm{\prime }}\mathit{x},\mathit{x}}(p)=RO{C}_2^{({\mathit{\beta }}^F{)}^{\mathrm{\prime }}\mathit{x},\mathit{x}}(p)a.s.\mathrm{\forall }{\mathit{\beta }}^F.$ Now, using Lemma A.4 we know that this is equivalent to

$$IRO{C}_1^{\mathit{x},({\mathit{\beta }}^F{)}^{\mathrm{\prime }}\mathit{x}}(q)=IRO{C}_2^{\mathit{x},({\mathit{\beta }}^F{)}^{\mathrm{\prime }}\mathit{x}}(q)a.s.\mathrm{\forall }{\mathit{\beta }}^F\mathrm{for}q\in (0,1).$$

By the definition of the inverted conditional ROC curve, that is the same as saying that

$$G_1((F_1^{{\mathit{\beta }}^F}{)}^{-1}(1-q|({\mathit{\beta }}^F{)}^{\mathrm{\prime }}\mathit{x})|\mathit{x})=G_2((F_2^{{\beta }^F}{)}^{-1}(1-q|({\mathit{\beta }}^F{)}^{\mathrm{\prime }}\mathit{x})|\mathit{x})a.s.\mathrm{\forall }{\mathit{\beta }}^F.$$

Using again the result of Lemma A.2, the previous statement is equivalent to

$$G_1^{{\mathit{\beta }}^G}((F_1^{{\mathit{\beta }}^F}{)}^{-1}(1-q|({\mathit{\beta }}^F{)}^{\mathrm{\prime }}\mathit{x})|({\mathit{\beta }}^G{)}^{\mathrm{\prime }}\mathit{x})=G_2^{{\mathit{\beta }}^G}((F_2^{{\mathit{\beta }}^F}{)}^{-1}(1-q|({\mathit{\beta }}^F{)}^{\mathrm{\prime }}\mathit{x})|({\mathit{\beta }}^G{)}^{\mathrm{\prime }}\mathit{x})a.s.\mathrm{\forall }{\mathit{\beta }}^F,{\mathit{\beta }}^G.$$

By definition, that is the same as saying that $IRO{C}_1^{({\mathit{\beta }}^G{)}^{\mathrm{\prime }}\mathit{x},({\mathit{\beta }}^F{)}^{\mathrm{\prime }}\mathit{x}}(q)=IRO{C}_2^{({\mathit{\beta }}^G{)}^{\mathrm{\prime }}\mathit{x},({\mathit{\beta }}^F{)}^{\mathrm{\prime }}\mathit{x}}(q)a.s.\mathrm{\forall }{\mathit{\beta }}^F,{\mathit{\beta }}^G$ and, using again Lemma A.4, that is equivalent to

$$RO{C}_1^{({\mathit{\beta }}^F{)}^{\mathrm{\prime }}\mathit{x},({\mathit{\beta }}^G{)}^{\mathrm{\prime }}\mathit{x}}(\tilde{p})=RO{C}_2^{({\mathit{\beta }}^F{)}^{\mathrm{\prime }}\mathit{x},({\mathit{\beta }}^G{)}^{\mathrm{\prime }}\mathit{x}}(\tilde{p})a.s.\mathrm{\forall }{\mathit{\beta }}^F,{\mathit{\beta }}^G\text{for }\tilde{p}\in (0,1),$$

where $F_i^{{\mathit{\beta }}^F}(c|({\mathit{\beta }}^F{)}^{\mathrm{\prime }}\mathit{x})=P(Y_i^F⩽c|({\mathit{\beta }}^F{)}^{\mathrm{\prime }}{\mathit{X}}^F=({\mathit{\beta }}^F{)}^{\mathrm{\prime }}\mathit{x})$ and $G_i^{{\mathit{\beta }}^G}(c|({\mathit{\beta }}^G{)}^{\mathrm{\prime }}\mathit{x})=P(Y_i^G⩽c|({\mathit{\beta }}^G{)}^{\mathrm{\prime }}{\mathit{X}}^G=({\mathit{\beta }}^G{)}^{\mathrm{\prime }}\mathit{x})$ for i = 1, 2.

Funding Statement

The research of A. Fanjul-Hevia is supported by the Ministerio de Educación, Cultura y Deporte (fellowship FPU14/05316), as well as by the Spanish Ministerio de Educación y Formación Profesional (Mobility Grant EST18/00673). A. Fanjul-Hevia, W. González-Manteiga and I. Van Keilegom acknowledge the support by the Grant PID2020-116587GB-I00 from Spanish Ministerio de Ciencia e Innovación (MCIN/AEI/ 10.13039/501100011033). J.C. Pardo-Fernández acknowledges financial support by the Grant PID2020-118101GB-I00 from Spanish Ministerio de Ciencia e Innovación (MCIN/AEI/10.13039/501100011033). I. Van Keilegom is financially supported by the European Research Council (2016-2021, Horizon 2020 / ERC grant agreement No. 694409).

Disclosure statement

No potential conflict of interest was reported by the author(s).