Performance Analysis of 3-class Classifiers: Properties of a 3D ROC Surface and the Normalized Volume Under the Surface for the Ideal Observer

Abstract

Classification of a given observation to one of three classes is an important task in many decision processes or pattern recognition applications. A general analysis of the performance of three-class classifiers results in a complex six-dimensional (6D) receiver operating characteristic (ROC) space, for which no simple analytical tool exists at present. We investigate the performance of an ideal observer under a specific set of assumptions that reduces the 6D ROC space to 3D by constraining the utilities of some of the decisions in the classification task. These assumptions lead to a 3D ROC space in which the true-positive fraction (TPF) can be expressed in terms of the two types of false-positive fractions (FPFs). We demonstrate that the TPF is uniquely determined by, and therefore is a function of, the two FPFs. The domain of this function is shown to be related to the decision boundaries in the likelihood ratio plane. Based on these properties of the 3D ROC space, we can define a summary measure, referred to as the normalized volume under the surface (NVUS), that is analogous to the area under the ROC curve (AUC) for a two-class classifier. We further investigate the properties of the 3D ROC surface and the NVUS for the ideal observer under the condition that the three class distributions are multivariate normal with equal covariance matrices. The probability density functions (pdfs) of the decision variables are shown to follow a bivariate log-normal distribution. By considering these pdfs, we express the TPF in terms of the FPFs, and integrate the TPF over its domain numerically to obtain the NVUS. In addition, we performed a Monte Carlo simulation study, in which the 3D ROC surface was generated by empirical “optimal” classification of case samples in the multi-dimensional feature space following the assumed distributions, to obtain an independent estimate of NVUS. The NVUS value obtained by using the analytical pdfs was found to be in good agreement with that obtained from the Monte Carlo simulation study. We also found that, under all conditions studied, the NVUS increased when the difficulty of the classification task was reduced by changing the parameters of the class distributions, thereby exhibiting the properties of a performance metric in analogous to AUC. Our results indicate that, under the conditions that lead to our 3D ROC analysis, the performance of a 3-class classifier may be analyzed by considering the ROC surface, and its accuracy characterized by the NVUS.

I. INTRODUCTION

Receiver operating characteristic (ROC) analysis has been a very important tool for the assessment of classification problems in which the ground truth is a binary reference standard (for example, presence or absence of disease), i.e., two-class classification problems. Many problems in pattern recognition involve multi-class classification, where the task can be defined as assigning a given observation vector to one of several classes. Some practical applications in the medical field in recent literature include: (1) diagnosis of oral cancer as high-grade squamous cell carcinoma, low-grade squamous cell carcinoma, leukoplakia, and normal squamous tissue [1]; (2) classification of the voxels in knee MRI volumes as tibial cartilage, femoral cartilage or the background [2]; (3) classification of ECG signals as normal, ventricular premature and supraventricular premature beats [3]; and (4) classification of pigmented skin lesions as benign, dysplastic nevi, or cutaneous melanoma [4]. In all of the applications above, the task of the designed classifier is to solve the multi-class discrimination task, as opposed to solving a number of binary decision tasks. In three of them, the percentage correct classification was used as the performance metric. From ROC analysis, it is known that the percentage correct classification is an incomplete, and sometimes misleading measure. This motivates us to investigate the performance limits for an ideal observer model for three-class classification, and whether a summary measure, analogous to the area under the ROC curve (AUC) in two-class problems can be defined for the three-class task.

The interest on three-class (or, more generally, multi-class) classification has prompted many researchers to investigate how ROC analysis can be generalized to multi-class problems [5–13]. Some of these investigations [7, 9–13] studied the three-class ROC problem when the decision variables are related to two likelihood ratios generated by the ideal observer, (see (3) below), while others studied the problem for one or two decision variables that may not necessarily represent the optimal decision variables [5, 6, 8]. Several of these studies involved assumptions that resulted in simplified ROC surfaces [5–8, 12, 13], while others investigated the ideal-observer performace in a general 6D ROC space [9–11]. We are interested in the application of three-class classifiers to computer-aided diagnosis (CAD), where many problems involve three possible outcomes. Examples include the classification of a region of interest (ROI) as malignant, normal, or benign in the problem of mammographic lesion detection/characterization, and the classification of a detected mass ROI as a spiculated malignant, non-spiculated malignant, or benign mass in the problem of mammographic lesion characterization.

In this study, we investigate the three-class ROC problem based on the decision variables used by the ideal observer. We reduce the 6D ROC space generated by these decision variables to 3D by constraining the utilities of some of the decisions in the classification task. This approach was previously taken by our research group [7, 12], as well as recently by others [13]. The difference between our approach and that of He et al. is that we chose constraints such that the 3D ROC analysis can be performed in terms of the trade-off between the true-positive fraction (TPF) and two false-positive fractions (FPFs), and therefore is analogous to the two-class ROC curve. The specific set of assumptions imposed by He et al. [13] results in a three-class ROC surface in which the axes are the correct decision fractions for the three classes. The area under the ROC curve plays an important role in the evaluation of two-class classifiers. In analogy, we propose a normalized volume under the ROC surface (NVUS) as a figure of merit (FOM) for 3-class classification under our utility constraints. We investigate the properties of the 3D ROC surface and the properties of the NVUS, and present simulation results under the equal covariance Gaussian assumption for the three classes.

II. METHODS

A. 3D ROC surface

In a two-class medical diagnosis task, the ROC curve is defined as the locus of TPF and FPF decision pairs, which represents the tradeoff between the sensitivity and specificity. For a three-class problem, a similar analysis of the tradeoff among different correct and incorrect decisions results in a 5-dimensional ROC hypersurface in a 6-dimensional probability space. A complete characterization of the ROC hypersurface even for simple class distributions is an important but difficult task because of the high dimensionality of the probability space. An avenue that might lead to a practical analysis method, as well as shed light on the general 3-class classification task, is to make simplifying assumptions that focus on a certain useful subclass while still preserving the nature of the problem. A specific problem of interest in the computer-aided diagnosis (CAD) community is that of classifying a case (or an ROI) as malignant, benign, and normal. In this study, we make simplifying assumptions that may result in a practical ROC analysis for the type of problems that satisfy the assumptions, explore the properties of the resulting ROC hypersurface, and define an FOM based on the ROC hypersurface that may be useful for summarizing the classification performance.

In a three-class classification problem, the task is to assign a given D-dimensional observation vector w⃗, to one of three classes. In this study, motivated by the classification of cases as malignant, benign, and normal, these classes are denoted by π_M, π_B, and π_N, respectively. Since there are three possibilities of true class membership, denoted by t_M, t_B, and t_N, and three possibilities of class assignment for the observation, denoted by d_M, d_B, and d_N, there are a total of nine distinct true membership-assignment pairs. When w⃗ is a random vector, the class assignment d is a random variable, that can take three distinct values, d = d_M, d = d_B, or d = d_N (random variables are denoted by boldface letters). The probability of d = d_i given t_j, i.e., the probability of deciding that the observation belongs to class π_i when the true class membership is π_j, is denoted by P_i ≡ P_j(d = _id | t_j), where i ∈ {M, B, N} and j ∈ {M, B, N} Since the sum of these probabilities add to unity for each true class (P_Mj + P_Bj + P_Nj = 1 for j ∈ {M, B, N}), the probability space in a three-class problem is six-dimensional.

In a Bayes’ test, one assigns a utility to each of these nine true membership-assignment pairs. Each decision made for a random vector w⃗ is therefore associated with a utility. The Bayes’ test is designed so that, on the average, the utility is as large as possible, i.e., the expected value of the utility is maximized. Let U_ij (i ∈ {M, B, N}, j ∈ {M, B, N}), denote the utility of d_i given t_j, i.e., the utility of deciding that the observation belongs to class π_i when the true class membership is π_j. If the random vector w⃗ belongs to class π_j, the expected utility conditioned on its class membership is

$$E\{U|{\pi }_j\}={\displaystyle \sum _{i\in \{M,B,N\}}}{P}_{\mathit{\text{ij}}}{U}_{\mathit{\text{ij}}},$$ (1)

The expected utility of the decision rule is then given by averaging over all classes π_j

$$E\{U\}={\displaystyle \sum _{j\in \{M,B,N\}}}P({\pi }_j){\displaystyle \sum _{i\in \{M,B,N\}}}{P}_{\mathit{\text{ij}}}{U}_{\mathit{\text{ij}}},$$ (2)

where P(π_j) is the a-priori probability of class π_j.

Let p_w⃗ (w⃗) denote the probability density function (pdf) of the random vector w⃗, and p_w⃗|πj (w⃗ | π_j) denote the conditional pdf of w⃗ given class π_j. Defining

$$\begin{array}{ccc}{\mathrm{\Lambda }}_{\mathit{\text{BM}}}(\overrightarrow{w})=\frac{p_{\overrightarrow{\mathbf{w}}|{\pi }_B}(\overrightarrow{w}|{\pi }_B)}{p_{\overrightarrow{\mathbf{w}}|{\pi }_M}(\overrightarrow{w}|{\pi }_M)}\hfill & \text{and}\hfill & {\mathrm{\Lambda }}_{\mathit{\text{NM}}}(\overrightarrow{w})=\frac{p_{\overrightarrow{\mathbf{w}}|{\pi }_N}(\overrightarrow{w}|{\pi }_N)}{p_{\overrightarrow{\mathbf{w}}|{\pi }_M}(\overrightarrow{w}|{\pi }_M)},\hfill \end{array}$$ (3)

one can show that the application of the Bayes’ test leads to the following decision rules [10, 14]:

• Assign d = d_M if and only if

  $$P({\pi }_M)(U_{\mathit{\text{MM}}}-U_{\mathit{\text{BM}}})+P({\pi }_B)(U_{\mathit{\text{MB}}}-U_{\mathit{\text{BB}}}){\mathrm{\Lambda }}_{\mathit{\text{BM}}}(\overrightarrow{w})+P({\pi }_N)(U_{\mathit{\text{MN}}}-U_{\mathit{\text{BN}}}){\mathrm{\Lambda }}_{\mathit{\text{NM}}}(\overrightarrow{w})\ge 0,\text{and}$$ (4)

  $$P({\pi }_M)(U_{\mathit{\text{MM}}}-U_{\mathit{\text{NM}}})+P({\pi }_B)(U_{\mathit{\text{MB}}}-U_{\mathit{\text{NB}}}){\mathrm{\Lambda }}_{\mathit{\text{BM}}}(\overrightarrow{w})+P({\pi }_N)(U_{\mathit{\text{MN}}}-U_{\mathit{\text{NN}}}){\mathrm{\Lambda }}_{\mathit{\text{NM}}}(\overrightarrow{w})\ge 0$$ (5)
• Assign d = d_B if and only if (4) is not satisfied, and

  $$P({\pi }_M)(U_{\mathit{\text{BM}}}-U_{\mathit{\text{NM}}})+P({\pi }_B)(U_{\mathit{\text{BB}}}-U_{\mathit{\text{NB}}}){\mathrm{\Lambda }}_{\mathit{\text{BM}}}(\overrightarrow{w})+P({\pi }_N)(U_{\mathit{\text{BN}}}-U_{\mathit{\text{NN}}}){\mathrm{\Lambda }}_{\mathit{\text{NM}}}(\overrightarrow{w})\ge 0$$ (6)

Assign d = d_N if and only if after performing the tests in (4)–(6), neither d = d_M nor d = d_B is assigned.

Equations (4)–(6) define the decision boundary lines in the likelihood-ratio plane, defined by Λ_BM (w⃗) and Λ_NM (w⃗) as the abscissa and ordinate, respectively. These three lines intersect at one point [14], and partition the decision plane (the likelihood-ratio plane) into three distinct decision regions. The relationship among these decision boundary lines has recently been investigated by Edwards et al. [10].

In a 2-class problem, the ROC curve for the ideal observer can be considered as the relationship between the sensitivity and (1-specificity) as the threshold of the likelihood ratio varies, i.e., as the likelihood ratio line is partitioned into two regions. The ROC hypersurface for a 3-class problem can similarly be considered as the relationship among various decision probabilities as the partitioning of the decision plane, defined by (4)–(6), varies. An independent set of three equations can be characterized by three slopes and three intercepts. However, since the decision boundary lines are not independent, but intersect at one point, one degree of freedom is lost, resulting in a five dimensional ROC hypersurface in a six-dimensional probability space.

In this study, we reduce the dimensionality of the ROC hypersurface by imposing constraints on (4)–(6). For the purpose of normalization, we assume that 0 ≤ U_ij ≤ 1. The utility of a correct decision is assumed to be at a maximum of 1, thus U_MM = 1, U_BB = 1, and U_NN = 1. If a malignant case is misdiagnosed as normal or benign, the utilities will be at a minimum of 0, thus U_BM = 0, and U_NM = 0. If a normal case is called benign or vice versa, it may not be very harmful or costly, so that the utilities are still high with U_BN = 1 and U_NB = 1. The utilities of misdiagnosing benign or normal to be malignant, U_MB and U_MN, depend on the further course of action, and will be variable between 0 and 1. Under these conditions, we note that (4) and (5) coincide, and can be written as:

$$P({\pi }_M)+P({\pi }_B)(U_{\mathit{\text{MB}}}-1){\mathrm{\Lambda }}_{\mathit{\text{BM}}}(\overrightarrow{w})+P({\pi }_N)(U_{\mathit{\text{MN}}}-1){\mathrm{\Lambda }}_{\mathit{\text{NM}}}(\overrightarrow{w})\ge 0,$$ (7)

while all coefficients in (6) equal zero and it becomes indeterminate. For the assumed values of the utilities, the classification problem reduces to deciding whether the observation belongs to π_M or not, i.e., assign w⃗ to class M if and only if (7) holds. However, since U_MB and U_MN are variable, and are not necessarily equal, there is still a distinction between the benign and normal classes, and the problem is still a three-class classification task.

With these assumptions, the expected utility in (2) becomes a function of only three of the P_ij's, namely, P_MM (probability of correctly deciding that the observation is malignant) P_MB (probability of falsely deciding that the observation is malignant when the observation belongs to the benign class), and P_MN (probability of falsely deciding that the observation is malignant when the observation belongs to the normal class). One can thus analyze how the TPF (P_MM) varies as a function of the two FPFs (P_MB and P_MN). To differentiate the resulting surface in 3D space from both the ROC curve for a 2-class problem (2D ROC curve) and from the general ROC hypersurface for a 3-class problem, we refer to it as the “3D ROC surface”. Note that, similar to a 2D ROC curve, our 3D ROC surface describes the variation of the TPF as a function of the FPFs, and results from the simplification of the general ROC hypersurface into 3D because of the assumptions made for the utilities.

We note that the assumptions for the utilities described above are only a special case of the choices for the utilities that will degenerate the three decision lines into a single line. Specifically, if

$$U_{\mathit{\text{BM}}}=U_{\mathit{\text{NM}}}={\upsilon }_1,U_{\mathit{\text{BB}}}=U_{\mathit{\text{NB}}}={\upsilon }_2,\text{and}{U}_{\mathit{\text{BN}}}=U_{\mathit{\text{NN}}}={\upsilon }_3$$ (8)

then (4) and (5) coincide, and (6) becomes undefined. These assumptions are summarized in Table 1, and imply that (i) the utility of classifying a malignant tumor as benign is the same as that of classifying it as normal, (ii) the utility of classifying a benign lesion as normal is the same as that of classifying it correctly, and (iii) the utility of classifying a normal region as a benign lesion is the same as that of classifying it correctly. In this case, the expected utility in (2) can be written as a function of P_MM (sensitivity), P_MB and P_MN (two kinds of FPFs):

$$E\{U\}=(U_{\mathit{\text{MM}}}-U_{\mathit{\text{BM}}})P({\pi }_M)P_{\mathit{\text{MM}}}+(U_{\mathit{\text{MB}}}-U_{\mathit{\text{BB}}})P({\pi }_B)P_{\mathit{\text{MB}}}+(U_{\mathit{\text{MN}}}-U_{\mathit{\text{BN}}})P({\pi }_N)P_{\mathit{\text{MN}}}+U_{\mathit{\text{BM}}}P({\pi }_M)+U_{\mathit{\text{BB}}}P({\pi }_B)+U_{\mathit{\text{BN}}}P({\pi }_N)$$ (9)

Table I.

The conditions for the utilities that will degenerate the three decision lines into a single line. For a problem such as classification of a case as malignant, benign and normal, the utilities may be chosen as υ₀=υ₂=υ₃=1 and υ₁=0.

True class membership	M	B	N
Assigned class membership
M	υ0	UMB	UMN
B	υ1	υ2	υ3
N	υ1	υ2	υ3

B. The likelihood-ratio plane

Figure 1 describes the decision line provided by (7) in the likelihood ratio plane, where the abscissa and ordinate are Λ_BM and Λ_NM, respectively. The equation for the decision line can be expressed as

$${\mathrm{\Lambda }}_{\mathit{\text{NM}}}=m{\mathrm{\Lambda }}_{\mathit{\text{BM}}}+n,$$ (10)

where m and n are the slope and intercept of the line, respectively

$$m=-\frac{P({\pi }_B)}{P({\pi }_N)}\frac{1-U_{\mathit{\text{MB}}}}{1-U_{\mathit{\text{MN}}}},$$ (11)

and

$$n=\frac{P({\pi }_M)}{P({\pi }_N)}\frac{1}{1-U_{\mathit{\text{MN}}}}.$$ (12)

Fig. 1.

The likelihood ratio (LR) plane and the decision line.

Since all utilities are by definition less than or equal to unity, we have − ∞ ≤ m ≤ 0, and 0 ≤ n ≤ ∞ as the utilities and the a-priori class probabilities vary.

From Fig. 1, we observe that for a given w⃗ we assign d = d_M if the two likelihood ratios are such that the point (Λ_BM(w⃗), Λ_NM(w⃗)) is within the triangle S, and assign d ≠ d_M otherwise. Since the likelihood ratios are functions of the random vector w⃗, Λ_BM(w⃗) and Λ_NM(w⃗) are random variables, and have a joint pdf under π_M, π_B, and π_N. The TPF and FPFs in which we are interested can be expressed as

$$P_{\mathit{\text{MM}}}={\displaystyle \underset{S}{\iint }}{p}_{{\mathbf{\Lambda }}_{\mathbf{\text{BM}}},{\mathbf{\Lambda }}_{\mathbf{\text{NM}}}|{\mathit{\pi }}_{\mathbf{M}}}({\mathrm{\Lambda }}_{\text{BM}},{\mathrm{\Lambda }}_{\text{NM}}|{\pi }_{\mathrm{M}})d{\mathrm{\Lambda }}_{\text{BM}}d{\mathrm{\Lambda }}_{\text{NM}}$$ (13)

$$P_{\mathit{\text{MB}}}={\displaystyle \underset{S}{\iint }}{p}_{{\mathbf{\Lambda }}_{\mathbf{\text{BM}}},{\mathbf{\Lambda }}_{\mathbf{\text{NM}}}|{\mathit{\pi }}_{\mathbf{B}}}({\mathrm{\Lambda }}_{\text{BM}},{\mathrm{\Lambda }}_{\text{NM}}|{\pi }_{\mathrm{B}})d{\mathrm{\Lambda }}_{\text{BM}}d{\mathrm{\Lambda }}_{\text{NM}}$$ (14)

$$P_{\mathit{\text{MN}}}={\displaystyle \underset{S}{\iint }}{p}_{{\mathbf{\Lambda }}_{\mathbf{\text{BM}}},{\mathbf{\Lambda }}_{\mathbf{\text{NM}}}|{\mathit{\pi }}_{\mathbf{N}}}({\mathrm{\Lambda }}_{\text{BM}},{\mathrm{\Lambda }}_{\text{NM}}|{\pi }_{\mathrm{N}})d{\mathrm{\Lambda }}_{\text{BM}}d{\mathrm{\Lambda }}_{\text{NM}}$$ (15)

In the ROC plots that we construct, we chose to express the TPF as a function of the two FPFs. To gain insight into the interdependence among these three quantities of interest, we first study how the two FPFs are related. Consider a fixed value of m = m^*. By a change of variables Λ_BM = u and Λ_NM = v + m^*u, (14) can be written as

$$P_{\mathit{\text{MB}}}={\displaystyle \underset{\mathrm{v}=0}{\overset{\mathrm{n}}{\int }}}\mathit{\text{dv}}{\displaystyle \underset{u=0}{\overset{-v/m^{*}}{\int }}}{p}_{{\mathbf{\Lambda }}_{\mathbf{\text{BM}}},{\mathbf{\Lambda }}_{\mathbf{\text{NM}}}|{\mathit{\pi }}_{\mathbf{B}}}(u,v+m^{*}u)\mathit{\text{du}}={\displaystyle \underset{\mathrm{v}=0}{\overset{\mathrm{n}}{\int }}}{f}_{B,m^{*}}(v)\mathit{\text{dv}}$$ (16)

where

$$f_{B,m^{*}}(v)={\displaystyle \underset{\mathrm{u}=0}{\overset{-\mathrm{v}/{\mathrm{m}}^{*}}{\int }}}{p}_{{\mathbf{\Lambda }}_{\mathbf{\text{BM}}},{\mathbf{\Lambda }}_{\mathbf{\text{NM}}}|{\mathit{\pi }}_{\mathbf{B}}}(u,v+m^{*}u)\mathit{\text{du}}.$$ (17)

Although the new expression for P_MB seems more complicated, it has a meaningful interpretation. f_B,m^* (v) can be interpreted as the integral of p_{ΛBM,ΛNM|πB} (Λ_BM, Λ_NM|π_B), i.e., the joint pdf of the likelihood ratios under class B, along a line that is parallel to the Λ_NM = m^*Λ_BM line, and that intersects the Λ_NM axis at Λ_NM = v (Fig. 2a). P_MB is then obtained by integrating f_B,m^* (v) up to n, similar to the integration one might perform in 2-class ROC analysis to obtain the false-negative fraction. Similarly, we also express P_MN as (Fig. 2b)

$$P_{\mathit{\text{MN}}}={\displaystyle \underset{\mathrm{v}=0}{\overset{\mathrm{n}}{\int }}}\mathit{\text{dv}}{\displaystyle \underset{u=0}{\overset{-v/m^{*}}{\int }}}{p}_{{\mathbf{\Lambda }}_{\mathbf{\text{BM}}},{\mathbf{\Lambda }}_{\mathbf{\text{NM}}}|{\mathit{\pi }}_{\mathbf{N}}}(u,v+m^{*}u)\mathit{\text{du}}={\displaystyle \underset{\mathrm{v}=0}{\overset{\mathrm{n}}{\int }}}{f}_{N,m^{*}}(v)\mathit{\text{dv}}$$ (18)

where

$$f_{N,m^{*}}(v)={\displaystyle \underset{\mathrm{u}=0}{\overset{-\mathrm{v}/{\mathrm{m}}^{*}}{\int }}}{p}_{{\mathbf{\Lambda }}_{\mathbf{\text{BM}}},{\mathbf{\Lambda }}_{\mathbf{\text{NM}}}|{\mathit{\pi }}_{\mathbf{N}}}(u,v+m^{*}u)\mathit{\text{du}}.$$ (19)

Fig. 2.

(a) The integral of the joint pdf of the likelihood ratios under class B, along lines with slope m^*. The resulting integral is a 1-D function, and is denoted as f_B,m^* (v). (b) The integral of the joint pdf of the likelihood ratios under class N, along lines with slope m^*. The resulting integral is a 1-D function, and is denoted as f_N,m^* (v). Note that f_B,m^* (v) and f_N,m^* (v) are not Gaussian; in particular, they are zero for v ≤ 0 and u ≤ 0, respectively.

With this interpretation, it can be seen that as n varies while m = m^* is kept constant, the relationship between P_MB and P_MN is similar to an ROC curve (Fig. 3) in the FPF plane, referred to as C_m^* below. In particular, we observe that n=0 implies P_MB = P_MN = 0, and as n → ∞, P_MB = P_MN = 1.

Fig. 3.

The interpretation of P_MN and P_MB in terms of the areas under f_B,m^* (v) and f_N,m^* (v) below a threshold n. This interpretation is similar to that for true-negative and false-negative fractions in a 2-class classification problem.

C. Properties of the ROC surface

In this subsection, we explore two properties of the ROC surface of the ideal observer obtained under the constraints given by (8). Specifically, we show that for a given FPF pair (P_MB, P_MN), multiple values for sensitivity (P_MM) cannot exist. In other words, P_MM is a function of P_MB and P_MN. We also determine the domain of this function. These properties are used in Section III to numerically evaluate the 3D ROC surface.

Property 1: Let P_MB (m₀, n₀) = P_MB (m₁, n₁) and m₁ ≥ m₀. Then P_MN (m₀, n₀) ≥ P_MN (m₁, n₁).

This property is proven in Appendix 1. The relative locations of the curves C_m0 and C_m1 as a result of Property 1 are shown in Fig. 4. Let us define C₀ (P_MB) as the loci of P_MN as P_MB is varied for m = 0, and C_−∞ (P_MB) as the loci of P_MN as P_MB is varied for m = −∞. Since − ∞ ≤ m ≤ 0, it can be concluded that the loci of P_MN in the FPF plane is bounded from above by the curve C_−∞ (P_MB) and below by the curve C₀ (P_MB).

Fig. 4.

The curve C_mi plots the relationship between P_MB and P_MN in the FPF plane for m = m_i. Four such curves are plotted: C_−∞, C_m0, C_m1, and C₀. The domain of the ROC surface is defined by the curves C_−∞ and C₀. The figure shows that when m₁ > m₀, the curve C_m1 is below the curve C_m0.

Property 2: If the pairs (m₀, n₀) and (m₁, n₁) are such that P_MB (m₀, n₀) = P_MB (m₁, n₁) and P_MN (m₀, n₀) = P_MN (m₁, n₁), then P_MM (m₀, n₀) = P_MM (m₁, n₁).

Proof: The proof of this claim is similar to that of Property 1, and is provided in Appendix 2. Property 2 asserts that P_MM can be expressed as a function of P_MB and P_MN, although it may be difficult to express the functional form φ(P_MB, P_MN) ≡ P_MM explicitly for given distributions of the malignant, normal, and benign classes.

Property 3: When P_MB is held constant, P_MM is a non-decreasing function of P_MN. Similarly, when P_MN is held constant, P_MM is a non-decreasing function of P_MB.

Proof: Please refer to Appendix 3.

D. Normalized volume under the surface

For a two-class classification task, the AUC is a commonly used FOM to measure the classification accuracy. AUC can be interpreted as the sensitivity averaged over all possible values of specificities. As described above, in a three-class classification task with the utilities assumed to be those in (8), the sensitivity can be expressed as a function of the two FPFs. It is then natural to define an FOM for a three-class classification task as a function of the average sensitivity over all possible values FPF pairs. The height of the surface P_MM = φ(P_MB, P_MN) is the sensitivity at a given value of the FPF pair (P_MB, P_MN). As described above, in general, the FPF pair (P_MB, P_MN) does not take all possible values in the square [0, 1] × [0, 1]. The total volume V_total under the surface P_MM therefore does not provide the average sensitivity over all possible values of FPF pairs. The domain (P_MB, P_MN) is bounded by the curves C_−∞ and C₀ in the FPF plane as shown in Fig. 4, and the average sensitivity over all possible values of FPF pairs is the volume under the surface P_MM, normalized by the area A_FPF bounded by the curves C_−∞ and C₀. The normalized volume under the surface (NVUS) is thus defined as

$$\mathit{\text{NVUS}}=\frac{{\displaystyle \underset{P_{\mathit{\text{MB}}}=0}{\overset{1}{\int }}}{\mathit{\text{dP}}}_{\mathit{\text{MB}}}{\displaystyle \underset{P_{\mathit{\text{MN}}}=C_0(P_{\mathit{\text{MB}}})}{\overset{C_{-\infty }(P_{\mathit{\text{MB}}})}{\int }}}\phi (P_{\mathit{\text{MB}}},P_{\mathit{\text{MN}}}){\mathit{\text{dP}}}_{\mathit{\text{MN}}}}{{\displaystyle \underset{P_{\mathit{\text{MB}}}=0}{\overset{1}{\int }}}{\mathit{\text{dP}}}_{\mathit{\text{MB}}}{\displaystyle \underset{P_{\mathit{\text{MN}}}=C_0(P_{\mathit{\text{MB}}})}{\overset{C_{-\infty }(P_{\mathit{\text{MB}}})}{\int }}}{\mathit{\text{dP}}}_{\mathit{\text{MN}}}}\equiv \frac{V_{\mathit{\text{total}}}}{A_{\mathit{\text{FPF}}}}$$ (20)

The need for normalization may also be deduced by noting that A_FPF is related to the separation between classes B and N. If the distributions of these two classes are similar, then the FPF values P_MB and P_MN will also be similar, regardless of how the decision line is selected. Therefore, both A_FPF and the spread of the surface P_MM = φ(P_MB, P_MN) will be small. On the other hand, relatively dissimilar distributions of the classes B and N will result in a larger value of A_FPF and a larger spread of the surface, which will lead to a larger volume under the surface, provided that the separation between classes M and B and between classes M and N remain constant. However, as seen from (9), the separation between classes B and N has no impact on the expected utility, and therefore the effect of the A_FPF on V_total needs to be removed by normalization.

Gaussian distributions

In the development of parametric ROC analysis for a two-class problem, a frequently made assumption is that the classifier scores (i.e., the decision variable) for the two classes have a binormal distribution. One can then either fit a maximum likelihood curve to the classifier scores [15] or to the likelihood ratio [16]. Although we do not attempt to fit surfaces to our 3D ROC plots in this work, we believe that it is important to examine the shape of the ROC surface and to demonstrate the feasibility of our approach to using NVUS as a performance metric for the important sub-category of Gaussian distributed data. We thus assume that the observation w⃗ is normally distributed under each class, π_M, π_B, and π_N. To facilitate the analysis, we also assume a simple case in which the covariance matrices of the Gaussian distribution under the three classes are equal, and the mean vectors are different. Without loss of generality, one can then diagonalize the covariance matrices, and translate the distributions under the three classes so that the mean for class π_A is 0 (Appendix 4). Then, under the equal covariance assumption, one can write

$$\begin{array}{c}\overrightarrow{\mathbf{w}}~\mathrm{N}(0,I)\text{under}{\pi }_M\hfill \\ \overrightarrow{\mathbf{w}}~\mathrm{N}({\overrightarrow{\mu }}_B,I)\text{under}{\pi }_B\hfill \\ \overrightarrow{\mathbf{w}}~\mathrm{N}({\overrightarrow{\mu }}_N,I)\text{under}{\pi }_N\hfill \end{array}$$ (21)

where μ⃗_B and μ⃗_N are the mean vectors under π_B and π_N, respectively, and I is the identity matrix. The likelihood ratios are then

$${\mathbf{\Lambda }}_{\mathbf{\text{BM}}}(\overrightarrow{\mathbf{w}})=\frac{p_{\overrightarrow{\mathbf{w}}|{\pi }_B}(\overrightarrow{w}|{\pi }_B)}{p_{\overrightarrow{\mathbf{w}}|{\pi }_M}(\overrightarrow{w}|{\pi }_M)}=\text{exp}\left\{-\frac{1}{2}\left[{(\overrightarrow{\mathbf{w}}-{\overrightarrow{\mu }}_B)}^T(\overrightarrow{\mathbf{w}}-{\overrightarrow{\mu }}_B)-{\overrightarrow{\mathbf{w}}}^T\overrightarrow{\mathbf{w}}\right]\right\}=\text{exp}\left\{{{\overrightarrow{\mu }}_B}^T\overrightarrow{\mathbf{w}}-{{\overrightarrow{\mu }}_B}^T{\overrightarrow{\mu }}_B/2\right\}$$ (22)

$${\mathbf{\Lambda }}_{\mathbf{\text{NM}}}(\overrightarrow{\mathbf{w}})=\frac{p_{\overrightarrow{\mathbf{w}}|{\pi }_N}(\overrightarrow{w}|{\pi }_N)}{p_{\overrightarrow{\mathbf{w}}|{\pi }_M}(\overrightarrow{w}|{\pi }_M)}=\text{exp}\left\{-\frac{1}{2}\left[{(\overrightarrow{\mathbf{w}}-{\overrightarrow{\mu }}_N)}^T(\overrightarrow{\mathbf{w}}-{\overrightarrow{\mu }}_N)-{\overrightarrow{\mathbf{w}}}^T\overrightarrow{\mathbf{w}}\right]\right\}=\text{exp}\left\{{{\overrightarrow{\mu }}_N}^T\overrightarrow{\mathbf{w}}-{{\overrightarrow{\mu }}_N}^T{\overrightarrow{\mu }}_N/2\right\}$$ (23)

Defining

$$\overrightarrow{\mathbf{z}}=\text{log}(\overrightarrow{\mathbf{\Lambda }})=\left(\begin{array}{c}\text{log}({\mathbf{\Lambda }}_{\mathbf{\text{BM}}})\hfill \\ \text{log}({\mathbf{\Lambda }}_{\mathbf{\text{NM}}})\hfill \end{array}\right),$$ (24)

it can be seen that z⃗ is a linear function of w⃗, with

$$\overrightarrow{\mathbf{z}}=P\overrightarrow{\mathbf{w}}+\mathrm{K},$$ (25)

where

$$P=\left(\begin{array}{c}{\overrightarrow{\mu }}_B^T\hfill \\ {\overrightarrow{\mu }}_N^T\hfill \end{array}\right),$$ (26)

and

$$K=-\frac{1}{2}\left(\begin{array}{c}{\overrightarrow{\mu }}_B^T{\overrightarrow{\mu }}_B\hfill \\ {\overrightarrow{\mu }}_N^T{\overrightarrow{\mu }}_N\hfill \end{array}\right).$$ (27)

Therefore, z⃗ is Gaussian under all three classes with covariance matrix

$${\mathrm{\Sigma }}_z={\mathit{\text{PP}}}^T=\left(\begin{array}{cc}{\overrightarrow{\mu }}_B^T{\overrightarrow{\mu }}_B\hfill & {\overrightarrow{\mu }}_B^T{\overrightarrow{\mu }}_N\hfill \\ {\overrightarrow{\mu }}_B^T{\overrightarrow{\mu }}_N\hfill & {\overrightarrow{\mu }}_N^T{\overrightarrow{\mu }}_N\hfill \end{array}\right)=\left(\begin{array}{cc}\hfill {\sigma }_1^2\hfill & \hfill r{\sigma }_1{\sigma }_2\hfill \\ \hfill r{\sigma }_1{\sigma }_2\hfill & \hfill {\sigma }_2^2\hfill \end{array}\right),$$ (28)

where

$${\sigma }_1=\sqrt{{\overrightarrow{\mu }}_B^T{\overrightarrow{\mu }}_B},{\sigma }_2=\sqrt{{\overrightarrow{\mu }}_N^T{\overrightarrow{\mu }}_N},\text{and}r=\frac{{\overrightarrow{\mu }}_B^T{\overrightarrow{\mu }}_N}{\sqrt{{\overrightarrow{\mu }}_B^T{\overrightarrow{\mu }}_B}\sqrt{{\overrightarrow{\mu }}_N^T{\overrightarrow{\mu }}_N}}$$ (29)

Note that σ₁ and σ₂ are the magnitudes of the vectors μ⃗_B and μ⃗_N, while r is the cosine of the angle between them. The mean of z⃗ under classes M, B, and N are calculated as

$$\begin{array}{c}{\overline{z}}_M=\left(\begin{array}{c}{\overline{z}}_M(1)\hfill \\ {\overline{z}}_M(2)\hfill \end{array}\right)=-\frac{1}{2}\left(\begin{array}{c}{\sigma }_1^2\hfill \\ {\sigma }_2^2\hfill \end{array}\right),\hfill \\ {\overline{z}}_B=\left(\begin{array}{c}{\overline{z}}_B(1)\hfill \\ {\overline{z}}_B(2)\hfill \end{array}\right)=\frac{1}{2}\left(\begin{array}{c}\hfill {\sigma }_1^2\hfill \\ \hfill 2r{\sigma }_1{\sigma }_2-{\sigma }_2^2\hfill \end{array}\right),\text{and}\hfill \\ {\overline{z}}_N=\left(\begin{array}{c}{\overline{z}}_N(1)\hfill \\ {\overline{z}}_N(2)\hfill \end{array}\right)=\frac{1}{2}\left(\begin{array}{c}\hfill 2r{\sigma }_1{\sigma }_2-{\sigma }_1^2\hfill \\ \hfill {\sigma }_2^2\hfill \end{array}\right),\hfill \end{array}$$ (30)

where z̅_j (i), i = 1, 2, denotes the i^th component of the mean vector of z⃗ under class j, j=M, B, N.

Since Λ⃗ = exp(z⃗), Λ⃗ follows a bivariate log-normal distribution. For given values of m and n (defined by (11) and (12)) the sensitivity and false-positive fractions can then be calculated using (13)–(15). For example,

$$P_{\mathit{\text{MM}}}={\displaystyle \underset{{\mathrm{\Lambda }}_{\mathit{\text{BM}}}=0}{\overset{-n/m}{\int }}}{\displaystyle \underset{{\mathrm{\Lambda }}_{\mathit{\text{NM}}}=0}{\overset{m{\mathrm{\Lambda }}_{\mathit{\text{BM}}}+n}{\int }}}{p}_{{\mathbf{\Lambda }}_{\mathbf{\text{BM}}},{\mathbf{\Lambda }}_{\mathbf{\text{NM}}}|{\mathit{\pi }}_{\mathbf{M}}}({\mathrm{\Lambda }}_{\mathit{\text{BM}}},{\mathrm{\Lambda }}_{\mathit{\text{NM}}}|{\pi }_{\mathrm{M}})d{\mathrm{\Lambda }}_{\mathit{\text{BM}}}d{\mathrm{\Lambda }}_{\mathit{\text{NM}}}$$ (31)

where

$$p_{{\mathbf{\Lambda }}_{\mathbf{\text{BM}}},{\mathbf{\Lambda }}_{\mathbf{\text{NM}}}|{\mathit{\pi }}_{\mathbf{M}}}({\mathrm{\Lambda }}_{\mathit{\text{BM}}},{\mathrm{\Lambda }}_{\mathit{\text{NM}}}|{\pi }_{\mathrm{M}})=\frac{1}{2\pi {\sigma }_1{\sigma }_2\sqrt{1-r^2}}\frac{1}{{\mathrm{\Lambda }}_{\mathit{\text{BM}}}}\frac{1}{{\mathrm{\Lambda }}_{\mathit{\text{NM}}}}\text{exp}\{\frac{-1}{2(1-r^2)}\times [\frac{{(\text{log}{\mathrm{\Lambda }}_{\mathit{\text{BM}}}-{\overline{z}}_M(1))}^2}{{\sigma }_1^2}-\frac{2r(\text{log}{\mathrm{\Lambda }}_{\mathit{\text{BM}}}-{\overline{z}}_M(1))(\text{log}{\mathrm{\Lambda }}_{\mathit{\text{NM}}}-{\overline{z}}_M(2))}{{\sigma }_1{\sigma }_2}+\frac{{(\text{log}{\mathrm{\Lambda }}_{\mathit{\text{NM}}}-{\overline{z}}_M(2))}^2}{{\sigma }_2^2}]\}$$ (32)

is the joint pdf for the bivariate log-normal random variables Λ_BM and Λ_NM. The equations for P_MB and P_MN are similar, with z̅_B and z̅_N replacing z̅_M in (32), respectively:

$$P_{\mathit{\text{MB}}}={\displaystyle \underset{{\mathrm{\Lambda }}_{\mathit{\text{BM}}}=0}{\overset{-n/m}{\int }}}{\displaystyle \underset{{\mathrm{\Lambda }}_{\mathit{\text{NM}}}=0}{\overset{m{\mathrm{\Lambda }}_{\mathit{\text{BM}}}+n}{\int }}}d{\mathrm{\Lambda }}_{\mathit{\text{BM}}}d{\mathrm{\Lambda }}_{\mathit{\text{NM}}}\frac{1}{2\pi {\sigma }_1{\sigma }_2\sqrt{1-r^2}}\frac{1}{{\mathrm{\Lambda }}_{\mathit{\text{BM}}}}\frac{1}{{\mathrm{\Lambda }}_{\mathit{\text{NM}}}}\text{exp}\{\frac{-1}{2(1-r^2)}\times [\frac{{(\text{log}{\mathrm{\Lambda }}_{\mathit{\text{BM}}}-{\overline{z}}_B(1))}^2}{{\sigma }_1^2}-\frac{2r(\text{log}{\mathrm{\Lambda }}_{\mathit{\text{BM}}}-{\overline{z}}_B(1))(\text{log}{\mathrm{\Lambda }}_{\mathit{\text{NM}}}-{\overline{z}}_B(2))}{{\sigma }_1{\sigma }_2}+\frac{{(\text{log}{\mathrm{\Lambda }}_{\mathit{\text{NM}}}-{\overline{z}}_B(2))}^2}{{\sigma }_2^2}\left]\right\}$$ (33)

$$P_{\mathit{\text{MN}}}={\displaystyle \underset{{\mathrm{\Lambda }}_{\mathit{\text{BM}}}=0}{\overset{-n/m}{\int }}}{\displaystyle \underset{{\mathrm{\Lambda }}_{\mathit{\text{NM}}}=0}{\overset{m{\mathrm{\Lambda }}_{\mathit{\text{BM}}}+n}{\int }}}d{\mathrm{\Lambda }}_{\mathit{\text{BM}}}d{\mathrm{\Lambda }}_{\mathit{\text{NM}}}\frac{1}{2\pi {\sigma }_1{\sigma }_2\sqrt{1-r^2}}\frac{1}{{\mathrm{\Lambda }}_{\mathit{\text{BM}}}}\frac{1}{{\mathrm{\Lambda }}_{\mathit{\text{NM}}}}\text{exp}\{\frac{-1}{2(1-r^2)}\times [\frac{{(\text{log}{\mathrm{\Lambda }}_{\mathit{\text{BM}}}-{\overline{z}}_N(1))}^2}{{\sigma }_1^2}-\frac{2r(\text{log}{\mathrm{\Lambda }}_{\mathit{\text{BM}}}-{\overline{z}}_N(1))(\text{log}{\mathrm{\Lambda }}_{\mathit{\text{NM}}}-{\overline{z}}_N(2))}{{\sigma }_1{\sigma }_2}+\frac{{(\text{log}{\mathrm{\Lambda }}_{\mathit{\text{NM}}}-{\overline{z}}_N(2))}^2}{{\sigma }_2^2}\left]\right\}$$ (34)

It is seen that for given m and n, the TPF and the FPFs can be completely described in terms of three parameters, σ₁, σ₂, and r. Since the 3D ROC surface is obtained by varying m and n over their allowable range (− ∞ ≤ m ≤ 0, and 0 ≤ n ≤ ∞), the 3D ROC surface is a function of only these three parameters.

Under the equal covariance Gaussian assumption, the upper and lower bounds for P_MN on the FPF plane (C_−∞ (P_MB) and C₀ (P_MB), respectively) can be determined analytically (Appendix 5):

$$C_{-\infty }(P_{\mathit{\text{MB}}})=\mathrm{\Phi }\left[{\mathrm{\Phi }}^{-1}(P_{\mathit{\text{MB}}})+\frac{{\overline{z}}_B(1)-{\overline{z}}_N(1)}{{\sigma }_1}\right]$$ (35)

and

$$C_0(P_{\mathit{\text{MB}}})=\mathrm{\Phi }\left[{\mathrm{\Phi }}^{-1}(P_{\mathit{\text{MB}}})+\frac{{\overline{z}}_B(2)-{\overline{z}}_N(2)}{{\sigma }_2}\right]$$ (36)

where

$$\mathrm{\Phi }(x)=\frac{1}{\sqrt{2\pi }}{\displaystyle \underset{-\infty }{\overset{x}{\int }}{e}^{-t^2/2}\mathit{\text{dt}}}$$ (37)

III. SIMULATION RESULTS

We have performed simulation studies under the equal covariance Gaussian assumption in order to study some properties of the ROC surface and compare the NVUS for different values of the parameters σ₁, σ₂, and r. We also implemented two different methods to construct the ROC surface, and compared the NVUS values obtained by these two methods.

We first investigated the characteristics of the C_m curves, i.e., the relationship between P_MB and P_MN for a given m. For this purpose, we numerically evaluated (33) and (34) for given n and m values to obtain P_MB and P_MN, and plotted the relationship between P_MB and P_MN as n is varied and θ = tan⁻¹ (m) is fixed. The resulting plots for different values of the parameters σ₁, σ₂, and r are shown in Fig. 5 for θ = −90°, −67.5°, −45°, −22.5°, and 0°. The curves for θ = −90° and 0° (defined as C_−∞ (P_MB) and C₀ (P_MB) above) are important because they define the domain of P_MM = φ(P_MB, P_MN). It is seen that this domain can take a number of different convex and non-convex shapes, and the area of the domain increases as r decreases. In the limiting case where μ⃗_B and μ⃗_N are equal (r=1 and σ₁ = σ₂), there is no distinction between the classes π_B and π_N, and the domain of the ROC surface reduces to the diagonal line that passes through (0,0) and (1,1) in the (P_MB, P_MN) plane and the ROC surface reduces to the 2-class ROC curve.

Fig. 5.

The relationship between P_MN and P_MB for different values of the slope of the decision line (m) in the likelihood ratio plane. Each graph shows five values for θ = tan⁻¹ (m): θ = −90°, −67.5°, −45°, −22.5°, and 0°. In the top row of graphs, ${\sigma }_1^2={\sigma }_2^2=3$, and in the bottom row of graphs, ${\sigma }_1^2=1\text{and}{\sigma }_2^2=3$. The value of r is −0.9, 0, and 0.9 for the graphs on the left, middle, and right, respectively.

Fig. 6 shows examples of the ROC surfaces obtained for two sets of values for the ROC parameters. In Fig. 6a, r=0.5 and ${\sigma }_1^2={\sigma }_2^2=3$ so that the resulting ROC surface is symmetric with respect to the P_MB = P_MN line. In Fig. 6b, r=0.5, ${\sigma }_1^2=1\text{and}{\sigma }_2^2=3$, so that the surface is not symmetric, although the domain is still convex. Figures 6c and 6d show the same surfaces as grayscale images, where the brightness of the pixel is proportional to the value of P_MM. From these examples, it is observed that P_MM is a non-decreasing function of P_MB and P_MN, which is a result of Property 3.

Fig. 6.

Examples of 3D ROC surfaces: (a) and (b) are surface representations for Gaussian distributions with r=0.5, ${\sigma }_1^2={\sigma }_2^2=3$, and r=0.5, ${\sigma }_1^2=1,{\sigma }_2^2=3$, respectively. (c) and (d) are grayscale representations of (a) and (b), respectively.

To evaluate the NVUS for different values of the parameters σ₁, σ₂, and r, we implemented two methods. In Method 1, which uses (31) to (34) for the pdf of Λ⃗ directly, we numerically integrated the volume over the domain of P_MM = φ(P_MB, P_MN). For this purpose, we used recursive adaptive Simpson quadrature [17], in the form that it was implemented in MATLAB for double integration. Adaptive Simpson quadrature attempts to minimize the number of function calculations by using small subdivisions of the interval only where required for the desired tolerance [18]. To perform the integration, one needs to numerically express P_MM as a function of P_MB and P_MN. For a given pair of P_MB and P_MN, we first numerically solved the non-linear equations (33) and (34) to determine which values of m and n result in the given FPFs. We then substituted these values into (31) and performed numerical integration to obtain P_MM. The desired tolerance in the recursive adaptive Simpson quadrature was set as 10⁻⁵ for both the integration to find P_MM and to find the volume under the surface. The area of the domain of φ(P_MB, P_MN) was likewise obtained by numerically integrating the area between the curves given by (35) and (36).

In Method 2, we used a Monte-Carlo method to compute the NVUS. For given values of the parameters σ₁, σ₂, and r, N/3 observations were randomly generated from each of the three classes π_M, π_B, and π_N according to the pdfs given by (21). We then computed the likelihood ratios Λ_BM and Λ_NM using (22) and (23) for each of the N observations. For a given value of the pair (m,n), the three quantities of interest (P_MM, P_MB and P_MN) can then be determined by how many (Λ_BM, Λ_NM) pairs from each class fall into the shaded region in Fig. 1. By systematically varying m between 0 and −∞, and n between 0 and ∞, one can generate pairs (P_MB, P_MN) that cover the region between C_−∞ (P_MB) and C₀ (P_MB), and the corresponding values of the sensitivity P_MM. To find the volume under the surface and the area of the domain of the 3D ROC surface, we used Delaunay triangulation, as described in Appendix 6. Since our purpose was to compare with Method 1 rather than to evaluate the small-sample dependence of the Monte-Carlo method, we used a large number (N=300,000) for the number of observations, which was experimentally chosen as a trade-off between computational efficiency and the accuracy of the estimated volume. One hundred fifty different values were used for both m and n to generate (P_MB, P_MN) pairs that cover the domain of the 3D ROC surface.

Fig. 7 plots the NVUS results for different values of σ₁ when ${\sigma }_2^2=1$. Three plots are shown for r=−0.5, r=0, and r=0.5, for both Methods 1 and 2. Figure 8 is the same as Fig. 7, except that ${\sigma }_2^2=6$. Similar trends were observed for other values of ${\sigma }_2^2$ between. 1 and 9. Methods 1 and 2 result in very similar NVUS values. For fixed r and σ₂, NVUS increases with σ₁. Also, comparing Figs. 7 and 8, it is seen that for fixed r and σ₁, NVUS increases with σ₂. For fixed values of σ₁ and σ₂, the dependence of NVUS values on r is shown in Fig. 9.

Fig. 7.

Comparison of the NVUS values obtained by the direct integration and Monte-Carlo methods for ${\sigma }_2^2=1$. The NVUS values are plotted as a function of ${\sigma }_1^2$ for three values of r (r=−0.5, 0, and 0.5).

Fig. 8.

Comparison of the NVUS values obtained by the direct integration and Monte-Carlo methods for ${\sigma }_2^2=6$. The NVUS values are plotted as a function of ${\sigma }_1^2$ for three values of r (r=−0.5, 0, and 0.5).

Fig. 9.

NVUS as a function of r for different combinations of values for ${\sigma }_1^2\text{and}{\sigma }_2^2$.

IV. DISCUSSION

From the results presented in Figs. 7–9, it can be observed that the trends for the NVUS are consistent with the characteristics of a proper FOM, namely, as the difficulty of the classification problem decreases, the value of the FOM for the optimal classifier increases. First, NVUS increases with σ₁ for fixed r and σ₂. From the definitions of these parameters, if σ₁ increases while the other parameters are fixed, the difference in the magnitude between the mean vectors of the malignant and benign classes increases. If the malignant and benign classes are farther apart, then the classification problem is easier and the value of the FOM should increase. Similarly, if σ₂ increases while the other parameters are fixed, the value of the FOM should increase, which is the trend observed by comparing Figs. 7 and 8. Finally, if r increases while σ₁ and σ₂ are fixed, the angle between the vectors μ⃗_B and μ⃗_N decreases, and the normal and benign classes become more similar to each other. Therefore, if all the other parameters are fixed while r increases, the portion of the malignant class that overlaps with the union of the normal and benign class distributions decreases. For large r, if the classifier works well for separating the normal and malignant classes, then it will have a similar performance in separating the benign and malignant classes as well. In contrast, if the angle between the vectors μ⃗_B and μ⃗_N is large (small r), then the benign and normal classes are quite different, hence a classifier that works well to differentiate the normal and malignant classes may not work well to differentiate the benign and malignant classes. In summary, a larger r increases the separation between the malignant class and the other two classes, while it decreases the separation between the benign and normal classes. Due to our choice of the utilities, the separation between the benign and normal classes has no effect on the expected utility (9), and the value of the NVUS is determined by the separation between the malignant and the other two classes. We therefore expect that the value of the FOM would usually increase with r, as demonstrated by the trend for the NVUS in Fig. 9.

With the utilities given as in (8), the decision boundary between the benign and normal classes is indeterminate. The expected utility in (9) does not contain terms related to the misclassification probabilities P_BN and P_NB, therefore, the decision between classes B and N does not affect the task performance. However, the expected utility, and thus the task performance, is uniquely determined by P_MM, P_MB and P_MN. Furthermore, since U_MB and U_MN are not necessarily equal, there is still a distinction between the benign and normal classes, and the problem remains a three-class classification task.

As in 2D ROC analysis, comparison of 3D ROC surfaces and the summary measures for two classification tasks requires caution. Similar to the condition that two 2D ROC curves may intersect, the 3D ROC surfaces for the ideal observer for two classification tasks may also intersect. Under this condition, one does not know whether the sensitivity for the task with the larger NVUS value is higher than that with the smaller NVUS value at all FPF values. An additional need for caution for the comparison of the NVUS values of two 3D ROC surfaces results from the possibility that the FPF domains for the two tasks may be different, as shown in Fig. 5 for six different tasks. Consider two tasks T₁ and T₂, and assume that the domain of the 3D ROC surface for T₁ is larger than and encloses the entire domain for T₂. It is conceivable that the sensitivity of the ideal observer may be higher for T₁ than T₂ at all FPF values in the domain of T₂, and despite this, the NVUS value for T₁ can be lower if the sensitivity for T₁ at FPF values outside the domain of T₂ is very low.

In our previous [7, 12] and current studies, our assumptions for the utilities were that U_MM = U_BB = U_NN = U_BN = U_NB = 1, U_BM = U_NM = 0, and that U_MB and U_MN are variable in [0,1]. These assumptions simplify the formulations of the slope and intercept of the decision line, (Fig. 1 and (11) and (12)) and facilitate the subsequent analyses. However, the conditions on the utilities can be more general, namely U_BM = U_NM, U_BB = U_NB, and U_BN = U_NN (8), and the three decision lines of the ideal observer will still degenerate into a single line that separates the malignant class from the other two classes. With the additional assumption that the utility of an incorrect decision is not larger than that of a correct decision (i.e, U_BM ≤ U_MM, U_MB ≤ U_BB, and U_MN ≤ U_NN), the slope and intercept for this line will satisfy −∞ ≤ m ≤ 0 and 0 ≤ n ≤ ∞, respectively. Since the properties of the 3D ROC surface found in this study depend only on the situation of a single decision line separating the malignant class from the other two classes in the likelihood ratio plane, the conclusions of our study is applicable to the more general condition given by (8).

V. CONCLUSION

In this study, we investigated the 3-class classification problem under some simplifying, but realistic, assumptions for the utilities that result in a single decision line in the likelihood ratio plane separating the malignant decisions from “non-malignant” decisions. Under these assumptions (8), we showed that the sensitivity of the ideal observer can be expressed as a function of two false-positive fractions, thereby one can define a 3D ROC surface above the FPF plane, and that the domain of the ROC surface in the FPF plane depends on the class distributions. We extended the definition of the area under the ROC curve for a two-class problem to a normalized volume under the ROC surface for a three-class problem, where the normalization is required because of the dependence of the domain of the 3D ROC surface on the classification task.

We studied the properties of the 3D ROC surface and the NVUS for the ideal observer under the condition that the three-class distributions in the feature space are multivariate normal with equal covariance matrices. Under this assumption for class distributions, it can be shown that the probability density functions of the likelihood ratios follow a bivariate log-normal distribution, and that they depend only on three parameters. We investigated two methods for calculating the NVUS, and found that the results were in good agreement. We also found that the NVUS satisfies some of the intuitive properties of a proper FOM for characterizing the classification performance. Although the analysis in this study was performed under simplifying assumptions, we expect that it will provide guidance for the challenging problem of designing performance metric and analysis for 3-class classification tasks.

APPENDIX 1

We first show that

$$P_{\mathit{\text{MB}}}={\displaystyle \underset{S}{\iint }{\mathrm{\Lambda }}_{\text{BM}}{p}_{{\mathbf{\Lambda }}_{\mathbf{\text{BM}}},{\mathbf{\Lambda }}_{\mathbf{\text{NM}}}|{\mathit{\pi }}_{\mathbf{M}}}}({\mathrm{\Lambda }}_{\text{BM}},{\mathrm{\Lambda }}_{\text{NM}}|{\pi }_{\mathrm{M}})d{\mathrm{\Lambda }}_{\text{BM}}d{\mathrm{\Lambda }}_{\text{NM}}$$ (A.1)

and

$$P_{\mathit{\text{MN}}}={\displaystyle \underset{S}{\iint }{\mathrm{\Lambda }}_{\text{NM}}{p}_{{\mathbf{\Lambda }}_{\mathbf{\text{BM}}},{\mathbf{\Lambda }}_{\mathbf{\text{NM}}}|{\mathit{\pi }}_{\mathbf{M}}}}({\mathrm{\Lambda }}_{\text{BM}},{\mathrm{\Lambda }}_{\text{NM}}|{\pi }_{\mathrm{M}})d{\mathrm{\Lambda }}_{\text{BM}}d{\mathrm{\Lambda }}_{\text{NM}}.$$ (A.2)

Let Λ_BM = f₁ (w⃗), Λ_NM = f₂ (w⃗), and let Ω(m, n) denote the set of vectors w⃗ that satisfy f₂(w⃗) − mf₁ (w⃗) − n < 0. Noting that p_w⃗|πB (w⃗ | π_B) = Λ_BM (w⃗)p_w⃗|πM, the false positive probability for the benign class can be written as the D-dimensional integral

$$P_{\mathit{\text{MB}}}={\displaystyle \underset{\mathrm{\Omega }(m,n)}{\int }{p}_{\overrightarrow{\mathbf{w}}|{\mathit{\pi }}_{\mathbf{B}}}}(\overrightarrow{\mathrm{w}}|{\pi }_{\mathrm{B}})d\overrightarrow{w}={\displaystyle \underset{\mathrm{\Omega }(m,n)}{\int }{\mathrm{\Lambda }}_{\mathit{\text{BM}}}(\overrightarrow{w})p_{\overrightarrow{\mathbf{w}}|{\mathit{\pi }}_{\mathbf{M}}}}(\overrightarrow{w}|{\pi }_{\mathrm{M}})d\overrightarrow{w}$$ (A.3)

The integrand in (A.3) can be integrated by first finding the set of all vectors w⃗ in the D-dimensional observation space such that

$${\mathrm{\Lambda }}_{\mathit{\text{BM}}}<f_1(\overrightarrow{w})\le {\mathrm{\Lambda }}_{\mathit{\text{BM}}}+\mathrm{\Delta }{\mathrm{\Lambda }}_{\mathit{\text{BM}}},\text{and}{\mathrm{\Lambda }}_{\mathit{\text{NM}}}<f_2(\overrightarrow{w})\le {\mathrm{\Lambda }}_{\mathit{\text{NM}}}+\mathrm{\Delta }{\mathrm{\Lambda }}_{\mathit{\text{NM}}}.$$ (A.4)

The sum of all p_w⃗|πM (w⃗ | π_M) such that (A.4) is satisfied is (by definition) p_{ΛBM,ΛNM|πM} (Λ_BM, Λ_NM | π_M)ΔΛ_BMΔΛ_NM, while w⃗ ∈ Ω(m, n) is equivalent to Λ_NM − mΛ_BM − n < 0. Integrating over the entire D-dimensional observation space, we find that

$${\displaystyle \underset{\mathrm{\Omega }(m,n)}{\int }{\mathrm{\Lambda }}_{\mathit{\text{BM}}}(\overrightarrow{w})p_{\overrightarrow{\mathbf{w}}|{\mathit{\pi }}_{\mathbf{M}}}(\overrightarrow{w}|{\pi }_M)d\overrightarrow{w}}={\displaystyle \underset{{\mathrm{\Lambda }}_{\mathit{\text{NM}}}-m{\mathrm{\Lambda }}_{\mathit{\text{BM}}}-n<0}{\iint }{\mathrm{\Lambda }}_{\mathit{\text{BM}}}{p}_{{\mathbf{\Lambda }}_{\mathbf{\text{BM}}},{\mathbf{\Lambda }}_{\mathbf{\text{NM}}}|{\mathit{\pi }}_{\mathbf{M}}}}({\mathrm{\Lambda }}_{\mathit{\text{BM}}},{\mathrm{\Lambda }}_{\mathit{\text{NM}}}|{\pi }_{\mathrm{M}})d{\mathrm{\Lambda }}_{\mathit{\text{BM}}}d{\mathrm{\Lambda }}_{\mathit{\text{NM}}}={\displaystyle \underset{S}{\iint }{\mathrm{\Lambda }}_{\mathit{\text{BM}}}{p}_{{\mathbf{\Lambda }}_{\mathbf{\text{BM}}},{\mathbf{\Lambda }}_{\mathbf{\text{NM}}}|{\mathit{\pi }}_{\mathbf{M}}}}({\mathrm{\Lambda }}_{\mathit{\text{BM}}},{\mathrm{\Lambda }}_{\mathit{\text{NM}}}|{\pi }_{\mathrm{M}})d{\mathrm{\Lambda }}_{\mathit{\text{BM}}}d{\mathrm{\Lambda }}_{\mathit{\text{NM}}}$$ (A.5)

Eq. (A.2) can be obtained in a similar manner.

We now show if P_MB (m₀, n₀) = P_MB (m₁, n₁) and m₁ ≥ m₀, then P_MN (m₀, n₀) ≥ P_MN (m₁, n₁).

For notational simplicity, let us define x = Λ_BM, y = Λ_NM, and f(x, y) = p_{ΛBM,ΛNM|πM} (Λ_BM, Λ_NM | π_M). First consider Case (A), in which the two lines y = m₀x + n₀ and y = m₁x + n₁ (i.e., the lines Λ_NM = m₀Λ_BM + n₀ and Λ_NM = m₁Λ_BM + n₁) do not intersect in the positive quadrant (Fig. A1). Using (A.1), and noting that f(x, y) is non-negative, P_MB (m₀, n₀) = P_MB (m₁, n₁) implies f(x, y) = 0 in the region D shown in Fig. A1. Using (A.2), this implies P_MN (m₀, n₀) = P_MN (m₁, n₁).

Fig. A1.

Case (A) for Property 1: The lines Λ_NM = m₀Λ_BM + n₀ and Λ_NM = m₁Λ_BM + n₁ do not intersect in the positive quadrant. The difference between P_MB (m₀, n₀) and P_MB (m₁, n₁) is the integral of the joint pdf of the likelihood ratios under class B in the shaded region D.

We now consider Case (B), in which the two lines y = m₀x + n₀ and y = m₁x + n₁ intersect at a point (x^*, y^*) in the positive quadrant (Fig. A2). Using (A.1), it is seen that

$$P_{\mathit{\text{MB}}}(m_0,n_0)=P_{\mathit{\text{MB}}}(m_1,n_1)\Rightarrow {\displaystyle \underset{S_0}{\iint }\mathit{\text{xf}}(x,y)\mathit{\text{dxdy}}=}{\displaystyle \underset{S_1}{\iint }\mathit{\text{xf}}(x,y)\mathit{\text{dxdy}},}$$ (A.6)

where the two regions S₀ and S₁ are the two shaded triangles in Fig. A2. For any x in S₀, we have x^* ≥ x, and for any x in S₁, we have x ≥ x^*. Therefore,

$$x^{*}{\displaystyle \underset{S_0}{\iint }f(x,y)\mathit{\text{dxdy}}\ge }{\displaystyle \underset{S_0}{\iint }\mathit{\text{xf}}(x,y)\mathit{\text{dxdy}}}={\displaystyle \underset{S_1}{\iint }\mathit{\text{xf}}(x,y)\mathit{\text{dxdy}}}\ge x^{*}{\displaystyle \underset{S_1}{\iint }f(x,y)\mathit{\text{dxdy}}},$$ (A.7)

which indicates

$$P_{\mathit{\text{MB}}}(m_0,n_0)=P_{\mathit{\text{MB}}}(m_1,n_1)\Rightarrow {\displaystyle \underset{S_0}{\iint }f(x,y)\mathit{\text{dxdy}}\ge }{\displaystyle \underset{S_1}{\iint }f(x,y)\mathit{\text{dxdy}}}.$$ (A.8)

Fig. A2.

Case (B) for Property 1: The lines Λ_NM = m₀Λ_BM + n₀ and Λ_NM = m₁Λ_BM + n₁ intersect at a point (x^*, y^*) in the positive quadrant. The difference between P_MB (m₀, n₀) and P_MB (m₁, n₁) is the difference of the integral of the joint pdf of the likelihood ratios under class B in the shaded regions S₀ and S₁.

Similarly, for any y in S₀, we have y ≥ y^*, and for any y in S₁, we have y^* ≥ y. Therefore,

$${\displaystyle \underset{S_0}{\iint }\mathit{\text{yf}}(x,y)\mathit{\text{dxdy}}\ge }{y}^{*}{\displaystyle \underset{S_0}{\iint }f(x,y)\mathit{\text{dxdy}}}\ge y^{*}{\displaystyle \underset{S_1}{\iint }f(x,y)\mathit{\text{dxdy}}}\ge {\displaystyle \underset{S_1}{\iint }\mathit{\text{yf}}(x,y)\mathit{\text{dxdy}}},$$ (A.9)

where the middle inequality is due to (A.8). We therefore conclude that

$${\displaystyle \underset{S_0}{\iint }\mathit{\text{yf}}(x,y)\mathit{\text{dxdy}}\ge }{\displaystyle \underset{S_1}{\iint }\mathit{\text{yf}}(x,y)\mathit{\text{dxdy}}},$$ (A.10)

which, along with (A.2), implies that P_MN (m₀, n₀) ≥ P_MN (m₁, n₁). We also note that P_MN (m₀, n₀) = P_MN (m₁, n₁) if and only if f(x, y) = 0 in S₀ ∪ S₁\(x^*, y^*).

APPENDIX 2

In this appendix, we show that a given (P_MB, P_MN) pair uniquely determines P_MM, i.e., we cannot have more than one sensitivity value for a given FPF pair. Let (m₀, n₀) and (m₁, n₁) be such that P_MB (m₀, n₀) = P_MB (m₁, n₁) and P_MN (m₀, n₀) = P_MN (m₁, n₁). Proceeding in a similar fashion to the proof of Property 1, we see that if the lines y = m₀x + n₀ and y = m₁x + n₁ do not intersect in the positive quadrant, or if m₁ = m₀, then P_MM (m₀, n₀) = P_MM (m₁, n₁). If these two lines intersect at one point in the positive quadrant, then, without loss of generality, let us assume m₁ > m₀ so that we can refer to Fig. A2. We have already shown (A.8) that if m₁ > m₀, then

$$P_{\mathit{\text{MB}}}(m_0,n_0)=P_{\mathit{\text{MB}}}(m_1,n_1)\Rightarrow {\displaystyle \underset{S_0}{\iint }f(x,y)\mathit{\text{dxdy}}\ge }{\displaystyle \underset{S_1}{\iint }f(x,y)\mathit{\text{dxdy}}}\Rightarrow P_{\mathit{\text{MM}}}(m_0,n_0)\ge P_{\mathit{\text{MM}}}(m_1,n_1)$$ (A.11)

Referring to Fig. A2, for any y in S₀, we have y ≥ y^*, and for any y in S₁, we have y^* ≥ y. Therefore,

$$P_{\mathit{\text{MN}}}(m_0,n_0)=P_{\mathit{\text{MN}}}(m_1,n_1)\Rightarrow {\displaystyle \underset{S_0}{\iint }\mathit{\text{yf}}(x,y)\mathit{\text{dxdy}}=}{\displaystyle \underset{S_1}{\iint }\mathit{\text{yf}}(x,y)\mathit{\text{dxdy}}}\Rightarrow {\displaystyle \underset{S_0}{\iint }{y}^{*}f(x,y)\mathit{\text{dxdy}}\le }{\displaystyle \underset{S_1}{\iint }{y}^{*}f(x,y)\mathit{\text{dxdy}}}\Rightarrow {\displaystyle \underset{S_0}{\iint }f(x,y)\mathit{\text{dxdy}}\le }{\displaystyle \underset{S_1}{\iint }f(x,y)\mathit{\text{dxdy}}}$$ (A.12)

from which we conclude

$$P_{\mathit{\text{MN}}}(m_0,n_0)=P_{\mathit{\text{MN}}}(m_1,n_1)\Rightarrow P_{\mathit{\text{MM}}}(m_0,n_0)\le P_{\mathit{\text{MM}}}(m_1,n_1)$$ (A.13)

Equations (A.11) and (A.13) imply that P_MM (m₀, n₀) = P_MM (m₁, n₁).

APPENDIX 3

We will only prove the first part, i.e., if P_MB (m₀, n₀) = P_MB (m₁, n₁) and P_MN (m₀, n₀) > P_MN (m₁, n₁), then P_MM (m₀, n₀) ≥ P_MM (m₁, n₁). The proof of the second part follows similar steps. The assumptions about P_MB and P_MN imply that m₁ > m₀. This is because if m₀ were larger than or equal to m₁, then, by Property 1, we would have P_MN (m₁, n₁) ≥ P_MN (m₀, n₀). We also note that the lines Λ_NM = m₀Λ_BM + n₀ and Λ_NM = m₁Λ_BM + n₁ intersect in the positive quadrant, because otherwise we would have P_MN (m₀, n₀) = P_MN (m₁, n₁). Under these conditions, we have already shown that (A.8)

$${\displaystyle \underset{S_0}{\iint }{p}_{{\mathbf{\Lambda }}_{\mathbf{\text{BM}}},{\mathbf{\Lambda }}_{\mathbf{\text{NM}}}|{\mathit{\pi }}_{\mathbf{M}}}({\mathrm{\Lambda }}_{\text{BM}},{\mathrm{\Lambda }}_{\text{NM}}|{\pi }_{\mathrm{M}})d{\mathrm{\Lambda }}_{\text{BM}}d{\mathrm{\Lambda }}_{\text{NM}}\ge {\displaystyle \underset{S_1}{\iint }{p}_{{\mathbf{\Lambda }}_{\mathbf{\text{BM}}},{\mathbf{\Lambda }}_{\mathbf{\text{NM}}}|{\mathit{\pi }}_{\mathbf{M}}}({\mathrm{\Lambda }}_{\text{BM}},{\mathrm{\Lambda }}_{\text{NM}}|{\pi }_{\mathrm{M}})d{\mathrm{\Lambda }}_{\text{BM}}d{\mathrm{\Lambda }}_{\text{NM}}},}$$

which implies that P_MM (m₀, n₀) ≥ P_MM (m₁, n₁).

APPENDIX 4

In this appendix, we show that, for the equal-covariance Gaussian distributions discussed in Section 2.E., diagonalization and translation do not affect the 3D ROC surface. This is equivalent to proving that a straight line in the original likelihood ratio plane remains a straight line after diagonalization. Let w⃗′ denote a random vector to be classified before the diagonalization, and let

$$\begin{array}{c}\overrightarrow{\mathbf{w}}\prime ~\mathrm{N}({\overrightarrow{\mu }}_M^{\prime },\mathrm{\Sigma }\prime )\text{under}{\pi }_M\hfill \\ \overrightarrow{\mathbf{w}}\prime ~\mathrm{N}({\overrightarrow{\mu }}_B^{\prime },\mathrm{\Sigma }\prime )\text{under}{\pi }_B\hfill \\ \overrightarrow{\mathbf{w}}\prime ~\mathrm{N}({\overrightarrow{\mu }}_N^{\prime },\mathrm{\Sigma }\prime )\text{under}{\pi }_N\hfill \end{array}$$ (A.14)

The likelihood ratio between π_B and π_M before the diagonalization and translation is given by

$${\mathbf{\Lambda }}_{\mathbf{\text{BM}}}^{\prime }(\overrightarrow{\mathbf{w}}\prime )=\frac{p_{\overrightarrow{\mathbf{w}}\prime |{\pi }_B}(\overrightarrow{w}\prime |{\pi }_B)}{p_{\overrightarrow{\mathbf{w}}\prime |{\pi }_M}(\overrightarrow{w}\prime |{\pi }_M)}=\text{exp}\left\{-\frac{1}{2}\left[{(\overrightarrow{\mathbf{w}}\prime -{\overrightarrow{\mu }}_B^{\prime })}^T{\mathrm{\Sigma }\prime }^{-1}(\overrightarrow{\mathbf{w}}\prime -{\overrightarrow{\mu }}_B^{\prime })-{(\overrightarrow{\mathbf{w}}\prime -{\overrightarrow{\mu }}_M^{\prime })}^T{\mathrm{\Sigma }\prime }^{-1}(\overrightarrow{\mathbf{w}}\prime -{\overrightarrow{\mu }}_M^{\prime })\right]\right\}=\text{exp}\left\{-\frac{1}{2}\left({\overrightarrow{\mu }}_B^{\prime T}{\mathrm{\Sigma }\prime }^{-1}{\overrightarrow{\mu }}_B^{\prime T}-{{\overrightarrow{\mu }}_M^{\prime }}^T{\mathrm{\Sigma }\prime }^{-1}{{\overrightarrow{\mu }}_M^{\prime }}^T\right)\right\}\text{exp}\left\{{\left({\overrightarrow{\mu }}_B^{\prime }-{\overrightarrow{\mu }}_M^{\prime }\right)}^T{\mathrm{\Sigma }\prime }^{-1}\overrightarrow{\mathbf{w}}\prime \right\}$$ (A.15)

We will show that the likelihood ratio between π_B and π_M after the diagonalization and translation, Λ_BM (w⃗), is proportional to ${\mathbf{\Lambda }}_{\mathbf{\text{BM}}}^{\prime }(\overrightarrow{\mathbf{w}}\prime )$. The transformation for diagonalization and translation between w⃗ and w⃗′ is

$$\overrightarrow{\mathbf{w}}={\mathrm{L}}^{-1/2}{V}^T(\overrightarrow{\mathbf{w}}\prime -{\overrightarrow{\mu }}_M^{\prime }),$$ (A.16)

where V is the matrix composed of the eigenvectors of Σ′, and L is the diagonal matrix constructed from the corresponding eigenvalues. The covariance matrix after the diagonalization is the identity matrix, and the class means are

$${\overrightarrow{\mu }}_M=0,{\overrightarrow{\mu }}_B={\mathrm{L}}^{-1/2}{V}^T({\overrightarrow{\mu }}_B^{\prime }-{\overrightarrow{\mu }}_M^{\prime }),\text{\hspace{1em}and\hspace{1em}}{\overrightarrow{\mu }}_N={\mathrm{L}}^{-1/2}{V}^T({\overrightarrow{\mu }}_N^{\prime }-{\overrightarrow{\mu }}_M^{\prime })$$ (A.17)

The likelihood ratio between π_B and π_M after the diagonalization and translation is

$$\begin{array}{c}{\mathbf{\Lambda }}_{\mathbf{\text{BM}}}(\overrightarrow{\mathbf{w}})=\frac{p_{\overrightarrow{\mathbf{w}}|{\pi }_B}(\overrightarrow{w}|{\pi }_B)}{p_{\overrightarrow{\mathbf{w}}|{\pi }_M}(\overrightarrow{w}|{\pi }_M)}\hfill \\ =\text{exp}\left\{-\frac{1}{2}\left[{(\overrightarrow{\mathbf{w}}-{\overrightarrow{\mu }}_B)}^T(\overrightarrow{\mathbf{w}}-{\overrightarrow{\mu }}_B)-{\overrightarrow{\mathbf{w}}}^T\overrightarrow{\mathbf{w}}\right]\right\}\hfill \\ =\text{exp}\left\{{{\overrightarrow{\mu }}_B}^T\overrightarrow{\mathbf{w}}-\frac{1}{2}{{\overrightarrow{\mu }}_B}^T{\overrightarrow{\mu }}_B\right\}\hfill \\ =\text{exp}\left\{-\frac{1}{2}{{\overrightarrow{\mu }}_B}^T{\overrightarrow{\mu }}_B\right\}\text{exp}\left\{{{\overrightarrow{\mu }}_B}^T\overrightarrow{\mathbf{w}}\right\}\hfill \\ =\text{exp}\left\{-\frac{1}{2}{{\overrightarrow{\mu }}_B}^T{\overrightarrow{\mu }}_B\right\}\text{exp}\left\{{{\overrightarrow{\mu }}_B}^T{\mathrm{L}}^{-1/2}{V}^T(\overrightarrow{\mathbf{w}}\prime -{\overrightarrow{\mu }}_M^{\prime })\right\}\hfill \\ =\text{exp}\left\{-\frac{1}{2}{{\overrightarrow{\mu }}_B}^T{\overrightarrow{\mu }}_B-{{\overrightarrow{\mu }}_B}^T{\mathrm{L}}^{-1/2}{V}^T{\overrightarrow{\mu }}_M^{\prime }\right\}·\text{exp}\left\{{({\overrightarrow{\mu }}_B^{\prime }-{\overrightarrow{\mu }}_M^{\prime })}^{T}V{\mathrm{L}}^{-1/2}·{\mathrm{L}}^{-1/2}{V}^T\overrightarrow{\mathbf{w}}\prime \right\}\hfill \\ =\text{exp}\left\{-\frac{1}{2}{{\overrightarrow{\mu }}_B}^T{\overrightarrow{\mu }}_B-{{\overrightarrow{\mu }}_B}^T{\mathrm{L}}^{-1/2}{V}^T{\overrightarrow{\mu }}_M^{\prime }\right\}·\text{exp}\left\{{({\overrightarrow{\mu }}_B^{\prime }-{\overrightarrow{\mu }}_M^{\prime })}^T\mathrm{\Sigma }{\prime }^{-1}\overrightarrow{\mathbf{w}}\prime \right\}\hfill \end{array}$$ (A.18)

Comparing to (A.15), we find that ${\mathbf{\Lambda }}_{\mathbf{\text{BM}}}(\overrightarrow{\mathbf{w}})={\mathrm{K}}_{\mathrm{B}}{\mathbf{\Lambda }}_{\mathbf{\text{BM}}}^{\prime }(\overrightarrow{\mathbf{w}}\prime )$, where K_B is a constant of proportionality. Similarly, it can be shown that ${\mathbf{\Lambda }}_{\mathbf{\text{NM}}}(\overrightarrow{\mathbf{w}})={\mathrm{K}}_{\mathrm{N}}{\mathbf{\Lambda }}_{\mathbf{\text{NM}}}^{\prime }(\overrightarrow{\mathbf{w}}\prime )$, and therefore a straight line in the original likelihood ratio plane remains a straight line after diagonalization.

APPENDIX 5

We will demonstrate the derivation of (36) here, i.e., the relationship between P_MB and P_MN for m=0. The integrals can be evaluated in a similar manner to find the relationship between P_MB and P_MN for m = −∞ (35). First, for m=0, and with a change of variables

$$\begin{array}{c}u=\text{log}{\mathrm{\Lambda }}_{\mathit{\text{BM}}}-{\overline{z}}_B(1)\hfill \\ v=\text{log}{\mathrm{\Lambda }}_{\mathit{\text{NM}}}-{\overline{z}}_B(2),\hfill \end{array}$$ (A.19)

Eq. (33) can be written as

$$P_{\mathit{\text{MB}}}={\displaystyle \underset{v=-\infty }{\overset{\text{log}n-{\overline{z}}_B(2)}{\int }}{\displaystyle \underset{-\infty }{\overset{\infty }{\int }}\frac{\mathit{\text{dudv}}}{2\pi {\sigma }_1{\sigma }_2\sqrt{1-r^2}}G(u,v)}},$$ (A.20)

where

$$\begin{array}{cc}G(u,v)\hfill & =\text{exp}\left\{\frac{-1}{2(1-r^2)}\left[\frac{u^2}{{\sigma }_1^2}-\frac{2\mathit{\text{ruv}}}{{\sigma }_1{\sigma }_2}+\frac{v^2}{{\sigma }_2^2}\right]\right\}\hfill \\ \hfill & =\text{exp}\left\{\frac{-1}{2(1-r^2)}{\left[\frac{u}{{\sigma }_1}-\frac{\mathit{\text{rv}}}{{\sigma }_2}\right]}^2-\frac{v^2}{2{\sigma }_2^2}\right\}.\hfill \end{array}$$ (A.21)

Therefore,

$$\begin{array}{cc}{P}_{\mathit{\text{MB}}}\hfill & =\frac{1}{2\pi {\sigma }_1{\sigma }_2\sqrt{1-r^2}}{\displaystyle \underset{v=-\infty }{\overset{\text{log}n-{\overline{z}}_B(2)}{\int }}\mathit{\text{dv}}\text{exp}(-v^2/2{\sigma }_2^2)}·{\displaystyle \underset{-\infty }{\overset{\infty }{\int }}\text{exp}\left\{\frac{-1}{2(1-r^2)}{\left[\frac{u}{{\sigma }_1}-\frac{\mathit{\text{rv}}}{{\sigma }_2}\right]}^2\right\}\mathit{\text{du}}}\hfill \\ \hfill & =\frac{1}{\sqrt{2\pi }{\sigma }_2}{\displaystyle \underset{v=-\infty }{\overset{\text{log}n-{\overline{z}}_B(2)}{\int }}\mathit{\text{dv}}\text{exp}(-v^2/2{\sigma }_2^2)}\hfill \\ \hfill & =\mathrm{\Phi }\left[\frac{\text{log}n-{\overline{z}}_B(2)}{{\sigma }_2}\right]\hfill \end{array}$$ (A.22)

Similarly,

$$P_{\mathit{\text{MN}}}=\mathrm{\Phi }\left[\frac{\text{log}n-{\overline{z}}_N(2)}{{\sigma }_2}\right]$$ (A.23)

Solving for log n in terms of P_MB in (A.22) and substituting into (A.23), we obtain (36).

APPENDIX 6

Delaunay triangulation is closely related to the Voronoi diagram for a set of Q points in the plane. Given Q points P_q, q = 1,…Q, one can partition the plane into Q partitions S_q, q = 1,…Q, where S_q^* is the region of the plane that is closer to the point P_q^* than to any of the other Q-1 points. S_q^* is called the Voronoi domain of the point q^*. If one draws a line between any two points whose Voronoi domains are adjacent to each other, a set of triangles is obtained, known as the Delaunay triangulation. The Delaunay triangulation has the property that the circumcircle of no triangle contains any points of the triangulation. Figure A3 shows a Delaunay triangulation for Q=9 points. The Voronoi cell S₇ for the point P₇ is shown as the shaded polygon. When the domain of the 3D ROC is convex, the area of the domain was approximated by the sum of all such triangles. When the domain was not convex, we excluded from the computation of the area any triangle whose interior contained any point outside the region bounded by C_−∞ (P_MB) and C₀ (P_MB), obtained empirically using the Monte Carlo simulation. The volume under the surface was approximated by summing up the volume of each prism whose base was given by a Delaunay triangle used in the area estimation.

Fig. A3.

Delaunay triangulation for Q=9 points in the plane. The Voronoi cell S₇ for the point P₇ is shown as the shaded polygon.