Optimizing area under the ROC curve using semi-supervised learning

Abstract

Receiver operating characteristic (ROC) analysis is a standard methodology to evaluate the performance of a binary classification system. The area under the ROC curve (AUC) is a performance metric that summarizes how well a classifier separates two classes. Traditional AUC optimization techniques are supervised learning methods that utilize only labeled data (i.e., the true class is known for all data) to train the classifiers. In this work, inspired by semi-supervised and transductive learning, we propose two new AUC optimization algorithms hereby referred to as semi-supervised learning receiver operating characteristic (SSLROC) algorithms, which utilize unlabeled test samples in classifier training to maximize AUC. Unlabeled samples are incorporated into the AUC optimization process, and their ranking relationships to labeled positive and negative training samples are considered as optimization constraints. The introduced test samples will cause the learned decision boundary in a multidimensional feature space to adapt not only to the distribution of labeled training data, but also to the distribution of unlabeled test data. We formulate the semi-supervised AUC optimization problem as a semi-definite programming problem based on the margin maximization theory. The proposed methods SSLROC1 (1-norm) and SSLROC2 (2-norm) were evaluated using 34 (determined by power analysis) randomly selected datasets from the University of California, Irvine machine learning repository. Wilcoxon signed rank tests showed that the proposed methods achieved significant improvement compared with state-of-the-art methods. The proposed methods were also applied to a CT colonography dataset for colonic polyp classification and showed promising results.¹

1. Introduction

Receiver operating characteristic (ROC) analysis is a standard methodology to evaluate the performance of a classification system [1–12]. It is applied extensively within clinical medicine [13–15]. The ROC curve is a two-dimensional plot which illustrates the relationship between the true positive rate (sensitivity) and the false positive rate (1 – specificity) of a binary classifier. In essence, a classifier seeks the optimal mapping of samples from a multi-dimensional feature space to a one-dimensional decision space during the training process. After the training process, the classifier can be applied to test samples whose labels are unknown and make a prediction for each test sample. The value of the prediction should be numerical (not binary categories) in order to make ROC analysis. Based on the predictions of the test set from a trained classifier, user of the classifier can select a specific diagnostic threshold to differentiate positive from negative samples for his or her specific application by finding the threshold along the ROC curve which maximizes sensitivity at the highest acceptable false positive rate (or cost).

The area under the ROC curve (AUC) is a univariate description of the ROC curve [1]. It ranges from 0.5 to 1, with larger values representing higher system performance. The AUC is equal to the probability that the decision value assigned to a randomly-drawn positive sample is greater than the value assigned to a randomly-drawn negative sample. Flach et al. proved that AUC is coherent and linearly related to expected loss [12]. The AUC statistic is commonly used to compare different classification systems. Previous studies have shown that AUC is statistically consistent and a more discriminative measure than classification accuracy [3,4].

Although some researchers have recommended the use of AUC for the evaluation of machine learning algorithms when a single performance metric needs to be used for the evaluation [1], others have pointed out some shortcomings of the use of the AUC. Lobo et al. cited a number of limitations of the use of AUC in evaluating the performance of species distribution (presence–absence) models [16] in ecology. Among the more general limitations are that the AUC summarizes performance over regions of the ROC space in which one would rarely operate, and that the goodness-of-fit of a model is ignored by the AUC. Hanczar et al. studied the problem of comparing estimates of AUC, true positive rate (TPR) and false positive rate (FPR) with true metrics when classifier training and performance estimation are performed on small-sample datasets [17]. They found that generally there is weak regression of the true metric on the estimated metric for all three figures of merit (AUC, TPR and FPR) studied. Clearly, AUC needs to be carefully considered as an endpoint in both classifier evaluation and classifier design. However, when a single figure of merit needs to be used for classifier design, and the operating point of the classifier (a specific desired FPR or TPR) is not defined a priori, AUC remains a strong alternative to other figures of merit. AUC continues to be a very widely used endpoint in classifier evaluation and design, and many approaches to classifier design only indirectly maximize the AUC by optimizing some other cost functions, such as classification accuracy [18]. Our study does not try to define the scenarios for which AUC is an appropriate metric, but to instead discuss and compare approaches for optimizing AUC when it is deemed appropriate. Direct optimization of the AUC for a binary classifier is an interesting problem that may lead to improved performance for such applications.

In previous work, Rakotomamonjy first showed that support vector machines (SVMs) can maximize both AUC and accuracy [5]. He proposed a quadratic programming-based algorithm for AUC maximization by considering the margins between positive and negative training samples. Hereafter, we will refer this method as “SVMROC”. Subsequently, Brefeld and Scheffer presented a rigorous derivation of an AUC-maximizing SVM by imposing a convex bound and a margin item to the optimization problem [6]. They not only gave a strict analytical solution to the AUC-maximizing problem, but also showed an approximate solution based on clustering the constraints for large datasets.

Learning by an ensemble of classifiers is a very effective learning mechanism and a mainstream scheme used in machine learning [19,20]. Ensemble learning refers to a collection of methods that learn a target function by training a number of individual learners and combining their predictions together. Bagging [21] and boosting [22] are two of the best-known ensemble learning methods. Inspired by the “collaborative filtering” problem of ranking movies for a user based movie ratings from other users, Freund et al. proposed an efficient algorithm, termed RankBoost, for combining preferences based on the boosting approach [23]. RankBoost was originally designed for ranking problems. AUC optimization promotes ranks of positive training samples and decreases ranks of negative training samples, and is therefore essentially a ranking problem. RankBoost can thus be applied to AUC optimization, and has been widely used as a baseline method for this problem.

To maximize AUC for large scale and high dimensional data, Gao et al. proposed a one-pass AUC optimization technique called OPAUC [24]. The most prominent feature of this technique is that it only scans the data once as a single sequence and, therefore, does not require storage of the whole training set. OPAUC employs a square loss to measure the ranking error between two instances from different classes. A regression based algorithm was developed to calculate the first and second-order statistics of the training data and store them in memory. By this way, the storage requirement of OPAUC is only determined by the dimension of the data, not the number of instances of the data.

In recent years, semi-supervised learning (SSL) has emerged as an alternate approach to supervised learning in machine learning with advantages in many real life applications. Semi-supervised learning falls between supervised and unsupervised learning [25,26]. It utilizes both labeled data (usually a small amount), in which the true class is known, and unlabeled data (usually many), in which the data class is unknown, during the training process. Semi-supervised learning algorithms were developed primarily because the labeling of data is typically expensive, and even impossible in some applications. It is especially useful for medical problems because the acquisition of labels is very expensive and time consuming for many clinical trials. Previous studies of semi-supervised learning focused on classification and clustering problems [25,26]. For classification problems, classification accuracy is a widely-used evaluation indicator to test semi-supervised learning methods.

Traditional AUC optimization techniques are supervised learning methods, which only utilize labeled data in classifier training. Previous studies on SSL have shown that by utilizing distribution or manifold information of test samples, SSL algorithms can achieve higher classification performance compared with supervised learning algorithms. Thus, one natural idea will be to apply the mechanism of SSL to the problem of AUC optimization. In addition, SSL also has a close connection to transductive learning. Traditional supervised learning algorithms attempt the difficult task of learning general rules from training data, but transductive learning reasons from observed training data to test cases directly [27,25]. This is quite different from traditional inductive learning, which only considers functions learned from a training set and ignores statistical connection between training and test sets. In transductive learning, an unlabeled test dataset is used during classifier training in order to predict class membership for the given test dataset based on the labels of training samples. Trans-ductive learning focuses on how to transfer the knowledge gained from the training samples to the unlabeled test samples in an efficient and accurate way. The motivation behind transductive learning is also applicable to the AUC optimization problem.

As an example of transductive learning, Sindhwani and Keerthi proposed semi-supervised linear support vector classifiers (named “SVMlin”) to handle partially-labeled large scale datasets with possibly very large and sparse features [28,29]. They applied modified finite Newton techniques to linear transductive SVM which is significantly more efficient and scalable than traditional dual optimization techniques for solving quadratic programming problems.

In the literature, there is little work on applying SSL or transductive learning to AUC optimization explicitly. Amini et al. proposed a boosting algorithm (“SSRankBoost”) for learning bipartite ranking functions with partially labeled data [30]. Bipartite ranking problem refers to a ranking problem which assigns higher scores to relevant examples than to irrelevant ones for a given dataset which has wide applications in document analysis area. Along the same line, Ralaivola proposed a semi-supervised bipartite ranking algorithm with the normalized Rayleigh coefficient [31]. Later, Usunier et al. proposed a multiview semi-supervised learning algorithm for ranking multilingual documents [32]. Since AUC optimization has close relationship with ranking problem, works on learning bipartite ranking functions can also be applied to AUC optimization problems.

In this work, inspired by semi-supervised and transductive learning, we propose two new AUC optimization algorithms hereby referred to as semi-supervised learning receiver operating characteristic algorithms (SSLROC1 and SSLROC2), which utilize unlabeled test samples for classifier training. Unlabeled test samples are incorporated into the AUC optimization process, and their ranking relationships to positive and negative training samples are considered as optimization constraints. The introduced test samples make the learned decision boundary in a multi-dimensional feature space to adapt not only to the distribution of labeled training data, but also to the distribution of unlabeled test data. We formulate the semi-supervised AUC optimization problem as a semi-definite programming (SDP) problem [33] based on the margin maximization theory.

The paper is organized as follows: we first introduce the AUC optimization problem in Section 2. The AUC optimization problem is then formulated as a semi-supervised learning problem based on the margin maximization theory and solved using semi-definite programming in Section 3. In Section 4, we list 34 datasets (from University of California, Irvine machine learning repository) which are used to evaluate the proposed method, and show comparisons with state-of-the-art classification or AUC optimization methods. We also show results from the proposed method for a colonic polyp classification problem based on a biomedical imaging dataset. In Section 5 we conclude our findings and discuss computational complexity issues and future research directions.

2. Maximizing AUC with large margin learning

For a two-class classification problem, given training samples {(x₁, y₁), …, (x_n, y_n)}, y_i ∈{−1, + 1}, the optimization problem for maximizing the area under the ROC curve is defined as follows:

Optimization problem 1

$$max\mathit{AUC}=\underset{\mathit{w}}{max}\frac{{\sum }_{i=1}^{n^{+}}{\sum }_{j=1}^{n^{-}}I({\xi }_{ij}>0)}{n^{+}\times n^{-}}$$ (1)

with ${\xi }_{ij}=\langle \mathit{w},\varphi (x_i^{+})\rangle -\langle \mathit{w},\varphi (x_j^{-})\rangle$, i = 1, 2, …, n⁺, j = 1, 2, …, n⁻, where n⁺ and n⁻ are the numbers of positive (+1) and negative (−1) training samples, respectively; ϕ: X → F denotes a mapping function which maps the input space X into a new feature space F; w is the weight of the linear classifier; I is the indicator function (1: when condition holds; 0: otherwise). The key idea of above formulation of AUC maximization is to assign higher prediction values for positive training samples compared with negative training samples and make the learned classifier work for as many positive–negative sample pairs as possible.

Since optimization problem 1 is not differentiable, Rakotomamonjy proposed the following approximately equivalent problem (1-norm or 2-norm) based on a large margin learning theory [5]:

Optimization problem 2

$$\underset{\mathit{w}}{min}\frac{1}{2}{\Vert \mathit{w}\Vert }^2+C{\sum }_{i=1}^{n^{+}}{\sum }_{j=1}^{n^{-}}{\xi }_{ij}^r$$ (2)

with

$$\begin{array}{l}\langle \mathit{w},\varphi (x_i^{+})\rangle -\langle \mathit{w},\varphi (x_j^{-})\rangle \ge 1-{\xi }_{ij},\\ {\xi }_{ij}\ge 0,\\ i=1,2,\dots ,n^{+},j=1,2,\dots ,n^{-},\\ r\in \{1,2\}.\end{array}$$

The above constrained quadratic programming optimization problem 2 can be solved using the Lagrange multiplier optimization method [5]. Optimization problem 2 attempts to identify a linear classifier in the reproducing kernel Hilbert space, which makes correct predictions for every positive–negative pair in the training set with certain relaxation ξ_ij ≥ 0, i = 1, 2, …, n⁺, j = 1, 2, …, n⁻.

3. A semi-supervised learning method for AUC optimization

In optimization problem 2 we consider only training samples during the AUC optimization process. Therefore, this is a supervised learning algorithm in essence. It has been shown in the semi-supervised learning literature that adding information from unlabeled test samples can be helpful in identifying a more accurate decision boundary in classification problems [25,26]. One natural question is how to best utilize the information contained in the unlabeled test set to help maximize the AUC during optimization in large margin leaning classifiers (e.g., SVMs).

To extend large margin learning to the semi-supervised learning domain, Bennett and Demiriz proposed a semi-supervised support vector machine (S³VM) [34]. S³VM minimizes both the classification accuracy and the function capacity based on available data in both training and test sets. The key idea in the formulation of S³VM is the incorporation of unlabeled test sample constraints within the large margin learning framework. Because the labels for the test samples are unknown, two constraints are imposed in the optimization problem for each test sample. This corresponds to the situation in which the unknown test sample is first assumed to be a positive sample, and then a negative sample. Later, Sindhwani and Keerthi proposed semi-supervised linear SVMs to handle large scale data [28,29].

Inspired by the above-mentioned work on semi-supervised SVMs, in this paper we proposed two new semi-supervised algorithms to solve the AUC optimization problem 2. The basic idea is to incorporate unlabeled data in the AUC optimization framework shown in problem 2 and guess labels of unlabeled data during the optimization process. For each test sample, we first assume it is positive and compare it with all negative training samples; then we assume it is negative and compare it with all positive training samples. By this way, we hope we can rank potential positive samples higher compared with potential negative samples in the test set with the guidance of labeled training samples. In another words, here we propose to utilize unlabeled test data which is the essence of semi-supervised learning and try to rank as many positive test samples higher (compared with negative samples) as possible which is the essence of AUC optimization. More specifically, for a two-class classification problem, given positive training samples $\{(x_1^{+},y_1^{+}),\dots ,(x_p^{+},y_p^{+})\},y_i^{+}\in \{+1\}$ i = 1, 2, …, p, negative training samples $\{(x_1^{-},y_1^{-}),\dots ,(x_q^{-},y_q^{-})\},y_j^{-}\in \{-1\}$, j = 1, 2, …, q, and test samples {(x₁), …, (x_r)} without labels, the optimization problem for maximizing the AUC under the semi-supervised learning settings is defined as

Optimization problem 3 (1-norm)

$$\begin{array}{l}\underset{\mathit{w},\mathit{\xi },\mathit{\eta },\mathit{u},\mathit{d}}{min}\frac{1}{2}{\Vert \mathit{w}\Vert }^2+\frac{C_1}{2}\sum _{i=1}^p\sum _{j=1}^q{\xi }_{ij}+\frac{C_2}{2}\sum _{m=1}^r\left(\sum _{j=1}^q{\eta }_{mj}+\sum _{i=1}^p{\mu }_{mi}\right)\\ \mathrm{s}.\mathrm{t}.(\langle \mathit{w},\varphi (x_i^{+})\rangle -\langle \mathit{w},\varphi (x_j^{-})\rangle )\ge 1-{\xi }_{ij},\\ {\xi }_{ij}\ge 0,i=1,2,\dots ,p,j=1,2,\dots ,q,\\ (\langle \mathit{w},\varphi (x_m)\rangle -\langle \mathit{w},\varphi (x_j^{-})\rangle )+M(1-d_m)\ge 1-{\eta }_{mj},\\ {\eta }_{mj}\ge 0,m=1,2,\dots ,r,j=1,2,\dots ,q,\\ -1\times (\langle \mathit{w},\varphi (x_m)\rangle -\langle \mathit{w},\varphi (x_i^{+})\rangle )+{Md}_m\ge 1-{\mu }_{mi},\\ {\mu }_{mi}\ge 0,m=1,2,\dots ,r,i=1,2,\dots ,p,\end{array}$$ (3)

where w is the linear classifier to be identified; margin size parameter M is a sufficiently large constant introduced to handle the margins between test samples and positive/negative training samples; C₁ and C₂ are trade-off parameters to balance classifier complexity, training error of training samples, and impact from unlabeled test samples; ξ_ij ≥ 0, i = 1, 2, …, p, j = 1, 2, …, q, are margins introduced to accommodate non-linear separable positive–negative pairs in the training set; η_mj ≥ 0, m = 1, 2, …, r, j = 1, 2, …, q, are margins introduced for test-negative sample pairs; μ_mi ≥ 0, m = 1, 2, …, r, i = 1, 2, …, p, are margins introduced for test-positive sample pairs. d_m ∈ {0, 1}, m = 1, 2, …, r, are estimated labels of the unlabeled test samples (0 means negative sample). The objective function shown in Eq. (3) contains three parts: the first part is a penalty item on the complexity of the classifier; the second part weighted by C₁ contains training errors on positive–negative pairs; the last part weighted by C₂ deals with the empirical errors from unlabeled test data. In the constraints shown above, there are also three parts: the first part shows the pair-wise empirical errors from the training set; the second/third part shows empirical errors when compare test samples with negative/positive training samples (labels of test samples are estimated during the optimization process). Based on d_m and M, for each test sample, although there are two constraints in Eq. (3), actually only one constraint will take effect. In our experiments, we kept C₁ and C₂ equal to make the algorithm simple. A key advance in this approach is the inclusion of manifold information of test samples as part of the AUC maximizing (or ranking) constraints regarding positive/negative training samples.

Theorem 1

The optimal solution for quadratic optimization problem 3 can be found by solving the following semidefinite programming (SDP) problem:

$$\begin{array}{l}\underset{{\mathit{d}}_{\mathbf{3}},\mathit{\psi },\mathit{\zeta }}{min}t\\ \mathrm{s}.\mathrm{t}.\left[\begin{array}{cc}\mathit{K}& \frac{({\mathit{d}}_{\mathbf{3}}-\mathit{\psi }+\mathit{\zeta })}{\sqrt{2}}\\ \frac{{({\mathit{d}}_{\mathbf{3}}-\mathit{\psi }+\mathit{\zeta })}^T}{\sqrt{2}}& t-{\mathit{\psi }}^T{\mathit{e}}_{\mathbf{3}}\end{array}\right]\ge 0,\\ \mathit{\psi }\ge 0,\mathit{\zeta }\ge 0,\end{array}$$ (4)

where

$$\begin{array}{l}{\mathit{d}}_{\mathbf{3}}=\left(\begin{array}{c}\mathit{e}\\ -{\mathit{d}}^N\\ -{\mathit{d}}^P\end{array}\right),\\ d_{(m-1)\ast q+j}^N=(M(1-d_m)-1),m=1,2,\dots ,r,j=1,2,\dots ,q,\\ d_{(m-1)\ast p+i}^P=({Md}_m-1),m=1,2,\dots ,r,i=1,2,\dots ,p,\\ K_{i_1{j}_1,i_2{j}_2}^{\mathit{PNPN}}=k_{i_1{i}_2}-k_{i_1{j}_2}-k_{j_1{i}_2}+k_{j_1{j}_2},\\ K_{{ij}_1,{mj}_2}^{\mathit{PNUN}}=k_{im}-k_{{ij}_2}-k_{j_{1}m}+k_{j_1{j}_2},\\ K_{i_{1}j,{mi}_2}^{\mathit{PNUP}}=k_{i_{1}m}-k_{i_1{i}_2}-k_{jm}+k_{{ji}_2},\\ K_{m_1{j}_1,m_2{j}_2}^{\mathit{UNUN}}=k_{m_1{m}_2}-k_{m_1{j}_2}-k_{j_1{m}_2}+k_{j_1{j}_2},\\ K_{m_{1}j,m_2,i}^{\mathit{UNUP}}=k_{m_1{m}_2}-k_{m_{1}i}-k_{{jm}_2}+k_{ji},\\ K_{m_1{i}_1,m_2{i}_2}^{\mathit{UPUP}}=k_{m_1{m}_2}-k_{m_1{i}_2}-k_{i_1{m}_2}+k_{i_1{i}_2},\end{array}$$

k_ij = 〈ϕ(x_i), ϕ(x_j) 〉, ϕ(x) is a chosen kernel function, x_i and x_j are samples from the set denoted by the corresponding superscripts, K^PNUP = K^UPPNT, K^PNUN = K^UNPNT, K^UNUP = K^UPUNT and

$$K=\left[\begin{array}{c}+{\mathit{K}}^{\mathit{PNPN}}+{\mathit{K}}^{\mathit{PNUN}}-{\mathit{K}}^{\mathit{PNUP}}\\ +{\mathit{K}}^{\mathit{UNPN}}+{\mathit{K}}^{\mathit{UNUN}}-{\mathit{K}}^{\mathit{UNUP}}\\ -{\mathit{K}}^{\mathit{UPPN}}-{\mathit{K}}^{\mathit{UPUN}}+{\mathit{K}}^{\mathit{UPUP}}\end{array}\right].$$

The proof of this theorem is shown in Appendix A. In the above definitions, P means positive training samples; N means negative training samples; U means unknown test samples. Each block in the block matrix K contains kernel function values from four datasets denoted by its superscript.

The AUC optimization problem using semi-supervised learning can also be formulated using 2-norm soft margin:

Optimization problem 4 (2-norm)

$$\begin{array}{l}\underset{\mathit{w},\mathit{\xi },\mathit{\eta },\mathit{\mu },\mathit{d}}{min}\frac{1}{2}{\Vert \mathit{w}\Vert }^2+\frac{C_1}{2}\sum _{i=1}^p\sum _{j=1}^q{\xi }_{ij}^2+\frac{C_2}{2}\sum _{m=1}^r\left(\sum _{j=1}^q{\eta }_{mj}^2+\sum _{i=1}^p{\mu }_{mi}^2\right)\\ \mathrm{s}.\mathrm{t}.(\langle \mathit{w},\varphi (x_i^{+})\rangle -\langle \mathit{w},\varphi (x_j^{-})\rangle )\ge 1-{\xi }_{ij}\\ {\xi }_{ij}\ge 0,i=1,2,\dots ,p,j=1,2,\dots ,q\\ (\langle \mathit{w},\varphi (x_m)\rangle -\langle \mathit{w},\varphi (x_j^{-})\rangle )+M(1-d_m)\ge 1-{\eta }_{mj}\\ {\eta }_{mj}\ge 0,m=1,2,\dots ,r,j=1,2,\dots ,q\\ -1\times (\langle \mathit{w},\varphi (x_m)\rangle -\langle \mathit{w},\varphi (x_i^{+})\rangle )+{Md}_m\ge 1-{\mu }_{mi}\\ {\mu }_{mi}\ge 0,m=1,2,\dots ,r,i=1,2,\dots ,p\end{array}$$ (5)

Theorem 2

The optimal solution for quadratic optimization problem 4 can be found by solving the following SDP problem:

$$\begin{array}{l}\underset{{\mathit{d}}_{\mathbf{3}},\mathit{\zeta }}{min}\\ \mathrm{s}.\mathrm{t}.\left[\begin{array}{cc}\mathit{K}& \frac{({\mathit{d}}_{\mathbf{3}}+\mathit{\zeta })}{\sqrt{2}}\\ \frac{{({\mathit{d}}_{\mathbf{3}}+\mathit{\zeta })}^T}{\sqrt{2}}& t\end{array}\right]\ge 0,\mathit{\zeta }\ge 0,\end{array}$$

where

$${\mathit{d}}_{\mathbf{3}}=\left(\begin{array}{c}\mathit{e}\\ -{\mathit{d}}^N\\ -{\mathit{d}}^P\end{array}\right),$$

and

$$K=\left[\begin{array}{c}+{\mathit{K}}^{\mathit{PNPN}}+{\mathit{K}}^{\mathit{PNUN}}-{\mathit{K}}^{\mathit{PNUP}}\\ +{\mathit{K}}^{\mathit{UNPN}}+{\mathit{K}}^{\mathit{UNUN}}-{\mathit{K}}^{\mathit{UNUP}}\\ -{\mathit{K}}^{\mathit{UPPN}}-{\mathit{K}}^{\mathit{UPUN}}+{\mathit{K}}^{\mathit{UPUP}}\end{array}\right].$$

The proof of Theorem 2 is similar to Theorem 1.

4. Experimental validation

4.1. Experimental settings

To evaluate the proposed SSLROC1 (1-norm) and SSLROC2 (2-norm) AUC optimization methods, we compared them with SVMs [35] and three state-of-the-art supervised AUC optimization methods: SVMROC [5], RankBoost [23] and OPAUC [24]. We also compare the proposed methods with two semi-supervised classifiers SSRank-Boost [30] and SVMlin [28,29] to show the advantages unlabeled data bring to the AUC optimization problem. For each tested method and dataset we used 5 × 2-fold cross validation, which contains 5 repetitions of 2-fold cross validation (CV). The validation method was inspired by Dietterich’s 5 × 2 CV paired t-test study [36], which has a low probability of incorrectly detecting a difference when no difference exists (type-I error), and a reasonable probability of detecting a difference when it exists (power). We calculated the AUC for each test fold of the 5 × 2-fold CV using prediction values from each method, and used the AUC average of the ten test folds to evaluate the performance of each method on each dataset. To determine whether two compared methods have a significant difference across multiple datasets, we used a Wilcoxon paired signed rank test.

For all datasets, we used z-score to normalize all features available such that each feature is centered to have mean of zero and scaled to have standard deviation of one. For SVMs, SVMROC, SSLROC1 and SSLROC2 (the four kernel based learning methods), we used a Gaussian radial basis function (RBF) as a kernel function for the similarity calculation, and the width factor σ was set as the 90th percentile of pairwise distances (in ascending order) between all instances or samples for each dataset. For SVMs and SVMROC, the classifier complexity and training error trade off parameter C was varied from 1 × 10⁻⁴ to 1 × 10², increasing linearly in log 10 scale. For RankBoost, we tuned the number of weak learners from 30 to 90 to identify the optimal parameter. We explored the same parameter space for OPAUC as the authors proposed in ref. [24] to identify the optimal parameter combinations: learning rate η changes from 2⁻¹² to 2¹⁰ and regularization parameter λ changes from 2⁻¹⁰ to 2² (change linearly in log 2 scale). For SVMlin [28,29], we tested the following parameter combinations: regularization parameter λ [10⁻⁴, 10⁴] and λ_u [10⁻², 10²] (linearly changing in the log 10 scale). The parameters used for SSRankBoost [30] are discount factor [0:0.2:1] and the number of unlabeled examples K [1:10]. For the proposed methods SSLROC1 and SSLROC2, trade off parameter C was set from 10⁻³ to 10¹ (change linearly in log 10 scale) and margin size parameter M was set as one of the three values: 0.1, 1 and 10.

Matlab was used as the programming environment in this study. We employed the public open source Matlab toolboxes SDPT3 [37], Sedumi [38] and YALMIP [39] as the SDP solver. For SVMROC and RankBoost we employed the SVM-KM kernel learning toolbox [40] (http://asi.insa-rouen.fr/~arakotom/toolbox/index). SVMlin was downloaded from http://vikas.sindhwani.org/svmlin.html. OPAUC was from Prof. Zhihua Zhou’s lab (http://lamda.nju.edu.cn). SSRankBoost was downloaded from http://ama.liglab.fr/~amini/SSRankBoost/.

4.2. Experimental results on UCI datasets

4.2.1. UCI datasets

To test the proposed ROC optimization algorithm and compare it with SVMs and traditional ROC optimization methods, we employed the University of California, Irvine (UCI) machine learning repository [41]. The UCI machine learning repository contains more than 200 datasets contributed from various application domains and is widely used in the machine learning community to evaluate various algorithms such as clustering, feature extraction, classification, and regression.

To determine the number of datasets needed for the experiments, we performed power analysis [42] using a Wilcoxon paired signed rank test. Power analysis showed that 34 datasets were needed in order to secure a 10% probability of getting a type I error and a 20% probability of getting a type II error (alpha=0.1, power=0.8) for the comparison between the proposed method and SVMs. Thus, we randomly selected 34 classification datasets from the UCI Repository. Note, we did not account for multiple hypotheses in our sample size calculation. All datasets had an attribute that could be used as a class label. Some were multi-class classification problems converted to binary classification problems based on previous published work using these datasets. In Table 1 we list all datasets used in this study along with the number of instances, attributes, and class label for each dataset. Due to computational considerations, we randomly selected 100 instances or samples from each dataset if it contained more than 100 instances.

Table 1.

Characteristics of the 34 UCI datasets employed in this study. Under the class labels, “rest” designates that it was a multi-class problem and that the rest of the classes were combined into one class.

Dataset	Name	#	Attr	Class label (+1)	Class label (−1)
1	Abalone	4177	8	Female	Male, infant
2	Arcene_train	100	10,000	Positive	Negative
3	Blood	748	5	Donated blood	Did not donate
4	Breast	106	9	Car, fad, mas	gla, con, adi
5	Bupaliver	345	6	>5 drinks	<5 drinks
6	Cancer_wbc	669	9	Malignant	Benign
7	Cardio	74	10	Alive after 1 year	Died before 1 year
8	cmc	1473	9	Long/use of contraceptives	No contraceptive use
9	cnae_9	1080	856	Category: range 1–5	Category: range 6–9
10	Credit_g	1000	20	Good credit	Bad credit
11	Derm	366	34	4,5,6	1,2,3
12	E.coli	336	6	pp	rest
13	Glass	214	9	7th type	Rest
14	Heart	270	14	Absence	Presence
15	Hepatitis	155	19	Die	Live
16	House	435	16	Democrat	Republican
17	Ionosphere	351	34	Bad	Good
18	Iris	150	4	setosa	Versicolor, virginica
19	Kidney_inflam	120	6	Bladder inflammation	No inflammation
20	Kr vs. kp	3196	36	White wins	White loses
21	Mushroom	8124	21	Edible	Poisonous
22	Parkinsons	197	23	Parkinson’s	Healthy
23	pima	768	8	Positive for diabetes	No diabetes
24	Post_op	90	8	Patient discharged (s)	Rest
25	sonar	208	60	Rock	Mine
26	Spectf	267	45	1	0
27	Statlog	690	14	Credit approved	Not approved
28	Survival	306	3	Survived 5+ years	Died within 5 years
29	Teach	151	5	Low	Medium, high
30	Tictactoe	958	9	x wins	x loses
31	Vehicle	846	18	van, bus	saab, opel
32	Weight	625	4	Right-leaning	Balanced/left-leaning
33	Wine	178	12	Cultivar 3	Cultivar 1 and 2
34	Zoo	101	17	Aquatic animals	Not aquatic

4.2.2. Results

In Table 2 we show the average AUC of the eight compared methods tested on each of the 34 UCI datasets using the 5 × 2-fold CV. For each dataset and each method tested, the AUC shown was from the optimal parameters which achieved the highest AUC performance. We also show the corresponding standard deviation for each method on each dataset. Standard deviation was calculated based on the ten AUC values from 5 × 2-fold CV. In Table 3 we list the numbers of win-tie-loss between the eight methods (pairwise) on the 34 UCI datasets. We observed that compared with state-of-the-art classification methods, SSLROC1 and SSLROC2 showed superior performance on more datasets. In Table 4 we show p values of the Wilcoxon signed rank tests between the eight methods (pairwise) on the 34 UCI datasets. Since the highest p-value is less than α=0.05, Hochberg’s method for multiple tests of statistical significance [43] indicates that SSLROC1 and SSLROC2 have significantly improved performance compared with other methods. Also from the table we find that the difference between the proposed methods SSLROC1 (1-norm) and SSLROC2 (2-norm) does not reach statistical significance.

Table 2.

Average AUCs of eight methods on the 34 UCI machine learning datasets.

Dataset	SVM		SVMROC		RankBoost		OPAUC		SVMlin		SSRankBoost		SSLROC1		SSLROC2
	Avg	Std	Avg	Std	Avg	Std	Avg	Std	Avg	Std	Avg	Std	Avg	Std	Avg	Std
1	0.700	0.078	0.699	0.079	0.620	0.083	0.701	0.075	0.704	0.078	0.654	0.080	0.703	0.077	0.702	0.078
2	0.806	0.092	0.806	0.092	0.659	0.086	0.701	0.126	0.811	0.094	0.707	0.083	0.815	0.072	0.818	0.069
3	0.735	0.039	0.735	0.036	0.689	0.072	0.716	0.056	0.725	0.045	0.691	0.077	0.737	0.042	0.736	0.048
4	0.906	0.026	0.927	0.038	0.946	0.015	0.877	0.030	0.926	0.031	0.915	0.026	0.910	0.038	0.930	0.026
5	0.631	0.086	0.627	0.088	0.631	0.059	0.641	0.082	0.648	0.095	0.652	0.088	0.638	0.081	0.632	0.084
6	0.993	0.006	0.993	0.006	0.974	0.030	0.993	0.006	0.994	0.006	0.984	0.013	0.994	0.005	0.993	0.005
7	0.984	0.013	0.979	0.013	0.963	0.045	0.984	0.015	0.982	0.013	0.978	0.016	0.983	0.013	0.979	0.012
8	1.00	0.001	1.00	0.001	1.00	0.000	1.00	0.001	1.00	0.001	1.00	0.000	0.999	0.001	1.00	0.001
9	0.932	0.046	0.938	0.049	0.904	0.056	0.958	0.021	0.959	0.025	0.927	0.043	0.943	0.043	0.943	0.043
10	0.734	0.092	0.734	0.092	0.700	0.096	0.725	0.088	0.719	0.099	0.699	0.087	0.733	0.092	0.734	0.091
11	0.993	0.008	0.994	0.007	0.960	0.032	0.994	0.007	0.991	0.011	0.981	0.012	0.996	0.005	0.996	0.005
12	0.948	0.037	0.952	0.044	0.860	0.112	0.931	0.052	0.917	0.070	0.922	0.069	0.947	0.050	0.951	0.047
13	0.963	0.044	0.964	0.045	0.905	0.110	0.954	0.061	0.964	0.034	0.934	0.063	0.970	0.030	0.970	0.036
14	0.923	0.039	0.925	0.036	0.880	0.051	0.928	0.038	0.923	0.038	0.892	0.050	0.925	0.037	0.925	0.038
15	0.856	0.061	0.853	0.061	0.753	0.080	0.842	0.056	0.854	0.052	0.803	0.065	0.854	0.057	0.851	0.060
16	0.988	0.012	0.988	0.011	0.983	0.016	0.989	0.010	0.989	0.010	0.984	0.016	0.991	0.006	0.990	0.009
17	0.951	0.037	0.946	0.044	0.914	0.049	0.889	0.049	0.933	0.023	0.901	0.059	0.964	0.026	0.962	0.028
18	1.00	0.000	1.00	0.000	0.998	0.006	1.00	0.000	1.00	0.000	0.999	0.002	1.00	0.000	1.00	0.000
19	1.00	0.000	1.00	0.000	1.00	0.000	1.00	0.000	1.00	0.000	1.00	0.000	1.00	0.000	1.00	0.000
20	0.884	0.035	0.884	0.046	0.934	0.042	0.873	0.045	0.891	0.049	0.934	0.042	0.892	0.033	0.888	0.041
21	0.954	0.037	0.957	0.036	0.955	0.040	0.954	0.039	0.949	0.041	0.956	0.035	0.961	0.023	0.961	0.023
22	0.878	0.040	0.878	0.041	0.885	0.047	0.879	0.043	0.877	0.037	0.885	0.039	0.878	0.040	0.879	0.041
23	0.766	0.066	0.767	0.066	0.729	0.060	0.766	0.076	0.778	0.045	0.736	0.043	0.784	0.054	0.777	0.049
24	0.604	0.070	0.606	0.072	0.593	0.059	0.607	0.069	0.595	0.082	0.606	0.062	0.607	0.081	0.606	0.065
25	0.830	0.059	0.850	0.045	0.842	0.048	0.839	0.045	0.833	0.043	0.837	0.055	0.845	0.039	0.849	0.047
26	0.852	0.064	0.852	0.064	0.736	0.101	0.852	0.068	0.834	0.081	0.783	0.058	0.849	0.063	0.852	0.065
27	0.928	0.032	0.925	0.027	0.887	0.036	0.924	0.030	0.926	0.034	0.885	0.046	0.928	0.028	0.927	0.031
28	0.646	0.102	0.644	0.101	0.616	0.067	0.646	0.086	0.642	0.106	0.616	0.077	0.647	0.069	0.658	0.084
29	0.708	0.074	0.710	0.086	0.684	0.062	0.670	0.079	0.675	0.093	0.700	0.068	0.731	0.053	0.726	0.049
30	0.750	0.105	0.744	0.110	0.762	0.091	0.679	0.070	0.692	0.083	0.765	0.089	0.774	0.074	0.782	0.064
31	0.949	0.033	0.951	0.036	0.859	0.050	0.850	0.074	0.937	0.041	0.877	0.034	0.956	0.029	0.966	0.024
32	0.988	0.009	0.989	0.009	0.958	0.028	0.974	0.013	0.988	0.010	0.913	0.034	0.988	0.013	0.988	0.013
33	1.00	0.001	1.00	0.001	0.976	0.053	1.00	0.001	1.00	0.001	0.994	0.008	1.00	0.000	1.00	0.000
34	0.989	0.010	0.990	0.009	0.992	0.007	0.990	0.010	0.991	0.011	0.991	0.009	0.991	0.008	0.991	0.008
Avg	0.875	0.043	0.877	0.044	0.845	0.053	0.863	0.045	0.872	0.044	0.856	0.045	0.880	0.038	0.881	0.038

Table 3.

Numbers of wins–ties–losses (superior–equal–inferior AUC) between the eight methods (pairwise) on the 34 UCI datasets. For three numbers shown in each entry of the table, the first is the number of wins of corresponding method shown on the left column compared with the corresponding method shown on the top; middle is the number of ties between them and the third is the number of losses. (M1:SVM; M2:SVMROC; M3:RankBoost; M4:OPAUC; M5:SVMlin; M6:SSRankBoost; M7:SSLROC1; M8:SSLROC2).

Method	M1	M2	M3	M4	M5	M6	M7	M8
M1	0–34–0	13–6–15	25–1–8	17–2–15	19–2–13	23–1–10	7–2–25	5–2–27
M2	15–6–13	0–34–0	26–1–7	19–2–13	18–2–14	27–1–6	7–2–25	6–2–26
M3	8–1–25	7–1–26	0–34–0	11–1–22	9–1–24	9–3–22	5–1–28	5–1–28
M4	15–2–17	13–2–19	22–1–11	0–34–0	13–3–18	21–1–12	7–2–25	9–2–23
M5	13–2–19	14–2–18	24–1–9	18–3–13	0–34–0	23–1–10	7–2–25	9–2–23
M6	10–1–23	6–1–27	22–3–9	12–1–21	10–1–23	0–34–0	5–1–28	5–1–28
M7	25–2–7	25–2–7	28–1–5	25–2–7	25–2–7	28–1–5	0–34–0	15–6–13
M8	27–2–5	26–2–6	28–1–5	23–2–9	23–2–9	28–1–5	13–6–15	0–34–0

Table 4.

p Values of the Wilcoxon signed rank tests between the four methods (pairwise). (M1:SVM; M2:SVMROC; M3:RankBoost; M4:OPAUC; M5:SVMlin; M6:SSRankBoost; M7:SSLROC1; M8:SSLROC2).

Method	M1	M2	M3	M4	M5	M6	M7	M8
M1	1.000	0.539	0.000	0.082	0.246	0.002	0.000	0.000
M2	0.539	1.000	0.000	0.026	0.145	0.000	0.003	0.000
M3	0.000	0.000	1.000	0.012	0.001	0.003	0.000	0.000
M4	0.082	0.026	0.012	1.000	0.117	0.098	0.001	0.001
M5	0.246	0.145	0.001	0.117	1.000	0.006	0.001	0.004
M6	0.002	0.000	0.003	0.098	0.006	1.000	0.000	0.000
M7	0.000	0.003	0.000	0.001	0.001	0.000	1.000	0.820
M8	0.000	0.000	0.000	0.001	0.004	0.000	0.820	1.000

For the proposed SSLROC1 and SSLROC2 methods, there are two critical parameters which control their generalization ability. They are training error trade-off parameter C and margin size parameter M. To identify the influence of C and M on the performance of the proposed methods, in Fig. 1 we show the average AUC of SSLROC1 and SSLROC2 on three example UCI datasets when different C and M were used in the experiment. From these four example cases, we can find a trend in the parameter combinations which leads to better performance. To reduce computation load, we only explored a small parameter space spanned by M and C. There were 15 combinations of them in total which are few. For example, in the work of OPAUC, the authors tested a parameter space spanned by the learning rate eta (2⁻¹²−2¹⁰) and regularization parameter lambda (2⁻¹⁰−2²), 299 parameter combinations in total. From the trends shown on the four UCI datasets, we see that there is a high probability that exploring a larger parameter space will lead to better AUC performance.

Fig. 1.

Average AUCs of 5 × 2-fold CV on three example UCI datasets. The first column corresponds to SSLROC1 and the second column corresponds to SSLROC2. Each row corresponds to one dataset. For both SSLROC1 and SSLROC2, we show results in the same parameter space spanned by log 10(C) and log 10(M).

4.3. Experimental results on CTC dataset

Colorectal cancer is the second-leading cause of cancer death in Americans [44]. Computed tomographic colonography (CTC), also known as virtual colonoscopy, provides a less invasive alternative to optical colonoscopy in screening patients for colonic polyps [45]. In Fig. 2, we show 3D volume rendering of a segmented colon and a typical colonic polyp on the fold. Previous studies showed that computer-aided detection systems can assist radiologists in CTC reading and improve their detection performance [46–49]. To show the effectiveness of our proposed methods and their potential applications in the CTC computer-aided detection system (CAD), we tested all four methods on a CTC dataset and analyzed the results using ROC analysis.

Fig. 2.

3D volume rendering of a segmented colon (left figure) with spine and ribs; a typical colonic polyp on the fold (right figure).

4.3.1. CTC datasets

Our dataset consisted of CTC examinations of 50 patients collected from three medical centers. Each patient had one or more polyps ≥6 mm confirmed by histopathological evaluation following optical colonoscopy (OC). Each patient was scanned in the supine and prone positions, and each scan was performed during a single breath hold using a 4- or 8-channel CT scanner. CT scanning parameters included 1.25- to 2.5-mm section collimation, 15 mm/s table speed, 1-mm reconstruction interval, 100 mAs, and 120 kVp. For each CT scan in the dataset, we segmented the colon first from the original 3D image [50]. Then we searched the inner surface of the colon to identify initial colonic polyp candidates. Our initial detection scheme based on surface curvature analysis reported 60 colonic polyps 5–30 mm in size and 5234 false positives. The labels of initial detections were determined by OC examination which is a golden standard in CTC. Each initial detection defined as a CAD detection represents a candidate polyp. After initial detection we extracted 157 3D geometric features from each colonic polyp candidate [47]. The polyps were confirmed by traditional optical colonoscopy. To make the problem computationally feasible we filtered the initial dataset to 100 CAD detections, which included 49 true detections and 51 false positives by removing true and false positives with low SVM vote values predicted by a SVM committee classifier [51]. 5 × 2-fold CV was performed on the filtered dataset and test set in CV was treated as unlabeled samples under our SSL learning framework.

4.3.2. Results

In Fig. 3, we show AUCs of eight methods on the CTC dataset. RankBoost showed the highest performance with an AUC of 0.914. The proposed SSLROC2 method was ranked as the second highest performance with AUC of 0.909. Please note that both SSLROC1 and SSLROC2 outperformed all other semi-supervised learning methods for AUC maximization. In Fig. 4, we show comparisons of SSLROC1 and SSLROC2 with different parameters C and M. They both achieved highest performance when log 10(C) = −1 and log 10(M) = 0.

Fig. 3.

Comparison of AUCs of eight methods on the CTC dataset.

Fig. 4.

Average AUC of SSLROC1 (a) and SSLROC2 (b) on the CTC dataset when different C values (classifier complexity and training error trade-off parameter) and M values (margin size parameter) were used in the experiment. (a) SSLROC1 and (b) SSLROC2.

5. Discussion and conclusion

We proposed two new AUC optimization methods called SSLROC1 and SSLROC2, which introduce test samples in the optimization of margins in a binary classification problem for the purpose of AUC maximization. We tested the proposed methods on 34 randomly selected UCI machine learning datasets. The SSLROC algorithms were found to have superior AUCs in a significantly larger fraction of UCI datasets compared with SVMs, SVMROC, RankBoost, OPAUC, SVMlin, and SSRankBoost which are state-of-the-art classification and AUC optimization methods. The proposed methods also showed advantages in a colonic polyp classification problem for a dataset of CT colonography cases compared with other methods except RankBoost.

SVMs have a complexity of O(kn²) for RBF kernels and O(kn) for linear kernels, where n and k are the number of training samples and features, respectively. For our proposed method the computational complexity will increase to O(25kn⁴/16) and O(25kn²/16) for RBF and linear kernels, respectively, due to the introduction of test samples during AUC optimization. Here we assume that the training and testing sets have the same number of instances, which is the case for 5 × 2-fold cross validation; we also assume that the number of positive and negative samples is equal. For SVMROC, the computational complexity are O(kn⁴/4) and O(kn²/4) for RBF and linear kernels, respectively, under the assumption that the number of positive and negative samples is equal. The computational complexity analysis shows that the computational complexity is two orders of magnitude higher for both SVMROC and SSLROC over SVMs. For this reason the proposed method was applied to only small datasets in our study. However, the increased complexity of our method is balanced by its significantly higher performance over the other techniques. In future work we will investigate how to develop a more computationally efficient algorithm, likely using more efficient algorithms to approximate the solution of the AUC maximization problem in large datasets [8].

Another potential disadvantage of the SSLROC method (and all transductive learning algorithms) is that when a new test dataset is acquired, the algorithm needs to be re-trained using the new test set as unlabeled data. This is in contrast to inductive learning algorithms (including all supervised algorithms), where the trained classifier can be directly applied to a new test dataset. In the field of computer-aided detection and diagnosis for radiological images, it is preferred to have a well trained CAD system and deploy it to hospitals or clinics without further training. Thus, a future research topic of interest will be to combine online and transductive learning to address the retraining issue in transductive AUC learning.

As we showed in the previous section, SSLROC1 and SSLROC2 did not reach statistical significance on the 34 UCI datasets. In the literature, Ng showed that sample complexity which is the minimum number of training examples required to train a good classifier grows only logarithmically as the number of irrelevant features increases in the dataset when L1 regularization is employed [52]; L2 regularization has a worst sample complexity that grows at least linearly. In the work of Zhu et al. on 1-norm SVMs, they also argue that 1-norm SVM has some advantages over 2-norm SVM when data contains redundant noise features [53]. For the proposed methods, the major difference is that we use different norms for the regularization. So based on studies shown above, SSLROC1 should beat SSLROC2 when data contain irrelevant noisy features. However, from experimental results shown in Table 2, we did not observe such kind of trend when we compare average AUCs of SSLROC1 and SSLROC2. We suspect that it might be related with small size data employed in this study. In the future, it will be interesting to investigate how the data size affects the generalization performance of the two proposed methods.

In conclusion, we developed new methods of AUC optimization based on semi-supervised learning and transductive learning that yield improved classifier performance on multiple public datasets. The proposed methods may lead to improved classification performance in diverse realms of data analysis including medical imaging and computer vision.

Biography

Dr. Shijun Wang received his PhD degree in Control Science and Engineering from Tsinghua University, China, where his research focused on machine learning and complex systems. He earned a BS in Electronic Engineering at Beihang University and an MS in Communication at Second Aerospace Science Academy, China. Dr. Shijun Wang’s current research interests in the Imaging Biomarkers and Computer-Aided Diagnosis Laboratory include machine learning, statistical image analysis and their applications in computer-aided diagnosis. He is an associate editor of Medical Physics and reviewer for IEEE TMI, IEEE TBME, Journal of Artificial Intelligence Research, Journal of Magnetic Resonance Imaging, Medical Physics, Pattern Analysis & Applications, and Journal of Theoretical Biology.

Appendix A. Proof of Theorem 1

Proof

By using the Lagrange multipliers optimization method [54], we transfer the constrained optimization problem 3 into the following unconstrained primal Lagrange function:

$$\begin{array}{l}{L}_p^1(\mathit{w},\mathit{\xi },\mathit{\eta },\mathit{\mu },\mathit{\alpha },\mathit{\beta },\mathit{\chi },\mathit{\gamma },\mathit{\kappa },\mathit{\lambda })\\ =\frac{1}{2}{\Vert \mathit{w}\Vert }^2+\frac{C_1}{2}\sum _{i=1}^p\sum _{j=1}^q{\xi }_{ij}+\frac{C_2}{2}\sum _{m=1}^r\left(\sum _{j=1}^q{\eta }_{mj}+\sum _{i=1}^p{\mu }_{mi}\right)\\ -\sum _{i=1}^p\sum _{j=1}^q{\alpha }_{ij}((\langle \mathit{w},\varphi (x_i^{+})\rangle -\langle \mathit{w},\varphi (x_j^{-})\rangle )+{\xi }_{ij}-1)-\sum _{i=1}^p\sum _{j=1}^q{\beta }_{ij}{\xi }_{ij}\\ -\sum _{m=1}^r\sum _{j=1}^q{\chi }_{mj}((\langle \mathit{w},\varphi (x_m)\rangle -\langle \mathit{w},\varphi (x_j^{-})\rangle )\\ +M(1-d_m)+{\eta }_{mj}-1)-\sum _{m=1}^r\sum _{j=1}^q{\gamma }_{mj}{\eta }_{mj}\\ -\sum _{m=1}^r\sum _{i=1}^p{\kappa }_{mi}(-1\times (\langle \mathit{w},\varphi (x_m)\rangle -\langle \mathit{w},\varphi (x_i^{+})\rangle )+{Md}_m+{\mu }_{mi}-1)\\ -\sum _{m=1}^r\sum _{i=1}^p{\lambda }_{mi}{\mu }_{mi}.\end{array}$$

The Karush–Kuhn–Tucker (KKT) conditions [54] for optimal primal variables w, ξ, η, and μ are

Stationarity:

$$\begin{array}{l}\frac{\partial L_p^1}{\partial \mathit{w}}=\mathit{w}-{\sum }_{i=1}^p{\sum }_{j=1}^q{\alpha }_{ij}(\varphi (x_i^{+})-\varphi (x_j^{-}))\\ -{\sum }_{m=1}^r{\sum }_{j=1}^q{\chi }_{mj}(\varphi (x_m)-\varphi (x_j^{-}))\\ -{\sum }_{m=1}^r{\sum }_{i=1}^p{\kappa }_{mi}(-1\times (\varphi (x_m)-\varphi (x_i^{+})))=0,\\ \frac{\partial L_p^1}{\partial \mathit{\xi }}=\frac{C_1}{2}\mathit{e}-\mathit{\alpha }-\mathit{\beta }=0\\ \frac{\partial L_p^1}{\partial \mathit{\eta }}=\frac{C_2}{2}\mathit{e}-\mathit{\chi }-\mathit{\gamma }=0,\\ \frac{\partial L_p^1}{\partial \mathit{\mu }}=\frac{C_2}{2}\mathit{e}-\mathit{\kappa }-\mathit{\lambda }=0;\end{array}$$

Primal feasibility:

$$\begin{array}{l}(\langle \mathit{w},\varphi (x_i^{+})\rangle -\langle \mathit{w},\varphi (x_j^{-})\rangle )\ge 1-{\xi }_{ij},{\xi }_{ij}\ge 0,i=1,2,\dots ,p,j=1,2,\dots ,q,\\ (\langle \mathit{w},\varphi (x_m)\rangle -\langle \mathit{w},\varphi (x_j^{-})\rangle )+M(1-d_m)\ge 1-{\eta }_{mj}{\eta }_{mj}\ge 0,m=1,2,\dots ,r,j=1,2,\dots ,q,\\ -1\times (\langle \mathit{w},\varphi (x_m)\rangle -\langle \mathit{w},\varphi (x_i^{+})\rangle )+{Md}_m\ge 1-{\mu }_{mi},{\mu }_{mi}\ge 0,m=1,2,\dots ,r,i=1,2,\dots ,p;\end{array}$$

Dual feasibility:

$$\begin{array}{l}{\alpha }_{ij}\ge 0,i=1,2,\dots ,p,j=1,2,\dots ,q,\\ {\beta }_{ij}\ge 0,i=1,2,\dots ,p,j=1,2,\dots ,q,\\ {\chi }_{mj}\ge 0,m=1,2,\dots ,r,j=1,2,\dots ,q,\\ {\gamma }_{mj}\ge 0,m=1,2,\dots ,r,j=1,2,\dots ,q,\\ {\kappa }_{mi}\ge 0,m=1,2,\dots ,r,i=1,2,\dots ,p,\\ {\lambda }_{mi}\ge 0,m=1,2,\dots ,r,i=1,2,\dots ,p;\end{array}$$

Complementary slackness:

$$\begin{array}{l}{\alpha }_{ij}((\langle \mathit{w},\varphi (x_i^{+})\rangle -\langle \mathit{w},\varphi (x_j^{-})\rangle )+{\xi }_{ij}-1)=0,\\ {\beta }_{ij}{\xi }_{ij}=0,i=1,2,\dots ,p,j=1,2,\dots ,q,\\ {\chi }_{mj}((\langle \mathit{w},\varphi (x_m)\rangle -\langle \mathit{w},\varphi (x_j^{-})\rangle )+M(1-d_m)+{\eta }_{mj}-1)=0,\\ {\gamma }_{mj}{\eta }_{mj}=0,m=1,2,\dots ,r,j=1,2,\dots ,q,\\ {\kappa }_{mi}(-1\times (\langle \mathit{w},\varphi (x_m)\rangle -\langle \mathit{w},\varphi (x_i^{+})\rangle )+{Md}_m+{\mu }_{mi}-1)=0,\\ {\lambda }_{mi}{\mu }_{mi}=0,m=1,2,\dots ,r,i=1,2,\dots ,p,\end{array}$$

where

$$\begin{array}{l}\mathit{\alpha }=\{{\alpha }_{11},{\alpha }_{12},\dots ,{\alpha }_{1q},{\alpha }_{21},{\alpha }_{22}\dots ,{\alpha }_{2q},\dots ,{\alpha }_{p1}\dots ,{\alpha }_{pq}\},\\ \mathit{\beta }=\{{\beta }_{11},{\beta }_{12}\dots ,{\beta }_{1q},{\beta }_{21},{\beta }_{22}\dots ,{\beta }_{2q},\dots ,{\beta }_{p1}\dots ,{\beta }_{pq}\},\\ \mathit{\chi }=\{{\chi }_{11},{\chi }_{12}\dots ,{\chi }_{1q},{\chi }_{21},{\chi }_{22}\dots ,{\chi }_{2q},\dots ,{\chi }_{r1}\dots ,{\chi }_{rq}\},\\ \mathit{\gamma }=\{{\gamma }_{11},{\gamma }_{12}\dots ,{\gamma }_{1q},{\gamma }_{21},{\gamma }_{22}\dots ,{\gamma }_{2q},\dots ,{\gamma }_{r1}\dots ,{\gamma }_{rq}\},\\ \mathit{\kappa }=\{{\kappa }_{11},{\kappa }_{12}\dots ,{\kappa }_{1p},{\kappa }_{21},{\kappa }_{22}\dots ,{\kappa }_{2p},\dots ,{\kappa }_{r1}\dots ,{\kappa }_{rp}\},\\ \mathit{\lambda }=\{{\lambda }_{11},{\lambda }_{12}\dots ,{\lambda }_{1p},{\lambda }_{21},{\lambda }_{22}\dots ,{\lambda }_{2p},\dots ,{\lambda }_{r1}\dots ,{\lambda }_{r_p}\},\end{array}$$

, e is a vector filled with all ones.

The optimal w can be achieved at:

$$\mathit{w}={\sum }_{i=1}^p{\sum }_{j=1}^q{\alpha }_{ij}(\varphi (x_i^{+})-\varphi (x_j^{-}))+{\sum }_{m=1}^r{\sum }_{j=1}^q{\chi }_{mj}(\varphi (x_m)-\varphi (x_j^{-}))+{\sum }_{m=1}^r{\sum }_{i=1}^p{\kappa }_{mi}(-1\times (\varphi (x_m)-\varphi (x_i^{+}))).$$

Let us define: $K_{i_1{j}_1,i_2{j}_2}^{\mathit{PNPN}}=k_{i_1{i}_2}-k_{i_1{j}_2}-k_{j_1{i}_2}+k_{j_1{j}_2},K_{{ij}_1,{mj}_2}^{\mathit{PNUN}}=k_{im}-k_{{ij}_2}-k_{j_{1}m}+k_{j_1{j}_2},K_{i_{1}j,{mi}_2}^{\mathit{PNUP}}=k_{i_{1}m}-k_{i_1{i}_2}-k_{jm}+k_{{ji}_2},K_{m_1{j}_1,m_2{j}_2}^{\mathit{UNUN}}=k_{m_1{m}_2}-k_{m_1{j}_2}-k_{j_1{m}_2}+k_{j_1{j}_2},K_{m_{1}j,m_{2}i}^{\mathit{UNUP}}=k_{m_1{m}_2}-k_{m_{1}i}-k_{{jm}_2}+k_{ji},K_{m_1{i}_1,m_2{i}_2}^{\mathit{UPUP}}=k_{m_1{m}_2}-k_{m_1{i}_2}-k_{i_1{m}_2}+k_{i_1{i}_2}$, k_ij = 〈ϕ(x_i), ϕ(x_j)〉, x_i and x_j are samples from the set denoted by the corresponding superscripts.

In the above definitions, P means positive training samples; N means negative training samples; U means unknown test samples. K^PNPN defines the kernel matrix between positive–negative training sample pairs; K^PNUN defines the kernel matrix between positive–negative training sample pairs and test-negative sample pairs; K^PNUP defines the kernel matrix between positive–negative training sample pairs and test-positive sample pairs; K^UNUN defines the kernel matrix between test-negative sample pairs; K^UNUP defines the kernel matrix between test-negative sample pairs and test-positive sample pairs; K^UPUP defines the kernel matrix between test-positive sample pairs. Here negative and positive samples are only from training set and test samples are only from test set.

Therefore:

$$\begin{array}{l}{L}_p^1(\mathit{w},\mathit{\xi },\mathit{\eta },\mathit{\mu },\mathit{\alpha },\mathit{\beta },\mathit{\chi },\mathit{\gamma },\mathit{\kappa },\mathit{\lambda })\\ =-\frac{1}{2}({\mathit{\alpha }}^T{\mathit{K}}^{\mathit{PNPN}}\mathit{\alpha }+2{\mathit{\alpha }}^T{\mathit{K}}^{\mathit{PNUN}}\mathit{\chi }-2{\mathit{\alpha }}^T{\mathit{K}}^{\mathit{PNUP}}\mathit{\kappa }+{\mathit{\chi }}^T{\mathit{K}}^{\mathit{UNUN}}\mathit{\chi }\\ -2{\mathit{\chi }}^T{\mathit{K}}^{\mathit{UNUP}}\mathit{\kappa }+{\mathit{\kappa }}^T{\mathit{K}}^{\mathit{UPUP}}\mathit{\kappa })+\sum _{i=1}^p\sum _{j=1}^q{\alpha }_{ij}\\ -\sum _{m=1}^r\sum _{j=1}^q{\chi }_{mj}(M(1-d_m)-1)\\ -\sum _{m=1}^r\sum _{i=1}^p{\kappa }_{mi}({Md}_m-1).\end{array}$$

After removing primal variables, we get dual representation of the optimization problem as follows:

$$\begin{array}{l}max{L}_p^1(\mathit{\alpha },\mathit{\chi },\mathit{\kappa })=-\frac{1}{2}\times ({\mathit{\alpha }}^T{\mathit{K}}^{\mathit{PNPN}}\mathit{\alpha }+2{\mathit{\alpha }}^T{\mathit{K}}^{\mathit{PNUN}}\mathit{\chi }-2{\mathit{\alpha }}^T{\mathit{K}}^{\mathit{PNUP}}\mathit{\kappa }\\ +{\mathit{\chi }}^T{\mathit{K}}^{\mathit{UNUN}}\mathit{\chi }-2{\mathit{\chi }}^T{\mathit{K}}^{\mathit{UNUP}}\mathit{\kappa }+{\mathit{\kappa }}^T{\mathit{K}}^{\mathit{UPUP}}\mathit{\kappa })\\ +{\mathit{\alpha }}^T\mathit{e}-\sum _{m=1}^r\sum _{j=1}^q{\chi }_{kj}(M(1-d_m)-1)-\sum _{m=1}^r\sum _{i=1}^p{\kappa }_{mi}({Md}_m-1),\end{array}$$

with constraints: 0 ≤ α ≤ C₁/2, 0 ≤ χ C₂/2, 0 ≤ κ C₂/2. Thus, the Lagrangian of the maximization problem can be defined as

$$\begin{array}{l}{L}_p^1(\mathit{\alpha },\mathit{\chi },\mathit{\kappa },\mathit{\nu },\mathit{o},\mathit{\theta },\mathit{\rho },\mathit{\sigma },\mathit{\tau })\\ =-\frac{1}{2}\times ({\mathit{\alpha }}^T{\mathit{K}}^{\mathit{PNPN}}\mathit{\alpha }+2{\mathit{\alpha }}^T{\mathit{K}}^{\mathit{PNUN}}\mathit{\chi }-2{\mathit{\alpha }}^T{\mathit{K}}^{\mathit{PNUP}}\mathit{\kappa }\\ +{\mathit{\chi }}^T{\mathit{K}}^{\mathit{UNUN}}\mathit{\chi }-2{\mathit{\chi }}^T{\mathit{K}}^{\mathit{UNUP}}\mathit{\kappa }+{\mathit{\kappa }}^T{\mathit{K}}^{\mathit{UPUP}}\mathit{\kappa })\\ +{\mathit{\alpha }}^T\mathit{e}-{\mathit{\chi }}^T{\mathit{d}}^N-{\mathit{\kappa }}^T{\mathit{d}}^p+{\mathit{\nu }}^T\left(\frac{C_1}{2}\mathit{e}-\mathit{\alpha }\right)+{\mathit{o}}^T\mathit{\alpha }\\ +{\mathit{\theta }}^T\left(\frac{C_2}{2}\mathit{e}-\chi \right)+{\mathit{\rho }}^T\mathit{\chi }+{\mathit{\sigma }}^T\left(\frac{C_2}{2}\mathit{e}-\mathit{\kappa }\right)+{\mathit{\tau }}^T\mathit{\kappa },\end{array}$$

where $d_{(m-1)\ast q+j}^N=(M(1-d_m)-1)$, m = 1, 2, …, r, j = 1, 2, …, q, $d_{(m-1)\ast p+i}^P=({Md}_m-1)$, m = 1, 2, …, r, i = 1, 2, …, p, s.t. ν ≥ 0, ο ≥ 0, θ ≥ 0, ρ ≥ 0, σ ≥ 0, τ ≥ 0.

Let us define

$$\begin{array}{l}\mathit{\omega }=\left(\begin{array}{c}\mathit{\alpha }\\ \mathit{\chi }\\ \mathit{\kappa }\end{array}\right),\mathit{\psi }=\left(\begin{array}{c}\mathit{\nu }\\ \mathit{\theta }\\ \mathit{\sigma }\end{array}\right),\mathit{\zeta }=\left(\begin{array}{c}\mathit{o}\\ \mathit{\rho }\\ \mathit{\tau }\end{array}\right)\text{and}\\ K=\left[\begin{array}{c}+{\mathit{K}}^{\mathit{PNPN}}+{\mathit{K}}^{\mathit{PNUN}}-{\mathit{K}}^{\mathit{PNUP}}\\ +{\mathit{K}}^{\mathit{UNPN}}+{\mathit{K}}^{\mathit{UNUN}}-{\mathit{K}}^{\mathit{UNUP}}\\ -{\mathit{K}}^{\mathit{UPPN}}-{\mathit{K}}^{\mathit{UPUN}}+{\mathit{K}}^{\mathit{UPUP}}\end{array}\right],\end{array}$$

where K^PNUP = K^UPPNT, K^PNUN = K^UNPNT, K^UNUP = K^UPUNT.

$$\begin{array}{l}{L}_p^1(\mathit{\alpha },\mathit{\chi },\mathit{\kappa },\mathit{\nu },\mathit{o},\mathit{\theta },\mathit{\rho },\mathit{\sigma },\mathit{\tau })=L_p^1(\mathit{\omega },\mathit{\psi },\mathit{\zeta })\\ =-\frac{1}{2}{\mathit{\omega }}^T\mathit{K}\mathit{\omega }+{\mathit{\omega }}^T\left(\left(\begin{array}{c}\mathit{e}\\ -{\mathit{d}}^N\\ -{\mathit{d}}^P\end{array}\right)-\mathit{\psi }+\mathit{\zeta }\right)+{\mathit{\psi }}^T\left(\begin{array}{c}\frac{C_1}{2}\mathit{e}\\ \frac{C_2}{2}\mathit{e}\\ \frac{C_2}{2}\mathit{e}\end{array}\right).\end{array}$$

Let us define

$${\mathit{d}}_{\mathbf{3}}=\left(\begin{array}{c}\mathit{e}\\ -{\mathit{d}}^N\\ -{\mathit{d}}^P\end{array}\right)\text{and}{\mathit{e}}_{\mathbf{3}}=\left(\begin{array}{c}\frac{C_1}{2}\mathit{e}\\ \frac{C_2}{2}\mathit{e}\\ \frac{C_2}{2}\mathit{e}\end{array}\right).$$

Therefore

$$\begin{array}{l}{L}_p^1(\mathit{\omega },\mathit{\psi },\mathit{\zeta })=-\frac{1}{2}{\mathit{\omega }}^T\mathit{K}\mathit{\omega }+{\mathit{\omega }}^T({\mathit{d}}_3-\mathit{\psi }+\mathit{\zeta })+{\mathit{\psi }}^T{\mathit{e}}_{\mathbf{3}},\\ \mathrm{s}.\mathrm{t}.\mathit{\omega }\ge 0,\mathit{\psi }\ge 0,\mathit{\zeta }\ge 0.\end{array}$$

Based on duality, we have the following equivalent problems:

$$\underset{\mathit{\omega }\ge 0\mathit{\psi }\ge 0,\mathit{\zeta }\ge 0}{\text{max}\text{min}}{L}_p^1(\mathit{\omega },\mathit{\psi },\mathit{\zeta })=\underset{\mathit{\psi }\ge 0,\mathit{\zeta }\ge 0\mathit{\omega }\ge 0}{minmax}{L}_p^1(\mathit{\omega },\mathit{\psi },\mathit{\zeta }).$$

The inner maximization could be achieved at:

$$\begin{array}{l}\frac{\partial L_p^1(\mathit{\omega },\mathit{\psi },\mathit{\zeta })}{\partial \mathit{\omega }}=-\mathit{K}\mathit{\omega }+({\mathit{d}}_{\mathbf{3}}-\mathit{\psi }+\mathit{\zeta })=0\\ \Rightarrow \mathit{\omega }={\mathit{K}}^{-1}({\mathit{d}}_3-\mathit{\psi }+\mathit{\zeta }).\end{array}$$

So

$$\begin{array}{l}\underset{\mathit{\psi }\ge 0,\mathit{\zeta }\ge 0\mathit{\omega }\ge 0}{\text{min}\text{max}}{L}_p^1(\mathit{\omega },\mathit{\psi },\mathit{\zeta })=\underset{\mathit{\psi }\ge 0,\mathit{\zeta }\ge 0}{min}-\frac{1}{2}{\mathit{\omega }}^T\mathit{K}\mathit{\omega }\\ +{\mathit{\omega }}^T({\mathit{d}}_{\mathbf{3}}-\mathit{\psi }+\mathit{\zeta })+{\mathit{\psi }}^T{\mathit{e}}_{\mathbf{3}}|\mathit{\omega }={\mathit{K}}^{-1}({\mathit{d}}_{\mathbf{3}}-\mathit{\psi }+\mathit{\zeta }).\\ =\underset{\mathit{\psi }\ge 0,\mathit{\zeta }\ge 0}{min}\frac{1}{2}{({\mathit{d}}_{\mathbf{3}}-\mathit{\psi }+\mathit{\zeta })}^T{\mathit{K}}^{-1}({\mathit{d}}_{\mathbf{3}}-\mathit{\psi }+\mathit{\zeta })+{\mathit{\psi }}^T{\mathit{e}}_{\mathbf{3}}.\end{array}$$

Let t ≥ 0 be the upper limit of the minimization problem:

$$t\ge \underset{\mathit{\psi }\ge 0,\mathit{\zeta }\ge 0}{min}\frac{1}{2}{({\mathit{d}}_{\mathbf{3}}-\mathit{\psi }+\mathit{\zeta })}^T{\mathit{K}}^{-1}({\mathit{d}}_{\mathbf{3}}-\mathit{\psi }+\mathit{\zeta })+{\mathit{\psi }}^T{\mathit{e}}_{\mathbf{3}}.$$

Using the Schur complement [54], we will get

$$\begin{array}{l}\left[\begin{array}{cc}\mathit{K}& \frac{({\mathit{d}}_{\mathbf{3}}-\mathit{\psi }+\mathit{\zeta })}{\sqrt{2}}\\ \frac{{({\mathit{d}}_{\mathbf{3}}-\mathit{\psi }+\mathit{\zeta })}^T}{\sqrt{2}}& t-{\mathit{\psi }}^T{\mathit{e}}_{\mathbf{3}}\end{array}\right]\ge 0,\\ \mathit{\psi }\ge 0,\mathit{\zeta }\ge 0.\end{array}$$

So we have the following SDP problem:

$$\begin{array}{l}\underset{{\mathit{d}}_{\mathbf{3}},\mathit{\psi },\mathit{\zeta }}{min}t\\ \mathrm{s}.\mathrm{t}.\\ \left[\begin{array}{cc}\mathit{K}& \frac{({\mathit{d}}_{\mathbf{3}}-\mathit{\psi }+\mathit{\zeta })}{\sqrt{2}}\\ \frac{{({\mathit{d}}_{\mathbf{3}}-\mathit{\psi }+\mathit{\zeta })}^T}{\sqrt{2}}& t-{\mathit{\psi }}^T{\mathit{e}}_{\mathbf{3}}\end{array}\right]\ge 0,\\ \mathit{\psi }\ge 0,\mathit{\zeta }\ge 0.\end{array}$$

In practice, we found that adding a regularizer diag(I₁/C₁, I₂/C₂, I₃/C₂) to K will increase the positive definiteness of K and lead to better performance, where diag is a diagonal matrix and I₁, I₂, and I₃ are identify matrices having the same size as K^PNPN, K^UNUN, and K^UPUP, respectively.