Multiple Ordinal Regression by Maximizing the Sum of Margins

Abstract

Human preferences are usually measured using ordinal variables. A system whose goal is to estimate the preferences of humans and their underlying decision mechanisms requires to learn the ordering of any given sample set. We consider the solution of this ordinal regression problem using a Support Vector Machine algorithm. Specifically, the goal is to learn a set of classifiers with common direction vectors and different biases correctly separating the ordered classes. Current algorithms are either required to solve a quadratic optimization problem, which is computationally expensive, or are based on maximizing the minimum margin (i.e., a fixed margin strategy) between a set of hyperplanes, which biases the solution to the closest margin. Another drawback of these strategies is that they are limited to order the classes using a single ranking variable (e.g., perceived length). In this paper, we define a multiple ordinal regression algorithm based on maximizing the sum of the margins between every consecutive class with respect to one or more rankings (e.g., perceived length and weight). We provide derivations of an efficient, easy-to-implement iterative solution using a Sequential Minimal Optimization procedure. We demonstrate the accuracy of our solutions in several datasets. In addition, we provide a key application of our algorithms in estimating human subjects’ ordinal classification of attribute associations to object categories. We show that these ordinal associations perform better than the binary one typically employed in the literature.

I. Introduction

Ordinal measurements play a key role in many applications, from plant biology [41] to users’ reviews [43] and visual perception [45]. For instance, objects can be ordered according to their features, functionality, etc. A clear example is in the perception of facial expressions of emotion by humans. These facial expressions are generally ordered according to the degree of the perceived emotion category, regardless of the actual category being expressed by the sender [27], [9], [8]. Unlike a classification problem, here, errors generated by misclassifications reflect the difference between the orderings, and unlike a regression problem, the labels define the discrete class ranks [16], [6]. In our facial expression example, the perceived difference (i.e., distance) of an emotion category shown in images a and b is given by the difference between two ordinal variables [27], [23]. Ordinal measurements only provide the order of the classes relative to each other.

In the present paper, we derive a multiple ordinal regression algorithm, where two or more ranking (unknown) functions are used to order the data. Our goal is to simultaneously estimate these multiple underlying ranking functions to minimize the global ordinal classification error. Existing ordinal classification algorithms described in the literature are either computationally complex or based on maximizing the number of correct pairwise rankings which is sensitive to outliers [13]. To resolve these problems, in the present paper, we derive a Support Vector Machine (SVM)-based formulation defined by a simple iterative approach which minimizes the global risk. We first present the single ranking solution and then derive the algorithm for multiple ranking functions. This allows us to weight the importance of the ranking problems with respect to each other. Finally, we derive an efficient, easy-to-solve iterative solution using a Sequential Minimal Optimization (SMO) based iterative procedure.

Extensive experimental results show the accuracy of the algorithm through several tests in the UCI datasets. Importantly, we illustrate an application of our algorithm in learning the visual feature rankings of human subjects, where we learn to rank objects based on user-specified, high-level descriptions (attributes). This allows developing detection and recognition algorithms that are based on high-level attribute ranking estimations provided by our algorithm. We illustrate this scheme in Fig. 1. In the experimental results section, we also provide an application of this multiple ranking algorithm to data visualization.

Fig. 1.

Here we illustrate the use of multiple ranking functions in the classification of an unseen sample x. Attribute-class associations are predefined according to human subjects’ attribute preferences (whiteness and brownness on a scale from 1 to 4) for a set of animal classes. Our multiple ranking SVM is used to learn the function f(.) from a training set to estimate the multiple attribute rankings from samples. The attribute associations of the test sample x are estimated with the multiple ranking functions, i.e. f₁(x) and f₂(x). These rankings are used to assign the sample to one of the unseen testing classes.

A. Background and significance

Several algorithms have been proposed to resolve the ordinal regression problem [26], [47], [12], [22], [1], [6], [38], [18], [13], [20], [17]. For example in [20] the authors learn a regression tree to estimate the ordinal values. The limitations of this approach are that the metric between the ordinal variables is not defined and that the regression algorithm assumes a uniform error.

To resolve the above stated problems, Shashua et al. [38] derive SVM-based ranking algorithms to address the single ranking problem. The goal of their algorithms is to learn a set of hyperplanes with common weight vector and different biases to order a set of ordinal classes. Shashua et al. present two possible solutions to this problem – a fixed margin strategy and a sum of margins strategy. Fig. 2 shows a typical solution of these two strategies. The fixed margin strategy maximizes the distance of the closest class pair. This solution is prone to errors, since the closest class pair may not define a correct direction of separation for all classes. In contrast, the second strategy maximizes the sum of the margins between every consecutive class while minimizing the global risk. Unfortunately, the formulation of [38] leads to several inequalities that are very difficult to solve and are inefficient. Both of these algorithms are in fact formulated as quadratic optimization problems and, hence, the number of samples that can be utilized is limited.

Fig. 2.

Different learning strategies result in distinct rankings. (a) Shows the fixed margin strategy, where the margin is defined by the closest classes. In this case the goal is to find the two hyperplanes with the common direction vector w and biases b₁ and b₂ that maximize the minimum margin. Our sum of margins strategy shown in (b) extracts the direction w that maximizes the sums of margins ${\sum }_{i=1}^2(b_i-a_i)$ with the four hyperplane biases of a₁, b₁, a₂, and b₂. Different colors correspond to classification regions defined by the ranking rule. The solution of fixed margin strategy is biased by the closest class pairs of 1 and 2, and the sum of margin strategy obtains a larger margin by maximizing the sum of margins.

Chu et.al. [6] address the above limitations by reformulating the fixed margin approach as a SMO¹ (Sequential Minimal Optimization). In addition, [6] extended the fixed margin solution to handle the implicit errors that would be generated when a sample is misranked further than two classes away from its correct class rank. Unfortunately, these solutions were based on the fixed margin strategy and are hence biased towards the classification direction given by the closest class pairs.

Other types of ranking formulations have been commonly used in the information retrieval community for ranking document retrieval results [17], [18], [1], [22]. In these cases, the ranking SVM problem is formulated as to find a solution that maximizes the number of correctly ordered pairwise samples. Note that, this problem is different than our ordinal classification learning or ordinal regression problem. While we are learning a function to estimate an ordinal variable, these approaches target to maximize correctly ordered pairwise samples. Considering all pairwise samples, this results in a large quadratic optimization with complexity squared with the number of samples. To address this issue, several efficient implementations of ranking SVM with a fixed margin strategy have been proposed [19], [4], [3], [49]. Boosting approaches defined in the ”Yahoo! Learning to Rank Challenge” [3], such as GBRANK [49], use a similar formulation to ranking SVM. Efficient ranking learning with primal optimization techniques have been defined in [36]. A recent approach by [7] derives ranking forests to address learning to rank problem with binary feedbacks. Again, all these approaches are defined for the information retrieval applications and target to learn a ranking SVM that maximize the correct pairwise orderings and are hence based on a fixed margin strategy, with the subsequent limitations already stated above.

In a recent study, [44] provides a detailed theoretical comparison between various ranking algorithms. It is shown that most of these approaches are related to the proportional log odds model in statistics [25]. Specifically, the fixed-margin strategy based solution of [6] with the implicit constraints on individual samples have the same asymptotic ranking function as the proportional log odds model. However, [44] points out that the dependency of this solution to the implicit constraints of individual sample rankings may result in poor performance in some applications. This provides a theoretical support for our observations on the inferior performance of the fixed margin strategy. [44] also emphasizes that the objective functions for ordinal regression and multiclass classification are significantly different, and multiclass classification would result in large errors if used in an ordinal regression problem. This is also reflected in our experimental results in Section IV. These methods are also to be compared to recent algorithms that search for a balance between the bias and variance of the regressors [47]. The major problem of this approach is its limitations when working with multiple ordinal variables.

There has also been a great interest on the theoretical properties of the learning to rank approaches from partial pairwise rankings. [10] provides a detailed review of the learning to rank algorithms and summarizes that the practical solutions to collect ranking information don’t have a relevant theory and the approaches that have theoretical support don’t have practical solutions. To address these limitations, [10] proposes a theory to identify partial preferences when the dataset is only partially labeled. [33] also provides a solution for the rank aggregation from pairwise data and derives theoretical results for the underlying statistical model. Note that, we provide the review of these recent approaches to clarify that the ordinal regression problem we are addressing in this paper is a different problem from the approaches that are designed for learning to rank problem.

Other recent ordinal regression approaches provide extensions for ensemble learning [29], [11], sampling problems [30], and semi-supervised learning [37]. [29] proposes an ensemble learning for ordinal regression by extracting multiple projection-based two class classifiers and three class ordinal regressors. The ensemble of the probability scores obtained from these functions are used to rank the test samples. [11] addresses the feature selection for ensemble learning of ordinal regressors. Negative correlation is used to find new features that provide additional information to the ensemble. Both of these ensemble solutions are shown to improve the accuracy of fixed margin strategy based solutions [6], Gaussian Processes Ordinal Regression [6] and SVM. [30] addresses the sampling problems in ordinal regression using a graph-based approach. Ranking classes that have large number of samples usually dominate the regression and results in misranking of classes when the number of samples is small. This is addressed by a graph-based approach to properly adjust the regressor weights and is shown to improve the accuracy of fixed margin solutions. [37] proposes a semi-supervised approach of Transductive Ordinal Regression. Unlabeled samples are utilized while learning the regressor. A training algorithm is proposed to estimate the labels of the unlabeled samples and minimize the loss function for the samples with ranking labels. This approach could be utilized when the data labels are missing and it is difficult to collect the rankings for all classes. These extensions are all proposed for ordinal regression problems and can utilize our sum-of-margin based multiple and single ordinal regression solutions proposed in the present paper. This increases the applicability of our solutions to a larger set of problems.

II. Single Ranking Learning with SVM

Assume we are given a set of samples ${\mathbf{x}}_i^j\in {ℝ}^d$, where i = {1, 2, …, n_j} specifies the sample, n_j is the total number of samples in class j, and the superscript j = {1, 2, …, R} is the class label which specifies the order of the sample.

Our goal is to learn a ranking function that estimates the ordering of a future (test) observation. Let this function be represented by a set of parallel hyperplanes, with w as the common direction vector. Let each class lie between two of the resulting R − 1 hyperplanes with biases b₁, b₂, …, b_R−1. We classify a test sample x to (class) rank r using the rule,

$$f(\mathbf{x})=\underset{r\in 1,2,\dots ,R}{min}r:{\mathbf{w}}^T\mathbf{x}<b_r,$$ (1)

Note that, without loss of generality, the rank R hyperplane is set at infinity with b_R = ∞.

Fig. 2(b) illustrates the geometric interpretation of the ranking problem. In this example, our goal is to estimate the ranking function that maximizes the separation of the samples in 3 classes (shown as circles, plus signs and squares in the figure) with 4 parallel hyperplanes. The vector w illustrates the common weight vector for the hyperplanes with biases a₁, b₁, a₂ and b₂. The margins between consecutive classes are defined as b_i − a_i. The averages of the a_i and b_i are used to define the classification rule, i.e., $c_i=\frac{a_i+b_i}{2}$. The classifier w^Tx ≤ c₁ specifies the ordering obtained by the first function, c₁ < w^Tx ≤ c₂ given by the second rank, and w^Tx > c₂ of the third. For this problem, our ranking function is f(x) = min_r=1,…,R w^Tx < c_r, with c_R = ∞ and R = 3. Note that, the hyperplanes w^Tx = a₁ and w^Tx = a₂ penalize the misranking errors of samples from classes 1 and 2 to classes with higher ranks, e.g., for samples from class 1 these are classes 2 and 3. Similarly, w^Tx = b₁ and w^Tx = b₂ penalize misorderings of samples from classes 2 and 3 to classes with lower rank. We now derive this geometric problem with a SVM type ranking function learning.

Formally, we have

$$\begin{array}{l}minF\left(\mathbf{w},{\epsilon }_i^j,{\epsilon }_i^{\ast j+1},a_i,b_i\mid C\right)=\frac{1}{2}{\Vert \mathbf{w}\Vert }^2+\\ C\sum _{j=1}^{R-1}\left(\sum _{i=1}^{n_j}{\epsilon }_i^j+\sum _{i=1}^{n_{j+1}}{\epsilon }_i^{\ast j+1}\right)+\sum _{j=1}^{R-1}(a_j-b_j)\\ \text{such}\text{that}\{\begin{array}{l}{\mathbf{w}}^T{\mathbf{x}}_i^j-a_j\le {\epsilon }_i^j,-{\mathbf{w}}^T{\mathbf{x}}_i^{j+1}+b_j\le {\epsilon }_i^{\ast j+1},\hfill \\ {\epsilon }_i^j\ge 0,{\epsilon }_i^{\ast j+1}\ge 0,\hfill \end{array}\\ \text{for}j=\{1,\dots ,R-1\},i=\{1,\dots ,n_j\},\end{array}$$ (2)

where ${\epsilon }_i^j$ and ${\epsilon }_i^{\ast j+1}$ are the slack variables that correspond to errors in the orderings if the samples ${\mathbf{x}}_i^j$ and ${\mathbf{x}}_i^{j+1}$ with respect to the hyperplanes with biases a_j and b_j, and where n_j is the number of samples in class j.

The first term in the optimization problem above corresponds to the complexity of the classifier and the common scale of the margins. The second term is the sum of errors generated by the hyperplanes, and the third term is the sum of margins between consecutive hyperplanes. The parameter C controls the importance of the error term with respect to the margin. The inequality conditions define the classification rule and the positivity of the slack variables ε.

The Lagrangian for the primal problem in (2) is given by,

$$\begin{array}{l}L=\frac{1}{2}{\Vert \mathbf{w}\Vert }^2+C\left(\sum _{i,j}{\epsilon }_i^j+\sum _{i,j}{\epsilon }_i^{\ast j+1}\right)\\ +\sum _{j=1}^{R-1}(a_j-b_j)+\sum _{i,j}{\lambda }_i^j\left({\mathbf{w}}^T{\mathbf{x}}_i^j-a_j-{\epsilon }_i^j\right)\\ +\sum _{i,j}{\delta }_i^j\left(-{\mathbf{w}}^T{\mathbf{x}}_i^{j+1}+b_j-{\epsilon }_i^{\ast j+1}\right)\\ -\sum _{i,j}{\gamma }_i^j{\epsilon }_i^j-\sum _{i,j}{\gamma }_i^{\ast j+1}{\epsilon }_i^{\ast j+1}\end{array}$$ (3)

where ${\lambda }_i^j,{\delta }_i^j,{\gamma }_i^j,{\gamma }_i^{\ast j+1}$ are the nonnegative Lagrange multipliers.

The KKT (Karush-Kuhn-Tucker) optimality conditions for the above problem are,

$$\frac{\partial L}{\partial \mathbf{w}}=\mathbf{w}+\sum _{i,j}{\lambda }_i^j{\mathbf{x}}_i^j-\sum _{i,j}{\delta }_i^j{\mathbf{x}}_i^{j+1}=0,$$ (4)

$$\frac{\partial L}{\partial a_j}=-\sum _i{\lambda }_i^j+1=0,\frac{\partial L}{\partial b_j}=\sum _i{\delta }_i^j-1=0,$$ (5)

$$\frac{\partial L}{\partial {\epsilon }_i^j}=-{\gamma }_i^j-{\lambda }_i^j+C=0,$$ (6)

$$\frac{\partial L}{\partial {\epsilon }_i^{\ast j+1}}=-{\gamma }_i^{\ast j+1}-{\delta }_i^j+C=0,$$ (7)

$$\begin{array}{r}{\lambda }_i^j\ge 0,{\delta }_i^j\ge 0,{\gamma }_i^j\ge 0,{\gamma }_i^{\ast j+1}\ge 0,\\ {\lambda }_i^j\left({\mathbf{w}}^T{\mathbf{x}}_i^j-a_j-{\epsilon }_i^j\right)=0,\\ {\delta }_i^j\left(-{\mathbf{w}}^T{\mathbf{x}}_i^{j+1}+b_j-{\epsilon }_i^{\ast j+1}\right)=0,\\ {\gamma }_i^j{\epsilon }_i^j=0,{\gamma }_i^{\ast j+1}{\epsilon }_i^{\ast j+1}=0,\forall i,j.\end{array}$$ (8)

We rewrite the primal problem such that the terms that cancel each other are closer to each other,

$$L=\frac{1}{2}{\Vert \mathbf{w}\Vert }^2+{\mathbf{w}}^T\left(\sum _{i,j}{\lambda }_i^j{\mathbf{x}}_i^j-{\delta }_i^j{\mathbf{x}}_i^{j+1}\right)$$ (9)

$$-\sum _{i,j}({\lambda }_i^j+{\gamma }_i^j){\epsilon }_i^j-\sum _{i,j}({\delta }_i^j+{\gamma }_i^{\ast j+1}){\epsilon }_i^{\ast j+1}$$ (10)

$$+C\left(\sum _{i,j}{\epsilon }_i^j+\sum _{i,j}{\epsilon }_i^{\ast j+1}\right)$$ (11)

$$+\sum _{j=1}^{R-1}(a_j-b_j)-\sum _{i,j}{\lambda }_i^j{a}_j+\sum _{i,j}{\delta }_i^j{b}_j.$$ (12)

Eq. 4 is applied to 9 and results to $-\frac{1}{2}{\Vert \mathbf{w}\Vert }^2$, Eqs. in 10 and 11 cancel each other when KKT conditions in Eqs. 6 and 7 are used, and the equality conditions in 5 are used to cancel the terms in 12. Conditions in Eqs. 6,7 and 8 results to, $0\le {\lambda }_i^j\le C$ and $0\le {\delta }_i^j\le C$. And Eqs. in 5 are used to obtain the sum to one conditions of ${\sum }_i{\lambda }_i^j=1,{\sum }_i{\delta }_i^j=1$.

To simplify notation, we rewrite w as w = Qμ, where Q = [−X_1… − X_R−1X_2…X_R], ${\mathbf{X}}_j=\{{\mathbf{x}}_1^j,\dots ,{\mathbf{x}}_{n_j}^j\}$ are the samples of class j, and $\mu =[{\mu }_1^1,\dots {\mu }_{R-1}^1{\mu }_1^2,\dots ,{\mu }_{R-1}^2]$ are all of the Lagrange multipliers, with ${\mu }_j^1=\{{\lambda }_1^j,\dots ,{\lambda }_{n_j}^j\}$, and ${\mu }_j^2=\{{\delta }_1^j,\dots ,{\delta }_{n_j}^j\}$ are the Lagrange multipliers for samples from classes j and j + 1 penalized by hyperplanes with biases a_j and b_j, respectively.

Importantly using these equalities in (3) and the KKT conditions results in the dual problem,

$$\begin{array}{c}maxW(\mu )=-{\mu }^T{\mathbf{Q}}^T\mathbf{Q}\mu 0\le {\mu }_i\le C,i=1,\dots ,N_L\\ {\mathbf{1}}^T{\mu }_j^1=1,{\mathbf{1}}^T{\mu }_j^2=1j=1,\dots ,R-1,\end{array}$$ (13)

where N_L = 2N − n₁ − n_r is the total number of Lagrange multipliers. Eq. (13) is a simple quadratic optimization problem. Unlike the algorithm proposed by [38]. This sum of margins maximizing algorithm only includes (additional) equality conditions.

The parameters

The parameter C in the above formulations control the importance of minimizing the training error versus maximizing the margin. Specifically, the amount of training error is controlled by the two conditions on the Lagrange multipliers in the dual problem, Eq. (13), i.e., the upper-bound C and the sum-to-one conditions. For instance, if $C=1/n_j^i$, where $n_j^i$ are the number of samples in a group of Lagrange multipliers from one of i = 1, 2, and j = 1, …, R − 1, then all the samples in this group will have a Lagrange multiplier equal to C to satisfy the sum-to-one condition ${\mathbf{1}}^T{\mu }_j^i=1$. This means that, all the corresponding training samples in this group will be incorrectly ordered with respect to the corresponding ranking function. On the other hand, if C is scaled to a larger value, $C=1/(\kappa n_j^i)$ with a scalar κ in 0 < κ < 1, then at most κ fractions of the Lagrange multipliers will be nonzero. Hence, we use a parameter 0 < κ ≤ 1 to control the parameters $C_j^i$ with respect to the number of samples in the corresponding group of multipliers, i.e., $C_j^i=\frac{1}{\kappa n_j^i}$, for each group of Lagrange multipliers. Herein, we keep a global parameter C for simplicity.

Comparison with the literature

The fixed margin strategy is defined as,

$$\begin{array}{l}\underset{\{\mathbf{w},{\epsilon }_i^j,{\epsilon }_i^{\ast j+1},b_i\}}{min}\frac{1}{2}{\Vert \mathbf{w}\Vert }^2+C\sum _{j=1}^{R-1}\left(\sum _{i=1}^{n_j}{\epsilon }_i^j+\sum _{i=1}^{n_{j+1}}{\epsilon }_i^{\ast j+1}\right)\\ \text{such}\text{that}\{\begin{array}{l}{\mathbf{w}}^T{\mathbf{x}}_i^j-b_j\le -1+{\epsilon }_i^j,\hfill \\ {\mathbf{w}}^T{\mathbf{x}}_i^{j+1}-b_j\ge 1-{\epsilon }_i^{\ast j+1},\hfill \\ {\epsilon }_i^j\ge 0,{\epsilon }_i^{\ast j+1}\ge 0,\forall i,j\hfill \end{array}\end{array}$$ (14)

where b_j are the biases of the hyperplanes separating the classes [38].

The main advantage of the fixed margin strategy is the simplicity of its dual problem. However, note that, in this case, the margin 1/||w|| is biased towards those two classes that are closest to each other. This is a typical problem in many machine learning algorithms [48], [15], [24], yielding suboptimal, biased solutions. Similarly, [6] derives an implicitly constrained fixed margin problem to improve on the accuracy of the fixed-margin strategy. Here, the errors from all samples are considered in the computation of every hyperplane, as opposed to just using the samples from neighboring classes as in the above. The limitation of the formulation is that it minimizes the error between neighboring classes, rather than the global error measure, which also results in biased solutions [15].

As an alternative to the fixed margin, the sum of margins strategy maximizes the margins between every neighboring class. More formally, [38] defined one possible objective as,

$$\begin{array}{l}\underset{\{\mathbf{w},{\epsilon }_i^j,{\epsilon }_i^{\ast j+1},a_i,b_i\}}{min}\sum _{j=1}^{R-1}(a_j-b_j)+C\sum _{j=1}^{R-1}\left(\sum _{i=1}^{n_j}{\epsilon }_i^j+\sum _{i=1}^{n_{j+1}}{\epsilon }_i^{\ast j+1}\right)\\ \text{such}\text{that}\{\begin{array}{l}{a}_j\le b_j,b_j\le a_{j+1},\hfill \\ {\mathbf{w}}^T{\mathbf{x}}_i^j-a_j\le {\epsilon }_i^j,{\mathbf{w}}^T{\mathbf{x}}_i^{j+1}-b_j\ge -{\epsilon }_i^{\ast j+1}\hfill \\ {\Vert \mathbf{w}\Vert }^2\le 1,{\epsilon }_i^j\ge 0,{\epsilon }_i^{\ast j+1}\ge 0,\forall i,j.\hfill \end{array}\end{array}$$ (15)

When compared to the objective function of the fixed margin strategy in Eq. 14, the above formulation can control the margin weights separately with the ${\sum }_{j=1}^{R-1}(a_j-b_j)$ term. However, it is missing the ||w||² term that corresponds to the global scale of the margins.

Furthermore, the dual formulation of Eq. 15 results in a quadratic optimization problem with several inequalities. As a consequence, the solution of the problem with an iterative procedure becomes computationally expensive because of the need to keep track of a large number of inequalities at every iteration.

On the other hand, the dual problem in our formulation is simple and can be readily solved using an unsophisticated iterative approach. Furthermore, our solution considers maximizing the margin between every consecutive class and the global scale defined by 1/||w||. In summary, we have eliminated the drawbacks of the two previous approaches while keeping their advantages.

III. Multiple Ranking SVM (MRSVM)

We now present the extension of the above approach to multiple ranking learning and derive a computationally efficient algorithm.

A. General formulation

Learning multiple ranking functions is done by solving for each of the ranking problems simultaneously. Formally, this requires us to extend (2) as follows,

$$minF\left({\mathbf{w}}_1,{\epsilon }_{i_1}^{j_1},{\epsilon }_{i_1}^{\ast j_1+1},a_{j_1},b_{j_1}\mid C_1\right)+F\left({\mathbf{w}}_2,{\epsilon }_{i_2}^{j_2},{\epsilon }_{i_2}^{\ast j_2+1},a_{j_2},b_{j_2}\mid C_2\right).$$

Note that, an important advantage of this extension is to be able to set the relative importance of the ranking problems by adjusting the weight factors C₁ and C₂ in a single problem. This allows us to determine a solution that balances the error between the two rankings. Furthermore, it may find a more sparse solution than solving two ranking problems separately, since the support vectors can be shared between the two. In the experimental results section, we show that the simultaneous solution performs better than separate solutions would.

B. SMO solution

The Sequential Minimal Optimization (SMO) algorithm is proposed as a fast, efficient, yet simple solution to the dual problem in SVM [31]. As the kernel matrix size is too large for problems with large number of samples, it becomes impractical to solve the SVM quadratic optimization. SMO handles this by iteratively updating the solution based on two samples (selected by heuristics) and a closed-form solution at each step. After a certain number of iterations, the optimal solution can be obtained. Our SMO-based formulation results in linear and quadratic times. Additionally, the memory used by SMO scales linearly with the number of training samples.

For the ranking solution derived in the previous section, the size of the quadratic problem in (13), scales twice as fast as the number of samples. Fortunately, we show that a SMO iterative procedure does not suffer from this limitation. Our solution is detailed in the following.

We rewrite (13) by isolating two of the Lagrange multipliers for the same class j concerning the same hyperplane i. Let these multipliers be ${\mu }_1={\mu }_{j_1}^i$ and ${\mu }_2={\mu }_{j_2}^i$. Then,

$$\underset{{\mu }_1,{\mu }_2}{min}-W=\sum _{k_1,k_2=1}^2{\mu }_{k_1}{\mu }_{k_2}{\mathbf{K}}_{k_1{k}_2}+2\sum _{k_1=1}^2{\mu }_{k_1}{\mathbf{z}}_{k_1}+z,$$ (16)

where K = Q^TQ, ${\mathbf{z}}_{k_1}={\sum }_{k_2=3}^{N_L}{\mu }_{k_2}{\mathbf{K}}_{k_1{k}_2}$, and $z={\sum }_{k_1,k_2=3}^{N_L}{\mu }_{k_1}{\mu }_{k_2}{\mathbf{K}}_{k_1{k}_2}$. From the constraints in (13),

$$0\le {\mu }_1,{\mu }_2\le C,{\mu }_1+{\mu }_2=D,$$

where $D=1-{\sum }_{k_1=3}^{n_j}{\mu }_{j_{k_1}}^i$. Now we insert μ₁ = D − μ₂ to Eq. (16), and set the derivative of W with respect to μ₂ to zero, we have $\frac{\partial W}{\partial {\mu }_2}=-2(D-{\mu }_2){\mathbf{K}}_{11}+2(D-2{\mu }_2){\mathbf{K}}_{12}+2{\mu }_2{\mathbf{K}}_{22}-2{\mathbf{z}}_1+2{\mathbf{z}}_2=0$. From this we can obtain the updated Lagrange multiplier as,

$${\mu }_2^{\mathit{new}}=\frac{D({\mathbf{K}}_{11}-{\mathbf{K}}_{12})+{\mathbf{z}}_1-{\mathbf{z}}_2}{{\mathbf{K}}_{11}+{\mathbf{K}}_{22}-2{\mathbf{K}}_{12}}.$$

For clarity, the above equation can be rewritten as

$${\mu }_2^{\mathit{new}}={\mu }_2+\frac{u_1-u_2}{{\mathbf{K}}_{11}+{\mathbf{K}}_{22}-2{\mathbf{K}}_{12}},$$ (17)

where u_k = K_1kμ₁ + K_2kμ₂ + z_k is the projection of the sample to the weighted direction w, i.e., $u_k={\mathbf{w}}^T{\mathbf{x}}_k^{j_2}$ for i = 1, and $u_k=-{\mathbf{w}}^T{\mathbf{x}}_k^{j_2+1}$ for i = 2, where i = {1, 2} indicates the Lagrange multipliers for samples that are penalized by the corresponding hyperplane. These indices are previously reflected in Eq. 13 and its derivations as the superscripts of ${\mu }_j^i$.

Next, we show that the updated multiplier satisfies the boundary conditions defined in (13). These conditions are given by the minimum and maximum possible solutions defined by the equality condition ${\sum }_{k=1}^{n_j}{\mu }_{j_k}^i=1$, i.e. μ₁ + μ₂ = D, within the feasible region 0 ≤ μ₁, μ₂ ≤ C. These are the lower- and upper-bounds, L = max(0, D − C) and H = min(C, D), respectively.

Following these derivations, we update ${\mu }_2^{\mathit{new}}$ to the closest point in the solution space. That is,

$${\mu }_2^{\mathit{new}}=\{\begin{array}{lll}L\hfill & \text{if}\hfill & {\mu }_2^{\mathit{new}}<L\hfill \\ {\mu }_2^{\mathit{new}}\hfill & \text{if}\hfill & L\le {\mu }_2^{\mathit{new}}\le H\hfill \\ H\hfill & \text{if}\hfill & {\mu }_2^{\mathit{new}}>H.\hfill \end{array}$$ (18)

${\mu }_1^{\mathit{new}}$ is given by the equality condition, ${\mu }_1^{\mathit{new}}=D-{\mu }_2^{\mathit{new}}$. After each iteration, the bias of the hyperplanes are updated if the multipliers satisfy $0<{\mu }_k^{\mathit{new}}<C$, i.e., $a_j={\mathbf{w}}^T{\mathbf{x}}_k^j$, for = 1, and $b_j={\mathbf{w}}^T{\mathbf{x}}_k^{j+1}$, for i = 2.

The main difference of the above derived SMO algorithm and the original SMO specifically designed for the two-class SVM classifier [31] or the novelty detection [35] or for estimating the support of a high-dimensional distribution [34] is the selection of the two Lagrange multipliers. Because of the sum-to-one condition for the Lagrange multipliers, the two multipliers should be selected from the group of samples from the same class rank and is penalized by the same hyperplane, i.e., the groups defined by ${\mu }_j^i$. This group-based multiplier selection property allows us to readily extend the SMO for single ranking SVM to that of multiple ordering SVM by simply considering a group from one of the ordering problems at each iteration. The group and the Lagrange multipliers to be optimized at each iteration are selected using the same heuristics as in SMO for SVM.

The derived approach is summarized in Algorithm 1.

Algorithm 1.

SMO solution for MRSVM

do

Find a variable μ2 that violates the KKT optimality conditions.

Search for the second variable μ1 within the same ranking problem (d = {1, 2}) and within the same group ${\mu }_j^i$ of μ2.

Select the second variable that maximize the step size (u1− u2 in (17)) and calculate the local minima for ${\mu }_2^{\mathit{new}}$ using (17).

Update ${\mu }_2^{\mathit{new}}$ to the closest point in the solution space using (18).

Calculate ${\mu }_1^{\mathit{new}}=D-{\mu }_2^{\mathit{new}}$

If $0<{\mu }_j^{\mathit{new}}<C_d$, update the corresponding hyperplane biases $a_j={\mathbf{w}}^T{\mathbf{x}}_k^j$ for i = 1 and $b_j={\mathbf{w}}^T{\mathbf{x}}_k^{j+1}$ for i = 2.

until all variables satisfy KKT optimality conditions

IV. Experiments

In this section, we provide experiments illustrating the efficiency and accuracy of the proposed algorithm. We compare the accuracy of the orderings given by the derived Multiple Ranking SVM (MRSVM) and Multiple Single Ranking SVM (MSRSVM) and compared them to those given by Support Vector Regression (SVR) [39], [42] and ranking SVM (SVM_rank) [19] algorithms using standard UCI datasets. SVM_rank provides state-of-the-art optimized implementation of fixed margin strategy based solutions.

Two applications of interest of the derived algorithm are in attribute-based object recognition and data visualization. These are given at the end of this section.

A. Computational Comparison

We illustrate the computational efficiency of Algorithm 1 using the example ranking problem shown in Fig. 2. The 3-class ranking problem is formulated as the quadratic optimization problem in (13) and the solutions are obtained by our MATLAB implementation of Algorithm 1 and the standard MATLAB implementation of the Quadratic Programming (QP) solver, both in a PC with 2.2GHz CPU. The QP solver is based on the active-set strategy [14]. Samples from 3 classes are obtained from 3 Gaussian distributions with means at (−10, 0) (5, 0) and (30, 0) with covariance matrices equal to 2I. We run 10-fold cross-validation experiments with {10, 20, …, 100} samples per class, which results in {27, 54, …, 270} training samples for each validation. The same solutions are obtained with both algorithms in all of the cross-validation tests, while showing different computational times as illustrated in Fig. 3. While the time complexity of the standard QP solver scales polynomially with the number of samples, Algorithm 1 does not show a significant change. This is because, Algorithm 1 is an iterative SMO-type approach where its convergence significantly depends on the complexity of the learning problem. If the ranking classes are well separated from each other the convergence may take only a few iterations. A problem that requires a highly nonlinear solution may have a computational complexity as much as a quadratic problem. The time complexity of our solution scales between linear and quadratic with respect to the dataset size. Complexity of the ranking problem defines the average time complexity. Since the difficulty of the ranking problem does not increase by adding samples to the training set, the convergence rate and computational expense of our solution stays the same.

Fig. 3.

Computational times required to solve the quadratic optimization problem in (13) for the 3-class ranking problem illustrated in Fig. 2. Algorithm 1 is compared to a standard MATLAB Quadratic Programming (QP) solver with {27, 54,…, 270} training samples in each of the 10-fold cross validation experiments. Average and standard deviations of the computational times are plotted.

TABLE I.

A comparison between the margins obtained using different approaches. The values in the parenthesis (a,b) mean: fixed margin strategy in Eq. 14 for (a) and MSRSVM algorithm for (b). A set of ranking problems are tested by varying the location of class 3 according to r and θ as shown in Fig. 4.

θ/r	15	20	25	30
0	(10.06, 11.39)	(12.88, 17.91)	(12.52, 18.28)	(11.29, 26.96)
π/8	(10.24, 12.45)	(14.44, 18.92)	(11.29, 21.39)	(12.86, 25.79)
π/4	(9.17, 9.72)	(11.36, 12.03)	(12.24, 26.5)	(11.72, 25.21)

B. Fixed Margin Strategy vs. MSRSVM

The fixed margin strategy objective formulated in Eq. 14 mainly focuses on maximizing the margin between the closest class pairs. This results in solutions that are driven by the closest classes, which means other classes do not contribute much in computing the solution. MSRSVM algorithm solves Eq. 2 and maximizes the sum of margins. It properly assigns the same importance to maximizing the margin between each class pairs.

We illustrate these two solutions to ordinal regression in a 3-class ranking experiment. For simplicity, we designed the experiment for linearly separable ranking problems. 50 instances are sampled from each of 3 Gaussian distributions with means at (−15, 0), (0, 0) and r(COS(θ), sin(θ)) where r = {15, 20, 25, 30} is the radius and θ = {0, π/8, π/4} is the angular degrees. Fig. 4 illustrates the configuration of these three-class ranking problem.

Fig. 4.

Here we illustrate the set of ranking problems that are used to compare the fixed margin strategy with MSRSVM. Classes are sampled from 3 Gaussian distributions of N(μ_I, 4I), where μ₁ = (−15, 0)^T, μ₂ = (0, 0)^T, and the covariance matrices with scaled identity matrices of 4I. Class 3 is simulated to be at multiple locations of μ₃ = r(cos(θ), sin(θ))^T, for r = {15, 20, 25, 30} and θ = {0, π/8, π/4}. The circles correspond to Gaussian distributions, with 4I as covariance matrices. Classes 1 and 2 are fixed and represented with circles of solid lines. Class 3 is set to each of the locations (r, θ), which are represented with dashed circles.

We solve these ranking problems using Eq. 14 and the derived MSRSVM algorithm, and calculate the total margin obtained by the solutions. This is done for ten random sample sets for each configuration of r and θ and average margins are calculated. With a fixed margin, the margin size is given by 4/||w||, while the sum of margins is calculated with Σ_i(b_i − a_i)/||w||. Note that, the size of the margin controls the upper bound of the VC dimension [46]. The larger the margin the smaller the upper bound, which means the solution has smaller generalization error, i.e. the solution with larger margin is expected to yield a smaller classification error when testing previously unseen feature vectors.

Table I compares the averaged margins that are obtained with the fixed margin approach and that derived in the present paper. As seen in this figure, for all the tested cases, the margin size obtained by fixed-margin strategy is around 10, which is approximately the margin between the closest class pairs. When class 3 moves away from class 2 the margins obtained by our solution is larger than the fixed margin strategy. This is pretty important when class 3 is rotated to a vertical location, where the 1-dimensional ranking function that has the largest margin is not aligned with the horizontal line. This was illustrated in Fig. 2, where the fixed margin is still tuned to the closest class pairs and obtains a solution that is biased by this class pair and neglects the separability between classes 2 and 3. On the other hand, MSRSVM considers the margin between each consecutive rank pair and obtains a solution that has significantly larger margins than the previously discussed solutions.

C. UCI datasets

We provide a comparison with three datasets typically employed in regression problems. These are from the UCI repository². For each dataset, multiple rankings are provided for two output variables. These ordinal variables are obtained by binning the continuous variables to equal frequencies, i.e., the number of samples for each class are the same. Specifically, in California housing medianIncome and medianHouseValue features, in Boston rm features that stands for the average number of rooms per dwelling and housing values of class features, and for auto-mpg cylinders and the mpg values in class features are converted to ordinal variables. In our experiments, we have used the radial basis kernel function. The parameters of the optimization problem C₁ and C₂ are selected from the set of {0.1, 1, 10, 100} using 5-fold cross-validation within the training set, both for the ordinal learning and regression problems.

Table II summarizes the dimensionality of the datasets, number of samples in training/testing, and average absolute error for each of the algorithms. The average error rates are obtained from 10 random partitions of the dataset into training and testing.

TABLE II.

Shown here are the mean absolute errors $\frac{1}{N}{\sum }_{i=1}^N\Vert r_i-{\widehat{r}}_i\Vert$ between the ground truth ranking of r_i and the estimated ranking of r̂_i using MRSVM, MSRSVM, SVR, and ranking SVM based on pairwise orders (SVM_rank). We also provide the training and testing partitions, the dimensionality (d) of the dataset, the labels of the database features, the number of ranking classes (R) and the ranking results (mean absolute errors) for individual dimensions and their average.

Dataset	train/test (d)	feature (R)	Methods
			MRSVM	MSRSVM	SVR	SVMrank
California Housing	1000/19640 (7)	medianIncome (4)	0.81	0.76	1.2	1.15
		medianHouseValue (4)	0.83	0.83	1.03	1.15
		avg	0.82	0.79	1.12	1.15
Boston	300/206 (12)	rm (4)	0.84	0.84	1.14	1.21,
		class (4)	0.48	0.51	0.88	1.05
		avg	0.66	0.68	1.01	1.13
Auto-mpg	192/200 (6)	cylinders (3)	0.08	0.09	0.25	0.33
		class (5)	0.41	0.50	0.68	1.19
		avg	0.24	0.29	0.46	0.76

In addition, we compared our results to SVM_rank [19], which has been commonly used in information retrieval. As we summarized in the introduction, this ranking algorithm considers the pairwise ordered samples. Hence, to run this algorithm, we convert our samples and the associated ordinal variables to a proper representation. A training sample with a ranking of r_i is represented with respect to its ranking relations to its 10 nearest neighbors, i.e., x_i^j with rankings of r_i^j for j = {1,…, 10}. This results in 10 pairwise preference constraints with relevance scores between the pairwise samples defined by the absolute values of the difference between the ranking labels |r_i − r_i^j|. The smaller this score is, the more similar the rankings of the pairwise samples are. A ranking function is learned to accurately sort these samples and their relation to each other. The parameters of the algorithm are selected with a 5-fold cross-validation test. The ranking of a testing sample is obtained from the rank of the training sample that has the closest ranking score to the testing sample. The mean absolute errors obtained with this approach are shown in Table II. SVM_rank performs worse than the proposed algorithms, since it is mostly concerned with correctly ranking individual samples, while our ranking algorithms attempt to maximize the correct ordering of classes.

D. Detecting Unseen Object Classes

A key feature that significantly differentiates the human visual system from machine vision is the generalization capability to unseen object classes. Towards this goal, Lampert et.al. [21] recently proposed attribute-based object classification. Objects are described with semantic attributes (i.e. high-level object descriptions) that are common across classes, such as physical properties of animals. Lampert et. al. propose to use binary SVM classifiers to estimate the existence of an attribute in a query object. However, the binary representation of class-attribute associations is usually insufficient, since one needs to describe an attribute with more than two levels, e.g., to consider anchovy, tuna and whale, we need three levels (small, medium, and large) to discriminate them. In addition, these high level descriptions are defined by human preferences, which require ordinal classification algorithms. This is because, the metric underlying human preferences is not known, while ranking of the attribute class associations can be accurately measured. [28] applied a formulation of the ranking SVM [18] to estimate the relative orders of the attributes. However, as summarized before, ranking SVM is not derived to estimate the ordinal variables. This was illustrated by the SVM_rank algorithm in the previous section.

Direct Attribute Prediction

As a baseline, we compare our algorithm with the Direct Attribute Prediction (DAP) algorithm proposed in [21]. DAP assumes that a set of binary high-level attributes are provided for a set of training and testing classes, i.e., binary class-attribute associations are known. The relation between the high-level attributes and low-level image features are learned using the training classes and samples. A test sample is classified to one of the unseen test classes based on the attribute predictions and known binary class-attribute associations. Specifically, the posterior probability of observing an attribute $a_m^c$ of the unseen test class c given a sample x is obtained by $p(a_m^c\mid x)=1/\mathit{exp}(A_m{g}_m(\mathbf{x})+B_m)$, where g_m(x) is a binary SVM classifier and the Platt scaling [32] coefficients A_m and B_m are estimated with a validation set to convert the binary classifier outputs to posterior probabilities with a sigmoid function. The class label of the unseen sample is estimated using these posterior probabilities of the attribute class associations, i.e., the sample x is classified to the class with maximum product of MAP predictions. Formally, $arg{max}_{c=1,\dots ,C}{\prod }_{m=1}^{M}p(a_m^c\mid \mathbf{x})$ is the decision rule that maps the input samples to one of the C unseen test classes. Note that, in this formulation, $a_m^c$ are the binary variables that are obtained by thresholding the attribute-class association matrix.

Direct Attribute Prediction with Multiple Ordinal Regression

We illustrated this application of the herein derived algorithm at the beginning of this paper, Fig. 1. We use the human orderings and the proposed MRSVM algorithm to learn a ranking function f_m(.) for each of m attributes. These ranking functions thus determine the ordinal classification. The estimated multiple rankings of a test sample x are used to assign the query to one of C classes using a probabilistic decision rule. To do this, we first estimate a Normal distribution 𝒩(μ_rm, σ_rm) from the ranking estimations f_m(x_v|r_m) of the same validation set samples x_v as used for DAP (i.e., the validation set samples that are used to estimate the Platt scaling above) and given by the ranking labels of r_m. The likelihood of a new sample x of rank r_m for attribute m is simply given by,

$$p(\mathbf{x}\mid r_m)=\frac{1}{\sqrt{2\pi {\sigma }_{r_m}^2}}\mathit{exp}\left(-\frac{{\Vert f_m(\mathbf{x}\mid r_m)-{\mu }_{r_m}\Vert }^2}{2{\sigma }_{r_m}^2}\right).$$ (19)

The MAP prediction of $p(r_m^c\mid \mathbf{x})$ is obtained by the Bayes’ rule,

$$p(r_m^c\mid \mathbf{x})=\frac{p(\mathbf{x}\mid r_m^c)p(r_m^c)}{{\sum }_{r_m=1}^{R_m}p(\mathbf{x}\mid r_m)p(r_m)},$$

where $r_m^c$ is the ordinal attribute value r_m of class c, the attribute rank priors p(r_m) are assumed to be equal, and the likelihood probability p(x|r_m) of observing sample x given attribute m and ranking r_m is obtained from (19).

A test sample x is classified to the class that has the maximum sum-of-posterior-probabilities,

$$arg\underset{c=1,\dots ,C}{max}\sum _{m=1}^{M}p(r_m^c\mid \mathbf{x}).$$

We call this approach Direct Attribute Prediction (DAP) with MSRSVM. Note that, the main advantage of our approach (compared to the baseline algorithm derived above) is its ability to learn multiple ranking functions f_m(.) instead of the binary classifiers g_m(.) employed by other approaches.

Dataset and Experimental Setup

We used the “Animals with Attributes” dataset [21] to test the performance of the algorithms detailed above. The dataset includes 50 animal classes with a partition of 40 training and 10 testing classes. In our experiments, we used 92 samples/class for a total of 3, 680 training and 920 testing samples. Each class is associated with each of 85 attributes with a continuous scalar variable collected from human subjects. Images of the objects are sampled from the Internet and are represented as a set of 6 features of RGB color histograms, SIFT, rgSIFT, PHOG, SURF and local self-similarity histograms.

DAP and DAP with MSRSVM are used to learn binary and multiple ranking functions between the 40 training classes and their corresponding attribute associations. The continuous attribute-class association scores are converted to binary variables via thresholding with the average value [21]. The multiple rankings are obtained by sorting the class association scores for each attribute and dividing into R = 4 ranking levels such that the R regions have the same area under the curve of sorted association scores. This conversion allows similar association scores to share the same ranking value. We use exponential-χ² kernels [40] and calculate the average of them to combine the kernel values calculated using 6 features of histograms. Formally, we use the kernel $k(\mathbf{x},\mathbf{y})=\frac{1}{6}{\sum }_{i=1}^6\mathit{exp}(-{\chi }^2({\mathbf{x}}_i,{\mathbf{y}}_i)/{\gamma }_i)$ that averages exponential-χ² kernels of exp(−χ²(x_i, y_i)/γ_i) with the kernel scale parameter of γ_i that are based on the χ² distances between the i^th feature vectors x_i and y_i, where ${\chi }^2({\mathbf{x}}_i,{\mathbf{y}}_i)={\sum }_{j=1}^d{({\mathbf{x}}_{ij}-{\mathbf{y}}_{\mathbf{ij}})}^2/({\mathbf{x}}_{ij}+{\mathbf{y}}_{\mathbf{ij}})$ and d is the dimensionality of the i^th feature vectors.

The parameters of the algorithms are selected by a 10-fold cross-validation experiment in the training set. This resulted in the selection of the exponential-χ² kernel widths γ_i to be the median of all pairwise χ² distances out of the set of parameters of scaled medians {5⁻¹, 5^−1/2,…, 5¹} * median. The parameter C as defined in the SVM formulation of LIBSVM [2] is selected to be 1 from the set of {0.1, 1, 10, 100}, and the parameter κ in MSRSVM is set to .001.

Results

Table III shows the recognition rates of unseen testing samples obtained with DAP and DAP with MSRSVM. We provided the experimental results for 10 independent validation sets, and their mean. We also show the average confusion tables over 10 runs on independent validation sets in Fig. 5. As seen in the figure, our proposed ranking solution provides less confusions than the binary classification approach. We also computed the average ratio of confused samples to correctly classified samples by $\frac{1}{10}{\sum }_{i=1}^{10}\frac{1-c_{ii}}{c_{ii}}$, where c_ii is the i^th diagonal element of the confusion table. This ratio is 25.81 for DAP and 2.83 for DAP with MSRSVM.

TABLE III.

Here we compare the recognition rates of unseen testing set samples obtained by DAP and DAP with MSRSVM algorithms which are trained with 10 independent validation sets of training set.

Method/Validation Set	1	2	3	4	5	6	7	8	9	10	mean
DAP	35.87	36.2	36.63	35.65	35.65	36.63	36.3	37.72	36.09	35.43	36.21
DAP w MSRSVM	38.48	39.13	38.26	40.76	40.65	39.67	39.57	38.15	39.24	38.7	39.26

Fig. 5.

Shown here are the average confusion tables for the (a) DAP and (b) DAP with MSRSVM approaches. These results are obtained by averaging 10 runs over independent validation sets. The indices of the rows and columns correspond to the indices of the testing classes.

Ordinal Attribute Regression

We have also experimented with several alternative ways of estimating the ordinal attribute values, i.e., learning various ranking functions f_m(.) from the low-level image features. We use ordinal attribute variables to represent the class-attribute associations. These ordinal variables are obtained by binning the continuous variables to equal frequencies. We have tested the accuracy of estimating these variables using three algorithms. Although Multiclass SVM (M-SVM) minimizes the misclassification rate rather than misranking, we provide comparisons to this algorithm to clarify the differences between the classification and ranking algorithms. Gaussian Processes for Ordinal Regression (GPOR) [6] is formulated using a Bayesian framework, which allows to use a gradient descent based optimization algorithm to estimate the model parameters such as ranking hyperplane biases, kernel parameter, and noise level. This parameter estimation may be useful in several problems. However, as pointed out by [6], the scalability of this algorithm is limited due to gradient descent algorithm that becomes the computational bottleneck in large scale problems. We compared the performance of these two algorithms to the proposed MSRSVM solution in estimating the ordinal attribute values. The parameters for M-SVM and MSRSVM are obtained with 10-fold cross validation on the training set, while GPOR utilizes a gradient descent based optimization to estimate its parameters. As seen in Table IV, MSRSVM obtained lower mean absolute errors than M-SVM and GPOR with smaller standard deviation over all attribute label estimations. M-SVM performed worst as its objective is not defined for ranking problems. The limitation of the GPOR performance is most likely due to approximations in model based solution. GPOR training time is much larger than SMO based solutions of M-SVM and MSRSVM. This is because the gradient descent technique with variable convergence rates depends on the complexity of the ranking problem.

TABLE IV.

Accuracy and time comparisons between Multiclass SVM (M-SVM), Gaussian Processes for Ordinal Regression (GPOR), and MSRSVM are provided in the task of estimating 85 ordinal attribute variables. Average mean absolute errors and its standard deviation σ_e are obtained over all attributes. The average training and testing times per attribute are reported along with the standard deviations σ_t.

Method	Average Mean Absolute Errors (σe)	Training and Testing Time in secs (σt)
M-SVM	0.6764(0.4985)	40.2286(21.96)
GPOR	0.4816(0.4382)	246.41(215.22)
MSRSVM	0.4706(0.3972)	61.2927(9.6012)

E. Visualizing Multiple Rankings of Attributes

Another application of MRSVM algorithm is to order objects based on multiple visual properties. This can be used, for example, to give visual feedback to a user’s search.

To illustrate this application, we have performed the following experiment using the “Animals with Attributes” data-set [21]. We selected the whiteness and brownness attribute class associations and converted these measurements to ordinal variables defined by 4 underlying ranking functions. Our goal is to learn the multiple ranking function in the training set classes such that it accurately orders the unseen testing class objects based on estimation of the selected class attribute. We used the training and testing sets that were described in the previous section, and use the 2, 688-dimensional multi-resolution RGB histograms to represent the images.

The mean absolute errors are shown in Table V, where we see a significant error rate for the derived approach. The MRSVM is then applied to unseen test samples to estimate the attribute rankings and visualize them with respect to the ranking scores. The samples from the classes with maximum relevance scores are also shown in Table VI. Brownness and whiteness of the animals increase from top to bottom and left to right, respectively. The number of animal images associated with a certain rank are shown in parentheses. The selected images of the 3 sample animals are in Fig. 6. This provides a visual interpretation of the order of the objects with respect to the attributes. We see that the animals that have darker skins (e.g. leopard, hippopotamus and chimpanzee) ranked 1 in whiteness (i.e., least white), as expected.

TABLE V.

Comparison of the mean absolute error rate of the proposed ranking algorithms, and Support Vector Regression.

Dataset	train/test (d)	ranking (R)	Methods
			MRSVM	MSRSVM	SVR
Animals with Attributes	3680/920 (2688)	1 (4)	1.08	1.08	1.15
		2 (4)	0.98	1.18	1.5
		avg	1.03	1.13	1.32

TABLE VI.

Shown here are the names of the animals in the testing set that have the most number of samples ranked at position (i, j) estimated with the ranking function learned with MRSVM. The number of samples out of 92/animal associated to a ranking are shown in parenthesis. There are no animals associated with the empty cells.

		Whiteness
		1	2	3	4
Brownness	1	-	humpback +whale (21)	-	-
	2	leopard (3)	-	-	raccoon (2)
	3	hippopotamus (10)	rat (50)	-	giant +panda (8)
	4	chimpanzee (4)	pig (23)	persian +cat (2)	-

Fig. 6.

Three sample images of the animals in each nonempty row and column of the multiple ranking table. Rows from left to right corresponds to column wise scan of the nonempty cells of Table VI, i.e. leopard, hippopotamus, chimpanzee, humpback whale, rat, pig, persian cat, raccoon, giant panda.

Recent studies in action recognition [5] are very similar to the set up defined above and are thus also poised to benefit from the algorithms derived in the present paper.

V. Conclusion

The goal of the algorithms derived in the present paper is to estimate a function that orders the objects with respect to a set of specified preferences. To the authors knowledge, this paper presents the first algorithm that can learn multiple ranking functions minimizing the misclassification risk between all neighboring classes, yielding superior results to state of the art algorithms. In particular, we have derived a SMO-based iterative solution that allows learning of the functions from very large databases. Applications in several data-sets shows that the proposed ranking algorithm performs better than Support Vector Regression and SVM_rank. An important application of the derived algorithm is to estimate the attribute associations of an object based on multiple features. This provides a rapid and efficient solution to one of the most classical (yet challenging) problems in machine learning and computer vision – object categorization.