Robust Multicategory Support Matrix Machines

Abstract

We consider the classification problem when the input features are represented as matrices rather than vectors. To preserve the intrinsic structures for classification, a successful method is the Support Matrix Machine (SMM) in [19], which optimizes an objective function with a hinge loss plus a so-called spectral elastic net penalty. However, the issues of extending SMM to multicategory classification still remain. Moreover, in practice, it is common to see the training data contaminated by outlying observations, which can affect the robustness of existing matrix classification methods. In this paper, we address these issues by introducing a robust angle-based classifier, which boils down binary and multicategory problems to a unified framework. Benefitting from the use of truncated hinge loss functions, the proposed classifier achieves certain robustness to outliers. The underlying optimization model becomes nonconvex, but admits a natural DC (difference of two convex functions) representation. We develop a new and efficient algorithm by incorporating the DC algorithm and primal-dual first-order methods together. The proposed DC algorithm adaptively chooses the accuracy of the subproblem at each iteration while guaranteeing the overall convergence of the algorithm. The use of primal-dual methods removes a natural complexity of the linear operator in the subproblems and enables us to use the proximal operator of the objective functions, and matrix-vector operations. This advantage allows us to solve large-scale problems efficiently. Theoretical and numerical results indicate that for problems with potential outliers, our method can be highly competitive among existing methods.

1. Introduction

Many popular classification methods are originally developed for data with a vector of covariates, such as linear discriminant analysis, logistic regression, support vector machine (SVM), and Adaboost [12]. Recent advances in technology enable the generation of a wealth of data with complex structures, where the input features are represented by multi-linear geometric objects such as matrices or tensors, rather than by the form of vectors or scalars. The matrix-type datasets are often encountered in a wide range of real applications, e.g., the face recognition [31] and the analysis of medical images, such as the electroencephalogram data [36].

One common strategy to handle the matrix data classification is to stack a matrix into a long vector, and then employ some existing vector-based methods. This approach has several drawbacks. First, after vectorization, the dimensionality of the resulting vector typically becomes exceedingly high, which in turn leads to the curse of dimensionality, i.e. the large p and small n phenomenon. Second, vectorization of matrix-type data can destroy informative structure and correlation of data matrix, such as the neighbor information and the adjacent relation. Third, under the statistical learning framework, the regularization of vector and matrix data should be different due to their intrinsic structures. To exploit the correlation among the columns or rows of the data matrix, several methods were developed, for example, [6], [27], [24], [14]. These methods are essentially built on the low-rank assumption. Another major direction is to extend regularization techniques commonly used in vector-based classification methods to the present matrix-type data, under certain sparsity assumptions. The regularization with the nuclear norm of matrix of parameters is popular in a variety of settings; see [7] for matrix completion with a low rank constraint, and [36] for matrix regression problems based on generalized linear models. Specifically, [19] proposed the Support Matrix Machine (SMM) which employs a so-called spectral elastic net penalty for binary classification problems. The spectral elastic net penalty is the combination of the squared Frobenius matrix norm and the nuclear norm, in parallel to the elastic net [37]. They showed that the SMM classifier enjoys the property of grouping effect, while keeping a low-rank representation.

Our approach and contribution:

Though the SMM model is simple yet effective, two major issues still remain. The first one is how to extend it to address the problem of multicategory classification. One may reduce the multicategory problem via a sequence of binary problems, for example, using one-versus-rest or one-versus-one techniques. However, the one-versus-rest method can be inconsistent when there is no dominating class, and one-versus-one method may suffer a tie-in-vote problem [17, 18]. Another issue is that existing classifiers may not be robust against outliers, and thus they may have unstable performance in practice [30]. To address these two issues, we propose a new multicategory angle-based SMM using truncated hinge loss functions, which not only provides a natural generalization of binary SMM methods, but also achieves certain robustness to outliers. Our proposed classifier can be viewed as a robust matrix counterpart of the robust vector-based classifier in [32]. We show that the proposed classifier enjoys Fisher consistency and other attractive theoretical properties.

Because the truncated hinge loss is nonconvex and the spectral elastic net regularization is not smooth, the optimization problem involved in our classifier is highly non-trivial. We first show that this problem admits a global optimal solution by exploiting special structures of the model. Next, we show that the optimization problem has a natural DC (difference of two convex functions) decomposition. Hence, one can apply a DC algorithm (DCA) [2] to solve this problem. However, the convex subproblem is rather complicated with nonsmooth objective functions and linear operators, and cannot be solved exactly. This prevents us from solely applying DCA to solve our nonconvex problem. We instead develop a new variant, namely the inexact proximal DCA, to solve this problem. By using the proximal term, we obtain a strongly convex subproblem. Then, to approximately solve this subproblem, we propose to use primal-dual first-order methods proposed in [8, 28]. These methods allow us to exploit the special structures of the problem by utilizing the proximal operator of the objective terms, and matrix-vector multiplications. One drawback of this approach is to match the number of inner iterations in the primal-dual scheme and the inexactness of the proximal DCA scheme. By exploiting the problem structure, we show how to estimate this number of inner iterations at each step of the DCA scheme to obtain a unified DCA algorithm for solving the nonconvex optimization problem. We prove that by adaptively controlling the number of iterations in the primal-dual routine, we can still achieve a global convergence of our DCA variant, which converges to a stationary point. Our method can be implemented efficiently and does not require to estimate any parameter with expensive computational cost. To our limited knowledge, we are not aware of any efficient method to solve SMM-type problems in the literature except the alternating direction method of multipliers (ADMM)-based scheme [5]. In order to examine the efficiency of our method, we compare it with an ADMM-based scheme [5]. As shown in Section 5, our method outperforms ADMM in terms of computational time, and our new model has highly competitive performance among existing methods in different aspects.

Paper organization:

The rest of the article is organized as follows. In Section 2, we briefly review some related works, and then introduce our proposed model and methodology. In Section 3, we describe a new inexact proximal DCA algorithm and investigate its convergence. Some statistical learning results, including Fisher consistency, risk and robustness analysis, are presented in Section 4. Numerical studies are given in Section 5 on both synthetic and real data. Section 6 concludes our work with some remarks, and theoretical proofs are delineated in the appendix.

Notation:

For a matrix $A\in {ℝ}^{p\times q}$ of rank r (r ≤ min(p, q)), $r(r\le \text{min}(p,q)),A=U_A{\Sigma }_A{V}_A^{\top }$ represents the condensed singular value decomposition (SVD) of A, where $U_A\in {ℝ}^{p\times r}$ and $V_A\in {ℝ}^{q\times r}$ satisfy $U_{\text{A}}^{\top }{U}_A=I_r$ and $V_A^{\top }{V}_A=I_r$, and Σ_A = diag·{σ₁(A), ⋯, σ_r(A)} with σ₁ (A) ≥ ⋯ ≥ σ_r (A) > 0. For each τ > 0, the singular value thresholding operator ${\mathcal{D}}_{\tau }(\cdot )$ is defined as follows:

$${\mathcal{D}}_{\tau }(A)=U_A{\mathcal{D}}_{\tau }\left({\Sigma }_A\right)V_A^{\top },$$

where ${\mathcal{D}}_{\tau }\left({\Sigma }_A\right)= diag \left\{{\left[{\sigma }_1(A)-\tau \right]}_{+},\cdots ,{\left[{\sigma }_r(A)-\tau \right]}_{+}\right\}$ with ${[a]}_{+}=\text{max}\{0,a\}$. For $A\in {ℝ}^{p\times q}$, $\Vert A{\Vert }_F=\sqrt{{\sum }_{i,j}{a}_{ij}^2}$ denotes the Frobenius norm of $A,\Vert A{\Vert }_{*}={\sum }_{i=1}^r{\sigma }_i(A)$ denotes the nuclear norm of A, and ∥A∥₂ = σ₁(A) stands for the spectral norm of A. The inner product between two matrices is defined as $\langle A,B\rangle =\sqrt{ tr \left(A^{\top }B\right)}=\sqrt{{\sum }_{i,j}{a}_{i,j}{b}_{i,j}}$. It is well-known that the nuclear norm ∥A∥_*, as a mapping from ${ℝ}^{p\times q}$ to $ℝ$, is not differentiable, but convex. Alternatively, one considers the subdifferential of ∥A∥_*, which is the set of subgradients and denoted by ∂∥A∥_*. For a matrix A, vec(A) denotes its vectorization. We use 〈·, ·〉 to denote the inner product.

For a proper, closed and convex function $\phi :{ℝ}^n\to ℝ\cup \{+\infty \}$, dom(φ) denotes the domain of φ, $pro{x}_{\phi }(\text{x})\triangleq \text{arg}{\text{min}}_{\text{y}}\left\{\phi (\text{y})+\frac{1}{2}\right\}\Vert \text{y}-\text{x}{\Vert }^2$ denotes its proximal operator, and ${\phi }^{*}(y)\triangleq \text{sup}\left\{{\text{x}}^{\top }\text{y}-\phi (\text{x})\right\}$ denotes its Fenchel conjugate. We say that φ has a “friendly” proximal operator if its proximal operator prox_φ can be computed efficiently by, e.g., closed-form or polynomial time algorithms. We say that φ is μ_φ-strongly convex if $\phi (\cdot )-\frac{1}{2}{\mu }_{\phi }\Vert \cdot {\Vert }_F^2$ is convex, where μ_φ ≥ 0. Given a nonnegative real number x, we denote ⌊x⌋ the largest integer that is less than or equal to x.

2. Methodology

Assume that the underlying joint distribution of $(X,\mathcal{Y})$ is $Pr(X,\mathcal{Y})$, where $X\in {ℝ}^{p\times q}$ is the matrix of predictors and $\mathcal{Y}$ is the label. We are given a set of training samples of matrix-type data ${\mathcal{T}}_N={\left\{X_i,y_i\right\}}_{i=1}^N$ collected independently and identically distributed (i.i.d.) from Pr, where $X_i\in {ℝ}^{p\times q}$ is the ith input sample and y_i is its corresponding class label. Here, we assume that X_i’s are zero-centered; otherwise we can make transformation by $X_i-\overline{X}$, where $\overline{X}=N^{-1}{\sum }_{i=1}^r{X}_i$. We take the structure into consideration and handle all X_i’s in the matrix form. Based on the given training set, the target of a classification problem is to estimate a classifier ${\mathcal{T}}_N$, by minimizing the empirical prediction error

$$\frac{1}{N}\sum _{i=1}^N\mathbb{I}\left(\widehat{y}\left(X_i\right)\ne y_i\right),$$

where $\mathbb{I}(\cdot )$ is the indicator function. Because $\mathbb{I}(\cdot )$ is discontinuous, in practice, we use some surrogate loss function to approximate it. As an example, in the case of the SVM, the hinge loss is adopted.

2.1. Review of the Support Matrix Machine

We take the binary problem as a special example with the encoded class labels set {+1, −1}. The optimization problem of [19]’s SMM can be expressed as

$$\underset{M_1,b}{\text{min}}\left\{\frac{1}{N}\sum _{i=1}^N\mathcal{l}\left(y_i\left(\langle M_1,X_i\rangle +b\right)\right)+\lambda \left(\frac{1}{2}\Vert M_1{\Vert }_F^2+\tau \Vert M_1{\Vert }_{*}\right)\right\},$$ (1)

where $M_1\in {\mathcal{R}}^{p\times q}$, and $\mathcal{l}(u)\triangleq [{[1-u]}_{+}=\text{max}\left\{1-u,0\right\}$ is the hinge loss, τ ≥ 0 controls the balance between the Frobenius norm and nuclear norm, and λ > 0 is a tuning parameter that balances the loss and regularization terms. The SMM (1) is a soft margin classifier, and it has a close connection to the ordinary SVM [4, 10]. With τ = 0, by vectorization of the coefficient matrix M₁, SMM reduces to the standard form of the SVM.

The penalty term, $J\left({\text{M}}_1\right)\triangleq \frac{1}{2}\Vert {\text{M}}_1{\Vert }_F^2+\tau \Vert {\text{M}}_1{\Vert }_{*}$, can be re-expressed as

$$J\left(M_1\right)=\sum _{i=1}^{\text{min}\left\{p,q\right\}}\frac{{\sigma }_i^2\left(M_1\right)}{2}+\tau \sum _{i=1}^{\text{min}\left\{p,q\right\}}{\sigma }_i\left(M_1\right).$$

Clearly, this term is essentially of the form of the elastic net penalty for all singular values of the regression matrix M₁, and thus is referred to as the spectral elastic net penalty. Such regularization encourages a low-rank constraint of the coefficient matrix. This can be better understood by the dual problem of (1), which is presented as follows:

$$\{\begin{array}{l}\underset{\alpha }{\text{min}}\left\{\frac{1}{2}{\Vert D_{\tau }\left(\sum _{i=1}^N{\alpha }_i{y}_i{X}_i\right)\Vert }_F^2-\sum _{i=1}^N{\alpha }_i\right\}\\ \text{s.t. }0\le {\alpha }_i\le C,i=1,\dots ,N;\sum _{i=1}^N{\alpha }_i{y}_i=0,\end{array}$$ (2)

where C = (Nλ)⁻¹, and the optimum satisfies $M_1={\mathcal{D}}_{\tau }\left({\sum }_{i=1}^N{a}_i{y}_i{X}_i\right)$. The derivation of (2) is given in the appendix. Under the low-rank assumption, small singular values of ${\sum }_{i=1}^N{a}_i{y}_i{X}_i$ are more likely to be noisy, and hence SMM could be more efficient than the SVM by thresholding with an appropriate choice of τ. Moreover, due to the use of the trace norm, [19] also showed that there is a stronger grouping effect in the estimation of M₁ than the ordinary SVM.

2.2. Robust Multicategory SMM

For extensions of the binary classification method to the multicategory case, a common approach is to use K classification functions to stand for the K categories, and the prediction rule is based on which function has the largest value. Recently, [32] showed that this approach can be inefficient and suboptimal, and proposed an angle-based classification framework that needs to train K 1 classification functions f = (f₁, ⋯ , f_K−1)^⊤. The angle-based classifiers can enjoy better prediction performance and faster computation [33, 34, 26]. Hence, we adopt this strategy here. For simplicity, we focus on linear learning.

To be more specific, consider a centered simplex with K vertices W = (w₁, ⋯, w_K) in ${ℝ}^{K-1}$, where these vertices are given by

$${\text{w}}_k=\{\begin{array}{ll}{(K-1)}^{-\frac{1}{2}}1\hfill & \text{ if }k=1,\hfill \\ -\frac{1+\sqrt{K}}{{(K-1)}^{\frac{3}{2}}}1+\sqrt{\frac{K}{K-1}{e}_{k-1}}\hfill & \text{ if }k\ge 2.\hfill \end{array}$$

Here, e_k is the unit vector of length K − 1 with the kth entry 1 and 0 otherwise, and 1 is the vector of all ones. One can verify that each vector w_k has Euclidean norm 1, and the matrix W introduces a symmetric simplex in ${ℝ}^{K-1}$. Each w_k represents the kth class label. Let M be the linear transformation matrix which maps an input X into a (K − 1)-variate vector f(X) = M · vec(X), where $M={\left[\text{ vec}\left(M_1\right),\cdots ,\text{ vec}\left(M_{K-1}\right)\right]}^{\top }\in {ℝ}^{(K-1)\times pq}$ and $M_j\in {ℝ}^{p\times q}$ for any j ∈ {1, ⋯ , K − 1}. The angle ∠(f(X), w_k) shows the confidence of the sample X belonging to class k. Thus the prediction rule is based on which angle is the smallest, i.e.,

$$\widehat{y}(X)=\underset{k\in \left\{1,\cdots ,K\right\}}{\text{arg min}}\angle \left(f(X),w_k\right).$$

It can also be verified that the least-angle prediction rule is equivalent to the largest inner product, i.e.,

$$\widehat{y}(X)=\underset{k\in \left\{1,\cdots ,K\right\}}{\text{arg max}}\langle f(X),w_k\rangle .$$

Here, we define $H_a(u)\triangleq {[a-u]}_{+}=\text{max}\left\{0,a-u\right\}$ and $G_a(u)\triangleq {[a-u]}_{+}=\text{max}\left\{0,a+u\right\}$. Based on the structure of matrix-type data, our proposed Robust Multicategory Support·Matrix ·Machine (RMSMM) solves

$$\underset{f\in \mathcal{F}}{\text{min}}\left[\frac{1}{N}\sum _{i=1}^N\left\{\gamma T_{(K-1)s}\left(\langle f\left(X_i\right),w_{y_i}\rangle \right)+(1-\gamma )\sum _{k\ne y_i}{R}_s\left(\langle f\left(X_i\right),w_k\rangle \right)\right\}+\lambda J(M)\right],$$ (3)

Where

• $\mathcal{F}\triangleq \left\{\text{f}|\text{f}(\text{X})= M\text{vec}(\text{X}),M\in {ℝ}^{(K-1)\times pq}\right\};$

• $f(X)\triangleq \left(f_1(X),\cdots ,f_{K-1}(X)\right)$ with f_j(X) = 〈M_j,X〉 for j = 1, ⋯,K − 1;

• $J(M)\triangleq {\sum }_{j=1}^{k-1}\left(\frac{1}{2}{\Vert M_j\Vert }_F^2+\tau {\Vert M_j\Vert }_{*}\right)$, where τ ≥ 0 is a balancing parameter;

• $T_s(u)\triangleq H_{K-1}(u)-H_s(u)$ and $R_s(u)\triangleq G_1(u)-G_s(u)$. The notation s ≤ 0 a parameter that controls the location of truncation, and γ ∈ [0, 1] is a convex combination parameter.

In (3), the loss term $\mathcal{L}(\text{X},y,\text{M})=\left\{\gamma T_{(K-1)s}\left(\langle \text{f}(\text{X}),{\text{w}}_y\rangle \right)+(1-\gamma )\sum _{k\ne y}{R}_s\left(\langle \text{f}(\text{X}),{\text{w}}_k\rangle \right)\right\}$ can be written as ${\mathcal{L}}_1(X,y,M)-{\mathcal{L}}_2(X,y,M)$, where

$${\mathcal{L}}_1(X,y,M)=\gamma H_{(K-1)}\left(\langle f(X),w_y\rangle \right)+(1-\gamma )\sum _{k\ne y}{G}_1\left(\langle f(X),w_k\rangle \right),\text{ and}$$

$${\mathcal{L}}_2(X,y,M)=\gamma H_{(K-1)s}\left(\langle f(X),w_y\rangle \right)+(1-\gamma )\sum _{k\ne y}{G}_s\left(\langle f(X),w_k\rangle \right).$$

The first term ${\mathcal{L}}_1$ of the above representation is a generalization of the reinforced multicategory loss function in the angle-based framework proposed by [33]. Note that ${\mathcal{L}}_1$ explicitly encourages 〈f(X), w_y〉 to be small for k ≠ y_i In parallel to [33], we will show later that this convex combination of hinge loss functions enjoys Fisher consistency with $\gamma \in \left[0,\frac{1}{2}\right]$ and s ≤ 0.

The use of the second term ${\mathcal{L}}_2$ is motivated by [30] to alleviate the effect of potential outliers, resulting in a truncated hinge loss. It can be seen that for any potential outlier (X, y) with a sizable 〈f(X), w_y〉, its loss $\mathcal{L}$ is upper bounded by a constant for any f. Thus, the impact of outliers can be alleviated by using $\mathcal{L}$. Note that when s > 0, T_s(u) and R_s(u) are constants within [−s, s]. In this case, the loss for some correctly classified observations is the same as that of those misclassified ones. Hence, it is more desirable to set s ≤ 0. As recommended by [32], the choice of s = −(K − 1)⁻¹ works well and will be used in our simulation study.

The truncated hinge loss is nonconvex, which makes the optimization problem (3) more involved than that of SMM. We next present an efficient algorithm to implement our RMSMM.

3. Optimization Algorithm

Since the optimization problem (3) admits a DC decomposition, we propose to apply DCA [2] to solve this problem. At each iteration of DCA, it requires to solve a convex subproblem, which does not have a closed form. We instead solve this convex subproblem up to a given accuracy and design an inexact variant of DCA so that it automatically adapts the accuracy of the subproblem to guarantee the overall convergence of the full algorithm.

3.1. A DC Representation of (3)

Problem (3) is nonconvex, but fortunately, it possesses a natural DC representation. Indeed, due to the relation $f(X)\triangleq M\cdot \text{vec} (X)$, we can write

$$\langle f(X),w\rangle =w^{\top }M\cdot \text{vec} (X)=a^{\top } \text{vec} (M),$$

where $a\triangleq \text{vec} (X)\otimes w$ with ⊗ denoting the Kronecker product. Let us define

$$a_i\triangleq \text{vec} \left(X_i\right)\otimes w_{y_i},\text{ and }{b}_{ik}\triangleq \text{vec} \left(X_i\right)\otimes w_k, i=1,\cdots ,N,k=1,\cdots ,K-1.$$ (4)

Then, we can rewrite problem (3) as

$$\underset{{}_{\text{M}\in {ℝ}^{(K-1)\times pq}}}{\text{min}}\left\{F(\text{M})\triangleq \frac{1}{N}\sum _{i=1}^N\left[\gamma T_{s(K-1)}\left(a_i^{\top } \text{vec} (\text{M})\right)+(1-\gamma )\sum _{k\ne y_i}{R}_s\left(b_{ik}^{\top } \text{vec} (\text{M})\right)\right]+\lambda J(\text{M})\right\}.$$ (5)

Problem (5) has a DC representation as follows:

$$\underset{{}_M}{\text{min}}\left\{F(M)\triangleq \Phi (M)-\Psi (M)\right\},$$ (6)

where

$$\{\begin{array}{l}\Phi (M)\triangleq \frac{1}{N}\sum _{i=1}^N\left[\gamma H_{K-1}\left(a_i^{\top } \text{vec} (M)\right)+(1-\gamma )\sum _{k\ne y_i}{G}_1\left(b_{ik}^{\top } \text{vec} (M)\right)\right]+\lambda J(M)\hfill \\ \Psi (M)\triangleq \frac{1}{N}\sum _{i=1}^N\left[\gamma H_{s(K-1)}\left(a_i^{\top } \text{vec} (M)\right)+(1-\gamma )\sum _{k\ne y_i}{G}_s\left(b_{ik}^{\top } \text{vec} (M)\right)\right].\hfill \end{array}$$ (7)

Here, both function Φ and Ψ are convex, but nonsmooth. In addition, Ψ is polyhedral. Note that we can always add any strongly convex function S to Φ and Ψ to write F = Φ − Ψ as

$$F(M)=\Phi (M)-\Psi (M)=[\Phi (M)+S(M)]-[\Psi (M)+S(M)],$$ (8)

to obtain a new DC representation. The latter representation shows that both convex functions Φ+S and Ψ+S are strongly convex. This representation also leads to a strongly convex subproblem at each iteration of DCA as we will see in the sequel. However, the choice of S is crucial, and also affects the performance of the algorithm. In our implementation, we simply add a convex quadratic function which leads to a proximal DCA.

Note that dom(Φ) ∩ dom(Ψ) ≠ ∅. Since problem (6) is nonconvex, any point $M^{*}\in {ℝ}^{(K-1)\times pq}$ satisfies

$$0\in \partial F\left(M^{*}\right)\equiv \partial \Phi \left(M^{*}\right)-\partial \Psi \left(M^{*}\right)$$ (9)

is called a stationary point of (6). If M^* satisfies ∂Φ (M^*) ∩ ∂Φ (M^*) ≠ ∅, then we say that M^* is a critical point (6). We show in the following theorem that (6) has a global optimal solution.

Theorem 1

If λ > 0, then problem (6) has at least one global optimal solution M^*.

Proof We first write the objective function F of (5) into the sum $F(\text{M})=\overline{F}(\text{M})+\frac{\lambda }{2}\Vert \text{M}{\Vert }_F^2$, where $\overline{F}$ is a function combining the sum of Ts(K−1), R_s, and the nuclear norm ${\sum }_{i=1}^{K-1}\tau \Vert {\text{M}}_j{\Vert }_{*}{}_i$ in J.

Next, we show that $\overline{F}$ is Lipschitz continuous. Indeed, using the fact that $\left[a\right]+=\text{max}\left\{0,a\right\}=\frac{1}{2}(a+\left|a\right|)$ we can show that $T_s(u)=H_{K-1}(u)-H_s(u)={\left[K-1-u\right]}_{+}-{[s-u]}_{+}$ and $R_s(u)=G_1(u)-G_s(u)={[1+u]}_{+}-{[s+u]}_{+}$ are both Lipschitz continuous. In addition, we have $\Vert M_j{\Vert }_F\le \Vert M_j{\Vert }_{*}\le {[\text{min}\{p,q\}]}^{1/2}\Vert M_j{\Vert }_F$ for j = 1, ⋯ K – 1. Hence, ${\sum }_{i=1}^{K-1}\tau \Vert {\text{M}}_j{\Vert }_{*}$ is also Lipschitz continuous. As a consequence, $\overline{F}$ defined above is Lipschitz continuous. That is, there exists $\overline{L}\in [0,+\infty )$ such that $\left|\overline{F}(M)-\overline{F}(\widehat{M})\right|\le \overline{L}\Vert M-\widehat{M}{\Vert }_F$ for all $\text{M},\widehat{M}\in {ℝ}^{(K-1)\times pq}$.

Using a fixed point ${\text{M}}^0\in {ℝ}^{(K-1)\times pq}$, we can bound F as

$$F(\text{M})\ge F\left({\text{M}}^0\right)-L_{\overline{F}}\Vert \text{M}-{\text{M}}^0{\Vert }_F+\frac{\lambda }{2}\Vert \text{M}{\Vert }_F^2\to +\infty ,\text{ as }\Vert \text{M}{\Vert }_E\to +\infty .$$

Hence, F is coercive, i.e., F(M) → +∞ as ∥M∥_F → +∞. Consequently, its sublevel set $\mathcal{L}(\beta )=\{\text{M}|F(\text{M})\le \beta \}$ is closed and bounded for any $\beta \in ℝ$. By the well-known Weierstrass theorem, (6) has at least one global optimal solution M^*. ◻

3.2. Inexact Proximal DCA Scheme

Let us start with the standard DCA scheme [2] and propose an inexact proximal DCA scheme to solve (6). The proximal DCA is equivalent to DCA applying to the DC decomposition (8) mentioned above, but often uses an adaptive strongly convex term S.

3.2.1. The Standard DCA Scheme and Its Proximal Variant

The DCA method for solving (6) is very simple. At each iteration t ≥ 0, given M^t, we compute a subgradient r (M^t) ∈ ∂Ψ(M^t) and form the subproblem:

$$\underset{\text{M}}{\text{min}}\left\{{\tilde{F}}_t(M)\triangleq \Phi (M)-\langle \nabla \Psi \left(M^t\right),M\rangle \right\},$$ (10)

to compute the next iteration M^t+1 as an exact solution of (10). The subproblem (10) is convex. However, it is fully nonsmooth and does not have a closed form solution.

In the proximal DC variant, we instead apply DCA to the DC decomposition (8) with $S(\text{M})\triangleq \frac{\rho }{2}\Vert \text{M}{\Vert }_F^2$, which leads to the following scheme:

$${\text{M}}^{t+1}\triangleq \underset{\text{M}}{\text{arg min}}\left\{{\tilde{F}}_t(\text{M})\triangleq \Phi (\text{M})-\langle \nabla \Psi ({\text{M}}^t,\text{M}\rangle +|\frac{{\rho }_t}{2}\Vert \text{M}-{\text{M}}^t{\Vert }_F^2\right\},$$ (11)

where ρ_t > 0 is a given proximal parameter. Clearly, M^t+1 is well-defined and unique.

3.2.2. Inexact Proximal DCA Scheme

Clearly the subproblem (11) in the proximal DCA scheme (11) does not have a closed form solution. We can only obtain an approximate solution of this problem. This certainly affects the convergence of (11). We instead propose an inexact variant of (11) by approximately solving

$$M^{t+1}:\approx \underset{M}{\text{arg min}}\left\{ {\tilde{F}}_t(M)\triangleq \Phi (M)-\langle \nabla \Psi \left(M^t\right),M\rangle +\frac{{\rho }_t}{2}\Vert M-M^t{\Vert }_F^2\right\},$$ (12)

where :≈ stands for the approximation between the approximate solution M^t+1 and the true solution ${\overline{M}}^{t+1}$ of the subproblem (12), and is characterized via the objective residual as

$${\tilde{F}}_t\left(M^{t+1}\right)-{\tilde{F}}_t\left({\overline{M}}^{t+1}\right)\le \frac{{\delta }_t^2}{2}.$$ (13)

We note that this condition is implementable if we apply first-order methods in convex optimization to approximately solving (12).

Clearly, by strong convexity, we have

$$\frac{{\rho }_t}{2}\Vert M^{t+1}-{\overline{M}}^{t+1}{\Vert }_F^2\le {\tilde{F}}_t\left(M^{t+1}\right)-{\tilde{F}}_t\left({\overline{M}}^{t+1}\right)\le \frac{{\delta }_t^2}{2}.$$

This leads to $\Vert M^{t+1}-{\overline{M}}^{t+1}{\Vert }_F\le {\delta }_t/\sqrt{{\rho }_t}$, which shows the difference between the approximate solution M^t+1 and the true one ${\overline{M}}^{t+1}$.

Under the inexact criterion (13), we can still prove the following descent property of the inexact proximal DCA scheme (12).

Lemma 1

Let Ψ be μ_Ψ-strongly convex with μ_Ψ ≥ 0. Let {M^t} be the sequence generated by the inexact DCA scheme (12) under the inexact criterion (13). Then

$$F\left(M^{t+1}\right)\le F\left(M^t\right)-\frac{\left({\rho }_t+{\mu }_{\Psi }\right)}{2}\Vert M^{t+1}-M^t{\Vert }_F^2+\frac{{\delta }_t^2}{2}.$$ (14)

Proof Using the optimality condition of (12), we have

$$\nabla \Phi \left({\overline{M}}^{t+1}\right)-\nabla \Psi \left(M^t\right)+{\rho }_t\left({\overline{M}}^{t+1}-{\text{M}}^t\right)=0,\text{ where }\nabla \Phi \left({\overline{M}}^{t+1}\right)\in \partial \Phi \left({\overline{M}}^{t+1}\right).$$

From the μ_Φ- and μ_Ψ-strong convexity of Φ and Ψ, respectively, we have

$$\begin{gathered} \Phi \left({\overline{M}}^{t+1}\right)\le \Phi \left({\text{M}}^t\right)+\langle \nabla \Phi \left({\overline{M}}^{t+1}\right),{\overline{M}}^{t+1}-{\text{M}}^t\rangle -\frac{{\mu }_{\infty }}{2}\Vert {\overline{M}}^{t+1}-{\text{M}}^t{\Vert }_F^2,-\Psi \left({\text{M}}^{t+1}\right)\le -\Psi \left({\text{M}}^t\right)-\langle \nabla \Psi \left({\text{M}}^t\right),{\text{M}}^{t+1}-{\text{M}}^t\rangle -\frac{{\mu }_w}{2}\Vert {\text{M}}^{t+1}-{\text{M}}^t{\Vert }_F^2 \\ =-\Psi \left(M^t\right)-\langle \nabla \Psi \left(M^t\right),{\overline{M}}^{t+1}-M^t\rangle +\langle \nabla \Psi \left(M^t\right),{\overline{M}}^{t+1}-M^{t+1}\rangle -\frac{{\mu }_{\Psi }}{2}\Vert M^{t+1}-M^t{\Vert }_F^2. \end{gathered}$$

Summing up the last two inequalities and using the above optimality condition, we obtain

$$\Phi \left({\overline{M}}^{t+1}\right)-\Psi \left(M^{t+1}\right)\le F\left(M^t\right)-{\rho }_t{\Vert {\overline{M}}^{t+1}-M^t\Vert }_F^2+\langle \nabla \Psi \left(M^t\right),{\overline{M}}^{t+1}-M^{t+1}\rangle -\frac{{\mu }_{\infty }}{2}{\Vert {\overline{M}}^{t+1}-M^t\Vert }_F^2-\frac{{\mu }_w}{2}{\Vert M^{t+1}-M^t\Vert }_F^2.$$

Here, F (M) = Φ(M) − Ψ(M). Next, using (13), we have

$$\Phi \left({\text{M}}^{t+1}\right)\le \Phi \left({\overline{\text{M}}}^{t+1}\right)-\langle \nabla \Psi \left(M^t\right),{\overline{\text{M}}}^{t+1}-{\text{M}}^{t+1}\rangle +\frac{{\delta }_t^2}{2}+\frac{{\rho }_t}{2}{\Vert {\overline{M}}^{t+1}-M^t\Vert }_F^2-\frac{{\rho }_t}{2}{\Vert M^{t+1}-M^t\Vert }_F^2.$$

Summing up the last two inequalities and using F = Φ − Ψ again, we obtain

$$F\left(M^{t+1}\right)\le F\left(M^t\right)-\frac{1}{2}\left[\left({\rho }_t+{\mu }_{\Phi }\right)\Vert {\overline{M}}^{t+1}-M^t{\Vert }_F^2+\left({\rho }_t+{\mu }_{\Psi }\right)\Vert M^{t+1}-M^t{\Vert }_F^2\right]+\frac{{\delta }_t^2}{2}.$$

This implies (14) by neglecting the term $-\frac{1}{2}\left({\rho }_t+{\mu }_{\Phi }\right)\Vert {\overline{M}}^{t+1}-M^t{\Vert }_F^2$

3.3. Solution of The Convex Subproblem

By rescaling the objective function by a factor of $\frac{1}{\lambda }$, we can rewrite the strongly convex subproblem (12) at the iteration t of the inexact proximal DCA scheme as follows:

$$\underset{\text{M}}{\text{min}}\left\{ {\tilde{F}}_t(M)\triangleq P_t(\mathcal{A}(M))+Q_t(M)\right\},$$ (15)

where

$$P_t(\mathcal{A}(M))\triangleq \frac{1}{\lambda N}\sum _{i=1}^N\left[\gamma H_{K-1}\left(a_i^{\top } \text{vec} (M)\right)+(1-\gamma )\sum _{k\ne y_i}{G}_1\left(b_{ik}^{\top } \text{vec} (M)\right)\right]-\frac{1}{\lambda }\langle \nabla \Psi \left(M^t\right),M\rangle ,$$

and

$$Q_t(M)\triangleq J(M)+\frac{{\rho }_t}{2}\Vert M-M^t{\Vert }_F^2=\sum _{j=1}^{K-1}\left[\frac{1}{2}\Vert M_j{\Vert }_F^2+\tau \Vert M_j{\Vert }_{*}+\frac{{\rho }_t}{2}\Vert M_j-M_j^t{\Vert }_F^2\right].$$

Here, $\mathcal{A}$ is a linear operator concatenating all vectors a_i and b_ik, and the subgradient ∇Ψ(M^t) in P_t, and P_t is a nonsmooth convex function, but has a “friendly” proximal operator that can be computed in linear time (see Subsection 3.5 for more details). Due to the strong convexity of J, (15) is strongly convex even for ρ_t = 0. However, one can adaptively choose ρ_t ≥ 0 such that we have a “good” strong convexity parameter. If we do not add a regularization term $\frac{1}{2}\Vert M{I}_j{\Vert }_F^2$, then (15) is strongly convex if ρ_t > 0. Since μ_Ψ = 0 in (6), to strictly get a descent property in Lemma 1, we require ρ_t > 0. The following lemma will be used in the sequel, whose proof is given in the appendix.

Lemma 2

The objective function P_t(·) of (15) is Lipschitz continuous, i.e., there exists L₀ ∈ (0, + ∞) such that $\left|P_t(u)-P_t(\widehat{u})\right|)\le L_0\Vert u-\widehat{u}{\Vert }_F$ for all u, $\widehat{u}$, where L₀ is independent of t. Consequently, the domain $\text{dom }\left(P_t^{*}\right)$ of the conjugate $P_t^{*}$ is bounded uniformly in t, i.e., its diameter $D_{P^{*}}\triangleq 2\text{sup}\left\{\Vert v\Vert |v\in \text{ dom }\left(P_t^{*}\right)\right\}$ is · finite and independent of t.

Denote by

$$\mathcal{L}(\beta )\triangleq \left\{\text{M}\in {ℝ}^{(K-1)\times pq}|F(\text{M})\le \beta \right\},$$ (16)

the sublevel set of (5). As we proved in Theorem 1, the sublevel set $\mathcal{L}(\beta )$ is closed and bounded for any $\beta \in ℝ$. We define

$$D_{\mathcal{L}}\triangleq 2\text{sup}\left\{\Vert M{\Vert }_F|F(M)\le F\left(M^0\right)\right\}$$ (17)

the diameter of this sublevel set, which is finite, i.e., $D_{\mathcal{L}}\in (0,+\infty )$.

3.3.1. Primal-dual Schemes for Solving (15)

Problem (15) can be written into a minimax saddle-point problem using the Fenchel conjugate of P_t. It is natural to apply primal-dual first-order methods to solve this problem. We propose in this subsection two different primal-dual schemes to solve (15).

Our first algorithm is the common Chambolle-Pock primal-dual method proposed in [8]. This method is described as follows. Starting from ${\widehat{M}}_0^t={\tilde{M}}_0^t=M^t$, and $Y_0^t=Y^t$ as an initial dual variable with Y⁰ = 0, set ${\text{M}}_0^t=0$, and at each inner iteration l ≥ 0, we perform

$$\{\begin{array}{l}{Y}_{l+1}^t= pro{x}_{{\sigma }_l^t{P}_t^{*}}\left(Y_l^t+{\sigma }_l^t\mathcal{A}\left({\widehat{M}}_l^t\right)\right),\\ {\tilde{M}}_{l+1}^t= pro{x}_{{\omega }_l^t{Q}_t}\left({\tilde{M}}_l^t-{\omega }_l^t{\mathcal{A}}^{*}\left(Y_{l+1}^t\right)\right),\\ {\theta }_l^t=\frac{1}{\sqrt{1+2\left(1+{\rho }_t\right){\omega }_l^t}}, {\omega }_{l+1}^t={\theta }_l^t{\omega }_l^t, {\sigma }_{l+1}^t=\frac{{\sigma }_l^t}{{\theta }_l^t},\\ {\widehat{M}}_{l+1}^t={\tilde{M}}_t^{l+1}+{\theta }_l^t\left({\tilde{M}}_{l+1}^t-{\tilde{M}}_l^t\right),\\ M_{l+1}^t=\left(1-s_l^t\right)M_l^t+s_l^t {\tilde{M}}_{l+1}^t,\text{ with }{s}_l^t=\frac{{\sigma }_l^t}{{\sum }_{j=1}^l{\sigma }_j^t}.\end{array}$$ (18)

Here, we use the index t for the DCA scheme as the outer iteration counter, and the index l for the inner iteration counter. The initial stepsizes are set to be ${\sigma }_0^t={\omega }_0^t=c\Vert \mathcal{A}{\Vert }^{-1}$, where $\Vert \mathcal{A}\Vert$ is the operator norm of $\mathcal{A}$, and c = 0.999; $\mathcal{A}*$ is the adjoint operator of $\mathcal{A}$ (i.e., when $\mathcal{A}$ is a matrix, $\mathcal{A}*$ is the transpose of $\mathcal{A}$), $prox{ }_{\sigma }{P}_t^{*}$ is the proximal operator of the Fenchel conjugate $P_t^{*}$ of P_t, and ${\text{prox}}_{\omega Q_t}$ is the proximal operator of ω · Q_t. Alternatively, we can also apply [28, Algorithm 2] to solve (15). Originally, [28, Algorithm 2] works directly on the primal space, and has a convergence guarantee on the primal sequence $\left\{M_l^t\right\}$ that is independent of the dual variable $\left\{M_l^t\right\}$ as we can see in Lemma 3 below. Let us describe this scheme here to solve (15). Starting from $M_0^t=M^t$, ${\tilde{M}}_0^t=M^t$ and $Y_0^t=Y^t$, at each inner iteration l ≥ 0, we update

$$\{\begin{array}{l}{Y}_{l+1}^t= pro{x}_{{\sigma }_l^t{P}_t^{*}}\left(Y_0^t+{\sigma }_l^t\mathcal{A}\left({\widehat{M}}_l^t\right)\right)\\ {\tilde{M}}_{l+1}^t= pro{x}_{Q_t/{\omega }_l^t{\beta }_l^t}\left({\tilde{M}}_l^t-\frac{1}{{\omega }_l^t{\beta }_l^t}{\mathcal{A}}^{*}\left(Y_{l+1}^t\right)\right)\\ M_{l+1}^t=\left(1-{\omega }_l^t\right)M_l^t+{\omega }_l^t{\tilde{M}}_{l+1}^t\\ {\omega }_{l+1}^t=\frac{{\omega }_i^t}{2}\left(\sqrt{{\left({\omega }_l^t\right)}^2+4}-{\omega }_l^t\right), {\sigma }_{l+1}^t=\frac{{\sigma }_i^t}{1-{\omega }_{l+1}^t}, {\beta }_{l+1}^t=\Vert \mathcal{A}{\Vert }^2{\sigma }_{l+1}^t,\\ {\widehat{M}}_{l+1}^t=M_{l+1}^t+\frac{{\omega }_{l+1}^t\left(1-{\omega }_l^t\right)}{{\omega }_l^t}\left(M_{l+1}^t-M_l^t\right).\end{array}$$ (19)

Here, the initial values ${\omega }_0^t=1$ and ${\sigma }_0^t=\frac{1}{2}\Vert \mathcal{A}{\Vert }^{-2}\left(1+{\rho }_t\right)$ are given.

Note that both schemes (18) and (19) look quite similar at first glance, but they are fundamentally different. First, the dual step $Y_l^t$ in (19) fixes $Y_0^t$ for all iterations l, while it is recursive with $Y_l^t$ in (18). Second, (18) has an extra averaging step at the last line, while (19) has a linear coupling step at the last line, where it works similarly as the accelerated gradient method of Nesterov [23]. Finally, the way of updating parameters in both schemes are really different.

In terms of complexity, (18) and (19) essentially have the same per-iteration complexity with one proximal operator ${\text{prox}}_s{}_{P_t^{*}}$, one proximal operator ${\text{prox}}_{r{Q}_t}$, one matrix-vector multiplication $\mathcal{A}(\text{M})$, and one adjoint operation ${\mathcal{A}}^{*}(Y)$.

The following lemma provides us conditions to design a stopping criterion for the inner loop (i.e., the l-iterative loop), whose proof is given in the appendix.

Lemma 3

Let ${\overline{M}}^{t+1}$ be the unique solution of (15) at the outer iteration t. Then, the sequence ${\left\{M_l^t\right\}}_{l\ge 0}$ generated by (18) satisfies

$${\tilde{F}}_t\left(M_l^t\right)-{\tilde{F}}_t\left({\overline{M}}^{t+1}\right)\le \frac{\left(1+{\rho }_t+\Vert \mathcal{A}\Vert \right)\Vert \mathcal{A}\Vert }{\left(1+{\rho }_t\right)l^2}\left(\Vert M_0^t-{\overline{M}}^{t+1}{\Vert }_F^2+\Vert Y_0^t-Y^{t+1}{\Vert }_F^2\right),$$ (20)

where ${\overline{Y}}^{t+1}$ is the corresponding exact dual solution of (15). Alternatively, the sequence ${\left\{M_l^t\right\}}_{l\ge 0}$ generated by (19) satisfies

$${\tilde{F}}_t\left(M_l^t\right)-{\tilde{F}}_t\left({\overline{M}}^{t+1}\right)\le \frac{4L_0\Vert \mathcal{A}\Vert }{{(l+1)}^2}\left[\frac{2L_0\Vert \mathcal{A}\Vert }{1+{\rho }_t}+\sqrt{3}\Vert M_0^t-{\overline{M}}^{t+1}{\Vert }_F\right]+\frac{3\left({\rho }_t+1\right)\Vert M_0^t-{\overline{M}}^{t+1}{\Vert }_F^2}{{(l+1)}^2},$$ (21)

where L₀ is given in Lemma 2

One advantage of (19) over (18) is that the right-hand side bound (21) does not depend on the dual variables $Y_0^t$ and ${\overline{Y}}^{t+1}$ as in (20).

3.3.2. The Upper Bound of the Inner Iterations

Our next step is to specify the maximum number of inner iterations l_max(t) to guarantee the condition (13) at each outer iteration t.

First, from both schemes (18) and (19), one can see that $\left\{Y_l^t\right\}\subset \text{ dom }\left(P_t^{*}\right)$. Hence, by Lemma 2, we can bound $\Vert Y_0^t-{\overline{Y}}^{t+1}{\Vert }_F\le D_{P^{*}}$. On the other hand, by Theorem 1, the sublevel set $\mathcal{L}\left(F\left(M^0\right)\right)$ defined by (16) is bounded. We can also bound $\Vert M_0^t-{\overline{M}}^{t+1}{\Vert }_F\le D_{\mathcal{L}}$, where and $D_{\mathcal{L}}$ is given by (17). Using these upper bounds (20), we can show that

$${\tilde{F}}_t\left(M_l^t\right)-{\tilde{F}}_t\left({\overline{M}}^{t+1}\right)\le \frac{\left(1+{\rho }_t+\Vert \mathcal{A}\Vert \right)\Vert \mathcal{A}\Vert }{\left(1+{\rho }_t\right)l^2}\left(D_{\mathcal{L}}^2+D_{P^{*}}^2\right).$$

Let ${\overline{K}}_t\triangleq {\left(1+{\rho }_t\right)}^{-1}\left(1+{\rho }_t+\Vert \mathcal{A}\Vert \right)\Vert \mathcal{A}\Vert$ be a constant. In order to guarantee (13), we require to choose the number of iterations l at most

$$l_{\text{max}}(t)\triangleq \left[\frac{1}{{\delta }_t}\sqrt{{\overline{K}}_t\left(D_{\mathcal{L}}^2+D_{P^{*}}^2\right)}\right]+1\text{ with }{\delta }_t=\frac{1}{{(t+1)}^{\alpha }}\sqrt{D_{\mathcal{L}}^2+D_{P*}^2}.$$ (22)

Here, α > 1 is a given constant specified by the user. With such a choice of δ_t, we have $l_{\text{max}}(t)=\left[\sqrt{{\overline{K}}_t}{(t+1)}^{\alpha }\right]+1$, which is independent of $D_{\mathcal{L}}$ and $D_{P^{*}}$.

If we apply (19) to solve (15), then we have the bound (21). Let ${\widehat{K}}_t\triangleq \frac{8L_0^2\Vert \mathcal{A}{\Vert }^2}{1+{\rho }_t}+4\sqrt{3}{L}_0\Vert \mathcal{A}\Vert D_{\mathcal{L}}+3\left({\rho }_t+1\right)D_{\mathcal{L}}^2$. Since $\Vert M_0^t-{\overline{M}}^{t+1}{\Vert }_F\le D_{\mathcal{L}}$, in order to achieve ${\tilde{F}}_t\left(M_l^t\right)- {\tilde{F}}_t\left({\overline{M}}^{t+1}\right)\le {\delta }_t^2/2$, we require ${(l+1)}^{-2}{\widehat{K}}_t\le {\delta }_t^2/2$ which implies $l+1\ge \sqrt{2{\widehat{K}}_t/{\delta }_t}$ Hence, we can choose

$$l_{\text{max}}(t)\triangleq \left[\frac{\sqrt{2{\widehat{K}}_t}}{{\delta }_t}\right]+1,\text{ with }{\delta }_t=\frac{C_0\sqrt{2{\widehat{K}}_t}}{{(t+1)}^{\alpha }}\text{ and }{C}_0\in (0,1),$$ (23)

to terminate the primal-dual scheme (19). With such a choice of δ_t, we can exactly evaluate $l_{\text{max}}(t)=\left[C_0^{-1}{(t+1)}^{\alpha }+1\right]$, which is also independent of $D_{\mathcal{L}}$.

Remark 1

By the choice of δ_t as in (22) or (23), the maximum number of inner iterations l_max(t) is independent of the two constants $D_{\mathcal{L}}$ and $D_{P^{*}}$. These constants only show up when we prove the convergence of Algorithm 1 in Theorem 2, but they do not need to be evaluated in Algorithm 1 below. Hence, in the implementation of Algorithm 1, we simply use $l_{\text{max}}(t)=\left[\sqrt{{\overline{K}}_t}{(t+1)}^{\alpha }\right]+1$ for (18), or $l_{\text{max}}(t)=\left[C_0^{-1}{(t+1)}^{\alpha }+1\right]$ for (19) to specify the maximum number of inner iterations, where α > 1 is a given number, e.g., α = 1.1.

Algorithm 1.

(Inexact Proximal DC Algorithm with primal-dual iterations)

1: Initialization:

2: Input an accuracy ε > 0. Choose an initial point ${\text{M}}^0\in {ℝ}^{(K-1)\times pq}$, and choose ${\text{Y}}^0\triangleq 0$.

3: Choose two parameters $0<\underset{\_}{\rho }<\overline{\rho }<+\infty$, and σ0=${\sigma }_0={\omega }_0=0.999\Vert \mathcal{A}\Vert$.

4: For t = 0 to T, perform

5: Evaluate a subgradient $\nabla \Psi \left(M^t\right)\in \partial \Psi \left(M^t\right)$ and choose ${\rho }_t\in [\rho ,\overline{\rho }]$.

6: Initialization of inner loop: Initialize $M_0^t, {\widehat{M}}_0^t, {\tilde{M}}_0^t, Y_0^t, {\sigma }_0^t$ and ${\omega }_0^t$ Compute lmax(t).

7: Inner loop: For l = 0, 1,···, lmax(t), perform either (18) or (19).

8: Terminate the inner loop: If l ≥ lmax(t), then set $M^{t+1}=M_{l_{\text{max}}(t)}^t$ and $Y^{t+1}=Y_{l_{\text{max}}(t)}^t$.

9: Stopping criterion: If $\Vert {\text{M}}^{t+1}-{\text{M}}^t{\Vert }_F\le \epsilon \text{ max}\left\{1,\Vert {\text{M}}^t{\Vert }_F\right\}$, then terminate and return Mt+1.

10: End for

3.4. The Overall Algorithm and Its Convergence Guarantee

We now combine the inexact proximal DCA scheme (12), and the primal-dual scheme (18) (or (19)) to complete the full algorithm for solving (5) as in Algorithm 1.

In the sequel, we will explicitly specify the evaluation of a subgradient ∇Ψ(M^t) of Ψ, the choice of ρ_t, and the evaluation of ${\text{ prox}}_s{}_{P_t^{*}}$ and ${\text{prox}}_{r{Q}_t}$. The number of maximum iterations T of the outer loop is not necessary to specify. However, we use T as a safeguard value to prevent the algorithm from an infinite loop. Practically, we can set T to be a relatively large value, e.g., T = 10³. Nevertheless, the stopping criterion at Step 9 will terminate Algorithm 1 earlier. For large-scale problems, we can evaluate the operation norm $\Vert \mathcal{A}\Vert$> of $\mathcal{A}$ by a power method.

We state the overall convergence of Algorithm 1 in the following theorem.

Theorem 2 (Overall convergence)

Let {M^t}· be the sequence generated by Algorithm 1 using (18) (respectively, (19)) for approximately solving (12) up to l_max(t) inner iterations as in (22) (respectively, (23)). Then, we have

$$\sum _{t=0}^{\infty }\Vert M^{t+1}-M^t{\Vert }_F^2<+\infty and it implies \underset{t\to \infty }{\text{lim}}\Vert M^{t+1}-M^t{\Vert }_F=0.$$

Moreover, the sequence ·{M^t} is bounded. Any cluster point M^* of {M^t} is a stationary point of (5). Consequently, the whole sequence{M^t} converges to a stationary point of (5).

Proof Since we apply (19) to solve the subproblem (12), with the choice of _t as in (23), we can derive from Lemma 1 that

$$\sum _{t=0}^T{\rho }_t\Vert M^{t+1}-M^t{\Vert }_F^2\le 2\left(F\left(M^0\right)-F\left(M^{T+1}\right)\right)+\sum _{t=0}^T{\delta }_t.$$

By Theorem 1, we have F (M^T+1) ≥ F (M^*) > −∞, the global optimal value of (5). Hence, using the fact that ${\rho }_t\ge \underset{\_}{\rho }>0$, we obtain

$$\underset{\_}{\rho }\sum _{t=0}^{\infty }\Vert M^{t+1}-M^t{\Vert }_F^2\le 2\left(F\left(M^0\right)-F\left(M^{*}\right)\right)+\sum _{t=0}^{\infty }{\delta }_t<+\infty .$$

Here, $\sum _{t=0}^{\infty }{\delta }_t<+\infty$ due to the choice of δ_t. This is exactly the first estimate in Theorem 2. The second limit in Theorem 2 is a direct consequence of the first one.

By Theorem 1 again, the sublevel set $\sum _{t=0}^{\infty }{\delta }_t<+\infty$ defined by (16) is bounded, and F (M^t+1) ≤ F (M^t) by Lemma 1, we have $\left\{M^t\right\}\subset \mathcal{L}\left(F\left(M^0\right)\right)$, which is bounded. For any cluster point M^* of {M^t}, there exists a subsequence $\left\{M^{t_s}\right\}$ that converges to M^*. Now, we prove that M^* is a stationary point of (5). Using the optimality condition of (12), we have

$$0\in \partial \Phi \left({\overline{M}}^{t+1}\right)-\nabla \Psi \left(M^t\right)+{\rho }_t\left({\overline{M}}^{t+1}-M^t\right).$$ (24)

Note that lim ${\text{lim}}_{t\to \infty }\Vert {\overline{M}}^{t+1}-M^{t+1}{\Vert }_F=0$ due to the choice of δ_t. Here, we can pass this limit to a subsequence if necessary. Using this limit and the fact that limt→∞∥M^t+1−M^t∥_F = 0, we can show ${\text{lim}}_{t\to \infty }\Vert {\overline{M}}^{t+1}-M^t{\Vert }_F=0$. In summary, we have ${\text{lim}}_{t\to \infty }{\overline{M}}^{t+1}={\text{lim}}_{t\to \infty }{M}^t=M^{*}$. Using the definition of Φ and Ψ, we can see that the subgradient ∇Ψ(M^t) of Ψ is uniformly bounded and independent of t. The subgradient $\nabla \Phi \left({\overline{M}}^{t+1}\right)$ can be represented as $\nabla \Phi \left({\overline{M}}^{t+1}\right)={\overline{S}}^{t+1}+\lambda {\overline{M}}^{t+1}$., where ${\overline{S}}^{t+1}$ is uniformly bounded and independent of t. By taking subsequence if necessary, both $\nabla \Phi \left({\overline{M}}^{t+1}\right)$ and ∇Ψ(M^t) converge to ∇Ψ(M^*) and ∇Ψ(M^*), respectively. By [25, Theorem 24.4], we have ∇Φ(M^*) ∈ ∂Φ(M^*) and ∇Ψ(M^*) ∈ ∂Ψ(M^*). Using this fact, ${\text{lim}}_{t\to \infty }{\overline{M}}^{t+1}={\text{lim}}_{t\to \infty }{M}^t=M^{*}$, and the boundedness of ρ_t, we can show that 0 ∈ ∂Φ(M^*) −∂Ψ(M^*) Hence, M^* a stationary point of (5). By the boundedness of {M¹} and limt→∞∥M^t+1−M^t∥_F = 0, one can then use routine techniques to show that the whole sequence {M^t} converges to M^*. ◻

While the convergence result given in Theorem 2 is rather standard and similar to those in [2], its analysis for the inexact proximal DCA seems to be new to the best of our knowledge. Note that the convex subproblem in DCA-type methods is often general and may not have closed-form solutions. It is natural to incorporate inexactness in an adaptive manner to guarantee the convergence of the overall algorithm.

3.5. Implementation Details and Comparison with ADMM

In Algorithm 1, we need to compute the proximal operator ${\text{prox}}_{{\sigma }_l^t{P}_t^{*}}$ of the Fenchel conjugate $P_t^{*}$ of P_t and ${\text{prox}}_{{\omega }_l^t}{}_{Q_t}$ of Q_t. In addition, in order to compare our method with other optimization methods, we specify the well-known ADMM to solve (12) as our comparison candidate.

3.5.1. Evaluation of Subgradient ∇Ψ(M^t) and The Choice of ρ_t

Using the definition of Ψ from (7), we have

$$\nabla \Psi \left(M^t\right)=\frac{1}{N}\sum _{i=1}^N\left[\gamma \nabla H_{s(K-1)}\left(a_i^{\top } \text{vec} M^t\right)a_i+(1-\gamma )\sum _{k\ne y_i}\nabla G_s\left(b_{ik}^{\top } \text{vec} M^t\right)b_{ik}\right],$$

where $\nabla H_{s(K-1)}(u)=\frac{1}{2}\cdot \text{ sign }(s(K-1)-u)-\frac{1}{2}$ and $\nabla G_s(v)=\frac{1}{2}\cdot \text{ sign }(s+v)+\frac{1}{2}$. Here, sign(·) is the common sign function.

To choose ρ_t, we first choose a range $[\underset{\_}{\rho },\overline{\rho }]$ in (0, +∞). For instance, we can choose $\underset{\_}{\rho }={10}^{-5}$ and $\overline{\rho }={10}^5$, and {ρ_t} is any sequence in $[\underset{\_}{\rho },\overline{\rho }]$. We can also fix ρ_t for all t as ${\rho }_t=\overline{\rho }>0$, e.g., ρ_t = 10⁻³. From our experience, we observe that if ρ_t is small, the strong convexity of (15) is 1+ρ_t, which is also small. Hence, the number of inner iterations l_max(t) is large. However, the number of outer iterations t may be small. In the opposite case, if ρ_t is large, then we need a small number l_max(t). Nevertheless, due to a short step M^t+1 −M^t, the number of outer iterations may increase. Therefore, trading-off the value of ρ_t is crucial and affects the performance of Algorithm 1.

3.5.2. Evaluation of Proximal Operators

To compute the proximal operator of $P_t^{*}$ in (18), we can use Moreau’s identity [3]:

$$pro{x}_{\sigma P_t^{*}}(z)=\{\begin{array}{l}{z}_j-\sigma pro{x}_{1/\sigma P_t}\left(z_j/\sigma \right)=z_j-\sigma \left[{\mathcal{S}}_{1/\sigma }\left(z_j+{\mu }_j\right)-{\mu }_j\right], j=1,\cdots ,2N,\hfill \\ z_j-\sigma pro{x}_{1/\sigma P_t}\left(z_j/\sigma \right)=(1-\sigma )z_{2N+1}+1,\hfill \end{array}$$

where ${\mathcal{S}}_r(v)= sign (u)\odot \text{max}\left\{\left|v\right|-r,0\right\}$ is the well-known soft-thresholding operator.

To compute the proximal operator of Q_t, we note that (here, τ_j = τ)

$$Q_t(M)\triangleq \sum _{j=1}^{K-1}\left[\frac{1}{2}\Vert M_j{\Vert }_F^2+{\tau }_j\Vert M_j{\Vert }_{*}+\frac{{\rho }_t}{2}\Vert M_j-M_j^t{\Vert }_F^2\right].$$

Hence, we have

$$pro{x}_{\omega Q_t}(M)={\left( pro{x}_{\omega Q_{t_j}}\left(M_j\right)\right)}_{j=1}^{K-1},$$

Where $Q_{t_j}\left({\text{M}}_j\right)\triangleq \frac{1}{2}\Vert {\text{M}}_j{\Vert }_F^2+{\tau }_j\Vert {\text{M}}_j{\Vert }_{*}+\frac{{\rho }_t}{2}\Vert {\text{M}}_j-{\text{M}}_j^t{\Vert }_F^2$, and

$$pro{x}_{\omega Q_{t_j}}\left(M_j\right)\triangleq \underset{{\widehat{M}}_j}{\text{arg min}}\left\{\omega {\tau }_j{\Vert {\widehat{M}}_j\Vert }_{*}+\frac{1+\omega \left(1+{\rho }_t\right)}{2}{\Vert {\widehat{M}}_j-\frac{\omega {\rho }_t{M}_j^t+M_j}{1+\omega \left({\rho }_t+1\right)}\Vert }_F^2\right\}.$$

This operator can be computed in a closed form using SVD of $\left(\omega {\rho }_t{M}_j^t+M_j\right)/\left[1+\omega \left({\rho }_t+1\right)\right]=U_j{\Sigma }_j{V}_j^{\top }$ as $pro{x}_{\omega Q_{t_j}}\left(M_j\right)=U_j{\mathcal{S}}_r\left({\Sigma }_j\right)V_j^{\top }$, where ${\mathcal{S}}_r$ is the soft-thresholding operator defined above with r = ωτ_j/[1 + ω (1 + ρ_t)].

3.5.3. ADMM Method for Solving (15)

In Algorithm 1, we can apply ADMM to solve the subproblem (15) instead of primal-dual methods. We split the nuclear norm in Q_t of (15) by introducing an auxiliary variable S and rewrite (15) as

$$\{\begin{array}{l}\underset{M,S}{\text{min}}\left\{{\left[P_t(\mathcal{A}(\text{M}))+\sum _{j=1}^{K-1}\left[\frac{1}{2}{\Vert {\text{M}}_j\Vert }_F^2+\frac{{\rho }_t}{2}{\Vert {\text{M}}_j-{\text{M}}_j^t\Vert }_F^t\right]\right]}_{B_t(\text{M})}+\sum _{j=1}^{K-1}{\tau }_j{\Vert {\text{S}}_j\Vert }_{*}\right\}\\ \text{s.t S}-\text{M}=0.\end{array}$$ (25)

We define the corresponding augmented Lagrangian function of (25) as

$${\mathcal{L}}_{\beta }(M,S,\text{Λ})\triangleq P_t(\mathcal{A}(M))+\sum _{j=1}^{K-1}\left[\frac{1}{2}\Vert M_j{\Vert }_F^2+\frac{{\rho }_t}{2}\Vert M_j-M_j^t{\Vert }_F^2\right]+\sum _{j=1}^{K-1}{\tau }_j\Vert S_j{\Vert }_{*}+ trace\left({\text{Λ}}^{\top }(S-M)\right)+\frac{\beta }{2}\Vert S-M{\Vert }_F^2,$$

where β > 0 is a penalty parameter. Starting from an initial point $M_0^t=M^t$, $S_0^t=M^t$, our ADMM scheme for solving (25) updates at the inner iteration l according to the following steps:

$$\{\begin{array}{l}{\text{M}}_{l+1}^t\triangleq \underset{\text{M}}{\text{arg min}}\left\{B_t(\text{M})+ trace \left({\left({\text{Λ}}^t\right)}_l^{\top }\left(S_l^t-\text{M}\right)\right)+\frac{\beta }{2}\Vert {\text{S}}_l^t-\text{M}{\Vert }_F^2\right\}\hfill \\ S_{l+1}^t\triangleq \underset{S}{\text{arg min}}\left\{\sum _{j=1}^{K-1}{\tau }_j\Vert S_j{\Vert }_{*}+ trace \left({\left({\text{Λ}}_l^t\right)}^{\top }\left(S-M_{l+1}^t\right)\right)+\frac{\beta }{2}\Vert S-M_{l+1}^t{\Vert }_F^2\right\}\hfill \\ {\text{Λ}}_{l+1}^t\triangleq {\text{Λ}}_l^t+\beta \left(S_{l+1}^t-M_{l+1}^t\right).\hfill \end{array}$$ (26)

In this scheme, the auxiliary sequence $\left\{S_l^t\right\}$ can be computed into a closed form using SVD as we have done in Subsection 3.5.2. The sequence $\left\{M_l^t\right\}$ requires to solve a general convex problem. However, this problem has a special structure so that its dual formulation becomes a boxed constrained convex quadratic program, which is very similar to (2). Hence, we solve this problem by coordinate descent methods, see, e.g., [29]. In summary, if we apply ADMM to solve (15), then our inexact proximal DCA has three loops: DCA outer iterations, ADMM inner iterations, and coordinate descent iterations for computing $\left\{M_l^t\right\}$.

Remark 2 (Convergence of the ADMM scheme (26))

Note that (15) is strongly convex, and both subproblems in $M_{l+1}^t$ and $S_{l+1}^t$ of (26) are strongly convex, and therefore, uniquely solvable. Consequently, this scheme converges theoretically as proved e.g., in [5, Appendix A]. Together with asymptotic convergence guarantees, the convergence rates of ADMM, where (26) is a special case, have been studied in e.g., [11, 13, 21]. We omit the details here.

4. Statistical Properties

In this section, we explore some statistical properties of our proposed classifier RMSMM (3). In the first part, we establish the Fisher consistency result for the RMSMM, and study the finite sample bound on the misclassification rate. In the second part, we analyze the robustness property of RMSMM via the breakdown point theory.

4.1. Classification Consistency

Fisher’s consistency is a fundamental property of classification methods. For an observed matrix-type data with fixed X, and denote by $P_k(X)= Pr(\mathcal{Y}=k|X)$ the class conditional probability of class k ∈ {1, 2, ⋯, K}. One can verify that the best prediction rule, namely, the Bayes rule, which minimizes the misclassification error rate, is ŷ_Bayes(X) = arg max_k P_k(X).

For a classifier, denote by ϕ (f(X), y) its surrogate loss function for classification using f as the classification function, and ŷ_f the corresponding prediction rule. Assume the conditional loss is L(X) = E[ϕ(f(X), y) | X], where the expectation is taken with respect to the marginal distribution of $(\mathcal{Y}|X)$. We denote the theoretical minimizer of the conditional loss as f*(X) = arg min_f L(X). When ŷ_f* (X) = ŷ_Bayes(X), we say the classifier is Fisher consistent. Let us denote by $\mathcal{L}(X,y,M)$ the loss function in (3). Then, we have the following result.

Theorem 3

The classifier with the loss $\mathcal{L}(X,y,M)$ is Fisher consistent when $\gamma \in \left[0,\frac{1}{2}\right]$ and s ≤ 0.

This result can be viewed as a generalization of Theorem 1 in [34] which is devised for vector-type observations. By this theorem, we know that our classifier RMSMM can achieve the best classification accuracy, given a sufficiently large matrix-type training dataset and a rich family $\mathcal{F}$. The following theorem provides an upper bound of the prediction error using the training dataset.

The proof of both Theorems 3 and 4 can be found in the appendix.

Theorem 4

Suppose that the conditional distribution of X given $\mathcal{Y}=k$ is the same as the distribution of C_k + E, where $C_k\in {ℝ}^{p\times q}$ is a constant matrix and the entries of E are i.i.d. random variables with mean zero and finite fourth moment. Let $\widehat{M}={\left[\text{vec}\left({\widehat{M}}_1\right),\cdots , \text{vec}\left({\widehat{M}}_{K-1}\right)\right]}^{\top }\in ℝ(K-1)\times pq$ denote the solution of (5). Then, with probability at least 1 − δ, the misclassification rate of the classifier ŷ corresponding to $\widehat{M}$ can be bounded as

$$E[\mathbb{I}\left\{\mathcal{Y}\ne \widehat{y}(X)\right\}]\le \frac{1}{N}\sum _{i=1}^N\mathbb{I}\left\{y_i\ne \widehat{y}\left(X_i\right)\right\}+\sqrt{\frac{\text{log}\left({\delta }^{-1}\right)}{N}}+\frac{ cr(\sqrt{p}+\sqrt{q})}{\sqrt{N}},$$ (27)

Where $r={\sum }_{j=1}^{K-1}\Vert {\widehat{M}}_j{\Vert }_{*}$, and c is a constant specified in the proof.

Theorem 4 measures the gap between the expectation error and the empirical error, which allows us to get a better understanding of the utility of the nuclear norm. For each category, the decision matrix contains p × q parameters, and therefore, if we only impose the Frobenius constraints [34] we would expect at best to obtain rates of the order $\sqrt{pq}$. By taking the low rank structure of the decision matrices into account, we use the nuclear norm penalty to control the singular values of the decision matrices. For the i-th singular vectors of the k-th decision matrix, there are p + q + 1 free parameters in total [22], one for the singular value σ_ki and the others for the orthogonal vectors with dimensions p and q. Its contribution to the gap will be $c{\sigma }_{ki}(\sqrt{p}+\sqrt{q})$. Hence, with the low-rank structure of the decision matrices, the nuclear-norm-penalized estimator achieves a substantially faster rate.

The rate in Theorem 4 can be further improved if we additionally impose some low-rank constraint on the noise term of X_i. For example, consider E = UΛV^⊤, where $\text{Λ}\in {ℝ}^{r_x\times r_x}$ is low-rank noise with all entries i.i.d. with mean zero and the finite fourth moment, U and V are orthogonal projection matrices independent of Λ. It can be verified that the term $\sqrt{p}+\sqrt{q}$ in the rate above can be replaced by $2\sqrt{r_x}$. Finally, as a side remark, consider a special case with q = 1, i.e., the features are vectors rather than matrices. In such a situation, the nuclear norm reduces to the quadratic norm, and the last term of the upper bound in (27) will become $cr(\sqrt{p}+1)/\sqrt{N}$, which is equivalent to existing results, for example, see [34].

4.2. Breakdown Point Analysis

Robustness theory has been developed to evaluate instability of statistical procedures since the 1960s [15]. The breakdown point theory focuses on the smallest fraction of contaminated data that can cause an estimator totally diverging from the original model. Here we consider the breakdown point analysis for multicategory classification models.

Let ${\mathcal{T}}_n$ be the original n observations, and ${\tilde{\mathcal{T}}}_{n,m}={\mathcal{T}}_{n-m}\cup {\mathcal{V}}_m$ be the contaminated sample with m observations of ${\mathcal{T}}_n$ contaminated, and $\tilde{M}=\widehat{M}\left({\tilde{\mathcal{T}}}_{n,m}\right)$ be the parameters estimated from the contaminated sample. We extend the sample angular breakdown point in [35] to the multicategory classification problem as

$${ϵ}^{\star }\left(\widehat{M},{\mathcal{T}}_n\right)=\text{min}\left\{\frac{m}{n} | \exists k, s\text{.}t\text{. }{w}_k^{\top }\widehat{M} {\tilde{M}}^{\top }{w}_k\le 0\right\},$$

where $\widehat{M}=\widehat{M}\left({\mathcal{T}}_n\right)$ is the estimated decision matrix from the original sample. Since the angle-based classifiers make the decision by comparing the angles between the (K − 1)-dimensional classification function f and the K vertices of the simplex ${\left\{w_k\right\}}_{k=1}^K$, it is reasonable to quantify the divergence between classifiers via the angles between the decision vectors $w_k^{\top }\tilde{M}$ and the original counterpart, $w_k^{\top }\widehat{M}$. When there exists one category k so that the angle between the two decision vectors is larger than π/2, the two classifiers would behave totally different at this category. Consequently, the classifier with contaminated samples would “break down”.

The following theorem compares the sample breakdown points of the proposed RMSMM and the multicategory SMM (MSMM) which generalizes [19]’s SMM using angle-based methods, say γ = 1/2 and s = −∞ in Eq. (3).

Theorem 5

Assume that $\widehat{M}\ne 0$. Then the breakdown point of MSMM is 1/n, while the breakdown point of RMSMM is not smaller than $\frac{{ϵ}_1}{2(K-1)(1-s)}$, where

$${ϵ}_1=\underset{\text{M}\in {\text{Δ}}^{-}}{\text{min}}F(\text{M})-\underset{\text{M}\in {\text{Δ}}^{+}}{\text{min}}F(\text{M})>0.$$

By this theorem, only one contaminated observation will make the MSMM classifier break down. In other words, this estimator may not work well in the presence of few outliers. In contrast, the breakdown point of our proposed RMSMM, benefitting from the use of truncated hinge loss functions, has a fixed lower bound. Thus, the RMSMM has high outlier-resistance compared to its counterpart without truncation. The robustness property will be carefully examined via numerical comparisons in the next section.

5. Numerical Experiments

In this section, we investigate the performance of our proposed robust angle-based SMM using simulated and real datasets. Our configuration of the algorithm is as follows. For the primal-dual method described in Algorithm 1, we use M⁰ = 0 and ρ_t = 0.01 for every t. We set the stop criterion as ∥M^t+1 − M^t∥_F ≤ 10⁻⁴ max {1, ∥M^t∥_F}. All the simulation results are obtained based on 100 replications.

5.1. Simulation Results

We generate simulated datasets by the following two scenarios. In the first scenario, the dimensions of input matrices are 50 × 50. For the kth category, to make the matrices low-rank, we randomly generate two 50 × 5 matrices, U_k and V_k, which are standard orthonormal. More precisely, we first generate two 50 × 5 matrices with all the entries i.i.d. from the standard normal distribution and obtain U_k and V_k by the Gram-Schmidt process. The center of each class is then specified by $C_k=U_k{V}_k^{\top };k=1,\cdots ,K$. The observations in each class are generated by C_k + E; k = 1,⋯, K, where E is a 50 × 50 normal random matrix with all entries i.i.d from $\mathcal{N}\left(0,{\sigma }^2\right)$. For the contaminated observations, we generate them by 3C₁ + E for $\mathcal{Y}\in \left\{1,\cdots ,K\right\}$.

In the second scenario, the dimensions of input matrices are fixed as 80 × 100. We follow the settings in [36] to generate the true array signals by $C_k=C_{k,1}{C}_{k,2}^{\top };k=1,\cdots ,K$, where each entry of C_k is 0 or 1 and $C_{k,i}\in {ℝ}^{p_i\times r}$, p₁ = 80 and p₂ = 100. To control the rank and the percentage of nonzero entries, we set r = 10 and generate C_k,i by setting each row to contain only one entry one and others zero, and the probabilities of entries being one are equal. All the entries of the noise matrix E are i.i.d. from σ · t(3), where t(3) denotes the Student’s t-distribution with three degrees of freedom. The outliers are generated by the same method as in the first scenario.

We use 10³ observations for training, 10⁴ observations for tuning and 10⁴ observations for testing. The contamination ratio in the training sample ρ, is chosen as 0%, 10%, and 20%. For training the truncated model, we use the solutions of the ordinary SMM as an initial point. Following the suggestion by [33], we choose γ = 1/2 as it can provide stable classification performance. The truncation parameter, s, is fixed at −1/(K − 1). The other hyper-parameters, C and τ, are selected via a grid search on the tuning set.

We first consider the binary classification problem, say K = 2. We compare our RMSMM with the SMM in [19]. We also include a naive benchmark, the standard SVM method which is applied to the stacked-up vectors. Fig. 1 presents the classification error rates of RMSMM, SMM, and SVM on the simulated data with Scenario (I) and K = 2. Three noise magnitudes are considered: σ = 0.5, 0.7 and 0.9. Both two “support-matrix-based” methods, RMSMM and SMM, perform much better than the SVM. It has been observed that RMSMM generally outperforms SMM when there exits outliers, and its advantage becomes more pronounced for larger ρ. All methods are affected by different values of σ, but the comparison conclusion still holds for various σ.

Fig. 1.

Classification error rates for RMSMM, SMM, and SVM on the simulated data with Scenario (I) and K = 2. Here, ρ stands for the percentage of data that are contaminated. SMM: [19]’s support matrix machine; SVM: the standard SVM applied to the stacked-up vectors.

Next we consider the multicategory case. Fig. 2 depicts the boxplots of the classification error rates for RMSMM and other competitors under Scenario (I) with K = 3 and 5. Three benchmarks are considered: the multicategory SMM using angle-based methods, MSMM; the angle-based multicategory SVM classifier [32] and its robust version RMSVM classifier [34]. In the case of ρ = 0, the RMSMM and its non-robust counterpart MSMM perform almost identically, which demonstrates that the truncation parameter, s, can adapt to the data structure and make the efficiency loss of RMSMM relative to MSMM minimal when there is no outlier. When ρ = 0.1 or ρ = 0.2, the advantage of RMSMM is clear: the means and standard variations of its classification error rates are generally smaller. From this figure, we can also observe that the use of the nuclear norm is prominent: the two SMM-based classifiers perform much better than the two SVM-based ones. Similar comparison conclusions can be drawn from Fig. 3, which reports the classification error rates of RMSMM and the other three methods under Scenario (II) with σ = 3, 4, and 5.

Fig. 2.

Classification error rates for RMSMM, SMM, and SVM on the simulated data with Scenario (I). The top three panels: the case with K = 3; the bottom three panels: the case with K = 5. MSMM: multicategory generalization of SMM using angle-based methods; MSVM: the angle-based multicategory SVM [32]; RMSVM: the robust angle-based multicategory SVM [34].

Fig. 3.

Classification error rates for RMSMM, SMM, and SVM on the simulated data with Scenario (II). The top three panels: the case of K = 3; the bottom three panels: the case of K = 5.

Finally, we present some comparison results of the ADMM and primal-dual algorithms for solving the RMSMM optimization problem (5). Fig. 4 reports the classification error rates and the corresponding computational time (in seconds) of the RMSMM using the two different primal-dual algorithms: (18) and (19) under Scenario (I) with σ = 0.7 and Scenario (II) with σ = 4 when K = 3. The bottom two panels record the total run time including the selection of tuning parameters. The tuning parameters λ and τ in the RMSMM are selected via a grid search. To be more specific, λ ∈ [0.1, 10⁴] and for each choice of λ, τ is tuned to make the decision matrix change from full-rank to rank one. One can see that the two algorithms perform very similarly in terms of classification rates, but the proposed primal-dual algorithm is significantly faster and the advantage is more remarkable as ρ increases. This is further confirmed by Fig. 5 which depicts the decay curves of the RMSMM objective function values versus the computational time until the two algorithms reach the desired accuracy. We consider the case under Scenario (II) with K = 3 and σ = 4 for a given combination of tuning parameters. In particular, we fix a combination of (λ, τ) and record the objective function values for each iteration. Clearly, the primal-dual algorithm is generally more stable and converges much faster than ADMM.

Fig. 4.

Comparison between the ADMM and primal-dual algorithms: Primal-Dual stands for (18), and Proximal-Alter stands for (19) for solving the RMSMM optimization problem (5). The top two panels: classification error rates under Scenario (I) with σ = 0.7 and Scenario (II) with σ = 4 when K = 3; The bottom two panels: the corresponding computational time (in seconds).

Fig. 5.

The decrease of the RMSMM objective values with respect to the computational time under Scenario (II) with K = 3 and σ = 4.

5.2. A Real-data Example

We apply the RMSMM model (5) to the Daily and Sports Activities Dataset [1] which can be found on the UCI Machine Learning Repository. The dataset comprises motion sensor data of 19 daily sport activities, each performed by 8 subjects (4 females, 4 males, between the ages of 20 and 30) in their own style for 5 minutes. The dataset was collected by several sensors. The input matrices are of dimension 125 × 45, where each column contains 125 samples of data acquired by a sensor over a period of 5 seconds at 25 Hz sampling frequency, and each row contains the data acquired from all of 45 sensor axes at a particular sampling instant.

To show the efficient performance of the proposed RMSMM model, we only select the first 10 categories of the dataset for simplicity. Thus the total number of instances is N = 10 × 8 × 60 = 4, 800. It is a 10-category and balanced classification problem with 480 instances in each category. We equally and randomly divide the data into three parts for training, tuning, and testing, and the sample size of each part is 1, 600.

We choose s = −K + 1, and select the other parameters by a grid search. We report the classification accuracy of RMSMM, MSMM, RMSVM, and MSVM in Fig. 6-(left). The two matrix-based methods achieve lower classification rates than the other two vector-based classifiers, due to the benefit of the nuclear norm. This improvement can be more clear in Fig. 7, which presents the heatmap of the decision matrices of RMSMM and RMSVM; the former has a more sparse structure than the latter.

Fig. 6.

Classification error rates for RMSMM, MSMM, RMSVM, and MSVM on the Daily and Sports Activities Dataset. The left and right panels present the results when the data are clean or contaminated, respectively.

Fig. 7.

Heatmaps of the first decision matrices of RMSMM (left panel) and RMSVM (right panel)

To demonstrate the effect of potential outliers on classification accuracy, we artificially contaminate the dataset with outliers by randomly relabeling 10% of the training set into another class. From Fig. 6-(right), we observe that the performances of all the methods are deteriorated by this manipulation, while the RMSMM performs the best. Both two robust classifiers, RMSMM and RMSVM, are less affected by the outliers, than the other two non-robust methods. All these numerical examples shown above suggest that the RMSMM is a practical and robust classier for a multicategory classification problem when the input features are represented as matrices.

6. Concluding Remarks

In this paper, we consider how to devise a robust multicategory classifier when the input features are represented as matrices. Our method is constructed in the angle-based classification framework, embedding a truncated hinge loss function into the support matrix machine. Although the corresponding optimization problem is nonconvex, it admits a natural DC (difference of two convex functions) representation. Hence, it is natural to apply DCA algorithms to solve this problem. Unfortunately, the convex subproblem in DCA is rather complex and does not have a closed form solution. Therefore, we develop an inexact proximal DCA variant to solve the underlying optimization problem. To approximately solve the convex subproblem, we propose to use primal-dual first-order methods. We combine both inexact proximal DCA and primal-dual methods to obtain a new proximal DCA scheme. We prove that our optimization model admits a global optimal solution, and the sequence generated by our DCA variant globally converges to a stationary point.

In terms of statistical learning perspective, we prove Fisher’s consistency and prediction error bounds. Numerical results demonstrate that our new classifiers are quite efficient and much more robust than existing methods in the presence of outlying observations. We conclude the article with two remarks. First, our unified framework is demonstrated using the linear classifier. Though it is well recognized that linear learning is an effective solution in many real applications, it may be sub-efficient especially for problems with complex feature structures. Thus it is of interest to thoroughly study nonlinear learning under the proposed framework. Second, our numerical results show that the proposed procedure works well under large-dimensional scenarios. Theoretical investigation to the necessary condition on which the statistical theoretical guarantee of RMSMM holds is another interesting topic for future study.

A Appendix: Proofs of Technical Results

In this appendix, we provide all the remaining proofs of the results presented in the main text.

A.1. Proof of Lemma 2: Lipschitz continuity and boundedness

Since [a]₊ = max {0, a} = (a + |a|)/2, the function P_t defined in (15) can be rewritten as $P_t(\mathcal{A}(z))=\Vert \widehat{\mathcal{A}}z+\mu {\Vert }_1+d_t^{\top }Z$ for some matrix $\widehat{\mathcal{A}}$ and vectors μ and d_t. Here, ${\text{d}}_t\triangleq \text{d}-{\lambda }^{-1}\text{vec}\left(\nabla \Psi \left(M^t\right)\right)$. However, ψ is also Lipschitz continuous due to its definition. This implies that ∇ψ(M^t) is uniformly bounded, i.e., there exists a constant C₀ ∈ (0, + ∞) such that ||∇ψ (M^t)||_F ≤ C₀ for all $M^t\in {ℝ}^{(K-1)\times pq}$. As a consequence, P_t is Lipschitz continuous with the uniform constant L₀ that is independent of t, i.e., $\left|P_t(u)-P_t(\widehat{u})\right|\le L_0\Vert u-\widehat{u}{\Vert }_F$ for all $u,\widehat{u}$. The boundedness of $dom \left(P_t^{*}\right)$ of the conjugate $P_t^{*}$ follows from [3, Corollary 17.19].

A.2. The proof of Lemma 3: The convergence of the primal-dual methods

Let $\mathcal{G}(M,Y)=Q_t(M)+\langle \mathcal{A}(M),Y\rangle -P_t^{*}(Y)$, where $P_t^{*}$ is the Fenchel conjugate of P_t. Applying [9, Theorem 4] with f = 0, for any M and Y, we have

$$\mathcal{G}\left(M_l^t,Y\right)-\mathcal{G}\left(M,{\overline{Y}}_l^t\right)\le \frac{1}{T_l}\left(\frac{\Vert M_0^t-M{\Vert }_F^2}{2{\omega }_0^t}+\frac{\Vert Y_0^t-Y{\Vert }_F^2}{2{\sigma }_0^t}\right),$$ 28)

where $T_l=\sum _{i=1}^l\frac{{\sigma }_{i-1}^t}{{\sigma }_0^t}$, and ${\overline{Y}}_l^t=\frac{1}{T_l}\sum _{j=1}^l\frac{{\sigma }_{j-1}^t}{{\sigma }_0^t}{Y}_j^t$.

By the update rule in (18), we have ${\omega }_{l+1}^t{\sigma }_{l+1}^t={\omega }_l^t{\sigma }_l^t$. Hence, by induction, we have ${\omega }_l^t{\sigma }_l^t={\omega }_0^t{\sigma }_0^t=\Vert \mathcal{A}{\Vert }^{-2}$. On the other hand, by [8, Lemma 2], with the choice of $\lambda =\Vert \mathcal{A}{\Vert }^{-1}\left(1+{\rho }_t\right)$, we have

$$\frac{\Vert \mathcal{A}\Vert }{1+{\rho }_t}+\frac{\Vert \mathcal{A}\Vert l}{\Vert \mathcal{A}\Vert +\left(1+{\rho }_t\right)}\le \frac{1}{\left(1+{\rho }_t\right){\omega }_l^t}\le \frac{\Vert \mathcal{A}\Vert }{1+{\rho }_t}+l.$$

Using this estimate and ${\sigma }_l^t=\Vert \mathcal{A}{\Vert }^{-2}{\omega }_l^{-t}$, we have

$$T_l=\sum _{i=1}^l\frac{{\sigma }_{i-1}^t}{{\sigma }_0^t}=\frac{1}{\Vert \mathcal{A}\Vert }\sum _{i=1}^l\frac{1}{{\omega }_{i-1}^t}\ge \sum _{i=1}^l\left(\frac{i-1}{1+c}+1\right)=\frac{l(l-1)}{2(1+c)}+l\ge \frac{l^2}{2(1+c)},$$

where $c=\Vert \mathcal{A}\Vert {\left(1+{\rho }_t\right)}^{-1}$. Hence, we can estimate T_l as $T_l\ge \frac{1}{2}{\left(1+{\rho }_t+\Vert \mathcal{A}\Vert \right)}^{-1}\left(1+{\rho }_t\right)l^2$. Using this estimate of T_l, $T_l,{\sigma }_0^t={\omega }_0^t=\Vert \mathcal{A}\Vert$, and ${\tilde{F}}_t\left(M_l^t\right)-{\tilde{F}}_t\left({\overline{M}}^{t+1}\right)\le \mathcal{G}\left(M_l^t,{\overline{Y}}^{t+1}\right)-\mathcal{G}\left({\overline{M}}^{t+1},{\overline{Y}}_l^t\right)$, we obtain from (28) that

$${\tilde{F}}_t\left(M_l^t\right)-{\tilde{F}}_t\left({\overline{M}}^{t+1}\right)\le \frac{\left(1+{\rho }_t+\Vert \mathcal{A}\Vert \right)\Vert \mathcal{A}\Vert }{\left(1+{\rho }_t\right)l^2}\left(\Vert M_0^t-{\overline{M}}^{t+1}{\Vert }_F^2+\Vert Y_0^t-{\overline{Y}}^{t+1}{\Vert }_F^2\right).$$

This is exactly (20).

Next, we prove (21). By introducing $Y=\mathcal{A}(M)$, we can reformulate the strongly convex subproblem (15) into the following constrained convex problem:

$${\tilde{F}}_t\left({\overline{M}}^{t+1}\right)=\underset{{}_{\text{M},\text{Y}}}{\text{min}}\left\{ {\tilde{F}}_t(\text{M},\text{Y})=P_t(\text{Y})+Q_t(\text{M})|\mathcal{A}(\text{M})-\text{Y}=0\right\}.$$ (29)

Note that Q_t is strongly convex with the strong convexity parameter 1 + ρ_t. We can apply [28, Algorithm 2] to solve (29). If we define

$${\text{Δ}}_{{\sigma }_i^t}\left({\text{M}}_{l+1}^t\right)=P_t\left({\text{Y}}_{l+1}^t\right)+Q_t\left({\text{M}}_{l+1}^t\right)+\frac{{\sigma }_l^t}{2}\Vert \mathcal{A}\left({\text{M}}_{l+1}^t\right)-{\text{Y}}_{l+1}^t{\Vert }_F^2-{\tilde{F}}_t\left({\overline{M}}^{t+1}\right).$$

then, from the proof of [28, Theorem 2], we can show that

$${\text{Δ}}_{{\sigma }_l^t}\left(M_{l+1}^t\right)\le \frac{2\left[{\sigma }_0^t\Vert \mathcal{A}{\Vert }^2+1+{\rho }_t\right]\Vert M_0^t-{\overline{M}}^{t+1}{\Vert }_F^2}{{(l+2)}^2}.$$ (30)

By Lemma 2, P_t is Lipschitz continuous with the Lipschitz constant L₀. Then we have

$$\begin{gathered} {\tilde{F}}_t\left(M_{l+1}^t\right)-{\tilde{F}}_t\left({\overline{M}}^{t+1}\right) \\ =P_t\left(\mathcal{A}\left(M_{l+1}^t\right)\right)+Q_t\left(M_{l+1}^t\right)-{\tilde{F}}_t\left({\overline{M}}^{t+1}\right)\le P_t\left(Y_{l+1}^t\right)+Q_t\left(M_{l+1}^t\right)-{\tilde{F}}_t\left({\overline{M}}^{t+1}\right)+L_0\Vert \mathcal{A}\left(M_{l+1}^t\right)-Y_{l+1}^t{\Vert }_F. \end{gathered}$$

Combining (30) and this estimate, we obtain

$$\begin{gathered} 0\le {\tilde{F}}_t\left(M_{l+1}^t\right)-{\tilde{F}}_t\left({\overline{M}}^{t+1}\right) \\ \le \frac{2\left[{\sigma }_0^t\Vert \mathcal{A}{\Vert }^2+1+{\rho }_t\right]\Vert M_0^t-{\overline{M}}^{t+1}{\Vert }_F^2}{{(l+2)}^2}+L_0\Vert \mathcal{A}\left(M_{l+1}^t\right)-Y_{l+1}^t{\Vert }_F-\frac{{\sigma }_l^t|}{2}\Vert \mathcal{A}\left(M_{l+1}^t\right)-Y_{l+1}^t{\Vert }_F^2. \end{gathered}$$

Similar to the proof of [28, Corollary 1], by using ${\sigma }_0^t=\frac{1+{\rho }_t}{2\Vert \mathcal{A}{\Vert }^2}$, the last inequality leads to

$$\Vert \mathcal{A}\left(M_{l+1}^t\right)-Y_{l+1}^t{\Vert }_F\le \frac{4\Vert \mathcal{A}\Vert }{{(l+1)}^2}\left[\frac{2L_0\Vert \mathcal{A}\Vert }{1+{\rho }_t}+\sqrt{3}\Vert M_0^t-{\overline{M}}^{t+1}{\Vert }_F\right].$$

Combining the two last estimates, we obtain

$$\tilde{F}\left(M_l^t\right)-\tilde{F}\left({\overline{M}}^{t+1}\right)\le \frac{4\Vert \mathcal{A}\Vert L_0}{{(l+1)}^2}\left[\frac{2L_0\Vert \mathcal{A}\Vert }{1+{\rho }_t}+\sqrt{3}\Vert M_0^t-{\overline{M}}^{t+1}{\Vert }_F\right]+\frac{3\left({\rho }_t+1\right)\Vert M_0^t-{\overline{M}}^{t+1}{\Vert }_F^2}{{(l+1)}^2},$$

which is exactly (21). ◻

A.3. Proof of statistical properties

We provide the proof of Theorems 3 and 4 in this section.

A.3.1. Proof of Theorem 3: Fisher’s consistency

In our RMSMM (3), one can abstract the truncated hinge loss function as

$$\varphi (f(X),y)=\gamma T_{(K-1)s}\left(\langle f(X),w_y\rangle \right)+(1-\gamma )\sum _{k\ne y}{R}_s\left(\langle f(X),w_k\rangle \right).$$

Then, the conditional loss can be rewritten as

$$L(X)\triangleq \sum _{k=1}^K\left[\gamma P_k{T}_{(K-1)s}\left(\langle f(X),w_k\rangle \right)+\left(1-P_k\right)R_s\left(\langle f(X),w_k\rangle \right)\right].$$

[34, Theorem 1] showed that for a vector data x, the robust classifier based on the loss function ϕ(f(x), y) is Fisher consistent with $\gamma \in \left[0,\frac{1}{2}\right]$ and s ≤ 0. By vectorizing the matrix data X to a new vector x = vec(X), then all settings here are the same as those of Theorem 1 in [34]. In this case, Fisher consistency results can naturally be transferred to matrix-type data. ◻

A.3.2. Proof of Theorem 4: Misclassification rates

First, we introduce the Rademacher complexity. Let $\mathcal{G}=\left\{g:X\times \mathcal{Y}\to ℝ\right\}$ be a class of loss functions. Given the sample $\mathcal{T}={\left\{\left(X_i,y_i\right)\right\}}_{i=1}^N$, we define the empirical Rademacher complexity of $\mathcal{G}$ as

$${\widehat{R}}_N(\mathcal{G})=E_{\sigma }\left\{\underset{g\in \mathcal{G}}{\text{sup}}\frac{1}{N}\sum _{i=1}^N{\sigma }_{i}g\left(X_i,y_i\right)\right\},$$

where $\sigma ={\left\{{\sigma }_i\right\}}_{i=1}^N$ are i.i.d. random variables with Pr(σ₁ = 1) = Pr(σ₁ = −1) = 1/2. The Rademacher complexity of $\mathcal{G}$ is defined as

$$R_N(\mathcal{G})=E_{\sigma ,\mathcal{T}}\left\{\underset{g\in \mathcal{G}}{\text{sup}}\frac{1}{N}\sum _{i=1}^N{\sigma }_{i}g\left(X_i,y_i\right)\right\}.$$

For our model, let

$$H=\left\{h(\text{X},y)={\text{min}}_{k\ne y}\left(\left(\text{f}(\text{X}),{\text{w}}_y-{\text{w}}_k\right)\right)|f\in \mathcal{F},\sum _j\Vert {\text{M}}_j{\Vert }_{*}\le r\right\},$$

and

$${\mathbb{I}}_{\kappa }(x)=\{\begin{array}{l}1x<0,\hfill \\ 1-\frac{1}{\kappa }x 0\le x\le \kappa ,\hfill \\ 0otherwise.\hfill \end{array}$$

To prove Theorem 4, we first recall the following lemma which provides a bound on $E\left[{\mathbb{I}}_{\kappa }\left\{h(X,y)\right\}\right]$ by the empirical error and the Rademacher complexity.

Lemma 4

For any h ∈ H, with probability at least 1 – δ, we have

$$E\left[{\mathbb{I}}_{\kappa }\left\{h(X,y)\right\}\right]\le \frac{1}{N}\sum _{i=1}^N{\mathbb{I}}_{\kappa }\left\{h\left(X_i,y_i\right)\right\}+2R_N\left({\mathbb{I}}_{\kappa }\circ H\right)+{\left\{\frac{\text{log}\left({\delta }^{-1}\right)}{N}\right\}}^{1/2}.$$

The proof of Lemma 4 can be found in [34].

Now, we need to derive the upper bound of the Rademacher complexity used in Lemma 4. Since ${\mathbb{I}}_{\kappa }$ is $\frac{1}{\kappa }$-Lipschitz, we have

$$\begin{gathered} R_N\left({\mathbb{I}}_{\kappa }\circ H\right)\le \frac{1}{\kappa }{E}_{\sigma ,\mathcal{T}}\left\{\underset{\Sigma \Vert M_j\Vert +\le r}{\text{sup}}\frac{1}{N}\sum _{i=1}^N{\sigma }_i\sum _{j=1}^{K-1} tr \left(M_j^{\top }{\tilde{X}}_i\right)\right\} \\ =\frac{r}{\kappa N}{E}_{\sigma ,\mathcal{T}}\left\{\Vert \sum _{i=1}^N{\sigma }_i{\tilde{X}}_i{\Vert }_2\right\}, \end{gathered}$$

where ${\tilde{X}}_i$ denotes ${\text{X}}_i-\overline{X}$ and $\overline{X}=N^{-1}{\sum }_{i=1}^N{X}_i$. The first inequality is due to Lemma 4.2 in [20], and the absolute values of the entries in w_y − w_k are all bounded by 1.

Firstly, by the assumption, we can write X = E(X) + E, where $E(X)={\sum }_{k=1}^{K}Pr(\mathcal{Y}=k)C_k$ and the variance and the fourth moment of the entries are σ² and ${\mu }_4^4$. Accordingly, ${\tilde{X}}_i=E_i-\overline{E}$, where $\overline{E}=N^{-1}{\sum }_{i=1}^N{E}_i$. Since ${\left\{\left(X_i,y_i\right)\right\}}_{i=1}^N$ are the i.i.d. copies of $(X,\mathcal{Y})$, we have

$$\Vert \sum _{i=1}^N{\sigma }_i{\tilde{X}}_i{\Vert }_2\le \left|\sum _{i=1}^N{\sigma }_i\right|\Vert \overline{E}{\Vert }_2+\Vert \sum _{i=1}^N{\sigma }_i{E}_i{\Vert }_2.$$

Because $E\left[{\left({\sum }_{i=1}^N{\sigma }_i{E}_i\right)}^2\right]=N{\sigma }^2$ and $E\left[{\left({\sum }_{i=1}^N{\sigma }_i{E}_i\right)}^4\right]=N{\mu }_4^4+3N(N-1){\sigma }^4$, by Theorem 2 in [16] we have

$$E_{\sigma ,\mathcal{T}}\left(\Vert \sum _{i=1}^N{\sigma }_i{E}_i{\Vert }_2\right)\le c\sigma N^{1/2}\left\{p^{1/2}+q^{1/2}+{(pq)}^{1/4}{\left[N{\mu }_4^4+3N(N-1){\sigma }^4\right]}^{1/4}/\left(\sigma N^{1/2}\right)\right\}\le c\sigma \left(1+\frac{3^{1/4}}{2}\right)N^{1/2}\left\{p^{1/2}+q^{1/2}\right\}+O\left(N^{1/4}\left(p^{1/2}+q^{1/2}\right)\right),$$

where c is a constant which does not depend on $\mathcal{T}$. By similar arguments, it is easy to see that

$$\begin{gathered} E_{\sigma ,\mathcal{T}}\left(\left|\sum _{i=1}^N{\sigma }_i\right|\Vert \overline{E}{\Vert }_2\right)\le \sqrt{E_{\sigma }\left\{{\left(\sum _{i=1}^N{\sigma }_i\right)}^2\right\}{E}_{\mathcal{T}}\left(\Vert \overline{E}{\Vert }_2\right)} \\ =N^{1/2}{E}_T\left(\Vert \overline{E}{\Vert }_2\right)=O\left(p^{1/2}+q^{1/2}\right). \end{gathered}$$

Accordingly, we obtain the upper bound of the Rademacher complexity as

$$R_N\left({\mathbb{I}}_{\kappa }\circ H\right)\le \frac{r}{\kappa \sqrt{N}}\left\{c\sigma \left(1+\frac{3^{1/4}}{2}\right)\left(p^{1/2}+q^{1/2}\right)\right\}.$$

The proof is completed by using Lemma 4 with this bound and the fact that the continuous indicator function ${\mathbb{I}}_{\kappa }$ is an upper bound of the indicator function for any κ. ◻

A.3.3. Proof of Theorem 5: Breakdown Point Analysis

Let F(M, $\mathcal{T}$) denote the loss function (3) with the sample $\mathcal{T}$, and

$${\text{Δ}}^{+}\triangleq \left\{M | \forall k, s.t. w_k^{\top }\widehat{M}{M}^{\top }{w}_k\rangle 0\right\}\text{ and }{\text{Δ}}^{-}\triangleq \left\{M | \exists k, s.t. w_k^{\top }\widehat{M}{M}^{\top }{w}_k\le 0\right\}.$$

For the MSMM classifier, we can choose the contaminated observation as (X^o, k) with $\text{vec} {\left(X^o\right)}^{\top }=-c{w}_k^{\top }\widehat{M}$. For any M ∈ Δ⁺, $w_k^{\top }\widehat{M}{M}^{\top }{w}_k>0$, then $w_k^{\top }M\text{vec} \left(X^o\right)=-c{w}_k^{\top }\widehat{M}{M}^{\top }{w}_k\to -\infty$ as c → ∞. In this situation, the loss term corresponding to this contaminated observation will tend to infinity. Hence, we have $\tilde{M}\in {\text{Δ}}^{-}$ and the classifier breaks down.

For the RMSMM, since $\widehat{M}\ne 0,\widehat{M}$ is an interior point of Δ⁺, the claim

$${ϵ}_1=\underset{M\in {\text{Δ}}^{-}}{\text{min}}F\left(M,{\mathcal{T}}_n\right)-\underset{M\in {\text{Δ}}^{+}}{\text{min}}F\left(M,{\mathcal{T}}_n\right)>0$$

is true. Note that the loss function

$$l(X,\mathcal{Y},M)=\gamma T_{s(K-1)}\left(w_y^{\top }M\text{vec} (X)\right)+(1-\gamma )\sum _{k\ne \mathcal{Y}}{R}_s\left(w_k^{\top }M\text{vec} (X)\right)$$

is bounded by (K − 1)(1 − s). For any m ≤ nϵ₁/[2(1 + δ)(K − 1)(1 − s)] with δ > 0 being any positive constant, any corresponding n −m clean subset ${\mathcal{T}}_{n-m}\subset {\mathcal{T}}_n$, and any $M\in {ℝ}^{p\times q}$, we have

$$0\le F\left(M,{\mathcal{T}}_n\right)-\frac{n-m}{n}F\left(M,{\mathcal{T}}_{n-m}\right)=\frac{1}{n}\sum _{i\in {\mathcal{T}}_n\backslash {\mathcal{I}}_{n-m}}l\left(X_i,y_i,M\right)\le \frac{m(K-1)(1-s)}{n}<\frac{{ϵ}_1}{2+2\delta }.$$

Therefore,

$$\left|\underset{\text{M}\in {\text{Δ}}^{-}}{\text{min}}F\left(\text{M},{\mathcal{T}}_n\right)-\underset{\text{M}\in {\text{Δ}}^{+}}{\text{min}}F\left(\text{M},{\mathcal{T}}_n\right)-\underset{\text{M}\in {\text{Δ}}^{-}}{\text{min}}F\left(\text{M}, {\tilde{T}}_{n,m}\right)+\underset{\text{M}\in {\text{Δ}}^{+}}{\text{min}}F\left(\text{M},{\tilde{T}}_{n,m}\right)\right|\le \frac{{ϵ}_1}{1+\delta },$$

and

$$\underset{\text{M}\in {\text{Δ}}^{-}}{\text{min}}F\left(\text{M},{\tilde{T}}_{n,m}\right)-\underset{\text{M}\in {\text{Δ}}^{+}}{\text{min}}F\left(\text{M},{\tilde{T}}_{n,m}\right)>\frac{{ϵ}_1\delta }{1+\delta }>0.$$

The last inequality reveals that $\tilde{M}\in {\text{Δ}}^{+}$ and thus the classifier would not break down when m ≤ nϵ₁/[2(1 + δ)(K − 1)(1 − s)] observations are contaminated. Finally, the proof is complete by setting δ → 0. ◻

A.4. Derivation of Eq. (2): The dual problem

Lemma 5

For a p × q real matrix A, the subdifferential of the nuclear norm ∥·∥_* is given as

$$\partial \Vert A{\Vert }_{*}=\left\{U_A{V}_A^{\top }+Z|Z\in {ℝ}^{p\times q},U_A^{\top }Z=0,Z{V}_A=0,\Vert Z{\Vert }_2\le 1\right\},$$

where U_AΣ_AV_A^⊤ is the SVD of A, and ∂ stands for the operator of subgradients.

Lemma 6

Suppose that $X\in {ℝ}^{p\times q}$, ∂G(X) = ρX − P + τ∂∥X∥_*, where $P\in {ℝ}^{p\times q}$ is a constant matrix w.r.t. X. Let the SVD of P be

$$P=U_0{\Sigma }_0{V}_0^{\top }+U_1{\Sigma }_1{V}_1^{\top },$$

where Σ₀ contains the singular values of P which are greater than τ, and Σ₁ contains the rest. Then, we have 0 ∈ ∂G(X*), where $X^{*}={\rho }^{-1}{\mathcal{D}}_{\tau }(P)={\rho }^{-1}{U}_0\left({\Sigma }_0-\tau I\right)V_0^{\top }$.

Lemma 6 can be verified by using Lemma 5 with $Z={\tau }^{-1}{U}_1{\Sigma }_1{V}_0^{\top }$.

Now we derive the dual problem (2) of (1). As in the classical SVM, by setting C = (Nλ)⁻¹, we can rewrite (1) into the following form:

$$\{\begin{array}{l}\underset{{}_{\text{M},b,\xi }}{\text{min}}\left\{\frac{1}{2} tr \left(M^{\top }M\right)+\tau \Vert M{\Vert }_{*}+C\sum _{i=1}^N{\xi }_i\right\}\hfill \\ \text{s.t }{\xi }_i\ge 0, y_i\left[ tr \left(M^{\top }{X}_i\right)+b\right]\ge 1-{\xi }_i, i=1,\cdots ,N.\hfill \end{array}$$

The corresponding Lagrange function of this problem can be written as

$$L_P(\text{M},b,\xi ,\alpha ,\mu )=\frac{1}{2} tr \left(M^{\top }M\right)+\tau \Vert M{\Vert }_{*}+C\sum _{i=1}^N{\xi }_i-\sum _{i=1}^N{\alpha }_i\left[y_i\left\{ tr \left(M^{\top }{X}_i\right)+b\right\}-1+{\xi }_i\right]-\sum _{i=1}^N{\mu }_i{\xi }_i,$$ (31)

where α_i ≥ 0 and μ_i ≥ 0 are corresponding Lagrange multipliers. By setting the derivatives w.r.t. b and ξ_i of this Lagrange function to zero, we get

$$\{\begin{array}{l}\sum _{i=1}^N{\alpha }_i{y}_i =0,\\ C-{\alpha }_i-{\mu }_i=0,i=1,\cdots ,N.\end{array}$$

Based on Lemma 6 and setting the derivative w.r.t. M to zero, we have $M={\mathcal{D}}_{\tau }\left({\sum }_{i=1}^N{\alpha }_i{y}_i{X}_i\right)$. Substituting these conditions into (31), we obtain

$$\{\begin{array}{l}\underset{\alpha }{\text{min}}\left\{\frac{1}{2}\Vert {\mathcal{D}}_{\tau }\left(\sum _{i=1}^N{\alpha }_i{y}_i{X}_i\right){\Vert }_F^2-\sum _{i=1}^N{\alpha }_i\right\}\hfill \\ \text{s.t }0\le {\alpha }_i\le C;i=1,\dots ,N,\sum _{i=1}^N{\alpha }_i{y}_i=0.\hfill \end{array}$$

This gives us the dual problem (2) of (1).