Multi-Stage Multi-Task Feature Learning

Abstract

Multi-task sparse feature learning aims to improve the generalization performance by exploiting the shared features among tasks. It has been successfully applied to many applications including computer vision and biomedical informatics. Most of the existing multi-task sparse feature learning algorithms are formulated as a convex sparse regularization problem, which is usually suboptimal, due to its looseness for approximating an ${\ell }_0$-type regularizer. In this paper, we propose a non-convex formulation for multi-task sparse feature learning based on a novel regularizer. To solve the non-convex optimization problem, we propose a Multi-Stage Multi-Task Feature Learning (MSMTFL) algorithm. Moreover, we present a detailed theoretical analysis showing that MSMTFL achieves a better parameter estimation error bound than the convex formulation. Empirical studies on both synthetic and real-world data sets demonstrate the effectiveness of MSMTFL in comparison with the state of the art multi-task sparse feature learning algorithms.

1 Introduction

Multi-task learning (MTL) exploits the relationships among multiple related tasks to improve the generalization performance. It has been applied successfully to many applications such as speech classification [16], handwritten character recognition [14, 17] and medical diagnosis [2]. One common assumption in multi-task learning is that all tasks should share some common structures including the prior or parameters of Bayesian models [18, 21, 24], a similarity metric matrix [16], a classification weight vector [6], a low rank subspace [4, 13] and a common set of shared features [1, 8, 10, 11, 12, 14, 20].

In this paper, we focus on multi-task feature learning, in which we learn the features specific to each task as well as the common features shared among tasks. Although many multi-task feature learning algorithms have been proposed, most of them assume that the relevant features are shared by all tasks. This is too restrictive in real-world applications [9]. To overcome this limitation, Jalali et al. (2010) [9] proposed an ${\ell }_1+{\ell }_{1,\infty }$ regularized formulation, called dirty model, to leverage the common features shared among tasks. The dirty model allows a certain feature to be shared by some tasks but not all tasks. Jalali et al. (2010) also presented a theoretical analysis under the incoherence condition [5, 15] which is more restrictive than RIP [3, 27]. The ${\ell }_1+{\ell }_{1,\infty }$ regularizer is a convex relaxation for the ${\ell }_0$-type one, which, however, is too loose to well approximate the ${\ell }_0$-type regularizer and usually achieves suboptimal performance (requiring restrictive conditions or obtaining a suboptimal error bound) [23, 26, 27]. To remedy the shortcoming, we propose to use a non-convex regularizer for multi-task feature learning in this paper.

Contributions

We propose to employ a capped-${\ell }_1,{\ell }_1$ regularized formulation (non-convex) to learn the features specific to each task as well as the common features shared among tasks. To solve the non-convex optimization problem, we propose a Multi-Stage Multi-Task Feature Learning (MSMTFL) algorithm, using the concave duality [26]. Although the MSMTFL algorithm may not obtain a globally optimal solution, we theoretically show that this solution achieves good performance. Specifically, we present a detailed theoretical analysis on the parameter estimation error bound for the MSMTFL algorithm. Our analysis shows that, under the sparse eigenvalue condition which is weaker than the incoherence condition in Jalali et al. (2010) [9], MSMTFL improves the error bound during the multi-stage iteration, i.e., the error bound at the current iteration improves the one at the last iteration. Empirical studies on both synthetic and real-world data sets demonstrate the effectiveness of the MSMTFL algorithm in comparison with the state of the art algorithms.

Notations

Scalars and vectors are denoted by lower case letters and bold face lower case letters, respectively. Matrices and sets are denoted by capital letters and calligraphic capital letters, respectively. The ${\ell }_1$ norm, Euclidean norm, ${\ell }_{\infty }$ norm and Frobenius norm are denoted by ∥ · ∥₁, ∥ · ∥, ∥ · ∥_∞ and ∥ · ∥_F, respectively. | · | denotes the absolute value of a scalar or the number of elements in a set, depending on the context. We define the ${\ell }_{p,q}$ norm of a matrix X as ${\Vert X\Vert }_{p,q}={\left({\sum }_i{\left({\left({\sum }_j{\mid x_{ij}\mid }^q\right)}^{1∕q}\right)}^p\right)}^{1∕p}$. We define ${\mathbb{N}}_n$ as {1, …, n} and N(μ, σ²) as a normal distribution with mean μ and variance σ². For a d×m matrix W and sets ${\mathcal{I}}_i\subseteq {\mathbb{N}}_d\times \left\{i\right\},\mathcal{I}\subseteq {\mathbb{N}}_d\times {\mathbb{N}}_d$, we let ${\mathrm{w}}_{{\mathcal{I}}_i}$ be a d × 1 vector with the j-th entry being w_ji, if $(j,i)\in {\mathcal{I}}_i$, and 0, otherwise. We also let $W_{\mathcal{I}}$ be a d × m matrix with the (j, i)-th entry being w_ji, if $(j,i)\in \mathcal{I}$, and 0, otherwise.

2 The Proposed Formulation

Assume we are given m learning tasks associated with training data {(X₁, y₁), …, (X_m, y_m)}, where $X_i\in {\mathbb{R}}^{n_i\times d}$ is the data matrix of the i-th task with each row as a sample; ${\mathrm{y}}_i\in {\mathbb{R}}^{n_i}$ is the response of the i-th task; d is the data dimensionality; n_i is the number of samples for the i-th task. We consider learning a weight matrix $W=[{\mathrm{w}}_1,\cdots ,{\mathrm{w}}_m]\in {\mathbb{R}}^{d\times m}$ consisting of the weight vectors for m linear predictive models: y_i ≈ f_i(X_i) = X_iw_i, ${\mathrm{y}}_i\approx {\mathbf{f}}_i\left(X_i\right)=X_i{\mathrm{w}}_i,i\in {\mathbb{N}}_m$. In this paper, we propose a non-convex multi-task feature learning formulation to learn these m models simultaneously, based on the capped-${\ell }_1,{\ell }_1$ regularization. Specifically, we first impose the ${\ell }_1$ penalty on each row of W, obtaining a column vector. Then, we impose the capped-${\ell }_1$ penalty [26, 27] on that vector. Formally, we formulate our proposed model as follows:

$$\underset{W}{\min }\left\{l\left(W\right)+\lambda \sum _{j=1}^d\min ({\Vert {\mathbf{w}}^j\Vert }_1,\theta )\right\},$$ (1)

where l(W) is an empirical loss function of (W; λ (> 0) is a parameter balancing the empirical loss and the regularization; θ (> 0) is a thresholding parameter; w^j is the j-th row of the matrix W. In this paper, we focus on the quadratic loss function: $l\left(W\right)={\sum }_{i=1}^m\frac{1}{m{n}_i}{\Vert X_i{\mathrm{w}}_i-{\mathrm{y}}_i\Vert }^2$.

Intuitively, due to the capped-${\ell }_1,{\ell }_1$ penalty, the optimal solution of Eq. (1) denoted as W* has many zero rows. For a nonzero row (w*)^k, some entries may be zero, due to the ${\ell }_1$-norm imposed on each row of W. Thus, under the formulation in Eq. (1), a certain feature can be shared by some tasks but not all the tasks. Therefore, the proposed formulation can leverage the common features shared among tasks.

The formulation in Eq. (1) is non-convex and is difficult to solve. To this end, we propose a Multi-Stage Multi-Task Feature Learning (MSMTFL) algorithm (see Algorithm 1). Note that if we terminate the algorithm with $\ell =1$, the MSMTFL algorithm is equivalent to the ${\ell }_1$ regularized multi-task feature learning algorithm (Lasso). Thus, the solution obtained by MSMTFL can be considered as a refinement of that of Lasso. Although Algorithm 1 may not find a globally optimal solution, the solution has good performance. Specifically, we will theoretically show that the solution obtained by Algorithm 1 improves the performance of the parameter estimation error bound during the multi-stage iteration. Moreover, empirical studies also demonstrate the effectiveness of our proposed MSMTFL algorithm. We provide more details about intuitive interpretations, convergence analysis and reproducibility discussions of the proposed algorithm in the full version [7].

3 Theoretical Analysis

In this section, we theoretically analyze the parameter estimation performance of the solution obtained by the MSMTFL algorithm. To simplify the notations in the theoretical analysis, we assume that the number of samples for all the tasks are the same. However, our theoretical analysis can be easily extended to the case where the tasks have different sample sizes.

We first present a sub-Gaussian noise assumption which is very common in the analysis of sparse regularization literature [23, 25, 26, 27].

Assumption 1 Let $\stackrel{‒}{W}=[{\stackrel{‒}{\mathrm{w}}}_1,\cdots ,{\stackrel{‒}{\mathrm{w}}}_m]\in {\mathbb{R}}^{d\times m}$ be the underlying sparse weight matrix and ${\mathrm{y}}_i=X_i{\stackrel{‒}{\mathrm{w}}}_i+{\delta }_i$, $\mathbb{E}{\mathrm{y}}_i=X_i{\stackrel{‒}{\mathrm{w}}}_i$, where ${\delta }_i\in {\mathbb{R}}_n$ is a random vector with all entries ${\delta }_{ji}(j\in {\mathbb{N}}_n,i\in {\mathbb{N}}_m)$ being independent sub-Gaussians: there exists σ > 0 such that ${\forall }_j\in {\mathbb{N}}_n,i\in {\mathbb{N}}_m,t\in \mathbb{R}:{\mathbb{E}}_{{\delta }_{ji}}\text{exp}\left(t{\delta }_{ji}\right)\le \text{exp}({\sigma }^2{t}^2∕2)$.

Remark 1 We call the random variable satisfying the condition in Assumption 1 sub-Gaussian, since its moment generating function is upper bounded by that of the zero mean Gaussian random variable. That is, if a normal random variable x ~ N(0, σ²), then we have $\mathbb{E}\text{exp}\left(tx\right)={\int }_{-\infty }^{\infty }\text{exp}\left(tx\right)\frac{1}{\sqrt{2\pi }\sigma }\text{exp}\left(-\frac{x^2}{2{\sigma }^2}\right)dx=\text{exp}({\sigma }^2{t}^2∕2){\int }_{-\infty }^{\infty }\frac{1}{\sqrt{2\pi }\sigma }\text{exp}(-{(x-{\sigma }^{2}t)}^2∕(2{\sigma }^2\left)\right)dx=\text{exp}({\sigma }^2{t}^2∕2)\ge {\mathbb{E}}_{{\delta }_{ji}}\text{exp}\left(t{\delta }_{ji}\right)$.

Remark 2 Based on the Hoeffding's Lemma, for any random variable x ∊ [a, b] and $\mathbb{E}x=0$, we have $\mathbb{E}\left(\text{exp}\right(tx\left)\right)\le \text{exp}\left(\frac{t^2{(b-a)}^2}{8}\right)$. Therefore, both zero mean Gaussian and zero mean bounded random variables are sub-Guassians. Thus, the sub-Gaussian noise assumption is more general than the Gaussian noise assumption which is commonly used in the literature [9, 11].

We next introduce the following sparse eigenvalue concept which is also common in the analysis of sparse regularization literature [22, 23, 25, 26, 27].

Definition 1 Given 1 ≤ k ≤ d, we define

$${\rho }_i^{+}\left(k\right)=\underset{\mathbf{w}}{\sup }\left\{\frac{{\Vert X_i\mathbf{w}\Vert }^2}{n{\Vert \mathbf{w}\Vert }^2}:{\Vert \mathbf{w}\Vert }_0\le k\right\},{\rho }_{max}^{+}\left(k\right)=\underset{i\in {\mathbb{N}}_m}{\max }{\rho }_i^{+}\left(k\right),$$

$${\rho }_i^{-}\left(k\right)=\underset{\mathbf{w}}{\inf }\left\{\frac{{\Vert X_i\mathbf{w}\Vert }^2}{n{\Vert \mathbf{w}\Vert }^2}:{\Vert \mathbf{w}\Vert }_0\le k\right\},{\rho }_{min}^{-}\left(k\right)=\underset{i\in {\mathbb{N}}_m}{\min }{\rho }_i^{-}\left(k\right).$$

Remark 3 ${\rho }_i^{+}\left(k\right)\left({\rho }_i^{-}\right(k\left)\right)$ is in fact the maximum (minimum) eigenvalue of ${\left(X_i\right)}_{\mathcal{S}}^T{\left(X_i\right)}_{\mathcal{S}}∕n$, where $\mathcal{S}$ is a set satisfying $\mid \mathcal{S}\mid \le k$ and ${\left(X_i\right)}_{\mathcal{S}}$ is a submatrix composed of the columns of X_i indexed by $\mathcal{S}$. In the MTL setting, we need to exploit the relations of ${\rho }_i^{+}\left(k\right)\left({\rho }_i^{-}\right(k\left)\right)$ among multiple tasks.

We present our parameter estimation error bound on MSMTFL in the following theorem:

Theorem 1 Let Assumption 1 hold. Define ${\stackrel{‒}{\mathcal{F}}}_i=\left\{\right(j,i):{\stackrel{‒}{w}}_{ji}\ne 0\}$ and $\stackrel{‒}{\mathcal{F}}={\cup }_{i\in {\mathbb{N}}_m}{\stackrel{‒}{\mathcal{F}}}_i$. Denote $\stackrel{‒}{r}$ as the number of nonzero rows of $\stackrel{‒}{W}$. We assume that

$$\forall (j,i)\in \stackrel{‒}{\mathcal{F}},{\Vert {\stackrel{‒}{\mathbf{w}}}^j\Vert }_1\ge 2\theta$$ (3)

$$\text{and}\frac{{\rho }_i^{+}\left(s\right)}{{\rho }_i^{-}(2\stackrel{‒}{r}+2s)}\le 1+\frac{s}{2\stackrel{‒}{r}},$$ (4)

where s is some integer satisfying $s\ge \stackrel{‒}{r}$. If we choose λ and θ such that for some $s\ge \stackrel{‒}{r}$:

$$\lambda \ge 12\sigma \sqrt{\frac{2{\rho }_{max}^{+}\left(1\right)\ln (2dm∕\eta )}{n},}$$ (5)

$$\theta \ge \frac{11m\lambda }{{\rho }_{min}^{-}(2\stackrel{‒}{r}+s)},$$ (6)

then the following parameter estimation error bound holds with probability larger than 1 − η:

$${\Vert {\widehat{W}}^{\left(\ell \right)}-\stackrel{‒}{W}\Vert }_{2,1}\le {0.8}^{\ell ∕2}\frac{9.1m\lambda \sqrt{\stackrel{‒}{r}}}{{\rho }_{min}^{-}(2\stackrel{‒}{r}+s)}+\frac{39.5m\sigma \sqrt{{\rho }_{max}^{+}\left(\stackrel{‒}{r}\right)(7.4\stackrel{‒}{r}+2.7\ln (2∕\eta \left)\right)∕n}}{{\rho }_{min}^{-}(2\stackrel{‒}{r}+s)},$$ (7)

where ${\widehat{W}}^{\left(\ell \right)}$ is a solution of Eq. (2).

Remark 4 Eq. (3) assumes that the ${\ell }_1$-norm of each nonzero row of $\stackrel{‒}{W}$ is away from zero. This requires the true nonzero coefficients should be large enough, in order to distinguish them from the noise. Eq. (4) is called the sparse eigenvalue condition [27], which requires the eigenvalue ratio ${\rho }_i^{+}\left(s\right)∕{\rho }_i^{-}\left(s\right)$ to grow sub-linearly with respect to s. Such a condition is very common in the analysis of sparse regularization [22, 25] and it is slightly weaker than the RIP condition [3, 27].

Remark 5 When $\ell =1$ (corresponds to Lasso), the first term of the right-hand side of Eq. (7) dominates the error bound in the order of

$${\Vert {\widehat{W}}^{Lasso}-\stackrel{‒}{W}\Vert }_{2,1}=O\left(m\sqrt{\stackrel{‒}{r}\ln (dm∕\eta )∕n}\right),$$ (8)

since λ satisfies the condition in Eq. (5). Note that the first term of the right-hand side of Eq. (7) shrinks exponentially as ℓ increases. When ℓ is sufficiently large in the order of $O\left(\text{ln}\right(m\sqrt{\stackrel{‒}{r}∕n})+\text{ln}\text{ln}(dm\left)\right)$, this term tends to zero and we obtain the following parameter estimation error bound:

$${\Vert {\widehat{W}}^{\left(\ell \right)}-\stackrel{‒}{W}\Vert }_{2,1}=O\left(m\sqrt{\stackrel{‒}{r}∕n+\ln (1∕\eta )∕n}\right).$$ (9)

Jalali et al. (2010) [9] gave an ${\ell }_{\infty ,\infty }$-norm error bound ${\Vert {\widehat{W}}^{\mathit{Dirty}}-\stackrel{‒}{W}\Vert }_{\infty ,\infty }=O\left(\sqrt{\text{ln}(dm∕\eta )∕n}\right)$ as well as a sign consistency result between $\widehat{W}$ and $\stackrel{‒}{W}$. A direct comparison between these two bounds is difficult due to the use of different norms. On the other hand, the worst-case estimate of the ${\ell }_{2,1}$-norm error bound of the algorithm in Jalali et al. (2010) [9] is in the same order with Eq. (8), that is: ${\Vert {\widehat{W}}^{\mathit{Dirty}}-\stackrel{‒}{W}\Vert }_{2,1}=O\left(m\sqrt{\stackrel{‒}{r}\text{ln}(dm∕\eta )∕n}\right)$. When dm is large and the ground truth has a large number of sparse (i.e., $\stackrel{‒}{r}$ is a small constant), the bound in Eq. (9) is significantly better than the ones for the Lasso and Dirty model.

Remark 6 Jalali et al. (2010) [9] presented an ${\ell }_{\infty ,\infty }$-norm parameter estimation error bound and hence a sign consistency result can be obtained. The results are derived under the incoherence condition which is more restrictive than the RIP condition and hence more restrictive than the sparse eigenvalue condition in Eq. (4). From the viewpoint of the parameter estimation error, our proposed algorithm can achieve a better bound under weaker conditions. Please refer to [19, 25, 27] for more details about the incoherence condition, the RIP condition, the sparse eigenvalue condition and their relationships.

Remark 7 The capped-${\ell }_1$ regularized formulation in Zhang (2010) [26] is a special case of our formulation when m = 1. However, extending the analysis from the single task to the multi-task setting is nontrivial. Different from previous work on multi-stage sparse learning which focuses on a single task [26, 27], we study a more general multi-stage framework in the multi-task setting. We need to exploit the relationship among tasks, by using the relations of sparse eigenvalues ${\rho }_i^{+}\left(k\right)\left({\rho }_i^{-}\right(k\left)\right)$ and treating the ${\ell }_1$-norm on each row of the weight matrix as a whole for consideration. Moreover, we simultaneously exploit the relations of each column and each row of the matrix.

4 Proof Sketch

We first provide several important lemmas (please refer to the full version [7] or supplementary materials for detailed proofs) and then complete the proof of Theorem 1 based on these lemmas.

Lemma 1 Let $\stackrel{‒}{{\rm Y}}=[{\stackrel{‒}{∊}}_1,\cdots ,{\stackrel{‒}{∊}}_m]$ with ${\stackrel{‒}{∊}}_i={[{\stackrel{‒}{∊}}_{1i},\cdots ,{\stackrel{‒}{∊}}_{di}]}^T=\frac{1}{n}{X}_i^T(X_i{\stackrel{‒}{\mathbf{w}}}_i-{\mathbf{y}}_i)(i\in {\mathbb{N}}_m)$. Define $\stackrel{‒}{\mathcal{H}}\supseteq \stackrel{‒}{\mathcal{F}}$ such that $(j,i)\in \stackrel{‒}{\mathcal{H}}(\forall i\in {\mathbb{N}}_m)$, provided there exists $(j,g)\in \stackrel{‒}{\mathcal{F}}$ ($\stackrel{‒}{\mathcal{H}}$ is a set consisting of the indices of all entries in the nonzero rows of $\stackrel{‒}{W}$). Under the conditions of Assumption 1 and the notations of Theorem 1, the followings hold with probability larger than 1 − η:

$${\Vert \stackrel{‒}{{\rm Y}}\Vert }_{\infty ,\infty }\le \sigma \sqrt{\frac{2{\rho }_{max}^{+}\left(1\right)\ln (2dm∕\eta )}{n}},$$ (10)

$${\Vert {\stackrel{‒}{{\rm Y}}}_{\stackrel{‒}{\mathcal{H}}}\Vert }_F^2\le m{\sigma }^2{\rho }_{\mathit{max}}^{+}\left(\stackrel{‒}{r}\right)(7.4\stackrel{‒}{r}+2.7\text{ln}(2∕\eta \left)\right)∕n.$$ (11)

Lemma 1 gives bounds on the residual correlation $\stackrel{‒}{{\rm Y}}$ with respect to $\stackrel{‒}{W}$. We note that Eq. (10) and Eq. (11) are closely related to the assumption on λ in Eq. (5) and the second term of the right-hand side of Eq. (7) (error bound), respectively. This lemma provides a fundamental basis for the proof of Theorem 1.

Lemma 2 Use the notations of Lemma 1 and consider ${\mathcal{G}}_i\subseteq {\mathbb{N}}_d\times \left\{i\right\}$ such that ${\stackrel{‒}{\mathcal{F}}}_i\cap {\mathcal{G}}_i=\varnothing (i\in {\mathbb{N}}_m)$. Let $\widehat{W}={\widehat{W}}^{\left(\ell \right)}$ be a solution of Eq. (2) and $\Delta \widehat{W}=\widehat{W}-\stackrel{‒}{W}$. Denote ${\widehat{\lambda }}_i={\widehat{\lambda }}_i^{(\ell -1)}={[{\lambda }_1^{(\ell -1)},\cdots ,{\lambda }_d^{(\ell -1)}]}^T$. Let ${\widehat{\lambda }}_{{\mathcal{G}}_i}={\min }_{(j,i)\in {\mathcal{G}}_i}{\widehat{\lambda }}_{ji}$, ${\widehat{\lambda }}_{{\mathcal{G}}_i}={\min }_{i\in {\mathcal{G}}_i}{\widehat{\lambda }}_{{\mathcal{G}}_i}$ and ${\widehat{\lambda }}_{0i}={\max }_j{\widehat{\lambda }}_{ji}$, ${\widehat{\lambda }}_0={\max }_i{\widehat{\lambda }}_{0i}$. If $2{\Vert {\stackrel{‒}{∊}}_i\Vert }_{\infty }\langle {\widehat{\lambda }}_{{\mathcal{G}}_i}$, then the following inequality holds at any stage ℓ ≥ 1:

$$\sum _{i=1}^m\sum _{(j,i)\in {\mathcal{G}}_i}\mid {\widehat{w}}_{ji}^{\left(\ell \right)}\mid \le \frac{2{\Vert \stackrel{‒}{{\rm Y}}\Vert }_{\infty ,\infty }+{\widehat{\lambda }}_0}{{\widehat{\lambda }}_{\mathcal{G}}-2{\Vert \stackrel{‒}{{\rm Y}}\Vert }_{\infty ,\infty }}\sum _{i=1}^m\sum _{(j,i)\in {\mathcal{G}}_i^c}\mid \Delta {\widehat{w}}_{ji}^{\left(\ell \right)}\mid .$$

Denote $\mathcal{G}={\cup }_{i\in {\mathbb{N}}_m}{\mathcal{G}}_i$, $\stackrel{‒}{\mathcal{F}}={\cup }_{i\in {\mathbb{N}}_m}{\stackrel{‒}{\mathcal{F}}}_i$ and notice that $\mathcal{F}\cap \mathcal{G}=\varnothing \Rightarrow \Delta {\widehat{W}}^{\left(\ell \right)}={\widehat{W}}^{\left(\ell \right)}$. Lemma 2 sasy that ${\Vert \Delta {\widehat{W}}_{\mathcal{G}}^{\left(\ell \right)}\Vert }_{1,1}={\Vert {\widehat{W}}_{\mathcal{G}}^{\left(\ell \right)}\Vert }_{1,1}$ is upper bounded in terms of ${\Vert \Delta {\widehat{W}}_{{\mathcal{G}}^c}^{\left(\ell \right)}\Vert }_{1,1}$, which indicates that the error of the estimated coefficients locating outside of $\stackrel{‒}{\mathcal{F}}$ should be small enough. This provides an intuitive explanation why the parameter estimation error of our algorithm can be small.

Lemma 3 Using the notations of Lemma 2, we denote $\mathcal{G}={\mathcal{G}}_{\left(\ell \right)}={\stackrel{‒}{\mathcal{H}}}^c\cap \left\{\right(j,i):{\widehat{\lambda }}_{ji}^{(\ell -1)}=\lambda \}={\cup }_{i\in {\mathbb{N}}_m}{\mathcal{G}}_i$ with $\stackrel{‒}{\mathcal{H}}$ being defined as in Lemma 1 and ${\mathcal{G}}_i\subseteq {\mathbb{N}}_d\times \left\{i\right\}$. Let ${\mathcal{J}}_i$ be the indices of the largest s coefficients (in absolute value) of ${\widehat{\mathbf{w}}}_{{\mathcal{G}}_i}$, ${\mathcal{I}}_i={\mathcal{G}}_i^c\cup {\mathcal{J}}_i$, $\mathcal{I}={\cup }_{i\in {\mathbb{N}}_m}{\mathcal{I}}_i$ and $\stackrel{‒}{\mathcal{F}}={\cup }_{i\in {\mathbb{N}}_m}{\stackrel{‒}{\mathcal{F}}}_i$. Then, the following inequalities hold at any stage ℓ ≥ 1:

$${\Vert \Delta {\widehat{W}}^{\left(\ell \right)}\Vert }_{2,1}\le \frac{\left(1+1.5\sqrt{\frac{2\stackrel{‒}{r}}{s}}\right)\sqrt{8m\left(4{\Vert {\stackrel{‒}{{\rm Y}}}_{{\mathcal{G}}_{\left(\ell \right)}^c}\Vert }_F^2+\sum _{(j,i)\in \stackrel{‒}{\mathcal{F}}}{\left({\widehat{\lambda }}_{ji}^{(\ell -1)}\right)}^2\right)}}{{\rho }_{\mathit{min}}^{-}(2\stackrel{‒}{r}+s)},$$ (12)

$${\Vert \Delta {\widehat{W}}^{\left(\ell \right)}\Vert }_{2,1}\le \frac{9.1m\lambda \sqrt{\stackrel{‒}{r}}}{{\rho }_{\mathit{min}}^{-}(2\stackrel{‒}{r}+s)}.$$ (13)

Lemma 3 is established based on Lemma 2, by considering the relationship between Eq. (5) and Eq. (10), and the specific definition of $\mathcal{G}={\mathcal{G}}_{\left(\ell \right)}$. Eq. (12) provides a parameter estimation error bound in terms of ℓ_2,1-norm by ${\Vert {\stackrel{‒}{{\rm Y}}}_{{\mathcal{G}}_{\left(\ell \right)}^c}\Vert }_F^2$ and the regularization parameters ${\widehat{\lambda }}_{ji}^{(\ell -1)}$ (see the definition of ${\widehat{\lambda }}_{ji}\left({\widehat{\lambda }}_{ji}^{(\ell -1)}\right)$ in Lemma 2). This is the result directly used in the proof of Theorem 1. Eq. (13) states that the error bound is upper bounded in terms of λ, the right-hand side of which constitutes the shrinkage part of the error bound in Eq. (7).

Lemma 4 Let ${\widehat{\lambda }}_{ji}=\lambda I({\Vert {\widehat{\mathbf{w}}}^j\Vert }_1\langle \theta ,j\in {\mathbb{N}}_d)$, $\forall i\in {\mathbb{N}}_m$ with some $\widehat{W}\in {\mathbb{R}}^{d\times m}$. $\stackrel{‒}{\mathcal{H}}\supseteq \stackrel{‒}{\mathcal{F}}$ is defined in Lemma 1. Then under the condition of Eq. (3), we have:

$$\sum _{(j,i)\in \stackrel{‒}{\mathcal{F}}}{\widehat{\lambda }}_{ji}^2\le \sum _{(j,i)\in \stackrel{‒}{\mathcal{H}}}{\widehat{\lambda }}_{ji}^2\le m{\lambda }^2{\Vert {\stackrel{‒}{W}}_{\stackrel{‒}{\mathcal{H}}}-{\widehat{W}}_{\stackrel{‒}{\mathcal{H}}}\Vert }_{2,1}^2∕{\theta }^2.$$

Lemma 4 establishes an upper bound of ${\Sigma }_{(j,i)\in \stackrel{‒}{\mathcal{F}}}{\widehat{\lambda }}_{ji}^2$ by ${\Vert {\stackrel{‒}{W}}_{\stackrel{‒}{\mathcal{H}}}-{\widehat{W}}_{\stackrel{‒}{\mathcal{H}}}\Vert }_{2,1}^2$, which is critical for building the recursive relationship between ${\Vert {\widehat{W}}^{\left(\ell \right)}-\stackrel{‒}{W}\Vert }_{2,1}$ and ${\Vert {\widehat{W}}^{(\ell -1)}-\stackrel{‒}{W}\Vert }_{2,1}$ in the proof of Theorem 1. This recursive relation is crucial for the shrinkage part of the error bound in Eq. (7).

4.1 Proof of Theorem 1

Proof For notational simplicity, we denote the right-hand side of Eq. (11) as:

$$u=m{\sigma }^2{\rho }_{\mathit{max}}^{+}\left(\stackrel{‒}{r}\right)(7.4\stackrel{‒}{r}+2.7\text{ln}(2∕\eta \left)\right)∕n.$$ (14)

Based on $\stackrel{‒}{\mathcal{H}}\subseteq {\mathcal{G}}_{\left(\ell \right)}^c$, Lemma 1 and Eq. (5), the followings hold with probability larger than 1 − η:

$${\Vert {\stackrel{‒}{{\rm Y}}}_{{\mathcal{G}}_{\left(\ell \right)}^c}\Vert }_F^2={\Vert {\stackrel{‒}{{\rm Y}}}_{\mathcal{H}}\Vert }_F^2+{\Vert {\stackrel{‒}{{\rm Y}}}_{{\mathcal{G}}_{\left(\ell \right)}^c\backslash \stackrel{‒}{\mathcal{H}}}\Vert }_F^2\le u+\mid {\mathcal{G}}_{\left(\ell \right)}^c\backslash \stackrel{‒}{\mathcal{H}}\mid {\Vert \stackrel{‒}{{\rm Y}}\Vert }_{\infty ,\infty }^2\le u+{\lambda }^2\mid {\mathcal{G}}_{\left(\ell \right)}^c\backslash \stackrel{‒}{\mathcal{H}}\mid ∕144\le u+(1∕144)m{\lambda }^2{\theta }^{-2}{\Vert {\widehat{W}}_{{\mathcal{G}}_{\left(\ell \right)}^c\backslash \stackrel{‒}{\mathcal{H}}}^{(\ell -1)}-{\stackrel{‒}{W}}_{{\mathcal{G}}_{\left(\ell \right)}^c\backslash \stackrel{‒}{\mathcal{H}}}\Vert }_{2,1}^2,$$ (15)

where the last inequality follows from $\forall (j,i)\in {\mathcal{G}}_{\left(\ell \right)}^c\backslash \stackrel{‒}{\mathcal{H}}$, ${\Vert {\left({\widehat{\mathbf{w}}}^{(\ell -1)}\right)}^j\Vert }_1^2∕{\theta }^2={\Vert {\left({\widehat{w}}^{(\ell -1)}\right)}^j-{\stackrel{‒}{\mathbf{w}}}^j\Vert }_1^2∕{\theta }^2\ge 1\Rightarrow \mid {\mathcal{G}}_{\left(\ell \right)}^c\backslash \stackrel{‒}{\mathcal{H}}\mid \le m{\theta }^{-2}{\Vert ({\widehat{\mathbf{W}}}_{{\mathcal{G}}_{\left(\ell \right)}^c\backslash \stackrel{‒}{\mathcal{H}}}^{(\ell -1)}-{\stackrel{‒}{W}}_{{\mathcal{G}}_{\left(\ell \right)}^c\backslash \stackrel{‒}{\mathcal{H}}})\Vert }_{2,1}^2$. According to Eq. (12), we have:

$${\Vert {\widehat{W}}^{\left(\ell \right)}-\stackrel{‒}{W}\Vert }_{2,1}^2={\Vert \Delta {\widehat{W}}^{\left(\ell \right)}\Vert }_{2,1}^2\le \frac{8m{\left(1+1.5\sqrt{\frac{2\stackrel{‒}{r}}{s}}\right)}^2\left(4{\Vert {\stackrel{‒}{{\rm Y}}}_{{\mathcal{G}}_{\left(\ell \right)}^c}\Vert }_F^2+\sum _{(j,i)\in \mathcal{F}}{\left({\widehat{\lambda }}_{ji}^{(\ell -1)}\right)}^2\right)}{{\left({\rho }_{\mathit{min}}^{-}\right(2\stackrel{‒}{r}+s\left)\right)}^2}\le \frac{78m\left(4u+(37∕36)m{\lambda }^2{\theta }^{-2}{\Vert {\widehat{W}}^{(\ell -1)}-\stackrel{‒}{W}\Vert }_{2,1}^2\right)}{{\left({\rho }_{\mathit{min}}^{-}\right(2\stackrel{‒}{r}+s\left)\right)}^2}\le \frac{312mu}{{\left({\rho }_{\mathit{min}}^{-}\right(2\stackrel{‒}{r}+s\left)\right)}^2}+0.8{\Vert {\widehat{W}}^{(\ell -1)}-\stackrel{‒}{W}\Vert }_{2,1}^2\le {0.8}^{\ell }{\Vert {\widehat{W}}^{\left(0\right)}-\stackrel{‒}{W}\Vert }_{2,1}^2+\frac{312mu}{{\left({\rho }_{\mathit{min}}^{-}\right(2\stackrel{‒}{r}+s\left)\right)}^2}\frac{1-{0.8}^{\ell }}{1-0.8}\le {0.8}^{\ell }\frac{{9.1}^2{m}^2{\lambda }^2\stackrel{‒}{r}}{{\left({\rho }_{\mathit{min}}^{-}\right(2\stackrel{‒}{r}+s\left)\right)}^2}+\frac{1560mu}{{\left({\rho }_{\mathit{min}}^{-}\right(2\stackrel{‒}{r}+s\left)\right)}^2}.$$

In the above derivation, the first inequality is due to Eq. (12); the second inequality is due to the assumption $s\ge \stackrel{‒}{r}$ in Theorem 1, Eq. (15) and Lemma 4; the third inequality is due to Eq. (6); the last inequality follows from Eq. (13) and 1 − 0.8^ℓ ≤ 1 (ℓ ≥ 1). Thus, following the inequality $\sqrt{a+b}\le \sqrt{a}+\sqrt{b}(\forall a,b\ge 0)$, we obtain:

$${\Vert {\widehat{W}}^{\left(\ell \right)}-\stackrel{‒}{W}\Vert }_{2,1}\le {0.8}^{\ell ∕2}\frac{9.1m\lambda \sqrt{\stackrel{‒}{r}}}{{\rho }_{\mathit{min}}^{-}(2\stackrel{‒}{r}+s)}+\frac{39.5\sqrt{mu}}{{\rho }_{\mathit{min}}^{-}(2\stackrel{‒}{r}+s)}.$$

Substituting Eq. (14) into the above inequality, we verify Theorem 1.

5 Experiments

We compare our proposed MSMTFL algorithm with three competing multi-task feature learning algorithms: ℓ₁-norm multi-task feature learning algorithm (Lasso), ℓ_1,2-norm multi-task feature learning algorithm (L1,2) [14] and dirty model multi-task feature learning algorithm (DirtyMTL) [9]. In our experiments, we employ the quadratic loss function for all the compared algorithms.

5.1 Synthetic Data Experiments

We generate synthetic data by setting the number of tasks as m and each task has n samples which are of dimensionality d; each element of the data matrix $X_i\in {\mathbb{R}}^{n\times d}(i\in {\mathbb{N}}_m)$ for the i-th task is sampled i.i.d. from the Gaussian distribution N(0, 1) and we then normalize all columns to length 1; each entry of the underlying true weight $\stackrel{‒}{W}\in {\mathbb{R}}^{d\times m}$ is sampled i.i.d. from the uniform distribution in the interval [−10, 10]; we randomly set 90% rows of $\widehat{W}$ as zero vectors and 80% elements of the remaining nonzero entries as zeros; each entry of the noise ${\delta }_i\in {\mathbb{R}}^n$ is sampled i.i.d. from the Gaussian distribution N(0, σ²); the responses are computed as ${\mathbf{y}}_i=X_i{\stackrel{‒}{\mathbf{w}}}_i+{\delta }_i(i\in {\mathbb{N}}_m)$

We first report the averaged parameter estimation error ${\Vert \widehat{W}-\stackrel{‒}{W}\Vert }_{2,1}$ vs. Stage (ℓ) plots for MSMTFL (Figure 1). We observe that the error decreases as ℓ increases, which shows the advantage of our proposed algorithm over Lasso. This is consistent with the theoretical result in Theorem 1. Moreover, the parameter estimation error decreases quickly and converges in a few stages.

Figure 1.

Averaged parameter estimation error ${\Vert \widehat{W}-\stackrel{‒}{W}\Vert }_{2,1}$ vs. Stage $\left(\ell \right)$ plots for MSMTFL on the synthetic data set (averaged over 10 runs). Here we set $\lambda =\alpha \sqrt{\ln \left(dm\right)∕n},\theta =50m\lambda$. Note that $\ell =1$ corresponds to Lasso; the results show the stage-wise improvement over Lasso.

We then report the averaged parameter estimation error ${\Vert \widehat{W}-\stackrel{‒}{W}\Vert }_{2,1}$ in comparison with four algorithms in different parameter settings (Figure 2). For a fair comparison, we compare the smallest estimation errors of the four algorithms in all the parameter settings [25, 26]. As expected, the parameter estimation error of the MSMTFL algorithm is the smallest among the four algorithms. This empirical result demonstrates the effectiveness of the MSMTFL algorithm. We also have the following observations: (a) When λ is large enough, all four algorithms tend to have the same parameter estimation error. This is reasonable, because the solutions $\widehat{W}$'s obtained by the four algorithms are all zero matrices, when λ is very large. (b) The performance of the MSMTFL algorithm is similar for different θ's, when λ exceeds a certain value.

Figure 2.

Averaged parameter estimation error ${\Vert \widehat{W}-\stackrel{‒}{W}\Vert }_{2,1}$ vs. λ plots on the synthetic data set (averaged over 10 runs). MSMTFL has the smallest parameter estimation error among the four algorithms. Both DirtyMTL and MSMTFL have two parameters; we set λ_s/λ_b = 1, 0.5, 0.2, 0.1 for DirtyMTL (1/m ≤ λ_s/λ_b ≤ 1 was adopted in Jalali et al. (2010) [9]) and θ/λ = 50m, 10m, 2m, 0.4m for MSMTFL.

5.2 Real-World Data Experiments

We conduct experiments on two real-world data sets: MRI and Isolet data sets. (1) The MRI data set is collected from the ANDI database, which contains 675 patients' MRI data preprocessed using FreeSurfer¹. The MRI data include 306 features and the response (target) is the Mini Mental State Examination (MMSE) score coming from 6 different time points: M06, M12, M18, M24, M36, and M48. We remove the samples which fail the MRI quality controls and have missing entries. Thus, we have 6 tasks with each task corresponding to a time point and the sample sizes corresponding to 6 tasks are 648, 642, 293, 569, 389 and 87, respectively. (2) The Isolet data set² is collected from 150 speakers who speak the name of each English letter of the alphabet twice. Thus, there are 52 samples from each speaker. The speakers are grouped into 5 subsets which respectively include 30 similar speakers, and the subsets are named Isolet1, Isolet2, Isolet3, Isolet4, and Isolet5. Thus, we naturally have 5 tasks with each task corresponding to a subset. The 5 tasks respectively have 1560, 1560, 1560, 1558, and 1559 samples (Three samples are historically missing), where each sample includes 617 features and the response is the English letter label (1–26).

In the experiments, we treat the MMSE and letter labels as the regression values for the MRI data set and the Isolet data set, respectively. For both data sets, we randomly extract the training samples from each task with different training ratios (15%, 20% and 25%) and use the rest of samples to form the test set. We evaluate the four multi-task feature learning algorithms in terms of normalized mean squared error (nMSE) and averaged means squared error (aMSE), which are commonly used in multi-task learning problems [28, 29]. For each training ratio, both nMSE and aMSE are averaged over 10 random splittings of training and test sets and the standard deviation is also shown. All parameters of the four algorithms are tuned via 3-fold cross validation.

Table 1 and Figure 3 show the experimental results in terms of averaged nMSE (aMSE) and the standard deviation. From these results, we observe that: (a) Our proposed MSMTFL algorithm out-performs all the competing feature learning algorithms on both data sets, with the smallest regression errors (nMSE and aMSE) as well as the smallest standard deviations. (b) On the MRI data set, the MSMTFL algorithm performs well even in the case of a small training ratio. The performance for the 15% training ratio is comparable to that for the 25% training ratio. (c) On the Isolet data set, when the training ratio increases from 15% to 25%, the performance of the MSMTFL algorithm increases and the superiority of the MSMTFL algorithm over the other three algorithms is more significant. Our results demonstrate the effectiveness of the proposed algorithm.

Table 1.

Comparison of four multi-task feature learning algorithms on the MRI data set in terms of averaged nMSE and aMSE (standard deviation), which are averaged over 10 random splittings.

measure	traning ratio	Lasso	L1,2	DirtyMTL	MSMTFL
nMSE	0.15	0.6651(0.0280)	0.6633(0.0470)	0.6224(0.0265)	0.5539(0.0154)
	0.20	0.6254(0.0212)	0.6489(0.0275)	0.6140(0.0185)	0.5542(0.0139)
	0.25	0.6105(0.0186)	0.6577(0.0194)	0.6136(0.0180)	0.5507(0.0142)
aMSE	0.15	0.0189(0.0008)	0.0187(0.0010)	0.0172(0.0006)	0.0159(0.0004)
	0.20	0.0179(0.0006)	0.0184(0.0005)	0.0171(0.0005)	0.0161(0.0004)
	0.25	0.0172(0.0009)	0.0183(0.0006)	0.0167(0.0008)	0.0157(0.0006)

Figure 3.

Averaged test error (nMSE and aMSE) vs. training ratio plots on the Isolet data set. The results are averaged over 10 random splittings.

6 Conclusions

In this paper, we propose a non-convex multi-task feature learning formulation based on the capped-${\ell }_1,{\ell }_{1w}$ regularization. The proposed formulation learns the specific features of each task as well as the common features shared among tasks. We propose to solve the non-convex optimization problem by employing a Multi-Stage Multi-Task Feature Learning (MSMTFL) algorithm, using concave duality. We also present a detailed theoretical analysis in terms of the parameter estimation error bound for the MSMTFL algorithm. The analysis shows that our MSMTFL algorithm achieves good performance under the sparse eigenvalue condition, which is weaker than the incoherence condition. Experimental results on both synthetic and real-world data sets demonstrate the effectiveness of our proposed MSMTFL algorithm in comparison with the state of the art multi-task feature learning algorithms. In our future work, we will focus on a general non-convex regularization framework for multi-task feature learning settings (involving different loss functions and non-convex regularization terms) and derive theoretical bounds.