Multiplicative Multitask Feature Learning

Abstract

We investigate a general framework of multiplicative multitask feature learning which decomposes individual task’s model parameters into a multiplication of two components. One of the components is used across all tasks and the other component is task-specific. Several previous methods can be proved to be special cases of our framework. We study the theoretical properties of this framework when different regularization conditions are applied to the two decomposed components. We prove that this framework is mathematically equivalent to the widely used multitask feature learning methods that are based on a joint regularization of all model parameters, but with a more general form of regularizers. Further, an analytical formula is derived for the across-task component as related to the task-specific component for all these regularizers, leading to a better understanding of the shrinkage effects of different regularizers. Study of this framework motivates new multitask learning algorithms. We propose two new learning formulations by varying the parameters in the proposed framework. An efficient blockwise coordinate descent algorithm is developed suitable for solving the entire family of formulations with rigorous convergence analysis. Simulation studies have identified the statistical properties of data that would be in favor of the new formulations. Extensive empirical studies on various classification and regression benchmark data sets have revealed the relative advantages of the two new formulations by comparing with the state of the art, which provides instructive insights into the feature learning problem with multiple tasks.

1. Introduction

Multitask learning (MTL) improves the generalization of the estimated models for multiple related learning tasks by capturing and exploiting the task relationships. It has been theoretically and empirically shown to be more effective than learning tasks individually. Especially when single task learning suffers from limited sample size, multitask learning reinforces a single learning process with the transferable knowledge learned from the related tasks. Multi-task learning has been widely applied in many scientific fields, such as robotics (Zhang and Yeung, 2010), natural language processing (Ando and Zhang, 2005), computer aided diagnosis (Bi et al., 2008), and computer vision (Kang et al., 2011).

Research efforts have been devoted to various MultiTask Feature Learning (MTFL) algorithms. One direction of these works directly learns the dependencies among tasks, either by modeling the correlated regression or classification noise (Greene, 2002), or assuming that the model parameters share a common prior (Yu et al., 2005; Lee et al., 2007; Xue et al., 2007; Yu et al., 2007; Jacob et al., 2008), or by examining the tasks’ covariance matrix (Bonilla et al., 2007; Zhang and Yeung, 2010; Guo et al., 2011; Yang et al., 2013). Another research direction relies on a basic assumption that the different tasks may share a substructure in the feature space. In order to exploit this shared substructure, Rai and Daume (2010) project the task parameters to explore the latent common substructure. Kang et al. (2011) form a shared low-dimensional representation of data by feature learning. More recent methods explore the latent basis that can be used to characterize the entire set of tasks and examine the potential clusters of tasks. For instance, Passos et al. (2012) assume that subsets of features may be shared only between tasks in the same cluster. Kumar and Daume III (2012) allow overlapping of tasks in different groups by having several bases in common. Maurer et al. (2013) exploit a dictionary allowing sparse representations of the tasks. Gong et al. (2012) detect if certain tasks are outliers from the majority of the tasks.

Regularization methods are widely used in MTFL to learn a shared subset of features. A common strategy is to impose a blockwise joint regularization (Meier et al., 2008; Zhao et al., 2009) on all task model parameters to shrink the effects of features across the tasks and simultaneously minimize the regression or classification loss. These methods employ the so-called ℓ_1,p matrix norm (Lee et al., 2010; Liu et al., 2009a; Obozinski and Taskar, 2006; Zhang et al., 2010; Zhou et al., 2010) that is the sum of the ℓ_p norms of the rows in a matrix. As a result, this regularizer encourages sparsity among the rows. If a row of the parameter matrix corresponds to a feature and a column represents an individual task, the ℓ_1,p regularizer intends to rule out the unrelated features across tasks by shrinking the entire rows of the matrix to zero. Typical choices for p are 2 (Obozinski and Taskar, 2006; Evgeniou and Pontil, 2004) and ∞ (Turlach et al., 2005). Effective algorithms have since then been developed for the ℓ_1,2 (Liu et al., 2009a) and ℓ_1,∞ (Quattoni et al., 2009) regularization. Later, the ℓ_1,p norm is generalized to include 1 < p ≤ ∞ with a probabilistic interpretation that the resultant MTFL method solves a relaxed optimization problem with a generalized normal prior for all tasks (Zhang et al., 2010). Although the matrix-norm based regularizers lead to convex learning formulations for MTFL, recent studies show that a convex regularizer may be too relaxed to approximate the ℓ₀-type regularizer for the shrinkage effects in the feature space and thus results in suboptimal performance (Gong et al., 2013). To address this problem, non-convex regularizers, such as the capped-ℓ₁, ℓ₁ regularizer (Gong et al., 2013), have been proposed for the multitask joint regularization. However, using non-convex regularizers may bring up computational challenges. For instance, non-convex formulations are usually difficult to solve, and require more complicated optimization algorithms to guarantee satisfactory performance.

For the existing MTFL methods based on joint regularization, a major limitation is that it either selects a feature as relevant to all tasks or excludes it from all models, which is very restrictive in practice where tasks may share some features but may also have their own specific features that are not relevant to other tasks. To overcome this limitation, one of the most effective strategies is to decompose the model parameters into either summation (Jalali et al., 2010; Chen et al., 2011; Gong et al., 2012) or multiplication (Xiong et al., 2006; Bi et al., 2008; Lozano and Swirszcz, 2012) of two components with separate regularizers applied to the two components. One regularizer is imposed on the component taking care of the task-specific model parameters and the other one is imposed on the component for mining the cross-feature sparsity. Specifically, for the methods that decompose the parameter matrix into summation of two matrices, the dirty model in (Jalali et al., 2010) employs ℓ_1,1 and ℓ_1,∞ regularizers to the two components. A robust MTFL method in (Chen et al., 2011) uses the trace norm on one component for mining a low-rank structure shared by tasks and a column-wise ℓ_1,2-norm on the other component for identifying task outliers. A more recent method applies the ℓ_1,2-norm both row-wisely to one component and column-wisely to the other (Gong et al., 2012). For these additive decomposition methods, it requires the corresponding entries in both components to be zero in order to exclude a feature from a task.

For the methods that work with multiplicative decompositions, the parameter vector of each task is decomposed into an element-wise product of two vectors where one is used across tasks and the other is task-specific. To exclude a feature from a task, the multiplicative decomposition only requires one of the components to be zero. Existing methods of this line apply the same regularization to both of the component vectors, by either the ℓ₂-norm penalty (Bi et al., 2008), or the sparse ℓ₁-norm (i.e., multi-level LASSO (Lozano and Swirszcz, 2012)). The multi-level LASSO method has been analytically compared to the dirty model (Lozano and Swirszcz, 2012), showing that the multiplicative decomposition creates better shrinkage on the global and task-specific parameters. The across-task component can screen out the features irrelevant to all tasks. An individual task’s component can further select features from those selected by the cross-task component for use in its corresponding model. Although there are different ways to regularize the two components in the product, no systematic work has been done to analyze the algorithmic and statistical properties of the different regularizers. It is insightful to answer the questions such as how these learning formulations differ from the early methods based on blockwise joint regularization, how the optimal solutions of the two components intervene, and how the resultant solutions are compared with those of other methods that also learn both shared and task-specific features. We highlight the contributions of this paper as follows:

• We propose and examine a general framework of the multiplicative decomposition that enables a variety of regularizers to be applied. The general form corresponds to a family of MTFL methods, including all early methods that decompose model parameters as a product of two components (Bi et al., 2008; Lozano and Swirszcz, 2012).

• Our theoretical analysis has revealed that this family of methods is actually equivalent to the joint regularization based approach but with a more general form of regularizers, including matrix-norm based and non-matrix-norm based regularizers. The non-matrix-norm based joint regularizers derived from the proposed framework have never been considered previously. If they are considered in the joint regularization form, the resultant optimization problems will be difficult to solve. However, our equivalent multiplicative MTFL framework in this case uses convex regularizers on the two components, which can be solved efficiently.

• Further analysis reveals that the optimal solution of the across-task component can be analytically computed by a formula of the optimal task-specific parameters. This analytical result facilitates a better understanding of the shrinkage effects of the different regularizers applied to the two components.

• Statistical justification is also derived for this family of formulations. It proves that the proposed framework is equivalent to maximizing a lower bound of the maximum a posterior solution under a probabilistic model that assumes generalized normal priors on model parameters.

• Two new MTFL formulations are derived from the proposed general framework. Unlike the existing methods (Bi et al., 2008; Lozano and Swirszcz, 2012) where the same kind of vector norm is applied to both components, the shrinkage of the global and task-specific parameters differs in the new formulations. We empirically illustrate the scenarios where the two new formulations are more suitable for solving the MTFL problems.

• An efficient blockwise coordinate descent algorithm is derived suitable for solving the entire family of the methods. Given some of the methods (including the two new formulations we study) correspond to non-matrix-norm based joint regularizers, our algorithm provides a powerful alternative to solving the related difficult optimization problems, allowing us to explore the behaviors and properties of these regularizers in an effective way. Convergence analysis is thoroughly discussed.

To depict the differences between our approach and previous methods, Table 1 summaries various regularizers used in the joint regularization based and model decomposition based MTFL methods. There are some fundamental connections among these methods. As studied in the present work, multiplicative MTFL models can be connected with the early blockwise joint regularization models. We also attempt to empirically compare the model behaviors between the multiplicative and additive MTFL methods, and particularly compare the applicability of the four different choices of regularization listed in Table 1 for multiplicative MTFL.

Table 1.

The regularization terms used in various MTFL methods.

	Model	Methods	Norms
Joint regularization models	A	Evgeniou and Pontil (2004)	ℓ1,2
		Turlach et al. (2005)	ℓ1,∞
		Lee et al. (2010)	both ℓ1,1 and ℓ1,2
		Zhang et al. (2010)	ℓ1,p, 1 < p ≤ ∞
		Gong et al. (2012)	capped ℓ1,1
Decomposed models	A = P + Q	Jalali et al. (2010)	ℓ1,1 on P, ℓ1,∞ on Q
		Gong et al. (2012)	ℓ1,2 on P, ℓ1,2 on QT
	A = diag(c) · B	The proposed framework    Bi et al. (2008) Lozano and Swirszcz (2012)   The new formulation 1   The new formulation 2	(ℓk)k on c, (ℓp)p on B k=2, p=2 k=1, p=1 k=1, p=2 k=2, p=1

The rest of the paper is organized as follows. Section 2 defines the mathematical notation and introduces the proposed models in detail. Section 3 discusses several important theoretical properties of the proposed models including equivalence analysis. Section 4 provides the statistical justification of the multiplicative MTFL models. In Section 5, we develop an efficient algorithm to solve the optimization problems in the proposed models with a convergence analysis. Section 6 shows the empirical results, in which simulations have been designed to examine the various feature sharing patterns for which a specific choice of regularizer may be preferred. Extensive experiments with a variety of classification and regression benchmark data sets are also described in Section 6. In Section 7, we conclude this work.

2. The Proposed Multiplicative MTFL

Given T tasks in total, for each task t, t ∈ {1,⋯, T} we have sample set $({\mathbf{X}}_t\in {\mathbb{R}}^{{\mathit{\ell }}_t\times d},{\mathbf{y}}_t\in {\mathbb{R}}^{{\mathit{\ell }}_t})$. The data set of X_t has ℓ_t examples, where the i-th row corresponds to the i-th example ${\mathbf{x}}_i^t$ of task t, i ∈ {1, ⋯ ℓ_t}, and each column represents a feature and there are totally d features. The vector y_t contains $y_i^t$, the label of the i-th example of task t. We consider functions of the linear form $y_t={\mathit{\alpha }}_t^{\top }{\mathbf{x}}_i^t$ where α_t ∈ ℝ^d, which corresponds to computing X_tα_t on the training data as the estimate of y_t. We define the parameter matrix or weight matrix A = [α₁, ⋯, α_T ] and denote the rows of A by α^j, j ∈ {1,⋯,d}.

The joint regularization based MTFL method with the ℓ_1,p regularizer minimizes

$${\displaystyle \sum _{t=1}^{T}L({\mathit{\alpha }}_t,{\mathbf{X}}_t,{\mathbf{y}}_t)+\mathrm{\lambda }{\displaystyle \sum _{j=1}^d{\Vert {\mathit{\alpha }}^j\Vert }_p}},$$ (1)

for the best α_{t: t=1, ⋯, T}, where L(α_t,X_t, y_t) is the loss function of task t which computes the discrepancy between the observed y_t and the model output X_tα_t, and λ is a tuning parameter to balance between the loss and the regularizer. Although any suitable loss function can be used in the formulation (1), convex loss functions are the common choices such as the least squares loss ${{\sum }_{i=1}^{{\mathit{\ell }}_t}(y_i^t-{\mathit{\alpha }}_t^{\top }{\mathbf{x}}_i^t)}^2$ for regression problems or the logistic loss ${\sum }_{i=1}^{{\mathit{\ell }}_t}\log (1+e^{-y_i^t({\mathit{\alpha }}_t^{\top }{\mathbf{x}}_i^t)})$ for classification problems. These loss functions are strictly convex with respect to the model parameters α_t. The ℓ_p norm is computed for each row of the matrix A corresponding to a feature (rather than a task) so to enforce sparsity on the features.

A family of multiplicative MTFL methods can be derived by rewriting α_t = diag(c)β_t where diag(c) is a diagonal matrix with its diagonal elements composing a vector c. The c vector is used across all tasks, indicating if a feature is useful for any of the tasks, and the vector β_t is only for task t. Let j index the entries in these vectors. We have ${\alpha }_j^t=c_j{\beta }_j^t$. Typically c comprises binary entries that are equal to 0 or 1, but the integer constraint is often relaxed to require just non-negativity (c ≥ 0). We minimize a regularized loss function as follows for the best c and β_t:t=1,⋯,T:

$$\underset{{\mathit{\beta }}_t,\mathbf{c}\ge 0}{\min }{\displaystyle \sum _{t=1}^{T}L(\mathbf{c},{\mathit{\beta }}_t,{\mathbf{X}}_t,{\mathbf{y}}_t)+{\gamma }_1{\displaystyle \sum _{t=1}^T{\Vert {\mathit{\beta }}_t\Vert }_p^p+{\gamma }_2{\Vert \mathbf{c}\Vert }_k^k}},$$ (2)

where L(c, β_t, X_t, y_t) is the same loss function used in Eq.(1) but with α_t replaced by the new vector of ${(c_j{\beta }_j^t)}_{j=1,\cdots ,d}$, ${\Vert {\mathit{\beta }}_t\Vert }_p^p={\sum }_{j=1}^d{|{\beta }_j^t|}^p$ and ${\Vert \mathbf{c}\Vert }_k^k={\sum }_{j=1}^d{(c_j)}^k$, which are the ℓ_p-norm of β_t to the power of p and the ℓ_k-norm of c to the power of k if p and k are positive integers. The tuning parameters γ₁, γ₂ are used to balance the empirical loss and regularizers. At optimality, if c_j = 0, the j-th variable is removed for all tasks, and the corresponding row vector α^j = 0; otherwise the j-th variable is selected for use in at least one of the α’s. Then, a specific β_t can rule out the j-th variable from task t if ${\beta }_j^t=0$.

If both p = k = 2, Problem (2) becomes the formulation used in Bi et al. (2008). Since the ℓ₂-norm regularization is applied on both β_t and c, this model does not impose strong sparsity on the model parameters. According to our empirical study, this model may be suitable for the scenarios where only a few features can be excluded from all of the tasks, and the different models (tasks) share a lot features between each other. There could exist features that, although irrelevant to most of the tasks, cannot be completely excluded only due to few tasks.

If p = k = 1, Problem (2) becomes the formulation used in Lozano and Swirszcz (2012), where the ℓ₁-norm regularization is applied on β_t and c and thus it induces very strong sparsity both on task-specific parameters and the across-task component to select the features. Compared to the model in Bi et al. (2008), this model is more suitable for learning from the tasks with persistently sparse models. For example, many features are irrelevant to any of the tasks, and only a few of the features selected by c are used by an individual task, indicating the sparse pattern in sharing the selected features among tasks.

Besides the above two existing models, any other choices of p and k will derive into new formulations for MTFL. Note that the two existing methods discussed in Bi et al. (2008); Lozano and Swirszcz (2012) use p = k in their formulations, which renders ${\beta }_j^t$ and c_j the same amount of shrinkage. To explore other feature sharing patterns among tasks, we propose two new formulations where p ≠ k.

Formulation 1

If p = 2 but k = 1 in Problem (2), then we obtain the following optimization problem

$$\underset{{\mathit{\beta }}_t,\mathbf{c}\ge 0}{\min }{\displaystyle \sum _{t=1}^{T}L(\mathbf{c},{\mathit{\beta }}_t,{\mathbf{X}}_t,{\mathbf{y}}_t)+{\gamma }_1{\displaystyle \sum _{t=1}^T{\Vert {\mathit{\beta }}_t\Vert }_2^2+{\gamma }_2{\Vert \mathbf{c}\Vert }_1}}.$$ (3)

When there exists a large subset of features irrelevant to any of the tasks, it requires a sparsity-inducing norm on c. However, within the relevant features selected by c, the majority of these features are shared between tasks. In other words, the features used in each task are not sparse relative to the features selected by c, which requires a non-sparsity-inducing norm on β. Hence, we use ℓ₁ norm on c and ℓ₂ norm on each β in our formulation (2).

Formulation 2

If p = 1 but k = 2 in Problem (2), we obtain the following optimization problem

$$\underset{{\mathit{\beta }}_t,\mathbf{c}\ge 0}{\min }{\displaystyle \sum _{t=1}^{T}L(\mathbf{c},{\mathit{\beta }}_t,{\mathbf{X}}_t,{\mathbf{y}}_t)+{\gamma }_1{\displaystyle \sum _{t=1}^T{\Vert {\mathit{\beta }}_t\Vert }_1+{\gamma }_2{\Vert \mathbf{c}\Vert }_2^2}}.$$ (4)

When the union of the features relevant to any given tasks includes many or even all features, the ℓ₂ norm penalty on c may be preferred. However, only a limited number of features are shared between tasks, i.e., the features used by individual tasks are sparse with respect to the features selected as useful across tasks by c. We can impose the ℓ₁ norm penalty on β.

Clearly, many other choices of p and k values can be used, such as those corresponding to higher order polynomials $(\mathrm{e}.\mathrm{g}.,{\sum }_{j=1}^d{c}_j^3)$. Our theoretical results in the next few sections apply to all positive value choices of p and k unless otherwise specified. In our empirical studies, however, we have implemented algorithms for the two existing models and the two new models for comparison. Some other choices of regularizers may require significant re-programming of our algorithms and we will leave them for more thorough individual examinations in the future.

3. Theoretical Analysis

We first extend the formulation (1) to allow more choices of regularizers. We introduce a new notation that is an operator applied to a vector, such as α^j. The operator ${\Vert {\mathit{\alpha }}^j\Vert }_p^{p/q}=\sqrt[q]{{\sum }_{t=1}^T{|{\alpha }_j^t|}^p}$, p, q ≥ 0, which corresponds to the ℓ_p norm if p = q and both are positive integers. A joint regularized MTFL approach can solve the following optimization problem with pre-specified values of p, q and λ, for the best parameters α_t:t=1,_⋯,_T:

$$\underset{{\mathit{\alpha }}_t}{\min }{\displaystyle \sum _{t=1}^{T}L({\mathit{\alpha }}_t,{\mathbf{X}}_t,{\mathbf{y}}_t)}+\mathrm{\lambda }{\displaystyle \sum _{j=1}^d\sqrt{{\Vert {\mathit{\alpha }}^j\Vert }_p^{p/q}}}.$$ (5)

Our main results of this paper include (i) a theorem that establishes the equivalence between the models derived from solving Problem (2) and Problem (5) for properly chosen values of λ, q, k, γ₁ and γ₂; (ii) a theorem that delineates the conditions for (2) to impose a convex (or concave) regularizer on the model parameter matrix A; and (iii) an analytical solution of Problem (2) for c which shows how the sparsity of the across-task component is relative to the sparsity of task-specific components.

Theorem 1 (Main Result 1)

Let ${\widehat{\mathit{\alpha }}}_t$ be the optimal solution to Problem (5) and $({\widehat{\mathit{\beta }}}_t,\widehat{\mathbf{c}})$ be the optimal solution to Problem (2). Then ${\widehat{\mathit{\alpha }}}_t=\mathit{diag}(\widehat{\mathbf{c}}){\widehat{\mathit{\beta }}}_t$ when $\mathrm{\lambda }=2\sqrt{{\gamma }_1^{2-\frac{p}{kq}}{\gamma }_2^{\frac{p}{kq}}}$ and $q=\frac{k+p}{2k}$ (or equivalently, $k=\frac{p}{2q-1}$).

Proof

Theorem 1 can be proved by establishing the following two lemmas and two theorems. The two lemmas provide the basis for the proofs of the two theorems and then from the first theorem, we conclude that the solution ${\widehat{\mathit{\alpha }}}_t$ of Problem (5) also minimizes the following optimization problem:

$$\underset{{\mathit{\alpha }}_t,\mathit{\sigma }\ge 0}{\min }{\displaystyle {\sum }_{t=1}^{T}L({\mathit{\alpha }}_t,{\mathbf{X}}_t,{\mathbf{y}}_t)+{\mu }_1{{\displaystyle {\sum }_{j=1}^d{\sigma }_j^{-1}\Vert {\mathit{\alpha }}^j\Vert }}_p^{p/q}+{\mu }_2{\displaystyle {\sum }_{j=1}^d{\sigma }_j}},$$ (6)

and the optimal solution of Problem (6) also minimizes Problem (5) when proper values of λ, μ₁ and μ₂ are chosen. The second theorem connects Problem (6) to the proposed formulation (2). We show that the optimal ${\widehat{\sigma }}_j$ is equal to ${({\widehat{c}}_j)}^k$, and then the optimal $\widehat{\mathit{\alpha }}$ can be computed as $\text{diag}(\widehat{\mathbf{c}}){\widehat{\mathit{\beta }}}_t$ from the optimal ${\widehat{\mathit{\beta }}}_t$.  ■

Note that when p = 2 and q = 1, the intermediate problem (6) uses a similar regularizer to that in Micchelli et al. (2013) where $\frac{{|{\alpha }^j|}^2}{{\sigma }_j}+{\sigma }_j$ is used to approximate |α^j| in the ℓ₁-norm regularizer. Problem (6) extends the discussion in Micchelli et al. (2013) to include more general regularizers according to p and q.

Lemma 2

For any given α_t:t=1,⋯,T, Problem (6) can be optimized with respect to σ by the following analytical solution

$${\sigma }_j={\mu }_1^{\frac{1}{2}}{\mu }_2^{-\frac{1}{2}}\sqrt{{\Vert {\alpha }^j\Vert }_p^{p/q}}.$$ (7)

Proof

By the Cauchy-Schwarz inequality, we can derive a lower bound for the sum of the two regularizers in Problem (6) as follows:

$${\mu }_1{\displaystyle \sum _{j=1}^d{\sigma }_j^{-1}{\Vert {\mathit{\alpha }}^j\Vert }_p^{p/q}+{\mu }_2{\displaystyle \sum _{j=1}^d{\sigma }_j\ge 2\sqrt{{\mu }_1{\mu }_2}{\displaystyle \sum _{j=1}^d\sqrt{{\Vert {\mathit{\alpha }}^j\Vert }_p^{p/q}}}}},$$ (8)

where the equality holds if and only if ${\sigma }_j={\mu }_1^{\frac{1}{2}}{\mu }_2^{-\frac{1}{2}}\sqrt{{\Vert {\mathit{\alpha }}^j\Vert }_p^{p/q}}$.

Using the method of proof by contradiction, suppose that σ* optimizes Problem (6) with a given set of α_t:t=1,⋯,T, but ${\sigma }_j^{\ast }\ne {\mu }_1^{\frac{1}{2}}{\mu }_2^{-\frac{1}{2}}\sqrt{{\Vert {\mathit{\alpha }}^j\Vert }_p^{p/q}}$. Thus, σ* does not make the regularization term reach its lower bound. Then, we can choose another $\tilde{\mathit{\sigma }}$ where ${\tilde{\sigma }}_j={\mu }_1^{\frac{1}{2}}{\mu }_2^{-\frac{1}{2}}\sqrt{{\Vert {\mathit{\alpha }}^j\Vert }_p^{p/q}}$ so $\tilde{\mathit{\sigma }}$ delivers the lower bound in Eq.(8). Because the lower bound (the right hand side of Eq.(8)) only depends on α, it is a constant for fixed α_t:t=1,⋯,T. Hence, since the loss function is also fixed for given α_t:t=1,⋯,T, $\tilde{\mathit{\sigma }}$ reaches a lower objective value of Problem (6) than that of σ*, which contradicts to the optimality of σ*. Therefore, the optimal σ always takes the form of Eq.(7).  ■

Remark 3

Based on the proof of Lemma 2, we also know that the objective function of (5) is the lower bound of the objective function of (6) for any given α_t (including the optimal ${\widehat{\mathit{\alpha }}}_t$) when $\mathrm{\lambda }=2\sqrt{{\mu }_1{\mu }_2}$, and the lower bound can be attained if and only if σ is set according to the formula (7). Hence, we can also conclude that if $({\widehat{\mathit{\alpha }}}_t,\widehat{\mathit{\sigma }})$ is the optimal solution of Problem (6), then ${\widehat{\sigma }}_j={\mu }_1^{\frac{1}{2}}{\mu }_2^{-\frac{1}{2}}\sqrt{{\Vert {\widehat{\mathit{\alpha }}}^j\Vert }_p^{p/q}}$.

Lemma 4

Let ${\alpha }_j^t=c_j{\beta }_j^t$ for all t and j. Replacing β_t by α_t in Problem (2) yields the following optimization problem

$$\underset{{\mathit{\alpha }}_t,\mathbf{c}\ge 0}{\min }{\displaystyle \sum _{t=1}^{T}L({\mathit{\alpha }}_t,{\mathbf{X}}_t,{\mathbf{y}}_t)+{\gamma }_1{\displaystyle {\sum }_{j=1}^d{c}_j^{-p}{\Vert {\mathit{\alpha }}^j\Vert }_p^p+{\gamma }_2{\displaystyle {\sum }_{j=1}^d{(c_j)}^k}}}.$$ (9)

For any given α_t:t=1,⋯,T, Problem (9) can be optimized with respect to c by the following analytical solution

$$c_j={({\gamma }_1{\gamma }_2^{-1}{\displaystyle \sum _{t=1}^T{({\mathit{\alpha }}_j^t)}^p})}^{\frac{1}{p+k}}.$$ (10)

Proof

This lemma can be proved following a similar argument in the proof of Lemma 2. By the Cauchy-Schwarz inequality, the sum of the two regularizers in (9) satisfies the following inequality

$${\gamma }_1{\displaystyle \sum _{j=1}^d{c}_j^{-p}{\Vert {\mathit{\alpha }}^j\Vert }_p^p+{\gamma }_2{\displaystyle \sum _{j=1}^d{c}_j^k\ge 2{\gamma }_1^{\frac{k}{p+k}}{\gamma }_2^{\frac{p}{p+k}}}}{\displaystyle \sum _{j=1}^d{({\Vert {\mathit{\alpha }}^j\Vert }_p^p)}^{\frac{k}{p+k}}},$$ (11)

and the equality holds if and only if ${\gamma }_1{c}_j^{-p}{\Vert {\mathit{\alpha }}^j\Vert }_p^p={\gamma }_2{c}_j^k$ (and note that all parameters γ₁, γ₂, ${\Vert {\widehat{\mathit{\alpha }}}^j\Vert }_p^p$ and c are non-negative), which yields the following formula

$$c_j={({\gamma }_1{\gamma }_2^{-1}{\displaystyle \sum _{t=1}^T{({\alpha }_j^t)}^p})}^{\frac{1}{p+k}}.$$

Through proof by contradiction, we know that the optimal c has to take the above formula.  ■

Based on Lemma 2, we will prove that Problem (5) is equivalent to Problem (6) in the sense that an optimal solution of Problem (5) is also an optimal solution of Problem (6) and vice versa when λ, μ₁, and μ₂ satisfy certain conditions.

Theorem 5

The solution sets of Problem (5) and Problem (6) are identical when $\mathrm{\lambda }=2\sqrt{{\mu }_1{\mu }_2}$.

Proof

First, if $\widehat{\mathbf{A}}=[{\widehat{\mathit{\alpha }}}_{t:t=1,\cdots ,T}]$ minimizes Problem (5), we show that the pair $(\widehat{\mathbf{A}},\widehat{\mathit{\sigma }})$ minimizes Problem (6) where ${\widehat{\sigma }}_j={\mu }_1^{\frac{1}{2}}{\mu }_2^{-\frac{1}{2}}\sqrt{{\Vert {\widehat{\mathit{\alpha }}}^j\Vert }_p^{p/q}}$.

By Remark 3, the objective function of (5) is the lower bound of the objective function of (6) for any given A (including the optimal solution $\widehat{\mathbf{A}}$). When ${\widehat{\sigma }}_j={\mu }_1^{\frac{1}{2}}{\mu }_2^{-\frac{1}{2}}\sqrt{{\Vert {\widehat{\mathit{\alpha }}}^j\Vert }_p^{p/q}}$, Problem (6) reaches the lower bound. In this case, Problem (5) at $\widehat{\mathbf{A}}$ and Problem (6) at $(\widehat{\mathbf{A}},\widehat{\mathit{\sigma }})$ have the same objective value. Now, suppose that the pair $(\widehat{\mathbf{A}},\widehat{\mathit{\sigma }})$ does not minimize Problem (6), there exists another pair $(\tilde{\mathbf{A}},\tilde{\mathit{\sigma }})\ne (\widehat{\mathbf{A}},\widehat{\mathit{\sigma }})$ that achieves a lower objective value than that of $(\widehat{\mathbf{A}},\widehat{\mathit{\sigma }})$. By Lemma 2, we have that ${\tilde{\sigma }}_j={\mu }_1^{\frac{1}{2}}{\mu }_2^{-\frac{1}{2}}\sqrt{{\Vert {\tilde{\mathit{\alpha }}}^j\Vert }_p^{p/q}}$, and then Problem (6), at $\tilde{\mathbf{A}}$, reaches the lower bound which is formulated as the objective of Problem (5) when $\mathrm{\lambda }=2\sqrt{{\mu }_1{\mu }_2}$. In other words, the objective values of (6) and (5) are identical at $\tilde{\mathbf{A}}$. Hence, $\tilde{\mathbf{A}}$ achieves a lower objective value than that of $\widehat{\mathbf{A}}$ for Problem (5), contradicting to the optimality of $\widehat{\mathbf{A}}$.

Second, if $(\widehat{\mathbf{A}},\widehat{\mathit{\sigma }})$ minimizes Problem (6), we show that $\widehat{\mathbf{A}}$ minimizes Problem (5).

Suppose that $\widehat{\mathbf{A}}$ does not minimize Problem (5), which means that there exists ${\tilde{\mathit{\alpha }}}^j$ (≠ ${\widehat{\mathit{\alpha }}}^j$ for some j) that achieves a lower objective value than that of ${\widehat{\mathit{\alpha }}}^j$. We set ${\tilde{\sigma }}_j={\mu }_1^{\frac{1}{2}}{\mu }_2^{-\frac{1}{2}}\sqrt{{\Vert {\tilde{\mathit{\alpha }}}^j\Vert }_p^{p/q}}$. Then $(\tilde{\mathbf{A}},\tilde{\mathit{\sigma }})$ is an optimal solution of Problem (6) as proved in the first paragraph, and will bring the objective function of Problem (6) to a lower value than that of $(\widehat{\mathbf{A}},\widehat{\mathit{\sigma }})$, contradicting to the optimality of $(\widehat{\mathbf{A}},\widehat{\mathit{\sigma }})$.

Therefore, Problems (5) and (6) have identical solution sets when $\mathrm{\lambda }=2\sqrt{{\mu }_1{\mu }_2}$.  ■

In order to link Problem (6) to our formulation (2), we let σ_j = (c_j)^k, k ∈ ℝ, k ≠ 0 and ${\alpha }_j^t=c_j{\beta }_j^t$ for all t and j, and derive an equivalent objective function of Problem (6) based on Lemmas 2 and 4.

Theorem 6

The optimal solution $(\widehat{\mathbf{A}},\widehat{\mathit{\sigma }})$ of Problem (6) is equivalent to the optimal solution $(\widehat{\mathbf{B}},\widehat{\mathbf{c}})$ of Problem (2) where ${\widehat{\alpha }}_j^t={\widehat{c}}_j{\widehat{\beta }}_j^t$ and ${\widehat{\sigma }}_j={({\widehat{c}}_j)}^k$ when ${\gamma }_1={\mu }_1^{\frac{kq}{2kq-p}}{\mu }_2^{\frac{kq-p}{2kq-p}}$, γ₂ = μ₂, and $k=\frac{p}{2q-1}$.

Proof

First, if ${\widehat{\alpha }}_j^t$ and ${\widehat{\sigma }}_j$ optimize Problem (6), we show that ${\widehat{c}}_j=\sqrt[k]{{\widehat{\sigma }}_j}$ and ${\widehat{\beta }}_j^t={\widehat{\alpha }}_j^t/{\widehat{c}}_j$ optimize Problem (2).

By a change of variables (replacing ${\widehat{\mathit{\alpha }}}_t$ and $\widehat{\mathit{\sigma }}$ by ${\widehat{\mathit{\beta }}}_t$ and $\widehat{\mathbf{c}}$ in Problem (6)), we know that ${\widehat{\mathit{\beta }}}_t$ and $\widehat{\mathbf{c}}$ minimize the following objective function

$$J({\widehat{\beta }}_j^t,{\widehat{c}}_j)={\displaystyle \sum _{t=1}^{T}L(\widehat{\mathbf{c}},{\widehat{\mathit{\beta }}}_t,{\mathbf{X}}_t,{\mathbf{y}}_t)}+{\mu }_1{\displaystyle \sum _{j=1}^d{\Vert {\widehat{\mathit{\beta }}}^j\Vert }_p^{p/q}{\widehat{c}}_j^{(p-kq)/q}}+{\mu }_2{\displaystyle \sum _{j=1}^d{({\widehat{c}}_j)}^k}.$$ (12)

By Lemma 2 and Remark 3, the optimal ${\widehat{\sigma }}_j={\mu }_1^{\frac{1}{2}}{\mu }_2^{-\frac{1}{2}}\sqrt{{\Vert {\widehat{\mathit{\alpha }}}^j\Vert }_p^{p/q}}$. Because ${\widehat{c}}_j=\sqrt[k]{{\widehat{\sigma }}_j}$, we derive that

$${\widehat{c}}_j={\left({\mu }_1{\mu }_2^{-1}{\Vert {\widehat{\mathit{\beta }}}^j\Vert }_p^{p/q}\right)}^{\frac{q}{2kq-p}}.$$

Substituting the formula of ${\widehat{c}}_j$ into Eq.(12) yields the same objective function of Problem (2) after replacing μ₁ and μ₂ by γ₁ and γ₂ with the equations ${\gamma }_1={\mu }_1^{\frac{kq}{2kq-p}}{\mu }_2^{\frac{kq-p}{2kq-p}}$, γ₂ = μ₂. Therefore, ${\widehat{\mathit{\beta }}}_t$ and $\widehat{\mathbf{c}}$ optimize Problem (2) because otherwise, if any other solution (β, c) can further reduce the objective value of Problem (2), then the corresponding α and σ will bring the objective function of Problem (6) to a lower value than $\widehat{\mathit{\alpha }}$ and $\widehat{\mathit{\sigma }}$.

Next, if ${\widehat{\beta }}_j^t$ and ${\widehat{c}}_j$ optimize Problem (2), we show that ${\widehat{\alpha }}_j^t={\widehat{c}}_j{\widehat{\beta }}_j^t$ and ${\widehat{\sigma }}_j={({\widehat{c}}_j)}^k$ optimize Problem (6).

Substituting ${\widehat{\alpha }}_j^t$, ${\widehat{\sigma }}_j$ for ${\widehat{\beta }}_j^t$, ${\widehat{c}}_j$ in Problem (2) yields an objective function

$$J({\widehat{\alpha }}_j^t,{\widehat{\sigma }}_j)={\displaystyle \sum _{t=1}^{T}L({\widehat{\mathit{\alpha }}}_t,{\mathbf{X}}_t,{\mathbf{y}}_t)}+{\gamma }_1{\displaystyle \sum _{j=1}^d{\Vert {\widehat{\mathit{\alpha }}}^j\Vert }_p^p{\widehat{\sigma }}_j^{-(p/k)}}+{\gamma }_2{\displaystyle \sum _{j=1}^d{\widehat{\sigma }}_j}.$$ (13)

We hence know that ${\widehat{\alpha }}_j^t$ and ${\widehat{\sigma }}_j$ minimize Eq.(13). Similar to the proof of Lemma 4, we can show that the optimal $\widehat{\mathit{\sigma }}$ takes the form of

$${\widehat{\sigma }}_j={({\gamma }_1{\gamma }_2^{-1})}^{\frac{k}{k+p}}{({\Vert {\widehat{\mathit{\alpha }}}^j\Vert }_p^p)}^{\frac{k}{k+p}}.$$

Substituting the formula into Eq.(13) and setting ${\gamma }_1={\mu }_1^{\frac{kq}{2kq-p}}{\mu }_2^{\frac{kq-p}{2kq-p}}$ and γ₂ = μ₂ transfer Eq.(13) to the objective function of Problem (6). Thus ${\widehat{\alpha }}_j^t$ and ${\widehat{\sigma }}_j$ optimize Problem (6).  ■

Now, based on the above two lemmas and two theorems, we can derive that when $\mathrm{\lambda }=2\sqrt{{\gamma }_1^{2-\frac{p}{kq}}{\gamma }_2^{\frac{p}{kq}}}$ and $q=\frac{k+p}{2k}$, the optimal solutions to Problems (2) and (5) are equivalent. Solving Problem (2) will yield an optimal solution $\widehat{\mathit{\alpha }}$ to Problem (5) and vice versa.

By the equivalence analysis, the proposed framework corresponds to a family of joint regularization methods as defined by Eq.(5). Assuming a convex loss function is used, this family includes some convex formulations when convex regularizers are applied to A in (5) and some other non-convex formulations when non-matrix-norm based regularizers are applied to A. Particularly, when q = p/2, the regularization term on α^j in (5) becomes the standard ℓ_p-norm. Correspondingly, when k = p/(p − 1) which is commonly not an integer except p = 2, our formulation (2) amounts to imposing a ℓ_1,p-norm on A. When both k and p take positive integers (except p = k = 2), Problem (2) is equivalent to using a non-matrix-norm regularizer in (5). Combinations of different p and k values will render the models different algorithmic behaviors.

Theorem 7 (Main Result 2)

For any positive k and p, if kp ≥ k+p, then the formulation (2) imposes a convex regularizer on the model parameter matrix A; or otherwise, it imposes a concave regularizer on A.

Proof

According to Theorem 1, the formulation (2) is equivalent to the formulation (5) that imposes the following regularizer on A:

$$\mathrm{\lambda }{\displaystyle \sum _{j=1}^d\sqrt{{\Vert {\mathit{\alpha }}^j\Vert }_p^{p/q}}}=\mathrm{\lambda }{\displaystyle \sum _{j=1}^d{({\Vert {\mathit{\alpha }}^j\Vert }_p)}^{\frac{p}{2q}}}=\mathrm{\lambda }{\displaystyle \sum _{j=1}^d{({\Vert {\mathit{\alpha }}^j\Vert }_p)}^{\frac{kp}{k+q}}}.$$ (14)

where $q=\frac{k+p}{2k}$ and $\mathrm{\lambda }=2\sqrt{{\gamma }_1^{2-\frac{p}{kq}}{\gamma }_2^{\frac{p}{kq}}}$.

A function x^a is a convex function in terms of x > 0 if a ≥ 1; or otherwise, it is concave. Hence, if kp ≥ k + p, Eq.(14) is a composite function of two functions: a convex function $x^{\frac{\mathit{kp}}{k+p}}$ and another convex function which is the ℓ_p vector norm of α^j. Thus, the overall regularizer is convex. Otherwise, if kp < k + p, the regularizer becomes a concave function in terms of α’s.  ■

Remark 8

For a particular choice of p = 2 and k = 2, Problem (2) is formulated as

$$\underset{{\mathit{\beta }}_t,\mathbf{c}\ge 0}{\min }{\displaystyle \sum _{t=1}^{T}L(\mathbf{c},{\mathit{\beta }}_t,{\mathbf{X}}_t,{\mathbf{y}}_t)}+{\gamma }_1{\displaystyle \sum _{t=1}^T{\Vert {\mathit{\beta }}_t\Vert }_2^2+{\gamma }_2{\Vert \mathbf{c}\Vert }_2^2},$$ (15)

which is used in Bi et al. (2008). This problem is equivalent to the following joint regularization method as used in Obozinski and Taskar (2006); Argyriou et al. (2007).

$$\underset{{\mathit{\alpha }}_t}{\min }{\displaystyle \sum _{t=1}^{T}L({\mathit{\alpha }}_t,{\mathbf{X}}_t,{\mathbf{y}}_t)}+\mathrm{\lambda }{\displaystyle \sum _{j=1}^d\sqrt{{\displaystyle \sum _{t=1}^T{({\alpha }_j^t)}^2}}},$$ (16)

when $\mathrm{\lambda }=2\sqrt{{\gamma }_1{\gamma }_2}$. Problem (16) uses the so called ℓ_1,2-norm to regularize the matrix A.

Remark 9

For a particular choice of p = 1 and k = 1, Problem (2) is formulated as

$$\underset{{\mathit{\beta }}_t,\mathbf{c}\ge 0}{\min }{\displaystyle \sum _{t=1}^{T}L(\mathbf{c},{\mathit{\beta }}_t,{\mathbf{X}}_t,{\mathbf{y}}_t)}+{\gamma }_1{\displaystyle \sum _{t=1}^T{\Vert {\mathit{\beta }}_t\Vert }_1+{\gamma }_2{\Vert \mathbf{c}\Vert }_1},$$ (17)

which is used in Lozano and Swirszcz (2012). This problem is equivalent to the following joint regularization method

$$\underset{{\mathit{\alpha }}_t}{\min }{\displaystyle \sum _{t=1}^{T}L({\mathit{\alpha }}_t,{\mathbf{X}}_t,{\mathbf{y}}_t)}+\mathrm{\lambda }{\displaystyle \sum _{j=1}^d\sqrt{{\displaystyle \sum _{t=1}^T|{\alpha }_j^t|}}},$$ (18)

when $\mathrm{\lambda }=2\sqrt{{\gamma }_1{\gamma }_2}$. Problem (17) imposes stronger sparsity on β and c (and correspondingly on α) than (15), which intends to shrink more model parameters to zero. Problem (18) uses a concave non-matrix-norm regularizer.

Remark 10

For a particular choice of p = 2 and k = 1, which corresponds to the new formulation (3). This problem is equivalent to the following joint regularization method

$$\underset{{\mathit{\alpha }}_t}{\min }{\displaystyle \sum _{t=1}^{T}L({\mathit{\alpha }}_t,{\mathbf{X}}_t,{\mathbf{y}}_t)}+\mathrm{\lambda }{\displaystyle \sum _{j=1}^d\sqrt[3]{{\displaystyle \sum _{t=1}^T{({\alpha }_j^t)}^2}}},$$ (19)

when $\mathrm{\lambda }=2{\gamma }_1^{\frac{1}{3}}{\gamma }_2^{\frac{2}{3}}$. Problem (19) uses a concave non-matrix-norm regularizer as well but the concavity is weaker than Problem (18) in the sense that the polynomial order is $\frac{2}{3}$ rather than $\frac{1}{2}$.

The proposed formulation (3) imposes stronger sparsity induction on the across-task component than on the task-specific component. Thus, it has stronger shrinkage effects to exclude many features for all the tasks. If the jth feature is considered as unrelated to most of the tasks, the model of p = k = 2 may shrink c_j to a small value but not zero, the new model (3) might shrink c_j to zero instead. Therefore this model would be more favorable to jointly learning from tasks where a large portion of noisy features exist that may be irrelevant or redundant to all of the tasks. Compared to the method in Lozano and Swirszcz (2012) where p = k = 1, the new formulation (3) may allow more selected features to be shared across different tasks.

Remark 11

For a particular choice of p = 1 and k = 2, which corresponds to the new formulation (4). This problem is equivalent to the following joint regularization method

$$\underset{{\mathit{\alpha }}_t}{\min }{\displaystyle \sum _{t=1}^{T}L({\mathit{\alpha }}_t,{\mathbf{X}}_t,{\mathbf{y}}_t)}+\mathrm{\lambda }{\displaystyle \sum _{j=1}^d\sqrt[3]{{\left({\displaystyle \sum _{t=1}^T|{\mathit{\alpha }}_j^t|}\right)}^2}},$$ (20)

when $\mathrm{\lambda }=2{\gamma }_1^{\frac{2}{3}}{\gamma }_2^{\frac{1}{3}}$. Problem (20) uses another concave non-matrix-norm regularizer that has the similar polynomial order of $\frac{2}{3}$ to Problem (19). In comparison with (19), the cross-task quadratic terms (e.g., $|{\alpha }_j^1||{\alpha }_j^2|$, $|{\alpha }_j^2||{\alpha }_j^3|$) are allowed inside the cube root in this formulation.

The new formulation (4) imposes stronger sparsity induction on task-specific component than on the across-task vector. This model can be favorable in the cases where few tasks share a limited number of selected features. As shown in our empirical results, for instance, when every two or three tasks share a limited subset of selected features but no common features are used by more than three tasks, this model performs the best among the four multiplicative MTFL formulations. In comparison to the model in Lozano and Swirszcz (2012), this model allows more of the features that are only relevant to a few tasks to be selected. In comparison with the model in Bi et al. (2008), this model can help to remove a lot of irrelevant or redundant features for individual tasks from the selected features.

We further derive another main result that characterizes the optimal across-task vector as a formula of the optimal task-specific vectors. This connection can be easily developed from the above equivalence analysis, and can help us understand the relationship and interaction between the two components.

Theorem 12 (Main Result 3)

Let ${\widehat{\mathit{\beta }}}_t$, t = 1, ⋯, T, be the optimal solutions of Problem (2), Let $\widehat{\mathbf{B}}=[{\widehat{\mathit{\beta }}}_t,\cdots ,{\widehat{\mathit{\beta }}}_T]$ and ${\widehat{\mathit{\beta }}}^j$ denote the j-th row of the matrix $\widehat{\mathbf{B}}$. Then,

$${\widehat{c}}_j={({\gamma }_1/{\gamma }_2)}^{\frac{1}{k}}\sqrt[k]{{\displaystyle \sum _{t=1}^T{({\widehat{\beta }}_j^t)}^p}},$$ (21)

for all j = 1, ⋯, d, is optimal to Problem (2).

Proof

This analytical formula can be directly derived from Theorem 5 and Theorem 6. When we set ${\widehat{\sigma }}_j={({\widehat{c}}_j)}^k$ and ${\widehat{\alpha }}_j^t={\widehat{c}}_j{\widehat{\beta }}_j^t$ in the proof of Theorem 6, we obtain ${\widehat{c}}_j={\left({\mu }_1{\mu }_2^{-1}{\Vert {\widehat{\mathit{\beta }}}^j\Vert }_p^{p/q}\right)}^{\frac{q}{2kq-p}}$. In the same proof, we also show that ${\mu }_1={\gamma }_1^{\frac{2kq-p}{kq}}{\gamma }_2^{\frac{p-kq}{kq}}$ and μ₂ = γ₂. Substituting these formulas into the formula of c yields ${\widehat{c}}_j={({\gamma }_1/{\gamma }_2)}^{\frac{1}{k}}{\Vert {\widehat{\mathit{\beta }}}^j\Vert }^{\frac{p}{2kq-p}}$. Moreover, to establish the equivalence between (2) and (5), we require $q=\frac{k+p}{2k}$, which leads to k = 2kq − p. Hence, ${\Vert {\widehat{\mathit{\beta }}}^j\Vert }^{\frac{p}{2kq-p}}=\sqrt[k]{{\displaystyle {\sum }_{t=1}^T{({\widehat{\beta }}_j^t)}^p}}$. We then obtain the formula (21).  ■

Remark 13

For particular choices of p and k, the relation between the optimal c and β can be computed according to Theorem 12. Table 2 summarizes the relation formula for the common choices when p ∈ {1, 2} and k ∈ {1, 2}.

Table 2.

The shrinkage effect of c with respect to β for four common choices of p and k.

p	k	c
2	2	${\widehat{c}}_j=\sqrt{{\gamma }_1{\gamma }_2^{-1}}\sqrt{{\displaystyle {\sum }_{t=1}^T{({\widehat{\beta }}_j^t)}^2}}$
1	1	${\widehat{c}}_j={\gamma }_1{\gamma }_2^{-1}{\displaystyle {\sum }_{t=1}^T|{\widehat{\beta }}_j^t|}$
2	1	${\widehat{c}}_j={\gamma }_1{\gamma }_2^{-1}{\displaystyle {\sum }_{t=1}^T{({\widehat{\beta }}_j^t)}^2}$
1	2	${\widehat{c}}_j=\sqrt{{\gamma }_1{\gamma }_2^{-1}}\sqrt{{\displaystyle {\sum }_{t=1}^T|{\widehat{\beta }}_j^t|}}$

4. Probabilistic Interpretation

In this section we show that the proposed multiplicative formalism is related to the maximum a posteriori (MAP) solution of a probabilistic model. Let p(A|Δ) be the prior distribution of the weight matrix A = [α₁, …, α_T] = [α^1⊤, …, α^d⊤]^⊤ ∈ ℝ^d×T, where Δ denote the parameter of the prior. Then the a posteriori distribution of A can be calculated via Bayes rule as

$$p(\mathbf{A}|\mathbf{X},\mathbf{y},\mathbf{\Delta })\propto p(\mathbf{A}|\mathbf{\Delta }){\displaystyle \prod _{t=1}^{T}p({\mathbf{y}}_t|{\mathbf{X}}_t,{\mathit{\alpha }}_t)}.$$ (22)

Denote $z~\mathcal{G}\mathcal{N}(\mu ,\rho ,p)$ the univariate generalized normal distribution, with the density function

$$p(z)=\frac{1}{2\rho \mathrm{\Gamma }(1+1/p)}\exp \left(-\frac{{|z-\mu |}^p}{{\rho }^p}\right),$$ (23)

in which ρ > 0, p > 0, and Γ(·) is the Gamma function (Goodman and Kotz, 1973). Now let each element of A, ${\alpha }_j^t$, follow a generalized normal prior, ${\alpha }_j^t~\mathcal{G}\mathcal{N}(0,{\delta }_j,p)$. Then with the independent and identically distributed assumption, the prior takes the form (also refer to Zhang et al. (2010) for a similar treatment)

$$p(\mathbf{A}|\mathbf{\Delta })\propto {\displaystyle \prod _{j=1}^d{\displaystyle \prod _{t=1}^T\frac{1}{{\delta }_j}\exp \left(-\frac{{|{\alpha }_j^t|}^p}{{\delta }_j^p}\right)}}={\displaystyle \prod _{j=1}^d\frac{1}{{\delta }_j^T}\exp \left(-\frac{{\Vert {\mathit{\alpha }}^j\Vert }_p^p}{{\delta }_j^p}\right)},$$ (24)

where ‖·‖_p denotes the vector p-norm. With an appropriately chosen likelihood function p(y_t|X_t, α_t) ∝ exp(−L(α_t, X_t, y_t)), finding the MAP solution is equivalent to solving the following problem:

$$\underset{\mathbf{A},\mathbf{\Delta }}{\min }J={\displaystyle \sum _{t=1}^{T}L({\mathit{\alpha }}_t,{\mathbf{X}}_t,{\mathbf{y}}_t)}+{\displaystyle \sum _{j=1}^d\left(\frac{{\Vert {\mathit{\alpha }}^j\Vert }_p^p}{{\delta }_j^p}+T\ln {\delta }_j\right)}.$$ (25)

Setting the derivative of J with respect to δ_j to zero, we obtain:

$${\delta }_j={\left(\frac{p}{T}\right)}^{1/p}{\Vert {\mathit{\alpha }}^j\Vert }_p.$$ (26)

Bringing this back to (25), we have the following equivalent problem:

$$\underset{\mathbf{A}}{\min }J={\displaystyle {\sum }_{t=1}^{T}L({\mathit{\alpha }}_t,{\mathbf{X}}_t,{\mathbf{y}}_t)}+T{\displaystyle {\sum }_{j=1}^d\ln {\Vert {\mathit{\alpha }}^j\Vert }_p.}$$ (27)

Now let us look at the multiplicative nature of ${\alpha }_j^t$ with different p ∈ [1, ∞).

When p = 1, we have:

$$\begin{array}{l}{\displaystyle \sum _{j=1}^d\ln }{\Vert {\mathit{\alpha }}^j\Vert }_1={\displaystyle \sum _{j=1}^d\ln }\left({\displaystyle \sum _{t=1}^T|{\alpha }_j^t|}\right)\\ ={\displaystyle \sum _{j=1}^d\ln }\left({\displaystyle \sum _{t=1}^T|c_j{\beta }_j^t|}\right)\\ ={\displaystyle \sum _{j=1}^d(\ln |c_j|+\ln {\displaystyle \sum _{t=1}^T|{\beta }_j^t|})}\\ \le {\displaystyle \sum _{j=1}^d|c_j|+}{\displaystyle \sum _{j=1}^d{\displaystyle \sum _{t=1}^T|{\beta }_j^t|}}-2d,\end{array}$$

where the inequality is because of the fact that ln z ≤ z − 1 for any z > 0. Therefore we can optimize an upper bound of J in (27),

$$\underset{\mathbf{A}}{\min }{J}_1={\displaystyle \sum _{t=1}^{T}L({\mathit{\alpha }}_t,{\mathbf{X}}_t,{\mathbf{y}}_t)}+T{\displaystyle \sum _{j=1}^d|c_j|+T{\displaystyle \sum _{j=1}^d{\displaystyle \sum _{t=1}^T|{\beta }_j^t|}}}-2dT,$$ (28)

which is equivalent to the multiplicative formulation (2) where {p, k} = {1,1}. This proves the following theorem:

Theorem 14

When {p, k} = {1, 1}, optimizing the multiplicative formulation (2) is equivalent to maximizing a lower bound of the MAP solution under probabilistic model (22) with p = 1 in the prior definition.

In the general case, we have:

$$\begin{array}{l}{\displaystyle \sum _{j=1}^d\ln }{\Vert {\mathit{\alpha }}^j\Vert }_p=\frac{1}{p}{\displaystyle \sum _{j=1}^d\ln }\left({\displaystyle \sum _{t=1}^T{|c_j{\beta }_j^t|}^p}\right)\\ =\frac{1}{p}{\displaystyle \sum _{j=1}^d\ln }({(c_j^k)}^{\frac{p}{k}}\cdot {\displaystyle \sum _{t=1}^T{|{\beta }_j^t|}^p})\\ =\frac{1}{k}{\displaystyle \sum _{j=1}^d\ln }{|c_j|}^k+\frac{1}{p}{\displaystyle \sum _{j=1}^d\ln }{\displaystyle \sum _{t=1}^T{|{\beta }_j^t|}^p}\\ \le \frac{1}{k}{\displaystyle \sum _{j=1}^d{|c_j|}^k}+\frac{1}{p}{\displaystyle \sum _{t=1}^T{\Vert {\mathit{\beta }}_t\Vert }_p^p-d(\frac{1}{k}+\frac{1}{p}).}\end{array}$$

These inequalities lead to an upper bound of J in (27). By minimizing the upper bound, the problem is formulated as:

$$\underset{\mathbf{A}}{\min }{J}_{p,k}={\displaystyle \sum _{t=1}^{T}L}({\mathit{\alpha }}_t,{\mathbf{X}}_t,{\mathbf{y}}_t)+\frac{T}{k}{\Vert \mathbf{c}\Vert }_k^k+\frac{T}{p}{\displaystyle \sum _{t=1}^T{\Vert {\mathit{\beta }}_t\Vert }_p^p},$$ (29)

which is equivalent to the general multiplicative formulation in (2). Therefore we have proved the following theorem:

Theorem 15

Optimizing the multiplicative formulation (2), in the form of (29), is equivalent to maximizing a lower bound of the MAP solution under probabilistic model (22) with p ∈ (1, ∞) in the prior definition.

5. Optimization Algorithm

Alternating optimization algorithms have been used in both of the early methods (Bi et al., 2008; Lozano and Swirszcz, 2012) to solve Problem (2) which alternate between solving two subproblems: solve for β_t with fixed c; solve for c with fixed β_t. The convergence property of such an alternating algorithm has been analyzed in Bi et al. (2008); Lozano and Swirszcz (2012) that it converges to a local minimizer. In these early methods, both of the two subproblems have to be solved using iterative algorithms such as gradient descent, linear or quadratic program solvers.

Besides the algorithm that solves for c and β_t alternatively and can be applied to our formulations, we design an alternating optimization algorithm that utilizes the closed-form solution for c we have derived in Lemma 4 and the property that both Problems (2) and (5) are equivalent to the intermediate Problem (6) (or Problem (9)). In the algorithm to solve Problem (2), we start from an initial choice of c. At iteration s, we start from c^s, and solve for ${\mathit{\beta }}_t^s$ with the fixed c^s. We then compute the value of ${\mathit{\alpha }}_t^s=\text{diag}({\mathbf{c}}^s){\mathit{\beta }}_t^s$, which is used to update c^s to c^s+1 according to Eq.(10) in Lemma 4. The overall procedure is summarized in Algorithm 1. As a central idea in designing this algorithm, at each iteration we update β to reduce the loss function, and update c to reduce the regularizer while maintaining the same loss function value. Note that if the value of α is fixed, the loss will remain the same.

To analyze the convergence property of Algorithm 1, we utilize the fact that Problem (2) and Problem (9) are equivalent. For notational convenience, we denote the objective function of Problem (2) by g(B, c) that takes inputs β_t and c. We denote the objective function of Problem (9) by f(A, c). Both objective functions comprise the sum of three parts. For instance, f can be written as follows:

$$f(\mathbf{A},\mathbf{c})=f_0(\mathbf{A},\mathbf{c})+f_{\mathbf{A}}(\mathbf{A})+f_{\mathbf{c}}(\mathbf{c}),$$ (31)

where $f_0(\mathbf{A},\mathbf{c})={\gamma }_1{\sum }_{j=1}^d(c_j^{-p}{\sum }_{t=1}^T{({\alpha }_j^t)}^p)$ is the part that involves both α and c, $f_{\mathbf{A}}(\mathbf{A})={\sum }_{t}L({\mathit{\alpha }}_t,{\mathbf{X}}_t,{\mathbf{y}}_t)$ is the part relying only on α, and $f_{\mathbf{c}}(\mathbf{c})={\mu }_2{\sum }_{j=1}^d{(c_j)}^k$ is the part for c only. Let z be the vector consisting of all variables in Problem (9). Similar to what has been defined in Tseng (2001), we define that the point z is a stationary point of f if z ∈ dom f where dom f is the feasible region of f, and

$$\underset{\mathrm{\lambda }\to 0}{\lim }[f(\mathbf{z}+\mathrm{\lambda }\mathbf{b})-f(\mathbf{z})]/\mathrm{\lambda }\ge 0,\forall \mathbf{b}\text{such that}(\mathbf{z}+\mathrm{\lambda }\mathbf{b})\in \mathit{dom}f.$$ (32)

Algorithm 1.

The blockwise coordinate descent algorithm for multiplicative MTFL

Input: Xt, yt, t = 1,⋯, T, as well as γ1, γ2, p and k

Initialize: cj = 1, ∀j = 1,⋯, d, and s = 1

repeat

Compute ${\tilde{\mathbf{X}}}_t={\mathbf{X}}_t\text{diag}({\mathbf{c}}^s)$, ∀ t = 1,⋯, T

for t = 1,⋯, T do

Solve the following problem for ${\mathit{\beta }}_t^s$

$$\underset{{\mathit{\beta }}_t}{\min }L({\mathit{\beta }}_t,{\tilde{\mathbf{X}}}_t,{\mathbf{y}}_t)+{\gamma }_1{\Vert {\mathit{\beta }}_t\Vert }_p^p$$ (30)

end for

Compute ${\mathit{\alpha }}_t^s=\text{diag}({\mathbf{c}}^s){\mathit{\beta }}_t^s$

Set s = s + 1

Compute cs+1 using ${\mathit{\alpha }}_t^s$ according to Eq.(10)

until ${max}_{t,j}(|{({\alpha }_j^t)}^s-{({\alpha }_j^t)}^{s-1}|)<\epsilon$ (or other proper termination rules)

Output: αt, c and βt, t = 1,⋯, T

where b denotes any feasible direction, (which corresponds to ∇f(z) = 0 if f is differentiable, or 0 ∈ ∂f(z) if f is non-differentiable where ∂f (z) is the subgradient of f at z). In our case, f is not differentiable at α = 0 or c = 0 when p is set to an odd number. We also define that a point z is a coordinate-wise minimum point of f if z ∈ dom f, and $\forall {\mathbf{b}}_k\in {\mathbb{R}}^{d_k}$ that makes (0,⋯, b_k,⋯, 0) a feasible direction, there exists a small ε > 0, such that for all positive λ ≤ ε,

$$f(\mathbf{z}+\mathrm{\lambda }(0,\cdots ,{\mathbf{b}}_k,\cdots ,0))\ge f(\mathbf{z}),$$ (33)

where k indexes the blocks of variables in our algorithm, which include α₁, ⋯, α_T and c, so k = {1, ⋯, T + 1}, and the (T + 1)-th block is for c, d_k is the number of variables in the kth coordinate block and in our case d_k = d. The vector (0,⋯, b_k,⋯,0) is a vector in ℝ^d×(T+1) and used to only vary z in the k-th block.

We first prove that for the sequence of points generated by Algorithm 1, the objective function f is monotonically non-increasing. Then we prove the sequence of points is bounded, because of which, the sequence will have accumulation points. We prove that each accumulation point is a coordinate-wise minimum point. Then according to Tseng (2001), if f is regular at an accumulation point z*, this z* is a stationary point.

Lemma 16

Let the sequence of iterates generated by Algorithm 1 be ${\mathbf{z}}^s={\{{\mathbf{A}}^s,{\mathbf{c}}^s\}}_{s=1,2,\cdots }$ and the function f is defined by (31), then f(A^s+1, c^s+1) ≤ f(A^s, c^s).

Proof

First, note that for each {A^s, c^s}, we accordingly have {B^s, c^s} where ${\mathit{\alpha }}_t^s=\text{diag}({\mathbf{c}}^s){\mathit{\beta }}_t^s$, ∀t, and we have f(A^s, c^s) = g(B^s, c^s). Because we start with c so at each iteration c gets updated first (the order dose not matter actually). At iteration s + 1, we compute c^s+1 based on A^s according to Eq.(10). According to Lemma 4, c^s+1 will reach the lower bound of f when A is fixed to A^s. Hence f(A^s, c^s+1) ≤ f(A^s, c^s). Moreover, when c is updated, there is an implicit new value of ${\tilde{\mathbf{B}}}^s$ that is just computed as ${({\tilde{\beta }}_j^t)}^s={({\alpha }_j^t)}^s/{(c_j)}^{s+1}$, ∀_j and t. Then, $g({\tilde{\mathbf{B}}}^s,{\mathbf{c}}^{s+1})=f({\mathbf{A}}^s,{\mathbf{c}}^{s+1})$. Next in Algorithm 1, we obtain B^s+1 by solving Problem (30), i.e., by optimizing g with respect B when c is fixed to c^s+1. Hence, $g({\mathbf{B}}^{s+1},{\mathbf{c}}^{s+1})\le g({\tilde{\mathbf{B}}}^s,{\mathbf{c}}^{s+1})$. Then A will be updated by A^s+1 = diag(c^s+1)B^s+1, which leads to f(A^s+1, c^s+1) = g(B^s+1, c^s+1). Overall, we have

$$f({\mathbf{A}}^{s+1},{\mathbf{c}}^{s+1})=g({\mathbf{B}}^{s+1},{\mathbf{c}}^{s+1})\le g({\tilde{\mathbf{B}}}^s,{\mathbf{c}}^{s+1})=f({\mathbf{A}}^s,{\mathbf{c}}^{s+1})\le f({\mathbf{A}}^s,{\mathbf{c}}^s).$$

This proves that f(A^s+1, c^s+1) ≤ f(A^s, c^s).  ■

Based on the proof of Lemma 16, we can also show that g(B^s+1, c^s+1) ≤ g(B^s, c^s) for the sequence of {B^s, c^s}_s=1,2,⋯ that is also created by Algorithm 1.

Lemma 17

The sequence of iterates generated by Algorithm 1, z^s = {A^s, c^s}_s=1,2,⋯, (or equivalently, {B^s, c^s}_s=1,2,⋯) is bounded.

Proof

According to Lemma 16, we know that f(A^s+1, c^s+1) ≤ f(A^s, c^s), and g(B^s+1, c^s+1) ≤ g(B^s, c^s), ∀s = 1, 2, ⋯. Hence, g(B^s, c^s) ≤ g(B¹, c¹), ∀s = 1, 2, ⋯. Let g(B¹, c¹) = C. Then, g(B^s, c^s) is upper bounded by C.

In our algorithm, we assume that the loss function in g can be either the least squares loss or the logistic regression loss of all the tasks. Hence, the loss terms are non-negative (actually most other loss functions, such as the hinge loss, are also non-negative). The two regularizers, one on β_t and the other on c, are both non-negative. Thus, we have that ${\Vert {\mathit{\beta }}_t^s\Vert }_p^p\le C/{\gamma }_1$ and ${\Vert {\mathbf{c}}^s\Vert }_k^k\le C/{\gamma }_2,$ ∀s = 1, 2, ⋯. This shows that the sequence of {B^s, c^s}_s=1,₂, ⋯ is bounded. Because ${\mathit{\alpha }}_t^s=\text{diag}({\mathbf{c}}^s){\mathit{\beta }}_t^s$, ${\Vert {\mathit{\alpha }}_t^s\Vert }_p^p={\sum }_{j=1}^d{|{({\alpha }_j^t)}^s|}^p={\sum }_{j=1}^d{|(c_j^s{({\beta }_j^t)}^s|}^p\le {\Vert {\mathbf{c}}^s\Vert }_{2p}^{2p}+{\Vert {\mathit{\beta }}_t^s\Vert }_{2p}^{2p}\le \tilde{C}$ where $\tilde{C}$ is a constant computed from C, γ₁ and γ₂. Thus, the sequence of z^s = {A^s, c^s}_s=1,2,⋯ is also bounded.  ■

Theorem 18

The sequence z^s = {A^s, c^s}_{s=1,2, ⋯} generated by Algorithm 1 has at least one accumulation point. For any accumulation point z* = {A*, c*}, z* is a coordinate-wise minimum point of f.

Proof

According to Lemma 17, the sequence z^s is bounded, so there exists at least one accumulation point z* and a subsequence of {z^s}_s=1,2,⋯ that converges to z*. Without loss of generality and for notational convenience, let us just assume that {z^s}_s=1,2,⋯ converges to z*. Because z* is an accumulation point, if it is the iterate at the current iteration s, then in the next iteration s + 1, the same iterate z* will be obtained. Hence, ${\mathit{\beta }}_t^{\ast }$ is the optimal solution of Problem (30) when c is set to c* (and all other ${\mathit{\beta }}_{k\ne t}={\mathit{\beta }}_k^{\ast }$). Correspondingly, c* is the optimal solution of Problem (2) when B is set to B*. Hence, for any feasible direction (0, ⋯, b_t, ⋯, 0), ∀t = 1,2,⋯,T + 1, we have

$$f({\mathbf{z}}^{\ast }+\mathrm{\lambda }(0,\cdots ,{\mathbf{b}}_t,\cdots ,0))\ge f({\mathbf{z}}^{\ast }),$$ (34)

for small λ values. Hence, z* is a coordinate-wise minimum point of f.  ■

Theorem 19

If both p and k are positive, and k ≥ 1, the accumulation point z* is a stationary point of f.

Proof

Due to Lemma 4, we know that the optimal solution of Problem (9) can only occur when c and α_t satisfy Eq.(10), so any other points can be excluded from discussion. Hence, we define a new level set $Z^0=\{\mathbf{z}|f(\mathbf{z})\le C\}\cap \{\mathbf{z}|c_j={({\gamma }_1{\gamma }_2^{-1}{\sum }_{t=1}^T{({\alpha }_j^t)}^p)}^{\frac{1}{p+k}},\forall {\alpha }_j^t\}$. We now prove that f is continuous on this set Z⁰ and the gradient of f exists when p is even (differentiable case), and examine 0 ∈ ∂f at z* for non-differentiable cases.

Given the definition of f, f is continuous with respect to ${\alpha }_j^t$, ∀_j and t. Because f₀ is a division term that has a divisor based on c_j, we have the division-by-0 issue, so f is in general not continuous with respect to c_j at 0 (but continuous and differentiable at other values). However, using L’Hospital’s rule (i.e., $\underset{\varphi \to 0}{\lim }\frac{f_1(\varphi )}{f_2(\varphi })=\underset{\varphi \to 0}{\lim }\frac{f_1^{\prime }(\varphi )}{f_2^{\prime }(\varphi })$, we show that f is continuous with respect to c_j at c_j =0 in the set Z⁰. When c_j = 0, ${\Vert {\mathit{\alpha }}^j\Vert }_p^p={\sum }_{t=1}^T{({\alpha }_j^t)}^p=0$ due to Eq.(10). Let $\varphi ={\Vert {\mathit{\alpha }}^j\Vert }_p^p$. Since the function f₀ is a ratio of two items both approaching 0 as functions of ϕ, we can apply L’Hospital’s rule as follows

$$\underset{\varphi \to 0}{\lim }\frac{{\Vert {\mathit{\alpha }}^j\Vert }_p^p}{{(c_j)}^p}=\underset{\varphi \to 0}{\lim }\frac{\varphi }{{({\gamma }_1{{\gamma }_2}^{-1}\varphi )}^{\frac{p}{p+k}}}=\underset{\varphi \to 0}{\lim }\frac{1}{\frac{p}{p+k}{({\gamma }_1{{\gamma }_2}^{-1})}^{\frac{p}{p+k}}{\varphi }^{-\frac{p}{p+k}}}=\underset{\varphi \to 0}{\lim }\frac{(p+k){\varphi }^{\frac{k}{p+k}}}{p{({\gamma }_1{{\gamma }_2}^{-1})}^{\frac{p}{p+k}}}=0.$$

We then compute the partial derivative of f₀ with respect to c_j, which is $\frac{\partial f_0}{\partial c_j}=\frac{-p{\Vert {\mathit{\alpha }}^j\Vert }_p^p}{{(c_j)}^{p+1}}$ for c_j ≠ 0. Now when c_j approaches 0, we can prove continuity using L’Hospital’s rule:

$$\frac{\partial f_0}{\partial c_j}{|}_{c_j\to 0}=\underset{\varphi \to 0}{\lim }\frac{-p{\Vert {\mathit{\alpha }}^j\Vert }_p^p}{{(c_j)}^{p+1}}=\underset{\varphi \to 0}{\lim }\frac{-p\varphi }{{({{\gamma }_1{\gamma }_2}^{-1}\varphi )}^{\frac{p+1}{p+k}}}=\underset{\varphi \to 0}{\lim }\frac{-p(p+k){\varphi }^{\frac{k-1}{p+k}}}{(p+1){({{\gamma }_1{\gamma }_2}^{-1})}^{\frac{(p+1)}{p+k}}}=0,$$

when k > 1. When k = 1, the limit is a finite number $-p(p+k)/((p+1){({{\gamma }_1{\gamma }_2}^{-1})}^{\frac{(p+1)}{p+k}})$. Note that when an odd p is taken, ${\Vert {\mathit{\alpha }}_t\Vert }_p^p=\sum {|{\alpha }_j^t|}^p$ is not differentiable at 0. However, the above limit exists no matter f is differentiable or not because we take ϕ as the varying parameter. With the above conditions, we use the results in Tseng (2001) that when each subproblem has a unique minimum, which is the case in our algorithm because Subproblem (30) is strictly convex (for our chosen loss functions) and we have already proved the unique analytical solution of c, z^* is a stationary point of f.  ■

We briefly discuss the computation cost of Algorithm 1. Subproblem (30) is essentially for single task learning, which can be solved by many existing efficient algorithms, such as gradient-based optimization methods. The second subproblem has a closed-form solution for c, which requires only a minimal level of computation. The computation cost of Algorithm 1 only linearly increases with the number of tasks. Due to the nature of Algorithm 1, it can be easily parallelizable and distributed to multiple processors when optimizing individual β_t. More efficient algorithms may be designed for specific choices of p in the future.

6. Experiments

We empirically evaluated the performance of the multiplicative MTFL algorithms on both synthetic data sets and a variety of real-world data sets, where we solved either classification (using the logistic regression loss) or regression (using the least squares loss) problems. In the experiments, we implemented and compared Algorithm 1 for four parameter settings: (p, k) = (2, 2), (1, 1), (2, 1), and (1, 2), corresponding to four multiplicative MTFL (MMTFL) methods as listed in Table 2. Although when the values of (p, k) were (2,2) and (1,1), the two models corresponded respectively to the same methods in Bi et al. (2008) and Lozano and Swirszcz (2012), they were solved differently from prior methods using our Algorithm 1 with higher computational efficiency. When (p, k) = (2, 1) and (1, 2), the resultant models corresponded to the two new formulations.

In our experiments, the multiplicative MTFL methods were also compared with the additive MTFL methods that decompose the model parameters into an addition of two components, such as the Dirty model (DMTL) (Jalali et al., 2010) and the robust MTFL (rMTFL) (Gong et al., 2012). Single task learning (STL) approaches were also implemented as baselines and compared with all of the MTFL algorithms in the experiments. We list the various methods used for comparison in our experiments as follows:

• STL_lasso : Learning each task independently with ‖α_t‖₁ as the regularizer.

• STL_ridge : Learning each task independently with ${\Vert {\mathit{\alpha }}_t\Vert }_2^2$ as the regularizer.

• DMTL (Jalali et al., 2010) : The dirty model with regularizers ‖P‖_1,1 and ‖Q‖_1,∞, where A = P + Q.

• rMTFL (Gong et al., 2012) : Robust multitask feature learning with the regularizers ‖P‖_1,2 and ‖Q^T‖_1,2, where A = P + Q.

• MMTFL{2, 2} (Bi et al., 2008) : Multiplicative multitask feature learning with regularizers ${\Vert \mathbf{B}\Vert }_F^2$ and ${\Vert \mathbf{c}\Vert }_2^2$, where A = diag(c)B.

• MMTFL {1, 1} (Lozano and Swirszcz, 2012) : Multiplicative multitask feature learning with regularizers ‖B‖_1,1 and ‖c‖₁, where A = diag(c)B.

• MMTFL {2, 1} (New formulation 1) : Multiplicative multitask feature learning with regularizers ${\Vert \mathbf{B}\Vert }_F^2$ and ‖c‖₁, where A = diag(c)B.

• MMTFL {1, 2} (New formulation 2) : Multiplicative multitask feature learning with regularizers ‖B‖_1,1 and ${\Vert \mathbf{c}\Vert }_2^2$, where A = diag(c)B.

In all experiments, unless otherwise noted, the original data set was partitioned to have 25%, 33% or 50% of the data in a training set and the rest used for testing. For each specified partition ratio (corresponding to a trial), we randomly partitioned the data 15 times and reported the average performance. The same tuning process was used to tune the hyperparameters (e.g.,γ₁ and γ₂) of every method in the comparison. In every trial, an internal three-fold cross validation (CV) was performed within the training data of the first partition to select a proper hyperparameter value for each of the methods from the choices of 2^k with k = −10, −9, ⋯,7. In the subsequent partitions of each trial, the hyperparameters were fixed to the values that yielded the best performance in the CV.

The regression performance was measured by the coefficient of determination, denoted by R², which measures the explained variance of the data by the fitted model. In particular, we used the following formula to report performance $R^2=1-\frac{{\sum }_{i=1}^n{(y_i-f_i)}^2}{{\sum }_{i=1}^n{(y_i-\overline{y})}^2}$ where $\overline{y}$ is the mean of the observed values of y and f_i is the prediction of the observed y_i. The reported values in our experiments were the averaged R² over all tasks. The R² value ranges from 0 to 1, and a higher value indicates better regression performance. The classification performance was measured by the F1 score, which is the harmonic mean of precision and recall. The numbers we reported in each trial was the F1 score averaged over all tasks. Similarly, the F1 score also ranges from 0 to 1 and higher values represent better classification performance.

6.1 Simulation Studies

We created three categories of data sets, all of which were designed for regression experiments to evaluate the behaviors of the different methods. The data sets in each category were created with a pre-specified feature sharing structure. The first two categories were designed to validate the scenarios that we hypothesized for our two new formulations to work. We performed sensitivity analyses using these two categories of data sets, i.e., studying how performance was altered when the number of tasks or the number of features varied. Because we also empirically compared with a few additive decomposition based methods, it would be interesting to see how multiplicative MTFL behaved in a scenario that was actually in favor of additive MTFL. Hence, in the third category, the feature sharing structure was generated following the assumption that the robust MTFL method used (Gong et al., 2012).

In all experiments, we created input examples X_t for each task t using a number of features randomly drawn from the standard multivariate Gaussian distribution $\mathcal{N}(\mathbf{0},\mathbf{1})$, and pre-defined A = [α₁,⋯,α_T] across all tasks. The responses for each task was computed by y_t = X_tα_t + ε_t where ${\mathit{\epsilon }}_t\sim \mathcal{N}(0,0.5)$ was the noise introduced to the model. If an entry of A was set to non-zero, its value was randomly sampled from a uniform distribution in the interval [0.5, 1.5]. As a side note, we transposed A in Figure 1 for better illustration.

Figure 1.

The true parameter matrix versus the parameter matrices constructed by the various methods on synthetic data. The figure shows the results when 200 examples and 100 features were created for 20 tasks each. Darker color indicates larger values in magnitude.

6.1.1 Synthetic Datasets

Category 1 (C1)

In the C1 experiments, 40% of the rows in the matrix A were set to 0. Since each row corresponded to a feature across the tasks, the zero rows made the features irrelevant to all tasks. All remaining features received non-zero values in A so that they were used in every task’s model although with different combination weights. Hence, the individual models were sparse with respect to all synthesized features, but not sparse with respect to the selected features. The pre-defined A is demonstrated in Figure 1a(top), where we transpose A to have columns representing features and a darker spot indicates that the particular element of A had a larger absolute value. Note that this synthetic data follows the assumption that has motivated the early block-wise joint regularized methods that used matrix-norm-based regularizers. Those methods were developed with an assumption that a subset of features was shared by all tasks. However, we observed that the level of shrinkage needed would be different for c and β, which corresponded to non-matrix-norm-based regularizers. We hypothesized that the proposed new formulation (3) would produce better regression performance than existing models on this data category.

To examine how the number of tasks influenced the performance of different methods, we varied the number of tasks from 1, 5, 10, 50, and 100 to 1000. For each task, we created 200 examples, each represented by 100 features. We also tested the different methods when increasing the number of features. In this set of experiments, we used 20 tasks and each task contained 1000 examples. Each example was represented by a number of features ranging from 200 to 1000 with a step size of 200.

Category 2 (C2)

In the C2 experiments, 10% of the rows in A were set to non-zero and these features were shared by all tasks. We then arranged the tasks to follow six different sparse structures (the staircases) as shown in Figure 1b(top), where we once again transpose A. Each of the remaining features except the 10% common features was used by a comparatively small proportion of the tasks. Consecutive tasks were grouped such that the neighboring groups of tasks shared 7% of the features besides the 10% common features, whereas the non-neighboring groups of tasks did not share any features. Therefore, no feature could be excluded from all tasks, but a majority of individual features (90%) was only useful for few tasks (i.e., the useful features for one task were sparse). In this case, the non-sparsity-inducing norm was suitable for regularizing c and sparsity-inducing norm was more suitable for regularizing β. We hypothesized that the new formulation (4) would produce better regression performance than the other models on this data set.

We created 20, 50, 100, 500 and 1000 tasks, respectively, to test the algorithms’ sensitivity to the number of tasks. The numbers of tasks were chosen to make sure enough tasks in each of the six groups. The number of tasks in each group ranged from 3 to 170. We generated 200 examples and 100 features for each task. We also created another set of C2 data sets with the number of features changing from 200 to 1000 with a step size of 200 for 20 tasks and 1000 examples for each task.

Category 3 (D3)

This category was synthesized following the model of the additive decomposition methods. It only contained one data set where 200 examples, each represented by 100 features, were generated for each of 20 tasks. The parameter matrix A = P + Q where 80 rows in P and 16 columns in Q were set to 0. The component P was used to indicate the subset of relevant features across all tasks, and the component Q was used to tell that there were outlier tasks that did not share features with other tasks. Given this simulation process, this data set would be in favor of the rMTFL model proposed in Gong et al. (2012). The designed model parameter matrix was shown in Figure 1c(top).

6.1.2 Performance on synthetic data sets

We first compared the regression performance of the different methods on the three categories of data sets. Table 3 shows the averaged R² values together with standard deviations for each method and each trial setting. The best results are shown in bold fonts. The results in Table 3 were obtained on synthetic data sets that had 20 tasks with 200 examples and 100 features for each task. We reported the test R² obtained on each data set when 25%, 33% and 50% of the data were used in training. From Table 3, we observe that the proposed formulation (3)(MMTFL(2,1)) consistently outperformed other models on C1 data sets, whereas the proposed model (4)(MMTFL(1,2)) consistently outperformed on C2 data sets. The results confirmed our hypotheses that the two proposed models could be more suitable for learning the type of sharing structures in C1 and C2. As anticipated, rMTFL model constantly outperformed other models on the C3 data set. Among the multiplicative MTFL methods, MMTFL(2,2) achieved similar performance to that of rMTFL (off only by 0.01 ~ 0.02 for average R²).

Table 3.

Comparison of the test R² values obtained by the different MTFL methods on synthetic data sets using different partition ratios for training data (where standard deviation 0 means that it is less than 0.01).

	Dataset	STL_lasso	STL_ridge	DMTL	rMTFL	MMTFL(2,2)	MMTFL(1,1)	MMTFL(2,1)	MMTFL(1,2)
	25%	0.32±0.02	0.45±0.01	0.60±0.02	0.58±0.02	0.64±0.02	0.54±0.03	0.73±0.02	0.42±0.04
C1	33%	0.54±0.03	0.62±0.02	0.73±0.01	0.61±0.02	0.79±0.02	0.76±0.01	0.86±0.01	0.65±0.03
	50%	0.70±0.02	0.86±0.01	0.75±0.01	0.66±0.01	0.86±0.01	0.88±0.01	0.90±0.01	0.84±0.01
	25%	0.42±0.03	0.33±0.01	0.36±0.01	0.46±0.01	0.45±0.01	0.35±0.05	0.46±0.02	0.49±0.02
C2	33%	0.77±0.02	0.63±0.01	0.42±0.02	0.63±0.03	0.69±0.02	0.75±0.01	0.67±0.03	0.83±0.02
	50%	0.95±0.01	0.89±0.01	0.81±0.01	0.83±0.01	0.91±0	0.95±0	0.92±0.01	0.97±0
	25%	0.28±0.03	0.43±0.03	0.50±0.04	0.55±0.03	0.54±0.03	0.31±0.02	0.43±0.02	0.34±0.02
C3	33%	0.47±0.01	0.56±0.02	0.60±0.02	0.65±0.02	0.64±0.02	0.45±0.04	0.60±0.04	0.47±0.03
	50%	0.77±0.01	0.78±0.01	0.76±0.02	0.83±0.01	0.81±0.02	0.75±0.01	0.79±0.02	0.76±0.01

In order to elucidate the different shrinkage effects of the different decomposition strategies and regularizers, we compared the true parameter matrix with the constructed parameter matrices by the six MTFL methods used in our experiments in Figure 1. From the results on the C1 and C2 data sets, we observe that only MMTFL(2,1), MMTFL(1,2) and MMTFL(1,1) produced reasonably sparse structures. The two additive decomposition methods could not yield a sufficient level of sparsity in the models. Although the unused features did receive smaller weights in general, they were not completely excluded. To evaluate the accuracy of feature selection, we quantitatively measured the discrepancy between the estimated models and the true model by computing the mean squared error (MSE) trace((A − A_est)^⊤(A − A_est)) where A_est was the matrix estimated by a method. We compared MSE values of individual mothods.

On the C1 data, MMTFL(2,1) learned better combination weights (darker areas) for the relevant features than MMTFL(1,1). MMTFL(1,1) appeared to be unnecessarily too sparse because the useful features received much smaller weights than needed (lighter than the true model). The smallest MSE was achieved by MMTFL(2,1) with a value of 0.1, and the second best model, MMTFL(1,1), had MSE = 0.2 whereas the rMTFL model had the largest MSE = 0.25.

On the C2 data, MMTFL(1,2) learned a model that was most comparable to the true model. Both MMTFL(1,2) and MMTFL(1,1) eliminated well the irrelevant features. However, if we compared the two rows corresponding to these two models in Figure 1, we could see that MMTFL(1,1) broke the staircases in several places (e.g., towards the lower right and the up left corners) by excluding more features than necessary. Note that the feature sharing patterns (particularly in synthetic data C2) may not be revealed by the recent methods on clustered multitask learning that cluster tasks into groups (Kang et al., 2011; Jacob et al., 2008; Zhou et al., 2011) because no cluster structure is present in Figure 1b. Rather, the sharing pattern in Figure 1b is actually in between the consecutive groups of tasks. MMTFL(1,2) had the smallest MSE = 0.05, which was smaller than that of the second best model MMTFL(1,1) by 0.025. DMTL received the largest MSE = 0.19.

Figure 1c shows the results on the C3 data set. MMTFL(2,1), MMTFL(1,2), and MMTFL(1,1) imposed excessive sparsity on the parameter matrix, which removed some useful features. The other three models, DMTL, rMTFL and MMTFL(2,2), produced similar parameter matrices, but rMTFL was originally designed to detect outlier tasks and thus was more favorable for this data set. The rMTFL model obtained the smallest MSE (0.03), and MMTFL(2,2) had a similar performance (MSE=0.04), which was the same as that of DMTL. On this data set, MMTFL(1,1) got the largest MSE (0.09). These results bring out an interesting observation that for the MTL scenarios that have outlier tasks but relevant tasks share the same set of features, MMTFL(2,2) (which corresponds to the very early joint regularized method using the ℓ_1,2 matrix norm) is most suitable among the multiplicative MTFL methods.

Figure 2 compares the performance of different methods when we vary the number of tasks in the C1 and C2 categories. On each data set, we used 33% of the data in training with 15 trials, and reported here the average R² values and standard deviation bars. From Figure 2, MMTFL(2,1) constantly performed the best among all methods on the C1 data sets (but not in the single task learning) whereas MMTFL(1,2) outperformed the other models on the C2 data sets. We also observed that on C2 data, MMTFL(1,1) obtained very similar performance to that of MMTFL(1,2) after the number of tasks reached 50. On this data category, almost all methods reached a stable level of accuracy after the number of tasks reached 50 except DMTL. DMTL continued to gain knowledge from more relevant tasks until it reached 500 tasks but it produced the lowest R² values among all methods. Overall, the results indicate that with the fixed dimension and sample size, when the number of tasks reaches a certain level, the transferable knowledge learned from the tasks can be saturated for a specific feature sharing structure. On C1 data, we observe that the performance was not always monotonically improved or non-degraded (for all methods) when more tasks were included, which may indicate that when an unnecessarily large number of tasks was used, it could add more uncertainty to the learning process.

Figure 2.

The regression performance of different models on synthetic data C1 and C2 when the number of tasks is varied.

Figure 3 compares the performance of different methods when we vary the number of features in the C1 and C2 categories. Obviously, when the problem dimension was higher, the learning problem became more difficult (especially when the number of tasks and sample size remained the same). All methods dropped their performance substantially with increasing numbers of features although MMTFL(2,1) and MMTFL(1,2) still outperformed other methods, respectively, on the C1 and C2 data sets. This figure also shows that MMTFL(1,1) performed well on the C2 data sets but much worse on the C1 data sets. DMTL produced good performance, close to that of MMTFL(2,1), on the C1 data sets.

Figure 3.

The regression performance of different models on synthetic data C1 and C2 when the number of features is varied.

6.2 Experiments with Benchmark Data

Extensive empirical studies were conducted on benchmark data sets where we tested the proposed multiplicative MTFL algorithms on ten real-world data sets. Among these data sets, three were for regression experiments and all others were for classification experiments. Characteristics of these data sets are summarized in the next section.

6.2.1 Benchmark Datasets

Sarcos (Argyriou et al., 2007)

Sarcos data were collected for a robotics problem of learning the inverse dynamics of a 7 degrees-of-freedom SARCOS anthropomorphic robot arm. Each observation has 21 features corresponding to 7 joint positions and their velocities and accelerations. We needed to map from the 21-dimensional input space to 7 joint torques, which corresponded to 7 tasks. For each task, we randomly selected 2000 cases for training and the remaining 5291 cases for test. Readers can consult with http://www.gaussianprocess.org/gpml/data/ for more details.

CollegeDrinking (Bi et al., 2013)

The college drinking data were collected in order to identify alcohol use patterns of college students and the risk factors associated with the binge drinking. The data set contained daily responses from 100 college students to a survey questionnaire measuring various daily measures, such as drinking expectation, negative affects, and level of stress, every day in a 30 day period. The goal was to predict the amount of nighttime drinks based on 51 daily measures for each student, corresponding to 100 regression tasks. Because there were only 30 records for each person, we used 66%, 75% and 80% of the records to form the training set, and the rest for test.

QSAR (Ma et al., 2015)

The quantitative structure-activity relationship (QSAR) methods are commonly used to predict biological activities of chemical compounds in the field of drug discovery. The data sets we used were collected from three different types of drug activities, including binding to cannabinoid receptor 1 (CB1), inhibition of dipeptidyl peptidase 4 (DPP4) and time dependent 3A4 inhibitions (TDI). For each activity, there were 200 molecule examples represented by 2618 features. Three regression models were constructed to simultaneously predict the targets −log(IC50)) of the CB1, DPP4 and TDI effectiveness based on the molecular features.

C.M.S.C. (Lucas et al., 2013)

The Climate Model Simulation Crashes (C.M.S.C.) data set contained records of simulated crashes encountered during climate model uncertainty quantification ensembles. The data set comprised 3 tasks. There were 180 examples for each task. Each example was represented by an 18-dimensional feature vector. Each task is formed by a binary classification problem, which was to predict simulation outcomes (either fail or succeed) from the input parameter values for a climate model.

Landmine (Xue et al., 2007)

The original Landmine data contained 29 data sets where sets 1–15 corresponded to the geographical regions that were highly foliated and sets 16–29 corresponded to the regions with bare earth or desert. Each data set could be used to build a binary classifier. We used the data sets 1–10 and 16–25 to form 20 tasks where each example was represented by 9 features extracted from radar images. The number of examples varied between individual tasks ranging from 445 to 690.

Alphadigits (Maurer et al., 2013)

This data set was composed of binary 20 × 16 images of the 10 digits and capital letters. We used all the images of digits to form 10 binary classification tasks. For each digit, there were 39 images in this data set. We labeled the images of a single digit as positive examples, and randomly selected other 39 images from other digits and labeled them as negative examples. All the pixels were concatenated to form a 320-dimensional feature vector for each image.

Underwatermine (Liu et al., 2009b)

This data set was originally used in the underwater mine classification problem that aimed to detect mines from non-mines based on the synthetic-aperture sonar images. The data set consisted of 8 tasks with sample sizes ranging from 756 to 3562 for each task, and each task was a binary classification problem. Each example was represented by 13 features.

Animal recognition (Kang et al., 2011)

This data set consisted of images from 20 animal classes. Each image was originally represented by 2000 features extracted using the bag of word descriptors from the Scale-invariant Feature Transform (SIFT), and then the dimensionality was reduced to 202 by a principal component analysis, retaining 95% of the data variance. For each animal class, there were 100 images. We formed 20 binary classification tasks where for each task, 100 positive examples were from a specific animal class and 100 negative examples were randomly sampled from other classes.

HWMA_base and HWMA_peak (Qazi et al., 2007; Bi and Wang, 2015)

The heart wall motion abnormality (HWMA) detection data set was used to analyze and predict if a heart had abnormal motion based on the image features extracted from stress test echocardiographs of 220 patients. The images were taken at the base dose and also peak dose of stress contrast. The wall of left ventricle was medically segmented into 16 segments, and 25 features were extracted from each segment. Every segment of every heart case was annotated by radiologists in terms of normal or abnormal motions. Thus, there were 16 binary classification tasks, each corresponding to one of the 16 heart segments, and each task comprised 220 examples.

6.2.2 Performance on real world data sets

Three real-world data sets, the Sarcos, CollegeDrinking and QSAR, were used in regression experiments. The performance of the different methods is summarized in Table 4 depicting the R² values averaged over the 15 re-partitions in each trial. MMTFL(2,1) achieved the best R² values on the Sarcos data set (in all of the 3 trials) and the CollegeDrinking data set (in 2 of the 3 trials). The Sarcos data appeared to be in favor of denser models given MMTFL(2,2) also performed reasonably well on this data set. MMTFL(1,2) models achieved the best performance on the QSAR data set consistently across all the 3 trial settings. On this data set, it was obvious that MMTFL(1,2) was more suitable, which indicated that most of the 2618 features were useful for some tasks, but the tasks shared few features between each other. The difference between the proposed models and the additively decomposed models ranged from 1 % to 10%, and most importantly, the trend was consistent for the proposed models to outperform on these data sets.

Table 4.

Comparison of the R² values of the different MTFL methods on the benchmark regression data sets.

Dataset		STL_lasso	STL_ridge	DMTL	rMTFL	MMTFL(2,2)	MMTFL(1,1)	MMTFL(2,1)	MMTFL(1,2)
SARCOS	25%	0.75±0.02	0.78±0.02	0.86±0	0.86±0	0.89±0	0.88±0	0.89±0.01	0.86±0.01
	33%	0.78±0.01	0.78±0.02	0.81±0.11	0.81±0.1	0.90±0	0.89±0	0.90±0.01	0.82±0.01
	50%	0.82±0.01	0.83±0.06	0.87±0.1	0.87±0.1	0.89±0	0.90±0.01	0.91±0.01	0.86±0.01
CollegeDrinking	66%	0.15±0.05	0.09±0.02	0.11±0.07	0.22±0.02	0.23±0.04	0.15±0.02	0.21±0.01	0.19±0.04
	75%	0.18±0.05	0.15±0.02	0.18±0.08	0.18±0.05	0.14±0.05	0.25±0.08	0.27±0.05	0.21±0.08
	80%	0.11±0.06	0.09±0.04	0.19±0.06	0.16±0.06	0.19±0.05	0.15±0.04	0.21±0.04	0.15±0.04
QSAR	25%	0.39±0.03	0.36±0.01	0.39±0.01	0.39±0.01	0.34±0.02	0.38±0.03	0.38±0.03	0.40±0.03
	33%	0.39±0.04	0.39±0.04	0.40±0.04	0.39±0.04	0.37±0.04	0.41±0.04	0.40±0.05	0.43±0.03
	50%	0.44±0.06	0.41±0.03	0.44±0.03	0.44±0.04	0.40±0.06	0.44±0.04	0.45±0.05	0.48±0.04

The other seven real-world data sets were used in classification experiments. Table 5 summarizes the results where the F1 scores were averaged across the 15 random splits in each trial together with standard derivations. MMTFL(2,1) models achieved consistently the best performance on the C.S.M.C. and Landmine data sets in comparison with other models. In particular, we observed both MMTFL(1,1) and the MMTFL(2,1) models produced the best F1 scores in the trial with 33% training split whereas MMTFL(2,1) outperformed all other models in the trial with other partition ratios. These two data sets may prefer across-task sparse models, indicating that many irrelevant features may exist in the data. For the remaining five data sets used in classification experiments, MMTFL(1,2) models showed generally better performance than all other models. The difference between the best model and other MMTFL models could reach 4% to 8%.

Table 5.

Comparison of the F1 scores of the different MTFL methods on the benchmark classification data sets.

Dataset		STL_lasso	STL_ridge	DMTL	rMTFL	MMTFL(2,2)	MMTFL(1,1)	MMTFL(2,1)	MMTFL(1,2)
C.S.M.C.	25%	0.30±0.04	0.31±0.03	0.36±0	0.26±0.01	0.40±0	0.39±0.01	0.43±0	0.39±0
	33%	0.37±0.02	0.37±0.06	0.37±0.01	0.39±0.01	0.40±0.01	0.45±0.01	0.45±0.01	0.39±0.01
	50%	0.40±0.02	0.46±0.01	0.37±0.01	0.44±0.01	0.48±0.01	0.46±0.01	0.49±0.01	0.46±0
Landmine	25%	0.14±0.01	0.06±0.01	0.04±0.01	0.06±0	0.1±0.01	0.17±0.01	0.17±0.01	0.17±0.01
	33%	0.13±0.01	0.15±0.01	0.16±0.01	0.16±0.01	0.1±0.01	0.18±0.01	0.18±0.01	0.12±0.01
	50%	0.15±0.01	0.15±0.01	0.18±0.01	0.21±0.01	0.11±0.01	0.19±0.01	0.23±0.01	0.15±0.01
Alphadigits	25%	0.86±0.01	0.77±0.01	0.86±0.01	0.86±0	0.85±0.01	0.84±0.01	0.78±0.01	0.88±0.01
	33%	0.90±0.01	0.87±0.01	0.90±0.01	0.90±0.01	0.87±0.01	0.85±0.01	0.81±0.01	0.91±0.01
	50%	0.90±0.01	0.87±0.01	0.91±0.01	0.92±0.01	0.91±0.01	0.90±0.01	0.85±0.01	0.93±0.01
Underwatermine	25%	0.24±0.01	0.27±0.02	0.21±0.01	0.27±0	0.29±0.01	0.24±0.01	0.25±0.01	0.27±0.01
	33%	0.25±0.01	0.31±0.02	0.31±0.01	0.29±0.01	0.3±0.01	0.3±0.01	0.27±0.01	0.32±0.01
	50%	0.27±0.01	0.28±0.01	0.32±0.01	0.34±0.01	0.32±0.01	0.34±0.01	0.35±0.01	0.37±0.01
Animal	25%	0.63±0.01	0.63±0.01	0.65±0.01	0.65±0	0.66±0.01	0.64±0.01	0.63±0.01	0.66±0.01
	33%	0.66±0.01	0.67±0.01	0.66±0.01	0.65±0.01	0.67±0.01	0.66±0.01	0.66±0.01	0.68±0.01
	50%	0.67±0	0.68±0.01	0.68±0.01	0.66±0.01	0.69±0.01	0.67±0.01	0.68±0.01	0.69±0.01
HWMA_base	25%	0.35±0.01	0.37±0.01	0.42±0.01	0.4±0	0.46±0.01	0.36±0.01	0.44±0.01	0.45±0.01
	33%	0.38±0.01	0.34±0	0.45±0.01	0.47±0.01	046±0.01	0.44±0.01	0.47±0.01	0.49±0.01
	50%	0.44±0.01	0.45±0.01	0.48±0.01	0.48±0.01	0.51±0.01	0.47±0.01	0.47±0.01	0.55±0.01
HWMA_peak	25%	0.41±0	0.48±0.01	0.47±0.01	0.47±0	0.53±0.01	0.53±0.01	0.51±0.01	0.54±0.01
	33%	0.42±0.01	0.47±0.01	0.47±0.01	0.5±0.01	0.56±0.01	0.55±0.01	0.52±0.01	0.53±0.01
	50%	0.44±0.02	0.50±0.02	0.49±0.01	0.51±0.01	0.56±0.01	0.56±0.01	0.55±0.01	0.58±0.01

In particular, for the Alphadigits data set, we used the raw pixels of the hand written digits as the features to build models. Each task aimed to learn a linear model in the original pixel dimensions to distinguish a digit from other nine digits. Thus, each model, or equivalently the weight vector of the linear model, could be re-shaped back into an image. Figure 4 compares the constructed models for each digit by each of the six MTFL methods. In the top of Figure 4, we illustrate some sample images. In the middle, we show the results by the four MMTFL methods whereas at the bottom we include the constructed additively decomposed models. Clearly, the additively decomposed models were much noisier and selected many undesirable features. Among the multiplicative MTFL methods, MMTFL(1,1) models were too sparse to trace out the digits. Overall, MMTFL(1,2) models were the closest to the shapes of the different digits. If we compare MMTFL(2,1) and (1,2), MMTFL(2,1) excluded too many features from all digits. It indicated that most of the pixels were useful for predicting a digit but not many pixels were shared by multiple digits.

Figure 4.

The models constructed by MTFL methods for individual digits using the pixels in hand written digit images. The models can be re-organized back into images. The pixel in the above images ranges from 0 to 1. The lighter, the closer to 1 for lucid illustration.

7. Conclusion

In this paper, we have studied a general framework of multiplicative multitask feature learning. By decomposing the model parameter of each task into a product of two components: the across-task feature indicator and task-specific parameters, and applying different regularizers to the two components, we can select features for individual tasks and also search for the shared features among tasks. We have examined the theoretical properties of this framework when different regularizers are applied and found that this family of methods creates models equivalent to those of the joint regularized multitask learning methods but with a more general form of regularization. Further, we show that this family consists of some convex and some non-convex formulations and specify the conditions to obtain convexity. An analytical formula is derived for the across-task component as related to the task-specific component, which sheds light on the different shrinkage effects in the various regularizers. An efficient algorithm is derived to solve the entire family of methods and also tested in our experiments for some chosen parameter values. Empirical results on synthetic data clearly show that there may not be a particular choice of regularizers that is universally better than other choices. We empirically show a few feature sharing patterns that are in favor of the two newly-proposed choices of regularizers. The extensive experimental results on real-world benchmark data sets also confirm the observation and demonstrate the advantages of the proposed two formulations over several existing methods.