Multi-task Vector Field Learning

Abstract

Multi-task learning (MTL) aims to improve generalization performance by learning multiple related tasks simultaneously and identifying the shared information among tasks. Most of existing MTL methods focus on learning linear models under the supervised setting. We propose a novel semi-supervised and nonlinear approach for MTL using vector fields. A vector field is a smooth mapping from the manifold to the tangent spaces which can be viewed as a directional derivative of functions on the manifold. We argue that vector fields provide a natural way to exploit the geometric structure of data as well as the shared differential structure of tasks, both of which are crucial for semi-supervised multi-task learning. In this paper, we develop multi-task vector field learning (MTVFL) which learns the predictor functions and the vector fields simultaneously. MTVFL has the following key properties. (1) The vector fields MTVFL learns are close to the gradient fields of the predictor functions. (2) Within each task, the vector field is required to be as parallel as possible which is expected to span a low dimensional subspace. (3) The vector fields from all tasks share a low dimensional subspace. We formalize our idea in a regularization framework and also provide a convex relaxation method to solve the original non-convex problem. The experimental results on synthetic and real data demonstrate the effectiveness of our proposed approach.

1 Introduction

In many applications, labeled data are expensive and time consuming to obtain while unlabeled data are abundant. The problem of using unlabeled data to improve the generalization performance is often referred to as semi-supervised learning (SSL). It is well known that in order to make semi-supervised learning work, some assumptions on the dependency between the predictor function and the marginal distribution of data are needed. The manifold assumption [15, 5], which has been widely adopted in the last decade, states that the predictor function lives in a low dimensional manifold of the marginal distribution.

Multi-task learning was proposed to enhance the generalization performance by learning multiple related tasks simultaneously. The abundant literature on multi-task learning demonstrates that the learning performance indeed improves when the tasks are related [4, 6, 7]. The key step in MTL is to find the shared information among tasks. Evgeniou et al. [12] proposed a regularization MTL framework which assumes all tasks are related and close to each other. Ando and Zhang [2] proposed a structural learning framework, which assumed multiple predictors for different tasks shared a common structure on the underlying predictor space. An alternating structure optimization (ASO) method was proposed for linear predictors which assumed the task parameters shared a low dimensional subspace. Arvind et al. [1] generalized the idea of sharing a subspace by assuming that all task parameters lie on a manifold.

In this paper, we consider semi-supervised multi-task learning (SSMTL). Although many SSL methods have been proposed in the literature [10], these methods are often not directly amenable to MTL extensions [18]. Liu et al. [18] proposed an SSMTL framework which encouraged related models to have similar parameters. However they require that related tasks share similar representations [9]. Wang et al. [19] proposed another SSMTL method under the assumption that the tasks are clustered [4, 14]. The cluster structure is characterized by task parameters of linear predictor functions. For linear predictors, the task parameters they used are actually the constant gradient of the predictor functions which form a first order differential structure. For general nonlinear predictor functions, we show it is more natural to capture the shared differential structure using vector fields.

In this paper, we propose a novel SSMTL formulation using vector fields. A vector field is a smooth mapping from the manifold to the tangent spaces which can be viewed as a directional derivative of functions on the manifold. In this way, a vector field naturally characterizes the differential structure of functions while also providing a natural way to exploit the geometric structure of data; these are the two most important aspects for SSMTL. Based on this idea, we develop the multi-task vector field learning (MTVFL) method which learns the prediction functions and the vector fields simultaneously. The vector fields we learned are forced to be close to the gradient fields of predictor functions. In each task, the vector field is required to be as parallel as possible. We say that a vector field is parallel if the vectors are parallel along the geodesics on the manifold. In extreme cases, when the manifold is a linear (or an affine) space, then the geodesics of such manifold are straight lines. In such cases, the space spanned by these parallel vectors is a simply one-dimensional subspace. Thus when the manifold is flat (i.e., with zero curvature) or the curvature is small, it is expected that these parallel vectors concentrate on a low dimensional subspace. As an example, we can see from Fig. 1 that the parallel field on the plane spans a one-dimensional subspace and the parallel field on the Swiss roll spans a two-dimensional subspace. For the multi-task case, these vector fields share a low dimensional subspace. In addition, we assume these vector fields share a low dimensional subspace among all tasks. In essence, we use a first-order differential structure to characterize the shared structure of tasks and use a second-order differential structure to characterize the specific parts of tasks. We formalize our idea in a regularization framework and provide a convex relaxation method to solve the original non-convex problem. We have performed experiments using both synthetic and real data; results demonstrate the effectiveness of our proposed approach.

Figure 1.

Examples of parallel fields. The parallel field on ℝ² spans a one dimensional subspace and the parallel field on the Swiss roll spans a two dimensional subspace.

2 Multi-task Learning: A Vector Field Approach

In this section, we first introduce vector fields and then present multi-task learning via exploring shared structure using vector fields.

2.1 Multi-task Learning Setting and Vector Fields

We first introduce notation and symbols. We are given m tasks, with n_l samples $x_i^l$, i = 1, …, n_l for the l-th task. The total number of samples is n = Σ_l n_l. For the l-th task, we assume the data { $x_i^l$} are on a d_l-dimensional manifold . All of these data manifolds are embedded in the same D-dimensional ambient space ℝ^D. It is worth noting that the dimensions of different data manifolds are not required to be the same. Without loss of generality, we assume the first $n_l^{\prime }(n_l^{\prime }<n_l)$ samples are labeled, with $y_j^l\in ℝ$ for regression and $y_j^l\in \{-1,1\}$ for classification, $j=1,\dots ,n_l^{\prime }$. The total number of labeled samples is $n^{\prime }={\sum }_l{n}_l^{\prime }$. For the l-th task, we denote the regression function or classification function by $f_l^{\ast }$. The goal of semi-supervised multi-task learning is to learn the function value on unlabeled data, i.e., $f_l^{\ast }(x_i^l),n_l^{\prime }+1\le i\le n_l$.

Given the l-th task, we first construct a nearest neighbor graph by either ε-neighborhood or k nearest neighbors. Let $x_i^l~x_j^l$ denote that $x_i^l$ and $x_j^l$ are neighbors. Let $w_{ij}^l$ denote the weight which measures the similarity between $x_i^l$ and $x_j^l$. It can be approximated by the heat kernel weight or the simple 0–1 weight. For each point $x_i^l$, we estimate its tangent space $T_{x_i^l}\mathcal{M}$ by performing PCA on its neighborhood. We choose the largest d_l eigenvectors as the bases since the tangent space $T_{x_i^l}\mathcal{M}$ has the same dimension as the manifold . Let $T_i^l\in {ℝ}^{D\times d_l}$ be the matrix whose columns constitute an orthonormal basis for $T_{x_i^l}\mathcal{M}$. It is easy to show that $P_i^l=T_i^l{T_i^l}^T$ is the unique orthogonal projection from ℝ^D onto the tangent space $T_{x_i^l}\mathcal{M}$ [13]. That is, for any vector a ∈ ℝ^m, we have $P_i^{l}a\in T_{x_i^l}\mathcal{M}$ and $(a-P_i^{l}a)\perp P_i^{l}a$.

We now formally define the vector field and show how to represent it in the discrete case.

Definition 2.1 ([16])

A vector field X on the manifold is a continuous map X : → T where T is the set of tangent spaces, written as p ↦ X_p, with the property that for each p ∈ , X_p is an element of T_p .

We can think of a vector field on the manifold as an arrow in the same way as we think of the vector field in the Euclidean space, with a given magnitude and direction attached to each point on the manifold, and chosen to be tangent to the manifold. A vector field V on the manifold is called a gradient field if there exists a function f on the manifold such that ∇f = V where ∇ is the covariant derivative on the manifold. Therefore, gradient fields are one kind of vector fields. It plays a critical role in connecting vector fields and functions.

Let V_l be a vector field on the manifold . For each point $x_i^l$, let $V_{x_i^l}$ denote the value of the vector field V_l at $x_i^l$. Recall the definition of vector field, $V_{x_i^l}$ should be a vector in the tangent space $T_{x_i^l}{\mathcal{M}}_l$. Therefore, we can represent it by the coordinates of the tangent space $T_{x_i^l}{\mathcal{M}}_l$ as $V_{x_i^l}=T_i^l{v}_i^l$, where $v_i^l\in {ℝ}^{d_l}$ is the local representation of $V_{x_i^l}$ with respect of $T_i^l$. Let f_l be a function on the manifold . By abusing the notation without confusion, we also use f_l to denote the vector $f_l={(f_l(x_l^1),\dots ,f_l(x_{n_l}^l))}^T$ and use V_l to denote the vector $V_l={\left({v_1^l}^T,\dots ,{v_{n_l}^l}^T\right)}^T\in {ℝ}^{d_l{n}_l}$. That is, V_l is a d_ln_l-dimensional big column vector which concatenates all the $v_i^l$’s for a fixed l. Then for each task, we aim to compute the vector f_l and the vector V_l.

2.2 Multi-task Vector Field Learning

In this section, we introduce multi-task vector field learning (MTVFL).

Many existing MTL methods capture the task relatedness by sharing task parameters. For linear predictors, the task parameters they used are actually the constant gradient vectors of the predictor functions. For general nonlinear predictor functions, we show it is natural to capture the shared differential structure using vector fields. Let f denote the vector ${(f_1^T,\dots ,f_m^T)}^T$ and V denote the vector ${(V_1^T,\dots ,V_m^T)}^T={({v_1^1}^T,\dots ,{v_{n_l}^m}^T)}^T$. We propose to learn f and V simultaneously:

• The vector field V_l should be close to the gradient field ∇f_l of f_l, which can be formularized as follows:

  $$\underset{f,V}{min}{R}_1(f,V)=\sum _{l=1}^m{R}_1(f_l,V_l):=\sum _{l=1}^m{\int }_{{\mathcal{M}}_l}{\Vert \nabla f_l-V_l\Vert }^2$$ (1)
• The vector field V_l should be as parallel as possible:

  $$\underset{V}{min}{R}_2(V)=\sum _{l=1}^m{R}_2(V_l):=\sum _{l=1}^m{\int }_{{\mathcal{M}}_l}{\Vert \nabla V_l\Vert }_{\text{HS}}^2$$ (2)

  where ∇ is the covariant derivative on the manifold, where ||·||_HS denotes the Hilbert-Schmidt tensor norm [11]. ∇V_l measures the change of the vector field, therefore minimizing ${\int }_{{\mathcal{M}}_l}{\Vert \nabla V_l\Vert }_{\text{HS}}^2$ enforces the vector field V_l to be parallel.

• All vector fields share an h-dimensional subspace where h is a predefined parameter:

  $$T_i^l{v}_i^l=u_i^l+{\mathrm{\Theta }}^T{w}_i^l,s.t.\mathrm{\Theta }{\mathrm{\Theta }}^T=I_{h\times h}.$$ (3)

Since these vector fields are assumed to share a low dimensional space, it is expected that the residual vector $u_i^l$ is small. We define another term R₃ to control the complexity as follows:

$$R_3(v_i^l,w_i^l,\mathrm{\Theta })=\sum _{l=1}^m\sum _{i=1}^{n_l}\alpha {\Vert u_i^l\Vert }^2+\beta {\Vert T_i^l{v}_i^l\Vert }^2$$ (4)

$$=\sum _{l=1}^m\sum _{i=1}^{n_l}\alpha {\Vert T_i^l{v}_i^l-{\mathrm{\Theta }}^T{w}_i^l\Vert }^2+\beta {\Vert T_i^l{v}_i^l\Vert }^2$$ (5)

Note that α and β are pre-specified coefficients, indicating the importance of the corresponding regularization component. Since we would like the vector field to be parallel, the vector norm is not expected to be too small. Besides, we assume the vector fields share a low dimensional subspace, the residual vector $u_i^l$ is expected to be small. In practice we suggest to use a small β and a large α. By setting β = 0, R₃ will reduce to the regularization term proposed in ASO if we also replace the tangent vectors by the task parameters. Therefore, this formulation is a generalization of ASO.

It can be verified that $w_i^{l^{\ast }}=\mathrm{\Theta }{T}_i^l{v}_i^l=arg{min}_{w_i^l}{R}_3(v_i^l,w_i^l,\mathrm{\Theta })$. Thus we have $u_i^l=T_i^l{v}_i^l-{\mathrm{\Theta }}^T{w}_i^l=(I-{\mathrm{\Theta }}^T\mathrm{\Theta })T_i^l{v}_i^l$. Therefore, we can rewrite R₃ as follows:

$$\begin{array}{l}{R}_3(V,\mathrm{\Theta })=\sum _{l=1}^m\sum _{i=1}^{n_l}\alpha {\Vert u_i^l\Vert }^2+\beta {\Vert T_i^l{v}_i^l\Vert }^2\\ =\sum _{l=1}^m\sum _{i=1}^{n_l}(\alpha {\Vert (I-{\mathrm{\Theta }}^T\mathrm{\Theta })T_i^l{v}_i^l\Vert }^2+\beta {\Vert T_i^l{v}_i^l\Vert }^2)\\ =\alpha V^T{A}_{\mathrm{\Theta }}V+\beta V^{T}HV\end{array}$$ (6)

where H is a block diagonal matrix with the diagonal blocks being ${T_i^l}^T{T}_i^l$, and A_Θ is another block diagonal matrix with the diagonal blocks being ${T_i^l}^T{(I-{\mathrm{\Theta }}^T\mathrm{\Theta })}^T(I-{\mathrm{\Theta }}^T\mathrm{\Theta })T_i^l={T_i^l}^T(I-{\mathrm{\Theta }}^T\mathrm{\Theta })T_i^l$. Therefore, the proposed formulation solves the following optimization problem:

$$\underset{f,V,\mathrm{\Theta }}{argmin}E(f,V,\mathrm{\Theta })=R_0(f)+{\lambda }_1{R}_1(f,V)+{\lambda }_2{R}_2(V)+{\lambda }_3{R}_3(V,\mathrm{\Theta })s.t.\mathrm{\Theta }{\mathrm{\Theta }}^T=I_{h\times h}$$ (7)

where R₀(f) is the loss function. For simplicity, we use the quadratic loss function $R_0(f)={\sum }_{l=1}^m{\sum }_{i=1}^{n_l^{\prime }}{(f_l(x_i^l)-y_i^l)}^2$.

2.3 Objective Function in the Matrix Form

To simplify Eq. (7), in this section we rewrite our objective function in the matrix form.

Using the discrete methods in [17], we have the following discrete form equations:

$$R_1(f_l,V_l)=\sum _i\sum _{j~i}{w}_{ij}^l{({(x_j^l-x_i^l)}^T{T}_i^l{v}_i^l-f_j^l+f_i^l)}^2$$ (8)

$$R_2(f_l,V_l)=\sum _{i~j}{w}_{ij}^l{\Vert P_i^l{T}_j^l{v}_j^l-T_i^l{v}_i^l\Vert }^2$$ (9)

Interestingly, with some algebraic transformations, we have the following matrix forms for our objective functions:

$$R_1(f_l,V_l)=2f_l^T{L}_l{f}_l+V_l^T{G}_l{V}_l-2V_l^T{C}_l{f}_l$$ (10)

where L_l is the graph Laplacian matrix, G_l is a d_ln_l × d_ln_l block diagonal matrix, and $C_l={[{C_1^l}^T,\dots ,{C_n^l}^T]}^T$ is a d_ln_l × n_l block matrix. Denote the i-th d_l × d_l diagonal block of G_l by $G_{ii}^l$ and the i-th d_l × n_l block of C_l by $C_i^l$, we have

$$G_{ii}^l=\sum _{j~i}{w}_{ij}^l(x_j^l-x_i^l){(x_j^l-x_i^l)}^T,C_i^l=\sum _{j~i}{w}_{ij}^l(x_j^l-x_i^l){s_{ij}^l}^T$$ (11)

where $s_{ij}^l\in {ℝ}^{n_l}$ is a selection vector of all zero elements except for the i-th element being −1 and the j-th element being 1. And R₂ becomes

$$R_2(V_l)=V_l^T{B}_l{V}_l$$ (12)

where B_l is a d_ln_l × d_ln_l sparse block matrix. If we index each d_l × d_l block by $B_{ij}^l$, then we have

$$B_{ii}^l=\sum _{j~i}{w}_{ij}^l(Q_{ij}^l{Q_{ij}^l}^T+I)$$ (13)

$$B_{ij}^l=\{\begin{array}{rr}\hfill -2w_{ij}^l{Q}_{ij}^l,& \text{if}{x}_i~x_j\hfill \\ \hfill 0,& \text{otherwise}\hfill \end{array}$$ (14)

where $Q_{ij}^l={T_i^l}^T{T}_j^l$. It is worth nothing that both R₁ and R₂ depend on tangent spaces $T_i^l$.

Thus we can further write R₁(f, V) and R₂(V) as follows

$$R_1(f,V)=\sum _{l=1}^m{R}_1(f_l,V_l)=2f^{T}Lf+V^{T}GV-2V^{T}Cf$$ (15)

$$R_2(V)=\sum _{l=1}^m{R}_2(V_l)=V^{T}BV$$ (16)

where L, G and B are block diagonal matrices with the corresponding l-th block matrix being L_l, G_l and B_l, respectively. C is a column block matrix with the l-th block matrix being C_l.

Let denote an n × n diagonal matrix where = 1 if the corresponding i-th data is labeled and = 0 otherwise. And let y ∈ ℝⁿ be a column vector whose i-th element is the corresponding label of the i-th labeled data and 0 otherwise. Then $R_0(f)=\frac{1}{n^{\prime }}{(f-y)}^T\mathbb{I}(f-y)$. Finally, we get the following matrix form for our objective function in Eq. (7) with the constraint ΘΘ^T = I_h×h as:

$$\begin{array}{l}E(f,V,\mathrm{\Theta })=R_0(f)+{\lambda }_1{R}_1(f,V)+{\lambda }_2{R}_2(V)+{\lambda }_3{R}_3(V,\mathrm{\Theta })=\frac{1}{n^{\prime }}{(f-y)}^T\mathbb{I}(f-y)+{\lambda }_1(2f^{T}Lf+V^{T}GV-2V^{T}Cf)+{\lambda }_2{V}^{T}BV+{\lambda }_3{V}^T(\alpha A_{\mathrm{\Theta }}+\beta H)V\\ =\frac{1}{n^{\prime }}{(f-y)}^T\mathbb{I}(f-y)+2{\lambda }_1{f}^{T}Lf+V^T({\lambda }_{1}G+{\lambda }_{2}B+{\lambda }_3(\alpha A_{\mathrm{\Theta }}+\beta H))V-2{\lambda }_1{V}^{T}Cf\end{array}$$

It is worth noting that matrices L, G, B, C depend on data, and only the matrix A_Θ is related to Θ.

3 Optimization

In this section, we discuss how to solve the following optimization problem:

$$\underset{f,V,\mathrm{\Theta }}{argmin}E(f,V,\mathrm{\Theta }),s.t.\mathrm{\Theta }{\mathrm{\Theta }}^T=I_{h\times h}$$ (17)

We use the alternating optimization to solve this problem.

• Optimization of f and V. For a fixed Θ, the optimal f and V can be obtained via solving

  $$\underset{f,V}{argmin}E(f,V,\mathrm{\Theta })$$ (18)
• Optimization of Θ. For a fixed V, the optimal Θ can be obtained via solving.

  $$\underset{\mathrm{\Theta }}{argmin}{R}_3(V,\mathrm{\Theta }),s.t.\mathrm{\Theta }{\mathrm{\Theta }}^T=I_{h\times h}$$ (19)

3.1 Optimization of f and V for a Given Θ

When Θ is fixed, the objective function is similar to that of the single task case. However, there are some differences we would like to mention. Firstly, when constructing the nearest neighbor graph, data points from different tasks are disconnected. Therefore when estimating tangent spaces, data points from different tasks are independent. Secondly, we do not require the dimension of tangent spaces from each task to be the same.

We note that

$$\frac{\partial E}{\partial f}=2\left(\frac{1}{n^{\prime }}\mathbb{I}+2{\lambda }_{1}L\right)f-2{\lambda }_1{C}^{T}V-2\frac{1}{n^{\prime }}y$$ (20)

$$\frac{\partial E}{\partial V}=-2{\lambda }_{1}Cf+2({\lambda }_{1}G+{\lambda }_{2}H+{\lambda }_3(\alpha A_{\mathrm{\Theta }}+\beta H))V$$ (21)

Requiring the derivatives to be vanish, we get the following linear system

$$\left(\begin{array}{cc}\frac{1}{n^{\prime }}\mathbb{I}+2{\lambda }_{1}L& -{\lambda }_1{C}^T\\ -{\lambda }_{1}C& {\lambda }_{1}G+{\lambda }_{2}B+{\lambda }_3(\alpha A_{\mathrm{\Theta }}+\beta H)\end{array}\right)\left(\begin{array}{c}f\\ V\end{array}\right)=\left(\begin{array}{c}\frac{1}{n^{\prime }}y\\ 0\end{array}\right)$$ (22)

Except for the matrix A_Θ, all other matrices can be computed in advance and will not change during the iterative process.

3.2 Optimization of Θ for a Given V

Since functions R₀(f), R₁(f, V) and R₂(V) are not related to the variable Θ, we only need to optimize R₃(V, Θ) subject to ΘΘ^T = I_h×h.

Recall Eq. (6), we rewrite R₃(V, Θ) as follows:

$$\begin{array}{l}\widehat{\mathrm{\Theta }}=\underset{\mathrm{\Theta }}{argmin}\sum _{l=1}^m\sum _{i=1}^{n_l}\alpha \left({\Vert (I-{\mathrm{\Theta }}^T\mathrm{\Theta })T_i^l{v}_i^l\Vert }^2+\frac{\beta }{\alpha }{\Vert T_i^l{v}_i^l\Vert }^2\right)\\ =\underset{\mathrm{\Theta }}{argmin}\alpha \text{tr}\left({\mathbf{V}}^T((1+\frac{\beta }{\alpha })I-{\mathrm{\Theta }}^T\mathrm{\Theta })\mathbf{V}\right)\\ =\underset{\mathrm{\Theta }}{argmax}\text{tr}(\mathrm{\Theta }{\mathbf{VV}}^T{\mathrm{\Theta }}^T)\end{array}$$ (23)

where $\mathbf{V}=(T_1^1{v}_1^1,\dots ,T_{n_m}^m{v}_{n_m}^m)$ is a D × n matrix with each column being a tangent vector. The optimal Θ̂ can be obtained by using singular value decomposition (SVD). Let $\mathbf{V}=Z_1\mathrm{\sum }{Z}_2^T$ be the SVD of V and we assume that the singular values are in a decreasing order in Σ. Then the rows of Θ̂ are given by the first h columns of Z₁.

3.3 Convex Relaxation

The orthogonality constraint in Eq. (23) is non-convex. Next, we propose to convert Eq. (23) into a convex formulation by relaxing its feasible domain into a convex set.

Let η = β/α. It can be verified that the following equality holds: (1 + η)I − Θ^TΘ = η (1 + η)(ηI + Θ^TΘ)⁻¹. Then we can rewrite R₃(V, Θ) as R₃(V, Θ) = αη (1 + η) tr V^T (ηI + Θ^T Θ)⁻¹V). Let M_e be defined as M_e = {M : M = Θ^TΘ, ΘΘ^T = I, Θ ∈ ℝ^h×d}. The convex hull [8] of M_e can be expressed as the convex set M_c given by $M_c=\{M:\text{tr}(M)=h,M\preccurlyeq I,M\in {\mathbb{S}}_{+}^d\}$ and each element in M_e is referred to as an extreme point of M_c.

To convert the non-convex problem Eq. (23) into a convex formulation, we replace Θ^T Θ with M, and naturally relax its feasible domain into a convex set based on the relationship between M_e and M_c presented above; this results in an optimization problem as

$$\underset{\mathrm{\Theta }}{argmin}{R}_3(\mathbf{V},M),s.t.,\text{tr}(M)=h,M\preccurlyeq I,M\in {\mathbb{S}}_{+}^d$$ (24)

where R₃(V, M) is defined as R₃(V, M) = αη (1 + η) tr V^T(ηI + M)⁻¹V). It follows from [3, Theorem 3.1] that the relaxed R₃ is jointly convex in V and M. After we obtain the optimal M, the optimal Θ can be approximated using the first h eigenvectors (corresponding to the largest h eigenvalues) of the optimal M.

4 Experiments

In this section, we evaluate our method on one synthetic data and one real data set. We compare the proposed Multi-Task Vector Field Learning (MTVFL) algorithm against the following methods: (a) Single Task Vector Field Learning (STVFL, or PFR), (b) Alternating Structure Optimization (ASO) and (c) its nonlinear version - Kernelized Alternating Structure Optimization (KASO). The kernel constructed in KASO uses both labeled data and unlabeled data. Thus it can be viewed as a semi-supervised MTL method.

4.1 Synthetic Data

We first construct a synthetic data to evaluate our method in comparison with the semi-supervised single task learning method (STVFL). We generate two data sets including Swiss roll and Swiss roll with hole embedded in 3-dimensional Euclidean space. The Swiss roll is generated by the following equations x = t₁ cos t₁; y = t₂; z = t₁ sin t₁ where t₁ ∈ [3π/2, 9π/2]; t₂ ∈ [0, 21]. The Swill roll with hole excludes points within t₁ ∈ [9, 12] and t₂ ∈ [9, 14]. The ground truth function is f(x, y, z) = t₁. This test is a semi-supervised multi-task regression problem. We randomly select a number of labeled data in each task and try to predict the value on other unlabeled data.

Each data set has 400 points. We construct a nearest neighbor graph for each task. The number of nearest neighbors is set to 5 and the manifold dimension is set to 2 as they are both 2 dimensional manifolds. The shared subspace dimension is set to 2. The regularization parameters are chosen via cross-validation. We perform 100 independent trials with randomly selected labeled sets. The performance is measured by the mean squared error (MSE). We also try ASO and KASO, however they perform poorly since the data is highly nonlinear. The averaged MSE over two tasks is presented in Fig. 2. We can observe that MTVFL consistently outperforms STVFL which demonstrates the effectiveness of SSMTL.

Figure 2.

(a) Performance of MTVFL and STVFL; (b) The singular value distribution.

We also show the singular value distribution of the ground truth gradient fields. Given the ground truth f, we can compute the gradient field V by taking derivatives of R₁(f, V) with respect to V. Requiring the derivative to vanish, we get the following equation GV = Cf. After obtaining V, the gradient vector $V_{x_i^l}$ at each point can be obtained as $V_{x_i^l}=T_i^l{v}_i^l$. Then we perform PCA on these vectors and the singular values of the covariance matrix of $V_{x_i^l}$ are shown in Fig. 2 (b). As can be seen from Fig. 2 (b), the number of dominant singular values is 2 which indicates that the ground truth gradient fields concentrate on a 2-dimensional subspace.

4.2 Landmine Detection

We use the landmine data set studied in [20]. There are totally 29 sets of data which are collected from various real landmine fields. Each data example is represented by a 9-dimensional vector with a binary label, which is either 1 for landmine or 0 for clutter. The problem of landmine detection is to predict the labels of unlabeled objects. Among the 29 data sets, 1–15 correspond to relatively highly foliated regions and 16–29 correspond to bare earth or desert regions. Following [20], we choose the data sets 1–10 and 16–24 to form 19 tasks.

The basic setup of all the algorithms is as follows. First, we construct a nearest neighbor graph for each task. The number of nearest neighbors is set to 10 and the manifold dimension is set to 4 empirically. These two parameters are the same for all 19 tasks. The shared subspace dimension is set to be 5 for both of MTVFL and ASO and the shared subspace dimension of KASO is set to 10. All the regularization parameters for the four algorithms are chosen via cross-validation. Note that KASO needs to construct a kernel matrix. We use Gaussian kernel in KASO and the Gaussian width is set to be optimal by searching within [0.01, 10].

We perform 100 independent trials with randomly selected labeled sets. We measure the performance by AUC which denotes area under the Receiver Operation Characteristic (ROC) curve. A large AUC value indicates good classification performance. Since the data have severely unbalanced labels, following [20], we do a special setting that assures there is at least one “1” and one “0” labeled sample in the training set of each task. The AUC averaged over the 19 tasks is presented in Fig. 3 (a). As can be seen, MTVFL consistently outperforms the other three algorithms. When the number of labeled data increases, KASO outperforms STVFL. ASO does not improve much when the amount of labeled data increases, which is probably because the data have severely unbalanced labels and the ground truth predictor function is nonlinear. We also show the singular value distribution of the ground truth gradient fields in Fig. 3 (b). The computation of the singular values is the same as in Section. 4.1. As can be seen from Fig. 3 (b), the number of dominant singular values is 5. The percentage of the sum of the first 5 singular values over the total sum is 91.34%, which indicates that the ground truth gradient fields concentrate on a 5-dimensional subspace.

Figure 3.

(a) Performance of various MTL algorithms; (b) The singular value distribution.

5 Conclusion

In this paper, we propose a new semi-supervised multi-task learning formulation using vector fields. We show that vector fields can naturally capture the shared differential structure among tasks as well as the structure of the data manifolds, both of which are crucial for semi-supervised multi-task learning. Our experimental results on synthetic and real data demonstrate the effectiveness of the proposed method. There are several interesting future directions suggested in this work. One is the relation between learning on task parameters and learning on vector fields. Ultimately, both of them are learning functions. Another direction is to apply other assumptions made in the multi-task learning community into vector field learning, e.g., the cluster assumption.