A Sparse Interactive Model for Matrix Completion with Side Information

Abstract

Matrix completion methods can benefit from side information besides the partially observed matrix. The use of side features that describe the row and column entities of a matrix has been shown to reduce the sample complexity for completing the matrix. We propose a novel sparse formulation that explicitly models the interaction between the row and column side features to approximate the matrix entries. Unlike early methods, this model does not require the low rank condition on the model parameter matrix. We prove that when the side features span the latent feature space of the matrix to be recovered, the number of observed entries needed for an exact recovery is O(log N) where N is the size of the matrix. If the side features are corrupted latent features of the matrix with a small perturbation, our method can achieve an ε-recovery with O(log N) sample complexity. If side information is useless, our method maintains a O(N^3/2) sampling rate similar to classic methods. An efficient linearized Lagrangian algorithm is developed with a convergence guarantee. Empirical results show that our approach outperforms three state-of-the-art methods both in simulations and on real world datasets.

1 Introduction

Matrix completion has been a basis of many machine learning approaches for computer vision [6], recommender systems [21, 24], signal processing [19, 27], and among many others. Classically, low-rank matrix completion methods are based on matrix decomposition techniques which require only the partially observed data in the matrix [15, 3, 14] by solving the following problem

$${\min }_{\mathbf{E}}{\Vert \mathbf{E}\Vert }_{\ast },\text{subject}\text{to}{R}_{\mathrm{\Omega }}(\mathbf{E})=R_{\mathrm{\Omega }}(\mathbf{F}),$$ (1)

where F ∈ ℝ^m×n is the partially observed low-rank matrix (with a rank of r) that needs to be recovered, Ω ⊆ {1, ⋯, m} × {1, ⋯, n} be the set of indexes where the corresponding components in F are observed, the mapping R_Ω(M): ℝ^m×n → ℝ^m×n (i, j)-th entry is M_i,j if (i, j) ∈ Ω (or 0 otherwise), and ||E||_* computes the nuclear norm of E. Early theoretical analysis [4, 5, 20] proves that O(Nr log² N) entries are sufficient for an exact recovery if the observed entries are uniformly sampled at random where N = max{n, m}.

Recent studies start to explore side information for matrix completion and factorization [1, 18, 7, 17, 8]. For example, to infer the missing ratings in a user-movie rating matrix, descriptors of the users and movies are often known and may help to build a content-based recommender system. For instance, kids tend to like cartoons, so the age of a user likely interacts with the cartoon feature of a movie. When few ratings are known, this side information could be the main source for completing the matrix. Although based on empirical studies, several works found that side features are helpful [17, 18], those methods are based on non-convex matrix factorization formulations without any theoretical guarantees. Three recent methods have focussed on convex nuclear-norm regularized objectives, which leads to theoretical guarantees on matrix recovery [13, 28, 9, 16]. These methods all construct an inductive model X^T GY so that R_Ω(X^T GY) = R_Ω(F) where the side matrices X and Y consist of side features, respectively, for the row entities (e.g., users) and column entities (e.g., movies) of a (rating) matrix. This inductive model has a parameter matrix G which is either required to be low rank [13] or to have a minimal nuclear norm ||G||_* [28]. Recovering G of a (usually) smaller size is argued to be easier than directly recovering the matrix F. With a very strong assumption on ‘perfect’ side information, i.e., both X and Y are orthonormal matrices and respectively in the latent column and row space of the matrix F, the method in [28] is proved to require much reduced sample complexity O(log N) for an exact recovery of F. Because most side features X and Y are not perfect in practice, a very recent work [9] proposes to use a residual matrix N to handle the noisy side features. This method constructs an inductive model X^T GY + N to approximate F and requires both G and N to be low rank, or have a low nuclear norm. It uses the nuclear norm of the residual to quantify the usefulness of side information, and proves O(log N) sampling rate for an ε-recovery when X and Y span the full latent feature space of F, and o(N) sample complexity when X and Y contain corrupted latent features of F. An ε-recovery is defined as that the expected discrepancy between the predicted matrix and the true matrix is less than an arbitrarily small ε > 0 under a certain probability.

In this paper, we propose a new method for matrix recovery by constructing a sparse interactive model X^T GY to approximate F where G can be sparse but does not need to be low rank. The (i, j)-th element of G determines the role of the interaction between the i-th feature of users and the j-th feature of products. The low-rank property of F is commonly assumed to characterize the observation that similar users tend to rate similar products similarly [4]. When using an inductive approximation F = X^T GY, rank(F) ≤ rank(G), so a low-rank requirement on G can be a sufficient condition on the low-rank condition of F. Previous relevant methods [13, 28, 9] all impose the low-rank condition on G, which is however not a necessary condition for F to be low rank (only becomes a necessary condition when X and Y are full rank). Given general side matrices $\mathbf{X}\in {\mathrm{ℝ}}^{d_1\times m}$ and $\mathbf{Y}\in {\mathrm{ℝ}}^{d_2\times n}$ where the numbers of features d₁, d₂ ≪ N, limiting the interactive model of $\mathbf{G}\in {\mathrm{ℝ}}^{d_1\times d_2}$ to be low rank can be an over-restrictive constraint. In our model, we use a low-rank matrix E to directly approximate F and then estimate E from the interactive model of X and Y with a sparse regularizer on G. We show empirically that a low-rank F can be recovered from a corresponding full (or high) rank G. Our contributions are summarized as follows:

1. We propose a new formulation that estimates both E and G by imposing a nuclear-norm constraint on E but a general regularizer on G, e.g., the sparse regularizer ||G||₁. The proposed model has recovery guarantees depending on the quality of the side features: (1) when X and Y are full row rank and span the entire latent feature space of F (but are not required to satisfy the much stronger condition of being orthonormal as in [28]), O(log N) observations are still sufficient for our method to achieve an exact recovery of F. (2) When the side matrices are not full rank and corrupted from the original latent features of F, i.e., X and Y do not contain enough basis to exactly recover F, O(log N) observed entries can be sufficient for an ε-recovery.

2. A new linearized alternating direction method of multipliers (LADMM) is developed to efficiently solve the proposed formulation. Existing methods that use side information are solved by standard block-wise coordinate descent algorithms which have convergence guarantee to a global solution only when each block-wise subproblem has a unique solution [26]. Our LADMM has stronger convergence property [29] and benefits from the linear convergence rate of ADMM [11,23].

3. Prior methods focus on the recovery of F, and little light has been shed to understand whether the interactive model of G can be retrieved. Because of the explicit use of E and G, our method aims to directly recover both. The unique G in the case of exact recovery of F can be attained by our algorithm. When G is not unique in the ε-recovery case, our algorithm converges to a point in the optimal solution set.

2 The Proposed Interactive Model

To utilize the side information in X and Y to complete F, we consider to build a predictive model from the observed components that predicts the missing ones. One can simply build a linear model: f = x^Tu + y^Tv + g, where x and y are the feature vectors respectively for a user and a product, and u, v and g are model parameters. In real life applications, interactive terms between the features in X and Y can be very important. For example, male users tend to rate science fiction and action movies higher than female, which can be informative when predicting their ratings. Therefore, a linear model considering no interactive terms can be oversimple and have low predictive power for missing entries. We hence add interactive terms by introducing an interaction matrix ${\mathbf{H}}^{d_1\times d_2}$ into the predictive model, which can be written as: f = x^THy + x^Tu + y^Tv + g. By defining $\overline{\mathbf{x}}={[{\mathbf{x}}^{T}1]}^T,\overline{\mathbf{y}}={[{\mathbf{y}}^{T}1]}^T$ and ${\mathbf{G}}^{(a=d_1+1)\times (b=d_2+1)}=\left(\begin{array}{cc}\mathbf{H}& \mathbf{u}\\ {\mathbf{v}}^T& g\end{array}\right)$ the above model can be simplified to: $f={\overline{\mathbf{x}}}^T\mathbf{G}\overline{\mathbf{y}}$. The following optimization problem can be solved to obtain the model parameter G.

$$\underset{\mathbf{G},\mathbf{E}}{\min }g(\mathbf{G})+{\mathrm{\lambda }}_E{\Vert \mathbf{E}\Vert }_{\ast },\text{subject}\text{to}{\overline{\mathbf{X}}}^T\mathbf{G}\overline{\mathbf{Y}}=\mathbf{E},R_{\mathrm{\Omega }}(\mathbf{E})=R_{\mathrm{\Omega }}(\mathbf{F}),$$

where E is a completed version of F, ${\overline{\mathbf{X}}}^{a\times m}$ and ${\overline{\mathbf{Y}}}^{b\times \mathrm{n}}$ are two matrices that are created by augmenting one row of all ones to X and to Y, respectively, and g(G) and ||E||_* are used to incorporate the (sparsity) prior of G and low rank prior of E. Because the side information data can be noisy and not all the features and their interactions are helpful to the prediction of F, a sparse G is often expected. Our implementation has used g(G) = ||G||₁. It is natural to impose low rank requirement on E because it is a completed version of a low rank matrix F. The tuning parameter λ_E is used to balance the two priors in the objective.

Without loss of generality and for convenience of notation, we simply use X and Y to denote the augmented matrices. Denote the Frobenius norm of a matrix by ||·||_F. To account for Gaussian noise, we relax the equality constraint X^T GY = E and replace it by minimizing their squared residual: ${\Vert {\mathbf{X}}^T\mathbf{GY}-\mathbf{E}\Vert }_F^2$ and solve the following convex optimization problem to obtain G and E:

$$\underset{\mathbf{G},\mathbf{E}}{\min }\frac{1}{2}{\Vert {\mathbf{X}}^T\mathbf{GY}-\mathbf{E}\Vert }_F^2+{\mathrm{\lambda }}_{G}g(\mathbf{G})+{\mathrm{\lambda }}_E{\Vert \mathbf{E}\Vert }_{\ast },\text{subject}\text{to}{R}_{\mathrm{\Omega }}(\mathbf{E})=R_{\mathrm{\Omega }}(\mathbf{F}).$$ (2)

where λ_G is another tuning parameter that together with λ_E balances the three terms in the objective. Especially, the regularizer g(·) in our theoretical analysis can take any general matrix norm that satisfies ||M||_* ≤ Cg(M)), ∀M, for a constant C, so for instance g(·) can be ||G||₁, or ||G||_F, or ||G||₂. Throughout this paper, the matrices X (and Y) refer to, i.e., either the original ${\mathbf{X}}^{d_1\times m}$ (and ${\mathbf{Y}}^{d_2\times n}$) or the augmented ${\overline{\mathbf{X}}}^{a\times m}$ (and ${\overline{\mathbf{Y}}}^{b\times \mathrm{n}}$) depending on the user-specified model.

Our formulation (2) differs from existing methods that make use of side information for matrix completion in several ways. Existing methods [28, 13, 9] solve the problem by finding $\widehat{\mathbf{H}}$ that minimizes ||H||_* subject to R_Ω(X^T HY) = R_Ω(F), but we expand it to include the linear term within the interactive model. The proposed model adds the flexibility to consider both linear and quadratically interactive terms, and allows the algorithm to determine the terms that should be used in the model by enforcing the sparsity in H (or G). Because E = X^T GY, the rank of G bounds that of E from above. The existing methods all control the rank of G (e.g. by minimizing ||G||_*) to incorporate the prior of low rank E (and thus low rank F) in their formulations. However, when the rank of G is not properly chosen during the tuning of hyperparameters, it may not even be a sufficient condition to ensure low rank E (if rank(E) ≪ the pre-specified rank(G)). It is easy to see that besides G a low-rank X or Y can lead to a low-rank E as well. Enforcing a low-rank condition on H or G may limit the search space of the interactive model and thus impair the prediction performance on missing matrix entries, which are demonstrated in our empirical results. Moreover, one can observe that when λ_G is sufficiently large, Eq.(2) is reduced to the standard matrix completion problem (1) without side information because G may be degenerated into a zero matrix, so our formulation is applicable when no access to useful side information.

3 Recovery Analysis

Let E₀ and G₀ be the two matrices such that R_Ω(F) = R_Ω(E₀) and E₀ = X^T G₀Y. In this section, we give our theoretical results on the sample complexity for achieving an exact recovery of E₀ and G₀ when X and Y are both full row rank (i.e., rank(X) = a and rank(Y) = b), and an ε-recovery of E₀ when the two side matrices are corrupted and less informative. The proofs of all theorems are given in supplementary materials.

3.1 Sample Complexity for Exact Recovery

Before presenting our results, we give a few definitions. Let $\mathbf{F}=\mathbf{U}\mathbf{\sum }{\mathbf{V}}^T,{\mathbf{X}}^T={\mathbf{U}}_{\mathbf{X}}{\mathbf{\sum }}_{\mathbf{X}}{\mathbf{V}}_{\mathbf{X}}^T$ and ${\mathbf{Y}}^T={\mathbf{U}}_{\mathbf{Y}}{\mathbf{\sum }}_{\mathbf{Y}}{\mathbf{V}}_{\mathbf{Y}}^T$ be the singular value decomposition of F, X^T and Y^T, respectively, where all Σ matrices are full rank, meaning that singular vectors corresponding to the singular value 0 are not included in the respective U and V matrices. Let

$$P_{\mathbf{U}}={\mathbf{UU}}^T\mathrm{\in }{\mathrm{ℝ}}^{m\times m},P_{\mathbf{V}}={\mathbf{VV}}^T\mathrm{\in }{\mathrm{ℝ}}^{n\times n},P_{\mathbf{X}}={\mathbf{U}}_{\mathbf{X}}{\mathbf{U}}_{\mathbf{X}}^T={\mathbf{X}}^T{\mathbf{V}}_{\mathbf{X}}{\mathbf{\sum }}_{\mathbf{X}}^{-2}{\mathbf{V}}_{\mathbf{X}}^T\mathbf{X}\mathrm{\in }{\mathrm{ℝ}}^{m\times m},P_{\mathbf{Y}}={\mathbf{U}}_{\mathbf{Y}}{\mathbf{U}}_{\mathbf{Y}}^T={\mathbf{Y}}^T{\mathbf{V}}_{\mathbf{Y}}{\mathbf{\sum }}_{\mathbf{Y}}^{-2}{\mathbf{V}}_{\mathbf{Y}}^T\mathbf{Y}\mathrm{\in }{\mathrm{ℝ}}^{n\times n},$$

where P_U, P_V, P_X and P_Y project a vector onto the subspaces spanned, respectively, by the columns in U, V and rows in X, and Y. For any matrix M^m×n that satisfies M = P_XMP_Y, we define two linear operators: P_T : ℝ^m×n → ℝ^m×n and P_T^⊥ : ℝ^m×n → ℝ^m×n as follows:

$$P_T(\mathbf{M})=P_{\mathbf{U}}\mathbf{M}{P}_{\mathbf{Y}}+P_{\mathbf{X}}\mathbf{M}{P}_{\mathbf{V}}-P_{\mathbf{U}}\mathbf{M}{P}_{\mathbf{V}}$$

$$P_{T^{\perp }}(\mathbf{M})=(P_{\mathbf{X}}-P_{\mathbf{U}})\mathbf{M}(P_{\mathbf{Y}}-P_{\mathbf{V}})=P_{{\mathbf{X}}^{\perp }}\mathbf{M}{P}_{{\mathbf{Y}}^{\perp }}.$$

Let μo and μ₁ be the two coherence measures of F and be defined as follows as discussed in [4, 16]:

$${\mu }_0=\max (\frac{m}{r}\underset{1\le i\le m}{\max }{\Vert P_{\mathbf{U}}{\mathbf{e}}_i\Vert }_2,\frac{n}{r}\underset{1\le j\le n}{\max }{\Vert P_{\mathbf{V}}{\mathbf{e}}_j\Vert }_2),{\mu }_1=\underset{i,j}{\max }\frac{\mathit{mn}}{r}{({[\mathbf{U}{\mathbf{V}}^T]}_{i,j})}^2,$$

where e_i is the unit vector with the ith entry equal to 1. Let μ_XY be the coherence measurement between X and Y and be defined as:

$${\mu }_{\mathbf{XY}}=\max (\underset{1\le i\le m}{\max }\frac{m{\Vert {\mathbf{x}}_i\Vert }_2^2}{a},\underset{1\le j\le n}{\max }\frac{n{\Vert {\mathbf{y}}_i\Vert }_2^2}{b}).$$

With the above definitions, we show in the following theorem that when X and Y are both full row rank, (G₀, E₀) is the unique solution to Eq.(2) with high probability as long as there are O(r log N) observed components in F. In other words, with a sampling rate of O(r log N), our method can fully recover both E₀ and G₀ with a high probability when X and Y are full row rank.

Theorem 1

Let μ = max(μ₀, μ_XY), $\sigma =\max ({\Vert {\mathbf{\sum }}_{\mathbf{X}}^{-1}\Vert }_{\ast },{\Vert {\mathbf{\sum }}_{\mathbf{Y}}^{-1}\Vert }_{\ast })$, N = max(m, n), $q_0=\frac{1}{2}(1+\text{log}a-\text{log}r),T_0=\frac{128p}{3}\sigma \mu \max ({\mu }_1,\mu )r(a+b)\text{log}N$ and $T_1=\frac{8p}{3}{\sigma }^2{\mu }^2(\mathit{ab}+r^2)\text{log}N$, where p is a constant. Assume T₁ ≥ q₀T₀, X and Y are both full row rank. For any p > 1, with a probability at least 1 − 4(q₀ + 1)N^−p+1 − 2q₀N^−p+2, (G₀, E₀) is the unique optimizer to Problem (2) with necessary sampling rate as few as O(r log N). More precisely, the sampling size |Ω| should satisfy that $|\Omega |\ge \frac{64p}{3}\sigma \mu \max ({\mu }_1,\mu )(1+\text{log}a-\text{log}r)r(a+b)\text{log}N$.

When r ≪ N and r = O(1), the sampling rate for the exact recovery of both E₀ and G₀ reduces to O(log N). A similar sampling rate for a full recovery of E₀ has been developed in [28] where both X and Y, however, need to be orthonormal matrices in their derivation. In Theorem 1, because σ is mainly determined by the smallest singular values of the side information matrices, and sampling rate increases when σ increases, it suggests that side information matrices of lower rank would require more observed F entries for a full recovery of F. An advanced model without the orthonormal assumption has been given in [9], but exact recovery is not discussed. In our case, the two matrices are only required to be full row rank. Moreover, the theoretical or empirical results in our work give the first careful investigation on the recovery of both G₀ and E₀.

3.2 Sample Complexity for ε-Recovery

The condition for full-rank side information matrices may not be satisfied in some cases to fully recover E₀ (or F). We analyze the error bound of our model and prove a reduced sample complexity in comparison with standard matrix completion methods for an ε-recovery when the side information matrices are not full row rank or their rank is difficult to attain.

Theorem 2

Denote ||E||_* ≤ α, ||G||₁ ≤ γ, ||X^T GY − E||_F ≤ ϕ and the perfect side feature matrices (containing latent features of F) are corrupted with ∆X and ∆Y where ||∆X||_F ≤ s₁, ||∆Y||_F ≤ s₂ and S = max(s₁,s₂). To ε-recover F that the expected loss $\mathbb{E}[l(f,\mathbf{F})]<\epsilon$ for a given arbitrarily small $\epsilon >0,O(\min (({\mathrm{\gamma }}^2+{\varphi }^2)\text{log}N,S^2\alpha \sqrt{N})/{\epsilon }^2)$ observations are sufficient for our model when corrupted factors of side information are bounded.

Theorem 2 can be inferred from the fact that the trace norm of E and the ℓ₁-norm of G affect sample complexity of our model. It meets the intuition that higher rank matrix ought to require more observations to recover. Besides, for the discovery of G, a sparse interactive matrix can lead to the decrease of the sample complexity, which implies that the side information, even though when it is not perfect, could be informative enough such that the original matrix can be compressed by sparse coding via the estimated interaction between the features of row and column entities of the matrix. Our empirical evaluations have confirmed the utility of even imperfect side features. When the rank of the original data matrix r = O(1) (r ≪ N), and correspondingly α = O(1), Theorem 2 points out that only O(log N) sampling rate is required for an ε-recovery. The classic matrix completion analysis without side information shows that under certain conditions, one can achieve O(N poly log N) sample complexity for both perfect recovery [4] and ε-recovery [25], which is higher than our complexity. However, the condition for these existing bounds is that the observed entries follow a certain distribution. Recent studies [22] found that if no specific distribution is pre-assumed for observed entries, O(N^3/2) sampling rate is sufficient for an ε-recovery. Compared to those results, our analysis does not require any assumption on the distribution of observed entries. When X and Y contain insufficient interaction information about F and ||E||_* = O(N), the sample complexity of our method increases to O(N³/²) in the worst case, which means that our model maintains the same complexity as the classic methods.

4 Adaptive LADMM Algorithm

In this section, we develop an adaptive LADMM algorithm [29] to solve problem (2). First, we show that the ADMM is applicable in our problem and we then derive LADMM steps. A convergence proof is established to guarantee the performance of our algorithm.

Because it requires separable blocks of variables in order to use ADMM, we first define C = E − X^T GY and use it in Eq.(2). Then the augmented Lagrangian function of (2) is given by

$$ℒ(\mathbf{E},\mathbf{G},\mathbf{C},{\mathbf{M}}_1,{\mathbf{M}}_2,\beta )=\frac{1}{2}{\Vert \mathbf{C}\Vert }_F^2+{\mathrm{\lambda }}_E{\Vert \mathbf{E}\Vert }_{\ast }+{\mathrm{\lambda }}_G{\Vert \mathbf{G}\Vert }_1+\langle {\mathbf{M}}_1,R_{\mathrm{\Omega }}(\mathbf{E}-\mathbf{F})\rangle ++\langle {\mathbf{M}}_2,\mathbf{E}-{\mathbf{X}}^T\mathbf{GY}-\mathbf{C}\rangle +\frac{\beta }{2}{\Vert R_{\mathrm{\Omega }}(\mathbf{E}-\mathbf{F})\Vert }_F^2+\frac{\beta }{2}{\Vert \mathbf{E}-{\mathbf{X}}^T\mathbf{GY}-\mathbf{C}\Vert }_F^2$$ (3)

where M₁, M₂ ∈ ℝ^m×n are Lagrange multipliers and β > 0 is the penalty parameter. Given C^k, G^k, E^k, ${\mathbf{M}}_1^k$ and ${\mathbf{M}}_2^K$ at iteration k, each group of the variables yields their respective subproblems:

$${\mathbf{C}}^{k+1}=\text{arg}\underset{\mathbf{C}}{\min }ℒ({\mathbf{E}}^k,{\mathbf{G}}^k,{\mathbf{M}}_2^k,\mathbf{C},{\beta }_k),{\mathbf{G}}^{k+1}=\text{arg}\underset{\mathbf{G}}{\min }ℒ({\mathbf{E}}^k,\mathbf{G},{\mathbf{M}}_2^k,{\mathbf{C}}^{k+1},{\beta }_k),{\mathbf{E}}^{k+1}=\text{arg}\underset{\mathbf{E}}{\min }ℒ(\mathbf{E},{\mathbf{G}}^{k+1},{\mathbf{M}}_1^k,{\mathbf{M}}_2^k,{\mathbf{C}}^{k+1},{\beta }_k),$$ (4)

After solving these subproblems, we update the multipliers M₁ and M₂ as follows;

$${\mathbf{M}}_1^{k+1}={\mathbf{M}}_1^k+{\beta }_k(R_{\mathrm{\Omega }}({\mathbf{E}}^{k+1}-\mathbf{F})),{\mathbf{M}}_2^{K+1}={\mathbf{M}}_2^K+{\beta }_k({\mathbf{E}}^{k+1}-{\mathbf{X}}^T{\mathbf{G}}^{k+1}\mathbf{Y}-{\mathbf{C}}^{k+1}).$$ (5)

We focus on demonstrating the iterative steps of the adaptive LADMM. Given C^k, G^k, E^k, ${\mathbf{M}}_1^k$ and ${\mathbf{M}}_2^K$, Algorithm 1 describes how to obtain the next iterate (C, E, G, M₁, M₂). A closed-form solution has been derived for each subproblem in the supplementary material.

Algorithm 1.

The adaptive LADMM algorithm to solve C^k, G^k, E^k, k = 1, …, K

Input: X, Y and RΩ(F) with parameters λG, λE, τA, τB, ρ and βmax.

Output: C, G, E; Initialize E0, G0, ${\mathbf{M}}_1^0$, ${\mathbf{M}}_2^0$. Compute A = YT ⊗ XT. k = 0, repeat; ${\mathbf{C}}^{k+1}=\frac{{\beta }_k}{{\beta }_k+1}({\mathbf{E}}^k-{\mathbf{X}}^T{\mathbf{G}}^k\mathbf{Y}+{\mathbf{M}}_2^k/{\beta }_k)$; ${\mathbf{G}}^{k+1}=\mathit{reshape}(\text{max}(|{\mathbf{g}}^k-f_1^k/{\tau }_A|-\frac{{\mathrm{\lambda }}_G}{{\tau }_A{\beta }_k},0)\odot \mathit{sgn}({\mathbf{g}}^k-f_1^k/{\tau }_A))$ where $f_1^k={\mathbf{A}}^T(\mathbf{A}{\mathbf{g}}^k+{\mathbf{c}}^k-{\mathbf{b}}_1^k)={\mathbf{A}}^T(\mathbf{A}{\mathbf{g}}^k+{\mathbf{c}}^k-{\mathbf{e}}^k-{\mathbf{m}}_2^k/{\beta }_k)$ and e = vec(E), g = vec(G), m = vec(M), c = vec(C). ${\mathbf{E}}^{k+1}=\mathit{SVT}({\mathbf{E}}^k-(f_2^k+f_3^k)/(2{\tau }_B),{\mathrm{\lambda }}_E/2({\beta }_k{\tau }_B))$ where $f_2^k=R_{\mathrm{\Omega }}({\mathbf{E}}^k-\mathbf{F}+{\mathbf{M}}_1^k/{\beta }_k);f_3^k={\mathbf{E}}^k-{\mathbf{X}}^T{\mathbf{G}}^{k+1}\mathbf{Y}-{\mathbf{C}}^k+M_2^k/{\beta }_k.$ ${\mathbf{M}}_1^{k+1}={\mathbf{M}}_1^k+{\beta }_k(R_{\Omega }({\mathbf{E}}^{k+1}-\mathbf{F})).$ ${\mathbf{M}}_2^{K+1}={\mathbf{M}}_2^k+{\beta }_k({\mathbf{E}}^{k+1}-{\mathbf{X}}^T{\mathbf{G}}^{k+1}\mathbf{Y}-{\mathbf{C}}^{k+1}).$ βk+1 = min (βmax, ρβk). k = k + 1 until convergence; Return C, G, E;

The adaptive parameter in Algorithm 1 is ρ > 1, and β_max controls the upper bound of {β_k}. The operator reshape(g) converts a vector g ∈ ℝ^ab into a matrix G ∈ ℝ^a×b, which is the inverse operator of vec(G). The operator SVT(E, t) is the singular value thresholding process defined in [3] for soft-thresholding the singular values of an arbitrary matrix E by a threshold t. The matrix A = Y^T ⊗ X^T where ⊗ indicates the Kronecker product. In the initialization step, ${\mathbf{M}}_1^0$, ${\mathbf{M}}_2^0$ are randomly drawn from the standard Gaussian distribution; we initialize E₀ and G₀ by the iterative soft-thresholding algorithm [2] and SVT operator respectively.

The adaptive LADMM can effectively solve the proposed optimization problem in several aspects. First, the convergence of the commonly-used block-wise coordinate descent (BCD) method, sometimes referred to as alternating minimization methods, requires typically that the optimization problem be strictly convex (or quasiconvex but hemivariate). The strongest result for BCD so far is established in [26] which requires the alternating subproblems to be optimized in each iteration to its unique optimal solution. This requirement is often restrictive in practice. Our convex (but not strictly convex) problem can be solved by the adaptive LADMM with the global convergence guarantee which is characterized in Theorem 3. Second, two of the subproblems are non-smooth due to the ℓ₁-norm or the nuclear norm, so it can be difficult to obtain a closed-form formula to efficiently compute a solution by standard optimization tools; however, adaptive LADMM utilizes the linearization technique which leads to a closed-form solution for each linearized subproblem, and significantly enhances the efficiency of the iterative process. Third, adaptive LADMM can be practically parallelizable by a similar scheme to that of ADMM. It is also noted that the convergence rate of LADMM [11] and parallel LADMM is O(1/k) [23] whereas the BCD method still lacks of clear theoretical results of its convergence rate.

Theorem3

Define the operators $\mathcal{A}$ and $ℬ$ as $\mathcal{A}(\mathbf{G})=\left(\genfrac{}{}{0ex}{}{0}{-{\mathbf{X}}^T\mathbf{GY}}\right),ℬ(\mathbf{E})=\left(\genfrac{}{}{0ex}{}{R_{\mathrm{\Omega }}(\mathbf{E})}{\mathbf{E}}\right)$, and let $\mathbf{M}=\left(\genfrac{}{}{0ex}{}{{\mathbf{M}}_1}{{\mathbf{M}}_2}\right)$. If β_k is non-decreasing and upper-bounded, ${\tau }_A>{\Vert \mathcal{A}\Vert }^2$, and ${\tau }_B>{\Vert ℬ\Vert }^2$, then the sequence {(C^k, G^k, E^k, M^k)} generated by the adaptive LADMM Algorithm 1 converges to a global minimizer of Eq. (2).

5 Experimental Results

We validated our method in both simulations and the analysis of two real world datasets: MovieLens (movie rating) and NCI-DREAM (drug discovery) datasets. Three most recent matrix completion methods that also utilized side information, MAXIDE[28], IMC[13] and DirtyIMC[9], were compared against our method. The design of our experiments focused on demonstrating the effectiveness of our method in practice. The performance of all methods was measured by the relative mean squared error (RMSE) calculated on missing entries: ${\Vert R_{\cancel{\mathrm{\Omega }}}({\mathbf{X}}^T\mathbf{GY}-\mathbf{F})\Vert }_2^2/{\Vert R_{\cancel{\mathrm{\Omega }}}(\mathbf{F})\Vert }_2^2$. For both synthetic and real-world datasets, we randomly set q percent of the components in each observed matrix F to be missing. The hyperparameters λ’s and the rank of G (required by IMC and Dirty IMC) were tuned via the same cross validation process: we randomly picked 10% of the given entries to form a validation set. Then models were obtained by applying each method to the remaining entries with a specific choice of λ from 10⁻³,10⁻²,…, 10⁴. The average validation RMSE was examined by repeating the above procedure six times. The hyperparameter values that gave the best average validation RMSE were chosen for each method. For IMC and DirtyIMC, the best rank of G was chosen from = 1 to 15 within each data split. For each choice of q, we repeated the above entire procedure six times and reported the average RMSE on the missing entries.

5.1 Synthetic Datasets

We created two different simulation tests with and without full row rank X and Y. For all the synthetic datasets, we first randomly created X and Y. In order to make our simulations reminiscent real situations where distributions of side features can be heterogeneous, data for each feature in both X and Y were generated according to a distribution that was randomly selected from Gaussian, Poisson and Gamma distributions. We created the sparse G matrices as follows. The location of the non-zero entries of G were randomly picked but their values were generated by multiplying a value drawn from $\mathcal{N}(0,100)$, which we repeated several times to chose the matrices that showed full or high rank. We then generated F with F = X^T GY + N where N represents noise and each component N_i,j was drawn from $\mathcal{N}(0,1)$. For each simulated F, we ran all methods with q ∈ [10% – 80%] with an increase step of 10%.

We compared the different methods in three settings, which were labeled as synthetic experiment I, II and III in our results. In the first setting, the dimension of X and Y was set to 15 × 50 and 20 × 140 and all features in these two matrices were randomly generated to make them full row rank. Both the last two settings corresponded to the second test where X and Y were not full row rank. The dimension of X and Y was set to 16 × 50, 21 × 140 and 20 × 50, 25 × 140, respectively, for these two settings where the first 15 features in X and 20 features in Y were randomly created, but the remaining features were generated by arbitrarily linear combinations of the randomly created features. For all three settings, we used 10 synthetic datasets and reported mean and standard deviation of RMSE on missing values as shown in Figure 1.

Figure 1.

The Comparison of RMSE for Experiments I, II, and III.

Our approach outperformed all other compared methods significantly in almost all these settings. When the missing rate q increased, the RMSE of our method grew much slower than other methods. We studied the rank of the recovered G and E in the first setting. For all methods, the corresponding G and E that gave the best performance were examined. The ranks of G and E from our method, MAXIDE, IMC, DirtyIMC were 15, 8, 1, 1 and 15, 15, 1, 2, respectively. These results suggested that incorporating the strong prior of low rank G might hurt the recovery performance. The retrieved model matrices G of all compared methods (when using q =10% of missing entries in one of the 10 synthetic datasets) together with the true G are plotted in Figure 2. Only our method was able to recover the true G and all the other methods merely found approximations.

Figure 2.

The heatmap of the true G and recovered G matrices in Synthetic Experiment I.

5.2 Real-world Datasets

We used two relatively large datasets that we could find as suitable for our empirical evaluation. Note that early methods employing side information were often tested on datasets with either X or Y but not both although some of them might be larger than the two datasets we used.

5.2.1. MovieLens

This dataset was downloaded from [12] and contained 100,000 user ratings (integers from 1 to 5) from 943 users on 1682 movies. There were 20 movie features such as genre and release date, as well as 24 user features describing users’ demographic information such as age and gender. We compared all methods with four different q values: 20–50%. The RMSE values of each method are shown in Table 1, which shows that our approach significantly outperformed other methods, especially when q was large. Figure 3 shows the constructed G matrix that shows some interesting observations. For instance, male users tend to rate action, science fiction, thriller and war movies high but low for children’ movies, exhibiting some common intuitions.

Table 1.

The Comparison of RMSE values of different methods on real-world datasets.

	MovieLens Data				NCI-Dream Challenge
Methods	20%	30%	40%	50%	20%	30%	40%	50%
Our approach	0.276 (± 0.001)	0.279 (± 0.002)	0.284 (± 0.001)	0.292 (± 0.001)	0.181 (± 0.069)	0.139 (± 0.010)	0.145 (± 0.018)	0.190 (± 0.031)
MAXIDE	0.424 (±0.016)	0.425 (±0.013)	0.419 (±0.008)	0.421 (±0.013)	0.268 (±0.036)	0.240 (±0.007)	0.255 (±0.016)	0.288 (±0.022)
IMC	0.935 (±0.001)	0.943 (±0.001)	0.945 (±0.001)	0.959 (±0.001)	0.437 (±0.031)	0.489 (±0.003)	0.557 (±0.013)	0.637 (±0.011)
DirtyIMC	0.705 (±0.001)	0.738 (±0.001)	0.775 (±0.001)	0.814 (±0.001)	0.432 (±0.033)	0.475 (±0.008)	0.551 (±0.018)	0.632 (±0.011)

Figure 3.

HeatMap of G for MovieLens

5.2.2 NCI-DREAM Challenge

The data on the reactions of 46 breast cancer cell lines to 26 drugs and the expression data of 18633 genes for all the cell lines were provided by NCI-DREAM Challenge [10]. For each drug, we had 14 features that describes their chemical and physical properties such as molecular weight, XLogP3 and hydrogen bond donor count, and were downloaded from National Center for Biotechnology Information (http://pubchem.ncbi.nlm.nih.gov/). For the cell line features, we ran principle component analysis (PCA) and used the top 45 principal components that accounted for more than 99.99% of the total data variance. We compared the four different methods with four different q values: 20–50%. The RMSE values of all methods are provided in Table 1 where our method again shows the best performance. We examined the ranks of both G and E obtained by all the methods. They were 15, 15, 1, 1 for G and 2, 15, 1, 2 for E, respectively, for our approach, MAXIDE, IMC and DirtyIMC in sequence. This demonstrates that a low rank E but a high rank G give the best performance on this dataset. In other words, requiring a low rank G may hurt the performance of recovering a low rank E.

The constructed G by our method is plotted in Figure 4, where columns represent cell line features (i.e., principle components) and rows represent drug features. Please refer to the supplementary material for the names of these features. According to this figure, drug features: XlogP (F2), hydrogen bond donor (HBD) (F3), Hydrogen bond acceptor (HBA) (F4) and Rotatable Bond number (F5) all played important roles in drug sensitivity. This result aligns well with biological knowledge, as all these four features are very important descriptors for cellular entry and retention.

Figure 4.

HeatMap of sign(G) log(|G|) for NCI-DREAM for a better illustration

6 Conclusion

In this paper, we have proposed a novel sparse inductive model that utilizes side features describing the row and column entities of a partially observed matrix to predict its missing entries. This method models the linear predictive power of side features as well as interaction between the features of row and column entities. Theoretical analysis shows that this model has advantages of reduced sample complexity over classical matrix completion methods, requiring only O(log N) observed entries to achieve a perfect recovery of the original matrix when the side features reflect the true latent feature space of the matrix. When the side features are less informative, our model requires O(log N) observations for an ε-recovery of the matrix. Unlike early methods that use a BCD algorithm, we have developed a LADMM algorithm to optimize the proposed formulation. Given the optimization problem is convex, this algorithm can converge to a global solution. Computational results demonstrate the superior performance of this method over three recent methods. Future work includes the examination of other types and quality of side information and the understanding of whether our method will benefit a variety of relevant problems, such as multi-label learning, and semi-supervised clustering etc.