Turbo-SMT: Parallel Coupled Sparse Matrix-Tensor Factorizations and Applications

Abstract

How can we correlate the neural activity in the human brain as it responds to typed words, with properties of these terms (like ’edible’, ’fits in hand’)? In short, we want to find latent variables, that jointly explain both the brain activity, as well as the behavioral responses. This is one of many settings of the Coupled Matrix-Tensor Factorization (CMTF) problem.

Can we enhance any CMTF solver, so that it can operate on potentially very large datasets that may not fit in main memory? We introduce Turbo-SMT, a meta-method capable of doing exactly that: it boosts the performance of any CMTF algorithm, produces sparse and interpretable solutions, and parallelizes any CMTF algorithm, producing sparse and interpretable solutions (up to 65 fold). Additionally, we improve upon ALS, the work-horse algorithm for CMTF, with respect to efficiency and robustness to missing values.

We apply Turbo-SMT to BrainQ, a dataset consisting of a (nouns, brain voxels, human subjects) tensor and a (nouns, properties) matrix, with coupling along the nouns dimension. Turbo-SMT is able to find meaningful latent variables, as well as to predict brain activity with competitive accuracy. Finally, we demonstrate the generality of Turbo-SMT, by applying it on a Facebook dataset (users, ’friends’, wall-postings); there, Turbo-SMT spots spammer-like anomalies.

1 Introduction

How is knowledge mapped and stored in the human brain? How is it expressed by people answering simple questions about specific words? If we have data from both worlds, are we able to combine them and jointly analyze them? In a very different scenario, suppose we have the social network graph of an online social network, and we also have additional information about how and when users interacted with each other. What is a comprehensive way to combine those two pieces of data? Both, seemingly different, problems may be viewed as instances of what is called Coupled Matrix-Tensor Factorization (CMTF), where a data tensor and matrices that hold additional information are jointly decomposed into a set of low-rank factors.

The CMTF framework is by no means a new concept, with one of its earliest applications dating back to 2007 where the authors of [11] define a clustering algorithm on data that form Matrix/Tensor couples and apply it to movie recommendation and newsgroup articles settings. Subsequently, there has been a lot of work, both on the algorithmic side [52, 4] and on the application side, ranging from metabolomics [7] to social network analysis [33], where usually the data were of small to medium scale. However, we envision applications of the framework where the data scale renders the current state of the art a bottleneck to the analysis. Such an example of application is the one of Neurosemantics, where people are shown stimuli (which can range from single words to entire books) and their brain activity is recorded. These measurements can easily span multiple gigabytes of data, and their size and grows as the complexity of the stimulus increases. To that end, we need fast and parallelizable methods for CMTF that are able to handle data that may not fit in the main memory of a machine. In this work, we introduce Turbo-SMT, a parallel, scalable, and sparsity promoting CMTF meta-algorithm. Our main contributions are the following:

• Parallel & triple-sparse algorithm: We provide an approximate, novel, scalable, and triple-sparse (see Sec. 3) meta-method, Turbo-SMT, that is able to parallelize and scale-up any CMTF core algorithm. Additionally, we improve upon the ALS core CMTF algorithm, making it faster and able to handle missing values (see Sec. 3 for details on the algorithm, and Sec. 7 for experimental evaluation).

• Effectiveness & Knowledge Discovery: As a first step of the Neurosemantics application, we analyze BrainQ, a brain scan dataset which is coupled to a semantic matrix. Furthermore, we analyze a social interaction dataset, coupled with friendship information between the users. (see Sec. 5 for details)

• Reproducibility: Our code is publicly available¹ ; the BrainQ and Facebook datasets we use (see Section 5) are also publicly available.

In the knowledge discovery part, the brain scan part of the dataset consists of fMRI scans originally used in [37], a work that first demonstrated that brain activity can be predictably analyzed into component semantic features. Here, we demonstrate a disciplined way to combine both datasets and carry out a variety of data mining/machine learning tasks, through this joint analysis. A snippet of our analysis is shown in Fig. 1, where we find groups of semantically similar nouns that activate the same brain regions; additionally, Turbo-SMT generally improves the CMTF approximation error, while parallelizing the execution of the baselines, working on smaller pieces of data, potentially enabling processing of datasets that may not fit in memory. Furthermore, we apply Turbo-SMT to a time-evolving social network with side information, discovering anomalies, thereby demonstrating the generality of the CMTF framework and, ultimately, the strength of Turbo-SMT.

Figure 1.

(a): Analyzing the BrainQ dataset for the Neurosemantics application, finding semantically similar nouns that activate the same part of the brain. In this case, the premotor cortex is associated with movements such as holding or picking small items up. (b): The proposed algorithm Turbo-SMT reduces the approximation error, while running in parallel, and operating on much smaller pieces of data at any given point in time.

An earlier version of this work was uploaded on Arxiv.org [44] for quick dissemination thereof and early feedback. The first published version of this work is included in the proceedings of SIAM SDM 2014 [45]. The present manuscript is an invited journal extension of [45] as part of the “best of SDM 2014” papers. The new contributions introduced here are 1) additional and extensive performance evaluation, 2) comparison of the Turbo-SMT against compression based techniques (which, to the best of our knowledge have not been published for the CMTF problem), 3) further improvement of the ALS algorithm for CMTF (which is one of the core solvers that can be used with Turbo-SMT), and 4) application of Turbo-SMT to a social network with side information.

2 Preliminaries

2.1 A note on notation

Table 1 contains symbols and essential notation that is henceforth used throughout the text, both in the present section, as well as in Section 3, where Turbo-SMT is introduced.

Table 1.

Table of symbols

Symbol	Description
CMTF	Coupled Matrix-Tensor Factorization
ALS	Alternating Least Squares
CMTF-OPT	Algorithm introduced in [4]
x, x;, X, X̲	scalar, vector, matrix, tensor (respectively)
A ⊙ B	Khatri-rao product.
A ⊗ B	Kronecker product.
A * B	Hadamard (elementwise) product.
A†	Pseudoinverse of A (see Sec. 2)
‖A‖F	Frobenius norm of A.
a ◦ b ◦ c	(a ◦ b ◦ c) (i, j, k) = a(i)b(j)c(k)
(i) as superscript	Indicates the i-th iteration
${\mathbf{A}}_1^i,{\mathbf{a}}_1^i$	series of matrices or vectors, indexed by i.
X̲(i)	i-th mode unfolding of tensor X̲ (see [27]).
ℐ	Set of indices.
x(ℐ)	Spanning indices ℐ of x.
nucleus	a biased random sample of the data tensor.

2.2 Introduction to Tensors

Matrices record dyadic properties, like “people recommending products”. Tensors are the n-mode generalizations, capturing 3- and higher-way relationships. For example “subject-verb-object” relationships, such as the ones recorded by the Read the Web - NELL project [1] (and have been recently used in this context [26] [41]) naturally lead to a 3-mode tensor. In this work, our working example of a tensor has three modes. The first mode contains a number of nouns; the second mode corresponds to the brain activity, as recorded by an fMRI machine; and the third mode identifies the human subject corresponding to a particular brain activity measurement. A very popular tensor decomposition, shown in Fig. 2 is the PARAFAC decomposition [24], where a tensor is decomposed into a sum of rank-one tensors (each rank-one tensor being an outer product of three vectors). For a thorough overview of all different flavors of tensor decompositions, we refer the interested reader to [31]

Figure 2.

PARAFAC decomposition [24] of a three-way tensor of a brain activity tensor as sum of F outer products (rank-one tensors), reminiscent of the rank-F singular value decomposition of a matrix. Each component corresponds to a latent concept of, e.g. “insects”, “tools” and so on, a set of brain regions that are most active for that particular set of words, as well as groups of persons.

In [41], the authors introduced a scalable and parallelizable tensor decomposition which uses biased mode sampling. In this work, we focus on a more general and expressive framework, that of Coupled Matrix-Tensor Factorizations, extending the idea of [41] for that setting. Besides [41], there is a line of work that employs sampling for tensor decomposition [36, 23], which is tangentially related to the present work, and is discussed in more detail in Section 8.3

2.3 Coupled Matrix-Tensor Factorization

Oftentimes, two tensors, or a matrix and a tensor, may have one mode in common; consider the example that we mentioned earlier, where we have a word by brain activity by human subject tensor, we also have a semantic matrix that provides additional information for the same set of words. In this case, we say that the matrix and the tensor are coupled in the ’word’ mode, and the problem is instance of Coupled Matrix-Tensor Factorization (CMTF) [11, 52, 4] which is an active field of research, aiming to jointly analyze datasets (in the form of tensors and matrices) that share a subset of their modes.

In this work we focus on three mode tensors, however, everything we mention extends directly to higher modes. In the general case, a three mode tensor X̲ may be coupled with at most three matrices Y_i, i = 1 ⋯ 3, in the manner illustrated in Figure 3 for one mode. The optimization function that encodes this decomposition is:

$$\underset{\mathbf{A},\mathbf{B},\mathbf{C},\mathbf{D},\mathbf{E},\mathbf{G}}{\text{min}}{\Vert \underset{¯}{\mathbf{X}}-\sum _k{\mathbf{a}}_k◦{\mathbf{b}}_k◦{\mathbf{c}}_k\Vert }_F^2+{\Vert {\mathbf{Y}}_1-\mathbf{A}{\mathbf{D}}^T\Vert }_F^2+{\Vert {\mathbf{Y}}_2-\mathbf{B}{\mathbf{E}}^T\Vert }_F^2+{\Vert {\mathbf{Y}}_3-\mathbf{C}{\mathbf{G}}^T\Vert }_F^2$$ (2.1)

where a_k is the k-th column of A. The idea behind the coupled matrix-tensor decomposition is that we seek to jointly analyze X̲ and Y_i, decomposing them to latent factors who are coupled in the shared dimension. For instance, the first mode of X̲ shares the same low rank column subspace as Y₁; this is expressed through the latent factor matrix A which jointly provides a basis for that subspace.

Figure 3.

Coupled Matrix - Tensor example: Tensors often share one or more modes (with thick, wavy line): X̲ is the brain activity tensor and Y is the semantic matrix. As the wavy line indicates, these two datasets are coupled in the ’word’ dimension.

2.4 Solving CMTF

A popular algorithm for solving CMTF (as shown in Figure 3) is the so-called Alternating Least Squares (ALS), a block-coordinate descent method [4] (Fig. 5); in the next subsection, we provide a short description of the ALS algorithm, since one of our contributions pertains to speeding up the ALS algorithm itself. Besides ALS, there exist other algorithms for CMTF. For example, [4] uses a first order optimization algorithm for the same objective. Throughout this work, we use both algorithms as core CMTF solvers for Turbo-SMT. The strength of Turbo-SMT, however, is that it can be used as-is with any underlying core CMTF implementation. Hence, Turbo-SMT is CMTF solver independent.

Figure 5.

This is a pattern extracted using Turbo-SMT, which shows what appears to be a spammer on the Facebook dataset: One person, posting to many different walls on a single day.

2.4.1 The Alternating Least Squares Algorithm

The work-horse algorithm for solving PARAFAC (as shown in Figure 2) is Alternating Least Squares (ALS) [50]; the basic idea is that by fixing two of the three factor matrices, we have a least squares problem for the third, and we thus do so iteratively, alternating between the matrices we fix and the one we optimize for, until the algorithm converges, usually when the relative change in the objective function between two iterations is very small.

Solving CMTF using ALS follows the same strategy, only now, we have up to three additional matrices in our objective. For instance, when fixing all matrices but A, the update for A requires to solve the following least squares problem:

$$\underset{\mathbf{A}}{\text{min}}{\Vert {\underset{¯}{\mathbf{X}}}_{(1)}-(\mathbf{B}\odot \mathbf{C}){\mathbf{A}}^T\Vert }_F^2+{\Vert {\mathbf{Y}}_1-\mathbf{D}{\mathbf{A}}^T\Vert }_F^2$$

In order to obtain initial estimates for matrices A, B, C we take the PARAFAC decomposition of X̲. As for matrix D (and similarly for the rest), it suffices to solve a simple Least Squares problem, given the PARAFAC estimate of A, we initialize as D = Y₁ (A^†)^T, where † denotes the Moore-Penrose Pseudoinverse which, given the Singular Value Decomposition of a matrix X = UΣV^T, is computed as X^† = VΣ⁻¹U^T.

In Algorithm 1, we provide a detailed outline of the baseline algorithm, ALS for CMTF, also shown in Figure 5 of [4] in less detail.

The Moore-Penrose Pseudoinverse of a matrix is computed as X^† = VΣ⁻¹U^T given the Singular Value Decomposition of a matrix X = UΣV^T. The Kronecker product of two matrices A, B (of sizes I × J and K × M respectively) is defined as

$$\mathbf{A}\otimes \mathbf{B}=[\begin{array}{ccc}\hfill \mathbf{A}(1,1)\mathbf{B}\hfill & \hfill \cdots \hfill & \hfill \mathbf{A}(1,J)\mathbf{B}\hfill \\ \hfill ⋮\hfill & \hfill \ddots \hfill & \hfill ⋮\hfill \\ \hfill \mathbf{A}(I,1)\mathbf{B}\hfill & \hfill \cdots \hfill & \hfill \mathbf{A}(I,J)\mathbf{B}\hfill \end{array}]$$

Algorithm 1.

Baseline: Alternating Least Squares (ALS) Algorithm for CMTF [4]

Input: X̲ of size I × J × K, matrices Yi, i = 1 ⋯ 3, of size I × I2, J × J2, and K × K2 respectively, number of factors F.
Output: A of size I × F, B of size J × F, C of size K × F, D of size I2 × F, G of size J2 × F, E of size K2 × F.
1:	Unfold X̲ into X̲(1), X̲(2), X̲(3) (see [27]).
2:	Initialize A, B, C using PARAFAC of X̲. Initialize D, G, E as discussed on the text (e.g. D = Y1 (A‡)T).
3:	while convergence criterion is not met do
4:	$\mathbf{A}={\left[\begin{array}{c}\hfill {\underset{¯}{\mathbf{X}}}_{(1)}\hfill \\ \hfill {\mathbf{Y}}_1\hfill \end{array}\right]}^T{\left({\left[\begin{array}{c}\hfill \mathbf{B}\odot \mathbf{C}\hfill \\ \hfill \mathbf{D}\hfill \end{array}\right]}^{†}\right)}^T$
5:	$\mathbf{B}={\left[\begin{array}{c}\hfill {\underset{¯}{\mathbf{X}}}_{(2)}\hfill \\ \hfill {\mathbf{Y}}_2\hfill \end{array}\right]}^T{\left({\left[\begin{array}{c}\hfill \mathbf{C}\odot \mathbf{A}\hfill \\ \hfill \mathbf{G}\hfill \end{array}\right]}^{†}\right)}^T$
6:	$\mathbf{C}={\left[\begin{array}{c}\hfill {\underset{¯}{\mathbf{X}}}_{(3)}\hfill \\ \hfill {\mathbf{Y}}_3\hfill \end{array}\right]}^T{\left({\left[\begin{array}{c}\hfill \mathbf{A}\odot \mathbf{B}\hfill \\ \hfill \mathbf{E}\hfill \end{array}\right]}^{†}\right)}^T$
7:	D = Y1 (A†)T, G = Y2 (B†)T, E = Y3 (C†)T
8:	end while

The Khatri-Rao product of two matrices A, B with sizes I × F and J × F is

$$\mathbf{A}\odot \mathbf{B}=[\mathbf{A}(:,1)\otimes \mathbf{B}(:,1)\cdots \mathbf{A}(:,F)\otimes \mathbf{B}(:,F)]$$

In Section 4.1, we improve upon Algorithm 1, introducing a few key modifications that allow for faster execution.

3 Proposed Method

3.1 Algorithm Description

There are three main concepts behind Turbo-SMT (outlined in Algorithm 2):

• Phase 1 Obtain a nucleus of our data by using biased sampling.

• Phase 2 Fit CMTF to the reduced data (possibly on more than one nuclei)

• Phase 3 stitch the partial results

Phase 1: Sampling

An efficient way to reduce the size of the dataset, yet operate on a representative subset thereof is to use biased sampling. In particular, given a three-mode tensor X̲ we sample as follows. We calculate three vectors as shown in equations (3.2), one for each mode of X̲ (and respectively Equations 3.3 and 3.4 for the two modes of the matrix).

$${\mathbf{x}}_A(i)=\sum _{j=1}^J\sum _{k=1}^K|\underset{¯}{\mathbf{X}}(i,j,k)|+\sum _{j=1}^{I_1}|{\mathbf{Y}}_1(i,j)|,{\mathbf{x}}_B(j)=\sum _{i=1}^I\sum _{k=1}^K|\underset{¯}{\mathbf{X}}(i,j,k)|+\sum _{i=1}^{I_2}|{\mathbf{Y}}_2(j,i)|,{\mathbf{x}}_C(k)=\sum _{i=1}^I\sum _{j=1}^J|\underset{¯}{\mathbf{X}}(i,j,k)|+\sum _{j=1}^{I_3}|{\mathbf{Y}}_3(k,j)|,$$ (3.2)

$${\mathbf{y}}_{1,A}(i)=\sum _{j=1}^{I_1}|{\mathbf{Y}}_1(i,j)|{\mathbf{y}}_{2,B}(j)=\sum _{i=1}^{I_2}|{\mathbf{Y}}_2(j,i)|,{\mathbf{y}}_{3,C}(k)=\sum _{j=1}^{I_3}|{\mathbf{Y}}_3(k,j)|$$ (3.3)

$${\mathbf{y}}_{1,D}(j)=\sum _{i=1}^I|{\mathbf{Y}}_1(i,j)|,{\mathbf{y}}_{2,G}(i)=\sum _{j=1}^J|{\mathbf{Y}}_2(j,i)|,{\mathbf{y}}_{3,E}(i)=\sum _{k=1}^K|{\mathbf{Y}}_3(k,i)|$$ (3.4)

These vectors, which we henceforth refer to as density vectors are the marginal absolute sums with respect to all but one of the modes of the tensor, and in essence represent the importance of each index of the respective mode. We then sample indices of each mode according to the respective density vector. For instance, assume an I × J × K tensor; suppose that we need a sample of size $\frac{I}{s}$ of the indices of the first mode. Then, we just define $p_{ℐ}(i)={\mathbf{x}}_A(i)/\sum _{i=1}^I{\mathbf{x}}_A(i)$ as the probability of sampling the i-th index of the first mode, and we simply sample without replacement from the set {1 ⋯ I}, using pℐ as bias. The very same idea is used for matrices Y_i. Doing so is preferable over sampling uniformly, since our bias makes it more probable that high density indices of the data will be retained on the sample, and hence, it will be more representative of the entire set.

Suppose that we call ℐ, 𝒥, 𝒦 the index samples for the three modes of X̲. Then, we may take X̲_s = X̲(ℐ, 𝒥, 𝒦) (and similarly for matrices Y_i) to form a nucleus of the data; which essentially is a small, yet representative, sample of our original dataset, where the high density blocks are more likely to appear on the sample. It is important to note that the indices of the coupled modes are the same for the matrix and the tensor, e.g. I randomly selects the same set of indices for X̲ and Y₁. This way, we make sure that the coupling is preserved after sampling.

In cases where the data have very different dimensions per mode (e.g. one mode is significantly smaller than the rest), we may use different sampling factors in order to account for this imbalance.

Finally, Phase 1 can be executed very efficiently, since both the calculation of sample biases, as well as the sampling of indices require only 2 passes on the non-zero elements of the (usually, highly sparse) data.

Phase 2: Fit CMTF to nuclei

The next step of Turbo-SMT is to fit a CMTF model to each nucleus, and then, based on the sampled indices, redistribute the result to the original index space. As we have already discussed, Turbo-SMT is not restricted in any way to a specific CMTF solver; in fact, we provide experiments using both an ALS and a first order gradient approach. In more detail, suppose that A_s is the factor matrix obtained by the aforementioned procedure, and that jointly describes the first mode of X̲_s and Y_1,s. The dimensions of A_s are going to be |ℐ| × F (where ||˙ denotes cardinality and F is the number of factors). Let us further assume matrix A of size I × F which expresses the first mode of the tensor and the matrix, before sampling; due to sampling, it holds that I ≫ |ℐ|. If we initially set all entries of A to zero and we further set A(ℐ, :) = A_s we obtain a highly sparse factor matrix whose non-zero values are a ’best effort’ approximation of the true ones, i.e. the values of the factor matrix that we would obtain by decomposing the full data.

So far, we have provided a description of the algorithm where only one repetition of sampling is used. However, the approximation quality of Turbo-SMT improves as we increase the number of nuclei. To that end, we allow for multiple sampling repetitions in our algorithm, i.e. extracting multiple sample tensors X̲_s and side matrices Y_i,s, fitting a CMTF model to all of them and combining the results in a way that the true latent patterns are retained. We are going to provide a detailed outline of how to carry the multi-repetition version of Turbo-SMT in the following.

While doing multiple repetitions, we keep a common subset of indices for all different samples. In particular, let p be the percentage of common values across all repetitions and ℐ_p denote the common indices along the first mode (same notation applies to the rest of the indices); then, all sample tensors X̲_s will definitely contain the indices ℐ_p on the first mode, as well as $(1-p)\frac{I}{s}$ indices sampled independently (across repetitions) at random. This common index sample is key in order to ensure that our results are not rank deficient, and all partial results are merged correctly. As we discuss in Phase 3, it suffices to keep a set of common “anchor” indices just for first mode, however, we also describe a method that uses common in all modes of the tensor (i,e, ℐ_p, 𝒥_p, 𝒦_p), that is, however, more expensive computationally.

We do not provide an exact method for choosing p, however, as a rule of thumb, we observed that, depending on how sparse and noisy the data is, a range of p between 0.2 and 0.5 works well. This introduces a trade-off between redundancy of indices that we sample, versus the accuracy of the decomposition; since we are not dealing solely with tensors, which are known to be relatively more well behaved in terms of decomposition uniqueness (in contrast to matrices), it pays off to introduce some data redundancy (especially when Turbo-SMT runs in a parallel system) so that we avoid rank-deficiency in our data.

Let r be the number of different sampling repetitions, resulting in r different sets of sampled matrix-tensor couples ${\underset{¯}{\mathbf{X}}}_s^{(i)}$ and ${\mathbf{Y}}_{j,s}^{(i)}$ (i = 1 ⋯ r, j = 1 ⋯ 3). For that set of coupled data, we fit a CMTF model, using a CMTF solver, obtaining a set of factor matrices A⁽ⁱ⁾ (and likewise for the rest).

Phase 3: Stitching partial results

After having obtained these r different sets of partial results, as a final step, we have to merge them together into a set of factor matrices that we would ideally get had we operated on the full dataset.

In order to make the merging work, we first introduce the following scaling on each column of each factor matrix: Let’s take A⁽ⁱ⁾ for example; we normalize each column of A by the ℓ₂ norm of the common part, as described in line 8 of Algorithm 2. By doing so, the common part of each factor matrix (for all repetitions) will be unit norm. This scaling is absorbed in a set of scaling vectors λ_A (and accordingly for the rest of the factors). The new objective function is shown in Equation 3.5

$$\underset{\mathbf{A},\mathbf{B},\mathbf{C},\mathbf{D},\mathbf{E},\mathbf{G}}{\text{min}}{\Vert \underset{¯}{\mathbf{X}}-\sum _k{\mathit{\lambda }}_A(k){\mathit{\lambda }}_B(k){\mathit{\lambda }}_C(k){\mathbf{a}}_k◦{\mathbf{b}}_k◦{\mathbf{c}}_k\Vert }_F^2+{\Vert {\mathbf{Y}}_1-\mathbf{A}\text{ diag}({\mathit{\lambda }}_A*{\mathit{\lambda }}_D){\mathbf{D}}^T\Vert }_F^2+{\Vert {\mathbf{Y}}_2-\mathbf{B}\text{ diag}({\mathit{\lambda }}_B*{\mathit{\lambda }}_E){\mathbf{E}}^T\Vert }_F^2+{\Vert {\mathbf{Y}}_3-\mathbf{C}\text{ diag}({\mathit{\lambda }}_C*{\mathit{\lambda }}_G){\mathbf{G}}^T\Vert }_F^2$$ (3.5)

A problem that is introduced by carrying out multiple sampling repetitions is that the correspondence of the output factors of each repetition is very likely to be distorted. In other words, say we have matrices A⁽¹⁾ and A⁽²⁾ and we wish to merge their columns (i.e. the latent components) into a single matrix A, by stitching together columns that correspond to the same component. It might very well be the case that the order in which the latent components appear in A⁽¹⁾ is not the same as in A⁽²⁾.

The sole purpose of the aforementioned normalization is to resolve the correspondence problem. In Algorithm 3, we merge the partial results while establishing the correct correspondence of the columns.

Algorithm 3 can be seen as a greedy algorithm for solving the correspondence problem between columns. Provided that the factor matrices are not collinear, the algorithm usually finds the correct correspondence. In reality, where data might be noisy, we can use the Hungarian Method [32], which will slightly increase the computational cost of the overall algorithm but solve the problem optimally.

In order to see why this happens, we follow the example of r = 2 of the previous paragraph, according to Algorithm 3, we compute the inner product of the common parts of each column of A⁽¹⁾ and A⁽²⁾. Since the common parts of each column are normalized to unit norm, then the inner product of the common part of the column of A⁽¹⁾ with that of A⁽²⁾ will be maximized (and exactly equal to 1) for the matching columns, and by the Cauchy-Schwartz inequality, for all other combinations, it will be less than 1. Additionally, elimination of the already used columns operates as a tie-breaker.

The non-collinearity assumption implies that the matrix/tensor pair of each nucleus contains data that fit the CMTF model (for the given decomposition rank) well. If either the data do not obey the model, or the rank is too high, the partial factors to be merge might end up being collinear, in which case we have to either decrease the decomposition rank, discard the particular nucleus that produces collinear factors, or readjust s and r so that the nucleus chosen has “good” structure (where “good” is used in the sense that it fits the CMTF model well).

Note that in the way that we describe the stitching, we only need to keep common “anchor” indices for the A matrix, establish the correspondence on A and then propagate the column permutation to the remaining factor matrices. Since Turbo-SMT can be used for data whose dimensions span millions or billions, a set of anchor indices can easily span thousands or even tens of thousands, thus, saving only a set of anchor indices for matrix A results in memory and communication savings. Phase 3, due to its low complexity, can be executed very efficiently. In particular, the StitchFactors algorithm requires O(rF²) steps, where, both r and F are, for most practical cases, very small, compared to the data dimensionality.

For the sake of completeness, here we also describe a theoretically more accurate, but more computationally and memory intensive way to stitch the components. Suppose we have a set of common indices for all A, B, C. We may, thus, perform the stitching jointly for A, B, C and then propagate the permutation that we computed to matrices D, E, G. This joint matching can be done as follows: Consider the following product:

$$\mathbf{A}({ℐ}_p,:)\odot \mathbf{B}({𝒥}_p,:)\odot \mathbf{C}({𝒦}_p,:),$$

where ⊙ is the Khatri-Rao product, as define earlier. This results in a matrix with F columns, where the f-th column is equal to

$$\mathbf{A}({ℐ}_p,f)\otimes \mathbf{B}({𝒥}_p,f)\otimes \mathbf{C}({𝒦}_p,f).,$$

where ⊗ is the Kronecker product. Because of the definition of the PARAFAC decomposition, the above is simply the vectorized f-th rank one component of the PARAFAC decomposition defined by A(ℐ_p, :), B(𝒥_p, :), C(𝒦_p, :), i.e. the pieces of the partial results that correspond to the same “anchor” indices. Thus, the problem of finding the correspondence between, say, the first nucleus and the second nucleus reduces to the problem of finding the optimal column correspondence between matrices

$${\mathbf{A}}^{(1)}({ℐ}_p,:)\odot {\mathbf{B}}^{(1)}({𝒥}_p,:)\odot {\mathbf{C}}^{(1)}({𝒦}_p,:)$$

and

$${\mathbf{A}}^{(2)}({ℐ}_p,:)\odot {\mathbf{B}}^{(2)}({𝒥}_p,:)\odot {\mathbf{C}}^{(2)}({𝒦}_p,:)$$

which can, again, be solved using the Hungarian Method [32]. However, as we mentioned earlier, the size of ℐ_p, 𝒥_p, 𝒦_p, in cases where we are dealing with big tensors and matrices, can be in the orders of thousands or tens of thousands. Thus, computing the above Khatri-Rao products may incur high computational and memory overhead during the stitching step, thus, in Algorithm 2 we show Turbo-SMT when using the lightweight version of stitching (Algorithm 3).

Algorithm 2.

Turbo-SMT: Sparse and parallel CMTF

Input: Tensor X̲ of size I × J × K, matrices Yi, i = 1 ⋯ 3, of size I × I2, J × J2, and K × K2 respectively, number of factors F,   sampling factor s, number of repetitions r.
Output: A of size I × F, b of size J × F, c of size K × F, D of size I2 × F, G of size J2 × F, E of size K2 × F. λA, λB, λC,   λD, λE, λG of size F × 1.
1:	Initialize A, B, C, D, E, G to all-zeros.
2:	Randomly, using mode densities as bias, select a set of 100p% (p ∈ [0, 1]) “anchor” indices ℐp to be common across all repetitions.
	For example, ℐp is sampled with probabilities with $p_{ℐ}(i)={\mathbf{x}}_A(i)/\sum _{i=1}^I{\mathbf{x}}_A(i)$.
3:	for i = 1 ⋯ r do
	{Phase 1: Obtain nuclei through biased sampling}
4:	Compute densities as in equations 3.2, 3.3, 3.4.
	Compute set of indices ℐ(i) as random sample without replacement of {1 ⋯ I} of size I/ (s (1 − p)) with probability    $p_{ℐ}(i)={\mathbf{x}}_A(i)/\sum _{i=1}^I{\mathbf{x}}_a(i)$. Likewise for 𝒥, 𝒦,, ℐ1, ℐ2, and ℐ3. Set ℐ(i) = ℐ ⋃ ℐp.
5:	Get nucleus ${\underset{¯}{\mathbf{X}}}_s^{(i)}=\underset{¯}{\mathbf{X}}({ℐ}^{(i)},{𝒥}^{(i)},{𝒦}^{(i)}),{\mathbf{Y}}_{1s}^{(i)}={\mathbf{Y}}_1({ℐ}^{(i)},{ℐ}_1^{(i)})$ and likewise for ${\mathbf{Y}}_{2s}^{(i)}$ and ${\mathbf{Y}}_{3s}^{(i)}$. Note that the same index   sample is used for coupled modes.
	{Phase 2: Fit the model on each nucleus}
6:	Run a CMTF solver for ${\mathbf{X}}_s^{(i)}$ and ${\mathbf{Y}}_{js}^{(i)}$, j = 1 ⋯ 3 and obtain As, Bs, Cs, Ds, Gs, Es.
7:	A(i)(ℐ(i), :) = As. Likewise for the rest.
8:	Calculate the ℓ2 norm of the columns of the common part: ${\mathit{\lambda }}_A^{(i)}(f)={\Vert {\mathbf{A}}^{(i)}({ℐ}_p,f)\Vert }_2$, for f = 1 ⋯ F. Normalize columns of A(i) using ${\mathit{\lambda }}_A^{(i)}$ (likewise for the rest). Note that the common part of each factor will now be normalized to unit norm.
9:	end for
	{Phase 3: Stitch partial results}
10:	A =StitchFactors ( ${\mathbf{A}}_1^i$). When stitching partial results for A, record the column correspondences for different nuclei
11:	Using the above correspondence, stitch the partial factors of B, C, D, E, G, without performing the full StitchFactors algorithm, but simply reordering the columns.
12:	Using the above column/component correspondence, reorder λA, λB, λC, λD, λE, λG for each repetition.
13:	λA = average of ${{\lambda }_A}_1^i$. Likewise for the rest.

3.2 Sparsity through Sampling

Besides data size reduction, one merit of sampling is sparsity on the latent factors. Every time Turbo-SMT does one repetition, it operates on a sub-sampled version of the data. Consequently, in the third phase of the algorithm, where the results are re-distributed to their indices in the original, high dimensional space, most of the indices of the latent factors are going to be exactly zero, thus resulting in latent factor sparsity. In this way, Turbo-SMT always operates on a sparse set of data, through the entire lifetime of the algorithm, a thing which is not true for the majority of the algorithms both for tensor and coupled decompositions, which usually operate on dense factors (even when the final output is sparse), and have very high and unnecessary storage needs.

Definition 3.1. (Triple-sparse)

An algorithm is triple-sparse when 1) the input of the algorithm is sparse, 2) the intermediate data during the lifetime of the algorithm is sparse, and 3) the final output of the algorithm is sparse.

In the above definition, the input of the algorithm need not necessarily be sparse; however, a triple-sparse algorithm still satisfies the second and third requirement, by operating on a sparse, representative subset of the data. We, thus, call Turbo-SMT, a triple-sparse algorithm. We have to note that we consider the intermediate data that the algorithm manipulates to be sparse with respect to the original data, and not necessarily sparse with respect to the sample size.

In addition to storage and efficiency, a major benefit of sparsity is interpretability: when most of the coefficients of the factors are zero, it is easier for a person to inspect the non-zeros and identify patterns in the results. Additionally, having sparse latent factors has been shown to be equivalent to higher order co-clustering [42, 43] which is a very useful and highly interpretable tool in exploratory data mining.

Algorithm 3.

StitchFactors: Given partial results of factor matrices, merge them correctly

Input: Factor matrices ${\mathbf{A}}_1^i$ of size I × F each, and r is the number of repetitions, ℐp: set of common indices.
Output: Factor matrix A of size I × F.
1:	Set A = A(1)
2:	Set ℓ = {1 ⋯ F}, a list that keeps track of which columns have not been assigned yet.
3:	for i = 2 ⋯ r do
4:	for f1 = 1 ⋯ F do
5:	for f2 in ℓ do
6:	Compute similarity v(f2) = (A(ℐp, f2))T (A(i)(ℐp, f1)))
7:	end for
8:	c* = arg maxc v(c) (Ideally, for the matching columns, the inner product should be equal to 1; conversely, for the rest of the   columns, it should be considerably smaller)
9:	A(:, c*) = A(i)(:, f1)|A(:,c*)=0, i.e. update the zero entries of the column.
10:	Remove c* from list ℓ.
11:	end for
12:	end for

3.3 Parallelization

Turbo-SMT is, by its nature, parallelizable; in essence, we generate multiple samples of the coupled data, we fit a CMTF model to each sample and then we merge the results. By carefully observing Algorithm 2, we can see that lines 3 to 9 may be carried out entirely in parallel, provided that we have a good enough random number generator that does not generate the very same sample across all r repetitions. In particular, the r repetitions are independent from one another, since computing the set of common indices (line 2), which is the common factor across all repetitions, is done before line 3.

4 Further Optimizations

4.1 Speeding up the ALS algorithm

In addition to our main contribution in terms of speeding CMTF in general, we are able to further speed the ALS algorithm up, by making a few careful interventions to the core algorithm (Algorithm 1).

Lemma 4.1

We may do the following simplification to each pseudoinversion step of the ALS algorithm (Algorithm 1):

$${\left[\begin{array}{c}\hfill \mathbf{A}\odot \mathbf{B}\hfill \\ \hfill \mathbf{M}\hfill \end{array}\right]}^{†}={({\mathbf{A}}^T\mathbf{A}*{\mathbf{B}}^T\mathbf{B}+{\mathbf{M}}^T*\mathbf{M})}^{†}[{(\mathbf{A}\odot \mathbf{B})}^T,{\mathbf{M}}^T]$$

Proof

For the Moore-Penrose pseudoinverse of the Khatri-Rao product, it holds that [13], [34]

$${(\mathbf{A}\odot \mathbf{B})}^{†}={({\mathbf{A}}^T\mathbf{A}*{\mathbf{B}}^T\mathbf{B})}^{†}{(\mathbf{A}\odot \mathbf{B})}^T$$

Furthermore [13] (A ⊙ B)^T (A ⊙ B) = A^TA * B^TB For a partitioned matrix $\mathbf{P}=\left[\begin{array}{c}\hfill {\mathbf{P}}_1\hfill \\ \hfill {\mathbf{P}}_2\hfill \end{array}\right]$, it holds that its pseudoinverse may be written in the following form [25]

$${\left[\begin{array}{c}\hfill {\mathbf{P}}_1\hfill \\ \hfill {\mathbf{P}}_2\hfill \end{array}\right]}^{†}={({\mathbf{P}}_1^T{\mathbf{P}}_1+{\mathbf{P}}_2^T{\mathbf{P}}_2)}^{†}[{\mathbf{P}}_1^T,{\mathbf{P}}_2^T]$$

Putting things together, it follows:

$${\left[\begin{array}{c}\hfill \mathbf{A}\odot \mathbf{B}\hfill \\ \hfill \mathbf{M}\hfill \end{array}\right]}^{†}={({\mathbf{A}}^T\mathbf{A}*{\mathbf{B}}^T\mathbf{B}+{\mathbf{M}}^T*\mathbf{M})}^{†}[{(\mathbf{A}\odot \mathbf{B})}^T,{\mathbf{M}}^T]$$

which concludes the proof.

The above lemma implies that substituting the naive pseudoinversion of $\left[\begin{array}{c}\hfill \mathbf{A}\odot \mathbf{B}\hfill \\ \hfill \mathbf{M}\hfill \end{array}\right]$ with the simplified version, offers significant computational gains to Algorithm 1. More precisely, if the dimensions of A, B and M are I × R, J × R and I × I₂, then computing the pseudoinverse naively would cost O (R² (IJ + I₂)), whereas our proposed method yields a cost of O (R² (I + J + I₂)) because of the fact that we are pseudoinverting only a small R × R matrix. We have to note here that in almost all practical scenarios R ≪ I, J, I₂.

Table 2 provides a solid impression of the speedup achieved on the core ALS algorithm, as a result of the simplification of the pseudo-inversion step, as derived above. In short, we can see that the speedup achieved is in most realistic cases 2× or higher, adding up to being a significant improvement on the traditional algorithm.

Table 2.

Pseudoinversion speedup (100000 runs)

R	I = 10	I = 100	I = 1000	I = 10000	I = 100000
1	2.4686 ± 0.3304	2.4682 ± 0.3560	2.4479 ± 0.2948	2.4546 ± 0.3214	2.4345 ± 0.3144
5	2.2496 ± 0.3134	2.2937 ± 0.1291	2.2935 ± 0.1295	2.2953 ± 0.1291	2.2975 ± 0.1318
10	2.6614 ± 0.1346	2.6616 ± 0.1368	2.6610 ± 0.1380	2.6591 ± 0.1377	2.6593 ± 0.1428

4.2 Making ALS robust to missing values

In many practical scenarios, we often have corrupted or missing data. For instance, when measuring brain activity, a few sensors might stop working, whereas the majority of the sensors produce useful signal. Despite these common data imperfections, it is important for a data mining algorithm to be able to operate. The work of Tomasi et. al [49] provides a very comprehensive study on how to handle missing values for plain tensor decompositions. A very clean and straightforward way of handling missing values is to ignore them throughout the optimization process, both with respect to the original data and with respect to the model. This can be achieved through masking the missing values, hiding them from the optimization. Notice that is not the same as simply zeroing out all missing values, since 0 might have a valid physical interpretation.

In this section, we show how this masking can be applied to the ALS algorithm for the CMTF model. In [4], the authors show how this can be achieved for the CMTF-OPT algorithm. In any case, this masking approach is very important because besides enabling the analysis of incomplete datasets, it can also serve as stepping stone for estimating those missing values: When obtaining a low rank model of the incomplete dataset after masking the missing values, we can reconstruct the original data and our reconstruction will contain imputations for the missing values. This can be seen as the first iteration of an Expectation-Maximization scheme introduced in [49].

Following the formulation of [6, 4] we define a ’weight’ tensor W̲ which has ’0’ in all coefficients where values are missing, and ’1’ everywhere else. Similarly, we introduce three weight matrices W_i for each of the coupled matrices Y_i. Then, the optimization function of the CMTF model becomes

$$\underset{\mathbf{A},\mathbf{B},\mathbf{C},\mathbf{D},\mathbf{E},\mathbf{G}}{\text{min}}{\Vert \underset{¯}{\mathbf{W}}*(\underset{¯}{\mathbf{X}}-\sum _k{\mathbf{a}}_k\circ {\mathbf{b}}_k\circ {\mathbf{c}}_k)\Vert }_F^2+{\Vert {\mathbf{W}}_1*({\mathbf{Y}}_1-\mathbf{A}{\mathbf{D}}^T)\Vert }_F^2+{\Vert {\mathbf{W}}_2*({\mathbf{Y}}_2-\mathbf{B}{\mathbf{E}}^T)\Vert }_F^2+{\Vert {\mathbf{W}}_3*({\mathbf{Y}}_3-\mathbf{C}{\mathbf{G}}^T)\Vert }_F^2$$

As we show in Algorithm 1, we may solve CMTF by solving six least squares problems in an alternating fashion. A fortuitous implication of this fact is that in order to handle missing values for CMTF, it suffices to solve

$$\underset{\mathbf{B}}{\text{min}}{\Vert \mathbf{W}*(\mathbf{X}-\mathbf{A}{\mathbf{B}}^T)\Vert }_F^2$$ (4.6)

where W is a weight matrix in the same sense as described a few lines earlier.

Our solution draws from a similar derivation in [43] (Section 3.B) On our way tackling the above problem, we first need to investigate its scalar case, i.e. the case where we are interested only in B(j, f) for a fixed pair of j and f. The optimization problem may be rewritten as

$$\underset{\mathbf{B}(j,f)}{\text{min}}\Vert \mathbf{W}(:,j)*\mathbf{X}(:,j)-(\mathbf{W}(:,j)*\mathbf{A}(:f))\mathbf{B}{(j,f)}^T\Vert$$ (4.7)

which is essentially a scalar least squares problem of the form: ${\text{min}}_b{\Vert \mathbf{x}-\mathbf{a}b\Vert }_2^2$ with solution in analytical form: $b=\frac{{\mathbf{x}}^T\mathbf{a}}{{\Vert \mathbf{a}\Vert }_2^2}$. The reader may notice that the dimensions of W and A are incompatible, however, in the scalar case of Eq. 4.7, we multiply element-wise the columns of W and A, which are of the same dimensions.

We may, thus, solve this problem of Equation 4.6 using coordinate descent, on B iteratively, until convergence. Therefore, with the aforementioned derivation, we are able to modify our original algorithm in order to take missing values into account.

So far, we have described how to make ALS robust against missing values, however a few modifications to Turbo-SMT are required in order to be able to use the robust ALS core solver. In particular, in order to use the masking scheme for Turbo-SMT, during sampling, we have to take into account that some values in the data are missing: when we compute the weights for the biased sampling (i.e. Equations 3.2, 3.3, 3.4) we have to hide the missing values. In order to do that, we compute the weights on

$${\underset{¯}{\mathbf{X}}}_m=\underset{¯}{\mathbf{W}}*\underset{¯}{\mathbf{X}}$$

and

$${\mathbf{Y}}_{1m}={\mathbf{W}}_1*{\mathbf{Y}}_1$$

where the weights W̲ and W₁ are defined above (and accordingly for Y₂, Y₃). This is equivalent to assuming that the missing values are equal to 0, and because computation of the weights is calculated through addition, treating the missing values as 0 in this context is equivalent to ignoring them. After these modifications, Turbo-SMT can be used along with the robust to missing values ALS algorithm for large and incomplete datasets.

5 Knowledge Discovery

5.1 Turbo-SMT on Brain Image Data With Additional Semantic Information

As part of a larger study of neural representations of word meanings in the human brain [37], we applied Turbo-SMT to a combination of datasets which we henceforth jointly refer to as BrainQ. This dataset consists of two parts. The first is a tensor that contains measurements of the fMRI brain activity of 9 human subjects, when shown each of 60 concrete nouns (5 in each of 12 categories, e.g. dog, hand, house, door, shirt, dresser, butterfly, knife, telephone, saw, lettuce, train). fMRI measures slow changes in blood oxygenation levels, reflecting localized changes in brain activity. Here our data is made up of 3×3×6mm voxels (3D pixels) corresponding to fixed spatial locations across participants. Recorded fMRI values are the mean activity over 4 contiguous seconds, averaged over multiple presentations of each stimulus word (each word is presented 6 times as a stimulus). Further acquisition and preprocessing details are given in [37]. This dataset is publicly available². The second part of the data is a matrix describing the semantics of these 60 nouns, using crowd-sourced numerical responses to 218 questions such as ”is it smaller than a golfball?” (see also samples in Fig. 4). This dataset has been used before in works such as [39], [40].

Figure 4.

Turbo-SMT finds meaningful groups of words, questions, and brain regions that are (both negatively and positively) correlated, as obtained using Turbo-SMT. For instance, Group 3 refers to small items that can be held in one hand, such as a tomato or a glass, and the activation pattern is very different from the one of Group 1, which mostly refers to insects, such as bee or beetle. Additionally, Group 3 shows high activation in the premotor cortex which is associated with the concepts of that group.

BrainQ’s size is 60×77775×9 with over 11 million non-zeros (tensor), and 60 × 218 with about 11000 non-zeros (matrix). The dimensions might not be extremely high, however, the data is very dense and it is therefore difficult to handle efficiently.

5.2 Simultaneous Clustering of Words, Questions and Regions of the Brain

One of the strengths of CMTF is its expressiveness in terms of simultaneously soft-clustering all involved entities of the problem. By taking a low rank decomposition of the BrainQ data (using r = 5 and s_I = 3, s_J = 86, s_K = 1 for the tensor and s_I for the questions dimension of the matrix)³, we are able to find groups that jointly express words, questions and brain voxels (we can also derive groups of human subjects; however, it is an active research subject in neuroscience, whether brain-scans should differ significantly between people, and is out of the scope of the present work).

In Figure 4, we display 4 such groups of brain regions that are activated given a stimulus of a group of similar words; we also display the most prominent words, along with groups of similar questions that were highly correlated with the words of each group. Moreover, we were able to successfully identify high activation of the premotor cortex in Group 3, which is associated with concepts that have conventional manual uses (such as holding, or picking up).

5.3 Predicting Brain Activity from Questions

In addition to soft-clustering, the low rank joint decomposition of the BrainQ data offers another significant result. This low dimensional embedding of the data into a common semantic space, enables the prediction of, say, the brain activity of a subject, for a given word, given the corresponding vector of question answers for that word. In particular, we use this linear transformation in order to map the question answer vector to the latent semantic space and then expanding it to the brain voxel space, we obtain a fairly good prediction of the brain activity. This linear transformation is reminiscent of the mapping done in Latent Semantic Analysis (LSA) [21], where the authors compute a low rank embedding of terms and documents, and given a query consisting of terms, they project it to the latent dimension, taking advantage of the compaction offered by the low rank approximation. In the case of LSA, the embeddings used are orthonormal (since they are computed using SVD), and thus the “mapping” is a proper projection to the subspace. In our case, we do not restrict the embeddings to be orthonormal, and thus we perform a linear transformation.

To evaluate the accuracy of these predictions of brain activity, we follow a leave-two-out scheme, where we remove two words entirely from the brain tensor and the question matrix; we carry out the joint decomposition, in some very low dimension, for the remaining set of words and we obtain the usual set of matrices A, B, C, D. Due to the randomized nature of Turbo-SMT, we did 100 repetitions of the procedure described below.

Let q_i be the question vector for some word i, and v_i be the brain activity of one human subject, pertaining to the same word. By left-multiplying q_i with D^T, we map q_i to the latent space of the decomposition; then, by left-multiplying the result with B, we map the result to the brain voxel space. Thus, our estimated (predicted) brain activity is obtained as v̂_i = BD^T q_i

Given the predicted brain activities v̂₁ and v̂₂ for the two left out words, and the two actual brain images v₁ and v₂ which were withheld from the training data, the leave-two-out scheme measures prediction accuracy by the ability to choose which of the observed brain images corresponds to which of the two words. After mean-centering the vectors, this classification decision is made according to the following rule:

$${\Vert {\mathbf{v}}_1-{\hat{\mathbf{v}}}_1\Vert }_2+{\Vert {\mathbf{v}}_2-{\hat{\mathbf{v}}}_2\Vert }_2<{\Vert {\mathbf{v}}_1-{\hat{\mathbf{v}}}_2\Vert }_2+{\Vert {\mathbf{v}}_2-{\hat{\mathbf{v}}}_1\Vert }_2$$

Although our approach is not designed to make predictions, preliminary results are very encouraging: Using only F=2 components, for the noun pair closet/watch we obtained mean accuracy of about 0.82 for 5 out of the 9 human subjects. Similarly, for the pair knife/beetle, we achieved accuracy of about 0.8 for a somewhat different group of 5 subjects. For the rest of the human subjects, the accuracy is considerably lower, however, it may be the case that brain activity predictability varies between subjects, a fact that requires further investigation.

6 Generality: Mining Social Networks with Additional Information

The CMTF framework, and Turbo-SMT by the same token, can be applied in a very diverse set of applications. In the previous subsection we demonstrated the expressive power of the framework for the BrainQ dataset; here we show how it can be applied in a completely different scenario, that of social network analysis with side information. To that end, we use Turbo-SMT to analyze a Facebook dataset, introduced in [51]⁴. This dataset consists of a 63890 × 63890 × 1847 (wall, poster, day) tensor with about 740,000 non-zeros, and a 63890 × 63890 who is friends with whom matrix, with about 1.6 million non-zeros. The tensor contains the amount of posts a user posted on some user’s Wall. In principle it contains counts, however, most of the values are either 1 or 0. In contrast to BrainQ, this dataset is very sparse (as one would expect from a social network dataset). However, Turbo-SMT works in both cases, demonstrating that it can analyze data efficiently, regardless of their density.

We decomposed the data into 25 rank one components, using s_I = 1000, s_J = 1000, s_K = 100 and s_I for both dimensions of the matrix, and manually inspected the results. A fair amount of components captured normal activity of Facebook users who occasionally post on their friends’ walls; here we only show one outstanding anomaly, due to lack of space: In Fig. 5 we show what appears to be a spammer, i.e. a person who, only on a certain day, posts on many different friends’ walls: the first subfigure corresponds to the wall owners, the second subfigure corresponds to the people who post on these walls, and the third subfigure is the time (measured in days); we thus have one person, posting on many peoples’ walls, on a single day.

We chose to include this particular result in order to showcase the modelling versatility of CMTF models, and thus of the Turbo-SMT algorithm, since we are able to express a variety of problems in the CMTF framework, and additionally, solve them very efficiently, and obtain sparse and interpretable solutions by using Turbo-SMT. The Facebook dataset merits its own detailed analysis, however, here we include a small, preliminary result, due to space considerations.

7 Experiments

7.1 Experimental Setup

We implemented Turbo-SMT in Matlab. Our implementation of the code is publicly available.⁵ For the parallelization of the algorithm, we used Matlab’s Parallel Computing Toolbox. For tensor manipulation, we used the Tensor Toolbox for Matlab [10] which is optimized especially for sparse tensors (but works very well for dense ones too). We use the CMTF-ALS and the CMTF-OPT [4] algorithms as baselines, i.e. we compare Turbo-SMT when using one of those algorithms as their core CMTF implementation, against the plain execution of those algorithms. In order to make this comparison possible, all datasets we used were within the capabilities of the Tensor Toolbox for Matlab.

We implemented our version of the ALS algorithm, and we used the CMTF Toobox⁶ implementation of CMTF-OPT. All experiments were carried out on a machine with 4 Intel Xeon E74850 2.00GHz, and 512Gb of RAM. The version of Matlab was R2013a (8.1.0.604) 64-bit. In the appendix, we show timing experiments on a less powerful machine, in order to demonstrate that the principle behind Turbo-SMT is applicable to a variety of platforms. Whenever we conducted multiple iterations of an experiment (due to the randomized nature of Turbo-SMT), we report error-bars along the plots. For all the following experiments we used either portions of the BrainQ dataset, or the whole dataset.

7.2 Run time

A major feature of Turbo-SMT is scaling up and boosting existing, potentially highly optimized and state of the art, CMTF solvers. Key facts that contribute to this scale-up are: 1) dimensionality reduction through sampling, 2) the fact that Turbo-SMT operates on sparse data throughout its lifetime, and 3) that Turbo-SMT is highly parallelizable. Figure 1 illustrates this behaviour. In this section, we measure the execution time on the BrainQ data for a variety of different configurations of Turbo-SMT, and for two different baselines (CMTF-OPT and CMTF-ALS). In particular, while keeping the number of repetitions r fixed to 4, we vary the sampling factor s for values {2, 5, 10, 20}. Because BrainQ is highly imbalanced, the actual sampling factors we use are $[\begin{array}{ccc}\hfill \frac{s}{2}\hfill & \hfill s\hfill & \hfill 1\hfill \end{array}]$. We also vary the decomposition rank F for 1, 5, and 10.

The reason why we keep r fixed to 4 is because we conducted the same experiment on a machine with 4 cores (showing the results in the appendix) and for consistency purposes we chose the same r for both experiments. In Section 7.3 we show, however, that increasing r is improving the accuracy and if we use a machine with as many cores as r, then we may do so entirely in parallel, without significantly deviating from the timings that we report in this section.

When we compare Turbo-SMT against one of the two baselines, we make sure that we use the corresponding baseline as the core solver for Turbo-SMT. We observed a certain variation of solutions achieved by the baselines, depending on the number of iterations of the algorithm. Thus, we chose to experiment for three different limits for the number of iterations for the baselines (also enforcing the same limit in the core solver of Turbo-SMT). In particular, all the experiments are executed for 100, 1000, and 10000 iterations⁷. Note that this number indicates the maximum number of iterations (denoted by iter), however the baseline algorithm may terminate before reaching this number of iterations, depending on its (random) initialization. The termination criterion for both baselines was either meeting the maximum number of iterations iter, or the difference of the objective function between two consecutive iterations to be less than 10⁻⁸. Finally, for every combination of rank, sampling factor, and number of iterations, we run Turbo-SMT and the corresponding baseline 5 times, in order to account for multiple local minima as well as variation in the execution times.

Figure 6 shows the run times for CMTF-OPT and CMTF-ALS. In particular, we show the average run time, as well as the maximum run time, in order to account for the worst case scenario. As expected, the run time for both baselines increases as the decomposition rank, and the number of iterations increase. There are a few cases where the runtime for iter=1000 is larger than that of iter=10000, however, since iter is merely the maximum number of iterations, and the problem being solved is a highly non-convex one, it is very likely that a run from the batch where iter was 10000 converged faster than a run from the batch of iter=1000. In Figure 7 we show the corresponding run times for Turbo-SMT, using both baselines. We observe the general trend of the run time increasing with the rank and the number of iterations, however, the run time decreases as the sampling factor decreases.

Figure 6.

Run time (in seconds) for the baselines, on the full BrainQ dataset. The results shown are over 5 full runs of the algorithm.

Figure 7.

Run time (in seconds) for Turbo-SMT when using each of the baselines as core solver. The results shown are over 5 full runs of the algorithm. For every value of F, the bars on the plot starting from left to right, correspond to the parameters shown on the legend. The number of repetitions r was set to 4.

7.3 Accuracy

During the experiment described in Section 7.2, besides run time, we also measured the reconstruction error for each method and each combination of parameters. In Figure 8 we show the reconstruction errors for the two baselines, and in Figure 9 the error for Turbo-SMT. A first observation is that the error is fairly high for both the baselines and Turbo-SMT, fact that possibly indicates that the structure of the data is not exactly the one being imposed by CMTF, however, a low rank approximation is still useful for exploratory analysis.

Figure 8.

Reconstruction error of the tensor and the matrix (as measured by the cost function of the problem) for the two baselines. for r = 4.

Figure 9.

Reconstruction error for Turbo-SMT.

Figures 9 assumes that the number of repetitions r is fixed to 4, which is a rather small number, considering the size of the data. When we, however, increase r, the approximation improves. In Figure 10 we demonstrate that the algorithm operates correctly, in the sense that it reduces the model error (Equation 2.1) when doing more repetitions. In particular, the vertical axis displays the relative error, i.e. $\frac{\text{Turbo}-\mathrm{S}\mathrm{M}\mathrm{T}\text{ error}}{\text{baseline error}}$ (with ideal being equal to 1) and the horizontal axis is the number of repetitions in the sampling. We use 60 × 200 × 9 portion of the BrainQ tensor and the entire matrix, to ensure that both the baseline (CMTF-OPT in this case) and Turbo-SMT run in a short amount of time. We set $s=[\begin{array}{ccc}\hfill 10\hfill & \hfill 20\hfill & \hfill 1\hfill \end{array}]$. We run the experiment 1000 times and we keep the solutions achieving the minimum error. As Figure 10 shows, Turbo-SMT’s relative error decreases as a function of the repetitions, indicating that Turbo-SMT generally reduces the CMTF approximation error with every repetition. As the decomposition rank increases, the number of repetitions needed increases as well, since the model to be captured becomes more complex. Because we are operating on smaller pieces of data, we aim for very low rank decompositions, in order to avoid overfactoring (i.e. choosing a higher number than the rank of the data) which may cause problems with the stitching and lead to instabilities. Additionally, due to the fact that Turbo-SMT operates on random samples of the data, convergence of the approximation error is noisier than deterministic methods such as ALS.

Figure 10.

The relative error of the model, as a function of the number of repetitions r is decreasing, which empirically shows that Turbo-SMT generally reduces the approximation error of the CMTF model.

It is important to note that given a large dataset, during the first few repetitions of Turbo-SMT, the accuracy may be comparable to the null model’s (i.e. the all-zero model), as shown in Figure 10; this is because Turbo-SMT starts with a null model and progressively builds it up. However, the important take home point here is that given a machine or a cluster of machines with enough amount of cores, we can enable the baselines to work on very large datasets that may not fit in main memory, and by running more repetitions, we explore more of the data and improve the approximation accuracy of the model.

7.4 Sparsity

One of the main advantages of Turbo-SMT is that, it is triple-sparse, i.e. starting from (possibly) sparse data, every intermediate result of Turbo-SMT is sparse, as well as the final output. In Fig. 11 we demonstrate the sparsity of Turbo-SMT’s results by introducing the relative sparsity metric; this intuitive metric is simply the ratio of the output size of the baseline algorithm, divided by the output size of Turbo-SMT. The output size is simply calculated by adding up the number of non-zero entries for all factor matrices output by the algorithm. We use a portion of the BrainQ dataset in order to execute this experiment. We can see that for the relatively dense BrainQ dataset, we obtained significantly more sparse results;, e.g. up to 65 times more sparse with almost same approximation error, for the case of CMTF-OPT. We observe a large difference of result sparsity when using CMTF-OPT, as opposed to ALS; most likely, this difference is due to the fact that, according to [4], CMTF-OPT converges to a better local minimum than ALS. The results of Fig. 11 indicate that our triple-sparse algorithm is able to capture the most useful variation of the data, successfully suppressing noise.

Figure 11.

The relative output size vs. the relative cost indicates that, even for very dense datasets such as BrainQ, we are able to get up to 65 fold (for CMTF-OPT) decrease in the output size, while maintaining almost same approximation error as the baseline.

7.5 Comparison to Tucker3 Compression

One existing approach used for speeding up the PARAFAC decomposition is based on the CANDELINC theorem [16] and has been used in [15, 14]; roughly, the method first uses Tucker3 in order to compress the original tensor to a smaller, core tensor, then fits the PARAFAC decomposition on the core tensor, and finally, projects the factor matrices to the original space. In Section 5.3 of [31], it is implied that one could do the same for the CMTF problem. Before proceeding with a brief sketch of the approach which, to the best of our knowledge, has not been published yet, we provide a brief overview of the Tucker3 decomposition.

Consider the I × J × K tensor X̲. Then, its {Q, R, P} Tucker3 decomposition consists of a P × Q × R core tensor, say, G̲ and three assorted, unitary, matrices U, V, Z with sizes I × P, J × Q and K × R respectively. The Tucker3 objective function is:

$$\underset{\underset{¯}{\mathbf{G}},\mathbf{A},\mathbf{B},\mathbf{C}}{\text{min}}{\Vert \underset{¯}{\mathbf{X}}-\sum _{p=1}^P\sum _{q=1}^Q\sum _{r=1}^R\underset{¯}{\mathbf{G}}(p,q,r){\mathbf{u}}_p\circ {\mathbf{v}}_q\circ {\mathbf{z}}_r\Vert }_F^2$$

and we may write the decomposition, compactly, as:

$$\underset{¯}{\mathbf{X}}\approx [{\underset{¯}{\mathbf{G}}}^{(P\times Q\times R)},{\mathbf{U}}^{(I\times P)},{\mathbf{V}}^{(J\times Q)},{\mathbf{Z}}^{(K\times R)}]$$

Having a tensor X̲ coupled with matrices Y_i, i = 1 ⋯ 3, we may first obtain the Tucker3 decomposition of X. Consequently, we may use U in order to project Y₁ to the compressed space, and respectively V for Y₂ and Z for Y₃. We, thus, obtain a new set of coupled data: the core tensor G̲, and the projected side matrices. Then, we fit a CMTF model to the compressed data, and as a final step, we use U in order to project A, D to their original dimension (and accordingly for the rest of the factor matrices).

This method, however, lacks a few key features that Turbo-SMT has:

• Tucker3 is now a bottleneck; its computation (even though there exist memory efficient implementations in the literature [10], [29], which we use in our implementation) is very costly, compared to the simple sampling scheme that Turbo-SMT is using. Even though alternatives to Tucker3 could be used for the compression (e.g. HOSVD [20]), the authors of [15] state that the algorithm is better off using Tucker3 compression, quality-wise, whenever Tucker3 is able to be computed.

• This method, in contrast to Turbo-SMT, is not parallelizable, at least not in an obvious way, that would make its computation more efficient.

• This compression-based technique is not triple-sparse: The output of Tucker3 is dense, hence the core tensor G̲ and the projected side matrices are going to be dense. Additionally, both ALS and CTMF-OPT [4] produce dense factors. Therefore, this technique is prone to storage and interpretability issues.

We implemented this compression-based technique, using Tensor Toolbox’s [10] memory efficient implementation of the Tucker3 decomposition. In Fig. 12, we illustrate the wall-clock time of this approach, compared to Turbo-SMT, on the entire BrainQ dataset; we chose s = 5 for Turbo-SMT, and we chose P = Q = R = 60 for the compression-based technique. We observe that Turbo-SMT performs significantly better, while, additionally, producing sparse outputs.

Figure 12.

Comparison of Turbo-SMT and a compression-based technique, that uses the Tucker3 decomposition, as described in Sec. 7.5. Turbo-SMT significantly outperforms the alternative method.

7.6 Performance of ALS with missing values

In order to measure resilience to missing values we define the Signal-to-Noise Ratio (SNR) as simply as $\mathrm{S}\mathrm{N}\mathrm{R}=\frac{{\Vert {\underset{¯}{\mathbf{X}}}_m\Vert }_F^2}{{\Vert {\underset{¯}{\mathbf{X}}}_m-{\underset{¯}{\mathbf{X}}}_0\Vert }_F^2}$, where X̲_m is the reconstructed tensor when a m fraction of the values are missing. In Figure 13, we demonstrate the results of that experiment; we observe that even for a fair amount of missing data, the algorithm performs reasonably well, achieving high SNR. Moreover, for small amounts of missing data, the speed of the algorithm is not degraded, while for larger values, it is considerably slower, probably due to Matlab’s implementation issues. However, this is encouraging, in the sense that if the amount of missing data is not overwhelming, Turbo-SMT is able to deliver a very good approximation of the latent subspace. This experiment was, again, conducted on a portion of BrainQ.

Figure 13.

This Figure shows the Signal-to-Noise ratio (SNR)-as defined in the main text- as a function of the percentage of missing values. We can observe that, even for a fair amount of missing values, the SNR is quite high, signifying that Turbo-SMT is able to handle such ill-conditioned settings, with reasonable fidelity.

8 Related Work

8.1 Coupled, Multi-block, Multi-set Models

Coupled Matrix-Tensor Factorizations belong to a family of models also referred to as Multi-block or Multi-set in the literature. Smilde et al. in [48] provided the first disciplined treatment of such multi-block models, in a chemometrics context. One of the earliest works that introduce the concept of coupling in data mining applications is [11], where the authors apply their algorithms to movie recommendation and newsgroup article clustering. In [47], Singh and Gordon introduce a collective matrix factorization framework, again in a data mining setting, where the coupling is between matrices. An important issue with these models is how to weigh the different data blocks such that scaling differences may be alleviated. In [52], Wilderjans et al. propose and compare two different weighing schemes.

Most related to the present work is the work of Acar et al. in [4], where a first order optimization approach is proposed, in order to solve the CMTF problem. In [5], Acar et. al apply the CMTF model, using the aforementioned first-order approach in a bioinformatics setting. In [3], Acar et. al introduce a coupled matrix decomposition, where two matrices match on one of the two dimensions, and are decomposed in the same spirit as in CMTF, while imposing explicit sparsity constraints (via ℓ₁ norm penalties); although Turbo-SMT also produces sparse factors, this so happens as a fortuitous byproduct of sampling, whereas in [3] an explicit sparsity penalty is considered. As an interesting application, in [55], the authors employ CMTF for Collaborative Filtering. On a related note, [53], [33], and [35] introduce models where multiple tensors are coupled with respect to one mode, and analyzed jointly. Finally, extending the CMTF framework, Acar et al. propose in [7] a new model where rank-one components can be shared by the matrix and the tensor, or belong solely to either one of the data pieces/blocks.

To the best of our knowledge, Turbo-SMT is the first algorithm that is able parallelize and sparsify any (possibly highly fine tuned) core algorithm for CMTF. The original paper introducing Turbo-SMT appeared at SIAM SDM 2014 [45].

8.2 Fast & Scalable Tensor Decompositions

The mechanics behind the Tensor Toolbox for Matlab [10] are introduced in [30] and later on in[9], where Bader and Kolda first show how to efficiently decompose sparse tensors, avoiding the materialization of prohibitively dense and large intermediate data. Bro and Sidiropoulos introduce a compression based method in [14] which is able to speed up PARAFAC, without however exploiting parallelization. In [41] the authors introduced a parallel algorithm for the regular PARAFAC decomposition, where a sampling scheme of similar nature as here is exploited; following a similar split and merge parallelization scheme, [46] introduces a fast and parallelizable algorithm for PARAFAC, based on compression using random projections, which provides guarantees with respect to the identifiability of the true latent factors. In [26], a scalable MapReduce implementation of PARAFAC is presented, where the intermediate data problem is avoided in the same way as in [30, 9], with specific considerations that pertain to the distributed implementation. Finally, in [54], the authors introduce a parallel framework in order to handle tensor decompositions efficiently. More recently, Beutel et al. [12] introduce a highly scalable Distributed Stochastic Gradient system for CMTF on MapReduce. Finally, in [19], the authors propose an alternative parallel architecture for decomposing tensors on a cluster, by exploiting the multilinear nature of the tensor data and mapping the computation in a similar multilinear fashion on the cluster’s machines. Finally, in [28] the authors introduce a method to vertically partition a multi-relational dataset (which can be viewed as a multiway tensor) using relational normalization, which results in independent decompositions of smaller tensors, that are combined at the end.

8.3 Sampling based techniques

Turbo-SMT draws its strength in reducing the problem size using biased sampling. In that sense, it is related to the line of work under the name of “CUR” [22]. In its matrix version, given a matrix X, CUR samples m columns of X which define matrix C, samples n rows of X that form matrix R and uses C, R to compute the “core” matrix U. The main advantage of doing so is that C, R contain actual pieces of the data, which makes the decomposition easily interpretable. There is also follow up work that generalizes CUR to tensors [36, 23] in the same spirit. The way sampling is used in Turbo-SMT is different, since it is used in order to obtain a small piece of the data which is later decomposed according to CMTF; thus the end result of the algorithm will be approximating the result of CMTF and not define a decomposition that contains actual pieces of the data.

Finally, in [41] a subset of the authors of the present work propose a biased sampling based PARAFAC decomposition that uses the same sample, parallelize, and merge framework.

8.4 Tensor Applications to Brain Data

There has been substantial related work, which utilizes tensors for analyzing brain data. In [2] Acar et al. apply the PARAFAC decomposition on EEG measurements in order to detect epileptic seizures. Morup et al. in [38] introduce a variation of the PARAFAC decomposition that is able to handle temporal shifts across brain measurements. In [17], the authors describe how one can use non-negative tensor factorization for source separation and analysis of brain signals, and finally, in [18] et al. introduce a constrained PARAFAC model in order to identify the connectivity between different brain regions.

8.5 Tensor Applications to Social Network Analysis

Social network analysis has also been tackled using tensors before. One of the first works that did so was the work of Bader et al. [8], where the authors detect cliques of dense interaction in the ENRON e-mail network, while tracking their activity over time. The findings of [8] have also been reproduced in [43] using different, albeit in the tensor regime, techniques. Lin et al. [33] employs coupled tensor factorization for community detection in social graphs with context information about the users. Finaly, in [41], the authors apply their tensor decomposition algorithm on a Facebook dataset, detecting normal and abnormal user interaction.

9 Conclusions

The main contributions of our work are:

• Parallel & triple-sparse algorithm: Turbo-SMT is able to parallelize any CMTF solver, producing much sparser results. Moreover, the way that Turbo-SMT is defined, it can operate on data that do not fit in main memory, thus enabling state of the art CMTF solvers to work on such data, even though they are not specifically designed to do so.

• Effectiveness and Knowledge Discovery: Turbo-SMT, applied to the BrainQ dataset, discovers meaningful triple-mode clusters: clusters of words, of questions, and of brain regions have similar behavior; as a by-product, Turbo-SMT predicts brain activity, with performance that matches state of the art predictors.

• Reproducibility: We make our code public, enabling reproducibility and re-usability of our work.

Appendix

A Run times of BrainQ on a less powerful machine

The run time experiments shown in Section 7.2 are executed on a powerful workstation with ample memory and processing power. In this appendix, we show results of the very same experiments, when executed on a much smaller, more accessible machine, an iMac with 4Gb of RAM and an Intel i5 processor. The Matlab version on that machine was 7.12.0.635 (R2011a) 64-bit. Due to timing considerations, for the ALS algorithm we only ran the experiment for 100 iterations of the core solver, since the machines memory and processing power seemed inadequate.

Figure 14 shows the run times of the baselines. As a general trend, we observe that for most of the configurations, the algorithms take more time to execute compared to the powerful workstation case of Section 7.2, which is expected. In Figure 15 we show the timings for Turbo-SMT using the two baselines as core solvers; Turbo-SMT is generally faster than the baselines, however the speedup yielded here is smaller. Again, this behavior is expected, since the all four cores of the processor were fully utilized during the entire time of the experiment, and the memory of the machine was rather small, leading to a large swap file which resulted in heavy disk I/O, further burdening the CPU cores.

Figure 14.

Run time (in seconds) for the baselines

Figure 15.

Run time (in seconds) for Turbo-SMT

However, the main principle of Turbo-SMT still holds: by breaking up the data into smaller pieces, Turbo-SMT is able to parallelize the computation, improving performance and potentially enabling the analysis of very large datasets that do not fit in main memory that the state of the art simply cannot handle.

B Run times and sparsity of Facebook

In this section of the appendix, we provide more details on the technical aspects of analyzing the Facebook dataset, that do not pertain to data analysis and discovery. We were not able to measure reconstruction error in this dataset due to Matlab’s memory limitations. Figure 16(a) shows the run time for CMTF-OPT and Turbo-SMT using CMTF-OPT for Facebook, for a variety of ranks (using s₁ = 10, s₂ = 10, and s₃ = 2, and 100 iterations for the core solver). Timing was computed on the same workstation that was used for Section 7.2. Because the Facebook dataset is extremely sparse and the baseline is able to handle it more efficiently, using sparse computations, the speedup observed is very small. However, as shown in Figure 16(b), Turbo-SMT yields results that are much sparser than those of the baseline (where the definition of sparsity is the same as in Section 7.4), which is very important for interpreting the results.

Figure 16.

Run times, sparsity, and approximation error for the Facebook dataset, using CMTF-OPT and Turbo-SMT with CMTF-OPT as a core solver.