Beyond Low Rank + Sparse: Multi-scale Low Rank Matrix Decomposition

Abstract

We present a natural generalization of the recent low rank + sparse matrix decomposition and consider the decomposition of matrices into components of multiple scales. Such decomposition is well motivated in practice as data matrices often exhibit local correlations in multiple scales. Concretely, we propose a multi-scale low rank modeling that represents a data matrix as a sum of block-wise low rank matrices with increasing scales of block sizes. We then consider the inverse problem of decomposing the data matrix into its multi-scale low rank components and approach the problem via a convex formulation. Theoretically, we show that under various incoherence conditions, the convex program recovers the multi-scale low rank components either exactly or approximately. Practically, we provide guidance on selecting the regularization parameters and incorporate cycle spinning to reduce blocking artifacts. Experimentally, we show that the multi-scale low rank decomposition provides a more intuitive decomposition than conventional low rank methods and demonstrate its effectiveness in four applications, including illumination normalization for face images, motion separation for surveillance videos, multi-scale modeling of the dynamic contrast enhanced magnetic resonance imaging and collaborative filtering exploiting age information.

I. INTRODUCTION

Signals and systems often exhibit different structures at different scales. Such multi-scale structure has inspired a wide variety of multi-scale signal transforms, such as wavelets [1], curvelets [2] and multi-scale pyramids [3], that can represent natural signals compactly. Moreover, their ability to compress signal information into a few significant coefficients has made multi-scale signal transforms valuable beyond compression and are now commonly used in signal reconstruction applications, including denoising [4], compressed sensing [5], [6], and signal separation [7]–[9]. By now, multi-scale modeling is associated with many success stories in engineering applications.

On the other hand, low rank methods are commonly used instead when the signal subspace needs to be estimated as well. In particular, low rank methods have seen great success in applications, such as biomedical imaging [10], face recognition [11] and collaborative filtering [12], that require exploiting the global data correlation to recover the signal subspace and compactly represent the signal at the same time. Recent convex relaxation techniques [13] have further enabled low rank model to be adaptable to various signal processing tasks, including matrix completion [14], system identification [15] and phase retrieval [16], making low rank methods ever more attractive.

In this paper, we present a multi-scale low rank matrix decomposition method that incorporates multi-scale structures with low rank methods. The additional multi-scale structure allows us to obtain a more accurate and compact signal representation than conventional low rank methods whenever the signal exhibits multi-scale structures (see Figure 1). To capture data correlation at multiple scales, we model our data matrix as a sum of block-wise low rank matrices with increasing scales of block sizes (more detail in Section II) and consider the inverse problem of decomposing the matrix into its multi-scale components. Since we do not assume an explicit basis model, multi-scale low rank decomposition also prevents modeling errors or basis mismatch that are commonly seen with multi-scale signal transforms. In short, our proposed multi-scale low rank decomposition inherits the merits from both multi-scale modeling and low rank matrix decomposition.

Fig. 1.

An example of our proposed multi-scale low rank decomposition compared with other low rank methods. Each blob in the input matrix is a rank-1 matrix constructed from an outer product of hanning windows. Only the multi-scale low rank decomposition exactly separates the blobs to their corresponding scales and represents each blob as compactly as possible.

Leveraging recent convex relaxation techniques, we propose a convex formulation to perform the multi-scale low rank matrix decomposition. We provide a theoretical analysis in Section V that extends the rank-sparsity incoherence results in Chandrasekaran et al. [17]. We show that the proposed convex program decomposes the data matrix into its multi-scale components exactly under a deterministic incoherence condition. In addition, in Section VI, we provide a theoretical analysis on approximate multi-scale low rank matrix decomposition in the presence of additive noise that extends the work of Agarwal et al. [18].

A major component of this paper is to introduce the proposed multi-scale low rank decomposition with emphasis on its practical performance and applications. We provide practical guidance on choosing regularization parameters for the convex method in Section IV and describe heuristics to perform cycle spinning [19] to reduce blocking artifacts in Section IX. In addition, we applied the multi-scale low rank decomposition on real datasets and considered four applications of the multi-scale low rank decomposition: illumination normalization for face images, motion separation for surveillance videos, compact modeling of the dynamic contrast enhanced magnetic resonance imaging and collaborative filtering exploiting age information. (See Section X for more detail). Our results show that the proposed multi-scale low rank decomposition provides intuitive multi-scale decomposition and compact signal representation for a wide range of applications.

Related work

Our proposed multi-scale low rank matrix decomposition draws many inspirations from recent developments in rank minimization [13], [14], [18], [20]–[24]. In particular, the multi-scale low rank matrix decomposition is a generalization of the low rank + sparse decomposition proposed by Chandrasekaran et al. [17] and Candès et al. [25]. Our multi-scale low rank convex formulation also fits into the convex demixing framework proposed by McCoy et al. [26]–[28], who studied the problem of demixing components via convex optimization. The proposed multi-scale low rank decomposition can be viewed as a concrete and practical example of the convex demixing problem. However, their theoretical analysis assumes that each component is randomly oriented with respect to each other, and does not apply to our setting, where we observe the direct summation of the components. Bakshi et al. [29] proposed a multi-scale principal component analysis by applying principal component analysis on wavelet transformed signals, but such method implicitly constrains the signal to lie on a predefined wavelet subspace. Various multi-resolution matrix factorization techniques [30], [31] were proposed to greedily peel off components of each scale by recursively applying matrix factorization. One disadvantage of these factorization methods is that it is not straightforward to incorporate them with other reconstruction problems as models. Similar multi-scale modeling using demographic information was also used in collaborative filtering described in Vozalis and Margaritis [32].

II. MULTI-SCALE LOW RANK MATRIX MODELING

In this section, we describe the proposed multi-scale low rank matrix modeling in detail. To concretely formulate the model, we assume that we can partition the data matrix of interest Y into different scales. Specifically, we assume that we are given a multi-scale partition ${\left\{P_i\right\}}_{i=1}^L$ of the indices of an M × N matrix, where each block b in P_i is an order magnitude larger than the blocks in the previous scale P_i−1. Such multi-scale partition can be naturally obtained in many applications. For example in video processing, a multi-scale partition can be obtained by decimating both space and time dimensions. Figures 2 and 4 provide two examples of a multi-scale partition, the first one with decimation along two dimensions and the second one with decimation along one dimension. In Section X, we provide practical examples on creating these multi-scale partitions for different applications.

Fig. 2.

Illustration of a multi-scale matrix partition and its associated multi-scale low rank modeling. Since the zero matrix is a matrix with the least rank, our multi-scale modeling naturally extends to sparse matrices as 1 × 1 low rank matrices.

Fig. 4.

Illustration of another multi-scale matrix partition and its associated multi-scale low rank modeling. Here, only the vertical dimension of the matrix is decimated. Since a 1 × N matrix is low rank if and only if it is zero, our multi-scale modeling naturally extends to group sparse matrices.

To easily transform between the data matrix and the block matrices, we then consider a block reshape operator R_b(X) that extracts a block b from the full matrix X and reshapes the block into an m_i × n_i matrix (Figure 3), where m_i × n_i is the ith scale block matrix size determined by the user.

Fig. 3.

Illustration of the block reshape operator R_b. R_b extracts block b from the full matrix and reshapes it into an m_i × n_i matrix. Its adjoint operator $R_b^{\mathrm{T}}$ takes an m_i × n_i matrix and embeds it into a full-size zero matrix.

Given an M × N input matrix Y and its corresponding multi-scale partition and block reshape operators, we propose a multi-scale low rank modeling that models the M × N input matrix Y as a sum of matrices ${\sum }_{i=1}^L{X}_i$, in which each X_i is block-wise low rank with respect to its partition P_i that is,

$$\begin{array}{l}Y={\displaystyle \sum _{i=1}^L{X}_i}\\ X_i={\displaystyle \sum _{b\in P_i}{R}_b^{\top }\left(U_b{S}_b{V}_b^{\top }\right)}\end{array}$$

where U_b, S_b, and V_b are matrices with sizes m_i ×r_b, r_b × r_b and n_i × r_b respectively and form the rank-r_b reduced SVD of R_b(X_i). Note that when the rank of the block matrix R_b(X_i) is zero, we have {U_b, S_b, V_b} as empty matrices, which do not contribute to X_i. Figure 2 and 4 provide illustrations of two kinds of modeling with their associated partitions.

By constraining each block matrices to be of low rank, the multi-scale low rank modeling captures the notion that some nearby entries are more similar to each other than global entries in the data matrix. We note that the multi-scale low rank modeling is a generalization of the low rank + sparse modeling proposed by Chandrasekaren et al. [17] and Candès et al. [25]. In particular, the low rank + sparse modeling can be viewed as a 2-scale low rank modeling, in which the first scale has block size 1 × 1 and the second scale has block size M × N. By adding additional scales between the sparse and globally low rank matrices, the multi-scale low rank modeling can capture locally low rank components that would otherwise need many coefficients to represent for low rank + sparse.

Given a data matrix Y that fits our multi-scale low rank model, our goal is to decompose the data matrix Y to its multi-scale components ${\left\{X_i\right\}}_{i=1}^L$. The ability to recover these multi-scale components is beneficial for many applications and allows us to, for example, extract motions at multiple scales in surveillance videos (Section X). Since there are many more parameters to be estimated than the number of observations, it is necessary to impose conditions on X_i. In particular, we will exploit the fact that each block matrix is low rank via a convex program, which will be described in detail in section III.

A. Multi-scale low rank + noise

Before moving to the convex formulation, we note that our multi-scale matrix modeling can easily account for data matrices that are corrupted by additive white Gaussian noise. Under the multi-scale low rank modeling, we can think of the additive noise matrix as the largest scale signal component and is unstructured in any local scales. Specifically if we observe instead the following

$$Y={\displaystyle \sum _{i=1}^L{X}_i+X_{\mathcal{Z}}}$$ (1)

where X_Z is an independent and identically distributed Gaussian noise matrix. Then we can define a reshape operator R_Z that reshapes the entire matrix into an MN × 1 vector and the resulting matrix fits exactly to our multi-scale low rank model with L+1 scales. This incorporation of noise makes our model flexible in that it automatically provides a corresponding convex relaxation, a regularization parameter for the noise matrix and allows us to utilize the same iterative algorithm to solve for the noise matrix. Figure 5 provides an example of the noisy multi-scale low rank matrix decomposition.

Fig. 5.

An example of the multi-scale low rank decomposition in the presense of additive Gaussian noise by solving the convex program (2).

III. PROBLEM FORMULATION AND CONVEX RELAXATION

Given a data matrix Y that fits the multi-scale low rank model, our goal is to recover the underlying multi-scale components ${\left\{X_i\right\}}_{i=1}^L$ using the fact that X_i is block-wise low rank. Ideally, we would like to obtain a multi-scale decomposition with the minimal block matrix rank and solve a problem similar to the following form:

$$\begin{array}{l}\stackrel{\text{minimize}}{X_1,\dots ,X_L}{\displaystyle \sum _{i=1}^L{\displaystyle \sum _{b\in P_i}\text{rank}\left(R_b\left(X_i\right)\right)}}\\ \text{Subject to}Y={\displaystyle \sum _{i=1}^L{X}_i}\end{array}$$

However, each rank minimization for each block is combinatorial in nature. In addition, it is not obvious whether the direct summation of ranks is a correct formulation as a 1-sparse matrix and a rank-1 matrix should intuitively not carry the same cost. Hence, the above non-convex problem is not a practical formulation to obtain the multi-scale decomposition.

Recent development in convex relaxations suggests that rank minimization problems can often be relaxed to a convex program via nuclear norm relaxation [13], [23], while still recovering the optimal solution to the original problem. In particular, Chandrasekaren et al. [17] and Candès et al., [25] showed that a low rank + sparse decomposition can be relaxed to a convex program by minimizing a nuclear norm + ℓ1-norm objective as long as the signal constituents are incoherent with respect to each other. In addition, Candès et al., [25] showed that the regularization parameters for sparsity and low rank should be related by the square root of the matrix size. Hence, there is hope that, along the same line, we can perform the multi-scale low rank decomposition exactly via a convex formulation.

Concretely, let us define ‖·‖_nuc to be the nuclear norm, the sum of singular values, and ‖·‖_msv be the maximum singular value norm. For each scale i, we consider the block-wise nuclear norm to be the convex surrogate for the block-wise ranks and define ‖·‖_(i) the block-wise nuclear norm for the ith scale as

$$\Vert ·{\Vert }_{\left(i\right)}={\displaystyle \sum _{b\in P_i}\Vert R_b(·){\Vert }_{\text{nuc}}}$$

Its associated dual norm $\Vert ·{\Vert }_{\left(i\right)}^{\ast }$ is then given by

$$\Vert ·{\Vert }_{\left(i\right)}^{\ast }=\underset{b\in P_i}{\text{max}}\Vert R_b(·){\Vert }_{\text{msv}}$$

which is the maximum of all block-wise maximum singular values.

We then consider the following convex relaxation for our multi-scale low rank decomposition problem:

$$\begin{array}{l}\underset{X_1,\dots ,X_L}{\text{minimize}}{\displaystyle \sum _{i=1}^L{\mathrm{\lambda }}_i\Vert X_i{\Vert }_{\left(i\right)}}\\ \text{subject}\text{to}Y={\displaystyle \sum _{i=1}^L{X}_i}\end{array}$$ (2)

where ${\left\{{\mathrm{\lambda }}_i\right\}}_{i=1}^L$ are the regularization parameters and their selection will be described in detail in section IV.

Our convex formulation is a natural generalization of the low rank + sparse convex formulation [17], [25]. With the two sided matrix partition (Fig. 2), the nuclear norm applied to the 1 × 1 blocks becomes the element-wise ℓ1-norm and the norm for the largest scale is the nuclear norm. With the one sided matrix partition (Fig. 4), the nuclear norm applied to 1 × N blocks becomes the group-sparse norm and can be seen as a generalization of the group sparse + low rank decomposition [21]. If we incorporate additive Gaussian noise in our model as described in Section II-A, then we have a nuclear norm applied to an MN × 1 vector, which is equivalent to the Frobenius norm.

One should hope that the theoretical conditions from low rank + sparse can be generalized rather seamlessly to the multi-scale counterpart. Indeed, in Section V, we show that the core theoretical guarantees in the work of Chandrasekaren et al. [17] on exact low rank + sparse decomposition can be generalized to the multi-scale setting. In section VI, we show that the core theoretical guarantees in the work of Agarwal et al. [18] on noisy matrix decomposition can be generalized to the multi-scale setting as well to provide approximate decomposition guarentees.

IV. GUIDANCE ON CHOOSING REGULARIZATION PARAMETERS

In this section, we provide practical guidance on selecting the regularization parameters ${\left\{{\mathrm{\lambda }}_i\right\}}_{i=1}^L$. Selecting the regularization parameters ${\left\{{\mathrm{\lambda }}_i\right\}}_{i=1}^L$ is crucial for the convex decomposition to succeed, both theoretically and practically. While theoretically we can establish criteria on selecting the regularization parameters (see Section V and VI), such parameters are not straightforward to calculate in practice as it requires properties of the signal components ${\left\{X_i\right\}}_{i=1}^L$ before the decomposition.

To select the regularization parameters ${\left\{{\mathrm{\lambda }}_i\right\}}_{i=1}^L$ in practice, we follow the suggestions from Wright et al. [33] and Fogel et al. [34], and set each regularization parameter λ_i to be the Gaussian complexity of each norm‖·‖, which is defined the expectation of the dual norm of random Gaussian matrix:

$${\mathrm{\lambda }}_i~E\left[\Vert G{\Vert }_{\left(i\right)}^{\ast }\right]$$ (3)

where ∼ denotes equality up to some constant and G is a unit-variance independent and identically distributed random Gaussian matrix.

The resulting expression for the Gaussian complexity is the maximum singular value of a random Gaussian matrix, which has been studied extensively by Bandeira and Handel [35]. The recommended regularization parameter for scale i is given by

$$\boxed{{\mathrm{\lambda }}_i~\sqrt{m_i}+\sqrt{n_i}+\sqrt{\text{log}\left(\frac{MN}{\text{max}\left(m_i,n_i\right)}\right)}}$$ (4)

For the sparse matrix scale with 1 × 1 block size, ${\mathrm{\lambda }}_i~\sqrt{\text{log}\left(MN\right)}$ and for the globally low rank scale with M × N block size, ${\mathrm{\lambda }}_i~\sqrt{M}+\sqrt{N}$. Hence this regularization parameter selection is consistent with the ones recommended for low rank + sparse decomposition by Candès et al. [25], up to a log factor. In addition, for the noise matrix with MN × 1 block size, ${\mathrm{\lambda }}_i~\sqrt{MN}$, which has similar scaling as in square root LASSO [36]. In practice, we found that the suggested regularization parameter selection allows exact multi-scale decomposition when the signal model is matched (for example Figure 1) and provides visually intuitive decomposition for real datasets.

For approximate multi-scale low rank decomposition in the presence of additive noise, some form of theoretical guarantees for the regularization selection can be found in our analysis in Section VI. In particular, we show that if the regularization parameters λ_i is larger than the Gaussian complexity of $\Vert ·{\Vert }_{\left(i\right)}^{\ast }$ in addition to some “spikiness” parameters, then the error between recovered decomposition and the ground truth ${\left\{X_i\right\}}_{i=1}^L$ is bounded by the block-wise matrix rank.

V. THEORETICAL ANALYSIS FOR EXACT DECOMPOSITION

In this section, we provide a theoretical analysis of the proposed convex formulation and show that if ${\left\{X_i\right\}}_{i=1}^L$ satisfies a deterministic incoherence condition, then the proposed convex formulation (2) recovers ${\left\{X_i\right\}}_{i=1}^L$ from Y exactly.

Our analysis follows similar arguments taken by Chandrasekaren et al. [17] on low rank + sparse decomposition and generalizes them to the proposed multi-scale low rank decomposition. Before showing our main result (Theorem V.1), we first describe the subgradients of our objective function (Section V-A) and define a coherence parameter in terms of the block-wise row and column spaces (Section V-B).

A. Subdifferentials of the block-wise nuclear norms

To characterize the optimality of our convex problem, we first look at the subgradients of our objective function. We recall that for any matrix X with {U, S, V} as its reduced SVD representation, the subdifferential of ‖·‖_nuc at X is given by [23], [37],

$$\partial \Vert X{\Vert }_{\text{nuc}}=\left\{U{V}^{\top }+W:W\text{and}X\text{have orthogonal row and column spaces and}\Vert W{\Vert }_{\text{msv}}\le 1\right\}$$

Now recall that we define the block-wise nuclear norm to be $\Vert ·{\Vert }_{\left(i\right)}={\sum }_{b\in P_i}\Vert R_b(·){\Vert }_{\text{nuc}}$. Then using the chain rule and the fact that $R_b\left(X_i\right)=U_b{S}_b{V}_b^{\top }$, we obtain an expression for the subdifferential of ‖·‖_(i) at X_i as follows:

$$\partial \Vert X_i{\Vert }_{\left(i\right)}=\left\{{\displaystyle \sum _{b\in P_i}{R}_b^{\top }\left(U_b{V}_b^{\top }+W_b\right):W_b\text{and}{R}_b\left(X_i\right)\text{have orthogonal row and column spaces and}\Vert W_b{\Vert }_{\text{msv}}\le 1}\right\}$$

To simplify our notation, we define $E_i={\displaystyle {\sum }_{b\in P_i}{R}_b^{\top }\left(U_b{V}_b^{\top }\right)}$ and T_i to be a vector space that contains matrices with the same block-wise row spaces or column spaces as X_i, that is,

$$T_i=\left\{{\displaystyle \sum _{b\in P_i}{R}_b^{\top }\left(U_b{X}_b^{\top }+Y_b{V}_b^{\top }\right):X_b\in {\mathrm{ℂ}}^{n_i\times r_i},Y_b\in {\mathrm{ℂ}}^{m_i\times r_i}}\right\}$$

where m_i ×n_i is the size of the block matrices for scale i and r_b is the matrix rank for block b. Then, the subdifferential of each ‖·‖_(i) at X_i can be compactly represented as,

$$\partial {\Vert X_i\Vert }_{\left(i\right)}=\left\{E_i+W_i:W_i\in T_i^{\perp }\text{and}\Vert W_i{\Vert }_{\left(i\right)}^{\ast }\le 1\right\}$$

We note that E_i can be thought of as the “sign” of the matrix X_i, pointing toward the principal components, and, in the case of the sparse scale, is exactly the sign of the entries.

In the rest of the section, we will be interested in projecting a matrix X onto T_i, which can be performed with the following operation:

$${\mathcal{P}}_{T_i}\left(X\right)={\displaystyle \sum _{b\in P_i}{R}_b^{\top }\left(U_b{U}_b^{\top }{R}_b\left(X\right)+R_b\left(X\right)V_b{V}_b^{\top }-U_b{U}_b^{\top }{R}_b\left(X\right)V_b{V}_b^{\top }\right)}$$

Similarly, to project a matrix X onto the orthogonal complement of T_i, we can apply the following operation:

$${\mathcal{P}}_{T_i^{\perp }}\left(X\right)={\displaystyle \sum _{b\in P_i}{R}_b^{\top }\left(\left(I-U_b{U}_b^{\top }\right)R_b\left(X\right)\left(I-V_b{V}_b^{\top }\right)\right)}$$

where I is an appropriately sized identity matrix.

B. Incoherence

Following Chandrasekaren et al. [17], we consider a deterministic measure of incoherence through the block-wise column and row spaces of X_i. Concretely, we define the coherence parameter for the jth scale signal component X_j with respect to the ith scale to be the following:

$${\mu }_{ij}=\underset{N_j\in T_j,\Vert N_j{\Vert }_{\left(j\right)}^{\ast }\le 1}{\text{max}}\Vert N_j{\Vert }_{\left(i\right)}^{\ast }$$ (5)

Using μ_ij as a measure of incoherence, we can quantitatively say that the jth scale signal component is incoherent with respect to the ith scale if μ_ij is small. In the case of low rank + sparse, Chandrasekaren et al. [17] provides excellent description of the concepts behind the coherence parameters. We refer the reader to their paper for more detail.

C. Main Result

Given the above definition of incoherence, the following theorem states our main result for exact multi-scale low rank decomposition:

Theorem V.1

If we can choose regularization parameters ${\left\{{\mathrm{\lambda }}_i\right\}}_{i=1}^L$ such that

$${\displaystyle \sum _{j\ne i}{\mu }_{\mathit{ij}}\frac{{\mathrm{\lambda }}_j}{{\mathrm{\lambda }}_i}<\frac{1}{2},\text{for}i=1,\dots ,L}$$

then ${\left\{X_i\right\}}_{i=1}^L$ is the unique optimizer of the proposed convex problem (2).

In particular when the number of scales L = 2, the condition on {μ₁₂, μ₂₁} reduces to μ₁₂μ₂₁ < 1/4 and the condition on {λ₁, λ₂} reduces to 2μ₁₂ < λ₁/λ₂ < 1/(2μ₂₁), which is in similar form as Theorem 2 in Chandrasekaren et al. [17].

The proof for the above theorem is given in Appendix A.

VI. THEORETICAL ANALYSIS FOR APPROXIMATE DECOMPOSITION

In this section, we provide a theoretical analysis for approximate multi-scale low rank decomposition when the measurement is corrupted by additive noise as described in Section II-A. Our result follows arguments from Agarwal et al. [18] on noisy 2-scale matrix decomposition and extends it to the multi-scale setting.

Instead of using the incoherence parameter μ_ij defined for the exact decomposition analysis in Section V, we opt for a weaker characterization of incoherence between scales for approximate decomposition, studied in Agarwal et al. [18]. Concretely, we consider spikiness parameters α_ij between the jth signal component X_j and ith scale norm ‖·‖_(i) such that,

$${\alpha }_{\mathit{ij}}=\Vert X_j{\Vert }_{\left(i\right)}^{\ast }$$

for each j ≠ i. Hence, if α_ij is small, we say X_j is not spiky with respect to the ith norm.

For analysis purpose, we also impose the constraints $\Vert X_j{\Vert }_{\left(i\right)}^{\ast }\le {\alpha }_{\mathit{ij}}$in the convex program. That is, we consider the solution from the following convex program:

$$\underset{X_1,\dots ,X_L,X_{\mathcal{Z}}}{\text{minimize}}{\displaystyle \sum _{i=1}^L{\mathrm{\lambda }}_i\Vert X_i{\Vert }_{\left(i\right)}+{\mathrm{\lambda }}_{\mathcal{Z}}\Vert X_{\mathcal{Z}}{\Vert }_{\text{fro}}}$$ (6)

$$\begin{array}{l}\text{subject to}Y={\displaystyle \sum _{i=1}^L{X}_i+X_{\mathcal{Z}}}\\ \Vert X_j{\Vert }_{\left(i\right)}^{\ast }\le {\alpha }_{\mathit{ij}}\text{for}j\ne i\end{array}$$ (7)

We emphasize that the additional constraints (7) are imposed only for the purpose of theoretical analysis and are not imposed in our experimental results. In particular, for our simulation example in Figure 5, the minimizer of the convex program (2), using the recommended regularization parameters in Section IV, satisfied the constraints (7) even when the constraints were not imposed.

Let us define ${\left\{{\mathrm{\Delta }}_i\right\}}_{i=1}^L$ and Δ_Z to be the errors between the ground truth components ${\left\{X_i\right\}}_{i=1}^L$ and X_Z and the minimizers of convex program (6). Then, equivalently, we can denote ${\left\{X_i+{\mathrm{\Delta }}_i\right\}}_{i=1}^L$ and X_Z + Δ_Z as the minimizers of the convex program (6). The following theorem states our main result for approximate decomposition.

Theorem VI.1

If we choose ${\left\{{\mathrm{\lambda }}_i\right\}}_{i=1}^L$ such that

$${\mathrm{\lambda }}_i\ge 2{\mathrm{\lambda }}_{\mathcal{Z}}\frac{\Vert X_{\mathcal{Z}}{\Vert }_{\left(i\right)}^{\ast }}{\Vert X_Z{\Vert }_{\text{fro}}}+{\displaystyle \sum _{j\ne i}2{\alpha }_{\mathit{ij}}}$$ (8)

and λ_Z such that

$${\mathrm{\lambda }}_{\mathcal{Z}}\ge \sqrt{64{\displaystyle \sum _{i=1}^L{\mathrm{\lambda }}_i^2}{\displaystyle \sum _{b\in P_i}{r}_b}}$$ (9)

then the error is bounded by

$${\displaystyle \sum _{i=1}^L\Vert {\mathrm{\Delta }}_i{\Vert }_{\text{fro}}}\lesssim \frac{\Vert X_{\mathcal{Z}}{\Vert }_{\text{fro}}}{{\mathrm{\lambda }}_{\mathcal{Z}}}{\displaystyle \sum _{i=1}^L{\mathrm{\lambda }}_i}\sqrt{{\displaystyle \sum _{b\in P_i}{r}_b}}$$

where ≲ denotes inequality up to a universal constant.

Hence, when the spikiness parameters are negligible and X_Z = σG, where G is an independent, identically distributed Gaussian noise matrix with unit variance and σ is the noise standard deviation, choosing ${\mathrm{\lambda }}_Z~E\left[\Vert G{\Vert }_{\text{fro}}\right]~\sqrt{MN}$ and ${\mathrm{\lambda }}_i~E\left[\Vert G{\Vert }_{\left(i\right)}^{\ast }\right]~\sqrt{m_i}+\sqrt{n_i}+\sqrt{\text{log}\left(MN/\text{max}\left(m_i,n_i\right)\right)}$ ensures the condition is satisfied with high probability. This motivates the recommended regularization selection in Section IV.

The proof for the above theorem is given in Appendix B and follows arguments from Agarwal et al. [18] on noisy matrix decomposition and Belloni et al. [36] on square root LASSO.

VII. AN ITERATIVE ALGORITHM FOR SOLVING THE MULTI-SCALE LOW RANK DECOMPOSITION

In the following, we will derive an iterative algorithm that solves for the multi-scale low rank decomposition via the Alternating Direction of Multiple Multipliers (ADMM) [38]. While the proposed convex formulation (2) can be formulated into a semi-definite program, first-order iterative methods are commonly used when solving for large datasets for their computational efficiency and scalability. A conceptual illustration of the algorithm is shown in Figure 6.

Fig. 6.

A conceptual illustration of how to obtain a multi-scale low rank decomposition. First, we extract each block from the input matrix and perform a thresholding operation on its singular value to recover the significant components. Then, we subtract these significant components from our input matrix, thereby enabling the recovery of weaker, previously submerged components.

To formally obtain update steps using ADMM, we first formulate the problem into the standard ADMM form with two separable objectives connected by an equality constraint,

$$\begin{array}{cc}\underset{X_i,Z_i}{\text{minimize}}& \mathrm{I}\left\{Y={\displaystyle \sum _{i=1}^L{X}_i}\right\}+{\displaystyle \sum _{i=1}^L{\mathrm{\lambda }}_i\Vert Z_i{\Vert }_{\left(i\right)}}\\ \text{subject to}& X_i=Z_i\end{array}$$ (10)

where I{·} is the indicator function.

To proceed, we then need to obtain the proximal operators [39] for the two objective functions $\mathbf{I}\left\{Y={\displaystyle {\sum }_{i=1}^L{X}_i}\right\}$ and ${\displaystyle {\sum }_{i=1}^L{\mathrm{\lambda }}_i\Vert Z_i{\Vert }_{\left(i\right)}}$. For the data consistency objective $\mathbf{I}\left\{Y={\displaystyle {\sum }_{i=1}^L{X}_i}\right\}$, the proximal operator is simply the projection operator to the set. To obtain the proximal operator for the multi-scale nuclear norm objective ${\displaystyle {\sum }_{i=1}^L{\mathrm{\lambda }}_i\Vert X_i{\Vert }_{\left(i\right)}}$, we first recall that the proximal operator for the nuclear norm ‖X‖_nuc with parameter λ is given by the singular value soft-threshold operator [23],

$${\text{SVT}}_{\mathrm{\lambda }}\left(X\right)=U\text{max}\left(\sum -\mathrm{\lambda },0\right)V^{\top }$$ (11)

Since we defined the block-wise nuclear norm for each scale i as ${\displaystyle {\sum }_{b\in P_i}\Vert R_b(·){\Vert }_{\text{nuc}}}$, the norm is separable with respect to each block and its proximal function with parameter λ_i is given by the block-wise singular value soft-threshold operator,

$${\text{BLOCKSVT}}_{{\mathrm{\lambda }}_i}\left(X\right)={\displaystyle \sum _{b\in P_i}{R}_b^{\top }\left({\text{SVT}}_{{\mathrm{\lambda }}_i}\left(R_b\left(X\right)\right)\right)}$$ (12)

which simply extracts every blocks in the matrix, performs singular value thresholding and puts the blocks back to the matrix. We note that for 1 × 1 blocks, the block-wise singular value soft-threshold operator reduces to the element-wise soft-threshold operator and for 1 × N blocks, the block-wise singular soft-threshold operator reduces to the joint soft-threshold operator.

Putting everything together and invoking the ADMM recipe [38], we have the following algorithm to solve our convex multi-scale low rank decomposition (2):

$$\boxed{\begin{array}{l}{X}_i\leftarrow \left(Z_i-U_i\right)+\frac{1}{L}\left(Y-{\displaystyle \sum _{i=1}^L\left(Z_i-U_i\right)}\right)\\ Z_i\leftarrow {\text{BLOCKSVT}}_{{\mathrm{\lambda }}_i/\rho }\left(X_i+U_i\right)\\ U_i\leftarrow U_i-\left(Z_i-X_i\right)\end{array}}$$ (13)

where ρ is the ADMM parameter that only affects the convergence rate of the algorithm.

The resulting ADMM update steps are similar in essence to the intuitive update steps in Figure 6, and alternates between data consistency and enforcing multi-scale low rank. The major difference of ADMM is that it adds a dual update step with U_i, which bridges the two objectives and ensures the convergence to the optimal solution. Under the guarantees of ADMM, in the limit of iterations, X_i and Z_i converge to the optimal solution of the convex program (2) and U_i converges to a scaled version of the dual variable. In practice, we found that ∼ 1000 iterations are sufficient without any visible change for imaging applications. Finally, we note that because the proximal operator for the multi-scale nuclear norm is computationally simple, other proximal operator based algorithms [39] can also be used.

VIII. COMPUTATIONAL COMPLEXITY

Given the iterative algorithm (13), one concern about the multi-scale low rank decomposition might be that it is significantly more computationally intensive than other low rank methods as we have many more SVD’s and variables to compute for. In this section, we show that because we decimate the matrices at each scale geometrically, the theoretical computational complexity of the multi-scale low rank decomposition is similar to other low rank decomposition methods, such as the low rank + sparse decomposition.

For concreteness, let us consider the multi-scale partition with two-sided decimation shown in Figure 2 and have block sizes m_i = 2ⁱ⁻¹ and n_i = 2ⁱ⁻¹. Similar to other low rank methods, the SVD’s dominate the per iteration complexity for the multi-scale low rank decomposition. For an M × N matrix, each SVD costs<monospace>#flops</monospace>(M ×N SVD) = O(MN²). The per iteration complexity for the multi-scale low rank decomposition is dominated by the summation of all the SVD’s performed for each scale, which is given by,

$$\begin{array}{l}\#\text{flops}\left(M\times N\text{SVD}\right)+4\#\text{flops}\left(M/2\times N/2\text{SVD}\right)+\dots \\ =O\left(M{N}^2\right)+O\left(M{N}^2\right)/2+O\left(M{N}^2\right)/4+\dots \\ \le 2O\left(M{N}^2\right)\approx \#\text{flops}\left(M\times N\text{SVD}\right)\end{array}$$ (14)

Hence, the per-iteration computational complexity of the multi-scale low rank with two-sided decimated partition is on the order of a M × N matrix SVD. In general, one can show that the per-iteration complexity for arbitrary multi-scale partition is at most log(N) times the full matrix SVD.

While theoretically, the computation cost for small block sizes should be less than bigger block sizes, we found that in practice the computation cost for computing the small SVD’s can dominate the per-iteration computation. This is due to the overhead of copying small block matrices and calling library functions repeatedly to compute the SVD’s.

Since we are interested in thresholding the singular values and in practice many of the small block matrices are zero as shown in Section X, one trick of reducing the computation time is to quickly compute an upper bound on the maximum singular value for block matrices before the SVD’s. Then if the upper bound for the maximum singular value is less than the threshold, we know the thresholded matrix will be zero and can avoid computing the SVD. Since for any matrix X, its maximum singular value is bounded by the square root of any matrix norm on X^⊤X [40], there are many different upper bounds that we can use. In particular, we choose the maximum row norm and consider the following upper bound,

$${\sigma }_{\text{max}}\left(X\right)\le \sqrt{\underset{i}{\text{max}}{\displaystyle \sum _j\left|X_{ik}{X}_{jk}\right|}}$$ (15)

Using this upper bound, we can identify many below-the-threshold matrices before computing the SVD’s at all. In practice, we found that the above trick provides a modest speedup of 3 ∼ 5×.

IX. HEURISTICS FOR TRANSLATION INVARIANT DECOMPOSITION

Similar to wavelet transforms, one drawback of the multi-scale low rank decomposition is that it is not translation invariant, that is, shifting the input changes the resulting decomposition. In practice, this translation variant nature often creates blocking artifacts near the block boundaries, which can be visually jarring for image or video applications. One solution to remove these artifacts is to introduce overlapping partitions of the matrix so that the overall algorithm is translation invariant. However, this vastly increases both memory and computation especially for large block sizes. In the following, we will describe a cycle spinning approach that we used in practice to reduce the blocking artifacts with only slight increase in per-iteration computation.

Cycle spinning [19] has been commonly used in wavelet denoising to reduce the blocking artifacts due to the translation variant nature of the wavelet transform. To minimize artifacts, cycle spinning averages the denoised results from all possible shifted copies of the input, thereby making the entire process translation invariant. Concretely, let S be the set of all shifts possible in the target application, SHIFT_s denote the shifting operator by s, and DENOISE be the denoising operator of interest. Then the cycle spinned denoising of the input X is given by:

$$\frac{1}{\left|S\right|}{\displaystyle \sum _{s\in S}{\text{SHIFT}}_{-s}\left(\text{DENOISE}\left({\text{SHIFT}}_s\left(X\right)\right)\right)}$$ (16)

In the context of multi-scale low rank decomposition, we can make the iterative algorithm translation invariant by replacing the block-wise singular value thresholding operation in each iteration with its cycle spinning counterpart. In particular, for our ADMM update steps, we can replace the Z_i step to:

$$Z_i\leftarrow \frac{1}{\left|S\right|}{\displaystyle \sum _{s\in S}{\text{SHIFT}}_{-s}\left({\text{BLOCKSVT}}_{{\mathrm{\lambda }}_i/\rho }\left({\text{SHIFT}}_s\left(X_i+U_i\right)\right)\right)}$$ (17)

To further reduce computation, we perform random cycle spinning in each iteration as described in Figueiredo et al. [41], in which we randomly shifts the input, performs block-wise singular value thresholding and then unshifts back:

$$Z_i\leftarrow {\text{SHIFT}}_{-s}\left({\text{BLOCKSVT}}_{{\mathrm{\lambda }}_i/\rho }\left({\text{SHIFT}}_s\left(X_i+U_i\right)\right)\right)$$ (18)

where s is randomly chosen from the set S.

Using random cycle spinning, blocking artifacts caused by thresholding are averaged over iterations and in practice, reduces distortion significantly. Figure 7 shows an example of the multi-scale low rank decomposition with and without random cycle spinning applied on a simulated data that does not fall on the partition grid. The decomposition with random cycle spinning vastly reduces blocking artifacts that appeared in the one without random cycle spinning.

Fig. 7.

An example of the multi-scale low rank decomposition with and without random cycle spinning. Each blob in the input matrix Y is a rank-1 matrix constructed from an outer product of hanning windows and is placed at random positions. Blocking artifacts can be seen in the decomposition without random cycle spinning while vastly diminished in the random cycle spinned decomposition.

X. APPLICATIONS

To test for practical performance, we applied the multi-scale low rank decomposition on four different real datasets that are conventionally used in low rank modeling: illumination normalization for face images (Section X-A), motion separation for surveillance videos (Section X-B), multi-scale modeling of dynamic contrast enhanced magnetic resonance imaging (Section X-C) and collaborative filtering exploiting age information (Section X-D). We compared our proposed multi-scale low rank decomposition with low rank + sparse decomposition for the first three applications and with low rank matrix completion for the last application. Randomly cycle spinning was used for multi-scale low rank decomposition for all of our experiments. Regularization parameters λ_i were chosen exactly as $\sqrt{m_i}+\sqrt{n_i}+\sqrt{\text{log}\left(MN/\text{max}\left(m_i,n_i\right)\right)}$ for multi-scale low rank and max(m_i, n_i) for low rank + sparse decomposition. Our simulations were written in the C programming language and ran on a 20-core Intel Xeon workstation. Some results are best viewed in video format, which are available as supplementary materials.

In the spirit of reproducible research, we provide a software package (in C and partially in MATLAB) to reproduce most of the results described in this paper. The software package can be downloaded from: https://github.com/frankong/multi_scale_low_rank.git

A. Multi-scale Illumination Normalization for Face Recognition Pre-processing

Face recognition algorithms are sensitive to shadows or occlusions on faces. In order to obtain the best possible performance for these algorithms, it is desired to remove illumination variations and shadows on the face images. Low rank modeling are often used to model faces and is justified by approximating faces as convex Lambertian surfaces [11].

Low rank + sparse decomposition [25] was recently proposed to capture uneven illumination as sparse errors and was shown to remove small shadows while capturing the underlying faces as the low rank component. However, most shadows are not sparse and contain structure over different lighting conditions. Here, we propose modeling shadows and illumination changes in different face images as block-low rank as illumination variations are spatially correlated in multiple scales.

We considered face images from the Yale B face database [42]. Each face image was of size 192×168 with 64 different lighting conditions. The images were then reshaped into a 32, 256 × 64 matrix and both multi-scale low rank and low rank + sparse decomposition were applied on the data matrix. For low rank + sparse decomposition, we found that the best separation result was obtained when each face image was normalized to the maximum value. For multi-scale low rank decomposition, the original unscaled image was used. Only the space dimension was decimated as we assumed there was no ordering in different illumination conditions. The multi-scale matrix partition can be visualized as in Figure 4.

Figure 8 shows one of the comparison results. Multi-scale low rank decomposition recovered almost shadow-free faces. In particular, the sparkles in the eyes were represented in the 1 × 1 block size and the larger illumination changes were represented in bigger blocks, thus capturing most of the uneven illumination changes. In contrast, low rank + sparse decomposition could only recover from small illumination changes and still contained the larger shadows in the globally low rank component.

Fig. 8.

Multi-scale low rank versus low rank + sparse on faces with uneven illumination. Multi-scale low rank decomposition recovers almost shadow-free faces, whereas low rank + sparse decomposition can only remove some shadows.

B. Multi-scale Motion Separation for Surveillance Videos

In surveillance video processing, it is desired to extract foreground objects from the video. To be able to extract foreground objects, both the background and the foreground dynamics have to be modeled. Low rank modeling have been shown to be suitable for slowly varying videos, such as background illumination changes. In particular, if the video background only changes its brightness over time, then it can be represented as a rank-1 matrix.

Low rank + sparse decomposition [25] was proposed to foreground objects as sparse components and was shown to separate dynamics from background components. However, sparsity alone cannot capture motion compactly and often results in ghosting artifacts occurring around the foreground objects as shown in Figure 9. Since video dynamics are correlated locally at multiple scales in space and time, we propose using the multi-scale low rank modeling with two sided decimation to capture different scales of video dynamics over space and time.

Fig. 9.

Multi-scale low rank versus low rank + sparse decomposition on a surveillance video. For the multi-scale low rank, body motion is mostly captured in the 16 × 16 × 16 scale while fine-scale motion is captured in 4 × 4 × 4 scale. Background video component is captured in the globally low rank component and is almost artifact-free. Low rank + sparse decomposition exhibits ghosting artifacts as pointed by the red arrow because they are neither globally low rank or sparse.

We considered a surveillance video from Li et al. [43]. Each video frame was of size 144×176 and the first 200 frames were used. The video frames were then reshaped into a 25, 344×200 matrix and both multi-scale low rank and low rank + sparse decomposition were applied on the data matrix.

Figure 9 shows one of the results. Multi-scale low rank decomposition recovered a mostly artifact free background video in the globally low rank component whereas low rank + sparse decomposition exhibits ghosting artifact in certain segments of the video. For the multi-scale low rank decomposition, body motion was mostly captured in the 16 × 16 × 16 scale while fine-scale motion was captured in 4 × 4 × 4 scale.

C. Multi-scale Low Rank Modeling for Dynamic Contrast Enhanced Magnetic Resonance Imaging

In dynamic contrast enhanced magnetic resonance imaging (DCE-MRI), a series of images over time is acquired after a T₁ contrast agent was injected into the patient. Different tissues then exhibit different contrast dynamics over time, thereby allowing radiologists to characterize and examine lesions. Compressed sensing Magnetic Resonance Imaging [44] is now a popular research approach used in three dimensional DCE-MRI to speed up acquisition. Since the more compact we can represent the image series, the better our compressed reconstruction result becomes, an accurate modeling of the dynamic image series is desired to improve the compressed sensing reconstruction results for DCE-MRI.

When a region contains only one type of tissue, then the block matrix constructed by stacking each frame as columns will have rank 1. Hence, low rank modeling [10], and locally low rank modeling [45] have been popular models for DCE-MRI. Recently, low rank + sparse modeling [46] have also been proposed to model the static background and dynamics as low rank and sparse matrices respectively. However, dynamics in DCE-MRI are almost never sparse and often exhibit correlation across different scales. Hence, we propose using a multi-scale low rank modeling to capture contrast dynamics over multiple scales.

We considered a fully sampled dynamic contrast enhanced image data. The data was acquired in a pediatric patient with 20 contrast phases, 1×1.4×2 mm³ resolution, and 8s temporal resolution. The acquisition was performed on a 3T GE MR750 scanner with a 32-channel cardiac array using an RF-spoiled gradient-echo sequence. We considered a 2D slice of size 154×112 were then reshaped into a 17, 248×20 matrix. Both multi-scale low rank and low rank + sparse decomposition were applied on the data matrix.

Figure 10 shows one of the results. In the multi-scale low rank decomposition result, small contrast dynamics in vessels were captured in 4 × 4 blocks while contrast dynamics in the liver were captured in 16 × 16 blocks. The biggest block size captured the static tissues and interestingly the respiratory motion. Hence, different types of contrast dynamics were captured compactly in their suitable scales. In contrast, the low rank + sparse modeling could only provide a coarse separation of dynamics and static tissue, which resulted in neither truly sparse nor truly low rank components.

Fig. 10.

Multi-scale low rank versus low rank + sparse decomposition on a dynamic contrast enhanced magnetic resonance image series. For the multi-scale result, small contrast dynamics in vessels are captured in 4 × 4 blocks while contrast dynamics in the liver are captured in 16 × 16 blocks. The biggest block size captures the static tissues and interestingly the respiratory motion. In contrast, the low rank + sparse modeling could only provide a coarse separation of dynamics and static tissue, which result in neither truly sparse nor truly low rank components.

D. Multi-scale Age Grouping for Collaborative Filtering

Collaborative filtering is the task of making predictions about the interests of a user using available information from all users. Since users often have similar taste for the same item, low rank modeling is commonly used to exploit the data similarity to complete the rating matrix [14], [22], [23]. On the other hand, low rank matrix completion does not exploit the fact that users with similar demographic backgrounds have similar taste for similar items. In particular, users of similar age should have similar taste. Hence, we incorporated the proposed multi-scale low rank modeling with matrix completion by partitioning users according to their age and compared it with the conventional low rank matrix completion. Our method belongs to the general class of collaborative filtering methods that utilize demographic information [32].

To incorporate multi-scale low rank modeling into matrix completion, we change the data consistency constraint in problem (2) to ${\left[Y\right]}_{\mathit{jk}}={\left[{\displaystyle {\sum }_{i=1}^L{X}_i}\right]}_{\mathit{jk}}$ for observed jk entries, and correspondingly, the update step for ${\left\{X_i\right\}}_{i=1}^L$ in equation (13) is changed to ${\left[X_i\right]}_{jk}\leftarrow {\left[\left(Z_i-U_i\right)+\frac{1}{L}\left(Y-{\displaystyle {\sum }_{i=1}^L\left(Z_i-U_i\right)}\right)\right]}_{jk}$ for observed jk entries and ${\left[X_i\right]}_{jk}\leftarrow {\left[Z_i-U_i\right]}_{jk}$ for unobserved jk entries. We emphasize that our theoretical analysis does not cover matrix completion and the presented collaborative filtering application is mainly of empirical interest.

To compare the methods, we considered the 100K Movie-Lens dataset, in which 943 users rated 1682 movies. The resulting matrix was of size 1682 × 943, where the first dimension represented movies and the second dimension represented users. The entire matrix had 93.7% missing entries. Test data was further generated by randomly undersampling the rating matrix by 5. The algorithms were then run on the test data and root mean squared errors were calculated over all available entries. To obtain a multi-scale partition of the matrix, we sorted the users according to their age along the second dimension and partitioned them evenly into age groups.

Figure 11 shows a multi-scale low rank reconstructed user rating matrix. Using multiple scales of block-wise low rank matrices, correlations in different age groups were captured. For example, one of the scales shown in Figure 11 captures the tendency that younger users rated Star Wars higher whereas the more senior users rated Gone with the Wind higher. The multi-scale low rank reconstructed matrix achieved a root mean-squared-error of 0.9385 compared to a root mean-squared-error of 0.9552 for the low rank reconstructed matrix.

Fig. 11.

Multi-scale low rank reconstructed matrix of the 100K MovieLens dataset. The extracted signal scale component captures the tendency that younger users rated Star Wars higher whereas the more senior users rated Gone with the Wind higher.

XI. DISCUSSION

We have presented a multi-scale low rank matrix decomposition method that combines both multi-scale modeling and low rank matrix decomposition. Using a convex formulation, we can solve for the decomposition efficiently and exactly, provided that the multi-scale signal components are incoherent. We provided a theoretical analysis of the convex relaxation for exact decomposition, which extends the analysis in Chandrasekaren et al. [17], and an analysis for approximate decomposition in the presence of additive noise, which extends the analysis in Agarwal et al. [18]. We also provided empirical results that the multi-scale low rank decomposition performs well on real datasets.

We would also like to emphasize that our recommended regularization parameters empirically perform well even with the addition of noise, and hence in practice does not require manual tuning. While some form of theoretical guarantees for the regularization parameters are provided in the approximate decomposition analysis, complete theoretical guarentees are not provided, especially for noiseless situations, and would be valuable for future work.

Our experiments show that the multi-scale low rank decomposition improves upon the low rank + sparse decomposition in a variety of applications. We believe that more improvement can be achieved if domain knowledge for each applications is incorporated with the multi-scale low rank decomposition. For example, for face shadow removal, prior knowledge of the illumination angle might be able to provide a better multi-scale partition. For movie rating collaborative filtering, general demographic information and movie types can be used to construct multi-scale partitions in addition to age information.

Biographies

Frank Ong is currently pursuing the Ph.D. degree in the department of Electrical Engineering and Computer Sciences at the University of California, Berkeley. He received his B.Sc. at the same department in 2013. His research interests lie in signal processing, medical image reconstruction, compressed sensing and low rank methods.

Michael (Miki) Lustig is an Associate Professor in EECS. He joined the faculty of the EECS Department at UC Berkeley in Spring 2010. He received his B.Sc. in Electrical Engineering from the Technion, Israel Institute of Technology in 2002. He received his Msc and Ph.D. in Electrical Engineering from Stanford University in 2004 and 2008, respectively. His research focuses on medical imaging, particularly Magnetic Resonance Imaging (MRI), and very specifically, the application of compressed sensing to rapid and high-resolution MRI, MRI pulse sequence design, medical image reconstruction, inverse problems in medical imaging and sparse signal representation.

APPENDIX A PROOF OF THEOREM V.1

In this section, we provide a proof of Theorem V.1 and show that if ${\left\{X_i\right\}}_{i=1}^L$ satisfies a deterministic incoherence condition, then the proposed convex formulation (2) recovers ${\left\{X_i\right\}}_{i=1}^L$ from Y exactly. Our proof makes use of the dual certificate common in such proofs. We will begin by proving a technical lemma collecting three inequalities.

Lemma A.1

For i = 1,…, L, the following three inequalities hold,

$$\begin{array}{ll}{\Vert {\mathcal{P}}_{T_i}(X)\Vert }_{(i)}^{\ast }\le {\Vert X\Vert }_{(i)}^{\ast }\hfill & \text{for any matrix }X\hfill \end{array}$$ (19)

$$\begin{array}{ll}{\Vert {\mathcal{P}}_{T_i^{\perp }}(X)\Vert }_{(i)}^{\ast }\le {\Vert X\Vert }_{(i)}^{\ast }\hfill & \text{for any matrix }X\hfill \end{array}$$ (20)

$$\begin{array}{cc}{\Vert N_j\Vert }_{(i)}^{\ast }\le {\mu }_{\mathit{ij}}{\Vert N_j\Vert }_{(j)}^{\ast }& \text{for }j\ne \end{array}i\text{and }{N}_j\in T_j$$ (21)

Proof

To show the first inequality (19), we recall that ${\Vert X\Vert }_{(i)}^{\ast }={\text{max}}_{b\in P_i}{\Vert R_b(X)\Vert }_{\text{msv}}$. Then, using the variational representation of the maximum singular value norm, we obtain,

$$\begin{array}{c}{\Vert {\mathcal{P}}_{T_i}(X)\Vert }_{(i)}^{\ast }=\underset{b\in P_i}{\text{max}}\underset{u,\upsilon }{\text{max}}{u}^{\top }{R}_b({\mathcal{P}}_{T_i}(X))\upsilon \\ =\underset{b\in P_i}{\text{max}}\underset{\underset{\upsilon \in \text{row}(R_b(X_i))}{u\in \text{col}(R_b(X_i))\text{or}}}{\text{max}}{u}^{\top }{R}_b(X)\upsilon \\ \le \underset{b\in P_i}{\text{max}}\underset{u,\upsilon }{\text{max}}{u}^{\top }{R}_b(X)\upsilon ={\Vert X\Vert }_{(i)}^{\ast }\end{array}$$

where col and row denote the column and row spaces respectively.

Similarly, we obtain the second inequality (20):

$$\begin{array}{l}\Vert P_{T_i^{\perp }}\left(X\right){\Vert }_{\left(i\right)}^{\ast }=\underset{b\in P_i}{\text{max}}\underset{\begin{array}{l}u\in {\text{col}}^{\perp }\left(R_b\left(X_i\right)\right)\text{and}\\ \nu \in {\text{row}}^{\perp }\left(R_b\left(X_i\right)\right)\end{array}}{\text{max}}{u}^{\top }{R}_b\left(X\right)\upsilon \\ \le \underset{b\in P_i}{\text{max}}\underset{u,\nu }{\text{max}}{u}^{\top }{R}_b\left(X\right)\upsilon =\Vert X{\Vert }_{\left(i\right)}^{\ast }\end{array}$$

The third inequality (21) follows from the incoherence definition that ${\mu }_{\mathit{ij}}\ge \Vert N_j{\Vert }_{\left(i\right)}^{\ast }/\Vert N_j{\Vert }_{\left(j\right)}^{\ast }$ for any non-zero N_j.■

Next, we will show that if we can choose some parameters to “balance” the coherence between the scales, then the block-wise row/column spaces ${\left\{T_i\right\}}_{i=1}^L$ are independent, that is ${\displaystyle {\sum }_{i=1}^L{T}_i}$ is a direct sum. Consequently, each matrix N in the span of ${\left\{T_i\right\}}_{i=1}^L$ has a unique decomposition $N={\displaystyle {\sum }_{i=1}^L{N}_i}$, where N_i ∈ T_i.

Proposition A.2

If we can choose some positive parameters ${\left\{{\mathrm{\lambda }}_i\right\}}_{i=1}^L$ such that

$${\displaystyle \sum _{j\ne i}{\mu }_{\mathit{ij}}\frac{{\mathrm{\lambda }}_j}{{\mathrm{\lambda }}_i}<1,\text{for}i=1,\dots ,L}$$ (22)

then we have

$$T_i\cap {\displaystyle \sum _{j\ne i}{T}_j=\left\{0\right\},\text{for}i=1,\dots ,L}$$ (23)

In particular when L = 2, the condition on {μ₁₂,μ₂₁} reduces to μ₁₂μ₂₁ < 1, which coincides with Proposition 1 in Chandrasekaren et al. [17]. We also note that given μ_ij, we can obtain ${\left\{{\mathrm{\lambda }}_i\right\}}_{i=1}^L$ that satisfies the condition ${\displaystyle {\sum }_{j\ne i}{\mu }_{\mathit{ij}}{\mathrm{\lambda }}_j<{\mathrm{\lambda }}_i}$ by solving a linear program.

Proof

Suppose by contradiction that there exists ${\left\{{\mathrm{\lambda }}_i\right\}}_{i=1}^L$ such that ${\displaystyle {\sum }_{j\ne i}{\mu }_{\mathit{ij}}{\mathrm{\lambda }}_j/{\mathrm{\lambda }}_i<1}$, but $T_i\cap {\displaystyle {\sum }_{j\ne i}{T}_j\ne \left\{0\right\}}$ Then there exists ${\left\{N_i\in T_i\right\}}_{i=1}^L$ such that ${\displaystyle {\sum }_{i=1}^L{\mathrm{\lambda }}_i{N}_i=0}$ and not all N_i zero. But this leads to a contradiction because for i = 1,…,L,

$$\begin{array}{l}\Vert N_i{\Vert }_{\left(i\right)}^{\ast }=\Vert -{\displaystyle \sum _{j\ne i}\frac{{\mathrm{\lambda }}_j}{{\mathrm{\lambda }}_i}{N}_j{\Vert }_{\left(i\right)}^{\ast }}\\ \le {\displaystyle \sum _{j\ne i}\frac{{\mathrm{\lambda }}_j}{{\mathrm{\lambda }}_i}{\mu }_{\mathit{ij}}\Vert N_j{\Vert }_{\left(j\right)}^{\ast }}\end{array}$$ (24)

$$\le \left({\displaystyle \sum _{j\ne i}\frac{{\mathrm{\lambda }}_j}{{\mathrm{\lambda }}_i}{\mu }_{\mathit{ij}}}\right)\underset{j\ne i}{\text{max}}\Vert N_j{\Vert }_{\left(j\right)}^{\ast }$$ (25)

$$<\underset{j\ne i}{\text{max}}\Vert N_j{\Vert }_{\left(j\right)}^{\ast }$$ (26)

where we have used equation (21) for the first inequality (24), Holder’s inequality for second inequality (25) and ${\displaystyle {\sum }_{j\ne i}{\mu }_{\mathit{ij}}{\mathrm{\lambda }}_j/{\mathrm{\lambda }}_i<1}$ for the last inequality. Hence, none of ${\left\{\Vert N_i{\Vert }_{\left(j\right)}^{\ast }\right\}}_{i=1}^L$ is the largest of the set, which is a contradiction. ■

Our next theorem shows an optimality condition of the convex program (2) in terms of its dual solution.

Theorem A.3

(Lemma 4.2 [33]). ${\left\{X_i\right\}}_{i=1}^L$ is the unique minimizer of the convex program (2) if there exists a matrix Q such that for i = 1,…,L,

• 1) ${\mathcal{P}}_{T_i}\left(Q\right)={\mathrm{\lambda }}_i{E}_i$

• 2) $\Vert {\mathcal{P}}_{T_i^{\perp }}\left(Q\right){\Vert }_{\left(i\right)}^{\ast }<{\mathrm{\lambda }}_i$

Proof

Consider any non-zero perturbation ${\left\{{\mathrm{\Delta }}_i\right\}}_{i=1}^L$ to ${\left\{X_i\right\}}_{i=1}^L$ such that ${\left\{X_i+{\mathrm{\Delta }}_i\right\}}_{i=1}^L$ stays in the feasible set, that is ${\displaystyle {\sum }_{i=1}^L{\mathrm{\Delta }}_i=0}$. We will show that ${\displaystyle {\sum }_{i=1}^L{\mathrm{\lambda }}_i\Vert X_i+{\mathrm{\Delta }}_i{\Vert }_{\left(i\right)}>{\displaystyle {\sum }_{i=1}^L{\mathrm{\lambda }}_i\Vert X_i{\Vert }_{\left(i\right)}}}$.

We first decompose Δ_i into orthogonal parts with respect to T_i, that is, ${\mathrm{\Delta }}_i={\mathcal{P}}_{T_i}\left({\mathrm{\Delta }}_i\right)+{\mathcal{P}}_{T_i^{\perp }}\left({\mathrm{\Delta }}_i\right)$. We also consider a specific subgradient $G={\left[G_1\cdots G_L\right]}^{\top }$ of ${\displaystyle {\sum }_{i=1}^L{\mathrm{\lambda }}_i\Vert ·{\Vert }_{\left(i\right)}}$ at ${\left\{X_i\right\}}_{i=1}^L$ such that $\Vert {\mathcal{P}}_{T_i^{\perp }}\left(G_i\right){\Vert }_{\left(i\right)}^{\ast }\le \mathrm{\lambda }$, and $\langle {\mathcal{P}}_{T_i^{\perp }}\left({\mathrm{\Delta }}_i\right),{\mathcal{P}}_{T_i^{\perp }}\left(G_i\right)\rangle ={\mathrm{\lambda }}_i\Vert {\mathcal{P}}_{T_i^{\perp }}\left({\mathrm{\Delta }}_i\right){\Vert }_{\left(i\right)}$. Then, from the definition of subgradient and the fact that ${\displaystyle {\sum }_{i=1}^L{\mathrm{\Delta }}_i=0}$, we have,

$$\begin{array}{l}{\displaystyle \sum _{i=1}^L{\mathrm{\lambda }}_i\Vert X_i+{\mathrm{\Delta }}_i{\Vert }_{\left(i\right)}\ge {\displaystyle \sum _{i=1}^L{\mathrm{\lambda }}_i\Vert X_i{\Vert }_{\left(i\right)}+\langle {\mathrm{\Delta }}_i,G_i\rangle }}\\ ={\displaystyle \sum _{i=1}^L{\mathrm{\lambda }}_i\Vert X_i{\Vert }_{\left(i\right)}+\langle {\mathrm{\Delta }}_i,G_i\rangle -\langle {\mathrm{\Delta }}_i,Q\rangle }\end{array}$$

Applying the orthogonal decomposition with respect to T_i and using ${\mathcal{P}}_{T_i}\left(G_i\right)={\mathcal{P}}_{T_i}\left(Q\right)={\mathrm{\lambda }}_i{E}_i$, we have,

$$\begin{array}{l}{\displaystyle \sum _{i=1}^L{\mathrm{\lambda }}_i\Vert X_i+{\mathrm{\Delta }}_i{\Vert }_{\left(i\right)}\ge {\displaystyle \sum _{i=1}^L{\mathrm{\lambda }}_i\Vert X_i{\Vert }_{\left(i\right)}+\langle {\mathcal{P}}_{T_i^{\perp }}\left({\mathrm{\Delta }}_i\right),{\mathcal{P}}_{T_i^{\perp }}\left(G_i\right)\rangle }}\\ -\langle {\mathcal{P}}_{T_i^{\perp }}\left({\mathrm{\Delta }}_i\right),{\mathcal{P}}_{T_i^{\perp }}\left(Q\right)\rangle \end{array}$$

Using Holder’s inequality and the assumption for the subgradient G_i, we obtain,

$$\begin{array}{l}{\displaystyle \sum _{i=1}^L{\mathrm{\lambda }}_i\Vert X_i+{\mathrm{\Delta }}_i{\Vert }_{\left(i\right)}\ge {\displaystyle \sum _{i=1}^L{\mathrm{\lambda }}_i\Vert X_i{\Vert }_{\left(i\right)}+{\mathrm{\lambda }}_i\Vert {\mathcal{P}}_{T_i^{\perp }}\left({\mathrm{\Delta }}_i\right){\Vert }_{\left(i\right)}}}\\ -\Vert {\mathcal{P}}_{T_i^{\perp }}\left(Q\right){\Vert }_{\left(i\right)}^{\ast }\Vert {\mathcal{P}}_{T_i^{\perp }}\left({\mathrm{\Delta }}_i\right){\Vert }_{\left(i\right)}\\ >{\displaystyle \sum _{i=1}^L{\mathrm{\lambda }}_i\Vert X_i{\Vert }_{\left(i\right)}}\end{array}$$

■

With Proposition A.2 and Theorem A.3, we are ready to prove Theorem V.1.

Proof of Theorem V.1

Since ${\displaystyle {\sum }_{j\ne i}{\mu }_{\mathit{ij}}{\mathrm{\lambda }}_j/{\mathrm{\lambda }}_i<1/2}$, by Proposition A.2, $T_i\cap {\displaystyle {\sum }_{j\ne i}{T}_j}=\left\{0\right\}$ for all i. Thus, there is a unique matrix Q in ${\displaystyle {\sum }_{i=1}^L{T}_i}$ such that ${\mathcal{P}}_{T_i}\left(Q\right)={\mathrm{\lambda }}_i{E}_i$. In addition, Q can be uniquely expressed as a sum of elements in T_i. That is, $Q={\displaystyle {\sum }_{i=1}^L{Q}_i}$ with Q_i ∈ T_i. We now have a matrix Q that satisfies the first optimality condition. In the following, we will show that it also satisfies the second optimality condition $\Vert {\mathcal{P}}_{T_i^{\perp }}Q{\Vert }_{\left(i\right)}^{\ast }<{\mathrm{\lambda }}_i$.

If the vector spaces ${\left\{T_i\right\}}_{i=1}^L$ are orthogonal, then Q_i is exactly λ_iE_i. Because they are not necessarily orthogonal, we express Q_i as λ_iE_i plus a correction term λ_i∈_i. That is, we express Q_i = λ_i(E_i + ∈_i). Putting Q_i’s back to Q, we have

$$Q={\displaystyle \sum _{i=1}^L{\mathrm{\lambda }}_i\left(E_i+{\mathrm{\epsilon }}_i\right)}$$ (27)

Combining the above equation (27) with the first optimality condition (A.3), ${\mathcal{P}}_{T_i}(Q)={\mathrm{\lambda }}_i{E}_i$, we have ${\displaystyle {\sum }_{j=1}^L{\mathrm{\lambda }}_j{\mathcal{P}}_{T_i}(E_j+{\epsilon }_j)=}{\mathrm{\lambda }}_i{E}_i$. Since ${\mathcal{P}}_{T_i}(E_i+{\epsilon }_i)=E_i+{\epsilon }_i$, rearranging the equation, we obtain the following recursive expression for ${\epsilon }_i$:

$${\epsilon }_i=-{\mathcal{P}}_{T_i}\left({\displaystyle \sum _{j\ne 1}\frac{{\mathrm{\lambda }}_j}{{\mathrm{\lambda }}_i}(E_j+{\epsilon }_j)}\right)$$ (28)

We now obtain a bound on ${\Vert {\mathcal{P}}_{T_i^{\perp }}(Q)\Vert }_{(i)}^{\ast }$ in terms of ε_i.

$$\begin{array}{l}{\Vert {\mathcal{P}}_{T_i^{\perp }}(Q)\Vert }_{(i)}^{\ast }={\Vert {\mathcal{P}}_{T_i^{\perp }}({\displaystyle {\sum }_{j\ne i}{\mathrm{\lambda }}_j(E_j+{\epsilon }_i)})\Vert }_{(i)}^{\ast }\\ \le {\Vert {\displaystyle \sum _{j\ne i}{\mathrm{\lambda }}_j}(E_j+{\epsilon }_j)\Vert }_{(i)}^{\ast }\end{array}$$ (29)

$$\le {\displaystyle \sum _{j\ne i}{\mu }_{\mathit{ij}}}{\mathrm{\lambda }}_j(1+{\Vert {\epsilon }_j\Vert }_{(j)}^{\ast })$$ (30)

$$\le ({\displaystyle \sum _{j\ne i}{\mu }_{\mathit{ij}}}{\mathrm{\lambda }}_j)\underset{j\ne i}{\text{max}}(1+{\Vert {\epsilon }_j\Vert }_{(j)}^{\ast })$$ (31)

where we obtain equation (29) from equation (20), equation (30) from equation (21) and the last inequality (31) from Holder’s inequality.

Similarly, we obtain a recursive expression for $1+{\Vert {\epsilon }_i\Vert }_{(i)}^{\ast }$ using equation (28)

$$\begin{array}{l}1+{\Vert {\epsilon }_i\Vert }_{(i)}^{\ast }=1+{\Vert {\mathcal{P}}_{T_i}({\displaystyle {\sum }_{j\ne i}\frac{{\mathrm{\lambda }}_j}{{\mathrm{\lambda }}_i}(E_j+{\epsilon }_j)})\Vert }_{(i)}^{\ast }\\ \le 1+{\Vert {\displaystyle {\sum }_{j\ne i}\frac{{\mathrm{\lambda }}_j}{{\mathrm{\lambda }}_i}(E_j+{\epsilon }_j)}\Vert }_{(i)}^{\ast }\end{array}$$ (32)

$$\le 1+{{\displaystyle {\sum }_{j\ne i}{\mu }_{\mathit{ij}}\frac{{\mathrm{\lambda }}_j}{{\mathrm{\lambda }}_i}(1+\Vert {\epsilon }_j\Vert }}_{(j)}^{\ast })$$ (33)

$$\le 1+({\displaystyle \sum _{j\ne i}{\mu }_{\mathit{ij}}\frac{{\mathrm{\lambda }}_j}{{\mathrm{\lambda }}_i}}\underset{j\ne i}{\text{max}}(1+{\Vert {\epsilon }_j\Vert }_{(j)}^{\ast })$$ (34)

where we obtain equation (32) from equation (19), equation (33) from equation (21) and the last inequality (34) from Holder’s inequality.

Taking the maximum over i on both sides and rearranging, we have

$$\underset{i}{\text{max}}(1+{\Vert {\epsilon }_i\Vert }_{(i)}^{\ast })\le \frac{1}{1-{\text{max}}_i{\displaystyle {\sum }_{j\ne i}{\mu }_{\mathit{ij}}\frac{{\mathrm{\lambda }}_j}{{\mathrm{\lambda }}_i}}}$$

Putting the bound back to equation (31), we obtain

$$\begin{array}{l}{\Vert {\mathcal{P}}_{T_i^{\perp }}(Q)\Vert }_{(i)}^{\ast }\le {\mathrm{\lambda }}_i\frac{{\displaystyle {\sum }_{j\ne i}{\mu }_{\mathit{ij}}\frac{{\mathrm{\lambda }}_j}{{\mathrm{\lambda }}_i}}}{1-{\text{max}}_i{\displaystyle {\sum }_{j\ne i}{\mu }_{\mathit{ij}}\frac{{\mathrm{\lambda }}_j}{{\mathrm{\lambda }}_i}}}\\ <{\mathrm{\lambda }}_i\end{array}$$ (35)

where we used ${\displaystyle {\sum }_{j\ne i}{\mu }_{\mathit{ij}}{\mathrm{\lambda }}_j/}{\mathrm{\lambda }}_i<1/2$ in the last inequality.

Thus, we have constructed a dual certificate Q that satisfies the optimality conditions (A.3) and ${\left\{X_i\right\}}_{i=1}^L$ is the unique optimizer of the convex problem (2).

APPENDIX B PROOF OF THEOREM VI

In this section, we provide a proof of Theorem VI, showing that as long as we can choose our regularization parameters accordingly, we obtain a solution from the convex program (6) that is close to the ground truth ${\left\{X_i\right\}}_{i=1}^L$.

We will begin by proving a technical lemma collecting three inequalities. Throughout the section, we will assume $X_{\mathcal{Z}}$ is non-zero for simplicity, so that the subgradient of ${\Vert X_{\mathcal{Z}}\Vert }_{\text{fro}}$ is exactly $X_{\mathcal{Z}}/{\Vert X_{\mathcal{Z}}\Vert }_{\text{fro}}$.

Lemma B.1

For i=1,…,L, the following three inequalities hold,

$${\Vert X_i\Vert }_{\text{(}i\text{)}}-{\Vert X_i+{\mathrm{\Delta }}_i\Vert }_{\text{(}i\text{)}}\le {\Vert {\mathcal{P}}_{T_i}({\mathrm{\Delta }}_i)\Vert }_{\text{(}i\text{)}}-{\Vert {\mathcal{P}}_{T_i^{\perp }}({\mathrm{\Delta }}_i)\Vert }_{\left(i\right)}$$ (36)

$${\displaystyle \sum _{i=1}^L{\mathrm{\lambda }}_i}{\Vert {\mathcal{P}}_{T_i^{\perp }}({\mathrm{\Delta }}_i)\Vert }_{\left(i\right)}\le 3{\displaystyle \sum _{i=1}^L{\mathrm{\lambda }}_i}{\Vert {\mathcal{P}}_{T_i}({\mathrm{\Delta }}_i)\Vert }_{\left(i\right)}$$ (37)

$${\Vert {\mathcal{P}}_{T_i}({\mathrm{\Delta }}_i)\Vert }_{\left(i\right)}\le \sqrt{2{\displaystyle \sum _{b\in {\mathcal{P}}_i}{r}_b}}{\Vert ({\mathrm{\Delta }}_i)\Vert }_{\text{fro}}$$ (38)

Proof

We will prove the inequalities in order.

Let us choose a subgradient $G_i=E_i+W_i$ of ${\Vert X_i\Vert }_{(i)}$ at X_i such that $\langle W_i,{\mathcal{P}}_{T_i^{\perp }}({\mathrm{\Delta }}_i)\rangle ={\Vert {\mathcal{P}}_{T_i^{\perp }}({\mathrm{\Delta }}_i)\Vert }_{(i)}$. Then, from the definition of the subgradient, we have,

$$\begin{array}{l}{\Vert X_i+{\mathrm{\Delta }}_i\Vert }_{\left(i\right)}\ge {\Vert X_i\Vert }_{\left(i\right)}+\langle G_i,{\mathrm{\Delta }}_i\rangle \\ ={\Vert X_i\Vert }_{\left(i\right)}+\langle E_i,{\mathcal{P}}_{T_i}({\mathrm{\Delta }}_i)\rangle +{\Vert {\mathcal{P}}_{T_i^{\perp }}({\mathrm{\Delta }}_i)\Vert }_{\left(i\right)}\\ ={\Vert X_i\Vert }_{\left(i\right)}-{\Vert {\mathcal{P}}_{T_i}({\mathrm{\Delta }}_i)\Vert }_{(i)}+{\Vert {\mathcal{P}}_{T_i^{\perp }}({\mathrm{\Delta }}_i)\Vert }_{\left(i\right)}\end{array}$$ (39)

where we used Holder’s inequality for the last inequality (39). Re-arranging, we obtain the first result (36).

For the second inequality, we note that since ${\displaystyle {\sum }_{i=1}^L{X}_i+{\mathrm{\Delta }}_i+X_{\mathcal{Z}}+{\mathrm{\Delta }}_{\mathcal{Z}}=Y}$, we have ${\mathrm{\Delta }}_{\mathcal{Z}}=-{\displaystyle {\sum }_{i=1}^L{\mathrm{\Delta }}_i}$.From the definition of subgradient, we obtain,

$$\begin{array}{l}{\mathrm{\lambda }}_{\mathcal{Z}}{\Vert X_{\mathcal{Z}}+{\mathrm{\Delta }}_{\mathcal{Z}}\Vert }_{\text{fro}}\ge {\mathrm{\lambda }}_{\mathcal{Z}}{\Vert X_{\mathcal{Z}}\Vert }_{\text{fro}}+{\mathrm{\lambda }}_{\mathcal{Z}}\langle \frac{X_{\mathcal{Z}}}{{\Vert X_{\mathcal{Z}}\Vert }_{\text{fro}}},{\mathrm{\Delta }}_{\mathcal{Z}}\rangle \\ ={\mathrm{\lambda }}_{\mathcal{Z}}{\Vert X_{\mathcal{Z}}\Vert }_{\text{fro}}-{\displaystyle \sum _{i=1}^L{\mathrm{\lambda }}_{\mathcal{Z}}}\langle \frac{X_{\mathcal{Z}}}{{\Vert X_{\mathcal{Z}}\Vert }_{\text{fro}}},{\mathrm{\Delta }}_i\rangle \\ \ge {\mathrm{\lambda }}_{\mathcal{Z}}{\Vert X_{\mathcal{Z}}\Vert }_{\text{fro}}-{\displaystyle \sum _{i=1}^L{\mathrm{\lambda }}_{\mathcal{Z}}}\frac{{\Vert X_{\mathcal{Z}}\Vert }_{(i)}^{\ast }}{{\Vert X_{\mathcal{Z}}\Vert }_{\text{fro}}}{\Vert {\mathrm{\Delta }}_i\Vert }_{(i)}\end{array}$$ (40)

$$\ge {\mathrm{\lambda }}_{\mathcal{Z}}{\Vert X_{\mathcal{Z}}\Vert }_{\text{fro}}-{\displaystyle \sum _{i=1}^L\frac{{\mathrm{\lambda }}_i}{2}}{\Vert {\mathrm{\Delta }}_i\Vert }_{\left(i\right)}$$ (41)

$$\begin{array}{l}\ge {\mathrm{\lambda }}_{\mathcal{Z}}{\Vert X_{\mathcal{Z}}\Vert }_{\text{fro}}-{\displaystyle \sum _{i=1}^L\frac{{\mathrm{\lambda }}_i}{2}}{\Vert {\mathcal{P}}_{T_i}({\mathrm{\Delta }}_i)\Vert }_{\left(i\right)}\\ -\frac{{\mathrm{\lambda }}_i}{2}{\Vert {\mathcal{P}}_{T_i^{\perp }}({\mathrm{\Delta }}_i)\Vert }_{\left(i\right)}\end{array}$$ (42)

where we obtain equation (40) from Holder’s inequality, equation (41) from the condition of ${\mathrm{\lambda }}_i$ (8) and equation (42) from the triangle inequality.

Since ${\{X_i+{\mathrm{\Delta }}_i\}}_{i=1}^L$ and $X_{\mathcal{Z}}+{\mathrm{\Delta }}_{\mathcal{Z}}$ achieves the minimum objective function, we have,

$$\begin{array}{l}{\mathrm{\lambda }}_{\mathcal{Z}}{\Vert X_{\mathcal{Z}}\Vert }_{\text{fro}}+{\displaystyle \sum _{i=1}^L{\mathrm{\lambda }}_i}{\Vert X_i\Vert }_{(i)}\\ \ge {\mathrm{\lambda }}_{\mathcal{Z}}{\Vert X_{\mathcal{Z}}+{\mathrm{\Delta }}_{\mathcal{Z}}\Vert }_{\text{fro}}+{\displaystyle \sum _{i=1}^L{\mathrm{\lambda }}_i}{\Vert X_i+{\mathrm{\Delta }}_i\Vert }_{(i)}\end{array}$$

Substituting equation (39) and (42), we obtain,

$$\begin{array}{l}{\mathrm{\lambda }}_{\mathcal{Z}}{\Vert X_{\mathcal{Z}}\Vert }_{\text{fro}}+{\displaystyle \sum _{i=1}^L{\mathrm{\lambda }}_i}{\Vert X_i\Vert }_{(i)}\\ \ge {\mathrm{\lambda }}_{\mathcal{Z}}{\Vert X_{\mathcal{Z}}\Vert }_{\text{fro}}-{\displaystyle \sum _{i=1}^L\frac{{\mathrm{\lambda }}_i}{2}}{\Vert {\mathcal{P}}_{T_i}({\mathrm{\Delta }}_i)\Vert }_{(i)}-\frac{{\mathrm{\lambda }}_i}{2}{\Vert {\mathcal{P}}_{T_i^{\perp }}({\mathrm{\Delta }}_i)\Vert }_{(i)}\\ +{\displaystyle \sum _{i=1}^L{\mathrm{\lambda }}_i}{\Vert X_i\Vert }_{(i)}-{\mathrm{\lambda }}_i{\Vert {\mathcal{P}}_{T_i}({\mathrm{\Delta }}_i)\Vert }_{(i)}+{\mathrm{\lambda }}_i{\Vert {\mathcal{P}}_{T_i^{\perp }}({\mathrm{\Delta }}_i)\Vert }_{(i)}\end{array}$$ (43)

Cancelling and re-arranging, we obtain the desired inequality (37),

$${\displaystyle \sum _{i=1}^L{\mathrm{\lambda }}_i}{\Vert {\mathcal{P}}_{T_i^{\perp }}({\mathrm{\Delta }}_i)\Vert }_{(i)}\le 3{\displaystyle \sum _{i=1}^L{\mathrm{\lambda }}_i}{\Vert {\mathcal{P}}_{T_i}({\mathrm{\Delta }}_i)\Vert }_{(i)}$$

For the third inequality, recall that for any rank-r matrix X, its nuclear norm ${\Vert X\Vert }_{\text{nuc}}$ is upper bounded by $\sqrt{r}{\Vert X\Vert }_{\text{fro}}$. Moreover, the projection of any matrix Y to the column and row space T of a rank r matrix is at most rank-2r, that is rank(PT (Y)) ≤ 2r. Hence, we obtain,

$$\begin{array}{l}{\Vert {\mathcal{P}}_{T_i}({\mathrm{\Delta }}_i)\Vert }_{(i)}={\displaystyle \sum _{b\in {\mathcal{P}}_i}{\Vert R_b({\mathcal{P}}_{T_i}({\mathrm{\Delta }}_i))\Vert }_{\text{nuc}}}\\ \le {\displaystyle \sum _{b\in {\mathcal{P}}_i}\sqrt{2r_b}{\Vert R_b({\mathrm{\Delta }}_i)\Vert }_{\text{fro}}}\\ \le \sqrt{{\displaystyle \sum _{b\in {\mathcal{P}}_i}2r_b}}{\Vert ({\mathrm{\Delta }}_i)\Vert }_{\text{fro}}\end{array}$$

where the last inequality follows from Cauchy-Schwatz inequality and the fact that ${\displaystyle {\sum }_{b\in {\mathcal{P}}_i}{\Vert R_b({\mathrm{\Delta }}_i)\Vert }_{\text{fro}}^2}={\Vert ({\mathrm{\Delta }}_i)\Vert }_{\text{fro}}^2$■.

With these three inequalities, we now proceed to prove Theorem VI.

Proof of Theorem VI

From the optimality of ${\{X_i+{\mathrm{\Delta }}_i\}}_{i=1}^L$, we have the following inequality,

$$\begin{array}{l}{\mathrm{\lambda }}_{\mathcal{Z}}{\Vert X_{\mathcal{Z}}+{\mathrm{\Delta }}_{\mathcal{Z}}\Vert }_{\text{fro}}+{\displaystyle \sum _{i=1}^L{\mathrm{\lambda }}_i}{\Vert X_i+{\mathrm{\Delta }}_i\Vert }_{\left(i\right)}\\ \le {\mathrm{\lambda }}_{\mathcal{Z}}{\Vert X_{\mathcal{Z}}\Vert }_{\text{fro}}+{\displaystyle \sum _{i=1}^L{\mathrm{\lambda }}_i}{\Vert X_i\Vert }_{\left(i\right)}\end{array}$$

Re-arranging, we obtain,

$$\begin{array}{l}{\Vert X_{\mathcal{Z}}+{\mathrm{\Delta }}_{\mathcal{Z}}\Vert }_{\text{fro}}-{\Vert X_{\mathcal{Z}}\Vert }_{\text{fro}}\\ \le \frac{1}{{\mathrm{\lambda }}_{\mathcal{Z}}}\sum _{i=1}^L{\mathrm{\lambda }}_i({\Vert X_i\Vert }_{\left(i\right)}-{\Vert X_i+{\mathrm{\Delta }}_i\Vert }_{\left(i\right)})\end{array}$$

For convenience, let us define ${\mathrm{\Lambda }}_T={\displaystyle {\sum }_{i=1}^L{\mathrm{\lambda }}_i}{\Vert {\mathcal{P}}_{T_i}({\mathrm{\Delta }}_i)\Vert }_{(i)}$ and ${\mathrm{\Lambda }}_{T^{\perp }}={\displaystyle {\sum }_{i=1}^L{\mathrm{\lambda }}_i}{\Vert {\mathcal{P}}_{T_i^{\perp }}({\mathrm{\Delta }}_i)\Vert }_{(i)}$. Then, from Lemma B.1 equation (36), we obtain,

$${\Vert X_{\mathcal{Z}}+{\mathrm{\Delta }}_{\mathcal{Z}}\Vert }_{\text{fro}}-{\Vert X_{\mathcal{Z}}\Vert }_{\text{fro}}\le \frac{1}{{\mathrm{\lambda }}_{\mathcal{Z}}}({\mathrm{\Lambda }}_T-{\mathrm{\Lambda }}_{T^{\perp }})$$ (44)

We would like to keep only ${\mathrm{\Delta }}_{\mathcal{Z}}$ on the left hand side and cancel $X_{\mathcal{Z}}$. To do this, we multiply both sides of equation (44) with ${\Vert X_{\mathcal{Z}}+{\mathrm{\Delta }}_{\mathcal{Z}}\Vert }_{\text{fro}}+{\Vert X_{\mathcal{Z}}\Vert }_{\text{fro}}$. Then, using (a+b)(a−b) = a² − b², we expand the left hand side as:

$$\begin{array}{l}\mathrm{L}.\mathrm{H}.\mathrm{S}.={\Vert X_{\mathcal{Z}}+{\mathrm{\Delta }}_{\mathcal{Z}}\Vert }_{\text{fro}}^2-{\Vert X_{\mathcal{Z}}\Vert }_{\text{fro}}^2\\ ={\Vert {\mathrm{\Delta }}_{\mathcal{Z}}\Vert }_{\text{fro}}^2+2\langle X_{\mathcal{Z}},{\mathrm{\Delta }}_{\mathcal{Z}}\rangle \end{array}$$

Recall that ${\mathrm{\Delta }}_{\mathcal{Z}}=-{\displaystyle {\sum }_{i=1}^L{\mathrm{\Delta }}_i}$, we obtain the following lower bound for the left hand side:

$$\begin{array}{l}\mathrm{L}.\mathrm{H}.\mathrm{S}.={\Vert {\mathrm{\Delta }}_{\mathcal{Z}}\Vert }_{\text{fro}}^2-2\langle X_{\mathcal{Z}},{\displaystyle \sum _{i=1}^L{\mathrm{\Delta }}_i}\rangle \\ \ge {\Vert {\mathrm{\Delta }}_{\mathcal{Z}}\Vert }_{\text{fro}}^2-2{\displaystyle \sum _{i=1}^L{\Vert X_{\mathcal{Z}}\Vert }_{\left(i\right)}^{\ast }{\Vert {\mathrm{\Delta }}_i\Vert }_{\left(i\right)}}\end{array}$$ (45)

$$\ge {\Vert {\mathrm{\Delta }}_{\mathcal{Z}}\Vert }_{\text{fro}}^2-\frac{{\Vert X_{\mathcal{Z}}\Vert }_{\text{fro}}}{{\mathrm{\lambda }}_{\mathcal{Z}}}{\displaystyle \sum _{i=1}^L{\mathrm{\lambda }}_i}{\Vert {\mathrm{\Delta }}_i\Vert }_{(i)}$$ (46)

$$\ge {\Vert {\mathrm{\Delta }}_{\mathcal{Z}}\Vert }_{\text{fro}}^2-\frac{{\Vert X_{\mathcal{Z}}\Vert }_{\text{fro}}}{{\mathrm{\lambda }}_{\mathcal{Z}}}({\mathrm{\Lambda }}_T+{\mathrm{\Lambda }}_{T^{\perp }})$$ (47)

where we used Holder’s inequality for equation (45), the condition for λ_i for equation (46), and the triangle inequality for (47).

We now turn to upper bound the right hand side. We know ${\Vert X_{\mathcal{Z}}+{\mathrm{\Delta }}_{\mathcal{Z}}\Vert }_{\text{fro}}\le {\Vert X_{\mathcal{Z}}\Vert }_{\text{fro}}+({\mathrm{\Lambda }}_T-{\mathrm{\Lambda }}_{T^{\perp }})/{\mathrm{\lambda }}_{\mathcal{Z}}$ from equation (44). Hence, we obtain,

$$\begin{array}{l}\mathrm{R}.\mathrm{H}.\mathrm{S}.=({\Vert X_{\mathcal{Z}}+{\mathrm{\Delta }}_{\mathcal{Z}}\Vert }_{\text{fro}}+{\Vert X_{\mathcal{Z}}\Vert }_{\text{fro}})\frac{1}{{\mathrm{\lambda }}_{\mathcal{Z}}}({\mathrm{\Lambda }}_T-{\mathrm{\Lambda }}_{T^{\perp }})\\ \le (2{\Vert X_{\mathcal{Z}}\Vert }_{\text{fro}}+\frac{1}{{\mathrm{\lambda }}_{\mathcal{Z}}}({\mathrm{\Lambda }}_T-{\mathrm{\Lambda }}_{T^{\perp }}))+\frac{1}{{\mathrm{\lambda }}_{\mathcal{Z}}}({\mathrm{\Lambda }}_T-{\mathrm{\Lambda }}_{T^{\perp }})\end{array}$$

Using Lemma B.1 equation (37), we have,

$$\begin{array}{l}\mathrm{R}.\mathrm{H}.\mathrm{S}.\le 2\frac{{\Vert X_{\mathcal{Z}}\Vert }_{\text{fro}}}{{\mathrm{\lambda }}_{\mathcal{Z}}}({\mathrm{\Lambda }}_T-{\mathrm{\Lambda }}_{T^{\perp }})+\frac{1}{{\mathrm{\lambda }}_{\mathcal{Z}}^2}{({\mathrm{\Lambda }}_T-{\mathrm{\Lambda }}_{T^{\perp }})}^2\\ \le 2\frac{{\Vert X_{\mathcal{Z}}\Vert }_{\text{fro}}}{{\mathrm{\lambda }}_{\mathcal{Z}}}({\mathrm{\Lambda }}_T-{\mathrm{\Lambda }}_{T^{\perp }})+16\frac{1}{{\mathrm{\lambda }}_{\mathcal{Z}}^2}{\mathrm{\Lambda }}_T^2\end{array}$$ (48)

Combining and simplifying the lower bound (47) and the upper bound (48), we obtain,

$${\Vert {\mathrm{\Delta }}_{\mathcal{Z}}\Vert }_{\text{fro}}^2\le 3\frac{\Vert X_{\mathcal{Z}}{\Vert }_{\text{fro}}}{{\mathrm{\lambda }}_{\mathcal{Z}}}{\mathrm{\Lambda }}_T+16\frac{1}{{\mathrm{\lambda }}_{\mathcal{Z}}^2}{\mathrm{\Lambda }}_T^2$$ (49)

We will now lower bound ${\Vert {\mathrm{\Delta }}_{\mathcal{Z}}\Vert }_{\text{fro}}^2={\Vert {{\displaystyle {\sum }_{i=1}^L\mathrm{\Delta }}}_i\Vert }_{\text{fro}}^2$ by individual terms:

$$\begin{array}{l}{\Vert {\displaystyle \sum _{i=1}^L{\mathrm{\Delta }}_i}\Vert }_{\text{fro}}^2={\displaystyle \sum _{i=1}^L{\Vert {\mathrm{\Delta }}_i\Vert }_{\text{fro}}^2}+\langle {\mathrm{\Delta }}_i,{\displaystyle \sum _{j\ne i}{\mathrm{\Delta }}_j}\rangle \\ \ge {\displaystyle \sum _{i=1}^L{\Vert {\mathrm{\Delta }}_i\Vert }_{\text{fro}}^2}-{\Vert {\mathrm{\Delta }}_i\Vert }_{(i)}{\displaystyle \sum _{j\ne 1}{\Vert {\mathrm{\Delta }}_i\Vert }_{(i)}^{\ast }}\end{array}$$

where we used Holder’s inequality for the last inequality.

Now, using the assumption that both ${\Vert X_j\Vert }_{(i)}^{\ast }$ and ${\Vert X_j+{\mathrm{\Delta }}_j\Vert }_{(i)}^{\ast }$ are bounded by α_ij. We have,

$$\begin{array}{l}{\Vert {\displaystyle \sum _{i=1}^L{\mathrm{\Delta }}_i}\Vert }_{\text{fro}}^2\ge {\displaystyle \sum _{i=1}^L{\Vert {\mathrm{\Delta }}_i\Vert }_{\text{fro}}^2}-{\Vert {\mathrm{\Delta }}_i\Vert }_{(i)}{\displaystyle \sum _{j\ne i}2{\alpha }_{\mathit{ij}}}\\ \ge {\displaystyle \sum _{i=1}^L{\Vert {\mathrm{\Delta }}_i\Vert }_{\text{fro}}^2}-{\mathrm{\lambda }}_i{\Vert {\mathrm{\Delta }}_i\Vert }_{(i)}\\ \ge {\displaystyle \sum _{i=1}^L{\Vert {\mathrm{\Delta }}_i\Vert }_{\text{fro}}^2}-{\mathrm{\Lambda }}_T-{\mathrm{\Lambda }}_{T^{\perp }}\end{array}$$ (50)

$$\ge {\displaystyle \sum _{i=1}^L{\Vert {\mathrm{\Delta }}_i\Vert }_{\text{fro}}^2}-4{\mathrm{\Lambda }}_T$$ (51)

where we used the triangle inequality for equation (50) and Lemma B.1 equation (37) for equation (51).

Substituting the lower bound back to equation (49), we have

$${\displaystyle \sum _{i=1}^L{\Vert {\mathrm{\Delta }}_i\Vert }_{\text{fro}}^2}\le (3\frac{\Vert X_{\mathcal{Z}}\Vert \text{fro}}{{\mathrm{\lambda }}_{\mathcal{Z}}}+4){\mathrm{\Lambda }}_T+16\frac{1}{{\mathrm{\lambda }}_{\mathcal{Z}}^2}{\mathrm{\Lambda }}_T^2$$ (52)

We now turn to upper bound the equation. From Lemma B.1 equation (38), we know that ${\mathrm{\Lambda }}_T\le {\displaystyle {\sum }_{i=1}^L{\mathrm{\lambda }}_i\sqrt{2{\displaystyle {\sum }_{b\in {\mathcal{P}}_i}{r}_b{\Vert {\mathrm{\Delta }}_i\Vert }_{\text{fro}}}}}$. Hence, we have,

$$\begin{array}{l}16{\mathrm{\Lambda }}_T^2\le 16{\left({\displaystyle \sum _{i=1}^L{\mathrm{\lambda }}_i}\sqrt{2{\displaystyle \sum _{b\in {\mathcal{P}}_i}{r}_b{\Vert {\mathrm{\Delta }}_i\Vert }_{\text{fro}}}}\right)}^2\\ \le 16\left({\displaystyle \sum _{i=1}^L{\mathrm{\lambda }}_i^{2}2}{\displaystyle \sum _{b\in {\mathcal{P}}_i}{r}_b}\right){\displaystyle \sum _{i=1}^L{\Vert {\mathrm{\Delta }}_i\Vert }_{\text{fro}}^2}\end{array}$$ (53)

$$\le \frac{1}{2}{\mathrm{\lambda }}_{\mathcal{Z}}^2{\displaystyle \sum _{i=1}^L{\Vert {\mathrm{\Delta }}_i\Vert }_{\text{fro}}^2}$$ (54)

where we used Cauchy-Schwartz’s inequality for equation (53) and the condition for ${\mathrm{\lambda }}_{\mathcal{Z}}$ for equation (54). Hence, substituting back to equation (52), rearranging and ignoring constants, we have, ${\displaystyle \sum _{i=1}^L{\Vert {\mathrm{\Delta }}_i\Vert }_{\text{fro}}^2}\lesssim \frac{{\Vert X_{\mathcal{Z}}\Vert }_{\text{fro}}}{{\mathrm{\lambda }}_{\mathcal{Z}}}{\displaystyle \sum _{i=1}^L{\mathrm{\lambda }}_i\sqrt{{\displaystyle \sum _{b\in {\mathcal{P}}_i}{r}_b}}}{\Vert {\mathrm{\Delta }}_i\Vert }_{\text{fro}}$

Completing the squares with respect to ${\Vert {\mathrm{\Delta }}_i\Vert }_{\text{fro}}$ gives us,

$$\begin{array}{l}{\displaystyle \sum _{i=1}^L{\left({\Vert {\mathrm{\Delta }}_i\Vert }_{\text{fro}}-\frac{{\Vert X_{\mathcal{Z}}\Vert }_{\text{fro}}}{{\mathrm{\lambda }}_{\mathcal{Z}}}{\mathrm{\lambda }}_i\sqrt{{\displaystyle \sum _{b\in {\mathcal{P}}_i}{r}_b}}\right)}^2}\\ \lesssim {\displaystyle \sum _{i=1}^L{\left(\frac{{\Vert X_{\mathcal{Z}}\Vert }_{\text{fro}}}{{\mathrm{\lambda }}_{\mathcal{Z}}}{\mathrm{\lambda }}_i\sqrt{{\displaystyle \sum _{b\in {\mathcal{P}}_i}{r}_b}}\right)}^2}\end{array}$$

Using the triangle inequality to lower bound the left hand side, we obtain

$$\begin{array}{l}{\displaystyle \sum _{i=1}^L{\Vert {\mathrm{\Delta }}_i\Vert }_{\text{fro}}-\frac{{\Vert X_{\mathcal{Z}}\Vert }_{\text{fro}}}{{\mathrm{\lambda }}_{\mathcal{Z}}}{\mathrm{\lambda }}_i\sqrt{{\displaystyle \sum _{b\in {\mathcal{P}}_i}{r}_b}}}\\ \lesssim \sqrt{{\displaystyle \sum _{i=1}^L{\left(\frac{{\Vert X_{\mathcal{Z}}\Vert }_{\text{fro}}}{{\mathrm{\lambda }}_{\mathcal{Z}}}{\mathrm{\lambda }}_i\sqrt{{\displaystyle \sum _{b\in {\mathcal{P}}_i}{r}_b}}\right)}^2}}\end{array}$$

Using the fact that ℓ1-norm is larger than the ℓ2-norm, and re-arranging give us the desired result,

$${\displaystyle \sum _{i=1}^L{\Vert {\mathrm{\Delta }}_i\Vert }_{\text{fro}}\lesssim \frac{{\Vert X_{\mathcal{Z}}\Vert }_{\text{fro}}}{{\mathrm{\lambda }}_{\mathcal{Z}}}{\displaystyle \sum _{i=1}^L{\mathrm{\lambda }}_i}\sqrt{{\displaystyle \sum _{b\in {\mathcal{P}}_i}{r}_b}}}$$

■