Convolutional Analysis Operator Learning: Dependence on Training Data

Abstract

Convolutional analysis operator learning (CAOL) enables the unsupervised training of (hierarchical) convolutional sparsifying operators or autoencoders from large datasets. One can use many training images for CAOL, but a precise understanding of the impact of doing so has remained an open question. This paper presents a series of results that lend insight into the impact of dataset size on the filter update in CAOL. The first result is a general deterministic bound on errors in the estimated filters, and is followed by a bound on the expected errors as the number of training samples increases. The second result provides a high probability analogue. The bounds depend on properties of the training data, and we investigate their empirical values with real data. Taken together, these results provide evidence for the potential benefit of using more training data in CAOL.

I. Introduction

LEARNING convolutional operators from large datasets is a growing trend in signal/image processing, computer vision, machine learning, and artificial intelligence. The convolutional approach resolves the large memory demands of patch-based operator learning and enables unsupervised operator learning from “big data,” i.e., many high-dimensional signals. See [1], [2] and references therein. Examples include convolutional dictionary learning [2], [3] and convolutional analysis operator learning (CAOL) [1], [4]. CAOL trains an autoencoding CNN in an unsupervised manner, and is useful for training multi-layer CNNs from many training images [1]. In particular, the block proximal gradient method using a majorizer [1], [2] leads to rapidly converging and memory-efficient CAOL [1]. However, a theoretical understanding of the impact of using many training images in CAOL has remained an open question.

This paper presents new insights on this topic. Our first main result provides a deterministic bound on filter estimation error, and is followed by a bound on the expected error when “model mismatch” has zero mean. (See Theorem 1 and Corollary 2, respectively.) The expected error bound depends on the training data, and we provide empirical evidence of its decrease with an increase in training samples. Our second main result provides a high probability bound that explicitly decreases with increasingly many i.i.d. training samples. The bound improves when model mismatch and samples are uncorrelated. (See Theorem 3.) Additional empirical findings provide evidence that the correlation can indeed be small in practice. Put together, our findings provide new insight into how using many samples can improve CAOL, underscoring the benefits of the low memory usage of CAOL.

II. Backgrounds and Preliminaries

A. CAOL with orthogonality constraints

CAOL seeks a set of filters that “best” sparsify a set of training images $\left\{x_l\in {ℂ}^N:l=1,\dots ,L\right\}$ by solving the optimization problem [1, §II-A] (see Appendix for notation):

$$\underset{D=\left[d_1,\dots ,d_K\right]}{\text{argmin}}\underset{{}_{\left\{z_{l,k}\right\}}}{\min }F\left(D,\left\{z_{l,k}\right\}\right),\text{ subj}.\text{to}D{D}^H=\frac{1}{R}\cdot I,$$ (P0)

$$F\left(D,\left\{z_{l,k}\right\}\right):=\sum _{l=1}^L\sum _{k=1}^K{\Vert d_k⊛x_l-z_{l,k}\Vert }_2^2+\alpha {\Vert z_{l,k}\Vert }_0,$$

where ⊛ denotes convolution, $\left\{d_k\in {ℂ}^R:k=1,\dots ,K\right\}$ is a set of K ≥ R convolutional kernels, $\left\{z_{l,k}\in {ℂ}^N:l=1,\dots ,L,k=1,\dots ,K\right\}$ is a set of sparse codes, α > 0 is a regularization parameter controlling the sparsity of features {z_l,k}, and ||·||₀ denotes the ℓ⁰-quasi-norm. We group the K filters into a matrix:

$$D:=\left[d_1\cdots d_K\right]\in {ℂ}^{R\times K}.$$ (1)

The orthogonality condition $D{D}^H=\frac{1}{R}I$ in (P0) enforces 1) a tight-frame condition on the filters, i.e., ${\sum }_{k=1}^K{\Vert d_k⊛x\Vert }_2^2=\Vert x{\Vert }_2^2$, ∀x [1, Prop. 2.1]; and 2) filter diversity when R = K, since $D{D}^H=\frac{1}{R}I$ implies $D^{H}D=\frac{1}{K}I$ and each pair of filters is incoherent, i.e., |〈d_k, d_k′〉|² = 0, ∀k ≠ k′. One often solves (P0) iteratively, by alternating between optimizing D (filter update) and optimizing {z_l,k : ∀l, k} (sparse code update) [1], i.e., at the i iteration, the current iterates are updated as $\left\{z_{l,k}^{(i+1)}\right\}={\mathrm{argmin}}_{\left\{z_{l,k}\right\}}F\left(D^{(i)},\left\{z_{l,k}\right\}\right)$ and $D^{(i+1)}={\mathrm{argmin}}_{D{D}^H=\frac{1}{R}\cdot I}F\left(D,\left\{z_{l,k}^{(i+1)}\right\}\right)$.

B. Filter update in a matrix form

The key to our analysis lies in rewriting the filter update for (P0) in matrix form, to which we apply matrix perturbation and concentration inequalities. Observe first that

$$d_k⊛x_l=\underset{=:{\mathrm{\Psi }}_l\in {ℂ}^{N\times R}}{\underbrace{\left[{\mathrm{\Pi }}^0{x}_l,\dots ,{\mathrm{\Pi }}^{R-1}{x}_l\right]}}{d}_k={\mathrm{\Psi }}_l{d}_k,l=1,\dots ,L,$$ (2)

where $\mathrm{\Pi }:=\left[\begin{array}{cc}0& I_{N-1}\\ 1& 0\end{array}\right]\in {ℂ}^{N\times N}$ is the circular shift operator and (·)ⁿ denotes the matrix product of its n copies. We consider a circular boundary condition to simplify the presentation of {Ψ_l} in (2), but our entire analysis holds for a general boundary condition with only minor modifications of {Ψ_l} as done in [1, §IV-A]. Using (2), the filter update of (P0) is rewritten as

$$D^{\star }=\underset{D}{\mathrm{argmin}}\sum _{l=1}^L{\Vert {\mathrm{\Psi }}_{l}D-Z_l\Vert }_F^2\text{,}\text{subj}.\text{to }D{D}^H=\frac{1}{R}\cdot I,$$ (P1)

where $Z_l:=\left[z_{l,1},\dots ,z_{l,K}\right]\in {ℂ}^{N\times K}$ contains all the current sparse code estimates for the lth sample, and we drop iteration superscript indices (·)⁽ⁱ⁾ throughout. The next section uses this form to characterize the filter update solution D^⋆.

III. Main Results:

Dependence of CAOL on Training Data

The main results in this section illustrate how training with many samples can reduce errors in the filter D^⋆ from (P1) and characterize the reduction in terms of properties of the training data. Throughout we model the current sparse codes estimates as

$$Z_l=\underset{=:Z_{\text{true },l}}{\underbrace{{\mathrm{\Psi }}_l{D}_{\text{true }}}}+E_l,l=1,\dots ,L$$ (3)

where D_true is formed from optimal (orthogonal) filters analogously to (1), and $E_l\in {ℂ}^{N\times K}$ captures model mismatch in the current sparse codes, e.g., due to the current iterate being far from convergence or being trapped in local minima.

The following theorem provides a deterministic characterization.

Theorem 1.

Suppose that both matrices

$$\sum _{l=1}^L{\mathrm{\Psi }}_l^H{Z}_l\in {ℂ}^{R\times K}and\sum _{l=1}^L{\mathrm{\Psi }}_l^H{Z}_{\text{true },l}\in {ℂ}^{R\times K}$$ (4)

are full row rank, where {Ψ_l, Z_l, Z_true,l : l = 1, …, L} are defined in (2)–(3). Then, the solution D^⋆ to (P1) has error with respect to D_true bounded as

$$\Vert D^{\star }-D_{\text{true }}{\Vert }_F^2\le 5\frac{\Vert {\sum }_{l=1}^L{\mathrm{\Psi }}_l^H{E}_l{\Vert }_F^2}{{\lambda }_{\text{min}}^2\left({\sum }_{l=1}^L{\mathrm{\Psi }}_l^H{\mathrm{\Psi }}_l\right)},$$ (5)

where λ_min(·) denotes the smallest eigenvalue of its argument.

The full row rank condition on (4) ensures that the estimated filters D^⋆ and the true filters D_true are unique, and it further guarantees that the denominator of (5) is strictly positive. When the model mismatches E₁, …, E_L are independent and mean zero, we obtain the following expected error bound:

Corollary 2.

Under the construction of Theorem 1, suppose that E_l is a zero-mean random matrix for l = 1, …, L, and is independent over l. Then,

$$\mathbb{E}{\Vert D^{\star }-D_{\text{true }}\Vert }_F^2\le 5{\overline{\sigma }}^2{\rho }^2,$$ (6)

where $\mathbb{E}(\cdot )$ denotes the expectation,

$$\begin{gathered} {\overline{\sigma }}^2:=\underset{{}_{l=1,\dots ,L}}{\text{max}}{\lambda }_{\text{max}}\left(\mathbb{E}\left\{E_l{E}_l^H\right\}\right), \\ {\rho }^2:=\frac{\mathrm{tr}\left({\sum }_{l=1}^L{\mathrm{\Psi }}_l^H{\mathrm{\Psi }}_l\right)}{{\lambda }_{\text{min}}^2\left({\sum }_{l=1}^L{\mathrm{\Psi }}_l^H{\mathrm{\Psi }}_l\right)}, \end{gathered}$$ (7)

λ_max(·) denotes the largest eigenvalue of its argument, and the expectation is taken over the model mismatch.

Given fixed K and R, it is natural to expect that ${\overline{\sigma }}^2$ is bounded by some constant independent of L, and so the expected error bound in (6) largely depends on ρ² in (7). When training samples are i.i.d., one may further expect $(1/L){\sum }_{l=1}^L{\mathrm{\Psi }}_l^H{\mathrm{\Psi }}_l$ to concentrate around its expectation, roughly resulting in ρ² ∝ 1/L, with a proportionality constant that depends on R and the statistics of the training data. Fig. 1 illustrates ρ² for various image datasets, providing empirical evidence of this decrease in real data.

Fig. 1.

Empirical values of ρ² in (7) show a decrease with L for different datasets and filter dimensions. (The fruit and city datasets with L =10 and N = 10⁴ were preprocessed with contrast enhancement and mean subtraction; see details of datasets and experiments in [1], [2] and references therein. For L < 10, the results are averaged over 50 datasets randomly selected from the full datasets.) Under the assumptions of Corollary 2, the decrease in this quantity leads to a better expected error bound in (5). Without preprocessing, the quantity ρ² increases by a factor of around 10³.

Our second theorem provides a probabilistic error bound via concentration inequalities, given i.i.d. training sample and model mismatch pairs (x₁, E₁), …, (x_L, E_L).¹ It removes the zero-mean assumption for the model mismatches {E_l : ∀l} in Corollary 2 that might be strong, e.g., if training data are not preprocessed to have zero mean.

Theorem 3.

Suppose that training sample and model mismatch pairs $\left(x_1,E_1\right),\dots ,\left(x_L,E_L\right)\stackrel{iid}{~}(x,E)$, where x and E are almost surely bounded, i.e.,

$$\Vert x{\Vert }_2\le \gamma and\Vert E{\Vert }_F\le \sigma ,$$ (8)

and the matrices in (4) are almost surely full row rank. Then, for any $0<\delta <{\lambda }_{\text{min}}(\overline{\mathrm{\Lambda }})/\left(2R{\gamma }^2\right)$, the solution D^⋆ to (P1) has error with respect to D_true bounded as

$${\Vert D^{\star }-D_{\text{true }}\Vert }_F^2\le 5{\left\{\frac{\sigma \sqrt{\mathrm{tr}(\overline{\mathrm{\Lambda }})/L}+{\Vert \mathbb{E}\left({\mathrm{\Psi }}^{H}E\right)\Vert }_F+2\sigma \gamma \sqrt{R}\delta }{{\lambda }_{\text{min}}(\overline{\mathrm{\Lambda }})-2{\gamma }^{2}R\delta }\right\}}^2,$$ (9)

with probability at least

$$1-3R\text{exp}\left(-L\frac{{\delta }^2/2}{3+\delta /3}\right),$$ (10)

where $\overline{\mathrm{\Lambda }}:=\mathbb{E}\left({\mathrm{\Psi }}^H\mathrm{\Psi }\right)$ and Ψ is constructed from x as in (2).

Taking δ sufficiently small, the high probability error bound (9) is primarily driven by

$$\overline{\rho }:=\frac{\sqrt{\mathrm{tr}(\overline{\mathrm{\Lambda }})/L}}{{\lambda }_{\text{min}}(\overline{\mathrm{\Lambda }})}\mathrm{and}\overline{\chi }:=\frac{{\Vert \mathbb{E}\left({\mathrm{\Psi }}^{H}E\right)\Vert }_F}{{\lambda }_{\text{min}}(\overline{\mathrm{\Lambda }})},$$ (11)

where $\overline{\rho }$ is analogous to ρ in (7), and $\overline{\chi }$ captures how correlated the model mismatch is to the training samples. As the number L of training samples increases, $\overline{\rho }$ decreases as $1/\sqrt{L}$. On the other hand, $\overline{\chi }$ is constant with respect to L and provides a floor for the bound. Fig. 2 illustrates $\overline{\chi }$ for CAOL iterates from different image datasets, and provides empirical evidence that this term can indeed be small in real data. If the model mismatch is sufficiently uncorrelated with the training samples, i.e., $\overline{\chi }$ is practically zero, then only the $\overline{\rho }$ term remains and this term decreases with L. Namely, if model mismatch is entirely uncorrelated with the training samples, then using many samples decreases the error bound to (effectively) zero.

Fig. 2.

Empirical estimate of $\overline{\chi }$ in (11) across iterations in the alternating optimization algorithm [1] that solves CAOL (P0) with α = 10⁻³. (The fruit and city datasets with L = 10 and N = 10⁴ were preprocessed with contrast enhancement and mean subtraction; see details of datasets and experiments in [1], [2] and references therein. The model mismatches $\left\{E_l^{(i)}:\forall l\right\}$ at the ith iteration were calculated every 50 iterations based on (3), where we use the converged filters for D_true.) Observe that ${\overline{\chi }}^{(i)}$ generally decreases over iterations; when $\overline{\chi }$ is small, the high probability error bound (9) in Theorem 3 depends primarily on $\overline{\rho }$ defined in (11).

IV. Related Works

Sample complexity [7] and synthesis (or reconstruction) error [8] have been studied in the context of synthesis operator learning (e.g., dictionary learning [9]); see the cited papers and references therein. A similar understanding for (C)AOL has however remained largely open; existing works focus primarily on establishing (C)AOL models and their algorithmic challenges [1], [10]–[13]. The authors in [14] studied sample complexity for a patch-based AOL method, but the form of their model differs from that of ours (P0). Specifically, they consider the following AOL problem: ${\text{min}}_D{\sum }_{l}f\left(D^T{\widehat{x}}_l\right)+g(D)$, where f (·) is a sparsity promoting function (e.g., a smooth approximation of the ℓ⁰-quasi-norm [14]), g(·) is a regularizer or constraint for the filter matrix D, and $\left\{{\widehat{x}}_l:l=1,\dots ,L\right\}$ is a set of training patches (not images).

V. Proof of Theorem 1

Rewriting (P1) yields that D^⋆ is a solution of the (scaled) orthogonal Procrustes problem [1, §S.VII]:

$$\underset{D}{\text{argmin}}{\Vert \tilde{\mathrm{\Psi }}D-\tilde{Z}\Vert }_F^2,\text{ subj}.\text{to}D{D}^H=\frac{1}{R}\cdot I,$$ (12)

where $\tilde{\mathrm{\Psi }}\in {ℂ}^{LN\times R}$ arises by stacking Ψ₁, …, Ψ_L vertically and $\tilde{Z}\in {ℂ}^{LN\times K}$ arises likewise from Z₁, …, Z_L. Similarly, since Ψ₁ D_true = Z_true,l as in (3), D_true is a solution of the analogous (scaled) orthogonal Procrustes problem

$$\underset{D}{\text{argmin}}{\Vert \tilde{\mathrm{\Psi }}D-{\tilde{Z}}_{\text{true}}\Vert }_F^2,\text{ subj}.\text{to}D{D}^H=\frac{1}{R}\cdot I,$$ (13)

where ${\tilde{Z}}_{\text{true }}\in {ℂ}^{LN\times K}$ arises by stacking Z_true,1, …, Z_true,L vertically.

By assumption, both ${\tilde{\mathrm{\Psi }}}^H\tilde{Z}$ and ${\tilde{\mathrm{\Psi }}}^H{\tilde{Z}}_{\text{true}}$ are full row rank and so (12) and (13) have unique solutions given by the unique (scaled) polar factors

$$D^{\star }=\frac{1}{\sqrt{R}}\text{Q}{\left({\tilde{Z}}^H\tilde{\mathrm{\Psi }}\right)}^H{D}_{\text{true}}=\frac{1}{\sqrt{R}}\text{Q}{\left({\tilde{Z}}_{\text{true}}^H\tilde{\mathrm{\Psi }}\right)}^H$$ (14)

where Q(·) denotes the polar factor of its argument, and can be computed as Q(A) = WV^H from the (thin) singular value decomposition A = WΣV^H.

Thus we have

$$\begin{gathered} {\Vert D^{\star }-D_{\text{true }}\Vert }_F^2 \\ =\frac{1}{R}{\Vert \text{Q}\left({\tilde{Z}}_{\text{true}}^H\tilde{\mathrm{\Psi }}\right)-\text{Q}\left({\tilde{Z}}^H\tilde{\mathrm{\Psi }}\right)\Vert }_F^2 \\ \le \frac{1}{R}{\Vert {\tilde{E}}^H\tilde{\mathrm{\Psi }}\Vert }_F^2\left\{{\left[\frac{2}{{\sigma }_R\left({\tilde{Z}}_{\text{true}}^H\tilde{\mathrm{\Psi }}\right)+{\sigma }_R\left({\tilde{Z}}^H\tilde{\mathrm{\Psi }}\right)}\right]}^2+{\left[\frac{1}{\text{max}\left\{{\sigma }_R\left({\tilde{Z}}_{\text{true}}^H\tilde{\mathrm{\Psi }}\right),{\sigma }_R\left({\tilde{Z}}^H\tilde{\mathrm{\Psi }}\right)\right\}}\right]}^2\right\} \\ \le \frac{1}{R}{\Vert {\tilde{E}}^H\tilde{\mathrm{\Psi }}\Vert }_F^2\left\{{\left[\frac{2}{{\sigma }_R\left({\tilde{Z}}_{\text{true }}^H\tilde{\mathrm{\Psi }}\right)}\right]}^2+{\left[\frac{1}{{\sigma }_R\left({\tilde{Z}}_{\text{true }}^H\tilde{\mathrm{\Psi }}\right)}\right]}^2\right\} \\ =\frac{5}{R}\frac{{\Vert {\tilde{\mathrm{\Psi }}}^H\tilde{E}\Vert }_F^2}{{\sigma }_R^2\left({\tilde{\mathrm{\Psi }}}^H{\tilde{Z}}_{\text{true}}\right)}=\frac{5}{R}\frac{{\Vert {\sum }_{l=1}^L{\mathrm{\Psi }}_l^H{E}_l\Vert }_F^2}{{\sigma }_R^2\left({\sum }_{l=1}^L{\mathrm{\Psi }}_l^H{Z}_{\text{true,}l}\right)} \end{gathered}$$ (15)

where $\tilde{E}=\tilde{Z}-{\tilde{Z}}_{\text{true}}$ is exactly E₁, …, E_L stacked vertically, and σ_r (·) denotes the rth largest singular value of its argument. The first inequality holds by the perturbation bound in [15, Thm. 3], and the second holds since ${\sigma }_R\left({\tilde{Z}}^H\tilde{\mathrm{\Psi }}\right)\ge 0$. Recalling that Z_true,l = Ψ_lD_true, we rewrite the denominator of (15) as

$$\begin{gathered} {\sigma }_R^2\left(\sum _{l=1}^L{\mathrm{\Psi }}_l^H{Z}_{\text{true },l}\right)={\sigma }_R^2\left(\sum _{l=1}^L{\mathrm{\Psi }}_l^H{\mathrm{\Psi }}_l{D}_{\text{true }}\right) \\ =\frac{1}{R}{\sigma }_R^2\left(\sum _{l=1}^L{\mathrm{\Psi }}_l^H{\mathrm{\Psi }}_l\right)=\frac{1}{R}{\lambda }_{\text{min}}^2\left(\sum _{l=1}^L{\mathrm{\Psi }}_l^H{\mathrm{\Psi }}_l\right), \end{gathered}$$ (16)

where the second equality holds because $D_{\text{trne}}{D}_{\text{true}}^H=(1/R)I$. Substituting (16) into (15) yields (5).

VI. Proof of Corollary 2

Taking the expectation of (5) over the model mismatch amounts to taking the expectation of the numerator of the upper bound in (5):

$$\begin{gathered} \mathbb{E}{\Vert \sum _{l=1}^L{\mathrm{\Psi }}_l^H{E}_l\Vert }_F^2 \\ =\sum _{l=1}^L\mathbb{E}{\Vert {\mathrm{\Psi }}_l^H{E}_l\Vert }_F^2=\sum _{l=1}^L\mathrm{tr}\left({\mathrm{\Psi }}_l^H\mathbb{E}\left\{E_l{E}_l^H\right\}{\mathrm{\Psi }}_l\right) \\ \le \sum _{l=1}^L{\lambda }_{\text{max}}\left(\mathbb{E}\left\{E_l{E}_l^H\right\}\right)\cdot {\Vert {\mathrm{\Psi }}_l\Vert }_F^2\le {\overline{\sigma }}^2\cdot \sum _{l=1}^L{\Vert {\mathrm{\Psi }}_l\Vert }_F^2, \end{gathered}$$ (17)

where the first equality holds by using the assumption that E_l is zero-mean and independent over l, the second equality follows by expanding the Frobenius norm then applying linearity of the trace and expectation, the first inequality holds since $v^{H}Mv\le {\lambda }_{\text{max}}(M)\cdot \Vert v{\Vert }_2^2$ for any vector v and Hermitian matrix M, and the last inequality follows from the definition of ${\overline{\sigma }}^2$. Rewriting (17) using the identity ${\sum }_{l=1}^L{\Vert {\mathrm{\Psi }}_l\Vert }_F^2={\sum }_{l=1}^L\text{tr}\left({\mathrm{\Psi }}_l^H{\mathrm{\Psi }}_l\right)=\mathrm{tr}\left({\sum }_{l=1}^L{\mathrm{\Psi }}_l^H{\mathrm{\Psi }}_l\right)$ yields the result (6).

VII. Proof of Theorem 3

We derive two high probability bounds, one each for the numerator and denominator of (5). Then, the bound (9) with probability (10) follows by combining the two via a union bound. Before we begin, note that (8) implies that $\Vert \mathrm{\Psi }{\Vert }_2\le \Vert \mathrm{\Psi }{\Vert }_F\le \gamma \sqrt{R}$ almost surely; our proofs use this inequality multiple times.

A. Upper bound for numerator

Observe first that

$${\Vert \sum _{l=1}^L{\mathrm{\Psi }}_l^H{E}_l\Vert }_F={\Vert L\mathbb{E}\left({\mathrm{\Psi }}_l^H{E}_l\right)+\sum _{l=1}^L\left\{{\mathrm{\Psi }}_l^H{E}_l-\mathbb{E}\left({\mathrm{\Psi }}_l^H{E}_l\right)\right\}\Vert }_F\le L{\Vert \mathbb{E}\left({\mathrm{\Psi }}_l^H{E}_l\right)\Vert }_F+{\Vert \sum _{l=1}^L{\xi }_l\Vert }_2,$$ (18)

where ${\xi }_l:=\mathrm{vec}\left\{{\mathrm{\Psi }}_l^H{E}_l-\mathbb{E}\left({\mathrm{\Psi }}_l^H{E}_l\right)\right\}\in {ℂ}^{RK}$ for l = 1, …, L. We next bound ${\Vert {\sum }_{l=1}^L{\xi }_l\Vert }_2$ via the vector Bernstein inequality [16, Cor. 8.44]. Note that ξ₁, …, ξ_L are i.i.d. with $\mathbb{E}{\xi }_l=0$ (by construction). Furthermore, ξ_l is almost surely bounded as

$$\begin{gathered} {\Vert {\xi }_l\Vert }_2={\Vert {\mathrm{\Psi }}_l^H{E}_l-\mathbb{E}\left({\mathrm{\Psi }}_l^H{E}_l\right)\Vert }_F \\ \le {\Vert {\mathrm{\Psi }}_l^H{E}_l\Vert }_F+{\Vert \mathbb{E}\left({\mathrm{\Psi }}_l^H{E}_l\right)\Vert }_F\text{(Triangle}\text{ineq}.) \\ \le {\Vert {\mathrm{\Psi }}_l^H{E}_l\Vert }_F+{\Vert \mathbb{E}\left({\mathrm{\Psi }}_l^H{E}_l\right)\Vert }_F\text{(Jensen}’\text{s ineq}.) \\ \le {\Vert {\mathrm{\Psi }}_l\Vert }_F{\Vert E_l\Vert }_F+\mathbb{E}{\Vert {\mathrm{\Psi }}_l\Vert }_F{\Vert E_l\Vert }_F \\ \le 2\sigma \gamma \sqrt{R}. \end{gathered}$$

Thus the vector Bernstein inequality [16, Cor. 8.44] yields that for any t > 0,

$${\Vert \sum _{l=1}^L{\xi }_l\Vert }_2\le \sigma \sqrt{L}\sqrt{\mathrm{tr}(\overline{\mathrm{\Lambda }})}+t,$$ (19)

with probability at least

$$1-\text{exp}\left\{\frac{-t^2/2}{3L{(2\sigma \gamma \sqrt{R})}^2+t(2\sigma \gamma \sqrt{R})/3}\right\}.$$ (20)

We obtained (19) by the following simplification:

$$\begin{gathered} \mathbb{E}{\Vert \sum _{l=1}^L{\xi }_l\Vert }_2\le \sqrt{\mathbb{E}{\Vert \sum _{l=1}^L{\xi }_l\Vert }_2^2}=\sqrt{L\mathbb{E}{\Vert {\xi }_l\Vert }_2^2} \\ =\sqrt{L\mathbb{E}{\Vert {\mathrm{\Psi }}_l^H{E}_l-\mathbb{E}\left({\mathrm{\Psi }}_l^H{E}_l\right)\Vert }_F^2} \\ =\sqrt{L\left\{\mathbb{E}{\Vert {\mathrm{\Psi }}_l^H{E}_l\Vert }_F^2-{\Vert \mathbb{E}\left({\mathrm{\Psi }}_l^H{E}_l\right)\Vert }_F^2\right\}} \\ \le \sqrt{L\mathbb{E}{\Vert {\mathrm{\Psi }}_l^H{E}_l\Vert }_F^2}\le \sqrt{L\mathbb{E}\left({\Vert {\mathrm{\Psi }}_l\Vert }_F^2{\Vert E_l\Vert }_F^2\right)} \\ \le \sqrt{L{\sigma }^2\mathbb{E}{\Vert {\mathrm{\Psi }}_l\Vert }_F^2}=\sigma \sqrt{L}\sqrt{\mathrm{tr}(\overline{\mathrm{\Lambda }})}, \end{gathered}$$

where the third equality holds by $\mathbb{E}\Vert A-\mathbb{E}A{\Vert }_F^2={\sum }_{i,j}\mathbb{E}{\left(A_{i,j}-\mathbb{E}{A}_{i,j}\right)}^2={\sum }_{i,j}\mathbb{E}{A}_{i,j}^2-{\left(\mathbb{E}{A}_{i,j}\right)}^2=\mathbb{E}\Vert A{\Vert }_F^2-\Vert \mathbb{E}A{\Vert }_F^2$. We obtained (20) by the following simplifications:

$$\underset{\Vert x{\Vert }_2\le 1}{\text{sup}}\mathbb{E}{\left|x^H{\xi }_l\right|}^2\le \mathbb{E}{\Vert {\xi }_l\Vert }_2^2\le {(2\sigma \gamma \sqrt{R})}^2,$$

$$\mathbb{E}{\Vert \sum _{l=1}^L{\xi }_l\Vert }_2\le \mathbb{E}\sum _{l=1}^L{\Vert {\xi }_l\Vert }_2\le L\mathbb{E}{\Vert {\xi }_l\Vert }_2\le L(2\sigma \gamma \sqrt{R}).$$

Applying (19) and (20) with $t=2\sigma \gamma \sqrt{R}L\delta$ to the square of (18) yields

$${\Vert \sum _{l=1}^L{\mathrm{\Psi }}_l^H{E}_l\Vert }_F^2\le L^2{\left\{\sigma \sqrt{\mathrm{tr}(\overline{\mathrm{\Lambda }})/L}+{\Vert \mathbb{E}\left({\mathrm{\Psi }}_l^H{E}_l\right)\Vert }_F+2\sigma \gamma \sqrt{R}\delta \right\}}^2,$$ (21)

with probability at least $1-\text{exp}\left(-L\frac{{\delta }^2/2}{3+\delta /3}\right)$.

B. Lower bound for denominator

Observe that ${\sum }_{l=1}^L{\mathrm{\Psi }}_l^H{\mathrm{\Psi }}_l=L\overline{\mathrm{\Lambda }}+{\sum }_{l=1}^L{\mathrm{\Lambda }}_l$, where ${\mathrm{\Lambda }}_l:={\mathrm{\Psi }}_l^H{\mathrm{\Psi }}_l-\overline{\mathrm{\Lambda }}$, so Weyl’s inequality [17] yields

$${\lambda }_{\text{min}}\left(\sum _{l=1}^L{\mathrm{\Psi }}_l^H{\mathrm{\Psi }}_l\right)\ge {\lambda }_{\text{min}}(L\overline{\mathrm{\Lambda }})-{\Vert \sum _{l=1}^L{\mathrm{\Lambda }}_l\Vert }_2,$$ (22)

and it remains to bound ${\Vert {\sum }_{l=1}^L{\mathrm{\Lambda }}_l\Vert }_2$. We do so by using the Matrix Bernstein inequality [16, Cor. 8.15].

Note that Λ₁, …, Λ_L are i.i.d. (since x₁, …, x_L are i.i.d.) and $\mathbb{E}{\mathrm{\Lambda }}_l=0$. Furthermore, Λ_l is almost surely bounded as

$$\begin{gathered} {\Vert {\mathrm{\Lambda }}_l\Vert }_2={\Vert {\mathrm{\Psi }}_l^H{\mathrm{\Psi }}_l-\mathbb{E}\left({\mathrm{\Psi }}_l^H{\mathrm{\Psi }}_l\right)\Vert }_2 \\ \le {\Vert {\mathrm{\Psi }}_l^H{\mathrm{\Psi }}_l\Vert }_2+{\Vert \mathbb{E}\left({\mathrm{\Psi }}_l^H{\mathrm{\Psi }}_l\right)\Vert }_2\text{(Triangle}\text{ineq}.) \\ \le {\Vert {\mathrm{\Psi }}_l^H{\mathrm{\Psi }}_l\Vert }_2+\mathbb{E}{\Vert {\mathrm{\Psi }}_l^H{\mathrm{\Psi }}_l\Vert }_2\text{(Jensen}’\text{s ineq}.) \\ ={\Vert {\mathrm{\Psi }}_l\Vert }_2^2+\mathbb{E}{\Vert {\mathrm{\Psi }}_l\Vert }_2^2\le 2{\gamma }^{2}R. \end{gathered}$$

Thus, the Matrix Bernstein inequality [16, Cor. 8.15] yields that for any t > 0,

$$\mathbb{P}\left\{{\Vert \sum _{l=1}^L{\mathrm{\Lambda }}_l\Vert }_2\ge t\right\}\le 2R\text{exp}\left\{\frac{-t^2/2}{L{\left(2{\gamma }^{2}R\right)}^2+2{\gamma }^{2}Rt/3}\right\},$$ (23)

where we use the following simplification:

$${\Vert \sum _{l=1}^L\mathbb{E}{\mathrm{\Lambda }}_l^2\Vert }_2=L{\Vert \mathbb{E}{\mathrm{\Lambda }}_l^2\Vert }_2\le L\mathbb{E}{\Vert {\mathrm{\Lambda }}_l\Vert }_2^2\le L{\left(2{\gamma }^{2}R\right)}^2.$$

Applying (23) with t = 2γ²RLδ to the square of (22) yields

$${\lambda }_{\text{min}}^2\left(\sum _{l=1}^L{\mathrm{\Psi }}_l^H{\mathrm{\Psi }}_l\right)\ge L^2{\left\{{\lambda }_{\text{min}}(\overline{\mathrm{\Lambda }})-2{\gamma }^{2}R\delta \right\}}^2.$$ (24)

with probability at least $1-2R\text{exp}\left(-L\frac{{\delta }^2/2}{1+\delta /3}\right)$.

C. Combined bound

Combining the bounds (21) and (24) via a union bound yields (9) with probability at least

$$1-\text{exp}\left(-L\frac{{\delta }^2/2}{3+\delta /3}\right)-2R\text{exp}\left(-L\frac{{\delta }^2/2}{1+\delta /3}\right),$$ (25)

which is greater than or equal to (10).