Hutch++: Optimal Stochastic Trace Estimation

Abstract

We study the problem of estimating the trace of a matrix A that can only be accessed through matrix-vector multiplication. We introduce a new randomized algorithm, Hutch++, which computes a (1 ± ε) approximation to tr(A) for any positive semidefinite (PSD) A using just O(1/ε) matrix-vector products. This improves on the ubiquitous Hutchinson’s estimator, which requires O(1/ε²) matrix-vector products. Our approach is based on a simple technique for reducing the variance of Hutchinson’s estimator using a low-rank approximation step, and is easy to implement and analyze. Moreover, we prove that, up to a logarithmic factor, the complexity of Hutch++ is optimal amongst all matrix-vector query algorithms, even when queries can be chosen adaptively. We show that it significantly outperforms Hutchinson’s method in experiments. While our theory requires A to be positive semidefinite, empirical gains extend to applications involving non-PSD matrices, such as triangle estimation in networks.

1. Introduction

A ubiquitous problem in numerical linear algebra is that of approximating the trace of a d×d matrix A that can only be accessed via matrix-vector multiplication queries. In other words, we are given access to an oracle that can evaluate Ax for any $\mathbf{x}\in {ℝ}^d$, and the goal is to return an approximation to tr(A) using as few queries to this oracle as possible. An exact solution can be obtained with d queries because $\mathrm{tr}(\mathit{A})={\sum }_{i=1}^d{\mathit{A}}_{ii}={\sum }_{i=1}^d{\mathbf{e}}_i^T\mathit{A}{\mathbf{e}}_i$, where e_i denotes the i^th standard basis vector. The goal is thus to develop algorithms that use far fewer than d matrix-vector multiplications.

Known as implicit or matrix free trace estimation, this problem arises in applications that require the trace of a matrix A, where A is itself a transformation of some other matrix B. For example, A = B^q, A = B⁻¹, or A = exp(B). In all of these cases, explicitly computing A would require roughly O(d³) time, whereas multiplication with a vector x can be implemented more quickly using iterative methods. For example, B^qx can be computed in just O(d²) time for constant q, and for well-conditioned matrices, B⁻¹x and exp(B)x can also be computed in O(d²) time using the conjugate gradient or Lanczos methods [Hig08]. Implicit trace estimation is used to approximate matrix norms [HMAS17, MNS⁺18], spectral densities [LSY16, CKSV18, BKKS20], log-determinants [BDKZ15, HMS15], the Estrada index [US18, WSMB20], eigenvalue counts in intervals [DNPS16], triangle counts in graphs [Avr10], and much more [Che16, LSTZ20]. In these applications, we typically have that A is symmetric, and often positive semidefinite (PSD).

1.1. Hutchinson’s Estimator

The most common method for implicit trace estimation is Hutchinson’s stochastic estimator [Hut90]. This elegant randomized algorithm works as follows: let $\mathit{G}=\left[{\text{g}}_1,\dots ,{\text{g}}_m\right]\in {ℝ}^{d\times m}$ be a matrix containing i.i.d. random variables with mean 0 and variance 1. A simple calculation shows that $\mathbb{E}\left[{\mathbf{g}}_i^T\mathit{A}{\mathbf{g}}_i\right]=\mathrm{tr}(\mathit{A})$ for each ${\mathbf{g}}_i\in {ℝ}^d$, and ${\mathbf{g}}_i^T\mathit{A}{\mathbf{g}}_i$ can be computed with just one matrix-vector multiplication. So to approximate tr(A), Hutchinson’s estimator returns the following average:

$$\mathrm{Hutchinson}’\mathrm{s}\mathrm{Estimator}:{\text{H}}_m(\mathit{A})=\frac{1}{m}\sum _{i=1}^m{\mathbf{g}}_i^T\mathit{A}{\mathbf{g}}_i=\frac{1}{m}\mathrm{tr}\left({\mathit{G}}^T\mathit{AG}\right).$$ (1)

Hutchinson’s original work suggests using random ±1 sign vectors for g₁, …, g_m, and an earlier paper by Girard suggests standard normal random variables [Gir87]. Both choices perform similarly, as both random variables are sub-Gaussian. For vectors with sub-Gaussian random entries, it can be proven that, when A is positive semidefinite, (1−ε)tr(A) ≤ H_m(A) ≤ (1+ε)tr(A) with probability ≥ 1 − δ if we use $m=O\left(\log \right(\raisebox{1ex}{$1$}\!\left/ \!\raisebox{-1ex}{$\delta $}\right.)/{\epsilon }^2)$ matrix-vector multiplication queries [AT11, RA15].¹ For constant δ (e.g., $\delta =\raisebox{1ex}{$1$}\!\left/ \!\raisebox{-1ex}{$10$}\right.$) the bound is O(1/ε²).

1.2. Our results

Since Hutchinson’s work, and the non-asymptotic analysis in [AT11], there has been no improvement on this O(1/ε²) matrix-vector multiplication bound for trace approximation. Our main contribution is a quadratic improvement: we provide a new algorithm, Hutch++, that obtains the same (1 ± ε) guarantee with O(1/ε) matrix-vector multiplication queries. This algorithm is nearly as simple as the original Hutchinson’s method, and can be implemented in just a few lines of code.

Algorithm 1.

Hutch++

input: Matrix-vector multiplication oracle for PSD matrix $\mathit{A}\in {ℝ}^{n\times n}$. Number m of queries.

output: Approximation to tr(A).

1: Sample $\mathit{S}\in {ℝ}^{n\times \frac{m}{3}}$ and $\mathit{G}\in {ℝ}^{n\times \frac{m}{3}}$ with i.i.d. {+1, −1} entries.

2: Compute an orthonormal basis $\mathit{Q}\in {ℝ}^{n\times \frac{m}{3}}$ for the span of AS (e.g., via QR decomposition).

3: return Hutch++ $(\mathit{A})=\mathrm{tr}\left({\mathit{Q}}^T\mathit{A}\mathit{Q}\right)+\frac{3}{m}\mathrm{tr}\left({\mathit{G}}^T\left(\mathit{I}-\mathit{Q}{\mathit{Q}}^T\right)\mathit{A}\left(\mathit{I}-\mathit{Q}{\mathit{Q}}^T\right)\mathit{G}\right)$.

Hutch++ requires m matrix-vector multiplications with A: m/3 to compute A·S, m/3 to compute A · Q, and m/3 to compute A · (I − QQ^T )G. It requires O(dm²) additional runtime to compute the basis Q and the product (I − QQ^T )G = G − QQ^T G. For concreteness, we state the method with random sign matrices, but the entries of S and G can be any sub-Gaussian random variables with mean 0 and variance 1, including e.g., standard Gaussians. Our main theorem on Hutch++ is:

Theorem 1.

If Hutch++ is implemented with $m=O(\sqrt{\log (1/\delta )}/\epsilon +\log (1/\delta ))$ matrix-vector multiplication queries, then for any PSD A, with probability ≥ 1 − δ, the output Hutch⁺⁺(A) satisfies:

$$(1-\epsilon )\mathrm{tr}(\mathit{A})\le Hutch++(\mathit{A})\le (1+\epsilon )\mathrm{tr}(\mathit{A}).$$

Hutch++ can be viewed as a natural variance reduced version of Hutchinson’s estimator. The method starts by computing an orthonormal span $\mathit{Q}\in {ℝ}^{d\times \frac{m}{3}}$ by running a single iteration of power method with a random start matrix S. Q coarsely approximates the span of A’s top eigenvectors. Then we separate A into its projection onto the subspace spanned by Q, and onto that subspace’s orthogonal compliment, writing tr(A) = tr(QQ^T AQQ^T ) + tr ((I − QQ^T )A(I − QQ^T)). By the cyclic property of the trace, the first term is equal to tr(Q^T AQ), which is computed exactly by Hutch++ with $\frac{m}{3}$ matrix-vector multiplications. The second term is approximated using Hutchinson’s estimator with the random vectors in G.

Thus, the error in estimating tr(A) is entirely due to approximating this second term. The key observation is that the variance when estimating this term is much lower than when estimating tr(A) directly. Specifically, it is proportional to ${\Vert \left(\mathit{I}-\mathit{Q}{\mathit{Q}}^T\right)\mathit{A}\left(\mathit{I}-\mathit{Q}{\mathit{Q}}^T\right)\Vert }_F^2$, which, using standard tools from randomized linear algebra [CEM⁺15, Woo14], we can show is bounded by ε tr(A)² with good probability when m = O(1/ε). This yields our improvement over Hutchinson’s method applied directly to A, which has variance bounded by tr(A)². The full proof of Theorem 1 is in Section 3.

Algorithm 1 is adaptive: it multiplies A by a sequence of query vectors r₁, …, r_m, where later queries depend on earlier ones. In contrast, Hutchinson’s method is non-adaptive: r₁, …, r_m are chosen in advance, before computing any of the products Ar₁, …, Ar_m. In addition to Algorithm 1, we give a non-adaptive variant of Hutch++ that obtains the same O(1/ε) bound. We complement these results with a nearly matching lower bound, proven in Section 4. Specifically, via a reduction from the Gap-Hamming problem from communication complexity, we show that any matrix-vector query algorithm whose queries have bounded bit complexity requires $m=\mathrm{\Omega }\left(\frac{1}{\epsilon \log (1/\epsilon )}\right)$ queries to estimate the trace of a PSD matrix up to a (1 ± ε) multiplicative approximation. We also prove a tight $m=\mathrm{\Omega }\left(\frac{1}{\epsilon }\right)$ lower bound for non-adaptive algorithms in the real RAM model of computation.

Empirical Results.

In Section 5 we complement our theoretical results with experiments on synthetic and real-world matrices, including applications of trace estimation to approximating log determinants, the graph Estrada index, and the number of triangles in a graph. We demonstrate that Hutch++ improves substantially on Hutchinson’s estimator, and on related estimators based on approximating the top eigenvalues of A. While our theory applies to positive semidefinite matrices, Hutch++ can be applied unmodified to non-positive semidefinite trace estimation, and continues to perform very well empirically. We note that Hutch++ is simple to implement and essentially parameter free – the only choice needed is the number of matrix-vector multiplication queries m.

1.3. Prior Work

Upper bounds.

A nearly tight non-asymptotic analysis of Hutchinson’s estimator for positive semidefinite matrices was given by Avron and Toledo using an approach based on reducing to Johnson-Lindenstrauss random projection [AT11, DG03, Ach03]. A slightly tighter approach from [RA15] obtains a (1±ε) multiplicative error bound with m = O(1/ε²) matrix-vector multiplication queries. This bound is what we improve on with Hutch++.

A number of papers suggest variance reduction schemes for Hutchinson’s estimator. Some take advantage of sparsity structure in A [TS11, SLO13] and others use a “decomposition” approach similar to Hutch++ [APJ⁺18]. Most related to our work are two papers which, like Hutch++, perform the decomposition by projecting onto some Q that approximately spans A’s top eigenspace [GSO17, Lin17]. The justification is that this method should perform much better than Hutchinson’s when A is close to low-rank, because tr(Q^T AQ) will capture most of A’s trace. Our contribution is an analysis of this approach which 1) improves on Hutchinson’s even when A is far from low-rank and 2) shows that a very coarse approximation to the top eigenvectors suffices (computed using one iteration of the power method). Finally, we note two papers which directly use the approximation tr(A) ≈ tr(Q^T AQ), where Q is computed with a randomized SVD method [SAI17, HL20]. Of course, this approach works best for nearly-low rank matrices.

Lower bounds.

Our lower bounds extend a recent line of work on lower bounds for linear algebra problems in the “matrix-vector query model” [SEAR18, SWYZ19, BHSW20]. [WWZ14] proves a lower bound of Ω(1/ε²) queries for PSD trace approximation in an alternative model that allows for adaptive “quadratic form” queries:${\mathbf{r}}_1^T\mathit{A}{\mathbf{r}}_1,\dots ,{\mathbf{r}}_m^T\mathit{A}{\mathbf{r}}_m$. This model captures Hutchinson’s estimator, but not Hutch++, which is why we are able to obtain an upper bound of O(1/ε) queries.

2. Preliminaries

Notation.

For $\mathbf{a}\in {ℝ}^d$, $\Vert \mathbf{a}{\Vert }_2={\left({\sum }_{i=1}^d{a}_i^2\right)}^{1/2}$ denotes the ℓ₂ norm and $\Vert \mathbf{a}{\Vert }_1={\sum }_{i=1}^d\left|a_i\right|$ denotes the ℓ₁ norm. For $\mathit{A}\in {ℝ}^{n\times d}$, ${\Vert \mathit{A}\Vert }_F={\left({\sum }_{i=1}^d{\sum }_{j=1}^d{\mathit{A}}_{ij}^2\right)}^{1/2}$ denotes the Frobenius norm. For square $\mathit{A}\in {ℝ}^{d\times d}$, $\mathrm{tr}(\mathit{A})={\sum }_{i=1}^d{\mathit{A}}_{ii}$ denotes the trace. Our main results on trace approximation are proven for symmetric positive semidefinite (PSD) matrices, which are the focus of most applications. Any symmetric $\mathit{A}\in {ℝ}^{d\times d}$ has an eigendecomposition A = V ΛV ^T, where $V\in {ℝ}^{d\times d}$ is orthogonal and Λ is a real-valued diagonal matrix. We let λ = diag(Λ) be a length d vector containing A’s eigenvalues in descending order: λ₁ ≥ λ₂ ≥ … ≥ λ_d. When A is PSD, λ_i ≥ 0 for all i. We use the identities tr(A) = ‖λ‖₁ and ‖A‖_F = ‖λ‖₂. Finally, we let A_k = argmin_B,rank(B)=k ‖A − B‖_F denote the optimal k-rank approximation to A. For a PSD matrix $\mathit{A},{\mathit{A}}_k=V_k{\mathbf{\Lambda }}_k{V}_k^T$, where $V_k\in {ℝ}^{d\times k}$ contains the first k columns of V and Λ_k is the k × k top left submatrix of Λ.

Hutchinson’s Analysis.

We require a standard bound on the accuracy of Hutchinson’s estimator:

Lemma 2.

Let $\mathit{A}\in {ℝ}^{d\times d}$, δ ∈ (0, ¹/2],$\ell \in \mathbb{N}$. Let H_ℓ(A) be the ℓ-query Hutchinson estimator defined in (1), implemented with mean 0, i.i.d. sub-Gaussian random variables with constant sub-Gaussian parameter. For fixed constants c, C, if ℓ > clog(¹/δ), then with probability ≥ 1 − δ,

$$\left|{\text{H}}_{\ell }(\mathit{A})-\mathrm{tr}(\mathit{A})\right|\le C\sqrt{\frac{\log (1/\delta )}{\ell }}\Vert \mathit{A}{\Vert }_F.$$

So, if $\ell =O\left(\frac{\log (1/\delta )}{{\epsilon }^2}\right)$ then, with probability ≥ 1 − δ, |H_ℓ(A) − tr(A)| ≤ ε‖A‖_F.

We refer the reader to [RV⁺13] for a formal definition of sub-Gaussian random variables: both normal $\mathcal{N}(0,1)$ random variables and ±1 random variables are sub-Gaussian with constant parameter. Lemma 2 is proven in Appendix A for completeness. It is slightly more general than prior work [RA15] in that it applies to non-PSD, and even asymmetric matrices, which will be important in the analysis of our non-adaptive algorithm. A similar result was recently shown in [CK20].

3. Complexity Analysis

We start by providing the technical intuition behind Hutch++. First note that, for a PSD matrix with eigenvalues λ, ‖A‖_F ≤ tr(A), so Lemma 2 immediately implies that Hutchinson’s estimator obtains a relative error guarantee with O(1/ε²) queries. However, this bound is only tight when ‖λ‖₂ ≈ ‖λ‖₁, i.e., when A has significant mass concentrated on just a small number of eigenvalues.

Hutch++ simply eliminates this possibility by approximately projecting off A’s large eigenvalues using a projection QQ^T. By doing so, it only needs to compute a stochastic estimate for the trace of (I−QQ^T )A(I−QQ^T ). The error of this estimate is proportional to ‖(I−QQ^T )A(I−QQ^T )‖_F, which we show is always much smaller than tr(A). In particular, suppose that Q = V_k exactly spanned the top k eigenvectors A and thus (I − QQ^T )A(I − QQ^T ) = A − A_k. Then we have:

Lemma 3.

Let A_k be the best rank-k approximation to PSD matrix A. Then,

$${\Vert \mathit{A}-{\mathit{A}}_k\Vert }_F\le \frac{1}{\sqrt{k}}\mathrm{tr}(\mathit{A}).$$

Proof.

We have ${\lambda }_{k+1}\le \frac{1}{k}{\sum }_{i=1}^k{\lambda }_i\le \frac{1}{k}\mathrm{tr}(\mathit{A})$, so:

$${\Vert \mathit{A}-{\mathit{A}}_k\Vert }_F^2=\sum _{i=k+1}^d{\lambda }_i^2\le {\lambda }_{k+1}\sum _{i=k+1}^d{\lambda }_i\le \frac{1}{k}\mathrm{tr}(\mathit{A})\sum _{i=k+1}^d{\lambda }_i\le \frac{1}{k}\mathrm{tr}{(\mathit{A})}^2.$$

This result immediately suggests the possibility of an algorithm with O(1/ε) query complexity: Set k = O(1/ε) and split tr(A) = tr(A_k) + tr(A − A_k). The first term can be computed exactly with O(1/ε) matrix-vector multiplication queries if V_k is known, since $\mathrm{tr}\left({\mathit{A}}_k\right)=\mathrm{tr}\left(V_k^T\mathit{A}{V}_k\right)$. By Lemma 3 combined with Lemma 2, the second can be estimated to error ±εtr(A) using just O(1/ε) queries instead of O(1/ε²). Of course, we can’t compute V_k exactly with a small number of matrix-vector multiplication queries, but this is easily resolved by using an approximate projection. Using standard tools from randomized linear algebra, O(k) queries suffices to find a Q with ‖(I − QQ^T )A(I − QQ^T )‖_F ≤ O(‖A − A_k‖_F ), which is all that is needed for a O(1/ε) query result.

Concretely, we use Lemma 3 to prove the following general theorem, from which Theorem 1 and our non-adaptive algorithmic result will follow as direct corollaries.

Theorem 4.

Let $\mathit{A}\in {ℝ}^{d\times d}$ be PSD, δ ∈ (0, ¹/₂), $\ell \in \mathbb{N}$,$k\in \mathbb{N}$. Let $\tilde{\mathit{A}}$ and Δ be any matrices with:

$$\mathrm{tr}(\mathit{A})=\mathrm{tr}(\tilde{\mathit{A}})+\mathrm{tr}(\mathbf{\Delta })\text{and}\Vert \mathbf{\Delta }{\Vert }_F\le 2{\Vert \mathit{A}-{\mathit{A}}_k\Vert }_F.$$

For fixed constants c, C, if ℓ > clog(¹/δ), then with probability 1−δ, $\mathit{Z}=\left[\mathrm{tr}(\tilde{\mathit{A}})+{\text{H}}_{\ell }(\mathbf{\Delta })\right]$ satisfies:

$$|\mathit{Z}-\mathrm{tr}(\mathit{A})|\le 2C\sqrt{\frac{\log (1/\delta )}{k\ell }}\cdot \mathrm{tr}(\mathit{A}).$$

In particular, if $k=\ell =O\left(\frac{\sqrt{\log (1/\delta )}}{\epsilon }+\log (1/\delta )\right)$, Z is a (1 ± ε) error approximation to tr(A).

Proof.

We have with probability ≥ 1 − δ:

$$\begin{gathered} |\mathit{Z}-\mathrm{tr}(\mathit{A})|=\left|{\text{H}}_{\ell }(\mathbf{\Delta })-\mathrm{tr}(\mathbf{\Delta })\right|\text{(since}\mathit{Z}=\mathrm{tr}(\tilde{\mathit{A}})+{\text{H}}_{\ell }(\mathbf{\Delta })\text{and}\mathrm{tr}(\mathit{A})=\mathrm{tr}(\tilde{\mathit{A}})+\mathrm{tr}(\mathbf{\Delta })) \\ \le C\sqrt{\frac{\log (1/\delta )}{\ell }\Vert \mathbf{\Delta }{\Vert }_F}(\text{by}\text{the}\text{standard}\mathrm{Hutchinson’s}\text{analysis},\text{Lemma 2}) \\ \le 2C\sqrt{\frac{\log (1/\delta )}{\ell }}{\Vert \mathit{A}-{\mathit{A}}_k\Vert }_F\text{(by}\text{the}\text{assumption}\text{that}\Vert \mathbf{\Delta }{\Vert }_F\le 2{\Vert \mathit{A}-{\mathit{A}}_k\Vert }_F) \\ \le 2C\sqrt{\frac{\log (1/\delta )}{k\ell }}\mathrm{tr}(\mathit{A}).\text{(by}\text{Lemma}\text{3)} \end{gathered}$$

As discussed, Theorem 4 would immediately yield an O(1/ε) query algorithm if we knew an optimal k-rank approximation for A. Since computing one is infeasible, our first version of Hutch++ (Algorithm 1) instead uses a projection onto a subspace Q which is computed with one iteration of the power method. We have:

Theorem 1 Restated.

If Algorithm 1 is implemented with $m=O(\sqrt{\log (1/\delta )}/\epsilon +\log (1/\delta ))$ matrix-vector multiplication queries, then for any PSD A, with probability ≥ 1−δ, the output Hutch⁺⁺(A) satisfies: (1 − ε)tr(A) ≤ Hutch⁺⁺(A) ≤ (1 + ε)tr(A).

Proof.

Let S, G, and Q be as in Algorithm 1. We instantiate Theorem 4 with $\tilde{\mathit{A}}={\mathit{Q}}^T\mathit{A}\mathit{Q}$ and Δ = (I − QQ^T )A(I − QQ^T ). Note that, since Q is orthogonal, (I − QQ^T ) is a projection matrix, so (I − QQ^T ) = (I − QQ^T )². This fact, along with the cyclic property of the trace, gives:

$$\mathrm{tr}(\tilde{\mathit{A}})=\mathrm{tr}\left(\mathit{A}\mathit{Q}{\mathit{Q}}^T\right)\text{and}\mathrm{tr}(\mathbf{\Delta })=\mathrm{tr}\left(\mathit{A}\left(\mathbf{I}-\mathit{Q}{\mathit{Q}}^T\right)\right),$$

and thus $\mathrm{tr}(\tilde{\mathit{A}})+\mathrm{tr}(\mathbf{\Delta })=\mathrm{tr}(\mathit{A})$ as required by Theorem 4. Furthermore, since multiplying by a projection matrix can only decrease Frobenius norm,$\Vert \mathrm{\Delta }{\Vert }_F^2\le {\Vert \mathit{A}\left(\mathit{I}-\mathit{Q}{\mathit{Q}}^T\right)\Vert }_F^2=\Vert \mathit{A}-\mathit{A}\mathit{Q}\mathit{Q}{\Vert }_F^2$.

Recall that Q is an orthogonal basis for the column span of AS, where S is a random sign matrix with $\frac{m}{3}$ columns. Q is thus an orthogonal basis for a linear sketch of A’s column space, and it is well known that Q will align with large eigenvectors of A, and $\Vert \mathit{A}-\mathit{A}\mathit{Q}\mathit{Q}{\Vert }_F^2$ will be small [Sar06, Woo14]. Concretely, applying Corollary 7 and Claim 1 from [MM20], we have that, as long as $\frac{m}{3}\ge O(k+\log (1/\delta ))$, with probability ≥ 1 − δ:

$${\Vert \mathit{A}-\mathit{A}\mathit{Q}{\mathit{Q}}^T\Vert }_F^2\le 2{\Vert \mathit{A}-{\mathit{A}}_k\Vert }_F^2.$$

Accordingly, $\Vert \mathbf{\Delta }{\Vert }_F^2\le 2{\Vert \mathit{A}-{\mathit{A}}_k\Vert }_F^2$ as required by Theorem 4. The result then immediately follows by setting $k=O(\sqrt{\log (1/\delta )}/\epsilon +\log (1/\delta ))$ and noting that $\text{Hutch}++(\mathit{A})=\left[\mathrm{tr}(\tilde{\mathit{A}})+{\text{H}}_{\ell }(\mathbf{\Delta })\right]$ where $\ell =O(\sqrt{\log (1/\delta )}/\epsilon +\log (1/\delta ))$. □

3.1. A Non-Adaptive Variant of Hutch++

As discussed in Section 1, Algorithm 1 is adaptive: it uses the result of computing AS to compute Q, which is then multiplied by A to compute the tr(Q^T AQ) term. Meanwhile, Hutchinson’s estimator is non-adaptive: it samples a single random matrix upfront, batch-multiplies by A once, and computes an approximation to tr(A) from the result, without any further queries.

Not only is non-adaptivity an interesting theoretical property, but it can be practically useful, since parallelism or block iterative methods often make it faster to multiply an implicit matrix by many vectors at once. With these considerations in mind, we describe a non-adaptive variant of Hutch++, which we call NA-Hutch++. NA-Hutch++ obtains nearly the same theoretical guarantees as Algorithm 1, although it tends to perform slightly worse in our experiments.

We leverage a streaming low-rank approximation result of Clarkson and Woodruff [CW09] which shows that if $\mathit{S}\in {ℝ}^{d\times m}$ and $R\in {ℝ}^{d\times cm}$ are sub-Gaussian random matrices with m = O(k log(1/_δ)) and c > 1 a fixed constant, then with probability 1 − δ, the matrix $\tilde{\mathit{A}}=\mathit{A}R{\left({\mathit{S}}^T\mathit{A}R\right)}^{+}{(\mathit{A}\mathit{S})}^T$ satisfies $\Vert \mathit{A}-\tilde{\mathit{A}}{\Vert }_F\le 2{\Vert \mathit{A}-{\mathit{A}}_k\Vert }_F$. Here ⁺ denotes the Moore-Penrose pseudoinverse. We can compute $\mathrm{tr}(\tilde{\mathit{A}})$ efficiently without explicitly constructing $\tilde{\mathit{A}}\in {ℝ}^{d\times d}$ by noting that it is equal to tr((S^T AR)⁺(AS)^T (AR)) via the cyclic property of the trace. This yields:

Algorithm 2.

NA-Hutch++ (Non-Adaptive variant of Hutch++)

input: Matrix-vector multiplication oracle for PSD matrix $\mathit{A}\in {ℝ}^{d\times d}$. Number m of queries.

output: Approximation to tr(A).

1: Fix constants c1, c2, c3 such that c1 < c2 and c1 + c2 + c3 = 1.

2: Sample $\mathit{S}\in {ℝ}^{d\times c_{1}m}$, $R\in {ℝ}^{d\times c_{2}m}$, and $\mathit{G}\in {ℝ}^{d\times c_{3}m}$ with i.i.d. {+1, −1} entries.

3: Compute Z = AR and W = AS.

4: return $\text{NA}-\text{Hutch}++(\mathit{A})=\mathrm{tr}\left({\left({\mathit{S}}^T\mathit{Z}\right)}^{+}\left(W^T\mathit{Z}\right)\right)+\frac{1}{c_{3}m}\left[\mathrm{tr}\left({\mathit{G}}^T\mathit{A}\mathit{G}\right)-\mathrm{tr}\left({\mathit{G}}^T\mathit{Z}{\left({\mathit{S}}^T\mathit{Z}\right)}^{+}{W}^T\mathit{G}\right)\right]$

NA-Hutch++ requires m matrix-vector multiplications with A. In our experiments, it works well with c₁ = c₃ = 1/4 and c₂ = 1/2. Assuming m < d, it requires O(dm²) further runtime, to perform the matrix multiplications on line 4 and to compute (S^T Z)⁺, which takes O(dm² + m³) time.

Theorem 5.

If NA-Hutch++ is implemented with m = O(log(¹/δ)/ε) matrix-vector multiplication queries and $\frac{c_2}{c_1}$ a sufficiently large constant, then for any PSD A, with probability ≥ 1 − δ, the output NA-Hutch++(A) satisfies: (1 − ε)tr(A) ≤ NA-Hutch++(A) ≤ (1 + ε)tr(A).

Proof.

We apply Theorem 4 with $\tilde{\mathit{A}}=\mathit{Z}{\left({\mathit{S}}^T\mathit{Z}\right)}^{+}{\mathbf{W}}^T$, $\mathbf{\Delta }=\mathit{A}-\tilde{\mathit{A}}$, k = O(1/ε) and $\ell =c_{3}m=O\left(\frac{\log (1/\delta )}{\epsilon }\right)$. $\mathrm{tr}(\mathit{A})=\mathrm{tr}(\tilde{\mathit{A}})+\mathrm{tr}(\mathbf{\Delta })$ and $\text{NA-Hutch++}(\mathit{A})=\left[\mathrm{tr}(\tilde{\mathit{A}})+{\text{H}}_{\ell }(\mathbf{\Delta })\right]$. By Theorem 4.7 of [CW09], since c₁m = O(k log(¹/δ)), ∥Δ∥_F ≤ 2∥A − A_k∥_F with probability ≥ 1 − δ as required. □

4. Lower Bounds

A natural question is if the O(1/ε) matrix-vector query bound of Theorem 1 and Theorem 5 is tight. In this section, we prove that it is up to a logarithmic factor, even for algorithms that perform adaptive queries like Hutch++. Our lower bound is via a reduction to communication complexity: we show that a better algorithm for PSD trace estimation would imply a better 2-party communication protocol for the Gap-Hamming problem, which would violate known adaptive lower bounds for that problem [CR12]. To prove this result we need to assume a fixed precision model of computation. Specifically we require that the entries in each query vector r are integers bounded in absolute value by 2^b, for some fixed constant b. By scaling, this captures the setting where the query vectors are non-integer, but have bounded precision. Formally, we prove in Section 4.1:

Theorem 6.

Any algorithm that accesses a positive semidefinite matrix A via matrix-vector multiplication queries Ar₁, …, Ar_m, where r₁, …, r_m are possibly adaptively chosen vectors with integer entries in {−2^b, …, 2^b}, requires $m=\mathrm{\Omega }\left(\frac{1}{\epsilon (b+\log (1/\epsilon ))}\right)$ such queries to output an estimate t so that, with $probability>2/3,(1-\epsilon )\mathrm{tr}(\mathit{A})\le t\le (1+\epsilon )\mathrm{tr}(\mathit{A})$.

For constant b our lower bound is $\mathrm{\Omega }\left(\frac{1}{\epsilon \log (1/\epsilon )}\right)$, which matches Theorem 1 and Theorem 5 up to a log(1/ε) factor. We also provide an alternative lower bound which holds in the real RAM model of computation (all inputs and arithmetic operations involve real numbers). This second lower bound is tight up to constants, but only applies to non-adaptive algorithms. It is proven using different information theoretic techniques – we reduce to a hypothesis testing problem involving negatively spiked covariance matrices [CMW15, PWBM18]. Formally, we prove in Appendix B:

Theorem 7.

Any algorithm that accesses a postive semidefinite matrix A through matrix-vector multiplication queries Ar₁, …, Ar_m, where r₁, …, r_m are real valued non-adaptively chosen vectors requires $m=\mathrm{\Omega }\left(\frac{1}{\epsilon }\right)$ such queries to output an estimate t so that, with probability > ¾, (1 − ε)tr(A) ≤ t ≤ (1 + ε) tr(A).

4.1. Adaptive lower bound

The proof of Theorem 6 is based on reducing the Gap-Hamming problem to trace estimation. This problem has been well studied in communication complexity since its introduction in [IW03].

Problem 1 (Gap-Hamming).

Let Alice and Bob be communicating parties who hold vectors s ∈ {−1, 1}ⁿ and t ∈ {−1, 1}ⁿ, respectively. The Gap-Hamming problem asks Alice and Bob to return:

$$\begin{array}{lll}1if\langle \text{s},\mathbf{t}\rangle \ge \sqrt{n}\hfill & and\hfill & -1if\langle \text{s},\mathbf{t}\rangle \le -\sqrt{n}.\hfill \end{array}$$

A tight lower bound on the unbounded round, randomized communication complexity of this problem was first proven in [CR12], with alternative proofs appearing in [Vid12, She12]. Formally:

Lemma 8 (Theorem 2.6 in [CR12]).

The randomized communication complexity for solving Problem 1 with probability ≥ 2/3 is Ω(n) bits.

With Lemma 8 in place, we have all we need to prove Theorem 6.

Proof of Theorem 6.

Fix a perfect square $n\in \mathbb{N}$. Consider an instance of Problem 1 with inputs $\mathbf{s}\in {ℝ}^n$ and $\mathbf{t}\in {ℝ}^n$. Let $\mathit{S}\in {ℝ}^{\sqrt{n}\times \sqrt{n}}$ and $T\in {ℝ}^{\sqrt{n}\times \sqrt{n}}$ contain the entries of s and t rearranged into matrices (e.g., placed left-to-right, top-to-bottom). Let Z = S + T and let A = Z^T Z. A is positive semidefinite and we have:

$$\mathrm{tr}(\mathit{A})=\Vert \mathit{Z}{\Vert }_F^2=\Vert \text{s}+\mathbf{t}{\Vert }_2^2=\Vert \text{s}{\Vert }_2^2+\Vert \mathbf{t}{\Vert }_2^2+2\langle \text{s},\mathbf{t}\rangle =2n+2\langle \text{s},\mathbf{t}\rangle .$$ (2)

If $\langle s,t\rangle \ge \sqrt{n}$ then we will have $\mathrm{tr}(\mathit{A})\ge 2(n+\sqrt{n})$ and if $\langle s,t\rangle \ge -\sqrt{n}$ then we will have $\mathrm{tr}(\mathit{A})\le 2(n-\sqrt{n})$. So, if Alice and Bob can approximate tr(A) up to relative error $(1\pm 1/\sqrt{n})$, then they can solve Problem 1. We claim that they can do so with just $O(m\cdot \sqrt{n}(\log n+b))$ bits of communication if there exists an m-query adaptive matrix-vector multiplication algorithm for positive semidefinite trace estimation achieving error $(1\pm 1/\sqrt{n})$.

Specifically, Alice takes charge of running the query algorithm. To compute Ar for a vector r, Alice and Bob first need to compute Zr. To do so, Alice sends r to Bob, which takes $O(\sqrt{n}\cdot b)$ bits since r has entries bounded by 2^b. Bob then computes Tr, which has entries bounded by $\sqrt{n}{2}^b$. He sends the result to Alice, using $O(\sqrt{n}(b+\log n))$ bits. Upon receiving Tr, Alice computes Zr = Sr + Tr. Next, they need to multiply Zr by Z^T to obtain Ar = Z^T Zr. To do so, Alice sends Zr to Bob (again using $O(\sqrt{n}(b+\log n))$ bits) who computes T^T Zr. The entries in this vector are bounded by 2n2^b, so Bob sends the result back to Alice using $O(\sqrt{n}(b+\log n))$ bits. Finally, Alice computes S^T Zr and adds the result to T^T Zr to obtain Z^T Zr = Ar. Given this result, Alice chooses the next query vector according to the algorithm and repeats.

Overall, running the full matrix-vector query algorithm requires $O(m\cdot \sqrt{n}(\log n+b))$ bits of communication. So, from Lemma 8 we have that $m=\mathrm{\Omega }(\sqrt{n}/(\log n+b))$ queries are needed to approximate the trace to accuracy 1 ± ε for $\epsilon =1/\sqrt{n}$, with $\text{probability}>2/3$. □

5. Experimental Validation

We complement our theory with experiments on synthetic matrices and real-world trace estimation problems. We compare four algorithms, including both our adaptive and non-adaptive methods:

• Hutchinson’s. The standard estimator run with {+1, −1} random vectors.

• Subspace Projection. The method from [SAI17], which computes an orthogonal matrix $\mathit{Q}\in {ℝ}^{d\times k}$ that approximately spans the top eigenvector subspace of $\mathit{A}\in {ℝ}^{d\times d}$ and returns tr(Q^T AQ) as an approximation to tr(A). A similar approach is employed in [HL20]. [SAI17] computes Q using subspace iteration, which requires k(q + 1) matrix-vector multiplications when run for q iterations. A larger q results in a more accurate Q, but requires more multiplications. As in [SAI17], we found that setting q = 1 gave the best performance, so we did so in our experiments. With q = 1, this method is similar to Hutch++, except that is does not approximate the remainder of the trace outside the top eigenspace.

• Hutch++. The adaptive method of Algorithm 1 with {+1, −1} random vectors.

• NA-Hutch++. The non-adaptive method of Algorithm 2 with c₁ = c₃ = ¼ and c₂ = 1/2 and {+1, −1} random vectors.

5.1. Synthetic Matrices

We first test the methods above on random matrices with power law spectra. For varying constant c, we let Λ be diagonal with Λ_ii = i^−c. We generate a random orthogonal matrix $\mathit{Q}\in {ℝ}^{5000\times 5000}$ by orthogonalizing a random Gaussian matrix and set A = Q^T ΛQ. A’s eigenvalues are the values in Λ. A larger c results in a more quickly decaying spectrum, so we expect Subspace Projection to perform well. A smaller c results in a slowly decaying spectrum, which will mean that ∥A∥_F ≪tr(A). In this case, we expect Hutchinson’s to outperform its worst case multiplicative error bound: instead of error ±εtr(A) after O(1/ε²) matrix-multiplication queries, Lemma 2 predicts error on the order of ±ε∥A∥_F. Concretely, for dimension d = 5000 and c = 2, we have ∥A∥_F = .63 · tr(A) and for c = .5 we have ∥A∥_F = .02 · tr(A). In general, unlike the Subspace Projection method and Hutchinson’s estimator, we expect Hutch++ and NA-Hutch++ to be less sensitive to A’s spectrum.

In Figure 1 we plot results for various c. Relative error should scale roughly as ε = O(m^−γ), where γ = ½ for Hutchinson’s and γ = 1 for Hutch++ and NA-Hutch++. We thus use log-log plots, where we expect a linear relationship between the error ε and number of iterations m.

Figure 1:

Relative error versus number of matrix-vector multiplication queries for trace approximation algorithms run on random matrices with power law spectra. We report the median relative error of the approximation t after 200 trials. The upper and lower bounds of the shaded region around each curve are the 25^th and 75^th percentile errors. Subspace Projection has consistently low variance, but as expected, only performs better than Hutchinson’s when c = 2 and there is very fast eigenvalue decay. Hutch++ and NA-Hutch++ typically outperform both methods.

The superior performance of Hutch++ and NA-Hutch++ shown in Figure 1 is not surprising. These methods are designed to achieve the “best of both worlds”: when A’s spectrum decays quickly, our methods approximate tr(A) well by projecting off the top eigenvalues. When it decays slowly, they perform essentially no worse than Hutchinson’s. We note that the adaptivity of Hutch++ leads to consistently better performance over NA-Hutch++, and the method is simpler to implement as we do not need to set the constants c₁, c₂, c₃. Accordingly, this is the method we move forward with in our real data experiments.

5.2. Real Matrices

To evaluate the real-world performance of Hutch++ we test it in the common setting where A = f(B). In most applications, B is symmetric with eigendecomposition V^T ΛV, and $f:ℝ\to ℝ$ is a function on real valued inputs. Then we have f(B) = V^T f(Λ)V where f(Λ) is simply f applied to the real-valued eigenvalues on the diagonal of Λ. When f returns negative values, A may not be postive semidefinite. Generally, computing f(B) explicitly requires a full eigendecomposition and thus Ω(n³) time. However, many iterative methods can more quickly approximate matrix-vector queries of the form Ar = f(B)r. The most popular and general is the Lanczos method, which we employ in our experiments [UCS17, MMS18].²

We consider trace estimation in three example applications, involving both PSD and non-PSD matrices. We test on relatively small inputs, for which we can explicitly compute tr(f(B)) to use as a baseline for the approximation error. However, our methods can scale to much larger matrices.

Graph Estrada Index.

Given the binary adjacency matrix B ∈ {0, 1}^d×d of a graph G, the Estrada index is defined as tr(exp(B)) [Est00, dlPGR07], where exp(x) = e^x. This index measures the strength of connectivity within G. A simple transformation of the Estrada index yields the natural connectivity metric, defined as $\log \left(\frac{1}{d}\mathrm{tr}(\exp (B))\right)$ [JBYJHZ10, EHB12].

In our experiments, we approximated the Estrada index of the Roget’s Thesaurus semantic graph, available from [BM06]. The Estrada index of this 1022 node graph was originally studied in [EH08]. We use the Lanczos method to approximate matrix multiplication with exp(B), running it for 40 iterations, after which the error of application was negligible compared to the approximation error of trace estimation. Results are shown in Figure 2.

Figure 2:

Relative error versus number of matrix-vector multiplication queries for trace approximations of transformed matrices, which were multiplied by vectors using the Lanczos method. We report median relative error of the approximation t after 100 trials. The upper and lower bounds of the shaded region around each curve are the 25^th and 75^th percentile errors. As expected, Subspace Project and Hutch++ outperform Hutchinson’s when A = exp(B), as exponentiating leads to a quickly decaying spectrum. On the other hand, Hutchinson’s performs well for A = log(B + λI), which has a very flat spectrum. Hutch++ is still essentially as fast, even though this matrix is not PSD. Subspace Project fails in this case because the top eigenvalues of A do not dominate its trace.

Gaussian Process Log Likelihood.

Let $B\in {ℝ}^{d\times d}$ be a PSD kernel covariance matrix and let λ ≥ 0 be a regularization parameter. In Gaussian process regression, the model log likelihood computation requires computing logdet(B + λI) = tr(f(B)) where f(x) = log(x + λ) [WR96, Ras04]. This quantity must be computed repeatedly for different choices of B and λ during hyperparameter optimization, and it is often approximated using Hutchinson’s method [BDKZ15, UCS17, HMAS17, DEN⁺17]. We note that, while B is positive semidefinite, log(B +λI) typically will not be. So our theoretical bounds do not apply in this case, but Hutch++ can be applied unmodified, and as we see in Figure 2, still gives good performance.

In our experiments we consider a benchmark 2D Gaussian process regression problem from the GIS literature, involving precipitation data from Slovakia [NM13]. B is the kernel covariance matrix on 6400 randomly selected training points out of 196,104 total points. Following the setup of [EMM20], we let B be a Gaussian kernel matrix with width parameter γ = 64 and regularization parameter λ = .008, both determined via cross-validation on ℓ₂ regression loss.

Graph Triangle Counting.

Given the binary adjacency matrix B ∈ {0, 1}^d×d of an undirected graph G, the number of triangles in G is equal to $\frac{1}{6}\mathrm{tr}\left(B^3\right)$. The triangle count is an important measure of local connectivity and extensive research studies its efficient approximation [SW05, BBCG08, PT12]. Popular approaches include applying Hutchinson’s method to A = B³ [Avr10], or using the EigenTriangle estimator, which is similar to the Subspace Projection method [Tso08].

In our experiments, we study approximate triangle counting on two common benchmark graphs: an arXiv.org collaboration network³ with 5,243 nodes and 48,260 triangles, and a Wikipedia administrator voting network⁴ with 7,115 nodes and 608,389 triangles. We again note that the adjacency matrix B is not positive semidefinite, and neither is A = B³. Nevertheless, we can apply Hutch++ and see very strong performance. In this setting we do not need to apply Lanczos for matrix-vector query computation: Ar can be computed exactly using three matrix-vector multiplications with B. Results are shown Figure 3 with graph spectral visualized in Figure 4

Figure 3:

Relative error versus number of matrix-vector multiplication queries for trace approximations of transformed matrices. We report the median relative error of the approximation t after 100 trials. The upper and lower bounds of the shaded region around each curve are the 25^th and 75^th percentile errors. Hutch++ still outperforms the baseline methods even though A is not PSD. We note that Subspace Project has somewhat uneven performance: increasing m will take into account a larger number of top eigenvalues when approximating the trace. However, since these may be positive or negative, approximation error does not monotonically decrease. Hutch++ is not sensitive to this issue since it does not use just the top eigenvalues: see Figure 4 for more discussion.

Figure 4:

Even when estimating the trace of a non-PSD matrix like A = B³, which for the triangle counting examples above will have both positive and negative eigenvalues, Hutch++ can far outperform Hutchinson’s method. It will approximately project off the largest magnitude eigenvalues from A (whether postive or negative), which will reduce the variance in estimating the trace $\mathrm{tr}(\mathit{A})={\sum }_{i=1}^d{\lambda }_i(B)$³.

A. Proof of Lemma 2

We start by stating the Hanson-Wright inequality for i.i.d sub-Gaussian random variables:

Imported Theorem 9 (Theorem 1.1 from [RV⁺13]).

Let $\mathbf{x}\in {ℝ}^n$ be a vector of mean 0, i.i.d. sub-Gaussian random variables with constant sub-Gaussian parameter C. Let $\mathit{A}\in {ℝ}^{n\times n}$ be a matrix. Then, there exists a constant c only depending on C such that for every t ≥ 0,

$$\mathrm{Pr}\left\{\left|{\mathbf{x}}^T\mathit{A}\mathbf{x}-\mathbb{E}\left[{\mathbf{x}}^T\mathit{A}\mathbf{x}\right]\right|>t\right\}\le 2\exp \left(-c\cdot \min \left\{\frac{t^2}{\Vert \mathit{A}{\Vert }_F^2},\frac{t}{\Vert \mathit{A}{\Vert }_2}\right\}\right).$$

Above, ∥A∥₂ = max_x ∥Ax∥₂/∥x∥₂ denotes the spectral norm. We refer the reader to [RV⁺13] for a formal definition of sub-Gaussian random variables: both normal $\mathcal{N}(0,1)$ random variables and ±1 random variables are sub-Gaussian with constant C.

Lemma 2 Restated.

Let $\mathit{A}\in {ℝ}^{d\times d}$, δ ∈ (0, ½],$\ell \in \mathbb{N}$. Let H_ℓ(A) be the ℓ-query Hutchinson estimator defined in (1), implemented with mean 0, i.i.d. sub-Gaussian random variables with constant sub-Gaussian parameter. For fixed constants c, C, if $\ell >c\log (1/\delta )$, then with prob. 1 − δ,

$$\left|{\text{H}}_{\ell }(\mathit{A})-\mathrm{tr}(\mathit{A})\right|\le C\sqrt{\frac{\log (1/\delta )}{\ell }}\Vert \mathit{A}{\Vert }_F.$$

Proof.

Let $\overline{\mathit{A}}\in {ℝ}^{\ell d\times \ell d}$ be a block-diagonal matrix formed from ℓ repetitions of A:

$$\overline{\mathit{A}}:=\left[\begin{array}{cccc}\mathit{A}& 0& \dots & 0\\ 0& \mathit{A}& \dots & 0\\ 0& 0& \ddots & ⋮\\ 0& 0& \dots & \mathit{A}\end{array}\right].$$

Let $\mathit{G}\in {ℝ}^{d\times \ell }$ be as in (1). Let g_i be G’s i^th column and let $\mathbf{g}=\left[{\mathbf{g}}_1,\dots ,{\mathbf{g}}_{\ell }\right]\in {ℝ}^{d\ell }$ be a vectorization of G. We have that $\ell \cdot {\text{H}}_{\ell }(\mathit{A})=\mathrm{tr}\left({\mathit{G}}^T\mathit{A}\mathit{G}\right)={\mathbf{g}}^T\overline{\mathit{A}}\mathbf{g}$. So, by Imported Theorem 9,

$$\mathrm{Pr}\{\left|{\mathbf{g}}^T\overline{\mathit{A}}\mathbf{g}-\mathbb{E}\right|{\mathbf{g}}^T\overline{\mathit{A}}\mathbf{g}]\mid >t\}\le 2\exp \left(-c\cdot \min \left\{\frac{t^2}{\Vert \overline{\mathit{A}}{\Vert }_F^2},\frac{t}{\Vert \overline{\mathit{A}}{\Vert }_2}\right\}\right).$$ (3)

We let t′ = t/ℓ, and substitute $\mathbb{E}\left[{\mathbf{g}}^T\overline{\mathit{A}}\mathbf{g}\right]=\mathrm{tr}(\overline{\mathit{A}})=\ell \mathrm{tr}(\mathit{A})$, $\Vert \overline{\mathit{A}}{\Vert }_F^2=\ell \Vert \mathit{A}{\Vert }_F^2$ and $\Vert \overline{\mathit{A}}{\Vert }_2=\Vert \mathit{A}{\Vert }_2$ into (3) to get:

$$\mathrm{Pr}\left\{\left|{\text{H}}_{\ell }(\mathit{A})-\mathrm{tr}(\mathit{A})\right|>t^{\prime }\right\}\le 2\exp \left(-c\min \left\{\frac{\ell t^{\prime 2}}{\Vert \mathit{A}{\Vert }_F^2},\frac{\ell t^{\prime }}{\Vert \mathit{A}{\Vert }_2}\right\}\right).$$

Now, taking $t^{\prime }=\sqrt{\frac{\ln (2/\delta )}{c\ell }}\Vert \mathit{A}{\Vert }_F$, we have:

$$\mathrm{Pr}\left\{\left|{\text{H}}_{\ell }(\mathit{A})-\mathrm{tr}(\mathit{A})\right|>\sqrt{\frac{\ln (2/\delta )}{c\ell }}\Vert \mathit{A}{\Vert }_F\right\}\le 2\exp \left(-\min \left\{\log (2/\delta ),\sqrt{c\ell \log (2/\delta )}\frac{\Vert \mathit{A}{\Vert }_F}{\Vert \mathit{A}{\Vert }_2}\right\}\right)$$

Since $\frac{\Vert \mathit{A}{\Vert }_F}{\Vert \mathit{A}{\Vert }_2}\ge 1$, if we take $\ell \ge \ln (2/\delta )/c$, we have that the minimum takes value $\log (2/\delta )$, so

$$\mathrm{Pr}\left\{\left|{\text{H}}_{\ell }(\mathit{A})-\mathrm{tr}(\mathit{A})\right|>\sqrt{\frac{\ln (2/\delta )}{c\ell }}\Vert \mathit{A}{\Vert }_F\right\}\le \delta .$$

The final result follows from noting that $\ln (2/\delta )\le 2\ln (1/\delta )$ for δ ≤ ½. □

B. Proof of Theorem 7

To prove our non-adaptive lower bound for the real RAM moodel we first introduce a simple testing problem which we reduce to estimating the trace of a PSD matrix A to (1 ± ε) relative error:

Problem 2.

Fix d, $n\in \mathbb{N}$such that d ≥ n and $n:=\frac{1}{\epsilon }$ for ε ∈ (0, 1]. Let D₁ = I_n and $D_2=\left(\begin{array}{l}{\mathit{I}}_{n-1}\\ 0\end{array}\right)$.⁵ Consider A = G^T DG generated by selecting $\mathit{G}\in {ℝ}^{n\times d}$ with i.i.d. random Guassian $\mathcal{N}(0,1)$ entries and D = D₁ or D = D₂ with equal probability. Then consider any algorithm which fixes a query matrix $U\in {ℝ}^{d\times m}$, observes $\mathit{A}U\in {ℝ}^{d\times m}$, and guesses if D = D₁ or D = D₂.

The reduction from Problem 2 to relative error trace estimation is as follows:

Lemma 10.

For any ε ∈ (0, 1] and sufficient large d, if a randomized algorithm $\mathcal{A}$ can estimate the trace of any d × d PSD matrix to relative error $1\pm \frac{\epsilon }{4}$ with success $probability\ge \frac{3}{4}$ using m queries, then $\mathcal{A}$ can be used to solve Problem 2 with success $probability\ge \frac{2}{3}$ using m queries.

Proof.

To solve Problem 2 we simply apply $\mathcal{A}$ to the matrix A = G^T DG and guess D₁ if the trace is closer to $\frac{d}{\epsilon }$ and D₂ if it’s closer to $\frac{d}{\epsilon }-d$. To see that this succeeds with probability 2/3, we first need to understand the trace of A. To do so, note that tr(A) = tr(G^T DG) is simply a scaled Hutchinson estimate for tr(D), i.e. tr(G^T DG) = d · H_d(D). So, via Lemma 2, for large enough d we have that with probability $\text{probability}\ge \frac{11}{12}$ both of the following hold:

$$\frac{1}{d}\mathrm{tr}\left({\mathit{G}}^T{D}_1\mathit{G}\right)\ge \left(1-\frac{\epsilon }{4}\right)\mathrm{tr}\left(D_1\right)\text{and}\frac{1}{d}\mathrm{tr}\left({\mathit{G}}^T{D}_2\mathit{G}\right)\le \left(1+\frac{\epsilon }{4}\right)\mathrm{tr}\left(D_2\right).$$

Additionally, with probability $\frac{3}{4}$, $\mathcal{A}$ computes an approximation Z with $\left(1-\frac{\epsilon }{4}\right)\mathrm{tr}(\mathit{A})\le \mathit{Z}\le \left(1+\frac{\epsilon }{4}\right)\mathrm{tr}(\mathit{A})$. By a union bound, all of the above events happen with $\text{probability}\ge \frac{2}{3}$. If D = D₁:

$$\mathit{Z}\ge \left(1-\frac{\epsilon }{4}\right)\mathrm{tr}(\mathit{A})\ge {\left(1-\frac{\epsilon }{4}\right)}^2\cdot d\cdot \mathrm{tr}\left(D_1\right)>\left(1-\frac{\epsilon }{2}\right)\cdot \frac{d}{\epsilon }.$$

On the other hand, if D = D₂,

$$\mathit{Z}\le \left(1+\frac{\epsilon }{4}\right)\mathrm{tr}(\mathit{A})\le {\left(1+\frac{\epsilon }{4}\right)}^2\cdot d\cdot \mathrm{tr}\left(D_2\right)={\left(1+\frac{\epsilon }{4}\right)}^2\cdot (1-\epsilon )\cdot \frac{d}{\epsilon }<\left(1-\frac{\epsilon }{2}\right)\cdot \frac{d}{\epsilon }.$$

Thus, with probability 2/3, Z is closer to $\frac{d}{\epsilon }$ when D = D₁ and closer to $\frac{d}{\epsilon }-d$ when D = D₂, so the proposed scheme guesses correctly.

In the remainder of the section we show that Problem 2 requires Ω(1/ε) queries, which combined with Lemma 10 proves our main lower bound, Theorem 7. Throughout, we let $X\stackrel{\text{dist}}{=}Y$ denote that X and Y are identically distributed. We first argue that for Problem 2, the non-adaptive query matrix U might as well be chosen to be the first m standard basis vectors.

Lemma 11.

For Problem 2, without loss of generality, we may assume that the query matrix U equals U = E_m = [e₁ … e_m], the first m standard basis vectors.

Proof.

First, we may assume without loss of generality that U is orthonormal, since if it were not, we could simply reconstruct the queries AU by querying A with an orthonormal basis for the columns of U. Next, by rotational invariance of the Gaussian distribution, if $\mathit{G}\in {ℝ}^{n\times d}$ is an i.i.d. $\mathcal{N}(0,1)$ matrix, and $\mathit{Q}\in {ℝ}^{d\times d}$ is any orthogonal matrix, then GQ is distributed identically to G. Let $\overline{U}\in {ℝ}^{d\times d-m}$ be any orthonormal span for the nullspace of U, so that $\mathit{Q}:=\left[\begin{array}{ll}U\hfill & \overline{U}\hfill \end{array}\right]$ is orthogonal. We have that $\mathit{Q}{\mathit{G}}^{T}D\mathit{G}{E}_m\stackrel{\text{dist}}{=}\mathit{Q}{\mathit{Q}}^T{\mathit{G}}^{T}D\mathit{G}\mathit{Q}{E}_m={\mathit{G}}^{T}D\mathit{G}U$. So, using the result G^T DGE_m of querying with matrix E_m, we can just multiply by Q on the left to obtain a set of vectors that has the exact same distribution as if U had been used as a 25aquery matrix.

With Lemma 11 in place, we are able to reduce Problem 2 to a simpler testing problem on distinguishing m random vectors drawn from normal distributions with different covariance matrices:

Problem 3.

Let $n=\frac{1}{\epsilon }$ and let $\mathbf{z}\in {ℝ}^n$ be a uniformly random unit vector. Let $N\in {ℝ}^{n\times m}$ contain m i.i.d. random Gaussian vectors drawn from an n-dimensional Gaussian distribution, $\mathcal{N}(\mathbf{0},C)$, where the covariance matrix C either equals I or I − zz^T , with equality probability. The goal is to use N to distinguish, with $probability>\frac{1}{2}$ what the true identity of C is.

Lemma 12.

Let $\mathcal{A}$ be an algorithm that solves Problem 2 with m queries and success probability p. Then $\mathcal{A}$ can be used to solve Problem 3 with m Gaussian samples and the same success probability.

Proof.

By Lemma 11, it suffices to show how to use the observed matrix N in Problem 3 to create a sample from the distribution G^T DGE_m where $\mathit{G}\in {ℝ}^{n\times d}$ has i.i.d. $\mathcal{N}(0,1)$ entries. Specifically, we claim that, if we sample $L\in {ℝ}^{n\times (d-m)}$ with i.i.d. $\mathcal{N}(0,1)$ entries, and compute

$$M=\left[\begin{array}{c}{N}^{T}N\\ L^{T}N\end{array}\right],$$

then M is identically distributed to G^T DGE_m. I.e, if we let ${\mathit{G}}_m\in {ℝ}^{n\times m}$ contain the first m columns of G and let G_d−m contain the remaining d − m columns, our goal is the show that $M\stackrel{\text{dist}}{=}{\left[{\mathit{G}}_m{\mathit{G}}_{d-m}\right]}^{T}D{\mathit{G}}_m$.

To see this is the case, let $\mathit{Z}\in {ℝ}^{n\times n}$ be a uniformly random orthogonal matrix and let D₁, D₂ be as in Problem 2. The first observation is that $N–\mathcal{N}(\mathbf{0},C)$ is identically distributed to ZDS where $\mathit{S}\in {ℝ}^{d\times m}$ has standard normal entries ~ $\mathcal{N}(0,1)$ and D = D₁ or D = D₂ with equal probability. This follows simply from that fact that ZD₁D₁Z^T = I and $\mathit{Z}{D}_2{D}_2{\mathit{Z}}^T=\mathit{I}-z_n{z}_n^T$, where z_n is the last row of Z, which is a uniformly random unit vector. It follows that $N^{T}N\stackrel{\text{dist}}{=}{\mathit{S}}^T{D}^T{\mathit{Z}}^T\mathit{Z}D\mathit{S}={\mathit{S}}^{T}D\mathit{S}\stackrel{\text{dist}}{=}{\mathit{G}}_m^{T}D{\mathit{G}}_m$. Next, observe that L^T Z is independent of G and has i.i.d. $\mathcal{N}(0,1)$ entries since Z is orthogonal (and Gaussians are rotationally invariant). So, $L^{T}N\stackrel{\text{dist}}{=}{L}^T\mathit{Z}D\mathit{S}\stackrel{\text{dist}}{=}{\mathit{G}}_{d-m}^{T}D\mathit{G}$ and overall:

$$M=\left[\begin{array}{c}{N}^{T}N\\ L^{T}N\end{array}\right]\stackrel{\text{dist}}{=}\left[\begin{array}{c}{\mathit{G}}_m^{T}D{\mathit{G}}_m\\ {\mathit{G}}_{d-m}^{T}D{\mathit{G}}_m\end{array}\right]={\left[\begin{array}{ll}{\mathit{G}}_m\hfill & {\mathit{G}}_{d-m}\hfill \end{array}\right]}^{T}D{\mathit{G}}_m.$$

Finally, we directly prove a lower bound on the number of samples m required to solve Problem 3, and thus, via Lemma 12, Problem 2. Combined with Lemma 10, this immediately yields our main lower bound on non-adaptive trace estimation, Theorem 7.

Lemma 13.

If $m<\frac{c}{\epsilon }$ for a fixed constant c, then Problem 3 cannot be solved with $probability\ge \frac{2}{3}$.

Proof.

The proof follows from existing work on lower bounds for learning “negatively spiked” covariance matrices [CMW15, PWBM18]. Let $\mathcal{P}$ be the distribution of N in Problem 3, conditioned on C = I, and let $\mathcal{Q}$ be the distribution conditioned on C = I − zz^T. These distributions fall into the spiked covariance model of [PWBM18], specifically the negatively spiked Wishart model (see Defn. 5.1 in [PWBM18]) with spike size β = −1, and spike distribution $\mathcal{X}$ the uniform distribution over unit vectors in ${ℝ}^n$. Let $D_{{\chi }^2}(\mathcal{P}\Vert \mathcal{Q})$ denote the ${\chi }^2$ divergence between $\mathcal{P}$ and $\mathcal{Q}$. Specifically,

$$D_{{\chi }^2}(\mathcal{Q}\Vert \mathcal{P})={\int }_{X\in {ℝ}_{d\times m}}{\left(\frac{\mathcal{Q}(X)}{\mathcal{P}(X)}\right)}^2\mathcal{P}(X)dX-1.$$

We have $D_{KL}(\mathcal{Q}\Vert \mathcal{P})\le D_{{\chi }^2}(\mathcal{Q}\Vert \mathcal{P})$, so to prove that $\mathcal{P}$, $\mathcal{Q}$ cannot be distinguished with good probability, it suffices to prove an upper bound on $D_{{\chi }^2}(\mathcal{Q}\Vert \mathcal{P})$. In [CMW15] (Lemma 7) it is proven that, letting v and v′ be independent random unit vectors in ${ℝ}^n$,

$$D_{{\chi }^2}(\mathcal{Q}\Vert \mathcal{P})=\underset{\mathbf{v},{\mathbf{v}}^{\prime }}{\mathbb{E}}\left[{\left(1-{\langle \mathbf{v},{\mathbf{v}}^{\prime }\rangle }^2\right)}^{-m/2}\right]-1.$$ (4)

Equation (4) uses the notation of Prop. 5.11 in [PWBM18], which restates and proves a slightly less general form of the equality from [CMW15]. Our goal is to prove that the expectation term in (4) is ≤ 1 + C for some small constant C when $m=\frac{c}{\epsilon }=cn$ for a sufficiently small constant c.

We first note that 〈v, v′〉 is identically distributed to x ∈ [−1, 1] where x is the first entry in a random unit vector in ${ℝ}^n$. It is well known that $\frac{x+1}{2}$ is distributed according to a beta distribution with parameters $\alpha =\beta =\frac{n-1}{2}$ [FKN90]. Specifically, this gives that x has density:

$$p(x)=\frac{\mathrm{\Gamma }(2\alpha )}{2\mathrm{\Gamma }{(\alpha )}^2}\cdot {\left(\frac{1-x^2}{4}\right)}^{\alpha -1}.$$

Plugging this density back in to the expectation term in (4) we obtain:

$$\begin{gathered} \underset{\mathbf{v},{\mathbf{v}}^{\prime }}{\mathbb{E}}\left[{\left(1-{\langle \mathbf{v},{\mathbf{v}}^{\prime }\rangle }^2\right)}^{-m/2}\right]={\int }_{-1}^1\frac{\mathrm{\Gamma }(2\alpha )}{2\mathrm{\Gamma }{(\alpha )}^2}\cdot {\left(\frac{1-x^2}{4}\right)}^{\alpha -1}{\left(1-x^2\right)}^{-m/2}dx \\ ={\int }_{-1}^1\frac{\mathrm{\Gamma }(2\alpha )}{2\mathrm{\Gamma }{(\alpha )}^2}\cdot {\left(\frac{1}{4}\right)}^{\alpha -1}{\left(1-x^2\right)}^{\alpha -1-m/2}dx \end{gathered}$$

Assume without loss of generality that n is an odd integer, and thus $\alpha =\frac{n-1}{2}$ is an integer. Let m/2 = cα for some constant c ≪ 1 such that cα is an integer and thus (1−c)α is an integer. Then:

$$\begin{array}{l}\underset{\mathbf{v},{\mathbf{v}}^{\prime }}{\mathbb{E}}\left[{\left(1-{\langle \mathbf{v},{\mathbf{v}}^{\prime }\rangle }^2\right)}^{-m/2}\right]=\frac{\frac{\mathrm{\Gamma }(2\alpha )}{\mathrm{\Gamma }{(\alpha )}^2}}{\frac{\mathrm{\Gamma }(2(1-c)\alpha )}{\mathrm{\Gamma }{((1-c)\alpha )}^2}}\cdot {\left(\frac{1}{4}\right)}^{c\alpha }\cdot {\int }_{-1}^1\frac{\mathrm{\Gamma }(2(1-c)\alpha )}{2\mathrm{\Gamma }{((1-c)\alpha )}^2}{\left(\frac{1}{4}\right)}^{(1-c)\alpha -1}{\left(1-x^2\right)}^{(1-c)\alpha -1}dx\\ =\frac{\frac{\mathrm{\Gamma }(2\alpha )}{\mathrm{\Gamma }{(\alpha )}^2}}{\frac{\mathrm{\Gamma }(2(1-c)\alpha )}{\mathrm{\Gamma }{((1-c)\alpha )}^2}}\cdot {\left(\frac{1}{4}\right)}^{c\alpha },\end{array}$$ (5)

where the equality follows because the term being integrated is the density of x where $\frac{x+1}{2}$ is distributed according to a beta distribution with parameters (1 − c)α, (1 − c)α. Since we have chosen parameters such that α is a positive integer, we have:

$$\frac{\mathrm{\Gamma }(2\alpha )}{\mathrm{\Gamma }{(\alpha )}^2}=\frac{(2\alpha -1)!}{(\alpha -1)!(\alpha -1)!}=\frac{\alpha }{2}\cdot \left(\begin{array}{c}2\alpha \\ \alpha \end{array}\right).$$

Similarly $\frac{\mathrm{\Gamma }(2(1-c)\alpha )}{\mathrm{\Gamma }{((1-c)\alpha )}^2}=\frac{(1-c)\alpha }{2}\cdot \left(\begin{array}{c}2(1-c)\alpha \\ (1-c)\alpha \end{array}\right)$. Each of the binomial coefficients in these expressions is a central binomial coefficient (i.e., proportional to a Catalan number), and we can use well known methods like Stirling’s approximation to bound them. In particular, we employ a bound given in Lemma 7 of [MS77], which gives $\frac{1}{2}\frac{4^z}{\sqrt{z}}\le \left(\begin{array}{c}2z\\ z\end{array}\right)\le \frac{1}{\sqrt{\pi }}\frac{4^z}{\sqrt{z}}$. for any integer z. Accordingly, we have

$$\frac{\frac{\mathrm{\Gamma }(2\alpha )}{\mathrm{\Gamma }{(\alpha )}^2}}{\frac{\mathrm{\Gamma }(2(1-c)\alpha )}{\mathrm{\Gamma }{((1-c)\alpha )}^2}}=\frac{1}{1-c}\cdot \frac{\left(\begin{array}{c}2\alpha \\ \alpha \end{array}\right)}{\left(\begin{array}{c}2(1-c)\alpha \\ (1-c)\alpha \end{array}\right)}\le \frac{1}{1-c}\cdot \frac{1/\sqrt{\pi }}{1/2}\frac{4^{\alpha }}{4^{(1-c)\alpha }}\cdot \sqrt{\frac{(1-c)\alpha }{\alpha }}=\frac{2}{\sqrt{\pi (1-c)}}\cdot 4^{c\alpha }.$$

Plugging into (5) and requiring c ≤ .1 we have:

$$\underset{\mathbf{v},{\mathbf{v}}^{\prime }}{\mathbb{E}}\left[{\left(1-{\langle \mathbf{v},{\mathbf{v}}^{\prime }\rangle }^2\right)}^{-m/2}\right]\le \frac{2}{\sqrt{\pi \cdot .9}}<\frac{6}{5}.$$

It follows that $D_{KL}(\mathcal{Q}\Vert \mathcal{P})\le D_{{\chi }^2}(\mathcal{Q}\Vert \mathcal{P})\le \frac{6}{5}-1=\frac{1}{5}$, and thus by Pinsker’s inequality that

$$D_{TV}(\mathcal{Q},\mathcal{P})\le \frac{1}{\sqrt{10}}<\frac{1}{3}.$$

Thus, no algorithm can solve Problem 3 with $\text{probability}\ge \frac{1}{2}+\frac{1/3}{2}=\frac{2}{3}$, completing the lemma. □