Geodesic Convexity of the Symmetric Eigenvalue Problem and Convergence of Steepest Descent

Abstract

We study the convergence of the Riemannian steepest descent algorithm on the Grassmann manifold for minimizing the block version of the Rayleigh quotient of a symmetric matrix. Even though this problem is non-convex in the Euclidean sense and only very locally convex in the Riemannian sense, we discover a structure for this problem that is similar to geodesic strong convexity, namely, weak-strong convexity. This allows us to apply similar arguments from convex optimization when studying the convergence of the steepest descent algorithm but with initialization conditions that do not depend on the eigengap $\delta$. When $\delta >0$, we prove exponential convergence rates, while otherwise the convergence is algebraic. Additionally, we prove that this problem is geodesically convex in a neighbourhood of the global minimizer of radius $\mathcal{O}(\sqrt{\delta })$.

Introduction

We consider the problem of computing the top k eigenvectors of a symmetric matrix $A\in {\mathbb{R}}^{n\times n}$, which has many applications in numerical linear algebra (low rank approximation), statistics (principal component analysis) and signal processing. Without loss of generality, we assume that A is also positive semidefinite. This is because A can be shifted as $A+c{I}_n$ for some constant c and this transformation does not change its eigenvectors.

We denote by ${\lambda }_1\ge {\lambda }_2\ge \cdots \ge {\lambda }_n$ the eigenvalues of A counted with multiplicity and by $\delta :={\lambda }_k-{\lambda }_{k+1}$ the eigengap for some k between 1 and $n-1$. We also denote ${\mathrm{\Lambda }}_{\alpha }=\text{diag}({\lambda }_1,\dots ,{\lambda }_k)$ and ${\mathrm{\Lambda }}_{\beta }=\text{diag}({\lambda }_{k+1},\dots ,{\lambda }_n)$.

A set of k leading eigenvectors of A can be found by minimizing the function

$$\begin{array}{c}\hfill f(X)=-\text{Tr}(X^{T}AX)\end{array}$$

over the set of $n\times k$ matrices with orthonormal columns. Indeed, from Fan’s trace minimization theorem (see, e.g., [18, Corollary 4.3.39]) we know that

$$\begin{array}{c}\hfill min\{f(X):X\in {\mathbb{R}}^{n\times k},X^{T}X=I_k\}=-({\lambda }_1+\cdots +{\lambda }_k)=-\text{Tr}({\mathrm{\Lambda }}_{\alpha })=:f^{\ast }.\end{array}$$ 1

Since A is symmetric, we can define the matrix $V_{\alpha }=\left[\begin{array}{c}{v}_1\cdots v_k\end{array}\right]$ such that $V_{\alpha }^T{V}_{\alpha }=I_k$ and with $v_i\in {\mathbb{R}}^n$ a unit-norm eigenvector corresponding to ${\lambda }_i$. If the eigengap $\delta$ is strictly positive, then $\text{span}(V_{\alpha })$ is unique; otherwise, we can choose any $v_k$ from a subspace with dimension equal to the multiplicity of ${\lambda }_k$. It is readily seen that $f(V_{\alpha })=-({\lambda }_1+\cdots +{\lambda }_k)$. In fact, all minimizers of (1) are of the form $V_{\alpha }Q$ with Q a $k\times k$ orthogonal matrix. We also define $V_{\beta }=\left[\begin{array}{c}{v}_{k+1}\cdots v_n\end{array}\right]$ that contains the eigenvectors corresponding to the eigenvalues ${\lambda }_{k+1},\dots ,{\lambda }_n$. Its columns span the orthogonal complement of $\text{span}(V_{\alpha })$ in ${\mathbb{R}}^n$ and thus $V_{\beta }^T{V}_{\beta }=I_{n-k}$ and $V_{\alpha }^T{V}_{\beta }=0_{k\times (n-k)}$.

Since $\text{span}(V_{\alpha })=\text{span}(V_{\alpha }Q)$, it is more natural to consider this problem as a minimization problem on the Grassmann manifold $\text{Gr}(n,k)$, the set of k-dimensional subspaces in ${\mathbb{R}}^n$. Let us therefore redefine the objective function as

$$\begin{array}{c}\hfill f(\mathcal{X})=-\text{Tr}(X^{T}AX)\text{where}\mathcal{X}=\text{span}(X)\text{for}X\in {\mathbb{R}}^{n\times k}\text{s.t.}{X}^{T}X=I_k.\end{array}$$ 2

This cost function can be seen as a block version of the standard Rayleigh quotient. An immediate benefit is that, if $\delta >0$, the minimizer of (2) is isolated since it is the subspace ${\mathcal{V}}_{\alpha }=\text{span}(V_{\alpha })$.

To minimize f on $\text{Gr}(n,k)$, we shall use the Riemannian steepest descent method (RSD) along geodesics in $\text{Gr}(n,k)$. Quite remarkably, for $\text{Gr}(n,k)$ these geodesics can be implemented efficiently in closed form.

For analyzing the convergence properties of steepest descent on $\text{Gr}(n,k)$, we extend results of the recent work [4], where it is shown that the Rayleigh quotient on the sphere enjoys favourable geodesic convexity-like properties, namely, weak-quasi-convexity and quadratic growth. In this work, we show that these convexity-like properties continue to hold in the more general case of the block Rayleigh quotient function $f:\text{Gr}(n,k)\to \mathbb{R}$. These results are of general interest, but also sufficient to prove a local convergence rate for steepest descent for minimizing f when started from an initial point outside the region of local convexity. For the latter, a crucial help is provided by the fact that the Grassmann manifold is non-negatively curved (see [33]).

In particular, assuming a strictly positive eigengap $\delta$ between ${\lambda }_k$ and ${\lambda }_{k+1}$, we prove an exponential convergence rate to the subspace spanned by the k leading eigenvectors, similar to the convergence of power method and subspace iteration (Theorem 5.1). If we do not assume any knowledge regarding the eigengap, then we can still prove a sub-exponential (polynomial) convergence rate of the function values to the global minimum (Theorem 5.2), but we cannot directly study the convergence to a global minimizer. This is in line with previous work but our analysis does not use standard notions of geodesic convexity and allows for an initial guess further from the global minimizer. In Appendix B we present related convergence results for steepest descent with a more tractable step size but at the expense of needing a slightly better initialization.

Related Work

The symmetric eigenvalue problem has been popular for several decades in the numerical linear algebra and optimization communities. When only a few eigenvalues are targeted, the main solvers for this problem have been based on subspace iteration and Krylov subspace methods. Less but still considerable attention has been given to the steepest descent method and its accelerated versions. Most works on steepest descent focus only on computing the first leading eigenvector of a symmetric matrix ($k=1$), using a Euclidean version of the algorithm. Asymptotic convergence rates are known for this setting since the 1950’s, see [15]. More recently, exact non-asymptotic estimates for the same Euclidean steepest descent with exact line search were proved in [20]. For a more comprehensive overview of this line of research, the reader can refer to [26] and the references therein. A recent result that takes a different route compared to the previous ones is [4]. There, a steepest descent algorithm on the sphere is analyzed using newly proved convexity-like properties of the spherical Rayleigh quotient.

Regarding the block version of the algorithm, where one targets multiple pairs of eigenvalues and eigenvectors, much less is known. We refer here to [27], which presents a steepest descent-like method for the multiple eigenvector problem using Ritz projections onto a 2k-dimensional subspace in each step. The convergence of this algorithm is proved to be linear, but computing the Ritz projections is quite expensive. Instead, in this work we consider a much cheaper version of steepest descent by directly choosing only one of the vectors in this 2k-dimensional subspace to update our algorithm. Some analysis for such a steepest descent (without Ritz projection) on the Grassmann manifold using a retraction and an Armijo step-size is provided in [2] (see Algorithm 3 and Theorem 4.9.1). Unfortunately this convergence rate is asymptotic, that is, a linear rate is achieved after an unknown number of iterations. The region in which the convergence happens cannot be quantified. Also, such a convergence rate does not yield an iteration complexity for the algorithm.

The optimization landscape provided by the block Rayleigh quotient on the Grassmann manifold has also received some attention lately. [32] provides many interesting properties of the critical points of this function and proves that all but the global optimum are strict saddles. This is later used to derive favourable convergence properties for a hybrid method consisting of Riemannian steepest descent in a first stage and a Riemannian Newton’s method in a final stage. [23] proves the so-called robust strict saddle property for this function, that is, the Hessian evaluated in each critical point except the global optimum has both positive and negative eigenvalues in a whole neighborhood. However, none of these papers talks about (generalized) convexity of any form, nor discusses any convergence rates for steepest descent.

Turning the discussion to the convexity properties of eigenvalue problems, there is a new line of research concerned by that. In [34], the authors prove (Theorem 4) that the Rayleigh quotient is geodesically gradient dominated in the sphere ($k=1$), that is, it satisfies a spherical version of the Polyak–Łojasiewicz inequality. In [4], it is shown that this result of [34] can be strengthened to a geodesic weak-quasi-convexity and quadratic growth property, which imply gradient dominance when combined. Finally, the recent paper [3] examines (among other contributions) the convexity structure of the same block version of the symmetric eigenvalue problem on the Grassmann manifold that we introduced above. Unfortunately, the characterization of the geodesic convexity region independently of the eigengap $\delta$ (Corollary 5 in [3]) is wrong (see our Appendix A for a counterexample). As we will prove in Theorem A.1, the geodesic convexity region of f (and the one of the equivalent cost function used in [3]) needs to depend on the eigengap, as appears also in [19, Lemma 7] in the case of the sphere ($k=1$).

To the best of our knowledge, the current work is the first that provides non-asymptotic convergence rates for the steepest descent algorithm for the multiple eigenvalue-eigenvector problem on the Grassmann manifold. We mainly rely on the work [4], which proves exponential convergence of steepest descent only in the case of $k=1$, that is, for the leading eigenvector. In this paper, we take a reasonable but highly non-trivial step forward by extending the convexity-like characterization of the spherical Rayleigh quotient to general k, that is, for a block of k leading eigenvectors. Again, the paper [9] is of high value for our current work regarding weakly-strongly-convex functions.

As mentioned above, the standard algorithm for computing the leading eigenspace of dimension k is subspace iteration (or power method when $k=1$).1 However, there are reasons to believe that, in certain cases, Riemannian steepest descent (and its accelerated version with non-linear conjugate gradients) should be preferred, especially in noisy settings [4] or in electronic structure calculations where the leading eigenspace of many varying matrices A needs to be computed.2 In particular, [4] presents strong experimental evidence that steepest descent is more robust to perturbations of the matrix–vector products than subspace iteration close to the optimum. While subspace iteration still behaves better at the start of the iteration, it asymptotically fails to converge to an approximation of the leading subspace that is as good as the one estimated by Riemannian steepest descent. While [4] dealt with a noisy situation due to calculations in a distributed setting with limited communication, exactly the same effect can be observed when we inject the matrix–vector products with Gaussian noise. Thus, we expect steepest descent to perform better than subspace iteration close to the optimum in any stochastic regime [14].

Regarding worst-case theoretical guarantees, the strongest convergence result for subspace iteration in the presence of a strictly positive eigengap $\delta$ is in terms of the largest principal angle between the iterates and the optimum [13], that is, the ${\ell }_{\infty }$-norm of the vector of principal angles. In contrast, our convergence result for steepest descent for $\delta >0$ (Theorem 5.1) is in terms of the ${\ell }_2$-norm of the same vector of angles, which is in general stronger. When $\delta =0$, it is known from [21, 28] that the largest eigenvalue ($k=1$) can still be efficiently estimated. We extend this result for $k>1$ and prove a convergence rate of steepest descent for the function values f (Theorem 5.2), relying only on weak-quasi-convexity (and thus using a different argument from [21, 28]).

Geometry of the Grassmann Manifold and Block Rayleigh Quotient

We present here a brief introduction into the geometry of the Grassmann manifold. The content is not new and for more details, we refer to [2, 7, 12].

The (n, k)-Grassmann manifold is defined as the set of all k-dimensional subspaces of ${\mathbb{R}}^n$:

$$\begin{array}{c}\hfill \text{Gr}(n,k)=\{\mathcal{X}\subseteq {\mathbb{R}}^n:\mathcal{X}\text{is a subspace and}dim(\mathcal{X})=k\}.\end{array}$$

Any element $\mathcal{X}$ of $\text{Gr}(n,k)$ can be represented by a matrix $X\in {\mathbb{R}}^{n\times k}$ that satisfies $\mathcal{X}=\text{span}(X)$. Such a representative is not unique since $Y=XQ$ for some invertible matrix $Q\in {\mathbb{R}}^{k\times k}$ satisfies $\text{span}(Y)=\text{span}(X)$. Without loss of generality, we will therefore always take matrix representatives X of subspaces $\mathcal{X}$ that have orthonormal columns throughout the paper. With some care, the non-uniqueness of the representatives is not a problem.3 For example, our objective function (2) is invariant to Q.

Riemannian structure. The set $\text{Gr}(n,k)$ admits the structure of a differential manifold with tangent spaces

$$\begin{array}{c}\hfill T_{\mathcal{X}}\text{Gr}(n,k)=\{G\in {\mathbb{R}}^{n\times k}:X^{T}G=0\},\end{array}$$ 3

where $\mathcal{X}=\text{span}(X)$. Since $X^{T}G=0$ if and only if ${(XQ)}^{T}G=0$, for any invertible matrix $Q\in {\mathbb{R}}^{k\times k}$, this description of the tangent space does not depend on the representative X. However, a specific tangent vector G will depend on the chosen X. With slight abuse of notation,4 the above definition should therefore be interpreted as: given a fixed X, we define tangent vectors $G_1,G_2,\dots$ of $\text{Gr}(n,k)$ at $\mathcal{X}=\text{span}(X)$.

This subtlety is important, for example, when defining an inner product on $T_{\mathcal{X}}\text{Gr}(n,k)$:

$$\begin{array}{c}\hfill {⟨G_1,G_2⟩}_{\mathcal{X}}=\text{Tr}(G_1^T{G}_2)\text{ with}{G}_1,G_2\in T_{\mathcal{X}}\text{Gr}(n,k).\end{array}$$

Here, $G_1$ and $G_2$ are tangent vectors of the same representative X. Observe that the inner product is invariant to the choice of orthonormal representative: If ${\overline{G}}_1=G_{1}Q$ and ${\overline{G}}_2=G_{2}Q$ with orthogonal Q, then we have

$$\begin{array}{c}\hfill {⟨{\overline{G}}_1,{\overline{G}}_2⟩}_{\mathcal{X}}=\text{Tr}({\overline{G}}_1^T{\overline{G}}_2)=\text{Tr}(Q^T{G}_1^T{G}_{2}Q)=\text{Tr}(G_1^T{G}_{2}Q{Q}^T)=\text{Tr}(G_1^T{G}_2).\end{array}$$

It is easy to see that the norm induced by this inner product in any tangent space is the Frobenius norm, which we will denote throughout the paper as $\Vert ·\Vert :={\Vert ·\Vert }_F$.

Exponential map. Given the Riemannian structure of $\text{Gr}(n,k)$, we can compute the exponential map at a point $\mathcal{X}$ as [1, Thm. 3.6]

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill {\text{Exp}}_{\mathcal{X}}:T_{\mathcal{X}}\text{Gr}(n,k)& \to \text{Gr}(n,k)\hfill \\ \hfill G& \mapsto \text{span}(XVcos(\mathrm{\Sigma })+Usin(\mathrm{\Sigma })),\hfill \end{array}\end{array}$$ 4

where $U\mathrm{\Sigma }{V}^T$ is the compact SVD of G such that $\mathrm{\Sigma }$ and V are square matrices.

The exponential map is invertible in the domain [7, Prop. 5.1]

$$\begin{array}{c}\hfill \left\{G,\in ,T_{\mathcal{X}},\text{Gr},(n,k),:,{\Vert G\Vert }_2,<,\frac{\pi }{2}\right\},\end{array}$$ 5

where ${\Vert G\Vert }_2$ is the spectral norm of G. The inverse of the exponential map restricted to this domain is the logarithmic map, denoted by $\text{Log}$. Given two subspaces $\mathcal{X},\mathcal{Y}\in \text{Gr}(n,k)$, we have

$$\begin{array}{c}\hfill {\text{Log}}_{\mathcal{X}}(\mathcal{Y})=U\text{atan}(\widehat{\mathrm{\Sigma }})V^T,\end{array}$$ 6

where $U\widehat{\mathrm{\Sigma }}{V}^T=(I-X{X}^T)Y{(X^{T}Y)}^{-1}$ is again a compact SVD. This is well-defined if $X^{T}Y$ is invertible, which is guaranteed if all principal angles between $\mathcal{X}$ and $\mathcal{Y}$ are strictly less than $\pi /2$ (see below). By taking $G={\text{Log}}_{\mathcal{X}}(\mathcal{Y})$, we see that $\mathrm{\Sigma }=\text{atan}(\widehat{\mathrm{\Sigma }})$.

Principal angles. The Riemannian structure of the Grassmann manifold can be conveniently described by the notion of the principal angles between subspaces. Given two subspaces $\mathcal{X},\mathcal{Y}\in \text{Gr}(n,k)$, the principal angles between them are $0\le {\theta }_1\le \cdots \le {\theta }_k\le \pi /2$ obtained from the SVD

$$\begin{array}{c}\hfill Y^{T}X=U_{1}cos\theta V_1^T\end{array}$$ 7

where $U_1\in {\mathbb{R}}^{k\times k},V_1\in {\mathbb{R}}^{k\times k}$ are orthogonal and the diagonal matrix $cos\theta =\text{diag}(cos{\theta }_1,...,cos{\theta }_k)$. Notice that the definition of principal angles is indeed independent of the specific orthonormal representatives of the subspaces in question.

We can express the Riemannian logarithm using principal angles and the intrinsic distance induced by the Riemannian inner product discussed above is

$$\begin{array}{c}\hfill \text{dist}(\mathcal{X},\mathcal{Y})=\Vert {\text{Log}}_{\mathcal{X}}(\mathcal{Y})\Vert =\Vert {\text{Log}}_{\mathcal{Y}}(\mathcal{X})\Vert =\sqrt{{\theta }_1^2+...+{\theta }_k^2}=\Vert \theta {\Vert }_2,\end{array}$$ 8

where $\theta ={({\theta }_1,\dots ,{\theta }_k)}^T$. For more details on these facts, the reader can refer to section 4.3 in [12] (arc length distance).

If $X\in {\mathbb{R}}^{n\times k}$ is an arbitrary matrix with orthonormal columns, then, generically, these columns will not be exactly orthogonal to the k leading eigenvectors $v_1,\dots ,v_k$ of A. Thus, we have with probability one that the principal angles between $\mathcal{X}$ and the space of k leading eigenvectors satisfy $0\le {\theta }_1\le \cdots \le {\theta }_k<\pi /2$.

Curvature. We can compute exactly the sectional curvatures in $\text{Gr}(n,k)$, but for our purposes we only need that they are everywhere non-negative [7, 33]. This means that the geodesics on the Grassmann manifold spread more slowly than in Euclidean space. This is consequence of the famous Toponogov’s theorem (see [11]) that we state here in the form of the following technical lemma, which will be important in our convergence analysis.

Lemma 3.1

Let $\mathcal{X},\mathcal{Y},\mathcal{Z}\in \text{Gr}(n,k)$, such that

$$max\{\text{dist}(\mathcal{X},\mathcal{Z}),\text{dist}(\mathcal{Y},\mathcal{Z})\}<\frac{\pi }{2}.$$

Then

$$\begin{array}{c}\hfill \text{dist}(\mathcal{X},\mathcal{Y})\le \Vert {\text{Log}}_{\mathcal{Z}}(\mathcal{X})-{\text{Log}}_{\mathcal{Z}}(\mathcal{Y})\Vert .\end{array}$$

Lemma 3.2

(Law of cosines) Let $\mathcal{X},\mathcal{Y},\mathcal{Z}$ as in Lemma 3.1. Then

$$\begin{array}{c}\hfill {\text{dist}}^2(\mathcal{X},\mathcal{Y})\le {\text{dist}}^2(\mathcal{Z},\mathcal{X})+{\text{dist}}^2(\mathcal{Z},\mathcal{Y})-2⟨{\text{Log}}_{\mathcal{Z}}(\mathcal{X}),{\text{Log}}_{\mathcal{Z}}(\mathcal{Y})⟩.\end{array}$$

Proof

Apply Lemma 3.1 and expand $\Vert {\text{Log}}_{\mathcal{Z}}(\mathcal{X})-{\text{Log}}_{\mathcal{Z}}{(\mathcal{Y})\Vert }^2$. $\square$

Block Rayleigh quotient. Our objective function for minimization is the block version of the Rayleigh quotient:

$$\begin{array}{c}\hfill f(\mathcal{X})=-\text{Tr}(X^{T}AX)\text{where}\mathcal{X}=\text{span}(X)\in \text{Gr}(n,k)\text{s.t.}{X}^{T}X=I_k.\end{array}$$

This function has ${\mathcal{V}}_{\alpha }=\text{span}(\left[\begin{array}{c}{v}_1\cdots v_k\end{array}\right])$ as global minimizer. This minimizer is unique on $\text{Gr}(n,k)$ if and only if $\delta >0$.

Given any differentiable function $f:\text{Gr}(n,k)\to \mathbb{R}$, we can define its Riemannian gradient as the vector field that satisfies

$$\begin{array}{c}\hfill df(\mathcal{X})(G)={⟨\text{grad}f(\mathcal{X}),G⟩}_{\mathcal{X}},\text{for any}G\in T_{\mathcal{X}}\text{Gr}(n,k).\end{array}$$

For a given representative X of $\mathcal{X}$, the Riemannian gradient of the block Rayleigh quotient satisfies

$$\begin{array}{c}\hfill \text{grad}f(\mathcal{X})=-2(I-X{X}^T)AX.\end{array}$$

Using the notions of the Riemannian gradient and Levi-Civita connection, we can define also a Riemannian notion of Hessian. For the block Rayleigh quotient f, the Riemannian Hessian $\text{Hess}f$ evaluated as bilinear form satisfies

$$\begin{array}{c}\hfill \text{Hess}f(\mathcal{X})[G,G]=2⟨G,G{X}^{T}AX-AG⟩,\end{array}$$ 9

for $G\in T_{\mathcal{X}}\text{Gr}(n,k)$; see [12, §4.4] or [2, §6.4.2].

Convexity-like Properties of the Block Rayleigh quotient

We now prove the new analytic properties of the block Rayleigh quotient $f(\mathcal{X})=-\text{Tr}(X^{T}AX)$. These are important in their own right but will also be used later for the convergence of the Riemannian steepest descent method.

Smoothness

A $C^2$ function defined on the Grassmann manifold is called $\gamma$-smooth if the maximum eigenvalue of its Riemannian Hessian is everywhere upper bounded by a positive constant $\gamma$. This is true for the block Rayleigh quotient, as we show in the next proposition:

Proposition 1

(Smoothness) The eigenvalues of the Riemannian Hessian of f on $\text{Gr}(n,k)$ are upper bounded by $\gamma :=2({\lambda }_1-{\lambda }_n)$.

Proof

Let G be a tangent vector of $\text{Gr}(n,k)$ at X. Then the Riemannian Hessian satisfies (see (9))

$$\begin{array}{c}\hfill \frac{1}{2}\text{Hess}f(\mathcal{X})[G,G]=\text{Tr}(G^{T}G{X}^{T}AX)-\text{Tr}(AG{G}^T).\end{array}$$

Since $A,X^{T}AX,G{G}^T$, and $G^{T}G$ are all symmetric and positive semi-definite matrices, standard trace inequality (see, e.g, [18, Thm. 4.3.53]) gives

$$\begin{array}{c}\hfill \text{Hess}f(\mathcal{X})[G,G]\le 2({\lambda }_{max}(X^{T}AX)-{\lambda }_{min}(A)){\Vert G\Vert }^2.\end{array}$$

Since X has orthonormal columns, ${\lambda }_{max}(X^{T}AX)\le {\lambda }_{max}(A)$; see, e.g., [18, Cor. 4.3.37]. The proof is now complete with the definition of ${\lambda }_1$ and ${\lambda }_n$. $\square$

The result in Prop. 1 is tight: Choosing $X=V_{\alpha }$ and $G=v_n{e}_1^T$, it is readily verified that the upper bound is attained. From now on, we refer to $\gamma$ as the specific value $2({\lambda }_1-{\lambda }_n)$. This value also features in a useful upper bound for the spectral norm of the gradient. This bound is independent of $\mathcal{X}$:

Lemma 4.1

For all $\mathcal{X}\in \text{Gr}(n,k)$ and $\gamma =2({\lambda }_1-{\lambda }_n)$, the Riemannian gradient of f satisfies

$$\begin{array}{c}\hfill {\Vert \text{grad}f(\mathcal{X})\Vert }_2\le \frac{\gamma }{2}.\end{array}$$

Proof

Since X has orthonormal columns, we can complete it to the orthogonal matrix $Q=\left[\begin{array}{cc}X& X_{\perp }\end{array}\right]$. Hence, ${\Vert \text{grad}f(\mathcal{X})\Vert }_2=\Vert 2(I-X{X}^T){AX\Vert }_2=2{\Vert X_{\perp }^{T}AX\Vert }_2$. The result now follows directly from [22, Thm. 2] since A is real symmetric and the definition of $\gamma =2({\lambda }_1-{\lambda }_n)$. $\square$

By the second-order Taylor expansion of f (see, e.g., [8], Corollary 10.54) it is easy to see that Proposition 1 implies

$$\begin{array}{c}\hfill f(\mathcal{X})\le f(\mathcal{Y})+⟨\text{grad}f(\mathcal{Y}),{\text{Log}}_{\mathcal{Y}}(\mathcal{X})⟩+\frac{\gamma }{2}{\text{dist}}^2(\mathcal{X},\mathcal{Y}),\end{array}$$ 10

for any $\mathcal{X},\mathcal{Y}\in \text{Gr}(n,k)$ such that ${\text{Log}}_{\mathcal{X}}(\mathcal{Y})$ is well-defined.

As in the introduction, denote the global minimum of f by $f^{\ast }$ which is attained at ${\mathcal{V}}_{\alpha }\in \text{Gr}(n,k)$. Inequality (10) leads to the following lemma:

Lemma 4.2

For any $\mathcal{X}\in \text{Gr}(n,k)$ and $\gamma =2({\lambda }_1-{\lambda }_n)$, we have

$$\begin{array}{c}\hfill f(\mathcal{X})-f^{\ast }\ge \frac{1}{2\gamma }{\Vert \text{grad}f(\mathcal{X})\Vert }^2.\end{array}$$

Proof

Since $f^{\ast }$ is a global minimum of f, we have from (10) that

$$\begin{array}{c}\hfill f^{\ast }\le f(\mathcal{X})\le f(\mathcal{Y})+⟨\text{grad}f(\mathcal{Y}),{\text{Log}}_{\mathcal{Y}}(\mathcal{X})⟩+\frac{\gamma }{2}{\Vert {\text{Log}}_{\mathcal{Y}}(\mathcal{X})\Vert }^2,\end{array}$$

for any $\mathcal{X},\mathcal{Y}\in \text{Gr}(n,k)$ such that ${\text{Log}}_{\mathcal{X}}(\mathcal{Y})$ is well-defined.

We set $\mathcal{X}:={\text{Exp}}_{\mathcal{Y}}\left(-,\frac{1}{\gamma },\text{grad},f,(\mathcal{Y})\right)$. By Lemma 4.1, we have that ${∥-,\frac{1}{\gamma },\text{grad},f,(\mathcal{Y})∥}_2<\frac{\pi }{2}$ and by equation (5) we have that ${\text{Log}}_{\mathcal{Y}}(\mathcal{X})$ is well-defined and equal to $-\frac{1}{\gamma }\text{grad}f(\mathcal{Y})$. Then,

the right hand side of the initial inequality becomes

$$\begin{array}{c}\hfill f^{\ast }\le f(\mathcal{Y})-\frac{1}{\gamma }{\Vert \text{grad}f(\mathcal{Y})\Vert }^2+\frac{1}{2\gamma }{\Vert \text{grad}f(\mathcal{Y})\Vert }^2=f(\mathcal{Y})-\frac{1}{2\gamma }{\Vert \text{grad}f(\mathcal{Y})\Vert }^2.\end{array}$$

Rearranging the last inequality and substituting $\mathcal{Y}=\mathcal{X}$, we get the desired result. $\square$

Weak-Quasi-Convexity and Quadratic Growth

We now turn our interest in the convexity properties of the block Rayleigh quotient function. We start by proving a property which is known in the literature as quadratic growth.

Proposition 2

(Quadratic growth) Let $0\le {\theta }_1\le \cdots \le {\theta }_k<\pi /2$ be the principal angles between the subspaces $\mathcal{X}$ and ${\mathcal{V}}_{\alpha }$. The function f satisfies

$$f(\mathcal{X})-f^{\ast }\ge c_Q\delta {\text{dist}}^2(\mathcal{X},{\mathcal{V}}_{\alpha })$$

where $c_Q=4/{\pi }^2>0.4$.

Proof

The spectral decomposition of $A=V_{\alpha }{\mathrm{\Lambda }}_{\alpha }{V}_{\alpha }^T+V_{\beta }{\mathrm{\Lambda }}_{\beta }{V}_{\beta }^T$ implies

$$\begin{array}{c}\hfill X^{T}AX=X^T{V}_{\alpha }{\mathrm{\Lambda }}_{\alpha }{V}_{\alpha }^{T}X+X^T{V}_{\beta }{\mathrm{\Lambda }}_{\beta }{V}_{\beta }^{T}X.\end{array}$$ 11

Since $f(\mathcal{X})=-\text{Tr}(X^{T}AX)$, we have

$$\begin{array}{cc}\hfill f(\mathcal{X})-f^{\ast }& =\text{Tr}({\mathrm{\Lambda }}_{\alpha })-\text{Tr}(X^T{V}_{\alpha }{\mathrm{\Lambda }}_{\alpha }{V}_{\alpha }^{T}X)-\text{Tr}(X^T{V}_{\beta }{\mathrm{\Lambda }}_{\beta }{V}_{\beta }^{T}X)\hfill \\ \hfill & =\text{Tr}({\mathrm{\Lambda }}_{\alpha })-\text{Tr}({\mathrm{\Lambda }}_{\alpha }{V}_{\alpha }^{T}X{X}^T{V}_{\alpha })-\text{Tr}({\mathrm{\Lambda }}_{\beta }{V}_{\beta }^{T}X{X}^T{V}_{\beta })\hfill \\ \hfill & =\text{Tr}({\mathrm{\Lambda }}_{\alpha }(I_k-V_{\alpha }^{T}X{X}^T{V}_{\alpha }))-\text{Tr}({\mathrm{\Lambda }}_{\beta }{V}_{\beta }^{T}X{X}^T{V}_{\beta }).\hfill \end{array}$$

From the definition (7) of the principal angles between X and $V_{\alpha }$, we recall that

$$\begin{array}{c}\hfill V_{\alpha }^{T}X=U_{1}cos\theta V_1^T,\end{array}$$ 12

where $cos\theta =\text{diag}(cos{\theta }_1,\dots ,cos{\theta }_k)$ is a diagonal matrix and $U_1,V_1$ are orthogonal matrices. Plugging this equality in, we get that the jth eigenvalue of the matrix $I_k-V_{\alpha }^{T}X{X}^T{V}_{\alpha }$ is equal to $1-{cos}^2{\theta }_j={sin}^2{\theta }_j\ge 0$. Thus, by standard trace inequality for symmetric and positive definite matrices (see, e.g., [18, Thm. 4.3.53]), the first summand above satisfies

$$\begin{array}{c}\hfill \text{Tr}({\mathrm{\Lambda }}_{\alpha }(I_k-V_{\alpha }^{T}X{X}^T{V}_{\alpha }))\ge {\lambda }_k\sum _{j=1}^k{sin}^2{\theta }_j.\end{array}$$

The matrix $V_{\beta }^{T}X{X}^T{V}_{\beta }$ has the same non-zero eigenvalues with the same multiplicity as the matrix

$$\begin{array}{c}\hfill X^T{V}_{\beta }{V}_{\beta }^{T}X=I_k-V_1{cos}^2\theta V_1^T=V_1{sin}^2\theta V_1^T\end{array}$$

where we used $V_{\beta }{V}_{\beta }^T=I_n-V_{\alpha }{V}_{\alpha }^T$ and the SVD of $V_{\alpha }^{T}X$. Thus the jth eigenvalue of $V_{\beta }^{T}X{X}^T{V}_{\beta }$ is ${sin}^2{\theta }_j\ge 0$. By trace inequality again, the second summand therefore satisfies

$$\begin{array}{c}\hfill \text{Tr}({\mathrm{\Lambda }}_{\beta }{V}_{\beta }^{T}X{X}^T{V}_{\beta })\le {\lambda }_{k+1}\sum _{j=1}^k{sin}^2{\theta }_j.\end{array}$$

Putting both bounds together, we get

$$\begin{array}{c}\hfill f(\mathcal{X})-f^{\ast }\ge ({\lambda }_k-{\lambda }_{k+1})\sum _{j=1}^k{sin}^2{\theta }_j\ge \delta \sum _{j=1}^k\frac{4}{{\pi }^2}{\theta }_j^2\end{array}$$

and the proof is complete by the definition (8) of $\text{dist}$. $\square$

We say that f is geodesically convex if for all $\mathcal{X}$ and $\mathcal{Y}$ in a suitable region it holds

$$f(\mathcal{X})-f(\mathcal{Y})\le ⟨\text{grad}f(\mathcal{X}),-{\text{Log}}_{\mathcal{X}}(\mathcal{Y})⟩.$$

This generalizes the classical convexity of differentiable functions on ${\mathbb{R}}^n$ to manifolds by taking the logarithmic map instead of the difference $\mathcal{X}-\mathcal{Y}$.

In Appendix A, we prove that our objective function f is only geodesically convex in a small neighbourhood of size $\mathcal{O}(\sqrt{\delta })$ around the minimizer ${\mathcal{V}}_{\alpha }$. Fortunately, our key result of this section shows that f satisfies a much weaker notion of geodesic convexity, known in the literature as weak-quasi-convexity, that does not depend on the eigengap $\delta$.

We first need the following lemma which is the general version of the CS decomposition but applied to our setting of square blocks.

Lemma 4.3

Let $X,Y\in {\mathbb{R}}^{n\times k}$ be such that $X^{T}X=Y^{T}Y=I_k$ with $k<n$. Choose $X_{\perp },Y_{\perp }\in {\mathbb{R}}^{n\times (n-k)}$ such that $X_{\perp }^T{X}_{\perp }=Y_{\perp }^T{Y}_{\perp }=I_{n-k}$ and $\text{span}(X_{\perp })=\text{span}{(X)}^{\perp }$, $\text{span}(Y_{\perp })=\text{span}{(Y)}^{\perp }$. Then there exist $0\le r,s\le k$ such that

$$\begin{array}{cccc}\hfill Y^{T}X& =U_1\left[\begin{array}{c}{I}_r\\ & C_s\\ & & O_{p\times p}\end{array}\right]V_1^T,\hfill & \hfill Y^T{X}_{\perp }& =U_1\left[\begin{array}{c}{O}_{r\times m}\\ & S_s\\ & & I_p\end{array}\right]V_2^T\hfill \\ \hfill Y_{\perp }^{T}X& =U_2\left[\begin{array}{c}{O}_{m\times r}\\ & S_s\\ & & I_p\end{array}\right]V_1^T,\hfill & \hfill Y_{\perp }^T{X}_{\perp }& =U_2\left[\begin{array}{c}-I_m\\ & -C_s\\ & & O_{p\times p}\end{array}\right]V_2^T\hfill \end{array}$$

with $p=k-r-s$ and $m=n-2k+r$, and we have

• orthogonal matrices $U_1,V_1$ of size k and $U_2,V_2$ of size $n-k$;

• identity matrices $I_q$ of size q;

• zero matrices $O_{q\times t}$ of size $q\times t$;

• diagonal matrices $C_s=\text{diag}({\alpha }_1,\dots ,{\alpha }_s)$ and $S_s=\text{diag}({\beta }_1,\dots ,{\beta }_s)$ such that $1>{\alpha }_1\ge \cdots \ge {\alpha }_s>0$, $0<{\beta }_1\le \cdots \le {\beta }_s<1$ and $C_s^2+S_s^2=I_s$.

Proof

Since $\left[\begin{array}{cc}X& X_{\perp }\end{array}\right]$ and $\left[\begin{array}{cc}Y& Y_{\perp }\end{array}\right]$ are orthogonal, the result follows directly from the CS decomposition of the orthogonal matrix $P={\left[\begin{array}{cc}Y& Y_{\perp }\end{array}\right]}^T\left[\begin{array}{cc}X& X_{\perp }\end{array}\right]$; see the Theorem of §4 in [29]. $\square$

Observe that the matrix $\text{diag}(I_r,C_s,O_{p\times p})$ in this lemma corresponds to the matrix $cos(\theta )$ in (7) with $\theta$ the vector of principal angles $0\le {\theta }_1\le \cdots \le {\theta }_k\le \pi /2$ between $\text{span}(X)$ and $\text{span}(Y)$. However, the lemma explicitly splits off the angles that are zero and $\pi /2$ so that it can formulate the related decompositions for $Y^T{X}_{\perp },Y_{\perp }^{T}X,$ and $Y_{\perp }^T{X}_{\perp }$ with $C_s$ and $S_s$.

We are now ready to state our weak quasi-convexity result. In the statement of the proposition below (and throughout this paper), we use the convention that $\frac{0}{tan0}=1$.

Proposition 3

(Weak-quasi-convexity) Let $0\le {\theta }_1\le \cdots \le {\theta }_k<\pi /2$ be the principal angles between the subspaces $\mathcal{X}$ and ${\mathcal{V}}_{\alpha }$. Then, f satisfies

$$2a(\mathcal{X})(f(\mathcal{X})-f^{\ast })\le ⟨\text{grad}f(\mathcal{X}),-{\text{Log}}_{\mathcal{X}}({\mathcal{V}}_{\alpha })⟩$$

with $a(\mathcal{X}):={\theta }_k/tan{\theta }_k$.

Proof

Take X and $V_{\alpha }$ matrices with orthonormal columns such that $\mathcal{X}=\text{span}(X)$ and ${\mathcal{V}}_{\alpha }=\text{span}(V_{\alpha })$. Since ${\theta }_k<\pi /2$, we know that $p=0$ in Lemma 4.3 and thus $s=k-r$ with r the number of principal angles that are equal to zero. Choosing a matrix $X_{\perp }$ with orthonormal columns such that $\text{span}(X_{\perp })=\text{span}{(X)}^{\perp }$, we therefore get from Lemma 4.3 that there exist orthogonal matrices $U_1,V_1$ of size k and $V_2$ of size $n-k$ such that

$$\begin{array}{cccc}\hfill V_{\alpha }^{T}X& =U_1\left[\begin{array}{c}{I}_r\\ & C_{k-r}\end{array}\right]V_1^T,\hfill & \hfill V_{\alpha }^T{X}_{\perp }& =U_1\left[\begin{array}{c}{O}_{r\times m}\\ & S_{k-r}\end{array}\right]V_2^T.\hfill \end{array}$$ 13

Comparing with (7), we deduce that $C_{k-r}=\text{diag}(cos{\theta }_{r+1},\dots ,cos{\theta }_k)$ and $S_{k-r}=\text{diag}(sin{\theta }_{r+1},\dots ,sin{\theta }_k)$ since $C_{k-r}^2+S_{k-r}^2=I$.

We recall from (6) that

$$\begin{array}{c}\hfill {\text{Log}}_{\mathcal{X}}({\mathcal{V}}_{\alpha })=U\text{atan}(\mathrm{\Sigma })V^T,\end{array}$$ 14

where $U\mathrm{\Sigma }{V}^T=(I_n-X{X}^T)V_{\alpha }{(X^T{V}_{\alpha })}^{-1}=:M$ is a compact SVD (without the requirement that the diagonal of $\mathrm{\Sigma }$ is non-increasing). Using $X_{\perp }$ from above, we can also write $M=X_{\perp }{X}_{\perp }^T{V}_{\alpha }{(X^T{V}_{\alpha })}^{-1}$. Substituting (13) and using that $U_1$ and $V_1$ are orthogonal gives

$$\begin{array}{c}\hfill M=X_{\perp }{V}_2\left[\begin{array}{c}{O}_{m\times r}\\ & S_{k-r}{C}_{k-r}^{-1}\end{array}\right]V_1^T=X_{\perp }{\tilde{V}}_2\left[\begin{array}{c}{O}_{r\times r}\\ & S_{k-r}{C}_{k-r}^{-1}\end{array}\right]V_1^T,\end{array}$$

where ${\tilde{V}}_2\in {\mathbb{R}}^{(n-k)\times k}$ contains the last k columns of $V_2$ in order. Note that this reformulation of the SVD of M holds always, regardless of the relationship between m and r. If $m\ge r$, the matrix $\left[\begin{array}{c}{O}_{m\times r}\\ & S_{k-r}{C}_{k-r}^{-1}\end{array}\right]$ has its first $m-r$ rows equal to 0, thus we can cut the first $m-r$ columns of $V_2$, since they do not contribute to the product. This yields a matrix ${\tilde{V}}_2$ with $n-k$ rows and $n-k-m+r=k$ of the last columns of $V_2$. If $m<r$, then the first $r-m$ columns of $\left[\begin{array}{c}{O}_{m\times r}\\ & S_{k-r}{C}_{k-r}^{-1}\end{array}\right]$ are 0 and now we can add $r-m$ columns in the beginning of the matrix $V_2$ that keep the derived matrix orthonormal. This again yields a matrix ${\tilde{V}}_2$ with $n-k$ rows and $n-k+r-m=k$ columns. Since the matrix $\left[\begin{array}{c}{O}_{r\times r}\\ & S_{k-r}{C}_{k-r}^{-1}\end{array}\right]$ occurs by adding $r-m$ zero rows at the beginning of $\left[\begin{array}{c}{O}_{m\times r}\\ & S_{k-r}{C}_{k-r}^{-1}\end{array}\right]$, the product does not change.

Since ${\theta }_1=\cdots ={\theta }_r=0$, we can therefore formulate the compact SVD of M using the vector $\theta$ of all principal angles as follows:

$$\begin{array}{c}\hfill M=U\mathrm{\Sigma }{V}^T\text{with}U=X_{\perp }{\tilde{V}}_2,\mathrm{\Sigma }=tan(\theta ),V=V_1.\end{array}$$

Hence from (14) we get directly that

$$\begin{array}{c}\hfill {\text{Log}}_{\mathcal{X}}({\mathcal{V}}_{\alpha })=X_{\perp }{\tilde{V}}_2\theta V_1^T,\end{array}$$ 15

where $\theta$ is a diagonal matrix.

We now claim that (15) also satisfies

$$\begin{array}{c}\hfill {\text{Log}}_{\mathcal{X}}({\mathcal{V}}_{\alpha })=X_{\perp }{X}_{\perp }^T{V}_{\alpha }{U}_1\frac{\theta }{sin\theta }{V}_1^T,\end{array}$$ 16

where $\frac{\theta }{sin\theta }$ is a diagonal matrix for which $\frac{0}{sin0}=1$. Indeed, recalling that ${\theta }_1=\cdots ={\theta }_r=0$ and using the identities

$$\begin{array}{c}\hfill X_{\perp }^T{V}_{\alpha }={\tilde{V}}_2\left[\begin{array}{c}{O}_{r\times r}\\ & S_{k-r}\end{array}\right]U_1^T,\frac{\theta }{sin\theta }=\left[\begin{array}{c}{I}_r\\ & S_{k-r}^{-1}\end{array}\right]\left[\begin{array}{c}{I}_r\\ & T_{k-r}\end{array}\right]\end{array}$$

where $T_{k-r}=\text{diag}({\theta }_{r+1},\dots ,{\theta }_k)$, we obtain

$$\begin{array}{cc}\hfill \text{rhs of (16)}& =X_{\perp }{\tilde{V}}_2\left[\begin{array}{c}{O}_{r\times r}\\ & S_{k-r}\end{array}\right]\left[\begin{array}{c}{I}_r\\ & S_{k-r}^{-1}\end{array}\right]\left[\begin{array}{c}{I}_r\\ & T_{k-r}\end{array}\right]V_1^T\hfill \\ & =X_{\perp }{\tilde{V}}_2\left[\begin{array}{c}{O}_{r\times r}\\ & T_{k-r}\end{array}\right]V_1^T=X_{\perp }{\tilde{V}}_2\theta V_1^T=\text{rhs of (15)}.\hfill \end{array}$$

Next, we work out

$$\begin{array}{c}\hfill s:=⟨\text{grad}f(\mathcal{X}),-{\text{Log}}_{\mathcal{X}}({\mathcal{V}}_{\alpha })⟩.\end{array}$$

Since $\text{grad}f(\mathcal{X})$ and ${\text{Log}}_{\mathcal{X}}({\mathcal{V}}_{\alpha })$, respectively, give tangent vectors for the same representative X of $\mathcal{X}$, the inner product above is the trace of the corresponding matrix representations. Using (16) with $I-X{X}^T=X_{\perp }{X}_{\perp }^T$, we therefore get

$$\begin{array}{cc}\hfill s& =2\langle (I-X{X}^T)AX,(I-X{X}^T)V_{\alpha }{U}_1\frac{\theta }{sin(\theta )}{V}_1^T\rangle \hfill \\ \hfill & =2\text{Tr}(\frac{\theta }{sin(\theta )}{U}_1^T{V}_{\alpha }^T(I-X{X}^T)AX{V}_1).\hfill \end{array}$$

Since $A{V}_{\alpha }=V_{\alpha }{\mathrm{\Lambda }}_{\alpha }$, we can simplify

$$\begin{array}{c}\hfill V_{\alpha }^T(I-X{X}^T)AX={\mathrm{\Lambda }}_{\alpha }{V}_{\alpha }^{T}X-V_{\alpha }^{T}X{X}^{T}AX.\end{array}$$ 17

Substituting in the expression above and using that $V_{\alpha }^{T}X=U_{1}cos\theta V_1^T$, we get

$$\begin{array}{cc}\hfill \frac{1}{2}s& =\text{Tr}(\frac{\theta }{sin(\theta )}{U}_1^T{\mathrm{\Lambda }}_{\alpha }{U}_{1}cos(\theta ))-\text{Tr}(\frac{\theta }{sin(\theta )}cos(\theta )V_1^T{X}^{T}AX{V}_1)\hfill \\ \hfill & =\text{Tr}(\frac{\theta }{tan(\theta )}(U_1^T{\mathrm{\Lambda }}_{\alpha }{U}_1-V_1^T{X}^{T}AX{V}_1)),\hfill \end{array}$$

with the convention $\frac{0}{tan0}=1$.

Denote the symmetric matrix

$$\begin{array}{c}\hfill S:=U_1^T{\mathrm{\Lambda }}_{\alpha }{U}_1-V_1^T{X}^{T}AX{V}_1.\end{array}$$ 18

We show below that all diagonal entries $S_{11},\dots ,S_{\mathit{kk}}$ of S are nonnegative. Hence, by diagonality of the matrix $\frac{\theta }{tan(\theta )}$, we obtain

$$\begin{array}{cc}\hfill \frac{1}{2}s& =\sum _j\frac{{\theta }_j}{tan{\theta }_j}{S}_{\mathit{jj}}\ge \underset{j}{min}\frac{{\theta }_j}{tan{\theta }_j}\text{Tr}(S)=\frac{{\theta }_k}{tan{\theta }_k}[\text{Tr}({\mathrm{\Lambda }}_{\alpha })-\text{Tr}(X^{T}AX)]\hfill \end{array}$$

since $U_1$ and $V_1$ are orthogonal matrices. We recover the desired result after substituting $f(\mathcal{X})=-\text{Tr}(X^{T}AX)$ and $f_{\ast }^{}=-\text{Tr}(V_{\alpha }^{T}A{V}_{\alpha })=-\text{Tr}({\mathrm{\Lambda }}_{\alpha })$.

It remains to show that $S_{\mathit{jj}}\ge 0$ for $j=1,\dots ,k$. Since $\text{span}(V_{\beta })=\text{span}{(V_{\alpha })}^{\perp }$, Lemma 4.3 gives us in addition to (13) also

$$\begin{array}{c}\hfill V_{\beta }^{T}X=U_2\left[\begin{array}{c}{O}_{m\times r}\\ & S_{k-r}\end{array}\right]V_1^T={\tilde{U}}_{2}sin\theta V_1^T,\end{array}$$ 19

where ${\tilde{U}}_2\in {\mathbb{R}}^{(n-k)\times k}$ contains the last k columns of the orthogonal matrix $U_2$ in order. A short calculation using (11) then shows that (18) satisfies

$$\begin{array}{c}\hfill S=U_1^T{\mathrm{\Lambda }}_{\alpha }{U}_1-cos\theta U_1^T{\mathrm{\Lambda }}_{\alpha }{U}_{1}cos\theta -sin\theta {\tilde{U}}_2^T{\mathrm{\Lambda }}_{\beta }{\tilde{U}}_{2}sin\theta \end{array}$$

with diagonal elements

$$\begin{array}{c}\hfill S_{\mathit{jj}}={sin}^2{\theta }_j{(U_1^T{\mathrm{\Lambda }}_{\alpha }{U}_1-{\tilde{U}}_2^T{\mathrm{\Lambda }}_{\beta }{\tilde{U}}_2)}_{\mathit{jj}}.\end{array}$$

Since $U_1$ and ${\tilde{U}}_2$ have orthonormal columns, we obtain

$$\begin{array}{c}\hfill {\lambda }_{min}(U_1^T{\mathrm{\Lambda }}_{\alpha }{U}_1)\ge {\lambda }_{min}({\mathrm{\Lambda }}_{\alpha })={\lambda }_k,{\lambda }_{max}({\tilde{U}}_2^T{\mathrm{\Lambda }}_{\beta }{\tilde{U}}_2)\le {\lambda }_{max}({\mathrm{\Lambda }}_{\beta })={\lambda }_{k+1},\end{array}$$

from which we get with Weyl’s inequality that

$$\begin{array}{c}\hfill {\lambda }_{min}(U_1^T{\mathrm{\Lambda }}_{\alpha }{U}_1-{\tilde{U}}_2^T{\mathrm{\Lambda }}_{\beta }{\tilde{U}}_2)\ge {\lambda }_{min}(U_1^T{\mathrm{\Lambda }}_{\alpha }{U}_1)-{\lambda }_{max}({\tilde{U}}_2^T{\mathrm{\Lambda }}_{\beta }{\tilde{U}}_2)\ge {\lambda }_k-{\lambda }_{k+1}\ge 0.\end{array}$$

Hence, the matrix

$$\begin{array}{c}\hfill U_1^T{\mathrm{\Lambda }}_{\alpha }{U}_1-{\tilde{U}}_2^T{\mathrm{\Lambda }}_{\beta }{\tilde{U}}_2\end{array}$$ 20

is symmetric and positive semi-definite. Its diagonal entries, and thus also $S_{\mathit{jj}}$, are therefore nonnegative. $\square$

We now arrive at a useful property of f that will later allow us to analyze the convergence of Riemannian steepest descent. It is a weaker version of strong geodesic convexity and can be proved easily using quadratic growth and weak-quasi-convexity.

Theorem 4.1

(Weak-strong convexity) Let $0\le {\theta }_1\le \cdots \le {\theta }_k<\pi /2$ be the principal angles between the subspaces $\mathcal{X}$ and ${\mathcal{V}}_{\alpha }$. Then, f satisfies

$$\begin{array}{c}\hfill f(\mathcal{X})-f^{\ast }\le \frac{1}{a(\mathcal{X})}⟨\text{grad}f(\mathcal{X}),-{\text{Log}}_{\mathcal{X}}({\mathcal{V}}_{\alpha })⟩-c_Q\delta {\text{dist}}^2(\mathcal{X},{\mathcal{V}}_{\alpha })\end{array}$$

with $a(\mathcal{X})={\theta }_k/tan{\theta }_k>0$, $c_Q=4/{\pi }^2>0.4$, and $\delta ={\lambda }_k-{\lambda }_{k+1}\ge 0$.

Proof

Combining Propositions 2 and 3 leads to

$$\begin{array}{c}\hfill c_Q\delta {\text{dist}}^2(\mathcal{X},{\mathcal{V}}_{\alpha })\le f(\mathcal{X})-f^{\ast }\le \frac{1}{2a(\mathcal{X})}⟨\text{grad}f(\mathcal{X}),-{\text{Log}}_{\mathcal{X}}({\mathcal{V}}_{\alpha })⟩.\end{array}$$

At the same time, Proposition 3 also implies

$$\begin{array}{cc}\hfill f(\mathcal{X})-f^{\ast }& \le \frac{1}{2a(\mathcal{X})}⟨\text{grad}f(\mathcal{X}),-{\text{Log}}_{\mathcal{X}}({\mathcal{V}}_{\alpha })⟩-c_Q\delta {\text{dist}}^2(\mathcal{X},{\mathcal{V}}_{\alpha })\hfill \\ \hfill & +c_Q\delta {\text{dist}}^2(\mathcal{X},{\mathcal{V}}_{\alpha }).\hfill \end{array}$$

Using the first inequality to bound the last term of the right hand side, we recover the desired result. $\square$

Remark 4.1

Theorem 4.1 is also valid when the eigengap $\delta =0$. In that case, ${\mathcal{V}}_{\alpha }$ is any subspace spanned by k leading eigenvectors of A and the theorem (almost) reduces to Proposition 3 (up to a scalar 2).

While not needed for our convergence proof, the next result is of independent interest and shows that f is gradient dominated in the Riemannian sense when the eigengap $\delta$ is strictly positive. This property is the Riemannian version of the Polyak–Łojasiewicz inequality and generalizes a result by [34] for the Rayleigh quotient on the sphere.

Proposition 4

(Gradient dominance) The function f satisfies

$$\begin{array}{c}\hfill {\Vert \text{grad}f(\mathcal{X})\Vert }^2\ge 4c_Q\delta a^2(\mathcal{X})(f(\mathcal{X})-f^{\ast })\end{array}$$

for all subspaces $\mathcal{X}$ that have a largest principal angle $<\pi /2$ with ${\mathcal{V}}_{\alpha }$.

Proof

We assume that $\delta >0$ since otherwise the statement is trivially true. By Theorem 4.1, we have

$$\begin{array}{c}\hfill f(\mathcal{X})-f^{\ast }\le \frac{1}{a(\mathcal{X})}⟨\text{grad}f(\mathcal{X}),-{\text{Log}}_{\mathcal{X}}({\mathcal{V}}_{\alpha })⟩-c_Q\delta {\text{dist}}^2(\mathcal{X},{\mathcal{V}}_{\alpha }).\end{array}$$

Since $⟨G_1,G_2⟩\le \frac{\rho }{2}\Vert G_1{\Vert }^2+\frac{1}{2\rho }{\Vert G_2\Vert }^2$ for all matrices $G_1,G_2$ and $\rho >0$, we can write (for any $\rho >0$) that

$$\begin{array}{c}\hfill ⟨\text{grad}f(\mathcal{X}),-{\text{Log}}_{\mathcal{X}}({\mathcal{V}}_{\alpha })⟩\le \frac{\rho }{2}\Vert \text{grad}f(\mathcal{X}){\Vert }^2+\frac{1}{2\rho }{\Vert {\text{Log}}_{\mathcal{X}}({\mathcal{V}}_{\alpha })\Vert }^2.\end{array}$$

Using that $\text{dist}(\mathcal{X},{\mathcal{V}}_{\alpha })=\Vert {\text{Log}}_{\mathcal{X}}({\mathcal{V}}_{\alpha })\Vert$ and choosing $\rho =1/(2c_Q\delta a(\mathcal{X}))$, we get the desired result. $\square$

Convergence of Riemannian Steepest Descent

We now have everything in place to prove the convergence of the Riemannian steepest descent (RSD) method on the Grassmann manifold for minimizing f. Starting from a subspace ${\mathcal{X}}_0\in \text{Gr}(n,k)$, we iterate

$$\begin{array}{c}\hfill {\mathcal{X}}_{t+1}={\text{Exp}}_{{\mathcal{X}}_t}(-{\eta }_t\text{grad}f({\mathcal{X}}_t)).\end{array}$$ 21

Here, ${\eta }_t>0$ is a step size that may depend on the iteration t and will be carefully chosen depending on the specific case, but always depending on $\gamma$, which equals $2({\lambda }_1-{\lambda }_n)$.

We start by a general result which shows that the distance to the optimal subspace contracts after one step of steepest descent. The step size depends on the smoothness and weak-quasi-convexity constants of f from Propositions 1 and 3. This is crucial since the constant $a(\mathcal{X})$ depends on the biggest principal angle between $\mathcal{X}$ and ${\mathcal{V}}_{\alpha }$ and bounding the evolution of distances of the iterates to the minimizer will help us also bound this constant.5 An alternative contraction property with a more tractable step size is presented in Proposition 6 of Appendix B.

Lemma 5.1

(Contraction of RSD) Let ${\mathcal{X}}_t$ and ${\mathcal{V}}_{\alpha }$ have principal angles $0\le {\theta }_1\le \cdots \le {\theta }_k<\pi /2$. Then, iteration (21) with $0\le {\eta }_t\le a({\mathcal{X}}_t)/\gamma$ satisfies

$$\begin{array}{c}\hfill {\text{dist}}^2({\mathcal{X}}_{t+1},{\mathcal{V}}_{\alpha })\le (1-2c_Q\delta a({\mathcal{X}}_t){\eta }_t){\text{dist}}^2({\mathcal{X}}_t,{\mathcal{V}}_{\alpha }).\end{array}$$

Observe that $\gamma =0$ implies $A={\lambda }_{1}I$ and any subspace $\mathcal{X}$ of dimension k will be an eigenspace of A with $\text{dist}(\mathcal{X},{\mathcal{V}}_{\alpha })=0$. We will therefore not explicitly prove this lemma and all forthcoming convergence results for $\gamma =0$ since the statements will be trivially true.

Proof of Lemma 5.1

By the assumption on the principal angles, we get that $0<a({\mathcal{X}}_t)={\theta }_k/tan{\theta }_k\le 1$. The hypothesis on ${\eta }_t$ and Lemma 4.1 then gives

$$\begin{array}{c}\hfill {\eta }_t\Vert \text{grad}f({\mathcal{X}}_t){\Vert }_2\le \frac{a({\mathcal{X}}_t)}{\gamma }{\Vert \text{grad}f({\mathcal{X}}_t)\Vert }_2\le \frac{1}{2}<\frac{\pi }{2}.\end{array}$$

By (5), this guarantees that the geodesic $\tau \mapsto \text{Exp}(-\tau {\eta }_t\text{grad}f({\mathcal{X}}_t))$ lies within the injectivity domain at ${\mathcal{X}}_t$ for $\tau \in [0,1]$. Hence, $\text{Exp}$ is bijective along this geodesic and thus ${\text{Log}}_{{\mathcal{X}}_t}({\mathcal{X}}_{t+1})=-{\eta }_t\text{grad}f({\mathcal{X}}_t)$. We can thus apply Lemma 3.1 to obtain

$$\begin{array}{cc}\hfill {\text{dist}}^2({\mathcal{X}}_{t+1},{\mathcal{V}}_{\alpha })& \le \Vert -{\eta }_t\text{grad}f({\mathcal{X}}_t)-{\text{Log}}_{{\mathcal{X}}_t}({\mathcal{V}}_{\alpha }){\Vert }^2\hfill \\ \hfill & ={\eta }_t^2{\Vert \text{grad}f({\mathcal{X}}_t)\Vert }^2+{\text{dist}}^2({\mathcal{X}}_t,{\mathcal{V}}_{\alpha })+2{\eta }_t\sigma \hfill \end{array}$$ 22

with

$$\begin{array}{c}\hfill \sigma :=⟨\text{grad}f({\mathcal{X}}_t),{\text{Log}}_{{\mathcal{X}}_t}({\mathcal{V}}_{\alpha })⟩.\end{array}$$

Theorem 4.1 and Lemma 4.2 together with Proposition 1 give

$$\begin{array}{cc}\hfill \frac{\sigma }{a({\mathcal{X}}_t)}& \le f^{\ast }-f({\mathcal{X}}_t)-c_Q\delta {\text{dist}}^2({\mathcal{X}}_t,{\mathcal{V}}_{\alpha })\hfill \\ \hfill & \le -\frac{1}{2\gamma }{\Vert \text{grad}f({\mathcal{X}}_t)\Vert }^2-c_Q\delta {\text{dist}}^2({\mathcal{X}}_t,{\mathcal{V}}_{\alpha }).\hfill \end{array}$$

Multiplying by $2a({\mathcal{X}}_t){\eta }_t$ and using ${\eta }_t\le a({\mathcal{X}}_t)/\gamma$, we get

$$\begin{array}{cc}\hfill 2{\eta }_t\sigma & \le -\frac{a({\mathcal{X}}_t){\eta }_t}{\gamma }{\Vert \text{grad}f({\mathcal{X}}_t)\Vert }^2-2c_Q\delta a({\mathcal{X}}_t){\eta }_t{\text{dist}}^2({\mathcal{X}}_t,{\mathcal{V}}_{\alpha })\hfill \\ \hfill & \le -{\eta }_t^2{\Vert \text{grad}f({\mathcal{X}}_t)\Vert }^2-2c_Q\delta a({\mathcal{X}}_t){\eta }_t{\text{dist}}^2({\mathcal{X}}_t,{\mathcal{V}}_{\alpha }).\hfill \end{array}$$

Substituting into (22), we obtain the first statement of the lemma. $\square$

Remark 5.1

When $\delta =0$, Lemma 5.1 still holds for any subspace ${\mathcal{V}}_{\alpha }$ spanned by k leading eigenvectors of A. In that case, the lemma only guarantees that the distance between the iterates of steepest descent and this ${\mathcal{V}}_{\alpha }$ does not increase.

dummy

Linear Convergence Rate Under Positive Eigengap

Lemma 5.1 features a contraction rate only for one step of the algorithm. In order to get a global convergence rate, one needs to bound the quantity $a({\mathcal{X}}_t)$ from below and independently of t. To that end, we need a stricter bound in the distance of the initial guess to the optimum. Such a bound guarantees that $a(X_t)$ remains always lower bounded by a positive number, or equivalently, that the iterates of the algorithm never get too close to a non-optimal critical point.

Theorem 5.1

If $\text{dist}({\mathcal{X}}_0,{\mathcal{V}}_{\alpha })<\pi /2$ then the iterates ${\mathcal{X}}_t$ of Riemannian steepest descent (21) with step size ${\eta }_t$ such that

$$\begin{array}{c}\hfill 0<\eta \le {\eta }_t\le cos(\text{dist}({\mathcal{X}}_0,{\mathcal{V}}_{\alpha }))/\gamma \end{array}$$

satisfy

$$\begin{array}{c}\hfill {\text{dist}}^2({\mathcal{X}}_t,{\mathcal{V}}_{\alpha })\le {\left(1,-,2,c_Q,cos,(\text{dist}({\mathcal{X}}_0,{\mathcal{V}}_{\alpha })),,\delta ,,\eta \right)}^t{\text{dist}}^2({\mathcal{X}}_0,{\mathcal{V}}_{\alpha }).\end{array}$$

Proof

We first claim that $\text{dist}({\mathcal{X}}_t,{\mathcal{V}}_{\alpha })\le \text{dist}({\mathcal{X}}_0,{\mathcal{V}}_{\alpha })$ for all $t\ge 0$. This would then also imply that ${\theta }_k({\mathcal{X}}_t,{\mathcal{V}}_{\alpha })<\pi /2$ for all $t\ge 0$ since

$$\begin{array}{c}\hfill {\theta }_k({\mathcal{X}}_t,{\mathcal{V}}_{\alpha })\le \sqrt{\sum _{i=1}^k{\theta }_i{({\mathcal{X}}_t,{\mathcal{V}}_{\alpha })}^2}=\text{dist}({\mathcal{X}}_t,{\mathcal{V}}_{\alpha }).\end{array}$$

For $t=0$, we have ${\theta }_k({\mathcal{X}}_0,{\mathcal{V}}_{\alpha })<\pi /2$ by hypothesis on ${\mathcal{X}}_0$ and thus

$$\begin{array}{c}\hfill a({\mathcal{X}}_0)=\frac{{\theta }_k({\mathcal{X}}_0,{\mathcal{V}}_{\alpha })}{tan({\theta }_k({\mathcal{X}}_0,{\mathcal{V}}_{\alpha }))}\ge cos({\theta }_k({\mathcal{X}}_0,{\mathcal{V}}_{\alpha }))\ge cos(\text{dist}({\mathcal{X}}_0,{\mathcal{V}}_{\alpha })).\end{array}$$

Since by construction ${\eta }_0\le cos(\text{dist}({\mathcal{X}}_0,{\mathcal{V}}_{\alpha }))/\gamma$, this implies that ${\eta }_0\le a({\mathcal{X}}_0)/\gamma$ and Lemma 5.1 guarantees that $\text{dist}({\mathcal{X}}_1,{\mathcal{V}}_{\alpha })\le \text{dist}({\mathcal{X}}_0,{\mathcal{V}}_{\alpha })$. In particular, we also have ${\theta }_k({\mathcal{X}}_1,{\mathcal{V}}_{\alpha })<\pi /2$.

Next, assume that

$$\begin{array}{c}\hfill \text{dist}({\mathcal{X}}_t,{\mathcal{V}}_{\alpha })\le \text{dist}({\mathcal{X}}_0,{\mathcal{V}}_{\alpha }),\end{array}$$

which implies ${\theta }_k({\mathcal{X}}_t,{\mathcal{V}}_{\alpha })<\pi /2$. Then by a similar argument like above, we have

$$\begin{array}{c}\hfill a({\mathcal{X}}_t)\ge cos(\text{dist}({\mathcal{X}}_t,{\mathcal{V}}_{\alpha }))\ge cos(\text{dist}({\mathcal{X}}_0,{\mathcal{V}}_{\alpha })).\end{array}$$ 23

By hypothesis on ${\eta }_t$, we observe

$$\begin{array}{c}\hfill {\eta }_t\le \frac{cos(\text{dist}({\mathcal{X}}_0,{\mathcal{V}}_{\alpha }))}{\gamma }\le \frac{cos(\text{dist}({\mathcal{X}}_t,{\mathcal{V}}_{\alpha }))}{\gamma }\le \frac{a({\mathcal{X}}_t)}{\gamma }.\end{array}$$

Applying Lemma 5.1 once again with the induction hypothesis proves the claim:

$$\begin{array}{c}\hfill \text{dist}({\mathcal{X}}_{t+1},{\mathcal{V}}_{\alpha })\le \text{dist}({\mathcal{X}}_t,{\mathcal{V}}_{\alpha })\le \text{dist}({\mathcal{X}}_0,{\mathcal{V}}_{\alpha }).\end{array}$$

The main statement of the theorem now follows easily: Since ${\eta }_t\le a({\mathcal{X}}_t)/\gamma$ and ${\theta }_k({\mathcal{X}}_t,{\mathcal{V}}_{\alpha })<\pi /2$ for all $t\ge 0$, Lemma 5.1 gives

$$\begin{array}{c}\hfill {\text{dist}}^2({\mathcal{X}}_{t+1},{\mathcal{V}}_{\alpha })\le \left(1,-,2,c_Q,a,({\mathcal{X}}_t),\delta ,{\eta }_t\right){\text{dist}}^2({\mathcal{X}}_t,{\mathcal{V}}_{\alpha }).\end{array}$$

Combining with (23) and ${\eta }_t\ge \eta$ shows the desired result by induction. $\square$

If the eigengap $\delta$ is strictly positive, then Theorem 5.1 gives an exponential convergence rate towards the optimum ${\mathcal{V}}_{\alpha }$. If $\delta =0$, then Theorem 5.1does not provide a convergence rate but rather implies that the intrinsic distances of the iterates to the optimum do not increase.

From Theorem 5.1 we get immediately the following iteration complexity.

Corollary 5.1

Let Riemannian steepest descent be started from a subspace ${\mathcal{X}}_0$ that satisfies $\text{dist}({\mathcal{X}}_0,{\mathcal{V}}_{\alpha })<\pi /2$ and with step-size $\eta$ satisfying the condition of Theorem 5.1. Then after at most

$$\begin{array}{c}\hfill T=2\frac{log(\epsilon )-log(\text{dist}({\mathcal{X}}_0,{\mathcal{V}}_{\alpha }))}{log(1-0.8cos(\text{dist}({\mathcal{X}}_0,{\mathcal{V}}_{\alpha }))\delta \eta )}+1\le \mathcal{O}\left(\frac{log(\text{dist}({\mathcal{X}}_0,{\mathcal{V}}_{\alpha }))-log(\epsilon )}{cos(\text{dist}({\mathcal{X}}_0,{\mathcal{V}}_{\alpha }))\delta \eta }\right)\end{array}$$

many iterations, ${\mathcal{X}}_T$ will satisfy $\text{dist}({\mathcal{X}}_T,{\mathcal{V}}_{\alpha })\le \epsilon$. With the maximal step size allowed in Theorem 5.1, we get

$$\begin{array}{c}\hfill T\le \mathcal{O}\left(\frac{{\lambda }_1-{\lambda }_n}{\delta },\frac{1}{{cos}^2(\text{dist}({\mathcal{X}}_0,{\mathcal{V}}_{\alpha }))},log,\left(\frac{\text{dist}({\mathcal{X}}_0,{\mathcal{V}}_{\alpha })}{\epsilon }\right)\right).\end{array}$$

Proof

In order to guarantee $\text{dist}({\mathcal{X}}_T,{\mathcal{V}}_{\alpha })\le ϵ$, it suffices to have

$$\begin{array}{c}\hfill {\left(1,-,2,c_Q,cos,(\text{dist}({\mathcal{X}}_0,{\mathcal{V}}_{\alpha })),,\delta ,,\eta \right)}^T{\text{dist}}^2({\mathcal{X}}_0,{\mathcal{V}}_{\alpha })\le {ϵ}^2.\end{array}$$

Taking the logarithm of both sides, we get

$$\begin{array}{c}\hfill Tlog(1-2c_{Q}cos(\text{dist}({\mathcal{X}}_0,{\mathcal{V}}_{\alpha }))\delta \eta )+2log(\text{dist}({\mathcal{X}}_0,{\mathcal{V}}_{\alpha }))\le 2log(ϵ),\end{array}$$

which gives

$$\begin{array}{c}\hfill T\ge 2\frac{log(ϵ)-log(\text{dist}({\mathcal{X}}_0,{\mathcal{V}}_{\alpha }))}{log(1-2c_{Q}cos(\text{dist}({\mathcal{X}}_0,{\mathcal{V}}_{\alpha }))},\end{array}$$

since $log(1-2c_{Q}cos(\text{dist}({\mathcal{X}}_0,{\mathcal{V}}_{\alpha }))$ is negative. By considering that $c_Q\ge 0.8$, we get

$$\begin{array}{c}\hfill T\ge 2\frac{log(ϵ)-log(\text{dist}({\mathcal{X}}_0,{\mathcal{V}}_{\alpha }))}{log(1-0.8cos(\text{dist}({\mathcal{X}}_0,{\mathcal{V}}_{\alpha }))\delta \eta )},\end{array}$$

and the smallest integer that satisfies this inequality is exactly

$$\begin{array}{c}\hfill T=2\frac{log(ϵ)-log(\text{dist}({\mathcal{X}}_0,{\mathcal{V}}_{\alpha }))}{log(1-0.8cos(\text{dist}({\mathcal{X}}_0,{\mathcal{V}}_{\alpha }))\delta \eta )}.\end{array}$$

The inequality part of the result follows by considering that

$$\begin{array}{c}\hfill log(1-0.8cos(\text{dist}({\mathcal{X}}_0,{\mathcal{V}}_{\alpha })\delta \eta )\ge -cos(\text{dist}({\mathcal{X}}_0,{\mathcal{V}}_{\alpha }))\delta \eta .\end{array}$$

The final bound for T follows by a simple substitution of $\eta =cos(\text{dist}({\mathcal{X}}_0,{\mathcal{V}}_{\alpha }))/\gamma$.

As expected, T depends inversely proportional on the eigengap $\delta$ and proportional to the spread of the eigenvalues. In addition, we also have an extra term $1/{cos}^2(\text{dist}({\mathcal{X}}_0,{\mathcal{V}}_{\alpha }))$ that depends on the initial distance $\text{dist}({\mathcal{X}}_0,{\mathcal{V}}_{\alpha })$, which is due to the weak-quasi-convexity property of f. This is a conservative overestimation, since this quantity improves as the iterates get closer to the optimum.

Remark 5.2

If $\delta >0$, the exponential convergence rate is in terms of the intrinsic distance on the Grassmann manifold, that is, the ${\ell }_2$ norm of the principal angles. Standard convergence results for subspace iteration are stated for the biggest principal angle, that is, the ${\ell }_{\infty }$ norm. This is weaker than the intrinsic distance. For subspace iteration with projection, the convergence result from [31, Thm. 5.2] shows that all principal angles ${\theta }_i$ converge to zero and eventually gives convergence of the ${\ell }_4$ norm of the principal angles. This is also weaker than the intrinsic distance.

Convergence of Function Values Without an Eigengap Assumption

When $\delta =0$, Theorem 5.1 still holds, but does not provide a rate of convergence as discussed above. Instead, we can prove the following result:

Theorem 5.2

If the distance $\text{dist}({\mathcal{X}}_0,{\mathcal{V}}_{\alpha })$ of the initial subspace ${\mathcal{X}}_0$ to the minimizer satisfies $\text{dist}({\mathcal{X}}_0,{\mathcal{V}}_{\alpha })<\pi /2$ for a subspace ${\mathcal{V}}_{\alpha }$ that is spanned by any k leading eigenvectors of A, then the iterates ${\mathcal{X}}_t$ of Riemannian steepest descent (21) with fixed step size

$$\begin{array}{c}\hfill \eta \le cos(\text{dist}({\mathcal{X}}_0,{\mathcal{V}}_{\alpha }))/\gamma \end{array}$$

satisfy

$$\begin{array}{c}\hfill f({\mathcal{X}}_t)-f^{\ast }\le \frac{2\gamma +\frac{1}{\eta }}{4(cos(\text{dist}({\mathcal{X}}_0,{\mathcal{V}}_{\alpha }))t+1)}{\text{dist}}^2({\mathcal{X}}_0,{\mathcal{V}}_{\alpha })=\mathcal{O}\left(\frac{1}{t}\right).\end{array}$$

Proof

Since we satisfy all the hypotheses of Theorem 5.1, we know that for all $t\ge 0$ it holds $\text{dist}({\mathcal{X}}_t,{\mathcal{V}}_{\alpha })\le \text{dist}({\mathcal{X}}_0,{\mathcal{V}}_{\alpha })<\pi /2$ and thus also that ${\mathcal{X}}_t$ is in the injectivity domain of $\text{Exp}$ at ${\mathcal{V}}_{\alpha }$. In addition, its proof states in (23) that

$$\begin{array}{c}\hfill a({\mathcal{X}}_t)\ge C_0:=cos(\text{dist}({\mathcal{X}}_0,{\mathcal{V}}_{\alpha }))>0,\end{array}$$

which implies that the function f is weakly-quasi-convex at every ${\mathcal{X}}_t$ with constant $2C_0$. Hence

$$\begin{array}{c}\hfill 2C_0{\mathrm{\Delta }}_t\le ⟨\text{grad}f({\mathcal{X}}_t),-{\text{Log}}_{{\mathcal{X}}_t}({\mathcal{V}}_{\alpha })⟩,\end{array}$$ 24

where we defined

$$\begin{array}{c}\hfill {\mathrm{\Delta }}_t:=f({\mathcal{X}}_t)-f^{\ast }.\end{array}$$

Similar to the proof of Theorem 5.1, by the hypothesis on the step size ${\eta }_t$, Lemma 5.1 shows that ${\mathcal{X}}_{t+1}$ is in the injectivity domain of $\text{Exp}$ at ${\mathcal{X}}_t$. Hence, by the definition of Riemannian steepest descent, we have

$$\begin{array}{c}\hfill {\text{Log}}_{{\mathcal{X}}_t}({\mathcal{X}}_{t+1})=-\eta \text{grad}f({\mathcal{X}}_t).\end{array}$$ 25

In addition, the smoothness property (10) of f gives

$$\begin{array}{c}\hfill {\mathrm{\Delta }}_{t+1}-{\mathrm{\Delta }}_t\le ⟨\text{grad}f({\mathcal{X}}_t),{\text{Log}}_{{\mathcal{X}}_t}({\mathcal{X}}_{t+1})⟩+\frac{\gamma }{2}{\text{dist}}^2({\mathcal{X}}_t,{\mathcal{X}}_{t+1}).\end{array}$$

Substituting (25), we obtain

$$\begin{array}{c}\hfill {\mathrm{\Delta }}_{t+1}-{\mathrm{\Delta }}_t\le \left(-,\eta ,+,\frac{\gamma }{2},{\eta }^2\right){\Vert \text{grad}f({\mathcal{X}}_t)\Vert }^2\le 0,\end{array}$$ 26

since $\eta \le C_0/\gamma$ with $0<C_0:=cos(\text{dist}({\mathcal{X}}_0,{\mathcal{V}}_{\alpha }))\le 1$ and $\gamma >0$.

Since $\text{Gr}(n,k)$ has nonnegative sectional curvature, Lemma 3.2 implies

$$\begin{array}{c}\hfill {\text{dist}}^2({\mathcal{X}}_{t+1},{\mathcal{V}}_{\alpha })\le {\text{dist}}^2({\mathcal{X}}_t,{\mathcal{X}}_{t+1})+{\text{dist}}^2({\mathcal{X}}_t,{\mathcal{V}}_{\alpha })-2⟨{\text{Log}}_{{\mathcal{X}}_t}({\mathcal{X}}_{t+1}),{\text{Log}}_{{\mathcal{X}}_t}({\mathcal{V}}_{\alpha })⟩.\end{array}$$

Substituting (25) into the above and rearranging terms gives

$$\begin{array}{c}\hfill 2\eta ⟨\text{grad}f({\mathcal{X}}_t),-{\text{Log}}_{{\mathcal{X}}_t}({\mathcal{V}}_{\alpha })⟩\le {\text{dist}}^2({\mathcal{X}}_t,{\mathcal{V}}_{\alpha })-{\text{dist}}^2({\mathcal{X}}_{t+1},{\mathcal{V}}_{\alpha })+{\eta }^2{\Vert \text{grad}f({\mathcal{X}}_t)\Vert }^2.\end{array}$$

Combining with (24), we get

$$\begin{array}{c}\hfill {\mathrm{\Delta }}_t\le \frac{1}{4C_0\eta }({\text{dist}}^2({\mathcal{X}}_t,{\mathcal{V}}_{\alpha })-{\text{dist}}^2({\mathcal{X}}_{t+1},{\mathcal{V}}_{\alpha }))+\frac{\eta }{4C_0}{\Vert \text{grad}f({\mathcal{X}}_t)\Vert }^2.\end{array}$$ 27

Now multiplying (26) by $\frac{1}{C_0}$ and summing with (27) gives

$$\begin{array}{cc}& \frac{1}{C_0}{\mathrm{\Delta }}_{t+1}-\left(\frac{1}{C_0},-,1\right){\mathrm{\Delta }}_t\le \frac{1}{4C_0\eta }({\text{dist}}^2({\mathcal{X}}_t,{\mathcal{V}}_{\alpha })-{\text{dist}}^2({\mathcal{X}}_{t+1},{\mathcal{V}}_{\alpha }))\hfill \\ \hfill & +\frac{1}{C_0}\left(-,\eta ,+,\frac{\gamma }{2},{\eta }^2,+,\frac{\eta }{4}\right){\Vert \text{grad}f({\mathcal{X}}_t)\Vert }^2.\hfill \end{array}$$ 28

By assumption $\eta \le C_0/\gamma$, where $0<C_0:=cos(\text{dist}({\mathcal{X}}_0,{\mathcal{V}}_{\alpha }))\le 1$ and $\gamma >0$. Since

$$\begin{array}{c}\hfill \frac{\eta }{C_0}\left(-,1,+,\frac{\gamma }{2},\eta ,+,\frac{1}{4}\right)\le \frac{\eta }{C_0}\left(\frac{C_0}{2},-,\frac{3}{4}\right)\le -\frac{1}{4}\frac{\eta }{C_0}<0.\end{array}$$

Inequality (28) can be simplified to

$$\begin{array}{c}\hfill \frac{1}{C_0}{\mathrm{\Delta }}_{t+1}-\left(\frac{1}{C_0},-,1\right){\mathrm{\Delta }}_t\le \frac{1}{4C_0\eta }({\text{dist}}^2({\mathcal{X}}_t,{\mathcal{V}}_{\alpha })-{\text{dist}}^2({\mathcal{X}}_{t+1},{\mathcal{V}}_{\alpha })).\end{array}$$

Summing from 0 to $t-1$ gives

$$\begin{array}{c}\hfill \frac{1}{C_0}{\mathrm{\Delta }}_t+\sum _{s=1}^{t-1}{\mathrm{\Delta }}_s-\left(\frac{1}{C_0},-,1\right){\mathrm{\Delta }}_0\le \frac{1}{4C_0\eta }\left({\text{dist}}^2,({\mathcal{X}}_0,{\mathcal{V}}_{\alpha }),-,{\text{dist}}^2,({\mathcal{X}}_t,{\mathcal{V}}_{\alpha })\right).\end{array}$$

From the smoothness property (10) at the critical point ${\mathcal{V}}_{\alpha }$ of f, we get

$$\begin{array}{c}\hfill {\mathrm{\Delta }}_0\le \frac{\gamma }{2}{\text{dist}}^2({\mathcal{X}}_0,{\mathcal{V}}_{\alpha }).\end{array}$$

Combining these two inequalities then leads to

$$\begin{array}{cc}\hfill \frac{1}{C_0}{\mathrm{\Delta }}_t+\sum _{s=0}^{t-1}{\mathrm{\Delta }}_s& \le \frac{1}{C_0}{\mathrm{\Delta }}_0+\frac{1}{4C_0\eta }{\text{dist}}^2({\mathcal{X}}_0,{\mathcal{V}}_{\alpha })\hfill \\ \hfill & \le \frac{1}{2C_0}\left(\gamma ,+,\frac{1}{2\eta }\right){\text{dist}}^2({\mathcal{X}}_0,{\mathcal{V}}_{\alpha }).\hfill \end{array}$$

Since (26) holds for all $t\ge 0$, it also implies ${\mathrm{\Delta }}_t\le {\mathrm{\Delta }}_s$ for all $1\le s\le t$. Substituting

$$\begin{array}{c}\hfill t{\mathrm{\Delta }}_t\le \sum _{s=0}^{t-1}{\mathrm{\Delta }}_s\end{array}$$

into the inequality from above,

$$\begin{array}{c}\hfill {\mathrm{\Delta }}_t\le \frac{1}{2C_0}\frac{\gamma +\frac{1}{2\eta }}{\frac{1}{C_0}+t}{\text{dist}}^2({\mathcal{X}}_0,{\mathcal{V}}_{\alpha })=\frac{\gamma +\frac{1}{2\eta }}{2(C_{0}t+1)}{\text{dist}}^2({\mathcal{X}}_0,{\mathcal{V}}_{\alpha }),\end{array}$$

we obtain the desired result. $\square$

Remark 5.3

This type of result is standard for functions that are geodesically convex (see, e.g. [35]). Our objective function does not satisfy this property, but we can still have a similar upper bound on the iteration complexity for convergence in function value. We note that this does not imply convergence of the iterates to a specific k-dimensional subspace, but only convergence of a subsequence of the sequence of the iterates.

Sufficiently Small Step Sizes

The convergence results in Theorems 5.1 and 5.2 require that the initial subspace ${\mathcal{X}}_0$ lies within a distance strictly less than $\pi /2$ from a global minimizer ${\mathcal{V}}_{\alpha }$. While this condition is independent from the eigengap (unlike results that rely on standard convexity, see appendix), it is also not fully satisfactory: it is hard to verify in practice, and it is unnecessarily severe in numerical experiments. In fact, this condition is only used to obtain a uniform lower bound on the weak-quasi-convexity constant $a({\mathcal{X}}_t)={\theta }_k^{(t)}/tan({\theta }_k^{(t)})$ with ${\theta }_k^{(t)}$ the largest principal angle between ${\mathcal{X}}_t$ and ${\mathcal{V}}_{\alpha }$. Since the Riemannian distance is the ${\ell }_2$ norm of the principal angles, a contraction in this distance leads automatically to ${\theta }_k^{(t)}<\pi /2$ if ${\theta }_k^{(0)}<\pi /2$. If one could guarantee by some other reasoning that ${\theta }_k^{(t)}$ does not increase after one step, the condition $\text{dist}({\mathcal{X}}_0,{\mathcal{V}}_{\alpha })<\pi /2$ would not be needed.

We now show that for sufficiently small step sizes ${\eta }_t$, the largest principal angle ${\theta }_k^{(t)}$ between ${\mathcal{X}}_t$ and ${\mathcal{V}}_{\alpha }$ does indeed not increase after each iteration of Riemannian steepest descent regardless of the initial subspace ${\mathcal{X}}_0$. While it does not explain what we observe in numerical experiments where large steps can be taken, it is a first result in explaining why we can initialize the iteration at a random initial subspace ${\mathcal{X}}_0$.

Proposition 5

Riemannian steepest descent started from a subspace ${\mathcal{X}}_t$ returns a subspace ${\mathcal{X}}_{t+1}$ such that

$$\begin{array}{c}\hfill {\theta }_k({\mathcal{X}}_{t+1},{\mathcal{V}}_{\alpha })\le {\theta }_k({\mathcal{X}}_t,{\mathcal{V}}_{\alpha }),\end{array}$$

for all step sizes $0\le \eta \le \overline{\eta }$ where $\overline{\eta }>0$ is sufficiently small.

For the proof of this proposition, we will need the derivatives of certain singular values. While this is well known for isolated singular values, it is possible to generalize to higher multiplicities as well by relaxing the ordering and sign of singular values [10]. For a concrete formula, we use the following result from Lemma A.5 in [24].

Lemma 5.2

Let ${\sigma }_1\ge \cdots \ge {\sigma }_n$ be the singular values of S $\in {\mathbb{R}}^{n\times n}$ with $u_1,\dots ,u_n$ and $v_1,\dots ,v_n$ the associated left and right orthonormal singular vectors. Suppose that ${\sigma }_j$ has multiplicity m, that is,

$${\sigma }_{j_0-1}>{\sigma }_{j_0}=\cdots ={\sigma }_j=\cdots ={\sigma }_{j_0+m-1}>{\sigma }_{j_0+m}.$$

Then, the jth singular value of $S+\eta$T satisfies

$$\begin{array}{c}\hfill {\sigma }_j(S+\eta T)={\sigma }_j+\eta {\lambda }_{j-j_0+1}+\mathcal{O}({\eta }^2),\eta \to 0^{+},\end{array}$$

where ${\lambda }_j$ is the jth largest eigenvalue of $\frac{1}{2}(U^{T}BV+V^T{B}^{T}U)$ with

$$\begin{array}{c}\hfill U=\left[\begin{array}{ccc}{u}_{j_0}& \cdots & u_{j_0+m-1}\end{array}\right]\text{and}V=\left[\begin{array}{ccc}{v}_{j_0}& \cdots & v_{j_0+m-1}\end{array}\right].\end{array}$$

Proof of Proposition 5

For ease of notation, let $X:=X_t$ and $X_{+}:=X_{t+1}$ such that ${\mathcal{X}}_t=\text{span}(X)$ and ${\mathcal{X}}_{t+1}=\text{span}(X_{+})$. By definition of the exponential map on Grassmann, the next iterate of the Riemannian SD iteration (21) with step $\eta$ satisfies

$$\begin{array}{c}\hfill X_{+}=XVcos(\eta \mathrm{\Sigma })V^T+Usin(\eta \mathrm{\Sigma })V^T\end{array}$$

where

$$\begin{array}{c}\hfill U\mathrm{\Sigma }{V}^T=-\text{grad}f({\mathcal{X}}_t).\end{array}$$

Since V is orthogonal, we can write

$$\begin{array}{c}\hfill Usin(\eta \mathrm{\Sigma })V^T=U(\eta \mathrm{\Sigma })V^{T}V\left(\frac{sin(\eta \mathrm{\Sigma })}{\eta \mathrm{\Sigma }}\right)V^T=-\eta \text{grad}f({\mathcal{X}}_t)V\left(\frac{sin(\eta \mathrm{\Sigma })}{\eta \mathrm{\Sigma }}\right)V^T\end{array}$$

where $1/\mathrm{\Sigma }:={\mathrm{\Sigma }}^{-1}$ and $\frac{sin0}{0}=1$. Taking Taylor expansions of $sin$ and $cos$,

$$\begin{array}{cc}\hfill Vcos(\eta \mathrm{\Sigma })V^T& =V\left(I,-,\mathcal{O},(,{\eta }^2,)\right)V^T=I-\mathcal{O}({\eta }^2)\hfill \\ \hfill V\frac{sin(\eta \mathrm{\Sigma })}{\eta \mathrm{\Sigma }}{V}^T& =V\left(I,-,\mathcal{O},(,{\eta }^2,)\right)V^T=I-\mathcal{O}({\eta }^2),\hfill \end{array}$$

we obtain

$$\begin{array}{cc}\hfill V_{\alpha }^T{X}_{+}& =V_{\alpha }^{T}X(I-\mathcal{O}({\eta }^2))+V_{\alpha }^T(-\eta \text{grad}f(\mathcal{X}))(I-\mathcal{O}({\eta }^2))\hfill \\ \hfill & =V_{\alpha }^T(X-\eta \text{grad}f({\mathcal{X}}_t))(I-\mathcal{O}({\eta }^2))\hfill \end{array}$$ 29

since $\Vert V_{\alpha }{\Vert }_2={\Vert X\Vert }_2=1$.

Let now $\theta$ be the vector of k principal angles between ${\mathcal{X}}_t$ and ${\mathcal{V}}_{\alpha }$. As in (12) and (19), we therefore have the SVDs

$$\begin{array}{c}\hfill V_{\alpha }^{T}X=U_{1}cos\theta V_1^T\text{and}{V}_{\beta }^{T}X={\tilde{U}}_{2}sin\theta V_1^T,\end{array}$$ 30

where $U_1,V_1\in {\mathbb{R}}^{k\times k}$ and ${\tilde{U}}_2\in {\mathbb{R}}^{(n-k)\times k}$ have orthonormal columns. Next, we write (29) in terms of

$$\begin{array}{c}\hfill M:={sin}^2\theta U_1^T{\mathrm{\Lambda }}_{\alpha }{U}_{1}cos\theta -cos\theta sin\theta {\tilde{U}}_2^T{\mathrm{\Lambda }}_{\beta }{\tilde{U}}_{2}sin\theta .\end{array}$$

Since $\text{grad}f({\mathcal{X}}_t)=-2(I-X{X}^T)AX$, the identity (17) gives

$$\begin{array}{c}\hfill V_{\alpha }^T(X-\eta \text{grad}f({\mathcal{X}}_t))=V_{\alpha }^{T}X+2\eta {\mathrm{\Lambda }}_{\alpha }{V}_{\alpha }^{T}X-2\eta V_{\alpha }^{T}X{X}^{T}AX.\end{array}$$

After substituting (11) and (30), a short calculation using ${cos}^2\theta =I-{sin}^2\theta$ and the orthogonality of $U_1$ and $V_1$ then shows

$$\begin{array}{c}\hfill V_{\alpha }^T(X-\eta \text{grad}f({\mathcal{X}}_t))=U_1(cos\theta +2\eta M)V_1^T.\end{array}$$

Relating back to (29), we thus obtain

$$\begin{array}{cc}\hfill V_{\alpha }^T{X}_{+}& =U_1(cos\theta +2\eta M)V_1^T(I-\mathcal{O}({\eta }^2))\hfill \\ \hfill & =U_1(cos\theta +2\eta M)(I-V_1^T\mathcal{O}({\eta }^2)V_1)V_1^T\hfill \\ \hfill & =U_1(cos\theta +2\eta M-\mathcal{O}({\eta }^2))V_1^T.\hfill \end{array}$$

The singular values of $V_{\alpha }^T{X}_{+}$ are therefore the same as the singular values of the matrix $cos\theta +2\eta M+\mathcal{O}({\eta }^2)$.

By Weyl’s inequality (see, e.g., [18, Cor. 7.3.5]), each singular value of $cos\theta +2\eta M+\mathcal{O}({\eta }^2)$ is $\mathcal{O}({\eta }^2)$ close to some singular value of $cos\theta +2\eta M$. Let $1\le j\le k$. Denote the jth singular value of $cos\theta +2\eta M$ by ${\sigma }_j(\eta )$ to which we will apply Lemma 5.2. Let m be the multiplicity of ${\sigma }_j(0)$. Hence, there exists $j_0$ such that ${\sigma }_{j_0}(0)=\cdots ={\sigma }_j(0)=\cdots ={\sigma }_{j_0+m-1}(0)$. Since $cos\theta$ is a diagonal matrix with decreasing diagonal, its $\ell$th singular value equals $cos{\theta }_{\ell }$ and its associated left/right singular vector is the $\ell$th canonical vector $e_{\ell }$. Denoting

$$\begin{array}{c}\hfill E=\left[\begin{array}{ccc}{e}_{j_0}& \cdots & e_{j_0+m-1}\end{array}\right],\end{array}$$

observe that $cos\theta E=cos{\theta }_{j_0}E$ (here, $cos\theta$ is a diagonal matrix and $cos{\theta }_{j_0}$ is a scalar) and likewise for $sin\theta E$. We thus get

$$\begin{array}{c}\hfill E^{T}ME={sin}^2{\theta }_{j_0}cos{\theta }_{j_0}(U_1^T{\mathrm{\Lambda }}_{\alpha }{U}_1-{\tilde{U}}_2^T{\mathrm{\Lambda }}_{\beta }{\tilde{U}}_2).\end{array}$$

In the proof of Proposition 3, we showed that the matrix in brackets above is symmetric and positive semi-definite (see (20)). Since $0\le {\theta }_{j_0}\le \pi /2$, the eigenvalues of $E^{T}ME$ are therefore all non-negative. Lemma 5.2 thus gives that ${\sigma }_j(\eta )\ge {\sigma }_j$ for sufficiently small and positive $\eta$. Since the singular values of $V_{\alpha }^T{X}_{+}$ are the cosines of the principal angles between ${\mathcal{V}}_{\alpha }$ and ${\mathcal{X}}_{t+1}$ with step size $\eta \ge 0$, we conclude that there exists $\overline{\eta }>0$ such that for all $\eta \in [0,\overline{\eta }]$ it holds

$$\begin{array}{c}\hfill {\theta }_j({\mathcal{X}}_{t+1},{\mathcal{V}}_{\alpha })\le {\theta }_j({\mathcal{X}}_t,{\mathcal{V}}_{\alpha }).\end{array}$$

Since j was arbitrary, this finishes the proof. $\square$

Numerical Experiment

We report on a small numerical experiment to verify the convergence rates proven above. The steepest descent iteration with fixed step size was implemented in Matlab using the geodesic formula (4).

As first test matrix, we took the standard 3D Laplacian on a unit cube, discretized with finite differences and zero Dirichlet boundary conditions. The size of the matrix A is $n=400$. We tested a few values for the block size k. They are depicted in the table below, together with other parameters that are relevant for Theorem 5.1.

k	$\delta$	$\text{dist}({\mathcal{X}}_0,{\mathcal{V}}_{\alpha })$
1	$0.0665\dots$	$0.113\dots$
6	$0.0665\dots$	$0.280\dots$
10	$0.0262\dots$	$0.350\dots$

In Fig. 1, the convergence of the Riemannian distance is visible in addition to the theoretical convergence rate of Theorem 5.1. We see that in all cases, these bounds on the convergence are valid (in particular, exponential) although they are rather conservative.

Fig. 1.

Steepest descent along geodesics for the block Rayleigh quotient of size k applied to a discretized 3D Laplacian matrix. The full lines correspond to the experimental values and the dashed lines to the theoretical upper bounds

For completeness, we implemented steepest descent starting from a subspace ${\mathcal{X}}_0$ far away from the optimum. In that case, Theorem 5.1 does not apply since, if $\text{dist}({\mathcal{X}}_0,{\mathcal{V}}_{\alpha })>\frac{\pi }{2}$, the step-size $\eta \le cos(\text{dist}({\mathcal{X}}_0,{\mathcal{V}}_{\alpha }))/\gamma$ is or will become eventually negative. However, a meaningful choice for $\eta$ is given by Proposition 7 of Appendix B, where we prove a local linear convergence rate for the function values of the iterates for step size $\eta =1/\gamma$.

We see in Fig. 2 that despite the seemingly bad initial guess, steepest descent converges globally with a linear rate. This reveals that the restriction of the initial guess in our theoretical results is probably unnecessary and constitutes a topic for future work.

Fig. 2.

Same matrix from Fig. 1 but such that $\text{dist}({\mathcal{X}}_0,{\mathcal{V}}_{\alpha })\gg \pi /2$ and with fixed step size $1/\gamma$

In the second test, we investigate the convergence when the eigengap $\delta$ is small or zero. In particular, we take $A=VD{V}^T\in {\mathbb{R}}^{1000\times 1000}$ with V a random orthogonal matrix and D contains the eigenvalues

$$\begin{array}{c}\hfill {\lambda }_1=3,{\lambda }_2=2,{\lambda }_3=1+{10}^{-2}+{10}^{-6},{\lambda }_4=1+{10}^{-6},{\lambda }_5={\lambda }_6=1.\end{array}$$

The other eigenvalues are equidistantly distributed between 0.1 and 0.2. The block size and other relevant parameters for the test are described below. Since the convergence for small $\delta$ slows down considerably after the first 5 iterations, we apply the bounds of Theorem 5.2 at iteration $t=6$ (and treat this as the start with $t=0$).

k	$\delta$	$\text{dist}({\mathcal{X}}_0,{\mathcal{V}}_{\alpha })$	$\text{dist}({\mathcal{X}}_6,{\mathcal{V}}_{\alpha })$
2	$0.99\dots$	$0.051\dots$	$0.001\dots$
3	${10}^{-2}$	$0.055\dots$	$0.031\dots$
4	${10}^{-6}$	$0.063\dots$	$0.045\dots$
5	0	$0.070\dots$	$0.054\dots$

The convergence in function value is visible in Fig. 3. Observe that we have displayed a logarithmic scale for both axes whereas before the figure had a logarithmic scale only for y-axis. Algebraic convergence like 1/t is therefore visible as a straight line. We see in the figure that the convergence is not easily described, and that there is no clear difference between zero or small gap. However, the upper bounds of Theorem 5.2 are again valid. In addition, when the gap is not small, the convergence is clearly faster.

Fig. 3.

Steepest descent along geodesics for the block Rayleigh quotient of size k applied to a random matrix with small eigengaps. The full lines correspond to the experimental values and the dashed lines to the theoretical upper bounds of Theorem 5.2. Each color corresponds to a certain eigengap $\delta$

As before, we test the behaviour of steepest descent starting from an initial guess far away from the optimum. We use again step size $1/\gamma$; see Theorem B.2. In Fig. 4 we show the convergence of steepest descent for the problem defined by matrix A with this step size.

Fig. 4.

Same matrices with small eigengap from Fig. 3 but such that $\text{dist}({\mathcal{X}}_0,{\mathcal{V}}_{\alpha })\gg \pi /2$ and with fixed step size $1/\gamma$

We observe again that the local nature of our theoretical results is quite pessimistic: the algorithm converges with an algebraic rate even with a bad initial guess but it shows eventually linear convergence.

Conclusion and Future Work

We provided the first non-asymptotic convergence rates for Riemannian steepest descent on the Grassmann manifold for computing a subspace spanned by k leading eigenvectors of a symmetric matrix A.

Our main idea was to exploit a convexity-like structure of the block Rayleigh quotient, which can be of much more general interest than for only analyzing steepest descent. One example is line search methods, which have usually favourable properties compared to vanilla steepest descent. Also, weakly-quasi-convex functions have been proven to admit accelerated algorithms [25], while accelerated or almost accelerated Riemannian algorithms have been developed in [5, 6, 36]. It would naturally be interesting to examine whether a provable accelerated method can be developed for the block Rayleigh quotient on the Grassmann manifold. This would hopefully reduce the dependence of the iteration complexity on the eigengap $\delta$ from $\mathcal{O}(1/\delta )$ to $\mathcal{O}(1/\sqrt{\delta })$.

Another interesting direction is to extend the analysis of [4] from the computation of just one leading eigenvector to computation of a whole subspace, using the generalized machinery developed in this work, or develop a noisy version of steepest descent and compare with noisy power method [14].

Appendix A: Geodesic convexity

Let $\delta >0$ and thus ${\mathcal{V}}_{\alpha }$ is the unique minimizer of f. Define the following neighbourhood of ${\mathcal{V}}_{\alpha }$ in $\text{Gr}(n,k)$:

$$\begin{array}{c}\hfill N_{\ast }^{}(\phi )=\{\mathcal{X}\in \text{Gr}(n,k):{\theta }_k(\mathcal{X},{\mathcal{V}}_{\alpha })<\phi \}\text{with}\phi \in [0,\pi /4].\end{array}$$ 31

Here, ${\theta }_k(\mathcal{X},{\mathcal{V}}_{\alpha })$ denotes the largest principal angle between $\mathcal{X}$ and ${\mathcal{V}}_{\alpha }$. Since ${\theta }_k$ is a metric on $\text{Gr}(n,k)$ (see [30]), any two subspaces $\mathcal{X},\mathcal{Y}\in N_{\ast }^{}(\phi )$ will satisfy ${\theta }_k(\mathcal{X},\mathcal{Y})<\pi /2$ by triangle inequality. They thus have a unique connecting geodesic. It is shown in [?, Lemma 2] that for any fixed $\phi \in [0,\pi /4]$ this geodesic remains in $N_{\ast }^{}(\phi )$. Each set $N_{\ast }^{}(\phi )$ is thus an open totally geodesically convex set as defined in, e.g., [8, Def. 11.16].

One of the main results in [3], namely Cor. 4, states that f is geodesically convex on $N_{\ast }^{}(\pi /4)$. This is unfortunately wrong and we present a small counterexample.

Counterexample for Cor. 4 in [3].

Here we use the notation of [3]. The reader is encouraged to take a look there for notational purposes.

Take $c:=cos(\pi /4)=\sqrt{2}/2$ and $0\le \epsilon <1$. Define the matrices

$$X_p:=\left(\begin{array}{cc}1& 0\\ 0& 1\\ 0& 0\\ 0& 0\end{array}\right),U_p:=\left(\begin{array}{cc}c& 0\\ 0& c\\ c& 0\\ 0& c\end{array}\right),M:=U_p\left(\begin{array}{cc}1& 0\\ 0& \epsilon \end{array}\right).$$

These matrices satisfy the conditions posed in [3]:

• Principal alignment: $X_p^T{U}_p=\left(\begin{array}{cc}c& 0\\ 0& c\end{array}\right)$.

• Principal angles between $X_p$ and $U_p$ are in $[0,\pi /4]$.

• $U=U_p$ since $Q=I$.

Now consider the following tangent vector of unit Frobenius norm:

$$\mathrm{\Delta }=\left(\begin{array}{cc}0& 0\\ 0& 0\\ 0& 1\\ 0& 0\end{array}\right).$$

It is clearly a tangent vector of $[X_p]$ since $X_p^T\mathrm{\Delta }=0$. The Hessian of $f_{\mathit{full}}$ at $[X_p]$ in the direction of $\mathrm{\Delta }$ satisfies (see equation (4.2) in [3])

$$\text{Hess}{f}_{\mathit{full}}([X_p])[\mathrm{\Delta },\mathrm{\Delta }]=-2\text{Tr}(M^T\mathrm{\Delta }{\mathrm{\Delta }}^T(I-X_p{X}_p^T)M)+{\Vert (\mathrm{\Delta }{X}_p^T+X_p{\mathrm{\Delta }}^T)M\Vert }_F^2.$$

Simple calculation shows that

$$\text{Hess}{f}_{\mathit{full}}([X_p])[\mathrm{\Delta },\mathrm{\Delta }]=-2c^2+(1+{\epsilon }^2)c^2.$$

Hence for $\epsilon <1$, we have $\text{Hess}{f}_{\mathit{full}}([X_p])[\mathrm{\Delta },\mathrm{\Delta }]<0$ and the $f_{\mathit{full}}$ is non-convex which is in contrast with Corollary 4. $\square$

Instead, our Theorem A.1 guarantees convexity when $\phi$ depends on the spectral gap. Since f is smooth, the function is geodesically convex on $N_{\ast }^{}(\phi )$ if and only if its Riemannian Hessian is positive definite on $N_{\ast }^{}(\phi )$; see, e.g., [8, Thm. 11.23]. We will therefore compute the eigenvalues of $\text{Hess}f$ based on its matrix representation. This requires us to first vectorize the tangent space.

From (3), a matrix G is a tangent vector if and only if $G^{T}X=0$. Hence, taking $X_{\perp }\in {\mathbb{R}}^{n\times (n-k)}$ orthonormal such that ${\mathcal{X}}^{\perp }=\text{span}(X_{\perp })$, we have the equivalent definition

$$\begin{array}{c}\hfill T_X\text{Gr}(n,k)=\{X_{\perp }M:M\in {\mathbb{R}}^{(n-k)\times k}\}.\end{array}$$

The matrix M above can be seen as the coordinates of $G=X_{\perp }M$ in the basis $X_{\perp }$. More specifically, by using the linear isomorphism $\text{vec}:{\mathbb{R}}^{n\times k}\to {\mathbb{R}}^{\mathit{nk}}$ that stacks all columns of a matrix under each other, we can define the tangent vectors of $\text{Gr}(n,k)$ as standard (column) vectors in the following way:

$$\begin{array}{c}\hfill \text{vec}(G)=\text{vec}(X_{\perp }M)=(I_k\otimes X_{\perp })\text{vec}(M).\end{array}$$

Here, the Kronecker product $\otimes$ appears due to [17, Lemma 4.3.1]. By well-known properties of $\otimes$ (see, e.g., [17, Chap. 4.2]), the matrix $I_k\otimes X_{\perp }$ has orthonormal columns. We have thus obtained an orthonormal basis for the (vectorized) tangent space. With this setup, we can now construct the Hessian.

Lemma A.1

Let $I_k\otimes X_{\perp }$ be the orthonormal basis for the vectorization of $T_{\mathcal{X}}\text{Gr}(n,k)$. Then the Riemannian Hessian of f at $\mathcal{X}$ in that basis has the symmetric matrix representation

$$\begin{array}{c}\hfill H_X=2(X^{T}AX\otimes I_{n-k}-I_k\otimes X_{\perp }^{T}A{X}_{\perp }).\end{array}$$ 32

Furthermore, with $1\le i\le k$ and $1\le j\le n-k$ its $k(n-k)$ eigenvalues satisfy

$$\begin{array}{c}\hfill {\lambda }_{i,j}(H_X)=2({\lambda }_i(X^{T}AX)-{\lambda }_j(X_{\perp }^{T}A{X}_{\perp })).\end{array}$$

Proof

Since $\text{vec}$ is a linear isomorphism, the symmetric matrix $H_X$ satisfies

$$\begin{array}{c}\hfill \text{Hess}f(X)[X_{\perp }M,X_{\perp }M]=⟨\text{vec}(M),H_X\text{vec}(M)⟩,\forall M\in {\mathbb{R}}^{n\times (n-k)},\end{array}$$

where $⟨·,·⟩$ is the Euclidean inner product. Define $m=\text{vec}(M)$. Plugging in the formula (9) for $\text{Hess}f$, we calculate

$$\begin{array}{cc}\hfill \text{Hess}f(X)[X_{\perp }M,X_{\perp }M]& =2⟨X_{\perp }M,X_{\perp }M{X}^{T}AX-A{X}_{\perp }M⟩\hfill \\ \hfill & =2⟨(I\otimes X_{\perp })m,(X^{T}AX\otimes X_{\perp })m-(I\otimes A{X}_{\perp })m⟩\hfill \\ \hfill & =2⟨m,{(I\otimes X_{\perp })}^T(X^{T}AX\otimes X_{\perp }-I\otimes A{X}_{\perp })m⟩\hfill \\ \hfill & =2⟨m,(X^{T}AX\otimes I-I\otimes X_{\perp }^{T}A{X}_{\perp })m⟩\hfill \end{array}$$

Here, we used typical calculus rules for the Kronecker product (see, e.g., [17, Chap. 4.2]). We recognize the matrix $H_X$ directly.

The eigenvalues of (32) can be directly obtained using [17, Thm. 4.4.5]. $\square$

Taking $X=V_{\alpha }$ and $X_{\perp }=V_{\beta }$, Lemma A.1 shows immediately that the minimal eigenvalue of $\text{Hess}f({\mathcal{V}}_{\alpha })$ is equal to $2\delta =2({\lambda }_k-{\lambda }_{k+1})$. Since $\delta >0$, $\text{Hess}f$ will remain strictly positive definite in a neighbourhood of ${\mathcal{V}}_{\alpha }$ by continuity. To quantify this neighbourhood, we will connect ${\mathcal{V}}_{\alpha }$ to an arbitrary $\mathcal{X}$ using a geodesic and see how this influences the bounds of Lemma A.1. This also requires connecting ${\mathcal{V}}_{\beta }$ to ${\mathcal{X}}^{\perp }$. The next lemma shows that both geodesics are closely related. Recall that $sin(t\theta )$ and $cos(t\theta )$ denote diagonal matrices of size $k\times k$. For convenience, we will denote by O a zero matrix whose dimensions are clear from the context and is not always square.

Lemma A.2

Let $X,Y\in {\mathbb{R}}^{n\times k}$ be such that $X^{T}X=Y^{T}Y=I_k$ with $k\le n/2$. Denote the principal angles between $\text{span}(X)$ and $\text{span}(Y)$ by ${\theta }_1\le \cdots \le {\theta }_k$ and assume that ${\theta }_k<\pi /2$. Choose $X_{\perp },Y_{\perp }\in {\mathbb{R}}^{n\times (n-k)}$ such that $X_{\perp }^T{X}_{\perp }=Y_{\perp }^T{Y}_{\perp }=I_{n-k}$ and $\text{span}(X_{\perp })=\text{span}{(X)}^{\perp }$, $\text{span}(Y_{\perp })=\text{span}{(Y)}^{\perp }$. Define the curves

$$\begin{array}{cccc}\hfill \gamma (t)& :[0,1]\to {\mathbb{R}}^{n\times k},\hfill & \hfill t& \mapsto X{V}_{1}cos(t\theta )+X_{\perp }{V}_2\left[\begin{array}{c}O\\ sin(t\theta )\end{array}\right],\hfill \\ \hfill {\gamma }_{\perp }(t)& :[0,1]\to {\mathbb{R}}^{n\times (n-k)},\hfill & \hfill t& \mapsto X_{\perp }{V}_2\left[\begin{array}{c}I\\ & cos(t\theta )\end{array}\right]-X{V}_1\left[\begin{array}{cc}O& sin(t\theta )\end{array}\right],\hfill \end{array}$$

where the orthogonal matrices $V_1,V_2$ are the same as in Lemma 4.3. Then $\text{span}(\gamma (t))$ is the connecting geodesic on $\text{Gr}(n,k)$ from $\text{span}(X)$ to $\text{span}(Y)$. Likewise, $\text{span}({\gamma }_{\perp }(t))$ is a connecting geodesic on $\text{Gr}(n,n-k)$ from $\text{span}(X_{\perp })$ to $\text{span}(Y_{\perp })$. Furthermore, $\gamma (t)$ and ${\gamma }_{\perp }(t)$ are orthonormal matrices for all t.

Proof

Assume ${\theta }_1=\cdots ={\theta }_r=0$, where $r=0$ means that ${\theta }_1>0$. Like in the proof of Prop. 3, the CS decomposition of X and Y from Lemma 4.3 can be written in terms of their principal angles ${\theta }_1,\dots ,{\theta }_k$. Since ${\theta }_k<\pi /2$ and $n\le k/2$, this gives after dividing certain block matrices the relations

$$\begin{array}{cccc}\hfill Y^{T}X& =U_{1}cos(\theta )V_1^T,\hfill & \hfill Y^T{X}_{\perp }& =U_1\left[\begin{array}{cc}{O}_{k\times (n-2k)}& sin(\theta )\end{array}\right]V_2^T\hfill \\ \hfill Y_{\perp }^{T}X& =U_2\left[\begin{array}{c}{O}_{(n-2k)\times k}\\ sin(\theta )\end{array}\right]V_1^T,\hfill & \hfill Y_{\perp }^T{X}_{\perp }& =U_2\left[\begin{array}{c}-I_{n-2k}\\ & -cos(\theta )\end{array}\right]V_2^T,\hfill \end{array}$$

where $U_1,V_1$ and $U_2,V_2$ are orthogonal matrices of size $k\times k$ and $(n-k)\times (n-k)$, resp.

Denote $\mathcal{X}=\text{span}(X)$ and $\mathcal{Y}=\text{span}(Y)$. By definition, the connecting geodesic $\gamma (t)$ is determined by the tangent vector ${\text{Log}}_{\mathcal{X}}(\mathcal{Y})$, which can be computed from (6). To this end, we first need the compact SVD of $M:=X_{\perp }{X}_{\perp }^{T}Y{(X^{T}Y)}^{-1}$. Substituting the results from above, we get (cfr. (15))

$$\begin{array}{c}\hfill M=X_{\perp }{V}_2\left[\begin{array}{c}{O}_{(n-2k)\times k}\\ sin(\theta )\end{array}\right]U_1^T{U}_1{(cos(\theta ))}^{-1}{V}_1^T=X_{\perp }{V}_2\left[\begin{array}{c}{O}_{(n-2k)\times k}\\ I_k\end{array}\right]tan(\theta )V_1^T.\end{array}$$

Observe that this is a compact SVD. Applying (6), we therefore get

$$\begin{array}{c}\hfill G:={\text{Log}}_{\mathcal{X}}(\mathcal{Y})=U\mathrm{\Sigma }{V}^T\text{with}U=X_{\perp }{V}_2\left[\begin{array}{c}O\\ I_k\end{array}\right],\mathrm{\Sigma }=\theta ,V=V_1\end{array}$$

and from (4), the connecting geodesic satisfies

$$\begin{array}{c}\hfill {\text{Exp}}_{\mathcal{X}}(tG)=\text{span}(X{V}_{1}cos(t\theta )+X_{\perp }{V}_2\left[\begin{array}{c}O\\ I_k\end{array}\right]sin(t\theta )).\end{array}$$

We have proven the stated formula for $\gamma (t)$. Verifying that $\gamma {(t)}^T\gamma (t)=I_k$ follows from a simple calculation that uses ${cos}^2(t\theta )+{sin}^2(t\theta )=I_k$.

Denote ${\mathcal{X}}^{\perp }=\text{span}(X_{\perp })$ and ${\mathcal{Y}}^{\perp }=\text{span}(Y_{\perp })$. To prove ${\gamma }_{\perp }(t)$, we proceed similarly by computing $G^{\perp }:={\text{Log}}_{{\mathcal{X}}^{\perp }}({\mathcal{Y}}^{\perp })$, which requires now the SVD of $M^{\perp }:=X{X}^T{Y}_{\perp }{(X_{\perp }^T{Y}_{\perp })}^{-1}$. Again substituting the results from the CS decomposition, we get

$$\begin{array}{cc}\hfill M^{\perp }& =X{V}_1\left[\begin{array}{cc}{O}_{k\times (n-2k)}& sin(\theta )\end{array}\right]U_2^T{U}_2{\left[\begin{array}{c}-I_{n-2k}\\ & -cos(\theta )\end{array}\right]}^{-1}{V}_2^T\hfill \\ \hfill & =X{V}_1\left[\begin{array}{cc}{O}_{k\times (n-2k)}& -tan(\theta )\end{array}\right]V_2^T\hfill \end{array}$$

Since (6) requires a compact SVD with a square $\mathrm{\Sigma }$, we rewrite this as

$$\begin{array}{c}\hfill M^{\perp }=\left[\begin{array}{cc}\tilde{X}& X{V}_1\end{array}\right]\left[\begin{array}{c}{O}_{(n-2k)\times (n-2k)}\\ & -tan(\theta )\end{array}\right]V_2^T\end{array}$$

where $\tilde{X}$ contains $n-2k$ columns that are orthonormal to X (the final result will not depend on $\tilde{X}$). Let ${\theta }_1^{\perp }\le \cdots \le {\theta }_{n-k}^{\perp }$ denote the principal angles between ${\mathcal{X}}^{\perp }$ and ${\mathcal{Y}}^{\perp }$. Up to zero angles, they are the same as those between $\mathcal{X}$ and $\mathcal{Y}$. Since $k\le n/2$, we thus have

$$\begin{array}{c}\hfill {\theta }_1^{\perp }=\cdots ={\theta }_{n-2k}^{\perp }=0,{\theta }_{n-2k+1}^{\perp }={\theta }_1,\dots ,{\theta }_{n-k}^{\perp }={\theta }_k.\end{array}$$

Applying (6) with these principal angles, we obtain

$$\begin{array}{c}\hfill G^{\perp }:={\text{Log}}_{{\mathcal{X}}^{\perp }}({\mathcal{Y}}^{\perp })=U\mathrm{\Sigma }{V}^T\text{with}U=-\left[\begin{array}{cc}\tilde{X}& X{V}_1\end{array}\right],\mathrm{\Sigma }={\theta }^{\perp },V=V_2.\end{array}$$

From (4), the corresponding geodesic satisfies

$$\begin{array}{cc}\hfill {\text{Exp}}_{{\mathcal{X}}^{\perp }}(t{G}^{\perp })& =\text{span}(X_{\perp }{V}_{2}cos(t{\theta }^{\perp })-\left[\begin{array}{cc}\tilde{X}& X{V}_1\end{array}\right]sin(t{\theta }^{\perp }))\hfill \\ \hfill & =\text{span}(X_{\perp }{V}_2\left[\begin{array}{c}{I}_{n-2k}\\ & cos(t\theta )\end{array}\right]-\left[\begin{array}{cc}{O}_{n\times (n-2k)}& X{V}_{1}sin(t\theta )\end{array}\right]).\hfill \end{array}$$

Rewriting the block matrix, we have proven ${\gamma }_{\perp }(t)$. Its orthonormality is again a straightforward verification. $\square$

With the previous lemma, we can now investigate the Riemannian Hessian of f near ${\mathcal{V}}_{\alpha }$ when it is given in the matrix form $H_X$ of Lemma A.1. Let $\mathcal{X}=\text{span}(X)\in \text{Gr}(n,k)$ with orthonormal X. Its principal angles with ${\mathcal{V}}_{\alpha }$ are ${\theta }_1\le \cdots \le {\theta }_k<\pi /2$. Use the substitutions $X\mapsto V_{\alpha },Y\mapsto X$ and $X_{\perp }\mapsto V_{\beta },Y_{\perp }\mapsto X_{\perp }$ in Lemma A.2 to define the geodesics $\gamma (t)$ and ${\gamma }_{\perp }(t)$ that connect ${\mathcal{V}}_{\alpha }$ to $\mathcal{X}$, and ${\mathcal{V}}_{\beta }$ to ${\mathcal{X}}^{\perp }$, resp. Denoting

$$\begin{array}{c}\hfill C:=cos(\theta ),S:=sin(\theta ),\tilde{C}:=\left[\begin{array}{c}I\\ & C\end{array}\right],\tilde{S}:=\left[\begin{array}{c}O\\ S\end{array}\right],\end{array}$$

we get the following expressions for the geodesics:

$$\begin{array}{c}\hfill \gamma (t)=V_{\alpha }{V}_{1}C+V_{\beta }{V}_2\tilde{S},{\gamma }_{\perp }(t)=V_{\beta }{V}_2\tilde{C}-V_{\alpha }{V}_1{\tilde{S}}^T.\end{array}$$

Recall that $H_X$ is defined using $X^{T}AX$ and $X_{\perp }^{T}A{X}_{\perp }$. Since $\gamma (1)=X{Q}_1$ and ${\gamma }_{\perp }(1)=X^{\perp }{Q}_2$ for some orthogonal matrices $Q_1,Q_2$, we can write with $A=V_{\alpha }{\mathrm{\Lambda }}_{\alpha }{V}_{\alpha }^T+V_{\beta }{\mathrm{\Lambda }}_{\beta }{V}_{\beta }^T$ that

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill Q_1^T{X}^{T}AX{Q}_1& =\gamma {(1)}^{T}A\gamma (1)\hfill \\ \hfill & =C(V_1^T{\mathrm{\Lambda }}_{\alpha }{V}_1)C+{\tilde{S}}^T(V_2^T{\mathrm{\Lambda }}_{\beta }{V}_2)\tilde{S}\hfill \\ \hfill Q_2^T{X}_{\perp }^{T}A{X}_{\perp }{Q}_2& ={\gamma }_{\perp }{(1)}^{T}A{\gamma }_{\perp }(1)\hfill \\ \hfill & =\tilde{C}(V_2^T{\mathrm{\Lambda }}_{\beta }{V}_2)\tilde{C}+\tilde{S}(V_1^T{\mathrm{\Lambda }}_{\alpha }{V}_1){\tilde{S}}^T.\hfill \end{array}\end{array}$$ 33

Here we used simplifications like $V_{\beta }^{T}A{V}_{\alpha }=V_{\beta }^T{V}_{\alpha }{\mathrm{\Lambda }}_{\alpha }=0$.

A simple bounding of the eigenvalues of the difference of these matrices results in the main result.

Theorem A.1

Let $k\le n/2$. Define the neighbourhood

$$\begin{array}{c}\hfill B_{\ast }^{}=\left\{\mathcal{X},\in ,\text{Gr},(n,k),:,{sin}^2,({\theta }_k(\mathcal{X},{\mathcal{V}}_{\alpha })),\le ,\frac{\delta }{{\lambda }_1+{\lambda }_k}\right\},\end{array}$$

then f is geodesically convex on $B_{\ast }^{}$.

Proof

Our aim is to show that ${\lambda }_{i,j}(H_X)$ remains positive given the bound on ${\theta }_k$. From Lemma A.1, we see that

$$\begin{array}{c}\hfill {\lambda }_{min}(H_X)\ge 0⟺{\lambda }_{min}(X^{T}AX)\ge {\lambda }_{max}(X_{\perp }^{T}A{X}_{\perp }).\end{array}$$ 34

Since $Q_1,Q_2$ are orthogonal in (33), it suffices to find a lower and upper bound of, resp.,

$$\begin{array}{cc}\hfill {\lambda }_{min}(X^{T}AX)& ={\lambda }_{min}(C(V_1^T{\mathrm{\Lambda }}_{\alpha }{V}_1)C+{\tilde{S}}^T(V_2^T{\mathrm{\Lambda }}_{\beta }{V}_2)\tilde{S})\hfill \\ \hfill {\lambda }_{max}(X_{\perp }^{T}A{X}_{\perp })& ={\lambda }_{max}(\tilde{C}(V_2^T{\mathrm{\Lambda }}_{\beta }{V}_2)\tilde{C}+\tilde{S}(V_1^T{\mathrm{\Lambda }}_{\alpha }{V}_1){\tilde{S}}^T).\hfill \end{array}$$

Standard eigenvalue inequalities for symmetric matrices (see, e.g., [18, Cor. 4.3.15]) give

$$\begin{array}{cc}\hfill {\lambda }_{min}(X^{T}AX)& \ge {\lambda }_{min}(C(V_1^T{\mathrm{\Lambda }}_{\alpha }{V}_1)C)+{\lambda }_{min}({\tilde{S}}^T(V_2^T{\mathrm{\Lambda }}_{\beta }{V}_2)\tilde{S})\hfill \\ \hfill {\lambda }_{max}(X_{\perp }^{T}A{X}_{\perp })& \le {\lambda }_{max}(\tilde{C}(V_2^T{\mathrm{\Lambda }}_{\beta }{V}_2)\tilde{C})+{\lambda }_{max}(\tilde{S}(V_1^T{\mathrm{\Lambda }}_{\alpha }{V}_1){\tilde{S}}^T).\hfill \end{array}$$

Recall that ${\lambda }_1\ge \cdots \ge {\lambda }_n$ are the eigenvalues of A. Since $\tilde{S}$ is a tall rectangular matrix, we apply the generalized version of Ostrowski’s theorem from [16, Thm. 3.2] to each term above6 and obtain

$$\begin{array}{cc}\hfill {\lambda }_{min}(C(V_1^T{\mathrm{\Lambda }}_{\alpha }{V}_1)C)& \ge {\lambda }_{min}(C^2){\lambda }_{min}({\mathrm{\Lambda }}_{\alpha })={cos}^2({\theta }_k){\lambda }_k\hfill \\ \hfill {\lambda }_{min}({\tilde{S}}^T(V_2^T{\mathrm{\Lambda }}_{\beta }{V}_2)\tilde{S})& \ge {\lambda }_{min}({\tilde{S}}^T\tilde{S}){\lambda }_{min}({\mathrm{\Lambda }}_{\beta })={sin}^2({\theta }_1){\lambda }_n,\hfill \end{array}$$

since the matrices $V_1,V_2$ are orthogonal and ${\theta }_1\le \cdots \le {\theta }_k<\pi /2$. Adding this gives the lower bound

$$\begin{array}{c}\hfill {\lambda }_{min}(X^{T}AX)\ge {cos}^2({\theta }_k){\lambda }_k+{sin}^2({\theta }_1){\lambda }_n\ge {cos}^2({\theta }_k){\lambda }_k.\end{array}$$ 35

Likewise, using the block structure of $\tilde{S}$, we get

$$\begin{array}{cc}\hfill {\lambda }_{max}(\tilde{C}(V_2^T{\mathrm{\Lambda }}_{\beta }{V}_2)\tilde{C})& \le {\lambda }_{max}(C^2){\lambda }_{max}({\mathrm{\Lambda }}_{\beta })={cos}^2({\theta }_1){\lambda }_{k+1}\hfill \\ \hfill {\lambda }_{max}(\tilde{S}(V_1^T{\mathrm{\Lambda }}_{\alpha }{V}_1){\tilde{S}}^T)& ={\lambda }_{max}(S(V_1^T{\mathrm{\Lambda }}_{\alpha }{V}_1)S)\hfill \\ \hfill & \le {\lambda }_{max}(S^2){\lambda }_{max}({\mathrm{\Lambda }}_{\alpha })={sin}^2({\theta }_k){\lambda }_1\hfill \end{array}$$

and thus

$$\begin{array}{c}\hfill {\lambda }_{max}(X_{\perp }^{T}A{X}_{\perp })\le {cos}^2({\theta }_1){\lambda }_{k+1}+{sin}^2({\theta }_k){\lambda }_1\le {\lambda }_{k+1}+{sin}^2({\theta }_k){\lambda }_1.\end{array}$$ 36

The condition (34) is thus satisfied when

$$\begin{array}{c}\hfill {cos}^2({\theta }_k){\lambda }_k={\lambda }_k-{sin}^2({\theta }_k){\lambda }_k\ge {\lambda }_{k+1}+{sin}^2({\theta }_k){\lambda }_1,\end{array}$$

which reduces to the bound on ${\theta }_k$ in the statement of the theorem.

It remains to show that $B_{\ast }^{}$ is an open totally geodesically convex set. Since ${\lambda }_1\ge {\lambda }_k\ge {\lambda }_{k+1}\ge 0$, we get

$$\begin{array}{c}\hfill \frac{{\lambda }_k-{\lambda }_{k+1}}{{\lambda }_1+{\lambda }_k}\le \frac{{\lambda }_k}{2{\lambda }_k}=\frac{1}{2}.\end{array}$$

Hence, $B_{\ast }^{}=N_{\ast }^{}(\phi )$ with $\phi \le \pi /4$ since ${sin}^2(\pi /4)=1/2$. $\square$

If $k=1$, the proof above can be simplified.

Corollary A.1

Let $k=1$ and define the neighbourhood

$$\begin{array}{c}\hfill B_{\ast }^{}=\left\{\mathcal{X},\in ,\text{Gr},(n,1),:,{sin}^2,({\theta }_1(\mathcal{X},{\mathcal{V}}_{\alpha })),\le ,\frac{\delta }{\delta +{\lambda }_1-{\lambda }_n}\right\}.\end{array}$$

Then f is geodesically convex on $B_{\ast }^{}$.

Proof

Since $k=1$, there is no need to simplify the bounds (35) and (36) as was done above. This gives that f is convex as long as

$$\begin{array}{c}\hfill {cos}^2({\theta }_1){\lambda }_1+{sin}^2({\theta }_1){\lambda }_n\ge {cos}^2({\theta }_1){\lambda }_2+{sin}^2({\theta }_1){\lambda }_1.\end{array}$$

Rewriting leads directly to the stated condition on ${sin}^2({\theta }_1)$. $\square$

Remark that optimizing f on $\text{Gr}(n,1)$ is equivalent to

$$\begin{array}{c}\hfill \underset{x\in {\mathbb{R}}^n}{min}-x^{T}Ax\text{s.t.}\Vert x\Vert =1,\end{array}$$ 37

which is the minimization of the Rayleigh quotient problem on the unit sphere $S^{n-1}=\{x\in {\mathbb{R}}^n:x^{T}x=1\}$. Cor. A.1 can therefore also be phrased in terms of a geodesically convex region for this problem. Denoting a unit norm top eigenvector of A by $v_1$ and using that ${sin}^2{\theta }_1=1-{cos}^2{\theta }_1$, we get that (37) is geodesically convex on

$$\begin{array}{c}\hfill \widehat{B}_{\ast }^{}=\left\{x,\in ,S^{n-1},:,{(x^T{v}_1)}^2,\ge ,1,-,\frac{\delta }{\delta +{\lambda }_1-{\lambda }_n}\right\}.\end{array}$$

This result can now be directly compared to [19, Lemma 7] where the corresponding region is defined as ${(x^T{v}_1)}^2\ge 1-\frac{\delta }{\delta +{\lambda }_1}$. This is a stricter condition and our result is therefore a small improvement.

Appendix B: Convergence of Steepest Descent with step $\frac{1}{\gamma }$

We now prove convergence of steepest descent with a more tractable choice of step-size compared to the analysis of the main paper. However, this requires a slightly better initialization at most $\frac{\pi }{2\sqrt{2}}$ away from the minimizer.

Maximum extent of the Iterates

We first prove that steepest descent with step-size at most $\frac{1}{\gamma }$ does not guarantee contraction on distances from step to step, but we can still bound the distance at step t with the initial distance up to a scalar:

Proposition 6

Consider steepest descent applied to f with step-size $\eta \le \frac{1}{\gamma }$. If the iterates ${\mathcal{X}}_t$ satisfy ${\theta }_k({\mathcal{X}}_t,{\mathcal{V}}_{\alpha })<\frac{\pi }{2}$, then they also satisfy

$$\begin{array}{c}\hfill {\text{dist}}^2({\mathcal{X}}_t,{\mathcal{V}}_{\alpha })\le 2{\text{dist}}^2({\mathcal{X}}_0,{\mathcal{V}}_{\alpha }).\end{array}$$

Proof

Consider the discrete Lyapunov function

$$\begin{array}{c}\hfill \mathcal{E}(t)=\frac{1}{\gamma }(f({\mathcal{X}}_t)-f^{\ast })+\frac{1}{2}{\text{dist}}^2({\mathcal{X}}_t,{\mathcal{V}}_{\alpha }).\end{array}$$

Then

$$\begin{array}{c}\hfill \mathcal{E}(t+1)-\mathcal{E}(t)=\frac{1}{\gamma }(f({\mathcal{X}}_{t+1})-f({\mathcal{X}}_t))+\frac{1}{2}({\text{dist}}^2({\mathcal{X}}_{t+1},{\mathcal{V}}_{\alpha })-{\text{dist}}^2({\mathcal{X}}_t,{\mathcal{V}}_{\alpha })).\end{array}$$

By $\gamma$-smoothness of f, we have

$$\begin{array}{cc}& f({\mathcal{X}}_{t+1})-f({\mathcal{X}}_t)\le ⟨\text{grad}f({\mathcal{X}}_t),{\text{Log}}_{{\mathcal{X}}_t}({\mathcal{X}}_{t+1})⟩+\frac{\gamma }{2}\text{dist}{({\mathcal{X}}_t,{\mathcal{X}}_{t+1})}^2\hfill \\ \hfill & =\left(-,\eta ,+,\frac{\gamma }{2},{\eta }^2\right){\Vert \text{grad}f({\mathcal{X}}_t)\Vert }^2.\hfill \end{array}$$

We also know by Proposition 3 that

$$\begin{array}{c}\hfill ⟨\text{grad}f(\mathcal{X}),-{\text{Log}}_{\mathcal{X}}({\mathcal{V}}_{\alpha })⟩\ge 0,\end{array}$$

for any $\mathcal{X}$ with ${\theta }_k(\mathcal{X},{\mathcal{V}}_{\alpha })<\pi /2$. By the fact that the sectional curvatures of the Grassmann manifold are non-negative, we have

$$\begin{array}{cc}\hfill {\text{dist}}^2({\mathcal{X}}_{t+1},{\mathcal{V}}_{\alpha })& \le {\text{dist}}^2({\mathcal{X}}_t,{\mathcal{V}}_{\alpha })+{\text{dist}}^2({\mathcal{X}}_{t+1},{\mathcal{X}}_t)-2⟨{\text{Log}}_{{\mathcal{X}}_t}({\mathcal{X}}_{t+1}),{\text{Log}}_{{\mathcal{X}}_t}({\mathcal{V}}_{\alpha })⟩\hfill \\ \hfill & ={\text{dist}}^2({\mathcal{X}}_t,{\mathcal{V}}_{\alpha })+{\eta }^2\Vert \text{grad}f({\mathcal{X}}_t){\Vert }^2+2\eta ⟨\text{grad}f({\mathcal{X}}_t),{\text{Log}}_{{\mathcal{X}}_t}({\mathcal{V}}_{\alpha })⟩\hfill \\ \hfill & \le {\text{dist}}^2({\mathcal{X}}_t,{\mathcal{V}}_{\alpha })+{\eta }^2{\Vert \text{grad}f({\mathcal{X}}_t)\Vert }^2.\hfill \end{array}$$

Thus

$$\begin{array}{cc}& \mathcal{E}(t+1)-\mathcal{E}(t)\le \left(-,\frac{\eta }{\gamma },+,\frac{{\eta }^2}{2}\right)\Vert \text{grad}f({\mathcal{X}}_t){\Vert }^2+\frac{{\eta }^2}{2}{\Vert \text{grad}f({\mathcal{X}}_t)\Vert }^2\hfill \\ \hfill & \le \left(-,\frac{\eta }{\gamma },+,{\eta }^2\right){\Vert \text{grad}f({\mathcal{X}}_t)\Vert }^2\le 0,\hfill \end{array}$$

because $\eta \le \frac{1}{\gamma }$. Since $\mathcal{E}(t)$ does not increase, we have

$$\begin{array}{cc}\hfill \frac{1}{2}{\text{dist}}^2({\mathcal{X}}_t,{\mathcal{V}}_{\alpha })& \le \mathcal{E}(t)\le \mathcal{E}(0)=\frac{1}{\gamma }(f({\mathcal{X}}_0)-f^{\ast })+\frac{1}{2}{\text{dist}}^2({\mathcal{X}}_0,{\mathcal{V}}_{\alpha })\hfill \\ \hfill & \le \frac{1}{2}{\text{dist}}^2({\mathcal{X}}_0,{\mathcal{V}}_{\alpha })+\frac{1}{2}{\text{dist}}^2({\mathcal{X}}_0,{\mathcal{V}}_{\alpha })={\text{dist}}^2({\mathcal{X}}_0,{\mathcal{V}}_{\alpha })\hfill \end{array}$$

and the desired result follows. $\square$

Convergence Under Positive Eigengap

When $\delta >0$, we can use gradient dominance to prove convergence of steepest descent to the (unique) minimizer in terms of function values:

Proposition 7

Steepest descent with step-size $\eta =1/\gamma$ initialized at ${\mathcal{X}}_0$ such that

$$\begin{array}{c}\hfill \text{dist}({\mathcal{X}}_0,{\mathcal{V}}_{\alpha })\le \frac{\pi }{4}\end{array}$$

satisfies

$$\begin{array}{c}\hfill f({\mathcal{X}}_t)-f^{\ast }\le {\left(1,-,0.32,c_Q,\frac{\delta }{\gamma }\right)}^t(f({\mathcal{X}}_0)-f^{\ast }).\end{array}$$

Proof

By the previous result and an induction argument to guarantee that the biggest angle between ${\mathcal{X}}_t$ and ${\mathcal{V}}_{\alpha }$ stays strictly less than $\pi /2$, we can bound the quantities $a({\mathcal{X}}_t)$ uniformly from below:

Since $\text{dist}({\mathcal{X}}_t,{\mathcal{V}}_{\alpha })\le \sqrt{2}·\text{dist}({\mathcal{X}}_0,{\mathcal{V}}_{\alpha })\le \frac{\sqrt{2}\pi }{4}$, we have

$$\begin{array}{c}\hfill a({\mathcal{X}}_t)\ge cos({\theta }_k({\mathcal{X}}_t,{\mathcal{V}}_{\alpha }))\ge cos(\text{dist}({\mathcal{X}}_t,{\mathcal{V}}_{\alpha }))\ge cos\left(\frac{\sqrt{2}\pi }{4}\right)\ge 0.4.\end{array}$$

By $\gamma$-smoothness of f, we have

$$\begin{array}{c}\hfill f({\mathcal{X}}_{t+1})-f({\mathcal{X}}_t)\le -\frac{\Vert \text{grad}f({\mathcal{X}}_t){\Vert }^2}{2\gamma }\end{array}$$

and applying gradient dominance (Proposition 4), we get the bound

$$\begin{array}{c}\hfill f({\mathcal{X}}_{t+1})-f({\mathcal{X}}_t)\le -\frac{2c_Q\delta a^2({\mathcal{X}}_t)}{\gamma }(f({\mathcal{X}}_t)-f^{\ast })\end{array}$$

thus

$$\begin{array}{c}\hfill f({\mathcal{X}}_{t+1})-f^{\ast }\le \left(1,-,2,c_Q,a^2,({\mathcal{X}}_t),\frac{\delta }{\gamma }\right)(f({\mathcal{X}}_t)-f^{\ast })\le \left(1,-,0.32,c_Q,\frac{\delta }{\gamma }\right)(f({\mathcal{X}}_t)-f^{\ast }).\end{array}$$

By induction the desired result follows. $\square$

We now state the iteration complexity of steepest descent algorithm:

Theorem B.1

Steepest descent with step-size $\frac{1}{\gamma }$ starting from a subspace ${\mathcal{X}}_0$ with distance at most $\frac{\pi }{4}$ from ${\mathcal{V}}_{\alpha }$ computes an estimate ${\mathcal{X}}_T$ of ${\mathcal{V}}_{\alpha }$ such that $\text{dist}({\mathcal{X}}_T,{\mathcal{V}}_{\alpha })\le ϵ$ in at most

$$\begin{array}{c}\hfill T=\frac{1}{0.32c_Q}\frac{\gamma }{\delta }log\frac{f({\mathcal{X}}_0)-f^{\ast }}{c_Q\delta {ϵ}^2}+1\le \mathcal{O}\left(\frac{\gamma }{\delta },log,\frac{f({\mathcal{X}}_0)-f^{\ast }}{\delta ϵ}\right).\end{array}$$

Proof

For $\text{dist}({\mathcal{X}}_T,{\mathcal{V}}_{\alpha })\le ϵ$, it suffices to have

$$\begin{array}{c}\hfill f({\mathcal{X}}_T)-f^{\ast }\le c_Q{ϵ}^2\delta \end{array}$$

by quadratic growth of f in Proposition 2. Using ${(1-c)}^T\le exp(-cT)$ for all $T\ge 0$ and $0\le c\le 1$, the previous result gives that it suffices to choose T as the smallest integer such that

$$\begin{array}{c}\hfill f({\mathcal{X}}_T)-f^{\ast }\le exp\left(-,0.32,c_Q,\frac{\delta }{\gamma },T\right)(f({\mathcal{X}}_0)-f^{\ast })\le c_Q{ϵ}^2\delta .\end{array}$$

Solving for T and substituting $c_Q=4/{\pi }^2$, we get the required statement. $\square$

Gap-Less Result

We also prove a convergence result for the function values when $\delta$ is assumed to be 0:

Theorem B.2

Steepest descent with step-size $\eta =\frac{1}{\gamma }$ initialized at ${\mathcal{X}}_0$ such that

$$\begin{array}{c}\hfill \text{dist}({\mathcal{X}}_0,{\mathcal{V}}_{\alpha })\le \frac{\pi }{4}\end{array}$$

satisfies

$$\begin{array}{c}\hfill f({\mathcal{X}}_t)-f^{\ast }\le \frac{f({\mathcal{X}}_0)-f^{\ast }+\frac{\gamma }{2}{\text{dist}}^2({\mathcal{X}}_0,{\mathcal{V}}_{\alpha })}{0.4t+1}=\mathcal{O}\left(\frac{1}{t}\right).\end{array}$$

Proof

By Proposition 6, we have that $\text{dist}(X_t,{\mathcal{V}}_{\alpha })\le \frac{\sqrt{2}\pi }{4}$ and f satisfies the weak-quasi-convexity inequality at any iterate ${\mathcal{X}}_t$ of steepest descent with constant $C_0:=0.4$.

Consider the discrete Lyapunov function

$$\begin{array}{c}\hfill \mathcal{E}(t)=\frac{C_{0}t+1}{\gamma }(f({\mathcal{X}}_t)-f^{\ast })+\frac{1}{2}{\text{dist}}^2({\mathcal{X}}_t,{\mathcal{V}}_{\alpha }).\end{array}$$

We have that

$$\begin{array}{cc}\hfill \mathcal{E}(t+1)-\mathcal{E}(t)=& \frac{C_{0}t+C_0+1}{\gamma }(f({\mathcal{X}}_{t+1})-f^{\ast })-\frac{C_{0}t+1}{\gamma }(f({\mathcal{X}}_t)-f^{\ast })\hfill \\ \hfill & +\frac{1}{2}({\text{dist}}^2({\mathcal{X}}_{t+1},{\mathcal{V}}_{\alpha })-{\text{dist}}^2({\mathcal{X}}_t,{\mathcal{V}}_{\alpha })).\hfill \end{array}$$

Now we have to estimate a bound for ${\text{dist}}^2({\mathcal{X}}_{t+1},{\mathcal{V}}_{\alpha })-{\text{dist}}^2({\mathcal{X}}_t,{\mathcal{V}}_{\alpha })$. By $\gamma$-smoothness of f and denoting ${\mathrm{\Delta }}_t=f({\mathcal{X}}_t)-f^{\ast }$ we have

$$\begin{array}{c}\hfill {\mathrm{\Delta }}_{t+1}-{\mathrm{\Delta }}_t\le ⟨\text{grad}f({\mathcal{X}}_t),{\text{Log}}_{{\mathcal{X}}_t}({\mathcal{X}}_{t+1})⟩+\frac{\gamma }{2}{\text{dist}}^2({\mathcal{X}}_t,{\mathcal{X}}_{t+1})=-\frac{\Vert \text{grad}f({\mathcal{X}}_t){\Vert }^2}{2\gamma }\end{array}$$

By $C_0$-weak–strong-convexity of f and the fact that the Grassmann manifold is of positive curvature, we have

$$\begin{array}{c}\hfill C_0{\mathrm{\Delta }}_t\le \frac{\gamma }{2}({\text{dist}}^2({\mathcal{X}}_t,{\mathcal{V}}_{\alpha })-{\text{dist}}^2({\mathcal{X}}_{t+1},{\mathcal{V}}_{\alpha }))+\frac{\Vert \text{grad}f({\mathcal{X}}_t){\Vert }^2}{2\gamma }\end{array}$$

Summing this to the previous inequality, we get

$$\begin{array}{c}\hfill {\text{dist}}^2({\mathcal{X}}_{t+1},{\mathcal{V}}_{\alpha })-{\text{dist}}^2({\mathcal{X}}_t,{\mathcal{V}}_{\alpha })\le \frac{2}{\gamma }((1-C_0)(f({\mathcal{X}}_t)-f({\mathcal{X}}_{t+1}))-C_0(f({\mathcal{X}}_{t+1})-f^{\ast })).\end{array}$$

Thus

$$\begin{array}{cc}\hfill \mathcal{E}(t+1)-\mathcal{E}(t)& \le \frac{C_{0}t+1}{\gamma }(f({\mathcal{X}}_{t+1})-f({\mathcal{X}}_t))+\frac{C_0}{\gamma }(f({\mathcal{X}}_{t+1})-f^{\ast })\hfill \\ \hfill & +\frac{1-C_0}{\gamma }(f({\mathcal{X}}_t)-f({\mathcal{X}}_{t+1}))-\frac{C_0}{\gamma }(f({\mathcal{X}}_{t+1})-f^{\ast })\hfill \\ \hfill & =\frac{C_{0}t+C_0}{\gamma }(f({\mathcal{X}}_{t+1})-f({\mathcal{X}}_t))\le 0.\hfill \end{array}$$

Thus $\mathcal{E}(t)\le \mathcal{E}(0)$ and the result follows. $\square$

Funding

Open access funding provided by University of Geneva.