Steerable Principal Components for Space-Frequency Localized Images*

Abstract

As modern scientific image datasets typically consist of a large number of images of high resolution, devising methods for their accurate and efficient processing is a central research task. In this paper, we consider the problem of obtaining the steerable principal components of a dataset, a procedure termed “steerable PCA” (steerable principal component analysis). The output of the procedure is the set of orthonormal basis functions which best approximate the images in the dataset and all of their planar rotations. To derive such basis functions, we first expand the images in an appropriate basis, for which the steerable PCA reduces to the eigen-decomposition of a block-diagonal matrix. If we assume that the images are well localized in space and frequency, then such an appropriate basis is the prolate spheroidal wave functions (PSWFs). We derive a fast method for computing the PSWFs expansion coefficients from the images' equally spaced samples, via a specialized quadrature integration scheme, and show that the number of required quadrature nodes is similar to the number of pixels in each image. We then establish that our PSWF-based steerable PCA is both faster and more accurate then existing methods, and more importantly, provides us with rigorous error bounds on the entire procedure.

1. Introduction

Principal component analysis (PCA), also known as Karhunen–Loeve transform, is a ubiquitous method for dimensionality reduction which is often utilized for compression, denoising, and feature extraction from datasets. Given a dataset, the basic idea behind classical PCA is to find the best linear approximation (in the least-squares sense) to the dataset using a set of orthonormal basis functions, thus allowing for processing methods adaptive to the dataset at hand. Due to increasing improvements in image acquisition and storage techniques, we often encounter the need to process very large datasets, which consist of many thousands of images of ever-growing resolutions. In addition, there exist image acquisition techniques that introduce known deformation types into the images, thus increasing the variability in the data.

In this work, we focus on the rotation deformation, and specifically, on the setting where each image was acquired through an unknown planar rotation. Therefore, it is only natural to include all planar rotations of all images when performing the PCA procedure. It is important to mention that, when handling large datasets, the naive approach of introducing a large number of rotated versions of all images into the dataset, and then performing standard PCA, is computationally prohibitive, and, moreover, is less accurate than considering the continuum of all rotations. In the literature, one can find numerous works concerning the study of deformations (such as rotations, translations, dilations, etc.) and their connections to invariant-feature extraction, through the theory of Lie groups and compact group theory [20, 4, 21, 22, 11]. In this context, we aim to incorporate the action of the group SO(2) into the framework of PCA of image datasets. We will refer to such a procedure as steerable PCA. By the theory of Lie groups, and specifically the action of the group SO(2), it is known that the resulting principal components (which best approximate all rotations of a set of images) are described by tensor products of radial functions and angular Fourier modes. Such functions are referred to as “steerable” [7], since they can be rotated (or “steered”) by a simple multiplication with a complex-valued constant, and hence the term “steerable” in steerable PCA. An early approach for computing the steerable PCA was proposed in [25], which used the SVD to obtain the principal components of a single template and its set of deformations. In [9], the authors presented an efficient steerable-PCA algorithm based on angular Fourier decomposition of the images, after they have been resampled to a polar grid. While this method considers the continuum of all planar rotations, it requires a nonobvious discretization of the images in the radial direction. Efficient methods for steerable PCA were also introduced in [27] and [10]. These methods considered a finite set of equiangular rotations of each image on a polar grid, which allows for efficient circulant matrix decompositions to be carried out when computing the principal components. Recently, [38] and [37] utilized Fourier–Bessel basis functions to expand the images, followed by applying PCA in the domain of the expansion coefficients, thus accounting for all (infinitely many) rotations. We also mention [35], where the authors present an accurate algorithm to obtain the steerable principal components of templates whose analytical form is known in advance.

As digital images are typically specified by their samples on a Cartesian grid, considering their rotations implicitly assumes that they were sampled from some underlying bivariate function. Rotation of a sampled image essentially interpolates this underlying function from the available Cartesian samples. We remark that while previous works provide algorithms for steerable PCA of discretized input images, they lack in describing rigorous connections between the results of the procedure and the images prior to discretization, i.e., the underlying bivariate functions. In this work, we assume that the digital images were sampled from bivariate functions that are essentially bandlimited and are sufficiently concentrated in an area of interest in space. These assumptions guarantee that if an image was sampled with a sufficiently high sampling rate, it can be reconstructed from its samples with high precision [26]. Such assumptions are very common in various areas of engineering and physics, and as all acquisition devices are essentially bandlimited and restricted in space/time, are expected to hold for a wide range of image datasets. Since we are interested in processing images with arbitrary orientations, it is only natural to consider a circular support area, both in space and frequency, instead of the (classical) square. We note that this model was implicitly assumed to hold in [38] and [37] for single-particle cryo-electron microscopy (cryo-EM) images.

Under the model assumptions mentioned above, our goal is to develop a fast and accurate steerable-PCA procedure which considers the continuum of all planar rotations of all images in our dataset. Particularly appealing basis functions, for expanding bandlimited functions which are also concentrated in space, are the prolate spheroidal wave functions (PSWFs) [33, 16, 17, 32, 24], defined as the strictly bandlimited set of orthogonal functions, which maximize the ratio between their ℒ² norm inside some finite region of interest and their ℒ² norm over the entire Euclidean space. Recently, [13] described an approximation scheme for functions localized to disks in space and frequency, using a series of two-dimensional PSWFs. We therefore incorporate the methods introduced in [13] into the framework of steerable PCA, providing accurate, scalable, and efficient algorithms. Our approach is in the spirit of [37], resulting in a similar block-diagonal covariance matrix. However, replacing the Fourier–Bessel with PSWFs turns out to be advantageous in terms of accuracy, available error bounds, speed, and statistical properties.

The contributions of this paper are the following. By utilizing theoretical and computational tools related to PSWFs, we are able to provide accuracy guarantees for our steerable-PCA algorithm, under the assumptions of space-frequency localization. This accuracy is in part related to a rigorous truncation rule we provide for the PSWFs series expansion, in contrast to the series truncation rules used in [38] and [37]. In addition, using a quadrature integration scheme optimized for integrating bandlimited functions on a disk [31], we present an algorithm which is in theory (and in practice) faster than [37] by a factor between 2 and 4. Finally, we also show that under some conditions on the space-frequency concentration of the images at hand, the transformation to the PSWFs expansion coefficients is nearly orthogonal.

The organization of this paper is as follows. In section 2 we introduce the PSWFs and their usage in expanding space-frequency localized images specified by their equally spaced Cartesian samples. In particular, we review the results of [13] on expanding a function into a series of PSWFs, evaluating the expansion coefficients, and bounding the overall approximation error. In section 3 we present a fast algorithm for approximating the expansion coefficients from section 2 up to an arbitrary precision. In section 4 we formalize the procedure of steerable PCA for the continuous setting (similarly to [35]), and combine it with our PSWFs-based approximation scheme. Section 5 then summarizes all relevant algorithms, and analyses in detail the computational complexities involved. In section 6 we provide some numerical results on the spectrum of the transformation to the PSWFs expansion coefficients, as this spectrum is of particular interest for noisy datasets, and in section 7 we compare our algorithm to that of [37] in terms of running time and accuracy. Finally, in section 8 we provide some concluding remarks and some possible future research directions.

2. Image approximation based on PSWFs expansion

Let f : ℝ² → ℝ be a square integrable function on ℝ². We define a function f(x) as c-bandlimited if its two-dimensional Fourier transform, denoted by F(ω), vanishes outside a disk of radius c. Specifically, if we denote D ≜ {x ∈ ℝ², |x| ≤ 1}, then f is bandlimited with bandlimit c if

$$f(x)={\left(\frac{1}{2\pi }\right)}^2{\int }_{c\mathbf{D}}F(\omega )e^{\iota \omega x}d\omega ,x\in {ℝ}^2.$$ (1)

Among all c-bandlimited functions, the prolate spheroidal wave functions (PSWFs) on D (the unit disk) are the most energy concentrated in D; that is, they satisfy

$$\frac{{\Vert \psi (x)\Vert }_{{\mathcal{L}}^2(\mathbf{D})}}{{\Vert \psi (x)\Vert }_{{\mathcal{L}}^2({ℝ}^2)}}\to \underset{\psi }{max},$$ (2)

while constituting an orthonormal system over ℒ²(D). The two-dimensional PSWFs were derived and analyzed in [32], and were shown to be the solutions to the integral equation

$$\alpha \psi (x)={\int }_{\mathbf{D}}\psi (\omega )e^{\iota c\omega x}d\omega ,x\in \mathbf{D}.$$ (3)

We denote the PSWFs with bandlimit c as ${\psi }_{N,n}^c(x)$, and their corresponding eigenvalues as. ${\alpha }_{N,n}^c$, which together form the eigenfunctions and eigenvalues of (3), with N ∈ ℤ and n ∈ ℕ ∪ {0}. In addition, it turns out that the functions ${\psi }_{N,n}^c(x)$ are orthogonal on both D and ℝ² using the standard ℒ² inner products on D and ℝ², respectively, and are dense in both the class of ℒ²(D) functions and in the class of c-bandlimited functions on ℝ². In polar coordinates, the functions ${\psi }_{N,n}^c(x)$ have a separation of variables and can be written in complex form as

$${\psi }_{N,n}^c(r,\theta )=\frac{1}{\sqrt{2\pi }}{R}_{N,n}^c(r)e^{\iota N\theta },N\in \mathbb{Z},n\in \mathbb{N}\cup \{0\},$$ (4)

where R_N,n(r) (defined explicitly in (66) in Appendix C) satisfies R_N,n(r) = R_{|N|, n} (r), and the eigenfunctions ${\psi }_{N,n}^c(x)$ are normalized to have an ℒ²(D) norm of 1. The indices N and n are often referred to as the angular index and the radial index respectively. Equation (4) also tells us that the PSWFs are steerable [7]; that is, rotating ${\psi }_{N,n}^c(r,\theta )$ is equivalent to multiplying it by a complex constant. This property is important for handling datasets which include rotations and in particular for the steerable-PCA procedure. A detailed numerical evaluation procedure for the two-dimensional PSWFs can be found in [31] and an illustration of the PSWFs for the several first index pairs (N, n) can be seen in Figure 1.

Figure 1. Illustration of the first few PSWFs (real part) with bandlimit c = 16π, ordered according to their eigenvalues (for every index N, the eigenvalues are ordered from largest to smallest as a function of the index n).

Since our images are assumed to be essentially bandlimited and sufficiently concentrated in a disk, the properties of the PSWFs mentioned above (and especially their optimal concentration property) make them suitable basis functions for expanding such images, as we consider next. Let us define a function f(x) as (ν, μ)-concentrated if its ℒ² norm outside a disk of radius ν is upper bounded by μ, that is

$$\sqrt{{\int }_{x\notin \nu \mathbf{D}}{|f(x)|}^{2}dx}\le \mu .$$ (5)

Using this definition, the class of c-bandlimited functions is a subclass of (c, δ_c)-concentrated functions in the Fourier domain (for any δ_c ≥ 0). We consider the two-dimensional functions from which our images were sampled to be (1, ε)-concentrated in space, with their Fourier transforms being (c, δ_c)-concentrated. This assumption always holds for some set of parameters c, δ_c, ε, and the general notion is that δ_c and ε are “small.” Since our images are given in their sampled form, we define the unit square Q ≜ [−1, 1] × [−1, 1], and assume that we are given the samples $\{f(x_k):x_k=\frac{k}{L}\in \mathbf{Q}\}$, where k is a two-dimensional integer index. These samples correspond to a Cartesian grid of (2L + 1) × (2L + 1) equally spaced samples, with sampling frequency of L in each dimension. We mention that in this setup, the Nyquist frequency corresponds to a bandlimit c of at most πL.

Given an image I(x) ∈ ℒ²(D), we can expand it as

$$I(x)=\sum _{N=-\infty }^{\infty }\sum _{n=0}^{\infty }{a}_{N,n}{\psi }_{N,n}^c(x),x\in \mathbf{D},a_{N,n}={\int }_{\mathbf{D}}I(x){({\psi }_{N,n}^c(x))}^{\ast }dx,$$ (6)

where (·)* denotes complex conjugation. As numerical algorithms cannot use infinite expansions, the image I(x) needs to be approximated by a finite sequence of PSWFs, while its expansion coefficients {a_N,n} are approximated using only the available samples ${\{I(\frac{k}{L})\}}_{\frac{k}{L}\in \mathbf{Q}}$.To this end, we follow [13], and define the truncated PSWFs expansion that approximates I(x) as

$$\widehat{I}(x)\triangleq \sum _{N,n\in {\mathrm{\Omega }}_T}{\widehat{a}}_{N,n}{\psi }_{N,n}^c(x),$$ (7)

where â_N,n are the approximated coefficients (to be described shortly), and Ω_T is a finite set of indices that is determined by a truncation parameter T, and is defined by

$${\mathrm{\Omega }}_T\triangleq \{(N,n):\sqrt{\frac{{\left|{\lambda }_{N,n}^c\right|}^2}{1-{\left|{\lambda }_{N,n}^c\right|}^2}}>T\},T>0,{\lambda }_{N,n}^c\triangleq \frac{c}{2\pi }{\alpha }_{N,n}^c,$$ (8)

where ${\alpha }_{N,n}^c$ is the eigenvalue from (3) corresponding to the eigenfunction ${\psi }_{N,n}^c$. The properties of the index set Ω_T are determined by the “normalized” eigenvalues ${\lambda }_{N,n}^c$, whose behavior is exemplified in Figure 2. The coefficients â_N,n of (7) are defined by

Figure 2.

Illustration of the normalized eigenvalues $|{\lambda }_{N,n}^c|$ for L = 64, c = πL, sorted in a nonincreasing order with a joint index k enumerating over (N, n). It is evident that for T ≪ 1 the set Ω_T consists of indices up to the right of the vertical dashed line, where the normalized eigenvalues rapidly decay to zero. On the other hand, for T ≫ 1 the set Ω_T consists of indices up to the left of the vertical line, where the normalized eigenvalues are extremely close to 1. The dashed vertical line corresponds to k = c²/4, which also approximately agrees with ${|{\lambda }_{N,n}^c|}^2=0.5$ (or equivalently with the rule T = 1).

$${\widehat{a}}_{N,n}\triangleq {\left|{\lambda }_{N,n}^c\right|}^2\frac{\langle I,{\psi }_{N,n}^c\rangle }{L^2}=\frac{{\left|{\lambda }_{N,n}^c\right|}^2}{L^2}\sum _{\frac{k}{L}\in \mathbf{D}}I\left(\frac{k}{L}\right){\left({\psi }_{N.n}^c\right(\frac{k}{L}\left)\right)}^{\ast },$$ (9)

where both I and ${\psi }_{N,n}^c$ stand for the appropriate vectors of samples of I(x) and ${\psi }_{N,n}^c(x)$ inside the unit disk, respectively. We note that for real-valued images we have that a_N,n = (a_−N,n)*, and thus it is sufficient to compute the coefficients â_N,n only for N ≥ 0.

Using our definitions in (7), (8), (9), and assuming that I(x) is (1, ε)-concentrated in space and (c, δ_c)-concentrated in Fourier domain (as defined earlier), we show in Appendix A that

$${\Vert I(x)-\sum _{N,n\in {\mathrm{\Omega }}_T}{\widehat{a}}_{N,n}{\psi }_{N,n}^c(x)\Vert }_{{\mathcal{L}}^2(\mathbf{D})}\le (\epsilon +\frac{{\delta }_c}{2\pi })(T+4)\triangleq \mathcal{E}(\epsilon ,{\delta }_c,T).$$ (10)

It is evident that the bound ℰ(ε, δ_c, T) in (10) depends only on the images' concentration properties and on the truncation parameter T, all of which can be determined a priori.

Lastly, it is also mentioned in [13] that the cardinality of the index set Ω_T of (8) is expected to be

$$|{\mathrm{\Omega }}_T|=\frac{c^2}{4}-\frac{2}{{\pi }^2}clog(c)log(T)+o(clog(c)).$$ (11)

The first term on the right hand-side of (11), which depends quadratically on c, is in fact (up to a constant of 1/π) what is known as the Landau rate [14] for stable sampling and reconstruction, and it is the term which dominates asymptotically the required number of basis functions for the approximation (see also Figure 2). A table listing the values of |Ω_T| for various L and T can be found in [13]. We remark that for c = πL (Nyquist sampling) and T = 1, which essentially means that the approximation error is of the order of the space-frequency localization, we have that |Ω_T| is about π²L²/4, which is smaller than the number of samples (pixels) in the unit disk.

3. Fast PSWFs coefficients approximation

Computing the expansion coefficients â_N,n of (9) directly results in a computational complexity of O(L⁴) operations. This is due to the fact that each image contains O(L²) samples, while there are about O(L²) different basis functions in the expansion. Since we aim to process large datasets of high resolution images, in what follows we describe an asymptotically more efficient method for evaluating the expansion coefficients in O(L³) operations.

Using (3), we can rewrite the PSWFs expansion coefficients (9) as

$${\widehat{a}}_{N,n}={\alpha }_{N,n}^c{\left(\frac{c}{2\pi L}\right)}^2\sum _{\frac{k}{L}\in \mathbf{D}}I\left(\frac{k}{L}\right){\left[{\int }_{\mathbf{D}}{\psi }_{N,n}^c\left(\frac{k}{L}\right)e^{\iota c\omega \frac{k}{L}}\right]}^{\ast }$$ (12)

$$={\alpha }_{N,n}^c{\left(\frac{c}{2\pi L}\right)}^2{\int }_{\mathbf{D}}{[{\psi }_{N,n}^c(\omega )]}^{\ast }{\varphi }^c(\omega )d\omega ,$$ (13)

where

$${\varphi }^c(u)\triangleq \sum _{\frac{k}{L}\in \mathbf{D}}I\left(\frac{k}{L}\right)e^{-\iota cu\frac{k}{L}},u\in {ℝ}^2.$$ (14)

Since both ${\psi }_{N,n}^c$ and ϕ^c are c-bandlimited functions, the product ${\psi }_{N,n}^c\cdot {\varphi }^c$ defines a function which is 2c-bandlimited. Therefore, we can compute the integral in (13) using a quadrature formula for 2c-bandlimited functions on a disk. Such an integration scheme was proposed in [31] using specialized quadrature nodes and weights. Thus, it follows that, by using such a quadrature formula, we can approximate the expansion coefficients of (13) by

$${\stackrel{\sim }{a}}_{N,n}\triangleq {\alpha }_{N,n}^c{\left(\frac{c}{2\pi L}\right)}^2\sum _{\ell =1}^{{\mathcal{N}}_r}\sum _{j=1}^{N_{\theta }^{\ell }}{\mathcal{W}}_{\ell ,j}^{2c}{[{\psi }_{N,n}^c({\omega }_{\ell ,j}^{2c})]}^{\ast }{\varphi }^c({\omega }_{\ell ,j}^{2c}),$$ (15)

where ${\omega }_{\ell ,j}^{2c}$ and ${\mathcal{W}}_{\ell ,j}^{2c}$ are the quadrature nodes and weights respectively, 𝒩_r is the number of different radii of the quadrature nodes, and ${\mathcal{N}}_{\theta }^{\ell }$ is the number of quadrature nodes per radius (which may vary for each radius). We use for ${\omega }_{\ell ,j}^{2c}$ and ${\mathcal{W}}_{\ell ,j}^{2c}$ the quadrature nodes and weights of [31] corresponding to bandlimit 2c. These quadrature nodes are equally spaced in the angular direction and nonequally spaced in the radial direction. In polar coordinates, the nodes and weights are given by

$${\omega }_{\ell ,j}^{2c}=(r_{\omega }^{\ell },{\theta }_{\omega }^{\ell ,j}),{\theta }_{\omega }^{\ell ,j}=\frac{2\pi j}{{\mathcal{N}}_{\theta }^{\ell }},{\mathcal{W}}_{\ell ,j}^{2c}=\frac{2\pi }{{\mathcal{N}}_{\theta }^{\ell }}{r}_{\omega }^{\ell }{\stackrel{\sim }{\mathcal{W}}}_{\ell }^{2c},$$ (16)

and the derivation of ${\stackrel{\sim }{\mathcal{W}}}_{\ell }^{2c}$ is detailed in [31]. A typical array of quadrature nodes is plotted in Figure 3.

Figure 3.

Quadrature nodes ${\omega }_{\ell ,j}^{2c}=(r_{\omega }^{\ell },{\theta }_{\omega }^{\ell ,j})$ for L = 16, c = πL, and accuracy of the order of machine precision. The nodes are arranged to be equally spaced in the angular direction (the specific phase of the allocation is arbitrary) with different numbers of nodes for each radius (rings closer to the origin require fewer angular nodes). The number of radial nodes is about c/π, and the number of angular nodes per radius $r_{\omega }^{\ell }$ is about $c{r}_{\omega }^{\ell }e$, where e is the base of the natural logarithm.

By employing (15), (16), and the analytical expression of the PSWFs (4), we finally arrive at

$${\stackrel{\sim }{a}}_{N,n}=\sqrt{2\pi }{\alpha }_{N,n}^c{\left(\frac{c}{2\pi L}\right)}^2\sum _{\ell =1}^{{\mathcal{N}}_r}{\stackrel{\sim }{\mathcal{W}}}_{\ell }^{2c}\frac{R_{N,n}^c(r_{\omega }^{\ell })r_{\omega }^{\ell }}{{\mathcal{N}}_{\theta }^{\ell }}\sum _{j=1}^{{\mathcal{N}}_{\theta }^{\ell }}{\varphi }^c({\omega }_{\ell ,j}^{2c})e^{-\frac{2\pi \iota jN}{{\mathcal{N}}_{\theta }^{\ell }}},$$ (17)

which is particularly appealing for efficient numerical evaluation, as we explain later in this section.

Clearly, replacing the integral in (13) with a quadrature formula results in an approximation error. In this respect, it is desirable to choose 𝒩_r and ${\mathcal{N}}_{\theta }^{\ell }$ such that this error is smaller than some prescribed accuracy ϑ_q (usually chosen as the machine precision). To this end, and according to [31], it is sufficient to satisfy the condition

$$\sum _{|j|>{\mathcal{N}}_{\theta }^{\ell }}{J}_j(2c{r}_{\omega }^{\ell }\rho )\le C_1{\vartheta }_q,$$ (18)

for all 1 ≤ ℓ ≤ 𝒩_r and ρ ∈ [0, 1], where J_j is the Bessel function of the first kind and order j, as well as the condition

$$\sum _{k=2{\mathcal{N}}_r+1}^{\infty }\frac{\left|{\lambda }_{0,k}^{2c}\right|}{2c}{\Vert R_{0,k}^{2c}(r)\Vert }_{\infty }{\Vert R_{0,k}^{2c}(r)\sqrt{r}\Vert }_{\infty }(1+\sum _{j=1}^{{\mathcal{N}}_r}|{\stackrel{\sim }{W}}_j^{2c}|)\le C_2{\vartheta }_q,$$ (19)

where ${\lambda }_{N,n}^c$ was defined in (8), ‖·‖_∞ stands for the max-norm in [0, 1], and both C₁ and C₂ are constants which depend on the bandlimit c. We mention that $|{\lambda }_{0,k}^{2c}|$ are ordered in a nonincreasing order with respect to the index k. In order to determine appropriate values for 𝒩_r and ${\mathcal{N}}_{\theta }^{\ell }$, one can proceed to solve the inequalities in conditions (18) and (19) numerically by directly evaluating these expressions. However, in order to analyze the computational complexity of the procedure described in this section, and to offer the reader simpler analytic expressions and some insight concerning conditions (18) and (19), we provide some further analysis of these conditions and the resulting number of quadrature nodes. In Appendix B, we prove that, in order to satisfy condition (18) for every 1 ≤ ℓ ≤ 𝒩_r, it is sufficient to choose the integers ${\mathcal{N}}_{\theta }^{\ell }$ as

$${\mathcal{N}}_{\theta }^{\ell }=\lceil c{r}_{\omega }^{\ell }e+log{\vartheta }_q^{-1}+log\frac{2}{C_1}+1\rceil ,$$ (20)

where $r_{\omega }^{\ell }$ are the radial quadrature nodes from (16), e is the base of the natural logarithm, and ⎾·⏋ is the rounding up operation. Then, it is evident that the term $c{r}_{\omega }^{\ell }e$ dominates condition (20), i.e., ${\mathcal{N}}_{\theta }^{\ell }\sim c{r}_{\omega }^{\ell }e$, since all other factors (prescribed error ϑ_q and the constant C₁) affect it only logarithmically. Therefore, we expect the overall number of quadrature nodes to be

$$\sum _{\ell =1}^{{\mathcal{N}}_r}{\mathcal{N}}_{\theta }^{\ell }\sim ce\sum _{\ell =1}^{{\mathcal{N}}_r}{r}_{\omega }^{\ell }\sim \frac{ce{\mathcal{N}}_r}{2},$$ (21)

assuming that the quadrature points $r_{\omega }^{\ell }$ are approximately symmetric about 0.5. We remark that numerical experiments reveal that there is a small difference between conditions (18) and (20), and specifically, choosing ${\mathcal{N}}_{\theta }^{\ell }$ directly via the numerical evaluation of condition (18) results in about 20% fewer angular nodes compared to choosing ${\mathcal{N}}_{\theta }^{\ell }$ according to (20).

With regard to condition (19), although it may seem somewhat daunting, the sum on the left-hand side is dominated by the values of $|{\lambda }_{0,k}^{2c}|$ and their decay properties. We argue in Appendix C that the number of non-negligible (relative to machine precision) values of $|{\lambda }_{0,k}^{2c}|$ is only about 2c/π, that is

$$|\{k:|{\lambda }_{0,k}^{2c}|>\epsilon \}|~\frac{2c}{\pi },$$ (22)

for some small ε and a sufficiently large c. This observation stems from the fact that the values of $|{\lambda }_{0,k}^{2c}|$ become arbitrarily small (and deca y at a superexponential rate) once k reaches 2c/π + O(logc). We provide numerical evidence for this claim in Figure 4.

Figure 4.

Behavior of $|{\lambda }_{0,n}^{2c}|$for c = πL (Nyquist sampling) and several values of L. For each value of L, the dotted vertical line represents n = 2c/π, and the dashed vertical line represents the first index n for which $|{\lambda }_{0,n}^{2c}|\le {10}^{-12}$, which we denote by $n_1^{2c}$. First, we observe that 2c/π is the exact index for which $|{\lambda }_{0,n}^{2c}|$ begins its rapid decay, and, second, it is evident that the difference between $n_1^{2c}$ and the estimate 2c/π grows extremely slowly with L, making the asymptotic estimate $n_1^{2c}~2c/\pi$ sufficiently accurate even for moderate values of L.

Therefore, it is evident that condition (19) can be satisfied if $|{\lambda }_{0,k}^{2c}|$ is sufficiently small for k > 2𝒩_r, which together with (22) implies that for c = πL (Nyquist sampling)

$${\mathcal{N}}_r~\frac{c}{\pi }=L.$$ (23)

In total, by (21) and (23), the number of quadrature nodes for c = πL is

$$\sum _{\ell =1}^{{\mathcal{N}}_r}{\mathcal{N}}_{\theta }^{\ell }~\frac{e}{2\pi }{c}^2=\frac{\pi e}{2}{L}^2\approx 4.27L^2,$$ (24)

which is only slightly larger than the number of sampling points in the unit square.

Let us now turn our attention to the computational complexity of computing ã_N,n of (17). The first step is to compute ${\varphi }^c({\omega }_{\ell ,j}^{2c})$ from (14) for all quadrature nodes $\{{\omega }_{\ell ,j}^{2c}\}$, which can be implemented efficiently using the nonequispaced fast fourier transform (NFFT) [3, 8, 28, 5]. The computational complexity of such algorithms is

$$O(L^{2}logL+log{\epsilon }^{-2}P),$$ (25)

where L is the sampling rate, ε is the required accuracy of the transform, and P is the number of evaluation points, which is equal to the number of quadrature nodes (24). Next, we notice that, for every value of ℓ the inner sum in (17)

$$C_{\ell }^N\triangleq \sum _{j=1}^{{\mathit{N}}_{\theta }^{\ell }}{\varphi }^c({\omega }_{\ell ,j}^{2c})e^{-\frac{2\pi \iota jN}{{\mathcal{N}}_{\theta }^{\ell }}}$$ (26)

can be computed for multiple values of N using the FFT in $O({\mathcal{N}}_{\theta }^{\ell }log{\mathcal{N}}_{\theta }^{\ell })$ operations, resulting in total of $O({\sum }_{l=1}^{{\mathcal{N}}_r}{\mathcal{N}}_{\theta }^{\ell }log{\mathcal{N}}_{\theta }^{\ell })$ operations for all required values of ℓ. By (24), this results in a computational complexity of O(L² log L) for c = πL. Lastly, the approximated expansion coefficients (17) are given by the remaining outer sum

$${\stackrel{\sim }{a}}_{N,n}=\sqrt{2\pi }{\alpha }_{N,n}^c{\left(\frac{c}{2\pi L}\right)}^2\sum _{\ell =1}^{{\mathcal{N}}_r}{\stackrel{\sim }{\mathcal{W}}}_{\ell }^{2c}\frac{R_{N,n}^c(r_{\omega }^{\ell })r_{\omega }^{\ell }}{{\mathcal{N}}_{\theta }^{\ell }}{C}_{\ell }^N,$$ (27)

which can be computed for all indices {N, n} ∈ Ω_T in 𝒩_r |Ω_T| operations, where |Ω_T| denotes the cardinality of the index set Ω_T. From (11) and (23), the computational complexity of the last step is essentially O(L³), which thus governs the computational complexity of the entire procedure described in this section. We end this discussion with the observation that the complexity of evaluating (27) can be further reduced by exploiting the rapid decay of the functions $R_{N,n}^c(r)$ with the radius r (see Figure 1b for an illustration), due to which a substantial number of the terms involving $R_{N,n}^{\ell }(r_{\omega }^{\ell })$ can be discarded for certain sets of the radii $\{r_{\omega }^{\ell }\}$ and indices {(N, n)}. To exemplify this point, for T = 1, L = 150, and c = πL, we have that about 30% of the values $\{|R_{N,n}^c(r_{\omega }^{\ell })|\}$ are below 10⁻¹², and therefore can be safely discarded from (27).

4. Steerable-PCA procedure

As discussed in the introduction, steerable PCA extends the classical PCA by artificially including in the analyzed dataset all (infinitely many) planar rotations of each image. In the previous sections, we have established a framework for expanding images using PSWFs, under the assumption that the images to be approximated are sufficiently concentrated in space and frequency. Since PSWFs constitute an excellent basis for expanding images which are localized in space and frequency (see (50), (10) and [13]), we use them in this section to construct optimized basis functions for a given set of images and their rotations.

Let us suppose that our dataset consists of M (sampled) images, where the mth image is given by the samples $I_m\left(\frac{k}{L}\right)$ of some function I_m(x) ∈ ℒ²(D). We denote by $I_m^{\phi }(x)$ the planar rotation of I_m(x) by an angle ψ ∈ [0, 2π). Now, ideally, we would like to obtain basis functions that best approximate (in the sense of ℒ²(D)) the images $I_m^{\phi }(x)$ for all rotation angles ψ ∈ [0, 2π). However, we do not have access to the underlying images I_m(x), nor their rotations $I_m^{\phi }(x)$. Therefore, we first replace them with their PSWFs-based approximations, denoted by Î_m(x) (see (7)) and ${\widehat{I}}_m^{\phi }(x)$. Then, we derive a steerable-PCA procedure for the set of approximated images, and finally we make the connection between the resulting steerable principal components of the approximated images and the original images.

The kth principal component g_k(x) ∈ ℒ²(D) for the set of approximated images $\{{\widehat{I}}_m^{\phi }\}$, m = 0, …, M – 1, is defined as

$$\begin{array}{c}{g}_k(x)=\underset{g(x)}{argmax}\frac{1}{M}\sum _{m=0}^{M-1}\frac{1}{2\pi }{\int }_0^{2\pi }{\left|{\langle {\widehat{I}}_m^{\phi }(x)-\widehat{\mu }(x),g(x)\rangle }_{{\mathcal{L}}^2(\mathbf{D})}\right|}^{2}d\phi ,\\ \mathrm{s}.\mathrm{t}.{\Vert g_k(x)\Vert }_{{\mathcal{L}}^2(\mathbf{D})}=1,{\langle g_k(x),g_i(x)\rangle }_{{\mathcal{L}}^2(\mathbf{D})}=0,0\le i\le k,\end{array}$$ (28)

where 〈·, ·〉_ℒ²(D) denotes the standard inner product on ℒ²(D), and μ̂(x), is the average of all approximated images and their planar rotations, given by

$$\widehat{\mu }(x)=\frac{1}{M}\sum _{m=0}^{M-1}\frac{1}{2\pi }{\int }_0^{2\pi }{\widehat{I}}_m^{\phi }(x)d\phi .$$ (29)

In other words, the function g_k(x) is expected to maximize the average projection norm (defined over all images and their rotations), such that {g_k(x)} forms a set of orthonormal functions over ℒ²(D). The functions {g_k(x)} are named the steerable principal components of our dataset. The formulation (28) differs from the classical formulation of PCA in the additional integration over the rotation angle φ, which has the interpretation of including all (infinitely many) rotations of all images in our data set.

After substituting (7) in (28) and exploiting the orthonormality and steerable structure of the PSWFs, the optimization problem (28) is reduced to a problem in the domain of the PSWFs expansion coefficients â_N,n, for which the derivation in [38] reveals that the solution is given by the eigenvectors of the |Ω_T| × |Ω_T| Hermitian positive semidefinite matrix, whose entries are

$$C_{(N,n),(N^{\prime },n^{\prime })}=\{\begin{array}{cc}\frac{1}{M}{\sum }_{m=0}^{M-1}{\widehat{c}}_{N,n}^m{\left({\widehat{c}}_{N^{\prime },n^{\prime }}^m\right)}^{\ast },& N=N^{\prime },\\ 0,& N\ne N^{\prime },\end{array}$$ (30)

where

$${\widehat{c}}_{N,n}^m=\{\begin{array}{cc}{\widehat{a}}_{N,n}^m-\frac{1}{M}{\sum }_{m^{\prime }=0}^{M-1}{\widehat{a}}_{0,n}^{m^{\prime }},& N=0,\\ {\widehat{a}}_{N,n}^m,& N\ne 0,\end{array}$$ (31)

and ${\widehat{a}}_{N,n}^m$, given by (9) (or (17)), are the approximated expansion coefficients for the mth image in our dataset, i.e., the coefficients of Î_m. We refer to C of (30) as our rotationally invariant covariance matrix (replacing the standard covariance matrix used in classical PCA). Since the matrix C enjoys a block-diagonal structure, we obtain its eigen-decomposition through the eigen-decomposition of its blocks. Let λ̂₁ ≥ λ̂₂ ≥ … ≥ λ̂_|ΩT| ≥ 0 be the eigenvalues of the matrix C and let ĝ_k be the eigenvector corresponding to eigenvalue λ̂_k. For each pair (λ̂_k, ĝ_k), the entries of ĝ_k, denoted by ${\widehat{g}}_{N,n}^k$, are nonzero only on some block of the matrix C corresponding to an angular index N_k, and it can be shown that the functions g_k(x) of (28) are recovered as

$$g_k(x)=\sum _{n=0}^{n_k}{\widehat{g}}_{N_k,n}^k{\psi }_{N_k,n}(x),$$ (32)

where n_k stands for the largest index n such that (N_k, n) ∈ Ω_T. Equation (32), together with (4), confirms that each g_k(x) is a steerable function (as in [7]).

By the formulation of (28), it follows that the functions g_k(x) form the optimal basis (in ℒ²(D)) for expanding the approximated images {Î_m(x)} and their rotations, such that if we define

$${\stackrel{\sim }{I}}_{K,m}^{\phi }(x)\triangleq \widehat{\mu }(x)+\sum _{k=1}^K{c}_{k,m}^{\phi }{g}_k(x),c_{k,m}^{\phi }\triangleq e^{-\iota N_k\phi }\sum _{n=0}^{n_k}{\widehat{c}}_{N_k,n}^m{\left({\widehat{g}}_{N_k,n}^k\right)}^{\ast },$$ (33)

then we have that

$$\frac{1}{M}\sum _{m=0}^{M-1}\frac{1}{2\pi }{\int }_0^{2\pi }{\Vert {\widehat{I}}_m^{\phi }(x)-{\stackrel{\sim }{I}}_{K,m}^{\phi }(x)\Vert }_{{\mathcal{L}}^2(\mathbf{D})}^{2}d\phi \le \sum _{k=K+1}^{|\mathrm{\Omega }|}{\widehat{\lambda }}_k,$$ (34)

where λ̂_k is the kth eigenvalue of C.

Since the steerable principal components were computed for the set of approximated images {Î_m(x)} and not for the set of underlying images {I_m(x)}, the bound in (34) holds only for the set of the approximated images. Nonetheless, the PSWFs approximation scheme from section 2 provides us with strict error bounds when expanding images localized in space and frequency. Therefore, when using (33) to approximate our images I_m(x), the error norm (averaged over all images and rotations) can be bounded by joining (34) and (10). In particular, for a truncation parameter T, we have that

Algorithm 1. Evaluating PSWFs expansion coefficients (direct method).

Required: An image $\{I(\frac{k}{L})\}$ sampled on a Cartesian grid of size (2L + 1) × (2L + 1) with $\frac{k}{L}\in \mathbf{Q}$, and Q = [−1, 1] × [−1, 1]. Precomputation: Choose a bandlimit c (≤ πL) and a truncation parameter T. Evaluate the PSWFs ${\psi }_{N,n}^c(\frac{k}{L})$ and their eigenvalues ${\alpha }_{N,n}^c$ according to [31] for (N, n) ∈ ΩT (see (8)), where $\frac{k}{L}\in \mathbf{D}$, and compute the normalized eigenvalues ${\lambda }_{N,n}^c=\frac{c}{2\pi }{\alpha }_{N,n}^c$. For all (N, n) ∈ ΩT, compute the expansion coefficients ${\widehat{a}}_{N,n}=\frac{{|{\lambda }_{N,n}^c|}^2}{L^2}\sum _{\frac{k}{L}\in \mathbf{D}}I(\frac{k}{L}){({\psi }_{N,n}^c(\frac{k}{L}))}^{\ast }$.

$$\frac{1}{M}\sum _{m=0}^{M-1}\frac{1}{2\pi }{\int }_0^{2\pi }{\Vert I_m^{\phi }(x)-{\stackrel{\sim }{I}}_{K,m}^{\phi }(x)\Vert }_{{\mathcal{L}}^2(\mathbf{D})}^{2}d\phi \le \sum _{k=K+1}^{|{\mathrm{\Omega }}_T|}{\widehat{\lambda }}_k+2\mathcal{E}(\epsilon ,{\delta }_c,T)\sqrt{\sum _{k=K+1}^{|{\mathrm{\Omega }}_T|}{\widehat{\lambda }}_k}+{\mathcal{E}}^2(\epsilon ,{\delta }_c,T),$$ (35)

where ${\stackrel{\sim }{I}}_{K,m}^{\phi }(x)$ is the expansion via the steerable principal components (33), and ε(ε, δ_c, T) is the approximation error term of (10) for the truncated series of PSWFs. In essence, (35) asserts that if the images are sufficiently localized in space and frequency, then for an appropriate truncation parameter T the error in expanding I_m(x) using the steerable principal components computed from Î_m(x) is close to the smallest possible error given by (34).

5. Algorithm summary and computational cost

We summarize the algorithms for evaluating the expansion coefficients â_N,n and ã_N,n (corresponding to the direct method from section 2 and the efficient method of section 3, respectively) in Algorithms 1 and 2 respectively. The steerable-PCA procedure described in section 4 is summarized in Algorithm 3.

We now turn our attention to the computational complexity of Algorithms 1, 2, and 3. We omit the precomputation steps from the complexity analysis as they can be performed only once per setup, and do not depend on the specific images.

Since we have O(L²) PSWFs in our expansions (corresponding to the indices in the set Ω_T) and O(L²) equally spaced Cartesian samples in the unit disk, computing the expansion coefficients of M images using the direct approach of Algorithm 1 requires O(ML⁴) operations. On the other hand, Algorithm 2 allows us to obtain the expansion coefficients in O(ML³) operations, since the NFFT in step 3 requires O(L²logL) operations and step 4 can be implemented using O(L² log L + L³) operations, for each image.

Although Algorithm 2 and the method described in [37] (based on Fourier–Bessel basis functions) have the same order of computational complexity, our method enjoys a twofold asymptotic speedup in the coefficients' evaluation, and often it runs about three to four times faster. The twofold asymptotic speedup is because we need asymptotically only half the number of radial nodes for the numerical integration (see analysis in section 3 and Appendix C) compared to the Gaussian quadratures used in [37], which dictates the constant in the leading term of the computational complexity. The greater speedup observed in practice stems from the fact that practically, the most time-consuming operation in both methods is the NFFT, which depends heavily on the total number of quadrature nodes. In our integration scheme, the total number of quadrature nodes is about a quarter of the number of nodes in [37].

Algorithm 2. Evaluating PSWFs expansion coefficients (efficient method).

Required: An image $\{I(\frac{k}{L})\}$ sampled on a Cartesian grid of size (2L + 1) × (2L + 1) with $\frac{k}{L}\in \mathbf{Q}$, and Q = [−1, 1] × [−1, 1]. Precomputation: Choose a bandlimit c (≤ πL) and a truncation parameter T. Choose the number of radial nodes 𝒩r and angular ${\mathcal{N}}_{\theta }^{\ell }$ nodes for 1 ≤ ℓ < 𝒩r (according to (18) and (19) or similar relaxed conditions). Compute the quadrature nodes ${\omega }_{\ell ,j}^{2c}=(r_{\omega }^{\ell },{\theta }_{\omega }^{\ell ,j})$ and weights for ${\mathcal{W}}_{\ell ,j}^{2c}$ for ℓ = 1, …, 𝒩r and $j=1,\dots ,{\mathcal{N}}_{\theta }^{\ell }$, as described in [31]. Evaluate the radial part of the PSWFs $R_{N,n}^c(r)$ at the radial quadrature nodes for $r_{\omega }^{\ell }$ (N, n) ∈ ΩT (see (8)). Compute ${\varphi }^c({\omega }_{\ell ,j}^{2c})$ from (14) by NFFT. For (N, n) ∈ ΩT, compute the expansion coefficients ãN,n via equations (26) and (27).

Algorithm 3. PSWFs-based steerable PCA.

Required: PSWFs expansion coefficients of M images ${\{{\widehat{a}}_{N,n}^m\}}_{m=0}^{M-1}$ for (N, n) ∈ ΩT and a bandlimit c. Precomputation: Evaluate the PSWFs ${\psi }_{N,n}^c(\frac{k}{L})$, $\frac{k}{L}\in \mathbf{D}$ and their eigenvalues ${\alpha }_{N,n}^c$ for (N, n) ∈ ΩT according to [31]. Compute the expansion coefficients of the mean image for ${\widehat{\mu }}_{0,n}=\frac{1}{M}{\sum }_{m=0}^{M-1}{\widehat{a}}_{0,n}^m$ for n = 0, …, n0 where n0 is the largest index n such that (0, n) ∈ ΩT. Update ${\widehat{a}}_{0,n}^m\leftarrow {\widehat{a}}_{0,n}^m-{\widehat{\mu }}_{0,n}$. Compute the eigenvalues λ̂1, …, λ̂|ΩT| and eigenvectors ĝ1, …, ĝ|ΩT| of the matrix C (from (30)) by diagonalizing each of its blocks separately. Compute the sampled basis functions $g_{\ell }(\frac{k}{L})={\sum }_{n=0}^{n_{\ell }}{\widehat{g}}_{N_{\ell },n}^{\ell }{\psi }_{N_{\ell },n}^{\ell }(\frac{k}{L})$, where nℓ stands for the largest index n such that (Nℓ, n) ∈ ΩT, and ${\widehat{g}}_{N_{\ell },n}^{\ell }$ are the entries of the eigenvector ĝℓ corresponding to the pair (Nℓ, n). Compute the coefficients of Im in the steerable-PCA basis by $c_{\ell ,m}={\sum }_{n=0}^{n_{\ell }}{\widehat{a}}_{N_{\ell },n}^m{\left({\widehat{g}}_{N_{\ell },n}^{\ell }\right)}^{\ast }$.

Next, as of the computational complexity of the steerable PCA (Algorithm 3), forming the blocks of the matrix C requires O(ML³) operations, and then the eigen-decomposition of each block requires O(L³) operations, where there are O(L) different blocks in the matrix. Therefore, obtaining the eigenvalues and eigenvectors in step 5 of Algorithm 3 requires O(ML³ + L⁴) operations. As pointed out in [38] and [37], the eigenvalues and eigenvectors can be also obtained from the SVD of the matrices of coefficients ${\widehat{c}}_{N,n}^m$ from which the blocks of C are obtained.

Following (32), if we have O(L²) basis functions g_k(x), their evaluation on the Cartesian grid requires O(L⁵) operations. Sometimes it may be more convenient to evaluate these basis functions on a polar grid instead, in which case the computational complexity reduces to O(L⁴).

Lastly, computing the expansion coefficients of the images in the steerable basis via (33) requires O(ML³) operations, since O(L) operations are required to compute a single expansion coefficient for every image.

We point out though that often only a small fraction of the basis functions are chosen for subsequent processing (via their eigenvalues), so the contribution of steps 6 and 7 in Algorithm 3 to the overall running time of the steerable PCA procedure is usually negligible.

To summarize, the computational complexity of the entire procedure (computing PSWFs expansion coefficients + steerable PCA), when using Algorithm 2 to compute the expansion coefficients, is O(ML³ + L⁵) when sampling the steerable principal components on the Cartesian grid and O(ML³ + L⁴) when sampling them on a polar grid. It is important to note that, although Algorithm 1 for evaluating PSWFs expansion coefficients suffers from an inferior order of computational complexity (compared to Algorithm 2), it is simpler to implement and may still run faster (due to optimized implementations of the scalar product on CPUs and GPUs), particularly for small values of L.

6. Steerable PCA in the presence of noise

Up to this point, we have presented a method for computing the steerable PCA of a set of images localized in space and frequency, sampled on a Cartesian grid. In many practical settings however, the images are corrupted with noise. Therefore, it is beneficial to understand the impact this noise has on the PSWFs expansion coefficients, and in particular, it is generally convenient if the transformation to the expansion coefficients does not alter the spectrum of the noise. In this section, we demonstrate numerically that for sufficiently localized images in space-frequency, for which we can choose a truncation parameter T ≫ 1 (see (8)), the transformation to the PSWFs expansion coefficients is essentially orthogonal. In particular, the higher the truncation parameter T, the closer is our steerable PCA to orthogonality.

Let us denote by I a column vector consisting of the clean (Cartesian) samples of an input image. Suppose that our image is corrupted by an additive noise such that

$$\stackrel{\sim }{I}=I+\xi ,$$ (36)

where ξ is zero mean noise vector with covariance matrix R_ξ. From (9), our approximated expansion coefficients are given (up to a constant) by

$${({\widehat{\psi }}^c)}^{\ast }\stackrel{\sim }{I}={({\widehat{\psi }}^c)}^{\ast }I+{({\widehat{\psi }}^c)}^{\ast }\xi ,$$ (37)

where the operator (·)* stands for the conjugate transpose, and ψ̂^c denotes the matrix whose columns contain samples of ${\psi }_{N,n}^c(x){|{\lambda }_{N,n}^c|}^2/L$ inside the unit disk, with different columns corresponding to different pairs of indices (N, n). If we define the vector of additive noise in the expansion coefficients as

$$\widehat{\xi }={({\widehat{\psi }}^c)}^{\ast }\xi ,$$ (38)

then its covariance matrix is provided by

$$R_{\widehat{\xi }}={({\widehat{\psi }}^c)}^{\ast }{R}_{\xi }{\widehat{\psi }}^c.$$ (39)

Now, if the noise in (36) is white (R_ξ = σ²I), and in order to preserve the covariance of the noise, we would require the matrix

$$H_c\triangleq {({\widehat{\psi }}^c)}^{\ast }{\widehat{\psi }}^c$$ (40)

to be as close as possible to the identity matrix, or equivalently that the eigenvalues of the matrix H_c be as close as possible to 1. As the matrix H_c is Hermitian and positive semidefinite, it has non-negative real-valued eigenvalues ν₁, ν₂, …, ν_ΩT. To determine how close H_c is to the identity matrix, we evaluated numerically the maximal distance (in absolute value) between the eigenvalues {ν_k} and 1, and the results are plotted in Figure 5 for various values of L, T, and the bandlimit c.

Figure 5.

Measured deviation of the eigenvalues of H_c from 1 for different values of the truncation parameter T, the bandlimit c, and the sampling resolution L. We notice that the deviation from orthogonality remains approximately constant for different values of L and c. Specifically, for T = 10⁶ we notice that H_c is practically orthogonal.

It is noteworthy that for T = 10⁶ the eigenvalues of H_c differ from 1 by about 10⁻¹². We have also observed that the spectrum of the matrix H_c is related to the truncation parameter T by

$$\underset{k}{max}|1-{\nu }_k|\approx 2T^{-2},$$ (41)

for T ≥ 10. This observation is exemplified numerically in Figure 6. As the matrix H_c in (40) is essentially orthogonal for sufficiently large values of T, it is important to mention that, if the images are sufficiently localized in space and frequency, it is often possible to choose the value of T as to enjoy the orthogonality of the transform, while keeping the bound in (10) sufficiently small.

Figure 6.

Measured deviation of the eigenvalues of H_c from 1 as a function of T, versus the function 2T⁻², for L = 96 and c = πL. We obsereve that the truncation parameter T gives us direct control over the spectrum of H_c with max_k |1 – ν_k| ≈ 2T⁻² for T ≥ 10.

7. Numerical experiments

We begin by demonstrating the running times of our algorithm for large datasets. We generated several sets of 20,000 white noise images, where the sets consist of images with different L for each set, and compared the running time of our algorithm with the FFBsPCA algorithm of [37], where the coefficients' evaluation for our method was carried out using both Algorithm 1 (direct method) and Algorithm 2 (efficient method). All of the algorithms were implemented in Matlab, and were executed on a dual Intel Xeon X5560 CPU (8 cores in total), with 96 GB of RAM running Linux. Whenever possible, all 8 cores were used simultaneously, either explicitly using Matlab's parfor, or implicitly, by employing Matlab's implementation of BLAS, which takes advantage of multicore computing. As for the NFFT implementation, we used the software package [12], with an oversampling of 2 and a truncation parameter m = 6 (which provides accuracy close to machine precision). The running times (in seconds) are shown in Table 1. As anticipated, Algorithm 1 runs faster than Algorithm 2 for small image sizes, but becomes significantly slower for larger values of L. As for the running time of our algorithm versus FFBsPCA [37], we have mentioned in section 5 that our algorithm is asymptotically two times faster, since the number of radial nodes in our quadrature scheme is asymptotically half that of FFBsPCA. In addition, the total number of quadrature nodes in our scheme is about a quarter of that of FFBsPCA, and since the NFFT procedure for evaluating the Fourier transform of the sampled images on a polar grid (see (14)) is the most time-consuming step of both algorithms, it is expected that our algorithm will be faster than FFBsPCA by a factor between 2 and 4, which indeed agrees with the results in Table 1. In all scenarios tested, most of the computation time was spent on evaluating the PSWFs expansion coefficients (Algorithm 2), and only a small fraction of the time on the eigen-decomposition of the rotationally invariant covariance matrix. Table 2 summarizes the time spent on the evaluation of the PSWFs expansion coefficients (using Algorithm 2) versus time spent on the eigen-decomposition of the rotationally invariant covariance matrix.

Table 1.

Comparison of algorithms' running times, for 20,000 images consisting of white noise, for T = 10 and several values of L. All timings are given in seconds.

L	PSWFs direct (Algs. 1+3)	PSWFs efficient (Algs. 2+3)	FFBsPCA [37]
32	6	25	47
64	95	59	151
96	491	117	363
128	1625	204	697

Table 2.

Running times of coefficients' evaluation and eigen-decomposition for the efficient PSWFs-based method. All timings are given in seconds.

L	PSWFs coefficients evaluation (Alg. 2)	Eigen-decomposition (Alg. 3 steps 3-6)
32	24.5	0.5
64	56.5	2.5
96	111	6
128	193	11

Next, we demonstrate our algorithms on simulated single-particle cryo-electron microscopy (cryo-EM) projection images. In single-particle cryo-EM, one is interested in reconstructing a three-dimensional model of a macromolecule (such as a protein) from its two-dimensional projection images taken by an electron microscope. The procedure begins by embedding many copies of the macromolecule in a thin layer of ice (hence the “cryo” in the name of the procedure), where due to the experimental setup the different copies are frozen at random unknown orientations. Then, an electron microscope acquires two-dimensional projection images of the electron densities of these macromolecules. This procedure of image acquisition can be modelled mathematically as the Radon transform of a volume function evaluated at random viewing directions. Due to the properties of the imaging procedure, each projection image generated by the electron microscope undergoes a convolution with a kernel, referred to as the “contrast transfer function” (CTF), which is known to have a Gaussian envelope [6]. Since the unknown volume is essentially limited in space, and since the behavior of the CTF dictates that all images are localized in Fourier domain, we conclude that projection images obtained by single-particle cryo-EM are essentially limited to circular domains in both space and frequency. We emphasize that although the goal in single particle cryo-EM is to reconstruct the three-dimensional structure of the macromolecule, the input to the reconstruction process is only the set of two-dimensional images. Now, since the in-plane rotation of each single-particle cryo-EM image in the detector plane is arbitrary and irrelevant for the reconstruction, the image processing methods applied to the input image dataset, such as denoising and classification, should be invariant to these in-plane rotations. This observation explains why these images are suitable for exemplifying our steerable-PCA algorithm.

To demonstrate the accuracy of our method, we simulated 10,000 clean projection images from a noiseless three-dimensional density map (EMD-5578 from The Electron Microscopy Data Bank (EMDB) [19]) using the ASPIRE package [1], and obtained steerable principal components using both our fast algorithm and FFBsPCA [37]. Then, we used different numbers of principal components to reconstruct the images, and compared the theoretical error predicted by the residual eigenvalues (see (34)) with the empirical error obtained by comparing the reconstructions to the original images. Obviously, the difference between the two errors is due to the error incurred by the images' expansion scheme (PSWFs or Fourier–Bessel functions). Typical projection images from this dataset can be seen in Figure 7. To simplify the setting and the exposition, we used the projection images as they were obtained from the given volume, and we did not crop, filter, or process them beforehand. Therefore, we assumed a bandlimit c = πL throughout the experiment. We note that in general it is possible to crop and filter the images (i.e., choose a smaller bandlimit c) without significantly degrading the quality of the images (up to some prescribed accuracy), thus reducing the computational burden of the algorithms. This can be accomplished either by power density estimation (as demonstrated in [37]), or by employing more sophisticated and dataset-specific estimation techniques for the B-factor (which governs the Gaussian envelope decay of the CTF); see for example [23]. Figure 8 depicts the first 12 eigenfunctions obtained by our steerable-PCA algorithm for this dataset. In Figure 9(a) we show the relative error norms for the FFBsPCA algorithm and our PSWFs-based algorithm, using several values of T and with different numbers of eigenfunctions in the reconstruction. As expected, the theoretical and empirical relative error norms coincide when a small number of principal components is used in the reconstruction, yet when a large number of principal components is used, the error incurred by the expansion of the images (either using PSWFs or Fourier–Bessel) comes into play and dominates the overall error. As guaranteed by (35) and (10) for space-frequency localized images, smaller values of T lead to smaller approximation errors, and we notice that our PSWFs-based method outperforms the FFBsPCA algorithm in terms of accuracy for T = 10 and T = 10⁻³. It is also important to mention that the number of PSWFs taking part in the approximation of each image does not increase significantly when lowering T (see (11)). On the other hand, if one allows the approximation error to be of the order of 10⁻³ to 10⁻⁴, then it is also possible to use a very large truncation parameter, such as T = 10⁵, for which the PSWFs-based transform enjoys superior orthogonality properties (see Figure 5), as well as shorter expansions. Often, such scenarios arise when handling noisy datasets. In a way, the truncation parameter provides us with flexibility to adapt the approximation scheme to the specific setting. In Figure 9(b) we show the approximation errors obtained by expanding the images in the PSWFs basis and in the Fourier–Bessel basis (without the rotationally invariant orthogonalization), where we sorted the basis functions according to their contribution to the expansion. It is evident that PSWFs are more adapted to expanding our image dataset than Fourier–Bessel, which is reasonable due to the underlying model behind these images, whereas both bases provide lower accuracy than the steerable principal components obtained from our PSWFs-based steerable PCA.

Figure 7. Sample of 3 simulated noiseless projection images at a resolution of L = 127 pixels.

Figure 8. The first 12 eigenfunctions with largest eigenvalues, obtained for T = 10⁻³.

Figure 9.

The ratio between the squared error norm and the squared norm of the images, when expanding the images with different numbers of basis functions. Panel (a) compares theoretical errors with empirical errors for our PSWFs-based algorithm (for several values of T) and the FFBsPCA algorithm of [37]. Curves which correspond to theoretical errors are obtained from the residual eigenvalues of the steerable-PCA procedure (see (34)), and empirical errors correspond to measured errors between the reconstructed and original projection images. Panel (b) illustrates the errors due to steerable-PCA expansions, PSWFs expansions, and Fourier–Bessel expansions. Note that the x axis, which counts the number of basis functions used in each expansion, counts only basis functions with non-negative angular indices (since these are sufficient for encoding real-valued images).

8. Summary and discussion

In this paper, we utilized PSWFs-based computational tools to construct fast and accurate algorithms for obtaining steerable principal components of large image datasets. The accuracy of our algorithms is guaranteed under the assumptions of localization of the images in space and frequency, which are natural assumptions for many datasets, particularly in the field of tomography. For M images, each sampled on a Cartesian grid of size (2L + 1) × (2L + 1), the computational complexity for obtaining the steerable principal components is O(ML³) operations, and their accuracy is of the order of the localization of the images in space and frequency, i.e., the norm of the images outside the unit disk in space, and the norm of their Fourier transform outside a disk of radius c, where c is the chosen bandlimit. We have compared our method with the FFBsPCA algorithm [37], which is considered state of the art for performing steerable PCA on single-particle cryo-EM projection images, and have shown that our method is both faster and more accurate (for sufficiently small values of T). In addition, our method enjoys rigorous error bounds throughout its various steps, whereas in contrast the FFBsPCA algorithm provides no analytic guarantees on its accuracy. We mention that classical operations of windowing and filtering can be subsumed in our procedure to ensure that the images fulfil the requirements of space-frequency localization.

As image resolutions get higher, investigating more efficient methods for processing image datasets is an important ongoing research task. As the running times of our algorithms are mostly dominated by the task of computing the PSWFs expansion coefficients, reducing the computational complexity of this step from O(L³) to O(L² log L) for each image, resulting in the same asymptotic complexity as the two-dimensional FFT, will be a significant progress. Since the radial part of the PSWFs is evaluated to a prescribed accuracy using a finite series of special polynomials which admit a recurrence relation (see [31]), and since the expansion coefficients in terms of these polynomials are obtained from the eigenvectors of a tridiagonal matrix, it is possible to employ the methods described in [34] to derive an O(L² log L) algorithm for computing the PSWFs expansion coefficients of a function. In addition, as mentioned earlier, the NFFT, which was employed to map the Cartesian grid samples to a polar grid, is a major time-consuming component. In this context, we mention that gridding methods (see for example [29]), when applied carefully, may accelerate such a mapping, and thus reduce the overall running time of our algorithms.

Appendix A. PSWFs expansion error bound

Let us define the restriction of an image I(x) to the unit disk by

$$\overline{I}(x)\triangleq \{\begin{array}{cc}I(x),& x\in \mathbf{D},\\ 0,& x\notin \mathbf{D},\end{array}$$ (42)

and denote the Fourier transform of Ī(x) by

$${\overline{I}}^{\mathcal{F}}(\omega )\triangleq \mathcal{F}\{\overline{I}\}(\omega )={\int }_{{\mathbb{R}}^2}\overline{I}(x)e^{\iota \omega x}\mathit{\text{dx}}.$$ (43)

Since the Fourier transform is a linear operator, we can write

$${\overline{I}}^{\mathcal{F}}(\omega )=\mathcal{F}\{I-(I-\overline{I})\}(\omega )=I^{\mathcal{F}}(\omega )+\mathcal{F}\{I-\overline{I}\}(\omega ),$$ (44)

where we defined the Fourier transform of I(x) by I^ℱ(ω). Now, it is clear that

$${\Vert {\overline{I}}^{\mathcal{F}}(\omega )\Vert }_{{\mathcal{L}}^2(\omega \notin c\mathbf{D})}\le {\Vert I^{\mathcal{F}}(\omega )\Vert }_{{\mathcal{L}}^2(\omega \notin c\mathbf{D})}+{\Vert \mathcal{F}\{I-\overline{I}\}(\omega )\Vert }_{{\mathcal{L}}^2(\omega \notin c\mathbf{D})}$$ (45)

$$\le {\Vert I^{\mathcal{F}}(\omega )\Vert }_{{\mathcal{L}}^2(\omega \notin c\mathbf{D})}+{\Vert \mathcal{F}\{I-\overline{I}\}(\omega )\Vert }_{{\mathcal{L}}^2({\mathbb{R}}^2)}.$$ (46)

Next, by employing Parseval's identity we obtain

$${\Vert \mathcal{F}\{I-\overline{I}\}(\omega )\Vert }_{{\mathcal{L}}^2({\mathbb{R}}^2)}=2\pi {\Vert I(x)-\overline{I}(x)\Vert }_{{\mathcal{L}}^2({\mathbb{R}}^2)},$$ (47)

and thus, by using our space-frequency concentration assumptions on the image I(x), we have that

$${\Vert {\overline{I}}^{\mathcal{F}}(\omega )\Vert }_{{\mathcal{L}}^2(\omega \notin c\mathbf{D})}\le {\delta }_c+2\pi \epsilon .$$ (48)

Therefore, we conclude that Ī(x) is (1, ε̄)-concentrated in space, and its Fourier transform Ī^ℱ(ω) is (c, δ̄_c)-concentrated, where

$$\overline{\epsilon }=0,{\overline{\delta }}_c={\delta }_c+2\pi \epsilon .$$ (49)

Now, it follows from [13] that the approximation error of Ī satisfies

$${\Vert \overline{I}(x)-\sum _{N,n\in {\mathrm{\Omega }}_T}{\widehat{a}}_{N,n}{\psi }_{N,n}^c(x)\Vert }_{{\mathcal{L}}^2(\mathbf{D})}\le (\overline{\epsilon }+\frac{{\overline{\delta }}_c}{2\pi })T+\eta {\left(\frac{c}{2\pi L}\right)}^2\sqrt{\sum _{\frac{k}{L}\notin \mathbf{D}}{|\overline{I}(\frac{k}{L})|}^2}+\frac{2}{\pi }{\overline{\delta }}_c,$$ (50)

where T is the truncation parameter defined in (8), and η is a constant no bigger than 2π²L/c. Finally, since I_m(x) and Ī_m(x) coincide on D, we get using (50)

$${\Vert I_m(x)-\sum _{N,n\in {\mathrm{\Omega }}_T}{\widehat{a}}_{N,n}^m{\psi }_{N,n}^c(x)\Vert }_{L^2(\mathbf{D})}\le (\epsilon +\frac{{\delta }_c}{2\pi })(T+4).$$ (51)

Appendix B. A bound for ${\mathcal{N}}_{\theta }^{\ell }$

Recall that we choose the number of angular nodes per radius, denoted ${\mathcal{N}}_{\theta }^{\ell }$, such that it satisfies

$$\sum _{|j|>{\mathcal{N}}_{\theta }^{\ell }}|J_j(2c{r}_{\omega }^{\ell }\rho )|<C_1{\vartheta }_q,$$ (52)

for every ρ ∈ [0, 1]. Using a bound for the Bessel functions [2] together with the fact that |ρ| ≤ 1, we get

$$|J_j(2c{r}_{\omega }^{\ell }\rho )|\le {\left(\frac{2c{r}_{\omega }^{\ell }\rho }{2}\right)}^j\frac{1}{\mathrm{\Gamma }(j+1)}\le {\left(c{r}_{\omega }^{\ell }\right)}^j\frac{1}{(j+1)!},$$ (53)

and by using Stirling's approximation [2] (for n > 1)

$$n!\ge \sqrt{2\pi n}{\left(\frac{n}{e}\right)}^n>e{\left(\frac{n}{e}\right)}^n$$ (54)

we obtain (for j > 0)

$$|J_j(2c{r}_{\omega }^{\ell }\rho )|\le \frac{1}{c{r}_{\omega }^{\ell }e}{\left(\frac{c{r}_{\omega }^{\ell }e}{j+1}\right)}^{j+1}.$$ (55)

Then, using the fact that the Bessel functions of the first kind satisfy

$$J_{-n}(x)={(-1)}^n{J}_n(x),$$ (56)

we can write

$$\sum _{|j|>{\mathcal{N}}_{\theta }^{\ell }}|J_j(2c{r}_{\omega }^{\ell }\rho )|=2\sum _{j>{\mathcal{N}}_{\theta }^{\ell }}|J_j(2c{r}_{\omega }^{\ell }\rho )|\le \frac{2}{c{r}_{\omega }^{\ell }e}\sum _{j>{\mathcal{N}}_{\theta }^{\ell }}{\left(\frac{c{r}_{\omega }^{\ell }e}{j+1}\right)}^{j+1}.$$ (57)

We define

$${\gamma }_{\ell }\triangleq c{r}_{\omega }^{\ell }e$$ (58)

and choose the number of quadrature nodes per radius as

$${\mathcal{N}}_{\theta }^{\ell }=\lceil {\gamma }_{\ell }\rceil +d,$$ (59)

where d is some positive integer (to be determined shortly) and ⌈·⌉ denotes the rounding up operation. It can be easily verified that

$$\frac{{\gamma }_{\ell }}{j+1}\le \frac{{\gamma }_{\ell }}{{\gamma }_{\ell }+d}$$ (60)

whenever $j>{\mathcal{N}}_{\theta }^{\ell }$. Using (57) and (60) we get that

$$\sum _{|j|>{\mathcal{N}}_{\theta }^{\ell }}|J_j(2c{r}_{\omega }^{\ell }\rho )|\le \frac{2}{{\gamma }_{\ell }}\sum _{j>{\mathcal{N}}_{\theta }^{\ell }}{\left(\frac{{\gamma }_{\ell }}{{\gamma }_{\ell }+d}\right)}^{j+1}\le 2{(1+\frac{d}{{\gamma }_{\ell }})}^{-{\mathcal{N}}_{\theta }^{\ell }}\le 2{(1+\frac{d}{{\gamma }_{\ell }})}^{-{\gamma }_{\ell }-d}\le 2e^{-d}$$ (61)

where we have used the inequality

$${(1+\frac{a}{b})}^{-b}<e^{-\frac{ab}{a+b}},$$ (62)

from [2], with $a=({\gamma }_{\ell }+d)\frac{d}{{\gamma }_{\ell }}$ and b = γ_ℓ + d. Finally, we can see that in order to satisfy (52) it is sufficient to choose

$$d=d({\vartheta }_q)\ge log{\vartheta }_q^{-1}+log\frac{2}{C_1},$$ (63)

and thus

$${\mathcal{N}}_{\theta }^{\ell }\ge c{r}_{\omega }^{\ell }e+log{\vartheta }_q^{-1}+log\frac{2}{C_1}+1.$$ (64)

Appendix C. Behavior and decay properties of ${\lambda }_{0,k}^{2c}$

We start by reviewing some known results on the behavior of the eigenvalues of the PSWFs in the one-dimensional setting, where they have been thoroughly investigated (see [24] and references therein). The most well-known characterization of these eigenvalues is that they can be divided into three distinct regions of behavior (as a function of their index n): a flat region, where the (normalized) eigenvalues are essentially 1, a transitional region, where they decay from values close to 1 to values close to 0, and a superexponential decay region, where they are very close to 0 and decay as ∼ e^−nlogn. In addition, it is known that if we choose all eigenvalues that are greater than some small ε, then there are about 2c/π eigenvalues from the flat region, $O(log(\frac{1-\epsilon }{\epsilon })logc)$ eigenvalues from the transitional region, and o(logc) eigenvalues from the decay region (see [18] for a precise formulation). Thus, the number of eigenvalues greater then ε is dominated by the number of eigenvalues in the flat region, which is 2c/π. As for the eigenvalues of the two-dimensional PSWFs, results in [30] indicate that as in the one-dimensional setting, the eigenvalues can be similarly divided into three distinct regions: flat, transitional, and superexponential decay regions. Correspondingly, the number of significant eigenvalues is dominated by the number of eigenvalues in the flat region. Since ${\lambda }_{0,k}^{2c}\propto {\alpha }_{0,k}^{2c}$ (see (8)), it is clear that in order to satisfy condition (19) we need to determine the number of terms in the flat region of $|{\lambda }_{0,k}^{2c}|$. To this end, we follow [15], which provides similar results for general non-Hermitian Toeplitz integral operators (see also [18] and [36]), and consider the sum

$$\sum _{k=0}^{\infty }{|{\lambda }_{0,k}^{2c}|}^2,$$ (65)

which is approximately equal to the number of values of $|{\lambda }_{0,k}^{2c}|$ which are close to 1 (denoted as the flat region). For simplicity of the presentation, we evaluate this sum for a bandlimit of c, and eventually, replace c with 2c. From [32], the radial functions $R_{N,n}^c(r)$ in (4) are real-valued, and are obtained as the solutions to the integral equation

$$\beta R(r)={\int }_0^{1}R(\rho )J_N(cr\rho )\rho d\rho ,r\in [0,1],$$ (66)

where J_N(x) is the Bessel function of the first kind of order N. The eigenvalues ${\alpha }_{N,n}^c$ and ${\beta }_{N,n}^c$ of (3) and (66) are related by

$${\alpha }_{N,n}^c=2\pi {\iota }^N{\beta }_{N,n}^c.$$ (67)

By substituting φ(r) = R(r)√r and γ = β√c into (66), we obtain the integral equation

$$\gamma \phi (r)={\int }_0^1\phi (\rho )J_N(cr\rho )\sqrt{cr\rho }d\rho ,r\in [0,1],$$ (68)

whose eigenvalues and eigenfunctions are ${\gamma }_{N,n}^c$ and ${\phi }_{N,n}^c(r)$, respectively. Equation (68) was analyzed in [32], where it is established that the eigenfunctions ${\phi }_{N,n}^c(r)$ constitute a complete orthonormal system in ℒ² [0, 1]. Therefore, it follows that we have the identity

$$J_N(cr\rho )\sqrt{cr\rho }=\sum _{k=1}^{\infty }{\gamma }_{N,n}^c{\phi }_{N,n}^c(r){\phi }_{N,n}^c(\rho ),$$ (69)

for r and ρ both in [0, 1]. If we notice that ${\lambda }_{N,n}^c=\sqrt{c}{\gamma }_{N,n}^c$, and take N = 0, we have

$$J_0(cr\rho )c\sqrt{r\rho }=\sum _{k=1}^{\infty }{\lambda }_{0,k}^c{\phi }_{0,k}^c(r){\phi }_{0,k}^c(\rho ).$$ (70)

Next, we take the squared absolute value of both sides of the equation above, followed by double integration (in r and ρ) to obtain

$${\int }_0^1{\int }_0^1{J}_0^2(cr\rho )c^{2}r\rho drd\rho =\sum _{k=0}^{\infty }{|{\lambda }_{0,k}^c|}^2.$$ (71)

By evaluating the left-hand side of (71) using known integral identities of the Bessel functions [2], and after some manipulation, one can verify that

$$\sum _{k=0}^{\infty }{|{\lambda }_{0,k}^c|}^2=\frac{c^2}{4}(J_0^2(c)-J_2(c)J_0(c)+2J_1^2(c)).$$ (72)

Now, using the asymptotic approximation (see [2])

$$J_m(x)~\sqrt{\frac{2}{\pi x}}cos(x-\frac{m\pi }{2}-\frac{\pi }{4}),$$ (73)

valid for x ≫ |m² − 1/4|, we can write

$$\sum _{k=0}^{\infty }{|{\lambda }_{0,k}^c|}^2~\frac{c}{2\pi }\left({cos}^2\right(c-\frac{\pi }{4})-cos(c-\pi -\frac{\pi }{4})cos(c-\frac{\pi }{4})+2{cos}^2(c-\frac{\pi }{2}-\frac{\pi }{4}\left)\right)=\frac{c}{2\pi }\left(2{cos}^2\right(c-\frac{\pi }{4})+2{sin}^2(c-\frac{\pi }{4}\left)\right)=\frac{c}{\pi }.$$ (74)

Therefore, the expression in (74) establishes that for large values of c, the number of terms in the set $\{|{\lambda }_{0,n}^{2c}|\}$ which are close to 1 is about 2c/π (see also Figure 4).