A Note on the Behavior of the Randomized Kaczmarz Algorithm of Strohmer and Vershynin

Abstract

In a recent paper by T. Strohmer and R. Vershynin [“A Randomized Kaczmarz Algorithm with Exponential Convergence”, Journal of Fourier Analysis and Applications, published online on April 25, 2008] a “randomized Kaczmarz algorithm” is proposed for solving systems of linear equations ${\{\langle a^i,x\rangle =b_i\}}_{i=1}^m$ . In that algorithm the next equation to be used in an iterative Kaczmarz process is selected with a probability proportional to ‖aⁱ‖ ². The paper illustrates the superiority of this selection method for the reconstruction of a bandlimited function from its nonuniformly spaced sampling values.

In this note we point out that the reported success of the algorithm of Strohmer and Vershynin in their numerical simulation depends on the specific choices that are made in translating the underlying problem, whose geometrical nature is “find a common point of a set of hyperplanes”, into a system of algebraic equations. If this translation is carefully done, as in the numerical simulation provided by Strohmer and Vershynin for the reconstruction of a bandlimited function from its nonuniformly spaced sampling values, then indeed good performance may result. However, there will always be legitimate algebraic representations of the underlying problem (so that the set of solutions of the system of algebraic equations is exactly the set of points in the intersection of the hyperplanes), for which the selection method of Strohmer and Vershynin will perform in an inferior manner.

1 Introduction

Kaczmarz’s algorithm [21] is a sequential projection method for the solution of linear systems of equations of the form Ax = b. It was rediscovered in the field of image reconstruction from projections in [13] where it is called the Algebraic Reconstruction Technique (ART), see, e.g., [5, 7, 17], and it is obtained also by applying to the hyperplanes described by the linear system the method of successive projections onto convex sets. The latter is called in the literature POCS (for projections onto convex sets), see, e.g., [25], and was originally published by Bregman [3] and further studied by Gubin, Polyak and Raik [15], see, e.g., Bauschke and Borwein [1] and Combettes [9] for reviews. It belongs also to the class of row-action methods described in [6], see also [7]. The literature on Kaczmarz’s algorithm is vast and ranges from beautiful mathematical theoretical results to many successful applications in important real-world problems. Kaczmarz’s method is also a special case of several other algorithmic structures, see, e.g., [4, 8, 12, 20].

In a recent paper [24] (see also [23]) Strohmer and Vershynin propose a randomized Kaczmarz’s algorithm and claim that “it outperforms all previously known methods on general extremely overdetermined systems.” In this note we point out that the reported success of the algorithm of [24] in their numerical simulation depends on the specific choices that are made in translating the underlying problem, whose geometrical nature is “find a common point of a set of hyperplanes”, into a system of algebraic equations. If this translation is carefully done, as in the numerical simulation provided by [24] for the reconstruction of a bandlimited function from its nonuniformly spaced sampling values, then indeed good performance may result. However, there will always be legitimate algebraic representations of the underlying problem (so that the set of solutions of the system of algebraic equations is exactly the set of points in the intersection of the hyperplanes), for which the selection method of [24] will perform in an inferior manner.

2 The randomized Kaczmarz’s algorithm

Each equation of the m× n system Ax = b gives rise to a hyperplane

$$H_i=\{x\in R^n|\langle a^i,x\rangle =b_i\}$$ (1)

in the n-dimensional Euclidean space Rⁿ, where aⁱ ∈ Rⁿ and b_i ∈ R are the i-th row of the matrix A and the i-th component of the right-hand side vector b ∈ R^m of the linear system, respectively. The Euclidean inner product is $\langle a^i,x\rangle :={\displaystyle {\sum }_{j=1}^n{a}_j^i{x}_j}$ where $a^i={(a_j^i)}_{j=1}^n$ and $x={\left(x_j\right)}_{j=1}^n$ and ‖aⁱ‖ denotes the Euclidean norm. The randomized Kaczmarz’s algorithm of Strohmer and Vershynin is as follows.

Algorithm 1 [24, Algorithm 1]

Initialization

x⁰ ∈ Rⁿ is arbitrary.

Iterative Step

Given the current iterate x^k, calculate the next iterate x^k+1 by

$$x^{k+1}=x^k+\frac{b_{r(i)-\langle a^{r(i)},x^k\rangle }}{\parallel a^{r(i)}{\parallel }^2}{a}^{r(i)},$$ (2)

where r (i) is chosen from the set {1, 2, …, m} at random with probability proportional to ‖a^{r (i)}‖ ².

Obviously, the equations can be scaled, i.e., both sides of each equation 〈aⁱ, x〉 = b_i can be divided through by an arbitrary nonzero real number, say, c_i, without changing the hyperplane H_i but changing only its algebraic presentation. The question is how do such permissible changes affect the behavior of Algorithm 1, if at all, and what does that mean for the superiority claim of this algorithm in [24].

3 The results of Strohmer and Vershynin

Strohmer and Vershynin use the scaled condition number of a matrix A, introduced by Demmel [10],

$$\kappa \left(A\right):=\parallel A{\parallel }_F\parallel A^{-1}{\parallel }_2,$$ (3)

where ‖A‖₂ and ‖A‖ _F are the spectral norm and the Frobenius norm of A, respectively. Their main result [24, Theorem 2] states that any iterative sequence ${\{x^k\}}_{k=o}^{\mathrm{\infty }},$ , generated by Algorithm 1, converges to the solution of the system Ax = b with an expected exponential rate, and that the rate of convergence depends on the scaled condition number κ (A).

In their numerical experiments they compare what they call the standard Kaczmarz method (in which the rows of A are selected in their “natural” order 1, 2, …) with a simple randomized Kaczmarz method (in which the rows of A are selected at random with equal probability and with the randomized Kaczmarz method of Algorithm 1 (where the rows of A are selected at random with probability proportional to the 2-norm of the rows). They plot the least squares error versus the number of projections for each algorithm and claim that “Algorithm 1 significantly outperforms the other Kaczmarz methods, demonstrating not only the power of choosing the projections at random, but also the importance of choosing the projections according to their relevance.”

4 Theory and practice

Clearly, Kaczmarz’s method is a geometric algorithm (this term was used by Gordon and Mansour [14]) in the sense that the sequence of iterates generated by it depends only on the hyperplanes defined by the equations and not on any particular algebraic representation of the hyperplanes. One can always use real nonzero numbers, say c_i, i = 1, 2, …, m, and divide through each equation 〈aⁱ, x〉 = b_i, without affecting the hyperplanes defined by the equations and the solution set of the system. Therefore, the rate of convergence of a geometric algorithm must depend on properties of the underlying geometric objects (the hyperplanes) and not on their algebraic representation. This geometric approach has been mathematically studied in several works, see, e.g., Hamaker and Solmon [16], Kayalar and Weinert [22], Bauschke, Borwein and Lewis [2] and Deutsch and Hundal [11] which consider, in various settings, the connection between the angles among the underlying sets and the rate of convergence of the alternating projections method. When specialized to hyperplanes, that algorithm coincides with Kaczmarz’s method and the results apply to the linear system of equations Ax = b.

A scaling of the equations will change the system matrix A and its scaled condition number κ(A) and, in the light of [24], it might be tempting to think that it is possible to control in such a way the convergence rate of Kaczmarz’s method. However, the geometric nature of Kaczmarz’s method precludes such a possibility. Given any probability distribution whatsoever to be used for selecting the next hyperplane in Algorithm 1, we can always use real nonzero numbers to divide through the equations so that the rule for selecting the rows in Algorithm 1 will be exactly according to that probability distribution.¹ Thus, the rule proposed in Algorithm 1 cannot be in general optimal (or even in any sense superior).

Thus, the results shown in Figure 1 of [24], cannot be fully explained by the presented theory there. It is true that the convergence behavior depends on the order-selection of hyperplanes (as is well-known), but this dependence is determined by the geometry of the hyperplanes (the angles among them) and not by their algebraic representation, see, e.g., [2, 11, 16, 18, 19, 22].

Stated differently, if, for any reason whatsoever, we believe that at iteration k hyperplane H_i should be picked with higher probability then hyperplane H_j, then this is the case however the equations representing H_i and H_j are scaled (since the step of projecting onto the selected hyperplane is independent of the scaling). Hence, a claim that states that an order-selection methodology that depends on the norms of the aⁱs in the representation of the hyperplanes is generally superior to another order-selection methodology must be inherently false.

To avoid inferior behavior by the randomized Kaczmarz algorithm, it is essential that the system of algebraic equations that represent the set of hyperplanes be carefully chosen. Indeed, this was done by Strohmer and Vershynin [24] in their numerical simulation in Section 4.1, see their equation (18). Had they selected a different algebraic representation, they would have obtained different convergence behavior. In particular, if they scaled their equations so that ‖aⁱ‖ = 1 after scaling, then the reported difference between the behaviors of simple randomized Kaczmarz and of Algorithm 1 would have disappeared. And, if by the luck of the draw, the algebraic representation happened to be such that the norm associated with one equation is very much larger than the norms associated with the other equations, then the progress made by Algorithm 1 towards a solution would be poor due to the fact that the random selection would keep selecting the same equation most of the time.