Remarks on Kaczmarz Algorithm for Solving Consistent and Inconsistent System of Linear Equations

Abstract

In this paper we consider the classical Kaczmarz algorithm for solving system of linear equations. Based on the geometric relationship between the error vector and rows of the coefficient matrix, we derive the optimal strategy of selecting rows at each step of the algorithm for solving consistent system of linear equations. For solving perturbed system of linear equations, a new upper bound in the convergence rate of the randomized Kaczmarz algorithm is obtained.

Introduction

Kaczmarz algorithm [28] is an iterative method for solving system of linear equations of the form

1

where has full column rank, and . In the consistent case, the solution of (1) can be regarded as the coordinate of the common point of hyperplanes defined by each single equation in (1):

2

where , , denotes the ith row of A and is the ith element of vector b.

The idea of the Kaczmarz type algorithms is to exploit the geometric structure of the problem (1), and the using a sequential of projections to seek the solution. The recursive process can be formulated as follows. Let be an initial guess to the solution of (1), then the classical Kaczmarz algorithm iteratively generates a sequence of approximate solutions by the recursive formula:

3

where . For a given , from (3) we can see that satisfies the ith equation in (1), i.e., . The updating formula (3) implicitly produces a solution to the following constraint optimization problem [21, 37]

which is equivalent to finding the projection of from the hyperplane . Two geometric explanations of the above process can be illustrated by Fig. 1:

Fig. 1.

Geometric illustrations of the classical Kaczmarz iterations with .

By comparing the projection processes displayed in Fig. 1, it is natural to have the intuition that convergence of the classical Kaczmarz algorithm highly depends on the geometric positions of the associated hyperplanes. If the normal vectors of every two successive hyperplanes keep reasonably large angles, the convergence of the classical Kaczmarz algorithm will be fast, whereas two nearly parallel consecutive hyperplanes will make the convergence slow down. The Kaczmarz algorithm can be regarded as a special application of famous von Neumann’s alternating projection [35] originally distributed in 1933. The fundamental idea can even trace the history back to Schwarz [38] in 1870s.

In the past few years, the Karzmarz algorithm has been interpreted as successive projection methods [4, 7, 8, 11–13], which are also known as projection onto convex sets (POCS) [9, 17, 18, 42–44] in the optimization community. Notice that each iteration of the Kaczmarz algorithm just need flops and the cost is independent with the number of equations, this type of algorithms are well-suited to problems with . Due to its simplicity and generality, Kaczmarz algorithms find viable applications in the area of image processing and signal process [19, 20, 24–26, 30, 36] under the name of algebraic reconstruction techniques (ART). Since 1980s, relaxation variants [11, 25, 41]

4

and the block versions [3, 33, 34]

5

of the Kaczmarz algorithm have been widely investigated, and some fruitful theoretical results have been obtained. In particular, for consistent linear systems, it is shown [5, 21, 31, 39] that the Kaczmarz iterations converges to the least square norm solution with any starting vector in the column space of . For inconsistent linear systems, the cyclic subsequences generated by the Kaczmarz algorithm converges to a weighted least squares solution when the relaxation parameter goes to zero [12].

As indicated in Fig. 1, convergence of the classical Kaczmarz algorithm depends on the sequence of successive projections, which relies upon the ordering of the rows in the matrix A. In some real applications, it is observed [25, 30] that instead of selecting rows of the matrix A sequentially at each step of the Kaczmarz algorithm, randomly selection can often improve its convergence. Recently, in the remarkable paper [39], Strohmer and Vershynin proved the rate of convergence for the following randomized Kaczmarz algorithm

where r(i) is chosen from with probabilities . In particular, the following bound on the expected rate of convergence for the randomized Kaczmarz method is proved

6

where , with be the scaled conditioned number of A introduced by J. Demmel [14]. Due to this pioneering work that characterized the convergence rate for the randomized Kaczmarz algorithms, the idea stimulated considerable interest in this area and various investigations [1, 2, 6, 10, 15] have been performed recently. In particular, some acceleration strategies have been proposed [6, 16, 22] and convergence analysis was performed in [21, 23, 27, 29, 31, 32]. See also [21, 23] for some comments on equivalent interpretations of the randomized Kaczmarz algorithms.

Optimal Row Selecting Strategy of the Kaczmarz Algorithm for Solving Consistent System of Linear Equations

In this section, we consider the case that system of linear equations (1) is consistent and x is a solution. If the ith row is selected at the th iteration of the Kaczmarz algorithm, i.e.,

then can be reformulated as

It follows that

7

and thus

8

From (7) and (8), we can see that

9

i.e.,

10

To this end, let us make the following orthogonal direct sum decomposition ,

11

where and is a normalized vector orthogonal to . Then coefficients and can be written as

where is the angle between the vectors and .

Substituting the above decomposition (11) into (7) gives

12

It follows that

13

From (13) we can see that the error norms generated by the Kaczmarz algorithm are monotonically nonincreasing. Moreover, the convergence can be optimized if is minimized at every iteration, which is equivalent to selecting the row that solves the optimization problem

As x is the unknown solution, the above minimization problems seems unsolvable. However, noting that consistent linear system (1) implies

and is fixed at the th iteration. The minimization problem can be tackled by maximizing , i.e.,

14

where

It is clear from (14) that the optimal updating strategy for the Kaczmarz algorithm is to select the row that satisfies

i.e., the index where has the largest entry in absolute value. We refer to the above row selection method as the optimal selecting strategy, and call the Kaczmarz algorithm with the optimal selecting strategy as the optimal Kaczmarz algorithm.

Next, we analyze the convergence of the optimal Kaczmarz algorithm for solving consistent system of linear equations. To simplify the analysis, we introduce two notations

and

where and .

Based on (13), the th error can be bounded as follows

15

where .

Notice that

we can theoretically divide the convergence history of the Kaczmarz algorithm into two periods:

• when , the algorithm converge exponentially,

• when , we have

  

  and thus,

  

  This implies that , i.e., solves the system of linear equation (1).

In summary, for solving consistent system of linear equations (1), there exists a theoretical optimal selecting strategy or optimal randomization strategy for Kaczmarz algorithm. With the strategy, the algorithm converges exponentially and will achieve convergence when

Randomized Kaczmarz Algorithm for Solving Inconsistent System of Linear Equations

Suppose (1) is a consistent system of linear equations and its right hand side is perturbed with a noise vector r as follows:

16

where (16) can be either consistent or inconsistent. In this section, we give some remarks on the convergence of randomized Kaczmarz algorithm for solving (16), which was investigated by D. Needell [32].

Firstly, we recall the Lemma 2.2 in [32].

Lemma 1

Let be the affine subspaces of consisting of the solutions to unperturbed equations, . Let be the solution spaces of the noisy equations, . Then

where .

Remarks: If the Lemma 1 is used to interpret the Kaczmarz algorithm for solving the perturbed and unperturbed equations, we need to introduce a vector in the as the orthogonal complement of the vector , and write as

where is a solution generated by Kaczmarz algorithm for solving the unperturbed equations, and is a vector in the orthogonal complement of .

Example 1. Consider the system of linear equations

and the perturbed equations

i.e., , and .

Let

and

If we use as the same initial guess for the perturbed and unperturbed linear system, then

and

Note that and . We have

i.e.,

In order to derive the convergence rate of randomized Kaczmarz algorithm for solving the perturbed linear equations (16), we need to make use of the established convergence results [39] for the unperturbed linear system (1), together with the relationship between the approximate solutions generated by the Kaczmarz algorithm [39] for perturbed and unperturbed linear equations. In [32], D. Needell analyzed the convergence rate and error bound of the randomized Kaczmarz algorithm for solving the perturbed linear equations, in which the author take the approximate solution to the perturbed linear equations as the guess for the unperturbed system, which make the derivation process simplified. However, the approximate solutions generated by applying the randomized Kaczmarz algorithm to the perturbed linear system may not converge to the solution of the unperturbed linear system.

In what follows, we will consider the convergence rate of the randomized Kaczmarz algorithm for solving (16) from a different perspective. We try to bound the difference between the solution for the unperturbed linear system (1) and approximate solutions generated by applying the randomized Kaczmarz algorithm to the perturbed linear system.

In the following discussion, we use and to denote the approximate solutions generated by applying the randomized Kaczmarz algorithm to (1) and (16), respectively. The recursive formulas can be written as

17

and

18

where the subscript is used to denote that the th row is selected with probability at the kth iteration.

Suppose the same initial guess is used as the starting vector. Then

and potentially

It follows that

19

In the next iteration, we have

where with .

Continue the above process, we have

20

where and .

Subtracting x on both sides of (20) gives

21

Based on Jensen’s inequality and (6), we have

22

where , with .

Taking norm on both sides of (21) and using triangle inequality, we have

where .

In conclusion, we have derived the following theorem.

Theorem 1

Let A be a matrix full column rank and assume the system is consistent. Let be the kth iterate of the noisy randomized Kaczmarz method run with , and let denote the rows of A. Then we have

where and .

Conclusions

In this paper, we provide a new look at the Kaczmarz algorithm for solving system of linear equations. The optimal row selecting strategy of the Kaczmarz algorithm for solving consistent system of linear equations is derived. The convergence of the randomized Kaczmarz algorithm for solving perturbed system of linear equations is analyzed and a new bound of the convergence rate is obtained from a new perspective.