Algorithm 937: MINRES-QLP for Symmetric and Hermitian Linear Equations and Least-Squares Problems

Abstract

We describe algorithm MINRES-QLP and its FORTRAN 90 implementation for solving symmetric or Hermitian linear systems or least-squares problems. If the system is singular, MINRES-QLP computes the unique minimum-length solution (also known as the pseudoinverse solution), which generally eludes MINRES. In all cases, it overcomes a potential instability in the original MINRES algorithm. A positive-definite pre-conditioner may be supplied. Our FORTRAN 90 implementation illustrates a design pattern that allows users to make problem data known to the solver but hidden and secure from other program units. In particular, we circumvent the need for reverse communication. Example test programs input and solve real or complex problems specified in Matrix Market format. While we focus here on a FORTRAN 90 implementation, we also provide and maintain MATLAB versions of MINRES and MINRES-QLP.

1. INTRODUCTION

MINRES-QLP [Choi 2006; Choi et al. 2011] is a Krylov subspace method for computing the minimum-length and minimum-residual solution (also known as the pseudoinverse solution) x to the following linear systems or least-squares (LS) problems:

$$\text{solve}Ax=b,$$ (1)

$$\text{minimize}{\Vert x\Vert }_2\text{such that}Ax=b,$$ (2)

$$\text{minimize}{\Vert x\Vert }_2\text{such that}x\in \text{arg}\underset{x}{\text{min}}{\Vert Ax-b\Vert }_2,$$ (3)

where A is an n × n symmetric or Hermitian matrix and b is a real or complex n-vector. Problems (1) and (2) are treated as special cases of (3). The matrix A is usually large and sparse, and it may be singular.¹ It is defined by means of a user-written subroutine Aprod, whose function is to compute the product y = Aυ for any given vector υ.

Let x_k be the solution estimate associated with MINRES-QLP’s kth iteration, with residual vector r_k = b − Ax_k. Without loss of generality, we define x₀ = 0. MINRES-QLP provides recurrent estimates of ‖x_k‖, ‖r_k‖, ‖Ar_k‖, ‖A‖, cond(A), and ‖Ax_k‖, which are used in the stopping conditions.

Other iterative methods specialized for symmetric systems Ax = b are the conjugate-gradient method (CG) [Hestenes and Stiefel 1952], SYMMLQ and MINRES [Paige and Saunders 1975], and SQMR [Freund and Nachtigal 1994]. Each method requires one product Aυ_k at each iteration for some vector υ_k. CG is intended for positive-definite A, whereas the other solvers allow A to be indefinite.

If A is singular, SYMMLQ requires the system to be consistent, whereas MINRES returns an LS solution for (3) but generally not the min-length solution; see [Choi 2006; Choi et al. 2011] for examples. SQMR without preconditioning is mathematically equivalent to MINRES but could fail on a singular problem. To date, MINRES-QLP is probably the most suitable CG-type method for solving (3).

In some cases, the more established symmetric methods may still be preferable.

1. If A is positive definite, CG minimizes the energy norm of the error ‖x − x_k‖_A in each Krylov subspace and requires slightly less work per iteration. However, CG, MINRES, and MINRES-QLP do reduce ‖x − x_k‖_A and ‖x − x_k‖ monotonically. Also, MINRES and MINRES-QLP often reduce ‖r_k‖ to the desired level significantly sooner than does CG, and the backward error for each x_k decreases monotonically. (See Section 2.4 and Fong [2011] and Fong and Saunders [2012].)

2. If A is indefinite but Ax = b is consistent (e.g., if A is nonsingular), SYMMLQ requires slightly less work per iteration, and it reduces the error norm ‖x − x_k‖ monotonically. MINRES and MINRES-QLP usually reduce ‖x −x_k‖ [Fong 2011; Fong and Saunders 2012].

3. If A is indefinite and well conditioned and Ax = b is consistent, MINRES might be preferable to MINRES-QLP because it requires the same number of iterations but slightly less work per iteration.

4. MINRES and MINRES-QLP require a preconditioner to be positive definite. SQMR might be preferred if A is indefinite and an effective indefinite preconditioner is available.

MINRES-QLP has two phases. Iterations start in the MINRES phase and transfer to the MINRES-QLP phase when a subproblem (see (8)) becomes ill-conditioned by a certain measure. If every subproblem is of full rank and well-conditioned, the problem can be solved entirely in the MINRES phase, where the cost per iteration is essentially the same as for MINRES. In the MINRES-QLP phase, one more work vector and 5n more multiplications are used per iteration.

MINRES-QLP described here is implemented in FORTRAN 90 for real double-precision problems. It contains no machine-dependent constants and does not use any features from later standards. It requires an auxiliary subroutine Aprod and, if a preconditioner is supplied, a second subroutine Msolve. We also provide a complex implementation for Hermitian problems. Precision other than double can be obtained by changing one line of code.

We also maintain a MATLAB implementation capable of solving both real and complex problems. All implementations are available at [SOL].

Table I lists the main notation used.

Table I.

Key Notation

‖ · ‖	matrix or vector two-norm
Ā	Ā = A − σ I (see also σ below)
cond(A)	condition number of A with respect to two-norm = $\frac{\text{max}|{\lambda }_i|}{{\text{min}}_{\lambda i\ne 0}|{\lambda }_i|}$
ei	ith unit vector
ℓ	index of the last Lanczos iteration when βℓ+1 = 0
n	order of A
null(A)	null space of A defined as {x ∈ ℝn | Ax = 0}
range(A)	column space of A defined as {Ax | x ∈ ℝn}
T	(right superscript to a vector or a matrix) transpose
x†	unique minimum-length least-squares solution of problem (3)
𝒦k(A, b)	kth Krylov subspace defined as span{b, Ab,…, Ak−1b}
ε	machine precision
σ	real scalar shift to diagonal of A

1.1. Least-Squares Methods

Further existing methods that could be applied to (3) are CGLS and LSQR [Paige and Saunders 1982a, 1982b], LSMR [Fong and Saunders 2011], and GMRES [Saad and Schultz 1986], all of which reduce ‖r_k‖ monotonically. The first three methods would require two products Aυ_k and Au_k each iteration and would be generating points in less favorable subspaces. GMRES requires only products Aυ_k and could use any nonsingular (possibly indefinite) preconditioner. It needs increasing storage and work each iteration, perhaps requiring restarts, but it could be more effective than MINRES or MINRES-QLP (and the other solvers) if few total iterations were required. Table II summarizes the computational requirements of each method.

Table II.

Comparison of Various Least-Squares Solvers on n × n Systems (3)

Solver	Storage	Work per Iteration	Products per Iteration	Systems to Solve per Iteration with Preconditioner
MINRES	7n	9n	1	1
MINRES-QLP	7n – 8n	9n – 14n	1	1
GMRES(m)	(m + 2)n	(m + 3 + 1/m)n	1	1
CGLS	4n	5n	2	2
LSQR	5n	8n	2	2
LSMR	6n	9n	2	2

Storage refers to memory required by working vectors in the solvers. Work counts number of floating-point multiplications. On inconsistent systems, all solvers below except MINRES and GMRES with restart parameter m return the minimum-length LS solution (assuming no preconditioner).

1.2. Regularization

We do not discourage using CGLS, LSQR, or LSMR if the goal is to regularize an ill-posed problem using a small damping factor λ > 0 as follows:

$$\underset{x}{\text{min}}\Vert \left[\begin{array}{c}\hfill A\hfill \\ \hfill \lambda I\hfill \end{array}\right]x-\left[\begin{array}{c}\hfill b\hfill \\ \hfill 0\hfill \end{array}\right]\Vert .$$ (4)

However, this approach destroys the original problem’s symmetry. The normal equation of (4) is (A² + λ² I)x = Ab, which suggests that a diagonal shift to A may well serve the same purpose in some cases. For symmetric positive-definite A, Ā = A − σ I with σ < 0 enjoys a smaller condition number. When A is indefinite, a good choice of σ may not exist, for example, if the eigenvalues of A were symmetrically positioned around zero. When this symmetric form is applicable, it is convenient in MINRES and MINRES-QLP; see (3), (5), and (15). We also remark that MINRES and MINRES-QLP produce good estimates of the largest and smallest singular values of Ā (via diagonal values of R_k or L_k in (7) and (11); see Choi et al. [2011, Section 4]).

Three other regularization tools in the literature (see Golub and Van Loan [1996, Sections 12.1.1–12.1.3] and Hansen [1998]) are LSQI, cross-validation, and L-curve. LSQI involves solving a nonlinear equation and is not immediately compatible with the Lanczos framework. Cross-validation takes one row out at a time and thus does not preserve symmetry. The L-curve approach for a CG-type method takes iteration k as the regularization parameter [Hansen 1998, Chap. 8] if both ‖r_k‖ and ‖x_k‖ are monotonic. By design, ‖r_k‖ is monotonic in MINRES and MINRES-QLP, and so is ‖x_k‖ when Ā is positive definite [Fong 2011]. Otherwise, we prefer the condition L-curve approach in Calvetti et al. [2000], which graphs cond(T_k) against ‖r_k‖. Yet another L-curve feasible in MINRES-QLP is $\Vert x_{k-2}^{(2)}\Vert$ against ‖r_k‖, since the former is also monotonic (but available two iterations in lag); see Section 2.4.

2. MATHEMATICAL BACKGROUND

Notation and details of algorithmic development from [Choi 2006; Choi et al. 2011] are summarized here. As noted earlier, “A” in (1)–(3) is treated as A− σ I.

2.1. Lanczos Process

MINRES and MINRES-QLP use the symmetric Lanczos process [Lanczos 1950] to reduce A to a tridiagonal form $\underset{¯}{T_k}$. The process is initialized with υ₀ ≡ 0, β₁ = ‖b‖, and β₁υ₁ = b. After k steps of the tridiagonalization, we have produced

$$p_k=A{\upsilon }_k-\sigma {\upsilon }_k,{\alpha }_k={\upsilon }_k^T{p}_k,{\beta }_{k+1}{\upsilon }_{k+1}=p_k-{\alpha }_k{\upsilon }_k-{\beta }_k{\upsilon }_{k-1},$$ (5)

where we choose β_k > 0 to give ‖υ_k‖ = 1. Numerically,

$$p_k=A{\upsilon }_k-\sigma {\upsilon }_k-{\beta }_k{\upsilon }_{k-1},{\alpha }_k={\upsilon }_k^T{p}_k,{\beta }_{k+1}{\upsilon }_{k+1}=p_k-{\alpha }_k{\upsilon }_k$$

is slightly better than (5) [Paige 1976], but we can express (5) in matrix form:

$$V_k\equiv [{\upsilon }_1\dots {\upsilon }_k],A{V}_k=V_{k+1}\underset{¯}{T_k},\underset{¯}{T_k}\equiv \left[\begin{array}{c}\hfill T_k\hfill \\ \hfill {\beta }_{k+1}{e}_k^T\hfill \end{array}\right],$$ (6)

where T_k = tridiag(β_i, α_i, β_i+1), i = 1, …, k. In exact arithmetic, the Lanczos vectors in the columns of V_k are orthonormal, and the process stops with k = ℓ when β_ℓ+1 = 0 for some ℓ ≤ n, and then AV_ℓ = V_ℓT_ℓ. The rank of T_ℓ could be ℓ or ℓ − 1 (see Theorem 2.2).

2.2. MINRES Phase

MINRES-QLP typically starts with a MINRES phase, which applies a series of reflectors Q_k to transform $\underset{¯}{T_k}$ to an upper triangular matrix $\underset{¯}{R_k}$:

$$Q_k[\underset{¯}{T_k}{\beta }_1{e}_1]=\left[\begin{array}{cc}\hfill R_k\hfill & \hfill t_k\hfill \\ \hfill 0\hfill & \hfill {\varphi }_k\hfill \end{array}\right]\equiv [\underset{¯}{R_k}{\mathrm{t̄}}_{k+1}],$$ (7)

where

$$Q_k=Q_{k,k+1}\left[\begin{array}{cc}\hfill Q_{k-1}\hfill & \hfill \hfill \\ \hfill \hfill & \hfill 1\hfill \end{array}\right],Q_{k,k+1}\equiv \left[\begin{array}{ccc}\hfill I_{k-1}\hfill & \hfill \hfill & \hfill \hfill \\ \hfill \hfill & \hfill c_k\hfill & \hfill s_k\hfill \\ \hfill \hfill & \hfill s_k\hfill & \hfill -c_k\hfill \end{array}\right].$$

In the kth step, Q_k,k+1 is effectively a Householder reflector of dimension 2 [Trefethen and Bau 1997, Exercise 10.4], and its action including its effect on later columns of T_j, k < j ≤ ℓ, is compactly described by

$$\left[\begin{array}{cc}\hfill c_k\hfill & \hfill s_k\hfill \\ \hfill s_k\hfill & \hfill -c_k\hfill \end{array}\right]\left[\begin{array}{ccc}\hfill {\gamma }_k\hfill & \hfill {\delta }_{k+1}\hfill & \hfill 0\hfill \\ \hfill {\beta }_{k+1}\hfill & \hfill {\alpha }_{k+1}\hfill & \hfill {\beta }_{k+2}\hfill \end{array}\right|\begin{array}{c}\hfill {\varphi }_{k-1}\hfill \\ \hfill 0\hfill \end{array}]=[\begin{array}{ccc}\hfill {\gamma }_k^{(2)}\hfill & \hfill {\delta }_{k+1}^{(2)}\hfill & \hfill {\epsilon }_{k+2}\hfill \\ \hfill 0\hfill & \hfill {\gamma }_{k+1}\hfill & \hfill {\delta }_{k+2}\hfill \end{array}\left|\begin{array}{c}\hfill {\tau }_k\hfill \\ \hfill {\varphi }_k\hfill \end{array}\right],$$

where the superscripts with numbers in parentheses indicate the number of times the values have been modified. The kth solution approximation to (3) is then defined to be x_k = V_ky_k, where y_k solves the subproblem

$$y_k=\text{arg}\underset{y\in {ℝ}^k}{\text{min}}\Vert \underset{¯}{T_k}y-{\beta }_1{e}_1\Vert =\text{arg}\underset{y\in {ℝ}^k}{\text{min}}\Vert \underset{¯}{R_k}y-{\mathrm{t̄}}_{k+1}\Vert .$$ (8)

When k < ℓ, R_k is nonsingular and the unique solution of this subproblem satisfies R_ky_k = t_k. Instead of solving for y_k, MINRES solves $R_k^T{D}_k^T=V_k^T$ by forward substitution, obtaining the last column d_k of D_k at iteration k. At the same time, it updates x_k ∈ 𝒦_k(A, b) (see Table I for definition) via x₀ ≡ 0 and

$$x_k=V_k{y}_k=D_k{R}_k{y}_k=D_k{t}_k=x_{k-1}+{\tau }_k{d}_k,{\tau }_k\equiv e_k^T{t}_k,$$ (9)

where one can show using V_k = D_kR_k that $d_k=({\upsilon }_k-{\delta }_k^{(2)}{d}_{k-1}-{\epsilon }_k{d}_{k-2})/{\gamma }_k^{(2)}$.

2.3. MINRES-QLP Phase

The MINRES phase transfers to the MINRES-QLP phase when an estimate of the condition number of A exceeds an input parameter trancond. Thus, trancond > 1/ε leads to MINRES iterates throughout (where ε ≈ 10⁻¹⁶ denotes the floating-point precision), whereas trancond = 1 generates MINRES-QLP iterates from the start.

Suppose for now that there is no MINRES phase. Then MINRES-QLP applies left reflections as in (7) and a further series of right reflections to transform R_k to a lower triangular matrix L_k = R_kP_k, where

$$P_k=P_{1,2}{P}_{1,3}{P}_{2,3}\cdots P_{k-2,k}{P}_{k-1,k},P_{k-2,k}=\left[\begin{array}{cccc}\hfill I_{k-3}\hfill & \hfill \hfill & \hfill \hfill & \hfill \hfill \\ \hfill \hfill & \hfill c_{k2}\hfill & \hfill \hfill & \hfill s_{k2}\hfill \\ \hfill \hfill & \hfill \hfill & \hfill 1\hfill & \hfill \hfill \\ \hfill \hfill & \hfill s_{k2}\hfill & \hfill \hfill & \hfill -c_{k2}\hfill \end{array}\right],P_{k-1,k}=\left[\begin{array}{ccc}\hfill I_{k-2}\hfill & \hfill \hfill & \hfill \hfill \\ \hfill \hfill & \hfill c_{k3}\hfill & \hfill s_{k3}\hfill \\ \hfill \hfill & \hfill s_{k3}\hfill & \hfill -c_{k3}\hfill \end{array}\right].$$

In the kth step, the actions of P_k−2,k and P_k−1,k are compactly described by

$$\left[\begin{array}{ccc}{\gamma }_{k-2}^{(5)}\hfill & \hfill & \hfill {\epsilon }_k\hfill \\ {\vartheta }_{k-1}\hfill & {\gamma }_{k-1}^{(4)}\hfill & \hfill {\delta }_k^{(2)}\hfill \\ \hfill & \hfill & \hfill {\gamma }_k^{(2)}\hfill \end{array}\right]\left[\begin{array}{ccc}\hfill c_{k2}\hfill & \hfill \hfill & \hfill s_{k2}\hfill \\ \hfill \hfill & \hfill 1\hfill & \hfill \hfill \\ \hfill s_{k2}\hfill & \hfill \hfill & \hfill -c_{k2}\hfill \end{array}\right]\left[\begin{array}{ccc}\hfill 1\hfill & \hfill \hfill & \hfill \hfill \\ \hfill \hfill & \hfill c_{k3}\hfill & \hfill s_{k3}\hfill \\ \hfill \hfill & \hfill s_{k3}\hfill & \hfill -c_{k3}\hfill \end{array}\right]=\left[\begin{array}{ccc}\hfill {\gamma }_{k-2}^{(6)}\hfill & \hfill \hfill & \hfill \hfill \\ \hfill {\vartheta }_{k-1}^{(2)}\hfill & \hfill {\gamma }_{k-1}^{(4)}\hfill & \hfill {\delta }_k^{(3)}\hfill \\ \hfill {\eta }_k\hfill & \hfill \hfill & \hfill {\gamma }_k^{(3)}\hfill \end{array}\right]\left[\begin{array}{ccc}\hfill 1\hfill & \hfill \hfill & \hfill \hfill \\ \hfill \hfill & \hfill c_{k3}\hfill & \hfill s_{k3}\hfill \\ \hfill \hfill & \hfill s_{k3}\hfill & \hfill -c_{k3}\hfill \end{array}\right]=\left[\begin{array}{ccc}\hfill {\gamma }_{k-2}^{(6)}\hfill & \hfill \hfill & \hfill \hfill \\ \hfill {\vartheta }_{k-1}^{(2)}\hfill & \hfill {\gamma }_{k-1}^{(5)}\hfill & \hfill \hfill \\ \hfill {\eta }_k\hfill & \hfill {\vartheta }_k\hfill & \hfill {\gamma }_k^{(4)}\hfill \end{array}\right].$$ (10)

The kth approximate solution to (3) is then defined to be x_k = V_ky_k = V_kP_ku_k = W_ku_k, where u_k solves the subproblem

$$u_k\equiv \text{arg}\underset{u}{\text{min}}\Vert u\Vert \mathrm{s}.\mathrm{t}.u\in \text{arg}\underset{u\in {ℝ}^k}{\text{min}}\Vert \left[\begin{array}{c}\hfill L_k\hfill \\ \hfill 0\hfill \end{array}\right]u-\left[\begin{array}{c}\hfill t_k\hfill \\ \hfill {\varphi }_k\hfill \end{array}\right]\Vert .$$ (11)

For k < ℓ, R_k and L_k are nonsingular because $\underset{¯}{T_k}$ has full column rank by Lemma 2.1 below. It is only when k = ℓ and b ∉ range(A) that R_k and L_k are singular with rank ℓ − 1 by Theorem 2.2, in which case one can show that ${\eta }_k={\gamma }_k^{(3)}={\vartheta }_k={\gamma }_k^{(4)}=0$ in (10) and $L_{\ell }=[\begin{array}{cc}\hfill L_{\ell -1}\hfill & \hfill 0\hfill \\ \hfill 0\hfill & \hfill 0\hfill \end{array}]$ with L_ℓ−1 nonsingular. In any case, we need to solve only the nonsingular lower triangular systems L_ku_k = t_k or L_ℓ−1u_ℓ−1 = t_ℓ−1. Then, u_k and y_k = P_ku_k are the min-length solutions of (11) and (8), respectively.

MINRES-QLP updates x_k−2 to obtain x_k by short-recurrence orthogonal steps:

$$x_{k-2}^{(2)}=x_{k-3}^{(2)}+{\mu }_{k-2}^{(3)}{w}_{k-2}^{(4)},\text{where}{x}_{k-3}^{(2)}\equiv W_{k-3}^{(4)}{\mu }_{k-3}^{(3)},$$ (12)

$$x_k=x_{k-2}^{(2)}+{\mu }_{k-1}^{(2)}{w}_{k-1}^{(3)}+{\mu }_k{w}_k^{(2)},$$ (13)

where w_j refers to the jth column of W_k = V_kP_k and μ_i is the ith element of u_k.

If this phase is preceded by a MINRES phase of k iterations (0 < k < ℓ), it starts by transferring the last three vectors d_k−2, d_k−1, d_k to w_k−2, w_k−1, w_k, and the solution estimate x_k from (9) to $x_{k-2}^{(2)}$ in (12). This needs the last two rows of L_ku_k = t_k (to give μ_k−1, μ_k) and the relations W_k = D_kL_k and $x_{k-2}^{(2)}=x_k-{\mu }_{k-1}{w}_{k-1}-{\mu }_k{w}_k$. The cheaply available right reflections P_k and the bottom right 3 × 3 submatrix of L_k (i.e., the last term in (10)) need to have been saved in the MINRES phase in order to facilitate the transfer.

2.4. Norm Estimates and Stopping Conditions

Short-term recurrences are used to estimate the following quantities (where we assume σ = 0 for simplicity):

$$\begin{array}{ccc}\hfill \Vert r_k\Vert \approx {\varphi }_k={\varphi }_{k-1}{s}_k,\hfill & \hfill {\varphi }_0=\Vert b\Vert & \hfill ({\varphi }_k\searrow )\\ \hfill \Vert A{r}_k\Vert \approx {\psi }_k={\varphi }_k\Vert [{\gamma }_{k+1}{\delta }_{k+2}]\Vert ,\hfill & \hfill & \hfill ({\psi }_{\ell }=0)\\ \hfill \Vert x_k^{(2)}\Vert \approx {\chi }_{k-2}^{(2)}=\Vert [{\chi }_{k-3}^{(2)}{\mu }_{k-2}^{(3)}]\Vert ,\hfill & \hfill {\chi }_{-2}={\chi }_{-1}=0& \hfill ({\chi }_{k-2}^{(2)}\nearrow )\\ \hfill \Vert x_k\Vert \approx {\chi }_k=\Vert [{\chi }_{k-2}^{(2)}{\mu }_{k-1}^{(2)}{\mu }_k]\Vert ,\hfill & \hfill {\chi }_0=0& \hfill ({\chi }_{\ell }=\Vert x^{†}\Vert )\\ \hfill \Vert A{x}_k\Vert \approx {\omega }_k=\Vert [{\omega }_{k-1}{\tau }_k]\Vert ,\hfill & \hfill {\omega }_0=0& \hfill ({\omega }_k\nearrow )\\ \hfill \Vert A\Vert \approx {𝒜}_k=\text{max}\{{𝒜}_{k-1},\Vert \underset{¯}{T_k}{e}_k\Vert ,{\mathrm{\gamma ̅}}_k\},\hfill & \hfill {𝒜}_0=0& \hfill ({𝒜}_k\nearrow \Vert A\Vert )\\ \hfill \text{cond}(A)\approx {\kappa }_k={𝒜}_k/{\mathrm{\gamma ̲}}_k,\hfill & \hfill {\kappa }_0=1& \hfill ({\kappa }_k\nearrow \text{cond}(A)),\end{array}$$

where γ̅_k and γ̲_k are the largest and smallest diagonals of L_k in absolute value. The up (down) arrows in parentheses indicate that the associated quantities increase (decrease) monotonically. The last two estimates tend to their targets from below; see Choi [2006] and Choi et al. [2011] for derivation.

MINRES-QLP has 14 possible stopping conditions in five classes that use the preceding estimates and optional user-input parameters itnlim, rtol, Acondlim, and maxxnorm:

• (C1)
  from Lanczos and the QLP factorization:

  • k = itnlim; β_k+1 < ε; $|{\gamma }_k^{(4)}|<\epsilon$
• (C2)
  normwise relative backward errors (NRBE) [Paige and Strakoš 2002]:

  • ‖r_k‖/ (‖A‖ ‖x_k‖ + ‖b‖) ≤ max(rtol, ε); ‖Ar_k‖/ (‖A‖ ‖r_k‖) ≤ max(rtol, ε);
• (C3)
  regularization attempts:

  • cond(A) ≥ min(Acondlim, 0.1/ε); ‖x_k‖ ≥ maxxnorm;
• (C4)
  degenerate cases:

  • β₁ = 0 ⇒ b = 0 ⇒ x = 0 is the solution;
  • β₂ = 0 ⇒ υ₂ = 0 ⇒ Ab = α₁b,

    • i.e., b and α₁ are an eigenpair of A, and x = b/α₁ solves Ax = b;
• (C5)
  erroneous inputs:

  • A not symmetric; M not symmetric; M not positive definite;

where M is a preconditioner to be described in the next section. For symmetry of A, it is not practical to check $e_i^{T}A{e}_j=e_j^{T}A{e}_i$ for all i, j = 1, …, n. Instead, we statistically test whether z = |x^T(Ay)−y^T(Ax)| is sufficiently small for two nonzero n-vectors x and y (e.g., each element in the vectors is drawn from the standard normal distribution). For positive definiteness of M, since M is positive definite if and only if M⁻¹ is positive definite, we simply test that $z_k^T{M}^{-1}{z}_k=z_k^T{q}_k>0$ each iteration (see Section 3).

We find that the recurrence relations for ϕ_k and ψ_k hold to high accuracy. Thus x_k is an acceptable solution of (3) if the computed value of ϕ_k or ψ_k is suitably small according to the NRBE tests in class (C2) above. When a condition in (C3) is met, the final x_k may or may not be an acceptable solution.

The class (C1) tests for small β_k+1 and ${\gamma }_k^{(4)}$ are included in the unlikely case in practice that the theoretical Lanczos termination occurs. Ideally, one of the NRBE tests should cause MINRES-QLP to terminate. If not, it is an indication that the problem is very ill-conditioned, in which case the regularization and preconditioning techniques of Sections 1.2 and 3 may be helpful.

2.5. Two Theorems

We complete this section by presenting two theorems from Choi et al. [2011] with slightly simpler proofs.

Lemma 2.1. $\text{rank}(\underset{¯}{T_k})=k$ for all 1 ≤ k < ℓ.

Proof. For 1 ≤ k < ℓ, we have β₂, …, β_k+1 > 0 by definition. Hence, $\underset{¯}{T_k}$ has full column rank.

Theorem 2.2. T_ℓ is nonsingular if and only if b ∈ range(A). Furthermore, rank(T_ℓ) = ℓ − 1 if b ∉ range(A).

Proof. We use AV_ℓ = V_ℓT_ℓ twice. First, if T_ℓ is nonsingular, we can solve T_ℓy_ℓ = β₁e₁ and then AV_ℓy_ℓ = V_ℓT_ℓy_ℓ = V_ℓβ₁e₁ = b. Conversely, if b ∈ range(A), then range(V_ℓ) ⊆ range(A). Suppose T_ℓ is singular. Then there exists z ≠ 0 such that V_ℓT_ℓz = AV_ℓz = 0. That is, 0 ≠ V_ℓz ∈ null(A). But this is impossible because V_ℓz ∈ range(A) and null(A) ∩ range(V_ℓ) = 0. Thus, T_ℓ must be nonsingular.

We have shown that if b ∉ range(A), $T_{\ell }=[\begin{array}{cc}\hfill \underset{¯}{T_{\ell -1}}\hfill & \hfill \begin{array}{c}\hfill {\beta }_{\ell }{e}_{\ell -1}\hfill \\ \hfill {\alpha }_{\ell }\hfill \end{array}\hfill \end{array}]$ is singular, and therefore ℓ > rank(T_ℓ) ≥ $\text{rank}(\underset{¯}{T_{\ell -1}})=\ell -1$ by Lemma 2.1. Therefore, rank(T_ℓ) = ℓ − 1.

By Lemma 2.1 and Theorem 2.2, we are assured that the QLP decomposition without column pivoting [Stewart 1999; Choi et al. 2011] for $\underset{¯}{T_k}$ is rank-revealing, which is a necessary precondition for solving a least-squares problem.

Theorem 2.3. In MINRES-QLP, x_ℓ is the minimum-length solution of (3).

Proof. y_ℓ comes from the min-length LS solution of T_ℓy_ℓ ≈ β₁e₁ and thus satisfies the normal equation $T_{\ell }^2{y}_{\ell }=T_{\ell }{\beta }_1{e}_1$ and y_ℓ ∈ range(T_ℓ). Now x_ℓ = V_ℓy_ℓ and Ax_ℓ = AV_ℓy_ℓ = V_ℓT_ℓy_ℓ. Hence, $A^2{x}_{\ell }=A{V}_{\ell }{T}_{\ell }{y}_{\ell }=V_{\ell }{T}_{\ell }^2{y}_{\ell }=V_{\ell }{T}_{\ell }{\beta }_1{e}_1=Ab$. Thus, x_ℓ is an LS solution of (3). Since y_ℓ ∈ range(T_ℓ), y_ℓ = T_ℓz for some z, and so x_ℓ = V_ℓy_ℓ = V_ℓT_ℓz = AV_ℓz ∈ range(A) is the min-length LS solution of (3).

3. PRECONDITIONING

Iterative methods can be accelerated if preconditioners are available and well-chosen. For MINRES-QLP, we want to choose a symmetric positive-definite matrix M to solve a nonsingular system (1) by implicitly solving an equivalent symmetric consistent system $M^{-\frac{1}{2}}A{M}^{-\frac{1}{2}}\mathrm{x̄}=\mathrm{b̄}$, where $M^{\frac{1}{2}}x=\mathrm{x̄},\mathrm{b̄}=M^{-\frac{1}{2}}b$, and $\text{cond}(M^{-\frac{1}{2}}A{M}^{-\frac{1}{2}})\ll \text{cond}(A)$. This two-sided preconditioning preserves symmetry. Thus, we can derive preconditioned MINRES-QLP by applying MINRES-QLP to the equivalent problem and setting $x=M^{-\frac{1}{2}}\mathrm{x̄}$.

With preconditioned MINRES-QLP, we can solve a singular consistent system (2), but we will obtain a least-squares solution that is not necessarily the minimum-length solution (unless M = I). For inconsistent systems (3), preconditioning alters the least-squares norm to ‖ · ‖_M⁻¹, and the solution is of minimum length in the new norm space. We refer readers to Choi et al. [2011, Sect. 7] for a detailed discussion of various approaches to preserving the two-norm “minimum length.”

To derive MINRES-QLP, we define

$$z_k={\beta }_k{M}^{\frac{1}{2}}{\upsilon }_k,q_k={\beta }_k{M}^{-\frac{1}{2}}{\upsilon }_k,\text{so that}M{q}_k=z_k.$$ (14)

Then, ${\beta }_k=\Vert {\beta }_k{\upsilon }_k\Vert =\Vert M^{-\frac{1}{2}}{z}_k\Vert ={\Vert z_k\Vert }_{M^{-1}}={\Vert q_k\Vert }_M=\sqrt{q_k^T{z}_k}$, where the square root is well defined because M is positive definite, and the following expressions replace the quantities in (5) in the Lanczos iterations:

$$p_k=A{q}_k-\sigma q_k,{\alpha }_k=\frac{1}{{\beta }_k^2}{q}_k^T{p}_k,z_{k+1}=\frac{1}{{\beta }_k}{p}_k-\frac{{\alpha }_k}{{\beta }_k}{z}_k-\frac{{\beta }_k}{{\beta }_{k-1}}{z}_{k-1}.$$ (15)

We also need to solve the system Mq_k = z_k in (14) at each iteration.

In the MINRES phase, we define ${\mathrm{d̄}}_k=M^{-\frac{1}{2}}{d}_k$ and update the solution of the original problem (1) by

$${\mathrm{d̄}}_k=(\frac{1}{{\beta }_k}{q}_k-{\delta }_k^{(2)}{\mathrm{d̄}}_{k-1}-{\epsilon }_k{\mathrm{d̄}}_{k-2})/{\gamma }_k^{(2)},x_k=M^{-\frac{1}{2}}{\mathrm{x̄}}_k=x_{k-1}+{\tau }_k{\mathrm{d̄}}_k.$$

In the MINRES-QLP phase, we define ${\mathrm{W̅}}_k\equiv M^{-\frac{1}{2}}{W}_k=(M^{-\frac{1}{2}}{V}_k)P_k$ and update the solution estimate of problem (1) by orthogonal steps:

$$\begin{array}{cc}\hfill {\mathrm{w̄}}_k=-(c_{k2}/{\beta }_k)q_k+s_{k2}{\mathrm{w̄}}_{k-2}^{(3)},\hfill & \hfill {\mathrm{w̄}}_{k-2}^{(4)}=(s_{k2}/{\beta }_k)q_k+c_{k2}{\mathrm{w̄}}_{k-2}^{(3)},\hfill \\ \hfill {\mathrm{w̄}}_k^{(2)}=s_{k3}{\mathrm{w̄}}_{k-1}^{(2)}-c_{k3}{\mathrm{w̄}}_k,\hfill & \hfill {\mathrm{w̄}}_{k-1}^{(3)}=c_{k3}{\mathrm{w̄}}_{k-1}^{(2)}+s_{k3}{\mathrm{w̄}}_k,\hfill \\ \hfill x_{k-2}^{(2)}=x_{k-3}^{(2)}+{\mu }_{k-2}^{(3)}{\mathrm{w̄}}_{k-2}^{(4)},\hfill & \hfill x_k=x_{k-2}^{(2)}+{\mu }_{k-1}^{(2)}{\mathrm{w̄}}_{k-1}^{(3)}+{\mu }_k{\mathrm{w̄}}_k^{(2)}.\hfill \end{array}$$

Let ${\mathrm{r̄}}_k=\mathrm{b̄}-M^{-\frac{1}{2}}(A-\sigma I)M^{-\frac{1}{2}}{\mathrm{x̄}}_k=M^{-\frac{1}{2}}{r}_k$. Then, $x_k=M^{-\frac{1}{2}}{\mathrm{x̄}}_k$ is an acceptable solution of (1) if the computed value of ϕ_k ≈ ‖r̄_k‖ = ‖r_k‖_M⁻¹ is sufficiently small.

We can now present our pseudocode in Algorithm 1. The reflectors are implemented in Algorithm 2 SymOrtho(a, b) for real a and b, which is a stable form for computing $r=\sqrt{a^2+b^2}\ge 0,c=\frac{a}{r}$, and $s=\frac{b}{r}$. The complexity is at most 6 flops and a square root. Algorithm 1 lists all steps of MINRES-QLP with preconditioning. For simplicity, w̄_k is written as w_k for all relevant k. Also, the output x solves (A − σ I)x ≈ b, but other outputs are associated with the preconditioned system.

4. IMPLEMENTATION AND TESTING

A detailed description of the contents and structure of the Fortran 90 package implementing MINRES-QLP for real and complex data may be found in the user manual that accompanies the software (also available as Choi and Saunders [2012]).

Our design spares users from implementing reverse communication, in which the solver would return control to the calling program whenever Aprod or Msolve were to be invoked. (While reverse communication is widely used in scientific computing with FORTRAN 77, the resulting code usually appears formidable and unrecognizable from the original pseudocode; see Dongarra et al. [1995] and Oliveira and Stewart [2006] for two examples of CG and numerical integration coded in FORTRAN 77 and 90, respectively.) Our MINRES-QLP implementation achieves the purpose of reverse communication while preserving code readability and thus maintainability. The FORTRAN 90 module structure allows a user’s Ax products and Mx = y solves to be implemented outside MINRES-QLP in the same way that MATLAB’s function handles operate.

In our development of FORTRAN 90 MINRES-QLP, we have created a suite of 117 test cases including singular matrices representative of real-world applications [Foster 2009; Davis and Hu 2011]. The test program outputs results to MINRESQLP.txt. If users need to modify subroutine MINRESQLP, they can run these test cases and search for the word “ appear” in the output file to check whether all tests are reported to be successful. For more sophisticated unit testing frameworks employed in large-scale scientific software development, see O’Boyle et al. [2008]. Further details of the numerical examples are given in Choi and Saunders [2012].

Finally, note that the choice of parameter values can have a critical effect on the convergence of iterative solvers. While the default parameter values in MINRES-QLP work well in most tests, they may need to be fine-tuned by trial and error, and for some applications it may be worthwhile to implement full or partial reorthogonalization of the Lanczos vectors [Simon 1984].

ALGORITHM 1.

Pseudocode of preconditioned MINRES-QLP for solving (A − σ I)x ≈ b. In the right-justified comments, $Ã\equiv M^{-\frac{1}{2}}(A-\sigma I)M^{-\frac{1}{2}}$.

	input: A, b, σ, M
1	z0 = 0, z1 = b, Solve Mq1 = z1, ${\beta }_1=\sqrt{b^T{q}_1,}{\varphi }_0={\beta }_1$		[Initialize]
2	w0 = w−1 = 0, x−2 = x−1 = x0 = 0
3	c0, 1 = c0, 2 = c0, 3 = −1, s0, 1 = s0, 2 = s0, 3 = 0, τ0 = ω0 = χ−2 = χ−1 = χ0 = 0
4	κ0 = 1, 𝒜0 = δ1 = γ−1 = γ0 = η−1 = η0 = η1 = ϑ−1 = ϑ0 = ϑ1 = μ−1 = μ0 = 0
5	k = 0
6	while no stopping condition is satisfied do
7		k ← k + 1
8		pk = Aqk − σqk, ${\alpha }_k=\frac{1}{{\beta }_k^2}{q}_k^T{p}_k$	[Preconditioned Lanczos]
9		$z_{k+1}=\frac{1}{{\beta }_k}{p}_k-\frac{{\alpha }_k}{{\beta }_k}{z}_k-\frac{{\beta }_k}{{\beta }_{k-1}}{z}_{k-1}$
10		Solve Mqk+1 = zk+1, ${\beta }_{k+1}=\sqrt{q_{k+1}^T{z}_{k+1}}$
11		if k = 1 then ρk = ‖ [αk βk+1] ‖ else ρk = ‖ [βk αk βk+1] ‖
12		${\delta }_k^{(2)}=c_{k-1,1}{\delta }_k+s_{k-1,1}{\alpha }_k$	[Previous left reflection…]
13		γk = sk−1, 1δk − ck−1, 1αk	[on middle two entries of $\underset{¯}{T_k}$ek…]
14		εk+1 = sk−1, 1βk+1	[produces first two entries in $\underset{¯}{T_{k+1}}$ek+1]
15		δk+1 = −ck−1, 1βk+1
16		$c_{k1},s_{k1},{\gamma }_k^{(2)}\leftarrow \text{SymOrtho}({\gamma }_k,{\beta }_{k+1})$	[Current left reflection]
17		$c_{k2},s_{k2},{\gamma }_{k-2}^{(6)}\leftarrow \text{SymOrtho}({\gamma }_{k-2}^{(5)},{\epsilon }_k)$	[First right reflection]
18		${\delta }_k^{(3)}=s_{k2}{\vartheta }_{k-1}-c_{k2}{\delta }_k^{(2)},{\gamma }_k^{(3)}=-c_{k2}{\gamma }_k^{(2)},{\eta }_k=s_{k2}{\gamma }_k^{(2)}$
19		${\vartheta }_{k-1}^{(2)}=c_{k2}{\vartheta }_{k-1}+s_{k2}{\delta }_k^{(2)}$
20		$c_{k3},s_{k3},{\gamma }_{k-1}^{(5)}\leftarrow \text{Symortho}({\gamma }_{k-1}^{(4)},{\delta }_k^{(3)})$	[Second right reflection…]
21		${\vartheta }_k=s_{k3}{\gamma }_k^{(3)},{\gamma }_k^{(4)}=-c_{k3}{\gamma }_k^{(3)}$	[to zero out ${\delta }_k^{(3)}$]
22		τk = ck1ϕk−1	[Last element of tk]
23		ϕk = sk1ϕk−1, ψk−1 = ϕk−1 ‖ [γk δk+1] ‖	[Update ‖r̄k‖, ‖Ãr̄k−1‖]
24		if k = 1 then γmin = γ1 else ${\gamma }_{\text{min}}\leftarrow \text{min}\{{\gamma }_{\text{min}},{\gamma }_{k-2}^{(6)},{\gamma }_{k-1}^{(5)},|{\gamma }_k^{(4)}|\}$
25		${𝒜}_k=\text{max}\{{𝒜}_{k-1},{\rho }_k,{\gamma }_{k-2}^{(6)},{\gamma }_{k-1}^{(5)},|{\gamma }_k^{(4)}|\}$	[Update ‖Ã‖]
26		ωk = ‖ [ωk−1 τk] ‖, κk ← 𝒜k/γmin	[Update ‖Ãxk‖, cond(Ã)]
27		$w_k=-(c_{k2}/{\beta }_k)q_k+s_{k2}{w}_{k-2}^{(3)}$	[Update wk−2, wk−1, wk]
28		$w_{k-2}^{(4)}=(s_{k2}/{\beta }_k)q_k+c_{k2}{w}_{k-2}^{(3)}$
29		$\mathbf{\text{if}}k>2\mathbf{\text{then}}{w}_k^{(2)}=s_{k3}{w}_{k-1}^{(2)}-c_{k3}{w}_k,w_{k-1}^{(3)}=c_{k3}{w}_{k-1}^{(2)}+s_{k3}{w}_k$
30		$\mathbf{\text{if}}k>2\mathbf{\text{then}}{\mu }_{k-2}^{(3)}=({\tau }_{k-2}-{\eta }_{k-2}{\mu }_{k-4}^{(4)}-{\vartheta }_{k-2}{\mu }_{k-3}^{(3)})/{\gamma }_{k-2}^{(6)}$	[Update μk−2]
31		$\mathbf{\text{if}}k>1\mathbf{\text{then}}{\mu }_{k-1}^{(2)}=({\tau }_{k-1}-{\eta }_{k-1}{\mu }_{k-3}^{(3)}-{\vartheta }_{k-1}^{(2)}{\mu }_{k-2}^{(3)})/{\gamma }_{k-1}^{(5)}$	[Update μk−1]
32		if ${\gamma }_k^{(4)}\ne 0$ then ${\mu }_k=({\tau }_k-{\eta }_k{\mu }_{k-2}^{(3)}-{\vartheta }_k{\mu }_{k-1}^{(2)})/{\gamma }_k^{(4)}$ else μk = 0	[Compute μk]
33		$x_{k-2}^{(2)}=x_{k-3}^{(2)}+{\mu }_{k-2}^{(3)}{w}_{k-2}^{(3)}$	[Update xk−2]
34		$x_k=x_{k-2}^{(2)}+{\mu }_{k-1}^{(2)}{w}_{k-1}^{(3)}+{\mu }_k{w}_k^{(2)}$	[Compute xk]
35		${\chi }_{k-2}^{(2)}=\Vert [{\chi }_{k-3}^{(2)}{\mu }_{k-2}^{(3)}]\Vert$	[Update ‖xk−2‖]
36		${\chi }_k=\Vert [{\chi }_{k-2}^{(2)}{\mu }_{k-1}^{(2)}{\mu }_k]\Vert$	[Compute ‖xk‖]
37	x = xk, ϕ = ϕk, ψ = ϕk‖ [γk+1 δk+2] ‖, χ = χk, 𝒜 = 𝒜k, ω = ωk
	output: x, ϕ, ψ, χ, 𝒜, κ, ω

ALGORITHM 2.

Algorithm SymOrtho.

	input: a, b
1	if b = 0 then s = 0,		r = |a|
2		if a = 0 then c = 1 else c = sign(a)
3	else if a = 0 then
4		c = 0,	s = sign(b),	r = |b|
5	else if |b| ≥ |a| then
6		τ = a/b,	$s=\text{sign}(b)/\sqrt{1+{\tau }^2}$,		c = sτ,	r = b/s
7	else if |a| > |b| then
8		τ = b/a,	$c=\text{sign}(a)/\sqrt{1+{\tau }^2}$,		s = cτ,	r = a/c
	output: c, s, r