On the cost of iterative computations

Abstract

With exascale-level computation on the horizon, the art of predicting the cost of computations has acquired a renewed focus. This task is especially challenging in the case of iterative methods, for which convergence behaviour often cannot be determined with certainty a priori (unless we are satisfied with potentially outrageous overestimates) and which typically suffer from performance bottlenecks at scale due to synchronization cost. Moreover, the amplification of rounding errors can substantially affect the practical performance, in particular for methods with short recurrences. In this article, we focus on what we consider to be key points which are crucial to understanding the cost of iteratively solving linear algebraic systems. This naturally leads us to questions on the place of numerical analysis in relation to mathematics, computer science and sciences, in general.

This article is part of a discussion meeting issue ‘Numerical algorithms for high-performance computational science’.

1. Introduction

In this work, we focus on solving linear algebraic systems Ax = b. Such problems are ubiquitous throughout the computational and data sciences, and thus designing methods, algorithms and software that aim to solve such problems accurately and efficiently is the focus of much effort in the field of high-performance computing and numerical mathematics. In particular, we are concerned with the case where an approximate solution to Ax = b is computed iteratively, which is the approach generally used when A is large and sparse (it need not even be explicitly available) or when a solution adequate for the application requires less accuracy than offered by the available machine precision.

In the era of supercomputing hardware capable of exascale-level computation, developing effective approaches for the iterative solution of such systems will require the development of new methods, algorithms and implementations capable of exploiting the underlying hardware of the machine in an effort to optimize the computational approach for a given machine and a given instance of data A and b. This invokes the seemingly simple question: What is the cost of an iterative computation?

In this work, we aim to illuminate long-standing results that are often missed or misused in efforts to optimize iterative computations. We claim no novel technical contribution; the contribution of this work is rather to explain and clarify important considerations for those working in related areas. Towards this goal, we use the example of the conjugate gradient method (CG). This choice is, in our opinion, appropriate for several reasons. First, the method has a prominent role as a representative of the Krylov subspace methods, which are ubiquitous in computational applications. Second, CG has incredibly deep roots and interconnections, being linked with classical mathematical objects such as continued fractions, orthogonal polynomials and Gauss quadrature. Furthermore, CG is used as an important example in various works aimed at reconsidering the foundations of numerical computations. CG is also now used as a benchmark to access the speed of supercomputers [1].

CG dates back to the work of Hestenes & Stiefel [2] and Lanczos [3]. There has since been a wealth of work towards understanding its mathematical properties and the numerical properties of algorithms which implement it. It is not the purpose of this work to give a historical account of the literature on CG, including seminal works by Axelsson, Brezinski, Golub, Grcar, Greenbaum, Meurant, O’Leary, Paige, Parlett, Saad, Simon, van der Sluis, van der Vorst, and many others. We refer the curious reader to the book [4] for historical notes and relevant background.

2. What is a computation?

In order to discuss the cost of an iterative computation, we must first carefully explain what we mean by a computation. We have deliberately chosen to avoid the word ‘algorithm’ in the title of this manuscript for reasons that we hope to make clear in this section.

The goal of any computation is to solve some problem. In the example, we will use throughout this manuscript, the problem is solving an N × N linear algebraic system

$$Ax=b.$$ 2.1

The definition of problem should, however, be considered in a wider context. The matrix A and vector b typically arise in some intermediate stage of the solution of the original problem. The original problem can come from many vastly different applications; for example, a partial differential equation (an infinite dimensional problem), or some discrete collected data. Regardless of the particular application, it is almost always the case that the origin of the problem imposes certain mathematical structure present in A and b (which may be immediately discernible in some cases and less apparent in others).

A computation involves some particular method for solving the problem. There are many possible methods for solving Ax = b to choose from depending on the structure of the matrix A and its numerical properties. In the case that A is of a moderate size and dense (i.e. it is not advantageous to consider zero entries or low-rank structure), a direct method such as LU factorization might be used. In the case we focus on here, where A is a very large, sparse and symmetric positive definite (SPD) matrix, CG is an obvious choice.

CG is in the class of Krylov subspace methods. A Krylov subspace method on the problem Ax = b implicitly constructs in each iteration an approximation of A by restricting and projecting the matrix viewed as the operator A onto the ith Krylov subspace

$${\mathcal{K}}_i(A,r_0)\equiv \text{span}\{r_0,A{r}_0,\dots ,A^{i-1}{r}_0\},$$

where r₀ = b − Ax₀ is the initial residual given initial approximate solution x₀. The matrix A can then be, on the ith Krylov subspace, represented by the tridiagonal Jacobi matrix T_i; see [4, (ch. 3)]. The CG approximate solution $x_i\in x_0+{\mathcal{K}}_i(A,r_0)$ is determined by the optimality condition

$$||x-x_i|{|}_A=\underset{z\in x_0+{\mathcal{K}}_i(A,r_0)}{min}||x-z|{|}_A,$$ 2.2

which can be equivalently written as

$$||x-x_i|{|}_A=||{\rho }_i^{\text{CG}}(A)(x-x_0)|{|}_A=\underset{\rho (0)=1,\text{deg}(\rho )\le i}{min}||\rho (A)(x-x_0)|{|}_A,$$ 2.3

with the ith CG polynomial ${\rho }_i^{\text{CG}}(\mathrm{\lambda })$ determined by the eigenvalues ${\theta }_1^{(i)},\dots ,{\theta }_i^{(i)}$ of the Jacobi matrix T_i via

$${\rho }_i^{\text{CG}}(\mathrm{\lambda })=\frac{(\mathrm{\lambda }-{\theta }_1^{(i)})\cdots (\mathrm{\lambda }-{\theta }_i^{(i)})}{{(-1)}^i{\theta }_1^{(i)}\cdots {\theta }_i^{(i)}}.$$ 2.4

Here and in the remainder of the paper, $||v|{|}_A=\sqrt{v^{\text{T}}Av}$ denotes the energy norm of a vector v.

In other words, in each iteration i, CG adds to the initial approximate solution the vector in ${\mathcal{K}}_i(A,r_0)$ that minimizes the energy norm of the error. The CG polynomial ${\rho }_i^{CG}$ is the unique polynomial that minimizes the energy norm of the error over the ith Krylov subspace. CG can equivalently be seen as the Vorobyev method of moments or as a matrix formulation of Gauss–Christoffel quadrature. These are all mathematically the same method. For details, see e.g. [4, (ch. 3)]. It is also important to consider the connection of the finite precision CG method with the infinite-dimensional CG method. For details, we refer the reader to [5]; see also [6] for an example of this connection for elliptic boundary value problems. It is common in the literature to use different polynomials, e.g. Chebyshev polynomials, in (2.3) to obtain an upper bound on the error in CG; we elaborate on this in §3.

Another aspect of the computation is the algorithm used to realize the particular method. For example, the most well-known algorithm for CG is that of Hestenes and Stiefel (HSCG) [2], displayed in algorithm 1, which uses three coupled two-term recurrences. Another algorithmic variant of CG, first developed by Stiefel [7] (see also [8,9]) uses two three-term recurrences. There are a number of other algorithms for CG (many motivated by reducing the cost per iteration) such as those that use different formulae for the computation of the coefficients (e.g. [10,11]), use auxiliary vectors (e.g. the variant of Chronopoulos & Gear [12] and the pipelined CG method of Ghysels & Vanroose [13]), and block iterations into groups of s (the so-called s-step variants; e.g. [14–16]). All the mentioned algorithms are realizations of CG; they are all mathematically equivalent, i.e. in exact arithmetic they produce exactly the same iterates and thus all exhibit the CG optimality property (2.3). As described in the next section, the cost per iteration cannot be reduced to an operation count, as is done in some influential early textbooks as well as recent publications such as [17,18].

A computation also involves an implementation, which involves the rewriting of the algorithm into instructions to be run on a computer. The implementation includes aspects such as the parallelization of the algorithm, the particular programming language, and also hardware aspects such as the machine precision.

Finally, there is the particular instance of data and machine involved in the computation. In our example, the data are a particular matrix A, vector b, initial approximate solution vector x₀, and the required accuracy of the approximation to the exact solution. The numerical properties and structure of the data in combination with the particular machine hardware ultimately determine the cost of the iterative computation, and thus ultimately the particular instance of data and machine must play a key role in developing an appropriate method, algorithm and implementation to use to solve the problem.

Achieving good performance at scale will thus require a revolutionary change in the entire computational process. It is not merely a question of using a different algorithm. This was formulated in a visionary way by von Neumann at the time of the dawn of the computer. Quoted by Grcar in his beautiful survey on the origins of modern numerical analysis [19], von Neumann writes in a letter to Maxwell Newman [20, p. 1],

I am convinced that the methods of ‘approximation mathematics’ will have to be changed very radically in order to use …[a computer] sensibly and effectively - and to get into the position of being able to build and use still faster ones.

Outside of the field of numerical mathematics, there is a tendency to emphasize the importance of the algorithm in defining the cost of a computation, which has led to work in extending the Turing machine model for discrete computations to the realm of numerical computations in order to rectify what some see as a lack of rigour in the field. For example, Blum et al. [[21, bib21], p. 21; [23, bib22], p. 23] write

There is not even a formal definition of algorithm in the subject … Thus we view numerical analysis as an eclectic subject with weak foundations …

By contrast, Baxter & Iserles conclude in their essay On the foundations of computational mathematics [23, p. 27, 30],

The purpose of computation is not to produce a solution with least error but to produce reliably, robustly and affordably a solution which is within a user-specified tolerance … The foundations of computational mathematics are the totality of mathematics.

We concur with the last quote and summarize:

In the field of computational mathematics (numerical mathematics/numerical analysis) the transformation of the data is the key concept, consisting of a problem, method, algorithm, implementation and particular instance of data and machine.

Algorithms are merely a part of this, and thus efforts to define the cost of an algorithm in isolation from these other aspects will not deal with the bigger picture. Moreover, in works focused on the algorithm as the primary (and essentially the only) object, the notion of cost or complexity of algorithms is reduced to counting arithmetic operations using specific assumptions on the data, and the characteristically nonlinear convergence behaviour of Krylov subspace methods such as CG and of their various algorithmic realizations is falsely described in terms of linear contractions.

It is also worth noting that the cost of a computation is well defined in terms of energy consumption (e.g. the kilowatt hours required to execute the computation) and/or in terms of runtime (e.g. the number of seconds required for a computation to run on a machine). These are meaningful metrics, important in practice as well as theory, which are measurable even in total absence of a formal model of real computation.

As for extrapolating the role of contraction arguments in the description of the convergence behaviour of Krylov subspace methods such as CG, one can only say that highly nonlinear phenomena can be meaningfully linearized only locally. In his article on describing rates of convergence published in 1977, Pták [24] makes a similar comment, noting that

…a description which fits the whole process, not only an asymptotic one, is only possible by means of suitable functions, not just numbers.

For a recent work which develops these ideas further, see [25]. We elaborate on this in the following section.

3. Describing the cost of an iterative computation

Various performance models have been developed to describe the cost of a computation in terms of the cost of floating point operations and the cost of moving blocks of data of a given size between levels of the sequential memory hierarchy and/or between parallel processors. For an overview of sequential memory models, see [26, Section 2.2.1]. In parallel models (assuming a distributed memory machine), data movement is between different parallel processors. Well-known parallel models include the PRAM model [27], the n_1/2 model [28], the Bulk Synchronous Parallel model [29], the LogP model [30] and the (α, β, γ)-model [31].

Unlike direct computations which perform a fixed number of floating point operations,¹ in iterative computations one cannot determine a priori (except in a few special cases) the number of iterations that will be required to converge to the prescribed accuracy. Assuming a constant cost per iteration, the cost of an iterative computation is then given by

$$\text{Cost}=(\text{Cost per iteration})\times (\text{Number of iterations}),$$ 3.1

where ‘Cost per iteration’ will depend on the efficiency of the implementation as well as specific structure of the problem and the underlying hardware, and ‘Number of iterations’ will depend on the accuracy requirements of the application, the numerical properties of the data, the method used and the finite precision behaviour of the particular algorithm chosen for the given method.

The effects of finite precision roundoff error can have a significant influence on the number of iterations required and cannot be ignored; different algorithms for the same method can result in widely varying costs, and in the extreme case, roundoff errors can prevent an algorithm from satisfying the prescribed accuracy requirements at all. In the following subsections, we briefly recall work towards understanding the behaviour of CG (of both the method itself and in finite precision algorithms) and examine various aspects deserving careful consideration.

(a). Bounding the norm of the error

As described in §2, the behaviour of CG is described by the minimization problem (2.3) where the polynomial that achieves this minimization is the CG polynomial ${\rho }_i^{\text{CG}}$ in (2.4). From (2.3), we can write

$$\frac{||x-x_i|{|}_A}{||x-x_0|{|}_A}\le \underset{\rho (0)=1,\text{deg}(\rho )\le i}{min}\underset{1\le j\le N}{max}|\rho ({\mathrm{\lambda }}_j)|,$$ 3.2

where λ_j, j = 1, …, N, are the eigenvalues of the SPD matrix A. This bound is sharp, meaning that if A has d distinct eigenvalues, then for each i = 0, …, d there is some initial approximation x₀ such that the left- and right-hand sides in (3.2) are equal. Further, for each i = 1, …, d − 1, there exist i + 1 distinct eigenvalues of A, ${\hat{\mathrm{\lambda }}}_1,\dots ,{\hat{\mathrm{\lambda }}}_{i+1}$, such that [32]

$$\underset{\rho (0)=1,\text{deg}(\rho )\le i}{min}\underset{1\le j\le N}{max}|\rho ({\hat{\mathrm{\lambda }}}_j)|={\left(\sum _{k=1}^{n+1}\prod _{j=1,j\ne k}^{i+1}\frac{|{\hat{\mathrm{\lambda }}}_j|}{|{\hat{\mathrm{\lambda }}}_j-{\hat{\mathrm{\lambda }}}_k|}\right)}^{-1}.$$ 3.3

This again demonstrates that the convergence behaviour of CG depends strongly on the distribution of eigenvalues of A and that CG is highly nonlinear.

As mentioned in §2, a frequently made oversimplification is to replace the set of increasingly ordered eigenvalues of A by the continuous interval [λ₁, λ_N] and upper bound the right-hand side of (3.2) using scaled and shifted Chebyshev polynomials. This leads to the often-used bound

$$\frac{||x-x_i|{|}_A}{||x-x_0|{|}_A}\le 2{\left(\frac{\sqrt{\kappa (A)}-1}{\sqrt{\kappa (A)}+1}\right)}^i,$$ 3.4

where κ(A) = λ_N/λ₁ is called the condition number of the SPD matrix A. The highly nonlinear convergence behaviour of CG is thus bounded by a linear contraction for any distribution of eigenvalues between λ₁ and λ_N and any distribution of the components of the initial residuals in the individual invariant subspaces. We further note that due to a result of Greenbaum [33], which we elaborate on in §3b(i), the bound (3.4) approximately applies also to finite precision HSCG. Because the specific distribution of eigenvalues is crucial in determining CG convergence, and, further, because the behaviour of finite-precision HSCG can differ significantly from the exact method, it is clear that (3.4) will often fail to give a good estimate of the CG convergence behaviour; see e.g. [4] and also the numerical experiment in §3c. For a recent demonstration of how inappropriate use of (3.4) can be misleading, see [6], which explains the limitations as well as the non-trivial requirements for the appropriate use of condition number arguments for a class of elliptic boundary value problems.

Despite the obvious shortcomings of the bound (3.4), condition number-based contraction is frequently identified with the CG ‘convergence rate’ in general in the literature. This has been extended to describing the complexity of CG in terms of the condition number (e.g. [17,18,34–36]). The justification for this is frequently misattributed to classical papers, which interpret the result differently. For example, in [17, pp. 8–9] (see also [37]) it is stated that

Soon after the introduction of κ(A) for error analysis, Hestenes and Stiefel [2] showed that this quantity also played a role in complexity analysis. More precisely, they showed that the number of iterations of the conjugate gradient method (assuming infinite precision) needed to ensure that the current approximation to the solution of a linear system attained a given accuracy is proportional to $\sqrt{\kappa (A)}$.

This follows from (3.4), which is not presented by Hestenes and Stiefel; in fact, it was clear through the connections they make to orthogonal polynomials, Gauss quadrature and continued fractions (see §2 and [2, Sections 14–18]) that the nonlinearity of CG was well understood. In the context of solving large-scale linear systems, Chebyshev polynomials were considered in the early works of Young, Lanczos, Rutishauser and others; see [4, Section 5.5.3] for a detailed description. However, these authors never had in mind that the upper bound for CG that can be derived using Chebyshev polynomials should be equated with a linear ‘rate of convergence’ of CG; the concept of a linear ‘rate of convergence’ is applicable to stationary iterative methods but makes little sense in the context of Krylov subspace methods. Daniel, who first published the explicit form of the bound (3.4) [38, Theorem 1.2.2], was careful to explain that

… assuming only that [the spectrum of the matrix A lies inside the interval [λ₁, λ_N]], we can do no better than [the bound (3.4)]. Under more specialized assumptions, it is possible to deduce more accurate estimates …

Similar cautions have been repeated by others. In [39, p. 28], Axelsson & Barker state

It should be kept in mind, however, that depending on the distribution of the interior eigenvalues [the bound (3.4)] may be quite pessimistic, hiding the true rate of convergence.

This is also echoed by van der Vorst [40, p. 50], who writes

Upperbounds as in [the bound (3.4)] show that we have global convergence behaviour, but they do not help us explain all sorts of local effects in the convergence behaviour of CG.

See also the comments of Hackbusch [41, Remark 9.4.13, p. 272].

Illustrative examples using SPD diagonal matrices A with N real, positive distinct eigenvalues were used in [42,43], published in 1991 and 1992, respectively (see also the more recent monograph [44]). Given the smallest eigenvalue λ₁ and the largest eigenvalue λ_N, the inner eigenvalues are defined by, for j = 2, …, N − 1

$${\mathrm{\lambda }}_j={\mathrm{\lambda }}_1+\frac{j-1}{N-1}({\mathrm{\lambda }}_N-{\mathrm{\lambda }}_1){\gamma }^{N-j},$$ 3.5

for a given parameter 0 < γ ≤ 1. Varying the value of γ generates matrices with the same condition number κ(A) but with different distributions of eigenvalues, which drastically affect CG convergence behaviour (including the effects of rounding errors).

Similarly, it is easy to demonstrate the importance of the right-hand side, which is not captured by the bound (3.4). For further work on the investigation of the role of the right-hand side in the convergence behaviour of CG; see e.g. [45].

Although the experiments using (3.5) can be considered as simple illustrations, they capture the points that are present in practical computations. More details can be found, e.g. in [4, (Section 5.6)] and in the literature quoted therein. Experiments using a discretized boundary value problem together with a detailed explanation of CG convergence and its relationship to the bounds based on the so-called adapted condition number can be found in [6]; see also §3c and [4, Section 5.9.2].

(b). Algorithms designed for high-performance computing

On modern computer architectures, the time to perform a floating point operation is much less than the time to move a word of data, both between levels of the memory hierarchy and between parallel processors. Thus in the setting of large-scale computations on large-scale machines, the number of floating point operations performed is often a poor measure of runtime and/or energy cost, which depends much more heavily on the communication (i.e. the data movement). This has motivated the study and development of algorithms that minimize the amount of communication, which becomes especially important at the exascale level (e.g. [46]). The difficulty of obtaining good performance in iterative solvers such as CG is highlighted by the HPCG benchmark [1], on which today’s top supercomputer obtains less than 2% of peak performance [47].

There has been a wealth of work, mostly in the context of direct numerical linear algebra, towards proving lower bounds on the amount of data an algorithm must move and in how many messages it must do so (e.g. [26,48–51]). The subsequent task is to find an approach that attains the proved lower bound on communication, potentially at the price of extra floating point operations (which have negligible cost). This line of research has resulted in a number of algorithms and implementations with improved runtimes on a variety of practical problems, many of which have since been implemented in libraries such as LAPACK [52].

The extension of such ideas to iterative methods, however, is tricky. There is a long history of developing algorithms that reduce the number of synchronizations per iteration in Krylov subspace methods; for an overview, see [53]. As mentioned in §2, two recent approaches are ‘s-step’ and ‘pipelined’ algorithms. The former reduces the number of synchronizations by a factor of s over a fixed number of iterations and the latter hides synchronization cost by overlapping global synchronizations with other useful work. For problems with certain sparsity structure, both approaches can offer faster runtime over a fixed number of steps.

The issue is with the phrase a fixed number of steps. As we have outlined, the cost of an iterative computation has two components; both the cost per iteration and the number of iterations to convergence to a prescribed accuracy. Especially in the case of those Krylov subspace methods for which algorithms typically use short recurrences (like CG), rounding errors can have a significant effect on the delay of convergence.

The finite precision effects evident in HSCG can be even worse in algorithms designed to reduce the cost per iteration. Despite being equivalent to HSCG in exact arithmetic, sometimes seemingly innocuous algorithmic modifications can cause finite precision roundoff errors to be amplified in a way that causes the numerical behaviour to differ, potentially significantly, from the classical HSCG algorithm. This means that the total number of iterations required for convergence to the prescribed accuracy, which is typically far from the maximum attainable accuracy, can be much greater than in HSCG. If the increase in the required number of iterations outweighs the reduction in cost per iteration, this can potentially negate any benefits of synchronization-reducing and synchronization-overlapping approaches.

Thus consideration of the effects of rounding errors, especially the potential for their amplification, is inextricable from the design of iterative algorithms. It makes little sense to declare algorithms to be ‘high-performance algorithms’ or ‘exascale algorithms’; whether or not they are suitable for high-performance computing depends heavily on the implementation and on the input data A, b, both in terms of their structure and numerical properties.

In the following subsection, we will summarize existing results on the finite precision behaviour of HSCG and related methods. We then briefly describe the s-step and pipelined variants, and for each, describe what is known and what remains open regarding their finite precision behaviour.

(i). (HSCG in finite precision)

We first review some results regarding the convergence delay and attainable accuracy in finite precision HSCG; for further details on this topic, see e.g. [54]. In the remainder of the paper, we use hats to denote quantities computed in finite precision whenever it is necessary to distinguish them from the exact quantities and ε to denote the machine unit roundoff.

In any iterative approach to solving linear systems, the accuracy $||x-{\hat{x}}_i||$ is generally not computable since it requires knowing the true solution x. However, since $x-{\hat{x}}_i=A^{-1}(b-A{\hat{x}}_i)$, it is common to use the size of the true residual, $||b-A{\hat{x}}_i||$, as a computable measure of accuracy.² Over the course of the iterations, rounding errors made in updating ${\hat{x}}_i$ and ${\hat{r}}_i$ cause the true residual $b-A{\hat{x}}_i$ to deviate from the recursively updated residual ${\hat{r}}_i$ computed in line 6 of algorithm 1. Writing $b-A{\hat{x}}_i={\hat{r}}_i+(b-A{\hat{x}}_i-{\hat{r}}_i)$, we can bound $||b-A{\hat{x}}_i||\le ||{\hat{r}}_i||+||b-A{\hat{x}}_i-{\hat{r}}_i||$. If the method is convergent, we eventually have $||{\hat{r}}_i||\to 0$, and thus the size of the true residual will depend on the size of the residual gap $f_i\equiv b-A{\hat{x}}_i-{\hat{r}}_i$. There is a wealth of work on bounding the attainable accuracy for classical variants of CG, including HSCG (the first of which was due to Greenbaum [55]) and also variants that use two coupled 3-term recurrences, e.g. [56]. In short, if the finite precision iterate updates in HSCG satisfy ${\hat{x}}_i={\hat{x}}_{i-1}+{\hat{\alpha }}_{i-1}{\hat{p}}_{i-1}+\delta x_i$ and ${\hat{r}}_i={\hat{r}}_{i-1}-{\hat{\alpha }}_{i-1}A{\hat{p}}_{i-1}+\delta r_i$, then using standard floating point rounding error analysis, a bound on the size of the residual gap f_i can be written in various forms [55,57,58], e.g.

$$||f_i||\le ||f_0||+O(\epsilon )\sum _{m=0}^i(1+N)||A||||{\hat{x}}_m||+||{\hat{r}}_m||.$$ 3.6

Describing the behaviour of HSCG in finite precision is a more difficult task. Finite precision errors cause orthogonality and eventually linear independence among the computed residual vectors (which are crucial in forming the direction vectors and updating the approximate solution) to be lost. The numerical rank deficiency of the subspace spanned by the residual vectors, which is, strictly speaking, no longer a Krylov subspace, can be related to the delay of convergence. Relatedly, in CG, the coefficients α₀, …, α_i and β₁, …, β_i determine through an easy calculation the entries of the Jacobi matrix T_i whose eigenvalues (called Ritz values) ${\theta }_{\ell }^{(i)}$, ℓ = 1, …, i, approximate those of A. Paige [59] has shown that rank deficiency (and thus convergence delay) goes hand-in-hand with the appearance of multiple copies of eigenvalues of A appearing as Ritz values in T_i. Paige’s results were subsequently used as the basis for the ‘backward-like’ error analysis of Greenbaum [33], who proved that the finite precision HSCG³ algorithm for matrix A behaves like the exact CG method for a larger matrix $\tilde{A}$ which has tight clusters of possibly many eigenvalues around the eigenvalues of A. The investigation of backward-like stability and convergence behaviour for even simple algorithms like HSCG remains an active area of research; see the recent works [60,61].

We return briefly to the connection of CG with Gauss–Christoffel quadrature mentioned in §2. The coefficients α₀, …, α_i and β₁, …, β_i determine distribution functions ω⁽ⁱ⁾(λ) which approximate, in terms of the ith Gauss–Christoffel quadrature, the distribution function ω(λ) determined by inputs A, b and x₀. The A-norm of the CG error for f(λ) = λ⁻¹ can thus be related to the scaled quadrature error, i.e.

$$\frac{||x-x_0|{|}_A^2}{||r_0|{|}^2}=\int {\mathrm{\lambda }}^{-1}d\omega (\mathrm{\lambda })=\sum _{\ell =1}^i{\omega }_{\ell }^{(i)}{\{{\theta }_{\ell }^{(i)}\}}^{-1}+\frac{||x-x_i|{|}_A^2}{||r_0|{|}^2};$$

(e.g. [4, Section 3.5]). For a particular CG algorithm variant in finite precision, the size of the error can be related to the distribution functions ${\hat{\omega }}^{(i)}$ determined by the computed quantities (more precisely, by the Jacobi matrices determined through the computed coefficients α_j and β_j). In other words, ${\hat{\omega }}^{(i)}(\mathrm{\lambda })$ can be associated with some distribution function $\hat{\omega }(\mathrm{\lambda })$ which is meaningfully related to the original distribution function ω(λ), i.e.

$$\int {\mathrm{\lambda }}^{-1}\text{d}\omega (\mathrm{\lambda })\approx \int {\mathrm{\lambda }}^{-1}\text{d}\hat{\omega }(\mathrm{\lambda })=\sum _{\ell =1}^i{\hat{\omega }}_{\ell }^{(i)}{\{{\hat{\theta }}_{\ell }^{(i)}\}}^{-1}+\frac{||x-{\hat{x}}_i|{|}_A^2}{||r_0|{|}^2}+F_i,$$ 3.7

where F_i is relatively small. For HSCG, this was proved by Greenbaum [33] (see also [43,62]). In this way, the delay of convergence due to roundoff is rigorously mathematically described through the error in the Gauss–Christoffel quadrature.

(ii). (s-step CG in finite precision)

The basic idea behind s-step variants of Krylov subspace methods is to expand the Krylov subspace O(s) dimensions at a time and then perform a block orthogonalization using a single global synchronization. We give a brief description of s-step CG; for a more detailed derivation and a historical overview of related work, see [63,64].

The s-step CG method, shown in algorithm 2, consists of an outer loop indexed by k and an inner loop which iterates over j ∈ {1, …, s}. In each outer loop k (each block of s iterations), we construct bases for the Krylov subspaces

$${\mathcal{K}}_{s+1}(A,p_{sk})=\text{span}\{p_{sk},A{p}_{sk},\dots ,A^s{p}_{sk}\}\text{and}{\mathcal{K}}_s(A,r_{sk})=\text{span}\{r_{sk},A{r}_{sk},\dots ,A^{s-1}{r}_{sk}\}.$$ 3.8

We let ${\mathcal{P}}_k$ and ${\mathcal{R}}_k$ denote matrices whose columns form bases for subspaces in (3.8) with starting vectors p_sk and r_sk, respectively, and define the N × (2s + 1) matrix ${\mathcal{Y}}_k=[{\mathcal{P}}_k,{\mathcal{R}}_k]$. Note that (assuming exact arithmetic) any polynomials can be used for constructing these bases. Letting ${\underset{\_}{\mathcal{Y}}}_k$ denote ${\mathcal{Y}}_k$ with columns s + 1 and 2s + 1 set to zero, we have the recurrence $A{\underset{\_}{\mathcal{Y}}}_k={\mathcal{Y}}_k{\mathcal{B}}_k$, where ${\mathcal{B}}_k$ a matrix of size (2s + 1) × (2s + 1) (and which is in general upper Hessenberg) containing the coefficients for the particular polynomial bases used. Subsequently, the inner products between basis vectors are computed in a single global synchronization by constructing the Gram matrix $G_k={\mathcal{Y}}_k^T{\mathcal{Y}}_k$. Within the inner loops iterations, rather than update the length-N vectors x, r and p, we instead update their coordinates in the bases stored in the columns of ${\mathcal{Y}}_k$, denoted x′, r′ and p′, respectively. In the basis ${\mathcal{Y}}_k$, the distributed matrix–vector multiplication Ap_i−1 can be computed via ${\mathcal{B}}_k{p}_{sk+j-1}^{\prime }$. Inner products computed during the iterations can be recovered using, e.g. $r_i^T{r}_i=r_{sk+j}^{\prime \text{T}}{\mathcal{Y}}_k^T{\mathcal{Y}}_k{r}_{sk+j}^{\prime }=r_{sk+j}^{\prime \text{T}}{G}_k{r}_{sk+j}^{\prime }$. Since s is assumed to be much smaller than N, the matrices ${\mathcal{B}}_k$ and G_k and the coordinate vectors can be stored on each processor locally, and thus there is no interprocessor communication required within the inner loop.

The attainable accuracy of s-step CG has been studied in [65]; see also [64]. In summary, in iteration sk + j, the residual gap can be bounded as

$$||f_{sk+j}||\le ||f_0||+\epsilon {\overline{\mathit{\Gamma }}}_k\sum _{\ell =1}^{sk+j}(1+N)||A||||{\hat{x}}_{\ell }||+||{\hat{r}}_{\ell }||,$$ 3.9

where ${\overline{\mathit{\Gamma }}}_k=\underset{m\le k}{max}{\mathit{\Gamma }}_m$ with ${\mathit{\Gamma }}_m=c||{\hat{\mathcal{Y}}}_m^{+}||||{\hat{\mathcal{Y}}}_m||$, where the superscript ‘+’ denotes the Moore–Penrose pseudoinverse and c is a low-degree polynomial in s. We can think of Γ_m as a type of condition number of the computed basis ${\hat{\mathcal{Y}}}_m$. The bound (3.9) says that the local rounding errors in each iteration can be amplified by ill-conditioned ${\hat{\mathcal{Y}}}_m$, linking the condition numbers of the polynomial bases computed in each outer iteration with the loss of accuracy in finite precision s-step CG.

The convergence delay in s-step CG has not been fully investigated theoretically, although it is clear that the basis condition number plays a large role. In [66], the analysis of Paige [59] for the classical Lanczos algorithm is extended to the s-step Lanczos algorithm, and it is shown that the conclusions of Paige for the classical Lanczos method regarding convergence of Ritz values and Ritz vectors also apply to the s-step Lanczos method as long as

$${\overline{\mathit{\Gamma }}}_k^2<{(24\epsilon (N+11s+15))}^{-1}.$$ 3.10

The results in [66] may provide a path forward for extending the results of Greenbaum [33] to the s-step CG algorithm. In particular, we conjecture that as long as (3.10) holds, the finite precision s-step CG algorithm on a matrix A can be associated with the exact algorithm on a larger matrix with eigenvalues in clusters around those of A, where the diameter of the clusters will depend on ${\overline{\mathit{\Gamma }}}_k$. We note that such a result would also imply that as long as (3.10) holds, the Chebyshev polynomial-based convergence bound (3.4) approximately holds even for s-step CG in finite precision. Other potential avenues for studying the convergence behaviour of s-step CG in finite precision are described in [53, (Section 4.2)].

In general, we expect the condition number of the computed bases to increase as s increases, the rate of which depends on the particular polynomials used. This presents a trade-off between the cost per iteration (which we generally expect to decrease with s up to some point) and the number of iterations (which we generally expect to increase with s).

In figure 1, we demonstrate the numerical behaviour of s-step CG with s = 2, 3, 4 in finite (double) precision using the matrix bcsstk01 from SuiteSparse [67]. Here, we use the monomial basis for the Krylov subspaces in (3.8). The right-hand side b has equal components in the eigenbasis of A with ‖b‖₂ = 1 and x₀ = 0. For comparison, we also plot HSCG in double precision as well as CG in (simulated) exact arithmetic. It is clear that both the convergence delay and the decrease in attainable accuracy in s-step CG grow worse with s. We note that the effects of roundoff error are visible even for the HSCG method.

Figure 1.

Convergence of HSCG and s-step CG (using monomial basis) with s = 2, 3, 4 in double precision and exact CG for matrix bcsstk01. (Online version in colour.)

It is well known that bounds on the condition number of the monomial basis grow exponentially with s, and thus it is expected that the numerical behaviour will quickly deteriorate with growing s. That the use of monomials leads to numerical instability was recognized even in early works on the s-step CG method (e.g. [12]). This led to the use of more well-conditioned bases, constructed using, e.g. Newton or Chebyshev polynomials, parameters for which can be approximated using rough estimates of the spectrum of A (which can be estimated using Ritz values from the first few iterations; e.g. the works [68–72]). In many cases, this has been shown to improve numerical behaviour, but we stress that this is not a universal rule. For example, for the test problem in figure 1, both Newton and Chebyshev bases constructed using estimates of the extremal eigenvalues of A perform worse than the monomial basis in terms of both convergence and accuracy, despite a smaller ${\overline{\mathit{\Gamma }}}_k$. The bound (3.9) of course still applies; it is merely that the attainable accuracy using the monomial basis is much better than what is predicted by the right-hand side of (3.9).

(iii). (Pipelined CG in finite precision)

The idea behind pipelined CG, shown in algorithm 3, is to decouple the sparse matrix–vector multiplication and the inner products so that they can be computed simultaneously using asynchronous communication collectives. This decoupling is accomplished by the introduction of auxiliary recurrences s_i ≡ Ap_i, w_i ≡ Ar_i and z_i ≡ Aw_i ≡ A² r_i, as well as the use of an alternative form for the computation of the scalar α_i. We limit our discussion to the unpreconditioned version of pipelined CG; the preconditioned variant requires the use of additional auxiliary vectors.

The pipelined CG algorithm, introduced by Ghysels & Vanroose [13], builds on a wealth of previous work such as the algorithm of Chronopoulos & Gear [12]. For a more complete description of the history and related methods, see [53]. Other pipelined variants of Krylov subspace methods have been developed, such as pipelined GMRES [73] and pipelined BiCGStab [74], as well as ‘deep-pipelined’ variants [73,75].

For variants such as pipelined CG that use auxiliary recurrences, the maximum attainable accuracy can be bounded using the methodology outlined in [53]. A detailed analysis of the maximum attainable accuracy in pipelined CG was given by Cools et al. [76] which follows this methodology. We briefly summarize the main result of their analysis.

Letting ${\delta }_i^p$, ${\delta }_i^s$, ${\delta }_i^z$, ${\delta }_i^x$, ${\delta }_i^r$ and ${\delta }_i^w$ denote the local errors due to finite precision computation in the associated vector updates, the size of the residual gap in pipelined CG can be expressed as $||f_i||\le ||f_0||+\sum _{j=1}^i(||A||||{\delta }_j^x||+||{\delta }_j^r||+|{\hat{\alpha }}_j|||g_j||)$, where

$$\begin{array}{rl}& g_j=\left(\prod _{k=1}^j{\hat{\beta }}_k\right)g_0+\sum _{k=1}^j\left(\prod _{\ell =k+1}^j{\hat{\beta }}_{\ell }\right)(A{\delta }_k^p-{\delta }_k^s+h_k),h_k=h_0-\sum _{\ell =0}^{k-1}(-A{\delta }_{\ell }^r+{\delta }_{\ell }^w+{\hat{\alpha }}_{\ell }{c}_{\ell })\\ \text{and}& c_{\ell }=\left(\prod _{m=1}^{\ell }{\hat{\beta }}_m\right)c_0+\sum _{m=1}^{\ell }\left(\prod _{n=m+1}^{\ell }{\hat{\beta }}_n\right)(A{\delta }_m^q-{\delta }_m^z);\end{array}$$

see [76, Section 2.3]. It is clear from the above expression that local rounding errors made in each iteration can be globally amplified by the values of the computed $\hat{\alpha }$’s and $\hat{\beta }$’s. In exact arithmetic, the β terms give the ratio between subsequent residual norms, which are not monotonically decreasing in CG. Thus, we might expect amplification of errors whenever there is residual oscillation; see [56] for an early description of this phenomenon.

In figure 2, we demonstrate the effect of finite precision errors in pipelined CG in double precision using the same problem as in figure 1. Relative to HSCG, pipelined CG suffers a delay of convergence and exhibits a drastic decrease in attainable accuracy.

Figure 2.

Convergence of HSCG and pipelined CG in double precision and exact CG for matrix bcsstk01. (Online version in colour.)

We note that figures 1 and 2 are not necessarily representative of the general numerical behaviour of the s-step and pipelined CG algorithms. In figure 3, we plot the convergence of HSCG, s-step CG with s = 2, 3, 4, and pipelined CG in double precision for a different problem using the matrix nos4 from SuiteSparse [67]. We also include CG in (simulated) exact arithmetic for reference. Again, the right-hand side b was set to have equal components in the eigenbasis of A with ‖b‖₂ = 1 and x₀ = 0. For this problem, the s-step and pipelined algorithms all behave similar to HSCG up until a relative accuracy of around 10⁻¹¹. Depending on the prescribed accuracy requirements of the application, any of the variants of HSCG could be perfectly suitable from a numerical perspective.

Figure 3.

Convergence of HSCG, s-step CG (using monomial basis) with s = 2, 3, 4, and Pipelined CG in double precision as well as exact CG for matrix nos4. (Online version in colour.)

We close this subsection with the comment that the focus thus far (in the mentioned variants of Krylov subspace methods and also in e.g. ‘inexact Krylov subspace methods’ [77]) has been almost exclusively on bounding the maximum attainable accuracy, as this is a much simpler task than determining effects on convergence delay. Although useful for determining the applicability of the method to a given problem, typically in practice the prescribed accuracy requirement is much, much above than the maximum attainable accuracy. The more important question is in determining whether such approaches are beneficial in terms of cost, which requires a study of the convergence behaviour in finite precision. One way forward may be through the mentioned connection to Gauss–Christoffel quadrature [53, Section 4.2].

(c). The effect of preconditioning

Preconditioning is a technique which involves transforming the linear system Ax = b into an equivalent one for which the iterative method will converge more rapidly. Given a preconditioner M, in the case of left preconditioning for example, the preconditioned system becomes M⁻¹Ax = M⁻¹b. There are a few considerations concerning the construction and application of preconditioners that require attention.

The name ‘preconditioning’ often invokes the argument that the condition number plays the dominant role in determining convergence behaviour, which is certainly not true; see §3a. Even when the distribution of eigenvalues is taken into account, statements regarding the convergence behaviour often rely on arguments about eigenvalue clustering. A frequent assertion is that the number of tight clusters of eigenvalues is equal to the number of iterations in which a Krylov subspace method can be expected to provide an approximation close to the exact solution. This is not true in general for any Krylov subspace method we are aware of.

In order to explain the point for CG and SPD matrices, we use the original analysis of Greenbaum [33] mentioned in §3b. Skipping some technical details, she proved that given the number of steps, finite precision CG on a problem with a matrix A (possibly with simple eigenvalues) behaves like CG in exact arithmetic on a problem with a larger matrix $\tilde{A}$ which has tight clusters of eigenvalues around those of A. It is evident that finite precision CG on a problem with the matrix A can behave very differently from exact CG for the same problem. Therefore, it cannot be true that in general exact CG for the problem with the matrix $\tilde{A}$ having tight clusters of possibly many eigenvalues behaves similarly to exact CG for the problem with the matrix A where each cluster is represented by a single eigenvalue. This disproves the generality of the clustering argument.⁴ We stress that the consideration of exact CG for the matrices $\tilde{A}$ and A does not include any effects of rounding errors. The location as well as the diameter of the clusters is important in determining convergence.

As a demonstration, in figure 4 we show CG convergence in exact arithmetic for diagonal matrices A with eigenvalues described by (3.5) with N = 25, λ₁ = 0.1, λ_N = 100 and γ = 0.65. Note that this test case was used in [4, e.g. fig. 5.15]. The dashed line shows convergence for the matrix as described with right-hand side b having identical entries and ‖b‖₂ = 1. The circles plot convergence for the matrix constructed by taking A and splitting its two smallest eigenvalues into clusters of five eigenvalues each, with the entries in b split accordingly. Similarly, the dotted line shows convergence for the matrix constructed in the same way except with the two largest eigenvalues split into clusters of five each. The dash-dotted line shows convergence for the matrix with every eigenvalue split into a cluster of five eigenvalues (again, with entries of b split accordingly). The distance between neighbouring eigenvalues within a cluster was 10⁻¹⁴.

Figure 4.

Convergence of CG with clusters of eigenvalues. (Online version in colour.)

Each of these matrices has the same number of clusters of eigenvalues, and yet the convergence behaviour can be different. Notice that the location of the clusters matters. Clusters of eigenvalues close to the origin do not affect the behaviour of CG, whereas the eigenvalue clusters around the high end of the spectrum visibly cause a convergence delay. We stress that these tests are run in (simulated) exact arithmetic; the differing convergence is not due to roundoff error. A detailed explanation with other experiments can be found in [4, Section 5.9.1]. To demonstrate the theory of Greenbaum [33], we include the solid line which shows convergence of HSCG run in double precision on the original matrix with simple eigenvalues. We can see that the behaviour is qualitatively similar to the behaviour of the exact method with each eigenvalue replaced by a cluster. Fully analogous arguments can be applied and the same conclusions can be drawn for problems with symmetric indefinite matrices solved, e.g. by the MINRES method.

The clustering notion as a general argument fails for non-symmetric problems and other Krylov subspace methods like GMRES as well. Even for normal matrices, many eigenvalues close to the origin are known to cause stagnation of GMRES convergence (e.g. [79] and the examples therein). In the non-normal case, it is even more clear that the clustering argument does not hold; it is possible to construct an example problem (or even a parametrized set of problems) with any prescribed spectrum such that GMRES exhibits any prescribed convergence behaviour, e.g. stagnation, until the final iteration [80–82]. Consequently, any fast convergence behaviour of GMRES is possible with the system matrix having eigenvalues widespread in the complex plane and any poor convergence behaviour is possible with real eigenvalues tightly clustered around one. Therefore, any statement that links convergence of GMRES solely with the distribution of the eigenvalues of the system matrix must be justified in relation to these results proved more than 20 years ago. This is in the literature done very rarely.

The clustering argument is often used in combination with the remarkable fact pointed out in [83] that a certain class of preconditioners leads to preconditioned matrices with a low degree minimal polynomial. Such preconditioners are very important in practice where they are often replaced by some more computationally feasible approximations, which can lead instead of a low degree minimal polynomial to a few tight clusters of eigenvalues. This subtle shift requires a rigorous analysis. In [83], the authors issue a strong warning that while replacing the low degree minimal polynomial argument with the argument that the preconditioned matrix has a few distinct eigenvalues one must include a strong assumption that the preconditioned matrix is either diagonalizable or the Jordan blocks have small size.⁵

While replacing single eigenvalues by tight clusters one must admit that a rigorous argument is involved⁶ and the behaviour in practice can depend, as pointed out above in the symmetric case, on the location of the clusters. Our point is that presenting rigorously assumptions under which a statement holds is imperative. Linking convergence of GMRES solely with the distribution of eigenvalues without considering the structure of invariant subspaces (a common practice) is not only mathematically incorrect, but closes the door for investigation of potentially important phenomena. It may happen that eigenvalues do determine GMRES convergence despite that the structure of the associated invariant subspaces does not allow the use of any existing bounds. In this case, we must investigate this structure—it almost certainly bears significant information about the problem. This can be a very positive outcome of the results in [80–82].

An unrelated issue is that much work in designing preconditioners focuses only on the convergence of a Krylov subspace method for the resulting system and ignores the effects on attainable accuracy in finite precision. Even if the preconditioned system is well conditioned, it must be kept in mind that we do not explicitly form this system in practice; the matrix–vector product and preconditioner application are computed separately. We can thus write the computed preconditioned vector as

$$\hat{z}=(M^{-1}A+\mathrm{\Delta })r,\text{where}\hspace{0.17em}||\mathrm{\Delta }|{|}_2\le O(\epsilon )N^{3/2}(\kappa (A)+\kappa (M))||M^{-1}A|{|}_2;$$

c.f. [84, Section 3]. If the original matrix is ill conditioned (which is often the case where we would want to use a preconditioner), any reasonably efficient preconditioner is likely also ill conditioned. Thus, large κ(A) and κ(M) can potentially introduce significant roundoff errors which can affect the numerical behaviour of the method despite κ(M⁻¹A) being small. This was investigated in [84, Section 3] for the case where M consists of the triangular factors from an LU factorization; it was shown there that if A is very ill conditioned (and thus M is also), GMRES provides a backward stable solution to the left-preconditioned system if double the working precision is used in the matrix–vector product and preconditioner application.

Finally, another important consideration is the increase in cost per iteration due to extra computations involved in preconditioning. Even if the preconditioner reduces the number of iterations required for convergence, each iteration may now be much more expensive due to additional communication and/or computation involved in applying the preconditioner, which is of particular importance in highly parallel environments. It is thus not always clear that even with a reduced number of iterations there will be an overall benefit to performance/energy cost. This trade-off must be carefully balanced in minimizing the cost of an iterative method (e.g. [85]) as well as the book of Meurant [86, ch. 8] which nicely describes this and other issues relevant to preconditioning.

4. The importance of data and adaptivity

The importance of considering the source of the problem in developing efficient iterative methods and algorithms has been stressed by many working in the field. In their textbook published in 1981 [9, p. x], Hageman & Young write

The choice of an effective iterative solution method for a particular problem depends heavily on the details peculiar to the problem and on the particular architecture of the computer to be used.

Along similar lines, in the influential textbook [87, p. xiii], Saad explains that

… it still cannot be stated that an arbitrary sparse linear system can be solved iteratively in an efficient way. If physical information about the problem can be exploited, more effective and robust methods can be tailored for the solutions.

The particular problem instance and the underlying application are also of great importance in consideration of the effects of finite precision computation. As Parlett concisely states in his article on the contributions of James Wilkinson [88, p. 20].

… tiny arithmetic inaccuracies leap sharply from being negligible in one domain to critical in the other.

Thus, the data must be a main focus in designing iterative schemes and evaluating the cost of iterative computations; designing iterative algorithms based on general assumptions about the numerical properties and structure of the data, or the statistical distribution of the data, is an effort that does not help in solving particular problems.

As we have seen throughout this article, ‘the data’ cannot be reduced to something as simple as the matrix condition number. For CG, the nonlinear adaptation to the data is the main principle behind the method; this is the beauty of Krylov subspace methods. This was appreciated early on by Lanczos and Einstein, who corresponded on the topic of the Lanczos method [4]:⁷

The reason why I am strongly drawn to such approximation mathematics problems is not the practical applicability of the solution, but rather the fact that a very ‘economical’ solution is possible only when it is very ‘adequate’. To obtain a solution in very few steps means nearly always that one has found a way that does justice to the inner nature of the problem.

— Cornelius Lanczos in a letter to Albert Einstein on 9 March 1947

Your remark on the importance of adapted approximation methods makes very good sense to me, and I am convinced that this is a fruitful mathematical aspect, and not just a utilitarian one.

— Einstein’s reply to Lanczos on 18 March 1947

Equating the (nonlinear) convergence of CG with the (linear) contraction bound based on condition number completely loses any notion of the adaptivity behind the method.⁸

The complex, nonlinear, data-dependent behaviour of Krylov subspace methods, combined with the effect of rounding errors, makes efficient use of such methods seem daunting. Software libraries which provide implementations of algorithms for such iterative methods often provide little or (usually) no documentation describing important aspects of numerical behaviour, leaving the user blindly guessing at the best approach. Baxter & Iserles [23, p. 30] warn that

… the impetus towards ‘general’ methods and ‘general’ software, which can cater to many different problems in a broad category, might be inimical to progress. Once we classify mathematical problems by their structural features, broad categories are much too coarse.

Given the view of the highly data-dependent nature of both the convergence behaviour and the cost per iteration, it seems infeasible to develop approaches with efficient cost for large classes of problems. There is, however, some hope offered by the possibility of explicitly adaptive approaches. Baxter & Iserles [23, p. 27] also write, on the topic of adaptivity,

… good computation requires the algorithm to respond to the data it is producing and change the allocation of computing resources (step size, size of the grid, number of iterations, even the discretization method itself) accordingly … All this emphasis on adaptivity is hardly ‘foundational’ and has been the focus of much effort in numerical analysis, scientific computing and software engineering … adaptivity is linked to important foundational issues and, ideally, in place of its current niche of half-mathematics, half-art, should be brought into the fully mathematic domain.

Echoing the quote above, there are numerous instances where explicitly controlled adaptivity has been incorporated in iterative algorithms to balance trade-offs between numerical behaviour and cost per iteration, in which decisions are made on-line depending on measured quantities and the particular input data. Some well-established examples in Krylov subspace methods include selective reorthogonalization in the Lanczos process and implicit restarting in the Arnoldi method. Baxter and Iserles are correct in pointing out that such algorithmic innovations are often lacking a full theoretical foundation. But some of them are well rooted in theoretical results, with both selective reorthogonalization and implicit restarting being good examples. With the emergence of multiprecision capabilities in hardware, another current area of interest is in the design of algorithms which adaptively vary the precision used throughout the computation. Such techniques present the opportunity for significant performance improvements and are deserving of rigorous theoretical analysis. We expect this to be a fruitful area of future research.

5. The place of numerical analysis, challenges and opportunities

The example of CG illuminates key aspects involved in evaluating the cost of iterative computations that will inevitably play a role in solving challenging numerical problems, and reveals the potential hazards in approaches that fail to take a holistic view of this cost. The primary theses are summarized below.

We stress that the mathematical transformation of the data represented by the iterative scheme is the key concept; algorithms are only a part of the consideration. The cost of a computation in terms of computational time or consumed energy is a well-defined metric, and does not require a formal model of real computation.

Further, the premise that the linearization of highly nonlinear phenomena can give insights only locally or under very specific circumstances is obviously valid in iterative computations. Thus, in the case of Krylov subspace methods, simplified analyses based on condition number and contraction arguments are in general incapable of giving realistic estimates of practical behaviour. Further, rounding errors can have a significant effect on the delay of convergence to the prescribed accuracy. Thus consideration of the effects of rounding errors, especially the potential for their amplification, is inextricable from the design and analysis of iterative schemes. Declaring roundoff issues to be ‘out of the scope’ of such work (which usually means ‘postponed indefinitely’) can only have deleterious effects on the development of efficient iterative computations. Formulae developed under the assumption of exact arithmetic, such as various estimates in quadratic forms (e.g. [44,62,89]) cannot safely be used in computations unless they are justified by an appropriate analysis of the effects of roundoff. This is, in most of the literature, simply neglected.

Finally, iterative computations, and in particular Krylov subspace methods, are used in a number of application areas, and the structure and numerical properties of the particular data A and b will depend on the underlying application problem. Thus, performing analyses of cost or designing algorithms based on abstract and restrictive assumptions solely on the algebraic level can lead to failures in many practical cases. The data must be a primary focus in designing iterative schemes, and this necessitates adaptive approaches.

Computational mathematics and numerical analysis historically lacks a definitive academic home. In some institutions, it lives within mathematics and in others within computer science. Sometimes, it is separated into its own ‘interdisciplinary’ institute. In recent years, the overlap with informatics and data science is becoming increasingly important. We again evoke the essay of Baxter & Iserles [23, p. 4], where they comment that

Numerical analysis lies at the meeting point of pure mathematics, computer sciences and application areas. It often attracts some degree of hostility from all three. Yet it is important to note that the complaints of pure mathematicians, computer scientists and the broad constituency of scientists and engineers are very different and often contradictory.

Indeed, in the area of iterative computations, where computations using Krylov subspace methods can serve as an instructive example, it is clear that efforts towards developing abstract complexity theory, developing algorithms designed for high-performance computing and developing preconditioners are all coming from vastly different academic perspectives. As we have attempted to demonstrate here, no approach isolated to one academic perspective can give a truly complete view of the cost in practical iterative computations. Interdisciplinary research is not only beneficial; it is critical to progress in this area.

There is an emerging trend connecting data science and data informatics with the realm of computational science. We agree fully with this development. Scientific computing consists of much more than solving PDEs; it also includes, e.g. tasks such as event identification and correlation in high-energy physics data, climate simulation and its validation using sensor data, determining patterns in astronomical data, analysis of data from social and other complex networks and genetic sequencing. Supercomputer hardware is now being designed with data-driven applications in mind (e.g. including more memory at the node level, the inclusion of accelerators/specialized processors for neural network computations, and the inclusion of reduced-precision capabilities in hardware).

As argued above, numerical analysis (computational mathematics) is by its nature interdisciplinary and so is its mathematical foundation. Although in this work we have pointed out many areas in which domain-specific viewpoints can lead to missing the bigger picture, we see this as a positive opportunity for further research and development. Towards this, it is imperative that assumptions and limitations are clearly stated, experiments are designed in accordance with the principles of the scientific method, and that results are interpreted honestly. In truth, negative results are often the most valuable in terms of furthering scientific research, because they eventually lead to big steps forward or even major breakthroughs. We invite the greater computational mathematics community, including those in computer science, data science, computer engineering and mathematics, to join together and adhere to these tenets in the push towards achieving unprecedented scales of computational power.

Data accessibility

This article has no additional data.

Authors' contributions

E.C. carried out the numerical experiments and drafted the manuscript. Both authors collaborated on the content and revision of the article. Both authors gave final approval for publication and agree to be held accountable for the work performed therein.

Competing interests

We declare we have no competing interests.

Funding

The first author is supported by Charles University Primus programproject PRIMUS/19/SCI/11. The second author is partially supported by the Grant Agency of the Czech Republic Project 2GA18-12719S.