On Ambiguity in Linear Inverse Problems: Entrywise Bounds on Nearly Data-Consistent Solutions and Entrywise Condition Numbers

Abstract

Ill-posed linear inverse problems appear frequently in various signal processing applications. It can be very useful to have theoretical characterizations that quantify the level of ill-posedness for a given inverse problem and the degree of ambiguity that may exist about its solution. Traditional measures of ill-posedness, such as the condition number of a matrix, provide characterizations that are global in nature. While such characterizations can be powerful, they can also fail to provide full insight into situations where certain entries of the solution vector are more or less ambiguous than others. In this work, we derive novel theoretical lower- and upper-bounds that apply to individual entries of the solution vector, and are valid for all potential solution vectors that are nearly data-consistent. These bounds are agnostic to the noise statistics and the specific method used to solve the inverse problem, and are also shown to be tight. In addition, our results also lead us to introduce an entrywise version of the traditional condition number, which provides a substantially more nuanced characterization of scenarios where certain elements of the solution vector are less sensitive to perturbations than others. Our results are illustrated in an application to magnetic resonance imaging reconstruction, and we include discussions of practical computation methods for large-scale inverse problems, connections between our new theory and the traditional Cramér-Rao bound under statistical modeling assumptions, and potential extensions to cases involving constraints beyond just data-consistency.

I. Introduction

This paper concerns the classical and well-studied finite-dimensional linear inverse problem of estimating a true unknown signal vector ${\mathbf{x}}^{\ast }\in {\mathbb{R}}^N$ from a noisy data vector $\mathbf{b}\in {\mathbb{R}}^M$ obtained as

$$\mathbf{b}=\mathbf{A}{\mathbf{x}}^{\ast }+\mathbf{n},$$ (1)

where $\mathbf{A}\in {\mathbb{R}}^{M\times N}$ is the known system matrix and $\mathbf{n}\in {\mathbb{R}}^M$ represents an unknown noise perturbation. In many practical scenarios, this inverse problem is ill-posed, meaning that the original value of ${\mathbf{x}}^{\ast }$ might be ambiguous even in the absence of noise, and/or that solutions may be highly sensitive to noise.

While there are many existing theoretical results that characterize ambiguity for this inverse problem, it is frequent to see characterizations phrased in terms of global bounds on the estimation error as a function of the size of the noise [1]-[5]. To be concrete, if we let $\widehat{\mathbf{x}}$ be the estimate of ${\mathbf{x}}^{\ast }$ obtained through some estimation procedure, it is common to see global error bounds of the form

$$\Vert \widehat{\mathbf{x}}-{\mathbf{x}}^{\ast }{\Vert }_2\le C\Vert \mathbf{n}{\Vert }_2,$$ (2)

where $\Vert \cdot {\Vert }_2$ denotes the standard euclidean norm and the constant $C$ will depend on the characteristics of the estimation procedure, the characteristics of the matrix $\mathbf{A}$, and potentially also the characteristics of the original vector ${\mathbf{x}}^{\ast }$ and the data vector $\mathbf{b}$. Examples of this type of bound include:

• If $\mathbf{A}$ has full column rank and $\widehat{\mathbf{x}}$ is obtained by least-squares as $\widehat{\mathbf{x}}={\mathbf{A}}^{†}\mathbf{b}$, then the spectral norm [1]-[6] provides the bound

  $$\Vert \widehat{\mathbf{x}}-{\mathbf{x}}^{\ast }{\Vert }_2\le {\sigma }_N^{-1}(\mathbf{A})\Vert \mathbf{n}{\Vert }_2,$$ (3)

  where ${\sigma }_i(\cdot )$ denotes the ith largest singular value of a matrix, and ${\sigma }_N^{-1}(\mathbf{A})=\Vert {\mathbf{A}}^{†}\Vert$, where $\Vert \cdot \Vert$ denotes the spectral norm and ${\mathbf{A}}^{†}$ denotes the Moore-Penrose pseudoinverse of $\mathbf{A}$.
• If $\mathbf{A}$ has full column rank and $\widehat{\mathbf{x}}$ is obtained by least-squares as $\widehat{\mathbf{x}}={\mathbf{A}}^{†}\mathbf{b}$, then condition number analysis shows that [1]-[5]

  $$\frac{\Vert \widehat{\mathbf{x}}-{\mathbf{x}}^{\ast }{\Vert }_2}{\Vert {\mathbf{x}}^{\ast }{\Vert }_2}\le \kappa (\mathbf{A})\frac{\Vert \mathbf{n}{\Vert }_2}{\Vert {\mathbf{b}}^{\ast }{\Vert }_2},$$ (4)

  where ${\mathbf{b}}^{\ast }\triangleq \mathbf{b}-\mathbf{n}$ is the ideal measurement that would have been obtained in the absence of noise,¹ and $\kappa (\mathbf{A})\triangleq {\sigma }_1(\mathbf{A})∕{\sigma }_N(\mathbf{A})$ is the condition number that provides a measure of the sensitivity of the inverse problem to perturbations of the data. The best (smallest) possible condition number is $\kappa (\mathbf{A})=1$, while large values of the condition number imply that even relatively small amounts of noise can lead to potentially large relative errors.

• Many error bounds have been derived that apply under special structural assumptions about ${\mathbf{x}}^{\ast }$, such as sparsity [7] or low-rank [8] assumptions. While we make no attempt to be comprehensive, a typical example is the work of Donoho [9], which showed that if $\Vert \mathbf{n}{\Vert }_2\le \epsilon$ and if ${\mathbf{x}}^{\ast }$ is sufficiently sparse, then using $\widehat{\mathbf{x}}=\text{arg}{\text{min}}_{\mathbf{x}\in {\mathbb{R}}^N}\Vert \mathbf{x}{\Vert }_1$ subject to $\Vert \mathbf{A}\mathbf{x}-\mathbf{b}{\Vert }_2\le \epsilon$ will result in a solution that obeys a bound with the same form as Eq. (2) for most large underdetermined matrices $\mathbf{A}$.

We refer to all of these as global bounds because they only provide insight into the aggregate error across all of the entries of the vector $\widehat{\mathbf{x}}$, in contrast to an entrywise bound that would provide insight into the potential errors for each of the $N$ individual entries of the vector $\widehat{\mathbf{x}}$.

In practical applications, there are many situations where certain entries of ${\mathbf{x}}^{\ast }$ are easier to estimate than others and also many situations where certain entries of ${\mathbf{x}}^{\ast }$ are more important to estimate accurately. In such situations, global performance bounds can be misleading, and it can be very valuable to know which entries of an obtained solution vector are more trustworthy. For example, this situation occurs frequently in biomedical imaging applications, where the data acquisition procedure can capture more information about certain image voxels than it does for others, leading to spatially-varying ambiguity about the reconstructed image. In such applications, this spatially-varying ambiguity has often previously been evaluated using statistical characterizations like variance maps or Cramér-Rao bounds [10]-[16]. Use of such techniques generally requires either explicit modeling of the noise statistics or multiple repetitions of the same data collection procedure to enable empirical variance estimation, and can involve restrictive assumptions about the matrix $\mathbf{A}$ (e.g., that it has full column rank).

In this work, we derive novel tight entrywise bounds that apply to every nearly data-consistent vector $\mathbf{x}$, can be used with arbitrary matrices $\mathbf{A}$ (including rank-deficient matrices), are agnostic to the estimation method used to obtain $\widehat{\mathbf{x}}$, and only require knowledge about the size of $\mathbf{n}$ (via an upper bound on $\Vert \mathbf{n}{\Vert }_2$) without requiring an accurate statistical model or multiple repetitions of the data collection procedure. To be precise about what we mean by “nearly data-consistent solutions,” we will define ${\Gamma }_{\epsilon }(\mathbf{A},\mathbf{b})$ as the set of potential solution vectors $\mathbf{x}$ that are consistent with the data vector $\mathbf{b}$ within a tolerance of $\epsilon$:

$${\Gamma }_{\epsilon }(\mathbf{A},\mathbf{b})=\left\{\mathbf{x}\in {\mathbb{R}}^N\mathrm{s}.\mathrm{t}.\Vert \mathbf{A}\mathbf{x}-\mathbf{b}{\Vert }_2\le \epsilon \right\}.$$ (5)

Our theoretical results will provide tight entrywise bounds that must be satisfied by all of the elements of ${\Gamma }_{\epsilon }(\mathbf{A},\mathbf{b})$. Notably, if $\Vert \mathbf{n}{\Vert }_2\le \epsilon$ and data is acquired according to Eq. (1), then the true value ${\mathbf{x}}^{\ast }$ must belong to ${\Gamma }_{\epsilon }(\mathbf{A},\mathbf{b})$, and our entrywise bounds on ${\Gamma }_{\epsilon }(\mathbf{A},\mathbf{b})$ must also be bounds for ${\mathbf{x}}^{\ast }$.

Our results (which will be given formally in the sequel as Thm. 1 and its corollaries) will allow us to make useful statements such as: If $\mathbf{x}\in {\Gamma }_{\epsilon }(\mathbf{A},\mathbf{b})$, then $x_i$, the ith entry of $\mathbf{x}$, must belong to a fixed interval $[L_i,U_i]\subset \mathbb{R}$, where the values of $L_i$, and $U_i$ can be computed explictly based on the values of $\mathbf{A}$, $\mathbf{b}$, $\epsilon$, and $i$. The values we obtain for $L_i$ and $U_i$ are also tight in the sense that we can easily identify specific values of $\mathbf{x}\in {\Gamma }_{\epsilon }(\mathbf{A},\mathbf{b})$ that achieve the minimal and maximal values of $x_i$. We are not aware of existing characterizations of this form, and we believe they enable fundamental new insights into the ambiguities associated with ill-posed inverse problems.

In addition, our results also allow us to define an entrywise condition number ${\kappa }_i(\mathbf{A})$ for the $i\text{th}$ entry of $\mathbf{x}$. This entrywise condition number (given formally in Corollary 4) will be used in a similar way the condition number $\kappa (\mathbf{A})$ from Eq. (4), but will provide bounds that are generally less pessimistic than Eq. (4) (with ${\kappa }_i(\mathbf{A})\le \kappa (\mathbf{A})$) and which explicitly capture the fact that some entries of $\mathbf{x}$ can be substantially better conditioned than others.

During the final stages of writing this paper, we became aware of mathematical literature involving componentwise condition numbers [17]-[21]. Componentwise theory is similar to our proposed entrywise theory in the sense that they both focus on the individual entries of the vector $\mathbf{x}$. However, there are also important differences, including the fact that componentwise theory is frequently used to obtain a single global bound that summarizes the worst-case relative error across all of the individual entries of the vector, while we provide a finer-grained analysis by defining seperate/distinct bounds for each of the $N$ entries of $\mathbf{x}$ individually. One exception to this is the componentwise theory by Chandrasekaran and Ipsen [19], who, like us, also focus on the individual entries of $\mathbf{x}$. Interestingly, Chandrasekaran and Ipsen also describe a version of a componentwise condition number that happens to perfectly match the definition of the entrywise condition number proposed in this work. Nevertheless, the literature on componentwise theory (including Ref. [19]) generally considers different problem setups, asks different kinds of questions, and uses very different proof techniques compared to our work, offering perspectives that are both distinct and complementary.

This paper is organized as follows. We start by introducing some additional notation in Sec. II. This is followed by a presentation of our principal theoretical results in Sec. III. To illustrate the usefulness of our theory, we apply it to characterize an inverse problem from a biomedical imaging application in Sec. IV. We conclude the paper with a discussion of several additional topics in Sec. V, including practical computational considerations for large problem sizes, connections between our proposed bounds and Cramér-Rao bounds under white Gaussian noise assumptions, and interesting potential extensions to common scenarios where additional constraints are available beyond just near data-consistency (e.g., sparsity constraints, low-rank constraints, manifold constraints, etc.).

II. Notation

In addition to the notation that we have already introduced, the following notation is also used throughout the paper. We use $ℛ(\mathbf{A})$ and $𝒩(\mathbf{A})$ to respectively denote the range space and nullspace of the matrix $\mathbf{A}$. For a subspace $𝒮$, we use ${𝒮}^{\perp }$ to denote its orthogonal complement, use ${\mathbf{P}}_{𝒮}$ to denote its orthogonal projection matrix, and use $\text{dim}(𝒮)$ to denote its dimension.

For a matrix $\mathbf{A}\in {\mathbb{R}}^{M\times N}$ of rank $r$, we can use the singular value decomposition (SVD) to express $\mathbf{A}$ as $\mathbf{A}=\mathbf{U}\Sigma {\mathbf{V}}^T$, where $\mathbf{U}\in {\mathbb{R}}^{M\times r}$ and $\mathbf{V}\in {\mathbb{R}}^{N\times r}$ are matrices with orthonormal columns (corresponding to the left and right singular vectors, respectively), and $\Sigma \in {\mathbb{R}}^{r\times r}$ is a diagonal matrix with $i\text{th}$ diagonal element equal to ${\sigma }_i(\mathbf{A})$. The columns of $\mathbf{U}$ form an orthonormal basis for $ℛ(\mathbf{A})$, while the columns of $\mathbf{V}$ form an orthonormal basis for $ℛ({\mathbf{A}}^T)$. We can also write the matrix $\mathbf{A}$ using the extended SVD as

$$\mathbf{A}=[\mathbf{U}{\mathbf{U}}_{\perp }]\left[\begin{array}{cc}\hfill \Sigma \hfill & \hfill 0\hfill \\ \hfill 0\hfill & \hfill 0\hfill \end{array}\right]\left[\begin{array}{c}\hfill {\mathbf{V}}^T\hfill \\ \hfill {\mathbf{V}}_{\perp }^T\hfill \end{array}\right],$$ (6)

where ${\mathbf{U}}_{\perp }\in {\mathbb{R}}^{M\times (M-r)}$ and ${\mathbf{V}}_{\perp }\in {\mathbb{R}}^{N\times (N-r)}$ are also matrices with orthonormal columns. The columns of ${\mathbf{U}}_{\perp }$ form an orthonormal basis for ${ℛ}^{\perp }(\mathbf{A})=𝒩({\mathbf{A}}^T)$ and satisfy ${\mathbf{U}}^T{\mathbf{U}}_{\perp }=0$, while the columns of ${\mathbf{V}}_{\perp }$ form an orthonormal basis for ${ℛ}^{\perp }(\mathbf{A})=𝒩(\mathbf{A})$ and satisfy ${\mathbf{V}}^T{\mathbf{V}}_{\perp }=0$.

We use ${\mathbf{I}}_T$ to denote the $T\times T$ identity matrix. We denote the Moore-Penrose pseudoinverse of the matrix $\mathbf{A}$ as ${\mathbf{A}}^{†}=\mathbf{V}{\Sigma }^{-1}{\mathbf{U}}^T$. If $\mathbf{A}$ is square and nonsingular, then ${\mathbf{A}}^{†}={\mathbf{A}}^{-1}$. If $\mathbf{A}$ has full column rank, then ${\mathbf{A}}^{†}=({\mathbf{A}}^T\mathbf{A}{)}^{-1}{\mathbf{A}}^T$. Note that ${\mathbf{P}}_{ℛ(\mathbf{A})}={\mathbf{UU}}^T=\mathbf{A}{\mathbf{A}}^{†}$, and that ${\mathbf{P}}_{{ℛ}^{\perp }(\mathbf{A})}={\mathbf{I}}_M-\mathbf{A}{\mathbf{A}}^{†}$.

For the positive semidefinite matrix ${\mathbf{A}}^T\mathbf{A}=\mathbf{V}{\Sigma }^2{\mathbf{V}}^T$, we define its square-root as $({\mathbf{A}}^T\mathbf{A}{)}^{\frac{1}{2}}\triangleq \mathbf{V}\Sigma {\mathbf{V}}^T$.

We will use ${\mathbf{e}}_i\in {\mathbb{R}}^N$ to denote the vector whose $i\text{th}$ entry is equal to one, with all the other entries equal to zero (i.e., ${\mathbf{e}}_i$ is the $i\text{th}$ column of ${\mathbf{I}}_N$).

III. Main Results

Our main theoretical results are given by the following theorem and its corollaries.

Theorem 1: Let ${\Phi }_{\epsilon }^i(\mathbf{A},\mathbf{b})$ denote the set of all possible values for $x_i$ (the $i\text{th}$ entry of $\mathbf{x}$) for vectors $\mathbf{x}$ that belong to the set of nearly data-consistent solutions ${\Gamma }_{\epsilon }(\mathbf{A},\mathbf{b})$:

$${\Phi }_{\epsilon }^i(\mathbf{A},\mathbf{b})=\left\{\begin{array}{c}\hfill x_i\in \mathbb{R}\mathrm{s}.\mathrm{t}.x_i={\mathbf{e}}_i^T\mathbf{x}\hfill \\ \hfill \text{for some}\mathbf{x}\in {\Gamma }_{\epsilon }(\mathbf{A},\mathbf{b})\hfill \end{array}\right\}.$$

If ${\mathbf{e}}_i$ is orthogonal to $𝒩(\mathbf{A})$, then ${\Phi }_{\epsilon }^i(\mathbf{A},\mathbf{b})$ is the finite interval ${\Phi }_{\epsilon }^i(\mathbf{A},\mathbf{b})=[L_i,U_i]\subset \mathbb{R}$, with

$$\begin{gathered} L_i=\underset{\mathbf{x}\in {\mathbb{R}}^N}{\text{min}}{\mathbf{e}}_i^T\mathbf{x}\mathrm{s}.\mathrm{t}.\Vert \mathbf{A}\mathbf{x}-\mathbf{b}{\Vert }_2\le \epsilon \\ ={\mathbf{e}}_i^T{\mathbf{A}}^{†}\mathbf{b}-{\Vert ({\mathbf{A}}^{†}{)}^T{\mathbf{e}}_i\Vert }_2\sqrt{{\epsilon }^2-\Vert {\mathbf{P}}_{{ℛ}^{\perp }(\mathbf{A})}\mathbf{b}{\Vert }_2^2}, \\ {U}_i=\underset{\mathbf{x}\in {\mathbb{R}}^N}{\max }{\mathbf{e}}_i^T\mathbf{x}\mathrm{s}.\mathrm{t}.\Vert \mathbf{A}\mathbf{x}-\mathbf{b}{\Vert }_2\le \epsilon \\ ={\mathbf{e}}_i^T{\mathbf{A}}^{†}\mathbf{b}+{\Vert ({\mathbf{A}}^{†}{)}^T{\mathbf{e}}_i\Vert }_2\sqrt{{\epsilon }^2-\Vert {\mathbf{P}}_{{ℛ}^{\perp }(\mathbf{A})}\mathbf{b}{\Vert }_2^2}, \end{gathered}$$

and midpoint $\frac{1}{2}(L_i+U_i)={\mathbf{e}}_i^T{\mathbf{A}}^{†}\mathbf{b}$. Conversely, if ${\mathbf{e}}_i$ is not orthogonal to $𝒩(\mathbf{A})$, then ${\Phi }_{\epsilon }^i(\mathbf{A},\mathbf{b})$ is unbounded with ${\Phi }_{\epsilon }^i(\mathbf{A},\mathbf{b})=\mathbb{R}$.

This theorem is valid for arbitrary $\mathbf{A}$, $\mathbf{b}$, and $\epsilon$, and shows that, depending on how the nullspace of the matrix $\mathbf{A}$ interacts with the vector ${\mathbf{e}}_i$, the value of $x_i$ is either restricted to a known interval of length $2{\Vert ({\mathbf{A}}^{†}{)}^T{\mathbf{e}}_i\Vert }_2\sqrt{{\epsilon }^2-\Vert {\mathbf{P}}_{{ℛ}^{\perp }(\mathbf{A})}\mathbf{b}{\Vert }_2^2}$ or is entirely unconstrained and can be an arbitrary real number.

While Thm. 1 applies to the general case, simplifications occur under assumptions about the rank of $\mathbf{A}$. specifically:

• If $\mathbf{A}$ has full column rank (i.e., $\text{rank}(\mathbf{A})=N$), then ${\mathbf{e}}_i$ is always orthogonal to $𝒩(\mathbf{A})$, and ${\Phi }_{\epsilon }^i(\mathbf{A},\mathbf{b})$ is always a finite interval.

• If $\mathbf{A}$ has full row rank (i.e., $\text{rank}(\mathbf{A})=M$), then $\Vert {\mathbf{P}}_{{ℛ}^{\perp }(\mathbf{A})}\mathbf{b}{\Vert }_2$ is always zero. This implies that ${\Phi }_{\epsilon }^i(\mathbf{A},\mathbf{b})$ is a finite interval of length $2\epsilon {\Vert ({\mathbf{A}}^{†}{)}^T{\mathbf{e}}_i\Vert }_2$ when ${\mathbf{e}}_i$ is orthogonal to $𝒩(\mathbf{A})$, and is unbounded with ${\Phi }_{\epsilon }^i(\mathbf{A},\mathbf{b})=\mathbb{R}$ when ${\mathbf{e}}_i$ is not orthogonal to $𝒩(\mathbf{A})$.

• If $\mathbf{A}$ is square and non-singular (i.e., $\text{rank}(\mathbf{A})=N=M$), then ${\Phi }_{\epsilon }^i(\mathbf{A},\mathbf{b})$ is always the finite interval of length $2\epsilon {\Vert ({\mathbf{A}}^{-1}{)}^T{\mathbf{e}}_i\Vert }_2$ with midpoint ${\mathbf{e}}_i^T{\mathbf{A}}^{-1}\mathbf{b}$.

It should also be noted that Thm. 1 will still be valid if the vector ${\mathbf{e}}_i$ is substituted everywhere with an arbitrary nonzero vector $\mathbf{w}\in {\mathbb{R}}^N$. This allows the use of these kinds of bounds in the case where we are potentially interested in the range of values for a weighted linear combination of the entries of $\mathbf{x}$ in the form ${\mathbf{w}}^T\mathbf{x}$. For example, in practical imaging applications, if voxel $x_i$ appears to have a larger value than a neighboring voxel $x_j$ in the reconstructed image, it may be worthwhile to know tight upper and lower bounds on the value of $x_i-x_j$ across all $\mathbf{x}\in {\Gamma }_{\epsilon }(\mathbf{A},\mathbf{b})$, which would allow direct insight into whether voxel $x_i$ is always larger than voxel $x_j$ within the class of nearly data-consistent images. This could easily be achieved using the weight vector $\mathbf{w}={\mathbf{e}}_i-{\mathbf{e}}_j$.

Before giving the proof of Thm. 1, we will first remark that minor manipulations of Thm. 1 also lead to the following corollaries. (As with Thm. 1, these corollaries are also valid if the vector ${\mathbf{e}}_i$ is replaced everywhere with an arbitrary nonzero vector $\mathbf{w}\in {\mathbb{R}}^N$).

Corollary 1: Assume that noisy data $\mathbf{b}$ is measured according to Eq. (1), and that the noise is known to obey $\Vert \mathbf{n}{\Vert }_2\le \epsilon$. Let $x_i^{\ast }\triangleq {\mathbf{e}}_i^T{\mathbf{x}}^{\ast }$ denote the $i\text{th}$ entry of the original true vector ${\mathbf{x}}^{\ast }$. Then $x_i^{\ast }\in {\Phi }_{\epsilon }^i(\mathbf{A},\mathbf{b})$, with ${\Phi }_{\epsilon }^i(\mathbf{A},\mathbf{b})$ defined as in Thm. 1.

Corollary 2: Assume that noisy data $\mathbf{b}$ is measured according to Eq. (1), and that the noise is known to obey $\Vert \mathbf{n}{\Vert }_2\le \epsilon$. Also assume that an estimation procedure is used to obtain an estimate $\widehat{\mathbf{x}}$ of ${\mathbf{x}}^{\ast }$, and that this estimate is nearly data-consistent such that $\widehat{\mathbf{x}}\in {\Gamma }_{\epsilon }(\mathbf{A},\mathbf{b})$. Let ${\widehat{x}}_i\triangleq {\mathbf{e}}_i^T\widehat{\mathbf{x}}$ and $x_i^{\ast }\triangleq {\mathbf{e}}_i^T{\mathbf{x}}^{\ast }$ denote the $i\text{th}$ entries of the estimate and the original true vector, respectively. Then if ${\mathbf{e}}_i$ is orthogonal to $𝒩(\mathbf{A})$, we must have

$$\begin{gathered} \mid {\widehat{x}}_i-x_i^{\ast }\mid \le {\Vert ({\mathbf{A}}^{†}{)}^T{\mathbf{e}}_i\Vert }_2\sqrt{{\epsilon }^2-\Vert {\mathbf{P}}_{{ℛ}^{\perp }(\mathbf{A})}\mathbf{b}{\Vert }_2^2} \\ +\mid {\widehat{x}}_i-{\mathbf{e}}_i^T{\mathbf{A}}^{†}\mathbf{b}\mid \\ \le 2{\Vert ({\mathbf{A}}^{†}{)}^T{\mathbf{e}}_i\Vert }_2\sqrt{{\epsilon }^2-\Vert {\mathbf{P}}_{{ℛ}^{\perp }(\mathbf{A})}\mathbf{b}{\Vert }_2^2}. \end{gathered}$$

Corollary 3: Assume that noisy data $\mathbf{b}$ is measured according to Eq. (1), that $\mathbf{A}$ has full column rank, and that $\widehat{\mathbf{x}}$ is obtained using $\widehat{\mathbf{x}}={\mathbf{A}}^{†}\mathbf{b}$. Then

$$\begin{gathered} \mid {\widehat{x}}_i-x_i^{\ast }\mid \le {\Vert ({\mathbf{A}}^{†}{)}^T{\mathbf{e}}_i\Vert }_2\sqrt{\Vert \mathbf{n}{\Vert }_2^2-\Vert {\mathbf{P}}_{{ℛ}^{\perp }(\mathbf{A})}\mathbf{b}{\Vert }_2^2} \\ \le {\Vert ({\mathbf{A}}^{†}{)}^T{\mathbf{e}}_i\Vert }_2\Vert \mathbf{n}{\Vert }_2. \end{gathered}$$

Note that Cor. 3 can be viewed an entrywise analogue of the spectral norm bound from Eq. (3). Moreover, Cor. 3 provides a bound that is generally better and never worse than Eq. (3), since we always have that ${\Vert ({\mathbf{A}}^{†}{)}^T{\mathbf{e}}_i\Vert }_2\le {\sigma }_N^{-1}(\mathbf{A})$.

Corollary 4: Assume that noisy data $\mathbf{b}$ is measured according to Eq. (1), that $\mathbf{A}$ has full column rank, and that $\widehat{\mathbf{x}}$ is obtained using $\widehat{\mathbf{x}}={\mathbf{A}}^{†}\mathbf{b}$. Then

$$\begin{gathered} \frac{\mid {\widehat{x}}_i-x_i^{\ast }\mid }{\Vert {\mathbf{x}}^{\ast }{\Vert }_2}\le {\kappa }_i(\mathbf{A})\frac{\sqrt{\Vert \mathbf{n}{\Vert }_2^2-\Vert {\mathbf{P}}_{{ℛ}^{\perp }(\mathbf{A})}\mathbf{b}{\Vert }_2^2}}{\Vert {\mathbf{b}}^{\ast }{\Vert }_2} \\ \le {\kappa }_i(\mathbf{A})\frac{\Vert \mathbf{n}{\Vert }_2}{\Vert {\mathbf{b}}^{\ast }{\Vert }_2}, \end{gathered}$$

where we define ${\kappa }_i(\mathbf{A})\triangleq {\Vert ({\mathbf{A}}^{†}{)}^T{\mathbf{e}}_i\Vert }_2{\sigma }_1(\mathbf{A})$ as the entrywise condition number for the $i\text{th}$ entry of $\mathbf{x}$.

Note that Cor. 4 can be viewed as an entrywise analogue of the condition number bound from Eq. (4). In addition, Cor. 4 provides a bound that is generally better and never worse than Eq. (4), since we always have that ${\kappa }_i(\mathbf{A})\le \kappa (\mathbf{A})$ (which can be established from the fact that ${\Vert ({\mathbf{A}}^{†}{)}^T{\mathbf{e}}_i\Vert }_2\le {\sigma }_N^{-1}(\mathbf{A})$.

The proof of Thm. 1 is given below.

Proof of Thm. 1

We first decompose $\mathbf{b}$ as $\mathbf{b}={\mathbf{P}}_{ℛ(\mathbf{A})}\mathbf{b}+{\mathbf{P}}_{{ℛ}^{\perp }(\mathbf{A})}\mathbf{b}={\mathbf{AA}}^{†}\mathbf{b}+{\mathbf{P}}_{{ℛ}^{\perp }(\mathbf{A})}\mathbf{b}$. Due to orthogonality between $ℛ(\mathbf{A})$ and ${ℛ}^{\perp }(\mathbf{A})$ (invoking the Pythagorean theorem for ${\mathbb{R}}^N$), we can rewrite ${\Gamma }_{\epsilon }(\mathbf{A},\mathbf{b})$ as

$$\begin{gathered} {\Gamma }_{\epsilon }(\mathbf{A},\mathbf{b})=\left\{\begin{array}{c}\mathbf{x}\in {\mathbb{R}}^N\hfill \\ \hfill \mathrm{s}.\mathrm{t}.\Vert \mathbf{A}\mathbf{x}-\mathbf{A}{\mathbf{A}}^{†}\mathbf{b}{\Vert }_2^2+\Vert {\mathbf{P}}_{{ℛ}^{\perp }(\mathbf{A})}\mathbf{b}{\Vert }_2^2\le {\epsilon }^2\hfill \end{array}\right\} \\ =\{\mathbf{x}\in {\mathbb{R}}^N\mathrm{s}.\mathrm{t}.\Vert \mathbf{A}\mathbf{x}-\mathbf{A}\mathbf{z}{\Vert }_2^2\le {\lambda }^2\}, \end{gathered}$$ (7)

with ${\lambda }^2\triangleq {\epsilon }^2-\Vert {\mathbf{P}}_{{ℛ}^{\perp }(\mathbf{A})}\mathbf{b}{\Vert }_2^2$ and $\mathbf{z}\triangleq {\mathbf{A}}^{†}\mathbf{b}$. Note that $\Vert {\mathbf{P}}_{{ℛ}^{\perp }(\mathbf{A})}\mathbf{b}{\Vert }_2$ must be smaller than $\epsilon$ for ${\Gamma }_{\epsilon }(\mathbf{A},\mathbf{b})$ to be a nonempty set.

Using the definition of the ${\ell }_2$-norm, we can simplify the previous expression to

$$\begin{gathered} {\Gamma }_{\epsilon }(\mathbf{A},\mathbf{b})=\left\{\begin{array}{c}\mathbf{x}\in {\mathbb{R}}^N \\ \hfill \\ \hfill \mathrm{s}.\mathrm{t}.{\Vert {\left(\frac{{\mathbf{A}}^T\mathbf{A}}{{\lambda }^2}\right)}^{\frac{1}{2}}(\mathbf{x}-\mathbf{z})\Vert }_2^2\le 1\hfill \end{array}\right\}. \end{gathered}$$ (8)

This set has the form of an ellipsoid [22], and it is well-known that if ${\mathbf{A}}^T\mathbf{A}$ is nonsingular, then the volume of this ellipsoid is [23]

$$\frac{{\pi }^{\frac{N}{2}{\lambda }^N}}{\Gamma (\frac{N}{2}+1)}\sqrt{\mid ({\mathbf{A}}^T\mathbf{A}{)}^{-1}\mid },$$ (9)

where $\Gamma (\cdot )$ denotes the gamma function (and should not be confused with the set ${\Gamma }_{\epsilon }(\mathbf{A},\mathbf{b})$). Clearly, if the volume of this ellipsoid is large, then there will be many vectors that are nearly data-consistent, and the solution to the inverse problem will have more potential ambiguity. If ${\mathbf{A}}^T\mathbf{A}$ were singular, then the ellipsoid is degenerate and must extend infinitely along directions aligned with the nullspace of $\mathbf{A}$, causing ${\Gamma }_{\epsilon }(\mathbf{A},\mathbf{b})$ to have infinite volume. However, these statements only provide insight into the global ambiguity of the solution, and our goal in this work is to characterize the ambiguities associated with the individual entries of the elements of ${\Gamma }_{\epsilon }(\mathbf{A},\mathbf{b})$, which may be much better for some entries than might be interpreted based on a global characterization.

Invoking the SVD $\mathbf{A}=\mathbf{U}\Sigma {\mathbf{V}}^T$, we have

$${\Gamma }_{\epsilon }(\mathbf{A},\mathbf{b})=\left\{\mathbf{x}\in {\mathbb{R}}^N\mathrm{s}.\mathrm{t}.{\Vert \frac{\Sigma {\mathbf{V}}^T}{\lambda }(\mathbf{x}-\mathbf{z})\Vert }_2^2\le 1\right\}.$$ (10)

Furthermore, because of the orthonormality and mutual orthogonality properties of $\mathbf{V}$ and ${\mathbf{V}}_{\perp }$, it is possible to uniquely represent an arbitrary vector $\mathbf{x}\in {\mathbb{R}}^N$ as $\mathbf{x}=\mathbf{V}\mathbf{p}+{\mathbf{V}}_{\perp }\mathbf{q}$, where $\mathbf{p}\in {\mathbb{R}}^r$ with $\mathbf{p}={\mathbf{V}}^T\mathbf{x}$, and $\mathbf{q}\in {\mathbb{R}}^{N-r}$ with $\mathbf{q}={\mathbf{V}}_{\perp }^T\mathbf{x}$. We can then write

$${\Gamma }_{\epsilon }(\mathbf{A},\mathbf{b})=\left\{\begin{array}{c}\hfill \mathbf{x}=\mathbf{V}\mathbf{p}+{\mathbf{V}}_{\perp }\mathbf{q}\hfill \\ \hfill \mathrm{s}.\mathrm{t}.\mathbf{p}\in {\mathbb{R}}^r,\mathbf{q}\in {\mathbb{R}}^{N-r},\hfill \\ \hfill \text{and}{\Vert \frac{\Sigma }{\lambda }(\mathbf{p}-{\mathbf{V}}^T\mathbf{z})\Vert }_2^2\le 1\hfill \end{array}\right\}.$$ (11)

A simple change of variables then results in

$${\Gamma }_{\epsilon }(\mathbf{A},\mathbf{b})=\left\{\begin{array}{c}\hfill \mathbf{x}=\lambda \mathbf{V}{\Sigma }^{-1}\mathbf{p}+{\mathbf{V}}_{\perp }\mathbf{q}+\mathbf{z}\hfill \\ \hfill \mathrm{s}.\mathrm{t}.\mathbf{p}\in {\mathbb{R}}^r,\mathbf{q}\in {\mathbb{R}}^{N-r},\hfill \\ \hfill \text{and}\Vert \mathbf{p}{\Vert }_2^2\le 1\hfill \end{array}\right\}.$$ (12)

Notably, the component $\mathbf{q}$ (which corresponds to the part of $\mathbf{x}$ that is in the nullspace of $\mathbf{A}$) is completely unconstrained, and can be arbitrarily large. This results in a degenerate ellipsoid as described previously.

Given this geometric characterization of the set of nearly data-consistent solutions, we can now consider the behavior of the individual entries of $\mathbf{x}$. Substituting the results of Eq. (12) into the definition of ${\Phi }_{\epsilon }^i(\mathbf{A},\mathbf{b})$ as given in Thm 1, we obtain

$${\Phi }_{\epsilon }^i(\mathbf{A},\mathbf{b})=\left\{\begin{array}{c}\hfill x_i=\lambda {\mathbf{e}}_i^T\mathbf{V}{\Sigma }^{-1}\mathbf{p}+{\mathbf{e}}_i^T{\mathbf{V}}_{\perp }\mathbf{q}+{\mathbf{e}}_i^T\mathbf{z}\hfill \\ \hfill \mathrm{s}.\mathrm{t}.\mathbf{p}\in {\mathbb{R}}^r,\mathbf{q}\in {\mathbb{R}}^{N-r},\hfill \\ \hfill \text{and}\Vert \mathbf{p}{\Vert }_2^2\le 1\hfill \end{array}\right\}.$$ (13)

At this point, there are two cases to consider.

1). Case 1:

First, consider the case where ${\mathbf{e}}_i$ is not orthogonal to $𝒩(\mathbf{A})$, such that ${\mathbf{e}}_i^T{\mathbf{V}}_{\perp }\ne 0$. Then, because $\mathbf{q}$ is unconstrained, we must have

$${\Phi }_{\epsilon }^i(\mathbf{A},\mathbf{b})=\mathbb{R},$$ (14)

indicating that the $i\text{th}$ entry of $\mathbf{x}$ is unbounded and could take on arbitrary values. In particular, for an arbitrary $\alpha \in \mathbb{R}$, we can achieve $\alpha =x_i={\mathbf{e}}_i^T\mathbf{x}$ for $\mathbf{x}\in {\Gamma }_{\epsilon }(\mathbf{A},\mathbf{b})$ by choosing $\mathbf{x}={\mathbf{V}}_{\perp }\mathbf{q}+\mathbf{z}$ with $\mathbf{q}=(\alpha -{\mathbf{e}}_i^T\mathbf{z}){\mathbf{V}}_{\perp }^T{\mathbf{e}}_i∕{\Vert {\mathbf{V}}_{\perp }^T{\mathbf{e}}_i\Vert }_2^2$.

2). Case 2:

In the second case, assume that ${\mathbf{e}}_i$ is orthogonal to $𝒩(\mathbf{A})$ such that ${\mathbf{e}}_i^T{\mathbf{V}}_{\perp }=0$. In this case, we have the simplification

$${\Phi }_{\epsilon }^i(\mathbf{A},\mathbf{b})=\left\{\begin{array}{c}\hfill x_i=\lambda {\mathbf{e}}_i^T\mathbf{V}{\Sigma }^{-1}\mathbf{p}+{\mathbf{e}}_i^T\mathbf{z}\hfill \\ \hfill \mathrm{s}.\mathrm{t}.\mathbf{p}\in {\mathbb{R}}^r\text{and}\Vert \mathbf{p}{\Vert }_2^2\le 1\hfill \end{array}\right\}.$$ (15)

The maximum and minimum possible values of $\lambda {\mathbf{e}}_i^T\mathbf{V}{\Sigma }^{-1}\mathbf{p}$ under the constraint that $\Vert \mathbf{p}{\Vert }_2^2\le 1$ are respectively given by $\pm {\sigma }_1\left(\lambda {\mathbf{e}}_i^T\mathbf{V}{\Sigma }^{-1}\right)=\pm \lambda \Vert {\Sigma }^{-1}{\mathbf{V}}^T{\mathbf{e}}_i{\Vert }_2=\pm \lambda {\Vert ({\mathbf{A}}^{†}{)}^T{\mathbf{e}}_i\Vert }_2$. Note also that the extremal values of $\lambda {\mathbf{e}}_i^T\mathbf{V}{\Sigma }^{-1}\mathbf{p}$ are achieved by taking $\mathbf{p}=\pm {\Sigma }^{-1}{\mathbf{V}}^T{\mathbf{e}}_i∕{\Vert {\Sigma }^{-1}{\mathbf{V}}^T{\mathbf{e}}_i\Vert }_2$, which satisfies $\Vert \mathbf{p}{\Vert }_2=1$, and that simple rescaling of this choice of $\mathbf{p}$ by a real-scalar with modulus less than one will allow us to achieve any real value in between the upper and lower bounds. We therefore find that ${\Phi }_{\epsilon }^i(\mathbf{A},\mathbf{b})$ is simply the finite interval ${\Phi }_{\epsilon }^i(\mathbf{A},\mathbf{b})=[L_i,U_i]\subset \mathbb{R}$, with $L_i$ and $U_i$ as given in the statement of the theorem.

This completes the proof of Thm. 1. □

While most of the corollaries are trivial manipulations of Thm. 1, we will note that the derivation of Cor. 4 is based on the combination of Cor. 3 with the fact that $\Vert {\mathbf{b}}^{\ast }{\Vert }_2=\Vert \mathbf{A}{\mathbf{x}}^{\ast }{\Vert }_2\le {\sigma }_1(\mathbf{A})\Vert {\mathbf{x}}^{\ast }{\Vert }_2$.

IV. Illustrative Example

We illustrate the utility of our new theoretical results by applying them to an example application: image reconstruction in multi-channel magnetic resonance imaging (MRI). Specifically, under the sensitivity encoding model of multi-channel MRI [11], [24], we assume that we are interested in reconstructing a discrete image defined on an $N$-voxel grid, where the value of the image at each voxel is complex-valued. We use $\mathbf{x}\in {\mathbb{C}}^N$ to represent the unknown voxel values. Data measurements are then obtained from this image from a set of $L$ channels, where the data ${\mathbf{b}}_{\ell }$ from the $\ell \text{th}$ channel is represented as

$${\mathbf{b}}_{\ell }={\mathbf{FS}}_{\ell }\mathbf{x}+{\mathbf{n}}_{\ell },\text{for}\ell =1,\dots ,L,$$ (16)

where ${\mathbf{n}}_{\ell }$ represents the noise for the $\ell \text{th}$ channel, ${\mathbf{S}}_{\ell }\in {\mathbb{C}}^{N\times N}$ is a diagonal matrix whose diagonal entries are equal to the corresponding values of the spatially-varying sensitivity profile for the $\ell \text{th}$ channel, and $\mathbf{F}\in {\mathbb{C}}^{P\times N}$ represents an operator that performs Fourier transformation followed by some form of sampling. Typically, the sampling rate is chosen to be below the Nyquist rate to allow accelerated data acquisition, resulting in $P<N$. This allows us to represent the inverse problem as

$$\mathbf{b}=\mathbf{A}\mathbf{x}+\mathbf{n},$$ (17)

where $\mathbf{b}\in {\mathbb{C}}^M$ is the vertical concatenation of the ${\mathbf{b}}_{\ell }$ vectors, $\mathbf{n}\in {\mathbb{C}}^M$ is the vertical concatenation of the ${\mathbf{n}}_{\ell }$ vectors, and $\mathbf{A}\in {\mathbb{C}}^{M\times N}$ is the vertical concatenation of the $\mathbf{F}{\mathbf{S}}_{\ell }$ matrices, with $M=LP$.

An important observation is that the matrices and vectors in our example inverse problem are all complex-valued, while our theoretical results are all given assuming the real-valued case. We chose to present our theory for the real-valued case because there exists a standard ordering for real-numbers that enables a simple discussion of lower- and upper-bounds, while the complex-valued case would be much more complicated. However, in order to apply our theory to this scenario, we need to transform the complex-valued problem statement into an equivalent real-valued problem statement. This is easily done by separating each complex-value into its real- and imaginary-components, and then working with an equivalent purely-real expression of the inverse problem (see, e.g., [25], [26]). We have used this real-valued transformation approach in the results that follow. For simplicity and to avoid introducing new notation, we will simply refer to these now real-valued variables using the same notation we used for the complex-valued case (i.e., $\mathbf{x}$, $\mathbf{A}$, $\mathbf{b}$, etc.). From this point forward, all variables should now be interpreted as real-valued.

In our illustrative example application, we use real fully-sampled 32-channel MRI brain data acquired at our institution, which we retrospectively subsample to emulate an accelerated data acquisition procedure as illustrated in Fig. 1, with $P\approx N∕3.3$. Note that even though the matrix $\mathbf{F}$ is underdetermined because $P<N$, the overall system $\mathbf{A}$ is still overdetermined because $LP>N$. We have determined numerically that the matrix $\mathbf{A}$ has full column rank.

Fig. 1.

The (a) original fully-sampled 32-channel MRI brain data and (b) the retrospectively subsampled version of the data used for the illustrative MRI reconstruction example. The top row shows coil-combined magnitude images, with zero-filling of the missing Fourier data samples in the subsampled case. The bottom row shows the corresponding coil-combined Fourier magnitudes. The Fourier sampling pattern captures every fourth phase-encoding line, while also measuring the central (low-frequency) 24 phase encoding lines.

As can be seen in Fig. 1, our subsampling scheme is designed such that undersampling only occurs along one dimension of the 2D Fourier domain (i.e., the phase encoding dimension in MRI terminology), while the remaining dimension (i.e., the readout dimension in MRI terminology) is fully sampled at the Nyquist rate. This structure implies that one can apply a unitary inverse Fourier transform operator along the fully-sampled readout dimension as a preprocessing step to completely decouple the reconstruction of each image row [27]. Reconstructing each image row independently enables substantial reductions in computational complexity, and the results we present later in this section are all based on this row-decoupled approach.

In our illustrative example, we have used ESPIRiT [28] with circular neighborhoods [29] to estimate the sensitivity profiles needed to form the ${\mathbf{S}}_{\ell }$ matrices from the fully-sampled central Fourier data. In addition, we incorporate a phase estimate directly into the sensitivity maps so that the reconstructed image will be approximately real-valued [25], [30]-[33]. Our formulation also uses prior knowledge of the support of the image and does not estimate voxel locations that are already known to be zero, which is achieved by removing the corresponding columns from the $\mathbf{A}$ matrix [11]. We have also used direct measurements of the multi-channel thermal noise to prewhiten the channels, which allows all thermal noise samples to be treated as independent and identically distributed zero-mean circularly-symmetric complex Gaussian noise [24].

Our illustration will start by calculating the theoretical lower and upper bounds from Thm. 1 for this inverse problem. However, before this can be done, it is important to select an appropriate value of $\epsilon$. Ideally, the noise perturbation $\mathbf{n}$ in MRI would be purely due to thermal (white Gaussian) noise, such that the distribution of $\Vert \mathbf{n}{\Vert }_2$ would follow the chi distribution, which has a well-characterized cumulative distribution function (CDF). If this were the case, then a reasonable approach for selecting $\epsilon$ might be to choose $\epsilon$ based on the CDF, e.g., choose $\epsilon$ so that the probability of observing $\Vert \mathbf{n}{\Vert }_2>\epsilon$ is less than 1%. Unfortunately, this approach for determining $\epsilon$ turns out to be inappropriate for this data, since we observe empircally that we would have $\Vert {\mathbf{P}}_{{ℛ}^{\perp }(\mathbf{A})}\mathbf{b}{\Vert }_2>\epsilon$ under this approach to choosing $\epsilon$. The reason this approach fails is that we are working with real data, and there are many sources of error in the data beyond just the thermal noise, including minor motion of the human subject during the scan, flow of the subject’s blood and cerebrospinal fluid during the scan, and spin relaxation during the acquisition. All of these contribute to $\mathbf{n}$, causing it to be much larger than would be predicted from the statistics of thermal noise. This example also helps to underscore the point from the introduction that it can be important to have theoretical bounds that are not heavily dependent on statistical noise modeling assumptions, since these assumptions can fail for practical real-world data.

To overcome this practical issue, we have used a heuristic (and likely suboptimal) approach for choosing $\epsilon$, that is nevertheless good enough for our illustration despite its imperfections. Specifically, we observe that, based on our inverse problem model from Eq. (1), the vector ${\mathbf{P}}_{{ℛ}^{\perp }(\mathbf{A})}\mathbf{b}$ should not contain any of the true signal, and consists entirely of the projection of the noise $\mathbf{n}$ onto the subspace ${ℛ}^{\perp }(\mathbf{A})$. If we make the assumption that the noise energy $\mathbf{n}$ distributes relatively evenly across all subspaces of ${\mathbb{R}}^M$ and that $\text{dim}({\mathbb{R}}^M)=M$ and $\text{dim}({ℛ}^{\perp }(\mathbf{A}))=M-N$ are large enough that concentration of measure principles can be applied, then we might reasonably expect that

$$\begin{gathered} \frac{\Vert \mathbf{n}{\Vert }_2^2}{\text{dim}({\mathbb{R}}^M)}\approx \frac{\Vert {\mathbf{P}}_{{ℛ}^{\perp }(\mathbf{A})}\mathbf{b}{\Vert }_2^2}{\text{dim}({ℛ}^{\perp }(\mathbf{A}))} \\ \Rightarrow \Vert \mathbf{n}{\Vert }_2\approx \sqrt{\frac{M}{M-N}}\Vert {\mathbf{P}}_{{ℛ}^{\perp }(\mathbf{A})}\mathbf{b}{\Vert }_2. \end{gathered}$$ (18)

In what follows, we therefore set $\epsilon =\sqrt{\frac{M}{M-N}}\Vert {\mathbf{P}}_{{ℛ}^{\perp }(\mathbf{A})}\mathbf{b}{\Vert }_2$. Before moving on, we would like to reemphasize that this choice is heuristic and not rigorously justified, and many other ways of choosing $\epsilon$ could have been used instead. Different choices of $\epsilon$ would result in a very simple and predictable change in the lower- and upper-bounds $L_i$ and $U_i$, and would have no effect on other properties that may be of interest such as the entrywise spectral norm-type bound $\Vert ({\mathbf{A}}^{†}{)}^T{\mathbf{e}}_i{\Vert }_2$ from Cor. 3 or the entrywise condition number ${\kappa }_i(\mathbf{A})$ from Cor. 4.

Figure 2 depicts the entrywise lower- and upper-bounds we obtain on the real- and imaginary-parts of the image based on our choice of $\epsilon$. In this example, the lower-bounds and upper-bounds are relatively far separated from one another for most voxels of the images, suggesting that near data-consistency constraints themselves may be insufficient to guarantee accurate estimation of the image voxel amplitudes. Of course, it should also be kept in mind that these are worst-case bounds that are independent of any specific image estimation method, and that the practical performance of a specific estimator in the presence of typical (i.e., not worst-case) $\mathbf{n}$ may be better. On the other hand, the fact that there can be this level of variation within the set of near data-consistent images may also encourage an experimenter to consider the use of noise-mitigation strategies so that $\epsilon$ can be reduced, and/or the redesign of the data acquisition matrix $\mathbf{A}$ so that $\Vert ({\mathbf{A}}^{†}{)}^T{\mathbf{e}}_i{\Vert }_2$ can be made smaller.

Fig. 2.

Illustration of spatial maps of the entrywise lowerbounds $L_i$ and upperbounds $U_i$ for the (a,b) real and (c,d) imaginary parts of nearly data-consistent images $\mathbf{x}\in {\Gamma }_{\epsilon }(\mathbf{A},\mathbf{b})$, as obtained by applying Thm. 1 to the multi-channel MRI reconstruction scenario.

An important thing to keep in mind is that the spatial maps of $L_i$ and $U_i$ shown in Fig. 2 may happen to look like realistic MRI images, but they are not themselves elements of ${\Gamma }_{\epsilon }(\mathbf{A},\mathbf{b})$. In particular, these maps only provide entrywise bounds, and it is unlikely that these bounds could be achieved by all voxels simultaneously. To illustrate this point, Fig. 3 gives examples of two images belonging to ${\Gamma }_{\epsilon }(\mathbf{A},\mathbf{b})$ that achieve the upper- or lower-bounds for specific choices of voxels (using the technique described in the proof of Thm. 1 to identify the extremal cases).

Fig. 3.

Examples of two different nearly data-consistent images $\mathbf{x}\in {\Gamma }_{\epsilon }(\mathbf{A},\mathbf{b})$ that achieve the (a) upper- and (b) lower-bounds from Fig. 2 for a specific column of voxels (indicated with red arrows). For simplicity, only the real parts of the images are shown (the imaginary parts of both images are close to zero).

As mentioned in Sec. III, we can also replace ${\mathbf{e}}_i$ in Thm. 1 with a nonzero vector $\mathbf{w}\in {\mathbb{R}}^N$ to construct bounds on linear combinations of the entries of $\mathbf{x}$. We illustrate this point by showing spatial maps of the upper- and lower-bounds on the difference between adjacent voxels ($\mathbf{w}={\mathbf{e}}_i-{\mathbf{e}}_j$ as discussed in Sec. III, where ${\mathbf{e}}_i$ corresponds to the $i\text{th}$ voxel position in the image and ${\mathbf{e}}_j$ corresponds to the voxel immediately to the right of the $i\text{th}$ voxel) in Fig. 4. Interestingly, we observe that there are only a few voxels that are consistently larger or smaller than their neighbors (i.e., the lower-bounds and upper-bounds have consistent sign) within the set of nearly data-consistent solutions.

Fig. 4.

Spatial maps of the (a) lower-bounds $L_i$ and (b) upper-bounds $U_i$ for the difference between adjacent voxels, with $\mathbf{w}={\mathbf{e}}_i-{\mathbf{e}}_j$.

In our final set of illustrations, we compare the global bounds from Eqs. (3) and (4) against our new entrywise bounds from Cors. 3 and 4. Specifically, Fig. 5 focuses on the spectral norm-type bounds from Eq. (3) and Cor. 3, while Fig. 6 focuses on the condition number bounds from Eq. (4) and Cor. 4. In order to show spatial maps of the global bounds in these figures, we have made use of the equivalence of norms on finite dimensional spaces [1], [5]. Specifically, this equivalence implies that

$$\Vert \mathbf{x}{\Vert }_{\infty }\le \Vert \mathbf{x}{\Vert }_2\le \sqrt{N}\Vert \mathbf{x}{\Vert }_{\infty }$$ (19)

for arbitrary $\mathbf{x}\in {\mathbb{R}}^N$, where $\Vert \cdot {\Vert }_{\infty }$ is the standard infinity norm. This relationship immediately allows a global bound in the form of Eq. (2) to be converted into a bound on the individual elements of $\mathbf{x}$:

$$\Vert \widehat{\mathbf{x}}-{\mathbf{x}}^{\ast }{\Vert }_{\infty }\le C\Vert \mathbf{n}{\Vert }_2,$$ (20)

which allows us to depict spatial maps of the global bounds.

Fig. 5.

Spatial maps of the spectral norm-type (a) global bounds ${\sigma }_N^{-1}(\mathbf{A})$ and (b) entrywise bounds $\Vert ({\mathbf{A}}^{†}{)}^T{\mathbf{e}}_i{\Vert }_2$. Note that because we use decoupling to solve for each image row independently, we have a different global bound for each image row.

Fig. 6.

Spatial maps of the (a) traditional condition number $\kappa (\mathbf{A})$ and the (b) entrywise condition number ${\kappa }_i(\mathbf{A})$. Note that because we use decoupling to solve for each image row independently, we have a different condition numbers for each image row.

As can be seen from Figs. 5 and 6, our proposed entrywise bounds are usually better and never worse than the global bounds, consistent with our theoretical expectations. In addition, the entrywise characterizations both do a good job of capturing the fact that the values of certain voxels can be substantially less sensitive to noise perturbations than others, while the traditional global characterizations lack this nuance and are more pessimistic.

V. DISCUSSION

A. Efficient Computations for Large Problems

The new theoretical characterizations presented in this paper all depend on the ability to compute the numerical value of ${\Vert ({\mathbf{A}}^{†}{)}^T{\mathbf{e}}_i\Vert }_2$. When the problem size is small enough, it is possible to explicitly calculate the SVD of $\mathbf{A}$, at which point this calculation becomes trivial. However, in many practical scenarios, the matrix $\mathbf{A}$ is too big to be stored in memory, and it is necessary to work with functions that compute matrix-vector multiplications with $\mathbf{A}$ or ${\mathbf{A}}^T$ rather than working directly with the matrix $\mathbf{A}$.

In situations like this, there exist well-established iterative procedures that allow computation of the quantity ${\mathbf{A}}^{†}\mathbf{m}$ for arbitrary vectors $\mathbf{m}\in {\mathbb{R}}^M$ and arbitrary matrices $\mathbf{A}$. For example, starting from zero initialization ${\mathbf{x}}^{(0)}=0$, the Landweber iteration [2], [34]-[36]²

$${\mathbf{x}}^{(k)}={\mathbf{x}}^{(k-1)}-\tau {\mathbf{A}}^T(\mathbf{A}{\mathbf{x}}^{(k-1)}-\mathbf{m})$$ (21)

will satisfy ${\mathbf{x}}^{(k)}\to {\mathbf{A}}^{†}\mathbf{m}$ as $k\to \infty$ as long as $0<\tau <2∕{\sigma }_1^2(\mathbf{A})$. Furthermore, if the value of ${\sigma }_r(\mathbf{A})$ is known where $r$ is the rank of $\mathbf{A}$, then it is also possible to use this value to precompute the number of iterations $k$ that are needed so that ${\mathbf{x}}^{(k)}$ approximates ${\mathbf{A}}^{†}\mathbf{m}$ within a prescribed level of accuracy.³ This type of approach allows for simple computation of ${\mathbf{A}}^{†}{\mathbf{e}}_i$.

However, this type of approach quickly becomes computationally burdensome if we wish to compute ${\Vert ({\mathbf{A}}^{†}{)}^T{\mathbf{e}}_i\Vert }_2$ for a large number of different $i$ values. Fortunately, it is also possible to use stochastic methods to approximately compute ${\Vert ({\mathbf{A}}^{†}{)}^T{\mathbf{e}}_i\Vert }_2$ for all $i$ simultaneously. In particular, let ${\mathbf{z}}_s\in {\mathbb{R}}^M$ for $s=1,\dots ,S$ be a collection of independent and identically distributed random vectors with zero mean and covariance matrix equal to ${\mathbf{I}}_M$, and let

$${\mathbf{g}}_S\triangleq \frac{1}{S}\sum _{s=1}^S\mid {\mathbf{A}}^{†}{\mathbf{z}}_s{\mid }^2,$$ (22)

where our notation assumes that the magnitude-squaring operation is applied separately to each entry of the vector. Then it is not hard to show that the expected value of $i\text{th}$ entry of ${\mathbf{g}}_S$ is equal to ${\Vert ({\mathbf{A}}^{†}{)}^T{\mathbf{e}}_i\Vert }_2^2$, and that ${\mathbf{g}}_S$ converges to its expected value in the limit as $S\to \infty$. This allows approximate calculation of ${\Vert ({\mathbf{A}}^{†}{)}^T{\mathbf{e}}_i\Vert }_2$ for all $N$ possible values of $i$ simultaneously, but only requiring the computation of ${\mathbf{A}}^{†}{\mathbf{z}}_s$ for $s=1,\dots ,S$. If $S\ll N$, then this can represent a massive improvement in computational complexity over directly calculating $\Vert ({\mathbf{A}}^{†}{)}^T{\mathbf{e}}_i{\Vert }_2$ for each $i=1,\dots ,N$. Notably, this approach is quite similar to stochastic techniques designed to efficiently estimate the diagonal entries of spatially-varying covariance matrices [14].

B. Relationship to Cramér-Rao bounds

The theoretical bounds we obtained in this paper were derived without making any assumptions about the statistical characteristics of $\mathbf{n}$, which can be useful when the distribution of $\mathbf{n}$ is unknown. Interestingly, if we make stronger assumptions about the statistical characteristics of $\mathbf{n}$, then we can establish direct links between the traditional Cramér-Rao bound from estimation theory [39] and a quantity that appears prominently in our theoretical bounds, namely $\Vert ({\mathbf{A}}^{†}{)}^T{\mathbf{e}}_i{\Vert }_2$.⁴

To derive Cramér-Rao bounds, we will now make the assumption that $\mathbf{n}$ is a zero-mean Gaussian random vector with covariance matrix ${\mathbf{I}}_M$. In this case, assuming that ${\mathbf{e}}_i\perp 𝒩(\mathbf{A})$, it can be shown that the entrywise Fisher information corresponding to the quantity $x_i={\mathbf{e}}_i^T\mathbf{x}$ is equal to $({\mathbf{e}}_i^T({\mathbf{A}}^T\mathbf{A}{)}^{†}{\mathbf{e}}_i{)}^{-1}$ [40]. This means that the entrywise Cramér-Rao bound for $x_i$ (which is this inverse of the entrywise Fisher information, as well as a lower bound on the variance of any unbiased estimator of $x_i$ [39]) is equal to ${\mathbf{e}}_i^T({\mathbf{A}}^T\mathbf{A}{)}^{†}{\mathbf{e}}_i=\Vert ({\mathbf{A}}^{†}{)}^T{\mathbf{e}}_i{\Vert }_2^2$. This establishes a clear parallel between Cramér-Rao bounds and our theoretical characterization.

This relationship is potentially interesting because minimization of the Cramér-Rao bound is a common objective in methods to design optimally-efficient $\mathbf{A}$ matrices [40]. For example, this type of Cramér-Rao optimization approach has been explored to optimize data acquisition protocols in MRI applications with linear data acquisition models [16], [41], [42]. While this approach has historically been motivated by statistical variance-minimization arguments, our theoretical results demonstrate that this approach can also be justified based on deterministic principles, without requiring strong assumptions about the nature of the noise $\mathbf{n}$. Specifically, minimizing the Cramér-Rao bound for $x_i$ under white Gaussian noise assumptions implicitly has the same effect as minimizing the length of the interval [$L_i,U_i$] for $x_i$ under near data-consistency constraints, regardless of the actual distribution of $\mathbf{n}!$

It should be noted that we have only established a relationship between our new entrywise bounds and entrywise Cramér-Rao bounds for linear models in the form of Eq. (1). There are also many applications where nonlinear data acquisition procedures are optimized by minimizing Cramér-Rao bounds (e.g., a few MRI examples include [43]-[50]). Based on our experience with the linear case, we suspect that it may be possible to rigorously justify the goodness of such design approaches for nonlinear models without making strong statistical assumptions, and believe that this could be an interesting topic for future research.

C. Extensions to Constrained Reconstruction?

One of the features of our theory is that our bounds rely entirely on near data-consistency constraints ${\Gamma }_{\epsilon }(\mathbf{A},\mathbf{b})$, and can be agnostic to the specific estimation methods that are used to obtain $\widehat{\mathbf{x}}$ or any additional prior information that we may have about ${\mathbf{x}}^{\ast }$. In practice, modern estimation methods often rely on more than just near data-consistency, and also impose other forms of constraints when solving an inverse problem. For example, in modern MRI reconstruction, it is common to look for reconstructed images that are not only nearly-data consistent, but which also obey sparsity constraints [51]-[55], low-rank constraints [56]-[61], autoregressive/structured low-rank constraints [62]-[66], or even manifold constraints that could be learned by applying machine learning methods to large databases of previous images [67]-[69]. (See also [70], [71] for additional historical context).

These kinds of additional constraints are all expected to reduce ambiguity in the solution to the inverse problem. As such, rather than just understanding entrywise ambiguity for the set of nearly data-consistent solutions $\mathbf{x}\in {\Gamma }_{\epsilon }(\mathbf{A},\mathbf{b})$, it may be even better to be able to identify entrywise ambiguity for solutions that are both nearly data-consistent and obey other constraints, i.e., $\mathbf{x}\in {\Gamma }_{\epsilon }(\mathbf{A},\mathbf{b})\cap 𝒦$, where $𝒦$ is a set that might represent a sparsity constraint, a low-rank constraint, a manifold constraint, etc.

This is a difficult problem and a rigorous investigation of the interplay between ${\Gamma }_{\epsilon }(\mathbf{A},\mathbf{b})$ and different types of complicated constraint sets $𝒦$ would go far beyond the scope of this paper, though we expect this to be a promising direction for future exploration. We especially believe that developing interval bounds of the form $x_i\in [L_i,U_i]$ for $\forall \mathbf{x}\in {\Gamma }_{\epsilon }(\mathbf{A},\mathbf{b})\cap 𝒦$ may be important for building trust in machine-learning methods designed to solve inverse problems, given the potential for instability and limited resolving power that has been observed with some of these kinds of methods [72], [73].

Biography

Justin P. Haldar (Senior Member, IEEE) received the Ph.D. in Electrical and Computer Engineering from the University of Illinois at Urbana-Champaign in 2011. He is now an Associate Professor in the Signal and Image Processing Institute and the Ming Hsieh Department of Electrical and Computer Engineering at the University of Southern California. His research interests include inverse problems, signal processing, computational imaging, and applications in biomedical imaging (especially magnetic resonance imaging). His research has been recognized with honors such as the NSF CAREER award, the IEEE ISBI best paper award, and the IEEE EMBC first-place student paper award, among others. He is an Associate Editor for the IEEE Transactions on Medical Imaging (2014-Present) and a Senior Area Editor for the IEEE Transactions on Computational Imaging (2021-Present; previously Associate Editor from 2018-2021), and has twice received (in 2019 and 2022) the Outstanding Editorial Board Award from the IEEE Signal Processing Society. He has also been active in the IEEE Signal Processing Society’s Technical Committees on Computational Imaging (Chair Elect, 2024-2025; Vice Chair, 2023; Advisory Member, 2022; Member, 2016-2021) and Bio Imaging and Signal Processing (Member, 2014-2019; Associate Member, 2011-2013).