Autoregression and Structured Low-Rank Modeling of Sinogram Neighborhoods

Abstract

Sinograms are commonly used to represent the raw data from tomographic imaging experiments. Although it is already well-known that sinograms posess some amount of redundancy, in this work, we present novel theory suggesting that sinograms will often possess substantial additional redundancies that have not been explicitly exploited by previous methods. Specifically, we derive that sinograms will often satisfy multiple simple data-dependent autoregression relationships. This kind of autoregressive structure enables missing/degraded sinogram samples to be linearly predicted using a simple shift-invariant linear combination of neighboring samples. Our theory also further implies that if sinogram samples are assembled into a structured Hankel/Toeplitz matrix, then the matrix will be expected to have low-rank characteristics. As a result, sinogram restoration problems can be formulated as structured low-rank matrix recovery problems. Illustrations of this approach are provided using several different (real and simulated) X-ray imaging datasets, including comparisons against a state-of-the-art deep learning approach. Results suggest that structured low-rank matrix methods for sinogram recovery can have comparable performance to state-of-the-art approaches. Although our evaluation focuses on competitive comparisons against other approaches, we believe that autoregressive constraints are actually complementary to existing approaches with strong potential synergies.

I. Introduction

In many tomographic imaging modalities, data acquisition can be modeled using the Radon transform. For simplicity, this work will consider the 2D Radon transform, which is defined for a 2D image f(x, y) via line integrals according to

$$\rho (u,\theta )={\int }_{L(u,\theta )}f(x,y)dl,$$ (1)

where the integral path L(u, θ) is the line defined by

$$x\text{cos}(\theta )+y\text{sin}(\theta )=u.$$ (2)

The function ρ(u, θ) is commonly known as the “sinogram,” and is comprised of different 1D projections of the image along different projection angles θ. Once the full sinogram is measured, the image f(x, y) can be reconstructed using methods such as classical filtered backprojection or more recent model-based iterative reconstruction methods [1].

In real applications, there are many situations where sinogram information is missing [2], [3], e.g., if there are malfunctioning detectors or detector gaps, if the acquisition geometry prevents certain angles from being acquired, if there are metal objects within the field of view that corrupt the values of certain sinogram samples, etc. In addition, sinograms can also be noisy, e.g., when using a small radiation dose. In such situations, methods that can accurately interpolate/extrapolate sinogram data can be valuable for both denoising and recovering missing information.

It has been previously established that sinograms exhibit various consistency conditions and other forms of redundancy [1], [3], [4] that can be exploited for sinogram restoration. One of the most common forms of this (e.g., [5], [6]) is based on the fact that sinograms are theoretically expected to approximately have a bowtie-shaped support in the Fourier domain [1], [7]. While this bowtie constraint is easy to use (e.g., the bowtie region can be computed based only on the radius of the object) and can be effective, it tends to use a very loose/conservative constraint on the Fourier-domain support, which does not fully exploit some of the additional object-dependent redundancies that may be present in the sinogram.

In this work, we demonstrate that sinograms possess a new form of redundancy that, to the best of our knowledge, has not been utilized by previous methods. Specifically, we show that if a sinogram possesses any form of limited support in the Fourier domain, then the sinogram samples should also be expected to approximately satisfy multiple localized linear shift-invariant autoregression relationships. Although these multiple autoregression relationships are expected to be object-dependent and may change from one acquisition to the next, we also demonstrate that they can be learned automatically from the approximate nullspace of an appropriately-constructed structured matrix formed from the sinogram samples themselves, even with missing or noisy data. Importantly, this mathematical structure also implies that sinogram restoration can be formulated as a structured low-rank matrix recovery problem.

The theory and methods we describe in this work are obtained as a generalization of results that were originally derived in the context of Fourier-domain interpolation for spatially support-limited images [8]–[10]. While similar autoregressive reconstruction and structured low-rank matrix recovery methods have emerged for the reconstruction of magnetic resonance imaging (MRI) data [9]–[14], we believe that our present contribution represents the first application of this kind of approach to sinogram restoration.

Preliminary accounts of portions of this work have been previously reported at recent conferences [15], [16]. Compared to those publications, the present contribution provides more detailed descriptions of our theory and methods and contains substantially-expanded experimental results.

This paper is organized as follows. Section II presents our novel theory describing the autoregressive structure of sinograms and relations to structured low-rank matrices and structured low-rank matrix recovery. Section III then describes an example formulation of sinogram restoration as a structured low-rank matrix recovery problem, and describes the details of the implementation that we use in subsequent experimental evaluations. Section IV presents evaluations of this approach, using both simulated and real tomography data. For reference, we compare our proposed approach against existing sinogram interpolation methods [17], [18] as well as a recent state-of-the-art deep learning approach [19]. Deep learning approaches to sinogram restoration have become popular in recent years (e.g., [19]–[21]) due to their excellent performance, although generally require the use of extensive training data. Notably, although this paper is merely intended to provide novel theoretical insights and proof-of-principle (i.e., not state-of-the-art) empirical demonstrations, our results suggest that the proposed structured low-rank modeling approach can achieve similar performance to deep learning methods while also having distinct characteristics and potential synergism with such approaches. Finally, our discussions and conclusions are presented in Sec. V.

II. Theory

A. Preliminaries

For simplicity, we assume that the continuous sinogram ρ(u, θ) has been uniformly sampled on a rectilinear grid, resulting in the discrete 2D representation

$$p[m,n]=\rho (m\Delta u,n\Delta \theta )\text{ for }\forall (m,n)\in {\mathbb{Z}}^2,$$ (3)

where Δu and Δθ are sampling intervals. Since sinograms satisfy ρ(u, θ + 2π) = ρ(u, θ) and ρ(u, θ + π) = ρ(−u, θ) [1], we also assume for simplicity that Δθ = 2π/N for some even integer N such that p[m, n] = p[m, n+N] and p[m, n] = p[−m, n + N/2] for $\forall (m,n)\in {\mathbb{Z}}^2$.

Our theoretical results depend on the Fourier characteristics of p[m, n]. Specifically, we consider the 2D Fourier representation of p[m, n] defined by

$$P\left(e^{j\omega },k\right)=\sum _{m\in \mathbb{Z}}\sum _{n=-N/2}^{N/2-1}p[m,n]e^{-j\omega m}{e}^{-j2\pi \frac{nk}{N}}$$ (4)

for $\omega \in ℝ$ and $k\in \mathbb{Z}$. This can be interpreted as applying a standard discrete-time Fourier transform (DTFT) along the first dimension and a standard discrete Fourier transform (DFT) along the second dimension. This is a natural choice given that p[m, n] is discrete along both dimensions and periodic along the second dimension. Since the continuous sinogram ρ(u, θ) is expected to have a bowtie-shaped support in the Fourier domain [1], [7], classic sampling theory suggests that the principal period of P(e^jω, k) (i.e., with ω ∈ [−π, π) and k ∈ [−N/2, …, N/2 − 1]) will also be expected to have a bowtie-shaped support if the sampling intervals Δu and Δθ are small enough to avoid aliasing.

For illustration, we plot real X-ray sinogram data from a phantom [22] and its corresponding Fourier representation in Fig. 1. As can be seen, the sinogram does indeed satisfy a bowtie-shaped Fourier domain support constraint. However, it should also be observed that the bowtie constraint is quite conservative, and that the actual spectral support is substantially smaller than what the bowtie implies.

Fig. 1.

An illustration of the spectral characteristics of typical sinograms using real X-ray tomography data from Ref. [22]. (a) The sinogram p[m, n]. Due to symmetry, we only show values corresponding to 0 ≤ n ≤ N/2 − 1. (b) Filtered backprojection reconstruction of the sinogram. (c) The Fourier transform P(e^jω, k) of the sinogram. It is observed that the support of P(e^jω, k) is very compact, as expected. (d) The same image of P(e^jω, k) from (c), on which we have overlaid (in red) the complement of the bowtie-shaped region computed based on the radius of the object [1], [7]. It can be observed that the support of P(e^jω, k) is contained within the bowtie-shaped region (as expected), although it is also observed that the bowtie constraint is highly conservative (i.e., the support of P(e^jω, k) occupies a very small fraction of the bowtie region).

B. Autoregressive Properties of Sinograms

Motivated by previous results [8]–[10], we observe in this work that if P(e^jω, k) is known to have limited support, then it should also be possible to define non-trivial functions H(e^jω, k) with complementary support such that P(e^jω, k)H(e^jω, k) ≈ 0 for all frequences in the spectral domain. This observation is significant because of the following:

Theorem 1:

Let ε be an arbitrary positive scalar. We have that

$$\frac{1}{2\pi N}{\int }_{-\pi }^{\pi }\sum _{k=-N/2}^{N/2-1}{\left|P\left(e^{j\omega },k\right)H\left(e^{j\omega },k\right)\right|}^{2}d\omega \le \epsilon$$ (5)

for some function H(e^jω, k) if and only if

$$\sum _{m\in \mathbb{Z}}\sum _{n=-N/2}^{N/2-1}{\left|\sum _{\ell \in \mathbb{Z}}\sum _{q=-N/2}^{N/2-1}h[\ell ,q]p\left[m-\ell ,<n-q{>}_N\right]\right|}^2\le \epsilon ,$$ (6)

where

$$h[m,n]\triangleq \frac{1}{2\pi N}{\int }_{-\pi }^{\pi }\sum _{k=-N/2}^{N/2-1}H\left(e^{j\omega },k\right)e^{j\omega m}{e}^{j2\pi \frac{nk}{N}}d\omega$$ (7)

and < · >_N denotes the length-N modulo operation (as used in circular convolution).

This theorem is similar to previous results derived in a different context [10], and as in that work, it can be trivially proven through the direct application of standard DTFT and DFT properties, namely Parseval’s theorem and the convolution property of the Fourier transform.

A consequential implication of Thm. 1 is that if P(e^jω, k) and H(e^jω, k) have complementary support such that P(e^jω, k)H(e^jω, k) ≈ 0 at all frequencies, then there must exist a corresponding filter h[m, n] (defined in Eq. (7)) such that convolution with the filter satisfies

$$\sum _{\ell \in \mathbb{Z}}\sum _{q=-N/2}^{N/2-1}h[\ell ,q]p\left[m-\ell ,<n-q{>}_N\right]\approx 0$$ (8)

for all points (m, n) in the sinogram domain. In the terminology of finite-rate-of-innovation theory [23], h[m, n] can be viewed as an (approximate) annihilating filter.

If P(e^jω, k) has limited support, then there will be a huge number of different choices of H(e^jω, k) that will satisfy P(e^jω, k)H(e^jω, k) ≈ 0, similar to the situation in previous contexts [8]–[10]. In what follows (and consistent with previous work [9], [10]), we will assume that at least some of these H(e^jω, k) spectral functions correspond to short finite impulse response (FIR) filters h[m, n]. Specifically, for the sake of concreteness and without loss of generality, we will assume that h[m, n] is FIR such that h[m, n] = 0 whenever (m, n) ∈ Λ, where the support set Λ is defined as $\Lambda \triangleq \left\{(m,n)\in {\mathbb{Z}}^2:\left(m^2+n^2\right)\le {\tau }^2\right\}$, where τ is a parameter that defines the radius of the sinogram-domain filter support. Note that this FIR assumption implies that the corresponding Fourier spectrum H(e^jω, k) must be smooth (i.e., formed only from low-frequency sinusoids, with the maximum frequency determined by the filter radius τ). Assuming that H(e^jω, k) is smooth should not be very restrictive if P(e^jω, k) has large contiguous gaps in its support set, since there should be many smooth functions whose support can approximately fit within such gaps [9], [10].

Under this FIR assumption, the relationship in Eq. (8) simplifies to

$$\sum _{(\ell ,q)\in \Lambda }h[\ell ,q]p\left[m-\ell ,<n-q{>}_N\right]\approx 0$$ (9)

for all points (m, n) in the sinogram domain. Importantly, Eq. (9) suggests that the sinogram samples from a small neighborhood are approximately linearly dependent, and therefore redundant. For example, if it is further assumed that h[0, 0] ≠ 0, then we must have that any sinogram sample can be approximately interpolated using a linear shift-invariant combination of its neighboring samples according to:

$$p[m,n]\approx -\sum _{(\ell ,q)\in \Lambda \setminus \left\{0,0\right\}}\frac{h[\ell ,q]}{h[0,0]}p\left[m-\ell ,<n-q{>}_N\right].$$ (10)

This autoregressive relationship is significant, because it can be used to interpolate/extrapolate missing or corrupted sinogram values if the h[m, n] coefficients can be learned [8]–[10]. Potential methods for learning these coefficients in practical scenarios will be discussed in the following subsections.

C. Low-Rank Modeling of Sinogram Neighborhoods

The results of the previous subsection imply that sinograms p[m, n] with support-limited Fourier transforms should be autoregressive, which would be useful if appropriate filter coefficients h[m, n] were known. However, such filter coefficients will be dependent on the support characteristics of P(e^jω, k), which are unlikely to be the same for different images. While image-independent loose bounds on the support of P(e^jω, k) may be available (e.g., the bowtie constraint), tighter and more informative image-dependent bounds are unlikely to be known in advance. This makes it desirable to be able to learn the h[m, n] filters directly from measured sinogram samples. In this section, we suggest an approach based on structured low-rank matrix modeling, which closely resembles the low-rank modeling of k-space neighborhoods (LORAKS) approach that was previously introduced in the context of MRI reconstruction [9].

Suppose that we have access to the values of p[m, n] for all points $(m,n)\in {\mathbb{Z}}^2$ satisfying −M ≤ m ≤ M and −N/2 ≤ n ≤ N/2 − 1, for some integer M with M > τ. Let Γ denote the set of all points $(m,n)\in {\mathbb{Z}}^2$ satisfying |m| ≤ M − τ and −N/2 ≤ n ≤ N/2 − 1, and let the elements of this set be placed in (arbitrary) order such that (m_i, n_i) denotes the ith element of Γ for i = 1, …, |Γ|. By construction, the cardinality of Γ is |Γ| = N(2M − 2τ+1). Also let the elements of the set Λ be placed in (arbitrary) order such that (ℓ_j, q_j) denotes the jth element of Λ for j = 1, …, |Λ|. Consider the matrix $C\in {ℝ}^{\left|\Gamma |\times |\Lambda \right|}$ that is constructed from the samples of the sinogram p[m, n] according to

$${[C]}_{ij}\triangleq p\left[m_i-{\ell }_j,<n_i-q_j{>}_N\right]$$ (11)

for i = 1, …, |Γ| and j = 1, …, |Λ|. Note that each row of C collects the set of sinogram samples from the local sinogram neighborhood (specified by Λ) that is centered at the point (m_i, n_i). In addition, each sample p[m, n] will generally appear multiple times within the C matrix, and C will have convolutional (block Toeplitz) structure if the sets Γ and Λ are ordered appropriately (e.g., placed in raster order).

Importantly, if the relationship in Eq. (9) is valid, then the structured matrix C must satisfy

$$\text{Ch}\approx 0,$$ (12)

where $h\in {ℝ}^{\left|\Lambda \right|}$ is the vector with jth element equal to h[ℓ_j, q_j] for j = 1, …, |Λ|. In other words, under the assumptions of the previous subsection, the vector h is an approximate nullspace vector of C, and the matrix C must be approximately rank-deficient. Further, in the common situation where there are multiple linearly-independent FIR filters that satisfy the assumptions of the previous section, then the matrix C will have multiple approximate nullspace vectors and will be approximately low-rank.

The low-rank structure of the C matrix can be quite useful. Specifically, if we have access to samples of p[m, n] but do not know the spectral support of P(e^jω, k) and do not know of appropriate filter coefficients h[m, n] that satisfy Eq. (9), then we can obtain estimates of these by first constructing the C matrix and then calculating its approximate nullspace vectors. Figures 2 and 3 show an illustration of this, in which we have constructed a C matrix using the full set of sinogram data from Fig. 1 using M = 1120, N = 360, and a neighborhood radius of τ = 7 (corresponding to a 15×15 FIR filter with the corners set to zero). As can be seen from Fig. 2, the singular values of this C matrix decay very rapidly, which confirms the theoretical expectation that C will be approximately low-rank. Figure 3 illustrates that we can use the nullspace vectors of the C matrix to directly provide FIR filters h[m, n] that satisfy P(e^jω, k)H(e^jω, k) ≈ 0 at all frequencies. Further, the figure also shows that by combining the spectral supports of all of the approximate nullspace vectors, it is possible to obtain a very tight estimate of the support of P(e^jω, k).

Fig. 2.

Singular values of the C matrix corresponding to the sinogram data from Fig. 1. We observe that the singular values decay very rapidly (i.e., the matrix has a maximum possible rank of 149, while the rank-7 approximation is sufficient to capture more than 99% of the total energy of the original matrix), confirming the expectation that C should be low-rank because P(e^jω, k) had a very small spectral support. While we have only illustrated this behavior for a single sinogram with a single value of τ, the characteristics observed in this case are quite typical.

Fig. 3.

(a-e) Five representative sinogram filters h[m, n] obtained from the approximate nullspace vectors of the C matrix. The filters are color-coded to indicate the sign of the filter coefficients (blue = positive, red = negative). (f-j) The corresponding spectral representations H(e^jω, k) of the filters, shown as red overlays on top of P(e^jω, k). As can be seen, we have in each case that P(e^jω, k) and H(e^jω, k) have approximately complementary support, as expected. (k) An estimate of the complement of the spectral support of P(e^jω, k) can be obtained by examining the spectral supports of all the filter functions obtained from approximate nullspace vectors. In this case, the red overlay shows the square-root of the sum-of-squares of 134 different frequency responses H(e^jω, k) corresponding to the approximate nullspace vectors of the C matrix, which provides a very tight delineation of the spectral support of P(e^jω, k).

The previous arguments and illustration suggest that the C matrix formed from sinogram samples is expected to be approximately low-rank in practical scenarios, and that the nullspace structure of this matrix implicitly encodes important object-dependent information about the spectral support of P(e^jω, k) that can be much tighter than the conventional bowtie constraint. However, the previous arguments are also all based on the assumption that we have direct access to the full set of ideal sinogram samples p[m, n], which will not actually be available in problems of interest where some of the p[m, n] values are missing and/or corrupted by noise. Fortunately, the fact that the ideal C should be approximately low-rank can still be useful in cases of noisy and/or missing sinogram information, due to the fact that it is frequently possible to accurately recover low-rank matrices from a subset of noisy entries [24]. In the next subsection, we leverage such ideas to formulate the sinogram restoration problem within the framework of low-rank matrix recovery.

D. Formulating Sinogram Restoration as Structured Low-Rank Matrix Recovery

In what follows, we assume that we are interested in reconstructing a vector of sinogram samples $p\in {ℝ}^{(2M+N/2)}$ corresponding to p[m, n] with −M ≤ m ≤ M and 0 ≤ n ≤ N/2 − 1. This corresponds to a half-sinogram (i.e., covering angles from 0° to 180°), but is easily converted into a full-sinogram $\tilde{p}\in {ℝ}^{(2M+N)}$ containing samples of p[m, n] with −M ≤ m ≤ M and −N/2 ≤ n ≤ N/2 − 1 (covering a full 360°) using the aforementioned symmetry constraint that p[m, n] = p[−m, n+N/2]. We use $F\in {ℝ}^{(2M+N)\times (2M+N/2)}$ to denote the matrix that converts the half-sinogram to the full-sinogram such that

$$\tilde{p}=Fp.$$ (13)

Although the half-sinogram contains everything we need to know (i.e., the full-sinogram is redundant due to periodicity), the F matrix will be useful, e.g., for constructing the C matrix (which, as described previously, is based on the full-sinogram).

The ideal (noiseless) forward model of sinogram data acquisition is given by

$$\tilde{d}=AFp,$$ (14)

where $\tilde{d}\in {ℝ}^K$ denotes the vector of measured sinogram samples (assuming K samples are acquired), and the matrix $A\in {ℝ}^{K\times (2M+N)}$ models the data acquisition process. In this work, we will focus on scenarios where certain sinogram samples are missing, in which case A is a subsampling matrix.

In practice, the real measured data will not be noiseless, and we use $d\in {ℝ}^K$ to denote the noise-perturbed version of $\tilde{d}$. We use $p(d\mid \tilde{d})$ to model the probability of measuring noisy data d given that noiseless data was $\tilde{d}$ (i.e., the likelihood function). Given this model, the objective of sinogram restoration is to obtain an estimate $\widehat{p}$ of the noiseless half-sinogram p from the noisy/incomplete measured data d.

A standard statistical approach to this problem [25] would seek to minimize a penalized maximum likelihood objective function, i.e.,

$$\widehat{p}=\text{arg}\underset{p}{\text{min}}L(d\mid AFp)+\lambda \Psi (p),$$ (15)

where $L(d\mid \tilde{d})=-\text{ln}p(d\mid \tilde{d})$ is the negative log-likelihood function that measures data consistency, λ is a regularization parameter, and Ψ(·) represents a regularization penalty function that is used to encourage the estimated sinogram to possess desirable characteristics. In this work, we will choose Ψ(·) to encourage the C matrix formed from the samples of the full-sinogram $\tilde{p}$ to have low-rank characteristics. Notably, forcing C to be low-rank is equivalent to there being multiple linearly-independent nullspace vectors h that satisfy Eq. (12), which in turn requires that the sinogram is simultaneously annihilated by a large number of linearly independent FIR filters h[m, n] (with the number of independent filters equal to the dimension of the approximate nullspace). As a result, low-rank constraints do not just use a single autoregression relationship. Instead, they implicitly impose multiple autoregression relationships simultaneously in a data-driven way, without ever explicitly needing to estimate the h[m, n] filters. Using $\mathcal{C}(\cdot ):{ℝ}^{2M+N}\to {ℝ}^{\left|\Gamma |\times |\Lambda \right|}$ to denote the linear operator that maps a vector of sinogram data into the corresponding convolution-structured matrix C, we will focus attention on regularization penalties that take the form

$$\Psi (p)=J(\mathcal{C}(Fp))$$ (16)

for an appropriate low-rank promoting penalty function J(·).

Several classes of penalty functions that promote low-rank reconstructions are compatible with our proposed framework, and have been previously used in various MRI image reconstruction scenarios. One class of methods directly enforces “strong” low-rank constraints, e.g., by taking

$$J(C)=\{\begin{array}{ll}0,\hfill & \text{rank}(C)\le r\hfill \\ \infty ,\hfill & \text{else},\hfill \end{array}$$ (17)

where r is a user-selected rank parameter. This type of strict low-rank constraint can be imposed using various methods like Cadzow’s algorithm or Incremented Rank PowerFactorization [11], [26], [27]. Although this choice of penalty can be effective, the constraint it imposes is very strong (i.e., requiring a strictly low-rank matrix, rather than the approximately low-rank matrix structure suggested in the previouos subsection), A somewhat “softer” low-rank constraint, proposed in Ref. [9], encourages C to be approximately rank-r (where r is again a user-selected parameter) by penalizing the size of the difference between C and its optimal rank-r approximation. Mathematically, this can be described as

$$J(C)=\underset{Y}{\text{min}}\parallel C-Y{\parallel }_F^2\text{subject}\text{to}\text{rank}(Y)\le r.$$ (18)

If σ_i(C) is used to denote the singular values of C (placed in decreasing order such that σ_i(C) ≥ σ_i+1(C) for all i), then this penalty function can be equivalently written as

$$J(C)=\sum _{i>r}{\left({\sigma }_i(C)\right)}^2,$$ (19)

which can be viewed as a truncated form of the squared Frobenius norm that ignores the largest r singular values. This penalty function has been used extensively in the context of MRI, and open source implementations of several different algorithm variations are available [28].

Other popular regularization penalties include the family of Schatten norms, which have also been used in structured low-rank matrix methods for MRI (e.g., [13], [29]). These penalties take the form

$$J(C)=\sum _i{\left({\sigma }_i(C)\right)}^q,$$ (20)

where q is a user selected parameter. The choice of q = 1 is known as the nuclear norm, and is special because it yields the convex envelope of rank(C) and is thus the convex function that best approximates the rank penalty [24]. The nuclear norm is one of the most frequently-used choices for low-rank matrix recovery, because its convex structure can simplify optimization, and also because strong theoretical performance guarantees have been derived for it [24]. However, in the context of low-rank methods for computational image reconstruction, many authors have preferred to use nonconvex penalties (e.g., the penalties in Eqs. (17) and (18) are both nonconvex) because they tend to yield better solutions and can sometimes offer reduced computation.¹ If q < 1 in Eq. (20), the corresponding penalty function is nonconvex and, like other nonconvex penalties, frequently has advantages over the nuclear norm in this kind of setting [29].

A final class of methods that was originally developed for MRI [12], [31]–[36] takes a two-step approach, and when translated to the context of sinogram recovery, relies on the assumption that certain regions of the sinogram have been fully sampled. Specifically, let $p_{zp}\in {ℝ}^{2M+N}$ denote an incomplete sinogram for which all missing samples have been zero-filled, and let $C_{zp}=\mathcal{C}\left(F{p}_{zp}\right)$. If certain regions of the sinogram have been fully sampled, then it can be possible to form a fully-sampled submatrix of C_zp, denoted as S_zp, by discarding all rows of C_zp containing missing (zero-filled) entries. Under our previously stated assumptions, every approximate nullspace vector of the C matrix should also be an approximate nullspace vector of the S_zp matrix, which could allow us to use the approximate nullspace vectors of S_zp as an estimate of the nullspace vectors of C. This can lead to greatly-simplified (convex, Tikhonov-style) regularization penalties, e.g., [12], [35], [36]

$$J(C)=\parallel C\widehat{H}{\parallel }_F^2,$$ (21)

where $\widehat{H}$ is the matrix of pre-estimated approximate nullspace vectors.

III. Illustrative Reconstruction Example: Formulation and Algorithm

As described in the previous section, our framework is compatible with many different formulations. However, for the sake of concreteness and without loss of generality, the remainder of this paper will focus attention on a specific imaging scenario, a specific problem formulation, and a specific optimization algorithm. Our goal for this work is to simply demonstrate proof-of-principle, and we have made no attempt to optimize the choices that we describe below. For simplicity, many of our choices resemble similar choices that were made for LORAKS reconstruction in the context of MRI reconstruction [9], and the code used in this work is a minor modification of the open-source LORAKS 2.0 code [28].

As described before, we assume an imaging scenario with missing sinogram samples, with reconstruction formulated as in Eq. (15). For concreteness, we now make the further assumption that the noise in the measured data is negligible, such that it is reasonable to strictly enforce data consistency constraints, i.e., taking

$$L(d\mid AFp)=\{\begin{array}{ll}0,\hfill & d=AFp\hfill \\ \infty ,\hfill & \text{else}.\hfill \end{array}$$ (22)

In addition, we choose regularization according to Eq. (16) using the “soft” low-rank regularization penalty from Eq. (18). For the empirical examples shown later, r was chosen manually as the point where the singular-value curve for the structured matrix S_zp starts to flatten out, which is a common rank selection heuristic. More advanced rank-selection approaches would also have been viable [37], [38].

With the problem formulation fully specified, it remains to describe the optimization algorithm used to obtain the solution. Fortunately, the choices we have made allow our problem formulation to match existing LORAKS problem formulations almost exactly, which allows us to simply reuse previous algorithms that were originally developed for MRI [28]. All of the results shown later in this paper are based on the FFT-based multiplicative half-quadratic LORAKS optimization algorithm with strict data consistency as described in Sections 2.2.2 and 2.2.3 of Ref. [28]. This algorithm is based on a majorize-minimize procedure, which alternates between imposing low-rank constraints and imposing data-consistency constraints until convergence, and replaces matrix-vector multiplications with FFTs by leveraging the convolutional structure of the C matrix. See also [29] and [39] for related structured low-rank algorithms that were also developed in the context of MRI.

The formulation we used is nonconvex, and is therefore subject to local minima. The results shown in this work are based on a naive initialization where missing sinogram samples were zero-filled.

IV. Results

A. Results with Real Phantom Data

We first show evaluations of the proposed structured low-rank sinogram restoration method by retrospectively subsampling the X-ray data shown previously in Fig. 1. We considered three different patterns of missing data (“bad bins,” “metal objects,” and “limited angles”) as illustrated in Fig. 4. As can be seen, in each of these cases, the missing data leads to severe artifacts in a naive filtered backprojection reconstruction.

Fig. 4.

Three missing data scenarios used for evaluation. The top row shows sinograms with missing data (retrospectively subsampled), where the missing sample locations have been marked in red. The bottom row shows corresponding filtered backprojection reconstructions.

We restored these sinograms using the proposed structured low-rank method. For comparison, we also implemented the approach from Ref. [6] (which we call the “bowtie” approach) that restores the sinogram by iteratively projecting the current sinogram estimate onto the space of sinograms that satisfy exact data consistency and the space of sinograms that satisfy the spectral-domain bowtie support constraint. Results are shown in Fig. 5. In each case, we observe that the bowtie approach is somewhat successful at restoring the sinogram, but is considerably less successful (i.e., demonstrating more substantial artifacts) than the proposed approach.

Fig. 5.

Restoration results for the three scenarios shown in Fig. 4. The top two rows show sinogram and filtered backprojection results for the bowtie approach, while the bottom two rows show results for the proposed approach.

B. Results with Real Ptychographic Tomography Data

We also assessed the proposed method in the context of nondestructive ptychographic imaging of integrated circuits [40]. In this application, images with nanometer-scale spatial resolution are formed through a two-stage reconstruction process. During data acquisition, for each projection angle, the sample was scanned using a coherent synchrotron X-ray beam from the Swiss Light Source, and diffraction patterns were measured for each beam position. In the first stage of image reconstruction, phase retrieval algorithms are applied to reconstruct complex projection images from these diffraction patterns, and the phases of the images (which are more sensitive to the material properties of the integrated circuit) from different projection angles are extracted and assembled into sinograms. In the second stage of image reconstruction, tomographic algorithms are applied to obtain full reconstructions of the integrated circuit from the computed sinograms. In the data we show in this work (which is the same data from Ref. [40]), we are interested in reconstructing a cylindrically-shaped pillar (10.9 μm in diameter) of an integrated circuit from data acquired with 1200 different projection angles.

One of the issues that arises in this application is that, when the X-ray propagation is parallel to the direction of long straight metal line features in the integrated circuit, the accumulated phase can exceed 2π, leading to artifacts in the reconstruction [41]. An example of this is shown in Fig. 6, where one portion of the sinogram has been corrupted by this kind of artifact, leading to streak artifacts if this portion of the sinogram is zero-filled. Note that we are showing a filtered version of the sinogram in this figure (i.e., after the differentiation and Hilbert transformation steps of filtered backprojection have been applied), which bypasses some of the inherent phase ambiguities associated with the phase retrieval step. As can be seen, the zero-filled reconstruction suffers from streak artifacts, while the proposed approach leads to substantial reductions in these artifacts.

Fig. 6.

Restoration results for the integrated circuit data. We show (top row) pre-filtered sinograms and (bottom row) filtered backprojection results for (left) the zero-filled sinogram and (right) the proposed approach. The corrupted sinogram sample locations are marked in red on the zero-filled sinogram.

C. Results with Simulated Security Data

We also evaluated the proposed approach in the context of a simulated security application, in which a tomographic X-ray scan of a suitcase-like object is taken, but some of the measured sinogram samples are corrupted by the presence of metal objects. Our basic simulation approach follows the one described in Ref. [19], which is based on a polyenergetic X-ray source model combined with energy-integrating detectors. The ideal projection data is subsequently degraded by data-dependent (Poisson) and electronic (Gaussian) noise, and the sinogram data is then log-normalized. Note that, by design, this simulation approach accounts for effects such as beam hardening, but does not account for other effects like photon scatter. See Ref. [19] for further discussion.

Following Ref. [19], we simulated multiple objects with and without metal objects, systematically varying the radius of the metal objects from 8 to 16 voxels. Representative examples are shown in Fig. 7. These metal objects cause corruption of the sinogram samples (with bigger objects leading to the corruption of a larger number of sinogram samples), and the restoration approach considered here is to discard the corrupted sinogram samples and recover the missing information using structured low-rank matrix methods.

Fig. 7.

Simulated suitcase-like objects (left) without and (middle, right) with metal objects (shown in red) of different radii: (middle) radius = 8 voxels; (right) radius = 14 voxels.

In addition to applying our proposed structured low-rank matrix approach, we also compared against a recent state-of-the-art deep learning method (i.e., the Deep Metal Artifact Reduction (Deep-MAR) method [19]), as well as benchmark methods based on linear interpolation (LI-MAR) [17] and weighted nearest-neighbor interpolation (WNN-MAR) [18]. For Deep-MAR, we used a pretrained network provided by the original Deep-MAR authors, with the training details described in Ref. [19]. Representative sinogram reconstruction results for some of the metal object radii are shown in Fig. 8, while the full set of sinogram reconstruction results for all radii can be found in Figs. S1 and S2 of the supplementary material, with quantitative normalized mean-squared error (NRMSE) values reported in Table I. As can be seen, the proposed method has the best (smallest) NRMSE values when the metal object is small, while Deep-MAR begins to slightly outperform the proposed method as the metal object radius grows larger (although the performance of both methods is roughly comparable in this range), and both the proposed method and Deep-MAR substantially outperform LI-MAR and WNN-MAR. Interestingly, while the proposed method and Deep-MAR often have similar NRMSE values, the error distributions for both methods are observed to have very different structure, indicating that these two approaches may be taking advantage of different sinogram features and are therefore potentially complementary.

Fig. 8.

Representative sinogram restoration results for the simulated security data for (top row) metal objects with radius = 8 voxels and (bottom row) metal objects with radius = 14 voxels. The left two columns show the sinograms for the metal-free case and the zero-filled sinograms for the cases with metal objects (with corrupted sinogram samples marked in red). The remaining four columns show error images for different sinogram restoration methods. Each pixel of each error image has been normalized by dividing by the corresponding value from the metal-free sinogram.

TABLE I.

NRMSE values corresponding to the sinogram restoration results shown in Fig. 8 and supplementary Figs. S1 and S2. The best value for each radius is indicated with bold.

Radius	LI-MAR	WNN-MAR	Deep-MAR	Proposed
8	0.011	0.012	0.012	0.006
10	0.015	0.016	0.009	0.008
12	0.019	0.019	0.010	0.011
14	0.024	0.024	0.012	0.015
16	0.029	0.028	0.014	0.020

In addition to evaluating errors in the sinogram domain, we also applied filtered backprojection reconstruction to the recovered sinograms and evaluated image-domain performance. Representative image results are shown in Fig. 9, with the full set of results shown in Figs. S3 and S4 of the supplementary material. To avoid visual confusion, we have masked the original locations of the metal objects in these images. Quantitative image-domain NRMSE values are reported in Table II. As can be seen, the results in the image domain follow similar trends to those observed in the sinogram domain, with the proposed method having the best performance when the metal object radius is small and Deep-MAR having the best performance when the metal object radius is large, but with each approach producing a distinct error pattern that suggests the potential for complementarity. Specifically, the proposed approach frequently has smaller-intensity artifacts in the vicinity of the metal objects than Deep-MAR, but also often has larger-intensity streak artifacts in regions that are distant from the metal objects.

Fig. 9.

Representative filtered backprojection results for the simulated security data (corresponding to the case of metal objects with radius = 8 voxels). The top row shows the images while the bottom shows errors with respect to the reconstruction of the (uncorrupted) metal-free sinogram. Each voxel of each error image has been normalized by dividing by the corresponding value from the metal-free reference image.

TABLE II.

NRMSE values corresponding to the filtered backprojection images shown in Fig. 9 and supplementary Figs. S3 and S4. The best value for each radius is indicated with bold.

Radius	LI-MAR	WNN-MAR	Deep-MAR	Proposed
8	0.046	0.072	0.068	0.037
10	0.058	0.083	0.060	0.045
12	0.068	0.092	0.060	0.055
14	0.080	0.102	0.063	0.067
16	0.092	0.110	0.065	0.081

When interpreting these results, it is also important to remember that the proposed method and Deep-MAR are based on very different principles. The proposed approach is based purely on structured low-rank modeling assumptions, and the reconstruction is based entirely on one set of sinogram data. In contrast, Deep-MAR is a deep learning approach that relies on a very substantial amount of training data. Given this mismatch in the amount of data each method relies upon, the fact that the proposed method can even compete with Deep-MAR (and even outperforms it in certain situations) is quite surprising, and suggests that the autoregressive nature of the sinogram can be a very powerful constraint. Of course, as our previous comments indicate, Deep-MAR and the proposed method appear to be complementary in various ways, and we suspect that a hybrid approach (which we leave to future work) is likely to produce even better results.

V. Discussion and Conclusions

In this work we derived novel theoretical results demonstrating that sinograms should be expected to obey data-dependent linear shift-invariant autoregressive relationships, which enable sinogram samples to be linearly predicted from their neighbors. Based on these theoretical principles, we also showed that these linear-predictability relationships can be automatically learned from incomplete sinogram data by formulating the sinogram restoration problem using appropriate structured low-rank regularization methods.

Our theoretical results were derived based on the assumption that the Fourier transfom of the sinogram is expected to be support limited (but with unknown support). All sinograms have some degree of limited spectral support (i.e., the bowtie constraint [1], [7]), so our autoregression-based theory and methods should be broadly applicable without requiring special assumptions. However, structured low-rank modeling should be especially beneficial when the spectral support of the sinogram is particularly small, and it would be insightful to know when this should be expected. It is already known theoretically that certain types of images (e.g., spatially-smooth images or circularly symmetric images [7]) will have sinograms with particularly small spectral support, although our experience suggests that sinograms with small spectral support are much more pervasive than this. A full theoretical understanding remains to be developed.

Our theoretical results for sinograms have strong similarities to previous results that suggest that spatial Fourier transform samples (i.e., k-space data) such as those acquired in MRI should also obey linear shift-invariant autoregressive relationships if the image happens to be support limited (again with unknown support) [8]–[10]. These previous theoretical observations also lead to powerful structured low-rank matrix reconstruction methods in the context of MRI [9]–[14], and as such, we anticipate that similar structured low-rank modeling approaches may be promising for any application where the data measurements are expected to have limited support in an appropriate spectral domain.

The results shown in this work were based on the assumption that data was collected with a 2D parallel-beam geometry. However, it is also known that 2D fan-beam geometries are also known to yield sinograms that possess bowtie-like spectral support characteristics [42], which suggests that the proposed approach is likely to apply directly to 2D fan-beam data in a natural way. In addition, it is well-established that fan-beam, cone-beam, and time-of-flight data can be rebinned to an equivalent parallel-beam data format [43]–[45], which should also enable use of the proposed structured low-rank matrix framework to these more general acquisition approaches.

Our results demonstrated that structured low-rank approaches can offer comparable NRMSE performance to state-of-the-art deep learning methods, but without any need for training. However, our structured low-rank algorithm implementation was slower than methods like Deep-MAR (ignoring the time Deep-MAR requires for training). This is not surprising, as we used an unoptimized MATLAB implementation – but even if our implementation were optimized, it is unlikely to be faster than Deep-MAR, which is very fast (i.e., ~200 milliseconds per reconstruction, which is orders of magnitude faster than the current structured low-rank implementation). Nevertheless, it should be emphasized that structured low-rank methods and deep learning methods do not need to be viewed adversarially (as we have done in this work) – they have the potential to be complementary, and hybrid approaches may offer substantial advantages. For instance, deep-learning reconstructions could be used to initialize structured low-rank methods (leading to faster convergence), or both methods could be used synergistically, e.g., under the framework of consensus equilibrium [46]. We believe that this is a very interesting direction for future work, and readers may also take interest in related papers that have explored connections and synergies between deep learning methods and structured low-rank matrix methods in the context of MRI [14], [47].

Similarly, while this paper has focused on a two-step reconstruction process (sinogram restoration followed by tomographic image reconstruction), it would be straightforward to synergistically combine these into a single step, e.g., using concepts such as variable splitting or consensus equilibrium [46]. In addition, since structured low-rank modeling can be implemented using simple regularization penalties, it is completely compatible with other forms of regularization constraints. Since the constraints imposed by sinogram-domain structured low-rank regularization penalties are generally distinct from and complementary to the constraints imposed by other regularization strategies, there is strong potential for these different regularization approaches to be synergistically combined for even better results, as has already been demonstrated in MRI contexts [48], [49].