Generalized Prediction Framework for Reconstructed Image Properties using Neural Networks

Abstract

Model-based reconstruction (MBR) algorithms in CT have demonstrated superior dose-image quality tradeoffs compared to traditional analytical methods. However, the nonlinear and data-dependent nature of these algorithms pose significant challenges for performance evaluation and parameter optimization. To address these challenges, this work presents an analysis framework for quantitative and predictive modeling of image properties in general nonlinear MBR algorithms. We propose to characterize the reconstructed appearance of arbitrary stimuli by the generalized system response function that accounts for dependence on the imaging conditions, reconstruction parameters, object, and the stimulus itself (size, contrast, location). We estimate this nonlinear function using a multilayer perceptron neural network by providing input and output pairs that samples the range of imaging parameters of interest. The feasibility of this approach was demonstrated for predicting the appearance of a spiculated lesion reconstructed by a penalized-likelihood objective with a Huber penalty in a physical phantom as a function of its location and reconstruction parameters β and δ. The generalized system response functions predicted from the trained neural network show good agreement with those computed from mean reconstructions, proving the ability of the framework in mapping out the nonlinear function for combinations of imaging parameters not present in the training data. We demonstrated utility of the framework to achieve desirable (e.g., non-blocky) lesion appearance in arbitrary locations in the phantom without the need for performing actual reconstructions. The proposed prediction framework permits efficient and quantifiable performance evaluations to provide robust control and understanding of image properties for general classes of nonlinear MBR algorithms.

1. INTRODUCTION

Medical imaging algorithms play a pivotal role in enabling new capabilities in image formation, post-processing, and analysis. CT Reconstruction in particular, has seen rapid proliferation of model-based reconstruction (MBR) algorithms. In contrast to their traditional analytical counterparts, MBR offers capability to integrate statistics of measurement noise, physics of the imaging system, and prior knowledge from sources such as prior images and dictionaries. These properties allow MBR to transcend traditional limitations of sampling and signal-to-noise requirements to achieve superior dose-image quality tradeoffs as demonstrated by numerous studies.

Despite these advantages, performance evaluation and optimization of MBR poses significant challenges as traditional analytical tools become inapplicable. MBR algorithms are typically nonlinear and carry numerous dependencies including imaging techniques, reconstruction parameters, patient anatomy, and the size, contrast, and location of the stimuli. Current approaches of image quality quantitation have various limitations. Contrast-dependent resolution¹ alone does not account for the full extent of data-dependency, e.g., the resolution of a particular contrast lesion can change with its location within the object. Moreover, edge-preserving methods are inherently shift-variant with potential abrupt changes in resolution at contrast boundaries. Noise, resolution, or task-based detectability² do not explicitly reveal texture information which may be important for diagnosis. The large number of dependencies also pose challenges for parameter optimization. For a small class of locally linearizable MBR [e.g., penalized likelihood (PL) reconstruction with a quadratic penalty], analytical derivations of local impulse response and covariance do encompass all dependencies and enable efficient, prospective analysis and optimization.³ However, for the majority of non-linearizable MBR, current algorithm characterization approaches frequently rely on retrospective, exhaustive, and qualitative evaluations of the reconstructions under different imaging conditions. Such approaches are highly subjective and difficult to scale up to complex clinical scenarios with many imaging parameters.

To address these challenges, this work proposes a novel image quality analysis framework that is quantitative, predictive, and applicable to general classes of nonlinear MBR algorithms. We seek to answer two questions: 1) how to quantify image quality in MBR, and 2) how to optimize MBR for specific image quality metrics.

We introduce the generalized system response (denoted as $\mathcal{H}$), a nonlinear function that quantifies system response to arbitrary perturbations. In the context of lesion detection, $\mathcal{H}$ corresponds to the reconstructed appearance of a specific lesion stimulus with the blurring and bias effects associated with particular imaging conditions and reconstruction algorithms. Such a metric captures the full system response and (together with the generalized noise response) allows the computation of any derivative metrics such as detectability, radiomic measures of texture, heterogeneity, shape, lesion boundaries, etc. We formulate the prediction of $\mathcal{H}$ as a function estimation problem and propose a data-driven method for learning this response function. Specifically, we leverage the universal approximation theorem⁴ and use a multilayer perceptron neural network as a nonlinear function estimator. While the approach is generally applicable to any nonlinear data processing, this work focuses on penalized-likelihood reconstruction with a Huber penalty in CT. We prove feasibility of the proposed approach for a spiculated lesion in a digital phantom and demonstrate utility in prospective regularization optimization.

2. METHODS

2.1. Generalized System Response for Model-Based Reconstruction

We define the generalized system response, $\mathcal{H}$, as the transfer function that characterizes the system response to arbitrary lesion stimuli, μ_s. Mathematically, $\mathcal{H}$ is equal to the difference between the mean reconstructions with and without the stimulus, i.e.,

$$\mathcal{H}({\mu }_s;\mu )=\overline{\widehat{\mu }(y(\mu +{\mu }_s))}-\overline{\widehat{\mu }(y(\mu ))}$$ (1)

where $\widehat{\mu }$, y, and μ denotes the reconstruction, projection data, and the true object without the stimuli. In traditional linear reconstruction algorithms, $\mathcal{H}$ carries dependencies on the specifics of data acquisition (e.g., dose, detector characteristics, geometry) and reconstruction (e.g., smoothing kernels); nonlinear algorithms introduce additional dependencies on the object and the stimuli.

We may similarly define the generalized noise response, which corresponds to the second moment of the posterior distribution of the reconstructed image, e.g., $\mathcal{K}({\mu }_s;\mu )=cov\{\widehat{\mu }(y(\mu +{\mu }_s))\}$. Together, the generalized system and noise response functions form essential pieces of composite image quality metrics such as task-based detectability index, and can be used to infer texture information. This work will focus on establishing a predictive framework for $\mathcal{H}$, with future work extending similar analysis to $\mathcal{K}$.

2.2. Penalized-Likelihood Reconstruction with a Huber Penalty

The particular reconstruction algorithm in this work involves a penalized-likelihood objective with Huber penalty:

$$\widehat{\mu }=\underset{\mu }{\text{arg max}}L(\mu ;y)-\beta \sum _j\sum _{k\in \mathcal{N}}{\psi }_H({\mu }_j-{\mu }_k)\text{where}{\psi }_H(x)=\{\begin{array}{cc}\frac{1}{2\delta }{x}^2\hfill & \mid x\mid \le \delta \hfill \\ \mid x\mid -\delta ∕2\hfill & \mid x\mid >\delta \hfill \end{array}\phantom{\}}$$ (2)

This approach has demonstrated improved performance compared to linearizable penalties (e.g., quadratic).⁵ However, edge enhancement effects of the Huber penalty introduce shift-variant resolution which complicates traditional resolution metrics such as edge spread or modulation transfer function (which may apply only locally and be feature-dependent). In addition, the interplay between the two reconstruction parameters, β and δ, introduce complex dependencies to the reconstructed appearance of the lesion. Figure 1 illustrates the dependencies of $\mathcal{H}$ on location within the object and (β, δ) for a spiculated lesion stimulus to demonstrate 1) the intrinsic nonlinearity of $\mathcal{H}$, 2) strong shift-variance of the system response (lesion boundary vs. lesion interior), and 3) the importance of modeling the full system response.

Figure 1:

Generalized system response ($\mathcal{H}$) of a spiculated lesion computed from mean reconstructions (Eq.1) following the PL with Huber penalty objective (Eq.2). $\mathcal{H}$ shows dependence on (a) locations and (b) reconstruction parameters β and δ. In addition, $\mathcal{H}$ is nonlinear and shows strong shift-variance within the region of interest around the stimulus.

2.3. Universal Approximation Theorem and Prediction Framework Implementation

For estimating the nonlinear $\mathcal{H}$, we leverage the universal approximation theorem, which states that a feedforward neural network with a single hidden layer that contains a finite number of hidden neurons can approximate any continuous functions on compact subsets of ${\mathbb{R}}^n$. This property of neural networks has been employed in control theory for the identification and prediction of nonlinear dynamical systems.⁴

2.3.1. Integrating Prior Knowledge of Reconstruction Properties

From Fessler,³ image properties of PL reconstruction with a quadratic penalty are dependent on the imaging system and the object, but may be estimated via the local Fisher information, A^TW Ae_j, where A, W, and e_j are the forward-projector, diagonal weighting matrix (equal to the measurement y), and an impulse at voxel location j. Guided by this knowledge, we provide a similar quantity, A^TW Aμ_s, as an input to the neural network in order to provide prior knowledge of the system geometry (through A), the attenuation properties of the object (through W), and the lesion stimulus (size and contrast through μ_s, location within the object through A and W). Furthermore, the generalized system response is highly localized. We therefore conjecture that we only need to supply the network with a local region-of-interest (ROI) (chosen to be 21×21 voxels) around the stimulus in A^TW Aμ_s. Each input case therefore contains 443 numbers (441 voxels, β, and δ).

2.3.2. Efficient Network Architectures

We adopted a multilayer perceptron network with three hidden layers to reduce the number of nodes (compared to a single hidden layer stated in the universal approximation theorem). We identified combinations of activation functions that yield the best prediction results without compromising training speed. The network architecture and an example input-output pair are illustrated in Fig.2(a).

Figure 2:

(a) Structure of the multilayer perceptron network (b) Range of parameters for training and testing

2.4. Training, Testing, and Data Generation

For initial investigations in this work, we demonstrate feasibility of the prediction framework as a function of reconstruction parameters (β and δ) and stimulus locations. We focus on a single type of stimulus: a spiculated lesion of 1.5 mm radius and 0.01 mm⁻¹ contrast to the background; and a single elliptical phantom with two circular inserts (Fig.1). Lesion and phantom variabilities will be subjects of future investigation. The combinations of location, β, and δ for training and testing are illustrated in Fig.2(b). The range of β and δ were wide enough to encompass clinically desirable image quality. The testing parameters are non-overlapping with those used in training to test the network’s ability to interpolate between unsampled multi-parameter space. Input and output data were generated using an in-house reconstruction toolbox. The data forward model assumed an idealized imaging system with no detector blur or readout noise.

2.5. Example Application: Regularization Design for Controlled Lesion Appearance

In MBR, reconstructed appearance of stimuli are often location-dependent. For specific MBR like quadratic PL, uniform spatial resolution may be achieved through spatially varying regularization.³ However, in Huber PL, system response is also shift-variant and spatial resolution is a poor measure of the effect of $\mathcal{H}$. Thus, Huber regularization design has traditionally relied on exhaustive parameter sweeps and retrospective assessments of reconstructions. Using the prediction framework developed above, we may perform efficient regularization design to achieve a target lesion appearance without the need for additional reconstructions.

We selected a target lesion appearance with visible spiculations. For each location, the $\mathcal{H}$ as a function of finely sampled β and δ (over the range of the training data at 0.02 interval in the exponent) was predicted using the trained neural network. Structural similarity index (SSIM) ⁶ was computed between the predicted $\mathcal{H}$ and the target lesion and the (β, δ) yielding the maximum SSIM was selected for that location.

3. RESULTS

The predictive ability of the network is illustrated in Fig.3 where the generalized system response computed from mean reconstructions (Eq.1) are compared with predictions from the neural network. Results were shown for the testing parameters illustrated in Fig.2(b). Good agreement was observed between the reconstructed output (current gold standard) and the prediction as a function of β, δ, and locations, demonstrating the network’s ability to interpolate and predict the complex, nonlinear, and location-dependent outcomes for unseen combinations of parameters. That is, by training the network will a subset of reconstructions, we can map out the nonlinear function $\mathcal{H}$ and perform predictions the for arbitrary combinations of imaging parameters without the need for performing actual reconstructions.

Figure 3:

Predictive ability of the neural network model. Generalized system response computed from mean reconstructions (current gold standard) are compared with predictions from the trained network. Comparisons are shown as β, δ, and locations vary, respectively.

The utility of the prediction framework is demonstrated in efficient, location-dependent regularization design to achieve a uniform target lesion appearance at two example locations (Fig.4). We identify nearly continuous ranges of (β, δ) (indicated by cyan masks) that can achieve similar lesion appearance (within 0.1% of the maximum SSIM), a finding that would be challenging to obtain with traditional analysis methods where image properties need to be assessed retrospectively from reconstructions.

Figure 4:

Example application of the prediction framework in fast optimization of location-dependent reconstruction parameters to achieve a target lesion appearance as determined by structural similarity index (SSIM). The (β, δ) that yields the maximum SSIM is indicated by the green star. The range of (β, δ) with similar SSIM (within 0.1%) to the maximum is indicated by the cyan mask.

4. DISCUSSION AND CONCLUSION

This work presents a novel analysis framework for predicting the generalized system response of nonlinear MBR algorithms. For initial investigations, we demonstrated feasibility of a multilayer perceptron network to yield accurate predictions of the reconstructed appearance of a spiculated lesion as a function of locations within the phantom and combinations of reconstruction parameters β and δ.

Ongoing work includes: 1) incorporating lesion variability (size, shape, contrast) and phantom variability in the prediction framework; 2) including realistic system models such that network trained with simulated data can be used to predict responses from physical imaging systems; 3) investigating more efficient network architectures specific to tomography; and 4) establishing a prediction framework for the generalized noise response to allow for the evaluation of composite metrics such as task-based detectability index and texture measures of lesions.

The proposed framework is applicable to a wide range of algorithms, and permits efficient and quantifiable performance evaluation to provide the robust control and understanding of imaging output necessary for reliable clinical application and regulatory oversight. Such a framework will enable not only proper parameter specification across imaging conditions but also advanced adaptive patient- and task-specific protocols that use state-of-the-art processing routines.