Data-dependent Nonlinearity Analysis in CT Denoising CNNs

Abstract

Recent years have seen the increasing application of deep learning methods in medical imaging formation, processing, and analysis. These methods take advantage of the flexibility of nonlinear neural network models to encode information and features in ways that can outperform conventional approaches. However, because of the nonlinear nature of this processing, images formed by neural networks have properties that are highly data-dependent and difficult to analyze. In particular, the generalizability and robustness of these approaches can be difficult to ascertain. In this work, we analyze a class of neural networks that use only piece-wise linear activation functions. This class of networks can be represented by locally linear systems where the linear properties are highly data-dependent - allowing, for example, estimation of noise in image output via standard propagation methods. We propose a nonlinearity index metric that quantifies the fidelity of a local linear approximation of trained models based on specific input data. We applied this analysis to three example CT denoising CNNs to analytically predict the noise properties in the output images. We found that the proposed nonlinearity metric highly correlates with the accuracy of noise predictions. The analysis proposed in this work provides theoretical understanding of the nonlinear behavior of neural networks and enables performance prediction and quantitation under certain conditions.

1. INTRODUCTION

Image quality assessment using quantitative metrics such as mean and covariance is critical for imaging system optimization and task-based performance evaluation. In shift-invariant and linear systems, the point spread function can completely characterize the mean and noise propagation from the input to the output with simple convolutions. In nonlinear systems, however, the image properties are highly data-dependent and the mean and covariance propagation is difficult to generalize.

Deep learning methods have been widely applied to medical image formation, processing, and analysis. Unlike the traditional algorithms that often use a more specific model customized to the imaging system of interest, deep learning algorithms adopt neural networks packing a large amount of parameters within a “black box”. The flexibility of neural networks allow them to better represent complex systems. However, over-parameterization and opacity also raise concern about the generalizability and robustness. Quantitative image quality assessment in deep learning approaches is desired to validate the robustness, improve interpretability of the neural networks and further translation to clinical applications.

Image property analysis in deep learning methods is more complicated compared with the analytical and model-based methods due to their increasingly nonlinear nature.¹ Many methods have been developed to understand model predictions by deploying a trained model in a post-hoc fashion.² For example, attribution methods like integrated gradient³ forward propagate the change of the input to the output and illustrate how input features influence the final inference estimates by exhaustively integrating the gradients from the reference instance to the instance of interest. If the gradients on the path remains constant, the system is then locally linearizable and propagation methods with a local linearization is again valid for image properties analysis. Such local linearity was previously investigated as the gradient stability and was regarded as an important feature of robust networks.⁴

In this work, we discuss the local linearity of a class of neural network models using piece-wise linear activation layers and explains that the nonlinearity of a trained neural network is dependent on particular testing dataset and proposed a data-dependent nonlinearity index that quantifies the fidelity of a local linear approximation specific to the trained model and the testing dataset. We applied this analysis to three example CT denoising CNNs and computed the nonlinearity indices with respect to different testing datasets. We further investigate the correlation between the proposed nonlinearity indices and the accuracy of noise prediction using a local linear approximation.

2. THEORY

2.1. Local linearity of neural networks

Generally, a neural network consists of two types of layers. The first type includes linear transformations between the input and output - e.g., a fully-connected layer or a convolutional layer. The second type features nonlinear operations including nonlinear transformations like the rectified linear unit (ReLU) or leaky ReLU. Consider a network block consisting of a linear layer and a ReLU activation layer. The prediction y with a given input x can be written as:

$$y=f(Wx+b),f(t)=\text{max}(0,t),$$ (1)

where W and b denote the weights and the biases of the linear layer, respectively. The ReLU activation function f(·) is an element-wise operator that only outputs non-negative values and returns zero when the input values are negative. This process is equivalent to multiplication with a diagonal matrix S, where the diagonal element s_j is 1 (the neuron is turned on) if the input t_j is non-negative or 0 otherwise (the neuron is turned off). When deploying a trained model to make inferences, the matrix of activation states are determined by the specific input instance. We can rewrite Equation (1) as

$$y(x)=S(x)(Wx+b)$$ (2)

If the activation states are identical among an ensemble of inputs, all of the outputs can be described with the same linear function. Consequently, analytical methods for linear systems can be adopted to evaluate the system performance locally within that ensemble. Specifically, the nonlinear network can be characterized as a piece-wise linear system, and a first-order Taylor expansion may be used to characterize the system locally:

$$y(x)={\mathcal{J}}_{x_0}\left(x-x_0\right)+y\left(x_0\right).$$ (3)

We use ${\mathcal{J}}_{x_0}\in {ℝ}^{I\times J}$ to denote the Jacobian matrix ∂y/∂x = S(x₀)W at x₀, where x₀ can be any input instance within an ensemble sharing the same activation state. We can therefore propagate noise using standard methods in linear systems:

$$\text{Cov}(y)=\mathcal{J}\text{Cov}(x){\mathcal{J}}^T,$$ (4)

where Cov(y) and Cov(x) are the covariance matrices of y and x.

2.2. Data-dependent nonlinearity metric

In the previous section, we describe the local linearity of a class of neural networks that use piecewise-linear activation functions and define the locality as an ensemble of inputs that share the same activation states. However, in practice, such an ensemble may be difficult to define or quantify. For example, small perturbations to a signal may result in a change of activation state, so may different noise realizations following the same distribution - i.e., inputs normally considered to be within the same class may be subjected to different linear transforms. To accurately characterize the system, one therefore needs to exhaustively find all states activated by the input instances of interest and find the linear transform for each instance. However, one can always find a linearity approximation locally and evaluate the deviation between the approximation and the ground truth. For inputs of interest following a distribution p(x), we propose the optimal linear approximation with a mean Jacobian matrix $\overline{\mathcal{J}}=\int {\mathcal{J}}_{x}p(x)dx$ and a data-dependent nonlinearity index as the expectation of the Frobenius distance between the Jacobian matrices at each instance and $\overline{\mathcal{J}}$

$$D^{\text{nonlinear }}=\int {\left|{\mathcal{J}}_x-\overline{\mathcal{J}}\right|}^{2}p(x)dx$$ (5)

If the linear approximation is accurate for the all inputs in p(x), the proposed nonlinearity index reaches the minimum 0; when a local linearity approximate is inaccurate, e.g., when different noise realization results in many changes in activation states, the D^nonlinear metric increases.

2.3. Illustration in a toy problem

A toy example is illustrated in Fig.1 where we trained a two-dimensional denoising network consisting of a single-layer of perceptrons with 5 hidden nodes. We simulated 1,000,000 pairs of training data according to the following rules. The output vectors, [y₁, y₂], are correlated with y₁ sampled from a uniform distribution, y₁ ~ Unif(0, 10), and y₂ = 2y₁. The input vectors, [x₁, x₂], are then sampled from Gaussian distributions with mean equal to [y₁, y₂] and covariance equal to [0.25,0.3;0.3,1.0].

Figure 1:

(a) Bivariate denoising PNN. (b) 3D surface plot of y₁(x₁, x₂) with two perturbation tests and projection in the (x₁, x₂) plane. (c) Scatter plots of 2 noisy clusters. (d) Histograms of x₁ and ${\widehat{y}}_1$.

Figure 1(b) shows the surface plot of predicted y₁ with (x₁, x₂) around (0,0). We observed that the trained model is an aggregation of 15 linear models, split with 5 lines in the input space corresponding to the five hidden neurons. We further tested the trained model with two perturbation paths, each consisting of 2,000 points linearly interpolated between (0,1) and (1,3) (green line) and between (0,1) and (1,−3) (red line). Linearity is well-preserved between (0,1) and (1,3) but not between (0,1) to (1,−3).

To illustrate the data-dependent nonlinearity with noise injection, we generated two testing data consisting of 10,000 samples with y₁ = 0 and y₁ = 10. The scatter plots are summarized in Figure 1(c), where the color of each scatter point represents the activation state it lies in. The normalized histograms of x₁ and ${\widehat{y}}_1$ are shown. In the y₁ = 10 testing dataset, all inputs fall within a single activation state. Therefore, for all inputs around y₁ = 10 can be described by a linear system, where normally distributed input results in normally distributed output. In the y₁ = 0 testing dataset, however, depending on the particular noise realization, the inputs lie in different activation states. Negative values are effectively subjected to a non-negativity constraint where the values are clipped at 0. Therefore, the system response for the y₁ = 0 testing dataset is nonlinear and the output distribution is non-Gaussian. With the above model and testing data examples, we demonstrate that certain classes of neural network are effectively piece-wise linear and the nonlinearity of the system is dependent on the particular input instance.

3. APPLICATION IN CT-DENOISING CNNS

3.1. CT image denoising using deep learning

We performed the linearity analysis for noise propagation in two example CNNs for CT image desnoising: 1) a simple 3-layer CNN structure with ReLU activation, each hidden layer consisting of 8 channels and 3 × 3 kernels with strides set to 1; 2) a more complex RED-CNN structure⁵ with 10 convolution layers/ 96 channels followed by ReLU activation, 5 × 5 kernels with strides equal to 1, and skip connections.

For training data, we selected 2,000 normal-dose images from the Cancer Imaging Archive (TCIA)⁶ as the ground truth anatomy and simulated noisy FBP reconstructions at low, medium, and high-dose levels corresponding to a barebeam fluence of I₀ =10⁴,10⁵,10⁶, respectively. The simple CNN were trained using 1,000,000 paired patches of size 32 × 32 pixels at different noise reduction levels: 1) CNN I: trained with (I₀ =10⁴, I₀ =10⁶) pairs, 2) CNN II with (I₀ =10⁵, I₀ =10⁶) pairs. The models were trained individually, each using 300 epochs of the Adam algorithm⁷ to minimize the mean-squared error between the denoised images and the normal-dose images.

3.2. Noise propagation with linear approximation

To evaluate the noise propagation, we selected a thorax CT slice from the TCIA (Figure 2). We simulated 100 noisy realization at two dose levels (I₀ =10⁴,10⁵). The variance maps of the denoised images using the trained CNN models were empirically measured with the 100 noisy instances and compared with analytical calculation following the local linear approximation using one noisy realization. To speed up computation, we took advantage of the finite size of the receptive field and computed the Jacobian matrices and the covariance within patches of 32 × 32 pixels at strides of 16 pixels. The variances were extracted from the covariance matrices and stitched together to generate a form variance map of the entire image. We also computed the nonlinearity index maps in a local region of interest (ROI shown in Figure 2) according to:

$$D_i^{\text{nonlinear }}=\frac{1}{NJ}\sum _j\sum _n{\left({\mathcal{J}}_{n,i,j}-\frac{1}{N}\sum _n\left({\mathcal{J}}_{n,i,j}\right)\right)}^2$$ (6)

where ${\mathcal{J}}_{n,i,j}$ denotes ∂y_i/∂x_j in the n^th noisy realization. These nonlinearity index maps were compared with the variance error maps (difference between empirical calculation and analytical prediction) to validate if the proposed nonlinearity indices can reflect the fidelity of the local linear approximation.

Figure 2:

2D thorax CT image and simulated noisy images at different dose levels. The yellow box in the ground truth image circumscribes the region of interest for nonlinearity quantification with the proposed nonlinearity index.

4. RESULTS

The variance maps of the noisy images and the denoised images are summarized in Figure 3. The variance of the denoised images is proportional to the variance of the noisy input images. Different denoising models result in different variance magnitudes and spatial distributions of noise. Comparing the two CNNs - the one trained with lower-dose inputs exhibits higher noise reduction and potential for streaking artifacts. The variance maps of the RED-CNN-denoised images are different from those of the input and both CNNs. Variance calculated using the linear approximation shows good agreement with measurements across different models and inputs.

Figure 3:

(a) The variance maps of the input at low- and medium-dose. Measured and predicted variance maps of denoised images with (b) low-dose and (c) medium-dose testing sets. The columns show results of CNN I, CNN II, RED-CNN, from left to right.

The nonlinearity index maps are summarized in Figure 4(a) and the the difference between the approximated and the measured variance maps in Figure 4(b). The display window for each network model is ± 1/10 of the maximum variance with low-dose inputs. The mismatch between the measured and the approximated variance increase where a higher nonlinearity index is reported. The proposed nonlinearity indices reveal the deviation of the neural network models from a local linear approximation w.r.t the noisy inputs. We observe that when the input images contain higher noise (lower dose), the data-dependent nonlinearity increases. This is expected because when the noise is higher, the inputs span a large range in the input space and can potentially activate different linear states. Among all trained networks, the RED-CNN exhibits the highest nonlinearity, potentially because of deeper network architecture that allows a larger number of activation states.

Figure 4:

(a) Nonlinear index maps. (b)Variance approximation errors compared with the measurements. Each row summarizes results generated with input noisy images at different dose levels. Each column corresponds to different denoising networks.

5. DISCUSSION AND CONCLUSION

We discussed the local linearity nature of neural networks using piece-wise constant activation functions and proposed a data-dependent nonlinearity index to evaluate when a linear approximation can be applied to the inputs of interest. Using local linearizations, we analytically propagated noise in pretrained desnoising neural networks for a range of testing cases. We showed that with local linearization can accurately approximate the magnitude and distribution of noise in the denoised images. The data-dependent nonlinearity indices correlate well with the accuracy of local linear approximations.

The proposed analysis was developed for networks with piece-wise linear activation functions. For networks that use other activation functions, e.g., the sigmoid function or the hyperbolic tangent function, the model is no longer piece-wise linear. The activation states cannot be properly discretized. However, the local linearization framework with gradients computation and the proposed nonlinearity indices are still valid. For example, a nonlinear but flat function can still be approximated with a locally linear function with a relatively low mismatch. Example networks with non-piece-wise-linear activation functions will be included in future work.

The proposed analysis provides theoretical understanding of the nonlinear and data-dependent nature of certain neural networks and bring us one step closer to rigorous quantitation and interpretability of network performance.