Reconstructing and analyzing the invariances of low‐dose CT image denoising networks

Abstract

Background

Deep learning‐based methods led to significant advancements in many areas of medical imaging, most of which are concerned with the reduction of artifacts caused by motion, scatter, or noise. However, with most neural networks being black boxes, they remain notoriously difficult to interpret, hindering their clinical implementation. In particular, it has been shown that networks exhibit invariances w.r.t. input features, that is, they learn to ignore certain information in the input data.

Purpose

To improve the interpretability of deep learning‐based low‐dose CT image denoising networks.

Methods

We learn a complete data representation of low‐dose input images using a conditional variational autoencoder (cVAE). In this representation, invariances of any given denoising network are then disentangled from the information it is not invariant to using a conditional invertible neural network (cINN). At test time, image‐space invariances are generated by applying the inverse of the cINN and subsequent decoding using the cVAE. We propose two methods to analyze sampled invariances and to find those that correspond to alterations of anatomical structures.

Results

The proposed method is applied to four popular deep learning‐based low‐dose CT image denoising networks. We find that the networks are not only invariant to noise amplitude and realizations, but also to anatomical structures.

Conclusions

The proposed method is capable of reconstructing and analyzing invariances of deep learning‐based low‐dose CT image denoising networks. This is an important step toward interpreting deep learning‐based methods for medical imaging, which is essential for their clinical implementation.

1. INTRODUCTION

Deep learning‐based methods have revolutionized the field of medical image formation in general and computed tomography (CT) in particular by delivering cutting‐edge solutions to a wide range of problems. These include noise reduction, ¹ , ² , ³ , ⁴ , ⁵ image reconstruction, ⁶ , ⁷ , ⁸ scatter estimation, ⁹ , ¹⁰ , ¹¹ and artifact reduction. ¹² , ¹³ Most of these problems, however, are not injective, meaning that a single target‐domain (e.g., artifact‐free) image can be derived from different source‐domain (e.g., artifact‐deteriorated) images. Therefore, a good network for these tasks must be invariant to some input features (e.g., image noise for low‐dose reconstruction) to some extent. ¹⁴ From a network architecture perspective, invariances can be realized by certain noninjective layers such as max‐pooling layers or convolutions with certain weight configurations.

In this study, we aim to investigate and interpret these invariances in low‐dose computed tomography (LDCT) image denoising networks — a prevalent application of deep learning in CT image formation. Such an analysis can provide valuable insights into the networks behavior and help in identifying potential biases or shortcomings of the networks and their training data. This is important in order to improve the interpretability and robustness of deep learning‐based methods for medical imaging, which is an essential step toward bridging the implementation gap of deep learning‐based methods in medical imaging. ¹⁵ , ¹⁶

1.1. Deep learning‐based low‐dose CT image denoising

While our method for reconstructing and analyzing invariances of image‐to‐image translation networks is applicable to a wide range of deep learning‐based applications for CT and other modalities, we here focus on the task of LDCT due to the abundance of publications in the field^* and the availability of open‐source datasets.

LDCT aims at providing an image $x$ with a lower dose than conventional CT acquisitions, which is typically accomplished by decreasing the tube current and consequently reducing the x‐ray flux. However, this approach increases noise in the projection data due to photon starvation. As a result, when these images are reconstructed using standard filtered back projection (FBP), they exhibit unwanted noise and streak artifacts, potentially reducing diagnostic value.

To mitigate these artifacts, advanced reconstruction techniques such as iterative reconstruction can be employed. These methods effectively suppress the artifacts but are computationally expensive, often limiting their clinical applicability in time‐critical scenarios, such as emergency rooms. On the other hand, denoising methods present a computationally efficient solution and can be integrated seamlessly into any existing reconstruction pipeline. These algorithms may be conventional, ¹⁷ , ¹⁸ , ¹⁹ , ²⁰ or data‐driven ¹ , ² , ²¹ , ²² , ²³ and can be applied in either projection domain, image domain, or both. Particularly, deep learning‐based methods applied to reconstructed images are prevalent in the literature since they do not require access to the (often proprietary) projection data.

Deep learning‐based image domain denoising methods usually learn a mapping $f_{\theta }:x^{(i)}\to y^{(i)}$ from low‐dose images $x^{(i)}$ (i.e., images reconstructed from low‐dose projections via FBP) to high‐dose images $y^{(i)}$, where $f_{\theta }$ is a deep neural network (DNN) with parameters $\theta$. Most methods optimize the parameters in a supervised fashion by minimizing some (typically pixel‐wise) loss $\mathcal{L}$ over the training set ${\{(x^{(i)},y^{(i)})\}}_{i=1}^N$

$$\begin{array}{c}\hfill {\theta }^{\ast }=\underset{\theta }{arg\min }\frac{1}{N}\sum _{i=1}^N\mathcal{L}\left(f_{\theta }(x^{(i)}),y^{(i)}\right).\end{array}$$ (1)

Numerous other works train $f_{\theta }$ unsupervised or self‐supervised. These include methods leveraging the image prior of convolutional neural networks (CNNs), ⁸ intrinsic similarities within the training data (e.g., across views or patches), ⁵ , ²⁴ , ²⁵ , ²⁶ , ²⁷ or methods from deep metric learning (DML). ²⁸ We refer the reader to Lei et al., 2024 ²⁹ for a comprehensive review of these methods.

For a fair comparison between denoising algorithms, we henceforth focus on methods trained using Equation 1 that vary in their architectural design of $f_{\theta }$ and the choice of $\mathcal{L}$ used for learning the parameters $\theta$.

1.2. Reconstructing invariances of DNNs

Previously, Rombach et al. ¹⁴ presented a method to reconstruct the invariances of some image classification network $f:{\mathbb{R}}^{n\times m}\to {\{0,1\}}^c$, with $n\times m$ being the image size and $c$ the number of classes, using conditional invertible neural networks (cINNs). Let $\overline{z}\in {\mathbb{R}}^d$ denote any internal latent representation (e.g., if $d=n\times m\times 64$ this could be the output of a zero‐padded convolutional layer with 64 filters) that we can get by decomposing $f$ into $f(x)=\mathrm{\Psi }(\overline{z})=\mathrm{\Psi }(\mathrm{\Phi }(x))$, where $\mathrm{\Phi }:{\mathbb{R}}^{n\times m}\to {\mathbb{R}}^d$ and $\mathrm{\Psi }:{\mathbb{R}}^d\to {\{0,1\}}^c$. To then find out which information about $x$ is captured in $\overline{z}$ and which is missing (i.e., the invariances of $\mathrm{\Phi }$), we need a compact data representation of $x$. The authors propose to learn such a data representation $z$ by training a variational autoencoder (VAE) comprised of an encoder $E$ and decoder $D$. Since $z=E(x)$ now not only contains the information of $x$ that is captured in $\overline{z}$, but also $\varphi$’s invariances $v$, we need to disentangle these two components. This is achieved by training a normalizing flow $t(·|\overline{z}):z\to v$ that maps between those two domains, conditioned on the network representation $\overline{z}$. Since $t$ is invertible, we can then sample from $p(v)$ (here assumed to be normal) and apply $t^{-1}$ to obtain samples from $p(z)$. Finally, we can reconstruct the invariances of $\mathrm{\Phi }$ in image space by applying the pretrained decoder $D$ to the samples $z$.

This method has later been adapted to reconstruct the invariances of CT image denoising networks. ³⁰ However, due to the fact that LDCT denoising networks exhibit fewer and more subtle invariances than image classification networks, some reconstructed invariances may be attributed to the VAE rather than the denoising networks. This is further exacerbated by the diversity of medical image data, which makes it difficult for the VAE to learn an almost complete data representation of the input data. In this work, we propose to reconstruct the invariances of LDCT denoising networks by training a conditional VAE, therefore improving its data representation compared to previous works. We also investigate the invariances of more recent and advanced denoising networks and introduce methods to analyze the sampled invariances.^†

2. METHODS

In the following we will present a method to sample and analyze the invariances of LDCT denoising networks. Note that the method presented herein is network and application‐agnostic and therefore potentially applicable to many other image‐to‐image translation tasks in medical imaging such as metal artifact correction or sparse view CT.

2.1. Dataset

For all our experiments, we use the 50 chest exams provided in the open‐source Low‐dose CT Image and Projection Dataset. ³¹ For each scan in the dataset, the authors simulated low‐dose reconstructions by inserting noise in the projection domain. These reconstructions correspond to a dose level of 10%.

We randomly split the respective acquisitions (on a patient level) into 70% training (35 patients), 20%validation (10 patients), and 10% (5 patients) test data. During training and validation, we employ a weighted sampling scheme, ensuring that every acquisition has an equal probability of being selected, regardless of the varying number of slices per acquisition. All data are normalized to have zero‐mean, unit‐variance before feeding them to the networks.

2.2. Denoising methods

We reconstruct the invariances of four different deep learning‐based image denoising algorithms, which are summarized in the following. We refer the reader to the respective publications for more details.

CNN‐10 ¹ One of the earliest deep learning‐based methods for LDCT image denoising. The authors propose a simple three‐layer CNN which receives low‐dose images as input and is trained using Equation 1 with the mean‐squared‐error loss.

RED‐CNN ² This method builds upon CNN‐10 by incorporating a deeper residual encoder‐decoder architecture but keeps the overall training procedure identical. In previous works, ³² it has been shown that this method outperforms many other (and notably newer) deep learning‐based denoising methods.

WGAN‐VGG ³ The authors improve on CNN‐10 by using a deeper network architecture and by training it together with a convolutional critic as Wasserstein‐GAN. ³³ Furthermore, they added a perceptual loss ³⁴ derived from a pretrained VGG to the overall generator loss. In comparison with traditional pixel‐wise loss functions, this approach leads to denoised samples that exhibit more refined details and authentic noise textures.

DU‐GAN ²² Similar to WGAN‐VGG, the authors employ an adversarial training scheme, but use a U‐Net‐based discriminator ³⁵ which allows for per‐pixel feedback to the generator network, for which they use the same structure as RED‐CNN.

All four methods are trained using the data as described in Section 2.1 and we use the best performing network on the validation data for subsequent invariance reconstruction. Additional training specifics for each method are provided in Supplementary Materials A.1.

2.3. Reconstructing invariances

Our pipeline to reconstruct invariances (Figure 1) comprises three components:

• (a)The LDCT denoising network $f_{\theta }$, that receives low‐dose images $x^{(i)}$ as input and predicts high‐dose images ${\widehat{y}}^{(i)}=f_{\theta }\left(x^{(i)}\right)$ (Section 2.2).
• (b)A conditional VAE $D_{\psi }\circ E_{\phi }$ that is trained to learn a complete data representation $z\in {\mathbb{R}}^M$ of the low‐dose images. We condition both encoder $E_{\phi }$ and decoder $D_{\psi }$ on predictions of the denoising network ${\widehat{y}}^{(i)}$, thereby improving their encoding/reconstruction capabilities (Section 2.3.1).
• (c)A cINN that disentangles the information in $z$ that the denoising network $f_{\theta }$ is invariant to from the one it is not invariant to. To reconstruct invariances, we then sample from the Gaussian distribution of invariances, apply the inverse cINN, and decode the samples using the (fixed) conditional decoder.

FIGURE 1.

Overview of our method to reconstruct invariances of LDCT image denoising networks. Solid arrows represent inputs/outputs to modules, dotted arrows represent conditional inputs to a module. We indicate points where loss functions $\mathcal{L}$ are calculated using $\mathcal{L}$. Training: (a) training of the denoising network $f_{\theta }$ using low‐dose images $x^{(i)}$ and corresponding high‐dose images $y^{(i)}$. ${\mathcal{L}}_{\text{LDCT}}$ can be some pixel‐wise or adversarial loss, or a combination of both (Equation 1); (b) training of the conditional VAE with encoder $E_{\phi }$ and decoder $D_{\psi }$ conditioned on the denoised images ${\widehat{y}}^{(i)}=f_{\theta }(x^{(i)})$. ${\mathcal{L}}_{\text{KL}}$ and ${\mathcal{L}}_{\text{Rec}}$ are the Kullback–Leibler divergence and reconstruction loss, respectively (Equation 4); and (c) training of the conditional INN $t_{\xi }$ to disentangle the invariances of the denoising network $f_{\theta }$ from the latent representation $z^{(i)}$ learned by the VAE. ${\mathcal{L}}_{\text{cINN}}$ is the loss function of the cINN (Equation 6). Inference: We sample new invariances $v^{(k)}$ from the Gaussian distribution of invariances and apply the inverse cINN to obtain samples $z^{(i,k)}$. We then decode the samples using the conditional decoder $D_{\psi }$ to obtain the invariance reconstructions ${\stackrel{\sim }{x}}^{(i,k)}$ (Section 2.3).

2.3.1. Training of the conditional VAE

In order to reconstruct which information of low‐dose images a given denoising network $f_{\theta }$ has learned to represent and which to ignore (i.e., its invariances), we first need to learn an (almost) complete representation of low‐dose images $x^{(i)}$. We do so by training a conditional variational autoencoder comprised of a conditional probabilistic encoder $E_{\phi }$ defining the distribution $q_{\phi }(z|x,\widehat{y})$ and conditional probabilistic decoder $D_{\psi }$ defining $p_{\psi }(x|z,\widehat{y})$. We assume a Gaussian prior $p(z|\widehat{y})$ on latent variables $z$ and approximate the posterior with a Gaussian $q_{\phi }(z|x,\widehat{y})$ with diagonal covariance. Let ${\mu }^{(i)},{\sigma }^{(i)}\in {\mathbb{R}}^M$ denote the mean and standard deviation predicted by the encoder $E_{\phi }$ for the $i$‐th sample $x^{(i)}$, conditioned on its respective denoised image ${\widehat{y}}^{(i)}$. Then,

$$\begin{array}{c}\hfill \ln q_{\phi }(z|x^{(i)},{\widehat{y}}^{(i)})=\ln \mathcal{N}(z;{\mu }^{(i)},diag({\sigma }^{(i)})).\end{array}$$ (2)

As for any variational autoencoder, ³⁶ both encoder and decoder are trained to maximize the expectation ${\mathbb{E}}_i\left[\text{ELBO}(x^{(i)},{\widehat{y}}^{(i)})\right]$ with the evidence lower bound (ELBO) being

$$\begin{array}{ccc}\hfill \text{ELBO}(x^{(i)},{\widehat{y}}^{(i)})& =& {\mathbb{E}}_{z\sim q_{\phi }(z|x^{(i)},{\widehat{y}}^{(i)})}\left[\ln p_{\psi }(x^{(i)}|z,{\widehat{y}}^{(i)})\right]\hfill \\ & & -D_{\text{KL}}\left(q_{\phi }(z|x^{(i)},{\widehat{y}}^{(i)})||p_{\psi }(z|{\widehat{y}}^{(i)})\right),\hfill \end{array}$$ (3)

where $D_{\text{KL}}(q||p)$ denotes the Kullback–Leibler (KL) divergence between distributions $q,p$. Using the fact that the KL divergence between two Gaussians can be computed analytically, we derive the loss function

$$\begin{array}{ccc}\hfill {\mathcal{L}}_{\text{VAE}}(\phi ,\psi )& =& -{\mathbb{E}}_i\left[\phantom{\sum _{m=1}^M}\ln p_{\psi }(x^{(i)}|z^{(i)},{\widehat{y}}^{(i)})\right.\hfill \\ & & +\left.\frac{1}{2}\sum _{m=1}^M\left(1+\ln {({\sigma }_m^{(i)})}^2-{({\mu }_m^{(i)})}^2-{({\sigma }_m^{(i)})}^2\right)\right]\hfill \\ & =& -{\mathbb{E}}_i\left[\underset{{\mathcal{L}}_{\text{Rec}}}{\underbrace{\parallel x^{(i)}-{\widehat{x}}^{(i)}\parallel }}+\underset{{\mathcal{L}}_{\text{KL}}}{\underbrace{\frac{1}{2}{\sum }_{m=1}^M\left(1+\ln {({\sigma }_m^{(i)})}^2-{({\mu }_m^{(i)})}^2-{({\sigma }_m^{(i)})}^2\right)}}\right],\hfill \end{array}$$ (4)

where $M=dim(z)$, ${\widehat{x}}^{(i)}=D_{\psi }({\mu }^{(i)}+{\sigma }^{(i)}\epsilon |{\widehat{y}}^{(i)})$ with $\epsilon \sim \mathcal{N}(0,I)$.

Conditioning the VAE on auxiliary information ³⁷ eases the task for both encoder and decoder, as they can focus on the information about the input image that is not contained in the auxiliary information (here: the denoised image $\widehat{y}$) already^‡.

In our experiments, both $E_{\phi }$ and $D_{\psi }$ are parameterized by DNNs, with $E_{\phi }$ being an ImageNet‐pretrained ResNet‐50 ³⁸ and $D_{\psi }$ based on BigGAN. ³⁹ To improve reconstruction quality, we use a perceptual loss ³⁴ and adversarial loss in addition to the pixel‐wise loss in Equation 4. We refer the reader to Supplementary Material A.2 for more details on the training procedure. For comparison, we also train a VAE without the conditioning on $\widehat{y}$ (as explored in previous works ¹⁴ , ³⁰ ), but otherwise identical architecture and training procedure.

2.3.2. Training of the conditional invertible neural network

The latent representation $z$ does not only contain invariances of the denoised image ${\widehat{y}}^{(i)}$ but also information about the input image $x$. Therefore, we need to disentangle these two components, that is, extract the invariances $v$ of the denoising networks' prediction ${\widehat{y}}^{(i)}$ from the other information in $z$. Thus, we need to learn a mapping $t_{\xi }$ from $z$ to some space of invariances, given a denoised image ${\widehat{y}}^{(i)}$. Let this space of invariances $p(v)$ be a standard Gaussian distribution, that is, $p(v)\sim \mathcal{N}(0,I)$. Then, $t_{\xi }:p(z|\widehat{y})\to p(v)$ allows us to generate $v^{(i)}=t_{\xi }(z^{(i)}|{\widehat{y}}^{(i)})$ for any given sample $i$. In our experiments, $t_{\xi }$ is realized by a conditional invertible neural network with parameters $\xi$, that is, a normalizing flow conditioned on ${\widehat{y}}^{(i)}$. ⁴⁰ , ⁴¹ , ⁴² , ⁴³

As for any cINN, ⁴³ we can find optimal parameters $\xi$ via standard maximum likelihood training. Using the change‐of‐variables formula gives us the likelihood

$$\begin{array}{c}\hfill q(z^{(i)}|{\widehat{y}}^{(i)},\xi )=p\left(t_{\xi }(z^{(i)}|{\widehat{y}}^{(i)})\right)\left|J^{(i)}\right|,\end{array}$$ (5)

with $J^{(i)}=det\left(\frac{\partial t_{\xi }(z^{(i)}|{\widehat{y}}^{(i)})}{\partial z^{(i)}}\right)$. The loss function over training samples $i$ then reads as

$$\begin{array}{cc}\hfill {\mathcal{L}}_{\text{cINN}}& ={\mathbb{E}}_i\left[-\ln q(z^{(i)}|{\widehat{y}}^{(i)},\xi )\right]\hfill \\ & ={\mathbb{E}}_i\left[-\underset{\ell (t_{\xi }(z^{(i)}|{\widehat{y}}^{(i)});0,1)}{\underbrace{\ln p\left(t_{\xi }(z^{(i)}|{\widehat{y}}^{(i)})\right)}}-\ln \left|J^{(i)}\right|\right]\hfill \\ & ={\mathbb{E}}_i\left[\frac{1}{2}{\parallel t_{\xi }(z^{(i)}|{\widehat{y}}^{(i)})\parallel }_2^2-\ln \left|J^{(i)}\right|\right],\hfill \end{array}$$ (6)

where in the last step we used the log‐likelihood over samples $x=\left\{x_1,x_2,\cdots ,x_N\right\}$ of a standard Gaussian distribution $\ell (x;\mu ,{\sigma }^2)=-N\ln \sigma -\frac{N}{2}\ln (2\pi )-\frac{1}{2{\sigma }^2}{\parallel x-\mu \parallel }_2^2$ and the assumption that $p(v)$ is a normal distribution with zero mean and unit variance. The first line in Equation 6 is the log‐likelihood of observing some representation $z^{(i)}$ given the corresponding denoised image ${\widehat{y}}^{(i)}$ under parameters $\xi$. After optimization of parameters $\xi$, we can sample from $p(v)$ and apply the inverse $t_{\xi }^{-1}$ to map invariances to the input data representation, conditioned on the denoised image ${\widehat{y}}^{(i)}$. We refer the reader to Supplementary Material A.3 for more details on the architecture and training of $t_{\xi }$.

2.3.3. Sampling invariances

Once conditional VAE and cINN are trained, we can generate invariance samples ${\stackrel{\sim }{x}}^{(i,k)}$ for a given sample $x^{(i)}$ from the test set and a (trained) denoising network $f_{\theta }$ as follows:

• 1.Denoise the image using the pretrained denoising network $f_{\theta }$: ${\widehat{y}}^{(i)}=f_{\theta }(x^{(i)})$.
• 2.Sample $v^{(k)}\sim \mathcal{N}(0,I)$ from the space of invariances.
• 3.Apply the inverse of the cINN $t_{\xi }^{-1}$ to the sampled invariance: $z^{(i,k)}=t_{\xi }^{-1}(v^{(k)}|{\widehat{y}}^{(i)})$.
• 4.Decode the samples using the (fixed) conditional decoder $D_{\psi }$ to obtain the invariance reconstructions ${\stackrel{\sim }{x}}^{(i,k)}=D_{\psi }(z^{(i,k)}|{\widehat{y}}^{(i)})$.

Every ${\stackrel{\sim }{x}}^{(i,k)}$ is then a sample from the distribution of invariances of the denoising network $f_{\theta }$ for the $i^{\mathrm{th}}$ low‐dose image $x^{(i)}$ and two images ${\stackrel{\sim }{x}}^{(i,k)},{\stackrel{\sim }{x}}^{(i,l)}$ differ only in their realization of invariances.

2.4. Analyzing invariances

In our experiments, we find that the most prominent invariances of LDCT denoising networks are related to the noise level and noise realization of the input images. While this is expected and a desirable property of any denoising algorithm, this does not answer our initial question of whether LDCT denoising networks are invariant to anatomical structures or other image content. Finding such differences in the pixel space is challenging, as differences in noise realizations can easily overshadow differences in content. We therefore propose to analyze the invariances in an embedding space instead and compare two different methods to do so (Figure 2). The first is based on an embedding learned by an unconditional VAE (Sections 2.4.1, and Figure 2a). The second is based on a learned embedding of the invariances using a DML approach (Sections 2.4.2, and Figure 2b).

FIGURE 2.

Overview of our methods to analyze sampled invariances. (a) based on the embedding of an unconditional VAE, with encoder $E_{\varphi }^{\prime }$, whose latent space is dominated by content‐related information. Applying the same encoder to the input image $x^{(i)}$ and sampled invariance ${\stackrel{\sim }{x}}^{(i,k)}$, we can measure the content similarity between the two and (b) based on a learned embedding which is trained with a triplet loss (Equation 8) to map low‐dose images $x^{(i)}$ closer to invariance samples ${\stackrel{\sim }{x}}^{(i,k)}$ corresponding to the same sample $i$ than to invariance samples ${\stackrel{\sim }{x}}^{(j,l)},j\ne i$.

2.4.1. Using an unconditional VAE

The conditional VAE is trained to learn a complete representation of the low‐dose images, which includes both their noise level and noise realization as well as the anatomical structures and other image content. However, since for the conditional VAE, the latent space follows the distribution $p(z|{\widehat{y}}^{(i)})$, we cannot compare different samples $i$ with another. Instead, we use an unconditional VAE for which $p(z)$ is standard Gaussian distributed but which is otherwise identical to the conditional one. Since noise is generally harder to model than content, we expect the learned representation $E_{\varphi }^{\prime }(x^{(i)})$ to be dominated by content‐related information.

We can then use differences in the latent space as a proxy for differences in anatomical content between invariance samples ${\stackrel{\sim }{x}}^{(i,k)}$ and low‐dose inputs $x^{(i)}$. To this end we compute the cosine similarity between the latent representations of the low‐dose input and invariance samples $z^{(i)}=E_{\varphi }^{\prime }(x^{(i)}),z^{(i,k)}=E_{\varphi }^{\prime }({\stackrel{\sim }{x}}^{(i,k)})$ as

$$\cos \left(z^{(i)},z^{(i,k)}\right)=\frac{z^{(i)}·z^{(i,k)}}{\parallel z^{(i)}\parallel \parallel z^{(i,k)}\parallel }.$$ (7)

2.4.2. Using a learned embedding

We can also learn an embedding of the invariances using a DML approach. Metric learning generally seeks to learn a metric function $h$ such that semantic relations between datapoints $x^{(i)},x^{(j)}\in \mathcal{X}$ are depicted by metric distances $d_h(x^{(i)},x^{(j)}):=d(h(x^{(i)}),h(x^{(j)}))$, with $d(·,·)$ being some distance, in the embedding $h(·)$. In DML, $h$ is typically parameterized by a deep neural network $h_{\nu }:=h$ with weights $\nu$ being learned by minimizing a loss function that encourages the network to map similar (w.r.t. some semantic relation) samples closer together than dissimilar ones. To do so, many different loss functions have been proposed, most popularly ranking‐based loss functions. ⁴⁴ , ⁴⁵ , ⁴⁶ We refer the reader to Roth et al., 2020 ⁴⁷ for a nice overview of training strategies in DML.

In our experiments, we use the triplet loss ⁴⁵ to learn an embedding in which low‐dose inputs $x^{(i)}$ are closer to invariance samples ${\stackrel{\sim }{x}}^{(i,k)},\forall k$ corresponding to the same sample $i$ (with same anatomy) than to invariances samples $x^{(j,l)},\forall l,j\ne i$ corresponding to different samples (with different anatomy). The loss function for $h_{\nu }$ then reads as

$$\begin{array}{c}\hfill {\mathcal{L}}_{\text{Triplet}}={\mathbb{E}}_{i,j}{\left[d_h\left(x^{(i)},{\stackrel{\sim }{x}}^{(i,k)}\right)-d_h\left(x^{(i)},{\stackrel{\sim }{x}}^{(j,l)}\right)+\alpha \right]}_{+},\end{array}$$ (8)

with $\alpha$ being some prespecified margin. In our experiments, we use a pretrained ResNet‐50 ³⁸ as $h_{\nu }$ and select triplets $(a,p,n):=(x^{(i)},{\stackrel{\sim }{x}}^{(i,k)},{\stackrel{\sim }{x}}^{(j,l)})$ using the semi‐hard triplet mining strategy. ⁴⁵ We refer the reader to Supplementary Material A.4 for more details on the training procedure.

3. RESULTS

3.1. Denoising of LDCT images

We first verify qualitatively that the denoising methods (compare Section 2.2) perform as expected and are able to denoise LDCT images similarly as reported in their respective publications. To this end, we show results for random axial slices of all five test patients in Figure 3. For each patient of the test set, we show the high‐dose image, the low‐dose image, and the respective denoised images. While all methods are able to reduce noise and streak artifacts compared to the low‐dose image, the results of WGAN‐VGG ³ and DU‐GAN ²² show more realistic noise structures and exhibit finer details compared to the two methods trained using a pixel‐wise loss exclusively. This is in line with the findings of Yang et al. ³ and Huang et al. ²² and can be attributed to the additional perceptual loss (for WGAN‐VGG) and adversarial loss (for both WGAN‐VGG and DU‐GAN).

FIGURE 3.

High‐dose, low‐dose, and denoising results for the four methods described in Section 2.2. We show results for random axial slices and crops of size $128\times 128\mathrm{px}$ for all five patients from the test set. Center ($C$) and width ($W$) are $C=-600\mathrm{HU}$, $W=1500\mathrm{HU}$.

Upon quantitative evaluation (Table 1), we find that RED‐CNN performs best in terms of the structural similarity index measure (SSIM), peak signal‐to‐noise ratio (PSNR), and root‐mean‐square error (RMSE). However, it is important to note that these metrics do not correlate well with human reader ratings (the gold standard in terms of medical image quality assessment) for computed tomography. ⁴⁸ , ⁴⁹ , ⁵⁰ Since this work is not concerned with the evaluation of the denoising methods themselves, but rather with their invariances, we do not further investigate the performance of the denoising methods and leave the development of better metrics for future work.

TABLE 1.

Quantitative evaluation of the denoising methods described in Section 2.2.

	SSIM	PSNR (dB)	RMSE (HU)
LD	0.312 ± 0.072	18.1 ± 2.5	236 ± 86
CNN‐10	0.56 ± 0.10	27.3 ± 2.1	72 ± 19
RED‐CNN	0.58 ± 0.10	28.0 ± 2.2	66 ± 18
WGAN‐VGG	0.505 ± 0.099	25.3 ± 2.2	91 ± 26
DU‐GAN	0.544 ± 0.096	26.3 ± 2.2	80 ± 22

Note: We report the mean and standard deviation of the SSIM, PSNR, and RMSE over all axial slices of the test set. Bold values highlight the best performing method for each metric.

3.2. VAE reconstructions

Next, we evaluate the reconstruction capabilities of the conditional VAE (Section 2.3.1) for random axial slices of all patients of the test set in Figure 4. We find that reconstructions of the conditional VAE (Figure 4; third row) are very similar to the input low‐dose images (Figure 4; second row) for all exam types. Additionally, we show the reconstructions of an unconditional VAE (Figure 4; last row) as it was used in previous work to reconstruct invariances of LDCT denoising networks ³⁰ for comparison. While the unconditional VAE is able to generate realistic low‐dose images that reflect, to some extent, the anatomical structures of the low‐dose input images, it fails to capture fine details and removes or hallucinates many of the anatomical structures in the reconstructions (compare red arrows in Figure 4). We show reconstruction results for all conditional VAEs (conditioned on different denoising networks) in Supplementary Material B.

FIGURE 4.

High‐dose, low‐dose, and VAE reconstructions for the conditional VAE (here conditioned on RED‐CNN) described in Section 2.3.1. Patients, axial slices and crops correspond to those shown in Figure 3. Additionally, we show the reconstructions of an unconditional VAE (as used in previous work ³⁰ ) for comparison. $C=-600\mathrm{HU}$, $W=1500\mathrm{HU}$.

3.3. Invariance reconstruction

Given the procedure described in Section 2.3.3 and shown in Figure 1, we sample 100 invariances for each of the four denoising networks on 1000 random crops of the test set. In Figure 5 we show three invariances for one of those random crops. We find that for all denoising networks sampled invariances mainly differ in terms of noise amplitude and realization. This is expected as the networks see many noise realizations as well as patients of different thickness (influencing the noise level) during the training. These differences in noise structure and amplitude overshadow possible differences in anatomical content between samples. We provide additional results in Supplementary Material B.

FIGURE 5.

Invariances for a random crop $i$ from the test set. Shown are low‐dose image $x^{(i)}$, high dose image $y^{(i)}$ and three reconstructed invariances ${\stackrel{\sim }{x}}^{(i,k)}$ for each of the four denoising methods. We also show standard deviations $\sigma \left({\{{\stackrel{\sim }{x}}^{(i,k)}\}}_{i=1}^K\right)$ over $K=100$ invariances. For CT Images: $C=-600\mathrm{HU}$, $W=1500\mathrm{HU}$, for standard deviations: $C=0\mathrm{HU}$, $W=400\mathrm{HU}$.

3.4. Analyzing invariances

In the VAE latent space

Next, we analyze sampled invariances in the latent space of the unconditional VAE. To this end we compute for each network and sampled crop $i$ the mean cosine similarity over sampled invariances $k=1,2,\cdots ,K$

$$\begin{array}{c}\hfill S_{\text{VAE}}^{(i)}=\frac{1}{K}\sum _{k=1}^K\cos \left(z^{(i)},z^{(i,k)}\right).\end{array}$$ (9)

In Figure 6 we show four crops, corresponding to the $\{0,1/3,2/3,1\}$ quantiles of the mean similarity $S_{\text{VAE}}^{(i)}$ over all test samples, for each network. Here we find that for samples with lower $S_{\text{VAE}}^{(i)}$ (left), most differences between $x^{(i)}$ and ${\stackrel{\sim }{x}}^{(i,k)}$ are in terms of anatomical content (red arrows). In contrast, for samples with higher $S^{(i)}$ (right) anatomical content is similar between $x^{(i)}$ and ${\stackrel{\sim }{x}}^{(i,k)}$ and differences are mainly in terms of noise amplitude and realization. This indicates that the latent space of the VAE is indeed dominated by anatomy‐related information and disentangling sampled invariances. We provide further analysis of the VAE latent space in Supplementary Material B.

FIGURE 6.

Invariances with increasing $S_{\text{VAE}}$ (left to right), that is, decreasing amount of content‐related invariances as measured in the VAE latent space, for each of the four denoising methods. $C=-600\mathrm{HU}$, $W=1500\mathrm{HU}$.

3.4.1. In the DML latent space

Next, we analyze sampled invariances using the DML‐based embedding. Similar as for the VAE, we measure similarity between $x^{(i)}$ and ${\stackrel{\sim }{x}}^{(i,k)}$ using the mean cosine similarity over sampled invariances $k=1,2,\cdots ,K$

$$\begin{array}{c}\hfill S_{\text{DML}}^{(i)}=\frac{1}{K}\sum _{k=1}^K\cos \left(h(x^{(i)}),h({\stackrel{\sim }{x}}^{(i,k)})\right)\end{array}$$ (10)

and show four samples with increasing mean similarity $S_{\text{DML}}^{(i)}$, again corresponding to the $\{0,1/3,2/3,1\}$ quantiles of the empirical distribution, in Figure 7. For all denoising networks, we find that samples with lower $S_{\text{DML}}^{(i)}$ (left) exhibit differences in terms of anatomical content (red arrows) while samples with higher $S_{\text{DML}}^{(i)}$ (right) mainly differ in terms of noise amplitude and realization.

FIGURE 7.

Invariances with increasing $S_{\text{DML}}$ (left to right), that is, decreasing amount of content‐related invariances as measured in the DML embedding space, for each of the four denoising methods. $C=-600\mathrm{HU}$, $W=1500\mathrm{HU}$.

Lastly, we compare the invariances of different denoising networks quantitatively using the pixel‐wise mean absolute difference (MD) between $x^{(i)}$ and ${\stackrel{\sim }{x}}^{(i,k)}$ as well as $S_{\text{VAE}}^{(i)}$ and $S_{\text{DML}}^{(i)}$ (Table 2). Note that, opposed to $S_{\text{VAE}}$ and $S_{\text{DML}}$, the MD acts in the pixel space and is therefore both a measure of content‐related and noise‐related invariances. We find that quantitatively, the invariances of the denoising networks are very similar, with RED‐CNN showing the highest amount of invariances (higher MD, lower $S_{\text{VAE}}$ and $S_{\text{DML}}$). Upon a statistical analysis using a one‐sided Mann–Whitney U test with Benjamini–Hochberg correction for multiple comparisons, we examine that this finding is significant for most invariance metrics and denoising methods (In Table 2, stars for some method indicate significance levels of the pairwise test that RED‐CNN has more invariances compared to this method).

TABLE 2.

Quantitative evaluation of invariances using the mean absolute difference (MD), mean cosine similarity in the VAE latent space ($S_{\text{VAE}}$), and mean cosine similarity in the learned embedding space ($S_{\text{DML}}$).

Invariances	MD $\uparrow$ noise + content	$S_{\text{VAE}}\downarrow$ content	$S_{\text{DML}}\downarrow$ content
CNN‐10	$182\pm {67}^{\ast \ast \ast }$	$0.978\pm 0.{020}^{\ast \ast }$	$0.997\pm 0.{004}^{\ast \ast \ast }$
RED‐CNN	$\mathbf{191}\pm \mathbf{67}$	$\mathbf{0}.\mathbf{976}\pm \mathbf{0}.\mathbf{022}$	$\mathbf{0}.\mathbf{996}\pm \mathbf{0}.\mathbf{007}$
WGAN‐VGG	$158\pm {64}^{\ast \ast \ast }$	$0.979\pm 0.{021}^{\ast \ast \ast }$	$\mathbf{0}.\mathbf{996}\pm \mathbf{0}.\mathbf{006}$
DU‐GAN	$178\pm {70}^{\ast \ast \ast }$	$0.979\pm 0.013$	$0.997\pm 0.004$

Note: For MD, higher values imply more noise and content‐related invariances (due to MD being pixel‐wise), for $S_{\mathrm{VAE}}$ and $S_{\mathrm{DML}}$ lower ($\downarrow$) values imply more anatomical invariances (since they measure similarity of anatomical content between sampled invariances and input images). Bold values indicate the denoising method with the highest amount of invariances ($\uparrow$ MD, $\downarrow S_{\mathrm{VAE}}$, $\downarrow S_{\mathrm{DML}}$). We indicate statistical significance of this finding with , , and .

4. DISCUSSION

In this work, we presented a method for reconstructing the invariances of deep learning‐based low‐dose CT image denoising algorithms. Upon reconstructing the invariances of four common denoising networks we found that the sampled invariances mainly differ in terms of noise amplitude and realization, while the anatomical content is largely preserved. This is expected and can be explained by the training procedure of these networks. To answer our initial question of whether LDCT denoising networks are invariant to anatomical structures or other image content, we further proposed two methods to analyze the sampled invariances. Both methods are based on measuring distances between sampled invariances and input images in a lower‐dimensional latent space. Using these methods, we found that all denoising networks are also invariant to anatomical structures to some extent. Quantitatively, the amount of invariances (both noise‐related and content‐related) of different denoising networks are very similar, with RED‐CNN showing the highest amount of invariances both in terms of noise and anatomical structures. In Supplementary Material B we provide additional results for an algorithm that has, by design, more invariances to anatomical structures.

Our method is similar to uncertainty quantification methods such as Monte‐Carlo dropout or moment propagation ⁵¹ , ⁵² in that it can improve the interpretability of deep learning‐based methods for medical imaging. However, both approaches provide orthogonal views of the network's behavior. While uncertainty quantification methods provide a measure of the network's confidence in its predictions, our method provides a measure of the network's invariances to the input features. There are many scenarios in which an algorithm can be confident in its prediction but still exhibit invariances to certain input features (e.g., the algorithm analyzed in Supplementary Material B; Case study: Algorithm with strong invariances by design). In such cases, our method can provide additional insights into the network's behavior. Lastly, our proposed approaches for analyzing the sampled invariances could also be helpful in analyzing systematic uncertainties quantified using the aforementioned methods, an interesting direction for future work.

5. CONCLUSIONS

Our work shows that common LDCT image denoising networks are invariant to certain input features. While these invariances are mostly dominated by noise, all networks investigated in this study are also invariant to anatomical structures to some extent. We believe that developing methods to reconstruct and analyze these invariances is an important step toward interpreting deep learning‐based methods for medical image formation.

Since the presented method is architecture agnostic, several natural extensions of our work come to mind: Promising research directions include (a) evaluating the impact of training data distribution on the invariances of LDCT denoising networks; (b) investigating invariances of other networks for medical imaging including other modalities such as PET and MR; and (c) relating invariances to the similar concept of hallucinations in medical imaging. Lastly, while the sampling of invariances using our method is very fast ($\approx 23\mathrm{m}\mathrm{s}$), future work should reduce the computational complexity of training the two networks, for example by disentangling the invariances in the VAE latent space directly, thus eliminating the need for training of a cINN.

CONFLICT OF INTEREST STATEMENT

The authors declare no conflicts of interest.

Supporting information

Supporting Information

Eulig E, Jäger F, Maier J, Ommer B, Kachelrieß M. Reconstructing and analyzing the invariances of low‐dose CT image denoising networks. Med Phys. 2025;52:188–200. 10.1002/mp.17413