Synth-by-Reg (SbR): Contrastive learning for synthesis-based registration of paired images

Abstract

Nonlinear inter-modality registration is often challenging due to the lack of objective functions that are good proxies for alignment. Here we propose a synthesis-by-registration method to convert this problem into an easier intra-modality task. We introduce a registration loss for weakly supervised image translation between domains that does not require perfectly aligned training data. This loss capitalises on a registration U-Net with frozen weights, to drive a synthesis CNN towards the desired translation. We complement this loss with a structure preserving constraint based on contrastive learning, which prevents blurring and content shifts due to overfitting. We apply this method to the registration of histological sections to MRI slices, a key step in 3D histology reconstruction. Results on two public datasets show improvements over registration based on mutual information (13% reduction in landmark error) and synthesis-based algorithms such as CycleGAN (11% reduction), and are comparable to registration with label supervision. Code and data are publicly available at https://github.com/acasamitjana/SynthByReg.

1. Introduction

Image registration is a crucial step to spatially relate information from different medical images. Unpaired registration aligns images of different subjects into a common space to perform subsequent analysis (e.g., population studies [13], voxel-based morphometry [4], or multi-atlas segmentation [30,18]). On the other hand, paired registration aligns different images from the same anatomy and finds application in image guided intervention (e.g., MR-CT in the prostate [15]); patient follow-up (e.g., pre- and post-operative scans [20]); or longitudinal [29] and multimodal studies (e.g., 3D histology reconstruction with MRI [27]).

Registration is often cast as an optimisation problem where a source image is deformed towards a target image such that it maximises a similarity metric of choice. Classical registration methods solve this problem independently for every pair of images with standard iterative optimisers [32]. Modern learning approaches predict a deformation directly from a pair of images using a convolutional neural network (CNN). Supervised learning methods use ground truth deformation fields in training, either synthetic [31] or derived from manual segmentations [7]. These have been superseded by unsupervised methods, in which CNNs are trained to optimise metrics like those used in classical registration, e.g., sum of squared differences (SSD) or local normalised cross-correlation (LNCC) [6,34], without wasting capacity in regions without salient features.

Widespread similarity functions like SSD or LNCC are well suited for intra-modality registration problems. However, the difficulty of designing accurate similarity functions across modalities hampers inter-modality registration. Mutual information (MI) is often used [21] but with unsatisfactory results in the nonlinear case, due to the excessive flexibility of the model [17]. Other metrics used in inter-modality registration are the Modality-Independent Neighbourhood Descriptor (MIND, [14], based on local patch similarities) or adversarial losses measuring whether two images are well aligned or not [12]. MIND is sensitive to initial alignment, bias field or rotations depending on the neighbourhood size, while adversarial losses are prone to missing local correspondences.

An alternative to inter-modality registrationis to convert the problem into an intra-modality task using a registration-by-synthesis framework: image-to-image (I2I) translation is first used to synthesise new source images with the target contrast, and then intra-modality registration (which is more accurate) is performed in the target domain. With accurate image synthesis, the errors introduced by the translation are outweighed by the improvement in registration [17]. In unsupervised synthesis, cycle-consistent generative adversarial networks (CycleGAN) can be used [33,36], but they lack structural consistency across views and may generate artefacts due to overfitting (e.g., flip contrast or even deform images). To mitigate this issue, additional losses between the original and synthetic images have been proposed, e.g., segmentation losses [16] or inter-modality similarities between the original and synthetic scans (e.g., MIND [37] or MI [35]).

Beyond CycleGAN, other approaches have attempted to enforce geometry consistency between the original and synthetic images via specific architectures or training schemes. An I2I translation model that explicitly learns to disentangle domain-invariant (i.e., content) from domain specific features (i.e., appearance) was proposed in [28]; the latent content features can then be used to train a registration network. More recently, a novel training scheme that forces the translation and registration steps to be commutative (thus discouraging deformation at synthesis) has been presented [2]. Nonetheless, GAN-based approaches are challenging to train, with well-known problems (e.g., vanishing gradients, instability [3]) and an increasing number of losses and hyperparameters. In this work, we turn the registration-by-synthesis framework around into a synthesis-by-registration (SbR) approach, where a registration network trained on the target domain (and frozen weights) is used in the loss for training an I2I network. This allows us to greatly simplify the objective function and avoid potentially unstable adversarial training. Moreover, we use contrastive learning at the patch level to ensure geometric consistency. The SbR model outputs both the translated image and the deformation field. The contribution of this work is threefold: (i) we develop a novel registration loss for paired I2I translation; (ii) we adapt the contrastive PatchNCE loss [26] for image registration as a geometry-preserving constraint; and (iii) we combine (i) and (ii) into an unsupervised SbR framework for inter-modality registration that does not require multiple encoders / decoders and therefore has low GPU memory requirements.

2. Methods

2.1. Overview

Let us consider two misaligned 2D images of the same anatomy (e.g., a histological section and a corresponding MRI plane): the source S(x) and target T(x); x represents spatial location. We further assume the availability of an intra-modality registration CNN $\mathcal{R}$ with weights ${\mathit{\theta }}_{\mathcal{R}}$, which predicts a deformation field Φ from two images of the target modality: $\Phi =\mathcal{R}\left(T,T^{\prime };{\mathit{\theta }}_{\mathcal{R}}\right)$, such that T(x) ≈ T′(Φ(x)). We also define an I2I translation CNN $\mathcal{G}$ (with weights ${\mathit{\theta }}_{\mathcal{G}}$) from contrast S to T that regresses the image intensities: $S^T=\mathcal{G}\left(S;{\mathit{\theta }}_{\mathcal{G}}\right)$, such that S^T resembles the anatomy in S, had it been acquired with modality T. The crucial observation is that, if S^T is well synthesised, and $\Phi =\mathcal{R}\left(T,S^T;{\mathit{\theta }}_{\mathcal{R}}\right)$, then T(x) ≈ S^T (Φ(x)). Specifically, we propose the following loss (Figure 1):

$$\mathcal{L}\left({\mathit{\theta }}_{\mathcal{G}}\right)={\mathcal{L}}_{\text{reg}}\left(S,T;{\mathit{\theta }}_{\mathcal{G}},{\mathit{\theta }}_{\mathcal{R}}\right)+{\lambda }_{\text{geo}}{\mathcal{L}}_{\text{geo}}\left(S,T;{\mathit{\theta }}_{\mathcal{G}},{\mathit{\theta }}_{\mathcal{R}}\right),$$ (1)

where ${\mathcal{L}}_{\text{reg}}$ is a “registration loss” measuring the similarity of T and (the deformed) S^T, ${\mathcal{L}}_{\text{geo}}$ is a geometric consistency loss that ensures that the contents of S and S^T are aligned, and λ_geo is a relative weight. A key implicit assumption of this framework is that, because the images are paired, there exists a spatial transform Φ that aligns S and T well, such that the synthesis does not need to shift or blur boundaries to minimise the error; the geometric consistency loss further discourages such mistakes. Crucially, the loss in Eq. 1 does not depend on ${\mathit{\theta }}_{\mathcal{R}}$: the registration CNN is trained on the target domain and its weights are frozen, such that gradients will backpropagate through these layers to improve the synthesis. This asymmetric scheme enables us to avoid using a distribution matching loss (e.g., CycleGAN) that may produce hallucination artefacts [9].

Fig. 1.

Overview of proposed pipeline, using histology and MRI as source and target contrasts, respectively.

2.2. Intra-modality registration network

One of the key points in SbR is the differentiable registration method used to train the image synthesis model. We use a U-Net [8] model (as in [10]) that learns a diffeomorphic mapping between images from the same modality. The model is trained on pairs of images from the target domain and outputs a stationary velocity field (SVF), ψ, at half the input resolution. Then, a scaling and squaring approach is used to integrate ψ into a half-resolution deformation field, which is linearly upsampled to obtain the final deformation Φ(x). Training uses LNCC as image similarity term, and the norm of the gradient of the SVF as regulariser:

$${\mathcal{L}}_{\mathcal{R}}\left(T,T^{\prime };{\mathit{\theta }}_{\mathcal{R}}\right)=\frac{1}{\left|\Omega \right|}\sum _{\mathit{x}\in \Omega }\mathit{\text{LNCC}}\left[T(\mathit{x}),T^{\prime }\left(\Phi \left(\mathit{x};{\mathit{\theta }}_{\mathcal{R}}\right)\right)\right]+2{\lambda }_{\mathcal{R}}{\Vert \nabla \mathit{\psi }\left(\mathit{x};{\mathit{\theta }}_{\mathcal{R}}\right)\Vert }_2,$$ (2)

where Ω is the discrete image domain, and ${\lambda }_{\mathcal{R}}$ is a relative weight. Since the goal is to learn registration of images with approximately the same anatomy, we train the CNN with pairs of images that are similar to each other – specifically, within 3 neighbours in the image stack. In order to prevent overfitting, which may be problematic due to the relatively limited number of combinations of pairs, we use random spatial transformations for data augmentation at each iteration in the source and target images, including small random similarity transforms and smooth nonlinear deformations. Once this CNN has been trained, its weights ${\mathit{\theta }}_{\mathcal{R}}$ are frozen during training of the rest of layers in our framework.

2.3. Image-to-Image translation using a registration loss

The modality translation is performed by a generator network, $\mathcal{G}$, with a similar architecture to [26] and trained using a combination of two losses: ${\mathcal{L}}_{\text{reg}}$ and ${\mathcal{L}}_{\text{geo}}$. The first component ${\mathcal{L}}_{\text{reg}}$ is the registration loss between the target and the translated, deformed source. In section 2.2 above, we used the LNCC metric, which is known to work well in learning-based, intra-modality registration registration of most modalities, and can handle bias field in MRI [6]. However, in I2I we need to explicitly penalise absolute intensity differences, since encouraging local correlation is not enough to optimise the synthesis. For this purpose, we use the ℓ₁-norm, which has been widely used in the synthesis literature, and which is more robust than ℓ₂ against violations of the assumption that the anatomy is perfectly paired in the source and target images. The registration loss is:

$${\mathcal{L}}_{\text{reg}}\left(S,T;{\mathit{\theta }}_{\mathcal{G}},{\mathit{\theta }}_{\mathcal{R}}\right)=\frac{1}{\left|\Omega \right|}\sum _{\mathit{x}\in \Omega }{\Vert T(\mathit{x})-S^T\left(\Phi \left(\mathit{x};{\mathit{\theta }}_{\mathcal{R}}\right);{\mathit{\theta }}_{\mathcal{G}}\right)\Vert }_1.$$ (3)

The second component of the loss ${\mathcal{L}}_{\text{geo}}$ seeks to enforce geometric consistency in the synthesis and is based on noise contrastive estimation (PatchNCE [26]). The idea behind PatchNCE is to maximise a lower bound on the MI between the pre- and post-synthesis images at the patch level. For this purpose, we define a “query” image q (e.g., S^T) and a “reference” image r (e.g., S), from which we extract patch descriptors from the stack of features computed by the encoding branch of the I2I CNN, $\mathcal{G}$. These descriptors are the output of L layers of interest, including: the input image, the downsampling convolutional layers and the first and last ResNet blocks. Specifically, we extract sets of features f_l at the layers of interest l = 1, …, L and N random locations x_l,n per layer, i.e., {f_l(x_l,n)}_{l=1, …, L;n=1, …, N} (in practice, a tissue mask is used when drawing x_l,n in order not to sample the background). Each of these f_l encodes different image features (with different number of channels), from different neighbourhoods (patches), and at different resolution levels.

Given these descriptors, the contrastive loss builds on the principle that ${\mathit{f}}_l^q\left({\mathit{x}}_{l,n}\right)$ (for the query) and ${\mathit{f}}_l^r\left({\mathit{x}}_{l,n^{\prime }}\right)$ (for the reference) should be similar for n = n′ and dissimilar for n ≠ n′. Rather than using the descriptors f directly, we follow in [26] and run them through two-layer perceptrons (which are different for the descriptors in every layer l, since they have different resolutions), followed by unit-norm normalisation layers. This yields a new representation {z_l,n}_{l=1, …, L;n=1, …, N}, with:

$${\mathit{z}}_{l,n}={\mathcal{Q}}_l\left({\mathit{f}}_l\left({\mathit{x}}_{l,n}\right);{\mathit{\theta }}_z\right),$$ (4)

where θ_z groups the parameters of these representation layers. Given z, the contrastive PatchNCE loss is given by a softmax function of cosine similarities:

$${\mathcal{L}}_{\text{PatchNCE }}\left(q,r;{\mathit{\theta }}_z,\tau \right)=-\frac{1}{N}\sum _{n=1}^N\sum _{l=1}^L\text{log}\left(\frac{\text{exp}\left({\mathit{z}}_{l,n}^q\cdot {\mathit{z}}_{l,n}^r/\tau \right)}{{\sum }_{n^{\prime }=1}^N\text{exp}\left({\mathit{z}}_{l,n}^q\cdot {\mathit{z}}_{l,n^{\prime }}^r/\tau \right)}\right),$$ (5)

where τ is a temperature parameter and (·) is the dot product. It can be shown that the lower bound on the MI becomes tighter with increasing N [25].

In practice, we use two PatchNCE losses: one between the source and translated images; and another between the registered and target images:

$$\begin{array}{c}{\mathcal{L}}_{\text{geo }}\left(S,T;{\mathit{\theta }}_{\mathcal{G}},{\mathit{\theta }}_{\mathcal{R}},{\mathit{\theta }}_z,\tau \right)={\mathcal{L}}_{\text{PatchNCE }}\left(S^T\left(\mathit{x};{\mathit{\theta }}_{\mathcal{G}}\right),S(\mathit{x});{\theta }_z,\tau \right)\\ +{\mathcal{L}}_{\text{PatchNCE }}\left(S^T\left(\Phi \left(\mathit{x};{\mathit{\theta }}_{\mathcal{R}}\right);{\mathit{\theta }}_{\mathcal{G}}\right),T(\mathit{x});{\theta }_z,\tau \right).\end{array}$$ (6)

Combining the registration and geometric consistency losses in Equations 3 and 7 yields the final loss for our meta-architecture:

$$\begin{array}{c}\mathcal{L}\left({\mathit{\theta }}_{\mathcal{G}},{\mathit{\theta }}_z\right)=\frac{1}{\left|\Omega \right|}\sum _{\mathit{x}\in \Omega }{\Vert T(\mathit{x})-S^T\left(\Phi \left(\mathit{x};{\mathit{\theta }}_{\mathcal{R}}\right);{\mathit{\theta }}_{\mathcal{G}}\right)\Vert }_1\\ +{\lambda }_{\text{geo}}{\mathcal{L}}_{\text{PatchNCE}}\left(S^T\left(\mathit{x};{\mathit{\theta }}_{\mathcal{G}}\right),S(\mathit{x});{\theta }_z,\tau \right)\\ +{\lambda }_{\text{geo}}{\mathcal{L}}_{\text{PatchNCE}}\left(S^T\left(\Phi \left(\mathit{x};{\mathit{\theta }}_{\mathcal{R}}\right);{\mathit{\theta }}_{\mathcal{G}}\right),T(\mathit{x});{\theta }_z,\tau \right),\end{array}$$ (7)

which we optimise with respect to ${\mathit{\theta }}_{\mathcal{G}}$ and θ_z – since τ is a fixed hyperparameter and ${\mathit{\theta }}_{\mathcal{R}}$ is frozen, as explained above.

3. Experiments and results

3.1. Data

We validate the presented methodology in the context of 3D histology reconstruction via registration to a reference MRI volume. We use two publicly available datasets with histological sections and an ex vivo 3D MRI of the same subject. A 3D similarity transform between the stack of histological sections and the MRI volume was used to align images from both domains [24]. The MRI volume was then resampled into the space of histological stack, which yields a set of paired images to register: histological sections and corresponding MRI resampled planes. The two datasets are:

• Allen Human Brain Atlas [11]: this dataset includes 93 sections with manual delineations of hundreds of brain structures, which we grouped into four coarse tissue classes: cerebral white matter (WM), cerebral grey matter (GM), cerebellar white matter (WMc), and cerebellar grey matter (GMc). An ex vivo MRI is available, which was segmented into the same four tissue classes with SPM [5]. In addition, J.E.I. manually annotated 13.8 ± 4.4 pairs of matching landmarks in the histological sections and corresponding resampled MRI planes, uniformly distributed across all spatial locations.

• BigBrain Initiative [1]: we considered one every 20 sections, i.e., one section every 0.4 mm (344 sections in total). As in the previous dataset, an ex vivo MRI is available and J.E.I. manually annotated 11.6 ± 1.7 landmark pairs in the histological sections and corresponding MRI planes. No segmentations are available for this dataset.

3.2. Experimental setup

In our experiments, we register each histological section to the corresponding (resampled) MRI slice. For quantitative evaluation, we report the average root-mean-squared landmark error (both datasets) and the Dice score on brain tissue classes (only for the Allen dataset).

Our proposed method, SbR, was trained with the following hyper-parameters: λ_geo = 0.02, τ = 0.05 and ${\lambda }_{\mathcal{R}}=1$, which were set from a subset of the Allen dataset and used elsewhere. We also tested three other configurations of our method: an ablated version without the structure preserving constraint, i.e., λ_geo = 0 (SbR-N); fine-tuning the result of SbR by unfreezing the registration parameters (SbR-R); and an extension (SbR-G) that includes an LSGAN loss [22] with a PatchGAN discriminator [19] to discriminate between synthesised and target images (S^T and T). SbR-G enables us to assess the potential benefits of adding a distribution matching loss in training.

In addition, we compare our method against a number of other methods, to test differences against: standard registration metrics, other synthesis-based approaches without specific geometric constraints, and supervision with labels and Dice scores. Specifically, the competing methods are: (i) Linear, the initial affine registration with NiftyReg [24]; (ii) NMI, unsupervised training using normalised mutual information (NMI) with 20 bins on the image intensities; (iii) NMIw, weakly supervised training using NMI and an additional Dice loss [23] on the segmentations; (iv) cGAN, a CycleGAN [38] approach combined with our registration loss; and (v) RoT, the state-of-the-art method presented in [2] that consists of alternating the registration and translation steps. All learning-based methods above (including ours) use the same architecture for registration, and also the same nonlinear spatial augmentation scheme (sampling 9 × 9 × 2 from zero-mean Gaussians and upsampling to full resolution)

3.3. Results

The quantitative results are summarised in Figure 2. The landmark errors show that our proposed method (SbR) outperforms all baseline approaches: 11%, 9 % and 7% error reduction with respect cGAN, RoT and NMIw in the Allen dataset and 23% and 33 % with respect cGAN and RoT in the BigBrain dataset; all improvements are statistically significant (p < 0.001) using a Wilcoxon signed-rank test. Interestingly, SbR is able to align tissue masks as well as NMIw, even though segmentations were not used in the training phase. The naive approach (SbR-N) suffers from synthetic artefacts in the generator, which degrades the results - thus highlighting the importance of including structure preserving constraints in the model. The other two extensions of the model, SbR-G and SbR-R, achieve similar performance to the initial configuration, without yielding any statistically significant additional benefits.

Fig. 2.

Landmark mean squared error on the Allen human brain atlas dataset (a) and the BigBrain dataset (b). Dice score coefficient for the Allen dataset is shown in (c).

In Figure 3, we show an example of the synthesised and registered images using SbR for each dataset. The method displays robustness against common artefacts, such as: cracks, missing tissue and inhomogeneous staining (in histology), or intensity inhomogeneity (in MRI). Our method is able to accurately register convoluted structures such as the cortex, as seen in Figure 4

Fig. 3.

Image examples from (a) the Allen human brain atlas, and (b) the BigBrain project, with the deformed and rectangular grid overlaid on the source and target spaces, respectively.

Fig. 4.

Section 170 from BigBrain, with cortical boundaries manually traced on the target domain (MRI) and overlaid on the histology, before and after registration.

4. Discussion and conclusion

We have presented Synth-by-Reg, a synthesis-by-registration framework for inter-modality registration, which we have validated on a histology-to-MRI registration task. The method uses a single I2I translation network trained with a robust registration loss (based on the ℓ₁-norm) and a geometric consistency term (based on contrastive learning). In histology-MRI registration, Synth-by-Reg enables us to avoid using a CycleGAN approach, which often falters in presence of histological artefacts – since it needs to learn to simulate them and subsequently recover from them. Future work will focus on adapting our method to the unpaired scenario, as well as to other imaging modalities. We believe that synthesis-by-registration can be a very useful alternative in difficult inter-modality registration problems when weakly paired data are available, e.g., MRI and histology.