ANHIR: Automatic Non-rigid Histological Image Registration Challenge

Abstract

Automatic Non-rigid Histological Image Registration (ANHIR) challenge was organized to compare the performance of image registration algorithms on several kinds of microscopy histology images in a fair and independent manner. We have assembled 8 datasets, containing 355 images with 18 different stains, resulting in 481 image pairs to be registered. Registration accuracy was evaluated using manually placed landmarks. In total, 256 teams registered for the challenge, 10 submitted the results, and 6 participated in the workshop. Here, we present the results of 7 well-performing methods from the challenge together with 6 well-known existing methods. The best methods used coarse but robust initial alignment, followed by non-rigid registration, used multiresolution, and were carefully tuned for the data at hand. They outperformed off-the-shelf methods, mostly by being more robust. The best methods could successfully register over 98% of all landmarks and their mean landmark registration accuracy (TRE) was 0.44% of the image diagonal. The challenge remains open to submissions and all images are available for download.

I. Introduction

IMAGE registration is one of the key tasks in biomedical image analysis, aiming to find a geometrical transformation between two or more images, so that corresponding objects appear at the same position in both/all images. This research area has received a lot of attention over the last several decades and literally hundreds of different registration algorithms exist [1]-[8]. An unbiased comparison of the registration algorithms is essential to help users choose the most suitable method for a given task.

A. Goals

The goal of this study is to report the results of the Automatic Non-rigid Histological Image Registration challenge (ANHIR)¹, at the IEEE International Symposium on Biomedical Imaging (ISBI) 2019. To the best of our knowledge, ANHIR was the first open comparison of image registration algorithms on microscopy images. Moreover, we describe the associated annotated dataset of histology images, which is freely available from the challenge website, and is one of very few microscopy image datasets suitable for testing registration algorithms.

B. Previous work on image registration evaluation

One of the earliest image registration comparisons was the Retrospective Image Registration Evaluation (RIRE) project [9], comparing rigid registration algorithms on computed tomography (CT), magnetic resonance (MRI), and positron emission tomography (PET) human brain data using fiducial landmarks. Later studies added more methods [10], non-linear transformations [11]-[13], and other organs, such as lung [14], or liver and prostate [15], as well as comparison on microscopy data [16]. Registration performance is usually measured using landmark distances [9], [13], [15]-[17] or segmentation overlap [11], [12], [18].

In some of the above mentioned studies, the comparison was done by the authors themselves by means of publicly available implementations [13], which is a good strategy to find the best out-of-the-box performing approach. In other cases [17], authors (re)implemented the methods themselves, eliminating the differences in the quality of the implementation.

C. Challenges

It can be argued that the authors of a method are the best equipped and motivated to implement, tune, and run their own methods. In a challenge, the organizers prepare the data and invite participants to apply their methods to provided data and submit the results, which are then evaluated by given criteria. It is widely believed that challenges have a very beneficial effect on the progress in a particular field by: (i) lowering the entry barrier by providing (usually large) annotated datasets, (ii) motivating participants to improve the performance of their methods, (iii) identifying the state-of-the-art approaches, and (iv) tracking the overall progress of the field.

In some of the previous image registration comparisons, only known authors of image registration methods were invited to participate. Other challenges were opened to all participants, including the following:

• The RIRE project [9] used masked fiducial points on MRI, CT and PET images. It is now fully open.

• The EMPIRE10 challenge [18] on pulmonary images.

• The recent CurlOUS challenge² on the registration between presurgical MRI and intra-operative ultrasound.

• The MOCO challenge asking for motion correction of myocardial perfusion MRI [19].

• The Continous Registration Challenge³, which uses eight different datasets including lung 4D CT and MRI brain images differs in requiring participants to integrate their code to a common platform.

As far as we know, ANHIR, described here, is the only registration challenge using microscopy images.

D. Histology image registration

Microscopy images in histology need to be registered for a number of reasons [20], most frequently to create a 3D reconstruction from scanned 2D thin slices [21]-[29]. Other applications include distortion compensation [30], creating a high resolution mosaic from small 2D tiles [31], fusing information from differently stained slices [32]-[35] or even from different modalities, such as histology and MRI [36]. Registration can assist in segmenting unknown stains using known stains [37], or to combine gene expression maps from multiple specimens [38].

Histology image registration is challenging because of the following aspects: (i) large sizes of some images, e.g. 80k × 60k pixels for the whole slide images (WSI), (ii) repetitive texture, making it hard to find globally unique landmarks, (iii) sometimes large non-linear elastic deformation, occlusions and missing sections due to sample preparation, (iv) differences in appearance due to different staining or acquisition methods, and (v) differences in local structure between slices. Registration methods developed for other modalities might therefore not work optimally and need to be modified for microscopy images. Changes in appearance can be handled by color normalization [39] or deconvolution [40], focusing on the parts or stains common to both images [33], segmentation [41], [42] or probabilistic segmentation [43], or using multimodal similarity features such as structural probability maps [29]. While most algorithms are intensity-based, some use keypoint features [44] or edge features [45], and some combine the two approaches [22]. The very large image size is often handled simply by working on downscaled images but better precision can be obtained by dividing the images into smaller parts (tiles), registering them separately, and combining the results [31], [46]. The availability of a large number of consecutive slices can be used to improve robustness by enforcing consistency between different pairs of slices [22].

There is a relative paucity of microscopy image datasets suitable for image registration evaluation. The exception are datasets for testing cell tracking and cell nuclei registration in fluorescence microscopy [47], [48], which differs significantly from histological images considered here. Evaluation has been performed on other datasets, which are however not publicly available, such as a comparison of histology and block-face optical images [49], and a 3D alignment of ssTEM slices [50].

This work can be regarded as a continuation of an earlier image registration evaluation [16], [51], which is significantly expanded here in terms of the number of methods, number of images and image types, image size and resolution, and the quality of annotations.

II. ANHIR Challenge

The ANHIR challenge focused on comparing the accuracy, robustness, and speed of fully automatic non-linear registration methods on microscopy histology images. The images to be registered are large and differ in appearance due to different staining.

A. Data

In total, we have assembled 8 datasets, please see Table I for an overview and Fig. 1 for examples. More details are available in Sections⁴ S.I and S.IV. The appearance differences due to staining are sometimes important (see Fig. S2). In each case, high-resolution (up to 40× magnification) whole slide images were acquired. The images are organized in sets such that any two images within a set could be meaningfully registered, as they come from spatially close slices. Different stains are used for each image in a set and the local structure often differs, see Fig. S3. In total, we obtained 49 sets with 3 to 9 images per set, the average size was 5 slides per set. There are 355 images in total with 18 different stains⁵. We generated 481 image pairs for registration (each pair within each set but dropping symmetric pairs) and split them into 230 training and 251 testing pairs. Note that the lung lesions and lung lobes datasets were used in [16] and were therefore already completely public. We have therefore decided to treat these datasets as training only and not use them for the evaluation, except when explicitly mentioned.

TABLE I.

Summary of the datasets and their properties. See the supplementary material for a more detailed description. (COAD=Colon Adenocarcinoma).

Name	Scanner	Mag.	μm/ pixel	Avg. size [pixels]	# sets	#train pairs	#test pairs
Lung lesions	Zeiss	40×	0.174	18k × 15k	3	30	0
Lung lobes	Zeiss	10×	1.274	11k × 6k	4	40	0
Mammary glands	Zeiss	10×	2.294	12k × 4k	2	38	0
Mouse kidney	Hamam.	20×	0.227	37k × 30k	1	15	18
COAD	3DHistech	10×	0.468	60k × 50k	20	84	153
Gastric	Leica	40×	0.2528	60k × 75k	9	13	40
Human breast	Leica	40×	0.253	65k × 60k	5	5	20
Human kidney	Leica	40×	0.253	18k × 55k	5	5	20

Fig. 1.

Example dataset images: (a) lung lesions, (b) COAD (colon adenocarcinoma), (c) mammary glands, (d) lung lobes, (e) mouse kidney, (f) gastric mucosa and adenocarcinoma, (g) human breast, (h) human kidney. Some images were cropped for visualization. Multiple stains exist for each tissue type.

Since the original images are very large, participants were suggested to register provided medium size images, chosen from a fixed set of scales so that their size is approximately 10k × 10k pixels. Most participants have themselves reduced the image size even further as part of the preprocessing. For convenience, we also provided small size images at 5% of the original size, around 2k × 2k pixels.

B. Landmarks

We have annotated significant structures in the images with landmarks spread over the tissue (see Figure 2 for an example). Each landmark position is known in all images from the same set. We have placed on the average 86 landmarks per image. There were 9 annotators. Annotating a set of images took around 2 hours on average, with 20 ~ 30% additional time for proofreading. In total, annotation and validation of the 355 images took about 250 hours. All images were annotated by at least two different people. The average distance between landmarks of the two annotators was 0.05% of the image diagonal (to be compared with the relative Target Registration Error, Section II-C). This corresponds to several pixels at the original resolution. Landmarks for some of the images were withheld by the organizers. Image pairs with landmarks for both images available to participants were used for training, while image pairs with landmarks for exactly one image known to the participants formed the set of testing pairs $\mathcal{T}$. See Section S.II in the Supplementary material for details.

Fig. 2.

An example of a set of lung lesion tissue images using Cc10, CD31, H&E, Ki67 and proSPC stains (from left to right) with landmarks shown as green dots.

C. Evaluation measures

For each pair of images (i, j), we have determined the coordinates of corresponding landmarks ${\mathrm{x}}_l^i$, ${\mathrm{x}}_l^j$, where l ∈ Lⁱ, and Lⁱ is a set of landmarks, guaranteed to occur in both i and j. The participants were asked to submit the coordinates ${\widehat{\mathrm{x}}}_l^j$ of landmark points in the coordinate system of image j corresponding to provided coordinates ${\mathrm{x}}_l^i$ in the image i. We then define the relative Target Registration Error (rTRE) as the Euclidean distance between the submitted coordinates ${\widehat{\mathrm{x}}}_l^j$ and the manually determined (ground truth) coordinates ${\mathrm{x}}_l^j$ (withheld from participants)

$${\text{rTRE}}_l^{ij}=\frac{{\Vert {\widehat{\mathrm{x}}}_l^j-{\mathrm{x}}_l^j\Vert }_2}{d_j}$$ (1)

normalized by the length of the image diagonal d_j. Figure 3 shows an example of successful and unsuccessful registrations.

Fig. 3.

Examples of (a) successful and (b) unsuccessful registration. We are showing the warped image overlaid over the reference image, with the original positions and true displacements in green, estimated positions and estimated displacements in blue, target positions in magenta, and registration errors (TRE) in red.

Our main criterion to evaluate the registration algorithms is the average rank of median rTRE (ARMrTRE)

$$\text{ARMrTRE}(m)=\underset{(i,j)\in \mathcal{T}}{\text{mean}}\underset{m\in \mathcal{M}}{\text{rank}}{\mu }^{i,j}(m)$$ (2)

$$\text{where}{\mu }^{i,j}(m)=\underset{l\in L^i}{\text{median}}{\text{rTRE}}_l^{i,j}(m),$$ (3)

$\mathcal{T}$ is the set of all test image pairs and $\mathcal{M}$ is the set of all methods (a method corresponds to a single submission). In other words, we evaluate the rTRE (1) for all landmarks in an image and calculate the median of all rTRE values in an image. Aggregation by a median was chosen to eliminate outliers. We rank all methods $m\in \mathcal{M}$ according to this median. Ranking was chosen to compensate for differences in difficulty between datasets and images, which may influence the scale of the rTRE. Finally, we average the ranks over all test image pairs $\mathcal{T}$.

We also calculate other criteria aggregating the rTRE, namely average median rTRE (AMrTRE) and median of median rTRE (MMrTRE)

$$\text{AMrTRE}(m)=\underset{(i,j)\in \mathcal{T}}{\text{mean}}{\mu }^{i,j}(m)$$ (4)

$$\text{MMrTRE}(m)=\underset{(i,j)\in \mathcal{T}}{\text{median}}{\mu }^{i,j}(m)$$ (5)

and similarly the average maximum rTRE (AMxrTRE)

$$\text{AMxrTRE}(m)=\underset{(i,j)\in \mathcal{T}}{\text{mean}}\underset{l\in L^i}{\max }{\text{rTRE}}_l^{i,j}(m)$$ (6)

and average rank maximum rTRE (ARMxrTRE)

$$\text{ARMxrTRE}(m)=\underset{(i,j)\in \mathcal{T}}{\text{mean}}\underset{m\in \mathcal{M}}{\text{rank}}\underset{l\in L^i}{\max }{\text{rTRE}}_l^{i,j}(m)$$ (7)

To evaluate robustness, we compare individual rTREs for each landmark with the relative Initial Registration Error (rIRE) before registration

$${\text{rIRE}}_l^{ij}=\frac{{\Vert {\mathrm{x}}_l^i-{\mathrm{x}}_l^j\Vert }_2}{d_j}$$ (8)

This is meaningful for our images, since the images i and j share the same or similar system of coordinates, i.e. ${\mathrm{x}}_l^i$ is a reasonable initial approximation for ${\mathrm{x}}_l^j$. Let us call K^i,j ⊆ Lⁱ the set of successfully registered landmarks, i.e. those for which the registration error decreases, rTRE_l < rIRE_l. We define the robustness as the relative number of successfully registered landmarks,

$$R^{i,j}(m)=\frac{\mid K^{i,j}\mid }{\mid L^i\mid }$$ (9)

and then, the average robustness over all test image pairs is

$$\overline{R}(m)=\underset{(i,j)\in \mathcal{T}}{\text{mean}}{R}^{i,j}(m)$$ (10)

Finally, we asked the participants to report the registration times t^ij in minutes for each registration, including loading input images and writing the output files. We report the average registration time

$$\overline{t}(m)=\underset{(i,j)\in \mathcal{T}}{\text{mean}}{t}^{ij}(m)$$ (11)

The average registration time should only be considered as indicative, because it has not been independently validated. The times are approximately normalized with respect to participant computer performance by running an identical calibration program provided by the organizers.

D. Pairwise method comparison

For each pair of methods m and m′ from $\mathcal{M}$, we determine if a method m is significantly better than method m′. More specifically, we compare the median rTRE μ (3) on all test image pairs $(i,j)\in \mathcal{T}$. Our null hypothesis is that μ^i,j (m) ≥ μ^i,j(m′). Clearly, the two μ values are dependent and not necessarily normally distributed, while the test image pairs are considered i.i.d. Hence, we employ the one-sided Wilcoxon signed-rank test and evaluate the p-value under the normal assumptions, since the number of samples $\mid \mathcal{T}\mid =251$ is sufficiently large. For p < 0.01, we reject the null hypothesis and conclude that the method m indeed performs significantly better than method m′. The alternative would be permutation testing, as in [52], which is however much more computationally demanding.

E. Organization

The challenge was accepted to ISBI 2019 in October 2018 and officially announced in December 2018, after a website was set up and an evaluation system prepared. The lung lesions, lung lobes, and mammary glands data [16] with 9 training sets (108 image pairs with all landmark coordinates available) was immediately made accessible to enable the participants to start developing. In February 2019, we released the remaining images, training landmarks (122 image pairs) and a subset of the test landmarks (251 image pairs), and opened the web-based submission and evaluation system. The remaining test landmarks were released in the middle of March. The submission system was closed on March 31, 2019. Participants performed the registration on their own computers.

The challenge was widely advertised and totally open, anybody was free to participate. In total, 256 teams registered for the challenge and 10 submitted the results. For comparison, we also used two “internal submissions” corresponding to the bUnwarpJ and NiftyReg methods, see Section III. The teams were also asked to submit a description of their method. Seven teams complied and were invited to the workshop on April 11, during the ISBI 2019 conference, and to participate in writing this article. Winners were announced at this workshop. One of the teams (TUB) was unable to travel to the workshop, so only 6 teams presented there. The results announced at the workshop are given in Table III (column ‘rank’); the results shown here may be based on submissions updated later. Note also that the teams ranked 7, 9 and 11 chose not to provide a description of their methods.

TABLE III.

Quantitative result of all methods on the evaluation data. The first and second rows (‘average’, ‘median’ etc.) correspond to the aggregation method within each image pair and over all image pairs, respectively. The table is sorted by ARMrTRE (in bold). Rank corresponds to workshop results, missing numbers = insufficient information.

method	rank	Average rTRE		Median rTRE		Max rTRE		Robustness		Median rTRE	Max rTRE	Average time [min]
		Average	Median	Average	Median	Average	Median	Average	Median	Average Rank
		(AArTRE)		(AMrTRE)		(AMxrTRE)		R		(ARMrTRE)	(ARMxrTRE)
initial		0.1340	0.0684	0.1354	0.0665	0.2338	0.1157	-	-	-		-
MEVIS	1	0.0044	0.0027	0.0029	0.0018	0.0251	0.0188	0.9880	1.0000	2.84	5.04	0.17
AGH	3	0.0053	0.0032	0.0036	0.0019	0.0283	0.0225	0.9821	1.0000	3.42	6.00	6.55
UPENN	2	0.0042	0.0029	0.0029	0.0019	0.0239	0.0190	0.9898	1.0000	3.47	4.29	1.60
CKVST	4	0.0043	0.0032	0.0027	0.0023	0.0239	0.0189	0.9883	1.0000	4.41	5.27	7.80
TUB	5	0.0089	0.0029	0.0078	0.0021	0.0280	0.0178	0.9845	1.0000	4.53	3.81	0.02
TUNI	6	0.0064	0.0031	0.0048	0.0021	0.0287	0.0204	0.9823	1.0000	5.32	5.80	9.73
DROP*		0.0861	0.0042	0.0867	0.0028	0.1644	0.0273	0.8825	0.9892	7.06	7.43	3.99
ANTs*		0.0991	0.0072	0.0992	0.0058	0.1861	0.0351	0.7889	0.9714	9.23	7.79	48.24
RVSS*		0.0472	0.0063	0.0448	0.0046	0.1048	0.0275	0.8155	0.9928	9.65	8.42	5.25
bUnwarpJ*		0.1097	0.0290	0.1105	0.0260	0.1995	0.0727	0.7899	0.9310	9.67	9.37	10.57
Elastix*		0.0964	0.0074	0.0956	0.0054	0.1857	0.0353	0.8477	0.9722	10.04	8.88	3.50
UA	10	0.0536	0.0100	0.0506	0.0082	0.1124	0.0353	0.8209	0.9853	10.28	8.83	1.70
NiftyReg*		0.1120	0.0372	0.1136	0.0355	0.2010	0.0714	0.7427	0.8519	11.08	10.08	0.14

^{* =}methods added by the organizers.

The challenge remains open to submissions and all images, as well as landmarks for the training image pairs, are available for download. The testing landmarks are kept confidential.

III. Registration methods

Let us briefly describe the evaluated registration methods. See also Table II for an overview. The participants were instructed to use the same settings for all test image pairs. Note, that the method names used here may differ from the team names found on the ANHIR webpage.

TABLE II.

Evaluated methods. NGF — normalized gradient field, NCC — normalized correlation coefficient, MI — mutual information, SSD — sum of squared differences, LM — Levenberg-Marquardt, L-BFGS — Limited memory Broyden–Fletcher–Goldfarb–Shanno, ASGD — Adaptive Stochastic Gradient Descent, CNN — convolutional neural network. Stars marks well-known methods added by the organizers.

Method	Freely available	Grayscale	Exhaustive search	Affine/rigid prealignment	Feature points	Non-linear transformation	Criterion	optimization/other
UA [54]		✓		✓		dense moving mesh	MI	gradient
TUNI [56]		✓		✓	✓	virtual springs	NCC	robust linear
CKVST		✓		✓		B-splines	NCC	L-BFGS
MEVIS [58]		✓	✓	✓		dense	NGF	L-BFGS
AGH	✓	✓	✓	✓	✓	various	various	various
UPENN [61]	✓	✓	✓	✓		dense	NCC	L-BFGS, gradient
TUB [53]		✓		✓		dense	NCC	CNN
bUnwarpJ* [22]	✓	✓				B-splines	SSD	LM
RVSS* [22]	✓	✓		✓	✓	B-splines	SSD	LM
NiftyReg* [69]	✓	✓		✓		B-splines	NCC, MI	conjugated-gradients
Elastix* [70]	✓	✓		✓		B-splines	NCC	ASGD
ANTs* [71]	✓	✓		✓		B-splines	MI, NCC	L-BFGS
DROP* [73]	✓	✓				B-splines	SSD	discrete optimization

UA: (U. Alberta)⁶ The images are rescaled to 1/40 of their size and converted to grayscale. A translation and rotation is estimated first, followed by a non-rigid registration using a moving mesh framework [54], which alternates gradient descent on densely represented deformation and correction steps imposing incompressibility. Both steps use a mutual-information similarity criterion.

TUNI: (U. Tampere)⁷ The images are converted to grayscale and histogram equalized by Contrast Limited Adaptive Histogram Equalization (CLAHE) [55]. The registration is done by Elastic Stack Alignment (ESA) [56], which first finds a rigid transformation using Random Sample Consensus (RANSAC) from Scale-Invariant Feature Transform (SIFT) features. A non-linear registration follows using normalized correlation and virtual springs to keep the transformation close to a rigid one. Good parameters for the ESA algorithm were found by a parallel grid search on a computational cluster based on [57].

CKVST: (Chengdu)⁸ The images are decomposed into channels by a stain deconvolution procedure [40] and only the hematoxylin channel is used. A rigid transformation (including scaling) is estimated first on downsampled images. The transformation is further refined by evaluating the similarity on a set of selected high resolution patches. Finally, a non-rigid registration represented by B-splines is found locally. The normalized cross correlation is used for both rigid and non-rigid registration, with gradient descent and limited-memory Broyden-Fletcher-Goldfard-Shanno (L-BFGS) optimizers, respectively.

MEVIS: (Fraunhofer)⁹ Normalized gradient field (NGF) similarity [58]-[60] of grayscale versions of the images is used in a three-step approach. First, centers of mass are aligned and a number of different rotations are tried. Second, an affine transformation is found by a Gauss-Newton method. Third, a dense, non-rigid transformation is found using curvature regularization and L-BFGS optimization. The method is accelerated by optimizing memory access and by precalculation of reduced-size images.

AGH: (AGH UST)¹⁰ The images are converted to grayscale, downsampled and histogram equalized (for some of the procedures). The key feature of the AGH method is to apply several different approaches and then automatically select the best result based on a similarity criterion. An initial similarity or rigid transformation was determined by RANSAC from feature points detected by SIFT, ORB, or SURF. If it failed, the initial transformation was found by aligning binary tissue masks calculated from the images. Second, a non-rigid transformation was found using local affine registration, various versions of the demons algorithms, or a feature-point-based thinplate spline interpolation. For most of the cases, the most effective procedure was the MIND-based Demons algorithm.

UPENN: (U. Pennsylvania) [61]¹¹ A stain deconvolution [40] is used to remove the DAB (diaminobenzidine) stain and make the appearance more uniform. The images are rescaled to 4% of the original size, converted to grayscale, and padded with a random noise. For robustness, 5000 random initial rigid transformations are evaluated. The registration itself is performed starting with an initial affine step using L-BFGS optimization, followed by the estimation of a detailed deformation field, using a fast multi-resolution version of the dense non-linear diffeomorphic registration with the Greedy tool [62] and a local NCC criterion.

TUB: (Tsinghua U.)¹² A combined standard deviation of all color channels in a Gaussian neighborhood with σ = 1 pixel is used as a scalar feature for subsequent registration. Image size is reduced to about 512 × 512 pixels and padded to keep the aspect ratio. The registration is performed by a convolutional neural network similar to the Volume Tweening Network [63]. First a rigid and then a dense non-linear transformation is estimated. The network is first trained in an unsupervised manner, maximizing an image correlation coefficient, then finetuned using provided landmark positions on the training data.

The seven methods described come from the ANHIR challenge as described in Section II-E. For completeness, we have also included several well known, freely available and often used registration methods, which we already tested in [16], to enable a better comparison with the state of the art. These methods (marked by * in Tables II and III) were not part of the ANHIR competitions but were run by the organizers on the same data and evaluation protocols.

bUnwarpJ: The images are registered using the bUnwarpJ ImageJ plugin [22] which calculates a non-linear B-spline-based deformation in a multiresolution way, minimizing a sum of squares difference (SSD) criterion, generalizing earlier B-spline registration methods [64], [65]. The key feature of bUnwarpJ is to simultaneously calculate the deformation in both directions, enforcing invertibility by a consistency constraint and providing strong regularization. Additional regularization is possible by penalizing the deformation field divergence and curl.

RVSS: Register virtual stack slices is another ImageJ registration plugin which extends the bUnwarpJ method [22] by incorporating also SIFT [66] feature points in both images to add robustness. We start by estimating a rigid transformation, which we then refine using a B-spline transformation.

NiftyReg: The NiftyReg software is used to first perform an affine registration based on block matching [67], [68] and then a B-spline non-linear registration [69].

Elastix: Elastix [70] is a registration software providing a simple to use command-line interface to the Insight Segmentation and Registration Toolkit (ITK). We use a B-spline deformation and MI similarity criteria with the adaptive stochastic gradient descent optimizer.

ANTs: Advanced Normalization Tools [71] is another registration toolkit using ITK as a backend. We first perform an affine registration by MI maximization, followed by a dense symmetric diffeomorphic registration (SyN) using cross-correlation.

DROP: Similarly to many previously mentioned approaches, DROP [72], [73] represents the deformation using B-splines, minimizing a sum of absolute differences (SAD) (ℓ₁) criterion. It differs by treating registration as a discrete optimization problem, solved efficiently in a multiresolution fashion using linear programming.

IV. Results

A. Relative target error comparison

Our main evaluation criterion is ARMrTRE, average rank of median relative target registration error (see Section II-C, eq. 2). We show the values of ARMrTRE for all methods in Fig. 6, together with AMrTRE (4), and MMrTRE (5). We see that MEVIS is most often the best method, with AGH and UPENN within the top four, and TUB, TUNI, and CKVST more variable but close behind. On the other hand, the general methods (NiftyReg, bUnwarpJ, Elastix, DROP and ANTs), not optimized for microscopy images, are always in the second half of the ranking. It turns out that the results are almost the same when the other criteria from Section II-C are employed. A graphical comparison of those as well as other criteria based on of aggregating the relative landmark error rTRE can be found in Fig. S1 in the Supplementary material.

Fig. 6.

Average rank of median rTRE (a), average median rTRE (b), and median of median rTRE (c) per method on the test data.

For quantitative values on test data, see Table III. Since some datasets contain only training and no testing data, for completeness we also provide quantitative results on all data combined in Supplementary material Table SIII.

B. Pairwise method comparison

Fig. 7 shows the color coded p-values for pairwise comparison of the methods, as described in Section II-D. It turns out that we could (topologically) sort the methods by performance from the best methods at the top to the worst performers at the bottom. All differences between methods are significant with the following exceptions: There are no significant differences between the first three methods (MEVIS, CKVST, AGH), between CKVST, AGH, and UPENN, between ANTs and RVSS, and between UA, ANTs and Elastix methods.

Fig. 7.

Color coded p-values for pairwise comparison of methods over all test image pairs in terms of median rTRE, where the median is taken over all landmarks in an image. Red/blue indicates, that the method in the row performs statistically better/worse than the column method, respectively, according to the one-sided Wilcoxon signed-rank test at significance level p = 0.01. Gray and shades of gray indicate no significant differences. Methods are topologically sorted by performance from top to bottom.

C. Differences between datasets

We may ask whether some datasets are more difficult than others and whether some methods are particularly suited for specific datasets. It seems that the answer is yes to both questions. For example, according to Fig. 4, the (CNN based) TUB method outperforms all other methods by an order of magnitude in terms of AMrTRE on some datasets (lung lesions, lung lobes). However, this could be also due to overfitting, since for these datasets all landmark positions were available for training since the beginning and none of them are part of the testing set. Note that evaluation was done on the testing pairs for all datasets except lung lesions and lobes, and mammary glands, which contain only training pairs.

Fig. 4.

Relative performance of the methods on the different datasets in terms of AMrTRE.

Fig. 5 compares the robustness (8) by datasets. We see that the best six methods have a very high robustness (> 98%), while others completely fail on some of the datasets (e.g. Elastix or TUNI). Fig. S4 in the Supplementary material compares the methods’ performance as a function of the combination of stains being registered.

Fig. 5.

Robustness (the relative number of landmarks which moved towards the ground truth location) of all methods on different tissues.

D. Training/testing

In Fig. S5, we compare the performance in terms of the distribution of the median rTRE and rank median rTRE between the training and testing data. For most methods, the distributions on training and testing data are rather similar. However, we note that TUB performs much better on training than on testing data, hinting again on possible overfitting. Interestingly, some methods, like MEVIS or CKVST, actually perform slightly better on the testing data.

V. Discussion and conclusions

Our experience with organizing the ANHIR challenge and a very positive reaction of the community, including a good attendance of the ANHIR workshop at ISBI 2019, confirmed that there is indeed a wide interest in such challenges.

However, organizing such a challenge requires a significant effort. The most time-consuming part was annotating the images with landmarks. Although the landmarks were sufficient to our purpose, there are sometimes large areas without salient corresponding structures, where very few landmarks can be identified. In the future, we would recommend that manually obtained landmarks are combined with other complementary means of evaluating the registration quality, such as measuring overlap using reference segmentations (as in NIREP [11]), or using synthetically generated (and therefore known) deformations [64]. We became aware that for the results to be meaningful, we must discourage the participants to overtrain on the provided datasets. Future challenges should also measure the speed of the algorithms more reliably, ideally by using a common hardware for all methods. Besides pairwise registration, we might consider other more advanced tasks such as 3D reconstruction by registering the histology slices, creating a large 2D mosaic from a set of smaller tile images, aligning 2D microscopy images with volumetric medical invivo images (e.g. MRI or ultrasound), or aligning standard histopathology with high-dimensional microscopy modalities such as MALDI [74]. For these applications, completely new evaluation approaches might be needed.

Perhaps surprisingly, most of the submitted methods (see Section III) used very similar and rather classical techniques. The best performing algorithms were originally developed for other modalities but were tuned for the data at hand. They consist of a carefully designed multi-step pipeline, which typically started with a coarse but robust prealignment, followed by a non-rigid registration step for fine tuning the results. Only one method (TUB) used the now ubiquitous CNNs; it was very fast and performed well, although not as well as the best performing methods and its generalization ability was limited. Interestingly, directly applying well-known off-the-shelf methods gave significantly worse results, although these methods contained very similar algorithmic ingredients as some of the best performing methods (e.g., MEVIS, CKVST, AGH). One explanation is that those general purpose methods were developed for other types of images, mostly 3D brain data, and possibly required more parameter tuning. Our observation is that the better performing methods often won not by being more accurate but by being more robust (see Table III), converging to the solution even from a far away initialization. The participants were clearly aware of this, since most submitted methods spend a significant effort to find a good initialization, for example by incorporating automatically detected landmarks (TUNI, AGH), trying many different starting points (MEVIS, UPENN), or even applying different methods in parallel (AGH). Another reason for the performance gap might be that the participants spent more time finding the optimal parameters for their own methods (even using a computing cluster in TUNI).

To summarize the evaluation, there was very little difference in terms of both robustness and accuracy between the first six methods (MEVIS, AGH, UPENN, CKVST, TUB, TUNI). Those six methods outperformed all others in all evaluation measures that we have tested.

On the other hand, there were large differences in the reported execution speed (Table III). With the exception of the extremely fast CNN method (TUB), the differences seem to be more due to the quality of engineering (implementation) and the choice of parameters, and not the choice of the key algorithms. While MEVIS and NiftyReg take on average a few seconds per registration, the majority of the methods need a few minutes, with the slowest method (ANTs) taking almost an hour. Note however that MEVIS uses precalculation extensively and NiftyReg might have taken advantage of the GPU.

No methods were developed explicitly for the ANHIR challenge or for this type of images. No evaluated method takes advantage of the full resolution or full color information. With a median landmark localization error measured in tens of pixels at the original resolution even for the best methods, none of the methods would be probably robust and accurate enough for routine, fully automatic use. However, these methods could certainly be used in a semi-supervised setting to substantially reduce the need for manual interaction in many tasks, such as fusing information from several differently stained images.

The above mentioned weaknesses will likely be addressed in future research, e.g. by creating algorithms capable of working on full resolution images or avoiding the overfitting in CNNs. We therefore believe that the need for objective performance evaluation will remain and that challenges such as ANHIR should be continued to stimulate and evaluate research in image registration.