Deep Adversarial Training for Multi-Organ Nuclei Segmentation in Histopathology Images

Abstract

Nuclei segmentation is a fundamental task for various computational pathology applications including nuclei morphology analysis, cell type classification, and cancer grading. Deep learning has emerged as a powerful approach to segmenting nuclei but the accuracy of convolutional neural networks (CNNs) depends on the volume and the quality of labeled histopathology data for training. In particular, conventional CNN-based approaches lack structured prediction capabilities, which are required to distinguish overlapping and clumped nuclei. Here, we present an approach to nuclei segmentation that overcomes these challenges by utilizing a conditional generative adversarial network (cGAN) trained with synthetic and real data. We generate a large dataset of H&E training images with perfect nuclei segmentation labels using an unpaired GAN framework. This synthetic data along with real histopathology data from six different organs are used to train a conditional GAN with spectral normalization and gradient penalty for nuclei segmentation. This adversarial regression framework enforces higher-order spacial-consistency when compared to conventional CNN models. We demonstrate that this nuclei segmentation approach generalizes across different organs, sites, patients and disease states, and outperforms conventional approaches, especially in isolating individual and overlapping nuclei.

I. Introduction

From its origin in gross anatomy prior to cell theory, to modern computer-aided digital pathology, the field of histopathology has evolved as a crucial element in the diagnosis and treatment of nearly every disease discovered [1]–[3]. Millions of tissue biopsies are performed annually, and in nearly every case the study of nuclear morphology and distribution provides critical clues to healthcare providers [4]. Indeed, containing vast amounts of genetic and epigenetic code that govern and regulate cellular type, morphology, and function, the nucleus is of paramount importance. Decades of modern cytological and histopathologic study have led to the development of cellular stains such as hematoxylin that fluorescently highlight nuclei [5], [6]. Since their advent, and until the nascent field of digital pathology, interpretation and determination of aberrant phenotypes in these stained tissue specimens has been accomplished by subjective human interpretation [7]–[9]. As the field of pathology migrates from subjective to automated objective interpretation, there is a critical need for accurate and precise but simple and computationally-efficient methods for nuclei segmentation.

The shape and distribution of cell nuclei in pathology images is used to determine cell, tissue, and cancer types, and is critical in cancer identification, grading, and prognosis [10]. For example, in blood smears, multi-lobed nuclei typically indicate neutrophils, oval and kidney-shaped nuclei mark monocytes, round nuclei with a high nuclear-to-cytoplasmic ratio mark lymphocytes, while fully developed red blood cells lack nuclei all together [11], [12]. “Cigar-shaped” nuclei help identify smooth muscle, and, when found out of place, leiomyosarcoma [2]. Nuclear distribution, morphology, presence, and absence helps identify the layers of epithelium on the exterior and interior of the body, while the identification of cytological penetration of characteristic epithelial layers by cancer such as melanoma can drastically alter treatment course and prognosis [13]. In breast cancer, the identification of nuclear and chromosomal mitotic figures provides an accurate prognostic indicator, and in adenocarcinoma of the prostate, cancer cells commonly demonstrate nuclear enlargement and prominent nucleoli [14], [15]. The importance of nuclear identification and subsequent detailed analysis can hardly be overstated.

The field of digital pathology is revolutionizing modern healthcare. The advent of whole-slide imagers coupled with telemedicine and cloud storage has facilitated efficient storage of vast amounts of tissue data and promises to facilitate diagnosis, prognosis and treatment planning [16]. Combining tissue data with electronic medical records, genetic and epigenetic sequencing, big data science, and epidemiologic studies, can facilitate further personalized healthcare and reveal interesting scientific discoveries [17], [18]. The application of precise computer vision techniques to this data is catalyzing the field’s development, increasing the efficiency of providers through rapid screening, aiding in education, and standardizing analysis to reduce observer variability [10], [19]–[21].

Given the importance of nuclear distribution and morphology, and the rapid growth of digital pathology, the task of precise nuclear segmentation in H&E images via computer algorithms provides a logical starting point for the rest of computer-aided tissue image analysis. Accurate segmentation of nuclei is a pivotal starting point for further feature extraction and classification within the nucleus itself, but also serves as a relatively simple basis to model cellular distribution, which can be used to classify tissue subtypes and identify abnormalities.

While identification of nuclei in a conventional H&E stain is routine for trained clinicians and researchers, chromatic stain variability, nuclear overlap and occlusion, variability in optical image quality, and differences in nuclear and cytoplasmic morphology and stain density provide challenges for computer-based segmentation algorithms [22]–[24]. Many techniques have been used to accomplish the task, but have been met with limited success in the more challenging of cases. For example, intensity thresholding methods generally fail with noisy images and clumped nuclei, marker-based watershed segmentation requires precise parameter selection, and active contour and deformable models are computationally expensive [23]–[29]. Machine learning-based nuclear segmentation approaches are generally more robust to these challenges, as they can learn to identify variations in nuclear morphology and staining patterns. More specifically, convolutional neural networks (CNNs) have recently demonstrated state-of-the-art performance in nuclei segmentation [24], [30], [31]. In this approach, images are passed through a trained two-class classifier to label pixels as nuclear or non-nuclear, and additional post-processing techniques to delineate clustered nuclei are subsequently applied [24]. A third class of inter-nuclear pixels can be added to eliminate the bulk of the post-processing work, which demonstrates significant advantage in separating crowded nuclei [31]. Most current deep learning-based multi-organ nuclei segmentation methods are limited by the amount of data available. For a single network to perfectly segment nuclei from different organs large, diverse, quality-annotated training data is required. Moreover, existing methods often fail to segment overlapping nuclei without complex post processing steps.

The primary goal of this work is to accurately segment nuclei in a diverse set of H&E stained histology images. Unfortunately, a significant barrier for data-driven approaches such as deep networks is the paucity of large and diverse annotated datasets. Although, individually trained nuclei segmentation networks may perform better for a specific organ, training a separate network for every organ in the human body while catering for the diversity that exists across patients, hospitals and disease states is tedious. A singular network that is independent of organ specific nuclei morphology is more broadly applicable while training on just a few organs. While there exist datasets for organ types such as breast tissue and prostate tissue, there is a lack of support for other important organ types. Moreover, even for supported cancer types, there is also a lack of coverage of possible disease grades. Overall, the presentation of histopathology tissue and their nuclei is organ-, disease-, and patient-specific. While specialized organ and disease human experts might perform better than general pathologists, for data-driven approaches such as deep networks, there is not enough training data currently to cover all possible modes of data, which makes computational pathology a domain adaptation problem. Stated precisely, domain adaptation is the generalization problem in which a model trained on data from one domain is often incapable of performing well on data from another domain due to subtle distinctions between the two domains. In computational pathology, models trained on a specific patient, H&E staining protocol, disease grade or organ would have a domain bias towards that training set, and might not cover the full distribution of testing cases found in histopathology tissue. Thus, as with previous work, in order to generalize to unknown organ types and disease grades, nuclei segmentation should be approached is unified manner.

Contributions:

In this work, we propose a method to overcome the diversity required in training data using synthetically generated data and then use a contex-aware adversarial network for nuclei segmentation. Our main contributions are highlighted below:

• Synthetic Pathology Images with Ground Truth: Because of the limited availability of diverse labeled nuclei segmentation data we generate a large dataset of perfectly-annotated synthetic histopathology images by generating random polygon masks and adding realism using unpaired cycle-consistent adversarial training.

• Adversarial Nuclei Segmentation: We propose to train a conditional GAN (cGAN) network with spectral normalization and gradient penalty for multi-organ nuclei segmentation. Instead of using post-processing steps which reinforce spacial contiguity in the nuclei segmentation mask we use adversarial term imposes higher-order spacial-consistency during the training process. Moreover, we pose the problem as a regression rather than a classification problem where the loss function is also learned during the training process.

• Quantitative Study: We validate our proposed nuclei segmentation paradigm on publicly available and newly created datasets. Our quantitative study demonstrates cross site, patient and organ adaptability of our proposed method.

II. Related Work

A. Deep Learning-based Nuclei Segmentation

Nuclei segmentation in histopathology images has been extensively studied using a variety of deep learning methods. However, there are several challenges associated with effectively using deep learning approaches for this task. Most work has focused on developing nuclei segmentation methods for single organs and specific application without addressing issues such as domain adaptation [32]. Histopathology images are diverse due to variations such as organ type, tissue site, and staining protocol. Clumped and chromatin-sparse nuclei are especially difficult to isolate or detect. The close proximity of epithelial cell and the random occurrence of mitotic figures make accurate boundary detection difficult. Most deep learning techniques estimate a probability map based on a two-class problem followed by post-processing. Neeraj et al. completed seminal work in multi-organ nuclei segmentation by posing the challenge as a three class problem (CNN-3C) [31]. Cui et al. developed a fully convolutional network-based nuclei segmentation approach [33] and Naylor et al. posed the problem as a regression task of estimating the nuclei distance map [34]. Khoshdeli et al. demonstrate that fusion of networks is better than multi-task learning for nuclei segmentation [35]. Salvi et al. demonstrate a multi-scale adaptive nuclei segmentation (MANA) method and show results on multi-organ nuclei segmentation. The performance of all of these approaches is limited by the size and quality of labeled datasets and the diversity required in the images to model the distribution of relevant tissue features. For a single method to be adaptable to multiple organs, data sites, patients, and disease states large amounts of diverse data are required. Moreover, simple CNN-models are generally not capable of handling the issue of overlapping nuclei and post-processing or alternate parallel processing is required to determine such overlaps. This is because CNNs minimize a per-pixel loss between the input image and the segmentation mask. Most previous work has posed the nuclei segmentation problem as a classification or as a combined classification and regression problem. We propose to pose this as an adversarial regression-based image-to-image translation problem. Such an approach is more context aware and globally consistent, i.e. the loss function is learned taking the entire image into consideration rather than just pixel-wise loss.

B. GANs for Medical Imaging Applications

GANs were introduced by Goodfellow et al. in [36] and have since been used for a variety of medical imaging applications including segmentation [37]–[39], detection [40], [41], reconstruction [42]–[44], domain adaptation [45]–[48]. The GAN framework can be seen as a two player min-max game where the first player (the generator), is tasked with transforming a random input to a specific distribution, such that the second player (the discriminator) cannot distinguish between the true and synthesized distributions [36], [49]. GANs have a generative and artistic ability to map random noise to realistic distributions [50]–[52]. However, for medical imaging tasks it is critical to constrain this artistic ability and thus conditional GANs (cGANs) are more applicable. cGANs have the ability to conditionally control the output of GAN training based on a class or an image. Image-to-image translation tasks i.e. situations where the GAN is conditioned by an input image, are more useful for medical imaging than class conditioning. Due to the limited availability of labeled medical imaging data generative adversarial networks (GANs) have been recently been used for synthetic data generation. For example, Dou et al. proposed to generate synthetic retinopathy data from retinal masks [37]. Schlegl [41] proposed generating pathology data using nuclei masks where the background and foreground are learned separately. Despite the wide spread use of GANs they are notoriously difficult to train. We use recently proposed GAN stability methods such as spectral normalization [53] and gradient penalty for improving the stability of cGAN-based image-to-image translation.

III. Methods

A. Datasets

There are very few nuclei segmentation datasets with pixel-level annotations. Careful annotation of nuclei boundaries is time-consuming, error-prone and may also suffer from subjective interpretation errors. Irshad et al. [54] showed that there was larger interoberver disparity among pathologists identifying nuclei in H&E data. Several datasets have been released for nuclei segmentation but often lack annotations of all nuclei in large portions of the data [55], [56]. This makes it difficult to identify false positives and to focus on clumped and overlapping nuclei. The only publicly available dataset with complete nuclei annotation and multi-organ pathology images was presented by Neeraj et al. in [31]. This dataset contains annotations of 30 1000 × 1000 pathology images from seven different organs (bladder, colon, stomach, breast, kidney, liver, and prostate). The raw data for this data was sourced from the NIH Cancer Genome Atlas (TCGA). Although, this is a large database it is still not enough to cater for the diversity required to train context aware methods that can determine overlapping nuclei with a high accuracy on object and pixel-level statistics. To increase the number of organs and test data available we complemented this existing dataset with four additional 1000 × 1000 pathology images (breast, prostate, ovary, and esophagus) from the TCGA database labeled by a pathologist at the Johns Hopkins Hospital. Since, this additional data is labeled by a different pathologist it contributes to the overall diversity of the data. Data from five different organs from the first dataset [31] along with synthetically generated data were used for training. Synthetically generated data to models a large diversity of nuclei sizes, overlapping morphology and characteristics that can enhance overall network performance. The trained models were evaluated on a combination of images from the first and second datasets and includes evaluation on four organs that were not used for training. For testing, two slides from breast, liver, kidney, prostate, bladder, colon and stomach and one slide from esophagus and ovary were used.

B. Stain Normalization

A significant amount of color variations exists in H&E stained histopathology images due to differences in manufacturing of stains, staining protocols used and response functions of digital scanners. Differences in stains can be a major issue in cross-site domain adaptability of CNN-based computational pathology methods. This is because CNNs mostly rely on the color and texture of H&E images for learning cues. Normalizing the images can significantly improve the performance of such CNN-based methods [57]. However, normalization methods developed for conventional vision applications [58] provide limited benefit in computational pathology applications because the two stains can be normalized against each other. Several methods have been presented for normalizing pathology images or adapting networks trained on data from one site to data from other locations [57], [59]–[61]. Vahadane et al. [62] proposed decomposing the pathology image into stain density maps in an unsupervised manner and then combining these sparse stain density maps with the stain colored basis of a target image. This approach preserves the structure in the source image while adapting the color to the target domain. We empirically found that sparse stain normalized H&E images following [62] performed better with our nuclei segmentation methods as compared to standard Reinhard et al. [58] and Macenko et al. [61] stain normalization. Details about this method can be found in [62] and examples of stain normalized images have been shown in Fig. 1a.

Fig. 1.

(a) Figure showing full-slide images, 1000×1000 cropped images and sparse stain normalization for nine different organs sources from the NIH TCGA database. (b) Unpaired synthetic data generation using randomly-generated polygon masks mapped to H&E images. The architecture includes a dual-GAN setup with cycle-consistency loss. Two generators learn mappings, G, and S between a mask (M) and a histology image (N) G : M → N and S : N → M and two discriminators classify the pairs of M and N as real or fake. (c) The conditional GAN setup for segmenting nuclei. The discriminator enhances the receptive field of the generator while learning an efficient loss function for the task.

C. Learning Preliminaries

The primary objective of this work is twofold:

1. Generate synthetic H&E images that model the distribution of cellular and extracellular spatial features represented in multiple organs.

2. Use both the synthetic and real histopathology data for training a context-aware CNN that can accurately segment nuclei.

This approach can be generalized as learning mapping functions between two domains: M (nuclei mask) and N (H&E images). Specifically, G maps from M → N, and is used for synthetic data generation, while S maps from N → N, and is used for nuclei segmentation. We denote m and n as training examples where m ∈ M and n ∈ N.

D. Synthetic Data Generation

Previous work on histopathology image synthesis has focused on generating the nuclei-free background and foreground separately [63]. In contrast we propose a relatively simple unpaired mapping-based approach, where cycle-consistency loss is used in a dual-GAN architecture to transfer between polygon masks (M) and histopathology images (N). The size, location and shape of the nuclei can vary significantly based on patients, clinical condition, organs, cell-cycle phase and aberrant phenotypes. Thus, we generate nuclei masks by building a dictionary of nuclei sizes and shapes from different organs and randomly perturbing size and shape parameters before placing them on a grid in a randomized fashion. We then use a cycleGAN-style [52] architecture to add realism to the polygon mask. The cycleGAN is trained on real H&E patches, once trained it can be used to add realism to randomly perturbed nuclei placed on a grid. These generated polygon masks and corresponding synthesized H&E images are subsequently used for training. The purpose of synthetic data generation is to make the data independent of biological nuclei morphology represented in a specific organ, patient or disease state. By slightly perturbing the nuclei on a scaffold we essentially domain randomize the network which allows the network to be more general. Domain randomization is recent concept commonly used by the robotics community to generalize training on synthetic data [64]. While the generated synthetic data does not have biological significance, it can be used to train a nuclei segmentation network which is more effective than a network entirely trained on realistic data.

The cycleGAN framework learns a mapping between randomly generated polygon masks and unpaired pathology images. Since cycleGAN is based on consistency-loss, the setup also learns a reverse mapping from pathology images to corresponding segmentation or polygon masks (Fig. 1b). However, since this is an unpaired mapping, the segmentation network is not efficient enough to be used. Thus, we use the reverse mapping only to train the forward mapping more effectively. Such an arrangement consists of four networks: G (random polygon mask to pathology image generator), S (pathology image to polygon mask generator), D_N (discriminator for G), and D_M (discriminator for S). To train this framework for synthetic data generation with unpaired data, the cycleGAN objective consists of an adversarial loss term ${\mathcal{L}}_{\text{GAN}}$ and a cycle-consistency loss term ${\mathcal{L}}_{\text{cyc}}$. The adversarial loss is used to match the distribution of translated samples to that of the target distribution and can be expressed for both mapping functions.

The cycle-consistency loss is used to constrain the generator in learning an invertible mapping from generated polygon masks to pathology tissue, in that, when learning a mapping function G that translates the nuclei mask into pathology tissue, we can learn a mapping function S as that can map that pathology tissue back into a nuclei mask (creating a forward cycle, N → M → N). As a small note, that only forward cycle consistency loss is maintained. We do not preserve backward cycle-consistency. For the mapping G : M → N with discriminator D_N, we can express the objective as the binary cross entropy (BCE) loss of D_N in classifying real or fake, in which D_N and G play a min-max game to maximize and minimize this loss term respectively. This objective for G : M → N, can be expressed as,

$${\mathcal{L}}_{\text{GAN}}\left(G,D_N\right)={\mathbb{E}}_{n~p_{\text{data }}\left(n\right)}\left[\text{log}{D}_N(n)\right]+{\mathbb{E}}_{m~p_{\text{data }}\left(m\right)}\left[\text{log}\left(1-D_N(G(m))\right)\right],$$ (1)

in which the generator G aims to generate nuclei images from random polygon masks such that they would be indistinguishable from real nuclei images, i.e., G(m) ≈ n, while the discriminator D_N aims to distinguish generated vs. real nuclei images. A similar objective can be expressed for S : N → M,

$${\mathcal{L}}_{\text{GAN}}\left(S,D_M\right)={\mathbb{E}}_{m~p_{\text{data }}\left(m\right)}\left[\text{log}{D}_M(m)\right]+{\mathbb{E}}_{n~p_{\text{data }}\left(n\right)}\left[\text{log}\left(1-D_M(S(n))\right)\right].$$ (2)

The cycle-consistency loss is used to incentivize a one-to-one mapping between samples in M and N, and facilitate the evolution of G and S to inverse functions of each other. Specifically, the ${\mathcal{L}}_{\text{cyc}}$ term ensures that the forward and back translations between the random polygon mask and nuclei image are lossless and cycle consistent, i.e., S(G(m)) ≈ m (forward cycle-consistency) and G(S(N)) ≈ N (backwards cycle-consistency). The forward cycle loss term helps with refining synthetic nuclei images to be more realistic, as the generated images should not only mimic the H&E tissue, but also create a diverse morphology of the nuclei for training a robust nuclei segmentation network. We can express the objective as,

$${\mathcal{L}}_{cyc}\left(G,S\right)={\lambda }_n{\mathbb{E}}_{n~p_{\text{data }}\left(n\right)}\left[\parallel G\left(S\left(n\right)\right)-n{\parallel }_1\right]+{\lambda }_m{\mathbb{E}}_{m~p_{\text{data }}\left(m\right)}\left[\parallel S\left(G\left(m\right)\right)-m{\parallel }_1\right]$$ (3)

where λ controls the importance of the forward and backward cycle constraints. For synthetic data generation, we relaxed the λ_m term, as a random polygon mask can represent multiple valid nuclei images. This randomization can also be seen as GAN input noise that contributes to generating the diversity in synthetic image.

The full objective for synthetic data generation can thus be written as,

$$\underset{G,S}{\text{arg}\text{min }}\underset{D_N,D_M}{\text{arg}\text{max }}{\mathcal{L}}_{\text{GAN}}\left(G,D_N\right)+{\mathcal{L}}_{\text{GAN}}\left(S,D_M\right)+{\mathcal{L}}_{cyc}\left(G,S\right)$$ (4)

E. Conditional GANs for Segmentation

One of the major challenges in nuclei segmentation is independent boundary detection and isolation of overlapping nuclei. Standard CNNs segment these nuclei as one object because these approaches typically rely on minimization of some pixel-wise loss. The contribution of a single misclassified pixel is insignificant to the overall loss but can subsequently lead to multiple nuclei segmented as one. This problem has previously been mitigated by contour prediction [31], [65], [66], concave point detection [67] and distance map regression [34].

For conventional vision, conditional random fields (CRFs) have been extensively used to enforce spatial contiguity as a post-processing step in segmentation problems [68]. Joint CNN-CRF models have also been explored for more global and context aware CNN training [46], [69], [70]. Despite the advances in CNN-CRF models, this approach is often limited to the use of pairwise CRFs, which only incorporate second order statistics. According to [71], [72] higher order potentials have also been useful for image segmentation. Using CRFs to incorporate higher order statistics renders a complex energy function. Adversarial training allows higher order consistency without being limited to a specific type of higher order potential (e.g. unary and pairwise in the case of CRFs). Since the adversarial model has a field-of-view that is a large portion of the image rather than just neighboring pixels or super-pixels, it can enforce a higher-order spacial-consistency that can neither be enforced using pair-wise terms in a CRF nor measured by pixel-wise loss.

The adversarial segmentation model also learns an appropriate loss function which circumvents manually engineered loss functions. This has been explored in detail in image-to-image translation methods such as [51]. Such a model is flexible enough to detect subtle differences in a range of higher order statistics between the predicted and ground truth nuclei segmentation masks. The adversarial setup can learn a loss, based on classifying the output image as real or fake, while iteratively training a segmentation model to minimize this learned loss. Each output pixel is usually considered conditionally independent from all other pixels whereas conditional GANs can learn a structured context-aware loss considering a larger receptive field. The adversarial loss used to penalize the generator is calculated from the discriminator, in which the discriminator classifies overlapping pairs of image and segmentation prediction patches as being real or unrealistic. By controlling the size of the patch, we can control the size of the non-locality, with bigger patches incorporating more global information in the image.

The cGAN framework learns a mapping S for nuclei segmentation, in which S can adapt H&E nuclei images to their segmentation masks. To train this framework for semantic segmentation with paired data, the conditional GAN objective consists of an adversarial loss term ${\mathcal{L}}_{\text{GAN}}$ and a per-pixel loss term ${\mathcal{L}}_1$ to penalize both the joint configuration of pixels and segmentation errors.

The adversarial loss in conditional GANs is similar to that of cycleGAN, in which the segmentation network S and discriminator D_M play a min-max game in respectively minimizing and maximizing the objective, ${\text{min}}_S{\text{max}}_{D_M}{\mathcal{L}}_{\text{GAN}}\left(S,D_M\right)$. Specifically, S translates nuclei images to realistic segmentation masks to minimize the cross-entropy loss of D_M. The adversarial loss can additionally be interpreted as a structured loss, in which S is penalized if the group configuration of the pixels in the predicted mask is unrealistic, i.e. masks that look like salt-and-pepper noise. Because the data is paired, D_M sees both the nuclei image and the predicted mask. We can express the GAN objective as,

$${\mathcal{L}}_{\text{GAN}}\left(S,D_M\right)={\mathbb{E}}_{m,n~p_{\text{data }}\left(m,n\right)}\left[\text{log}{D}_M\left(m,n\right)\right]+{\mathbb{E}}_{n~p_{\text{data }}\left(n\right)}\left[\text{log}\left(1-D_M\left(m,S\left(n\right)\right)\right)\right]$$ (5)

An additional ${\mathcal{L}}_1$ loss term is used to bring the output closer to the ground truth and stabilize GAN training,

$${\mathcal{L}}_1\left(S\right)={\mathbb{E}}_{m,n~p_{\text{data }}}\left(m,n\right)\left[\parallel m-S\left(n\right){\parallel }_1\right].$$ (6)

Based on Eq. 5 and 6 the full objective for conditional GAN-based segmentation can be expressed as,

$$\text{arg }{\text{min}}_S{\text{max}}_{D_M}{\mathcal{L}}_{\text{GAN}}\left(S,D_M\right)+{\mathcal{L}}_1\left(S\right).$$ (7)

Because the discriminator works on overlapping patches (i.e., PatchGAN Markovian classifier), it penalizes structure at a patch level rather than over the entire image. This approach draws focus of the network on portions of the image in which the nuclei boundaries are likely to be missed. The overlapping patches mean that the same nuclei of the image contribute to the learned loss multiple times in a different context and varying neighboring environments. To resolve individual, touching and overlapping nuclei, we use a distance transform that is computed on the probability map, which essentially yields a global minimum in the case of individual nuclei and multiple separate local minimas for overlapping nuclei. This post processing technique is similar to [27], [73].

F. Spectral Normalization for GAN Stability

Discriminator normalization can improve stability. From an optimization point-of-view, such normalization leads to more efficient gradient flow. Various forms of normalization have been proposed, for example batch normalization (BN) was proposed for GAN frameworks in [74]. BN is done on a batch level, and normalizes pre-activations of nodes in a layer to the mean and standard deviation of the parameters learned for each node in the layer. Since a neural network can be seen as a composition of non-linear mappings with spectral properties. Spectral normalization was first suggested in [53] for improving GAN stability and entails dividing each weight matrix, including the matrix representing convolutional kernels, by their spectral norm. We use spectral normalization for stabilizing GAN training for both synthetic data generation and segmentation. Spectral normalization is used to control the Lipschitz constant of the discriminator such that the spectral norm σ of the convolution weights W in the network would be bounded by the Lipschitz constraint: σ(W) = 1. As a result, the discriminator is more stable during training and can avoid irregular gradients.

IV. Experiments and Results

A. Implementation Details

Dataset Preprocessing

The data obtained from [31] and our manually labeled data were normalized using sparse stain normalization to match a standard slide from the breast. Collectively, these data were sourced from nine different organs, 34 different patients and collected at different hospitals. Four slides from breast, liver, kidney and prostate were used for training. For testing, two slides from breast, liver, kidney, prostate, bladder, colon and stomach and one slide from esophagus and ovary were used. Thus, our test data includes images from four organs that the network was not trained on. All data were decomposed into large patches of size 256 × 256 for training the network efficiently. The training data was then supplemented with synthetically generated data containing 4, 650 patches. The overall training data had 4906 256 × 256 patches.

Synthetic Data Generation

Network Architectures:

The generator architectures contain two stride-2 convolutions, nine residual blocks and two functionally constrained convolutions with a stride of $\frac{1}{2}$. Reflection padding was used to minimize artifacts. The discriminator architecture was a simple classifier with three layers and the output was 70×70 with the aim to classify weather these overlapping patches were real or fake. As suggested in [52] a patch level discriminator has fewer parameters and is more easily applicable to various image sizes. We observed that larger size images needed more residual blocks for efficient convergence. The GAN training was stabilized to prevent mode collapse by using spectral normalization [53]. Additional details of the network architectures is discussed in the supplement.

Training Details:

The training code was implemented using Pytorch 0.4. For all experiments λ_n = 70 and λ_m = 10. These values were chosen after a log-linear grid search. Adam solver [75] was used to solve the optimization problem with a batch size of 1, which was experimentally determined. A total of 300 epochs was used. The learning rate was set to 0.0002 for the first 150 epochs and linearly decayed to zero for the remaining 150 epochs. Since the purpose of this architecture was to construct an accurate generator, we divided the objective function by two when optimizing the discriminator i.e. to give it a lower learning rate. All networks were trained from scratch with no prior knowledge and weights were initialized from a Gaussian distribution with a mean and standard deviation of 0 and 0.02 respectively.

Nuclei Segmentation

Network Architectures:

We use an encoder-decoder architecture with skip connections (U-Net [76]) for the generator. Skip connections are added between the i^th layer and the (n − i)^th layer, where each skip concatenates channels at the i^th and the (n − i)^th layers, where n is the total number of layers. After the last layer in the decoder, a convolution is applied to map the output segmentation mask followed by a Tanh function. Leaky ReLUs [77] were employed for the encoder with a slope of 0.2 and regular ReLUs were used for the decoder. A 70×70 patch Markovian discriminator, similar to the one described previously [51], was employed but with leaky ReLUs with a slope of 0.2. Additional details about this network architecture are discussed in the supplementary material.

Training Details:

During training, random jitter was applied to the data by resizing the 256 patches to 286 × 286 and then cropping them back to 256 × 256. The Adam optimizer was used to solve the objective function and all networks were trained from scratch where the weights were initialized from a Gaussian distribution of mean 0 and standard deviation 0.02. The training was run for 400 epochs and the learning rate was set to 0.0002 for the first 200 epochs and linearly decayed to zero for the remaining 200 epochs. Spectral normalization was used for increasing the stability of the adversarial training. A pooling history of randomly selected 64 patch pairs from the segmented output and ground truth was used in the discriminator.

B. Evaluation Criteria

For efficient evaluation of nuclei segmentation, both object-level and pixel-level errors must be penalized. Object-level penalty is associated with nuclei detection and pixel-level penalty is important for preserving fine boundaries, which influence the shape and size of the nuclei mask. As described in [31] any quantitative nuclei segmentation metric should penalize four different possible erroneous outcomes: a) Missed nuclei b) Detection of ghost objects as nuclei c) Under-segmentation (i.e., segmentation of overlapping nuclei as one) d) Over-segmentation (i.e., segmentation of a single nucleus as many). For our quantitative study we use three different evaluation metrics that assess these outcomes:

• Average Pompeiu–Hausdorff distance (aHD): The aHD is a pixel-level metric that calculates the greatest of all the distances from a point in the ground truth segmentation mask to the closest point in the predicted mask [79]. It can be calculated as $P\left(x,y\right)=\text{max}\left(\overrightarrow{p}\left(h,y\right),\overrightarrow{p}\left(y,h\right)\right)$ where $\overrightarrow{p}={\max }_a{\min }_b\Vert a-b\Vert$ and a ∈ h and b ∈ y. A lower HD indicates the two segmentation masks being compared are closer to each other.

• F₁ Score: The F₁ Score is an object-level metric defined by the harmonic mean between precision and recall, $F_1=\frac{\mathit{\text{Recall}}\times \mathit{\text{Precision}}}{\mathit{\text{Recall}}+\mathit{\text{Precision}}}$. A higher F₁ score indicates a better intersection between the ground truth and the predicted segmentation masks.

• Aggregated Jaccard Index (AJI): AJI was proposed in [31] specifically for quantitative evaluation of nuclei segmentation. AJI is an extension of the global Jaccard index which computes the ratio of aggregated intersection cardinality and aggregated union cardinality in the ROI. Assuming, $G={\cup }_{i=1,\mathrm{2..}K}{G}_i$ is the ground truth of the nuclei pixels and $P={\cup }_{j=1,\mathrm{2..}L}{P}_j$ are the prediction results, the AJI can then be defined as, $AJI=\frac{{\sum }_{i=1}^L\left|G_i\cap P_j^{*}\left(i\right)\right|}{{\sum }_{i=1}^K\left|G_i\cup P_j^{*}(i)\right|+{\sum }_{k\in U}\left|P_k\right|}$. Where, $P_j^{*}\left(i\right)$ is the connected component from the prediction result for which the Jaccard index is maximized, and U is the set of indices of detected ghosts that are not assigned to any component in the ground truth. A higher AJI indicates better results while penalizing all four possible errors mentioned above.

C. Quantitative Study and Results

We used data from nine different organs for evaluating the trained nuclei segmentation network. No data used for testing was used for training. To assess generalizability of the developed algorithms, we tested on four organs that were not represented in the real or synthetic training data. Table I summarizes the results of the testing on three different metrics mentioned above. A comparative analysis with standard segmentation architectures [82], [83] such as U-Net [76], Fully Convolutional Network (FCN) [68] and Mask R-CNN [84] was performed using an identical test and train split. These architectures were used for comparison because of their wide spread usage for a variety of different segmentation tasks including nuclei segmentation [85], [86]. Besides these standard architectures, we also make explicit comparisons with DIST [34] and CNN-3C [31] two state-of-the-art nuclei segmentation methods. We also compare with nuclei segmentation toolboxes available in Cell Profiler [81] and ImageJ-Fiji [80]. These comparisons and representative patches from different organs are shown in Fig. 2. The overlap between ground truth annotations from a pathologist and those detected by our method are highlighted in green and the difference between the ground truth and the prediction are shown in red. Our segmentation networks demonstrates a 29.19% improvement in AJI as compared to DIST and 42.98% as compared to CNN-3C. In terms of standard architectures there is a 44.27% improvement over standard Mask R-CNN and 73.19% over a U-Net. Among the 3224 overlapping nuclei 2769 were accurately segmented using our method so the segmentation rate among overlapping nuclei was 85.88% as compared to 79.46% with DIST and 77.38% with CNN-3C. Additional comparative analysis of the proposed Nuclei Segmentation Framework with Fused ENet [35] and MANA [78] has been shown in Table II. It should also be noted that There was an 8.68% improvement in the AJI by using sparse stain normalization.

TABLE I.

Comparative Analysis of the Proposed Nuclei Segmentation Framework Against Standard Architectures From the Test Dataset

Organ	Aggregated Jaccard Index (AJI) ↑				Average Hausdorff distance ↓				F1-Score ↑
	FCN	U-Net	Mask R-CNN	Proposed	FCN	U-Net	Mask R-CNN	Proposed	FCN	U-Net	Mask R-CNN	Proposed
Breast	0.339	0.485	0.483	0.686	8.17	7.927	6.286	4.761	0.794	0.774	0.749	0.881
Ovary	0.322	0.471	0.513	0.728	8.48	8.142	7.236	4.221	0.675	0.702	0.725	0.876
Esophagus	0.293	0.392	0.433	0.759	8.71	8.349	7.394	3.963	0.667	0.698	0.733	0.891
Liver	0.364	0.422	0.469	0.692	7.93	7.463	6.927	4.173	0.683	0.697	0.741	0.794
Kidney	0.472	0.532	0.542	0.724	7.34	6.858	6.538	4.492	0.775	0.783	0.779	0.829
Prostate	0.281	0.276	0.536	0.731	8.94	8.347	7.912	4.013	0.794	0.802	0.784	0.857
Bladder	0.461	0.491	0.477	0.768	7.11	7.423	7.612	4.273	0.746	0.791	0.769	0.904
Colorectal	0.247	0.218	0.391	0.686	9.27	8.729	7.743	4.712	0.719	0.733	0.691	0.836
Stomach	0.383	0.437	0.629	0.721	7.52	7.314	7.518	4.017	0.866	0.874	0.843	0.922
Overall	0.351	0.414	0.497	0.721	8.163	7.839	7.136	4.291	0.746	0.761	0.757	0.866

Fig. 2.

Representative patches from six different organs and corresponding nuclei segmentation masks predicted by our proposed method, overlaid on ground truth segmentation masks. The green region represents an overlap between the prediction and manually labeled ground truth whereas the red region represents a disparity between the two. The predominance of the green region demonstrates accurate labeling. The bar charts compare the AJI for all test patches of corresponding organs with state-of-the-art methods (DIST [34] and CNN-3C [31]) as well as commonly-used segmentation tools in Fiji [80] and Cell Profiler [81].

TABLE II.

Comparative Analysis of the Proposed Nuclei Segmentation Framework With Fused ENet [35] and MANA [78]

Metric	Fused ENet [35]	MANA [78]	Ours
Aggregated Jaccard Index (AJI)↑	0.601	0.673	0.721
Average Hausdorff distance↓	6.219	4.882	4.291
F1-Score↑	0.733	0.839	0.866

V. Discussion and Conclusion

Objective, accurate, and automated analysis of H&E slides has the potential to revolutionize clinical practice in pathology, by improving the detection, grading, classification, and quantitative analysis of aberrant phenotypes. Towards realizing this potential, this work addresses one of the most fundamental tasks in computational pathology nuclear segmentation. Nuclear morphology and distribution are paramount for the analysis of histopathology slides by computational pathology. In addition to providing foundational features, nuclei shape and position can be used to enhance network attention.

We propose a single network that is trained with four organs and synthetically generated pathology data. Our network is trained using an adversarial pipeline which has a larger receptive field as compared to standard CNNs and captures more global information. This approach captures higher-order statistics from the image and the resulting networks are more context-aware. We pose the segmentation problem as an image-to-image translation task rather than a classification task. Doing so allows us to learn a complex loss function between the output and the ground truth rather than having to use a manually engineered one. We demonstrate that this approach performs better than standard architectures, state-of-the-art methods and general purpose tools such as Fiji and Cell Profiler for nuclei segmentation.

Future work will involve adapting this approach to other medical imaging modalities, as well as fusing nuclear morphology information with other features for networks that can improve detection, classification, grading and prognosis from histopathology images. Future work will also explore the adversarial training paradigm for structured prediction and generalizing the concept for other applications. The source code and trained models are publically available: http://github.com/mahmoodlab/NucleiSegmentation