Deep fusion of contextual and object-based representations for delineation of multiple nuclear phenotypes

Abstract

Motivation

Nuclear delineation and phenotypic profiling are important steps in the automated analysis of histology sections. However, these are challenging problems due to (i) technical variations (e.g. fixation, staining) that originate as a result of sample preparation; (ii) biological heterogeneity (e.g. vesicular versus high chromatin phenotypes, nuclear atypia) and (iii) overlapping nuclei. This Application-Note couples contextual information about the cellular organization with the individual signature of nuclei to improve performance. As a result, routine delineation of nuclei in H&E stained histology sections is enabled for either computer-aided pathology or integration with genome-wide molecular data.

Results

The method has been evaluated on two independent datasets. One dataset originates from our lab and includes H&E stained sections of brain and breast samples. The second dataset is publicly available through IEEE with a focus on gland-based tissue architecture. We report an approximate AJI of 0.592 and an F1-score 0.93 on both datasets.

Availability and implementation

The code-base, modified dataset and results are publicly available.

Supplementary information

Supplementary data are available at Bioinformatics online.

1 Introduction

Nuclear segmentation from histology sections is an important step prior to any morphological feature extraction. Due to variations in sample preparation and biological heterogeneity, nuclear segmentation is challenging. In addition, the presence of overlapping nuclei makes the problem even more difficult. Numerous studies have been done on automatic nuclear segmentation (Irshad et al., 2014; Xing and Yang, 2016) with most recent research focused on deep convolutional neural networks (CNNs) (Sirinukunwattana et al., 2016; Xie et al., 2015; Xing et al., 2016). In Khoshdeli et al. (2018), we showed that fusion of deep encoder–decoder networks could outperform delineation of various nuclear phenotypes when compared to other deep learning models (e.g. multitask, U-Net). In this Application-Note, we fuse the region-based segmentation with contextual information to enhance the delineation of nuclei by defining a potential-field function. A deep learning model with a regression loss function trains this potential-field function. As a result, perceptual boundaries between adjacent nuclei are completed, and the amount of final post-processing is reduced significantly.

2 Materials and methods

A summary of the framework and rationale for introducing the potential-field follows. The framework for the proposed nuclei segmentation is shown in Supplementary Figure S1. First, the potential-field function is integrated with the segmentation mask using the second derivative of the distance transform. Second, a deep convolutional model is trained to learn potential-field values. Third, the output of the region-based segmentation and the learned potential-field function are integrated via a fusion model, which is also a deep network model. These steps are described in further detail below.

Integration of the potential-field function is inspired by the ability to predict the perceptual boundary between adjacent nuclei. As a result, each nucleus can be delineated from a clump of touching nuclei. The potential-field function should have a high value at the location of perceptual boundaries and zero elsewhere. In this context, the potential-field function is computed with the second derivative of the distance transform of the annotated masks. Supplementary Figure S2 indicates the steps for computing the potential function.

The prediction of the potential-field function is achieved by training a deep regression model. For this purpose, a deep convolutional encoder–decoder, based on the ENet (Paszke et al., 2016) model, is selected. ENet has been shown to outperform other architectures in our previous research (Khoshdeli et al., 2018).

The ENet model is modified with: (i) the loss function defined as the mean square error to better suit the regression problem, (ii) the number of layers is decreased from 17 to 10 to determine the simplest network without sacrificing performance and (iii) limiting filter sizes to 3-by-3 to enable high throughput segmentation that is necessary for whole slide images.

Integration of the potential-field function with the output of region-based segmentation is performed by training a deep fusion model. This fusion model is a convolutional network that takes the potential-field and region-based probabilities as the input features and produces the final nuclear segmentation. The architecture of the fusion model consists of four layers of 3-by-3 convolution, followed by a ReLU activation function. Supplementary Table S1 indicates the design of the fusion model.

3 Results and discussion

In this section, we report the comparison of the fused potential-field model with other methods via a visual example consisting of multiple nuclear phenotypes, as shown in Supplementary Figure S3.

The comparative analysis includes (i) Fused-ENet (Khoshdeli et al., 2018); (ii) deep regression of the distance map (Naylor et al., 2019) and (iii) the CNN model proposed in Kumar et al. (2017). Table 1 shows the performance on the IEEE-TMI (Kumar et al., 2017) dataset, and both AJI and F1-score are improved by the fused potential-field model. Table 2 compares the performance between the Fused-ENet (Khoshdeli et al., 2018) and the proposed model using our previously published dataset (Khoshdeli et al., 2018). One of the innovations of our approach is that the utility of the watershed method has been significantly reduced through the integration of contextual information. In addition, previous researchers have opted to set an arbitrary threshold of 0.5 for the probability map, however, as shown in Supplementary Table S2, we have learned that tuning the threshold for the probability map improves performance. Finally, we measured the performance of the proposed method after post-processing with watershed again and observed a slightly improved performance, as shown in Table 3.

Table 1.

The proposed method yields an improved AJI and F1-score on the IEEE-TMI dataset (Kumar et al., 2017)

Experiment #	Approach	AJI	F1-score
1	Proposed	0.5702	0.8756
2	Deep regression of the distance map	0.5598	0.7863
3	Kumar’s model	0.5083	0.8267

Note: The use boldface is to accentuate the best score.

Table 2.

The proposed method yields an improved AJI and F1-score on the BMC dataset (Khoshdeli et al., 2018) without extensive post-processing

Experiment #	Approach	AJI	F1-score
1	Proposed	0.5844	0.9241
2	Fused-ENet	0.5687	0.9124

Note: The use boldface is to accentuate the best score.

Table 3.

Post-processing with watershed improves the performance for both datasets

Experiment #	dataset	AJI	F1-score
1	IEEE-TMI (Kumar et al., 2017)	0.5896	0.9267
2	BMC (Khoshdeli et al., 2018)	0.5981	0.9281

Note: The use boldface is to accentuate the best score.

Funding

This manuscript was supported in part by NIH R15CA235430.

Conflict of Interest: none declared.