Sphere estimation network: three-dimensional nuclei detection of fluorescence microscopy images

Abstract.

Purpose: Fluorescence microscopy visualizes three-dimensional subcellular structures in tissue with two-photon microscopy achieving deeper penetration into tissue. Nuclei detection, which is essential for analyzing tissue for clinical and research purposes, remains a challenging problem due to the spatial variability of nuclei. Recent advancements in deep learning techniques have enabled the analysis of fluorescence microscopy data to localize and segment nuclei. However, these localization or segmentation techniques would require additional steps to extract characteristics of nuclei. We develop a 3D convolutional neural network, called Sphere Estimation Network (SphEsNet), to extract characteristics of nuclei without any postprocessing steps.

Approach: To simultaneously estimate the center locations of nuclei and their sizes, SphEsNet is composed of two branches to localize nuclei center coordinates and to estimate their radii. Synthetic microscopy volumes automatically generated using a spatially constrained cycle-consistent adversarial network are used for training the network because manually generating 3D real ground truth volumes would be extremely tedious.

Results: Three SphEsNet models based on the size of nuclei were trained and tested on five real fluorescence microscopy data sets from rat kidney and mouse intestine. Our method can successfully detect nuclei in multiple locations with various sizes. In addition, our method was compared with other techniques and outperformed them based on object-level precision, recall, and $F1$ score. Our model achieved 89.90% for $F1$ score.

Conclusions: SphEsNet can simultaneously localize nuclei and estimate their size without additional steps. SphEsNet can be potentially used to extract more information from nuclei in fluorescence microscopy images.

1. Introduction

Fluorescence microscopy is optical microscopy using visible light and a combination of lenses to image magnified subcellular structures in tissue in 3D.¹ Especially, two-photon microscopy, using two photons simultaneously for fluorescence excitation, can image deeper into tissue.² Fluorescence microscopy enables analyzing intravital studies of organs.³ An essential process of quantifying microscopy images is nuclei detection. In this paper we define “nuclei detection” as a task of finding locations and characteristics of each nucleus, which is important for clinical and research purposes.⁴ Detecting the size of nuclei is crucial to understand their cell functions.⁵

An initial step for nuclei detection is to find locations of nuclei accurately in microscopy images. A thresholding technique to separate foreground (nuclei) and background⁶ can be used to find locations of nuclei. A progressive weighted mean curve of a grayscale histogram was used to select a thresholding value to automatically segment cell membranes and nuclei in cardiospheres presented in fluorescence microscopy volumes.⁷ Some microscopy data sets may suffer from inhomogeneous intensity in volumes which can be corrected using adaptive histogram equalization as a preprocessing step.⁸ Detecting locations of nuclei on fluorescence microscopy images is still challenging because some nuclei in a volume may touch or overlap.⁴ After the thresholding process, watershed-based techniques⁹ have been used to separate touching nuclei. A method to find watershed seeds based on mathematical morphology in time-lapse microscopy was introduced.¹⁰ Watershed-based techniques can be sensitive to irregular biological structures or noise presented in microscopy volumes leading to overdetection.

Active contours, also known as snakes, have been developed to detect structures in irregular shapes to avoid overdetection.¹¹ Active contour can be used on microscopy images because it can capture nuclei in various sizes and shapes.¹² A method of coupling multiple active surfaces by penalizing overlaps and constraining volumes was introduced to separate touching nuclei¹³ using region-based active surfaces,¹⁴ showing promising results of detecting objects with poorly defined boundaries. This method was further developed using the watershed technique, a non-PDE-based energy minimization, and the Radon transform.¹⁵ A nuclei segmentation technique based on active contours with inhomogeneity correction was described to overcome intensity inhomogeneity in flourescence microscopy volumes.¹⁶ Alternatively, a model-based method by identifying nuclei with four primitives representing boundaries and delineating them using region growing was designed.¹⁷ A method using the fast radial symmetry transform and a dilation-based nonmaximum suppression to detect nuclei in various sizes and shapes was introduced.¹⁸ However, these techniques cannot distinguish nuclei from other subcellular structures causing many false-detection.

Recently, convolutional neural networks (CNNs)¹⁹ have become powerful tools to solve computer vision problems, such as classification,²⁰ detection,²¹ and segmentation.²² Using CNNs, nuclei detection has made a big progress to find the location of nuclei.²³ A patch-based method with max-pooling layers was introduced to detect mitosis in breast histology images.²⁴ A window is slid over an image, and a CNN estimates the probability of the center pixel being mitosis. After smoothing the probability map, mitosis are detected by nonmaximal suppression. Another patch-based method to detect tyrosine hydroxylase-containing cells in zebrafish images was described.²⁵ A set of training patches are selected on both cell region and background region using a support vector machine classifier.²⁶ Patch-based methods by sliding a window centered at every pixel are computationally expensive due to repeated convolutional operations when a majority portion of a current patch overlaps with previous patches. To avoid redundant convolutional operations in overlapping patches, fast scanning²⁷ was employed into CNNs by performing convolutional operations on the entire image. With the fast scanning technique, a nucleus localization method in microscopy images, called deep voting, was developed by extracting voting offsets and voting confidences to estimate the voting density map.²⁸ Alternatively, a CNN-based structured regression model trained by a proximity mask assigning higher value to nuclei centers was described.²⁹ A spatially constrained CNN was introduced with a parameter estimation layer and a spatially constrained layer for spatial regression to more accurately locate nuclei in hematoxylin and eosin-stained histopathology images.³⁰ A cell nuclei detection method, known as vector oriented confidence accumulation (VOCA), was developed by accumulating localization vectors pointing to nuclei center locations to accurately detect the center locations.³¹ These CNN-based methods can successfully detect nuclei locations but they do not provide further characteristics of nuclei such as the size.

To extract size information of objects, a faster R-CNN³² and a mask R-CNN³³ were developed to estimate bounding boxes surrounding objects and masks on objects, respectively. A cell proposal network, similarly done to the faster R-CNN,³² was described to detect cells on microscopy images within bounding boxes.³⁴ Cells are generally not in rectangular shapes so detecting cells in bounding boxes may not provide useful information. A mask R-CNN³³ was used to detect locations of nuclei within masks in distinct labels in microscopy images.³⁵ Alternatively, a cell detection method by object probability predictions and radian distances in multiple orientations to generate star-convex polygons was described.³⁶ Furthermore, a multi-input multi-output network was designed by utilizing multiple resolutions to segment cells in various sizes in fluorescence microscopy images.³⁷ These methods can only process microscopy data in 2D and cannot use depth information in microscopy volumes.

To utilize the entire information in microscopy 3D volumes, 3D CNNs have been developed. A 3D CNN for nuclei detection and segmentation on microscopy images was described.³⁸ The 3D CNN was composed of a 3D U-Net³⁹ for segmentation followed by a series of convolutional layers for detection. The 3D CNN was trained by a manually labeled training set which may be tedious to produce for various data sets. Data augmentation approaches are widely used to multiply a small set of labeled ground truth volumes.²⁰ Simple transformations, such as translation, reflection, rotation, and color jitter, can increase the number of training volumes. For example, a cell detection and segmentation plugin developed in ImageJ⁴⁰ using manual labels with data augmentation approaches was presented.⁴¹ Our goal is to train 3D CNNs without labeling real ground truth volumes to reduce a burden of the 3D labeling process. We introduced a technique⁴² to automatically generate synthetic microscopy volumes containing multiple ellipsoids with blurring⁴³ and noise⁴⁴ operations without using real ground truth volumes to train a 3D segmentation CNN. In addition, we presented a 3D detection-segmentation method to label each nucleus distinctly.⁴⁵ Specifically, nuclei center coordinates were first detected by a 3D distance transform⁴⁶ to select center candidates and a 3D classification CNN to remove center candidates on non-nuclei structures. A 3D segmentation CNN trained by synthetic microscopy volumes similarly done as our previous technique⁴² labeled individual nuclei by processing subvolumes centered at the coordinates from the detection stage. One problem in our previous works^42,45 is that the synthetic microscopy volumes do not look realistic. To achieve better performance, generating more realistic synthetic volumes is necessary.

Generative adversarial networks (GANs) are developed to generate realistic synthetic images and volumes to be used during training.⁴⁷ A cycle-consistent adversarial network, known as a CycleGAN, was introduced where a cycle consistency term was included without requiring any real ground truth volumes.⁴⁸ More recently, we developed a spatially constrained cycle-consistent adversarial network, known as a SpCycleGAN, including a spatial constraint term to accurately match synthetic nuclei locations to their corresponding nuclei mask locations.⁴⁹ By training CNNs using synthetic microscopy volumes from the SpCycleGAN⁴⁹ without real ground truth volumes, we distinctly segmented nuclei.⁵⁰

To further analyze and understand nuclei in fluorescence microscopy volumes, we present a 3D nuclei detection network called Sphere Estimation Network (SphEsNet) to find both center coordinates and radii of nuclei simultaneously using our single network. Instead of detecting nuclei in bounding boxes,³² our proposed network can detect nuclei in spheres with proper sizes because most cell nuclei are known to be round.⁵ Manually generating ground truth volumes to train our SphEsNet by annotating center locations of nuclei and especially measuring their radii in 3D can be tedious and time-consuming. We train our model using synthetic microscopy volumes automatically generated from the SpCycleGAN we developed.⁴⁹ Note no real ground truth volumes are required while generating synthetic volumes. Our work presented in this paper is different from our previous works focusing on nuclei segmentation.^45,50 Our previous works require two steps, nuclei center localization followed by segmentation, using multiple CNNs. The biological and clinical goal of the analysis is not segmenting nuclei but extracting characteristics of nuclei. Thus, additional postprocessing steps would be necessary to extract characteristics of nuclei from segmentation masks. For example, an additional least square fitting method⁵¹ on segmentation masks would be required to estimate the size of nuclei. The main contribution of this work is to detect the location of nuclei and to estimate radii of nuclei simultaneously using our single CNN without additional postprocessing steps. We process two-photon microscopy data from rat kidney and mouse intestine. Our evaluation using a ground truth volume on real fluorescence microscopy data shows that our SphEsNet can successfully estimate both locations and sizes of nuclei.

2. Proposed Method

Figure 1 is a block diagram of the proposed 3D nuclei detection method. To denote 3D image volumes, we use the notation we defined in our previous works.^42,45,49,50 In this paper, we denote $I$ as a 3D image volume of size $X\times Y\times Z$ voxels and $I_{z_p}$ as the $p$’th 2D focal plane image of the 3D image volume, $I$, along the $z$-direction, where $p\in \{1,\dots ,Z\}$. For example, $I^{\mathrm{orig}}$ is an original 3D volume and $I_{z_{50}}^{\mathrm{orig}}$ is the 50th focal plane image of the volume. In addition, we denote $I_{(q_i:q_f,s_i:s_f,p_i:p_f)}$ as a 3D subvolume of $I$, whose $x$ coordinate is $q_i\le x\le q_f$, $y$ coordinate is $s_i\le y\le s_f$, $z$ coordinate is $p_i\le z\le p_f$, where $q_i\in \{1,\dots ,X\}$, $q_f\in \{1,\dots ,X\}$, $s_i\in \{1,\dots ,Y\}$, $s_f\in \{1,\dots ,Y\}$, $p_i\in \{1,\dots ,Z\}$, and $p_f\in \{1,\dots ,Z\}$. It is required to have $q_i\le q_f$, $s_i\le s_f$, and $p_i\le p_f$. For example, $I_{(193:\mathrm{320,193}:\mathrm{320,31}:94)}^{\det }$ is a subvolume of a detection volume, $I^{\det }$, whose $x$ coordinate is $193\le x\le 320$, $y$ coordinate is $193\le y\le 320$, and $z$ coordinate is $31\le z\le 94$. Lastly, we denote $\mathbf{v}$ as a voxel location.

Fig. 1.

Block diagram of our proposed method for 3D nuclei detection of fluorescence microscopy images. Our SphEsNet is trained by synthetic microscopy volumes, which are automatically generated using an SpCycleGAN.⁴⁹ After training is done, an original microscopy volume is processed to detect nuclei in 3D.

The goal of this paper is to simultaneously estimate the location and size of nuclei presented in an original fluorescence microscopy 3D volume, $I^{\mathrm{orig}}$, using our SphEsNet. To train the SphEsNet, we produce nuclei center volumes, $I^{\mathrm{ctr}}$, to estimate the location of nuclei, nuclei radius volumes, $I^{\mathrm{rad}}$, to estimate the size of nuclei, and nuclei mask volumes, $I^{\text{mask}}$, to generate synthetic microscopy volumes, $I^{\mathrm{syn}}$, using an SpCycleGAN.⁴⁹ Note $I^{\text{mask}}$ is composed of multiple spherical masks where the size range of the masks matches to the size range of nuclei in the input microscopy volume, $I^{\mathrm{orig}}$. The location and size of synthetic nuclei in $I^{\mathrm{syn}}$ match to the location and size of spherical masks in $I^{\text{mask}}$. Once our SphEsNet model, $M$, is trained by $I^{\mathrm{syn}}$, $I^{\mathrm{ctr}}$, and $I^{\mathrm{rad}}$, it can be used on the real microscopy volume, $I^{\mathrm{orig}}$, to detect nuclei. After processing $I^{\mathrm{orig}}$ using our SphEsNet, the final detection volume, $I^{\det }$, contains color-coded estimated spheres representing nuclei in the microscopy volume. Multiple SphEsNet models can be trained to detect nuclei in other real microscopy volumes based on various nuclei size ranges. Our SphEsNet is implemented in PyTorch.⁵²

2.1. Synthetic Volume Generation

The process of manually labeling ground truth for microscopy volumes in 3D is tedious. Thus, a method of automatically generating synthetic volumes to train the SphEsNet is necessary. In this section, we use an SpCycleGAN we developed⁴⁹ to generate synthetic volumes for training. Note no real ground truth volumes are required in this technique. The synthetic volume generation stage is composed of two steps: the synthetic ground truth volume generation step and the synthetic microscopy volume generation step. The synthetic ground truth volume generation step generates a set of nuclei center volumes, $I^{\mathrm{ctr}}$, a set of nuclei radius volumes, $I^{\mathrm{rad}}$, and a set of nuclei mask volumes, $I^{\text{mask}}$. The synthetic microscopy volume generation step generates a set of synthetic microscopy volume, $I^{\mathrm{syn}}$, from $I^{\text{mask}}$ using the SpCycleGAN.⁴⁹ Three sets of volumes, $I^{\mathrm{syn}}$, $I^{\mathrm{ctr}}$, and $I^{\mathrm{rad}}$, are then used to train our SphEsNet. In this stage, 20 synthetic volumes with size of $128\times 128\times 128\text{voxels}$ are generated.

2.1.1. Synthetic ground truth volume generation

During the synthetic ground truth volume generation step, $I^{\text{mask}}$ is first generated. Note $I^{\text{mask}}$ is used in the synthetic microscopy volume generation step to generate the corresponding $I^{\mathrm{syn}}$ where each mask matches to a synthetic nucleus. To produce $I^{\text{mask}}$ with multiple masks, $N$ mask candidate volumes are generated. We denote the $i$’th mask candidate volume as $I^{\text{mask},i}$ where $1\le i\le N$. Each mask candidate volume contains one spherical mask which can be potentially included in $I^{\text{mask}}$. Specifically, $I^{\text{mask},i}$ is a binary volume where a mask is centered at a random location ${\mathbf{c}}_i=(x_i,y_i,z_i)$ with a random radius, $r_i$:

$$I^{\text{mask},i}(\mathbf{v})=\{\begin{array}{ll}1,& \text{if}{\Vert \mathbf{v}-{\mathbf{c}}_i\Vert }_2^2\le r_i^2\\ 0,& \text{otherwise}\end{array},$$ (1)

where $1\le x_i\le X$, $1\le y_i\le Y$, $1\le z_i\le Z$, and $r_{\min }\le r_i\le r_{\max }$. $r_{\min }$ and $r_{\max }$ are the minimum possible radius of a synthetic nucleus in a volume and the maximum possible radius of a synthetic nucleus in a volume, respectively. $N$ can be selected based on the density of nuclei in $I^{\mathrm{orig}}$, and $r_{\max }$ and $r_{\min }$ can be selected based on the size of nuclei in $I^{\mathrm{orig}}$.

After mask candidate volumes are generated, $I^{\text{mask}}$ is generated by adding $I^{\text{mask},i}$ in a sequential order. First of all, $I^{\text{mask}}$ is initialized to zeros. Note nuclei cannot be overlapped physically in a biological structure. For $1\le i\le N$, the $I^{\text{mask},i}$ is added to $I^{\text{mask}}$ if $I^{\text{mask},i}$ has no overlap with other masks already added in $I^{\text{mask}}$. If $I^{\text{mask},i}$ is overlapped with one of the masks in $I^{\text{mask}}$, $I^{\text{mask},i}$ is not included in $I^{\text{mask}}$. Once this process is done, then $I^{\text{mask}}$ contains $N_{\text{mask}}$ number of nuclei where $N_{\text{mask}}\le N$. This technique is similar to our previous synthetic binary volume generation technique.⁴²

To generate $I^{\mathrm{ctr}}$ and $I^{\mathrm{rad}}$, we first define “central regions.” Note the central regions are defined to more accurately find the location of nuclei centers using our CNN. For the $j$’th nucleus, where $1\le j\le N_{\text{mask}}$, whose center location is located at ${\mathbf{c}}_j=(x_j,y_j,z_j)$ and radius is $r_j$, the $j$’th central region is defined within a sphere centered at the ${\mathbf{c}}_j$ with a radius of $a{r}_j$. $a$ is a parameter determining the range of the central region where $0\le a\le 1$. In this work, we set $a=0.5$. Within the $j$’th central region, $I^{\mathrm{ctr},j}$ contains information of the location of the $j$’th nucleus in $I^{\text{mask}}$ and $I^{\mathrm{rad},j}$ contains information of the radius of the $j$’th nucleus in $I^{\text{mask}}$. Voxel values within the central region of $I^{\mathrm{ctr},j}$ become 1 and of $I^{\mathrm{rad},j}$ become $r_j$ whereas other voxel values not in the central region of both $I^{\mathrm{ctr},j}$ and $I^{\mathrm{rad},j}$ become 0:

$$I^{\mathrm{ctr},j}(\mathbf{v})=\{\begin{array}{ll}1,& \text{if}{\Vert \mathbf{v}-{\mathbf{c}}_j\Vert }_2^2\le {(a{r}_j)}^2\\ 0,& \text{otherwise}\end{array},$$ (2)

$$I^{\mathrm{rad},j}(\mathbf{v})=\{\begin{array}{ll}{r}_j,& \text{if}{\Vert \mathbf{v}-{\mathbf{c}}_j\Vert }_2^2\le {(a{r}_j)}^2\\ 0,& \text{otherwise}\end{array}.$$ (3)

Then, $I^{\mathrm{ctr}}$ and $I^{\mathrm{rad}}$ are the sum of $I^{\mathrm{ctr},j}$ and $I^{\mathrm{rad},j}$ for all $j$, respectively:

$$I^{\mathrm{ctr}}=\sum _{j=1}^{N_{\text{mask}}}{I}^{\mathrm{ctr},j},$$ (4)

$$I^{\mathrm{rad}}=\sum _{j=1}^{N_{\text{mask}}}{I}^{\mathrm{rad},j}.$$ (5)

2.1.2. Synthetic microscopy volume generation

In the synthetic microscopy volume generation step, $I^{\text{mask}}$ is used to generate the corresponding $I^{\mathrm{syn}}$ using the SpCycleGAN.⁴⁹ The SpCycleGAN contains generative networks, $G_1$, $G_2$, and $H$, and discriminative networks, $D_1$ and $D_2$, where the generative networks produce realistic synthetic images, and the discriminative networks determine if the images are real or synthetic. The training loss function of the SpCycleGAN is defined as a sum of four loss terms:⁴⁹

$$\begin{gathered} L(G_1,G_2,H,D_1,D_2)=L_{\mathrm{GAN}}(G_1,D_1,I^{\text{mask}},I^{\mathrm{orig}})+L_{\mathrm{GAN}}(G_2,D_2,I^{\mathrm{orig}},I^{\text{mask}}) \\ +{\lambda }_1{L}_{\text{cycle}}(G_1,G_2,I^{\mathrm{orig}},I^{\text{mask}})+{\lambda }_2{L}_{\mathrm{sp}}(G_1,H,I^{\mathrm{orig}},I^{\text{mask}}), \end{gathered}$$ (6)

where ${\lambda }_1$ and ${\lambda }_2$ are weight coefficients. In Eq. (6), $G_1$ is a generative network generating $I^{\mathrm{orig}}$ using $I^{\text{mask}}$, $G_2$ is a generative network generating $I^{\text{mask}}$ using $I^{\mathrm{orig}}$, $D_1$ is a discriminative network distinguishing between $I^{\mathrm{orig}}$ and $G_1(I^{\text{mask}})$, $D_2$ is a discriminative network distinguishing between $I^{\text{mask}}$ and $G_2(I^{\mathrm{orig}})$, and $H$ is a generative network generating a binary mask, $H[G_1(I^{\text{mask}})]$, using $G_1(I^{\text{mask}})$. The first loss term in Eq. (6) is the adversarial loss⁴⁷ between $G_1$ and $D_1$:

$$L_{\mathrm{GAN}}(G_1,D_1,I^{\text{mask}},I^{\mathrm{orig}})=E_{I^{\mathrm{orig}}}\{\log [D_1(I^{\mathrm{orig}})]\}+E_{I^{\text{mask}}}(\log \{1-D_1[G_1(I^{\text{mask}})]\}).$$ (7)

The second loss term in Eq. (6) is the adversarial loss⁴⁷ between $G_2$ and $D_2$:

$$L_{\mathrm{GAN}}(G_2,D_2,I^{\mathrm{orig}},I^{\text{mask}})=E_{I^{\text{mask}}}\{\log [D_2(I^{\text{mask}})]\}+E_{I^{\mathrm{orig}}}(\log \{1-D_2[G_2(I^{\mathrm{orig}})]\}).$$ (8)

The third loss term in Eq. (6) is the cycle consistency loss:⁴⁸

$$\begin{gathered} L_{\text{cycle}}(G_1,G_2,I^{\mathrm{orig}},I^{\text{mask}}) \\ =E_{I^{\text{mask}}}\{{\Vert G_2[G_1(I^{\text{mask}})]-I^{\text{mask}}\Vert }_1\}+E_{I^{\mathrm{orig}}}\{{\Vert G_1[G_2(I^{\mathrm{orig}})]-I^{\mathrm{orig}}\Vert }_1\}, \end{gathered}$$ (9)

where ${\Vert .\Vert }_1$ is $L_1$ norm. The last loss term in Eq. (6) is the spatial constraint loss:⁴⁹

$$L_{\mathrm{sp}}(G_1,H,I^{\mathrm{orig}},I^{\text{mask}})=E_{I^{\text{mask}}}\{{\Vert H[G_1(I^{\text{mask}})]-I^{\text{mask}}\Vert }_2\},$$ (10)

where ${\Vert .\Vert }_2$ is $L_2$ norm. Including the spatial constraint loss to the training loss function of SpCycleGAN reduces spatial shifts between nuclei masks in $I^{\text{mask}}$ and synthetic nuclei in $I^{\mathrm{syn}}$.⁴⁹ In this work, we use the same generative network architecture and the same discriminative network architecture from a CycleGAN.⁴⁸ To train the SpCycleGAN, the Adam optimizer⁵³ is used with a learning rate of 0.001 with the training loss function in Eq. (6) for 200 epochs, and we set ${\lambda }_1=10$ and ${\lambda }_2=10$. After training the SpCycleGAN, $G_1$ is used to produce synthetic images, $I^{\mathrm{syn}}$, from labeled images, $I^{\text{mask}}$. Figure 2 shows an example of $I^{\mathrm{orig}}$, $I^{\mathrm{syn}}$, $I^{\text{mask}}$, $I^{\mathrm{ctr}}$, and $I^{\mathrm{rad}}$.

Fig. 2.

A set of examples of $I^{\mathrm{orig}}$, $I^{\mathrm{syn}}$, $I^{\text{mask}}$, $I^{\mathrm{ctr}}$, and $I^{\mathrm{rad}}$ with size of $128\times 128\text{pixels}$ generated by an SpCycleGAN⁴⁹ (a) a 2D image of $I^{\mathrm{orig}}$, (b) a 2D image of $I^{\mathrm{syn}}$. (c) the corresponding 2D image of $I^{\text{mask}}$, (d) the corresponding 2D image of $I^{\mathrm{ctr}}$, and (e) the corresponding 2D image of $I^{\mathrm{rad}}$.

2.2. Sphere Estimation Network

Figure 3 shows the architecture of our SphEsNet. The proposed network is different from our previous works to distinctly segment nuclei.^45,50 Our previous works consist of two steps, where the first step is to find center locations of nuclei and the second step is to segment individual nuclei. Using our single network, we can simultaneously find center locations of nuclei and estimate their radii.

Fig. 3.

Architecture of our SphEsNet. Our SphEsNet is composed of a backbone network, which is a modified 3D U-Net and two branches. CTR branch finds the center locations of nuclei and RAD branch estimates the radii of nuclei.

Our architecture consists of a backbone network and two branches. The first branch estimates center locations of nuclei and the second branch estimates radii of nuclei. In this paper, we denote the first branch as the CTR branch and the second branch as the RAD branch. For the backbone network, we employ a modified 3D U-Net architecture. Our backbone network has the same number of convolutional layers, max-pooling layers, and transposed convolutional layers in the 3D U-Net,³⁹ but the number of channels are reduced. A 3D convolutional layer is a series of a convolutional function using filters with size of $3\times 3\times 3$ and with padding of 1, a 3D batch normalization,⁵⁴ and a rectified-linear unit (ReLU) as an activation function. A 3D max-pooling layer using a window with size of $2\times 2\times 2$ and with a stride of 2 is used as a downsampling operation in the encoder, and a 3D transposed convolutional layer is used as an upsampling operation in the decoder. A concatenation operation between the encoder and the decoder is included to preserve spatial information. The final feature map intensities of the CTR branch are between 0 and 1 to classify each voxel to center location or not, so the Sigmoid function is used as an activation function. The final feature map intensities of the RAD branch are greater than 0 so the ReLU function is used as an activation function. More details on training and inference are provided below.

2.2.1. Training

During training, $I^{\mathrm{syn}}$, $I^{\mathrm{ctr}}$, and $I^{\mathrm{rad}}$ generated from the synthetic volume generation stage are used as an input volume, a ground truth volume in the CTR branch, and a ground truth volume in the RAD branch, respectively. Note that 20 sets of volumes with size of $128\times 128\times 128\text{voxels}$ are generated in the synthetic volume generation stage. We divide each volume to eight volumes with size of $64\times 64\times 64\text{voxels}$, so 160 sets of volumes are used to train the model, $M$. Our training loss function, $L$, is a linear combination of the binary cross entropy (BCE) loss, $L^{\mathrm{BCE}}$, at the CTR branch and the mean squared error (MSE) loss, $L^{\mathrm{MSE}}$, at the RAD branch:

$$L(I^{\mathrm{ctr}},I^{\mathrm{rad}},I^{\mathrm{ctr},\text{out}},I^{\mathrm{rad},\text{out}})=L^{\mathrm{BCE}}(I^{\mathrm{ctr}},I^{\mathrm{ctr},\text{out}})+\lambda L^{\mathrm{MSE}}(I^{\mathrm{rad}},I^{\mathrm{rad},\text{out}}),$$ (11)

where $I^{\mathrm{ctr}}$ is the ground truth volume in the CTR branch, $I^{\mathrm{rad}}$ is the ground truth volume in the RAD branch, $I^{\mathrm{ctr},\text{out}}$ is the output volume in the CTR branch, and $I^{\mathrm{rad},\text{out}}$ is the output volume in the RAD branch, respectively. $\lambda$ is a weight coefficient that can emphasize one loss term over the other loss term. For example, if $\lambda$ is small, then the BCE loss at the CTR branch will be more emphasized. Also, if $\lambda$ is big, then the MSE loss at the RAD branch will be more emphasized. In this work, we set $\lambda =1$ unless specified. More specifically, the BCE loss is defined as

$$L^{\mathrm{BCE}}(I^{\mathrm{ctr}},I^{\mathrm{ctr},\text{out}})=-\frac{1}{V}\sum _{\mathbf{v}=1}^V\{I^{\mathrm{ctr}}(\mathbf{v})\log I^{\mathrm{ctr},\text{out}}(\mathbf{v})+[1-I^{\mathrm{ctr}}(\mathbf{v})]\log [1-I^{\mathrm{ctr},\text{out}}(\mathbf{v})]\},$$ (12)

and the MSE loss is defined as

$$L^{\mathrm{MSE}}(I^{\mathrm{rad}},I^{\mathrm{rad},\text{out}})=\frac{1}{V}\sum _{\mathbf{v}=1}^V{[I^{\mathrm{rad}}(\mathbf{v})-I^{\mathrm{rad},\text{out}}(\mathbf{v})]}^2,$$ (13)

where $V$ is the number of voxels in a volume. To train our model, the Adam optimizer⁵³ is used with a learning rate of 0.001.

2.2.2. Inference

Our SphEsNet is trained on volumes with size of $64\times 64\times 64\text{voxels}$ so an input volume to the model should be $64\times 64\times 64\text{voxels}$. If an input microscopy volume is bigger than $64\times 64\times 64\text{voxels}$ then our model can only process a subvolume of the input with size of $64\times 64\times 64\text{voxels}$. To process the entire input volume, a 3D window is slid with size of $64\times 64\times 64\text{voxels}$, similarly done to our previous work.⁴² First, the input microscopy volume is reflection-padded by 16 voxels. Then, we slide the 3D window to $x$, $y$, and $z$ directions by 32. Note that nuclei, which are partially included on the boundary of the 3D window, may generate inaccurate result. Therefore, only the central volume with size of $32\times 32\times 32\text{voxels}$ is used for our final output. We repeat this until the entire input microscopy volume is processed.

Once the process is done, then we have two output volumes, $I^{\mathrm{ctr},\text{out}}$ and $I^{\mathrm{rad},\text{out}}$. Note $I^{\mathrm{ctr},\text{out}}$ is the output volume from the CTR branch, and $I^{\mathrm{rad},\text{out}}$ is the output volume from the RAD branch. First of all, a local maximum of $I^{\mathrm{ctr},\text{out}}$, ${\mathbf{c}}_k$, is selected as the final center coordinate of the $k$’th nucleus. We avoid selecting more than two center coordinates within a cube with length of $r_{\min }$ because we set the smallest radius of nuclei in a volume is $r_{\min }$. Sometimes, we observed that multiple local maxima with the same intensity are connected as a 3D region. In this case, a centroid of the 3D region is selected as the final center coordinate. After selecting center coordinates of nuclei, the radius for the $k$’th nucleus is estimated as $I^{\mathrm{rad},\text{out}}({\mathbf{c}}_k)+r_{\inf }$ to generate a sphere centered at ${\mathbf{c}}_k$. We observed that the real nuclei in $I^{\mathrm{orig}}$ are not perfectly spherical, and the radius is generally estimated to the shortest distance from the estimated center to the boundary of nuclei. So we increment the estimated radius, $I^{\mathrm{rad},\text{out}}({\mathbf{c}}_k)$, by $r_{\inf }$ to increase the overlapping region between the estimated sphere and the nucleus. In this work, we set $r_{\inf }=1$. As a final step, any other center locations within a sphere are removed because a center location of a nucleus cannot physically be in another nucleus. The final detection volume, $I^{\det }$, is generated after color-coding spheres distinctly to clearly visualize 3D nuclei detection results. The computational time for processing a volume with size of $512\times 512\times 512\text{voxels}$ using our model was approximately 3 min using NVIDIA’s GeForce GTX Titan X.

3. Experimental Results

Our method is tested on five various real fluorescence microscopy data sets consisting of grayscale volumes. Data-I, Data-II, Data-III, and Data-IV are imaged from rat kidney where Hoechst 33342 stain is used to label nuclei. Data-I consists of $Z=512$ images, Data-II of $Z=45$, Data-III of $Z=32$, and Data-IV of $Z=23$, with $X=512\times Y=512$. Data-V is imaged from mouse intestine where DAPI is used to label nuclei, and the size of the volume is $X=512\times Y=930\times Z=157$. The resolution in $z$ direction is different from the resolutions in $x$ and $y$ directions for Data-II. To make the shape of nuclei in Data-II spherical, the original volume is interpolated in $z$ direction by a factor of 2. The range of nuclei radii is 4 to 6 voxels, 8 to 10 voxels, 8 to 12 voxels, 3 to 5 voxels, and 8 to 10 voxels in Data-I, Data-II, Data-III, Data-IV, and Data-V, respectively.

In this work, we trained three models, $M^{\text{Data}\text{-}\mathrm{I}}$, $M^{\text{Data}\text{-}\mathrm{II}}$, and $M^{\text{Data}\text{-}\mathrm{III}}$, where parameters of synthetic volumes to train the models are shown in Table 1. Here, $r_{\min }$, $r_{\max }$, and $N$ are selected based on the minimum radius of a nucleus, the maximum radius of a nucleus, and the density of nuclei presented in an original fluorescence microscopy volume, respectively. The size of nuclei may vary between real fluorescence microscopy volumes. Thus, we trained three models representing three different size ranges of nuclei. We observed that Data-III contains heavy noise so we selected $\lambda =0.1$ to emphasize more on nuclei location to avoid false detection. Data-I was tested on $M^{\text{Data-}\mathrm{I}}$, Data-II on $M^{\text{Data-}\mathrm{II}}$, and Data-III on $M^{\text{Data-}\mathrm{III}}$, respectively. In addition, Data-IV was tested on $M^{\text{Data-}\mathrm{I}}$ and Data-V was tested on $M^{\text{Data-}\mathrm{II}}$ because the size of nuclei in Data-IV is similar to the size of nuclei in Data-I, and the size of nuclei in Data-V is similar to the size of nuclei in Data-II. Figure 4 shows that our method can successfully detect nuclei on various data sets. It is shown that our trained models can detect nuclei on other data sets which were not used during the synthetic volume generation stage.

Table 1.

Parameters of synthetic volumes to train $M^{\text{Data-}\mathrm{I}}$, $M^{\text{Data-}\mathrm{II}}$, and $M^{\text{Data-}\mathrm{III}}$.

Model	$I^{\mathrm{orig}}$	$r_{\min }$	$r_{\max }$	$N$
$M^{\text{Data-}\mathrm{I}}$	Data-I	4	6	1000
$M^{\text{Data-}\mathrm{II}}$	Data-II	8	10	50
$M^{\text{Data-}\mathrm{III}}$	Data-III	8	12	200

Fig. 4.

Nuclei detection of multiple microscopy data sets. (a) $I_{z_{50}}^{\mathrm{orig}}$ of Data-I, (b) $I_{z_{50}}^{\det }$ of Data-I, (c) $I_{z_{16}}^{\mathrm{orig}}$ of Data-II, (d) $I_{z_{16}}^{\det }$ of Data-II, (e) $I_{z_4}^{\mathrm{orig}}$ of Data-III, (f) $I_{z_4}^{\det }$ of Data-III, (g) $I_{z_8}^{\mathrm{orig}}$ of Data-IV, (h) $I_{z_8}^{\det }$ of Data-IV, (i) $I_{z_{132}}^{\mathrm{orig}}$ of Data-V, and (j) $I_{z_{132}}^{\det }$ of Data-V.

We numerically compared our SphEsNet to other nuclei detection methods. An object-based metric is used for our evaluation. First of all, $N_{\mathrm{TP}}$ is the number of true positives defined as detected nuclei overlapping at least 50% of the corresponding ground truth nuclei. Otherwise, the detected nuclei are defined as false positives, and we denote the number of false positives as $N_{\mathrm{FP}}$. Lastly, $N_{\mathrm{FN}}$ is the number of false negatives defined as ground truth nuclei overlapping less than $50\%$ of the corresponding detected nuclei or without having their corresponding detected nucleus. Precision, recall, and $F1$ score, denoted as $P$, $R$, and $F1$, respectively, are defined as^55,56

$$P=\frac{N_{\mathrm{TP}}}{N_{\mathrm{TP}}+N_{\mathrm{FP}}},$$ (14)

$$R=\frac{N_{\mathrm{TP}}}{N_{\mathrm{TP}}+N_{\mathrm{FN}}},$$ (15)

$$F1=\frac{2PR}{P+R}.$$ (16)

We used a subvolume of Data-I with size of $128\times 128\times 64\text{voxels}$ to evaluate multiple nuclei detection methods. A 3D ground truth volume, $I_{(193:\mathrm{320,193}:\mathrm{320,31}:94)}^{\mathrm{gt}}$, was manually generated using ITK-SNAP⁵⁷ and our 3D ground truth volume contains 283 nuclei. We removed components whose number of voxels are less than $50\text{voxels}$ from the ground truth volume and the detection subvolumes to avoid partially included nuclei on the boundary.

We tested multiple training loss functions on our CNN, defined as Eq. (11), by tuning a weight coefficient, $\lambda$. Note that the BCE loss, $L^{\mathrm{BCE}}$, was used for center candidate classification and the MSE loss, $L^{\mathrm{MSE}}$, was used for radius regression. Table 2 shows the comparison between various training loss functions. We observed that precision decreased when $\lambda =0.1$ and recall decreased when $\lambda =10$. We selected $\lambda =1$ as our weight coefficient for the training loss function which gives the highest $F1$ score for Data-I.

Table 2.

Precision, recall, and $F1$ score for various $\lambda$ values for training loss functions for Data-I.

	Precision (%)	Recall (%)	$F1$ score (%)
$\lambda =0.1$	77.51	97.33	86.29
$\mathbf{\lambda }=\mathbf{1}$	84.04	96.63	89.90
$\lambda =10$	84.79	95.24	89.81

Note: Bold characters highlight the best result based on F1 scores.

We compared our method with various methods listed as follows. First, we used watershed technique⁹ which can be used to separate touching nuclei in microscopy volumes. Here, an original volume was binarized by an empirically selected threshold value of 64. Due to inhomogeneous intensity presented in microscopy volumes, we observed that noise in the central region was detected, causing low precision. To correct inhomogeneity, a 3D adaptive histogram equalization⁸ followed by binarization with an empirically selected threshold value of 192 was used as a preprocessing step before watershed technique. We will denote the method of using a 3D adaptive histogram equalization and watershed technique as ${\text{Watershed}}^{+}$. The 3D adaptive histogram equalization helped removing noise in the microscopy volume but non-nuclei subcellular structures were still captured. We observed that overdetection was occurred during watershed due to irregular equalized structures and enhanced noise causing a lower precision. A mask R-CNN³³ is a 2D segmentation method which can both detect and segment objects in distinct labels. A pretrained mask R-CNN with ResNet-50⁵⁸ and a feature pyramid network⁵⁹ was fine-tuned using the same training images used for the SphEsNet. To combine 2D segmentation results into 3D, we used a $z$-directional combination process as a postprocessing step. The $m$’th mask on $I_{z_p}$ whose centroid located at $(x_{m,p},y_{m,p})$ and the $n$’th mask on $I_{z_{p+1}}$ whose centroid located at $(x_{n,p+1},y_{n,p+1})$ were combined as the same nucleus if the distance between $(x_{m,p},y_{m,p})$ and $(x_{n,p+1},y_{n,p+1})$ was less than $\frac{r_{\max }}{2}$. We will denote this method as mask ${\mathrm{R}\text{-}\mathrm{CNN}}_z$. A 2D mask R-CNN cannot utilize depth information, so it performed poor segmentation. In addition, we observed the mask R-CNN was overfitted. We investigated on our previous two-stage method for 3D nuclei detection-segmentation.⁴⁵ We will denote this method as Det-Seg. The 3D segmentation CNN was trained by a set of synthetic volumes generated by blurring and noise operations. The synthetic volumes used in our previous work⁴⁵ were not realistic so segmentation masks were not accurate causing a low precision. More recently, we generated an accurate binary segmentation mask using a 3D CNN trained by a set of synthetic volumes generated by a SpCycleGAN.⁴⁹ To separate touching nuclei in the binary segmentation mask, morphological operations were used as a postprocessing step. We used a 3D erosion, a 3D connected component (for color-coding), and a 3D dilation to separate touching nuclei. A ball with radius of 1 was empirically selected as a structuring element for the 3D erosion and the 3D dilation. We will denote this method as Seg-Morph. We observed that some nuclei were still not separated by morphological operations leading to a low recall. Our method detected the locations of nuclei with their radii accurately so issues caused by overdetection or touching nuclei can be avoided. Note the locations of nuclei and their radii are simultaneously estimated using our SphEsNet without any postprocessing steps. Table 3 shows our method outperforms numerically. The original volume, the ground truth volume, and 3D results are visualized using Voxx⁶⁰ shown in Fig. 5.

Table 3.

Precision, recall, and $F1$ score for multiple nuclei detection methods and our proposed method for Data-I.

	Precision (%)	Recall (%)	$F1$ score (%)
Watershed9	51.14	92.13	65.78
${\text{Watershed}}^{+}$8,9	41.31	97.17	57.97
Mask $\mathrm{R}\text{-}{\mathrm{CNN}}_z$33	20.85	39.20	27.22
Det-Seg45	68.35	90.22	77.78
Seg-Morph49	91.20	82.01	86.36
Our method	84.04	96.63	89.90

Note: Bold characters highlight the best result based on F1 scores.

Fig. 5.

Nuclei detection results for Data-I by a 3D visualization tool, Voxx.⁶⁰ (a) Original microscopy volume, (b) the corresponding ground truth volume, (c) Watershed,⁹ (d) ${\text{Watershed}}^{+}$ composed of a 3D adaptive histogram equalization⁸ and Watershed,⁹ (e) mask ${\mathrm{R}\text{-}\mathrm{CNN}}_z$ composed of a 2D mask R-CNN³³ and a $z$-directional combination process, (f) Det-Seg composed of a 3D detection and segmentation,⁴⁵ (g) Seg-Morph composed of a 3D binary segmentation⁴⁹ and morphological operations, and (h) our proposed method using a SphEsNet.

4. Conclusions

In this paper, we presented a CNN for 3D nuclei detection, called SphEsNet. Training was done using a synthetic training set generated by spatially constrained cycle-consistent adversarial networks without using any real ground truth volumes. Our network can successfully detect locations and sizes of nuclei presented in fluorescence microscopy volumes without any additional postprocessing steps. Our network outperformed other detection/segmentation techniques. We showed that our network can be used on other real fluorescence microscopy volumes when the size of nuclei in an input microscopy volume was similar to the size of nuclei in a training set. One limitation of our work was that the size range of nuclei was an important parameter so that we were not able to train a combined model, which can be used on all testing microscopy volumes. In the future, we plan to extract more characteristics from nuclei in more diverse shapes and sizes.

Biographies

David Joon Ho is currently a machine learning scientist at Memorial Sloan Kettering Cancer Center. He received his BS and MS degrees in electrical and computer engineering from the University of Illinois at Urbana-Champaign in 2010 and 2012, respectively, and his PhD in electrical and computer engineering from Purdue University in 2019. His current research interests include machine learning and computer vision on biomedical images. He is a member of IEEE.

Daniel Mas Montserrat graduated from Polytechnic University of Catalonia in 2015. He is currently a PhD candidate at Purdue University under the supervision of Professor Edward J. Delp. He has worked as a research assistant at the Video and Image Processing Laboratory. His main areas of research are deep learning, image processing, and media forensics.

Chichen Fu received his PhD from Purdue University. He is a video algorithm engineer at Zoom Video Communications. His main research interests include image processing, video processing, and computer vision.

Paul Salama received his PhD in electrical engineering from Purdue University in 1999. In 1999, he joined the Department of Electrical and Computer Engineering, in the Purdue School of Engineering and Technology, at Indiana University-Purdue University Indianapolis, Indianapolis, Indiana, where he is currently a professor of electrical and computer engineering and the associate dean for graduate programs within the school. His research interests include image analysis, image and video compression, machine learning, statistical signal processing, and medical imaging. He is a senior member of IEEE and member of SPIE.

Kenneth W. Dunn received his PhD in biology from SUNY Stony Brook in 1986. He then took a position as an NIH postdoctoral fellow at Columbia University. In 1995, he joined the faculty in the Department of Medicine at Indiana University and was promoted to full professor in 2014. His research is broadly focused on the development and application of methods of microscopy, particularly intravital microscopy to the study of cell biology and physiology.

Edward J. Delp is currently the Charles William Harrison distinguished professor of electrical and computer engineering and a professor of biomedical engineering at Purdue University. His research interests include image and video compression, multimedia security, medical imaging, multimedia systems, communication, and information theory. He has published and presented more than 500 papers. He is a fellow of IEEE, a fellow of SPIE, a fellow of the Society for Imaging Science and Technology (IS&T), and a fellow of the American Institute of Medical and Biological Engineering.

Disclosures

No conflicts of interest, financial or otherwise, are declared by the authors.