Small Blob Detector Using Bi-Threshold Constrained Adaptive Scales

Abstract

Recent advances in medical imaging technology bring great promises for medicine practices. Imaging biomarkers are discovered to inform disease diagnosis, prognosis, and treatment assessment. Detecting and segmenting objects from images are often the first steps in quantitative measurement of these biomarkers. The challenges of detecting objects in images, particularly small objects known as blobs, include low image resolution, image noise and overlap among the blobs. This research proposes a Bi-Threshold Constrained Adaptive Scale (BTCAS) blob detector to uncover the relationship between the U-Net threshold and the Difference of Gaussian (DoG) scale to derive a multi-threshold, multi-scale small blob detector. With lower and upper bounds on the probability thresholds from U-Net, two binarized maps of the distance are rendered between blob centers. Each blob is transformed to a DoG space with an adaptively identified local optimum scale. A Hessian convexity map is rendered using the adaptive scale, and the under-segmentation typical of the U-Net is resolved. To validate the performance of the proposed BTCAS, a 3D simulated dataset (n=20) of blobs, a 3D MRI dataset of human kidneys and a 3D MRI dataset of mouse kidneys, are studied. BTCAS is compared against four state-of-the-art methods: HDoG, U-Net with standard thresholding, U-Net with optimal thresholding, and UH-DoG using precision, recall, F-score, Dice and IoU. We conclude that BTCAS statistically outperforms the compared detectors.

I. Introduction

Imaging biomarkers play a significant role in medical diagnostics and in monitoring disease progression and response to therapy [1]–[5]. The development and validation of imaging biomarkers involves the detection, segmentation and classification of imaging features. Deep learning tools have been recently developed to perform these functions. For example, convolutional neural networks (CNNs) have been applied to magnetic resonance (MR) and X-ray computed tomography (CT) images [6]–[11] and recurrent neural networks (RNN) have been applied to functional and molecular images such as positron emission tomography (PET) [12]–[14] for image classification. A convolutional RNN model was designed to detect mitosis from cell videos [15], and a CNN model was developed to generate probability maps to initialize and model cell nuclear shape and fine-tune nuclei for segmentation in optical images [16]. An ensemble of CNN models with differently sized filters was used to detect pulmonary nodules in CT images [17]. However, deep learning tools are strongly affected by the quality of the images.

Recently, imaging tools have been developed to precisely map and measure individual glomeruli in the kidney using an injected contrast agent, (cationic ferritin, CF), which binds to the glomerular basement membrane and creates a dark spot in gradient-echo MR images [18]–[20]. The emerging field of CFE-MRI provides comprehensive, 3D measurements of histologic features of the kidney that may aid in early detection of kidney pathology [21]–[23]. Glomeruli appear as small blobs in CFE-MR images. A number of blob detectors have been developed for small blob detection such as nuclei detection [24], [25], cell detection [26]–[28], among which scale-space based blob detectors have attracted great attention. For example, Kong et al. proposed the generalized Laplacian of Gaussian (gLoG) [29], which accurately detected blobs of various scales, shapes and orientations from histologic and fluorescent microscopic images. Zhang et al. developed the Hessian-based blob detectors HLoG [30] and HDoG [31] to automatically detect glomeruli in CFE-MR images with high accuracy and efficiency. However, these blob detectors are not robust to noise [32], leading to high false positive rates. Deep learning has recently been applied to detect and segment blob-like objects. One approach is to apply a CNN to identify patches enclosing objects first, and then perform post-processing to segment the objects. For example, Ciresan et al. [33] applied a CNN to automatically detect cells in histologic images of breast cancer. Based on the probable location of the centroid of the cell, derived from the CNN, they used non-maxima suppression to identify the cells. Images can also be first pre-processed and then divided into patches, which are then confirmed by a CNN. For example, Khoshdeli et al. [34] used non-negative matrix factorization and a Laplacian of Gaussian (LoG) filter to initially identify blobs, and used a CNN model to detect nuclei. Fully convolutional networks (FCNs) [35] have been proposed for segmentation at the pixel or voxel level. FCNs transfer the CNN’s fully connected layers to convolutional layers, providing a map at pixel or voxel scale to detect the object [36]. However, FCNs often require large datasets for training, limiting their potential use in medical applications where sample sizes are often small. To resolve this issue, U-Net [37], a modified version of FCN, was developed to achieve fast and accurate segmentation [38]–[40].

CFE-MRI provides unique challenges for image segmentation to identify and measure individual glomeruli in the kidney. First, the size of a glomerulus is on the order of the image voxel (~ 100 μm) and the spatial frequency of glomeruli is close to that of image noise, requiring algorithms with good de-noising ability. Since most existing blob detectors suffer from over-detection, post-pruning is often employed to correct false positives [30]. Second, a large fraction of glomeruli overlap in the images. A single threshold applied to the probability map derived from U-Net may not separate overlapping glomeruli, leading to under-segmentation and a high false negative rate. To address both over-detection and under-segmentation, the UH-DoG detector was proposed to take advantage of the complementary properties of U-Net and HDoG [40]. The probability map from U-Net provides blob likelihood in the whole image, and the Hessian map from HDoG indicates local convexity among a group of neighborhood pixels or voxels. Joining the two maps was initially promising for detecting glomeruli [40]. However, UH-DoG employs a single threshold-single scale approach, which may pose the following challenges: (1) A single threshold applied to the U-Net probability map may not be sensitive to noise to minimize under-segmentation. (2) A single optimum scale applied to the DoG space may overlook large variations in blob size. One possible solution is to exhaustively explore multiple thresholds in U-Net and multiple scales in DoG. Unfortunately, the massive number of glomeruli (> 1 million in a human kidney) with varying sizes makes such attempts computationally prohibitive.

This work aims to uncover the relationship between the U-Net threshold and the DoG scale to derive a multi-threshold, multi-scale small blob detector. We first prove the monotonicity of the U-Net probability map, laying the foundation for the proposed detector. With lower and upper bounds on the probability thresholds, we then render two binarized maps of the distance between blob centers. Since the true blob will fall between the two distance maps with a specified level of certainty, the search space for the DoG scales is bounded. Each blob can then be transformed to an optimum local DoG space locally, instead of by a single global optimum scale. A Hessian convexity map is rendered using an adaptive scale, and the under-segmentation typical of the U-Net is resolved. We term this approach the Bi-Threshold Constrained Adaptive Scale (BTCAS) blob detector. To validate the performance of BTCAS blob detector, we first study a 3D simulated dataset (n=20) where the locations of blobs are known. Four methods are chosen from the literature: HDoG [31], U-Net with standard thresholding [37], U-Net with optimal thresholding [32], and UH-DoG [40] for comparison. Next, we compare blob detection using these methods applied to a 3D image of three human kidneys and a set of 3D image of mouse kidneys from CFE-MRI against the HDoG, UH-DoG and stereology.

II. Methods

Our proposed Bi-Threshold Constrained Adaptive Scale (BTCAS) blob detector consists of two steps to detect blobs (glomeruli) from CFE-MRI of the kidney: (1) Training U-Net to generate a probability map to detect the centroids of the blobs, and then deriving two distance maps with bounded probabilities; (2) Applying the Difference of Gaussian (DoG) with an adaptive scale constrained by the bounded distance maps, followed by Hessian analysis for final blob segmentation.

A. Bi-Threshold Distance Maps from U-Net

U-Net consists of an encoding path (left) and a decoding path (right), (Fig. 1). The encoding path has four blocks. Within each block, there are two 3×3 convolutional layers (Conv 3×3), a rectified linear unit (ReLU) layer, and a 2×2 max-pooling layer (Max pool 2×2). After each max-pooling layer, the resolution of the feature maps is halved and the channel is doubled. The input images are compressed by layer, through the encoding path. The corresponding decoding path performs the inverse operation to reconstruct the output as a probability map of the same size as the input images. The resolution is increased by layer through the decoding path. To transfer information from the encoding path to the decoding path, concatenation paths are added between them, marked by black arrows in Fig. 1. The final layer is a 1×1 convolutional layer, followed by a sigmoid function. This sigmoid function ensures that the resultant output is a probability map. In supervised learning applications where the output labeling is known, U-Net can be directly used as a model for segmentation. When the output labeling is unknown, U-Net can be used to process and denoise the images [41]–[44]. Here, since the ground truth is unknown, we investigate the denoising capabilities of U-Net. It is common to use autoencoders to denoise images. However, in CFE-MRI of the kidney, the glomeruli are extremely small, similar to noise that can be potentially removed by autoencoders. The major difference between U-Net and autoencoders is that U-Net has concatenation paths, which can transfer fine-grained information from low layers to high layers to increase the performance of the segmentation results. Therefore, U-Net may have the advantage over autoencoder model by removing background noise from the MR images and simultaneously enhancing the glomerular detection.

Fig. 1.

Architecture of U-Net Model.

Let $X\in {[0,1]}^{I_1\times I_2\times I_3}$ be the input image and $Y\in {[0,1]}^{I_1\times I_2\times I_3}$ be the image after being denoised. For simplicity, assume input image X has Gaussian noise ε. We have

$$X=Y+\epsilon ,\epsilon ~\mathcal{N}\left(0,{\sigma }^{2}I\right).$$ (1)

U-Net is to obtain a function $\mathcal{F}(\cdot )$ mapping X to Y by learning and optimizing the parameters Θ of convolutional and deconvolutional kernels. This is achieved by minimizing the global loss function:

$$\mathcal{L}(\Theta )=\frac{1}{N}{\sum }_{i=1}^n\mathit{loss}(Y,\mathcal{F}(X;\Theta )),$$ (2)

Where N is the sample size, $\mathcal{F}(X;\Theta )\in {[0,1]}^{I_1\times I_2\times I_3}$ is the probability map followed by the sigmoid activation function, loss(·) is a binary cross entropy loss function defined as:

$$\begin{gathered} loss(Y,\mathcal{F}(X;\Theta ))=-\frac{1}{I_1{I}_2{I}_3}{\sum }_{k=1}^{I_1\times I_2\times I_3}{y}_k\cdot \log \left({\mathcal{F}}_k(X;\Theta )\right)+ \\ \left(1-y_k\right)\cdot \mathit{log}\left(1-{\mathcal{F}}_k(X;\Theta )\right), \end{gathered}$$ (3)

where $y_k\in \{0,1\}$ is the true label and ${\mathcal{F}}_k(X;\Theta )\in [0,1]$ is the predicted probability for voxel k. After denoising, the output of $\mathcal{F}(\cdot )$ approximates Y:

$$\mathcal{F}(X;\Theta )\approx Y.$$ (4)

Glomeruli in CFE-MR images are roughly spherical in shape, with varying image magnitudes. Based on this observation, we develop the Proposition 1 in Appendix.

The first use of Proposition 1 is to identify the centroid of any blob. From Proposition 1, the centroid of any bright blob reaches maximum probability. Therefore, a regional maximum function RM can be applied to the probability map U(x, y, z) to find voxels with maximum probability from the connected neighborhood voxels as blob centroids:

$$RM(U)=\underset{\begin{array}{c}u\in U(x,y,z)\\ \Delta u\in (-k,k)\end{array}}{\min }U(u+\mathrm{\Delta }u)\},$$ (5)

where k is the Euclidean distance between each voxel with its neighborhood voxels. The blob centroid set $C={\left\{C_i\right\}}_{i=1}^N$ is defined as:

$$C=\{(x,y,z)\mid (x,y,z)\in \arg RM(U(x,y,z))\}.$$ (6)

Here, k = 1. Each blob centroid C_i ∈ C has maximum probability within 6-connected neighborhood voxels.

A second use of Proposition 1 is to binarize the probability map with a confidence level. We first use Otsu’s thresholding [45] to remove noise and voxels in the blob centroids, and to extract the probability distribution of blob voxels. Next, instead of using single threshold, we apply the two-sigma rule to the distribution to identify the lower probability δ_L and higher probability δ_H covering 95% range of the probabilities. As a result, the probability map can be binarized to $B_L(x,y,z)\in {\{0,1\}}^{I_1\times I_2\times I_3}$ and $B_H(x,y,z)\in {\{0,1\}}^{I_1\times I_2\times I_3}$.

$$B_{L/H}(x,y,z)=\{\begin{array}{ll}1,\hfill & U(x,y,z)\ge {\delta }_{L/H}\hfill \\ 0,\hfill & U(x,y,z)<{\delta }_{L/H}\hfill \end{array}$$ (7)

From Proposition 2, B_L(x, y, z) will approximate a blob with larger size and B_H(x, y, z) will approximate a blob with smaller size. Without loss of generality, let B(x, y, z) be a binarized probability map and define Ω = {(x, y, z)|B(x, y, z) = 1} as the set of blob voxels and ∂Ω the set of boundary voxels. d(·) is the Euclidean distance function of any two voxels. The Euclidean distance of each voxel with the nearest boundary voxels is:

$$d(p,\partial \Omega )=\underset{\begin{array}{c}p\in \Omega \\ q\in \partial \Omega \end{array}}{\min }d(p,q).$$ (8)

Given B_L(x, y, z) and B_H(x, y, z), two distance maps are derived, $D_L(x,y,z)\in R^{I_1\times I_2\times I_3}$ and $D_H(x,y,z)\in R^{I_1\times I_2\times I_3}$ respectively. Fig. 2 illustrates the process from the probability map, Bi-Threshold (lower and upper bound) to binarized distance maps.

Fig. 2.

Approach to derive the distance maps from probability map: (a) probability distribution of probability map. (b) visualization of probability map. (c) probability distribution after applying Otsu’s thresholding. (d) visualization of blob’s probability. (e) binarized probability map B_L under low threshold δ_L. (f) binarized probability map B_H under low threshold δ_H. (g) distance map D_L derived from B_L. (h) distance map D_H derived from B_H

For each blob centroid C_i ∈ C (see (9)), we approximate radius r_i of blob i as:

$$r_i\in \left(D_H\left(C_i\right),D_L\left(C_i\right)\right).$$ (9)

As proved in [46], the smoothing scale in DoG is positively correlated with the blob radius. Here we will use this bounded radius information in (9) to constrain the adaptive scales in DoG imaging smoothing, as described in the next section.

B. Bounded Adaptive Scales in DoG and Hessian Analysis

For a normalized 3D image $X(x,y,z)\in {[0,1]}^{I_1\times l_2\times I_3}$, a DoG filter is

$$DoG(x,y,z;s)=X(x,y,z)*\frac{(G(x,y,z;S+\mathrm{\Delta }s)-G(x,y,z;s))}{\mathrm{\Delta }s},$$ (10)

where s is the scale value, ∗ is convolution operator, and Gaussian kernel $G(x,y,z;s)=\frac{1}{{\left(2\pi s^2\right)}^{\frac{3}{2}}}{e}^{-\frac{\left(x^2+y^2+z^2\right)}{2s^2}}$. The DoG filter smooths the image more efficiently in 3D than the LoG filter does [31]. However, determining the optimum DoG scale in blob detection is challenging. Zhang et al. [30] proposed to use a single global optimal scale for all blobs by identifying the maximum DoG in the whole image. This approach guarantees efficient smoothing when all blobs are similar in size. For blobs with a wide range of sizes, if blob size is smaller than the DoG scale, the blob will be smoothed; if the blob size is larger than the DoG scale, only part of the blob will be smoothed. Adaptive scales have been proposed to alleviate this issue. One example is from Yousef et al. [47] where a distance map was generated from the binarized map and graph-cut was applied to constrain the range of LoG scale. With the constrained range, each blob has an optimum scale when the LoG is at a local maximum. The authors acknowledged that one potential issue from graph-cut is under-segmentation [47]. In addition, the LoG is less computationally affordable compared to the DoG for 3D images.

Recognizing the merits and the challenges from the adaptive scales developed in [47]. Here we apply the distance maps (D_L and D_H) from U-Net to constrain the DoG scale for scale inference. Specifically, for a d-dimensional images, the DoG will reach a maximum response under scale $s=r/\sqrt{\text{d}}$ [46]. In a 3D image, let the range of scale for each blob be $s_i\in \left(s_i^L,s_i^H\right)$. By substituting r with (9), we get:

$$s_i^L=D_H\left(C_i\right)/\sqrt{3}$$ (11)

$$s_i^H=D_L\left(C_i\right)/\sqrt{3}$$ (12)

For each blob, a normalized DoG_nor(x, y, z; s_i) with multi-scale $s_i\in \left(s_i^L,s_i^H\right)$ is applied on a small 3D window with size N × N × N (N > 2 ∗ D_L(C_i)) and window center is the blob centroid C_i ∈ C. For each voxel (x, y, z) in DoG_nor(x, y, z; s_i) at scale s_i, the Hessian matrix for this voxel is:

$$\begin{array}{l}H(Do{G}_{nor}(x,y,z;s_i)\\ =\left[\begin{array}{ccc}\frac{{\partial }^{2}Do{G}_{nor}\left(x,y,z;s_i\right)}{\partial x^2}& \frac{{\partial }^{2}Do{G}_{nor}\left(x,y,z;s_i\right)}{\partial x\partial y}& \frac{{\partial }^{2}Do{G}_{nor}\left(x,y,z;s_i\right)}{\partial x\partial z}\\ \frac{{\partial }^{2}Do{G}_{nor}\left(x,y,z;s_i\right)}{\partial x\partial y}& \frac{{\partial }^{2}Do{G}_{nor}\left(x,y,z;s_i\right)}{\partial y^2}& \frac{{\partial }^{2}Do{G}_{nor}\left(x,y,z;s_i\right)}{\partial y\partial z}\\ \frac{{\partial }^{2}Do{G}_{nor}\left(x,y,z;s_i\right)}{\partial x\partial z}& \frac{{\partial }^{2}Do{G}_{nor}\left(x,y,z;s_i\right)}{\partial y\partial z}& \frac{{\partial }^{2}Do{G}_{nor}\left(x,y,z;s_i\right)}{\partial z^2}\end{array}\right]\end{array}$$ (13)

In a normalized DoG-transformed 3D image, each voxel of a transformed bright or dark blob has a negative or positive definite Hessian [31]. Taking a bright blob as an example, we define the Hessian convexity window, HW(x, y, z; s_i), a binary indicator matrix:

$$\begin{gathered} HW\left(x,y,z;s_i\right)= \\ \{\begin{array}{ll}1\hfill & H\left(Do{G}_{nor}\left(x,y,z;s_i\right)\right)isnegativedefinite\hfill \\ 0\hfill & otherwise\hfill \end{array} \end{gathered}$$ (14)

For each blob with centroid C_i ∈ C, let the average DoG value for each window BW_DoG be:

$$B{W}_{DoG}\left(s_i\right)=\frac{{\sum }_{(x,y,z)}\mathit{DoG}(x,y,z)HW\left(x,y,z;s_i\right)}{{\sum }_{(x,y,z)}HW\left(x,y,z;s_i\right)}.$$ (15)

The optimum scale $s_i^{*}$ for each blob is determined if $B{W}_{\mathit{DoG}}\left(s_i^{*}\right)$ is maximum with $s_i\in \left(s_i^L,s_i^H\right)$. We derive the optimum scale $s_i^{*}$ for each blob with centroid C_i ∈ C. The final segmented blob set S_blob is:

$$\begin{array}{l}{S}_{blob}=\{(x,y,z)\mid (x,y,z)\in \\ Do{G}_{nor}\left(x,y,z;s_i^{*}\right),HW\left(x,y,z;s_i^{*}\right)=1\}.\end{array}$$ (16)

The details of proposed BTCAS blob detector are summarized in Table I.

Table I.

Pseudocode for BTCAS Blob Detector

1. Use a pretrained model to generate a probability map of blobs from original image.

2. Initialize probability range (δL, δH) and thresholding probability map to get binarized map B(x, y, z) and distance map D(x, y, z)

3. Calculate the blob centroids set C from probability map U(x, y, z). For each blob with centroid Ci ∈ C, get the scale range $\left(s_i^L,s_i^H\right)$.

4. For each blob with centroid Ci ∈ C, transform raw image window of blob to multi-scale DoG space with scale $s_i\in \left(s_i^L,s_i^H\right)$.

5. Calculate the Hessian matrix based on normalized DoG smoothed window and generate the Hessian convexity window HW(x, y, z; si).

6. Calculate average DoG intensity of each window $B{W}_{DoG}(s)=\frac{{\Sigma }_{(x,y,z)}DoG(x,y,z)HW(x,y,z;s)}{{\Sigma }_{(x,y,z)}HW(x,y,z;s)}$ and find the optimum scale for each blob by $s_i^{*}=argmaxB{W}_{DoG}\left(s_i\right)$

7. Get the optimum Hessian convexity window HW(x, y, z; $HW\left(x,y,z;s_i^{*}\right)$) under scale $s_i^{*}$.

8. Identify the final segmented blob voxels set Sblob.

III. Experiments and Results

A. Training Dataset and Data Augmentation

We used a public dataset [48] of optical images of cell nuclei to train U-Net. This dataset contains 141 pathology images (2,000 × 2,000 pixels). The 12,000 ground truth annotations were provided by a domain expert, which involved delineating object boundaries over 40 hours. Since we aimed to facilitate U-Net to denoise our blobs images based on the ground truth labeled images, we generated Gaussian distributed noise with μ_niose = 0 and ${\sigma }_{noise}^2=0.01$, which were added to the labeled images, resulting in 141 synthetic training images as shown in Fig. 3 (g–i). Data were augmented to increase the invariance and robustness of U-Net. We generated the augmented data by a combination of rotation shift, width shift, height shift, shear, zoom, and horizontal flip. The trained model is validated using 3D synthetic image data and 3D MR image data.

Fig. 3.

U-Net training dataset: (a-c) original images. (d-f) ground truth labeled images for (a-c). (g-i) synthetic training images based on (d-f).

B. Experiment I: Validation Experiments using 3D Synthetic Image Data

We simulated 20, 3D images with 10 different numbers of blobs and two different levels of noise. From each 3D image (sized 256×256×256), blobs were generated using the Gaussian function with parameter s = 1 for blob size. The radii of the blobs was approximated as (2 × s + 0.5) voxels, based on observation. Blobs were spread on the images at random locations. The number of blobs (N) ranged from 5,000 to 50,000 with a step size of 5,000. Noise was generated by the Gaussian function with μ_niose = 0 and ${\sigma }_{noise}^2$ defined by:

$${\sigma }^2{}_{noise}=\frac{{\sigma }^2{}_{image}}{{10}^{\frac{SNR}{10}}}.$$ (17)

The signal-to-noise ratio (SNR) was set at 1dB and 5dB for high noise and low noise, respectively. As the quantity of blobs increased, so did blob density, which resulted in a large number of blobs being closely clumped together (see Fig. 4). We derived the ratio of overlap (O) of blobs in the 3D image:

$$O=\frac{N_O}{N_T}.$$ (18)

Fig. 4.

The 3D synthetic images dataset in Experiment I. Slice 100 (of 256) from simulated 3D blob images with different parameter settings on the number of blobs and signal-to-noise ratio (SNR)(dB) (a) 3D blob image with N = 5,000 and SNR = 1dB, O = 0.04; (b) 3D blob image with N = 10,000 and SNR = 5dB, O = 0.07; (c) 3D bob image with N = 20,000 and SNR = 5dB, O = 0.14; and (d) 3D blob image with N = 50,000 and SNR = 1dB, O = 0.31.

Five methods were applied to the synthetic 3D blob images: the HDoG [31], U-Net with standard thresholding [37], U-Net with optimal thresholding (OT U-Net) [32], the UH-DoG [40], and our proposed BTCAS blob detector. The parameter settings of the DoG were as follows: window size N was 7. γ was 2. Δs was 0.001. To denoise the images of the 3D blobs using a trained U-Net, we first resized each 256×256 slice to 512×512 and each slice was fed into U-Net. We used the Adam optimizer in U-Net with a learning rate set to 0.0001. The dropout rate was set to 0.5. The threshold for the U-Net probability map in UH-DoG was set to 0.5. U-Net was implemented on a NVIDIA TITAN XP GPU with 12 GB of memory. Here we used a 2D U-Net and 2D probability maps were rendered on each slice then stacked together to form a 3D probability map.

C. Evaluating the Number of Blobs Detected

First, we compared the number of blobs detected from different algorithms and noisy image (Fig. 5) settings. The HDoG suffered from significant over-detection, yielding a high error rate in both experiments. In other methods, for the experiment on images with low noise, as the number of true blobs increased from 5,000 to 50,000, error rates for the U-Net, OT U-Net, and UH-DoG ranged from 4.96–38.78%, 4.28–32.22%, and 1.36–12.60% respectively. Our proposed BTCAS’s error rates were significantly lower, ranging from 0.06–1.44%. For the experiment using images with high noise, as the number of true blobs increased from 5,000 to 50,000, error rates for the U-Net, OT U-Net, UH-DoG ranged from 4.68–39.87%, 4.08–32.96%, 1.38–12.79%. BTCAS had error rates of 0.08–10.20%. By integrating U-Net, the detection error decreased, and over-detection was reduced. However, both U-Net and OT U-Net detected fewer blobs than the ground truth. This can be explained by overlapping blobs; If the probability values at the boundaries of overlapping blobs are larger than the threshold, under-segmentation occurs, leading to fewer detected blobs. OT U-Net used Otsu’s thresholding to find the optimal threshold to somewhat reduce under-segmentation. With Hessian analysis, under-segmentation can be eliminated. The UH-DoG and BTCAS outperformed both U-Net and OT U-Net. The error rate of BTCAS slowly increased when the number of blobs increased from 5,000 to 50,000 with low noise and from 5,000 to 40,000 with high noise. Although the error rate of BTCAS increased when the number of blobs increased from 40,000 to 50,000 under high noise, this error rate was significantly lower than for UH-DoG. We conclude that BTCAS is much more robust in the presence of noise compared to the other four methods.

Figure 5.

A. Comparison of blob detection error rate (%) of HDoG, U-Net, OT U-Net, UH-DoG and BTCAS in 3D synthetic blob images with low noise (SNR = 5DB). Number of true blobs (overlap ratio) ranges from 5000 (0.04) to 50000 (0.31). B. Comparison of blob detection error rate (%) of HDoG, U-Net, OT U-Net, UH-DoG and BTCAS in 3D synthetic blob images with high noise (SNR = 1DB). Number of true blobs (overlap ratio) ranges from 5000 (0.04) to 50000 (0.31).

D. Evaluating Blob Detection and Segmentation Accuracy

Next we evaluated algorithm performance by precision, recall, F-score, Dice coefficient, and Intersection over Union (IoU). For detection, Precision measures the fraction of retrieved candidates confirmed by the ground-truth. Recall measures the fraction of ground-truth data retrieved. F-score is an overall performance of precision and recall. For segmentation, the Dice coefficient measures the similarity between the segmented blob mask and the ground truth. IoU measures amount of overlap between the segmented blob mask and the ground truth. Ground truth voxels and blob locations (the coordinates of the blob centers) were already generated when synthesizing the 3D blob images. A candidate was considered as a true positive if the centroid of its magnitude was in a detection pair (i, j) for which the nearest ground truth center j had not been paired and the Euclidian distance D_ij between ground truth center j and blob candidate i was less than or equal to d. To avoid duplicate counting, the number (#) of true positives TP was calculated by (19). Precision, recall, F-score were calculated by (20), (21), (22).

$$\begin{gathered} TP=\min \{\#\left\{(i,j):{\min }_{i=1}^m{D}_{ij}\le d\right\},\#\{(i,j):{\min }_{j=1}^n{D}_{ij}\le \\ d\left\}\right\}, \end{gathered}$$ (19)

$$precision=\frac{TP}{n},$$ (20)

$$recall=\frac{TP}{m}\text{,}$$ (21)

$$F-score=2\times \frac{precision\times recall}{\text{(}precision\text{+}recall)},$$ (22)

where m is the number of true glomeruli and n is the number of blob candidates; d is a thresholding parameter set to a positive value (0, +∞). If d is small, fewer blob candidates are counted since the distance between the blob candidate centroid and ground-truth should be small. If d is too large, more blob candidates are counted. Here, since local intensity extremes could be anywhere within a small blob with an irregular shape, we set d to the average diameter of the blobs: $\sqrt{\frac{{\sum }_{(x,y)}I(x,y;s)}{\pi }}$. The Dice coefficient and IoU were calculated by comparing the segmented blob mask and ground truth mask by (23) and (24).

$$Dice(B,G)=\frac{2|B\cap G|}{\left|B\right|+\left|G\right|},$$ (23)

$$IoU(B,G)=\frac{B\cap G}{B\cup G},$$ (24)

where B is the binary mask for segmentation result and G is the binary mask for the ground truth.

Comparisons between the models are shown in Tables II and III. ANOVA test was performed with Tukey’s HSD multi-comparison at significance level 0.05. BTCAS significantly outperforms other four methods on Recall, F-Score for images with low and high noises. Compared to UH-DoG, BTCAS provides better performance on Recall, F-Score and is comparable on Precision, Dice and IoU. In this synthetic data, the blobs were generated with similar size (s = 1); we can thus still conclude that BTCAS can resolve under-segmentation by U-Net.

Table II.

Comparison (avg ± std) and ANOVA using Tukey’s HSD pairwise test of BTCAS, HDoG, UH-DoG, U-Net, OT U-Net on 3D synthetic images under SNR = 5db (low noise)

Metrics	BTCAS	HDoG	U-NET	OT U-NET	UH-DoG
Precision	1.00±0.00	0.10±0.07 (*< 0.0001)	0.98±0.01 (*<0.0001)	1.00±0.00 (*<0.0001)	1.00±0.00 (0.172)
Recall	0.99±0.00	0.99±0.01 (* 0.041)	0.76±0.12 (*< 0.001)	0.81±0.09 (*<0.0001)	0.93±0.04 (*<0.001)
F-score	1.00±0.00	0.18±0.11 (*<0.0001)	0.85±0.08 (*< 0.001)	0.89±0.06 (*<0.001)	0.96±0.02 (*<0.001)
Dice	0.96±0.03	0.26±0.14 (*<0.0001)	0.52±0.00 (*< 0.0001)	0.60±0.04 (*<0.0001)	0.97±0.02 (*<0.0001)
IoU	0.92±0.05	0.16±0.09 (*<0.0001)	0.35±0.00 (*< 0.0001)	0.43±0.04 (*<0.0001)	0.94±0.04 (*<0.0001)

^*significance p < 0.05

Table III.

Comparison (avg ± std) and ANOVA using Tukey’s HSD pairwise test of BTCAS, HDoG, UH-DoG, U-Net, OT U-Net on 3D synthetic images under SNR = 1db (high noise)

Metrics	BTCAS	HDoG	U-NET	OT U-NET	UH-DoG
Precision	0.98±0.03	0.09±0.06 (*< 0.0001)	0.98±0.01 (0.338)	1.00±0.00 (0.063)	1.00±0.00 (* 0.035)
Recall	0.99±0.00	0.99±0.01 (* 0.026)	0.76±0.12 (*<0.001)	0.81±0.10 (*<0.001)	0.93±0.04 (*<0.001)
F-score	0.99±0.02	0.17±0.10 (*<0.0001)	0.85±0.08 (*<0.001)	0.89±0.06 (*<0.0001)	0.96±0.02 (*<0.001)
Dice	0.92±0.08	0.26±0.13 (*<0.0001)	0.51±0.01 (*<0.0001)	0.61±0.03 (*<0.0001)	0.94±0.04 (0.063)
IoU	0.85±0.13	0.15±0.09 (*<0.0001)	0.34±0.00 (*<0.0001)	0.44±0.03 (*<0.0001)	0.89±0.07 (0.061)

^*significance p < 0.05

E. Experiment II: Validation using 3D Human Kidney CFE-MR Images

In this experiment we investigated blob segmentation applied to 3D CFE-MR images to measure number (Nglom) and apparent volume (aVglom) of glomeruli in healthy and diseased human donor kidneys that were not accepted for transplant. Three human kidneys were obtained at autopsy through a donor network (The International Institute for the Advancement of Medicine, Edison, NJ) after receiving Institutional Review Board (IRB) approval and informed consent from Arizona State University [21]. They were imaged by CFE-MRI as described in [18]–[21].

Each human MR image has pixel dimensions of 896×512×512. We applied the HDoG, UH-DoG and proposed BTCAS blob detector to segment glomeruli. The parameter settings of DoG are as follows: window size N = 7. γ = 2. Δs = 0.001. We first generated 14,336 2D patches, with each patch 128×128 in size and each patch was then fed into U-Net. The threshold for the U-Net probability map in UH-DoG was 0.5. We performed quality control by visually checking the identified glomeruli, visible as black spots in the images. For illustration, example results from CF2 which has more heterogenous pattern are shown in Fig. 6. As seen, the BTCAS blob detector performed better than the HDoG and the UH-DoG in segmentation. Several example glomeruli are marked with orange, green and blue circles. In Fig. 6(e–h), orange circles show that some noise is detected as false positives by the HDoG and UH-DoG, but the BTCAS blob detector performed well using the denoising provided by U-Net. Green circles show some blobs that are under-segmented in UH-DoG due to the fixed probability threshold. The BTCAS blob detector captured these. Blue circles show that for blobs with a range of sizes, the BTCAS blob detector delineated all voxels of blobs with the adaptive optimum DoG scale.

Fig. 6.

Glomerular segmentation results from 3D MR images of human kidney (CF2 slice 256). (a) Original magnitude image. (b) Glomerular segmentation results of HDoG. (c) Glomerular segmentation results of UH-DoG. (d) Glomerular segmentation results of BTCAS blob detector. (e-h) Magnified regions (yellow box) from (a-d).

Nglom and aVglom are reported in Tables IV and V, where the HDoG, UH-DoG and proposed BTCAS blob detector are compared to data from unbiased dissector-fractionator stereology. representing a ground truth in the average measurements in each kidney. We used the stereology data from [21] and the method of calculating aVglom from [19]. The differences between the results of the HDoG, UH-DoG, BTCAS methods and stereology data are also listed in Tables IV and V. Compared to stereology, the HDoG identified more glomeruli and the difference with stereology is much larger than the other two methods, indicating over-detection under the single optimal scale of DoG and lower mean aVglom than stereology. UH-DoG identified fewer glomeruli due to under-segmentation when using the single thresholding (0.5) on the probability map of U-Net combined with the Hessian convexity map. BTCAS provided the most accurate measurements of Nglom and mean aVglom than the other two methods.

Table IV.

Human Kidney glomerular segmentation (Nglom) from CFE-MRI using HDoG, UH-DoG and the proposed BTCAS blob detectors compared to dissector-fractionator stereology

Human Kidney	Nglom (× 106) (Stereology)	Nglom (× 106) (BTCAS)	Difference Ratio (%)	Nglom (× 106) (UH-DoG)	Difference Ratio (%)	Nglom (× 106) (HDoG)	Difference Ratio (%)
CF 1	1.13	1.16	2.65	0.66	41.60	2.95	>100
CF 2	0.74	0.86	16.22	0.48	35.14	1.21	63.51
CF 3	1.46	1.50	2.74	0.85	41.78	3.93	>100

Table V.

Human Kidney glomerular segmentation from CFE-MRI (mean aVglom) using HDoG, UH-DoG and the proposed BTCAS blob detectors compared to dissector-fractionator stereology

Human Kidney	Mean aVglom(× 10−3mm3) (Stereology)	Mean aVglom (× 10−3mm3) (BTCAS)	Difference Ratio (%)	Mean aVglom(× 10−3mm3) (UH-DoG)	Difference Ratio (%)	Mean aVglom (× 10−3mm3) (HDoG)	Difference Ratio (%)
CF 1	5.01	5.32	6.19	7.36	46.91	4.8	4.19
CF 2	4.68	4.78	2.14	5.62	20.09	3.2	31.62
CF 3	2.82	2.55	9.57	3.73	32.37	3.2	13.48

F. Experiment III: Validation using 3D Mouse Kidney CFE-MR Images

We conducted experiments on CF-labeled glomeruli from a dataset of 3D MR images to measure Nglom and aVglom of glomeruli in healthy and diseased mouse kidneys. This dataset includes chronic kidney disease (CKD, n=3) vs. controls (n=6), acute kidney injury (AKI, n=4) vs. control (n=5). The animal experiments were approved by the Institutional Animal Care and Use Committee (IACUC) under protocol #3929 on 04/07/2020 at the University of Virginia, in accordance with the National Institutes of Health Guide for the Care and Use of Laboratory Animals. They were imaged by CFE-MRI as described in [49].

Each MRI image has pixel dimensions of 256×256×256. We applied the HDoG, HDoG with VBGMM, UH-DoG and proposed BTCAS blob detector to segment glomeruli. The parameter settings of DoG were: window size N = 7. γ = 2. Δs = 0.001. To denoise the 3D blob images by using trained U-Net, we first resized each slice to 512×512 and each slice was fed into U-Net. The threshold for the U-Net probability map in UH-DoG was 0.5.

Nglom and mean aVglom are reported in Table VI and Table VII, where the HDoG, UH-DoG and proposed BTCAS blob detector are compared to HDoG with VBGMM from [31]. The differences between the results are also listed in Tables IV and V. Compared to HDoG with VBGMM, the HDoG identified more glomeruli and the difference with HDoG with VBGMM is much larger than for the other two methods, indicating over-detection under the single optimal scale of the DoG and lower mean aVglom than HDoG with VBGMM. UH-DoG identified fewer glomeruli and larger mean aVglom due to under-segmentation when using the single thresholding (0.5) on the probability map of U-Net combined with the Hessian convexity map. BTCAS provided the most accurate measurements of Nglom and mean aVglom compared to the other two methods.

Table VI.

Mouse Kidney glomerular segmentation (N_glom) from CFE-MRI using HDoG, UH-DoG and the proposed BTCAS compared to HDoG with VBGMM method

Mouse kidney		Nglom (HDoG with VBGMM)	Nglom (BTCAS)	Difference Ratio (%)	Nglom (UH-DoG)	Difference Ratio (%)	Nglom (HDoG)	Difference Ratio (%)
CKD	ID 429	7,656	7,719	0.82	7,346	4.05	10,923	42.67
	ID 466	8,665	8,228	5.04	8,138	6.08	9,512	9.77
	ID 467	8,549	8,595	0.54	8,663	1.33	12,755	49.20
	Avg	8,290	8,181	2.13	8,049	2.91	11,063	33.88
	Std	552	440		663		1626
Control for CKD	ID 427	12,724	12,008	5.63	12,701	0.18	15,515	21.93
	ID 469	10,829	11,048	2.02	11,347	4.78	15,698	44.96
	ID 470	10,704	10,969	2.48	11,309	5.65	13,559	26.67
	ID 471	11,943	12,058	0.96	12,279	2.81	16,230	35.90
	ID 472	12,569	13,418	6.75	12,526	0.34	17,174	36.64
	ID 473	12,245	12,318	0.60	11,853	3.20	15,350	25.36
	Avg	11,836	11,970	3.07	12,003	1.41	15,588	31.91
	Std	872	903		595		1193
AKI	ID 433	11,046	10,752	2.66	11,033	0.12	12,315	11.49
	ID 462	11,292	10,646	5.72	10,779	4.54	17,634	56.16
	ID 463	11,542	11,820	2.41	10,873	5.80	20,458	77.25
	ID 464	11,906	12,422	4.33	11,340	4.75	25,233	>100
	Avg	11,447	11,410	3.78	11,006	3.85	18,910	64.21
	Std	367	858		246		5401
Control for AKI	ID 465	10,336	10,393	0.55	10,115	2.14	13,473	30.35
	ID 474	10,874	11,034	1.47	11,157	2.60	16,934	55.73
	ID 475	10,292	9,985	2.98	10,132	1.55	12,095	17.52
	ID 476	10,954	11,567	5.60	10,892	0.57	15,846	44.66
	ID 477	10,885	11,143	2.37	11,335	4.13	14,455	32.80
	Avg	10,668	10,824	2.59	10,726	0.54	14,561	36.21
	Std	325	630		572		1908

Table VII.

Mouse Kidney glomerular segmentation from CFE-MRI (mean aV_glom) using HDoG, UH-DoG and the proposed BTCAS compared to HDoG with VBGMM Method

Mouse kidney		Mean aVglom (HDoG with VBGMM)	Mean aVglom (BTCAS)	Difference Ratio (%)	Mean aVglom (UH-DoG)	Difference Ratio (%)	Mean aVglom (HDoG)	Difference Ratio (%)
CKD	ID 429	2.57	2.63	2.33	2.92	11.99	2.46	4.28
	ID 466	2.01	2.01	0.00	2.06	2.43	1.75	12.94
	ID 467	2.16	2.20	1.85	2.32	6.90	1.9	12.04
	Avg	2.25	2.28	1.40	2.43	7.67	2.04	9.75
	Std	0.29	0.32		0.44		0.37
Control for CKD	ID 427	1.49	1.57	5.37	1.61	7.45	1.49	0.00
	ID 469	1.91	1.95	2.09	2.20	13.18	1.76	7.85
	ID 470	1.98	2.05	3.54	2.04	2.94	1.73	12.63
	ID 471	1.5	1.58	5.33	1.56	3.85	1.4	6.67
	ID 472	1.35	1.36	0.74	1.49	9.40	1.35	0.00
	ID 473	1.5	1.56	4.00	1.58	5.06	1.39	7.33
	Avg	1.62	1.68	3.51	1.75	7.16	1.52	5.75
	Std	0.26	0.26		0.30		0.18
AKI	ID 433	1.53	1.64	7.19	1.63	6.13	1.38	9.80
	ID 462	1.34	1.41	5.22	1.48	9.46	1.3	2.99
	ID 463	2.35	2.4	2.13	2.61	9.96	1.94	17.45
	ID 464	2.31	2.36	2.16	2.40	3.75	1.78	22.94
	Avg	1.88	1.95	4.18	2.03	7.27	1.60	13.29
	Std	0.52	0.50		0.56		0.31
Control for AKI	ID 465	2.3	2.46	6.96	2.40	4.17	2.11	8.26
	ID 474	2.44	2.34	4.10	2.52	3.17	2.14	12.30
	ID 475	1.74	1.86	6.90	1.70	2.35	1.58	9.20
	ID 476	1.53	1.57	2.61	1.62	5.56	1.49	2.61
	ID 477	1.67	1.68	0.60	1.70	1.76	1.61	3.59
	Avg	1.94	1.98	4.23	1.99	2.62	1.79	7.19
	Std	0.41	0.40		0.43		0.31

G. Discussion of Computation Time

Our proposed method uses U-Net for pre-processing, followed by the DoG where the scales vary depending on sizes of the glomeruli. The computational time of U-Net is satisfactory. For example, it takes < 5 minutes for training and <1 second per slice or per patch for testing. Therefore, we focus on the discussion of computation efforts related to the DoG implementation. Given a 3D image in N₁ × N₂ × N₃, and a convolutional filtering kernel size as r₁ × r₂ × r₃, the computational complexity of HDoG is O(N₁N₂N₃(r₁ + r₂ + r₃)[30]. Considering our proposed method BTCAS, let N_S be the number of scales searched (N_S >1), the computational complexity is O(N_SN₁N₂N₃(r₁ + r₂ + r₃)). We conclude BTCAS requires more computing effort comparing to HDoG [31] since N_S>1. Yet, HDoG [30], the single scale approach suffers from performances as shown in the comparison experiments (see Fig. 6 in Section II.E, Tables II–VII in Section II.C, E and F). Exhaustively searching optimum scale for each glomerulus however is computational prohibitive. Table VIII summarizes the computational time for DoG under exhaustive search on scales (note the scale ranges [0, 1.5] using stereology knowledge) for each glomerulus and that for BTCAS. As seen, BTCAS saves about 30% computing time.

Table VIII.

Comparison of computation time between DoG under glomerulus-specific optimla scale and proposed BTCAS method

Human Kidney	DoG under glomerulus-specific optimal scale (second)	BTCAS (second)	Difference Ratio (%)
CF 1	51,238	34,792	32.10
CF 2	39,616	28,156	28.93
CF 3	59,703	41,425	30.61
AVG ± STD	50,186 ± 10,085	34,791 ± 6,635	30.55 ± 1.59

IV. Conclusion

In this research, we develop a new small blob detector (BTCAS). This work provides three main contributions to the literature. First, U-Net reduces over-detection when it was used in the initial de-noising step. This results in a probability map with the identified centroid of blob candidates. Second, distance maps were rendered with lower and upper probability bounds, which are used as the constraints for local scale search for the DoG. Third, a local optimum DoG scale was adapted to the range of blob sizes to better separate touching blobs. In two experiments, this adaptive scale based on deep learning greatly decreases under-segmentation by U-Net with over 80% increase in Dice and IoU and decreases over-detection by DoG with over 100% decrease in error rate of blob detection.

While the results of this study are encouraging, there is room for improvement. First, the proposed method consists of two sequential steps, where adaptive DoG and Hessian analysis are based on the probability and distance maps predicted from U-Net. This approach may not be computationally efficient. It is our intention to integrate the DoG and Hessian analysis as layers of in the overall deep learning network for comprehensive glomerular segmentation. Second, we used 2D U-Net instead of 3D U-Net to perform on 3D images, so each slice is processed independently. The performance might be different under a 3D U-Net. We also plan to explore semi-supervised learning by incorporating domain knowledge of glomeruli to further improve glomerular detection and segmentation. All the future work is built upon the BTCAS which we believe has shown to be an adaptive and effective tuning-free detector for blob detection and segmentation and has the potential for kidney biomarker identification for clinical use.

APPENDIX

Proposition 1.

For any blob, let the normalized intensity distribution of the blob be $I_b(x,y,z)\in {[0,1]}^{I_1\times I_2\times I_3}$, and the centroid of the blob be (μ_x, μ_y, μ_z), assuming that the blob (after denoising) follows a rotationally symmetric Gaussian distribution,

$$I_b(x,y,z)=\frac{1}{2\pi {\sigma }^2}\exp \left(-\frac{{\left(x-{\mu }_x\right)}^2+{\left(y-{\mu }_y\right)}^2+{\left(z-{\mu }_z\right)}^2}{2{\sigma }^2}\right).$$ (25)

The probability predicted by U-Net increases or decreases monotonically from the centroid to the boundary of the dark or bright blob.

Proof.

Here we focus on bright blobs. Let the input intensity distributions of a blob with noise be $I_N\in {[0,1]}^{I_1\times I_2\times I_3}$. We have

$$I_N=I_b+\epsilon ,\epsilon ~\mathcal{N}\left(0,{\sigma }^{2}I\right).$$

We define the probability map from U-Net as $U(x,y,z)\in {[0,1]}^{I_1\times I_2\times I_3}$, which indicates the probability of each voxel belonging to any blob. The probability of blob U_b can approximate the intensity distribution of the blob based on (4):

$$U_b(x,y,z)={\mathcal{F}}_b\left(I_N;\Theta \right)={\mathcal{F}}_b\left(I_b+\epsilon ;\Theta \right)\approx I_b(x,y,z).$$ (27)

The probabilities from U_b(x, y, z) thus follow a Gaussian distribution and the probabilities monotonically decrease from the centroid to the boundary of a blob, with U_b(μ_x, μ_y, μ_z) reaching maximum probability.

Proposition 2.

Given a binarized probability map, a blob can be identified with a radius r. With B_L(x, y, z) and B_H(x, y, z), we obtain $r_{{\delta }_L}$, $r_{{\delta }_H}$ respectively. B_L(x, y, z) marks a larger blob region extending to the boundaries with low probability and B_H(x, y, z) marks a smaller blob region extending the boundary with high probability, that is $r_{{\delta }_L}>r_{{\delta }_H}$.

Proof.

From (25) and (27), we get:

$$U_b(x,y,z)\approx I_b(x,y,z)=\frac{1}{2\pi {\sigma }^2}\exp \left(-\frac{{\left(x-{\mu }_x\right)}^2+{\left(y-{\mu }_y\right)}^2+{\left(z-{\mu }_z\right)}^2}{2{\sigma }^2}\right).$$ (28)

Let the radius of a blob be r(δ) ∈ R. The distance between the thresholding pixel (x_δ, y_δ, z_δ) and the centroid of blob can be approximated by the radius of the blob:

$$r(\delta )\approx \sqrt{{\left(x_{\delta }-{\mu }_x\right)}^2+{\left(y_{\delta }-{\mu }_y\right)}^2+{\left(z_{\delta }-{\mu }_z\right)}^2}.$$ (29)

Given high probability threshold δ_H and low probability threshold δ_L,

$$U_b\left(x_{{\delta }_H},y_{{\delta }_H},z_{{\delta }_H}\right)={\delta }_H,$$ (30)

and

$$U_b\left(x_{{\delta }_L},y_{{\delta }_L},z_{{\delta }_L}\right)={\delta }_L.$$ (31)

From Proposition 1, the blob centroid has the maximum probability and the probability monotonically decreases from the centroid to the boundary:

$$U_b\left(x_{{\delta }_L},y_{{\delta }_L},z_{{\delta }_L}\right)<U_b\left(x_{{\delta }_H},y_{{\delta }_H},z_{{\delta }_H}\right)<U_b\left({\mu }_x,{\mu }_y,{\mu }_z\right),$$ (32)

and

$$r\left({\delta }_L\right)>r\left({\delta }_H\right)>r\left(U_b\left({\mu }_x,{\mu }_y,{\mu }_z\right)\right)=0.$$ (33)