Quantifying morphologies of developing neuronal cells using deep learning with imperfect annotations

Abstract

The functionality of human intelligence relies on the interaction and health of neurons, hence, quantifying neuronal morphologies can be crucial for investigating the functionality of the human brain. This paper proposes a deep learning (DL) based method for segmenting and quantifying neuronal structures in fluorescence microscopy images of developing neuronal cells cultured in vitro. Compared to the majority of supervised DL-based segmentation methods that heavily rely on creating exact corresponding masks of neuronal structures for the preparation of training samples, the proposed approach allows for imperfect annotation of neurons, as it only requires tracing the centrelines of the neurites. This ability accelerates the preparation of training data by several folds. Our proposed framework is built on a modified version of PSPNet with an EfficientNet backbone pre-trained on the CityScapes dataset. To handle the imperfectness of training samples, we incorporated a weighted combination of two loss functions, namely the Dice loss and Lovász loss functions, into our network. We evaluated the proposed framework and several other state-of-the-art methods on a published dataset of approximately 900 manually quantified cultured mouse neurons. Our results indicate a close correlation between the proposed method and manual quantification in terms of neuron length and the number of branches while demonstrating improved analysis speed. Furthermore, the proposed method achieved high accuracy in neuron segmentation, as evidenced by the evaluation of the neurons’ length and number of branches.

Graphical Abstract

1. Introduction

Understanding the properties of neurons is crucial for the diagnosis and treatment of neurological disorders. Neurons are the fundamental units of the nervous system, and their proper functionality depends on various factors, including their shape, size, branching, and morphological features (Rosso et al., 2005, Meinertzhagen et al., 2009. Any impairment in these factors can lead to a range of neurological disorders, such as Alzheimer’s disease, Parkinson’s disease, and epilepsy Andrade and Ramalho (2018).

The use of advanced imaging techniques, such as fluorescence microscopy, can provide a detailed view of the neuronal structure and function Sancataldo et al. (2019). Although microscopy imaging, and more specifically, (epi)fluorescence microscopy imaging is a particularly effective technique for analyzing the structure and function of neurons, accurate numerical evaluation and analysis of such data can still be challenging. In fact, accurate quantification of neurons is possible through precise segmentation of neuronal images. Precise segmentation of neuronal images requires accurate annotation of the images by experienced experts. This issue often involves creating binary masks for each image, such that each mask accurately colocalizes with the corresponding neuron. Precise annotation of the entire neuronal mask, however, can be a time-consuming and tedious process. To overcome this bottleneck and to accelerate training data preparation, we developed a segmentation approach that allows for imperfect annotations of the samples, using only the centrelines of the neurites. To clarify, our suggested annotation scenario allows annotators to simply trace the centrelines of the neurites rather than requiring them to label the entire neurons precisely. The weakly labeled neurons (here, the 1-pixel-width skeletons of neurites) are then uniformly dilated using morphology filters. Moreover, higher-order branches of the neurons, which are thin and/or short in size, have been discarded by the annotator during the annotation process. Along with the raw images of neurons, the corresponding dilated structures are then fed into the downstream segmentation network during the training process.

An example neuron image is presented in Fig. 1-a, along with its manual annotation by an expert in Fig. 1-b. The annotator only marked the centrelines of the neuronal structures to accelerate the annotation process. Moreover, higher-order branches of the neurons, which are thin and/or short in size have been discarded by the annotator during the annotation process. Subsequently, these centrelines were uniformly dilated and used to train the proposed deep learning-based segmentation network. As evident from the figure, the data is weakly annotated, often resulting in the final mask fed into the network for training being either thicker (indicated by red dashed boxes) or thinner (shown by blue solid boxes) than the actual neurites.

Fig. 1.

An example of imperfect annotation of neuronal data. (a) A 2D micrograph of a neuron image. (b) The expert annotates the neuron by marking only the centrelines of the neuronal structures. To expedite the process of neuron annotation, the annotator disregarded the smaller and/or finer neurites during the annotation process. The annotated centrelines will then be uniformly dilated and fed into the proposed deep learning-based segmentation network for training. (c) The dilated centerlines overlaid on the micrograph. The resulting masks are often either thinner (red dashed boxes) or thicker (blue solid boxes) than the actual neurites, leading to an imperfect set of training samples that weakly supervise the deep learning network.

The proposed neuron segmentation network is based on a modified version of PSPNet (Pyramid Scene Parsing Network) Zhao et al. (2017). PSPNet is a well-established deep learning architecture that has shown state-of-the-art performance in semantic image segmentation tasks in computer vision Zhu et al. (2021). In order to tailor it to our particular task of segmenting neurons with imperfect annotation, we introduce several contributions, such as leveraging a combined loss function, replacing ResNet He et al. (2016) with EfficientNet Tan and Le (2019) as the backbone of the PSPNet architecture, and using the pyramid pooling module which is added on top of the EfficientNet. EfficientNet is a computationally efficient deep learning architecture that has demonstrated success in various computer vision tasks, such as object detection Tan et al. (2020), image recognition Xie et al. (2020), and panoptic segmentation De Carvalho et al. (2022). Our decision to use EfficientNet as the backbone is based on its ability to achieve high accuracy while being computationally efficient Tan and Le (2019). Specifically, we utilize an EfficientNet model that has been pre-trained on the CityScapes dataset. Despite the CityScapes dataset consisting of high-resolution images of urban scenes unrelated to biology, we discovered that the pre-trained network was a fitting option for performing transfer learning and extracting features from our microscopy images. This modification enables us to capture more complex features and achieve higher accuracy in our neuron segmentation task.

Furthermore, to address the challenge of imperfect training samples that leads to a weakly-supervised machine learning problem, we propose using a weighted combination of Dice loss and Lovász loss as the loss function for our model during training. This approach takes into account both the overlap between predicted and ground-truth segmentation masks (Dice loss) and the geometric properties of the segmented object (Lovász loss). By properly weighing the two losses, we aim to improve the accuracy of our model in segmenting neurons even when the training data contains imperfect or noisy samples.

We present the performance evaluation of our technique on 2D fluorescence micrographs of primary hippocampal neurons obtained from mice, which were initially quantified manually in Brosig et al. (2019a), and subsequently through unsupervised machine learning in Zehtabian et al. (2022a). We compare the supervised neurite segmentation achieved by our proposed fully-automated approach with ground truth results obtained through manual image analysis in terms of neuron length as well as the number of branches. The analysis of our technique was found to be in agreement with that of experienced users, and it was capable of simultaneously analyzing hundreds to thousands of images in a short time without requiring manual interventions.

The primary contributions of this work are summarized as follows: .

• 1.Addressing Imperfect Annotation: Unlike the majority of previous related works that have only focused on perfect annotation scenarios, this study proposes a solution to tackle imperfect annotation. Specifically, the approach involves annotating only the centerlines of the neuronal components, which helps mitigate the challenges posed by manual annotation efforts that are time-consuming and tedious.
• 2.EfficientNet Integration: To strike a balance between speed and accuracy, we adopted the EfficientNet algorithm and incorporated it as a backbone in the PSPNet architecture. This integration not only ensures an accurate segmentation but enhances the computational efficiency of the proposed method.
• 3.Weighted Loss Function: We suggest a weighted combination of two well-known loss functions in this study. This enables the neural network to accurately detect finer structures in the neuronal image, further improving the quality and integration of segmentation results.

The rest of the article is as follows: in Section 2, we will review related articles on neuron segmentation. In Section 3, we will explain our proposed neuron segmentation approach in more detail, including the main network architecture and the suggested loss function. Section 4 will present our experimental results and performance comparisons with five other state-of-the-art methods. We will also provide a discussion of the achieved results. Finally, in Section 5, we will conclude the article by summarizing the contributions of our proposed approach and suggesting potential applications.

2. Literature Review

The segmentation of neuronal structures in microscopy images is still a crucial task in neuroscience research. Over the years, various methods have been developed to address this issue, and more specifically, to automate the segmentation process. These methods span from unsupervised approaches Zehtabian et al. (2022a) and traditional machine learning methods Dong and Xie (2005) to more recent deep learning-based techniques Bilodeau et al. (2021). This section provides an overview of some of the most related works, highlighting their strengths and limitations.

Sholl analysis Sholl (1953) is one of the pioneers in quantifying neuronal morphology. This method approximates dendritic branching by counting the number of times neurites cross concentric circles at increasing distances from the soma, allowing for quantification of neuronal morphology. In 2000, Ascoli and Krichmar developed L-Neuron, a software package that enables the creation and analysis of neuronal analogs for quantifying neurons. L-Neuron employs a set of recursive rules to accurately and efficiently describe dendritic geometry and topology. This is achieved by locally inter-correlating morphological parameters such as branch diameter and length. Back in 2000, L-Neuron was able to accurately generate neuronal structure models that were both realistic and computationally efficient for further analysis Ascoli and Krichmar (2000). In Schmitz et al. (2011), the authors developed an automated image analysis routine that utilized steerable filters and deconvolutions to analyze dendrite and synapse characteristics in immuno-fluorescence images automatically and was able to detect and quantify a wide range of neuronal organelles. Authors in Quan et al. (2016) proposed an automated technique for tracing neuronal processes in 3D image stacks using the constrained principal curves. The algorithm extracts the centerline of neuronal processes by tracing a series of points and also identifies collisions, crossings, and bifurcations between two processes. In their study, Wang et al. (2017) developed an ensemble neuron tracing method that automatically reconstructs complex neuron morphology from 3D image data. Their approach incorporated multiple neuron tracers to generate optimal tracing results by leveraging the strengths of different tracers. In their work Ikeno et al. (2018), the authors presented a method that combines several software tools for image masking and filtering with an existing tool for dendritic segmentation and tracing to process confocal microscope images and reconstruct neuronal structures. The authors evaluated the performance of their proposed method by comparing it against ground-truth reconstructions from the BigNeuron Project. In 2018, Kofahi et al. introduced a single-channel cell segmentation algorithm for microscopy images that utilized less staining in the nucleus Al-Kofahi et al. (2018). The algorithm integrated deep learning to predict the locations of cells and nuclei, which was subsequently combined with thresholding and watershed-based segmentation. A method for detecting and segmenting neurons from microscopy acquisition was proposed by the authors in Baglietto et al. (2018). The process commenced with the automatic detection of the soma through multiscale blob enhancement filtering. Subsequently, an active contour approach was employed to achieve precise segmentation of the identified cell body. The outcome of this segmentation served as the initial seed for the second part of the approach, which proposed a method for tracing dendrite arborization.

StarDist Schmidt et al. (2018) is a supervised deep learning network that utilizes the U-Net architecture and was initially designed for cell and nuclei segmentation. It also showed its efficiency in the segmentation of other similar structures Zehtabian et al. (2022b). In general, StarDist excels at segmenting blob-like objects with a star-convex shape, although its performance may be less optimal for strongly elongated shapes such as neurons. To achieve better results with such shapes, it may be necessary to increase the number of rays used by the model. A technique for automated reconstruction and quantification of microglia, the brain’s resident immune cells, was proposed by Abdolhoseini et al. (2019). The method starts with utilizing multilevel thresholding to segment soma structures and identify foreground areas. Seed points are subsequently extracted from the foreground areas and used to generate the branches’ skeleton via the tracing process. Finally, the reconstructed data is quantified. In the study Soltanian-Zadeh et al. (2019), the authors developed a three-component segmentation algorithm that includes preprocessing steps to prepare two-photon microscopy data for analysis by CNN, a core 3D CNN architecture called STNeuroNet, which generates a probability map of potential masks for active neurons, and a set of postprocessing steps to infer the location and mask of individual active neurons from the outputs of STNeuroNet.

SplineDist Mandal and Uhlmann (2021) is an extension of the StarDist framework that utilizes parametric spline curves to model objects, instead of star-convex polygons used in StarDist. This makes SplineDist a potentially better alternative to StarDist for segmenting elongated or tubular objects such as neurons. The study conducted by Shih et al. (2021) introduced a new automatic algorithm named NeuroRetriever for unbiased large-scale segmentation of confocal fluorescence images of single neurons in the adult Drosophila brain. NeuroRetriever employs a high-dynamic-range thresholding method to segment the three-dimensional morphology of single neurons using branch-specific structural features. Gallart et al. showed that deep learning architectures could be utilized for automatic segmentation of dendritic spines using light microscopy Vidaurre-Gallart et al. (2022). To achieve this, the authors constructed a sizable dataset of ground truth data and evaluated the potentially effective DL biomedical segmentation architectures. Authors in Liu et al. (2022) conducted a review of automated neuron segmentation and tracing in diverse scenarios, with particular emphasis on the growing use of deep learning enhanced methods. They also discussed semi-automatic techniques for single neuron tracing in mammalian whole brains, along with the resultant datasets.

Authors in Bilodeau et al. (2022) tackled a relatively similar issue to the concern of this study, which is related to dealing with imperfect annotation. They introduced a deep learning network, called MICRA-Net, for the analysis of microscopy data under weak supervision. The approach is particularly useful in situations where achieving precisely annotated datasets is challenging or not feasible. In their proposed approach, the learned information is extracted using gradient class activation maps to generate detailed feature maps of the biological structures of interest. Although their proposed network has been originally trained on a simple main classification task with image-level annotations, it has shown its ability to solve more complex tasks such as semantic segmentation, even when precise annotations are not available.

3. Proposed Method

3.1. Model Architecture

Our proposed neuron segmentation framework, as shown in Fig. 2, is built upon a modified variant of the PSPNet Zhao et al. (2017), which is a modern deep learning architecture for achieving accurate semantic image segmentation in computer vision. PSPNet is equipped with a pyramid pooling module to capture contextual information at multiple scales, thereby enriching the feature maps. This allows the network to gain a deeper understanding of the spatial relationship between objects in the image (i.e. the various components of the neurons), and consequently, to better capture fine-grained features that play a crucial role in achieving precise segmentation. In general, PSPNet can be built on top of different backbone architectures, including various versions of ResNet. More specifically, PSPNet often utilizes the ResNet-50 and ResNet-101 backbone architectures. The choice of the backbone architecture depends on factors such as the complexity of the data (here, the complexity and morphology of the neurons), available computational resources, and desired accuracy.

Fig. 2.

General overview of the proposed segmentation network architecture.

Regarding an outstanding accomplishment of the PSPNet in recent years Yan et al. (2021), we adapt this architecture to the specific task of 2D neuron segmentation using imperfect training samples by introducing modifications to the network design. One of the primary modifications we made to the architecture is to utilize EfficientNet-B0, which is the most lightweight implementation of EfficientNet Tan and Le (2019) and was trained on the CityScapes dataset Cordts et al. (2016), as the backbone in place of ResNet. EfficientNet is a deep learning architecture that is designed to be computationally efficient while still achieving state-of-the-art performance on a wide range of computer vision applications, including image recognition, object detection, and semantic segmentation. EfficientNet achieves its efficiency through a combination of several techniques, one of which is compound scaling. This method scales the depth, width, and resolution of the network in a principled way, resulting in an improved trade-off between accuracy and computational efficiency. By utilizing this technique, EfficientNet is able to achieve state-of-the-art performance while maintaining a small computational footprint. Furthermore, EfficientNet employs Squeeze-and-Excitation (SE) blocks as an additional technique to enhance its performance. These SE blocks are designed to selectively amplify important features and reduce less relevant information, resulting in improved accuracy and efficiency of the network without adding significant computational loss to the network. EfficientNet also uses the Swish activation function, which has been shown to outperform traditional activation functions such as ReLU on our desired neuron segmentation task Ramachandran et al. (2017).

The pyramid pooling module, which is added on top of the EfficientNet backbone, constitutes the second part of the PSPNet architecture Zhao et al. (2017). The pyramid pooling module in PSPNet is composed of parallel pooling layers with different kernel sizes, typically 1, 2, 3, and 6. Each pooling layer applies a different pooling operation to the feature maps, which allows the network to capture contextual information at multiple scales. The four pooling operations are average pooling, max pooling, square root of average pooling, and the square root of max pooling. Each pooling layer produces a fixed-sized output, regardless of the size of the input image, which allows for efficient computation. The outputs of the pooling layers are then concatenated and passed through a convolutional layer to produce the final segmentation map.

The final layer in the PSPNet architecture is typically a convolutional layer that is used to up-sampling the feature maps back to their original image size. This layer restores the resolution of the feature maps that were reduced during the pooling operations in the pyramid pooling module. Additionally, the final layer is equipped with a softmax activation function, which generates a probability distribution over the classes for every pixel in the input image. The output of the final layer is a segmentation map, where each pixel is assigned a class label (here, either the foreground neuron or the background) based on the highest probability in the distribution. In some variations of PSPNet, additional post-processing steps such as Conditional Random Fields (CRF) can be applied to the segmentation map to improve the accuracy of the final output. This step is omitted in the proposed framework in order to reduce the complexity.

Ultimately, the presented architecture produces a 2D segmentation map of the neurons that has identical dimensions as the input image. This segmentation map provides a representation of the location and boundaries of the neurons present in the image, enabling further analysis and interpretation of the neural structures.

3.2. Combined Loss Function

One of the main contributions of the presented work is the suggestion to use a weighted combination of Dice loss Sudre et al. (2017) and Lovász loss Berman et al. (2018) as the model’s loss function during training. Several sets of experiments confirmed that the network trained solely on Dice loss would converge once the general structure of the neurons is approximated. However, it may not be able to accurately detect the finer structures in the neuronal image. The Lovász loss, on the other hand, does not necessarily result in quick convergence of the network, unlike the Dice loss. In other words, it is resistant to premature convergence, allowing for the detection of finer neurites, such as neuronal branches. Additionally, it provides a better estimate of the width of different neuronal structures.

The computation of the suggested combined loss function is demonstrated in Equation (1), where χ represents the contribution of Dice loss, and φ represents the contribution of Lovász loss. Additionally, y and $\hat{y}$ denote the indicator functions for the true and estimated labels, respectively.

$$\mathrm{Loss}(y,\hat{y})=\chi L_{\mathrm{Dice}}(y,\hat{y})+\phi L_{\text{Lovász}}(y,\hat{y})$$ (1)

The equation defining $L_{\mathrm{Dice}}(y,\hat{y})$ is as follows:

$$L_{\mathrm{Dice}}(y,\hat{y})=\frac{1}{N}{\sum }_{i=1}^N\left(1-\frac{2y_i{\hat{y}}_i}{y_i+{\hat{y}}_i}\right)$$ (2)

Here, y_i represents the indicator function for the true label of pixel i, while ${\hat{y}}_i$ represents the indicator function for the estimated label of pixel i. Furthermore, L_Lovász is defined using the following equation:

$$\text{Lovász}=\frac{1}{C}{\sum }_{c\in C}\mathrm{\Delta }{\overline{J}}_c(m(c))$$ (3)

In this equation, C denotes the number of classes, and $\mathrm{\Delta }{\overline{J}}_c$ is defined as follows:

$$\mathrm{\Delta }{J}_c\left(y,\hat{y}\right)=1-J_c\left(y,\hat{y}\right)$$ (4)

In the above equation, J_c is the Jaccard function which is defined by the following equation:

$$J_c\left(y,\hat{y}\right)=\frac{∣\{\hat{y}=c\}\cap \{y=c\}∣}{∣\{\hat{y}=c\}\cup \{y=c\}∣}$$ (5)

The value of m for each pixel i in class c is computed using the following equation:

$$m_i\left(c\right)=\left\{\begin{array}{cc}1-f_i(c)\hfill & \mathrm{if}c=\widehat{y_i}\hfill \\ f_i(c)\hfill & \mathrm{otherwise}\hfill \end{array}\right)$$ (6)

Here, f_i refers to the class probabilities f_i(c) ∈ [0,1] that are defined as follows:

$$f_i\left(c\right)=\frac{e^{F_i(c)}}{{\sum }_{c^{\prime }\in C}{e}^{F_i(c^{\prime })}}\forall i\in \left[1,p\right],\forall c\in C$$ (7)

In this context, the function F_i refers to the softplus function, which is defined as $F_i(c)=\log (1+e^c)$, where c is a scalar value showing the class of pixel i.

To optimize the changing weights during training, the polynomial learning rate policy Mishra and Sarawadekar (2019) is utilized for adjusting the learning rate:

$$\lambda ={\lambda }_0*{(1-\frac{\iota }{{\theta }_{\iota }})}^{\delta }$$ (8)

In this equation, λ represents the running iteration learning rate, where λ₀ is the initial learning rate, and ι denotes the number of iterations. There are a total of epochs times the number of iterations in each epoch, which gives θ_ι, and δ is a hyperparameter that determines the shape of the polynomial curve.

4. Experimental Results

In this section, we present the results of applying our proposed neuron segmentation approach and other competing neuron segmentation approaches to raw 2D fluorescence micrographs of neurons. The purpose of this analysis is to evaluate the effectiveness of our proposed method against existing approaches in accurately segmenting individual neurons from 2D fluorescence microscopy image data, particularly when dealing with imperfect annotations. To achieve this goal, we evaluate each method’s performance by comparing their segmentation results to the ground truth data, considering the length of the neurons and the number of branches.

4.1. Dataset

For our research, we utilized a publicly available dataset Brosig et al. (2019b) consisting of 2D images of neurons, initially presented in Brosig et al. (2019a). Containing approximately 900 images of neurons with varying sizes and shapes, the dataset was recorded from primary hippocampal neurons of either C57 BL/6 or C57 Bl/6 PRG2 mice. For visualization of individual cells, neurons were transfected with a plasmid encoding for green fluorescent protein (GFP) after two days in culture, fixed after five days in culture, and immunolabeled for the axon marker Tau and GFP, following which they were imaged using a Nikon Eclipse Ti epifluorescence microscope with a 40x objective Brosig et al. (2019a). This must be noted that the neurons in this dataset were imaged at an early developmental stage when dendrites are not fully developed, resulting in relatively short dendritic structures.

The process of manually tracing and annotating neurites has been carried out using NeuronJ Meijering et al. (2004), an ImageJ plugin that is widely used for its ability to perform semi-automated centerline tracing of 2D neuron structures. Using NeuronJ, an expert has classified each pixel in the ground truth data into two classes, that are the foreground and the background. The foreground class consists of the centerline of the neuron’s components, including its axon, dendrites, and their respective branches. Subsequently, we manually identified and incorporated the somas (cell bodies) of the neurons into the ground truth masks. The ratio of foreground to background areas in the ground truth data had an average of 0.04, suggesting a significant imbalance in the segmentation problem.

Once the ground truth data was prepared, the dataset was partitioned into two subsets: training and testing. The training set comprised 850 images, where 50 of them were allocated for validation during the training phase. The testing set contained 50 images unseen to the network, which were used for evaluating the model’s performance.

4.2. Parameters Setting

The stochastic gradient descent (SGD) algorithm was employed for training the model with a momentum of 0.9 and a start learning rate of 0.001.

A polynomial learning rate policy Mishra and Sarawadekar (2019) was utilized for optimizing the learning rate during the training with the combined loss function where φ and χ values were set to 0.1 and 0.9 to adjust the impact of Lovász loss and Dice loss in Equation (1), respectively.

Additionally, the number of epochs and batch size were set to 10 and 4, respectively. We observed that increasing the number of epochs did not result in improved quantitative outcomes, which could be attributed to transfer learning. Furthermore, through an extensive set of experiments, it was observed that smaller weights for the Lovász loss led to discontinuities in segmented neurons. At the same time, larger values resulted in unwanted thicker neurites in the segmentation outcomes.

All of our training experiments were conducted on an NVIDIA Tesla T4 with 16 GB of VRAM, utilizing PyTorch bfloat16 precision during training.

4.3. Criteria

While a broader spectrum of quantitative measures, such as the region occupied by the soma and a few more complicated morphological features of neurons, can also be simply calculated, the focus of this study is primarily on the total length of the neurites and the number of branches, which are considered the two most significant readouts in our study. All pertinent visual and numerical outcomes are stored for subsequent statistical analysis by executing the suggested method.

4.4. Performance Comparison Results

The proposed model’s performance and a few existing competing supervised segmentation methods have been evaluated on a test set of 50 randomly taken images from the dataset. It should be noted that these images were not used during the training process and are new to all the networks, ensuring that they are unseen by the system. The existing supervised segmentation methods used in performance comparisons include U-Net Ronneberger et al. (2015), Adapted-U-Net, StarDist Schmidt et al. (2018), SplineDist Mandal and Uhlmann (2021), and MICRA-Net Bilodeau et al. (2021).

U-Net is a prevalent deep learning method for image segmentation, often employed in medical imaging. It utilizes a U-shaped encoder-decoder network with skip connections, enabling it to capture both local and global information. The skip connections also help retain spatial information lost during downsampling, enhancing the segmentation accuracy Ronneberger et al. (2015).

To enhance the U-Net architecture’s ability to handle imperfectly annotated training samples, we incorporated our proposed weighted combination of Dice loss and Lovász loss as the loss function. This resulted in a modified U-Net segmentation architecture, referred to as the “Adapted-U-Net”, hereafter.

StarDist Schmidt et al. (2018), which was initially developed for cell and nucleic segmentation, is a supervised deep machine learning network with the U-Net architecture as its foundation. Moreover, SplineDist Mandal and Uhlmann (2021) is an extension of the StarDist framework. Unlike StarDist, which uses star-convex polygons to represent objects, SplineDist models object using parametric spline curves. Finally, MICRA-Net Bilodeau et al. (2021) is a deep learning approach designed to analyze microscopy data under weak supervision, addressing the challenge of acquiring precisely annotated datasets that can be difficult or unfeasible to obtain.

To ensure a fair comparison of the different methods, we conducted a thorough grid search to optimize the parameters of the competing methods with respect to our data.

Figure 3 visually demonstrates the performance of our proposed method compared to the manual centreline detection of neurites used as the ground truth. As shown in the figure, our proposed method reproduces the manual analysis with a high degree of similarity.

Fig. 3.

Visualization of the lengths and branch numbers of 50 neurons, comparing the ground truth (horizontal axis) with the predictions of the proposed method (vertical axis), with the correlations quantified by R² scores. The lengths and branch numbers of neurons are expressed in pixel units.

To quantitatively evaluate the results, we computed morphological readouts of the segmented neurons and compared them to those obtained by manual analysis performed by the expert user. The results, shown in Fig. 3-a, include the total lengths of all reconstructed neurites as a measure of neuron size and the number of branches as a measure of complexity. Our analysis of total length measurements demonstrates a strong correlation (R² = 91.12%) between the automated reconstruction and manual reconstruction of the overall size of neurons.

The method being used categorizes the points where neurites intersect as branches. In Fig. 3-b, the number of branches computed by the proposed method for each test image is displayed alongside the results obtained through manual analysis. Despite having a lower level of correlation (R² = 80.30%) than the total length of the neuron, the proposed method is still considered acceptable and suitable for counting the number of branches in various applications, including the detection of phenotypes following biological treatments.

Table 1 presents a comparison of the performance of the proposed method with other competing methods in terms of accuracy in calculating the neuron length and in counting the number of neuronal branches. To achieve this, the correlations between the obtained results for each method and the ground-truth values are calculated in terms of R2 and presented in the table.

Table 1.

Comparison of the proposed method’s performance with other methods in terms of neuron length and branch count. Correlations with ground truth values are presented in terms of R2.

Method	Lengths R2	Branches R2
U-Net, by Ronneberger et al. (2015)	0.0	0.0
Adapted-U-Net	0.2600	0.0604
StarDist2D, by Schmidt et al. (2018)	0.4708	0.3390
SplineDist, by Mandal and Uhlmann (2021)	0.6438	0.5630
MICRA-Net, by Bilodeau et al. (2021)	0.8379	0.8168
Proposed Method	0.9112	0.8030

The table clearly indicates that the traditional U-Net method is unable to address the challenge of weak supervision caused by imperfectly annotated training samples. Furthermore, despite incorporating our proposed weighted combination of Dice loss and Lovász loss as the loss function in the “Adapted-U-Net” approach, the method was still unable to accurately segment the neuron images. While the results were superior to those of the traditional U-Net, the network was not able to learn enough from the training samples in which only the centrelines had been annotated.

Compared to the two aforementioned networks, the StarDist method demonstrated superior segmentation of the neuronal images, while the SplineDist method performed slightly better than the StarDist. We hypothesize that this performance difference could be due to the fact that the StarDist method is specifically designed to segment objects with a star-shaped appearance, whereas the SplineDist method utilizes spline-based curve fitting techniques. Given that the morphology of neurons can be approximated by a series of spline curves, the SplineDist method has been demonstrated to be more effective in segmenting neurons, compared with StarDist. However, both methods still struggle with imperfect annotations where the entire mask of the neuron has not been annotated by the annotator, and only the centrelines are available for training.

In contrast, the MICRA-Net method demonstrated significantly better performance than the four aforementioned networks. Although the method was not specifically designed to address the type of weak annotation presented in this paper, its ability to efficiently extract latent information under weak spatial supervision led to superior performance.

Compared to other approaches, the proposed approach demonstrated the best accuracy in determining the size of neurons. However, in terms of accurately counting the number of branches, the proposed method ranked second among the evaluated methods.

Furthermore, we evaluated the performance and number of parameters of the proposed method by switching the backbone between ResNet101 and EfficientNet. The experimental results, presented in Table 2, indicate that the proposed PSPNet-based method achieved slightly lower accuracy in approximating the neuronal size when equipped with EfficientNet. However, the number of parameters was approximately three times lower compared to using ResNet-101 as the backbone. For our experiments, an Apple MacBook Pro with M2 Apple silicon and 8 GB RAM was used for the run-time tests.

Table 2.

Comparison of ResNet-101 and EfficientNet-B0 backbone architectures in the proposed neuron segmentation approach for measuring neuron length and branch count. PSPNet-ResNet-101 performs slightly better than EfficientNet-B0 in accuracy, but has three times more parameters.

Architecture	Backbone	Parameters	Run Time	Lengths R2	Branches R2
PSPNet	ResNet-101	46 M	3.1 s	0.9120	0.8109
PSPNet	EfficientNet-B0	15.2 M	1.0 s	0.9112	0.8030

Figure 4 presents example results of applying the proposed method (PSPNet+EfficientNet-B0) to raw epifluorescent images randomly selected from the test set. The figure shows that the proposed segmentation approach is in good agreement with human manual tracing of the centerlines of neuronal structures.

Fig. 4.

Example results of the proposed method applied to randomly selected raw epifluorescent images from the test set, demonstrating compatibility of the proposed segmentation approach and human manual tracing of the centerlines of neuronal structures. From left to right: raw 2D images of developing neuronal cells, expert-created ground-truth data, and predictions made by the proposed network. The resulting centerlines are uniformly dilated for better visualization, as was done for the ground-truth data.

Finally, we illustrate the training progression of our network through its learning curve. Fig. 5 displays this curve for the weighted combined loss, plotted for both the total number of training iterations and epochs, to provide a dual perspective and enhance interpretability. As can be seen in the figure, the learning curve demonstrates a significant early drop in weighted loss, with diminishing improvements after ten epochs. Our data indicates that additional training after this point does not substantially enhance the loss reduction. Therefore, we limit training to ten epochs, balancing computational efficiency with the accuracy of key measures which are the neuron length and branch count. Our decision, supported by thorough experimental evidence, ensures both computational expediency and the preservation of quantitative accuracy.

Fig. 5.

Learning curves for weighted combined loss during network training, illustrating model convergence over (A) training iterations and (B) epochs.

4.5. Discussion on the limitations of the proposed approach

While the proposed method has shown its effectiveness in addressing the significant challenge of neuronal image segmentation with imperfect annotations, it is pertinent to recognize and discuss certain limitations associated with our approach.

Firstly, the applicability of our schema is confined to 2D images of neurons. It is not designed for, nor does it accommodate, 3D neuronal data. This limitation is a critical consideration, especially in contexts where three-dimensional data analysis is essential.

Moreover, the focus of our study is predominantly on early developing neurons, characterized by elaborated axons and relatively underdeveloped axonal branches as well as short dendrites. This specificity means that our method may not be as effective when applied to mature neurons, which typically exhibit denser networks and more complex structural intricacies. As such, the applicability of our approach in scenarios involving mature neuronal networks remains unexplored and potentially limited.

Additionally, although our framework shows proficiency in segmenting neuronal structures with perfect annotations, it is important to note that its performance in such scenarios is comparable to that of other existing segmentation methods but does not demonstrate superiority. This observation is a direct consequence of the design and optimization of our approach, which is intentionally specialized for handling imperfect annotations. This specialization, while beneficial in certain contexts, might not yield additional advantages when applied to datasets with perfect annotations.

Lastly, the nature of the annotations used in training our algorithm significantly influences the output. Specifically, training with our generated set of imperfect annotations leads to predictions in the form of neuronal skeletons, as opposed to segmentation masks that result from training with conventional perfect annotations. This distinction in output formats might limit the scope of our method’s applicability, depending on the specific requirements of a given analysis or research objective.

5. Conclusion

This paper demonstrated the effectiveness of deep learning for fluorescence microscopic neuron image segmentation tasks. We have shown that our proposed model can achieve high accuracy on a dataset with weak annotations. The centerline labeling has been done via NeuronJ, which results in imperfect annotations. In order to ameliorate the performance of the PSPNet, an EfficientNet backbone has been utilized, which is pre-trained on the CityScapes dataset. We measured neuronal morphology from segmented images by extracting essential parameters, such as neuron length and branch number. These parameters are fundamentally relevant to neuroscience. We tested the proposed method on published data comprising manually quantified cultured mouse neurons. Our proposed fully-automated method can be used to analyze fluorescence microscopy images of 2D neurons in a high-throughput manner, thus facilitating the screening of several biological treatments and the analysis of a larger number of neurons for detecting changes in phenotypes following different biological treatments.

CRediT authorship contribution statement

AMN developed the core image processing algorithms, designed the software, and visualized the results. AZ formulated the project idea, contributed to the software development, and assisted with result analysis and interpretation. HH provided administration and supervision for the projec and confirmed the results. All authors contributed to the manuscript’s composition and approved the final version.