Co-localization of fluorescent signals using deep learning with Manders overlapping coefficient

Abstract

Object-based co-localization of fluorescent signals allows the assessment of interactions between two (or more) biological entities using spatial information. It relies on object identification with high accuracy to separate fluorescent signals from the background. Object detectors using convolutional neural networks (CNN) with annotated training samples could facilitate the process by detecting and counting fluorescent-labeled cells from fluorescence photomicrographs. However, datasets containing segmented annotations of colocalized cells are generally not available, and creating a new dataset with delineated masks is label-intensive. Also, the co-localization coefficient is often not used as a component during training with the CNN model. Yet, it may aid with localizing and detecting objects during training and testing. In this work, we propose to address these issues by using a quantification coefficient for co-localization called Manders overlapping coefficient (MOC)¹ as a single-layer branch in a CNN. Fully convolutional one-state (FCOS)² with a Resnet101 backbone served as the network to evaluate the effectiveness of the novel branch to assist with bounding box prediction. Training data were sourced from lab curated fluorescence images of neurons from the rat hippocampus, piriform cortex, somatosensory cortex, and amygdala. Results suggest that using modified FCOS with MOC outperformed the original FCOS model for accuracy in detecting fluorescence signals by 1.1% in mean average precision (mAP). The model could be downloaded from https://github.com/Alphafrey946/Colocalization-MOC.

1. INTRODUCTION

Fluorescence microscopy has been extensively used in biomedical research for decades and is often considered a gold standard method for studying cellular and subcellular anatomy and function. Commonly, multiple fluorescent probes are used to identify specific cellular or subcellular targets via the coincident expression of more than one target molecule. Interpretation of whether different molecules occur at the same spatial location (i.e co-localization) is critical for evaluating the chemical regulation of biological systems and the site of action.³ To quantify the co-localization, the object molecules are first identified, and then spatial analysis among these detected objects is performed using either co-occurrence or correlation analysis.^4,5 On one hand, however, object-based methods heavily rely on separating objects from the background, either manually or automatically. Manual delineation of cell segmentation for identifying co-localization events within a field of view is labor-intensive, requires domain knowledge, and is susceptible to observer subjectivity and inter-observer variation.⁶ Automated methods used thresholds and filters to separate objects from the background. The performances of those method are heavily depending on image quality and those methods are sensitive to background noise.^7–9 For certain in vitro biomarkers, most current methods would yield poor results.

To address these challenges, segmentation based convolutional neural networks (CNN) has been proposed for object detection of cells.¹⁰ However, detailed pixel-level labels still require annotation effort for preparing training datasets.⁶ By comparison, weakly supervised object detection with weak annotations, such as using bounding boxes^11,12 and points,^13,14 do not need a large amount of pixel-wise segmentation data and reduces the required annotation effort for preparing training datasets. In the training phase, the performance of bounding box regression can be improved by either a better network design,¹⁵ or a new branch for extracting reliable local features.¹⁶

In this work, we treated object-based co-localization analysis as an object detection task utilizing a CNN model. Bounding boxes were generated from point annotations created by a domain expert, thereby reducing the annotation burden for training the CNN model. First, fluorescence microscopy images were separated into individual wavelength channels and then overlaid on images taken at different wavelength channels to create combinations. In our datasets, fluorescent images of DAPI stained nuclei were overlaid with fluorescent images of NeuN immunoreactivity, a biomarker of neurons, and 3-nitrotyrosine (3-NT), a well-established marker of oxidative damage. Each point annotation served as the center for a corresponding bounding box annotation, which were defined as the coordinates of the rectangular border surrounding one colocalized cell, shown in Fig. 1. We propose a single-layer branch using Manders coefficient to measure the degree of co-occurrence, which calculate the spatial overlap between detected fluorophores, as defined in Eq. 1. Our approach provides more local information for bounding box localization and improves detection accuracy.

Figure 1.

A Representative fluorescence photomicrograph displaying three data channels (red, green and blue) with a bounding box (white box) which encloses a potential colocalized candidate.

The main contributions of this paper are summarized as:

• Introducing fluorescent labeled cell co-occurrence information as a single-layer branch to help filtering of bounding boxes determined to be low-quality, in order to improve detection accuracy.

• Showing improvement over original state-of-the-art FCOS model using lab curated dataset.

2. APPROACH

2.1. Detection Network

A fully convolutional one-state (FCOS) object detection network² was modified with Resnet101¹⁷ as the backbone in this work because of its state-of-the-art efficiency in both time and computation resources.² We introduced a single-layer branch in subnets called MOC, in parallel with classification and center-ness branch as shown in Fig. 2. For fair comparison, we modified the FCOS module implemented from mmdetection¹⁸ toolbox and used the nonmodified version from mmdetection¹⁸ as benchmark.

Figure 2.

The network architecture of modified FCOS. Compare to original FCOS network, a new MOC branch (one layer) is added to the subsets.

2.2. Manders Overlapping Coefficient

A single-layer branch using Manders coefficient is added in parallel with the center-ness and the classification branch. The co-occurrent signals were measured using Manders coefficient¹ within each bounding box instead of the correlation, which normally uses Pearson’s or Spearman’s coefficient.¹⁹ A higher MOC indicates that channels are highly colocalized. The funcion is

$$MOC=\frac{\sum \left(I_1\times I_2\times \dots \times I_n\right)}{\sqrt{\sum {\left(I_1\right)}^2\times \sum {\left(I_2\right)}^2\times \dots \times \sum {\left(I_n\right)}^2}},$$ (1)

where I_n are the intensity values within the detected bounding boxes that indicate target cell locations at channel n. The MOC, within the range of 0 to 1, indicates the fraction of colocalizing signal intensity among channels I_n, and the degree of co-localization. An advantage of the MOC is that it is not sensitive to differences in signal intensity due to photobleaching, different imaging setting, or different fluorescent probes.²⁰

MOC is trained with binary cross entropy (BCE) loss during training, then this loss is added to the loss function proposed from the original FCOS model.² The new loss function for our modified FCOS module is defined as

$$\mathit{\text{Loss}}=FL+\alpha \times IOU+\beta \times CE+MOC,$$ (2)

where FL stands for Focal loss, interaction-over-union (IOU) stands for IOU loss, CE stands for Cross Entropy, and and are hyperparameters.²

The MOC is also used to rank the detected bounding boxes during testing. The final ranking score is modified from multiplying the predicted center-ness with the corresponding classification score² to

$$\frac{\mathit{\text{center}}-\mathit{\text{ness}}+MOC}{c}\times \mathit{\text{classification}}.$$ (3)

By our experiments, we found out that 1.5 would be suitable value for c. Such procedure could down weight the score of bounding boxes containing less overlapping fluorophores. As a result, these bounding boxes with lower MOC scoring might be filtered out, improving the detection performance.

3. EXPERIMENTS

3.1. Training

A training dataset was generated by our lab.²¹ Appendix A contains the preparation and data acquisition methods in detail. Single channel grayscale images of different fluorescent probes were combined into RGB images. Image arrangement was as follows: NeuN labeled neurons (red) channel with DAPI stained nuclei (blue) channel and 3-NT (green) channel. For training purposes, the acquired raw data were cropped and resized to have their shorter side being 800 and their longer side less or equal to 1333 (800*1333). The dataset, including 4416 images, consisted of approximately 60% training data, 20% validation data, and 20% testing data.

Point annotations were made in the center of each cell determined by a domain expert. Bounding boxes were generated from the point labels, which served as the center for each individual bounding box for the target cell. Bounding box size was defined by the average size of the targeted cell (20 μm) multiplied by the fluorescence micrograph pixel resolution (2.981 pixel/μm). Those bounding boxes were also plotted back on the images for quality control by a team member.

The network was trained with stochastic gradient descent (SGD), with weight decay of 0.0001 and momentum of 0.9. The network was trained over 24 epochs with the initial rate of 0.005 with 8 images per batch (4 images per GPU). After a first warm up epoch, the learning rate is reduced by a factor of 10 at epoch 17 and 23, respectively. We used the hyper-parameters from the original FCOS paper to make sure the right inference setting was used.² Since Tian et al. did not test FCOS with fluorescence photomicrograph images from the original paper, we also compared our result with another state-of-the-art object detection method, FreeAnchor.²² The FreeAnchor was trained over 24 epochs with 8 images per iteration.

3.2. Evaluations

Performance was evaluated using standard practice of evaluating object detection.²³ Average precision (AP) is defined as the area under the recall-precision curve for detection precision, given a certain IOU threshold value between predicted bounding boxes with ground truth. We reported the mean average precision (mAP) value, which calculated the mean of AP using a range of IOU values. The range of IOU values was from 0.05 to 0.95 with an increment value of 0.05. The AP at IOU value of 0.5 and 0.75, denoted as AP50 and AP75 respectively, was also determined. Reported results evaluated the performance on training with our dataset using original FCOS as baseline, and our modified model as comparison.

4. RESULT

Table 1 shows the improvement in object detection accuracy using our modified model. The mAP, AP50, and AP75 increased by 1.1%, 0.3%, and 3.3% respectively, compared with the original proposed FCOS model. We also shows that both FCOS (with and without MOC) outperformed FreeAnchor. Thus, training with the MOC branch improves the performance using our datasets.

Table 1.

FCOS vs. our modified model (FCOS+MOC) and Free Anchor. S is the size of the shorter side of the image, and S × 1.66 implies that the image is rescaled such that its longer side cannot exceed 1.66 × S.²⁴ FCOS+MOC outperforms the original FCOS by 1.1% mAP, 0.3%AP50 and 3.3% AP75 with the same backbone.

Module	Backbone	Training Size	Test Size	AP	AP50	AP75
FreeAnchor22	X-101	800 × 1.66	800 × 1.66	47.0	87.9	42.6
FCOS2	X-101	(640, 800) × 1.66	800 × 1.66	49.2	90.4	44.7
Our Method	X-101	(640, 800) × 1.66	800 × 1.66	50.3	90.7	48.0

5. CONCLUSION

In this study, fluorescence photomicrographs obtained from our lab were used to train a state-of-the-art object detection network and used MOC as a subnet branch for cell co-localization analysis. The task of co-localizing different wavelength fluorescent signals was carried out by deep learning-based object detection using bounding boxes based on point annotation. We report that training with MOC improved detection performance compared to the baseline. This approach is promising for improving accuracy in fluorescent signal co-localization tasks. In the future, we plan to determine whether the MOC can boost the performance on different network architectures for object detection as well as object counting.

APPENDIX A. STAINING METHOD AND DATA ACQUISITION

Adult male Long Evans rats were exposed to 10 mg/kg/d chlorpyrifos (CPF, s.c.) or vehicle (10% Ethanol+90% Neobee M-5) for up to 21 d. A subset of animals were pretreated with the antioxidant Trolox (1 mg/kg/d i.p.) beginning 5 d prior to and continuing throughout CPF exposure. Brains were collected and sliced into 2 mm thick coronal sections. Tissue sections were then post-fixed in 4% paraformaldehyde (PFA) for 24 h at 4°C before being transferred to 30% sucrose in PBS at 4°C. Sections were embedded and frozen in Tissue-Plus^™ O.C.T. compound, and cryosectioned into 10 μm thick sections and stored at −80°C. To perform immunohistochemistry staining, slides were air-dried at room temperature, washed with PBS, and then placed in sodium citrate buffer (10 mM C6H5Na3O7 . 2H2O, pH 6.0) and heated for 30 min in a rice cooker for antigen retrieval. Following antigen retrieval, sections were incubated in blocking buffer [PBS with 10% normal goat serum, 1% bovine serum albumin and 0.03% Triton X-100] for 1 h at room temperature, and then incubated with primary antibody in blocking buffer at 4°C overnight. Primary antibodies included mouse anti-NeuN (1:1000) and rabbit anti-nitrotyrosine (3-NT, 1:200). Sections were washed in PBS with 0.03% Triton X-100 on the next day before incubated in secondary antibody in blocking buffer for 2 h at room temperature. Goat-anti mouse IgG1(γ1) conjugated to Alexa Fluor 568 nm (1:1000, A21124, Life Technologies) and goat-anti rabbit Ig conjugated to Alexa Fluor 488 nm (1:500, Life Technologies A11034) were used to detect anti-NeuN and anti-3-NT, respectively. Fluorescent images were acquired by the ImageXpress Micro XLS Widefield High-Content Analysis System (Molecular Devices).