Efficient implementation of convolutional neural networks in the data processing of two-photon in vivo imaging

Abstract

Motivation

Functional imaging at single-neuron resolution offers a highly efficient tool for studying the functional connectomics in the brain. However, mainstream neuron-detection methods focus on either the morphologies or activities of neurons, which may lead to the extraction of incomplete information and which may heavily rely on the experience of the experimenters.

Results

We developed a convolutional neural networks and fluctuation method-based toolbox (ImageCN) to increase the processing power of calcium imaging data. To evaluate the performance of ImageCN, nine different imaging datasets were recorded from awake mouse brains. ImageCN demonstrated superior neuron-detection performance when compared with other algorithms. Furthermore, ImageCN does not require sophisticated training for users.

Availability and implementation

ImageCN is implemented in MATLAB. The source code and documentation are available at https://github.com/ZhangChenLab/ImageCN.

Supplementary information

Supplementary data are available at Bioinformatics online.

1 Introduction

Monitoring the activities of every neuron in a microcircuit is essential in understanding the working principles of the brain. Increasingly, functional imaging using two-photon (2p) microscopy has been used to capture data from neurons that have been labeled with calcium indicators (Helmchen and Denk, 2005; Svoboda et al., 1997). This powerful tool provides real-time monitoring of hundreds of individual neurons in a neural microcircuit using single-neuron resolution. However, the processing of 2p image data is challenging due to the enormous volume of data and the low signal–noise ratio. Many supervised [e.g. convolutional neural network (CNN)-based classification and the RobustBoost algorithm] and unsupervised (e.g. PCA/ICA, CNMF, NeuroSeg) methods are generated to analyze the large amount of data (Guan et al., 2018; Klibisz et al., 2017; Mukamel et al., 2009; Pnevmatikakis et al., 2016; Valmianski et al., 2010; Xu et al., 2016). Supervised algorithms require manually annotated ground truth data to train models, whereas unsupervised algorithms do not. Different approaches have been applied to detect the locations of neurons. Morphology-based methods could extract morphological properties of neurons, whereas neurons with low intensity and blurred outlines but clear calcium spikes are difficult to detect. Activity-based methods detect neurons based on the localized spatiotemporal activity of each neuron, although the temporarily silent neurons might be easily missed. To overcome these disadvantages, we developed an analysis pipeline that combines the fluctuation method and the CNNs model to process 2p image data. This system, called ImageCN, has a superior ability to analyze 2p functional imaging data and requires minimal user experience.

2 Materials and methods

All imaging experiments were performed as previously reported (Jiang et al., 2017; Su et al., 2016; Tian et al., 2018; Wei et al., 2016). A detailed description of the methods is available at Bioinformatics online.

3 Results

3.1 Workflow for data processing of 2p imaging

The basic strategy of ImageCN is shown in Figure 1A. Briefly, for this study, the image stack was transformed into a reference image and an average-projection image, which represent the activity feature and morphological structure of each neuron, respectively. Regions of interest (ROIs) were extracted and segmented into patches by the local adaptive thresholding algorithm and the watershed algorithm. Two pre-trained CNN models were used to classify each patch from the reference image and average-projection image, respectively. Subsequently, a similar CNN-based strategy was utilized to pick up every spike for every identified neuron.

Fig. 1.

Schematic workflow and performance tests of ImageCN. (A) The flowchart of cell detection and spike extraction based on CNNs. (B) A set of 48 manually labeled cell and non-cell patches from the reference image received using the CNNs. (C) The receiver operating characteristic curve and area under curve of exhaustive grid search, the GA, the BPNN and the CNNs. (D) Comparison of ImageCN performance with manually annotated data. Scale bar, 50 μm

3.2 The extraction and segmentation of ROIs

To detect neurons from raw imaging data, the reference images and average-projection images were generated using the fluctuation method. The fluctuation method was based on the assumption that the active neurons possessed significantly larger signal fluctuations than the inactive neurons and the background. Compared to images from the maximum or average-projection methods, the reference images generated via the fluctuation method showed rigid delineations that marked the responsive cells (Supplementary Fig. S1). The average-projection image was generated to detect the morphological structure of each neuron. The ROIs of these two images were detected by local adaptive thresholding algorithm and segmented using the watershed algorithm (Supplementary Figs S2 and S3). However, only a portion of the ROIs were actual neurons; therefore, each ROI was extracted from images and segmented into a small patch for further classification using pre-trained CNN models (Fig. 1B).

3.3 Comparison between CNN and classic models

For further classification of each patch, we used CNNs to extract neurons from the background and neuropils (Supplementary Fig. S7). The two CNN models were trained with two datasets, one containing 3403 patches (1885 neurons and 1518 non-neurons) and the other containing 3247 patches (1802 neurons and 1445 non-neurons). The F1 score (balanced F score) was calculated as the harmonic mean of recall and precision, which has been widely used to assess the performance of a binary classifier, and it was used to quantify the performance of the neuron-detection. We compared the performance of CNNs with other classic algorithms, such as exhaustive grid search, the back propagation neural network (BPNN) and the genetic algorithm (GA), and our results indicated a higher performance of CNNs (Fig. 1C). The F1 scores of the BPNN algorithm increased with the training data and converged when the training number exceeded 100; the recall and precision rates reached 86.58 ± 0.43% and 85.12 ± 0.31%, respectively (Supplementary Fig. S6). In contrast, the F1 scores of the CNN algorithm continually increased; the recall and precision rates reached 91.03 ± 0.22% and 90.44 ± 0.16%, respectively (Supplementary Fig. S7). The GA F1 scores increased rapidly during optimization but remained relatively low once the optimization process reached a plateau; the recall and precision rates reached 85.54 ± 0.48% and 75.41 ± 0.34%, respectively (Supplementary Fig. S6).

3.4 Comparison between ImageCN and other algorithms

To evaluate the detection performance of our model, we manually annotated the ground truth of nine different sets of imaging data and compared the detection performance of ImageCN to other algorithms: deep-calcium, a fully convolutional networks-based algorithm, and CNMF, a spatiotemporal processing algorithm. As demonstrated in Table 1, ImageCN showed a significantly higher performance than the other two algorithms. Moreover, when we evaluated the ability to detect morphology and activity features separately, deep-calcium and CNMF showed advantages in one aspect but not the other. These results showed the necessity of combining these two significant features of 2p imaging data.

Table 1.

Comparison of the performance of different algorithms

Method	Recall	Precision	F1 score
ImageCN	0.89±0.02	0.82±0.02	0.85±0.01
Deep-calcium	0.71±0.02	0.72±0.04	0.70±0.02
CNMF	0.68±0.05	0.54±0.06	0.59±0.04

3.5 Detection of calcium spikes

After the detection of the neurons, the fluorescence trace of each neuron was extracted and processed using the piecewise linear representation method, which is a type of high-level, time-series representation used to detect turning points (peaks or troughs) of time-series data by extracting important points (Fink and Pratt, 2004) (Supplementary Fig. S8). We constructed another CNN model to pick the spikes, Figure 1A shows an example of the final output of our analysis pipeline processing in the typical 3D (x, y and time) functional imaging data format. Compared with manually annotated ground truth of nine different sets of imaging data, the recall rate, precision and F1 score of calcium transient detection reached 88.82 ± 1.03%, 78.64 ± 0.76% and 82.58 ± 0.45%, respectively.

4 Conclusions

We developed a semiautomatic analysis pipeline for 2p imaging data with ImageCN. This pipeline offers a valuable tool for mining the calcium signals from time-lapse 2p imaging data and converting them to digitalized data. Both active and inactive neurons are detected automatically, and our results demonstrated that the combination of activity features and morphological structures significantly boosts the performance of detection (Fig. 1D). ImageCN also offers a spike detection-function, which generates spike trains for further analysis.

Funding

This work was supported by grants from the National Basic Research Program of China [2017YFA0105201]; the National Science Foundation of China [31670842]; the Beijing Municipal Science & Technology Commission [Z181100001518001]; Support Project of High-level Teachers in Beijing Municipal Universities in the Period of 13th Five–year Plan [CIT&TCD20190334]; and Beijing Natural Science Foundation Program and Scientific Research Key Program of Beijing Municipal Commission of Education [KZ201910025025].

Conflict of Interest: none declared.