A fully-automated multiscale kernel graph cuts based particle localization scheme for temporal focusing two-photon microscopy

Abstract

The temporal focusing two-photon microscope (TFM) is developed to perform depth resolved wide field fluorescence imaging by capturing frames sequentially. However, due to strong nonignorable noises and diffraction rings surrounding particles, further researches are extremely formidable without a precise particle localization technique. In this paper, we developed a fully-automated scheme to locate particles positions with high noise tolerance. Our scheme includes the following procedures: noise reduction using a hybrid Kalman filter method, particle segmentation based on a multiscale kernel graph cuts global and local segmentation algorithm, and a kinematic estimation based particle tracking method. Both isolated and partial-overlapped particles can be accurately identified with removal of unrelated pixels. Based on our quantitative analysis, 96.22% isolated particles and 84.19% partial-overlapped particles were successfully detected.

1. Introduction

Temporally focused two-photon microscopy (TFM), as an attractive imaging technique, has the ability to capture not only transverse plane but also depth-resolved image sequentially by exciting a large field-of-view without raster scanning of the focal spot. Due to its simplicity and rapid image acquisition capability, TFM has great potential to be adapted on biomedical imaging applications such as 3D super-resolution imaging, cellular dynamics imaging (CDIM) and phosphorescence lifetime imaging (PLIM) [1, 2]. Our TFM is first characterized with fluorescent nanospheres (hereinafter to be referred as particles) with diameter of 100 nm to observe the peripheral fringes in point spread function (PSF). The nanospheres we used are green fluorescent spheres with the excitation and emission peaks at 505 and 515 nm respectively (FluoSpheres carboxylate-modified, F8803, Thermo Fisher Scientific, Waltham, MA, USA). Spheres’ initial concentrations were diluted with distilled water at the ratio of 1:1000. The sphere samples are well suspended in the solution with the help of a vortex mixer before they are prepared in the slides and cover glasses. The whole process was finished at a dark environment to preserve the optical properties of samples and we scanned the sample in axial dimension over a range of 100 μm at 2 μm step.

However, the original video obtained by our TFM needs further processing for future research due to large amount of noises (including shot noise and read out noise) and diffraction rings from out-of-focus particles in captured images. In this study, we developed a fully-automated scheme to denoise TFM videos by a hybrid noise reduction technique with the combination of correlation and Kalman filter, segment isolated and partial-overlapped particles through multiscale kernel graph cuts algorithm and track particles vertically based on kinematic estimation method.

2. Materials and methods

2.1 Materials

Our video was captured through temporal focusing two-photon fluorescence microscope with 1144 frames based on continuous top-to-bottom scanning with the time interval 0.03s. Then we manually separated 4 volumes from the original video and each volume contains 286 frames from the top to the bottom end.

2.2 Methods

Our proposed localization scheme for particles is organized by the flowchart shown in Figure 1. For each independent volume, in order to decrease electrical and fluorescence noises, a correlation method combined with Kalman filter was applied every three frames to achieve rough noise reduction. Then an automated multiscale kernel graph cuts based segmentation algorithm was proposed to identify particles for each frame. Global cut and local cut were implemented individually to achieve high-quality segmentation results. Finally, we tracked the well-segmented particles through a kinematic estimation based particle tracking algorithm.

Figure 1.

Flowchart of our proposed 3D localization scheme

2.2.1. Noise reduction

Because of high noises levels in our original video, noise reduction is necessary to be served as the first step. In this subsection, we utilized a temporal correlation method, that is, each pixel with the same coordinates from three adjacent frames was multiplied with each other, which largely decrease random electrical noises without damaging the original shape of particles. Besides, a Kalman filter was used on processed frames to maximally take advantage of adjacent information. The iteration process includes predicting particles’ positions; estimating uncertainties based on the current frame and modifying the estimations when it marches to the next frame [3]. In this study, we set the noise variance estimate is 0.1 and the bias of the prediction is 0.8 respectively. After rough noise reduction, the contrast within particles was lower. However, graph cuts theory can solve the issue of low contrast through minimizing energy function [4].

2.2.2. Multiscale kernel graph cuts based particle segmentation

Large amount of noises are removed based on our hybrid noise reduction technique. However, there still remain residual noises as well as diffraction rings surrounding particles. In order to accurately segment particles, our particle segmentation technique can be separated into two steps. Firstly, globally locate particles in each frame with high tolerance of noises. Then segment individually for each local subregion obtained from global segmentation to achieve the refinement results.

Graph cuts formulate image segmentation work as an object labelling problem through energy minimization with a high tolerance of noises, which is an ideal solution to achieve initial segmented particles [5]. The conventional graph cuts method regards Gaussian distribution as the approximate distribution of data. However, it may cause imprecise result if the optimal model of data does not meet Gaussian distribution. In order to avoid restrictions caused by using general linear models, we utilized kernel graph cuts method as our proposed segmentation algorithm to optimally segment different particles. Kernel function aims to transform data in our observation field to a higher dimension so that some common linear models such as Gaussian distribution can be available in our images. The goal for graph cuts is to discover global and local weight value for each pixel within our image and then to maximize the energy function during label assignment based on max-flow min-cut theorem [6, 7]. In this paper, let δ be the kernel mapping transformation by RBF kernel function, G be the global energy and L be the local energy respectively. α is a positive coefficient between 0 and 1, θ assigns each pixel of an image to a certain label and k is total number of labels. The energy function of general graph cuts method can be denoted by Equation 1, 2 and 3:

$$E(\theta )=\alpha \ast \delta \left(G(\theta )\right)+\delta \left(L(\theta )\right)$$ (1)

$$G(\theta )={\mathrm{\sum }}_p{D}_p\left(\theta (p)\right)={\mathrm{\sum }}_l{\mathrm{\sum }}_p\left|{\mu }_l-I_p\right|,l=1,2,\dots ,k$$ (2)

$$L(\theta )={\mathrm{\sum }}_{\{p,q\}\in N}\left|I_p-I_q\right|$$ (3)

Assuming the mean value of each segmentation label is μ_l, the global energy G(θ) can be defined as the sum of conditional probabilities regarding to each pixel p in each label l. The intensity of each pixel is marked by I_p. Moreover, the local energy L(θ) was designated as sum of absolute intensity difference of pixel p and q in an eight-neighborhood set N.

Since the gray intensities of each particle in corresponding frame and the contrast between particles and background are different, there is little-to-no possibility to relay on one specific coefficient α alone. For global segmentation in this study, we set the value of α range from 0.003 to 0.012 with the interval 0.001 to maximally segment all the candidate objects and the number of labels k is 5. Viewing all the segmentation graphs as a whole, graphs with relative small α value includes large amount of noises and blurred particles with low contrast. On the contrary, large α value can identify bright particles without noises. In order to optimally erase noises and keep corresponding particles, we drew the energy diagram with cool and warm colors indicate low frequency and high frequency respectively. Based on our experiments, we regarded f= 4 as the threshold, that is, only the segmentation objects with the frequency higher than 4 will be maintained.

However, a lot of over-segmentation conditions occurred if only based on global segmentation, especially among short-distance particles. Therefore, the local segmentation was implemented to isolate particles and remove unrelated structures individually. For each subregion from global segmentation results, if the area of current subregion is larger than 200, local segmentation based on kernel graph cuts will be exploited. As a consequence, regions contain short-distance particles can be separated and interferential structures such as diffraction rings will be erased as well. We set the number of label k is 3 and α is 0.1 to distinguish background, unrelated structures and particles in this experiment.

2.2.3. Three dimensional particle tracking

After precise particle segmentation, a three-volume best-estimate tracking algorithm using kinematic prediction was utilized to tracking the 3D trajectory for segmented particles [8]. Firstly, it predicts the position of particle based on the former two volumes and considers that the moving length and direction of current particle is the same as previous one. Then it explores all the surrounding objects with predetermined threshold of radius R to find the optimal object with shortest deviation of distance between estimated and actual position. If one particle cannot be connected with any particles in the next frame, a new tracking will be created and initialized to track current particle [3, 9]. The algorithm repeats the two steps above until it comes to the last frame in current volume. In this paper, R was set to 20.

3. Results

Our proposed particle localization scheme was applied to video frames contained fluorescent nanospheres scanned by our temporal focusing two-photon microscope. All the computations were applied on a Windows 7 desktop with Intel Core^™2 Duo Processor E8600 @ 3.33 GHz and the average processing time for each frame is 26 seconds. The results are showed as follows. Figure 2 shows the noise reduction result of a random frame (frame 248 in volume 3, t = 8.27s). In Figure 2b, majority of electrical noises are decreased, and the bright objects with diffraction rings are fluorescent nanospheres. Figure 3 displays the result of global segmentation using multiparameter kernel graph cuts with an energy diagram (see Figure 2b). The warmer color indicates the higher frequency. Figure 4 represents the combination results of our global and local segmentation technique (marked with red dashed curves) using multiscale kernel graph cuts. Figure 4b and Figure 4c are comparisons of global segmentation and local segmentation. Furthermore, the subregions emphasized by solid blue squares in Figure 4b need to undergo local segmentation and corresponding results are shown in Figure 4c marked with solid green squares. The magnified views of corresponding subregions are enlarged in Figure 4d₁ and 4d₂. According to our results, our segmentation method can isolate partial-overlapped particles (subregion B and D in Figure 4) and remove unrelated pixels caused by noises and diffraction rings (subregion A, C and E in Figure 4) simultaneously. The particle tracking results for all four volumes are shown in Figure 5. (a), (b), (c) and (d) indicate the trajectory of particles with different colors in volume 1, 2, 3 and 4 respectively.

Figure 2.

Result of noise reduction. (a) Original frame. (b) Processed result

Figure 3.

Result of global segmentation using multiparameter kernel graph cuts. (a) Unsegmented image. (b) Energy diagram. (c) Segmentation result.

Figure 4.

Combination result of global and local segmentation

Figure 5.

Result of particle tracking

The accuracy of localization was evaluated by calculating the percentage of isolated and partial-overlapped particles that were correctly identified compared to the ground truth (manual segmentation). We randomly selected 100 frames with 1456 isolated and 329 partial-overlapped particles. After processing by our scheme, 1401 isolated and 277 partial-overlapped particles were precisely located so that the accuracy of detecting isolated and partial-overlapped particles are 0.9622 and 0.8419.

4. Conclusions

In this study, we developed a fully-automated particle localization scheme for temporal focusing two-photon microscope without human intervention. First, our hybrid noise reduction process reduces majority of noises from original video. In addition, multiscale kernel graph cuts based global and local segmentation technique identifies both isolated particles and particles in short distance with overlapping diffraction rings. In general, most particles can be precisely located from original video with large amount of noises based on our scheme. Despite the promising results, some limitations of our scheme still exist. Due to noises, the shapes of some segmented particles are slightly bigger than original particles. Besides, the accuracy of detecting partial-overlapped particles with similar intensity is relatively low. Overcoming these limitations forms one of our future goals. Last but not least, due to the unavoidable exposure time from camera, the particles’ positions between adjacent scanning frames are not perfectly matched. Therefore, vertical alignment based on tracking could be another improvement in the future to reconstruct the shape of biomedical microorganisms.