Uncertainty-Gated Min-Cost Flows for In Vivo NanoScale Synaptic Plasticity Tracking

Abstract

Synapses are the fundamental unit of neural connectivity. They exhibit dynamic functional and structural changes that enable the brain to learn, adapt, and form memories. Recent advances in endogenous protein fluorescent labeling offer an opportunity to image synaptic strength in vivo and thus study the mechanisms underlying adaptive neural computation in living mice. Studying synaptic dynamics requires tracking individual signals of small, densely packed synapses over days while they change in size, position, and intensity between imaging sessions, and may even appear/disappear entirely. Tracking >100,000 dynamic, submicrometer particles is difficult even for state-of-the-art algorithms. Moreover, most algorithms rely on an isotropic uncertainty ball, assigning equal weight to the lateral plane (XY) and to the noisier axial dimension (Z), leading to poorer performance. To address these challenges and accurately track synapses in vivo, we developed SynTrack. We formulated SynTrack as a Maximum A Posteriori estimation problem under the anisotropic uncertainty ball, along with a fully temporally connected spatio-temporal graph to overcome long-term occlusions. SynTrack achieves a mean track length of 0.51 μm with a Multiple Object Tracking Accuracy (MOTA) score of 88.8%, on par with MOTA scores of expert annotators but with massively increased speed and scalability. Over two weeks, we successfully track 65,000 synapses in 5.6 out of 8 imaging sessions on average, with 20,000 synapses being tracked in at least seven sessions. We present SynTrack as a state-of-the-art algorithm capable of high-resolution and fidelity tracking of synapse dynamics in behaving mice with unprecedented detail.

I. INTRODUCTION

Synapses are densely packed sub-micron structures that connect pairs of neurons and enable propagation of electrical signals across vast ensembles of neurons via synaptic neurotransmission. As animals form memories, adapt to their environment, and learn to perform new tasks, the molecular composition of synapses changes by strengthening or weakening, as well as creating or deleting synaptic connections between specific neurons. This process of dynamic remodeling of synapses is called synaptic plasticity, which has long been a central focus in neuroscience and is thought to underlie many forms of learning, memory, and behavioral adaptation.

AMPA-type glutamate Receptors (AMPARs) are known to be heavily involved in synaptic plasticity, specifically in long-term potentiation and depression [Malinow and Malenka, 2002, Huganir and Nicoll, 2013]. Given the critical role plasticity plays in the brain, the ability to track the timecourse and spatial orchestration of synaptic plasticity in living animals would significantly advance our understanding of this central function of the healthy brain, allow us to observe how synaptic deficits are manifested in neurological disease, and pave the way to understand new learning rules in artificial systems. However, studies of the computational impacts of plasticity have largely been theoretical [Morris et al., 1989, Fusi et al., 2000, Bicknell and Latham, 2025, Aitchison et al., 2021] due to the difficulty of observing and following the dynamics of hundreds of thousands of synapses in vivo across long time-scales. Only recently have microscopy, biological, and algorithmic tools advanced to the point of enabling the resolution of submicrometer-sized structures across multiple days, thus opening the door to such studies.

Nevertheless, the difficulties posed by in vivo data add significant resolution and motion challenges. Previous work imaging changes in AMPAR expression during learning tasks have been restricted to observing a sparse set of synapses (less than ≈1000 synapses per mouse [Roth et al., 2020]) or ex vivo studies using confocal microscopy in postmortem tissue [Rao-Ruiz et al., 2021]. More recently, new transgenic mouse lines [Graves et al., 2021], combined with multiphoton imaging [Helmchen and Denk, 2005] and novel algorithms for image enhancement [Xu et al., 2023] and segmentation [Chen et al., 2025], have reported longitudinally imaging of hundreds of thousands of synapses in vivo (Fig. 1). Such datasets finally provide faithful observations of large-scale synaptic changes, though identifying individual synapses and tracking their changes in strength over time (i.e. synaptic plasticity) with high fidelity remains a significant hurdle.

Fig. 1:

The synapse tracking problem. A) Synapse imaging, detection and tracking pipeline: after longitudinal 2-photon (2p) imaging in vivo, imaging volumes are aligned with affine registration, denoised with XTC [Xu et al., 2023], and synapses are detected with MR-CNN [Chen et al., 2025]. SynTrack was developed to link individual synapse detection between days to create longitudinal tracks. B) Snapshot of 3D imaging volume comprising of 65,000 synapses in a SEP-GluA2 mouse cortex. Pink and orange squares respectively represent the location of the XY and XZ planes shown in C. C) Longitudinal images in the XY (top) and XZ (bottom) planes after affine registration. Sum projection of 4 slices (1.33 μm thickness total). The lower Z resolution resulting from the Point Spread Function (PSF) of 2p microscopy underscores the need for SynTrack to take into account the larger positional uncertainty of synapses in the Z dimension.

Several key challenges make tracking densely labeled synapses in vivo difficult. First, the Point-Spread Function (PSF) inherent to optical imaging achieves poorer resolution along the axial dimension (Z) compared to XY planes, rendering an anisotropic image. This results in higher uncertainty of synapse structure and position in Z, and a tracking algorithm must be able to appropriately down-weight this noisier Z-dimension. While the dense signals of modern transgenic lines that label all excitatory synapses in the brain provide a unique approach to understand how synaptic plasticity drives cognition, the density of labeled synapses presents a challenge for matching and tracking hundreds of thousands of individual synapses across imaging sessions (Fig. 1b). Moreover, individual synapses lack distinctive features, and thus identity information is principally derived from their spatiotemporal location relative to other synapses (Fig. 1c) rather than from distinguishing shape or textures. The high dimensionality and limited individuality of synapse shape is further exacerbated by the limited Signal-to-Noise Ratio (SNR) inherent to in vivo imaging, which can impact the accuracy of localizing or even detecting a given synapse on one or more sessions. Thus, a tracking algorithm must be able to be re-identify missing synapses between days. Finally, as data are acquired by imaging large volumes of brain tissue, synapse tracking must be rendered in 3D.

Given these challenges and the novelty of large-scale synaptic imaging, the published synapse tracking literature has focused on manually tracking a small number of synapses from sparsely labeled samples by identifying their location relative to an anatomical anchor, e.g., a spine [Roth et al., 2020]. Beyond synapse tracking, there is a rich body of adjacent literature on tracking other objects, such as cells, wherein the most common paradigm consists of running an object detector followed by a linking algorithm, e.g., Nearest Neighbor [Sugawara et al., 2022], Hungarian Matching [Jaqaman et al., 2008, Fazli et al., 2018], or Deep Learning [Arbelle et al., 2018, Ben-Haim and Raviv, 2022]. Brain-wide imaging and tracking of extremely large numbers of neurons has been achieved, [Chen et al., 2024, Winnubst et al., 2019] lending support to the potential feasibility of tracking similar numbers of synapses in living mice across time. Unfortunately, current methods fall short of enabling accurate synaptic tracking. Current trackers, often designed for other domains such as tracking pedestrians, cannot simultaneously handle the lower signal-to-noise levels, anisotropic uncertainty from the imaging system, large number of objects to track, and lack of distinguishing features within synapse imaging datasets.

To meet the challenges of accurate synaptic tracking in living mice, We introduce a min-cost flow algorithm for in vivo tracking of hundreds of thousands of synapses: SynTrack. Our novel algorithm a) directly estimates an anisotropic PSF uncertainty ball from either labeled or unlabeled data, b) solves the min-cost-flow problem by using a cost-scaling algorithm to find the optimal association using information from all frames, and c) is computationally efficient enough to align hundreds of thousands of synapses in minutes. These qualities make SynTrack especially powerful in handling missing detections and re-identifications of densely packed synapses in living mice.

Contributions:

We introduce a novel parametrically flexible uncertainty estimation and gating procedure (Proposition 1) for arbitrary motion distributions. This is used to generate a sparse spatial, dense temporal min cost flow tracking graph which enables accurate synaptic tracking. We introduce a block coordinate descent procedure to perform label-free estimation of the uncertainty ball, which performs on par with labeled data from human experts. Finally, we use SynTrack on a longitudinal imaging dataset to successfully extract tracks for hundreds of thousand synapses over two weeks, achieving an unprecedentedly detailed visualization of synaptic changes over time in living mice. We demonstrate that tracks extracted via SynTrack recover a behavioral perturbation given to the mouse.

II. BACKGROUND AND RELATED WORK

Object tracking has a rich history. Starting with the seminal filtering works of [Kalman, 1960] on recursive bayesian posterior state estimation for a single target with gaussian noise and [Doucet et al., 2000] which approximates the posterior of a single target by particles to handle nonlinear transitions and noise. This was soon extended to multi-object tracking by Multiple Hypothesis Tracking framework of [Reid, 1979] and Joint Probabilistic Data Association framework of [Fortmann et al., 1983]. Note that these approaches were primarily designed for an online setting, and often restricted to tracking very few objects.

As an alternative to online methods, min-cost-flow methods were originally proposed in the context of offline pedestrian tracking. The classic paper by [Zhang et al., 2008] introduced network flow for use in multi-object tracking. Note that the Min-Cost-Flow-Based Algorithm in [Zhang et al., 2008] cannot be used off the shelf because it was designed to track pedestrians with appearance-based similarity loss that does not work to model synaptic motion. Unlike pedestrian tracking, which includes far more time points than objects, synaptic tracking involves extremely large numbers of objects over fewer time points. Thus, we need to explicitly handle a large numbers of missing detections. [Pirsiavash et al., 2011] and [Butt and Collins, 2013] reported results on min-cost flow with occlusion modeling for k frames to handle missing detections. We adopted a similar approach for SynTrack.

While literature in the field of in vivo synaptic tracking is essentially non-existent due to the novelty of such datasets, there a rich body in the related field of in vivo cell tracking. The simplest strategy in this domain utilizes nearest neighbor for data association [Sugawara et al., 2022]. Additional work has used greedy pairwise matching between frames based on Hungarian Matching [Jaqaman et al., 2008, Fazli et al., 2018]. More recent approaches employed deep-learning approaches for linking [Ben-Haim and Raviv, 2022]. While deep-learning algorithms have achieved the state of the art in segmentation, algorithms for linking detections requires dense annotations that are challenging to acquire for volumetric data. Furthermore, such methods also demonstrate poor generalizability to datasets that are “out of distribution” with respect to training data. In fact, a recent 10-year benchmark on the famous cell tracking challenge [Maška et al., 2023] has reported that traditional approaches for data association can be as effective as deep learning. Along with the relative homogeneity of synapse appearance, this was one driving factor behind our choice to follow a hybrid data association strategy (using continuous normalizing flow for learning motion distribution and min-cost flow for data association). Padfield et al. [2011] and Löffler et al. [2021] proposed a min-cost-flow tracking algorithm with appear/disappear nodes to handle cells appearing and disappearing due to microscope movement. Note that this approach does not handle re-identifications, which are quite common in longitudinal synapse imaging.

We note that the majority of the algorithms discussed here have used an isotropic uncertainty ball that treats movement in each dimension equally. The isotropic assumption, while helpful for simplifying the model and appropriate in many applications, is insufficient for modeling 3D in vivo synaptic data. Specifically, the optical properties of the point-spread function (PSF) stretch the data axially, producing an anisotropic uncertainty ball. Moreover, while past methods have relied on object features for tracking, such as appearance-based models [Zhang et al., 2008], synapses have minimal distinguishing features. Finally, many prior methods require labeled data, which is difficult to accurately annotate in volumetric synapse tracking at scale. Thus we require that our tracking solution have 1) the ability to capture anisotropic uncertainty, 2) be minimally reliant on object features, and 3) require minimal-to-no labeled data, in addition to 4) being scalable to 100,000+ objects and 5) be able to re-identify objects that are missing at a handful of intermediary time-points. By achieving all of these qualities, SynTrack makes a significant contribution to the state of the art.

III. METHOD

A. Registration Formulation and Denoising

Given a collection of volumetric images $ℐ=\left\{{ℐ}_1,\dots ,{ℐ}_T\right\}$, we denoised each dataset independently using a previously developed autoencoder that was specifically designed to enhance in vivo two-photon synaptic imaging [Xu et al., 2023]. The image enhancement reduces background fluorescence levels and shot noise, making synapses more readily detectable. Once each dataset was denoised, we performed gross affine registration of the imaged volumes between sessions. Affine registration corrects for the large-scale motion in in vivo imaging due to, e.g., breathing or blood pressure changes. Specifically, given a set of voxel grid coordinates $\mathrm{\Omega }\subset {\mathbb{R}}^3$, we solve:

$${\widehat{A}}_t,{\widehat{b}}_t=\mathrm{a}\mathrm{r}\mathrm{g}\underset{A_t,b_t}{\mathrm{m}\mathrm{i}\mathrm{n}}\sum _{x\in \mathrm{\Omega }}{({ℐ}_{\text{temp}}(x)-{ℐ}_t\left({\varphi }_t^{-1}(x)\right))}^2$$ (1)

Here ${\varphi }^{-1}$ is the affine mapping function parameterized as ${\varphi }_t^{-1}(x):=A_{t}x+b_t$. $I_t\circ {\varphi }^{-1}$ is computed via interpolation for locations off the grid. While affine registration roughly aligns sessions, primarily based on large-scale features such as dark areas corresponding to vasculature, it does not account for the microscopic movements and fluorescence changes that must be tracked using SynTrack.

B. Synapse detection

To detect and segment synapses, we used our previously developed modified Mask-R-CNN (MR-CNN) [Chen et al., 2025] that is designed to efficiently identify hundreds-of-thousands of objects in volumetric data. MR-CNN extracts each synapse’s location and brightness information by simultaneously identifying which voxels in the volume belong to any synapse, along with a per-synapse mask. From these masks we extracted the synapse’s location as defined by the centroid of the 3-dimensional segmentation, converted to physical coordinates.

C. Tracking: Network flow

1). Probabilistic Model:

For tracking, we begin with the formulation of Zhang et al. [2008], and expand on this work to address two main challenges: 1) handling anisotropic uncertainty in synapse locations and 2) re-identifying synapses that disappear and reappear despite the lack of distinct appearance cues. In this formulation we are given a set of detections $𝒟=\left\{d_1,\dots ,d_I\right\},d_i=\left(x_i,t_i\right)$ with $x_i\in {\mathbb{R}}^3$ being the spatial coordinate and $t_i\in \{1,\dots ,T\}$ the time associated with the $i$’th detection. We define the following two functions: space : $𝒟\to {\mathbb{R}}^3,\mathrm{s}\mathrm{p}\mathrm{a}\mathrm{c}\mathrm{e}\left(d_i\right)=x_i$ to extract the 3d position of a detection and the function time : $𝒟\to \{1,\dots ,T\}$, $\mathrm{time}\left(d_i\right)=t_i$ to get the associated time. Because of the homogeneity of synapse appearances, we primarily focus on the spatial location for tracking.

We begin with the definition of tracks. Let $𝒯=\{{𝒯}^1,\dots ,{𝒯}^K\}$ be a set of $K$ non-overlapping tracks where each track is defined as ${𝒯}^k=({𝒯}_1^k,{𝒯}_2^k,\dots ,{𝒯}_{n_k}^k),{𝒯}_l^k\in 𝒟$. We enforce time ordering of tracks: $\mathrm{time}({𝒯}_l^k)<\mathrm{time}({𝒯}_{l+1}^k)\forall l=\left\{1,\dots ,n_k-1\right\},\forall k=\{1,\dots ,K\}$. We also disallow two tracks from covering the same detection, i.e ${𝒯}^k\cap {𝒯}^{k^{\prime }}=\varnothing ,\forall k\ne k^{\prime },k,k^{\prime }=\{1,\dots ,K\}$.

Two sources of uncertainty can influence how tracks are inferred from the data: spatial uncertainty in the location of detections, and the probability that a synapse is detected at all in a given session. To model these sources of uncertainty we define 1) $\beta$ as the probability of successfully detecting a synapse and 2) $\mathrm{\Sigma }$ as the parameter associated with spatial uncertainty. $\mathrm{\Sigma }$ can encode, for example, a spatial covariance matrix or weights of a neural network encoding a nonlinear uncertainty probability distribution. $\mathrm{\Sigma }$ and $\beta$ effectively parametrize the prior over tracks, and we are interested in using said prior to compute and optimize the posterior of tracks given the data $p(\mathrm{\Sigma },\beta ,𝒯\mid 𝒟)$ so that we can estimate the tracks and parameters via maximum a posteriori (MAP) estimation. Specifically we can use Bayes rule as:

$$p(𝒯,\mathrm{\Sigma },\beta \mid 𝒟)\propto p(𝒟\mid 𝒯)p(𝒯,\mathrm{\Sigma },\beta )$$ (2)

$$p(𝒯,\mathrm{\Sigma },\beta )=p(𝒯\mid \mathrm{\Sigma },\beta )p(\mathrm{\Sigma })p(\beta )$$ (3)

$$p(𝒯,\mathrm{\Sigma },\beta \mid 𝒟)\propto p(𝒟\mid 𝒯)p(𝒯\mid \mathrm{\Sigma },\beta )p(\mathrm{\Sigma })p(\beta ).$$ (4)

Note that in this formulation the data depends only on the tracks, which in turn depends on the uncertainty parameters.

Leveraging this posterior thus requires defining the likelihood and prior probabilities. Detections are assumed to be IID distributed: $p(𝒟\mid 𝒯)=\prod _{d_i\in 𝒟}p\left(d_i\mid 𝒯\right)$. The likelihood of each detection can be written as a Bernoulli distribution ${\theta }_i^{{\mathrm{𝟙}}_{𝒫}\left(d_i\right)}{\left(1-{\theta }_i\right)}^{1-{\mathrm{𝟙}}_{𝒫}\left(d_i\right)}$ depending on whether or not it belongs the partial cover of tracks $𝒫={\cup }_{k=1}^K{𝒯}_k,𝒫\subseteq 𝒟$, where ${\theta }_i$ encodes the object detector confidence.

To capture the probabilistic continuation of a track into subsequent datasets, we model the prior probability of ${𝒯}^k$ in Equation (2) as a factorizable Markov chain $p(𝒯\mid \mathrm{\Sigma },\beta )=\prod _{k=1}^{K}p({𝒯}^k\mid \mathrm{\Sigma },\beta )$. Each track can further be written as: $p({𝒯}^k\mid \mathrm{\Sigma },\beta )=p_{\text{birth}}\left(\prod _{l=1}^{n_k-1}p({𝒯}_{l+1}^k\mid {𝒯}_l^k,\mathrm{\Sigma },\beta )\right)p_{\text{death}}$. By defining the spatial gap as $\mathrm{\Delta }{x}_{k,l}=\mathrm{s}\mathrm{p}\mathrm{a}\mathrm{c}\mathrm{e}({𝒯}_{l+1}^k)-\mathrm{s}\mathrm{p}\mathrm{a}\mathrm{c}\mathrm{e}({𝒯}_l^k)$ and temporal gap as $\mathrm{\Delta }{t}_{k,l}=\mathrm{time}({𝒯}_{l+1}^k)-\mathrm{time}({𝒯}_l^k)$, we further factor the conditional density as the product of the probabilities over temporal and spatial displacements:

$$p\left(T_k^{l+1}\mid T_k^l\right):=p_{\mathrm{\Delta }x}\left(\mathrm{\Delta }{x}_{k,l}\mid \mathrm{\Sigma }\right)\mathrm{G}\mathrm{e}\mathrm{o}\mathrm{m}\left(\mathrm{\Delta }{t}_{k,l}\mid \beta \right)$$ (5)

The above expressions define the probability of tracks given the time- and space- separation of detections across sessions. To practically use these probabilities we need to a) estimate an unknown spatial conditional density $p_{\mathrm{\Delta }x}\left(\mathrm{\Delta }{x}_{k,l}\mid \mathrm{\Sigma }\right)$ so that we can properly compute graph weights and b) decide on a good threshold to gate weights based on this density to sparsify the graph and improve the computational efficiency when scaling to extremely large numbers of tracks. In the case where $p_{\mathrm{\Delta }x}\left(\mathrm{\Delta }{x}_{k,l}\mid \mathrm{\Sigma }\right)$ is normally distributed, we would have exact expressions for the probabilities and could perform a simple $3\sigma$ truncation to retain the majority of the probability mass. However, $p_{\mathrm{\Delta }x}\left(\mathrm{\Delta }{x}_{k,l}\mid \mathrm{\Sigma }\right)$ is unknown and not necessarily analytically tractable, let alone normally distributed, removing the utility of a simple $3\sigma$ threshold. Instead, we leverage advances in normalizing flows [Kobyzev et al., 2020] to allow us to start with a simpler Gaussian density to capture the uncertainty in more complex situations.

Proposition 1 (Nonlinear gating via normalizing flow).

Given a random variable $\mathrm{\Delta }z\in {\mathbb{R}}^3$ with density $p(\mathrm{\Delta }z)=𝒩\left(0,I_{3\times 3}\right)$ and a diffeomorphic transformation $\varphi :{\mathbb{R}}^3\to {\mathbb{R}}^3,\mathrm{\Delta }x=\varphi (\mathrm{\Delta }z)$. Consider a $r$ radius ball $S(r)=\left\{\mathrm{\Delta }z\in {\mathbb{R}}^3\Vert \Vert \mathrm{\Delta }z\Vert \le \right.r\}$ as well as $r^{\alpha }$, the radius of ball containing $\alpha$ probability mass $r^{\alpha }={\mathrm{i}\mathrm{n}\mathrm{f}}_r\{r\in {\mathbb{R}}_{\ge 0}\mid {\int }_{S(r)}{p}_{\mathrm{\Delta }z}(\mathrm{\Delta }z)d(\mathrm{\Delta }z)\ge \alpha \}$. The ball corresponding to this radius is given by: $B_{\mathrm{\Delta }z}^{\alpha }=S\left(r^{\alpha }\right)$. The density $p_{\mathrm{\Delta }x}$ defined via change of variables $p_{\mathrm{\Delta }x}=\left(p_{\mathrm{\Delta }z}\circ \right.\left.{\varphi }^{-1}\right)\left|\mathrm{d}\mathrm{e}\mathrm{t}D{\varphi }^{-1}\right|$ has $B_{\mathrm{\Delta }x}^{\alpha }=\varphi \left(B_{\mathrm{\Delta }z}^{\alpha }\right)$ as an $\alpha$ likelihood ball.

Remark 1 implies that one way to perform $\alpha$ confidence truncation of $p_{\mathrm{\Delta }x}$ is to check if $\mathrm{\Delta }x\in B_{\mathrm{\Delta }x}^{\alpha }$ (or equivalently ${\varphi }^{-1}(\mathrm{\Delta }x)\in B_{\mathrm{\Delta }z}^{\alpha }$).

Proof.

Consider small volumes $d(\mathrm{\Delta }z)$ and $d(\mathrm{\Delta }x)$ in the latent space and ambient space, respectively. These are related by $d(\mathrm{\Delta }z)=\left|\mathrm{d}\mathrm{e}\mathrm{t}D{\varphi }^{-1}\right|d(\mathrm{\Delta }x)$. Thus, the likelihood inside the ball $B_{\mathrm{\Delta }x}^{\alpha }$ can be written as

$$\begin{gathered} {\int }_{B_{\mathrm{\Delta }x}^{\alpha }}{p}_{\mathrm{\Delta }x}(\mathrm{\Delta }x)d(\mathrm{\Delta }x)={\int }_{\varphi \left(B_{\mathrm{\Delta }z}^{\alpha }\right)}\left(p_{\mathrm{\Delta }z}\circ {\varphi }^{-1}\right)\left|\mathrm{d}\mathrm{e}\mathrm{t}D{\varphi }^{-1}\right|d(\mathrm{\Delta }x) \\ ={\int }_{B_{\mathrm{\Delta }z}^{\alpha }}{p}_{\mathrm{\Delta }z}(\mathrm{\Delta }z)d(\mathrm{\Delta }z)=\alpha \end{gathered}$$ (6)

□

2). Examples of ${\mathit{B}}_{\mathbf{\Delta }\mathit{x}}^{\mathit{\alpha }}$ for different parameterizations of $\mathit{\varphi }$:

Proposition 1 allows us to perform gating for arbitrary probability densities beyond the normal distribution. In this section, we consider three examples of diffeomorphic transformations and how to perform truncation in each case:

1. A Diagonal Scaling (Isotropic) Transformation: $\varphi (\mathrm{\Delta }z)=\sigma I\mathrm{\Delta }z$. The $\alpha$ confidence truncation for this is given by the isotropic ball: $B_{\mathrm{\Delta }x}^{\alpha }=\left\{\mathrm{\Delta }x\in {\mathbb{R}}^3|\Vert \mathrm{\Delta }x|\mid \le \right.\left.r^{\alpha }\sigma \right\}$.

2. A Linear (Anistropic) Transformation: $\varphi (\mathrm{\Delta }z)={\mathrm{\Sigma }}^{\frac{1}{2}}\mathrm{\Delta }z$ with $\mathrm{\Sigma }={\mathrm{\Sigma }}^{\frac{1}{2}}{\mathrm{\Sigma }}^{\frac{1}{2}}$. The $\alpha$ confidence truncation of this is given by the Mahalanobis Ball: $B_{\mathrm{\Delta }x}^{\alpha }=\{\mathrm{\Delta }x\in \left.{\mathbb{R}}^3|{\left(\mathrm{\Delta }{x}^T{\mathrm{\Sigma }}^{-1}\mathrm{\Delta }x\right)}^{\frac{1}{2}}\le r^{\alpha }\right\}$.

3. A Continuous Normalizing Flow: Given a smooth trajectory $l:[0,1]\to {\mathbb{R}}^3,t\mapsto l(t)$ governed by the stationary dynamics $\frac{dl}{dt}=v(l)$, we parameterize $\varphi$ as a flow of this ODE: $\varphi \left(\mathrm{\Delta }{z}_0\right)=l(1)$, $l(0)=\mathrm{\Delta }{z}_0\forall z_0\in {\mathbb{R}}^3$ [Chen et al., 2018]. Unlike the other two cases, the $\alpha$ confidence truncation of this does not have a closed formed expression and involves computing ${\varphi }^{-1}$ via flowing the ODE backwards $B_{\mathrm{\Delta }x}^{\alpha }=\left\{\mathrm{\Delta }x\in {\mathbb{R}}^3‖{\varphi }^{-1}(\mathrm{\Delta }x)‖\le r^{\alpha }\right\}$.

3). Formulating MAP estimation as Min Cost Flow Problem:

We represent this probabilistic model as a directed graph $G=(V,E)$. We have a set of non negative capacities $u$ associated with each edge. We also have edge costs $c$ and a supply-demand $b$ on each vertex.

We first define two special nodes source at time 0 and sink at time T. We duplicate the set of detection nodes and call this ${𝒟}^{\mathrm{*}}$. Define the set of vertices: $V=source\cup 𝒟\cup {𝒟}^{\mathrm{*}}\cup \mathit{sink}$

We next define edges as per the following procedure: We first start with edges coming out of source $E_{\mathit{source}}=\{\left(source,v\right)\forall v\in 𝒟\}$. There is an edge from every node to sink $E_{\mathit{sink}}=\left\{\left(v^{\mathrm{*}},sink\right)\forall v^{\mathrm{*}}\in {𝒟}^{\mathrm{*}}\right\}$. We use detection edges to encode confidence of object detector $E_{\mathit{det}}=\left\{\left(v,v^{\mathrm{*}}\right)\forall v\in \right.\left.𝒟,v^{\mathrm{*}}\in {𝒟}^{\mathrm{*}},v=v^{\mathrm{*}}\right\}$. We connect temporal edges to all future time points: $E_{\text{temporal}}=\left\{\left(v,v^{\mathrm{*}}\right)\mid v^{\mathrm{*}}\in {𝒟}^{\mathrm{*}},v\in 𝒟,v[1]<\right.\left.v^{\mathrm{*}}[1],v^{\mathrm{*}}[1]-v[1]\in B_{\mathrm{\Delta }x}^{\alpha }\right\}$. The set of edges is thus defined as $E=E_{\mathit{source}}\cup E_{\mathit{sink}}\cup E_{\mathit{det}}\cup E_{\mathit{temporal}}$.

We also define an edge capacity function $c:E\to \mathbb{R}$ with $c(e)=-\mathrm{l}\mathrm{o}\mathrm{g}p(d\mid 𝒯)$ if $e\in E_{\det }$ where $e=(d,d^{\mathrm{*}}),c(e)=-\mathrm{l}\mathrm{o}\mathrm{g}{p}_{\mathrm{birth}}$ if $e\in E_{\mathrm{source}},c(e)=-\mathrm{l}\mathrm{o}\mathrm{g}{p}_{\mathrm{death}}$ if $e\in E_{\mathrm{sink}}$, and $c(e)=-\mathrm{l}\mathrm{o}\mathrm{g}p\left(d^{\prime }\mid d\right)$ if $e\in E_{\mathrm{temporal}}$ where $e=\left(d,d^{\prime }\right)$. We also define a node supply function $b:V\to \mathbb{R}$ with $b(v)=0$ if $v\in 𝒟\cup {𝒟}^{\mathrm{*}},b(v)=k$ if $v=\mathrm{s}\mathrm{o}\mathrm{u}\mathrm{r}\mathrm{c}\mathrm{e}$, and $b(v)=-k$ if $v=\mathrm{s}\mathrm{i}\mathrm{n}\mathrm{k}$. Let $f:E\to R$ such that $f(e)$ is the flow associated with edge $e$. Give a vertex $j\in V$, we define $\mathrm{i}\mathrm{n}\mathrm{f}\mathrm{l}\mathrm{o}\mathrm{w}(j)=\{(i,j)\in E\mid i\in V\}$ and $\mathrm{o}\mathrm{u}\mathrm{t}\mathrm{f}\mathrm{l}\mathrm{o}\mathrm{w}(j)=\{(j,k)\in E\mid k\in V\}$. The minimum cost flow problem is then defined as:

$$\begin{array}{l}\underset{f}{\mathrm{m}\mathrm{i}\mathrm{n}}\sum _{e\in E}c(e)f(e)\mathit{s.t.}0\le f(e)\le c(e)\\ \sum _{e^{\prime }\in \mathrm{outflow}(j)}f\left(e^{\prime }\right)-\sum _{e\in \mathrm{inflow}(j)}f(e)=b(i)\forall i\in V\end{array}$$ (7)

In order to perform label free estimation, we start with the log of the posterior

$$\begin{gathered} \mathrm{l}\mathrm{o}\mathrm{g}p(𝒯,\mathrm{\Sigma },\beta \mid 𝒟)=\mathrm{l}\mathrm{o}\mathrm{g}p(𝒟\mid 𝒯)+\mathrm{l}\mathrm{o}\mathrm{g}p(𝒯\mid \mathrm{\Sigma },\beta ) \\ +\mathrm{l}\mathrm{o}\mathrm{g}p(\mathrm{\Sigma })+\mathrm{l}\mathrm{o}\mathrm{g}p(\beta )+\mathrm{c}\mathrm{o}\mathrm{n}\mathrm{s}\mathrm{t}\mathrm{a}\mathrm{n}\mathrm{t} \end{gathered}$$

Optimizing this one variable at a time, we arrive at the following block coordinate ascent algorithm:

Algorithm 1.

Unlabeled Estimation of Motion Parameters

Require: Detections $D=\left\{d_i=\left(x_i,t_i\right)\right\}$, detection confidences $\left\{{\theta }_i\right\},{\mathrm{\Sigma }}^0,{\beta }^0$

Ensure: Estimated tracks $𝒯$, spatial uncertainty $\mathrm{\Sigma }$

1: repeat

2:  Track Update: ${𝒯}^{t+1}\leftarrow \arg {\mathrm{m}\mathrm{a}\mathrm{x}}_{𝒯}\mathrm{l}\mathrm{o}\mathrm{g}p(𝒟\mid 𝒯)+\mathrm{l}\mathrm{o}\mathrm{g}p\left(𝒯\mid {\mathrm{\Sigma }}^t,{\beta }^t\right)$

3:  Uncertainty Update: ${\mathrm{\Sigma }}^{t+1}\leftarrow \arg {\mathrm{m}\mathrm{a}\mathrm{x}}_{\mathrm{\Sigma }}\mathrm{l}\mathrm{o}\mathrm{g}p({𝒯}^{t+1}\mid \left.\mathrm{\Sigma },{\beta }^t\right)+\mathrm{l}\mathrm{o}\mathrm{g}p(\mathrm{\Sigma })$

4:  Detector Prob. Update: ${\mathrm{\beta }}^{t+1}\leftarrow \arg {\mathrm{m}\mathrm{a}\mathrm{x}}_{\beta }\log p({𝒯}^{t+1}\mid \left.{\mathrm{\Sigma }}^{t+1},\beta \right)+\log p\left(\beta \right)$

5:  $t\leftarrow t+1$

6: until convergence

IV. RESULTS

We tested SynTrack on both simulated and real datasets. Simulated particle flows allowed us to experiment with point spread function and test SynTrack’s ability to adapt to anisotropic PSFs in a controlled manner. Furthermore, annotating data manually is highly time-consuming and simulations offer a direct way to generate a large amount of test data with ground truth. We then tested on real biological data comprising of: a) one ground truth dataset with 198 tracks on synapses annotated independently by two expert annotators b) a much bigger dataset with ≈ 65,000 unannotated synapses.

We evaluated SynTrack using three standard metrics: #Switches counting how many times the tracker incorrectly swaps an object’s identity during tracking, IDF1 evaluating how coherently the tracker maintains temporal identity and MOTA measuring overall tracking accuracy by combining errors like missed detections, false positives, and ID switches [Bernardin and Stiefelhagen, 2008].

A. Simulation

We generated synthetic synaptic data by sampling a template of $I$ synapses over a unit square [0, 1]² from a homogeneous Poisson process. We first generate $x_i^{\mathit{template}}\sim \mathrm{P}\mathrm{o}\mathrm{i}\mathrm{s}\mathrm{s}\mathrm{o}\mathrm{n}\mathrm{D}\mathrm{i}\mathrm{s}\mathrm{k}\left(d_{\mathit{thresh}}\right)$ with $d_{\mathit{thresh}}=0.1$ to prevent particles from being to close together. The synapses exhibit small Brownian motion between days $x_i^t\sim 𝒩(x_i^{\mathit{template}},{\sigma }^{2}I)$ with $\sigma =0.02$. For all simulations, we generated 100 days of data. Of this, 50 were used for estimating the uncertainty distributions, while the remaining 50 were used for evaluation For each day, detections are sampled based on the imaging Point Spread Function (PSF): $p\left(y_1^t,\dots ,y_I^t\mid x_1^t,\dots ,x_I^t\right)=\prod _{i=1}^I\phi \left(y_i^t\mid x_i^t\right)$.

We modeled $\phi \left(y_i^t\mid x_i^t\right)$ as a normalized probability distribution, using one of two parameterizations: The first a gaussian $\phi \left(y_i^t\mid x_i^t\right)=𝒩\left(x_i^t,\mathrm{\Sigma }\right)$. In Figure 5, top row, we use a scaled parameterization with $\mathrm{\Sigma }=\left[\begin{array}{cc}0.01& 0.0001\\ 0.0001& 0.001\end{array}\right]$. True synapse locations were overlayed above with the PSF density for each synapse. Sampled detections are shown below. We computed differences for each synapse between successive days to compute the motion uncertainty ball (bottom left) which is normally distributed (Henze-Zirkler test). As shown in Table I, the isotropic model performed poorly across all three metrics. The anisotropic-Gaussian model gave less weight to the comparatively noisier Z dimension and relied more on XY and thus performed the second best, followed by the continuous normalizing flow-based model that had the highest MOTA and IDF1 and the lowest number of switches.

Fig. 5:

A) Synthetic data with normal gaussian PSF. Larger z uncertainty implies we should give lesser weight to z for tracking. The conditional motion distribution is also a multivariate normal distribution. B) Synthetic data with Laplace PSF and corresponding motion distribution. Conditional distribution of motion is non normal.

TABLE I:

Tracking Metrics for Gaussian PSF — Mean ± Std Dev across 100 trials. #Switches: ID swaps, MOTA: Multi Object Tracking Accuracy.

Algorithm	MOTA (%)	IDF1 (%)	#Switches
Anisotropic Gauss.	95.51 ± 1.66	89.18 ± 4.44	30.30 ± 12.66
Cont. Norm. Flow	95.59 ± 1.61	89.54 ± 4.24	30.25 ± 11.65
Isotropic Gauss.	90.89 ± 2.63	81.33 ± 4.42	55.71 ± 15.93

Next, we experimented with a Laplace PSF to see how well SynTrack adapted to non gaussian PSFs: $\phi \left(y_i^t\mid x_i^t\right)=ℒ(x_{i,0}^t,s_0)ℒ(x_{i,1}^t,s_1)$. In Figure 5, bottom row, we use $s_0=0.075,s_1=0.0375$. The uncertainty ball, shown in bottom left is not normally distributed (Henze-Zirkler test), $p<0.01$. Results for this experiment are reported in Table II. Here, the gap between continuous normalizing flow model and anisotropic gaussian is especially striking since the PSF and motion distributions are non normal.

TABLE II:

Tracking Metrics for Laplace PSF — Mean ± Std Dev across 100 trials.

Algorithm	MOTA (%)	IDF1 (%)	#Switches
Anisotropic Gauss.	88.18 ± 2.70	77.03 ± 4.19	75.25 ± 17.99
Cont. Norm. Flow	89.18 ± 2.50	77.22 ± 3.84	72.07 ± 16.40
Isotropic Gauss.	86.72 ± 2.74	75.23 ± 4.57	84.37 ± 17.67

B. Synapse imaging in mouse cortex

We next evaluated SynTrack on real, multi-day synapse imaging data of living mice. The complete dataset consisted of imaging fluorescently labeled synapses using two-photon microscopy. Briefly, this SEP-GluA2 transgenic mouse line enabled visualization of all GluA2-containing excitatory synapses throughout the brain [Graves et al., 2021, Xu et al., 2023]. This was achieved by transgenic labeling of all endogenous AMPAR GluA2 subunits with a green fluorescent tag: Super Ecliptic pHluorin (SEP). Thus, when excited by 910 nm light from a two-photon microscope through a cranial window surgically implanted above brain regions of interest, hundreds of thousands of individual GluA2-containing synapses appear as fluorescent puncta, whose fluorescent intensity is directly correlated with the functional strength of each synapse [Graves et al., 2021]. Data were collected over a 100μm³ volume for 8 imaging sessions spanning two weeks. Because the linear parameterization performed on par with nonlinear, we restricted ourselves to scaling/linear parameterizations for the synapse experiments.

A critical part of SynTrack is the uncertainty of how synapse locations changed between recording days, which was encoded in the spatial covariance matrix $\mathrm{\Sigma }$. Accordingly, estimating $\mathrm{\Sigma }$ was an essential step, and we explored both supervised and unsupervised approaches to estimate the spatial covariance matrix. In the supervised case, we used labeled tracks provided by two human expert annotators. We then took the difference between consecutive timepoints to estimate the uncertainty in motion between successive days. These are ${\mathrm{\Sigma }}_1^1,{\mathrm{\Sigma }}_1^2$. Next, we used block coordinate descent iterations described in Algorithm 1. Estimated covariance from this is referred to as ${\mathrm{\Sigma }}_{\mathrm{u}}^{\mathrm{*}}$.

To model missed detections, we fit a geometric distribution to the ground-truth tracks manually annotated by two experts. The fit distribution was used to estimate a missing probability from Expert 1, Expert 2, and Unlabeled Estimate ${\beta }_1^1=0.8,{\beta }_1^2=0.9,{\beta }_{\mathrm{u}}^{\mathrm{*}}=0.8$, which was used in the geometric distribution in the transition function.

Given $\beta$, we used the following heuristic to estimate the number of tracks. Assuming a coin flip model with $k$ true objects on a day, the number of observed detections $N$ will be $k\mathrm{*}\beta$. We thus set $k$ to be $\frac{N}{\beta }$. In the ground truth dataset of Table III, this was equivalent to setting $k=128$. In order to efficiently find pairs of neighbors $x_i$ and $x_j$ that are within $r^{\alpha }$ Mahalanobis ball of each other, we converted them to euclidean distances by linear transformation $z_i={\mathrm{\Sigma }}^{-\frac{1}{2}}{x}_i$ and a CKDTree implementation in scipy. In all experiments, we used $r^{\alpha }=3$ to achieve $\alpha =0.997$ probability truncation.

TABLE III:

Comparison against classical algorithms and human annotators.

Method	MOTA (%)	IDF1 (%)	#Switches
Expert 2 → Expert 1	87.8	82.7	91
SynTrack Labeled	89.8	83.3	76
SynTrack Unlabeled	89.0	80.4	82
SynTrack Isotropic	88.0	77.4	89
Hungarian Matching	84.7	74.3	114
Nearest Neighbor	76.5	73.3	174
Ilastik Tracking with Learning	24.6	43.7	120

Tracks generated by SynTrack have a mean spatial displacement of approximately 0.51 μm between consecutive days and were detected on an average of 5.6 out of 8 imaging sessions. Notably, the temporal and spatial statistics of SynTrack’s output closely match those derived from expert annotations, suggesting that the method produced biologically meaningful and consistent results, though at vastly improved speed compared to human annotators (a few seconds vs. 6–10 hours per imaging volume comprising of 200 synapses over 8 imaging sessions) .

Comparing Trackers:

To test SynTrack’s ability to “fit” a human annotator, in Table III, we used the estimate of our first Expert ${\mathrm{\Sigma }}_1^1$ as the uncertainty ball in SynTrack, using Expert 1’s own tracks as ground truth. As expected, SynTrack was able to fit these tracks well and achieve a MOTA of 89.8% with only 76 ID swaps. In contrast, Expert 2 compared with Expert 1 achieved a MOTA of 87.8% with 91 ID swaps, indicating more dissimilarity. Thus, SynTrack is closer to Expert 1 than Expert 2 compared with Expert 1.

In order to benchmark SynTrack, we compared it against the following baselines: Hungarian Matching (adapted from [Jaqaman et al., 2008] and augmented to handle missing detections), Nearest neighbor and Ilastik tracking with learning.

As seen in Table III, SynTrack outperformed other algorithms like Hungarian Matching (MOTA 84.7%) and Nearest Neighbor (MOTA 75.8%). Out of 198 total tracks in the 8-day ground truth dataset, 91 tracks were perfect matches with SynTrack, 90 tracks were perfect matches between the two annotators, while Hungarian only perfectly tracked 41. Comparing consecutive days where synapses were present on both days, we found that SynTrack performed better than Hungarian, which gradually decreased in accuracy over time. This was observed to be in line with previous findings in cell tracking literature [Maška et al., 2023] where global data association methods using information from multiple frames tends to outperform greedy methods.

One interesting feature of SynTrack was its ability to reidentify “missing” synapses between imaging sessions. In Figure 4, we compared an XY slice from the ground truth dataset against SynTrack. SynTrack achieved good correspondence with ground truth and was able to reidentify the highlighted synapse that went missing on Day 8 and Day 12, as defined by both expert annotators.

Fig. 4:

SynTrack successfully reidentifies synapses that go missing and reappear on consecutive days (highlighted by green box). Top row shows ground truth tracks labeled by expert annotator. Bottom row shows the corresponding synapses automatically tracked by SynTrack. Synapse segmentations are color-coded according to their assigned track, with matching colors across columns indicating the same synapse track.

We were interested in assessing the effect of temporal component of our spatiotemporal loss (Eq 5) in reducing ID swaps (i.e. synapse matching errors across imaging sessions). In Table IV, we add results for both a spatiotemporal loss as described in equation 5 as well as a purely spatial version of the loss without the geometric distribution. The results in Table IV, reveal that the geometric link function consistently limited ID swaps and increased the MOTA score, possibly by penalizing tracks with much larger gaps between trajectories.

TABLE IV:

Effect of different spatial uncertainty balls and geometric link function on SynTrack performance. All gating was done at 3σ. lab: labeled spatial uncertainty ball; un: unlabeled spatial uncertainty ball; E1: Expert 1; E2: Expert 2.

Method	Ref	MOTA (%)	IDF1 (%)	#Switches
E2	E1	87.8	82.7	91
SynTrack ${\mathrm{\Sigma }}_{\text{lab}}^1,{\beta }^1=0.8$	E1	89.8	83.3	76
SynTrack ${\mathrm{\Sigma }}_{\text{lab}}^1$	E1	89.4	83.9	79
SynTrack ${\mathrm{\Sigma }}_{\mathrm{l}}^2,{\beta }^2=0.9$	E1	88.7	80.0	84
SynTrack ${\mathrm{\Sigma }}_{\text{lab}}^2$	E1	88.6	80.1	85
SynTrack ${\mathrm{\Sigma }}_{\mathrm{u}\mathrm{n}}^{\mathrm{*}},{\beta }^{\mathrm{*}}=0.8$	E1	89.0	80.4	82
SynTrack ${\mathrm{\Sigma }}_{\text{un}}^{\mathrm{*}}$	E1	88.4	80.2	86
E1	E2	88.8	82.7	83
SynTrack ${\mathrm{\Sigma }}_{\text{lab}}^2,{\beta }^1=0.9$	E2	87.6	78.2	92
SynTrack ${\mathrm{\Sigma }}_{\text{lab}}^2$	E2	86.8	78.0	98
SynTrack ${\mathrm{\Sigma }}_{\text{lab}}^1,{\beta }^1=0.8$	E2	88.0	81.2	89
SynTrack ${\mathrm{\Sigma }}_{\text{lab}}^1$	E2	86.8	80.9	98
SynTrack ${\mathrm{\Sigma }}_{\mathrm{u}\mathrm{n}}^{\mathrm{*}},{\beta }^{\mathrm{*}}=0.8$	E2	87.0	77.4	97
SynTrack ${\mathrm{\Sigma }}_{\mathrm{u}\mathrm{n}}^{\mathrm{*}}$	E2	86.3	76.9	102

Next, we demonstrated that applying SynTrack to unlabeled data yields synapse tracking with accuracy on par with human annotators. As seen in Table IV, comparing human Expert 2 to Expert 1 as ground truth, Expert 2 had a MOTA of 87.8% along with 91 ID swaps. On the other hand, SynTrack with uncertainty ball ${\mathrm{\Sigma }}_u^{\mathrm{*}}$ achieved a MOTA of 89% with just 82 ID swaps. Thus SynTrack performed as well as agreement between two expert annotators. Furthermore, SynTrack with motion learned from the second annotator ${\mathrm{\Sigma }}_1^2$ also does well, with MOTA of 88.7% and 84 ID swaps, indicating that both labeled and unlabeled versions of SynTrack were effective at generalizing to a different annotator.

To further validate SynTrack, in Figure 3, we ran it on a larger volume. For each synapse within a given imaging session, synaptic strength was defined for each individual synapse as the summed fluorescence intensity of all voxels within the automatically segmented puncta. Synapses were then matched across imaging sessions, using SynTrack to “connect-the-dots” across sessions, generating 65,000 trajectories of each individual synapse. Changes in fluorescent intensity across each trajectory was used to report synaptic plasticity with single-synapse resolution (Figure 7), which has never been achieved at this scale. In order to perform dimensionality reduction of these trajectories, a natural tool of choice is non negative matrix factorization. In Figure 6, we show results from performing non-negative matrix factorization (NMF) on the tracks.

Fig. 3:

Qualitative Results from SynTrack. Synapse segmentations are color-coded according to their assigned track, with matching colors across columns indicating the same synapse track. White squares in the top row mark the regions shown as zoomed XY images in the middle row. Dotted line highlights the coordinate of the XY and XZ slices.

Fig. 6:

Low rank decomposition from NMF. Consistent pattern of decreasing activity in fluorescence in P1 along with spikes on Day 4 in P2 and Day 7 in P3 are observed.

We tried ranks from 1 to 4. A rank of 3 was found to be appropriate. The rank 3 factorization revealed three interesting patterns. P1 represented a steady drop in luminance, possibly encoding some sort of bleaching phenomena caused by multiple imaging sessions. P2 reached its peak activation on Day 4 while P3 had a peak on Day 7. P2 and P3 are especially functionally significant, since days 4 and 7 represent behavioral days where the strongest stimuli was presented to the mouse.

The CostScaling algorithm used by SynTrack has a polynomial computational complexity of $𝒪\left(n^{2}m\mathrm{l}\mathrm{o}\mathrm{g}(nC)\right)$ [Király and Kovács, 2012] for a graph with $n$ nodes and $m$ edges and $C$ being the largest edge weight. For efficiency, we used a C++ implementation of the Cost Scaling algorithm provided by the LEMON library. This resulted in a highly efficient tracker: running SynTrack on a dataset of 65,000 synapses over 8 imaging days took roughly 20 minutes on a laptop.

V. DISCUSSION

In this paper, we present SynTrack, an in vivo Synaptic Plasticity Tracking algorithm. Our results showcase the novel ability to accurately track changes in synaptic strength across dense networks of tens of thousands of submicron synapses across two weeks in living mice. Put simply, this is the most detailed and comprehensive illustration of synaptic plasticity ever published. Beyond the proof-of-concept demonstration here, SynTrack can be used to track the synaptic foundations of any behavior by imaging any brain region of interest, provided sufficient optical access using in vivo two-photon microscopy. More broadly, the ability to accurately track longitudinal changes in fluorescent puncta intensity could be extended to other similar questions in neuroscience and biology. For instance, SynTrack could be used to follow changes in dendritic spine morphology and number to report structural plasticity associated with learning. Alternatively, these tools could be useful in tracking axonal boutons across either learning or development, or other fluorescently labeled proteins of interest, illuminating how complex biological systems form and remodel over time.

Despite the efficacy of SynTrack, there were several limitations. One of the big challenges was handling the large detector missing probability ${\beta }_{\mathit{det}}=0.8$. We partially alleviated this by allowing skip connections with full temporal connectivity in the min cost flow graph. Next, we relied on a heuristic strategy for selecting number of tracks. We are interested in exploring approaches to estimate number of tracks from data in the future. We also restricted ourselves to linear registration which may still have some residual motion. Future work should focus on algorithms which combine tracking, nonlinear registration and detections, possibly by fitting images directly, and which also handle nonlinear registrations to further stabilize the motions.

Fig. 2:

A) We use a triangular template to simulate 6 noisy detections with deletes and reappearances:. The detection at $x_1$ disappears at Day 1 reappears at Day 4 at the location $x_5$. B) SynTrack deals with both misses and reappearances by allowing flows to skip ahead in time and C) recovers the template morphology.