A multi-ensemble mean-field reduction method for networks of globally coupled phase oscillators with arbitrary parameter distributions

Abstract

Understanding the dynamical properties of coupled phase oscillator systems with heterogeneous oscillator frequencies has been a long-standing challenge of complex systems theory. While the seminal work of Ott and Antonsen dramatically improved our theoretical understanding of coupled phase oscillators for a small family of oscillator frequency distributions, we here present a mean-field reduction method for arbitrary frequency distributions. Our method leverages the drastic dimensionality reduction obtained for Lorentzian frequency distributions, and combines it with a data-driven multi-ensemble approach. As such, the method renders the Ott-Antonsen equations directly applicable to empirical distributions of phase oscillator frequencies, often achieving a drastic dimensionality reduction and allowing to study real-world physical and biological systems by means of stability, sensitivity, and bifurcation analyses.

Introduction —

Coupled phase oscillators have been broadly applied as a mathematical framework for studying physical, biological, and chemical systems with interacting periodic processes, such as power grids [1–3], chemical reactions [4–7], or neural networks [8–13]. As individual oscillators are generally non-identical, it is a key question across all these systems, how heterogeneity in the intrinsic oscillator frequencies affects the system dynamics. In this letter, we introduce a mean-field method for studying the dynamics of phase oscillator systems with arbitrary frequency distributions, which naturally lends itself to the study of oscillator heterogeneity.

We consider networks of globally coupled phase oscillators, where the phase of each oscillator evolves according to

$${\dot{\theta }}_i={\omega }_i+\frac{K}{N}\sum _{j=1}^{N}G\left({\theta }_j,{\theta }_i\right),$$ (1)

and is governed by an intrinsic frequency ${\omega }_i$ that introduces quenched disorder at the level of the oscillators. For the family of systems where the function $G$ takes the form

$$G\left({\theta }_j,{\theta }_i\right)=c_{-1}\left({\theta }_j\right)e^{-\text{i}{\theta }_i}+c_0\left({\theta }_j\right)+c_{+1}\left({\theta }_j\right)e^{\text{i}{\theta }_i},$$ (2)

the dynamics of the network converge to the OttAntonsen manifold [14–16]. The results of this letter are restricted to phase oscillator networks of the form given by Eqs. (1) and (2). Note that this family includes the set of systems where phase coupling only depends on the phase difference, i.e. $G\left({\theta }_j,{\theta }_i\right)=F\left({\theta }_j-{\theta }_i\right)$, for which the Ott-Antonsen ansatz conditions are met for coupling functions of the form $F(\varphi )={\text{Γ}}_0+r\sin (\varphi +\beta )$ [14, 16].

Assuming the special case of a Lorentzian distribution of intrinsic oscillator frequencies

$$\rho (\omega )=\frac{\text{Δ}}{\pi \left({[\omega -\overline{\omega }]}^2+{\text{Δ}}^2\right)},$$ (3)

the dynamics of the network on the OA manifold can be derived from Eq. (1), yielding

$$\dot{Z}=(\text{i}\overline{\omega }-\text{Δ})Z+\frac{K}{2}Z\left(1-|Z{|}^2\right),$$ (4)

with $Z(t)=\frac{1}{N}{\sum }_i\exp \left(\text{i}{\theta }_i\right)$ representing the complex-valued Kuramoto order parameter. This special case reduces the complexity of the network dynamics considerably, causing the network to approach either a fixed point that corresponds to a partially or completely synchronized state, or a fixed point that corresponds to an asynchronous state [14].

Since the seminal work of Ott and Antonsen, it has been a widely debated question whether globally coupled phase oscillator networks with frequency distributions $\rho (\omega )$ other than Eq. (3) also permit for a low-dimensional mean-field description and how their macroscopic dynamics differ from the Lorentzian case [15–20]. Scenarios that have been studied include Gaussian, uniform [19], bimodal [17], and trimodal oscillator frequency distributions [18], for example. In this letter, we consider a much broader family of oscillator frequency distributions $\rho (\omega )$, including multi-modal and non-symmetric distributions.

We provide a general mean-field reduction method for this class of networks that permits a drastic reduction in system dimensionality and enables the study of spatiotemporal pattern formation via methods from dynamical systems theory. We demonstrate our mean-field reduction method on representative model systems and on an open-source dataset of in vitro electrophysiological recordings from mouse neurons. We show that our method can (a) faithfully capture electrophysiological parameter distributions of different neuron types in different cortical layers, (b) predict the macroscopic dynamics of networks of recurrently coupled spiking neurons governed by those parameter distributions, and (c) reduce these networks to a low-dimensional set of mean-field equations. Using the latter, we characterize the bifurcation structure of spiking neural networks across different parameter distributions, demonstrating both the critical role of parameter heterogeneity in neural network dynamics and how our method allows it to be studied systematically. As our method applies to any system with sufficiently many globally coupled phase oscillators that are governed by Eqs. (1) and (2), it provides a powerful approach for studying the impact of the oscillator-intrinsic parameter distribution $\rho (\omega )$ on the the dynamics of such systems.

Lorentzian Mixture Approach —

Extending an approach we previously established for spiking neural networks [21], we approximate arbitrary distributions via a sum of Lorentzian distributions:

$$\rho (\omega )\approx {\rho }_M^{*}(\omega )=\frac{1}{\pi }\sum _{m=1}^M{w}_m\frac{{\text{Δ}}_m}{{\left(\omega -{\overline{\omega }}_m\right)}^2+{\text{Δ}}_m^2},$$ (5)

where ${\overline{\omega }}_m$ and ${\text{Δ}}_m$ are the centers and half-widths of $M$ individual Lorentzian distributions, respectively, and $w_m$ are the scalar weights associated with each Lorentzian distribution that satisfy $w_m\ge 0.0$ and ${\sum }_m{w}_m=1.0$. For any set of i.i.d. samples ${\omega }_1,{\omega }_2,\mathrm{..},{\omega }_N$ from $\rho (\omega )$, we define the loss function as the Cramer-von Mises statistic $W^2$:

$$𝓛\left(F,F_M^{*}\right)=W^2={\int }_{-\infty }^{\infty }{\left[F(\omega )-F_M^{*}(\omega )\right]}^{2}d{F}_M^{*}(\omega ),$$ (6)

i.e. the squared difference between the cumulative distribution function (CDF) $F_M^{*}(\omega )={\int }_{-\infty }^{\omega }{\rho }_M^{*}(\omega )d\omega$ and the empirical CDF $F(\omega )=\frac{1}{N}{\sum }_{i=1}^N\mathbf{1}\left({\omega }_i\le \omega \right)$. We treat the number of distributions $M$ as a hyperparameter, selected by greedy search: for each candidate $M$ we minimize Eq. (6) via gradient descent; we then compute an outer loss ${𝓛}_{\text{outer}}(M)=𝓛\left(F,F_M^{*}\right)+\lambda M$ for that $M$. Starting from small $M$, we increment it until the outer loss plateaus or $M$ reaches a maximum allowed value $M_{\max }$. The penalty $\lambda M$ in the outer loss discourages large $M$; thus the meta parameter $\lambda$ balances complexity of the Lorentzian mixture against fit accuracy.

Importantly, the Lorentzian mixture admits a closed-form CDF

$$F_M^{*}(\omega )=\sum _{m=1}^M{w}_m{\text{Ω}}_m(\omega ),$$ (7)

$${\text{Ω}}_m(\omega )=\frac{1}{2}+\frac{1}{\pi }\arctan \left(\frac{\omega -{\overline{\omega }}_m}{{\text{Δ}}_m}\right).$$ (8)

Gradient descent with respect to the parameters $\mathbf{\text{u}}=\left(w_1,w_2,\dots ,w_m,{\overline{\omega }}_1,{\overline{\omega }}_2,\dots ,{\overline{\omega }}_m,{\text{Δ}}_1,{\text{Δ}}_2,\dots ,{\text{Δ}}_m\right)$ requires the evaluation of the partial derivatives

$$\frac{\partial W^2}{\partial {\mathbf{\text{u}}}_i}=\frac{\partial W^2}{\partial F_M^{*}(\omega )}\frac{\partial F_M^{*}(\omega )}{\partial {\mathbf{\text{u}}}_i},$$ (9)

which we also obtained analytically (see Appendix A).

Together, gradient descent minimization of Eq. (6) can be performed with a fully analytic gradient, using Eq. (8) for the analytical Lorentzian mixture CDF. Additional optimization constraints are $w_m\ge 0$ for all $m\in 1,2,\mathrm{..},M$, ${\sum }_{m=1}^M{w}_m=1$, and ${\text{Δ}}_m\ge 0$ for all $m\in 1,2,\mathrm{..},M$. For a detailed description of the constrained optimization algorithm that we used throughout this letter, see Appendix B.

LMMF Equations for Globally Coupled Kuramoto Oscillators —

As a first demonstration of our method, we consider a network of globally coupled Kuramoto oscillators, i.e. Eq (1) with $G\left({\theta }_j,{\theta }_i\right)=\sin \left({\theta }_j-{\theta }_i\right)$, where we select the distribution of intrinsic oscillator frequencies to be a mixture of two Gaussian distributions:

$$\rho (\omega )=c_1𝒩\left(\omega ,{\mu }_1,{\sigma }_1\right)+c_2𝒩\left(\omega ,{\mu }_2,{\sigma }_2\right),$$ (10)

with $c_1+c_2=1.0$.

Using our optimization algorithm, $\rho (\omega )$ can be approximated by a Lorentzian mixture ${\rho }_M^{*}(\omega )$. If we consider each Lorentzian distribution of the mixture as a distinct ensemble of Kuramoto oscillators with a Lorentzian frequency distribution, then we may derive a system of mean-field equations using the Ott-Antonsen ansatz in the limit $N\to \infty$. We obtain

$${\dot{R}}_m=-{\text{Δ}}_m{R}_m+\frac{1-R_m^2}{2}K\sum _{l=1}^M{w}_l{R}_l\cos \left({\text{Ψ}}_l-{\text{Ψ}}_m\right),$$ (11)

$${\dot{\text{Ψ}}}_m={\text{Ω}}_m+\frac{1+R_m^2}{2R_m}K\sum _{l=1}^M{w}_l{R}_l\sin \left({\text{Ψ}}_l-{\text{Ψ}}_m\right),$$ (12)

for the average phase coherence $R_m$ and the average phase ${\text{Ψ}}_m$ of each ensemble, where the mixture weights $w_m$ enter the equations as the relative contributions of each ensemble to the overall mean-field drive. We call this system the Lorentzian mixture mean-field (LMMF) model.

Comparing the dynamics of the mean-field Eqs. (11) and (12) to a system of $N=5000$ Kuramoto oscillators for different $M_{max}$ and penalties $\lambda$, we find that the LMMF equations can (i) faithfully capture the Kuramoto network dynamics, and (ii) achieve a considerable dimensionality reduction (see Fig. 1). Given sufficient flexibility in terms of $M_{max}$ and $\lambda$, the optimization procedure consistently stops at $M\approx 6-8$ across different random realizations of the empirical oscillator distribution. Whereas $M=2$ suffices to predict the existence of the periodic solution that the Kuramoto system expresses (see Fig. 1c), $M\approx 6-8$ captures the average phase coherence dynamics of the finite-size Kuramoto system almost exactly. Note that the optimal $M$ depends on the choice of the meta parameter $\lambda$ (see Fig. 1a).

FIG. 1.

Lorentzian mixture mean-field (LMMF) model accurately captures the macroscopic dynamics of a network of Kuramoto oscillators with multimodal intrinsic frequency distribution. (a) LMMF model-fitting performance as a function of meta parameters $M_{\text{max}}$ and $\lambda$. The color represents the root mean square error between the Fourier transformed average phase coherence dynamics of the two models. The numbers in each cell represent the number of Lorentzian distributions $M$ that led to the best fit. (b-d) Example LMMF fits for different values of $M_{max}$ and $\lambda$. The top row depicts the fit to the empirical distribution of oscillator frequencies, whereas the bottom row depicts the average phase coherence dynamics of the Kuramoto oscillator network and the LMMF equations.

Rational Frequency Distributions —

We next compare our approach to previous works that use an ensemble approach to capture the dynamics of globally coupled Kuramoto oscillator systems with a Gaussian frequency distribution [19, 20] (see also [22, 23] for related work on quadratic integrate-and-fire neurons with Gaussian parameter distributions). In [20], the authors uses a truncated power expansion to approximate the Gaussian distribution with a weighted sum of rational distributions. They report a high accuracy in locating the synchronization threshold, but find a considerable difference in the transient dynamics at finite truncation orders. In [19], Skardal extends the Ott-Antonsen ansatz to globally coupled Kuramoto oscillators with rational frequency distributions, showing that the reduced dynamics remain low-dimensional but require a number of equations that grows linearly with the distribution’s order.

In our previous example, we found that our model captured the transient dynamics of the Kuramoto system with high accuracy. To examine this property of our model more systematically, we therefore examined its performance on the rational frequency distributions studied by Skardal [19]. As in the previous section, we consider systems of globally coupled Kuramoto oscillators, but with a rational oscillator frequency distribution

$$\rho (\omega )=g_n(\omega )=\frac{n\sin (\pi /2n){\text{Δ}}^{2n-1}}{\pi \left({\omega }^{2n}+{\text{Δ}}^{2n}\right)},$$ (13)

which reduces to the Lorentzian distribution for $n=1$ and converges to the uniform distribution in the limit $n\to \infty$.

For different choices of $n$, we randomly sample individual oscillator frequencies ${\omega }_i$ from Eq. (13), and then compare the average phase coherence dynamics $R(t)$ of the Kuramoto system, the mean-field equations derived by Skardal [19], and our LMMF equations fitted to the distribution of samples ${\omega }_i$. We consistently find that the LMMF equations capture $R(t)$ more closely than the Skardal mean-field equations (see Fig. 2). Moreover, at high $n$ the LMMF equations are of considerably lower dimensionality than the Skardal equations, thus providing a more accurate representation of the Kuramoto system with a smaller number of mean-field equations (see Fig. 2b).

FIG. 2.

Low-dimensional LMMF equations capture finite size bias in Kuramoto oscillator networks with rational frequency distributions. (a) Number of complex equations in the LMMF model (red) and the Skardal mean-field model (blue) [19]. (b) Performance of the LMMF equations vs. the Skardal mean-field equations in predicting the microscopic phase coherence dynamics. (c-f) Examples of oscillator frequency distribution fits (left column) and phase coherence dynamics (right column) for different sample sizes $N$ and distribution exponents $n$. In the left column, the blue line depicts the probability density $g_n(\omega )$ used by the Skardal model, the grey-filled bars depict the distribution of the microscopic samples drawn from $g_n(\omega )$, and the red line depicts the Lorentzian mixture fit of the sample distribution.

This result seems counterintuitive given the exact nature of the Skardal solution. However, whereas the Skardal mean-field equations are fully accurate for the frequency density function given by Eq. (13) as $N\to \infty$, even $N=5000$ randomly drawn samples ${\omega }_i$ from the density function (13) may introduce a finite size bias. Our results suggest that the LMMF equations capture these small deviations of the empirical distribution in a way that better predicts the behavior of the finite-size system, despite the fact that the LMMF equations assume the limit $N_m\to \infty$. Hence, we conclude that the LMMF equations are particularly well-suited to study the emergent dynamics of finite-size systems of globally coupled phase oscillators with heterogeneous oscillator frequencies.

Oscillator Heterogeneity —

Nodal heterogeneity, i.e. quenched disorder of parameters that control the behavior of nodes within large interacting networks, has been shown to play an important, non-trivial role for the dynamics of complex systems [24]. In coupled phase oscillators, for example, heterogeneity in oscillator frequencies can have either a synchronizing or a de-synchronizing effect, as well as promote or inhibit chimera states [25–29]. This has been shown to have direct implications for real-world problems that include interacting periodic processes, such as power grid stability [2], active particle motion [30], and neural population coding [31]. As a final result, we report that our LMMF approach permits study of the role of parameter heterogeneity in globally coupled phase oscillator systems.

We demonstrate our approach using the example of a mixture of four Gaussian distributions, and extend this to real neurophysiology data in Appendix C. We first sample $N=5000$ oscillator frequencies ${\omega }_i$ from the Gaussian mixture, and fit this empirical distribution with the Lorentzian mixture ${\rho }_M^{*}(\omega )$ (Fig. 3a). To study the effect of heterogeneity in this system, we then re-scale the centers and half-widths of the individual Lorentzian distributions, such that

$${\overline{\omega }}_m^{*}=h_{\overline{\omega }}{\overline{\omega }}_m+\left(1-h_{\overline{\omega }}\right)\sum _{l=1}^M{w}_l{\overline{\omega }}_l,$$ (14)

$${\text{Δ}}_m^{*}=h_{\text{Δ}}{\text{Δ}}_m.$$ (15)

FIG. 3.

Global heterogeneity parameters for analyzing Lorentzian mixture fits. (a) The empirical distribution (grey) of ${\omega }_i$ and the Lorentzian mixture fit (red). (b) Effects of changes in $h=h_{\overline{\omega }}=h_{\text{Δ}}$ on the Lorentzian mixture density. (c) 1D bifurcation diagram for the oscillator heterogeneity $h$. The steady-state solution branch (blue) and bifurcation points were obtained via parameter continuation of the synchronized fixed point solution of the LMMF equations, whereas the envelope of the quasi-periodic torus was approximated via numerical simulations of the LMMF equation dynamics in response to a slow, bidirectional ramp of $h$. (d) Average phase coherence dynamics of the LMMF equations and the Kuramoto oscillator network in response to steps in $h$. (e-f) 2D bifurcation diagrams of the Hopf and fold bifurcation loci in the $h-J$ and $h_{\overline{\omega }}-h_{\text{Δ}}$ parameter spaces, respectively. Shaded regions represent the regions in parameter space where the Kuramoto oscillator network converged to an asynchronous or synchronized fixed point.

This way, the LMMF approach allows for a systematic study of the system dynamics as a function of two global heterogeneity parameters: the distance $h_{\overline{\omega }}$ of the Lorentzian centers to the global distribution center, and the global scaling of the individual half-widths $h_{\text{Δ}}$. Choosing $h_{\overline{\omega }}=f_{\overline{\omega }}(h)$ and $h_{\text{Δ}}=f_{\text{Δ}}(h)$, the two can be studied as functions of a single, global heterogeneity parameter (see Fig. 3b for the simple case $h_{\overline{\omega }}=h_{\text{Δ}}=h$).

Analyzing the fixed point solutions of the fitted LMMF equations with respect to changes in $h$ via the bifurcation analysis software PyCoBi [32], we find that a stable, synchronized fixed point exists at small values of $h$, then loses stability through a fold bifurcation and eventually gives rise to a quasi-periodic torus via a supercritical Hopf bifurcation (see Fig. 3c–d). Tracing the loci of these codimension 1 bifurcations in the 2D parameter spaces spanned by $h-K$ and $h_{\text{Δ}}-h_{\overline{\omega }}$, we obtain detailed mechanistic insight how distortions of the oscillator frequency distribution $\rho (\omega )$ affect the system dynamics (see Fig. 3e–f).

Conclusion —

In this letter, we have introduced the Lorentzian mixture mean-field (LMMF) approach for studying the dynamics of coupled phase oscillator systems governed by arbitrary parameter distributions. We have shown that our method can provide drastic dimensionality reductions in terms of the system equations, while still capturing the macroscopic dynamics of finite size oscillator systems accurately. Finally, we have established a systematic method for studying the dynamic effects of complex distortions of the shape of parameter distributions within our LMMF framework.

Appendix A: Cramer-von Mises Loss Gradient —

In the following, we will provide the analytical derivation of the gradient with respect to all model parameters parameters ${\mathbf{\text{u}}}_i$. For ordered sets of $N$ i.i.d. samples ${\omega }_i$, the gradient with respect to the mixture weights $w_m$ is

$$\frac{\partial W^2}{\partial w_m}=\frac{\partial W^2}{\partial F_M^{*}(\omega )}\frac{\partial F_M^{*}(\omega )}{\partial w_m}$$ (16)

$$=2\sum _{i=1}^N\frac{\partial F_M^{*}\left({\omega }_i\right)}{\partial w_m}\left[F\left({\omega }_i\right)-F_M^{*}\left({\omega }_i\right)\right]$$ (17)

$$=2\sum _{i=1}^N\text{Ω}\left({\omega }_i\right)\left[F\left({\omega }_i\right)-F_M^{*}\left({\omega }_i\right)\right],$$ (18)

with $\text{Ω}\left({\omega }_i\right)$ given by Eq. (8). The gradient with respect to the Lorentzian centers ${\overline{\omega }}_m$ is

$$\frac{\partial W^2}{\partial {\overline{\omega }}_m}=\frac{\partial W^2}{\partial F_M^{*}(\omega )}\frac{\partial F_M^{*}(\omega )}{\partial {\overline{\omega }}_m}$$ (19)

$$=2\sum _{i=1}^N\frac{\partial F_M^{*}\left({\omega }_i\right)}{\partial {\overline{\omega }}_m}\left[F\left({\omega }_i\right)-F_M^{*}\left({\omega }_i\right)\right]$$ (20)

$$=-2w_m\sum _{i=1}^N\frac{\partial \text{Ω}\left({\omega }_i\right)}{\partial {\overline{\omega }}_m}\left[F\left({\omega }_i\right)-F_M^{*}\left({\omega }_i\right)\right],$$ (21)

$$=-2w_m\sum _{i=1}^N{\rho }_m\left({\omega }_i\right)\left[F\left({\omega }_i\right)-F_M^{*}\left({\omega }_i\right)\right],$$ (22)

with $\rho \left({\omega }_i\right)$ given by Eq. (3). For the gradient with respect to the Lorentzian half-widths, we obtain

$$\frac{\partial W^2}{\partial {\text{Δ}}_m}=\frac{\partial W^2}{\partial F_M^{*}(\omega )}\frac{\partial F_M^{*}(\omega )}{\partial {\text{Δ}}_m}$$ (23)

$$=2\sum _{i=1}^N\frac{\partial F_M^{*}\left({\omega }_i\right)}{\partial {\text{Δ}}_m}\left[F\left({\omega }_i\right)-F_M^{*}\left({\omega }_i\right)\right]$$ (24)

$$=-2w_m\sum _{i=1}^N\frac{\partial \text{Ω}\left({\omega }_i\right)}{\partial {\text{Δ}}_m}\left[F\left({\omega }_i\right)-F_M^{*}\left({\omega }_i\right)\right]$$ (25)

$$=-\frac{2w_m}{\pi }\sum _{i=1}^N\frac{{\omega }_i-{\overline{\omega }}_m}{{\left({\omega }_i-{\overline{\omega }}_m\right)}^2+{\text{Δ}}_m^2}\left[F\left({\omega }_i\right)-F_M^{*}\left({\omega }_i\right)\right].$$ (26)

Appendix B: Lorentzian Mixture Optimization Algorithm —

Here, we present a nested 2-level optimization algorithm for the Lorentzian mixture given by Eq. (5), where the outer level performs the selection of the optimal number of ensembles $M$ and the inner level minimizes Eq. (6) for a given $M$ via gradient descent.

The outer loop selects $M\in 1,\dots ,M_{max}$ via a greedy search with two early-stopping criteria, whereas the inner loop uses the Sequential Least Squares Programming (SLSQP) algorithm with multiple seeding points for gradient descent [33], as it allows to implement boundary and equality constraints such that all solutions to the minimization problem ${\min }_{\mathbf{\text{u}}}𝓛\left(F,F_M^{*}(\mathbf{\text{u}})\right)$ obey $w_m\ge 0$ for all $m\in 1,2,\mathrm{..},M$, ${\sum }_{m=1}^M{w}_m=1$, and ${\text{Δ}}_m\ge 0$ for all $m\in 1,2,\mathrm{..},M$. Since the inner loop is minimizing the Cramer-von Mises loss $W^2$, we can make use of the test statistic $T=N{W}^2$ for the difference between the empirical distribution and the model distribution, which has been tabulated for finite $N$ [34]. Starting from $M=1$, the outer optimization loop uses the SLSQP optimization to find the set of parameters $\mathbf{\text{u}}$ that minimizes Eq. (6). The outer loop stops if $T<1-\alpha$, i.e. if the empirical and model distributions are statistically indistinguishable at the significance level $\alpha$. As a second stopping criterion, the outer loss ${𝓛}_{\text{outer}}(M)={𝓛}_{\text{inner}}\left(F,F_M^{*}\right)+\lambda M$ is used, where ${𝓛}_{\text{inner}}\left(F,F_M^{*}\right)$ is the final loss after the SLSQP optimization for a given $M$. If ${𝓛}_{\text{outer}}(M+k)\le {𝓛}_{\text{outer}}(M)$ for $k=[1,\dots ,K]$, the outer loop is stopped and the solution for the $M$ with the lowest ${𝓛}_{\text{outer}}(M)$ is returned.

This concludes an algorithm for approximating arbitrary parameter distributions $\rho (\omega )$ via a Lorentzian mixture given by Eq. (5). It leverages the analytical properties of the Lorentzian distributions to fit the CDF of the Lorentizan mixture to empirical parameter distributions via the Cramer-von Mises loss given by Eq. (6), for which the gradients with respect to the parameters $\mathbf{\text{u}}$ can be obtained analytically. Finally, the algorithm exploits that the mixture combines all Lorentzian distributions linearly, allowing for a greedy search strategy with early stopping over the parameter $M$.

Appendix C: Dynamics of Globally Coupled Spiking Neurons from Different Cortical Layers —

The relationship between neural heterogeneity, synchronization of neural dynamics and population coding has been a particular subject of increasing research interest, not only within the field of neuroscience [24, 35–38], but also in machine learning [39, 40] and analog computing [41, 42]. A recent idea in this area has been that dynamic changes of neural heterogeneity – caused by neuron-intrinsic adaptation processes or neuromodulatory signals – could serve to control neurocomputational regimes [24, 43]. While a range of studies examined the macroscopic dynamics of networks of spiking neurons as a function of the width of a parameter distribution [43–48], an application of these ideas to empirical data sets of neural heterogeneity is still missing.

Here, we present the LMMF approach as a novel tool for (i) establishing a direct connection between neural network models and empirical data on neural heterogeneity, and (ii) the systematic study of changes in empirical parameter distributions. Using open-source electrophysiological provided by the Allen Brain Institute for cortical neurons in mice [49–51], we obtained empirical distributions of the distance-to-threshold $D_i=V_{\theta ,i}-V_{r,i}$, where $V_{\theta ,i}$ and $V_{r,i}$ refer to the spike threshold and resting membrane potential of the $i^{\text{th}}$ neuron in a data set, respectively. Specifically, we extracted empirical distributions of $D_i$ for fast-spiking interneurons in cortical layers 2/3 $(N=46)$ and 5/6 $(N=126)$ available through the Allen Software Development Kit [52]. Using the LMMF approach, we would like to study how differences in the distributions between cortical layers translate to differences in the population dynamics. To this end, we consider QIF networks of the form

$${\dot{V}}_i=\left(V_i-V_r\right)\left(V_i-V_{\theta ,i}\right)+Js+I(t),$$ (27)

$$s=\frac{1}{N}\sum _{i=1}^N{\int }_0^{\infty }g(\tau )\sum _{k\setminus t_i^k<t-\tau }\delta \left(t-\tau -t_i^k\right)\text{d}\tau ,$$ (28)

where $V_i$ is the membrane potential of neuron $i$, $t_i^k$ is the time of the $k^{th}$ spike emitted by neuron $i$, and $s$ is a convolution of the global spike rate with the synaptic response kernel $g(s)$. A neuron emits a spike when $V_i\ge V_p$, i.e. when its membrane crosses a peak value, after which the membrane potential is reset to $V_i\leftarrow V_0$. To inform the QIF model Eqs. (27) and (28) by the empirical distributions, we choose a global resting membrane potential $V_r={\overline{V}}_r$ and defined $V_{\theta ,i}={\overline{V}}_r+D_i$, thus collapsing heterogeneity in $V_{r,i}$ and $V_{\theta ,i}$ into a single parameter.

We then used our LMMF approach to obtain a low-dimensional set of mean-field equations for the QIF network. First, we fitted the Lorentzian mixture to the empirical distributions of $D_i$. Then, we followed the derivation described in [53] to obtain a set of mean-field equations for each QIF ensemble $m$ of the Lorentzian mixture:

$${\dot{r}}_m=\frac{{\text{Δ}}_m{\sigma }_m}{\pi }\left(v_m-{\overline{V}}_r\right)+r_m\left(2v_m-{\overline{V}}_r-{\overline{V}}_{\theta ,m}\right),$$ (29)

$${\dot{v}}_m=\left(v_m-{\overline{V}}_r\right)\left(v_m-{\overline{V}}_{\theta ,m}\right)+Js+I(t)$$ (30)

$$-\pi r_m\left({\text{Δ}}_m{\sigma }_m+\pi r_m\right),$$ (31)

$$s=\sum _{m=1}^M{w}_m{\int }_0^{\infty }g(\tau )r_m(t-\tau )\text{d}\tau ,$$ (32)

where ${\sigma }_m=\mathrm{sign}\left(v_m-{\overline{V}}_r\right)$ reflects a state-dependent pole switch for solving the integral over the Lorentzian distribution

$$\rho \left(V_{\theta ,m}\right)=\frac{1}{\pi }\frac{{\text{Δ}}_m}{{\left[V_{\theta ,m}-{\overline{V}}_{\theta ,m}\right]}^2+{\text{Δ}}_m^2}.$$ (33)

The mean-field derivation builds on previous results, showing that globally coupled QIF neurons permit the application of the Ott-Antonsen ansatz in the limits $V_p\to \infty$ and $V_0\to -\infty$ [8, 16, 21, 54–57]. Using the LMMF equations, we analyzed the equilibria of the QIF system, their stability, as well as their bifurcation structure via the parameter continuation software PyCoBi [32].

We find subtle differences in the neural synchronization properties of fast-spiking interneurons in layer 2/3 vs. layer 5/6, which can be explained by the differences in their average distance-to-threshold (see Fig. 4a–b). Generally, both distributions reflect a high degree of neural heterogeneity that favor asynchronous neural dynamics; only at a substantial collapse in heterogeneity (parameterized through $h$) does the asynchronous fixed point loose stability via a supercritical Hopf bifurcation, giving rise to limit cycle oscillations (see Fig. 4c–e).

Following the same procedure as described for the fast-spiking interneurons, we obtained $D_i$ for excitatory pyramidal cells of cortical layers 2/3 $(N=75)$ vs. 5/6 $(N=488)$ via the Allen Software Development Kit [52]. Again, we analyzed the equilibria of recurrently coupled QIF neurons of the pyramidal cell type governed by those distributions by fitting the LMMF model to the empirical distributions and identifying the equilibria and bifurcations of Eqs. (29)–(32). We find differences in the input-output steady-state curves of layer 2/3 vs. layer 5/6 pyramidal cells, including the loci of fold bifurcations in the $J-I$ parameter space (see Fig. 5). The more uniform distribution of $D_i$ in the layer 5/6 pyramidal cells requires less external input for the population to transition into an activated state in contrast to the multi-modal distribution of the layer 2/3 pyramidal cells. At low values of $h_{\text{Δ}}$, multi-stable dynamic regimes exist due to the multimodal distributions over $D_i$, allowing extrinsic inputs to switch between multiple stable firing rate configurations (see Fig. 5c–d). Increased heterogeneity around the centers ${\overline{V}}_{\theta ,m}$, implemented through $h_{\text{Δ}}$, reduce the number of co-existing stable states.

These findings confirm previous results on the role of neural heterogeneity for multi-stability and neural synchronization [24, 43, 47, 48, 53] and shows that the LMMF approach can reveal such heterogeneity effects in empirically measured parameter distributions with complex shapes (see Fig. 5a–b).

FIG. 4.

LMMF equations reveal heterogeneity-related synchronization of fast-spiking interneurons. (a-b) Lorentzian mixture fits to the empirical distributions of $D_i$ in layer 2/3 and layer 5/6 fast-spiking interneurons, respectively. (c) 1D bifurcation diagram of the globally coupled interneuron populations for $J=-100$ and $I=430$. (d) 2D bifurcation diagram depicting the Hopf bifurcation curves in the $I-h$ parameter space. (e) Synaptic activation dynamics of the mean-field equations and corresponding QIF networks in response to a decrease in neural heterogeneity.

FIG. 5.

LMMF equations reveal heterogeneity-related multi-stability of pyramidal cells. (a-b) Lorentzian mixture fits to the empirical distributions of $D_i$ in layer 2/3 and layer 5/6 pyramidal cells, respectively. (c) 1D bifurcation diagram of the pyramidal cell populations for $J=100$. (d) 2D bifurcation diagram depicting the fold curves in the $I-h_{\text{Δ}}$ parameter space. (e) Average firing rate dynamics of the mean-field equations and corresponding QIF networks in response to a stepping input.