Before and beyond the Wilson–Cowan equations

Abstract

The Wilson–Cowan equations represent a landmark in the history of computational neuroscience. Along with the insights Wilson and Cowan offered for neuroscience, they crystallized an approach to modeling neural dynamics and brain function. Although their iconic equations are used in various guises today, the ideas that led to their formulation and the relationship to other approaches are not well known. Here, we give a little context to some of the biological and theoretical concepts that lead to the Wilson–Cowan equations and discuss how to extend beyond them.

INTRODUCTION

Wilson and Cowan published the first of their two classic papers in 1972 (Wilson and Cowan 1972), wherein they obtained a set of coupled mean field equations providing a coarse-grained description of the dynamics of a network of excitatory and inhibitory neuron populations. The second paper in 1973 included spatial dependence (Wilson and Cowan 1973). Their work provided a demarcation in the practice of theoretical neuroscience. Before Wilson and Cowan, the nascent field of computational neuroscience was still grappling with how best to model neural dynamics. Thereafter, it began to coalesce onto a common set of neural network equations, and studying their properties became the focus.

By the mid-twentieth century, it became an established idea that at least some information processing in the brain is performed via global (population) activity (as opposed to at the single neuron level) (Feldman and Cowan 1975). A number of studies looked into the dynamics of neuronal populations in order to find a theoretical framework for studying the collective behavior of neurons (Beurle 1956; Griffith 1963a, 1965). Given the complex nature of the brain and our lack of access to details of the microscopic processes, building a dynamical theory naturally called for the use of statistical approaches, including coarse-graining and mean field theory.

Course-graining considers the system at a lower resolution, thereby reducing the degrees of freedom of the system. In the process, some information will be lost. The strategy is to ensure that the lost information is not crucial to understanding the phenomenon of interest. The classic example is the dynamics of gas molecules in a closed room. If we are only interested in whether the room will be comfortable, then the coarse-grained measures of temperature, pressure, and relative densities of the constituent molecules (nitrogen, oxygen, water, carbon dioxide, etc.) are sufficient. The actual microscopic dynamics of the over 10²³ molecules is irrelevant. However, in systems that are highly coupled, such as the brain, it is not clear what information is important. Mean field theory is a coarse-graining approach that captures the relevant mean dynamics while discounting (hopefully irrelevant) fluctuations around the mean.

Mean field theory was originally developed for magnetism (Le Bellac et al. 2004; Peliti 2011), which involves the alignment of atomic spins under the competing effects of coupling between the spin magnetic moments of each atom and thermal fluctuations. Reminiscent of neurons in the brain, the dynamics are highly complex, since a perturbation of one spin may lead to unending reverberating loops where each spin influences other spins that in turn affect themselves. Mean field theory cuts through this complexity by imposing self-consistency between the role of a spin as the influencer of other spins and the one being influenced by other spins. If the number of coupled spins is large enough and the spins are uncorrelated enough, then the fluctuations will go to zero (by the central limit theorem) and the mean will be the only relevant quantity. The neglect of correlations and higher-order statistics is the hallmark of mean field theory.

The original Wilson–Cowan equations were a coarse-grained mean field system for a continuous activity variable representing the proportion of a local population of neurons that are firing or active at any moment of time. However, this interpretation is not rigidly adhered to, and often the activity variable is deemed a physical quantity like the mean firing rate or an electrochemical potential of a single neuron or a group of neurons. The Wilson–Cowan and related equations became ubiquitous because they were able to model many elements of observed neural phenomena. Here we present a brief (and grossly incomplete) review of the history of some key ideas that led to their inception. We discuss when the assumptions of the equations are valid, when they break down, and how to go beyond them to take into account fluctuations and correlations.

WHAT ARE THE WILSON–COWAN EQUATIONS?

The use of neural network activity equations is pervasive. They have been referred to as Wilson–Cowan equations (Cowan et al. 2016; Kilpatrick 2013; Wilson and Cowan 1972), Cohen–Grossberg equations (Lu and Chen 2003), Amari equations (Potthast 2013), rate equations (Shriki et al. 2003; Sompolinsky et al. 1988; Vogels et al. 2005), graded-response equations (Hopfield 1984; Hopfield and Tank 1986), continuous short-term memory (STM) equations (Cohen and Grossberg 1983; Grossberg 1969b, 1988), and neural field equations (Bressloff 2012; Coombes 2005). Here we will adopt the term “activity equations.” Most of these equations come in one of three forms. The activity equations in Wilson and Cowan’s 1972 paper for a set of discrete local populations are

$$\tau {\dot{a}}_i=-a_i+\left(1-r{a}_i\right)f_j\left({\displaystyle \sum _j{w}_{ij}{a}_j+I_i\left(t\right)}\right)$$ (1)

where a_i is neural activity of a local population indexed by i, f_i is a nonlinear activation function (also referred to as a response function or gain function), w_ij is a coupling weight matrix, I_i is an external input, τ is a decay time constant, and r is a refractory period. Wilson and Cowan explicitly separated the effects of excitatory and inhibitory neurons. Their 1973 paper introduced a continuum version:

$$\tau {\partial }_{t}a\left(x,t\right)=-a\left(x,t\right)+\left(1-ra\left(x,t\right)\right)f\left({\displaystyle {\int }_{\Omega }w\left(x-y\right)}a\left(y,t\right)dy+I\left(y,t\right)\right)$$

where x is a spatial variable on a domain Ω. The other two common forms of neural activity equations are

$$\tau {\dot{s}}_i=-s_i+{\displaystyle \sum _j{w}_{ij}{f}_j\left(s_j\right)+I_i}$$ (2)

$$\tau {\dot{s}}_i=-s_i+\left(B_i-s_i\right){\displaystyle \sum _j{w}_{ij}{f}_j\left(s_j\right)+I_i}$$ (3)

which Grossberg (1988) calls the additive and shunting short-term memory (STM) equations, respectively. Grossberg and colleagues use Eqs. 2 and 3 with excitatory and inhibitory effects separated. Amari and subsequent neural field papers use the continuum version of Eq. 2, often with excitation and inhibition lumped together so neurons no longer obey Dale’s law, but this simplifies the analysis.

The term “Wilson–Cowan equations” often refers to either Eq. 1 with r = 0 or Eq. 2. For constant external input, these two equations are actually identical (Grossberg 1988), as can be seen by setting s_i = I_i + Σ_jw_ija_j. Assuming τ = 1 and taking the derivative yields

$$\begin{array}{c}{\dot{s}}_i={\displaystyle \sum _j{w}_{ij}{\dot{a}}_j}={\displaystyle \sum _j{w}_{ij}}\left[-a_i+f_i\left({\displaystyle \sum _j{w}_{ij}{a}_j+I_i}\right)\right]\\ =-s_i+{\displaystyle \sum _j{w}_{ij}{f}_j}\left(s_j\right)+I_i\end{array}$$ (4)

From this perspective, if a is neural activity then s is best viewed as a synaptic input or drive (Pinto et al. 1996). We will use the term “Wilson–Cowan (WC) equations” to exclusively refer to Eq. 1. For Eq. 2 we will use the term “STM model.” For a quasi-stationary state where s_i ≈ Σ_jw_ijf_j(s_j) + I_i, a_i can be interpreted as a neuronal mean firing rate

$$a_i=f_i\left(s_i\right)$$

and f_i(s) is a single-neuron activation function (FI curve) specifying rate as a function of input. Equation 1 would then be considered a rate-based model equivalent to the STM model (Eq. 2).

BEFORE THE WILSON–COWAN EQUATIONS

There are three essential ingredients that activity equations include: 1) a continuous time-dependent activity variable, 2) a linear weighted sum over these variables in the input, and 3) a nonlinear activation function linking input to output. Underlying these ingredients are coarse-graining and mean field theory. Here we explore some of the history behind these ideas. A list of milestones is in Table 1.

Table 1.

Milestones in the development of activity equations

1943	McCulloch and Pitts	Neural networks as computational units; foundation of neural networks
1948	Shimbel and Rapoport	Statistical approach to theory of central nervous system
1956	Beurle	First neural field theory of purely excitatory neurons
1963	Griffith	Role of inhibition in stabilizing neural activity
1963,1965	Griffith	Reaction-diffusion neural field theory
1966–	Grossberg	Continuous models of associative memory and pattern recognition
1972	Wilson and Cowan	Mean field theory of coarse-grained neural activity with excitation and inhibition
1973	Wilson and Cowan	Wilson–Cowan equations with spatial dependence
1975	Feldman and Cowan	Wilson–Cowan equations can be derived from microscopic activity under quasistationarity
1977	Amari	Neural field theory with local excitation and lateral inhibition
1983	Cohen and Grossberg	Lyapunov function for general class of continuous activity equations
1995–	Gerstner	Spike-response formalism

It was well known throughout the last century that neurons fire action potentials if given sufficient stimulus. Thus it was reasonable to model them as binary-state machines that activate when the inputs exceed a threshold. In 1943, McCulloch and Pitts (McCulloch and Pitts 1943) showed that a network of such binary threshold neurons with both excitatory and inhibitory connections is capable of producing any logical operation and thus performing universal computations. This launched the study of neural networks in computer science, artificial intelligence, and neuroscience.

The success of McCulloch and Pitts led to the idea that the function of the nervous system is fully reducible to the structure of logical gates, which are defined by the connection weights. Shortly thereafter, a dissenting opinion emerged that sprouted a branch of research that would eventually lead to the Wilson–Cowan equations and dynamical systems modeling of the brain. Shimbel and Rapoport (Shimbel and Rapoport 1948) argued that the McCulloch and Pitts “neural net” approach fell short of building plausible theories for neural systems. They argued that countless numbers of equivalent networks could produce the same desired output and that neural nets were not robust to the failure of a few neurons. Additionally, it was unlikely that genes predetermine the details of the microscopic structure of biological systems but rather impose statistical traces on the macroscopic properties. They proposed that the goal should be to seek the statistical properties that govern the evolution and function of a biological system, rather than proposing specific network connectivity to perform a particular task. Their idea was echoed by Ashby and von Neumann (Rosenblatt 1958), which eventually impacted Rosenblatt’s approach in developing his perceptron idea (Rosenblatt 1958, 1961). Thus the argument against the logical circuit approach instigated advances in machine learning as well.

Shimbel and Rapoport derived a dynamic equation governing the firing probability of neurons located in a given region of a brain network in terms of a recursive map. Like McCulloch and Pitts, they assumed binary neurons and supposed that a particular synapse receives input from n axon terminals, which they called “bulbs” partitioned into p “bulb groups.” The probability that a neuron will fire depends on an integer number of bulbs exceeding a threshold h. Neuron firing dynamics are decomposed into a set of probabilities governing the connectivity and synaptic properties. The fraction of neurons at location x firing at time t = 1 was presumed to be

$$A_1\left(x\right)={\displaystyle \sum _{n,p,h}P\left(x;n,p,h\right)}{P}_{nph}\left[I_0\left(x\right)\right]I_0\left(x\right)={\displaystyle {\int }_{\Omega }O\left(x,x\prime \right)}{A}_0\left(x\prime \right)\text{d}x\prime$$

where A₀(x) is the firing probability of neurons at x on a domain Ω at time t = 0, O(x,x′) is the probability that a bulb group at x originates from x′, P(x;n,p,h) is the probability that a neuron at x has a firing threshold h and synapses of type (n,p), and P_nph[I₀(x)] denotes the conditional probability that an active neuron at x is of (n,p,h) type. There were two crucial ideas for these equations. The first is a Markov property where the probability of firing at a given time only depends on the probability of firing in the previous time. The second is that the firing probability does not depend on higher-order statistics beyond the mean, which is a mean field theory assumption.

Although P_nph[I₀(x)] are derivable in principle, it would be a daunting task because the probability of activation depended on the microscopic details of synaptic configurations. Beurle (Beurle 1956) introduced coarse-graining, which resolved the synaptic structure problem. He was inspired by experimental findings on the distribution of dendritic fibers as a function of distance from the cell body (Sholl 1953; Uttley 1955). Anatomical studies had started to focus on connectivity patterns of neuronal aggregates (as opposed to the connections of individual neurons), providing probability distributions instead of specific circuits (Sholl 1953, 1955). Berule’s coarse-graining scheme represented neural activity as a continuous field in space and time and was the first neural field theory (Coombes 2005). Beurle assumed a network of purely excitatory cells with uniform thresholds and refractory period. A neuron becomes active if it receives input from a threshold number of active neurons. If q is the threshold number, then the rate at which the cells become active at a time t + Δt is

$$A\left(t+\Delta t\right)=Q{P}_{\left(q-1\right)}Ak$$

where A(x,t) is the proportion of cells active in any region per unit time, k is a scale constant, and Q is the proportion of cells that are sensitive to input, and

$$P_{\left(q-1\right)}=e^{-s}\frac{S_{\left(q-1\right)}}{\left(q-1\right)!}$$

is the proportion of cells that have an integrated excitation of (q − 1) and are ready to spike with the arrival of one further impulse. The mean rate of arriving impulses at x obeys

$$s\left(x,t\right)={\displaystyle {\int }_{-\infty }^0{\displaystyle {\int }_{-\infty }^{\infty }w\left(x-x\prime \right)}}A\left(x\prime ,t\prime \right)\alpha \left(t-t\prime \right)\text{d}x\prime \text{d}t\prime$$

where α(t) is the temporal response function and w(x) = be^−|x|/a is a spatially dependent connection function inspired by experimental studies (Sholl 1955; Uttley 1955). Again, this was a mean field formulation where the mean activity does not depend on higher moments of the activity.

Beurle analyzed spontaneous random activity and waves. He was interested in waves of activity because of then-recent electroencephalograph studies (Burns 1951; Chang 1951; Lilly and Cherry 1954; Walter and Shipton 1951). He showed that his model with minimal structure was capable of supporting plane, spherical, and circular waves as well as vortices. In addition, he was driven by the idea that switching of neural waves may be related to shifting of attention (or other perceptions). However, his model also showed that the stationary fixed point is unstable; a slight deviation from the critical point leads the activity to either cease or saturate (because of refractoriness). Therefore, he erroneously concluded that there is no stable continuous random activity. Despite the achievements of the theory, the lack of inhibition as well as some technical flaws drove others to look for alternative formulations. Nevertheless, Beurle established most of the concepts that eventually led to the Wilson–Cowan equations.

The next milestone was Griffith (Griffith 1963a, 1965). After establishing the role of inhibition in stabilizing brain activity (Griffith 1963b), Griffith took a different approach and derived a reaction-diffusion equation for neural activity. He desired a classical field theory with a field representing the synaptic excitation or input to the cells and a source that quantifies the activity of the cells (e.g., spiking frequency). He built a phenomenological model by enforcing a set of constraints on the equations.

Griffith adopted a linear inhomogeneous differential equation for s(x,t) of the form

$$\widehat{H}s\left(x,t\right)=\kappa A\left(x,t\right)$$

where $\widehat{H}$ is a linear operator and κ is a constant. He defined A as

$$A\left(x,t\right)\equiv f\left[{\displaystyle {\int }_{-\infty }^{+\infty }s\left(x,t-\tau \right)}\alpha \left(\tau \right)\text{d}\tau \right]$$

where f is a nonlinear activation function and the temporal response function α(τ) obeys causality. Assuming that s is differentiable and quasi-stationary, Griffith obtained the following general form for A by Taylor expanding to second order:

$$A\approx f\left(s\right)+C_1\frac{\partial s}{\partial t}+C_2\frac{{\partial }^{2}s}{\partial t^2}$$

where C₁ and C₂ are constants. He then assumed that the field operator $\widehat{H}$ obeyed translational and rotational invariance, which implied

$$\widehat{H}=a+b\frac{\partial }{\partial t}+c\frac{{\partial }^2}{\partial t^2}+{\nabla }^2$$ (5)

Substituting a field solution ansatz with constant velocity v,

$$s\left(x,t\right)={\displaystyle \int A\left(x,t-\left|x-x\prime \right|/v\right)}w\left(\left|x-x\prime \right|\right)\text{d}x\prime$$

where w is a connectivity function, into Eq. 5, he obtained

$$\begin{array}{c}{\nabla }^{2}s=\frac{1}{4}{\beta }^{2}s+\left(\frac{\beta }{v}-4\pi C_1\right)\frac{\partial s}{\partial t}\\ +\left(\frac{1}{v^2}-4\pi C_2\right)\frac{{\partial }^{2}s}{\partial t^2}-4\pi f\left(s\right)\end{array}$$ (6)

for $w\left(x\right)\propto x^{-1}{e}^{-\frac{1}{2}\beta x}$, which Griffith argued was compatible with Sholl’s experimental findings (Sholl 1955). Rescaling and assuming a shallow activation function where $\partial f/\partial s$ varies slowly, in the limit of v → ∞, Eq. 6 can be reduced to

$${\nabla }^{2}s=-\mu s+\dot{s}-f\left(s\right)$$ (7)

where μ > 0 is a constant and f is also redefined as a general nonlinear function. From Eq. 7 he obtained a spatially homogeneous mean field equation

$$\dot{s}=-\mu s+f\left(s\right)$$

This is the STM model (Eq. 2) for uniform connectivity, which as shown above is equivalent to the Wilson–Cowan equation.

Griffith showed that Beurle’s formalism is equivalent to his if the activation function f(s) is adjusted appropriately. He originally assumed a linear response function, namely f(s) = f(s_c) + b(s − s_c), but argued that a different activation function would result in the same unstable fixed point as in Beurle’s theory. Griffith was the first to show the possibility of stable spontaneous activity depending on the gain function. However, he did not account for excitatory and inhibitory synaptic connections explicitly.

THE WILSON AND COWAN EQUATIONS

By the mid-1960s most of the concepts that would be incorporated into the Wilson–Cowan equations had been proposed, although no one had put them together into a cohesive whole. Wilson and Cowan developed a coarse-grained description of neuronal activity where the distinction between excitatory and inhibitory cells was taken into account explicitly. They were motivated by physiological evidence from Hubel and Wiesel (Hubel and Wiesel 1965; Wiesel and Hubel 1963), Mountcastle (Mountcastle 1957), Szentágothai and Lissak (Szentágothai and Lissak 1967), and Colonnier (Colonnier 1965), which suggested the existence of certain populations of neurons with similar responses to external stimuli.

In line with Sholl (Sholl 1955) and Beurle (Beurle 1956), Wilson and Cowan argued that a microscopic (single neuron) description of neural activity is probably not well suited for understanding higher-level functions that entail more complex processes such as sensory processing, memory, and learning. Using dynamical systems analysis, they showed that their equations exhibit multistability and hysteresis, which could serve as a substrate for memory (Cragg and Temperley 1955; Fender and Julesz 1967), and limit cycles, where the frequency of the oscillation is a monotonic function of stimulus intensity. The combination of mathematical tractability and dynamical richness is the reason for the lasting legacy of their equations.

Wilson and Cowan derived their activity equations from first principles using a probabilistic framework on an aggregate of heterogeneous threshold neurons coupled with excitatory and inhibitory synapses with random connectivity by which spatial interactions could be neglected. They extended the model to the case of spatial inhomogeneity the following year (Wilson and Cowan 1973).

They assumed that neurons had a sensitive phase where input exceeding a threshold would cause them to fire and a refractory period in which they would not fire regardless of input. Defining A_i as the proportion of cells of type i ∈ {E,I} (excitatory or inhibitory) active at time t, the fraction that are refractory will be ${\displaystyle {\int }_{t-r}^t{A}_i\left(t\prime \right)}\text{d}t$, with refractory period r and the fraction that are sensitive is then $1-{\displaystyle {\int }_{t-r}^{t}A{}_i\left(t\prime \right)}\text{d}t$.

The activation function f(z) gives the expected proportion of cells that would respond to a given level of input z if nonrefractory. If this and the fraction of sensitive neurons are independent, then the updated fraction of active cells would be

$$\left[1-{\displaystyle {\int }_{t-r}^t{A}_i}\left(t\prime \right)\text{d}t\prime \right]f_i\left(z_i\right)\delta t$$

where

$$z_i\equiv {\displaystyle {\int }_{\infty }^t\alpha \left(t-t\prime \right)}\left[W_{i\text{E}}{A}_{\text{E}}\left(t\prime \right)-W_{i\text{I}}{A}_{\text{I}}\left(t\prime \right)+I_i^{\text{ext}}\left(t\prime \right)\right]\text{d}t\prime$$

$I_i^{\text{ext}}$ is external input, α(t − t′) is a response function governing the time evolution of the spike arrivals, and W_jk is the average number of synapses from type k to j. Wilson and Cowan noted that the input to a cell and the sensitive fraction could be correlated, which would violate mean field theory since both involve A_i. They argued that this correlation could be negligible for highly interconnected networks because of the presence of spatial and temporal fluctuations in the average excitation within the population, and also because of the variability of thresholds supported by experiments (Geisler et al. 1958; Rall and Hunt 1956; Verveen and Derksen 1969).

To estimate the shape of the activation function, they assumed that even if all the neurons receive the same input any variability in other parameters such as the firing threshold or the number of afferent connections could lead to a probability distribution for the number of spiking neurons. Assuming that the variability mainly stems from the threshold, the expected proportion of neurons that receive at least threshold excitation (per unit time) would be¹

$$f\left(x\right)={\displaystyle {\int }_0^{x\left(t\right)}p\left(\theta \right)}\text{d}\theta$$

where p(θ) is the threshold probability density. If p(θ) is unimodal then f(x) would be a sigmoidal function. Later, Feldman and Cowan (Feldman and Cowan 1975) expanded on that interpretation, showing that the activation function in the Wilson–Cowan equations can be regarded as the average of single-neuron activation functions.

Putting this all together gives

$$\begin{array}{c}{A}_{\text{E}}\left(t+\tau \right)=\left[1-{\displaystyle {\int }_{t-r}^t{A}_{\text{E}}\left(t\prime \right)}\text{d}t\prime \right]\\ \times f_{\text{E}}\left({\displaystyle {\int }_{-\infty }^t\alpha \left(t-t\prime \right)}\left[W_{\text{EE}}{A}_{\text{E}}\left(t\prime \right)-W_{\text{EI}}{A}_{\text{I}}\left(t\prime \right)+I_{\text{E}}^{\text{ext}}\left(t\prime \right)\right]\text{d}t\prime \right)\end{array}$$ (8)

$$\begin{array}{c}{A}_{\text{I}}\left(t+\tau \right)=\left[1-{\displaystyle {\int }_{t-r}^t{A}_{\text{I}}\left(t\prime \right)}\text{d}t\prime \right]\\ \times f_{\text{I}}\left({\displaystyle {\int }_{-\infty }^t\alpha \left(t-t\prime \right)}\left[W_{\text{IE}}{A}_{\text{E}}\left(t\prime \right)-W_{\text{II}}{A}_{\text{I}}\left(t\prime \right)+I_{\text{I}}^{\text{ext}}\left(t\prime \right)\right]\text{d}t\prime \right)\end{array}$$ (9)

After obtaining the general Eqs. 8 and 9, Wilson and Cowan derived a set of differential equations that carried the biologically relevant aspects of their general equations. Defining time coarse-grained variables

$$a_i\left(t\right)=\frac{1}{s}{\displaystyle {\int }_{t-s}^t{A}_i\left(t\prime \right)}\text{d}t\prime$$

they argued that if α(t) ≈ 1 for 0 ≤ t ≤ r and decays rapidly for t > r, then

$$\begin{array}{l}{\displaystyle {\int }_{t-r}^{t}A\left(t\prime \right)}\text{d}t\prime \to ra\left(t\right)\\ {\displaystyle {\int }_{-\infty }^t\alpha \left(t-t\prime \right)}A\left(t\prime \right)\text{d}t\prime \to ka\left(t\right)\end{array}$$

where k and r are constants. Inserting into Eqs. 8 and 9, Taylor expanding, and rescaling gives

$${\tau }_{\text{E}}\frac{\text{d}{a}_{\text{E}}}{\text{d}t}=-a_{\text{E}}+\left(1-r{a}_{\text{E}}\right)f_{\text{E}}\left[W_{\text{EE}}{a}_{\text{E}}-W_{\text{EI}}{a}_{\text{I}}+I_{\text{E}}^{\text{ext}}\left(t\right)\right]$$ (10)

$${\tau }_{\text{I}}\frac{\text{d}{a}_{\text{I}}}{\text{d}t}=-a_{\text{I}}+\left(1-r{a}_{\text{I}}\right)f_{\text{I}}\left[W_{\text{IE}}{a}_{\text{E}}-W_{\text{II}}{a}_{\text{I}}+I_{\text{I}}^{\text{ext}}\left(t\right)\right]$$ (11)

After deriving the original mean field equations, Wilson and Cowan considered spatial inhomogeneity. Inspired by Beurle (Beurle 1956), they extended their original equations to

$$\tau \frac{\partial A_{\text{E}}\left(\mathbf{x},t\right)}{\partial t}=-A_{\text{E}}\left(\mathbf{x},t\right)+\left(1-r{A}_{\text{E}}\left(\mathbf{x},t\right)\right)\times f_{\text{E}}[{\displaystyle {\int }_{\Omega }\text{d}\mathbf{x}\prime }{\rho }_{\text{E}}\left(\mathbf{x}\prime \right)W_{\text{EE}}\left(\mathbf{x}-\mathbf{x}\prime \right)A_{\text{E}}\left(\mathbf{x}\prime ,t\right)-{\displaystyle {\int }_{\Omega }\text{d}\mathbf{x}\prime }{\rho }_{\text{I}}\left(\mathbf{x}\prime \right)\text{d}\mathbf{x}\prime W_{\text{EI}}\left(\mathbf{x}-\mathbf{x}\prime \right)A_{\text{I}}\left(\mathbf{x}\prime ,t\right)+I_{\text{E}}^{\text{ext}}\left(\mathbf{x},t\right)]$$

$$\tau \frac{\partial A_{\text{I}}\left(\mathbf{x},t\right)}{\partial t}=-A_{\text{I}}\left(\mathbf{x},t\right)+\left(1-r{A}_{\text{I}}\left(\mathbf{x},t\right)\right)\times f_{\text{I}}[{\displaystyle {\int }_{\Omega }\text{d}\mathbf{x}\prime }{\rho }_{\text{E}}\left(\mathbf{x}\prime \right)W_{\text{IE}}\left(\mathbf{x}-\mathbf{x}\prime \right)A_{\text{E}}\left(\mathbf{x}\prime ,t\right)-{\displaystyle {\int }_{\Omega }\text{d}\mathbf{x}\prime }{\rho }_{\text{I}}\left(\mathbf{x}\prime \right)W_{\text{II}}\left(\mathbf{x}-\mathbf{x}\prime \right)A_{\text{I}}\left(\mathbf{x}\prime ,t\right)+I_{\text{I}}^{\text{ext}}\left(\mathbf{x},t\right)]$$

where the activity variables are

$$\begin{array}{l}{A}_{\text{E}}\left(\mathbf{x},t\right)={\displaystyle {\int }_{-\infty }^t\text{d}t\prime \alpha \left(t-t\prime \right)}{n}_{\text{E}}\left(\mathbf{x},t\prime \right)\\ A_{\text{I}}\left(\mathbf{x},t\right)={\displaystyle {\int }_{-\infty }^t\text{d}t\prime \alpha \left(t-t\prime \right)}{n}_{\text{I}}\left(\mathbf{x},t\prime \right)\end{array}$$

α(t − t′) = α₀e^{−(t − t′)/τ}, W_ij(x − x′) is the spatially dependent connection weight, ρ_i(x) defines the density of neurons in a small volume around x, and n_E and n_I represent the proportions of excitatory and inhibitory neurons activated per unit time.

Wilson and Cowan developed the model to incorporate spatial dependence, but x could be generalized to represent any quantity. The seminal work of Ermentrout and Cowan (1979), which explained the emergence of visual hallucination patterns in terms of a Turing instability, considered a Wilson–Cowan model in two spatial dimensions. Others (Ben-Yishai et al. 1995; Bressloff et al. 2002) have taken x to be an orientational angle. We refer the interested reader to Coombes (2005) on more applications of the Wilson–Cowan model.

STM MODELS

The artificial intelligence branch of neural networks following McCulloch and Pitts focused initially on the computational capabilities of the binary neuron (e.g., Caianiello 1961; Orbach 1962; Widrow 1962). However, Hartline and Ratliff (Hartline and Ratliff 1958) considered the continuous firing rate of single cells in a neural network to successfully model the Limulus retina, which could be directly measured. In the 1960s, adaptive networks were shown to be able to perform recognition tasks, among those Rosenblatt’s perceptron (Rosenblatt 1958) and the adaptive linear machine, Adaline (Widrow 1960). These were followed by Grossberg, whose initial goal was to understand how the behavior of an individual can adapt stably in real time to complex and changing environmental conditions (Grossberg 1988). His approach led to a class of continuous neural networks defined by the nonlinear coupling between activity and synaptic (adaptive) weights for which he deduced many important properties of neural networks (Grossberg 1968, 1969a, 1969b, 1988).

Grossberg first considered model Eq. 2, which he termed the additive STM equation since the equation exhibits bistability as shown by Griffith. He then added dynamics for the synaptic weights to model long-term memory. Inspired by the structure of the Hodgkin–Huxley model, Grossberg next derived an equation for neural networks that more closely modeled the shunting dynamics of individual neurons, resulting in Eq. 3 (Grossberg 1964, 1968, 1969b). The shunting STM model is approximated by the additive STM model when the activities s_i in Eq. 3 are far from saturation. These networks are capable of performing a rich repertoire of behaviors including content-addressable memory and oscillations (Cohen and Grossberg 1983; Grossberg 1988). Cohen and Grossberg (Cohen and Grossberg 1983) proved that under general conditions including symmetric weights, general activity equations possess a Lyapunov function, indicating that all orbits will flow to local minima and can serve as a basis for models of associative memory.

In 1982, Hopfield, using the analogy to the statistical mechanics of spin glasses, showed that a discrete time binary neuron model with symmetric connections also has a nonincreasing Lyapunov function and thus stable attractors that can act as memory states (Hopfield 1982). The foundations of this work had been set in the early 1960s, when several studies demonstrated that (artificial) adaptive networks could perform recognition tasks (Rosenblatt 1961; Widrow 1960). These advances were followed by a number of studies on associative memory and pattern recognition, including those by Grossberg (Grossberg 1968, 1969a, 1969b) and Amari (Amari 1977, 1983; Potthast 2013). Little and Shaw (Little 1974; Little and Shaw 1978) pointed out the analogy between neural networks and spin glasses. Hopfield’s contributions (Hopfield 1982; Hopfield and Tank 1986) attracted a lot of interest from the physics community (Amit et al. 1985; Sompolinsky 1988), but it is rarely acknowledged that his discovery was a special case of the general Cohen–Grossberg theorem and that there was much work in this area that preceded him.

SPIKE-RESPONSE FORMALISM

The Wilson–Cowan equations were highly impactful in modeling a number of neural phenomena such as pattern formation, waves, and slow oscillations. However, Feldman and Cowan showed that the Wilson–Cowan equations (Eq. 1) are only valid for quasi-stationary activity (Feldman and Cowan 1975). Gerstner and van Hemmen (Gerstner and van Hemmen 1992a) also showed that for stationary activity any single-neuron model can be reduced to a network of graded response neurons, but this is not true for coherent oscillations (Gerstner and van Hemmen 1992a).

To address this deficiency, Gerstner sought a general formulation of global dynamics whereby one can systematically estimate the accuracy of rate-based models. To that end, Gerstner (1995) developed the spike-response formalism, which is a derivation of mean field population dynamics given that aspects of the single-neuron dynamics are known. The approach echoes the ideas underlying the Wilson–Cowan equations, with a focus on a realistic model of single-neuron dynamics.

In the spike-response formalism, the membrane potential of a neuron is modeled by the combination of the time-dependent refractory response of the neuron to its own activity and the summed responses to the incoming spikes (Gerstner et al. 2014; Gerstner and van Hemmen 1992a). As a result, the synaptic potential of a single neuron can be described by the integro-differential equation

$$s_i\left(t,\tau \right)=I_i^{\text{ext}}+{\displaystyle \sum _{f=1}^F{\eta }^{\text{refr}}}\left(t-t_i^f\right)+{\displaystyle \sum _j{W}_{ij}}{\displaystyle {\int }_0^{\infty }\kappa \left(\tau ,t\prime \right)}{S}_j^{\left(F\right)}\left(t-t\prime \right)\text{d}t\prime$$

where s_i(t,τ) is the membrane potential of neuron i that has fired its last spike at t − τ, η^refr is the refractory function, $t_i^f$ denotes the spike times for neuron i, and κ(τ, t′) denotes the response kernel in which τ is the time that has passed since the last postsynaptic spike represented by $S_j^{\text{(}F\text{)}}$ (Gerstner et al. 2014). Different models of single-neuron dynamics can be reduced to a spike-response model with appropriate kernel functions (Gerstner 1995; Gerstner and van Hemmen 1992a, 1992b).

To find the connection to rate-based models, Gerstner applied mean field theory. He considered a uniform population of neurons for which the activity A_p(t) of a single pool p is defined as the proportion of spiking neurons per unit time within the pool. Neurons in a pool are equivalent; connection weights and the response kernel κ_p(τ,t′) only depend on the pool identity. However, the addition of noise, which turned out to be essential in his formulation, does cause variability among spike trains within a single pool. To formulate his pool dynamics, Gerstner assumed that the spike trains could be approximated as renewal processes if the synapses are weak |W_pq|≪ 1. This led to a closed set of mean field equations describing the synaptic input s_p(t), pool activity A_p(t), and the firing noise probability imposed by the dynamics. His formalism is suited to model neural network dynamics at arbitrarily short timescales.

In line with previous studies (Feldman and Cowan 1975; Gerstner and van Hemmen 1992a), Gerstner showed that differential equation activity models, while excellent for modeling asynchronous firing, break down for fast transients and coherent oscillations. He derived a correction to the quasi-stationarity assumption, estimating the error of rate-based models. He showed that in order for the rate-based description to be valid one should have $〈\Delta t_{\text{isi}}〉\dot{A}\left(t\right)/A\left(t\right)\ll 1$, where Δt_isi denotes the mean interspike interval (inverse of mean firing rate). In addition, he also rederived the Wilson–Cowan equations by adjusting the appropriate kernels and provided a derivation wherein the noise-induced neuron firing probability plays the same role as the sigmoidal activation function in the original Wilson–Cowan derivation.

MEAN FIELD THEORY FOR KNOWN NEURAL DYNAMICS

The Wilson–Cowan equations and variants have been highly successful, but the question remains as to how quantitatively accurate they are in modeling a neural system with predefined microscopic neuron and synaptic dynamics. Here, we explicitly derive a mean field theory for the neural activity of a deterministic network of coupled spiking neurons (Buice and Chow 2013a; Qiu and Chow 2018).

Consider a network of N conductance-based neurons:

$$\begin{array}{c}{\tau }_V\frac{\text{d}{V}_i}{\text{d}t}=h_V\left(V_i,m_i,s_i\right)\\ {\tau }_m\frac{\text{d}m}{\text{d}t}=h_m\left(V_i,m_i,s_i\right)\\ {\tau }_s\frac{\text{d}{s}_i}{\text{d}t}=-s_i+{\displaystyle \sum _{j=1}^N{w}_{ij}}A\left(V_j\right)\end{array}$$

where V_i is the membrane potential of neuron i, m_i represents a single or set of auxiliary variables, s_i represents the synaptic input or drive, h’s are continuous functions specifying the dynamics of the respective variables, τ’s are time constants, w_ij is a matrix of synaptic weights between neurons, and A is a function that activates whenever V_j exceeds a threshold indicating an action potential. We do not explicitly distinguish between excitatory or inhibitory neurons, but this is reflected in the parameter values, which can vary from neuron to neuron, and the synaptic weight matrix (e.g., obey Dale’s law).

If the individual neurons have limit cycle dynamics, as expected for spiking neurons, and the coupling between individual neurons is not excessively strong (i.e., a single spike does not strongly perturb the spiking dynamics, although many can and will), then the neuron dynamics can be reduced to a phase variable around or near the limit cycle (Brown et al. 2004). The system takes the simpler form of

$${\dot{\theta }}_i=F_i\left({\theta }_i,s_i\right)$$ (12)

$${\tau }_s{\dot{s}}_i=-s_i+\frac{1}{N}{\displaystyle \sum _j{w}_{ij}\delta \left(t-t_s^j\right)}$$ (13)

where θ_i is the phase of neuron i, F_i is the phase velocity, and $t_s^j$ are the spiking times of neuron j, which we set to θ_j = π.

Our goal is to generate a mean field description of the network specified by Eqs. 12 and 13. We quantify the neuron activity in terms of a density of the neuron phase:

$${\eta }_i\left(\theta ,t\right)=\delta \left(\theta -{\theta }_i\left(t\right)\right)$$

where δ(·) is the Dirac delta function. This allows us to write

$$\delta \left(t-t_s^j\right)={\dot{\theta }}_j{\eta }_j\left(\pi ,t\right)=F_j\left(\pi ,s_j\right){\eta }_j\left(\pi ,t\right)$$ (14)

which is also the firing rate of the neuron, i.e., the neuron velocity or flux at π. Inserting Eq. 14 into Eq. 13 gives

$${\tau }_s{\dot{s}}_i=-s_i+\frac{1}{N}{\displaystyle \sum _j{w}_{ij}{F}_j\left(\pi ,s_j\right){\eta }_j}\left(\pi ,t\right)$$ (15)

We obtain the dynamics of η_i by imposing local neuron conservation. Although η_i is not differentiable, we can still formally write

$${\partial }_t{\eta }_i\left(\theta ,t\right)+{\partial }_{\theta }\left[F_i\left(\theta ,s_i\right){\eta }_i\left(\theta ,t\right)\right]=0$$ (16)

which is called the Klimontovich equation in the kinetic theory of plasmas (Ichimaru 1973; Nicholson 1983). Equations 16 and 15 fully specify the spiking neural network dynamics but are no simpler to solve than the original equations. They need a regularization scheme to make them useful. One approach is to average over the population. This was the strategy employed in before the wilson and cowan equations, the wilson and cowan equations, and spike-response formalism and in mean field treatments of networks (Brunel 2000; Strogatz and Mirollo 1991) where a population average is taken. In the limit of N → ∞, the phases are assumed to be sufficiently asynchronous so that η is differentiable and Eq. 16 becomes a well-behaved partial differential equation. With the addition of Gaussian white noise, it takes the form of a Fokker–Planck equation.

An alternative is to average over an ensemble of networks, each prepared with a different initial condition drawn from a specified distribution (Buice et al. 2010; Buice and Chow 2007, 2013a, 2013b, 2013c; Buice and Cowan 2007; Hildebrand et al. 2007). Taking the ensemble average $〈\cdot 〉$ over Eqs. 15 and 16 gives

$$\begin{array}{l}{\partial }_t〈{\eta }_i\left(\theta ,t\right)〉+{\partial }_{\theta }〈F_i\left(\theta ,s_i\right){\eta }_i\left(\theta ,t\right)〉=0\\ {\tau }_s〈{\dot{s}}_i〉=-〈s_i〉+\frac{1}{N}{\displaystyle \sum _j{w}_{ij}〈F_j\left(\pi ,s_j\right){\eta }_j\left(\pi ,t\right)〉}\end{array}$$

We see that the dynamics of the mean of η_i and s_i depend on the higher-order moment $〈F_i\left(\theta ,s_i\right){\eta }_i〉$. We can construct an equation for this by differentiating F_i(θ,s_i)η_i, inserting into Eqs. 15 and 16, and taking the average again, but this will include even higher-order moments. Repeating will result in a moment hierarchy that is more complicated than the original equations.

However, if all higher-order moments factorize into products of the means (i.e., cumulants are zero), then we obtain the mean field theory of the spiking network Eqs. 12 and 13:

$${\partial }_t〈{\eta }_i\left(\theta ,t\right)〉+{\partial }_{\theta }\left[F_i\left(\theta ,〈s_i〉\right)〈{\eta }_i\left(\theta ,t\right)〉\right]=0$$ (17)

$${\tau }_s〈{\dot{s}}_i〉=-〈s_i〉+\frac{1}{N}{\displaystyle \sum _j{w}_{ij}{F}_j\left(\pi ,〈s_j〉\right)〈{\eta }_j\left(\pi ,t\right)〉}$$ (18)

which does not match any of the mean field activity equations in what are the wilson–cowan equations?. However, since $F_j\left(\pi ,〈s_j〉\right)〈{\eta }_j\left(\pi ,t\right)〉$ is the mean firing rate of neuron j, then we see that Eq. 18 has the form of Eq. 2 with time-dependent firing rate dynamics given by Eq. 17. If $〈{\eta }_i〉$ were to go to steady state, then Eq. 17 with ${\partial }_t〈{\eta }_i\left(\theta ,t\right)〉$ set to zero can be solved to yield

$$〈{\eta }_i\left(\theta \right)〉=\frac{C_j}{〈F_j\left(\theta ,〈s_j〉\right)〉}$$

where C_j is determined by the normalization condition ${\displaystyle {\int }_{\theta }{\eta }_j\left(\theta \right)}\text{d}\theta =1$. We can then obtain Eq. 2 with gain function

$$f_j\left(s_i\right)=\frac{C_j{F}_j\left(\pi ,〈s_j〉\right)}{〈F_j\left(\theta ,〈s_j〉\right)〉}$$

Thus the WC equations are a description of mean field theory of the steady state, although Eq. 17 is dissipation free and thus will not relax to steady state from all initial conditions without a noise source or finite-size effects (Buice and Chow 2007, 2013a; Hildebrand et al. 2007).

This formalism could be applied to any type of neuron and synaptic dynamics. The neuron model (Eqs. 12 and 13) was chosen for simplicity and also so that the mean field equations would be similar to those of what are the wilson–cowan equations?. The inclusion of higher-order time derivatives or nonlinear terms could result in very different mean field equations. A similar argument to arrive at mean field theory could be applied with the population average, although this will be further complicated by the neuron-dependent weight w_ij. No matter what average is used, we still do not know if or when mean field theory is valid. To answer this question, we need to compute the corrections to mean field theory and see when they are small.

BEYOND MEAN FIELD THEORY

Although mean field theory is highly successful in describing neural activity (Coombes 2005), there are reasons for trying to go beyond it. First, correlations have been found to be prevalent in cortical activity (Averbeck et al. 2006; Doiron et al. 2016; Salinas and Sejnowski 2001; Schneidman et al. 2006). Second, the number of neurons in any neural population is finite, which can make any mean field approximation susceptible to finite-size effects (Buice and Chow 2007, 2013a; Hildebrand et al. 2007). Third, mean field theory may not qualitatively describe the mean dynamics correctly. For example, in the Kuramoto coupled oscillator model, the steady state is marginally stable in mean field theory, yet simulations show that the system relaxes to steady state rather quickly because of finite-size effects beyond mean field theory (Buice and Chow 2007; Hildebrand et al. 2007). Finally, mean field theory cannot provide a systematic method for evaluating its validity and accuracy.

An inherent assumption of mean field theory is that there is a probability distribution over which averages can be taken. In the case of neural activity, such a probability distribution could be high and even infinite dimensional, and the validity of any mean field theory relies on the assumption that the probability is highly localized around a maximum, so that the fluctuations are negligible. Consider again the Klimontovich formulation of the spiking network Eqs. 15 and 16

$${\mathcal{G}}_{\eta }\left[{\eta }_i\right]\equiv {\partial }_t{\eta }_i\left(\theta ,t\right)\text{+}{\partial }_{\theta }{F}_i\left(\theta ,s_i\right){\eta }_i\left(\theta ,t\right)-\delta \left(t-t_0\right){\eta }_i^0\left(\theta \right)=0$$ (19)

$${\mathcal{G}}_s\left[s_i\right]\equiv {\dot{s}}_i\left(t\right)\text{+}{s}_i\left(t\right)-\frac{1}{N}{\displaystyle \sum _{j=1}^N{w}_{ij}}{F}_j\left(\pi ,t\right){\eta }_j\left(\pi ,t\right)-\delta \left(t-t_0\right)s_i^0=0$$ (20)

where we have expressed the initial conditions as explicit forcing terms and set τ_s = 1.

We can formally define an empirical probability density functional (PDF) for η_i(θ,t) and s_i(t) as Dirac delta functionals constrained to Eqs. 19 and 20, to wit

$$\begin{array}{c}\mathcal{P}\left[\eta ,s|{\eta }^0,s^0\right]={\displaystyle \prod _i\delta \left[{\partial }_t{\eta }_i+{\partial }_{\theta }{F}_i\left(\theta ,s_i\right){\eta }_i-\delta \left(t-t_0\right){\eta }_i^0\left(\theta \right)\right]}\\ \times \delta \left[{\dot{s}}_i+s_i-\frac{1}{N}{\displaystyle \sum _j{w}_{ij}{F}_j\left(\pi ,s_j\right)}{\eta }_j\left(\pi ,t\right)-s_i^0\delta \left(t-t_0\right)\right]\end{array}$$

This formal expression can be rendered useful with the functional Fourier transform expression of the Dirac delta functional (Chow and Buice 2015):

$$\begin{array}{c}\delta \left[{\mathcal{G}}_{\eta }\right]={{\displaystyle \int \mathcal{D}\tilde{\eta }e}}^{-{\displaystyle \int \text{d}t\text{d}}\theta {\tilde{\eta }}_i\left(\theta ,t\right){\mathcal{G}}_{\eta }\left[{\eta }_i\left(\theta ,t\right)\right]}\\ \delta \left[{\mathcal{G}}_{\mathcal{S}}\right]={{\displaystyle \int \mathcal{D}\tilde{s}e}}^{-{\displaystyle \int \text{d}t}{\tilde{s}}_i\left(t\right){\mathcal{G}}_{\mathcal{S}}\left[s_i\left(t\right)\right]}\end{array}$$

where ${\tilde{\eta }}_i\left(\theta ,t\right)$ and s̃_i(t) are functional Fourier transform variables, called response fields, and the integral is over all possible paths or histories of these functions. We have absorbed the factors of 2π into the functional differentials $\mathcal{D}\tilde{\eta }$ and $\mathcal{D}\tilde{s}$. This results in the functional or path integral expression

$$\mathcal{P}\left[\eta ,s|{\eta }^0,s^0\right]={\displaystyle \int \mathcal{D}\tilde{\eta }\mathcal{D}\tilde{s}{e}^{-{\Sigma }_i{\displaystyle \int \text{d}t\text{d}\theta {\tilde{\eta }}_i}{\mathcal{G}}_{\eta }\left[{\eta }_i\right]-{\Sigma }_i{\displaystyle \int \text{d}t{\tilde{s}}_i{\mathcal{G}}_{\mathcal{S}}\left[s_i\right]}}}$$

At this point, the PDF has no uncertainty and thus all cumulants beyond the mean are zero, which is expected since we started with a fully deterministic system. However, uncertainty can be introduced into a deterministic system if we consider averaging over an ensemble of initial conditions. Assuming fixed $s_i^0$ and a functional distribution for the initial phase density $\mathcal{P}\left[{\eta }^0\right]$,we can integrate or marginalize over the initial condition densities:

$$\mathcal{P}\left[\eta ,s\right]={\displaystyle \int \mathcal{D}{\eta }^0\mathcal{P}\left[\eta ,s|{\eta }^0,s^0\right]\mathcal{P}\left[{\eta }^0\right]}$$ (21)

If we set ${\eta }_i^0\left(\theta \right)=\delta \left(\theta -{\theta }_i\left(t=0\right)\right)$, the distribution over initial densities is given by the distribution over the initial phase ${\rho }_i^0\left(\theta \right)$, which implies $\mathcal{D}{\eta }^0\mathcal{P}\left[{\eta }^0\right]={\displaystyle {\prod }_i\text{d}{\theta }_i{\rho }_i^0\left({\theta }_i\right)}$. The initial condition contribution is then given by the integral

$$\begin{array}{c}{e}^{W_0\left[\tilde{\eta }\right]}={\displaystyle \int {\displaystyle \prod _i\text{d}{\theta }_i{\rho }_i^0}\left({\theta }_i\right)}{e}^{{\Sigma }_i{\tilde{\eta }}_i\left({\theta }_i{t}_0\right)}\\ ={\displaystyle \prod _i{\displaystyle \int \text{d}\theta {\rho }_i^0}}\left(\theta \right)e^{{\tilde{\eta }}_i\left({\theta }_i{t}_0\right)}\\ =e^{{\Sigma }_i\text{ln}\left(1-{\displaystyle \int \text{d}\theta }{\rho }_i^0\left(\theta \right)\left(e^{{\tilde{\eta }}_i\left({\theta }_i{t}_0\right)}-1\right)\right)}\end{array}$$

where W₀ is a cumulant generating functional (see appendix) for the initial neuron density η. Thus network dynamics defined by Eqs. 19 and 20 are mapped to the PDF

$$\mathcal{P}\left[\eta ,s\right]={\displaystyle \int \mathcal{D}{\eta }^0\mathcal{D}\tilde{\eta }\mathcal{D}s{e}^{-NS}}$$

where

$$S=\frac{1}{N}[{\displaystyle \sum _i{\displaystyle {\int }_{t_0}^{t_1}\text{d}t{\displaystyle {\int }_{-\pi }^{\pi }\text{d}\theta {\tilde{\eta }}_i\left(\theta ,t\right)}}}\left[{\partial }_t{\eta }_i\left(\theta ,t\right)+{\partial }_{\theta }{F}_i\left(\theta ,s_i\right){\eta }_i\left(\theta ,t\right)\right]+{\displaystyle {\int }_{t_0}^{t_1}\text{d}t{\tilde{s}}_i\left(t\right)}\left({\dot{s}}_i+s_i-\frac{1}{N}{\displaystyle \sum _j{w}_{ij}{F}_j\left(\pi ,s_j\right)}{\eta }_j\left(\pi ,t\right)-\delta \left(t-t_0\right)s_i^0\right)+W_0]$$

is called the action. The exponential of $\tilde{\eta }$ in W₀ (which has the form of a generating function for a Poisson distribution) can be simplified via the Doi–Peliti–Janssen transformation (Buice et al. 2010; Buice and Chow 2007, 2013a, 2013b, 2013c; Buice and Cowan 2007; Hildebrand et al. 2007):

$$\begin{array}{c}{\psi }_i={\eta }_i\text{exp}\left(-{\tilde{\eta }}_i\right)\\ {\tilde{\psi }}_i=\text{exp}\left({\tilde{\eta }}_i\right)-1\end{array}$$

resulting in the transformed action

$$S=\frac{1}{N}{\displaystyle \sum _i[{\displaystyle \int \text{d}\theta dt{\tilde{\psi }}_i\left(\theta ,t\right)}}\left[{\partial }_t{\psi }_i\left(\theta ,t\right)+{\partial }_{\theta }{F}_i\left(\theta ,s_i\right){\psi }_i\left(\theta ,t\right)\right]+{\displaystyle \int \text{d}t{\tilde{s}}_i\left(t\right)}({\dot{s}}_i\left(t\right)+\beta s_i-\delta \left(t-t_0\right)s_i^0-\frac{1}{N}{\displaystyle \sum _j{w}_{ij}{F}_j\left(\pi ,s_j\right)}\left({\tilde{\psi }}_j\left(\pi ,t\right)+1\right){\psi }_j\left(\pi ,t\right))+\text{ln}\left(1-{\displaystyle \int \text{d}}\theta {\rho }_i^0\left(\theta \right){\tilde{\psi }}_i\left(\theta ,t_0\right)\right)]$$ (22)

where noncontributing terms that arise after integration by parts are ignored.

Given the PDF specified by the action (Eq. 22), the generating functional (GF) can be constructed and all moments or cumulants are obtained from functional derivatives of the GF (see Generating Functional, appendix). However, except for a few cases (see Complex Gaussian Integral and Wick’s Theorem, appendix), the GF does not have a closed-form solution and the computation of the moments hinges on constructing a perturbative expansion. If the action is divided by a small parameter ϵ, then the action can be expanded around its critical point as a series in ϵ via a functional version of Laplace’s method (see Laplace’s Method, appendix). In quantum field theory, this is known as a semiclassical expansion where ϵ is Planck’s constant. The expansion utilizes the property that the Gaussian integral, for real or complex variables, exists in any dimension including infinite dimensional functional spaces (see appendix). All moments or cumulants of the Gaussian are reducible to combinations of the first and second moments, which have closed-form expressions.

Mean field theory is the lowest order of the semiclassical expansion. Corrections at each successive power of ϵ require the computation of integrals that grow in complexity and number quickly, but the procedure can be systematized by the use of Feynman diagrams or graphs where the vertices represent the nonquadratic terms and the edges are the (co)variances, which for dynamical systems are Green’s functions or propagators (Zinn-Justin 1996). The semiclassical expansion is also called a loop expansion because each Feynman graph contains a closed path or loop for each order in ϵ (beyond the lowest). Mean field theory has no loops, and all graphs pertaining to mean field theory are directed acyclic graphs or tree graphs.

The small parameter ϵ quantifies a deviation from mean field theory and could be the inverse of the number of neurons in a network of spiking neurons or a more abstract quantity such as a measure of the inverse distance of a network parameter to a bifurcation or critical point of the network dynamics (Buice et al. 2010; Buice and Cowan 2007). The path integral approach has been used to explore the impact of network connectivity and the shape of the transfer function on neural dynamics (Crisanti and Sompolinsky 2018; Ocker et al. 2017).

EFFECTIVE ACTION AND GENERALIZED ACTIVITY EQUATIONS

Although the mean and higher-order moments can be obtained to any desired order by a loop expansion, the procedure is unwieldy if there is no closed-form solution for the mean field equations. A strategy to circumvent this complication is to construct a set of closed equations of motion for the desired moments directly (e.g., mean and second moment). This is achieved through a Legendre transformation of the cumulant generating functional into an “effective action” in terms of the desired moments. The Legendre transformation converts a convex function of a set of variables into a function of the derivatives with respect to those variables, while preserving the extrema of the function. For example, the effective action for the mean a is given by

$$\Gamma \left[a\right]=J\cdot a-W\left[J\right]$$ (23)

subject to constraints

$$a_i^{\alpha }={\left.\frac{\delta W}{\delta J_i^{\alpha }}\right|}_{J=0}$$ (24)

$$J_i^{\alpha }=\frac{\delta \Gamma \left[a\right]}{\delta a_i^{\alpha }}$$ (25)

where a is a multivector of activity fields, J is a multivector of auxiliary fields, and W[J] = lnZ[J] is the cumulant GF. The equation of motion for the true mean a, which includes all corrections, is given by Eq. 25 with J = 0. In field theory, Eq. 23 is referred to as the one-particle irreducible (1PI) effective action.

Setting Φ = a + Ψ in the action of the generating functional yields

$$\Gamma \left[a\right]=\frac{1}{N}\text{ln}{\displaystyle \int \mathcal{D}\psi e^{\left(NS\left[a\right]+O\left(\left|\psi \right|\right)\right)}}$$ (26)

and thus Γ[a] = S[a] + O(|Ψ|) and mean field theory is given by the limit of Ψ = 0. Using Eq. 25 with J = 0 gives the mean field theory equations of motion

$$\frac{\delta S\left[a\right]}{\delta a}=0$$

Higher-order terms in 1/N, which are finite size corrections, can be obtained via a loop expansion.

The effective action formalism can be generalized to obtain a similar set of hierarchical self-contained equations of motion in terms of the mean and covariances or even higher-order moments. Consider the moment generating functional where we include an explicit auxiliary field for second moments:

$$W\left[J,K\right]=\text{ln}{\displaystyle \int \mathcal{D}\Phi \mathcal{P}\left[\Phi \right]}{e}^{J\cdot \Phi +\frac{1}{2}{\Phi }^T\cdot K\cdot \Phi }$$ (27)

Although this extra field is not necessary to compute the second moments, it is used to generate a so called two-particle irreducible (2PI) effective action in terms of the mean and covariances. A Legendre transformation yields

$$\begin{array}{c}\Gamma \left[\Psi ,C\right]=-W\left[J,K\right]+{\displaystyle \sum _{\alpha }{\displaystyle \int \text{d}x{J}^{\alpha }\left(x\right)}}{\Phi }^{\alpha }\left(x\right)\\ +{\displaystyle \sum _{\alpha ,\beta }\frac{1}{2}{\displaystyle \int \text{d}x\text{d}x\prime }}\left[{\Phi }^{\alpha }\left(x\right){\Phi }^{\beta }\left(x\prime \right)+C^{\alpha \beta }\left(x,x\prime \right)\right]K^{\alpha \beta }\left(x,x\prime \right)\end{array}$$ (28)

under the constraints

$$\frac{\delta W}{\delta J^{\alpha }}={\Psi }^{\alpha }$$ (29)

$$\frac{\delta W}{\delta J^{\alpha }\delta J^{\beta }}=\frac{1}{2}\left[{\Psi }^{\alpha }{\Psi }^{\beta }+C^{\alpha \beta }\right]$$ (30)

and

$$\frac{\delta \Gamma }{\delta a_{\alpha }}=J^{\alpha }+\frac{1}{2}{a}_{\beta }\left[K^{\alpha \beta }+K^{\beta \alpha }\right]$$ (31)

$$\frac{\delta \Gamma }{\delta C_{\alpha \beta }}=\frac{1}{2N}{K}^{\alpha \beta }$$ (32)

An expansion to first order in 1/N yields (Buice and Chow 2013c):

$$\Gamma \left[a,C\right]=S\left[a\right]+\frac{1}{2N}\text{Tr\hspace{0.17em}ln}\left(C^{-1}\right)+\frac{1}{2N}\text{Tr\hspace{0.17em}}{L}^{\alpha \beta }{C}_{\alpha \beta }$$ (33)

where

$$\text{Tr\hspace{0.17em}}{A}^{\alpha \beta }{B}_{\alpha \beta }={\displaystyle \int \text{d}x\text{d}x}\prime A^{\alpha \beta }\left(x,x\prime \right)B_{\alpha \beta }\left(x,x\prime \right)$$

and

$${{\displaystyle \int \text{d}x\prime \left(C^{-1}\right)}}^{\alpha \gamma }\left(x,x\prime \right)C_{\gamma \beta }\left(x\prime ,x_0\right)={\delta }_{\alpha \beta }\delta \left(x\prime -x_0\right)$$

Applying Eq. 32 on Eq. 33 yields the equations of motion up to N⁻¹ with J = 0 and K = 0:

$$\begin{array}{l}\frac{\delta S\left[a\right]}{\delta a_i}+\frac{1}{2N}\frac{\delta }{\delta a_i}\text{Tr\hspace{0.17em}}{L}^{\alpha \beta }{C}_{\alpha \beta }=0\\ \frac{1}{2N}\left[L^{\alpha \beta }-{\left(C^{-1}\right)}^{\alpha \beta }\right]=0\end{array}$$

These equations represent a set of generalized activity equations for the mean activity and correlations between pairs of neurons. The procedure could be extended to any set of cumulants.

FINAL WORDS

We have provided a brief and biased view of the ideas behind the Wilson–Cowan equations and their descendants. We show that one can arrive at a similar set of activity equations with first-order time derivatives, a nonlinear activation (gain) function, and a linear sum over inputs, from various starting points. Despite the fact that the equations are not generally quantitatively accurate for a network of spiking neurons, they seem to capture many facets of neural activity and function. They are a set of effective equations that have stood the test of time. Mean field theory is implicitly assumed in all of these equations. Correlations are deliberately ignored to make the equations tractable, but given that they are omnipresent in the brain, there has been ongoing effort to account for them. Nonetheless, we predict that forty-seven years from now, the Wilson–Cowan equations will still have great utility.

GRANTS

This work was supported by the Intramural Research Program of the National Institute of Diabetes and Digestive and Kidney Diseases, NIH.

DISCLOSURES

No conflicts of interest, financial or otherwise, are declared by the authors.

AUTHOR CONTRIBUTIONS

C.C.C. and Y.K. conceived and designed research; C.C.C. and Y.K. analyzed data; Y.K. prepared figures; C.C.C. and Y.K. drafted manuscript; C.C.C. and Y.K. edited and revised manuscript; C.C.C. and Y.K. approved final version of manuscript.

APPENDIX

Generating Functional

For a normalized PDF, $\mathcal{P}\left(x\right)$, the moment generating function is defined as

$$Z\left(J\right)=〈e^{Jx}〉={\displaystyle \int \mathcal{P}\left(x\right)}{e}^{Jx}\text{d}x$$

which is the Laplace transform of $\mathcal{P}\left(x\right)$. Moments of the PDF are obtained by

$${\left.〈x^n〉=\frac{d^n}{d{J}^n}Z\left(J\right)\right|}_{J=0}$$

The cumulant generating function is defined as W = lnZ. The same notion can be generalized to an infinite dimensional PDF of a function. Given a probability density functional, $\mathcal{P}\left[\varphi \left(x,t\right)\right]$, of an arbitrary field ϕ(x,t), the generating functional is defined as

$$Z\left[J\right]={\displaystyle \int \mathcal{D}\varphi \mathcal{P}\left[\varphi \left(x,t\right)\right]}{e}^{J\cdot \varphi }$$

where

$$J\cdot \varphi \equiv {\displaystyle \int J\left(x,t\right)}\varphi \left(x,t\right)\text{d}x\text{d}t$$

and $\mathcal{D}\varphi \equiv {\text{lim}}_{n\to \infty }{\displaystyle {\prod }_i^n\text{d}{\varphi }_i}$. Moments are given by

$${〈{\displaystyle \prod _j\varphi \left(x_j,t_j\right)}〉={\displaystyle \prod _j\frac{\delta }{\delta J_j}}Z\left[J\right]|}_{J=0}$$

where the functional derivative is defined as

$$\frac{\delta F\left[f\left(x\right)\right]}{\delta f\left(x\prime \right)}=F\prime \left[f\right]\delta \left(x-x\prime \right)$$

Similarly, the cumulants are obtained by derivatives of the cumulant generating functional.

$${{〈{\displaystyle \prod _j\varphi \left(x_j,t_j\right)}〉}_c={\displaystyle \prod _j\frac{\delta }{\delta J\left(x_j{t}_j\right)}},W\left[J\right]|}_{J=0}$$

Complex Gaussian Integral and Wick’s Theorem

Consider the complex Gaussian integral

$$\frac{1}{2\pi \gamma }{\displaystyle \int \text{d}\tilde{x}\text{d}x{e}^{-\frac{1}{\gamma }\tilde{x}x}}={\displaystyle \int \text{d}x\delta \left(x\right)}=1$$

where x̃ is pure imaginary. The generating functional for the complex Gaussian is

$$Z\left[J,\tilde{J}\right]=\frac{1}{2\pi \gamma }{\displaystyle \int \text{d}\tilde{x}\text{d}x{e}^{-\frac{1}{\gamma }\tilde{x}x+J\tilde{x}+\tilde{J}x}}=e^{\gamma \tilde{J}J}$$

This generalizes directly to the functional case

$$Z\left[J,\tilde{J}\right]={\displaystyle \int \mathcal{D}\varphi \mathcal{D}\tilde{\varphi }{e}^{-{\displaystyle \int \text{d}{z}_1\text{d}{z}_2}\tilde{\varphi }\left(z\prime \right){\Delta }^{-1}\left(z_1,z_2\right)\varphi \left(z_2\right)+{\displaystyle \int \text{d}zJ}\left(z\right)\tilde{\varphi }\left(z\right)+{\displaystyle \int \text{d}z\dot{J}\left(z\right)}\varphi \left(z\right)}}=e^{{\displaystyle \int \text{d}{z}_1\text{d}{z}_2\tilde{J}\left(z_1\right)}\Delta \left(z_1{z}_2\right)J\left(z_2\right)}$$ (A1)

where we have absorbed the normalization factors into the functional differentials $\mathcal{D}\varphi$ and $\mathcal{D}\tilde{\varphi }$. Since the exponent in Eq. A1 is bilinear in J and $\tilde{J}$, the only nonzero moments are bilinear in $\tilde{\varphi }$ and ϕ. Second moments are given by

$$〈\varphi \left(x,t\right)\tilde{\varphi }\left(x\prime ,t\prime \right)〉={\left.\frac{\delta }{\delta \tilde{J}\left(x,t\right)}\frac{\delta }{\delta J\left(x\prime ,t\prime \right)}Z\left[J,\tilde{J}\right]\right|}_{J=0,\tilde{J}=0}=\Delta \left(x,t,x\prime t\prime \right)$$

and completely specify the distribution (since the mean is zero). Each bilinear pair contributes a factor of Δ in higher moments, combined in all possible pairings

$$〈{\displaystyle \prod _{i=1}^m{\displaystyle \prod _{j=1}^m\phi \left(x_i,t_i\right)\tilde{\phi }\left(x_j^{\prime },t_j^{\prime }\right)}}〉={\displaystyle \sum _{\text{all\hspace{0.17em}possible\hspace{0.17em}pairings}}{\Delta }_{i_1,j_2}\dots {\Delta }_{i_{2m--1}{j}_{2m}}}$$

where ${\Delta }_{i_1,j_2}\equiv \Delta \left(x_{i_1},t_{i_2},x_{j_2},t_{j_2}\right)$. For instance, the fourth-order moment $〈{\varphi }_1{\varphi }_2{\varphi }_3{\varphi }_4〉={\Delta }_{12}{\Delta }_{34}+{\Delta }_{13}{\Delta }_{24}+{\Delta }_{14}{\Delta }_{23}$.

Laplace’s Method

Laplace’s method, also called the method of steepest descent when extended to the complex domain, can be used to compute functional integrals of the form

$$I={\displaystyle {\int }_{\mathcal{L}}\mathcal{D}\varphi e^{-\frac{1}{ϵ}{\displaystyle \int \text{d}zS\left[\varphi \left(z\right)\right]}}}$$ (A2)

where ϵ ≪1 and the integrand of Eq. A2 is analytic. If S has a critical point (minimum) on the contour $\mathcal{L}$ then exp(−S/ϵ) will be sharply peaked around that critical point and the integral can be completed as a series expansion in ϵ. A critical point of S satisfies the condition δS/δϕ(z) = 0. If ϕ₀ is the critical point and we set ψ = ϕ − ϕ₀, then we can expand

$$\begin{array}{c}S\left[\varphi \right]=S\left[{\varphi }_0\right]+\frac{1}{2!}{\displaystyle \int \text{d}{z}_1\text{d}{z}_2{S}^{\left(2\right)}}\left(z_1,z_2\right)\psi \left(z_1\right)\psi \left(z_2\right)\\ +\frac{1}{3!}{\displaystyle \int \text{d}{z}_1\text{d}{z}_2\text{d}{z}_3{S}^{\left(3\right)}}\left(z_1,z_2,z_3\right)\left({\varphi }_0\right)\psi \left(z_1\right)\psi \left(z_2\right)\psi \left(z_3\right)+\dots \end{array}$$

where

$$S^{\left(n\right)}\left(z_1,\dots ,z_n\right)=\frac{{\delta }^{n}S\left({\varphi }_0\right)}{\delta \varphi \left(z_1\right)\dots \delta \varphi \left(z_n\right)}$$

The integral Eq. A2 now has the form

$$\begin{array}{c}I=e^{-S\left({\varphi }_0\right)}{\displaystyle {\int }_{\mathcal{L}}\mathcal{D}\psi e^{-\frac{1}{2ϵ}{\displaystyle \int \text{d}{z}_1\text{d}{z}_2{S}^{\left(2\right)}\left(z_1,z_2\right)\psi \left(z_1\right)\psi \left(z_2\right)}}}\\ \times \left(1+\frac{1}{3!ϵ}{\displaystyle \int \text{d}{z}_4\text{d}{z}_5\text{d}{z}_6{S}^{\left(3\right)}}\left({\varphi }_0\right)\psi \left(z_4\right)\psi \left(z_5\right)\psi \left(z_6\right)+\dots \right)\end{array}$$

Hence, the integral Eq. A2 has been transformed into a sum over moments of the complex Gaussian (see Complex Gaussian Integral and Wick’s Theorem). Each term in the series will reduce to a set of integrals over products of the covariances ϵ(S⁽²⁾)⁻¹.

GLOSSARY

a_i

Mean neural activity of unit i (local population or single neuron)

A_i

Proportion of activated neurons of population i

C_ij

Correlation function between neuron i and j

f_i

Nonlinear activation function of unit i

F_i(θ_i,s_i)

Phase velocity function for neuron i

I₀(x)

Probability bulb group of a neuron at x originates from a neuron firing at t = 0

O(x,x′)

Probability bulb group at x originates from x′

$\mathcal{P}$

Probability density functional

P_nph[I₀(x)]

Probability neuron at x of bulb group p has firing threshold h and n axon terminals

Q

Proportion of cells sensitive to input

r

Refractory period

S

Action

$S_J^F$

Postsynaptic spike train (spike-response formalism)

s_i

Synaptic drive to unit i

W[J,J̃]

Cumulant generating functional

Z[J,J̃]

Moment generating functional

α(t − t′)

Temporal response function

Δ_ij(x,t,x′,t′)

Propagator

η^refr(t − t′)

Refractory function (spike-response formalism)

η(θ,t)

Phase density function

κ(τ,t′)

Response kernel (spike-response formalism)

Γ[a]

Effective action