Predicting how and when hidden neurons skew measured synaptic interactions

Abstract

A major obstacle to understanding neural coding and computation is the fact that experimental recordings typically sample only a small fraction of the neurons in a circuit. Measured neural properties are skewed by interactions between recorded neurons and the “hidden” portion of the network. To properly interpret neural data and determine how biological structure gives rise to neural circuit function, we thus need a better understanding of the relationships between measured effective neural properties and the true underlying physiological properties. Here, we focus on how the effective spatiotemporal dynamics of the synaptic interactions between neurons are reshaped by coupling to unobserved neurons. We find that the effective interactions from a pre-synaptic neuron r′ to a post-synaptic neuron r can be decomposed into a sum of the true interaction from r′ to r plus corrections from every directed path from r′ to r through unobserved neurons. Importantly, the resulting formula reveals when the hidden units have—or do not have—major effects on reshaping the interactions among observed neurons. As a particular example of interest, we derive a formula for the impact of hidden units in random networks with “strong” coupling—connection weights that scale with $1/\sqrt{N}$, where N is the network size, precisely the scaling observed in recent experiments. With this quantitative relationship between measured and true interactions, we can study how network properties shape effective interactions, which properties are relevant for neural computations, and how to manipulate effective interactions.

Author summary

No experiment in neuroscience can record from more than a tiny fraction of the total number of neurons present in a circuit. This severely complicates measurement of a network’s true properties, as unobserved neurons skew measurements away from what would be measured if all neurons were observed. For example, the measured post-synaptic response of a neuron to a spike from a particular pre-synaptic neuron incorporates direct connections between the two neurons as well as the effect of any number of indirect connections, including through unobserved neurons. To understand how measured quantities are distorted by unobserved neurons, we calculate a general relationship between measured “effective” synaptic interactions and the ground-truth interactions in the network. This allows us to identify conditions under which hidden neurons substantially alter measured interactions. Moreover, it provides a foundation for future work on manipulating effective interactions between neurons to better understand and potentially alter circuit function—or dysfunction.

Introduction

Establishing relationships between a network’s architecture and its function is a fundamental problem in neuroscience and network science in general. Not only is the architecture of a neural circuit intimately related to its function, but pathologies in wiring between neurons are believed to contribute significantly to circuit dysfunction [1–15].

A major obstacle to uncovering structure-function relationships is the fact that most experiments can only directly observe small fractions of an active network. State-of-the-art methods for determining connections between neurons in living networks infer them by fitting statistical models to neural spiking data [16–25]. However, the fact that we cannot observe all neurons in a network means that the statistically inferred connections are “effective” connections, representing some dynamical relationship between the activity of nodes but not necessarily a true physical connection [24–33]. Intuitively, reverberations through the network must contribute to these effective interactions; our goal in this work is to formalize this intuition and establish a quantitative relationship between measured effective interactions and the true synaptic interactions between neurons. With such a relationship in hand we can study the effective interactions generated by different choices of synaptic properties and circuit architectures, allowing us to not only improve interpretation of experimental measurements but also probe how circuit structure is tied to function.

The intuitive relationship between measured and effective interactions is demonstrated schematically in Fig 1. Fig 1A demonstrates that in a fully-sampled network the directed interactions between neurons—here, the change in membrane potential of the post-synaptic neuron after it receives a spike from the pre-synaptic neuron—can be measured directly, as observation of the complete population means different inputs to a neuron are not conflated. However, as shown in Fig 1B, the vastly more realistic scenario is that the recorded neurons are part of a larger network in which many neurons are unobserved or “hidden.” The response of the post-synaptic neuron 2 to a spike from pre-synaptic neuron 1 is a combination of both the direct response to neuron 1’s input as well as input from the hidden network driven by neuron 1’s spiking. Thus, the measured membrane response of neuron 2 due to a spike fired by neuron 1—which we term the “effective interaction” from neuron 1 to 2—may be quite different from the true interaction. It is well-known that circuit connections between recorded neurons, as drawn in Fig 1C, are at best effective circuits that encapsulate the effects of unobserved neurons, but are not necessarily indicative of the true circuit architecture. The formalized relationship we will establish in the Results is given in Fig 2.

Fig 1. The hidden unit problem.

A. In a hypothetical circuit consisting of just two recorded neurons (no hidden neurons), we can measure the strength and time course of the directed interactions between neurons by measuring the response of the post-synaptic neuron’s membrane potential to a spike from the pre-synaptic neuron. B. Realistically, there are many more neurons in the network that are unrecorded and hence “hidden.” In this schematic, only two neurons are observed. The hidden neurons are driven by input from the presynaptic neuron labeled 1, and provide input to the recorded post-synaptic neuron labeled 2. Because the activity of the hidden neurons is not controlled, the membrane response reflects a combination of neuron 1’s direct influence on neuron 2 and its indirect influence through the hidden network. C. The “effective” 2 neuron network observed experimentally.

Fig 2. Expansion of effective interactions into contributions from hidden paths.

A. Graphical representation of Eq (2). The linear response of the hidden network, ${\widehat{\Gamma }}_{h,h^{\prime }}\left(\omega \right)$, has been expanded as a series (corresponding to the grey hidden nodes and links between them), such that each term in the overall series can be interpreted as a contribution from a path through which the pre-synaptic neuron r′ is able to send a signal to post-synaptic neuron r via 1, 2, etc. hidden neurons. This expression holds for any pair of neurons in the recorded subset. B. Quantitative expressions for each diagram in the series can be read off by assigning the shown factors for each hidden neuron node and each link between neurons, recorded or hidden, and multiplying them together. (No factor is assigned to the recorded neuron nodes). γ_h is the gain of neuron h and ${\widehat{J}}_{i,j}\left(\omega \right)$ is the true interaction from j to i in the frequency domain.

Even once we establish a relationship between the effective and true connections, we will in general not be able to use measurements of effective interactions to extrapolate back to a unique set of true connections; at best, we may be able to characterize some of the statistical properties of the full network. The obstacle is that several different networks—different both in terms of architecture and intrinsic neural properties—may give rise to the same network behaviors, a theme of much focus in the neuroscience literature [34–39]. That is, inferring the connections and intrinsic neural properties in a full network from activity recordings from a subset of neurons is in general an ill-posed problem, possessing several degenerate solutions. Several statistical inference methods have been constructed to attempt to infer the presence of, and connections to, hidden neurons [28, 40–42]; the subset of the degenerate solutions that each of these methods finds will depend on the particular assumptions of the inference method (e.g., the regularization penalties applied). As an example, we demonstrate two small circuit motifs that give rise to nearly identical effective interactions, despite crucial differences between the circuits (Figs 3 and 4).

Fig 3. 3 neuron feedforward inhibition circuit.

A: A 3-neuron circuit displaying feedforward inhibition. Neuron 1 provides excitatory input to neurons 2 and 3, while neuron 3 provides inhibitory input to neuron 2. Neuron 3 also has a self-history coupling, denoted by an autaptic loop, which implements a refractory period in this circuit model. B: Leftmost, the effective interaction from neuron 1 to 2 when neuron 3 is unobserved. Subsequent plots decompose this interaction into contributions from neuron 1’s direct input to neuron 2, and its indirect input through neuron 3. The indirect input through neuron 3 also takes account of neuron 3’s self-history interaction. C. Leftmost, the effective interaction (membrane response) from neuron 1 to 2, subsequently decomposed into contributions from the direct interaction and the indirect interaction from 1 to 2.

Fig 4. Different complete circuits may underly similar effective circuits.

A: A circuit very similar to that in Fig 3, except that neuron 1 also provides excitatory input to neuron 4, which in turn provides inhibitory input to neuron 3. The self-history coupling of neuron 3 to itself has also been removed in this example. B: Leftmost, the effective interaction from neuron 1 to 2, which is qualitatively and quantitatively similar to the effective interaction shown in Fig 3. Subsequent plots indicate each path through the circuit that neuron 1 can send a signal to neuron 2 through the hidden neurons 3 and 4. C. Leftmost, the effective interaction from neuron 1 to 2. Subsequent plots decompose this interaction into contributions from the paths shown above in B.

Understanding the effect of hidden neurons on small circuit motifs is only a piece of the hidden neuron puzzle, and a full understanding necessitates scaling up to large circuits containing many different motifs. Having an analytic relationship between true and effective interactions greatly facilitates such analyses by directly studying the structure of the relationship itself, rather than trying to extract insight indirectly through simulations. In particular, in going to large networks we focus on the degree to which hidden neurons skew measured interactions (Fig 5), and how we can predict the features of effective interactions we expect to measure when recording from only a subset of neurons in a network with hypothesized true interactions (Fig 6).

Fig 5. Relative changes in interaction strength due to hidden neurons for three network types.

We quantify relative changes in interaction strength between effective (${\mathcal{J}}_{r,r^{\prime }}^{\text{eff}}$) and true (${\mathcal{J}}_{r,r^{\prime }}$) interactions by the (sample) root-square-mean deviation, $\sigma [{\mathcal{J}}_{r,r^{\prime }}^{\text{eff}}-{\mathcal{J}}_{r,r^{\prime }}]$, normalized by the true synaptic weight (sample) standard deviation $\sigma \left[{\mathcal{J}}_{r,r^{\prime }}\right]$. We do so for three (sparse) network types: Left. An Erdős-Réyni (ER) network with “mixed synapses” (i.e., Dale’s law not imposed) with normally distributed synaptic weights. Middle. An ER network with Dale’s law imposed, (i.e., each neuron’s outgoing synaptic weights all have the same sign). Right. A Watts-Strogatz (WS) small world network with 30% rewired connections and mixed synapses. All network types yield qualitatively similar results. In each plot solid lines are numerical estimates of the sample standard deviation of the difference between effective coupling weights ${\mathcal{J}}_{r,r^{\prime }}^{\text{eff}}$ and true coupling weights ${\mathcal{J}}_{r,r^{\prime }}$ between neurons r ≠ r′, normalized by the standard deviation of ${\mathcal{J}}_{r,r^{\prime }}$. These estimates account for all paths through hidden neurons. Purple lines correspond to synaptic weights with standard deviation $J_0/\sqrt{pN}$ (strong coupling), while grey lines correspond to synaptic weights with standard deviation J₀/pN (weak coupling). For weak 1/N coupling (grey), the ratio of standard deviations is $\mathcal{O}(1/\sqrt{N})$. For strong $1/\sqrt{N}$ coupling (purple) the ratio is $\mathcal{O}\left(1\right)$ and grows in strength as the fraction of recorded neurons N_rec/N decreases or the typical synaptic strength J₀ increases. The dashed black lines in the left plot show theoretical estimates accounting only for hidden paths of length-3 connecting recorded neurons (Eq (4). Deviations from the length-3 prediction at small f and large J₀ indicate that contributions from circuit paths involving many hidden neurons are significant in these regimes.

Fig 6. Effective interactions between recorded neurons differ qualitatively from true interactions.

Effective interactions $J_{r,r^{\prime }}^{\text{eff}}\left(t\right)$ (solid purple) versus true coupling filters (dashed black) for N_rec = 3 recorded neurons in a network of N = 1000 total neurons. Inset labels i ← j indicate the interaction is from neuron j to i, for i, j ∈ {1, 2, 3}. The simulated network has an Erdős-Réyni connectivity with sparsity p = 0.2 and normally distributed non-zero weights with zero mean and standard deviation $1/\sqrt{pN}$. Although the network is sparse, the effective interactions are not: non-zero effective interactions develop where no direct connection exists. The effective interactions can differ qualitatively from the true interactions, as evidenced by the biphasic 3 ← 2 effective interaction, whereas the true 3 ← 2 is purely excitatory.

Establishing a theoretical relationship between measured and “true” interactions will thus enable us to study how one can alter the network properties to reshape the effective interactions, and will be of immediate importance not only for interpreting experimental measurements of synaptic interactions, but for elucidating their role in neural coding. Moreover, understanding how to shape effective interactions between neurons may yield new avenues for altering, in a principled way, the computations performed by a network, which could have applications for treating neurological diseases caused in part by pathological synaptic interactions.

Results

Overview

Our goal is to derive a relationship between the effective synaptic interactions between recorded neurons and the true synaptic interactions that would be obtained if the network were fully observed. This makes explicit how the synaptic interactions between neurons are modified by unobserved neurons in the network, and under what conditions these modifications are—or are not—significant. We derive this result first, using a probabilistic model of network activity in which all properties are known. We then build intuition by applying our result to two simple networks: a 3-neuron feedforward-inhibition circuit in which we are able to qualitatively reproduce measurements by Pouille and Scanziani [43], and a 4-neuron circuit that demonstrates how degeneracies in hidden networks are handled within our framework.

To extend our intuition to larger networks, we then study the effective interactions that would be observed in sparse random networks with N cells and strong synaptic weights that scale as $1/\sqrt{N}$ [44–47], as has been recently observed experimentally [48]. We show how unobserved neurons significantly reshape the effective synaptic interactions away from the ground-truth interactions. This is not the case with “classical” synaptic scaling, in which synaptic strengths are inversely proportional to the number of inputs they receive (assumed $\mathcal{O}\left(N\right)$), as we will also show. (The case of classical scaling has also been studied previously using a different approach in [49–52]).

Model

We model the full network of N neurons as a nonlinear Hawkes process [53], often referred to as a “Generalized linear (point process) model” in neuroscience, and broadly used to fit neural activity data [16–23, 54]. Here we use it as a generative model for network activity, as it approximates common spiking models such as leaky integrate and fire systems driven by noisy inputs [55, 56], and is equivalent to current-based leaky integrate-and-fire models with soft-threshold (stochastic) spiking dynamics (see Methods).

To derive an approximate model for an observed subset of the network, we partition the network into recorded neurons (labeled by indices r) and hidden neurons (labeled by indices h). Each recorded neuron has an instantaneous firing rate λ_r(t) such that the probability that the neuron fires within a small time window [t, t + dt] is λ_r(t)dt, when conditioned on the inputs to the neuron. The instantaneous firing rate in our model is

$${\lambda }_r\left(t\right)={\lambda }_0\varphi ({\mu }_r+\sum _{r^{\prime }}{J}_{r,r^{\prime }}*{\dot{n}}_{r^{\prime }}\left(t\right)+\sum _h{J}_{r,h}*{\dot{n}}_h\left(t\right)),$$ (1)

where λ₀ is a characteristic firing rate, ϕ(x) is a non-negative, continuous function, μ_r is a tonic drive that sets the baseline firing rate of the neuron, and $J_{i,j}*{\dot{n}}_j\left(t\right)\equiv {\int }_{-\infty }^{\infty }d{t}^{\prime }{J}_{i,j}(t-t^{\prime }){\dot{n}}_j\left(t^{\prime }\right)$ is the convolution of the synaptic interaction (or “spike filter”) J_i,j(t) with spike train ${\dot{n}}_j\left(t\right)$ from pre-synaptic neuron j to post-synaptic neuron i, for neural indices i and j that may be either recorded or hidden. In this work we take the tonic drive to be constant in time, and focus on the steady-state network activity in response to this drive. We consider interactions of the form $J_{i,j}\left(t\right)\equiv {\mathcal{J}}_{i,j}{g}_j\left(t\right)$, where the temporal waveforms g_j(t) are normalized such that ${\int }_0^{\infty }dt{g}_j\left(t\right)=1$ for all neurons j. Because of this normalization, the weight ${\mathcal{J}}_{i,j}$ carries units of time. We include self-couplings J_i,i(t) not to represent autapses, but to account for intrinsic neural properties such as refractory periods (${\mathcal{J}}_{i,i}<0$) or burstiness (${\mathcal{J}}_{i,i}>0$). The firing rates for the hidden neurons follow the same expression with indices h and r interchanged.

We seek to describe the dynamics of the recorded neurons entirely in terms of their own set of spiking histories, eliminating the dependence on the activity of the hidden neurons. This demands calculating the effective membrane response of the recorded neurons by averaging over the activity of the hidden neurons conditioned on the activity of the recorded neurons. In practice this is intractable to perform exactly [57–60]. Here, we use a mean field approximation to calculate the mean input from the hidden neurons (again, conditioned on the activity of the recorded neurons). The value of deriving such a relationship analytically, as opposed to simply numerically determining the effective interactions, is that the resulting expression will give us insight into how the effective interactions decompose into contributions of different network features, how tuning particular features shapes the effective interactions, and conditions under which we expect hidden units to skew our measurements of connectivity in large partially observed networks.

As shown in detail in the Methods, the instantaneous firing rates of the recorded neurons can then be approximated as

$${\lambda }_r\left(t\right)\approx {\lambda }_0\varphi ({\mu }_r^{\text{eff}}+\sum _{r^{\prime }}{J}_{r,r^{\prime }}^{\text{eff}}*{\dot{n}}_{r^{\prime }}\left(t\right)+{\xi }_r\left(t\right)).$$

The effective baselines ${\mu }_r^{\text{eff}}={\mu }_r+{\sum }_h{\mathcal{J}}_{r,h}{\nu }_h$, are simply modulated by the net tonic input to the neuron, so we do not focus on them here. The ξ_r(t) are effective noise sources arising from fluctuation input from the hidden network. At the level of our mean field approximation these fluctuations vanish; corrections to the mean field approximation are straightforward and yield non-zero noise correlations, but will not impact our calculation of the effective interactions (see the Methods and SI), so as with the effective baselines we will not focus on the effective noise here.

The effective coupling filters are given in the frequency domain by

$${\widehat{J}}_{r,r^{\prime }}^{\text{eff}}\left(\omega \right)={\widehat{J}}_{r,r^{\prime }}\left(\omega \right)+\sum _{h,h^{\prime }}{\widehat{J}}_{r,h}\left(\omega \right){\widehat{\Gamma }}_{h,h^{\prime }}\left(\omega \right){\widehat{J}}_{h^{\prime },r^{\prime }}\left(\omega \right).$$ (2)

These results hold for any pair of recorded neurons r′ and r, and any choice of network parameters for which the mean field steady state of the hidden network exists. Here, the ν_h are the steady-state mean firing rates of the hidden neurons and ${\widehat{\Gamma }}_{h,h^{\prime }}\left(\omega \right)$ is the linear response function of the hidden network to perturbations in the input. That is, Γ_h,h′(t − t′) is the linear response of hidden neuron h at time t due to a perturbation to the input of neuron h′ at time t′, and incorporates the effects of h′ propagating its signal to h through other hidden neurons, as demonstrated graphically in Fig 2. Both ν_h and ${\widehat{\Gamma }}_{h,h^{\prime }}\left(\omega \right)$ are calculated in the absence of the recorded neurons. In deriving these results, we have neglected both fluctuations around the mean input from the hidden neurons, as well as higher order filtering of the recorded neuron spikes. For details on the derivations and justification of approximations, see the Methods and Supporting Information (SI).

The effective coupling filters are what we would—in principle—measure experimentally if we observe only a subset of a network, for example by pairwise recordings shown schematically in Fig 1. For larger sets of recorded neurons, interactions between neurons are typically inferred using statistical methods, an extremely nontrivial task [16–23, 28, 40, 41], and details of the fitting procedure could potentially further skew the inferred interactions away from what would be measured by controlled pairwise recordings. We will put aside these complications here, and assume we have access to an inference procedure that allows us to measure $J_{r,r^{\prime }}^{\text{eff}}\left(t\right)$ without error, so that we may focus on their properties and relationship to the ground-truth coupling filters.

Structure of effective coupling filters

The ground-truth coupling filters ${\widehat{J}}_{r,r^{\prime }}\left(\omega \right)$ (as defined in Eq (1)) are modified by a correction term ${\sum }_{h,h^{\prime }}{\widehat{J}}_{r,h}\left(\omega \right){\widehat{\Gamma }}_{h,h^{\prime }}\left(\omega \right){\widehat{J}}_{h^{\prime },r^{\prime }}\left(\omega \right)$. The linear response function ${\widehat{\Gamma }}_{h,h^{\prime }}\left(\omega \right)$ admits a series representation in terms of paths through the network through which neuron r′ is able to send a signal to neuron r via hidden neurons only.

We may write down a set of “Feynmanesque” graphical rules for explicitly calculating terms in this series [53]. First, we define the input-output gain of a hidden neuron h, ${\gamma }_h\equiv {\lambda }_0{\varphi }^{\prime }({\mu }_h+{\sum }_{h^{\prime }}{\mathcal{J}}_{h,h^{\prime }}{\nu }_{h^{\prime }})$, calculated in the absence of recorded neurons. The contribution of each term can then be written down using the following rules, shown graphically in Fig 2: i) for the edge connecting recorded neuron r′ to a hidden neuron h_i, assign a factor ${\widehat{J}}_{h_i,r^{\prime }}\left(\omega \right)$; ii) for each node corresponding to a hidden neuron h_i, assign a factor ${\gamma }_{h_i}/(1-{\gamma }_{h_i}{\widehat{J}}_{h_i,h_i}\left(\omega \right))$; iii) for each edge connecting hidden neurons h_i ≠ h_j, assign a factor ${\widehat{J}}_{h_j,h_i}\left(\omega \right)$; and iv) for the edge connecting hidden neuron h_j to recorded neuron r, assign a factor ${\widehat{J}}_{r,h_j}\left(\omega \right)$. All factors for each path are multiplied together, and all paths are then summed over.

The graphical expansion is reminiscent of recent works expanding correlation functions of linear models of network spiking in terms of network “motifs” [61–63]. Computationally, this expression is practical for calculating the effective interactions in small networks involving only a few hidden neurons (as in the next section), but is generally unwieldy for large networks. In practice, for moderately large networks the linear response matrix ${\widehat{\Gamma }}_{h,h^{\prime }}\left(\omega \right)$ can be calculated directly by numerical matrix inversion and an inverse Fourier transform back into the time domain. The utility of the path-length series is the intuitive understanding of the origin of contributions to the effective coupling filters and our ability to analytically analyze the strength of contributions from each path. For example, one immediate insight the path decomposition offers is that neurons only develop effective interactions between one another if there is a path by which one neuron can send a signal to the other.

Feedforward inhibition and degeneracy of hidden networks in small circuits

Effective interactions & emergent timescales in a small circuit

To build intuition for our result and compare to a well-known circuit phenomenon, we apply our Eq (2) to a 3-neuron circuit implementing feedforward inhibition, like that studied by Pouille and Scanziani [43]. Feedforward inhibition can sharpen the temporal precision of neural coding by narrowing the “window of opportunity” in which a neuron is likely to fire. For example, in the circuit shown in Fig 3A, excitatory neuron 1 projects to both neurons 2 and 3, and 3 projects to 2. Neuron 1 drives both 2 and 3 to fire more, while neuron 3 is inhibitory and will counteract the drive neuron 2 receives from 1. The window of opportunity can be understood by looking at the effective interaction between neurons 1 and 2, treating neuron 3 as hidden. We use our path expansion (Fig 2) to quickly write down the effective interaction we expect to measure in the frequency domain,

$${\widehat{J}}_{2,1}^{\text{eff}}\left(\omega \right)={\widehat{J}}_{2,1}\left(\omega \right)+\frac{{\widehat{J}}_{2,3}\left(\omega \right){\gamma }_3{\widehat{J}}_{3,1}\left(\omega \right)}{1-{\gamma }_3{\widehat{J}}_{3,3}\left(\omega \right)}.$$ (3)

The corresponding true synaptic interactions and resulting effective interaction are shown in Fig 3B. The effective interaction matches qualitatively the observed changes measured by Pouille and Scanziani [43], and shows a narrow window after neuron 2 receives a spike in which the change in membrane potential is depolarized and neuron 2 is more likely to fire. Following this brief window, the membrane potential is hyperpolarized and the cell is less likely to fire until it receives more excitatory input.

The effective interaction from neuron 1 to 2 in this simple circuit also displays several features that emerge in more complex circuits. Firstly, although the true interactions are either excitatory (positive) or inhibitory (negative), the effective interaction has a mixed character, being initially excitatory (due to excitatory inputs from neuron 1 arriving first through the monosynaptic pathway), but then becoming inhibitory (due to inhibitory input arriving from the disynaptic pathway).

Secondly, emergent timescales develop due to reverberations between hidden neurons with bi-directional connections, represented as loops between neurons in our circuit schematics (e.g., between neurons 3 and 4 in Fig 4). This includes self-history interactions such as refractoriness, schematically represented by loops like the 3 → 3 loop shown in Fig 3, corresponding to the factor $1/(1-{\gamma }_3{\widehat{J}}_{3,3}\left(\omega \right))$). In the particular example shown in Fig 3, in which we use a self-history interaction $J_{33}\left(\tau \right)={\mathcal{J}}_{33}{\beta }_{33}\text{exp}(-{\beta }_{33}\tau )$, a new timescale ${\beta }_{33}^{-1}{(1-{\gamma }_3{\mathcal{J}}_{33})}^{-1}$ develops. Other choices of interactions can generate more complicated emergent timescales and temporal dynamics, including oscillations. For example, in the 4-neuron circuit discussed below (Fig 4), the choice $J_{3,4}\left(\tau \right)=J_{4,3}\left(\tau \right)=-\left|\mathcal{J}\right|{\alpha }^2\tau e^{-\alpha \tau }$ yields effective interactions with new decay and oscillatory timescales equal to $\left(\alpha \right(1-{\lambda }_0{\left|\mathcal{J}\right|\left)\right)}^{-1}$ and $(\alpha {\lambda }_0{\left|\mathcal{J}\right|)}^{-1}$. In the larger networks we consider in the next section, inter-neuron interactions must scale with network size in order to maintain network stability. Because emergent timescales depend on the synaptic strengths of hidden neurons, we typically expect emergent timescales generated by loops between hidden neurons to be negligible in large random networks. However, because the magnitudes of the self-history interaction strengths need not scale with network size, they may generate emergent timescales large enough to be detected.

It is worth noting explicitly that only the interaction from neuron 1 to 2 has been modified by the presence of the hidden neuron 3, for the particular wiring diagram shown in Fig 3. The self-history interactions of both neurons 1 and 2, as well as the interaction from neuron 2 to 1 (zero in this case) are unmodified. The reason the hidden neuron did not modify these interactions is that the only link neuron 3 makes is from 1 to 2. There is no path by which neuron 1 can send a signal back to itself, hence its self-interaction is unmodified, nor is there a path that neuron 2 can send signals to neuron 3 or on to neuron 1, and hence neuron 2’s self-history interaction and its interaction to neuron 1 are unmodified.

Degeneracy of hidden networks giving rise to effective interactions

It is well known that different networks may produce the same observed circuit phenomena [34–39]. To illustrate that our approach may be used to identify degenerate solutions in which more than one network underlies observed effective interactions, we construct a 4-neuron circuit that produces a quantitatively similar effective interaction between the recorded neurons 1 and 2, shown in Fig 4. Specifically, in this circuit we have removed neuron 3’s self-history interaction and introduced a second inhibitory hidden network that receives excitatory input from neuron 1 and provides inhibitory input to neuron 3. By tuning the interaction strengths we are able to produce the desired effective interaction. This demonstrates that intrinsic neural properties such as refractoriness can trade off against inputs from other hidden neurons, making it difficult to distinguish the two cases from one another (or from a potential infinity of other circuits that could have produced this interaction; for example, a qualitatively similar interaction is produced in the N = 1000 network in which only three neurons are recorded, shown below in Fig 6). Statistical inference methods may favor one of the possible underlying choices of complete network consistent with a measured set of effective interactions, suggesting there may be some sense of a “best” solution. However, the particular “best” network will depend on many factors, including the amount and fidelity of data recorded, regularization choices, and how well the fitted model generalizes to new data (i.e., how “close” the fitted model is to the generative model). Potentially, if these conditions were met, with enough data the slight quantitative differences between the effective interactions produced by different hidden networks (including higher order effective interactions, which we assume to be negligible here; see SI), could help distinguish different hidden networks. However, the amount of data required to perform this discrimination and validate the result may be impractically large [36, 64–66]. It is thus worth studying the structure of the observed effective interactions directly in search of possible signatures that elucidate the statistical properties of the complete network.

Strongly coupled large networks

Constructing networks that produce particular effective interactions is tractable for small circuits, but much more difficult for larger circuits composed of many circuit motifs. Not only can combinations of different circuit motifs interact in unexpected ways, one must also take care to ensure the resulting network is both active and stable—i.e., that firing will neither die out nor skyrocket to the maximum rate. Stability in networks is often implemented by either building networks with classical (or “weak”) synapses whose strength scales inversely with the number of inputs they receive, assumed here to be proportional to network size, and hence ${\mathcal{J}}_{i,j}\sim 1/N$, or by building balanced networks in which excitatory and inhibitory synaptic strengths balance out, on average, and scale as ${\mathcal{J}}_{i,j}\sim 1/\sqrt{N}$ [44, 48] (but note the distinction that we use a “soft threshold” firing model with nonlinearity that is fixed as N varies, whereas previous work has typically used hard threshold models). In both cases the synapses tend to be small in value in large networks, but are compensated for by large numbers of incoming connections. In the case of 1/N scaling, neurons are driven primarily by the mean of their inputs, while in “strong” balanced $1/\sqrt{N}$ networks neurons are driven primarily by fluctuations in their inputs.

Our goal is to understand how the interplay between the presence of hidden neurons and different synaptic scaling or network architectures shapes effective interactions. Previous work has studied the hidden-neuron problem in the weak coupling limit [49–52] using a different approach to relate inferred synaptic parameters to true parameters; here we use our approach to study the $1/\sqrt{N}$ strong coupling limit, theoretically predicted to be an important feature that supports computations in networks in a balanced regime [44–47]. Moreover, experiments in cultured neural tissue have been found to be more consistent with the $1/\sqrt{N}$ scaling than 1/N [48], indicating that it may have intrinsic physiological importance.

We analytically determine how significantly effective interaction strengths are skewed away from the true interaction strengths as a function of both the number of observed neurons and typical synaptic strength. We consider several simple networks ubiquitous in neural modeling: first, an Erdős-Réyni (ER) network with “mixed synapses” (i.e., a neuron may have both positive and negative synaptic weights), a balanced ER network with Dale’s law imposed (a neuron’s synapses are all the same sign), and a Watts-Strogatz (WS) small world network with mixed synapses. Each network has N neurons and connection sparsity p (only 100p% of connections are non-zero). Connections in ER networks are chosen randomly and independently, while connections in the WS network are determined by randomly rewiring a fraction β of the connections in a (pN)^th-nearest-neighbor ring network. such that the overall network has a backbone of local synaptic connections with a web of sparse long-range connections. In each network N_rec neurons are recorded randomly.

For simplicity we take the baselines of all neurons to be equal, μ_i = μ₀ (such that in the absence of synaptic input the probability that a neuron fires in a short time window Δt is λ₀Δt exp(μ₀)). We choose the rate nonlinearity to be exponential, ϕ(x) = e^x; this is the “canonical” choice of nonlinearity often used when fitting this model to data [16–18, 20, 67]. We will further assume exp(μ₀) ≪ 1, so that we may use this as a small control parameter. For i ≠ j, the non-zero synaptic weights between neurons ${\mathcal{J}}_{i,j}$ are independently drawn from a normal distribution with zero mean and standard deviation J₀/(pN)^a, where J₀ controls the overall strength of the weights and a = 1 or 1/2, corresponding to “weak” and “strong” coupling. For simplicity, we do not consider intrinsic self-coupling effects in this part of the analysis, i.e., we take ${\mathcal{J}}_{i,i}=0$ for all neurons i. For the Dale’s law network, the overall distribution of synaptic weights follows the same normal distribution as the mixed synapse networks, but the signs of the weights correspond to whether the pre-synaptic neuron is excitatory or inhibitory. Neurons are randomly chosen to be excitatory and inhibitory, the average number of each type being equal so that the network is balanced. Numerical values of all parameters are given in Table 1.

Table 1. Network connectivity parameter values for Figs 5–8 and S1–S3.

See individual captions for other figures.

Number of neurons N	1000
Number of hidden neurons Nhid	{1, 90, 190, 290, 390, 490, 590, 690, 790, 890, 990}
Number of recorded neurons Nrec	N-Nhid
Baselines μi	-1.0, ∀i
Sparsity p	0.2
Coupling weights ${\mathcal{J}}_{ij}(i\ne j)$	$\mathcal{N}(0,J_0^2/{\left(pN\right)}^{2a})$
Self-coupling weights ${\mathcal{J}}_{ii}$	0
Coupling regime a	1 (weak coupling) or 1/2 (strong coupling)
Rewiring probability β (Watts-Strogatz only)	0.3
Characteristic synaptic weight J0	{0.25, 0.5, 0.75, 1.0}
Firing frequency λ0	1.0

We seek to assess how the presence of hidden neurons can shape measured network interactions. We first focus on the typical strength of the effective interactions as a function of both the fraction of neurons recorded, f = N_rec/N, and the strength of the synaptic weights J₀. We quantify the strength of the effective interactions by defining the effective synaptic weights ${\mathcal{J}}_{r,r^{\prime }}^{\text{eff}}\equiv {\int }_0^{\infty }d\tau J_{r,r^{\prime }}^{\text{eff}}\left(\tau \right)={\widehat{J}}_{r,r^{\prime }}^{\text{eff}}(\omega =0)$; c.f. ${\mathcal{J}}_{r,r^{\prime }}={\int }_0^{\infty }d\tau J_{r,r^{\prime }}\left(\tau \right)$ for the true synaptic weights. We then study the sample statistics of the difference, ${\mathcal{J}}_{r,r^{\prime }}^{\text{eff}}-{\mathcal{J}}_{r,r^{\prime }}$, averaged across both subsets of recorded neurons and network instantiations, to estimate the typical contribution of hidden neurons to the measured interactions. The mean of the synaptic weights is near zero (because the weights are normally distributed with zero mean in the mixed synapse networks and due to balance of excitatory and inhibitory neurons in the Dale’s law network), so we focus on the root-mean-square of ${\mathcal{J}}_{r,r^{\prime }}^{\text{eff}}-{\mathcal{J}}_{r,r^{\prime }}$. This measure is a conservative estimate of changes in strength, as $J_{r,r^{\prime }}^{\text{eff}}\left(\tau \right)$ may have both positive and negative components that partially cancel when integrated over time, unlike J_r,r′(τ). An alternative measure we could have chosen that avoids potential cancellations is ${\int }_0^{\infty }d\tau |J_{r,r^{\prime }}^{\text{eff}}\left(\tau \right)-J_{r,r^{\prime }}\left(\tau \right)|$, i.e., the integrated absolute difference between effective and true interactions. However, this will depend on our specific choices of waveform g(τ) in our definition $J_{i,j}\left(\tau \right)={\mathcal{J}}_{i,j}g\left(\tau \right)$, whereas ${\mathcal{J}}_{r,r^{\prime }}^{\text{eff}}-{\mathcal{J}}_{r,r^{\prime }}$ does not due to our normalization ${\int }_0^{\infty }d\tau g\left(\tau \right)=1$. As |∫ dτ f(τ)| ≤ ∫ dτ |f(τ)|, for any f(τ), we can consider our choice of ${\mathcal{J}}_{r,r^{\prime }}^{\text{eff}}-{\mathcal{J}}_{r,r^{\prime }}$ as a lower bound on the strength that would be quantified by ${\int }_0^{\infty }d\tau |J_{r,r^{\prime }}^{\text{eff}}\left(\tau \right)-J_{r,r^{\prime }}\left(\tau \right)|$.

Numerical evaluations of the population statistics for all three network types are shown as solid curves in Fig (5), for both strong coupling and weak coupling. All three networks yield qualitatively similar results. The vertical axes measure the root-mean-square deviations between the statistically expected true synaptic ${\mathcal{J}}_{r,r^{\prime }}$ and the corresponding effective synaptic weight ${\mathcal{J}}_{r,r^{\prime }}^{\text{eff}}$, normalized by the true root mean square of ${\mathcal{J}}_{r,r^{\prime }}$. Thus, a ratio of 0.5 corresponds to a 50% root-mean-square difference in effective versus true synaptic strength. We measure these ratios as a function of both the fraction of neurons recorded (horizontal axis) and the parameter J₀ (labeled curves).

There are two striking effects. First, deviations are nearly negligible ($\mathcal{O}(1/\sqrt{pN})$) for 1/N scaling of connections (gray traces in Fig 5). Thus, for large networks with synapses that scale with the system size, vast numbers of hidden neurons combine to have negligible effect on effective couplings. This is in marked contrast to the case when coupling is strong ($1/\sqrt{N}$ scaling), when hidden neurons have a pronounced $\mathcal{O}\left(1\right)$ impact (purple traces in Fig 5). This is particularly the case when f ≪ 1—the typical experimental case in which the hidden neurons outnumber observed ones by orders of magnitude—or when J₀ ≲ 1.0, when typical deviations become half the magnitude of the true couplings themselves (upper blue line). For J₀ ≳ 1.0, the network activity is unstable for an exponential nonlinearity.

To gain analytical insight into these numerical results, we calculate the standard deviation $\sigma [{\mathcal{J}}_{r,r^{\prime }}^{\text{eff}}-{\mathcal{J}}_{r,r^{\prime }}]$, normalized by $\sigma \left[{\mathcal{J}}_{r,r^{\prime }}\right]$, for contributions from paths up to length-3, focusing on the case of the ER network with mixed synapses (the Dale’s law and WS networks are more complicated, as the moments of the synaptic weights depend on the identity of the neurons). For strong $1/\sqrt{N}$ coupling we find

$$\begin{array}{cc}\hfill \frac{\sigma [{\mathcal{J}}_{r,r^{\prime }}^{\text{eff}}-{\mathcal{J}}_{r,r^{\prime }}]}{\sigma \left[{\mathcal{J}}_{r,r^{\prime }}\right]}& \approx {\lambda }_0{J}_0{e}^{{\mu }_0}\sqrt{1-f}\hfill \\ & \times (1+\frac{3}{2}{\left({\lambda }_0{J}_0{e}^{{\mu }_0}\right)}^2(1-f)),\hfill \end{array}$$ (4)

corresponding to the black dashed curves in Fig 5 left. Eq (4) is a truncation of a series in powers of ${\lambda }_0{J}_0{e}^{{\mu }_0}\sqrt{1-f}$, where f = N_rec/N is the fraction of recorded neurons. The most important feature of this series is the fact that it only depends on the fraction of recorded neurons f, not the absolute number, N. Contributions from long paths remain finite, even as N → ∞. In contrast, the corresponding expression for $\sigma [{\mathcal{J}}_{r,r^{\prime }}^{\text{eff}}-{\mathcal{J}}_{r,r^{\prime }}]/\sigma \left[{\mathcal{J}}_{r,r^{\prime }}\right]$ in the case of weak 1/N coupling is a series in powers of ${\lambda }_0{J}_0{e}^{{\mu }_0}\sqrt{(1-f)/\left(pN\right)}$, so that contributions from long paths are negligible in large networks N ≫ 1. (See [67] for derivation and results for N = 100.) Deviations of Eq (4) from the numerical solutions in Fig 5 indicate that contributions from truncated terms are not negligible when f ≪ 1. As these terms correspond to paths of length-4 or more, this shows that long chains through the network contribute significantly to shaping effective interactions.

The above analysis demonstrates that the strength of the effective interactions can deviate from that of the true direct interactions by as much as 50%. However, changes in strength do not give us the full picture—we must also investigate how the temporal dynamics of the effective interactions change. To illustrate how hidden units can skew temporal dynamics, in Fig 6 we plot the effective vs. true interactions between N_rec = 3 neurons in an N = 1000 neuron network. Because the three network types considered in Fig 5 yield qualitatively similar results, we focus on the Erdős-Réyni network with mixed synapses.

Four of the true interactions between neurons shown in Fig 6 are non-zero ($J_{1,2}^{\text{eff}}\left(t\right)$, $J_{3,2}^{\text{eff}}\left(t\right)$, $J_{3,1}^{\text{eff}}\left(t\right)$, and $J_{2,3}^{\text{eff}}\left(t\right)$). Of these, three exhibit only slight differences between the true and effective interactions: $J_{1,2}^{\text{eff}}\left(t\right)$ and $J_{3,1}^{\text{eff}}\left(t\right)$ have slightly longer decay timescales than their true counterparts, while $J_{2,3}^{\text{eff}}\left(t\right)$ has a slightly shorter timescale, indicating the contribution of the hidden network to these interactions was either small or cancelled out. However, the interaction $J_{3,2}^{\text{eff}}\left(t\right)$ differs significantly from the true interaction, becoming initially excitatory but switching to inhibitory after a short time, as in our earlier example case of feedforward inhibition. This indicates that neuron 2 must drive a cascade of neurons that ultimately provide inhibitory input to neuron 3.

Contrasting the true and effective interactions shown in Fig 6 highlights many of the ways in which hidden neurons skew the temporal properties of measured interactions. An immediately obvious difference is that although the true synaptic connections in the network are sparse, the effective interactions are not. This is a generic feature of the effective interaction matrix, as in order for an effective interaction from a neuron r′ to r to be identically zero there cannot be any paths through the network by which r′ can send a signal to r.¹ In a random network the probability that there are no paths connecting two nodes tends to zero as the network size N grows large. Note that this includes paths by which each neuron can send a signal back to itself, hence the neurons developed effective self-interactions, even though the true self-interactions are zero in these particular simulations.

Discussion

We have derived a quantitative relationship between “ground-truth” synaptic interactions and the effective interactions (interpreted here as post-synaptic membrane responses) that unobserved neurons generate between subsets of observed neurons. This relationship, Eq (2) and Fig 2, provides a foundation for studying how different network architectures and neural properties shape the effective interactions between subsets of observed neurons. Our approach can be also be used to study higher order effective interactions between 3 or more neurons, and can be systematically extended to account for corrections to our mean-field approximations and investigate effective noise generated by hidden neurons (using field theoretic techniques from [53], see SI), as well as time-dependent external drives or steady-states.

Here, as first explorations, we focused on the effective interactions corresponding to linear membrane responses. We first demonstrated that our approach applied to small feedforward inhibitory circuits yields effective interactions that capture the role of inhibition in shortening the time window for spiking, and are qualitatively similar to experimentally observed measurements [43]. Moreover, we used this example to demonstrate explicitly that different hidden networks can give rise to the same effective interactions between neurons. We then showed that the influence of hidden neurons can remain significant even in large networks in which the typical synaptic strengths scale with network size. In particular, when the synaptic weights scale as $1/\sqrt{N}$, the relative influence of hidden neurons depends only on the fraction of neurons recorded. Together with theoretical and experimental evidence for this scaling in cortical slices [44–48], this suggests that neural interactions inferred from cortical activity data may differ markedly from the true interactions and connectivity.

Dealing with degeneracy

The issue of degeneracy in complex biological systems and networks has been discussed at length in the literature, in the context of both inherent degeneracies—multiple different network architectures can produce the same qualitative behaviors [34, 37–39], as well as degeneracies in our model descriptions—many models may reproduce experimental observations, demanding sometimes arbitrary criteria for selecting one model over another. All have implications for how successfully one can infer unobserved network properties. One kind of model degeneracy, “sloppiness” [35, 65], describes models in which the behavior of the model is sensitive to changes in only a relatively small number of directions in parameter space. Many models of biological systems have been shown to be sloppy [35]; this could account for experimentally observed networks that are quite different in composition but produce remarkably similar behaviors. Sloppiness suggests that rather than trying to infer all properties of a hidden network, there may be certain parameter combinations that are much more important to the overall network operation, and could potentially be inferred from subsampled observations.

Another perspective on model degeneracy comes from the concepts of “universality” that occur in random matrix theory [68, 69] and Renormalization Group methods of statistical physics [64]. Many bulk properties of matrices (e.g., the distribution of eigenvalues) whose entrees are combinations random variables, such as our ${\mathcal{J}}_{r,r^{\prime }}^{\text{eff}}$, are universal in that they depend on only a few key details of the distribution that individual elements are drawn from [70]. Similarly, one of the central results of the Renormalization Group shows that models with drastically disparate features may yield the same coarse-grained model structure when many degrees of freedom are averaged out, as in our case of approximately averaging out hidden neurons. Different distributions (in the case of random matrix theory) or different models (in the case of the Renormalization group) that yield the same bulk properties or coarse-grained models are said to be in the same “universality class.” Measuring particular quantities under a range of experimental conditions (e.g., different stimuli) may be able to reveal which universality class an experimental system belongs to and eliminate models belonging to other universality classes as candidate generating models of the data, but these measurements cannot distinguish between models within a universality class.

Purely feedforward networks and recurrent networks are simple examples of broad universality classes in this context. In any randomly sampled feedforward network, only the feedforward interactions are modified or generated; no lateral or feedback connections develop because there is no path through hidden neurons that a recorded neuron can send signals to recorded neurons in the same or previous layers. Thus, the feedforward structure—a topological property of the network—is preserved. However, adding even a single feedback connection can destroy this topological structure if it joins two neurons connected by a feedforward path—i.e., such a link creates a cycle within the network, and it is no longer feedforward. If this network is heavily subsampled (f ≪ 1) the resulting effective interactions $J_{r,r^{\prime }}^{\text{eff}}\left(t\right)$ can even be fully recurrent. The majority of the effective interactions may be very weak, but nonetheless from a topological perspective the network has been fundamentally altered. Accordingly, any interactions $J_{r,r\prime }^{\text{eff}}\left(t\right)$ that represent a purely feedforward network could not have come from a network with recurrent interactions. In practice, we expect few, if any, cortical networks to be purely feedforward, so most networks will be recurrent if we consider only the network connections and not the connection strengths. Thus, a more interesting question is how the statistics and dynamics of synaptic weights further partition topologically-defined universality classes; for example, whether the distribution of synaptic weights can split the sets of $J_{r,r^{\prime }}^{\text{eff}}\left(t\right)$ that arise from recurrent networks and predominantly feedforward networks with sparse feedback and lateral interactions into different universality classes. A thorough investigation of such phenomena will be the focus of future work.

Inference of hidden network features

Despite the many possible confounds network degeneracy produces, much of the work on inference of hidden network properties has focused on inferring the individual interactions between neurons, with varying degrees of success. Both Dunn and Roudi [40] as well as Tyrcha and Hertz [41] studied inference of hidden activity in kinetic Ising models, sometimes used as simple minimal models of neuronal network activity. They found that the synaptic weights between pairs of observed neurons and observed-hidden pairs could be recovered to within reasonably small mean-squared-error when the number of hidden neurons was less than the number of observed neurons. However, both methods also found it difficult to infer connections between pairs of hidden neurons, resorting to setting such connections to zero in order to stabilize their algorithms. Tyrcha and Hertz also note their method recovers only an equivalence class of connections due to degeneracy in the possible assignment of signs of the synaptic weights and hidden neuron labels. This suggests inferring hidden network structure will be nearly impossible in the realistic limit N_rec ≪ N_hid.

A series of papers by Bravi and Sollich perform theoretical analyses of hidden dynamics inference in chemical reaction networks, modeled by a system of Langevin equations [57–60]. Although the applications the authors have in mind are signaling pathways such as epidermal growth factor reaction networks, one could imagine re-interpreting or adapting these equations to describe rate models. The authors develop a variety of approaches, including Plefka expansions [57–59] and variational Gaussian approximations [60], to study how observations constrain the inferred hidden dynamics, assuming particlar properties of the network structure. Ref. [60] in particular takes an approach most similar to ours, deriving an effective system of Langevin equations for the subsampled dynamics of the chemical reaction network. The effective system of equations contains a memory kernel that plays a role analogous to the correction to the interactions between neurons in our work (second term in our Eq (2). However, the structure of the memory kernel in [60] has a rather different form, being exponentially dependent on the integral of the hidden-hidden interactions, in contrast to our Γ_h,h′(t − t′), which depends on the inverse of δ_h,h′δ(t − t′) − γ_hJ_h,h′(t − t′) (see Methods). Though Bravi and Sollich do not expand their memory kernel in a series as we do, it would admit a similar series and interpretation in terms of paths through the hidden network, as in our Fig 2A. However, due to the exponential dependence on the hidden-hidden interactions, long paths of length ℓ through hidden networks are suppressed by factors ℓ!, suggesting the hidden network may have less influence in such networks compared to the network dynamics we study here.

Closest to our choice of model, Pillow and Latham [28] and Soudry et al[25] both use modifications of nonlinear Hawkes models to fit neural data with unobserved neurons. Pillow and Latham outline a statistical approach for inferring not just interactions with and between hidden neurons, but also the spike trains of hidden neurons, testing the method on a network of two neurons (one hidden). To properly infer the spike train of the hidden neurons, the model must allow for acausal synaptic interactions. This is acceptable if the goal is inferring hidden spike trains: for example, if the hidden neuron were to make a strong excitatory synapse onto the observed neuron, then a spike from the observed neuron increases the probability that the hidden neuron fired a spike in the recent past. An acausal synaptic interaction captures this effect, but is of course an unphysical feature in a mechanistic model, precluding physiological interpretation of such an interaction.

Soudry et al are concerned with the fact that common input from hidden neurons will skew estimates of network connectivity. To get around this issue, they present a different take on the hidden unit problem: rather than attempt to infer connectivity in a fixed subsample of a network, they propose a shotgun sampling method, in which a sequence of overlapping random subsets of the network are sampled over a long experiment. Under this procedure, a large fraction of the network can be sampled, just not contiguously in time, and reconstruction of the entire network could in principle be accomplished. Soudry et al show this strategy works in their simulated networks (even when the generative model is a hard-threshold leaky-integrate-and-fire rather than the nonlinear Hawkes model, which can be interpreted as a soft-threshold leaky-integrate-and-fire model; see SI). However, sampling the entire network may only be feasible in vitro; sampling of neurons in vivo, such as in wide-field calcium imaging studies, will still necessarily miss neurons not in the field of view or too deep in the tissue; in such cases our work provides the means to properly interpret the inferred effective interactions obtained with such a method.

Although a thorough treatment of statistical inference of hidden network properties is beyond the scope of our present work, we may make some general remarks on future work in these directions. The nonlinear Hawkes model we use here is commonly used to fit neural population activity data, and one could infer the effective baselines ${\mu }_r^{\text{eff}}$, interactions $J_{r,r\prime }^{\text{eff}}\left(t\right)$, and noise ξ_r(t) using existing techniques. In particular, Vidne et al. [22] explicitly fit the noise, which is likely important for proper inference, as otherwise effects of the noise could be artificially inherited by the effective interactions. Once such estimates are obtained, one could then in principle infer certain hidden network properties by combining a statistical model for these properties with the relationships between effective and true interactions derived in this work, such as Eq (2). (Detailed physiological measurements of ground-truth synaptic interactions in small volumes of neural tissue can be used to refine estimates). As we have stressed throughout this paper, inferring the exact connections between hidden neurons may be impossible due to a large number of degenerate solutions consistent with observations. However, one may be able to infer bulk properties of the network, such as the parameters governing the distribution of hidden-network connections, or even more exotic properties such as the eigenvalue distribution of the hidden network connection weight matrix. We leave these ideas as interesting directions for future work.

Hidden neurons and dimensionality reduction

Given the challenges that hidden network inference poses, one might wonder if there are network properties that can be reliably measured even with subsampled neural activity. Collective, low-dimensional dynamics have emerged as a possible candidate: recent work has investigated the effect that subsampled measurements have on estimating collective low-dimensional dynamics of trial-averaged network activity (using, e.g., principal components analysis). During a task, the effective dimensionality of a network’s dynamics is constrained [71, 72], opening the possibility that the subsampled population may be sufficient to accurately represent these task-constrained low-dimensional dynamics. Indeed, under certain assumptions—in particular that the collective dynamical modes are approximately random superpositions of neural activity and that sampled neurons are statistically representative of the hidden population—Gao et al. [72] calculate a conservative upper bound on the number of sampled neurons necessary to reconstruct the collective dynamics, finding it is often less than the effective dimensionality of the network.

The assumption that the collective dynamics are random superpositions of neural activity is crucial, because it means that each neuron’s trial-averaged dynamics are in turn a superposition of the collective modes. Hence, every neuron’s activity contains some information about the collective modes, and if only a few of these modes are important, then they can be extracted from any sufficiently large subset of neurons.

While modes of collective activity alone may be sufficient for answering certain questions, such as decoding task parameters or elucidating circuit function, explaining the structure of these modes—and in particular how the dynamical patterns that emerge under different task conditions or sensory environments are related—will ultimately require an understanding of the distribution of possible underlying network properties, which remain difficult to estimate from subsampled populations. We may be able to establish such structure-function relationships using our theory of effective interactions presented in this work: if we can relate the collective dynamics extracted from subsampled neurons to the properties of the effective interactions $J_{rr\prime }^{\text{eff}}\left(t\right)$, then we can link them to the true interactions through our Eq (2). With an understanding of how network properties shape such collective dynamics, we can begin to understand what network manipulations achieve desired patterns of activity, and therefore circuit function.

Implications beyond experimental limitations

The fact that many different hidden networks may yield the same set of effective interactions or low-dimensional dynamics suggests that the effective interactions themselves may yield direct insight into a circuit’s functions. For instance, many circuits consist of principal neurons that transmit the results of circuit computation to downstream circuitry, but often do not make direct connections with one another, instead interacting through (predominantly inhibitory) intermediaries called interneurons. From the point of view of a downstream circuit, the principal neurons are “recorded” and the interneurons are “hidden.” A potential reason for this general arrangement is that direct synaptic interactions alone are insufficient to produce the membrane responses required to perform the circuit’s computations, and the network of interneurons reshapes the membrane responses of projection neurons into effective interactions that can perform the desired computations—it may thus be that the effective interactions should be of primary interest, not necessarily the (possibly degenerate choices of) physiological synaptic interactions. For example, in the feedforward inhibitory circuits of Figs 3 and 4, the roles of the hidden inhibitory neurons may simply be to act as interneurons that reshape the interaction between the excitatory projection neurons 1 and 2, and the choice of which particular circuit motif is implemented in a real network is determined by other physiological constraints, not only computational requirements.

One of the greatest achievements in systems neuroscience would be the ability to perform targeted modifications to a large neural circuit and selectively alter its suite of computations. This would have powerful applications for both studying a circuit’s native computations, but also repurposing circuits or repairing damaged circuitry (due to, e.g., disease). If the computational roles of circuits are indeed most sensitive to the effective interactions between principal neurons, this suggests we can exploit potential degeneracies in the interneuron architecture and intrinsic properties to find some circuit that achieves a desired computation, even if it is not a physiologically natural circuit. Our main result relating effective and true interactions, Eq (2), provides a foundation for future work investigating how to identify sets of circuits that perform a desired set of computations. We have shown in this work that it can be done for small circuits (Figs 3 and 4), and that the effective interactions in large random networks can be significantly skewed away from the true interactions when synaptic weights scale as $1/\sqrt{N}$, as observed in experiments [48]. This holds promise for identifying principled approaches to tuning or controlling neural interactions, such as by using neuromodulators to adjust interneuron properties or inserting artificial or synthetic circuit implants into neural tissue to act as “hidden” neurons. If successful, this could contribute to the long term goal of using such interventions to aid in reshaping the effective synaptic interactions between diseased neurons, and thereby restore healthy circuit behaviors.

Methods

Model definition and details

The firing rate of a neuron i in the full network is given by

$${\lambda }_i\left(t\right)={\lambda }_0\varphi ({\mu }_i+{\mu }_i^{\text{ext}}\left(t\right)+\sum _j{\int }_{-\infty }^{\infty }d{t}^{\prime }{J}_{ij}(t-t^{\prime }){\dot{n}}_j\left(t^{\prime }\right)),$$ (5)

where λ₀ is a characteristic rate, ϕ(x) ≥ 0 is a nonlinear function, μ_i (potentially a function of some external stimulus θ) is a time-independent tonic drive that sets the baseline firing rate of the neuron in the absence of input from other neurons, ${\mu }_i^{\text{ext}}\left(t\right)$ is an external input current, and J_ij(t − t′) is a coupling filter that filters spikes ${\dot{n}}_j\left(t^{\prime }\right)$ fired by presynaptic neuron j at time t′, incident on post-synaptic neuron i. We will take ${\mu }_i^{\text{ext}}\left(t\right)=0$ for simplicity in this work, focusing on the activity of the network due to the tonic drives μ_i (which could be still be interpreted as external tonic inputs, so the activity of the network need not be interpreted as spontaneous activity).

While we need not attach a mechanistic interpretation to these filters, a convenient interpretation is that the nonlinear Hawkes model approximates the stochastic dynamics of a leaky integrate-and-fire network model driven by noisy inputs [55, 56]. In fact, the nonlinear Hawkes model is equivalent to a current-based integrate-and-fire model in which the deterministic spiking rule (a spike fires when a neuron’s membrane potential reaches a threshold value V_th) is replaced by a stochastic spiking rule (the higher a neuron’s membrane potential, the higher the probability a neuron will fire a spike). (It can also be mapped directly to a conductance-based in special cases [73]). For completeness, we present the mapping from a leaky integrate-and-fire model with stochastic spiking to Eq (5) in the Supporting Information (SI).

Derivation of effective baselines and coupling filters

To study how hidden neurons affect the inferred properties of recorded neurons, we partition the network into “recorded” neurons, labeled by indices r (with sub- or superscripts to differentiate different recorded neurons, e.g., r and r′ or r₁ and r₂) and “hidden” neurons labeled by indices h (with sub- or superscripts). The rates of these two groups are thus

$${\lambda }_r\left(t\right)={\lambda }_0\varphi ({\mu }_r+\sum _{r^{\prime }}{J}_{r,r^{\prime }}*{\dot{n}}_{r^{\prime }}+\sum _h{J}_{r,h}*{\dot{n}}_h),$$

$${\lambda }_h\left(t\right)={\lambda }_0\varphi ({\mu }_h+\sum _r{J}_{h,r}*{\dot{n}}_r+\sum _{h^{\prime }}{J}_{h,h^{\prime }}*{\dot{n}}_{h^{\prime }}).$$

To simplify notation, we write $J_{i,j}*{\dot{n}}_j={\int }_{-\infty }^{\infty }d{t}^{\prime }{J}_{i,j}(t-t^{\prime }){\dot{n}}_j\left(t^{\prime }\right)$. If we seek to describe the firing of the recorded neurons only in terms of their own spiking history, input from hidden neurons effectively acts like noise with some mean amount of input. We thus begin by splitting the hidden input to the recorded neurons up into two terms, the mean plus fluctuations around the mean:

$$\sum _h{J}_{r,h}*{\dot{n}}_h\left(t\right)=\sum _h{J}_{r,h}*\mathbb{E}\left[{\dot{n}}_h\left(t\right)|\right\{{\dot{n}}_r\left\}\right]+{\xi }_r\left(t\right),$$

where $\mathbb{E}\left[{\dot{n}}_h\left(t\right)|\right\{{\dot{n}}_r\left\}\right]$ denotes the mean activity of the hidden neurons conditioned on the activity of the recorded units, and ξ_r(t) are the fluctuations, i.e., ${\xi }_r\left(t\right)\equiv {\sum }_h{J}_{r,h}*({\dot{n}}_h-\mathbb{E}[{\dot{n}}_h\left(t\right)|\left\{{\dot{n}}_r\right\}\left]\right)$. Note that ξ_r(t) is also conditional on the activity of the recorded units.

By construction, the mean of the fluctuations is identically zero, while the cross-correlations can be expressed as

$$\mathbb{E}\left[{\xi }_r\left(t\right){\xi }_{r^{\prime }}\left(t^{\prime }\right)\right]={\int }_{-\infty }^{\infty }d{t}_{1}d{t}_2\sum _{h_1,h_2}{J}_{r,h_1}(t-t_1)J_{r^{\prime },h_2}(t^{\prime }-t_2)C_{h_1,h_2}(t_1,t_2),$$

where $C_{h_1,h_2}(t_1,t_2)$ is the cross-covariance between hidden neurons h₁ and h₂ (conditioned on the spiking of recorded neurons). If the autocorrelation of the fluctuations (r = r′) is small compared to the mean input to the recorded neurons (${\sum }_h{J}_{r,h}*\mathbb{E}\left[{\dot{n}}_h\left(t\right)|\right\{{\dot{n}}_r\left\}\right]$), or if J_r,h is small, then we may neglect these fluctuations and focus only on the effects that the mean input has on the recorded subnetwork. At the level of the mean field theory approximation we make in this work, the spike-train correlations are zero. One can calculate corrections to mean field theory (see SI) to estimate the size of this noise. Even when this noise is not strictly negligible, it can simply be treated as a separate input to the recorded neurons, as shown in the main text, and hence will not alter the form of the effective couplings between neurons. Averaging out the effective noise, however, would generate new interactions between neurons; we leave investigation of this issue for future work.

In order to calculate how hidden input shapes the activity of recorded neurons, we need to calculate the mean $\mathbb{E}\left[{\dot{n}}_h|\right\{{\dot{n}}_r\left\}\right]$. This mean input is difficult to calculate in general, especially when conditioned on the activity of the recorded neurons. In principle, the mean can be calculated as

$$\mathbb{E}\left[{\dot{n}}_h|\right\{{\dot{n}}_r\left\}\right]=\mathbb{E}\left[{\lambda }_0\varphi \right({\mu }_h+\sum _r{J}_{h,r}*{\dot{n}}_r+\sum _{h^{\prime }}{J}_{h,h^{\prime }}*{\dot{n}}_{h^{\prime }}\left)\right|\left\{{\dot{n}}_r\right\}].$$

This is not a tractable calculation. Taylor series expanding the nonlinearity ϕ(x) reveals that the mean will depend on all higher cumulants of the hidden unit spike trains, which cannot in general be calculated explicitly. Instead, we again appeal to the fact that in a large, sufficiently connected network, we expect fluctuations to be small, as long as the network is not near a critical point. In this case, we may make a mean field approximation, which amounts to solving the self-consistent equation

$$\mathbb{E}\left[{\dot{n}}_h|\right\{{\dot{n}}_r\left\}\right]={\lambda }_0\varphi ({\mu }_h+\sum _r{J}_{h,r}*{\dot{n}}_r+\sum _{h^{\prime }}{J}_{h,h^{\prime }}*\mathbb{E}[{\dot{n}}_{h^{\prime }}|\left\{{\dot{n}}_r\right\}\left]\right).$$ (6)

In general, this equation must be solved numerically. Unfortunately, the conditional dependence on the activity of the recorded neurons presents a problem, as in principle we must solve this equation for all possible patterns of recorded unit activity. Instead, we note that the mean hidden neuron firing rate is a functional of the filtered recorded input $I_h\left(t\right)\equiv {\sum }_r{J}_{h,r}*{\dot{n}}_r$, so we can expand it as a functional Taylor series around some reference filtered activity $I_h^0\left(t\right)={\sum }_r{J}_{h,r}*{\dot{n}}_r^0$,

$$\begin{array}{cc}\hfill \mathbb{E}\left[{\dot{n}}_h\left(t\right)|\right\{I_h\left(t\right)\left\}\right]& =\mathbb{E}\left[{\dot{n}}_h\left(t\right)|\right\{I_h^0\left(t\right)\left\}\right]\hfill \\ & +\int d{t}_1\sum _{h_1}\frac{\delta \mathbb{E}\left[{\dot{n}}_h\left(t\right)|\right\{I_h^0\left(t\right)\left\}\right]}{\delta I_{h_1}\left(t_1\right)}(I_{h_1}\left(t_1\right)-I_{h_1}^0\left(t_1\right))\hfill \\ & +\frac{1}{2}\int d{t}_{1}d{t}_2\sum _{h_1,h_2}\frac{{\delta }^2\mathbb{E}\left[{\dot{n}}_h\left(t\right)|\right\{I_h^0\left(t\right)\left\}\right]}{\delta I_{h_2}\left(t_2\right)\delta I_{h_1}\left(t_1\right)}(I_{h_2}\left(t_2\right)-I_{h_2}^0\left(t_2\right))(I_{h_1}\left(t_1\right)-I_{h_1}^0\left(t_1\right))\hfill \\ & +\dots \hfill \end{array}$$

Within our mean field approximation, the Taylor coefficients are simply the response functions of the network—i.e., the zeroth order coefficient is the mean firing rate of the neurons in the reference state $I_h^0\left(t\right)$, the first order coefficient is the linear response function of the network, the second order coefficient is a nonlinear response function, and so on.

There are two natural choices for the reference state $I_h^0\left(t\right)$. The first is simply the state of zero recorded unit activity, while the second is the mean activity of the recorded neurons. The zero-activity case conforms to the choice of nonlinear Hawkes models used in practice. Choosing the mean activity as the reference state may be more appropriate if the recorded neurons have high firing rates, but requires adjusting the form of the nonlinear Hawkes model so that firing rates are modulated by filtering the deviations of spikes from the mean firing rate, rather than filtering the spikes themselves. Here, we focus on the zero-activity reference state. We present the formulation for the mean field reference state in the SI.

For the zero-activity reference state $I_h^0\left(t\right)=0$, the conditional mean is

$$\begin{array}{cc}\hfill \mathbb{E}\left[{\dot{n}}_h\left(t\right)|\right\{I_h\left(t\right)\left\}\right]& =\mathbb{E}\left[{\dot{n}}_h|0\right]+\int d{t}_1\sum _{h_1}\frac{\delta \mathbb{E}\left[{\dot{n}}_h\left(t\right)|0\right]}{\delta I_{h_1}\left(t_1\right)}{I}_{h_1}\left(t_1\right)\hfill \\ & +\frac{1}{2}\int d{t}_{1}d{t}_2\sum _{h_1,h_2}\frac{{\delta }^2\mathbb{E}\left[{\dot{n}}_h\left(t\right)|0\right]}{\delta I_{h_2}\left(t_2\right)\delta I_{h_1}\left(t_1\right)}{I}_{h_2}\left(t_2\right)I_{h_1}\left(t_1\right)+\dots .\hfill \end{array}$$

The mean inputs $\mathbb{E}\left[{\dot{n}}_h|0\right]$ are the mean field approximations to the firing rates of the hidden neurons in the absence of the recorded neurons. Defining ${\nu }_h\equiv \mathbb{E}\left[{\dot{n}}_h|0\right]$, these firing rates are given by

$${\nu }_h={\lambda }_0\varphi ({\mu }_h+\sum _{h^{\prime }}{\mathcal{J}}_{h,h^{\prime }}{\nu }_{h^{\prime }});$$

in writing this equation we have assumed that the steady-state mean field firing rates will be time-independent, and hence the convolution $J_{h,h^{\prime }}*{\nu }_{h^{\prime }}={\mathcal{J}}_{h,h^{\prime }}{\nu }_{h^{\prime }}$, where ${\mathcal{J}}_{h,h^{\prime }}={\int }_0^{\infty }dt{J}_{h,h^{\prime }}\left(t\right)$. This assumption will generally be valid for at least some parameter regime of the network, but there can be cases where it breaks down, such as if the nonlinearity ϕ(x) is bounded, in which case a transition to chaotic firing rates ν_h(t) may exist (c.f. [74]). The mean field equations for the ν_h are a system of transcendental equations that in general cannot be solved exactly. In practice we will solve the equations numerically, but we can develop a series expansion for the solutions (see below).

The next term in the series expansion is the linear response function of the hidden unit network, ${\Gamma }_{h,h^{\prime }}(t-t^{\prime })\equiv \frac{\delta \mathbb{E}\left[{\dot{n}}_h\left(t\right)|0\right]}{\delta I_{h^{\prime }}\left(t^{\prime }\right)},$ given by the solution to the integral equation

$${\Gamma }_{h,h^{\prime }}(t-t^{\prime })={\gamma }_h({\delta }_{h,h^{\prime }}\delta (t-t^{\prime })+\sum _{h^{\prime \prime }}{\int }_0^{\infty }d{t}^{\prime \prime }{J}_{h,h^{\prime \prime }}(t-t^{\prime \prime }){\Gamma }_{h^{\prime \prime },h^{\prime }}(t^{\prime \prime }-t^{\prime })).$$

The “gain” γ_h is defined by

$${\gamma }_h\equiv {\lambda }_0{\varphi }^{\prime }({\mu }_h+\sum _{h^{\prime }}{\mathcal{J}}_{h,h^{\prime }}{\nu }_{h^{\prime }}),$$

where ϕ′(x) is the derivative of the nonlinearity with respect to its argument.

For time-independent drives μ_r and steady states ν_h (and hence γ_h), we may solve for Γ_h,h′(t − t′) by first converting to the frequency domain and then performing a matrix inverse:

$${\widehat{\Gamma }}_{h,h^{\prime }}\left(\omega \right)={[\mathbb{I}-\widehat{\mathbf{V}}\left(\omega \right)]}_{h,h^{\prime }}^{-1}{\gamma }_{h^{\prime }},$$

where ${\widehat{V}}_{h,h^{\prime }}\left(\omega \right)={\gamma }_h{J}_{h,h^{\prime }}\left(\omega \right)$.

If the zero and first order Taylor series coefficients in our expansion of $\mathbb{E}\left[{\dot{n}}_h\left(t\right)\right|\left\{{\dot{n}}_r\right\}]$ are the dominant terms—i.e., if we may neglect higher order terms in this expansion—then we may approximate the instantaneous firing rates of the recorded neurons by

$${\lambda }_r\left(t\right)\approx {\lambda }_0\varphi ({\mu }_r^{\text{eff}}+\sum _{r^{\prime }}{J}_{r,r^{\prime }}^{\text{eff}}*{\dot{n}}_{r^{\prime }}\left(t\right)),$$

where

$${\mu }_r^{\text{eff}}={\mu }_r+\sum _h{\mathcal{J}}_{r,h}{\nu }_h$$

are the effective baselines of the recorded neurons and

$${\widehat{J}}_{r,r^{\prime }}^{\text{eff}}\left(\omega \right)={\widehat{J}}_{r,r^{\prime }}\left(\omega \right)+\sum _{h,h^{\prime }}{\widehat{J}}_{r,h}\left(\omega \right){\widehat{\Gamma }}_{h,h^{\prime }}\left(\omega \right){\widehat{J}}_{h^{\prime },r^{\prime }}\left(\omega \right)$$

are the effective coupling filters in the frequency domain, as given in the main text. In addition to neglecting the higher order spike filtering terms here, we have also neglected fluctuations around the mean input from the hidden network. These fluctuations are zero within our mean field approximation, but we could in principle calculate corrections to the mean field predictions using the techniques of [53]; we do so to estimate the size of the effective noise correlations in the SI.

In the main text, we decompose our expression for ${\widehat{J}}_{r,r^{\prime }}^{\text{eff}}\left(\omega \right)$ into contributions from all paths that a signal can travel from neuron r′ to r. To arrive at this interpretation, we note that we can expand ${\widehat{\Gamma }}_{h,h^{\prime }}\left(\omega \right)$ in a series over paths through the hidden network. To start, we note that if $\left|\right|\widehat{\mathbf{V}}\left(\omega \right)\left|\right|<1$ for some matrix norm ||⋅||, then the matrix ${[\mathbb{I}-\mathbf{V}(\omega \left)\right]}^{-1}$ admits a convergent series expansion [75]

$${[\mathbb{I}-\widehat{\mathbf{V}}\left(\omega \right)]}^{-1}=\sum _{\ell =0}^{\infty }\widehat{\mathbf{V}}{\left(\omega \right)}^{\ell },$$

where $\widehat{\mathbf{V}}{\left(\omega \right)}^{\ell }$ is a matrix product and $\widehat{\mathbf{V}}{\left(\omega \right)}^0\equiv \mathbb{I}$. We can write an element of the matrix product out as

$${\left[\widehat{\mathbf{V}}{\left(\omega \right)}^{\ell }\right]}_{h,h^{\prime }}=\sum _{h_1,\dots ,h_{\ell }}{\widehat{V}}_{h,h_1}\left(\omega \right){\widehat{V}}_{h_1,h_2}\left(\omega \right)\dots {\widehat{V}}_{h_{\ell -1},h_{\ell }}\left(\omega \right){\widehat{V}}_{h_{\ell },h^{\prime }}\left(\omega \right);$$

inserting ${\widehat{V}}_{h_i,h_j}\left(\omega \right)={\gamma }_{h_i}{\widehat{J}}_{h_i,h_j}\left(\omega \right)$ yields

$${\left[\widehat{\mathbf{V}}{\left(\omega \right)}^{\ell }\right]}_{h,h^{\prime }}=\sum _{h_1,\dots ,h_{\ell }}{\gamma }_h{\widehat{J}}_{h,h_1}\left(\omega \right){\gamma }_{h_1}{\widehat{J}}_{h_1,h_2}\left(\omega \right)\dots {\gamma }_{h_{\ell -1}}{\widehat{J}}_{h_{\ell -1},h_{\ell }}\left(\omega \right){\gamma }_{h_{\ell }}{\widehat{J}}_{h_{\ell },h^{\prime }}\left(\omega \right).$$

This expression can be interpreted in terms of summing over paths through network of hidden neurons that join two observed neurons: the ${\widehat{J}}_{h_i,h_j}\left(\omega \right)$ are represented by edges from neuron h_j to h_i, and the ${\gamma }_{h_i}$ are represented by the nodes. In this expansion, we allow edges from one neuron back to itself, meaning we include paths in which signals loop back around to the same neuron arbitrarily many times before the signal is propagated further. However, such loops can be easily factored, contributing a factor ${\sum }_{m=0}^{\infty }{\left({\gamma }_h{\widehat{J}}_{h,h}\left(\omega \right)\right)}^m=1/(1-{\gamma }_h{\widehat{J}}_{h,h}\left(\omega \right))$. We thus remove the need to consider self-loops in our rules for calculating effective coupling filters by assigning a factor γ_h/(1 − γ_h J_{h, h}(ω)) to each node, as discussed in the main text and depicted in Fig 2. (The contribution of the self-feedback loops can be derived rigorously; see the SI for the full derivation).

Although we have worked here in the frequency domain, our formalism does adapt straightforwardly to handle time-dependent inputs; however, among the consequences of this explicit time-dependence are that the mean field rates ν_h(t) are not only time-dependent, but solutions of a system of nonlinear integral equations, and hence more challenging to solve. Furthermore, quantities like the linear response of the hidden network, Γ_h,h′(t, t′), will depend on both absolute times t and t′, rather than just their difference, t − t′, and hence we must also (numerically) solve for Γ_h,h′(t, t′) directly in the time domain. We leave these challenges for future work.

Model network architectures

Our main result, Eq (2), is valid for general network architectures with arbitrary weighted synaptic connections, so long as the hidden subset of the network has stable dynamics when the recorded neurons are removed. An example for which our method must be modified would be a network in which all or the majority of the hidden neurons are excitatory, as the hidden network is unlikely to be stable when the inhibitory recorded neurons are disconnected. Similarly, we find that synaptic weight distributions with undefined moments will generally cause the network activity to be unstable. For example, ${\mathcal{J}}_{i,j}$ drawn from a Cauchy distribution generally yield unstable network dynamics unless the weights are scaled inversely with a large power of the network size N.

Specific networks—Common features

The specific network architectures we study in the main text share several features in common: all are sparse networks with sparsity p (i.e., only a fraction p of connections are non-zero) and non-zero synaptic weight strengths drawn independently from a random distribution with zero population mean and population standard deviation J₀/(pN)^a; the overall standard deviation of weights, accounting for the expected 1 − p fraction of zero weights is $\sqrt{p}{J}_0/{\left(pN\right)}^a$. The parameter a determines whether the synaptic strengths are “strong” (a = 1/2) or “weak” (a = 1). In most of our analytical results we only need the mean and variances of the weights, so we do not need to specify the exact distribution. In simulations, we use a normal distribution. The reason for scaling the weights as 1/(pN)^a, as opposed to just 1/N^a, is that the mean incoming degree of connections is p(N − 1) ≈ pN for large networks; this scaling thus controls for the typical magnitude of incoming synaptic currents.

For strongly coupled networks, the combined effect of sparsity and synaptic weight distribution yields an overall standard deviation of $\sqrt{p}{J}_0/\sqrt{pN}=J_0/\sqrt{N}$. Because the sparsity parameter p cancels out, it does not matter if we consider p to be fixed or k₀ = pN to be fixed—both cases are equivalent. However, this is not the case if we scale ${\mathcal{J}}_{i,j}$ by 1/k₀, as the overall standard deviation would then be $\sqrt{p}{J}_0/k_0$, which only corresponds to the weak-coupling limit if p is fixed. If k₀ is fixed, the standard deviation would scale as $1/\sqrt{N}$.

It is worth noting that the determination of “weak” versus “strong” coupling depends not only on the power of N with which synaptic weights scale, but also on the network architecture and correlation structure of the weights ${\mathcal{J}}_{i,j}$. For example, for an all-to-all connected matrix with symmetric rank-1 synaptic weights of the form ${\mathcal{J}}_{i,j}={\zeta }_i{\zeta }_j$, where the ζ_i are independently distributed normal random variates, the standard deviation of each ζ must scale as $1/\sqrt{N}$ in order for hidden paths to generate $\mathcal{O}\left(1\right)$ contributions to effective interactions, such that ${\mathcal{J}}_{i,j}$ scales as 1/N but the coupling is still strong.

Specific networks—Differences in architecture and synaptic constraints

Beyond the common features outlined above, we perform our analysis of the distribution of effective synaptic interaction strengths for three network architectures commonly studied in network models. These architectures are not intended to be realistic representations of neuronal network structures, but to capture basic features of network architecture and therefore give insight into the basic features of the effective interaction networks.

Erdős-Réyni + mixed synapses—The first network we consider (and the one we perform most of our later analyses on as well) is an Erdős-Réyni random network architecture with “mixed synapses.” That is, each connection between neurons is chosen randomly with probably p. By “mixed synapses” we mean that each neuron’s outgoing synaptic weights are chosen completely independently. i.e., in this network there are no excitatory or inhibitory neurons; each neuron make make both excitatory and inhibitory connections. The corresponding analysis is shown in Fig 5A.

Erdős-Réyni + Dale’s law imposed—Real neurons appear to split into separate excitatory and inhibitory classes, a dichotomy know as “Dale’s law” (or alternatively, “Dale’s principle” to highlight that it is not really a law of nature). Neurons in a network that obeys this law will have coupling filters J_i,j(t) that are strictly positive for excitatory neurons and strictly negative for inhibitory neurons. This constraint complicates analytic calculations slightly, as the moments of the synaptic weights now depend on the identity of the neuron, and more care must be taken in calculating expected values or population averages. We instead impose this numerically to generate the results shown in Fig 5B. The trends are the same as in the network with mixed synapses, with the resulting ratios being slightly reduced.

As a technical point, because our analysis requires calculation of the mean field firing rates of the hidden network in absence of the recorded neurons, random sampling of the network may, by chance, yield hidden networks with an imbalance of excitatory neurons, for which the mean field firing rates of the hidden network may diverge for our choice of exponential nonlinearity. This is the origin of the relatively larger error bars in Fig 5B: less random samplings for which the hidden network was stable were available to perform the computation. One way this artifact can be prevented is by choosing a nonlinearity that saturates, such as ϕ(x) = c/(1 + exp(−x)), which prevents the mean-field firing rates from diverging and yields stable network activity (see Fig 8). Another is to choose a different reference state of network activity around which we perform our expansion of $\mathbb{E}\left[{\dot{n}}_h\right|\left\{{\dot{n}}_r\right\}]$, such as the mean field state discussed in the SI.

Fig 8. Same as Fig 5B in the main text, but for a sigmoidal nonlinearity ϕ(x) = 2/(1 + e^−x).

Because the sigmoid is bounded the mean field solution cannot diverge, yielding better results.

Watts-Strogatz network + mixed synapses—Finally, although Erdős-Réyni networks are relatively easy to analyze analytically, and are ubiquitous in many influential computational and theoretical studies, real world networks typically have more structure. Therefore, we also consider a network architecture with more structure, a Watts-Strogatz (small world) network. A Watts-Strogatz network is generated by starting with a K-nearest neighbor network (such that fraction of non-zero connections each neuron makes is p = K/(N − 1)) and rewiring a fraction β of those connections. The limit β = 0 remains a K-nearest neighbor network, while β → 1 yields an Erdős-Réyni network. We generated the adjacency matrices of the Watts-Strogatz networks using code available in [76]. Here we consider only a Watts-Strogatz network with mixed synapses; a network with spatial structure and Dale’s law would become sensitive to both the spatial distribution of excitatory and inhibitory neurons in the network as well as the way in which the neurons are sampled, an investigation we leave for future work. The results for the Watts-Strogatz network with mixed synapses are shown in Fig 5C, and are qualitatively similar to the Erdős-Réyni network with mixed synapses.

Because all three network types we considered yield qualitatively similar results, for the remainder of our analyses, we focus on the Erdős-Réyni + mixed synapses network for simplicity in both simulations and analytical calculations.

Parameter values used to generate our networks are given in Table 1.

Choice of nonlinearity ϕ(x)

The nonlinear function ϕ(x) sets the instantaneous firing rate for the neurons in our model. Our main analytical results (e.g., Eq (2) hold for arbitrary choice of ϕ(x). Where specific choices are required in order to perform simulations, we used ϕ(x) = max(x, 0) for the results presented in Figs 3 and 4 and ϕ(x) = exp(x) otherwise. The rectified linear choice is convenient for small networks, as high-order derivatives are zero, which eliminates corresponding high-order “loop corrections” to mean field theory [53]. The exponential function is the “canonical” choice of nonlinearity for the nonlinear Hawkes process [16–18, 20]. The exponential has particularly nice theoretical properties, but is also convenient for fitting the nonlinear Hawkes model to data, as the log-likelihood function of the model simplifies considerably and is convex (though some similar families of nonlinearities also yield convex log-likelihood functions).

An important property that both choices of nonlinearity possess is that they are unbounded. This property is necessary to guarantee that a neuron spikes given enough input. A bounded nonlinearity imposes a maximum firing rate, and neurons cannot be forced to spike reliably by providing a large bolus of input. The downside of an unbounded nonlinearity is that it is possible for the average firing rates to diverge, and the network never reaches a steady state. For example, in a purely excitatory network (all ${\mathcal{J}}_{i,j}\ge 0$) with an exponential nonlinearity, neural firing will run away without a sufficiently strong self-refractory coupling to suppress the firing rate. This will not occur with a bounded nonlinearity, as excitation can only drive neurons to fire at some maximum but finite rate.

This can be a problem in simulations of networks obeying Dale’s law. For unbounded nonlinearities, the mean field theory for the hidden network occasionally does not exist due to an imbalance of excitatory and inhibitory neurons caused by our random selection of recorded of neurons. However, the Dale’s law network is stable if the nonlinearity is bounded. We demonstrate this below in Figs 7 and 8, comparing simulations of the effective interaction statistics in Erdős-Réyni networks with and without Dale’s law for a sigmoidal nonlinearity ϕ(x) = 2/(1 + e^−x).

Fig 7. Same as Fig 5A in main text, but for a sigmoidal nonlinearity ϕ(x) = 2/(1 + e^−x).

Another consequence of unbounded nonlinearities is that the mean firing rates are either finite or they diverge. Bounded nonlinearities, on the other hand, may allow for the possibility of a transition to chaotic dynamics in the mean-field firing rate dynamics (cf. the results of the [74]).

Specific choices of network properties used to generate figures

Feedforward-inhibitiory circuit model details

3 neuron circuit (Fig 3).

Using our graphical rules (Fig 2), we calculated the effective interaction from neuron 1 to 2 for the circuit shown in Fig 3A, giving Eq 3. In principle, our mean field approximation would not be expected to hold for such a small circuit; in particular, loop corrections [53] to our calculation of the rate ν₃ and associated gain γ₃ might be significant. However, as loop corrections depend on derivatives of the nonlinearity ϕ(x), we can minimize these errors by choosing ϕ(x) = max(x, 0), for which ϕ′(x) = Θ(x), the Heaviside step function. Accordingly, we can solve for ${\nu }_3={\lambda }_0{\mu }_3/(1-{\lambda }_0{\mathcal{J}}_{33})$ and γ₃ = λ₀ for this particular network.

To generate the plots shown in Fig 3C, we take the inter-neuron couplings to have the form $J_{i,j}\left(\tau \right)={\mathcal{J}}_{i,j}{\alpha }_{i,j}^2\tau e^{-{\alpha }_{i,j}\tau }$ and the self-history couplings to have the form $J_{i,i}\left(\tau \right)={\mathcal{J}}_{i,i}{\beta }_{i,i}{e}^{-{\beta }_{i,i}\tau }$.

Using Mathematica to perform the inverse Fourier transform, we obtain an explicit expression for the effective interaction,

$$\begin{array}{cc}\hfill J_{2,1}^{\text{eff}}\left(\tau \right)& ={\mathcal{J}}_{21}{\alpha }_{21}^2\tau e^{-{\alpha }_{21}\tau }\hfill \\ & +{\mathcal{J}}_{23}{\mathcal{J}}_{31}{\alpha }_{23}^2{\alpha }_{31}^2\times \hfill \\ & [\frac{{\beta }_{33}{\mathcal{J}}_{33}}{{({\alpha }_{23}-{\beta }_{33}(1-{\lambda }_0{\mathcal{J}}_{33}))}^2{({\alpha }_{31}-{\beta }_{33}(1-{\lambda }_0{\mathcal{J}}_{33}))}^2}{e}^{-{\beta }_{33}(1-{\lambda }_0{\mathcal{J}}_{33})\tau }\hfill \\ & +\frac{(-2{\alpha }_{31}^2+{\beta }_{33}{\alpha }_{31}(4-{\lambda }_0{\mathcal{J}}_{33})-2{\beta }_{33}^2(1-{\lambda }_0{\mathcal{J}}_{33})-{\beta }_{33}{\mathcal{J}}_{33}{\alpha }_{23})}{{({\alpha }_{23}-{\alpha }_{31})}^2{({\alpha }_{31}-{\beta }_{33}(1-{\lambda }_0{\mathcal{J}}_{33}))}^2}{e}^{-{\alpha }_{31}\tau }\hfill \\ & +\frac{(-2{\alpha }_{23}^2+{\beta }_{33}{\alpha }_{23}(4-{\lambda }_0{\mathcal{J}}_{33})-2{\beta }_{33}^2(1-{\lambda }_0{\mathcal{J}}_{33})-{\beta }_{33}{\mathcal{J}}_{33}{\alpha }_{31})}{{({\alpha }_{31}-{\alpha }_{23})}^2{({\alpha }_{23}-{\beta }_{33}(1-{\lambda }_0{\mathcal{J}}_{33}))}^2}{e}^{-{\alpha }_{23}\tau }\hfill \\ & +\frac{{\alpha }_{23}-{\beta }_{33}}{{({\alpha }_{23}-{\alpha }_{31})}^2{({\alpha }_{23}-{\beta }_{33}(1-{\lambda }_0{\mathcal{J}}_{33}))}^2}\tau e^{-{\alpha }_{23}\tau }\hfill \\ & +\frac{{\alpha }_{31}-{\beta }_{33}}{{({\alpha }_{31}-{\alpha }_{23})}^2{({\alpha }_{31}-{\beta }_{33}(1-{\lambda }_0{\mathcal{J}}_{33}))}^2}\tau e^{-{\alpha }_{31}\tau }].\hfill \end{array}$$

In order for the inverse Fourier transform to converge and result in a causal function, we require that $1-{\lambda }_0{\mathcal{J}}_{33}>0$.

Parameter values used to generate the plots in Fig 3C are given in Table 2.

Table 2. Parameter values for Fig 3C.

Setting λ₀ = 1.0 simply sets the units of frequency and time to be measured relative to λ₀ (e.g., the value α₃₁ = 1.8 really means α₃₁ = 1.8λ₀ and ${\mathcal{J}}_{31}=2.0$ really means ${\mathcal{J}}_{31}=2.0/{\lambda }_0$).

Parameter	value
λ0	1.0
${\mathcal{J}}_{21}$	1.0
${\mathcal{J}}_{23}$	−2.0
${\mathcal{J}}_{31}$	2.0
${\mathcal{J}}_{33}$	−0.9
α21 = α23 = β33	1.0
α31	1.8

4 neuron circuit (Fig 4).

Like for the 3-neuron circuit, we can use our graphical rules (Fig 2) to calculate the effective interaction for our 4-neuron circuit (Fig 4A) in the frequency domain:

$$\begin{array}{cc}\hfill {\widehat{J}}_{21}^{\text{eff}}\left(\omega \right)-{\widehat{J}}_{21}\left(\omega \right)& ={\widehat{J}}_{23}\left(\omega \right)\left[\sum _{m=0}^{\infty }\right({\gamma }_3{\widehat{J}}_{34}\left(\omega \right){\gamma }_4{\widehat{J}}_{43}\left(\omega \right){)}^m]{\widehat{J}}_{31}\left(\omega \right)\hfill \\ & +{\widehat{J}}_{23}\left(\omega \right)\left[\sum _{m=0}^{\infty }{\left({\gamma }_3{\widehat{J}}_{34}\left(\omega \right){\gamma }_4{\widehat{J}}_{43}\left(\omega \right)\right)}^m\right]{\gamma }_3{\widehat{J}}_{34}\left(\omega \right){\gamma }_4{\widehat{J}}_{41}\left(\omega \right)\hfill \\ & ={\widehat{J}}_{23}\left(\omega \right)\frac{1}{1-{\gamma }_3{\widehat{J}}_{34}\left(\omega \right){\gamma }_4{\widehat{J}}_{43}\left(\omega \right)}{\widehat{J}}_{31}\left(\omega \right)\hfill \\ & +{\widehat{J}}_{23}\left(\omega \right)\frac{1}{1-{\gamma }_3{\widehat{J}}_{34}\left(\omega \right){\gamma }_4{\widehat{J}}_{43}\left(\omega \right)}{\gamma }_3{\widehat{J}}_{34}\left(\omega \right){\gamma }_4{\widehat{J}}_{41}\left(\omega \right)\hfill \\ & ={\widehat{J}}_{23}\left(\omega \right){\widehat{J}}_{31}\left(\omega \right)+{\widehat{J}}_{23}\left(\omega \right){\gamma }_3{\widehat{J}}_{34}\left(\omega \right){\gamma }_4{\widehat{J}}_{41}\left(\omega \right)\hfill \\ & +\frac{{\widehat{J}}_{23}\left(\omega \right){\gamma }_3{\widehat{J}}_{34}\left(\omega \right){\gamma }_4{\widehat{J}}_{43}\left(\omega \right){\widehat{J}}_{31}\left(\omega \right)}{1-{\gamma }_3{\widehat{J}}_{34}\left(\omega \right){\gamma }_4{\widehat{J}}_{43}\left(\omega \right)}+\frac{{\widehat{J}}_{23}\left(\omega \right){\gamma }_3{\widehat{J}}_{34}\left(\omega \right){\gamma }_4{\widehat{J}}_{43}\left(\omega \right){\gamma }_3{\widehat{J}}_{34}\left(\omega \right){\gamma }_4{\widehat{J}}_{41}\left(\omega \right)}{1-{\gamma }_3{\widehat{J}}_{34}\left(\omega \right){\gamma }_4{\widehat{J}}_{43}\left(\omega \right)};\hfill \end{array}$$

in going to the last equality we have separated the terms out into contributions from each of the paths, in order, shown in Fig 4B.

To generate the plots in Fig 4C, we choose ϕ(x) = max(x, 0), which gives γ_i = λ₀, as in Fig 3C, and interaction filters $J_{2,1}\left(\tau \right)={\mathcal{J}}_{21}{\alpha }_{21}^2\tau e^{-\alpha \tau }$ for the direct interaction and $J_{i,j}\left(\tau \right)={\mathcal{J}}_{ij}{\alpha }^2\tau e^{-\alpha \tau }$ for all other interactions shown—i.e., all other interactions have the same decay time α⁻¹ for simplicity.

Inverting the Fourier transform using Mathematica yields

$$\begin{array}{cc}\hfill J_{2,1}^{\text{eff}}\left(\tau \right)& ={\mathcal{J}}_{21}{\alpha }^2\tau e^{-\alpha \tau }-\frac{{\mathcal{J}}_{23}{\mathcal{J}}_{31}\alpha e^{-\alpha \tau }}{2|{\mathcal{J}}_{34}{|}^{3/4}{\left|{\mathcal{J}}_{43}\right|}^{3/4}}\left(\text{sin}\left(\alpha \right(|{\mathcal{J}}_{34}\left|\right|{\mathcal{J}}_{43}{\left|\right)}^{1/4}\tau )-\text{sinh}(\alpha \left(\right|{\mathcal{J}}_{34}\left|\right|{\mathcal{J}}_{43}{\left|\right)}^{1/4}\tau )\right)\hfill \\ & +\frac{{\mathcal{J}}_{23}{\mathcal{J}}_{41}\alpha e^{-\alpha \tau }}{2|{\mathcal{J}}_{34}{|}^{1/4}{\left|{\mathcal{J}}_{43}\right|}^{5/4}}(2\alpha \left(\right|{\mathcal{J}}_{34}\left|\right|{\mathcal{J}}_{43}{\left|\right)}^{1/4}\tau -\text{sin}\left(\alpha \right(|{\mathcal{J}}_{34}\left|\right|{\mathcal{J}}_{43}{\left|\right)}^{1/4}\tau )-\text{sinh}(\alpha \left(\right|{\mathcal{J}}_{34}\left|\right|{\mathcal{J}}_{43}{\left|\right)}^{1/4}\tau ))\hfill \end{array}$$

In order for this result to converge, we require $|{\mathcal{J}}_{34}\left|\right|{\mathcal{J}}_{43}|<1$. Splitting this result up into the contributions to each plot in Fig 4C, using the specific parameter choices λ₀ = 1 and ${\mathcal{J}}_{34}={\mathcal{J}}_{43}\equiv \mathcal{J}$, gives

$$\begin{array}{cc}\hfill 2\leftarrow 3\leftarrow 1:& \frac{1}{6}{\alpha }^4{\mathcal{J}}_{23}{\mathcal{J}}_{31}{\tau }^3{e}^{-\alpha \tau },\hfill \\ \hfill 2\leftarrow 3\leftarrow 4\leftarrow 1:& -\frac{1}{120}{\alpha }^6\left|\mathcal{J}\right|{\mathcal{J}}_{23}{\mathcal{J}}_{41}{\tau }^5{e}^{-\alpha \tau },\hfill \\ \hfill 2\leftarrow 3\leftrightarrow 4\leftarrow 3\leftarrow 1:& \frac{\alpha {\mathcal{J}}_{23}{\mathcal{J}}_{31}\left(\text{cosh}\right(\alpha \tau )-\text{sinh}(\alpha \tau \left)\right)(-2{\alpha }^3{\left|\mathcal{J}\right|}^{3/2}{\tau }^3-6\text{sin}\left(\alpha \sqrt{\left|\mathcal{J}\right|}\tau \right)+6\text{sinh}\left(\alpha \sqrt{\left|\mathcal{J}\right|}\tau \right)}{{12\left|\mathcal{J}\right|}^{3/2}},\hfill \\ \hfill 2\leftarrow 3\leftrightarrow 4\leftarrow 1:& \frac{\alpha e^{-\alpha \tau }{\mathcal{J}}_{23}{\mathcal{J}}_{41}(\alpha \sqrt{\left|\mathcal{J}\right|}\tau (120+{\alpha }^4{\mathcal{J}}^2{\tau }^4)-60\text{sin}\left(\alpha \sqrt{\mathcal{J}}\tau \right)-60\text{sinh}\left(\alpha \sqrt{\left|\mathcal{J}\right|}\tau \right))}{{120\left|\mathcal{J}\right|}^{3/2}}.\hfill \end{array}$$

Parameter values used to generate the plots in Fig 4C are given in Table 3.

Table 3. Parameter values for Fig 4C.

Setting λ₀ = 1.0 simply sets the units of frequency and time to be measured relative to λ₀ (e.g., the value α = 1.294 really means α = 1.294λ₀ and ${\mathcal{J}}_{23}=-3.0$ really means ${\mathcal{J}}_{23}=-3.0/{\lambda }_0$).

Parameter	value
λ0	1.0
${\mathcal{J}}_{21}={\mathcal{J}}_{31}={\mathcal{J}}_{41}$	1.0
${\mathcal{J}}_{23}$	−3.0
${\mathcal{J}}_{34}={\mathcal{J}}_{43}=\mathcal{J}$	−0.9
α21	1.0
α	1.294

Large networks

To generate the results in Fig 6 in the main text, we choose the coupling filters to be $J_{i,j}\left(t\right)={\mathcal{J}}_{i,j}{\alpha }^{2}t{e}^{-\alpha t}$, for i ≠ j, which has Fourier transform

$${\widehat{J}}_{i,j}\left(\omega \right)=\frac{{\mathcal{J}}_{i,j}{\alpha }^2}{{(\alpha +i\omega )}^2},$$

using the Fourier convention

$$\widehat{f}\left(\omega \right)={\int }_{-\infty }^{\infty }dt{e}^{-i\omega t}f\left(t\right).$$

The weight matrix $\mathcal{J}$ is generated as described in “Model network architectures,” choosing J₀ = 1.0. We partition this network up into recorded and hidden subsets. For a network of N neurons, we choose neurons 1 to N_rec to be recorded, and the remainder to be hidden, hence we define (using an index notation starting at 1; indices should be subtracted by 1 for 0-based index counting)

$${\mathcal{J}}^{\text{RR}}=\mathcal{J}[1:N_{\text{rec}},1:N_{\text{rec}}],$$

$${\mathcal{J}}^{\text{RH}}=\mathcal{J}[1:N_{\text{rec}},(N_{\text{rec}}+1):N],$$

$${\mathcal{J}}^{\text{HR}}=\mathcal{J}[(N_{\text{rec}}+1):N,1:N_{\text{rec}}],$$

and

$${\mathcal{J}}^{\text{HH}}=\mathcal{J}[(N_{\text{rec}}+1):N,(N_{\text{rec}}+1):N].$$

We numerically calculate the linear response matrix $\widehat{\mathbf{\Gamma }}\left(\omega \right)$ by evaluating

$$\widehat{\mathbf{\Gamma }}\left(\omega \right)={[\mathbb{I}-{\widehat{\mathbf{V}}}^{\text{HH}}\left(\omega \right)]}^{-1}\text{diag}\left(\overrightarrow{\gamma }\right),$$

where ${\widehat{V}}_{h,h^{\prime }}^{\text{HH}}\left(\omega \right)={\gamma }_h{\mathcal{J}}_{h,h^{\prime }}\left(\omega \right)$ and diag($\overrightarrow{\gamma })$ is an N_hid × N_hid diagonal matrix with elements γ_h.

The effective coupling filter in the frequency domain can then be evaluated pointwise at a desired set of frequencies ω by matrix multiplication,

$${\widehat{\mathbf{J}}}^{\text{eff}}\left(\omega \right)=\frac{{\alpha }^2}{{(\alpha +i\omega )}^2}{\mathcal{J}}^{\text{RR}}+{\left(\frac{{\alpha }^2}{{(\alpha +i\omega )}^2}\right)}^2{\mathcal{J}}^{\text{RH}}\widehat{\Gamma }\left(\omega \right){\mathcal{J}}^{\text{HR}}.$$

We then return to the time domain by inverse Fourier transforming the result, achieved by treating ${\widehat{\mathbf{J}}}^{\text{eff}}\left(\omega \right)$ as an N_rec × N_rec × N_freq array (where N_freq is the number of frequencies at which we evaluate the effective coupling) and multiplying along the frequency dimension by an N_freq × N_time matrix E with elements E_ω,t = exp(iωt)Δt/(2π), for N_time sufficiently small time bins of size δt = 0.1/α, for α = 10, as listed in S1 Table.

To generate Fig 5, we focus on the zero-frequency component of ${\widehat{J}}^{\text{eff}}\left(\omega \right)$, which is also equal to the time integral of J^eff(t). As in the main text, we label the elements of this component ${\mathcal{J}}_{r,r^{\prime }}^{\text{eff}}={\widehat{J}}_{r,r^{\prime }}^{\text{eff}}(\omega =0)$, which is equal to

$${\mathcal{J}}_{r,r^{\prime }}^{\text{eff}}={\mathcal{J}}_{r,r^{\prime }}+\sum _{h,h^{\prime }}{\mathcal{J}}_{r,h}{\widehat{\Gamma }}_{h,h^{\prime }}\left(0\right){\mathcal{J}}_{h^{\prime },r^{\prime }}.$$

We do not need to simulate the full network to study the statistics of ${\mathcal{J}}_{r,r^{\prime }}^{\text{eff}}$. We only need to generate samples of the matrix $\mathcal{J}$ and evaluate $\widehat{\mathbf{\Gamma }}\left(0\right)$. This is where the choice of an Erdős-Réyni network that is not restricted to obey Dale’s law becomes convenient. Because the weights ${\mathcal{J}}_{i,j}$ are i.i.d. and the sign of the weight is random, population averages will be equivalent to expected values. i.e., the sample mean

$${\tilde{\mathcal{J}}}_{\text{mean}}=\frac{1}{N_{\text{rec}}(N_{\text{rec}}-1)}\sum _{r\ne r^{\prime }}{\mathcal{J}}_{r,r^{\prime }}^{\text{eff}}$$

and sample variance

$${\tilde{\mathcal{J}}}_{\text{var}}=\frac{1}{N_{\text{rec}}(N_{\text{rec}}-1)-1}\sum _{r\ne r^{\prime }}{({\mathcal{J}}_{r,r^{\prime }}^{\text{eff}}-{\tilde{\mathcal{J}}}_{\text{mean}})}^2$$

will tend to the expected values $\mathbb{E}\left[{\mathcal{J}}_{r,r^{\prime }}^{\text{eff}}\right]$ and $\text{var}\left[{\mathcal{J}}_{r,r^{\prime }}^{\text{eff}}\right]$ for large networks. We have explicitly removed the diagonal elements from these averages because these elements will have slightly different statistics from the off-diagonal elements due to the fact that all ground-truth self-couplings are set to zero, ${\mathcal{J}}_{r,r}=0$. This allows us to compare the population variance, plotted in Fig 5 (after normalization by the population variance of the true off-diagonal weights), to the expected variance calculated analytically below.

The error bars in Fig 5 are generated by first drawing a single sample of true weights $\mathcal{J}$, and then taking 100 random subsets of N_rec = {10, 110, 210, 310, 410, 510, 610, 710, 810, 910, 999} recorded neurons. For this analysis, random subsets were generated by permuting the indices of the full weight matrix $\mathcal{J}$ and taking the last N_rec neurons to be recorded. For each random subset of the network we calculate the population statistics. The standard error of, for example, the population variance ${\tilde{\mathcal{J}}}_{\text{var}}$ across subsets gives an estimate of the error. However, if we only use a single sample of the network architecture and weights ${\mathcal{J}}_{i,j}$, this estimate may depend on the particular instantiation of the network. To average over the effects of global network architecture, we draw a total of 10 network architecture samples, and average a second time over these samples to obtain our final estimates of the population variance of ${\mathcal{J}}_{r,r^{\prime }}^{\text{eff}}$. We note that for an Erdős-Réyni network with mixed synapses, this second stage of averaging is probabilistically unnecessary: for a large enough network random subsets of a single large network are statistically identical to random subsets drawn from several samples of full Erdős-Réyni networks (i.e., the network is self-averaging). However, this will not be true for networks with more structure, such as the Watts-Strogatz or Dale’s law networks we also considered, for which the second stage of averaging over the global network architecture is necessary to average over network configurations.

Series approximation for the mean field firing rates for the case of exponential nonlinearity ϕ(x) = e^x

The mean field firing rates for the hidden neurons are given by

$${\nu }_h={\lambda }_0\text{exp}({\mu }_h+\sum _{h^{\prime }}{\mathcal{J}}_{h,h^{\prime }}{\nu }_{h^{\prime }}),$$

where we focus specifically on the case of exponential nonlinearity ϕ(x) = exp(x). For this choice of nonlinearity, γ_h = ν_h, so we do not need to calculate a separate series for the gains.

This system of transcendental equations generally cannot be solved analytically. However, for small exp(μ_h) ≪ 1 we can derive, recursively, a series expansion for the firing rates. We first consider the case of μ_h = μ₀ for all hidden neurons h. Let ϵ = exp(μ₀). We may then write

$${\nu }_h={\lambda }_0ϵ\sum _{\ell =0}^{\infty }{a}_h^{\left(\ell \right)}{\left({\lambda }_0ϵ\right)}^{\ell }.$$

Plugging this into the mean field equation,

$$\begin{array}{cc}\hfill \sum _{\ell =0}^{\infty }{a}_h^{\left(\ell \right)}{\left({\lambda }_0ϵ\right)}^{\ell }& =\text{exp}\left(\sum _{h^{\prime }}{\mathcal{J}}_{h,h^{\prime }}\sum _{\ell =0}^{\infty }{a}_{h^{\prime }}^{\left(\ell \right)}{\left({\lambda }_0ϵ\right)}^{\ell +1}\right)\hfill \\ & =1+\sum _{m=1}^{\infty }\frac{1}{m!}(\sum _{h^{\prime }}{\mathcal{J}}_{h,h^{\prime }}\sum _{\ell =0}^{\infty }{a}_{h^{\prime }}^{\left(\ell \right)}{\left({\lambda }_0ϵ\right)}^{\ell +1}{)}^m\hfill \\ & =1+\sum _{m=1}^{\infty }\frac{1}{m!}\sum _{{\ell }_1,\dots ,{\ell }_m,h_1^{\prime },\dots ,h_m^{\prime }}{\mathcal{J}}_{h,h_1^{\prime }}{a}_{h_1^{\prime }}^{\left({\ell }_1\right)}\dots {\mathcal{J}}_{h,h_m^{\prime }}{a}_{h_m^{\prime }}^{\left({\ell }_m\right)}{\left({\lambda }_0ϵ\right)}^{{\ell }_1+\cdots +{\ell }_m+m}\hfill \\ & =1+\sum _{\ell =1}^{\infty }\{\sum _{m=1}^{\infty }\frac{1}{m!}\sum _{{\ell }_1,\dots ,{\ell }_m,h_1^{\prime },\dots ,h_m^{\prime }}{\mathcal{J}}_{h,h_1^{\prime }}{a}_{h_1^{\prime }}^{\left({\ell }_1\right)}\dots {\mathcal{J}}_{h,h_m^{\prime }}{a}_{h_m^{\prime }}^{\left({\ell }_m\right)}{\delta }_{\ell ,{\ell }_1+\cdots +{\ell }_m+m}\}{\left({\lambda }_0ϵ\right)}^{\ell }.\hfill \end{array}$$

Thus, matching powers of λ₀ϵ on the left and right hand sides, we find $a_h^{\left(0\right)}=1$ and

$$a_h^{\left(\ell \right)}=\sum _{m=1}^{\infty }\frac{1}{m!}\sum _{{\ell }_1,\dots ,{\ell }_m,h_1^{\prime },\dots ,h_m^{\prime }}{\mathcal{J}}_{h,h_1^{\prime }}{a}_{h_1^{\prime }}^{\left({\ell }_1\right)}\dots {\mathcal{J}}_{h,h_m^{\prime }}{a}_{h_m^{\prime }}^{\left({\ell }_m\right)}{\delta }_{\ell ,{\ell }_1+\cdots +{\ell }_m+m}$$

for ℓ > 0.

For ℓ = 1, the sum in m truncates at m = 1 (as ${\delta }_{\ell ,{\ell }_1+\cdots +{\ell }_m+m}$ is zero for m > ℓ, as all indices are positive). Thus,

$$\begin{array}{cc}\hfill a_h^{\left(1\right)}& =\sum _{h_1^{\prime }}{\mathcal{J}}_{h,h_1^{\prime }},\hfill \\ \hfill a_h^{\left(2\right)}& =\sum _{h_1^{\prime },h_2^{\prime }}\{{\mathcal{J}}_{h,h_1^{\prime }}{\mathcal{J}}_{h_1^{\prime },h_2^{\prime }}+\frac{1}{2}{\mathcal{J}}_{h,h_1^{\prime }}{\mathcal{J}}_{h,h_2^{\prime }}\},\hfill \\ \hfill a_h^{\left(3\right)}& =\sum _{h_1^{\prime },h_2^{\prime },h_3^{\prime }}\{{\mathcal{J}}_{h,h_1^{\prime }}{\mathcal{J}}_{h_1^{\prime },h_2^{\prime }}{\mathcal{J}}_{h_2^{\prime },h_3^{\prime }}+\frac{1}{2}{J}_{h,h_1^{\prime }}{\mathcal{J}}_{h_1^{\prime },h_2^{\prime }}{\mathcal{J}}_{h_1^{\prime },h_3^{\prime }}+{\mathcal{J}}_{h,h_1^{\prime }}{\mathcal{J}}_{h,h_2^{\prime }}{\mathcal{J}}_{h_2^{\prime },h_3^{\prime }}\hfill \\ & \hfill +\frac{1}{3!}{\mathcal{J}}_{h,h_1^{\prime }}{\mathcal{J}}_{h,h_2^{\prime }}{\mathcal{J}}_{h,h_3^{\prime }}\}.\end{array}$$

With this we have calculated the firing rates to $\mathcal{O}\left({ϵ}^4\right)$.

The analysis can be straightforwardly extended to the case of heterogeneous μ_h, though it becomes more tedious to compute terms in the (now multivariate) series. Assuming ϵ_h ≡ exp(μ_h) ≪ 1 for all h, to $\mathcal{O}\left({ϵ}^3\right)$ we find

$${\nu }_h={\lambda }_0{ϵ}_h(1+\sum _{h^{\prime }}{\mathcal{J}}_{h,h^{\prime }}{\lambda }_0{ϵ}_{h^{\prime }}+\sum _{h_1^{\prime },h_2^{\prime }}\{{\mathcal{J}}_{h,h_1^{\prime }}{\mathcal{J}}_{h_1^{\prime },h_2^{\prime }}+\frac{1}{2}{\mathcal{J}}_{h,h_1^{\prime }}{\mathcal{J}}_{h,h_2^{\prime }}\}{\lambda }_0{ϵ}_{h_1^{\prime }}{\lambda }_0{ϵ}_{h_2^{\prime }}+\dots ).$$

Variance of the effective coupling to second order in N_rec/N & fourth order in ${\lambda }_0{J}_0{e}^{{\mu }_0}$ (exponential nonlinearity)

To estimate the strength of the hidden paths, we would like to calculate the variance of the effective coupling ${\mathcal{J}}_{r,r^{\prime }}^{\text{eff}}$ and compare its strength to the variance of the direct couplings ${\mathcal{J}}_{r,r^{\prime }}$, where ${\mathcal{J}}_{r,r^{\prime }}^{\text{eff}}\equiv {\int }_0^{\infty }dt{J}_{r,r^{\prime }}^{\text{eff}}\left(t\right)$ and ${\mathcal{J}}_{r,r^{\prime }}\equiv {\int }_0^{\infty }dt{J}_{r,r^{\prime }}\left(t\right)$, as in the main text.

We assume that the synaptic weights ${\mathcal{J}}_{i,j}$ are independently and identically distributed with zero mean and variance $\text{var}\left(\mathcal{J}\right)=p\frac{J_0^2}{{\left(pN\right)}^{2a}}$ for i ≠ j, where a = 1 corresponds to weak coupling and a = 1/2 corresponds to strong coupling. We assume no self-couplings, ${\mathcal{J}}_{i,i}=0$ for all neurons i. The overall factor of p in $\text{var}\left[\mathcal{J}\right]$ comes from the sparsity of the network. For example, for normally distributed non-zero weights with variance $J_0^2/N^{2a}$, the total probability for every connection in the network is

$${\rho }_{ER\times J}\left(\mathcal{J}\right)=(1-p)\delta \left(\mathcal{J}\right)+p\frac{\text{exp}(-\frac{N^{2a}}{2}\frac{{\mathcal{J}}^2}{J_0^2})}{\sqrt{2\pi J_0^2/N^{2a}}}.$$

Because the ${\mathcal{J}}_{i,j}$ are i.i.d., the mean of ${\mathcal{J}}_{r,r^{\prime }}^{\text{eff}}$:

$$\begin{array}{cc}\hfill \overline{{\mathcal{J}}_{r,r^{\prime }}^{\text{eff}}}& =\overline{{\mathcal{J}}_{r,r^{\prime }}}+\sum _{h,h^{\prime }}\overline{{\mathcal{J}}_{r,h}{\widehat{\Gamma }}_{h,h^{\prime }}{\mathcal{J}}_{h^{\prime },r^{\prime }}}\hfill \\ & =0+\sum _{h,h^{\prime }}\overline{{\mathcal{J}}_{r,h}}\overline{{\widehat{\Gamma }}_{h,h^{\prime }}}\overline{{\mathcal{J}}_{h^{\prime },r^{\prime }}}\hfill \\ & =0,\hfill \end{array}$$

where we used the fact that ${\widehat{\Gamma }}_{h,h^{\prime }}\equiv {\widehat{\Gamma }}_{h,h^{\prime }}\left(0\right)$ depends only on the hidden neuron couplings ${\mathcal{J}}_{h,h^{\prime }}$, which are independent of the couplings to the recorded neurons, ${\mathcal{J}}_{r,h}$ and ${\mathcal{J}}_{h^{\prime },r^{\prime }}$. This holds for any pair of neurons (r, r′), including r = r′ because of the assumption of no self-coupling.

The variance of ${\mathcal{J}}_{r,r^{\prime }}^{\text{eff}}$ is thus equal to the mean of its square, for r ≠ r′,

$$\begin{array}{cc}\hfill \text{var}\left[{\mathcal{J}}_{r,r^{\prime }}^{\text{eff}}\right]& =\overline{{\left({\mathcal{J}}_{r,r^{\prime }}^{\text{eff}}\right)}^2}\hfill \\ & =\overline{{\left({\mathcal{J}}_{r,r^{\prime }}\right)}^2}+\overline{{\left(\sum _{h,h^{\prime }}{\mathcal{J}}_{r,h}{\widehat{\Gamma }}_{h,h^{\prime }}{\mathcal{J}}_{h^{\prime },r^{\prime }}\right)}^2}\hfill \\ & =\text{var}\left[\mathcal{J}\right]+\sum _{h_1,h_1^{\prime },h_2,h_2^{\prime }}\overline{{\mathcal{J}}_{r,h_1}{\widehat{\Gamma }}_{h_1,h_1^{\prime }}{\mathcal{J}}_{h_1^{\prime },r^{\prime }}{\mathcal{J}}_{r,h_2}{\Gamma }_{h_2,h_2^{\prime }}{\mathcal{J}}_{h_2^{\prime },r^{\prime }}}\hfill \\ & =\text{var}\left[\mathcal{J}\right]+\sum _{h,h^{\prime }}\overline{{\mathcal{J}}_{r,h}^2}\overline{{\widehat{\Gamma }}_{h,h^{\prime }}^2}\overline{{\mathcal{J}}_{h^{\prime },r^{\prime }}^2}\hfill \\ & =\text{var}\left[\mathcal{J}\right]+\text{var}{\left[\mathcal{J}\right]}^2\sum _{h,h^{\prime }}\overline{{\widehat{\Gamma }}_{h,h^{\prime }}^2}\hfill \end{array}$$

In this derivation, we used the fact that $\overline{{\mathcal{J}}_{r,h_1}{\mathcal{J}}_{r,h_2}}=\overline{{\mathcal{J}}_{r,h_1}^2}{\delta }_{h_1,h_2}$ due to the fact that the synaptic weights are uncorrelated. We now need to compute $\overline{{\widehat{\Gamma }}_{h,h^{\prime }}^2}$. This is intractable in general, so we will resort to calculating this in a series expansion in powers of ϵ ≡ exp(μ₀) for the exponential nonlinearity model. Our result will also turn out to be an expansion in powers of J₀ and 1 − f ≡ N_hid/N.

The lowest order approximation is obtained by the approximation ν_h ≈ λ₀ϵ and Γ_h,h′ ≈ ν_hδ_h,h′, yielding

$$\begin{array}{cc}\hfill \frac{\text{var}\left[{\mathcal{J}}_{r,r^{\prime }}^{\text{eff}}\right]}{\text{var}\left[\mathcal{J}\right]}& =1+{\left({\lambda }_0ϵ\right)}^2{N}_{\text{hid}}\text{var}\left[\mathcal{J}\right]\hfill \\ & =1+{\left({\lambda }_0{J}_0ϵ\right)}^2(1-f)\frac{1}{{\left(pN\right)}^{2a-1}}.\hfill \end{array}$$ (7)

This result varies linearly with f, while numerical evaluation of the variance shows obvious curvature for f ≪ 1 and J₀ ≲ 1, so we need to go to higher order. This becomes tedious very quickly, so we will only work to $\mathcal{O}\left({ϵ}^4\right)$ (it turns out $\mathcal{O}\left({ϵ}^3\right)$ corrections vanish).

We calculate $\overline{{\widehat{\Gamma }}_{h,h^{\prime }}^2}$ using a recursive strategy, though we could also use the path-length series expression for ${\widehat{\Gamma }}_{h,h^{\prime }}\left(\omega \right)$, keeping terms up to fourth order in ϵ. We begin with the expression

$${\widehat{\Gamma }}_{h,h^{\prime }}={\nu }_h{\delta }_{h,h^{\prime }}+\sum _{h^{\prime \prime }}{\nu }_h{\mathcal{J}}_{h,h^{\prime \prime }}{\widehat{\Gamma }}_{h^{\prime \prime },h^{\prime }}$$

and plug it into itself until we obtain an expression to a desired order in ϵ. In doing so, we note that ${\nu }_h\sim \mathcal{O}\left(ϵ\right)$, so we will first work to fourth order in ν_h, and then plug in the series for ν_h in powers of ϵ.

We begin with

$$\begin{array}{cc}\hfill {\widehat{\Gamma }}_{h,h^{\prime }}^2& ={\nu }_h^2{\delta }_{h,h^{\prime }}+2{\delta }_{h,h^{\prime }}\sum _{h^{\prime \prime }}{\nu }_h^2{\mathcal{J}}_{h,h^{\prime \prime }}{\widehat{\Gamma }}_{h^{\prime \prime },h^{\prime }}+{\left(\sum _{h^{\prime \prime }}{\nu }_h{\mathcal{J}}_{h,h^{\prime \prime }}{\widehat{\Gamma }}_{h^{\prime \prime },h^{\prime }}\right)}^2\hfill \\ & ={\nu }_h^2{\delta }_{h,h^{\prime }}+2{\delta }_{h,h^{\prime }}\sum _{h^{\prime \prime }}{\nu }_h^2{\mathcal{J}}_{h,h^{\prime \prime }}{\widehat{\Gamma }}_{h^{\prime \prime },h^{\prime }}+\sum _{h_1,h_2}{\nu }_h^2{\mathcal{J}}_{h,h_1}{\mathcal{J}}_{h,h_2}{\widehat{\Gamma }}_{h_1,h^{\prime }}{\widehat{\Gamma }}_{h_2,h^{\prime }}\hfill \\ & \approx {\nu }_h^2{\delta }_{h,h^{\prime }}+2{\delta }_{h,h^{\prime }}\sum _{h^{\prime \prime }}{\nu }_h^2{\mathcal{J}}_{h,h^{\prime \prime }}\{{\nu }_{h^{\prime \prime }}{\delta }_{h^{\prime \prime },h^{\prime }}+\sum _{h_2}{\nu }_{h^{\prime \prime }}{\mathcal{J}}_{h^{\prime \prime },h_2}{\nu }_{h_2}{\delta }_{h_2,h^{\prime }}\}\hfill \\ & +\sum _{h_1,h_2}{\nu }_h^2{\nu }_{h^{\prime }}^2{\mathcal{J}}_{h,h_1}{\mathcal{J}}_{h,h_2}{\delta }_{h_1,h^{\prime }}{\delta }_{h_2,h^{\prime }}\hfill \\ & ={\nu }_h^2{\delta }_{h,h^{\prime }}+2{\delta }_{h,h^{\prime }}\{{\nu }_h^2{\nu }_{h^{\prime }}{\mathcal{J}}_{h,h^{\prime }}+\sum _{h^{\prime \prime }}{\nu }_h^2{\nu }_{h^{\prime \prime }}{\mathcal{J}}_{h,h^{\prime \prime }}{\mathcal{J}}_{h^{\prime \prime },h^{\prime }}{\nu }_{h^{\prime }}\}+{\nu }_h^2{\nu }_{h^{\prime }}^2{\mathcal{J}}_{h,h^{\prime }}^2\hfill \\ & =\{{\nu }_h^2+2{\nu }_h^2{\nu }_{h^{\prime }}{\mathcal{J}}_{h,h^{\prime }}+2\sum _{h^{\prime \prime }}{\nu }_h^2{\nu }_{h^{\prime \prime }}{\mathcal{J}}_{h,h^{\prime \prime }}{\mathcal{J}}_{h^{\prime \prime },h^{\prime }}{\nu }_{h^{\prime }}\}{\delta }_{h,h^{\prime }}+{\nu }_h^2{\nu }_{h^{\prime }}^2{\mathcal{J}}_{h,h^{\prime }}^2\hfill \\ & =\{{\nu }_h^2+2\sum _{h^{\prime \prime }}{\nu }_h^3{\nu }_{h^{\prime \prime }}{\mathcal{J}}_{h,h^{\prime \prime }}{\mathcal{J}}_{h^{\prime \prime },h}\}{\delta }_{h,h^{\prime }}+{\nu }_h^2{\nu }_{h^{\prime }}^2{\mathcal{J}}_{h,h^{\prime }}^2\hfill \end{array}$$

The third order term ${\nu }_h^3{\mathcal{J}}_{h,h^{\prime }}{\delta }_{h,h^{\prime }}$ vanished because we assume no self-couplings. We have obtained ${\widehat{\Gamma }}_{h,h^{\prime }}^2$ to fourth order in ν_h; now we need to plug in the series expression for ν_h to obtain the series in powers of λ₀ϵ. We will do this order by order in ν_h. The easiest terms are the fourth order terms, as

$${\nu }_h^2{\nu }_{h^{\prime }}^2\approx {\left({\lambda }_0ϵ\right)}^4\text{and}{\nu }_h^3{\nu }_{h^{\prime \prime }}\approx {\left({\lambda }_0ϵ\right)}^4.$$

The second order term is

$$\begin{array}{cc}\hfill {\nu }_h^2& \approx {\left({\lambda }_0ϵ\right)}^2(1+\sum _{h_1}{\mathcal{J}}_{h,h_1}{\lambda }_0ϵ+\sum _{h_1,h_2}{a}_{h,h_1,h_2}^{\left(2\right)}{\left({\lambda }_0ϵ\right)}^2)\hfill \\ & \times (1+\sum _{h_1^{\prime }}{\mathcal{J}}_{h,h_1^{\prime }}{\lambda }_0ϵ+\sum _{h_1^{\prime },h_2^{\prime }}{a}_{h,h_1^{\prime },h_2^{\prime }}^{\left(2\right)}{\left({\lambda }_0ϵ\right)}^2)\hfill \\ & \approx {\left({\lambda }_0ϵ\right)}^2(1+2(\sum _{h_1}{\mathcal{J}}_{h,h_1}{\lambda }_0ϵ+\sum _{h_1,h_2}{a}_{h,h_1,h_2}^{\left(2\right)}{\left({\lambda }_0ϵ\right)}^2)+{\left(\sum _{h_1}{\mathcal{J}}_{h,h_1}{\lambda }_0ϵ\right)}^2)\hfill \\ & ={\left({\lambda }_0ϵ\right)}^2(1+2\sum _{h_1}{\mathcal{J}}_{h,h_1}{\lambda }_0ϵ+\sum _{h_1,h_2}\{2a_{h,h_1,h_2}^{\left(2\right)}+{\mathcal{J}}_{h,h_1}{\mathcal{J}}_{h,h_2}\left\}{\left({\lambda }_0ϵ\right)}^2\right),\hfill \end{array}$$

where $a_{h,h_1,h_2}^{\left(2\right)}={\mathcal{J}}_{h,h_1}{\mathcal{J}}_{h_1,h_2}+\frac{1}{2}{\mathcal{J}}_{h,h_1}{\mathcal{J}}_{h,h_2}$. We need the average $\overline{{\nu }_h^2}$. The third-order term will vanish upon averaging, and

$$\overline{2a_{h,h_1,h_2}^{\left(2\right)}+{\mathcal{J}}_{h,h_1}{\mathcal{J}}_{h,h_2}}=\overline{2{\mathcal{J}}_{h,h_1}{\mathcal{J}}_{h_1,h_2}+2{\mathcal{J}}_{h,h_1}{\mathcal{J}}_{h,h_2}}=2\text{var}\left[\mathcal{J}\right]{\delta }_{h_1,h_2}(1-{\delta }_{h,h_1}),$$

using the fact that synaptic weights are independent (giving the ${\delta }_{h_1,h_2}$ factor) and self-couplings are zero (giving the $1-{\delta }_{h,h_1}$ factor). We thus obtain

$$\overline{{\nu }_h^2}={\left({\lambda }_0ϵ\right)}^2+2{\left({\lambda }_0ϵ\right)}^4(N_{\text{hid}}-1)\text{var}\left[\mathcal{J}\right].$$

The first fourth order term in ${\widehat{\Gamma }}_{h,h^{\prime }}^2$, $2{\sum }_{h^{″}}{\nu }_h^3{\nu }_{h^{″}}{\mathcal{J}}_{h,h^{″}}{\mathcal{J}}_{h^{″},h}{\delta }_{h,h^{\prime }}$, will vanish upon averaging because matching indices requires h^′′ = h = h′ and we assume no self-couplings. The second fourth order term is ${\mathcal{J}}_{h,h^{\prime }}^2$, which averages to $\text{var}\left[\mathcal{J}\right](1-{\delta }_{h,h^{\prime }})$, where the factor of (1 − δ_h,h′) again accounts for the fact that this term does not contribute when h = h′ due to no self-couplings. We thus arrive at

$$\begin{array}{cc}\hfill \overline{{\widehat{\Gamma }}_{h,h^{\prime }}^2}& =({\left({\lambda }_0ϵ\right)}^2+2{\left({\lambda }_0ϵ\right)}^4(N_{\text{hid}}-1)\text{var}\left[\mathcal{J}\right]){\delta }_{h,h^{\prime }}+{\left({\lambda }_0ϵ\right)}^4\text{var}\left[\mathcal{J}\right](1-{\delta }_{h,h^{\prime }})\hfill \\ & =({\left({\lambda }_0ϵ\right)}^2+{\left({\lambda }_0ϵ\right)}^4(2N_{\text{hid}}-3)\text{var}\left[\mathcal{J}\right]){\delta }_{h,h^{\prime }}+{\left({\lambda }_0ϵ\right)}^4\text{var}\left[\mathcal{J}\right];\hfill \end{array}$$

Putting everything together,

$$\begin{array}{cc}\hfill \frac{\text{var}\left[{\mathcal{J}}_{r,r^{\prime }}^{\text{eff}}\right]}{\text{var}\left[\mathcal{J}\right]}& =1+\text{var}\left[\mathcal{J}\right]\sum _{h,h^{\prime }}\overline{{\widehat{\Gamma }}_{h,h^{\prime }}^2}\hfill \\ & =1+\text{var}\left[\mathcal{J}\right]\left[\sum _h\right\{{\left({\lambda }_0ϵ\right)}^2+{\left({\lambda }_0ϵ\right)}^4(2N_{\text{hid}}-3)\text{var}\left[\mathcal{J}\right]\}+\sum _{h,h^{\prime }}{\left({\lambda }_0ϵ\right)}^4\text{var}\left[\mathcal{J}\right]]\hfill \\ & =1+\text{var}\left[\mathcal{J}\right]\left[N_{\text{hid}}\right\{{\left({\lambda }_0ϵ\right)}^2+{\left({\lambda }_0ϵ\right)}^4(2N_{\text{hid}}-3)\text{var}\left[\mathcal{J}\right]\}+N_{\text{hid}}^2{\left({\lambda }_0ϵ\right)}^4\text{var}\left[\mathcal{J}\right]]\hfill \\ & =1+N_{\text{hid}}\text{var}\left[\mathcal{J}\right][{\left({\lambda }_0ϵ\right)}^2+{\left({\lambda }_0ϵ\right)}^4(2N_{\text{hid}}-3)\text{var}\left[\mathcal{J}\right]+N_{\text{hid}}{\left({\lambda }_0ϵ\right)}^4\text{var}\left[\mathcal{J}\right]]\hfill \\ & =1+N_{\text{hid}}\text{var}\left[\mathcal{J}\right][{\left({\lambda }_0ϵ\right)}^2+{\left({\lambda }_0ϵ\right)}^4(3-\frac{3}{N_{\text{hid}}}\left)N_{\text{hid}}\text{var}\left[\mathcal{J}\right]\right]\hfill \end{array}$$

For weak coupling, this tends to 1 in the N ≫ 1 limit, as $N_{\text{hid}}\text{var}\left[\mathcal{J}\right]=(1-f)J_0^2/N\to 0$, for fixed fraction of observed neurons f = N_rec/N. For strong coupling, $N_{\text{hid}}\text{var}\left[\mathcal{J}\right]=(1-f)J_0^2$, which is constant as N → ∞, and hence

$$\frac{\text{var}\left[{\mathcal{J}}_{r,r^{\prime }}^{\text{eff}}\right]}{\text{var}\left[\mathcal{J}\right]}=1+{\left({\lambda }_0{J}_0ϵ\right)}^2(1-f)+3{\left({\lambda }_0{J}_0ϵ\right)}^4{(1-f)}^2+o\left({\left({\lambda }_0{J}_0ϵ\right)}^4{(1-f)}^2\right),$$ (8)

where we have used little-o notation to denote that there are higher order terms dominated by (λ₀ J₀ϵ)⁴(1 − f)². With this expression, we have improved on our approximation of the relative variance of the effective coupling to the true coupling; however, the neglected higher order terms still become significant as f → 0 and J₀ → 1, indicating that hidden paths have a significant impact when synaptic strengths are moderately strong and only a small fraction of the neurons have been observed.

Because the synaptic weights ${\mathcal{J}}_{i,j}$ are independent, we may rewrite Eq (8) as

$$\frac{\text{var}[{\mathcal{J}}_{r,r^{\prime }}^{\text{eff}}-{\mathcal{J}}_{r,r^{\prime }}]}{\text{var}\left[\mathcal{J}\right]}\approx {\left({\lambda }_0{J}_0ϵ\right)}^2(1-f)+3{\left({\lambda }_0{J}_0ϵ\right)}^4{(1-f)}^2;$$

or, in terms of the ratio of standard deviations,

$$\frac{\sigma [{\mathcal{J}}_{r,r^{\prime }}^{\text{eff}}-{\mathcal{J}}_{r,r^{\prime }}]}{\sigma \left[\mathcal{J}\right]}\approx \left({\lambda }_0{J}_0ϵ\right)\sqrt{1-f}(1+\frac{3}{2}{\left({\lambda }_0{J}_0ϵ\right)}^2(1-f)),$$

where we used the approximation $\sqrt{1+x}\approx 1+x/2$ for x small.

In the main text, we plotted results for N = 1000 total neurons (Fig 5A). For strongly coupled networks, the results should only depend on the fraction of observed neurons, f = N_rec/N, while for weak coupling the results do depend on the absolute number N. To demonstrate this, in Fig 9 we remake Fig 5 for N = 100 neurons. We see that the strongly coupled results have not been significantly altered, whereas the weakly coupled results yield stronger deviations (as the deviations are $\mathcal{O}(1/\sqrt{N})$).

Fig 9. Same as Fig 5, but for N = 100 neurons and N_rec = {1, 11, 21, 31, 41, 51, 61, 71, 81, 91, 99} recorded neurons.

Because we plot the relative deviations of the coupling strength against the fraction of observed neurons, the curves for the strongly coupled case are the same as for N = 1000, as expected, while the weakly coupled case yields stronger deviations.

Supporting information

S1 Fig. Empirical estimates of average neuron firing rates from simulations plotted against mean firing rates predicted by mean field theory.

The fact that the data lies along the identity line demonstrates validity of the mean field theory approximation up to J₀ = 1.0.

(EPS)

S2 Fig

Top row: scatter plot comparing ν_h, the mean field firing rates of the hidden neurons in the absence of recorded neurons, to empirically estimated firing rates in simulations of the full network, for four different values of typical synaptic strength, J₀ = 0.25, 0.5, 0.75, and 1.0. The data lie along the identity line, demonstrating a strong correlation between ν_h and the empirical data. However, the spread of data around the identity line indicates that deviations of the mean firing rates away from ν_h, caused by coupling to the recorded neurons, is significant. Bottom row: Comparison of the first order approximation of the firing rates of hidden neurons, which accounts for the effects of recorded neurons, to the empirical rates. The data lie tightly along the identity with very little dispersion, demonstrating that higher order spike filtering is unnecessary even up to J₀ = 1.0, for N_rec = 100.

(EPS)

S3 Fig. Same as S2 Fig but for N_rec = 500 recorded neurons out of a total of N = 1000.

Demonstrates validity of linear approximation (neglecting higher order spike filtering) up to J₀ = 1.0, for N_rec = 500. The zeroth order approximation (top row) is quite poor, indicating the necessity of accounting for feedback from the recorded neurons. This first order approximation (bottom row) lies tightly along the identity line, indicating that even when the recorded and hidden populations are of comparable size, higher order spike filtering may not be significant. However, there appears to be some deviation for J₀ = 1.0, indicating that accounting for higher order spike filtering may be beneficial in this parameter regime.

(EPS)

S1 Table. Network activity simulation parameter values.

(PDF)

S1 Text. Supporting information.

(PDF)

Data Availability

Scripts and data files used to produce figures are available online at https://github.com/bradenbrinkman/hiddenneurons.

Funding Statement

Support provided by the Sackler Scholar Program in Integrative Biophysics (BAWB), Collaborative Research in Computational Neuroscience (CRCNS, https://crcns.org/) grant DMS-1208027 (ESB, FR), National Institute of Health (NIH, https://www.nih.gov/) grant EY11850 and the Howard Hughes Medical Institute (http://www.hhmi.org) (FR). This work was partially based on work supported by the Center for Brains, Minds and Machines (CBMM, https://cbmm.mit.edu/), funded by National Science Foundation (NSF, https://www.nsf.gov) STC award CCF-1231216 (MAB). MAB and ESB wish to thank the Allen Institute for Brain Science founders, Paul G. Allen and Jody Allen, for their vision, encouragement, and support. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.