Adjusted Regularization of Cortical Covariance

Abstract

It is now common to record dozens to hundreds or more neurons simultaneously, and to ask how the network activity changes across experimental conditions. A natural framework for addressing questions of functional connectivity is to apply Gaussian graphical modeling to neural data, where each edge in the graph corresponds to a non-zero partial correlation between neurons. Because the number of possible edges is large, one strategy for estimating the graph has been to apply methods that aim to identify large sparse effects using an L₁ penalty. However, the partial correlations found in neural spike count data are neither large nor sparse, so techniques that perform well in sparse settings will typically perform poorly in the context of neural spike count data. Fortunately, the correlated firing for any pair of cortical neurons depends strongly on both their distance apart and the features for which they are tuned. We introduce a method that takes advantage of these known, strong effects by allowing the penalty to depend on them: thus, for example, the connection between pairs of neurons that are close together will be penalized less than pairs that are far apart. We show through simulations that this physiologically-motivated procedure performs substantially better than off-the-shelf generic tools, and we illustrate by applying the methodology to populations of neurons recorded with multielectrode arrays implanted in macaque visual cortex areas V1 and V4.

1. Introduction

The rapid growth in the number of neurons being recorded simultaneously (Ahrens et al., 2015; Alivisatos et al., 2013; Kerr and Denk, 2008; Kipke et al., 2008) creates an urgent need for statistical procedures that can identify the structure of covariation in neural network activity (Shadlen and Newsome, 1998; Brown et al., 2004; Cunningham and Yu, 2014; Stevenson and Kording, 2011; Song et al., 2013; Yatsenko et al., 2015; Cohen and Maunsell, 2009; Cohen and Kohn, 2011; Efron et al., 2001; Kelly and Kass, 2012; Mitchell et al., 2009; Vinci et al., 2016). An appealing approach to network analysis begins by representing multivariate activity as a graph, that is, a set of nodes together with a specification of which nodes are connected by edges (Bassett and Sporns, 2017). In the case of multi-neuron recordings, each node would correspond to a neuron. Because these recordings are typically noisy, capturing in full detail the interactions among neurons, which can occur at multiple timescales, is very difficult. An initial simplification is to consider the vector of spike counts, within a time interval of several hundred milliseconds, to be a Gaussian random vector X whose covariance matrix Σ defines a graph based on the inverse matrix Ω = Σ⁻¹ (assuming Σ is invertible). Specifically, the edges in the graph correspond to the non-zero off-diagonal elements of Ω = [ω_ij], that is, an edge between nodes i and j is absent if and only if ω_ij·= 0. Furthermore, ω_ij= 0 if and only if the corresponding partial correlation satisfies ρ_ij = 0, and ρ_ij = 0 if and only if the i and j nodes are independent conditionally on all the other nodes, that is in our case, an edge exists between pairs of neurons that have a unique component of covariation that is not associated with all the other neurons. The time-scale, imposed by the scientific questions of interest or by objective choice of bin size, such as experimental task durations, affects the elements of Σ and hence the conclusions one might draw from the graph.

Such Gaussian graphical models are widely applied and studied (Murphy, 2012). However, even this simple case becomes challenging as the number of neurons grows: although, for typical spike count data, estimation of any single correlation coefficient may incur a relatively small error, compounding thousands of such small errors produces an unstable estimate of the matrix Σ. Thus, some form of regularization in the estimation of Σ is usually applied. In recent years, the most commonly-applied form of covariance regularization has been the Graphical lasso (Glasso) (Yuan and Lin, 2007; Friedman et al., 2008; Banerjee et al., 2008; Rothman et al., 2008; Mazumder and Hastie, 2012). To define it, we write the Gaussian likelihood function as$L\left(\Omega ;{\mathbb{X}}_n\right)$, where ${\mathbb{X}}_n=\left\{X^{(1)},\dots ,X^{(n)}\right\}$ with X^(r) representing the Gaussian random vector of spike counts of d neurons on trial r, for r = 1,... ,n, we assume the number of non-zero elements of Ω is comparatively small (so the matrix is sparse), and we maximize the penalized log likelihood function

$$\widehat{\Omega }(\lambda )=\arg \underset{\Omega \succ 0}{\max }\log L\left(\Omega ;{\mathbb{X}}_n\right)-\lambda \Vert \Omega {\Vert }_1,$$ (1)

where $\Vert \Omega {\Vert }_1={\sum }_{i,j=1}^d|{\omega }_{ij}|$ is the L₁ matrix norm of Ω (with or without the diagonal entries) (Yuan and Lin, 2007; Friedman et al., 2008; D’Aspremont et al., 2008; Rothman et al., 2008; Mazumder and Hastie, 2012) and Ω is assumed to be positive definite. The magnitude of the regularization parameter λ > 0 controls the degree of sparsity.

Glasso performs well in the presence of a small number of large effects, i.e., a small number of large non-zero offdiagonal elements of Ω, which corresponds to large signals relative to noise. In microelectrode array recordings, however, we expect instead to find a large number of small and noisy effects. Indeed, using realistic settings for a numerical simulation (spike counts on coarse time scales 300 ~ 1000ms) we found that Glasso and existing variants perform poorly (see Fig. 1 and the simulation section). We therefore sought to enhance off-the-shelf regularization by including information that is specific to the neural setting. In this paper we introduce a variant of Glasso that takes advantage of known neurophysiology: the covariation of pairs of neurons’ spike counts depends on their distance apart and their tuning curve correlation (Smith and Kohn, 2008; Smith and Sommer, 2013; Goris et al., 2014; Vinci et al., 2016). We use a Bayesian formulation of the problem to allow the penalty to vary with each neuron pair, separately, so that edges can become less likely to be placed between neurons as their distance apart increases or their tuning curve correlation decreases - the relationship of the penalty to these two covariates is learned from the data. We call the method Graphical lasso with Adjusted Regularization (GAR). Fig. 1 illustrates the typical benefit of applying GAR in comparison with Glasso. We provide an extensive simulation study to compare GAR with several variants of Glasso that have appeared in the literature. We also show how the Bayesian approach provides an elegant framework to construct the graph, in a manner similar to false discovery rate regression (Scott et al., 2015). Finally, we apply the method to populations of neurons recorded with multielectrode arrays implanted in macaque visual cortex areas V1 and V4 to illustrate aspects of network behavior that can be discovered with this approach.

Fig. 1.

Glasso and GAR graph estimation performances for a simulated network. The true graph contains d = 49 neurons and 118 edges, and is based on parameter settings derived from cortical data. Blue and red edges denote positive and negative partial correlations and the size of each node is proportional to the number of its connections. The GAR and Glasso estimates displayed are based on a sample size of n = 200 (see simulation Section 2.4 for details). For clarity, we show only the 118 strongest estimated connections (Glasso estimated 642 edges and GAR 204). ROC curves were obtained for 50 repeat simulated data; all 50 curves are plotted as thin lines and their averages as thick lines. The average area under the curves (AUC) is written in parentheses ± 2 simulation standard error: GAR is more accurate than Glasso.

2. Results

We first describe several penalized likelihood methods for estimating Ω, including GAR (Sections 2.1 and 2.2). We then explain how to infer a neuronal network connectivity graph from the estimate of Ω (Section 2.3). Finally, we illustrate the properties of these methods in an extensive simulation study (Section 2.4), and we estimate the connectivity graphs of populations of neurons recorded with multielectrode arrays implanted in macaque visual cortex areas V1 and V4 (Section 2.5).

2.1. Estimating the precision matrix Ω

The Glasso estimate of Ω is obtained in Eq. (1) where

$$\log L\left(\Omega ;{\mathbb{X}}_n\right)=\frac{n}{2}(\log \det (\Omega )-\mathrm{trace}(\widehat{\Sigma }\Omega )-d\log (2\pi )),$$ (2)

$\widehat{\Sigma }$is the sample covariance matrix of${\mathbb{X}}_n$, and λ > 0 is chosen according to one of several possible criteria (Yuan and Lin, 2007; Liu et al., 2010; Foygel and Drton, 2010). Here we use the criterion of Fan et al. (2009), taking λ to minimize the cross-validated risk of estimating Ω with$\widehat{\Omega }(\lambda )$, calculated as the average loss, $-\log L({\widehat{\Omega }}_{\text{train}}(\lambda );{\mathbb{X}}_{\text{test}})$, across 500 random (90%, 10%) splits of the data ${\mathbb{X}}_n=\left({\mathbb{X}}_{\text{train}},{\mathbb{X}}_{\text{test}}\right)$ where ${\widehat{\Omega }}_{\text{train}}(\lambda )$ is the estimate in Eq. (1) based on${\mathbb{X}}_{\text{train}}$.

A variant is the adaptive Glasso (AGlasso) (Fan et al., 2009) given by

$$\widehat{\Omega }(\lambda )=\arg \underset{\Omega \succ 0}{\max }\log L\left(\Omega ;{\mathbb{X}}_n\right)-\lambda \Vert Q\odot \Omega {\Vert }_1,$$ (3)

that is Eq. (1) but with the penalty ‖Ω‖₁ replaced by ‖Q⊙Ω‖₁, where ⊙ denotes the entry-wise matrix multiplication and Q is a matrix containing values inversely related to the absolute values of an initial estimate $\widehat{\Omega }$, for example $Q=[{|{\widehat{\omega }}_{ij}|}^{-1/2}]$ with $\widehat{\Omega }=\left[{\widehat{\omega }}_{ij}\right]$ the inverse of the sample covariance matrix $\widehat{\Sigma }$ (Fan et al., 2009). Hence, the AGlasso aims to penalize less/more the large/small entries of Ω. However, a reliable initial estimate of Ω is not always available; for instance, when the number of neurons d is greater than the number of trials n, $\widehat{\Sigma }$ requires modifications to be inverted, such as adding a small constant to its diagonal to make it positive definite. Because the AGlasso estimate depends on the quality of the initial estimate of Ω, it does not necessarily outperform the Glasso estimate.

The Bayesian Adaptive Glasso (BAGlasso; Wang (2012)) supplements the Gaussian likelihood with a prior distribution for Ω

$$\pi (\Omega |\Lambda )\propto {\displaystyle \prod _{i,j=1}^d{e}^{-{\lambda }_{ij}|{\omega }_{ij}|}}\times I(\Omega \succ 0)$$ (4)

where Λ = [λ_ij] is a symmetric matrix that contains a different penalty for each w_ij, to make it possible to penalize less/more the large/small w_ij. The data automatically tunes the penalties if we assume a sufficiently flexible hyperprior for Λ, for example independent Gamma distributions for each λ_ij (Wang, 2012). The BAGlasso estimate of Ω is taken to be the mean (posterior expectation) or the mode (maximum a posteriori, a.k.a. MAP) of $\pi (\Omega |{\mathbb{X}}_n)$, the posterior distribution of Ω. The BAGlasso estimator is not necessarily better than the Glasso or the AGlasso estimator because the added model flexibility also induces more variability. Note that the simple case where λ_ij· = λ for all (i, j) is known as the Bayesian Glasso (BGlasso; Wang (2012)); furthermore the BGlasso MAP estimate of Ω for a fixed λ is the Glasso estimate in Eq. (1).

The Glasso framework in Eq. (1) can also be extended into the Sparse-Low rank model (SPL)

$$(\widehat{S},\widehat{L})=\underset{\underset{\mathrm{rank}(L)\le q}{S-L\succ 0,}}{\arg \max }\log L\left(S-L;{\mathbb{X}}_n\right)-\lambda \Vert S{\Vert }_1.$$ (5)

where Ω = S — L ≻ 0 is assumed to be the combination of a sparse component S representing the dependence structure of the recorded neurons conditionally on all other recorded and latent neurons in the network, and a low-rank component $L\underset{\_}{\succ }0$ aimed at capturing the network effect of latent neurons on the recorded ones (Chandrasekaran et al., 2012; Giraud and Tsybakov, 2012; Yuan, 2012; Yatsenko et al., 2015). The parameters λ and q in Eq. (5) may be selected via cross-validation, analogously to Glasso. If L is set to zero, then Eq. (5) is equivalent to Glasso in Eq. (1). Partial correlations based on the component S alone would represent the conditional dependence structure of the observed neurons in an unknown larger network containing a set of unobserved units of intrisic dimensionality assumed to be smaller than q. However, for small sample sizes n and large numbers of neurons d, the sparse and low-rank components may become too expensive to estimate accurately so that SPL might perform no better than other methods (Giraud and Tsybakov, 2012; Yatsenko et al., 2015).

Proposed methods

The AGlasso and BAGlasso penalize less the Ω matrix entries that are anticipated to be larger, where the “anticipation” is explicitly garnered from an initial estimate of Ω for AGlasso, or implicitly data driven in the BAGlasso. Sometimes we have available additional variables that carry information about the strength of the dependence between neurons, for example inter-neuron distance and tuning curve correlation, which have been observed to regulate the shared activity of neuron pairs (Smith and Kohn, 2008; Smith and Sommer, 2013; Goris et al., 2014; Yatsenko et al., 2015; Vinci et al., 2016). Our proposed methods take advantage of these known, strong effects by allowing the penalties to depend on them: for example, pairs of neurons that are close together will be penalized less than pairs that are far apart (see data analysis, Section 2.5).

Let $W_{ij}\in {ℝ}^m$ be a vector of m auxiliary quantities that carry information about the strength of the dependence between neurons i and j. The Smooth Adaptive Glasso (SAGlasso) is a “smooth” variant of the AGlasso, where the entries q_ij of the weight matrix Q in Eq. (3) are taken to be smooth functions of W_ij. This smoothing will both reduce the noise in Q and introduce the information carried by W_ij.into the regularized estimation of Ω. To proceed, we regress $q_{ij}/\sqrt{q_{ii}{q}_{jj}}$ on W_ij using a local smoother such as smoothing splines or local polynomials, where we divide the q_ij by $\sqrt{q_{ii}{q}_{jj}}$ so that they are on the scale of the partial correlations

$${\rho }_{ij}=-\frac{{\omega }_{ij}}{\sqrt{{\omega }_{ii}{\omega }_{jj}}}.$$ (6)

To ensure that the resulting weights q_ij are positive, we either perform a Gamma regression (Algorithm 1, Appendix), or log transform $q_{ij}/\sqrt{q_{ii}{q}_{jj}}$ first if a Gaussian regression is used.

The Graphical lasso with Adjusted Regularization (GAR) is a variant of the BAGlasso, where we impose additional structure on λ_ij in Eq. (4). Specifically, we assume that the penalties are functions of the auxiliary variables W_ij according to:

$${\lambda }_{ij}={\alpha }_i{\alpha }_{j}g\left(W_{ij}\right),i<j,$$ (7)

where the function g(W_ij) is fitted to the data rather than pre-specified, and the a_i ‘s are positive parameters that mimic the scaling components ${\omega }_{ii}^{-1/2}$ in Eq. (6), so that g(W_ij) is on the scale of the partial correlation 𝜌_ij. We render g identifiable by setting ${\lambda }_{ii}={\alpha }_i^2$, which is reasonable since α_i mimics ${\omega }_{ii}^{-1/2}$, and the mean parameter of in Eq. (4) is ${\lambda }_{ii}^{-1}$. Then, while the BAGlasso postulates a hyperprior for Λ in Eq. (4), here we assume a hyperprior on the parameters of Λ in Eq. (7), that is on Θ = {α, g}, where α = (α₁,…,α_d), and g is the m-variate function of the auxiliary variables W. The GAR full Bayes estimate of Ω is taken to be the mean or the mode of$\pi (\Omega |{\mathbb{X}}_n,W)$, the posterior distribution of Ω given the data ${\mathbb{X}}_n$ and auxiliary covariates W = {W_ij}. Ideally, g should be as general a function as possible. However, the model is complex enough to make the posterior difficult to calculate or simulate from so we opted to use the simplest of non-parametric functions: a step function. In Section 2.2.2 we describe a Gibbs sampler to simulate from$\pi (\Omega ,\Theta |{\mathbb{X}}_n,W)$, and thus obtain sample mean or mode estimates of Ω. An alternative that allows a general form of g is to take an empirical Bayes approach to obtain a point estimate of Θ as the maximizer of the likelihood of Θ given ${\mathbb{X}}_n$ and W, and calculate the posterior distribution of Ω conditional on that estimate, that is$\pi (\Omega |{\mathbb{X}}_n,W,\widehat{\Theta })$, instead of calculating the full Bayes posterior. We implement this approach in Section 2.2.3 and apply it to our data in Section 2.5, taking 9 to be a regression spline with knots at the quartiles of the auxiliary variables. In the end, we have the full Bayes and the empirical Bayes GAR variants, and we can take the estimate of Ω to be either the mean or the mode of the corresponding posterior distribution. In our simulations (Section 2.4) we show that the improvement provided by GAR can be substantial compared to the competing methods, especially for values of d and n typically found in experimental neural data.

The proposed and existing methods considered here are similarly scalable. All methods aim to estimate the d(d + 1)/2 parameters of the precision matrix Ω, but their regularizations have different complexities: Glasso uses a single regularization parameter, λ in Eq. 1, and SPL uses two, λ and q in Eq. 5, while AGlasso and BAGlasso allow d(d + 1)/2 different penalties across the entries of Ω. SAGlasso and GAR also allow a different penalty for each entry of Ω, but their effective number is smaller than d(d + 1)/2. Indeed, the SAGlasso penalties all depend on W through a regression function that depends only on a limited number of regression coefficients. For GAR, all penalties are functions only of g and the d parameters αi (Eq. 7), where g is a regression function with a few degrees of freedom, e.g. splines with 3 or 4 knots. Morever, SAGlasso and GAR gain statistical efficiency when the auxiliary variables W are informative.

Finally, we note that a simple way to combine GAR with SPL (GAR-SPL) consists of replacing the penalty λ‖S‖₁ in Eq. (5) by$\xi \Vert \widehat{\Lambda }\odot S{\Vert }_1,$, where $\widehat{\Lambda }$ is a GAR estimate of$\Lambda |\left({\mathbb{X}}_n,W\right)$, and ξ and q are selected via cross-validation. In our simulations, GAR-SPL outperformed SPL but not GAR. A full Bayesian treatment of GAR-SPL, where the penalty matrix is estimated in direct combination with S rather than Ω, might provide a better performance; this is a topic of future research.

2.2. GAR estimation

In Sections 2.2.2 and 2.2.3 we describe Full and Empirical Bayes implementations of GAR, which both involve a data augmentation sampler that we present in Section 2.2.1. Algorithms and details are in Appendix.

2.2.1. Data augmentation

Both Full and Empirical Bayes implementations of GAR involve drawing samples from the posterior distribution

$$\pi (\Omega |{\mathbb{X}}_n,\Lambda )\propto \pi (\Omega |\Lambda )\times L\left(\Omega ;{\mathbb{X}}_n\right),$$ (8)

where the likelihood $L\left(\Omega ;{\mathbb{X}}_n\right)$ and the prior π(Ω | Λ) are defined in Eqs. (2) and (4). We proceed using a data augmentation strategy (Wang, 2012) where we introduce the nuisance random quantity$\mathcal{T}={\left\{{\tau }_{ij}\right\}}_{i<j}$, and jointly sample $(\Omega ,\mathcal{T})$ from

$$\pi (\Omega ,\mathcal{T}|{\mathbb{X}}_n,\Lambda )\propto \pi (\Omega ,\mathcal{T}|\Lambda )\times L\left(\Omega ;{\mathbb{X}}_n\right)$$ (9)

using the block Gibbs Sampler in Appendix, Algorithm 2, where

$$\pi (\Omega ,\mathcal{T}|\Lambda )\propto \pi (\Omega |\mathcal{T},\Lambda )\times \pi (\mathcal{T}|\Lambda ),$$ (10)

$$\pi (\Omega |\mathcal{T},\Lambda )={\displaystyle \prod }_{i<j}\phi \left({\omega }_{ij}|0,{\tau }_{ij}{\lambda }_{ij}^{-2}/2\right)\times {\displaystyle \prod }_{i=1}^d\gamma \left({\omega }_{ii}|1,{\lambda }_{ii}\right)\times I(\Omega \succ 0)\times C(\mathcal{T},\Lambda ),$$ (11)

$$\pi (\mathcal{T}|\Lambda )\propto C{(\mathcal{T},\Lambda )}^{-1}{\displaystyle \prod }_{i<j}\gamma \left({\tau }_{ij}|1,1\right),$$ (12)

φ(u | μ,σ²) is the Gaussian p.d.f. with mean μ and variance σ², γ(z|a,b) is the Gamma p.d.f. with shape and rate parameters a and b, and $C(\mathcal{T},\Lambda )$ is the finite normalizing constant of Eq. (11). Then, because the Laplace distribution is a Gaussian scale mixture (Andrews and Mallows, 1974; West, 1987), we can write Eq. 8 as the integral of Eq. 9:

$$\pi (\Omega |{\mathbb{X}}_n,\Lambda )=\int \pi (\Omega ,\mathcal{T}|{\mathbb{X}}_n,\Lambda )d\mathcal{T},$$

so that the matrix Ω in a samp le $(\Omega ,\mathcal{T})$ drawn from Eq. (9) is a sample from the posterior distribution in Eq. 8.

2.2.2. Full Bayes estimation

Let g in Eq. (7) be a step function with K steps and value β_k > 0 in step k, that is:

$$g\left(W_{ij}\right)={\displaystyle \sum _{k=1}^K{\beta }_k}{I}_{A_k}\left(W_{ij}\right),$$ (13)

where I is the indicator function, ∀h ≠ l, A_h ⋂ A_l = ∅, and ${\cup }_{k=1}^K{A}_k={ℝ}^m$. Let α = (α₁,…,α_d), β = (β₁,…,β_K) and Θ = {α,β}, where${\lambda }_{ii}={\alpha }_{ii}^2$. Following the augmentation strategy in Section 2.2.1, Eq. (10) reduces to

$$\begin{gathered} \pi (\Omega ,\mathcal{T}|\Theta ,W) \\ ={\displaystyle \prod }_{i<j}\phi \left({\omega }_{ij}|0,\frac{{\tau }_{ij}}{2{\alpha }_i^2{\alpha }_j^2{\sum }_{k=1}^K{\beta }_k^2{I}_{A_k}\left(W_{ij}\right)}\right)\gamma \left({\tau }_{ij}|1,1\right) \\ \times {\displaystyle \prod }_{i=1}^d\gamma \left({\omega }_{ii}|1,{\alpha }_i^2\right)\times I(\Omega \succ 0)\times G(\Theta ) \end{gathered}$$ (14)

where G(Θ) is the finite normalizing constant. We further assume the hyperprior density on Θ

$$\pi (\Theta )\propto {\displaystyle \prod }_{i=1}^d{\alpha }_i^{r-1}{e}^{-s{\alpha }_i^2}\times {\displaystyle \prod }_{k=1}^K{\beta }_k^{r^{\prime }-1}{e}^{-s^{\prime }{\beta }_k^2}\times G{(\Theta )}^{-1}$$ (15)

with r = s = 1, r′ = 0.01, and s′ = 0.00001; results were not sensitive to the choice of these parameters. We use the Gibbs sampler in Appendix, Algorithm 3, to sample from the full joint posterior distribution

$$\pi (\Omega ,\Theta ,\mathcal{T}|{\mathbb{X}}_n,W)\propto \pi (\Omega ,\Theta ,\mathcal{T}|W)\times L\left(\Omega ;{\mathbb{X}}_n\right),$$ (16)

where the likelihood $L\left(\Omega ;{\mathbb{X}}_n\right)$ is defined in Eq. (2), and the prior joint distribution $\pi (\Omega ,\Theta ,\mathcal{T}|W)$ is the product of Eqs. (14) and (15).

In practice, we take the number of steps K in Eq. (13) to be relatively small, e.g. 4 or 5, if we expect the penalties to change relatively slowly with W. Otherwise, to increase the flexibility of g while ensuring that enough data points contribute to estimating each β_k, we can afford of take $K\approx \sqrt{d}$ since d(d + 1)/2 values of W are available. We further locate the steps at evenly spaced empirical quantiles of the W’s, using a hierarchical quantile splitting when W is multi-dimensional, so that each step contains approximately the same number of W’s. If no auxiliary quantity W is available, full Bayes GAR can still be applied by setting K = 1. If we wanted to constrain g to be monotonic increasing, we could enrich the prior in Eq. (15) with the factor ${\prod }_{k=2}^{K}I\left({\beta }_{k-1}<{\beta }_k\right)$, which requires β_k to be sampled from the same distribution at step 2 of Algorithm 3, but truncated to be within the interval (β_k-1, β_k+1), where β₀ = 0 and β_K+1 = ∞. Different features of g may be imposed in similar ways.

2.2.3. Empirical Bayes estimation

By assuming Eq. (7) and${\lambda }_{ii}={\alpha }_i^2$, Eq. (4) reduces to

$$\pi (\Omega |\Theta ,W)={\displaystyle \prod _{i<j}{\alpha }_i}{\alpha }_{j}g\left(W_{ij}\right)e^{-2{\alpha }_i{\alpha }_{j}g\left(W_{ij}\right)|{\omega }_{ij}|}\times {\displaystyle \prod _{i=1}^d{\alpha }_i^2}{e}^{-{\alpha }_i^2{\omega }_{ii}}\times I(\Omega \succ 0)\times G(\Theta )$$ (17)

where Θ = {α, g}, g is a positive function of any form estimable in a Gamma regression, and G(Θ) is the normalizing constant. We further assume the prior density

$$\pi (\Theta )\propto p(\Theta )\times G{(\Theta )}^{-1}$$ (18)

where p (Θ) is a density on Θ. We estimate Θ by maximizing the posterior density

$$\pi (\Theta |{\mathbb{X}}_n,W)={\displaystyle {\int }_{\Omega \succ 0}\pi }(\Omega ,\Theta |{\mathbb{X}}_n,W)d\Omega$$ (19)

using an Expectation-Maximization algorithm (Dempster, 1977; Gelman et al., 2004):

• –
  E-step: Given the current estimate Θ^old, we compute the expectation

  $$\mathbb{E}[\log \pi (\Omega ,\Theta |{\mathbb{X}}_n,W)|{\mathbb{X}}_n,{\Theta }^{\text{old}},W],$$ (20)

  with respect to $\Omega ~\pi (\Omega |{\mathbb{X}}_n,{\Theta }^{\text{old}},W)$, which reduces to c + Q(Θ | Θ^old), where c is a constant of Θ and

  $$\begin{gathered} Q(\Theta |{\Theta }^{\text{old}})= \\ {\displaystyle \sum _{i<j}\left\{\log \left[{\alpha }_i{\alpha }_{j}g\left(W_{ij}\right)\right]-2{\alpha }_i{\alpha }_{j}g\left(W_{ij}\right){\overline{\omega }}_{ij}\right\}} \\ +{\displaystyle \sum _{i=1}^d\left\{2\log {\alpha }_i-{\alpha }_i^2{\overline{\omega }}_{ii}\right\}}+\log p(\Theta ), \end{gathered}$$

  where ${\overline{\omega }}_{ij}=\mathbb{E}\left[|{\omega }_{ij}||{\mathbb{X}}_n,{\Theta }^{\text{old}},W\right]$ can be approximated using the Gibbs sampler (Section 2.2.1).
• –
  M-STEP: Q(Θ | Θ^old) is concave with respect to g and α_i, i = 1,…,n, so we can maximize it with respect to Θ by circularly optimizing with respect to α and g until convergence, as follows: assuming p(Θ) ∝ 1 in Eq. (18), $\frac{\partial Q}{\partial {\alpha }_i}=0$ subject to α_i > 0 yields the maximizer

  $${\alpha }_i=\left(\sqrt{{\eta }^2+8{\overline{\omega }}_{ii}(d+1)}-\eta \right)/\left(4{\overline{\omega }}_{ii}\right),$$ (21)

  where $\eta =2{\sum }_{j\ne i}{\alpha }_{j}g\left(W_{ij}\right){\overline{\omega }}_{ij}$, and the maximizing function g is obtained by regressing $y_{ij}=2{\alpha }_i{\alpha }_j{\overline{\omega }}_{ij}$ on W_ij, i < j in a Gamma regression model.

We summarize the procedure for the case p(Θ) ∝ 1 in Appendix, Algorithm 4.

2.3. Estimating a connectivity graph

Here, we explain how to estimate graphs based on partial correlation estimates, so as to control the edge false discovery and false nondiscovery rates.

Let E = {e_ij} be a true graph, where e_ij = 1 when nodes i and j are connected by an edge, and e_ij = 0 otherwise. We build graph estimators of two kinds:

1. A δ-graph has edges ${\widehat{E}}_{\delta }=\left\{{\widehat{e}}_{ij}(\delta )\right\}$ such that, for a threshold δ ∈ [0,1),

   $${\widehat{e}}_{ij}(\delta )=\{\begin{array}{l}1,\text{if}|{\widehat{\rho }}_{ij}|>\delta \hfill \\ 0,\text{otherwise}\hfill \end{array}$$ (22)

   where ${\widehat{\rho }}_{ij}=-{\widehat{\omega }}_{ij}/\sqrt{{\widehat{\omega }}_{ii}{\widehat{\omega }}_{jj}}$ is a point estimate of the partial correlation in Eq. (6). That is, an edge is present in the graph estimate if the corresponding estimated absolute partial correlation $|{\widehat{\rho }}_{ij}|$ exceeds some threshold δ.
2. A (p, δ)-graph has edges ${\widehat{E}}_{p,\delta }=\left\{{\widehat{e}}_{ij}(p,\delta )\right\}$ such that, for thresholds p, δ ∈ [0,1),

   $${\widehat{e}}_{ij}(p,\delta )=\{\begin{array}{l}1,\text{if}{\pi }_{ij}(\delta )>p\hfill \\ 0,\text{otherwise}\hfill \end{array}$$ (23)

   where

   $${\pi }_{ij}(\delta )=\mathbb{P}(|{\rho }_{ij}|>\delta |{\mathbb{X}}_n,W)$$ (24)

   is the edge posterior probability, that is the posterior probability that the magnitude of the partial correlation ρ- exceeds δ. The (p, δ)-graph uses the full posterior probability of the partial correlations rather than the point estimates ${\widehat{\rho }}_{ij}$, setting an edge to zero if the edge posterior probability (Eq. 24) is smaller than p. In our simulations, (p, δ)-graphs were often more precise than δ-graphs.

One must choose values for δ and p. Using δ = 0 in Eq. (22) could produce a completely dense graph, while using a large value would only identify strong edges, which is not sensible for neural networks since connections can be small and numerous. Ideally, δ should be as close as possible to the minimum magnitude of the true non-zero partial correlations; we take a robust estimate of that minimum to be the 5-th quantile of the magnitudes of the non-zero MAP partial correlation estimates. We use the same δ in Eq. (23), and choose p to control the false discovery and non-discovery rates of the graph edges (FDR and FNR, respectively), that is the rate of false detected edges (number of (i, j) such that ${\widehat{e}}_{ij}=1$ but e_ij = 0) out of all detections (number of ${\widehat{e}}_{ij}=1$) and the rate of true missed connections (number of (i, j) such that ${\widehat{e}}_{ij}=0$ but e_ij = 1) out of all non-detections (number of${\widehat{e}}_{ij}=0$):

$$\text{FDR}=\mathbb{E}\left[\frac{{\sum }_{i<j}{\widehat{e}}_{ij}\left(1-e_{ij}\right)}{{\sum }_{i<j}{\widehat{e}}_{ij}}\right]$$ (25)

and

$$\text{FNR}=\mathbb{E}\left[\frac{{\sum }_{i<j}\left(1-{\widehat{e}}_{ij}\right)e_{ij}}{{\sum }_{i<j}\left(1-{\widehat{e}}_{ij}\right)}\right],$$ (26)

where the expectations are taken with respect to the data ${\mathbb{X}}_n=\left\{X^{(1)},\dots ,X^{(n)}\right\}$. Then for a fixed δ, a (p, δ)-graph can be selected by choosing either

$$p^{*}=\min \{p:\mathrm{FDR}\le C\}\text{or}{p}^{**}=\max \{p:\text{FNR}\le C\}$$

for some desired upper bound C, depending on whether we want to control the FDR or the FNR, or choose p to balance the FDR and FNR, that is FDR ≈ FNR (the intersection of the green and red curves in Fig. 2). However, the FDR and FNR are easy to approximate in simulations when the true graph E is known, but not otherwise. An alternative is to calculate their Bayesian counterparts, obtained by Eqs. (25) and (26) but with expectation taken conditionally on the data ${\mathbb{X}}_n$, which yields

$${\text{FDR}}_{II}=\frac{{\sum }_{i<j}{\widehat{e}}_{ij}\left(1-{\pi }_{ij}(\delta )\right)}{{\sum }_{i<j}{\widehat{e}}_{ij}}$$ (27)

and

$$FN{R}_{II}=\frac{{\sum }_{i<j}\left(1-{\widehat{e}}_{ij}\right){\pi }_{ij}(\delta )}{{\sum }_{i<j}\left(1-{\widehat{e}}_{ij}\right)},$$ (28)

where π_ij (δ) is the edge posterior probability defined in Eq. (24). Equation (27) has the same form as the Bayesian FDR in multiple hypothesis testing (Efron et al., 2001; Efron, 2007) and FDR-regression (Scott et al., 2015), obtained as the average of the local FDRs, i.e. posterior probabilities that the hypotheses are null, across the rejections. In our framework (1 — π_ij (δ)) is the local FDR for the pair (i,j). In Fig. 2 we show by simulation that bounding FDR_II or FNR_II also appears to bound their frequentist counterparts. This result is not surprising because empirical estimates of the Bayesian FDR are typically upward biased estimates of the frequentist FDR (Efron and Tibshirani, 2002).

Fig. 2.

Bayesian FDR and FNR, FDR_II and FNR_II, plotted on the x- axes, control their frequentist counterparts FDR and FNR, plotted on the y-axes, in a simulation based on d = 100 neurons and sample size n = 500 (Section 2.4). The black lines are the first bisectors x = y.

ROC curves summarize graph estimation performance

Other useful measures of detection error are sensitivity and specificity

$$\text{SENS}=\frac{{\sum }_{i<j}{\widehat{e}}_{ij}{e}_{ij}}{{\sum }_{i<j}{e}_{ij}}$$ (29)

and

$$\text{SPEC}=\frac{{\sum }_{i<j}\left(1-{\widehat{e}}_{ij}\right)\left(1-e_{ij}\right)}{{\sum }_{i<j}\left(1-e_{ij}\right)},$$ (30)

which give the proportions of true edges correctly identified and of missing edges correctly omitted, respectively. By tuning the parameters defining the estimated edges ${\widehat{e}}_{ij}$’s, for example λ in Eq. (1) and p in Eq. (23), we can obtain the curve of SENS versus 1 — SPEC, known as Receiver Operating Characteristic (ROC) curve (see Figs. 1 and 3). A point above the ROC curve denotes an edge detection performance that cannot be achieved by the estimator, i.e. no values of the tuning parameters could make the estimator produce that outcome. Therefore, a larger area under a ROC curve (AUC) indicates a better edge detection performance; we use this metric to compare graph estimators in our simulations (Figs. 1 and 3).

Fig. 3.

Performance of graph estimates, measured by areas under ROC curves (AUC), and of partial correlation estimates, measured by MSEs, for d = 50 neurons simulated according to Eq. (31) as functions of σ_ϵ in (a), where σ_ϵ ≈ 1 (shaded grey area) represents a real data scenario, and as functions of the sample size n in (b). (c) AUC and MSE for d = 50 observed neurons conditionally on q = 20 latent ones as functions of n. (d) AUC when methods are applied to non-Gaussian Poisson-lognormal (PLN) data without (top) and with (bottom) latent neurons, as functions of n. The horizontal intevals on the left of ah curves are 95% simulation intervals.

2.4. Statistical properties of the estimates of Ω in simulated data

We compared the performance of the various estimators of Ω in an extensive simulation study. We simulated data sets of n d-dimensional Gaussian vectors $X^{(1)},\dots ,X^{(n)}\stackrel{\text{iid}}{~}N\left(\mu ,{\Omega }^{-1}\right)$, with μ chosen to match typical values found in experimental data, and with Ω = [ω_ij] generated as follows: for i < j,

$$\{\begin{array}{l}{\omega }_{ij}=Z_{ij}\exp \left(b{W}_{ij}+{ϵ}_{ij}\right)\hfill \\ Z_{ij}~I(z=-1)\eta +I(z=1)(1-\eta )\hfill \\ {ϵ}_{ij}~N\left(0,{\sigma }_{ϵ}^2\right)\hfill \end{array}$$ (31)

where I (A) = 1 if A is true and I (A) = 0 otherwise, so that Z_ij ∈ {—1,1} makes ω_ij negative with probability η and positive with probability 1 — η. We use a simple auxiliary variable W_ij, which we take to be the physical distance between neurons i and j on a 4mm × 4mm Utah electrode array, and used b = –2, η = 0.75, and σ_𝜖 = 1 to generate values of ω_ij that are consistent with the experimental data analyzed in Section 2.5. We then symmetrized Ω by setting ω_ji = ω_ij, and set the diagonal entries ω_ii to the smallest positive ω* that rendered Ω positive definite. We achieved sparsity by resetting to zero the smallest half of the partial correlations so that half of the graph edges were null, and rescaled Σ = Ω⁻¹ so its diagonal elements would have magnitude similar to the experimental data analyzed in Section 2.5.

Fig. 1 shows a simulated network of d = 49 neurons with dependence structure in Eq. (31), the Glasso graph estimate, and the empirical Bayes GAR estimate (Algorithm 4) with g (Eq. 7) taken to be a regression spline: GAR uses the spatial information of the inter-neuron distances and yields more accurate connectivity graph estimates. Here and in the rest of the paper, all GAR parameters, including the penalty parameters, are assigned a prior distribution and are thus selected implicitly. Similarly for the other Bayesian methods. For the non-Bayesian methods, Glasso, AGlasso, and SPL, the penalty parameters are selected by cross validation (Fan et al., 2009).

Fig. 3a shows the AUC and mean squared error (MSE) of the partial correlation matrix estimate for a simulated network of d =50 neurons and sample size n = 200, for moderate deviations from 𝜎_𝜖 = 1 in Eq. (31), where the MSE is the average of the squared error, ${\sum }_{i<j}{({\widehat{\rho }}_{ij}-{\rho }_{ij})}^2/(2d(d-1))$, across repeat simulations. We show only the performance of the full Bayes GAR estimate; it is comparable to that of the empirical Bayes GAR estimate but faster to compute. Fig. 3b shows how AUC and MSE vary with the sample size n for fixed 𝜎_𝜖 = 1 and d = 50. GAR outperforms all other methods in Figs. 3a and 3b. The SAGlasso is the next best method. It is easier to implement than GAR but provides only a modest improvement over other methods.

Figs. 3a and 3b also show the performance of GAR*, the full Bayes estimate that uses a constant g(w) in Eq. (7). Comparing GAR and GAR* shows the added benefit of including auxiliary information within our novel Bayesian framework. Note that GAR* appears to outperform the Glasso and its variants even without adapting the regularization to the covariate. This may be due to the parameters α_i (Eq. 7) forcing the penalty to be on the standardized scale of the partial correlations rather than on the scale of the precision matrix entries, which are known to be highly sensitive to the variance of the Gaussian vectors X (Yuan and Lin, 2007). It is also possible that, while the Glasso variants AGlasso and BAGlasso also attempt to attenuate the data scaling effect on the regularization, they involve d(d +1) /2 regularization parameters (the entries of Q in Eq. 3 and the λ_ij ‘s in Ea. 4), whereas GAR* achieves the same goal with only d +1 parameters (the α_i’s and the constant g), which might reduce the variance of the partial correlation estimates.

The problem of estimating pairwise dependences conditionally on both observed and latent units (here neurons) has been dealt with previously by applying SPL in Eq. (5). In Fig. 3c we compare the performances of GAR, Glasso, SPL, and GAR-SPL to estimate the dependence structure of d =50 recorded neurons when an additional q = 20 neurons belong to the network but their activity is not observed. GAR outperformed SPL, likely because the information extracted from the inter-neuron distance overcompensated the missed information about the activity of the latent variables. SPL outperformed Glasso only in terms of MSE. The variant GAR-SPL outperformed SPL but not GAR, likely because of the additional variance due to the estimation of the low-rank component. However, a full Bayesian treatment of GAR-SPL might provide a better performance; this is a topic of future research.

Fig. 3d repeats the AUC curves of Figs. 3bc but with non-Gaussian data generated from the multivariate Poisson- lognormal distribution (Vinci et al., 2016)

$$\begin{gathered} X_i^{(r)}|Z_i^{(r)}~\mathrm{Poisson}\left(e^{Z_i^{(r)}}\right),i=1,\dots ,d \\ {\left(Z_1^{(r)},\dots ,Z_d^{(r)}\right)}^{\prime }~N(\mu ,\Sigma ), \end{gathered}$$ (32)

where the spike counts $X^{(r)}=(X_1^{(r)},\dots ,X_d^{(r)})$ on trial r are independent given their latent log-rates$Z_1^{(r)},\dots ,Z_d^{(r)}$, and μ and Σ were set to match typical values found in experimental spike count data (μ_i ≈ 2, Σ_ii ≈ 0.25, implying about 8.37 spikes/s on average). Under this model, dependences between spike counts are weaker than those of their latent rates due to Poisson-noise corruption (Vinci et al., 2016; Behseta et al., 2009). Since the dependences between the log-rates likely provide a better representation of input correlation (Vinci et al., 2016), we estimate the neuronal graph based on the partial correlations Ω = Σ⁻¹ of the latent rates, applying first the square root transformation to the spike counts to improve their fit to a Gaussian distribution (Kass et al., 2014; Georgopoulos and Ashe, 2000; Yu et al., 2009). Fig. 3d shows that GAR outperformed all other methods, presumably because it successfully extracted the connectivity information carried by inter-neuron distance despite the Poisson noise. Also as in Fig. 3c, GAR-SPL outperformed SPL but not GAR. We repeated the simulation of Fig. 3d with values of μ_i small enough to induce neurons’ firing rates of about 0.1 spikes/s and, compared to Fig. 3d, noted a loss of AUC of about 5% for GAR and GAR-SPL, and about 20% for all other methods that do not use auxiliary information.

2.5. Estimating neural connectivity in macaque visual cortex

Spike data were recorded from the V1 and V4 visual cortex of two rhesus macaque monkeys using 100-electrode Utah arrays. For the V1 data (Kelly et al., 2010; Scott et al., 2015; Cowley et al., 2016), visual stimuli were presented to an anesthetized animal. The stimuli were either a 30s sequence of drifting sinusoidal gratings (98 different orientations and two blanks, 300ms each), or blank gray screen (Fig. 4a). The 30s stimuli sequence was randomly ordered, and then repeated in that same order 120 times. For the V4 data, the visual stimuli were either vertical drifting sinusoidal gratings or a blank gray screen (Fig. 4b). Each trial began with the animal fixating a small dot for 150ms before the grating or blank screen was presented for 1000ms. Then the stimulus and fixation point were extinguished and the animal received a liquid reward for making an eye movement to a target 8 degrees from fixation in a random direction. Each stimulus (the vertical grating or the blank) was repeated 126 times. All procedures were approved by the Institutional Animal Care and Use Committee of the University of Pittsburgh and Albert Einstein College of Medicine, and were in compliance with the guidelines set forth in the National Institutes of Health’s Guide for the Care and Use of Laboratory Animals.

Fig. 4.

Experiments. (a) The V1 stimuli consist of 30 second sequences of 98 randomly ordered oriented drifting gratings, or a blank gray screen. (b) The V4 stimuli consist of a blank gray screen or a vertical drifting sinusoidal grating appearing in the aggregate receptive field of the V4 neurons (RFs, indicated by the dashed circle).

We applied GAR to the neurons’ spike counts in repeated trials of duration 300ms and 1000ms in the V1 and V4 experiments, respectively, to estimate the connectivity at the trial time scale. We square-root transformed the spike counts to mitigate the dependence between variance and mean and thus improve their fit to Gaussian distribution we assume in this paper.

V1 data

We obtained recordings for 128 candidate neuronal units by sorting the voltage signals of the 76 electrodes with the best signal to noise ratio (Kelly et al., 2007), in response to a sequence of 98 drifting sinusoidal gratings and blank screen (Fig. 4a). Previous analyses of these data have been published (Kelly et al., 2010; Scott et al., 2015; Cowley et al., 2016). To produce easily readable graphs, we analysed only 100 neurons, selected as follows: we first retained the highest spiking neuron on each of the 76 electrodes that had identifiable action potentials, and then added the 24 highest spiking remaining neurons. The firing rates of these 100 neurons ranged from 0.61 to 31.97 spikes/s, with mean 6.27, and 2.5th and 97.5th percentiles 1.12 and 16.20.

Neurons in V1 are driven by drifting gratings of orientation θ ∈ (0, 360] (Scott et al., 2015), and their average firing rates are usually described by sinusoidal tuning functions of θ, with similar tuning in diametrically opposite orientations θ and θ + 180 degrees (Butts and Goldman, 2006; Smith and Kohn, 2008; Scott et al., 2015; Vinci et al., 2016). Hence maximal firing rates occur at θ* and θ * + 180, for some θ*, and minimal firing rates occur in the orthogonal orientations, θ* ± 90. The tuning similarity of two neurons can be quantified by their tuning curve correlation (TCC), computed as the Pearson’s correlation of the two neurons’ tuning curves across stimuli (Smith and Kohn, 2008; Smith and Sommer, 2013; Ecker et al., 2014; Kass et al., 2014; Vinci et al., 2016). The strength of the dependence between two neurons’ activities has been observed to increase with TCC and decrease with inter-neuron distance (DIST) (Smith and Kohn, 2008; Smith and Sommer, 2013; Goris et al., 2014; Vinci et al., 2016), so we considered these two auxiliary quantities for our GAR connectivity graph estimation.

Fig. 5 shows the results of GAR applied to 300 ms square root transformed spike counts for the vertical grating (θ = 0) and blank screen conditions. The estimated penalty functions g(W₁,W₂) in Eq. (7) (fitted using Algorithm 4, with g a bivariate regression splines) are plotted in Fig. 5a: they increase with W₁ = DIST and decrease with W₂ = TCC, which is consistent with previous analyses of V1 macaque data (Goris et al., 2014; Smith and Kohn, 2008; Scott et al., 2015) where neurons dependences in macaque V1 were observed to decrease with inter-neuron distance and increase with tuning curve similarity. Fig. 5b shows that the corresponding edge posterior probabilities (Eq. (24)) decrease with DIST and increase with TCC, on average; the horizontal lines denote probability thresholds that lead to different graph FDR_II controls (Eqs. (23), (25), and (27)). We obtained similar results for all grating orientations. Finally, Fig. 6a displays the estimated (p, δ)-graphs (Eq. (23)): the graph for the vertical grating contains 1160 edges (859 positive and 301 negative connections) with 10% FDR_n, and 350 edges (307 positive, 43 negative) with 5% FDR_II; the graph under blank screen contains 1246 edges (937 positive, 309 negative) with 10% FDR_n, and 405 edges (362 positive, 43 negative) with 5% FDR_II.

Fig. 5.

(a) Estimated penalty functions (Eq. 7) and (b) edge posterior probabilities (Eq. (24)) for ad pairs of V1 neurons, with average plotted in red, under vertical grating and blank conditions. The penalty g(W₁, W₂) increases with W₁ = inter-neuron distance and decreases with W₂ = tuning curve correlation. The average edge posterior probability decreases with W₁ and increases with W₂. The horizontal Lines are the probability thresholds that lead to the corresponding 5, 10, 15, or 20% FDR_II controls in (p, δ)-graphs (Eqs. (23), (25), and (27)).

Fig. 6.

Connectivity of V1 neurons. (a) Estimated connectivity (ρ,δ)-graphs under vertical grating and blank conditions, with respective number of edges 1160 ± 87 and 1246 ± 87 (95% bootstrap intervals) at 10% FDR_II, and 350 ± 29 and 405 ± 30 at 5% FDR_II. The node positions represent the individual active channels on the 4×4 mm electrode array, blue and red edges denote positive and negative partial correlations, and a node size is proportional to the number of its connections. (b) Average across neuron pairs of absolute correlation (blue, Eq. 33) and mutual information (red, Eq. 34) with 95% posterior probability bands, as function of grating orientation θ. Larger values imply stronger connectivity. Both measures show maximal values about 20% above their minimum values. (c) Proportions of rejections across θ values of the 5%-level permutation tests that compare connectivities between orientations Δθ apart. Connectivity changes smoothly as a function of Δθ, and the connectivity at an arbitrary orientation θ differs maximally from the connectivity at the orthogonal orientations θ ± Δθ with Δθ = 90.

To investigate the extent to which covariation is tuned to orientation we computed the average absolute correlation

$$\text{AC}(\theta )=\frac{1}{d(d-1)}{\displaystyle \sum _{i\ne j}|c_{ij}(\theta )|},$$ (33)

and the multivariate Gaussian Mutual Information (Shannon, 1964; Guerrero, 1994; Cover and Thomas, 2012)

$$\text{MI}(\theta )=-\frac{1}{2}\log \det C(\theta ),$$ (34)

where C(θ) = diag(Σ(θ))^−1/2 × Σ(θ) × diag(Σ(θ))^−1/2 = [c_ij(θ)] is the correlation matrix of the square-rooted spike counts at orientation θ, and Σ(θ) = Ω(θ)^-1. Larger AC(θ) and MI(θ) imply stronger connectivity. Fig. 6b shows the posterior means of AC(θ) and MI(θ) as functions of θ, with 95% posterior probability bands; both measures display similar variations in connectivity and it appears that grating orientations θ and θ + 180 yield graphs with similar connectivity. To confirm this, we considered the network connectivity at orientations Δθ apart: given some orientation θ and some Δθ, we (i) calculated

$$D(\Delta \theta )={\displaystyle \sum _{i<j}|c_{ij}(\theta )-c_{ij}(\theta +\Delta \theta )|},$$ (35)

(ii) obtained a permutation test p-value (Kass et al., 2014) of the null hypothesis that the connectivity is the same at θ and θ + Δθ, i.e. D(Δθ) = 0, and repeated (i) and (ii) for all 98 values of θ ∈ [0,360]. Fig. 6c shows how the proportion of 5%-level test rejections varies with Δθ: the connectivity changes smoothly as a function of Δθ, and the connectivity at an arbitrary orientation θ differs maximally from the connectivity at the orthogonal orientations θ ± 90, and minimally at orientation θ + 180.

We applied GAR to estimate the V1 neuron network because our simulation study suggests it is the most accurate method. For comparison’s sake, we also applied AGlasso (Eq. 3) and drew qualitatively similar conclusions: 681 edges were identified for vertical grating and more, specifically 945, for blank screen (although the resulting graphs look quite different since only about 50% of these edges were also identified by GAR at 10% FDR_n within each condition) and the mutual information and average absolute correlation showed variations with orientation that were similar to, but more volatile than in Fig. 6b.

V4 data

V4 data were recorded in response to a vertical grating (θ = 0) and a blank screen (Fig. 4b). We selected 100 out of the 152 available candidate neuronal units, according to the same criteria used for the V1 data. The firing rates of these 100 neurons ranged from 0.04 to 38.54 spikes/s, with mean 6.64, and 2.5th and 97.5th percentiles 0.09 and 27.33. We applied GAR (Algorithm 4 with a univariate regression spline) to the 1000 ms square root transformed spike counts with inter-neuron distance (DIST) as an auxiliary quantity, and estimated the connectivity in the two conditions. The estimated penalty function g(W) (Eq. (7), Fig. 7a) increases with W = DIST and the edge posterior probability (Eq. (24), Fig. 7b) decreases with DIST, which is consistent with the previous analyses in Smith and Sommer (2013) and Vinci et al. (2016). Note that the posterior probability of non-zero partial correlations (|ρ| > δ = .005) remains substantial at 70–75% between neurons that are over 2mm apart, but because these correlations have magnitude (absolute value of posterior mean) close to the threshold δ (not shown), they do not appear in the (p, δ)-graphs displayed in Fig. 7c, at 10% and 5% FDR_II. The graph under vertical grating contains 333 edges (229 positive and 104 negative) with 10% FDR_II, and 59 edges (53 positive and 6 negative) with 5% FDR_II the graph under blank screen contains 573 edges (400 positive and 173 negative) with 10% FDR_II, and 143 edges (115 positive and 28 negative) with 5% FDR_II. These results suggest that neural connectivity is denser in the spontaneous activity induced by the blank screen. This was confirmed by a permutation test (Kass et al., 2014) based on the statistic $D(a,b)={\sum }_{i<j}|c_{ij}(a)-c_{ij}(b)|$, analogous to Eq. (35), with null hypothesis D(a, b) = 0: the p-value was smaller than 10^-8.

Fig. 7.

(a) Estimated penalty functions (Eq. 7) and (b) edge posterior probabilities (Eq. (24)) for all pairs of V4 neurons, with average plotted in red, under vertical grating and blank conditions. The penalty g(W) increases with W = inter-neuron distance. The average edge posterior probability decreases with W. The horizontal lines are the probability thresholds that lead to the corresponding 5, 10, 15, or 20% FDR_II controls in (p,d)-graphs (Eqs. (23), (25), and (27)). (c) Estimated connectivity (p, δ)-graphs under the two conditions, with respective number of edges 333 ± 79 and 573 ± 89 (95% bootstrap intervals) at 10% FDR_II, and 59 ± 19 and 143 ± 21 at 5% FDR_II. The node positions represent the individual active channels on the 4 × 4 mm electrode array, blue and red edges denote positive and negative partial correlations, and a node size is proportional to the number of its connections.

For comparison sake, we also estimated the V4 neuron network using the AGlasso (Eq. 3) and found 297 and 492 edges for vertical grating and blank screen, respectively; about 42% of AGlasso’s edges were also discovered by GAR at 10% FDR_II in the vertical grating condition, and about 44% in the blank screen condition.

Remarks

In the two datasets we analysed here, the estimated function g (Eq. (7)) increased with inter-neuron distance and decreased with tuning curve correlation; conversely, the edge posterior probabilities (Eq. (24) decreased and increased with inter-neuron distance and tuning curve correlation, respectively (Figs 5 and 7). Because g was fitted using splines, the dependence of the regularization on the auxiliary variables was estimated from the data rather than prespecified. Hence, GAR automatically extracted the neurons functional connectivity information carried by the auxiliary variables and incorporated it into the estimation of the partial correlations and dependence graphs. These are the main results of our data analyses. Note that the penalty function g is not constrained to be monotonic. While lateral connectivity in a region or within a single cortical area decreases with distance, which was encoded in our two data examples by g increasing with distance, there are many long neural pathways in the brain (see Van Den Heuvel and Sporns (2011)). If these also induce functional connectivity, GAR will extract that information and fit a penalty function g that varies accordingly.

We further note that in both analyses reported here, the number of positive connections were approximately two to four times the number of negative ones. This result is consistent with previous analyses in macaque visual cortex where the majority of pairwise correlations were positive (Smith and Kohn, 2008; Smith and Sommer, 2013), which in turn suggests that finding a majority of positive partial correlations is reasonable (if a positive-definite covariance matrix has mostly positive entries, then its inverse has mostly negative entries, and consequently partial correlations are mostly positive according to Eq. (6)); a majority of positive partial correlations has also been observed in mouse visual cortex (Yatsenko et al., 2015). Moreover, it is also known that in macaque visual cortex the ratio of excitatory to inhibitory neurons is about 80/20 (Markram et al., 2004), which, depending on the relative proportion of inhibitory to excitatory neurons recorded and their connection strengths and probabilities, might favors positive functional connections. We also found that the partial correlations in areas V1 and V4 had similar magnitudes (Fig. 8a), but that the correlations were larger and the connectivity denser in V1 (Figs. 8b and 8c), which is consistent with previous findings in V1 (Smith and Kohn, 2008) and V4 (Smith and Sommer, 2013; Vinci et al., 2016). Denser connectivity and higher correlations in V1 may be due to differences in time-scales and correlation between cortical layers (Smith et al., 2013), since the V1 data were targeted at more superficial layers than V4, as well as other differences in connectivity structure between the two cortical regions. In addition, slow fluctuations in activity due to anesthesia may have played a role in the higher correlation values in V1 (Ecker et al., 2014), although the values present in the data analyzed here are similar to other V1 reports in awake animals (Gutnisky and Dragoi, 2008; Poort and Roelfsema, 2009; Samonds et al., 2009; Rasch et al., 2011).

Fig. 8.

Distributions of partial correlations, correlations, and number of connections per neuron in areas V1 and V4. Partial correlations in areas V1 and V4 have similar magnitudes. Correlations and number of connections per neuron are larger in area V1.

3. Discussion

We have derived and implemented a method for pairwise covariate adjustment of the regularization penalty in the Graphical lasso, and have shown that it can greatly improve identification of the functional connectivity graph in high - dimensional neural spike count data. We have also illustrated the use of this technique in studying network activity by analyzing data from cortical areas V1 and V4, where the covariates were distance between neurons and tuning curve correlation. We expect this approach to be applicable to neural activity throughout cortex, and in subcortical areas as well. At the very least, partial correlation may be expected to change with inter-neuron distance, regardless of where these neurons are located, although cell-type shape specific information may also be useful to include, for example through a group-lasso type regularization (Yuan and Lin, 2006), to adjust for anisotropies of neurons’ axonal projections (Sin- cich and Blasdel, 2001). In addition, even in the absence of a well-defined tuning curve, it is reasonable to expect partial correlation to depend on other characterizations of trial- averaged responses across experimental conditions (based, for example, on the PSTH), or anatomical connectivity and genetic information about neurons. Thus, we suggest the incorporation of this kind of covariate information is likely to be helpful in a wide range of problems involving neural functional connectivity.

While we are inclined, based on the research reported here, to think that the general idea of incorporating covariate information into regularization is a good one, there are many different ways to carry it out. These could involve alternative forms of regularization (e.g. an elastic net may be better suited than an L₁ penalty to regularize a large number of small effects), within both Bayesian and nonBayesian frameworks, as well as regularization combined with dimensionality reduction (Chandrasekaran et al., 2012; Yuan, 2012; Yatsenko et al., 2015). These are topics for future research. Also, in previous work we noted that when spike count correlation is viewed as resulting from underlying firing-rate correlation after corruption by Poisson-like noise (Vinci et al., 2016; Behseta et al., 2009), the firing- rate correlation can be much larger than spike count correlation, and may be expected to be more sensitive to experimental manipulation because it likely provides a better representation of input correlation (Vinci et al., 2016). A natural next step, therefore, is to nest GAR within the hierarchical model of Vinci et al. (2016). We are currently working on that extension to the method developed here. In addition, covariate-adapted regularization may be applied to high-dimensional point process models of neural spike trains (Kass et al., 2014) and time series models for local field potentials and other continuous-time neural signals. We hope to investigate the utility of the general idea in these diverse neural contexts.

4. Appendix

4.1. SAGlasso algorithm

There are several ways to build the weight matrix Q of SAGlasso. We used Gamma regression, as described in Algorithm 1, which can be implemented efficiently with standard statistical software, e.g. R packages glm (Dobson and Barnett, 2008; Hastie and Pregibon, 1992; McCullagh and Nelder, 1989; Venables and Ripley, 2002), mgcv (Wood, 2011), and gam (Hastie and Tibshirani, 1990). Note that in Eq. (3), Q is typically estimated by the square rooted absolute entries of the inverse sample covariance matrix. In SAGlasso, we observed a slightly better performance without applying any transformation.

Algorithm 1.

SAGlasso

Input: ${\mathbb{X}}_n$, W, λ > 0, and $\widehat{\Omega }$, preliminary estimate of Ω.

1. Obtain {Yij}i<j, where $y_{ij}=|{\widehat{\omega }}_{ij}|/{\left({\widehat{\omega }}_{ii}{\widehat{\omega }}_{jj}\right)}^{1/2}$.

2. Fit Gamma regression of Yij on Wij with rate g(Wij).

3.Obtain $\widehat{Q}=\left[{\widehat{q}}_i{}_j\right]$ where ${\widehat{q}}_{ij}=\widehat{g}\left(W_{ij}\right)/{\left({\widehat{\omega }}_{ii}{\widehat{\omega }}_{jj}\right)}^{1/2}$, for i ≠ j, and ${\widehat{q}}_{ii}={|{\widehat{\omega }}_{ii}|}^{-1}$, for i = 1,..., d.

4. Solve Eq. (3) with Q = $\widehat{Q}$ and λ.

Output: Estimate$\widehat{\Omega }(\lambda )$.

4.2. GAR algorithms

GAR Algorithms 2–4 are derived in Section 2.2, and implemented in our R package “GARggm” available in ModelDB.

In Algorithm 2, U ~ InvGaussian(a, b) has p.d.f. $p(u)={\left(\frac{b}{2\pi u^3}\right)}^{1/2}\exp \left\{-b{(u-a)}^2/\left(2a^{2}u\right)\right\}$. Moreover, given a matrix M, M_ij is the i-th row and j-th column entry of M; M-ij is the j-th column of M without the i-th entry; Mi-j is the i-th row of M without the j-th entry; and M-i-j is the submatrix obtained by removing the i-th row and the j - th column from M. Algorithms 3 and 4 both produce posterior samples of Ω whose average approximates the posterior mean of Ω. The posterior mode of Ω can be obtained by solving Eq. (1) with λ||Ω|| replaced by $\Vert \widehat{\Lambda }\odot \Omega {\Vert }_1$, where $\widehat{\Lambda }$ is the estimated penalty matrix from either Algorithm 3 or 4, and ⊙ denotes the entry-wise matrix multiplication. This optimization can be performed using R functions such as glasso (package glasso, Friedman et al. (2008)) with argument rho set equal to $2\widehat{\Lambda }/n$; see also the R package QUIC, Hsieh et al. (2011). We solve the SPL problem in Eq. (5) by the EM algorithm of Yuan (2012) involving Glasso, and we impose the GAR penalty matrix $\widehat{\Lambda }$ on S in the Maximization step to obtain the GAR-SPL estimate. For d ~ 100, we suggest to run the Gibbs samplers for at least B = 2000 iterations, including a burn-in period of 300 iterations. The Gamma regression in step 2b of Algorithm 4 can be implemented either parametrically or nonparametrically by using standard statistical software (e.g. R packages glm (Dobson and Barnett, 2008; Hastie and Pregibon, 1992; McCullagh and Nelder, 1989; Venables and Ripley, 2002) and mgcv (Wood, 2011)); in the data analyses we used splines (Kass et al., 2014).

Algorithm 2.

Block Gibbs sampler for $\mathrm{\Omega }~\mathrm{\pi }(\mathrm{\Omega }|{\mathbb{X}}_{\mathrm{n}},\mathrm{\Lambda })$.

Input: $S={\sum }_{r=1}^n{\left(X^{(r)}-\overline{X}\right)}^{\prime }\left(X^{(r)}-\overline{X}\right)$ and Λ; start value of Ω; number of iterations B.

For b =1,..., B :

1. For i < j: sample ${\tau }_{ij}^{-1}~\mathrm{Inv}\text{Gaussian}({({\lambda }_{ij}|{\omega }_{ij}|)}^{-1},2)$.

2. For i = 1, ...,d: compute ${\Omega }_{-ii}={\Omega }_{i-i}^{\prime }≔\eta$ and

${\omega }_{ii}:=\xi +{\eta }^{\prime }{\Omega }_{-i-i}^{-1}\eta$, where

ξ ~ Γ(n/2 + 1, Sii/2 + λii), and η ~ N(−AS−ii,A), with

$A={\left[\left(S_{ii}+2{\lambda }_{ii}\right){\Omega }_{-i-i}^{-1}+D\right]}^{-1}$ and

$D=2\mathrm{diag}\left(\frac{{\lambda }_{1i}^2}{{\tau }_{1i}},\dots ,\frac{{\lambda }_{(i-1)i}^2}{{\tau }_{(i-1)i}},\frac{{\lambda }_{(i+1)1}^2}{{\tau }_{(i+1)i}},\dots ,\frac{{\lambda }_{di}^2}{{\tau }_{di}}\right)$.

3. Set Ω(b) = Ω.

Output: Sequence Ω(1),..., Ω(B).

Algorithm 3.

GAR - Full Bayes

Input: ${\mathbb{X}}_n$and W; parameters r,r’, s, s’, K; sets${\left\{A_k\right\}}_{k=1}^K$; start values of Ω,α, and β; number of iterations B.

For b = 1,..., B:

1. For i = 1, ...,d:

${\alpha }_i^2|\text{rest}~\Gamma \left((r+d+1)/2,s+C_i\right)$,

where $C_i={\omega }_{ii}+{\sum }_{j\ne i}{\alpha }_j^2{\sum }_{k=1}^K{\beta }_k^2{I}_{A_k}\left(W_{ij}\right){\omega }_{ij}^2{\tau }_{ij}^{-1}$.

2. For k = 1,..., K:

${\beta }_k^2|\text{rest}~\Gamma \left(\left(r^{\prime }+D_k\right)/2,s^{\prime }+E_k\right)$,

where $D_k={\sum }_{i<j}{I}_{A_k}\left(W_{ij}\right)$, and

$E_k={\sum }_{i<j}{I}_{A_k}\left(W_{ij}\right){\omega }_{ij}^2{\alpha }_i^2{\alpha }_j^2{\tau }_{ij}^{-1}$.

3. Do steps 1–2 of Algorithm 2 with current Λ.

4. Set Ω(b) = Ω and Θ(b) = {α, β}.

Output: Sequences {Ω(1),Θ(1)},…, {Ω(B),Θ(B)}.

Algorithm 4.

GAR - Empirical Bayes

Input: ${\mathbb{X}}_n$ and W; start values of Ω and Θ.

1. E-step: For i < j, approximate ${\overline{\omega }}_{ij}=\mathbb{E}\left[|{\omega }_{ij}||{\mathbb{X}}_n,\Theta ,W\right]$ by Algorithm 2.

2. M-STEP: Iterate a)-b) until convergence:

a) For i = 1,..., d, update αi according to Eq. (21).

b) Obtain g as the rate function of the Gamma regression of $y_{ij}=2{\alpha }_i{\alpha }_j{\overline{\omega }}_{ii}$ on Wij, i < j.

3. Iterate 1–2 until convergence.

Output: Estimate of Θ.

4.3. Computational efficiency of estimators

Table 1 contains the computation times of the graph estimators considered for d = 50,100 and n = 200,500, using the programming language R, CPU Quad-core 2.6 GHz Intel Core i7, and RAM 16 GB 2133 MHz DDR4. These times could be improved substantially by using a lower level language such as C++. Glasso, AGlasso, SPL, and SAGlasso are fitted with tuning parameter optimization based on tenfold cross-validation involving 500 random splits over a fine grid of 20 values of the tuning parameter about its optimal value. GAR full Bayes (Algorithm 3; K = [$\sqrt{d}$]) involved B = 2000 iterations, where the Gibbs sampler converged after about 300 iterations. GAR empirical Bayes (Algorithm 4; splines with 3 knots) involved 30 EM iterations, each including 500 iterations of Gibbs sampler for the E-step; the efficiency of this method may be improved by replacing the Gibbs sampler with some alternate faster approximation of Eq (20)

Table 1.

Computational time in seconds with 95% confidence intervals for d = 50,100 and n = 200,500.

Method	d = 50		d = 100
	n = 200	n = 500	n = 200	n = 500
Glasso	150 ± 1	160 ± 1	1198 ± 6	1057 ± 10
AGlasso	65 ± 1	87 ± 1	410 ± 3	501 ± 7
SPL	541 ± 19	641 ± 12	4626 ± 41	2709 ± 23
BAGlasso	96 ± 3	97 ± 2	825 ± 5	824 ± 5
SAGlasso	91 ± 1	116 ± 2	550 ± 18	626 ± 8
GAR-FB	92 ± 1	92 ± 1	827 ± 11	823 ± 3
GAR-EB	654 ± 22	724 ± 5	4163 ± 38	4337 ± 40