Rapid fluctuations in functional connectivity of cortical networks encode spontaneous behavior

Abstract

Experimental work across species has demonstrated that spontaneously generated behaviors are robustly coupled to variation in neural activity within the cerebral cortex. Functional MRI data suggest that temporal correlations in cortical networks vary across distinct behavioral states, providing for the dynamic reorganization of patterned activity. However, these studies generally lack the temporal resolution to establish links between cortical signals and the continuously varying fluctuations in spontaneous behavior observed in awake animals. Here, we used wide-field, mesoscopic calcium imaging to monitor cortical dynamics in awake mice and developed an approach to quantify rapidly time-varying functional connectivity. We show that spontaneous behaviors are represented by fast changes in both the magnitude and correlational structure of cortical network activity. Combining mesoscopic imaging with simultaneous cellular resolution 2-photon microscopy also demonstrated that correlations among neighboring neurons and between local and large-scale networks also encode behavior. Finally, the dynamic functional connectivity of mesoscale signals revealed subnetworks that are not predicted by traditional anatomical atlas-based parcellation of the cortex. These results provide new insight into how behavioral information is represented across the neocortex and demonstrate an analytical framework for investigating time-varying functional connectivity in neural networks.

Introduction

Cognitive functions such as perception and attention require the dynamic activity of neuronal networks defined by synaptic connectivity over local and long-range spatial scales^1–3. Moreover, animals cycle through behavioral states categorized by a variety of physical markers, including pupil dilation, facial movement, and locomotion, that are associated with changes in cognitive performance^4–9. Such variation in motor behaviors and arousal are themselves strongly coupled to fluctuations in neural activity, and several recent studies have demonstrated clear modulation of spontaneous and sensory-evoked firing rates associated with transitions between behavioral states^10–14. Work in both human and non-human subjects suggests that the spatiotemporal correlations between neural signals in large-scale networks spanning multiple brain regions also co-vary with state^1,14–20. These correlations are often viewed as functional connectivity between nodes in a network, reflecting either true structural (synaptic) connections or common inputs²¹. From these results, behavioral state fluctuations might be viewed as shifts between distinct network configurations optimized for contextually-relevant cognitive functions.

While many analyses of state-dependent correlations in cortical networks rely on binning data within identified epochs (e.g., sleep versus wakefulness or quiescence versus arousal), others have explored continuously time-varying functional connectivity^15,22. However, methodological challenges to monitoring large-scale brain dynamics at high spatial and temporal resolution have obscured the investigation of rapid co-variation in behavioral state and network connectivity. As neurons can be exquisitely sensitive to patterned or synchronized input²³, it seems reasonable to hypothesize that fast changes in network correlations are closely linked to the integrative function of single cells. Furthermore, the short-term correlation between two neuronal signals is a non-linear, second-order function of those signals that expresses the dynamics of their connectivity. Thus, temporal dynamics of activity and short-term correlations might differentially represent behaviorally relevant information. Nevertheless, the relative contributions of dynamic neural activity versus dynamic inter-node correlations to decoding behavior is largely unknown.

To explore this question, we used widefield mesoscopic and cellular resolution 2-photon calcium (Ca2+) imaging, both independently and simultaneously, to monitor neural activity in the awake, head-fixed mouse. We developed a methodological strategy that views the dynamic correlations between individual cortical regions and/or neurons as a “graph of graphs”, extracting their time-varying latent variables through non-Euclidean diffusion embedding²⁴. We then asked how accurately these signals could be used to decode spontaneous fluctuations in behavior (measured by pupil dilation, facial movements, and locomotion), and sensory input. Our results show that time-varying network correlations carry significant information about behavioral metrics that can outperform analyses based on the dynamics of activity alone. In addition, the dynamic multimodal correlations between local cellular and large-scale networks are also significantly predictive of behavior. We then show that both cortical parcels and single neurons are dynamically correlated to one of two broad mesoscale networks that are distinct from traditional anatomical boundaries. Overall, these findings demonstrate that rapid fluctuations in functional connectivity across spatial scales provide a robust representation of spontaneous behavior.

Results

Monitoring spontaneous behavior and cortical dynamics

We carried out widefield, mesoscopic calcium imaging³ in awake, head-fixed mice expressing the red fluorescent indicator jRCaMP1b²⁵ (Figure 1a). Indicator expression was mediated by neonatal injection of wild-type mice with AAV9-Syn-jRCaMP1b (see Methods)^26,27. We simultaneously monitored cortical activity and spontaneous behavioral metrics including pupil diameter, facial movements, and locomotion (Figure 1b–g, see Methods)^13,28. Pupil diameter and locomotion were extracted from video recording of the eye and running wheel-speed, respectively. Facial movements were derived from video data with FaceMap²⁸, and the first extracted component was used for most subsequent analyses. Although some studies have used categorical definitions of behavioral state according to thresholding of motor signals^9,13,14,29, our data suggest that these metrics are continuously distributed across a range of rapidly varying values (Calinski-Harabasz Index values vs. # of clusters for K-means clustering using 2–6 clusters, p=0.99, ANOVA, Figure 1b–c). We also find that these signals are only modestly correlated with each other (Figure 1d, Supplemental Figure S1). These results suggest pupil, locomotion, and facial metrics may reflect at least partially distinct, highly dynamic latent variables, such as different neuromodulatory influences¹⁴. Here, we focused on the general ability of functional connectivity to encode behavior and therefore separately compared the relationships of time-varying network activity and correlations to each of these observable variables.

Figure 1. Mesoscopic imaging of cortical activity and functional connectivity.

a, Schematic illustrating the setup for simultaneous behavioral monitoring and mesoscopic calcium imaging. b, Scatter plot illustrating the distribution of Z-scored behavioral metric values (locomotion, facial movement, and pupil size) over a 400 second window for the example mouse shown in (e-i). c, Population data showing Calinski-Harabasz index values for K-means clustering of behavioral metrics for all subjects. d, Population data (n=6 independent mice) showing average (±SEM) Pearson’s R² values for the relationships between wheel (W), pupil (P), and facial movements (F) for all subjects. e, Example time series from one animal showing cortical activity across the cortex. Each trace corresponds to one LSSC-based parcel. f, Heat map illustrating the time-series of pairwise correlations between each parcel from (e). Data are sorted by increasing standard deviation. g, Time series of behavioral metrics corresponding to the data shown in (e) and (f). h, Example LSSC-based functional parcellation of the neocortex for the data shown above. Left and right images are for the timepoints indicated by vertical red lines. i, Example pairwise correlation matrices for the data in (e) at the time points indicated.

After normalization and hemodynamic correction of imaging data (see Methods)^14,30, we segmented the cortex into functional parcels using a graph theory-based approach that relies on spatiotemporal co-activity between pixels (LSSC, Figure 1h, Supplemental Figure S2)³¹. This approach yields comparable reconstruction errors for a similar number of parcels in comparison to either the Allen Institute CCFv3 anatomy-based atlas³² or a uniform grid-based segmentation (see Methods). Principal component analysis yielded similar reconstruction errors as LSSC, and localized semi-nonnegative matrix factorization (LocalNMF)³³ gave near-zero errors, though both approaches generate non-disjoint parcels that may be more difficult to interpret (Supplemental Figure S2). We next extracted ${\mathit{x}}_t$, the time-varying fluctuations in the fluorescence signal associated with each parcel (Figure 1e, see Methods). As expected, variation in activity appeared to be coupled to changes in behavioral metrics over rapid (sub-second) time scales (Figure 1e–h). We then calculated the time-varying, pair-wise correlations ${\hat{C}}_t$ between LSSC parcels using a sliding 3-second window (0.1 second step-size, Figure 1 f–i, see Methods). On average, correlations across the cortex were high but with large fluctuations over time (average r=0.6±0.03, average CV_r=0.4±0.1, n=6 mice). Indeed, the moment-to-moment values also appeared to co-vary with rapid behavioral changes (Figure 1f–i).

Dynamic functional connectivity encodes spontaneous behavior

We next began with a basic assumption that behavior can be represented as some function of multi-dimensional, time-varying neural activity. That is:

$$b_t=f\left({\mathit{x}}_t\right)$$ (1)

Here, ${\mathit{x}}_t$ is an N-dimensional vector corresponding to the time-varying neural activity across $N$ cortical parcels at time $t$. This relationship can be approximated using the first two terms of a standard Taylor expansion (see Methods):

$$b_t\approx {\beta }_0+{\mathit{\beta }}_1^T{\mathit{x}}_t+{\sum }_i{\sum }_j{\beta }_2(i,j){\hat{C}}_t(i,j)+ϵ$$ (2)

Again, ${\mathit{x}}_t$ is the time-varying neural activity and the second-order term ${\hat{C}}_t(i,j)$ corresponds to the time-varying pairwise sample correlations between parcels $i$ and $j$. Representing behavior as a linear combination of time-varying neural signals is a common approach^9,34–37, and we hypothesized that combining both a linear term in activity ${\mathit{x}}_t$ and correlations ${\hat{C}}_t$, which are a nonlinear second-order function of ${\mathit{x}}_t$, would significantly improve decoding accuracy.

In an initial effort, fitting a linear ridge regression model for behavioral dynamics whose predictors are time-varying cortical activity and functional connectivity led to poor predictive power (Supplemental Figure S2) due to over-fitting caused by the high-dimensionality of pairwise correlations ${\hat{C}}_t$ (~10³ pairs per animal). Therefore, we developed a novel strategy to extract a lower dimensional representation, ${\varphi }_t$, capturing the intrinsic dynamics of the correlational signals using Riemannian geometry and diffusion embedding²⁴. Each correlation matrix over a short temporal window (e.g., 3 seconds) can be viewed as a graph whose $N$ nodes are cortical parcels connected by weighted edges equal to the instantaneous pairwise correlation coefficients between parcels. Sliding the window over time (Δt=0.1 seconds) produces a series of correlation matrices, which can also be viewed as a time-varying graph (Figure 2a). We then built a “graph of graphs”, where each node is now a time-point represented by the correlation matrix at that time (see Methods). A distance measure between correlation matrices is necessary to set the edge weights of this temporal graph. Since correlation matrices lie on a non-Euclidean Riemannian manifold, Euclidean distances do not properly represent similarity in this space (Figure 2b). Therefore, we used Riemannian geometry to calculate pairwise geodesic distances between correlation matrices. We applied diffusion embedding to the graph of graphs and extracted the low dimensional representation, ${\varphi }_t$, of the temporal dynamics of functional connectivity (Figure 2c).

Figure 2. Dynamic functional connectivity encodes rapid behavioral variations.

a, Example sequential pairwise, parcel-based correlation matrices, derived from a sliding window applied to neural activity across the cortex. b, Left, Schematic illustrating the cone-shaped Riemannian manifold used to calculate distances between correlation matrices. The Riemannian measurement reflects geodesic distance that is ignored when using Euclidean distance. Right, Illustration of a “graph of graphs”, whose nodes are individual matrices and edges are weighted by the length of the geodesic arc along the Riemannian cone, that is used to extract diffusion embedding (${\varphi }_t$) components. c, Example diffusion embedding components capturing dynamics of functional connectivity ${\varphi }_t$. d, Time series for behavioral metrics corresponding to data in (c). e, Example behavioral data (blue traces) from (d) showing fluctuations in pupil diameter, facial movement, and locomotion superimposed on predicted behavior (red traces) estimated using a joint model based on time-varying activity and embedded correlations. f, Population data (n=6 independent mice) showing average (±SEM) prediction accuracy (R²) for modeling behavior variables using a joint model of activity and embedded correlations (blue), joint model with shuffled ${\varphi }_t$ (yellow), joint model with shuffled activity (red), single predictor model using activity (pale yellow), and single predictor model using ${\varphi }_t$ (pale red). * indicates p<0.05 for two sided paired t-test (see main text). Full model compared to shuffling ${\varphi }_t^{\left(20\right)}$: p=0.001 for pupil, p=0.002 for face and p=0.01 for wheel. Full model compared to shuffling $x_t$: p=0.02 for pupil, p=0.006 for face, p=0.003 for wheel.

We constructed a cross-validated linear regression model combining the cortical activity for all LSSC parcels (${\mathit{x}}_t$, ranging from 48–53 parcels per animal across 6 mice) and the first 20 leading components of the embedded correlations, denoted by ${\varphi }_t^{\left(20\right)}$ to predict the continuously varying behavioral signals for pupil diameter, facial movement, or locomotion (Figure 2d–e). We found that behavior can be robustly decoded by this joint model (Pupil: R²=0.52±0.04; Face: R²=0.59±0.04; Wheel: R²=0.45±0.06; n=6 mice, Figure 2e–f). Prediction accuracy fell off rapidly for higher-order components of facial movement (Supplemental Figure S2). Additionally, accuracy was impaired when using either raw correlations, Euclidean rather than Riemannian distance for the diffusion embedding, or PCA for dimensionality reduction (Supplemental Figure S2). Smoothing or diffusion embedding of activity did not significantly impact the analysis (Supplemental Figure S2). Moreover, predictive performance was robust to changes in model parameters, did not improve with the inclusion of more than 20 embedding components, and did not vary appreciably for window lengths of 3–10 seconds (Supplemental Figure S2). We also compared model prediction accuracy for different methods of cortical segmentation. Analyses with LSSC was similar to grid-based and anatomical (CCFv3) parcels as well as non-disjoint segmentations derived from either PCA or LocalNMF (Supplemental Figure S2). Finally, to further quantify the potential for decoding behavior from our neural data, we considered a fully connected artificial neural network (see Methods). However, this non-linear approach showed no improvement over our joint linear model (Supplemental Figure S2).

To investigate the relative contributions of activity versus connectivity dynamics in modeling behavior, we recreated the joint model while temporally (circularly) shuffling one of the two predictors. Shuffling either term significantly impaired prediction accuracy relative to the unshuffled full model (Pupil: R²=0.2±0.04, p=0.001 for shuffling ${\varphi }_t^{\left(20\right)}$, R²=0.48±0.04, p=0.02 for shuffling ${\mathit{x}}_t$; Face: R²=0.38±0.05, p=0.002 for shuffling ${\varphi }_t^{\left(20\right)}$, R²=0.49±0.06, p=0.006 for shuffling ${\mathit{x}}_t$; Wheel: R²=0.31±0.07, p=0.01 for shuffling ${\varphi }_t^{\left(20\right)}$, R²=0.32±0.05, p=0.003 for shuffling ${\mathit{x}}_t$; Paired t-test, Figure 2f). Surprisingly, models in which correlational data were preserved performed similarly or better at decoding behavior than activity-preserved models, reaching significance for variations in pupil diameter (Pupil: p=0.002; Face: p=0.06; Wheel: p=o.37; Paired t-test, Figure 2f). To further examine the ability of activity or connectivity signals to independently predict behavioral signals, we generated single predictor models which produced similar results (Figure 2f).

Finally, we note that, while the time-averaged activity and pairwise correlations significantly differ for high versus low behavioral state epochs (see Methods), the temporal dynamics of averaged cortical activity (across all parcels) or correlations (across all pairs of parcels) are poorly predictive of the rapid fluctuations in behavior (Supplemental Figure S2). Altogether, these findings indicate that inclusion of rapidly time-varying functional connectivity significantly improves decoding power for modeling of behavioral state, suggesting that cortical network function relies not only on the absolute amount of activity but also on the dynamic coordination of activity across widespread areas.

Sensory inputs modestly influence network connectivity

Spontaneous cortical activity likely reflects latent signals corresponding to internally generated brain processes. Thus, we asked whether extrinsic sensory information was similarly represented by large-scale networks. First, we looked at responses to visual stimulation outside of any learning or task-based conditions. We presented the mouse with a series of drifting sinusoidal gratings (see Methods) and quantified evoked activity using mesoscopic calcium imaging. Contrast-dependent responses were largest in visual areas but were also observed broadly across other cortical regions. However, evoked responses had minimal impact on the correlational structure of activity across the cortex (Supplemental Figure S3). Linear modeling showed that the stimulus could be robustly decoded using activity in visual cortex, with prediction accuracy exhibiting strong contrast-dependence (Spearman’s R=0.9±0.1, p=0.0001). However, prediction accuracy using mesoscopic correlational structure was significantly lower (p=5.6e-5, paired T-test) and not coupled to stimulus contrast (Spearman’s R=0.1+0.2, p=0.25, Supplemental Figure S3).

We then asked whether network correlational structure was more affected by sensory inputs with learned association to behavior. We adopted a visually cued eyeblink task in which the drifting grating stimulus is paired with an aversive corneal air puff (see Methods). Our previous work showed that task performance is contrast-dependent and requires V1 activity⁹. As with untrained animals, the visual stimulus evoked a large response in V1 and surrounding areas but did not produce significant changes in pairwise correlations between areas. However, linear modeling based on either V1 activity or mesoscopic correlations could robustly predict trial occurrence in a contrast-dependent manner (Spearman’s R=0.91±0.05, p=2.0e-5 for activity and 0.59+0.17, p=0.01 for embedded correlations), although the former again significantly outperformed the latter (p=5.8e-5, paired t-test, Supplemental Figure S3). These results suggest that brief sensory inputs drive large fluctuations in cortical activity, with their impact on network correlations exhibiting sensitivity to the behavioral relevance of the stimulus.

Dynamic correlations of cellular networks predict behavior

To determine the generalizability of our approach and also examine encoding by neural correlations at a different spatial scale, we monitored local circuit activity using cellular resolution 2-photon calcium imaging of GCaMP6s-expressing neurons³⁸ in the primary visual cortex (see Methods, Figure 3a–b). As above, we looked at both time-varying activity ${\mathit{x}}_t$ and embedded pair-wise correlations ${\varphi }_t$ for identified neurons simultaneously with measurements of pupil diameter, facial movement, and locomotion (Figure 3c–f). Unlike large-scale network signals, correlations between neurons were broadly distributed around zero (average r=−0.001±0.006, average CV_r=6.1±1.47, n=6 mice).

Figure 3. Local circuit dynamics encode spontaneous behavioral variation.

a, Schematic illustrating the setup for simultaneous behavioral monitoring and 2-photon calcium imaging. b, Example field of view showing individual GCaMP6s-expressing neurons in visual cortex. Similar results were obtained for each of 6 mice. c, Example time series showing neuronal activity for all neurons in the field of view. d, Heat map illustrating the time-series of pairwise correlations between each neuron from (c). Data are sorted by increasing standard deviation. e, Example of the first six diffusion embedding components based on data in (d). f, Time-series for behavioral metrics corresponding to data in (c-d). g, Population data (n=6 independent mice) showing average (±SEM) prediction accuracy (R²) for modeling behavior variables using a joint model of activity and embedded correlations (blue), joint model with shuffled ${\varphi }_t$ (yellow), joint model with shuffled activity (red), single predictor model using activity (pale yellow), and single predictor model using ${\varphi }_t$ (pale red). * indicates p<0.05 for two sided paired t-test (see main text). Full model compared to shuffling ${\varphi }_t^{\left(20\right)}$: p=0.04 for pupil, p=0.03 for face and p=0.02 for wheel. Full model compared to shuffling ${\mathit{x}}_t$: p=0.0002 for pupil, p=0.006 for face p=0.003 for wheel.

We then generated a cross-validated linear model combining activity and embedded correlation dynamics across cells and attempted to predict rapid fluctuations in behavior. As with mesoscopic imaging, cellular data also robustly predicted behavior (Pupil: R²=0.59±0.04; Face: R²=0.44±0.08; Wheel: R²=0.39±0.1; n=6 mice, Figure 3). Again, modeling performance was poorer using raw correlations and Euclidean distances for embedding but was robust to changes in model parameters (Supplemental Figure S4).

To calculate the relative contributions of activity versus correlations in the joint model, we similarly shuffled one of the two predictors. As above, shuffling either variable significantly impaired prediction accuracy relative to the unshuffled model (Pupil: R²=0.53±0.05, p=0.04 for shuffling ${\varphi }_t^{\left(20\right)}$, R²=0.42±0.05, p=0.0002 for ${\mathit{x}}_t$; Face: R²=0.39±0.09, p=0.004 for shuffling ${\varphi }_t^{\left(20\right)}$, R²=0.32±0.05 p=0.03 for shuffling ${\mathit{x}}_t$; Wheel: R²=0.32±0.1, p=0.04 for shuffling ${\varphi }_t^{\left(20\right)}$, R²=0.26±0.07, p=0.02 for shuffling ${\mathit{x}}_t$; Paired t-test, Figure 3g). Models preserving either the activity or correlational data gave similar accuracy, with activity-based analysis showing modestly better performance for pupil fluctuations (p=0.025 for Pupil, p=0.14 for Face, p=0.15 for Wheel, Paired t-test, Figure 3g). Single-predictor models again produced similar results (Figure 3g). In summary, applying our approach for quantifying time-varying correlations in neural data to cellular resolution imaging, we again find that including dynamic functional connectivity significantly enhances prediction accuracy in models linking neural signals to fluctuations in behavioral state.

Dynamic connectivity suggests distinct cortical subnetworks

The improved accuracy of behavioral prediction using embedding of mesoscopic correlation matrices suggests they may reflect underlying principles of structural organization in large-scale cortical networks. We therefore examined the spatial interpretation of ${\varphi }_t$ by asking how the time-varying correlation for each pair of parcels is represented by the overall embedding. This approach allows us to determine whether the embedding is primarily capturing subsets of pairwise correlations. We quantified the goodness-of-fit using ${\varphi }_t^{\left(20\right)}$ to model the correlation time series between a target parcel and each of the other parcels across the cortex (Figure 4a–b, see Methods). Averaging these goodness-of-fit matrices across all animals (n=6 mice) revealed substantial within-subject spatial heterogeneity that was conserved across different individuals. The overall embedding primarily represented correlations between each target parcel and one or both of a posterior and anterolateral subdivision of the cortex (Figure 4c, Supplemental Figure S5). This spatial pattern was clearly evident after making a grand average across all parcels and animals (Figure 4d). Intuitively, this result indicates that independent of behavior, the dynamic large-scale correlations of cortical areas are dominated by the interrelationship of each cortical parcel with one or both of these two subnetworks. Surprisingly, this functional organization is distinct from the cortical segmentation defined by traditional, anatomy-based atlases such as the CCFv3 (Figure 4d). In particular, the anterolateral region includes rostral representations in primary motor cortex as well as upper limb, mouth, and nose representations in primary somatosensory cortex. The posterior region includes visual, auditory, and parietal association areas.

Figure 4. Dynamic functional connectivity reveals distinct cortical subnetworks.

a, Left illustration of LSSC-based parcellation, highlighting two parcels corresponding approximately to supplemental motor cortex (MOs) and primary visual cortex (VISp) based on CCFv3. Right, example components of correlation embedding for one animal (black), pairwise time-varying correlation between VISp and MOs (blue), and the predicted VISp-MOs correlation based on embedding. b, Example matrix from one animal showing the goodness of fit (R²) for modeling the time-varying correlations between each pair of parcels using ${\varphi }_t^{\left(20\right)}$. c, Average (n=6 mice) maps showing mean R² values for modeling the pairwise correlations of each cortical parcel with the indicated target parcel (shown in white). d, Left, grand average map showing R² values as in (c) collapsed across all animals (n=6) and all cortical parcels. Right, schematic illustrating the anterolateral (red) and posterior subnetworks (blue) derived from data in (c). Red dashed lines indicate angles for bisecting LSSC parcels into arbitrary subnetworks, with solid line (30°) corresponding to anterolateral/posterior division. e, Left, population data showing the average (±SEM) prediction accuracy for modeling pupil fluctuation based on time-varying activity in two subnetworks determined by bisecting lines in (d). Right, average (±SEM) prediction accuracy for pupil fluctuation based on time-varying correlation between two subnetworks determined by lines in (d).

To further examine whether coordinated activity across this anterolateral/posterior partition preferentially encodes spontaneous behavioral variation, we systematically bisected the cortex with a line rotated about the midpoint (Figure 4d). We then modeled behavior using either (1) the activity in these two parcels or (2) the correlations between the two parcels and quantified the prediction accuracy as a function of the angle of division. Accuracy using time-varying activity was independent of angle, suggesting limited spatial heterogeneity in how behavioral metrics are encoded by fluctuations in the magnitude of cortical signals (Figure 4e, Supplemental Figure S5). In contrast, accuracy using the time-varying correlations between the two parcels was significantly higher for the angle matching the anterolateral-posterior division noted above (Figure 4d). This result was relatively insensitive to the window length used for embedding (Supplemental Figure S5). Overall, these findings suggests that encoding of spontaneous behaviors by large-scale correlations is spatially organized into distinct subnetworks in the neocortex.

Dynamic connectivity across spatial scales encodes behavior

Our analyses revealed that spontaneous behaviors can be accurately decoded from the temporal dynamics of correlations between neural signals at both local circuit and mesoscopic spatial scales. However single neurons are embedded in large-scale networks by virtue of long-range synaptic connections. Thus, we wondered whether the functional connectivity across these scales was similarly dynamic. To this end, we carried out simultaneous wide-field and cellular 2-photon imaging (Figure 5a–b, see Methods)²⁶. We first analyzed these multimodal data sets separately as described above and quantified the accuracy with which linear models based on time-varying activity ${\mathit{x}}_t$ or embedded correlations ${\varphi }_t^{\left(20\right)}$ could predict behavior. Activity was predictive for mesoscopic (Pupil: R²=0.13±0.03; Face: R²=0.14±0.04; Wheel: R²=0.12±0.04; n=7 mice) and cellular (Pupil: R²=0.31±0.04; Face: R²=0.4±0.07; Wheel: R²=0.5±0.08) data. Similarly, embedded correlations were also predictive for mesoscopic (Pupil: R²=0.35±0.03; Face: R²=0.38±0.06; Wheel: R²=0.51±0.06) and cellular (Pupil: R²=0.14±0.03; Face: R²=0.12±0.05; Wheel: R²=0.15±0.06) data (Figure 5c). As above, correlations-based prediction accuracy was greater for mesoscopic data (Pupil: p=0.002; Face: p=0.004; Wheel: p=0.0002; paired t-test, n=7 mice) and activity-based prediction accuracy was greater for cellular data (Pupil: p=0.0001; Face: p=0.001; Wheel: p=0.0004; paired t-test, n=7 mice).

Figure 5. Functional connectivity across spatial scales encodes behavior.

a, Schematic illustrating the setup for simultaneous mesoscopic and 2-photon imaging. b, Left, example mesoscopic imaging frame and schematic of microprism placement in the contralateral hemisphere. Right, example 2-photon imaging frame collected through the prism. c, Population data (n=7 independent mice) showing average (±SEM) prediction accuracy (R²) for modeling behavior variables using either activity (yellow) or ${\varphi }_t$ (red) for mesoscopic or 2-photon data. * indicates p<0.05 for two sided paired t-test (see main text). Comparing ${\varphi }_t^{\left(20\right)}$ to ${\mathit{x}}_t$: for mesoscopic data, pupil: p=0.002; face: p=0.004; wheel: p=0.0002. For cellular data, pupil: p=0.0001; face: p=0.001; wheel: p=0.0004. d, Example sequential multimodal correlation matrices, derived from a sliding window applied to neural activity from mesoscopic (parcels) and 2-photon (cells) imaging, used for diffusion embedding. e, Dynamic multimodal correlation time series for three example cells, where each row represents a mesoscopic parcel. The standard deviation of correlation values over time, averaged across all rows is indicated. f, Example of the first 20 diffusion embedding components from the same animal (n=243 cells, 47 parcels). g, Time series for behavioral metrics corresponding to data in (e-g). h, Population data (n=7 independent mice) showing average (±SEM) prediction accuracy (R²) for modeling behavior variables using ${\varphi }_t$ derived from the embedding of dual mesoscopic and 2-photon correlations. i, Example maps for the cells in (e) showing R² values for modeling the correlation of the cell with each parcel using the overall diffusion embedding. j, Grand average map showing R² values as in (i) collapsed across all animals (n=6) and all cells and cortical parcels.

To explore functional connectivity across spatial scales, we developed a strategy to calculate the time-varying correlations between cells and parcels for a sliding 3-second window followed by diffusion embedding (Figure 5d–e, see Methods). This analysis revealed considerable heterogeneity in the degree of multimodal correlation dynamics exhibited by different cells, measured as the standard deviation of correlation fluctuations averaged for a single cell across all brain parcels across the imaging session (Figure 5e, Supplemental Figure S6, see Methods). The variation in both the correlations and the embedding components appeared to track with behavioral metrics (Figure 5e–g). Indeed, the embedding of the correlations for the dual imaging data could predict fluctuations in pupil diameter, facial movement, and locomotion (Figure 5h), with results robust across a range of model parameters (Pupil: R²=0.29±0.06; Face: R²=0.16±0.03; Wheel: R²=0.22±0.06; n=7 mice, Supplemental Figure S6). Shuffling any one of the correlation signals (mesoscopic, cellular, or dual) did not significantly reduce model accuracy, suggesting a substantial amount of overlap in the information represented across spatial scales.

We explored the spatial interpretation of the dual mesoscopic-cellular embedding by quantifying the accuracy with which the correlations between a single neuron and the mesoscopic parcels are represented by the overall embedding. In general, cells with the most dynamic correlations (largest standard deviation) exhibited the strongest spatial heterogeneity in their modeling accuracy (Figure 5i, Supplemental Figure S6). However, the spatial pattern was generally conserved for all cells and was clearly evident after averaging across the population (Figure 5j), again showing a division of the cortex into anterolateral and posterior subnetworks, with cells in visual cortex being dynamically correlated most strongly with the latter. These results provide an additional independent demonstration that these subdivisions reflect a fundamental organizing principle in the cortex.

Discussion

Our results show that functional connectivity in cortical networks is highly dynamic, varying on a sub-second time-scale that tracks with continuous metrics of spontaneous behaviors. Including these dynamic correlations in a linear model predicted behavioral fluctuations with high accuracy. This result was true for both large-scale networks monitored with mesoscopic calcium imaging and local networks monitored at cellular resolution with 2-photon microscopy. Moreover, combining these modalities revealed that behavior was also accurately represented by the dynamic correlations between local and cortex-wide networks. The spatial organization of dynamic correlations between either parcels or neurons and cortex-wide activity revealed two distinct subnetworks not obviously predicted from standard anatomical divisions.

The representation of behavioral information by time-varying cortical signals has been a focus of recent studies using diverse approaches to monitor brain activity^{1,2,10,15,22,39}. In rodent models, variations in behavioral state or arousal are coupled to changes in firing rate, pairwise correlations, and neuromodulatory signaling^{11–14,40,41}. In particular, arousal- and motor-related variables (e.g., pupil diameter, locomotion, whisking) are represented at the cellular and network scale^11,12,14. Several groups have also demonstrated that the spatial patterns of large-scale activity in the neocortex are markedly different when comparing across state^14,16,20. For example, spontaneous cortical activity can be decomposed using various methods into repeating spatiotemporal motifs that may correspond to sensory or motor signals²⁰. Similarly in human subjects, shifts in wakefulness correspond to changes in average resting state connectivity⁴², resting state fluctuations predict somatosensory perception⁴, and working memory-based task performance corresponds to spatially heterogeneous variation in timescales of patterned activity^43,44. Time-varying functional connectivity in fMRI studies has received considerable recent focus, with ongoing debates as to the mechanisms and behavioral relevance of coordinated signals over large-scale networks¹⁵. Indeed, a recent study also demonstrated the utility of Euclidean manifold learning to reduce the dimensionality of connectivity matrices²², suggesting the generality of this approach.

While some prior efforts to characterize spontaneous behaviors have relied on categorical definitions of state, our data indicate that variations in pupil diameter, facial movement, and locomotion do not appear to cluster into distinct regimes, a result more consistent with continuously and rapidly varying states. The exact mapping of these observed motor signals onto latent neural variables (and whether they are independent) is not well understood. For example, all three behavioral metrics are linked to variation in cholinergic and adrenergic activity^41,45,46. We also recently found that acetylcholine release was more robustly linked to facial movement (e.g., whisking) than locomotion¹⁴. Both Stringer et al.¹² and Musall et al.¹¹ demonstrated that rapid dynamics of behavior variables could accurately encode neural activity across the cortex. Here, we further explored the dynamics of functional connectivity expressed as the correlation between cortical parcels or neurons. Intuitively, time-varying activity and pairwise correlations can be viewed as first- and second-order terms in a Taylor expansion of a function relating behavior to neural signals. Thus, correlations cannot be linearly derived from the underlying activity and represent a potential mechanism to encode an independent component of behavioral dynamics. This conclusion is strongly supported by our results, where shuffling either activity or correlations significantly reduces modeling accuracy for both mesoscopic and 2-photon data.

The temporal scale of the neural and behavioral dynamics is similar to synaptic integration windows for single cells, suggesting the hypothesis that neurons may be sensitive to convergent synaptic input driven by correlated large-scale activity. Our results combining mesoscopic and 2-photon imaging demonstrate that the functional connectivity across these divergent spatial scales is also dynamic and accurately predictive of fluctuations in behavior. Thus, we propose that network activity within and across spatial scales in the neocortex is coordinated as a function of spontaneous behaviors. In the near future, ongoing development of multi-modal approaches, combining fluorescence imaging, fMRI, and electrophysiology^26,47–49, will likely drive additional discoveries into the functional organization of brain networks in diverse systems.

Our method for viewing time-varying functional connectivity in cortical networks as a graph of graphs provides a conceptual and analytical framework for extracting the intrinsic dynamics of short-term correlations and uses Riemannian geometry to correctly evaluate distances between correlation matrices extracted at different time points (we note that similar analyses using Euclidean geometry yielded substantially poorer prediction accuracy). These distances are then used to set the weights of a graph-of-graphs, allowing us to extract a low-dimensional representation for the manifold of the correlations and capture their underlying dynamics. A similar strategy has also been applied to fMRI data²². Using this approach and including both first-order (activity) and nonlinear second-order (embedded functional connectivity) terms for modeling behavior enabled us to significantly improve decoding power. Our method was generalizable for three different data sets (mesoscopic and 2-photon imaging alone and in combination) and yielded the surprising finding that higher-order statistics (i.e., correlational signals) can produce similar or better predictive accuracy than time-varying changes in activity.

Several distinct strategies have been developed to analyze the spatiotemporal organization of network activity, including singular value decomposition and non-negative matrix factorization^16,33. Here, we show that parcellation of cortical regions³¹ followed by embedding of time-varying correlations based on Riemannian geometry, provides a robust means to quantify dynamic functional connectivity that accurately decodes spontaneous fluctuations in behavior. Notably, our overall findings were similar for a number of either structural or functional approaches for segmenting the cortex. We did find that LSSC-based parcellation gave robust results with a good balance of reconstruction error and number of disjoint parcels. With the increasing interest in analysis of neural manifolds, our results also highlight the necessity of considering the geometry of the manifold (i.e., Riemannian versus Euclidean) on which the data lie to accurately reveal their intrinsic representation. Finally, our approach modestly out-performed a neural network-based model. This finding suggests that much of the information linking neural signals and spontaneous behavior is present in the dynamics of the activity and correlations.

Surprisingly, the representation of task-independent sensory information in mesoscopic correlations was relatively weak. However, the representation of conditioned sensorimotor event was more robust, indicating that functional connectivity may be altered by external inputs with learned behavioral relevance. These finding are consistent with recent work suggesting spontaneous behavior and external stimuli can be represented in orthogonal dimensions¹², but task-relevant behaviors are broadly present in large-scale patterns of network activity^11,29,50. Additionally, we and others have shown that training can modify the sensory and motor representations by single cortical neurons^9,51–53, and future studies must continue to explore how development, experience, and learning shape the functional organization of large-scale networks.

Finally, our results show that the cortex can be spatially segmented into two broad subnetworks, an anterolateral and posterior division, a functional division that emerges from analysis of spontaneous activity but also reflects variation in behavioral state metrics. We previously demonstrated such a division based on correlations between single cell activity and mesoscopic cortical signals²⁶, a result that is also present in our dual mesoscopic and 2-photon data here. Work from other labs using widefield imaging also provided evidence for the existence of spatially similar state-dependent subnetworks^17,48. Intriguingly, large-scale network activity may also be cell type-specific, as recent findings found distinct patterns for different populations of layer 5 projection neurons^29,50. Interestingly, our functional subnetworks do not map readily onto standard anatomical segmentation of the cortex, such as the CCFv3³². For example, the anterolateral division distinctly omits more caudal motor and somatosensory representations, while the posterior division broadly encompasses visual, auditory, and parietal regions while omitting retrosplenial areas. The mechanisms driving these partitions are unclear, but may reflect poorly mapped intracortical connections, heterogeneous neuromodulatory signaling, or indirect connections through subcortical hubs such as the thalamus^14,54. Indeed, recent work using brain-wide individual animal connectome sequencing (BRICseq) found anatomical evidence for inter-region connectivity that can support such disjoint subnetworks⁵⁵. Finally, we hypothesize that the dynamic modulation and plasticity of synaptic strength may support the translation between such structural and functional views of connectivity in cortical networks, a hypothesis that awaits experimental validation.

Methods

All animal handling and experiments were performed according to the ethical guidelines of the Institutional Animal Care and Use Committee of the Yale University School of Medicine. Some of the mesoscopic imaging data were collected as part of a previous study¹⁴, with experimental details provided below for clarity. Analysis results presented here represent wholly new findings and have not appeared elsewhere.

Animals

Adult (P60–100) male and female c57/Bl6 mice (n=19 mice in total) were kept on a 12h light/dark cycle, provided with food and water ad libitum, and housed individually following headpost implants. The animal housing facility was kept between 22–24 degrees C and 60–70% humidity. Imaging experiments were performed during the light phase of the cycle. For most mesoscopic imaging experiments, brain-wide expression of jRCaMP1b²⁵ was achieved via postnatal sinus injection as described previously^26,27. Briefly, Po-P1 litters were removed from their home cage and placed on a heating pad. Pups were kept on ice for 5 min to induce anesthesia via hypothermia and then maintained on a metal plate surrounded by ice for the duration of the injection. Pups were injected bilaterally with 4 ul of AAV9-hSyn-NES-jRCaMP1b (2.5×10^13gc/ml, Addgene). Mice also received an injection of AAV9-hSyn-GRAB_ACh3.0 to express the genetically encoded cholinergic sensor GRAB_ACh3.0⁵⁶, although these data were not used in the present study. Once the entire litter was injected, pups were returned to their home cage. For two-photon imaging experiments and eyeblink conditioning experiments, a similar procedure was used to drive cortex-wide expression of GCaMP6s³⁸. For dual mesoscopic and two-photon imaging experiments, adult (P60–70) mice transgenically expressing GCaMP6s in cortical excitatory neurons (CaMK2a-tTA; tetO-GCaMP6s; VIP-Cre)⁵⁷ were used. These animals were also injected with AAV driving Cre-dependent GCaMP6s and Cre-dependent tdTomato, though all red fluorescent cells were excluded from the present analysis. For eyeblink conditioning experiments, we used adult (P6o-70) CaMK2a-tTA; tetO-GCaMP6s.

Surgical procedures

All surgical implant procedures were performed on adult mice (>P50). Mice were anesthetized using 1–2% isoflurane and maintained at 37°C for the duration of the surgery. For mesoscopic imaging, the skin and fascia above the skull were removed from the nasal bone to the posterior of the intraparietal bone and laterally between the temporal muscles. The surface of the skull was thoroughly cleaned with saline and the edges of the incision secured to the skull with Vetbond. A custom titanium headpost was secured to the skull with transparent dental cement (Metabond, Parkell), and a thin layer of dental cement was applied to the entire dorsal surface of the skull. Next, a layer of cyanoacrylate (Maxi-Cure, Bob Smith Industries) was used to cover the skull and left to cure ~30 min at room temperature to provide a smooth surface for transcranial imaging. A similar procedure was used to prepare mice for two-photon imaging, with the addition of a dual-layer glass window implanted into a small (~4 mm square) craniotomy placed over the left primary visual cortex. The edges of the window were then sealed to the skull with dental cement. For dual mesoscopic and two-photon imaging, a 2mm glass microprism (Tower Optical) was placed on top of a dual-layer glass window implanted over the right primary visual cortex²⁶.

Mesoscopic imaging

Widefield mesoscopic calcium imaging was performed using a Zeiss Axiozoom with a 1x, 0.25 NA objective with a 56 mm working distance (Zeiss). Epifluorescent excitation was provided by an LED bank (Spectra X Light Engine, Lumencor) using two output wavelengths: 395/25 (isosbestic for GRAB_aCh3.0) and 575/25 nm (jRCaMP1b). Emitted light passed through a dual camera image splitter (TwinCam, Cairn Research) then through either a 525/50 (GRAB_ACh3.0 or GCaMP6s) or 630/75 (jRCaMP1b) emission filter (Chroma) before it reached two sCMOS cameras (Orca-Flash V3, Hamamatsu). Images were acquired at 512×512 resolution after 4x pixel binning, and each channel was acquired at 10 Hz with 20 ms exposure using HCImage software (Hamamatsu).

Two-photon imaging

Two-photon imaging was performed using a MOM microscope (Sutter Instruments) coupled to a 16x, o.8 NA objective (Nikon). Excitation was driven by a Titanium-Sapphire Laser (Mai-Tai eHP DeepSee, Spectra-Physics) tuned to 920 nm. Emitted light was collected through a 525/50 filter and a gallium arsenide phosphide photomultiplier tube (Hamamatsu). Images were acquired at 512×512 resolution at 30 Hz using a galvo-resonant scan system controlled by ScanImage software (Vidrio).

Dual mesoscopic and two-photon imaging

Dual imaging was carried out using a custom microscope combining a Zeiss Axiozoom (as above) and a Sutter MOM (as above), as described previously²⁶. To image through the implanted prism, a long-working distance objective (20x, o.4 NA, Mitutoyo) was used. Frame acquisitions were interleaved with an overall rate of 9.15 Hz, with each cycle alternating sequentially between a 920nm two-photon acquisition (512×512 resolution), a 395/25nm widefield excitation acquisition, and a 470/20nm widefield excitation acquisition. Widefield data were collected through a 525/50nm filter into a sCMOS camera (Orca Fusion, Hamamatsu) at 576×576 resolution after 45x pixel binning with 20ms exposure.

Behavioral monitoring

All imaging was performed in awake, behaving mice that were head-fixed so that they could freely run on a cylindrical wheel. A magnetic angle sensor (Digikey) attached to the wheel continuously monitored wheel motion. Mice received at least three wheel-training habituation sessions before imaging to ensure consistent running bouts. During widefield imaging sessions, the face (including the pupil and whiskers) was illuminated with an IR LED bank and imaged with a miniature CMOS camera (Blackfly s-USB3, Flir) with a frame rate of 10 Hz using Spinview software (Flir).

Visual stimulation

For visual stimulation experiments, sinusoidal drifting gratings (2 Hz, 0.04 cycles/degree, 20 degrees of visual field) with varied contrast were generated using custom-written functions based on Psychtoolbox in Matlab and presented on an LCD monitor at a distance of 20 cm from the right eye. Stimuli were presented for 2 seconds with a 5 second inter-stimulus interval.

Eyeblink Conditioning

Conditioning was carried out as previously published⁹. Briefly, mice were acclimated to head-fixation on the running wheel for several days. For training, each trial started with the onset of a 500 ms visual stimulus comprising a sinusoidal drifting grating (2 Hz, 0.04 cycles/degree, 20 degrees of visual field) that co-terminated with a 50 ms air puff directed to the cornea. Each training day comprised 60 pairings at 100% contrast. After reaching stable performance for 2–3 consecutive days, mice were moved to the next stage where contrast was varied randomly across trials. For all sessions, trials were separated by an exponentially distributed inter-stimulus interval ranging from 18–33 seconds.

Data analysis

All analyses were conducted using custom-written scripts in MATLAB (Mathworks). SVM classifiers were trained using publicly available software⁵⁸. No statistical methods were used to pre-determine sample sizes, but our sample sizes are similar to those reported in previous publications^14,26. Data distributions were assumed to be normal, but this was not formally tested. All animals were selected randomly from our colony for inclusion in the study, and there were no comparisons across subject groups. No blinding was used in the study given the lack of experimental groups. No data points were excluded from the study.

Preprocessing of behavior data

Pupil diameter and facial movements were extracted from face videography using FaceMap²⁸. For subsequent analysis, facial movement is defined as the the first component of FaceMap-based decomposition. Singular value decomposition (SVD) was applied to the face movie to extract the principal components (PCs) explaining the distinct movements apparent on the mouse’s face. Wheel position was obtained from a linear angle detector attached to the wheel axle by unwrapping the temporal phase and then computing the traveled distance (cm). Locomotion speed was computed as the differential of the smoothed distance (cm/sec) using a 0.4 second window. Epochs of sustained locomotion and quiescence were extracted using change-point detection as described¹⁴. High/low Pupil and Face epochs were extracted from within quiescence segments where z-score normalized values exceeded high/low thresholds of 60%/40% quantiles.

Preprocessing of mesoscopic imaging data

Imaging frames for green and red collection paths were grouped and down-sampled from 512X512 to 256X256 followed by an automatic ‘rigid’ transformation (imregtform, Matlab). In some cases, registration points were manually selected and a ‘similarity’ geometric transformation was applied. Detrending was applied using a low pass filter ($N=100,f_{\text{cutoff}}=0.001Hz$). Time traces were obtained using $(\Delta F/F{)}_i=\left(F_i-F_{i,o}\right)/F_{i,o}$ where $F_i$ is the fluorescence of pixel $i$ and $F_{i,o}$ is the corresponding low-pass filtered signal.

Hemodynamic correction

Hemodynamic artifacts were removed using a linear regression accounting for spatiotemporal dependencies between neighboring pixels¹⁴. We used the approximate isosbestic excitation of GCaMP6s or GRAB_ACh3.0 (395 nm) as a means of measuring activity-independent fluctuations in fluorescence associated with hemodynamic signals. Briefly, given two $p\times 1$ random signals $y_1$ and $y_2$ corresponding to $\Delta F/F$ of $p$ pixels for two excitation wavelengths “green” and““U””, we consider the following linear model:

$$\begin{array}{c}{y}_1=x+z+\eta ,\\ y_2=Az+\xi ,\end{array}$$

where $x$ and $z$ are mutually uncorrelated $p\times 1$ random signals corresponding to $p$ pixels of the neuronal and hemodynamic signals, respectively. $\eta$ and $\xi$ are white Gaussian $p\times 1$ noise signals and $A$ is an unknown $p\times p$ real invertible matrix. We estimate the neuronal signal as the optimal linear estimator for $x$ (in the sense of Minimum Mean Squared Error):

$$\hat{x}=H\left(\genfrac{}{}{0pt}{}{y_1}{y_2}\right),H={\Sigma }_{xy}{\Sigma }_y^{-1},$$

where $y=\left(\genfrac{}{}{0pt}{}{y_1}{y_2}\right)$ is given by stacking $y_1$ on top of $y_2,{\Sigma }_y=\mathbb{E}\left[y{y}^T\right]$ is the correlation matrix between $y$ and ${\Sigma }_{xy}=\mathbb{E}\left[x{y}^T\right]$ is the correlation matrix between $x$ and $y$. The matrix ${\Sigma }_y$ is estimated directly from the observations, and the matrix ${\Sigma }_{xy}$ is estimated by¹⁴:

$${\Sigma }_{xy}=\left({\Sigma }_{y_1}-{\sigma }_{\eta }^{2}I-{\left({\Sigma }_{y_1{y}_2}{\left({\Sigma }_{y_2}-{\sigma }_{\xi }^{2}I\right)}^{-1}{\Sigma }_{y_2}^{-1}{\Sigma }_{y_1{y}_2}^T\right)}^{T}0\right),$$

where ${\sigma }_{\eta }^2$ and ${\sigma }_{\xi }^2$ are the noise variances of $\eta$ and $\xi$, respectively, and $I$ is the $p\times p$ identity matrix. The noise variances ${\sigma }_{\eta }^2$ and ${\sigma }_{\xi }^2$ are evaluated according to the median of the singular values of the corresponding correlation matrices ${\Sigma }_{y_1}$ and ${\Sigma }_{y_2}$⁵⁹. This analysis is usually performed in patches where the size of the patch, $p$, is determined by the amount of time samples available and estimated parameters. In the present study, we used a patch size of $p=9$. The final activity traces were obtained by z-scoring the corrected $\Delta F/F$ signals per pixel.

Parcellation of mesoscopic data using LSSC

Functional parcellation of mesoscopic data was performed primarily using Local Selective Spectral Clustering (LSSC)³¹. Briefly, this method identifies areas of co-activity by building a graph where nodes are pixels and edge weights are determined by pairwise similarities between activity traces of pixels obtained by the following kernel:

$$\mathit{K}(i,j)=\text{exp}\left\{-{∥(\Delta F/F{)}_i-(\Delta F/F{)}_j∥}^2/{\sigma }^2\right\}$$

where $\sigma$ is a parameter expressing a similarity radius. A row-stochastic matrix $\mathit{P}$ is obtained by normalizing the rows such that $\mathit{P}={\mathit{D}}^{-1}\mathit{K}$, where $\mathit{D}(i,i)={\sum }_j\mathit{K}(i,j)$. The matrix $\mathit{P}$ can be viewed as a transition matrix of a Markov chain of the graph where $\mathit{P}(i,j)$ is the probability to jump from node (pixel) $i$ to node (pixel) $j$. We obtain a non-linear embedding of pixels by calculating the $d$ right eigenvectors with the largest eigenvalues of $\mathit{P}$:

$$(\Delta F/F{)}_i\to {\psi }^{\left(n\right)}(i)=\left(\begin{array}{c}{\psi }^1\left(i\right)\\ ⋮\\ {\psi }^n\left(i\right)\end{array}\right)$$

Overall, by taking $n$ to be significantly smaller than the number of time samples, every pixel is represented by a lower dimensional embedding ${\psi }^{\left(n\right)}$.

We evaluate the embedded representation ${\psi }^{\left(n\right)}$ and calculate the spectral embedding norm⁶⁰ of every pixel $s_i=∥{\psi }^{\left(n\right)}\left(i\right)∥$. LSSC uses an iterative approach for parcellating the brain where the inputs are the embedded representation of all pixels ${\psi }^{\left(n\right)}$ and their corresponding norms, $s_i,i=1,\dots ,p$, and lastly, a list of all pixels sorted by decreasing order of the embedding norm denoted by $l$. On each iteration the following operations are performed until coverage of at least $\vartheta$ percent of the mask brain pixels is assigned to parcels:

1. Select the first item on the list $l$ (the pixel having the maximal norm, noted by $i^{*}$)

2. Select the axes in which $i^{*}$ has the largest values, i.e., the subset: $L_{i*}=\left\{{\ell }_1,{\ell }_2,\dots ,{\ell }_{d_i}\right\}$ such that $\left|{\psi }^{{\ell }_1}\left(i\right)\right|\ge \left|{\psi }^{{\ell }_2}\left(i\right)\right|\ge \left|{\psi }^{{\ell }_3}\left(i\right)\right|\ge \cdots$

3. Obtain the pixels whose embeddings are closer to ${\psi }^{\left(n\right)}\left(i\right)$ than to the origin based on the axes $L_{i*}$ and assign them to the cluster $k$, i.e.:

   $$C_k=\left\{j\mid {∥{\psi }^{L_i}\left(i\right)-{\psi }^{L_i}\left(j\right)∥}^2<{∥{\psi }^{L_i}\left(j\right)∥}^2\right\}$$

4. Remove the set $C_k$ from the list $l:l\leftarrow l\setminus C_k$

5. $k\leftarrow k+1$

6. If at least $\vartheta$ percent of the mask of the brain is assigned to a specific parcel, then break.

The output is therefore a set of clusters $\left\{C_k\right\}$ where each clusters contains the pixels in that cluster. To increase robustness, we divided every session into 10 disjoint segments (folds), extracted the embedding on every fold and evaluated the embedding norm as the maximal value across all 10 folds. We refined the brain parcellation by merging parcels whose time traces are correlated more than a given threshold. Overlapping pixels were assigned to the parcel with closest centroid (in the embedding space). Additionally, unassigned isolated pixels (if any) were assigned to the (spatially) closest parcel. Isolated pixels within the borders of more than one parcel were assigned to the closest cluster (in the embedding space). Each animal and session was parcellated to reach a 95% coverage of the mask of the brain where clusters were merged based on a threshold of o.99, resulting in ~45 parcels per hemisphere. Time series for parcels were extracted by averaging values for all pixels within the parcel (see preprocessing of mesoscopic data above).

As a comparison to LSSC we used PCA to reduce the dimension of the widefield signal and used Localized semi-nonnegative matrix factorization (LocaNMF)³³ for parcellation of mesoscopic data. As an additional comparison, we spatially decimated the mesoscopic data by factor of 8, which is equivalent to using a fixed grid dividing the brain to patches of 8×8 pixels, used as “Grid” parcels.

Cellular ROI extraction by LSSC

We also used LSSC to identify cell bodies from the two-photon imaging data. The overall approach is similar to the parcellation process except for the stopping condition, where iterations continue until a maximal number of cells is reached. In the refinement stage, identified cells that smaller than 15 pixels were discarded and overlapping regions were resolved by de-mixing³¹.

Taylor expansion for estimating behavior as a function of neuronal activity

We formulate the link between temporal dynamics of neuronal activity ${\mathit{x}}_t\in {\mathbb{R}}^N$ and an observed behavior $b_t$ as:

$$b_t=f\left({\mathit{x}}_t\right)$$

where $f$ is an unknown function. Assuming that $f\left({\mathit{x}}_t\right)$ is 2 times differentiable, we can write its second-order Taylor’s expansion as:

$$b_t=f\left({\mathit{x}}_t\right)\approx f\left(\stackrel{-}{\mathit{x}}\right)+{\left.\sum _i\frac{\partial f}{\partial x_t\left(i\right)}\right|}_{\stackrel{-}{\mathit{x}}}\left(x_t\left(i\right)-\bar{x}\left(i\right)\right)+{\left.\frac{1}{2}\sum _i\sum _j\frac{{\partial }^{2}f}{\partial x_t\left(i\right)\partial x_t\left(j\right)}\right|}_{\stackrel{-}{\mathit{x}}}\left(x_t\left(i\right)-\bar{x}\left(i\right)\right)\left(x_t\left(j\right)-\bar{x}\left(j\right)\right)+ϵ$$

where $\stackrel{-}{\mathit{x}}$ is the average neuronal activity (across time), and $ϵ$ is the error of neglecting higher orders of ${\mathit{x}}_t$. Simplifying this equation leads to:

$$b_t\approx {\beta }_0+\sum _i{\beta }_1\left(i\right)x_t\left(i\right)+\sum _i\sum _j{\beta }_2(i,j)C_t(i,j)+ϵ$$ (3)

where $C_t(i,j)=\left(x_t\left(i\right)-\bar{x}\left(i\right)\right)\left(x_t\left(j\right)-\bar{x}\left(j\right)\right)$ is the time trace of the instantaneous interaction between brain region $i$ and brain region $j$ and ${\beta }_n,n=\mathrm{0,1},2$ are the model parameters. Overall eq. (3) proposes a linear model for behavior based on two temporal signals–- the activity $x_t\left(i\right)$ and the pairwise interaction $C_t(i,j)$, which is a nonlinear second-order function of elements of ${\mathit{x}}_t$. Since the elements $x_t\left(i\right)$ and $C_t(i,j)$ are linearly independent for all $i,j\in [1,N]$ can measure the decoding power of each of these two components ${\mathit{x}}_t$ and ${\mathit{C}}_t$ independently.

In eq. (3) the instantaneous interactions $C_t(i,j)$ are evaluated based on a single time point. In practice, estimating all pairwise interactions at a single point is highly sensitive to noise. Thus, we evaluate the interactions over a short sliding time window to obtain the sample covariance ${\hat{C}}_t(i,j)$ as a smoothed and more robust estimation for the temporal evolution of $C_t(i,j)$:

$${\hat{C}}_t(i,j)=\frac{\sum _{\tau =t-N_t/2}^{t+N_t/2}\left(x_{\tau }\left(i\right)-{\bar{x}}_t\left(i\right)\right)\left(x_{\tau }\left(j\right)-{\bar{x}}_t\left(j\right)\right)}{\sqrt{\sum _{\tau =t-N_t/2}^{t+N_t/2}{\left(x_{\tau }\left(i\right)-{\bar{x}}_t\left(i\right)\right)}^2}\sqrt{\sum _{\tau =t-N_t/2}^{t+N_t/2}{\left(x_{\tau }\left(j\right)-{\bar{x}}_t\left(j\right)\right)}^2}}$$ (4)

where ${\bar{x}}_t\left(j\right)$ is the smoothed averaged activity:

$${\bar{x}}_t\left(i\right)=\frac{1}{N_t}\sum _{\tau =t-N/2}^{t+N_t/2}{x}_{\tau }\left(i\right)$$

Inserting ${\hat{C}}_t(i,j)$ into (3) leads to eqn. (1). Overall, $\widehat{\mathit{C}}$ is a three-dimensional tensor of parcels by parcels by time, where each element ${\hat{C}}_t(i,j)$ is a time trace of the instantaneous correlation coefficient between parcel $i$ and parcel $j$. For most analyses, $N_{\text{t}}$ was 30 (corresponding to a 3 second moving window). In all cases, the time-step was set to be 1 frame (0.1 second).

Riemannian projection of correlation matrices

Correlation matrices are Symmetric and Positive Definite (SPD, i.e., symmetric and full rank) and whose underlying geometry is a manifold shaped like a cone with a Riemannian metric (Supplemental Figure 2)^61,62. The distances between two correlation matrices on this cone is defined by the geodesic distance, the length of the arc connecting these matrices, whereas the Euclidean distance is not an accurate measure for this geodesic distance. To accurately capture distances between SPD matrices, Riemannian geometry is often used to project them onto a tangent Euclidean space where the geodesic length is approximated by the Euclidian distances between the corresponding projections. This evaluation becomes more accurate if the plane is tangent to the cone at a point that is relatively close to all relevant matrices, usually taken as their Riemannian mean.

Briefly, let $\left\{{\mathit{C}}_k\right\}$ be a set of $K$ SPD matrices. Denote $\stackrel{-}{\mathit{C}}$ as the Riemannian mean of the set and $\stackrel{-}{\mathit{S}}$ as its equivalent in the tangent plane. $\stackrel{-}{\mathit{C}}$ and $\stackrel{-}{\mathit{S}}$ are calculated using the following iterative equations:

$$\begin{array}{c}{\stackrel{-}{\mathit{S}}}_n=\frac{1}{K}\sum _{k=1}^K{\stackrel{-}{\mathit{C}}}_n^{1/2}\text{log}\left({\stackrel{-}{\mathit{C}}}_n^{-1/2}{\mathit{C}}_k{\stackrel{-}{\mathit{C}}}_n^{-1/2}\right){\stackrel{-}{\mathit{C}}}_n^{1/2}\\ {\stackrel{-}{\mathit{C}}}_{n+1}={\stackrel{-}{\mathit{C}}}_n^{1/2}\text{exp}\left({\stackrel{-}{\mathit{C}}}_n^{-1/2}{\stackrel{-}{\mathit{S}}}_n{\stackrel{-}{\mathit{C}}}_n^{-1/2}\right){\stackrel{-}{\mathit{C}}}_n^{1/2}\end{array}$$

Where log (⋅) and exp (⋅) are the matrix logarithm and matrix exponential, respectively, and where the Euclidean mean is used to initialize: ${\stackrel{-}{\mathit{C}}}_0=\frac{1}{K}{\sum }_{k=1}^K{\mathit{C}}_k$. Convergence is obtained when the Frobenius norm of ${\stackrel{-}{\mathit{S}}}_n$ is smaller than a pre-set parameter $\epsilon :{\left|{\stackrel{-}{\mathit{S}}}_n\right|}_F<\epsilon$.

The projections of $\left\{{\mathit{C}}_k\right\}$ onto the tangent plane to the cone at the Riemannian mean are given by:

$${\mathit{S}}_k={\stackrel{-}{\mathit{C}}}^{1/2}\text{log}\left({\stackrel{-}{\mathit{C}}}^{-1/2}{\mathit{C}}_k{\stackrel{-}{\mathit{C}}}^{-1/2}\right){\stackrel{-}{\mathit{C}}}^{1/2},k=1,\dots ,K$$

As presented previously⁶³, the pairwise distances, $d_R^2\left({\mathit{C}}_k,{\mathit{C}}_l\right)$ on the cone between correlation matrices $\left\{{\mathit{C}}_k\right\}$ can be approximated by the Euclidean distances between their corresponding projections $\left\{{\mathit{S}}_k\right\}$:

$$d_R^2\left({\mathit{C}}_k,{\mathit{C}}_l\right)\cong {∥{\tilde{\mathit{S}}}_k-{\tilde{\mathit{S}}}_l∥}^2,$$

where ${\tilde{\mathit{S}}}_{\mathit{k}}=\text{log}\left({\stackrel{-}{\mathit{C}}}^{-1/2}{\mathit{C}}_k{\stackrel{-}{\mathit{C}}}^{-1/2}\right)$. This method requires that all matrices $\left\{{\mathit{C}}_k\right\}$ would be full rank^64,65. In practice this is not always the case if the number of time points for evaluation of the correlation matrices is smaller than $p$, i.e. $N_T<p$. Therefore, we add a regularization term, $\lambda \mathit{I}$, to each correlation matrix ${\mathit{C}}_k$⁶⁶ where $\lambda$ is set to the median of the singular values of ${\mathit{x}}_t$⁵⁹.

Dimensionality reduction by diffusion embedding

The series of matrices ${\mathit{C}}_t$ are symmetrical and therefore the dimension of their projections, $\left\{{\mathit{S}}_t\right\}$, is equal to $\left(\genfrac{}{}{0pt}{}{p}{2}\right)$ resulting in a high dimensional signal. To analyze the dynamics of this signal, we used diffusion geometry to obtain a low dimensional representation, capturing the dynamical properties of the correlation traces. Unlike LSSC where we reduce the dimension across time samples, in this case we reduce the dimension of parcels; we evaluated the $N_T\times N_T$ kernel matrix of $\left\{{\mathit{S}}_t\right\}$:

$$\mathit{A}(i,j)=\text{exp}\left\{-{∥{\mathit{S}}_i-{\mathit{S}}_j∥}^2/{\sigma }^2\right\}$$ (5)

where $\sigma$, which is a scale parameter evaluated as the median of pairwise distances between each projected matrix and its $k$-nearest neighbors where $k=20$. Note that our results are highly robust to variation in this parameter (Supplemental Figure 2).

Normalizing the kernel $\mathit{A}$ to be row-stochastic and taking the right eigenvectors leads to the low dimensional representation for the correlation traces:

$${\mathit{S}}_t\to {\varphi }_t^{\left(n\right)}=\left(\begin{array}{c}{\varphi }_t^1\\ ⋮\\ {\varphi }_t^n\end{array}\right)$$

The size of the kernel matrix $\mathit{A}$ is determined by the available time points recorded on each experiment ($N_T\approx {10}^4$ for all sessions). To reduce computational complexity involving eigenvalue decomposition of large matrices we used Nyström out-of-sample extension⁶⁷ as follows. We randomly choose a smaller subset of time points ${\left\{t_i\right\}}_{i=1}^N$, where $N<N_T$. We obtain the normalized kernel using these time points and extract its eigen-decomposition ${\lambda }_k,{\varphi }_{t_i}^k,k=1,\dots ,n$. We then extend this low dimensional representation to all time points:

$${\varphi }_t^k=\frac{1}{{\lambda }_k}\sum _{i=1}^N\text{exp}\left\{-{∥{\mathit{S}}_t-{\mathit{S}}_{t_i}∥}^2/{\sigma }^2\right\}{\varphi }_{t_i}^k$$

For comparison to using Riemannian geometry for calculating distances in the kernel, we carried out similar diffusion embedding based on Euclidean distances (Supplemental Figure S2).

Dimensionality reduction by principal component analysis

As a comparison to LSSC, we also used principal component analysis to reduce the dimensionality of widefield data (Supplemental Figure S2). Principal components were derived using the ‘pca’ function in Matlab.

Visual response analysis

Visual responses for unconditioned and conditioned experiments were evaluated, per parcel, as the difference between peak response during stimulus presentation and the average activity during the preceding two seconds. The responses were averaged per contrast value and normalized by the response to 100% contrast. To quantify the accuracy with which visual responses are encoded by visual activity, or embedded network activity/correlations, we trained a binary classifier (linear SVM, libsvm) to separate the visual response and the two seconds prior to stimulus onset. We used 10-fold cross validation to estimate the classification accuracy for every contrast value based on each predictor.

Modeling Behavior

Behavioral variables (pupil, facial movements, running speed) were modeled using linear ridge regression (unless otherwise mentioned) with 10-fold cross validation. Each session was divided into 10-disjoint continuous segments, where on each fold one segment was set aside for testing and the other segments were used for training. We assessed the predictive power of neuronal activity using the following predictors: raw activity and smoothed activity (using a 3 second moving window) and diffusion embedding based on Euclidean distances. For pairwise correlations, we used: raw correlation traces, PCA with number of components selected to account for >95% variance, and diffusion embedding of correlation traces using either Euclidean or Riemannian distances. To directly compare the predictive power of activity versus embedded correlations, we combined these predictors and evaluated the goodness of fit of the joint model. We then circularly shuffled either activity or embedded correlations through time and trained the resulting model to assess $R_{\text{shuffled activity}}^2$ and $R_{\text{shuffled}{\varphi }_t}^2$.

As a comparison to linear modeling we trained three fully connected Artificial Neural Networks for prediction of pupil/FaceMap/running speed based on neuronal activity traces as predictors. We performed a grid-search for the optimal configuration for prediction of behavior, based on 10-fold cross validation where on each fold 80% was used for training, 10% for parameters tunning and 10% for testing. Overall, the optimal performance was achieved with two inner layers of 64 and 4 units, learning rate of 0.05 and Relu activation.

Modeling correlations data by embedding

To quantify the relationship between the embedding of functional connectivity across the cortex ${\varphi }_C$ and the time-varying correlation between specific pairs of parcels, we used linear regression (10-fold cross validation) and obtained an R² value for every pair-wise correlation trace. To match LSSC parcels across animals, we identified the LSSC parcels whose center of mass were closest to each Allen Atlas brain parcel (23 parcels overall in a single hemisphere) and extracted a 23 × 23 matrix of R² values per session. We averaged these matrices across animals and extracted the rows corresponding to individual parcels. Each row was then represented as a separate brain map image, color-coded by the R² value corresponding to the correlation between the target (specific to that image) and each of the other parcels.

Evaluating predictive dynamics of cortical subnetworks

We evaluated the average activity (across parcels) of two sub-cortical networks using an arbitrary partition of the brain using a line bisecting the neocortex. We then used these two time traces to predict pupil size, facial movement, and locomotion speed. By rotating the line in 30, 60, 90, 120, 150 degrees we measured the R² values for prediction of behavior variables based on different ways for partition of the brain into sub-networks. We then evaluated the time trace of correlation between these two sub-networks using a sliding window of 3sec. This time we used the correlation trace to predict each behavior and evaluate the R² values as a function of the angle. For the correlations we also evaluated the R² as a function of the analysis window length (varying from 0.5sec to 15sec) for angle of 30degrees.

Instantaneous Multimodal Connectivity

Denote ${\mathit{x}}_t^{\left(1\right)}\in {\mathbb{R}}^{N_1}$ as the neuronal activity of $N_1$ brain parcels at time $t$ and denote ${\mathit{x}}_t^{\left(2\right)}\in {\mathbb{R}}^{N_2}$ as the neuronal activity of $N_2$ cells at time $t$. We define ${\Sigma }_t(i,j)$ as a 3-dimensional tensor of $N_1$ over $N_2$ over time, expressing the dynamics of multimodal connectivity between cells and parcels:

$${\Sigma }_t\left(i,j\right)=\frac{\sum _{\tau =t-\frac{N_t}{2}}^{t+\frac{N_t}{2}}\left(x_{\tau }^1\left(i\right)-{\bar{x}}_t^1\left(i\right)\right)\left(x_{\tau }^2\left(j\right)-{\bar{x}}_t^2\left(j\right)\right)}{\sqrt{\sum _{\tau =t-\frac{N_t}{2}}^{t+\frac{N_t}{2}}{\left(x_{\tau }^1\left(i\right)-{\bar{x}}_t^1\left(i\right)\right)}^2}\sqrt{\sum _{\tau =t-\frac{N_t}{2}}^{t+\frac{N_t}{2}}{\left(x_{\tau }^2\left(j\right)-{\bar{x}}_t^2\left(j\right)\right)}^2}},i=1,\dots ,N_1,j=1,\dots ,N_2,t=1,\dots ,N_T$$

where $N_T$ is the numer of time points. For a given time point $t$, the matrix ${\Sigma }_t$, is non-symmetric and therefore not bound to the Riemannian cone (as opposed to the sample covariance matrices ${\widehat{\mathit{C}}}_t$, which are SPD). Therefore, we used Euclidean distances to evaluate the diffusion kernel between correlation matrices related to different time points:

$$\tilde{\mathit{A}}\left(t,t^{\prime }\right)=\text{exp}\left\{-{∥{\Sigma }_t-{\Sigma }_{t^{\prime }}∥}^2/{\sigma }^2\right\}$$ (6)

where $\sigma$ is a scale parameter evaluated as the median of pairwise distances between each correlation matrix and its $k$-nearest neighbors, where $k=200$. Note that as for embedding of correlation matrices, our results here are also highly robust to variation in $k$ (Supplemental Figure 6). Here we again reduce computational complexity using Nyström extension and evaluate the kernel $\tilde{\mathit{A}}$ based on a smaller randomly selected sub-set of time points, $t^{\prime }=1,\dots ,N$, where $N<N_T$. We normalize the kernel matrix to be row-stochastic and obtain its eigen-decomposition:

$$\begin{gathered} {\Sigma }_{t^{\prime }}\to {\phi }_{t\prime }^{\left(n\right)}=\left(\begin{array}{c}{\varphi }_{t^{\prime }}^1\\ ⋮\\ {\varphi }_{t^{\prime }}^n\end{array}\right) \\ {\lambda }_k,k=1,\dots ,n \end{gathered}$$

We then extend this representation to all time points $t=1,..,N_T$:

$${\phi }_t^k=\frac{1}{{\lambda }_k}\sum _{t^{\prime }=1}^N\text{exp}\left\{-{∥{\Sigma }_t-{\Sigma }_{t^{\prime }}∥}^2/{\sigma }^2\right\}{\phi }_{t\prime }^k$$

Standard Deviation of Multimodal Connectivity

We estimate the average variability of correlations between a given cell and the mesoscopic cortical network as:

$${\sigma }_j=\frac{1}{N_1}\frac{1}{N_t}\sum _{i=1}^{N_1}\sum _{t=1}^{N_t}{\left({\Sigma }_t(i,j)-\bar{\Sigma }(i,j)\right)}^2,j=1,\dots ,N_2$$

where $\bar{\Sigma }(i,j)=\frac{1}{N_T}{\sum }_{t=1}^{N_T}{\Sigma }_t(i,j)$.

Data Availability

The full datasets generated and analyzed in this study are available from the corresponding authors on reasonable request. Data for mesoscopic imaging experiments with CCFv 3-based parcellation have been deposited on the figshare archive: https://figshare.com/projects/Benisty_Higley_2023/175317.