Natural behavior is the language of the brain

Abstract

The breadth and complexity of natural behaviors inspires awe. Understanding how our perceptions, actions, and internal thoughts arise from evolved circuits in the brain has motivated neuroscientists for generations. Researchers have traditionally approached this question by focusing on stereotyped behaviors, either natural or trained, in a limited number of model species. This approach has allowed for the isolation and systematic study of specific brain operations, which has greatly advanced our understanding of the circuits involved. At the same time, the emphasis on experimental reductionism has left most aspects of the natural behaviors that have shaped the evolution of the brain largely unexplored. However, emerging technologies and analytical tools make it possible to comprehensively link natural behaviors to neural activity across a broad range of ethological contexts and time scales, heralding new modes of neuroscience focused on natural behaviors. Here we describe a three-part roadmap that aims to leverage the wealth of behaviors in their naturally occurring distributions, linking their variance with that of underlying neural processes to understand how the brain is able to successfully navigate the everyday challenges of animals’ social and ecological landscapes. To achieve this aim, experimenters must harness one challenge faced by all neurobiological systems, namely variability, in order to gain new insights into the language of the brain.

In Brief:

Miller et al. argue that natural behavior can serve as a powerful lens into the full range of computations that brains perform by leveraging the dynamic covariance between behavior and neural activity.

Introduction.

Whereas the heart functions to pump blood through the circulatory system and the lungs to extract oxygen from the air around us, the nervous systems function to support an animal’s active participation in the world. Nervous systems have been shaped evolutionarily across species to successfully govern how individuals of a species navigate their unique ecological and social landscapes; a relationship elegantly articulated by Briscoe and Ragsdale¹:“The evolution of vertebrate nervous systems is in large part the evolution of neural circuitry — whether through qualitative changes to circuit construction, increase or reduction of neuronal number at particular nodes, or through the reorganization of circuit components in space — with behavior as the target of selection, made manifest through neuronal activity”. Brains are not general-purpose machines: the cells, circuits, and areas of the brain have been collectively optimized to support behaviors in species’ repertoires.

Recent technological advances in the neurosciences have pushed the envelope in our understanding of the intricate relationship that exists between neural systems and behavior^2,3. Technological development is necessary: without measuring specific dimensions of behavior or nervous system function, we are running blind. But an appreciation of these newly available data is tempered by the realization that tools alone will not illuminate the mysteries of the brain on their own⁴, no matter how ‘big’ the data. To meaningfully interpret such considerable datasets, integrating these diverse empirical observations into a higher-order understanding of complex systems mandates a solid theoretical foundation. If we are to understand brains as the dynamic biological systems they are, theory firmly rooted in behavior — the target of evolutionary selection — offers the most fertile ground for discovery of the brain’s computational principles, the generalizable neural dynamics that underlie if, when and how natural behaviors manifest across different contexts, including both the degree of plasticity and mechanistic constraints that shape the distributions of potential behavioral outcomes.

To address this considerable challenge, we propose a conceptual merger between two neuroscience sub-disciplines — systems neuroscience and neuroethology — that have at times been viewed as at odds but that are in fact complementary, each with its own theoretical advantages^5,6. These disciplines are distinguished in large part by their distinct perspectives on the role of behavior to elucidate facets of brain function. Systems neuroscience has tended towards reductionistic paradigms designed to control behavior through conditioning^7–9. From this perspective, behavior is viewed as one of many available lenses through which to view how brains work, with neural processes taking primacy. In contrast, neuroethology has focused on naturally occurring behaviors in a species’ repertoire, biasing towards highly specialized behaviors that result from intense evolutionary selection, such as survival and mate attraction^10–13. For neuroethologists, natural behaviors are typically the anchor point for inquiry into brain function. While both frameworks have been powerful engines of discovery for how brains work, each has in its own way skewed our understanding of neural systems by focusing on stereotyped behaviors that have as little variability as possible. This focus follows to generate the large numbers of identical or nearly identical trials or behavioral events to gain repeatability and statistical power for traditional forms of data analysis. Whether isolated behaviors of the type studied in the laboratory are representative ‘models’ of all behaviors and brain function is debatable¹⁴. In fact, such stereotyped behaviors reflect only a small fraction of the behavioral repertoire (Figure 1). In other words, even for the most well studied species, we know embarrassingly little about how brains support what animals are doing most of their day.

Figure 1. Animals have remarkably diverse behavioral repertoires.

The stereotyped conditioned and specialized behaviors that are typically studied in systems neuroscience and neuroethology are a small part of the range of behaviors they produce, many of which are shared across species (social interaction, predator avoidance, prey capture, and so on) but for which we know very little about the underlying neural mechanisms. For example, with the exception of spatial representations in the medial temporal lobe first pioneered by O’Keefe^15–19, relatively few data are available that detail the effects of navigation and exploration on perceptual and cognitive functions^20–22 despite the fact that the ability to move through space has both been a foundational pressure on brain evolution and routinely accounts for large portions of daily activity in all animals. Similarly, the neuroscience of the song learning behavior in oscines has been studied extensively generating unique insights into sequence motor learning but has taught us very little about brain mechanisms involved in natural vocal exchanges²³. Spatial exploration and natural communication behaviors, however, are highly variable, making them difficult to study in conventional frameworks.

How can neuroscientists interested in natural modes of brain function gain traction in understanding behaviors that are not isolated and stereotyped? How can we study the neural mechanisms underlying dynamic, variable, and infrequent behaviors? One important factor is to reconsider what constitutes a good or valid neuroscience experiment. The academic view of the scientific method emphasizes the virtue of high statistical repeatability and restricted outcomes for the pursuit of well-formed hypotheses. While this formulaic mode of scientific investigation has great value for many problems, it is not the only path to scientific inquiry, and it may not be well suited for studying many elements of natural behavior. In neurobiology, mechanistic hypotheses about brain circuits are often constrained by the narrow potential outcomes of a particular experimental paradigm. However, the long-term value of pursuing such hypotheses must ultimately be linked to an animal’s natural behavioral repertoire, including when that animal is a human. Thus, the distinct challenges posed by studying brain activity in animals undertaking natural behaviors are at least partially offset by the knowledge that the brain is operating within its normal range; the motivations for behaviors to manifest when they do are centered in the challenges the brain evolved to solve, rather than the experimentally compelled motivations like satiating thirst or avoiding pain that typify traditional neuroscientific approaches. Given the influence of an animal’s state on neural processes and behaviors^24–28, such differences in behavioral motivation are likely to have profound effects on brain activity.

We consider that while natural behaviors include the range of observable actions that manifest in the daily lives of animals, behavior is not limited to motor actions. Instead, we endorse a more holistic view of behavior comprising events and processes that occur between actions that include — but are not limited to — sleep, decision-making (including the decision to suppress a behavior), planning, and related internal states of the individual. This view emerges from an appreciation of the fact that natural behaviors occur along overlapping short (millisecond, second) and long (circadian, lifecycle) time scales, each impacting neural computations and processes independently and collectively.

In this review, we present a three-part roadmap aimed at addressing the challenges of studying natural behaviors and brains. We champion using the full distribution of behavioral and neural data — rather than their central tendency — to evaluate the neuroscience of natural behaviors. The concept of distributions, or variance, as primary targets of experimental inquiry is a foundational tenet of our roadmap. We use the term variance broadly, to encompass different outcomes (behavioral output, neural activity, and so on) that occur within ostensibly similar conditions, such as repeated events, as well as across a range of contexts that often remain unexplored. We argue that keystone principles of brain function can only be discovered when animals execute dynamic behavioral sequences, whose details are subject to variation within a range of natural parameters. After all, variance is an inherent feature of all biological systems. Variability not only distinguishes biology from other sciences²⁹, but it is one of the cornerstones of evolution by natural selection. Ultimately, the full mapping of covariance between the distributions of behaviors and brains is a key first step towards elucidating the core principles of neural computations that govern the breadth of behaviors in a repertoire.

Hence, rather than continue the tradition of neuroscience to curtail variability, our first challenge here is to embrace the distributions of the natural modes of behavior. Our second challenge addresses the experimental feasibility of an approach focused on dynamic, natural behaviors that we champion here. In this section, we describe how neural and behavioral recording advances have set the stage to optimize experimental designs that balance considerations of variance and control. The final challenge is one of interpretability. The recent step increase in the quality of data collection methods does not, by itself, guarantee a deeper understanding of brain function. New conceptual insights into the multi-dimensional relationship between complex brain signals and behaviors, and its bearing on core principles of brain function, require that scientists develop methods to harness the variance inherent to such rich datasets. For this, biological observations of behaviors and brains must be integrated into new theoretical frameworks that provide hitherto unseen perspectives on neural computations. Natural behaviors are not simply a useful source of variance. Natural behaviors encompass the map of possible neural computations that we seek to understand in neuroscience - how we interact with and navigate the world around us.

Importantly, here we distinguish between neural computations — processes in the brain itself, the representation and transformation of external and internal variables that govern behaviors — and computational methods — the analysis approaches that neuroscientists apply to characterize our observations and measurements of processes in the brain. Our ambition to develop theoretical principles from analysis approaches that fully capture neural computations within natural behavior remains the primary goal, but there are few, if any, convincing examples at present. A key motivation for the roadmap described here is to outline a path to that end.

Challenge 1: embrace the diversity of natural behaviors

Every action taken by an animal is unique. The Greek philosopher Heraclitus captured this principle in his famous adage, “No man ever steps in the same river twice, for it is not the same river and he is not the same man”. Thus, the brain is designed to achieve its goals amid an ever-changing set of circumstances. Nonetheless, neuroscience has traditionally sought to recreate, as precisely as possible, actions and behavioral conditions for the benefit of precise repeatability. This understandable pursuit has resulted in our view of the brain being filtered through a ‘mean/median’ framework, in which we record an event multiple times, averaging neural activity across those ‘trials/events’ and consider the mean/mode/median as ‘representative’. In fact, experimental approaches have often gone to great lengths to reduce the variance to focus the scope of inquiry to this central tendency outcome.

This traditional conceptual framework, which encapsulates many, though of course not all approaches to analyzing brain function, puts aside the important fact that variation in both the environmental context and the execution of a behavior is a core feature of brain function. Brains, however, must operate in a graded, analog fashion that continually adjusts actions and expectations with respect to the outside world. This natural variation should not be construed as noise against which the brain must contend, but rather the basic parameters of its operation. Harnessing the natural distribution of natural behaviors to study the brain in action — rather than focusing only on the mean/median — requires a tactical shift with new practical and theoretical considerations. This includes an appreciation of low-frequency neural or behavioral events as meaningful parts of the distribution, rather than as outliers that can be statistically ignored. Foremost is the capacity to measure and quantify the diverse modes of behavioral variance that mark natural experience across the full range that behaviors manifest.

Each instance of a given behavioral action is different because of changing environmental conditions^30–32, ecological contexts (such as the presence or absence of particular conspecifics or predators^12,33–36), internal cognitive states^37–39, and a range of other factors^40,41. One important source of variance stems from the observation that behaviors often occur within a continuous sequence, rather than as discrete events⁴². A lioness hunt is not only the attack, but a series of complementary actions including visually tracking, stalking, hiding and positioning, typically in coordination with group members⁴³. Thus the significance of a given behavioral component, and the neural mechanisms supporting it, can depend on what came immediately before. This ubiquitous principle provides one avenue for dissecting contextual versus invariant neural contributions to a given behavior. It is exemplified in classic studies of human speech production, where the articulation of a word depends strongly on the preceding and subsequent syllables^44,45(Figure 2).

Figure 2. Context-dependent variability of behavior, demonstrated by co-articulation of speech.

The formant frequency representation for single words (‘other’) can change dramatically in different sentences (A,B) and be distinct when spoken in isolation (C). This context dependence, and the contrast between natural context and repeated trials in isolation, is also mirrored in the sensory domain. For example, in vision an edge at a given orientation can appear in many different visual contexts, and such stimuli within natural scenes evoke different responses than the standard repeated presentation of a similar stimulus in isolation^55,56.

The neural processes that generate natural behaviors must coordinate over both short and long time scales to accommodate sequential behavioral actions. Behavioral exchanges, and their underlying neural mechanisms, typically transpire over several seconds or longer. For example, the activity of frontal cortex neurons in marmosets prior to hearing a conspecific vocalization almost perfectly predicted whether the monkeys would engage in a conversational exchange, despite the fact that call response occurred up to 10 seconds after hearing a conspecific vocalization⁴⁶. In other words, variance in the state of the neural population at the time an ‘event’ occurred (conspecific vocalization was heard) was a key source of variance in the probability of the subsequent social behavior several seconds later. Although experimental conventions focus analyses to select time periods at the time of a key event, designing tasks that involve extended time windows — including different sequences that comprise the same behavior — and the variance that emerges in these epochs can be harnessed to illuminate facets of brain activity that can at times have profound effects on behavior⁴⁷. An important challenge will be determining how to define the windows of analysis, from the millisecond timescale of spiking activity, up to seconds for individual actions, and hours or days for state variability.

In addition to the retrospective dependency emphasized above, certain events in a sequence can have prospective consequences. The inclusion of a particular event, such as a goal-oriented decision, can be a key pivot point that marks a divergence of the subsequent sequence’s trajectory⁴⁸. Decision-making illustrates this point as it is traditionally framed as the endpoint after weighing multiple options, a fact reflected in the various experimental paradigms conventionally used to address this issue^9,49–52. Under natural conditions, however, decision-making more closely resembles a continuous, ongoing process. Decisions are made constantly, ranging from the choice to not walk into a wall to strategizing about climbing the social hierarchy. These types of decisions, and their effect on what follows, are not well captured in laboratory tests of decision-making that are most often discrete and dichotomous by nature⁵³.

In natural behavior, choices are often expressed on a graded scale and at a chosen point in a continuous sequence. This principle is obvious in navigation, where local decisions in space and time often alter the course of what follows⁵⁴. Moreover, each iteration of a certain kind of choice may have a different consequence. For example, a foraging bat, upon reaching a second feeding tree, may be more satiated and choose to socialize before consuming food. Thus, the way in which decision-making shapes the continuous flow of experience can provide natural experimental sources of variance for studying the neural principles underlying natural behavior.

Challenge 2: optimize experimental design for natural distributions.

From a practical perspective, addressing the points above requires significant changes in the acquisition, analysis, and conceptualization of data, particularly with respect to experimental design. Foremost is the continuous collection of behavioral and physiological data. Behavioral, neural, autonomic, and other signals must be acquired with sufficient temporal resolution to match the temporal dynamics of the behavioral motifs being investigated. At the same time, acquisition often needs to remain continuous over a period long enough to capture rare events and to study slower physiological processes. This contrast to trial-based paradigms reflects the desire to sample physiological activity during the broad distribution of natural behaviors. These requirements, combined with the desire to simultaneously sample a breadth of neural and other signals, place high demands on data recording, storage, and transmission. Fortunately, recent technological advances for data acquisition allow us to meet this challenge. For the quantification of behavior, machine learning approaches allow quick and continuous assessment of myriad behavioral variables. For physiological recordings, large-scale electrophysiological arrays and multiscale neuroimaging approaches allow for the simultaneous monitoring of hundreds, and sometimes thousands, of neurons in real time.

Here we discuss the bearing that these advances are beginning to have on experimental design. Importantly, this does not just simply mean acquiring larger quantities of data, but matching experimental design to the distributions that are naturally relevant: for example, sensory stimuli that represent the range an animal experiences in the real world, behavioral tasks that allow the broad repertoire of actions an animal can take, and recordings that capture the full diversity of neural cell types, response properties, and dynamics that underlie each of these.

New types of data — behavior

Tapping into the full richness of behavior will be facilitated by incorporating several key factors that have been traditionally eliminated in behavioral paradigms. First, studying natural behaviors that lack traditionally stereotyped characteristics, such as being innate and highly repeatable, or learned and over-trained, will allow exploration of the full space of possible mapping between neural activity and behavior. Second, task designs that have a continuous readout of behavior, rather than a single decision point, will both capture the richness of true behavior, which rarely has an isolated decision, and greatly expand the volume of data that can be acquired in a single session, which is necessary for many recent ‘data-hungry’ analysis methods. And last, natural behaviors take into account each species’ unique view of the world, its umwelt⁵⁷, and what it is offered by that world, its affordances⁵⁸ — with an ethological correspondence between the task variables and behavioral output, ensuring that meaningful computations are being studied.

Whenever constraints on the behavior are released, new phenomena are likely to be observed. Although the benefits may seem apparent, a combination of technical limitations and sociological issues have historically hampered researchers from taking such unconstrained approaches to behavior. Nonetheless, pioneering studies showed the richness of sensory processing during natural behavior^59,60. More recently, it was found that simply allowing a mouse to run in place on a treadmill, rather than constraining it to sit, results in striking changes in visual and auditory processing that have expanded our view of cortical function^20,61,62. These examples also emphasize the importance of active sensing^63–65: in the real world animals determine how they acquire sensory information, as opposed to laboratory paradigms where the experimenter is in charge. Allowing naturalistic, active sensing therefore has the potential to introduce tremendous variability in both the sensory input and motor output, with concordant technical challenges. This also reflects the domain in which sensory systems are optimized to function²² which includes ongoing movements and time-varying sensory stimuli.

While one potential approach to incorporating natural behavior is to move on entirely from current reductionist paradigms, releasing all constraints and allowing an animal to behave freely, there are also more step-wise, transitional approaches available. For example, studies of active vision can begin with free viewing within a head-fixed paradigm⁶⁶, or two-alternative forced choice paradigms can be modified to include richer stimuli and naturalistic response actions. Likewise, the study of natural scene statistics itself provides context for understanding the types of coding schemes the visual scene may employ⁶⁷, although the utility of the natural scenes approach in probing vision experimentally has been debated^68,69. As a further step, closed-loop experimental paradigms, which determine the timing and occurrence of stimuli and/or rewards based on a subject’s ongoing behavior, offer a path to balancing experimental control with the continuous nature of ethological behaviors⁷⁰. One notable concern with such step-wise approaches, however, is whether each iterative step away from traditional reductionistic paradigms will eventually lead to truly natural behavior, or whether distinct challenges will emerge at each step limiting progress to this eventual goal. Thus, both the transitional approach of incorporating ethological components into current paradigms and the direct approach of pursuing purely natural behaviors are certain to be informative and challenging⁷¹.

New types of data — recording brain signals

Measurements of neural activity offer a notably narrow window into brain function. With very few exceptions, a nearly infinitesimal subset of the brain’s neurons is recorded over a brief period of time in the animal’s life, generally minutes to hours. From these small subsamples, we hope to infer a general population code for brain function. Traditionally, the solution to this has been to narrow down the subject of inquiry and corresponding experimental paradigm to match this window with the neural activity. In restricting visual processing to motion of moving dots, for example, neural recordings are often then limited to areas associated with motion processing (such as visual area MT) during a two-alternative forced choice motion discrimination task, and further restricted by matching the specific stimulus parameters to a subset of neurons in that particular recording session. As discussed above, however, brains rarely operate in such an isolated regime, and therefore as we increase the complexity of the behavioral context, we must broaden this window so that the dimensionality of our neural data matches the dimensionality of the behavior or computations under investigation.

Increasing the throughput of recordings, in terms of number of simultaneous neurons as well as the neural populations — for example, cell types, circuits and brain regions — from which these neurons are sampled, ensures that there is a large and diverse enough population to represent the full behavioral or sensory repertoire being presented. Increasing the duration of recordings, whether over hours or days, not only increases the total data, but also allows sampling across multiple behavioral contexts and examination of variability over time (Figure 3). Some of these methods, such as two-photon calcium imaging, also allow estimation of coverage — the fraction of cells within a population that are responding in a given context — as opposed to methods such as single tungsten electrodes that will miss inactive neurons, leading to a ‘dark matter’ conundrum⁷⁴.

Figure 3. Behavioral and neural signatures of variability.

. (A) Zebrafinches sing to conspecifics of the opposite sex (directed song) as well as in isolation (non-directed song). While only small acoustic differences are evident between these contexts, the underlying neural activity shows striking variability⁷². Adapted with permission from Hessler and Doupe⁷². (B) Head-fixed mice on a spherical treadmill spontaneously alternate between a stationary state and locomotion. Responses to identical visual stimuli result in approximately two-times greater response when the mouse is running. Without quantification of behavioral state, this would appear as unexplained variance^20,72. Adapted with permission from Niell and Stryker²⁰. (C) In recordings of monkeys performing a visual task over several hours, task performance and arousal (as measured by pupil diameter) vary slowly. This is accompanied by a slow drift in the first principal component of neural activity, evident at the population level⁷³. Adapted with permission from Cowley et al.⁷³.

One example of the insights that can be gained from such an approach is the elucidation of a ‘slow drift’ signal in macaque V4 and PFC⁷³, which found that activity distributed across populations of neurons in these areas was modulated over the course of minutes and hours during performance of visual attention task (Figure 3C). The amplitude of this spontaneous modulation was up to five-times larger than the effect of attention that was the basis of the task design, and co-varied with behavioral performance, thus revealing a large-scale determinant of brain activity and behavior that would not be detected if averaging over trials or recording a single neuron at a time. Furthermore, this study in primates parallels findings on the effects of behavioral state, particularly locomotion and arousal, across a range of model species from flies to mice^20,75–78.

Challenge 3: analyses to harness the variance.

In the section above, we have proposed dramatically expanding the distribution of experimental data across many dimensions — but how then do we extract interpretable findings? In addition to enabling the acquisition of more diverse data, recent advances will hopefully yield a richer understanding of the data, with data analysis methods and models that consider more than tuning curves or choice probabilities to allow a full mapping between neural activity and behavior. These analysis techniques offer powerful approaches that both distill the complexity of natural brain function into cogent models while still capturing the richness of these dynamic datasets. Ultimately, this delicate balance is key to harnessing the variance inherent to the natural brain and elucidating the fundamental principles that underlie how countless species successfully navigate the world. Given the need to account for the many sources of variance that act on different levels of the brain–behavior interactions, answers to these questions in this framework need not resemble current descriptions of neural circuits that attempt to assign a specific computational role to individual neuron classes in highly constrained contexts.

Quantifying behavior

Over the past several years, machine-learning based approaches to quantifying behavior through video analysis, such as DeepLabCut, and LEAP^79,80, and acoustic analysis^81,82, have rapidly become pervasive in neuroscience. The tremendous strength of these approaches is that they allow easy quantification of many kinematic variables during an animal’s action. This contrasts with previous approaches, which required either reducing measurements down to one or a few task variables (binary choice, reaction time); laborious manual or semi-automated tracking of a small number of body points; or categorical, often qualitative, assessment into an ethogram. Hence, these new methods improve our ability to capture the true variance of natural behavior.

But this wealth of data raises the question of how to effectively make use of them for subsequent analysis. After all, although these behavioral quantification tools have dramatically increased our ability to annotate behavior, the path to translating these data into novel insight is not yet clear. One route might be to directly relate these measures to neural activity. However, raw kinematics are generally not the parameter space that is directly relevant for behavioral analysis. Instead, a number of methods have been developed that map raw kinematics into behavioral motifs or ethograms that are both more directly related to brain function and, ideally, more interpretable^83–85 for researchers. In addition to providing labels for behaviors that can be correlated with neural activity, the transitions between these states may provide a way to divide continuous behavior into ‘decision points’ that allow current analysis methods to be applied.

Is this high-throughput method of annotation just a way of avoiding the labor of manual tracking / ethogram coding or will there be fundamental new insights emerge? Achieving this may depend on incorporating this rich ‘annotation’ into models that will allow fundamental principles of behavior to be derived that might not result from more limited datasets, such as those that may arise from more classical ethological treatments. For example, Calhoun et al.³⁷ used a combined GLM–HMM (Generalized Linear Model–Hidden Markov Model) to determine latent states underlying shifts in Drosophila courtship song behavior. Depending on state, the same feedback cues from a conspecific elicit different distributions of behavioral output in terms of song production. Notably, this approach allowed identification of a specific cell type associated with one of the latent states rather than with the behavioral outputs. B-SOiD (Behavioral Segmentation of Open-field In DeepLabCut), likewise, provides a dimensionality reduction method for clustering behaviors and sequences of behaviors that might not be evident using more traditional ethological approaches⁸⁶.

Importantly, these types of modeling are not the standard post hoc approaches often used to recapitulate experimental findings. Rather, in these approaches the model itself is an analysis tool that casts large-scale and complex data into interpretable terms, which can then be used to directly interrogate the system further through manipulation of the behavior or the brain. Translating these types of insight about specific behaviors into more broad theoretical frameworks that can provide a ‘first principles’ understanding of behavior is likely to remain a challenge for some time.

Analyzing neural activity

Current methods for recording neural activity during behavior generate a tremendous amount of data — countless individual spikes in many neurons across the duration of recording. As with behavioral data, the challenge is utilizing all these neural data⁸⁷. What types of analyses are needed to extract the meaning from large-scale neural recordings? Traditional analyses often average across trials to obtain the mean response as a function of one or a few parameterized variables (stimulus orientation or frequency, task response). Similarly, in studying populations of neurons, traditional analyses often compute responses of neurons individually or after collapsing the data into pairwise interactions.

There are examples of classic studies that incorporate more sophisticated analyses, but a recent proliferation of advanced quantitative methods is now primed to make them more commonplace across systems neuroscience. Many of these approaches, such as PCA (Principal Components Analysis), non-linear manifold embedding and more generally data-driven statistical regularization techniques, involve dimensionality reduction: finding patterns in the raw high dimensional data that allows it to be represented by a much smaller number of variables^87–90.

In contrast, other approaches, such as Canonical Component Analysis, directly interrogate the relationship between brain activity and behavior⁹¹. This has multiple potential advantages, including facilitating single trial analysis because multiple neurons provide more robust signal than individual neurons, incorporating data over groups of correlated neurons rather than examining individuals, and potentially leading to a set of interpretable factors that can be related to aspects of behavior and brain state.

The power of dimensionality reduction approaches in revealing principles of neural coding range from rotational dynamics in motor cortex⁹² to the temporal evolution of olfactory coding⁹³, and across species from worms⁹⁴ to primates⁹⁵. These approaches can also reveal primary determinants of neural activity that may not be directly task related, such as the ‘slow drift’ described above and the impact of spontaneous movement on cortical dynamics^96,97. Finally, these approaches are likely to be essential in determining how downstream regions use this information, as demonstrated by studies of the ability to decode visual stimulus properties from large populations of V1 neurons^98,99.

Analysis at the level of single trials, rather than averaged responses across multiple trials, is particularly valuable in the context of natural behavioral and variability, as the exact same behavior is never repeated twice. A number of methods have been developed to facilitate this approach¹⁰⁰, including analysis of second-order statistics (such as power spectra) that can be combined even when individual time series cannot be aligned¹⁰¹, point process models of individual neurons¹⁰², and GLMs (Generalized Linear Models) and related techniques that allow incorporation of multiple variables into the responses of populations of neurons^103,104. Single trial analysis thus allows accounting for the unique combination of stimulus parameters, contextual and state variables, and motor output on a given trial (Figure 4). Furthermore, even in cases when the behavior itself is relatively reproducible, as in a two-alternative forced choice task, the underlying neural dynamics may have variability that is lost or even corrupted by focusing on the mean response. For example, while averaging activity of neurons across multiple trials of an information integration task shows gradual ramping activity, analysis at the level of single trials suggests that individual trials show discrete steps of activity at variable times, which then average out to a ramp over time¹⁰⁵. Developing statistical approaches that allow rigorous assessment of rare or singular events that make up the long tail of variance in natural behavior remains a key challenge.

Figure 4. Pathways from natural behaviors to neural computations.

Marmoset prey capture comprises several visual processes that have heretofore been studied in isolation, rather than as different components of a single integrative, visuo-motor behavior. Employing complementary machine vision technologies to annotate the behavior of the animal and high-density neural recordings provides is a powerful strategy to identify behavioral components in different contexts through quantitative ethograms, while employing analytical approaches that reduce the dimensionality and identify critical covariance between brain and behavior. This approach is made possible by modern technologies for quantifying behaviors at time scales that mirror brain signals, longitudinal imaging and neural recordings and powerful modeling tools.

Conclusions

Ludwig Wittgenstein once wrote “If a lion could speak, we could not understand him”¹⁰⁶. While this famous passage is often highlighted in discussions of communication and meaning in language, its connotations are directly relevant to our discussion here. Brains have many signals — spikes, oscillations, neuromodulators, and so on — that alone or together communicate an assemblage of information between cells, circuits and substrates across a range of different time scales and varying degrees of impact. Not unlike Willard Quine’s explorer who, upon hearing a local exclaim ‘Gavagai!’ in an unknown language, is left to ponder the meaning of the word — the indeterminacy of translation^107–109 — we neuroscientists are tasked with both measuring various brain signals and interpreting their ‘meaning’. We are not, however, entirely without a Rosetta Stone. The computational principles that govern brains are routinely and elegantly illustrated in the natural behavioral repertoires of animals, amongst their dynamic, multidimensional distributions. In fact, they are so clearly evident because it is these very dynamics that have been sculpted and optimized over evolution for the myriad biological timescales across which animals operate. The language of the brain is natural behavior: the observable orchestral outcome of many brain signals working in concert. The impetus to combine natural, variable behaviors, large-scale neural recording and computational analysis to elucidate brain functions has been pursued by several neuroscience pioneers in previous decades^101,110,111. What is different at this point in our field, however, is that we now have the tools to translate and leverage this prodigious diversity — both within single behaviors and across repertoires — as a powerful engine of discovery unto itself.

The significance of behavioral distributions for elucidating the principles governing brain functions is evident not only during natural behaviors, but also in more traditional paradigms. Consider first the highly reductionist studies of motion perception in the primate brain^112,113. Subjects fixate a central spot, view an ambiguous dynamic moving dot display in their visual periphery, and then communicate their decision about the net direction of dot motion by making a discrete motor response to one of two response targets. While early generations of this task used a fixed duration of visual motion^114,115, this behavioral parameter was iteratively extended to allow subjects to make their response as soon as they had made up their mind¹¹⁶. This modest change effectively grants volitional control of viewing duration to the experimental subject, increasing the natural variance of the behavior and yielding new insights into each single decision event¹¹⁷. Theoretical frameworks developed to account for these data therefore needed to graduate to more dynamic GLM approaches that revealed the complex combinations of sensory, decision and motor signals involved even in this highly stereotyped behavior¹⁰⁴.

The need for brains to support behavioral diversity is further highlighted by the increasing awareness that mixed selectivity is likely the rule rather than the exception for neural coding, at least in vertebrate brains^118–122. There is no reason to assume that the response of a single neuron should have a simple interpretation in terms of single sensory or behavioral variables that we impose upon them. There are simply too few cells in brains and too many actions they support for the vast majority of single neurons to be highly selective. The participation of individual neurons in many seemingly unrelated operations should not come as a surprise. At the same time, studies of mixed selectivity have typically relied on more conventional behavioral paradigms with correspondingly low-dimensional task parameters that do not fully reflect the breadth of operations that these cells, circuits and brains perform. These same limitations, however, do not extend to natural behaviors and repertoires. By examining the responses of neurons across multiple behavioral variables and/or contexts — thereby increasing the scope of behavioral distributions tested within individual neurons — these pioneering experiments have laid the groundwork for genuinely new perspectives on the neural computations governing flexible, variable behaviors.

Ethological repertoires offer perhaps the most powerful opportunity for insight into core neural computations that support brains. This suite of behaviors illustrates not only the particular challenges each species faces, but the distinct manner in which they are solved. Brains are analog systems, reflective of the very world in which they were selected to function over evolution. Information is not binned in nervous systems, but represented along covarying distributions in a multidimensional space.

The continued reliance on low-dimensional, stereotyped behaviors readily lends itself to testing binary hypotheses about cell types and circuits, but not necessarily to discovering the principles governing the full distribution of actions and processes that they support⁴. Ultimately, we do not record the activity of a single neuron in isolation, rather, we measure the perspective of a single neuron within a larger population. As the coding dynamics of that population change with respect to immediate behavioral and/or contextual challenges, it will be reflected in the distribution of a neuron’s activity covarying with behavioral distributions. As a result, neural responses to stimuli under more conventional conditions may not be representative of the putative natural analog because differences in behavioral context affect the animal’s perception of those signals in meaningful ways that are reflected in the patterns of neural activity¹²³.

Similar to the challenge of indeterminacy in ascribing meaning to words in language^108,109, one cannot identify the function of a brain cell, circuit, or substrate from limited contexts, but by embracing the myriad distributions of natural behaviors. For example, we have long measured sensory neuron responses to a battery of stimuli in order to determine the features that modulate activity: ‘sensory tuning’. Why not measure the covariance between neural activity and a large corpus of behaviors and/or sequences of behaviors that naturally occur within an animal’s repertoire? ‘Behavior tuning’ neurons in this way could yield unique insights into brains’ computational principles that generalize across events, as well as more idiosyncratic neural dynamics that emerge in response to distinct contextual challenges¹²³ (Figure 4).

Developing a true theory of the brain requires experiments that take advantage of the breadth of distributions that only natural behaviors can offer. Such comparisons may be the only way to account for the indeterminacy between brain activity and function. As for the significance of cross-species comparisons for identifying generalizable principles of neural functions at the synaptic and cellular levels^2,13, comparing neural processes across behaviors may reveal generalizable principles of neural function that resolve the indeterminacy of translation for brain signals and computations.

As neuroscientists we have traditionally sought to simplify brains and attribute single functions to single neurons. However, the single neuron focus fails to account for how animals survive and flourish. The path to discovery towards understanding how animals perform the impressive breadth of behaviors evident in natural repertoires is through an appreciation of the multidimensional relationships that exist between distributions of brains and behaviors. We anticipate that answers will include a broader set of principles than equating single neural cell types with responses to a specific stimulus feature or behavioral motif. Answers will associate the context-dependent dynamics of neural populations with generalizable neural computations essential for interacting with the natural social and ecological environments. Armed with the impressive modern tools to characterize behaviors and environmental factors at timescales on par with brains, data sciences approaches to quantify and model these multi-dimensional relationships, along with impressive molecular technologies to dissect these substrates, neuroscience is poised to transform our understanding of how brains actually function in the real world.