The What, How, and Why of Naturalistic Behavior

Abstract

In the past few years, advances in machine learning have fueled an explosive growth of descriptive and generative behavior models of animal behavior. These new approaches offer higher levels of detail and granularity than has previously been possible, allowing for fine-grained segmentation of animals’ actions and precise quantitative mappings between an animal’s sensory environment and its behavior. How can these new methods help us understand the governing principles shaping complex and naturalistic behavior? In this review, we will recap ways in which our ability to detect and model behavior have improved in recent years, and consider how these techniques might be used to revisit classical normative theories of behavioral control.

Introduction

Animal behavior is shaped by an interaction of past experience, internal motivational state, and external environmental cues. Behaviors critical for survival, such as feeding and social interaction, are furthermore influenced by the genetically encoded wiring of species-specific neural circuitry. Given this diverse set of influences, the analysis of behavior informs scientific progress in fields spanning genetics, ecology, neuroscience, economics, and robotics. The theories and models of behavior generated by these fields vary in their objective, language, and level of mechanistic detail. They can be broadly divided into categories of descriptive models that identify what an agent did, generative models that tell us how an agent can be expected to behave given a set of conditions, and normative models that identify why a behavior occurred by defining the underlying principles that inform its structure.

Machine learning tools for animal posture and behavior analysis are becoming increasingly accessible to the research community, fueling new approaches to the quantification and study of animal behavior. Many of these newly developed methods are descriptive models that improve the throughput and granularity with which we can quantify behavior, bringing its analysis to the realm of big data. An open question is how these methods can give us insight into the organization and control of behavior itself, in the form of generative and normative models. In this review, we will briefly recount scientific applications of descriptive behavioral analysis, before turning to how detailed pose and behavior data might also be used to build generative models and test classical theories about the principles that shape behavioral organization.

Descriptive models: turning video files into data

In many applications, the use of computational methods for behavior analysis can be likened to automation of manufacturing: the things produced are not fundamentally different from what was around before, but now they can be produced with greater precision and at scale. Several recent reviews^1–8 have described the use of modern computer vision or machine learning methods to characterize the pose, kinematics, or actions of behaving agents. At the heart of many approaches is markerless pose estimation, which characterizes animal posture in terms of a set of experimenter-defined 2D or 3D “keypoints” ^9–12. Pose estimates or other features extracted from behavior video can be used to quantify postural dynamics of animals^13–15, or paired with supervised (experimenter-defined) behavior detection^16–18 or unsupervised (data-defined) behavior discovery^19–23 methods to segment continuous posture data into actions (Figure 1A).

Figure 1.

Unsupervised behavior analysis and its evaluation. A) A typical unsupervised behavior analysis pipeline takes in continuous data (eg pose estimates) and segments it into motifs. Usually data is first pre-processed to remove sources of variance that are not meaningful for behavior (such as the absolute positions of the animals) and extract some representation of animal movement. These processed features are then segmented into motifs using some form of clustering algorithm. B) Different design choices in an unsupervised analysis produce different motifs: there is therefore a pressing need for ways to evaluate the “usefulness” of unsupervised analysis methods. Three commonly used approaches are comparison of motifs to manual annotations, eg in terms of Precision and Recall (left), ability of motifs to predict neural activity (center), and ability to distinguish between animal species, strains, or treatment groups based on the occurrence or patterning of motifs (right).

The practical advantage of these methods is twofold. First, they accelerate data processing when applied at scale. By training algorithms to detect behaviors of interest, researchers can distill hundreds of hours of video into precise and detailed records of animals’ actions that are ready for downstream analysis. In addition to saving human effort, this approach can improve annotation quality: whereas human annotators show substantial within-individual and between-individual variability in labeling behavior start and stop times¹⁷, automated algorithms are self-consistent. And second, computational behavior analyses can produce high temporal resolution, highly granular descriptions of animals’ actions, surpassing what is feasible for a human annotator^19–21.

The combined impact of automation and increased granularity is beginning to change the way behavioral experiments are designed and analyzed, with a growing number of labs opting to collect long-term, high-throughput recordings of animals in complex environments. Automating the frame-by-frame annotation of animal behavior has enabled high-throughput screening of hundreds of animals across strains, mutant lines, or experimental perturbations^15,23–30. Differences between animal cohorts in these assays can most simply be detected by identifying a change in the proportion of time animals express one or more behaviors, or by a change in the transition probability between pairs of behaviors^24,25. Difference in behavior can also be correlated with other within-cohort experimental measures, like gene expression²⁹.

Beyond the level of individual behaviors, differences between cohorts may be identified by training a classifier to distinguish animal groups given either postural features or a histogram of expressed behaviors^24,25,27. The magnitude of differences between cohorts can also be quantified and visualized to infer relationships between them^25,27. Interestingly, some studies have used dimensionality reduction methods (like Principal Component Analysis or Fisher’s Linear Discriminant Analysis) to identify different behavioral modes across individuals, mutant lines, or drug treatments^25,26,29. These analyses suggest a low-dimensional “behavior manifold” might be learned that can capture the variability in behavior expressions across individuals.

Generative model and forecasting: the Laplacian demon in the details

If we knew an animal’s entire sensory environment and history, could we predict what it would do next? How far into the future would our predictions hold? Generative models can help us better understand how animals integrate sensory cues and their own recent history to determine their ongoing behavior. A generative model captures the relationship between two probability distributions, specifically the probability distribution of one signal conditioned on the value of another. In terms of behavior, this might translate to learning a distribution over possible behavioral responses given an animal’s past actions or its sensory environment.

Generative models can be used as a means of categorizing behavior: for example, autoregressive Hidden Markov models capture the statistics of time series evolution, but their inferred hidden states can also be used for unsupervised behavioral segmentation²⁰. But generative models have also been used to uncover how animals’ sensory environment, state, and history shape expression of behavior^31–34. For example, fly courtship song type, previously thought to vary at random, can be predicted by postural cues from the courted female³¹, and furthermore this sensory-evoked behavior is better predicted when taking into account an estimate of the singing male’s internal state³³.

Animals often show preparatory movements or gradual escalation of interactions from which we can predict of future movements—for example, mice dart and tail-rattle before initiating an attack. As their name implies, generative models can also be used to generate “realistic-looking” behavioral data, predicting an animal’s actions given initial conditions^32,34; in machine learning, this area of research is called imitation learning. Model-synthesized trajectories predict how an animal would respond given an environment, provided that environment is close to the conditions the model was trained on: for example, simulated flies follow walls and perform wing extensions when they encounter other simulated flies³⁴. Trained models that can forecast behavior are often hierarchical, allowing them to capture structure in behavior on multiple timescales. One exciting promise of these models is that they provide continuous, behavior-related signals of different degrees of granularity, that can be contrasted with recorded neural dynamics to identify neural correlates of behavior or internal states³⁵.

Normative models: what’s my motivation?

Why does behavior have the structure it does? A normative model is one that uses some measure of value or utility to derive predictions of how a system “ought to” behave if driven by that utility. At its highest level, we expect animal behavior to either directly or indirectly further the survival of that animal’s genetic material. While this maxim is quite broad, principles of survival have informed many more specific theories of naturalistic behavior control. We will review examples from single-agent and dyadic behaviors, and discuss how computational behavior analysis might enrich our experimental investigation of these models. There is also a rich literature on principles of collective behavior^36,37, which will not be covered here.

Behavior of an agent in an environment.

Exploration of new environments is a fundamental behavioral drive, be it to establish territory, identify threats, or discover resources. Normative theories predict that exploring animals balance the reward of information gained with the cost of time passage³⁸ or risk of injury or predation. Foraging behavior, an extension of exploration, also encompasses an assortment of value-based decision-making tasks^39–41. During foraging, an animal makes dietary choices as to what resources to pursue⁴², and also must efficiently use time and energy when collecting resources. In a model environment of food patches, the marginal value theorem determines the precise time to migrate to new food sources, given full knowledge of the underlying average reward rate of the environment^43,44 (Figure 2A). At the opposite extreme, if nothing is known about the structure of the environment, random Lévy flight motion is optimal⁴⁵. Factors such as food source distribution, prey movement, and competition with conspecifics also impact strategy. Foraging under risk of predation introduces the further wrinkle of balancing the energetic cost of monitoring the environment with the survival cost of being caught by a predator^46,47.

Figure 2.

Normative models make predictions about the temporal organization of behavior. A) The Marginal Value Theorem predicts that animals foraging in a patchy environment should migrate to a new patch when their net rate of energy accumulation within a patch (blue line) matches the mean rate of energy accumulation across the environment (parallel red lines). Figure adapted from Charnov, 1976. B) In the resource holding potential model of aggressive escalation, animals integrate evidence to estimate the difference in fitness between fighters, and flee the encounter once their estimate passes a threshold. One possible formulation of this strategy is shown here, adapted from Enquist et al., 1990. The war of attrition model further hypothesizes that animals’ motivation to obtain resources can change their wilingness to flee, indicated here by the shift of the red line.

Behavior in pairs of interacting agents.

Dyadic behavior can include both symmetric interactions between members of a species and asymmetric interactions such as predator-prey behavior. Predatory imminence theory suggests a topology of defensive behavior, in which the type of behavior a prey species expresses is shaped by the immediacy and magnitude of predator threat⁴⁸ (Figure 3A). This escalating organization of defensive behaviors is built around a key conflict: that predators can be evaded either by reducing detectability (through freezing) or by overt escape actions that transiently increase detectability. Because escape behavior draws attention and interrupts ongoing foraging by the prey, an optimal strategy finds a balance between these two defensive behaviors.

Figure 3.

Behavioral escalation in predator-prey and within-species aggression contexts. A) Predator imminence theory describes an organization of defensive strategies in which level of predatory threat shifts the behavior animals express. Figure adapted from Fanselow & Lester, 1988. B) Escalation of aggressive encounters between conspecifics shows a similar scaling of behavior, a strategy thought to minimize risk of injury. Escalation could serve to establish animals’ relative resource holding potentials, or could be a “war of attrition” process of animals communicating their level of commitment to a contested resource.

Outside of predator-prey conflict, violence in nature occurs predominantly between members of the same species, and rarely results in serious injury. The survival value of intra-species aggression has been questioned since Darwin, who hypothesized that aggression ensures that the fittest members of a species obtain greater territories and resources. Subsequent theories further posed that aggression could help balance distribution of animals across available environment⁴⁹. In social species, establishment of dominance hierarchies is thought to stabilize groups and create networks that shape the flow of information among group members^50,51. An extensive literature in game theory has explored the structure of aggression, and gave rise to the concept of “evolutionarily stable strategies”: behavioral strategies in a species that are stable to small perturbations in the form of mutant strategies⁵².

Normative models of aggression often focus on the observation that, like defensive behavior, aggressive behaviors vary in their intensity (Figure 3B). Escalation of aggression could serve several functions. In the “resource holding potential” hypothesis, escalation allows animals to exchange information about relative fitness while minimizing risk of injury, by only escalating if the relative fitness of the pair remains unclear (Figure 2B) ^53,54. This hypothesis predicts that animals more closely matched in size escalate aggression further than more asymmetric pairs, a prediction that holds to varying extents across species^55,56. Alternatively, the “war of attrition” model⁵⁷ poses that escalation signals the cost an animal is willing to pay for a resource, explaining how manipulations such as food deprivation might motivate an animal to escalate further or faster⁵⁸. Escalation may also simply reflect the animal’s relative levels of aggression and fear⁵⁹. Models of aggressive escalation were extensively investigated in the 1970’s, 80’s, and 90’s, in species ranging from dung beetles to hermit crabs to red deer; this work is excellently reviewed in ⁶⁰.

Going forward.

Computational behavior analysis is experiencing a period of explosive growth, fueling and fueled by a push among neuroscientists to study more complex and naturalistic animal behaviors. Yet existing normative models of behavior largely precede the computer vision revolution in animal tracking. New behavior quantification methods will allow more rigorous testing of predictions made by classical normative models, whereas generative modeling provides methods for capturing behavioral control policies based on precise quantification of animals’ sensory environment and behavior. An open challenge for modern behavior analysis is to re-evaluate our normative theories of behavior in the light of more plentiful data, to determine how useful they are as descriptors of animals’ actions.

Towards this goal, the theoretical framework that has been established around exploration and foraging strategies provides rich ground for experimental exploration. Exploratory behavior and learning in neuroscience are often studied in conceptual tasks that do not require physical exploration, permitting neural correlates to be measured^61,62. An exciting promise of computational behavior analysis is that it allows exploration to be studied in complex physical environments such as mazes^63,64, where—in striking contrast to conceptual exploration—mice rapidly learn long sequences of actions, and can show one-shot learning of their home path out of the maze⁶⁴. By tracking exploring animals’ posture, researchers can detect deliberative actions such as “vicarious trial and error”, when animals pause to investigate options at a choice point^65,66. Quantification of deliberative actions may help study another normative theory of exploration, the principle of effort minimization, which predicts that as an environment or scenario becomes familiar animals may swap between decision strategies, balancing low-effort habit and high-effort planning and simulation⁶⁷.

Analysis and generative modeling of animal actions and sensory environments during naturalistic foraging could also help determine how animals construct and update their internal model of food availability in their environment. For example, recent work has demonstrated that imitation learning can capture learned foraging behavior in a head-fixed task⁶². Elimination of the behavior annotation bottleneck could also allow for higher volume experiments testing the sensitivity of foraging behavior to combinations of environmental parameters such as degree of patch structure, presence of threat or conspecific cues, choices between food types, or changes in food availability.

Normative theories of dyadic interactions have often focused on the rules governing transitions between behaviors: either the switch from freezing to escape in predator imminence theory, or the gradual escalation of aggression in intra-species encounters. These progressions are challenging to study in the lab, as animals show rapid habituation and priming after a few presentations of a threating stimulus or conspecific. With automated analyses, classical assays like the resident-intruder paradigm could be expanded to longer timescales in enriched environments, so that animals have time and space for more naturalistic encounters to unfold⁶⁸.

Finally, one exciting area for exploration is to develop theories and models for how competing motivational signals should interact to shape behavior. Animal behavior in the wild strikes a balance between competing drives: hunger and thirst, predator defense, and social (reproductive, parenting, and territorial) motivations. These drives can be thought of as low-dimensional intervening variables between sensation and action⁶⁹: any single drive can be affected by multiple sensory cues- for example, both mating experience⁷⁰ and social isolation⁷¹ make mice more aggressive, and both dry food and salt make mice thirsty. And drives can be linked to any learned behavior: both aggression and thirst can be used to motivate nose-poking or lever-pressing in operant tasks^72,73.

We cannot measure an animal’s aggressive motivation directly, but we can apply experimental manipulations that we know will alter aggression—and also hunger, thirst, stress, reproductive state, or time of day—and measure resulting changes in behavior. How might these different state manipulations interact? Before we can generate hypotheses of how the brain is controlling behavior, it is good to stop and ask: how complex is behavior in the first place? If the annotation bottleneck can be resolved, we might begin to use high-throughput behavioral studies to ask whether intervening variables of hunger, thirst, fear, or aggression are indeed the one-dimensional signals we call them, or whether behavioral control policies and their underlying neural drives are more complex.

Box 1: Future challenges for descriptive models of behavior.

Performance.

A particular challenge facing unsupervised behavior discovery is deciding what constitutes a “good” representation of behavior⁷⁴. Common metrics for evaluating the quality of an unsupervised behavior discovery method include agreement with human annotations, ability to account for variance in neural data, and utility in distinguishing between strains or conditions (Figure 1B), however many paper apply these only to in-house datasets, making it difficult to compare methods. Establishing benchmark datasets for evaluating behavior algorithms may help overcome this issue⁷⁵.

Generalization.

Behavior classifiers and pose estimators trained in one lab rarely perform well out-of-the-box when used by another lab, unless the two groups standardize their data acquisition setup. Creating algorithms that can be tuned to new environments, or that are trained on larger datasets, would help ensure replicability of the results of computational analyses.

Interpretability.

A common complaint about machine learning methods is that they are a black box. However, automated methods could actually become more transparent in their choices than human annotators, who can struggle to communicate their decision process during annotation. Efforts to make human-interpretable behavior classification tools may help scientists have greater confidence in their results⁷⁶.

Long timescales.

Behavior has structure over multiple scales: from the coordination of actions into sequences⁷⁷, to effects of priming⁷⁸, habituation⁷⁹, satiety^80,81, and time of day^82,83. Long timescales of behavior can be captured intrinsically by sequence learning⁸⁴ or hierarchical models³⁴, or may be recovered post-hoc by coarse-graining of action sequences⁸⁵. But there is still ample space for exploration in this area.

Level of granularity.

Neural recording datasets introduce another challenge for behavior analysis: not just finding the representation of behavior that best captures observed postural variance, but finding the right level of granularity to account for observed neural activity. The optimal behavior representation in this sense will likely be different for different brain regions, with some regions better explained by fine-grained, sub-second action motifs, and other regions more correlated with an animal’s overall behavioral objective (such as aggression or reproduction.) How best to integrate quantitative behavior analysis with neural imaging datasets is an exciting area of ongoing research.

Interacting agents.

Multi-agent behavior poses unique challenges for analysis, as these datasets include times when animals are interacting and times when each individual is behaving independently. This affects unsupervised methods in particular, as the relative positioning of agents is sometimes critical for interpreting behavior and sometimes doesn’t matter at all. We hypothesize that one way to overcome this challenge is to base behavior discovery methods on forecasting models, which could be trained to incorporate postural information from an agent’s partner only when that information is relevant to predicting what the agent does next.