Body language signals for rodent social communication in rodents

Abstract

Integration of social cues to initiate adaptive emotional and behavioral responses is a fundamental aspect of animal and human behavior. In humans, social communication includes prominent nonverbal components, such as social touch, gestures and facial expressions. Comparative studies investigating the neural basis of social communication in rodents has historically been centered on olfactory signals and vocalizations, with relatively less focus on non-verbal social cues. Here we outline two exciting research directions: First, we will review recent observations pointing to a role of social facial expressions in rodents. Second, we will outline observations that point to a role of “non-canonical” rodent body language: body posture signals beyond stereotyped displays in aggressive and sexual behavior. In both sections, we will outline how social neuroscience can build on recent advances in machine learning, robotics and microengineering to push these research directions forward towards a holistic systems neurobiology of rodent body language.

Introduction

Many social cues are nonverbal (a smile, a raised eyebrow, a shrug). A failure to correctly process and interpret social cues is thought to underlie social dysfunction in many neuropsychiatric conditions, from negatively biased interpretations of social signals in depression (Weightman et al., 2014) to a near-complete breakdown of social understanding in some individuals with autism spectrum disorder (Klin et al., 2002). A comparative investigation in rodents – where we have advanced tools for monitoring and manipulating neural activity during behavior – could be a powerful way to advance our understanding of the evolution and function of neural circuits for processing social cues (Anderson and Adolphs, 2014).

In general, we know little about the use of posture and gesture in orchestrating social group behavior. A comparative study of body language is an old idea (Darwin, 1872), but the systems neuroscience of rodent body language is still in early days. It is clear that rodents make use of stereotyped body postures and movements in sexual courtship (e.g., female rats darting) and in aggression and dominance (e.g., rat boxing) (Schweinfurth, 2020). However, compared to our detailed knowledge about the processing of socially significant olfactory signals in aggressive (Anderson, 2016), sexual (Lenschow and Lima, 2020) and parental (Kohl, 2020) behaviors, we know much less about how body language signals (touch, movement, postures) are integrated by the rodent brain.

Facial expressions and whisking

Mice and rats display a variety of facial expressions. Both mice (Langford et al., 2010) and rats (Sotocina et al., 2011) make stereotyped expressions (‘grimaces’) with their facial musculature in response to pain and stress: tightening of orbital muscles, squinting eyes and retraction of the ears (Figure 1a). Rats also make facial expressions (forward movement and blushing of the ears) (Finlayson et al., 2016) (Figure 1b) and jumps (Ishiyama and Brecht, 2016) when experiencing positive emotions, such as after tickling. In wild mice, ear posture correlates with their behavior in tests that are thought to measure the animals’ emotional state: approaching a novel odor and exploring the open arms in an elevated plus-maze. Mice with retracted ears behave more cautiously than mice with their ears in an upright, forward position (Lecorps and Féron, 2015). Rats also display different facial expressions when presented with tastants that evoke different emotional responses (e.g. bitter, unpleasant quinine, and sweet and palatable sucrose) (Grill and Norgren, 1978a, 1978b).

Figure 1: Rodent facial expressions.

(a) Pain grimace in rats: Orbital tightening, cheek flattening, folded, curled ears angled forwards or outwards (Sotocina et al., 2011) (b) Altered facial expression after tickling: Ear blushing and ears angled backwards (Finlayson et al., 2016) (c) In a mouse resident-intruder paradigm, the resident and intruder mice display two different facial expressions maintained during fighting: The resident displays tightened eyes and flattened ears, while the intruder displays widened eyes and erect ears (Defensor et al., 2012) (d) Examples of distinguishable facial expressions in mice: expressions after drinking sweet and bitter liquid, pain and freezing behavior (Dolensek et al., 2020) (e) In rats, whiskers are more protracted in social facial interactions before an aggressive interaction than in social facial interactions before nonaggressive interactions. (Wolfe et al., 2011). Figure permissions pending. Permissions: (a) Reproduced from (Sotocina et al., 2011) under a CC BY 2.0 license, (b) reproduced from (Finlayson et al., 2016) under a CC BY 4.0 license, (c) reproduced from (Defensor et al., 2012) with permission from Elsevier (d) reproduced from (Dolensek et al., 2020) with permission from AAAS (e) reproduced from (Wolfe et al., 2011) with permission from APA.

If rodent facial expressions differ between emotional states, that raises the possibility that these facial cues could be perceived by conspecifics and play a role in social communication. Increasing evidence across species suggests that facial expressions are displayed in social situations, and distinguishable by conspecifics. For example, ear wiggling is a social signal displayed by female rats during courtship (Erskine, 1989; Vreeburg and Ooms, 1985). Naked mole rats – a eusocial rodent species – have an extensive vocabulary of non-verbal body language, including elaborate facial interactions (e.g. head-on pushing, mouth gaping and tooth fencing) (Lacey et al., 2017). These facial interactions are involved in the control of ‘lazy’ workers (Reeve, 1992) and help maintain reproductive suppression (Clarke and Faulkes, 1997). A landmark study found that when an intruder mouse was placed into the cage of the resident mouse, the two mice displayed two different facial expressions, which they maintained, even during fighting: The resident displayed tightened eyes and flattened ears, while the intruder displayed widened eyes, erect ears and an open mouth (Defensor et al., 2012) (Figure 1c). A recent study found that it was possible to train an image classifier to distinguish between facial expressions in head fixed mice in a wide range of situations (aversive and palatable tastants, LiCl-indcued nausea, painful electric shock, freezing) (Dolensek et al., 2020) (Figure 1d).

Rats are nocturnal (Barnett, 1975), have modest visual acuity (Prusky et al., 2000), and often encounter conspecifics head-on in burrows (Blanchard et al., 2001). This suggests that ethologically they more often would sense faces of conspecifics with the whiskers rather than by vision. Beyond palpating the face of a social interaction partner, the whiskers themselves might also convey information, social or otherwise. During rat social facial interactions, whiskers are more protracted in aggressive than in nonaggressive interactions (Figure 1e), and female rats whisk with a lower amplitude when meeting male conspecific than when meeting female conspecific (Wolfe et al., 2011). During social facial interactions, cessation of sniffing by a subordinate rat decreases the likelihood that a dominant rat will initiate antagonistic behaviors (Wesson, 2013a). It is still unclear what aspects of rat behavior communicate subordination during such a facial interaction: The cessation of sniffing itself (Wesson, 2013b), altered patterns of ultrasonic vocalizations (Assini et al., 2013; Rao et al., 2014), whisking (Wolfe et al., 2011), body posture (Barnett, 1975) or – perhaps – some combination of the recently described, sniff-locked nose-twitching and head-bobbing (Kurnikova et al., 2017).

Positioning, motion, and asymmetry of the mouth, nose and whiskers (Dominiak et al., 2019; Towal and Hartmann, 2006) and rapid whisker twitches (whisker ‘pumps’) (Wallach et al., 2020) are predictive of upcoming motor behavior (e.g., running, turning). Whisker pumps might serve as a social cue during facial interactions. Rats have also been shown to display contagious yawning (Moyaho et al., 2014). Yawning is a social signal in many species (Guggisberg et al., 2010), but we know little about if and how yawning functions as a social signal in rats (Moyaho et al., 2017).

Mice also spontaneously engage in social facial whisker touch (Heckman et al., 2017) and neonatal whisker trimming leads to social behavior deficits in adult (Soumiya et al., 2016). Mice also perform an interesting whisking-related social dominance behavior referred to as ‘whisker barbering’: dominant mice will pin down subordinates, grab their vibrissae by the teeth and pull them out by the roots with a hard tug (Sarna, 2000; Strozik and Festing, 1981).

Detailed knowledge about the facial musculature (Haidarliu et al., 2010, 2011, 2012, 2013, 2014; Hill et al., 2008), 3D facial anatomy (Belli et al., 2018; Huet and Hartmann, 2014; Knutsen et al., 2008), and whisker biomechanics (Yang et al., 2019; Zweifel et al., 2019) (Figure 2a-d) might together provide an understanding of the topology of facial expression space, and predict the range of facial expressions that a rodent can produce (Hill et al., 2008; Luo et al., 2020; Sherman et al., 2013; Simony et al., 2010). This would enable the description of a kind of ‘natural scene statistics’ of a facial expressions, a powerful analytical framework pioneered in classic investigations of visual cortex (Geisler, 2008). Building on recent approaches in robotic methods of delivering complex, 3D sensory stimuli to whiskers (Goldin et al., 2018; Jacob et al., 2010; Ramirez et al., 2014) (Figure 2e), it might be possible to present complex, naturalistic, and/or full-field patterns of whisker stimulation to estimate facial expression receptive fields in the whisker system. Such an approach would allow us to understand if and how the neural encoding of socially significant whisker stimuli (Bobrov et al., 2014; Ebbesen et al., 2017, 2019; Lenschow and Brecht, 2015; Rao et al., 2014) differ from the encoding of non-social stimuli, such as objects and textures (Maravall and Diamond, 2014; Petersen, 2019).

Figure 2: Towards a holistic systems neuroscience of social facial expressions.

(a) Anatomical drawing of the rat whisker motor plant (Haidarliu et al., 2010) (b) Estimating the functional anatomy of the rat whisker musculature (Hill et al., 2008) (c) Biomechanical model of the rat whisker motor plant (Haidarliu et al., 2011) (d) A computational model of a realistic rat whisker field interacting with a complex scene (Zweifel et al., 2019) (e) A robotic system for simultaneous stimulation of 24 individual whiskers (Goldin et al., 2018) (f) A head-mounted camera system for mice (Meyer et al., 2018) (g,h) Head-mounted, physiological sensors systems for recording of sniffing, nose movements and head-bobbing in rats (Kurnikova et al., 2017) (i) Automatic whisker tracking in a freely moving mouse (Gillespie et al., 2019) (j) Using fluorescent dye to visualize single whiskers within the whisker field (Rigosa et al., 2017). Permissions: (a) reproduced from (Haidarliu et al., 2010) with permission from Wiley and Sons (b) reproduced from (Hill et al., 2008) with permission, copyright 2008 Society for Neuroscience, (c) reproduced from (Haidarliu et al., 2011) with permission from Wiley and Sons, (d) With permission from N. Zweifel and M. J. Z. Hartmann, (e) reproduced from (Goldin et al., 2018) under a CC BY 4.0 license, (f) reproduced from (Meyer et al., 2018) under a CC BY 4.0 license, (g-h) reproduced from (Kurnikova et al., 2017) with permission from Elsevier, (i) reproduced from (Gillespie et al., 2019) with permission from Elsevier, (j) reproduced from (Rigosa et al., 2017) under a CC BY 4.0 license.

Engineering advances in miniaturization make it now possible to record gaze direction and eye movements – and likely also whisker movements – by head-mounted cameras in freely-moving rats (Wallace et al., 2013) and mice (Meyer et al., 2018; Sattler and Wehr, 2020) (Figure 2f). Using this approach, it was recently reported that mice close their eyes when a conspecific is within close distance (Meyer et al., 2020) – an unexpected and interesting observation in the context of making and recognizing facial expressions. With miniaturized, head-mounted thermocouples, accelerometers, gyrometers, and Hall-effect probes, it is possible to quantify sniffing patterns, nose movements, and head-bobbing in freely-moving animals (Kurnikova et al., 2017; Wesson, 2013a) (Figure 2g-h). It remains unclear how these aspects of facial behavior vary during social interactions. Alternatively to using head-mounted cameras to record facial behavior in freely moving animals, a recently described method combines real-time tracking with motorized cameras to capture high-resolution ‘close-up’ images of animals moving in a large 3D arena (Nourizonoz et al., 2020).

Quantifying whisker movements during social facial interactions remains a challenge. Whisker tracking of solitary animals has reached high levels of accuracy. In head-fixed mice with most whiskers trimmed, simultaneous measurements of the three-dimensional shapes and kinematics of eight whiskers can be obtained automatically (Petersen et al., 2020). However, in socially interacting animals (with full, un-trimmed whisker fields), overlapping and occluded whiskers remain a major problem, and thus far social whisking patterns have either been tracked manually (Bobrov et al., 2014; Lenschow and Brecht, 2015; Wolfe et al., 2011) or approximated by automatically tracking the average movement of the whisker field as a whole (Ebbesen et al., 2017). A promising path towards automatic whisker tracking in socially-interacting animals is to combine recent advances in automatic whisker tracking in freely-moving animals (Gillespie et al., 2019) (Figure 2i) with techniques for tracking the movement of single whiskers despite overlaps and occlusions by painting single whiskers with a fluorescent dye (Nashaat et al., 2017; Rigosa et al., 2017) (Figure 2j).

Posture and movement as body language signals

A role of body language in signaling distress

Several studies have shown that rats will actively help conspecifics in distress. Rats will press a lever to lower a distressed and wriggling rat dangling in a harness (prodded with a sharp pencil if it did not exhibit sufficient signs of discomfort) (Rice and Gainer, 1962)_(Figure 3a), rats will press a lever to remove a conspecific from a water tank (Rice, 1965), and rats will leave a dark and comforting hiding place and stay in a brightly lit, open arena to ensure that a nearby conspecific does not receive a painful shock (Preobrazhenskaya and Simonov, 1974)_(Figure 3b). More recent studies have shown that rats (Bartal et al., 2011) and mice (Ueno et al., 2019a) will open a door to release a conspecific trapped in a small plastic tube (Figure 3c), and that rats will open a door that lets cagemate escape a pool of water (Sato et al., 2015)(Figure 3d).

Figure 3: Investigating body language signals of distress.

(a) A classic paradigm to investigate actibe helping in rats: Aversive dangling from a harness. (Rice and Gainer, 1962) (b) Another classic paradigm: One rat has to decide between a brightly lit arena and a dark and comforting shelter. Entering the shelter delivers a shock to a nearby conspecific. (Preobrazhenskaya and Simonov, 1974). (c) The aversive-restraint-tube paradigm: One rat can open a door that lets a conspecific escape a transparent tube. (Bartal et al., 2011) (d) The aversive-swimming-pool paradigm: One rat can open a door that lets a conspecific escape a pool of water. (Sato et al., 2015).

What drives the behavior of the helper animal? In rats, restraint-tube-opening behavior depends on familiarity with the strain of rat in distress (Bartal et al., 2014). Behavioral changes occur after drugging the helper rat, with benzodiazepine sedation leading to longer opening latency (Bartal et al., 2016) and heroin abolishes opening(Tomek et al., 2019). Door-opening latency changes if there are multiple potential helpers, and depends on if these ‘bystanders’ are sedated (Havlik et al., 2020). In voles, oxytocin receptor knockout delays door-opening for a soaked conspecific (Kitano et al., 2020). Multiple studies have varied the rescue paradigms to clarify what emotional states might motivate door opening and helping behavior. Helpers might be motivated by empathic concern for the distressed, may desire rewarding social interactions, open the door out of curiosity or boredom, or might be irritated by aversive cues from the trapped animal, among other hypotheses (Blystad et al., 2019; Carvalheiro et al., 2019; Cox and Reichel, 2020; Hachiga et al., 2018; Hiura et al., 2018; Schwartz et al., 2017; Silberberg et al., 2014; Silva et al., 2020; Ueno et al., 2019b; Vasconcelos et al., 2012).

Relatedly, the behaviors of the distressed animal might also be an important factor. Trapped animals produce lower-frequency distress calls in the first restraint sessions (Bartal et al., 2011, 2014), but might also display other signs or signals of stress, such as seen in pain and sickness (Barnett, 1975; Kolmogorova et al., 2017). This raises the possibility that other signals such as olfactory cues (Bredy and Barad, 2009; Kiyokawa et al., 2006) or elements of body language such as gesture and posture could be used to signal distress and solicit help. Several studies have found rodents are indeed sensitive to body language signals of distress, such as freezing (Atsak et al., 2011; Cruz et al., 2020), and rats prefer a room decorated with images of conspecifics in a neutral pose rather than a room decorated with images of conspecifics in pain (i.e., facial grimaces and hunched posture) (Nakashima et al., 2015).

Controlled experiments involving robotic animals (Abdai et al., 2018) or virtual animals (Naik et al., 2020) is powerful way to probe the sensitivity of animals to visual social stimuli. Some studies have simulated body language distress signals by robotic animals (Abdai et al., 2018). Rats will work to release a moderately rat-like robot from a restraint tube, and rats seem to discriminate between robots based on behavior (Quinn et al., 2018). There is ongoing work to develop more complex rat robots, capable of realistic postures and movement patterns (Ishii et al., 2013; Li et al., 2020; Shi et al., 2015).

Instructing social partners through body language

Several studies have investigated the behavior of rats in artificial social games which also might involve body language. Rats will cooperate at rates above chance level in iterated prisoner’s dilemma games (Gardner et al., 1984; Viana et al., 2010; Wood et al., 2016), but the interpretation of such games is complex, since an iterated prisoner’s dilemma can be dominated without any theory of mind (Press and Dyson, 2012). Body language is usually not quantified but a classic study reported that cooperation would break down if the animals could not see each other, and that rats would engage in specific left- or right-turning feint behaviors apparently to influence the behavior of the partner animal (Gardner et al., 1984). The importance of visual observation has been highlighted in another social coordination nose-poke task (Łopuch and Popik, 2011).

Multiple studies have found that rats will work to deliver food to conspecifics (Dolivo and Taborsky, 2015; Rutte and Taborsky, 2007, 2008; Schneeberger et al., 2012; Schweinfurth and Taborsky, 2018a, 2018b), and that – when given the option to donate food to conspecifics at no extra cost to themselves – rats will prefer that conspecifics receive food also (Hernandez-Lallement et al., 2015; Kentrop et al., 2020; Márquez et al., 2015; Oberliessen et al., 2016). Many of these studies report observations that are in line with the supposition that rats use body language signals to communicate what they want the ‘chooser’ animal to do, and that the chooser animal is sensitive to these signals. In one study, the likelihood of donating food by the chooser rat incurring no extra cost, was modulated by the display of food-seeking behavior by the prospective recipients, expressed as poking a nose port and by social interactions through a mesh (Márquez et al., 2015). In another study, where rats could work to deliver food to a conspecific only, subject rats provide food correlated with the intensity of movements and body postures displayed by the prospective recipients. These putative body language signals included stretching their paws towards the food, sniffing through the mesh in the direction of the food, and other attention-grabbing behaviors directed at the subject rat (Schweinfurth and Taborsky, 2018a).

Studies investigating the behavior of groups of mice in complex environments have found marked individual differences in displays of social postures and movements (Forkosh et al., 2019; Torquet et al., 2018), and patterns in social interaction partnering (König et al., 2015; Peleh et al., 2019; Shemesh et al., 2013; Weissbrod et al., 2013). However, while postures and movement patterns correlate with social dominance (Forkosh et al., 2019; Wang et al., 2014), it is still unclear if and how body language cues might help establish, maintain, or adjust the dominance hierarchy (Forkosh et al., 2019) or social network of co-habituation (König et al., 2015).

Individual differences in movement and postures during co-housing or colony dynamics might mirror the observation that animals tend to take on different behavioral roles. When rats are moving together in dyads, some become ‘leaders’ and some become ‘followers’ (Weiss et al., 2015). In a test where rats have to dive underwater to collect morsels of food, some become ‘divers’ (swimming and collecting food), and other rats become non-divers which wait for the other animal to bring them food (Grasmuck and Desor, 2002; Krafft et al., 1994). In wild mice performing collective nest building, some mice will become nest-builders (carrying out the vast majority of the work in collecting nesting material) and some mice will only participate weakly or not at all (Serra et al., 2012).

One important set of behavioral roles in group-housed animals is co-parenting and caretaking of infant rodent pups. Parenting behavior and active care for pups is orchestrated by innate circuits to some degree (Kohl, 2020). In the context of body language, however, maternal female mice can solicit the help of sexually experienced males (Liang et al., 2014; Liu et al., 2013; Tachikawa et al., 2013) and virgin females (Ehret et al., 1987; Krishnan et al., 2017; Marlin et al., 2015) for aspects of pup caretaking (e.g., nest building, pup retrieval, crouching, and pup grooming). The parental behaviors expressed by males and female virgins develop with exposure or experience with pups (concaveation) and during co-housing with a dam and litter. The presence of experienced dams accelerates concaveation (Carcea et al., 2020; Marlin et al., 2015) indicating that dams engage in some behaviors or interactions that affect the emergence of co-parenting abilities in males or virgin females. Olfactory and auditory cues from the dam play a role. Blockade of these signals delay the development of co-parenting in males and – even without visual input from the dam – replay of dam vocalizations or dam odors can induce parenting in males (Liu et al., 2013). Body language and motor activity of the dam also contribute, as dams will actively engage virgins in maternal care by ‘shepherding’ the virgins to the nest and pups. Furthermore, dams demonstrate maternal behavior in spontaneous pup retrieval episodes that allow virgin females to learn by observation (Carcea et al., 2020). Active social engagement and demonstration by dams might be a key driver in facilitation fast social learning of co-parenting. Free-living, wild dams selectively choose to communally nurse (Ferrari et al., 2018; Harrison et al., 2018), but we do not yet know the role of body language signals in coordinating co-parenting between dams in outdoor colonies.

The apparently active demonstration of parenting is in contrast to other studies reporting that wild rats do not rapidly acquire new foraging techniques by observation, even if they are performed by conspecifics (Galef, 1982). It is, however, in line with other reported examples of rodents learning by observation (Petrosini et al., 2003). For example, rats can learn to solve a Morris water maze by observing a trained conspecific swim to the hidden platform (Leggio et al., 2000), mice can learn to solve a complex ‘puzzle box’ by observing conspecifics (Carlier and Jamon, 2006), and rats will imitate joystick movements that they have seen a conspecific make to receive a food reward (even though their joystick movements do not actually affect their reward at all) (Heyes et al., 1994).

Pose estimation and quantitative analysis of body language

Methodological advances in computer vision and machine learning provide new ways to monitor and analyze body language signals for social behavior. Multiple open source packages for machine-learning based markerless tracking of posture and body parts in single animals have recently been developed, including DeepLabCut (Mathis et al., 2018), LEAP (Pereira et al., 2020), DeepPoseKit (Graving et al., 2019), OptiFlex (Liu et al., 2020), DeepGraphPose (Wu et al., 2020) and others (von Ziegler et al., 2020). However, translating single animal tracking to multiple animals is not straightforward, for at least two reasons. First, the camera view on a specific animal might be occluded by other animals (especially if any have neural implants). Second, even if all body parts are visible, the body parts have to be ‘grouped’ correctly and assigned to the correct animal.

One straightforward way to distinguish two interacting animals is to use animals that are physically marked or of a different coat color (Hong et al., 2015; Nilsson et al., 2020; Segalin et al., 2020) (Figure 4a-c). This is a robust method, but excludes some use-cases (e.g., studies of behavioral genetics that require a specific background or where it is important that animals come from the same litter). Another method have used deep neural networks to recognize body parts and metrics of spatiotemporal continuity to group body parts and maintain tracking of animal identities, in unmarked animals of the same coat color (Pereira et al., 2020) (Figure 4d). Another approach maintains the identities of multiple animals by training a network to recognize subtle differences in the appearance of individual animals (Pérez-Escudero et al., 2014; Romero-Ferrero et al., 2019; Walter and Couzin, 2020) (Figure 4e). Another approach combines the use of implanted RFID chips(Kritzler et al., 2007; Peleh et al., 2019) and the use of depth videography (Aguilar-Rivera et al., 2018; Gerós et al., 2020; Hong et al., 2015; Matsumoto et al., 2013; Sheets et al., 2013; Wiltschko et al., 2015) to track movements patterns and body postures in multiple mice, in real time (Chaumont et al., 2019) (Figure 4f). The RFID-based identity tracking provides a robust cross-validation of animal position (when sufficiently separated), but may interfere with electrophysiological recordings. We have taken a related approach, building on pioneering work in tracking by physical modeling in rats (Aguilar-Rivera et al., 2018; Matsumoto et al., 2013) (Figure 4g), that combines deep learning-based keypoint detection and depth videography in a robust tracking algorithm capable of automatically tracking a 3D model of the posture of interacting mice. This method is compatible with electrophysiology (robust to occlusions and camera artifacts due to wires and a neural recording implant carried by the mouse on the right) (Figure 4h) (Ebbesen and Froemke, 2020).

Figure 4: New computational methods for automatically estimating body postures in socially interacting rodents.

(a,b) Disambiguating body parts of two mice by their coat color (a: (Nilsson et al., 2020), b: (Segalin et al., 2020)). (c) Imaging mice of different coat colors and estimating their body postures by approximating the animals as ellipses in a simultaneously acquired depth image (only depth image shown) (Hong et al., 2015). (d) Tracking animals of the same coat color by using a spatiotemporal loss function to assign detected body pars to the correct animals (Pereira et al., 2020) (e) Tracking the identity of multiple animals by training a network to recognize subtle differences in each individual animal’s appearance (Romero-Ferrero et al., 2019) (f) Combining depth videography with implanted RFID-chips to track and disambiguate multiple mice in real time (Chaumont et al., 2019) (g) Combined depth videography and physical modeling in a computational tool for semi-automatic tracking of body postures in interacting rats (Aguilar-Rivera et al., 2018; Matsumoto et al., 2013). (h) Combining deep learning, physical modeling and a particle-filter based tracking algorithm with spatiotemporal constraints to automatically track the body postures of interacting mice, compatible with electrophysiology (robust to occlusions and camera artifacts due to wires and a neural recording implant carried by the mouse on the right) (Ebbesen and Froemke, 2020). Permissions: (a) With permission from S. R. O. Nilsson & S. A. Golden. (b) With permission from A. Kennedy. (c) reproduced from (Hong et al., 2015) with permission from National Academy of Sciences (d) With permission from T. Pereira & J. Shaevitz. (e) With permission from the idtracker.ai team, (f) reproduced from (Chaumont et al., 2019) with permission from Springer Nature, (g) reproduced from (Matsumoto et al., 2013) under a CC BY license.

Beyond recording raw postural and movement data, machine learning methods have also provided new ways to segment raw tracking data into behavioral categories in a principled and objective manner, and to discover behavioral structure – the building blocks of body language – in a purely data-driven way. The latter is especially promising, because it could allow discovery of new postures and movement patterns, purely from statistical properties in the behavioral kinematics and agnostic to potential observer bias.

A very effective way of automatically segmenting raw tracking data is to use a supervised approach and train a classifier to reproduce human annotation of behavioral categories (Hong et al., 2015; Nilsson et al., 2020; Segalin et al., 2020). This approach will , when using modern, deep-leaning based classifiers and large training sets (Nilsson et al., 2020; Segalin et al., 2020), provide a precise way to automatically annotate behavioral data. Unsupervised approaches learn the behavioral categories from the data itself. Tracked behavioral features from an animal (e.g,. 3D coordinates of many body parts) is a high-dimensional time series. To find structure in such, it is possible to draw from a recent work in laboratory studies of worm and insect behavior (Brown and Bivort, 2018; Calhoun and Murthy, 2017) and field ethology (Patterson et al., 2017; Smith and Pinter-Wollman, 2020).

One approach to discover behavioral categories is to look for ‘building blocks’ of the observed behaviors that re-occur. To this end, an elegant and robust approach is to perform a nonlinear projection from the high-dimensional space of all tracked body part coordinates (often augmented with derived features, such as time derivatives and spectral components) down to a low-dimensional 2D (Berman et al., 2014, 2016; Braun et al., 2010; Johnson et al., 2020; Klibaite and Shaevitz, 2019; Klibaite et al., 2017; Pereira et al., 2019; Werkhoven et al., 2019; York et al., 2020) or 3D manifold (Hsu and Yttri, 2019; Mearns et al., 2020) in a manner that preserves local similarity (e.g. t-SNE (Maaten and Hinton, 2008)). On this low-dimensional manifold, similar behaviors will form clusters, that can be identified by density-based clustering algorithms. The generated clusters are manually inspected and curated (e.g. merged or split) and assigned names (e.g., ‘locomotion’, ‘grooming’, etc.).

Another approach to discover behavioral categories is to define a generative model – e.g. some flavor of state space model – and fit this model to the high-dimensional time series of tracked body features (Adam et al., 2019; Calhoun et al., 2019; DeRuiter et al., 2016; Ebbesen and Froemke, 2020; Heiligenberg, 1973; Katsov et al., 2017; Macdonald and Raubenheimer, 1995; Markowitz et al., 2018; Tao et al., 2019; Wiltschko et al., 2015, 2020). This approach is attractive, because it is highly expressive: It is possible to define very complex models, e.g. by adding autoregressive terms (Wiltschko et al., 2015), by allowing for complex hidden dynamics (Linderman et al., 2019), by incorporating nested structures (Tao et al., 2019), and by including nonlinear transformations (Calhoun et al., 2019). It is also possible to explicitly incorporate knowledge about the animals anatomy, by writing a full generative model of the animal’s body itself, akin to (Johnson et al., 2020; Merel et al., 2019). However, these methods also have drawbacks. First, complex models quickly become prohibitively computationally expensive to fit to data. Fitting can be accelerated, e.g. by using fast modern and efficient sampling algorithms (Leos-Barajas and Michelot, 2018) or GPU-accelerated variational inference(Ebbesen and Froemke, 2020), but even these methods often show poor mixing/convergence for complex models. Second, even if a model is well fit to data, there is no principled way to discover what the ‘true’ latent structure is (e.g., the true number of hidden states or transition graph structure) (Adam et al., 2019; Fox et al., 2010; Li and Bolker, 2017; Pohle et al., 2017). Thus, for example, the number of hidden states in a state space model of behavior – i.e. the number of different behavioral categories – has to be set using a heuristic, e.g., by fitting a model with the number of latent states as a free parameter and then choosing a cutoff (Markowitz et al., 2018; Wiltschko et al., 2015, 2020), by fixing the number of states based on inspection of raw data and the desired coarseness of the model (Ebbesen and Froemke, 2020; Katsov et al., 2017; Tao et al., 2019), or – in a very elegant approach – by comparing models with different latent structure according to their ability to capture multiple aspects of the observed data, such both the most likely state and transitions between states (Calhoun et al., 2019).

Machine learning based approaches for behavioral tracking and analysis are in continual development along several directions that are of particular interest to the analysis of socially-interacting animals. For example, there are several methods for estimating 3D locations of body parts by triangulation of multiple simultaneous 2D views of the animal (Bala et al., 2020; Günel et al., 2019; Nath et al., 2019; Zimmermann et al., 2020), but such triangulation methods are sensitive to occlusions and thus difficult to use in interacting animals. A recent report showed, that after having collected one good ‘ground-truth’ multi-view 3D dataset, it was possible to train a network to predict the 3D posture of an animal from a single 2D view only (Gosztolai et al., 2020). Building upon work in humans, it might even be possible to learn 3D body skeletons from only 2D views, i.e., without the need to capture a ground truth 3D data from multiple cameras in the first place (Novotny et al., 2019). Such methods for estimating the 3D posture from a single 2D view could be a very powerful way to deal with camera occlusions in studies of interacting animals.

An elegant way to improve unsupervised behavioral clustering is to do everything in a single operation, whereby a deep neural net simultaneously learns to project the data onto a low-dimensional manifold and estimate an optimal number of latent clusters according to single objective function (Graving and Couzin, 2020; Luxem et al., 2020). Another promising approach is to use a dictionary-based approach to identify behavioral categories as sequence ‘motifs’ in the raw tracking data (Reddy et al., 2020).

When analyzing the behavior of single animals, some studies have eschewn body part tracking altogether and identified behavioral categories by fitting state space models directly to video data (Batty et al., 2019; Markowitz et al., 2018; Wiltschko et al., 2015, 2020) or by training a network to replicate human labeling directly from raw video (Bohnslav et al., 2020). It would be very useful if these approaches can be modified to handle multiple animals in the same video. This challenge is difficult, not just due to occlusions, but because multiple animals are interaction and thus will have complicated between-animal statistics. Writing a generative model of two animals is more challenging than utilizing two copies of a generative model of a solitary animal. In fact, to understand the structure of rodent body language, and its neural basis, these between-animal statistics are critical to document in high resolution. For example, running towards a conspecific or running away from a conspecific have a very different social “meaning”, but may be identical in the kinematic space the single animal, if that animal is modeled in isolation.

Mathematical methods for understanding the behavior of interacting animals are still in active development, with many important and open questions to work on. Recent reports have used unsupervised methods to elucidate how the behavior of interacting Drosophila depends on the relative spatial location of the interacting animals (Klibaite et al., 2017) and the animals’ behavioral state (e.g., courting or not) (Klibaite and Shaevitz, 2019). In ethology, there is related work in the use of information theory (Pilkiewicz et al., 2020), modeling (Sumpter et al., 2012) and network theory (Weiss et al., 2020) to understand the role of social interactions in determining collective movement, e.g., in fish (Rosenthal et al., 2015) and baboons (Strandburg-Peshkin et al., 2015). One recent study combined information-theory and presentation of robotic conspecifics to understand the statistics of dyadic interactions in zebrafish (Karakaya et al., 2020) and another study used purely statistical methods to discover that rats rely on social information from conspecifics when exploring a maze (Nagy et al., 2020).

Conclusions

As outlined above, some methods for automated behavioral analysis – e.g., those that discover behavioral structure directly from raw video (Batty et al., 2019; Bohnslav et al., 2020; Wiltschko et al., 2015) – do not return an explicit physical ‘body model’ of the animal; only discrete behavioral categories. For many biological questions, such an “ethogram-centric” view has no drawbacks, but when relating neural data to behavior, continuous information about movement and posture kinematics can be critical. Neural activity is modulated by motor signals (Georgopoulos et al., 1982; Kropff et al., 2015; Parker et al., 2020) and vestibular signals (Angelaki et al., 2020; Kalaska, 1988; Mimica et al., 2018) in many brain areas. To understand how neural circuits process body language cues during social interactions, these “low-level” motor and posture related confounds must be regressed out. For example, it is in principle not enough to know that activity in a brain region is different during mutual allogrooming than during boxing to conclude that a neural region is responding to a difference in social ‘meaning’ (e.g., aggressive, but not agonistic behaviors). Differences in neural activity between behavioral categories might just as well be related simply to “low-level” differences in movements and postures made by the animals in those different behavioral categories.

Regressing out confounding low-level motor and postural signals is a difficult task. Without a body model, it is possible to regress out some variance by regressing the neural activity onto variance in the raw video itself, e.g. ,regress out activity related to face movement by regressing onto principal components of a video of the face (Musall et al., 2019; Stringer et al., 2019). However, movement and posture signals are generally aligned to the animals own body in some form of egocentric frame of reference, e.g. to muscles, posture or movement trajectories (Omrani et al., 2017). The transformation from body to video is highly non-linear and difficult to discover automatically.

There are related complexities when interpreting differences in neural activity in social situations that are associated with sensory input, e.g., social events that include vocalizations and social touch. Social touch widely modulates the brain from the hypothalamus (Tang et al., 2020) to frontal and sensory cortices (Ebbesen et al., 2019). Moreover – in context of understanding the structure of body language – it is likely essential to know if a close contact between animals included a social touch or not. While behavioral tracking methods that can estimate the animal 3D posture as a “skeleton” of body points are suited for regressing out signals due to the animals’ own posture and movement, a full, deformable 3D surface model of the animal is required to measure social body touch. To this end, there are also promising machine learning methods on the horizon. For example, starting from a detailed, deformable 3D model of the animal’s shape and color, it is possible to extract a detailed 3D model of an animal’s body surface from a single 2D view, even in complex images (Badger et al., 2020; Biggs et al., 2018; Kearney et al., 2020; Zuffi et al., 2017, 2018, 2019).

Rodents display a wide range of facial expressions (grimaces and whisker movements), including during social interactions. Multiple observations suggest that body postures and movements of conspecifics function as an important social signal. Recent major advances machine-learning methods for behavioral analysis and microengineering of behavioral sensors are making it possible to quantify facial expressions and body postures during complex, social interactions. These data will reveal new questions about the neural basis of social cognition in rodents to understand the comparative neurobiology of body language.