Automated Pose Estimation in Primates

Abstract

Understanding the behavior of primates is important for primatology, for psychology, and for biology more broadly. It is also important for biomedicine, where primates are an important model organism, and whose behavior is often an important variable of interest. Our ability to rigorously quantify behavior has, however, long been limited. On one hand, we can rigorously quantify low-information measures like preference, looking time, and reaction time; on the other, we can use more gestalt measures like behavioral categories tracked via ethogram, but at high cost and with high variability. Recent technological advances have led to a major revolution in behavioral measurement that offers affordable and scalable rigor. Specifically, digital video cameras and automated pose tracking software can provide measures of full body position (i.e., pose) of primates over time (i.e., behavior) with high spatial and temporal resolution. Pose-tracking technology in turn can be used to infer behavioral states, such as eating, sleeping, and mating. We call this technological approach behavioral imaging. In this review, we situate the behavioral imaging revolution in the history of the study of behavior, argue for investment in and development of analytical and research techniques that can profit from the advent of the era of big behavior, and propose that primate centers and zoos will take on a more central role in relevant fields of research than they have in the past.

Graphical Abstract

Introduction

The Minnesota Zoo in Apple Valley, MN, has the largest public collection of Japanese macaques (Macaca fuscata) in the United States (Figure 1). Every morning, the zoo’s twenty-seven macaques emerge from their dormitory and enter a large, beautifully architectured open enclosure. There are many ways to describe what they do next, but one way to say it is that they proceed to generate an enormous amount of data. That is, each of the monkeys moves each of its limbs in a specific way, moves its body position along a particular and often complex path, and interacts with multiple items in the pen. They also interact with each other in complex ways, play, explore, foraging, relax, eat, and so on.

Figure 1.

Camera tracking system at the Minnesota Zoo. Relatively affordable cameras, such as GoPro cameras, when combined with computer vision tracking software, can provide estimates of pose, although they depend on having long-term access and a safe vantage point.

The ability to track and analyze the actions of primates like the ones at the Minnesota Zoo has potential relevance to researchers in biology and biomedicine, as well as for neuroscience, psychology, comparative biology, and research into animal welfare (e.g., Rudebeck et al., 2019; Buffalo et al., 2019; Bliss-Moreau and Rudebeck, 2020; Santos and Rosati, 2015; Periera et al., 2020). And yet, nearly all the rich data these monkeys generate largely slips past without being registered.

Instead, characterization of complex primate behavior is limited to what humans can annotate, laboriously, in hand-crafted ethograms. Such measures can only capture information when observers are present – that is, they do not work for behavior that occurs after staff leave for the day, in the evening, or on vacations, or even when the annotators’ attention is momentarily diverted (Brando and Buchanan-Smith, 2018). It is also possible to characterize simple behaviors more rigorously using computerized systems. For example, we can quantify monkeys’ interactions with specialized response systems, such as levers and buttons. Such systems produce data that are several orders of magnitude lower in information rate than the animals’ full behavioral repertoire. For example, a measure of gaze direction or preference, typical in many psychological studies, gives one bit of information (left vs. right) per trial. Those data can, of course, be used to test important hypotheses. But they come with a high opportunity cost: they ignore much more data than they capture.

In contrast, behavioral tracking can produce high quantities of data (for example, thirteen body landmarks in three dimensions sampled at 30 Hz, Bala et al., 2020) without human intervention or even oversight. Indeed, without tracking, our ability to collect behaviors with high technical precision and reproducibility is so limited that it might be described as looking at the world through a drinking straw.

This is all changing, and at a rapid pace. A recent series of technological advances has made it possible to collect a good deal of the data these monkeys produce. These technologies are generalizable, meaning that, when appropriately trained, they can provide tracking for any species of primate, including arboreal ones. And not just these monkeys - other species and other locations can also be tracked (Dankert et al., 2009; Mathis et al., 2020; Joska et al., 2021; Walter and Couzin, 2021; Bala et al., 2020; Marshall et al., 2021). We call this technological approach behavioral imaging. We use this term deliberately by comparison to other forms of imaging, which can give detailed information available in the visual domain through use of advanced technology.

The subject of study in this approach has been called big behavior (von Ziegler et al., 2020). By our lights, little behavior refers to measures of behavior that involve highly reduced, low-information tracking of what the animal is doing. Little behavior would include reaction times, preferences expressed through choice, gaze directions, pupil size, and reaches - measures that give a small amount of information per unit time. Little behavior would also include ethograms made by trained annotators – not because they measure small amounts of information but because that information is filtered through the narrow aperture of human annotation abilities.

Big behavior, by contrast, includes full information about the position of all major joints in the entire body and involves continuous movement over an extended period of time. The extent of that time can be very large. Scientists can now use multiple high-resolution cameras to continuously capture every fine-grained behavior over several weeks, months, and years. They can track pose of both individuals and groups. As with any big data situation, the big behavior revolution raises unprecedented challenges, especially in managing, analyzing, and understanding the data in a fully automatic fashion. However, the benefits it offers are so great that many teams are working to solve them.

The present review will describe how we got to this point, talk about the state of the art, make a few predictions about the near future, and sketch out some of the potential benefits. We will discuss some of the specific scientific problems that big behavior for primates is likely to affect. Likewise, we will argue that primate centers and zoos will take on much larger roles in research than they have in the past.

The challenges of big behavior in primates

Behavioral tracking in primates has recently become possible through parallel developments in computer vision, machine learning, and robotics. The story starts at the beginning of the last decade, when technical breakthroughs in deep learning enabled software to recognize objects in an image by use of convolutional neural networks (CNNs, Krizhevsky et al, 2012). Subsequent works improved on the design of CNNs to enable automatic tracking of humans both live and from videos (Cao et al., 2019 Newell et al., 2016; Wei et al., 2016; Fang et al., 2017). That work in turn inspired and facilitated work allowed for tracking of animals such as flies, mice, and horses from videos (Mathis et al., 2018; Pereira et al., 2019; Mathis et al, 2020; Marks et al., 2021). The tools that allowed these animals to be tracked leveraged the copious capacity of CNNs to learn the visual and geometric relationship between landmark locations (body joints). They also relied on relatively cheap and robust digital cameras and standard computer vision techniques.

Relative to other animals, non-human primates have been much more difficult to track (Bala et al., 2020; Negrete et al., 2021; Labuguen et al., 2020; Testard et al., 2021). There are three reasons for this. First, their body joints are highly flexible and thus generate a large number of distinctive body postures. Each joint has multiple degrees of freedom and these complex joint positions are essential for tracking and identifying activities such as bipedal/quadrupedal locomotion, hanging, and dexterous object manipulation. This fact distinguishes primates from non-primate animals such as rodents and insects - the poses of these animals can be characterized by a small basis set. Second, primates’ bodies are covered by a thick fur, which is a characteristic failure case of computer vision-based tracking algorithms. For example, the pelvis location is highly ambiguous while performing a sedentary activity because there are no visually or geometrically salient features to identify the pelvis joint. Third, primates’ body movements are fundamentally three dimensional. The range of motion of bodies spans full three-dimensional space, and more importantly, the motion often involves 3D interactions with objects, peers, and environments. This dimensionality raises two major problems – (1) considerable occlusion and (2) broad variation in appearances.

We have proposed that, given these difficulties, non-human primate tracking cannot be achieved using existing tracking paradigms (Bala et al., 2020). In particular, off the shelf solutions like DeepLabCut need orders of magnitude more data than they do using other species, and so bespoke solutions currently work best. We need innovations in both hardware and software that are tailored to reflect the primate characteristics. For the readers’ benefit, we next describe several of the needed innovations, and several of the major factors that must be considered and the obstacles that must be overcome.

(1). 2D vs. 3D representation.

Primates move in three dimensions. Although other animals also move in three dimensions, primates tend to make use of all three quite a bit more than other popular model organisms. A rat’s major axes of movement are on a plane along the ground and a worm’s behavior is well studied in a petri dish. By contrast, a monkey moves vertically, but, more importantly, its limbs move with much more freedom relative to its central body axis.

In most computer vision solutions, tracking occurs on the XY plane, without regard to depth (i.e., the Z plane); doing so is much easier to implement and costs little with plane-moving species. Continuing with the rodent example, a typical rodent tracking system uses an overhead camera and tracks XY locations of landmarks. This 2D representation, however, shows limited expressibility to describe the behaviors of primates due to their 3D body movement over 3D scenes. The 2D representation is a camera projection of 3D behaviors where its location drastically varies as the viewpoint changes. A 3D representation (XYZ) is a viable solution that requires a specialized system such as a depth camera or a multiview camera system. For instance, a system of multiview cameras is used to develop OpenMonkeyStudio that enables tracking 3D body movements of macaques (Bala et al., 2020, Figure 2). The resulting 3D representation is invariant to the viewpoint change, which allows modeling coherent behavioral clusters compared with the 2D representation as shown in Figure 3.

Figure 2.

Automated pose-tracking software, such as OpenMonkeyStudio (Bala et al., 2020), can provide high quality tracking of poses in primates. The OMS system is based on multiview capture, which can bypass problems associated with occlusion.

Figure 3.

With OpenMonkeyStudio, we compare 2D and 3D representations in macaques (Bala et al., 2020). We compared our systems’ ability to recognize semantic actions (standing, walking, climbing, climbing supine, sitting, and jumping). A. The poses are clustered by using UMAP. Each cluster is correlated with specific actions. B. With the clusters, we recognize actions using the k nearest neighbor search and visualize the transitions between actions. C. In contrast, the 2D representation provides the clusters that are driven by the pose and viewpoint.

(2). Target-specific model vs. generalizable model.

The ability of a CNN model to track primates is limited by our ability to train the model. Our ability to train the model is limited by the size and quality of the training set. Thus, the success of tracking algorithms is predicated on the existence of a large, accurate, and well-curated datasets. For instance, large scale annotated datasets such as COCO and MPII have been used to train CNNs to reliably detect human poses (Lin et al., 2014; Andriluka et al., 2014). However, such large datasets do not exist for most animal species.

This data challenge has been addressed by training a target-specific model facilitated by annotation tools. For example, user interfaces provided by DeepLabCut and LEAP have enabled effective pose annotations by use of a very small number of annotations (as few as 10–1000) in a video (Mathis et al., 2018; Pereira et al., 2019). These methods, which make use of transfer learning or its variants, work for relatively simple organisms like flies, worms, and mice (e.g., Mathis et al., 2018; Nath et al., 2019).

Despite its remarkable performance, this approach is not suited to track the behaviors of non-human primates because visual appearance significantly varies as a function of viewpoint, pose, and species (Bala et al., 2020). As a result, a CNN trained using a dataset from one viewpoint does not generalize to that from another viewpoint. Instead of learning such target-specific models, we have proposed a new paradigm that aims to learn a generalizable model by collecting a large, curated dataset. This curated dataset must include diverse images across species, background, illuminations, poses, viewpoints, and interactions, where the size of data is comparable to the human data size (order of millions). The MacaquePose dataset that is made of 17,000 annotated internet images (Labugen et al, 2021). It is an inspiration for the OpenMonkeyChallenge project (Yao et al., 2021). It includes a large variety of appearances, poses, and viewpoints of macaques in the wild, which can be combined with state-of-the-art pose detection models such as DeepLabCut and LEAP. The performance was unprecedented: no existing macaque detection model was able to achieve that level of accuracy and generalization, due to the scale of the data. Nonetheless, it still lags behind the accuracy of human pose estimation. OpenMonkeyChallenge is designed to address this limitation by creating a 6 fold-bigger-dataset across multiple species to enable learning a shared representation between them (Yao et al., 2021).

Once the large dataset is collected, a generalizable CNN can be trained to detect primate poses from out-of-sample videos regardless of camera configuration, background, and species. Such networks can also be trained to use models of the animals to draw inferences about positions of landmarks that are occluded or that are not in the original training set (Bala et al., 2021). For example, we have collected more than 100,000 images with 17 landmark annotations portraying primates in natural habitats. The images can be obtained from the internet, national primate research centers, and from zoos, and include 26 species (20 monkeys and 6 apes) as shown in Figure 4. This is, by far, the largest collection for non-human primates. With this dataset, we recently opened a new benchmark challenge called OpenMonkeyChallege (http://openmonkeychallenge.com/) that facilitates an annual competition to develop a generalizable pose detection model. We believe that this dataset, and others like it, will be crucial for developing widely usable models for tracking non-human primates.

Figure 4.

Images from our new dataset, OpenMonkeyChallenge, illustrating the range of species, individuals, and backgrounds.

(3). Multi-view camera system:

Most existing animal tracking approaches are designed to draw pose inferences from an image stream from a single camera. Due to the 3D nature of primate behaviors, multiview camera systems provide several benefits. These include:

• Disocclusion: in tracking, occlusion is the main source of pose detection error. In particular, when a primate interacts with objects and other primates, those other things will cover some of the relevant landmarks. For example, occlusion is often an unavoidable problem with arboreal primates, such as gibbons, because the trees in which they spend time naturally cover their landmarks. Indeed, one side of the animal’s body will almost always occlude the other, just as the limbs will often occlude the more central parts of the body. Some of this occlusion can be mitigated by a multiview camera system; even when the cameras do not offer views of the occluded parts, the parallax views they provide can greatly facilitate inferences about the true positions of the relevant landmarks. The benefits of disocclusion are particularly valuable when considering social interactions, which are often naturally occluded by the agents involved in the interaction.

• Robust estimation: Multiple simultaneous observations reinforce robust estimation. The multiview images of a common primate are visually and geometrically distinctive, and therefore, aggregating the detections from these multiview images allows more accurate and robust estimation in the presence of measurement noise.

• Data augmentation: Multiview images provide trivial data augmentation. Annotations from two viewpoints can be reconstructed in 3D, and in turn, projected onto other images, which forms additional annotations without manual effort. This leads to significant improvement on annotation efficiency (Marshall et al., 2021; Bala et al., 2020). For instance, in OpenMonkeyStudio, two to three images are manually annotated and nearly 50–60 multiview images are automatically annotated using the multiview geometry. This results in a generalizable model trained by a large, annotated dataset (~200K images).

• 3D representation: Multiview geometry offers a 3D representation. This is different from the 3D lifting approach that reconstructs 3D pose from a single view image of which reconstruction is defined up to scale and orientation (Günel et al., 2019). With multiview images, full metric scale reconstruction can be obtained.

(4). Identifying individuals:

Primates are social animals. Indeed, their social nature makes them an appealing model organism for humans, including both healthy social development and diseases associated with impaired social function, such as autism and Alzheimer’s Disease. The fact that they are social means that any technology deployed to understand their behavior must work with groups. Indeed, both zoos and primate centers often have groups as a standard organizational unit. Standard pose tracking software can typically be extended to include tracking multiple individuals (see above and Bala et al., 2020). However, these systems need to have advanced features to ensure individuation. That is, systems may obtain good performance on pose estimation through interpolation and deduction, but these tricks can lead to confusion over identity. Identity confusion can cause critical problems with experiments when the goal is to look at changes in individuals longitudinally. Consider, for example, a study to monitor the success of a physical therapy regime for an individual monkey who has a limp. If the tracking system confused him with a conspecific, it will appear that the limp has healed spontaneously, when in reality, it is just tracking a different individual. Even if there is a small probability of misidentification, this probability may be effectively compounded over time – possibly to near certainty - in long-lasting studies.

Individuation is a deceptively tricky problem and is one that is poorly suited for pose tracking systems. The reason is that, without secondary checks, the method is susceptible to catastrophic failures – even rare loss of coverage can lead to confusion and major problems with data. Thus, individuation often can benefit from complementary approaches, such as identification of individuals by face, body shape, movement type, or other distinguishing features. For example, one possible solution is to use technology such as radio frequency identification (RFID) tags to define individuals. That in turn would require placing RFID tags onto individuals’ bodies, which may be impractical. Finally, a third approach involves humans checking the results of the tracking system to ensure that individuation is maintained. This approach is costly but may still be orders of magnitude more achievable than full pose tracking by humans. There is some work on the use of trackers in primates (for example, Gelardi et al., 2020; Gazes et al., 2019).

(5). General technical issues:

In addition to these problems there are several basic technical problems that need to be overcome and whose difficulty should not be understated. We list a few of them here. Anyone interested in tracking primates must have enough cameras to cover the area of interest. Typically, tracking is considerably more useful if the user can achieve 3-D tracking, which in turn means multiple (often four or more) cameras. In either case, if the recording is longitudinal, these installations will need to be weatherproofed and will need to be protected from the prying hands of the primates themselves. This level of coverage may be daunting at large zoo exhibits and even moreso in field cages at primate centers.

These cameras will need to be synchronized to the precision of a single frame and must maintain synchronization across the period of recording. All cameras must be calibrated, both intrinsically and extrinsically, and this calibration must take place often (typically at least once a day). The cameras will typically generate large volumes of data (often over 1 TB per hour), and this data stream must be managed and stored. That information must be transported from the recording site to the location where the servers are stored. Often it is impractical to build them near the site where the primates live, so long high-quality fiber-optic wires, possibly requiring laying new wiring, may be unavoidable. The needs for storage can be reduced if some computations can occur online or at the site of recording, but this in turn raises other technical challenges. Ultimately, these things can be costly – a simple webcam will likely not offer the full benefits of tracking. Moreover, the installation must be coordinated with the stakeholders of the site, and if it is a zoo or primate center, they may have other needs that supersede those of the scientists who want to do the tracking.

Automated behavioral state identification (“ethogramming”)

Automated behavioral sampling and identification (briefly, “automated ethogramming”) procedures offer the promise of using a computer to perform the process of identification of behavior, rapidly, cheaply, and on a massive scale (Marks et al., 2021; Voloh et al., 2021). Because it is much cheaper and faster, automated behavioral identification promises many things that are uneconomic with human annotators, such as massive throughput testing, or monitoring behavior twenty-four hours per day, or testing unlikely hypotheses that would be too costly to test with standard approaches.

Since it is a new technology, it is difficult to predict how behavioral imaging will be best used. Here, we will make some educated guesses. However, like many others, we believe that because of its ability to tell us about behavior, the promise is high (Datta et al., 2019; Krakauer et al., 2017; Calhoun and El Hady, 2021; Calhoun and Hayden, 2015). Specifically, we envision that longitudinal and quantifiable analysis from years long recordings of primates will enable us to discover the reproducible behavioral patterns and characteristics, which has been impossible from a small-scale data analysis. Pose (see above) can tell us where the landmarks and major joints are positioned in space, but the key dependent variable for many studies is what behavior the animal is performing (Anderson and Perona, 2014). In other words, the ideal output of behavioral imaging is the category of behavior that describes the animal at every moment in time. Such a time series would be the starting point for a great deal of research.

For illustrative purposes, we consider a couple examples. Suppose we have developed a potential animal model of clinical depression and wish to validate it (for example, see work by Shively and Willard, 2012). It could be validated by comparing the amount of time the animal is immobile, sleeping, or stays in its dormitory – behaviors that would have clear face validity to symptoms of human depression. As noted above, these behaviors can, in principle, be scored by a human observer, but only in small quantities because of the cost. Another example would be anxiety - behavioral identification could be used to determine whether the animal shows behaviors with face validity to human anxiety disorders such as excessive grooming, fastidiousness, unusual behaviors around doorways – behaviors with face validity to human anxiety. Moreover, the ability to monitor for both types of behavior efficiently means that we could then engage in unsupervised classification of possible depression analogues by the presence or absence of specific symptoms and/or by their cross-symptomology with anxiety homologues. This clustering could then differentiate different subtypes of depression correlates.

Another example would be in the form of precursors to drug abuse. Monkeys are an important model organism for drug abuse for several reasons, including their long lifespan, their closeness to human body size (compared to rodents), and the similar architecture of their prefrontal cortex. Within the world of addiction, one important question is how strongly one can predict susceptibility to drug abuse based on past behaviors. This kind of search is essentially a needle in a haystack type of search – we must have access to large amounts of highly quantified (and ideally unbiased) measures of behavior that we can analyze and correlate with subsequent behavior following drug exposure. Behavioral imaging with state identification provides an ideal input to such studies.

More generally, behavioral states could be used as a dependent variable in studies of how some intervention affects the way the animal distributes its behavioral state over time. Or they could be used as an independent variable, such as in studies of how an animal’s behavior affects that of a conspecific. Identifying behavior based on pose is a non-trivial problem and remains an area of active research in many species. Most existing work has been done for animals like flies, worms, and mice. In primates, automated behavioral identification is relatively unexplored. However, the general principles developed for other animals ought to generalize relatively straightforwardly. In general, such approaches take the raw pose data, subject it to some preprocessing (such as centering the animal and rotating it to a canonical orientation), and then performing some unsupervised procedure, such as t-SNE or UMAP, to identify behavioral states.

Automated behavioral state identification an alternative to hand-scored behavioral sampling procedures that produce ethograms. Automating the process addresses the key limitations in this behavioral sampling approach by providing a computational solution that is cost-effective, consistent, fast, accurate, and fine-grained (Anderson and Perona, 2014).

1. Cost-effectiveness: the human-scored ethogram approach involves highly trained human annotators who watch live animals or videos and mark down the behaviors they observe based on a predetermined rubric. These humans are skilled in identifying behaviors; indeed, the need to carefully train these observers is a major cost of the ethogram approach and one that makes it very expensive to implement. Being able to do that takes education and training and does not scale with large datasets. Big behavior automates behavioral annotations, which reduces the cost of training, management, and labor.

2. Consistency and speed: different human raters tend to have different subjective criteria, meaning the studies are often not reproducible with the same data if the raters are no longer available (Anderson and Perona, 2014; Levitis et al., 2009). Even well-trained individual human observers can often be somewhat inconsistent. This is especially likely to be a problem with novel and unusual primate behaviors, or ones that are poorly characterized.

3. Accuracy: human raters are intrinsically fallible - they get bored, especially after many hours of video footage, they use criteria that change subtly over time, they have blind spots, systematic patterns of behavior that they are less likely to detect. Consider for example how often even professional sports referees make mistakes, or even disagree, and that in sports, categories like fair and foul are rigorously defined.

4. Bias: human raters tend to exhibit systematic biases (Kardish et al., 2014). Indeed, careful experimental work has shown that human observers interpret behaviors through their own expectations (Tuyttens et al., 2014). More broadly, a great deal of behavioral identification is subjective, meaning it may be difficult to evaluate bias so that it can be reduced or avoided; even ostensibly objective measures ought to be blinded, and seldom are (Holman et al., 2015).

5. Longitudinal perspective: Humans are especially poor at detecting rare behaviors, and at detecting the kinds of behaviors that involve gradual change over long periods of time (Biggs et al., 2014; Kryszczuk and Boyce, 2002; Wolfe et al., 2005). Automated ethogramming offers the opportunity to take the long view, to accumulate data over long periods, and draw inferences about rare and slowly changing behaviors. We anticipate this benefit may be especially important for developmental trajectories, such as skill-learning, aging, and adolescence.

6. Annotation resolution: Humans have a sensitivity limit. We can only detect a certain number of behaviors and can only detect a certain granularity of behavior. For example, it may take some effort to be able to identify the different gaits of a running horse, especially subtly different ones. Moreover, it’s possible that there are some patterns in the data that are real and measurable, but humans simply are not adapted to notice them without help. For example, some behaviors may reflect certain patterns of simultaneous activity across multiple limbs and may only be detectable and classifiable after certain dimensionality reduction processes that humans lack the cognitive capacity to perform.

What behavioral imaging of primates offers biology and biomedicine

Behavioral imaging of primates can be of great benefit to all manner of scientists – from those interested in the biology of primates to those who are interested in using primates as model organisms for disease. It is important to note that these categories are not mutually exclusive, nor does biomedical research necessarily make use of invasive procedures on unhealthy animals. Indeed, there is a great deal of important biomedical research that can and should be done healthy animals; this work would seek to understand the biology of these animals without any use of disease models or simulacra of disease processes. Thus, the study of healthy animals poses an opportunity for biomedicine as well as for biology.

Improvements in diagnosis:

Diagnosis typically requires specialized expertise in humans; in animals, who cannot talk and often disguise their symptoms, it’s more difficult. Many conditions can only be diagnosed following conspicuous presentation of behavioral symptoms. Many diagnoses require advanced veterinary intuition. In many cases, animals deliberately camouflage their symptoms. For these reasons, diagnosis is likely to be a major focus for behavioral imaging. This is not to say that computers will monitor all behavior and replace veterinary expertise. Instead, we imagine that behavioral imaging will provide a complementary measure that will boost trained medical opinion.

First, it will provide much more information, over a longer period of time. Second, it will offer the ability to identify subtle and hidden behavioral patterns, that may be ignored or missed. For instance, a monkey may slightly drop her shoulder while walking when her arm is wounded. Such differences represent symptoms, which have not been coded and listed as a behavioral marker by the experts. Third, it will offer acute measures, with greater quantity, accuracy, and sensitivity than human observers can provide. Finally, it will provide a ready and quantifiable control group, including individuals in the same facility, to say whether the individual in question differs from their peers.

Finer-grained diagnoses:

Many diseases are clusters of distinct conditions with somewhat different symptom profiles and different best treatments. To give a well-known example, there are many effective psychopharmaceutical treatments for clinical depression, and different patients respond differently to different ones. Trying each one takes several months, so if diagnosis could be sped up, a great deal of suffering could be eliminated. It is hoped that finer grained diagnosis, based on behavioral presentation, would provide an indicator of which treatment should be tried first. The ability to characterize behavior with high bandwidth promises greater information that can be clustered and specific disease subtypes identified. Even modest improvements in classification of diseases into different subtypes, which in turn can motivate specialized study of treatment.

To give one well-known example, deep brain stimulation (DBS) treatment for depression in humans is sometimes focused on the subgenual anterior cingulate sulcus (Mayberg et al., 2005; Lozano et al., 2008). This is a brain region whose responses are known to be associated with depression, with negative outcome monitoring, and with depression severity (Alexander et al., 2019; Roberts and Clarke, 2019; Drevets et al., 2008; Azab and Hayden, 2018; Maisson et al., 2021). However, despite promising early treatment results, follow-up studies have been equivocal. It has been proposed that the core issue is that we are treating two different diseases with different anatomical bases (Drysdale et al., 2017; McGrath et al., 2013). Improved diagnosis in humans would help us test these hypotheses faster, and more valid animal models (see above) would help even more.

Validating animal models of disease:

Primate models of disease are crucial to modern biomedical research. In many cases, models are imperfect or are only relevant to a subset of symptoms. In other cases, the validity of primate models is unknown. For example, the limitations of models are especially well delineated in the case of psychiatric illness, which often relies on personal reports. Consider that many of the major criteria for depression and obsessive-compulsive disorder rely on subjective descriptions of feelings (Goodman et al., 1989; Beck et al., 1988). Ascertaining the validity of an animal disease model can be a surprisingly ad hoc procedure, based on superficial assessment of major symptoms (Geyer and Markou, 1995; Koob and Zimmer, 2012). In many cases, establishing a valid animal model for some disease or symptom class takes many years - even decades - to result in a consensus across the field. Despite these problems, some diseases, or some symptom classes may be characterized by a specific set of behaviors and behavioral patterns; it is possible that the particular constellation of behaviors can serve as an ethological fingerprint of the disease or symptom. While this possibility is speculative, it cannot readily be tested until we have behavioral tracking. Optimistically, these fingerprints can be compared across species to test specific hypotheses about the behavioral validity of specific models.

Improvements in treatment:

Many diseases have well defined treatments that require parameter-setting (Johnson et al., 2013). For example, Parkinson’s Disease is well treated by DBS although the parameter space is very large. The optimization of parameters takes place through what is more or less a gradient descent procedure, that is, through trial and error (Schiff, 2010). The slow part is the assessment of state in the patient. Primate models can improve that process, but they still have a slow assessment stage - often even slower in primates because they lack the verbal modality. Big behavior offers a solution to this problem - it is both high bandwidth and computerized, so it can be made very fast, and potentially improve treatment time by orders of magnitude. It is possible - albeit speculative at this point - that behavioral imaging can lead to rapid closed loop parameters for treatment. We can test specific parameters and read out the effects of those parameters - whether they are stimulation parameters in DBS, anatomical positioning of stimulation, different drug doses, or any other of a large number of possibilities.

Replacement for other dependent variables:

To be a bit more speculative, behavioral imaging could potentially offer new dependent variables that could reduce the need for invasive measures, leading to benefits in welfare, and increases in the efficiency of research. A good deal of research may require some measure of efficacy of some intervention, but not depend on any particular measure. For example, consider a team of researchers who want to rapidly screen several dozen candidates for headache medicine, most of which will be inert. In that case, behavioral imaging may provide information sufficient to move on to the second stage. For example, animals without headaches may be more active overall, may engage in more social behaviors, may spend more time in well-lit areas, and so on. And it may do so at a price that is much cheaper, and with less intervention, than other measures (such as measures of internal physiological functioning).

Or, to give an example drawn from the psychological literature, scientists are sometimes interested in knowing the way that an animal will respond to seeing a reflection of itself in the mirror (Gallup et al., 2002). For example, a researcher may want to know whether an animal differentiates seeing itself in the mirror from seeing another conspecific. This work may in turn be biased by observation or may be slow and may benefit from running dozens of animals instead of a few. Automated tracking could make high-throughput mirror testing feasible. It could also make it possible to compare many species in a larger clade, rather than forcing psychologists to pick one or two exemplar species.

Indeed, in such cases, big behavior offers the possibility of performing such experiments in animals that are inaccessible to other measures, including rare and endangered animals, and highly intelligent animals, such as apes. Even for standard laboratory primates, behavioral imaging may be an order of magnitude cheaper than other measures (see above) and may therefore be a preferred alternative. For example, it may require less specialized equipment, trained technicians, or built laboratory environments. Finally, even if it is not cheaper, it may have fewer welfare costs than invasive measures, and may be preferred for that reason.

Natural behaviors:

Finally, behavioral imaging allows scholars to monitor naturalistic - or at least relatively naturalistic behaviors. When primates perform tasks that resemble those for which they have evolved, their behavior is more likely to reflect their normal response biases, and thus to be more ethologically valid, and likely more interpretable (Pearson et al., 2014; Hayden, 2018). For example, we have argued that both risk and impulsivity measures in primates are biased by standard laboratory tasks (Hayden, 2016; Blanchard et al., 2014; Eisenreich et al., 2019). While we may currently lack even strong hypotheses about the types of scientific questions that can best be asked by studying natural behavior at scale, the best ideas often come after the data are collected and while they are being analyzed. And analysis of new data or old data are better scales is often the best way to drive the development of novel theories, hypotheses, and tests. For these reasons we believe that automated recording of natural behavior is likely, by itself, to produce important insights.

What behavioral imaging offers zoos and primate centers

Much of the behavioral imaging work to be done can occur at specialized sites such as zoos and primate centers. These places have their own agendas and their own group of engaged stakeholders. While research is often a major goal, it is not the only one, and even when research is a priority, there are many researchers with competing demands. It is worth exploring, therefore, how the benefits that behavioral imaging offers can also benefit zoos and primate centers.

Indeed, many of these benefits are overlapping. For example, zoos and primate centers have a great interest in keeping their animals healthy, and that means that any improvements in diagnosis and detection of latent disease states can help them in their own mission. Consider, for example, the benefits offered by an always-on behavioral tracking system that identifies pose for all primates in an enclosure, and then quantifies behavioral state. This system could perform monitoring at night and holidays when staff are absent and send warnings to on-call veterinary staff. It could also serve as a tracking history for an animal that needs diagnosis and ask how long a particular behavior has been evident, how long it has taken to progress whether this animal is showing improvement, and so on. It could free up time for the veterinary staff to focus on other issues, such as enrichment, that would directly improve the lives of the animals on site, thus leading to secondary welfare benefit.

Another example would come from enrichment interventions. Zoo and primate center staff typically have at hand multiple possible enrichment opportunities, but very little direct data from their own subjects about which enrichment opportunities have the greatest desired effect. And the desired effect may be species dependent, and it may be individual dependent, and may even be context dependent. When an animal is anxious, the best enrichment may be different than when the animal is recovering from a broken leg. The ability to measure behavior and automatically evaluated the efficacy of specific enrichment interventions is likely to lead to fast advances in welfare.

It is important to note that these benefits go hand in hand with the research benefits –the tracking that benefits welfare and health can also produce data that is of interest to scientists, thus making scientific exploration more readily available to scholars interested in it.

Finally, it is worth noting that there are potential conservation benefits to behavioral imaging. Many insights obtained from artificial environments can be used to gain insights into how animals behave in their natural environments and develop insights that can be used to protect them in those environments. Second, they can be used to test hypotheses about how best to reintegrate animals into natural environments, and speed and improve that process. And third, by improving welfare in unnatural environments, imaging can help keep animals healthy and productive, so they (or their progeny) are ready to return to the environment at a later time.

Limitations of behavioral imaging

The present review is meant to highlight the virtues of behavioral imaging. Of course, it has many limitations as well. First, it is non-trivial to implement pose tracking in any environment, even in a laboratory. It is undoubtedly more difficult in environments tailored to other goals. Successful installation of a working system may require multiple computer scientists and electrical engineers. This level of complexity is probably unavoidable right now, but may be made simpler in the future as tracking systems – both software and hardware - improve.

Second, there is a basic limit on the amount of information that can be gained solely from behavioral imaging. A good deal of what goes on inside an animal is not visible to the outside world. For example, an animal hiding in the corner may be afraid, may be depressed, may be tired, or may just be actively observing the environment from a safe vantage point. It is very possible even in theory that there is no way given the behavior alone to draw a robust inference about the animal’s true state. Indeed, animals may expend effort to hide or conceal their status, such as if they are injured. That is, there may be some information – perhaps a great deal – of information that is not detectable by behavior alone.

Third, even among behavior that is expressed, some of it requires high quality tracking that may be impractical in zoo environments. To give a few examples, there may be important information available about the animal’s internal state that is visible in facial expressions, fine hand movements, and gaze directions. Indeed, eyes can give information about the locus of attention, about intentions, about priorities, and about beliefs; these things can in turn provide information about the animal’s internal state. Even the pupil size of the animal can give a great deal of information about the animal’s beliefs, expectations, surprise, reward function, attention, arousal, and urges. This is information that is available in behavior but extremely difficult to extract in practice.

Finally, it is worth emphasizing again that zoos and primate centers have many competing goals, and that benefiting scientists is not the only one. As such, behavioral imaging must satisfy the constraints imposed by the competing goals of the recording site. These can include unobtrusiveness, limits to number of cameras, limits to power, and storage. It can also involve political calculations, which may involve coordination among stakeholders.

New roles for primate centers and zoos in the era of big behavior

A great deal of primate research takes place in the laboratory. In contrast, zoos historically been used for research less than they could. And while primate centers typically have a strong research role, this role conflicts with their other roles, including as a supplier of primates, and with their own research agenda, which may conflict with that of any given scientist. We believe that both zoos and primate centers resources are poised to become more useful in the era of big behavior.

The underutilization of primate centers and zoos stems from several specific factors that, on balance, give the laboratory distinct advantages. These include the fact that laboratories offer freedom to perform invasive experiments, give access to research animals full time, and the ability to use animals solely for research. Laboratories also offer the ability to focus on any of a small number of well-studied research organisms, with the benefits that standardization of research animals brings. While primate centers have some of these benefits, the primates located there are often needed for breeding, and competition for access to them can be strong. We believe that behavioral imaging mitigates some of the limitations of zoos and primate centers, and will make them more attractive locations to perform research.

Zoos and primate centers are designed for observing natural behavior:

Zoos are designed with public audiences in mind and are architectured to maximize unobtrusive viewing from many vantage points (Figure 5). Indeed, they are laid out to have viewing in a way that does not bother animals and is also safe from their interference. This is ideal for big behavior. It allows measuring natural behaviors including social interactions with minimal interference (see above). While primate centers are often not designed for the public, they typically have observational capacity for another reason - to allow staff to rapidly scan the housing for checking on animals and ascertaining if there are any problems. This is an advantage over many research laboratory environments, which are often constructed generically and without observation in mind. Many laboratories have tight space restrictions, which can result in imaging that is difficult or distorted. This is also a major advantage of zoos and primate centers relative to field research sites, which pose many practical problems for observation. For example, field sites are often remote and far from power and repair facilities, and studies can last multiple weeks.

Figure 5.

The orangutan enclosure at the Toronto Zoo allows for panoramic viewing of the research subjects, who have a large and enriched space to move in. Such enclosures are particularly useful for behavioral imaging.

Zoos and primate centers have many individuals:

While laboratory studies of primates typically focus on few individuals (in many cases, two per study), zoos and primate centers have access to many more. Even in zoos, in which numbers may be limited for a given species, this number is augmented by the national network of exchange of animals between zoos. Of course, laboratories can compensate for low numbers with a high degree of training and instrumentation per subject. But with tracking, these needs can be reduced. The larger number of individuals is important because it can reduce Type II errors, can improve generalizability, and can facilitate individual difference studies. Moreover, the relative ease of collecting behavioral imaging data means that data from multiple zoos can in principle be combined, thereby further increasing numbers. In contrast, laboratory research is supported solely by funding for the research itself, so the cost of adding additional animals to a study is borne by the research funds.

Zoos (and sometimes primate centers) have many species:

Individual laboratories specialize in a handful of model organisms and focus on deep search into the biology of those individuals. This lack of species diversity is a limit for biomedical research laboratories, as it reduces generalizability. In contrast, zoos typically have a large number of species. This breadth of species has several benefits. Perhaps the greatest is for comparative studies. Indeed, the lack of variety of species has been identified as a limitation of major biological and biomedical research in the modern era, and a limit on the generalizability of the resulting science. The variety is limited for practical reasons by the complexity of having to learn the intricacies of multiple animal species. Zoos already bear this cost. We can only imagine the benefits to research of having access to a large number of species, each of which can be selected for a particular study tailored to the specific needs of that study (cf. Stevens et al., 2005).

Zoos and primate centers have excellent records about their animals:

Zoos and primate centers have access to their animals for long periods of time, which makes it possible to answer important questions about their ancestry, behavioral history, their DNA, endocrine profile, and so on. This is an advantage relative to field studies, where animal subjects are often catch-as-catch-can. Even in cases where the same individuals are followed for many years, there is an enormous cost to developing this knowledge, and that cost makes it harder to explore new species or groups of individuals. The record-keeping in zoos and primate centers is also better than the laboratory, where most animals are procured from suppliers with unknown details, or, in the case of primates from breeding sites where this information is often unavailable.

Zoo and primate center environments are designed for welfare:

Zoos and primate centers have a keen interest in keeping their animals happy and unstressed. Aside from statutory and regulatory reasons, these places have a strong incentive to do this. Zoo audiences like seeing happy animals, and happy animals are more usable by primate center researchers and purchasers. In contrast, laboratory environments are designed to optimize data collection. As a result, zoos provide opportunities for ethologically valid behavior. That type of behavior is invaluable. For example, psychiatric studies often rely on the assumption that animal models are psychiatrically normal - deviations introduce biases which can in turn reduce the efficacy of treatment effects. Even non-psychiatric studies, however, can be hampered by the introduction of stressors that serve as uncontrolled variables (at best) or as competitors to variables of interest.

Conclusion

Behavioral imaging in primates is poised to provide many research benefits to biologists and medical researchers. The data such methods provide is greatly enriched compared to data derived from earlier methods, such as preference, reaction time, and gaze direction, which involve low amount of information about animals’ internal states. This is not to minimize the value of such methods. Measure of such low information behavior has been critically important for many studies and has led to many important insights. Indeed, that approach has been a mainstay of our own labs. Our labs, among others, have been interested in studying more complex and naturalistic behaviors and have begun to do so, although, still using very simple measures like positions of single landmarks and gaze (Yoo et al., 2020; 2021a and b). Clearly, big questions can be asked with conventional approaches.

However, it’s still looking at the world through a keyhole. With new technologies and new approaches, we are now able to open the door and step into the world and see behavior fully. It’s still hard to predict all the effects that change will cause. That’s partly because most of the analyses to study these data have not been invented. They will require new mathematical techniques that are only beginning to be delineated. Moreover, it’s not clear how much of the animals’ internal states leak out through their behavior. Nonetheless, we believe that these changes will lead to great advances in our understanding of the machinery of living things.

Data availability statement:

No data were used for this paper.