Behavioral tracking gets real

Abstract

A deep-learning-based software package called DeepLabCut rapidly and easily enables video-based motion tracking in any animal species. Such tracking technology is bound to revolutionize movement science and behavioral tracking in the laboratory and is also poised to find many applications in the real world.

Physically interacting with the world triggers neural activities throughout much of the nervous system¹. But to understand movement and how it relates to brain function, we need ways of quantifying it with high spatial and temporal accuracy. Breakthroughs in motion capture technology have driven many advances in movement science. For instance, Muybridge’s high-frequency filming resolved the debate as to whether a running horse flies with four feet disengaged from the ground, and has inspired researchers to focus on the kinematics of humans and animals since 1872². The French scientist E.-J. Marey was the first to propose analyzing human and animal motion by photographing marks or light bulbs attached to the body³. In the 1970s, G. Johansson video recorded reflective patches on joints and opened the new era of computer modeling of motion capture⁴. More recently, technologies based on passive and active markers have enabled precise and fast motion tracking and led to a broad range of insights about the brain. Yet these approaches are complex, and only specialized labs study the fine details of movement produced during behavior. In this issue of Nature Neuroscience, Mathis and colleagues introduce DeepLabCut⁵, a new tool that makes fine motion tracking of any animal species readily accessible: all you need is to install the software, spend a few hours labeling exemplars, and then run the code (Fig. 1).

Fig. 1 |. Motion capture examples.

DeepLabCut produces meaningful results across many animal species, including mice, fruit flies and even young humans. Top left, top right and bottom left reprinted with permission from ref.⁵, Springer Nature.

The limits of motion tracking technologies are felt all over the field. For instance, hand reaching, which is easily tracked, is the most popular model in human motor control studies. However, focusing on hand movements, often confined in the horizontal plane by a fixed shoulder, obscures the real beauty of human movement control: we walk, run, jump, dance, and interact with other people and animals, both indoors and outdoors. All these rich behaviors are largely left unexplored and potentially misunderstood by extrapolating from findings on small hand movements. Similarly, we know relatively little about the movement of rodents, in large part because no good standard movement tracking systems exist. Real-world motion tracking should also be easy to use, free of interference with behavior, and based solely on video recordings. DeepLabCut answers all those needs. This may sound very much like magic, so let us go through what has made DeepLabCut such an accurate and easy-to-use tool for motion tracking.

Motion capture based on videos started in the 1970s⁶, but only with recent advances in machine learning has it become ready for a variety of research areas that are in dire need of large-scale movement data: for example, ecology⁷ and neuroscience⁸. The Mathis study⁵ is exciting because it harnesses the power of deep learning to enable motion tracking by machines with human-like accuracy without needing much training data. To build DeepLabCut, the authors first pretrained their deep neural network on ImageNet, a massive open database of images used for image recognition research⁹. The aim of this first step was to reduce the amount of data needed for training. This allowed the algorithm to achieve close-to-human performance in capturing movements with only a few hundred labeled training images. Second, DeepLabCut uses an architecture (DeeperCut) that has been extensively optimized for the estimation of pose from photographs of humans¹⁰, which was found to generalize to animals. Third, the system does not detect just one body part, but all of them at the same time. This allows the network to effectively use the location of each body part to assist in the localization of the others. Like so often in modern machine learning, the combination of a broad set of intuitive ideas allows the building of superior systems. Fourth, DeepLabCut was tested and calibrated across species—for example, rats and flies—and across activities—for example, locomotion and hand reaching. Fifth, and arguably most important, the code is provided free on GitHub (https://github.com/AlexEMG/DeepLabCut), easy to use and of high quality. The result is that many people, including us, have already started using it. In the future, not every laboratory will need to go to the trouble of training their models: there will be standard models, say for tracking rodents. In fact, we started to adapt the code to our needs well before this paper was in press and found that it was easy to understand and a pleasure to work with—for example, for tracking babies. It is easy to download, has limited dependencies, and pretty much works right out of the box.

Like all supervised learning algorithms, DeepLabCut starts with a labeled dataset. The scientist using it will start with a few videos. If she wants to track a dinosaur, she will take a few frames from the video database and indicate where the head, the neck and each of the paws are located, simply with a few mouse clicks. The deep learning algorithm can then use this information to effectively estimate how probable each image region is to be a given body part—say, the right front paw. All these developments make it attractive for any lab in the world to use the code for real-world motion tracking.

Notably, DeepLabCut allows the conversion of any video containing movement into motion capture information and thus opens up a huge treasure trove of data for movement science. We are indeed sitting on massive movement data if we consider existing video databases, such as YouTube (> 10⁹ videos and growing), which have a high proportion of videos of human and animal movements. In the future, given the ease of video shooting, we predict that motion capture will move from an expensive and difficult task restricted to the laboratory to an effortless daily routine for everyone.

Motion capture has long been key to improving performance among top athletes^11,12. However, it requires cumbersome and technically intensive tracking approaches and specialist knowledge for interpretation, rendering it expensive and only available to elite athletes and the wealthy. We can imagine a future in which everyone can receive professional-grade guidance on sports training based on automatic movement analysis. Technologies such as DeepLabCut make motion capture effectively free and easy, which could allow people everywhere to learn sports and other movement skills faster.

Motion capture also has a rich history in physical medicine and rehabilitation. Gait laboratory analysis can reveal problems with walking or running, but the procedure is expensive (often in the $1,000 range). In some medical areas, doctors rely on the subjective coding of behaviors (that is, visual inspection of movements) for diagnosis of, for example, autism¹³, or for evaluating recovery of motor functions¹⁴. While certified doctors or physical therapists draw on a deep trove of experience, their services are often only available to relatively wealthy patients and are not readily available in many countries. Ubiquitous and effortless motion capture promises to bring movement science into medical practice. In the future, motion analysis by video will likely become a first and ubiquitous step for anyone suffering from just about any movement issue.

Retrospectively, movement science has been elevated to new heights whenever cutting-edge motion capture technology has opened our eyes to the complexity and beauty of animal movements. In the era of artificial intelligence and blessed by automatic video-based motion capture, we are ready for another wave of exciting discoveries in behavioral sciences. We predict that this time it will enable movement science to transcend the boundaries of laboratories and become an essential part of everything involving movement, as tracking-based, artificial intelligence–enabled movement trainers allow us to learn sports faster while avoiding injuries, and provide diagnostic information that will be crucial for precision medicine.