Fig. 4. Zero-shot behavioral quantification with SuperAnimal.

a Workflow overview for behavioral analysis with SuperAnimal. Artwork by Gil Costa. b Images in (b) are adapted from Lukas von Ziegler, Oliver Sturman, and Johannes Bohacek (2020) “Videos for deeplabcut, noldus ethovision X14, and TSE multi conditioning systems comparisons’’. Zenodo. 10.5281/zenodo.3608658 under a CC-BY license: https://creativecommons.org/licenses/by/4.0 with their DeepLabCut “in distribution" model (left) and our SuperAnimal zero-shot, out-of-distribution, results (right). c Ethogram comparing ground truth annotations vs. zero-shot predictions from SuperAnimal-TopViewMouse. d F1 score computed across IID (Sturman) and SuperAnimal with, or without CEBRA on the two behavioral classes (n = 20 videos). Box plots’ middle lines indicate the median. The bounds of the box indicate the first and third quartiles and the whiskers are drawn to the farthest datapoint within 1.5*IQR. e CEBRA⁴⁸ embedding on Sturman keypoints and SuperAnimal-based keypoints in 3D, transformed with FastICA for visualization. f Correlation matrix that demonstrates the correlation between SuperAnimal-TopViewMouse and ground-truth annotations averaged across three annotators and across the model and keypoint configurations. g We analyzed 30 horse videos where every frame had a ground truth (GT) annotation of keypoints¹⁶ (left) vs. our SuperAnimal-Quadruped model (right). The right limbs (closest to the camera) from one representative gait trial are shown. Swing and stance phases are colored in light gray and black zones, respectively. h Histogram delineating the number of videos where the ground contact by the hoof was identical to the GT vs. over or under-counted by one stride (no error larger than one was found). i: We computed the error between the GT stride length vs. model prediction for the hoofs (i.e., right_back_paw vs. Offhindfoot, etc). Each dot represents a stride, color denotes hindlimb vs. forelimb, near legs only.