Fig. 1. Keypoint trajectories exhibit sub-second structure.

a, Left: simultaneous depth and 2D infrared (IR) recording setup. Middle: pose representations using the depth data (top) or IR (bottom, tracked keypoints indicated). Right: Example syllable sequences from MoSeq applied to depth data (referred to as ‘MoSeq (depth)’) or to keypoint data (referred to as ‘MoSeq (keypoints)’). Figure created with SciDraw under a CC BY 4.0 license. b, Keypoint change scores or low-confidence detection scores, relative to the onset of MoSeq transitions (x axis) derived from either depth (gray) or keypoint (black) data. Differences in each case were significant (P = 2 × 10⁻⁷ over N = 20 model fits, Mann–Whitney U test; plots show mean and range across model fits). c, Comparison of syllable durations for MoSeq (keypoints) and MoSeq (depth), showing mean and inter-95% confidence interval range across N = 20 model fits. d, Left: keypoint detection errors, including high-frequency fluctuations in keypoint coordinates (top row) and error-induced syllable switches (bottom row). Right: keypoint coordinates before (frame1) and during (frame2) an example keypoint detection error. This error (occurring in the tail keypoint) causes a shift in egocentric alignment, hence changes across the other tracked keypoints. e, 5-s interval during which the mouse is immobile yet the keypoint coordinates fluctuate. Left: egocentrically aligned keypoint trajectories. Right: path traced by each keypoint during the 5-s interval. f, Variability in keypoint positions assigned by eight human labelers. g, Cross-correlation between various features and keypoint fluctuations at a range of frequencies. Each heat map represents a different scalar time series (such as ‘transition probability’—the likelihood of a syllable transition on each frame). Each row shows the cross-correlation between that time series and the time-varying power of keypoint fluctuations at a given frequency.