Skeleton motion topology-masked prediction and contrastive learning for self-supervised human action recognition

Abstract

To address the limitations in data augmentation and neglect of joint dependencies in self-supervised human action recognition, this paper proposes a hybrid framework that integrates topology-masked motion modeling with contrastive learning. The proposed motion topology-masking technique jointly encodes skeletal topology and motion dynamics, preventing the model from over-focusing on temporally salient regions of prominent motions. We employ a multi-stage hybrid augmentation strategy, combining conventional and extreme augmentation methods to generate diverse, enriched positive pairs for contrastive learning. Additionally, we introduce a trajectory-guided feature dropping module, which selectively discards critical features based on trajectory attention maps, preventing the model from avoiding excessive focus on local joint trajectories. This approach effectively leverages large-scale unlabeled skeleton data through self-supervised learning, significantly reducing reliance on costly annotated datasets. Extensive experiments on NTU-60, NTU-120, and PKU-MMD demonstrate that the proposed model achieves superior performance in both occluded scenarios under complex environments and low-supervision conditions. It effectively mitigates visual interference and annotation scarcity while substantially improving action recognition accuracy.

Introduction

As an pivotal research direction in the field of computer vision, human action recognition has witnessed significant performance improvements in recent years with the development of deep learning and sensing technologies, and has shown great potential in intelligent applications.This technology has been widely applied in medical rehabilitation¹, intelligent interaction², security surveillance³, and sports science⁴, demonstrating significant application value across these domains.While prior research predominantly focused on appearance-based action recognition, alternative approaches have demonstrated the advantages of using posture information.As a key input for human action recognition, skeletal sequences inherently encode rich multi-dimensional spatiotemporal information.Compared to RGB videos, 3D skeleton sequences provide inherent advantages through their low-dimensional topological representation that mitigates privacy concerns regarding appearance disclosure, while their compact data format enhances computational efficiency and demonstrates intrinsic robustness against cluttered backgrounds, abrupt illumination changes, and clothing variations. These characteristics have established skeleton-based representation as a predominant research focus in action recognition, catalyzing substantial advancements in the field. However, current approaches remain constrained by their heavy reliance on annotated data, the acquisition of which is both labor-intensive and time-consuming. To address this limitation, this paper presents a novel investigation into self-supervised representation learning for 3D action recognition.

Current self-supervised human action recognition research primarily evolves around two paradigms: (1) Masked modeling-based approaches learn fine-grained feature representations by reconstructing randomly masked input data, inspired by Masked Autoencoders (MAE)⁵, these methods randomly mask up to 80% of joints or frame intervals in skeleton sequences, compelling the model to recover occluded content through spatiotemporal context reasoning. Such methods employ pixel-wise reconstruction loss to drive encoders in capturing fine-grained motion patterns, demonstrating outstanding performance in dense prediction tasks like action detection. (2) Contrastive learning-based approaches leverage augmentation techniques (e.g., skeleton cropping, motion jittering) to generate diverse positive sample pairs. By optimizing the InfoNCE loss to maximize feature consistency across different views of the same action, these methods learn discriminative sequence-level representations (e.g., global action semantics), demonstrating superior performance in skeleton-based motion representation.

Despite remarkable progress made by self-supervised learning in the field of human action recognition, existing approaches still confront several critical challenges.First, random masking strategies fail to effectively capture critical motion patterns and overlook joint dependency relationships, thereby limiting the model’s capacity to learn deeper-level features⁶. Meanwhile, existing contrastive learning approaches predominantly rely on conventional augmentation strategies, which often induce model overfitting and exhibit limited generalization capability⁷. Consequently, enhancing model generalization and robustness without relying on extensive annotated data has emerged as a pivotal research challenge in the field.

To address the limitations of conventional augmentations prone to overfitting, extreme augmentations causing information bias and critical feature dependency, and random masking strategies failing to capture essential motion patterns in contrastive learning, this paper proposes a novel framework integrating contrastive learning with motion topology modeling. By utilizing motion topology modeling to accurately focus on critical motion characteristics and integrating contrastive learning to build multi-perspective positive sample pairs for feature distribution optimization, this framework coercively guides the model to explore multi-dimensional features and learning more universal motion patterns

The main contributions of this work can be summarized as follows:

• This research proposes a methodological integration of contrastive learning with motion topology modeling, developing a novel masking mechanism that synthetically incorporates both skeletal topological structure and motion intensity metrics. The designed framework preserves static inter-joint constraints through topological graphs while applying kinematic-aware masking to motion-salient regions using dynamic motion information as auxiliary supervision, with adaptive parameter adjustment to accommodate emerging action patterns.

• Building upon conventional augmentation techniques, extreme augmentation strategies are employed to construct multi-perspective positive sample pairs, introducing richer motion patterns and more comprehensively exploiting latent information within the training data

• This paper proposes a trajectory-guided feature dropout module that specializes in analyzing and processing the kinematic trajectory characteristics of individual joints in skeletal data, thereby enabling the model to learn more generalized and transferable feature representations.

• Comprehensive experiments and ablation studies conducted on three large-scale benchmark datasets systematically validate the effectiveness of the proposed skeleton-based action recognition approach.

Related work

Skeleton-based action recognition

Initial approaches for skeleton-based action recognition primarily utilized hand-crafted feature extraction techniques, exemplified by the STIP method⁸ which detected spatiotemporal interest points in 3D space. While providing baseline performance, these methods showed limited generalization capability.

For temporal sequence modeling, recurrent architectures have demonstrated superior performance due to their inherent sequence processing capabilities. Long Short-Term Memory networks⁹ and Gated Recurrent Units¹⁰ effectively capture temporal dynamics and long-range dependencies in skeleton sequences. Subsequent developments introduced attention mechanisms (VA-RNN¹¹) and two-stream architectures¹² to jointly model temporal dynamics and spatial configurations. The IndRNN architecture¹³ further improved training stability by addressing gradient vanishing/explosion issues. However, these RNN-based approaches remain fundamentally limited in spatial feature extraction.

For spatial modeling, Convolutional Neural Networks have achieved strong performance through various skeleton representations. Wang et al.’s Joint Trajectory Maps¹⁴ encode motion trajectories as three-channel images, while PoseC3D¹⁵ directly processes raw skeleton data using 3D convolutions. Caetano et al. developed alternative representations using geometric algebra¹⁶ and tree structures¹⁷. Although effective, these CNN-based methods typically require converting skeleton data into pseudo-image formats, potentially compromising the inherent spatiotemporal relationships in the original data.

In recent years, Graph Convolutional Networks (GCNs), with their inherent adaptability to graph-structured data, have gradually become an important tool for skeleton-based action recognition. Yan et al.’s ST-GCN¹⁸ pioneered joint modeling of spatial and temporal features through graph representations. Subsequent improvements include AS-GCN’s action-specific dependencies¹⁹ and InfoGCN’s information bottleneck optimization²⁰. While these methods effectively exploit graph-structured data, their reliance on fixed topologies limits adaptability to diverse actions and long-range dependencies.

In parallel, the Transformer architecture has attracted widespread attention due to its powerful modeling capabilities, especially in handling temporal data. The ViViT model proposed by Arnab et al.²¹ employs spatiotemporal self-attention on video patches, while MTCN²² captures multimodal temporal contexts. Although these approaches excel at modeling long-range dependencies, their heavy reliance on annotated data poses challenges for skeleton-based tasks where labeling costs are prohibitive, especially for complex actions.

Despite significant advances in skeleton-based action recognition, current methods remain heavily dependent on costly manual annotations. To address this limitation, we propose a self-supervised framework integrating contrastive learning with masked skeleton modeling, significantly reducing annotation requirements while maintaining recognition performance.

Self-supervised masked modeling

Masked modeling has emerged as a powerful self-supervised paradigm, initially pioneered by BERT’s MLM approach in NLP²³. In computer vision, ViT²⁴ established the foundation for Transformer-based image processing, enabling subsequent masked approaches. MAE⁵ introduced an asymmetric encoder-decoder that reconstructs heavily masked images, while BEiT²⁵ advanced this through discrete token prediction. MaskFeat²⁶ further extended the framework by predicting feature descriptors from masked patches. These methods demonstrate masked modeling’s versatility across modalities while maintaining strong representation learning capabilities.

Masked modeling techniques have also been successfully applied to video data and 3D action recognition tasks. VideoMAE²⁷, an extension of MAE⁵ to video data, learns video representations by randomly masking partial regions of video frames and reconstructing the missing content. This method fully leverages the temporal continuity of video data, significantly improving the effectiveness of video representation learning.

In the field of 3D action recognition, MAMP (Masked Motion Prediction)²⁸ proposed a masked modeling method specifically for skeleton sequences. This method learns skeleton representations by masking partial joints in the skeleton sequence and predicting their motion information. By predicting the temporal motion of the masked joints, MAMP²⁸ enhances the model’s ability to model contextual motion information, significantly improving the performance of Transformer in 3D action recognition. Furthermore, SkeletonMAE²⁹ as a mask-based autoencoder for skeleton sequences, learns skeleton representations by masking partial joints in the skeleton sequence and reconstructing their original positions, further advancing the development of 3D action recognition technology.

Although the aforementioned methods have significantly improved the performance of visual representation learning by introducing masking mechanisms and Transformer architectures, they still face three limitations: high computational costs, strong dependence on large-scale data, and insufficient generalization ability. Additionally, previous works adopt topology-based masking strategies that lack flexibility, and the use of random masking fails to effectively capture key motion information, neglecting the inherent joint topological dependencies of human actions.

Current skeleton-based masking methods often rely on random masking or fixed topological strategies, failing to adequately consider the semantic importance and dynamic characteristics of joint movements, which limits the model’s ability to learn critical motion information. To construct more challenging and semantically relevant data views, introducing auxiliary information-guided masking strategies has become an important research direction. Such adaptive guidance mechanisms using multi-source information have been validated in various computer vision fields. For instance, in image dehazing, Su et al.³⁰ fused multiple prior-based dehazing results through an image quality-guided adaptive weighting scheme; in RGB-thermal tracking, FMTrack³¹ utilized inter-modal frequency characteristics to construct adaptive filters for cross-modal interaction, while TDAT³² enhanced feature discriminability by modeling target-background dependencies with global agents. The common advantage of these methods lies in their content-aware guidance mechanisms, enabling models to transcend reliance on raw data or random strategies and focus more precisely on key information regions. Inspired by this, our motion topology masking strategy integrates human topological priors with dynamic motion information to establish a content-aware masking guidance mechanism. This approach directs the masking process toward information-rich joints and temporal segments, thereby facilitating the learning of more discriminative action representations during self-supervised pretraining.

Self-supervised contrastive learning

Contrastive learning, as an important branch of self-supervised learning, learns feature representations by constructing positive and negative sample pairs and has shown excellent performance on skeleton data. Its core lies in generating diverse positive sample pairs through data augmentation and optimizing contrastive loss to learn consistent representations.

Recent advances in contrastive learning have significantly advanced self-supervised skeleton representation learning. MoCo³³ introduces a dynamic dictionary with momentum updates, while SimCLR³⁴ demonstrates effective view-invariant learning without memory banks. For skeleton data, Chen et al.³⁵ and Guo et al.⁷ developed specialized augmentation strategies, with the latter successfully adapting MoCo v2. Zhang et al.³⁶ enhanced spatiotemporal modeling through topological mixing, while Gao et al.³⁷ proposed motion trend-aware contrastive learning (ST-CL). CMD³⁸ addresses false negatives through hierarchical invariance learning, and STARS³⁹ replaces augmentation with feature-space neighbors. However, while contrastive learning excels at discriminative feature learning, it often overlooks intrinsic data semantics and structural relationships

Existing methods learn joint features and their relationships from unlabeled skeleton data to generate discriminative motion representations. Studies demonstrate that integrating heterogeneous network architectures can effectively enhance feature learning capabilities. For instance, models such as T-Net⁴⁰ and CLEAR⁴¹ proposed by Bui et al. combine Swin Transformer and Vanilla ViT to construct dual-stream backbone networks, incorporating channel-aware self-attention and cross-fusion modules to jointly model global and local features. Such approaches validate that functionally complementary network branches offer novel insights for learning discriminative representations, also providing references for designing multi-view and multi-granularity feature learning in self-supervised frameworks.

In contrastive learning, data augmentation is key to constructing positive sample pairs. However, samples generated by conventional augmentation strategies often exhibit high similarity, which may lead to model overfitting; while extreme augmentation can improve diversity, it may compromise semantic consistency. Thus, balancing diversity and semantic fidelity becomes a central issue—a challenge also present in weakly-supervised learning. For example, in weakly-supervised image dehazing, Wang et al.⁴² introduced a discriminator integrating spatial and frequency information to more comprehensively evaluate the discrepancy between generated images and the real distribution, thereby enhancing the realism and clarity of results. This inspires the need for adaptive optimization of positive sample construction and feature representation from multiple perspectives in contrastive learning.

Based on the above analysis, this paper proposes a multi-level augmented blending strategy. Building upon conventional augmentations, extreme augmentations are introduced to construct positive pairs for contrastive learning, thereby incorporating richer motion patterns.

Method

Framework overview

This work proposes a method that combines contrastive learning with motion-topology masked skeleton modeling, enabling simultaneous learning of both sequence-level semantic features and joint-level discriminative characteristics. This study employs a hybrid strategy that integrates conventional augmentation with extreme augmentation to capture the diversity and richness of complex actions, construct positive sample pairs from different perspectives, and uncover a broader spectrum of motion patterns. In addition, by introducing a trajectory-guided feature dropping module, this method conducts in-depth analysis of motion trajectories, amplifies subtle differences in movements, identifies and removes important features, thereby alleviating the potential overfitting problem during extreme augmentation. In the masked modeling module, we introduce a motion-aware strategy to augment the topology-based masking approach. By leveraging motion information as auxiliary guidance, this method prioritizes masking motion-rich regions, thereby integrating both static joint topology and dynamic motion patterns into the masking process.The proposed method uses a three-layer Bi-GRU as its core encoder in a self-supervised setup. This architecture is highly parameter-efficient, reducing overfitting risks when annotated data is scarce. Furthermore, its gating mechanism effectively captures long-term motion dependencies, which is crucial for reconstructing masked joints and discerning action semantics. This robust modeling of bidirectional temporal dynamics provides a strong foundation for understanding motion. For the decoder, we employ a two-layer GRU. The overall framework is illustrated in Fig. 1.

Fig. 1.

Overall framework diagram.

In Fig. 1, z,,, and represent the embedding vector, normal-view embedding vector, extreme-augmented-view embedding vector, discarded extreme-augmented-view embedding vector, and feature vector, respectively.

The proposed framework consists of two main stages: (1) pre-training and (2) action recognition. During the pre-training stage, a self-supervised learning approach integrating masked skeleton modeling with contrastive learning is adopted. This allows the encoder to capture both motion patterns and semantic information inherent in skeleton sequences, ultimately yielding an encoder that can extract high-quality, robust, and discriminative action features. Upon completion of pre-training, the encoder parameters are fixed. In the action recognition stage, the pretrained encoder processes input skeleton sequences to generate feature representations, which are then classified into action categories through a classification head.

Masked skeleton modeling

This paper proposes a motion-topology masking (MTM) strategy, which enhances masked skeleton modeling by incorporating joint motion intensity priors. The method introduces motion-aware masking into the topological masking framework through a weighting mechanism, enabling the masking process to jointly consider both skeletal connectivity and dynamic motion patterns.

The specific process is shown in Fig. 2. First, joint groups are predefined according to human anatomical topology to produce grouped skeletons as candidate masking regions. Then, motion intensity I is computed to generate a motion-aware masking probability distribution . The motion intensity and uniform distribution probabilities are blended to obtain mixed probabilities distribution. According to these mixed probabilities distribution, k joints are sampled and mapped to their affiliated body parts. All joints within the selected parts are masked (set to 0) during the current time segment, yielding the masked skeleton sequence.

Fig. 2.

Pipeline of the masked skeleton modeling.

Topology-based joint partitioning

First, the input skeleton sequence is divided into grouped sequence by partitioning joints according to human body topology, as shown in Fig. 3. The joints are categorized into the following groups: trunk, left arm, right arm, left leg, and right leg. Joints within each group exhibit close connectivity (e.g., left arm joints including shoulder, elbow, and hand). We apply uniform masking with shared probability p per group,making topological masking probability distribution , Meanwhile, the sequence is divided into different segments along the time dimension, with each segment having a length of l.

Fig. 3.

Topology-based partitioning.

Motion-aware masking probability computation

The proposed method calculates motion-aware masking probabilities using a motion-sensitive strategy. In contrast to the joint-level motion masking used in MAMP²⁷, our method operates at the body-part level of granularity.Specifically: First, the grouped skeleton sequence is divided into non-overlapping temporal segments . Based on this, the initial motion intensity I is derived by calculating the absolute differences in joint positions between adjacent frames. Subsequently, Multi-channel intensities are averaged to derive per-joint motion importance in each spatiotemporal segment.

1

where i is the temporal offset. The motion intensities are normalized into probability distributions using a temperature parameter:

2

The topology-based masking strategy incorporates motion intensity by weighted fusion of topological masking probabilities distribution and motion-intensity probabilities distribution, yielding comprehensive masking probabilities distribution. Each joint’s masking probability thus depends on both its topological position and motion intensity, given by:

3

Where is the weighting parameter balancing topological and motion information. The motion-topology mixing mechanism simultaneously highlights high-motion regions and prevents key joint overfitting.

The original skeleton input undergoes masking following the aforementioned strategy.This masked skeleton is then processed by encoder to extract feature representations, while the decoder reconstructs the skeleton to produce predicted skeleton The mean squared error (MSE) between original and predicted is optimized specifically for masked regions:

4

where N is the total number of masked joints, and M is the binary mask (1 for visible joints, 0 for masked joints). When , indicates the joint requires prediction; when , denotes the joint is known.

Multi-level augmentation hybrid strategy

Conventional approaches, typically reliant on a single augmentation strategy, offer limited positive sample diversity. This work introduces a Multi-level Augmentation Hybrid (MAH) strategy to systematically combine techniques of different granularities and intensities. The framework incorporates both semantics-preserving conventional augmentations and extreme augmentation aimed at discovering new motion patterns. Cooperation with the TGFD module (Section 3.4) alleviates potential noise from extreme transformations, thereby improving feature robustness and discriminability. The MAH strategy, which extends local- and mixed-skeleton augmentations with extreme variants, facilitates a more comprehensive exploration of motion patterns.

Local-skeleton augmentation

Local-skeleton augmentation primarily utilizes Temporal crop-resize⁴³ and Shear⁴⁴ on individual skeleton sequences for transformation learning, aiming to generate diverse samples. This helps the model learn feature representations with stronger generalization ability. By altering the spatiotemporal structure of skeleton sequences, the model is forced to learn more robust features, enabling it to better capture spatiotemporal relationships and improve its recognition capability.

Specifically, for input skeleton sequence S, positive sample pairs via temporal crop-resize, shear, and joint jittering transformations. Each pair is encoded by query encode and key encoder , then projected through and to yield features . A memory queue stores contrastive learning negatives, with network optimization via InfoNCE loss⁴⁵:

5

Where represents the i-th negative sample feature in the memory queue, is the temperature hyperparameter, and computes their similarity. Normalized and are processed. Post-training, batch samples update the memory queue M following the FIFO policy.

Mixed-skeleton augmentation

Mixed-skeleton augmentation employs CutMix⁴⁶, ResizeMix⁴⁷, and Mixup⁴⁸ to blend pairs of skeleton sequences, generating samples with diverse patterns and actions, thereby enhancing data diversity. Moreover, it enables the model to better understand the relationships between different actions and improves the model’s recognition capability.

Specifically, for two given skeleton sequences and , the mixing method is selected from the three augmentation methods by randomly determining the operation probability to obtain mixed skeleton data . The corresponding embedded features are then extracted via while component is optimized with the InfoNCE loss Eq.5.

Extreme augmentation

Conventional augmentation generates similar positive samples that constrain motion diversity exploration. Therefore, the extreme augmentation strategy proposed by AimCLR⁹ is further introduced to explore richer movement patterns. This approach combines spatial (translation/shearing/flipping/rotation), temporal (axis masking), and spatiotemporal (Gaussian noise/blur) transformations to produce significantly altered views. To mitigate identity distortion in original sequences and training instability requiring extended duration and sophisticated strategies, we selectively exclude the most aggressive transformations while retaining beneficial extreme augmentation. To mitigate identity distortion in original sequences and training instability requiring extended duration and sophisticated strategies,the extreme augmentation method is still adopted, but some overly extreme methods are excluded.

Specifically, to prevent information loss from axial masking (entire-axis occlusion hinders effective representation learning) and motion detail erosion from excessive Gaussian blur, the extreme augmentation in this paper uses three spatial augmentations: Shear⁴⁶, Spatial Flip, and Rotate; two temporal augmentations: Crop and Temporal Flip; and one spatiotemporal augmentation: Gaussian Blur, which produces the final extremely-augmented sequence. Subsequently, the embedded data is obtained through . Similarly, is optimized using the InfoNCE loss, as shown in Eq.5.

To counter the identity information loss from extreme augmentations, a dual distribution divergence minimization loss⁶ is employed between normal- and extreme-augmented views. This loss preserves motion semantics while enhancing feature discriminability, utilizing Kullback-Leibler (KL) divergence to measure distributional differences via two components: for normal versus extreme views, and for normal versus discard-based views:

6

7

8

the final loss:

9

This approach effectively leverages novel motion patterns from extreme augmentation while gently minimizing distribution discrepancies, thereby preventing performance degradation caused by overly aggressive transformations.

Trajectory-guided feature dropping module

The proposed trajectory-guided feature dropping (TGFD) module actively eliminates salient features to compel the model to learn more comprehensive representations and mitigate over-dependence on specific local features. By implementing a trajectory-aware attention mechanism, the module computes attention maps through temporal dimension compression, deriving global trajectory representations that focus on channel-wise motion patterns in skeletal data. This approach establishes long-term spatial dependencies between joints while amplifying subtle motion distinctions, thereby enhancing the model’s ability to discriminate semantically similar actions. The detailed pipeline is presented in Fig. 4.

Fig. 4.

Trajectory-guided feature dropping module.

The Trajectory-guided Feature Dropping (TGFD) module identifies and removes over-relied features by analyzing global joint trajectories. It uses temporal average pooling to create global descriptors, highlighting long-term spatial patterns. These descriptors are then processed by two separate MLPs to produce trajectory-based attention maps that purely reflect spatial importance from global motion.The input feature undergoes temporal average pooling to extract each joint’s global representation across all time steps. Two separate MLPs (each with two fully connected layers and ReLU activation) then non-linearly transform into Query () and Key () representations. Channel-wise element-wise multiplication between Query and Key yields initial attention scores, which are normalized via softmax over the joint dimension to produce the final trajectory-aware attention map:

10

The trajectory-based attention map and keep parameter jointly generate spatial attention masks and temporal attention masks , where keep controls feature retention by preserving low-attention-weight features while discarding high-attention-weight ones.

The spatial mask is applied to input feature , zeroing out important spatial features followed by normalization to yield feature , Temporal mask then processes , to produce the final output from the trajectory-guided feature dropping module.

The feature , after the dropping process, is projected through embedded projectors and to obtain its corresponding representation . Similarly, the InfoNCE loss is applied to optimize and . separately, as formulated in Eq.5.Unlike traditional regularization methods such as random dropping or low-attention feature selection, the proposed module actively suppresses high-attention features. This design breaks the model’s over-reliance on dominant local patterns, forcing the encoder to explore neglected complementary cues.

Obtaining classification results

The proposed framework trains the encoder via a synergistic mechanism combining contrastive learning with masked prediction to enhance feature representation quality.The masked skeleton prediction branch generates positive samples for contrastive learning, while gradient information - derived from discriminating action sequence features during contrastive learning - is backpropagated to update the reconstruction decoder in the prediction branch.

This framework jointly optimizes contrastive learning and masked prediction during pre-training to produce universal feature representations with superior action recognition capability. The co-optimization compels the encoder to learn discriminative motion patterns and semantic information from skeleton sequences. Through iterative optimization, the total loss progressively converges until encoder parameters stabilize at their feature-extraction-optimal state, after which the encoder becomes frozen.

The total loss function can be expressed as:

11

The hyperparameters , and act as weighting coefficients to balance contrastive learning loss , dual distribution divergence loss , and mask reconstruction loss . This weighted combination preserves contrastive learning’s semantic discriminability while enforcing stronger joint-level reconstruction constraints.

In the action recognition phase, input skeleton sequence X is processed through a bidirectional GRU encoder, producing hidden state tensor h with shape , where represents GRU hidden layer size and N denotes batch size. This encoder output is then fed into a fully-connected layer for classification, generating the final output y:

12

Here, W is the weight matrix of the fully connected layer, and b is the bias term, after which the output y is normalized through softmax to produce the probability distribution :

13

The value denotes the classification score for the i-th category among C total categories. The model’s predicted class for each sample is determined by selecting the category with maximum probability:

14

During this phase, the cross-entropy loss quantifies the difference between predicted outputs and ground-truth labels during this phase, driving parameter updates to optimize model training:

15

Where N denotes the dataset size and C the category count, represents the predicted probability of sample i belonging to class j, while the ground-truth label indicates actual class membership (1 for true, 0 otherwise).

The overall flow of the algorithm is shown in Algorithm 1:

Algorithm 1.

Procedure of the proposed method.

Experiment

Dataset

NTU RGB+D 60⁴⁹: This dataset comprises 60 action categories, covering three major scenarios: daily behaviors, medical condition-related actions, and two-person interactions, with a total of 56,578 samples. The standard evaluation protocols include cross-subject (xsub) and cross-view (xview).

NTU RGB+D 120⁵⁰: This dataset includes 120 categories of actions, covering four major scenarios: daily behaviors, medical diagnosis actions, two-person interactions, and device interactions, with a total of 114,480 samples. Its standard evaluation protocols consist of two types: cross-subject (xsub) and cross-setup (xset).

PKU-MMD⁵¹: The dataset is divided into two phases: Phase 1 comprises 1,076 long video sequences covering 51 action categories, while Phase 2 contains 1,009 short video sequences encompassing 41 action categories.

Experimental setup

Data Preprocessing:During data preprocessing, all skeleton sequences were uniformly aligned to 300 frames through zero-padding or truncation. This fixed length was empirically determined to preserve the essential information of most sequences in the dataset. Spatial-temporal joint normalization was applied to eliminate unit differences, with the data ultimately restructured into a normalized tensor format (3D coordinates 300 frames 25 joints 2 subjects) for direct model input.

Network Architecture:The model adopts a three-layer bidirectional GRU (Bi-GRU) as the encoder backbone, where the hidden dimension is set to and the number of encoder layers to 3. The decoder uses a two-layer GRU, with the number of decoder layers set to 2 and the hidden dimension to , which reconstructs the input data. An MLP serves as the projection head to map features into a 128-dimensional embedding space.

Pre-training:During self-supervised pre-training, we employ the SGD optimizer with 0.0001 weight decay and initialize the EMA parameter to 0.999. The model undergoes 429 training epochs with an initial learning rate of 0.02 and decayed to 0.002 at epoch 350. All experiments are implemented using PyTorch on NVIDIA RTX 3080 GPUs with a total batch size of 128.

Computational Efficiency:To evaluate the practical applicability of the proposed method, we further analyze its computational cost. The total number of model parameters is approximately 19.9 million, with a computational complexity of 2.7 GFLOPs. When pre-trained using two NVIDIA RTX 3080 GPUs with a batch size of 128, the process takes about 62 hours, and the peak memory usage per GPU reaches 14.2 GB. During the inference phase, memory usage is significantly reduced to 2.1 GB.

Reporting and Significance Analysis: The experimental results presented in this paper represent the mean accuracy obtained from five independent runs with different random seeds.

Evaluation and comparison

To validate the model effectiveness, experiments are performed on the NTU RGB+D 60, NTU RGB+D 120, and PKU-MMD benchmark datasets with evaluation conducted under three protocols: linear evaluation, fine-tuning, and transfer learning.

Linear Evaluation Protocol:Under the linear evaluation protocol, the pre-trained encoder parameters remain frozen for feature extraction, trained for 100 epochs using an SGD optimizer (momentum=0.9, batch size=256) with an initial learning rate of 0.1. Evaluations on NTU-60 and NTU-120 datasets demonstrate the proposed method’s superior performance compared to state-of-the-art approaches (Table 1).Compared with contrastive learning methods, it demonstrates obvious superiority by adopting the technology that combines contrastive learning with mask prediction. Particularly on the NTU-60 xview dataset, the performance of this method achieves the highest accuracy among all stream inputs, which further proves the superiority of the combination of contrastive learning and mask prediction. By using only joint data as input, it avoids the noise problem introduced by cross-modal fusion and can learn effective feature representations with a smaller amount of computation.

Table 1.

Performance comparison results under the linear evaluation protocol.Single-stream: joints; Three-stream: joints+bones+motion. The best and second-best accuracy rates are highlighted in bold and underlined, respectively.

Method	Input	NTU-60		NTU-120
		xsub (%)	xview (%)	xsub (%)	xset (%)
LongTGAN52	Single-stream	39.1	48.1	35.6	39.7
H-Transformer53	Single-stream	69.3	72.8	–	–
ML54	Single-stream	52.6	–	–	–
ISC55	Single-stream	76.3	85.2	67.1	67.9
SkeletonMAE29	Single-stream	74.8	77.7	72.5	73.5
CRRL5	Single-stream	67.7	73.8	56.2	57.0
GL-Transformer45	Single-stream	76.3	83.8	66.0	68.7
HaLP56	Single-stream	79.7	86.8	71.1	72.2
CPM57	Single-stream	78.7	84.9	68.7	69.6
P&C58	Single-stream	50.7	75.3	42.7	41.7
CMD38	Single-stream	79.4	86.9	70.3	71.5
SkeletonCLR59	Three-stream	75.0	79.8	60.7	62.6
AimCLR7	Three-stream	78.9	83.8	68.2	68.8
PSTL60	Three-stream	79.1	82.6	69.2	70.3
SkeAttnCLR61	Three-stream	82.0	86.5	77.1	80.0
HYSP62	Three-stream	79.1	85.2	64.5	67.3
RVTCLR+63	Three-stream	79.7	84.6	68.0	68.9
ViA64	Two-stream	78.1	85.8	69.2	66.9
Skeleton-logoCLR65	Single-stream	82.4	87.2	72.8	73.5
Ours	Single-stream	80.9	87.8	73.5	74.7

Fine-tuned Evaluation Protocol:Under the fine-tuning protocol, we attach an MLP head to the pre-trained backbone and fine-tune the entire network for 100 epochs (batch size=128) with an initial learning rate of 0.1 following cosine decay. Table 2 demonstrates our evaluation on NTU-60 and NTU-120 datasets. Compared with self-supervised approaches (HYSP⁶², SSL⁶⁶, ViA⁶⁴), our method achieves superior performance on NTU-60 xview and NTU-120 xset benchmarks and second-best accuracy on NTU-60 xsub and NTU-120 xsub. These results validate that our mask prediction-enhanced contrastive learning framework improves supervised generalization, while the pre-trained features exhibit strong discriminative power and transferability.

Table 2.

Performance Comparison under Fine-tuning Protocol.Single-stream: joints; Three-stream: joints+bones+motion. The best and second-best accuracy rates are highlighted in bold and underlined, respectively.

Method	Input	NTU-60		NTU-120
		xsub (%)	xview (%)	xsub (%)	xset (%)
SkeletonMAE29	Single-stream	86.6	92.9	76.8	79.1
CrosSCLR51	Three-stream	84.6	90.5	75.0	77.9
CPM57	Single-stream	84.8	91.1	87.4	78.9
AimCLR7	Three-stream	83.9	90.4	74.6	77.2
HYSP62	Three-stream	89.1	95.2	84.5	86.3
SSL66	Single-stream	92.8	96.5	84.8	85.7
ViA64	Two-stream	89.6	96.4	85.0	86.5
FreqMixFormer67	Single-stream	91.5	96.0	87.9	89.1
BlockGCN68	Single-stream	90.9	95.4	86.9	88.2
Ours	Single-stream	90.7	96.6	85.8	86.5

Transfer Learning Protocol: This protocol assesses representation transferability through a two-stage framework: self-supervised pre-training on source datasets (NTU-60/NTU-120) followed by supervised fine-tuning on the target dataset (PKU-II). As shown in Table 3, the model demonstrates reasonable but limited transfer performance.Compared to methods such as MAMP, our approach still has room for improvement in terms of transfer performance on PKU-MMD II. This can be primarily attributed to the significant domain gap between the NTU dataset and the PKU-MMD dataset. The NTU dataset contains a large number of “two-person interaction” actions (e.g., ’handshaking’, ’hugging’), whereas PKU-MMD (particularly in its second partition) focuses more on “individual daily actions” (e.g., ’drinking water’, ’typing on a keyboard’). This discrepancy in action category distribution and interaction complexity may lead to insufficient discriminative capability for certain fine-grained actions in PKU-MMD by models pre-trained on NTU. Furthermore, the masking strategy applied to the source data simulates continuous occlusion scenarios, while occlusions in the target dataset are more random and irregular. This inconsistency in occlusion patterns also contributes to the suboptimal transfer performance.

Table 3.

Performance Comparison under Transfer Learning Protocol.Source datasets: NTU-60 and NTU-120. Best and second-best accuracy are highlighted in bold and underlined, respectively.

Method	To PKU-II
	NTU-60 (%)	NTU-120 (%)
LongTGAN52	44.8	–
MS2L53	45.8	–
ISC55	51.1	52.3
CMD38	56.0	57.0
SkeletonMAE29	58.4	61.0
MAMP28	70.6	73.2
Ours	59.5	60.1

Ablation study

This section provides an in-depth analysis of the proposed method. The results are obtained under the linear evaluation protocol.

Module Effectiveness Analysis:To validate the contribution of each proposed module, we conduct comprehensive experiments on the NTU-60 xview dataset (Table 4). The model achieves competitive action recognition performance when using motion-topology masking (MTM) with conventional augmentation (CA). Switching to random masking (RM) and multi-level augmentation hybrid (MAH) yields significant improvement, while combining motion topology masking with hybrid augmentation further refines performance. The optimal results are attained by integrating the trajectory-guided feature dropping (TGFD) module, demonstrating that our three strategies collectively enable the encoder to learn more robust and task-adaptive features.

Table 4.

Ablation Study on Proposed Modules. indicates module usage, best results in bold.

MTM	MAH	TGFD	RM	CV	NTU-60 (%)
					84.8
					85.9
					86.1
					87.8

Masking Strategy Analysis:The proposed Motion Topology Masking (MTM) strategy combines motion-aware and topological masking to create more challenging data views. Comparative results on NTU-60 xview and NTU-120 xset datasets (Table 5) confirm its efficacy: random masking (RM) is the least effective, while introducing either topology-based (TRM) or motion-aware (MAM) masking individually boosts performance to similar levels, implying complementary supervisory signals. The complete MTM strategy achieves state-of-the-art results, proving that motion cues act as a powerful prior for semantic-rich masking. This synergy with topological masking avoids over-dependence on motion alone and delivers the highest performance.

Table 5.

Ablation Study on Masking Strategies. best results in bold.

Method	NTU-60(%)	NTU-120 (%)
RM	86.6	73.2
TRM	87.2	73.9
MAM	87.1	73.5
MTM	87.8	74.7

Masking Ratio Analysis: this paper conducts the ablation study on the masking ratio using the NTU-60 XView dataset. As a widely adopted benchmark in skeleton-based action recognition, the evaluation results from this dataset are recognized for being generalizable and comparable. As shown in Fig. 5, our experiments with different masking ratios reveal that both excessively high and low ratios degrade performance on the NTU-60 xview dataset. Empirical results demonstrate that a 40% masking ratio achieves optimal performance, balancing sufficient data augmentation with information preservation.

Fig. 5.

Ablation Study on Masking Ratios.

Ablation Study on the Masking Weight : To validate the balancing mechanism between topological priors and motion information in our motion topology masking strategy, we conducted an ablation study on the mixture weight under the NTU-60 xview protocol. As shown in Table 6), as increases from 0 (pure motion masking) to 1 (pure topology masking), model performance first rises and then declines, peaking at 87.8% when. This optimal configuration indicates that the topological structure provides a slightly dominant prior (weight 0.6), while motion information serves as an essential dynamic semantic supplement (weight 0.4). The synergy between the two enables the generation of masked views that are both semantically challenging and physiologically plausible, confirming the theoretical advantage of the linear combination in Eq. (4).

Table 6.

Ablation Analysis of the Mixture Weight on NTU-60 xview,best results in bold.

	Topological Weight	Motion Weight	NTU-60 (%)
0.0	0.0	1	86.6
0.2	0.2	0.8	87.0
0.4	0.4	0.6	87.5
0.6	0.6	0.4	87.8
0.8	0.8	0.2	87.3
1.0	0.0	1.0	86.9

Extreme Augmentation Combination: To systematically evaluate the contribution of each component in our extreme augmentation strategy, we conducted component-level ablation studies on the NTU-60 xview dataset. As shown in Table 7, based on the complete set of six transformations (including Shear, Spatial Flip, Rotate, Crop, Temporal Flip, and Gaussian Noise), the separate introduction of the excluded Axial Masking and Gaussian Blur resulted in significant accuracy drops of 2.7% and 1.8%, respectively. This indicates that Axial Masking compromises representation learning by disrupting the spatial topology of skeletons, while Gaussian Blur impairs motion discriminability by excessively smoothing temporal features. Our experiments demonstrate that the adopted six transformations collectively achieve an optimal balance between enhancing data diversity and preserving essential action semantics.

Table 7.

Ablation Study for the Extreme Augmentation Combination.

Augmentation Strategy	NTU-60 (%)
complete set	87.8
incorporating Axial Masking	85.1
incorporating Gaussian Blur	86.0

Enhanced Performance Analysis: As evidenced in Table 8, comprehensive experiments on NTU-60 xview demonstrate distinct performance patterns across augmentation strategies: conventional augmentation (CA) alone achieves 84.8% accuracy, while exclusive extreme augmentation (EA) degrades performance due to excessive feature distortion. The hybrid approach combining conventional and extreme augmentations yields superior results to conventional-only, confirming their complementary effects. While standard Dropout (SD) improved accuracy to 86.3%, and ordinary attention (OA) further increased it to 86.9%, and trajectory-based attention (TBA) achieving optimal performance through enhanced motion pattern learning from extreme-augmented contrastive samples. These results systematically validate three key insights: first, standalone extreme augmentation carries inherent risks of feature destruction; second, balanced hybrid strategies are essential for optimal performance; third, trajectory-guided attention provides unique representational advantages in skeleton-based learning.

Table 8.

Ablation Study of Augmentation Methods. indicates module usage, best results in bold.

CA	EA	SD	OA	TBA	NTU-60 (%)
					84.8
					81.3
					86.1
					86.9
					86.9
					87.8

Qualitative Comparison: Figure 6 presents t-SNE visualizations comparing our method with CMD³⁸ and MAMP²⁸ on NTU60’s 15 action classes.t-SNE visualization, a widely adopted qualitative evaluation tool in skeleton-based action recognition, effectively reveals the distribution structure of features in low-dimensional space by clearly demonstrating intra-class compactness and inter-class separation. CMD’s cross-modal distillation yields semantically-driven but dispersed distributions, while MAMP enforces strict inter-cluster separation with intra-cluster compactness. Our approach demonstrates distinct cluster formation (e.g., well-separated “giving object” (gray) and “shaking hands” (dark blue)), reflecting effective inter-class discrimination, while maintaining intra-class cohesion (e.g., tight “pick up” (yellow) clusters).Remaining challenges appear in semantically similar actions (“reading” (blue)/“phone use” (lavender)) where shared motion patterns (“bending”,“grasping”) cause overlap, suggesting needs for finer-grained feature learning.

Fig. 6.

t-SNE Visualization of Feature Embeddings.

Hyperparameter analysis

To systematically evaluate the model’s sensitivity to key hyperparameters, we conducted a comprehensive parameter analysis on the NTU-60 xview dataset. All parameters were optimized through grid search following a systematic parameter scanning strategy.As shown in Table.9 , with fixed at 1.0, the optimal balance is achieved when = 0.3 and = 30. This configuration effectively coordinates the learning processes of different pre-training tasks while preventing any single task from dominating the training. For the motion topology masking weight , the model performs best at , with performance degradation limited to within 0.5% across the range , demonstrating its robustness. In the TGFD module, the optimal keep ratio is 0.7, indicating that retaining 70% of high-attention features achieves an effective balance between breaking model dependency and preserving essential semantic information. The temperature parameter , which controls the smoothing of similarity distributions in contrastive learning, reaches its optimal value at = 0.07, maintaining effective distinction between positive and negative samples while ensuring training stability.

Table 9.

Sensitivity Analysis of Key Hyperparameters.

Hyperparameters	Search Range	Optimal Value	Sensitive Interval
	[1.0, 2.0]	1.0	[0.7, 1.3]
	[0.1, 1.0]	0.3	[0.2, 0.5]
	[10, 50]	30	[25, 35]
	[0.3, 0.9]	0.6	[0.5, 0.7]
keep	[0.05, 0.9]	0.7	[0.65, 0.75]
	[0.05, 0.09]	0.07	[0.06, 0.09]

Limitations and future work

This work has demonstrated notable performance gains, though several limitations remain. Addressing these limitations outlines a clear path for future work:

the model demonstrates sensitivity to data quality and exhibits unstable performance when processing noisy data. the representation learning framework shows limited capacity in modeling multi-level semantic features, making it challenging to effectively capture fine-grained motion characteristics. the model’s cross-domain generalization capability remains constrained by structural priors.

Our future work will focus on three primary directions: performing an in-depth qualitative analysis (by visualizing TGFD attention and MTM masks) to inform model refinement; integrating the pre-training framework with powerful backbone networks such as Transformer and GCN; and exploring a robust pre-training paradigm that incorporates fine-grained attention and adaptive topology learning, coupled with asymmetric architecture design to enhance training efficiency.

Conclusion

This study presents a self-supervised framework that synergizes motion-topology masked prediction with contrastive learning to enhance action recognition.The motion-topology masking strategy selectively focuses on high-intensity motion regions while maintaining topological constraints, enabling effective learning of discriminative motion patterns and joint dependencies.A hybrid strategy of normal and extreme augmentations is adopted to introduce novel motion patterns, increasing the diversity of contrastive learning and helping the model learn richer feature representations. To address the potential loss of original identity information caused by extreme augmentations, a trajectory-guided feature dropping module is introduced to enable the model to learn more comprehensive features and enhance recognition accuracy.Comprehensive ablation studies and comparative evaluations confirm the method’s superior recognition performance.

Author contributions

Y.H. conceived the methodology and experiment(s), F.L. provided the resources, F.L., Y.H. and X.H. conducted the experiment(s),Y.H. and C.S. analysed the results,X.G reviewed the results. X.G.,and C.S. contributed equally to this work. All authors have read and agreed to the published version of the manuscript.

Data availability

The datasets used and/or analysed during the current study available from the corresponding author on reasonable request.

Declarations

Competing interests

The authors declare no competing interests.