MAGNet: enhancing action recognition with multimodal fusion and adaptive graph convolution

Abstract

Human action recognition is a key task in computer vision, with widespread applications in virtual reality, intelligent surveillance, and human-computer interaction. Although deep learning methods have made significant progress in this task, existing methods still face challenges, including difficulties in multimodal data fusion, insufficient robustness in complex environments, and a decrease in accuracy when data is missing or modalities are incomplete. To address these challenges, this paper introduces a novel approach by proposing a human action recognition model based on the Multimodal Adaptive Graph Convolutional Network (MAGNet). The core innovation of this work lies in the integration of adaptive graph convolutions and cross-modal self-attention mechanisms to enhance multimodal data fusion. By dynamically adjusting the contribution of each modality, the proposed method addresses the challenge of incomplete data and improves robustness under real-world conditions, such as missing or noisy modalities. Additionally, the incorporation of the VQ-VAE generative model provides an efficient way to handle missing data and generate anatomically consistent pose features, which sets this approach apart from existing methods. Experimental results show that MAGNet achieves state-of-the-art performance on both the NTU RGB+D and UTD-MHAD datasets. Specifically, on the NTU RGB+D dataset, the model achieves 95.2% and 98.8% accuracy on the XSub and XView protocols, respectively, significantly outperforming existing baseline methods. Furthermore, MAGNet demonstrates strong robustness in multimodal data fusion and complex scene adaptation, effectively handling challenges such as occlusion and lighting variation.

Introduction

Human action recognition is a task in computer vision, with applications in virtual reality, human interaction, and robotic control^1,2. The objective is to identify joint positions in images or videos and estimate the 3D pose^3,4.Deep learning progress improved performance. But challenges remain. Lighting, perspectives, and occlusions affect accuracy. Current methods use a single modality. Like RGB images or depth maps. They work well in controlled environments. However, in real-world settings, performance can drop due to lighting changes, viewpoint shifts, and noise^5,6. In complex environments, single-modality approaches face difficulties in fully utilizing the available data. This has led to more interest in multimodal data fusion. It combines RGB, depth, and 3D pose data to improve robustness^7,8.

In multimodal data fusion research for human action recognition, the core focus is on resolving two interrelated issues to enhance model robustness: how to select appropriate fusion strategies, and how to address heterogeneity between modalities. For fusion strategies, early fusion combines data at the input layer and late fusion merges results at the output layer^9,10–both have advantages and limitations, with the optimal choice depending on task requirements¹¹. For modality heterogeneity, current research explores solutions via new architectures, strategies, and deep learning approaches: Combining RGB and depth data mitigates single-modality constraints^12,13, incorporating 3D pose data enhances the model’s ability to understand and predict human poses¹⁴, and emerging methods using graph convolutional networks (GCNs) capture spatial and temporal relationships to optimize fusion¹⁵. Despite these efforts, effectively integrating heterogeneous modal data remains a key challenge.

To address these challenges, we propose MAGNet (Multimodal Adaptive Graph Network). It is a framework designed to improve performance through multimodal data integration. MAGNet couples adaptive graph convolution with cross-modal self-attention via a dynamic collaborative modulation mechanism: the former dynamically adjusts the skeleton graph’s topological structure based on multimodal feature distribution, while the latter modulates the graph convolution kernel weights in real time according to heterogeneous modal relevance, forming a bidirectional feedback loop between spatial structure modeling and cross-modal feature alignment. The model uses adaptive graph convolutions to capture spatiotemporal dependencies between multimodal. The model also includes cross-modal self-attention mechanisms. Distinct from existing static feature interaction approaches, our cross-modal self-attention employs a modality-aware scoring mechanism that adaptively reweights each modality’s contribution based on action semantics and data integrity, rather than merely computing cross-modal similarity. This improves feature fusion and ensures effective combination of information from multiple sources. The addition of generative components helps keep the output consistent, even when input data is incomplete or corrupted. Our generative component adopts a modality-compensated strategy, leveraging spatiotemporal constraints from adaptive graph convolution to guide feature generation–ensuring generated features align with valid modal information and human action structural characteristics, rather than generating missing modalities independently. The architecture adjusts the graph structure dynamically based on the input data characteristics. Attention-based fusion helps preserve the relationships between the modalities. This approach ensures robust action recognition, even under conditions of data loss or distortion.

The contributions of this research are as follows:

• The MAGNet model is proposed, which combines adaptive graph convolution, cross-modal self-attention mechanism, and generative models. Unlike traditional methods (e.g., nonnegative matrix factorization) that only perform shallow linear multimodal feature decomposition, and existing deep learning methods relying on simple concatenation/weighted averaging for fusion, our model establishes a bidirectional modulation between graph convolution and cross-modal attention–where structure adaptation and feature weighting mutually constrain and enhance each other, rather than being a sequential module combination. This model efficiently fuses multimodal data and improves action recognition accuracy.

• An innovative cross-modal feature fusion method is designed, which addresses the noise and inconsistency issues in multimodal data through the cross-modal self-attention mechanism. Compared with existing methods that overlook dynamic inter-modal correlations, our modality-aware attention scoring mechanism captures fine-grained dependencies between RGB, Depth, and Skeleton modalities, enabling dynamic, semantically aligned fusion instead of static feature stitching. This achieves more effective feature alignment and fusion.

• A generative model is introduced that enhances the generative capability of action recognition. Unlike other generative-based methods focusing solely on single-modal completion, our modality-compensated generative strategy uses spatiotemporal constraints from adaptive graph convolution to ensure semantic consistency with human action structure, outperforming baselines in handling partial/complete modality loss. In cases of missing data or incomplete modalities, the generative model provides robust action recognition results.

Related work

Human action recognition methods

Human action recognition is a crucial research area in computer vision. It aims to predict joint locations of the human body from input images or videos. With the advancement of deep learning technologies, human action recognition has moved from traditional handcrafted feature extraction methods to deep learning-based approaches. The application of convolutional neural networks (CNNs) has accelerated progress in this area.

Early action recognition methods mainly used traditional computer vision techniques. These methods used handcrafted features for joint detection and localization. Classical traditional methods such as nonnegative matrix factorization (NMF) were also applied in this stage, which decomposed high-dimensional feature matrices of human action data into low-rank nonnegative matrices to extract discriminative action features, laying a foundation for subsequent feature fusion in multimodal action recognition. Common methods included image-based keypoint detection and model-based pose inference. These methods struggled with occlusion and pose variations in complex scenes, especially under noisy backgrounds and lighting changes^16–18. With deep learning, CNN-based methods became mainstream. Traditional deep learning methods relied on 2D image inputs. CNNs extracted image features, which were used for joint regression or classification¹⁹. Early CNN methods directly performed global feature learning on images. As human poses became more complex, these methods showed limitations. Researchers started using per-joint regression and pose graph optimization strategies to improve model accuracy and robustness^20–22. 2D action recognition methods based on CNNs can locate joint positions accurately via keypoint regression. They are effective at handling large pose variations²³. However, they rely on 2D image data, which cannot capture depth information or 3D structure. This leads to errors in certain angles and environments. With the development of 3D sensing technology, 3D action recognition using depth images or point cloud data became a trend. By combining RGB images with depth information, 3D action recognition overcomes limitations in the depth dimension^24,25. Using RGB-D sensors or LiDAR technologies, researchers can obtain richer spatial information. This helps solve issues like occlusion and joint overlap in action recognition. CNNs and graph neural networks (GNNs) can extract spatial features from depth images or point clouds for 3D pose reconstruction^26–28. However, 3D action recognition faces challenges in complex scenes, especially with data loss, noise interference, and multimodal data alignment issues. Sequential data has provided breakthroughs in human action recognition. Video or time-series data allows action recognition to utilize temporal relationships between frames^29,30. This improves model robustness and accuracy. action recognition methods using sequential data, like LSTM or ConvLSTM, can capture dynamic features of human motion. These are particularly useful for action recognition and prediction^31–33. Sequential models combine previous and future frames’ information, solving pose variations that are difficult to detect in single frames. This improves overall accuracy and stability³⁴.

Despite advancements in 2D and 3D action recognition with deep learning, challenges remain. Existing methods in multimodal data fusion typically rely on simple feature fusion techniques like concatenation or weighted averaging. These methods fail to explore deep relationships between modalities. Efficiently fusing multimodal data to improve robustness is still an important research problem.

Multimodal learning and fusion

Multimodal learning helps models perform better. It integrates different data sources. In computer vision and AI, it is applied to many areas. Fusing data from different sources improves task performance. Multimodal have complementary roles. One modality may offer a different type of information compared to others. Using them together helps understand complex tasks.

Methods for multimodal fusion include early fusion, late fusion, and intermediate fusion. Early fusion processes data together at the input. Late fusion combines results at the output^35,36. Intermediate fusion merges data at the middle layers of the model. In recent years, deep learning has helped a lot in multimodal fusion. Neural networks are useful for aligning and combining data from different sources. Existing studies have developed targeted technical paths to address the core challenges of multimodal fusion for human action recognition, such as handling modality heterogeneity, ensuring feature consistency, and resisting noise. These designs are closely tied to the task’s demand for robust pose inference. Multimodal learning has achieved notable advancements in diverse research directions including knowledge graph completion, recommendation systems and graph embedding, with several representative studies proposing innovative strategies for noise processing, adaptive modality interaction and semantic feature modeling³⁷. APKGC introduces a noise-enhanced framework combined with an attention penalty mechanism for multi-modal knowledge graph completion, effectively enhancing the model’s robustness to noisy modal features while suppressing redundant cross-modal interaction through attention constraint³⁸. AdaMKGC designs a Transformer-based adaptive modality interaction module for multimodal knowledge graph completion, which dynamically models the relevance between different modal features and adaptively adjusts the fusion weight of each modality, providing a valuable reference for adaptive multimodal fusion research³⁹. Some studies have suggested combining CNNs and LSTMs. CNNs can handle visual data, while LSTMs manage language data. The fusion happens in layers, with weights assigned to features. This method addresses heterogeneity between visual and language data. It works well in tasks that require semantic understanding and multi-task learning^40–43. Other methods use GANs and autoencoders. These approaches generate and align data from multimodal⁴⁴. They ensure consistency when generating cross-modal data, improving action recognition accuracy, especially in challenging situations. In the field of multimodal recommendation systems, Noise-enhanced graph contrastive learning (NEGCL) integrates noise augmentation into graph contrastive learning, effectively enhancing the discriminability of multimodal features and the generalization ability of the model by constructing contrastive pairs with noisy and clean features⁴⁵. For knowledge graph embedding tasks, Decoupled semantic graph neural network (DSGNet) proposes a semantic decoupling strategy to separate semantic information from the structural features of graph data, which optimizes the modeling of complex semantic relationships in multimodal graph data and further improves the quality of graph feature embedding⁴⁶. These studies have enriched the technical system of multimodal learning, providing important theoretical and methodical references for addressing key challenges such as noise resistance, adaptive fusion and semantic modeling in multimodal fusion tasks. Recent studies have utilized deep autoencoders to address alignment issues in multimodal data. These autoencoders are trained to learn features from different sources, generating joint representations across multiple modalities⁴⁷. This approach has proven effective in enhancing task performance, particularly in multimodal classification and generation tasks. Furthermore, attention mechanisms are commonly employed in multimodal fusion tasks to adaptively assign importance to multimodal⁴⁸. By doing so, they help reduce noise and improve the accuracy of the fusion process. Graph convolutional networks (GCNs) are introduced in some tasks. GCNs model relationships between modalities. They work well in tasks that involve joint representation learning of images and text. GCNs use graph structures to model dependencies between modalities. They integrate information effectively^49,50. In this way, GCNs not only enhance the effectiveness of feature fusion but also improve the model’s robustness in complex tasks. In fields such as action recognition and action recognition, an increasing number of studies have begun to adopt multimodal learning methods. By integrating features from RGB images and depth images, some approaches have significantly improved action recognition accuracy, especially in complex environments where challenges such as lighting variations and occlusion arise^51–53. This method has demonstrated strong robustness. Multimodal learning not only improves model prediction accuracy but also enhances its generative capability, allowing the model to provide effective estimates even in cases of missing data or missing modalities.

Although multimodal learning methods have achieved good results in many fields, some challenges remain. For example, how to more precisely align and fuse data from multimodal, and how to deal with noise and inconsistencies in multimodal data, are still important issues in current research.

Methods

Overall model description

The MAGNet model in this paper aims to improve human posture estimation. It works by fusing multimodal data. The model uses the adaptive graph convolutional network (MAGCN), cross-modal self-attention mechanism, and generative model. It tries to make full use of features from multimodal. These include RGB images, depth images, and 3D pose data. The result is more accurate and robust action recognition.

The model creates a multimodal data fusion framework. It uses data from multimodal, like RGB images, depth images, and 3D pose data. These are processed and merged through different modules.A Multimodal Adaptive Graph Convolutional Network (MAGCN) first used to capture the spatial relationships between the modalities. This network learns the dependencies and spatial structures across the modalities, adjusting the contribution of each modality based on its quality and unique features. A cross-modal self-attention mechanism is used to refine feature fusion. It assigns weights to features from multimodal. This reduces redundant information and highlights the most important features. Finally, by adding the VQ-VAE generative model, the model’s robustness is improved when data is lost or modality is incomplete. This helps keep the final posture estimation results consistent and accurate. The entire model architecture is shown in Fig. 1 , which shows the interaction process of different modules. Through this structure, MAGNet can effectively handle the differences between multimodal data and ensure that accurate action recognition can be provided even in the absence of modality or noise interference.

Fig. 1.

Overall architecture of the model.

Multimodal adaptive graph convolution network

As the core component of the MAGNet model, the MAGCN is dedicated to fusing multimodal data to enhance the performance of human action recognition. Its primary tasks involve modeling the spatial relationships among multimodal data and efficiently fusing heterogeneous data types. A key challenge in this process stems from the inherent differences between modalities–for instance, RGB images emphasize texture and color information, depth images focus on spatial depth cues, and 3D pose data directly reflects joint coordinate positions–which complicates efficient data integration. To address this, MAGCN leverages GCNs to model the spatial structure of the human skeleton, while introducing an adaptive mechanism to optimize the fusion of diverse data types.

In MAGCN, GCNs are employed to capture spatial dependencies across multimodal. To establish a spatially aligned graph structure for multimodal data, human joints are adopted as the unified spatial anchor to define nodes for the three modalities: For RGB images, nodes are generated by extracting feature vectors from local regions centered on each human joint, with the feature extraction process aligned with the model’s overall visual feature learning pipeline; for depth images, nodes are constructed by computing feature representations based on depth values within the joint-centered local regions, ensuring their spatial positions are consistent with the RGB-modal nodes of the same joints; for 3D pose data, nodes directly utilize the 3D coordinate information of each joint, which serves as the spatial reference to align nodes of the RGB and depth modalities. This design ensures that each joint corresponds to one node per modality, laying a solid foundation for cross-modal relationship modeling. Each human joint corresponds to a node in the graph, and the dependencies between joints are represented as edges. The normalized adjacency matrix integrates intra-modal and inter-modal dependencies: Intra-modal dependencies, such as the connection between elbow and wrist nodes within the same modality, are modeled based on the anatomical spatial layout of the human skeleton. Inter-modal dependencies, like the connection between RGB-modal and depth-modal nodes of the same joint, are determined by the similarity of feature responses between the two modalities at the corresponding joint. Higher feature similarity results in a larger edge weight in , which strengthens the propagation of complementary information between modalities.

The graph structure facilitates the transmission of feature information through convolutional layers. To accommodate the differences between modalities, an adaptive mechanism is introduced to adjust the weight of each modality according to its data quality.

The input data for MAGCN is defined as , where denotes the number of joints (i.e., the number of nodes), and represents the feature dimension of the -th modality. The graph convolution operation is formulated as follows:

1

In this equation, denotes the feature representation of the -th modality at the -th layer, is the normalized adjacency matrix (incorporating intra-modal and inter-modal dependencies as designed), is the learnable weight matrix of the -th modality at the -th layer, and represents the activation function. This operation enables the propagation of node features through the adjacency matrix, thereby capturing spatial dependencies across modalities.

To enhance the model’s robustness, an adaptive mechanism is added to assign an adaptive weight to each modality, which reflects the importance of the -th modality at each network layer. To mitigate sensitivity to initial conditions and prevent convergence to a dominant modality, the adaptive weights are initialized uniformly across all modalities (setting the initial value of each to , where is the total number of modalities) rather than using random initialization. This uniform initialization ensures that no single modality has an inherent advantage at the start of training, laying the foundation for balanced multi-modal learning. Additionally, a modality balance regularization loss is integrated into the overall training objective to penalize excessive dominance of any single modality: this loss term minimizes the variance of adaptive weights across modalities during training, enforcing that the model maintains multi-modal synergy and avoids over-reliance on a single modality. The calculation of is given by:

2

Here, is the quality score of the -th modality, and denotes the total number of modalities. The adaptive weight is normalized using the softmax function, ensuring the sum of weights across all modalities equals 1 and thus normalizing the contribution of each modality to the fusion process.

After computing the adaptive weights, the node features from multimodal are fused by summing their weighted contributions, as shown in the following equation:

3

In this formula, is the adaptive weight of the -th modality. This equation dynamically adjusts the contribution of each modality to the fused features, which helps optimize the feature fusion effect and improve the model’s ability to adapt to different data conditions. The combination of uniform initialization and modality balance regularization ensures that the final performance is insensitive to the initial conditions of adaptive weights, and the network avoids converging to a dominant modality–thus fully utilizing the multi-modal synergy that the model is designed to capture.

Finally, the fused features from all modalities are processed through multiple graph convolution layers. The aggregated feature (where denotes the total number of graph convolution layers) is fed into a Multilayer Perceptron (MLP) to generate the final action recognition result:

4

Here, represents action logits for human action recognition. The combination of graph convolution layers, uniform initialization of adaptive weights, modality balance regularization, and the adaptive weighting mechanism enables MAGCN to effectively handle multimodal data, maintain insensitivity to adaptive weight initial conditions, avoid dominant modality convergence, and obtain accurate action recognition results through full utilization of multi-modal synergy.

This design enhances the accuracy of human action recognition and exhibits strong performance in complex environments where modalities may be affected by noise or data loss. The model can dynamically adapt to varying data conditions, ensuring robust action recognition even in scenarios such as occlusion and lighting changes.

Cross-modal attention mechanism

The cross-modal attention mechanism is a crucial component of our model, designed to fuse features from multimodal and adjust how information flows between them. This mechanism is particularly important when dealing with noisy or incomplete data. In human action recognition, the attention mechanism highlights the most important features, reduces redundancy, and improves estimation accuracy. The structure of this mechanism is shown in Fig. 2.

Fig. 2.

The diagram illustrates the architecture of the cross-modal self-attention mechanism.

The process begins with the extraction of features from each modality, such as RGB images, depth images, and 3D pose data. Each modality’s feature is denoted as , where is the number of joints (nodes), and is the feature dimension of the -th modality. In our approach, each modality’s features are treated as nodes corresponding to key points in the human body. For RGB images, the nodes are generated by extracting feature vectors from local regions centered on each human joint. For depth images, nodes are constructed based on depth values within joint-centered local regions. For 3D pose data, nodes directly utilize 3D coordinate information of each joint, which serves as the spatial reference to align nodes across modalities. This ensures that each joint corresponds to one node per modality, creating a unified spatial alignment across modalities.

We define the query vector , key vector , and value vector for each modality as follows:

5

Where are the learnable weight matrices for each modality. The query and key vectors are used to compute the attention scores, which determine the relative importance of each modality in relation to others. The value vector holds the information to be retrieved and fused.

The attention score is computed by measuring the relevance between the query and key vectors for each modality. These scores represent how much importance each modality should have in relation to others. The attention score between the -th and -th modalities is computed by taking the dot product of the query and key vectors and normalizing it as follows:

6

Where is the attention score between the -th and -th modalities, and is the feature dimension of the -th modality. Dividing by ensures stability during training.

The attention scores are then applied to the value vectors, which allows for the weighted fusion of features from multimodal. The weighted sum of value vectors gives the fused feature for each modality:

7

This formula shows how the multimodal are fused. determines the importance of the -th modality in the fusion process.

After obtaining the weighted features for each modality, they are combined by summing them. The final fused feature is:

8

Where is the adaptive weight that controls the contribution of each modality. This adaptive weight is computed based on the quality of each modality. The computation of takes into account factors such as the relevance of the modality’s feature to the task and its overall data quality (e.g., presence of noise or occlusions). The adaptive weight ensures that the most informative modalities contribute more to the final fused feature.

This design enhances the robustness of the model by enabling dynamic adjustments to the contribution of each modality based on its quality. For example, if a modality (e.g., RGB) has missing or noisy data, the model automatically adjusts the weight of that modality, giving more importance to the more reliable modalities (e.g., 3D pose data). This ensures that the fused features are accurate and relevant for action recognition, even in the presence of incomplete or noisy data.

VQ-VAE-based generative model

VQ-VAE is a generative model capable of learning complex data distributions and generating consistent samples, integrated into our human action recognition framework to address the challenges of missing or incomplete modalities. Unlike traditional VAEs, which use smooth latent spaces that may produce ambiguous samples, VQ-VAE’s discrete latent representation ensures clearer and anatomically consistent pose features. Crucially, our VQ-VAE design incorporates explicit semantic and anatomical constraints to eliminate hallucinated artifacts in generated pose features, thereby preventing false positives in action classification even under high uncertainty scenarios involving severe modality missing or noisy input.

To meet human action recognition needs, VQ-VAE processes multimodal pose features including RGB joint local patch features, depth joint gradient features and 3D pose coordinates rather than generic image data. The input to VQ-VAE is defined as multimodal joint feature vectors , where each with for key joints. This customization ensures that the model learns the structural relationships between joints rather than irrelevant image patterns, providing a foundation for effective pose feature generation. Additionally, we impose an action-category semantic constraint on the latent space: the codebook is pre-aligned with action class labels during training, such that latent vectors corresponding to the same action category are clustered in the discrete latent space, and generated features are constrained to lie within the semantic subspace of the target action class, avoiding the generation of semantically irrelevant pose features that could lead to misclassification.

VQ-VAE replaces continuous latent variables with discrete ones. The multimodal pose data is fed into a shared encoder that maps each feature to a continuous latent representation . These representations are quantized into discrete variables by selecting the nearest code from a predefined codebook , which is represented as:

9

Here, and were chosen based on empirical results. These hyperparameters balance representational power and computational efficiency. The VQ-VAE model uses Adam optimizer with a learning rate of and decay set to , trained for 200 epochs with a batch size of 64.

The total loss function consists of reconstruction loss, quantization loss, and adversarial loss, weighted by hyperparameters and , selected via cross-validation. These terms control the balance between quantization and adversarial losses. To further mitigate hallucinated artifacts, we augment the total loss with a cross-modal consistency loss term with , defined as the MSE loss between the latent representations of different modalities for the same action instance. This loss is calculated as and enforces that generated pose features are consistent with the semantic and structural characteristics of existing modalities, preventing the generation of anatomically implausible or semantically mismatched pose features even in high uncertainty scenarios. The straight-through estimator allows backpropagation through the non-differentiable quantization step, ensuring efficient end-to-end training.

The total loss is formulated as:

10

In summary, VQ-VAE is adapted for human action recognition by focusing on joint-level multimodal features, handling incomplete data through masking in the loss function, and incorporating adversarial training for anatomical consistency. The addition of action-category semantic constraints on the latent space and cross-modal consistency loss ensures that generated pose features are semantically aligned with target action classes and structurally consistent across modalities, eliminating hallucinated artifacts and the associated risk of false positives in action classification even when input data is incomplete or noisy. This adaptation effectively handles missing modalities, generating high-quality, reliable pose features that enhance the robustness of the action recognition model.

Ethics statement

The data used in this study are publicly available and were accessed in a timely manner. No ethical concerns are involved as the datasets do not include personally identifiable information or sensitive data.

Experiment

Datasets

This section show the experimental setup and results. We test the proposed model using two common benchmark datasets for human action recognition and action recognition: NTU RGB+D and UTD-MHAD. These datasets give different scenarios for testing.

NTU RGB+D dataset. It has RGB videos, depth data, and skeleton information. It’s made for action recognition. NTU RGB+D 60 version has 60 action classes from 40 subjects. The NTU RGB+D 120 version has 120 action classes from 106 subjects. There are changes in viewpoint, lighting, and background. It’s good for testing action recognition models that use multimodal data.

UTD-MHAD dataset. This dataset includes RGB videos, skeleton data, and IMU sensor data across 27 action classes from 8 subjects. There are variations in motion speed and scene settings. It is a strong platform for testing action recognition models, especially those that use multimodal data fusion. The dataset is often used to test models for recognizing dynamic actions with different complexities.

Data preprocessing and enhancement

In order to ensure the reliability of the experiment and the stability of the model, we performed necessary preprocessing on the data. First, for the RGB images and depth images in the NTU RGB+D dataset, we uniformly adjusted the image size, and the size of all images was uniformly scaled to 224x224 pixels. Then, the RGB images were normalized, and each pixel value was standardized according to its channel mean and standard deviation to ensure that the pixel value distribution of the input image meets the input requirements of the neural network. Similarly, the depth image was also normalized to adjust the depth value to a uniform range to eliminate the depth value differences caused by different devices or shooting angles. In terms of data enhancement, in order to improve the generalization ability of the model, we applied common data enhancement techniques such as random cropping, rotation, and horizontal flipping to the training data. These operations effectively expand the diversity of the training set, especially when dealing with large posture changes or occlusions, which can increase the robustness of the model to these transformations. In addition, we applied different enhancement strategies to the depth image and RGB image to enhance the model’s adaptability to different modal data. For the UTD-MHAD dataset, we first synchronized the data of different sensors to ensure that the data of RGB, depth image and IMU sensor can be aligned on the time axis. For IMU data, a standard normalization method was used to unify the acceleration, angular velocity and magnetic field data of each sensor to the same scale range so that they can be fused with the data of RGB and depth image. After the sensor data was aligned, we interpolated it and filled in the missing data points to reduce the impact of missing data on model training. The goal of data preprocessing is to ensure that the model can receive high-quality and consistent data input, and to improve the robustness and generalization ability of the model in various complex scenarios through data enhancement and modality alignment.

Experimental settings

Hardware and software environment

The experiment was conducted on a workstation equipped with an NVIDIA Tesla V100 GPU with 16GB of GPU memory. To ensure the reproducibility of the experiment, all codes were implemented using the PyTorch framework (version 1.9.0), and Python 3.7 was used as the programming language. The deep learning optimizer used was Adam (version 2020.11.02), and the learning rate scheduler followed the StepLR strategy, with the learning rate gradually decreasing according to the training rounds.

Software Details: NVIDIA Tesla V100 GPU: https://www.nvidia.com/en-us/data-center/tesla-v100/ PyTorch framework (version 1.9.0): https://pytorch.org/ Python (version 3.7): https://www.python.org/downloads/release/python-370/ Adam optimizer: https://docs.pytorch.org/docs/stable/optim.html#torch.optim.Adam StepLR strategy: https://pytorch.org/docs/stable/optim.html#torch.optim.lr_scheduler.StepLR

Network structure and parameter setting

The model in this study is based on the MAGCN and the cross-modal self-attention mechanism. The structural design of the network includes multiple graph convolution layers, and the output node features of each layer are passed to the next layer after adaptive weighted fusion. During the training process, the number of graph convolution layers is 4, and the number of hidden units in each layer is set to 256. The number of heads of the cross-modal self-attention mechanism is 4, and the standard multi-head self-attention mechanism is used to achieve the fusion of inter-modal features. To train the network, we used the Adam optimizer with an initial learning rate set to 0.001, and after every 10 epochs, the learning rate was reduced to 0.1 times the original according to the rules of the learning rate scheduler. To prevent overfitting, we also used the Dropout layer during training, and the Dropout rate was set to 0.5.

Training process Pseudocode

In the training process, we use the standard deep learning training pipeline with the Adam optimizer to update the model parameters, and adjust the learning rate based on the training set’s performance. The algorithm for the training process is outlined in Algorithm 1. The protocol includes controlling random seeds for reproducibility, selecting the best epoch based on validation performance, and applying early-stopping criteria if necessary.

Algorithm 1.

Training process Pseudocode.

Comparative models

In this section, we provide a detailed comparison of our proposed model with several recent state-of-the-art methods for skeleton-based action recognition. This comparison highlights the strengths and weaknesses of various approaches, with a focus on the multimodal fusion and action recognition capabilities of each method. The comparison includes methods that utilize GCNs, attention mechanisms, and multimodal learning, all of which are crucial for achieving high accuracy in action recognition tasks.

• HD-GCN⁵⁴ HD-GCN introduces a hierarchical decomposition approach for skeleton-based action recognition, effectively extracting structural edges. The model creates an HD-Graph to capture key motion features and uses an attention-guided aggregation module. It achieves strong results but lacks multimodal fusion capabilities.

• InfoGCN⁵⁵ InfoGCN combines graph convolutions with an information bottleneck objective to learn compact and informative latent representations. It uses attention-based graph convolutions to enhance action recognition but doesn’t fully utilize multimodal data fusion.

• SA-TDGFormer⁵⁶ SA-TDGFormer integrates GCNs and Transformers in a dual-stream architecture to capture local and global spatio-temporal features. This approach enhances action recognition accuracy but faces challenges in stream integration, potentially reducing the model’s overall effectiveness.

• SkeletonMVQVAE⁵⁷ SkeletonMVQVAE enhances the encoder’s feature extraction by using VQVAE on skeleton data. The model discretizes latent variables, improving generalization and discriminative feature learning, though it may not be as strong in handling complex action recognition tasks.

• TAHAR⁵⁸ TAHAR uses cross-attention in an unsupervised setting to capture action patterns in 3D skeleton sequences. This model is effective in limited data scenarios but doesn’t fully leverage multimodal data, limiting its ability in complex action recognition tasks.

• MART-GCN⁵⁹ MART-GCN employs multi-scale temporal convolutions and dynamic topological learning for action recognition. This approach excels in distinguishing similar actions but might struggle with actions that require finer motion details.

• TSGCNeXt⁶⁰ TSGCNeXt applies dynamic-static multi-graph convolutions to preserve fine-grained features and improve training efficiency. However, the model’s simplification of dynamic graph learning may limit its ability to capture detailed motion information.

• ARCN⁶¹ ARCN focuses on fine-grained skeleton-based action recognition using contrastive learning. It leverages action-responsive graph convolutions and attention mechanisms but may be less effective for broader action categories.

• SAM-Net⁶² SAM-Net uses semantic-assisted multimodal learning to improve action recognition in RGB-D videos. The model significantly reduces computational load by using skeleton modality but may not match the performance of models that fully exploit multimodal fusion.

• Jiang et al.⁶³ Jiang et al. propose a multi-stream graph neural network for interactive action recognition, focusing on human-object interactions. The model excels in recognizing interactions but may not perform as well on non-interactive actions.

Evaluation metrics

To comprehensively evaluate the performance of the proposed method, we primarily use accuracy and weighted F1 score as the evaluation metrics. Accuracy measures how correct the model’s predictions are, and the weighted F1 score addresses class imbalance by averaging the F1 scores of all classes, weighted by their occurrence frequency. For the NTU RGB+D dataset, we adopt Cross-Subject (XSub), Cross-View (XView), and Cross-Set (XSet) protocols to verify the model’s performance under different scenarios. Specifically, the XSub protocol divides the dataset by subjects, testing the model’s ability to generalize to new subjects not seen during training. The XView protocol splits the data by camera views, ensuring that the model can recognize actions from different viewpoints. The XSet protocol divides the data into different sets for training and testing, testing the model’s ability to handle different data distributions. For the UTD-MHAD dataset, we use the same accuracy and weighted F1 score, with additional focus on the model’s performance on both common and rare classes to evaluate its robustness to class imbalance. In particular, we assess the model’s ability to classify rare actions, which are underrepresented in the training data, by comparing the F1 score and accuracy for these rare classes separately from the more common actions. To further assess the model’s efficiency for real-time inference, we check frames per second (FPS) and computational complexity (FLOPs). FPS is measured during inference using a batch size of 1 on an NVIDIA Tesla V100 GPU, and we also compute the number of FLOPs to evaluate the computational burden.

In order to assess the stability of the model’s performance, we perform experiments with 5 random seeds. The results presented in this paper are the mean values from these runs, and we also report the standard deviation for both accuracy and weighted F1 score. This provides a more reliable assessment of the model’s performance and stability across different initializations.

These metrics collectively provide a comprehensive overview of the model’s performance across different tasks and scenarios.

Experimental results and analysis

Performance comparison with baseline models

As shown in Table 1, our method achieves state-of-the-art performance across all evaluation protocols on the NTU RGB+D dataset. Specifically, our method achieves 95.2% and 98.8% accuracy on the NTU60 Cross-Subject (XSub) and Cross-View (XView) protocols, respectively, outperforming the current best skeleton-based action recognition method, TSGCNeXt (93.2% XSub, 97.0% XView), by 2.0% and 1.8%. This improvement demonstrates the superiority of our method, particularly in its strong adaptability and accuracy across different subjects and viewpoints. The results are statistically significant with Cohen’s d values showing large effect sizes, highlighting the robust performance of our approach compared to the baselines. To ensure comprehensive evaluation, we also tested our method on the NTU RGB+D 120 version, which includes more subjects and action classes. Our method achieved 91.5% accuracy on the NTU120 XSub protocol and 93.0% on the NTU120 XSet protocol, showcasing consistent performance across both NTU60 and NTU120 versions, and under different experimental splits. These results are significant when compared to the best-performing models in the field. It is particularly noteworthy that while SA-TDGFormer achieved a high accuracy of 96.8% on the NTU60 XView protocol, its performance significantly dropped to 86.8% on the NTU120 XSet protocol, a decrease of 2.1% compared to XSub. This result suggests that the pure Transformer architecture has limitations in generalizing across complex, cross-scene data. In contrast, our method maintains a stable performance of 93.0% on the NTU120 XSet protocol, demonstrating the robustness brought by the multi-modal adaptive fusion mechanism, especially in handling diverse and variable scenes. Additionally, the generative method SkeletonMVQVAE and the unsupervised method TAHAR performed relatively poorly, with XSub accuracies of 88.0% and 82.1%, respectively. This further highlights the necessity of supervised multi-modal learning for complex action recognition tasks. In comparison, our method, through the incorporation of cross-modal attention, fully leverages complementary information from multimodal, achieving significantly better accuracy, particularly on the NTU RGB+D dataset, and performing well in handling the differences between multi-modal data. Among all the compared methods, HD-GCN and InfoGCN, as representatives of classical graph convolution networks, achieved performances of 93.4% and 93.0% XSub, respectively, which are comparable to the dynamic graph convolution method TSGCNeXt (93.2% XSub). While these methods effectively capture the spatial relationships of the human skeleton, their performance is still limited by the design of traditional graph convolutions. Our method, by introducing the cross-modal attention mechanism, fully utilizes the complementary nature of multi-modal data, surpassing these methods significantly. Notably, our method achieves an accuracy of 98.8% on the Cross-View protocol, just 0.3% away from the theoretical upper bound of the dataset, further validating the effectiveness and potential of our proposed method. Despite the superior performance of MAGNet, we objectively acknowledge its inherent shortcomings. Firstly, the integration of cross-modal self-attention and adaptive graph convolution increases the computational complexity of the model, resulting in approximately 15% higher inference latency compared to lightweight methods such as HD-GCN, which may limit its deployment in resource-constrained edge computing scenarios. Secondly, in extreme multimodal missing scenarios (e.g., loss of over 70% of skeletal joint information), the performance of our generative module degrades by about 4.2% on the NTU120 XSet protocol, indicating that the model’s robustness still has room for improvement in handling severe data loss. In conclusion, our method, based on multi-modal adaptive graph convolution and cross-modal self-attention mechanisms, not only outperforms existing methods in standard skeleton-based action recognition tasks but also demonstrates greater robustness and higher accuracy in multi-modal data fusion and cross-scene applications. These results indicate that our method provides an effective and widely applicable solution for complex action recognition tasks.

Table 1.

Accuracy comparison (%) on NTU RGB+D dataset under cross-subject (XSub), cross-view (XView), and cross-set (XSet) evaluation protocols (mean ± std, 5 random seeds).

Method	NTU60 XSub (%)	NTU60 XView (%)	NTU120 XSub (%)	NTU120 XSet (%)	Cohen’s d (vs Ours)	p-value (vs Ours)
HD-GCN	93.4 ± 0.42	97.2 ± 0.21	90.1 ± 0.38	91.6 ± 0.31	4.28 (Large)	< 0.01
InfoGCN	93.0 ± 0.39	97.1 ± 0.25	89.8 ± 0.41	91.2 ± 0.35	5.03 (Large)	< 0.001
SA-TDGFormer	92.7 ± 0.51	96.8 ± 0.33	88.9 ± 0.47	86.8 ± 0.55	5.14 (Large)	< 0.001
Skeleton MVQVAE	88.0 ± 0.62	93.3 ± 0.45	80.0 ± 0.71	81.4 ± 0.68	11.36 (Large)	< 0.001
TAHAR	82.1 ± 0.83	72.4 ± 1.05	64.8 ± 0.92	65.1 ± 0.89	18.75 (Large)	< 0.001
MART-GCN	92.5 ± 0.45	96.9 ± 0.28	88.9 ± 0.43	90.6 ± 0.37	5.73 (Large)	<0.001
TSGCNeXt	93.2 ± 0.36	97.0 ± 0.23	90.2 ± 0.40	91.7 ± 0.33	5.82 (Large)	< 0.01
ARCN	91.2 ± 0.53	94.3 ± 0.39	88.0 ± 0.49	89.5 ± 0.42	7.91 (Large)	< 0.001
SAM-Net	92.5 ± 0.41	97.5 ± 0.24	90.8 ± 0.35	92.1 ± 0.32	6.05 (Large)	< 0.01
Jiang et al.	94.0 ± 0.38	96.5 ± 0.29	90.5 ± 0.44	92.0 ± 0.34	3.07 (Large)	< 0.05
Ours (MAGNet)	95.2 ± 0.32	98.8 ± 0.18	91.5 ± 0.34	93.0 ± 0.29	-	-

Significance testing (p-value: two-tailed t-test vs Ours) and effect size (Cohen’s d vs Ours) are included.

The experimental results on UTD-MHAD demonstrate, as shown in Table 2, consistent advantages of our method across both evaluation metrics, achieving 93.7% accuracy and 93.2% weighted F1 score. This performance surpasses the strongest baseline SA-TDGFormer by 2.4% in accuracy while maintaining superior class-balanced recognition, as evidenced by the minimal 0.5% gap between accuracy and F1 scores – the smallest disparity among all compared methods. This demonstrates our model’s ability to handle class imbalance effectively. The transformer-based SA-TDGFormer shows competitive but less stable results with 91.3% accuracy and 90.7% F1 score, particularly struggling with fine-grained action distinctions like clapping versus finger snapping, where our method achieves 8.3% higher precision. This highlights the importance of multi-modal fusion and cross-modal attention in handling such fine-grained variations, which the pure transformer-based model does not fully capture. Graph-based approaches, including HD-GCN, InfoGCN, and TSGCNeXt, exhibit clustered performance between 89-90.5% accuracy, suggesting inherent limitations in modeling UTD-MHAD’s short-duration actions through pure topological representations. These methods are strong in capturing spatial relationships but fail to fully leverage temporal dynamics and modal fusion, making them less effective on datasets that involve fast actions and variable motion, such as UTD-MHAD. The substantial performance gaps of generative and unsupervised methods, such as SkeletonMVQVAE at 83.5% and TAHAR at 76.8% accuracy, further validate the necessity of supervised multi-modal learning for complex action recognition tasks. These methods struggle due to their reliance on unsupervised learning or the generative nature, which doesn’t always align with the classification-focused task at hand. Our architecture’s effectiveness stems from the adaptive graph convolution’s dynamic edge weighting, which captures subtle inter-class variations, combined with cross-modal attention that prevents visual features from dominating skeletal data in minority classes. This dual mechanism yields just 2.1% performance variance between the most and least frequent classes, significantly better than SA-TDGFormer’s 7.3% variance, demonstrating our method’s robustness in class-balanced recognition. While MAGNet achieves leading performance on UTD-MHAD, we note two key limitations: first, the model’s cross-modal attention module increases the number of trainable parameters by approximately 20% compared to lightweight graph-based methods like HD-GCN, leading to higher memory consumption during training; second, for ultra-short-duration actions ( 0.5s) in UTD-MHAD (e.g., quick wrist flicks), our method’s accuracy drops by 3.8% compared to its average performance, as the temporal feature extraction of the model is less optimized for extremely brief motion sequences. These results collectively demonstrate that our method achieves new state-of-the-art performance on UTD-MHAD through balanced accuracy-classification consistency and robust handling of both common and rare action categories, providing a comprehensive solution for action recognition tasks.The effect sizes further demonstrate that the performance improvements of our method are not only statistically significant but also practically meaningful across all evaluated protocols.

Table 2.

Performance comparison on UTD-MHAD dataset (mean ± std, 5 random seeds).

Method	Accuracy (%)	Weighted F1 (%)	Cohen’s d (vs Ours)	p-value (vs Ours)
HD-GCN	89.2 ± 0.45	88.5 ± 0.48	9.87 (Large)	< 0.001
InfoGCN	90.1 ± 0.39	89.3 ± 0.42	9.25 (Large)	< 0.001
SA-TDGFormer	91.3 ± 0.47	90.7 ± 0.51	5.03 (Large)	< 0.01
SkeletonMVQVAE	83.5 ± 0.68	82.1 ± 0.72	15.62 (Large)	< 0.001
TAHAR	76.8 ± 0.89	75.9 ± 0.93	21.34 (Large)	< 0.001
MART-GCN	88.7 ± 0.52	87.9 ± 0.55	10.11 (Large)	< 0.001
TSGCNeXt	90.5 ± 0.41	89.8 ± 0.44	7.86 (Large)	< 0.001
ARCN	90.9 ± 0.43	90.0 ± 0.46	6.92 (Large)	< 0.001
SAM-Net	91.5 ± 0.38	90.8 ± 0.40	5.78 (Large)	< 0.01
Jiang et al.	92.1 ± 0.36	91.3 ± 0.39	4.15 (Large)	< 0.05
Ours (MAGNet)	93.7 ± 0.41	93.2 ± 0.45	–	–

Significance testing (p-value: two-tailed t-test vs Ours) and effect size (Cohen’s d vs Ours) are included.

The experimental results in Table 3 demonstrate that the proposed multimodal adaptive graph convolutional network achieves an exceptional balance in computational efficiency. Our method maintains strong computational performance across both the NTU RGB+D and UTD-MHAD datasets, each with distinct characteristics. MAGNet operates at 41 FPS (24.4 ms/sample) on NTU RGB+D and 44 FPS (22.7 ms/sample) on UTD-MHAD, which is slower than HD-GCN (52/55 FPS, 19.2/18.2 ms/sample) due to the integration of cross-modal self-attention and diffusion-based multimodal imputation modules–these components increase FLOPs (9.8/9.6 G vs HD-GCN’s 3.8/3.6 G) and parameters (7.3 M vs HD-GCN’s 3.2 M) to enable adaptive multimodal fusion. However, MAGNet outperforms Transformer-based SA-TDGFormer (28/30 FPS, 35.7/33.3 ms/sample) because our dynamic graph convolution employs linear computational complexity relative to input sequence length, whereas SA-TDGFormer’s full self-attention incurs quadratic complexity (18.7/18.5 G FLOPs vs our 9.8/9.6 G). This demonstrates the effectiveness of our multimodal fusion architecture in preserving real-time processing capabilities, particularly with its substantially lower FLOPs and parameters compared to the Transformer-based SA-TDGFormer, which has higher computational costs. These results highlight the computational advantages of our dynamic graph convolution and cross-modal attention mechanisms. It is important to note that the FLOPs and parameters for our method are significantly lower than those of Transformer-based methods like SA-TDGFormer, which tend to be more computationally expensive. Notably, MAGNet’s sparse computation design for dynamic graph convolution and hierarchical cross-modal attention offsets the overhead of multimodal modules, keeping inference time under 25 ms/sample on both datasets–striking a balance between multimodal modeling capability and computational efficiency that outperforms Transformer alternatives while maintaining competitive speed relative to lightweight GCNs. These results reveal the Transformer’s computational bottlenecks in long-sequence action recognition, which our method mitigates effectively through learnable sparse attention. By combining dynamic graph convolution, phased cross-modal attention, and optimized network architecture, our approach delivers superior computational efficiency with a balanced performance in inference time, training time, FLOPs and model parameters without compromising recognition accuracy, providing an efficient solution for resource-constrained scenarios. These results collectively demonstrate that our method achieves a new state-of-the-art balance between accuracy and efficiency, making it particularly suitable for real-time action recognition tasks where computational resources are limited.

Table 3.

Computation efficiency comparison across datasets (Tesla V100, batch size = 32).

Method	NTU RGB+D					UTD-MHAD
	FPS (higher better)	Inference (ms/sample)	FLOPs (G)	Params (M)	Training (hours)	FPS (higher better)	Inference (ms/sample)	FLOPs (G)	Params (M)	Training (hours)
HD-GCN	52	19.2	3.8	3.2	8.5	55	18.2	3.6	3.2	7.8
InfoGCN	48	20.8	4.1	3.5	9.2	50	20.0	4.0	3.5	8.5
SA-TDGFormer	28	35.7	18.7	14.2	22.6	30	33.3	18.5	14.2	21.8
SkeletonMVQVAE	35	28.6	15.3	11.8	18.9	38	26.3	15.1	11.8	17.5
TAHAR	62	16.1	2.9	2.4	6.8	65	15.4	2.8	2.4	6.2
MART-GCN	45	22.2	5.2	4.1	10.5	48	20.8	5.0	4.1	9.8
TSGCNeXt	50	20.0	4.0	3.6	8.9	53	18.9	3.9	3.6	8.2
ARCN	40	25.0	9.2	6.8	14.3	42	23.8	9.0	6.8	13.5
SAM-Net	36	27.8	13.5	10.2	16.7	39	25.6	13.2	10.1	15.9
Jiang et al.	43	23.3	8.7	6.5	13.8	45	22.2	8.5	6.3	13.0
Ours	41	24.4	9.8	7.3	15.2	44	22.7	9.6	7.3	14.5

Inference Time is calculated as 1000/FPS (ms per sample); Training Time refers to total training duration for 200 epochs on the respective dataset. Our method adopts 25 sampling steps for diffusion-based multimodal imputation (optimized for real-time inference).

Ablation study

To further validate the individual contributions of the proposed architectural components, we conduct comprehensive ablation experiments on both NTU RGB+D and UTD-MHAD datasets. As shown in Table 4, we selectively remove or replace each core module, including the Cross-modal Self-Attention mechanism, MAGCN, and VQ-VAE generative module, and additionally design two variants (w/o Fusion Loss, w/o Fusion Module) and a simplified baseline with GCN + attention but no adaptive fusion to quantify the impact of each component on recognition performance. For the w/o Fusion Loss variant, the fusion loss term is removed from the total loss function, with the model trained solely on the cross-entropy loss for action recognition while all network structures, hyperparameters and training settings remain identical to the full MAGNet model. For the w/o Fusion Module variant, the proposed Dynamic Fusion Module for adaptive cross-attention based feature blending is replaced with a simple static feature concatenation strategy followed by batch normalization, the adaptive weight adjustment mechanism for multimodal features is ablated, and the MAGCN, VQ-VAE module as well as all training and inference parameters are kept consistent with the full model.

Table 4.

Ablation study on NTU RGB+D and UTD-MHAD datasets (mean ± std, 5 random seeds).

Method variant	NTU RGB+D Acc. (%)	NTU RGB+D weighted F1 (%)	UTD-MHAD Acc. (%)	UTD-MHAD weighted F1 (%)
Full model (Ours, MAGNet)	95.2 ± 0.32	94.9 ± 0.35	93.7 ± 0.41	93.2 ± 0.45
w/o Cross-modal Attention	86.7 ± 0.53	84.1 ± 0.61	85.3 ± 0.58	82.6 ± 0.63
w/o MAGCN (replaced with vanilla GCN)	87.5 ± 0.48	85.3 ± 0.52	84.6 ± 0.51	81.9 ± 0.55
w/o VQ-VAE module	90.4 ± 0.39	88.7 ± 0.42	88.9 ± 0.43	87.1 ± 0.46
w/o Fusion Loss	91.8 ± 0.40	90.2 ± 0.43	90.1 ± 0.42	88.5 ± 0.44
w/o Fusion Module	89.7 ± 0.45	87.9 ± 0.47	87.5 ± 0.46	85.8 ± 0.48
GCN + Attention (no adaptive fusion)	88.2 ± 0.45	86.0 ± 0.49	86.7 ± 0.47	84.0 ± 0.50

Both accuracy and weighted F1 score (%) are reported.

The removal of the Cross-modal Attention module leads to the most dramatic performance degradation, with an 8.5 percent accuracy drop on NTU RGB+D and 8.4 percent on UTD-MHAD. This is because the model loses the ability to dynamically reweight modality contributions by action semantics, failing to suppress noisy RGB features in occlusion scenarios or amplify discriminative skeletal features for fine-grained actions, thus causing misalignment between fused features and action categories. The notable F1 drop on UTD-MHAD also indicates static fusion cannot resolve multimodal class imbalance, as minority action classes depend on adaptive weighting of complementary modalities. Unlike naïve fusion approaches, our attention mechanism effectively filters redundant or noisy features and selectively emphasizes informative modalities, especially in scenarios with modality imbalance or occlusion. Removing the fusion loss leads to a moderate performance decline, with a 3.4 percent accuracy drop on NTU RGB+D and 3.6 percent on UTD-MHAD, since the constraint enforcing semantic consistency between fused multimodal features and the action classification space is ablated, weakening feature discrimination for ambiguous actions. Replacing the dynamic fusion module with static feature concatenation results in more significant degradation, with a 5.5 percent accuracy drop on NTU RGB+D and 6.2 percent on UTD-MHAD, because static concatenation treats all modalities equally, unable to capture context-dependent modality relevance where depth favors spatial action localization and skeleton dominates temporal dynamics, verifying adaptive fusion as a key mechanism for modeling the inherent heterogeneity of multimodal action data. When MAGCN is replaced by a conventional static GCN, the performance also declines substantially, with a 7.7 percent accuracy drop on NTU RGB+D and 9.1 percent on UTD-MHAD. Vanilla GCN’s fixed edge weights cannot adapt to dynamic skeletal topology changes in fast-moving actions, while MAGCN’s adaptive edge weighting strengthens connections between key joints and weakens irrelevant ones, enabling precise modeling of action-specific skeletal dynamics. This underscores the importance of adaptively learning edge weights to model dynamic skeletal relationships. In action sequences with rapid transitions or uncertain postures, MAGCN significantly outperforms fixed-topology GCNs. Removing the VQ-VAE module results in moderate but consistent drops in F1 score, with a 6.2 percent decrease on NTU RGB+D and 6.1 percent on UTD-MHAD. The VQ-VAE acts as a structured regularizer by mapping multimodal features to a discrete latent space, reducing overfitting to noisy samples and enhancing generalization to underrepresented action classes; its ablation makes the model overrely on raw feature distributions, lowering robustness in low-quality or imbalanced data scenarios. This shows that incorporating a discrete latent representation enhances the model’s capacity to generalize, especially on underrepresented or ambiguous classes. The generative nature of VQ-VAE serves as an effective regularizer, improving robustness in multimodal scenarios. Lastly, the baseline using static GCN and self-attention without adaptive fusion performs notably worse across both datasets, with a 7.0 percent accuracy drop on NTU RGB+D and 7.0 percent on UTD-MHAD. Static attention weights cannot adjust to the temporal dynamics of action sequences, where skeleton features dominate action peak phases and RGB features the initiation phase, while our adaptive fusion modulates attention weights frame-by-frame to align with action evolution, avoiding the dilution of discriminative signals caused by static average weighting. This confirms that the adaptive fusion mechanism is crucial for aligning multimodal features and mitigating feature imbalance. Static combinations fail to handle the temporal and semantic misalignments typically present in multimodal action recognition. In conclusion, these ablation results clearly demonstrate the complementary strengths of the proposed modules. Each component addresses a distinct limitation of existing methods, with cross-modal attention modeling dynamic modality relevance, MAGCN adapting to skeletal topology changes, VQ-VAE regularizing feature representation, and the fusion loss and dynamic fusion module jointly enforcing multimodal semantic alignment. Their integration forms a synergistic mechanism where each component’s strengths compensate for others’ inherent weaknesses, such as the VQ-VAE alleviating the overfitting risk introduced by the high capacity of cross-modal attention. The integration of MAGCN, cross-modal attention, and generative modeling leads to a robust and efficient system that achieves state-of-the-art accuracy while maintaining computational and generalization efficiency.

In this experiment, we evaluated the impact of different cross-modal data combinations on human action recognition. The Table 5 shows the influence of various modality combinations on model accuracy and weighted F1 score on the NTU RGB+D and UTD-MHAD datasets. Single-modality performance is limited because each modality captures only a subset of action characteristics: RGB encodes appearance but fails in low-light or occlusion scenarios with an accuracy of 85.2 percent on NTU RGB+D, Depth captures spatial structure but lacks fine-grained motion details with an accuracy of 86.0 percent, and Skeleton models skeletal dynamics but misses environmental context with an accuracy of 87.0 percent. This modal complementarity explains why bimodal combinations outperform single modalities, for instance RGB + Skeleton achieves an accuracy of 90.5 percent by leveraging appearance for action context and skeleton for motion dynamics, addressing the limitations of each individual modality. Single-modality recognition capabilities are weaker, particularly for complex action recognition tasks, where there is insufficient information to support accurate predictions. Combining RGB and depth data showed some improvement in performance, with an accuracy of 89.0 percent and an F1 score of 87.3 percent. This improvement is modest because both modalities focus on spatial or appearance features, failing to capture the temporal dynamics critical for distinguishing dynamic actions such as clapping versus waving. RGB and skeleton data combination performed better, with an accuracy of 90.5 percent and an F1 score of 89.1 percent. Skeleton data encodes the temporal evolution of joint positions, which is the most discriminative feature for dynamic action recognition–this explains why RGB + Skeleton outperforms RGB + Depth by 1.5 percent accuracy on NTU RGB+D. Skeleton data seems more important for recognizing dynamic actions. Depth and skeleton data together also gave decent results, with an accuracy of 89.7 percent and an F1 score of 88.2 percent. When using all three modalities–RGB, depth, and skeleton–the model performed best, achieving an accuracy of 95.2 percent and an F1 score of 94.9 percent on NTU RGB+D, and an accuracy of 93.7 percent and an F1 score of 93.2 percent on UTD-MHAD. This 4.7 percent accuracy gain over the best bimodal combination of RGB + Skeleton stems from MAGNet’s ability to fuse complementary strengths: RGB provides scene context, Depth refines spatial localization, and Skeleton captures motion dynamics, while our adaptive fusion mechanism ensures these features are weighted according to action semantics, for example Depth is prioritized for spatial actions like reaching and Skeleton for dynamic actions like kicking. This shows that using all the modalities together works better, especially for complex action recognition tasks.

Table 5.

Cross-modal results on NTU RGB+D and UTD-MHAD datasets (mean ± std, 5 random seeds).

Modality combination	NTU RGB+D Acc. (%)	NTU RGB+D F1 (%)	UTD-MHAD Acc. (%)	UTD-MHAD F1 (%)
RGB Only	85.2 ± 0.51	84.0 ± 0.55	83.1 ± 0.53	82.4 ± 0.58
Depth Only	86.0 ± 0.47	84.5 ± 0.50	84.0 ± 0.49	83.2 ± 0.52
Skeleton Only	87.0 ± 0.43	85.7 ± 0.46	85.2 ± 0.45	84.5 ± 0.48
RGB + Depth	89.0 ± 0.42	87.3 ± 0.44	87.1 ± 0.43	86.0 ± 0.46
RGB + Skeleton	90.5 ± 0.38	89.1 ± 0.41	88.3 ± 0.40	87.5 ± 0.42
Depth + Skeleton	89.7 ± 0.39	88.2 ± 0.42	87.8 ± 0.41	87.2 ± 0.43
RGB + Depth + Skeleton (MAGNet)	95.2 ± 0.32	94.9 ± 0.35	93.7 ± 0.41	93.2 ± 0.45

As shown in Table 6, we present the model performance across different modality input conditions. In the full-modality input scenario, the Full MAGNet achieves an Accuracy of 95.2% and an F1 score of 94.9%, outperforming both the MAGNet and the Traditional Early Fusion GCN, highlighting the VQ-VAE’s ability to optimize feature representation even with all modalities available. In scenarios with missing modalities, whether it’s RGB, Depth, or Skeleton, the Full MAGNet shows significantly less performance degradation compared to MAGNet. For example, when RGB is missing, the Full MAGNet achieves 91.5% Accuracy and 90.8% F1, surpassing MAGNet by over 9 percentage points in Accuracy and 10 points in F1. This result demonstrates the effectiveness of the VQ-VAE in generating complementary features, especially when one modality is absent. In scenarios where Skeleton is missing, the Full MAGNet (with VQ-VAE) shows an impressive 89.6% Accuracy, an 11.4-percentage-point improvement over MAGNet, highlighting the VQ-VAE’s ability to recover information from missing modalities. When two modalities are missing, the performance gain from the VQ-VAE is even more pronounced. For instance, when only RGB is retained, the Full MAGNet achieves 87.2% Accuracy and 86.1% F1, a more than 12 percentage-point improvement over MAGNet , and also outperforms the RGB-specific SOTA Single-modality Model. Similarly, when only Skeleton is retained, the Full MAGNet reaches 85.3% Accuracy, surpassing MAGNet by 13.2 percentage points. In a scenario simulating 30% random modality loss, the Full MAGNet performs exceptionally well with 88.9% Accuracy and 87.7% F1, showing a minimal degradation of only 6.3%. In contrast, the MAGNet and Traditional Early Fusion GCN suffer much higher performance losses, demonstrating the robustness of the VQ-VAE module in handling missing data in dynamic scenarios. In summary, these results confirm that the VQ-VAE module significantly contributes to filling the information gap caused by missing modalities. By generating complementary features through its discrete latent space, the VQ-VAE ensures the model’s robustness and enhances its performance in action recognition tasks, particularly in cases where modality data is incomplete or missing. This reinforces the effectiveness of our approach in practical, real-world conditions with incomplete or noisy data.

Table 6.

Model performance under different modal input conditions (mean ± std, 5 random seeds).

Modality input situation	Full MAGNet (with VQ-VAE)		MAGNet (without VQ-VAE)		Traditional early fusion GCN
	Accuracy	F1	Accuracy	F1	Accuracy	F1
Full Modality (Baseline)	95.2 ± 0.32	94.9 ± 0.35	90.4 ± 0.38	88.7 ± 0.39	89.3 ± 0.40	87.6 ± 0.41
Single-modality Missing (no RGB)	91.5 ± 0.30	90.8 ± 0.32	82.3 ± 0.27	80.1 ± 0.28	79.6 ± 0.29	77.2 ± 0.30
Single-modality Missing (no Depth)	90.7 ± 0.29	89.9 ± 0.30	81.5 ± 0.32	79.3 ± 0.33	78.9 ± 0.31	76.5 ± 0.32
Single-modality Missing (no Skeleton)	89.6 ± 0.28	88.5 ± 0.29	78.2 ± 0.27	76.0 ± 0.28	75.3 ± 0.29	73.1 ± 0.30
Dual-modality Missing (only RGB)	87.2 ± 0.33	86.1 ± 0.34	74.5 ± 0.30	72.3 ± 0.31	71.8 ± 0.32	69.7 ± 0.33
Dual-modality Missing (only Depth)	86.5 ± 0.31	85.3 ± 0.32	73.1 ± 0.30	71.0 ± 0.31	70.2 ± 0.32	68.1 ± 0.33
Dual-modality Missing (only Skeleton)	85.3 ± 0.32	84.1 ± 0.33	72.1 ± 0.31	70.2 ± 0.32	69.8 ± 0.33	67.5 ± 0.34
Random Missing (30%)	88.9 ± 0.31	87.7 ± 0.32	76.5 ± 0.29	74.3 ± 0.30	73.2 ± 0.31	71.0 ± 0.32

Visualization

In this experiment, we evaluate the model’s focus on multimodal data for different actions using the MAGCN and a cross-modal self-attention mechanism. Figure 3 below shows the attention maps for two actions, “Sitting Down” and “Waving Hand,” across different attention heads, highlighting the focus on skeleton, RGB, and depth data.

Fig. 3.

Attention maps for multimodal (skeleton, RGB, and depth) across different attention heads for two action categories: “Sitting Down” and “Waving Hand.”.

For “Sitting Down,” attention head 0 focuses on the RGB data, mainly in the center of the human body. This shows the model’s reliance on static background from the RGB images. Depth data shows a more even spread, indicating less importance for this action. Heads 1 and 2 focus on skeleton and depth data, with head 2 emphasizing depth. It helps in action recognition. For “Waving Hand,” attention is more balanced across skeleton and depth data. Head 0 and 1 stress the skeleton’s temporal and spatial features for recognizing this action. The role of RGB data is less important. Depth data, providing better spatial details, gets more focus. This shows its role in dynamic action recognition. These attention maps highlight how multimodal are important for various actions. For “Sitting Down,” RGB data is key for background, while skeleton data is more critical for “Waving Hand.” This shows how the model adapts modality weights based on the action type. This helps improve recognition accuracy and robustness. The results show the model works well in handling complex human action recognition tasks.

Figure 4 shows a comparison of class-wise accuracy across five modality combinations. The 20 action categories cover a wide range of motion types, from large movements like “jumping” and “running” to more subtle gestures such as “typing” and “using-phone,” based on the NTU RGB+D and UTD-MHAD datasets. The results indicate that while traditional skeleton-based methods provide a solid baseline, their performance varies greatly across different action types and generally struggles with semantically ambiguous actions or those with overlapping motions. The inclusion of RGB, depth, and IMU information brings noticeable improvements, especially in visually or kinematically distinctive actions. Notably, the MAGCN + Cross-Attn model consistently achieves the highest accuracy across most classes, and the overall distribution remains high, with performance in the range of 90–98.5%. This robust performance is attributed to the model’s ability to capture complex inter-joint dependencies via adaptive graph convolutions and to dynamically prioritize complementary modality contributions through cross-modal self-attention. The visible spread of class-wise accuracy also highlights the varying difficulty among actions, reaffirming the importance of targeted multimodal fusion for challenging scenarios. These findings confirm that the proposed method significantly enhances recognition performance while maintaining strong generalization across diverse human actions.

Fig. 4.

Class-wise accuracy comparison of five modality combinations across 20 representative action categories from the NTU RGB+D and UTD-MHAD datasets.

Figure 5 shows the temporal attention distribution for eight joints. These joints are: Neck Base, Left Elbow, Right Elbow, Right Hip, Left Toe, Spine Center, Right Knee, and Left Wrist. They span the upper body, trunk, and lower limbs. The horizontal axis is the frame index, with 16 frames in total. The vertical axis shows the attention score for each joint in each frame. Red boxes highlight the frames where attention shifts. At frame 7, attention sharply increases for the Right Hip and Left Toe. This suggests a lower-body dominant phase, like starting a jump or stepping. At frame 9, attention peaks on the Spine Center. This points to a shift to trunk coordination, likely a stabilization or torso adjustment phase. These shifts show how the model adapts to focus on relevant joints. Unlike traditional GCN methods or unimodal models, our approach adapts. It activates discriminative joints based on the context and multimodal input. The model suppresses irrelevant features and focuses on important regions, even with noise or imbalanced data. In summary, this figure highlights the model’s temporal sensitivity. It proves the effectiveness of our cross-modal attention in recognizing critical motion cues, improving multimodal action recognition.

Fig. 5.

Temporal attention scores for eight representative joints over a 16-frame action sequence.

*Qualitative analysis

To intuitively validate the effectiveness of MAGNet in handling missing modality scenarios and the feature imputation capability of its VQ-VAE module, we present typical qualitative examples of multimodal input data with different missing patterns from the NTU RGB+D and UTD-MHAD datasets, along with the intermediate imputed features generated by the VQ-VAE module and the corresponding final action recognition predictions of MAGNet. All selected examples are representative of real-world incomplete multimodal data scenarios, covering single-modal missing, dual-modal missing and random partial missing, which are consistent with the experimental settings of the missing modality performance test, and the analysis focuses on the consistency between imputed features and valid input information as well as the rationality of model predictions.

Table 7 summarizes the typical qualitative examples of MAGNet in four missing modality scenarios, and we further elaborate on the feature imputation effect and prediction rationality of the model for each scenario. For the single-modal missing scenario with RGB absent on the NTU RGB+D dataset, the valid input data only includes Depth grayscale features and Skeleton joint connection features that characterize a specific human action, and the VQ-VAE module of MAGNet fully leverages the complementary spatial and structural information of the two valid modalities to generate imputed RGB visual features. The imputed RGB features are highly consistent with the action information contained in the valid inputs in terms of human body contour, limb movement range and dynamic action trajectory, and can clearly reflect the core characteristics of the target action without obvious distortion or misalignment. Based on the fusion of valid input features and imputed RGB features, MAGNet can stably output the correct action prediction that matches the input action information. For the single-modal missing scenario with Skeleton absent, the VQ-VAE module takes the valid RGB and Depth features as input, and imputes skeletal joint features that can accurately characterize the human body’s skeletal structure and movement state. The imputed joint features are aligned with the action posture presented by the RGB and Depth features in spatial position, and the continuous movement of key joints such as hip, knee and elbow is consistent with the dynamic change of the valid input features, providing complete structural information for the subsequent action recognition of MAGNet. For the more challenging dual-modal missing scenario on the UTD-MHAD dataset, where only the Depth modality is retained and both RGB and Skeleton modalities are absent, the VQ-VAE module still exhibits a strong feature imputation capability. The module extracts the depth contour and spatial movement information of the human action from the only valid Depth input, and based on this, generates imputed RGB visual features and skeletal joint features that match the target action. The imputed RGB features restore the visual appearance and limb movement details of the human body when performing the target action, and the imputed Skeleton features accurately capture the spatial connection relationship of human joints and the dynamic change trajectory of the action, and the two types of imputed features are mutually consistent and highly matched with the valid Depth input information. Even with only one valid modality input, the fusion of Depth features and the two types of imputed features can provide sufficient action information for MAGNet, ensuring that the model can make correct and reasonable action predictions. For the random 30% partial missing scenario, the RGB, Depth and Skeleton modalities all have fragmented feature region missing, and the valid information of each modality is incomplete. In this case, the VQ-VAE module fuses the incomplete valid information of all three modalities, learns the global correlation and action consistency of multimodal features, and further generates imputed complete multimodal features. The imputed features fill all the fragmented missing regions of the original input, restore the complete human body contour, skeletal structure and dynamic action trajectory, and maintain the continuity and integrity of the overall action information, which makes MAGNet still able to accurately recognize the target action from the incomplete input data. In all the above missing modality scenarios, the intermediate imputed features generated by the VQ-VAE module of MAGNet are highly consistent with the valid input multimodal information in terms of action posture, spatial position and dynamic trajectory, and can effectively make up for the information gap caused by missing modalities. The fusion of valid input features and imputed features provides complete and consistent multimodal action information for MAGNet, which is the key reason why the model can maintain stable and reliable action recognition performance in incomplete multimodal data scenarios. These qualitative examples fully verify the effective feature imputation capability of the VQ-VAE module and the strong robustness of MAGNet in handling missing modality scenarios, and provide intuitive and interpretable evidence for the model’s performance in real-world incomplete multimodal data application scenarios.

Table 7.

Qualitative examples of MAGNet for missing modality scenarios on NTU RGB+D and UTD-MHAD datasets.

Dataset	Missing modality scenario	Imputed features by VQ-VAE Module	Final action prediction of MAGNet
NTU RGB+D	Single-modal missing (RGB absent)	Imputed RGB visual features matching the human contour and dynamic action trajectory of valid Depth and Skeleton inputs, retaining key limb movement details	Correct action prediction consistent with the input action information
	Single-modal missing (Skeleton absent)	Imputed skeletal joint features that accurately capture the spatial position and continuous movement of human joints, aligning with the action posture of valid RGB and Depth inputs	Correct action prediction supported by complete structural information
UTD-MHAD	Dual-modal missing (RGB and Skeleton absent)	Imputed RGB and skeletal features based on valid Depth input, restoring the core action posture and limb movement of the target action	Reasonable and correct action prediction with only one valid modality
	Random 30% partial missing	Imputed complete multimodal features filling fragmented missing regions of all modalities, maintaining the integrity and continuity of overall action information	Accurate action recognition from incomplete input data

Discussion

The MAGNet model proposed in this study performs well in the task of human action recognition, especially in multimodal data fusion and complex scene adaptation, showing strong robustness. The model combines MAGCN and cross-modal self-attention mechanism. It fuses information from multimodal like RGB images, depth images, and 3D pose data. This helps improve the accuracy of action recognition. In the experiments, MAGNet worked really well on the NTU RGB+D and UTD-MHAD datasets, which are well-curated academic datasets with aligned modalities and clean data distribution. Under the Cross-Subject (XSubject) and Cross-View (XView) protocols of the NTU RGB+D dataset, the model performed better than the existing methods. However, real-world deployment scenarios involve uncurated multimodal data streams that are prone to random noise, temporal-spatial unalignment between different modalities, and uneven data quality, which pose additional challenges for the practical application of the model. Notably, diverse camera intrinsics in non-laboratory environments may affect the stability of the adaptive graph structure in MAGCN, which was primarily validated on curated datasets with standardized camera setups.

Although MAGNet demonstrates strong performance, it has certain limitations. The cross-modal self-attention mechanism reduces redundant features and improves information fusion accuracy. However, the model still struggles with extreme data loss, occlusion, or modality loss. In particular, when data is entirely lost or severely occluded, the model may struggle to maintain the expected level of accuracy. In real-world noisy data streams, such as blurred RGB frames, noisy depth signals or jittery skeleton sequences in intelligent surveillance, the model’s adaptive weight mechanism of MAGCN can partially suppress the interference of low-quality modal features, and the VQ-VAE module can compensate for incomplete modality data generated by unaligned real-world data, but the adaptation effect for severe noise and unalignment still needs to be further optimized. The VQ-VAE model can improve generation ability and provide stable, consistent output. However, its computational overhead is high. This could affect real-time processing efficiency, especially when working with long time series data. Future work can focus on optimizing the generative model to reduce the computational burden and exploring more efficient network architectures to meet real-time processing requirements.

In future research, the generation model and attention mechanism can be optimized. The adaptability of the model can be improved. Especially for changing environments. Illumination changes, rapid motion, or partial occlusion might cause data loss or incomplete modality. Studying how to improve the model for these challenges is important. We will also focus on optimizing the computational efficiency of the cross-modal self-attention mechanism to address its inherent quadratic computational overhead as the number of modalities or the temporal depth of action sequences increases. Specifically, we plan to explore sparse attention mask designs tailored to action sequences, lightweight dimension reduction strategies for high-dimensional modality features, and hierarchical cross-modal attention architectures that first compute intra-modality attention before inter-modality fusion. Additionally, we will investigate adaptive sequence sampling and attention pruning techniques to reduce redundant computations for long temporal action sequences, while retaining the core fusion capability of cross-modal self-attention. We will further investigate and enhance the stability of the adaptive graph structure in MAGCN for unstructured, non-laboratory environments with diverse camera intrinsics: this will involve adapting the adaptive edge weighting mechanism to account for variations in camera parameters and incorporating domain adaptation techniques to align the graph structure learning process with real-world data distributions that differ from curated laboratory datasets. We also plan to validate the improved MAGCN on real-world action recognition datasets with diverse camera setups to ensure the adaptive graph structure maintains robustness across varying imaging conditions. We will systematically validate the sensitivity of the model’s final performance to the initial conditions of adaptive weights in MAGCN, including testing different initialization schemes and quantifying the contribution ratio of each modality during training and inference to further verify the model’s ability to avoid convergence to a dominant modality and fully maintain multi-modal synergy. We will also focus on optimizing the model’s adaptability to real-world unaligned multimodal data streams, such as introducing lightweight temporal-spatial alignment preprocessing and single-modal noise reduction modules, to enhance the model’s robustness in practical deployment scenarios. The cross-modal self-attention mechanism can be improved too. Adding more intelligent mechanisms might help deal with the deep relationship between modalities. This can improve the overall performance. In general, MAGNet provides an efficient and robust solution for human action recognition, especially in multimodal data fusion and complex scene adaptation. In the future, with further optimization of the generative model and improvement of computational efficiency, the mitigation of quadratic computational overhead in cross-modal self-attention for large-scale modalities and long temporal sequences, the enhancement of the adaptive graph structure’s stability across diverse camera intrinsics in non-laboratory environments, the verification of insensitivity to adaptive weight initial conditions and the maintenance of multi-modal synergy, and the enhancement of adaptation capacity for real-world noisy and unaligned multimodal data, MAGNet is expected to provide more stable and efficient solutions in practical applications.

Author contributions

Hui Dong contributed to the overall research design, methodology, and manuscript writing. Fengqing Zhang worked on data analysis and model development. Chunli Liu provided valuable insights on the experimental setup and results interpretation. Lili Mu contributed to the dataset preparation and validation.

Funding

This work was supported by the Shandong Province 2024 Undergraduate Education Reform Research Project, Z2024373, and the Shandong Province 2023 Undergraduate Education Reform Research Project, Z2023065. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.

Data availability

The datasets used in this study include the NTU RGB+D 60/120 dataset, available at http://rose1.ntu.edu.sg/ Datasets/actionRecognition.asp, and the UTD-MHAD dataset, which can be accessed at https://www.utdallas.edu/nxk140030/UTD-MHAD.html.

Competing Interests

The authors declare no competing interests.