A Survey on 3D Skeleton-Based Action Recognition Using Learning Method

Abstract

Three-dimensional skeleton-based action recognition (3D SAR) has gained important attention within the computer vision community, owing to the inherent advantages offered by skeleton data. As a result, a plethora of impressive works, including those based on conventional handcrafted features and learned feature extraction methods, have been conducted over the years. However, prior surveys on action recognition have primarily focused on video or red-green-blue (RGB) data-dominated approaches, with limited coverage of reviews related to skeleton data. Furthermore, despite the extensive application of deep learning methods in this field, there has been a notable absence of research that provides an introductory or comprehensive review from the perspective of deep learning architectures. To address these limitations, this survey first underscores the importance of action recognition and emphasizes the significance of 3-dimensional (3D) skeleton data as a valuable modality. Subsequently, we provide a comprehensive introduction to mainstream action recognition techniques based on 4 fundamental deep architectures, i.e., recurrent neural networks, convolutional neural networks, graph convolutional network, and Transformers. All methods with the corresponding architectures are then presented in a data-driven manner with detailed discussion. Finally, we offer insights into the current largest 3D skeleton dataset, NTU-RGB+D, and its new edition, NTU-RGB+D 120, along with an overview of several top-performing algorithms on these datasets. To the best of our knowledge, this research represents the first comprehensive discussion of deep learning-based action recognition using 3D skeleton data.

Introduction

Action analysis, a pivotal and vigorously researched topic in the field of computer vision, has been under investigation for several decades [1–4]. The ability to recognize actions is of paramount importance, as it enables us to understand how humans interact with their surroundings and express their emotions [5,6]. This recognition can be applied across a wide range of domains, including intelligent surveillance systems, human–computer interaction, virtual reality, and robotics [7–9]. In recent years, the field of skeleton-based action recognition has made significant strides, surpassing conventional hand-crafted methods. This progress has been chiefly driven by substantial advancements in deep learning methodologies [10–19].

Traditionally, action recognition has relied on various data modalities, such as red-green-blue (RGB) image sequences [20–24], the depth image sequences [25,26], videos, or a fusion of these modalities (e.g., RGB combined with the optical flow) [27–31]. These approaches have yielded impressive results through various techniques. Compared to skeleton data, which offers a detailed topological representation of the human body through joints and bones, these alternative modalities often prove computationally intensive and less robust when confronted with complex backgrounds and variable conditions. This includes challenges posed by variations in body scales, viewpoints, and motion speeds [32,33].

Furthermore, the availability of sensors like the Microsoft Kinect [34] and advanced human pose estimation algorithms [35–38] has facilitated the acquisition of accurate 3-dimensional (3D) skeleton data [39]. Figure 1 provides a visual representation of human skeleton data. In this case, 25 body joints are captured for a given human body. Skeleton sequences possess several advantages over other modalities, characterized by 3 notable features: (a) Intraframe spatial information, where strong correlations exist between joints and their adjacent nodes, enabling the extraction of rich structural information. (b) Interframe temporal information, which captures strong and clear temporal correlations between frames of each body joint, enhancing the potential for action recognition. (c) A co-occurrence relationship between spatial and temporal domains when considering joints and bones, offering a holistic perspective. These unique attributes have catalyzed substantial research endeavors in human action recognition and detection. The escalating integration of skeleton data is anticipated to pervade diverse applications in the field.

Fig. 1.

Examples of skeleton data in NTU RGB+D / NTU RGB+D 120 datasets [131,132]. (A) Configuration of 25 body joints in the dataset. (B) RGB+joints representation of the human body.

The recognition of human actions using skeleton sequences predominantly hinges on a temporal dimension, transforming it into both a spatial and temporal information modeling challenge. As a result, traditional approaches in skeleton-based methods focus on extracting motion patterns from these sequences, prompting extensive research into handcrafted features. [31,40–44]. These features often entail capturing the relative 3D rotations and translations among different joints or body parts [13,45]. However, it has become evident that handcrafted features perform well only on specific datasets [46], highlighting the challenge that features tailored for one dataset may not be transferable to others. This issue hampers the generalization and broader application of action recognition algorithms.

With the remarkable development and outstanding performance of deep learning methods in various computer vision tasks, such as image classification [47,48] and object detection [49,50], the application of deep learning to skeleton data for action recognition has gained prominence. Nowadays, deep learning techniques utilizing recurrent neural networks (RNNs) [51], convolutional neural networks (CNNs) [52], graph convolutional networks (GCNs), and Transformer-based methods have emerged in this field [53,54]. Figure 2 provides an overview of the general pipeline for 3D skeleton-based action recognition (3D SAR) using deep learning, starting from raw RGB sequences or videos and culminating in action category prediction. RNN-based methods leverage skeleton sequences as natural time series data, treating joint coordinates as sequential vectors, aligning well with the RNN’s capacity for processing time series information. To enhance the learning of temporal context within skeleton sequences, variants like long short-term memory (LSTM) and gated recurrent unit (GRU) have been employed. Meanwhile, CNNs complement RNN-based techniques, as they excel at capturing spatial cues in the input data, which RNNs may lack. Additionally, a relatively recent approach, the GCNs has gained attention for its ability to model skeleton data in a natural topological graph structure, with joints and bones as vertices and edges, respectively, offering advantages over alternative formats like images or sequences. Transformer-based methods [55–60] capture the spatial-temporal relation of the input 3D skeleton data mainly based on its core multihead self-attention (MSA) mechanism.

Fig. 2.

The general pipeline of skeleton-based action recognition using deep learning methods. Firstly, the skeleton data was obtained in 2 ways, directly from depth sensors or from pose estimation algorithms. The skeleton will be sent into RNNs, CNNs, GCNs, or Transformer-based neural networks. Finally, we get the accurate action category.

All these 3 kinds of deep learning-based architectures have already gained unprecedented performance, but most review works just focus on traditional techniques or deep learning-based methods just with the RGB image or RGB-D data method. Ronald Poppe et al. [61] firstly addressed the basic challenges and characteristics of this domain and then gave a detailed illumination of basic action classification methods about direct classification and temporal state-space models. Daniel and Remi et al. [62] showed an overall overview of the action representation only in both spatial and temporal domains. Though the methods mentioned above provide some inspiration that may be used for input data preprocessing, neither skeleton sequence nor deep learning strategies were taken into account. Recently, Wu et al. [63] and Herath et al. [64] offered a summary of deep learning-based video classification and captioning tasks, in which the fundamental structure of CNN, as well as RNNs, was introduced, and the latter made a clarification about common deep architectures and quantitative analysis for action recognition. To our best knowledge, [65] is the first work recently giving an in-depth study in 3D SAR, which concludes this issue from the action representation to the classification methods. In the meantime, it also offers some commonly used datasets such as UCF, MHAD, MSR daily activity 3D, etc. [66–69], while it does not cover the emerging GCN based methods. Finally, [46] proposed a new review based on Kinect-dataset-based action recognition algorithms, which organized a thorough comparison of those Kinect-dataset-based techniques with various types of input data including RGB, Depth, RGB+Depth, and skeleton sequences. Reference [70] presented an overview of the action recognition across all the data modalities but without presenting the Transformer-based methods. In addition, all these works mentioned above also ignore the differences and motivations among CNN-based, RNN-based, GCN-based, and Transformer-based methods, especially when taking the 3D skeleton sequences into account.

To address these issues comprehensively, this survey aims to provide a detailed summary of 3D SAR employing 4 fundamental deep learning architectures: RNNs, CNNs, GCNs, and Transformers. Additionally, we delve into the motivations behind the choice of these models and offer insights into potential future directions for research in this field.

In summary, our study encompasses 4 key contributions:

• A comprehensive introduction about the superiority of 3D skeleton sequence data and characteristics of 3 kinds of fundamental deep architectures are presented in a detailed and clear manner, and a general pipeline in 3D SAR using deep learning methods is illustrated.

• Within each type of deep architecture, numerous contemporary methods leveraging skeleton data are introduced, focusing on data-driven approaches. These encompass spatial-temporal modeling, innovative skeleton data representation, and methods for co-occurrence feature learning.

• The discussion begins by addressing the latest challenging datasets, notably the NTU-RGB+D 120, along with an exploration of several top-ranked methods. Subsequently, it delves into envisaged future directions in this domain.

• The initial study comprehensively examines 4 foundational deep architectures, encompassing RNN-based, CNN-based, GCN-based, and Transformer-based methods within the domain of 3D SAR.

3D SAR with Deep Learning

While existing surveys have offered comprehensive comparisons of action recognition techniques based on RGB or skeleton data, they often lack a detailed examination from the perspective of neural networks. To bridge this gap, we provide a concise introduction to the fundamental properties of each architecture (Preliminaries: Basic properties of RNNs, CNNs, GCNs, and Transformers). Then our survey provides an exhaustive discussion and comparison of RNN-based (RNN-based methods), CNN-based (CNN-based methods), GCN-based (GCN-based methods), and Transformer-based (Transformer-based methods) methods for 3D SAR. We will explore these methods in-depth, highlighting their strengths and weaknesses, and introduce several latest related works as case studies, focusing on specific limitations or classic spatial-temporal modeling challenges associated with these neural network models.

Preliminaries: Basic properties of RNNs, CNNs, GCNs, and Transformers

Before delving into the specifics of each method, we provide a brief overview of the fundamental architecture, outlining their respective advantages, disadvantages, and coarse selection criteria under the 3D SAR setting.

RNNs

RNNs are ideal for capturing temporal dependencies in sequences of joint movements over time and are suited for modeling action sequences due to their ability to retain temporal information. However, RNNs are also vulnerable to long-term dependencies, potentially missing complex relationships in lengthy sequences, and are computationally inefficient due to sequential processing, leading to longer training times for large-scale datasets.

CNNs

CNNs are not only effective in capturing spatial patterns from the joint coordinates, recognizing spatial features within individual frames of the 3D skeleton data but also great for local spatial relationships among joints. However, CNNs are limited to capturing temporal evolution in sequences, potentially missing out on the temporal dynamics crucial for action recognition.

GCNs

GCNs are designed to manage graph-structured data such as skeletal joint connections in action recognition, enabling the learning of relationships between joints and their connectivity while integrating spatial and temporal information. However, GCNs can be sensitive to noisy or irregular connections among joints, potentially impacting recognition accuracy, particularly in complex actions.

Transformers

Transformers is not only efficient at capturing long-range dependencies without the vanishing/exploding gradient issue but also versatile in handling multiple modalities and learning global relationships. However, it is also computationally intensive due to attention mechanisms, potentially requiring substantial computational resources. What is more, compared to RNNs, it is also limited to sequential locality

RNN-based methods

Recursive connections within the RNN structure are established by feeding the output of the previous time step as the input to the current time step, as demonstrated in prior work [71]. This approach is known to be effective for processing sequential data. In a similar vein, models like the standard RNN, LSTM, and GRU were introduced to address limitations such as gradient-related issues and the modeling of long-term temporal dependencies that were present in the standard RNN.

From the first aspect, spatial-temporal modeling can be seen as the principle in action recognition tasks. Due to the weakness of the spatial modeling ability of RNN-based architecture, the performance of some related methods generally could not gain a competitive result [72–74]. Recently, Hong et al. [75] proposed a novel 2-stream RNN architecture to model both temporal dynamics and spatial configurations for skeleton data. Figure 3 shows the framework of their work. An exchange of the skeleton axes was applied for the data level preprocessing for the spatial dominant learning. Unlike [75], Jun et al. [76] stepped into the traversal method of a given skeleton sequence to acquire the hidden relationship of both domains. Compared with the general method which arranges joints in a simple chain so that ignores the kinetic dependency relations between adjacent joints, the mentioned tree-structure-based traversal would not add false connections between body joints when their relation is not strong enough. Then, using an LSTM with a trusted gate, the input is treated discriminately, through which, if the tree-structured input unit is reliable, the memory cell will be updated by importing input latent spatial information. Inspirited by the property of CNN, which is extremely suitable for spatial modeling. Li et al. [77] incorporated an attention RNN with a CNN model to enhance the complexity of spatial-temporal modeling. Initially, they introduced a temporal attention module within a residual learning module, allowing for the recalibration of temporal attention across frames within a skeleton sequence. Subsequently, they applied a spatial-temporal convolutional module to this first module, treating the calibrated joint sequences as images. Furthermore, in the work by Lin et al. [78], an attention recurrent relation LSTM network was employed. This network combines a recurrent relation network for spatial features with a multilayer LSTM to capture temporal features within skeleton sequences.

Fig. 3.

Examples of mentioned methods for dealing with spatial modeling problems. (A) Two-stream framework that enhances the spatial information by adding a new stream [75]. (B) Data-driven technique that addresses the spatial modeling ability by giving a transform toward original skeleton sequence data [76].

The second aspect involves the network structure, serving as a solution to address the limitations of standard RNNs. While RNNs are inherently suitable for sequence data, they often suffer from well-known problems like gradient exploding and vanishing. Although LSTM and GRU have alleviated these issues to some extent, the use of hyperbolic tangent and sigmoid activation functions can still result in gradient decay across layers. In response, new types of RNN architectures have been proposed [79–81]. Shuai et al. [81] introduced an independently recurrent neural network (IndRNN) designed to address gradient exploding and vanishing problems, making it feasible and more robust to construct longer and deeper RNNs for high-level semantic feature learning. This modification for RNNs is not limited to skeleton-based action recognition but can also find applications in other domains, such as language modeling. In the IndRNN structure, neurons in one layer operate independently of each other, enabling the processing of much longer sequences.

Finally, the third aspect is the data-driven pipeline. In the consideration that not all joints are informative for an action analysis, [82] add global context-aware attention to LSTM networks, which selectively focus on the informative joints in a skeleton sequence. Figure 4 illustrates the visualization of the proposed method, from the figure we can conclude that the more informative joints are addressed with a red circle color area, indicating those joints are more important for this special action. In addition, because the skeletons provided by datasets or depth sensors are not perfect, which would affect the result of an action recognition task, [83] transform skeletons into another coordinate system for the robustness to scale, rotation and translation first and then extract salient motion features from the transformed data instead of sending the raw skeleton data to LSTM. Figure 4B shows the feature representation process.

Fig. 4.

Data-driven based method. (A) Different importance among different joints for a given skeleton action [82]. (B) Feature representation processes, from left to right are original input skeleton frames, transformed input frames, and extracted salient motion features, respectively [83].

Numerous valuable works have utilized RNN-based methods to address challenges related to large viewpoint changes and the relationships among joints within a single skeleton frame. However, it is essential to acknowledge that in specific modeling aspects, RNN-based methods may exhibit limitations compared to CNN-based approaches. In the following sections, we delve into an intriguing question: How do CNN-based methods perform temporal modeling, and how can they strike the right balance between spatial and temporal information in action recognition?

CNN-based methods

While CNNs offer efficient and effective high-level semantic cue learning, they are primarily tailored for regular image tasks. However, action recognition from skeleton sequences presents a distinct challenge due to its inherent time-dependent nature. Achieving the right balance and maximizing the utilization of both spatial and temporal information within a CNN-based architecture remains a challenging endeavor.

Typically, from the spatial-temporal modeling aspect, most of the CNN-based methods explored the representation of 3D skeleton sequences. Specifically, to accommodate the input requirements of CNNs, 3D-skeleton sequence data undergoes the transformation from a vector sequence to a pseudo-image format. However, achieving a suitable representation that effectively combines both spatial and temporal information can be challenging. Consequently, many researchers opt to encode skeleton joints into multiple 2D pseudo-images, which are subsequently fed into CNNs to facilitate the learning of informative features [84,85]. Wang et al. [86] proposed the joint trajectory maps, which represent spatial configuration and dynamics of joint trajectories into 3 texture images through color encoding. However, this kind of method is a little complicated and also loses importance during the mapping procedure. To tackle this shortcoming, Li et al. [87] used a translation-scale invariant image mapping strategy which firstly divided human skeleton joints in each frame into 5 main parts according to the human physical structure, then those parts were mapped to 2D form. This method makes the skeleton image consist of both temporal information and spatial information. However, though the performance was improved, there is no reason to take skeleton joints as isolated points, cause in the real world, imitate connection exists among our body, for example, when waiting for the hands, not only the joints directly within the hand should be taken into account, but also other parts such as shoulders and legs are considerable. Li et al. [88] proposed the shape-motion representation from geometric algebra, which addressed the importance of both joints and bones and fully utilized the information provided by the skeleton sequence. Similarly, [13] also use the enhanced skeleton visualization to represent the skeleton data, and Carlos et al. [89] also proposed a new representation named SkeleMotion based on motion information that encodes the temporal dynamics by explicitly computing the magnitude and orientation values of the skeleton joints. Figure 5A shows the shape-motion representation proposed by [88], while Fig. 5B illustrates the SkeleMotion representation. What is more, similarly to SkeleMotion, [90] uses the framework of SkeleMotion but is based on tree structure and reference joints for a skeleton image representation.

Fig. 5.

Examples of the proposed representation skeleton image. (A) Skeleton sequence shape-motion representations [88] generated from "pick up with one hand" on Northwestern-UCLA dataset [133]. (B) SkeleMotion representation workflow [90].

Commonly, CNN-based methods represent a skeleton sequence as an image by encoding temporal dynamics and skeleton joints as rows and columns, respectively. However, this simplistic approach may limit the model’s ability to capture co-occurrence features, as it considers only neighboring joints within the convolutional kernel and may overlook latent correlations involving all joints. Consequently, CNNs might fail to learn the corresponding and useful features. In response to this limitation, Chao et al. [91] introduced an end-to-end framework designed to learn co-occurrence features through a hierarchical methodology. This approach gradually aggregates different levels of contextual information, beginning with the independent encoding of point-level information, which is then assembled into semantic representations within both temporal and spatial domains.

Besides explorations in the representation of 3D skeleton sequences, there also exist some other problems in CNN-based techniques. For example, to find a balance between the model size and the corresponding inference efficiency, DD-Net [14] was proposed to model double feature and double motion via CNN for efficient solutions. Kim et al. [92] proposed to use the temporal CNN for modeling the interpretable spatiotemporal cues [93]. As a result, the point-level feature of each joint is learned. In addition, 2-stream and 3-stream CNN-based heavy models are also proposed for improving the representation learning ability for spatial-temporal modeling [94]. So the skeleton-based action recognition using CNN is still an open problem waiting for researchers to dig in.

GCN-based methods

Drawing inspiration from the inherent topological graph structure of human 3D-skeleton data, distinct from the sequential vector or pseudo-image treatments in RNN-based or CNN-based methods. Recently, the graph convolution network has been adopted in this task frequently due to the effective representation of the graph structure data. Generally, 2 kinds of graph-related neural networks can be found, i.e., the graph neural networks and RNNs, and graph and convolutional neural networks (GCNs). In this survey, we mainly pay attention to the latter. This focus yielded compelling results, as evidenced by the performance of the GCN-based method on the rank board. Furthermore, merely encoding the skeleton sequence into a vector or 2D grid fails to fully capture the interdependence among correlated joints from the skeleton’s perspective. Conversely, GCNs present adaptability to diverse structures, such as the skeleton graph. Nonetheless, the principal challenge within GCN-based approaches persists in the handling of skeleton data, particularly in structuring the original data into a coherent graph format. Yan et al. [53] first presented a novel model, the spatial-temporal graph convolutional networks (ST-GCNs), for skeleton-based action recognition. Specifically, the approach first involved the creation of a spatial-temporal graph, wherein the joints functioned as graph vertices, establishing inherent connections within the human body structure and across temporal sequences as the graph edges. Following this step, the ST-GCN’s higher-level feature maps on the graph underwent classification using a standard Softmax classifier, assigning them to their respective action categories. This work has notably directed attention toward employing GCNs for skeleton-based action recognition, resulting in a surge of recent related research [95–100].

Built upon GCNs, 2 main common aspects are explored, i.e., more representative manner for the construction of the skeleton data and more effective designs of the GCN-based model [101,102].

From the first aspect, [101] proposed the action-structural graph convolutional networks (AS-GCNs) could not only recognize a person’s action but also use a multitask learning strategy to output a prediction of the subject’s next possible pose. The constructed graph in this work can capture richer dependencies among joints by 2 modules called Actional Links and Structural Links. Figure 6 shows the feature learning and its generalized skeleton graph of AS-GCN. Multitask learning strategy used in this work may be a promising direction because the target task would be improved by the other task as a complementary. To capture and enhance richer feature representations, Shi et al. [95] introduced the 2s-AGCN, which incorporates an adaptive topology graph. This approach allows for automatic updates leveraging the neural network’s backpropagation algorithm, effectively enhancing the characterization of joint connection strengths. Liu et al. [103] proposes MS-G3D which constructs a unified spatial-temporal graph. This big spatial-temporal graph is composed of several subgraphs, and each subgraph represents the spatial relationships of joints on a certain frame. This form of the adjacent matrix can effectively model the relationship between different joints in different frames. Similarly, there are also a lot of following-up methods proposed for constructing more representative graphs [104–106].

Fig. 6.

Feature learning with generalized skeleton graphs [101].

From the second aspect, traditional GCNs operate as straight feed-forward networks, limiting low-level layers’ access to semantic information from higher-level layers. To address this, Yang et al. [107] introduce the feedback graph convolutional network (FGCN) aimed at incrementally acquiring global spatial-temporal features. Departing from direct manipulation of the complete skeleton sequence, FGCN adopts a multistage temporal sampling strategy to sparsely extract a sequence of input clips from the skeleton data. Furthermore, Bian et al. [108] introduces a structural knowledge distillation scheme aimed at mitigating accuracy loss resulting from low-quality data, thereby enhancing the model’s resilience to incomplete skeleton sequences. Fang et al. [109] presents the spatial-temporal slow-fast graph convolutional network (STSF-GCN), which conceptualizes skeleton data akin to a unified spatial-temporal topology, reminiscent of MS-G3D.

From the preceding introduction and discussion, it is evident that the predominant concern revolves around data-driven approaches, seeking to uncover latent insights within 3D skeleton sequence data. In the realm of GCN-based action recognition, the central query persists: “How do we extract this latent information?” This question remains an ongoing challenge. Particularly noteworthy is the inherent temporal-spatial correlation within the skeleton data itself. The optimal utilization of these 2 aspects warrants further exploration. There remains substantial potential for enhancing their effectiveness, calling for deeper investigation and innovative strategies to maximize their utilization.

Transformer-based methods

Transformers [110] demonstrated their overwhelming power on a broad range of language tasks (e.g., text classification, machine translation, or question answering [110,111]), and the vision community follows it closely and extends it for vision tasks, such as image classification [48,112,113], object detection [49,50], segmentation [114], image restoration [115,116], and point cloud registration [117–119]. The emergence of transformer algorithms marks a pivotal shift in point-centric research. These transformer-based methods are gradually challenging the dominance of GCN methods, showcasing promising advancements in computational efficiency and accuracy. Upon analysis, we firmly believe that transformer-based approaches retain robust potential and are poised to become the mainstream technique in the future.

The core module in Transformer, MSAs [48,110] aggregate sequential tokens with normalized attention as: ${\mathbf{z}}_j={\sum }_i\text{Softmax}$${\left(\frac{\mathit{QK}}{\sqrt{d}}\right)}_i{\mathit{V}}_{i,j}$ where Q, K and V are query, key and value matrices, respectively. d is the dimension of query and key, and z_j is the j-th output token. This step usually represents the context relation computation and update of the overall 3D skeleton features. Building upon the MSA from the Transformer for solving the 3D-SAR problem, there are lots of transformer architecture-based solutions are proposed.

In particular, Cho et al. [120] proposed a novel model called Self-Attention Network (SAN) that completely utilizes the self-attention mechanism to model spatial-temporal correlations. Shi et al. [121] proposed a decoupled spatial-temporal attention network (DSTA-Net) that contains spatial-temporal attention decoupling, decoupled position encoding, and global spatial regularization. DSTA-Net decouples the skeleton data into 4 streams, namely, spatial-temporal stream, spatial stream, slow-temporal stream, and fast-temporal stream; each data stream focuses on expressing a particular aspect of the action. Plizzari et al. [122] proposed a novel Spatial-Temporal Transformer network (ST-TR) in which the spatial self-attention module and temporal self-attention module are used to capture the correlation between different nodes in a frame and the dynamic relationship between the same node in the whole frames. To handle action sequences of varying lengths proficiently, Ibh et al. [123] proposed TemPose, which leaves out the padded temporal and interaction tokens in the attention map. At the same time, Tempose codes the position of the player and the position of the badminton ball to predict the action class together.

The Transformer-based approach effectively mitigates the issue of solely concentrating on local information and excels in capturing extensive dependencies over long sequences. When applied to tasks involving skeleton-based human behavior recognition, the Transformer architecture demonstrates adeptness in capturing temporal relationships. However, its efficacy in modeling spatial relationships remains constrained due to limitations in capturing and encoding the intricate high-dimensional semantic information inherent in skeleton data [124,125]. Simultaneously, numerous approaches have emerged that amalgamate the Transformer with GCNs or CNNs, thereby forming hybrid architectures. These models are designed with the aspiration of harnessing the strengths inherent in each fundamental architecture. By combining the Transformer’s capabilities with the specialized strengths of RNNs, CNNs, or GCNs, these hybrid models aim to achieve a more comprehensive and powerful framework for diverse tasks [56,126–128].

Latest Datasets and Performance

Skeleton sequence datasets such as MSRAAction3D [129], 3D Action Pairs [130], and MSR Daily Activity3D [68] have be analyzed in lots of previous surveys [46,64,65]. In this survey, we mainly address the following 2 recent datasets, NTU-RGB+D [131] and NTU-RGB+D 120 [132].

The NTU-RBG+D dataset, introduced in 2016, stands as a significant resource, comprising 56,880 video samples gathered through Microsoft Kinect-v2. This dataset holds a prominent position as one of the largest collections available for skeleton-based action recognition. It furnishes the 3D spatial coordinates of 25 joints for each human depicted in an action, as illustrated in Fig. 1A. For assessing the proposed methods, 2 evaluation protocols are suggested: Cross-Subject and Cross-View. The Cross-Subject setting involves 40,320 samples, with 16,560 allocated for training and evaluation, employing a split of 40 subjects into training and evaluation groups. In the case of Cross-View, comprising 37,920 and 18,960 samples, the evaluation uses camera 1 while training is conducted using cameras 2 and 3. Recently, an extended version of the original NTU-RGB+D dataset known as NTU-RGB+D 120 has been introduced. This extended dataset comprises 120 action classes and encompasses a total of 114,480 skeleton sequences, significantly expanding the scope. Additionally, the viewpoints have increased to 155.

In Tables 1 and 2, we present the performance of recent skeleton-based techniques relevant to NTU-RGB + D and NTU-RGB + D 120 datasets, respectively. Note that in NTU-RGB+D, “CS” stands for Cross-Subject, and “CV” stands for Cross-View. For NTU-RGB + D120, there are 2 settings, i.e., Cross-Subject (C-Subject), and Cross-Setup (C-Setup).

Table  1.

The performance of the latest state-of-the-art 3D skeleton-based methods on NTU-RGB+D dataset

NTU-RGB+D dataset
Rank	Paper	Year	Accuracy (C-View)	Accuracy (C-Subject)	Method
1	Wang et al. [55]	2023	98.7	94.8	Two-stream Transformer
2	Duan et al. [134]	2022	97.5	93.2	Dynamic group GCN
3	Liu et al. [135]	2023	96.8	92.8	Temporal decoupling GCN
4	Zhou et al. [56]	2022	96.5	92.9	Transformer
5	Chen et al. [136]	2021	96.8	92.4	Topology refinement GCN
6	Zeng et al. [137]	2021	96.7	91.6	Skeletal GCN
7	Liu et al. [103]	2020	96.2	91.5	Disentangling and unifying GCN
8	Ye et al. [138]	2020	96.0	91.5	Dynamic GCN
9	Shi et al. [139]	2019	96.1	89.9	Directed graph neural networks
10	Shi et al. [95]	2018	95.1	88.5	Two-stream adaptive GCN
11	Zhang et al. [140]	2018	95.0	89.2	LSTM-based RNN
12	Si et al. [141]	2019	95.0	89.2	AGC-LSTM(Joints&Part)
13	Hu et al. [142]	2018	94.9	89.1	Nonlocal S-T + frequency attention
14	Li et al. [101]	2019	94.2	86.8	GCN
15	Liang et al. [143]	2019	93.7	88.6	3S-CNN + multitask ensemble learning
16	Song et al. [144]	2019	93.5	85.9	Richly activated GCN
17	Zhang et al. [145]	2019	93.4	86.6	Semantics-guided GCN
18	Xie et al. [77]	2018	93.2	82.7	RNN+CNN+Attention

Table 2.

The performance of the latest state-of-the-art 3D skeleton-based methods on NTU-RGB+D 120 dataset

NTU-RGB+D 120 dataset
Rank	Paper	Year	Accuracy (C-Subject)	Accuracy (C-Setup)	Method
1	Wang et al. [55]	2023	92.0	93.8	Two-stream Transformer
2	Xu et al. [146]	2023	90.7	91.8	Language knowledge-assisted
3	Zhou et al. [56]	2022	89.9	91.3	Transformer
4	Duan et al. [134]	2022	89.6	91.3	Dynamic group GCN
5	Chen et al. [136]	2021	88.9	90.6	Topology refinement GCN
6	Chen et al. [147]	2021	88.2	89.3	Spatial-temporal GCN
7	Liu et al. [103]	2020	86.9	88.4	Disentangling and unifying GCN
8	Cheng et al. [148]	2020	85.9	87.6	Shift GCN
9	Caetano et al. [90]	2019	67.9	62.8	Tree structure + CNN
10	Caetano et al. [89]	2019	67.7	66.9	SkeleMotion
11	Liu et al. [149]	2018	64.6	66.9	Body pose evolution map
12	Ke et al. [150]	2018	62.2	61.8	Multitask CNN with RotClips
13	Liu et al. [151]	2017	61.2	63.3	Two-stream attention LSTM
14	Liu et al. [12]	2017	60.3	63.2	Skeleton visualization (single stream)
15	Jun et al. [152]	2019	59.9	62.4	Online+Dilated CNN
16	Ke et al. [153]	2017	58.4	57.9	Multitask learning CNN
17	Jun et al. [82]	2017	58.3	59.2	Global context-aware attention LSTM
18	Jun et al. [76]	2016	55.7	57.9	Spatiotemporal LSTM

Based on the observation of the performance of these 2 datasets, we find that it is evident that existing algorithms have achieved impressive performances in the original NTU-RGB+D dataset. However, the newer NTU-RGB+D 120 poses a significant challenge, indicating that further advancements are needed to effectively address this more complex dataset. It is worth noting that the GCN-based methods achieved the leading results compared to the other 2 architectures. In addition to the very fundamental architectures (i.e., RNNs, CNNs, and GCN), the most recent Transformer [110] based methods also show their promising performance on both datasets. It is also easy to find that a hybrid Transformer and other architectures also further boost the overall performance of the 3D SAR.

Discussion

Considering the performance and attributes of the aforementioned deep architectures, several critical points warrant further discussion concerning the criteria for architecture selection. In terms of accuracy and robustness, GCNs demonstrate potential excellence by adeptly capturing spatial and temporal relationships among joints. RNNs exhibit proficiency in capturing temporal dynamics, while CNNs excel in identifying spatial features. When evaluating computational efficiency, CNNs boast faster processing capabilities owing to their parallel processing nature, contrasting with RNNs’ slower sequential processing. Additionally, RNNs tend to excel in recognizing fine-grained actions, where temporal dependencies play a crucial role, while CNNs may better suit the recognition of gross motor actions based on spatial configurations. Considering factors like dataset size and hardware resources, the choice becomes more adaptable, contingent on the final model’s scale. The size of the dataset and available computational resources for training become pivotal considerations, as different architectures might entail varying requirements. In summary, when recognizing actions reliant on temporal sequences, RNNs prove suitable for capturing the nuanced temporal dynamics within joint movements. In contrast, CNNs excel in identifying static spatial features and local patterns among joint positions. However, for comprehensive action recognition, leveraging both spatial and temporal relationships among joints, GCNs offer a beneficial approach when dealing with 3D skeletal data.

A possible in-practical solution can be also proposed to integrate not only one architecture but also a combination of them. This may make the final model absorb the advantages of each fundamental architecture. Furthermore, beyond the choice of deep architectures, the trajectory of 3D skeleton action recognition (SAR) navigation is a crucial consideration. Building upon our earlier discussions, we deduce that long-term action recognition, optimizing 3D-skeleton sequence representations, and achieving real-time operation remain significant open challenges. Moreover, annotating action labels for given 3D skeleton data remains exceptionally labor-intensive. Exploring avenues such as unsupervised or weakly-supervised strategies, along with zero-shot learning, may pave the way forward.

Conclusion

This paper presents an exploration of action recognition using 3D skeleton sequence data, employing 4 distinct neural network architectures. It underscores the concept of action recognition, highlights the advantages of skeleton data, and delves into the characteristics of various deep architectures. Unlike prior reviews, our study pioneers a data-driven approach, providing comprehensive insights into deep learning methodologies, encompassing the latest algorithms spanning RNN-based, CNN-based, GCN-based, and Transformer-based techniques. Specifically, our focus on RNN and CNN-based methods centers on addressing spatial-temporal information by leveraging skeleton data representations and intricately designed network architectures. In the case of GCN-based approaches, our emphasis lies in harnessing joint and bone correlations to their fullest extent. Furthermore, the burgeoning Transformer architecture has garnered significant attention, often employed in conjunction with other architectures for action recognition tasks. Our analysis reveals that a fundamental challenge across diverse learning structures lies in effectively extracting pertinent information from 3D skeleton data. The topology graph emerges as the most intuitive representation of human skeleton joints, a notion substantiated by the performance metrics observed in datasets like NTU-RGB+D. However, this does not negate the suitability of CNN or RNN-based methods for this task. On the contrary, the introduction of innovative strategies, such as multi-task learning, shows promise for substantial improvements, particularly in cross-view or cross-subject evaluation protocols. Nevertheless, achieving further accuracy enhancements on datasets like NTU-RGB+D presents increasing difficulty due to the already high-performance levels attained. Hence, redirecting focus toward more challenging datasets, such as the enhanced NTU-RGB+D 120 dataset, or exploring other fine-grained human action datasets becomes imperative. Finally, we delve into an exhaustive discussion on the selection of foundational deep architectures and explore potential future pathways in 3D skeleton-based action recognition.

Data Availability

Data are available upon reasonable request.