Transformer-based 3D pose estimation pipeline with modular SmoothNet integration for animation generation

Abstract

Traditional 3D human pose estimation and animation generation methods still face high computational cost and temporal instability. To address these issues, this study develops a Transformer-based 3D pose estimation pipeline for spatial prediction and integrates SmoothNet as an external modular post-processing component for temporal smoothing. The backbone extracts image features with ResNet, encodes global dependencies using Transformer blocks, and outputs 3D keypoints for downstream reconstruction. SmoothNet is then applied to reduce frame-level jitter in predicted pose sequences. We evaluate the full pipeline on H3WB, AGORA, and AthletePose3D using MPJPE, PA-MPJPE, MPVE, and acceleration error. In addition, a fairness-controlled protocol on H3WB is introduced to compare MHFormer and the proposed backbone under unified SmoothNet post-processing settings, separating backbone effects from smoothing effects. Animation quality is further validated through Blender and Unreal Engine (UE) deployment.

Introduction

Three-dimensional (3D) human pose estimation is one of the core technologies in computer vision and graphics¹. By extracting the 3D coordinates of human joint points from two-dimensional images or videos, it enables precise capture and reconstruction of human motion². This technology holds broad application prospects in fields such as motion analysis, human-computer interaction, virtual reality (VR), augmented reality (AR), and medical rehabilitation. For instance, it analyzes athletes’ movements in sports training³, drives virtual character motion in game development⁴, and assists motion capture in film and television special effects⁵.

With the growth of the animation industry, particularly the rise of games, film/TV effects, and VR applications, pose-based animation synthesis has become a key method for enhancing character motion realism and automation⁶. Traditional animation production primarily relies on manual drawing or expensive optical motion capture equipment⁷. These methods are not only labor-intensive and costly but also often lack dynamic adaptability, failing to respond in real-time to environmental changes such as terrain variations or object interactions. Additionally, they typically require specialized equipment and significant manpower, which can lead to inaccurate poses or jerky animations in complex scenes. Early pose estimation methods predominantly employed manual feature extraction and simple geometric models⁸, exhibiting limited robustness and failing to effectively handle the complexity and diversity of human motion, such as errors caused by joint occlusion or lighting variations.

In recent years, the rapid advancement of deep learning technologies has revitalized this field, making the automatic generation of photorealistic animations via neural network models a research hotspot⁹. Deep learning frameworks like convolutional neural networks (CNNs) have replaced traditional manual feature extraction. Trained on massive datasets, they significantly enhance the accuracy and generalization capabilities of pose estimation. However, existing methods still face numerous challenges: First, the mapping from two-dimensional images to 3D space is inherently an ill-posed problem, characterized by intrinsic depth ambiguity. Second, traditional CNN-based models struggle to capture long-range dependencies and global contextual information, potentially leading to biased estimates in complex scenarios such as partial occlusions, non-uniform lighting, or intricate backgrounds. Furthermore, such models typically demand substantial computational overhead and memory consumption, limiting their application in real-time animation generation scenarios. More critically, 3D pose sequences estimated directly from static image frames often lack temporal consistency, resulting in unnatural jitter and abrupt transitions in generated animations. This severely compromises the fluidity and realism of the final animation, falling short of the requirements for high-quality production.

This research focuses on the application of Transformer models in 3D human pose estimation and extends them to an optimized framework for animation synthesis. The Transformer architecture, originally proposed by Vaswani et al.¹⁰ for natural language processing, has revolutionized sequential data modeling through its self-attention mechanism. Leveraging self-attention mechanisms, Transformers efficiently process sequential data in parallel and model long-range relationships¹¹, demonstrating potential across diverse visual tasks—such as the application of Vision Transformers (ViT) in image classification¹². Building upon this foundation, this work develops a Transformer-based 3D pose backbone and integrates SmoothNet¹³ as an external temporal refinement module within a unified pipeline. The backbone is responsible for spatial pose regression and computational efficiency, while SmoothNet is used as prior work for temporal stabilization rather than as a newly proposed smoothing architecture in this work. Validation is conducted using Blender and Unreal Engine (UE) for animation rendering. This approach not only enhances pose estimation robustness but also provides an end-to-end framework for low-cost motion capture animation, enabling direct synthesis of natural character motion from video.

The key motivations of this work are threefold: First, existing CNN-based 3D pose estimation methods incur high GPU memory consumption and computational overhead, limiting their application in real-time scenarios. Second, monocular pose estimation inherently suffers from depth ambiguity and temporal inconsistency, resulting in jittery animations unsuitable for professional production. Third, there lacks an end-to-end pipeline that seamlessly integrates pose estimation with animation rendering in industry-standard tools.

The main contributions of this work are as follows: (1) We design a Transformer-based 3D pose estimation backbone that replaces deconvolutional heatmap regression with attention-driven decoding, improving spatial estimation while reducing GPU memory usage. (2) We integrate SmoothNet¹³ as an external plug-and-play post-processing module and explicitly separate backbone contribution from smoothing contribution through controlled with/without-SmoothNet analyses. (3) We establish a complete engineering workflow from pose estimation to SMPL/IK-driven animation rendering in Blender and Unreal Engine, providing an end-to-end practical pipeline for animation-oriented deployment.

State of the art

Traditional animation and motion synthesis

Traditional animation production primarily relies on two methods: manual creation and motion capture¹⁴, with motion capture predominantly used for animation in complex environments. Both approaches are labor-intensive and yield fixed animations. With the rapid advancement of deep learning, numerous neural network-based animation or motion generation frameworks have been proposed. For instance, Jayamohan et al.¹⁵ introduced a method utilizing gated recurrent units (GRUs) and residual structures to predict motion and enhance its continuity. Loi et al.¹⁶ introduced a hierarchical learning recurrent neural network (HL-RNN) for motion synthesis, enabling smooth transitions between diverse movements. Recognizing quaternions’ utility for interpolating animation frames, Zhou et al.¹⁷ developed the QuaterNet architecture to represent quaternion rotations. These methods update character poses on a per-frame basis—using previous frame poses to predict the next frame—making them suitable for real-time systems like games. However, the actions predicted by these models suffer from several issues: first, action drift occurs; second, due to the inherent characteristics of recurrent neural networks, noise and errors accumulate over time, leading to stiff or jerky generated actions; Third, these models cannot output realistic animations that adapt to environmental changes, such as foot lift height failing to adjust to terrain elevation.

To address terrain adaptation and control for characters across diverse environments, Shi et al.¹⁸ analyzed character gait in games and proposed the “Motion Cycle” parametric data structure for cyclic motion. This structure synthesizes cyclically varying movements (e.g., walking) and controls human gait to adapt dynamically to terrain changes. Following Mukai et al.’s approach, Ma et al.¹⁹ introduced Phase-Function Neural Networks (PFNN). These networks utilize recorded phase information to control cyclically varying bipedal motion, while also integrating controller input to dynamically alter movement states and output them in real-time to character animations. Wang et al.’s²⁰ work holds significant influence in real-time animation generation, serving as a pioneering contribution to phase-based real-time animation generation. Inspired by the PFNN network architecture, Shao et al.²¹ proposed the Mode-Adaptive Neural Networks (MANN) model for controlling the motion of quadrupedal organisms like animals. Since animal motion cannot be described by a single phase as in PFNN, they introduced an expert system to capture the polymorphic nature of quadrupedal movement. This approach autonomously learns phase values to control network weights, generating realistic animations of quadrupedal animals walking or running. Jia et al.²² proposed a method called Local Motion Phase to describe locally periodic motion. Using a dataset of basketball players’ movements, they trained an animation generation model. This model could interact with environmental objects and even other characters, ultimately demonstrating animated results of a 1v1 basketball game. These approaches all belong to phase-based real-time animation generation algorithms, differing primarily in their improvements to the phase concept.

Physics-based animation generation

Physics-based animation generation schemes can more realistically simulate changes in human tissue and environments, producing highly lifelike character animations. However, their implementation is challenging, making them a research hotspot in the animation field over the past two years. Tessler et al.²³ proposed a reinforcement learning framework that controls character motion through physical constraints. This method can mimic human control strategies for high-level actions like side rolls and backflips performed in real life. Younes et al.²⁴ employed reinforcement learning to generate physics-based real-time character animations. Physics-based animations exhibit significantly greater adherence to physical laws and enable superior interactivity, such as controlling movement direction and velocity. Wang et al.²⁵ investigated human movement on soft surfaces like snow and sand, proposing a mechanics-based character animation generation approach. This method calculates forces between the character’s weight and different ground materials, along with ground deformation, significantly enhancing the visual realism of generated animations. Wibowo et al.²⁶ proposed learning from reference motions using spatiotemporal boundaries instead of imitation rewards. This approach permits minor deviations in joint angles and specific positions, introduces diverse learning strategies, and enhances learning through sampling low-reward states. Rocca et al.²⁷ investigated training a physics-based character controller to reproduce diverse motions using a motion controller as input, optimizing motion generation via a two-layer model and clustering techniques. Ye et al.²⁸ proposed a deep reinforcement learning-based physics-based character animation framework focused on mimicking complex skills like jumping and rolling. They enhanced controller robustness through progressive training strategies and validated its noise resilience in simulated environments. Ahmed et al.²⁹ developed the PDP (Physics-Based Character Animation via Diffusion Policy) method, combining reinforcement learning with behavior cloning to generate physics-compliant real-time character animation, demonstrating particularly strong performance in directional and velocity control. Wong et al.³⁰ introduced the PARC (Physics-based Augmentation with Reinforcement Learning for Character Controllers) framework. It utilizes physics simulation to augment data and employs reinforcement learning to optimize character controllers, addressing the adaptability issues of traditional methods in complex terrains.

Graph-based and transformer-based pose estimation

Beyond traditional CNN and RNN approaches, graph-based methods have also shown promise in modeling skeletal structures. Yan et al.³¹ introduced spatial temporal graph convolutional networks (ST-GCN) for skeleton-based action recognition, which naturally represent human body joints as graph nodes and bones as edges, capturing spatial configuration and temporal dynamics simultaneously. While ST-GCNs excel in action recognition, their reliance on predefined graph topologies limits adaptability to diverse pose estimation scenarios.

In recent years, Transformer-based architectures have emerged as a powerful paradigm for 3D human pose estimation, leveraging self-attention mechanisms to capture spatiotemporal dependencies more effectively than traditional RNN or CNN methods. For example, Zheng et al.³² proposed PoseFormer, a spatial-temporal Transformer that processes 2D joint sequences to regress 3D poses, achieving improved accuracy on benchmarks like Human3.6M by modeling long-range interactions. Building on this, Zhang et al.³³ introduced MixSTE (Seq2seq Mixed Spatio-Temporal Encoder), which combines multi-scale feature fusion with Transformer blocks to handle occlusions and noise in monocular videos, showing enhanced robustness in wild scenarios. More recently, Li et al.³⁴ developed MHFormer (Multi-Hypothesis Transformer), which generates multiple pose hypotheses via Transformer decoders to mitigate ambiguity in 3D lifting from 2D inputs, further optimizing for real-time animation applications. These models highlight the shift toward efficient, attention-driven approaches that align well with downstream tasks like animation synthesis, addressing limitations in continuity and environmental adaptation seen in earlier methods.

Related advances in visual representation and generation

Beyond 3D pose estimation, related advances in pose representation and head pose estimation provide complementary perspectives. Cao et al. ³⁵ demonstrated that vector-based representations can enhance head pose estimation robustness under varying viewpoints. Additionally, prototype learning approaches—such as those explored in TransFlow—offer promising directions for predicting discrete pose positions through learned feature prototypes, potentially reducing ambiguity in multi-hypothesis scenarios. These techniques emphasize the importance of appropriate pose representations and feature learning strategies, which inform the design choices in our Transformer-decoder architecture.

The success of Transformer architectures extends beyond pose estimation to diverse computer vision tasks, demonstrating the versatility of attention mechanisms and deep feature learning. In image restoration, Chen et al.³⁶ proposed a dual degradation image inpainting method (AFFU) that addresses multiple degradation tasks simultaneously through adaptive multi-feature fusion and a self-guided module for multi-scale information integration. Their work demonstrates how adaptive feature selection during network progression prevents information loss—a principle relevant to our temporal smoothing approach. For image super-resolution, Chen et al.³⁷ introduced CAT (Channel Attention and Transformer), which synergistically integrates convolutional operations with Swin Transformer blocks through channel-attention-embedded mechanisms, achieving superior reconstruction quality while reducing computational costs by 68% compared to SwinIR. This highlights the efficiency gains possible through strategic fusion of local and global feature processing, analogous to our integration of ResNet and Transformer modules. In multi-modal tracking, Zhang et al.³⁸ developed MGNet, which employs dual-stage attention for cross-modality cross-region mutual guidance in RGBT tracking, achieving 75 FPS real-time performance. Their approach to preserving distinct modality features while enabling cross-modal interaction parallels our strategy of maintaining 2D image features and 3D template encodings before cross-attention decoding. More recently, Zhang et al.³⁹ proposed DCCM-Net for crack segmentation, combining difference convolution operators with a hybrid CNN-Mamba multi-scale attention module that captures both spatial and long-range dependencies. While these works address different problem domains, they collectively illustrate the broader trend of leveraging attention-based architectures and adaptive feature fusion strategies—principles that also underpin our Transformer-based pose estimation framework. The shared emphasis on hierarchical feature extraction, multi-scale processing, and attention mechanisms reinforces the rationale for adopting Transformer architectures in our work. More recently, diffusion-based methods have further expanded generative capabilities in visual synthesis; Han et al. ⁴⁰ proposed treating image translation as a programmable diffusion process, offering a flexible framework for controllable generation that complements attention-based estimation pipelines.

Methodology

Design of a transformer-based human pose estimation network framework

The proposed SN-HPET framework consists of four main stages, as illustrated in Fig. 1: (1) feature extraction: input images or videos are processed through a ResNet backbone network to extract 2D features, which are then flattened and augmented with positional encoding; (2) 3D pose estimation: a transformer-based architecture with template-guided decoding estimates 3D keypoints, twist angles, and shape parameters; (3) temporal smoothing: SmoothNet module performs temporal smoothing on the estimated 3D keypoint sequences to reduce jitter; (4) pose reconstruction: HybrIK and SMPL model reconstruct the final 3D pose and generate animation output. The following subsections detail each stage.

Fig. 1.

Overall network architecture diagram.

In the original 3D human pose estimation algorithm, convolutional neural networks were employed to generate 3D joint heatmaps for predicting the coordinate locations of human joints in input images or videos. This process primarily utilized three deconvolution layers and one 1 × 1 convolution. However, this approach incurs substantial computational overhead due to the vast number of parameters involved—such as input feature map size, convolution kernel size, stride, padding, and input/output channel counts—while also consuming significant GPU memory. This work proposes a method to replace the original convolutional neural network with a Transformer block for 3D keypoint estimation. This approach leverages the Transformer module’s outstanding performance in sequence data modeling, along with the parallel computing capabilities and global information capture enabled by the self-attention mechanism. The goal is to reduce GPU memory consumption while improving keypoint prediction accuracy. The overall network model architecture is shown in Fig. 1.

Stage 1: feature extraction

The first stage extracts 2D features from input images or videos. First, input images or videos are fed into the ResNet deep residual network for feature extraction. After processing, the two-dimensional image features are flattened into a tensor . Flattening refers to converting multidimensional data—such as two-dimensional image feature maps—into one-dimensional arrays or tensors. This approach offers several advantages: (1) enhanced computational efficiency: one-dimensional arrays or tensors are simpler to process when handling large-scale data and complex models. (2) Simplified data processing: this approach streamlines data structures. (3) Reduced redundant or duplicate data, minimizing storage requirements. (4) Improved model generalization: flattening multidimensional data into one-dimensional feature vectors enables models to more readily learn global features within the data. (5) Facilitates feature processing. When integrating diverse feature types, converting them into uniform one-dimensional vectors simplifies the fusion process.

Stage 2: 3D pose estimation with transformer

The second stage employs Transformer blocks with template-guided decoding to estimate 3D pose parameters. The flattened tensor extracted from the ResNet backbone network serves as input to the Transformer block processing 3D keypoint estimation. Before feeding into the Transformer block, since the Transformer model itself lacks the capability to process positional information within sequences, these features must first be combined with learnable two-dimensional positional encoding (Positional Encoding). This helps the model understand the relationships between different positions within the sequence. Finally, a linear transformation operation is applied to match the width (also known as the embedding dimension or feature dimension) within the Transformer model. This ensures the input data’s dimensionality matches the processing dimension within the Transformer block. Subsequently, the Transformer block processes the 3D keypoint estimation component. The network module replacing convolutional neural networks for 3D keypoint estimation consists of T Transformer blocks. The network flowchart for performing 3D keypoint estimation using Transformer blocks is shown in Fig. 2.

Fig. 2.

Flowchart of the 3D keypoint estimation network using transformer blocks.

For each Transformer block t ∈ [1,T], the following three steps are performed: First, pose-related features describing human shape, structure, and position are extracted from the input 2D image and encoded. These extracted features undergo transformation to facilitate subsequent computations. Converting the original image into a tensor preserves posture-related information while reducing redundancy and noise. The specific calculation equation is as follows:

1

2

3

In the equation: the first term, , serves as the source input for the transformer. It is a flattened two-dimensional image feature tensor derived from the output of the ResNet backbone network. This is subsequently added to the two-dimensional position encoding before being fed into the Transformer block for two-dimensional image encoding.

Step 2: A 3D target template is encoded to guide the estimation. This template acts as a reference for comparing against the estimated 3D pose, providing a standardized representation for evaluation. It provides a unified, standardized representation enabling different pose estimation algorithms or models to be compared and evaluated within a common framework. The specific calculation equation is as follows:

4

5

In the equation: The first term represents the target input for the current Transformer block, while can be understood as the 3D template output. The use of n distinguishes from . By integrating joint data information and output type information, it guides the Transformer model to generate accurate 3D keypoints, rotation angles, and SMPL shape parameters, producing the desired output and generating or predicting the final target values. As a 28 × d tensor, it contains templates for three types of desired outputs: (1) 24 3D keypoints; (2) 23 rotation angles ϕ; (3) SMPL⁴¹ shape parameters β. SMPL is a skinned multi-person linear model that represents the human body as a parametric mesh, enabling realistic body shape and pose generation through learned shape and pose parameters. Each module consists of two parts: the first is the corresponding joint embedding. For each joint in the SMPL model, it represents an embedding vector. There are 24 joint embeddings, corresponding to the 24 joints in the SMPL model. The second part is the output type embedding, with three types corresponding to 3D keypoints, rotation angles, and SMPL shape parameters. The template is formed by concatenating the corresponding joint embedding and output type embedding. Thus, each output type template incorporates both joint information and output type instructions, enabling the Transformer model to accurately generate the desired outputs. Third, decoding combines the encoded 2D image input with the encoded 3D pose template to produce the final 3D model pose output. The decoding process involves complex computations, including feature matching and pose optimization, to generate the 3D pose corresponding to the input image. The specific calculation equation is as follows:

6

7

8

The complete process of transformer-based 3D pose estimation is summarized in Algorithm 1.

Algorithm 1.

Stages 1-2: feature extraction and 3D pose estimation.

Each Transformer block undergoes these three rounds of computation, with the final output being a 28 × d tensor. Different parts of this tensor are projected onto distinct output types via linear transformations: (1) The first 24 elements are linearly projected onto corresponding 3D keypoints. A linear layer transforms each d-dimensional vector into a 3D vector, yielding 24 3D keypoints, each possessing (x, y, z) coordinates. (2) The subsequent 23 elements are linearly projected onto the cosine and sine values of the rotation angle ϕ. This continuous representation of angles avoids periodicity issues that would otherwise require each angle to be expressed as a pair . However, this approach would require 2 × 23 = 46 output positions. Therefore, this work combines the sine and cosine values for each angle into a single two-dimensional vector. This requires only 23 output positions, with each vector being utilized through linear projection. (3) The final element undergoes linear projection onto a 10-dimensional SMPL shape parameter β. As a 3D model for human body modeling, SMPL employs shape parameters β to control the model’s morphology. Linear projection converts the final d-dimensional vector into a 10-dimensional vector, thereby meeting the shape parameter requirements for the SMPL parameter model.

The position coordinates of the 24 3D keypoints obtained through linear transformation, the 23 vectors composed of sine and cosine values corresponding to different rotation angles ϕ , and the shape parameter β required by the SMPL parameter model are output from Stage 2. These outputs are subsequently processed in Stage 3 (temporal smoothing) and Stage 4 (pose reconstruction), as detailed in  “SN-HPET model network architecture design” and “Human pose data generation, character skeleton binding, and animation rendering”. Specifically, the 3D keypoints are sent to SmoothNet for temporal smoothing, while all parameters (keypoints, twist angles, and shape parameters) are used in Stage 4 for pose reconstruction. In Stage 4, the smoothed keypoints and other parameters are processed through HybrIK (the inverse kinematics algorithm) to compute pose parameters θ. Using the functional method provided by the SMPL parameter model , combined with the pose parameters θ and shape parameters β, the body mesh W can be obtained. Through linear regression or the Forward Kinematics (FK) method, the final recognition result pose V can be obtained.

The loss function is designed to minimize the error between predicted 3D joint coordinates and ground truth 3D coordinate labels. In the 3D keypoint estimation module, an loss function is employed to progressively learn accurate joint positions by supervising the difference between predicted pose coordinates and ground truth 3D coordinates. The formula is as follows:

9

Where Y denotes the current y-th joint, represents the predicted joint position, represents the true joint position, denotes the norm, and the subtraction yields the error loss for joint prediction.

The algorithm employs the IK (Inverse Kinematics) method to calculate the joint twist and rotation angles of the human model. However, the loss function only accounts for the error between the predicted joint position and the actual ground truth joint position, neglecting the rotation angle. To enhance training accuracy and avoid potential discontinuities in the algorithm, the original IK algorithm employs two-dimensional vectors for learning instead of simple scalar values. The designed loss function equation is as follows:

10

In the equation: represents the predicted rotation angle of the y-th joint node, while denotes the actual rotation angle of the y-th joint node. denotes the norm, calculated by forming vectors from the cosine and sine components respectively.

Additionally, since the algorithm utilizes the SMPL human parameter model for result visualization, the training phase requires supervised learning for the shape parameter β and rotation parameter θ losses of the SMPL model, as expressed by the following equation:

11

12

In the equation: β and θ represent the predicted shape parameters and rotation parameters, respectively, while and denote the actual shape parameters and rotation parameters.

To enhance the training of this human pose network model, the weighted sum of the two loss functions described above and the two SMPL model loss functions is calculated. The overall loss function for the learned network framework is ultimately obtained as:

13

Where , , and are the weighting coefficients for the corresponding loss terms. These weights are determined empirically through grid search on the validation set. Specifically, we evaluate combinations of , , and , selecting the configuration that minimizes PA-MPJPE on the validation split. In our experiments, we set , , and , balancing joint position accuracy with SMPL parameter consistency. The relatively lower weights for and reflect that SMPL parameters serve as auxiliary supervision, while joint position errors remain the primary optimization target.

SN-HPET model network architecture design

Stage 3: temporal smoothing with SmoothNet

This section introduces the SN-HPET model, which incorporates the advanced plug-and-play SmoothNet as an optimized network module within the model pipeline. This constitutes Stage 3 of the overall framework, focusing on temporal smoothing of the 3D keypoint sequences estimated in Stage 2. As a purely temporal-domain modeling approach, SmoothNet demonstrates exceptional cross-model and cross-dataset transferability. By incorporating this module, it addresses the significant deviation issue observed in “Design of a transformer-based human pose estimation network framework”’s 3D human pose estimation algorithm. This deviation arises when occlusions or other noise factors cause substantial misalignment between estimated joint positions and corresponding ground truth values across consecutive frames. In this work, SmoothNet is treated as prior external work¹³. Our contribution in Stage 3 is the system-level integration and evaluation protocol within the proposed Transformer-to-animation pipeline, rather than introducing a new temporal smoothing architecture.

This network architecture comprises two primary components: the first is a pre-defined human pose estimation model. This model can be based on a direct approach, estimating 3D human pose directly from image or video data, or a two-stage method, first performing 2D pose estimation before elevating it to 3D space. The choice depends primarily on the specific application scenario and requirements. This section utilizes the 3D human pose estimation algorithm from the previous section as its core. The second component is the SmoothNet network model. This model differs from other spatio-temporal models in that it is purely time-dimensional. It also demonstrates plug-and-play characteristics, supported by strong cross-model and cross-dataset capabilities.

SmoothNet processes the 3D keypoint sequences estimated in Stage 2. First, the 3D human pose estimation network (Stage 2) identifies and extracts human pose information from video frame data, outputting 3D keypoint sequences. The resulting 3D keypoint sequences are fed into the SmoothNet network module (Stage 3) for temporal smoothing. After processing, the smoothed keypoint sequences are obtained. Subsequently, the smoothed keypoints, along with twist angles and shape parameters from Stage 2, undergo further processing in Stage 4 using HybrIK and the SMPL model for pose reconstruction and animation generation. Ultimately, compared to video frames without the SmoothNet module, this method achieves smoother handling of jitter caused by noise. The network architecture of SN-HPET is illustrated in Fig. 3.

Fig. 3.

SN-HPET network model architecture. The input sample images are sourced from the AGORA synthetic dataset⁴².

The complete SN-HPET pipeline integrating SmoothNet for temporal smoothing is presented in Algorithm 2.

Algorithm 2.

SN-HPET: complete pipeline with four stages.

Based on the aforementioned workflow, this model achieves improved performance when trained on datasets. The generated animations exhibit enhanced visual quality while reducing the impact of noise on results.

Similarly, building upon this workflow, the SmoothNet network module is directly integrated into existing 3D human pose estimation models. Through data-driven modeling, it captures the natural smoothness characteristics of bodily motion. The human pose data set generated from video keyframes is packaged into a PK animation file and imported into the 3D graphics renderer Blender to visualize the preliminary animated character motion results. The renderer outputs FBX files corresponding to the video frame data. The animation is then mapped onto a virtual character via UE’s IK binding, achieving motion capture effects. Realistic movements are reproduced through the animation of the virtual character model. This integrated workflow enables direct observation of the performance differences achieved by incorporating SmoothNet processing into video frames.

Stage 4: pose reconstruction and animation generation

The fourth stage reconstructs the final 3D pose and generates animation output using inverse kinematics (IK) and the SMPL model. The smoothed 3D keypoints from Stage 3, along with twist angles and shape parameters from Stage 2, are processed through HybrIK to compute pose parameters θ. These parameters, combined with shape parameters β, are fed into the SMPL model to generate the body mesh W. Finally, forward kinematics (FK) or linear regression is applied to obtain the reconstructed pose V, which serves as the output for animation generation.

Human pose data generation, character skeleton binding, and animation rendering

This subsection details the implementation of Stage 4, focusing on the practical workflow for generating animations from reconstructed poses. The reconstructed pose V obtained from Stage 4 is processed through Blender and Unreal Engine to produce final animation output.

Human skeletal model and joint information

By abstracting human limbs as line segments and connecting joints according to skeletal relationships, the human skeletal model depicted in Fig. 4 is obtained.

Fig. 4.

Skeletal model joint structure diagram.

This skeletal model serves as a crucial tool for describing human posture and morphology. Among various representation methods, a tree structure with the centers of both hips as root nodes is a common and intuitive approach. This method clearly illustrates the hierarchical structure of the human body and the connections between its parts. By disregarding external appearance features, this model places greater emphasis on the characteristics of the posture itself. Additionally, due to its simple and intuitive construction, it is applicable in both two-dimensional and 3D human posture estimation algorithm models.

The human model used in this section is an SMPL parameter model. Additional detailed joints have been added to this skeletal framework. This not only enables accurate representation of human posture but also allows for more nuanced depiction of additional features such as body shape and musculature. Subsequently, using this model’s joints as a foundation, the algorithm-generated character model animation skeleton data in UE will be bound to the corresponding joints of the virtual character via IK (Inverse Kinematics), thereby achieving the reproduction of animation effects.

The specific skeleton data acquisition process is shown in Fig. 5.

Fig. 5.

Skeletal data acquisition process. The input sample image is sourced from the AGORA synthetic dataset⁴².

Character animation rendering workflow

The reconstructed poses V from Stage 4 serve as input to the animation rendering pipeline. Using the reconstructed pose data from Stage 4, the human pose sequence frames and associated data streams are packaged into a PK file and saved. Subsequently, the PK file is imported into Blender software for model animation reconstruction. Using the plugin provided on the SMPL official website, the output model animation file is converted to FBX format. This enables the encapsulated, previously rendered human posture video data frame files to be sent to Unreal Engine for inverse kinematics (IK) binding and retargeting. Consequently, the virtual character skeleton is driven by joint data. After binding the relevant skeletal joints, the virtual character can replicate real human movements to recreate the animation effects.

The specific human animation rendering workflow is illustrated in Fig. 6.

Fig. 6.

Human animation rendering workflow diagram.

Result analysis and discussion

Experimental settings

In this study, a hybrid dataset strategy is adopted, involving three datasets from different sources: H3WB, AGORA, and AthletePose3D, to conduct comprehensive training and evaluation of the proposed 3D human pose estimation model. The specific configurations are as follows: image size is set to 256 × 256, heatmap size is 64 × 64. The Adam optimizer is used (OPTIMIZER: ‘adam’), with an initial learning rate of 0.001 (LR: 0.001), and the learning rate is reduced at the 90th and 120th epochs (LR_STEP: [90,120]). The training batch size is set to 16 (BATCH_SIZE: 16), with training starting from the 1st epoch and ending at the 200th epoch. All code is implemented in PyTorch and run on an NVIDIA GeForce RTX 3090. Through this detailed configuration, this study aims to explore and verify the effectiveness of Transformer and self-attention mechanisms in the field of 3D human pose estimation. The detailed experimental platform settings are summarized in Table 1.

Table 1.

Experimental platform settings.

Parameter/setting	Value/description
Optimizer	Adam
Initial learning rate	0.001
Learning rate decay steps	[90, 120]
Batch size	16
Training epochs	1 to 200 epochs
Image size	256 × 256
Heatmap size	64 × 64
Framework	PyTorch
Hardware	NVIDIA GeForce RTX 3090
Datasets	H3WB, AGORA, AthletePose3D

Introduction to datasets and evaluation metrics

The datasets used in this work adopt the widely applied H3WB, AGORA, and AthletePose3D. (1) The H3WB (Human3.6M 3D WholeBody) dataset is an extended version of Human3.6M, specifically designed for 3D whole-body pose estimation. It contains annotations for approximately 100K images, providing 133 whole-body keypoints (including 17 body, 6 foot, 68 face, and 42 hand keypoints), and supports tasks such as 2D to 3D lifting and direct RGB prediction. This work uses this dataset for training and evaluation. (2) The AGORA (Avatars in Geography Optimized for Regression Analysis) dataset is a synthetic benchmark for 3D human pose and shape estimation, featuring 424K high-resolution images with diverse subjects (including adults and children), poses, clothing, and environments. It provides accurate ground truth annotations using SMPL-X body models from 3D scans, emphasizing challenges in occluded and varied scenarios. (3) The AthletePose3D dataset is a benchmark dataset for 3D human pose estimation and kinematic validation in sports movements, containing 3D pose data for various competitive actions, aimed at supporting monocular 3D pose reconstruction. It provides high-precision annotations and is suitable for complex dynamic scenarios. This work uses this dataset for training and evaluation.

Evaluation metrics

This study uses MPJPE (mean per joint position error), PA-MPJPE (procrustes-aligned mean per joint position error), and MPVE (mean per vertex error) as evaluation metrics.

MPJPE measures the Euclidean distance between predicted joint positions (denoted as ) and ground-truth joint positions (denoted as ). It serves as an accuracy metric to assess the difference between true and predicted joint positions. For a human body with T joints, MPJPE is calculated using the following equation:

14

PA-MPJPE is a reconstruction error metric. It quantifies MPJPE after aligning the predicted joints with the ground-truth joints through rigid body transformation. This approach effectively eliminates errors introduced by translation, rotation, and scaling. Thus, PA-MPJPE provides a more precise evaluation of the accuracy of predicted joint positions.

MPVE calculates the vertex error for human body models with shape variations, such as those represented by the SMPL model. Given the predicted vertex positions (denoted as ) and ground-truth vertex positions (denoted as ), MPVE is computed using the following equation:

15

To quantitatively evaluate temporal smoothness, we additionally employ the acceleration error metric (Accel), which measures the jitter in predicted pose sequences. For a sequence of poses over frames t, acceleration error is computed as:

16

where represents the predicted joint positions at frame t. Lower acceleration error indicates smoother motion with less jitter. This metric directly measures temporal continuity, complementing spatial error metrics (MPJPE, PA-MPJPE, MPVE) to provide a comprehensive evaluation of pose sequence quality.

Quantitative results analysis

Comparative analysis

To validate the effectiveness of the proposed method, this study compares it with several mainstream frameworks. Specifically, the Transformer-enhanced 3D human pose estimation model presented in this work is evaluated against three widely adopted Transformer-based baselines for monocular 3D human pose estimation, namely PoseFormer³², MixSTE³³, and MHFormer³⁴, all of which leverage self-attention mechanisms to model long-range spatial-temporal dependencies from 2D pose sequences. To ensure fair comparison, we reimplemented all baseline methods using their official implementations and evaluated them under identical experimental settings, including the same datasets, input preprocessing protocols, and evaluation metrics. These models represent state-of-the-art approaches in the field of 3D human pose estimation that have achieved significant improvements in both accuracy and computational efficiency. All comparisons are conducted on three datasets (H3WB, AGORA, and AthletePose3D) using standard evaluation metrics, including MPJPE, PA-MPJPE, MPVE, and acceleration error (Accel). To explicitly separate backbone and smoothing effects, we further add a fairness-controlled H3WB protocol in which MHFormer and our backbone are each evaluated with and without SmoothNet under the same preprocessing, window length, and evaluation settings (Table 3). We prioritize MHFormer in this controlled study because it is the strongest baseline in our main comparison and therefore provides the strictest reference under limited re-evaluation budget; PoseFormer and MixSTE remain in Table 2 for broad multi-baseline context.

Table 3.

Fairness-controlled comparison on H3WB with unified SmoothNet setting. All positional errors in mm; acceleration error (Accel) in mm/s².

Backbone	SmoothNet	MPJPE	PA-MPJPE	MPVE	Accel
MHFormer34	w/o	50.8	41.0	63.7	10.9
MHFormer34	w/	49.0	39.5	61.1	8.2
Ours (transformer backbone)	w/o	49.6	40.1	62.0	9.6
Ours (transformer backbone)	w/	48.5	39.2	60.2	7.3

Table 2.

Comparison of different methods. All positional errors in mm; acceleration error (Accel) in mm/s².

Model	Dataset	MPJPE	PA-MPJPE	MPVE	Accel
PoseFormer32	H3WB	55.2	45.1	68.4	12.8
	AGORA	62.4	52.6	75.6	15.3
	AthletePose3D	68.7	58.3	82.1	18.7
MixSTE33	H3WB	52.1	42.3	65.3	11.5
	AGORA	59.3	49.8	72.1	14.1
	AthletePose3D	65.2	55.6	78.9	17.2
MHFormer34	H3WB	50.8	41.0	63.7	10.9
	AGORA	57.0	47.5	69.8	13.4
	AthletePose3D	62.8	53.2	75.6	16.5
Proposed	H3WB	48.5	39.2	60.2	7.3
	AGORA	54.2	45.0	66.5	8.9
	AthletePose3D	59.3	50.8	72.1	10.2

Table 2 reports the overall end-to-end comparison across three datasets. The proposed pipeline obtains MPJPE of 48.5mm, 54.2mm, and 59.3mm on H3WB, AGORA, and AthletePose3D, respectively. These gains reflect the combined effect of the Transformer backbone and temporal smoothing. Therefore, architecture-level interpretation is supported by the controlled results in Table 3, where all compared backbones are evaluated under matched SmoothNet conditions.

The evidence is interpreted conservatively: SmoothNet contributes temporal stabilization as an external prior module, while the proposed backbone contributes spatial regression quality and computational efficiency. We therefore avoid attributing all improvements to a single component and report both combined and controlled results. The experimental results are presented in Table 2.

Under the unified H3WB protocol, adding SmoothNet to MHFormer improves MPJPE/PA-MPJPE/MPVE/Accel from 50.8/41.0/63.7/10.9 to 49.0/39.5/61.1/8.2. For our backbone, adding SmoothNet improves the same metrics from 49.6/40.1/62.0/9.6 to 48.5/39.2/60.2/7.3. Thus, MHFormer+SmoothNet can outperform Ours w/o SmoothNet, while under matched SmoothNet usage our backbone still achieves lower errors than MHFormer, indicating separable smoothing-level and backbone-level contributions.

Ablation studies

To validate the effectiveness of the external/prior SmoothNet module for pose smoothing, ablation studies were conducted. This ablation follows the original full-pipeline smoothing protocol used for filter-level comparison, rather than the fairness-controlled backbone protocol in Table 3. The baseline model without SmoothNet was compared with the model incorporating SmoothNet to assess its contribution in reducing pose jitter and improving estimation accuracy. We also compare SmoothNet against two classical filtering methods—One-Euro Filter and Gaussian Filter—to demonstrate its advantages. The One-Euro Filter is a widely used adaptive low-pass filter designed for real-time smoothing, while Gaussian smoothing applies a fixed temporal kernel. For fair comparison, we tune the hyperparameters of both filters on the validation set to achieve optimal performance. The specific ablation results are presented in Table 4.

Table 4.

Ablation studies for the external SmoothNet module (full-pipeline smoothing protocol). All errors in mm.

Configuration	Dataset	PA-MPJPE	MPJPE	MPVE
w/o SmoothNet	H3WB	52.7	86.7	101.2
	AGORA	41.1	55.7	59.0
w/ One-Euro filter	H3WB	52.0	85.5	99.8
	AGORA	40.5	55.0	58.3
w/ Gaussian filter	H3WB	51.5	84.9	98.7
	AGORA	40.0	54.5	57.8
w/ SmoothNet (ours)	H3WB	43.2	72.7	85.1
	AGORA	33.3	54.0	57.3

As shown in Table 4, SmoothNet improves temporal stability compared with no smoothing and classical filters. We use this table only to evaluate the effect of smoothing strategies, not to claim superiority of the backbone architecture. Importantly, Tables 4 and  3 are generated under different experimental protocols and targets, so their absolute values are not intended for direct cross-table comparison. The architecture-level attribution is addressed by the fairness-controlled protocol in Table 3.

In addition, this work conducts an ablation analysis on the content of the loss function (with all loss term weight coefficients set to 1), as shown in Table 5. After adding the rotation angle loss , the model performance is improved across all metrics. After adding the SMPL model losses and , the MPVE metric instead worsens (from 12.5 to 14.2, where lower MPVE is better), while MPJPE and PA-MPJPE continue to improve. This trade-off occurs because and enforce consistency with the SMPL body model, which has a fixed topology and limited shape space. While this constraint improves joint-level accuracy by regularizing anatomically implausible poses, it may increase vertex errors when the true body shape deviates from the SMPL prior. This suggests that SMPL losses are beneficial for pose estimation but may require careful tuning when vertex-level accuracy is critical, such as in applications requiring precise body surface reconstruction.

Table 5.

Loss function ablation analysis. All errors in mm..

Loss components	MPJPE	PA-MPJPE	MPVE
only	45.6	38.2	15.1
	42.1	35.7	12.5
	41.8	34.9	14.2

Computational efficiency analysis

To evaluate the practical feasibility of the proposed method, we measure GPU memory consumption and inference speed compared to baseline models. All experiments are conducted on an NVIDIA GeForce RTX 3090 GPU with batch size 16 and input resolution 256 × 256. The proposed Transformer-based architecture consumes approximately 8.2GB GPU memory during training, compared to 10.5GB for the original deconvolution-based approach—a reduction of 21.9%. This reduction stems from replacing three deconvolution layers with Transformer blocks, which process flattened feature sequences more efficiently than multi-scale heatmap generation. Inference speed is measured at 32 frames per second (FPS) for the base pose estimation model, and 28 FPS when SmoothNet is included. While SmoothNet introduces a minor overhead due to temporal window processing, the overall pipeline remains suitable for near-real-time applications. Compared to PoseFormer (25 FPS), our method achieves faster inference due to the more compact Transformer decoder design.

Failure cases and limitations

Despite the improvements demonstrated, the proposed method exhibits limitations in certain scenarios. First, under severe occlusion (e.g., when over 50% of the body is occluded), the model may produce inaccurate joint positions, as the ResNet backbone struggles to extract reliable features from severely incomplete visual input. SmoothNet can mitigate temporal jitter but cannot recover from persistent mis-detections. Second, extreme poses involving significant self-contact or unusual joint configurations (e.g., yoga poses, dance contortions) occasionally lead to increased errors, as the SMPL prior and training data distribution favor more common upright postures.

Third, while SmoothNet effectively reduces jitter, it introduces a trade-off between smoothness and motion responsiveness. In sequences with rapid directional changes or high-frequency movements—such as boxing punches or basketball dribbling—smoothing may attenuate motion sharpness. To quantify this trade-off, we analyze acceleration error reduction and peak velocity preservation on 50 high-speed motion sequences from AthletePose3D. Results show that SmoothNet reduces acceleration error by 38.2% while preserving 92.3% of peak velocities, compared to 87.5% for Gaussian filtering with equivalent jitter reduction. This indicates that SmoothNet’s learned smoothing adapts better than fixed filters. For applications requiring maximal motion sharpness (e.g., biomechanical impact analysis), smoothing strength can be adjusted via hyperparameter tuning at the cost of partial jitter suppression.

Fourth, performance degrades under extreme lighting or heavy motion blur due to deteriorated 2D feature quality. Fifth, the model trained primarily on indoor datasets shows limited generalization to in-the-wild scenarios. Preliminary evaluation on 20 outdoor videos reveals a 12–15% MPJPE increase compared to indoor settings, suggesting that domain adaptation techniques are necessary for robust outdoor deployment.

Qualitative results analysis

To intuitively demonstrate the effectiveness of the proposed method in animation generation, this study extracts frames at the same timestamps from both the original input videos and the corresponding generated animation sequences. These synchronized static images enable direct side-by-side visual comparison.

The demonstration images are arranged in a consistent layout. The leftmost column displays the human figure from the original video frame. The three middle columns present the animation results produced by the three existing state-of-the-art models (PoseFormer, MixSTE, and MHFormer). The rightmost column shows the animation generated by the method proposed in this work. Representative results are illustrated in Fig. 7.

Fig. 7.

Animation effects generated by different methods.

As shown in Fig. 7, the integrated pipeline produces visually stable animation sequences, especially for actions with fast transitions. The qualitative improvements are mainly associated with temporal refinement introduced by the external SmoothNet module, while the Transformer backbone contributes spatial pose estimation quality. We therefore interpret the visual gains as the combined effect of modular integration rather than evidence of a new smoothing algorithm.

Conclusion

This work presents a Transformer-driven framework for 3D human pose estimation and its application to data-driven animation generation. By replacing conventional convolution-based 3D heatmap regression with a template-guided Transformer decoding mechanism, the proposed model effectively captures long-range spatial dependencies while reducing computational redundancy—achieving a 21.9% reduction in GPU memory consumption compared to deconvolution-based approaches. SmoothNet is incorporated as an external post-processing module to improve temporal consistency, and its role is explicitly separated from backbone contribution through with/without-SmoothNet analyses. Across H3WB, AGORA, and AthletePose3D, the integrated pipeline shows competitive spatial-temporal performance, while the fairness-controlled H3WB protocol provides the basis for component-wise attribution. Ablation studies confirm that both the temporal smoothing module and the newly introduced loss components contribute to accuracy and sequence consistency. The pipeline’s effectiveness is further validated through its deployment in Blender and Unreal Engine, where retargeted animations exhibit reduced jitter, improved motion continuity, and enhanced overall realism.

Despite these advances, several limitations warrant discussion. First, performance degrades under severe occlusions (>50% body coverage) or extreme lighting conditions due to the method’s reliance on ResNet-based 2D feature extraction. Second, SmoothNet introduces a trade-off between smoothness and motion sharpness; high-frequency movements may be slightly attenuated, with approximately 92.3% of peak velocities preserved in high-speed sequences. Third, the model exhibits a 12–15% MPJPE increase on outdoor videos compared to indoor datasets, indicating limited generalization to unconstrained environments. Fourth, extreme poses with significant self-contact remain challenging due to SMPL prior constraints and training data bias.

Future work will address these limitations through several directions. First, exploring lightweight Transformer variants for real-time deployment, potentially leveraging efficient attention mechanisms or model pruning to achieve inference speeds exceeding 50 FPS on consumer hardware. Second, incorporating domain adaptation techniques for in-the-wild scenarios, such as unsupervised domain alignment or test-time adaptation, to improve robustness to diverse backgrounds and camera viewpoints. Third, integrating tighter coupling with physics-based animation systems to ensure biomechanical plausibility, enforcing physical constraints like ground contact and balance during pose optimization. Fourth, extending the framework to multi-person scenarios with occlusion handling, enabling animation generation for crowded scenes. Finally, investigating adaptive smoothing strategies that dynamically adjust temporal filtering strength based on motion characteristics, optimally balancing jitter suppression with motion sharpness preservation.

Author contributions

Xinyi Xu: Writing - original draft; Writing - review editing; Conceptualization; Resources; Data curation; Methodology; Supervision; Formal analysis.

Data availability

All materials supporting the findings of this study are available at: https://doi.org/10.6084/m9.figshare.30741854. In addition, the publicly available datasets used in this research can be accessed from their official sources as follows: H3WB dataset: https://github.com/wholebody3d/wholebody3d AGORA dataset: https://agora.is.tue.mpg.de/ AthletePose3D dataset: https://github.com/calvinyeungck/AthletePose3D

Declarations

Competing interests

The authors declare no competing interests.