Empowering Efficient Spatio-Temporal Learning with a 3D CNN for Pose-Based Action Recognition

. 2024 Nov 30;24(23):7682. doi: 10.3390/s24237682

Algorithm 1 Details of pose-based action recognition framework with PoseTransformer3D

Input:

V_{R G B}

V_{S k e l e t o n}

Output: Action recognition classification.

1:
Stage1: Generated 3D heatmap volumes
2:
for each $R G B a n d S k e l e t o n$ do
3:
for each $S e g m e n t S$ do
4:
for each $S a m p l i n g T$ do
5:
$V_{2 D P o s e} (T) = f_{D E} (V_{R G B} (T), V_{S k e l e t o n} (T)) \leftarrow$ Detection + Pose estimation
6:
end for
7:
$V_{3 D h e a t m a p v o l u m e s} = f_{S P} (V_{2 D P o s e}) \leftarrow$ Stack + Preprocessing
8:
end for
9:
end for
10:
Stage2: Pre-training a 3D-CNN for classifying 3D heatmap volumes
11:
for each $S a m p l i n g S$ do
12:
$p (c ∣ V) = log \frac{e^{(W_{c} V + b_{c})}}{\sum_{c_{i} = 1}^{C} e^{(w_{c_{i}} V + b_{c_{i}})}} \leftarrow$ Features extraction with 3D-CNN
13:
end for
14:
Stage3: PoseTransformer3D classifies 3D heatmap volumes based on 3D-CNN pre-training
15:
for each $S a m p l i n g S$ do
16:
$p {(c ∣ V)}_{3 D - C N N} = log \frac{e^{(W_{c} V + b_{c})}}{\sum_{c_{i} = 1}^{C} e^{(w_{c_{i}} V + b_{c_{i}})}} \leftarrow$ Features extraction with 3D-CNN Backbone
17:
$p {(c ∣ q)}_{G C B} = log \frac{e^{(W_{c} q + b_{c})}}{\sum_{c_{i} = 1}^{C} e^{(w_{c_{i}} q + b_{c_{i}})}} \leftarrow$ Features extraction with GCB
18:
$p (c ∣ (V, q)) = λ_{3 D - C N N} p {(c ∣ V)}_{3 D - C N N} + λ_{G C B} \sum_{m = 1}^{M} p {(c ∣ q)}_{G C B} \leftarrow$ Features extraction with PoseTransformer3D
19:
end for
20:
return action classification