Deep learning-based 2D keypoint detection in alpine ski racing – A performance analysis of state-of-the-art algorithms applied to regular skiing and injury situations

Abstract

Objectives

In this study, we examined the practicability of deep learning-based 2D keypoint detection applied to regular skiing and injury situations (i.e., out-of-balance situations and fall situations) on an alpine ski racing track.

Methods

We therefore created a regular skiing- and injury situation-specific dataset (hereinafter called "Injury Ski Dataset"), on which the state-of-the-art keypoint detection algorithms OpenPose, Mask-R-CNN, AlphaPose and DCPose were compared. The performance of each keypoint detector was evaluated by calculating the mean per joint position error (MPJPE) and the percentage of correct keypoints (PCK). Failure cases and common error patterns were further investigated by a visual analysis.

Results

We observed the best results for regular skiing, with 81%–92% of all keypoints detected correctly at an MPJPE of 9 (2) to 14 (3) pixels. In injury situations, self-occlusions and rare poses became more likely, similar to occlusions due to snow spray and motion blur. As a result, the performance in out-of-balance situations decreased to 68%–80% (PCK), while in fall situations, only 35%–54% of all keypoints were detected correctly, with mean errors of 26–36 pixels. Among all algorithms, AlphaPose was the most robust and achieved the best results.

Conclusions

PCK and MPJPE for regular skiing were in the range of manual annotation errors and can be considered low enough for further biomechanical analysis. For fall situations, keypoint detection should be further improved. Regarding the development of a deep learning tool for injury analysis in alpine skiing in the future, we propose to fine-tune a well-performing keypoint detector, such as AlphaPose, on a ski- and injury-specific dataset, such as ours.

1. Introduction

Alpine ski racing is considered an extreme sport, exposing its athletes to a very high risk of injury [[1], [2], [3], [4]]. While previous studies found every third World Cup athlete to be injured in one season, every seventh athlete suffers from a severe injury, leading to at least 28 days of absence in training and competition [2,5]. In addition, a recent study showed an increase in severe injuries among Austrian World Cup and European Cup athletes from approximately 11 in 1997 to 23 in 2019, with approximately one more injured athlete among 100 athletes every two seasons [1]. Among these injuries, the knee was found to be the most frequently injured body part [2,[5], [6], [7]], with rupture of the anterior cruciate ligament (ACL) being the most frequent diagnosis, accounting for half of all severe injuries [1].

Working towards effective prevention measures, further scientific efforts are necessary to gain a deeper understanding of risk factors and injury mechanisms in alpine ski racing. For many such research projects, reliable kinematic data of real injury events are essential. On the one hand, having a full kinematic description can help to better understand the injury mechanisms and identify the events leading to an injury-prone situation [8,9]. On the other hand, they may serve as input for biomechanical simulations, such as finite element [10] or musculoskeletal simulations [11], which provide additional information such as the underlying muscle tension or the loadings of the ACL. However, operating in an outdoor environment, it is very challenging to collect reliable kinematic data for regular alpine skiing turns [[12], [13], [14], [15]] and injury situations in particular [8,9]. Current methods primarily make use of systems with multiple synchronized video cameras, i.e., video-based stereophotogrammetry [13,[16], [17], [18], [19]], wearable measurement technologies [8,12,[20], [21], [22], [23]] or biomechanical simulations [11,[24], [25], [26]].

As real injury events cannot be reproduced under experimental conditions, the abovementioned methods are unfeasible for capturing such events unless an accidental injury occurs during a regular experiment [8]. In most cases, the only data available are video recordings provided by television broadcasters or trainers. This means that an injury situation is usually recorded from only a single perspective, with an unknown camera position and no calibration. Under these circumstances, the recovery of kinematic data is very challenging and has thus far only been accomplished by time-consuming manual matching techniques [27]. Applying such a manual matching procedure, Bere et al. [27,28] were able to reconstruct the kinematics of 2 ACL injury cases. However, the annotations required weeks to months, which obviously limits the use of this method to the analysis of a small number of very short and specific video sequences.

Deep learning-based human pose estimation may overcome the limitations mentioned above, as it can be applied to reconstruct kinematic features incredibly fast given a single-view video only. These algorithms are designed to locate human body parts within an image or video (2D keypoint detection) and subsequently reconstruct a three-dimensional representation from these detections (3D human pose estimation). As they require no sensory data other than videos or a sequence of single images, they are already used in a wide range of applications, including human–computer interaction, motion analysis, augmented reality, and video surveillance [29]. While 3D human pose estimation still faces many challenges, such as depth ambiguities and a lack of 3D training data for rare and complex motions [29], state-of-the-art 2D keypoint detection algorithms [[30], [31], [32], [33]] are already capable of estimating human joint locations in real time with impressive accuracy [29].

In particular, this applies to standard motions, where large amounts of training data are available. Activities that greatly differ from such standard motions and for which ground truth data are hard to collect, e.g., alpine skiing or fall situations, remain challenging [29,34]. For alpine ski racing, only two relevant public datasets exist. The publicly available ski 2DPose dataset created by Bachmann et al. [35] covers approximately 2.5k video frames of alpine ski racers in various disciplines and weather conditions, plus their corresponding 2D poses. Second, Spörri et al. [36] created a ski-specific 3D dataset containing 20k images and their respective 3D poses, including skis and poles, all recorded on a single giant slalom course in a capture volume of 3 subsequent turns. Rhodin et al. [34] were the first to assess a skier's 3D pose from a single view camera using a neural network. Building on this work, Ostrek et al. [37] showed that the performance of a deep learning-based approach can be viable for detecting relevant kinematic differences in the context of alpine skiing [37]. Bachmann et al. [35] implemented a 3D bundle adjustment pipeline to reconstruct a skier pose in a multicamera setting. As part of their pipeline, they trained a keypoint detection algorithm on their ski-specific 2D dataset [31]. However, to date, it remains unclear how well state-of-the-art keypoint detection algorithms work in injury-related out-of-balance and fall situations.

The main objectives of this research were to create an injury-specific test dataset (hereinafter called "Injury Ski Dataset") and to evaluate the performance of state-of-the-art keypoint detection algorithms applied to regular skiing and injury situations, including out-of-balance situations and fall situations. We further aimed to identify and understand difficulties and error patterns to improve keypoint detection within the scope of developing a deep learning-based human motion capture pipeline for injury situations in alpine ski racing.

2. Methods

2.1. Injury Ski Dataset

As poses and motions in an injury situation may differ greatly from regular skiing turns, an injury-specific 2D dataset was created within the scope of this research. Over the last twenty years, we were able to gather a large collection of injury recordings in alpine ski racing. This very diverse collection covers over 150 videos of male and female ski athletes in all disciplines, with video qualities ranging from very poor to high definition. To evaluate the performance of the different 2D keypoint detectors, seven injury recordings were selected. Providing an adequate representation of all available injury videos, each of the alpine skiing disciplines, slalom (SL), giant slalom (GS), super-g (SG) and downhill (DH), as well as male and female athletes, are represented in this selection. Additionally, a wide range of video qualities, spanning from low to very high, and skier sizes were included in the study. The skier sizes varied from approximately 100 to over 600 pixels, considering a standard resolution of 1280x720 pixels. Only videos with extremely low quality that could not be annotated manually in a meaningful way were excluded. Depending on the length and frame rate of each injury recording, 50 to 100 frames per video were sampled in equidistant time steps and annotated manually using a custom LabVIEW script for digitalizing video frames. To facilitate and speed up the digitalization process, each joint was annotated for all frames per video, while images were automatically centered around the position of the previous joint coordinates. Making it easier to focus on the respective joint, this method obviously increases the annotation accuracy. However, manual annotation still comes with high uncertainty that greatly depends on the image quality, the skier's resolution, and occlusions. While some joints, such as the elbow, wrist, knee, or ankle, are relatively easy to determine, also in high-quality images, it is very challenging to provide high accuracy when dealing with joints such as the hip or neck. As a rule of thumb, the neck joint was identified as the intersection point of the shoulder axis and the cervical spine, while the hip joints were defined by the intersection points of the hip axis and the femur. Following the skier model of [35] (Fig. 1), 24 keypoints were annotated per frame, including 14 body joints, both skis (8 keypoints), and poles (2 keypoints). Table 1 shows an overview of the annotated video sequences. In total, 533 frames were annotated. This Injury Ski Dataset was made publicly available for further research and can be downloaded here at https://sport1.uibk.ac.at/mz/cv.

Fig. 1.

Skier model with 24 keypoints, including 16 body joints, 8 ski and two pole keypoints, as used by Bachmann et al. [35].

Table 1.

Overview of all seven selected injury recordings. Frames were annotated equidistantly. The number of annotated frames N_A depended on the total number of frames N_F and frame rate. In total, 533 frames were annotated.

Subject	Sex	Discipline	NA	NF	Skier Size (pixel)	Resolution
Subject 1	m	SL	75	150	116	Low/medium
Subject 2	w	SL	50	100	109	Low
Subject 3	m	GS	50	250	239	High
Subject 4	m	DH	100	300	248	Medium/high
Subject 5	m	GS	100	300	145	Medium/high
Subject 6	w	GS	62	125	629	High
Subject 7	m	SG	96	480	438	High

Keypoint detection algorithms should perform well on frames where the pose and training samples are similar, e.g., motions where the person is mostly upright [30,31]. In fall situations, however, skiers’ poses and training samples differ greatly. To check the performance of the models for different skiing situations, the frames of each selected video were split into regular skiing, out-of-balance and fall situations. As shown in Fig. 2, regular skiing situations contain all situations where the skier shows controlled skiing. Out-of-balance situations cover situations where the skier is out of control but still on his skis, trying to regain his balance. Finally, fall situations that contain all frames where the skier is already hitting the ground or having no ground contact at all. A visual analysis as well as a quantitative evaluation of performance was conducted regarding this categorization, as described in the following section.

Fig. 2.

Examples of (a) regular skiing, (b) out-of-balance and (c) fall situations. The detected 2D pose is visualized as a skeleton overlay with detected limbs colored.

2.2. Selection of keypoint detection algorithms

We compared four state-of-the-art keypoint detection algorithms on our injury-specific test dataset. The detection algorithms OpenPose [31], Mask-R-CNN [32], DCPose [30] and AlphaPose [33] were chosen. All algorithms as well as corresponding pretrained models are publicly available online. OpenPose and Mask-R-CNN are probably the most widely used state-of-the-art keypoint detection algorithms. AlphaPose was chosen because it achieved impressive results on benchmark datasets and was constantly improved after publication [38]. DCPose tackles some of the challenges that arise when dealing with video recordings by taking temporal information across a series of neighboring video frames into account and was pretrained on the video-based Posetrack dataset.

2.3. Evaluation of keypoint detection algorithms

All experiments were run on a desktop computer housing an Nvidia RTX2060 graphics card. After setting up a virtual Anaconda environment for each keypoint detector and downloading the respective GitHub repositories, installation was completed following the installation manuals. The videos of our injury dataset, as described above, were processed by all four networks. Mean per joint position errors (MPJPE) as well as the percentage of correct keypoints (PCK) were evaluated using custom Python3 scripts regarding the three distinguished categories regular skiing, out-of-balance situations, and falls. To calculate the MPJPE, all per joint position errors (PJPE), defined as the Euclidean distance between the predicted and true joint locations in pixels, were calculated for all visible keypoints. Taking the mean over a respective set of frames gives the MPJPE. The PCK was obtained by dividing the number of keypoints with the PJPE under a certain threshold by the total number of (visible) keypoints. In our case, the threshold was set as 0.2 times the maximal distance between the hip and shoulder joints. To compare the results visually, the predicted keypoints were overlaid with each corresponding input frame. By going through all single frames individually, the strengths and weaknesses of the different models were observed.

3. Results

As shown in Table 2, Table 3, all keypoint detection algorithms investigated in this study show a high performance for regular skiing, with 81%–92% of all keypoints detected correctly. While in out-of-balance situations, PCK dropped to 68%–80%, only one-third to half of all keypoints in fall situations were detected correctly. Across all categories, AlphaPose performed best, with an overall PCK of 81%, followed by DCPose and Mask-R-CNN. OpenPose could only detect 70% of all keypoints correctly. Accordingly, we found the lowest MPJPE for all keypoint detection algorithms for regular skiing situations, with mean errors ranging from 9 to 14 pixels. At an average skier size of about 280 pixels, this allows a rough estimate of the real-world error to be just under 5 ​cm. The MPJPE values roughly doubled for out-of-balance situations and peaked in fall situations with 36 pixels (OpenPose) to 27 pixels (AlphaPose) of mean error. Across all categories, AlphaPose again showed the lowest MPJPE (16 pixels), closely followed by DCPose (17 pixels), Mask-R-CNN (19 pixels) and OpenPose (21 pixels).

Table 2.

Mean per joint position error (MPJPE) in pixels regarding each model and category as well as across all categories. The standard error is given in parentheses.

Model/Situation	Regular skiing	Out-of-balance	Falls	All
OpenPose	14 (3)	21 (4)	36 (10)	21 (4)
Mask-R-CNN	11 (2)	20 (3)	33 (7)	19 (3)
DCPose	9 (1)	17 (2)	31 (5)	17 (2)
AlphaPose	9 (2)	17 (3)	27 (8)	16 (3)

Table 3.

Percentage of correct keypoints (PCK) and standard error regarding each model and category. The standard error is given in parentheses.

Model/Situation	Regular skiing	Out-of-balance	Falls	All
OpenPose	0,81 (8)	0,68 (8)	0,35 (9)	0,70 (8)
Mask-R-CNN	0,87 (5)	0,75 (4)	0,42 (6)	0,75 (6)
DCPose	0,89 (4)	0,80 (5)	0,44 (7)	0,78 (6)
AlphaPose	0,92 (2)	0,77 (5)	0,54 (8)	0,81 (4)

To obtain an estimate of the annotation accuracy for this particular dataset, we digitized a small subset of 100 frames twice by two different annotators and calculated the MPJPE between these two annotated sets. We thereby found a mean deviation of 5 pixels.

Looking at all cases where a large deviation between prediction and annotation was observed, carefully, we identified three major error patterns. An example of each error pattern is shown in Fig. 3. Most failures were attributed to occlusions, which occurred when the skier was hidden from view by external objects such as gates, trees, or the terrain. Furthermore, we found many body parts hidden due to snow spray, and depending on camera angle and pose, some occlusions were caused by the skier itself (self-occlusion). Especially in out-of-balance and fall situations, as the skier became twisted and crunched, self-occlusion and occlusions due to snow spray became increasingly popular. However, in some frames, poses were detected incorrectly, even if the skier was completely visible. In most of these cases, the skiers’ poses were rather unusual, e.g., when he/she was inverted or horizontal during a fall or the pose got twisted, crunched, or crossed limbs. Additionally, recordings with poor image quality (low resolution, small skier size or motion blur) were a major challenge for all keypoint detection algorithms.

Fig. 3.

Most common error patterns across all keypoint detectors due to occlusions, unusual poses, and low-quality video recordings.

4. Discussion

The main findings of the study were as follows: (1) State-of-the-art keypoint detection algorithms performed very well on high-quality videos of regular skiing. (2) Their performance decreased towards out-of-balance situations and dropped in fall situations. In these conditions, occlusions and rare poses were found to become more frequent. (3) Among all four keypoint detection algorithms investigated in this study, AlphaPose performed best.

The datasets on which neural networks are trained are of great importance. These datasets are a large collection of images depicting humans, paired with the corresponding keypoints, i.e., shoulder, hip, knee, etc., of each visible person. While only a few large motion capture datasets exist that cover all kinds of human motions in the wild [[38], [39], [40]], there is only one publicly available 2D dataset of alpine ski racers. This dataset created by Bachmann et al. [35] contains 2.5k video frames of alpine ski racers in various disciplines and weather conditions. However, all these samples show regular skiing situations and no out-of-balance or even fall images. As poses and motions in an injury situation may differ greatly from regular skiing turns, a major contribution of this study was the collection of the first injury-related 2D dataset for alpine ski racing. The Injury Ski Dataset was made publicly available for further research. In this first step, the dataset was used to evaluate the performance of state-of-the-art keypoint detection algorithms.

Visual and quantitative analyses revealed good results for regular skiing, while the network performance decreased in out-of-balance situations, and most predictions failed in fall situations. With a PCK between 81% and 92% and MPJPEs of 9–14 pixels, our results for regular skiing can be considered high enough for further biomechanical research. In particular, the MPJPE values for DCPose and AlphaPose in this category were on average just four pixels above manual annotation accuracy, which was determined by annotating a small subset of our dataset twice. Furthermore, MPJPE results for OpenPose during regular skiing are in good agreement with the findings of Bachmann et al. [35]. Depending on the training configuration, they reported MPJPE errors down to 11 pixels at a higher PCK of up to 98%. The difference in PCK might be explained by the use of a different PCK metric, which does not use 20% of the torso diameter but 5% of the full body diagonal as a threshold. In out-of-balance situations, the algorithms still performed quite well, showing a PCK between 68% and 80%. Visual analysis revealed that most poses in this scenario resemble training samples, and occlusions due to snow spray are less frequent than in fall situations. Many injuries, in particular ACL injuries, occur in this phase and may therefore be well analysed with present keypoint detectors. However, in fall situations, athletes can become inverted and crunched, and therefore, self-occlusions and unusual poses become more frequent, similar to occlusions due to snow spray and motion blur. Such fall situations pose a major challenge for keypoint detection algorithms, leading to a drop in PCK (35%–54%) and an increase in MPJPE up to 36 pixels.

Similarly, other authors [30,31] identified occlusions, rare pose appearances in the training datasets and high-speed motions as the main difficulties in 2D keypoint detection. To analyse ski injuries that accompany a fall, further improvement in keypoint detection is needed. Comparing all four algorithms, AlphaPose performed best in PCK, followed by DCPose. Regarding the MPJPE metric, AlphaPose and DCPose performed similarly for regular skiing and out-of-balance situations, while AlphaPose again takes the lead in fall situations. This result was initially surprising as DC Pose was the only one of the examined algorithms that took temporal constraints into account, and thus should have provided better results, especially in difficult and complex cases. However, additional methods [41] have been developed that exploit the temporal continuity of movements to provide better keypoint estimates, particularly in falling situations. With these methods, a reduction in error of up to 30% has been achieved in fall situations.

The impact of 2D keypoint detection on biomechanical analysis is manifold. 2D keypoint detectors can be directly applied to address biomechanical problems that can be analyzed in one plane. In alpine skiing, this includes motions such as jump landings or take-off behavior. For instance, in the sagittal plane, one can analyze parameters such as jump height, distance, joint angles or forward/backward lean, while in the frontal plane, asymmetrical landing positions can be measured. For more complex movements that require a consideration beyond a single plane, 3D kinematics are necessary. In this regard, 2D keypoint detection algorithms can assist in accelerating biomechanical analysis in two different ways. First, automatic keypoint detection can be combined with classical reconstruction methods such as Direct Linear Transformation (DLT) to automate the detection of joints in these procedures. Second, it serves as an essential initial step in many fully computer vision-based pipelines, significantly influencing the overall method's error. Our results indicate that particularly for less complex movements and high-quality videos, a similar error can be expected as with manual digitization, but with a substantial gain in time and potential applications.

A major limitation of this study was the small size of our injury dataset. With only 533 frames in total and only a fraction of these frames in each category (regular skiing, out-of-balance, and falls), the selection of videos has a great impact on the results. Choosing different injury recordings, e.g., showing more/fewer occlusions or more/fewer complex poses, would greatly change the overall performance as well as the performance in each category. Furthermore, only pretrained models provided by the authors were used in this study. Knowing the performance using pretrained models might be useful. However, it would also be very interesting to train all algorithms on ski-specific data first and compare them afterwards.

5. Conclusion

Based on the findings of this study, we conclude that state-of-the-art keypoint detection algorithms provide viable 2D kinematics for regular skiing situations, particularly if the video quality is high. This already opens many applications, such as performance or even injury analysis, as many injuries (e.g., knee injuries) arise, while the skier is still skiing regularly or just slightly out-of-balance. Injuries that occur in a fall situation, e.g., fractures, concussions, or head injuries, however, cannot be captured by standard keypoint detection algorithms in their pretrained configuration. For these scenarios, we propose to fine-tune a well-performing keypoint detector, such as AlphaPose and DCPose, on a ski- and injury-specific dataset. Our injury-specific dataset can make an important contribution here. Additionally, independent postprocessing, as described by [41]^, was shown to improve keypoint detection in difficult frames and fall situations. The development of a deep learning-based motion capture tool for injury analysis in alpine ski racing will enable us to collect a large amount of injury kinematics and will therefore be a key step for injury prevention in alpine ski racing.

Funding

The authors disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This work was supported by the University of Innsbruck within the Doctoral Scholarships Support Program for Junior Scientists.

Declaration of competing interest

We have no conflicts of interest to disclose.