A Multimodal Dataset for Clinical Gait Analysis compromising Optical Motion Capture and Textual Gait Analyses from Patients undergoing Total Knee Arthroplasty

Abstract

We present a multimodal dataset¹ for automated gait analysis in patients undergoing Total Knee Arthroplasty (TKA). The dataset combines quantitative motion capture data with expert clinical gait assessments in textual form. The dataset includes recordings from 23 patients during a standardized six-minute walking test (6MWT), a clinical protocol used to assess walking endurance and functional mobility. Of these patients, 15 were recorded both pre-operatively and six weeks post-operatively, while the remaining patients were recorded only pre-operatively. Motion data were acquired using a markerless SIMI Motion system and are provided as 3D keypoint trajectories and joint angle time series. In addition, parametric body representations based on the Skinned Multi-Person Linear (SMPL) model were obtained through a separate post-processing pipeline applied to the multi-view recordings. For each recording session, three to six clinical gait assessments were obtained through structured interviews with physiotherapists. These annotations describe gait characteristics, deviations, and compensatory mechanisms. The dataset enables research on linking biomechanical motion data with clinical descriptions, supporting applications such as automated gait assessment, anomaly detection, and analysis of early post-operative recovery.

Background & Summary

Gait analysis provides essential insights into human locomotion and is widely used in clinical diagnostics, rehabilitation planning, and the development of assistive technologies². A detailed understanding of a patient’s walking, including deviations from normative gait, is important for correct assessment and potential treatment, especially in total knee arthroplasty³.

Several recent datasets present motion capture data from walking patterns and analyses of gait anomalies. Reis et al.⁴ contains motion capture data from patients before and after knee arthroplasty, but their analysis focused primarily on spatiotemporal parameters such as stride length and cadence. Hwang et al.⁵ applied machine learning methods to classify abnormal gait patterns, but their data were collected from healthy participants instructed to imitate pathological gait, rather than from clinically diagnosed patients. To represent human motion, a variety of modeling approaches are commonly used. Skeleton based models, such as the Halpe Body keypoint format⁶, capture only the 3D trajectories of anatomical landmarks. In contrast, parametric human body models like SMPL⁷ describe motion through joint rotations together with individualized body shape. Large scale datasets such as AMASS⁸ unify multiple motion capture collections into the SMPL representation and provide large amounts of human motion data across more than 300 participants. Among these, for instance the KIT Motion language dataset⁹ contains walking sequences at different speeds and includes short action descriptions. Building on such resources, the HumanML3D dataset¹⁰ pairs human motion sequences with textual action labels. However, these descriptions remain high level, and detailed characteristics of gait are often not captured. Moreover, none of the existing datasets combine motion trajectories with expert clinical gait assessment.

The dataset provides a clinically annotated multimodal representation of gait, including partial longitudinal coverage with pre- and post-operative recordings at six weeks for a subset of patients following total knee arthroplasty. It comprises motion capture data in two complementary representations: (i) full-body 3D keypoint trajectories obtained from the markerless SIMI Motion system, and (ii) SMPL-based parametric body models derived through post-processing. The SIMI data provide temporally resolved 3D joint trajectories, while the SMPL representation encodes body shape and pose parameters, together with the fitted 3D keypoints used during the optimization process. In addition, the dataset includes detailed clinical gait assessments from experts, available in both the original German and an English translation. Our dataset¹ supports exploratory research in areas such as gait analysis, anomaly detection, and the development of motion generation or motion understanding models. It is particularly suited for proof-of-concept studies and early-stage investigations of gait characteristics, including early post-operative changes. By combining markerless motion capture with expert derived clinical annotations, it offers a valuable resource for research in human motion science.

Parts of the dataset have already been utilized in prior work. Specifically, a subset consisting of 11 pre-operative patients was used in our study¹¹ to evaluate automated gait assessment approaches. This initial use case represents only a limited subset of the available data and serves to illustrate the broader research potential of the dataset. The prior use does not affect the completeness of the dataset presented here.

Methods

Motion data recordings

In cooperation with Klinikum Schloss Werneck and SIMI Motion¹², a total of 23 patients undergoing knee arthroplasty were recorded before surgery, and 15 of them were recorded again six weeks post-operatively. The average age of the participants was 67.52 years (with standard deviation of 6.15), and the gender distribution was balanced, with 12 females and 11 males. For 13 patients, the affected knee was on the right side, while 10 patients had left sided arthroplasty.

The recordings were conducted in a closed indoor environment (see Fig. 1a), where the start and end markers for the 6MWT were placed approximately six meters apart. During each recording session, participants walked continuously for six minutes, repeatedly moving in a straight line from the starting point to the end marker, turning around, and walking back. During this time nine recordings were made: three directly at the beginning, three after 3 minutes and three at 6 minutes. For the dataset, only the forward-walking segments were manually extracted from these continuous recordings for further processing. In some trials, portions of the trajectory beyond the end line are retained in the dataset. This results from the recording procedure, as recordings were not always terminated exactly at the end marker, leading to slight variations across trials. This does not affect the analysis, as the relevant phases of gait initiation and steady forward walking are consistently captured.

Fig. 1.

Overview of the recording setup and corresponding skeletal rendering. (a)Setup and room of the recordings before the operation (rendered by EasyMocap¹⁴). (b) Skeleton rendering generated with SIMI Motion¹².

The post-operative recordings took place six weeks after surgery. These recordings were performed in a different room, although the experimental setup and calibration remained unchanged. These factors may introduce additional variability and should be considered when interpreting the data. Due to participant dropout, the dataset contains complete longitudinal (pre/post) recordings for 15 patients. We provide information about the participants metadata in the MetaData_Werneck.xlsx file.

SIMI Motion¹² provided and processed the optical markerless motion capture system including eight different perspectives on the subject. The system operated at 80 Hz, and the external calibration as well as the camera parameters used for triangulation remained unchanged throughout each participant’s recording session. The outcome consisted of skeletal keypoint trajectories and joint angle time series (see Fig. 4a). Joint angles are reported as Euler angles following an XYZ rotation sequence (flexion/extension, abduction/adduction, internal/external rotation). Since SIMI Motion uses their standard protocol that is not aligned with the models commonly used in computer vision or animation, we additionally fitted the motion to the SMPL parameters⁷. The advantage of SMPL over the SIMI format is that there exist many motion datasets, such as the AMASS dataset⁸, containing a variety of motions from healthy subjects including normal gait sequences. Additionally¹³, offers the capability to convert the SMPL to in sports and health widley used OpenSIM skeleton. To retrieve the SMPL parameters we used the freely available EasyMocap¹⁴ tool, which produced a SMPL compatible reconstruction of each walking sequence based on the eight synchronized camera views. A validation of the EasyMocap output is given in section Technical Validation. Our study was approved by the Ethical Committee of the German Research Center for Artificial Intelligence, and all records comply with the Declaration of Helsinki.

Fig. 4.

Location of the anatomical keypoint definitions from (a) SIMI Motion and (b) the SMPL model with the corresponding indices. (a) Keypoint locations (CSV) adapted from SIMI Motion¹². (b) Keypoint locations of the 3D coordinates in the SMPL files (NPZ).

Textual gait analysis descriptions

In addition to motion capture recordings, we collected three to six high quality professional clinical gait assessments for each patients’ recording session, both before and after surgery. These assessments were obtained through structured expert interviews conducted in cooperation with Klinikum Schloss Werneck and Klinikum Hescuro Bad Kissingen/Bad Bocklet in Germany.

Experts evaluated privacy preserving skeletal renderings shown simultaneously in three standardized views: posterior, left lateral, and right lateral (Fig. 1b). These perspectives were chosen as they are the preferred views of physiotherapists. A total of 41 clinical practitioners, primarily physiotherapists with an average of 13.28 years of experience (with standard deviation of 10.12 years), participated.

Each session involved two experts working pairwise the following way:

1. The first physiotherapist independently viewed the skeletal renderings without time constraints and provided an oral clinical gait assessment.

2. The second expert reviewed the first expert’s description together with the skeleton video to ensure contextual alignment, and both experts then engaged in a collaborative discussion.

The oral analyses were recorded, transcribed, and cleaned for readability. The cleaning process included removing unnecessary filler words and summarizing the content into bullet points instead of plain text. Expert identities were anonymized, and all original audio recordings were deleted after transcription. Gait analysis was performed on a single representative forward walking trial per patient. This selection was motivated by the expert annotation process, which is time-intensive and requires multiple clinicians. While additional trials were recorded and are included in the dataset, they were not annotated. Therefore, the dataset is not intended to capture within-subject variability, but to provide consistent expert assessments for representative gait sequences. While these annotations provide valuable qualitative insights, they also come with certain limitations. The resulting annotations reflect qualitative, expert-derived clinical assessments and are inherently subjective, as is typical for clinical gait analysis. They are not intended to replace standardized outcome measures, but to capture clinically meaningful movement descriptions and expert reasoning in natural language. The experts evaluated only privacy preserving skeletal renderings (Fig. 1b), without access to the raw video recordings. To ensure privacy and anonymity, only anonymized skeletal or avatar representations were shown, and all transcript content has been fully anonymized.

Processing

Due to the recording setup, participants were asked to remain stationary for two seconds on the starting line before starting to walk straight. These initial frames were manually removed from both SIMI and SMPL motion capture data. Joint angles and keypoint positions are stored as CSV files and were processed using SIMI Motion software¹².

The SMPL parameters generated with EasyMocap¹⁴ were further processed as follows. The pose parameters are represented in axis-angle format and stored as vectors of length 156, where the first 72 values correspond to the standard 24 SMPL joints. The remaining entries are zero-padded to match the SMPL-H format, which is commonly used in human motion datasets and embedding frameworks. In addition, body shape coefficients (betas) and global pelvis translation (trans) are included for each frame. Since betas are not estimated by EasyMocap, the present data use the default SMPL values and do not include dynamic blend shapes. We applied a zero lag fourth order Butterworth low pass filter (cutoff frequency: 6 Hz, sampling rate: 80 Hz) using filtfilt¹⁵ to smooth the pose parameters (applied on five motion sequences). To detect pelvis orientation outliers, we estimated the main forward direction of each walking sequence and identified frames whose pelvis orientation exhibited large angular deviations from this main direction. In addition, we flagged frames with unrealistic frame to frame velocity changes to capture further unnatural rotations. Detected anomalies across all dimensions were corrected using spherical linear interpolation (SLERP), as pose parameters are represented in axis-angle format. Unlike linear interpolation, SLERP ensures smooth and physically consistent transitions by interpolating rotations along the shortest path on the unit sphere. Detected pelvis orientation outliers and frames with unrealistic velocity changes were rare and did not occur regularly (applied on 37 motion sequences). These corrections were applied to keep the SMPL sequences smooth and consistent over time. Afterwards, the entire SMPL sequence was globally rotated so that the model’s forward direction aligned with the positive X-axis, matching the KIT Motion Language dataset.

The EasyMocap pipeline does not provide the global orientation in the first three pose parameters as in the standard SMPL format. Instead, it outputs 69 pose parameters (23 joints) and an additional ‘Rh’ vector encoding the global rotation (pelvis). To maintain compatibility with the SMPL convention, we modified ‘Rh’ to obtain the standard 72 parameter SMPL pose vector corresponding to 24 joints. EasyMocap additionally provides the detected 3 d keypoints from the processing pipeline which we additionally included as an additional key 3d_positions to the NPZ files, a compressed NumPy-based file format commonly used to store multiple arrays in a single file. Note, that the 3D keypoints included in the NPZ files are not part of the standard SMPL model. These coordinates are produced by EasyMocap during the fitting process and represent the joint centers that EasyMocap estimates from the multi-view input videos. As a result, their anatomical definitions differ from both the official SMPL joint locations and the SIMI Motion keypoints. They should therefore be interpreted only as fitting related outputs rather than as SMPL-standard joints. To ensure compatibility with the AMASS dataset, we padded the pose parameter vector with zeros to reach the 156 dimensional SMPL-H format. Note, that the EasyMocap 3D keypoint positions, only partially correspond with the positions of the SIMI skeleton (see Fig. 4). In order to avoid inconsistency among the other available datasets, which include also textual descriptions of the performed actions, only the SMPL parameters were globally rotated to remain consistency with other datasets such as the KIT Motion Language Model. Because SMPL follows conventions common in computer vision research, most notably in AMASS, we adopted this alignment. The original 3D keypoints were not rotated to maintain consistency with the original SIMI Motion output. For instance, the KIT dataset provide a variety of walking sequences, and therefore serves as an important reference for comparison with our pathological gait recordings.

The audio recordings from structured expert interviews were transcribed and manually written into bullet point summaries. Filler words and redundant statements were removed to enhance clarity. The resulting expert assessments were translated into English using the help of a Large Language Model and were verifed after the translation. All processed data and annotations described above are provided in the dataset¹ (see Data Record section).

Ethics declarations

All recordings were conducted with approval from the ethical committees of the German Research Center for Artificial Intelligence. Written, informed consent was obtained from all participants prior to participation, including explicit consent for the anonymized publication and sharing of the collected data. Participants agreed that their data may be used in anonymized form for research dissemination. Only privacy preserving skeletal/SMPL data are provided. Raw videos are not included. Expert transcripts have been fully anonymized.

Data Record

The dataset is available at Zenodo¹. All data are provided as a compressed archive (Data.zip) containing motion capture data, SMPL parameters, keypoint trajectories, expert gait analysis annotations, and participant metadata. The dataset follows a hierarchical folder structure organized by patient, recording session, and trial, as illustrated in Fig. 2. At the top level, the archive contains:

• Metadata_Patients.xlsx: A spreadsheet containing participant-level metadata, including age, gender, height, affected side, and session participation.

• Patient<patientID>: One folder per participant.

• Each patient folder is structured as follows:

• PraeOperativ/: Contains pre-operative recordings.

• PostOperativ/: Contains post-operative recordings (if available).

• Within each session folder, the data are organized into trial-specific subfolders:

• Trial<TrialID>_Data/: Contains all data associated with a single straight forward walking.

Fig. 2.

Folder Structure of the provided dataset¹.

Each trial folder includes the following files:

• Prae_Patient<patientID>_Trial<TrialID>_KeypointsCenter.csv: Contains 3D keypoint trajectories for each frame.

• Prae_Patient<patientID>_Trial<TrialID>_JointAngles.csv: Contains joint angle time series (in degrees) for flexion/extension, abduction/adduction, and internal/external rotation.

• Prae_Patient<patientID>_Trial<TrialID>_SMPL.npz: Contains SMPL-based motion parameters.

The SMPL NPZ files include the following variables:

• poses: Joint angles in axis-angle representation

• betas: Body shape coefficients

• trans: Global pelvis translation

• 3d_positions: Estimated 3D joint positions

• mocap_framerate: Sampling frequency

• gender: Reported gender

• id: Unique identifier in the format Patient<patientID>_Trial<TrialID>

Expert gait analysis annotations are stored separately:

• Gait_Analysis_Text/: Contains TXT files with clinical gait assessment.

  Each annotation file follows the naming convention:

• Patient<patientID>_PraeOperativ_Expert_<n>.txt

where n denotes a sequential file index used to distinguish multiple annotations associated with the same patient and session. The index does not correspond to specific experts. These files contain anonymized textual summaries of gait assessments corresponding to the first trial. The column headers for the keypoint trajectories are listed in Fig. 3. Note that the keypoint indexing in the CSV files does not correspond to the SMPL kinematic hierarchy (see Fig. 4) .

Fig. 3.

Column headers for the keypoints centers (CSV). Joint angles contain [1–15]. Please note that the indexing in the keypoints.csv files does not correspond to the SMPL kinematic hierarchy or SMPL joint numbering. For the correct SMPL joint definitions, refer to⁷. The locations of the SIMI keypoints can be seen in Fig. 4a.

Technical Validation

The technical validation of this dataset¹ addresses (i) the SIMI Motion capture system (CSV files), (ii) the positional agreement between the SIMI trajectories (CSV files) and the 3d_positions stored within the SMPL parameters (NPZ files), and (iii) the expert gait annotations.

• (i).The SIMI Motion CSV trajectories serve as the reference system in our validation. All recordings were acquired using an eight camera SIMI Motion optical markerless configuration, a widely used commercial system for clinical and sports biomechanics in Germany. Although formal validation studies are currently being prepared (one scheduled for release in December 2025 and another planned for next year), the SIMI system is routinely used in practice and provides 3D kinematic data suitable for gait analysis. In this work, the SIMI trajectories are treated as the reference representation against which the SMPL-based reconstructions are compared.
• (ii).To obtain a full-body parametric representation, all sequences were processed with EasyMocap to estimate the SMPL pose and shape parameters. During this process, EasyMocap determines the 3D positions of each joint, and these positions form the basis for fitting the SMPL motion. Assessing the positional agreement of these 3D keypoint centers with the SIMI trajectories is therefore a necessary step. Since certain keypoints follow different anatomical definitions, perfect correspondence is not expected for all joints. However, a subset of joints (e.g., wrists, elbows, hips, knees, ankles, and eyes) can be considered broadly comparable in terms of their spatial location, enabling a joint-wise analysis. Figure 4 illustrates schematic joint definitions and does not reflect exact anatomical positions. The comparison of 3D trajectories suggests general spatial agreement for several joints, while also indicating systematic deviations that may arise from differing joint definitions and modeling assumptions (see Fig. 5).

  To quantify the differences between both systems, we compute the Euclidean distance between the SIMI 3D keypoint centers (reference) and the EasyMocap/SMPL 3D keypoint centers for every frame and every selected joint. For each motion sequence, we first computed the mean positional deviation per joint, and then compute the mean error. The distribution of these computed values is shown in the boxplots in Fig. 7. The results show that the errors are consistently distributed across all selected joints, with median errors remaining below 5.3 cm for every joint. These values are of a similar order of magnitude as deviations reported for multi-camera markerless motion capture systems in the literature¹⁶. However, it should be noted that most prior studies evaluate markerless systems against marker-based reference systems, whereas in this work two markerless representations are compared. Consequently, the reported differences reflect discrepancies between two modeling approaches rather than absolute reconstruction errors with respect to a physical ground truth. In this context, the SIMI system is treated as a reference representation, while acknowledging that it is not a strict ground truth. Additionally, we computed the distribution separately for the pre-operative and post-operative groups (see Fig. 7b,c). The resulting boxplots show broadly similar distributions across both sessions. However, variations between recordings may be influenced by factors such as lighting conditions, recording setup, or fitting stability, and cannot be attributed solely to the underlying motion.

  The percentile for 25, 50 and 90 percent of the position differences are summarized in Fig. 6. Across all joints, median values lie approximately between 4 cm and 6 cm. Lower median differences are observed for joints such as eyes and wrists, while slightly higher values occur for hips and knees. The P90 values remain below 10 cm for all joints, indicating that larger deviations were still bounded within this range.

• (iii).Each patient was independently evaluated by multiple clinical experts. All annotators were experienced experts (mean professional experience: 13.28 years). After the initial gait description, pairwise discussions were conducted to clarify or refine the observations. This procedure helped to clarify ambiguous observations and reduce potential inter-rater variability. Importantly, the clinical gait analysis was performed based on the SIMI data.

Fig. 5.

Anatomical positions of the SIMI keypoints (black) and the 3d_position (blue) stored in the NPZ files.

Fig. 7.

Comparison of SMPL and SIMI 3D Coordinates: Validation and Reliability. (a) Validation: Pre- and Post-operative Comparison Against the Reference System. (b) Reliability: Pre-operative Comparison with the Reference System. (c) Reliability: Post-operative Comparison with the Reference System.

Fig. 6.

Percentile tables (all values are given in cm).

The SMPL representations are provided as an additional modality and are commonly used in machine learning applications such as human motion modeling, embedding, and motion-to-text tasks. However, given the observed deviations and differences in joint definitions, the SMPL-based representations should be used with caution for biomechanical interpretation. The availability of the SIMI 3D joint trajectories enables the possibility of applying alternative or improved SMPL fitting procedures in future work. Overall, the dataset¹ provides a multimodal representation of human gait before and after knee arthroplasty, combining clinical annotations with kinematic data and parametric body models. This facilitates research across biomechanics, rehabilitation, computer vision, and data-driven analysis, while requiring careful consideration of the respective representation limitations.

Usage Notes

The SMPL parameters included in this dataset¹ follow the standard conventions of the SMPL model and can be processed or visualized using commonly available tools. All joint rotations are given in axis angle representation, and the forward direction of each sequence has aligned with the positive X-axis during preprocessing. Users working with both the SIMI Motion and SMPL representations should note that the SIMI keypoint ordering does not directly correspond to the SMPL kinematic hierarchy. The dataset¹ does not contain dynamic shape parameters (dmpls), as these are not produced by the EasyMocap pipeline. The CSV files provided by SIMI Motion include joint angles expressed in flexion/extension, abduction/adduction, and internal/external rotation.

The expert gait assessments, available in both the original German and in English translation, contain clinically common abbreviations such as LWS (lumbar spine), HWS (cervical spine), and BWS (thoracic spine). These assessments were based solely on anonymized skeletal renderings in three standardized views. While this provides a standardized and privacy-preserving representation of gait, it may limit the visibility of appearance-based cues, pain-related behavior, and other clinical signs that could be observed in real patient videos. Therefore, assessments based on skeletal renderings should not be interpreted as fully equivalent to assessments based on original video recordings. In addition, no pain scores, perceived disability scores, or functional status measures are included. The actual clinical condition of the patients is therefore only indirectly reflected through the recorded gait patterns and the textual expert assessments. Users should consider this when interpreting the dataset. The value of the dataset¹ lies in combining quantitative 6MWT-based motion recordings with several independent clinical gait descriptions per patient.

Not all patients participated in both recording sessions. The metadata file indicates which subjects include pre- and post-operative recordings. Furthermore, all motion sequences were preprocessed by removing stationary frames, correcting extreme pelvis orientation deviations, and applying smoothing filters, as described in section Methods. Due to data protection and privacy constraints, the raw videos of patients performing the 6MWT cannot be made available.

Funding

This project has received funding from the BMBF project OrthoSuPer (project ID 13GW0564A F and 13GW0564E), from the European Union’s Horizon Europe research and innovation programme under grant agreement No. 101092889 (Embodied Social Experiences in Hybrid Shared Spaces [SHARESPACE]), and from the European Union’s Horizon Europe research and innovation programme under grant agreement No. 101298864 (Virtual-physical Integration and Consistency through mulTimodal, Open, Real-time XR [VICTOR-XR]).

Data availability

The dataset is available under: 10.5281/zenodo.19728114.

Code availability

Custom code was used for data processing, including filtering, SMPL parameter conversion, and outlier correction. This code is not publicly available. However, all processing steps are described in detail in section Methods to ensure reproducibility. The study also relies on publicly available tools, including EasyMocap¹⁴, which was used for SMPL parameter estimation. The SIMI Motion data were processed using SIMI software version 10.2.0.

Competing interests

The authors declare no competing interests.