A neural network to predict the knee adduction moment in patients with osteoarthritis using anatomical landmarks obtainable from 2D video analysis

Abstract

Objective:

The knee adduction moment (KAM) can inform treatment of medial knee osteoarthritis; however, measuring the KAM requires an expensive gait analysis laboratory. We evaluated the feasibility of predicting the peak KAM during natural and modified walking patterns using the positions of anatomical landmarks that could be identified from video analysis.

Method:

Using inverse dynamics, we calculated the KAM for 86 individuals (64 with knee osteoarthritis, 22 without) walking naturally and with foot progression angle modifications. We trained a neural network to predict the peak KAM using the 3-dimensional positions of 13 anatomical landmarks measured with motion capture (3D neural network). We also trained models to predict the peak KAM using 2-dimensional subsets of the dataset to simulate 2-dimensional video analysis (frontal and sagittal plane neural networks). Model performance was evaluated on a held-out, 8-person test set that included steps from all trials.

Results:

The 3D neural network predicted the peak KAM for all test steps with r²=0.78. This model predicted individuals’ average peak KAM during natural walking with r²=0.86 and classified which 15° foot progression angle modifications reduced the peak KAM with accuracy=0.85. The frontal plane neural network predicted peak KAM with similar accuracy (r²=0.85) to the 3D neural network, but the sagittal plane neural network did not (r²=0.14).

Conclusion:

Using the positions of anatomical landmarks from motion capture, a neural network accurately predicted the peak KAM during natural and modified walking. This study demonstrates the feasibility of measuring the peak KAM using positions obtainable from 2D video analysis.

Introduction

Knee osteoarthritis (OA) affects approximately 14 million people in the United States¹ and is a leading cause of disability worldwide². The medial compartment of the knee is most commonly affected³, likely because it bears a greater proportion of total knee contact force compared to the lateral compartment⁴. Joint loading cannot be directly measured in vivo in an osteoarthritic knee, so the peak knee adduction moment (KAM) is a common surrogate measure of medial knee loading due to its relationship to the medio-lateral distribution of force in the knee⁵. The peak KAM, which typically occurs during the first half of the stance phase, is correlated with the presence⁶, severity^7,8, and progression⁹ of medial compartment knee osteoarthritis.

Measurements of the peak KAM enhance clinical decision making. Measuring the peak KAM during natural walking aids in diagnosing OA^10–12, evaluating a patient’s risk of OA progression⁹, and predicting surgical outcomes¹³. Gait modifications such as increasing trunk lean^14,15, avoiding contralateral pelvic drop¹⁶, medializing the knee (medial thrust)^17,18, and changing the foot progression angle^19,20 can reduce the peak KAM and pain. However, personalizing interventions based on how they affect each individual’s peak KAM is critical for maximizing the achievable reduction in loading and avoiding interventions that cause a harmful increase in loading^18,21–23. Personalized foot progression angle modifications (toe-in or toe-out), in particular, are an effective and subtle way to reduce the peak KAM²³.

The KAM is calculated using inverse dynamics from ground reaction forces and kinematics measured in a gait analysis laboratory equipped with force plates and a motion capture system (Figure 1A). The expensive equipment and technical expertise necessary to operate a gait laboratory are inaccessible to most clinicians, which excludes potentially valuable KAM measurements from routine clinical practice. The peak KAM has been calculated using mobile sensors (inertial measurement units and force-instrumented shoes) and an inverse dynamics approach²⁴; however, this method still requires expensive equipment, limiting its scalability. A simpler, cheaper way to assess the peak KAM is needed.

Figure 1.

A) The gold standard, laboratory-based workflow for measuring the knee adduction moment (KAM). After motion capture data is collected and semi-manually pre-processed, it is combined with force plate data to compute the KAM using inverse dynamics. B) The workflow for the current study. We use the coordinates of 13 anatomical landmarks from motion capture (to simulate video keypoints) as inputs into a neural network trained to predict the peak KAM. C) Our proposed future workflow for the automated measurement of the KAM. After collecting 2D video of gait, keypoints (e.g., joint positions) could be detected automatically using OpenPose²⁵. A neural network would predict the peak KAM using these keypoints as input.

The ability to predict the peak KAM with 2-dimensional (2D) video analysis would make these measurements available to clinicians, researchers, and patients using only a video camera. Recently-developed automated keypoint detection algorithms such as OpenPose²⁵ extract the positions of anatomical landmarks (e.g., joint locations) from 2D video (Figure 1C). But the small number of keypoints on each body segment and the imperfect anatomical accuracy of keypoints (ranging from 14–30mm)^26,27 limit their utility as a replacement for motion capture in an inverse dynamics analysis. Machine learning models are a promising solution for predicting complex biomechanical outputs using low-fidelity inputs like video-based keypoints^28,29.

Machine learning models have been trained to predict the KAM, but many of these models rely on input features from equipment that is not readily available in clinical settings, such as force plates, inertial measurement systems, pressure-sensing insoles, and electromyography systems^30–33. Other models have used kinematic features such as synthesized images of motion capture trajectories³⁴ and Euler angles³⁵ to predict the KAM curve in healthy individuals and individuals with alkaptonuria. These models demonstrate the promise of using kinematics alone to predict the KAM curve; however, the ability to predict the peak KAM in individuals with knee OA using kinematic features identifiable from video analysis remains unknown.

The objective of our study is to evaluate the feasibility of predicting the peak KAM in individuals with and without knee OA using keypoints that could be extracted from video. In the absence of a dataset with synchronized 2D video and joint kinetics, we simulate video-based keypoints using the positions of anatomical landmarks measured with motion capture from a previously-collected dataset. Furthermore, we simulate keypoints from 2D video by projecting 3-dimensional (3D) anatomical landmark positions onto a 2D plane. Anatomical keypoint detection is a rapidly maturing field, so evaluating a machine learning model’s ability to predict the peak KAM using motion-capture-based keypoints will demonstrate the potential for a video-based solution in the future.

We aimed to evaluate the feasibility of predicting the first KAM peak using machine learning models that use only the positions of anatomical landmarks as inputs. We evaluated model performance on two clinical decision-making tasks. The first task was predicting an individual’s first peak KAM during natural walking with a mean absolute error (MAE) less than 0.5% bodyweight*height (BW*H); this threshold was chosen because it is the minimum of the clinically-relevant range of 0.5–2.2%BW*H related to diagnosing OA and evaluating the risk of progression^9–12. The second task was classifying whether a foot progression angle modification increases or decreases the first peak KAM. To evaluate the feasibility of a video-based solution that uses a single camera, we compared the performance of models that use only sagittal or frontal plane anatomical landmark positions to the model that uses 3D positions.

Methods

Data Collection

Eighty-six individuals (64 with medial knee OA and 22 without OA²³) participated in this study after providing written consent in compliance with the Stanford University Institutional Review Board (Table 1). We included individuals with and without knee OA to establish a larger dataset and to train a model that is generalizable to a broad spectrum of OA severities. While assessing the KAM in individuals without OA is not currently a part of clinical practice, an inexpensive screening tool, like the pipeline proposed here (Figure 1C), could help identify asymptomatic individuals with a large KAM who may be at risk of developing medial knee OA¹². Inclusion criteria for the medial knee OA group were (i) Kellgren-Lawrence grade 1–3 and smaller medial compared to lateral joint space width assessed from an anterior-posterior weight-bearing radiograph by a radiologist (GEG); (ii) medial pain of 3 or higher out of 10 on the numeric rating scale and medial pain greater than patellofemoral or lateral pain; (iii) the ability to walk safely for 25 minutes on a treadmill without ambulatory aids; and (iv) body mass index (BMI) less than 35. Individuals without osteoarthritis were included if they had no lower-extremity pain or history of lower-extremity injury in the past year.

Table 1.

Population information for participants with and without osteoarthritis (OA). Data are presented as mean ± standard deviation.

Characteristic	Without OA	With OA
Number of subjects	22	64
Gender	9F, 13M	43F / 21M
Age (years)	24.7 ± 3.2	64.6 ± 8.81
Height (m)	1.75 ± 0.13	1.68 ± 0.10
Weight (kg)	69.4 ± 15.3	77.2 ± 15.0
BMI (kg/m2)	22.3 ± 2.2	27.2 ± 3.7
Preferred Walking Speed (m/s)	1.15 ± 0.10	1.16 ± 0.13
Kellgren Lawrence grade	N/A	I: 12, II: 32, III: 20

Participants walked on an instrumented treadmill (Bertec Corporation, Columbus, OH, USA) at their self-selected speed in an 11-camera motion capture volume (Motion Analysis Corporation, Santa Rosa, CA, USA) with a retro-reflective marker set sufficient to compute the 3D kinematics of the lower limbs and trunk²³. Participants first completed a static trial to identify ankle and knee joint axes and joint centers and then performed bilateral hip circumduction trials to determine the hip joint centers³⁶. After 5 minutes of treadmill familiarization, participants completed a 2-minute baseline trial of natural walking. Participants then received vibrotactile biofeedback (described in Uhlrich et al.²³) that instructed them to practice toeing-in and toeing-out by 5° and 10° relative to their natural foot progression angle for a minimum of 1 minute per angle. They subsequently performed four 2-minute modification trials where they targeted each of the foot progression angle modifications. The baseline and modification trials were used for analysis.

Ground reaction forces and marker positions were low-pass filtered at 8 Hz using a fourth-order, zero lag, Butterworth filter. The KAM was calculated using inverse dynamics in MATLAB (Mathworks Corporation, Natick, MA, USA) as the frontal plane component of the 3D knee moment expressed in the proximal tibial reference frame²³ and normalized by bodyweight and height. The peak KAM was defined as the maximum value of the KAM curve during the first half of the stance phase; the stance phase was defined as the time during which the vertical ground reaction force was greater than 30N.

Predictive Statistical Models

The inputs to our machine learning model were the 3D positions of 13 anatomical landmarks from motion capture and a binary value identifying the stance leg (Figure 1B). From the motion capture marker set, we selected a subset of anatomical landmarks that are similar to the keypoints tracked using 2D video analysis algorithms like OpenPose²⁵. The anatomical landmarks chosen included four from each lower extremity (2nd metatarsal head, posterior calcaneus, lateral malleolus, lateral femoral epicondyle), four pelvic landmarks (right and left anterior superior iliac spine and posterior superior iliac spine), and the C7 vertebrae. We defined the midpoint of the posterior superior iliac spine markers as the origin of the anatomical landmark positions to make these input positions independent of the participant’s location in the laboratory frame. Positions were then normalized by participant height. Additionally, the medio-lateral positions of left steps were reflected across the body midline, so that all data appeared to be from the right leg, making the input more consistent³⁷. Since our goal was to predict the first peak KAM, which occurs during the first half of the stance phase, we sampled the input positions 8 times during the first half of the stance phase. This sample rate was sufficient to capture the frequency content of the 8Hz-lowpass-filtered inputs during the stance phase, which lasts less than 1 second. The final input matrix for each step was of size 40 × 8 ((13 markers · 3 dimensions + 1 leg binary) · 8 timesteps).

Participants were randomly divided 80%−10%−10% into training, development, and test sets, such that all of a participant’s data resided in only one set. The training set consisted of 91,245 steps from 70 individuals (18 without OA, 52 with OA), the development set consisted of 9,810 steps from 8 individuals (2 without OA, 6 with OA), and the test set consisted of 11,675 steps from 8 individuals (2 without OA, 6 with OA). The performance of the model on the development set was used to select the model architecture and hyperparameters. The test set was used only to evaluate the accuracy of the final model.

Models were trained in Python (version 3.6.8) using Keras (version 2.2.4) with 1 GPU (NVIDIA GeForce GTX 960). We evaluated the following architectures: linear regression, a convolutional neural network (three 1-dimensional convolutional layers, 3 fully connected layers)³⁸, a long short-term memory network (2 long short-term memory layers, 1 fully connected layer)³⁹, and a fully connected neural network (10 hidden layers with 100 neurons each). Neural network weights were initialized with Xavier initialization⁴⁰, and Adam gradient descent⁴¹ was used to minimize the root-mean-squared-error. The final layer of each model had a single neuron with linear activation to output the single peak KAM value. We selected a fully connected neural network architecture due to its superior performance on the development set (Table S1, Supplemental Information). We refer to this model as the 3D neural network, as its inputs are 3D anatomical landmark positions.

We used lasso regression⁴² to reduce redundant information in the flattened input vector before training the 3D neural network (glmnet package⁴³ in R⁴⁴). To select the lambda value for lasso regression, we performed 10 runs of 10-fold cross-validation using the training set. We used the one-standard-error rule for selecting lambda: for each run, we selected the lambda value that yielded a prediction error that was one standard error greater than the minimum prediction error, thereby removing more features than the lambda that minimized prediction error. We then removed all but one binary leg feature from the flattened input vector. The input vector size was reduced from 320 to 299 features.

After a random search, we selected the following 3D neural network hyperparameters based on model performance on the development set: 1 hidden layer (tested 1–20), 800 neurons in the hidden layer (tested 0–1000), and 0.01 probability of dropout (tested 0–0.25). Neurons in the hidden layer used a rectified linear unit activation function⁴⁵, while the output neuron had a linear activation function. To reduce overfitting, we used early stopping with 5 epochs of patience and used model weights from the epoch with the best development-set performance.

To identify which anatomical landmark positions were most influential on the peak KAM predictions, we computed the saliency of the input features. In general, saliency analysis identifies how changes in each input feature relate to changes in the output by computing the partial derivative of the output with respect to each input feature⁴⁶. For this analysis, we trained a model with the same architecture and input features as the 3D neural network but without lasso regression in order to account for every feature at each timestep. We averaged saliency (Python Keras-vis package⁴⁷) over 2000 examples randomly selected from the test set, then averaged over all timesteps for each of the 39 anatomical landmark positions (Equation 1):

$$Salienc{y}_{x_i}={\text{mean}}_k\left({\text{mean}}_j\left(\left|\frac{\partial KA{M}^{\left(j\right)}}{\partial x_i^{\left(j,k\right)}}\right|\right)\right)$$ (Equation 1)

where x_i=1:39 input positions, j=1:2000 examples, and k=1:8 timesteps. Finally, we normalized the saliency value for each feature by the saliency value of the most salient feature (Equation 2).

$$Salienc{y}_{x_i,norm}=\frac{Salienc{y}_{x_i}}{{\max }_i\left(Salienc{y}_{x_i}\right)}$$ (Equation 2)

We then evaluated the performance of our models on the clinical decision-making tasks. To test a model’s ability to predict an individual’s baseline peak KAM during natural walking, we predicted the peak KAM for all steps during the baseline trial, computed the average value for an individual leg, then compared this value to the average of the peak KAM values from inverse dynamics. We analyzed both legs for the 8 individuals in the test set, yielding 16 conditions. To test the model’s ability to identify changes in the peak KAM from varying degrees of foot progression angle modifications, we identified steps for which individuals toed-in or toed-out by 5±2.5°, 10±2.5°, or 15±2.5°. For each modification, there were 32 possible conditions (8 subjects, 2 legs, 2 angles), but not all legs achieved the target angles, leaving between 20 and 31 conditions for analysis. Changes in the peak KAM were computed by subtracting a leg’s average baseline peak KAM from the peak KAM from a modified step, using KAM values from either the neural network or inverse dynamics.

For our final aim, we simulated 2D video input by using only sagittal or frontal plane anatomical landmark positions as model inputs. We removed the anterior-posterior landmark positions (in the laboratory frame) for the frontal plane neural network and the medio-lateral components for the sagittal plane neural network. There were 209 inputs for these planar models (13 markers · 2 dimensions · 8 timesteps + 1 leg binary). We used the same model architecture and training procedures for the planar models as the 3D neural network, excluding lasso regression.

Analytical Statistics

All statistical analyses were performed in MATLAB. We evaluated model performance using r² (Pearson’s correlation coefficient) and the mean absolute error (MAE; Equation 3):

$$MAE=\frac{1}{m}{\sum }_{i=1}^m\left|y_i-{\widehat{y}}_i\right|$$ (Equation 3)

where m=number of examples; y_i=inverse dynamics peak KAM; and ${\widehat{y}}_i$=neural network-predicted peak KAM. As a supplemental analysis, we added three types of virtual noise to the dataset to simulate various sources of error in video-based keypoint identification, with error magnitudes (14–30mm) based on the previously-reported accuracy of pose recognition algorithms^26,27 (details in Table S2).

To estimate the uncertainty in our performance estimates based on test-set subject selection, we used percentile bootstrapping⁴⁸ to compute 95% confidence intervals (CI) for r² and MAE. For all bootstrapped distributions, we trained one model and resampled (10,000 times) the subjects in the test set with replacement from the 8 subjects allocated to the test set, then calculated the test statistics for each resampled set. When evaluating model performance on all steps in the test set (Figures 2 and 6), we downsampled the test set to have an equal number of steps (400, randomly selected) for each leg, which gave each individual an equal effect on the test statistic, regardless of walking speed or step frequency. We computed the sample r² and MAE from this downsampled test set. To evaluate if our model predicted the baseline peak KAM with MAE less than 0.5%BW*H (Figure 4), we computed the average baseline peak KAM for each leg in the test set and computed the 95% CI from a bootstrapped distribution with per-subject resampling.

Figure 2.

The predicted peak knee adduction moment (KAM) from the neural network (NN) using 3D anatomical landmark positions as input (3D neural network) vs. the peak KAM calculated from inverse dynamics (ID) plotted against the y=x line. Presented data are for test subjects from the baseline and foot progression angle modification trials. Each point represents a single step, and a single color represents the steps from both legs of a subject.

Figure 6.

The performance of neural networks that use planar projections of anatomical landmark positions as inputs. A) The frontal plane neural network predicts the peak KAM with similar accuracy to the 3D neural network (r²=0.85). B) The sagittal plane neural network is less accurate (r²=0.14) than the 3D or frontal plane neural networks. Presented data are for test subjects from the baseline and foot progression angle modification trials. A point represents a single step, and a single color represents the steps from both legs of a subject.

Figure 4.

The peak knee adduction moment (KAM) estimated by the 3D neural network (NN) and inverse dynamics (ID) from the baseline (natural walking) trial. Data are averaged over all baseline steps for each leg of each subject (represented by a color) in the test set plotted against the y=x line.

Results

The 3D neural network predicted the first peak KAM for all steps in the test set (Figure 2) with r²=0.78 (95% CI=[0.44, 0.89]) and MAE=0.53%BW*H (95% CI=[0.39, 0.67]). The training set performed with r²=0.88 and the development set with r²=0.72. The saliency analysis (Figure 3) showed that for the 3D neural network, the peak KAM prediction was most sensitive to changes in positions of the swing and stance-leg anterior superior iliac spines (anterior-posterior), the stance-leg knee (medio-lateral), the C7 vertebrae (medio-lateral), and the swing-leg anterior superior iliac spine (anterior-posterior). From the supplemental virtual noise simulation (Table S2), the 3D neural network performance did not degrade substantially with constant error added to a coordinate across all subjects or with 14mm of error that changed on a subject-by-subject basis. Model performance did degrade with 30mm of subject-by-subject noise and any magnitude of noise that changed randomly at each timestep.

Figure 3.

The top five most salient features (features that, when changed, have the greatest effect on the predicted peak KAM) normalized by the most salient feature for the 3D, frontal plane, and sagittal plane neural networks (left). The positions and Cartesian coordinate directions of the most salient features (right) where x corresponds to the anterior-posterior direction, y to medio-lateral, and z to superior-inferior.

The 3D neural network predicted the average peak KAM during baseline (natural) walking with r²=0.86 (95% CI=[0.62, 0.94]) and MAE=0.37%BW*H (95% CI=[0.23, 0.51]) (Figure 4). The MAE confidence interval minimally overlapped with our most stringent clinically-meaningful accuracy threshold of 0.5%BW*H. Predictions of the baseline peak KAM were, on average, 0.31%BW*H (95% CI=[0.10, 0.51]) greater than the inverse dynamics values (Bland Altman analysis in Figure S1) with an absolute error range of 0.02–1.09%BW*H. The 3D neural network classified if 5°, 10°, and 15° toe-in or toe-out gait modifications increased or reduced the peak KAM with accuracies of 0.65, 0.71, and 0.85, respectively (Figure 5, Table 2).

Figure 5.

The average change in the peak knee adduction moment (KAM) estimated by the 3D neural network (NN) vs. inverse dynamics (ID) for 5°, 10°, and 15° foot progression angle modifications for each leg of each subject in the test set. The accuracy (acc.) of classification is increases with increasing degrees of foot progression angle modification.

Table 2.

The performance of the 3D and frontal plane neural networks in classifying whether changes in foot progression angle (FPA) increase or reduce the peak knee adduction moment (Figure 5). A reduction in peak KAM was considered a positive result. For example, sensitivity represents the number of cases that the neural network predicted as a reduction in the peak KAM divided by the number of cases that ID identified as a reduction in the peak KAM. Both models predict the effects of larger FPA modifications more accurately than small modifications.

	Accuracy	Sensitivity	Specificity	Precision
3D Neural Network
5° FPA modification	0.65	0.59	0.78	0.87
10° FPA modification	0.71	0.70	0.75	0.87
15° FPA modification	0.85	0.73	1.0	1.0
Frontal Plane Neural Network
5° FPA modification	0.58	0.68	0.33	0.71
10° FPA modification	0.79	0.90	0.50	0.80
15° FPA modification	0.80	1.0	0.73	0.73

The frontal plane neural network predicted the peak KAM for all steps in the test set (Figure 6A) with r²=0.85 (95% CI=[0.56, 0.91]) and MAE=0.49%BW*H (95% CI=[0.39, 0.59]), which was not statistically different from the performance of the 3D neural network. The frontal plane neural network predicted the average peak KAM during baseline walking with r²=0.86 (95% CI=[0.49, 0.96]) and MAE=0.40%BW*H (95% CI=[0.23, 0.59]), and predicted the change in KAM resulting from 5°, 10°, and 15° foot progression angle modifications with accuracy of 0.58, 0.79, and 0.80, respectively (Table 2).

The sagittal plane neural network predicted the peak KAM for all steps in the test set (Figure 6B) with r²=0.14 (95% CI=[0.02, 0.45]) and MAE=0.85%BW*H (95% CI=[0.63, 1.06]), which was less accurate than the frontal plane neural network and had minimal confidence-interval overlap with the 3D neural network.

Discussion

The purpose of this study was to evaluate the feasibility of using video to predict the peak KAM in individuals with and without knee OA during walking. To do this, we trained machine learning models to predict the peak KAM using the positions of anatomical landmarks from motion capture that are similar to those that could be measured with video analysis in the future. We found that a neural network that uses 3D positions accurately predicted the peak KAM for people with and without knee OA (r²=0.78). This model accurately predicted individuals’ peak KAM during natural walking as well as changes in peak KAM that resulted from large (15°) foot progression angle modifications, suggesting that it performs well enough to inform clinical decision making in many cases. A neural network that only uses frontal plane positions also predicted the peak KAM accurately, suggesting that it might be feasible to use a front-facing camera and video analysis tools to estimate the peak KAM in the future.

The 3D and frontal plane neural networks predicted the peak KAM with r²=0.78 and r²=0.85, which is similar to previous techniques that predicted the peak KAM with a reduced set of inputs compared to a full gait analysis laboratory. The 95% confidence intervals around the performance of our models overlap with the performance of machine learning approaches that used inertial measurement units (r²=0.71)³² or anthropometric measures and force plate data (r²=0.59)³⁰ as well as an inverse dynamics approach that used instrumented force shoes and inertial measurement units (r²=0.80)²⁴. Other studies have predicted the KAM curve during walking using a reduced set of inputs^34,35,49, but it is worth noting that predicted curves that are highly correlated with the reference curve do not always yield accurate peak predictions. For example, Favre et al. predicted the KAM curve with r²=0.94, but the peak KAM extracted from the curve was less accurate (r²=0.59)³⁰, demonstrating the importance of evaluating model performance on clinically-meaningful outcome metrics, like the peak KAM. Although our models do not directly enable mobile measurements, the frontal plane neural network demonstrates the feasibility of predicting the peak KAM from a limited set of anatomical landmark positions that are identifiable from 2D video. Secondarily, the model could serve as the basis for a transfer learning approach that uses keypoints from front-facing video as inputs.

The 3D input positions that the saliency analysis identified as most influential for KAM predictions correspond with kinematic changes that have been shown to affect the KAM. First, the superior-inferior positions of the posterior superior iliac spines relates to contralateral pelvic drop, which increases the KAM¹⁶. Second, the medial position of the stance-limb knee relates to the frontal-plane knee angle. Medializing the knee is a suggested mechanism for KAM-reducing interventions such as medial-thrust gait¹⁷, toe-in gait⁵⁰, and variable stiffness shoes⁵¹. Third, the medio-lateral position of the C7 vertebrae relates to the trunk-sway angle. Trunk-sway influences the KAM by altering the medio-lateral position of the center of mass¹⁵. Finally, the anterior-posterior position of the swing-leg knee could relate to stride length. A slower walking speed or decreased stride length can reduce the peak KAM^10,17. Some of the salient features in the frontal and sagittal plane models were different from the 3D model’s salient features in these planes, which may be explained by the redundancy of input features, complex nonlinear relationships between features, and the stochastic nature of model training. Even small perturbations to input data can lead to dramatically different saliency results, and thus, different interpretations of the model⁵², so the relationships between gait modifications and the salient features remains speculative.

Both the 3D and frontal plane neural networks performed with sufficient accuracy to inform clinical decision making in most cases. The models predicted the peak KAM during natural walking with MAE=0.37–0.49%BW*H, which is accurate enough to evaluate the risk of OA progression (differences between the peak KAM of progressors and non-progressors is 2.1%BW*H)⁹. Our models also predicted the baseline peak KAM with sufficient accuracy to classify between patients who are likely or unlikely to benefit from a high tibial osteotomy (individuals with a 2.2%BW*H lower pre-surgical peak KAM had better post-surgical outcomes)^13,53. Additionally, our models classified whether 15° foot progression angle modifications increased or reduced the peak KAM with an accuracy of 0.80–0.85. However, they were less accurate in predicting smaller 5° and 10° modifications, potentially due to subtler changes to both input kinematics and the output peak KAM. Notably, toeing-in typically reduces the first peak KAM⁵⁰, but it is not effective for all individuals²³; the 3D neural network correctly identified the five of the six legs that reduced the first peak KAM by toeing-in by 15° and the four of the five that did not. If a future video-based model performs with similar accuracy, any clinician with a smartphone would be able to make clinically-actionable biomechanical measurements without purchasing expensive equipment.

It is important to identify the limitations of this study. First, machine learning models may not generalize well to conditions not represented in the training data. Our models were trained on individuals with and without knee OA, with a BMI below 35, who were walking with varying foot progression angles. Thus, our models will likely lose accuracy when predicting the peak KAM for new populations performing different activities. Fortunately, we trained accurate models with a reasonably-sized dataset (n=86), suggesting that new models predicting different parameters can likely be trained using pre-existing datasets. Second, the inputs to our models were motion capture marker positions that were very similar, but not identical to the positions commonly used in video pose-recognition algorithms (e.g., lateral femoral epicondyle versus knee joint center). However, model performance did not change dramatically when a constant offset was added to the motion capture positions (Table S2), indicating that the models may not be sensitive to the small differences in the definitions of anatomical landmarks between motion capture and video-based keypoints. Finally, our models do not directly facilitate mobile measurement of the peak KAM since they rely on anatomical landmark positions from motion capture rather than keypoints from video. The robustness of our pipeline to several sources of error that may arise from video keypoints (Table S2) adds confidence that a video-based approach may be feasible. Nonetheless, the performance of our models trained on motion capture positions will likely exceed the accuracy of a similar model trained with a similarly-sized dataset of video keypoints. Even with a slight decrease in performance, an inexpensive, video-based solution would provide clinical value compared to other mobile solutions that require equipment that costs thousands of dollars^24,30,54.

In summary, we developed a model to predict the first peak KAM in individuals with and without knee OA using the positions of anatomical landmarks. Our model accurately predicts the peak KAM during natural walking as well as changes in the KAM that result from gait modifications. Since anatomical landmark positions projected onto a plane are similar to the output from 2D video keypoint detection algorithms, our results support the feasibility of predicting the peak KAM from a 2D frontal plane video as a promising next step. These results support the utility of computer vision and machine learning as tools that can bring biomechanical measurements into the clinic or home. We envision that models like the ones presented here will soon enable scientists to easily monitor joint loading in large cohorts and clinicians to prescribe personalized treatments for musculoskeletal pathologies.