Association Between Markerless Motion Capture Screenings and Musculoskeletal Injury Risk for Military Trainees: A Large Cohort and Reliability Study

Abstract

Background:

Markerless motion capture (MMC) systems used to screen for musculoskeletal injury (MSKI) risk have become popular in military and collegiate athletic settings. However, little is known regarding the test-retest reliability or, more importantly, the ability of these systems to accurately identify individuals at risk for MSKI.

Purpose:

To determine the association between scores from a proprietary MMC movement screen test and the likelihood of suffering a subsequent MSKI and establish the test-retest reliability of the MMC system used.

Study Design:

Cohort study; Level of evidence, 3.

Methods:

Trainees for the Air Force Special Warfare program underwent MMC screenings immediately before entering the 8-week training course. MSKI data were extracted from a database for the surveillance period for each trainee. Logistic regression analyses were performed to identify associations between baseline MMC scores and the likelihood of suffering any MSKI or, specifically, a lower extremity MSKI. The test-retest portion of the study collected MMC scores from 10 separate participants performing 4 trials of the standard test procedures. Reliability was assessed using intraclass correlation coefficients by a single rater.

Results:

Overall, 1570 trainees, of whom 800 (51%) suffered an MSKI, were included in the analysis. MMC scores poorly predicted the likelihood of any or a lower extremity MSKI (odds ratio, 1.01-1.02). Further, receiver operating characteristic curve analyses demonstrated poor sensitivity and specificity for prediction of MSKI with MMC scores (area under the curve = 0.53). Finally, intraclass correlation coefficients from the test-retest analysis of MMC scores ranged from 0.157 to 0.602.

Conclusion:

This MMC system displayed poor to moderate test-retest reliability and did not demonstrate the ability to discriminate between individuals who were and were not likely to suffer an MSKI.

Musculoskeletal injury (MSKI) is the medical condition having the greatest effect on the US military, accounting for 2.4 million encounters and 25 million limited-duty days, affecting over 900,000 active-duty servicemembers each year. ^16,30 Airmen in the US Air Force (USAF) Special Warfare (SW) community have been shown to suffer MSKIs at a higher rate compared with those in other USAF career fields. ^21,34 Lower extremity (LE) injuries are the most common type of MSKI and are among the leading causes of disability, morbidity, and lost training time within the USAF SW training environment. ²¹

Thus, the US Department of Defense has invested considerable resources to reduce the rate and burden of MSKI in its members and is poised to expand these initiatives. ^14,32 A key component of these efforts is identifying and procuring tools that identify individuals at risk for MSKI, which will help to focus and tailor preventive interventions to susceptible individuals. ³² One tool that has become popular recently in military training and collegiate athletics is markerless motion capture (MMC) assessments. ^8,20 These devices have been approved by the US Food and Drug Administration, and, when used per manufacturer recommendations, they capture joint motion data and come with the promise of assessing joint vulnerability in the workflow. ⁸ During this assessment, patients perform movements and functional tasks such as squatting, lunging, jumping, and overhead arm motions that are captured by multiple video cameras. ²⁵ These recordings are analyzed, and the data are converted to a report, which quantifies the quality and quantity of movement and an individual’s risk for MSKI. ^9,25 Compared with marker-based systems, markerless systems are less costly and can be administered quickly, allowing large numbers of individuals to be screened in a short time frame. ²³ In addition, unlike more traditional movement screens, such as the functional movement screen or the Y-balance test, the assessments are automated and, therefore, not influenced by the subjectivity of the individual administering the test. ²⁸ Although MMC assessments offer the benefits of efficiency, automation, and minimized operator-dependent variation, their true ability to identify individuals at risk for MSKI remains unknown. The output is reported using novel scales (range, 0-100; higher scores indicate greater MSKI risk) and use terms such as “vulnerability scores,” which are poorly defined and of uncertain clinical significance. Finally, the test-retest reliability of this device remains unknown.

The MMC system used in the USAF SW training environment is the DARI Motion 3-dimensional (3-D) MMC (Dynamic Athletic Research Institute, LLC). ⁹ The purpose of this study was to determine the association between 2 proprietary MMC scores derived by the DARI Motion device and the risk of MSKI in USAF SW trainees during an 8-week course. Further, we explored the test-retest reliability of this device in a separate substudy conducted at a separate site.

Methods

Design and Setting

This study consisted of 2 components: a retrospective cohort analysis (N = 1570) for assessment of predictive value of the MMC system and a test-retest reliability component (separate cohort, N = 10). The cohort arm of the study was conducted at the USAF SW Training Wing at Joint Base San Antonio, Texas. Participants were USAF airmen (all male) entering the SW training pipeline through an 8-week preparatory course (SW Prep). This course is designed to prepare airmen for the rigors of the SW training pipeline, which has high attrition and injury rates, and it involves 8 weeks of extensive physical, didactic, and psychological training. Trainees typically spend >3 hours each day in physical training events such as ruck-marching, swimming, and resistance training. Before the start of each course, trainees undergo a series of screening tests to include evaluations by certified athletic trainers to screen for active injuries that may affect training. Trainees with active injuries were removed from training and not included in this analysis. MMC assessments were performed within 3 days before the start of the course as part of routine screening for injury risk and movement dysfunction. Data used in this study were collected between October 2017 and April 2020 and included 14 cohorts of the SW Prep course.

The test-retest reliability component of this study was conducted at Luke Air Force Base, Arizona. Participants for this portion were staff members who volunteered to participate. Both components of this study received institutional review board approval.

Testing Procedures and MSKI Surveillance

MMC Assessment Procedures

The DARI Motion system was used for all MMC assessments. ⁹ We positioned 8 Blackfly/FLIR GigE cameras (50 frames per second) circumferentially above and around the patient to record movements. Markerless 3-D motion capture was performed in a rectangular room that measured 6 m × 6 m, was 3 m in height, and had green screen flooring. All tests were administered by certified athletic trainers. Participants were instructed to stand with feet a shoulder-width apart and toes pointing forward with their arms raised to 90° of abduction and their elbows flexed to 90° with their forearms parallel to the ground and palms facing downward. The participants stood still while the software created and superimposed a 3-D silhouette to digitally track and record movement data. Participants were then guided through 8 movements: shoulder internal/external rotation, shoulder flexion/extension, overhead squat, unilateral squat right/left, vertical jump, unilateral vertical jump right/left. Session data were uploaded to the DARI Vault server, which analyzed kinematic and kinetic data and produced a report for each evaluation quantifying movement quality and injury risk. These reports, which included suggestions for corrective exercises, were shared with participants so they could review on their own. No formal interventions or follow-ups were conducted based on the results of the screenings. The 2 proprietary composite scores used for this study were the DARI overall vulnerability (DOV) score and the DARI lower body vulnerability (DLV) score. The DLV score was included in our analyses because LE injuries are the predominant injury type seen in SW. ¹¹ Finally, the test-retest component of this study was carried out using the same movements by a single assessor (SA). Participants performed the test routine 4 times with 5 minutes of rest between tests.

Injury Surveillance and Classification

We were interested in 2 outcomes: whether or not a trainee suffered any MSKI during the 8-week training course and whether or not he suffered an LE MSKI. Injury data were obtained from the Medical Health System Management Analysis and Reporting Tool, which is a centralized data repository that captures and catalogs data input into the Military Health System’s electronic medical records. International Classification of Diseases (10th Revision, Clinical Modification) codes corresponding to MSKIs were used to identify trainees who suffered an MSKI. A classification matrix based on a previously established scheme was used to further classify each MSKI into 2 categories: any MSKI (yes/no) and LE MSKI (yes/no). ^12,24 Subsequent encounters for the same injury were not counted. Individuals could be categorized for >1 injury type (any MSKI and LE MSKI).

A total of 1570 SW trainees (all male) underwent MMC assessment immediately before the SW Prep course between October 2018 and April 2020 and were included in the analysis. See Figure 1 for the flow of patients through the study.

Figure 1.

Flowchart accounting for course enrollees and those lost to follow-up. MMC, markerless motion capture; MSKI, musculoskeletal injury; Prep, Special Warfare Preparatory Course; SW, special warfare.

Rationale for Variable Selection

Our 2 primary predictor variables of interest were the DOV and DLV scores. ⁹ Both are scored on a 0 to 100 scale, with higher scores reflecting a higher risk of suffering an MSKI during training.

Previous MSKI has been shown to be a strong predictor of future injury; therefore, we included as a covariate whether or not the trainee had suffered an MSKI before the primary injury surveillance period. ²⁹ Previous MSKI was defined as an injury sustained in the 14-week period before the start of SW Prep. We chose this duration to ensure we captured the period when the trainees were in basic military training (8 weeks) and any time between completion of basic training and the start of SW Prep when there was an administrative delay. Previous MSKIs were categorized as any MSKI (yes/no) and LE MSKI (yes/no) using the methods described previously. Body mass index (BMI) and age were also included as covariates in our analyses, as they have both been shown to be associated with increased risk of MSKI in military populations. ¹⁷

Statistical Analysis

Descriptive statistics were performed for each measure. We compared age, BMI, previous MSKI rates (any/LE MSKI), DOV scores, and DLV scores between both primary outcomes during the SW Prep course. Independent t tests were used to compare continuous measures (DOV and DLV scores, BMI, age), and Cohen’s d effect size statistics were calculated to measure the magnitude of differences in means. Chi-square analyses were used for comparing frequency counts (previous MSKIs). We performed logistic regression analyses in 2 phases. In the first phase, we conducted unadjusted univariate logistic regression analyses to determine if there were associations between individual predictor variables (previous injury, age, BMI, DOV and DLV scores) and the likelihood of each MSKI outcome during the SW Prep course. Receiver operating characteristic (ROC) curves were calculated for the logistic regression results of DOV scores and MSKI outcomes to demonstrate the predictive ability of these scores. ¹⁸ In phase 2, to explore the associations between DARI scores and MSKI while controlling for the effects of age, BMI, and previous injury status, we conducted adjusted univariate logistic regression analyses. All analyses were assessed for significance at a level of P < .05.

Test-retest reliability of the data was assessed using intraclass correlation coefficients (ICCs) (3,1; 2-way mixed effects, absolute agreement, single rater). The ICC analyses were set up using a sample size of 10 and 4 observations per participant, as that would allow us to detect ICCs ≥0.5. ⁵ Stata Version 16.1 (Stata), R Version 4.0 (The R Project for Statistical Computing), and SPSS Version 25 (IBM Corp) were used for the analyses.

Results

Overall, 800 (51%) of trainees suffered an MSKI during training, and 287 (18%) suffered a previous MSKI in the 14 weeks preceding the start of the course. Sample descriptive statistics broken down by total sample and status on any MSKI and LE MSKI are presented in Table 1. Small but statistically significant differences in BMI were observed between trainees who did and did not suffer an MSKI (mean difference [MD], 0.34 [95% confidence interval (CI), 0.13-0.55]; Cohen’s d = 0.16 [95% CI, 0.06-0.27]) and between those who did and did not suffer an LE MSKI (MD, 0.35 [95% CI, 0.14-0.56]; Cohen’s d = 0.17 [95% CI, 0.06-0.27]). Those who suffered an MSKI during SW Prep had higher rates of previous (any) MSKI (26% vs 10%) and previous LE MSKI (22% vs 8%) than those who did not suffer an injury during SW Prep. Likewise, participants who suffered an LE MSKI during SW Prep also had higher rates of previous (any) MSKI (28% vs 11%) and previous LE MSKI (24% vs 9%) compared with those who did not suffer an LE MSKI during SW Prep. Mean differences in DOV scores were statistically significant between those who did and did not suffer an MSKI (MD, 0.92 [95% CI, 0.19-1.64]; Cohen’s d = 0.13 [95% CI, 0.03-0.22]) and those who did and did not suffer an LE MSKI (MD, 1.01 [95% CI, 0.28-1.74]; Cohen’s d = 0.14 [95% CI, 0.04-0.24]) (Figure 2). Likewise, MDs in DLV scores were statistically significant between those who did and did not suffer an MSKI (MD, 1.13 [95% CI, 0.27-2.00; Cohen’s d = 0.13 [95% CI, 0.03-0.23]) and those who did and did not suffer an LE MSKI during SW Prep (MD, 1.02 [95% CI, 0.14-1.90]; Cohen’s d = 0.12 [95% CI, 0.02-0.22]).

Table 1.

Sample Characteristics Broken Down by Total Patients, any MSKI, and LE MSKI ^a

		Any MSKI			LE MSKI
	Total (n=1,570)	No (n=770)	Yes (n=800)	P-value	No (n=910)	Yes (n=660)	P-value
Age (y)	21.5 (3.4)	21.4 (3.2)	21.5 (3.6)	.53	21.5 (3.4)	21.4 (3.5)	.50
Height (cm)	177.3 (7.6)	177.5 (8.6)	177.0 (6.6)	.48	177.5 (8.4)	176.8 (6.4)	.080
Weight (kg)	77.7 (8.5)	78.1 (8.6)	77.2 (8.3)	.062	78.2 (8.6)	77.0 (8.2)	.013
BMI	24.7 (2.0)	24.9 (2.0)	24.6 (2.1)	.002	24.9 (2.0)	24.5 (2.1)	.002
Previous MSKI	287 (18%)	77 (10%)	211 (26%)	<.001	103 (11%)	184 (28%)	<.001
Previous LE MSKI	236 (15%)	60 (8%)	176 (22%)	<.001	78 (9%)	158 (24%)	<.001
DARI score b
DOV	37.6 (7.3)	37.2 (7.0)	38.0 (7.5)	.013	37.2 (7.1)	38.2 (7.6)	.007
DLV	33.6 (8.7)	33.0 (8.3)	34.1 (9.1)	.010	33.2 (8.3)	34.2 (9.3)	.023

^a Data are presented as mean ± SD for continuous measures and n (%) for frequency counts. Bold P values indicate statistically significant difference between the “no” and “yes” groups (P < .05). BMI, body mass index; DARI, Dynamic Athletic Research Institute; DLV, DARI lower body vulnerability; DOV, DARI overall vulnerability; LE, lower extremity; MSKI, musculoskeletal injury.

^b Higher values indicate higher risk of injury.

Figure 2.

Box plots with overlaid distribution of DOV scores broken down by status on any MSKI and LE MSKI. Higher scores purport to indicate increased vulnerability to MSKI. DARI, Dynamic Athletic Research Institute; DOV, DARI overall vulnerability; LE, lower extremity; MSKI, musculoskeletal injury.

Unadjusted Univariate Logistic Regression Analyses

BMI, previous MSKI, and previous LE MSKI all had significant univariate associations with both MSKI and LE MSKI. DOV and DLV scores had statistically significant associations with both any MSKI and LE MSKI; however, the odds ratios (ORs) were exceedingly modest (Table 2). ⁶ The areas under the curve (AUCs) for the unadjusted DOV scores and MSKI and DOV scores and LE MSKI were 0.53 and 0.54, respectively (Figure 3).

Table 2.

Unadjusted Univariate Logistic Regression Analyses Examining the Association Between Baseline Factors and MSKI Outcome Categories ^a

	Any MSKI		LE MSKI
	OR (95% CI)	P	OR (95% CI)	P
Age	1.01 (0.98-1.04)	.526	0.99 (0.96-1.02)	.501
BMI	0.92 (0.88-0.97)	.002	0.92 (0.88-0.97)	.002
Previous (any) MSKI	3.20 (2.41-4.25)	<.001	3.03 (2.32-3.95)	<.001
Previous LE MSKI	3.34 (2.44-4.56)	<.001	3.36 (2.50-4.50)	<.001
DARI score
DOV	1.02 (1.00-1.03)	.013	1.02 (1.01-1.03)	.007
DLV	1.01 (1.00-1.03)	.011	1.01 (1.00-1.03)	.023

^a Bold P values indicate statistical significance (P < .05). BMI, body mass index; DARI, Dynamic Athletic Research Institute; DLV, DARI lower body vulnerability; DOV, DARI overall vulnerability; LE, lower extremity; MSKI, musculoskeletal injury; OR, odds ratio.

Figure 3.

ROC curve analysis with area under the curve demonstrating the ability of DOV scores to predict any MSKI and LE MSKI. DARI, Dynamic Athletic Research Institute; DOV, DARI overall vulnerability; LE, lower extremity; MSKI, musculoskeletal injury; ROC, receiver operator characteristic.

Adjusted Univariate Logistic Regression Analyses

When logistic regression analyses were run while controlling for previous injury status, BMI, and age, the DLV score no longer demonstrated statistically significant associations with either any MSKI or LE MSKI, and the association of the DOV score with any MSKI also failed to reach statistical significance (Table 3). The association between DOV score and LE MSKI remained statistically significant (P = .011) in the adjusted analysis; however, the ORs were again very small (OR, 1.02 [95% CI, 1.00-1.04]). ⁶

Table 3.

Adjusted Univariate Logistic Regression Analysis Examining the Association Between DARI Scores and MSKI During an 8-Week Training Course ^a

	Model 1: Any MSKI b		Model 2: LE MSKI c
DARI scores	OR (95% CI)	P	OR (95% CI)	P
DOV	1.01 (1.00-1.03)	.106	1.02 (1.00-1.04)	.011
DLV	1.01 (1.00-1.02)	.166	1.01 (1.00-1.02)	.077

^a Bold P value indicates statistical significance (P < .05). DOV and DLV scores were analyzed independently for each model. BMI, body mass index; DARI, Dynamic Athletic Research Institute; DLV, DARI lower body vulnerability; DOV, DARI overall vulnerability; LE, lower extremity; MSKI, musculoskeletal injury; OR, odds ratio.

^b Model 1: adjusted for age, BMI, and previous MSKI.

^c Model 2: adjusted for age, BMI, and previous LE MSKI.

Test-Retest Reliability

The ICCs for vulnerability scores as estimated across 4 trials are shown in Table 4. The ICCs ranged from 0.157 to 0.602, indicating poor to moderate test-retest reliability.

Table 4.

ICCs for DOV and DLV Scores by Body Region ^a

Body Region	ICC
Shoulder
Right	0.507
Left	0.339
Hip
Right	0.396
Left	0.345
Knee
Right	0.308
Left	0.416
Ankle
Right	0.492
Left	0.366
Upper spine	0.602
Lower spine	0.157

^a For each participant, 4 trials of the movement screen were conducted. DARI, Dynamic Athletic Research Institute; DLV, DARI lower body vulnerability; DOV, DARI overall vulnerability; ICC, intraclass correlation coefficient.

Discussion

Over the past 10 years, there has been increased focus on biomechanical motion capture analysis to screen for MSKI risk in athletes. ^10,33 Whereas there have been isolated studies showing promise for motion analysis to identify athletes at risk for specific injuries (eg, anterior cruciate ligament rupture), these findings have been inconsistent across studies. ^3,19,26 More importantly, there have been no studies to date that support using motion capture (either marker based or markerless) to screen large cohorts of athletes or servicemembers for risk of MSKI. Given the multifactorial cause of MSKI, some authors have suggested that developing a movement screening tool to identify injury-prone individuals is an unrealistic goal. ^2,26 However, despite this dearth of supportive evidence in the literature and the skepticism among authors regarding the feasibility of screening for MSKI using motion analysis, proprietary MMC systems have emerged as a popular choice among military and collegiate athletic programs eager to identify vulnerable individuals.

To the best of the authors’ knowledge, this is the first study that has assessed the ability of a proprietary MMC system to differentiate between individuals who were and were not likely to suffer an MSKI in a military training setting. Our analysis found that neither the DOV nor the DLV scores were helpful in discriminating between those who were and were not likely to suffer an MSKI. While there were significant univariate associations between both DARI scores and MSKI during SW Prep, these associations were not clinically relevant. ⁶ This lack of clinical relevance was also seen in ROC curve AUC values. An ROC curve is an analytical tool used widely to assess the ability of a continuous measure to discriminate between a binary outcome such as the presence of a disease or condition. ¹⁵ The minimal AUC value for a test to provide acceptable discriminatory ability (ie, identify individuals with and without a condition) is 0.7. ²² The DARI scores had AUC values of 0.53 and 0.54, suggesting no clinically meaningful ability to discriminate between those who were and were not likely to suffer an MSKI (Figure 2). ²²

When DARI scores were adjusted for age, BMI, and previous injury status, the only associations that remained significant of the 4 that were analyzed were the DOV scores and LE MSKI. This small, but significant, association was most likely due to the large sample size (n = 1587), which allowed us to detect small, but clinically meaningless, relationships. Our findings also indicated that those SW trainees with lower BMI values, as well as those who had previously suffered an MSKI before starting SW Prep, were more prone to suffer an MSKI during the surveillance period. These findings are consistent with previous studies on MSKI risk in military populations. ^17,31

The DARI motion device tested displayed poor reliability, with the majority of vulnerability scores displaying ICCs <0.5. It is therefore plausible that the lack of predictive capacity of the DOV and DLV scores is a function of their poor repeatability and general test-specific variation. Whether this is a device-related issue or a function of the complex movements utilized for testing as specified by the manufacturer remains to be examined. ⁴ The accuracy of MMC systems varies according to the data collection method and technology used. ⁷ Further, the validity of markerless systems compared with marker-based systems has yet to be fully established across various movements. ⁷ As a result of these compounding limitations, this MMC system lacks the ability to consistently identify faulty movement in the first place and, thus, appears to be unable to detect faulty movements indicative of future risk of injury.

Limitations

There are 2 primary limitations to this study. The first is that this was a retrospective study design, which introduced a risk of bias. ¹³ However, this risk was somewhat mitigated in our study because the baseline variables (DARI scores, age, BMI, previous MSKI) were collected before the MSKI surveillance period (SW Prep). In addition, we had no missing data for the variables collected. ¹ The second limitation is that we did not consider whether or not there was a duty-limiting profile associated with the injuries that were captured. ²⁷ Including all injuries regardless of limited duty status introduced the possibility that minor injuries that did not affect training were included in the analyses, which could have potentially confounded the results.

Conclusion

The US Food and Drug Administration—approved MMC system assessed in this study, when used per manufacturer specifications, generated scores that had no clinical utility and exhibited poor to moderate test-retest reliability. Until these issues are addressed, it is unlikely such devices present any utility for the screening of tactical athletes for the risk of future MSKI.