Injury Risk Estimation Expertise

Abstract

Background:

Simple observational assessment of movement is a potentially low-cost method for anterior cruciate ligament (ACL) injury screening and prevention. Although many individuals utilize some form of observational assessment of movement, there are currently no substantial data on group skill differences in observational screening of ACL injury risk.

Purpose/Hypothesis:

The purpose of this study was to compare various groups’ abilities to visually assess ACL injury risk as well as the associated strategies and ACL knowledge levels. The hypothesis was that sports medicine professionals would perform better than coaches and exercise science academics/students and that these subgroups would all perform better than parents and other general population members.

Study Design:

Cross-sectional study; Level of evidence, 3.

Methods:

A total of 428 individuals, including physicians, physical therapists, athletic trainers, strength and conditioning coaches, exercise science researchers/students, athletes, parents, and members of the general public participated in the study. Participants completed the ACL Injury Risk Estimation Quiz (ACL-IQ) and answered questions related to assessment strategy and ACL knowledge.

Results:

Strength and conditioning coaches, athletic trainers, physical therapists, and exercise science students exhibited consistently superior ACL injury risk estimation ability (+2 SD) as compared with sport coaches, parents of athletes, and members of the general public. The performance of a substantial number of individuals in the exercise sciences/sports medicines (approximately 40%) was similar to or exceeded clinical instrument-based biomechanical assessment methods (eg, ACL nomogram). Parents, sport coaches, and the general public had lower ACL-IQ, likely due to their lower ACL knowledge and to rating the importance of knee/thigh motion lower and weight and jump height higher.

Conclusion:

Substantial cross-professional/group differences in visual ACL injury risk estimation exist. The relatively profound differences in injury risk estimation accuracy and their potential implications for risk screening suggest the need for additional training and outreach (see http://www.ACL-IQ.org).

Clinical Relevance:

Parents and sport coaches would likely benefit from training or use of decision support tools such as the ACL nomogram to assess ACL injury risk. In addition, physicians and other sports medicine professionals may also benefit from improving risk estimation performance to reach clinical biomechanical standards.

Sport-related anterior cruciate ligament (ACL) injuries are a significant economic and global health problem that disproportionately affect young female athletes.^1,12,15,30 The economic burden in the United States alone has been estimated to exceed $3 billion annually.¹³ Athletes sustaining an ACL injury lose substantial time out of sport and school and are at greater risk for reinjury and osteoarthritis.^1,15,30 Prevention techniques such as physical or neuromuscular training have been shown to be effective for reducing ACL injuries.^19,26 However, the time and resources involved in administering large-scale prevention programs are nontrivial.^7,8,21

One potential solution to reduce prevention time and increase effectiveness would be to ensure that practitioners have the ability to accurately and reliably assess ACL injury risk via real-time observation. First, having individuals who are skilled at predicting risk of injury without the use of additional tools or augmentation would significantly reduce screening time and cost over current biomechanical instrument-based methods.^10,18 Second, successful injury prevention programs emphasize biomechanical technique correction or feedback.⁹ To provide such feedback, the observer must have the ability to detect anomalies in movement patterns that would place an individual at risk for injury.

Limited research using a small number of physical therapists has begun to answer questions related to an individuals’ ability to accurately assess ACL injury risk via observation.^{4,20,25,28,29} Three of these studies used a limited number of raters (1 observer in the study by Stensrud et al,²⁵ 3 observers in that by Nilstad et al²⁰ and Ekegren et al⁴), and hence, have ignored the impact of individual differences on risk estimation ability. When a larger sample of observers were studied by Whatman and colleagues,²⁹ initial evidence for skill-based differences in risk estimation ability emerged. However, the ability to generalize from these studies to ACL injury risk estimation in the real world is limited due to the confounding judgment task instructions, criterion choice (ie, knee medial to toe is not the best predictor of actual ACL injury risk), and representativeness of stimuli (ie, individuals aged approximately 11 years are not at greatest risk for ACL injury). Moreover, all these studies utilized physical therapists and so it is unknown whether results will be generalizable to other individuals who would benefit from assessing ACL injury risk, including physicians, athletic trainers, sport coaches, strength and conditioning coaches, parents of athletes, and the athletes themselves.

Building on this work, Petushek and colleagues²³ developed and validated a brief test of ACL injury risk estimation skill: the ACL Injury Risk Estimation Quiz (ACL-IQ; http://www.ACL-IQ.org). The test is intended to measure an individual’s ability to visually estimate an athlete’s risk for an ACL injury by watching videos of drop vertical jumps where the responses are compared with concurrent 3-dimensional (3D) biomechanical measurement of an ACL injury risk factor. Drawing on a large sample of individuals (600+ participants) that included coaches, sport medicine practitioners, and computer-literate individuals from the general population, initial studies documented stable, superior, and reproducible skill-based differences in ACL injury risk estimation. Preliminary results further suggested that some individuals may even outperform clinical-based biomechanical screening methods (eg, ACL injury risk nomogram; see Myer et al¹⁸ for development and validation). Nevertheless, research has yet to precisely analyze and compare performance differences among specific subgroups (eg, physicians, athletic trainers, physical therapists, coaches, and parents) who may differ in their level of expertise with respect to ACL injury risk estimation.

The purpose of this study was to assess skill-based differences in visual estimation of ACL injury risk across various groups who would benefit from or be likely to use observational movement analysis for ACL injury risk assessment. Based on our previous research, we know that exercise science professionals, as a whole, can more accurately estimate ACL injury risk compared with non–exercise science individuals (eg, general public, sport coaches, parents, and athletes).²³ However, do subgroup differences in ACL injury risk estimation skill exist? Are coaches better than parents? Are physical therapists better than physicians? In addition, how does performance of the subgroups compare with an optimized clinically available instrument-based screening method (ACL nomogram)? We hypothesized that groups specializing in sports medicine/rehabilitation and having extensive knowledge/experience in ACL injury/prevention such as physicians, physical therapists, and athletic trainers will perform best and similar to the ACL nomogram. Subgroups hypothesized to have some experience and knowledge regarding ACL injury/prevention such as sport coaches, strength and conditioning coaches, and exercise science academics/students will perform better than parents and other general population individuals but poorer than the aforementioned sports medicine/rehabilitation professionals.

Based on recent perceptual-cognitive modeling of ACL-IQ skill, various factors such as ACL knowledge (eg, location, function, risk factors) and cue utilization (eg, focusing on the amount of medial knee motion and ignoring jump height and weight of the athlete) appear to account for major performance differences.²² Accordingly, subgroup analysis of these factors may provide new insight into specific areas for targeted risk estimation training.

Methods

Participants

Participants included 428 individuals who were previously part of the development and validation of the ACL-IQ (see Table 1 for demographic and occupational data).²³ Of the physicians sampled, 81% specialized in orthopaedics/sports medicine and 19% in family medicine. Participants were recruited via email, through personal networks, listserv/blog/social media posts, and from a paid web panel. Institutional review board approval was obtained through both Cincinnati Children’s Hospital and Michigan Technological University.

Table 1.

Participant Demographic Characteristics^a

Group	n	Age, y, Mean ± SD	Sex, Within-Group %
			Female	Male
ExSci student	27	24.00 ± 3.32	52	48
ExSci academic	21	38.05 ± 9.56	33	67
S&C coach	34	30.09 ± 6.70	24	77
Athletic trainer	50	31.52 ± 7.96	50	50
Physical therapist	46	35.26 ± 9.09	35	65
Physician	36	45.83 ± 12.30	17	83
Sport coach	32	31.19 ± 9.47	63	38
Parent of athlete	26	44.92 ± 9.31	77	23
Female athleteb	11	20.82 ± 1.40	100	0
General public	145	36.49 ± 12.87	53	47

^aExSci, exercise science; S&C, strength and conditioning.

^bAge ≤25 years.

Procedures

Participants completed the web-based, 5-item ACL-IQ (see http://www.ACL-IQ.org and Petushek et al²³ for development, reliability, and validation). The ACL-IQ is composed of five 3- to 5-second video clips of female athletes (mean ± SD: age, 15.9 ± 1.3 years; height, 163.6 ± 9.9 cm; body mass, 57 ± 12.1 kg) performing a 2-footed drop vertical jump from a 31 cm–high box. The athletes featured in the videos participated in landing and cutting sports. Coincidentally, these athletes also served as the participants in the development and validation of the clinical ACL nomogram.¹⁷ The ACL nomogram is a clinic-based ACL injury prediction algorithm that uses 2 standard video cameras, measuring tape, and isokinetic dynamometry to identify female athletes with high knee abduction moment. The algorithm works by measuring the amount of knee valgus and flexion motion during the drop vertical jump as well as tibia length, body mass, and quadriceps-to-hamstrings isokinetic torque ratio. The variables are then recorded and summed on a clinician-friendly nomogram to compute the probability of high knee abduction moment. All 5 of the current video clips used in the ACL-IQ had concurrent ACL nomogram scores.

After viewing each drop jump, participants were asked immediately to rate the risk for future ACL injury on a scale from 1 to 10 (Figure 1). Following risk assessment of each jump, participants also indicated the confidence in their risk rating on a scale from 1 to 10. No other instructions or training (eg, what to focus on) was used. Participants’ risk rating responses were compared with the athletes’ concurrent 3D biomechanical analysis of knee abduction moment. For example, the knee abduction moments were linearly transformed on a scale from 1 to 10 so that a simple difference score could be calculated and summed for a total error score (and inverted to show percentage correct). After completing the ACL-IQ, participants completed a brief survey where they reported the importance of 11 visual cues (arm motion, landing symmetry, inward/outward knee and thigh motion, lateral trunk motion, landing stiffness, foot alignment, height of individual, weight of individual, jump height, and jump alignment) on a scale from 1 to 10 for making their risk assessment decision. For example, if an individual thought that landing stiffness was a very important cue for making their injury risk assessment, they would rate this cue toward 10. Participants also answered 11 questions related to the ACL location, function, and risk factors for ACL injury to capture ACL-specific knowledge.

Figure 1.

Example item from the ACL Injury Risk Estimation Quiz. Images are snapshots of a video sequence. ACL, anterior cruciate ligament.

Statistical Analysis

Separate univariate 1-way analyses of variance were used to compare ACL-IQ, mean confidence, ACL knowledge, and visual cue importance ratings across the 10 groups (exercise science students, exercise science academics, physical therapists, athletic trainers, physicians, strength and conditioning coaches, female athletes, sport coaches, parents, and general public). Post hoc pairwise comparisons using the Tukey honestly significant difference test were used to follow up significant main effects. Independent single-sample t tests were conducted to compare ACL-IQ scores of each subgroup with the ACL nomogram score. Monte Carlo simulations with 10,000 iterations revealed that average “guessing” performance level on the ACL-IQ was 52%. Thus, independent single-sample t tests were also conducted to compare ACL-IQ scores of each subgroup with guessing or chance performance. All statistical analyses were conducted using SPSS version 21 (IBM Corp). The a priori alpha level was set at P < .05.

Results

Specific occupation ACL-IQ scores are depicted in Figure 2 (top panel) and in a standardized mean difference (Z score) form (Figure 2, bottom panel). The parents and general public performed worse than all other groups (P < .05). Female athletes performed worse than exercise science students (P < .05) (only 11 non–exercise science/sports medicine female athletes were included in the sample; thus, mean estimates are imprecise). Sport coaches displayed lower ACL-IQ scores than exercise science students and academics, physicians, strength and conditioning coaches, athletic trainers, and physical therapists (P < .05). There was no statistically significant difference in ACL-IQ between exercise science students and academics, physicians, strength and conditioning coaches, athletic trainers, or physical therapists (P < .05). All occupational/professional subgroups displayed similar mean confidence ratings across the 5 items/video clips in the ACL-IQ (P = .17) (Figure 3).

Figure 2.

(A) ACL Injury Risk Estimation Quiz (ACL-IQ) means and (B) standardized mean differences across various professions/groups. ACL, anterior cruciate ligament; ExSci, exercise science; S&C, strength and conditioning.

Figure 3.

Confidence in chosen risk rating across various professions/groups. ExSci, exercise science; S&C, strength and conditioning.

When scores were compared with ACL nomogram performance, the ACL nomogram performed better than all groups except exercise science students, t(26) = –1.01, P = .32 (Figure 2, top panel with nomogram line). However, a substantial number of individuals performed similar to or better than the ACL nomogram (Table 2). Parents and the general public groups were not statistically different from guessing performance.

Table 2.

Proportion of Individuals At or Above ACL Nomogram Performance^a

Group	At or Above ACL Nomogram, Within-Group %
ExSci student	56
Athletic trainer	48
Physical therapist	44
S&C coach	41
Physician	25
ExSci academic	19
Sport coach	19
Female athlete	18
General public	3
Parent of athlete	0

^aACL, anterior cruciate ligament; ExSci, exercise science; S&C, strength and conditioning.

Anterior cruciate ligament knowledge scores across subgroups are depicted in Figure 4. Sport coaches, parents, female athletes, and the general public displayed less ACL knowledge compared with exercise science students and academics as well as strength and conditioning coaches, athletic trainers, physical therapists, and physicians (P < .05). There were no significant differences in ACL knowledge between exercise science students and academics, physicians, strength and conditioning coaches, athletic trainers, or physical therapists (P > .05).

Figure 4.

Anterior cruciate ligament (ACL) knowledge across various professions/groups. ExSci, exercise science; S&C, strength and conditioning.

The importance ratings of various cues are depicted in Figure 5. Exercise science students and academics, physicians, strength and conditioning coaches, athletic trainers, and physical therapists rated knee/thigh motion as more important for assessing injury risk compared with the general public group (which was not statistically different from parents, female athletes, or sport coaches) (P < .05). Strength and conditioning coaches and athletic trainers rated trunk motion as more important for assessing injury risk compared with the general public subgroup (P < .05). No statistically significant differences between groups were displayed for the importance rating of height. The general public group rated weight as more important for assessing injury risk compared with athletic trainers (P < .05). Parents rated weight as more important for assessing injury risk compared with exercise science students and academics, physicians, athletic trainers, and physical therapists. Sport coaches rated weight as more important for assessing injury risk compared with athletic trainers (P < .05). Parents and general public groups rated jump height as more important for assessing injury risk compared with exercise science students and academics, physicians, strength and conditioning coaches, athletic trainers, and physical therapists. Sport coaches rated jump height as more important for assessing injury risk compared with exercise science students and academics, physicians, athletic trainers, and physical therapists. The general public group rated jump alignment as more important for assessing injury risk compared with exercise science academics (P < .05). Finally, parents rated jump alignment (eg, jump takeoff angle) as more important for assessing injury risk compared with exercise science students and academics (P < .05).

Figure 5.

Visual cue importance ratings across various professions/groups. The groups are depicted in order of ACL-IQ score from lowest (left) to highest (right) as indicated by the arrow. ACL-IQ Score, ACL Injury Risk Estimation Quiz; ExSci, exercise science; S&C, strength and conditioning.

Discussion

This investigation measured cross-professional differences in estimating ability of ACL injury risk. Specifically, parents, sport coaches, and individuals not in the sport medicine/exercise science fields (general public), on average, performed poorly and similar to chance performance. Interestingly, and contrary to our hypotheses, strength and conditioning coaches performed similar to sports medicine/therapy professionals. Exercise science students, physical therapists, and athletic trainers performed at levels that were roughly equivalent to that of the clinical instrument-based ACL injury risk assessment method (ie, ACL nomogram). Conversely, only 25% of physicians performed similar to or better than the ACL nomogram, but they were not statistically different from these superior performers. The lack of statistical significance is likely due to multiple comparison error correction as visual inspection of the standardized scores reveals that exercise science academics and physicians performed around “average” while physical therapists and exercise science students performed around 1 SD above the average.

The high level of sensitivity and accuracy achieved using a 10-point system (as opposed to a commonly used dichotomous system^4,28) is surprising given that no training or instructions on how to assess injury risk were given and raters were allowed only a single viewing. Although the overall level of performance was relatively high, the majority of the sample studied did not reach the performance level of the ACL nomogram, and thus, may benefit from using this tool for future ACL risk assessments. Future research will focus on developing training systems or other decision support tools such as checklists or decision trees to improve injury risk estimation ability. Despite considerable cross-professional/group differences, confidence ratings remained relatively high and stable across all groups (mean, 7.5/10). Put differently, the lesser skilled groups remained confident in their risk assessment despite being inaccurate, adding evidence to the unskilled and unaware phenomena.¹⁴

The conclusions from this cross-sectional analysis parallel recent mediation and moderation results that determined the specific perceptual-cognitive factors (eg, visual cues and knowledge) influencing performance.²² That is, parents, sport coaches, and general public have lower ACL-IQ likely because of their lower ACL knowledge and rating the importance of knee/thigh motion lower and weight and jump height higher. The slightly higher ACL-IQ of sport coaches over the general public group is likely due to the slightly higher ACL knowledge and higher rating of knee/thigh motion. These 4 factors (ACL knowledge and cue importance ratings for knee/thigh motion, jump height, and weight) have been shown to be the dominant factors influencing ACL-IQ performance even when considering ACL injury risk assessment experience, educational level, personality, and domain general perceptual-cognitive abilities such as general mental rotation and risk literacy.²² Theoretically, modifying any of these factors should improve performance. However, the most efficient and effective method for training visual assessment skill has not yet been investigated. Future research may benefit from utilizing higher fidelity cognitive process tracing methods such as eye-tracking or verbal protocol analysis to more reliably and accurately assess the judgment and decision-making strategies to develop optimal training or decision support systems.^{2,3,5,6,11,27}

Athletes are surrounded by many individuals and support staff who aim to enhance performance and reduce injuries. Coaches and parents have the most player contact time and are likely the most influential people for changing behavior. However, these individuals are lacking the skill to accurately identify athletes at risk for ACL injury. Of the sports medicine professions who are likely to have the greatest influence on injury prevention, physicians need the largest improvement in performance—especially if they want to attain performance levels similar to the ACL nomogram. It will therefore be important to target these individuals for improving risk assessment performance or to adopt the ACL nomogram to aid their injury risk assessment in practice.

A limitation to this study is that the criterion used to assess ACL injury risk was quantified using 3D biomechanical analysis as opposed to actual injury cases. Furthermore, additional studies are needed to reproduce the findings that knee abduction moment is a significant risk factor for ACL injury in young female athletes. However, previous prospective, cross-sectional, and cadaveric studies support knee abduction moment as a modifiable risk factor for ACL injury in young female athletes.^10,16,18,24 In addition, in order for screening to reduce ACL injuries, an appropriate intervention must be implemented, such as neuromuscular training.²⁶ Larger samples of individuals within the groups currently investigated (eg, female athletes) in addition to using more liberal post hoc tests such as the Fisher least significant difference may likely reveal other statistically significant cross-professional/group differences. Finally, it is essential to note the importance of detailed instruction for performing the drop vertical jump to ensure valid risk assessment (eg, stance width, drop height, and overhead target).

Conclusion

Overall, the findings from this study identified the groups that need the most improvement in their ACL injury risk assessment ability. Sport coaches and parents may benefit from improvement of visual assessment, and even sports medicine practitioners—especially physicians—would benefit from improvement to reach the level of clinical-based biomechanical assessment systems. The ACL-IQ is an assessment technology and feedback system for ACL injury risk prediction ability. Individuals can assess their ACL injury risk prediction ability with a short, free, and online (http://www.ACL-IQ.org) tool. Future research will focus on developing efficient methods to improve observational risk prediction performance as well as establishing evidence that individuals with high ACL-IQ can reduce ACL injuries.