Skip to main content
PMC home page
NIHPA Author Manuscripts logoLink to NIHPA Author Manuscripts
. Author manuscript; available in PMC: 2016 Nov 9.
Published in final edited form as: J Sport Rehabil. 2010 Feb;19(1):41–56. doi: 10.1123/jsr.19.1.41

Expert Versus Novice Interrater Reliability and Criterion Validity of the Landing Error Scoring System

James Onate 1, Nelson Cortes 1, Cailee Welch 1, Bonnie Van Lunen 1
PMCID: PMC5102003  NIHMSID: NIHMS274774  PMID: 20231744

Abstract

Context

A clinical assessment tool that would allow for efficient large-group screening is needed to identify individuals potentially at risk for anterior cruciate ligament (ACL) injury.

Objective

To assess the criterion validity of a jump-landing assessment tool compared with 3-dimensional (3D) motion analysis and evaluate interrater reliability across an expert vs novice rater using the Landing Error Scoring System (LESS).

Design

Validity protocol.

Setting

Controlled, laboratory.

Participants

Nineteen female (age 19.58 ± .84 y, height 1.67 ± .05 m, mass 63.66 ± 10.11 kg) college soccer athletes volunteered.

Main Outcome Measurement

Interrater reliability between expert rater (5 y LESS experience) vs novice rater (no LESS experience). LESS scores across 13 items and total score. 3D lower extremity kinematics were reduced to dichotomous values to match LESS items.

Interventions

Participants performed drop-box landings from a 30-cm height with standard video-camera and 3D kinematic assessment.

Results

Intrarater item reliability, assessed by kappa correlation, between novice and experienced LESS raters ranged from moderate to excellent (κ = .459–.875). Overall LESS score, assessed by intraclass correlation coefficient, was excellent (ICC2,1 = .835, P < .001). Statistically significant phi correlation (P < .05) was found between rater and 3D scores for knee-valgus range of motion; however, percent agreement between expert rater and 3D scores revealed excellent agreement (range of 84–100%) for ankle flexion at initial contact, knee-flexion range of motion, trunk flexion at maximum knee flexion, and foot position at initial contact for both external and internal rotation of tibia. Moderate agreement was found between rater and 3D scores for trunk flexion at initial contact, stance width less than shoulder width, knee valgus at initial contact, and knee-valgus range of motion.

Conclusions

Our findings support moderate to excellent validity and excellent expert vs novice interrater reliability of the LESS to accurately assess 3D kinematic motion patterns. Future research should evaluate the efficacy of the LESS to assess individuals at risk for ACL injury.

Keywords: screening, jump landing, anterior cruciate ligament

Despite great efforts by the sports medicine community to reduce gender disparity in anterior cruciate ligament (ACL) injuries, female ACL injury rates continue to be significantly higher than those of their male counterparts.1–8 The ACL-injury gender-disparity trend has been demonstrated in several comparable sports1,2,8,9 and nonathletic populations of similarly trained backgrounds (ie, military-related training). 4,10 ACL injuries typically result from noncontact mechanisms (70–75%),2,11,12 with the most common event reported to be during a jump-landing maneuver.11,13–15 Improper technique during jump-landing maneuvers may place significant force and strain on the ACL, hence causing the ligament to rupture. Because jump landing is the most common mechanism of noncontact ACL injury, it is believed that the assessment of jump-landing technique may serve as an important marker for identifying individuals at risk for noncontact ACL injury.

Several ACL injury-prevention programs have been developed in the hope of preventing this debilitating problem.16–21 The promising results of these ACL injury-prevention programs have resulted in an increased demand for prevention programs in the sports medicine community, but the feasibility of long-term implementation of these programs is costly because of manpower and time constraints. Recent recommendations from the International Olympic Committee concerning the need for baseline assessments to determine individuals most at risk for injury are provided with the concept of targeting tailor-made ACL injury-prevention programs to the most at-risk population.22 The concept of identification methods for targeting individuals at risk for injury in the preseason has gained recent exposure in the sports medicine literature.23 The investigation of jump-landing biomechanical patterns relative to increased ACL injury risk has also increased tremendously over the past decade, yet valid and reliable objective tests that can be used during clinical assessments are lacking.

Various sophisticated 3-dimensional motion-analysis assessments utilizing costly equipment have been conducted in attempts to identify factors for increased ACL injury risk. This costly and sophisticated approach provides outstanding objective kinematic and kinetic information, yet the cost, time, manpower, and accessibility of this type of assessment limit its value for clinical application. In 2003, Ford et al24 noted that no clinical system for accurate and practical screening and identification of individuals who may be at risk for ACL injury is currently available. McLean et al25 also indicated that being able to identify specifically at-risk individuals during their initial ground contact would allow greater opportunities to intervene with these individuals versus interventions in which some people may not need the training. In 2005, Noyes et al26 sought to develop a simple videographic test that would measure the distance between the hips, knees, and ankles in the coronal plane. They used the drop-jump screening test to assess gender and neuromuscular training differences in female athletes. The intraclass correlation coefficients for hip, knee, and ankle separation all showed values greater than .90, thus indicating excellent reliability. Nonetheless, 2 important considerations of this drop-jump screening test preclude its use across large-scale multicenter sites; one was the omission of the validation of this drop-jump screening method to known gold-standard criteria such as 3-dimensional motion-analysis instrumentation because of inability to assess multiple views with a single-camera setup, and the other was the time requirement to affix markers, digitize coordinates, and analyze the data. A clinically useful identification tool using a standard video camera that would allow for rapid and efficient large-group screening would be beneficial because it would be able to identify individuals potentially at risk for ACL injury before they begin sports participation to allow for time for an intervention.

A simple objective clinical tool specifically developed as a large-group ACL injury-risk-factor screening tool for military academy students, the Landing Error Scoring System (LESS), was created to identify individuals with poor jump-landing technique who may be at risk for noncontact ACL injury.27–29 The LESS is a relatively new clinical qualitative assessment tool for jump-landing technique, identifying potentially faulty movement patterns and poor technique as 2 cameras videotape the sagittal and frontal planes.27–29 An investigator watches the tape and records the error on a standard LESS scoring sheet that involves scoring individual joint motions at various moments in the landing sequence.27 Each component of the LESS is based on previous research that identified specific movements that facilitate the forces and strain acting on the lower extremity, specifically increasing ACL injury risk. These components were selected based on the influence of kinematic and kinetic studies assessing the relationship between movement patterns and ACL injury. The trained observer documents the knee-flexion angle at initial ground contact and throughout the entire range of motion, knee-valgus angle at initial contact and throughout the range of motion, foot–ground contact symmetry, trunk-flexion angle at initial contact, ankle-flexion angle at initial contact, foot stance width, and overall joint motion during a jump-landing task.

The purpose of this research study was to assess the validity of a simple clinical jump-landing movement-assessment tool, the LESS, in identifying subjective 2-dimensional jump-landing motion analysis compared with 3-dimensional high-speed motion-analysis assessment. In addition, the goal was to assess interrater reliability across expert and novice health care professionals (certified athletic trainers [ATCs]). We hypothesized that the LESS component scores would have a moderate to strong relationship with a 3-dimensional motion-analysis tool. We also hypothesized that the interrater reliability between expert and novice health care professional rater would be good to excellent.

Methods

Subjects

Nineteen female (age 19.58 ± .84 y, height 1.67 ± .05 m, mass 63.66 ± 10.11 kg) college soccer athletes from a Division I institution were selected based on a sample of convenience that was considered tenable based on evaluation of previous studies. 26–30 All subjects were free of hip, low back, knee, or ankle injuries within the last 6 months or surgeries within the last 2 years.30 The dominant leg was defined as the leg that the subject would use to kick a soccer ball as far as possible and was the leg tested for analysis purposes.24,31 All subjects signed an informed-consent form that was approved by the institutional review board before data collection.

Experimental Procedure

Subjects reported to the sports medicine research laboratory and completed a questionnaire to determine their eligibility to participate in the study. Those who were eligible and agreed to participate signed an informed-consent form before data collection. Subjects wore spandex shorts, sports bra, and other tight-fitting clothing and the team running shoes provided at the beginning of the season. A 10-minute warm-up consisting of cycling and self-directed stretching was given before data collection. Weight and height were obtained, and a caliper was used to measure knee width, ankle width, elbow width, wrist width, and hand thickness. A measuring tape was used to measure leg length, taken from the greater trochanter to the medial malleolus. The same researcher made all anatomical measures. After the warm-up period, 39 reflective markers were placed on specific body landmarks according to the Helen Hayes model.32,33 A static trial was then collected for later calculation of the kinematic model.

Drop-Jump Task

After the static trial, the subjects were given instructions regarding the drop-jump task. Each subject performed 3 practice jumps off a 30-cm box and landed with both feet on the force plates. A double-leg landing was performed to assess the relationship of the 2 limbs during the landing task. The box was placed 30 cm from the force plates (Figure 1). While standing on the box subjects were instructed to not jump off the box but to shift their weight and lean and drop from the box as vertically as possible, in an attempt to standardize jump-height-landing requirements. The purpose of this approach was to minimize upward vertical displacement off the box, effectively trying to standardize the downward vertical displacement across subjects. Subjects were asked to land on the force plates, jump as high as they could straight up in the air as if they were doing a soccer header, and finally land back on the force plates. At that time the entire foot needed to be on the force plate, with each foot on a separate force plate. The initial landing from the box was used for the purpose of analyses, with the second landing from the maximal vertical jump being discarded. Each subject performed 3 successful trials, with 1-minute rest periods between trials to minimize the effects of fatigue.

Figure 1.

Figure 1

Open in a new tab

Lab setup for data collection. Box placed 30 cm from force plates with 30 cm height. Eight VICON MX-F40 cameras and 2 Sony video cameras.

Instrumentation

Eight high-speed video cameras (Vicon Motion Systems Ltd, Oxford, England) were used to monitor the motion analysis of the lower extremity, with a sampling rate of 500 Hz. Two Bertec force plates, Model 4060-NC (Bertec Corp, Columbus, OH, USA) with a sampling rate of 500 Hz were used to measure ground-reaction forces. Two Sony Mini-DV Handycam camcorder DCR-HC40 (Sony Corp, New York, NY, USA) video cameras with a sampling rate of 30 Hz were used to record the trials; one on the mediolateral right side (all subjects were right-side kick-leg dominant) of the subject, and the other showing the anterior view of the subject.

From the standing trial a full-body kinematic model was created for each subject using Visual 3D (C-Motion, Rockville MD, USA). This kinematic model was used to quantify motion at the trunk, hip, knee, and ankle joints in the sagittal and frontal planes (Figure 2). The calculated kinematics for each joint using Visual 3D were determined from the standing calibration trial. A Cardan angle sequence (x, y, and z, which represent flexion–extension, abduction–adduction, and internal–external rotation, respectively) was used to calculate joint angles34; thus, positive values represent hip flexion, ankle dorsiflexion, trunk flexion, and knee extension. The pelvis was modeled as a cylinder, and the lower extremity, as frustums of cones. A frequency analysis using the Welch averaged, modified periodogram method with a 512-data-point Hanning window was calculated for residual frequency analysis. This analysis determined 25 Hz as an optimal cutoff frequency for both kinematic and kinetic data. Thus, kinematic and kinetic data were low-pass filtered through a fourth-order Butterworth zero-lag filter with a 25-Hz cutoff frequency.

Figure 2.

Figure 2

Open in a new tab

Example of 2-dimensional Landing Error Scoring System and 3-dimensional kinematic analyses.

Data Reduction and Analysis

The 3-dimensional kinematic and kinetic data were reduced using custom-made K2DS (kinematic and kinetic data simplification) MATLAB 6.1 (The MathWorks, Inc, Natick, MA, USA) software to export into a Microsoft Excel spreadsheet. The 3 trials were exported into SPSS 16.0 (SPSS Inc, Chicago, IL, USA) for data analysis and then averaged into a single score. The kinematic data (ankle flexion, ankle eversion, knee flexion, knee valgus, hip flexion, and trunk flexion) were coded as 0 or 1 to correspond to the LESS scores. As an example, if a subject landed with less than 30° of knee flexion at initial contact measured by the VICON, a score of 1 would be given to that subject; otherwise, a 0 would be given. For item 6 on the LESS (symmetric foot contact), vertical ground-reaction force was analyzed. For the purpose of this study the initial contact was determined as the point when the force plates recorded a value equal to or greater than 10 N. If both force plates had a value equal to or greater than 10 N at the same time frame, a 0 was given; otherwise, a 1 would be given to indicate an asymmetric landing. When this latter case occurred, the time difference between the feet contacting the force plates was recorded for each trial and kept for analysis.

Two ATCs scored the 3 drop-jump trials using the LESS instrument. The expert rater had 15 years of experience and the novice rater less than 1 year experience as an ATC. The expert rater was involved in the development of the LESS scoring form and provided a 1-hour training session to the novice rater, who had never completed a LESS assessment. The training session consisted of reviewing each item on the LESS scoring form and conducting sample landing analyses of a pilot subject to evaluate the novice rater’s responses. After the training session, the raters separately viewed the trials on their personal laptop computers as many times as desired, with an average scoring time ranging from approximately 2 to 4 minutes per individual. Each rater’s first of the 3 trials was used for the analysis to reduce possible biases from the raters while scoring each of the subjects. To assess the agreement between raters, a kappa statistical analysis was used. Kappa statistics measure interrater agreement for categorical variables (dichotomous), which is more robust than a percentage agreement because the agreement occurring by chance is taken into account. However, when the raters have high agreement on the scoring (ie, 1 score given to all subjects by either of the raters falling in a single cell) then it is only possible to compute percentage agreement. This is a paradox in kappa statistics—high agreement can yield low kappa values (ie, chance of agreement).35–38 Finally, an intraclass correlation coefficient (ICC2,1) was calculated for the overall score of the LESS between the novice and expert raters.

Phi-correlation-coefficient analysis measures the association between 2 dichotomous variables. Originally 1 of the variables is continuous and dichotomized for analysis purposes. The VICON data were dichotomized into 0 and 1 to match the LESS scoring system. As an example, if the 3-dimensional data for knee flexion presented a value greater than 30° then a score of 0 was given to dichotomize that variable. Thus, the same dichotomization score was used between the criterion and the rater to allow for statistical analysis. Phi is a form of Pearson product–moment correlation (r), given by a 2 × 2 contingency table.39–41 A paradox similar to that with the kappa statistic is observed with phi correlation coefficients. This is because both statistical analyses use a 2 × 2 contingency table, creating analogous paradoxes. For phi correlation, when all frequency is observed within a single cell it is not possible to compute the phi-correlation value, and only percentage agreement is reported. It should be noted that an item with low correlation could still yield a high percentage agreement.41,42 This means that most of the data fall within a unique cell, and there is not a frequency distribution between various cells. This statistical analysis was conducted to evaluate the degree of association between the LESS scores and the Vicon data.

The scores of the expert rater were used to calculate the phi coefficient correlations between the LESS scores and the 3-dimensional analysis. An alpha level of P < .05 was set a priori for statistical significance. An individual item analysis assessment of percent agreement, based on the number of equal scores between the expert rater and the 3D-instrument dichotomous score, was calculated and defined as poor (less than 50% agreement), moderate (51–79% agreement), or excellent (80% and above agreement). As an example, if the expert rater and the 3-dimensional instrument equally scored the same 10 out of 19 subjects, then the percentage agreement would be 10 divided by 19, which would equal 52.6%.

Results

Interrater Agreement (Novice vs Expert)

See Table 1. Items 1, 5, 8, and 10 had perfect agreement between raters (100%), with the same score given to all subjects by both raters; thus no kappa statistics are reported for these items. The raters had significant agreement on items 4 (κ = .459, P < .015, 90% observed agreement), 6 (κ = .875, P < .001, 95% observed agreement), 7 (κ = .643, P = .002, 95% observed agreement), 9 (κ = .615, P = .003, 80% agreement), 12 (κ = .643, P = .002, 95% observed agreement), and 14 (κ = .769, P = .001, 85% observed agreement). A perfect observed agreement (100%) was found between the raters for items 11 (κ = 1.0, P < .001) and 13 (κ = 1.0, P < .001). There was moderate agreement between raters for item 15 (κ = .553, P = .011, 65% observed agreement). Finally, there was excellent reliability between novice and expert LESS overall scores (ICC2,1 = .835, P < .001).

Table 1.

Kappa Values and Percentage Agreement for Interrater Agreement on the Categorical Landing Error Scoring System Items

Item Kappa P Agreement
4: Knee-flexion range of motion >30° .459 .015 90%
6: Initial foot contact (symmetry) .875 <.001 95%
7: Foot position at initial contact: toes > 30° of external rotation .643 .002 95%
9: Stance width at initial contact less than shoulder width .615 .003 80%
11: Knee valgus at initial contact: knees over midfoot 1.0 <.001 100%
12: Lateral trunk flexion at initial contact .643 .002 95%
13: Knee-valgus range of motion: greater than great toe 1.0 <.001 100%
14: Joint displacement (sagittal plane) .769 .001 85%
15: Overall impression .553 .011 65%
Open in a new tab

LESS Validity

See Table 2. For item 1, analysis of ankle flexion at initial contact, the LESS rater identified all subjects in a plantar-flexion position, whereas the 3-dimensional tool identified 2 subjects in dorsiflexion and 17 in plantar flexion, which represents a total of 89.5% of correct scores by the rater. There was no statistically significant correlation between instruments for item 2 (rφ = .118, P = .608). The mean for knee flexion at initial contact measured by the 3-dimensional system was 18.31° ± 7.40°, with a range of 10° to 37°. Only 10.5% had a knee-flexion value equal to or above 30° at initial contact as measured by the 3-dimensional tool. In addition, a poor (21%) agreement between LESS score and 3-dimensional system was found for knee flexion at initial contact.

Table 2.

Landing Error Scoring System Phi Correlation and Percent Agreement Between Expert Rater and 3-Dimensional Motion-Analysis System

Item Correlation Agreement 3-Dimensional,
Mean ± SD		 Phi
1: Ankle flexion at initial contact 89% (excellent) −10.13° ± 12.02° a
2: Knee flexion at initial contact p = .608 21% (poor) 18.31° ± 7.40° rφ = .118
3: Trunk flexion at initial contact p = .582 74% (moderate) −0.2° ± 5.9° rφ = −.130
4: Knee-flexion range of motion 84% (excellent) 65.31° ± 11.53° a
5: Trunk flexion at maximal knee flexion 95% (excellent) 6.14° ± 5.14° a
6: Initial foot contact 42% (poor) 7-ms difference a
7: Foot position at initial contact: toes >30° external rotation p = .310 85% (excellent) 22.69 ± 14.88 rφ = −.233
8: Foot position at initial contact: toes >30° internal rotation 100% (excellent) 22.69 ± 14.88 a
9: Stance width < shoulder width 74% (moderate) 6 ± 3.6 cm a
10: Stance width > shoulder width 100% (excellent) 6 ± 3.6 cm a
11: Knee valgus at initial contact p = .414 68% (moderate) −0.67° ± 3.77° rφ = –.188
12: Lateral trunk flexion at initial contact 10% (poor) 1.0 ± 0.8 cm a
13: Knee-valgus range of motion p = .047* 74% (moderate) 0.94° ± 9.31° rφ = .456
Open in a new tab
a

Could not be calculated because of lack of variance in either 3-dimensional instrument or rater score.

*

Significant correlation, P < .05.

Trunk flexion at initial contact (item 3) did not present a statistically significant correlation between instruments (rφ = −.130, P = .582). There was 74% agreement between the rater and the 3-dimensional system, with the same score being equally given to 14 out of 19 subjects. For item 4, knee-flexion range of motion higher than 30°, a mean of 65.31° ± 11.53° was found for the 3-dimensional instrument, meaning all subjects had more than 30° of knee range of motion. The rater correctly scored 16 subjects (84.3% agreement) as having a range of motion greater than 30°. For trunk flexion at maximum knee flexion (item 5), the mean angle was 6.14° ± 5.14°, with 88% of the subjects being properly scored by the rater.

As for vertical ground-reaction force at initial contact (item 6), all subjects had asymmetric foot contact, with values ranging from 1 to 15 milliseconds, whereas the rater only accurately scored 42% as asymmetric, although there was a minimal mean difference between the time that the feet contacted the force plates (7 ± 4 ms). There was no significant correlation between the 3-dimensional instrument and the rater score on ankle eversion, which corresponds to LESS item 7 concerning foot position at initial contact: toes >30° external rotation of foot (rφ = −.233, P = .310). Three subjects were identified by the rater as being higher than 30°, and 6 were scored as such by the 3-dimensional data, so the rater correctly scored 85% of the subjects. For item 8 (ankle inversion greater than 30° corresponding to foot position at initial contact: toes >30° internal rotation of foot) there was absolute agreement (100%) between the rater and the values from the 3-dimensional system.

Items 9 and 10, stance width at initial contact less or greater than shoulder width, were computed by using the distance between the calcaneous markers for stance width and between the acromion markers for shoulder width. Based on the 3-dimensional analysis, all subjects had a smaller stance width than shoulder width, but the mean difference between the shoulder and stance width was only 6 ± 3.6 cm. The rater accurately scored 14 subjects as having smaller stance width than shoulder, which represents 74% of the sample. No statistically significant correlation was found for knee valgus at initial contact (item 11) between the rater and the 3-dimensional instrument (rφ = −.188, P = .414). Both the rater and the 3-dimensional instrument identified the same number of subjects (16) as being in knee valgus. However, there was a match in 13 subjects between rater and 3-dimensional instrument, which yields a moderate agreement of 68.42%.

Lateral trunk flexion was calculated as the difference in the x-axis location between the midpoint of the markers on the anterosuperior iliac spine and the sternum marker. All subjects had lateral trunk flexion when analyzed by the VICON system, resulting in poor agreement of 10%. The rater only identified 2 subjects as having lateral trunk flexion. It should be noted that the mean lateral trunk flexion was merely 1 ± 0.8 cm. The last LESS item for kinematic-analysis comparison between rater and 3-dimensional instrument, knee-valgus range of motion (item 13), was determined by the locations of the knee marker (estimated joint center) and the toe marker in the mediolateral axis. There was a statistically significant correlation between the rater and the kinematic analysis (rφ =.456, P = .047). The 3-dimensional instrument identified 5 subjects with knee-valgus range of motion greater than the great toe, whereas the expert rater identified 12 subjects with this motion. The rater correctly identified 74% of the cases when compared with the 3-dimensional instrument.

Discussion

The most important findings of this study were that interrater reliability of the LESS was excellent and the validity of the LESS instrument compared with the 3-dimensional motion analysis is strictly item dependent. Phi-correlation-coefficient analysis could not be completed across each test item because of the paradox previously mentioned, yet assessment of frequency percent agreement between subjective expert-rater scores and objective 3-dimensional instrumentation values revealed excellent agreement (range of 84–100% agreement) for ankle flexion at initial contact, knee-flexion range of motion, trunk flexion at maximum knee flexion, and foot position at initial contact for both external and internal rotation of lower leg. In addition, statistically significant correlations were found between rater and 3-dimensional instrumentation scores of knee-valgus range of motion. Moderate agreement was found between rater and 3-dimensional scores for trunk flexion at initial contact, stance width less than shoulder width, knee valgus at initial contact, and knee-valgus range of motion. These findings support a moderate to excellent level of validity of the LESS to accurately assess 3-dimensional kinematic motion pattern, yet specific item analysis is required to determine the most accurate items for subjective jump-landing assessment tools. This supports previous reports that have shown the LESS to be a valid instrument for subjectively assessing jump-landing movement patterns. Padua et al29 found good to excellent concurrent validity for distinguishing males from females and low- versus high-LESS-score participants with kinematic parameters. Our findings go one step farther by evaluating the specific item-rated scores with 3-dimensional motion-analysis system reduced dichotomous scores. Two of the most commonly deemed important risk factors for ACL injury, knee valgus and knee flexion, showed moderate to excellent agreement, and 1 had a significant correlation, which further supports the use of the LESS for prospective assessment of jump-landing motion. Future work needs to be done to reduce or eliminate the portions of the LESS that are not valid. For instance, sagittal-plane knee angle at initial contact is theorized to be very important for assessing ACL-injury risk, yet our findings show a poor agreement (21%) between rater and 3-dimensional analysis. Future studies should focus on assessing and improving the clinical assessment of knee-flexion angle at initial contact, in addition to the assessment of knee-valgus angle at initial contact. One possible validation approach would be to create a range of values for knee angle at initial contact agreement (eg, 25–35°) because a small kinematic difference of 0.25° could have resulted in poor validation agreement. Nonetheless, caution is warranted with this approach because small differences can still result in agreement problems between dichotomous variables that are true/false and kinematic values. In addition, the improvement and affordability of clinical tools such as video cameras and digital cameras may create improved clinical scores for these 2 important items of assessment. Another consideration for the future is the ability to visually observe poor or good landing patterns in a real-time setting (eg, clinician’s office) and observe the agreement that trained clinicians can have with the LESS and kinematic analyses.

It appears that the most difficult points to subjectively assess motion parameters during a high-speed jump-landing movement are whether the feet land symmetrically at initial contact and lateral trunk flexion at initial contact. Symmetrical landing is quite difficult to estimate visually because the time difference between both feet landing ranged only from 1 to 15 milliseconds, much faster than the human eye or 2-dimensional videographic data can capture. Caution should be taken against constantly encouraging individuals to land with both feet at the same time because it may be quite difficult to achieve. Even though the time-difference range is quite small (1–15 ms), this may be enough of a time window to allow for ACL injury to occur. Yasuda et al43 conducted one of the few studies of time to ACL failure by laterally loading a cadaveric ACL and medial collateral ligament model and reported that failures occurred up to a 70-millisecond time frame. This speculative time of up to 70 milliseconds until ACL failure has been used for noncontact ACL-injury occurrence, yet it has not been supported through noncontact methods.44 Nonetheless, the time it takes for ACL injury to occur is quite short, and the results of our study dispute the concept of being able to accurately assess foot-landing symmetry and, indeed, whether both feet can actually land at the same time. Future studies should evaluate foot-landing symmetry across different tasks to assess whether asymmetrical foot landings are only specific to the box-drop protocol used in this study. Future studies should also evaluate the normal time frame of asymmetry for foot landing and determine whether individuals who land outside of a certain time frame (eg, >15 ms) might be more at risk for ACL injury.

Researchers often disagree on the most important kinematic factors that potentially contribute to ACL injury.45–49 Yu et al50,51 proposed that sagittal-plane kinematics, most notably knee and hip flexion at or near initial contact, are the most important kinematic factors to consider for ACL injury. They also alluded to the fact that increased proximal anterior tibial shear force is perhaps the most important risk factor, yet the ability to estimate these forces using unsophisticated equipment for large prospective multicenter studies is quite limited. Certain institutions and private-practice facilities may be able to afford expensive sophisticated 3-dimensional motion-analysis systems, but besides being cost prohibitive to the general public, the amount of time to collect, reduce, and analyze 3-dimensional kinematic and kinetic motion patterns is prohibitive to integration throughout the at-risk ACL-injury population (eg, youth athletes and military soldiers). Contrary to the notion of sagittal-plane kinematics as the leading risk factor for ACL injury, McLean et al52 reported on simulation studies that indicate that sagittal-plane kinematics alone cannot injure the ACL. Thus, the opposing model to sagittalplane kinematics supports the viewpoint of knee valgus at or near initial contact as the major kinematic risk factor to assess during baseline examinations.24,52,53 Our results concerning trunk–hip flexion at initial contact, knee-flexion range of motion, knee valgus at initial contact, and knee-valgus range of motion showed moderate to excellent validity and high interrater reliability. Thus, we propose the use of simple 2-dimensional videotape assessment with the LESS instrument as both a cost-effective and an easy method to implement across sports medicine clinics as a tool for prospective analysis of jump-landing technique and associated injury risk.

The development of objective clinical assessment tools is of paramount importance for large-group baseline screening. Large prospective multicenter studies have been recommended for the continued study of ACL injury,54 yet the development of a reliable, valid, standardized, cost-effective, and time-efficient method to conduct ACL-related risk-factor assessment is needed. The LESS has been previously demonstrated to have good reliability, and the results of our study further support these findings. The additional knowledge that the reliability between an expert health care professional rater (J.O.), who codeveloped the LESS, and a novice health care professional rater was excellent further illustrates the ability of the LESS to be used across various levels of clinician experience. Unlike a previous study concerned with reliability for multicenter study development that required extensive weeks of study,54 twelve 2-hour practice sessions over 4 weeks for structural alignment education, our 1-hour training session of a novice health care professional rater resulting in similar scores as an expert rater provides further proof of the ease with which this subjective assessment tool can be reliably scored by a novice rater. We propose 3 main reasons for the good reliability and relative ease of training to properly score the LESS. First, the clearly defined time points of analysis easily allow for a standardized time of assessment. In reality, the LESS is scored during 3 time-instant snapshots of the jump-landing movement: initial foot contact, maximum knee-flexion angle, and maximum knee-valgus angle. Because using a standard video camera creates an approximately 30-frame/s shot of the jump-landing movement, choosing the proper frame for analysis remains relatively easy for the novice rater. Second, the movement descriptors are easy to follow because they are standardized in a dichotomous fashion as either yes or no. For example, the knee-valgus angle was either less than great toe or greater than great toe, thus creating a simple choice instead of requiring estimation of how far in angular degrees or whether the knee was only slightly in valgus. The visual decision-making process is reduced in this fashion and helps eliminate some of the estimation differences between raters to improve interrater reliability. Third, the novice rater had a background in movement assessment since being credentialed as an ATC. Thus, the novice rater may have been a novice in conducting LESS assessments but was trained and nationally certified in an area of work where evaluating faulty movement patterns in the physically active is quite common. Future studies should be conducted to assess the interrater reliability across other professions related to injury prevention and performance enhancement in the physically active (eg, certified strength and conditioning specialists, physical therapists, physicians).

The strong interrater reliability and moderate to excellent validity of the LESS instrument create the need for further evaluation of its efficacy in the prediction of ACL or other lower extremity (eg, ankle sprain or patellofemoral problems) injury risk factor. Studies need to be conducted concerning the specificity and sensitivity of the LESS for detecting injury risk of the lower extremity. In addition, investigations should focus on making the LESS process even easier by perhaps eliminating the videotape portion and conducting purely visual assessment of the errors, similar to the postural-control Balance Error Scoring System recommended as a clinical tool for mild head-injury assessment.55 One possibility is to evaluate the raters’ ability to score the LESS on a 0–2 scale (0 = good, 1 = average, 2 = poor) to allow clinicians to simply recommend that an individual proceed for further testing (ie, 2-dimensional video camera or 3-dimensional motion analysis) or recommend an intervention plan. This would further reduce the time for test administration and, we believe, increase the clinical use of this tool across various medical disciplines (eg, family medicine, orthopedics, athletic training, physical therapy) to help reduce manpower for ACL-injury prevention programs by targeting the individuals potentially most at risk for injury. Another consideration is to create a nationally networked system for simple jump-landing movement evaluation, following the YouTube concept, to allow individuals to post their own videotape of jump-landing movement for quick assessment from qualified health care personnel. This could also set the basis for using video for other types of movement assessments commonly done in the sporting world for performance enhancement, but it may also be beneficial for injury prevention or rehabilitation (eg, gait, throwing, lifting).

The LESS showed excellent expert versus novice rater reliability and moderate to excellent item validity in assessing a drop-jump landing task. Clinicians engaging in ACL-injury prevention programs or researchers initiating large prospective multicenter studies associated with ACL-injury prevention should consider using the LESS as a measure of dynamic jump-landing motion technique. The excellent interrater reliability with minimal training time required is advantageous to large settings and multiple clinicians. In addition, the minimal setup time and relatively quick posttest analysis time are ideally suited to venues with minimal resources or time. The low cost and ease of applicability to numerous settings make the LESS an ideally suited component of any baseline examination for developing prevention programs associated with reducing lower extremity injury (eg, ACL or ankle sprains).

References

  • 1.Agel J, Arendt EA, Bershadsky B. Anterior cruciate ligament injury in national collegiate athletic association basketball and soccer: a 13-year review. Am J Sports Med. 2005;33(4):524–530. doi: 10.1177/0363546504269937. [DOI] [PubMed] [Google Scholar]
  • 2.Arendt EA, Agel J, Dick R. Anterior cruciate ligament injury patterns among collegiate men and women. J Athl Train. 1999;34(2):86–92. [PMC free article] [PubMed] [Google Scholar]
  • 3.Bjordal JM, Arnly F, Hannestad B, Strand T. Epidemiology of anterior cruciate ligament injuries in soccer. Am J Sports Med. 1997;25(3):341–345. doi: 10.1177/036354659702500312. [DOI] [PubMed] [Google Scholar]
  • 4.Cox JS, Lenz HW. Women midshipmen in sports. Am J Sports Med. 1984;12(3):241–243. doi: 10.1177/036354658401200315. [DOI] [PubMed] [Google Scholar]
  • 5.Malone TR, Hardaker WT, Garrett WE, Feagin JA, Bassett FH. Relationship of gender to anterior cruciate ligament injuries in intercollegiate basketball players. J South Orthop Assoc. 1993;2(1):36–39. [Google Scholar]
  • 6.Mountcastle SB, Posner M, Kragh JF, Jr, Taylor DC. Gender differences in anterior cruciate ligament injury vary with activity: epidemiology of anterior cruciate ligament injuries in a young, athletic population. Am J Sports Med. 2007;35(10):1635–1642. doi: 10.1177/0363546507302917. [DOI] [PubMed] [Google Scholar]
  • 7.Parkkari J, Pasanen K, Mattila VM, Kannus P, Rimpela A. The risk for a cruciate ligament injury of the knee in adolescents and young adults: a population-based cohort study of 46 500 people with a 9 year follow-up. Br J Sports Med. 2008;42(6):422–426. doi: 10.1136/bjsm.2008.046185. [DOI] [PubMed] [Google Scholar]
  • 8.Gomez E, DeLee JC, Farney WC. Incidence of injury in Texas girls’ high school basketball. Am J Sports Med. 1996;24(5):684–687. doi: 10.1177/036354659602400521. [DOI] [PubMed] [Google Scholar]
  • 9.Messina DF, Farney WC, DeLee JC. The incidence of injury in Texas high school basketball. a prospective study among male and female athletes. Am J Sports Med. 1999;27(3):294–299. doi: 10.1177/03635465990270030401. [DOI] [PubMed] [Google Scholar]
  • 10.Gwinn DE, Wilckens JH, McDevitt ER, Ross G, Kao TC. The relative incidence of anterior cruciate ligament injury in men and women at the United States Naval Academy. Am J Sports Med. 2000;28(1):98–102. doi: 10.1177/03635465000280012901. [DOI] [PubMed] [Google Scholar]
  • 11.Boden BP, Dean GS, Feagin JA, Jr, Garrett WE., Jr Mechanisms of anterior cruciate ligament injury. Orthopedics. 2000;23(6):573–578. doi: 10.3928/0147-7447-20000601-15. [DOI] [PubMed] [Google Scholar]
  • 12.Engstrom B, Johansson C, Tornkvist H. Soccer injuries among elite female players. Am J Sports Med. 1991;19(4):372–375. doi: 10.1177/036354659101900408. [DOI] [PubMed] [Google Scholar]
  • 13.Griffin LY, Albohm MJ, Arendt EA, et al. Understanding and preventing noncontact anterior cruciate ligament injuries: a review of the Hunt Valley II meeting, January 2005. Am J Sports Med. 2006;34(9):1512–1532. doi: 10.1177/0363546506286866. [DOI] [PubMed] [Google Scholar]
  • 14.Shimokochi Y, Shultz SJ. Mechanisms of noncontact anterior cruciate ligament injury. J Athl Train. 2008;43(4):396–408. doi: 10.4085/1062-6050-43.4.396. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 15.Yu B, Garrett WE. Mechanisms of non-contact ACL injuries. Br J Sports Med. 2007;41(Suppl 1):i47–i51. doi: 10.1136/bjsm.2007.037192. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 16.Hewett TE, Lindenfeld TN, Riccobene JV, Noyes FR. The effect of neuromuscular training on the incidence of knee injury in female athletes. a prospective study. Am J Sports Med. 1999;27(6):699–706. doi: 10.1177/03635465990270060301. [DOI] [PubMed] [Google Scholar]
  • 17.Hewett TE, Stroupe AL, Nance TA, Noyes FR. Plyometric training in female athletes. decreased impact forces and increased hamstring torques. Am J Sports Med. 1996;24(6):765–773. doi: 10.1177/036354659602400611. [DOI] [PubMed] [Google Scholar]
  • 18.Mandelbaum BR, Silvers HJ, Watanabe DS, et al. Effectiveness of a neuromuscular and proprioceptive training program in preventing anterior cruciate ligament injuries in female athletes: 2-year follow-up. Am J Sports Med. 2005;33(7):1003–1010. doi: 10.1177/0363546504272261. [DOI] [PubMed] [Google Scholar]
  • 19.Holm I, Fosdahl MA, Friis A, Risberg MA, Myklebust G, Steen H. Effect of neuromuscular training on proprioception, balance, muscle strength, and lower limb function in female team handball players. Clin J Sport Med. 2004;14(2):88–94. doi: 10.1097/00042752-200403000-00006. [DOI] [PubMed] [Google Scholar]
  • 20.Myklebust G, Engebretsen L, Braekken IH, Skjolberg A, Olsen OE, Bahr R. Prevention of anterior cruciate ligament injuries in female team handball players: a prospective intervention study over three seasons. Clin J Sport Med. 2003;13(2):71–78. doi: 10.1097/00042752-200303000-00002. [DOI] [PubMed] [Google Scholar]
  • 21.Myklebust G, Maehlum S, Holm I, Bahr R. A prospective cohort study of anterior cruciate ligament injuries in elite Norwegian team handball. Scand J Med Sci Sports. 1998;8(3):149–153. doi: 10.1111/j.1600-0838.1998.tb00185.x. [DOI] [PubMed] [Google Scholar]
  • 22.Renstrom P, Ljungqvist A, Arendt E, et al. Non-contact ACL injuries in female athletes: an International Olympic Committee current concepts statement. Br J Sports Med. 2008;42(6):394–412. doi: 10.1136/bjsm.2008.048934. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 23.Plisky PJ, Rauh MJ, Kaminski TW, Underwood FB. Star Excursion Balance Test as a predictor of lower extremity injury in high school basketball players. J Orthop Sports Phys Ther. 2006;36(12):911–919. doi: 10.2519/jospt.2006.2244. [DOI] [PubMed] [Google Scholar]
  • 24.Ford KR, Myer GD, Hewett TE. Valgus knee motion during landing in high school female and male basketball players. Med Sci Sports Exerc. 2003;35(10):1745–1750. doi: 10.1249/01.MSS.0000089346.85744.D9. [DOI] [PubMed] [Google Scholar]
  • 25.McLean SG, Walker K, Ford KR, Myer GD, Hewett TE, van den Bogert AJ. Evaluation of a two dimensional analysis method as a screening and evaluation tool for anterior cruciate ligament injury. Br J Sports Med. 2005;39(6):355–362. doi: 10.1136/bjsm.2005.018598. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 26.Noyes FR, Barber-Westin SD, Fleckenstein C, Walsh C, West J. The drop-jump screening test: difference in lower limb control by gender and effect of neuromuscular training in female athletes. Am J Sports Med. 2005;33(2):197–207. doi: 10.1177/0363546504266484. [DOI] [PubMed] [Google Scholar]
  • 27.Boling M, Thigpen CA, Padua DA, Marshall SW. Item specific reliability analyses of the landing error scoring system (LESS) Med Sci Sports Exerc. 2005;37:S124. [Google Scholar]
  • 28.Padua D, Marshall SW, Beutler AI, DeMaio M, Onate JA, Guskiewicz KM. Sex comparison of jump landing kinematics and technique. Med Sci Sports Exerc. 2004;36(5):S348. [Google Scholar]
  • 29.Padua DA, Marshall SW, Onate JA, et al. Reliability and validity of the landing error scoring system: implications on ACL injury risk assessment. J Athl Train. 2004;39:S110–S111. [Google Scholar]
  • 30.Onate JA, Guskiewicz KM, Marshall SW, Giuliani C, Yu B, Garrett WE. Instruction of jump-landing technique using videotape feedback: altering lower extremity motion patterns. Am J Sports Med. 2005;33(6):831–842. doi: 10.1177/0363546504271499. [DOI] [PubMed] [Google Scholar]
  • 31.Hewett TE, Myer GD, Ford KR, et al. Biomechanical measures of neuromuscular control and valgus loading of the knee predict anterior cruciate ligament injury risk in female athletes: a prospective study. Am J Sports Med. 2005;33(4):492–501. doi: 10.1177/0363546504269591. [DOI] [PubMed] [Google Scholar]
  • 32.Kadaba MP, Ramakrishnan HK, Wootten ME. Measurement of lower extremity kinematics during level walking. J Orthop Res. 1990;8(3):383–392. doi: 10.1002/jor.1100080310. [DOI] [PubMed] [Google Scholar]
  • 33.Kadaba MP, Ramakrishnan HK, Wootten ME, Gainey J, Gorton G, Cochran GV. Repeatability of kinematic, kinetic, and electromyographic data in normal adult gait. J Orthop Res. 1989;7(6):849–860. doi: 10.1002/jor.1100070611. [DOI] [PubMed] [Google Scholar]
  • 34.Grood ES, Suntay WJ. A joint coordinate system for the clinical description of three-dimensional motions: application to the knee. J Biomech Eng. 1983;105(2):136–144. doi: 10.1115/1.3138397. [DOI] [PubMed] [Google Scholar]
  • 35.Lantz CA, Nebenzahl E. Behavior and interpretation of the kappa statistic: resolution of the two paradoxes. J Clin Epidemiol. 1996;49(4):431–434. doi: 10.1016/0895-4356(95)00571-4. [DOI] [PubMed] [Google Scholar]
  • 36.Viera AJ, Garrett JM. Understanding interobserver agreement: the kappa statistic. Fam Med. 2005;37(5):360–363. [PubMed] [Google Scholar]
  • 37.Cicchetti DV, Feinstein AR. High agreement but low kappa: II. resolving the paradoxes. J Clin Epidemiol. 1990;43(6):551–558. doi: 10.1016/0895-4356(90)90159-m. [DOI] [PubMed] [Google Scholar]
  • 38.Feinstein AR, Cicchetti DV. High agreement but low kappa: I. the problems of two paradoxes. J Clin Epidemiol. 1990;43(6):543–549. doi: 10.1016/0895-4356(90)90158-l. [DOI] [PubMed] [Google Scholar]
  • 39.Kohout FJ. Statistics for Social Scientists. New York: John Wiley & Sons; 1974. [Google Scholar]
  • 40.Hays WL. Statistics for the Social Sciences. New York: Holt, Rinehart & Winston; 1973. [Google Scholar]
  • 41.Lewin LM, Wakefield JA. Percentage agreement and phi: a conversion table. J Appl Behav Anal. 1979;12(2):299–301. doi: 10.1901/jaba.1979.12-299. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 42.Birkimer JC, Brown JH. Back to basics: percentage agreement measures are adequate, but there are easier ways. J Appl Behav Anal. 1979;12(4):535–543. doi: 10.1901/jaba.1979.12-535. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 43.Yasuda K, Erickson AR, Beynnon BD, Johnson RJ, Pope MH. Dynamic elongation behavior in the medial collateral and anterior cruciate ligaments during lateral impact loading. J Orthop Res. 1993;11(2):190–198. doi: 10.1002/jor.1100110206. [DOI] [PubMed] [Google Scholar]
  • 44.Ireland ML. The female ACL: why is it more prone to injury? Orthop Clin North Am. 2002;33(4):637–651. doi: 10.1016/s0030-5898(02)00028-7. [DOI] [PubMed] [Google Scholar]
  • 45.Blackburn JT, Padua DA. Influence of trunk flexion on hip and knee joint kinematics during a controlled drop landing. Clin Biomech (Bristol, Avon) 2008;23:313–319. doi: 10.1016/j.clinbiomech.2007.10.003. [DOI] [PubMed] [Google Scholar]
  • 46.Chappell JD, Limpisvasti O. Effect of a neuromuscular training program on the kinetics and kinematics of jumping tasks. Am J Sports Med. 2008;36(6):1081–1086. doi: 10.1177/0363546508314425. [DOI] [PubMed] [Google Scholar]
  • 47.Cortes N, Onate JA, Abrantes J, Gagen L, Dowling E, Van Lunen B. Effects of gender and foot-landing techniques on lower extremity kinematics during drop-jump landings. J Appl Biomech. 2007;23(4):289–299. doi: 10.1123/jab.23.4.289. [DOI] [PubMed] [Google Scholar]
  • 48.McLean SG, Felin RE, Suedekum N, Calabrese G, Passerallo A, Joy S. Impact of fatigue on gender-based high-risk landing strategies. Med Sci Sports Exerc. 2007;39(3):502–514. doi: 10.1249/mss.0b013e3180d47f0. [DOI] [PubMed] [Google Scholar]
  • 49.Pollard CD, Davis IM, Hamill J. Influence of gender on hip and knee mechanics during a randomly cued cutting maneuver. Clin Biomech (Bristol, Avon) 2004;19(10):1022–1031. doi: 10.1016/j.clinbiomech.2004.07.007. [DOI] [PubMed] [Google Scholar]
  • 50.Yu B, McClure SB, Onate JA, Guskiewicz KM, Kirdendall DT. Age and gender effects on landing kinematics of youth soccer players during a stop-jump task. Med Sci Sports Exerc. 2004;36(5):S151–S152. doi: 10.1177/0363546504273049. [DOI] [PubMed] [Google Scholar]
  • 51.Yu B, McClure SB, Onate JA, Guskiewicz KM, Kirkendall DT, Garrett WE. Age and gender effects on lower extremity kinematics of youth soccer players in a stop-jump task. Am J Sports Med. 2005;33(9):1356–1364. doi: 10.1177/0363546504273049. [DOI] [PubMed] [Google Scholar]
  • 52.McLean SG, Huang X, Su A, Van Den Bogert AJ. Sagittal plane biomechanics cannot injure the ACL during sidestep cutting. Clin Biomech (Bristol, Avon) 2004;19(8):828–838. doi: 10.1016/j.clinbiomech.2004.06.006. [DOI] [PubMed] [Google Scholar]
  • 53.Hewett TE, Myer G, Gregory D, et al. Neuromuscular control and valgus loading of the knee predict ACL injury risk in female athletes. Med Sci Sports Exerc. 2004;36(5):S287. [Google Scholar]
  • 54.Shultz SJ, Nguyen AD, Windley TC, Kulas AS, Botic TL, Beynnon BD. Intratester and intertester reliability of clinical measures of lower extremity anatomic characteristics: implications for multicenter studies. Clin J Sport Med. 2006;16(2):155–161. doi: 10.1097/00042752-200603000-00012. [DOI] [PubMed] [Google Scholar]
  • 55.Guskiewicz KM, Bruce SL, Cantu RC, et al. Research based recommendations on management of sport related concussion: summary of the National Athletic Trainers’ Association position statement. Br J Sports Med. 2006;40(1):6–10. doi: 10.1136/bjsm.2005.021683. [DOI] [PMC free article] [PubMed] [Google Scholar]

RESOURCES