Is the Landing Error Scoring System Reliable and Valid? A Systematic Review

Abstract

Context:

The Landing Error Scoring System (LESS) is a clinical tool often used in research and practice to identify athletes presenting high injury-risk biomechanical patterns during a jump-landing task.

Objective:

To systematically review the literature addressing the psychometric properties of the LESS.

Data Sources:

Three electronic databases (PubMed, Web of Science, and Scopus) were searched on March 28, 2018, using the term “Landing Error Scoring System.”

Study Selection:

All studies using the LESS as main outcome measure and addressing its reliability, validity against motion capture system, and predictive validity were included. Original English-language studies published in peer-reviewed journals were reviewed. Studies using modified versions of the LESS were excluded.

Study Design:

Systematic literature review.

Level of Evidence:

Level 4.

Data Extraction:

Study design, population, LESS testing procedures, LESS scores, statistical analysis, and main results were extracted from studies using a standardized template.

Results:

Ten studies met inclusion criteria and were appraised using Newcastle-Ottawa Quality Assessment Scale adapted for cross-sectional studies. The overall LESS score demonstrated good-to-excellent intrarater (intraclass correlation coefficient [ICC], 0.82-0.99), interrater (ICC, 0.83-0.92), and intersession reliability (ICC, 0.81). The validity of the overall LESS score against 3-dimensional jump-landing biomechanics was good when individuals were divided into 4 quartiles based on LESS scores. The validity of individual LESS items versus 3-dimensional motion capture data was moderate-to-excellent for most of the items addressing key risk factors for anterior cruciate ligament (ACL) injury. The predictive value of the LESS for ACL and other noncontact lower-extremity injuries remains uncertain based on the current scientific evidence.

Conclusion:

The LESS is a reliable screening tool. However, further work is needed to improve the LESS validity against motion capture system and confirm its predictive validity for ACL and other noncontact lower-extremity injuries.

Increased participation in physical activities increases the likelihood of sustaining sport-related injuries.⁵⁴ Noncontact mechanisms explain approximately 18% of injuries in game situations and 37% of injuries in practice or training situations.²² Neuromuscular and biomechanical factors are key in the prevention of noncontact injuries, as modifiable through targeted training interventions.^17,42

The Landing Error Scoring System (LESS; Appendix Table A1, available in the online version of this article) is a clinical assessment tool⁴⁰ often used in research to identify individuals at high risk of sustaining noncontact injuries and to quantify changes in neuromuscular and biomechanical performance subsequent to intervention across sports^14,15,18 and performance levels.^27,49 The LESS has also been used in participants with a history of injury^24,30 and after anterior cruciate ligament (ACL) reconstruction^2,20 to quantify residual functional impairments and outcomes from rehabilitation.

It is essential that testing methods provide outcomes that are reproducible and valid so that changes in scores reflect meaningful changes in function of individuals and identify individuals with differing abilities. In the LESS, lower scores should reflect a reduction in injury risk and high injury-risk movement patterns. The LESS was previously addressed in critically appraised topics^32,44,47 and literature reviews^4,7,19,34,48; however, no systematic review has critically appraised and summarized research on its psychometrics properties (reliability and validity). Such a systematic review is warranted to ensure the justified use of the LESS in large-scale screening initiatives, monitoring changes in risk factors, establishing the effects of injury prevention programs, and identifying athletes at high risk of injuries. Therefore, our aim was to systematically review and critically appraise studies addressing the psychometric properties of the LESS.

Methods

Protocol and Registration

Literature review methods and inclusion criteria were specified in advance and registered on PROSPERO (CRD42018107210).

Eligibility Criteria

Studies that used the LESS as a main outcome measure and examined its reliability, validity against motion capture, and predictive validity were included for review regardless of participant, intervention, or study design characteristics. Only original research published in English in peer-reviewed (abstract available) journals were considered. Letters to the editor, symposium publications, conference abstracts, books, expert opinions, critically appraised topics, and literature reviews were excluded. Studies using modified versions of the original LESS protocol (eg, iLESS, real-time LESS, and automated quantification of the LESS) were excluded.^23,33,50

Information Sources and Search Strategy

Three electronic databases (Figure 1) were searched using the keywords “Landing Error Scoring System” on March 28, 2018. Psychometric property terms were not included in the search strategy to favor an all-inclusive approach and avoid missing studies that addressed the reliability or validity of the LESS as part of their methodology without its being a primary aim. In addition, a hand search of the reference lists of all included studies was conducted.

Figure 1.

Flowchart of the search strategy and study selection process.

Study Selection

The electronic search was conducted by 1 reviewer. Duplicate hits were removed first. Titles, abstracts, and full texts were screened sequentially for inclusion and exclusion criteria. In case of uncertainty regarding inclusion, a second reviewer was consulted.

Data Collection Process

Data concerning study design, population (number, sex, age, and activity level), LESS testing procedures, LESS scores, statistical analysis, and main results were extracted from studies using a standardized template by 1 reviewer, with the completeness of extraction verified by a second reviewer. The study design was reported according to Parab and Bhalerao.⁴¹ Studies were categorized into the following subcategories: reliability, validity against motion capture, and LESS predictive value for injury incidence. Four authors were contacted by email to request additional information regarding 8 of the included studies. Three authors responded, with 2 authors providing additional data for 2 studies.

Risk of Bias in Individual Studies

Two reviewers independently assessed the methodological quality of included studies (n = 10) using the Newcastle-Ottawa Quality Assessment Scale (NOS) adapted for cross-sectional studies.³⁶ Potentially identifiable information from studies was removed prior to quality appraisal to reduce assessment bias. The 2 reviewers achieved a consensus rating on all quality scores without the need for a third reviewer.

The NOS uses a “star system,” wherein more stars indicates a superior methodological quality. The NOS adapted for cross-sectional studies awards a maximum of 10 stars: 5 for selection (representativeness of the sample, sample size, nonrespondents, and ascertainment of the exposure), 2 for comparability, and 3 for outcome (assessment of outcome and statistical test). Reviewers agreed that for the statistical test item, the highest star rating would be allocated for the reporting of confidence intervals, quartiles, or limits of agreement. The methodological quality of studies was divided into 3 groups based on the number of stars awarded: weak (0-3 stars), moderate (4-6 stars), and strong (7-10 stars). Given that the reliability for NOS adapted for cross-sectional studies has not yet been determined, we assessed intra- and interrater reliability of our scores. Based on intraclass correlation coefficient (ICC) and confidence interval values, intrarater (ICC, 0.98; 95% CI, 0.94-0.99) and interrater (ICC, 0.94; 95% CI, 0.80-0.98) reliability of scores was excellent.³⁷ Finally, the level of evidence for each study was determined using the Oxford Centre for Evidence-Based Medicine 2011 table.

Summary Measures

Descriptive statistics were computed using Microsoft Office Excel 2016 and expressed in terms of means and SDs, minimum to maximum ranges (min to max), percentages (%), and counts (n). Weighted mean ± SD values based on sample size were computed to describe age and LESS scores of participants across studies. ICC values were interpreted according to Koo and Li²⁶ using thresholds of 0.50, 0.75, and 0.90 to indicate moderate, good, and excellent reliability, respectively, with ICCs <0.50 indicating poor reliability. Standard error of measurement and minimal detectable change values were calculated for reliability studies when possible. Percentage of agreement between individual LESS items or categories based on LESS score (excellent, good, moderate, and poor) and 3-dimensional (3D) motion capture was reported for construct validity.

Results

Study Selection

Figure 1 illustrates the search strategy and study selection process. A total of 10 studies met inclusion requirements and were reviewed.

Study Characteristics

Since the majority of studies did not report descriptive characteristics of the sample used for reliability assessment, study characteristics were calculated from all participants tested with the LESS in included studies (Tables 1 and 2). The study sample size ranged from 13 participants⁴⁹ to 2691 participants.⁴⁰ A total of 3835 participants were represented across the 10 studies. All studies tested physically active populations. Sex distribution was described in all studies, totaling 2102 males (55%) and 1733 (45%) females. The mean age was reported in all but one⁴⁰ of the 10 studies. The weighted mean age was 15.7 ± 2.0 years (minimum mean age 13.9 years³⁹ and maximum mean age 28.5 years⁹). Overall LESS scores were reported in all but one³⁸ of the 10 studies. Note that for interventional studies, LESS score before intervention was included in calculation. The calculated weighted mean for overall LESS score was 5.2 ± 1.7 errors (minimum mean LESS score of 4.4 errors³⁹ and maximum mean LESS score 6.5 errors¹). Only 3 studies reported the range of LESS scores,^9,18,24 with the smallest and largest values being 0.0 errors^9,18 and 13.3 errors.²⁴

Table 1.

Summary of studies reporting reliability of the Landing Error Scoring System (LESS)

Study (Year)	Quality a (Category)	Sample Size [Study Sample Size]	Population	Age, y b	LESS Score b	Intrarater	Interrater	Intersession
Padua et al (2009)40	6 stars (moderate)	50 (M: 25, F: 25) [2691]	US military freshmen	Not specified	4.9 ± 1.7	ICC = 0.91 SEM = 0.42 MDC = 1.16	ICC = 0.84 SEM = 0.71 MDC = 1.97	×
Onate et al (2010)38	4 stars (moderate)	19 (M: 0, F: 19) [19]	Division I college soccer players	19.6 ± 0.8	Not specified	×	ICC = 0.84	×
Smith et al (2012)53	8 stars (strong)	10 [92]	College and high school athletes	18.3 ± 2.0	5.1 ± 1.95 c	ICC = 0.97 SEM = 0.52 MDC = 1.44	ICC = 0.92	×
Beese et al (2015)1	8 stars (strong)	40 (M: 0, F: 40) [40]	Soccer players and multisport athletes	15.2 ± 1.2	6.5 ±1.9	ICC = 0.91 SEM = 0.48 MDC = 1.33	×	×
Wesley et al (2015)55	7 stars (moderate)	5 [36]	Athletes	19.3 ± 1.15 c	5.0 ± 2.0	ICC = 0.99 SEM = 0.19 MDC = 0.53	×	×
James et al (2016)24	6 stars (moderate)	34 (M: 19, F: 15) [34]	NCAA Division I soccer players	19.6 ± 1.2	5.4	ICC = 0.95	×	×
Fox et al (2017)18	6 stars (moderate)	10 (M: 0, F: 10) [32]	Subelite netball players	23.2 ± 3.1	4.9 ± 2.3	Cohen κ = 0.93	Cohen κ = 0.75	×
Scarneo et al (2017)49	9 stars (strong)	13 (M: 2, F: 11) [13]	Recreationally active population	21 ± 2	6.2 ± 1.7	×	×	ICC = 0.81 SEM = 0.81 MDC = 2.25
Dar et al (2019)9	8 stars (strong)	10 [49]	Recreationally active population	28.5 ± 5.6	4.8 ± 2.3	ICC = 0.82 (CI = 0.571-0.947)	ICC = 0.83 (CI = 0.451-0.956)	×

F, females; ICC, intraclass correlation coefficient; M, males; MDC, minimal detectable change; NCAA, National Collegiate Athletic Association.

^aMethodology quality assessment score based on Newcastle-Ottawa Quality Assessment Scale adapted for cross-sectional studies, weak (0-3 stars), moderate (4-6 stars), and strong (7-10 stars).³⁶

^bMean ± SD values for age (years) and LESS scores (errors) are shown for all the participants in the study, not the subsample assessed for reliability.

^cWeighted mean and weighted SD.

Table 2.

Summary of studies exploring Landing Error Scoring System (LESS) predictive value for injury incidence

Study (Year)	Quality a (Category)	Study Sample Size	Type of Lower-Extremity Injury	Population	Age, y b	LESS Score Uninjured b	LESS Score Injured b	Significance Value
Smith et al (2012)53	8 stars (strong)	92 (U: 64, I: 28)	Noncontact ACL	College and high school athletes	18.3 ± 2.0	5.0 ± 2.0	5.5 ± 1.9	P = 0.320
Padua et al (2015)39	8 stars (strong)	829 (U: 822, I: 7)	Noncontact ACL	Elite youth soccer players	13.9 ± 1.8	4.4 ± 1.7	6.2 ± 1.8	P = 0.005
James et al (2016)24	6 stars (moderate)	34 (U: 11, I: 10)	Any injury that causes a player to miss 1 or more practices	NCAA Division I soccer players	19.6 ± 1.2	5.8 ± 3.4	5.5 ± 2.5	P = 0.830

ACL, anterior cruciate ligament; I, injured; NCAA, National Collegiate Athletic Association; U, uninjured.

^aMethodology quality assessment score based on Newcastle-Ottawa Quality Assessment Scale adapted for cross-sectional studies, weak (0-3 stars), moderate (4-6 stars), and strong (7-10 stars).³⁶

^bMean ± SD values for age (years) and LESS (errors).

Risk of Bias Within Studies

Quality scores, levels of evidence, and study designs are presented in Appendix Table A2 (available online). Overall, studies were of moderate quality based on the NOS adapted for cross-sectional studies (10-point scale: mean 7.0 ± 1.5 points; range, 4-9). The level of evidence ranged between 2 and 4 based on the Oxford Centre for Evidence-Based Medicine 2011 table.

Psychometric Properties of the LESS

Reliability

Reliability values for LESS were reported in 9 studies^{1,9,18,24,38,40,49,53,55} and derived from a total of 191 participants (Table 1). Both intra- and interrater reliability of the total LESS score was good to excellent based on ICCs (0.82-0.99 and 0.83-0.92, respectively). In addition to data reported in Table 1, Onate et al³⁸ reported percentage agreement and kappa statistics between novice and expert raters for all individual LESS items, except for hip flexion at initial contact (IC) and hip flexion at maximal knee flexion, which were not clearly addressed. There was no significant agreement between raters for knee and trunk flexion at IC, and moderate agreement (65% agreement; κ = 0.533; P = 0.011) between raters for overall impression. For the remaining items, agreement between novice and expert raters ranged from 80% to 100% (κ = 0.459-1.0; P < 0.015). Only 1 study assessed the intersession reliability of the overall LESS score,⁴⁹ which was good (ICC, 0.81).

Validity Against Motion Capture System

Two studies reported the validity of the LESS against “gold standard” 3D motion capture.^38,40 Padua et al⁴⁰ found that numerous lower-extremity kinematics and kinetics jump-landing measures significantly differed between participants subdivided into 4 quartiles based on excellent (LESS ≤ 4), good (4 < LESS ≤ 5), moderate (5 < LESS ≤ 6), and poor (LESS > 6) LESS performances. Accordingly, the authors concluded the LESS is a valid clinical assessment tool for detecting poor jump-landing biomechanics.⁴⁰ Onate et al³⁸ dichotomized 3D motion capture data into 0 and 1 to match LESS scoring and investigated the association between LESS scores and 3D motion data using phi correlations and percentage agreements. The last 2 items (joint displacement and overall impression) were not included because of the subjective nature of scoring, and the authors did not report any results for the hip flexion at IC and hip flexion at maximal knee flexion. Poor agreement (10% to 42%) was found for knee flexion at IC, lateral trunk flexion at IC, and symmetric foot contact at IC. Moderate agreement (68% to 74%) was found for trunk flexion at IC, knee valgus at IC, stance width (narrow), and knee valgus displacement. The remaining LESS items showed excellent agreement (84% to 100%). Hence, Onate et al concluded that validity of the LESS was item dependent.

LESS Predictive Value for Injury Incidence

Two studies with total sample size of 921 participants reported the predictive value of the LESS for noncontact ACL injury incidence (Table 2).^39,53 The longitudinal cohort study³⁹ concluded that the LESS has a screening potential for noncontact ACL injury, identifying 5 errors as an optimal cutoff point, yielding a sensitivity of 86% (95% CI, 42% to 99%) and specificity of 64% (95% CI, 62% to 67%). On the other hand, the case-control study⁵³ did not find any significant relationship between LESS score and risk of suffering ACL injury, either for all participants as a combined group (P = 0.32) or for subgroups of male (P = 0.67), female (P = 0.16), high school (P = 0.37), or college (P = 0.66) participants. Likewise, there was no significant relationship between LESS as categorical variable (ie, poor, moderate, good, and excellent performances) and noncontact ACL injury incidence (P = 0.35).

One study²⁴ compared preseason LESS scores between soccer players with no previous lower-extremity injury who suffered an injury in the subsequent season and those who remained uninjured (Table 2). Lower-extremity injury was defined as any injury that caused players to miss 1 or more practices or games. There were no statistically significant differences (P = 0.83) in preseason LESS scores between those who sustained a lower-extremity injury and those who remained uninjured.

Discussion

Reliability

Findings from 9 studies of relatively strong methodological quality indicated good-to-excellent intrarater, interrater, and intersession reliabilities. Noteworthy is that reliability of the overall LESS score was derived from uninjured military freshmen and sportspeople with a mean age between 15 and 28 years. Although LESS has been used in younger^14,46 and injured^20,29 individuals; generalizability of reliability findings across these population groups is not confirmed. Furthermore, the findings showed that some individual items in LESS scoring are less reliable than others.³⁸ More specifically, no significant agreement between raters was found for knee and trunk flexion at IC and moderate agreement for overall impression. Given that knee flexion during landing and stiff landing with less ankle, knee, hip, and trunk flexion are key risk factors for knee injury incidence,^13,31,43 the lack of agreement in these LESS metrics is of concern. In the LESS, a knee flexion at IC of less than 30° defines an error (Appendix Table A1, available online). Accurately determining an angular measure from visual observations is challenging,^16,25 which can explain the lower reliability of the knee flexion angle at IC item. The overall impression item is subjective in nature, and the representation of an excellent, average, and poor landing may differ between raters. One solution could be to use a video analysis software to objectively assess angle-related items during LESS scoring to decrease the subjective nature of items. Although outcomes from such assessments might be more accurate, the scoring process would take more time. To decrease scoring time and improve consistency of LESS ratings, the automated quantification of the LESS using markerless motion capture technology has been developed recently.^9,33 The markerless method is as reliable as expert LESS raters.³³ The time and cost saving benefits of the markerless method, however, need to be weighed against the additional hardware and software expenditures.

Validity Against Motion Capture System

Some of the most frequently addressed biomechanical risk factors linked with ACL injury include increased knee valgus angle and moment^{8,11,12,21,28,35} and increased ground-reaction force resulting from stiff landing.^{5,6,10,13,45,51,52} Padua et al⁴⁰ associated poor LESS scores with decreased peak knee and hip flexion angles and increased peak knee valgus angles and moments. Onate et al³⁸ found that the validity of LESS items was strictly item dependent. Again, one of the main concerns is that angles are difficult to estimate visually. A small kinematic difference (eg, knee angle 29° = error present; knee angle 30° = error absent) in performance can result in poor agreement between clinical LESS scoring and motion capture scoring. Onate et al suggested creating a range of acceptable angular values (eg, knee angle at IC between 25° to 30°) to improve scoring validity, although no further studies were undertaken.

The validity of the LESS was strictly item dependent,³⁸ which might mean modification of the original LESS scoring template. The fact that most of the key factors for ACL injury had moderate and excellent agreement nonetheless supports the LESS as a valid screening tool for assessing ACL injury risk jump-landing biomechanics.

LESS Predictive Value for Injury Incidence

Two studies of strong methodological quality explored the predictive value of the LESS for ACL injury incidence and reported equivocal results.^39,53 Padua et al³⁹ employed a prospective study design. Of the 829 elite young soccer players with a mean age of 13.9 ± 1.8 years, 7 participants suffered a noncontact ACL injury during the 2.5-year observation period. Based on the data, 5 errors were identified as an optimal cut point for distinguishing between athletes with low and high risk of ACL injury. Sample size calculations and post hoc power analyses were not reported.

The study conducted by Smith et al⁵³ assessed a population of college and high school athletes from a range of sports with a mean age of 18.3 ± 2.0 years. The study was designed as a prospective cohort with 5047 screenings within a 3-year period. Smith et al did not find any significant relationship between LESS scores and risk of ACL injury.

The lack of agreement in the predictive value of the LESS between studies can be explained by differences in sampled populations in terms of age, main sporting events, and exclusion criteria (Smith et al⁵³ excluded athletes with a history of ACL injury, whereas Padua et al³⁹ did not). Furthermore, the lack of statistical power in both studies^39,53 is a limiting factor to the generalization of results.

One moderate quality study²⁴ recorded preseason LESS scores of soccer players without lower-extremity injury history and did not find any significant difference between participants who sustained an injury during the subsequent season and those who remained uninjured. The study is deemed to have several limitations that could explain the lack of association between LESS and lower-extremity injuries, including the relatively small sample size (n = 34), short follow-up period, and lack of differentiation between contact and noncontact mechanisms of injury. Fundamentally, the LESS was developed as an injury risk screening tool that identifies poor biomechanical control, which is a risk factor for noncontact, rather than contact, lower-extremity injuries. Thus, the relevance of the findings from James et al²⁴ is questioned.

Based on the current scientific evidence, the predictive value of the LESS for noncontact lower-extremity injury incidence remains uncertain. Studies with a greater statistical power are needed to affirm the relationship between LESS scores and noncontact ACL injury incidence, as well as incidence of other noncontact lower-extremity injuries.

Limitations

This systematic review combines data across studies to detail the reliability, validity against motion capture, and predictive validity of the LESS. The main limitations of this review were the varied methodological quality of studies and the inability to perform meta-analysis of data due to lack of detail. Another important limitation is that populations included, testing protocols, and calculations of total LESS score varied across studies. Population characteristics—for example, age, sex, and activity level—can significantly influence LESS scores,^3,27,53 and therefore the results of this literature review are most relevant to uninjured military freshmen and sportspeople with a mean age between 14 and 28 years.

Conclusion

The current evidence indicates that the overall LESS score has good-to-excellent intrarater, interrater, and intersession reliabilities. The validity of the individual LESS items against 3D motion capture is item dependent; however, the validity of LESS items addressing key knee-injury risk factors is moderate to excellent. The LESS predictive value for noncontact ACL injury and other noncontact lower-extremity injury incidence cannot be ascertained based on the current scientific evidence. Larger scale multicenter studies are needed to confirm the association between LESS scores and ACL injury incidence, as well as other noncontact lower-extremity injury incidence.