The Knee Injury and Osteoarthritis Outcome Score Does Not Have Adequate Structural Validity for Use With Young, Active Patients With ACL Tears

Abstract

Background

The Knee Injury and Osteoarthritis Outcome Score (KOOS) is well known and commonly used to assess young, active patients with ACL injuries. However, this application of the outcome measure has been called into question. There is currently no evidence supporting the structural validity of the KOOS for this patient population. Structural validity refers to whether a questionnaire meant to provide scores on different subscales behaves as intended in the populations of interest. Structural validity should be assessed for all questionnaire measures with multiple items or subscales.

Questions/purposes

Does the KOOS demonstrate adequate structural validity in young, active patients with ACL tears, when evaluated using (1) exploratory and (2) confirmatory factor analyses?

Methods

Between January 2014 and March 2017, 1033 patients were screened for eligibility in the Stability 1 randomized controlled trial from nine centers in Canada and Europe. Patients were eligible if they had an ACL deficient knee, were between 14 and 25 years old, and were thought to be at higher risk of reinjury based on the presence of two or more of the following factors: participation in pivoting sports, presence of a Grade 2 pivot shift or greater, generalized ligamentous laxity (Beighton score of 4 or greater), or genu recurvatum greater than 10°. Based on this criteria, 367 patients were ineligible and another 48 declined to participate. In total, 618 patients were randomized into the trial. Of the trial participants, 98% (605 of 618) of patients had complete baseline KOOS questionnaire data available for this analysis. Based on study inclusion criteria, the baseline KOOS data from the Stability 1 trial represents an appropriate sample to investigate the structural validity of the KOOS, specifically for the young, active ACL deficient population.

A cross sectional retrospective secondary data analysis of the Stability 1 baseline KOOS data was completed to assess the structural validity of the KOOS using exploratory and confirmatory factor analyses. Exploratory factor analysis investigates how all questionnaire items group together based on their conceptual similarity in a specific sample. Confirmatory factor analysis is similar but used often in a second stage to test and confirm a proposed structure of the subscales. These methods were used to assess the established five-factor structure of the KOOS (symptoms [seven items], pain [nine items], activities of daily living [17 items], sport and recreation [five items], and quality of life [four items]) in young active patients with ACL tears. Incremental posthoc modifications, such as correlating questionnaire items or moving items to different subscales, were made to the model structure until adequate fit was achieved. Model fit was assessed using chi-square, root mean square error of approximation (RMSEA) and an associated 90% confidence interval, comparative fit index (CFI), Tucker-Lewis index (TLI), as well as standardized root mean square residual (SRMR). Adequate fit was defined as a CFI and TLI > 0.9, and RMSEA and SRMR < 0.08.

Results

Structural validity of the KOOS was not confirmed when evaluated using (1) exploratory or (2) confirmatory factor analyses. The exploratory factor analysis, where the 42 KOOS items were allowed to group naturally, did not reflect adequate fit for a five-factor model (TLI = 0.828). Similarly, the confirmatory factor analysis used to investigate the KOOS structure as it was originally developed, revealed inadequate fit in our sample (RMSEA = 0.088 [90% CI 0.086 to 0.091]). Our analysis suggested a modified four-factor structure may be more appropriate in young, active ACL deficient patients; however, the final version presented here is not appropriate for clinical use because of the number and nature of post-hoc modifications required to reach adequate fit indices.

Conclusion

The established five-factor structure of the KOOS did not hold true in our sample of young, active patients undergoing ACL reconstruction, indicating poor structural validity.

Clinical Relevance

We question the utility and interpretability of KOOS subscale scores for young, active patients with ACL tears with the current form of the KOOS. A modified version of the KOOS, adjusted for this patient population, is needed to better reflect and interpret the outcomes and recovery trajectory in this high-functioning group. A separate analysis with a defined a priori development plan would be needed to create a valid alternative.

Introduction

The Knee Injury and Osteoarthritis Outcome Score (KOOS) is one of the most well known and commonly used patient-reported outcome measures (PROMs) for patients with ACL injuries [11]. Recently, the utility of the KOOS for patients with ACL tears has been called into question. Although studies have reported adequate reliability of the KOOS in patients with ACL deficiencies [10, 12], these studies have ceiling effects when used with young, active patients experiencing ACL injuries, which makes detecting differences between patients more difficult [11]. Additionally, the KOOS does not address knee stability and instability, a hallmark of ACL injury, recovery, and reconstruction success [11]. The Panther Symposium ACL Injury Clinical Outcomes Consensus Group recommends being conscious of the limitations of the KOOS for patients after an ACL injury or reconstruction to avoid overlooking treatment effects [11]. Despite the known limitations, the KOOS is a consistently used PROM in ACL research.

A substantial gap in the evaluation of the psychometric properties of outcome measures in orthopaedic research is a lack of structural validity assessment. Structural validity is defined as “the degree to which the scores of a PROM are an adequate reflection of the dimensionality of the construct to be measured”, or how appropriate the questionnaire subscales are [6]. This type of validity should be assessed for all multiitem outcome measures and provides valuable information on the acceptability of outcome subscales [6]. There is currently no evidence supporting the structural validity of the KOOS for patients with ACL injuries [11]. Factor analytic methods were not used to verify the KOOS structure during development, and thus a lack of knowledge exists as to whether the empirically examined factor structure would match the intended structure.

Therefore, we asked: Does the KOOS demonstrate adequate structural validity in young, active patients with ACL tears when evaluated using (1) exploratory and (2) confirmatory factor analyses?

Patients and Methods

Study Design, Setting, and Participants

This study was a cross sectional, retrospective secondary analysis of data obtained during the Stability study, a randomized clinical trial of young, active patients undergoing ACL reconstruction [4]. Between January 2014 and March 2017, 1033 patients were screened for eligibility in the Stability 1 trial from nine centers (seven in Canada and two in Europe). The inclusion criteria for the trial were patients between 14 and 25 years old undergoing primary ACL reconstruction who were deemed to have a high risk of surgical failure and reinjury based on at least two of the following criteria: a Grade 2 or higher pivot shift, a desire to return to high-risk or pivoting sports, and/or generalized ligamentous laxity [4]. Based on this criteria, 367 patients were ineligible and 48 declined to participate. In total, 618 patients were randomized into the trial. Reconstruction procedures were performed using a hamstring tendon autograft, and patients were randomized to undergo reconstruction with or without lateral extraarticular tenodesis, a stabilizing procedure intended to reduce postoperative rotational knee laxity [4]. Patients were followed for 2 years postoperatively, with data collected for several patient-reported, functional, and clinical outcomes. A detailed study protocol and trial results are available elsewhere [3, 4]. Importantly, baseline (postinjury and presurgery) data were used for this analysis; therefore, the intervention group that the patients were randomized to is not relevant to the current analyses. Of the randomized trial participants, 98% (605 of 618) of patients had complete baseline KOOS questionnaire data available for this analysis (full-item correlation matrix in Appendix 1; http://links.lww.com/CORR/A746). Based on study inclusion criteria, the baseline KOOS data from the Stability 1 trial represents an appropriate sample to investigate the structural validity of the KOOS, specifically for the young active ACL deficient population.

Ethical Approval

The Stability 1 study obtained ethical approval from the Western University Health Sciences Research Ethics Board (IRB#: 104524) and informed consent was obtained from all study participants.

Descriptive Data

Forty-eight percent (297 of 618) of the patients included in the analyses were male and 51% (318 of 618) were female, with a mean age of 19 ± 3 years and mean BMI of 24 ± 4 kg/m². The most played sports among the study patients were as follows: 51% (313 of 618) played soccer, 33% were runners (206 of 618), 29% played basketball (177 of 618), 22% played volleyball (134 of 618), 17% played hockey (108 of 618), 15% were downhill skiers (95 of 618), 14% played football (85 of 618), 12% played rugby (77 of 618), and 10% played baseball or softball (64 of 618).

KOOS Outcome Measure

We identified the KOOS as the outcome measure to be assessed for structural validity in our target population of young, active patients with ACL tears. The KOOS has five subscales (symptoms [seven items], pain [nine items], activities of daily living [ADLs; 17 items], sport and recreation [five items], and quality of life [QOL; four items], Fig. 1A) [8]. Each item has five possible ordinal responses; however, the response option labels are not consistent across all 42 items [8]. The subscales are evaluated by summing the score of each item (0 to 4) and then transforming the sum to a score from 0 (extreme knee problems) to 100 (no knee problems) [7]. These five subscales represent the original KOOS structure that we assessed for structural validity.

Fig. 1.

A-B These path diagrams represent the KOOS in young, active patient with ACL tears, including (A) the intended KOOS model, as described in the original development paper; and (B) acceptable modified KOOS model in the current dataset of young, active patients with ACL injuries, with standardized factor loadings. Correlations between latent variables are not depicted; in (B), these correlations range from 0.45 to 0.80 (p < 0.05). Items are labelled as follows: 7 symptoms items (s1-s7), 9 pain items (p1-p9), 17 ADL items (a1-a17), 5 sport and recreation items (sp1-sp5), 4 QOL items (q1-4); ADLs = activities of daily living; sport = sport and recreation function; QOL = quality of life.

Overview of Experimental Design/Statistical Analysis

The following is a breakdown of the analytical methods used throughout this study to assess the structural validity of the KOOS. Structural validity of a PROM asks the question: Are the items/questions in this outcome measure grouped appropriately into subscales (that is, factors), which reflect the intended structure and will reflect the outcomes of my target population? We used both exploratory and confirmatory factor analyses as outlined below to answer this question.

Exploratory Factor Analysis

An exploratory factor analysis will investigate how individual questionnaire items (questions) group together into constructs (or factors), such as pain or QOL. These factors suggest how the items should be grouped into subscales on PROMs. This analysis also provides evidence regarding the correct number of subscales for a questionnaire and groups items based on the strength of the relationship between them. The strength of the relationship between items and underlying factors are represented by standardized regression coefficients referred to as factor loadings, with large values indicating that the items are good exemplars of the underlying factors. Ideally, each item will have a high loading (with a minimum value of 0.30) on one of the factors but low values on other factors, indicating that it unambiguously defines one particular subscale. Otherwise, there is ambiguity about which subscale the item belongs to, and it complicates outcome scoring and interpretation. The magnitude of correlation between subscales themselves should not exceed 0.85 (standardized regression coefficient) because very high correlations suggest that the factors and their associated subscales are conceptually too similar and should be combined.

We conducted a five-factor exploratory factor analysis to evaluate the extent to which the grouping of the 42 KOOS items in our sample would match the developed questionnaire structure. We established a five-factor structure, and we identified items that belonged to particular factors (factor loadings > 0.3). We compared the resultant five-factor structure with the intended structure of the KOOS and assessed for model fit.

Confirmatory Factor Analysis

Confirmatory factor analysis is similar to exploratory factor analysis, but it is often used in a second stage of analysis to test and confirm a proposed structure. A confirmatory factor analysis involves testing the fit of the intended structure (or the intended subscales on a questionnaire) of an outcome measure by specifying which items group to uniquely define subscales, rather than allowing the items to group themselves (as in the exploratory analysis). This is achieved by submitting to analysis (a complex estimation procedure), a hypothesized structure and inspecting the quantitative fit indices indicating the extent to which the proposed structure is valid. Modification indices are also provided to indicate how the proposed structure can be improved. Our hypothesized structure consisted of the original five KOOS subscales of symptoms, pain, ADLs, sport, and QOL. If the original KOOS structure demonstrates structural validity within our sample, we would expect to find good model fit with this predefined structure.

Assessment of Fit/Structural Validity

Overall, quantitative model fit indices evaluate how well the proposed model provides a good fit to the data from our sample. These include a chi-square test of fit, root mean square error of approximation (RMSEA) and associated 90% confidence interval, comparative fit index (CFI), Tucker-Lewis index (TLI) and standardized root mean square residual (SRMR). A statistically significant chi-square suggests poor model fit (that is, the model differs from the data); however, it has been recognized that this test is too sensitive for samples larger than 200. Therefore, adequate fit was defined as a CFI and TLI > 0.9, and RMSEA and SRMR < 0.08 [5].

Modifications

Confirmatory factor analysis also provides information about where the misfit occurs in the model and modifications that could improve the model fit. These modification indices typically suggest whether a particular item belongs to a different factor or if it is ambiguous and belongs to more than one factor. A second example is finding some additional similarity between items within a factor and adding a correlation term between those items. In addition to the indices, items with poor loadings can be removed, and factors that correlate too highly (such as, above 0.85) can be combined. We made modifications one at a time, and we reanalyzed the model before pursuing additional modifications.

All analyses were conducted using the lavaan software package in R Studio [9]. The robust estimates of all indices are reported.

Results

Structural Validity of KOOS Using Exploratory Factor Analysis

The five-factor exploratory factor analysis in our sample did not match the five-factor structure of the original KOOS (fit indices: χ² = 1237.03; p < 0.001, TLI = 0.828, RMSEA = 0.076 (90% CI 0.073 to 0.079), and SRMR = 0.03) (Table 1). The analysis revealed adequate factor loadings for all items; however, some items had loadings greater than 0.3 on two factors (Table 1). Additionally, the fifth factor was defined by only two items, suggesting that a four-factor model (changing the KOOS to include four subscales versus five) may be more appropriate. The first factor was defined by 25 items, encompassing most symptoms, pain, and ADL-related items collapsed into one factor reflecting a subscale we labelled as “impairment” (Table 1). The second factor was defined by all five sports items (representing the sport subscale as intended) (Table 1). The third factor was defined by the remaining symptoms and pain items, relating to difficulty or pain with bending and straightening the knee (reflecting symptoms) (Table 1). The fourth factor was defined by all four QOL items (representing the QOL subscale as intended) (Table 1). The fifth factor was defined by only two items, reflecting difficulty putting on and removing socks, stockings, and shoes (Table 1). This factor should likely be removed as every subscale should be defined by at least three questionnaire items.

Table 1.

Subscale structure of the KOOS in young active patients with ACL tears based on a five-factor exploratory factor analysis

Factor 1 Impairment	Factor 2 Sport	Factor 3 Symptoms	Factor 4 Quality of life	Factor 5 Recommend removing
Symptoms (7 items)
		1
2
3
		4
		5
6		6
7		7
Pain (9 items)
1
		2
		3
		4
5
6
7
8
9
ADL (17 items)
1
2
3
4
5
6
7
8
				9
10
				11
12
13
14
15				15
16
17
Sport and recreation (5 items)
	1
	2
	3
	4
	5
QOL (4 items)
			1
			2
			3
			4

Fit indices: χ² = 1237.03; p < 0.001, TLI = 0.828, RMSEA = 0.076 (90% CI 0.073 to 0.079), and SRMR = 0.03. Items were assigned to a factor (column) if their loading onto that factor > 0.3; if the five-factor exploratory analysis was to support the intended structure of the KOOS, all items in one subscale would be grouped into one column, with no items from other subscales in that column, no items fitting into more than one column; TLI = Tucker-Lewis index, RMSEA = root mean square error of approximation, SRMR = standardized root mean square residual, ADL = activities of daily living, QOL = quality of life.

Structural Validity of KOOS Using Confirmatory Factor Analysis

The original KOOS model did not converge properly or have adequate fit in our sample (fit indices: χ² = 4617.55, p < 0.001, CFI = 0.909, TLI = 0.903, RMSEA = 0.088 [90% CI 0.086 to 0.091] and SRMR = 0.067). A negative item residual variance (QOL item 4; general difficulty with knee) and unacceptably high correlations between the symptoms and pain factors (0.91) and between the pain and ADL factors (0.97) indicated that the proposed structure was mathematically inappropriate in our dataset. Therefore, modifications to this model were needed to obtain an acceptable fit (Table 2). Nine modifications in total were performed iteratively, resulting in a four-factor model. The fit indices of the final model were CFI = 0.929, TLI = 0.925, RMSEA = 0.079 (90% CI 0.077 to 0.082), and SRMR = 0.062, with no correlations > 0.85 and no factor loadings < 0.3 (Table 2). Although only the final model converged properly with acceptable fit indices, this model would not be appropriate for clinical practice due to the number and type of modifications made. The final model structure contains four factors: symptoms, impairment, sport, and QOL, with three sets of correlated item residuals (Fig. 1).

Table 2.

Summary of KOOS model modifications and resultant fit indices following a confirmatory factor analysis of the KOOS in young, active patients with ACL tears

	Fit indices/indicators of an acceptable model structure
Summary of modifications	CFI > 0.9?	TLI > 0.9?	RMSEA < 0.08?	SRMR < 0.08?	Covariances < 0.85?	Loadings > 0.3?
Intended KOOS model	Yes	Yes	No	Yes	No	Yes
Remove QOL item, general difficulty with knee because of problematic negative variance	Yes	Yes	No	Yes	No	Yes
Combine pain and ADL factors to create “impairment” factor because of high covariance (0.97)	Yes	Yes	No	Yes	Yes	Yes
Combine symptoms into impairment factor because of high covariance (0.83)	Yes	No	No	Yes	Yes	Yes
Revert to four-factor model (better fit than three-factor) and allow correlation between residuals of two ADL items: difficulty putting on socks and difficulty taking off socks	Yes	Yes	No	Yes	Yes	Yes
Allow correlation between residuals of two ADL items: difficulty ascending stairs and descending stairs	Yes	Yes	No	Yes	Yes	Yes
Allow correlation between residuals of two pain items: pain walking on flat surfaces and difficulty walking on flat surfaces	Yes	Yes	No	Yes	Yes	Yes
Allow pain item, pain with twisting or pivoting on knee, to load on the sport factor	Yes	Yes	No	Yes	Yes	No
Remove pain with twisting or pivoting on knee from impairment factor because of problematic low loading (0.06)	Yes	Yes	Yes	Yes	Yes	Yes
Add QOL item general difficulty with knee, back into the model and allow it to load on the impairment factor to preserve all items	Yes	Yes	Yes	Yes	Yes	Yes

Only the last model listed had acceptable fit indices, covariances, and factor loadings; acceptable values: CFI and TLI > 0.9, RMSEA and SRMR < 0.08, all factor covariances < 0.85, and all factor loadings > 0.3; CFI = comparative fit index, TLI = Tucker-Lewis index, RMSEA = root mean square error of approximation, SRMR = standardized root mean square residual, ADL = activities of daily living, QOL = quality of life.

Discussion

Although the KOOS is commonly used to assess young, active patients after ACL injury, this application of the outcome measure has been questioned, and there is no evidence to support the structural validity of the questionnaire in this population. We conducted this study to investigate whether the proposed structure of the KOOS and the composition of its subscales is appropriate for young, active patients with ACL tears based on testing model fit with factor analyses. Our exploratory and confirmatory factor analyses suggest poor structural validity of the KOOS questionnaire in young patients with ACL tears. Neither the five-factor exploratory factor analysis nor the confirmatory factor analysis we conducted in this sample was consistent with the intended KOOS model structure. An alternative version of the KOOS is likely needed to accurately assess young, active patients with ACL injuries.

Limitations

Our study is limited by the sample in which the analyses were conducted. It is possible that the specific characteristics of our sample made it difficult to replicate the intended KOOS structure. Our sample consisted of a specific subset of patients with ACL injuries: young, active individuals with criteria placing them at an increased risk of re-injury who were willing to enroll in a randomized clinical trial. This sample may not represent or be generalizable to the wider population of patients with ACL deficiencies; however, it would be important for an ACL PROM to show validity across all possible presentations of ACL injuries. Additionally, this analysis was not conducted for male and female participants separately, and therefore, we cannot assume the results would apply equally to each sex.

Other researchers have used Rasch analysis to investigate KOOS structure or for similar purposes, which can provide more informative details for each item and may deal with ordinal items more successfully. Our rationale for using factor analysis rather than a Rasch modeling approach is that we wanted a method that would allow us to investigate the KOOS subscales and their appropriateness in our target population (the dimensionality of the instrument). One of the important assumptions of Rasch analysis or related item response models is unidimensionality of the measure (that it contains only one factor or scale). In the context of the KOOS, it was clear that a unidimensional model may not necessarily provide the best model. If we had found that a one-factor model provided the best fit in the exploratory analyses, then it would be acceptable to proceed with a Rasch analysis. Importantly, we did use an analytic approach that models categorical-ordinal items, similar to Rasch analysis.

Discussion of Key Findings

To our knowledge, this is the first study to use factor analyses to assess the structural validity of the KOOS in young patients with ACL injuries. Based on our assessment of the structural validity of the KOOS (that is, the appropriateness of how questions are grouped into the five subscales), we cannot recommend its use for young, active patients with ACL deficiencies. The proposed domain structure was not confirmed, and therefore it is difficult to have confidence in and to interpret the subscale scores as representing their intended constructs. Developing alternative versions such as short-form and age-based adaptations has been suggested as a strategy to improve the validity of outcome measures [2]. It may be worthwhile to build a simplified version of the KOOS that would be more appropriate for use in young, athletic patients with ACL injuries.

Using exploratory factor analysis, the underlying factor structure in our sample was clearly incongruent with the intended KOOS structure. The sport and QOL items loaded cleanly onto their intended domains. However, the pain and symptoms items grouped together, suggesting overlap between the domains. Based on this observation, we cannot be confident that the KOOS pain subscale score solely reflects pain and not some symptoms and vice versa. Most ADL items loaded cleanly onto one independent factor, except for a9 (difficulty putting on shoes and stockings) and a11 (difficulty removing shoes and stockings), which loaded onto their own unique factor. These are good examples of items that are likely too easy for young patients with ACL injuries and could be removed from the KOOS for this high-functioning patient population. The low median and SD values in this domain compared with that of the sport or QOL domains are further evidence of this. Additionally, a subscale should have at least three items to be well defined and improve internal consistency reliability. The fifth factor was not well defined, and a four-factor model may be more appropriate.

In the initial confirmatory factor analysis of the intended KOOS structure, inadequate fit indices, negative item variance and high covariance between factors were identified as problematic. Therefore, several modifications were made iteratively. A final four-factor model of the KOOS with acceptable fit was created after nine modifications to the initial model. Based on our analyses, the four remaining factors are symptoms, impairment, sport and recreation function, and QOL (Fig. 1). The impairment domain was created by combining the pain and ADL factors to describe the pain and difficulty associated with common activities and movements. In addition to the almost-perfect covariance between these factors, it made conceptual sense to combine them because every pain item (except for the first pain item, “how often is your knee painful?”) has a counterpart ADL item that discussed the identical action (for example, walking on a flat surface or ascending stairs). The second pain item (pain with twisting or pivoting on the knee) fits well within the sport domain because the twisting and pivoting motions described are most commonly performed during sports. The crossloading of the QOL item “overall, how much difficulty do you have with you knee?” to the impairment domain also makes sense conceptually.

Importantly, our final model cannot be considered confirmatory of the KOOS because no amount of conceptual rationale or theory would predict the correlated residuals or crossloadings required to obtain the resultant fit. Although modifications were partly informed by the magnitude of the modification indices, we made changes only when they seemed justified and after considering the conceptual coherence of particular items and factors. At the same time, we would not have been able to hypothesize the specific changes needed a priori. A well-fitted and clinically appropriate model would require only a few modification indices [1], while the final model in this study required a substantial number. Additionally, the correlated residuals between three sets of items prevent the use of simple scoring (summation of items in each group to get a subscale score) and therefore scoring would not be practical with this model. It would not be appropriate to use our final acceptable KOOS model with patients.

Altogether, these results suggest that the hypothesized KOOS model clearly does not fit our sample data well. Future confirmatory factor analyses would be required with new samples to test the original model as well as alternative models based on theoretical/clinical rationale and an a priori development plan for the target population of young, active patients with ACL tears. The International Knee Documentation Committee Subjective Knee Form (IKDC-SKF) is another popular patient reported outcome measure used for patients with ACL injury. This outcome similarly has no evidence supporting its structural validity specifically for this population [11]. An analysis of the IKDC-SKF structure in the ACL patient population is warranted.

Conclusion

The established five-factor structure of the KOOS was not confirmed during either the exploratory or confirmatory factor analysis in our sample, indicating poor structural validity. Therefore, we question the utility and interpretability of KOOS subscale scores for young, active patients with ACL tears. A modified version of the KOOS, adjusted for this patient population, is needed to better reflect and interpret the outcomes and recovery trajectory in this high-functioning group of patients. A separate analysis with a defined a priori development plan would be required to create a valid alternative.