KOOS-JR Demonstrates Psychometric Limitations in Measuring Knee Health in Individuals After ACL Reconstruction

Abstract

Background:

Measurement properties of the short form of the Knee injury and Osteoarthritis Outcome Score (KOOS-JR) are not established in individuals after anterior cruciate ligament reconstruction (ACLR). The purpose of this study was to determine the extent to which the KOOS-JR measures the construct of knee health in individuals post-ACLR using Rasch analysis.

Hypothesis:

The KOOS-JR will fit the Rasch model, but significant ceiling effects will be present.

Study Design:

Cross-sectional study.

Level of Evidence:

Level 3.

Methods:

Rasch analysis of the KOOS-JR from 166 individuals 10 months post-ACLR was conducted. Unidimensionality, a key criterion of the Rasch model, was evaluated using confirmatory factor analysis. Model fit of the rating scale, items, and persons were evaluated. Mean square fit statistics ≥1.6 and standardized z-scores ≥2.0 were indicative of person or item misfit. Additionally, reliability indicators including person reliability and separation indices were examined.

Results:

The KOOS-JR fit the criteria of unidimensionality. All items demonstrated model fit; however, ceiling effects were noted (n = 36; 22%). Person reliability was low (0.47). Calculation of person strata revealed that the KOOS-JR did not separate participants into more than 1 stratum. The mean person measure was 3.56 logits higher than the mean item measure, indicating that this sample is skewed toward increased knee health.

Conclusion:

Although the KOOS-JR represented a unidimensional construct with items and persons fitting the Rasch model, several limitations were noted: ceiling effects, low person reliability, and poor person differentiation. Ceiling effects indicate that many individuals in this sample experienced better knee health than the KOOS-JR items were able to measure.

Clinical Relevance:

Evaluating the measurement properties of the KOOS-JR is necessary to determine its clinical value in sports medicine. In later stages after ACLR recovery, administration of the KOOS-JR may not be adequate.

Complete functional recovery is not always achieved after anterior cruciate ligament reconstruction (ACLR), resulting in 25% to 45% of individuals failing to return to preinjury activity levels.^2,6 Recovery is multifaceted and can be clinically challenging to quantify in these individuals. With the ongoing paradigm shift toward a patient-centered model, more researchers and clinicians are acknowledging the importance of assessing recovery from the perspective of the patient, particularly through administering patient-reported outcomes (PROs).

The Knee injury and Osteoarthritis Outcome Score (KOOS) is one of the most common knee-related PROs used to assess perceived knee health and function in individuals who have undergone ACLR. However, the orthopaedic community has expressed concerns with the lengthy nature of the KOOS (42 items with responses in a 5-level Likert-type scale structure). Thus, shorter forms are beginning to emerge¹³; the KOOS-JR is a 7-item questionnaire intended to measure the construct of knee health.¹⁴ It was originally developed using individuals with end-stage osteoarthritis scheduled to undergo total knee replacements.¹⁴ More recently, however, the KOOS-JR is being administered to individuals after ACLR. While the KOOS-JR does have the potential to decrease respondent burden and promote efficiency in administration, theoretical and methodological investigation must be undertaken to determine its clinical usefulness in the ACLR population.

Ordinal response structures can be evaluated using item response theory and, more specifically, Rasch modeling.²⁰ Rasch modeling is described as “currently the closest generally accessible approximation of fundamental measurement principles of the human sciences.”⁵ It uses probabilities to investigate the relationship between difficulty of the items and ability of the persons within the sample, taking into consideration which items are more or less difficult and which persons have more or less ability.⁵ Many PROs used in ACLR and other orthopaedic conditions, including the KOOS-JR, have not been evaluated in this way. It is important to determine whether KOOS-JR data adhere to the Rasch model prior to widespread implementation for research and clinical use.¹⁰

The purpose of this investigation was to evaluate the dimensionality and psychometric properties of the KOOS-JR using Rasch analysis in individuals after ACLR. Since the KOOS-JR was developed using Rasch for individuals with osteoarthritis, we hypothesized that it will continue to fit the Rasch model when used for individuals post-ACLR. We also hypothesized that the use of the KOOS-JR in individuals post-ACLR will result in ceiling effects, since individuals who sustain ACL injuries tend to be younger and higher functioning than individuals experiencing osteoarthritis. Should our hypotheses be confirmed, we can use the KOOS-JR with more confidence that it is a useful indicator of knee health after ACLR.

Methods

Data Sources

This was a collaborative project involving a retrospective analysis of cross-sectional KOOS-JR data pooled from 4 different studies taking place at 3 research centers (University of North Carolina at Chapel Hill, Michigan State University, and Medical University of South Carolina). Individuals who completed this questionnaire had a history of unilateral ACLR. No questionnaires were excluded from analysis based on respondent demographics. A sample size of at least 100 surveys was needed to conduct the forthcoming analyses.⁴ The current project was approved as “not human subjects” research (retrospective analysis of deidentified survey data) by the institutional review board of the primary author’s institution.

The KOOS-JR Questionnaire

The KOOS-JR includes a total of 7 items that inquire about stiffness, pain, and activities of daily living intended to represent the construct of knee health.¹⁴ Creators of the KOOS-JR retained instructions and wording of items from the original KOOS questionnaire, and like the original, a 5-level Likert-type scale is used for responses. Responses are “none” to “extreme” difficulty and are coded 0 to 4, respectively. The KOOS-JR is scored by summing the raw scores of the 7 items (0-28) and converting them to a Rasch-based interval score ranging from 0 to 100, where 0 represents “total knee disability” and 100 represents “perfect knee health.” These interval scores were generated using Rasch-based scores from a validation cohort.¹⁴ The KOOS-JR and conversion table for scoring can be accessed from the following URL: https://www.hss.edu/hoos-jr-koos-jr-outcomes-surveys.asp.

Statistical Analysis

Unidimensionality

One of the primary assumptions of the Rasch model is unidimensionality. In other words, all items must represent a single construct. To test this assumption, we conducted an ordered-category confirmatory factor analysis (CFA) with diagonal-weighted least squares estimation using the package “lavaan” in the statistical software R.¹⁸ In accordance with best practice in CFA, we examined multiple types of fit indicators, including those reflective of absolute fit (standardized root mean square residual [SRMR], <0.08), those with parsimony corrections (root mean square error of approximation [RMSEA], <0.08), and comparative fit indicators (comparative fit index [CFI], >0.95; Tucker-Lewis index [TLI], >0.95). Acceptable model fit for these indicators was defined a priori.⁷ Factor loadings were considered to be significant if they were at least |0.32|, as this equates to approximately 10% overlapping variance with other items.¹⁹

Rasch Analysis

Rasch analysis was conducted to test the psychometric properties of the KOOS-JR using Winsteps Rasch Measurement (version 4.0.1). Specifically, joint maximum likelihood estimation was used with a partial-credit Rasch model. A partial-credit Rasch model was used because the KOOS-JR contains 2 conceptually different groups of items: pain/stiffness in questions 1 through 5 and function in questions 6 and 7.

A multistep approach was used to determine the measurement properties of the KOOS-JR. First, appropriateness of the rating scale was evaluated using the following criteria: (1) 10 or more observations in each rating scale category, collapsed across all items; (2) monotonicity of rating scale categories (ie, increase in average category difficulty with increasing category value); and (3) outfit mean square <2.0. Second, item fit statistics were used to identify misfitting items, where mean square standardized residuals ≥1.6 and standardized z-scores ≥2.0 indicated misfit.²¹ Point measure correlations were also evaluated to investigate the extent to which each item of the KOOS-JR related to the other items. Correlations of r > 0.75 were considered as strong, 0.25 < r < 0.75 as moderate, and r < 0.25 as weak.¹⁷ Moderate correlations are desired, indicating minimal inter-relationship among items. Third, person fit statistics were used to identify persons with response patterns that did not adhere to the Rasch model, with mean square standardized residuals ≥1.6 and standardized z-scores ≥2.0 indicating person misfit.²¹ Fourth, reliability indicators were evaluated, including (1) person reliability, which represents the reproducibility of person ordering, and (2) separation index, which is used to calculate the number of statistically distinct person ability strata in the sample.²² Person reliability was considered adequate for values ≥0.50, good for values ≥0.80, and high for values ≥0.90. The number of person strata is calculated as $\frac{(4G)+1}{3}$where G is the person separation index.²² Last, we evaluated test targeting and test coverage (ie, presence of ceiling or floor effects) via visual observation of person-item maps. Ceiling or floor effects are revealed if patients show extreme scores (higher or lower scores than what can be measured by the items). An acceptable maximum percentage of individuals in the ceiling or floor without affecting results is 15%.¹⁶

Results

Participant Demographics and KOOS-JR Outcomes

KOOS-JR questionnaires from 166 individuals after ACLR were included in this analysis, an adequate sample size to conduct the methods described above.⁴ The Rasch-derived score of the KOOS-JR for the entire sample was 79.9. No individuals were excluded based on any participant demographics (Table 1).

Table 1.

Respondent demographics (n = 166)

Age, y, median (range)	20 (14-41)
Time since surgery, mo, median (range)	10 (2-161)
Male, n (%)	68 (41)
Female, n (%)	98 (59)

Unidimensionality

The CFA of the KOOS-JR items revealed adequate fit to the unidimensional model (SRMR, 0.05; RMSEA, <0.01; CFI, 1.00; TLI, 1.11), and all items had significant (>|0.32|) factor loadings. This finding supported the requirement of unidimensionality for conducing Rasch analysis.

Rasch Analysis

Results revealed that each rating scale category had at least 10 observations, demonstrated monotonicity with the other categories, and had outfit mean square statistics <2.0, indicating that the rating scale had adequate fit to the Rasch model.

Second, item fit to the Rasch model was evaluated (Table 2), and all items met infit and outfit criteria for adequate fit. Point measure correlations were of moderate strength, indicating that the items were not excessively interrelated.

Table 2.

Individual item statistics^a

		Infit		Outfit
	Logit Measure (SE)	MnSq	ZStd	MnSq	ZStd	r
Standing	1.47 (0.23)	1.10	0.60	0.66	−1.30	0.52
Bend to floor	0.23 (0.19)	0.90	−0.5	0.86	−0.6	0.57
Straighten knee	0.12 (0.16)	1.27	1.8	1.07	0.5	0.60
Rise from sit	−0.04 (0.18)	0.91	−0.4	0.79	−1.0	0.60
Up/down stairs	−0.28 (0.15)	1.06	0.5	0.99	0.0	0.63
Stiffness	−0.69 (0.14)	0.95	−0.3	0.94	−0.4	0.67
Twist/pivot	−0.81 (0.14)	1.08	0.7	1.07	0.6	0.67

MnSq, mean square standardized residual; r, point-measure correlation; SE, standard error; ZStd, standardized z-scores.

^aItems listed in order of item difficulty.

Third, evaluation of person fit revealed only 3 of 166 individuals (1.8%) had responses that did not fit the Rasch model. These individuals were young females who had some of the lowest person measures in the sample. Examination of reliability indicators revealed that person reliability was 0.47, indicating less than adequate reproducibility of person measures. Additionally, the person separation index was 0.94, which was input into the formula for strata calculation.²² The strata calculation revealed that the KOOS-JR failed to divide the sample into 2 strata (strata, 1.59). Substantively, this result indicates that, based on these measurement properties, the KOOS-JR has limited ability to separate individuals who have problems with “knee health” from those who do not.

Last, test targeting and coverage were examined by inspecting the distribution of item and person measures on person-item maps (Figure 1). The mean person measure was 3.56 logits (SE, 1.01) higher than the mean of the items, suggesting that the present sample was skewed toward good knee health. Ceiling effects of the KOOS-JR were substantial, with 36 individuals (22%) who achieved maximum extreme scores, although no floor effects were found.

Figure 1.

Person-item map. The right side of the map represents the 7 items of the KOOS-JR, anchored at 0. The left side of the map represents persons. Persons are a mean of 3.56 logits higher than the mean item measure, which is anchored at 0. Each “#” is 3 persons; each “.” is 1 to 2 persons. KOOS-JR, short form of the Knee injury and Osteoarthritis Outcome Score.

Discussion

The purpose of this study was to determine the extent to which the KOOS-JR measures the construct of knee health in individuals post-ACLR. Presence of substantial ceiling effects indicates that many individuals in this sample experienced better knee health than the KOOS-JR items were able to measure. These results suggest that the value of the KOOS-JR in evaluating rehabilitation outcomes post-ACLR for research and clinical purposes may be improved by the addition of more difficult items, which would allow for better differentiation of individuals with and without limitations in knee health.

The use of the KOOS-JR is appealing because of its shortened length, high internal consistency, and ease of scoring. However, findings of this study suggest the need to reevaluate its use in ACL injury and surgery, as it was originally developed for individuals with end-stage osteoarthritis, who have greater disability and functional limitations. Research shows that the KOOS-JR functions well to measure knee health in the osteoarthritis population.^12,14 In fact, the Comprehensive Care for Joint Replacement model currently recommends use of the KOOS-JR for patients with osteoarthritis undergoing total knee replacement. Mandated use of PROs is anticipated to occur within 4 to 5 years.⁸ It is likely that these standards for clinical care and research in ACL injury and reconstruction will follow suit, as this area is closely related to that of osteoarthritis and total knee replacement. As health care research and practice progresses, it is likely that stakeholders will continue to promote the use of PROs for both reimbursement purposes and improvements in quality of care.

Substantial ceiling effects make it difficult to differentiate person ability levels and use scores to inform the development of treatment plans. As such, this questionnaire may not be sensitive enough to detect changes in function after rehabilitation interventions. In the context of rehabilitation, PROs should provide supportive information on the patient’s functioning to help clinicians understand patient perception of their own recovery, as well as identify deficits that can be targets for intervention to facilitate optimal return of function. After ACLR, patients experience limitations in recovery, observed via clinical- and laboratory-based measures (eg, muscle strength, functional performance).^1,20 In addition to clinical- and laboratory-based measures, patients experience problems with psychological readiness to return to activity,³ self-efficacy and motivation,¹¹ and kinesiophobia (patient report of fear of moving/reinjury).⁹ Effusion, giving way, muscle strength symmetry, return to sport, and patient-reported function were key criteria for evaluating a “successful outcome” after treatment for ACL injury.¹⁵ As such, it is imperative that researchers and clinicians ensure that patient-reported function is evaluated with efficient and precise tools to allow for better understanding of responses to treatment. Although the KOOS-JR asks questions about functioning, such as twisting/pivoting and knee bending, our results suggest that the instrument is unlikely to be effective in detecting changes resulting from post-ACLR rehabilitation treatment.

An alternative to the KOOS-JR could be the Function/Sports/Recreation and Quality of Life subscales from the full KOOS questionnaire, as these 2 subscales have shown fit to the Rasch model in a previous study.¹⁰ These 2 subscales together are only 9 items, allowing for better time efficiency and decreased respondent burden.

Limitations

First, KOOS-JR responses were derived from the full KOOS questionnaire (42 items), and it is unknown whether outcomes would have differed. However, we do not feel that this would have substantially changed the findings of this study. Second, this study was cross-sectional in nature and not prospective. Therefore, individuals included in this analysis were a median of 10 months post-ACLR, which may partly explain why individuals in this sample achieved such high scores.

Conclusion

The KOOS-JR represents a unidimensional construct and has adequate person and item fit to the Rasch measurement model for measuring knee health in individuals post-ACLR. However, it has psychometric limitations. The KOOS-JR demonstrated substantial ceiling effects for this sample of individuals who were, on average, nearly 1 year post–surgical intervention. The KOOS-JR may not be the best tool for rehabilitation research or treatment of ACLR, particularly for individuals in the later stages of recovery.