Development of the KOOS_global patient-reported outcome measurement platform to assess patient-reported outcomes after anterior cruciate ligament reconstruction

Abstract

Background:

The Knee Injury and Osteoarthritis Outcome Score (KOOS) has demonstrated inferior psychometric properties when compared to the International Knee Documentation Committee Subjective Knee Form (IKDC). The KOOS, Joint Replacement (KOOS,JR) is a validated short-form instrument to assess patient-reported outcome measures (PROs) after knee arthroplasty, and the purpose of this study was to determine if augmenting the KOOS,JR with additional KOOS items would allow for the creation of a short-form KOOS-based global knee score for patients undergoing anterior cruciate ligament (ACL) reconstruction with psychometric properties similar to that of the IKDC.

Hypothesis:

We hypothesized that an augmented version of the KOOS,JR could be created that would demonstrate convergent validity with the IKDC but avoid the ceiling effects and limitations previously noted with several of the KOOS subscales.

Study Design:

Cohort study. Level of Evidence: Diagnostic, Level II.

Methods:

Using pre- and 2-year postoperative responses to the KOOS questionnaires from a sample of 1,904 primary ACL reconstruction patients, an aggregate score combining the KOOS,JR and the 4 KOOS Quality of Life questions, termed the KOOS_global, was developed. Again using the pre- and postoperative responses from the 1,904 ACL reconstruction patients, psychometric properties of the KOOS_global were compared to the IKDC Subjective Score. Convergent validity between the KOOS_global and IKDC was assessed with a Spearman correlation (rho). Responsiveness of the 2 instruments was assessed by calculating the pre- to effect size (ES) and relative efficiency (RE). Finally, the presence of a preoperative floor or postoperative ceiling effect was defined using the threshold of 15% of patients reporting either the worst possible (0 for both KOOS_global and IKDC) or best possible scores (100 for both KOOS_global and IKDC), respectively.

Results:

The newly developed KOOS_global was responsive after ACL reconstruction and demonstrated convergent validity with the IKDC. The KOOS_global significantly correlated with IKDC scores (rho=.91, p<.001), explained 83% of the variability in IKDC scores, and was similarly responsive (RE=.63). While there was a higher rate of perfect postoperative scores with the KOOS_global (213/1904, 11%) than IKDC (6%), KOOS_global was still below the 15% ceiling effect threshold.

Conclusions:

The large ceiling effect limits the ability to use of several of the KOOS subscales in the younger, more active ACL patient population. However, by creating an aggregate score from the KOOS,JR and the 4 KOOS Quality of Life questions, the 11-item KOOS_global offers a responsive PRO tool after ACL reconstruction that converges with the information captured with the IKDC. Also, by offering the ability to calculate multiple scores from a single questionnaire, the 11-item KOOS_global may potentially provide the orthopedic community a single PRO platform to be used across knee-related subspecialties.

Introduction

Patient-reported outcomes measures, such as the Knee Injury and Osteoarthritis Outcome Score (KOOS), have provided invaluable information to researchers about the relative success of orthopedic interventions.^{3, 21, 31, 32} However, while important research tools, to date these tools have not been practical for clinicians to incorporate into their daily practices due to the number of questions and both the time required of the patient to complete as well as the time of the clinic staff to record. In anticipation of mandatory physician quality reporting for arthroplasty surgeons and the ensuing burden that will create, Lyman et al. recently validated the KOOS Joint Replacement, or KOOS,JR.²⁴ The KOOS,JR is a global score generated from a subset of seven questions from the full version of the 42-question KOOS that provides clinicians with an efficient and responsive tool to evaluate postoperative patient-reported outcomes (PROs) following total knee arthroplasty (TKA) procedures.²⁴

There are clear advantages of a valid and responsive outcome measure that can be generated from a limited number of questions. Meaningful clinical data can be generated but with less burden to the patient, surgeon, and clinical staff. This also creates the potential for PROs to more easily be incorporated into electronic medical records systems. Despite these advantages, the logistical challenges of using multiple PRO tools across knee-related subspecialties continues to be a barrier to widespread collection of PROs in the clinical setting.

The ideal scenario would be a single PRO platform that could be used across knee-related subspecialties to reduce logistical barriers to PRO collection. The KOOS,JR has been demonstrated to be a valid and efficient measure of patient-reported outcomes after knee arthroplasty; however, it is unlikely the KOOS,JR would be a valid and responsive PRO tool when administered to a younger, more active population of patients undergoing anterior cruciate ligament (ACL) reconstruction since the KOOS,JR was developed with knee arthroplasty patients. Therefore, the purpose of this study was to determine if augmenting the KOOS,JR with additional KOOS items would allow for the creation of a short-form KOOS-based global knee score for patients undergoing anterior cruciate ligament (ACL) reconstruction with psychometric properties similar to that of the IKDC. We hypothesized that an augmented version of the KOOS,JR could be created that would demonstrate convergent validity with the IKDC but avoid the ceiling effects and limitations previously noted with several of the KOOS subscales.

Methods

The study included 2,020 patients that underwent unilateral primary ACL reconstruction between 2002 and 2008 that had consented to participate in a multi-center, IRB-approved prospective cohort (Vanderbilt University IRB protocol #990426). All patients were included with the exception of patients undergoing revision ACL reconstruction or those that incurred a subsequent graft failure or additional ipsilateral reoperation at the time of their 2-year follow-up. Patients were not excluded based on age, sex, race, or the presence of concomitant injury. There was a subset of 116 patients with incomplete IKDC information that were excluded leaving a total sample of 1,904 patients in the current analysis (1904/2020 = 94% of the available sample).

Methods similar to the original KOOS,JR validation study were employed to determine if an augmented version of the KOOS,JR could be created that would demonstrate similar psychometric properties as the IKDC.²⁴ First, the specific KOOS subscales that did not correlate well with the KOOS,JR were identified. This was done on the premise that questions from these subscales would provide unique information not already provided by questions already included in the KOOS,JR as evidenced by the lower correlations with the KOOS,JR score. Individual questions within the identified subscale(s) were then selected if > 67% of patients had preoperative responses of “moderate” or greater.²⁴ Once additional questions were identified, an aggregate score of the seven KOOS,JR questions and any additional questions, termed the KOOS_global, was calculated using methods similar to a recent study involving the Hip disability and Osteoarthritis Outcome Score.¹³ We applied a Rasch measurement model²⁹ to the pre-operative and post-operative data using both the KOOS,JR and KOOS_global. The data were stacked^{35, 36} in order to examine the entire range of patient outcomes rather than examine pre-operative and post-operative separately, and the analysis was conducted using the Rasch Partial Credit model^{26, 38} available in Winsteps Rasch Measurement Software version 3.81.0 (Winsteps, Beaverton, OR) These analyses can be performed with missing values, and as such, we were able to include all 2,020 patients. Details regarding the methods used in the Rasch analysis, item inclusion and exclusion, and comparative results of the KOOS,JR and KOOS_global can be found in the accompanying electronic Supplemental files.^{1, 20, 33, 37}

Both the convergent validity and responsiveness of the KOOS_global in relation to the IKDC were assessed.^{10, 11, 16, 17, 25, 27, 34} The convergent validity and equivalence of the KOOS_global were examined using the IKDC Subjective Score as the “gold standard.” In order to be considered to be equivalent to the IKDC, the KOOS_global must demonstrate a Spearman correlation > 0.90 with the IKDC.^{10, 16, 25} The correlation was then squared (rho²) to determine the percentage of the variability in the IKDC that was explained by the KOOS_global.

The pre- to postoperative responsiveness of the KOOS_global was assessed and compared to the IKDC by calculating effect size (ES) and relative efficiency (RE).^{16, 18, 27} ES represents the average pre- to postoperative change divided by the standard deviation of the preoperative scores. Large ES are defined as those > 0.8, and ES differences between the KOOS_global and IKDC < 0.3 were considered to be representative of similar responsiveness.^{8, 16} RE is a ratio of the pre- to postoperative t-statistics of the KOOS_global and IKDC [RE = (t_KOOSglobal/t_IKDC)²].^{17, 18} RE values > 0.6 were considered to be indicative of the KOOS_global being similarly responsive as the IKDC.¹⁶ The presence of a preoperative floor or postoperative ceiling effect was defined using the threshold of 15% of patients reporting either the worst possible (0 for both KOOS_global and IKDC) or best possible scores (100 for both KOOS_global and IKDC), respectively.¹⁹

Results

One additional question from the KOOS Sport/Recreation subscale and four questions from the KOOS Quality of Life subscales were identified due to the low correlations between the KOOS,JR and these items. More than 67% of patients had preoperative responses of “moderate” or worse to all four questions from the Quality of Life subscale, thus all four items met the threshold for inclusion. While 71% of patients reported at least moderate difficulty when responding to the Sport/Recreation question regarding twisting and pivoting, consistent with the methods of Lyman et al.,²⁴ this question was not added to the KOOS_global as the KOOS,JR already included a question related to pain during this specific task.

Pre- and postoperative KOOS_global scores were then calculated by combining the responses from the four Quality of Life questions with the seven KOOS,JR questions. Similar to the scoring of the KOOS,JR, the responses to the 11 KOOS_global were summed to generate a raw score, and the raw score was then converted using a logit transformation. The KOOS_global questionnaire, scoring instructions, and details related to the logit transformation used to transform raw scores can be found in Figures 1 and 2. For the KOOS_global score, the lowest possible raw score which would be indicative of no pain or dysfunction (0/44) was scaled to 100 and the highest possible raw score (44/44), which is indicative of extreme pain and dysfunction was scaled to 0.

Figure 1.

KOOS_global questionnaire. Questions identified with an asterisk are those that would be scored to calculate the KOOS,JR.

Figure 2.

KOOS_global scoring instructions.

The KOOS_global met the thresholds for both convergent validity and responsiveness to be considered sufficiently equivalent to the IKDC. The KOOS_global significantly correlated with IKDC scores (rho = 0.91, p < 0.001), and explained 83% of the variability in IKDC scores. KOOS_global scores significantly increased between the preoperative and 2-year postoperative follow-ups (Preoperative = 53.3 ± 9.7, Postoperative = 73.3 ± 14.6, p < 0.001, ES = 2.1). Furthermore, the responsiveness of the KOOS_global did not dramatically differ from that of the IKDC (RE = 0.63). While there was a higher rate of perfect postoperative scores with the KOOS_global (213/1904, 11%) than IKDC (6%), KOOS_global was still below the 15% ceiling effect threshold. There was no evidence of a ceiling effect preoperatively (4/1904, <1%), or a floor effect either prior to (0/1904) or following surgery (0/1904) with KOOS_global (Table 1).

Table 1.

Pre- and 2-year postoperative KOOS, JR, KOOS_global, and IKDC scores (mean ± standard deviation) from 1,904 ACL reconstruction patients

	KOOSglobal	IKDC
Preoperative	53.3 ± 9.7	51.7 ± 17.0
Scores of 0 (n, %)	0 (0%)	1 (<1%)
Scores of 100 (n, %)	4 (<1%)	4 (<1%)
Postoperative	73.3 ± 14.6	83.2 ± 15.3
Scores of 0 (n, %)	0 (0%)	0 (0%)
Scores of 100 (n, %)	213 (11%)	122 (6%)
Pre/post t statistic	59.71	74.96
Effect size	2.1	1.8

Discussion

By creating an aggregate score of the KOOS,JR and KOOS Quality of Life questions, the 11-item KOOS_global demonstrated convergent validity with the 19-item IKDC and was responsive in a population of ACL reconstruction patients. The current results have implications both for those implementing PROs into their clinical practice but also for groups that have collected KOOS questionnaires from large volumes of ACL patients. For example, several national ACL reconstruction registries utilize the KOOS to assess patient-reported outcomes. As indicated in Part I of this two-part series, the KOOS ADL, Sports, and Symptom subscales have psychometric limitations when administered in the ACL reconstruction patient population. In addition, the traditional KOOS scoring does not allow for a single score to be calculated to represent the overall condition of the knee. The KOOS_global appears to avoid the psychometric limitations of the ADL, Sports, and Symptoms subscales while providing information that is substantially similar to that provided by the IKDC. As a result, groups or registries that have utilized the KOOS for a number of years may now have an efficient method to compare their results to those that have utilized the IKDC, while avoiding the ceiling effects and large minimal detectable change values previously demonstrated by some of the KOOS subscales when administered after ACL reconstruction.

In 2016, the American Academy of Orthopaedic Surgeons’ Quality Outcomes Data Work Group released a list of recommended PRO tools based on anatomical region and diagnosis.¹⁴ The IKDC was identified as the consensus recommended tool to assess outcomes for ACL-injured or reconstructed patients, whereas the KOOS and KOOS,JR were recommended for those with knee osteoarthritis.¹⁴ The Work Group stated that the initial list was intended to steer data collection and reporting, but further stated that they anticipated that the list could change over time.¹⁴ While these recommendations provided initial guidance, barriers to implementation arise if clinically adopting these recommendations to longitudinally follow ACL reconstruction patients over time when we know many ACL patients will progressively transition into the osteoarthritis patient population.⁶ This issue arises when one asks the question, “When does an ACL patient become and osteoarthritis patient?” Similarly then, when is it appropriate to transition from one tool to the other? This designation has historically been based on radiographic changes or degenerative defects noted on MRI.⁶ However, one of the advantages of longitudinal PRO collection is that it can be done remotely and without the need for additional imaging. As such, it is simply not feasible to tie the decision of whether to use the IKDC or KOOS based on imaging results. In the absence of a clear definition of when to stop the use of the IKDC and begin using the KOOS or KOOS,JR with a given patient, the KOOS_global could seamlessly bridge the gap as patients transition from being considered an ACL patient to being considered an osteoarthritis patient. Furthermore, this can be accomplished in an efficient manner with the 11-item KOOS_global compared to 19-item IKDC or 42-item KOOS. Longer PRO questionnaires have been associated with reduced patient response rates when compared to shorter questionnaires provided that both tools provide similar content.³⁰ The brevity of the KOOS_global does not appear to come at a cost in terms of lost information as the KOOS_global converged with the IKDC and showed similar responsiveness. An added bonus of a validated KOOS_global that is only 11 questions in length is the possibility of alternative platforms for soliciting short, intermediate, and long-term follow-up such as text messaging. Anthony et al. evaluated text messaging for delivery of the patient questionnaires in the early postoperative period, and reported > 85% follow-up at virtually no cost.² Similarly, Blocker et al. utilized text messaging to follow arthroplasty patients at greater than two years follow-up and concluded that text messaging is viable avenue for following patients long-term.⁴

It is important to note that we are in no way suggesting that the KOOS_global platform be considered a stand-alone replacement for the full KOOS in the research arena, nor do we believe that the KOOS_global should somehow supplant the KOOS,JR as a PRO tool in the arthroplasty patient population. The volume of TKAs has continued to rise in the United States, with more than 700,000 patients undergoing the procedure on an annual basis.^{5, 15} This patient volume, combined with potential mandated reporting of PROs as part of the CMS pay-for-performance measures,⁷ may place a significant burden on orthopedic practices in terms of both the costs and resources necessary to collect this information.⁹ The KOOS,JR was developed to provide arthroplasty surgeons with an efficient method to capture clinically-meaningful PROs.²⁴ While not as prevalent as osteoarthritis, ACL injuries are common, affecting approximately 250,000 people in the United States each year.¹² ACL injury appears to initiate the cascade of post-traumatic osteoarthritis, with more than 50% demonstrating arthritic changes five to 15 years after injury.^{22, 23, 28} As ACL injury appears to be a possible first step down the path towards osteoarthritis, few global PRO assessment tools allow for evaluation across the spectrum of disease progression.

The 11 questions identified in the current study allow the clinician to independently calculate a KOOS_global (scored using all 11 questions), KOOS,JR score (7 questions) and KOOS-Quality of Life subscale (4 questions). The principal advantage of using the KOOS_global questionnaire as a PRO platform is the versatility to be used across a variety of knee patient populations. In addition, the KOOS_global potentially alleviates some of the logistical barriers to routine PRO collection. These 11 questions take less than five minutes to complete, minimizing the burden to the patient. A single PRO questionnaire may also reduce the burden to the orthopedic practice. Front office staff would not need to differentiate which PRO questionnaire to give each patient when compared to a scenario in which multiple PRO tools were being used to evaluate different patient populations. For example, the front office staff for a clinician that normally sees both sports and osteoarthritis patients during the course of clinic day would not have to determine which PRO tool to give each knee patient, but rather, could use the KOOS_global questionnaire for adult knee patients. Similarly, for a multi-subspecialty practice in which surgeons of different subspecialties utilize the same staff and clinic space on different days, the front office staff would not have to remember which PRO tool to use on Monday versus Tuesday, but again, could use the KOOS_global questionnaire for adult knee patients. A single questionnaire can be given to adult knee patients, and then scored differently based on whether the patient had an ACL reconstruction (scored using all 11 KOOS_global questions) or had a total knee arthroplasty (scored using the 7 KOOS,JR questions within the KOOS_global questionnaire).

Furthermore, KOOS_global may allow information technology personnel to incorporate a single PRO platform within a practice’s electronic medical records system. If found to be a valid instrument with other common knee diagnoses, all knee patients could complete the 11 KOOS_global questions with three potential scores being created (KOOS,JR, KOOS Quality of Life, and KOOS_global). As pay-for-performance reporting mandates are implemented, the selection of which score to calculate and report for a given patient would not fall on the surgeon or orthopedic staff, but could be programmatically tied to procedure or diagnosis codes. This could allow for the same series of 11 questions to be scored differently for an ACL reconstruction (KOOS_global) and arthroplasty patient (KOOS,JR).

This study was not without limitations. The use of the KOOS_global has been validated in an ACL patient population and allows for the calculation of the KOOS,JR, which has been validated in the TKA population. While these validation studies include the “anchors” on the continuum of post-traumatic osteoarthritis (i.e. ACL injury and TKA), future studies are necessary to determine if the current results are generalizable to those with mild to moderate knee osteoarthritis or other common knee diagnoses.

Conclusions

By creating an aggregate score from the KOOS,JR and KOOS Quality of Life questions, the KOOS_global offers a responsive PRO tool after ACL reconstruction that converges with the information captured with the IKDC. Also, by offering the ability to calculate multiple scores from a single questionnaire, the 11-item KOOS_global may potentially provide the orthopedic community a single PRO platform to be used across knee-related subspecialties.

What is known about the subject:

Marked ceiling effects have been noted in some of the KOOS subscales following ACL reconstruction.

What this study adds to the existing knowledge:

By creating an aggregate score from the KOOS,JR and the 4 KOOS Quality of Life questions, the KOOS_global offers a responsive PRO tool after ACL reconstruction that converges with the information captured with the IKDC.