Can We Stop Asking So Many Questions? Comparing the Responsiveness of the Global Rating Scale to Legacy Knee Outcome Scores: a Delaware-Oslo Cohort Study

Abstract

Background

The selection of patient reported outcome measures (PROMs) is essential for obtaining meaningful information to manage a patient, determine a plan of care, and make clinical decisions, however, the process for selection of PROMs for clinical care is difficult, with the need to balance these multiple factors. Variation makes it difficult to compare data across providers and studies.

Hypothesis/Purpose

The objective of this study was to determine the responsiveness of four PROMs via effect size and the presence of a ceiling effect in the five years after ACL reconstruction. We hypothesized that the Global Rating Scale (GRS) would have similar effect size and ceiling effect to the commonly used legacy PROMs.

Study Design

Secondary analysis of a cohort study.

Methods

Of the 300 participants, 218 chose to have an ACLR, completed post-operative progressive criterion-based rehabilitation early after surgery, and were followed for five years. We collected the GRS, the Knee Outcome Survey-Activities of Daily Living Scale (KOS-ADLS), the International Knee Documentation Committee-Subjective Knee Form (IKDC-SKF), and the Knee injury and Osteoarthritis Outcome Score (KOOS) at pre- and post-training, and at 6, 12, 24, and 60 months after ACLR.

Results

The IKDC-SKF had the largest effect sizes and lowest ceiling effects. The GRS had a similar size and change in both effect size and ceiling effect compared to the longer PROMs. The GRS and IKDC had a correlation of 0.72, and the GRS had a MDC of 2.9/4.8.

Conclusion

The GRS responds similarly to the IKDC-SKF, KOS-ADLS, and KOOS and is responsive to patient change. The ease of use and patient-specific nature of the question means that it may be appropriate to use the GRS in clinical care as a consistent measure throughout the course of rehabilitation.

Introduction

While originally designed for use in research, patient-reported outcome measures (PROMs) are now also used by healthcare providers to assess the effects of clinical care and are designed to measure the patient’s perspective on their symptoms, function, and health-related quality of life.² The selection of PROMs is essential for obtaining meaningful information to manage a patient, determine a plan of care⁵ and make clinical decisions,³⁵ however, the process for selection of PROMs for clinical care is not easy. The relevance to the patient, psychometric properties, including reliability, validity, and responsiveness, as well as provider and patient burden are all important factors to consider in selection of a PROM.

A 2010 systematic review by Wang et al. identified and evaluated 24 separate PROMs for the knee and recommended different measures depending on diagnosis; while the International Knee Documentation Committee-Subjective Knee Form (IKDC-SKF) was the most generalizable, no measure was applicable across the spectrum of diagnoses or patient group.³⁶ Even within a diagnosis such as anterior cruciate ligament (ACL) injury and reconstruction, there are multiple recommended outcome measures.^12,19–21 While having multiple PROMs allows clinicians and researchers flexibility in selecting the appropriate measure for their patient, this variation makes it difficult to compare data across providers and studies.¹²

While the number of questions or length of time to complete PROMs that patients will accept varies,²⁵ longer PROMs place more strain on respondents and administrators and may lead to errors in responding to items and missing data,¹¹ as well as lower response rates.²⁵ Additionally, there may be higher burden if multiple surveys with similar concepts are administered at the same time,²⁵ as is often the case in research studies and at initial clinical visits.

Previous work on other single-item scales has found they are valid in those with knee injuries.^28,39 The single assessment numeric evaluation (SANE) asks respondents: On a scale from zero to 100, how would you rate your knee today (with 100 being normal)? In comparison, the Global Rating Scale (GRS) asks respondents: Rate your current knee function from 0% to 100%, with 100% equaling preinjury function.

The critical properties of outcome measures are patient-relevancy, user-friendliness, reliability, validity, and responsiveness to clinical change.¹⁰ Kirshner & Guyatt describe three applications of patient-reported measures; discriminative, used to distinguish between individuals on an underlying dimension, predictive, used as a screening or diagnostic instrument, and evaluative, used to quantitate treatment benefit and must be able to detect a clinically meaningful difference over time.¹⁶ This work falls into the last category, assessing the ability to detect real changes in functional impairments and quality of life. Within the context of assessing change, a valid measure must be able to detect a clinically-important change³⁰ and responsiveness is an instrument’s ability to detect real changes in the construct that it is intended to measure.³⁰ Construct validity may be accessed with correlations other measures of the construct.^14,32 Responsiveness may be measured with effect size,³¹ a standardized measure of change in a group,¹⁵ (mean change score in baseline standard deviation units; with pre-training as baseline)¹⁵ and the presence of a ceiling effect (percentage of patients with the maximum score).³⁴ A measure that has a ceiling effect is unable to identify a further improvement in a high functioning individual.³⁴

In a survey of health care providers regarding markers of a successful outcome after ACL injury and reconstruction, Lynch et al. found a consensus (≥80%) that PROMs are an important marker of success, but they did not find a consensus for the preferred use of any individual measure.¹⁸ The highest consensus PROM was for the Global Rating Scale (GRS) with 45% of respondents reporting it as an important measure. The Knee Outcome Survey-Activities of Daily Living Scale (KOS-ADLS), the IKDC-SKF, and the Knee injury and Osteoarthritis Outcome Score (KOOS) were rated as important by 41%, 38%, and 37% of respondents respectively.¹⁸ This lack of consensus among providers makes the selection of an appropriate outcome measure increasingly difficult.

Regardless, having one simple measure that can be used effectively throughout the rehabilitation timeline could improve plan of care decision-making as well as monitor progress across the continuum of care. If GRS has similar effect size and ceiling effect to longer measures, its use in isolation could decrease burden on patients and providers.

The objective of this study was to assess the validity of the GRS compared to the IKDC, determine the minimal detectable change (MDC) for the GRS, and to determine the responsiveness of four PROMs as identified by Lynch et al¹⁸ via effect size and the presence of a ceiling effect in a prospective cohort of ACL reconstructed followed for 5 years. We hypothesized that the GRS would have similar effect size and ceiling effect to the most commonly use legacy PROMs.

Methods

This is a secondary analysis of an ongoing prospective observational study.^8,17 The study was approved by the ethical/human subjects committees at the Regional Ethics Committee for South-Eastern Norway and the University of Delaware and all patients provided written informed consent, or parental consent with written assent if under 18 years old at enrollment. Participants were enrolled at both centers between 2007 and 2012. The Delaware-Oslo ACL Cohort Study is supported by grant R37 HD037985 from the National Institutes of Health

Participants

At enrollment, all participants had a complete unilateral ACL rupture confirmed by 3-mm or greater difference in anterior tibial excursion with instrumented arthrometry⁴ (KT1000; MEDmetric Corporation, San Diego, CO) within the previous seven months. Patients were athletes 13 to 55 years of age and regular participants in cutting and pivoting activities for at least 50 hours per year before their injury. Exclusion criteria included a repairable meniscus, symptomatic grade III injury to other knee ligaments, or greater than 1-cm² full-thickness articular cartilage lesion.

Study design

Following study enrollment, participants completed ten pre-operative rehabilitation sessions. Of the 300 participants, 218 chose to have an ACLR, completed post-operative progressive criterion-based rehabilitation early after surgery, and were followed for five years. We collected the GRS, KOS-ADLS, IKDC-SKF, and KOOS at pre- and post-training, and at 6, 12, 24, and 60 months after ACLR.

Global Rating Score

The GRS is a single item designed to assess current knee functional performance; patients are asked to rate their current knee function from 0% to 100%, with 100% equaling preinjury function.

Knee Outcome Survey-Activities of Daily Living Scale

The KOS-ADLS consists of 14 questions and was designed to determine symptoms and functional limitations in usual daily activities caused by various knee pathologies.¹² The KOS contains activities such as walking, stair climbing, and kneeling, and symptoms rated on their impact on these activities. It has questions related to recreational or sporting activities. Scores range between 0–100%, a greater symptoms and lower level of function resulting in a lower score.¹²

International Knee Documentation Committee-Subjective Knee Form

The IKDC-SKF has 18 items and was designed to detect improvement or deterioration in symptoms, function, and sports activities in a variety of knee conditions.¹³ It is reliable and valid for use in ligament and meniscal injuries, articular cartilage lesions, arthritis, and patellofemoral populations.¹³ Scores range between 0–100%, with higher scores representing lower levels of symptoms and higher levels of function and sports activity.¹³

Knee injury and Osteoarthritis Outcome Score

The KOOS has 42 questions arranged in five subscales: Symptoms, Pain, Activities of Daily Living (KOOS-ADL), Sport and Recreation function (Sport/Rec), and knee-related Quality of Life (KOOS-QOL) with scores ranging between 0–100%, with 100% equaling no difficulties.²⁶

Statistics

We used Excel (Microsoft, Seattle WA) to calculate effect sizes as: (mean score at time-point minus the mean score at baseline, divided by baseline standard deviation; pre-training as baseline)¹⁵ and the presence of a ceiling effect as a percentage of patients with the maximum score. Effect size and ceiling effects cut-offs were set a priori: 0.5 for medium effect size and 0.8 for large effect size,³ and ≥15% of participants having a maximum score for the presence of a ceiling effect.²³ We correlated with GRS with the IKDC-SKF with a Pearson correlation. We calculated the MDC for the GRS as the values associated with 20%²⁷ and 33%⁴¹ of the standard deviation of the measure at baseline. Power for the clinical trial was calculated a priori.

Results

Of the 300 total participants, 218 (100 women) completed training and chose to have an ACLR (106 hamstring grafts, 42 patellar tendon grafts, and 62 allografts); demographics at enrollment are in Table 1. Results shown are for all available data (Table 2); there was no change when the analysis used only those participants with a complete data set (n=114). We did not begin collecting KOOS responses from study participants until midway through enrollment, thus the analysis of the KOOS subscales included only the 69 participants with baseline KOOS scores.

Table 1:

Demographics of participants at enrollment

	Mean (range)
Age (years)	25.0 (13–52)
Height (cm)	174.5 (148–195)
Weight (kg)	75.7 (43.3–139)
Body Mass Index (BMI) (kg/m2)	24.7 (18.6–40.2)
Time from injury to enrollment (weeks)	8.3 (1–38)

Table 2:

Means, standard deviations, and percentage at ceiling for each PROM by time point.

Time point	Outcome	N	Mean	Std. Deviation	Effect size	Percentage at ceiling
Pre	KOS-ADLS	216	84.5	10.3	n/a	3%
	GRS	216	77.6	14.7	n/a	6%
	IKDC-SKF	210	69.6	12.6	n/a	0%
	KOOS-Pain	69	84.0	10.8	n/a	6%
	KOOS-Symptom	69	75.8	13.5	n/a	7%
	KOOS-ADL	69	93.5	7.4	n/a	23%
	KOOS-Sport/Rec	69	66.5	18.7	n/a	1%
	KOOS-QOL	69	51.0	19.1	n/a	0%
Post	KOS-ADLS	202	89.5	8.6	0.49	8%
	GRS	202	83.3	14.3	0.39	8%
	IKDC-SKF	188	76.8	12.8	0.57	1%
	KOOS-Pain	67	89.0	8.9	0.46	15%
	KOOS-Symptom	67	83.7	12.1	0.59	4%
	KOOS-ADL	67	97.1	3.7	0.48	39%
	KOOS-Sport/Rec	67	74.4	16.1	0.42	4%
	KOOS-QOL	67	54.7	18.8	0.20	0%
6m	KOS-ADLS	198	91.4	9.6	0.67	14%
	GRS	195	84.7	15.9	0.49	9%
	IKDC-SKF	188	82.6	12.5	1.03	4%
	KOOS-Pain	66	91.0	8.5	0.65	23%
	KOOS-Symptom	66	83.8	12.2	0.60	8%
	KOOS-ADL	66	96.8	5.1	0.44	47%
	KOOS-Sport/Rec	66	72.7	20.2	0.34	6%
	KOOS-QOL	66	58.1	20.0	0.37	3%
12m	KOS-ADLS	184	93.4	8.7	0.86	20%
	GRS	184	92.1	10.5	1	21%
	IKDC-SKF	181	89.2	11.8	1.54	11%
	KOOS-Pain	66	92.6	8.7	0.80	27%
	KOOS-Symptom	66	86.1	14.7	0.77	17%
	KOOS-ADL	66	98.0	4.8	0.60	59%
	KOOS-Sport/Rec	66	80.6	20.0	0.76	14%
	KOOS-QOL	66	70.8	19.7	1.04	6%
24m	KOS-ADLS	166	94.0	8.1	0.92	28%
	GRS	166	93.8	8.2	1.11	31%
	IKDC-SKF	166	90.7	11.1	1.66	22%
	KOOS-Pain	63	93.9	9.6	0.92	43%
	KOOS-Symptom	63	90.0	11.7	1.05	21%
	KOOS-ADL	63	98.2	5.1	0.63	68%
	KOOS-Sport/Rec	63	84.2	17.7	0.95	30%
	KOOS-QOL	63	75.2	20.9	1.26	13%
60m	KOS-ADLS	169	93.3	9.2	0.86	24%
	GRS	169	91.9	12.1	0.98	30%
	IKDC-SKF	169	88.8	12.4	1.52	13%
	KOOS-Pain	60	93.4	8.4	0.87	40%
	KOOS-Symptom	60	88.6	13.0	0.95	20%
	KOOS-ADL	60	98.0	4.9	0.60	70%
	KOOS-Sport/Rec	60	84.6	18.6	0.97	32%
	KOOS-QOL	60	73.7	21.3	1.18	8%

Effect Sizes

Effect sizes for all PROMs peaked at the 24-month time-point, with minimal differences observed between 12m, 24m, and 60m (Figure 1, Table 2). The IKDC-SKF had the largest effect sizes at all time-points (Figure 1). The GRS had a similar effect size and change in effect size compared to the longer PROMs.

Figure 1.

Effect sizes. Medium effect size ≥0.5 (dashed line), large effect size ≥0.8 (dotted line). Post=post-training time-point

Ceiling Effect

The KOOS-QOL subscale did not demonstrate a ceiling effect at any time-point and the IKDC-SKF only showed a ceiling effect at the 24m time-point (Figure 2, Table 2). All other measures had a ceiling effect at at least one time-point (most at 12m, 24m, and 60m) while the KOOS-ADL subscale had a ceiling effect at all time-points, and the KOOS-Pain subscale had ceiling effects at all but pre-training. The GRS had a similar effect size and change in ceiling effect compared to the longer PROMs.

Figure 2.

Ceiling effects. The dashed line is the 15% cutoff for ceiling effect.

Pre=pre-training time-point. Post=post-training time-point

Validity

The Pearson correlation coefficient between the GRS and the IKDC-SKF was 0.72 (p<0.001) when data were pooled across all time-points.

Minimally Important Change

The MDC for the GRS was 2.9 when using an 20%²⁷ and 30%⁴¹ of the standard deviation of the scores, which correspond to a small and small to medium effect.

Discussion

In the current study, we examined the effect sizes and ceiling effects of four PROMs after an ACL injury and reconstruction. The IKDC-SKF, GRS, KOS-ADLS, and the KOOS-QOL subscale demonstrated large effect sizes at 12m, 24m, and 60m after ACL reconstruction. These values indicated that these PROMs are sensitive to detecting clinical change.¹⁵ While the GRS had a small effect size post-training and at 6m, the overall trajectory of change is similar among all PROMs.

Gangel et al. correlated the SANE with the Lysholm knee score in 130 college-age patients after ACL reconstruction and found a good correlation (0.75, p<0.001).³⁹ Shelbourne et al. correlated the same SANE to the IKDC-SKF after ACL reconstruction and found an ICC of 0.66 and that the precision of agreement (calculated as the mean difference ±1 standard deviation), was met for 81% of respondents and limits of agreement (calculated as the mean difference ±2 SD) met for 94%.²⁸ In this study, the correlation between GRS and IKDC-SKF was 0.72 (0.69–0.75), p<0.001, which is a moderately strong relationship that provides some evidence of construct validity for the GRS in this population.

As may be expected, the large effect sizes at the later time-points correspond to the time-points with ceiling effects. All measures except the KOOS QOL had a ceiling effect at at least one time-point (most at 12m, 24m, and 60m) while the KOOS-ADL subscale had a ceiling effect at all time-points, and the KOOS-Pain subscale had ceiling effects at all but pre-training. The GRS had a similar size and change in ceiling effect compared to the longer PROMs. However, it is worth debating if the GRS can truly have a ceiling effect. The GRS asks a respondent to rate their current knee functional performance compared to pre-injury performance, so a score of 100% indicates the full resolution of any impairments or limitations relevant to that respondent. This may mean that, regardless of the percentage of respondents with a score of 100%, the GRS does not have a ceiling effect, but a performance asymptote, or the greatest true value that can be demonstrated³⁷. In a measure with a ceiling effect, a true score can only be observed if it is less than or equal to the ceiling threshold. So a patient may score 100% on the IKDC-SKF or the KOOS but still have functional limitations because not every activity or symptoms is covered on those PROM. In contrast, we would not expect there to be a score above the 100% scored on the GRS, this is an asymptote and not a ceiling, which differentiates it from the other measures.

Clinical decision making requires that health care providers choose the best tool for patient evaluation²⁵ and previous authors have suggested a combination of PROMs may be necessary to assess functional success.^5,18 In practice, a clinician could administer the GRS frequently, with the IKDC-SKF (or other measure as desired) given at the initial visit, then at 3 month (initiation of running²⁴/return to participation)¹ 6 months (return to sport activities)¹, 9 months (return to sport⁹/return to performance)¹, and at 12 and 24 months after ACLR. Based on our estimation of the MDC of 2.8/4.9, if a participant reports GRS scores of 60, 65, 65, and 70, a therapist should continue to progress treatment as indicated by stage of healing and objective criteria. If, however, a patient reports weekly scores of 60, 65, 65, and then 40, it would be appropriate to administer other tests/measures to identify causes for an unexpected change in status. While single-item measures are reliable and valid,^28,39 their simplicity comes at the cost of detail.²⁹ To gain this detail, clinicians could use outcome measures related to ADLs and functional limitations (KOS-ADLS¹² or IKDC-SKF¹³), fear (Tampa Scale of Kinseophobia⁴⁰ or the ACL-Return to Sports after Injury³⁸) or activity participation (Tegner³³ or Marx²²) as appropriate for the stage of recovery and goals of rehabilitation. Additionally, no PROM alone is sufficient to base all clinical decisions upon. Quadriceps strength, performance measures, and biological healing are all important factors to consider after an ACL injury.^1,5,7,9

Additionally, when a patient is nearing return to sport/performance, current recommendations suggest both objective and subjective criteria for clearance to return.^7,9 At the 12m time-point, the GRS had a large effect size, indicating it is responsive to a clinically meaningful change from baseline to the time of return-to-sport decision-making. Achieving a GRS of 90⁹ may indicate that the patient is ready for objective criteria testing.^7,9 As previously discussed, no one PROM is applicable to all patients at all time-points.³⁶ Clinicians can use the GRS for repeated, frequent assessment of change and reserve longer, more burdensome PROM for significant milestones during recovery.

Strengths and Limitations:

The study population was active athletes after ACL injury and subsequent ACLR. These patients were not enrolled immediately after their ACL injury, but at a “quiet knee” stage of recovery (8.3±5 weeks, range 1–38 weeks after injury). We, therefore, do not have data from the patient’s initial encounter with a health professional, nor do we have immediate post-operative data and so are unable to assess effect sizes with an early post-operative baseline. Additionally, previous research comparing this cohort to a comparable subgroup of the MOON cohort found our participants had statistically and clinically significantly higher baseline and two-year IKDC-SKF and KOOS scores⁶.

We assessed validity, responsiveness, and determined the MDC of the GRS in this study. Future research should establish test-retest reliability, standard error of the measurement. Also, while none of our measures are ACL specific, we only analyzed patients with ACL rupture and subsequent ACLR. We do not know the validity, responsiveness or MDC in patients with other knee injuries/surgeries.

Also, because we did not begin using the KOOS until midway through enrollment, analysis of the KOOS was only done in the 69 participants who had baseline KOOS scores.

We were unable to directly assess response burden in our participants. The difference in the length and thus the time needed to complete each PROMs is however much shorter in the GRS than the other three measures. The simplicity of administering the GRS may decrease burden on respondents and administrators while still providing responsive and valid information regarding patient status.

Conclusions

The IKDC-SKF has the largest effects sizes while the KOOS-QOL had the smallest ceiling effects. The GRS, however, responds similarly to the IKDC-SKF, KOS-ADLS, and KOOS measures and is responsive to patient change, with evidence of construct validity and a small MDC. The ease of use and patient-specific nature of the question means that, for clinical practice, it may be appropriate to use the GRS as a frequent measure throughout the course of rehabilitation, with different measures used at the beginning of treatment and other measures used at the later stages, or specific scales based on patient’s deficits or goals.

What is known about the subject:

There are more than 20 PROM for the knee alone, many are reliable and valid. However, the lack of consensus among providers makes the selection of an appropriate outcome measure increasingly difficult.

What this study adds to existing knowledge:

The Global Rating Scale was found to have similar effect sizes and presence of ceiling effects as commonly used legacy PROMs

Teaser:

Can we stop asking so many questions? The single-item Global Rating Scale responds similarly to commonly used PROMs; it may be a simple way to assess patient change throughout the course of rehabilitation.