Comparing Responsiveness of Six Common Patient-Reported Outcomes to Changes Following Autologous Chondrocyte Implantation

Abstract

Objective:

To compare the responsiveness of six common patient-reported outcomes (PROs) following autologous chondrocyte implantation (ACI).

Design:

A systematic search was conducted to identify reports of PROs following ACI. Study quality was evaluated using the modified Coleman Methodology Score (mCMS). For each outcome score, pre- to postoperative paired Hedge’s g effect sizes were calculated with 95% confidence intervals (CIs). Random effects meta-analyses were performed to provide a summary response for each PRO at time points (TP) I (<1 year), II (1 year to <2 years), III (2 years to <4 years), IV (≥4 years), and overall.

Results:

The mean mCMS for the 42 articles included was 50.9 ± 9.2. For all evaluated instruments, none of the mean effect size CIs encompassed zero. The International Knee Documentation Committee Subjective Knee Form (IKDC) had increasing responsiveness over time with TP-IV, demonstrating greater mean effect size [confidence interval] (1.78 [1.33, 2.24]) than TP-I (0.88 [0.69, 1.07]). The Knee Injury and Osteoarthritis Outcome Score–Sports and recreation subscale (KOOS-Sports) was more responsive at TP-III (1.76 [0.87, 2.64]) and TP-IV (0.98 [0.81, 1.15]) than TP-I (0.61 [0.44, 0.78]). Overall, the Medical Outcomes Study 36-Item Short Form Health Survey Physical Component Scale (0.60 [0.46, 0.74]) was least responsive. Both the Lysholm Scale (1.42 [1.14, 1.72]) and the IKDC (1.37 [1.13, 1.62]) appear more responsive than the KOOS-Sports (0.90 [0.73, 1.07]). All other KOOS subscales had overall effect sizes ranging from 0.90 (0.74, 1.22) (Symptoms) to 1.15 (0.76, 1.54) (Quality of Life).

Conclusions:

All instruments were responsive to improvements in function following ACI. The Lysholm and IKDC were the most responsive instruments across time. IKDC and KOOS-Sports may be more responsive to long-term outcomes, especially among active individuals.

Introduction

The limited ability of articular cartilage to heal on its own has been a topic of discussion for more than 200 years.¹ The treatment and management of articular cartilage damage can be particularly challenging in the knee joint where such defects are frequently observed during arthroscopy.^2-5 Restorative and reparative treatment of these defects is highly desirable to prevent the progression of osteoarthritis.⁶ Over the last three decades, approaches to treating chondral defects have shifted toward cell-based therapies. One of the most frequently used and well studied is the autologous chondrocyte implantation (ACI).⁷

Treatment Evaluation

As new methods for treating cartilage are developed, it is necessary to evaluate these treatments to determine their effectiveness. Although second look arthroscopies with cartilage biopsies may provide the most diagnostic method of evaluating cartilage repair, they are not always feasible or ethical to perform. Furthermore, biopsies allow for the assessment of the histological tissue repair but cannot be used to evaluate patient-oriented outcomes such as pain and function. To evaluate patient-oriented outcomes, investigators have relied on patient-reported outcome (PRO) instruments. Many PROs have been developed to address outcomes associated with a specific body part or region, a specific disease, or health-related quality of life as a whole. Numerous PROs have been used to document patient response to cartilage repair. Although the widespread use of PROs is beneficial for documenting treatment outcomes, the wide variety in PROs makes comparison across studies and instruments difficult. Ideally, a standard instrument or battery of instruments would be advantageous for reliable and valid assessment of patient response to treatment.

Some of the most commonly used PROs to evaluate articular cartilage repair outcomes include the Medical Outcomes Study 36-Item Short Form Health Survey (SF-36),^8-10 the International Knee Documentation Committee Subjective Knee Form (IKDC),¹¹ the Lysholm Knee Scale (Lysholm),^12,13 the Modified Cincinnati Knee Rating System (MCKRS),¹⁴ the Western Ontario and McMaster Universities Osteoarthritis Index (WOMAC),^15,16 and the Knee Injury and Osteoarthritis Outcome Score (KOOS).¹⁷ Test–retest reliability and validity among cartilage patients has previously been established for a version of each of these instruments.^18-21 Although all these instruments have been widely used to evaluate ACI treatment efficacy, there is no clear standard regarding which outcome instrument is ideal for evaluating treatment effects following ACI.PRO responsiveness is the evaluation of change in the instrument score over time in response to treatment.¹⁸ The reported responsiveness in self-reported function following ACI has not been compared among instruments. Identification of the most responsive instrument for an ACI population will provide clinicians and researchers with a specific PRO instrument to compare treatment effects between therapies.

The purpose of this study was to systematically review and summarize the scientific literature evaluating changes in PRO scores after ACI. For analysis, we selected commonly used outcome instruments in cartilage repair studies, including the IKDC, Lysholm, MCKRS, KOOS, WOMAC, and SF-36. The outcome of interest was PRO responsiveness following ACI treatment. Meta-analyses of PRO score changes (“Hedge’s g effect sizes with 95% CIs) were plotted among instruments to visually reflect the responsiveness of each instrument at specified postoperative time points (TPs) forest plots of mean effect sizes for each instrument at each TP were used to provide a graphical representation of how responsive each instruments is to changes in self-reported knee function at varying postoperative TPs. A better understanding of the responsiveness of each instrument will allow for improved selection of outcome instruments in future cartilage research.

Methods

This nonregistered review was prepared in accordance with the “The PRISMA Statement for Reporting Systematic Reviews and Meta-Analyses of Studies that Evaluate Health Care Interventions.”²²

Evidence Acquisition

Search strategy

In February 2011, investigators conducted a systematic search of the literature using CINAHL (from 1981), Medline (from 1966), and SPORTDiscus (from 1800) to identify reports of PROs following autologous chondrocyte implantation/transplantation. Search terms used were autologous, chondrocyte, outcome, and knee. All abstracts were then reviewed for study inclusion/exclusion. In the event the abstract did not provide sufficient information to determine study eligibility, the full manuscript was reviewed. Additionally, the reference lists of all included studies were reviewed to identify other potentially eligible studies.

Selection criteria

All studies were required to meet the following inclusion criteria: (a) publication in English; (b) investigations with human participants; (c) prospective evaluation of patient outcomes following cell-based treatment of articular cartilage defects with some form of cultured autologous chondrocytes; (d) utilization of at least one of the following PRO instruments: IKDC, Lysholm, MCKRS as described by Browne et al.,¹⁴ KOOS, WOMAC, or SF-36 Physical Component Scale (SF-36 PCS) preoperatively and at a minimum of one postoperative TP; and (e) reporting of statistics from which effect sizes and 95% confidence intervals (CIs) could be calculated.

Assessment of Methodological Quality and Level of Evidence

The quality of all studies included was assessed using the Coleman Methodology Score modified by Kon, Verdonk, and others.^23,24 This assessment tool was specifically adapted to evaluate the quality of cartilage repair studies and includes 11 parameters on a 100-point scale (100 = highest quality): study sample size (10 points), average follow-up (10), number of concomitant surgical procedures (10), study design (15), description of the surgical procedure (5), description of postoperative rehabilitation (5), the inclusion of MRI outcome (10), the inclusion of histological outcome (10), outcome criteria (5), procedure for assessing clinical outcomes (7), and description of subject selection process (8).²⁴

Level of evidence was evaluated based on criteria from the Centre for Evidence Based Medicine (CEBM).²⁵ Using this taxonomy, the quality of the evidence for the studies included was determined and a grade of recommendation was generated for the use of each PRO as a measure of ACI treatment effect.²⁵ Methodological quality assessment and the rating of the level of evidence were assessed independently by two investigators. Discrepancies in scoring were discussed until a consensus score was agreed upon.

Data Extraction

The primary outcome variables of interest were scores on six specified PROs: the IKDC, Lysholm, MCKRS, KOOS, WOMAC, and SF-36 PCS. Because of the variation in Modified Cincinnati Knee Rating Systems reported in the literature, only the MCKRS presented by Browne et al.¹⁴ was reviewed. To avoid inappropriate comparisons of various versions of the MCKRS, the studies included had to have either published the scale in the article or provided a clear reference. From each study all data that could be used to calculate effect sizes for PROs were extracted.

For each outcome score, individual pre to postoperative standardized effect sizes were calculated using bias-corrected Hedge’s g for paired samples²⁶ with 95% confidence intervals (CIs) to examine the magnitude and precision of the difference between pre- and postoperative PRO scores. These effect sizes are unitless measures, corrected to represent a parametric distribution of the effects. For the purpose of this study, Hedge’s g effect sizes were used as a measure of responsiveness with larger effect sizes representing increased responsiveness of an instrument—that is, greater change in the instrument score over time.¹⁸ Separate meta-analyses were performed to provide a summary response for each PRO at specified TPs. For the purposes of analysis, follow-up TPs were grouped into four categories, TP-I (<1 year), TP-II (1 year to <2 years), TP-III (2 years to <4 years), and TP-IV (4 years or more). Most studies made multiple comparisons across separate TPs, and each comparison was treated independently during statistical analyses. If a study reported multiple results within a given TP category (e.g., 3 months and 6 months are both within the <1 year TP category), only the latest data point (i.e., 6 months) was analyzed. Therefore, within each study, only a single result per instrument was included for a given TP category. For each PRO, an additional meta-analysis to determine the overall responsiveness across all TPs was conducted using the pooled standardized effects averaged across all available TPs. For each meta-analysis, a random effects model was employed.²⁶

Effect sizes, 95% CIs, and Z-distribution P values were calculated in Comprehensive Meta Analysis (Comprehensive Meta Analysis Version 2.0, Biostat, Englewood, NJ). A positive effect size indicated improvement in postoperative PRO score compared with preoperative score. Effect sizes for which CIs did not overlap were considered to be substantially different. Effect size values were interpreted as small if they were between 0.20 and 0.49, moderate if between 0.50 and 0.79, and large if more than 0.80.

Assessment of Bias

Methodological bias was assessed using part B of the modified Coleman Methodology Score.²⁴ To assess the likelihood of publication bias, a funnel plot of all measures was generated by plotting standard error against Hedge’s g effect size for each included study. To assess the robustness of the observed overall effects of the variations in study design on PRO score, Orwin’s Fail-Safe N test was employed.²⁷ For this test, a Hedge’s g effect size of 0.1 was assumed for all missing studies, or studies excluded due to publication bias, and the number of missing studies necessary to reduce the overall mean effect size for each instrument to a 0.4 was calculated.

Results

Study Selection

The initial literature search yielded 216 results. Application of inclusion and exclusion criteria resulted in the inclusion of 42 articles.^14,28-68 Study selection and inclusion is depicted in fig 1. The included studies are summarized in Table 1. A total of 2,016 patients with a mean age of approximately 34.5 years are reported on across all studies. Overall, 16 studies reported outcomes using the IKDC, 11 used the KOOS (2 reporting only total KOOS scores), 18 used the Lysholm, 12 used the MCKRS, 9 studies used the SF-36 PCS, and 2 studies used the WOMAC. A single study reported on four instruments.⁵⁸ All other studies reported on three or fewer instruments.

Figure 1.

Study selection results.

Table 1.

Descriptive Variables for Autologous Chondrocyte Implantation Studies Included in Systematic Review and Meta-Analyses

Study	Mean Age	Procedure Includeda	Follow-up Time (years)	Instrument Included in Reviewb	Total N Analyzed	Lesion Locationsc	Average Lesion Size (or Largest Lesion Size) (cm2)	Level of Evidence25	Mod. Coleman Methodology Scored,24 (Part B)	Potential Conflict of interest Disclosed
Basad et al. 201028	33	MACI	0.5, 1.5, 2	L	39	FC, Troc, Pat		1b	45 (13)	No
Behrens et al. 200629	35	MACI	2.87, 5	L	33	MFC, LFC, Pat	4.08	2b	52 (13)	No
Bhosale et al. 200730	43	ACI-C w/Meniscus allograft transplant	1	L	8	MFC, LFC, Kissing	9.7 femoral, 3.7 tibial (median)	2b	47 (10)	No
Briggs et al. 200331	30	ACI-C	2.825	L	14	MFC, LFC, Troc, Pat	2.46	2b	44 (14)	No
Browne et al. 200514	37	ACI-P	5	M, MP, MS	87	MFC, LFC, Troc	4.9	2b	61 (17)	Yes
de Windt et al. 200932	35	ACI—multiple versions	3	K—Total only	25	MFC, LFC	3.25	2b	35 (13)	No
Della Villa et al. 201033	24.3	MACI in athletic compared to nonathletic males	1, 5	I	65	MFC, LFC, Troc	2.25	2b	49 (10)	No
Ebert et al. 200834	36.9	MACI w/accelerated rehabilitation or traditional rehabilitation	0.25	K, S	62	MFC, LFC	3.3	2b	56 (13)	No
Erggelet et al. 200035	33.7	ACI-P	0.5, 1	M	13	MFC, LFC, Troc, Pat	6.27	2b	36 (10)	No
Farr et al. 200736	36.9	ACI-P c/Meniscus allograft transplant	4.5	L, M	29	MFC, LFC, Kissing	MFC 6.36, LFC 5.35, troc 4.77, other 2.68	2b	46 (13)	Yes
Gobbi et al. 200637	30.5	MACI patellofemoral	2	I	32	Pat, Troc	4.7	2b	61 (13)	No
Gobbi et al. 200938	31.2	MACI patellofemoral	2, 6.29	I	34	Pat, Troc	4.45	2b	61 (13)	No
Henderson and Lavigne 200641	33.6	ACI-P patellofemoral with or without realignment	0.75, 1, 2	I, M, S	44	FC, Troc, Pat	3.07	2b	46 (13)	No
Henderson et al. 200539	41	ACI-P	1, 2	I	53	MFC, LFC, Troc, Pat	3.7	2b	58 (10)	No
Henderson et al. 200640	38.8	ACI-P with or without reoperation	3.52	I, M, S	170	MFC, LFC, Troc, Pat	3.45	2b	50 (10)	No
Horas et al. 200342	31.4	ACI-P	0.5, 1, 2	L	20	MFC, LFC, PFJ	3.86	2b	60 (11)	No
Knutsen et al. 200443	33.3	ACI-P	1, 2	L, S	39	MFC, LFC	5.1	1b	68 (17)	No
Kon et al. 200944	29	MACI	5	I	40	MFC, LFC, Troc	2.2	2b	51 (13)	No
Kreuz et al. 200945	35	MACI	0.5, 1, 4	I, K	19	MFC, LFC, Pat	4	2b	49 (10)	No
Mandelbaum et al. 200746	37.1	ACI-P	4.91	M, MP, MS	40	Troc	4.5	2b	44 (11)	Yes
Marcacci et al. 200547	37.6	MACI	1.41, 3.17	I	141	FC, Troc, Pat, TP	3.5	2b	49 (10)	Yes
McNickle et al. 200948	30.3	ACI-P	4.3	I, K, L	122	MFC, LFC, Troc, Pat, Troc and Pat	4.21	2b	52 (13)	Yes
Micheli et al. 200649	15.5	ACI-P	4.3	M, MP, MS	32	MFC, LFC	5.4	2b	35 (7)	Yes
Minas and Bryant 200550	36.9	ACI-P patellofemoral	3.95	M, S, W	45	Pat, Troc, Pat and Troc, FC and Pat, FC and Troc, FC and Pat and Troc	10.45	2b	56 (17)	Yes
Mithöfer et al. 200551	15.9	ACI-P among adolescent athletes	3.91	L	20	MFC, LFC, Troc, Pat, TP	6.4	2b	43 (10)	Yes
Moseley et al. 201053	37	ACI-P	9.2	M, MP, MS	72	MFC, LFC, Troc	5.2	2b	52 (13)	Yes
Nehrer et al. 200652	33	MACI	1, 3	I, L, M	36	MFC, LFC, Pat, TP	1.5-8 (range)	2b	39 (10)	No
Niemeyer et al. 2010 (a)54	39.4	ACI-C to those 40 and older and younger than 40	0.5, 2	I, L	74	MFC, LFC, Troc, Pat	Not reported	2b	42 (13)	No
Niemeyer et al. 2010 (b)55	37.4	ACI-C	0.5, 1	I, L	66	MFC, LFC, Troc, Pat	4.3	2b	49 (13)	No
Ochi et al. 200256	26.4	Atelocollagen-associated ACI with periosteom flap	2	L	28	MFC, LFC, Pat	2.93	2b	45 (10)	Yes
Ossendorf et al. 200757	36	MACI	0.5, 1, 2	K	40	MFC, LFC, PAT, Troc, TP	4.6	2b	61 (10)	Yes
Pascual-Garrido et al. 200958	31.8	ACI-P patellofemoral	4	I, K, L, S	52	PAT, Troc, bipolar, Troc and MFC	4.2	2b	46 (10)	Yes
Peterson et al. 201059	33.3	ACI-P	12.8	L	58	FC, Pat	7	2b	39 (10)	No
Robertson et al. 200760	37.4	ACI-C	0.5, 1, 2	K	27	MFC, LFC, Pat	1-10 (range)	2b	56 (13)	No
Rosenberger et al. 200861	48.6	ACI-P over age 45	2, 3	M, S, W	56	MFC, LFC, Troc, Pat, MTP, LTP	4.7	2b	61 (22)	Yes
Rue et al. 200862	23.4	ACI-P w/Meniscal allograft transplant	2	I, K, L	15	MFC, LFC	3.93	2b	43 (10)	Yes
Saris et al. 200864	33.9	CCI	0.5, 1.5	K	51	FC	2.6	1b	68 (17)	Yes
Saris et al. 200963	33.9	CCI	3	K—Total only	41	FC	2.6	1b	62 (13)	Yes
Selmi et al. 200865	30	MACI	1, 2	I	17	FC	3	2b	64 (17)	No
Tohyama et al. 200966	>20	Atelocollagen-associated ACI with periosteom flap	0.5, 1, 2	L	27	MFC, LFC, Pat	3.2	2b	39 (13)	Yes
Zaslav et al. 200967	34.5	ACI-P following prior failed treatment w/n previous 3 years	0.5, 1, 3, 4	M, K, S	150	MFC, LFC, Troc	4.63	2b	52 (18)	Yes
Zeifang et al. 201068	29.1	ACI-P, MACI	1	I, L, S	21	MFC, LFC	4.20	2b	67 (13)	Yes

^aMACI: Matrix Assisted Autologous Chondrocyte Implantation, ACI-C Collagen covered Autologous Chondrocyte Implantation, ACI-P: Periostoum covered Autologous Chondrocyte Implantation; CCI: Characterized Chondrocyte Implantation.

^bI: International Knee Documentation Committee Subjective Knee Form (IKDC), L: Lysholm Knee Scale (Lysholm), K: Knee Injury and Osteoarthritis Outcome Score (KOOS), M: modified Cincinnati Knee Rating System (MCKRS) Patient Perspective: MP:MCKRS – Pain Scale; MS- MCKRS Swelling Scale, S: Medical Outcomes Study 36-Item Short Form Health Survey Physical Component Scale (SF-36 PCS), W: Western Ontario and McMaster Universities Osteoarthritis Index (WOMAC).

^cFC: Femoral Condyle, Troc: Trochlear, Pat: Patellar, MFC: Medial Femoral Condyle, LFC: Lateral Femoral Condyle, TP: Tibial Plateau, PFJ: Patellofemoral Joint, MTP: Medial Tibial Plateau, LTP: Lateral Tibial Plateau.

^dScored 0 to 100 with 100 representing best methodology; Part B subscore is presented as an assessment of study bias scored 0 to 22 with 22 representing least bias.

Methodology Scoring and Level of Evidence

The mean modified Coleman Methodology Score for all included articles was 50.9 ± 9.2 (range = 35-68). The mean modified Coleman Methodology Score for studies using each PRO instrument was as follows: IKDC 51.4 (standard deviation [SD] = 7.5), KOOS-Sports 51.9 (5.8), KOOS-all other subscales 53.7 (7.7), Lysholm 49.2 (8.8), MCKRS 48.2 (8.8), and SF-36 PCS 56.2 (7.8). The least reported parameters were inclusion of MRI and histological outcomes and description of the subject selection process. CEBM level of evidence was 2b for 38 articles and 1b for 4 articles included. Based on the consistent reporting of level 2 studies, a grade B recommendation was made for the use of the IKDC, KOOS, Lysholm, MCKRS, SF-36 PCS, and WOMAC as outcome measures following ACI.²⁵

Assessment of Bias

The mean score for Part B of the modified Coleman Methodology Score assessing individual study bias was 13 ± 3 out of a possible 22 points. The assessment of publication bias revealed an asymmetrical distribution of studies with a disproportionate number of studies above the mean effect size at the bottom of the funnel (Fig. 2). This indicated a slight publication bias toward studies demonstrating large treatment effects, particularly for studies with smaller sample sizes. However, the results of Orwin’s Fail Safe N test (Table 2) demonstrate that an additional 14 (SF-36 PCS) to 196 (KOOS) studies with a trivial effect size of 0.10 are necessary to reduce the mean effect size for any of the PROs to a weak value of 0.40. Therefore, the observed overall effects are very robust and not likely to be artificially influenced by this potential publication bias.

Figure 2.

Funnel plot to evaluate publication bias. The observed funnel plot suggests a slight publication bias toward studies demonstrating larger effect sizes, with an asymmetrical distribution of studies at the bottom of the funnel.

Table 2.

Orwin’s Fail Safe N Analysis to Evaluate Publication Bias

Instrument	Na
IKDC	95
Lysholm	83
KOOS	196
MCKRS	48
SF-36 PCS	14
Overall across all instruments	399

IKDC = International Knee Documentation Committee Subjective Knee Form; Lysholm = Lysholm Knee Scale; KOOS = Knee Injury and Osteoarthritis Outcome Score; MCKRS = Modified Cincinnati Knee Rating System; SF-36 PCS = Medical Outcomes Study 36-Item Short Form Health Survey Physical Component Scale.

^aNumber of studies with an effect size of 0.1 needed to reduce the overall mean effect size to 0.4.

Responsiveness of PROs

Mean effect sizes and 95% CIs for each instrument at each of the four TPs are reported in the forest plots in fig 3. For an instrument to be included in the meta-analysis at a given TP, a minimum of four individual data points must have been reported. The WOMAC did not meet this requirement at any time point, and the SF-36 PCS only met this requirement at TP-III. The MCKRS could only be evaluated at TP-III and TP-IV and only the patient perception scale could be evaluated.

Figure 3.

Effect sizes by patient-reported outcome instrument. Random effects model summary mean effect sizes for each patient-reported outcome instrument by time point. IKDC = International Knee Documentation Committee Subjective Knee Form; Lysholm = Lysholm Knee Scale; KOOS-ADL = Knee Injury and Osteoarthritis Outcome Score Activities of Daily Living subscale; KOOS-pain = Knee Injury and Osteoarthritis Outcome Score Pain subscale; KOOS-QOL = Knee Injury and Osteoarthritis Outcome Score Quality of Life subscale; KOOS-Sports = Knee Injury and Osteoarthritis Outcome Score Sports and Recreation subscale; KOOS-Symptoms = Knee Injury and Osteoarthritis Outcome Score Symptom subscale; MCKRS Patient = Modified Cincinnati Knee Rating System Patient Perspective; SF-36 PCS = Medical Outcomes Study 36-Item Short Form Health Survey Physical Component Scale; WOMAC = Western Ontario and McMaster Universities Osteoarthritis Index.

Responsiveness by Instruments across TPs

For all evaluated instruments, none of the mean effect sizes or CIs encompassed zero. This indicated that there was evidence of positive treatment effects following ACI regardless of the PRO used (Fig. 3). The IKDC increased responsiveness over time, with TP-IV demonstrating a greater mean effect size (mean effect size [95% CI]: 1.78 [1.33, 2.24]) than TP-I (0.88 [0.69, 1.07]). The responsiveness of the Lysholm varied little across TPs, with mean effect sizes only ranging from 1.29 to 1.69. There was also no difference in responsiveness for the MCKRS between TP-II and TP-III. Finally, the only KOOS subscale to show noticeable improvements in responsiveness over time was the KOOS-sports and recreation subscale (KOOS-Sports) for which TP-III (1.76 [0.87, 2.64]) and TP-IV (0.98 [0.81, 1.15]) demonstrated larger effect sizes than TP-I (0.61 [0.44, 0.78]). Effect sizes for the remaining KOOS subscales did not change over time and fell in the following ranges: KOOS-Activities of Daily Living 0.78 [0.27, 1.29] to 1.90 [1.02, 2.78], KOOS-Pain subscale 0.75 [0.40, 1.10] to 1.88 [1.12, 2.63], KOOS-Quality of Life 0.88 [0.32, 1.44] to 2.38 [1.20, 3.56], and KOOS-Symptoms 0.75 [0.50, 1.00] to 1.60 [0.79, 2.41].

Responsiveness by TP

At TP-I, the Lysholm (1.52 [0.92, 2.11]) appears more responsive than the KOOS-Sports subscale (0.61 [0.44, 0.78]). At TP-II, both the IKDC (1.37 [0.93, 1.80]) and the Lysholm (1.53 [0.96, 2.11]) were more responsive than the KOOS-Sports subscale (0.57 [0.23, 0.92]). There were no identifiable differences between any of the instruments at TP-III. Finally, at TP-IV the IKDC (1.78, [1.33, 2.24]) demonstrated a larger effect size than the KOOS-Sports subscale (0.98 [0.81, 1.15]).

Overall Responsiveness

The final comparison was of the overall responsiveness of each instrument with data from all available TPs averaged (Fig. 4). Both the Lysholm (1.43 [1.14, 1.72]) and the IKDC (1.37 [1.13, 1.62]) had overall mean effect sizes that were greater than the overall mean effect size for the KOOS-Sports subscale (0.90 [0.73, 1.07]) and SF-36 PCS (0.78 [0.52, 1.05]). CIs for all other instruments and all other KOOS subscales can be observed to overlap, suggesting no differences in responsiveness.

Figure 4.

Overall effects sizes for each patient-reported outcome. Random effects model summary mean effect sizes for each patient-reported outcome instrument across all time points combined. IKDC = International Knee Documentation Committee Subjective Knee Form; Lysholm = Lysholm Knee Scale; KOOS-ADL = Knee Injury and Osteoarthritis Outcome Score Activities of Daily Living subscale; KOOS-Pain = Knee Injury and Osteoarthritis Outcome Score Pain subscale; KOOS-QOL = Knee Injury and Osteoarthritis Outcome Score Quality of Life subscale; KOOS-Sports = Knee Injury and Osteoarthritis Outcome Score Sports and Recreation subscale; KOOS-Symptoms = Knee Injury and Osteoarthritis Outcome Score Symptom subscale; MCKRS Patient = Modified Cincinnati Knee Rating System Patient Perspective; SF-36 PCS = Medical Outcomes Study 36-Item Short Form Health Survey Physical Component Scale; WOMAC = Western Ontario and McMaster Universities Osteoarthritis Index.

Discussion

Our purpose was to evaluate the responsiveness of common PROs to the treatment effects of ACI. An underlying assumption was that ACI would have a common effect across studies and varying ACI procedures. Evaluating ACI efficacy was not a purpose of this review, and the results are in agreement with previous reviews documenting ACI to be a viable procedure resulting in positive patient outcomes.^24,69,70 The large mean effect sizes and narrow CIs observed in this review support the use of ACI for the generalized treatment of articular cartilage defects.

Responsiveness

The results of this review demonstrate that regardless of the duration of postoperative follow-up all instruments were responsive to patient improvement; however, the IKDC and Lysholm may be more responsive than the MCKRS, KOOS, or SF-36 PCS. There was insufficient data to adequately evaluate the WOMAC.

Responsiveness of PROs across TPs

The Lysholm demonstrated large mean effect sizes (1.30-1.70) with little variation across time (Fig. 2). The observed CIs for the Lysholm at all TPs overlap by more than 50%, suggesting little changes in responsiveness as time since ACI progresses. Common rehabilitation recommendations following ACI restrict return to sports participation for 12 to 18 months following surgery.^71-73 This delayed return to physical activity may result in lower scores on instruments that emphasize higher sport demands. Because the Lysholm primarily assesses everyday activities (walking, squatting, stair-climbing) and does not address sports activity, delayed return to higher level physical activity has little influence on Lysholm score. Therefore, the Lysholm scale may be ideal for evaluating short-term outcomes or outcomes among patients not intending to return to sports, but less responsive to changes during long-term recovery as individuals return to higher demand activities. Additionally, it should be noted that some authors and investigations have called into question the current weighting system for Lysholm items.^74,75

The IKDC also demonstrated large effect sizes. With a noticeable increase in mean effect size observed between TP-I and TP-IV, with mean effect size increasing from 0.88 to 1.78 with no overlap between CIs. This difference demonstrates increased treatment effects over time when evaluating outcomes with the IKDC. Greco et al.²⁰ observed a similar trend with responsiveness of the IKDC increasing between 6 and 12 months in a cohort of surgical cartilage patients. It has previously been reported^76-78 that functional and structural improvements following cartilage repair continue beyond 1 year postoperatively. The observed increases in mean effect size over time may represent the IKDC’s responsiveness to continual improvements in function that occur in the years following ACI. The responsiveness of the IKDC to continued improvements over time can be considered a strength of this instrument and may be due to its inclusion of sporting activities. The IKDC allows for continued improvement as individuals initiate return to strenuous activity and sports participation beyond 1 year postoperatively.

The KOOS-Sports subscale had the lowest mean effect at TP-I and TP-II, whereas the KOOS-Symptoms subscale had the lowest mean effect of all the KOOS scales at TP-III and TP-IV. Effect sizes for the KOOS-Sports subscale were lower at TP-I compared with TP-III and TP-IV. These results are similar to those observed with the IKDC, and this progressive increase in effect sizes over time may be related to the slow, progressive return to sports following ACI. For all other KOOS subscales no changes were seen for mean effect size between TPs. Overall, the KOOS was responsive to changes following ACI; however, the KOOS-Sports subscale was the only subscale to demonstrate increasing responsiveness over time, suggesting that it responded to increasing treatment effects as healing progressed and may be more sensitive to improvements in function among active individuals than other instruments or KOOS subscales.

There were only sufficient data to evaluate the MCKRS at TP-III and TP-IV, limiting any conclusions that can be drawn regarding the changes in its effect sizes over time. Our results suggest the MCKRS is responsive to changes in patient function following ACI; however, caution is urged regarding the use of this instrument. Many different versions of the MCKRS exist and many authors fail to reference the version of MCKRS they use. Several articles were excluded, at least in part, because the authors did not reference the version of the MCKRS used, or because a different version than the one presented by Browne et al.¹⁴ was used as an outcome measure.^76,79-87 Because of ambiguity regarding the use of “modified” Cincinnati Knee Rating Systems, the developers of the original Cincinnati Knee Rating Scale discourage the use of any modified versions.⁸⁸ However, because of the frequency with which the Browne et al.¹⁴ version of the MCKRS has been clearly referenced in ACI outcomes studies, it was chosen for inclusion in this review.

Both the SF-36 PCS and the WOMAC had limited data available for analysis. For the SF-36 PCS, there was only sufficient data for analysis of responsiveness at TP-III. For this TP, the SF-36 PCS did demonstrate a positive mean effect 2 to 4 years following ACI with an effects size of 0.92 [0.55, 1.28]. There were insufficient data to include the WOMAC in any of the meta-analyses performed.^50,61 Although additional studies have included the WOMAC as an outcome measure, the results were only reported using nonparametric statistics and/or without the reporting of means and standard deviations or other data necessary for calculating effect sizes.^89-92 As a result no clear conclusions regarding the responsiveness of the WOMAC as an outcome instrument can be reached based on this review.

Responsiveness between PROs

The forest plots of PRO instruments for each TP can be seen in fig 3, whereas the overall mean effect sizes averaged across all TPs can be seen in fig 4. The IKDC and the KOOS-Sports were the only instruments to demonstrate changes in effect sizes over time. These changes may be related to activity restriction and gradual return to sports following ACI. The restrictions on sporting activity during the first year post-ACI may also explain the significant differences observed between the responsiveness of the KOOS-Sports and the Lysholm at TP-I and TP-II. At TP-II and TP-IV, the IKDC appears more responsive than the KOOS-Sports. These differences may be the result of the wider range of physical functioning addressed in the IKDC as compared with the KOOS-Sports. The responsiveness of the MCKRS was not different from any other instrument evaluated at both TP-III and TP-IV. The SF-36 PCS had the lowest responsiveness overall and at TP-III. This finding is not surprising as the SF-36 was the only included instrument not specifically designed for the knee.

The Lysholm and the IKDC demonstrated the largest overall effect sizes, regardless of TP. These had appreciably greater responsiveness than the KOOS-Sports subscale and the SF-36 PCS (Fig. 4). Should investigators or clinicians wish to explore patient outcomes for individual constructs (quality of life, activities of daily living, sports, etc.), the KOOS via its subscales is the only instrument that allows for this multifaceted investigation, and along with the IKDC has been recommended for use by the International Cartilage Repair Society.⁷⁵ Although both the KOOS and IKDC include sports participation as components of evaluating knee function, the IKDC appears more responsive to overall changes in function following ACI (Fig. 4). This overall difference, combined with the observed differences in responsiveness between the IKDC and KOOS-Sports subscales at TP-II and TP-IV, leads us to propose that the IKDC may be the preferred outcome instrument for evaluating long-term outcomes following ACI, particularly among patients whose goals include return to sporting activity. Although all KOOS subscales are responsive to treatment effects following ACI, the IKDC and Lysholm are shorter instruments with single score outcomes and overall are more responsive to change than some subscales included in the KOOS. Based on these observations, the IKDC and the Lysholm may be preferable to the KOOS for documenting treatment effects following ACI.

Study Quality

The mean modified Coleman Methodology Score (50.9 ± 9.2) among studies was comparable to other recent reviews of ACI and other cartilage repair procedures.^44,69,93 Although the modified Coleman Methodology Score provides a set of standardized criteria by which to evaluate cartilage research, it is not without limitations. The scale is heavily weighted toward diagnostic, clinician-oriented outcomes, with up to 25% of the score dependent on MRI and histological evaluation. The relationship between MRI and clinical outcomes is not definitive; some authors observed low to moderate correlations between MRIs and PROs,^60,94,95 and others failed to observe such a relationship.^96,97 Similarly, histological analysis can involve a wide variety of techniques and may not be ethical in cases where reoperation is not otherwise indicated. Of the 42 studies included, only a single study⁶⁵ received full credit for both histological and MRI outcomes, suggesting that the requirement of these outcomes may not be applicable in a clinical research setting. Furthermore, only five studies scored a full 10 points for >90% of subjects undergoing one surgical procedure with <10% undergoing concomitant procedures.^{29,42,43,62,65} Notably, although concomitant procedures reduced the overall methodological score, studies that include concomitant procedures are more generalizable to real clinical practice than studies of single isolated defects.⁹⁸

In future research, more well-designed, well-documented, high-level clinical trials that use PROs with comprehensive data reporting are needed. Adopting uniform methodological reporting requirements for cartilage repair studies will improve the quality of the body of literature in this area. This review may provide a basis for this effort.

Limitations

The results of this review are limited by the quantity, quality, and strength of the studies and PROs selected for inclusion. Any recommendations made are based solely on the available evidence, and it should be noted that the IKDC and Lysholm were used in the literature more often than other instruments, strengthening the validity of recommendations regarding these two instruments and limiting our ability to draw conclusions regarding other PROs. As evidenced by the low modified Coleman Methodology Score observed in this review and others, the quality of reporting in cartilage outcomes studies is variable and generally poor. A random effects analysis was used to account for the variability between studies allowing our results to be generalized to a broad clinical population.

A statistical limitation of our study is the use of multiple measures at multiple TPs from within the same study populations. We acknowledge that outcome scores obtained from within the same sample are likely correlated, but given that the correlation between outcome measures and TPs is rarely reported and no studies documented all instruments at all TPs, correction for this relationship was not feasible. Fortunately, the observed mean effect sizes are so large and the CIs so small for the included outcome instruments that we do not believe this assumption violation significantly influences the overall conclusions of this review.

Conclusions

Evidence for the use of ACI as a treatment for chondral defects consists primarily of level 2b observational cohort studies. The methodological quality of many of these studies is limited by the absence of diagnostic outcomes such as MRI and histological analyses, small sample size, short follow-up, and high frequency of concomitant procedures. In addition, documentation of recruitment rate and investigator independence was lacking from many studies. The IKDC, Lysholm, KOOS, MCKRS, and SF-36 PCS were all responsive to improvements in function following ACI. A positive treatment effect for ACI was observed using all instruments with follow-up ranging from <1 year to beyond 4 years. The Lysholm and the IKDC were the most responsive instruments across time. The Lysholm was highly responsive as early as <1 year following ACI and was consistently responsive throughout follow-up. However, this instrument may not be responsive to changes in function associated with the resumption of higher demand activities such as sports that occurs after 1 year. For the evaluation of long-term outcomes among patients with intent to return to physical activity, this review supports the use of the IKDC, which was able to detect increasing treatment effects overtime. The use of the Lysholm and IKDC together represents a responsive combination of PRO instruments that are able to efficiently document both short-term and long-term treatment effects among patients of a variety of activity levels following ACI.