Influence of response shift on early patient-reported outcomes following autologous chondrocyte implantation

Abstract

Purpose

Response shift is the phenomenon by which an individual's standards for evaluation change over time. The purpose of this study was to determine whether patients undergoing autologous chondrocyte implantation (ACI) experience response shift.

Methods

Forty-eight patients undergoing ACI participated. The “then-test” method was used to evaluate response shift in commonly used patient-reported outcome measures (PROMs)—the SF-36 Physical Component Scale (SF-36 PCS), WOMAC, IKDC, and Lysholm. Each PROM was completed pre- and 6 and 12 months post-surgery. At 6 and 12 months, an additional “then” version of each form was also completed. The “then” version was identical to the original except that patients were instructed to assess how they were prior to ACI. Traditional change, response shift adjusted change, and response shift magnitude were calculated at 6 and 12 months. T tests (p < 0.05) were used to compare traditional change to response-shift-adjusted change, and response shift magnitude values to previously established minimal detectable change.

Results

There were no differences between traditional change and response-shift-adjusted change for any of the PROMs. The mean response shift magnitude value of the WOMAC at 6 months (15 ± 14, p = 0.047) was greater than the previously established minimal detectable change (10.9). The mean response shift magnitude value for the SF-36 PCS at 12 months (9.4 ± 6.8, p = 0.017) also exceeded the previously established minimal detectable change (6.6).

Conclusions

There was no evidence of a group-level effect for response shift. These results support the validity of pre-test/post-test research designs in evaluating treatment effects. However, there is evidence that response shifts may occur on a patient-by-patient basis, and scores on the WOMAC and SF-36 in particular may be influenced by response shift.

Level of evidence

II.

Introduction

To assess function or health-related quality of life (HRQL), patients are often asked to evaluate their well-being using a self-report instrument to document patient-reported outcome measures (PROMs). However, PROMs may be influenced by response shift [44]. Response shift is the phenomenon by which an individual's self-evaluation of a construct changes due to a change in internal standards of measurement (recalibration), a change in values or priorities (reprioritization), or a personal redefinition of the target construct (reconceptualization) [48]. Response shift may interfere with the ability to accurately detect changes in patient's health. Response shift has been observed among terminal illness and chronic disease patients where physical health deteriorates, yet their self-reported HRQL remains stable [16, 34, 45, 47, 50, 53, 55]. It has been hypothesized that these changes may be a result of changing values, standards, and priorities [44]. Additionally, response shift has been previously observed among knee arthroplasty and microfracture patients [1, 37, 38, 57].

While early results for autologous chondrocyte implantation (ACI) [5] outcomes are promising, the existing literature primarily reports outcomes using PROMs [4, 6–9, 11–14, 17, 23–25, 27, 31–33, 36, 41, 52, 56]. Although PROMs are used frequently in the orthopaedic literature, the traditional pre–post-test research designs used may be influenced by response shift phenomenon. In particular, the extended preparation and rehabilitation required for ACI, combined with the inherent expectations associated with surgery, may make patients prone to response shift [39]. If the PROMs frequently used to evaluate ACI outcomes are subject to a response shift, then reported outcomes may under- or over-estimate the effectiveness of existing articular cartilage treatments, calling into question the validity of much of the existing ACI outcomes literature.

The purpose of this study was to determine whether patients undergoing ACI experience response shift. The International Knee Documentation Committee (IKDC) Subjective Knee Evaluation Form, the Western Ontario and McMaster Osteoarthritis Index (WOMAC), and the medical outcomes study 36-item short-form health survey Physical Component Scale (SF-36 PCS) rely heavily on subjective evaluations of function and pain levels, which may be influenced by response shift. Therefore, it was hypothesized that these common ACI outcome instruments would demonstrate the evidence of a response shift. In contrast, the Lysholm Knee Scale (Lysholm) focuses on the capacity to perform specific tasks rather than the ease or pain associated with task performance; therefore, it was hypothesized that Lysholm scores would not be influenced by response shift.

Materials and methods

Patients were prospectively recruited from an active cartilage centre within a public university affiliated sports medicine clinic. Inclusion criteria were the following: planned ACI surgery to the knee, willingness to participate, and no uncorrectable contraindications to ACI such as extensive degenerative joint disease, insufficient meniscus, or unstable knee. There were no exclusions based on the limb malalignment if the malalignment was corrected prior to or at the time of surgery via high tibial osteotomy or tibial tubercle transfer. Patients undergoing concomitant meniscal transplant were excluded. A total of 56 consecutive patients who met eligibility requirements were approached to participate in this study (Fig. 1). The final participating enrolment for this study was 48 patients (29 males, 19 females, 35.1 ± 8.0 yrs, 180.7 ± 31.7 cm, 92.4 ± 20.3 kg). Among these patients, 24 underwent ACI to the patellofemoral joint with a tibial tubercle transfer, 2 underwent ACI to the lateral femoral condyle, and the remaining 22 underwent ACI to the femoral condyle, of which 4 also had a concomitant high tibial osteotomy. The mean number of defects treated per patient was 1.5 ± 0.6 with an average treatment area of 8.7 ± 6.9 cm² (range 1.9–39.0 cm²) as measured intraoperatively. All participants signed a university-approved institutional review board consent form.

Fig. 1.

Study enrolment and follow-up

Surgical procedures and rehabilitation

All patients underwent a two-step ACI procedure performed by the same surgeon (XX). During the first procedure, a limited chondroplasty was performed and the lesion was evaluated arthroscopically. At this time, a biopsy was obtained from the intercondylar notch (100–200 mg cartilage). This sample was sent to a commercial laboratory where it was cultured and expanded (Carticel, Genzyme Corp., Cambridge, MA). In a second surgical procedure, chondrocyte implantation was performed using a formal arthrotomy. First, the defect or defects were prepared using a curette to debride down to the subchondral plate with stable edges. A type I/III collagen membrane (Geistlich Bio-Gide^(R), Geistlich Pharma North America Inc., Princeton, New Jersey) was shaped to match the defect. Sutures and fibrin glue (Tisseel, Baxter Healthcare Corp., Deerfield, IL) were used to adhere the membrane over the defect to form a water-tight seal. The chondrocytes in suspension were then injected beneath the membrane into the defect through a small portal remaining at the edge of the collagen membrane. The portal was then closed and sealed with sutures and additional fibrin glue.

All patients followed standardized rehabilitation protocols following surgery [26]. All patients were braced in full extension and were non-weight bearing for 2 weeks postoperatively. Toe-touch weight bearing was permitted from 2 to 4 weeks with partial weight bearing from 4 to 6 weeks and progression to full weight bearing between weeks 6 and 12. Continuous passive motion was prescribed for all patients for 6–8 h per day for 6 weeks. For defects in the tibiofemoral joint, knee braces were gradually unlocked between 2 and 4 weeks as quadriceps control was gained. For defects to the patellofemoral joint, knees were braced in full extension for weight bearing through 4 weeks postoperative and then were gradually unlocked as quadriceps control was gained between weeks 4 and 6. Once good quadriceps control was gained, all patients were transitioned to a hinged knee sleeve. All patients were recommended to abstain from high-intensity cutting or pivoting activity until at least 12 months post-ACI.

Outcome measures

Patient-reported outcomes

The PROMs used in this study were the SF-36 PCS [29, 30, 54], the WOMAC [3], the IKDC [21], and the Lysholm [49]. Reliability among cartilage patients has been previously evaluated for each of these instruments [3, 21, 22, 28, 29, 40]. A researcher independent of the treating physician reviewed each instrument with the patients and was available to answer any questions they may have had. All PROMs were completed prior to implantation and at 6 and 12 months post-surgery.

Assessment of response shift

One of the most common statistical approaches for measuring response shift is the then-test method (Fig. 2) [18, 19]. This approach is identical to a traditional pre-test/post-test method with the exception that subjects complete an additional “then-test” assessment at the same session as their post-test assessment. For the then-test, subjects are instructed to assess how they were at the time of the pre-test, prior to the intervention. The rationale for this design is that by completing the then-test and the post-test at the same time, subjects will provide responses utilizing the same frame of reference and calibration standards for both. In a pre-/post-design, traditional change (TC) is the difference between post-test and pre-test scores and is the only variable of interest. With the then-test method, response shift is calculated as the difference between the then-test and the pre-test and the response-shift-adjusted change (RSAC) is considered to be the difference between the post-test and the then-test.

Fig. 2.

Then-test method for assessing response shift. For the then-test method, patients are requested to complete an outcome instrument three times. First pre-treatment (pre-test), again at a specified post-treatment time point (post-test), and at that same post-treatment time point, they also complete a then-test on which they are asked to retrospectively rate how they were at the pre-treatment time point. From these three scores, response shift, response shift magnitude, traditional change, and response-shift-adjusted change can then be calculated. In the present study, post and then evaluations were completed at 6 and 12 months postoperatively

Statistical analysis

Apriori power analysis using previously published 12-month response shift values in orthopaedic knee patients [37] (with PROM standard deviation values that were comparable to those previously observed in our own internal cartilage and ligament patient registry) demonstrated a need to enrol 35 patients to achieve sufficient power (0.80) to detect a response shift with α = 0.05.

Main outcome measures

The dependent variables of response shift, response shift magnitude, TC, and RSAC were calculated for the IKDC, Lysholm, SF-36 PCS, and WOMAC from pre-operation to 6 and 12 months post-ACI as described in Fig. 2.

Group-level effect

To investigate the occurrence of a group-level response shift, paired t tests were used to compare then-test with pre-test scores and to compare TC with RSAC for each PROM. Significant t test results (p < 0.05) would support the occurrence of a group-level effect with a consistent response shift occurring across patients.

Individual-level effect

To investigate the occurrence of an individual-level (patient-by-patient) effect for response shift, response shift magnitude was calculated as the absolute value of the response shift for each PROM. One-sample t tests were used to compare the response shift magnitude with previously established minimal detectable changes (MDCs) for each PROM instrument (p < 0.05). The MDC at 6- and 12-month follow-up has been previously established among patients post-ACI for the IKDC (15.6 points at 6 months; 13.7 points at 12 months), WOMAC (10.9, 15.3), and SF-36 PCS (8.3, 6.6) [15]. For the Lysholm scale, an MDC of 14 was calculated from previously published reliability and ICC values among patients awaiting surgery for chondral defects [2, 22].

Results

Study enrolment and follow-up is presented in Fig. 1. Main outcome measures are reported in Table 1. No group-level effect for response shift was observed. There were no differences between pre-test and then-test scores for any of the PROMs evaluated. There were also no differences between RSAC and TC, and none of the mean response shift values exceeded previously established MDC values.

Table 1.

Main outcome variables for response shift among autologous chondrocyte implantation patients

Measure	IKDC		Lysholm		SF-36 PCS		WOMAC
	Mean (95 % CI)	p values*	Mean (95 % CI)	p values	Mean (95 % CI)	p values	Mean (95 % CI)	p values
Pre-test	38.1 (34.5, 41.6)		46 (41, 51)		37.5 (35.0, 40.0)		33 (29, 38)
Post-test 6 months	51.1 (45.6, 56.6)		62 (55, 69)		43.5 (40.7, 46.3)		22 (17, 28)
Post-test 12 months	56.2 (46.5, 65.9)		65 (58, 73)		43.8 (40.1, 47.4)		20 (14, 27)
Then-test at 6 months	39.9 (33.9, 45.8)	n.s.	41 (36, 47)	n.s.	38.7 (35.5, 41.8)	n.s.	36 (39, 44)	n.s
Then-test at 12 months	39.4 (33.5, 45.2)	n.s.	43 (37, 48)	n.s.	39.3 (36.1, 42.5)	n.s.	35 (29, 41)	n.s
Response shift at 6 months	1.4 (–4.4, 7.2)		–5 (–11, 1)		1.0 (-2.2, 4.2)		5 (–2, 11)
Response shift at 12 months	0.0 (–6.3, 6.4)		–4 (–10, 1)		1.4 (-2.3, 5.1)		2 (–3, 8)
Response shift magnitude at 6 months	13.65 (9.7, 17.6)	n.s.	15 (11, 19)	n.s.	8.3 (6.3, 10.2)	n.s.	15† (11, 20)	0.047
Response shift magnitude at 12 months	13.64 (10.0, 17.3)	n.s.	14 (10, 17)	n.s.	9.4† (7.2, 11.5)	0.017	13 (9, 16)	n.s.
Traditional change at 6 months	13.3 (8.1, 18.4)		16 (11, 22)		6.3 (3.5, 9.0)		–10 (–15, –6)
Traditional change at 12 months	17.6 (12.1, 23.1)		19 (13, 25)		5.9 (3.4, 8.5)		–13 (–17, –8)
Response-shift-adjusted change at 6 months	11.7 (3.3, 20.0)	n.s.	21 (14, 28)	n.s.	4.9 (0.6, 9.2)	n.s.	–15 (–23, –6)	n.s.
Response-shift-adjusted change at 12 months	17.3 (8.1, 26.4)	n.s.	24 (16, 31)	n.s.	4.5 (0.6, 8.4)	n.s.	–15 (–21, –9)	n.s.
Minimal detectable change at 6 months	15.6 [15]		14 [22]		8.3 [15]		10.9 [15]
Minimal detectable change at 12 months	13.7 [15]		14 [22]		6.6 [15]		15.3 [15]

^*Comparisons made for each instrument at both 6 and 12 months: then-tests to pre-tests; response shift magnitude to minimal detectable change; traditional change to response-shift-adjusted change; n.s. [non-significant (p > 0.05)]

^†p < .05

Response shift = then-test – pre-test

Response shift magnitude = absolute value of response shift

Traditional change = post-test – pre-test

Response shift adjusted change = post-test – then-test

Individual-level analysis

Response shift magnitude values were used to determine the number of subjects that experienced a response shift beyond the MDC at the 6- and 12-month time points for each PROM instrument. At 6 months, it was observed that there was a response shift beyond the MDC for 17 patients assessed via the IKDC, 16 patients for the SF-36 PCS, 15 patients for the Lysholm, and 23 patients for the WOMAC. At the 12-month time point, 10 patients for the IKDC, 23 patients for the SF-36 PCS, 17 patients for the Lysholm, and 14 patients for the WOMAC experienced response shifts that exceeded the MDC. Overall, 13 patients at 6 months and 7 patients at 12 months demonstrated the evidence of a response shift on at least 3 of the four instruments utilized. The only PROMs to show a significant response shift at an individual level across patients were the total WOMAC score at 6 months and the SF-36 PCS at 12 months. The mean response shift magnitude value for the WOMAC at 6 months was 15 ± 14, which was significantly greater than the MDC over 6 months of 10.9 established by Greco et al. [15] (p = 0.047). The mean response shift magnitude value for the SF-36 PCS (9.4 ± 6.8) at 12 months also exceeded the previously established MDC (6.6) [15] over a 12-month follow-up (p = 0.017).

Discussion

The key finding of the present study was that traditional pre–post-test methods for evaluating PROMs appear valid for assessing group-level treatment effects following ACI. No group-level effects for response shift were observed. These results fail to support the hypothesis that response shift would be evident for the IKDC, SF-36 PCS, and WOMAC, but the results do support the hypothesis that no response shift would be observed for the Lysholm.

A significant difference between pre-test and then-test scores for the WOMAC [37, 38] and the SF-36 PCS [37] has been previously reported at 6 and 12 months following knee arthroplasty. Similarly, a response shift was reported using the Lysholm scale among patients with a median of 34 months following knee microfracture [1]. Upon initial review, our failure to observe a group-level response shift is in disagreement with the previous work [1, 37, 38] in orthopaedic knee patients. However, upon further examination, the values observed in the present study are very similar to those reported elsewhere. In the present study, mean RS values of –5 ± 19 and −4 ± 18 for the Lysholm were observed compared to a median RS of –7 by Balain et al. [1] Similarly, we observed mean RS values of 5 ± 20 and 2 ± 17 for the WOMAC and 1.0 ± 10.4 and 1.4 ± 11.6 for the SF-36 PCS, compared with previously reported mean WOMAC RS values of 3.8 ± 19.5, 5.5 ± 16.9, and 6.7 ± 15.5 and SF-36 PCS values of –1.7 ± 8.1 and –3.2 ± 7.9 [37, 38]. In all cases, the mean or median differences between then-test and pre-test scores were less than the previously established MDC scores for each instrument, and standard deviations or reported ranges were quite high. However, the larger sample sizes in the previous studies, ranging from 53 [1] to 234 [37], resulted in statistically significant RS values, leading the authors to conclude that a response shift had occurred.

By examining actual mean RS values and standard deviations, it can be concluded that the group effect for response shift observed in previous studies was no more clinically meaningful than those observed in the present study. This conclusion was reiterated by the previous authors who conceded that although a statistically signifi-cant response shift had occurred, adjusting for the response shift did not change clinical conclusions regarding treatment efficacy [1, 37, 38]. Based on the present study and previous reports, a slight group effect for response shift may occur among postoperative orthopaedic knee patients; however, this response shift is not substantial enough on a group level to invalidate the use of traditional pre–post-outcomes assessment methods.

In comparing response shift magnitude values with previously established MDC values for articular cartilage patients, a statistically significant response shift was observed on an individual level for the WOMAC at 6 months (p = 0.047) and the SF-36 PCS at 12 months (p = 0.017). Although the WOMAC and SF-36 PCS scores did not demonstrate a group-level effect for response shift, the mean response shift magnitude observed on WOMAC and SF-36 PCS scores did exceed MDC values—meaning individual patients did exhibit a response shift. However, some patients’ then-test scores recalibrated positively (then-test > pre-test), while others shifted negatively (then-test < pre-test) as a result mean response shift values were not statistically significant, but WOMAC and SF-36 response shift magnitude values were significant. However, response shift magnitude values suggest that WOMAC and SF-36 PCS scores are susceptible to response shift on the individual patient level. If WOMAC and/or SF-36 PCS scores are being used to track treatment progress of an individual patient, response shift should be taken into consideration.

Additional analyses using MDC values suggested that some individual patients may experience a clinically relevant response shift across PROM instruments with 13 patients at 6 months and 7 patients at 12 months observed to have response shift magnitude values exceeding MDC values on at least 3 out of 4 PROMs. The direction of the response shift is important on a group level to evaluate the influence of response shift on interpretation of overall treatment effects across patients. However, because it is clear that patients may experience either a positive or negative response shift, averaging RS values across patients may obscure the occurrence of a true, albeit nonuniform, response shift.

Question structure may contribute to the WOMAC and SF-36 PCS being more influenced by response shift than the IKDC or Lysholm. The versions of the WOMAC and the SF-36 included in this study rely heavily on 5- or 6-item Likert-like response choices. For example, WOMAC response choices include “none”, “mild”, “moderate”, “severe”, or “extreme”. This type of scale can be highly subjective and may be prone to scale recalibration and situational interpretation [35]. Depending on the patient's prior experiences, mild and moderate may have different meanings over time as the patient has more information and new experiences for comparison. While other PROM instruments contain some similarly structured questions, the WOMAC and SF-36 provide significantly less context from which the patient is asked to answer the questions. In contrast, the IKDC and the Lysholm provide the patient with reference criteria creating meaningful standards around which one can anchor his or her internal scale. For example, the IKDC asks “What is the highest level of activity you are able to perform without significant giving way in your knee?” and in addition to providing response choices such as “very strenuous” or “strenuous” examples of each level of activity are provided, such as “very strenuous activities like jumping or pivoting as in basketball or soccer”. By placing the dysfunction of giving way in the participation context of soccer or basketball, the instrument is cueing the patient to a specific sample of relevant experiences or activities from which to evaluate his or her own function. By providing scale anchors and directing the patient towards a specific sample of experiences, the IKDC and Lysholm appear to reduce the risk of significant variation in scale and conceptualization between and within patients over time.

Personal and environmental factors may explain why among cartilage patients response shift seems to be an individual and not a group phenomenon. Unlike a terminal disease, which will likely impact every aspect of life, the impact of physical limitations secondary to knee surgery vary from person to person depending on factors such as employment status, pre-injury activity level, social support, and preoperative expectations. These contextual factors have previously been referred to as “antecedents” in Spranger and Schwartz's model of response shift and HRQL [48]. This model of response shift stresses the importance of variables such as personality, sociodemographics, access to care, physical environment, expectations, and spiritual identity on health outcomes. All of these factors may vary from person to person, further explaining the great variability observed and why evidence of a response shift may exist on an individual level, but not on the group level. Awareness of this individual response shift may be highly relevant to clinicians as they try to reconcile changes in patient-reported health (or lack thereof) with observations of changes in physical health and performance [20]. Clinicians may strongly benefit by having an awareness of what factors may make an individual prone to a response shift and how those factors can be utilized to provide the individual with the highest possible self-perceived HRQL.

By asking patients to recall their level of function 6–12 months prior, the then-test method may be prone to recall bias [42]. However, the then-test method has been demonstrated as having convergent validity with more complicated methods of evaluating response shift including structural equation modelling and analysis of covariance, which require much larger samples sizes than were available in this investigation [42, 51]. Additional research has demonstrated that recall bias alone was unable to explain changes in then-test scores observed among multiple sclerosis patients or human immunodeficiency virus/acquired immune disease syndrome patients, and at least a portion of observed changes could be attributed to response shift [43, 46]. Finally, the use of the then-test method allowed for direct comparison to previous investigations of response shift in orthopaedic knee patients.

Additionally, it is cautioned that the conclusions drawn from this study must be limited to the patient population and time points investigated. The study population included ACI patients undergoing a variety of concomitant procedures. While this patient sample may seem heterogeneous relative to those reported commonly in the literature for ACI, we believe including complex patients in our sample actually increases the generalizability of our findings to true ACI populations treated in clinical practice [10]. Similarly, we only examined response shift within the first year following ACI and cannot draw conclusions concerning the effect of response shift on longer-term outcomes.

Conclusions

These results support the validity of traditional pre-test/post-test research designs in evaluating short-term treatment effects following cartilage repair. However, there is evidence that response shift may occur on an individual level on a patient-by-patient basis, and short-term scores on the WOMAC and SF-36 PCS in particular may be influenced by response shift. On a clinical level, recognizing the occurrence of a response shift may be key in evaluating short-term treatment progress for individual patients. This is particularly true for treatments such as ACI where physicians depend heavily on patient self-report and appraisal of progress because tools for diagnostic evaluation are limited and not always feasible or cost-effective.