The Lack of Association Between Changes in Functional Outcomes and Work Retention in a Chronic Disabling Occupational Spinal Disorder Population: Implications for the Minimum Clinical Important Difference

Abstract

Study Design

A prospective study in a chronic pain/disability population, relating changes in the Oswestry Disability Inventory (ODI), as well as the Mental Component Summary (MCS) and Physical Component Summary (PCS) of the SF-36, to work retention (WR) status at one year post-rehabilitation.

Objectives

To explore the relationship between WR status and change in ODI, and the MCS and PCS of the SF-36, and determine if an MCID can be identified utilizing WR as an external criterion for the group of patients under consideration.

Summary of Background Data

Clinically meaningful change may be defined through self-report, physician-based, or objective criteria of improvement, although most assessments have been based on self-report assessment of improvement. The disability occurring after work-related spinal disorders lends itself to anchoring self-report measures to objective work status outcomes 1 year post-treatment. Additional research is needed to evaluate the relationship between change and objective markers of improvement.

Methods

A consecutive cohort of patients (n=2,024) with chronic disabling occupational spinal disorders (CDOSDs) completed an interdisciplinary functional restoration program, and underwent a structured clinical interview for objective, socioeconomic outcomes at one year post-treatment. The average percent change in the ODI, as well as the MCS and PCS of the SF-36, were calculated for patients who successfully retained work and those who had not after completing a functional restoration program. Predictive ability of the percent change scores were evaluated through logistic regression analysis.

Results

No percent difference variables were strong predictors of work retention status one-year following treatment.

Conclusions

The current analyses suggest that the ODI and SF-36 MCS and PCS measures are not responsive at the individual patient level when WR data are employed as the external criterion utilizing an anchor-based approach. This finding contrasts to reports of responsiveness based on distributional methods, or methods using self-report anchors of change.

Methods for assessing validity and reliability of health related quality of life instruments (HRQLs) are well delineated. However, these psychometric properties do not address questions regarding the responsive properties, or clinical importance of a measure, a concept aimed at defining the amount of change in an instrument that is associated with a clinically meaningful change. As a relatively new concept, definitions and methods for calculating meaningful change vary.¹ One commonly used measurement is the minimum clinical important difference (MCID).¹ The MCID is defined as “the smallest change or difference in an outcome measure that is beneficial and would lead to a change in the patient’s medical management, assuming an absence of excessive side effects and costs”.²

The wide array of techniques for assessing the MCID have been extensively reviewed by Wells³, Crosby⁴, and Copay⁵, and may be grouped into 2 basic methods: (1) distributional; and (2) anchor-based (see these papers for detailed description of approaches). Distributional measures are based upon statistical distributions (e.g. effect size), whereas in an anchor-based approach an external criteria is used to define improvement. External criteria may be physician-based, patient-based, or objective indicators of health.

Several methodological issues have been raised concerning all methodologies.^5,6 One of the major problems with the application of the MCID concept is that it may vary based on numerous factors. Distribution-based determinations may vary from anchor-based⁷,and, within anchor-based approaches, MCIDs may vary based on choice of anchor.⁸ Given the wide variability in MCIDs, the Initiative on Methods, Measurement, and Pain Assessment in Clinical Trials recently recommended at least 2 methods should be used to evaluate the clinical importance of improvement or worsening for chronic pain clinical trial outcome measures. One approach that has received little attention is using objective external criterion in an anchor-based MCID approach. Patient-based anchors are most commonly used and, although the patient’s perspective of what constitutes an important change is undeniably an important aspect of treatment outcomes, the information gained from understanding what magnitude of change on a specific measure is associated with objective markers of success is of interest from a clinical, economic, and patient perspective. One of the primary issues encountered by injured workers is failure to return to work^9,10 which leads to lost wages, productivity costs, disability settlements and pensions, increased healthcare utilization, and decreased quality of life^10,11 making it an important outcome to consider in treatment efficacy from multiple perspectives.

Despite the relative importance of this perspective, the authors are aware of no previously published studies that have explored the MCID of any HRQLs utilizing work status as an anchor in a chronic workers’ compensation population. The primary purpose of the current study was, therefore, to evaluate the clinical importance of two HRQLs frequently used in the occupational musculoskeletal disorder population, with work retention status 1 year following treatment as the external criterion. The 2 measures evaluated were: 1) The Oswestry Disability Index (ODI)¹²; and 2) the Mental Component Score (MCS) and Physical Component Score (PCS) of the Short-Form-36 (SF-36).¹³

METHODS

Subjects

The study consisted of a consecutive cohort of patients (N=2,024) with chronic disabling occupational musculoskeletal disorders admitted to an interdisciplinary functional restoration program from 1999–2004. All patients consented to collection of data for purposes of rehabilitation management, workers compensation documentation, and research. The research protocol for this study was approved by our Institutional Review Board. Patients were included if: they had more than four months partial/total disability since a work-related injury; failure of non-operative care to achieve functional recovery; surgery that had not produced resolution; and ability to speak English or Spanish. Patients in the program were chronically disabled (average length of disability = 18.2 months), and had received limited to no success from traditional pain interventions (e.g., medication, single-discipline therapies, therapeutic injections, and/or surgery). Inclusion criteria were completing treatment, and being out of work prior to the start of treatment. Exclusion criteria included missing data and loss to follow-up, as pre and post data was necessary to determine the percent change, and follow-up work status was needed to assess return to work.

Procedure

All patients participated in an intake interview that consisted of an initial evaluation of medical history, physical examination, psychological assessment, medical case management, disability assessment, and a quantitative physical/functional capacity evaluation.^14–21 Interdisciplinary treatment consisted of quantitatively directed physical exercise progression and multimodal disability management. Patients were provided with some individualized combination of individual counseling, group therapy, stress management, biofeedback, coping skills training, and education focusing on disability management, vocational reintegration and future fitness maintenance.^22,23 Program duration lasted between 4–10 weeks based on level of disability and scheduling availability.

Variables

External Criterion

One year post-treatment, patients were contacted for a structured telephone interview, at which time work status was evaluated.²⁴ Patients successfully working in any capacity (part time, modified work schedule, or full time) at one year were included in the work retention group (WR, n=520). All other patients were included in the no work retention group (NWR, n=181). The reliability of these structured interview data have not been published, but 1- and 2-year telephonic interview findings for patients were evaluated in a previous 2 year follow-up study²⁵. An r-value of 0.92 was obtained as test-retest reliability coefficient for number of visits to healthcare professionals, and coefficients were similarly high for all other outcome variables.

Predictor Variables

The ODI (version 1)¹² has 10 items that involve a variety of daily activities, such as self-care and walking. Each item is scored from 0–5, and scores are calculated as simple percentages, with high scores indicating high functional loss. The SF-36 (version 1)¹³, a general health questionnaire developed to be used across diseases (e.g., diabetes, chronic pain) consists of 36 questions, results in an 8-scale profile, and physical component (PCS) and mental component (MCS) summary scores. Lower scores are indicative of greater disability. The current study utilized both the PCS and MCS. Scale scores were calculated utilizing the transformed scale formula as suggested in the 1993 SF-36 manual.²⁶ Patients completed HRQLs prior (PRE) and following treatment (POST). Percent improvement was defined as percent change (pre-post), relative to the possible improvement (pre- best score). Percent worsening was defined by percent declined (pre-post), relative to the most possible decline (worst possible score - pre). These 2 variables were then combined into a “percent change” variable used for analyses. A percent change score > 0 is associated with improvement, and < 0 with decline. Percent change scores were utilized based on evidence that MCID estimations vary based on pre-treatment level of severity.^27,28

Statistical Analyses

Demographics

Chi-square statistics were calculated for categorical variables, and Student’s t tests for continuous variables. Groups were compared on age, length of disability in months (LOD), race, and gender.

MCID Approach

Classification analyses were utilized to evaluate the responsive properties of selected HRQLs (ODI, MCS, and PCS), utilizing work retention as the external criterion. Sequential logistic regressions (one each for the ODI, MCS and PCS scales of the SF-36), with demographic (age, gender, LOD) and percent difference variables as the predictors, and work retention status as the dependent variable, were applied to the data. The first step included the appropriate percent difference variable, and the 2^nd step the set of demographic variables. Overall performance of the models was evaluated through Nagelkerke pseduo R² estimates²⁹, classification performance (sensitivity and specificity), and positive (PV+) and negative (PV−) predictive values. Cohen’s d effect sizes³⁰ were also calculated for each measure utilizing the following formula:

$$(\mathrm{X}\text{Pr}e-\mathrm{X}\mathit{\text{Post}})/\mathit{\text{SD}}\text{Pr}e.$$

RESULTS

Demographics

A total of 2,024 patients were enrolled in the treatment program, of which 432 (21.3%) did not complete treatment and thus did not meet inclusion criteria. Of these 1,592 cases, 235 were not out of work prior to treatment, making the population of interest a total of 1,357 cases (67.1% of total enrolled). Of these cases, 180 were missing 1 year work status and an additional 476 were missing ODI or SF-36 pre or post data and were excluded, thus the final sample analyzed was 701 (51.7% of cases meeting inclusion criteria).

Table 1 presents demographic variables for patients meeting inclusion criteria, but excluded from analysis (n=656), and those meeting inclusion criteria and included in analysis (n=701). Groups were similar in age, gender, and LOD. The groups differed in race (χ²= 38.43, p < .001) with patients missing data more likely to be Hispanic.

Table 1.

Demographic variables for the patients that met inclusion criteria and included in analyses and those meeting inclusion criteria but excluded (due to missing data) from analyses.

	Included	Excluded	t-test or χ2 p value
N	701	656
Age (mean±SD)	45.91±9.58	45.72±9.86	.80
Gender (n, % male)	367 (52.4)	331 (51.3)	.60
Race (n, %)			<.001
Caucasian	390 (55.6)	290 (45.0)
Black	179 (25.5)	138 (21.4)
Hispanic	117 (16.7)	201 (31.2)
Other	15 (2.1)	16 (2.5)
LOD1 (mean±SD)	17.72±16.68	17.89±20.15	.80

¹LOD is Length of Disability in Months

No differences were detected between WR and no work retention NWR groups in race; however, three notable significant demographic differences were: 1) NWR patients were more likely to have been female (χ² = 7.62, p = .006), and were older (t₆₉₉ = 6.2, p < .001) and had been injured longer on average (,t₆₉₉ = 4.6, p < .001) as compared to WR patients (see Table 2).

Table 2.

Demographic Variables for the Patients in the Work Retention (WR) and No Work Retention (NWR) Groups.

	WR (n=520)	NWR (n=181)	t-test or χ2 p value
Age (mean±SD)	44.54±9.10	49.49±9.66	<.001
Gender (n, % male)	286 (55.5)	78 (43.1)	.006
Race (n, %)
Caucasian	276 (53.1)	111 (61.3)	.31
Black	141 (27.1)	44 (24.3)
Hispanic	91 (17.5)	22 (12.2)
Other	12 (2.3)	4 (2.2)
LOD1 (mean±SD)	16.17±15.18	22.82±21.45	<.001

¹LOD is Length of Disability in Months

Descriptive Statistics and Correlation Analysis

The average pre, post, and percent difference for the HRQLs for WR and NWR groups are presented in Table 3. All groups averaged an improvement on each of the three measures, and effect sizes for all variables for both WR and NWR were moderate to large based on Cohen’s standards (ranging from .63-1.07). For the ODI, 11.8% (n = 83) reported a decline, 4.3% (n = 30) no change (as defined by 0 +/− SE of percent difference), and 83.9% (n = 588) an improvement. The average percent change in ODI among WR patients was 35.26±30.20, as compared to 31.07±27.26 in the NWR group. For the MCS, 25.4% (n = 178) reported a decline, 2.0% (n = 14) no change, and 72.6% (n = 509) improvement, and patients in the WR group averaged 11.61±18.95 percent change, as compared to 7.77±21.0 percent change in the NWR group. For the PCS, 23.5% (n = 165) scored worse, 3.0 % (n = 21) no change, and 73.5% (n = 515) an improvement, and patients in the WR group averaged 7.23±14.30 percent change, as compared to 4.21±16.23 percent change in the NWR group.

Table 3.

Pre, Post, and Percent Change Descriptive Statistics for Work Retention (WR) and No Work Retention (NWR) Groups.

Work Retention
Mean, SD	WR (n=520)	NWR (n=181)	P Value
ODI
Pre	39.60±15.17	44.71±15.16	<.001
Post	25.19±13.58	31.23±14.89	<.001
Percent Change	35.26±30.20	31.07±27.26	.10
Effect Size	.95	.89
SF-36 MCS
Pre	40.43±9.33	39.53±9.84	.27
Post	48.47±9.39	45.73±10.42	.001
Percent Change	11.61±18.95	7.77±21.0	.02
Effect Size	.86	.63
SF-36 PCS
Pre	31.40±5.69	30.46±6.38	.06
Post	37.53±7.72	35.35±7.53	.001
Percent Change	7.23±14.30	4.21±16.23	.02
Effect Size	1.07	.77

Regression Analyses

Percent difference was not a significant predictor of work retention status for the ODI only model (χ² (1, n = 701) = 2.68, p = .10). The percent difference only models for the MCS (χ² (1, n = 700) = 5.20, p = .02; see Table 4), and PCS (χ² (1, n = 700) = 5.44, p = .02) were marginally significant; however, an estimate of R² commonly used in logistic regression²⁹, Nagelkerke R², indicated only 1.1% of the variance was accounted for by either the MCS only or PCS only models.

Table 4.

Logistic Regression Analysis of Work Retention Status as a Function of Percent Difference HRQL and Demographic Variables.

Measure	B	Wald χ2	df	p value	Odds Ratio (95% CI)
ODI
1) Percent Difference	.005	2.68	1	.10	1.01 (1.0, 1.01)
2) Percent Difference	.005	1.18	1	.28	1.01 (1.0, 1.01)
+ Age	−.05	24.41	1	<.001	.95 (1.0, 1.01)
+ LOD	−.02	9.65	1	.002	.99 (.98, .99)
+ Gender	−.40	4.87	1	.03	.67 (.47, .96)
MCS
1) Percent Difference	.01	5.23	1	.02	1.01 (1.0, 1.02)
2) Percent Difference	.01	5.63	1	.02	1.01 (1.0, 1.02)
+ Age	−.05	25.26	1	<.001	.95 (.93, .97)
+ LOD	−.02	9.68	1	.002	.99 (.98, .99)
+ Gender	−.42	5.30	1	.021	.66 (.46, .94)
PCS
1) Percent Difference	.01	5.45	1	.02	1.01 (1.0, 1.03)
2) Percent Difference	.01	3.47	1	.06	1.01 (1.0, 1.02)
+ Age	−.05	24.94	1	<.001	.95 (.93, .97)
+ LOD	−.02	17.28	1	<.001	.98 (.97, .99)
+ Gender	−.43	5.83	1	.02	.65 (.46, .92)

Addition of the demographic variables did improve the fit of each of the models. Age, LOD, and gender were significant predictors for all 3 models (Table 4). In a demographic only model (χ² (3, n = 700) = 52.13, p < .001), people in the WR group were younger, more likely to be male, and had decreased length of disability as compared to the NWR group. The model had good sensitivity (96.93%), but poor specificity (7.82%), with a PV+ of 75.37%, and PV− of only 17%.

DISCUSSION

In an occupational health setting, return to work and work retention after a full course of interventional and non-surgical treatments is anticipated, and are generally regarded as the most “gold standard” of outcomes. The purpose of the current study was to evaluate the responsive properties of the ODI and SF-36 when used in a workers compensation population, with the end-goal of identifying an amount of change (MCID) that is associated with employment 1 year following interdisciplinary treatment. None of the percent change variables predicted more than 1% of the variance in work retention status. Thus, the ODI and SF-36 do not appear to be responsive measures as applied to a chronic musculoskeletal workers compensation population. The lack of sensitivity of these measures in detecting meaningful change implies these are not optimal for assessing treatment outcomes in this population.

Certain limitations with the use of work retention as an objective measure of success may be raised by some readers. The binary approach of work retention/no work retention does not take into consideration individual differences within this variable, such as whether patients were able to return to pre-treatment capacity (if they are working part time, full time, etc). In the current sample, however, 77% of the work retention group did return and maintain full time work, suggesting the binary variable provided adequate context for the current study. Another complexity with work retention is that it may be affected by factors unrelated to functional improvement. Results confirm previous reports that age, LOD, and gender are significant predictors of work retention status. ^31–33 Possible reasons to account for these effects are greater degenerative changes in patients with more prolonged disability and/or older patients, easier access to alternative funding sources such as retirement, greater social acceptability for women to not work, and psychosocial factors such as greater work fatigue.³² Of note, however, is that although age, gender, and LOD were significant predictors, the odds ratios were small and Nagelkerke’s estimate of R² was only 10.5%, suggesting that, although there was a relationship, these variables are modest predictors at best.

Another limitation is the significant loss of data due to patients missing either pre or post SF-36 or ODI data, or 1 year work retention data. Analysis comparing these 2 groups indicates that patients missing data, and thus excluded from analysis, are more likely to be Hispanic. One potential explanation for this finding is that there is a language barrier, resulting in lower contact rate for patients coming from primarily Spanish speaking homes. Consequently, results reported in this study may not generalize to the Hispanic population. Additional research is needed to identify underlying factors in increased missing data in this population.

Notably, there is a lack of congruency between our results using work retention as an external criterion, and other anchor- and distribution-based studies.^34–37 One critical issue that must be addressed in the MCID literature is this discrepancy among methods of calculating responsiveness. While the search for the best way to measure and define clinically important change continues, it is clear that controversy is rampant when discussing this endeavor. However, there are reasonable explanations for why controversy remains about the most appropriate statistical method to gauge clinically meaningful changes. On a basic level, this may be an attempt to avoid the fact that the term important in MCID cannot yet be unequivocally and operationally defined as a reliable construct. What is important to a physician and important to a patient may vary greatly, and assessment of an MCID needs to be explored from multiple perspectives. The advantage to using an objective indicator such as work retention is that in a workers compensation population, it is relevant to all parties.

Furthermore, as recently reviewed by Copay et al. (2007)⁵, the two major approaches to defining an MCID are not without significant problems themselves. For instance, distribution-based approaches can only define some minimum value below which a change score on a self-report may likely be due to measurement error. Therefore, they usually only provide a minimum detectable change that indicates nothing about clinical importance. In the current study, all three measures evaluated met Cohen’s large-effect size criteria for both WR and NWR groups (see Table 3). Although frequently cited in clinical trials as a measure of the amount of change, Cohen’s notion of small, medium, and large effect sizes was provided as a rule of thumb within the social science domain and does not necessarily apply to the area of medicine. As is evident by the lack of correlation between pre to post change in these measures and 1 year work status, the measures of effect size are meaningless in relation to the definition of “clinical importance” in this study as defined by 1 year work status. On the other hand, anchor-based approaches to the MCID are only as good as the external criterion on which they are based, and the methodology used to define clinical importance.⁵ Given the latter problem, it is therefore not surprising that the literature is rife with a wide range of MCID values for a given self-report instrument.³⁸

Most crucially, the important aspects of psychometric theory and methodology should not be ignored in the search for a reliable and sound method of documenting and interpreting clinical change. As highlighted by the panel convened to discuss the issues related to interpreting the minimal important change (MIC)³⁸, a review on this topic revealed “little (or no) theoretical or empirical justification was provided for the study design, anchor or method used for estimating MICs in the identified studies” (p. 91). Furthermore, the idea behind the MCID is to document some type of raw or percent change elicited from a groups-based analysis, and then apply this single numeric value as an index of important change at the level of the individual patient. This approach ignores the most basic concept of variability of a given individual response with respect to the larger sample or population in which this individual was observed in.³⁹

While the authors of the present study do not claim to have a final solution to the problems associated with the MCID, future research attempting to elicit MCID values for common self-report measures should utilize objective anchors relevant to the population of interest, such as health care utilization, case settlement, and work retention status in a workers’ compensation population. The inclusion of an objective perspective provides clinically relevant information to the patient, provider, and third-party payers, and circumvents some of the methodological issues outlined above.

CONCLUSIONS

The results of the present study suggest that, in a chronically disabled workers’ compensation setting, there is no relationship between improvement in self-report scores of the ODI and SF-36 MCS and PCS following rehabilitation and 1-year work status. The discrepancy among results obtained in the current paper using an objective anchor, other anchor-based approaches using self-report anchors, and distributional approaches, highlight the importance of needing further recommendations regarding the optimal method for calculating anchor-based MCIDs, with emphasis on the role of both clinical and statistical interpretation.

Key Points

1. Percent change in the ODI and SF-36 have been linked to clinical improvement as defined by patient reports of whether a treatment was effective or not.

2. Using work retention as a more objective marker of treatment efficacy shows there is no relationship between percent change in the ODI and SF-36 scales and treatment efficacy.

3. The results of this study suggest that, based on an objective marker of success, the ODI and SF-36 are not responsive measures for predicting individual patient improvement, and there is a lack of consistency among calculation methodologies for assessing responsiveness.

4. No justification is found for use of the ODI and SF-36 pre- and post-treatment to define MCID in clinical research on medical devices in this population.

Table 5.

Sensitivity/Specificity and Positive/Negative Predictive Value for Regression Models.

Measure	Classification		Predictive Value
	Sens- itivity	Spec- ificity	Positive	Negative
ODI
1) Percent Difference	100	0	74.14	0
2) Percent Difference
+ Age
+ LOD	97.69	8.84	75.45	13
+ Gender
MCS
1) Percent Difference	100	0	74.14	0
2) Percent Difference
+ Age
+ LOD	97.30	11.05	75.83	15
+ Gender
PCS
1) Percent Difference	100	0	74.14	0
2) Percent Difference
+ Age
+ LOD	96.72	9.39	75.37	18
+ Gender