Determining the Minimal Clinically Important Difference on the Oxford Shoulder Instability Score in Patients Undergoing Arthroscopic Bankart Repair for Shoulder Instability

Abstract

Background:

The literature on minimal clinically important differences (MCIDs) for patient-reported outcome measures assessing shoulder instability is limited, with none addressing the Oxford Shoulder Instability Score (OSIS). The OSIS was developed to provide a standardized method for assessing shoulder function after surgery for shoulder instability, and previous studies have demonstrated its high reliability, low interrater variability, and ease of administration.

Purpose:

To identify the MCID for the OSIS after arthroscopic Bankart repair for recurrent shoulder instability.

Study Design:

Case series; Level of evidence, 4.

Methods:

A longitudinally maintained institutional registry was queried for patients who underwent arthroscopic Bankart repair from 2010 to 2016 for recurrent shoulder instability secondary to a Bankart lesion without significant glenoid bone loss. The OSIS was completed preoperatively and at 1 year postoperatively. Patients were categorized into “expectations met” and “expectations unmet” groups using a questionnaire evaluating expectation fulfilment. The MCID of the OSIS at 1 year was calculated using 3 anchor-based approaches and a distribution-based approach. The 3 anchor-based approaches comprised (1) simple linear regression analysis, (2) receiver operating characteristic curve analysis, and (3) calculation of mean differences in change for the OSIS between the “expectations met” and “expectations unmet” groups.

Results:

The study cohort comprised 68 men and 11 women aged 29.9 ± 12.7 years (mean ± SD). Duration of follow-up for all patients exceeded 1 year. The MCIDs for the OSIS based on the 4 calculation approaches yielded a narrow range of values, ranging from 7.7 to 8.5 for the anchor-based methods and 8.6 for the distribution-based method.

Conclusion:

Study results indicated that patients with recurrent shoulder instability without significant bone loss who undergo primary arthroscopic Bankart repair and have at least 8.6 points of improvement on their OSIS experience a clinically significant change at 1 year postoperatively.

Arthroscopic Bankart repair is a commonly utilized surgical procedure ⁹ for isolated labral tears without significant glenoid bone loss. One way of evaluating successful outcomes after this procedure is through patient-reported outcome measures (PROMs). Several outcome scores been developed for the assessment of shoulder instability. Disease-specific outcome scores for shoulder instability include the Subjective Patient Outcome for Return to Sports score, ³ Oxford Shoulder Instability Score ⁸ (OSIS), Subjective Shoulder Value, ¹⁹ and the Western Ontario Shoulder Instability Index. ³¹ Outcome scores provide patients with evidence-based information to make decisions about their care and give them quality assurance regarding their surgery. They also provide surgeons with a benchmark to assess the efficacy of an intervention.

High variability within each outcome measure poses challenges in instituting practical guidelines in management of the uncomplicated Bankart injury. Furthermore, while changes in PROMs confer certain benefits to the surgeons for appraising their surgical outcomes, a key challenge is determining what constitutes a significant change in scores. Statistically significant changes in scores may not translate to clinical significance in fulfillment of the patient’s satisfaction and expectation. ²⁵

The concept of the minimal clinically important difference (MCID) ^6,11 in orthopaedic procedures ^{2,5,20,33,35,41} has thus gained traction in recent years as a tool to correlate improvements in PROMs with patient perceptions. Jaeschke et al ¹⁵ defined the MCID as “the smallest difference in score in the domain of interest which patients perceive as beneficial and which would mandate, in the absence of troublesome side effects and excessive cost, a change in patient’s management.” A recent systematic review identified 14 clinical studies reporting anchor-based MCID values for shoulder-specific PROMs. ¹⁷ To date, most MCID studies usually report on PROMs in the context of rotator cuff tears and osteoarthritis.

There is a lack of MCID thresholds for PROMs ^4,45 after arthroscopic Bankart repair. Literature specific to Bankart repairs for shoulder instability is limited, and MCID estimates have been established in some PROMs, such as the Rowe score and Western Ontario Shoulder Instability Index, ²⁶ but not for the OSIS. ²⁶ The OSIS is a comprehensive questionnaire consisting of 12 questions, which has been validated and proven to have high reliability and low interrater variability in prior clinical studies. ^39,42,44

In this study, we aimed to address this paucity in the current literature and determine the MCID using anchor- and distribution-based methods ²⁶ for the OSIS after arthroscopic Bankart repair for shoulder instability.

Methods

Patient Selection

The protocol for this study was approved by an institutional review board. A longitudinally maintained shoulder registry from a single institution was retrospectively reviewed by 2 fellowship-trained surgeons (W.S.F. and G.J.Z.) to retrieve records of patients who underwent arthroscopic Bankart repair from 2010 to 2016. Inclusion criteria for surgery were recurrent shoulder instability secondary to a Bankart lesion (diagnosed on magnetic resonance imaging) without significant glenoid bone loss (>25%). We excluded patients with concomitant rotator cuff tears or fractures, as well as those with prior stabilization procedures to the shoulder. A total of 79 patients were included in the study. A prior study ²² evaluating the MCID of outcome scores after rotator cuff surgery had a similar sample size.

Patients were prospectively evaluated for a minimum of 1 year and assessed by an independent physical therapist. This study utilized the original OSIS (designed in 1999), ⁸ with a total score ranging from 12 (least impaired) to 60 (most impaired). In addition, all patients were provided a shoulder-specific questionnaire for evaluation of the OSIS preoperatively and at 1 year postoperatively. Baseline demographic data (age, sex, and body mass index), surgical technique, and length of stay were evaluated as well.

Surgical Methods

Surgery was performed arthroscopically with the patient in the beach-chair position. General anesthesia was used. Repair of Bankart lesions was done arthroscopically using bioabsorbable suture anchors (Gryphon and Lupine; both from Mitek). Postoperatively, patients were placed on an arm sling, with gradual physical therapy commenced. Patients were taught pendular exercises postoperatively before discharge. Passive range of motion and strengthening exercises began at the 6-week mark in the outpatient physical therapy clinics. Full return to sport was allowed at 3 months postoperatively.

Defining the MCID

Anchor-Based Approaches

An anchor question was administered 1 year postoperatively to evaluate the patient’s expectation fulfillment after surgery. This was an external criterion for an anchor-based approach for determination ^7,30,40 of the MCID. The anchor question was “Has the surgery for your shoulder condition met your expectations so far?” Patients who answered “yes, totally,” “yes, almost totally,” and “yes, quite a bit” were allocated a score of 1, 2, and 3, respectively, and represented the group with expectations met (minimal change group). Patients who responded “more or less,” “no, not quite,” “no, far from it,” and “no, not at all” were allocated scores ranging from 4 to 7 and represented the group with expectations unmet (no change).

From the anchor-based approach, the MCID was derived through 3 methods. For the first method, ⁴⁷ simple linear regression analysis was performed, using the pre- to postoperative change in OSIS as the dependent variable and dichotomized scores on the anchor question (1-3 vs 4-7) as the independent variable. The MCID corresponded to the coefficient of expectation fulfillment. The second method used the receiver operating characteristic (ROC) curve approach, where the MCID corresponded to the point closest to the top-left corner of the ROC graph. This point corresponded to a threshold of OSIS change that is associated with the smallest amount of misclassification. ⁴⁶ ROC curve analysis was used to discriminate between anchor scores of 1 to 3 and 4 to 7 for change in OSIS. The discriminative ability was determined by the area under the curve. The MCID by the third method was the mean difference in the change in OSIS between the groups with expectations met and unmet.

Distribution-Based Approach

To mitigate the weaknesses of an anchor-based approach, a distribution-based approach was used to determine the MCID. This was performed according to the statistical characteristics of our sample and, consequently, the statistically significant changes in relation to the probability that the change occurred by chance. ⁴⁶ Based on available literature, the MCID can be calculated through the following means: half a standard deviation from mean baseline scores, ¹² small Cohen effect size, standard error of measurement, and minimum detectable change. We used the standard error of measurement method, which was validated in a prior systematic review measuring shoulder function. ²⁹

The effect size is a standardized measurement of change, and it is defined as the change in outcome scores from baseline to post surgery, divided by the standard deviation of the baseline score. It is based on the variability of patients’ improvement according to postoperative scores. ^5,10,18 A greater degree of pre- to postoperative improvement will thus lead to a larger effect size, ²⁸ and a larger effect size signifies a greater proportion of the treatment group surpassing the MCID as opposed to a control group. Conventionally, differences of 0.2, 0.5, and 0.8 standard deviations were considered small, medium, and large effect sizes, respectively. ²¹ A small effect size was used in calculating the MCID, with methodology drawn from a study ³² that also selected an effect size benchmark of 0.20 as approximating the MCID.

Statistical Analysis

Statistical analysis was performed with consultation with a biostatistician at our institution. Descriptive analysis for the pre- and postoperative OSIS was conducted, while the paired t test was used to compare the pre- and postoperative OSIS for the entire cohort and by sex. Comparison of these variables across anchor question responses was performed using analysis of variance. We compared the change in OSIS between the study groups (expectations met vs unmet) via a 2-sample t test. Statistical analysis was performed using R software (Version 3.5.1; R Core Team), and statistical significance was defined as P < .05.

Results

A total of 79 patients diagnosed with recurrent anterior shoulder instability secondary to a Bankart lesion were enrolled in the study. All cases were diagnosed using magnetic resonance imaging of the involved shoulder. They comprised 68 men (86%) and 11 women (14%) aged 29.9 ± 12.7 years (mean ± SD). A statistically significant difference in age was observed between men (28.2 ± 10.9 years) and women (43 ± 17.3 years; P < .001). No statistically significant difference was observed in body mass index between men and women. A significant improvement was observed in the OSIS at 1 year, overall and by sex (P ≤ .001 for all) (Table 1).

Table 1.

Comparison of Pre- and Postoperative OSIS Overall and by Sex ^a

	OSIS, Mean ± SD
	Preoperative	Postoperative	Mean Difference (95% CI)	P Value
Overall (N = 79)	35.1 ± 9.4	20.6 ± 8.3	–14.4 (–16.8 to –12)	<.001
Females (n = 11)	37.6 ± 10.7	23.5 ± 5.9	–14.2 (–20.9 to –7.5)	.001
Males (n = 68)	34.7 ± 9.2	20.2 ± 8.6	–14.5 (–17.1 to –11.8)	<.001

^a Bold P values indicate statistically significant difference between pre- and postoperative OSIS (P < .05, paired t test). OSIS, Oxford Shoulder Instability Score.

Table 2 shows the distribution of participants and mean OSIS according to responses to the anchor question. Of 79 patients, 71 (90%) reported that their expectations were met (score, 1-3): 24 patients had total fulfillment of expectations, 35 had almost fulfilled expectations, and 12 had fair fulfillment of expectations. Eight patients scored between 4 and 7 and were considered to have unmet expectations. There was a statistically significant difference in absolute 1-year postoperative OSIS and in change in OSIS between patients with a score of 1 to 3 and 4 to 7.

Table 2.

Preoperative, Postoperative, and Change in OSIS Across the Levels of Expectation ^a

	Responses to Expectations Question, b Mean ± SD
OSIS	1 (n = 24)	2 (n = 35)	3 (n = 12)	4-7 (n = 8)	P Value
Preoperative	37.2 ± 8.7	33.1 ± 8.6	36 ± 10.9	36 ± 12.3	.393
Postoperative, 1 y	15.7 ± 4.1	20.3 ± 7.5	26.5 ± 8.7	28.5 ± 10.4	.001
Change	–21.5 ± 8.7	–12.9 ± 8.3	–9.5 ± 13.3	–7.5 ± 11.3	.002

^a Bold P values indicate statistically significant difference between responses (P < .05, analysis of variance). OSIS, Oxford Shoulder Instability Score.

^b Based on pain relief and functional outcomes at 1 year postoperatively: “Has the surgery for your shoulder condition met your expectations so far?” Responses were scored as 1 (yes, totally), 2 (yes, almost totally), 3 (yes, quite a bit), 4 (more or less), 5 (no, not quite), 6 (no, far from it), and 7 (no, not at all).

Anchor-Based Approaches

The MCIDs calculated according to the 3 anchor-based approaches are summarized in Table 3. For the linear regression approach, the MCID based on plotting mean change in OSIS against level of expectation was 7.7 (95% CI, –15.5 to 0.1). Based on this method, at postoperative 12 months, a minimal change of 7.7 in OSIS was required to attain fulfillment of expectations. Of 79 patients, 64 (81%) reported a change greater or equal to this proposed MCID. For the second anchor-based method, the ROC curve yielded an MCID value of 8.5, although our data obtained only an acceptable performance of discrimination (area under the curve, 71%; 95% CI, 48.6%-93.5%) (Figure 1). The MCID calculated as the mean difference in OSIS improvement between the expectation groups (met vs unmet) was 7.7.

Table 3.

MCIDs for the OSIS as Calculated Using the 3 Anchor-Based Approaches ^a

Linear regression analysis
MCID (95% CI)	7.7 (–15.5 to 0.1)
P value	.052
ROC curve analysis
MCID based on expectation	8.5
Sensitivity, % (95% CI)	81.7 (70.7-89.9)
Specificity, % (95% CI)	62.5 (24.5-91.5)
Change in OSIS between groups
Expectations met (n = 71)	–15.2 ± 10.4
Expectations unmet (n = 8)	–7.5 ± 11.3
P value	.101
MCID	7.7

^a MCID, minimal clinically important difference; OSIS, Oxford Shoulder Instability Score; ROC, receiver operating characteristic.

Figure 1.

ROC curves for dichotomized responses to the expectation anchor question using change in Oxford Shoulder Instability Score at 12 months for calculation of the 12-month MCID. The MCID corresponded to the point closest to the top-left corner of the ROC graph (red dot). AUC, area under the curve; MCID, minimal clinically important difference; ROC, receiver operating characteristic.

Distribution-Based Approach

The MCID by the distribution-based approach (ie, standard error of measurement) used the following formula: s × SD(baseline OSIS) × √(1 – r), where s was set at 1 for a small effect size, 1.96 for moderate, or 2.77 for large ²³ and r was the test-retest intraclass correlation coefficient, which was calculated using 12-month postoperative OSIS measurements yielding an r value of 0.18. Based on a small effect size, the distribution-based approach yielded an MCID of 8.6.

Discussion

In this study, we determined the 12-month postoperative MCID estimates for the OSIS to be from 7.7 to 8.6. With the upper limit as the proposed value, the MCID for the OSIS was calculated as 8.6. There were no identifiable associations of the MCID with age or sex.

Mean OSIS after arthroscopic Bankart repair ^13,34,36 has been reported between 13.5 and 33.7. This wide variability in postoperative PROMs makes interpretation of the results contentious. Surgeons often favor objective means of determining success, which include range of motion, visual analog scale for pain, or measured laxity. Patients conversely measure surgical success by their overall health state, where they consider themselves to be at an acceptable endpoint. ³⁰

The closest study to ours was by van der Linde et al, ⁴³ who examined the minimally important change for the OSIS in 2017. In that study, the mean age of patients was 32 years and 70% were men. Our study had similar demographic data, with a population mean age of 29.9 ± 12.7 years, where 86% of them were men. Van der Linde et al found the minimally important change for the OSIS to be about 6 points; however, only 36% of patients were treated surgically and follow-up was just 6 months.

A recent systematic review ¹⁴ identified limited studies with high-credibility MCID estimates for the following PROMs: Constant score, Simple Shoulder Test, visual analog scale for pain, DASH questionnaire (Disabilities of the Arm, Shoulder and Hand), Oxford Shoulder Score, and Short Form Health Survey–12. Moser et al ²⁴ calculated the MCID for the OSIS as 6.5 after nonoperative treatment for shoulder instability. MCIDs of 8 to 10 for the OSIS have also been reported in patients diagnosed with superior labral anterior and posterior (SLAP) lesions. ^36,37 Findings of this study mirror clinically important change estimates in reports on SLAP lesions. ^36,37 Pathology for SLAP and Bankart lesions may have some similarities, but we still advocate having condition-specific outcome cutoffs for each procedure. ^36,38

The use of anchor-based approaches aids in deriving an association between OSIS and patients’ subjective determination of success. ²⁷ Anchor-based approaches typically do not factor in the precision of the measuring instrument but are based on external criteria, such as retrospective judgment of change. ¹⁶ The addition of a distribution-based approach focuses on the statistical characteristics of a patient sample and supports the estimates from anchor-based approaches. ²⁸ In spite of the high variability, the 4 methods of estimating the MCID in our study yielded a consistent range of values. Using multivariate linear and logistic regression modeling to calculate the MCID also reduced bias. ¹

This study adds to the existing literature by proposing an MCID cutoff for the OSIS after arthroscopic Bankart repair. Our results provide clinicians with information to determine if clinically meaningful improvement was achieved after surgery based on the OSIS. The MCID values identified in this study may be used as an adjunct to power calculations for future comparative studies.

Limitations

The following are limitations to this study. All patients were recruited from a single institution with variable characteristics. We did not have data on the number of dislocations, the time from initial dislocation to surgery, or the sport played, and we had only 1 year of postoperative follow-up for our patients. We also used the small effect size for calculation of the MCID by a distribution-based approach. The MCID derived from this analysis is not a definite attribute and is subject to fluctuations based on what is interpreted as important to the patient. As these interpretations are subjective and may vary across populations, the proposed values in our study cannot be transferred across patient populations or disease-specific states. ⁴⁶

There is a plethora of methods for calculating the MCID, which can give rise to high variability in values, and the lack of a universally accepted methodology for doing so may reduce the applicability of our results. ⁴⁶ Regardless, the derived results from this study remain relevant and valuable, as different methods yielded similar values. This proposed MCID helps bring the patient’s perspective to light and can help guide the clinician to make informed clinical decisions, while bearing its potential methodological limitations.

Conclusion

We propose an MCID value of 8.6 on the OSIS to represent a significant patient-reported outcome after arthroscopic Bankart repair at 1 year postoperatively. The findings of this study provide clinicians with an objective marker on which they can determine the success of arthroscopic Bankart repair. The MCID of the OSIS can contribute to power calculations in future randomized studies pertaining to shoulder arthroscopy and the treatment of shoulder instability.