Reliability, validity, and responsiveness of a novel subscapularis outcome score: the Baltimore Orthopedic Subscapularis Score

Abstract

Background

Available surveys that evaluate shoulder strength and pain often combine rotator cuff muscles making the test unable to differentiate subscapularis tears from other pathology including concomitant supraspinatus, infraspinatus tears. The purpose of this study was to validate a subscapularis-specific shoulder survey (Baltimore Orthopedic Subscapularis Score) as a viable clinical outcome assessment through analysis of psychometric properties.

Methods

A 5-question survey was given to a study population of 390 patients, 136 of whom had full thickness rotator cuff tears with a minimum score of 5 (better) and a maximum score of 25 (worse). Surveys were given during the initial consultation, preoperative visit, and postoperative visit. Content validity, construct validity, test-retest reliability, responsiveness to change, internal consistency, and minimal clinically important difference using distribution and anchor-based methods were determined for our subscapularis function survey.

Results

A high correlation was reported on test-retest reliability (intraclass correlation coefficient = 0.89). An acceptable internal consistency was reported for all patients surveyed (Cronbach alpha = 0.91). Floor and ceiling effects for patients with rotator cuff pathology were minimized (1% for both). Patients with an isolated subscapularis tear scored worse than supraspinatus/infraspinatus tears and exhibited similar dysfunction as patients with a supraspinatus/infraspinatus/subscapularis tear. An acceptable construct validity was reported with subscapularis-involved tears demonstrating higher scores with significance (P < .05). There was excellent responsiveness to change with a standardized response mean of 1.51 and effect size of 1.27 (large > 0.8). The minimal clinically important difference using a distribution and anchor-based method was 4.1 and 4.6, respectively. Among patients with rotator cuff tears in this population, a score of 22 or higher predicts a subscapularis tear 75% of the time, in spite of its low overall prevalence.

Conclusion

The subscapularis shoulder score demonstrated acceptable psychometric performance for outcomes assessment in patients with rotator cuff disease. This survey can be used as an effective clinical tool to assess subscapularis function.

Shoulder specific patient-reported outcome measures are critical for evaluating operative success and patient satisfaction. Numerous psychometrically validated shoulder and upper extremity patient-reported outcome measures currently exist such as the American Shoulder and Elbow Surgeons (ASES) score, Penn Shoulder Score, Western Ontario Shoulder Instability Index, Simple Shoulder Test, and Disabilities of the Arm, Shoulder and Hand score.^8,9,11,15,18 Of the numerous outcome measures available, none specifically assess subscapularis function. The available outcome instruments often combine rotator cuff muscles when evaluating shoulder strength and pain, making it difficult to determine if a subscapularis injury is present.^19,20,30

Assessment of subscapularis function is important in a variety of clinical settings. Controversy exists concerning the optimal method of subscapularis takedown during total shoulder arthroplasty (TSA), and a validated subscapularis-specific score could help provide clarity on which method gives patients the best outcome postoperatively.^3,26 Inverted L-shaped subscapularis tenotomy approaches in anterior glenohumeral instability operations often lead to postoperative subscapularis insufficiency which could possibly be better identified and subsequently treated with the advent of a subscapularis-specific score.²⁵ Patients with subscapularis failure after TSA who report trouble with internal rotation but who still show improvement from preoperative visits on the ASES survey present a challenging diagnosis because of the difficulties with physical examination and imaging ambiguities.^19,20,28 A subscapularis-specific survey would be of great clinical benefit for detecting and differentiating subscapularis tears from other types of rotator cuff tears in a fast and affordable way.

Ultimately, the use of outcome measurements in clinical settings must only be done after validating its psychometric properties. These properties include validity, reliability, and responsiveness to change.¹¹ Validity questions if the outcome instrument actually measures what it is supposed to measure. It is made up of content and construct validity. Content validity is how well an instrument measures the theoretical construct in addition to floor and ceiling effects. Construct validity is measured by the assessment of whether the instrument follows accepted hypotheses.^1,11,27 Reliability is commonly assessed by measuring the reproducibility.^1,11,27 Responsiveness to change is determined by establishing an instrument’s ability to appropriately detect change over time.²¹

Therefore, the purpose of this study was to validate a subscapularis-specific shoulder survey (Baltimore Orthopedic Subscapularis Score [BOSS]) as a viable clinical outcome assessment through analysis of psychometric properties. We hypothesized that the survey would demonstrate acceptable psychometric properties including validity, reliability, and responsiveness to change. Our secondary hypothesis was that there would be a quantifiable and significant minimal clinically important difference (MCID) identified from survey results.

Materials and methods

Development of the Baltimore Orthopedic Subscapularis Score (BOSS) survey

The BOSS survey questionnaire was developed based on face validity and the authors’ expert consensus regarding the symptoms of subscapularis insufficiency. The survey included 5 questions pertaining to the function of the subscapularis muscle. Each question allows for a patient-reported outcome measurement ranging from 1 to 5. The minimum and maximum possible scores were 5 (better function) and 25 (worse function), respectively. Refer to Table I for a representation of the survey.

Table I.

Survey questions.

Question	No difficulty	Mild difficulty	Moderate difficulty	Severe difficulty	Unable to perform
Using a knife to cut food	1	2	3	4	5
Scrub surfaces	1	2	3	4	5
Wash dishes	1	2	3	4	5
Tuck in your shirt	1	2	3	4	5
Pour liquid into a glass	1	2	3	4	5

Study groups, data collection, and statistical analysis

Institutional review board approval was granted for this study. A total of 390 patients were recruited for this study at multiple clinical sites, 136 of whom had full thickness rotator cuff tears defined by magnetic resonance imaging (MRI) interpreted by an attending orthopedic surgeon (Fig. 1). Inclusion criteria for reliability were patients with existing shoulder pathology. Inclusion criteria for validity were patients with full thickness rotator cuff tears. Inclusion criteria for responsiveness were those patients with full thickness rotator cuff tears involving the subscapularis who underwent arthroscopic repair. There were no exclusion criteria for establishing reliability. Exclusion criteria for validity and responsiveness included patients with glenohumeral arthritis, post-traumatic arthritis, infection, neurologic deficit, adhesive capsulitis, proximal humerus fracture, previous rotator cuff repair, or previous shoulder arthroplasty.

Figure 1.

Flow diagram of included and excluded patients.

Patients in this study included those with subscapularis-only tears; supraspinatus-only tears; supraspinatus/subscapularis tears; supraspinatus/infraspinatus tears; and combined supraspinatus/infraspinatus/subscapularis tears. The BOSS survey was given at the initial office visit, preoperative visit, and postoperative visit. A minimum of 6-month follow-up visit was also conducted either in-person or by telephone because of the COVID-19 pandemic. Statistical analysis was performed with SPSS software (IBM Corp., Armonk, NY, USA).

Test-retest reliability and internal consistency

Test-retest reliability was determined by identifying the intraclass correlation coefficient (ICC) between patient surveys collected at the initial clinic visit and preoperative visit with no change in shoulder status. Internal consistency was calculated by computing the Cronbach alpha score for the whole cohort (all), as well as only patients with rotator cuff tears.

Validity

Content and construct validity were analyzed in this study. Content validity was determined by first vetting the questions which make up the survey with 5 fellowship-trained shoulder and elbow surgeons, then analyzing the presence of floor effects (BOSS = 5) and ceiling effects (BOSS = 25). Floor and ceiling effects were deemed minimized if no more than 15% of participants had the highest or lowest possible score.¹ This score was calculated in patients with rotator cuff tears. Construct validity was analyzed by comparing the ability of the BOSS to recognize subscapularis dysfunction in 4 separate patient groups (“hypotheses”) categorized based on their respective rotator cuff pathology defined by MRI:

• 1)supraspinatus/infraspinatus/subscapularis versus supraspinatus/infraspinatus
• 2)subscapularis only versus supraspinatus only
• 3)subscapularis only versus supraspinatus/infraspinatus
• 4)supraspinatus/infraspinatus/subscapularis versus supraspinatus only

These groups were specifically analyzed because of their clinical relevance. A P-value < .05 was deemed as statistically significant.

Responsiveness

Responsiveness to change was assessed by comparing the BOSS survey results of patients with rotator cuff tears between preoperative and postoperative visits. This result was measured by calculating the effect size (ES) and standardized response mean (SRM) using Cohen’s d statistic.⁵ The ES and SRM were interpreted as small if = .20, medium if = .50, or large if > .80.⁵

Minimal clinically important difference

The MCID for this study was defined as the minimal change in BOSS score that an individual patient would identify as significant and thus would warrant a change in clinical management. A distribution-based MCID was calculated using 0.8 multiplied by the baseline (preoperative) standard deviation of patient scores with subscapularis pathology.^5,29 An anchor-based method was also used to calculate the MCID defined as the mean change in score of patients who rated themselves as a “little better” or a “little worse” as opposed to “about the same” on a 5-question Likert scale questionnaire.²⁹ Lastly, a diagnosis as an anchor was used to calculate MCID comparing subscapularis-only tears versus supraspinatus-only tears.

Accuracy

The accuracy of the BOSS was determined by calculating the number of correct assessments (true positives + true negatives) divided by the number of all assessments. True positives and true negatives were cross-checked with MRI results and/or operative reports. The highest accuracy point was determined to be a score greater than 20.

Results

The BOSS survey was administered to 390 patients. The mean age was 56.2 years (range, 18 to 85). Fifty-two percent of the study subjects were female (n = 201). Refer to Table II for detailed participant characteristics. Seven patients had subscapularis-only tears. Fifty-seven patients had supraspinatus-only tears. Eighteen patients had supraspinatus/subscapularis tears. Thirty-one patients had supraspinatus/infraspinatus tears. Twenty-three patients had supraspinatus/infraspinatus/subscapularis tears. Refer to Fig. 2 for the score breakdown by rotator cuff tear. The 18 patients with supraspinatus/subscapularis tears were not compared in the validity hypothesis calculations because these tears would perform similarly on the survey to isolated subscapularis tears.

Table II.

Participant characteristics

Characteristic	Respondents (N = 390)
Age (years)
Range	18 to 25
Mean (SD)	56.2 (13.0)
Female, n (%)	201 (52)
Race, n (%)
Hispanic	7 (2)
American Indian or Alaskan	1 (0)
Unknown	10 (3)
Asian	142 (36)
Black	197 (51)
White	33 (8)
BMI
Range	16 to 53
Mean (SD)	31.1 (6.4)

SD, standard deviation; BMI, body mass index.

Figure 2.

Score breakdown by rotator cuff tear. Abbreviations: Subscap, subscapularis; Supra, supraspinatus; Infra, infraspinatus.

When analyzing the test-retest reliability of BOSS results, patients were given the survey at their initial clinic visit and then at their preoperative visit with no change in shoulder status (n = 36). An ICC of 0.89 was reported, indicating good reliability. When reliability calculations were rerun in patients with rotator cuff tears only (n = 23), an ICC of 0.85 was reported, indicating good reliability as well.

The internal consistency measured via Cronbach alpha score for all patients (N = 390) was 0.91, indicating the survey questions are closely related as a group. The Cronbach alpha score for those patients with rotator cuff tears was 0.91, also indicating excellent reliability.

Floor and ceiling effects were calculated in patients with rotator cuff tears. One patient had a maximum score of 25 and 1 patient had a minimum score of 5 out of 136 patients (1%), with acceptable floor and ceiling effects (<15%), indicating excellent content validity for the BOSS.

Construct validity was determined by testing the 4 different hypotheses mentioned in the above methods section. Patients with combined supraspinatus/infraspinatus/subscapularis tears scored an average of 3.28 points higher than those with combined supraspinatus/infraspinatus tears (P = .01). Patients with isolated subscapularis tears scored an average of 5.64 points higher than those with isolated supraspinatus tears (P = .003). Patients with isolated subscapularis tears scored an average of 4.17 points higher than those with combined supraspinatus/infraspinatus tears (P = .04). Patients with combined supraspinatus/infraspinatus/subscapularis tears scored an average of 4.76 points higher than those with isolated supraspinatus tears (P < .0001). Refer to Table III for a representation of construct validity analysis.

Table III.

Construct validity demonstrated by comparison of scores based on cuff tear.

Rotator cuff tear group comparison	Mean score difference (95% CI)	P Value
Subscap vs. supra	5.64 (1.92 to 9.36)	.003
Subscap vs. supra, infra	4.17 (0.28 to 8.05)	.04
Supra, infra subscap vs. supra	4.76 (2.46 to 7.05)	<.0001
Supra, infra, subscap vs. supra, infra	3.28 (0.72 to 5.83)	.01

Subscap, subscapularis; Supra, supraspinatus; Infra, infraspinatus; CI, confidence interval.

Responsiveness to change from preoperatively to a minimum of 6 months postoperatively to allow for adequate healing was analyzed in patients with subscapularis pathology (n = 20). The average follow-up of this group was 18 months (range, 6 to 31 months). The BOSS showed an ES of 1.27 and SRM of 1.51 which are both classified as large using Cohen’s thresholds for grading SRM and ES (Table IV).⁵

Table IV.

Responsiveness demonstrated by comparison of preoperative and postoperative scores.

Rotator cuff tear	Preoperative score	Postoperative score	Change score	Standardized response mean	Effect size
Rotator cuff tears involving subscapularis (n = 20)	16.0 ± 5.2	9.42 ± 5.0	6.9 ± 4.4	1.51	1.27

The MCID using a distribution-based method was 4.1 using 0.8 multiplied by the baseline (preoperative) standard deviation given a large effect size.^5,29 The anchor-based method MCID was 4.6 calculated using the mean change in score of patients who rated themselves as a “little better” or a “little worse” as opposed to “about the same” using postoperative questionnaires that accompanied the subscapularis survey. Diagnosis as an anchor was used to calculate MCID and was 3.6 for subscapularis-only tears and 3.8 for supraspinatus-only tears.

Accuracy was calculated in multiple groups. In patients with subscapularis-only tears, a score greater than 20 on the BOSS predicted subscapularis pathology with 88% accuracy (2 true positives, 118 true negatives, and 136 total assessments), sensitivity 28%, specificity 91%, positive predictive value (PPV) 15%, and negative predictive value (NPV) 95%. In patients with isolated subscapularis tears and combined supraspinatus/infraspinatus/subscapularis tears, a score greater than 20 on the BOSS predicted subscapularis pathology with 79% accuracy (7 true positives, 100 true negatives, and 136 total assessments), sensitivity 23%, specificity 94%, PPV 53%, and NPV 81%. In patients with any form of subscapularis tear, a score greater than 20 on the BOSS predicted subscapularis pathology with 67% accuracy (8 true positives, 83 true negatives, and 136 total assessments), sensitivity 16%, specificity 94%, PPV 62%, and NPV 67%.

Discussion

The BOSS is a patient-reported outcome measurement tool for assessing subscapularis-specific pathology. It was developed to specifically assess functional deficits seen with subscapularis pathology. Survey questions were formulated based on the authors’ expert consensus on symptoms of subscapularis insufficiency based on patient reports, common complaints, and physical examination findings of patients with subscapularis failure.

The BOSS demonstrated acceptable psychometric properties in this validation study. Excellent test-retest reliability and internal consistency were reported. Acceptable floor and ceiling effects were demonstrated, illustrating excellent content validity. The survey was able to demonstrate a large responsiveness to change for patients with subscapularis tears who underwent arthroscopic repair. This study further showed a MCID of 4.1.

The BOSS also showed clinically relevant construct validity. Subscapularis pathology correctly produced higher scores in 4 scenarios: when isolated subscapularis tears were compared to isolated supraspinatus tears and supraspinatus/infraspinatus tears, and when combined supraspinatus/infraspinatus/subscapularis tears were compared to isolated supraspinatus tears and supraspinatus/infraspinatus tears. These higher scores were greater than our measured MCID (4.1) in 3 scenarios: when isolated subscapularis tears were compared to isolated supraspinatus tears and combined supraspinatus/infraspinatus tears, and when combined supraspinatus/infraspinatus/subscapularis tears were compared to isolated supraspinatus tears.

The study results did not show a statistically significant difference when isolated subscapularis tears were compared to supraspinatus/subscapularis tears. This finding might be related to the well-known challenge of distinguishing isolated subscapularis tears from combined supraspinatus/subscapularis tears because of similar presentations.¹² In addition, no statistically significant difference was demonstrated in survey results when supraspinatus/subscapularis tears were compared to supraspinatus/infraspinatus tears, although it is unclear how much clinical relevance exists when comparing these two groups. Isolated subscapularis tears scored significantly higher on the BOSS when compared to combined supraspinatus/infraspinatus tears with a mean score difference of 4.17. This finding is notable considering supraspinatus/infraspinatus tears are likely clinically worse as those patients with isolated subscapularis tears often maintain reasonable overall function.^7,29,31 Nevertheless, this survey still accurately captures isolated subscapular tears.

A subscapularis-specific survey can provide value. For instance, although rotator cuff tears after TSA have a relatively rare incidence of approximately 1%, the subscapularis accounts for nearly half of these injuries and it is often challenging to diagnose these patients.^6,20 A validated, subscapularis-specific survey such as the one developed for this study, offers an inexpensive resource to potentially identify these individuals. Additionally, clinical outcome studies have not shown consistent agreement on the optimal subscapularis management technique in TSA.2, 3, 4^,10,14 Lapner et al¹³ indicated no difference in outcome between tenotomy and peel arms in a randomized controlled trial but did not use a validated subscapularis-specific score. This subscapularis-specific questionnaire can shed light on which approach is best by offering a quantifiable measurement of subscapularis function postoperatively rather than more global assessments such as the ASES, where subscapularis-specific pathology is diluted.

Previous studies have attempted to use established scoring systems for evaluating subscapularis dysfunction. Two studies found that the difference in ASES score between full thickness subscapularis tears and normal tendons was less than the ASES MCID.^16,24 Similarly, Park et al²³ found no statistically significant difference in preoperative ASES score between intact, partial, and partial-to-full subscapularis tendon tears. Therefore, considering there is no gold standard subscapularis-specific survey, a criterion validity was not reported in this study. We believe that this novel survey is, to the best of our knowledge, the first to demonstrate construct validity in subscapularis insufficiency patients.

This validation study does have limitations. First, our cohort included a limited number of patients with subscapularis-only tears. Isolated subscapularis tears do have a low prevalence of less than 10%; therefore, this small subset of patients was an expected limitation.^17,22 Moreover, to be included in the study of responsiveness, the patient had to have a subscapularis tear that was fixed and not part of a tendon transfer or arthroplasty, and had to be seen for follow-up. To accommodate these limitations, combined group hypothesis testing was employed for analyzing construct validity and responsiveness. In addition, for calculating responsiveness, only 42% of patients with cuff tears involving the subscapularis were successfully surveyed at 6 months or later postoperatively. Patient numbers were limited not only by a loss to follow-up but because not all patients with cuff tears involving the subscapularis went on to rotator cuff repair. This limitation may have contributed to a nonresponse bias. A future study involving multiple centers to increase the number of patients, comparing isolated subscapularis tears to isolated supraspinatus tears, would help to further prove validity. Future directions include using the BOSS as an outcome assessment tool in reverse shoulder arthroplasty studies to evaluate the need for subscapularis repair and to more accurately analyze and define subscapularis dysfunction after TSA.

Conclusion

The subscapularis shoulder score demonstrated acceptable psychometric performance for outcomes assessment in patients with rotator cuff disease. This survey can be used as an effective clinical tool to assess subscapularis function.

Disclaimers:

Funding: No funding was disclosed by the authors.

Conflicts of interest: The authors, their immediate families, and any research foundation with which they are affiliated have not received any financial payments or other benefits from any commercial entity related to the subject of this article.