Agreement between the American Shoulder and Elbow Surgeons Society Standardized Shoulder Assessment score (ASES) and the Oxford Shoulder Score (OSS) in patients presenting with shoulder pathology: A cohort analysis of the Clinical Quality Registry for Outcomes in Shoulder and Elbow Pathology (CROSEP) registry

Abstract

Background

The choice of patient-reported outcome measure (PROM) used in shoulder studies varies based on clinician's preference and location. This creates difficulties when attempting to compare studies which have used different PROMs as their outcome measure. This study aims to assess the agreement between the American Shoulder and Elbow Surgeons score (ASES) and the Oxford Shoulder Score (OSS), and identify factors associated with agreement.

Methods

Patients with shoulder pathology were identified from a multi-cohort observational practice registry. 1050 paired ASES and OSS pre-treatment scores were prospectively collected. Linear regression was performed to assess the agreement between the PROMs. Mixed-effects analysis of variance was performed to assess the influence of factors associated with agreement.

Results

Regression for mean total and mean function ASES and OSS demonstrated good fit (adjusted R² 57.7%, P < 0.001; and 63.9%, P < 0.001). Mean pain subscore demonstrated a poorer fit (adjusted R² 39.4%, P < 0.001). Crosswalks to convert between mean scores were produced with reasonable precision. Veterans RAND 12-Item Health Survey score, age and diagnosis cohort influenced agreement.

Conclusion

Mean total and mean function ASES and OSS scores agree well with each other. This allows for a more informed comparison of studies using either PROMs as their outcome measure.

Introduction

Patient reported outcome measures (PROMs) which evaluate pain and function provide a method of assessing the effect of disease on the life of individuals from a physical and psychological perspective. ¹ They are utilised to standardise assessment and provide an objective measure of outcomes after surgery.^2,3 The use of PROMs has extended to the arena of healthcare policy to assess the effectiveness of treatments and the value of which they offer to the patient and clinician.^4,5

There are more than 30 different PROMs available to measure shoulder symptoms and function. ⁶ There is no “gold standard” PROM to assess general shoulder function. ² The choice of PROM used in different studies varies based on clinician's preference, with a clear geographical variation demonstrated in published literature. The American Shoulder and Elbow Surgeons Society Standardized Shoulder Assessment score (ASES) is commonly used in the United States of America and Canada. ⁷ In comparison, the Oxford Shoulder Score (OSS) is widely used in Europe and the United Kingdom. ⁸ It is also the shoulder arthroplasty outcome measurement questionnaire used by the New Zealand National Joint Registry. ⁹

One method of addressing this issue is to have patients complete multiple PROMs at each follow up time-point. The use of multiple PROMs and lengthy questionnaires have raised concerns regarding respondent burden and survey fatigue due to increased questionnaire lengths.^10–13 This has been proposed to lower response rates and negatively impact data quality.^14,15

There is little published literature comparing the results of these two questionnaires. Hapuarachchi and Poon demonstrated that total ASES score had good agreement with total OSS score in a population of 69 patients with cuff tear arthropathy that was managed with shoulder arthroplasty. ¹⁶ Wijeratna et al. highlighted that these questionnaires also showed good agreement in patients with glenohumeral osteoarthritis managed with shoulder arthroplasty. ¹⁷ Neither of these studies included an analysis of the scubcores of either outcome instruments in their findings. The sparse data makes it difficult to objectively compare and correlate the results of one study that utilizes the ASES with another study that uses the OSS.

The aim of this study is to assess the agreement between the ASES and OSS (total and subscore components) in patients diagnosed with various shoulder pathologies within a multi-surgeon private orthopaedic upper limb clinic. The second aim was to identify factors associated with agreement between the questionnaires in this population. It was hypothesised that (i) ASES and OSS total and subscore components would show significant agreement with each other and (ii) the amount of agreement between questionnaires would be significantly associated with patient and clinical factors.

Materials and methods

The analysis represents a cross-sectional study of baseline outcomes of shoulder pathology reported according to the RECORD guidelines ¹⁸ (Supplementary A). An observational analysis was performed on treatment records identified from a patient registry within a multi-surgeon private shoulder and elbow orthopaedic specialist practice. The practice is primarily based within an inner-city of a major capital city within Australia, with the participating surgeons also consulting at regional locations within the same state. This multi-cohort observational registry is registered on the Australian and New Zealand Clinical Trials Registry (ANZCTR), with ethics approval from an accredited Human Research Ethics Committee. Prospective data collection began in July 2017, with data recorded on all eligible patients up to data extraction in December 2019.

Inclusion and exclusion criteria

All patients who underwent baseline assessment for shoulder pathology, prior to definitive treatment, were included in the study. The exclusion criteria is highlighted in Table 1. In particular, repeat PROM records for patients who received definitive operative or non-operative treatment subsequent to baseline assessment were excluded from the analysis. Records with incomplete retrieval of PROM data were also excluded.

Table 1.

Inclusion and exclusion criteria.

Inclusion criteria	Patients in the practice registry receiving baseline assessment of shoulder pathology by their treating surgeon, prior to definitive treatment.
Exclusion criteria	Patients who revoked consent for research use of personal data.
	Patients with language difficulties which precluded retrieval of PROM data.
	Patients with clinically diagnosed mental illness which precluded retrieval of PROM data.
	Records with incomplete retrieval of PROM responses at pre-treatment assessment
	Repeat PROM assessments obtained following treatment.

The ASES, OSS and Veterans RAND 12-Item Health Survey (VR-12) were provided to patients via email or in the waiting room on a tablet device connected to a dedicated software package (Socrates, v3.9, 360KneeSystems, Australia). All patients underwent a clinical assessment by one of two fellowship trained orthopaedic shoulder surgeons, where a formal diagnosis was made. The patients were then placed into one of six cohorts based on this diagnosis (Table 2). Each cohort had a unique protocol for the collection of further outcome data, including any additional PROMs used and the timing of assessments. Where simultaneous pathologies were present, the cohort assignment was made based on the surgeon's opinion of the pathology that was responsible for the majority of the patient's presenting symptoms.

Table 2.

Patient diagnosis cohort.

Diagnosis Cohort	Diagnostic Algorithm
Acromioclavicular instability	Characterized by full or partial thickness tears of the acromioclavicular and/or coracoclavicular ligaments.
Glenohumeral osteoarthritis	Characterized by loss of articular cartilage, glenoid bone loss, and increased glenoid retroversion with posterior subluxation of the humeral head in advanced cases.
Glenohumeral instability	Characterized by loss of function or pain associated with excessive translation of the humeral head in the glenoid fossa. Includes traumatic capsulo-ligamentous injury, dissociation with associated glenoid fracture and atraumatic or voluntary dislocation, and recurrent dislocation.
Rotator cuff pathology	Characterized by shoulder pain during abduction of the arm between 60° and 120°. Includes tendinopathy, impingement of the cuff within the subacromial space, and full or partial thickness tears of any rotator cuff tendons.
Trauma	Proximal humerus, scapula or clavicle fractures.
General	Pathology not suitable for any of the other cohorts. Includes acromioclavicular arthritis, capsular contracture, and shoulder pain with no clear diagnosis.

Patient-reported outcome measurements

The ASES score was designed to be applicable to patients with shoulder disorders regardless of diagnosis.^6,19 It consists of a patient self-evaluation section (pASES) and a clinical examination section (cASES). The pASES comprises 11 questions relating to pain (1 item) and function (10 items). ¹ Both function and pain subscores are weighted equally (50% each) and combined for a maximum total score of 100% (100% being best). ¹ The minimum clinically important difference (MCID) for the total ASES varies according to the pathology present. It has been reported to be 12–17 points for rotator cuff pathology and 21 points for shoulder arthroplasty.^20,21

The OSS is a 12-item questionnaire that was designed to be applicable to all shoulder pathologies, other than shoulder instability. ²² It consists of four questions regarding pain and eight regarding function during activities of daily living. ²² The scoring system for the OSS was revised in 2009 so that individual scores are added to a total possible sum of 48 points, with 48 being the best.^6,23 This study uses the modified score. The MCID for the modified OSS has been variously reported to be from 2.7 to 14.7.^24,25

The Veterans RAND 12-Item Health Survey (VR-12) is a PROM used to estimate health-related quality of life, disease burden and disease-specific benchmarks with other populations. ²⁶ It asks 12 questions related to seven different health domains. The results are summarised into a physical component score (PCS) and a mental component score (MCS). The scores are standardised as a T score metric with a mean of 50.0 and a standard deviation of 10. ²⁶

Data access and cleaning

The research team had access to the full registry dataset collected within the practice from its inception to the date of retrieval. Exports from systems that underpin the registry (registry database, practice management system) were concatenated using custom scripts and assessed for completeness, consistency and response formatting. Outliers identified in continuous variables (body mass index, patient age). Diagnosis was recoded from comma delimited text arrays to categorical variables for rotator cuff tear, other rotator cuff pathology, glenohumeral pathology, and pathologies affecting the shoulder girdle (including acromioclavicular and sternoclavicular joints). A final combined dataset was then exported for further processing specific to this analysis. Questionnaire scores calculated by the registry database were transformed from strings to numeric format. Additional subscore calculations were made for the ASES and the OSS. Presentation type was recoded to primary; non-primary and insurance status was recoded to compensable; non-compensable. Hand dominance and index side were compared to indicate whether the treated shoulder was the dominant or non-dominant side for the patient, and was recoded to dominant; non-dominant. Missing data for PROMs were addressed with list-wise deletion. A sensitivity analysis was conducted on missing data by comparing those with complete ASES-OSS responses to those without complete responses (Supplementary B).

Statistical analysis

Patient demographics were summarised with median and interquartile ranges for continuous variables and proportions for categorical data. The groups were compared with univariate (independent t-test and Χ² analysis) and binary logistic regression to assess differences in VR-12 scores, age, sex, hand dominance, surgeon and insurance status. In the dataset with complete responses, ASES and OSS data were divided into total scores, as well as function and pain subscores. ²⁷ Linear regression was visualised with a fitted line plot to assess the relationship between the ASES and OSS scores. Goodness of fit was assessed using adjusted R², with 95% confidence and prediction intervals of the regression fit also calculated. Model residuals from the fitted lines were used to quantify agreement between the two scores. The residuals for each regression fit (total, function and pain) were transformed using a Box-Cox transformation with optimal lamba. A mixed-effects analysis of variance (ANOVA) was performed to assess the influence of factors on the agreement between scores. Diagnosis cohort, gender, surgeon, index side and insurance status were included in ANOVA as fixed effects. Age, date of examination and VR-12 PCS and MCS were included as covariates. Patient identifier was included as a random factor in the model. Model fit was assessed with adjusted R² and an alpha of 5% was used to indicate significance for all tests. Statistical analyses were conducted using a dedicated software package (Minitab v18, Minitab Inc, USA).

Results

Paired ASES and OSS outcome scores were available from 1050 patients after exclusion criteria were applied (Figure 1). The patient cohort was predominantly male (59.6%) with a median age of 55 years. Rotator cuff pathology was the most common diagnosis (45.9%) (Table 3).

Figure 1.

STROBE flow diagram.

Table 3.

Baseline characteristics of patients presenting with shoulder pathology.

Cohorts	Sample Size (N) (% of total)	Age	Female (%)	BMI (kg/m2)	Dominant side pathology (%)
Acromioclavicular instability	51 (4.8)	41 (30–55)	19.6	24.3 (22.9–27.7)	51
Glenohumeral osteoarthritis	78 (7.4)	66 (60–73.3)	50	28.4 (25.9–33.7)	41
Glenohumeral instability	112 (10.7)	25 (18–34)	25.6	26.4 (22–30.2)	42
Rotator cuff pathology	482 (45.9)	59 (51–69)	39	28.3 (24.8–32.1)	50.4
Trauma	102 (9.7)	47 (22.5–61.5)	48	24.9 (21.5–28.9)	41.2
General	225 (21.4)	50 (40–58)	53	26 (23.5–31.2)	45.8
Total	1050	55 (41–64)	41.3	27.1 (23.9–31.4)	47

Median and interquartile ranges used for continuous variables.

Agreement between ASES and OSS scores

The regression for total ASES and OSS scores exhibited good fit with an adjusted R² of 57.7% (P < 0.001). ASES and OSS function subscores also demonstrated good fit, with an adjusted R² of 63.9% (P < 0.001). In contrast, the pain subscore regression demonstrated a poorer fit (adjusted R² = 39.4%, P < 0.001) (Figure 2). The confidence intervals around the regression fits were sufficiently narrow to calculate crosswalks between mean scores for the total score and subscores of the two questionnaires (Table 4). However, the prediction intervals were too wide to allow these calculations to convert between individual scores. For example, the standard error of the estimate for the regression models ranged from 7.4 points on the ASES scale (function) to 14.1 (total score). Similarly, predictions for single future observations with respect to the absolute residual error between an individual score and the regression fit (crosswalk from OSS to ASES) ranged from 5.6 points (95% CI −3.7 to 15) for ASES function, up to 11.2 points (95%CI −5.7 to 28.1) for ASES total score (Figure 3).

Figure 2.

Baseline assessment of the distribution of ASES and OSS total, function and pain scores. CI = confidence interval, PI = prediction interval, R-Sq = R², R-Sq(adj) = adjusted R².

Table 4.

Crosswalk equations between mean total, function and pain ASES and OSS scores.

Score Type	ASES to OSS	OSS to ASES
Mean total score	7.2 + 0.4 ASES­­­total	9.7 + 1.5 OSStotal
Mean function subscore	25.8 + 1.7 ASESfunction	−2.0 + 0.4 OSSfunction
Mean pain subscore	7.1 + 1.0 ASESpain	13.2 + 0.4 OSSpain
ASEStotal = 1.9414 OSStotal – 5.8661* OSStotal = 0.4288 ASEStotal + 7.7199*

*Regression equation produced by Hapuarachchi and Poon. ³¹

Figure 3.

Model residuals from fitted line plots for rescaled ASES and OSS total, function and pain components.

Factors associated with agreement between scores

The mixed effects ANOVA demonstrated that agreement between total scores was influenced by both VR-12 MCS and PCS. Agreement between scores for pain was reasonably well explained by baseline VR-12 MCS, while agreement for the function subscore was significantly affected by VR-12 PCS, age, and cohort. (Table 5).

Table 5.

Mixed-effects analysis of variance for agreement between PROM scores (total, function and pain).

Model	Model fit (R2-adjusted %)	Predictors	Predictor P-value	Predictor coefficients (SE)
Total score agreement	36	VR-12 PCS VR-12 MCS	0.048 0.004	1.0 (0.5) 1.4 (0.5)
Function subscore agreement	38.1	VR-12 PCS Age at initial examination Diagnosis cohort Age × Cohort	<0.001 0.001 0.001 0.002	0.7 (0.3) -1.3 (0.4) GH instability -2.7 (1.3), Trauma 1.5 (0.6) GH instability -3.0 (0.9), Trauma 1.5 (0.6)
Pain subscore agreement	37.9	VR-12 MCS	<0.001	1.4 (0.4)

SE = standard error.

Discussion

The study demonstrates good agreement between mean total ASES and OSS scores, and their respective function subscores, in patients presenting with shoulder pathology to an orthopaedic specialist practice. Baseline VR-12 scores, age, and diagnosis cohort had a combined moderate influence (Table 5) on the agreement between the two questionnaires. These results indicate that other patient and/or clinical factors which may influence agreement have not yet been identified. The results of this study provide a method to convert the mean total and mean function ASES scores of a study population into their respective OSS scores (and vice-versa). This allows for the direct comparison of studies that have used either questionnaires as their primary outcome measure. It also aids in the interpretation of pooled data when conducting meta-analyses and systematic reviews. Moreover, the breakdown into “total”, “function” and “pain” subscores provides readers the flexibility to compare specific domains as required. The findings of this study can address the concerns with giving patients multiple, lengthy questionnaires resulting in response fatigue and reduced data quality. If a proposed trial is intending to collect both the ASES and OSS, then the current study suggests that only one of either two needs to be included.

The data showed good agreement between mean scores in a large population. At an individual level however, there was considerable variation in the agreement between the two questionnaires. When converting OSS to ASES in an individual patient, the absolute variance between the observed score and the regression fit (crosswalk from OSS to ASES) was up to 11.2 points for the ASES total score, which is close to the minimal clinically important difference for the ASES in rotator cuff pathology. ²⁰ This individual variability suggests further implications for the use of the ASES and OSS in the assessment of an individual patient's level of dysfunction. Specifically, caution is required if one intends to compare an isolated individual ASES or OSS to a mean score for a population, and then use that as a surrogate for clinical assessment. Any interpretation of this comparison must consider the variability associated with the agreement observed in the present findings. This is especially relevant given that outcome scores are being incorporated into arthroplasty registries, and have been used to drive health policy and to determine healthcare quality.^28,29

The mixed-effect ANOVA analysis demonstrated that VR-12 MCS, the patient's primary pathology and their age made some contribution to the variance in the agreement between the ASES and OSS. Overall the factors identified were able to explain 36–38% of this variance. The inconsistency between scores must thus be partly driven by other factors that this study has not been able to identify. Other authors who have examined agreement between different outcome scores have examined agreement between defined subgroups, such as age, gender or underlying diagnosis,^30–32 rather than run mixed-effect ANOVA analyses. Further work is required to identify what other factors influence agreement in different population subgroups.

Strengths and limitations

The strength of this study is its large patient cohort with various shoulder pathologies. Other studies of smaller patient populations have shown agreement between the ASES and OSS scores in shoulder arthroplasty.^16,17 The association between ASES and OSS demonstrated by Wijeratna et al. and Hapuarachchi and Poon's studies were only applicable to niche indications (glenohumeral joint osteoarthritis and cuff tear arthropathy respectively).^16,17 Our study of over 1000 patients, further subdivides this relationship into the respective PROM subscores, and demonstrates that this agreement holds for a variety of other shoulder pathologies.

Although the present study adds new information to patient-centred experiences of shoulder pathology, it should be interpreted in light of its limitations. Firstly, the present analysis only included questionnaire responses collected from patients prior to definitive treatment recommendations from their treating surgeon. There is data to suggest that ASES scores can take up to two years to reach their maximum value following rotator cuff surgery,^33,34 but it is not clear what the “maturation time” post treatment is for the ASES and OSS across all the respective shoulder pathologies. There were multiple different pathologies in the study population, therefore, it is possible that the agreement between the two scores would be different at a given time-point in the early period after treatment. This study was conducted using pre-operative data to avoid this potential confounding factor.

Secondly, the order in which the PROMs were completed was not monitored. Thus, the effect of question order bias cannot be excluded. This issue is not unique to this study and potentially affects all papers that examine agreement between various outcome scores. Some authors have examined the effect of positive or negative phrasing on a patient's response to outcome questionnaires,^35,36 however little is published on the relationship between responses to multiple PROMs and the order in which they are presented.

Thirdly, the subset of patients excluded from the analysis due to incomplete data may have introduced bias into the results and may limit the generalisability of the findings. Other authors that have examined the agreement between different PROMs have only reported the results of patients with complete data and have not mentioned how many cases of incomplete data were excluded.^30–32,37 The patients in this study were all presented with both questionnaires, but nearly a third of the otherwise eligible patients failed to complete one or both questionnaires. The concern is that the exclusion of incomplete data may have introduced “response bias”, resulting in a patient group that is not representative of the overall patient population. The subanalysis of the incomplete data group compared to the complete data group did show that incomplete data was more likely in male patients with rotator cuff pathology (Supplementary B). However, the incomplete data group showed no difference in mean ASES, and only small differences in mean OSS and VR-12 scores that were within the reported MCID threshold.^24,25 Although these differences were small, the authors believed it was appropriate to exclude the incomplete data group from the analysis. This study was designed as a benchmarking study to establish the relationship between the ASES and OSS. Thus, the authors believe that it was inappropriate to use statistical techniques such as imputation to “fill in the gaps” in the missing data group and then include them in the analysis.

Incomplete PROM data is a common issue that affects other regional and national registries,^38–47 reflecting the difficulty in obtaining complete patient data even when multi-level data collection systems are put in place. The finding that incomplete data was slightly more likely to be an issue in male patients with rotator cuff pathology may be of use in identifying patients that are “at risk” of incomplete data and directing more resources at them to ensure the data is correctly completed.

Conclusion

The study confirms the hypothesis that there is good agreement between mean ASES and OSS scores in patients with shoulder pathology, particularly for total scores and function subscore. This agreement allows for the conversion of one mean score to another. Clinicians will be able to compare studies which have used either PROMs as primary outcome measures. The findings of this study can also potentially reduce the number of questionnaires required from patients at each follow up time-point, thereby avoiding respondent burden and data fatigue. Baseline VR-12 scores, age and diagnosis cohort had combined moderate influence on the agreement between the questionnaires.