Minimal clinically important difference, substantial clinical benefit, and patient acceptable symptom state of PROMIS upper extremity after total shoulder arthroplasty

Abstract

Background

The Patient-Reported Outcomes Measurement Information System minimal clinically important difference (PROMIS MCID), substantial clinical benefit (SCB), and patient acceptable symptom state (PASS) of patient-reported outcome measures provide clinical significance to patient-reported outcome measures scores. The goal of this study is to measure the MCID, SCB, and PASS of PROMIS Upper Extremity v2.0 (PROMIS UE) in patients undergoing total shoulder arthroplasty (TSA).

Methods

All patients who underwent TSA since October 2017 were identified from our institutional database. Patients who had completed the PROMIS UE outcome measure before surgery were asked to complete a PROMIS UE and anchor survey that contained two transition questions to assess patient satisfaction and change in symptoms since treatment. The anchor-based MCID, SCB, and PASS were calculated as the change in PROMIS UE score that represented the optimal cutoff for a receiver operating characteristic curve. The distribution-based MCID was calculated as a range between the average standard error of measurement multiplied by 2 different constants: 1 and 2.77.

Results

This study enrolled 165 patients. The anchor-based MCID for PROMIS UE was calculated to be 8.05 with an AUC of 0.814. The anchor-based SCB was calculated to be 10.0 with an AUC of 0.727. The distribution-based MCID was calculated to be between 3.12 and 8.65. The PASS was calculated to be 37.2 with an AUC of 0.90.

Conclusions

The establishment of MCID, SCB, and PASS for PROMIS UE scores after shoulder arthroplasty provides meaningful and objective clinical interpretation of the improvements in outcome scores after TSA.

Patient-reported outcome measures (PROMs) play an important role in orthopedic surgery, allowing clinicians to measure aspects of a patient’s condition that cannot be captured by a physical examination and/or radiographic study. For this reason, numerous PROMs have been developed over the last few decades, with over 30 dedicated to measuring shoulder health and function alone.¹ In an effort to improve and standardize outcome reporting in clinical research and practice, the National Institutes of Health created the Patient-Reported Outcomes Measurement Information System (PROMIS) in 2004.²⁷ PROMIS uses item response theory and computer-adaptive testing to accurately capture patient outcomes with the minimum number of questions possible, thus reducing redundancy, response burden, and time of administration.⁴⁴ Scores are reported on a normalized 0-to-100 scale, where 50 represents the population average with a standard deviation of 10.^14,39 PROMIS instruments are designed to measure one of the domains of physical, mental, and social health, with the physical health domain further divided into physical function, pain intensity, pain interference, fatigue, and sleep disturbance.²⁷ PROMIS Upper Extremity (PROMIS UE) was designed to specifically focus on measurement of the upper extremity physical function subdomain.³⁹ The widespread adoption of this instrument in orthopedic surgery research will better enable investigators to compare and synthesize the outcomes of research studies on a variety of upper extremity conditions and their treatments.⁸

In order to properly use PROMIS UE in research and clinical practice, it is necessary for clinicians to be able to meaningfully interpret changes in PROMIS UE scores.²¹ Because PROMs are designed to measure a patient’s subjective experience of their condition, the resulting PROM scores are subject to natural variability between timepoints even if the patients' underlying condition has not changed in any clinically meaningful way.³ It is therefore essential to establish benchmarks against which clinicians can evaluate the clinical relevance of changes in a PROM score. Benchmarks commonly used for this purpose are the minimal clinically important difference (MCID), the substantial clinical benefit (SCB), and the patient acceptable symptom state (PASS). The MCID is defined as the minimum change in a PROM score that signifies a clinically significant improvement, whereas the SCB is the change in score that signifies a substantial or optimal improvement.^31,32,35,37 PASS is defined as the final PROM score at which patients are satisfied with their symptoms.^29,31,32

The two most commonly used approaches for the calculation of MCID are distribution-based and anchor-based.^19,28,35 Distribution-based approaches calculate an MCID that is based on the statistical variability in a group’s baseline PROM scores and is meant to represent the smallest measurable change that is unlikely to be due to random variation.^18,21 Anchor-based approaches use an external scale against which a change in PROM score can be compared to determine the clinical significance of the change. One commonly used anchor is a global rating of change survey that asks the patient to rate how their condition has changed from baseline using a Likert scale.¹⁹ The point in the scale that represents a clinically meaningful change—usually “a little better” or “better”—can then be used to establish the MCID.³⁵ SCB can be calculated using an anchor-based approach by selecting a reported change (ie, “much better”) that represents an optimal improvement.³² PASS can be similarly calculated using a separate self-assessment question that asks patients to rate their satisfaction with their current symptoms or outcome of their treatment.^29,32

MCID, SCB, and PASS are context-dependent values that vary based on the condition, treatment, and patient population in which they are calculated.^29,35 It is therefore necessary to calculate a PROMs MCID, SCB, and PASS values in a variety of patient populations to provide clinicians with appropriate benchmarks for use in research and clinical practice. To date, there have been relatively few studies on the MCID, SCB, and PASS values for PROMIS UE in specific upper extremity orthopedic patient populations, with the available literature limited to rotator cuff repair,¹⁷ biceps tenodesis,¹³ distal radius fractures,³⁴ carpal tunnel release,^4,20 and nonshoulder hand and upper extremity patients.²¹ Given the scarcity of published data in this area, the goal of our study is to calculate the MCID, SCB, and PASS for PROMIS UE in patients undergoing total shoulder arthroplasty (TSA) using both distribution- and anchor-based methods.

Methods

Ethics

This study was approved by the NYU Institutional Review Board. All study subjects provided informed consent before enrollment.

Study design

This study prospectively collected data on a retrospective cohort of patients that had undergone either anatomic or reverse TSA at NYU Langone Health between October 1, 2018, and February 1, 2020. Subjects for this study were recruited via a telephone call. Patients were considered for inclusion if they were older than 18 years, had completed the PROMIS UE instrument preoperatively, were not undergoing a revision surgery, had no subsequent upper extremity surgery or trauma, and were willing and able to provide informed consent. Subjects were excluded if they were unable to communicate in English or if they had a mental handicap that prevented them from providing informed consent or completing the study surveys.

Data collection

Subjects were asked to complete three surveys. The first asked if patients had any upper extremity injury surgery since the date of their shoulder arthroplasty surgery. The second and third surveys were the PROMIS UE v2.0 instrument and the anchor survey. Study data were collected either over the phone or online, based on patient preference, and managed using REDCap electronic data capture tools hosted at NYU Langone Health.^15,16 The option to complete the surveys by phone was included to avoid response and sampling bias due to the potential exclusion of patients that are unable or unwilling to complete the surveys online.^10,24,33,43

Patient-reported outcome measures

All subjects completed the PROMIS UE v2.0, an upper extremity–focused subset of questions from the PROMIS Physical Function computer-adaptive testing.

Anchor questions

The anchor survey asked patients two separate questions. The first question asked patients to assess the change in their physical function since surgery using a global rating of change question. The question was phrased, “Compared to before surgery, how would you describe the physical function of your shoulder now?” Responses were chosen from a 7-point Likert scale as follows: Much Worse, Worse, Slightly Worse, No Change, Slightly Better, Better, or Much Better. The second question asked patients to rate their satisfaction with the outcome of their treatment. The question was phrased, “How satisfied are you with the treatment result of your shoulder?”. Responses were chosen from a list of 5 choices: Very Satisfied, Satisfied, Neither Satisfied or Dissatisfied, Dissatisfied, or Very Dissatisfied.

Statistical analysis

All statistical analysis was performed with R version 3.6.2. The distribution-based MCID was calculated as a range using the standard error of measurement of the baseline PROMIS UE scores multiplied by two constants: 1 and 2.77.³⁵ The anchor-based MCID and SCB were calculated using receiver operating characteristic (ROC) analysis. This method calculates the MCID and SCB as the threshold change in score that can most accurately classify patients reporting a specific transition rating on the anchor question, which for the present study was set as “Better” for MCID and “Much Better” for SCB.³⁵ Although there is no gold standard methodology for calculating anchor-based MCID and SCB, ROC analysis is often considered to be the most unbiased and valid method and is thus used in a number of studies and systematic reviews seeking to calculate these benchmark values.^{6,9,26,29,32,40,41} PASS was similarly calculated as the postoperative PROMIS UE score that can most accurately classify patients reporting the transition ratings “Satisfied” or “Very Satisfied”. The choice of these transition ratings is in line with previous MCID and SCB calculations in shoulder surgery⁹ and lower back pain²³ patient populations and with PASS calculations in proximal interphalangeal joint arthroplasty.²⁹ The optimal cutoff point was determined using Youden’s Index.^11,18,28,35 Cutoff point calculations were deemed to be “acceptable” if the area under the curve (AUC) was greater than 0.7, with values exceeding 0.8 considered “excellent”.¹¹

Results

A total of 201 subjects met inclusion/exclusion criteria for this study, and 165 (82.1%) completed the study surveys (Fig. 1). Patient demographics are shown in Table I.

Figure 1.

Patient recruitment diagram.

Table 1.

Demographics.

	Overall (N = 165)
Age (yr)
Mean (SD)
Sex
Female	93 (56.4%)
Male	72 (43.6%)
Race
American Indian/Alaska Native	0 (0%)
Asian	2 (2.4%)
Black or African American	7 (8.3%)
White	58 (69.0%)
More than one race	1 (1.2%)
Unknown/not reported	16 (19.0%)
BMI
Mean (SD)	30.1 (6.99)
Smoking status
Never	78 (47.3%)
Current	8 (4.8%)
Former	79 (47.9%)
Affected shoulder
Left	74 (44.8%)
Right	91 (55.2%)
Days since surgery
Mean (SD)	472 (258)
Median [min, max]	484 [79.0, 931]

SD, standard deviation; BMI, body mass index.

Out of the 165 subjects who completed the anchor-based survey, 104 reported that their condition was “much better”, 24 reported “better”, 27 reported “slightly better”, 6 reported “no change”, 1 reported “slightly worse”, and 3 reported “worse”. No subjects reported that their condition was “much worse”. For the question to assess patient satisfaction, 118 reported “very satisfied”, 32 reported “satisfied”, and 11 reported “neither satisfied nor dissatisfied”, 3 reported “dissatisfied”, and 1 reported “very dissatisfied”. The anchor-based MCID calculated for subjects reporting a change of at least “better” was 8.05, with an excellent AUC of 0.814. The distribution-based MCID was calculated to be between 3.12 and 8.65. The SCB calculated for subjects reporting a change of “much better” was 10.0 with an acceptable AUC of 0.797. The PASS calculated for patients reporting “very satisfied” or “satisfied” was 37.2 with an excellent AUC of 0.900. These values are summarized in Table II. Plots displaying the optimal cutpoint analysis and ROC curves for the MCID, SCB, and PASS calculations can be seen in Figure 2, Figure 3, Figure 4, respectively.

Table 2.

MCID, PASS, and SCB.

	Value	Accuracy	Sensitivity	Specificity	AUC
MCID	8.05	0.752	0.734	0.811	0.814
SCB	10.0	0.727	0.673	0.820	0.797
PASS	37.2	0.764	0.747	0.933	0.900

MCID, minimal clinically important difference; PASS, patient acceptable symptom state; SCB, substantial clinical benefit; AUC, area under curve.

Figure 2.

MCID optimal cutpoint analysis and receiver operating characteristic curve. (A) MCID optimal cutpoint analysis and class distribution. Subjects in the “Improved” group reported a change of “Better” or “Much Better”. Subjects in the “Not Improved” group reported a change of “Slightly Better” or Lower. (B) Receiver operating characteristic curve for the MCID calculation. MCID, minimal clinically important difference; PROMIS UE, Patient-Reported Outcomes Measurement Information System Upper Extremity.

Figure 3.

SCB optimal cutpoint analysis and receiver operating characteristic curve. (A) SCB optimal cutpoint analysis and class distribution. Subjects in the “Much Improved” group reported a change of “Much Better”. Subjects in the “Not Much Improved” group reported a change of “Better” or Lower. (B) Receiver operating characteristic curve for the SCB calculation. SCB, substantial clinical benefit; PROMIS UE, Patient-Reported Outcomes Measurement Information System Upper Extremity.

Figure 4.

PASS optimal cutpoint analysis and receiver operating characteristic curve. (A) PASS optimal cutpoint analysis and class distribution. Subjects in the “Satisfied” group reported that they were “Satisfied” or “Very Satisfied”. Subjects in the “Not Satisfied” group reported they were “Neither Satisfied nor Dissatisfied” or Lower. (B) Receiver operating characteristic curve for the PASS calculation. PASS, patient acceptable symptom state; PROMIS UE, Patient-Reported Outcomes Measurement Information System Upper Extremity.

Discussion

PROMIS UE is becoming an increasingly popular PROM in shoulder and elbow surgery. Statistical significance of outcome scores does not always equate with clinical significance, and consequently, establishment of MCID, SCB, and PASS is critical for the meaningful and objective interpretation of outcome measure in published studies and in clinical practice. In this study, we have established the MCID, SCB, and PASS for the upper PROMIS UE outcome scores in patients who underwent TSA.

MCID provides a context for the evaluation of the clinical relevance of PROM score changes reported in the literature, as even minor improvements in PROM scores can be made statistically significant if the sample size is sufficiently large.^29,35 It also helps in the planning of new studies, where researchers can use the MCID to help select a clinically relevant effect size for use in power analysis and later during data analysis to analyze the impact of patient-specific factors on the likelihood of response to a particular treatment.²⁸

Complementary to MCID, which sets a lower bound for clinically meaningful improvement, SCB represents the threshold for improvement that patients would consider optimal. Together, these values can be used to define a range of outcomes—from the smallest clinically meaningful change to the ideal treatment response—that add additional context to the meaning of PROM score changes.³¹ While MCID and SCB aid in the interpretation of PROM score changes, PASS on the other hand represents the absolute PROM score beyond which patients would be satisfied with their symptoms.³⁸ It can be used by researchers, for example, to classify subjects into “responder” and “nonresponder” groups, and by clinicians to help set treatment goals and manage patient expectations.^30,38 The applicability of these values is context-specific, dependent on both the method of calculation used and the patient population used for calculation.^2,12 Ideally, these values should be calculated using multiple approaches in a patient population that is similar to the population in which they will be applied.^25,35 As there are no currently published values for MCID, SCB, and PASS for PROMIS UE v2.0 after TSA, our study aims to establish these benchmark values using prospectively collected data on a retrospective cohort.

Our anchor-based estimate for the MCID and SCB for PROMIS UE after TSA is 8.05 and 10, respectively. PASS was calculated to be 37.2. Both the MCID and PASS values were deemed to be “excellent” with AUCs greater than 0.8, while the SCB calculation was found to be “acceptable” with an AUC of 0.797. The distribution-based MCID calculated in this study, which ranges between 3.12 and 8.65, helps to serve as an independent confirmation that our anchor-based MCID calculation falls within the proper range of expected values.^9,35 While the anchor-based MCID of 8.05 for PROMIS UE in TSA calculated here is larger than those previously calculated in rotator cuff repair (4.87),¹⁷ biceps tenodesis (4.02),¹³ distal radius fractures (3.6 to 4.6),³⁴ and nonshoulder hand and upper extremity patients (2.1),²¹ it is similar in magnitude to the MCID (6.3 to 8.0) calculated for carpal tunnel release.⁴ Interstudy variations such as these in the calculation of these clinical benchmark values are expected given the differences in the patient populations and calculation methods used.¹⁹

There are several limitations to this study. First, as there is no clear consensus on the best methodology to calculate MCID, SCB, and PASS, one limitation of the present study is that there is no clear way to reconcile differences between the values presented here and those calculated in other studies.^12,35 The use of a different statistical method to calculate these values may lead to different results, as could changes in the wording of the anchor questions.^20,36 The inclusion of the option to complete the surveys over the phone may also have an effect on the PROMIS UE scores collected. However, although one recent study did find that the method of survey administration has a clinically significant effect on the measurement of PROMIS UE scores,⁴² numerous studies of a variety of PROMIS instruments found no such difference.^5,7,22 Another limitation of the present study is the mean follow-up period of 472 days. A study with a shorter or longer mean follow-up period may yield different results, due to both the longitudinal changes in subjects’ physical function after surgery as well as the increase in recall bias inherent to longer follow-up periods in anchor-based MCID calculations.^18,19 Finally, there is an inherent limitation in the utility of these benchmark values in the assessment of clinical meaningful change. In particular, it is important to remember when applying these values in clinical settings that they are not meant to be universal thresholds. The definition of what constitutes a clinically meaningful change varies between patients and should therefore be judged in a patient-specific fashion when dealing with individuals.¹⁹

Conclusion

The MCID, SCB, and PASS values for PROMIS UE after TSA proposed in this study are a useful addition to the growing body of work aimed at improving the meaningful clinical utility and interpretability of PROMIS UE in everyday clinical practice and research settings. These values can aid orthopedic surgeons in the interpretation of published studies and the planning of future research and can help improve their understanding of their own patients’ PROMIS UE scores after surgery.

Disclaimers

Funding: No funding was disclosed by the author(s).

Conflicts of interest: The authors, their immediate families, and any research foundation with which they are affiliated did not receive any financial payments or other benefits from any commercial entity related to the subject of this article.