Does the Patient-Reported Outcomes Measurement Information System Correlate to Legacy Scores in Measuring Mental Health in Young Total Hip Arthroplasty Patients?

Abstract

Background

Mental health is becoming increasingly important in patient outcomes. The patient reported outcome measurement information system (PROMIS) was developed by the NIH to collect outcome data in rapid dynamic fashion on electronic platforms. The potential role of PROMIS in monitoring young total hip arthroplasty (THA) patients is under-investigated. The purpose of this study is to investigate correlations between PROMIS Depression or Anxiety with SF-12 mental component score.

Methods

We identified 223 hips (200 patients) who underwent primary THA over a 30-month period at a single institution. Patients without preoperative PROMIS or SF-12 mental scores, or >50yo were excluded. All data was collected preoperatively and included age, sex, BMI, ASA, PROMIS Depression, PROMIS Anxiety, and SF-12 Mental component score. We considered floor and ceiling effects as significant if >15% of patients responded with the lowest or highest possible score, respectively. Relationships between SF-12 and PROMIS were investigated using correlation (R), and were considered strong if R>0.7.

Results

Mean age was 41-years-old, mean ASA category was 2, mean BMI was 30kg/m2, and 54% were female. None of the PROMs showed any floor/ceiling effects at baseline. PROMIS Depression showed a strong correlation to SF-12 Mental (R=-0.72) while PROMIS Anxiety showed a moderate correlation to SF-12 Mental (R=-0.58). Negative linear relationships were observed because a lower PROMIS Depression/Anxiety values indicates less depressive/anxious feelings (inverse of SF-12).

Conclusion

PROMIS Anxiety and Depression correlate well with SF-12 mental. These PROMIS domains may be attractive alternatives to legacy mental health instruments in young THA patients.

Level of Evidence: III

Introduction

As the link between mental health and clinical outcomes after orthopaedic surgery grows stronger,^1-4 the need for well-established gold standard patient-reported outcome measures (PROMs) has become exceedingly necessary.⁵ Outcome measures are believed to provide an standardized and easily interpretable way for patients to communicate their mental health status to care providers. Unfortunately, over the last few decades, PROMs with varying scales have been proposed and used across medicine.^6-11 This creates barriers to efficient and effective patient care for physicians unfamiliar with the available PROMs, while also placing burden on patients who are required to complete multiple PROMs at every clinic visit, regardless of that provider’s area of practice. Furthermore, many of the current mental health PROMs used in orthopaedic surgery are broad in scope, clumping all mental health aspects into one all-encompassing score.¹² To better understand how the various facets of mental health affect clinical outcomes, we must possess the ability to measure them accurately and separately.

There are few established PROMs that measure some characteristic of mental health in current orthopaedic practice.^13,14 One such PROM is the Short Form 12 Health Survey (SF-12), which is a 12-item health survey given to patients as a paper form to complete pre-operatively and throughout their follow-up. The individual item raw scores are transformed to a 0-100 scale, which is then standardized using linear T-score transformation with a general population mean of 50, and a standard deviation (SD) of 10.^15-18 A higher score on the SF-12 indicates that a patient has “better health”. By weighing the separate item scores, a physical component score (PCS) and a mental component score (MCS) can also be obtained. Unlike item scores, these scores are calculated using all the questions for a comprehensive view of the patient’s mental or physical health status. The SF-12MCS is considered one of the gold-standards of mental health PROMs for orthopaedic surgery due to its versatility and easy-to-understand scores.^13,19 Unfortunately, the SF-12MCS only gives a global summary of the patient’s mental health, with a low score interpreted as meaning the patient has “frequent psychological distress, substantial social and role disability due to emotional problems.”^15-18 Therefore a low score can’t help in determining if the distress is more related to anxious feelings or those of depressive thoughts.¹³

Patient-Reported Outcomes Measurement Information System (PROMIS) was created by the National Institute of Health (NIH) to offer all healthcare providers a standardized global PROM that can be utilized for all patients, irrespective of their conditions or the physician treating them.^20,21 This new set of PROs utilize computer adaptive testing (CAT) and item-response theory (IRT) to quickly and accurately assess a patient’s health status with the least number of questions possible.²² Over 90 different PROMIS domains have been created so far ranging from generalized outcomes like the Global Health to highly detailed and specialized outcomes such as PROMIS-Cancer Fatigue. This presents healthcare providers a new opportunity to identify and monitor potential factors that may play a role in their patient’s health. PROMIS Depression (PROMIS-DEP) measures negative mood, views of self, social cognition, and decreased positive affect and engagement. PROMIS Anxiety (PROMIS-ANX) measures fear, anxious misery, hyperarousal, and somatic symptoms related to arousal. PROMIS uses T-score standardization with all PROMIS-DEP and PROMIS-ANX scores calibrated for and centered on a general population mean of 50 and SD of 10. Subgroup means and SD have also been reported by the NIH and are available online at the HealthMeasures website.²³

As with any new PROM, assessment of the validity and feasibility is paramount for its accurate use in various populations.^5,6,24 This can be achieved by validating the PROM in the general population and establishing the psychometric properties; a set of “quality criteria” have previously been proposed,²⁵ and more recently adapted for orthopaedic surgery.²⁴ These criteria have been used to demonstrate the validity of PROMIS across multiple patient populations^26-28 and medical specialties.^29-32 Nevertheless, some of those properties must be continually evaluated for a PROM to be deemed valid in a target population.^5,6 Little is known regarding the use of mental health domains of PROMIS in young patients undergoing primary THA, or their correlation to established gold standard mental health PROMs.

Due to the known association between mental health and outcomes after orthopaedic procedures, it is important to define the psychometric properties of PROMIS mental health domains, like PROMIS-DEP and PROMIS-ANX, as well as their relationship to SF-12 in individual patient populations. This may ensure better monitoring of various facets of mental health and facilitate future surgical decision-making and patient counseling. Therefore, in this study we 1) investigate the correlation of PROMIS-DEP and PROMIS-ANX with the SF-12 MCS, 2) provide validity for use of PROMIS-DEP and PROMIS-ANX in patients 50 years old and younger receiving a primary THA by analyzing baseline psychometric property values, and 3) establish baseline values for PROMIS-DEP and PROMIS-ANX in this cohort. We hypothesize that SF-12MCS will correlate strongly with PROMIS-DEP, moderately strong with PROMIS-A, and that both PROMs will demonstrate their validity. The reason for stronger correlation for depression but only moderately strong correlation for anxiety is because SF-12MCS identifies depressive symptoms better than anxiety symptoms.¹³

Methods

This study was approved by our institutional IRB and all patients included in the study provided consent to be involved. We conducted a retrospective cohort study of hips that underwent a primary THA at our single academic institution for any diagnosis over a 30-month period between May 2016 and October 2018. The review of hips was conducted by one of 3 fellowship-trained hip arthroplasty surgeons. Those older than 50 years at the time of surgery or without scores for both the SF-12 MCS and at least one of the mental health PROMIS domains (Depression and Anxiety) were excluded. All demographics and PRO were collected as standard of care at the preoperative visit during check-in. After the patient provided consent, was scheduled for surgery, and agreed to participate in the study, the data was loaded into our research database. Demographics consisted of age at surgery, sex, operative side, BMI, and prior ipsilateral surgery. PROs collected preoperatively included PROMIS-DEP, PROMIS-ANX, SF-12MCS, SF-12PCS, UCLA activity score, WOMAC Pain, WOMAC Physical Function, and WOMAC Stiffness. PROMIS-DEP and PROMIS-ANX scores were obtained by patient input on a tablet computer device (iPad mini 16GB, Apple, Cupertino, CA) preloaded with the PROMIS-DEP (version 1.0) and PROMIS-ANX (version 1.0) computer adaptive tests (CATs) using an internally developed software while the remaining PROMs were collected via patient-completed paper questionnaire packets.

We assessed the data for descriptive statistics to summarize patient characteristics, and to ensure normal distribution before additional analysis. Presence of any apparent floor or ceiling effects were assessed using histograms; floor or ceiling effects were considered significant if 15% or more of the patients responded with the lowest or highest possible score, respectively.²⁵ We further analyzed all score distributions for the effect from demographic characteristics. Correlations between all PROMS were assessed using Spearman-rank correlation (R) and simple linear regression was conducted between PROMIS-DEP and PROMIS-ANX against SF-12MCS to investigate correlation, variation accounted for by the correlation, standard error (SE), significance, confidence intervals, general linear relationships, and floor or ceiling effects not identified on histograms. Residual plots were investigated to determine the presence of confounding variables. Correlation coefficients were interpreted as follows: 0 to 0.3 representing a negligible correlation, 0.3 to 0.5 representing weak correlation, 0.5 to 0.7 representing moderate correlation, and >0.7 representing strong correlation.³³ The amount of variation accounted for by the correlation was determined by the coefficient of determination (R²). The value of R² ranges from 0.0 to 1.0 where 1.0 represents 100% of the variation seen in the sample was explained by the correlation. We established through a priori power analysis that a sample size of 85 hips is necessary to show a weak correlation (R = 0.3) with 80% power and statistical significance. Construct validity was assessed by testing pre-determined hypotheses of how the PROMIS domains would correlate to PROMs measuring varying aspects of health and function. Interpretability was assessed with the baseline means, SD, and the coefficients of variation (CV, =SD/ Mean) of the cohort to investigate how they relate to the general population, as well as to the same properties in SF-12MCS. Due to the demographic characteristics of our cohort, we chose to compare all our data to the general population values for PROMIS and forego detailed analysis based on the subgroup distributions. A p-value <0.012 was considered statistically significant according to Bonferroni correction.

Results

Of the 1124 hips with primary THA, 223 total hips (in 200 patients) met inclusion criteria and were included in the final analysis (Figure 1). The mean age at surgery was 40.8 years old (±8.7 years), with a mean BMI of 30.4 (±6.4). From the 223 hips included, 97 (43.5%) were male, 109 (48.9%) had surgery on the right hip, and 69 (30.9%) had surgery on their index hip prior to receiving THA (Table 1). The majority (144 hips, 65%) presented with degenerative osteoarthritis as their sole diagnosis, 28 (12.6%) presented due to avascular necrosis and 6 (2.7%) presented post-trauma (Table 2).

Figure 1.

Flowchart demonstrating reasons for patient exclusions. THA, Total Hip Arthroplasty; Sx, Surgery; Pts, Patients; PROMIS, Patient-Reported Outcomes Measurement Information System.

Table 1.

Demographic Characteristics of the Cohort

Mean (SD)
Characteristic	Cohort (N= 223)
Age at Surgery	40.8 (8.7)
ASA	2 (0.5)
BMI	30.4 (6.4)
	No. (%)
Male Sex	97 (43.5%)
Right Side	109 (48.9%)
Prior I/L Sx	69 (30.9%)

No, Number; SD, Standard Deviation; Sx, Surgery; BMI, Body Mass Index; I/L, Ipsilateral.

Table 2.

Diagnoses of the Cohort

Dx	No. (%)
Degenerative OA	144 (65.0%)
Hip Dysplasia	38 (17.0%)
AVN	28 (12.6%)
Post-Traumatic	6 (2.7%)
Epiphyseal Dysplasia	5 (2.2%)
Impingement	1 (0.4%)
Rheumatoid Arthritis	1 (0.4%)

Diagnoses of the cohort. No, Number.

Regression analysis revealed that PROMIS DEP demonstrated a strong correlation with SF-12MCS at baseline (R=-0.71, 95% CI -0.67 to -0.88) while PROMIS ANX only had a moderate correlation (R=-0.58, 95% CI -0.59 to -0.89) with SF-12MCS at baseline (Figures 2A and 2B). The moderate R² value for PROMIS DEP and SF-12MCS (R²=0.50) demonstrates that the correlation and regression account for 50% of the variance between scores. Although these 2 PROMs are measuring roughly half of the same parameters, PROMIS-DEP is likely obtaining information that the SF-12MCS does not. The R² value for PROMIS ANX and SF-12MCS (R²=0.33) was even lower, indicating that these PROMIS-ANX is likely capturing information that SF-12MCS is not.

Figure 2A.

Scatter plot of pre-operative PROMIS Depression scores versus SF-12 Mental scores with linear regression line which yielded R2=0.5008. R2, Goodness of fit measure; R, Correlation Coefficient; SE, Standard Error; SF-12 MCS, Mental Component Score; PROMIS, Patient-Reported Outcomes Measurement Information System.

Figure 2B.

Scatter plot of pre-operative PROMIS Anxiety scores versus SF-12 Mental scores with linear regression line which yielded R2=0.333. R2, Goodness of fit measure; R, Correlation Coefficient; SE, Standard Error; SF-12 MCS, Mental Component Score; PROMIS, Patient-Reported Outcomes Measurement Information System.

Correlations of PROMIS-DEP, PROMIS-ANX, and SF-12MCS with SF-12PCS, UCLA activity score, WOMAC Pain, WOMAC Physical function, and WOMAC Stiffness are shown in Table 3. When comparing the correlations for PROMIS-DEP, PROMIS-ANX, and SF-12MCS (each of these against any one specific legacy PROM for pain or physical function), all R values fall within the same tier of correlation grading and also demonstrate overlapping confidence intervals, signifying no significant difference between the 3 mental scores and their various correlations to physical function and pain PROMs (Table 3).

Table 3.

Correlation of All PROMs

	R
PROM	PROMIS- DEP	PROMIS- ANX	SF-12MCS
SF-12MCS	-0.71	-0.58	1.00
SF-12PCS	-0.11	-0.18	0.01
UCLA Activity Score	-0.19	-0.12	0.21
WOMAC Pain	-0.17	-0.20	0.22
WOMAC Physical Function	-0.22	-0.23	0.29
WOMAC Stiffness	-0.18	-0.22	0.18

Correlation of pre-operative scores for PROMIS-DEP, PROMISANX, and SF-12MCS against WOMAC Pain, WOMAC Physical Function, WOMAC Stiffness, UCLA activity score, and SF-12PCS. R, Correlation coefficient; SD, Standard Deviation; PROM, Patient Reported Outcome Measure; PROMIS-DEP, Patient-Reported Outcomes Measurement Information System Depression; PROMISANX, Patient-Reported Outcomes Measurement Information System Anxiety; SF-12MCS, Short Form 12 Health Survey Mental Component Score; SF-12PCS, Short Form 12 Health Survey Physical Component Score; WOMAC, Western Ontario & McMaster Universities Osteoarthritis Index.

At baseline, all 3 of the PROMs demonstrated means within 1 SD of 50 (PROMIS-DEP=49.5, PROMISANX=54.3, SF-12MCS=52.1, Table 4). No ceiling effects were seen in any PROM, and only PROMIS DEP showed a floor effect with 16.59% (37/223) of hips achieving the lowest possible score (34.2)(Figure 3A). Distribution was otherwise normal for all PROMs, as seen in the histograms (Figures 3B and 3C) and confirmed with Shapiro-Wilk analysis. Demographic characteristics analyzed show no effect on score distribution. Scores were also stratified by age groups and showed no significant differences (Table 4).

Table 4.

Cohort Scores, All Hips and by Age Subgroup

	Mean (SD)
PROM	All Hips	18-34	35-44	45-50	P-value
PROMIS-DEP	49.5 (10.5)	50.5 (10.4)	47.6 (12)	50.2 (9.3)	0.360
PROMIS-ANX	54.3 (9)	54.3 (9.2)	53.4 (9.5)	54.8 (8.3)	0.336
SF-12MCS	52.1 (11.6)	51.2 (11.9)	52.3 (11.5)	52.5 (11.5)	0.819

Characteristics of pre-operative scores for PROMIS-DEP, PROMIS-ANX, and SF-12MCS. SD, Standard Deviation; PROM, Patient Reported Outcome Measure; PROMIS-DEP, Patient-Reported Outcomes Measurement Information System Depression; PROMIS-ANX, Patient-Reported Outcomes Measurement Information System Anxiety; SF-12MCS, Short Form 12 Health Survey Mental Component Score.

Figure 3A.

Histogram of PROMIS Depression scores. Orange dots signify the percentage of hips from the cohort that obtained that respective score. Those with > 15% were considered to demonstrate a floor or ceiling effect. PROMIS, Patient-Reported Outcomes Measurement Information System.

Figure 3B.

Histogram of PROMIS Anxiety scores. Orange dots signify the percentage of hips from the cohort that obtained that respective score. Those with > 15% were considered to demonstrate a floor or ceiling effect. PROMIS, Patient-Reported Outcomes Measurement Information System.

Figure 3C.

Histogram of SF-12 Mental Component scores. Orange dots signify the percentage of hips from the cohort that obtained that respective score. Those with > 15% were considered to demonstrate a floor or ceiling effect. PROMIS, Patient-Reported Outcomes Measurement Information System.

Discussion

Recent research has demonstrated the importance of considering mental health in patients undergoing surgery. Nonetheless, very little information is currently available on the effects that mental health status can have on clinical outcomes of patients undergoing orthopaedic surgery. Furthermore, the availability of PROMs that measure different aspects of mental health, or global mental health is limited. Current gold-standard PROMs of mental health include the SF-12MCS and the EQ-5D, both of which give a single score that encompasses the entirety of the patient’s mental health. To our knowledge, this is the first study to analyze the correlation of PROMIS mental health domains with SF-12MCS scores in patients <50-years-old receiving primary THA.

Our objectives in this study were to investigate the correlation of PROMIS-DEP and PROMIS-ANX with the SF-12MCS , to validate PROMIS-DEP and PROMIS-ANX for use in patients 50 years old and younger receiving a primary THA by analyzing baseline psychometric property values, and to establish baseline values for PROMIS-DEP and PROMIS-ANX in this cohort. Based on our findings, we believe that PROMIS-ANX is valid for use in this cohort to establish a baseline anxiety status and that the average patient in this cohort falls within an acceptable range of normal anxiety when compared to the general population. With regards to PROMIS-DEP, the data suggests that it is partially valid for use in this population and may still be useful for monitoring some patients, especially those with initially high scores. Despite showing stronger correlation than PROMIS-ANX, a lack of floor effect places some limitation to the validity of PROMIS-DEP for use in this population as discussed later. Nevertheless, PROMIS-DEP still demonstrated a mean in our population that is within an acceptable range of the normal general population. Although these findings support the use of PROMIS mental health domains as an alternative to legacy mental health measures in young patients undergoing THA, additional work is needed to obtain follow-up scores in this population to fully validate PROMIS-DEP and PROMIS-ANX.

The strong correlation of PROMIS-DEP to the SF-12MCS lends to its validity, while the moderate goodness-of-fit demonstrates that PROMIS-DEP and SF-12MCS are measuring many, but not all, of the same parameters. The weak and negligible correlations to the remaining PROMs provide further validity, showing that PROMIS-DEP is indeed capturing information about the patient’s depressive symptoms and not using other health aspects as surrogates. Unfortunately, PROMISDEP did demonstrate a floor effect, which places some limitations to its use in patients with very low scores pre-operatively and should be the subject of future research.

Meanwhile, the moderate correlation with small R² for PROMIS-ANX and SF-12MCS demonstrates that these 2 PROMs are measuring somewhat different parameters of mental health. This finding was expected and agrees with reports that SF-12MCS does a better job at picking up depressive symptoms more than anxiety symptoms.¹³ PROMIS-ANX was able to demonstrate correlations to PROMs of physical health that were comparable to SF-12MCS and PROMIS-DEP. It also showed no floor or ceiling effects, making it an effective tool for establishing baseline anxiety status. Some groups have demonstrated the validity of PROMIS-ANX by utilizing the EQ-5D in other patient populations but due to the lack of availability of a true gold-standard legacy measure for anxiety in orthopaedic populations, we were limited in the PROMs used for correlation and validation.

To utilize these PROMs in our specific populations, it’s necessary to establish baseline values and to investigate their interpretability when compared to the general population. Previous literature has shown that the average patients presenting for primary THA have depression and anxiety symptoms at rates similar to or less than the general population, indicated by their SF-12MCS scores.^34,35 Both PROMIS domains had means within 5 points (1/2SD) of the general population means and smaller SDs than SF-12MCS. This shows that PROMIS domains may have the ability to detect smaller changes in health status and likely have a smaller MCID than SF-12MCS, although actual MCID values must still be established in this cohort. It also shows that when it comes to feelings and thoughts of anxiety or depression, this cohort performs similarly to the general population. The CV for PROMIS-DEP (0.21) and PROMIS-ANX (0.17) compare well with the general populations (0.2), signifying that PROMIS-DEP exhibits distribution in this cohort that is nearly identical to the general population, and PROMIS-ANX exhibits less variation in this cohort when compared to the general population. Taken together with the easy-to-understand scoring system and standardized means and SD, PROMIS-DEP and PROMIS-ANX prove to be easily interpretable PROMs. A cursory analysis of our PROMIS values stratified by age and compared to values of PROMIS for the appropriate subgroups, rather than general population values, further supported the above findings.

Limitations

Two large limitations exist that constrain our abilities to fully validate these PROMs in our cohort. First, the lack of follow-up scores prevents us from performing other important validation criteria- reliability and responsiveness, or from establishing true MCID values. Second, the lack of a gold-standard mental health PROM in orthopaedic surgery for anxiety limited our ability to evaluate the validity of PROMIS-ANX, although this has been done previously in other populations.

Conclusion

Our findings support the use of PROMIS mental health domains as an attractive alternative to current mental health legacy scoring instrument in young patients undergoing THA, given their moderate to strong correlations with SF-12MCS, demonstrated validity, and easily interpretable scores. Further work is necessary to obtain follow-up scores in this cohort and to fully validate PROMIS-DEP and PROMIS-ANX for use in this cohort.