A comparison of PROMIS Global Health-Mental and legacy orthopedic outcome measures for evaluating preoperative mental health status

Abstract

The influence of mental health on postoperative outcomes necessitates comparison of the mental health constructs captured by patient reported outcome measures (PROMs). 1,026 patients completed preoperative PROMs including the PROMIS Physical Function (PF), PROMIS Global Health (GH), and a subspecialty-specific PROM. Across specialties, PROMIS GH-Physical, PF-CAT, and PF-SF scores correlated more strongly with all legacy PROMs than the GH-Mental score. Moderate to strong correlation between the PF-SF, PF-CAT, and GH-Physical questionnaires and subspecialty-specific PROMs across specialties suggest they assess similar concepts. Lower correlation between the GH-Mental and subspecialty-specific PROMs suggest that the GH-Mental questionnaire captures distinct patient-reported mental health.

1. Introduction

The increasingly common administration of validated patient-reported outcomes measures (PROMs) in clinical orthopedic settings, exemplified by proliferation of Patient-Reported Outcomes Measurement Information System (PROMIS),¹ has yielded a significant amount of patient-centered data. PROM-centered orthopedic outcomes research has targeted pre-operative subjective assessments—in conjunction with objective clinical assessments—that might predict post-operative outcomes. Further evaluation of pre-operative PRO dimensions is essential to enhance the efficiency of how these instruments are utilized in clinical practice.

PROMs are used to assess latent constructs such as impacts on physical function, daily living, emotions, or anxiety. The resulting score from a well-developed and validated PROM reveals information about how much of a latent construct is present, or how much it varies.² Therefore, the correlations between scores on one PROM with another can give us a sense of the association between the concepts evaluated by those instruments.

Two main correlation coefficients used in biomedical research are Pearson's product-moment correlation coefficient (r) and Spearman's rank correlation coefficient (rho). Assumptions for calculating Pearson's r include continuous-level variables, linearity, and normality (i.e., a normal distribution of scores) of both variables. Pearson's r is affected by outliers, while Spearman's rho is more appropriate when one or both scores are not normally distributed and/or ordinal, or when outliers are known or expected.

Several studies have revealed the prevalence of mental health conditions in patients with orthopedic conditions as well as the impact of psychologic distress on patient-reported outcomes following surgery.3, 4, 5 The goal of this study was to evaluate differences in how generic and subspecialty-specific PROMs assess pre-operative patient-reported mental health.

2. Methods

All patients scheduled for orthopedic surgery at a single high volume orthopedic clinic from September 2018 through June 2019 were invited to complete a panel of PROMs through the MyChart (EPIC Systems®, Verona, WI, version 2018) patient portal. Activation of a MyChart account to view the PROMs and their completion were entirely voluntary and had no impact on the provision of care. Scheduled patients had 30 days to complete the assigned PROMs, and all data was uploaded and securely stored in an electronic medical records (EMR) database for analyses. The protocol for retrospective evaluation of PROM scores was reviewed by the internal clinical research committee of the hospital and determined to be exempt from IRB review.

2.1. PROMIS

All patients were asked to complete the PROMIS Physical Function short form v1.2 (PF-SF) or the PROMIS Physical Function Computer Adaptive Test v2.0 (PF-CAT) version if scheduled after an upgrade was completed to the EMR, and the PROMIS Global Health, which returned a Physical (GH-P) and Mental (GH-M) score. Additionally, the PF-SF was replaced by the PROMIS Upper Extremity CAT v2.0 (UE-CAT) for upper extremity patients following the EMR upgrade.

PROMIS questionnaires are the result of a collaboration between the National Institutes of Health and academic researchers to develop a psychometrically sound subjective questionnaire that can be used across a wide range of conditions, diseases, and population.⁶ Through item response theory, large item banks were distilled into short forms of 4 to 12 items as well as CAT formats that can be completed in as few as 4 items. CAT scoring algorithms are designed to capture a higher correlation with the full item bank with fewer items answered.

Response options are presented in a Likert scale. The PROMIS T-score metric is standardized to a normative US population, where a T-score of 50 represents the average score, with a standard deviation of 10. Higher scores indicate a greater amount of the underlying concept being measured. PROMIS questionnaires and scoring manuals are publicly available on the Health Measures website. All PROMIS scores were converted to T-scores.

2.2. Subspecialty-specific measures

Along with PROMIS questionnaires, patients completed one PROM specific to the subspecialty or surgery, which was one of the following: the Knee Disability and Osteoarthritis Outcomes Score for Joint Replacement (KOOS-JR),⁷ the Hip Disability and Osteoarthritis Outcomes Score for Joint Replacement (HOOS-JR),⁸ the Disabilities of the Arm, Shoulder and Hand brief version (QuickDASH),⁹ the Oswestry Disability Index (ODI),¹⁰ or the Quality of Life (QoL) and Activities of Daily Living (ADL) sub-scores of the Foot and Ankle Outcome Score (FAOS).¹¹

2.3. FAOS

The FAOS is a 42-item PROM designed to evaluate the foot- and ankle-related symptoms, pain, and impacts on daily living, sports and recreation, and (health-related) quality of life. Scores are calculated per subscale rather than for the full questionnaire. The FAOS uses Likert response choices scored from 0 to 4. Higher scores are associated with fewer symptoms or impacts.¹¹

2.4. QuickDASH

The 30-item Disabilities of the Arm, Shoulder, and Hand (DASH) questionnaire assesses impacts on upper extremity function from the patient perspective.¹² The 11-item QuickDASH (QD) was developed by selecting the highest-rated and highest-correlated concepts from the DASH with the goal of reducing response burden while maintaining validity and reliability, and was shown to have high baseline and follow-up agreement (intraclass correlation, ICC = 0.96, 0.97) with the DASH.⁹ Lower scores are associated with increased shoulder and upper extremity function.

2.5. ODI

The ODI is an older but oft-used orthopedic PROM, developed almost 40 years ago to evaluate functional disability in persons with low back pain10. Concepts assessed include impacts on social life, personal care, employment, and other activities of daily living. Each item is a series of response options describing six levels of increasing disability, scored from 0 to 5. A summed score is transformed into a percentage that can be interpreted as one of five levels of disability; lower scores indicate greater functioning.

2.6. KOOS-JR and HOOS-JR

Both the KOOS-JR and HOOS-JR were developed from longer questionnaires using Rasch analysis. The KOOS-JR is a 7-item PROM, while the HOOS-JR has 6 items. Both are scored by coding response options from 0 (none) to 4 (extreme), and convert raw summed scores into an interval level, interpreted as 0 (total disability) to 100 (perfect joint health). The administration of these two PROMs is often recommended by clinical groups such as the American Joint Replacement Registry.¹³

2.7. Statistical approach

Demographic variables were analyzed with descriptive statistics. Pearson's r and Spearman's rho correlation coefficients between PROMs and PROM items were calculated, respectively, and coefficients were interpreted using the Hinkle et al. (2003) methodology (Table 1).¹⁴ The null hypothesis (H₀) was that there were no correlations between PROM scores or items, while the alternative hypothesis (H₁) was that there was a moderate to strong correlation between scores or items. Statistical significance for correlations was set to p < .01. Particular focus was placed on GH-M item correlations (rho) which was expected to correlate with the following dimensions of subspecialty specific PROMs: FAOS-QoL and FAOS-ADL sub-scores, ODI #9 (“social life”), QD #7 (“social activities”), and QD #8 (“daily activities”).

Table 1.

Correlation coefficient interpretation.

Correlation (+or -)	Interpretation
.90–1.00	Very high/very strong
.70–.89	High/strong
.50–.69	Moderate
.30–.49	Low/weak
.00–.29	Negligible/none

All subspecialty-specific PROMs were scored following the developer's instructions. Patients with missing scores on both PROMIS measures or on the subspecialty-specific measure were excluded from this analysis. PROM score histograms were analyzed for score distributions, floor and ceiling effects, and to assess normality.

3. Results

A total of 1,026 patients across seven orthopedic specialties completed at least two PROMs; 19 patients were excluded for missing scores. During the study period, there were 4,698 surveys sent to activated MyChart accounts, equating to a response rate of 22.2%. Participant mean age was 59.1 (SD: 14.16) years, and 543 (55.8%) were female (see Table 2: Demographic Characteristics).

Table 2.

Participant demographic characteristics.

Participant Demographics
N	1026
Demographic data available (%)	974 (94.9%)
Mean age in years (SD)	59.1 (14.16)
Sex
Female (%)	543 (55.8%)
Male	430 (44.2%)
Race
White (%)	822 (84.5%)
Black or African American	84 (8.6%)
Asian	3 (0.3%)
Declined to answer	50 (5.1%)
Ethnicity
Not Hispanic or Latino	883 (90.8%)
Hispanic or Latino	17 (1.7%)
Declined to answer	73 (7.5%)

The sample size, mean scores, standard deviations (SD), and interquartile ranges (IQR) for PROMIS questionnaires by subspecialty are reported in Table 3; matching data for the subspecialty-specific legacy measures is in Table 4. Nearly all PROM scores showed normal distributions, with the exception of the QuickDASH; all skewness (i.e., symmetry) values were within ±1.0, and all kurtosis (i.e., tailedness) values were within ±2.0. Score histograms showed normal distribution with minimal floor or ceiling effects. Across all PROMIS measures, spine patients reported the lowest mean preoperative scores.

Table 3.

PROMIS scores across subspecialties.

PROM Orthopedic subspecialty	N	Mean	SD	IQR
PF-SF T-Score/PF-CAT T-Score
Foot & Ankle	70	41.6	8.70	37.3–46.0
Spine	94	37.5	6.88	33.6–41.1
Hip Arthroscopy	43	41.3	6.04	37.0–45.1
Knee Arthroscopy	115	40.4	7.47	35.7–44.6
Hip Arthroplasty	166	38.2	6.03	34.3–41.9
Knee Arthroplasty	233	38.8	5.11	35.7–41.9
Upper Extremity	216	44.2	8.68	38.2–50.1
GH-M Score
Foot & Ankle	72	53.0	7.80	48.3–59.0
Spine	98	48.9	8.85	43.5–53.3
Hip Arthroscopy	45	49.7	7.01	43.5–56.0
Knee Arthroscopy	119	53.0	7.59	48.3–59.0
Hip Arthroplasty	171	51.3	8.55	45.8–56.0
Knee Arthroplasty	241	51.2	7.80	45.8–56.0
Upper Extremity	280	52.3	8.35	45.8–59.0
GH-P Score
Foot & Ankle	72	44.2	8.09	37.4–50.8
Spine	98	40.4	7.38	34.9–44.9
Hip Arthroscopy	45	42.4	7.39	37.4–47.7
Knee Arthroscopy	119	45.3	7.45	39.8–50.8
Hip Arthroplasty	171	42.5	7.29	37.4–47.7
Knee Arthroplasty	241	42.1	6.34	37.4–44.9
Upper Extremity	280	45.3	7.76	39.8–50.8

Table 4.

Subspecialty-specific PROM scores.

Subspecialty-Specific PROM Orthopedic specialty	N	Mean	SD	IQR
KOOS-JR
Knee Arthroscopy	119	55.1	15.63	44.9–63.8
Knee Arthroplasty	241	51.3	13.00	44.9–59.4
HOOS-JR
Hip Arthroscopy	45	58.8	13.76	49.9–67.5
Hip Arthroplasty	171	54.5	15.70	46.7–64.7
ODI
Spine	98	37.2	18.12	48.0–24.5*
FAOS QoL
Foot & Ankle	72	29.2	20.27	11.5–43.8
FAOS ADL
Foot & Ankle	72	72.4	22.56	57.0–91.5
QuickDASH
Upper Extremity	280	40.0	22.57	56.8–22.7*
PROMIS PF Upper Extremity CAT
Upper Extremity	48	35.7	9.29	28.8–41.0

*Scores on these PROMs are reverse-calculated (i.e., lower score = better).

Correlation coefficients (r) of the GH-M, GH-P, and PF-SF/CAT with subspecialty-specific, legacy PROMS are shown in Table 5. The p-values for most correlation coefficients were <0.0001. The GH-M had a lower range of significant correlations (r = −0.2894 to 0.4578; p < .0001) with subspecialty-specific PROMs compared to the GH-P and PF-SF/CAT (r = 0.5485 to 0.8435; p < .0001). The GH-M correlated more strongly with the GH-P and PF-SF/CAT.

Table 5.

Correlations between PROMIS and subspecialty-specific PROMs.

Orthopedic Subspecialty PROM	PROM Score Correlations (Pearson's r)*
	GH-M	GH-P	PF-SF/CAT
Foot & Ankle
PF-SF/CAT	0.4257**	0.7027	–
GH-P	0.5763	–	–
FAOS-QoL	0.3576***	0.5591	0.7414
FAOS-ADL	0.3327***	0.6581	0.7462
Spine
PF-SF/CAT	0.3207***	0.6532	–
GH-P	0.6296	–	–
ODI	−0.3944	−0.6728	−0.7986
Hip Arthroplasty
PF-SF/CAT	0.5163	0.7075	–
GH-P	0.6798	–	–
HOOS-JR	0.4578	0.7564	0.7205
Hip Arthroscopy
PF-SF/CAT	0.3581+	0.8435	–
GH-P	0.3190+	–	–
HOOS-JR	0.1597+	0.7546	0.7221
Knee Arthroplasty
PF-SF/CAT	0.4382	0.6939	–
GH-P	0.5346	–	–
KOOS-JR	0.2894	0.5937	0.5485
Knee Arthroscopy
PF-SF/CAT	0.3008***	0.7172	–
GH-P	0.5846	–	–
KOOS-JR	0.1463+	0.5754	0.5851
Upper Extremity
PF-SF/CAT	0.3767	0.7052	–
UE-CAT	0.4503***	0.5528	–
GH-P	0.6133	–	–
QD	−0.3173	−0.6468	−0.6671

Values significant at p < .0001 unless otherwise noted.

**Values significant at p < .001.

***Values significant at p < .01.

⁺Values not significant at p ≤ .01.

Item-to-item correlation coefficients (rho) between the four items on the GH-M and three related subspecialty-specific PROM items were negligible to low, ranging from −0.1852 to −0.3702 (all p < .01) (Table 6). The ODI's “Social Life” item had the comparatively highest levels of correlation (rho) with GH-M items, ranging from −0.2574 to −0.3702 (all p < .01).

Table 6.

Item-to-item correlations.

Subspecialty-Specific PROM Item	Item-item correlations*
GH-M Item	Spearman's rho
ODI “Social Life” (#9)
GH02	−0.3702**
GH04	−0.2574***
GH05	−0.2702***
GH10r	−0.3177***
QuickDASH “Social Activities (#7)
GH02	−0.2737
GH04	−0.1854***
GH05	−0.2393
GH10r	−0.2394
QuickDASH “ADL Limits” (#8)
GH02	−0.2211**
GH04	−0.1852***
GH05	−0.2206**
GH10r	−0.3053

Values significant at p < .0001 unless otherwise noted.

**Values significant at p < .001.

***Values significant at p < .01.

NOTE: GH02 = Quality of Life; GH04 = Mental health self-rating; GH05 = Social activities; GH10r = Anxiety, depression, irritability.

4. Discussion

We sought to analyze how several frequently used PROMs assess preoperative mental health, using correlation coefficients as a proxy for estimating conceptual similarities between questionnaires. Our results suggest that generic physical function PROMs and subspecialty-specific, legacy PROMs are not fully capturing patient-reported preoperative mental health.

The influence of preoperative mental health on positive post-surgical outcomes implies that valid assessment of this concept is vital to patient-centered data collection. A retrospective evaluation of preoperative PROMIS Anxiety, Pain Interference (PI), Physical Function (PF), and Depression module scores in foot and ankle surgical patients found those reporting higher pre-operative anxiety and distress had higher pre-operative pain and greater deficits in physical functioning, and had more post-operative residual pain and lower functioning as measured by the corresponding PROMIS modules (P < 0.001). A retrospective cohort study by Beleckas et al. (2017)¹⁵ reported lower pre-operative PROMIS Pain Interference and Depression computer adaptive testing (CAT) scores from patients undergoing carpal tunnel release surgery were associated with increased post-operative encounters related to their surgery. Beyond emotional or mental health, research by Baumhauer et al. (2017)¹⁶ found a pre-operative PROMIS Physical Function (PF) T-score had a high predictability (84.2%; Chi square = 117.8; p < 0.01) for determining whether or not a patient undergoing a foot & ankle procedure would have a minimal clinical important difference (MCID) in post-operative PF T-score.

Our data for patients undergoing spine surgery mirrored prior studies by those such as Beleckas et al. (2018),¹⁷ who found spine patients reported the greatest deficits in pre-op mental health; our results showed they also have the lowest reported physical functioning and global physical health. Mean preoperative scores on the ODI reported by our study can be interpreted as moderate disability, bordering on severe.¹⁸ Regarding spine patient mental health, cross-sectional study has strongly suggested that the PROMIS-Anxiety CAT should be administered pre-operatively, given that 20% of spine patients had above-threshold scores on that PROM and the Generalized Anxiety Disorder-7 questionnaire.¹⁷ This study further contended that because the PROMIS-Anxiety CAT correlated more strongly with the PROMIS-Physical Function, it is a valuable inclusion in a set of pre-operative PROMs. However, strong correlations suggest similar conceptual coverage, and our study found significant positive correlations between all PROMIS modules including the GH-M.

In the orthopedic literature, there is a paucity of PROM item-level evaluation. Item-to-item rank correlations observed in this study were noticeably low. The spine-specific ODI's “Social Life” item (#9) showed low or weak correlation with the GH-M “quality of life” item, and negligible to low correlation with the other three GH-M items. The large standard deviation (SD) and IQR typically seen in ODI scores might be culpable; in a previous study of the performance of the PF-SF and ODI in spine patients,¹⁹ mean change recorded by the ODI ranged from 17 to 19 points, and the calculated minimal clinical important difference (MCID) ranged from 7 to 51. In upper extremity patients, the overall negligible to weak correlation between the GH-M and the QD “Social Activities” and “Limits to ADL” items suggest that the QD evaluates disparate concepts.

Curiously, the hip-specific HOOS-JR had much higher correlation (r = 0.4578) with the GH-M in hip arthroplasty patients than hip arthroscopy patients (r = 0.1597). The HOOS-JR and knee focused KOOS-JR items exclusively evaluate physical function, which may in turn lead to arbitrary correlations.

There are several limitations to this study that should be noted. First, voluntary, self-reported, and subjective data is susceptible to patient misinterpretation, misunderstanding, and inattention. While the response rate (22.2%) is within the range of published voluntary electronic PROM data collection rates,²⁰ this study's PROM data may not be generalizable to the entire set of patients undergoing orthopedic surgery at our clinic during the study period. Second, the smaller sample sizes of Foot & Ankle and Hip Arthroscopy patients, and related PROM coefficient p values near or above .01 suggest that these groups may not be fully accurate representations of the greater patient population. Finally, a longitudinal evaluation including postoperative PROM scores from a majority of this study's population would bolster the statistical approach to evaluating conceptual coverage.

5. Conclusion

Generic physical function and legacy orthopedic PROMs appear to neglect some preoperative disease-related impacts on mental health. Further, subspecialty-specific PROM items evaluating impacts on social activities, quality of life, and limitations of activity of daily living also do not adequately capture mental health in patients before their orthopedic surgery. Including a valid assessment of mental health such as the GH-M is an important component of understanding preoperative function and potentially postoperative recovery for orthopedic patients.