Abstract
Background:
Patient satisfaction has garnered interest as a tool to measure healthcare quality. However, orthopaedic studies in total joint arthroplasty, spine, and hand patients have offered conflicting relationships between Press Ganey® (PG) satisfaction metrics and patient-reported outcome (PRO) measures. No prior study has assessed the relationship between PG and PROs in foot and ankle patients. Here, we tested if there was an association between PROMIS outcomes and PG satisfaction scores.
Methods:
PG and PROMIS outcomes data for new patient visits to an orthopaedic foot and ankle clinic between 2015 and 2017 were retrospectively analyzed. Patients that completed PG satisfaction surveys were included for study. All patients who completed PG surveys and completed one or both PROMIS physical function (PF) or pain interference (PI) metrics administered by computerized adaptive testing were included. Negative binomial regressions were used to compare PRO scores to PG overall satisfaction and PG satisfaction with care provider, adjusting for patient characteristics. Results were reported as dissatisfaction score ratios, which represented the amount of PG dissatisfaction associated with a 10-point increase in PROMIS PF or PI.
Results:
Of the 3,984 new patient visits, only 441 completed the PG survey (11.3% response rate). Ceiling effects were seen with PG data: 64% of patients reported perfect satisfaction with care provider and 27% had perfect overall satisfaction. Higher function on the PROMIS PF was weakly associated with increased overall satisfaction (Ratio=0.82, 95% CI: 0.68–0.99, P=0.039) and increased satisfaction with care provider (Ratio=0.60, 95% CI: 0.40–0.92, P=0.019). However, pain (PROMIS PI) was not associated with overall satisfaction or with satisfaction with care provider.
Conclusions:
Based on our data here, patient satisfaction was weakly related to patient-reported function but not pain interference among this subset of new patients presenting to a foot and ankle clinic. Given our essentially negative findings, further study is needed to determine which aspects of the PG satisfaction and PROMIS scores track similarly. Further, our findings add to the growing literature showcasing limitations of the PG tool including low response rates and notable ceiling effects. If satisfaction metrics and patient-reported outcomes capture differing aspects of the patient experience, we need to better understand how that influences the measurement of healthcare quality and value.
Level of Evidence:
III
Keywords: Press Ganey® satisfaction, PROMIS, foot and ankle
Introduction:
Patient satisfaction metrics – like the Press Ganey® (PG) survey – have gained increasing popularity in the assessment of quality of healthcare.10, 11 The PG tool was designed to assess outpatient patient satisfaction by probing multiple domains of the patient experience including clinic wait-times and overall assessment of the provider practice.18 As PG has generated interest for its use in clinical practice, the Patient-Reported Outcomes Measurement Information System (PROMIS) has also attracted attention for its ability to efficiently capture outcomes data on various clinical domains including patient-reported function and pain interference.12 Whether satisfaction and outcomes instruments, though, measure similar or differing aspects of the patient-experience is unclear.
Several total joint replacement studies have failed to identify meaningful links between PG satisfaction and patient-reported outcomes (PROs) like function and pain.4,5,13 There was no association in spine patients between self-reported functional disability and patient satisfaction.2 In upper extremity patients, PG satisfaction was associated with psychological health as well as with function and pain.22 No work to date, though, has explored the relationship between PG satisfaction and PROs in foot and ankle patients. The purpose of this study was to test whether patient-reported levels of pain and function (as captured by PROMIS) were associated with PG overall satisfaction and satisfaction with care provider scores.
Materials and Methods
Following Institutional Review Board approval, new patient clinic visits to one of four orthopedic foot and ankle surgeons at a single academic center from January 2015 to December 2017 were retrospectively studied. Patients ≥18 years-old were included for study. Demographic data were gathered from electronic chart review and included age, sex, race/ethnicity, insurance status, and provider. For insurance status, patients were categorized as having commercial (i.e. private) or non-commercial (i.e. Government-funded) insurance.
During new patient visits, patients completed PROMIS Physical Function (PF) and Pain Interference (PI) data via electronic tablets. PROMIS scores were immediately tabulated and entered into the medical record. Briefly, to better understand PROMIS score calculation, a value of 50 indicates a population average, and 10-point changes signify a standard deviation change. For PROMIS PF, higher scores represent more patient-reported function, and for PROMIS PI, larger scores indicate greater pain.
All new patients were then contacted to electronically complete the Press Ganey® survey. The PG survey includes 24 items arranged into 6 scales (Access, Moving Through the Visit, Nurse/Assistant, Care Provider, Personal Issues, and Overall Assessment). Each item was scored from 0 – 100, with a score of 100 signifying perfect satisfaction. Each scale score was calculated by taking the mean for all questions within that scale. The total score was generated by taking the mean from all the six scales, weighted equally. Like prior work, we used the answers to the questions surveying patients’ “overall satisfaction” and “satisfaction with care provider” for our analyses.(22) Using unique identifiers via the electronic medical record, we were able to confirm that PG data and PROMIS scores were linked to the same new patient visit.
Patient characteristics including age, sex, race/ethnicity, insurance status, and baseline PROMIS PF and PI scores were summarized descriptively for all patients, then stratified based on perfect versus less than perfect satisfaction. Perfect scores were 100 for either overall satisfaction or satisfaction with care provider. Less than perfect scores were anything below 100 for either satisfaction metric. Continuous variables (i.e. age, PROMIS PF, and PROMIS PI) were summarized as mean (standard deviation: SD), median (interquartile range: IQR), and range whereas categorical variables were summarized as frequency and percentage. The dichotomous stratification of perfect versus non-perfect scoring was done for descriptive purposes only and not used in formal analyses.
Associations between PROMIS PF and PI with both overall satisfaction and satisfaction with care provider (“PG scores”) were assessed using multivariable regression models. Because the distributions of the PG score outcome variables were left-skewed, we first transformed them to PG’, a measure of dissatisfaction, using the formula: PG’ = 100 – PG, where 100 represented the maximum satisfaction survey score. Each PG’ outcome (i.e. PG’ overall dissatisfaction and PG’ dissatisfaction with care provider) was then modeled using negative binomial regression with a log link function, so that the exponentiated regression coefficients represented the ratio in the amount of dissatisfaction associated with a 10-point increase in PROMIS PF or PI. For interpretation of these analyses, note that a 10% decrease in dissatisfaction cannot be interpreted as a 10% increase in satisfaction but, rather, 10% closer to perfect satisfaction.
All models were adjusted for age, sex, provider, and insurance type.8, 15, 21 Race/ethnicity were not included as covariates in the models due to the homogenous study population. Regression results were reported as score ratios with 95% Confidence Intervals (CIs) and P-values. Statistical significance was assessed at P < 0.05 level, and all tests were two-tailed. Analyses were conducted using R version 3.5 (R Foundation for Statistical Computing, Vienna, Austria).
Results:
During our study timeframe, 3,894 new patients were seen in our orthopedic foot and ankle clinics. Of those, 441 completed the PG survey (with an 11.3% response rate). All the patients who completed the PG survey had either one or both of the PROMIS PF and PI scores. Missing outcome and demographic data included PROMIS PF (N=4), PROMIS PI (N=1), race/ethnicity (N=2) and insurance status (N=3). Descriptive statistics were reported in Table 1 for all patients and then sub-classified based on perfect versus less than perfect satisfaction scores. Again, Table 1 was utilized to showcase descriptive differences between the groups (i.e. perfect versus non-perfect responders) but not used in formal analysis.
Table 1:
Summary of baseline patient characteristics.
| Variable | All Patients (N=441) | Overall Satisfaction | Satisfaction with Care Provider | ||
|---|---|---|---|---|---|
| Non-Perfect Scores (N=322) | Non-Perfect Scores (N=157) | ||||
| Age (years) | |||||
| Mean (SD) | 57.4 (13.9) | 58.4 (13.8) | 57.0 (13.9) | 58.1 (13.8) | 56 (13.8) |
| Median (IQR) | 59 (49–68) | 61 (48.5–69) | 58.5 (49– 67) | 60.5 (50–68) | 58 (47–66) |
| Range | 19.0–88.0 | 20.0–84.0 | 19.0–88.0 | 19.0–86.0 | 22.0–88.0 |
| Sex (% Male) | 156 (35.4%) | 44 (37.0%) | 112 (34.8%) | 100 (35.2%) | 56 (35.7%) |
| Ethnicity (% Non-Hispanic/Latino) | 431 (98%) | 118 (99.2%) | 313 (97.5%) | 280 (98.6%) | 151 (96.8%) |
| Race (% White/Caucasian) | 421 (95.7%) | 116 (97.5%) | 305 (95%) | 272 (95.8%) | 149 (95.5%) |
| Insurance Status (% Commercial) | 290 (66.2%) | 74 (63.2%) | 216 (67.3%) | 183 (65.1%) | 107 (68.2%) |
| PROMIS Physical Function | |||||
| Mean (SD) | 42.3 (8.7) | 42.9 (10.1) | 42.1 (8.1) | 43.0 (9.3) | 41.0 (7.3) |
| Median (IQR) | 41.5 (37.1–48.2) | 41.6 (36.5–50.1) | 41.5 (37.5–47.6) | 42.5 (37.5–49.8) | 40.5 (37.1–46.2) |
| Range | 15.4–73.3 | 17.7–73.3 | 15.4–66.0 | 15.4–73.3 | 17.7–62.4 |
| PROMIS Pain Interference | |||||
| Mean (SD) | 58.9 (7.5) | 59.7 (8) | 58.5 (7.3) | 58.2 (7.9) | 60.0 (6.7) |
| Median (IQR) | 58.5 (53.3–64.2) | 60.5 (52.7–65.5) | 57.7 (53.7–63.0) | 57.7 (52.6–64.2) | 59.1 (54.6–64.2) |
| Range | 38.7–80.1 | 38.7–80.1 | 38.7–78.9 | 38.7–80.1 | 38.7–78.9 |
There were significant correlations between PG overall satisfaction and PG satisfaction with care provider (r=0.80, 95% CI: 0.77, 0.83; P<0.001) as well as between PROMIS PF and PI (r=-0.69, 95% CI: −0.74, −0.64; P<0.001). The magnitude of the correlation between PROMIS PF and PI was comparable to previously published evidence.17 Comparing the dichotomized satisfaction scores with patient characteristics revealed an association between age and overall satisfaction (P=0.038) but no relationship between age and satisfaction with care provider (P=0.13). There were no significant relationships with sex, provider, and insurance type for either PG satisfaction score (all P>0.05). However, we adjusted for these variables in multivariable analyses comparing satisfaction scores with PROMIS scores because we had planned to do so a priori, and because they could still act as potential confounders despite not achieving statistical significance in these descriptive comparisons.
A 10-point increase in PROMIS PF was weakly associated with both overall satisfaction (Ratio=0.82, 95% CI: 0.68–0.99, P=0.039) (Figure 1a; Table 2) and satisfaction with care provider (Ratio=0.60, 95% CI: 0.40–0.92, P=0.019) (Figure 1b; Table 2), in multivariable regression models adjusting for age, sex, provider, and insurance. A 10-point increase in PROMIS PI was not associated with overall satisfaction (Ratio=1.15, 95% CI: 0.94–1.40, P=0.18) or with satisfaction with care provider (Ratio=1.46, 95% CI: 0.96–2.19, P=0.08) in multivariable analysis (Table 2).
Figure 1:
Predicted relationships between PROMIS PF and Press Ganey overall satisfaction (1a) and satisfaction with care provider (1b). Solid lines and shaded areas represent mean and interquartile ranges, respectively.
Table 2:
Association between patient-reported outcomes and patient satisfaction scores
| Satisfaction Outcome | PROMIS Predictor | Score Ratio (95% CI)* | P-Value* |
|---|---|---|---|
| l00-Overall Satisfaction | PROMIS Physical Function | 0.82 (0.68–0.99) | 0.039 |
| PROMIS Pain Interference | 1.15 (0.94–1.40) | 0.18 | |
| 100-Satisfaction with Care Provider | PROMIS Physical Function | 0.60 (0.40–0.92) | 0.019 |
| PROMIS Pain Interference | 1.46 (0.96–2.19) | 0.08 |
Results are presented from multivariate regression analyses comparing either overall satisfaction or satisfaction with care provider to either PROMIS outcome score, adjusting for age, sex, provider, and insurance status. Separate analyses were run for PROMIS physical function and PROMIS pain interference. Data are displayed as score ratios based on 10-point increases in each respective PROMIS score. To clarify for PROMIS physical function, lower scores mean greater functional impairment whereas with PROMIS pain interference, higher scores equate to more pain.
Discussion:
This study probed for relationships between patient satisfaction and PROMIS metrics in foot and ankle patients. Among new patient visits to one of four orthopedic foot and ankle surgeons, lower PROMIS PF scores were weakly associated with lower overall satisfaction as well as lower satisfaction with care provider. Despite the high correlation between PROMIS PF and PROMIS PI, higher PROMIS PI scores were not associated with either overall satisfaction or with satisfaction with care provider. This suggests that PG and PROMIS measure differing aspects of the patient-experience.
Low response rates are a known shortcoming of the Press Ganey instrument. In this study, only 11.3% of patients presenting to an orthopaedic foot and ankle surgeon completed PG data. While low, the response rate in our study parallels the published literature. Rane et al. recently encountered an 8.9% response rate in orthopedic upper extremity patients.19 Further, according to Lee et al., the response rate for the PG survey at our own institution in fiscal years 2013 and 2014 was 12.6%, which was a noticeable drop from 19.1% in 2011.13 Our response rate of 11.3% appears to coincide, then, with the declining rates seen at our own institution over the past decade. PG response rates are not consistently reported in orthopedic PG studies, but when they are, the rates have been low ranging from 12.6%-28%.2,4,5,7,9,13–15,21 PG data, whether here or elsewhere, appear to be limited by non-response bias, thereby raising concerns about the instruments’ ability to accurately measure patient satisfaction.19,21
In this study, we encountered notable ceiling effects with 64% of our patients reporting perfect satisfaction with care provider and 27% exhibiting perfect overall satisfaction. Similarly, in an analysis of hospital-wide visits, Presson et al. found ceiling effects of 29.3% for PG overall satisfaction and for specific PG items, ceiling effects ranged from 55.4–84.1%.18 In the same study, Presson et al. further showed that with a ceiling effect of 29.3%, a provider’s average PG score could drop from the 100th percentile to the 71st percentile with only a half point change in PG raw score (from 100 to 99.5).18 Despite the fact that ceiling effects can negatively impact an instruments’ reliability and validity, the PG tools are increasingly being used to evaluate physician performance. It is important for orthopaedic foot and ankle surgeons to better understand the PG tools’ limitations and how those shortcomings may impact how their healthcare delivery is being valued.
No prior studies have assessed for relationships between PG satisfaction and outcomes in orthopedic foot and ankle patients. However, prior work in other orthopaedic disciplines has asked similar questions. Kohring et al. failed to identify meaningful correlations between PG satisfaction and patient-reported outcomes in total knee arthroplasty patients.13 Similarly, Chughtai et al. did not find relevant associations between traditional outcome measures and PG overall hospital satisfaction after either total hip4 or knee5 replacements. Abtahi showed no relationship between functional disability in spine patients to patient satisfaction2 but did reveal that psychological distress was associated with PG satisfaction.1 Given that we found essentially no relationship between PG satisfaction and PROMIS pain and function, we were unable to establish a conclusive relationship between satisfaction and PROs in foot and ankle patients. Our conflicting data, then, largely coincide with the published literature.
In this study, we only identified weak relationships between patient-reported function with patient satisfaction – but not between pain and satisfaction. However, multiple outcome metrics exist beyond those included in this study including tools that assess psychological and mental health like the PROMIS anxiety instrument.22 In a hand and upper extremity patient population, Tyser et al. identified stronger relationships between PG satisfaction and PROMIS anxiety than with PROMIS PF and PI.22 Multiple lines of evidence have shown that psychological well-being impacts foot and ankle outcomes.3,16,20 Further study, then, in orthopaedic foot and ankle patients using different instruments that query psychological health might elucidate stronger links between PROs and PG satisfaction.
Like all studies, our report has limitations. We studied a homogenous population that was 96% White/Caucasian and 98% Non-Hispanic/Latino. Minority populations have a different patient experience than majority groups6,9 but due to the low number of non-Caucasian patients, we were unable to adequately study the relationship between satisfaction and outcomes within individual racial or ethnic groups. Further, perhaps patient satisfaction and outcomes are more closely linked in specific conditions (i.e. hallux valgus).17 Our study pooled all diagnoses, though, and was underpowered to perform disease-specific analyses.
Conclusion:
In conclusion, patient satisfaction and patient-reported physical function – but not patient-reported pain – were at best weakly related for new patients presenting to a foot and ankle clinic. No prior studies have explored PG satisfaction and PROs in the foot and ankle population, so our conflicting findings warrant further investigation. It remains unclear which aspects of the patient-experience PG satisfaction and PROMIS scores similarly measure and which components they uniquely capture. It appears from our data that PG and PROMIS measure differing components of patient care. Further, PG data remains handcuffed by its low response rate and potential for ceiling effects. If satisfaction metrics and patient-reported outcomes capture differing aspects of the patient experience, we need to better understand then how that may influence the measurement of healthcare quality and value. Therefore, until we achieve a broader understanding of the relationship between patient satisfaction and patient health status, we would caution against the use of PG data in the assessment of quality of care, outcomes, or reimbursement for orthopaedic foot and ankle surgery.
Acknowledgements:
We thank Drs. Timothy Beals, Alexej Barg, and Charles Saltzman for their editorial assistance.
Funding:
This investigation was supported by the University of Utah Population Health Research (PHR) Foundation, with funding in part from the National Center for Research Resources and the National Center for Advancing Translational Sciences, National Institutes of Health, through Grant UL1TR002538 (formerly 5UL1TR001067-05, 8UL1TR000105 and UL1RR025764).
References:
- 1.Abtahi AM, Brodke DS, Lawrence BD, Zhang C, Spiker WR. Association between patient-reported measures of psychological distress and patient satisfaction scores after spine surgery. J Bone Joint Surg Am. 2015;97(10):824–8. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 2.Abtahi AM, Lyman KS, Brodke DS, Lawrence BD, Zhang C, Spiker WR. Patient Satisfaction is Not Associated With Self-reported Disability in a Spine Patient Population. Clin Spine Surg. 2017;30(8):E1165–E8. [DOI] [PubMed] [Google Scholar]
- 3.Awale A, Dufour AB, Katz P, Menz HB, Hannan MT. Link Between Foot Pain Severity and Prevalence of Depressive Symptoms. Arthritis Care Res (Hoboken). 2016;68(6):871–6. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 4.Chughtai M, Gwam CU, Khlopas A, Sodhi N, Delanois RE, Spindler KP, et al. No Correlation Between Press Ganey Survey Responses and Outcomes in Post-Total Hip Arthroplasty Patients. J Arthroplasty. 2018;33(3):783–5. [DOI] [PubMed] [Google Scholar]
- 5.Chughtai M, Patel NK, Gwam CU, Khlopas A, Bonutti PM, Delanois RE, et al. Do Press Ganey Scores Correlate With Total Knee Arthroplasty-Specific Outcome Questionnaires in Postsurgical Patients? J Arthroplasty. 2017;32(9S):S109–S12. [DOI] [PubMed] [Google Scholar]
- 6.Chung S, Mujal G, Liang L, Palaniappan LP, Frosch DL. Racial/ethnic differences in reporting versus rating of healthcare experiences. Medicine (Baltimore). 2018;97(50):e13604. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 7.Compton J, Glass N, Fowler T. Evidence of Selection Bias and Non-Response Bias in Patient Satisfaction Surveys. Iowa Orthop J. 2019;39(1):195–201. [PMC free article] [PubMed] [Google Scholar]
- 8.Delanois RE, Gwam CU, Mistry JB, Chughtai M, Khlopas A, Yakubek G, et al. Does gender influence how patients rate their patient experience after total hip arthroplasty? Hip Int. 2018;28(1):40–3. [DOI] [PubMed] [Google Scholar]
- 9.Garcia LC, Chung S, Liao L, Altamirano J, Fassiotto M, Maldonado B, et al. Comparison of Outpatient Satisfaction Survey Scores for Asian Physicians and Non-Hispanic White Physicians. JAMA Netw Open. 2019;2(2):e190027. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 10.Graham B. Defining and Measuring Patient Satisfaction. J Hand Surg Am. 2016;41(9):929–31. [DOI] [PubMed] [Google Scholar]
- 11.Graham B, Green A, James M, Katz J, Swiontkowski M. Measuring patient satisfaction in orthopaedic surgery. J Bone Joint Surg Am. 2015;97(1):80–4. [DOI] [PubMed] [Google Scholar]
- 12.Hunt KJ, Alexander I, Baumhauer J, Brodsky J, Chiodo C, Daniels T, et al. The Orthopaedic Foot and Ankle Outcomes Research (OFAR) network: feasibility of a multicenter network for patient outcomes assessment in foot and ankle. Foot Ankle Int. 2014;35(9):847–54. [DOI] [PubMed] [Google Scholar]
- 13.Kohring JM, Pelt CE, Anderson MB, Peters CL, Gililland JM. Press Ganey Outpatient Medical Practice Survey Scores Do Not Correlate With Patient-Reported Outcomes After Primary Joint Arthroplasty. J Arthroplasty. 2018;33(8):2417–22. [DOI] [PubMed] [Google Scholar]
- 14.Lee VS, Miller T, Daniels C, Paine M, Gresh B, Betz AL. Creating the Exceptional Patient Experience in One Academic Health System. Acad Med. 2016;91(3):338–44. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 15.Martin L, Presson AP, Zhang C, Ray D, Finlayson S, Glasgow R. Association between surgical patient satisfaction and nonmodifiable factors. J Surg Res. 2017;214:247–53. [DOI] [PubMed] [Google Scholar]
- 16.Nakagawa R, Yamaguchi S, Kimura S, Sadamasu A, Yamamoto Y, Sato Y, et al. Association of Anxiety and Depression With Pain and Quality of Life in Patients With Chronic Foot and Ankle Diseases. Foot Ankle Int. 2017;38(11):1192–8. [DOI] [PubMed] [Google Scholar]
- 17.Nixon DC, McCormick JJ, Johnson JE, Klein SE. PROMIS Pain Interference and Physical Function Scores Correlate With the Foot and Ankle Ability Measure (FAAM) in Patients With Hallux Valgus. Clin Orthop Relat Res. 2017;475(11):2775–80. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 18.Presson AP, Zhang C, Abtahi AM, Kean J, Hung M, Tyser AR. Psychometric properties of the Press Ganey(R) Outpatient Medical Practice Survey. Health Qual Life Outcomes. 2017;15(1):32. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 19.Rane AA, Tyser AR, Presson AP, Zhang C, Kazmers NH. Patient Satisfaction in the Hand Surgery Clinic: An Analysis of Factors That Impact the Press Ganey Survey. J Hand Surg Am. 2019;44(7):539–47 e1. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 20.Shivarathre DG, Howard N, Krishna S, Cowan C, Platt SR. Psychological factors and personality traits associated with patients in chronic foot and ankle pain. Foot Ankle Int. 2014;35(11):1103–7. [DOI] [PubMed] [Google Scholar]
- 21.Tyser AR, Abtahi AM, McFadden M, Presson AP. Evidence of non-response bias in the Press-Ganey patient satisfaction survey. BMC Health Serv Res. 2016;16(a):350. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 22.Tyser AR, Gaffney CJ, Zhang C, Presson AP. The Association of Patient Satisfaction with Pain, Anxiety, and Self-Reported Physical Function. J Bone Joint Surg Am. 2018;100(21):1811–8. [DOI] [PMC free article] [PubMed] [Google Scholar]