Use of Patient-Reported Outcomes and Satisfaction for Quality Assessments

Abstract

Objectives

Recent focus on patient-reported outcomes (PROs) has created a new challenge as we learn how to integrate them into practice along with other quality metrics. We investigated the relationship between PROs and satisfaction among spine surgery patients. We hypothesized that there would be significant disparities between patient satisfaction and PROs at both the hospital and patient level.

Study Design

Retrospective cohort study of adults undergoing elective lumbar spine surgery at 12 hospitals participating in the Spine Surgical Care and Outcomes Assessment Program (Spine SCOAP).

Methods

Satisfaction, pain, and function scores collected at 30–90 days postoperatively were compared across hospitals to determine the relationship between PROs and satisfaction at the hospital-level. These scores were also collected one year postoperatively along with clinical information to determine the relationship between PROs and satisfaction at the patient-level.

Results

At the hospital-level (n=1,397 patients), satisfaction was high (88%, range 79–94%) but less than half of patients were satisfied and had a positive outcome in pain or function. At the patient-level (n=520 patients), the majority of patients (81%) were satisfied. Pain and function improvement were associated with satisfaction but even among patients who did not improve, 59% were satisfied.

Conclusion

Satisfaction was high across all hospitals, but PROs were much more variable. This disparity indicates that the “quality” profile of a hospital is highly dependent on the domain of measurement. Understanding the relationship between PROs, satisfaction, and quality is the first step in drawing meaningful conclusions that can then be translated into policy.

Introduction

Traditionally, clinical outcomes such as the rates of adverse events or mortality served as the metrics by which we measured treatment success or failure. While these objective measures of quality allow for comparisons across hospitals and physicians, the sole use of clinical outcomes may provide an incomplete appraisal of healthcare quality. Within the last few years, payers, policy makers, and healthcare quality organizations have focused on patient-reported outcomes (PROs) as an important marker of quality distinct from an assessment of patient satisfaction and clinical outcomes.(1–4) PROs are defined as measures of health that are directly reported by the patient, rather than interpreted by a clinician or reported in the medical record.(5) As clinicians and hospitals are charged with adding PRO measures to existing clinical and satisfaction metrics, the challenge moving forward is to integrate PROs in a manner that both improves patient care and provides a meaningful assessment of quality.

Spine surgery represents one clinically relevant scenario that could benefit from a robust integration of both PRO and clinical outcome measures. Functional impairment and pain are often the primary drivers for patients considering spine surgery, therefore, understanding which patients are most likely to see improvement in these symptoms can inform the decision between surgical and non-surgical options. Utilizing PROs for an individual patient rather than as a general population measure in this type of clinical scenario remains a newer effort, but one increasingly called for in today’s healthcare environment.(6, 7) In Washington State, healthcare providers are experiencing demands to implement and interpret PROs in spine surgery due to recent policy recommendations from the Bree Collaborative, proposed in 2015, that call for the use of these measures for all patients undergoing single-level lumbar spine fusion procedures. This lumbar fusion policy is intended to allow appropriate selection of surgical candidates among the vast population of adults with low back pain and to limit expensive surgical interventions to those patients most likely to experience benefit from the procedure.(8) Other organizations such as the Centers for Medicare and Medicaid Services have chosen to integrate patient satisfaction metrics into their reimbursement policies.(9) Given the differences between PROs, which typically focus on functional ability and pain, and patient satisfaction measures, it is not clear that one or the other necessarily provides a true measure of outcome, nor that these measures are robust enough to allow for policy and reimbursement decisions.

One mechanism for PRO collection is through the Comparative Effectiveness Research Translation Network (CERTAIN), a research and analytic platform aligned with the Spine Surgical Care and Outcomes Assessment Program (Spine SCOAP).(6, 10, 11) Spine SCOAP is a collaborative of hospitals within Washington State whose mission is to aggregate data from hospitals, healthcare providers, and patients to generate evidence-based best practice measures that can be dispersed among the collaborative to improve the care of patients in Washington State and beyond.

In this study, we investigated the relationship between PRO measures and patient satisfaction as collected by Spine SCOAP. We analyzed this relationship at a hospital-level to determine the utility of evaluating hospital quality based solely on satisfaction scores versus other PROs. We also analyzed this relationship at an individual patient-level to determine the correlation between satisfaction and PRO scores. We hypothesized that there would be significant disparities between patient satisfaction and PRO measures for both individual patients and entire hospitals, suggesting that the use of satisfaction scores as a primary measure of quality is flawed.

Study Data and Methods

We conducted a retrospective cohort study of adults undergoing elective lumbar spine surgery from 2012–2014 at 12 Spine SCOAP hospitals who also participated in CERTAIN PRO data collection. Eligible participants completed a survey that included measures of pain, function, and satisfaction both before surgery and at pre-determined time points postoperatively (30– 90days and one year). Pain was measured using a ten-point numeric rating scale with “0” representing “no pain” and “10” representing “worst pain possible.” Function was measured using the Oswestry Disability Index, a composite measure that grades function on a scale of 0–100, with higher scores indicating worse disability.(12) Improvement in pain or function was established by a minimally clinically important difference between baseline and follow-up (for pain, a change of at least two points from baseline; for function, a change of at least 15 points from baseline).(13) We measured satisfaction using a single question: “How satisfied are you with the overall result of your spine operation so far?” The satisfaction question was scored using a four-point modified Likert scale.

Two separate analyses were conducted. The first was focused on understanding how PROs and satisfaction are related for the purposes of hospital quality assessments, and the second was focused on determining the relationship between satisfaction and PROs at the individual patient level. The University of Washington Human Subjects Division approved the retrospective review of this data. All analyses were carried out using Stata version 11 (STATA Corp, College Station, TX).

Hospital Level Analysis

Patient-reported satisfaction, pain, and function scores collected at 30–90 days postoperatively were compared to baseline measures for each hospital. This time point was selected as it corresponds to the timeframe in which hospital satisfaction surveys are distributed to patients. For each hospital, we determined the proportion of patients with a positive outcome in each domain, as well as a composite measure reporting the proportion of patients who had a positive outcome in all three domains.(Table 1) A positive outcome was defined as one that met the minimally clinically important difference for that domain. We then compared individual hospital performance to the mean score across all hospitals and described the number of hospitals that performed either above or below average on each domain. To further investigate the association between these measures, we used a linear regression model to determine the degree of correlation between patient-reported satisfaction, pain, and function.

Table 1.

Description of outcome classification based on responses to numeric rating scale, Oswestry disability index, and single question satisfaction assessment.

	Positive Outcome	Negative Outcome
Pain	Increase of ≥2 points on numeric rating scale pain score compared to baseline score.	A change of < 2 points in numeric rating scale pain score, or a decrease in numeric rating scale pain score, compared to baseline score.
Function	Increase of ≥15 points on Oswestry disability index score compared to baseline score.	A change of <15 points on Oswestry disability index score, or decrease in Oswestry disability index score, compared to baseline score.
Satisfaction	Report of “very satisfied” or “somewhat satisfied” in response to single question of satisfaction with care.	Report of “very unsatisfied” or “somewhat unsatisfied” in response to single question of satisfaction with care.
Composite	Had a positive outcome for pain, function, and satisfaction as described above.	Had a negative outcome in pain, function, and/or satisfaction as described above.

Patient Level Analysis

In the second analysis, we analyzed patient-reported satisfaction, pain, and function at one year postoperatively to determine the association between patient characteristics and a positive outcome in any or all domains. Patient characteristics were obtained through Spine SCOAP and included information about sociodemographics, underlying diagnosis (e.g. spinal stenosis, herniated disc), and operative characteristics (e.g. fusion). An invasiveness index based on the type of intervention at each vertebral level as well as the number of operated levels was included as this has been shown to be correlated with clinical outcomes in spine surgery patients.(14, 15) The American Society of Anesthesiologist Score was included as an additional measure of physical status at the time of the operation. This is a commonly used score applied to every patient undergoing surgery and is intended to provide an additional description of patient risk.(16) We selected the one year postoperative time point because PROs and satisfaction in the early postoperative period are not necessarily reflective of findings at one year,(17, 18) which are felt to be a more reliable indicator of long-term satisfaction.

To determine the relationship between the three domains, we used a Poisson regression model stratified by pain and function improvement that was predictive of satisfaction (somewhat or very satisfied) at twelve months following spine surgery. A subset of patients (n=69, 13%) did not have clinical information available through Spine SCOAP and were excluded from the regression analysis. To account for the fact that other patient characteristics are known to be correlated with overall satisfaction,(19) we included sociodemographic factors, diagnostic factors, and operative characteristics as measured by SCOAP in the regression model. The resulting model was clustered by hospital to account for correlated data. In cases such as this where multiple observations are drawn from the same hospital, there is a concern that these results from within each individual hospital are correlated due to extrinsic factors that are not measured, and that this effect varies between hospitals. Failure to account for correlation can lead result in a biased estimate and an incorrect inference. A complete description of the regression model is included in the Appendix.

Results

Hospital Level Analysis

At the 30–90 day follow-up point, 1,397 patients treated at 12 hospitals had complete survey data. Overall satisfaction was high across hospitals with 88% of patients reporting that they were somewhat or very satisfied with their spine surgery result (range 79%–94% by hospital). More than two thirds of patients (69%) met the minimally clinically important difference for a positive outcome for pain alone (range 63%–82%) while 48% of patients had a positive outcome for function alone (range 21%–61%). Less than half of patients (40%, range 21%–52%) had a positive outcome in all three domains of pain, function, and satisfaction. At the hospital level, percentage of patients with pain improvement, function improvement, and satisfaction were poorly correlated (R²=0.38). Two hospitals were above average on all domains, while two hospitals were below average on all domains. (Figure 1)

Figure 1.

Proportion of patients who were satisfied, had improved pain, had improved funciton, or were satisfied and had improved function and pain at 30–90 days post-operatively, stratified by hospital.

Patient Level Analysis

At the one year follow-up point, 520 patients had complete baseline and follow-up survey data, and 451 (87%) had clinical information available from spine SCOAP. More than half of patients reported that they were very satisfied (n=301, 58%) while one quarter of patients reported that they were somewhat satisfied (n=124, 24%) with their spine surgery result. Table 2 demonstrates the interaction between patient-reported pain and function improvement and patient satisfaction. Patients with a higher magnitude of pain improvement more often reported that they were somewhat or very satisfied: 62% patients with no pain improvement or worse pain were satisfied, while 96% of those with five or more points of pain improvement were satisfied. We found a similar trend in between satisfaction and function, with 61% of those with no functional improvement or worse function reporting satisfaction, and 80% of those who function improved more than 30 points reporting satisfaction. Interestingly, even among patients who had no change in their symptoms or reported that they were worse, 59% of patients were still somewhat or very satisfied.

Table 2.

Percent of patient satisfaction by patient-reported pain and function improvement at twelve months following spine surgery. Color key from gray to white indicates satisfaction from low to high, respectively. Numeric rating scale (NRS) scored zero to ten, with higher scores indicating worse pain. Oswestry disability score (ODI), range from zero to one hundred. Higher scores indicate worse disability.

	Pain Improvement (NRS score reduction from baseline)
Functional Improvement (ODI score reduction from baseline)	None/Worse	Minimal	Minor	Major	Total
	(−5 to 0)	(1)	(2 to 4)	(≥5)
None/worse (≤0)	59% (n=58)	48% (n=23)	72% (n=18)	80% (n=15)	61% (n=114)
Minimal (1 to 14)	51% (n=35)	83% (n=23)	74% (n=35)	95% (n=20)	73% (n=113)
Minor (15 to 29)	88% (n=17)	69% (n=13)	92% (n=79)	98% (n=50)	92% (n=159)
Major (≥30)	80% (n=5)	83% (n=12)	93% (n=43)	100% (n=47)	94% (n=107)
Total	62% (n=115)	69% (n=71)	87% (n=175)	96% (n=132)	81% (n=493)

In our stratified analysis adjusting for patient characteristics, patients who had no improvement in back pain were 1.7 times (95%CI: 1.27–2.16) and 2.3 times (95%CI: 1.71–3.11) more likely to be satisfied if they experienced a minor improvement in function (ODI score reduced between 15 to 29 points) and major improvement in function (ODI score reduction of 30 points or more), respectively, compared to patients who had no improvement or worse function outcomes. Of the patients with improved back pain at one year, a mild improvement in function is associated with 1.3 times (95%CI: 1.00–1.71) the likelihood of being satisfied, and a major improvement in function is associated with 1.4 times (95%CI: 1.08–1.81) the likelihood of being satisfied, compared to patients who had no improvement or worse function outcomes. (Table 3)

Table 3.

Poisson regression models clustered by hospital to estimate changes in satisfaction between functional improvement levels at 12 months following surgery. IRR is Incidence relative risk ratio. CI is confidence interval. Poisson models adjusted for age, sex, smoking status, insurance, diagnosis, baseline function score, ASA status, and surgery invasiveness

	No Improvement in back pain n=159			Improvement in back pain n=265
Functional Improvement	% Satisfied	Unadjusted IRR (95%CI)	Adjusted IRR (95%CI)	% Satisfied	Unadjusted IRR (95%CI)	Adjusted IRR (95%CI)
None/worse ≤0)	55.55%	Ref	Ref	76.47%	Ref	Ref
Minimal (1 to 14)	63.79%	1.29** (1.17–1.42)	1.14 (0.95–1.36)	82.14%	1.15 (0.89–1.48)	1.16 (0.88 – 1.52)
Minor (15 to 29)	80.00%	1.71** (1.26–2.33)	1.66** (1.27–2.16)	94.74%	1.33* (1.02–1.74)	1.31 (1.00–1.71)
Major (≥30)	82.35%	1.98** (1.44–2.74)	2.30** (1.71–3.11)	95.96%	1.44* (1.14–1.84)	1.40* (1.08 – 1.81)

^**=<0.001;

^*<0.05

Discussion

In this analysis, we sought to determine how PRO measures compare to other quality measures such as patient-reported satisfaction. We analyzed data from a statewide clinical registry encompassing a collaborative of hospitals in Washington State that allowed for an assessment of both hospital-level and patient-level outcomes. In the hospital level analysis, satisfaction was high across all hospitals, but PROs for pain and function were much more variable. This disparity indicates that the “quality” profile of a hospital is highly dependent on the domain of measurement, and that a quality assessment based on one domain alone would be incomplete or misleading. This result was confirmed by the lack of correlation between patient-reported satisfaction and positive outcomes in pain or function. In the patient level analysis, the majority of patients reported satisfaction with their spine surgery result and we found a strong association between patient-reported improvements in pain and function and satisfaction. Despite this understandable finding, more than half of the patients with no improvement or worse outcomes in pain or function were satisfied with their surgery. This incongruence suggests that satisfaction ratings may be based on non-clinical aspects of care not captured by this survey.

Quality has traditionally been defined as a successful clinical outcome and evaluations have primarily focused on complications and hospital-based data points. An assessment of the patient’s perception of care is now recognized as a key component in the assessment of healthcare quality. This newer metric is becoming financially relevant as a result of the Merit-Based Incentive Payment System, passed as part of the Medicare Access and CHIP Reauthorization Act of 2015.(20) PROs have been proposed by policy-makers and payers as a required component of quality assessments and reimbursement schedules.(21, 22) For example, in 2015 the Centers for Medicare and Medicaid Services announced that patient experience surveys will be included as a component of the Comprehensive Care for Joint Replacement Payment Model, which provides bundled payments for certain lower extremity joint replacement operations.(23)

Many PRO scores assess a patient’s perception of improvement after an intervention, but others aim to quantify a patient’s satisfaction with their care. However, it is not clear that these patient satisfaction measures correlate with the delivery of safe and effective care as defined by clinicians and caregivers. Satisfaction surveys are often criticized for focusing on the “wrong” outcomes, such as the hospital noise level or the variety of food choices offered to patients, rather than on the clinical result of a treatment or intervention.(24) Patient satisfaction also may not reflect the total patient experience, as one patient might report high levels of satisfaction with their overall care despite a poor clinical outcome(25) while another patient might report dissatisfaction with their care while simultaneously benefiting from a positive clinical outcome.

Measurement of PROs has been confined primarily to the research setting but there is growing focus on incorporating such measures into clinical care, shared decision-making, and quality of care reporting. Emerging guidance from federal agencies and professional organizations,(1, 26, 27) provides a framework for implementation of PROs into practice, but there remain unresolved issues that prevent broad acceptance by the medical community.(28) Frequently cited concerns from practicing clinicians range from the logistics of implementing PROs, the acceptability of the measures by clinicians, and the methodology of data collection and interpretation.(28, 29) In order to justify the implementation and adoption of PROs in a clinical setting, their utility in healthcare quality assessments at both the hospital and individual patient level needs to be demonstrated. Another concern is how the metrics will translate into improved patient decision-making: when faced with a variety of quality metrics, which should a patient use when determining where to undergo surgery or seek care for their disease?

Previous studies have explored the relationship between patient satisfaction and other outcome measures (such as clinical outcomes and patient-reported functional outcomes) but conclusions have varied among studies. In one prospective study, Godil et al, reported that patient satisfaction measures at 90-days post-operatively are not correlated with clinical outcomes such as complications and re-admissions and as such should not be used as a proxy for overall quality.(30) These findings were confirmed in a subsequent review article.(22) The time frame of measurement is important, as well: PROs three months post-operatively are not necessarily predictive of PROs at one year.(17, 18) This latter finding may be clinically relevant as satisfaction metrics are frequently measured at time points close to the date of care delivery, rather than at later time points which might better reflect long-term improvements or outcomes. These studies highlight the unresolved issues related to the use of PROs and satisfaction data in quality assessments.

More generally, there is evidence that satisfied patients are more likely to utilize healthcare resources, thereby, incur costs, but it is not clear that the increased healthcare utilization is related to better outcomes.(31) Among surgical patients, one study found no association between satisfaction and measures of quality as defined by the Surgical Care Improvement Program, such as adequate antibiotic prophylaxis before surgery,(32) again pointing to the concern that patients base satisfaction ratings on non-clinical factors that are not reflected in clinical or administrative data. Our analysis found correlation between patient-reported improvement in pain and function and satisfaction in the patient-level analysis, but there was a relatively large proportion of individuals who reported that they were somewhat or very satisfied who also had a negative outcome in either pain or function. The variation in hospital performance suggests this as well, since some hospitals had above average performance on satisfaction, but were below average on pain and function. Again, those hospitals may be offering something to patients that subjectively improves their experience without actually improving the quality of care. Alternatively, an early time point of assessment may be inappropriate given the dynamic nature of recovery following surgery, since functional outcomes at 30–90 days postoperatively may not be reflective of functional outcomes at one year or beyond.

The survey utilized in this study was created under the auspices of quality improvement to help surgeons and hospitals understand more about their patients. The evidence is still emerging as to the complete set of questions that are relevant to support patient- and hospital-level decision-making, as well as to provide quality assessments. One limitation of this analysis is the potential inability to capture all domains that are important to patients, and thus may be biased to those areas deemed clinically relevant to the survey creators. A second limitation is the restriction of the analysis to lumbar surgery patients alone, which means that the results may not be reflective of all clinical conditions. We encourage future research endeavors that focus on how measures of the patient experience correlate in other health conditions. Because this analysis included only those with complete data, these results may not be representative of outcomes from non-responders. Because not all patients had complete follow-up data at both the early and late time points, the population included in the hospital level analysis is not necessarily the same population as those included in the patient-level analysis. We do not have information regarding non-responders, and it may be that patients who respond to surveys are systematically different from those who do not. Finally, the survey supported through CERTAIN is currently only available in English which may further bias our results.

Conclusion

In Washington State, current policy and payment recommendations require health systems to capture PRO data for patients undergoing lumbar spine surgery. It is imperative that we understand how to interpret this PRO information to ensure that patients get the right care at the right time. With the ongoing concerns about PRO implementation, collection, interpretation, and application,(28, 29)it is necessary to focus on how this important data can inform decisions about patient care. Successful implementation depends not only on the available infrastructure for collection and analysis, but also on the commitment of healthcare providers that PROs add value to the ongoing quality discussion. Without this focus we risk more measurement and more data without the ability to transform patient care. There is considerable work to be done before we will fully understand the appropriate role of PROs in healthcare.

The implications of this work are far-reaching, especially with the recent passage of the Merit-Based Incentive Payment System which emphasizes quality and value over volume.(20) How can we improve healthcare delivery if we do not yet understand the relationship between results from quality assessments and the overall patient experience? Understanding the relationship between PROs, satisfaction, and general quality is the first step in drawing meaningful conclusions that can then be translated to policy. Furthermore, this work requires the inclusion of patients, providers, payers, and policy makers to ensure that we are selecting the appropriate domains on which to judge quality.

TAKE-AWAY POINTS.

The findings from this study indicate that the “quality” profile of a hospital is highly dependent on the domain of measurement.

• Recent focus on patient-reported outcomes (PROs) has created a new challenge as we learn how to integrate them into practice along with other quality metrics.

• Understanding the relationship between PROs, satisfaction, and quality is the first step in drawing meaningful conclusions that can then be translated into policy.