Bias in patient satisfaction surveys: a threat to measuring healthcare quality

Abstract

Patient satisfaction surveys are an increasingly common element of efforts to evaluate the quality of healthcare. Many patient satisfaction surveys in low/middle-income countries frame statements positively and invite patients to agree or disagree, so that positive responses may reflect either true satisfaction or bias induced by the positive framing. In an experiment with more than 2200 patients in Nigeria, we distinguish between actual satisfaction and survey biases. Patients randomly assigned to receive negatively framed statements expressed significantly lower levels of satisfaction (87%) than patients receiving the standard positively framed statements (95%—p<0.001). Depending on the question, the effect is as high as a 19 percentage point drop (p<0.001). Thus, high reported patient satisfaction likely overstates the quality of health services. Providers and policymakers wishing to gauge the quality of care will need to avoid framing that induces bias and to complement patient satisfaction measures with more objective measures of quality.

Summary box.

• Patient satisfaction responses tend to be overwhelmingly positive.

• Patient satisfaction ratings are easy to manipulate with the framing of the questions.

• The most standard framing of patient satisfaction questions in low/middle-income country contexts—inviting patients to agree or disagree with positive statements—overstates the quality of health services.

• Providers and policymakers will need creative strategies to gauge actual patient satisfaction.

  • Providing a mix of positively and negatively framed statements would attenuate the overall bias, although bias would still be present in the responses to each individual statement.

  • Avoiding agree/disagree, yes/no response formats would also reduce acquiescence bias.

• For comprehensive measures of quality, patient satisfaction must be supplemented with more objective measures.

Introduction

As access to at least some level of health services increases in low/middle-income countries, the focus of policymakers shifts to quality: How can we ensure that patients receive high-quality care? But even while measuring the provision of care is challenging in systems with limited data, measuring the quality of care invites a host of new complications. How can we regularly, systematically measure the quality of medical attention and advice? The simplest, most direct approach seems to be to ask the patients themselves. To gauge the quality of care, many policymakers and researchers turn to the patient satisfaction survey.

In high-income countries, results from patient satisfaction surveys are used to identify gaps and to inform quality improvement plans in healthcare organisations and health systems,¹ as well as in research.^{2 3} Moreover, patient satisfaction is often used as a performance indicator that influences hospital reimbursements and, more and more frequently, physician compensation.^{4 5} In low/middle-income countries, these surveys are increasingly used. For example, in Africa alone, patient satisfaction instruments have been used in Kenya,⁶ South Africa,^{7 8} Nigeria⁹ and Tanzania,^{10 11} among others.

These surveys often provide patients with a statement and then ask them to agree or disagree with that statement, such as ‘This health facility is clean. Do you agree or disagree?’ If patients answer these questions favourably, does that actually reflect high levels of patient satisfaction, or rather does it reflect a bias? Patients in low-income environments with few options for health services may value any services, and indeed, other work indicates high reported patient satisfaction even in the face of relatively low-quality services.¹² Alternatively, patients may agree with the interviewer to be agreeable (‘acquiescence bias’), or because ‘I agree’ is the first option offered and so choosing it requires the least cognitive effort (‘satisficing’).¹³

This is a substantive issue: of 26 recent patient satisfaction surveys in low/middle-income countries, more than three-quarters phrased their statements positively. Specifically, of the 26 studies included in the World Bank’s Central Microdata Catalog that used patient satisfaction questions,¹⁴ 20 (77%) were phrased positively and only six (23%) were phrased negatively or neutrally. This potential framing bias adds to other previously identified challenges with patient satisfaction surveys, such as that patient satisfaction measured at clinics is rated much higher than patient satisfaction measured at home.¹⁵

How can we distinguish true patient satisfaction from bias induced by the survey?

We implemented an experiment in Nigeria to distinguish between actual satisfaction with health services and survey biases. The study was implemented in 80 primary healthcare centres in six Nigerian states: Anambra, Bauchi, Cross River, Ekiti, Kebbi and Niger. Patient exit interviews were administered to all patients who visited the primary healthcare centres at the time of data collection. Surveys were administered face-to-face by trained enumerators with tablet computers in eight monthly rounds between June 2014 and February 2015. Interviewers arrived unannounced as part of a larger randomised controlled trial that involved helping clinic staff to identify gaps in the quality of service delivery and to set goals to close those gaps.¹⁶ The patient exit interview did not mention the larger quality improvement intervention. In total, 2222 patients were interviewed, or roughly 28 patients per facility on average. In addition to patient satisfaction measures, data were collected on a set of demographic and socioeconomic characteristics of the patients (including age, gender, education, employment and income).

Each patient was presented with 11 statements on the quality of care and asked to agree or disagree with each statement. Patients were randomly assigned to receive one of three treatments: the standard, positively framed statements (table 1 Set A), a set of equivalent negatively framed statements (table 1 Set B), or a random mix of the two. As expected with randomisation and a large sample of patients, patients were statistically indistinguishable across groups on age, gender, education and employment.

Table 1.

Positive and negative framed patient satisfaction statements

Set A: Positively framed statement	Set B: Negatively framed statement
1. The lab fees today were reasonable.	The lab fees today were too expensive.
2. This health facility is clean.	This health facility is dirty.
3. The waiting time was appropriate.	The waiting time was too long.
4. The fees for medicines or drugs you received today were reasonable.	The fees for medicines or drugs received today were too expensive.
5. The staff at this facility is courteous and respectful.	The staff at this facility is rude and disrespectful.
6. The staff did a good a job of explaining your condition.	The staff did a poor job of explaining your condition.
7. You had enough privacy during your visit.	You had too little privacy during your visit.
8. The registration fees of this visit to the health facility were reasonable.	The registration fees of this visit to the health facility were too expensive.
9. The staff spent a sufficient amount of time with you.	The staff spent too little time with you.
10. The hours this facility is open are adequate to meet your needs.	The hours this facility is open are too short to meet your needs.
11. You completely trust the staff’s decision about medical treatment in this facility.	You do not completely trust the staff’s decision about medical treatment in this facility.

How patient satisfaction questions are framed makes a big difference

With positively framed statements, patients report extremely high levels of satisfaction. There is no item for which approval is lower than 88%; for more than half of the statements, agreement exceeds 94%. However, when patients are presented with negatively framed questions, satisfaction drops significantly on 10 out of 11 questions, with an average drop of 7.5 percentage points across all questions and including drops as large as 18.9 and 11.6 percentage points (figure 1 and table 2). (See online supplementary appendix for details on the specification and the robustness analysis.)

Figure 1.

Patient satisfaction is easily manipulated by framing of questions. Percentage of patients who respond ‘I agree’ to a statement about the quality of care that they received at a primary healthcare facility. Based on a sample of 2222 patients across six Nigerian states.

Table 2.

Impact of framing on patient satisfaction—the simple specification

	(1)	(2)	(3)	(4)	(5)	(6)	(7)	(8)	(9)	(10)	(11)	(12)
	Lab fees	Drug fees	Registration fees	Clean	Wait time	Respect	Explain	Privacy	Staff time	Open hours	Trust	Overall
Negative	−0.189 (0.002)	−0.116 (<0.001)	−0.020 (0.275)	−0.053 (<0.001)	−0.071 (<0.001)	−0.025 (<0.001)	−0.040 (<0.001)	−0.111 (<0.001)	−0.109 (<0.001)	−0.076 (<0.001)	−0.102 (<0.001)	−0.075 (<0.001)
Positive (control mean)	0.886	0.892	0.942	0.930	0.918	0.986	0.974	0.888	0.965	0.974	0.988	0.949
Observations (n)	178	1004	784	2219	2219	2213	2204	2209	2219	2144	2193	19 586
Missing values	2	7	37	3	3	9	18	13	3	78	29	202
Number of individuals who received the question	180	1011	821	2222	2222	2222	2222	2222	2222	2222	2222	19 788

Dependent variable=1 if the patient responded favourably in stage 1 (ie, ‘agree’ on positively framed questions or ‘disagree’ on negatively framed questions), 0 otherwise. ‘Negative’ refers to a negatively framed item. ‘Positive’ refers to a positively framed item. The numbers reported below the coefficients are p values. The total patients asked each question differs because certain questions only applied to a subset of patients.

When we separate the results for those who received positive and negative statements within a mixed battery of statements from those who received purely positive or purely negative statements, the pattern remains the same (table 3). For 10 out of the 11 statements (and on average), the impact of negative statements with all negative statements is negative and statistically significant. Patients who received a negative statement in the mixed battery of statements were also less likely to respond favourably. Here, 8 of the 11 statements show significant effects.

Table 3.

Impact of framing on patient satisfaction—the detailed specification

	(1)	(2)	(3)	(4)	(5)	(6)	(7)	(8)	(9)	(10)	(11)	(12)
	Lab fees	Drug fees	Registration fees	Clean	Wait time	Respect	Explain	Privacy	Staff time	Open hours	Trust	Overall
Negative with negative	−0.204 (0.002)	−0.105 (<0.001)	−0.028 (0.092)	−0.056 (<0.001)	−0.064 (<0.001)	−0.021 (0.004)	−0.029 (0.002)	−0.094 (<0.001)	−0.106 (<0.001)	−0.066 (<0.001)	−0.081 (<0.001)	−0.067 (<0.001)
Negative with mixed	−0.165 (0.113)	−0.148 (0.004)	−0.120 (0.022)	−0.036 (0.163)	−0.110 (<0.001)	−0.032 (0.055)	−0.084 (0.001)	−0.212 (<0.001)	−0.132 (<0.001)	−0.143 (<0.001)	−0.218 (<0.001)	−0.124 (<0.001)
Positive with mixed	−0.035 (0.731)	0.028 (0.433)	−0.227 (0.003)	0.001 (0.945)	−0.004 (0.846)	0.010 (0.161)	0.011 (0.303)	−0.006 (0.820)	−0.006 (0.697)	−0.018 (0.271)	0.001 (0.940)	−0.005 (0.454)
Observations (n)	178	1004	784	2219	2219	2213	2204	2209	2219	2144	2193	19 586
Missing values	2	7	37	3	3	9	18	13	3	78	29	202
Number of individuals who received the question	180	1011	821	2222	2222	2222	2222	2222	2222	2222	2222	19 788

Dependent variable=1 if the patient responded favourably in stage 1 (ie, ‘agree’ on positively framed questions or ‘disagree’ on negatively framed questions), 0 otherwise. ‘Negative with negative’ refers to a negatively framed item with a battery of negatively framed items. ‘Negative with mixed’ refers to a negatively framed item with a random mix of negative and positive items. The numbers reported below the coefficients are p values. The total patients asked each question differs because certain questions only applied to a subset of patients.

To probe the robustness of the results, we explore whether these results are consistent when we control for patient gender, age, income levels, or education levels. In all cases we obtain very similar results. We also introduce interaction terms for these variables, for which we do not see systematically significant results. That is, the pattern of acquiescence bias that we uncovered seems to affect patients irrespective of their income, education and experience with private facilities.

These results suggest that high-reported patient satisfaction likely overstates the quality of health service provision in resource-constrained environments, adding to evidence that patient satisfaction is imperfectly related to health outcomes.⁴ Inflated patient satisfaction reports can potentially distort decisions about effort and resource allocation.

Conclusion

There is broad consensus that improving patients’ experience as they obtain healthcare is an intrinsically desirable goal. Some elements of that improved experience are likely to be universal: patients value short waiting times and clean facilities, and they appreciate providers that respond to their needs and treat them with respect. Other elements may vary across contexts, such as the extent to which patients value being involved in the medical decision-making process. Routine measurement of patient experience and satisfaction is becoming commonplace in healthcare organisations in both high-income countries and low/middle-income countries.

Our results demonstrate that patient satisfaction measurements are deeply sensitive to the framing of the questions. Specifically, we find strong evidence of acquiescence bias, or the tendency of individuals to agree to the statement they are presented, irrespective of its content. As such, the standard (‘positive’) formulation results in consistently inflated measures of patient satisfaction. This highlights the need to supplement patient satisfaction with other measures to provide an overall indication of service quality. These may include the measurement of actual health outcomes, as well as the use of vignettes to gauge provider knowledge and standardised patients to gauge provider effort. Furthermore, there may be significant ceiling effects with positively framed questions, since the average tends to be so high that it is difficult to distinguish across performance levels.^{17 18}

The main implication of our study is that designers of patient satisfaction surveys should avoid using all positively phrased statements. Providing a mix of positively and negatively framed statements would attenuate the overall bias, although bias would still be present in the responses to each individual statement. Avoiding agree/disagree, yes/no response formats would also reduce acquiescence bias. Several major patient satisfaction surveys in use already incorporate these recommendations. For example, the 15-item Picker Patient Experience Questionnaire avoids agree/disagree statements altogether,¹⁹ and the Patient Experience Questionnaire has agree/disagree statements but includes both positive and negative framing.²⁰

Reduced bias would make patient satisfaction measures more meaningful, allowing better distinguishing across facilities, and would be beneficial for programmes wishing to use patient satisfaction to identify gaps and areas where changes are needed.