Racial/ethnic differences in reporting versus rating of healthcare experiences

Abstract

Asians are reported to have poorer healthcare experience than non-Hispanic Whites (NHWs), but the sources of the differences are not understood. One explanation is Asian's reluctance to choose extreme responses in survey. We thus sought to compare NHW-Asian differences in responses to healthcare experience surveys when asked to report versus rate their experiences. Patients of an outpatient care system in 2013 to 2014 in the United States were studied. Patient experience surveys were sent after randomly selected clinic visits. Responses from 6 major Asian subgroups and NHWs were included (N = 61,115). The surveys used a combined questionnaire of Clinician and Group Consumer Assessment of Healthcare Providers and Systems (CG-CAHPS) and Press Ganey surveys. CG-CAHPS questions are framed as “reporting” and Press Ganey questions as “rating” of experiences. We compared the proportion of favorable (or top box) responses to 2 related questions, one from CG-CAHPS and another from Press Ganey, and assessed racial/ethnic differences when using each of the 2 related questions, using a Pearson chi-squared test for independence. All Asian subgroups were less likely to select top box than NHWs for all questions. The Asian-NHW differences in "rating” questions were larger than the difference in related “reporting” questions. Of those who chose top box to CG-CAHPS questions (e.g., “Yes” on a question asking “Waited < 15 minutes”), their responses to related Press Ganey questions varied widely: 47% to 57% of Asian subgroups versus 67% of NHWs rated wait time as “Very good.” The extent of racial/ethnic differences in patient-reported experiences varies based on how questions are framed. The observed poorer experiences by Asians are in part explained by their worse rating of similar objectively measurable experiences.

1. Introduction

There is consistent evidence that Asians report worse healthcare experiences than non-Hispanic Whites (NHWs),^[1–12] but it is largely unknown what accounts for the racial/ethnic differences in reported experiences. Since a patient's report of their care continues to be an integral component of healthcare quality assessment and measurement of healthcare provider performance, a deeper understanding of the sources of the racial/ethnic differences is urgently needed.

The differences may in part reflect objectively measurable differences in quality of care, which may be explained by system-level barriers (e.g., language barriers, discrimination, access to services) that limit Asian patients’ ability to obtain the type of care they need.^[13,14] The differences may also reflect patient's subjective perception and expectation, given similar care they received. Evidence to date suggests that objectively measureable service quality may not be the main driver of the lower scores among Asians. Studies have shown that the Asian-NHW differences persist even when both demographic groups received care in the same setting.^[8–11,15] Despite lower ratings, Asians were less likely to change providers.^[16] Further, studies using a vignette to assess responses to the same hypothetical healthcare experiences found that Asians, but Hispanic and African Americans, rated the experiences as worse than NHWs did.^[17,18]

One explanation for this difference may be that Asians are culturally less likely to choose top box or best choice in responding to patient experience surveys.^[16,18–20] Existing literature on survey methodology has demonstrated that Asians are less likely to choose the top box, exhibiting lower extreme response tendency.^[21–24] That is, even when perceived experience is the same, Asians may respond differently than other groups, shifting the distribution downward away from extreme points of the rating scale.

The extent to which the tendency to prefer or avoid extreme response options (“response style” hereafter) manifests may depend on how the questions are worded and how response choices are framed. Evidence from survey methodology suggests that culturally differential response style is pronounced when the questions are framed as subjective ratings, often using Likert-type response scale and that the tendency is reduced by asking questions about objectively measurable experiences.^[25–27] It is thus expected that the distinctive response style of Asians in healthcare experience surveys can be mitigated by asking to “report” measurable experiences and providing response options that are objectively anchored. To our knowledge, this question has not been addressed using empirical data from healthcare experience surveys.

In this study, we examined differences between Asians and NHWs in their healthcare experiences and whether the differences are reduced when the experience is measured in terms of reporting observable experiences versus subjective ratings of the experience. In addressing the question, we take advantage of a unique dataset from routinely collected patient experience surveys, in which the questionnaire to each individual included 2 sets of questions asking about a similar aspect of care but one was framed as “reporting” and another as “rating.”

2. Methods

2.1. Study Setting and Data Sources

We used patient experience of care survey data from a large multispecialty, multisite outpatient care organization in northern California serving more than a million unique patients each year. Patients have a mix of insurance plans: 67% preferred provider organizations / fee-for-services, 15% health maintenance organization, 13% Medicare, and 5% other (Medicaid, other insurances, or self-pay) and most patients continue with their providers regardless of changes in insurance type. Patient demographic characteristics reflect the local population,^[28] representing significant diversity in racial/ethnic backgrounds (17% Asian, 11% Hispanic/Latino, 3% African American, 57% NHW, and 12% other). Patient race/ethnicity was self-reported.^[29,30] Individual survey responses were linked anonymously using electronic health record (EHR) data for the encounter to which survey refers. Other patient characteristics were also linked to EHR data, deidentified, and shared with the researchers under an IRB-approved protocol with a health insurance portability and accountability act waiver.

2.2. Patient Experience of Care Surveys

For more than a decade, the healthcare organization has been routinely conducting patient experience of care surveys for quality improvement purposes, and in the past few years Press Ganey, a nation-wide patient survey organization (www.pressganey.com), has been conducting the survey. Surveys were mailed to patients after randomly selected clinic visits, with no more than 1 survey sent to a patient within 90 days. The organization sought 30 responses per physician per 6-month period or roughly 5 per month. The overall response rate was about 18%, which is comparable to the response rate of mail-only patient surveys reported elsewhere.^[31] Response rate varied by patient demographics including age, sex, and race/ethnicity, as is reported in other settings.^[32–37] Potential response bias does not affect our study because our focus was on interpreting differences in completed surveys and comparing responses within each individual, rather than identifying the magnitude of disparities.

We used the survey data from 2013 to 2014 when a combined questionnaire of the Clinician and Group Consumer Assessment of Healthcare Providers and Systems (CG-CAHPS) Visit survey (version 2.0) and Press Ganey Medical Practice survey was sent to each patient. The CG-CAHPS survey was developed by the Agency for Healthcare Research and Quality (AHRQ), and is used as the standard survey tool for assessing and comparing healthcare providers from patients’ experiences.

The CG-CAHPS survey asks about care experiences with a specific provider in the past 12 months and during the most recent office visit. The survey asks about experiences with various dimensions of care, including getting timely appointments, care, and information; how well providers communicate with patients; the degree to which office staff are helpful, courteous, and respectful; and overall rating of the provider. The Press Ganey survey asks patients to rate their experiences with clinicians and staff during the most recent visit, which was specified in the survey, to a specific provider.

There is substantial overlap in the dimensions of care asked by the 2 surveys in 1 questionnaire; however, the way the survey questions are asked differs between the 2 surveys. Answers of most CG-CAHPS questions are worded as “reports,” using various scales including “Yes/No” and “Yes, definitely/Yes, somewhat/No,” while all the Press Ganey questions are framed as “ratings,” asking the degree to which patients are satisfied with the care with a 5-point scale: Very poor, Poor, Fair, Good, and Very good. We examined responses on 2 questions asking about a similar aspect of care but framed differently, offering different response options (see Table 1).

Table 1.

Related questions from CG-CAHPS and Press Ganey surveys.

2.3. Study sample

Among the surveys by NHW and Asian respondents (N = 61,115), we included English language surveys only (N = 60,038) and further limited it to respondents whose primary language was English (N = 57,985) to eliminate language and translation effects that can confound the racial/ethnic differences in survey responses. Asians in the United States are diverse with varying immigration histories and cultural and behavioral norms,^[38] which may influence how they respond to survey questions. We therefore disaggregated 6 subgroups of Asians (Chinese, Asian Indian, Filipino, Japanese, Korean, and Vietnamese) to examine variation among Asian subgroups.

2.4. Analytical approaches

First, we compared the distribution of answers to a pair of questions that are related but framed differently (see Table 1) across racial/ethnic subgroups. Second, for each of the paired questions, among those who gave a favorable response to a CG-CAHPS survey question, the distribution of answers to a related Press Ganey question was compared across racial/ethnic groups. For each survey question, a “favorable” response is defined as the top box rating/reporting, following the conventional scoring approach for CG-CAHPS survey recommended by AHRQ.^[39]

The unit of analysis was each survey response. Differences in the scores of each Asian subgroup and NHW were assessed with a Pearson chi-squared test for independence. Clustering across multiple observations within respondent was adjusted using Huber–White standard errors. STATA 13 was used for data management, statistical analysis, and graphical presentations of the results.

3. Results

A majority of respondents included in the study were NHW (83.1%) followed by Chinese (6.6%), Asian Indian (4.7%), Filipino (2.5%), Japanese (2.0%), Korean (0.5%), and Vietnamese (0.5%) patients (Table 2). All Asian subgroups were less likely to give favorable responses than NHWs to all the Press Ganey survey questions (across various domains) that are framed as ratings. In most Press Ganey questions, the NHW-Asian difference was largest for Asian Indians and smallest for Japanese. The difference was largest for the question asking the experience of wait time (ranging from 7% to 18% points) and smallest for the question asking provider's communication (ranging from 4% to 16% points).

Table 2.

Survey responses by respondent race/ethnicity.

The NHW-Asian differences for questions from CG-CAHPS were much smaller but statistically significant (P < .001) for Chinese (1% to 7% points lower across 5 questions) and Asian Indian (3% to 5% points lower across 5 questions) as compared with NHW. The differences were smaller for Filipino, Japanese, Korean, and Vietnamese and turned out statistically insignificant for most questions (see Table 2).

Of note is that the ceiling effect was pronounced with CG-CAHPS. That is, the likelihood of providing favorable responses (or % top box) was higher with CG-CAHPS than with related Press Ganey questions. For NHW, the CG-CAHPS score ranged from 88% (Wait time) to 97% (Provider communication) as compared with related Press Ganey score ranging from 59% (Wait time) to 85% (Recommend practice). The difference between CG-CAHPS and Press Ganey scores for related questions was larger for all Asian subgroups than for NHWs.

Of those who gave a favorable response to a CG-CAHPS question, Asians of all subgroups (Japanese and Filipino less so) were less likely to rate their experience as “Very good” in a related Press Ganey question. For example, NHW-Chinese difference ranged from 18.5% points (Wait time) to 10.3% points (Provider communication), despite their favorable response to the related CG-CAHPS question (Fig. 1).

Figure 1.

Distribution of responses to a Press Ganey question among those who gave “favorable” response to a related CG-CAHPS question. The NHW-Asian (of each subgroup) difference in the proportion of “Very good” responses was statistically significant at P < .001.

4. Discussion

By comparing individual responses to 2 different survey questions asking a related aspect of care, we assessed the extent to which the way the questions and answers are framed can mitigate the apparent Asian effects (or their tendency not to choose top box) in patient-reported experiences of care surveys. Consistent with prior studies,^[8–11,15] we found that Asians who received care in the same healthcare setting rated or reported their experience much poorer than other groups. In this study, we further identified that Asians exhibited a greater tendency of avoiding a top-box response to questions framed as “rating” versus “reporting” and that the magnitude of response style can be reduced considerably by using questions (reporting vs rating) and answer scales that target more objectively defined experiences.

Not all Asians are alike, however. While most Asians show similar patterns, some groups such as Japanese and Filipinos, who have longer history of US immigration than other Asian subgroups,^[40] responded to the CG-CAHPS questions more similarly to NHWs than other Asian subgroups. To our knowledge, this is the first study investigating subgroups of Asians in a disaggregated way to estimate Asian subgroup differences in patient-reported experiences and further to explore the contribution of question framing and response style to observed disparities in patient experiences.

By design, potential confounders in the relationship between race/ethnicity and patient-reported experience were controlled in our study. The patients of diverse background are from a large healthcare system who were cared for in a similar way, thereby minimizing the possibility of system barriers (e.g., access to care, quality of services received) explaining the observed racial/ethnic differences. Further, as responses to 2 related questions compared were from one questionnaire, that is, from the same person referring to the same encounter, there is no difference in care received between the 2 responses. Thus, any difference we detected is entirely attributable to the differences in how questions and responses are framed.

While the response scales used in CG-CAHPS scale is preferable in terms of mitigating Asian-NHW differences attributable to response style, it comes with potential limitations. The ability to capture the variation in patient's subjective rating or satisfaction of care received may be limited when using CG-CAHPS “reporting” questions than those using “rating” scales. Further, CG-CAHPS scale is more susceptible to ceiling effects. A majority chose the top box in CG-CAHPS reporting questions, and of those who chose top box, responses to a related Press Ganey rating question varied widely.

Many of those who chose top box in the CG-CAPHS reporting may still have experienced poorer than desired level of care. In a separate qualitative study, we conducted interviews with Asian Indians and Chinese patients in the study setting, and asked the reasons for not providing top-box responses in a Press Ganey question while giving top-box responses to a parallel CG-CAHPS question.^[41] They pointed out unmet expectations and offered numerous areas for improvement (e.g., discomfort in discussing with doctors about culturally based health practices, desire for no wait time at reception area).

Put together, no single question format or answer scale would be the best in all occasions; the selection of reporting versus rating questions should be based on the purpose of survey and application of the survey responses. When comparing providers who serve a different mix of populations (e.g., in terms of proportion of Asian patients in their panel), one might use the CG-CAHPS scale that is less susceptible to race/ethnicity-based response style. On the other hand, to assess patient experience of certain subgroups, rating scales (such as those used in Press Ganey survey) that can better discriminate subjective evaluation of similar, objectively measurable experiences would be preferred. While it would be desirable to reduce differences in scores attributable to measurement issues, rating of services or patient satisfaction is subjective in nature and is an important aspect to capture in patient experience surveys. For the “rating” of services, semantic-differential format or visual analog scale, which has shown to produce more equivalent distribution across cultural and linguistic groups and is less prone to a ceiling effect than Likert scale,^[42,43] may be preferred to minimize bias due to response style.

Response style is just one explanation of the observed lower patient-reported experiences among Asians. Asians are an extremely heterogeneous group with diverse backgrounds. Many Asians in the United States are recent immigrants and may have different perspectives, perceptions and expectations about healthcare services as compared to US born Americans.^[41] Asians who are less acculturated may perceive similar care as poorer than NHW if the service is less culturally tailored. Lack of a sense of belongingness to a healthcare system and lack of connection to providers who are racial/ethnic or culturally discordant may have contributed to their poorer experiences and thus survey responses. Consistently, we found that responses from Asians who have lived in the US longer who may be less acculturated (e.g., Filipino as a group; we did not have a measure of the length of stay in the United States for each individual) were closer to NHW's while Asians who have immigrated to the United States more recently (e.g., Korean as a group) gave far lower scores. Nevertheless, we did not have direct measures of perceived experiences to compare whether there was higher level of unmet needs among Asians, justifying their lower satisfaction ratings. Identifying sources of the differences and their relative importance is beyond the scope of the study but is important in order to effectively address the racial/ethnic differences in patient-reported experiences.

Patient experience and satisfaction of care, influenced by personal experiences, cultural norms, and values, may vary across other racial and ethnic groups not addressed in this article, for example, other Asian subgroups, Mid-eastern, Eastern European. Individuals with other race/ethnicity, language, or cultural backgrounds may be distinctively different in their use of rating scales as well as their perceive experience and expectations of health care. Future work is needed to understand and determine the social and cultural influences on different response tendencies of patient experience in other less represented minority groups. With full understanding of sources of differences across diverse groups, the data collected with patient experience surveys can provide more accurate and actionable information on the current status disparities in patient experiences, and ultimately, can help reduce the disparities.

In conclusion, caution should be exercised when choosing patient experience measures and interpreting results from racial/ethnically diverse populations. Questions asking about objectively measurable experience, which are less susceptible to response style (e.g., reluctance or tendency to choose extreme responses), would reduce unwarranted differences based on a respondent's cultural background, and thus may be more adequate for assessing and comparing providers who serve diverse populations. Subjective ratings of similar objectively measurable experiences are much poorer among Asians than NHWs, which may explain the observed racial/ethnic differences in patient-reported experiences seen in subjective ratings.

Acknowledgments

The authors thank Meghan Halley, a former collaborator at the Palo Alto Medical Foundation Research Institute, for her feedback on an earlier draft and presentation material. The authors also appreciate feedback from participants of Palo Alto Foundation Medical Group monthly meetings.

Author contributions

Conceptualization: Sukyung Chung, Dominick L. Frosch.

Data curation: Sukyung Chung.

Formal analysis: Sukyung Chung.

Investigation: Sukyung Chung, Lily Liang.

Methodology: Sukyung Chung, Lily Liang, Dominick L. Frosch.

Project administration: Sukyung Chung.

Resources: Sukyung Chung.

Software: Sukyung Chung.

Supervision: Sukyung Chung.

Visualization: Sukyung Chung.

Writing – original draft: Sukyung Chung, Gabriella Mujal.

Writing – review & editing: Gabriella Mujal, Lily Liang, Latha Palaniappan, Dominick L Frosch.