Survey Response Style and Differential Use of CAHPS Rating Scales by Hispanics

Abstract

Background

Prior CAHPS studies have shown that Hispanics report care that is similar to or less positive than for non-Hispanic Whites, yet have more positive ratings of care.

Objective

To examine differential use of the 0–10 rating scales in the CAHPSR health plan survey by Hispanic ethnicity and insurance status (Medicaid vs. commercial managed care).

Data

CAHPS 2.0H adult Medicaid and commercial data submitted to the National Committee for Quality Assurance (NCQA).

Measures

The dependent variables are the CAHPS 2.0 ratings of care: personal doctor or nurse, specialists, and health care received. Ratings were categorized into four levels: 0–4, 5–8, 9, and 10. The independent variable is a four-level categorical variable: Hispanic Medicaid, Hispanic commercial, (non-Hispanic) White Medicaid, and (non-Hispanic) White commercial. Six potential confounders were controlled: gender, age, education, self-rated health, survey mode, and survey language.

Analysis

Multinomial logistic regression was used to test for differences in extreme response styles.

Results

Hispanics exhibited a greater tendency towards extreme responding in the CAHPS ratings than non-Hispanic Whites—in particular, they were more likely than Whites in commercial plans to endorse a “10,” and often, scores of 4 or less, relative to an omitted category of “5”–“8.”

Conclusions

The observed higher Hispanic ratings may be partially attributed to differences in response style rather than superior care. This suggests caution in the use of central tendency measures and the proportion of 10 ratings when examining racial/ethnic differences in CAHPS ratings of care. It is advisable to consider pooling responses at the top end (e.g. 9 and 10) and lower end (e.g. 0–6) of the response scale when making racial/ethnic comparisons.

Introduction

The Consumer Assessments of Healthcare Providers and Systems (CAHPS®) surveys are widely used for measuring consumer experiences with providers and health plans and include both global ratings and reports of care. The CAHPS health plan surveys were designed to provide information that would allow consumers to compare health care plans and select the plan and services that are most appropriate to their needs (1). Reports of care capture the specific experiences with care in terms of what did or did not happen from the consumer’s perspective. CAHPS 2.0H (National Committee on Quality Assurance (NCQA) HEDIS version) reports assess five domains of health plan performance through specific items with verbal labels on each response option: getting needed care (access to care), timeliness of care (promptness of care), provider communication, staff helpfulness, and plan customer service.

The four global ratings ask for overall evaluations and use an 11-point (0–10) response scale, verbally anchored at 0 (“worst possible care”) and 10 (“best possible care”). CAHPS 2.0H global rating items evaluate the personal doctor or nurse (if applicable), specialists (if applicable), health care received, and health plan. CAHPS global ratings tend to be negatively skewed, with almost 50% of ratings in the highest 2 of 11 categories. The CAHPS consortium has established three alternative methods of collapsing the 0–10 scales for health plan reporting: a) 0–8, 9, 10; b) 0–7, 8–9, 10; c) 0–6, 7–8, 9–10. These different categorizations have the main goal of maximizing health plan differentiation. However, there has been limited research examining the impact of these categorizations on racial/ethnic comparisons (2).

Prior CAHPS studies have shown that Hispanics report care that is similar to or less positive than for non-Hispanic Whites, yet have more positive ratings of care (3–7). Yet to date there have been no studies examining this apparent Hispanic paradox. One potential explanation for this paradox is that reports of care are more objective and better capture differences in care, whereas ratings are more subjective and may be influenced by expectations (8) and skepticism (9). Ratings of care may obscure differences in care if Hispanics have lower expectations (5). Another contributing factor may be response style differences or differential use of the 0–10 global rating scales by Hispanics and non-Hispanic Whites. Response tendencies play a bigger role in more subjective items such as the ratings of care. Understanding cultural differences in the use of rating scales is critical in evaluating racial/ethnic differences in consumer assessments of care. In this study we focus on response style differences between Hispanics and non-Hispanic Whites.

There may also be differences in the use of CAHPS rating scales by type of health insurance. Members of Medicaid managed care plans are more likely to be lower in socio-economic status than those in commercial managed care plans. Damiano et al. (2) showed that Medicaid enrollees were more likely than state employees in Iowa to use the extreme ends of the CAHPS global ratings, even after adjusting for socio-demographic differences in the populations. The purpose of this study is to examine differential use of the 0–10 scales in the CAHPS® health plan survey by Hispanic ethnicity and insurance status (Medicaid vs. commercial managed care)

Response Style and Bias in Survey Scales

Response style refers to a tendency to “respond systematically to questionnaire items on some basis other than what the items were designed to measure” (10, p. 43). As a result, response styles can result in bias in survey research by inflating or deflating respondents’ scores. One of these response styles is the extreme response style or the tendency for survey respondents to select both endpoints of a response scale. Extreme response tendency (ERT) can result in inflated scores (more positive) and negatively skewed measures (10).

ERT in cross-cultural research can be measured by comparing response patterns for different racial/ethnic groups. Prior research has shown that African Americans and Hispanics in the U.S. are more likely than Whites to use the extreme responses in the scale (11–13). In contrast, Asians are more likely to use the midpoint of the scale than Whites (14). Whites tend to have response patterns that are less extreme than Hispanics and African Americans, but more extreme than Asians. These cultural differences have been attributed to variations in cultural conversational norms, such as differences in emphasis on sincerity versus modesty in social interactions (15). African Americans and Hispanics may emphasize sincerity and as a result be more willing to respond in stronger terms, while Asians may emphasize modesty and therefore be more cautious in their responding.

In the presence of differences in ERT, racial/ethnic comparisons that focus on means of CAHPS ratings may misinterpret higher scores as reflecting only better experiences with care. If Hispanics are more likely than Whites to select extreme responses to CAHPS global rating items, then the observed higher ratings of Hispanics may be in part a result of using the 0–10 scale differently rather than wholly a result of more positive experiences than Whites.

Methodology

Data

We analyzed CAHPS 2.0H adult Medicaid and commercial data submitted to the National Committee for Quality Assurance (NCQA). This file includes CAHPS 2.0H data from 96 Medicaid managed care plans in 33 states and 370 commercial managed care plans from all 50 states and the District of Columbia in the year 2000. Our data includes both plans that are non-publicly reporting to NCQA, as well as those publicly reported in Quality Compass® 2001. The survey population was made up of currently enrolled members who were at least 18 years old. The data were collected by telephone or mail, and surveys were administered in Spanish and English. Prior studies provide support for the cultural and linguistic appropriateness of the Spanish version of the CAHPS® 2.0 survey (16–18).

As per the U.S. Decennial Census, race/ethnicity was established from an item asking about Hispanic ethnicity and a second item with five options regarding race (White, Black/African American, Asian, Pacific Islander, American Indian/Native Alaskan, Other). Subjects endorsing Hispanic ethnicity and/or White race were retained in the analytic dataset. Other racial/ethnic groups were excluded from the analysis.

Dependent and Independent Variables

The three dependent variables were the CAHPS 2.0 ratings of care: personal doctor or nurse, specialists, and health care received. Although there are four CAHPS 2.0 ratings, one (health plan rating) was dropped from the analysis because it violated the assumptions of the model needed to test for ERT (see Analysis Approach section).

Ratings were categorized into four levels to highlight any variations in extreme responding: ‘0–4,’ ‘5–8,’ ‘9,’ and ‘10,’ with ‘5–8’ as the reference category. This categorization follows the approach used by Damiano et al. (2).

A dichotomous indicator of belonging to a Medicaid managed care plan distinguished such enrollees from those enrolled in a commercial managed care plan as the reference category. The indicator of Hispanic ethnicity was used to distinguish Hispanics from non-Hispanic Whites. These two variables were combined to construct an unordered four-level categorical variable with values of: Hispanic Medicaid, Hispanic commercial, (non-Hispanic) White Medicaid, and (non-Hispanic) White commercial (reference).

Six potential confounders were controlled for in the models: gender, age (18–24, 25–34, 35–44, 45–54, 55–64, 65–74 or 75 or older), education (8^th grade less, some high school, high school graduate, some college, four-year college graduate or more than four years of college), self-rated health (excellent, very good, good, fair or poor), survey mode (phone or mail) and survey language (English or Spanish). These characteristics are known to be systematically related to differences in the rating of health plans by enrollees (19, 20). All variables used the same categorical forms in which they were elicited on the CAHPS survey. Missing data for the covariates ranged from 0.3% (gender) to 0.8% (education). Approximately 1% of those completing each global rating were omitted from multivariate models because of one or more missing independent variables.

Analytic Approach

Multinomial logistic regression was used to test for differences in extreme response tendency. This approach, unlike ordinal logistic regression, allows one to model an ordered or unordered polychotomous dependent variable without assuming that independent variables have proportionate effects on all levels of the outcome. With multinomial logistic regression, a given independent variable may increase or decrease the probability of both low and high values of an ordinal outcome relative to the middle, making it suitable to assess ERT.

The multinomial logistic regression assumes that choices are independent from irrelevant alternatives (IIA), meaning that the ratio of the probabilities of choosing any two alternatives is independent of any other alternative. We used the Hausman test to evaluate the IIA assumption (21), and found no evidence of IIA violation for three CAHPS ratings: personal doctor, specialist, and health care. On the other hand, for the health care rating we found a substantial violation of IIA. As a result, we decided not to include the health plan rating in the analysis.

For each of the three outcomes, multinomial logistic regression predicted the global rating from the three ethnicity/insurance dummies and the covariates described above and odds ratios are provided for race/ethnicity and insurance groups.(22)

Non-response weights, computed as the inverse of health plan response rates, were used to account for variation in response rate by plan. As a result, respondents belonging to plans with lower response rates received larger weights, and all respondents within the same plan were assigned the same weight. All analyses accounted for the variance inflation effects of weighting on estimates using the Taylor linearization method in Stata (23).

Results

The overall response rate for all plans was 46% (Range- 15%–73%). For commercial plans the response rate was 48% (Range: 15%–73%), while for Medicaid plans the response rate was 38% (Range: 18%–62%). The analytic sample consists of 215,712 respondents: 178,793 non-Hispanic Whites enrolled in commercial plans, 15,538 non-Hispanic Whites in Medicaid plans, 16,310 Hispanics in commercial plans, and 5,071 Hispanics enrolled in Medicaid plans. Lower sample sizes for Hispanics than Whites reflect existing levels of enrollment, given that CAHPS uses simple random samples within plans. Larger sample sizes for commercial enrollees relative to Medicaid beneficiaries reflect the inclusion of more commercial plans.

Table 1 describes the distribution of the case-mix variables (age, gender, education, self-rated health, survey language, and survey mode) by the four ethnic/insurance groups (Hispanic Medicaid, Hispanic Commercial, White Medicaid, White Commercial). Respondents in Medicaid were younger and less educated, especially Hispanics in Medicaid. In addition, respondents in Medicaid were more often female and had poorer self-rated health. Compared to Whites, a higher proportion of Hispanics completed phone surveys. A higher proportion of Hispanics in Medicaid (19%) completed Spanish surveys compared to all other ethnic/insurance groups.

Table 1.

Characteristics of the Sample by Race/Ethnicity and Insurance

Variable	Hispanic		White		Total
	Medicaid	Commercial	Medicaid	Commercial
N	5,071	16,310	15,538	178,793	215,712
Age (%)
(Chi-square= 8883.9*)
18–24 Years	18.9	7.1	15.0	4.5	5.8
25–34 years	22.1	23.3	22.1	15.9	17.1
35–44 years	22.6	28.3	24.7	26.1	26.1
45–54 years	14.9	23.4	16.6	28.5	26.9
55–64 years	10.6	14.0	12.7	20.4	19.2
65–74 years	6.3	3.0	4.7	3.3	3.4
75 years or older	3.6	0.5	3.6	0.9	1.1
Missing	1.2	0.5	0.6	0.4	0.4
Gender (%)
(Chi-square= 1865.6*)
Male	21.8	40.0	24.0	38.5	37.2
Female	77.6	59.6	75.5	61.2	62.5
Missing	0.6	0.4	0.5	0.3	0.3
Education (%)
(Chi-square= 31213.3*)
Eighth grade or less	22.3	5.4	8.3	0.7	2.1
Some high school	22.5	7.7	18.3	3.4	5.2
High school graduate	30.8	28.9	38.2	27.5	28.4
Some college	18.9	35.0	26.5	33.9	33.1
College graduate	2.1	12.7	4.3	17.9	16.2
More than four years of college	1.1	8.9	2.8	15.9	14.1
Missing	2.4	1.3	1.7	0.7	0.9
Self-Rated Health (%)
(Chi-square= 16372.3*)
Excellent	13.1	17.0	10.4	18.7	17.9
Very Good	20.5	34.8	22.9	41.4	39.1
Good	30.6	35.0	31.5	31.0	31.3
Fair	26.0	10.6	24.1	7.1	9.0
Poor	8.9	1.7	10.2	1.1	2.0
Missing	0.9	0.9	0.8	0.8	.8
Survey Language
(Chi-square= 26168.9*)
English	80.9	98.9	99.9	99.8	99.3
Spanish	19.2	1.1	0.1	0.2	.7
Survey Mode
(Chi-square= 513.2*)
Phone	24.2	23.4	17.1	17.3	17.9
Mail	75.8	76.7	82.9	82.7	82.1

^*p < .0001

Multinomial logistic regressions results for the three global ratings are shown in Table 2 in the form of odds ratios. These are interpreted as the amount by which the odds of the specified outcome category are multiplied relative to the referent outcome category (‘5–8’) and the referent category of a predictor, controlling for other independent variables. We use the odds ratios to measure ERT effect size, and in most instances we obtained large odds ratios for our ethnic comparisons.(24) For example, the odds ratio of 1.42 in the cell of Table 2 corresponding to Hispanic Commercial and the ‘0–4’ rating for personal doctors means that the odds of a rating of ‘0–4’ relative to the odds of a rating of ‘5–8’ are 1.42 times as high for a Hispanic Commercial respondent when compared to the same ratio of ‘0–4’ ratings to ‘5–8’ ratings in White Commercial (reference) respondents, after controlling for age, gender, education, self-rated health, survey mode, and survey language. These odds ratios are thus “ratios of ratios.”

Table 2.

Odds Ratios for Personal Doctor, Specialist, and Health Care Ratings Relative to 5–8 by Race/Ethnicity and Insurance

Variable	Proportions of Ratings	Hispanic		White
		Medicaid	Commercial	Medicaid	Commercial
Personal Doctor Rating (N = 179,059)
0–4	3.4%	1.44***	1.42***	1.21*	(1.00)
5–8	46.4%	(1.00)	(1.00)	(1.00)	(1.00)
9	18.8%	1.08	1.07*	1.03	(1.00)
10	32.8%	1.80***	1.29***	1.50***	(1.00)
Specialist Rating (N = 108,246)
0–4	4.6%	1.16	1.24***	1.10	(1.00)
5–8	39.8%	(1.00)	(1.00)	(1.00)	(1.00)
9	21.4%	0.84	0.97	0.91*	(1.00)
10	35.3%	1.35***	1.18**	1.33***	(1.00)
Health Care Rating (N = 182,104)
0–4	4.2%	1.39***	1.37***	1.17*	(1.00)
5–8	48.9%	(1.00)	(1.00)	(1.00)	(1.00)
9	23.5%	1.06	1.07*	1.00	(1.00)
0	24.8%	1.98***	1.30***	1.58***	(1.00)

Reference Category: 5–8

^***p< 0.001;

^**<0.01;

^*p<0.05

Adjusting for age, education, gender, self-rated health, survey language, and survey mode.

For personal doctor rating, Hispanic Medicaid enrollees were significantly more likely than White commercial enrollees to use 0–4 (O.R.= 1.44) and 10 (O.R. = 1.80), while Hispanic commercial were significantly more likely to endorse 0–4 (O.R.= 1.42) and 10 (O.R. 1.29). White Medicaid enrollees were significantly more likely than White commercial enrollees to endorse 0–4 (O.R.= 1.21) and 10 (O.R.= 1.50) (Table 2).

For specialist rating, Hispanic Medicaid enrollees were significantly more likely than White commercial enrollees to endorse ‘10’ (O.R.= 1.35), while Hispanic commercial enrollees were significantly more likely to endorse ‘0–4’ (O.R.= 1.24) and ‘10’ (O.R.= 1.18). White Medicaid enrollees were significantly more likely than White commercial enrollees to endorse 10 (O.R.= 1.33).

For health care rating, Hispanic Medicaid were significantly more likely to endorse ‘0–4’ (O.R.= 1.39) and ‘10’ (O.R.= 1.98) than White commercial enrollees, while Hispanic commercial enrollees were significantly more likely to use ‘0–4’ (O.R.= 1.37) and ‘10’ (O.R.= 1.30). White Medicaid enrollees were significantly more likely than White commercial enrollees to endorse ‘0–4’ (O.R.= 1.17) and ‘10’ (O.R.= 1.58).

Results suggest substantial ethnic differences in scale use. The odds ratios for the ‘0–4’ and ‘10’ categories for Hispanics are significantly greater than 1 in 11 of 12 instances across the three ratings and two Hispanic subgroups. Furthermore, there was a median value 1.4 for the 12 odds ratios contrasting Hispanic selection of ‘0–4’ or ‘10’ with the selections by Whites, which suggests that the odds of ERT are 40% higher among Hispanics compared to Whites.

Conclusions

Prior studies using CAHPS data have shown that, compared to Whites, Hispanics have similar or less positive reports of care, yet have higher ratings of care. Analysis of this data (results not shown) confirms that Hispanics report lower scores for timeliness of care, access to care, and staff helpfulness, yet have higher ratings than Whites for all ratings: personal doctor, specialist, health care, and health plan. Results from this study suggest that this apparent paradox may be partially attributable to response tendencies associated with both ethnicity and socio-economic status (as shown by Medicaid insurance status). Hispanics exhibited a greater tendency towards extreme responding in the CAHPS ratings than non-Hispanic Whites, where they were more likely than Whites in commercial plans to endorse a ‘10,’ and often, scores of ‘4’ or less, relative to an omitted category of ‘5–8.’ Furthermore, we observed a gradient wherein Hispanics in Medicaid often had a greater probability of endorsing the top rating ‘10’ but a lower probability of endorsing the bottom ratings ‘0–4’ than Hispanics in commercial plans. Therefore, the observed higher Hispanic ratings in disparities research of patient experiences may be partially attributed to differences in response styles rather than variations in care. Many ratings of 10 rather than 9 by Hispanics and Medicaid enrollees may reflect ERT rather than especially positive experiences of care. Evidence of ERT on both ends of the response scale is the key to disambiguating the interpretation.

These results suggest that the more subjective global ratings may not be the best metric in racial/ethnic disparities research. If used, there should be caution in the use of central tendency measures and the proportion of 10 ratings in examining racial/ethnic differences in CAHPS ratings of care. If extreme response tendency is of a substantial magnitude, then reporting means or proportions of 10s may substantially overestimate true Hispanic healthcare experiences. For example, based on mean comparisons, health plans with a higher proportion of Hispanics may appear to perform better than other plans as a result of the Hispanic response style. Therefore, it is suggested that racial/ethnic comparisons across providers be based on an examination of score distributions rather than measures of central tendency (13).

Current Agency for Healthcare Research and Quality (AHRQ) guidelines suggest two alternative approaches for analyzing CAHPS ratings, by either combining the ‘9’ and ‘10’ ratings or reporting the ‘10’ rating separately (25). This study suggests that reporting combined proportions of ‘9’ or ‘10’ ratings may be preferable to reporting the proportion of ‘10’ rating alone, as the former approach may be less sensitive to extreme response style. Such a change might make comparisons more equitable when comparing plans with very different proportions of Hispanic respondents. The tradeoff would be that combining the ‘9’ and ‘10’ ratings would reduce plan differentiation within race/ethnic groups.

Study findings also showed that Whites in Medicaid plans were more likely than Whites in commercial plans to endorse a ‘10,’ and often, scores of ‘4’ or less, relative to an omitted category of ‘5–8.’ Hispanics in Medicaid had a higher probability of endorsing the top rating ‘10’ and bottom ratings ‘0–4’ than Whites in Medicaid. These findings confirm that response style differences are not limited to ethnicity but are also influenced by socio-economic status (as shown by Medicaid insurance status). This also suggests that the prior recommendation of reporting combined proportions of ‘9’ or ‘10’ ratings might be helpful also when comparing Medicaid and commercial plans.

Future research should address the limitations of this study. First, there may be other factors, besides differences in response styles, which may explain the observed higher Hispanic ratings in patient experiences with care. More research is needed examining the role of respondent expectations and skepticism on the observed racial/ethnic differences in patient experiences of care. Second, this study did not explore the causes of ERT by Hispanics; it exposed the presence of ERT by Hispanics in CAHPS ratings. Studies that examine the cultural dimensions associated with Hispanics’ extreme response style in CAHPS ratings are needed. Johnson et al. argues than an extreme response style is congruent with a “motivation to achieve clarity, precision, and decisiveness in one’s explicit verbal statements” (26, p. 266); attributes that may be more valued in certain cultures. For example, prior cross-cultural research has shown that persons in cultures characterized by high masculinity (emphases on assertiveness and decisive behavior), high power distance, and an individualistic orientation are more likely to engage in extreme responding (26, 27). However, these studies have been limited to international comparisons between countries. Additional research is needed examining the cultural attributes that may explain differences in response styles between US racial/ethnic groups. Finally, both reports and ratings of care may be subject to extreme response tendency (ERT). Future research should examine ERT in CAHPS reports of care.