Possibilities for Shortening the CAHPS Clinician and Group Survey

Abstract

Background:

The Consumer Assessment of Healthcare Providers and Systems (CAHPS) Clinician & Group adult survey (CG-CAHPS) includes 34 items used to monitor the quality of ambulatory care from the patient’s perspective. CG-CAHPS includes items assessing access to care, provider communication, and courtesy and respect of office staff. Stakeholders have expressed concerns about the length of the CG-CAHPS survey.

Objectives:

This paper explores the impact on reliability and validity of the CAHPS domain scores of reducing the numbers of items used to assess the three core CG-CAHPS domains (Provider Communication, Access to Care, and Courteous and Helpful Office Staff).

Research Design:

CG-CAHPS data reported here consist of 136,725 patients across four datasets including ambulatory clinics, patient-centered medical homes, and accountable care organizations. Analyses are conducted in parallel across the four settings to allow evaluations across data source.

Analyses:

Multiple regression and ANOVA techniques were used to evaluate reliability for shorter sets of items. Site-level correlations with the overall rating of the provider were compared to evaluate the impact on validity. The change in practices’ rank-ordering as a function of domain revision is also reported.

Results:

Findings suggest that the Provider Communication (6-items) and Access (5-items) domains can be reduced to as few as two-items each and Office Staff (2-items) can be reduced to a single item without a substantial loss in reliability or content.

Conclusions.

The performance of several of the reduced-length options for CG-CAHPS domains closely matches the full versions and may be useful in healthcare settings where the full-length survey is impractical due to time or cost constraints.

Introduction

The Consumer Assessment of Healthcare Providers and Systems (CAHPS^®) surveys are used by health plans to assess quality and for quality improvement initiatives, and by consumers and patients to assist in selecting among healthcare professionals, group practices, and health plans [1]. The CAHPS Clinician & Group (CG-CAHPS) survey is reported on the Centers for Medicare & Medicaid Services Physician Compare website, and a variant of CG-CAHPS is being used to evaluate Accountable Care Organizations participating in the Medicare Shared Savings Program. CG-CAHPS^® surveys have been administered to over 1.5 million patients from over 5,000 US medical practices [2], making it among the most frequently adopted survey for assessing patient experiences with care received from providers and staff in primary, specialty, and ambulatory care settings.

The full-length CG-CAHPS^® survey is perceived to be lengthy by some healthcare organizations (sponsors) who administer the instrument. The adult, 12-month CG-CAHPS^® survey includes 34 questions on a 5-page survey that assesses domains such as Provider Communication, Access to Care, and Courteous and Helpful Office Staff and can be completed in approximately 15 minutes. A shortened version of the CG-CAHPS^® survey would reduce patient and administrator burden [3], potentially increase response rates [4], and may enhance its use and impact on the delivery of healthcare.

We report findings from four separate implementations of the CG-CAHPS^® survey to explore the effects of shortening it on the reliability and validity of measurement. The CG-CAHPS^® survey is standardized to ensure comparability across providers and groups. Because of this, it is important to ensure that any potential revision maintains the content of the domains and does not adversely affect the statistical properties of the measure.

Methods

Data Collection and Procedures

Survey data were collected using a combination mail and telephone modes of administration. Data reported here consisted of responses from 136,725 participants obtained from four separate CG-CAHPS collection efforts. 1) The physician group setting consists of 53 ambulatory clinic locations, and 62 individual physicians, from which 63,441 respondents (response rate = 37%) were sampled from May 2005 to January 2009 [5]. 2) The safety net setting includes data from 7,230 participants (response rate approximately = 11%) who visited 28 practices in southern California from August 2012 to March 2014. 3) The third setting reported here includes responses from 2,740 participants visiting 6 health maintenance organizations that implement the patient-centered medical home (PCMH) model of care delivery (response rate = 37%; [6]). 4) The sample reported here consists of 63,415 beneficiaries (response rate = 54%) from 152 Accountable Care Organizations (“ACO”) who were sampled in January 2013 to March 2013. ACOs are collections of doctors, hospitals, and health care providers that have organized with an emphasis on performance measurement within populations they serve.

Demographics

The majority of the study participants were White (78%), Female (59%), had completed at least some college (67%), and self-reported good, very good or excellent general health (73%). Table 1 presents demographic comparisons across setting. The safety net sample had substantially more non-whites (49%) and less than high school educated (31%) respondents than the other samples. The ACO sample had by far the oldest respondents (47% 75 or older).

TABLE 1.

Beneficiary Demographic Characteristics and Health Status Across 4 CG-CAHPS Samples

Characteristics	Physician Group (% Nonmissing) (N =63,441)	Safety Net (% Nonmissing) (N = 7192)	PCMH (% Nonmissing) (N = 2740)	ACO (% Nonmissing) (N = 63,415)
Age (y)
18–24	598 (2)	635 (9)	82 (3)	32 (0)
25–34	2394 (8)	1234 (17)	144(5)	308 (0)
35–44	3166 (10)	1169 (16)	310 (11)	804 (1)
45–54	4727 (15)	1621 (23)	484 (18)	2222 (4)
55–64	6607 (21)	1880 (26)	701 (26)	4358 (7)
65–74	7029 (22)	471 (7)	641 (24)	24,909 (40)
Z75	7028 (22)	182 (3)	359 (13)	29,176 (47)
Missing	31892	0	11	1606
Sex
Male	25,362 (41)	2373 (33)	1048 (38)	27,098 (43)
Female	37,095 (59)	4818 (67)	1692 (62)	36,317 (57)
Missing	984	1	—	—
Race/ethnicity
White	46,206 (75)	3051 (51)	1890 (71)	49,016 (85)
Nonwhite	15,026 (25)	2945 (49)	760 (29)	8590 (15)
Missing	2209	1196	90	5809
Education
Less than high school	2910 (5)	2064 (31)	234 (9)	9128 (15)
High school graduate	6792 (11)	1557 (23)	583 (22)	20,266 (33)
Some college	16,948 (27)	2087 (31)	1152 (43)	16,186 (26)
4 y degree or more	35,420 (57)	1039 (15)	735 (27)	15,756 (26)
Missing	1371	445	36	2079
General health status
Excellent	8077 (13)	900 (13)	301 (11)	4207 (7)
Very good	18,569 (30)	1744 (25)	897 (33)	15,301 (25)
Good	20,219 (33)	2378 (34)	1063 (39)	22,919 (37)
Fair	11,515 (19)	1678 (24)	377 (14)	14,589 (24)
Poor	3624 (6)	383 (5)	72 (3)	4313 (7)
Missing	1437	109	30	2086
Mental health status
Excellent	19,335 (31)	1873 (26)	703 (26)	12,766 (21)
Very good	20,334 (33)	1980 (28)	950 (35)	19,197 (31)
Good	15,276 (25)	1874 (27	737 (27)	19,127 (31)
Fair	6026 (10)	1061 (15)	280 (10)	8542 (14)
Poor	1181 (2)	284 (4)	280 (2)	1814 (3)
Missing	1289	120	24	1969

ACO indicates Accountable Care Organization; CG-CAHPS, Consumer Assessment of Healthcare Providers and Systems Clinician and Group adult survey; PCMH, patient-centered medical home.

Measures

The CG-CAHPS Adult Survey includes 13 items that form three composites: Communication (6 items), Access (5 items), and Office Staff (2 items) [7, 8,9]. The survey also includes a single item asking respondents to provide an overall rating of their provider on a 0–10 scale (see Table 2).

TABLE 2.

Items and Abbreviations From CG-CAHPS Core Domains

Scales and Items	Item Abbreviation
How Well Your Providers Communicate
How often did this provider show respect for what you had to say?	Respect
How often did this provider listen carefully to you?	Listen
How often did this provider give you easy to understand information about these health questions or concerns?	Information
How often did this provider explain things in a way that was easy to understand?	Understand
How often did this provider spend enough time with you?	Spend enough time
How often did this provider seem to know the important information about your medical history?	Medical history
Getting Timely Care, Appointments, and Information
When you phoned this providers office during regular office hours, how often did you get an answer to your medical question that same day?	During office hours
When you phoned this providers office after regular office hours, how often did you get an answer to your medical question as soon as you needed?	After office hours
How often did you see this provider within 15 min of your appointment time?	Timely
When you phoned this providers office to get an appointment for care you needed right away, how often did you get an appointment as soon as you needed?	Urgent care
When you made an appointment for a check-up or routine care with this provider, how often did you get an appointment as soon as you needed?	Routine care
Helpful, Courteous, and Respectful Office Staff
How often were clerks and receptionists at this providers office as helpful as you thought they should be?	Helpful office staff How
often did clerks and receptionists at this providers office treat you with courtesy and respect?	Courtesy and respectful
Provider Rating	Provider Rating
Using any number from 0 to 10, where 0 is the worst provider possible and 10 is the best provider possible, what number would you use to rate this provider?

CG-CAHPS indicates Consumer Assessment of Healthcare Providers and Systems Clinician and Group adult survey.

Most of the data reported here were collected using the CAHPS survey version 2.0 [10], which utilizes a 12-month recall period and 4-point response options for all response items (Never, Sometimes, Usually, and Always). The ACO survey uses a 6-month recall period. The physician group sample was administered the C-G CAHPS 1.0 survey [11]. This version of the survey includes minor wording differences for the Access items, uses items that refer to the “doctor” rather than “provider”, and has a 6-point response scale (Never, Almost never, Sometimes, Usually, Almost always, and Always). To maintain consistency with CAHPS 2.0, we recoded “Almost never” as “Never” and “Almost always” as “Always.” The safety net sample was administered the visit version of the survey that had a 3-point response scale (“no”, “yes, somewhat”, and “yes, definitely’) for the Communication and Office Staff items; we dichotomized these into “yes” or “no” response.

Analytic Approach.

The purpose of this study was to determine 1) the minimum length (or number of items) of each domain while maintaining core domain content and site-level reliability and 2) to provide some options for short item subsets that perform well in comparison to the original domains. We evaluated $\left(\genfrac{}{}{0pt}{}{n}{k}\right)$ combinations of items in a given subset length where n is the total number of items per domain and k is the number of items in a given subset (i.e., the length of the subset). For example, in the 6-item Communication domain there are 62 possible item subsets because there are n – 1 possible lengths of each domain.

A two-step process was used to evaluate each item combination. The first step in the process was conducted to identify how short the domains could reasonably be without impacting reliability or unduly limiting the content of the domains. Step 1: First, all possible combinations for the Communication, Access, and Office Staff domains, we regressed the CAHPS full-length domain score on each combination of item subsets across samples, which provided an R² for each item subset [12]. We then estimated the practice-level reliability of each item subset considered using ANOVA to partition between versus within practice variance. The results of step 1 provided recommendations for the minimum length of each domain based on the variance accounted for in the full-length version, the site-level reliability, and the content represented by the subset.

The second step in the analysis was conducted on the item combinations for the minimum domain lengths obtained in step 1 in order to provide the analytic properties of some shortened domains. Step 2: Because the number of participants assigned to each practice or group varied across samples, we used intraclass correlations to estimate sample sizes needed to obtain practice-level reliabilities of 0.70, 0.80, and 0.90 for each recommended subset. We estimated practice-level correlations between the CAHPS single-item overall rating of the provider with CAHPS composite scores estimated from the full-length and recommended shortened item sets. Finally, we compared the percentile rank of the ACO practice scores using scores estimated from the original and the various recommended reduced domains. The difference in the percentile rank is an index of how much a revision to a domain affects the rank-ordering of the practices; as a summary of the difference we also report the average of the absolute value of the difference and the range of differences across practices.

Results

Identifying reduced-length measures

Table 3 presents the results for only the most informative item combination from each reduced length option. Results for both Communication and Access indicate that reduced domains with as few as two items remain closely related to the full-length domains. Note that because Office Staff contains only two items, it is not evaluated in this step. For two-item combinations the percentage of variance accounted for across samples ranges from 81%−92% and 86%−90%, for Communication and Access, respectively. In addition, the most informative two-item combinations are highly reliable across samples and are similar to the reliability of the original domains (0.79 to 0.96 and 0.87 to 0.98, for Communication and Access, respectively). Note that the variance in the original Communication and Access domains is substantially reduced in both one-item options.

TABLE 3.

Summary of the Percentage of Variance Accounted for the in the Original Domains by the Reduced Domains Across Item Combinations

	Variance Accounted for in the Original Domain (%)					Site-level Reliability
Domain and Length of Item Subsets	ACO	PCMH	Physician Group: Physician	Physician Group: Practice	Safety Net	ACO	PCMH	Physician Group: Physician	Physician Group: Practice	Safety Net
Provider Communication (original domain)	—	—	—	—	—	0.80	0.80	0.95	0.97	0.87
5 items (Understand, Listen, Information, Medical history, Spend enough time)	99	99	99	99	99	0.80	0.82	0.95	0.97	0.87
4 items (Understand, Listen, Medical history, Spend enough time)	97	98	98	98	96	0.80	0.81	0.95	0.97	0.87
3 items (Understand, Medical history, Spend enough time)	93	94	96	96	91	0.79	0.86	0.95	0.97	0.87
2 items (Understand, Spend enough time)	86	86	92	92	81	0.79	0.87	0.95	0.96	0.85
1 item (Spend enough time)	67	70	80	80	62	0.75	0.81	0.95	0.97	0.84
Access (original domain)	—	—	—	—	—	0.77	0.88	0.95	0.96	0.84
4 items (Urgent care, Routine care, During office hours, Timely)	99	98	99	99	97	0.88	0.92	0.95	0.96	0.89
3 items (Urgent care, Routine care, Timely)	94	94	95	95	91	0.90	0.86	0.94	0.96	0.91
2 items (Routine care, Timely)	88	88	90	90	86	0.94	0.87	0.97	0.98	0.91
1 item (Timely)	66	64	73	73	59	0.96	0.71	0.98	0.99	0.87

Results presented here were selected from the item subset among all possible combinations within a given length that resulted in the highest site-level reliability. The items representing the highest site-level reliability for a given length are provided in parenthesis. Complete item text can be found in Table 2.

ACO indicates Accountable Care Organization; PCMH, patient-centered medical home.

Having identified 2 items as the minimum length for the Communication and Access domains, we reviewed all possible combinations of 2-item subsets with each domain to consider content along with reliability. Among all combinations, Table 4 includes the 2-item subsets that have both the highest reliabilities and the content essential to represent the Communication and Access domains. For the Communication domain, the spend enough time item performs well when paired with either understand or listen items and results in two-item composites that provide nearly equivalent site-level measurement precision as the 6-item full-length composite. For the Access domain, the timely item performs well when paired with either routine care or during office hours items and results in two-item composites that in some settings are more reliable than the 5-item original measure. The single-item versions of Office Staff are somewhat more inconsistent and the degree to which the single-item versions impact reliability is less clear.

TABLE 4.

Reliabilities of the Recommended Minimum Communication, Access, and Courteous and Helpful Office Staff Reduced Domains

Domain and Item Subsets	Site-level Reliability					Site-level Sample Size Associated With Reliability at 0.70
	ACO	PCMH	Physician Group: Physician	Physician Group: Practice	Safety Net	ACO PCMH		Physician Group: Physician	Physician Group: Practice	Safety Net
Provider Communication (Original domain)	0.80	0.80	0.95	0.97	0.87	210	265	111	90	78
Understand, Spend enough time	0.79	0.87	0.95	0.96	0.85	221	155	116	97	90
Listen, Spend enough time	0.78	0.70	0.95	0.97	0.86	235	447	111	91	82
Access (Original domain)	0.77	0.88	0.95	0.96	0.84	250	141	117	102	93
Routine care, Timely	0.94	0.87	0.97	0.98	0.91	53	156	68	61	51
During office hours, Timely	0.91	0.89	0.97	0.98	0.82	79	126	65	55	109
Courteous and Helpful Office Staff (Original domain)	0.87	0.85	0.95	0.97	0.55	127	189	190	194	405
Helpful office staff	0.85	0.83	0.95	0.96	0.39	141	219	168	150	791
Courteous and respectful	0.84	0.82	0.93	0.96	0.66	152	236	193	212	254

ACO indicates Accountable Care Organization; PCMH, patient-centered medical home.

Preliminary validity of reduced-length measures

Table 5 presents the correlations for the recommended 2-item combination options of Communication and Access and the single-item versions of Office Staff with the original full-length versions of the domains and with the Provider Rating item. The recommended minimum-length domain options yield scores that are closely related to their original full-length versions. Across samples and item subsets the Pearson correlations range from 0.92–0.98, 0.80–0.96, and 0.97–1.00, for Communication, Access, and Office Staff, respectively. Compared to the original domains, there is only a slight reduction in correlations between the reduced domains and the Provider Rating, though there is little reduction for Office Staff. Note that the PCMH correlation results are based on very few sites (n =6) and are thus are excluded.

TABLE 5.

Correlations Between the Recommended Communication, Access, and Courteous and Helpful Office Staff Reduced Domains With the Original Domains and Overall Provider Rating

Domain and Item Subsets	Site-level Correlations With the Original Domain				Site-level Correlations With Overall Provider Rating
	ACO	Physician Group: Physician	Physician Group: Practice	Safety Net	ACO	Physician Group: Physician	Physician Group: Practice	Safety Net
Provider Communication	—	—	—	—	0.86	0.88	0.87	0.86
Understand, Spend enough time	0.97	0.97	0.98	0.98	0.84	0.81	0.82	0.82
Listen, Spend enough time	0.97	0.99	0.98	0.98	0.83	0.82	0.81	0.81
Access	—	—	—	—	0.65	0.58	0.64	0.50
Routine care, Timely	0.80	0.82	0.86	0.83	0.57	0.31	0.40	0.57
During office hours, Timely	0.84	0.86	0.94	0.91	0.59	0.37	0.49	0.51
Courteous and Helpful Office Staff	—	—	—	—	0.67	0.53	0.51	0.04
Helpful office staff	0.98	0.99	0.99	1.00	0.65	0.52	0.50	0.00
Courteous and respectful	0.97	0.98	0.98	1.00	0.66	0.54	0.50	0.07

ACO indicates Accountable Care Organization.

Finally, we evaluated the difference in the ACO sample percentiles between site-level scores from the original and reduced domains. The average difference can be interpreted as the absolute value of the expected change in the percentile for a given ACO when reporting a reduced domain. For the Communication domain the two options (understand and spend enough time; listen and spend enough time) both result in an average absolute percentile difference of 6% with a range across sites of 0–39% and 0–31% for both options, respectively. The average percentile differences and ranges are somewhat larger in magnitude for the Access revision options (average percentile differences = 13% and 15%, ranges = 0–68% and 0–72% for routine care and timely, and during office hours and timely, respectively). The average percentile differences and ranges is similarly small for the Office Staff single-item options (average percentile differences = 3% and 6%, ranges = 0–13% and 0–22% for helpful office staff and courtesy and respectful, respectively).

Discussion

This paper presents evidence supporting possible reductions in three core domains of the CG-CAHPS survey. Results indicate that the measures can be shortened while maintaining the general content measured by the original, full-length scales. Because CAHPS instruments are routinely used by consumers to evaluate healthcare options and by providers to evaluate the care being given, it is critical that the measures maintain standards of reliability at the site or program-level. Results presented here suggest the Communication and Access domains can be reduced to a minimum of 2 items each, and Office Staff to a single item, without loss in reliability and while maintaining the validity of the original domains.

Though the results presented here are promising, potential users of these reduced domain options should be aware of several limitations. The domain options presented here were derived from analytic findings and substantive judgment; obtaining feedback from patients may result in a different set of options [13]. In addition, the breadth of content measured by the original domains is an important consideration. Reducing the scales to two items (or a single item in the case of Office Staff) necessarily reduces the aspects of patient experiences that each domain measures, though the benefits of shorter length scales may offset this issue in certain contexts. Also, the intended use of reduced scales, as with all CAHPS measures, is at the level of the group/practice/site. Users intending to evaluate patient-level scores should be aware that shorter scales will result in lower participant-level reliability. In addition, we note that due to the potential shift in site-level scores that may accompany a substantial reduction in the number of items in a given domain, caution is needed in evaluating trends over time if one switches from the standard CAHPS survey to a shorter measure. The results presented here are indications of how the reduced-length measures would theoretically perform as reduced measures, but these estimates are based on prior administrations that included the original full-length survey. Future implementations of these measures are needed to evaluate the generalizability of the reliability evidence presented here and to provide an assessment of the potential impact of the revised measures across race, ethnic, and cultural groups [14, 15].

The reduced domain options presented here are a response to a perceived need for shorter surveys many users of CAHPS have expressed. The possible reductions to the survey include seven core reporting items and four screeners associated with the response items. This reduction of the CG-CAHPS survey from 34 to 23 items is estimated to reduce response burden by 25% and would translate to cost savings associated with administering the survey via telephone or mail. Finally, we note that the reduced domain options reported here are presented only as recommendations; based on these findings, some users of the CAHPS survey may prefer to maximize the reliability of the scales, while other users, based on program needs, may select scale options that contain particularly relevant domain content.