Differences in Consumer Assessment of Healthcare Providers and Systems (CAHPS®) Clinician and Group Survey Scores by Recency of the Last Visit: Implications for Comparability of Periodic and Continuous Sampling

Abstract

Background

Patient experience data can be collected by sampling patients periodically (e.g., patients with any visits over a one-year period) or sampling visits continuously (e.g., sampling any visit in a monthly interval). Continuous sampling likely yields a sample with more frequent and more recent visits, possibly affecting the comparability of data collected under the two approaches.

Objective

To explore differences in Consumer Assessment of Healthcare Providers and Systems Clinician and Group survey (CG-CAHPS®) scores using periodic and continuous sampling.

Research Design

We use observational data to estimate case-mix adjusted differences in patient experience scores under 12-month periodic sampling and simulated continuous sampling.

Subjects

29,254 adult patients responding to the CG-CAHPS survey regarding visits in the past 12 months to any of 480 physicians, 2007-2009.

Measures

Overall doctor rating and 4 CG-CAHPS composite measures of patient experience: doctor communication, access to care, care coordination, and office staff.

Results

Compared to 12-month periodic sampling, simulated continuous sampling yielded patients with more recent visits (by definition), more frequent visits (92% of patients with 2+ visits, compared to 76%), and more positive case-mix adjusted CAHPS scores (2 - 3 percentage points higher).

Conclusions

Patients with more frequent visits reported markedly higher CG-CAHPS scores, but this causes only small to moderate changes in adjusted physician-level scores between 12-month periodic and continuous sampling schemes. Caution should be exercised in trending or comparing scores collected via different schemes.

Introduction

Patient experience measures are key components of health care quality assessment in the United States (U.S.). Many policy initiatives for the improvement of care quality and the facilitation of patient choice include public reporting of providers’ performances on patient experience measures; some initiatives have incentivized providers to improve care through pay-for-performance programs (1). For example, the Patient Protection and Affordable Care Act expanded pay-for-performance in hospitals by establishing a Value-Based Purchasing program that rewards hospitals for how well they perform on quality measures and performance improvements (2). Further, ambulatory patient experience performance measurement is included in metrics related to governmental and commercial accountable care contracts encouraged by the Medicare Access and Children’s Health Insurance Program Reauthorization Act (3). Patient experience measures that are collected using psychometrically sound instruments, that employ adequate sample sizes and adjustment procedures, and that are implemented according to standard protocols are intrinsically meaningful and appropriate complements to clinical process and outcome measures in public reporting and pay-for-performance programs (4).

The Consumer Assessment of Healthcare Providers and Systems (CAHPS®) surveys are the most widely used patient experience measures in the U.S. (5–7). The CAHPS Clinician and Group Survey (CG-CAHPS) is used to assess care from physician groups and/or individual providers (8–10). Health care organizations can either conduct periodic or continuous sampling. The periodic sampling approach samples patients who have had a doctor’s office visit in the prior 12 (or 6) months with a named provider’s office. The continuous sampling approach samples patients on an ongoing basis, selecting patients who have had a recent visit and asking them about experiences with both their most recent visit and in the last 12 months. By design, continuous sampling is more likely than periodic sampling to survey patients closer to the time of a recent visit; the most recent visit in periodic sampling could have happened up to 12 months prior to the survey. While continuous sampling can facilitate more rapid feedback to providers than periodic sampling, it is unknown whether it yields patient experience scores that are comparable to those from periodic sampling.

In sampling procedures with shorter periods, patients with higher utilization (more frequent health care visits) are overrepresented relative to those in a longer deduplicated period (such as that used in periodic sampling) (11, 12). Overrepresentation increases with both the heterogeneity of utilization and the brevity of the sampling interval. Because greater intensity of health care utilization is associated with more positive perceptions of some aspects of care (11), continuous sampling could yield a different mix of patients (less healthy, higher utilization) and result in different CG-CAHPS scores than periodic sampling. If these differences are fully captured by standard CAHPS case-mix adjustment (CMA) models (13), then these compositional differences would not influence reported CG-CAHPS scores. However, if high utilizers differ in unmeasured ways or if being surveyed more proximally to one’s last visit affects responses, even case-mix adjusted scores may differ by sampling method. The sensitivity of case-mix adjusted scores to the frequency and recency of the last visit is of great interest for benchmarking, cross-sectional comparisons, trending over time, and public reporting.

As described in the following section, this study uses information about visit frequency and recency among survey respondents from two one-year rounds of 12-month periodic sampling to estimate differences in CG-CAHPS patient characteristics and case-mix adjusted patient reports about care associated with 12-month periodic and continuous sampling methods.

Methods

CG-CAHPS Survey Data

We analyze data from the CG-CAHPS 12-Month survey administered to adult patients of 480 primary care and specialty physicians in a large integrated health system, in which patients were asked about care received in the prior 12 months. Patients were eligible if they had at least one visit with their primary care or specialist physician (named in the survey) during the 12 months prior to the survey fielding date. The survey was administered using the periodic sampling method covering adult patients with at least one visit from March 2007 through January 2009. The most recent patient visits ranged from less than 3 months to 12 months before the survey fielding date (median = 6 months), permitting assessment of the associations of CG-CAHPS scores with last visit recency (in continuous months, assessed via administrative data) and self-reported visit frequency.

A random sample of 135 patients 18 years and older was drawn by the health care organization’s survey vendor for every primary care or specialist physician who had seen 100 or more unique patients in the prior 12 months, according to the professional billing system which includes patients covered by any insurance (Medicare, Medicaid, private insurance, etc.). The sampling procedure ensured that no more than one patient per household was sampled and that any sampled patient was included in only one physician’s sample. Patients were not allowed to be sampled more than once per 60 days or more than twice per year. Each patient who had seen more than one physician during the reference period was assigned to the physician whom they had visited most during the prior three months; ties were broken in favor of physicians with fewer patients to maximize the number of physicians with adequate sample sizes. Patients in the analytic sample confirmed having had at least one visit with that physician in the 12 months prior to the survey administration date. All surveys were administered in English by mail with telephone follow-up of mail non-respondents. A 37 percent response rate was obtained, with 29,254 completes (mean per physician = 60.9, SD= 24.5, Min=5, Max=126).

CG-CAHPS Measures

We examined 5 CG-CAHPS measures: the overall rating of the physician using a 0 (“worst provider possible”) to 10 (“best provider possible”) response scale and 4 composite measures assessing doctor communication (5 items), access to care (3 items), care coordination (3 items), and office staff (2 items) using a 6-level response scale (never, almost never, sometimes, usually, almost always, or always). Scores were linearly transformed to a 0–100 possible range with 100 representing the most positive experiences with care. Ordinal composite items coded a=1 to b=6 and overall physician ratings on an a=0 to b=10 were transformed using the formula: score_new = 100*(score_old − a)/(b- a). This transformation increases comparability to other transformed CAHPS survey scores, including the CG-CAHPS 3.0 survey, which uses a 4-level response scale, and facilitates the use of the CAHPS score difference magnitude criteria developed elsewhere (18).

Periodic and Simulated Continuous Samples

Table 1 describes the data used to obtain estimates corresponding to each of the sampling types considered in this paper. Estimates for the 12-month periodic sample employed all patients in the dataset, as 12-month periodic sampling was used to collect the data. All estimates for the 12-month periodic sample were unweighted.

Table 1:

Sample Inclusion

Sample	N	Sampling Type Represented	Patients Included	Weighting and Estimation
A	29,254	12-month periodic sampling	All patients with a visit <12 months before the survey	Unweighted
B	29,254	Simulated continuous sampling	All patients with a visit <12 months before the survey	Weighted by visit frequency†, projected to a mean visit recency of 1.5 months for estimation of CAHPS scores

^†5+ visits coded as 6

Continuous sampling is defined as sampling visits with equal probability, which is equivalent to sampling patients proportionately to their number of visits. In expectation, this is also equivalent to weighting deduplicated patients proportionately to their number of visits (14)- the expected mean characteristics of patients sampled continuously and the mean characteristics via such weighting are the same Continuous sampling will differ from periodic sampling in two ways: first, the patients who are selected will differ, because the method produces different probabilities of sample inclusion; second, patients will be surveyed much closer to the time of their encounter. We simulate these two aspects of continuous sampling through a combination of two methods. First, we use weighting, as described, to reproduce a sample of patients equivalent to those who would have been selected using continuous sampling. Second, we use linear regression to adjust for the effect of recency (a characteristic of the encounter, rather than the patient) on responses to CAHPS items.

To simulate and approximate results from continuous sampling due to differences in patients’ sample inclusion probabilities, we used all respondents from the 12-month periodic sample and frequency-weighted each observation with the number of self-reported visits in the past 12 months. This approach mimics continuous sampling by effectively sampling visits rather than deduplicated patients; a patient’s probability of selection is proportionate to their number of visits within a given period. Visit frequency was self-reported with response options 1, 2, 3, 4, 5-9, or 10+ visits in the past 12 months. When the number of visits was used as a frequency weight to simulate continuous sampling, responses of 5-9 and 10+ visits were approximated as 6; sensitivity analyses (not shown) suggest that our conclusions are not sensitive to this choice.

We then compared the estimates from the simulated continuous sampling approach to those from the 12-month periodic approach to calculate differences that might be observed if the different sampling methods were used for the same patient population and aggregated over a year of data collection.

Analysis

We estimated the characteristics of patients obtained under each sampling scheme. We then fit linear regression models to assess the average patient-level contribution of visit recency and frequency to any differences in CAHPS scores obtained under the different sampling approaches. These models regressed (ordinary least squares) each CAHPS measure on visit recency (in continuous months) and indicators for visit frequency categories. Scores were adjusted for year of data collection, patient gender, and standard CAHPS case-mix adjustors: patient age, education, and self-rated general and mental health (15). Models also included physician-level fixed effects to control for unmeasured physician characteristics.

Finally, we computed case-mix adjusted CAHPS scores obtained under each scheme. Physician-level effect sizes for differences in CAHPS scores were computed as the difference in means between sampling methods divided by the standard deviation of physician-level scores from the 12-month periodic sample. Scores were adjusted for year of data collection, patient gender, and the standard CAHPS case-mix adjustors listed above.

For the simulated continuous sample, weighting by visit frequency simulated the sample inclusion probabilities for each patient. We projected estimates for the simulated continuous sample to a mean recency of 1.5 months, a typical amount of time from visit to survey receipt for a monthly continuous sampling scheme. To do this, we used the recency coefficient from the CAHPS models described above, adjusting for the partial effect of recency on CAHPS scores after accounting for visit frequency and case-mix.

Statistical significance tests for the difference between the continuous and 12-month periodic samples, or the difference between frequency-weighted and unweighted estimates from the same data, were performed by testing whether the frequency weights were associated with the patient characteristic or CAHPS score. For differences in CAHPS scores, test statistics were adjusted for the additional uncertainty introduced by the regression-based projection to a mean visit recency of 1.5 months for the continuous sample.

Results

Fifty-five percent (n=16,143 patients) were surveyed less than 6 months after their most recent visit. Table 2 presents patient characteristics for the two sampling schemes. Compared to the 12-month periodic sample, the simulated continuous sample patients were far less likely to have had only one visit in the past 12 months (7.9% vs. 24.1%), essentially by definition. They tended to be significantly older, less educated, in worse general health, more likely to be Hispanic or Black, and less likely to be White.

Table 2:

Characteristics of patient population by sampling approach

		12-month Periodic Sample (Unweighted) [Sample A] n=29,254	Simulated Continuous Sample, Weighted by Visit Frequency, Projected to Mean 1.5 Months Recency [Sample B]	Difference [Sample B – Sample A]	p-value1
Recency of last visit (months)		Mean=6.04 SD=1.67	Projected Mean=1.50	−4.54	n/a2
Frequency of visits to this physician in last 12 months (%)	1	24.1	7.9	−16.2	n/a2
	2	23.5	15.4	−8.1	n/a2
	3	17.2	16.9	−0.3	n/a2
	4	14.1	18.4	4.3	n/a2
	5-9	15.1	29.7	14.6	n/a2
	10+	6.0	11.7	5.7	n/a2
Age (%)	18-24	1.6	1.5	−0.1	0.005
	25-34	7.5	7.5	−0.1	0.57
	35-44	10.1	10.0	−0.1	0.40
	45-54	14.6	14.3	−0.2	0.06
	55-64	21.1	20.6	−0.5	0.001
	65-74	22.1	21.7	−0.5	0.001
	75 or older	23.0	24.4	1.4	<0.001
Education (%)	8th grade or less	1.9	2.2	0.3	<0.001
	Some high school	2.5	2.8	0.3	<0.001
	High school	10.6	11.5	0.9	<0.001
	Some college	26.5	27.3	0.8	<0.001
	Bachelor’s degree (BA)	22.6	22.2	−0.4	0.002
	Higher than BA	35.8	34.0	−1.8	<0.001
Self-rated general health (%)	Excellent	13.7	12.2	−1.5	<0.001
	Very good	30.6	28.9	−1.7	<0.001
	Good	32.0	32.9	0.9	<0.001
	Fair	18.2	19.8	1.6	<0.001
	Poor	5.5	6.2	0.6	<0.001
Self-rated mental health (%)	Excellent	31.7	30.2	−1.5	<0.001
	Very good	32.4	31.8	−0.6	<0.001
	Good	24.3	25.4	1.2	<0.001
	Fair	9.8	10.6	0.8	<0.001
	Poor	1.8	1.9	0.1	0.02
Male (%)		39.6	38.7	−0.8	<0.001
Race/ethnicity (%)	Hispanic	10.4	11.2	0.8	<0.001
	White	68.3	66.6	−1.6	<0.001
	Black	5.6	6.1	0.5	<0.001
	Asian/Pacific Islander	11.6	11.8	0.2	0.09
	American Indian/Alaska Native	0.2	0.2	0.0	0.35
	Other/multiracial	3.9	4.0	0.2	0.02

¹p-value for test of association between characteristic and visit frequency weights

²Statistically significant by definition

Table 3 presents coefficients from adjusted models predicting each CAHPS measure from visit recency and frequency. A 1-month increase in time from last visit to survey was associated with a less positive CAHPS score for all 5 measures, with differences of −0.3 percentage points for each measure. Higher visit frequency in the past 12 months was associated with more positive CAHPS scores for all measures, with monotonically increasing coefficients for 2, 3, 4, 5-9, and 10+ visits compared to the 1-visit reference group. For example, the average adjusted difference in CAHPS scores between patients with 10 or more visits in the last 12 months and patients with only one visit in the last 12 months ranges from 6.7 (doctor communication) to 14.4 percentage points (access to care), p<0.001 for all.

Table 3:

Parameter estimates from models predicting CAHPS measures from visit recency and frequency, adjusted for case-mix and physician

	Recency of last visit (months)			Frequency of visits (comparison group is 1 visit)
				2 Visits			3 Visits			4 Visits			5-9 Visits			10+ Visits
	β	SE	p-value	β	SE	p-value	β	SE	p-value	β	SE	p-value	β	SE	p-value	β	SE	p-value
Global rating of doctor	−0.33	0.07	<0.001	3.99	0.29	<0.001	4.94	0.32	<0.001	6.64	0.34	<0.001	7.31	0.34	<0.001	9.38	0.49	<0.001
Patient experience composites
Doctor communication	−0.31	0.07	<0.001	3.14	0.30	<0.001	3.67	0.33	<0.001	4.74	0.35	<0.001	4.93	0.35	<0.001	6.67	0.50	<0.001
Access to care	−0.27	0.13	0.03	1.36	0.53	0.01	2.77	0.58	<0.001	4.68	0.62	<0.001	8.43	0.62	<0.001	14.35	0.88	<0.001
Coordination of care	−0.28	0.10	0.004	3.38	0.40	<0.001	3.97	0.44	<0.001	5.81	0.47	<0.001	7.01	0.47	<0.001	10.60	0.67	<0.001
Office staff	−0.28	0.08	<0.001	1.77	0.34	<0.001	2.42	0.37	<0.001	3.52	0.40	<0.001	4.88	0.39	<0.001	6.72	0.56	<0.001

Note: Each model predicts the CAHPS measure from visit recency, visit frequency, year, patient gender, case-mix adjustors (patient age, education, and self-rated general and mental health), and physician effects

Table 4 shows mean scores for the 5 CAHPS measures by sampling method using standard case-mix adjustment. Estimates for continuous sampling were significantly higher than those for 12-month periodic sampling for all 5 CAHPS measures, with mean differences of 2.1 to 2.9 percentage points (p<0.001 for all). Differences of 1 percentage point are considered small in the CAHPS context, and differences of 3 percentage points are considered moderate (16, 17).

Table 4:

Case-mix adjusted CG-CAHPS scores for continuous sampling relative to 12-month periodic sampling

		Recency of last visit
	N	12-month Periodic Sample (Unweighted) [Sample A]	Simulated Continuous Sample, Weighted by Visit Frequency, Projected to Mean 1.5 Months Recency [Sample B]1	Adjusted difference [Sample B - Sample A]	95% Confidence Interval [Sample B – Sample A]	p-value2	Physician- level SD3	Difference [Sample B – Sample A] in physician-level SDs	Percentile misranking Sample B, Sample A when true score is near 50th percentile
Global rating of doctor	28,799	90.17	92.88	2.70	(2.46, 2.94)	<0.001	6.04	0.45	17%
Patient experience composites
Doctor communication	29,002	91.10	93.32	2.22	(1.92, 2.52)	<0.001	5.96	0.37	15%
Access to care	28,770	63.10	65.97	2.88	(2.54, 3.21)	<0.001	12.72	0.23	9%
Coordination of care	28,747	83.23	85.74	2.51	(2.21, 2.81)	<0.001	7.35	0.34	13%
Office staff	28,831	86.13	88.26	2.13	(1.83, 2.44)	<0.001	5.41	0.39	15%

Note: Below, “residuals” refers to residuals from a model that predicts the CAHPS measure from year, patient gender, case-mix adjustors (patient age, education, and self-rated general and mental health), and physician effects

¹Grand mean plus visit-frequency-weighted mean of residuals, plus regression-based projection to a mean visit recency of 1.5 months, controlling for visit frequency and case-mix

²p-value for test of association between residuals and visit frequency weights, adjusted for additional uncertainty from regression-based projection to mean visit recency of 1.5 months

³Standard deviation (SD) of physician-level case-mix adjusted scores, not adjusted for visit recency or frequency

These differences can also be characterized as effect sizes on the scale of physician-level standard deviations. Differences in CAHPS scores between the continuous and 12-month periodic samples corresponded to effect sizes of 0.23 to 0.39 for the 4 composites, and 0.45 for the global doctor rating. Cohen (18) characterizes an effect size of 0.2 as small and an effect size of 0.5 as medium.

If physicians evaluated using continuous samples were compared without adjustment to physicians evaluated using 12-month periodic sampling, a physician whose true score was at the median (50^th percentile) of the distribution would be mis-ranked by 9 to 17 percentile points using continuous sampling.

Discussion

Healthcare organizations have several survey administration choices when collecting CG-CAHPS patient experience data: sampling strategy (periodic or continuous), reference period (6 months, 12 months, single visit, or hybrid), and survey mode (telephone, mail, web, etc.). These choices are made based on the organization’s desired frequency of data collection for quality improvement, benchmarking, trending, public reporting and other reasons such as cost of data collection. This flexibility allows users to take advantage of the relative benefits of the available survey options. It is important to understand whether these choices affect the resulting CAHPS scores.

Our results regarding differences in patient characteristics and case-mix adjusted CAHPS scores across sampling methods have direct implications for use of CAHPS surveys. Compared to 12-month periodic sampling, we estimated that patients selected via continuous sampling would have notably more frequent visits than those in a 12-month periodic sample, and somewhat more positive CAHPS scores (small to medium effect sizes, 0.23 to 0.45 at the physician level).

Visit recency and frequency were associated with substantial differences in CAHPS measures at the patient level. Longer time since the most recent visit was associated with less positive CAHPS scores, while more frequent visits were associated with more positive CAHPS scores. Consistent with our findings, past studies have documented that patient reports and ratings of care are associated with the length of time since the last visit: survey administration modes or sampling designs that increase the proximity of the patient response to the patient visit produce more favorable assessments (19–24). We conducted analyses of a dataset of 7,093 randomly selected adult enrollees (mean age=51; 65% female) in a group practice association located on the west coast (25) and found consistently positive associations between a more recent last visit and patient evaluations of care. For example, those with a most recent visit within the last month compared to more than 12 months ago reported more positive experiences with the doctor, wait time, and access to care ranging from 0.2 to 0.4 of a standard deviation at the patient level.

Although we find that more recent visits are associated with more positive survey responses, it is unclear whether they represent more accurate assessments, and findings are mixed, with some studies suggesting equivalent accuracy, and others suggesting better accuracy for more recent encounters (26, 27). One’s recollections of the visit may be more accurate for a more recent encounter, but the clarity and usefulness of provider communication in addressing the patient’s concerns or ongoing health conditions may only become fully apparent weeks after the encounter. There is some evidence that the correlation of visit proximity to survey administration with reports about care may be due to patient expectations (28, 29). One study (24) found that the presence of unmet expectations was associated with less positive perceptions of care. At 2 weeks and 3 months post treatment, experiencing symptomatic improvement and improved function were associated with more positive perceptions of care. It may be that the longer the time between being surveyed and the last visit, the more likely a patient is to have an unmet need, thus explaining the less favorable experiences reported if the last visit was longer ago.

Our study findings regarding the association between visit frequency and patient experience may reflect endogeneity. Case-mix adjustment corrects for differences between physicians in the characteristics of their patient populations that are not under physician control, but which may influence patient experience scores, such as age and education (13, 30). Adjustment for endogenous characteristics that may reflect quality of care, such as utilization (31), is not recommended. In this context, positive patient experiences might encourage return visits and negative ones might discourage return visits, encourage switching doctors or gaining a second opinion.

Our findings should be considered considering study limitations. First, the report items in the CG-CAHPS 3.0 survey have a 4-point response scale but the items in this study were administered using a 6-point response scale. But the pattern of our findings should still apply to CG-CAHPS data measured on a 4-point scale. Second, although we believe the relationships of patient experience reports with visit recency and frequency are unlikely to have changed rapidly over time, there may be value in replicating our analyses with more recent data. Third, while participating patients were randomly selected to participate in the CG-CAHPS survey, their willingness to participate involves self-selection that can confound the relationship between survey scores, recency, and frequency of visit. Fourth, encounters might differ across physicians or practices as a function of unmeasured confounders that could explain some of the observed differences. While the third and fourth limitations could affect the interpretation of the role of recency, it does not bias comparisons of the effects of different sampling approaches. Fifth, because recency in our data ranges from 2.6 to 12 months, our projection to a mean visit recency of 1.5 months for the simulated continuous sample extrapolates beyond the observed range of the data. However, we adjusted the standard errors of test statistics to account for the additional uncertainty due to this estimation. Finally, the simulation of continuous sampling relies on self-reported visit frequency and exact visit numbers were not available for those with more than four visits. This truncation may result in a small underrepresentation of the highest-utilization patients relative to true continuous sampling. Setting a ceiling on the number of times a patient can be sampled in twelve months would have an effect in the opposite direction, more so as the fraction of patients sampled increases. As such the estimates regarding continuous sampling should be viewed as approximations.

In summary, because continuous sampling methods produce samples with more recent and more frequent visits on average than annual sampling methods, they may result in more positive scores than periodic sampling under standard CG CAHPS case-mix adjustment. For high-stakes comparisons across providers or trending over time, patient pools should ideally be similar in terms of sampling methods, or should adjust for differences in approaches used by different providers or at different times (13).