Including Disenrollees in CAHPS^® Managed Care Health Plan Assessment Reporting

Abstract

This study addresses the question whether basing plan quality ratings on the data of both enrollees and disenrollees provides more accurate reporting. We test whether including disenrollee data in calculating the CAHPS^® scores, reported in health plan assessments, will result in lower scores indicating lower plan quality. Adding disenrollees results in statistically significant score decreases. Factors likely to result in larger score changes when disenrollees are included are explored. The study concludes that though final determinations should rest on more extensive data, that there is evidence to suggest that including disenrollee feedback improves accuracy.

Introduction

Measuring quality of care from the perspective of patients and health plan beneficiaries is an important component of overall quality assessment (Hurtado, Swift, and Corrigan, 2001). This measurement has been mandated, and use of the reports is becoming more widespread among both beneficiaries, or their counselors, who wish to evaluate or compare health plans available to them, as well as by health plans themselves as they seek to improve quality of care. Initially, the CAHPS^® measures were only calculated on a sample of enrollees from each plan. Concern arose that evaluating health plans by surveying only their enrolled membership would not be accurate, because the members who are most dissatisfied with the plan have disenrolled and are no longer eligible for the survey (U.S. General Accounting Office, 1997).

The present process of disenrollment includes the following steps:

• If disenrolling to another managed care plan, the beneficiary need only enroll in the other plan and disenrollment in their former plan will take place automatically.

• If disenrolling to original Medicare, the beneficiary need only notify the plan of their desire to leave the plan and on being disenrolled by the plan, the beneficiary is automatically put into original Medicare.

If “lock-in” is eventually put into place, the period of time when beneficiaries can disenroll will be curtailed to a once-a-year specified period.

Not all who are dissatisfied with their plan disenroll. Dissatisfied Medicare beneficiaries may choose to stay enrolled for a number of reasons including the desire to retain coverage of key benefits especially prescription drugs, interest in maintaining continuity with existing providers, lack of knowledge about other available insurance options, or having limited financial flexibility to change to a more expensive plan.

Now that disenrollees have been included in the annual surveys, it is of interest to see whether these concerns are indeed realized. Disenrollees are hypothesized to provide lower ratings of plan quality, thus including recently disenrolled beneficiaries in the sample to obtain ratings should lower health plan ratings, and in the process make them more accurate by not excluding a portion of the plans' membership population. In this article, we report on the results of a study to determine the validity of this hypothesis in the Medicare managed care (MMC) population. Although our findings are limited to Medicare beneficiaries, they have important implications for the assessment of Medicaid and privately funded health plans as well, because the number of persons covered, the technical problems, and the cost of surveying disenrollees in private and Medicaid plans are even greater than they are for Medicare.

Background

Since 1998, Medicare has sponsored annual surveys of beneficiaries enrolled at least 6 months in their MMC plans to assess the performance of those plans from the beneficiary's perspective. The MMC CAHPS^® produces average ratings of both the health care received and of the managed care plan overall. It also produces a frequency distribution of scores on each of five multi-item composites for each plan. The two overall rating scores and the five composite distributions are presented to the public on CMS' Medicare Compare Web site (http://www.medicare.gov), to assist beneficiaries and their advisors in choosing among Medicare+Choice (M+C) options. More detailed tabulations are given to each Medicare health plan in data books to assist in performance monitoring and improvement activities.

In the few years since their development, CAHPS^® measures have come to play an important place in health plan assessment (McIntyre, Rogers, and Heier, 2001). They have been used to assess MMC plan quality (Goldstein et al., 2001), to associate plan characteristics with higher quality (Landon et al., 2001), and by purchasers of health plans to monitor performance and to aid their beneficiaries and employees in decisionmaking (Ginsberg and Sheridan, 2001; Zema and Rogers, 2001). That CAHPS^® is an important set of measures in this arena is further evidenced by the extent to which it has been refined to level the playing field among plans (Zaslavsky et al., 2001), boosting the legitimacy of the reported measures, as well as its use in research in the area of health plan performance measures (Lied and Sheingold, 2001).

The question of whether to continue including disenrollees in calculating the reported measures involves disenrollment rates, as plans with higher rates feel more impact of disenrollee assessment on their overall CAHPS^® assessment scores. While disenrollment rates have been consulted by many in attempting to measure dissatisfaction with health plans and overall quality (U.S. General Accounting Office, 1997; Riley, Ingber, and Tudor, 1997), and this use has been disputed (Schlesinger, Druss, and Thomas, 1999), no one to date has discussed the angle of this study examining whether the inclusion of disenrollees in health plan assessment contributes to more accurate scores.¹

As with any publicly distributed performance assessment, inaccurate or erroneous scores could be devastating to the quality of service received by an individual consumer and to the reputation and financial success of providers. Pertaining to this study, if certain plans received higher quality ratings than they actually should if the full true population (enrollees and disenrollees) were used for the assessment, beneficiaries could choose such plans over others that may, in fact, give them better quality of care. Moreover, if reporting is not accurate, plans will not get the guidance they need to improve their care, resulting in poorer care for many of their members. In fact, if a particular area of care or service was plagued by such a problem that everyone who encountered it disenrolled, then the problem would not be brought to the attention of plan administrators with only enrollees involved in performance measurement, since the members remaining in the plan had not encountered that problem and could not, therefore, provide the needed feedback. In addition, if readers of the reports come to doubt or discount the validity of the results, it may lead to a backlash in which consumers ignore important information about plan quality and make decisions exclusive of such information, again perhaps leading to less than optimal health care. Thus, credibility of methods and accuracy of results are essential to the acceptance of the Medicare CAHPS^® scores. The unresolved question about the potential impact of disenrollee responses on CAHPS^® scores led Medicare and AHRQ to develop an instrument, survey methodology, and reporting strategy for a survey of Medicare health maintenance organization (HMO) disenrollees. The Balanced Budget Act of 1997 mandated the development of a survey for disenrollees that required annual reporting of disenrollment rates for each M+C plan to the public. Raw, total disenrollment rates are determined from administrative data, but the total disenrollment rate can be misleading because it does not explain why beneficiaries left the plan. Many leave for reasons unrelated to performance, for example, due to a change in residence, a plan leaving their area, or their desired health care provider contracted with a different plan. Thus, the design project developed questions about the reasons for disenrollment, as well as the CAHPS^® performance assessment questions modified for disenrollees (Guess et al., 2000).

Methods

Instrument and Item Content

As the purpose of the disenrollee assessment survey is to duplicate as closely as possible the managed care enrollee instrument, we only changed question wording or reference periods where absolutely necessary for the questionnaire intelligibility for the intended respondents. Thus, we started with the 1998 MMC instrument. Questions were reworded to ask about beneficiaries' experience in the past tense where necessary. But this was the only wording change from the survey used with the enrollees. The reference period for questions remained the same.

Because a single instrument encompassing both the assessment and reasons questions was envisioned for the field test, an instrument using the full set of enrollee assessment questions would have been too long. Thus, the disenrollee field test examined the two enrollee global rating items and only two of the five enrollee composites. The two global rating items ask for the respondent's overall assessment of the health care he or she received through the health plan and of the plan's overall performance. Both of these are rated on an 11-point scale, 0-10. The composites are summary statistics representing two or more questions that measure the same dimensions of health care or health insurance plan services. Each composite developed for reporting CAHPS^® survey items measures one of five health care experience dimensions: (1) getting the care they need; (2) getting care quickly; (3) how well their doctors communicate; (4) courtesy, respect, and helpfulness of office staff; and (5) health plan customer service, information, and paperwork. The disenrollee field test only included the questions needed to form the “Getting Care Needed” and “Doctor Communicates Well” composites—two of the more important composites in the evaluation of quality of health care.

Sample Design and Data Collection

The field test samples were drawn from the beneficiaries of four Medicare HMOs. The four plans are a convenience sample for the purposes of the field test of the survey under development, but they represented markets with multiple health plans on both the west and east coast. Two plans were based in California and two in Florida. (Demographic information on the four plans is available from the lead author.) While some minor differences exist among the plans, by and large, the disenrollee samples do not differ dramatically from enrollee samples. Importantly, distributions of self-reported health status do not differ significantly among the four plans, so no plan had a population with greater health risks.

Within each plan, the populations from which the sample was drawn included beneficiaries who had been in their plan continuously for at least 6 months ending in December 1998 and who had subsequently disenrolled between January and March 1999. The data were collected over a 7-week period from May through early July 1999. Respondents who had been in a plan for less than 6 months—classified as rapid disenrollees—were excluded from the sample so that the disenrollees matched the enrollees. Although disenrollees are a small proportion of the enrollees for a single period, the field test samples were chosen to be large enough to produce acceptably precise estimates for disenrollees alone, so that we could study the properties of CAHPS^® assessments by disenrollees. We used the enrollee data from the same four Medicare HMOs in the 1998 Managed Care Medicare Satisfaction Survey, collected over a 13-week period ending in January 1999.

Analytic Approach

Our analysis compared the enrollee-only data to what we call the “pooled sample” including the enrollee and disenrollee data together. We chose this approach because Medicare intended to report CAHPS^® assessments from disenrollees only as part of a pooled estimate for enrollees and disenrollees combined. The objective in obtaining CAHPS^® assessments from disenrollees is to reduce the favorable bias hypothesized to result from excluding them. Thus, comparing enrollees with the pooled, enrollee and disenrollee, population is the appropriate test of whether including disenrollee survey results would noticeably change plan quality reports. Two analytic approaches were undertaken to examine how inclusion of disenrollees might affect reporting.

Significance Test

The first is simply a statistical test to see theoretically whether the mean ratings or scores of interest differ in the two groups, and estimate the size of that difference. We hypothesize that the inclusion of disenrollees will lower the mean ratings and composite scores. Though the dependency between the two groups—enrollees are included in both groups—prohibits the use of a regular t-test, it is still possible to assess differences using a modified statistical test. A discussion of the adjusted F-test that was used for these tests is found in Technical Note A. SUDAAN^® software (Shah, Barnwell, and Bieler, 1997) was used to account for the design and sampling non-response weights in the F-test.

To form both the composites, since person-level CAHPS^® composite scores are not defined, we took the individual mean of the non-missing data for the items defining each of the composites (Technical Note B). We assigned integer values of 1-3 to the big problem, small problem, not a problem responses; and values of 1-4 to the never, sometimes, usually, and always responses.

CAHPS^® Macro Analysis and Output

Our second complementary analysis then was to undertake separately, in both groups, the usual rating and composite score analysis using the CAHPS^® macro, version 2.0 (U.S. Department of Health and Human Service, 1999). The CAHPS^® macro is a statistical software program designed to analyze multi-item composites and single items from CAHPS^® survey data, and includes the facility to handle disproportionate selection and the clustering inherent in our sampling.

The output from the CAHPS^® macro program includes two different summaries, commonly referred to as the “Bar Chart,” and the “Star Chart.” The Bar Chart provides purely descriptive output, presenting absolute levels of plan performance. It reports, for each rating item or composite, the respondent percentages responding within each of the three standardized response category groups. The Star Chart indicates relative performance identifying health plans whose mean performance is statistically higher or lower than the overall performance of all health plans, using a significance test.

Computing the mean plan composite scores in the comparative analysis is not simply a matter of finding the weighted mean person-level composite score because person-level scores are not defined. If at least one of the questionnaire items has been answered by a respondent, that person's data is used in a calculation developed to compute a mean score at the plan level, for each questionnaire item, using all the usable responses collected. Equally spaced integers, starting with one, are then applied to the various response categories, and the proportions developed are multiplied by these integers to get the mean. The overall composite score is then an equally weighted mean of the plan-level mean scores.

The inclusion of disenrollees is expected to shift the response distributions toward the lower response group, and probably not change the relative standing of plans or the findings of significant differences, since they are a relative measure.

We weighted the data up to population totals for this study because we oversampled disenrollees relative to enrollees to obtain greater precision in mean rating and composite estimates. We used population counts for our four HMOs from the time the 1998 Managed Care Medicare Satisfaction Survey samples were drawn, creating a pooled population ex post facto, and computing weights using the population and sample counts. However, as previously described, the CAHPS^® bar graphs produced by the standard CAHPS^® 2.0 analytic software do not use weights.

Results

Significance Tests

Table 1 presents the descriptive and other statistical test results for our comparison of the mean enrollee rating or composite with the mean rating or composite of the pooled sample containing both enrollees and disenrollees. The statistical comparisons controlled for plan, that is, we simultaneously tested the enrollee-disenrollee comparison within each of the four plans. The statistical tests reported were a contrast, comparing the mean value for the ratings or scores for the pooled sample with that of the enrollees only within each group.

Table 1. Comparison of Enrollees with Pooled Samples for the Two Ratings and Two Composites Within Each of the Four Plans: 1998.

Composite/Rating	Sample Size Weighted		Weighted Mean Ratings		Enrollees-Pooled Mean		Percent Enrollee Weighted
	Pooled	Enrollees	Pooled	Enrollees	Difference	Standard Error
Health Care Rating1
All Plans	487,830	462,463	8.61	8.70	0.08	0.0025	0.95
Plan 1	23,843	22,715	8.56	8.62	0.06	0.0097	0.94
Plan 2	260,972	241,261	8.53	8.65	0.12	0.0081	0.92
Plan 3	47,909	45,170	8.47	8.54	0.08	0.0085	0.94
Plan 4	155,105	153,317	8.81	8.84	0.03	0.0054	0.99
Health Plan Rating2
All Plans	642,111	615,838	8.58	8.66	0.08	0.0019	0.96
Plan 1	30,103	28,952	8.87	8.94	0.06	0.0075	0.96
Plan 2	337,510	317,302	8.34	8.44	0.11	0.0075	0.94
Plan 3	62,656	59,641	8.55	8.65	0.09	0.0064	0.95
Plan 4	211,842	209,943	8.92	8.95	0.02	0.0042	0.99
Getting Needed Care Composite3
All Plans	588,624	560,274	2.75	2.82	0.07	0.0004	0.95
Plan 1	28,445	27,181	2.76	2.83	0.07	0.0017	0.96
Plan 2	316,076	294,211	2.70	2.80	0.10	0.0011	0.93
Plan 3	57,433	54,231	2.70	2.78	0.09	0.0015	0.94
Plan 4	186,671	184,651	2.83	2.85	0.02	0.0008	0.99
Doctor Communication Composite4
All Plans	494,866	470,718	3.54	3.58	0.04	0.0007	0.95
Plan 1	24,343	23,254	3.50	3.53	0.03	0.0029	0.96
Plan 2	264,391	245,839	3.52	3.57	0.06	0.0028	0.93
Plan 3	48,585	45,846	3.47	3.51	0.04	0.0025	0.94
Plan 4	157,547	155,779	3.60	3.61	0.01	0.0018	0.99

¹Wald F (df: 4, 4,432) = 57.61, p<0.0001.

²Wald F (df: 4, 4,432) = 93.85, p<0.0001.

³Wald F (df: 4, 4,432) = 2418.17, p<0.0001.

⁴Wald F (df: 4, 4,432) = 153.46, p<0.0001.

SOURCE: Consumer Assessment of Health Plan Study (CAHPS^®) Medicare Disenrollee Field-test survey conducted in 1998 by Research Triangle Institute for the Agency for Healthcare Research and Quality and the Centers for Medicare & Medicaid Services.

Table 1 presents the comparison of the mean enrollee rating with the mean rating of the pooled sample for the two ratings and two composites. A statistically significant difference between the two groups for the rating of health care was found (Wald F statistic (df; 4, 4,432) = 57.61, p<0.0001). Individual t tests of the comparison within each plan, for this rating as well as the health plan rating and two composites, were all significant at p<0.0001. The mean differences between the pooled sample and enrollees-only ranges from 0.03 to 0.12, and the standard errors of those mean differences range from 0.0025 to 0.0097. With the health plan rating, again there is a statistically significant difference between the mean ratings of the two groups (Wald F statistic (df: 4, 4,432) = 93.85, p< 0.0001). The mean differences only range from 0.02 to 0.11, and the standard errors of those mean differences range from 0.0042 to 0.0075.

For the Getting Care Needed composite there is a statistically significant difference between the mean ratings of the pooled sample and the enrollees only (Wald F statistic (df: 4, 4,432) = 2,418.17, p< 0.0001). The average differences between the two groups ranges from 0.02 to 0.10, and the standard errors of those mean differences range from 0.0008 to 0.0017. For the Doctor Communication composite there is also a statistically significant difference between the pooled and enrollees= mean ratings (Wald F statistic (df: 4, 4,432) = 153.46, p<0.0001). The mean differences range from 0.01 to 0.06, and the standard errors of those mean differences range from 0.0018 to 0.0029.

Though significant differences were found in both ratings and the two composites, the mean differences are fairly small. For each of the ratings the difference is generally about 0.07+0.05. For the Getting Care Needed composite the difference is about 0.08+0.02, while for the Doctor Communication composite it is about 0.04+0.03.

CAHPS^® Macro Runs

Tables 2 and 3 present the CAHPS^® macro Bar and Star Chart output for the two ratings items and two composites. The tables show sample size information, descriptive breakdown of the responses, mean ratings, differences of each plan from the overall mean, standard errors, and rating indicating the significance test outcomes for the plan differences. In examining the tables, it is apparent that the distribution of responses differs in our two groups in only negligible ways. Generally a smaller percentage of the pooled sample, than the enrollees-only, chooses the highest rating category (and often the middle category), and a larger percentage of the pooled sample chooses the lowest rating category—all consistent with generally lower ratings among disenrollees. The unadjusted means are all smaller in the pooled samples. With few exceptions, plan differences from the overall mean tend to be a bit more extreme in the pooled samples. The standard errors of the difference either remain the same or drop in the larger pooled sample. The ratings remained the same in every case except one, in the Getting Needed Care composite for Plan 2. So in general the results followed our expectations.

Table 2. CAHPS^® Macro Output Descriptive Results (Bar Chart): 1998.

Composite/Rating	Group	Sample Size Analyzed	Bars
			Percentage Answering
			0-7	8-9	10
Health Care
Plan 1	Enrollees	295	19.7	35.3	45.1
	Pooled	495	20.2	35.1	44.7
Plan 2	Enrollees	1,212	18.0	34.7	47.4
	Pooled	1,331	19.3	34.4	46.4
Plan 3	Enrollees	334	22.5	31.1	46.1
	Pooled	612	23.3	31.1	45.6
Plan 4	Enrollees	685	14.2	41.5	44.4
	Pooled	863	14.5	41.4	44.2
Overall	Enrollees	2,526	—	—	—
	Pooled	3,301	—	—	—
Health Plan
Plan 1	Enrollees	376	14.9	29.8	55.3
	Pooled	580	15.7	29.9	54.5
Plan 2	Enrollees	1,594	21.5	34.3	44.2
	Pooled	1,716	22.7	34.4	42.9
Plan 3	Enrollees	441	19.5	37.0	43.5
	Pooled	747	20.9	36.6	42.5
Plan 4	Enrollees	938	13.0	36.7	50.3
	Pooled	1,127	13.3	36.6	50.0
Overall	Enrollees	3,349	—	—	—
	Pooled	4,170	—	—	—
Getting Care Needed
Plan 1	Enrollees	353	3.5	12.3	84.2
	Pooled	577	5.4	12.5	82.2
Plan 2	Enrollees	1,463	6.3	11.4	82.3
	Pooled	1,595	9.0	12.0	79.0
Plan 3	Enrollees	397	6.6	12.9	80.5
	Pooled	721	9.0	13.2	77.9
Plan 4	Enrollees	819	3.5	10.3	86.2
	Pooled	1,020	4.0	10.4	85.6
Overall	Enrollees	3,032	—	—	—
	Pooled	3,913	—	—	—
Doctor Communication
Plan 1	Enrollees	301	9.5	27.1	63.4
	Pooled	492	9.8	27.8	62.3
Plan 2	Enrollees	1,228	9.1	23.6	67.2
	Pooled	1,339	9.9	24.8	65.2
Plan 3	Enrollees	338	9.6	28.7	61.7
	Pooled	616	10.3	29.5	60.2
Plan 4	Enrollees	694	5.4	28.3	66.2
	Pooled	869	5.7	28.5	65.9
Overall	Enrollees	2,561	—	—	—
	Pooled	3,316	—	—	—

NOTE: CAHPS^® is Consumer Assessment of Health Plan Study.

SOURCE: Consumer Assessment of Health Plan Study (CAHPS^®) Medicare Disenrollee Field-test survey conducted in 1998 by Research Triangle Institute for the Agency for Healthcare Research and Quality and the Centers for Medicare & Medicaid Services.

Table 3. CAHPS^® Macro Output Significance Test Results (Star Chart): 1998.

Composite/Rating	Group	Unadjusted Means****	Stars
			Plan Difference from Overall Mean	Standard Error of Difference	Star Rating
Health Care
Plan 1	Enrollees	8.624	-0.039	0.084	**
	Pooled	8.588	-0.029	0.082	**
Plan 2	Enrollees	8.645	-0.017	0.057	**
	Pooled	8.570	-0.046	0.055	**
Plan 3	Enrollees	8.545	-0.117	0.084	**
	Pooled	8.490	-0.126	0.081	**
Plan 4	Enrollees	8.835	0.173	0.058	***
	Pooled	8.816	0.200	0.057	***
Overall	Enrollees	8.662	—	—	—
	Pooled	8.616	—	—	—
Health Plan
Plan 1	Enrollees	8.939	0.195	0.069	***
	Pooled	8.890	0.205	0.067	***
Plan 2	Enrollees	8.444	-0.300	0.051	*
	Pooled	8.358	-0.328	0.050	*
Plan 3	Enrollees	8.646	-0.098	0.068	**
	Pooled	8.567	-0.118	0.066	**
Plan 4	Enrollees	8.946	0.202	0.048	***
	Pooled	8.926	0.241	0.048	***
Overall	Enrollees	8.744	—	—	—
	Pooled	8.685	—	—	—
Getting Care Needed
Plan 1	Enrollees	2.807	0.023	0.017	**
	Pooled	2.768	0.025	0.016	**
Plan 2	Enrollees	2.761	-0.023	0.013	**
	Pooled	2.701	-0.043	0.012	*
Plan 3	Enrollees	2.740	-0.044	0.020	*
	Pooled	2.689	-0.055	0.019	*
Plan 4	Enrollees	2.827	0.043	0.013	***
	Pooled	2.816	0.073	0.013	***
Overall	Enrollees	2.783	—	—	—
	Pooled	2.743	—	—	—
Doctor Communication
Plan 1	Enrollees	3.525	-0.026	0.028	**
	Pooled	3.509	-0.022	0.027	**
Plan 2	Enrollees	3.567	0.016	0.018	**
	Pooled	3.535	0.004	0.018	**
Plan 3	Enrollees	3.506	-0.045	0.027	**
	Pooled	3.479	-0.051	0.026	**
Plan 4	Enrollees	3.606	0.055	0.019	***
	Pooled	3.598	0.068	0.018	***
Overall	Enrollees	3.551	—	—	—
	Pooled	3.530	—	—	—

^*Plan mean is significanty lower than overall mean.

^**Plan mean does not differ significantly from overall mean.

^***Plan mean is significantly higher than overall mean.

^****No case mix.

NOTE: CAHPS^® is Consumer Assessment of Health Plan Study.

SOURCE: Consumer Assessment of Health Plan Study (CAHPS^®) Medicare Disenrollee Field-test survey conducted in 1998 by Research Triangle Institute for the Agency for Healthcare Research and Quality and the Centers for Medicare & Medicaid Services.

Discussion

We undertook the comparison of the enrollee mean ratings and composites to the mean values of a pooled sample of enrollees and disenrollees in this analysis to inform the question of whether it is necessary to include disenrollee plan satisfaction data along with enrollee data in CAHPS^® plan reporting. The question is to what degree would reporting based on enrollee data alone be biased upward, because it does not include disenrollees who would almost certainly give lower mean ratings, since many may have left the plan due to dissatisfaction. Reporting the plan satisfaction for plans with large dissatisfied disenrollee populations, exclusive of the disaffected population's data, may mislead consumers into thinking a plan better satisfies its members than it actually does. The very possibility of such a scenario could lead the users of CAHPS^® reports to be concerned with the validity of scores that are based only on enrollee data.

For these four plans, comparing Table 2 and 3 results from a sample including disenrollees, to that excluding them, seems to indicate that the inclusion of disenrollees does not make a large difference. The percentages in Table 2 do not show sizable differences, and it is hard to imagine that these differences, generally less than 2 percentage points, would amount to something meaningful in the minds of beneficiaries examining them. However, it would be hard to come to a clean-cut decision regarding our study hypothesis using the descriptive chart because the nature of this display does not suggest a simple criteria for identifying when a meaningful difference is occurring.

Including disenrollees only changed significance test results for a single plan on a single composite: Plan 3 for the Getting Needed Care composite. This particular mean difference was not the largest mean difference between the two groups across all the ratings and composites we examined for these four plans. However, it did happen to have unusually low variability, a very large sample, and a fairly large percentage of disenrollees.

The nearly identical results for the two groups in the significance tests summarized in Table 3 stems from the relative nature of those tests. Each plan mean is compared with an overall mean of all the included plans. When disenrollees are included, the overall mean drops along with the individual plan means, this statistic of relative standing does not alter much. As long as the relative drop in mean ratings is roughly the same in all four plans, the significance tests will not change. If a particular plan has both disenrollee assessments that differ dramatically from the enrollees, and the proportion of disenrollees is large enough, it could change the significance tests.

Though the hypothesis in question would suggest a significance test comparing the enrollees with the pooled (enrollee and disenrollee) sample, such a test would violate assumptions of independence between the two groups being compared. However, derivations show (Technical Note A) that the significance test required would be equivalent to a t-test between enrollees and disenrollees.

Our analysis revealed significant differences between these two groups for both the health care and health plan ratings and for the Getting Needed Care and Doctor Communication composites. The fact that we found significant differences should not be a surprise. We know that in the aggregate the disenrollee ratings of health plan and health care tend to be lower than those of enrollees, and, given the samples sizes we had to work with, there was ample power to detect differences. However, as a result of this power, the tests identified actual mean differences between the enrollee and the pooled samples that are not generally of great practical significance.

The results of the significance test alone would still seem to suggest that disenrollee data be included in published CAHPS^® reports, as significant differences were found between the enrollees and the pooled group of enrollees and disenrollees for all four of the mean ratings and composites tested, though differences were small. However, there are a couple of caveats that should be considered when applying these results to the question at hand:

• The circumstances under which the enrollees and disenrollees completed these surveys differed. The data from the enrollees came from a satisfaction survey that asked about experience with the plan, while that of the disenrollees came from a disenrollment survey that focused on reasons for leaving the plan. The nature of the latter survey focuses respondents on the negative aspects of their plan experience, possibly putting them in a negative frame of mind. Comparing the ratings of disenrollees in a negative frame of mind with those of enrollees in a more neutral frame of mind is not likely to be a very fair comparison. The differences we saw may shrink if disenrollees were asked to assess their health care and health plan in a more neutral (i.e., less negative) frame of mind.

• The four plans selected for this analysis do not constitute a random sample of plans from across the Nation, nor is four a very big sample size to extrapolate from. The average mean difference in the ratings that we report may not be representative of what would be found across all the plans or even for a representative sample. Stronger arguments could be made with a larger and more representative sample of the plans.

If the CAHPS^® reports do not indicate that large differences exist, can it be concluded that including disenrollees is not necessary? Or do the caveats about the data available for this study, just reviewed, force us to look for other data? We asked what the factors are that are involved in these statistics, and how much they would need to change to affect the results.

There are two factors that determine how much change will be seen in the reported CAHPS^® scores when disenrollees are included: (1) the actual mean difference in rating between disenrollees and enrollees, and (2) the relative number of disenrollees vis-a-vis the enrollees (the disenrollment rate). Thus, including disenrollees in the data is most likely to make a difference if the disenrollees have a substantially lower mean rating, and if their proportion of the population is substantial compared with the enrollees. The fact that the size of the disenrollee population is determined by the sampling window over time,² while the size of the enrollee population stays relatively stable, means some serious consideration should be given to the way the disenrollee population is defined for sampling. The relative proportion of disenrollees can be arbitrarily set to a wide range of proportions by adjusting the period of time over which disenrollees are sampled.

To see what kind of change would be needed to change one of our results, we ran a small simulation adjusting up the percent of disenrollees for Plan 3, and examined the results for the two ratings items. We found that increasing the percentage of disenrollees changed the results of the significance tests of plan differences. The percent of disenrollees for Plan 3 for the health care rating in our data was 5.7 percent. We randomly re-sampled, without replacement, the enrollees and disenrollees to obtain a new sample with 8.55 percent disenrollees. When the CAHPS^® macro was run on that simulated data, Plan 3's mean health care rating became significantly different (lower) than the overall mean. Similarly, with the health plan rating where in our data Plan 3 had 4.8 percent disenrollees. We created a simulation data set with 7.2 percent disenrollees and got the same results: the non-significant difference from the overall mean became significant.

These simulated rates are not at all outside the range of disenrollment rates one might see across plans nationally. Indeed rates in this range and higher would be commonly encountered. Nationally for 2000, CMS reports 12 percent of plan members disenrolled, excluding plan withdrawals. Among the reported individual plan disenrollment rates in 2000, many plans have rates in the teens and higher. And as a point of comparison, to see that these rates are not outside the historical norm, testimony to the U.S. Senate reported disenrollment rates in HMOs in the Los Angeles and Miami areas ranged from 4-42 percent in 1995 (U.S. General Accounting Office, 1997). So the disenrollment rates in our simulation are not larger than we would expect to find for a good number of plans in a national sample. If anything they are a little bit on the low side. This information combined with our simulation results make a good case for including disenrollees despite finding only small differences in the four plans examined here, as they may not be representative of the true range of plans nationally. Larger and more significant changes in scoring may be found if disenrollees were excluded from plan assessment in many plans nationally.

In light of these findings, to include disenrollee data, would seem prudent to us. It may not always make a difference, but in those cases where it would make a difference (plans with large numbers of unsatisfied disenrollees) it is vital for consumers to be provided with unbiased assessments. If reporters of CAHPS^® assessment items wish to exclude disenrollees, this should be justified by a similar study with a larger and randomly selected national sample of health plans to better determine the impact of their inclusion or exclusion on reporting. We believe there is sufficient evidence to suggest that including disenrollees could impact reported CAHPS^® ratings and scores.

Technical Note A

The statistical test that we require, a test between dependent samples, is:

$$\mathrm{\Delta }={\mathrm{\mu }}_{\text{enrollees}}-{\mathrm{\mu }}_{\text{enrollees}}{+}_{\text{disenrollees}}={\mathrm{\mu }}_{\mathrm{e}}-{\mathrm{\mu }}_{\mathrm{e}+\mathrm{d}}$$ (1)

The mean of the combined sample, μ_e+d, can be written, using λ as the proportion of enrollees, as:

$${\mathrm{\mu }}_{\mathrm{e}+\mathrm{d}}=\mathrm{\mu }({\mathrm{\mu }}_{\mathrm{e}})+(1-\mathrm{\lambda }){\mathrm{\mu }}_{\mathrm{d}},$$ (2)

Substituting (2) into (1) and rearranging terms we arrive at this newly written difference:

$$\mathrm{\Delta }=(1-\mathrm{\lambda })({\mathrm{\mu }}_{\mathrm{e}}-{\mathrm{\mu }}_{\mathrm{d}}).$$ (3)

Assuming a constant λ and equal variances in both populations (σ²_p), the variance of Δ is

$$\text{SD}(\mathrm{\Delta })={(\text{Var}(\mathrm{\Delta }))}^{.5}={[{(1-\mathrm{\lambda })}^2({{\mathrm{\sigma }}^2}_{\mathrm{p}}/{\mathrm{n}}_{\mathrm{e}}+{{\mathrm{\sigma }}^2}_{\mathrm{p}}/{\mathrm{n}}_{\mathrm{d}})]}^{.5}.$$ (4)

These population values can be estimated by sample values leading to a t-test:

$$t={\mathrm{D}}_{\text{est}}/\text{SD}({\mathrm{D}}_{\text{est}})=[(1-{\mathrm{p}}_{\mathrm{e}})({\overline{\mathrm{x}}}_{\mathrm{e}}-{\overline{\mathrm{x}}}_{\mathrm{d}})]/{[{(1-{\mathrm{p}}_{\mathrm{e}})}^2({{\mathrm{s}}^2}_{\mathrm{p}}/\text{ne}+{{\mathrm{s}}^2}_{\mathrm{p}}/{\mathrm{n}}_{\mathrm{d}})]}^{.5}=({\overline{\mathrm{x}}}_{\mathrm{e}}-{\overline{\mathrm{x}}}_{\mathrm{d}})/{({{\mathrm{s}}^2}_{\mathrm{p}}/{\mathrm{n}}_{\mathrm{e}}+{{\mathrm{s}}^2}_{\mathrm{p}}/{\mathrm{n}}_{\mathrm{d}})}^{.5}$$ (5)

where p_e is the proportion of enrollees, estimating λ.

This same reasoning can be applied to any additive contrast to derive an F statistic, such as the comparison of enrollees and pooled enrollees-plus-disenrollees within each of the four plans (equation 6). Such a contrast between dependent samples can be validly tested in SAS^® or SUDAAN^® by a test between the two independent samples, with the reported differences and standard errors adjusted using the sample estimate of the proportion of enrollees in the pooled sample (p_e). Due to the weighting and sampling design we used SUDAAN^® to run an analysis of variance, with a specified contrast between the enrollees and disenrollees in each plan.

The model tested in Tables 2 and 3 was an ANOVA with a nested design, group (enrollee versus disenrollee) within plan. We used the SUDAAN^® procedure Proc Regress to test, using a Wald F statistic, the contrast of interest was:

$$\mathrm{y}=({\mathrm{\mu }}_{\text{Plan}1\text{enrollees}}-{\mathrm{\mu }}_{\text{Plan}1\text{disenrollees}})+({\mathrm{\mu }}_{\text{Plan}2\text{enrollees}}-{\mathrm{\mu }}_{\text{Plan}2\text{disenrollees}})+({\mathrm{\mu }}_{\text{Plan}3\text{enrollees}}-{\mathrm{\mu }}_{\text{Plan}3\text{disenrollees}})+({\mathrm{\mu }}_{\text{Plan}4\text{enrollees}}-{\mathrm{\mu }}_{\text{Plan}4\text{disenrollees}})$$ (6)

Technical Note B

Questionnaire Items Used in Forming CAHPS^® Composites

Items included in the Getting Care Needed Composite:

• Q6—With the choices your health plan gives (gave) you, how much of a problem, if any, was it to get a personal doctor or nurse you are (were) happy with?

• Q10—In the last 12 (6) months, how much of a problem, if any, was it to get a referral to a specialist that you needed to see?

• Q22—In the last 12 (6) months, how much of a problem, if any, was it to get the care you or your doctor believed necessary?

• Q23—In the last 12 (6) months, how much of a problem, if any, were delays in health care while you waited for approval from your health plan?

Items included in the Doctor Communication Composite:

• Q27—In the last 12 (6) months, how often did doctors or other health providers listen carefully to you?

• Q28—In the last 6 months, how often did doctors or other health providers explain things in a way you could understand?

• Q29—In the last 12 (6) months, how often did doctors or other health providers show respect for what you had to say?

• Q30—In the last 12 (6) months, how often did doctors or their health providers spend enough time with you?