Incentives for Caution: The In-Center Hemodialysis Consumer Assessment of Healthcare Providers and Systems Survey and Experience of Care

Dialysis treatment was heroic in the 1950s and pioneering in the 1960s; treatment for chronic kidney failure was available only to a select few deemed young, socially valuable, and otherwise healthy enough to justify the enormous investment (1). Public funding for ESRD came in the 1970s and with it, the expectation that the public’s money would be spent well. Measures of dialysis quality have evolved from case review in the 1970s and 1980s to comparison of process metrics, including measures of dialysis dose, the hemoglobin measure du jour, vascular access, mineral bone disorder treatment, and outcome metrics, including standardized mortality, hospitalization, and transplant ratios. Next is patient experience of care.

When Congress funded chronic dialysis in 1972, it was anticipated that this Medicare expansion might presage a national health insurance program. It was not to be. Instead, the 1980s and 1990s saw indemnity insurance give way to competition among prepaid group health insurance plans. Restriction of patient choice became a prominent theme of these plans’ cost containment efforts. The corporate, union, and government officials who purchased health insurance for their employees, members, and citizens needed to show that they were getting good value. Beneficiary questionnaires regarding the quality of care proliferated, and in response, in 1995, the Agency for Health Care Policy and Research (now the Agency for Healthcare Research and Quality [AHRQ]) sponsored the Consumer Assessment of Health Plans Study (CAHPS). The CAHPS was born.

The CAHPS began as an effort to develop a tool for comparing health plans, but it has evolved to become a family of surveys assessing patients’ experiences of their care not only by health plans but at hospitals and in physician offices, nursing facilities, and now, hemodialysis facilities. The brand remains the same, but CAHPS now denotes Consumer Assessment of Healthcare Providers and Systems. With academic and private sector vendor support, these surveys are developed and maintained by AHRQ and implemented by the Centers for Medicare and Medicaid Services (CMS). It should be no surprise that, after more than 15 years of investment in the CAHPS surveys by the Department of Health and Human Services, the CMS would use the In-Center Hemodialysis (ICH) CAHPS Survey (ICH-CAHPS or CAHPS-ICH) to satisfy the legislative requirement that “the ESRD QIP [Quality Incentive Program] shall include, to the extent feasible, a measure … of patient satisfaction as the Secretary shall specify. Information on patient experience with care at a facility is an important quality indicator to help providers/facilities improve services to their patients and to assist patients in choosing a provider/facility” (2). The CMS has recently announced that dialysis facilities’ ICH-CAHPS scores will be publicly reported on the Dialysis Facility Compare website (3). Nephrologists and dialysis providers should understand that the ICH-CAHPS is here to stay.

Work on the ICH-CAHPS began in 2002, and a 2006 report to the CMS by investigators from the American Institutes for Research, the RAND Corporation, Harvard Medical School, Westat, and ESRD Network 15 describing a quality improvement initiative using the ICH-CAHPS data has been intermittently available on the internet (4). Remarkably, however, no description of the instrument’s performance or its psychometric qualities has appeared; the work reported by Wood et al. (5) in this issue of CJASN represents the first peer-reviewed study of ICH-CAHPS. Wood et al. (5) sent the survey to consecutive hemodialysis patients referred to them by collaborating nephrologists, who reported clinical information and characteristics of the structure and process of care at the facilities at which the patients were treated. As the authors note, this population is a highly selected one (5); it would be unwise to interpret the results as the definitive ICH-CAHPS benchmarks. However, these potential biases do not necessarily impeach the study’s internal validity. The ICH-CAHPS comprises 58 items, of which 32 items yield three multi-item scales or composite measures: nephrologists’ communication and caring (NCC), quality of dialysis center care and operations (QoC), and providing information to patients (PI); 3 of 58 items are single-item global ratings of kidney doctor, dialysis center staff, and dialysis center.

Wood et al. (5) find most individual items contributing to NCC and QoC to correlate well (coefficient>0.4) with the total scale score (good convergence) and not to correlate too highly with another scale (good discrimination). Cronbach α-value is a measure of internal consistency: whether respondents, indeed, give similar and consistent answers to a set of questions designed to elicit similar information (6,7). It was 0.81 for NCC and 0.9 for QoC; a value of >0.7 is considered satisfactory for group-level comparisons. These scales also met or came close to meeting confirmatory factor analysis standards for comparative fit index and root-mean-square error of approximation. These measures of model fit indicate how well the observed data correlate with the hypotheses about the expected results used to formulate the scales. NCC and QoC thus pass psychometric muster. The results for PI, however, clearly indicate its failure. The individual items correlate poorly with the PI scale, with coefficients of 0.08–0.39, and α is 0.55. Comparative fit index for PI is too low, and root-mean-square error of approximation is too high. On the basis of the only information available at this time, provided by this study, the PI scale is not psychometrically valid. Unless and until future studies rehabilitate it, the usefulness of PI as a standard for dialysis care is questionable.

Many responses were clustered at the highest possible values on the scales and global ratings (the ceiling), but few were at the lowest level (the floor), meaning that these measurements will be less helpful in distinguishing excellent dialysis care from good care than in differentiating not-so-great care from very bad care. Therefore, the ICH-CAHPS may not be as effective at incentivizing excellence as it is at punishing poor performance. One of the most interesting findings is that intraclass correlation, determined by ANOVA, showed that the one parameter that differentiated dialysis facilities from one another most clearly was patients’ global assessment of the dialysis facility: “Using any number from 0 to 10, where 0 is the worst dialysis center possible and 10 is the best dialysis center possible, what number would you use to rate this dialysis center?” This single item had an intraclass correlation coefficient of 0.31, whereas intraclass correlation coefficients for the other global items and scales were 0.13–0.19. One interpretation of this finding is that a single global and quite subjective patient evaluation distinguishes among facilities better than the QoC scale, which sums patient observations about 17 different aspects of what happens at the center. The other interpretation is that, even for the best ICH-CAHPS measure, less than one third of score variability is attributable to true differences between facilities, and for the others, one tenth to one fifth.

What can one say about the construct validity of the findings reflected in the correlations between dialysis center characteristics and the ICH-CAHPS scores? If the physician reported taking care of fewer hemodialysis patients, the patient assigned the dialysis center staff and the center itself higher global scores, but neither the nephrologist communication scale nor the global kidney doctor rating varied. At facilities in which patients were reported to wait longer, NCC and QoC were rated lower, but the global rating of the dialysis staff, who are usually responsible for the daily operation of dialysis facilities, did not vary. Do these relationships make sense?

This analysis gives an important first look at ICH-CAHPS psychometrics, but to understand the whole story, we must return to the realm of policy and economics. In the early 1980s, the ESRD composite rate and monthly capitation payment represented Medicare’s first forays into prospective payment, and in 2013, dialysis facilities received the first payments under a QIP. It seems quite likely that dialysis facilities will be the first Medicare providers to which reimbursement will be determined, in part, by reports of patient experience of care. It will be important to minimize unintended consequences. Incentives to facilities receiving high patient ratings are certainly commendable. However, the finding that African Americans gave lower ratings on the global dialysis center rating (5) may be significant. If we penalize resource-poor facilities struggling to treat minority populations, the attempt to hear the patient’s voice could paradoxically set off a vicious cycle of underfunding–dissatisfaction–worsening underfunding. It is possible that the CMS’s plan to adjust reported results for patient mix (3) can mitigate this risk, but it will be important that value-based purchasing not exacerbate health care disparities. At this point, on the basis of very limited data, it is not clear which ICH-CAHPS scores would be an appropriate basis for a QIP standard or how such a standard should be implemented. It is time for caution.

Disclosures

None.