Abstract
Background and objectives
The US Centers for Medicare and Medicaid Services (CMS) End Stage Renal Disease Prospective Payment System and Quality Incentive Program requires that dialysis centers meet predefined criteria for quality of patient care to ensure future funding. The CMS selected the Consumer Assessment of Healthcare Providers and Systems In-Center Hemodialysis (CAHPS-ICH) survey for the assessment of patient experience of care. This analysis evaluated the psychometric properties of the CAHPS-ICH survey in a sample of hemodialysis patients.
Design, setting, participants, & measurements
Data were drawn from the Adelphi CKD Disease Specific Program (a retrospective, cross-sectional survey of nephrologists and patients). Selected United States–based nephrologists treating patients receiving hemodialysis completed patient record forms and provided information on their dialysis center. Patients (n=404) completed the CAHPS-ICH survey (comprising 58 questions) providing six scores for the assessment of patient experience of care. CAHPS-ICH item-scale convergence, discrimination, and reliability were evaluated for multi-item scales. Floor and ceiling effects were estimated for all six scores. Patient (demographics, dialysis history, vascular access method) and facility characteristics (size, ratio of patients-to-physicians, nurses, and technicians) associated with the CAHPS-ICH scores were also evaluated.
Results
Item-scale correlations and internal consistency reliability estimates provided support for the nephrologists’ communication (range, 0.16–0.71; α=0.81) and quality of care (range, 0.16–0.76; α=0.90) composites. However, the patient information composite had low internal consistency reliability (α=0.55). Provider-to-patient ratios (range, 2.37 for facilities with >36 patients per physician to 2.8 for those with <8 patients per physician) and time spent in the waiting room (3.44 for >15 minutes of waiting time to 3.75 for 5 to <10 minutes) were characteristics most consistently related to patients’ perceptions of dialysis care.
Conclusions
CAHPS-ICH is a potentially valuable and informative tool for the evaluation of patients’ experiences with dialysis care. Additional studies are needed to estimate clinically meaningful differences between care providers.
Keywords: hemodialysis, quality of care, CAHPS-ICH, reimbursement, patient experience of care
Introduction
As standards in healthcare administration and delivery shift to value-based purchasing of health care services, evidence-based measures of performance are required to drive high standards and improve the quality of care. The outcomes of such measures are increasingly used to determine future reimbursement payments. They also provide information to help care providers and clinicians identify specific areas in which care experiences could be improved.
The US Centers for Medicare and Medicaid Services (CMS) recently implemented an End Stage Renal Disease Prospective Payment System (ESRD PPS) for Medicare reimbursement of outpatient dialysis-related services along with a Quality Incentive Program (QIP) to monitor and ensure dialysis centers maintain quality patient care (1). There are currently two types of QIP metrics: clinical measures and reporting measures. The clinical measures have specific outcome standards that dialysis centers must meet, whereas the reporting measures require only collection of specific data. Failure to achieve these standards results in reductions to dialysis payments for future payment years.
The CMS selected the Consumer Assessment of Healthcare Providers and Systems (CAHPS) In-Center Hemodialysis (CAHPS-ICH) survey for assessing patient experience of care (2). The CAHPS-ICH development team created the survey by conducting a literature review, examining 13 existing surveys, convening a technical expert panel, soliciting public comment, conducting patient focus groups, and performing 71 cognitive interviews with draft items. The CAHPS-ICH survey was field tested with 1454 respondents (46% response rate) using a mixed-mode data collection effort (mail with phone follow-up); however, the results have not yet been published. Patient experience of care was included as a QIP reporting metric for 2014 (1). Although the CMS makes this recommendation, it also notes that the cost burden may be quite high based on the volume of patients treated. The CMS estimates a $10 million annual burden to the system (3).
This study evaluates the psychometric properties of the CAHPS-ICH survey in a sample of hemodialysis patients.
Materials and Methods
Data Source
Data were drawn from the Adelphi CKD Disease Specific Program (DSP), a cross-sectional survey of real-world clinical practice conducted in the United States in the first quarter of 2012 (4). The objective of the DSP is to provide a holistic understanding of the management of patients with CKD, including their treatment environment.
The DSP is a patient record-based survey of nephrologists in the United States and their patients with CKD. Nephrologists, identified from public lists of healthcare professionals, who manage at least 20 patients with CKD per month on dialysis were eligible to participate in the research and were drawn from different types of centers across the United States. Nephrologists were asked to provide information on their dialysis center and complete a patient record form for their next 10 CKD dialysis patients seen. Information captured included patient demographics, history and monitoring, symptoms, concomitant conditions, treatments, management, and hospitalization details.
The same patients were invited to fill out a patient self-completed questionnaire that included consultation history, symptoms, and the CAHPS-ICH (Supplemental Material).
The research was performed in full accordance with the US Health Insurance Portability and Accountability Act of 1996 (5). Each patient provided informed consent for the deidentified and aggregated reporting of research findings. All data were fully deidentified and aggregated before analysis. Full details of the methodology were previously published (4).
CAHPS-ICH Scoring System
The CAHPS-ICH survey has 58 items (questions), including 3 that ask for a global rating of the patients’ kidney doctor, the dialysis center staff, and the dialysis center itself. Patients are asked to rate each using a 0–10 scale that is subsequently collapsed to a 1–3 scale, with a score of 1 comprising responses of 0–6 on the original scale, a score of 2 comprising responses 7 and 8, and a score of 3 comprising responses 9 and 10. Three multi-item scales assess the following: nephrologists’ communication and caring (NCC scale; 6 questions with a final score between 1 and 4), quality of dialysis center care and operations (QoC scale; 17 questions with a final score between 1 and 4), and providing information to patients (PI scale; 9 questions with a final score between 0 and 1) (6). All scale scores were calculated according to the CAHPS-ICH scoring methodology (6).
Statistical Analyses
The three multi-item scales were evaluated using several different psychometric techniques (7). In step 1, item convergence (or the correlation of each item with the scale score), correcting for item overlap, was estimated to ensure that each item was assigned to the appropriate scale. A cut-off of ≥0.4 was used to indicate a large enough correlation (8).
In step 2, item discrimination was evaluated by comparing correlations of an item (correcting for item overlap) with its hypothesized scale, against its correlations with the other two scales, to ensure each item in a scale does not correlate as highly or more highly with other scales.
In step 3, the internal consistency reliability of each scale was also evaluated to provide a single summary measure of the extent to which the items had acceptable reliability. A threshold of ≥0.7 was used to indicate an acceptable level of reliability (7).
In step 4, floor and ceiling effects were estimated to determine whether the scales and global rating items distinguished adequately between respondents. All other factors being equal, measures are better when there are few patients scoring at the top (ceiling effect) and bottom (floor effect) of the scales.
Finally, in step 5, three separate confirmatory factor analysis (CFA) models were estimated to evaluate the overall fit of the hypothesized three scales to the data. Separate models were estimated to help identify any potential issues or problems with the multi-item scales. For each CFA model, two goodness-of-fit statistics were reported: the comparative fit index (CFI; 0–1) and the root mean square error of approximation (RMSEA; 0–1), in which a higher CFI and a lower RMSEA indicate better fit of the hypothesized model to the observed data. CFI values ≥0.95 and RMSEA values ≤0.06 are desirable (9).
In addition, to assess the extent to which patients within the same clinic share similar patient experiences and how well the CAHPS scales and global rating items distinguish between clinics, intraclass correlation coefficients (ICCs) and reliabilities at the clinic level were estimated using one-way ANOVA models.
Finally, F tests were performed to assess whether significant differences exist between the mean global ratings and composite scales across subgroups of patients defined by, for example, demographic characteristics and disease status. A Bonferroni adjusted P value of <0.0005 was used as the threshold for statistical significance to account for the multiple tests being performed.
Results
Demographics
Physicians and patients at 76 dialysis centers across all regions of the United States were included in this study. A total of 819 patients receiving dialysis were recruited. Of these, 404 patients were eligible for analysis based on having received hemodialysis for >3 months and having completed the CAHPS-ICH survey. Table 1 provides an overview of the characteristics of the recruited sample (n=819), and those included (n=404) and not included (n=415) in the analysis. Chi-squared tests and Fisher’s exact tests were performed and there were no statistically significant differences between the two groups. The mean (± SD) age of eligible patients was 57±14.7 years and the median time on dialysis was 48 months (interquartile range, 24–28 months).
Table 1.
Characteristics of the sample comparing patients included and excluded from the analysis
| Demographic | Recruited Sample (n=819) | No PSC Returned or Did Not Meet Inclusion Criteria (n=415) | Eligible Patients (n=404) | P Value |
|---|---|---|---|---|
| Age (yr) | ||||
| 0–19 | 1 (0.1) | 1 (0.2) | 0 (0.0) | 0.81 |
| 20–44 | 162 (19.8) | 77 (18.6) | 85 (21.0) | |
| 45–64 | 392 (47.9) | 206 (49.6) | 186 (46.0) | |
| 65–74 | 154 (18.8) | 73 (17.6) | 81 (20.0) | |
| ≥75 | 110 (13.4) | 58 (14.0) | 52 (12.9) | |
| Sex | ||||
| Male | 452 (55.3) | 225 (54.3) | 227 (56.2) | 0.62 |
| Female | 366 (44.7) | 189 (45.7) | 177 (43.8) | |
| Ethnicity | ||||
| White/Caucasian | 340 (41.6) | 181 (43.6) | 159 (39.6) | 0.19 |
| Black/African American | 302 (37.0) | 150 (36.1) | 152 (37.8) | |
| Hispanic/Latino | 110 (13.5) | 47 (11.3) | 63 (15.7) | |
| Other (including unknown) | 65 (8.0) | 37 (8.9) | 28 (7.0) | |
| Employment status | ||||
| Employed | 174 (22.3) | 88 (22.1) | 86 (22.6) | 0.20 |
| Unemployed | 328 (42.1) | 164 (41.2) | 164 (43.0) | |
| Retired | 255 (32.7) | 139 (34.9) | 116 (30.4) | |
| Other | 22 (2.8) | 7 (1.8) | 15 (3.9) | |
| BMI (kg/m2) | ||||
| <20 | 19 (2.3) | 7 (1.7) | 12 (3.0) | 0.42 |
| 20–24.9 | 187 (22.8) | 93 (22.4) | 94 (23.3) | |
| 25–29.9 | 336 (41.0) | 187 (45.1) | 149 (36.9) | |
| ≥30 | 235 (28.7) | 110 (26.5) | 125 (30.9) | |
| Not recorded | 42 (5.1) | 18 (4.3) | 24 (5.9) | |
| Time since diagnosis (yr) | ||||
| <2 | 132 (20.2) | 81 (24.0) | 51 (16.2) | 0.07 |
| 2 to <4 | 163 (25.0) | 80 (23.7) | 83 (26.4) | |
| 4 to <6 | 142 (21.8) | 72 (21.3) | 70 (22.3) | |
| ≥6 | 215 (33.0) | 105 (31.1) | 110 (35.0) |
Data are presented as n (%). BMI, body mass index; PSC, patient self-completion form.
Psychometric Properties of the CAHPS-ICH
NCC Scale.
Five of the six NCC items correlated strongly with the scale score, and all items had a stronger correlation with the NCC scale than with the other two scales (Table 2). Internal consistency reliability (coefficient α) was 0.81. The CFA model demonstrated a good fit (CFI, 0.96; RMSEA, 0.10), although the factor loading (the correlation of the item to the hypothesized composite) for item Q9 (physician fully informed) was only 0.17 (Table 3). Factor loadings for the other five items in the composite were higher, ranging from 0.68 to 0.83.
Table 2.
Item-scale correlations and internal consistency reliability of the CAHPS-ICH composites
| Item | Scale | ||
|---|---|---|---|
| Nephrologists' Communication | Quality of Care | Patient Information | |
| Q3 (physician listens carefully) | 0.68a | 0.23 | 0.09 |
| Q4 (physician ease of understanding) | 0.60a | 0.30 | 0.24 |
| Q5 (physician respect) | 0.67a | 0.24 | 0.05 |
| Q6 (physician time) | 0.70a | 0.35 | 0.23 |
| Q7 (physician feeling of care) | 0.71a | 0.29 | 0.20 |
| Q9 (physician fully informed) | 0.16a | 0.09 | 0.07 |
| Q10 (dialysis staff listen carefully) | 0.34 | 0.75a | 0.35 |
| Q11 (dialysis staff ease of understanding) | 0.31 | 0.69a | 0.31 |
| Q12 (dialysis staff respect) | 0.38 | 0.76a | 0.35 |
| Q13 (dialysis staff time) | 0.41 | 0.75a | 0.27 |
| Q14 (dialysis staff feeling of care) | 0.32 | 0.75a | 0.34 |
| Q15 (dialysis staff comfort) | 0.35 | 0.72a | 0.25 |
| Q16 (dialysis staff privacy of information) | −0.02 | 0.22a | 0.19 |
| Q17 (dialysis staff comfort) | 0.14 | 0.39a | 0.19 |
| Q21 (dialysis staff little pain with needles) | 0.09 | 0.46a | 0.14 |
| Q22 (dialysis staff closely monitor) | 0.30 | 0.75a | 0.26 |
| Q24 (dialysis staff manage problems) | 0.19 | 0.52a | 0.02 |
| Q25 (dialysis staff professional manner) | 0.19 | 0.65a | 0.24 |
| Q26 (dialysis staff dietary information) | 0.02 | 0.16a | 0.23b |
| Q27 (dialysis staff test explanations) | 0.24 | 0.46a | 0.36 |
| Q33 (center punctuality) | 0.16 | 0.51a | 0.21 |
| Q34 (center cleanliness) | 0.12 | 0.51a | 0.14 |
| Q43 (center problem handling) | −0.06 | 0.46a | −0.03 |
| Q19 (knowledge of access care) | 0.19b | 0.14b | 0.08a |
| Q28 (provided with written info about patient rights) | 0.05 | 0.21b | 0.18a |
| Q29 (dialysis staff review of patient rights) | 0.13 | 0.27 | 0.39a |
| Q30 (dialysis staff gave health problem advice) | 0.12 | 0.31 | 0.32a |
| Q31 (dialysis staff gave emergency procedure when on hemodialysis machine) | 0.09 | 0.09 | 0.15a |
| Q36 (treatment option information was received at amount desired) | 0.15 | 0.20 | 0.31a |
| Q38 (transplant eligibility discussion) | 0.02 | 0.17 | 0.38a |
| Q39 (peritoneal dialysis as an option discussion) | 0.09 | 0.16 | 0.24a |
| Q40 (treatment choice involvement was received at amount desired) | 0.02 | 0.21 | 0.23a |
| Cronbach's α coefficient | 0.81 | 0.90 | 0.55 |
Data are presented as correlation coefficients.
Indicates which items correspond to each scale.
Indicates that the item correlates better with another scale rather than its own.
Table 3.
Confirmatory factor analysis of the hypothesized three-composite structure
| Item | Standardized Factor Loadings | 95% Confidence Interval | P Value |
|---|---|---|---|
| Nephrologists’ communication (n=285)a | |||
| Q3 (listening carefully) | 0.73 | 0.67 to 0.78 | <0.0001 |
| Q4 (ease of understanding) | 0.68 | 0.61 to 0.74 | <0.0001 |
| Q5 (respect) | 0.75 | 0.70 to 0.81 | <0.0001 |
| Q6 (time) | 0.77 | 0.72 to 0.82 | <0.0001 |
| Q7 (feeling of care) | 0.83 | 0.79 to 0.87 | <0.0001 |
| Q9 (fully informed) | 0.17 | 0.07 to 0.28 | 0.0001 |
| Quality of care (n=339)b | |||
| Q10 (listening carefully) | 0.78 | 0.74 to 0.83 | <0.0001 |
| Q11 (ease of understanding) | 0.74 | 0.69 to 0.79 | <0.0001 |
| Q12 (respect) | 0.82 | 0.78 to 0.86 | <0.0001 |
| Q13 (time) | 0.82 | 0.78 to 0.86 | <0.0001 |
| Q14 (feeling of care) | 0.81 | 0.77 to 0.85 | <0.0001 |
| Q15 (dialysis comfort) | 0.79 | 0.75 to 0.83 | <0.0001 |
| Q16 (privacy of information) | 0.19 | 0.09 to 0.30 | <0.0001 |
| Q17 (comfort) | 0.42 | 0.33 to 0.51 | <0.0001 |
| Q21 (pain with needles) | 0.56 | 0.49 to 0.64 | <0.0001 |
| Q22 (closely monitor) | 0.79 | 0.74 to 0.83 | <0.0001 |
| Q24 (problems) | 0.81 | 0.77 to 0.85 | <0.0001 |
| Q25 (professional manner) | 0.65 | 0.59 to 0.72 | <0.0001 |
| Q26 (dietary information) | 0.17 | 0.06 to 0.28 | 0.002 |
| Q27 (explanations) | 0.46 | 0.37 to 0.54 | <0.0001 |
| Q33 (punctuality) | 0.52 | 0.44 to 0.60 | <0.0001 |
| Q34 (cleanliness) | 0.51 | 0.43 to 0.60 | <0.0001 |
| Q43 (problem solving) | 0.82 | 0.78 to 0.86 | <0.0001 |
| Patient information (n=353)c | |||
| Q19 (equipment) | 0.14 | 0.03 to 0.26 | 0.01 |
| Q28 (patient rights) | 0.26 | 0.15 to 0.37 | <0.0001 |
| Q29 (review of rights) | 0.45 | 0.35 to 0.56 | <0.0001 |
| Q30 (health problems) | 0.46 | 0.36 to 0.56 | <0.0001 |
| Q31 (emergency procedure) | 0.26 | 0.14 to 0.37 | <0.0001 |
| Q36 (treatment information) | 0.43 | 0.33 to 0.53 | <0.0001 |
| Q38 (transplant eligibility) | 0.85 | 0.76 to 0.93 | <0.0001 |
| Q39 (peritoneal dialysis) | 0.40 | 0.30 to 0.50 | <0.0001 |
| Q40 (treatment choices) | 0.34 | 0.23 to 0.45 | <0.0001 |
Likelihood ratio test of model versus saturated: chi square (9)=42.93, P>chi square<0.001.
Likelihood ratio test of model versus saturated: chi square (119)=354.81, P>chi square<0.001.
Likelihood ratio test of model versus saturated: chi square (27)=120.81, P>chi square<0.001.
QoC Scale.
Fourteen of the 17 QoC items correlated strongly with the scale score, with most items correlating more highly with QoC than the other two scales (Table 2). The coefficient α for the 17-item scale was 0.90. The CFA model did not quite demonstrate the desired fit (CFI, 0.92; RMSEA, 0.08). Factor loadings for items Q16 (dialysis staff privacy of information) and Q26 (dialysis staff dietary information), were only 0.19 and 0.17, respectively (Table 3), with the loadings for the other 15 items ranging from 0.42 to 0.82, thus indicating a stronger correlation with the hypothesized scale.
PI Scale.
All of the nine items in the PI scale correlated poorly with the scale score (Table 2). The coefficient α was only 0.55. The CFA model did not fit the data well (CFI, 0.75; RMSEA, 0.10) and the factor loadings ranged from 0.14 to 0.85 (Table 3).
Floor and Ceiling Effects
Floor effects (or lowest possible scores) were minimal, ranging from 0% to 8%, whereas ceiling effects (or highest possible scores) varied from 15% (QoC composite) to 75% (kidney doctor global rating). This suggests that the ICH measures may be limited in their ability to distinguish between patients experiencing the most positive levels of care.
Reliability at the Hemodialysis Center
The dialysis center item was the most reliable of the three global rating items and three multi-item scales (ICC, 0.31), indicating that within dialysis center scores are more consistent for this item than the other CAHPS-ICH scores. All other global ratings and composites had ICCs <0.2. The number of patient responses required for reliable results (defined as a reliability of 0.70) varied from 5 (dialysis center global rating) to 16 (NCC composite).
Associations of Patient and Dialysis Center Characteristics with CAHPS-ICH Scores
A higher ratio of patients per nurse was associated with lower PI scores (ranging from 0.84 for facilities with >12 patients per nurse to 0.91 for facilities with <5 patients per nurse; P=0.0001; Table 4), whereas a higher ratio of patients to physician was associated with worse global ratings of both the dialysis center (ranging from 2.37 for facilities with >36 patients per physician to 2.80 for facilities with <8 patients per physician; P=0.0001) and the dialysis center staff (ranging from 2.44 for facilities with >36 patients per physician to 2.78 for facilities with <8 patients per physician; P=0.0001). Longer waiting room times were associated with lower scores on the NCC composite (ranging from 3.68 for facilities with waiting times >15 minutes to 3.97 for facilities with waiting times of 5 to <10 minutes; P=0.0001) and the QoC composite (ranging from 3.44 for facilities with waiting times >15 minutes to 3.75 for facilities with waiting times of 5 to <10 minutes; P<0.001). This suggests that a more positive experience of care is demonstrated by patients treated in dialysis centers that are appropriately staffed.
Table 4.
Relationship between CAHPS-ICH scores and global rating items and key dialysis center characteristics
| Dialysis Center Characteristic | Composite Scores | Global Ratings | ||||
|---|---|---|---|---|---|---|
| Nephrologist Communication (1–4) | Quality of Care (1–4) | Patient Information (0–1) | Kidney Doctor (1–3) | Dialysis Center Staff (1–3) | Dialysis Center (1–3) | |
| Patients/technician (n) | ||||||
| <3 | 3.70 [0.05] | 3.54 [0.05] | 0.92 [0.03] | 2.76 [0.05] | 2.55 [0.08] | 2.53 [0.08] |
| 3 to <4 | 3.65 [0.04] | 3.43 [0.04] | 0.84 [0.02] | 2.67 [0.05] | 2.57 [0.06] | 2.56 [0.06] |
| 4 | 3.69 [0.03] | 3.48 [0.04] | 0.82 [0.02] | 2.71 [0.04] | 2.54 [0.05] | 2.60 [0.05] |
| >4 | 3.77 [0.04] | 3.40 [0.06] | 0.82 [0.04] | 2.81 [0.05] | 2.38 [0.08] | 2.45 [0.08] |
| Overall | 3.70 (P=0.27; n=374) | 3.46 (P=0.37; n=374) | 0.85 (P=0.04; n=372) | 2.74 (P=0.37; n=373) | 2.51 (P=0.43; n=370) | 2.54 (P=0.60; n=373) |
| Patients/nurse (n) | ||||||
| <5 | 3.69 [0.05] | 3.56 [0.04] | 0.91 [0.02] | 2.77 [0.05] | 2.66 [0.06] | 2.67 [0.06] |
| 5 to <8 | 3.69 [0.03] | 3.52 [0.04] | 0.84 [0.01] | 2.68 [0.04] | 2.58 [0.05] | 2.62 [0.05] |
| 8 to <12 | 3.68 [0.04] | 3.39 [0.05] | 0.76 [0.02] | 2.68 [0.05] | 2.37 [0.06] | 2.40 [0.06] |
| ≥12 | 3.71 [0.03] | 3.41 [0.04] | 0.84 [0.02] | 2.77 [0.04] | 2.51 [0.06] | 2.52 [0.06] |
| Overall | 3.69 (P=0.97; n=371) | 3.47 (P=0.04; n=371) | 0.84 (P=0.001; n=369)a | 2.72 (P=0.39; n=370) | 2.53 (P=0.03; n=367) | 2.55 (P=0.05; n=370) |
| Patients/physician (n) | ||||||
| <8 | 3.70 [0.04] | 3.56 [0.05] | 0.89 [0.01] | 2.81 [0.04] | 2.78 [0.05] | 2.80 [0.05] |
| 8 to <20 | 3.71 [0.04] | 3.46 [0.04] | 0.82 [0.02] | 2.70 [0.05] | 2.50 [0.06] | 2.59 [0.06] |
| 20 to <36 | 3.73 [0.03] | 3.42 [0.04] | 0.82 [0.02] | 2.74 [0.04] | 2.46 [0.06] | 2.47 [0.06] |
| ≥36 | 3.66 [0.04] | 3.45 [0.05] | 0.82 [0.02] | 2.73 [0.05] | 2.44 [0.06] | 2.37 [0.06] |
| Overall | 3.70 (P=0.69; n=348) | 3.47 (P=0.23; n=348) | 0.84 (P=0.001; n=346) | 2.74 (P=0.49; n=347) | 2.54 (P=0.001; n=344)a | 2.56 (P=0.001; n=348)a |
| Shifts at center (n) | ||||||
| 2 | 3.49 [0.06] | 3.46 [0.06] | 0.90 [0.02] | 2.41 [0.09] | 2.38 [0.10] | 2.37 [0.10] |
| 3 | 3.67 [0.04] | 3.49 [0.04] | 0.85 [0.01] | 2.71 [0.05] | 2.62 [0.05] | 2.65 [0.05] |
| 4 | 3.75 [0.04] | 3.47 [0.04] | 0.83 [0.02] | 2.81 [0.05] | 2.52 [0.06] | 2.53 [0.07] |
| ≥5 | 3.75 [0.05] | 3.34 [0.06] | 0.77 [0.02] | 2.75 [0.06] | 2.33 [0.09] | 2.37 [0.09] |
| Overall | 3.66 (P=0.02; n=377) | 3.44 (P=0.39; n=377) | 0.84 (P=0.01; n=375) | 2.67 (P=0.02; n=376) | 2.46 (P=0.03; n=373) | 2.48 (P=0.01; n=376) |
| Patient capacity (n) | ||||||
| <54 | 3.59 [0.04] | 3.45 [0.04] | 0.84 [0.02] | 2.65 [0.05] | 2.50 [0.06] | 2.55 [0.06] |
| 54 to <75 | 3.68 [0.04] | 3.56 [0.04] | 0.86 [0.02] | 2.78 [0.04] | 2.67 [0.05] | 2.67 [0.05] |
| 75 to <112 | 3.69 [0.03] | 3.40 [0.04] | 0.84 [0.02] | 2.66 [0.05] | 2.46 [0.05] | 2.45 [0.06] |
| ≥112 | 3.80 [0.03] | 3.46 [0.04] | 0.80 [0.02] | 2.77 [0.05] | 2.48 [0.07] | 2.56 [0.06] |
| Overall | 3.69 (P=0.01; n=377) | 3.47 (P=0.11; n=377) | 0.84 (P=0.23; n=375) | 2.72 (P=0.15; n=376) | 2.53 (P=0.03; n=373) | 2.56 (P=0.09; n=376) |
| Time spent in waiting room (min) | ||||||
| 5 to <10 | 3.97 [0.03] | 3.75 [0.04] | 0.90 [0.03] | 2.91 [0.07] | 2.56 [0.09] | 2.82 [0.08] |
| 10 to <15 | 3.69 [0.03] | 3.59 [0.03] | 0.81 [0.02] | 2.63 [0.05] | 2.54 [0.06] | 2.61 [0.06] |
| ≥15 | 3.68 [0.03] | 3.44 [0.03] | 0.84 [0.02] | 2.74 [0.04] | 2.52 [0.05] | 2.52 [0.05] |
| Overall | 3.78 (P=0.001; n=361)a | 3.59 (P=0.001; n=361)a | 0.85 (P=0.13; n=359) | 2.76 (P=0.03; n=360) | 2.54 (P=0.93; n=357) | 2.65 (P=0.03; n=361) |
| Time between meetings with a nephrologist (d) | ||||||
| 1 to <8 | 3.69 [0.05] | 3.46 [0.05] | 0.83 [0.05] | 2.70 [0.05] | 2.52 [0.14] | 2.56 [0.14] |
| 8 to <15 | 3.74 [0.05] | 3.52 [0.05] | 0.87 [0.05] | 2.85 [0.05] | 2.67 [0.15] | 2.67 [0.15] |
| ≥15 | 3.73 [0.05] | 3.60 [0.05] | 0.94 [0.05] | 2.47 [0.05] | 2.58 [0.26] | 2.52 [0.26] |
| Overall | 3.72 (P=0.80; n=368) | 3.53 (P=0.28; n=368) | 0.88 (P=0.46; n=366) | 2.68 (P=0.001; n=367)a | 2.59 (P=0.42; n=364) | 2.58 (P=0.60; n=367) |
Data are presented as adjusted mean scores, with the SEM of the difference [between adjusted mean score and overall score].
Indicates significant results.
Black/African-American patients reported worse global ratings of the dialysis center than white/Caucasian and Hispanic/Latino patients (ranging from 2.40 for African Americans versus 2.72 for whites/Caucasians and 2.68 for Hispanics/Latinos; P=0.0001 overall; Table 5). Finally, patients receiving anemia education reported more positive experiences regarding PI (ranging from 0.89 for educated patients versus 0.78 for patients not educated; P=0.0001).
Table 5.
Relationship between CAHPS-ICH scores and global rating items and key patient characteristics
| Patient Characteristic | Composite Scores | Global Ratings | ||||
|---|---|---|---|---|---|---|
| Nephrologist Communication (1–4) | Quality of Care (1–4) | Patient Information (0–1) | Kidney Doctor (1–3) | Dialysis Center Staff (1–3) | Dialysis Center (1–3) | |
| Age (yr) | ||||||
| <40 | 3.60 [0.05] | 3.43 [0.05] | 0.87 [0.02] | 2.75 [0.05] | 2.33 [0.08] | 2.49 [0.07] |
| 40 to <55 | 3.70 [0.04] | 3.44 [0.04] | 0.86 [0.02] | 2.77 [0.05] | 2.57 [0.06] | 2.54 [0.06] |
| 55 to <70 | 3.74 [0.03] | 3.49 [0.04] | 0.83 [0.02] | 2.70 [0.04] | 2.56 [0.05] | 2.60 [0.05] |
| ≥70 | 3.64 [0.05] | 3.49 [0.04] | 0.82 [0.02] | 2.67 [0.05] | 2.59 [0.06] | 2.58 [0.06] |
| Overall | 3.67 (P=0.13; n=384) | 3.46 (P=0.68; n=384) | 0.85 (P=0.19; n=382) | 2.72 (P=0.56; n=383) | 2.51 (P=0.13; n=380) | 2.55 (P=0.70; n=383) |
| Sex | ||||||
| Male | 3.69 [0.02] | 3.49 [0.02] | 0.85 [0.01] | 2.70 [0.03] | 2.54 [0.03] | 2.63 [0.03] |
| Female | 3.69 [0.02] | 3.43 [0.02] | 0.83 [0.01] | 2.75 [0.03] | 2.53 [0.03] | 2.48 [0.03] |
| Overall | 3.69 (P=0.87; n=379) | 3.46 (P=0.24; n=379) | 0.84 (P=0.52; n=377) | 2.72 (P=0.39; n=378) | 2.53 (P=0.91; n=375) | 2.55 (P=0.03; n=378) |
| BMI (kg/m2) | ||||||
| <20 | 3.59 [0.09] | 3.42 [0.12] | 0.88 [0.04] | 2.56 [0.11] | 2.42 [0.18] | 2.27 [0.17] |
| 20 to <25 | 3.66 [0.04] | 3.38 [0.06] | 0.86 [0.02] | 2.71 [0.06] | 2.43 [0.08] | 2.41 [0.08] |
| 25 to <30 | 3.70 [0.04] | 3.48 [0.05] | 0.85 [0.02] | 2.72 [0.05] | 2.57 [0.07] | 2.67 [0.07] |
| ≥30 | 3.71 [0.05] | 3.47 [0.05] | 0.84 [0.02] | 2.78 [0.05] | 2.55 [0.08] | 2.59 [0.07] |
| Overall | 3.66 (P=0.70; n=356) | 3.44 (P=0.41; n=356) | 0.86 (P=0.88; n=354) | 2.69 (P=0.47; n=355) | 2.49 (P=0.37; n=352) | 2.48 (P=0.01; n=355) |
| Ethnicity | ||||||
| White/Caucasian | 3.72 [0.04] | 3.53 [0.04] | 0.87 [0.02] | 2.76 [0.05] | 2.63 [0.05] | 2.72 [0.05] |
| African American | 3.63 [0.04] | 3.40 [0.04] | 0.82 [0.02] | 2.67 [0.05] | 2.41 [0.06] | 2.40 [0.06] |
| Hispanic/Latino | 3.77 [0.05] | 3.55 [0.06] | 0.82 [0.03] | 2.81 [0.06] | 2.66 [0.07] | 2.68 [0.07] |
| Other | 3.66 [0.06] | 3.27 [0.07] | 0.83 [0.03] | 2.55 [0.09] | 2.33 [0.09] | 2.30 [0.09] |
| Overall | 3.69 (P=0.17; n=377) | 3.44 (P=0.01; n=377) | 0.84 (P=0.10; n=375) | 2.70 (P=0.10; n=376) | 2.51 (P=0.003; n=373) | 2.53 (P=0.001; n=376)a |
| Employment status | ||||||
| Employed | 3.72 [0.03] | 3.43 [0.04] | 0.84 [0.02] | 2.76 [0.04] | 2.59 [0.06] | 2.65 [0.06] |
| Unemployed | 3.72 [0.04] | 3.54 [0.04] | 0.86 [0.02] | 2.73 [0.04] | 2.60 [0.05] | 2.61 [0.05] |
| Retired | 3.62 [0.04] | 3.42 [0.04] | 0.85 [0.02] | 2.65 [0.05] | 2.41 [0.06] | 2.46 [0.06] |
| Other | 3.74 [0.04] | 3.38 [0.08] | 0.78 [0.03] | 2.76 [0.06] | 2.54 [0.09] | 2.41 [0.08] |
| Overall | 3.70 (P=0.24; n=361) | 3.44 (P=0.12; n=361) | 0.83 (P=0.14; n=359) | 2.73 (P=0.44; n=360) | 2.53 (P=0.08; n=357) | 2.53 (P=0.07; n=360) |
| Time diagnosed with CKD (yr) | ||||||
| <2 | 3.61 [0.06] | 3.56 [0.05] | 0.87 [0.02] | 2.65 [0.07] | 2.61 [0.08] | 2.58 [0.08] |
| 2 to <4 | 3.64 [0.05] | 3.43 [0.05] | 0.84 [0.02] | 2.62 [0.06] | 2.50 [0.06] | 2.54 [0.07] |
| 4 to <6 | 3.69 [0.05] | 3.51 [0.05] | 0.86 [0.02] | 2.74 [0.06] | 2.58 [0.06] | 2.56 [0.06] |
| ≥6 | 3.71 [0.04] | 3.39 [0.04] | 0.82 [0.02] | 2.78 [0.05] | 2.42 [0.06] | 2.53 [0.06] |
| Overall | 3.66 (P=0.53; n=295) | 3.47 (P=0.09; n=295) | 0.85 (P=0.38; n=295) | 2.70 (P=0.23; n=294) | 2.53 (P=0.23; n=291) | 2.55 (P=0.97; n=294) |
| Dialysis history (yr) | ||||||
| <1 | 3.72 [0.04] | 3.57 [0.04] | 0.86 [0.02] | 2.70 [0.05] | 2.65 [0.05] | 2.66 [0.06] |
| 1 to <2 | 3.76 [0.04] | 3.56 [0.04] | 0.86 [0.02] | 2.78 [0.05] | 2.55 [0.06] | 2.60 [0.06] |
| 2 to <3 | 3.65 [0.04] | 3.43 [0.05] | 0.87 [0.02] | 2.73 [0.05] | 2.61 [0.06] | 2.54 [0.07] |
| ≥3 | 3.65 [0.04] | 3.42 [0.04] | 0.80 [0.02] | 2.73 [0.04] | 2.48 [0.05] | 2.53 [0.05] |
| Overall | 3.69 (P=0.19; n=355) | 3.49 (P=0.04; n=355) | 0.85 (P=0.03; n=355) | 2.74 (P=0.76; n=354) | 2.57 (P=0.21; n=351) | 2.58 (P=0.42; n=354) |
| Vascular access | ||||||
| IV catheter | 3.76 [0.04] | 3.58 [0.04] | 0.84 [0.02] | 2.72 [0.05] | 2.59 [0.06] | 2.63 [0.06] |
| AV fistula | 3.68 [0.03] | 3.45 [0.03] | 0.84 [0.01] | 2.73 [0.04] | 2.55 [0.05] | 2.56 [0.05] |
| Other | 3.65 [0.05] | 3.37 [0.05] | 0.84 [0.02] | 2.68 [0.05] | 2.40 [0.07] | 2.46 [0.07] |
| Overall | 3.70 (P=0.20; n=379) | 3.47 (P=0.03; n=379) | 0.84 (P=0.94; n=377) | 2.71(P=0.82; n=378) | 2.51 (P=0.23; n=375) | 2.55 (P=0.36; n=378) |
| Ever received anemia education | ||||||
| Yes | 3.74 [0.02] | 3.51 [0.03] | 0.89 [0.01] | 2.73 [0.03] | 2.60 [0.03] | 2.62 [0.03] |
| No | 3.65 [0.02] | 3.42 [0.03] | 0.78 [0.01] | 2.71 [0.03] | 2.46 [0.03] | 2.49 [0.03] |
| Overall | 3.69 (P=0.05; n=356) | 3.47 (P=0.08; n=356) | 0.84 (P=0.001; n=354)a | 2.72 (P=0.71; n=355) | 2.53 (P=0.04; n=352) | 2.56 (P=0.06; n=356) |
Data are presented as adjusted mean scores, with the SEM of the difference [between adjusted mean score and overall score]). AV, arteriovenous; IV, intravenous.
Indicates a significant result.
Discussion
Our analysis demonstrates good item convergence and item discrimination, and satisfactory internal consistency reliability for the CAHPS-ICH NCC and QoC composites. The PI composite had less than desirable internal consistency reliability (α<0.7) and the data did not fit the CFA model well. The ICCs at the center level ranged from 0.13 to 0.31 and the estimates of the number of responses required to obtain a reliability of 0.70 were relatively low, ranging from 5 to 16, which suggests that reliable information can be obtained with relatively few patients. Using a Bonferroni adjusted P value of <0.0005, we identified some patient and dialysis center characteristics that were associated with the CAHPS-ICH scales. These included the number of patients per nurse, which differentiated scores on the PI composite (P=0.0001), and the ratio of patients per physician, for which a higher ratio was associated with lower mean scores on the dialysis center and the dialysis center staff global ratings (P=0.0001 for both).
These results are consistent with recent patient experiences of care in United States hospitals (10,11). Jha and colleagues examined data from the CAHPS Hospital Survey (HCAHPS), which captures the performance of hospitals in the United States in terms of patient experience (10). Hospitals with higher ratios of nurses to patient-days performed better than those with the lowest ratios, with 70% and 64% of patients, respectively, indicating that they “would definitely recommend” the hospital (P<0.001). In 2009, Kutney-Lee and colleagues examined the relationship between nursing and HCAHPS scores across 430 United States hospitals (11). They also found that nurse-to-patient staffing ratios were significantly associated with multiple HCAHPS domains, including whether patients were likely to recommend a hospital.
CAHPS surveys are used in several therapeutic areas to inform patient experiences with care in a variety of settings such as ESRD (12), women’s health (13), nursing home residents (14), family member experiences of nursing home care (15), and dental care (16). An evaluation of the CAHPS Medicare survey for patients with ESRD conducted in 2009–2010 was recently reported (12). The cohort included 3794 patients with ESRD drawn from a database of 823,564 Medicare beneficiaries. The survey evaluated access to care, physician communication, customer service, and access to prescription drugs and drug information as well as ratings for overall care, personal physician, specialist physician, and the prescription drug plan. Overall, the care experience for patients with ESRD was at least as positive as that reported by those without ESRD. Within the ESRD cohort, patients who were black or had less education reported worse experiences than other patients with ESRD. A similar pattern was observed in this analysis of the CAHPS-ICH, with black/African-American patients reporting worse experiences than white/Caucasian and Hispanic/Latino patients in terms of the dialysis center. Bagchi and colleagues recently explored this apparent disparity through a series of focus groups with African Americans, Asian Indians, Latinos, and whites (17). They found that racial/ethnic groups were more likely to regard cultural competency and the provision of a holistic approach to care as important to healthcare quality.
Patient experience of care reflects the value of the interpersonal aspects of their health care experience and complements health-related quality of life evaluations in assessing care from the patient’s perspective (18). Information derived from direct evaluation of patient experience of care can be used to identify areas for improvement and support changes in care provision with the aim of improving the overall quality of care for patients. The recent inclusion of patient experience of care evaluation as a component of the ESRD PPS for Medicare reimbursement of outpatient dialysis-related services means that individual dialysis centers need to attempt to administer the survey to their patients and report that they did so in order to meet one of the three reporting measures that together comprise 10% of facility’s total QIP score (19). The CAHPS-ICH is a key element of this quality-led evaluation of care provision and this analysis supports the utility of this tool in ensuring that dialysis centers strive for and achieve delivery of high-quality care through the use of public funds (20,21). Utilization of the CAHPS-ICH should support comparisons between dialysis centers in terms of the quality of care they offer and sharing of best practices between centers. However, some issues such as the ethnic/racial distribution of the patient population may have an effect on between-center comparisons, as previously discussed.
The survey was conducted in a limited number of centers. To assess the generalizability of the results, we compared the patient cohort included in the current analysis with the US Renal Data System 2011 dialysis population and found the populations to be similar in terms of age and sex (22). When considering the results presented here, the limitations of the survey approach should be considered. We were unable to compare patients who did and did not complete the survey in terms of relevant variables that may have affected their experience of care and the likelihood of their completing the survey.
The CAHPS-ICH provides a valuable and informative tool for the evaluation of patient-perceived quality of care. This analysis suggests that at the center level, staff-to-patient ratios and the information provided to patients may be of particular relevance to patients in determining their perception of the quality of care they receive. However, measurement properties are specific to the setting in which they were assessed, so it will be important to repeat the analysis in other settings in order to test the replicability and thus the generalizability of the findings. Furthermore, although this analysis has focused on the composite scores, individual item results could also be used to assess aspects of care and may serve as markers for specific quality improvement endeavors. Additional tools will be needed to evaluate other aspects of quality of care, such as health-related quality of life, with clinical elements such as anemia education and management and vascular access, which form a significant part of the dialysis treatment routine. Additional psychometric evaluation would be useful before using CAHPS-ICH as a CMS QIP metric. Equally, further work to establish minimally important cut-off scores and changes or differences in scores that are clinically important may also be required. The CAHPS-ICH has the potential to assess the quality of service in renal dialysis units in the United States, yet more research is warranted to elicit the clinically meaningful difference between dialysis centers.
Disclosures
R.W., G.T.-S., and J.P. are employed by Adelphi Real World Ltd. C.J.P. is employed by Amgen, Inc. R.D.H. is a consultant to Amgen, Inc. and the Critical Path Institute. M.G. was employed by Amgen, Inc. at the time this study was initiated and conducted.
Supplementary Material
Acknowledgments
Medical writing assistance was provided by Tracey Lonergan of Adelphi Real World Ltd., and this support was funded by Amgen, Inc. Adelphi Real World Ltd. sponsored and conducted the research. After the research was completed, Amgen, Inc. subscribed for access to the data analyses. Amgen, Inc. participated in the data analysis and interpretation, and review and approval of the manuscript. All authors were involved in the analysis of the data, development, and revision of the manuscript, and decision to submit the manuscript for publication.
Footnotes
Published online ahead of print. Publication date available at www.cjasn.org.
This article contains supplemental material online at http://cjasn.asnjournals.org/lookup/suppl/doi:10.2215/CJN.10121013/-/DCSupplemental.
See related editorial, “Incentives for Caution: The In-Center Hemodialysis Consumer Assessment of Healthcare Providers and Systems Survey and Experience of Care,” on pages 1005–1006.
References
- 1.US Centers for Medicare and Medicaid Services: End Stage Renal Disease (ERSD) Quality Initiative. Available at: http://www.cms.gov/Medicare/End-Stage-Renal-Disease/ESRDQualityImproveInit/index.html Accessed February 6, 2013
- 2.US Centers for Medicare and Medicaid Services. US Department of Health and Human Services : Medicare program; end-stage renal disease prospective payment system and quality incentive program; ambulance fee schedule; durable medical equipment; and competitive acquisition of certain durable medical equipment prosthetics, orthotics and supplies. Final rule. Fed Regist 76: 70228–70316, 2011 [PubMed] [Google Scholar]
- 3.US Centers for Medicare and Medicaid Services. US Department of Health and Human Services : Medicare program; end-stage renal disease prospective payment system, quality incentive program, and durable medical equipment, prosthetics, orthotics, and supplies; proposed rule. Fed Regist 78: 40836–40890, 2013 [PubMed] [Google Scholar]
- 4.Anderson P, Benford M, Harris N, Karavali M, Piercy J: Real-world physician and patient behaviour across countries: Disease-specific programmes - a means to understand. Curr Med Res Opin 24: 3063–3072, 2008 [DOI] [PubMed] [Google Scholar]
- 5.US Department of Health and Human Services: Health information privacy. Available at: http://www.hhs.gov/ocr/privacy/ Accessed February 8, 2013
- 6.Agency for Healthcare Research and Quality: CAHPS® In-Center Hemodialysis Survey and Reporting Kit 2009. Available at: http://cahps.ahrq.gov/hemodialysis/ichsurvey.pdf Accessed February 8, 2013
- 7.Nunnally JC: Psychometric Theory, 2nd Ed., New York, McGraw-Hill, 1978 [Google Scholar]
- 8.Fayers PM, Machin D: Multi-Item Scales. In: Quality of Life: Assessment, Analysis and Interpretation, Chichester, UK, John Wiley & Sons, 2000, pp 72–79 [Google Scholar]
- 9.Hooper D, Coughlan J, Mullen MR: Structural equation modelling: Guidelines for determining model fit. Electron J Bus Res Methods 6: 53–60, 2008 [Google Scholar]
- 10.Jha AK, Orav EJ, Zheng J, Epstein AM: Patients’ perception of hospital care in the United States. N Engl J Med 359: 1921–1931, 2008 [DOI] [PubMed] [Google Scholar]
- 11.Kutney-Lee A, McHugh MD, Sloane DM, Cimiotti JP, Flynn L, Neff DF, Aiken LH: Nursing: A key to patient satisfaction. Health Aff (Millwood) 28: w669–w677, 2009 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 12.Paddison CA, Elliott MN, Haviland AM, Farley DO, Lyratzopoulos G, Hambarsoomian K, Dembosky JW, Roland MO: Experiences of care among Medicare beneficiaries with ESRD: Medicare Consumer Assessment of Healthcare Providers and Systems (CAHPS) survey results. Am J Kidney Dis 61: 440–449, 2013 [DOI] [PubMed] [Google Scholar]
- 13.Anderson RT, Weisman CS, Scholle SH, Henderson JT, Oldendick R, Camacho F: Evaluation of the quality of care in the clinical care centers of the National Centers of Excellence in Women’s Health. Womens Health Issues 12: 309–326, 2002 [DOI] [PubMed] [Google Scholar]
- 14.Sangl J, Buchanan J, Cosenza C, Bernard S, Keller S, Mitchell N, Brown J, Castle N, Sekscenski E, Larwood D: The development of a CAHPS instrument for Nursing Home Residents (NHCAHPS). J Aging Soc Policy 19: 63–82, 2007 [DOI] [PubMed] [Google Scholar]
- 15.Frentzel EM, Sangl JA, Evensen CT, Cosenza C, Brown JA, Keller S, Garfinkel SA: Giving voice to the vulnerable: The development of a CAHPS nursing home survey measuring family members’ experiences. Med Care 50[Suppl]: S20–S27, 2012 [DOI] [PubMed] [Google Scholar]
- 16.Bader JD, Shugars DA: Dental care plan assessment using the CAHPS Dental Plan Survey. J Public Health Dent 72: 295–301, 2012 [DOI] [PubMed] [Google Scholar]
- 17.Bagchi AD, Af Ursin R, Leonard A: Assessing cultural perspectives on healthcare quality. J Immigr Minor Health 14: 175–182, 2012 [DOI] [PubMed] [Google Scholar]
- 18.Browne K, Roseman D, Shaller D, Edgman-Levitan S: Analysis & commentary. Measuring patient experience as a strategy for improving primary care. Health Aff (Millwood) 29: 921–925, 2010 [DOI] [PubMed] [Google Scholar]
- 19.US Centers for Medicare and Medicaid Services; US Department of Health and Human Services: Fiscal year 2012. Justification of estimates for appropriations committees. Available at: http://www.hhs.gov/about/FY2012budget/cmsfy12cj_revised.pdf Accessed February 13, 2013
- 20.Davies E, Shaller D, Edgman-Levitan S, Safran DG, Oftedahl G, Sakowski J, Cleary PD: Evaluating the use of a modified CAHPS survey to support improvements in patient-centred care: Lessons from a quality improvement collaborative. Health Expect 11: 160–176, 2008 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 21.Rodriguez HP, von Glahn T, Elliott MN, Rogers WH, Safran DG: The effect of performance-based financial incentives on improving patient care experiences: A statewide evaluation. J Gen Intern Med 24: 1281–1288, 2009 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 22.US Renal Data System: Incident and prevalence counts by quarter, Q2 2012. Available at: http://www.usrds.org/qtr/default.aspx Accessed June 3, 2013
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.