Abstract
Background
The National Quality Forum endorsed a 3-item Care Transitions Measure (CTM-3), part of the Hospital Consumer Assessment of Healthcare Providers and Systems (HCAHPS) survey, for evaluating hospital care transitions performance.
Objective
To explore whether CTM-3 scores are a suitable proxy for quality of transitional care.
Design
Retrospective cohort study.
Participants
A random sample of 48,384 adults discharged from medical or surgical wards in all 113 acute care hospitals in Alberta, Canada, between April 2011 and March 2016.
Main Measures
CTM-3 scores and their associations with all-cause emergency department (ED) visits or non-elective readmissions at 30 days, 3 months, and 12 months anywhere in the province.
Results
CTM-3 scores were significantly lower (all p < 0.01) for females, older patients, those discharged from medical wards or teaching hospitals, and those with longer length of stay, higher Charlson scores, prior ED visits/hospitalizations, or who did not return to independent living after discharge. CTM-3 scores were not significantly associated with outcomes at 30 days (mean score 77.5 in those who subsequently had an ED visit/readmission vs. 77.9 in those who did not, p = 0.13, aOR 0.99, 95% CI 0.99–0.99). Although CTM-3 scores were significantly lower in patients who subsequently had ED visit/readmission at 3 months (77.5 vs. 78.5) and 12 months (77.6 vs. 79.5), the magnitude of risk was small: for every 10 point decrease in the CTM-3 score, the risk of ED visit/readmission was 2.6% higher (aOR 1.03, 95% CI 1.01–1.05) at 3 months and 4.0% higher (aOR 1.04, 95% CI 1.01–1.08) at 12 months.
Conclusions
The CTM-3 score is influenced by baseline patient and hospital factors, is not associated with 30-day post-discharge outcomes, and is only weakly associated with 3- and 12-month outcomes. These findings suggest that the CTM-3 score is not a good performance measure for the quality of transitional care.
Electronic supplementary material
The online version of this article (10.1007/s11606-019-05260-8) contains supplementary material, which is available to authorized users.
KEY WORDS: transitions of care, quality, patient experience of care questionnaires, readmissions, Hospital Consumer Assessment of Healthcare Providers and Systems survey
The Hospital Readmissions Reduction Program created in the Affordable Care Act imposes financial penalties on hospitals with higher readmission rates for heart failure, acute myocardial infarction, or pneumonia. As one of the few remediable factors associated with readmission rates are transitions of care,1 there is increasing interest in assessing and improving transitions of care. The National Quality Forum (www.qualityforum.org) endorsed a 3-item Care Transitions Measure (CTM-3) for evaluation of hospital performance in care transitions, and this was incorporated into the Hospital Consumer Assessment of Healthcare Providers and Systems (HCAHPS) patient satisfaction survey.2
Scores on the CTM-3 have been reported to be inversely correlated with risk of emergency department (ED) visits or readmissions within 30 days.3–5 However, the studies demonstrating this were small (200 and 192 patients with COPD, stroke, hip fracture, heart failure, or diabetes)3, 4 or highly selected (2963 patients with PCI or CABG from a single center).5 Moreover, all 3 of these studies could only capture ED visits or readmissions within the same hospital system—a substantial limitation since up to one-quarter of readmissions in North America are at hospitals other than the original one and these readmissions exhibit poorer outcomes.6–10 Thus, it is unknown whether the CTM-3 exhibits the same relationship to ED visits or readmissions in more diverse patient populations and with complete capture of subsequent healthcare resource use. Furthermore, although multiple studies have examined the psychometric properties of patient experience questionnaires such as the HCAHPS and the NHS Inpatient Survey (which contain questions analogous to those in the CTM-3 in the “Leaving Hospital” subsection), few studies have examined whether specific elements of patient experience questionnaires actually measure what they are intended to.11 As recently pointed out by Atreya and Pack, transitional care programs that improve CTM-3 scores do not necessarily reduce recidivism rates to the ED or the hospital suggesting that “there is substantial work yet to be done with the CTM-3 before it is ready for prime time use”.12
Despite these caveats, several health plans (including Medicare under the Value-Based Purchasing program) have created pay-for-performance initiatives built around hospital performance on patient experience of care measures such as CTM-3 scores.13 Thus, there is an urgent need to establish whether CTM-3 scores really are a valid proxy for quality of transitional care.14 To that end, we designed this study to examine CTM-3 scores in a random sample of all patients discharged from acute care hospitals in a universal access, publicly funded healthcare system where all interactions with the healthcare system are recorded (permitting 100% capture of subsequent ED visits or readmissions to any hospital in the entire region of study).
METHODS
Participants
We included all adults discharged from any of the 113 acute care hospitals in Alberta between April 2011 and March 2016 who completed the HCAHPS survey within 6 weeks of discharge. Alberta Health Services (AHS) is the single healthcare organization for all acute care hospitals and EDs in Alberta. AHS uses trained interviewers to conduct telephone surveys between 3 and 6 weeks after discharge on a random sample of 10% of all patients 18 years or older discharged to home, long-term care residence, or hospice after an inpatient stay of at least 24 h. Patients discharged from psychiatric units or with psychiatric consultations, and any hospitalizations associated with still-births, dilation and curettage procedures, or linked to a newborn with length of stay greater than 6 days are excluded. For the purposes of this study, we excluded any obstetric or gynecologic hospitalizations to focus on medical and surgical discharges and randomly selected one hospitalization for any patients with multiple HCAHPS-eligible hospitalizations. The patient flow chart is in Figure 1.
Figure 1.
Cohort selection flow chart.
Survey Instrument
Although AHS administers a 51-item modified version of the HCAHPS, it does include all 3 questions in the CTM-3: “during the hospital stay, staff took my preferences and those of my family or caregiver into account in deciding what my healthcare needs would be when I left”, “when I left the hospital I had a good understanding of the things I was responsible for in managing my health”, and “when I left the hospital, I clearly understood the purpose for taking each of my medications”. There are 4 response options for each question ranging from strongly disagree to strongly agree (and a “not applicable” option for the third question). The telephone interviewers use a standard script and data is captured using computer-assisted telephone interview software (Voxco; Montreal, Canada). We converted the CTM-3 results to a 100-point scale in two ways. In the first, we calculated the average CTM-3 score for each respondent after assigning points to each of their answers (from 1 for strongly disagree to 4 for strongly agree) and averaging across the 3 questions (https://caretransitions.org/wp-content/uploads/2015/08/CTM-3-SCORING.pdf). In the second, we calculated the proportion of respondents who strongly agreed with all 3 questions (https://caretransitions.org/wp-content/uploads/2015/08/CTM3Specs0807.pdf).
Data Linkage
We linked HCAHPS survey results with the Discharge Abstract Database (which records whether the hospitalization was elective or unplanned, admission date, discharge date, most responsible diagnosis, and up to 25 other diagnoses for all acute care hospitalizations in Alberta) and the National Ambulatory Care Reporting System (which captures all patient visits to any of the 104 EDs in Alberta with coding for up to 10 conditions) using each patient’s unique personal health number. We used the index hospitalization and all hospitalizations/ED visits in the prior 12 months to establish comorbidity profiles for each patient using ICD-10 codes and case definitions previously validated in these databases.15 We classified patients as being rural or urban residents based on their primary residence postal code in the Alberta Health Registry.
Outcome Measures
We examined all-cause ED visits and all-cause non-elective readmissions at 30 days, 3 months, and 12 months after index discharge date. The primary outcome was all-cause ED visit or non-elective readmission at 3 months after discharge.
Statistical Analyses
We compared baseline characteristics and outcomes between patients who did vs. did not complete the CTM-3 questions, broken down by whether they were discharged after a medical hospitalization or a surgical hospitalization, by standardized effect sizes (Cohen’s d of 0.2, 0.5, and 0.8 represent small, medium, and large effect sizes, respectively). We compared outcome rates between non-respondents and respondents. We examined the relationship between the CTM-3 score and each of the outcomes of interest. In multivariate analyses, we created a series of logistic regression models adjusted for baseline characteristics (age, sex, urban/rural residence, and any other variables in Table 1 with differences of greater than 0.1 based on the standard difference between respondents and non-respondents [except teaching hospital status since it was collinear with urban residence], using a backward stepwise selection method) and LACE (length of stay, acuity, comorbidity, and emergency department use) score16 to test whether CTM-3 scores were independently associated with our outcomes of interest. In a sensitivity analysis, we excluded any patients with events in the first 42 days to eliminate potential confounding from early events (i.e., before the patient completed the HCAHPS survey) influencing CTM-3 scores and examined the relationship between CTM-3 scores and readmissions/ED visits between 43 and 365 days after discharge using a series of logistic regression models.
Table 1.
Characteristics of Patients
| Medical Discharges | Surgical Discharges | ||||||
|---|---|---|---|---|---|---|---|
| CTM-3 respondents N = 11,689 |
Non-respondents N = 14,219 |
Standardized difference (respondent vs. not) | CTM-3 respondents N = 7318 |
Non-respondents N = 15,158 |
Standardized difference (respondent vs. not) | ||
| Mean age (SD) | 62.20 (18) | 61.49 (18) | 0.04 | 55.25 (16) | 57.57 (16) | 0.16 | |
| Male | 5147 (44.0) | 6643 (46.7) | 0.06 | 3131 (42.8) | 6914 (45.6) | 0.18 | |
| Rural resident | 5725 (49.0) | 4262 (30.0) | 0.04 | 1687 (23.1) | 2755 (18.2) | 0.16 | |
| Charlson Comorbidity Index Score | Mean (SD) | 2.25 (3) | 2.20 (3) | 0.02 | 1.25 (2) | 1.37 (2) | 0.06 |
| 0 | 3988 (34.1) | 4737 (33.3) | 4296 (58.7) | 8065 (53.2) | |||
| 1–2 | 3735 (32.0) | 4814 (33.9) | 1871 (25.6) | 4463 (29.4) | |||
| 3+ | 3966 (33.9) | 4668 (32.8) | 1151 (15.7) | 2630 (17.4) | |||
| Length of stay for index hospitalization | Days | 6.80 (10) | 6.86 (11) | 0.005 | 4.63 (10) | 4.78 (9) | 0.02 |
| Mean LACE index score at time of discharge (SD) | 11.44 (3) | 11.39 (3) | 0.01 | 7.74 (4) | 8.05 (4) | 0.09 | |
| 1+ ED visits, prior 6 months | 9815 (83.9) | 11,533 (81.1) | 0.08 | 5713 (78.1) | 11,926 (78.7) | 0.01 | |
| 1+ hospitalizations, prior year | 5142 (44.0) | 5446 (38.3) | 0.12 | 1620 (22.1) | 3222 (21.3) | 0.02 | |
| Admission urgency for index hospitalization | Elective | 1136 (9.7) | 1253 (8.8) | 0.05 | 4927 (67.3) | 9996 (65.9) | 0.17 |
| Non-elective | 10,533 (90.1) | 12,966 (91.1) | 2391 (32.7) | 5162 (34.1) | |||
| Index hospital characteristics |
Large Medium Small Teaching |
1170 (10.0) 1815 (15.5) 7907 (67.6) 783 (6.7) |
3941 (27.7) 2031 (14.3) 5374 (37.8) 2479 (17.4) |
0.66 |
2310 (31.6) 2234 (30.5) 1511 (20.6) 1263 (17.3) |
6896 (45.5) 2693 (17.8) 1392 (9.2) 4177 (27.6) |
0.52 |
| Admitted to the intensive care unit | 520 (4.4) | 1007 (7.1) | 0.11 | 661 (9.0) | 2222 (14.6) | 0.18 | |
| Discharge disposition after index hospitalization | |||||||
| Home without support | 10,340 (88.5) | 12,413 (87.3) | 0.04 | 6702 (91.6) | 13,684 (90.3) | 0.05 | |
| Home with support | 1237 (10.6) | 1689 (11.9) | 608 (8.3) | 1441 (9.5) | |||
| Long-term care/skilled nursing facility | 112 (1.0) | 117 (0.8) | 8 (0.1) | 33 (0.2) | |||
Ethics
As all analyzed data was de-identified and there was no contact between study personnel and patients, a waiver of consent was granted by the University of Alberta ethics board due to the retrospective nature of this study (Pro00081121). Of note, respondents to the HCAHPS are informed at the time of the telephone survey that their anonymized data could be used for quality assurance and/or research purposes.
RESULTS
In the 5 years (2011–2016) we studied, 48,384 (8.5%) of all 569,648 adult Albertans with medical or surgical hospital discharges were contacted for the HCAHPS survey and the response rate was 39.3% (45.2% for patients discharged from medical wards and 32.6% for patients discharged from surgical wards). The 19,007 patients who completed the CTM-3 were more likely to be female, rural residents, and from smaller non-teaching hospitals than the 29,377 patients who did not respond but age, length of stay (LOS) for the index hospitalization, Charlson score, LACE score, admitting service, and prior health resource use were very similar in both groups (Table 1). Amongst CTM-3 respondents, patients discharged from medical units were older (62.2 vs. 55.3 years, p < 0.001), had longer LOS (6.8 vs. 4.6 days, p < 0.0001), higher Charlson scores (2.3 vs. 1.3, p < 0.0001), and greater LACE scores (11.4 vs. 7.7, p < 0.0001) than those discharged from surgical units (Table 1).
Amongst medical discharges, the proportions of patients with all-cause ED visits were similar between CTM-3 respondents and non-respondents: 56.9% vs. 57.1% at 30 days, 77.5% vs. 77.2% at 3 months, and 92.7% vs. 92.1% at 12 months (Table 2). However, the all-cause readmission rates were significantly higher for CTM-3 respondents after medical discharges: 8.8% vs. 7.9% at 30 days, 17.6% vs. 16.0% at 3 months, and 36.5% vs. 33.9% at 12 months (Table 2). The opposite pattern was seen for surgical discharges with fewer CTM-3 respondents (compared with non-respondents) having all-cause ED visits (61.5% vs. 64.1% at 30 days, 77.6% vs. 81.8% at 3 months, and 89.0% vs. 91.4% at 12 months, Table 2) or readmissions (4.1% vs. 4.7% at 30 days, 6.8% vs. 7.6% at 3 months, and 14.2% vs. 14.5% at 12 months, Table 2). Similar patterns were seen when expressed as rates per 100 person years (Online Appendix, eTable 1) but all of the standardized differences were very small (all less than 0.09, and most less than 0.04). Our sensitivity analysis excluding patients with events that happened within the first 42 days demonstrated the same pattern as described above: after medical discharges, CTM-3 respondents exhibited higher event rates than non-respondents but after surgical discharges, CTM-3 respondents exhibited lower event rates than non-respondents (Online Appendix, eTable 2).
Table 2.
Post-discharge Outcomes
| Outcome | Medical discharges | Surgical discharges | ||||
|---|---|---|---|---|---|---|
| CTM-3 respondents (n = 11,689) | Non-respondents (n = 14,219) | p value | CTM-3 respondents (n = 7318) | Non-respondents (n = 15,158) | p value | |
| 30 days | ||||||
| All-cause ED visit | 6657 (56.95%) | 8119 (57.10%) | 0.81 | 4503 (61.53%) | 9802 (64.07%) | < 0.0001 |
| All-cause non-elective readmission | 1030 (8.81%) | 1116 (7.85%) | 0.005 | 297 (4.06%) | 710 (4.68%) | 0.03 |
| All-cause ED visit or non-elective readmission | 6690 (57.23%) | 8168 (57.44%) | 0.73 | 4509 (61.62%) | 9822 (64.80%) | < 0.0001 |
| 3 months | ||||||
| All-cause ED visit | 9053 (77.45%) | 10,977 (77.20%) | 0.63 | 5678 (77.59%) | 12,401 (81.81%) | < 0.0001 |
| All-cause non-elective readmission | 2059 (17.61%) | 2273 (15.99%) | 0.0005 | 500 (6.83%) | 1149 (7.58%) | 0.04 |
| All-cause ED visit or non-elective readmission | 9080 (77.68%) | 11,003 (77.38%) | 0.57 | 5681 (77.63%) | 12,410 (81.87%) | < 0.0001 |
| 12 months | ||||||
| All-cause ED visit | 10,836 (92.70%) | 13,099 (92.12%) | 0.08 | 6510 (88.96%) | 13,848 (91.36%) | < 0.0001 |
| All-cause non-elective readmission | 4270 (36.53%) | 4813 (33.85%) | < 0.0001 | 1037 (14.17%) | 2194 (14.47%) | 0.54 |
| All-cause ED visit or non-elective readmission | 10,850 (92.82%) | 13,112 (92.21%) | 0.06 | 6511 (88.97%) | 13,853 (91.39%) | < 0.0001 |
Number of patients with each outcome (and percentage)
CTM-3 scores were significantly lower (all p < 0.01) for females, older patients, those discharged from medical vs. surgical units, those hospitalized in teaching hospitals, patients who did not return to independent living after discharge, and those with longer LOS, higher Charlson scores, higher LACE scores, or prior ED visits or hospitalizations (Table 3). It is important to note that all of these factors are known to influence post-discharge outcomes, and thus these factors confound the relationship between CTM-3 scores and outcomes after discharge. Even without adjusting for these confounders, there was no significant association between CTM-3 scores and outcomes at 30 days (mean score 77.5 in those who subsequently had an ED visit or readmission vs. 77.9 in those who did not, p = 0.13, aOR 0.99, 95% CI 0.986–0.004) (Table 4). The multivariate model to predict 30-day readmissions using age and LACE score alone had a c-statistic of 0.65; adding CTM-3 scores to that model only changed the c-statistic to 0.655 which was not statistically significantly different. CTM-3 scores were significantly lower (both p < 0.01, Table 4) in patients who subsequently had non-elective readmission/ED visit at 3 months (77.5 vs. 78.5) and 12 months (77.6 vs. 79.5). However, the magnitude of outcome differences was small even for large changes in the CTM-3 score: for every 10 point decrease in the CTM-3 score, the risk of readmission/ED visit was 2.6% higher (aOR 1.026, 95% CI 1.01–1.05) at 3 months and 4.0% higher (aOR 1.04, 95% CI 1.01–1.08) at 12 months. After excluding events that happened within the first 42 days, we found that CTM-3 scores were still significantly associated with readmissions/ED visits between 43 and 365 days after discharge but the magnitude of the association again remained small: for every 10 point decrease in the CTM-3 score, the risk of readmission/ED visit at 12 months was only 4.4% higher (aOR 1.04, 95% CI 1.01–1.08).
Table 3.
Mean CTM-3 Scores by Baseline and Hospital Characteristics
| Mean (SD) score for all CTM-3 questions after assigning points to each response option | p value | Proportion (count) who answered strongly agree to all CTM-3 questions | p value | |
|---|---|---|---|---|
| Age | ||||
| -65 and older | 77.04 (16) | 0.015 for trend | 19.80 (1605) | < 0.001 for trend |
| -50 to 64 years | 78.93 (18) | 25.42 (1382) | ||
| -Younger than 50 years | 77.38 (19) | 22.66 (1238) | ||
| Sex | ||||
| -Male | 78.43 (17) | < 0.01 | 22.75 (1883) | 0.13 |
| -Female | 77.13 (18) | 21.82 (2341) | ||
| Residence | ||||
| -Rural | 78.02 (17) | 0.04 | 21.99 (1630) | 0.5 |
| -Urban | 77.50 (18) | 22.38 (2595) | ||
| Length of hospital stay | ||||
| -Greater than median | 76.98 (17) | < 0.01 | 20.99 (1928) | < 0.001 |
| -Median or less | 78.38 (17) | 23.39 (2297) | ||
| Charlson Comorbidity Index Score | ||||
| -0 | 77.97 (18) | < 0.01 | 22.66 (1877) | 0.02 |
| -1–2 | 77.92 (17) | 22.83 (1280) | ||
| -3 or more | 77.01 (17) | 20.87 (1068) | ||
| LACE index score at time of discharge | ||||
| -Greater than median | 76.79 (17) | < 0.01 | 2066 (1725) | < 0.001 |
| -Median or less | 78.41 (17) | 23.46 (2500) | ||
| 1+ ED visits, prior 6 months | ||||
| -Yes | 77.47 (17) | < 0.01 | 22.04 (3418) | 0.13 |
| -No | 78.70 (17) | 23.20 (807) | ||
| 1+ hospitalizations, prior year | ||||
| Yes | 76.93 (18) | < 0.01 | 21.19 (1433) | 0.01 |
| No | 78.12 (17) | 22.80 (2792) | ||
| Admitting service | ||||
| -Medicine | 76.80 (18) | < 0.01 | 20.44 (2389) | < 0.001 |
| -Surgery | 79.13(17) | 25.09 (1836) | ||
| Admission urgency for index hospitalization | ||||
| -Elective | 79.72 (17) | < 0.01 | 26.01 (1577) | < 0.001 |
| -Urgent | 76.75 (18) | 20.46 (2648) | ||
| Index hospital characteristics | ||||
| Large | 78.37 (18) | < 0.01 | 24.28 (845) | 0.01 |
| Medium | 76.52 (18) | 20.65 (836) | ||
| Small | 78.09 (17) | 22.26 (2096) | ||
| Teaching | 77.16(18) | 21.80 (446) | ||
| Admitted to the intensive care unit during index hospitalization | 0.50 | 0.20 | ||
| -Yes | 78.03 (18) | 23.88 (282) | ||
| -No | 77.68 (17) | 22.12 (3943) | ||
Table 4.
Mean CTM-3 Scores by Outcomes
| Mean (SD) score for all CTM-3 questions after assigning points to each response option | p value | Proportion (count) who answered strongly agree to all CTM-3 questions | p value | |
|---|---|---|---|---|
| Discharge disposition | < 0.01 | < 0.001 | ||
| Independent | 77.94 (17) | 22.62 (3855) | ||
| Home with homecare | 75.74 (17) | 19.02 (351) | ||
| Long-term care/skilled nursing facility | 73.37 (16) | 15.83 (19) | ||
| 30-day all-cause ED visit or non-elective readmission | 0.13 | 0.50 | ||
| Yes | 77.54 (18) | 22.06 (2470) | ||
| No | 77.93 (17) | 22.48 (1755) | ||
| 3-month all-cause ED visit or non-elective readmission | < 0.01 | 0.30 | ||
| Yes | 77.47 (17) | 22.06 (3256) | ||
| No | 78.50 (17) | 22.82 (969) | ||
| 12-month all-cause ED visit or non-elective readmission | < 0.01 | 0.02 | ||
| Yes | 77.53 (17) | 22.01 (3822) | ||
| No | 79.46 (17) | 24.48 (403) | ||
| 30-day non-elective readmission | < 0.01 | < 0.01 | ||
| Yes | 75.01 (19) | 19.07 (253) | ||
| No | 77.90(17) | 22.47 (3972) | ||
| 3-month non-elective readmission | < 0.01 | < 0.01 | ||
| Yes | 75.8 3(18) | 19.38 (496) | ||
| No | 77.99 (17) | 22.67 (3729) | ||
| 12-month non-elective readmission | < 0.01 | < 0.01 | ||
| Yes | 76.44 (18) | 20.24 (1074) | ||
| No | 78.19 (17) | 23.00 (3151) | ||
DISCUSSION
In order for a performance measure to be a good quality indicator, it must be feasible to measure in a timely fashion, comparable across settings (i.e., risk adjustment is possible to permit fair comparisons), valid (i.e., truly discriminates quality for the measure it is the supposed proxy for), reliable (i.e., reproducible in different hands), precise (i.e., clear, well-specified definitions with wide variation between patients or settings and sensitive to change), and actionable.17–19
We believe our findings suggest that the CTM-3 score is not a suitable performance measure for the quality of transitional care for several reasons. First, while the CTM-3 score is feasible to measure and reproducible (since it consists of 3 standardized survey questions), and was shown to improve after implementation of a care transition program in a randomized trial, that same trial demonstrated that the improvement in CTM-3 scores was not associated with a change in 30-day readmission or ED visit rates.20 We believe that identifying problems in preparedness to transition back to self-care 4 to 6 weeks after discharge is too late and outside the actionable window; while rates of early recidivism (within 1–2 weeks) seem more likely to reflect the quality of transitional care, this is before CTM-3 scores are often collected. Second, we demonstrated that CTM-3 scores are influenced by patient and hospital factors at baseline, none of which are readily modifiable, and to the best of our knowledge (and others),12 there are no models to risk adjust CTM-3 scores for these confounders at this time. Indeed, 2 prior studies in the USA also reported that age, gender, and baseline health status influenced CTM-3 scores.5,14 Third, the fact that CTM-3 scores were not associated with 30-day post-discharge outcomes, and were only weakly associated with 3- and 12-month outcomes, raises questions about whether the CTM-3 truly is a valid proxy for the quality of transitional care or whether other confounders influence late outcomes more than the transitional care 3–12 months earlier. Mixon and colleagues also recently reported that the CTM-3 score was not significantly associated with 30-day or 3-month death/readmission in 1239 patients with cardiovascular diagnoses in Tennessee—the c-statistic for the CTM-3 score was only 0.52 for 30-day outcomes in their study.21 While an examination of the Medicare Hospital Compare database reported statistically significant inverse relations between CTM-3 scores and 30-day readmission rates after hospitalizations for heart failure, pneumonia, or acute myocardial infarction, all of the correlations were weak (with correlation coefficients between − 0.20 and − 0.30).22
We found that patients who received the HCAHPS survey differed from those who were not surveyed in several baseline factors—this was not surprising since AHS selected medical and surgical patients and excluded psychiatric patients and certain obstetric cases from the HCAHPS surveys. We also found that CTM-3 respondents were different from non-respondents in several baseline factors and had different outcome rates. This is again not surprising since response bias is common in any survey—more satisfied patients are more likely to respond. Regardless, it means that we may have underestimated the magnitude of the inverse association between scores and event rates in our study, although the fact that the differences in post-discharge outcome rates were so small argues against a substantive effect.
Although we were able to examine CTM-3 scores for a wide spectrum of patients discharged from general medical or surgical units in an entire Canadian province, and were able to identify and capture all non-elective readmissions or ED visits anywhere in the province, there are some limitations to our work. First, the CTM-3 does not capture data on socioeconomic status or other social determinants of health which influence readmission risk and ED usage. Second, in examining the association between CTM-3 scores and post-discharge outcomes, we were not able to adjust for post-discharge outpatient visits or continuity of care, both of which are known to influence rates of readmission or ED visits.23–25 Third, the CTM-3 does not assess patient understanding of medication instructions, yet medication adherence is known to be a key driver of post-discharge outcomes.26 A fourth limitation, which is common to all studies examining the relationship between CTM-3 scores and post-discharge events, is that patients with early readmissions/ED visits may complete the HCAHPS survey after the second event, thereby biasing their answers. We do not have data on when each respondent in this cohort completed the CTM-3 survey but prior work in Alberta revealed that the vast majority of surveys were completed within 4 weeks after hospital discharge.27 For this reason, we looked beyond the 30-day post-discharge period to consider event rates at 3 and 12 months and in a sensitivity analysis, we excluded events within the first 6 weeks of discharge. Fifth, although selection and response biases are always a concern in survey studies, earlier work has demonstrated that those patients AHS chooses to administer the HCAHPS to, and those who respond, are representative of the broader Alberta medical/surgical inpatient population.27 Furthermore, the comorbidity profiles and resource utilization patterns of our cohort are consistent with reports of patients discharged from medical and surgical units in other Canadian provinces and Boston.28 Finally, any survey study, particularly one conducted by telephone,29 is at risk for social desirability response bias, which could inflate CTM-3 scores and thereby potentially reduce any apparent associations between CTM-3 scores and outcomes.
In conclusion, there is increasing enthusiasm for the use of patient experience of care measures to judge hospital quality and drive pay for performance programs, and there is evidence that some of these measures do correlate with important outcomes during and after hospitalization such as mortality, all-cause readmissions/ED visits, or other patient safety indicators.30,31 However, we do not think the CTM-3 score meets the evidentiary threshold for a performance measure of the quality of transitional care since it is influenced by baseline patient and hospital factors, is not associated with 30-day post-discharge outcomes, and is only weakly associated with 3- and 12-month outcomes.
Electronic Supplementary Material
(DOCX 16 kb)
Authors’ Contributions
Study concept and design: FM, KK, HQ.
Acquisition of data: FM, ML, JB.
Statistical analysis: ML, JB.
Interpretation of data and first draft of manuscript: FM.
Critical revisions of manuscript and approval of submission: all authors.
Funding
No project-specific funding. FM holds the Alberta Health Services Chair in Cardiovascular Outcomes at the University of Alberta.
Compliance with Ethical Standards
Conflict of Interest
The authors declare that they do not have a conflict of interest.
Footnotes
Prior Presentations
None.
Publisher’s Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Contributor Information
Finlay A. McAlister, Phone: (780) 492-9824, Email: Finlay.McAlister@ualberta.ca.
Mu Lin, Email: lmu@ualberta.ca.
Jeff Bakal, Email: jbakal@ualberta.ca.
Kyle A. Kemp, Email: kkemp@ucalgary.ca.
Hude Quan, Email: hquan@ucalgary.ca.
References
- 1.Kansagara D, Englander H, Salanitro A, et al. Risk prediction models for hospital readmission: a systematic review. JAMA. 2011;306:1688–1698. doi: 10.1001/jama.2011.1515. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 2.http://www.hcahpsonline.org. Centers for Medicare & Medicaid services, Baltimore, MD. Accessed on May 17, 2019.
- 3.Coleman EA, Mahoney E, Parry C. Assessing the quality of preparation for posthospital care from the patient’s perspective: the care transitions measure. Med Care. 2005;43:246–255. doi: 10.1097/00005650-200503000-00007. [DOI] [PubMed] [Google Scholar]
- 4.Coleman EA, Parry C, Chalmers SA, Chugh A, Mahoney E. The central role of performance measurement in improving the quality of transitional care. Home Health Care Serv Q. 2007;26:93–104. doi: 10.1300/J027v26n04_07. [DOI] [PubMed] [Google Scholar]
- 5.Goldstein JN, Hicks LS, Kolm P, Weintraub WS, Elliott DJ. Is the care transitions measure associated with readmission risk? Analysis from a single academic center. J Gen Intern Med. 2016;31:732–738. doi: 10.1007/s11606-016-3610-9. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 6.Staples J. A., Thiruchelvam D., Redelmeier D. A. Site of hospital readmission and mortality: a population-based retrospective cohort study. CMAJ Open. 2014;2(2):E77–E85. doi: 10.9778/cmajo.20130053. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 7.Kind AJM, Bartels C, Mell MW, Mullahy J, Smith M. For-profit hospital status and rehospitalizations at different hospitals: an analysis of Medicare data. Ann Intern Med. 2010;153:718–727. doi: 10.7326/0003-4819-153-11-201012070-00005. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 8.Hempstead K, DeLia D, Cantor JC, Nguyen T, Brenner J. The fragmentation of hospital use among a cohort of high utilizers. Implications for emerging care coordination strategies for patients with multiple chronic conditions. Med Care. 2014;52:S67–S74. doi: 10.1097/MLR.0000000000000049. [DOI] [PubMed] [Google Scholar]
- 9.Bourgeois FC, Olson KL, Mandl KD. Patients treated at multiple acute health care facilities. Quantifying information fragmentation. Arch Intern Med. 2010;170:1989–1995. doi: 10.1001/archinternmed.2010.439. [DOI] [PubMed] [Google Scholar]
- 10.McAlister FA, Youngson E, Kaul P. Patients with heart failure readmitted to the original hospital have better outcomes than those readmitted elsewhere. JAHA. 2017;6:e004892. doi: 10.1161/JAHA.116.004892. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 11.Beattie M, Murphy DJ, Atherton I, Lauder W. Instruments to measure patient experience of healthcare quality in hospitals: a systematic review. Syst Rev. 2015;4:97. doi: 10.1186/s13643-015-0089-0. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 12.Atreya AR, Pack QR. Care Transitions Measure score and coronary revascularizaton related readmission: Ready for primetime use? J Gen Intern Med. 2016;31:707–709. doi: 10.1007/s11606-016-3691-5. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 13.https://data.medicare.gov/Hospital-Compare/Hospital-Value-Based-Purchasing-HVBP-Patient-Exper/avtz-f2ge. Last accessed May 17, 2019.
- 14.Parry C, Mahoney E, Chalmers SA, Coleman EA. Assessing the quality of transitional care. Further applications of the Care Transitions Measure. Med Care. 2008;46:317–322. doi: 10.1097/MLR.0b013e3181589bdc. [DOI] [PubMed] [Google Scholar]
- 15.Quan H, Sundararajan V, Halfon P, et al. Coding algorithms for defining comorbidities in ICD-9-CM and ICD-10 administrative data. Med Care. 2005;43:1130–1139. doi: 10.1097/01.mlr.0000182534.19832.83. [DOI] [PubMed] [Google Scholar]
- 16.van Walraven C, Dhalla IA, Bell C, et al. Derivation and validation of an index to predict early death or unplanned readmission after discharge from hospital to the community. CMAJ. 2010;182:551–557. doi: 10.1503/cmaj.091117. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 17.Spertus JA, Eagle KA, Krumholz HM, Mitchell KR, Normand SL. American College of Cardiology and American Heart Association methodology for the selection and creation of performance measures for quantifying the quality of cardiovascular care. J Am Coll Cardiol. 2005;45:1147–1156. doi: 10.1016/j.jacc.2005.03.011. [DOI] [PubMed] [Google Scholar]
- 18.Dimick JB. What makes a “good” quality indicator? Arch Surg. 2010;145:295. doi: 10.1001/archsurg.2009.291. [DOI] [PubMed] [Google Scholar]
- 19.Spertus JA, Bonow RO, Chan P, et al. ACCF/AHA New insights into the methodology of performance measurement. J Am Coll Cardiol. 2010;56:1767–1782. doi: 10.1016/j.jacc.2010.09.009. [DOI] [PubMed] [Google Scholar]
- 20.Englander H, Michaels L, Chan B, Kansagara D. The care transitions innovation (C-Train) for socioeconomically disadvantaged adults: results of a cluster randomized controlled trial. J Gen Intern Med. 2014;29:1460–1467. doi: 10.1007/s11606-014-2903-0. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 21.Mixon AS, Goggins K, Bell SP, et al. Preparedness for hospital discharge and prediction of readmission. J Hosp Med. 2016;11:603–609. doi: 10.1002/jhm.2572. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 22.Dong N, Eisenberg JD, Dharmarajan K, Spatz ES, Desai NR. Relationship between patient-reported hospital experience and 30-day mortality and readmission rates for acute myocardial infarction, heart failure, and pneumonia. J Gen Intern Med. 2019;34:526–528. doi: 10.1007/s11606-018-4746-6. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 23.McAlister FA, Youngson E, Bakal JA, Kaul P, Ezekowitz J, van Walraven C. Impact of physician continuity on death or urgent readmission after discharge among patients with heart failure. CMAJ. 2013;185:e681–e689. doi: 10.1503/cmaj.130048. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 24.Gruneir A, Bronskill SE, Maxwell CJ, et al. The association between multimorbidity and hospitalization is modified by individual demographics and physician continuity of care: a retrospective cohort study. BMC Health Serv Res. 2016;16:154. doi: 10.1186/s12913-016-1415-5. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 25.Pereira Gray DJ, Sidaway-Lee K, White E, Thorne A, Evans PH. Continuity of care with doctors – a matter of life and death? A systematic review of continuity of care and mortality. BMJ Open. 2018;8:e021161. doi: 10.1136/bmjopen-2017-021161. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 26.Sokol M, McGuigan K, Verbrugge R, et al. Impact of medication adherence on hospitalization risk and healthcare cost. Med Care. 2005;43:521–530. doi: 10.1097/01.mlr.0000163641.86870.af. [DOI] [PubMed] [Google Scholar]
- 27.Kemp Kyle A., Chan Nancy, McCormack Brandi. The Alberta Inpatient Hospital Experience Survey: Representativeness of Sample and Initial Findings. Survey Practice. 2015;8(3):1–10. doi: 10.29115/SP-2015-0012. [DOI] [Google Scholar]
- 28.van Walraven C, McAlister FA, Bakal J, Hawken S, Donze J. External validation of the Hospital-patient One-year Mortality Risk (HOMR) score for predicting death within 1 year after hospital admission. CMAJ. 2015;187:725–733. doi: 10.1503/cmaj.150209. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 29.Elliott MN, Zaslavsky AM, Goldtein E, et al. The effects of survey mode, patient mix, and nonresponse on CAHPS survey scores. Health Serv Res. 2009;44:501–518. doi: 10.1111/j.1475-6773.2008.00914.x. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 30.Wang DE, Tsugawa Y, Figueroa JF, Jha AK. Association between the centers for Medicare and Medicaid services hospital star rating and patient outcomes. JAMA Intern Med. 2016;176:848–850. doi: 10.1001/jamainternmed.2016.0784. [DOI] [PubMed] [Google Scholar]
- 31.Kemp KA, Santana MJ, Southern DA, McCormack B, Quan H. Association of inpatient hospital experience with patient safety indicators: a cross-sectional, Canadian study. BMJ Open. 2016;6:e011242. doi: 10.1136/bmjopen-2016-011242. [DOI] [PMC free article] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.
Supplementary Materials
(DOCX 16 kb)