Drivers of Inpatient Hospital Experience Using the HCAHPS Survey in a Canadian Setting

Abstract

Objective

To identify factors associated with patients’ overall rating of inpatient hospital care.

Data Sources

Two years of patient interview data (April 1, 2011 to March 31, 2013), linked to inpatient administrative records.

Study Design

Patients rated their overall health on a scale of 0 (worst care) to 10 (best care) using the HCAHPS instrument administered via telephone, up to 42 days postdischarge. Logistic regression was used to generate odds ratios for each independent predictor.

Data Extraction

HCAHPS data were linked to inpatient records based on health care numbers and dates of service. The outcome (overall health experience) was collapsed into two groups (10 vs. 0–9).

Principal Findings

Overall hospital experience of 0–9 was associated with younger age, male gender, higher level of education, being born in Canada, urgent admission, not having a family practitioner as the most responsible provider service, and not being discharged home. A length of stay of less than 3 days was protective. The c-statistic for the multivariate model was 0.635.

Conclusions

Our results are novel in the Canadian population. Several questions for future research have been generated, in addition to opportunities for quality improvement within our own organization.

Many health care organizations now include a measure of patient experience among their list of key performance indicators. However, in the infancy of this movement, organizations developed their own instruments to measure patient experience, which may have satisfied an organizational need, but due to a lack of standardized survey instruments and data collection methodologies, did not permit for valid comparisons across organizations.

To enable an “apples to apples” comparison, the Hospital-Consumer Assessment of Healthcare Providers and Systems (HCAHPS) survey was developed and issued inz06. To encourage public transparency, data accessibility, and to provide an incentive for provider improvement, HCAHPS data from the United States are now publicly available on the Internet (Centers for Medicare and Medicaid Services 2014). Aggregate HCAHPS data are also available via an annual National Healthcare Quality Report from the Agency for Healthcare Research and Quality (AHRQ) (Agency for Healthcare Research and Quality 2013).

Recently, research has shown that a better patient experience is linked to improved outcomes (Glickman et al. 2010; Isaac et al. 2010; Meterko et al. 2010). Additional works have also shown that some HCAHPS items may be associated with patient-level characteristics such as gender (Elliott et al. 2012), health status, education level, age (Goldstein et al. 2010; Elliott et al. 2012), and ethnicity (Elliott et al. 2012). To date, however, no research or quality improvement initiatives of a similar format have been conducted in a Canadian setting. Evidence to document factors contributing to patient experience in a publicly funded universal health care setting may prove useful in the design of targeted interventions to improve patient experience and/or the development of subsequent outcomes.

Therefore, the purpose of the present project was to identify factors that may be associated with a patient’s overall rating of his/her hospital experience during an inpatient stay within a universal health care setting.

Methods

Study Population

This retrospective data analysis encompassed 18,213 completed HCAHPS surveys, which were conducted from April 1, 2011 to March 31, 2013 in the province of Alberta, Canada. In Alberta, universal health care services are publicly funded, and they are provided to approximately 4 million residents by Alberta Health Services (AHS). For the present project, a research/data sharing agreement was in place between AHS and the University of Calgary. Research Ethics approval was obtained from the Conjoint Health Research Ethics Board at the University of Calgary.

Data Sources

Patient experience was captured using the HCAHPS survey. The current version of the inpatient HCAHPS survey is comprised of 32 core items. Twenty-one questions encompass nine key topics: communication with doctors, communication with nurses, responsiveness of hospital staff, pain management, communication about medicines, discharge information, cleanliness of the hospital environment, quietness of the hospital environment, and transition of care. The survey also includes four screener questions and seven demographic items, which are used for adjusting the mix of patients across hospitals and for analytical purposes (Centers for Medicare and Medicaid Services 2013). The Alberta version of the HCAHPS instrument was a 51-item survey, which included the 32 core items, as well as 19 additional questions which were added to address AHS-specific policies and procedures. These additional questions were asked immediately after the core HCAHPS questions, except in the case where an additional question could be included in a given section. For example, all questions pertaining to nursing care were asked consecutively, regardless of core or additional status. The survey was completed via computer-assisted telephone interview using a standard script, requiring 8–15 minutes to complete. Interview answers were captured using Voxco software (Montreal, Canada). Patients were contacted up to 42 days postdischarge from one of the 93 acute inpatient facilities in the province. Following the application of exclusion criteria (Table1), a random sample of 5 percent of all remaining eligible discharges was captured, stratified at the facility level. Surveys were completed in English only, as our department does not have the resources for multilingual administration. In a given year, this results in the exclusion of approximately 1 percent of numbers called.

Table 1.

HCAHPS Survey Exclusion Criteria within Alberta Health Services

Less than 24-hour inpatient stay

Patient is under 18 years of age (HCHAPS is only validated in adult population)

Patient died during inpatient stay (no proxy interviews permitted)

Psychiatric physician or unit (any) in inpatient record (HCAHPS has not been validated for the mental health population)

Possible dilation and curettage (D&C) procedure (excluded out of consideration)

Day surgery or ambulatory procedures (HCAHPS is only validated for inpatient stays)

Possible still birth (excluded out of consideration)

Visit tied to a baby with length of stay greater than 6 days (e.g., complication/NICU stay) (excluded out of consideration)

For this project, the HCAHPS item relating to overall hospital experience comprised the outcome variable. The item was scored from 0 (worst possible score) to 10 (best possible score). The standard wording for the question was as follows:

We want to know your overall rating of your stay at <HOSPITAL NAME>. This is the stay that ended around <DATE>. Please do not include any other hospital stays in your answer. Using any number from 0 to 10, where 0 is the worst hospital possible and 10 is the best hospital possible … What number would you use to rate this hospital during your stay?

Based on each patient’s personal health number, facility codes, and service dates, each HCAHPS record was linked to a validated national health database, The Discharge Abstract Database (DAD), which encompasses all inpatient hospital discharges. The DAD database is maintained by the Canadian Institute for Health Information, with Alberta Health Services retaining a copy of their own provincial data. DAD data entry was completed by trained health information professionals according to strict coding rules, with the dataset undergoing systematic quality checks. Linkage of HCAHPS and DAD data was possible for 17,653 of the 18,127 available HCAHPS surveys which had a valid overall hospital experience score (97.4 percent linkage). These 17,653 records comprised the final sample included in the analyses. Figure1 shows the breakdown of available data, from number of HCAHPS surveys complete to number included in the final analyses.

Figure 1.

Flowchart of Data Available and Present in Analyses

Study Variables

The independent (predictor) variables for the present study were comprised of demographic (age group, sex, marital status, education level, patient born in Canada) and clinical (admission type, length of hospital stay, most responsible provider service, discharge disposition, number of medical comorbidities) measures.

Patient age groups were as follows: 18–29 years; 30–39 years; 40–49 years; 50–59 years; 60–69 years; and 70 years and older. Marital status was coded as single (never married); married/common law/living with partner; divorced/separated/widowed. Education level was coded as elementary or junior high; senior high; college/technical school; undergraduate level; postgraduate degree complete. Admission type was classified as urgent or elective, according to the DAD. Most responsible diagnosis was collapsed into five groups, according to ICD-10-CA coding in the DAD: neoplasm (malignant or benign); circulatory diseases; musculoskeletal diseases; pregnancy and childbirth; and all others. Total length of stay was classified as either less than 3 days (median in this cohort), or 3 days or longer. Most responsible provider service was classified according to the DAD as family practitioner versus all others. Discharge disposition was classified into three groups: transferred to another/within the same facility; left against medical advice/did not return from a pass; discharged home with or without support. Comorbidity profiles were generated using the Elixhauser Comorbidity Index (Elixhauser et al. 1998). To determine the presence of 30 common medical comorbidities in the sample, a validated list of ICD-10-CA codes was searched for in each corresponding inpatient record (Quan et al. 2005). For determination of the number of comorbidities present, diagnosis types “M” (most responsible diagnosis) and “2” (postadmission comorbidity) were excluded. Number of comorbidities was classified into three groups: none; 1 or 2; and 3 or more.

Analysis

The overall rating of hospital care (outcome variable) was dichotomized into two groups: those responding with a rating of 10 (best hospital possible) versus 0–9, which represented records where there was a potential opportunity for improvement. This method was in contrast with the current framework for Alberta Health Services’ public reporting of strategic performance measures (Alberta Health Services 2014), which reports groups of 8, 9, and 10, versus 0–7. Response frequencies were calculated for all predictor variables. Univariate logistic regression analyses were performed looking at each independent predictor. Following this, a multivariate regression analysis was performed, with the model containing all variables. In all cases, odds ratios and corresponding c-statistics were reported. All analyses were performed using SAS Network Version 9.3 for Windows (Cary, NC, USA). In all cases, statistical significance was determined a priori as a p-value less than .05.

Results

The distribution of the sample’s overall patient experience scores is shown in Figure2. From this, 6,632 of the 17,653 (37.6 percent) overall patient experience responses were 10 of 10. Approximately, 92 percent (16,245 of 17,653 responses) were 7 of 10 or greater. The frequencies of the demographic and clinical variables are provided in Table2. The sample was predominantly female (65.4 percent), married, common law, or living with a partner (69.5 percent), and born in Canada (85.6 percent). From a clinical perspective, 60 percent of admissions to hospital were urgent ones. About half (51.5 percent) of patients had a family practitioner as their most responsible provider service. The majority of patients (95.4 percent) were discharged directly home with or without support services. Just over half of the sample (56.6 percent) had no documented medical comorbidities, according to the Elixhauser coding algorithm.

Figure 2.

Distribution of HCAHPS Overall Hospital Experience Scores in the Sample (n = 17,653)

Table 2.

Frequency Table (n = 17,653 Unless Otherwise Stated)

Independent Variable	n	%
Age (in years)
18–29	2,877	16.30
30–39	2,733	15.48
40–49	1,803	10.21
50–59	2,675	15.15
60–69	3,035	17.19
70 and older	4,530	25.66
Sex
Male	6,111	34.62
Female	11,542	65.38
Marital status (n = 17,550)
Single (never married)	1,840	10.48
Married/common law/living with partner	12,192	69.47
Divorced/separated/widowed	3,518	20.05
Education level (n = 16,742)
Elementary or junior high	2,160	12.90
Senior high (some or complete)	5,552	33.16
College/technical school (some or complete)	5,457	32.59
Undergraduate level (some or complete)	2,810	16.78
Postgraduate degree complete	763	4.56
Patient born in Canada (n = 17,641)
Yes	15,103	85.61
No	2,538	14.39
Admission type (n = 17,652)
Urgent	10,627	60.20
Elective	7,025	39.80
Length of hospital stay
Less than 3 days	8,557	48.47
3 days or greater	9,096	51.53
Most responsible provider service
Family practitioner	9,100	51.55
Other	8,553	48.45
Discharge disposition
Transferred to other or same facility	694	3.93
Left against medical advice/no return from pass	129	0.73
Discharged home with/without support	16,830	95.34
Number of documented Elixhauser comorbidities
None	9,982	56.55
1 or 2	5,929	33.59
3 or more	1,742	9.87

Table3 illustrates the results of the univariate and multivariate logistic regression analyses. Independently, an increased odds of having an overall hospital experience rating less than 10 (e.g., 0–9 of 10) was associated with age of 18–69 years (when compared to 70 and older), being single, or married/common law/living with a partner (compared to divorced/separated/widowed), being born in Canada, having a length of stay of less than 3 days, not having a family practitioner as the most responsible provider service, leaving against medical advice/not returning from a pass (compared with being discharged home), and having fewer than three documented Elixhauser comorbidities. The relationship between education level and overall experience was a linear one. Having an urgent admission (vs. elective) was found to be protective. Independently, the three highest c-statistic values obtained were 0.596 (education level), 0.594 (age), and 0.540 (most responsible provider service).

Table 3.

Logistic Regression Results (Odds of Having an Overall Hospital Experience Rating of 0–9)

Independent Variable	Univariate			c	Multivariate
	OR	95% CI	p		OR	95% CI	p
Age (in years)
18–29	2.21	2.00–2.44	<.0001	0.594	2.28	2.01–2.59	<.0001
30–39	2.33	2.11–2.58	<.0001		2.08	1.83–2.36	<.0001
40–49	2.06	1.83–2.31	<.0001		1.81	1.59–2.05	<.0001
50–59	1.65	1.49–1.81	<.0001		1.42	1.27–1.58	<.0001
60–69	1.19	1.08–1.30	0.0003		1.04	0.95–1.16	0.3984
70 and older	1.00	—	—		1.00	—	—
Sex
Male	0.94	0.88–1.00	0.0488	0.507	1.14	1.06–1.23	0.0005
Female	1.00	—	—		1.00	—	—
Marital status
Single (never married)	1.56	1.39–1.76	<.0001	0.531	1.00	0.87–1.14	0.9584
Married/common law/living with partner	1.36	1.26–1.47	<.0001		1.03	0.94–1.12	0.5662
Divorced/separated/widowed	1.00	—	—		1.00	—	—
Education level
Elementary or junior high	0.30	0.25–0.36	<.0001	0.596	0.35	0.29–0.42	<.0001
Senior high (some or complete)	0.49	0.41–0.58	<.0001		0.49	0.41–0.59	<.0001
College/technical school (some or complete)	0.69	0.58–0.82	<.0001		0.67	0.56–0.80	<.0001
Undergraduate level (some or complete)	0.93	0.78–1.12	0.4600		0.92	0.76–1.10	0.3437
Postgraduate degree complete	1.00	—	—		1.00	—	—
Patient born in Canada
Yes	1.15	1.05–1.25	0.0020	0.508	1.28	1.17–1.41	<.0001
No	1.00	—	—		1.00	—	—
Admission type
Urgent	0.81	0.76–0.86	<.0001	0.525	1.09	1.01–1.18	0.0285
Elective	1.00	—	—		1.00	—	—
Length of hospital stay
Less than 3 days	1.10	1.03–1.16	0.0036	0.511	0.86	0.80–0.92	<.0001
3 days or greater	1.00	—	—		1.00	—	—
Most responsible provider service
Family practitioner	1.00	—	—	0.540	1.00	—	—
Other	1.37	1.29–1.46	<.0001		1.19	1.11–1.28	<.0001
Discharge disposition
Transferred to other or same facility	1.04	0.89–1.21	0.6581	0.503	1.24	1.05–1.48	0.0123
Left against medical advice/no return from pass	2.19	1.44–3.32	0.0003		2.62	1.67–4.13	<.0001
Discharged home with/without support	1.00	—	—		1.00	—	—
Number of documented Elixhauser comorbidities
None	1.39	1.25–1.54	<.0001	0.531	0.89	0.79–1.01	0.0694
1 or 2	1.14	1.02–1.27	0.0193		1.00	0.89–1.12	0.9791
3 or more	1.00	—	—		1.00	—	—

The results of the multivariate analysis, which included all predictors in the model, showed that an increased odds of having an overall hospital experience rating of 0–9 of 10 was independently associated with age of 18–59 years (when compared to 70 and older), male gender, being born in Canada, having an urgent admission, not having a family practitioner as the most responsible provider service, and being transferred to another facility or leaving against medical advice/not returning from a pass (compared with being discharged home). As in the univariate analysis, education was linearly related to overall experience. A length of stay of fewer than 3 days was found to be protective factors. The c-statistic for the complete multivariate model was 0.635.

Discussion

The present study is, to our knowledge, the first that examines a comprehensive list of demographic and clinic factors which may be associated with inpatient hospital experience in a Canadian setting. Our results showed a number of associations at the univariate and multivariate levels. Regardless of the method, age (particularly those between 18 and 59 years old) and those with higher levels of education appeared to have an increased odds of having lower ratings of their overall inpatient experience. Patients who were not discharged home (with or without support services), and those who did not have a family practitioner as their most responsible provider also reported lower levels of overall inpatient experience. Although our findings are novel with respect to the health care setting examined (e.g., a Canadian province with universal health coverage), they are in agreement with previous HCAHPS-related literature. Previous works from the United States have consistently shown that younger individuals tend to have a less positive health care experience, as do those with higher levels of education (Elliott et al. 2001; Zaslavsky et al. 2001; Roohan et al. 2003; O’Malley et al. 2005). Although the present study did not specifically examine race or ethnicity, we also found that patients born in Canada had a less positive inpatient experience. This finding is in contrast to previous American literature, which suggests that certain minority groups tend to report less positive experiences with their hospital or health plan (Elliott et al. 2001; Morales et al. 2001; Roohan et al. 2003; Weech-Maldonado et al. 2003, 2004). This contrast may be due in part to our own inclusion/exclusion criteria, in which non-English speakers are excluded from our interviewing protocol.

One result that was somewhat counterintuitive was that medical comorbidities did not have an association with overall patient experience scores. This may be due to the fact that we aggregated both this potential predictor into a count of comorbidities present. In their study, O’Malley and colleagues individually tested 20 different medical conditions and found that circulatory problems had an association with patient experience (O’Malley et al. 2005).

Another potential explanation for our lack of conclusive findings relative to comorbidities is that HCAHPS does not take into account patient expectations. Given that our multivariate c-statistic was 0.635, there is room for improvement with the model, but we may argue that the answer may not lie within administrative data. Recently, Bjertnaes, Sjetne, and Iversen (2012) looked at the association between patient-reported experiences and fulfillment of their expectations of care. They concluded that patient expectations were the second most important predictor of overall experience (after patient-reported experiences with nursing services). Our intention is to explore the potential association of individual comorbidities as well as patient expectations with overall experience in future works.

There are a couple of key strengths to the present study. First, to our knowledge, this is the first peer-reviewed manuscript that employs data linkage to explore factors which may be associated with HCAHPS responses within a Canadian inpatient setting. Given the universal nature of health care services in our jurisdiction (Alberta, Canada), we are able to track patients across different institutions and settings (e.g., emergency department, inpatient visits, etc.). This increased opportunity for data linkage overcomes a significant limitation of organizations that typically utilize HCAHPS. For example, in the United States, a given health care provider may indeed use HCAHPS to capture patient experience; however, the provider in question would have no inclination as to whether a given patient has obtained service elsewhere (e.g., a competing hospital/health care system).

Second, as we have used HCAHPS methodology, we have used a validated tool with a standard script and prompts to assess inpatient experience. This, when compared to many generic or ad-hoc instruments that are developed to measure experience, allows for significant comparison to other jurisdictions, while ensuring validity and repeatability of our results.

There are some limitations to the present study that warrant discussion. First, it is acknowledged that the potential relationships between predictors and the outcome variable may not be a static one throughout the range of possible responses. Simply said, a given predictor will not have the same relationship for an overall response of 10 versus 0–9, 8–10 versus 0–7, etcetera. In addition, the question remains as to how to best define a “successful experience.” For the purposes of this manuscript, we have defined this as an overall experience rating of 10 of 10. For public reporting, Alberta Health Services reports this as a rating of 8 or greater of 10 (Alberta Health Services 2014), while the Health Quality Council of Saskatchewan (a neighboring province) reports this as a rating of 9 or 10 of 10 (Health Quality Council of Saskatchewan 2013). This latter method is the same one used by CMS in the United States, which is often referred to as the “top box” rating.

To examine the potential impact of imposing different “roll-up” scores, we repeated our analyses with an overall rating of (a) 9 or 10 of 10 versus 0–8, as well as (b) 8, 9, or 10 of 10 versus 0–7. For the multivariate analyses, findings relative to sex and the number of comorbidities (persons with 1–2 comorbidities) were no longer statistically significant in both cases. Being born in Canada was no longer a significant predictor for the 8, 9, or 10 of 10 rating scenario. This demonstrates the importance of selecting an appropriate indicator for reporting, given that this data may be used to make meaningful clinical and patient-level decisions. For example, some may feel that 10 of 10 is too high of a goal, as some individuals may never give such a rating, despite receiving exemplary care. These individuals may inherently feel that there is always an opportunity for improvement, and give a 9 of 10. In this case, the use of the HCAHPS “top box” rating would be appropriate.

Secondly, as we have only captured a sample of our provincial population, we would advise caution in extrapolating the results to all patients in our province as well as those in other jurisdictions. The authors advise treating our results as a “snapshot” of inpatient hospital care within our province; one that should not influence decision making in isolation, but rather in combination with other key performance measures. One must also make the important distinction between statistical and clinical significance.

Thirdly, as we have linked our primary data (HCAHPS survey) with administrative data, the strength of the potential associations is subject to the imposed limitations and sources of error of administrative data. These may include, but are not limited to, coding errors, including misclassification, as well as the potential for underreporting certain conditions. Reabstraction studies performed on the DAD data at the national level have shown that agreement for the most responsible diagnosis to be at 94 percent (Canadian Institute for Health Information 2012).

A final limitation stems from the method of survey administration. Previous research has shown that to the mode of survey administration (e.g., mail, phone) may impact the responses that are received on the HCAHPS survey, with telephone respondents typically reporting more positive experiences (de Vries et al. 2005; Elliott et al. 2009). Although the reasons for this are not entirely understood, social desirability is believed to be a contributor. In comparing results from national HCAHPS surveys, CMS applies a mode adjustment to account for response differences from telephone and mail surveys (Centers for Medicare and Medicaid Services 2008).

In conclusion, the results of the present study provide novel results that have produced several questions for future research as well as opportunities for quality improvement within our organization. This preliminary manuscript has identified the association between age, education level, discharge disposition, most responsible provider service, and length of stay with overall inpatient experience in our cohort. To our knowledge, this is the first manuscript of its kind to explore factors that may contribute to overall patient experience in Canada — a setting where universal health care is provided to all citizens. Therefore, the implications for future research have great potential. For example, projects and initiatives that address the factors described above may prove useful in improving patient experience. Given the increased capability of data linkages, further research may also examine the potential cost-benefit, health behaviors (e.g., medication/treatment adherence), and other outcomes (e.g., mortality, hospital readmission rate, quality of life, etc.) that may be related to patient experience in a Canadian setting.

Supporting Information

Additional supporting information may be found in the online version of this article: