Looking Through the Patients’ Eyes

Abstract

Objective:

Patient satisfaction is a personal evaluation of health-care services that is often used as an indicator of quality of care. The aim of this study was to identify aspects of hospital care that affect patient satisfaction by examining the structural and convergent validity of an in-house questionnaire.

Methods:

The sample consisted of 3320 patients discharged from an Italian public hospital. The questionnaire included items exploring communication with nurses and physicians, pain management, quality of accommodation, and discharge information. Data were analyzed using the Rasch model.

Results:

From the patients’ perspective, the number of response options was excessive and the questionnaire proved to have both medical and accommodation dimensions. Patients, on average, gave higher satisfaction scores to the medical dimension over the accommodation dimension. Higher satisfaction was associated with kindness and courtesy of the nursing staff, doctors’ courtesy, and the quality of bed linen.

Conclusion:

The results support the administration of the questionnaire but suggest change in the hospital’s analytical procedures in order to match the drivers of satisfaction as seen by the patients.

Introduction

Patient satisfaction is an important and commonly used indicator of quality in health care (1,2). In a number of countries, including Denmark, Norway, Netherlands, England, Canada, and United States, surveying patient satisfaction is considered an important part of any systematic program of quality assurance undertaken at regular intervals using standardized instruments (3). Health-care managers often use satisfaction ratings to identify areas for delivery improvement and to establish a relationship of trust with patients. Research has demonstrated that satisfied patients are more willing to comply with doctors’ instructions, thereby improving positive health-care outcomes (4 –6). Moreover, in managed health systems, aspects of hospital reimbursement are often directly impacted by patient satisfaction levels (7).

A variety of questionnaires have been developed to assess this important outcome indicator (8). Indeed, mandated assessments might require the use of externally developed instruments, imposed by outside regulatory or systemic bodies. The United States and England have by far the longest tradition of assessing patients’ experiences through the American Consumer Assessment of Healthcare Providers and Systems (CAHPS) surveys and the surveys of the Picker Institute Europe for the English National Healthcare Service (NHS; 3).

But patient satisfaction is not always recorded systematically, nor always included in health-care planning, as many consider those assessments difficult to interpret, being based on a number of implicit assumptions about the nature, and meaning, of expressions of “satisfaction” (9).

In order to assess patient satisfaction, it is important to establish sound psychometric properties for the specific instrument being used. Most of the traditional data analysis techniques routinely used for patient satisfaction analysis (eg, factor analyses or structural equation modeling) require interval level data input for their calculations (10). Unfortunately, the interval nature of the data is almost always presumed, rather than checked or tested empirically. It is clear that responses to Likert scale provide merely ordinal level data (10,11). Modern test theory, including the Rasch model for measurement, was introduced to address potential problems related to the routine use of summary scores that do not take into account the relative importance to patients of the different items comprising any scale (12,13). Although Rasch analysis was developed initially to analyze reading, intelligence, and achievement tests, recently its use has been extended to other areas, such as health-related quality of life and the measurement of patient health-care experience (14 –16).

In Italy, health care is provided to all citizens and residents who receive those services by a mixed public–private provision. The public part, the NHS, is funded by the central government and administered on a regional basis. Italy does not have programs specifically designed to collect information systematically with respect to evaluation of quality of care from the perspective of patients. Those user satisfaction surveys that are adopted in health care are often different, even within the same region, and the general orientation is for each institution to adopt self-made assessment tools (17).

The purpose of this study is to use the Rasch model to identify the aspects that affect satisfaction with hospital experience from the patient’s perspective and to provide evidence for the structural and convergent validity of an in-hospital questionnaire designed to assess patient satisfaction in an Italian public hospital.

Methods

Setting and Patients

Randomly selected patients aged 18 years and over were interviewed within 15 to 20 days of their discharge from a leading teaching hospital in the Emilia Romagna region of northern Italy. Patients discharged from the Neonatal Unit were excluded from the survey. The interviews were conducted via a Computer-Assisted Telephone Interviewing (CATI) system by an external marketing research organization. The target number of interviews was calculated using a quota sample design based on the number of patients discharged in the previous year from each hospital unit. Written informed consent was not required as no unique patient identifier information was collected and participation in the interview was completely voluntary. The survey was undertaken as routine part of the activities of the Research, Innovation, Clinical Governance, and Evaluation division of the health-care performance office of the hospital. These activities, aimed at improving the quality of care, are specifically exempted from notification to the institutional ethics committee as they are regarded as routine audit activities.

Survey Instrument

Data from the Customer Satisfaction Audit (CSA) questionnaire developed specifically for the hospital, and subsequently refined to improve the quality of the wording and patient understanding, were analyzed. The questionnaire comprised 22 items concerning: communication with nurses at hospital admission (2 items), communication with health-care staff during the hospital stay (9 items), pain management (1 item), quality of hospital accommodation (8 items), efforts of health-care staff (1 item), and discharge information (1 item) (see Table A1 [Appendix]). In the first step, patients were asked to respond to the items on an ordered 3-point scale: “It did not meet my expectations,” “It was in line with my expectations,” and “It exceeded my expectations.” Those who responded: “It did not meet my expectations” or “It exceeded my expectations” were then asked to specify to what degree hospital performance had not met/exceeded their expectations using a 5-point scale (from 1 = “unimportant” to 5 = “important”). For the purpose of the analyses, patients’ answers were scored on twice; once considered only the initial answer on 3-point scale and the second used the finer scale of 11 points that included the answer about the importance of the disjunction between hospital performance and patient expectation. The questionnaire also included 2 items assessing overall patient satisfaction and willingness to recommend the hospital to others, as well as an additional section recording sociodemographic patient characteristics—age, sex, level of education, residence, and occupation.

Analysis

The Rasch model adopted for the analyses of these satisfaction data was the Rasch rating scale model (RSM), which is appropriate for analyzing a set of Likert-style items with common response opportunities involving more than 2 ordered response categories (11). With the Rasch model, the response probabilities of each person to each of the individual items are modeled as a logistic function of the latent satisfaction trait (θ). This model yields person (βn) and item (δi) satisfaction estimates, as well as estimates of a set of between response category thresholds common to all items. Item estimates <0 are relatively easy for the sample to endorse, corresponding to higher patient satisfaction; item estimates >0 indicate lower satisfaction.

The threshold estimates for both 3- and 11-category response options were examined in order to verify whether patients actually discriminated between the available ordered response categories.

The scale performance and the dimensionality of the instrument were assessed using both Rasch fit statistics and the results of the principal component analysis (PCA) of those Rasch residuals (18,19). Fit statistics <0.6 indicate items that overfit the model, usually because they share a certain component of meaning with other items. Conversely, fit statistics >1.4 indicate underfitting (or erratic) items whose response patterns are too unpredictable to produce quality measures (20). For the PCA of the Rasch residuals, the criteria used to determine the presence of more than 1 measurement dimension are variance explained by the Rasch model <60%, eigenvalues <3, and variance explained by the first contrast ≥5% (21,22). For each measurement subscale implied by PCA, a t test on mean score was calculated to confirm a potential multidimensional structure in the data (18).

Scale reliability was evaluated using the person separation index (comparable to Cronbach α; 23). Person separation indicates how well the questionnaire is able to differentiate between groups of hospital patients with different levels of satisfaction. A minimally acceptable value for separation is 2.

Convergent validity was assessed by comparing the Rasch person satisfaction estimates with responses to each of the 2 items concerning overall satisfaction judgments. It was hypothesized that mean person estimates would have a positive relationship with overall satisfaction and willingness to recommend the hospital.

Rasch person estimates were used to examine possible associations with patient demographic characteristics. Data are presented as a mean (Standard deviation [SD]) of the person estimates. In order to compare mean estimates, a Wilcoxon-Mann-Whitney test or a Kruskal-Wallis test was calculated, as appropriate, with the significance level set at P < .05. Kolmogorov-Smirnov and Shapiro-Wilk tests were used to verify normal distribution of estimates.

Winsteps software (Winstep 3.80.1) was used to perform Rasch analysis and SAS System version 9.3 (SAS, Cary, North Carolina) for all other analyses.

Results

The study sample consisted of 3320 participants. Mean age was 57 (range: 18-99 years), 57% were females, and 44% were retired. The majority (67%) of patients lived in the urban area, where the hospital was located, or in neighboring areas (Table 1).

Table 1.

Person Satisfaction Measures by Sociodemographic Characteristics for Medical Subscale (MS) and Accommodation Subscale (AS).^a

Patient Characteristics	n (%)	MS (Mean [SD])	P	AS (Mean [SD])	P
Gender
Male	1426 (42.95)	0.45 (2.45)	.006	0.33 (1.83)	<.0001
Female	1894 (57.05)	0.20 (2.51)		0.05 (1.94)
Age
< 65 years	1999 (60.3)	0.43 (2.49)	<.0001	0.12 (1.93)	.006
≥ 65 years	1316 (39.7)	0.13 (2.47)		0.25 (1.85)
Level of education
Degree	475 (14.4)	0.28 (2.45)	<.0001	−0.05 (1.92)	.002
Diploma	1116 (33.9)	0.37 (2.50)		−0.08 (1.89)
School-leaving certificate	862 (26.2)	0.59 (2.52)		0.32 (1.96)
Elementary school-leaving certificate/no study	840 (25.5)	−0.05 (2.42)		0.25 (1.81)
Occupation
Retired	1458 (44.2)	0.13 (2.40)	.02	0.18 (1.83)	.04
Employed	590 (17.9)	0.36 (2.51)		−0.09 (1.90)
Worker/agriculturist/artisan	317 (9.6)	0.58 (2.57)		0.46 (2.15)
Housewife	309 (9.4)	0.48 (2.62)		0.27 (1.97)
Commercial and service activity	299 (9.1)	0.41 (2.50)		0.05 (1.76)
Teacher/professor	121 (3.7)	0.43 (2.50)		0.07 (2.10)
Manager/entrepreneur	64 (1.9)	0.46 (2.58)		0.10 (1.84)
Unemployed	141 (4.3)	0.62 (2.50)		0.23 (1.87)
Residence
Hospital’s city	1386 (42)	0.01 (2.45)	<.0001	0.09 (1.82)	.001
Periphery of the hospital’s city	820 (24.8)	0.17 (2.45)		0.03 (1.83)
Other city	1096 (33.2)	0.79 (2.49)		0.37 (2.04)

Abbreviation: SD, standard deviation

^aHigher measures correspond to higher satisfaction. Measures are expressed in logits (Log odds units).

The threshold measures for each response category in the 11-point response scale were not ordered (respectively: − 0.02, 0.02, 0.01, 0, 0.02, 0.32, 0.57, 0.58, 0.70, 0.87, and 1.06 logits) along the satisfaction measure (Figure 1A). Investigation of the alternative 3-point scale revealed that those response category thresholds were located at −2.78 and 2.78 logits, and the mean satisfaction values for the 3 response categories increased monotonically (− 0.75, 0.55, and 3.08 logits) (Figure 1B) along the variable. These findings indicate that the patients did not use the 11-point scale as the hospital had presumed, but that the 3 response categories were ordered and responded to, as expected.

Figure 1.

A, Probability response curves for items with 11 response categories. B, Probability response curves for items with 3 response categories.

The Rasch model was implemented with all 22 CSA items, but items 13_cleanliness, and 17_food were under fitting, that is their performance was unacceptably unpredictable, denoting violation of the unidimensionality requirement for Rasch measurement (Table 2). The PCA analysis of the residuals posits suggested a 2-dimensional structure for the CSA questionnaire. The variance explained by the Rasch model was 52.3%, and variance explained by the first contrast was 5.9% (2.7 eigenvalue units). On the basis of those first residual component loadings, items were apparently clustered, in Figure 2, into 2 subsets: items with a positive loading reflected a “Medical subscale” (MS), and those with a negative loading, an “Accommodation subscale” (AS). Item “i” (3, privacy), with a loading close to zero, was allocated to the MS because it concerned respect for patient privacy during surgical procedures or medical examinations. The first MS subset includes all 14 items regarding the hospital admission, hospital stay, and discharge sections of the questionnaire, whereas the second AS subset includes 8 items relating to accommodation in hospital.

Table 2.

Rasch Fit Statistics for the Overall Scale (22 Items) and Rasch Item Estimates and Fit Statistics for Medical Subscale (MS) and Accommodation Subscale (AS).

		Overall Scale		MS			AS
No.	Items	Infit mnsq	Outfit mnsq	Item Estimate ± SE	Infit mnsq	Outfit mnsq	Item Estimate ± SE	Infit mnsq	Outfit mnsq
1	Nurses’ kindness	0.96	0.99	−2.59 ± 0.05	1.08	1.07
2	Ward information	0.97	1.00	−1.09 ± 0.05	1.12	1.06
3	Privacy	0.72	0.64	−0.68 ± 0.05	0.89	0.78
4	Consent information	0.8	0.72	−0.81 ± 0.05	0.91	0.78
5	Nurses’ courtesy	0.9	0.85	−2.30 ± 0.05	0.99	0.87
6	Doctors’ courtesy	0.78	0.70	−1.92 ± 0.05	0.81	0.67
7	Doctors’ availability	0.9	0.82	−1.61 ± 0.05	0.97	0.85
8	Nurses’ availability	0.96	0.92	−1.85 ± 0.05	1.09	1.01
9	Nurses’ attention	0.86	0.78	−1.70 ± 0.05	0.95	0.81
10	Doctors’ attention	0.87	0.80	−1.62 ± 0.05	0.89	0.75
11	Pain	0.89	0.83	−1.43 ± 0.05	1.05	0.94
12	Health status information	0.87	0.77	−1.32 ± 0.05	0.94	0.84
13	Cleanliness	1.64a	1.63a				−0.55 ± 0.04	1.42a	1.35
14	Bed	1.22	1.17				−0.53 ± 0.04	1.03	0.95
15	Comfort	1.09	1.03				−0.56 ± 0.04	0.92	0.85
16	Security	0.96	0.89				−0.51 ± 0.05	0.89	0.80
17	Food	1.48a	1.56a				0.49 ± 0.04	1.18	1.17
18	Menu	1.14	1.12				0.06 ± 0.05	0.93	0.85
19	Meal times	0.79	0.78				0.08 ± 0.05	0.67	0.58a
20	Sheet	1.04	0.99				−1.09 ± 0.04	0.84	0.80
21	Staff’s efforts	0.89	0.82	−1.05 ± 0.05	1.10	1.01
22	Discharge information	0.93	0.85	−1.80 ± 0.05	1.02	0.89

Abbreviations: mnsq, mean square; SE, standard error.

^aMisfitting (erratic) items.

Figure 2.

Loading and item difficulty estimate from principal component analysis. Items (upper case): A = 10_Doctors’ attention, B = 6_Doctors’ courtesy, C = 7_Doctors’ availability, D = 12_Health status information, E = 5_Nurses’ courtesy, F = 9_Nurses’ attention, G = 21_staffs’ efforts, H = 8_Nurses’ availability, I = 1_Nurses’ kindness, J = 11_Pain, and K = 22_Discharge information. Items (lower case): a = 19_Meal times, b = 17_Food, c = 18_Menu, d = 14_Bed, e = 20_Sheet, f = 15_Comfort, g = 16_Security, h = 13_Cleanliness, i = 3_Privacy, j = 2_Ward information, and k = 4_Consent information.

Consequently, Rasch analysis was applied again to each of the 2 dimensions separately, and the fit statistics improved. All items in the MS scale showed good item fit statistics, whereas, in the AS scale, 2 items—13_cleanliness and 19_meal times—exhibited borderline misfit statistics (Table 2). Overall, the mean item satisfaction estimates of the 2 subscales were significantly different (MS:−1.55 [0.05] vs AS: −0.33 [0.05]; P = .03), indicating that, on average, patients were considerably more satisfied with the medical aspects of their hospitalization (MS) than with their accommodation (AS).

The satisfaction measures for MS items ranged from − 2.59 logits (ie, most satisfactory: item 1_nurses’ kindness) to − 0.68 (ie, least satisfactory: item 3_privacy), while the satisfaction measures for AS items ranged from − 1.09 (most satisfactory: item 20_sheets) to 0.49 (least satisfactory: item 17_food). From the patients’ perspective, items 3_privacy and 4_consent information proved to be the least satisfied within the MS items, whereas the items related to meals, 17_food, 18_menu, and 19_meal times, were the least satisfied of the AS items. Conversely, higher satisfaction was associated with the kindness of the nursing staff at admission and the courtesy of nursing staff and doctors during the hospital stay (1_nurses’ kindness, 5_nurses’ courtesy, 6_doctors’ courtesy). Within the AS items, the quality of bed linen provided the highest satisfaction level (20_sheets; Table 2).

Person separation was 2.59 for MS and 1.65 for AS. These 2 indices correspond to moderate reliability coefficients of 0.87 and 0.73, respectively. This suggests, there are 3 measurably distinct strata of patients on satisfaction with the MS, but only the minimally acceptable, 2, separate strata for AS.

Overall satisfaction with hospital care was positively related to Rasch person measures. In particular, patients who were less satisfied with the hospital had significantly lower average Rasch satisfaction estimates (Mean [SD]), on both subscales, than did patients who were more satisfied (MS: not satisfied −3.81 [1.97], satisfied −0.91 [1.47], very satisfied 2.24 [2.02]; P < .0001; and AS: not satisfied –1.69 [1.86], satisfied − 0.49[1.25], very satisfied 1.18 [2.06]; P < .0001). Similarly, patients who would recommend the hospital to another needing the same medical service had significantly higher mean Rasch satisfaction estimates than did patients who would not recommend it (MS: 0.49 [2.35] vs −3.86 [1.99]; P < .001; and AS: 0.24 [1.87] vs −1.45 [1.99]; P < .001).

Being male and living at a distance from the city in which the hospital is located were patient characteristics associated with higher levels of satisfaction on both subscales. Whereas older respondents reported being marginally less satisfied on the MS than were younger respondents (0.13 [2.47] vs 0.43 [2.49], P < .0001), they claimed to be more satisfied on the AS (0.25 [1.8] vs 0.12 [1.93], P = .006). Level of education, on average, presented the same pattern as age, with lower levels of education corresponding to older respondents, and higher levels of education associated with younger respondents. There was no clear relationship between occupation and satisfaction, although retired patients were the least satisfied with MS and employed patients were the least satisfied with AS (Table 1).

Discussion

The evidence from this study suggests that the CSA questionnaire has adequate psychometric properties for the hospital’s quality assurance and improvement purposes.

The first result concerns response categories and their thresholds. Unfortunately, as detailed threshold analysis is often omitted in similar research, it remains unclear just how many response categories are needed by hospitals to collect adequate patient satisfaction data. The implicit assumption appears to be: more categories give better precision, and, consequently, improved measures and interpretation. We found that 3 response categories were sufficient to capture patients’ judgments of satisfaction effectively. This response format also has the advantage of requiring a reduced burden on the postdischarge patient and less time for each interview. The implication for those quantifying patient satisfaction is that the performance of the response options needs to be examined and verified empirically.

Another important result concerns the performance of the CSA items.

The PCA of the Rasch fit residuals revealed the presence of 2 dimensions in the CSA from the patients’ perspective: one measuring hospital care (MS) and the other related to hospital accommodation (AS). This is consistent with the literature showing that satisfaction is often a multidimensional construct (24). Overall, the medical scale, MS, performed better psychometrically and showed a greater range of patient satisfaction than did the AS. Our study revealed that, in the patient’s eyes, being treated with courtesy and kindness by nursing staff and the doctor were particularly strong drivers of satisfaction, while communication and explanation about privacy and consent information were the most critical points for lower satisfaction (25). With regard to accommodation aspects, patients reported higher satisfaction with the quality of the room, and it is clear that patients report that the hospital should focus efforts on improving the quality of food. However, the extent to which improving the quality of food is an important aspect of quality hospital care, remains open.

The third result concerns the convergent validity of the CSA questionnaire: patients reporting higher satisfaction on MS and AS dimensions also had higher scores on overall satisfaction and were more inclined to recommend the hospital to others (26).

As reported in several studies, satisfaction differed between older and younger patients (27, 28). We confirmed, further, that men were likely to report being more satisfied than were women, consistent with other findings from the literature (29).

Our results should be interpreted in light of several limitations. One is that the survey was restricted to a single hospital, and most patients were in late middle age; therefore generalization of our results should be made with caution. Second, there might be relevant patient characteristics, for instance living conditions or perception of health status (which were not collected in the present study) that might also impinge on satisfaction with hospital care. As well as the inclusion of psychological and medical variables that inform us on the patient’s personality and the actual patient’s state of health would be useful to better understand the study results. Furthermore, the sample size was insufficient to support robust satisfaction findings stratified by hospital ward.

Although this research supports the ongoing routine administration of the CSA by the hospital, 2 clear recommendations for change are based directly on the analyses of hospital performance from the patient perspective: one for data collection procedures and a second for questionnaire structure and analyses. (a) Although the instrument developers offered a plethora of response options for the telephone survey, patients’ responses reveal that they do not differentiate that number of ordered response categories. Three response categories were recommended to the hospital as a sufficient and parsimonious item format and (b) Instead of reporting a single overall satisfaction indicator, the hospital was recommended to ask the analysis provider to report on 2 satisfaction subscales: AS and MS. Conflating patients’ responses to single overall CSA satisfaction indicators risks the continued confounding of 2 distinctly different aspects of hospital performance in the eyes of its patients. Accommodation scale enhancements might be easier to implement, and thereby, improvements in patients’ AS satisfaction are likely to be easier to gain, but this hospital remains more likely to focus on patient satisfaction with medical aspects (MS), although these are likely to be costly and more difficult to attain. Ethical, medical, and pragmatic issues undoubtedly influence hospital policies, and patient satisfaction will remain one of the key outcomes to pursue, keeping in mind budget constraints and the need to achieve an adequate balance of available resources.

With a view to continuous quality improvement in hospital care, the quality of data, patient response burden, different satisfaction levels by item and by demographic group must remain crucial considerations. Moreover, validity and reliability are not “all or nothing” concepts; rather, they are a matter of degree (30). Evidence on psychometric testing needs to be strengthened over time in order to reduce the margin of error and to reexplore changing in the features of hospital care that are important to patients.

Author Biographies

Elisa Carretta is PhD fellow at the Department of Biomedical and Neuromotor Sciences, University of Bologna in Italy. Her responsibility includes data management and statistical analysis on clinical and epidemiology studies.

Trevor G Bond is an adjunct professor in the College of Arts, Society and Education at James Cook University in Australia. He specialises in the application of the Rasch model across the human sciences, and was the founding Chair of the Pacific Rim Objective Measurement Society.

Giuseppe Cappiello is an assistant professor at the Department of Management, University of Bologna in Italy. His research focuses primarily on three main themes: pricing policies, management of services and services innovation.

Maria Pia Fantini, MD, is professor in Health Services Planning and Organization at the Department of Biomedical and Neuromotor Sciences, University of Bologna in Italy. She is director of the Specialty School in Hygiene and Preventive Medicine.

Appendix A

Table A1.

CSA Questionnaire (Questionnaire Developed by the Hospital to Monitor Patient Satisfaction).

No.	Short Name	Item
1	Nurses’ kindness	Kindness of the nursing staff at admission
2	Ward information	Clarity of the information about ward organization provided at admission
3	Privacy	Respect of privacy during examination or tests
4	Consent information	Completeness and clarity of the information provided by the doctors while asking you to sign the written consent
5	Nurses’ courtesy	Courtesy of the nursing staff during your stay in hospital
6	Doctors’ courtesy	Courtesy of the doctors during your stay in hospital
7	Doctors’ availability	Availability of the doctors in case of need
8	Nurses’ availability	Availability of the nursing staff in case of need
9	Nurses’ attention	Attention of the nursing staff to your questions and needs
10	Doctors’ attention	Attention of the doctors to your questions and needs
11	Pain	Attention of the nursing staff in the control and reduction of your pain
12	Health status information	Completeness and clarity of the information about your health status and treatment provided by the doctors
13	Cleanliness	Room and toilet cleanliness
14	Bed	Comfort of the bed
15	Comfort	Room comfort (change of air, good furniture, room temperature)
16	Security	Ward security (control of access of outsiders)
17	Food	Quality of the food
18	Menu	Food choices on menus that take account of the patient health status and the doctor’s prescription.
19	Meal times	Meal times
20	Sheet	Quality of the bed sheet
21	Staff’s efforts	Staff’s effort in solving the reason of your hospitalization
22	Discharge information	Completeness and clarity of the information reported on the discharge letter and provided verbally about what to do once at home
23	Overall satisfaction	Overall satisfaction with the hospital
24	Recommend the hospital	Would you recommend the hospital to someone who needs the same treatment as yours?

Abbreviation: CSA, Customer Satisfaction Audit.