Is Scores Derived from the Most Internationally Applied Patient Safety Culture Assessment Tool Correct?

Abstract

Background

Hospital Survey on Patient Safety Culture, known as HSOPS, is an internationally well known and widely used tool for measuring patient safety culture in hospitals. It includes 12 dimensions with positive and negative wording questions. The distribution of these questions in different dimensions is uneven and provides the risk of acquiescence bias. The aim of this study was to assess the questionnaire against this bias.

Methods

Three hundred nurses were assigned into study and control groups randomly. Short form of HSOPS was distributed in the control group and totally reversed form of it was given to the study group. Percent positive scores and t-test were applied for data analysis. Statistical analyses were conducted using SPSS Version 16.

Results

Finally a total of 272 nurses completed the questionnaire. All dimensions with positive wording items in both groups had higher scores compared with their negative worded format. The first dimension “organizational learning and continued improvement” which had the only statistically significant difference, got 16.2% less score in the study group comparing the other group. In addition six out of 18 differences in questions were statistically significant.

Conclusion

The popular and widely used HSOPS is subject to acquiescence bias. The bias might lead to exaggerate the status of some patient safety culture composites. Balancing the number of positive and negative worded items in each composite could mitigate the mentioned bias and provide a more valid estimation of different elements of patient safety culture.

Introduction

Safety and quality issues in health care has gained lots of attentions since the beginning of 21th millennium (1) and many countries around the world has initiated different initiatives in this important element of patient care. It is believed that beside making structural interventions, hospitals should create a Patient Safety Culture (PSC) among their staff in order to improve quality and safety in health care (2), because an inappropriate culture is the biggest challenge to moving towards patient safety (3). Safety culture is defined as “the product of individual and group values, attitudes, perceptions, competencies, and patterns of behavior that determine the commitment to, and the style and proficiency of, an organization’s health and safety management” (4). Knowing the current status of each dimension of patient safety culture in health facilities is a major step towards improving it and for this reason PSC assessment has become a top priority for health systems especially in developed countries (5). Health policy makers, managers and the related officials rely on these data to recognize areas of strength and also areas of potential for improvement in their organization’s patient safety culture, and to make appropriate decisions for establishing a good culture.

Several self-report questionnaires have been developed to measure patient safety culture in hospitals (6–9). Hospital Survey on Patient Safety Culture (HSOPS) introduced by the US Agency for Healthcare Research and Quality (AHRQ) is one of these instruments (10). It measures safety culture from a hospital staff perspective and has 42 questions in positive and negative wordings (24 and 18 respectively), which covers seven unit-level composites of safety culture, three hospital level composites and four outcome variables (the questionnaire details is explained further in the methods section). Since the release of this questionnaire in November 2004, there has been a growing interest in using it among different countries around the world. As of February 2012, the number of HSOPS international users includes 45 countries, with 24 different translations (11).

This survey like most of the other studies of attitudes across countries relies on a comparison of aggregated mean scores to Likert-scale questions. These kinds of studies presuppose that “when respondents complete a questionnaire, their answers are based on the substantive meaning of the items to which they respond. “However, people’s responses are also influenced by their response styles” (12). Response styles refer to a respondent’s tendency to systematically answer to questionnaire items regardless of question content (13). Studies have shown that there are major differences in response styles between countries base on their power distance, collectivism, uncertainty avoidance, extraversion and so on, which all significantly influence response styles such as acquiescence and extreme response styles (12). Acquiescence bias is an important type of measurement error that is the tendency of some respondents to agree with a statement independent of its content. Schuman and Scott defined acquiescence or agreeing-response bias as “a presumed tendency for respondents to agree with attitude statements presented to them”(14). The harm of this kind of bias has been documented by questionnaire design researchers. Weiksner says” three findings are relevant: First, agree/disagree questions are highly susceptible to acquiescence bias. Second, the bias is substantial. “Third, acquiescence increases when respondents are less motivated to optimize their answers” (15). Weiksner in his study also concluded that acquiescence bias causes significant measurement error (15). Respondents are 10-20% more likely to agree than disagree with contentless statements (16). Moreover, 10-15% of study participants will agree with both an item and its opposite but few people disagree with both (15). These findings show that if items with positive wording in a questionnaire or each composite of them are more than negative ones, results will tend to be better than actual level of the studied phenomenon. These situations are quite different when negative wording items exceed the positive ones, because in this case, agreement will indicate an undesirable outcome.

As far as we are aware, most of the studies which had used HSOPS for measuring patient safety culture, has found “organizational learning-continued improvement” and “teamwork within units” among the highest scored patient safety culture dimensions, while “non punitive response to error”, “staffing” and “hospital handoffs and transitions” has got the lowest scores (1, 17–25). It is interesting to know that all of the questions in the two higher scored composites have positive wordings, while 10 items out of 11 ones in the three lower scored dimensions have negative wordings. These observations magnify the possibility of acquiescence bias in HSOPS and create a question in minds that: Is the well known and highly used Hospital Survey on Patient Safety Culture a biased questionnaire?

The present study aimed to assess the existence of this important bias in HSOPS.

Material & Methods

Participants

This cross-sectional study was conducted on nurses from two general teaching hospitals (Emam Khomeini and Shariati) in Tehran, Iran - in the second half of 2010. These hospitals were selected purposefully according to their size, units, and also appropriate cooperation with the study team. Nurses were chosen as the study population, because they have an important role in delivering health services and also their good participation and response rates in patient safety culture surveys (1, 26–28). Three hundred nurses (out of 887 nurses in the study population) from seven hospital units were selected and divided in two groups randomly (each group contained 150 nurses). Group A was the study group, while B was the control group.

The questionnaire

HSOPS covers seven unit-level composites of safety climate, three hospital level composites and four outcome variables:

Unit-level aspects of safety culture

• Supervisor/Manager Expectations & Actions Promoting Safety (4 items- 2 positive and 2 negative wording questions),

• Organizational Learning—Continuous Improvement (3 items- all of them have positive wordings),

• Teamwork Within Units (4 items- all of them have positive wordings),

• Communication Openness (3 items- 2 positive and 1 negative wording questions),

• Feedback and Communication About Error (3 items- all of them have positive wordings),

• Non punitive Response to Error (3 items- all of them have negative wordings), and

• Staffing (4 items- 1 positive and 3 negative wording questions).

Hospital-level aspects of safety culture

• Hospital Management Support for Patient Safety (3 items- 2 positive and 1 negative wording questions),

• Teamwork Across Hospital Units (4 items- 2 positive and 2 negative wording questions), and

• Hospital Handoffs and Transitions (4 items- all of them have negative wordings).

Outcome variables

• Overall Perceptions of Safety (4 items- 2 positive and 2 negative wording questions),

• Frequency of Event Reporting (3 items- all of them have neutral wordings),

• Patient Safety Grade (of the Hospital Unit) (1 item), and

• Number of Events Reported (1 item) (10).

The five-point Likert is utilized for the responses in agreement and frequency scales (strongly disagree, disagree, neither, agree, or strongly agree / never, rarely, sometimes, most of the time, or always). This questionnaire had been translated in Farsi and validated for applying in Iranian hospital setting prior to this study (26).

At the first stage, in order to explore the possibility of acquiescence bias existence in HSOPS, Five dimensions of the questionnaire (including 18 positively and negatively worded questions) which had the highest and lowest scores in most of the related studies, and also unequal distribution of the mentioned question wordings were selected (“organizational learning and continued improvement”; “teamwork within units”; “non punitive response to error”; “staffing”;“hospital handoffs and transitions”). Other seven composites related questions were deleted from the questionnaire to make it shorter and easier to response. HSOPS designers indicated that if somebody wants to shorten this questionnaire, they can remove some dimensions and all the questions which are related to them, and in this way their hospital’s results on the remaining composites still can be compared to other hospitals that use the survey (10). This shortened questionnaire was distributed in the control group (group B). The totally reversed form of the mentioned short questionnaire was given to the study group (group A). In this questionnaire each positive item were transformed into negative and vice versa. A panel of four experts, who were familiar with safety culture issues, reversed each item in a way that the validity of the questionnaire would not be negatively affected. After that the reversed form of the short questionnaire were distributed among 20 nurses from a general teaching hospital which were not in our study. That nurses were again asked to complete the questionnaire two weeks after the first distribution. After comparing the two round results, Chronbach Alpha coefficient were between 0.84 and 0.66, which showed good reliability of the reversed questionnaire.

Data analysis

Percent positive scores and t-test were applied for data analysis. For obtaining percent positive scores, all of the agreements (strongly agree and agree) with positively worded questions and disagreements (strongly disagree and disagree) with negatively worded items were calculated and then divided by the total number of respondents to that item. Statistical analyses were conducted using SPSS Version 16.

Results

Finally a total of 272 nurses had successfully completed the questionnaire (total response rate: 90.66%). Totally, 131 respondents were belonged to the study group, while the number of participants in the control group were 141 (response rate for each group were 87.33% and 94% respectively).

Professional characteristics

The majority of study participants in both groups were working in medicine and surgery units (43.5% for group “A” and 46.1 for group “B”). 45.8% of respondents in the group “A” and 48.2% in the other group reported having more than 6 years of experience at their current hospital, while having such an experience at current unit, was quite different in both groups (12.2% for group “A”, and 17% for “B”). About 64% of respondents in group “A” were working more than 40 hours a week in hospital, while near to 71% of their counterparts in the second group fell in this category (Table 1).

Table 1.

The professional characteristics of the respondents

Professional features	Study group (A)	Control group (B)
	Work Area/Unit
Medicine	27 (20.6)*	31 (22)
Surgery	30 (22.9)	34 (24.1)
Obstetrics	21 (16)	23 (16.3)
Psychiatry	20 (15.3)	23 (16.3)
Intensive care unit	15 (11.5)	17 (12.1)
Emergency	18 (13.7)	13 (9.2)
Experience in current hospital
Less than 1 year	23 (17.6)	7 (5)
1-5	30 (22.9)	41 (29.1)
6-10	18 (13.7)	25 (17.7)
11 years or more	60 (45.8)	68 (48.2)
Experience in current hospital work area/unit
Less than 1 year	39 (29.8)	18 (12.8)
1-5	53 (4.5)	72 (51.1)
6-10	23 (17.6)	27 (19.1)
11 years or more	16 (12.2)	24 (17)
	Working hours per week
Less than 20 hours	5 (3.8)	3 (2.1)
20-39 hours	42 (32.1)	38 (26.9)
40-59 hours	76 (58)	88 (62.4)
More than 60 hours	7 (5.3)	12 (8.5)

* The numbers in brackets indicates percent

Patient safety culture scores

Percent positive scores in the selected dimensions are shown in Table 2. In this table both questionnaires which had been applied for the control and study groups are seen. The control group (group B) had higher total percent positive scores in “Organizational learning and continued improvement” (dimension 1) and “teamwork within units” (dimension 2) compared with the study group (group A) (71.2% VS 55%, and 69.2% VS 63.6% respectively). This situation was quite reversed about the other three dimensions, where group “A” got better scores in comparison with the other group (18.8% VS 16.1%, 25.5% VS 21.8%, and 52.3% VS 49.1%).

Table 2.

Percent positive scores in each dimension and question in both groups

Short original HSOPS	Question wording	Positive scores group B (%)	short Reversed HSOPS	Question wording	Positive scores group A (%)
1- Organizational culture continued imp rovement		71.2			55*
A6. We are actively doing things to improve patient safety	+	84.4	We aren’t actively doing things to improve patient safety	–	69.5*
A9. Mistakes have led to positive changes here	+	54.6	Mistakes have not led to positive changes here	–	39.7*
A13. After we make changes to improve patient safety, we evaluate their effectiveness	+	74.5	We don’t evaluate the effectiveness of changes which had made to improve patient safety	–	55.7*
2- Teamwork within units		69.2			63.6
A1. People support one another in this unit	+	78.7	People don’t support one another in this unit	–	67.2*
A3. When a lot of work needs to be done quickly, we work together as a team to get the work done	+	67.4	When a lot of work needs to be done quickly, we rarely work together as a team to get the work done	–	67.1
A4. In this unit, people treat each other with respect	+	89.4	In this unit, people don’t treat each other with respect	–	83.2
A11. When one area in this unit gets really busy, others help out	+	41.1	When one area in this unit gets really busy, others don’t help	–	35.9
3- Non punitive response to error		16.1			18.8
A8. Staff feel like their mistakes are held against them	–	15.6	Staff feel like nobody hold their mistakes against them	+	18.3
A12. When an event is reported, it feels like the person is being written up, not the problem	–	17.7	When an event is reported, it feels like the problem is being considered, and the person isn’t being written up.	+	20.6
A16. Staff worry that mistakes they make are kept in their personnel file		14.9	Staff don’t worry that mistakes they make are kept in their personnel file	+	17.6
4- Staffing		21.8			25.5
A2. We have enough staff to handle the workload	+	16.3	We have not enough staff to handle the workload		11.5
A5. Staff in this unit work longer hours than is best for patient care	-	22	Staff in this unit work adequate hours, that is best for patient care	+	20.6
A7. We use more agency/temporary staff than is best for patient care	–	32.6	We use enough agency/temporary staff that is best for patient care	+	44.3*
A14. We work in “crisis mode” trying to do too much, too quickly	–	16.3	We work in “normal mode” and there aren’t too much work to do too quickly	+	25.2*
5- Hospital handoffs and transitions		49.1			52.3
F3. Things “fall between the cracks” when transferring patients from one unit to another	–	35.4	Nothing “fall between the cracks” when transferring patients from one unit to another	+	36.6
F5. Important patient care information is often lost during shift changes	–	61.7	Important patient care information is rarely lost during shift changes	+	71
F7. Problems often occur in the exchange of information across hospital units	–	41.8	Problems rarely occur in the exchange of information across hospital units	+	40.5
F11. Shift changes are problematic for patients in this hospital	–	57.4	Shift changes are not problematic for patients in this hospital	+	61.1

* Significantly different t test at p- 0.05

The highest difference in the composite percent positive scores was found in the first dimension, where respondents in the control group gave 16.2% more positive scores than their counterparts in the study group (71.2% VS 55%). The lowest difference was detected in the third dimension- “non punitive response to error”- where group “A” had 2.7% more positive scores than the other group. The maximum difference in the questions positive scores was related to A13- “After we make changes to improve patient safety, we evaluate their effectiveness”- in the first dimension, where participants in the control group gave 18.8% more positive scores than their peers in the study group (74.5% VS 55.7%). The minimum difference was belonged to A3- “When a lot of work needs to be done quickly, we work together as a team to get the work done”- in the second composite, where participants in the group “B” gave only 0.3% more positive scores compared with the other group (67.4% VS 67.1%).

Totally all selected positively worded questions (8 questions) in the original HSOPS got lower percent positive scores when they were changed into negative. On the other hand, all but 3 negatively worded questions (10 questions) had higher scores when they reworded into positive. These three exceptions were A2 and A5 from the “staffing”, and F7 from the “hospital handoffs and transitions” composites. Analysis showed that only 5 out of these observed 18 differences were statistically significant.

Discussion

This study examined the existence of acquiescence bias in the popular and widely used Hospital Survey on Patient Safety Culture (HSOPS). The questionnaire uses Likert scale with ordered scale anchors (“strongly disagree” to “strongly agree”). This anchor might be at risk of acquiescence bias as respondents are tend to agree (12). Moreover, observations showed that in the majority of studies around the world, negative dimensions had considerably lower scores compared with positive composites (1, 17–25). Also a study on about one thousand staff in Iranian hospitals which hasn’t published yet, showed similar results. Although there are exceptions, for example in Turkish study the positive composite -”organizational learning and continued improvement”- was among lower scores dimensions, while the negative composite -”hospital handoffs and transitions”- got the third highest score among all dimensions (28), or in studies which had been conducted in Lebanese and Egyptian hospitals, “hospitals handoffs and transitions” achieved the fourth highest score out of 12 dimensions(29, 30), these exceptions are minimal. The present study results showed that reversing the wording of questions could influence responses. All dimensions with positive wording items in both groups had higher scores compared with their negative worded format. This result confirmed our initial hypothesis that acquiescence bias is responsible for a part of HSOPS scores in some dimensions. Although further analysis showed that only one of the observed differences between group “A” and “B” in composites (“organizational learning and continued improvement”), and 6 out of 18 differences in questions (A1, A6, A7, A9, A13 and A14 in Table 2) were statistically significant, it is still important to pay particular attention to the impact of this bias on all unequally distributed positive and negative worded items composites.

It is difficult to assess which amount of scores is caused by an acquiescence bias and which amount truly shows a strong opinion about the patient safety culture (12). However, balancing the number of items with positive and negative wording in each dimension could attenuate the impact of acquiescence bias on composite scores and provide a more accurate estimation of the status of each patient safety culture element. Valid estimation of PSC status in each composite could help managers and related officials to detect strengths and potential for improvement areas of patient safety culture and compare them with other hospitals in national and international level and then make appropriate decisions towards improving this culture.

“regardless of what remedy is used to eliminate or alleviate response bias, the first step towards finding a solution is acknowledging that response bias can be a serious threat to valid comparisons across countries”(12).

Finally, the results of this study showed that the popular and widely used HSOPS is subject to acquiescence bias. This is because of the unequal distribution of positive and negative worded questions in different dimensions. The bias might lead to exaggerate the status of some patient safety culture composites. In dimensions with more positively worded questions, it might contribute to more percent positive scores, while in other dimensions which the number of negatively worded questions exceeds the positive ones, it could lead to less percent positive scores, below the real status of patient safety culture in those dimensions. Also the effect of this bias varied on different questions. In some items the difference between the studied groups was more than 10 percent, while in some other questions there were trivial and negligible differences. Considering that the prevalence of acquiescent answers is varied among countries based on their different cultures, it is important to be cautious in cross cultural comparisons of HSOPS results. Balancing the number of positive and negative worded items in each composite could mitigate the mentioned bias and provides a more valid estimation of different elements of patient safety culture and consequently allows more sound comparisons among countries.

Study limitations

Several limitations to this study need to be acknowledged. First of all, although 300 is not a small sample size, it seems that a larger scale study with different health care staff could better investigate this bias in HSOPS and also provides a more representative evidence for it. secondly, all participants in this study were nurses and almost all of them were female, while studies has been revealed that some demographic characteristics like gender, level of education and socioeconomic status affect response styles (e.g. males have been shown to have a higher level of acquiescence than females) (31–33). It means that if a relatively equal gender distribution had been applied, the observed differences between group A and B might have increased. Thirdly, in this study there were two different randomly assigned groups (study and control groups) which two mentioned sets of questionnaire were distributed among them in one round. Although this method is an approach towards comparing different things, it might be exposed to probable individual differences and also it could not show the real amount of acquiescence bias effect on results. Using two round of questionnaire distribution with an interval period of about one month in the study and control groups, where applying the short form of HSOPS for both groups in the first round, and in the next round giving the reversed form only to the study group, while applying the same first round questionnaire to the control group, may be a better option for investigating the bias in HSOPS.

Ethical considerations

Ethical and scientific approval was obtained from the Deputy of Research of the School of Public Health, Tehran University of Medical Sciences (TUMS). Participants were informed of the aims of the study and assured confidentiality. Verbal consents were obtained from all participants before asking them to complete the questionnaires.

Ethical issues (Including plagiarism, Informed Consent, misconduct, data fabrication and/or falsification, double publication and/or submission, redundancy, etc) have been completely observed by the authors.