Comparing NICU teamwork and safety climate across two commonly used survey instruments

Abstract

Background and objectives

Measurement and our understanding of safety culture are still evolving. The objectives of this study were to assess variation in safety and teamwork climate and in the neonatal intensive care unit (NICU) setting, and compare measurement of safety culture scales using two different instruments (Safety Attitudes Questionnaire (SAQ) and Hospital Survey on Patient Safety Culture (HSOPSC)).

Methods

Cross-sectional survey study of a voluntary sample of 2073 (response rate 62.9%) health professionals in 44 NICUs. To compare survey instruments, we used Spearman's rank correlation coefficients. We also compared similar scales and items across the instruments using t tests and changes in quartile-level performance.

Results

We found significant variation across NICUs in safety and teamwork climate scales of SAQ and HSOPSC (p<0.001). Safety scales (safety climate and overall perception of safety) and teamwork scales (teamwork climate and teamwork within units) of the two instruments correlated strongly (safety r=0.72, p<0.001; teamwork r=0.67, p<0.001). However, the means and per cent agreements for all scale scores and even seemingly similar item scores were significantly different. In addition, comparisons of scale score quartiles between the two instruments revealed that half of the NICUs fell into different quartiles when translating between the instruments.

Conclusions

Large variation and opportunities for improvement in patient safety culture exist across NICUs. Important systematic differences exist between SAQ and HSOPSC such that these instruments should not be used interchangeably.

Introduction

Despite a renewed focus on patient safety over the last decade there has been limited progress in ensuring that patients receive safer care in hospitals.^{1 2} Development of a culture of safety is considered to be foundational to achieving safer care.³ The Joint Commission requires that hospitals assess their safety culture on an ongoing basis.⁴ A culture of safety is the shared values, attitudes, perceptions and patterns of behaviour that determine the observable degree of effort with which organisational members direct their attention and actions towards minimising patient harm.⁵

Several survey instruments have been developed to measure safety culture. The Safety Attitudes Questionnaire (SAQ), is one of the two most widely used.⁶ In prior work we used SAQ in a small sample of 12 neonatal intensive care units (NICUs) and demonstrated good psychometric properties and significant variation in safety culture.^{7 8} The other commonly used survey is the Hospital Survey on Patient Safety Culture (HSOPSC).⁹ This instrument has shown good psychometric properties in adult healthcare settings¹⁰ but has not yet been studied in the NICU setting. Head to head comparisons between these instruments are uncommon. Etchegaray and Thomas¹¹ found similar reliability and predictive validity between SAQ and HSOPSC. However, this study was conducted within a single health system, limiting generalisability. Here we expand this literature by comparing these instruments across a large sample of NICUs.

The NICU setting is an excellent domain in which to explore these relations due to the compromised physiological state of patients and the complexity of interventions that are required. Preterm infants are fragile and exposed to complex and prolonged intensive healthcare interventions. Furthermore, care of preterm infants requires the coordination of and transition between prenatal, perinatal and postnatal caregivers from highly subspecialised medical, nursing and ancillary care disciplines. This study evaluates how well the two different survey instruments capture safety culture in this complex environment. The two objectives of this study were,

1. to assess variation and characteristics of safety and teamwork climate in the NICU setting, and

2. to compare NICU performance on safety and teamwork climate between SAQ and HSOPSC.

Methods

Overview

This survey study was performed among a voluntary sample of NICUs participating in a Delivery Room Management Quality Improvement Collaborative organised by the California Perinatal Quality Care Collaborative (CPQCC).¹² For the current study of NICUs, we assembled a survey to investigate safety culture and workforce engagement using existing validated metrics from multiple instruments (detailed below) and combined them with routinely collected hospital and NICU characteristics.

Sample

Of 61 NICUs who participated in a quality improvement initiative organised by CPQCC, 44 accepted an invitation to complete the survey at its onset (between June and September 2011). Of these 44 NICUs, 10 (22.7%) were designated as regional NICUs, 28 (63.6%) as community NICUs and 6 (13.6%) as intermediate NICUs as defined by the California Department of Healthcare Services. These designations are roughly equivalent to designations by the American Academy of Pediatrics as levels IV, III and II, respectively.¹³

Staff with 0.5 full-time equivalent or more in the NICU for at least the four consecutive weeks prior to survey administration was invited to participate. Paper-based surveys were administered during regular staff meetings, together with a pencil and sealable return envelope to maintain confidentiality. Individuals not captured in pre-existing meetings were hand-delivered a survey, pencil and return envelope. The CPQCC organised administration of the survey and linked the data with routinely collected organisational data.

Measures

Patient safety culture

The SAQ is associated with clinical outcomes,^14–17 and contains 30 items that load on six domains: teamwork climate, safety climate, job satisfaction, perceptions of management, stress recognition and working conditions with response scales ranging from 1 (disagree strongly) to 5 (agree strongly). Here we report on the safety and teamwork climate scales. SAQ also captures respondent characteristics including job position, years in specialty, gender and predominant work shift. Job position included attending physicians (Medical Doctors; MDs), fellow MDs, neonatal nurse practitioners, registered nurses, respiratory care practitioners and others.

HSOPSC¹⁸ is composed of 42 items that load on 12 domains. In order to limit respondent burden or the current study, we included four domains: Overall Perceptions of Safety, Teamwork within Units, Communication Openness, and Error Feedback and Communication. For consistency, and to save room on the paper-based survey administration, the HSOPSC items were scaled using the SAQ Likert options. This differed slightly from the HSOPSC standard, which uses several Likert options (eg, Never/Rarely/Sometimes/Most of the times/Always), but did not change the phrasing of the items. The change to a standard response scale improved the internal consistency reliability of the scales when compared with the literature.¹⁹

Organisational characteristics

We had access to the following organisational characteristics. NICU level was defined according to the 2012 American Academy of Pediatrics (AAP) Policy Statement¹³ and obtained in 2012 as part of a survey of NICU directors. Hospital ownership was also obtained from this survey. Annual number of NICU admissions, number of NICU beds and hospital teaching status were derived from the California Department of Public Health's Regional Perinatal Program of California data set.

Statistical analysis

Objective 1—Variation and characteristics: We used descriptive analyses such as frequencies, percentages, means (±SD) and graphs to describe respondent characteristics and organisational characteristics. Safety and teamwork climate scale and item scores were calculated for individual NICUs by taking the average of the scaled items, and then calculating the percentage of respondents within a NICU who reported positively (ie, proportion of those who agreed slightly or strongly).⁵ Scale reliability was measured via Cronbach's α. One sample two-tailed t tests were performed on the NICU-level scale scores.

Objective 2—Comparison between SAQ and HSOPSC: We assessed similarity in the rankings between SAQ and HSOPSC using Spearman's rank correlation coefficient. In addition, paired sample t tests were used to compare teamwork scales between instruments and safety scales, as well as conceptually similar items. We then ranked NICUs by their scale-level performance (per cent positive, ie, the per cent of respondents in a NICU reporting good climate) and classified them into quartiles. We then determined how often NICUs changed performance quartiles using the two instruments. Finally, we evaluated the effect of respondent and hospital characteristics on teamwork and safety climate scale scores using hierarchical multivariate regression models.

Results

Sample

Of 3294 administered surveys in the 44 NICUs, 2073 were returned for an overall response rate of 62.9%. NICU response rates ranged from 21.7% to 100% with an average of 69.7% (SD=19.8%), and the mean number of respondents per NICU was 47.1 (SD=24.7), ranging from 10 to 144. Table 1 lists respondent and hospital and unit characteristics of our sample. The majority of respondents were nurses (72%) and female (85%). A third of respondents had more than 20 years work experience. NICUs represented were of high acuity (88% level III or IV) and of medium to small size (96% <100 beds).

• Objective 1—Variation and characteristics of safety and teamwork climate

Table 1.

Respondent and neonatal intensive care unit (NICU) characteristics

	N (%)
Respondent characteristics
Sex (n=2003)
Male	305 (15)
Female	1697 (85)
Primary shift (n=1868)
Days	894 (48)
Evenings	79 (4)
Nights	602 (32)
Variable	293 (16)
Job position (n=2041)
Fellow physician	31 (2)
Physician	204 (10)
Neonatal nurse practitioner	35 (2)
Registered nurse	1464 (72)
Respiratory care provider	286 (14)
Other	21 (1)
Work experience (n=1970)
Less than 6 months	20 (1)
6–11 months	27 (1)
1–2 years	74 (4)
3–4 years	192 (10)
5–10 years	476 (24)
11–20 years	538 (27)
More than 20 years	643 (33)
Hospital and unit characteristics
Urban location (n=2072)
No	128 (6)
Yes	1944 (94)
Level of care (n=2072)
Level II	248 (12)
Level III	1151 (56)
Level IV	673 (32)
NICU beds (n=1924)
Less than 50	1153 (60)
50–99	694 (36)
More than 99	77 (4)
Annual NICU admissions (n=1947)
Less than 500	945 (49)
500–999	714 (37)
More than 999	288 (15)

Figure 1 shows a box and whisker plot of NICU performance on the safety and teamwork climate scales of SAQ and the four safety culture scales of HSOPSC. Respondent and NICU-level variability by scale and item are exhibited in table 2. Internal reliability was good for all scales, with Cronbach's α ranging from 0.71 to 0.86. With SAQ, on average, about two-thirds of the respondents in a given NICU reported good teamwork climate (66%) and good safety climate (65%). Yet, there is opportunity for improvement with regards to appropriately resolving disagreements, creating a culture that makes it easy to discuss and learn from errors.

Figure 1.

Safety and teamwork climate scales (Safety Attitudes Questionnaire, SAQ and Hospital Survey on Patient Safety Culture, HSOPSC) across 44 California neonatal intensive care units (NICUs). Box and whisker plots representing the mean, SD, and range per cent positive response for each scale.

Table 2.

Item-level descriptive results

Items	%Neg	%Neutral	%Pos
SAQ
Safety climate—per cent positive NICU means (SD; range) 65.2 (12.8; 33–95); Cronbach's α=0.81
The culture in this NICU makes it easy to learn from the errors of others	16.7	19.1	64.2
Medical errors are handled appropriately in this NICU	6.2	9.8	84.0
I know the proper channels to direct questions regarding patient safety in this NICU	3.2	6.8	90.0
I am encouraged by others in this NICU, to report any patient safety concerns I may have	7.4	12.2	80.4
I receive appropriate feedback about my performance	12.3	15.5	72.3
I would feel safe being treated here as a patient	7.0	11.7	81.4
It is difficult to discuss errors*	24.8	16.5	58.7
Teamwork climate—per cent positive NICU means (SD; range) 65.8 (13.8; 36–100); Cronbach's α=0.80
It is easy for personnel here to ask questions when there is something they do not understand	7.1	7.5	85.3
I have the support I need from others in this NICU to care for patients	5.6	7.3	87.1
Nurse input is well received in this NICU	11.4	11.9	76.7
It is difficult to speak up if I perceive a problem with patient care*	19.0	11.0	70.0
Disagreements in this NICU are resolved appropriately	16.6	20.9	62.5
The physicians and nurses here work together as a well coordinated team	10.8	10.2	79.0
HSOPSC (adapted)
Overall perceptions of safety—per cent positive NICU means (SD; range) 56.3 (15.4; 18–93); Cronbach's α=0.75
Patient safety is never sacrificed to get more work done	21.4	14.8	63.8
Our procedures and systems are good at preventing errors from happening	7.5	16.5	76.0
It is just by chance that more serious mistakes don't happen around here*	15.9	17.7	66.5
We have patient safety problems in this NICU*	16.2	16.1	67.7
Teamwork within units—per cent positive NICU means (SD; range) 74.0 (13.1; 44–100); Cronbach's α=0.86
People support one another in this NICU	9.7	11.7	78.6
When a lot of work needs to be done quickly, we work together as a team to get the work done	3.7	5.5	90.8
In this unit, people treat each other with respect	13.9	14.5	71.5
When one area in this NICU gets really busy, others help out	6.4	8.3	85.3
Communication openness—per cent positive NICU means (SD; range) 49.3 (12.7; 25–71); Cronbach's α=0.71
Staff will freely speak up if they see something that may negatively affect patient care	8.3	12.5	79.2
Staff feel free to question the decisions or actions of those with more authority	21.4	18.8	59.8
Staff are afraid to ask questions when something does not seem right*	23.3	17.1	59.6
Error feedback and communication—per cent positive NICU means (SD; range) 49.2 (14.4; 18–77); Cronbach's α=0.80
We are given feedback about changes put into place based on event reports	18.4	26.6	55.0
We are informed about errors that happen in this NICU	23.1	18.9	58.0
In this NICU, we discuss ways to prevent errors from happening again	11.3	13.8	74.9

The per cent positive (% pos) results include the range and overall mean per cent positive, which is the per cent agreement (agree slightly plus agree strongly) within a given NICU. The per cent negative (% neg) results include the range and overall mean per cent negative, which is the per cent disagreement (disagree slightly plus disagree strongly) within a given NICU.

*Item reverse coded.

HSOPSC, Hospital Survey on Patient Safety Culture; NICU, neonatal intensive care unit; SAQ, Safety Attitudes Questionnaire.

With HSOPSC, on average, slightly more than half reported good overall perceptions of safety (56%). Teamwork within units was the scale with the highest scores with three quarters of respondents reporting good teamwork (74%). About half of respondents reported good communication openness (49%) and error feedback communication (49%), mirroring item-level concerns regarding these areas in SAQ. T tests demonstrated significant variation across NICUs (p<0.001) in each one of the teamwork and safety scales on SAQ and HSOPSC.

• Objective 2—Comparison of safety and teamwork climate scales between SAQ and HSOPSC

We examined the degree to which SAQ and HSOPSC instruments provide similar information on safety and teamwork climate. NICU-level correlation coefficients for the SAQ and HSOPSC scale score per cent positives suggest that they are related but distinct metrics. Safety scales (safety climate and overall perception of safety) and teamwork scales (teamwork climate and teamwork within units) of the two instruments correlated strongly (safety r=0.72, p<0.001; teamwork r=0.67, p<0.001).

Figure 2 plots NICU-level performance for the safety climate and teamwork scales for both instruments. For each scale, using the two instruments resulted in similar NICU-level performance. However, for several NICUs performance differed strongly. For example, in comparing safety climate scales, one NICU, which ranked 23rd on HSOPSC, would have ranked 3rd on SAQ. Another, which ranked 25th on HSOPSC, would have ranked 40th on SAQ. Similar findings are true for the teamwork scale.

Figure 2.

Per cent positive responses for safety (A) and teamwork (B) climate by NICU for the SAQ and HSOPSC instruments. Both figures are sorted by increasing performance on HSOPSC. While there is concordance for most NICUs on the two instruments, some (examples noted with an asterisk) have widely differing performance. SAQ, Safety Attitudes Questionnaire; HSOPSC, Hospital Survey on Patient Safety Culture; Safe Clim, Safety Climate; TW Clim, Teamwork Climate; OPS, Overall Perceptions of Safety; TW, Teamwork Within Units; NICU, neonatal intensive care unit.

To further examine the relation between NICU performance on the two instruments, we ranked NICUs by quartiles of teamwork climate and teamwork within units. We found that 23 (52%) fell into different quartiles of teamwork between the instruments, 5 of which were off by two or more quartiles. Similarly, for NICUs ranked by quartiles of safety climate and overall perceptions of safety, we found that 21 (48%) fell into different quartiles of safety between the instruments, 5 of which were off by two or more quartiles.

So about half the time NICUs were classified to a different quartile, based on the use of the most similar scale from a different instrument—the quartile disconnects were larger still between teamwork climate and communication openness and between safety climate and ‘non-punitive response to error’.

Table 3 shows comparisons between SAQ and HSOPSC safety and teamwork climate scales and select conceptually similar items from the two instruments; all comparisons showed significant differences (p<0.002). We investigated potential reasons for these findings based on respondent or hospital characteristics. Table A in the eAppendix shows results of hierarchical models created for each of the SAQ and HSOPSC safety and teamwork scales. This table reveals significant associations of respondent and hospital characteristics with these scales. Women, non-physician providers and those with work experience exceeding more than 1 year exhibited lower safety culture ratings. With regards to hospital characteristics, larger NICUs exhibited lower safety and teamwork climate ratings. Overall though, both safety culture instruments exhibited similar patterns, providing little explanation for the significant differences in scale scores as a result of hospital and respondent demographics.

Table 3.

Comparisons of SAQ and HSOPSC teamwork and safety climate scales and select items

Variable (SAQ/HSOPSC)	r Value	Paired samples t test	Mean per cent positives	p Value
Safety climate/overall perceptions of safety	0.72	5.47	65.24/56.35	<0.001
Safety climate/non-punitive response to error	0.72	10.28	65.24/49.34	<0.001
Teamwork climate/teamwork within units per cent positive	0.67	−4.95	65.88/74.10	<0.001
Teamwork climate/communication openness	0.53	8.51	65.88/49.35	<0.001
The culture in this NICU makes it easy to learn from the errors of others./Our procedures and systems are good at preventing errors from happening.	0.73	−10.48	64.90/77.46	<0.001
I have the support I need from others in this NICU to care for patients./People support one another in this NICU.	0.70	7.54	87.98/79.99	<0.001
It is easy for personnel here to ask questions when there is something that they do not understand./Staff will freely speak up if they see something that may negatively affect patient care.	0.64	5.76	85.77/79.82	<0.001
In this NICU, it is difficult to speak up if I perceive a problem with patient care./REVERSE Staff will freely speak up if they see something that may negatively affect patient care.	0.49	−6.92	70.32/79.82	0.001
I would feel safe being treated here as a patient./Reverse coded We have patient safety problems in this NICU.	0.47	6.95	81.44/69.65	<0.001
It is easy for personnel here to ask questions when there is something that they do not understand./REVERSE Staff are afraid to ask questions when something does not seem right.	0.35	17.50	85.77/58.96	<0.001
The physicians and nurses here work together as a well-coordinated team./When a lot of work needs to be done quickly, we work together as a team to get the work done.	0.34	−6.05	79.88/91.41	<0.001
Medical errors are handled appropriately in this NICU./We are informed about errors that happen in this NICU.	0.23	11.26	84.26/59.30	<0.001

HSOPSC, Hospital Survey on Patient Safety Culture; NICU, neonatal intensive care unit; SAQ, Safety Attitudes Questionnaire.

Discussion

This study of NICU safety and teamwork climate included more NICUs and providers than previous studies combined^{8 11 20 21} and contributes to the literature by directly comparing hospital performance across the two most commonly used safety culture surveys, SAQ and HSOPSC. While we found areas of congruence between the instruments, we highlight important distinctions that warrant avoiding direct comparisons for internal or external performance assessments.

For internal comparisons, this study provides a rough but practical translation from one safety culture instrument to the other. Although switching instruments is not encouraged, it is relatively common and leaves organisations with a lack of continuity and a current year of data that can be misleading when compared with prior years using a different instrument. Progress in quality and safety is often assessed through safety culture instruments and instrument transitions leave quality and safety professionals blind to the relative changes that are taking place, while providing an unfortunate reason to minimise the usefulness of current results.

We found that switching instruments translates to a change in quartiles for about half of the NICUs, and that about half of the variance in one scale is not accounted for in the conceptually similar scale on the other instrument. Moreover, every cross-instrument comparison whether scale level or item level, yielded t tests that were significant and often quite large. Simply stated, switching from SAQ to HSOPSC would appear to cause your per cent positive teamwork to improve over 8 points, while your per cent positive safety norms would appear to decrease by 9 points.

Conceptually similar item comparisons across instruments mirrored the scale score results. Inspection of item content did not reveal obvious sources in the variation of these associations. For example, it is unclear why the correlation coefficient of r=0.49 for the two seemingly similar ‘speak-up’ items fails to account for three-fourths of the variance in each other (r² provides the per cent of the variance in one variable accounted for by the other). Perhaps it is due to the changing valence of the items, subtle phrasing differences, or the respondent interpretation, that cause these seemingly similar items to elicit such different results from the same respondents. In addition, commonly measured respondent and hospital characteristics did not explain these findings.

Etchegaray and Thomas¹¹ had previously conducted a head to head analysis of SAQ and HSOPSC in a single healthcare system. Correlations between safety and teamwork scales between the two instruments were higher in this study compared with the study by Etchegaray et al (Safety Climate 0.78 vs 0.63; Teamwork 0.70 vs 0.52). This difference may be in part explained by the more homogenous sample consisting only of NICUs in our study, or by our transformation of HSOPSC response scales to fit SAQ's. We have previously reported that this transformation improved HSOPSC's psychometric properties for these scales.¹⁹

In addition to the direct comparisons between instruments, our findings also affirm and expand the current literature on safety culture assessments in the NICU setting. We found significant variation in safety culture between NICUs and opportunity for improvement across all measured dimensions of safety culture.

An understanding of safety culture is important because it has been linked to clinical and operational outcomes,^22–27 as well as burn-out among NICU staff.²⁸ In addition, effective interventions to improve safety culture exist^{19 29} and have proven sustainable.³⁰ In previous work, we demonstrated significant variation and opportunity for improvement in safety culture among a small (n=12) group of NICUs⁸ and highlighted lower perceptions of safety culture among nurses. Here, we replicated these findings but found two notable differences to our prior work.

First, safety climate ratings in this sample of NICUs were about 6–10% higher than in the previous sample across all domains of SAQ. Since the two samples don't overlap, this finding may be due to differences in samples (the sample in this study self-selected to participate in a quality improvement initiative), time trends or repeat administration of the survey. Mean SAQ scores among participants who noted prior completion of the survey did not differ from those who completed it for the first time (71.6 vs 72.4, p=0.34), making this explanation less likely.

Second, safety culture ratings tended to decrease as the number of admissions and beds goes up, during evening shifts and after the 1st year on the job. These findings mirror those of Sexton et al²⁶ who describe a similar relation with regards to hospital size among 127 adult ICUs who participated in the keystone project. This association is concerning, considering the recent trend of consolidation of hospital systems in the wake of the Affordable Care Act.³¹

Our findings must be viewed in light of the study design. Survey studies may be subject to bias. We used the well established administration procedures by Sexton et al⁶ to assure adequate response rates. In addition, our transformation of HSOPSC response scales to fit SAQ may change the behaviour and meaning of these scales. However, we have found improved psychometric properties without rewording the actual questions. Finally, our voluntary sample may have been biased in that these NICUs had signed up to participate in a quality improvement collaborative. Alas, whether this suggests better or worse than average safety culture ratings is not certain given that our collaboratives attract some NICUs that are particularly challenged in the clinical improvement topic for which they sign up. In addition, any bias could be expected to affect ratings on both survey instruments equally.

Conclusion

Safety culture assessments among NICUs vary widely. SAQ and HSOPSC yield strongly correlated results and the scales elicit conceptually similar content. Nevertheless, in this one-to-one comparison of the instruments, we failed to find a single set of scales or items that yielded interchangeable results. We recommend caution in translating and or transitioning between instruments.